Scylla

An intelligent proxy pool for humanities, only supports Python 3.6. Key features:

• Automatic proxy ip crawling and validation
• Easy-to-use JSON API
• Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
• Could get started with only 2 commands minimally
• Straightforward programmable API
• (Additional) Headless browser crawling

Get started

Install with Docker (highly recommended)

Install directly via pip

pip install scylla
scylla # Run the cralwer and web server for JSON API

Use from source

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or yarn install
npm build # or yarn build

python -m scylla

Programmatically

For more details, please read the API Documentation.

Roadmap

• Basic RESTFul API
• Advanced RESTFul API (Sorting, paging, conditions)
• Programmable API
• Docker integration
• Smart web proxy server
• More statistical views of proxies
• Quality statistics for different providers
• Over 20 data sources

Development and Contribution

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or yarn install

Testing

pip install -r tests/requirements-test.txt
pytest -n 10 tests

License

Apache License 2.0. For more details, please read the LICENSE file.