Crawlab Python SDK

Python SDK for Crawlab

Installation

pip install crawlab-sdk

Usage

CLI

crawlab-cli

Scrapy Integration

In settings.py, add the following:

ITEM_PIPELINES = {
    'crawlab.CrawlabPipeline': 300
}

Development

Install dependencies

pip install -r requirements.txt

Compile gRPC

# Set the environment variable CRAWLAB_PROTO_PATH to the path of the gRPC proto files
export CRAWLAB_PROTO_PATH=/path/to/grpc/proto/files

# Compile gRPC to Python code
./compile_grpc.sh