Sphinx is a rate limiting HTTP proxy implemented in Go built using leaky buckets.
The name for this project "Sphinx" comes from the ancient Greek word sphingien, which means "to squeeze" or "to strangle." Sphinx would stand by the road and stop passers by. She would then ask them a riddle. If they could not answer, she would strangle them. Sphinx was thought of as a guardian often flanking the entrances to temples.
Rate limiting API's is often required to ensure that clients do not abuse the available resources and that the API is reliably available when multiple clients are requesting data concurrently. Buckets can be created based on various parameters of an incoming request (eg. Authorization, ip address etc) to configure how requests are grouped for limiting.
Rate limiting functionalities are available in some proxies (eg. Nginx, HAProxy) but they often use in-memory stores that make predictable rate-limiting really hard when running multiple proxies for load balancing or reliability. The configuration for setting up these limits also gets complex since it includes many actions including routing, request/response re-writing, rate-limiting.
-
This service is NOT focused on preventing Denial of Service (DoS) attacks or requests from malicious clients. The goal is to expose rate limiting information to clients and enforce balanced use by API clients.
-
Sphinx only allows for very simplistic forwarding. This would primarily be forwarding to a single host per instance of the rate limiter. Any advanced routing or request handling should be handled by a real proxy (eg. Nginx, HAProxy).
-
Sphinx does not currently listen over HTTPS, this keeps the burden of configuring SSL certificates and security outside of Sphinx. Hopefully there is real load balancing and HTTPS termination before a request hits Sphinx.
Sphinx will update HTTP response headers for requests that match limits to include details about the rate limit status. Headers are canonicalized, but clients should assume header names are case insensitive.
- X-RateLimit-Reset: Unix timestamp when the rate limit counter will be reset.
- X-RateLimit-Limit: The total number of requests allowed in a time period.
- X-RateLimit-Remaining: Number of requests that can be made until the reset time.
- X-RateLimit-Bucket: Name of the rate-limit bucket this request belongs to in the configuration.
limit names and response body can be set in the Sphinx Configuration.
Request:
HOST example.com
GET /resource/123
AUTHORIZATION Basic ABCD
Response Headers:
Status: 200 OK
X-RateLimit-Limit: 200
X-RateLimit-Remaining: 199
X-RateLimit-Reset: 1394506274
X-RateLimit-Bucket: authorized-users
In case your application hits a rate limit, a HTTP Status Code 429 Too Many Requests with an error message will be returned.
Request:
HOST example.com
GET /resource/123
AUTHORIZATION Basic ABC
Response Headers:
Status: 429 Too Many Requests
X-RateLimit-Limit: 200
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1394506274
X-RateLimit-Bucket: authorized-users
Response body:
{
"error": "API rate limit reached."
}
Sphinx is built and tested against Go 1.2.
Ensure this is the version of Go you're running with go version.
Make sure your GOPATH is set, e.g. export GOPATH=~/go.
Clone the repository to a location outside your GOPATH, and symlink it to
$GOPATH/src/github.com/Clever/sphinx.
If you have gvm installed, you can
make this symlink by running the following from the root of where you
have cloned the repository: gvm linkthis github.com/Clever/sphinx.
If you have done all of the above, then you should be able to run
make
If you'd like to see a code coverage report, install the cover tool
(go get code.google.com/p/go.tools/cmd/cover), make sure $GOPATH/bin
is in your PATH, and run:
COVERAGE=1 make
If you'd like to see lint your code, install golint (go get github.com/golang/lint/golint) and run:
LINT=1 make