:class:`~tornado.queues.Queue` example - a concurrent web spider
.. currentmodule:: tornado.queues
Tornado's tornado.queues module implements an asynchronous producer / consumer pattern for coroutines, analogous to the pattern implemented for threads by the Python standard library's queue module.
A coroutine that yields Queue.get pauses until there is an item in the queue. If the queue has a maximum size set, a coroutine that yields Queue.put pauses until there is room for another item.
A ~Queue maintains a count of unfinished tasks, which begins at zero. ~Queue.put increments the count; ~Queue.task_done decrements it.
In the web-spider example here, the queue begins containing only base_url. When a worker fetches a page it parses the links and puts new ones in the queue, then calls ~Queue.task_done to decrement the counter once. Eventually, a worker fetches a page whose URLs have all been seen before, and there is also no work left in the queue. Thus that worker's call to ~Queue.task_done decrements the counter to zero. The main coroutine, which is waiting for ~Queue.join, is unpaused and finishes.
.. literalinclude:: ../../demos/webspider/webspider.py