diff --git a/demos/webspider/webspider.py b/demos/webspider/webspider.py index 97cdd62dd4..34fbdb5042 100644 --- a/demos/webspider/webspider.py +++ b/demos/webspider/webspider.py @@ -1,16 +1,3 @@ -"""A trivial web-spider that crawls all the pages in http://tornadoweb.org. - -``spider()`` downloads the page at `base_url` and any pages it links to, -recursively. It ignores pages that are not beneath `base_url` hierarchically. - -This function demonstrates `queues.Queue`, especially its methods -`~queues.Queue.join` and `~queues.Queue.task_done`. -The queue begins containing only -`base_url`, and each discovered URL is added to it. We wait for -`~queues.Queue.join` to complete before exiting. This ensures that -the function as a whole ends when all URLs have been downloaded. -""" - # start-file import HTMLParser import time diff --git a/docs/guide/queues.rst b/docs/guide/queues.rst index a4bf97c5e9..12726f486f 100644 --- a/docs/guide/queues.rst +++ b/docs/guide/queues.rst @@ -14,12 +14,13 @@ until there is room for another item. A `~Queue` maintains a count of unfinished tasks, which begins at zero. `~Queue.put` increments the count; `~Queue.task_done` decrements it. -In the web-spider example here, when a worker fetches a page it parses the -links and puts new ones in the queue, then calls `~Queue.task_done` to -decrement the counter once. Eventually, a worker fetches a page whose URLs have -all been seen before, and there is also no work left in the queue. Thus that -worker's call to `~Queue.task_done` decrements the counter to zero. The main -coroutine, which is waiting for `~Queue.join`, is unpaused and finishes. +In the web-spider example here, the queue begins containing only base_url. When +a worker fetches a page it parses the links and puts new ones in the queue, +then calls `~Queue.task_done` to decrement the counter once. Eventually, a +worker fetches a page whose URLs have all been seen before, and there is also +no work left in the queue. Thus that worker's call to `~Queue.task_done` +decrements the counter to zero. The main coroutine, which is waiting for +`~Queue.join`, is unpaused and finishes. .. literalinclude:: ../../demos/webspider/webspider.py :start-after: # start-file