advanced-python/14-Generators.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generators\n",
    "Generators are functions that can be paused and resumed on the fly, returning an object that can be iterated over. Unlike lists, they are lazy and thus produce items one at a time and only when asked. So they are much more memory efficient when dealing with large datasets.  \n",
    "A generator is defined like a normal function but with the `yield` statement instead of `return`.\n",
    "```python\n",
    "def my_generator():\n",
    "    yield 1\n",
    "    yield 2\n",
    "    yield 3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execution of a generator function\n",
    "Calling the function does not execute it. Instead, the function returns a generator object which is used to control execution. Generator objects execute when `next()` is called. When calling `next()` the first time, execution begins at the start of the function and continues until the first `yield` statement where the value to the right of the statement is returned. Subsequent calls to `next()` continue from the `yield` statement (and loop around) until another `yield` is reached. If `yield` is not called because of a condition or the end is reached, a `StopIteration exception` is raised:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting\n",
      "3\n",
      "2\n",
      "1\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-3941498e0bf0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;31m# this will raise a StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcd\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m: "
     ]
    }
   ],
   "source": [
    "def countdown(num):\n",
    "    print('Starting')\n",
    "    while num > 0:\n",
    "        yield num\n",
    "        num -= 1\n",
    "\n",
    "# this will not print 'Starting'\n",
    "cd = countdown(3)\n",
    "\n",
    "# this will print 'Starting' and the first value\n",
    "print(next(cd))\n",
    "\n",
    "# will print the next values\n",
    "print(next(cd))\n",
    "print(next(cd))\n",
    "\n",
    "# this will raise a StopIteration\n",
    "print(next(cd))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting\n",
      "3\n",
      "2\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# you can iterate over a generator object with a for in loop\n",
    "cd = countdown(3)\n",
    "for x in cd:\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting\n",
      "6\n",
      "Starting\n",
      "[1, 2, 3]\n"
     ]
    }
   ],
   "source": [
    "# you can use it for functions that take iterables as input\n",
    "cd = countdown(3)\n",
    "sum_cd = sum(cd)\n",
    "print(sum_cd)\n",
    "\n",
    "cd = countdown(3)\n",
    "sorted_cd = sorted(cd)\n",
    "print(sorted_cd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Big advantage: Generators save memory!\n",
    "Since the values are generated lazily, i.e. only when needed, it saves a lot of memory, especially when working with large data. Furthermore, we do not need to wait until all the elements have been generated before we start to use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "499999500000\n",
      "8697464 bytes\n"
     ]
    }
   ],
   "source": [
    "# without a generator, the complete sequence has to be stored here in a list\n",
    "def firstn(n):\n",
    "    num, nums = 0, []\n",
    "    while num < n:\n",
    "        nums.append(num)\n",
    "        num += 1\n",
    "    return nums\n",
    "\n",
    "sum_of_first_n = sum(firstn(1000000))\n",
    "print(sum_of_first_n)\n",
    "import sys\n",
    "print(sys.getsizeof(firstn(1000000)), \"bytes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "499999500000\n",
      "120 bytes\n"
     ]
    }
   ],
   "source": [
    "# with a generator, no additional sequence is needed to store the numbers\n",
    "def firstn(n):\n",
    "    num = 0\n",
    "    while num < n:\n",
    "        yield num\n",
    "        num += 1\n",
    "\n",
    "sum_of_first_n = sum(firstn(1000000))\n",
    "print(sum_of_first_n)\n",
    "import sys\n",
    "print(sys.getsizeof(firstn(1000000)), \"bytes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Another example: Fibonacci numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 1, 2, 3, 5, 8, 13, 21]\n"
     ]
    }
   ],
   "source": [
    "def fibonacci(limit):\n",
    "    a, b = 0, 1 # first two fibonacci numbers\n",
    "    while a < limit:\n",
    "        yield a\n",
    "        a, b = b, a + b\n",
    "\n",
    "fib = fibonacci(30)\n",
    "# generator objects can be converted to a list (only used for printing here)\n",
    "print(list(fib))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator expressions\n",
    "Just like list comprehensions, generators can be written in the same syntax except with parenthesis instead of square brackets. Be careful not to mix them up, since generator expressions are often slower than list comprehensions because of the overhead of function calls (https://stackoverflow.com/questions/11964130/list-comprehension-vs-generator-expressions-weird-timeit-results/11964478#11964478)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\n",
      "4272\n"
     ]
    }
   ],
   "source": [
    "# generator expression\n",
    "mygenerator = (i for i in range(1000) if i % 2 == 0)\n",
    "print(sys.getsizeof(mygenerator))\n",
    "\n",
    "# list comprehension\n",
    "mylist = [i for i in range(1000) if i % 2 == 0]\n",
    "print(sys.getsizeof(mylist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept behind a generator\n",
    "This class implements our generator as an iterable object. It has to implement `__iter__` and `__next__` to make it iterable, keep track of the current state (the current number in this case), and take care of a `StopIteration`. It can be used to understand the concept behind generators. However, there is a lot of boilerplate code, and the logic is not as clear as with a simple function using the `yield` keyword. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "499999500000\n"
     ]
    }
   ],
   "source": [
    "class firstn:\n",
    "    def __init__(self, n):\n",
    "        self.n = n\n",
    "        self.num = 0\n",
    "        \n",
    "    def __iter__(self):\n",
    "        return self\n",
    "    \n",
    "    def __next__(self):\n",
    "        if self.num < self.n:\n",
    "            cur = self.num\n",
    "            self.num += 1\n",
    "            return cur\n",
    "        else:\n",
    "            raise StopIteration()\n",
    "             \n",
    "firstn_object = firstn(1000000)\n",
    "print(sum(firstn_object))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pythonengineer_env]",
   "language": "python",
   "name": "conda-env-pythonengineer_env-py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}