05_module_parallelization.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:ed90984b5d193aa4b34195f088463624484294ef0e1b2ce9b748527b9b763340"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Module parallelization"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Parallelization of your module could be the solution to obtain the results in an acceptable amount of time.\n",
      "But you have to be careful to identify the true bottleneck of your module. \n",
      "Other scientific tasks like for example numerical modelling commonly look as follows:\n",
      "                          \n",
      "    +------------------+                        +---------------+\n",
      "    | input data       |                        |               |\n",
      "    | with the initial +----------------------->| final results |\n",
      "    | condition        |    long computation    |               |\n",
      "    +------------------+        tasks           +---------------+\n",
      "\n",
      "In GIS, we are commonly dealing with rather simple chained computations that are applied to (massive) input data requiring several heavy partial results before to obtain the result:\n",
      "\n",
      "    +------------+\n",
      "    |            |       +---------+    +---------+    +---------+\n",
      "    | input data +------>| partial +--->| partial +--->| partial +\n",
      "    |            |       | results |    | results |    | results |\n",
      "    +------------+       +---------+    +---------+    +----+----+\n",
      "                                                            |\n",
      "                                                            |\n",
      "    +---------------+    +---------+    +---------+    +---------+\n",
      "    |               |<---+ partial |<---+ partial |<---+ partial |\n",
      "    | final results |    | results |    | results |    | results |\n",
      "    |               |    +---------+    +---------+    +----+----+\n",
      "    +---------------+                                        \n",
      "\n",
      "With respect to other parallelization tasks, in GIS processing often the main bottleneck is the maximal hard disk read/write speed occuring in the data intensive computation steps.\n",
      "Before starting to use all cores of your CPU, be sure to check if you have already saturated the read/write capabilites of your system. This can e.g. be done with a system monitor tool which generates small graphs of the resource consumption."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "ParallelModuleQueue class"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A simple way to execute several modules in parallel (developed by S\u00f6ren Gebbert) is the ParallelModuleQueue class.\n",
      "The basic idea is to create a queue with all the modules that must be execute in parallel. The ParallelModuleQueue class is based on the Module class of the pygrass library, here a small example for viewshed calculation:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# import the necessary libraries\n",
      "from copy import deepcopy\n",
      "\n",
      "from grass.pygrass.modules import Module, ParallelModuleQueue\n",
      "from grass.pygrass.vector import VectorTopo"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# define the global variables with the inputs\n",
      "TMP_VIEWSHED = 'tmp_viewshed_{:03}'\n",
      "ELEV = 'elevation'\n",
      "POINTS = 'view_points'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# we set the region to elevation map\n",
      "region = Module('g.region', raster='elevation')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# initialize an empty queue and list\n",
      "queue = ParallelModuleQueue(nprocs=4)\n",
      "viewsheds = []"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# initialize a module instance with shared inputs\n",
      "viewshed = Module('r.viewshed', input=ELEV, observer_elevation=3,\n",
      "                  run_=False, overwrite=True)\n",
      "\n",
      "# open the input vector map and start cycling over the vector points\n",
      "with VectorTopo(POINTS, mode='r') as points:\n",
      "    for point in points:\n",
      "        # create a copy of the module and set the remaining parameters\n",
      "        print(point.id)\n",
      "        m = deepcopy(viewshed)(output=TMP_VIEWSHED.format(point.id),\n",
      "                               coordinates=point.coords())\n",
      "        viewsheds.append(m)\n",
      "        queue.put(m)\n",
      "    queue.wait()\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "viewsheds"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}