notebooks/21_Intro_DeepLearning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 21 -  Introduction to Deep Learning\n",
    "\n",
    "by [Alejandro Correa Bahnsen](albahnsen.com/)\n",
    "\n",
    "version 0.1, May 2016\n",
    "\n",
    "## Part of the class [Practical Machine Learning](https://github.com/albahnsen/PracticalMachineLearningClass)\n",
    "\n",
    "\n",
    "\n",
    "This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US)\n",
    "\n",
    "Based on the slides and presentation by [Alec Radford](https://www.youtube.com/watch?v=S75EdAcXHKk) [github](https://github.com/Newmu/Theano-Tutorials/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this class you must install theno\n",
    "\n",
    "```pip install theano```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Motivation\n",
    "\n",
    "How do we program a computer to recognize a picture of a\n",
    "handwritten digit as a 0-9?\n",
    "\n",
    "![1](d1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What if we have 60,000 of these images and their label?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from load import mnist\n",
    "X_train, X_test, y_train2, y_test2 = mnist(onehot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y_train = np.argmax(y_train2, axis=1)\n",
    "y_test = np.argmax(y_test2, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],\n",
       " [0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],\n",
       " [0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[1].reshape((28, 28)).round(0).astype(int)[:, 4:26].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pylab import imshow, show, cm\n",
    "import matplotlib.pylab as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def view_image(image, label=\"\", predicted='', size=4):\n",
    "    \"\"\"View a single image.\"\"\"\n",
    "    plt.figure(figsize = (size, size))\n",
    "    plt.imshow(image.reshape((28, 28)), cmap=cm.gray, )\n",
    "    plt.tick_params(axis='x',which='both',  bottom='off',top='off', labelbottom='off')\n",
    "    plt.tick_params(axis='y',which='both',  left='off',top='off', labelleft='off')\n",
    "    show()\n",
    "    if predicted == '':\n",
    "        print(\"Label: %s\" % label)\n",
    "    else:\n",
    "        print('Label: ', str(label), 'Predicted: ', str(predicted))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADtCAYAAABTTfKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfel22lzTbDNPnuL4fe7//r54BDMac37klFIq9RbgJI63\n1LXWXhI4MRhU6qm6d+dwOFggEMgH3X/9BgKBwHkI0gYCmSFIGwhkhiBtIJAZgrSBQGYI0gYCmaFf\n98NOpxP1oEDgH+FwOHS858PSBgKZIUgbCGSGIG0gkBmCtIFAZgjSBgKZIUgbCGSGIG0gkBmCtIFA\nZgjSBgKZIUgbCGSGIG0gkBmCtIFAZgjSBgKZIUgbCGSGIG0gkBmCtIFAZgjSBgKZIUgbCGSGIG0g\nkBmCtIFAZgjSBgKZIUgbCGSGIG0gkBmCtIFAZgjSBgKZIUgbCGSG2m1BAs1Fp9MpHXHOz/PS57x/\nb2Z2OByKo557j71/r78nUEaQtmUA0brdboWY3W7Xer1e7VEXP29m9v7+bvv93t7f3ysLz6eO7+/v\nJTLr48BPBGlbBCZrioCDwcD6/X5lDQYD6/V6xWPv3Mzs7e3N3t7ebL/fl46ptdvtSv+HSd7pdEpk\nDvxEkLZlYIvKq9vt2mAwsOFwWBx1DQaD0tLnDoeD7Xa75Nput8XSx9vttiAuW2Czny5yp9MJ4v5/\nBGlbhk6nUxCVrWSv17PhcGjj8dhGo1HyOBwOS0c+PxwOtt1ubbPZVI681uu1rdfr4rzX6xU3k7e3\nt+JoZiVXOfATQdoWQWNXdYdBzul0apPJpFh4PB6PbTwel855HQ6HgpDeWq1WxVoul7ZarQrCwopy\ncutwOFi32y1c5bC0PxGkbRmYtByvDgaDgpDT6dRms1llTafTgsDe8XA4lAipx9fXV3t9fbXhcFhY\nd5ARcSsTlmPbwC8EaVsETkCxi4z4dDQa2WQysdlsZpeXl3ZxcVE5emTGOhwOBTG9NR6PizgYhDX7\nmXF+e3srXGB2iff7fZBWEKRtKLT+amaFOwyCarw6nU7t8vKytEDYU0hrZm7CCgvWVctHuJn0+/0i\nzt1sNmZmQVwHQdqGwau94nwwGCRdW1jY2WxWIqcSFfEsyNnv94vXQAwK6z0ajWy/35diUSTC+v1+\nybpPp1N7fX215XJZHFH7DdKWEaRtGFI12E6nU8SsTMaLi4tiIWZNLU5AjUajws0FudhiDofDonTD\nP2OXnBNfs9nMFouFzefz4vceDgfb7/e22WyK1wgEaRuFY8omuMBwea+urkprNpuVssOaMUaJB2Ue\ntbRmVhJpgLBeDM2EXa1Wtl6vbTqd2mAwKAj69vZm2+3WBoNBWFpCkLZhYOvKMSRKOrC0V1dXdnNz\nU1oXFxeV2qtHVI5bcVNQ9xjJJHaHOZ7ebDY2mUxK9dvJZFJyiTebja1Wq+LGEPiJIG2D4NVhWYoI\n0l5cXBSkvb29te/fv9vt7a3NZrMKMfnoSRiZUPzaeAyr+/b2VhBWVVE4TiaTwiXebre2Wq1ssVhY\nv98PS0sI0jYMHmlhFeEeo6QD0t7d3dnd3Z1dXFy4umMmq9co4Flajl+5YYC1xrpGo1HhEoOw4/E4\nSCsI0jYMnIhi8QRnatXS3t3d2X///WcXFxelGNjr8Em16elre+132tmj58PhsETY5+fnIuEV7vEv\nBGkzhFeDNbMKQXUh4YQklB5ns1mldspEPOe9aX8twF07er7f7wvllVdaQkZZf2/b5I1B2szg9cLi\nMeqwvJD5nUwmdn19bbe3t3Zzc2OXl5dFtng4HJYSSl4DvEKJ4hHH+zdqfTnrrPJKlI546e/gntvU\n+2gagrQZIVWDxeKY1dMPX11d2bdv304mLV5TiasESREm9VjFFkAqFmfPQRvrteeWf19TCRykzQyp\nfther1cST3hSxKurK7u+vrbr62u7urqy6XRq4/HYFUmkrGzdOBn+ed259zfVNTJwO6AXD6deq6md\nQUHajOBd3HyRs3gC8StIen197Yr/J5NJYWlPsbJmx2c81RFYY14+8g2I43MsZJfhEUAiyf222jDf\nROIGaTODdumwGB8KIxDz5uamcIe/fftml5eXlfa6lKX1jgol7yluszcMzrsZ4W9S9xgN8pzoQqkJ\nGuWmT7oI0mYG7+LGhc06XpR0vn37Zt+/f7fv37+XFE/QD8PtZNLyaymOTVZMkRdE8m4Mx1oG2T3W\nJnlYWTzPhG0qcYO0GSEV++HCRkwL9xjZ4u/fv9v//vc/u7i4KFkwPvdIq1B391gWVx9zppgfa4LN\ni2lBXK9JHhaWf9ZkFzlI+8WQqsF2Op1SO5s3w4mbALxaLMQT3jqlDquE9MakMoH1yDpktqhMZm4o\n4KTacrk0M3NnTiHGfXt7q7yu3kyagCDtP4S6n3W9sKjD8vwmneV0dXVlt7e3RdKJE01sTVlAUVeL\n9aAWzlM58VRFPu90OpXpjjjniZDj8bhw8bfbbZEhnkwmtfOnIJHU1waZm0LcIO0/gpfkUTVSXR3W\nW5eXl0XiCQontNSxHlgVT+dArSwIAl1x3ep2u5VhcGZWKJ2UtEzYXq9XtPFh5hSfD4dD22w2lZGt\nu92uZH2bgCDtP0Aqe+r1wrL7ytlhHgMD11fdYRZPqODfUz+dAiUsW7PdbldyW9WV7Xa7NpvNbLfb\nFSUaZMDhPiMLPpvNSoQdDoc2m82KyRY8d+r19dUGg0ExpgYLfxNmUDUFQdpPhlcDrcug8hEusdZh\ncQRReUQMW1rPyp7rGrOlZTcU3TnogeWRqTjv9XqlAW6wrJiZzI8xcwqEnU6ntlgsigWJJhoKuEzE\nUx7hAZzrUXxlBGn/ATzCqtKJL0KceyUdXpeXl+68Yh1Z6sXLp0ITUGxld7tdQVCe9YRztOmZlQkL\nIsPqsts8HA6LZvmLiwubz+dFfZkb89krYcLiZtIkBGn/ITxxvlfOQemDLe3l5aVdX1/bt2/f7Pb2\n1m5vb0t1WJ04oaTl1z8Hp1ra19fXkmVcLBZFczwTdjKZFMomZI/H43ExHmcymRQ3heVyWRAWCSz+\nu7gt8P39vYhreVxrExCk/Uc4hbBap0Ssx3VYkJab2HVsKc5V2KDnx+CVepi4iGlB2vl8bvP53F5e\nXuzl5aXk7oOQyPiye4yblZZvVqtVibBefVkt7Hq9/lDC7SsjSPuJ0ESTnqtkT/fNYdG/V4+dzWaV\neBjn5/TEnvJ3eGALzCSGy9zv90vJIt54Cwkt3MCwCx97A/1+v3KT8NZmsymNxwnSBj4MXIxqAdm6\nsLxQF+uJkXiaTqelcaY6DPxPXqxaisINAaWU3W5Xcud1ODliTZAbxINbzW4uv5bKH1mk4X2O/Lof\nCQG+OoK0nwhcbKnJEjxTmGuZeIwEFCwtZhVzdlgJ+7eIy1Mk8Pxut6vdTYBLMF6ZyBtzczgcimMq\nYcdSTn7tv3Hj+goI0n4ikGhhcb+nbPJ2p+MkFJd1uFOH3eDfUT2loKRh0na73QppU+TxLC1iT457\nD4dDIXNkS6uWvs7SfqS09dURpP1EsH7Ym/SPuuq5S6f9f7QOe+rfAMLgMTwIuMc6wZHdVM08q3u8\n3+8LsQW/nj72JlyolQ9LG/htsHsMS8sif7i7sKpqiVMbO+tOdL9Thz3lb8DfwQRCiaXOPeaGA3aP\nQdput1tkjfFa3W63tLUIW1pvlpRn5YO0gQ/Ds7RILkEvrPvn8L6wiF3VkmmW9HdrsXXvH24q98fC\nemoSSokL8qFc5LnH+lr7/b6U7DpmaT3CNo24QdpPhMa0IC36Xq+vr0sbOGtDAGqtaklTrvDfuFCZ\nuGbldj3sIlBnabE8S8suNz4rFvqnatpcy05lrZuEIO1fAls6ADI93S2O6686TZHJOxgMal9TJ0bA\nsnk/80bB4P2mZJb6t+Gcra7+3V5XEBOVvQR+P1zW8YirCTfOOGsSrmkI0v5hpGLKTqfjlne8rLFK\n9M4RRXDTd11TunduZiUrpYuJiKNaWixtkYOIoi6bjOQU1FG4yaHbhz9j73ETCeohSPsH4QkDeKXq\nslz+wc80C3rqBckE4KP2vvLikS26K95wODQzq4gjvKVaZCXtbrdLCi08wkLuqJa27QjS/mFoooRj\nKxZVeJYWpNXY7FQ3zyOPEqlu4cbCgg6zX7G4vo5ab4+w2pCu8TD/PwxoY8JyK5/3GbSRxEHaPwxc\ndF4/rGdhlbieDPBc99gjkFo+77zb7dp0OnV7XgElLC/vRgAri8W/R5VR3J6HLTFx4+FBcPxZtxFB\n2j8IT2bHZQjP0jJpoWzidY57rLpekIYJqhMl+LlUkzpIw1DCplxjfW28T05K4eeHw6GUqMN754Qa\nf9ZtRZD2D0PLEVxH9GJadY+19vqRmFZnNjFJ0WXDQ9HwWJvUcaNh0nqW1hvs5sW0IKbXPA9rzsIT\nWFove9xmBGn/IFQX69URPfeYE1Gp8sWpUFeVBfk8AsZbiFtB2NFoVOp3BVLETWWP+cahhB0MBrbZ\nbArS4rNYr9dH3WP97NuCIO0H4dUt4Q6nZhNDqsgCCt2HVcX+Xr2xruQCdZE3o+nYc3pzYY9gvV5X\n4lYl6nw+L6ZU8MREEHC73ZZ+R7/fLzLKqNXOZrPi3+92u8KK82ed6vThUMRTjqVEIbkhSHsmtPbK\npOKRMN7x+vra7u7uin116raa5NcC1Lrp+WazKWYy6YwmkAjk0Xh2t9uZmRWTJ7TshL+vbkTqYrGw\n+/t7u7+/t8fHR3t+frbFYmGr1ap4nf1+X8TOWhPmiYrcHK/uMXswXBLabDaVmjdrtHn+MWe++eaX\nA4K0ZyJVg+10OqWtJrl7B4v3h+W5xJpwqhP6a72VH6/Xa1ssFu58psViYev1uuLC8mNc+Ov12pbL\nZWWyYb/fL02c0LVYLOzp6cmenp7s8fHRXl5eCqsLF51JijJTirRsaVl1xaTln3uE5SkgSGql9rbN\nBUHaM6CumR55q0nu3kkttbQpwnrKIa+8gtlMLy8vxWwmPsLFTS2zX5aWGxDwut1utzRXWOcM47V5\nqaXlOF3jdiYt7xZQZ2n5+e12WyKtEhc3AW9vW/ydOSBIeyb4gtM5TLz/jDfiFETluNZzj818zW8q\n84q1XC5tsVjYfD635+dne35+tqenp+IcJRf8fg/QA7NskS9wjoU1LuZB4mzxYWlRC06pxobDYSn+\n9WJa/uz1eZCW2xjZRR4Oh4V1x/BydpVzQZD2DNQlQZi0cIVvbm6KXetub2/t8vKyUpdl99gT6yu8\nUS24yEHal5cXe35+tsfHx2I9PT2VRrp4lq7T6ZQ2tNIJE4fDoSAnYmQ9ptZmsyn2ldUwAI9BWo5p\nU+6xtur1+33bbreVrDxbWuw6wJ4LWgbD0jYYHmm5DsudOyDt3d2d3d3d2eXl5dENqPAa/HqAWlpO\nKiEOxehSkPbh4cEeHh7s/v7ettttZRo/H7VTR7ty3t/fi+QWb82Bx0w4PSIJlfIklLQcb3vuMX8X\nmNK42+2SiSgsDTVA2JwQpD0T7NZxqQEZSs/S3t3d2X///WeXl5elRIxmT4/VY1lbzONC2T1l9/jp\n6ckeHh7sx48f9uPHD9tut6VsKp+rIF8Ju16vbb/fV5JbnPTSRJcutmSeJHE0Gp2UiOLvAVMzMDmj\nLhGF5gdtVAjSNghq8XT6n4584YZ13SQL8azWX9lV9Nwzfs5TOXnxpJc9ns/nxf8dj8elhAzXLFVe\niOQQ4kEvO43Hm82m0lXEx2Pup25V6WV2PeKChJ6AhUf3IKbWG1NujfJBWoHntnEdFjI7L3bC7nXc\nZqdTElNCCSRCUsIJMyvcTE4CeXVZztaCCKpc0psFxBmoxbLbz+UVfh0WQfBrMNn+VpzokQzeD285\nwnsfISbnMhY+jyBtxtDkCB9ZzO5NRWTFE7fZpVrsPFVOXb+qaoiR5NH4Ukmr1k8vUnYvU4oiZGtZ\n5cTxp2cddf2t7wtgvTRPB7m4uCjKXUxYJNm2222QNlcwYb2lpOWha9PptDJRkfeGTWmItdSgUyb4\nXF1itrRMWhBKVUVMWoAbDDTG1pLWfr8v1WVTNwZ9/3/je+L3j8c87YLLb/gsuB6Lz4FnU+WCIK2g\nTkDBG2Gx6wXreszScvxlVnWBUxJFJS1ni0FY1fzWWVp2iVmowX+vJx55f38vfqceWdj/N60sx/6c\nCeZMMsIYfEcoN/HnivfOw/JyQZDWgSp2QDodysbqJ14gLTcCsFjBrBq7Kkm9ftVUTKvucSoDy1Yd\nj/E3svAhJYBA3OstTjZ9hp4X5D3FPebPAEk2bkfU7+arI0hL0IK/9rXyxcAaY9657lgiSrO1Xotb\n6siWVt1jZHFZzOC5x3i9TqdTWF1OSnn6Z05Ypd4n3xQ0Vv/T5PUIa+a7x+wBsIVFRty7oX51BGkF\nbF20A4Wzx+oeM2mPJaLYPVUipJoBmLTHLC23w3mWFkdVXnmiDr2YPZderepnKIvUNTarZo/ZAzCz\n4jNEaIHh7+EeZwzUYrkvlpVLOjyca7E4ZyurhDUryxC5J9UbxKbnq9Wq1LPKRPXI6rmuZnn2kKag\nNxomrpJ3s9mUhBYfmcH1FRCkNavcqVMD2GazWbHNJI4pwnIvqtmvLK2OX2FxvBJNY8bValUIJfjo\nxbGeBLAN8CoAGuZ4bZA5odWkTd2lkXlkNQ2STtxah8QTbzupd3IV3yMJwrVOuLocg3o9r2h/UzEF\ndL98A/CSQ22Al5fwFifb8P9yQWtJ631ZTFpYVm5i9zLFGsfqCFQmLSwtSKqCexbKs+XFUeNZPcf/\nV9Lm1Hb2O+DvVMtVOnYmpVDLAa0kbSoBo6RV6wqCskuMhd7Y1FwikBbJJBb3Y61Wq9pxLtqKp6Nj\nePsNL5ZtA86xtNrVlAtaSVozn7gp0l5fX9vNzU3Jqmoz+8XFRanFjo/qHiN7uVgsigb15+dnWy6X\n7iBxL+b13GdP7NBW91grAHwzDUubGVJlDY+0vKMdT55I7WyHnlRvmf3q0oGlxSgYNKovFotkogr9\nqF5tlEtG6h57g8abDhWJREzbAChxPdIiCcUD2XjTZ2+3dswNTtUyOaZlS/v4+Gj39/c2n89dd5cn\nGdYBAnh1j9viIuuN0rO0StiwtJnA+3I7nU6pMM+Cc05AeVtT8pwntm4qltD+ViyMhwFpU/EqpizU\nKZZUnNEm19is6j1p+cdTfeWGVpG2roMHDQF1e+2onliVTtDmpkT18/nc7u/v7eHhoRi4Np/P3f5U\n/D+2mJAdnkLaNlnYtqFVpDVLyxRPJS3+jbd/LNzTVGlmPp+XBq1htClrhjm55Kma2Dp41iLVIhdo\nDlpJWpYrcqxTt+mzWllvK0pYWiSaWBOMgWuwsGxpWdXkKaI0oaTExdHrFPqMjpvA56JVpNUEhe4B\nk5oxpLvaeeUDWFpWPGGcKS/EsPyY3WOvUYBJm6oxM2n/dhN64N+iVaQ1q7rHvBest3GWElcbxbkZ\nQC0tjzLlbTK8iYZop6vrp9W/wzvXaRdB3OahVaSts7S6f6xH3PF4nGwS15gWJR0eY/ry8pLUDkNY\noS17XvubZjw90nrTMQLNQKtIa2aVJJTuH6vJKHWPPbf0mKUFaZ+fn0tT93W/WN6qwjueiyBrM9Eq\n0mq/pSabIEnU/WC4a8fMHxVzOBxK84i94eHPz8+lTav4HM3qgTI0DDjlRpRj7fUctI60iGN5zhOk\niJAqojd2PB5XxpF4UxJx5FIPn/NzXIdtq8zwd3AqcZuM1pFWGwK4Y+fm5saur6/t8vKyIC0sLLvA\nqRlJSlJvsWhCZzcFfKRi+LrPrMnWtnWk5QFtrC1GJw9bWogouFOHVUc6EsazsnrOwok2ts6dilNI\n11ar20rSsnuM1jveoZ3dY7W0OvSblUt1hMXyCB8Z3uNgqaY+37bPrlWkrWty//btW+Eap2JajmVB\nWI5RPZIygVGH9VzrQBWetT2HpE11kVtFWnaPOaaFa3xzc1OaCaW7tJuVt9HwdrA7lojS5FUol+px\nCnHbZm1bR1qUe1KWVicwptxjdo25hOMRlh+b/f1h3k3DqcRtqmVVNJK0nvCh0+lUpIo6cBwjY1Cb\nhWvMVlYtrWdlvfJO1GH/PrR+rjLQ1K5+uaFxpK3bi0Yb13m3cO3e8faU9ca66C52OnuYJ9wHPgaP\nWPocE5ZJypMt9XvJdVBAo0irI0Z0YTQMq52YtNrczr2yZuULAkko3Y09tflVThfFV8IphOXnU9l9\nbzherjXyxpJWW+gGg0GFsCpTRPzqzRICUqTlbTm4HhuW9uM4lbC6j5Bnab1B7rlm7xtFWjMrkZZd\nXiatZ2l1GoW3ezuXe9jlYveYJ/23dfbw38IphGVry5ZWV84jZhtFWp5Kwd07sKbeQDa1tLqpMse0\nx9zjlKXN6YL4KkjFrKc8V2dpU/vp5oTGkVab29kNRkyrSShORHkD37yYNpWI4pg2LO2fwbmE5Rsr\ne0R1iaic0CjSmlXdYx6HCsJ6xIWl9UarnhLTqnvMZZ4cL4yvimOfYx1x6xJROX0/jSKtzrnVvWb5\nyKUdTj5pk7vZrwtFLwRNRGn2mC+OwE/UjYDVwXkcophV67D6nOYZ+GaK4XpeWS6376dRpGV4Y0ZT\ni/8d4KmVPLLybux1u7DndCf/0+DPOFWO6/V6xZ6/GESQ6mdmCSif4+b5+vpaDB/AIHg+LhaL4nvC\nd5TT99M40uqkg5S76xGXoVlJM0taWRAXpI067U/od6G7tOviLivenFs3MVOFE875Bqqk1TnTy+Wy\nNDEkp++ncaQ1O826plZqMBrki8dIC7es7e6xF2Zwl5WW20ajUWlnQq+f2cxK34POhlbSYkwtk1bd\n5CDtF8Q5hFWoRtXTG6t7HIqo0/f/1aF5vDMhLC26rDxLqxtve5YWhH18fCz2AGbJaZD2C8GLV8+1\ntHXF+pSlRcePkrZt8Ijr9TNjmN5sNiti2mP9zKlyzjH3eLFYFN8Pf09B2i+AVJfPOZY21SmSIi3c\nrlRpIaeL4negcSyOKdLyjoQYQsCW1ktEsXAC3wNPwKwjrVpnnOf0/TSOtN4d/lSyavbYS3ykGgWw\nvFirTaQ1O23/X+1nvr6+rlja1IwulpIyaeuyx4+Pj/b6+lqZGpLjRMzGkZbhETf171LlHo+4nnum\n42RyVtx8FJyh99ojvR7my8vLEmm9+VzdbjdJVq3F8g2UF/5NampITt9Ro0n7O9C4Fs95e+x45Ydc\n1TYfASeauNmCj/1+v5gtDYLqYteYZaVmv+qznqAF1hU1WBVRsAbc+05y+36CtCcilZzydrhrC2Hr\n6rCsbMI5kk0ecdktRjyrVvb9vbwrIawriMrbhrJ4wpMs5mhhgSBtAt6Xy4Tl5gHPwqY2z2oKvDps\np9Nx67DQeGNcrUfY6+trm81mpf8H0pqZm0/w4lfe65ctbVMIaxakdeG1hXmETbnI+m+bhro6LLTe\n3KSBOJZ3cfCs7WQyKenC1T3mQXpMWhAWllbFE2xpzara5dy+oyCtQGMdTxlV5xqnrHPTcGodlmux\nKOt4Vvbq6somk0mp2QNHM9/SsnsM9RO7x15Ma5b/FMwgbQIeWb1sspLXuzByvTg8nFKHZUsLsmqW\n2Fuj0Sj5WmppUzVZbtxQS5t7AgoI0jpI3ZFPsbb6O5qIujos75XEpOX9klLWdjgcVvICKmzxYlp2\nj3VWFxRPOc6CSiFIS1BSskhCFTSpbHFTkLKoXv0V5yCpWle2sCjroBbL+uJut1t8nqxawhFlHaz5\nfF7EsYhl4TZ700OagiCtQMXoPH6zbpxMU6BqMW5nhPuL/letx04mk4r15HNWOmHXBhCUP1fWBbNO\n+PX11Z6enuzp6amQJj4/P5cyxtrL3LTvxyxIW4JaWAjRcQHVjUhtipVlC6pzsrQXVqddTqfTwrJ6\nCyWd8XhcIu12u7Vut2vv7++lLhxd6gp7brHOgmpivTxIS1DS6k54dSNSmwCNTfWIBBNcWu2HBWnh\nFusRXTtQSLGlPRwO9vb2VjRewN1NPUbCic8hpvDGpDYJQVoBWr9S+la4ak0d3MbWVV1h7YPlIXmc\ndOLOHV6j0ajifnP8utlsCusJC8pHjlm9Iw8eaHLDRpCWwKJ0FqYfG5HaFPdLLa3u1ICMMGqvvC0o\nElAgLDY04/PBYOBmhXG+Wq3s5eWlFLPyen19Lbwcb+12u2TtvEkI0hK8mLZuRGoT3S+PtDxDGhYV\nRNSlP+PHvV6vsq8OLC2yw5g28fDwYPf39/bw8FAstD567Y9qVXXoW5OIG6QVpGLatgwj50QU74PE\nW4Ry8zonmkBStrr8uNPp2Hq9LpoAkChC9hglnaenJ3t4eLAfP37Yjx8/7P/+7//sx48ftlwuKxri\nlMa7qeIWsxaSNtVTq0jpj+v+TQ6om+bBYn89cqKJFxMXUxR5JwfeIwmfIXIGm82m1Fr3/PxcLLjI\nvFar1T/+9L4GWkdaszRx2TVklxAJp+12a8vl0t10+tSbwb+Eur5edlj3OeLHsLCpuFVb6mBFuVMH\nQghkg/kxJkyg9orEX27jYP42WklaszRx2S0cDoelOU+bzaY0BZ+3xMwBnU6nICe7vppoSi1OQMHl\n5XMQFuUcuMBmVlhX1Fe13sqjYVgwEaStopGkTc19qvt3eMxWh7Wwh8OhIC0srbcd5lcGSIsbEhb+\nHu7M8TLEvCm3t9BKp+UceCmbzaYkQ8TEf5xDkogjxBJtnR2dQiNJew686QucNeUk03a7LcVpObrH\nWnNV91eTSJxMggRRN+XGsdfrVTK2POZltVoVFlXliJiWCGUTd+o0sWzzO2g9ac2qgnhYWr67dzqd\npKXNyT3udrulLUDZgrLQn2NVnvoPD8RbZlaqcXMWHmonHSDO6/X1tTKXONzjKlpB2lOsIE9f6Pf7\nJQsLC1VH2lwtLU9GxDhTLzPsyRD1HHG/mZW6dUA+b+o/12NfX18riqY2b62SQitICxwjllpaxGa4\n0Jm06h7nAE5EsSTRG2XqrfF4XMk882NYRMSwSEShtMPN6iDt/f293d/fF3VYHenT5OkfH0WrSHsM\nKuPDhQKpK0jxAAAJ+klEQVSxAdcsNR6cTCbFRes1zeOxB34+NTCN31/ded3/HY1GSTE/k5YtLZd2\nRqNRpVWPz/G3pGY5cZKJmwB4rlPgOFpF2lPu1tpLCtJ5ovnZbGZXV1dFlwncatXU4nHdRAx9bY8U\n2oGTaqFLLUxEZCLy4ufrxpjivernhp5Xrb8yOb09fJukKPsMtIK0p1wQau2YtEBqHxq0hQ0GA7fL\nRPf08SR2h8Mh2cuq7qgOAcd5isQ4H41GldoqLySkcNQJ/xinkxpet91uK2TFQi3Wm5QY2eHz0ArS\nHoNHWHX7zH6RFpI+WFpYjX6/XxHE83ndtEaQtm6n9LoNmVXs4R2Hw2GFmHxEvM5lHAglzKzkOXjL\ns7RaewVpuZUuLO15aD1pvYtFyYrnPEvLF1+/36+MSEECC4kaby4yzlmNpdaUpxym9MFIjLE19vph\nvXiclV58E1BLW7cg+k9ZWZ4woU0XgdPRSNKeOiYzRVh2Vfk5L+vK1mIwGBRWZL1eV+q4+D2p9jHU\nUD2LCpmhRzotQ6WWTpvQx56lxmKxP9deefGWn6jJ8krtsxOW9jw0krSnoO4i4SQLX7BqaVViB2It\nl8tKOej9/b2Q9nm7FDBpWcDA53DLWQvMR+6oUcLrXCf9/Xi/qc+JBwR4rj9maB2LZ1P77ARpT0cr\nSXsKYWEV+aJV0vIF1+l0KgTgWBAuYCq7zKRNub8s2vcWxA9KRnZ71d3mx51OpzKAXY86ioeH3/HM\nJi+mxbgYnrYYiajz0TrSnlr2ARHVXWbispVlEYaOFoW7yfvXesTo9Xq17ivPFf4oaVUUwQt/J9+M\nOPvNkzxAPj4ul8uKO8wlH3aLecxpqJ3OQ+tI+xEwcTmpMxqNShedtvUhBgXZdNq9WluUZbwGdLbw\nqewv4lIvHuaSkNmvAXbs9rL76x15i0md5gGRvzc1kQfieaN6QvF0HoK0CYCo/Nhr22PVFP+MCTud\nTos2s9RQM5CWCeud12V/PReYF2fEOWsN8LwmjVl10oS303rdOYspdJeGIOx5CNLWwCMuW1omrDdL\nabVa2XQ6LdxJqH+OkbZuqevMR67VatmHG9M5GcaLJ1CCYHyu2WHPouJvVRfaG3Ha1C1V/jaCtEfA\nxOXWtmOE3Ww2Np1OKy1mqSSUkjaVQT5HXKHlGzMrLCvIyzE1x606nhSusWaF+TFP+Pd+D5NV9/MN\nnI4g7QngjLI2ErDogS2TZ614c2NeWqfV5BGfe8kutqh1MkbNhusYUiabt58OEk3ethzz+dxWq5Ub\nC3N/redlBGnPQ5D2DMBqcSbZK4fwRcvnnoVld5VJm7Kknia5rmFAx7/A0rJ15RiWSzKcHYZEUUfF\n8HG9Xlc+Dz5Ptd0Fcc9DkPYMwNJq6x5ffDo8m491VpZljB5ZtfbLxOTHOOf3jOPb21vxWMUSSlKN\nTdfrdaWJXY/r9drVVPNnhHPvGDgNjSMtXyzsinnxlC7OpKYIUNdI//7+XrTneZbGI2qKtJ4Aoq7Z\n3iOCEkfJmUoYpc5haTWWRZkHUysCfxeNIq1aPM6GQrhft3h0jOdingKvQ0gTWSCouopoSuD4VC0q\n/z4+1xuVtyA1xEJJRrc88eJZzHz22uuibPO5aBRpzaxiXXe73UmERUeO15NqZq4u14PX0sfPHw6H\nEnFTrXkapwKeJcVjds89Fx1xKeSGekRZSpNorC3W8k6Q9vPRONJyKQOkRSyYIisuUm5tY5KyIqoO\nKn/UTiHWMythmbScZPJcci9u1A4cTYZhv5xUkzqTNpVIY0VUWNp/h0aRli9eXMBsrVJkxWPEj5rN\nPGdwm2qVcY4uH36fai3xWqlssP5fFUiwmN9by+XSneqP8/V67e5ExyUh/QyDtJ+PRpHWrOoe8wV/\nzD0eDof2/v6zL9asah1Phf4/ttJ1GVUvjvZuGEp4/ZtZ2M9H3uSKt+BIlWw0iZZqywvSfi4aRVq+\niHEB889SogF+rEkjnRN1DN7AM7x+3ZH/faqUowkoTTxx8g1kZXd2Pp9XNmrmtV6vK8kxzXBrvBz6\n4c9Ho0hrVo5p2dq9v7/XxrSwtGZlwmKri3PglYv0PaYep0jP/zaVKWb3mDtvkGiCRcX+rxgWjgn/\nGGFaV1v1NMvneiKB30OjSKuWVn+mO7vjYp7P5zYej83MCjniaDSqtI9BdO8JHI7VcBl1ZPbcZ/27\nUksb0TU7rC6xxrfr9fqjH33gE9Eo0ppVLRGTlzOgi8WiNCLUzGy9Xrvtb3hOs8sqzj+VtHXvPRVP\nskyS3WB1ibU1jhf2gMWANZ2IGMgDjSUtLkImkpKWCbvf7221WhW9qZjDxP2qqRlMZudlmOvee2pw\nmva5eg0JWpLRxWNgmLQRk+aFxpHWrEpcPOZ9ZZSw6BflaRA4X6/XJeKifxW//9xkVd37Pla2SYn5\n+Tmvr5VVUCqmCNLmhUaRVskK5RF+BkvLhMW0BpBZN1Fer9dFXywGerM7Cb3wnyItssDcceMRUM8h\ndtAyz7EmALjHQdp80CjSmpWznGZlsQPIqYTlTaJ4UBoudh2TgtfgQW9/6r3rtMNTRrlgsXZYj/x3\nqHsdljYvNI60Zj5xEduqS4zkE6YdsnjeG0bGvxOE/ZMXPQsk+GZSt1iGmCImE9TTJwdp80HjSKsK\nI+9nICxcZSSXQFq2sDo90KxsYcfj8R+76D33GAkkHkvKrXH8HGuHvYX3mZJCBvJA40jL0AsRpGNy\n9Ho922631uv1bLfbudJAnWeUUjWNRiP3faTa6hSbzaaylQbXUlOEZdJ65SA8DmI2A40mrQe1Mkwo\n1u3yMDTEmqo4gmuKnc05tj21ZstE2m63FZdXz1k4wZtZMUm14T7I2iy0jrRmvyytAp0s3pYePPsX\nsSYIi8QVarYfxW63qySZNPGkC668zhKOwWnNRatI6+lluSQEN1ItLGdzeZK+CjD6/fLHeczaKpne\n3t6SI2D4Oa+0w/vihKVtNlpFWsBTTHFrHB6rO+ztr8Pr1OkWKcA9P2dxZrhueFwQtzloHWn54kVM\nq5lmJSwPWUvtH4vNrVLw2usU3LPqaYs1saT7w6bGlAaahVaTlsnKW2aAsDz2pa5RgJ/7HWjG2juv\nO3ptc/o3B/JH60hrVu4TZQurYoxz1598b3U9q8d+pr8v0Cy0krQMtbyBwFfH7/eTBQKBT0WQNhDI\nDEHaQCAzBGkDgcwQpA0EMkOQNhDIDEHaQCAzBGkDgcwQpA0EMkOQNhDIDEHaQCAzBGkDgcwQpA0E\nMkOQNhDIDEHaQCAzBGkDgcwQpA0EMkOQNhDIDEHaQCAzBGkDgcwQpA0EMkOQNhDIDEHaQCAzBGkD\ngcwQpA0EMkOQNhDIDEHaQCAzBGkDgczQiU2nAoG8EJY2EMgMQdpAIDMEaQOBzBCkDQQyQ5A2EMgM\n/w9YLzOtMWebRAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54704c02e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: 0\n"
     ]
    }
   ],
   "source": [
    "view_image(X_train[1], y_train[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADtCAYAAABTTfKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnetWIksShQMREVQU+8z7v9/0UeQOAjI/Zu3qzSayqrBp\n26TiWytXFbbipdlEZMattd/vLQiCfLj62z9AEASnEaINgswI0QZBZoRogyAzQrRBkBkh2iDIjOuy\nf2y1WhEPCoK/xH6/b3kfD0sbBJkRog2CzAjRBkFmhGiDIDNCtEGQGSHaIMiMEG0QZEaINggyI0Qb\nBJkRog2CzAjRBkFmhGiDIDNCtEGQGSHaIMiMEG0QZEaINggyI0QbBJkRog2CzAjRBkFmhGiDIDNC\ntEGQGSHaIMiMEG0QZEaINggyI0QbBJkRog2CzAjRBkFmhGiDIDNCtEGQGSHaIMiMEG0QZEaINggy\nI0QbBJkRog2CzAjRBkFmhGiDIDNCtEGQGSHaIMiMEG0QZEaINggyI0QbBJkRog2CzAjRBkFmhGiD\nIDNCtEGQGSHaIMiMEG0QZEaINggyI0QbBJkRog2CzLj+2z9A8P1otVrJx61Wq3is997n6vN4n6+f\nV8V+v0/ef3x8HFz1/hII0TYYTywQkSfOVqtlV1dXpUu/XlfV11XBAuT18fFhu93OttutbTYb2263\nB/ebzcY+Pj6K58lZwCHazDnVSlU9DwTEQsJ9u90u1vX19dG9ilDv+eu956r6XSDO1NpsNrZer221\nWh1dIXY8D75XjuIN0WZMmSta9mJMicMTHK/r62u7vr62TqdTLH4M4bKA+Z6/3rteXZUfsez3e9vt\ndkcLVna9Xtt8PrfFYmHz+dyur6+t1WoVgvb+Rq1WKzvhhmgzpUywqY9VfQ4sq4oP106nYzc3N8kF\n4ZZZVP5c72vL+Pj4sO12W7jBfL/b7Wy5XNpkMrHpdFq8CUCw7Xbbtttt8Vy5CtYsRJslnmBTItUX\nZdUhE1tIXTc3N9btdu329vbgintYTLi7et/pdA4+H/dYVaJlsfJeFdf5fG7dbvfIwq5Wq8KD8Fzj\n3MQbos2YKsGm/i0lXBWtiu/m5sZub2+t1+sVVyyIkN1dvYdY+Tn4/vq6/OW42+1ss9nY+/u7bTab\nYuHxdDq1TqdzJNibm5uDPTMLF49zIkSbKWWhFoVflGVWmt1jdmkhPra0/X7f7u7urN/vFwvWNrW6\n3a71+33r9XoHV9x3Op3S33m73dr7+3tyjcfjI8HO5/PC9c5ZqEyI9g+QEpAXQqkjOv56PdH1nisV\nx+Tn8H4OHBaxdeTHEJ2KFY+73W4h7jLReoLt9/u1LK0n1vV6be/v72ZmtlwubT6fF28iNzc3xf7W\n+xvnKN4Q7Seo2hfqPV9ThzypFxXjncjy8wBOJNBrnZBMal/a7XYP3GF2j3u9XiHWlIsMK43DJ4ip\n6tRY/9bqwiP+ym8W+N6pOHCu+1mzEO3JlGX+lGULsSXThRdZHdGWHfTgBZhaLM7UNRWLbbfb1ul0\njg6h+KoWWn9GuNewyByfrRtv5t+B35xardaRaPFze6LV/W1Owg3RnkBqD8j3qWVmxWEOhzv4vsri\nQDhejLTT6RQhjlQaHyw9C8pLkDgl5NPtdg8spyd2L+TDn1/X0vLfk0WLj/Hzqmix+G+Rk1CZEG1N\nqoSKx7rX5Md6mKPhj6qQh77odalosfAYP4Nn5b2sJl36puElV5QlWHjJFSzaOiLC33S/3xdWGt+n\njnuMxWLPTbwh2hqkwibe0jRAvmrYRFfVQcz19bUrdo5zIjvIS/ODtUwtdiW9dEa1ynr18on5PuV6\nf8Y9xj1+L7jf+iakZwYq3BwJ0Vag+6CylXrBYuEEVUMmuFaFPLCn1MMgXGFpObUPgt3tdoU1Si0V\nUOp3LLPIp/yNeJ3y/8GihdXd7/e19rSprUtOhGhLSP2HllnXsgX3mEV7f39v9/f39vDwUCnam5ub\ng1CJhk7a7babk4t1dXV1lIXEh0May9Tfv2prkPraOmcBddHvy6fjfE7gWdvU98rN8oZohbpWoio5\n3tsPdrvdQqAs1j8hWhXsbrc7OsFV4Va559+FVDir7OQ8F0HWIY//pS8C+66ycIgewOhhTsritlot\n63a7By4xFh5Xifb6+vrILUbcE6fH+FkhUhYwH9boPjQXN1EP2vh0HKV4SGtEIYHW4DI5ijpES/CB\nUSozyHMr8VhPK/UKS5laVaJtt9tuoj2LFi9mHErx3pYPoli0uQiWxarexMfHhytaFq5XQJ8jIVoC\nlhYvbI2jckaQJstDdFUhn1S45/b2tjLkA/fWW3jD8KwQnx5r8kPZXu87gppar0SPRQvhlgk2RJs5\nHNZAaIatGYTJbq2um5ub0j0xJ95rLSlOb8uApeTYqBaRp3ojcXKFrlNOb/8m3FYGFT9Y2+22EO16\nvT6ytCnB5ijcEK0dl6axaNma4sAIazAYHDzudrulp6ssOi+Vscra4Wcri3dWvTi9g7OcrCyEixpa\nWNT39/cj97jM0uL5cqTxouUXLF7IEBGsK05nHx4e7OnpyR4fH4sr39/e3h48pxfjLTvoqhJPVViJ\nwxapFye7w+rK54A2cONKH3WPdU+bco1zE2+jRevFIT1L2+/37f7+3gaDgT09PdlwOLTn52d7fn4u\n7ofDofV6vcrvkwru1xVOmSVXUi9GL26aA9p1EcKFYLH09DjlHue6r22saFOJA2xpIVwN0aiLPBgM\nXEtb9T3PTdULMPXvdd3FOp7A7+D9HPwxFelyubTFYmHL5bLoDzWbzWw+n9tyuSz2tmppc6exogVq\nBWFpWbR8EMVlZeySYt/Ez/m3LFnqRV/2eR7eG5ven/t3Sx2kfXx82Pv7e9FpEeLkNR6P7fX11cbj\ncfHvq9Wq6Hl8CVbWrKGiTYmKT1c5LsvJDBAtH/6Y/dprpfaIXyXc1J42Jd4qF5rvU0I91+/GAvUK\nH1arlS0WC5vNZjadTo/WeDy2t7c3G4/HNp1ObbFYHFjbS9jPmjVUtGb+C5FPVTk0w0n6mkihltZL\nffxq6uzfUqmAZW9onKOrb0znFC63SuUFd3g2m9lkMrHxeFysyWRStE/FWiwWrqXlvxH/7rnQONGm\nLAiLjLsPwj1WSwvRAk4VZLGqGL4CFWqdFyz/nGWC/VNixc/AcVgcNmGxpYVoR6ORjUYje3t7s8lk\nUrjPi8XiwNJ6f4NcaZxoFX0Bao6xtv1kS8uxUbzY9vt9IVhkKH21YHHV7CgvTlklWm/xm9W535T0\nhJgTKNQ9fnt7s9FoZC8vL/b6+mrT6fTgFBlru90ejQXRnz8nGilaz1qwe8yWVnv9pg6ikKDPz6lC\n+cp9rQo2dRCTEm0qOQSP+Xf/E3taLxaLlqiwtG9vb/b6+movLy/277//2nQ6PeqFzEkWuYsVNFK0\nwHuBepaWM6M8S2v2yz3W5/7KF0dKlNp6pszipqwri/VPvhGpe6yi9dzjl5cX+/nzp81ms6PSRL7m\nLFSm0aL14JNLDeKjt27KGnD1jNeVUFMGvfs6sdTUfar6RVP5ysIfVTXBWqrIP/854rR8EMV/W02g\nwKEULC9W1X7+EmisaL3/RCShr9drWywWB4Oc9vu9LRYLN9G/7uJ2LqlVdkjEFlPDIXzKqoOpvPI0\n76q50al7Xvv9vnhjOgf8+/CeFoUAXMFTVXZ3SUJlGiladuv4P5bHSSwWiwPBbrfbYsSELgiYD620\nUB19gb3WNN7hjrdSp6reYCrvHvs6r1MjanC5Vti74vflRBK0Mz3X/01qT8v71Dq1siHaC0EFy/tO\nWNrVanXQ/BszZLght3dF7ydt2oaQA9qcplrV4GfyXFi8kPmFm7r3HuOF7nVpxNLOGF4DuW63W/xs\neLM5134x9eYE99ir4EnlFvNzXhqNEy1QwZr9srTr9bqwsBDKcrks+ijpnhXr9va2yEleLpf28PBQ\nWAMzK06YvSbeujf12qqwm8gvYn2cWhCtdmnEfafTOcqzxlKLBsHi0O5clLnH6hqnLC3+jl6I5xJo\npGhVsBxrhaVlC7tcLg8GH6dqWnu9nj0+PhbJ6hCJmRUupCd4foGVWUKcoC6Xy4MDGVxV0CpmTp73\nDqo6nc5BnTCnAOrfDqGxc57MVoV81NJq2R3+1nguvl4SjRStmT+ACe4xBLter225XB4IzDtFxbXf\n79tyuTywama/itf3+32xD/YsRKvVckMV7Brj1NRbLGY+bcW9upS6ut1u8abDn7/f74/qcDnV85zh\nlJRo1ZPgNEf+O3r/x5dGY0ULPPcYAuGEgjq9je/u7g5eUGa/BItWNHihdTqdoxca/7ueCO92u+JU\nm6tc+IpcW7a+fK81pnrafHt76ybZt1qto1k/ECy/OZ3j/8Jzj9XNV/f40uKwVTRetAy7ZylSnRav\nrq5su90eub7sDkOsOr8Vj+uIFvFIjk+WiZav7+/vpZYWsWYu/i+zaqe6oFWWEL+zWlec5uP30z5Q\nTRKsWYj20+i+mF9wGGyMzCm4vRwy0pNnHOboARFfUU+qSfHsHvMLPdUD2Es4YLeX32y0rjjVN/mU\nvxuuKnxNpMDfcTqdFhU8eHPC78aN25pCiPZEysIJEBbHeSHY7XZbDNlKWWJ8ri6IDW8IKWuaOkH2\nRMvehOZes2C1hew5+iZ74SwVrWY8odwOXSngOZzTPc+FEO1voALWOK+ZFYJdr9cHISPv9BnP44V8\n+HmqToi9WC1bpCpLq1ZW59DqKMlTUhhVqPr74U0Pb3xcIMCWlntBhaUNKlGx4kULS4t0RRYaUiBT\nnRjx4mdBqUXy2oZ64kwt3vvx82olD1v/Kve4ThdJ/ltphheLtsrSTiaT4uNw/8M9DirxkjLwcVha\nzqTCC1DHhmgKIxfNe9k9eH5PjPzx1CGW98KuY2nr7mk/Y2l1z15nT8shLHaPQ7RBKSnhwtKyhdUY\nb52CAXwPPZ31spjUWvFS9xN4nSnM7KiemGuKWbDn2NOqW4w3nqo9rbdfD0sb1MI7kIJIttvtQTiI\nCwJUKHqf+h547Lm3nsBTIRmvuB3XMkvLwv2dWUCepfUSKVJ7Wi2ECEsb/BZ14rx/Ez0l1n01VyRh\nedP5uBvlKSEfdvG9ggccNGl7VO5t7Ln93/Xv/acI0TYIlNGlYsVouj4YDOz+/r6oWOIZuHpyfIp7\nzNsGTrnE/nUymdjLy4uNRiMbj8fJpuMaa26SlTUL0TYKpCOyu8tWFLOJMJ3+7u7uoCRPXeNT3WPE\nYtn11WbjEC2mBXDxRaqQv2mEaBsEW1qtme31esWsIlhaFq1X5XTqflZDYDgVxkKzcbRD5ewnr6LH\n29M3gRBtg4Bo0WFSa2dhaVGah0J+trSp+HIdOAwG0aINKlxinBKrpdXOG2Fpg0bAc4owWIxn7pa5\nx7C0Xny5rrXlemUWLVqgjsfjI7d5sVgc5RinTs6bQoi2Qail7fV6ByM82dLiIEr7PfMJtIaPqmD3\nGEkTEO3Pnz9tPB67Bf5ajO+FvppEiLZB6J5W5+4+PT0VnStgafv9/kHeMT8XX+uAgyh1j19eXuy/\n//2vjcdjN3lCa5TxXE0lRHuBpDKeOCWR97TcYiYV6qnTJtVL5uCP6XxZdoWxl9XEiabWzJYRor0w\nvP0mrnpiDHHy8k6LT6mXTRU8fHx8uNVImkPtpWI20QUuI0R7QWhNrPazUrF6wtV5Rd5khBQsUBUg\nOm9od0ivbUyqUD/4PyHaCwP7Vi2wR5qiWlpccVLM84rY0p4S1vFSDbkWWOt+tW9Vk2OwdQjRXhCa\n9M99qNCIvMw1xqGTDhk7JbdYG7Px3hTldFX9i9XaBoeEaC8IDsdwATuumgHliZc//7OWVrspevN4\n1MqWtUQN4R4Sor0wtMcT5xmnDqKQEdXv991hW6daWhWsNhsvc49TB1Ah3F+EaC+IVLuYlGA9S+uN\nLfkdS8uCTU2+8/oXx2FUmhBthqTisKjgUZFCoEia4MQJreLR02cWbJV41MrqTFmegpBq7+oVAwSH\nhGgzoywOi3xiLgLgx4PBwJ6fn+3x8bEQLk8C1N5VukAqeQJ9nlAvqyNMUBCAnGKvz5NnXUO4h4Ro\nM6IqDnt7e1tkOMGioiAA6YpcFMBZTzoBPpVTnGpnw6Ll3s88AQGi5eodTlH0Qj0h2GNCtJlRFoft\n9Xp2d3dXCBSpiXzPi0WrnShSVtasfOg14rGcpohSO7a0XsPxEGw9QrQZURWHhSusVhUVPBAqu81s\nadHDuEywZseiheBSlpYL3cvcY/R6anoVTxUh2oyoE4flAoDHx0cbDofFur+/Lw6eePGelg+3UlYW\nVz3lTRUETKdTG4/HR5aWh2hxG9QI85QTos2Msjgsu8cPDw/29PRkw+HQfvz4Yc/Pz3Z/f198Poud\n97Rea9eUe6z9mtTSLpfLomcxRMvF7TzaIw6g6hOizYiqOKznHg+HQ3t+frZ//vnH7u7ujgZ/6QAw\nfB+P1F6WUxfLLO1kMjloh5pyj4NyQrQZwYdQ2pwNBe3cPoZPkBHi8eKwHDoqQ6t4vPm5Ok8WlhX9\njHk6vcZog3qEaL8pau1YsGWN2bghGyp2OIe47gmxBwTrFalvNpuDyfQ8M5ctq5dUEW7waYRovyFl\ns3ZYtNjDamgH3Se0NrZOHLYMb+/KKYkqWF6wsKmZuSHc+oRovxl6esv3CPPg0Im7KWonRVhardb5\nrJUFmqbIS5uPe5ZWCwVCsKcTov2GeIJKucdeppNnaU+Jw6Zg91hPiVer1cEMHrayLFotEAhLezoh\n2m+Miuvq6qoI07BoEd7RLooq2jpx2DLYPdY5snzopK4xi5aFmpqbG5QTov1meBaW3WPvIIotLVf1\neO5xnThsipSl9ebypA6ivNK7sLSnEaL9pnhWFpaW47K6p+URld5BFJ77s3ild5jY7o2oZPGuVqtk\nwUFQnxDtN0KtK5/28oR2rZlFQsXd3V1RF6vN2U6Jw+JeP4aTX3WJeVI7hJsK8QS/T4j2m5Eqv4OV\n5cUpiWjI5g19PjUOm1rL5bIQJlfuINtJ62V1cFZwHkK03wy1stz2xRMsj+zQwc+nTmo3O+w+oVfO\nJebxlCxcryAgRHteQrTfDBWs5gerhWUry0OfPzNDVg+aONuJM554HCXPlUVusSfa4HyEaL8RevAE\nC6uCLXOPz9WUzeuiiPxhT7Bvb282nU6L/W5Y2j9HiPaboQdQXCBQtZ/Voc/amK2KVEgHB0o8KEv3\ntG9vbzabzQqB8wFUiPa8hGi/GVWWlgXriZe/Xu/rwKLlkA72s2pp2drO5/MjlzqKAs5PiPYb4YV6\nUsLlsZVsaX83eQJZSpqmqCV2XNgO93ixWBxlO+E+RHs+QrR/ERUTt5FhMWLpdHadH3t9Xf7fWScO\nqwkTLFScEE8mEzfPeLlcHmQ4Rf/iP0OI9otJpSi2Wq2DCh5vYVr7YDA4aMrGXSfKqIrDIumfEyX4\nnvs8cRtUbhcTbWP+PCHaL0ZPiLUYgHs96YJoUcmD3GJkPNWhKg6L/SrvW/VjGouNvsVfS4j2C2EL\nqymKV1dXR4UA2qcY7VA9S1sngaIqDrtYLArLyllOmjjBqYpqaUOwf54Q7RfjpSnini0t18lyFQ+6\nVGAI9Gfc41QcVhuL45CJQzpctaOx2Ohb/DWEaL+YqjRFdo/Zug6Hw6MBWqe6x1VxWBw4wS3GqfBo\nNCpCOtqtQicE4PvwNTgvIdovJhWHbbfbR72fIFr0Lh4MBgf1sii9q2tpzarjsOwej8djG41GxUIc\n1nOtEdaJA6g/T4j2C6mKw8LS8p728fHRnp+fC9F6sdlTLG1VHJYPoNjSvry8HMVhNR4bbVC/hhDt\nXyDlInN+MSwuC/j+/v4ouQJWutVquVaOP5Ya2+HFYRHWwcLBUyoOG4L9OkK0X0yqPWqZkDkTqqqC\nR4di8WKxcjqiHj5FHPZ7E6L9i6RaypSlLnLpnbZE1TEdujDsGRbWq4eNOOz3J0T7Fyhr3qbVPSra\nqtI7nquj+06IlutiEc4ZjUY2mUwiDpsBIdq/jGdtWZRaR6staFS0OmuHM5489xgnxK+vrzaZTCIO\nmwEh2r+A1zC8rI6Whcufp1+vYyd15g67x4jHsmin02nEYTMgRPvFaGjGE69nabFSggc6enK73RbF\n6Cpado9fXl5sNptFHDYDQrR/ibqWVkWLr+XnATqKkkWnoR7P0s5ms4jDZkCI9g+QCudwqqJ2nLi5\nubF+v38wolJrZdvtduX3Viurw7KQtsiZUNxUPOKw358Q7ZnRIc18j0HQOgUACymLKAro9/snpSlC\nYHwI5RUGcO8mtqIRh82DEO0Z4dNfPQVGbjHnDvPMnX6/X6QsPj4+FvWyp+YWp9xjPljCfpWn17Fw\nI6zzvQnRnpFUQzas29tbt7gdazAY2HA4LETb7/dPqpc1O3aPdfgzRMvCDcHmRYj2zLArrIurd5BL\nzFed5n6qpU0dRME1LrO0vG+NOOz3JkR7RtjSetMAeModxInCdq2VRb3sOdxj3c+ycNU1xnPwNfhe\nhGjPjCdaHDrpaMqnp6fi+vT0ZPf390V1D66f7UzBmVAq3NSeNg6g8iBEe0Y8S4u6Vy2z444Uz8/P\nNhwO7e7uzm1Efsqetso91uQJtrQR1smDEO0ZgWhZsGjUBivr7WOx7u7ujg6vIFiu4jHz950sTp0h\nyz2K0V5Gs52CPAjR/gaamYTkCW0bg30q712xZ9WG41rBwwUBZWV3u92u6FvMIzuQ9cRX1MquVqsQ\nbYaEaD9BqpAdVhZZTtoKFUkTOikAw6BZtFoUANDjSfet3LdYi9q1DepsNouhzxkTov0kmqJoZsVe\n1rO0OCmGG5yytF7ZnVpa9HjSxVPttA0qRlGqmxyizY8Q7YmUjfVgS+t1VeSwTirH2GtizqV32k3R\nmx/LrjEEC9HCLUatLErvQrT5EKL9JKmSupR7DEuLj7GlZfe4bEQl8opRbqdFADqlnS3taDSy2Wzm\nJlqEpc2LEO0nKCurw8mxusewtJpzrO6xVyere1ptgYqJdZ5ouQ3qfD4vrDRfQ7R5EaI9ET0xZouo\nB1HsHsPSclUPrKyKNvX9dEIAitq57xO7x2ppF4vFwWkzX0O0+RCiPYFUFQ/uy4oBEObhZuOcPIHn\nSZEa6QHBwsJyv2KNz6JvMS8vEyr43oRoS1CrhzisNgzH/cPDQ5HdhJNiHDrBDdaYbKo5m97rfpYF\nq3NkNYEi2p9eFiHaBN5+EqLlAna+DgYD+/HjR5FTjJGUEC0OnFS0nmBVWPv93rWyGE/JE9phVSFa\nbswWos2fEK1DWfd/nrejC/WwKUuLip26UwJ08QGU9jBm0XLPYu2mGEUB+ROiFbykCe3xhHxiiJLL\n7TBHlhMpOLSj3Sx0L5vab7J7zLnFLNq67jF/H9wH+RCidSjr/s+WFqEcHfiMHGNvT6uDpNU9Nju2\ntHziq6EezYKCa+y5x1zFE4LNlxBtCV7yBFtaiJZdYi5kx5VF6yVPYKnbqgUC3ohKboc6mUwOui2m\nmo3z9wjyI0SbIJVAwZYWVnU4HNo///xTZDzxYtFq32LvsMuzsF7zcW9PiwkB3H1R3eMgf0K0BKyp\nJu3jHtZTezqxe4xMJ97Hciy2DG1/qtU8XvyV47CLxeKoiCAyni6PxovWC+mk1v39vf348cOen5+L\nsA4Om7jUTmOx3umw3kOw2jmR12QysdfXVxuNRsUcWT10YrFH4sRl0ljReqM12P31YrEPDw/F/vXp\n6cmtj0XGUyoWW9Z5QhMn9DqdTu3l5aUQLYd3tPeT1xo1uAwaKVpPsJw7nIrDPjw8FE3YPEvLVlZF\ny5TFYd/f34u9Kru+GAI9Go2KWbKepYV7rQ3bgsuhkaIFGtrxToe5nxNCOdz6VC2t19/Jy3ry4rFs\naTWUw5U7uHqWVgsBQrSXR+NEW5Y84Z0Oe3FYXhrW8UaClKUpsrDU0k6n04MidmQ8IYkCnSg4vKPP\nGSmLl0fjRGvml9fpkCyOw3IRgFbt4B4jPLTjRJml1cFXbGkXi0XReeL19dVeX19tPB4XrjJf2dKm\nXO/gcmikaM3SnSc8SzscDu0///mPDQaDo30uitr7/X4x9BnPz9/HLL2X5TisZ2lfX1/t58+fNplM\n3AMqWNrtdlt8HxCCvTwaJ1qvcF1d41RTNsRhdQoAx2LLSLnGXnNx7lmMie2TyeSgVYy2jIlm482g\nUaJli6rF7FdXV0U3CQgS97xSjdjq4llZPu3V6e3c04lFqhMCwqI2h0aJ1uzX0GeEY/jKbWBUrPi4\nV8iuyRNVlKUo8vR2TqzQgdAh2ObSKNGypeXsJ27Gllqa7VQWhy0jtZeta2m1IVskTzSPRonW7Hiq\nHbeMUevqLY3DpsrrPKoEy9aWLS2LlnORORYbNIdGiVZbnaJzIlIPPeuqLrJXwH6Ka2yWFi8Lli0t\nC1cTJ6IYoHk0TrSee6yCZZGqcFn4egJdh5RgPeF6lpYzqFItZILLplGiNTPX0qKlqXdyrCIui8PW\npeogSq0sCxdf712DZtAo0er4Dh7czM3DtS8xl+eVkcop5ioeFqMKE7nEyHLi9ETEYoOgUaI1s4O8\nYJ4lm+r4f+pBkzeG0pvK7l3H47H9+++/RRUPT7YLaxqARonWm7nDIzy85AlN+C9DJwDo4r0pL1hU\nrpedTCZFMQBqY4PArGGiNTPXPeaDqFTGU11Ly/nDaLDG92VrPp/b29tbkbI4n8+LOtmwtAFolGh5\nFk9KtJxA4fUlLgPuMaynFrHD3UWLU73nIVpcdheWNmBCtBWW9pQ0RS5k9wZjoWAdB03co5jv+XHs\naQOlUaI1S7vH2j3xMwUBXF7H3Sfg8qI9TGqtVqtkrnFY2gA0SrRlB1FqaTlN8bOWFsOxxuOxvb6+\nFvtUr/8T9q+cnsjpjWFpA9Ao0TJecgM//mymEZfbaZiHD6ZgXbGHnc1mRbuYVKw3CMwaJlqc7uo8\nHBw68X4X1vfUZt+8b8ZzwZrf3NzY+/t76ZjLaBMTVNFY0WKsBgTLLVSRcNHv94tMpM+IViuJut2u\nrdfrg07LchMIAAACO0lEQVSNPMcnRBvUodGihWDxb1dXVweCRYy0bqE575k90ZZlWqXEGsINlMaK\nFodFsHK73a5oOdPv94sOh7/jHmv5nydanEyrYPU+CEAjRYsECBbsZrOxdrtdtE7Vrv11LS2X/mn5\nHxcfeJY25scGdWiUaHnGq9mxu4zJeA8PDwdDmf+UpU01Mk89DgKzhomWRQrLttlsDg6GBoPBQaJD\naiCzxzn2tEFQRaNEa/bL2uK+3W7bdru1drtt3W63SHZA/i9m6Dw8PFSKar1eF1/LucZcE8ujKKMx\nW/AZGidas1/7x1ardbCP3Gw2tlqtbDab2Xg8Ppjevtvt7O7urvR5N5vNwYBnLHwM2VFcDABrHqIN\n6tI40Wq2EbPZbGy5XNpsNrO3tze7vr62VqtV5BP3er3S595ut271Du65gIBn8ESaYnAKjRLtfr8/\nSmSAcJE3DEuLuTwQ7GKxsNvb29Lnx6l0qtCdR32EpQ0+S6NEa3YoXLjISKxg91gFO5lM7ObmpvS5\nOQbsXZGFxYXxsLRRxRPUpXGiNTuMgXIaIdxj7HVZsL1er7Kxm/aE0pUa9RGWNjiFxolWBYur2f9D\nNqvVyj4+Pmy9XhfFBNzJouq5vUns2teYRR2ld8GptMpeLK1W6+JfSTpgmhuRf6YheSodsay1agzR\nCjz2+737gmu8aIPgu5ISbf1xb0EQfAtCtEGQGSHaIMiMEG0QZEbpQVQQBN+PsLRBkBkh2iDIjBBt\nEGRGiDYIMiNEGwSZ8T/9Bj3DHeCbKAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f546e1111d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: 7\n"
     ]
    }
   ],
   "source": [
    "view_image(X_train[40000], y_train[40000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive model\n",
    "\n",
    "For each image, find the “most similar” image and guess\n",
    "that as the label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def similarity(image, images):\n",
    "    similarities = []\n",
    "    image = image.reshape((28, 28))\n",
    "    images = images.reshape((-1, 28, 28))\n",
    "    for i in range(images.shape[0]):\n",
    "        distance = np.sqrt(np.sum(image - images[i]) ** 2)\n",
    "        sim = 1 / distance\n",
    "        similarities.append(sim)\n",
    "    return similarities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.random.seed(52)\n",
    "small_train = np.random.choice(X_train.shape[0], 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADtCAYAAABTTfKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHpZJREFUeJztnWlXIlmzhSOZVGRQy+7///d6vVUlMsok90Pfne4M4iTY\nhUgm+1nrrEywREvdRJw4MWS73c6EENWh8d3fgBDic0i0QlQMiVaIiiHRClExJFohKoZEK0TFaJV9\nMMsynQcJ8U3sdrssel6WVoiKIdEKUTEkWiEqhkQrRMWQaIWoGBKtEBVDohWiYki0QlQMiVaIiiHR\nClExJFohKoZEK0TFkGiFqBgSrRAVQ6IVomJItEJUDIlWiIoh0QpRMSRaISqGRCtExZBohagYEq0Q\nFUOiFaJiSLRCVAyJVoiKIdEKUTEkWiEqhkQrRMWQaIWoGBKtEBVDohWiYki0QlQMiVaIiiHRClEx\nJFohKoZEK0TFkGiFqBgSrRAVQ6IVomJItEJUDIlWiIoh0QpRMSRaISqGRCtExZBohagYEq0QFUOi\nFaJiSLRCVAyJVoiKIdEKUTFa3/0NfAVZllmWZUfd82PPbrc7+LX8v8Hj3W4X3pe99jH/RohaiRYi\nbDQapavZbIb3EK4XXHRl+GPv7+/h2m634WulnjskenG91Eq0ZmaNRsNarZY1m8188eNWq5VcWZbt\niSYSkVlsFSHOzWYTXt/f3w++Pv8b3EdfT1wvtRItrGyz2bR2u22tVqtwxep0OuG10WgUxHJIQF7E\n2+3W1uu1rddr22w2+T0WrG30upGVNrPC1xXCrMaibbVa1ul09tbNzU24bm9vrdFoFETjhXTIbd5s\nNrZarWy1WtlyuSzcN5tN22w2e68biXW73Rb22LvdLvcChKiVaM0+3GNYUBYl1t3dXX7l+2azmYvG\ni8i7tmb7e9LNZmNvb2+FBcE2Gg1br9fJPS++zna7zf8veP0oSCaul1qJNsuywt4VooUwu91uYd3f\n3xcet1qtXDgsVjyX2uPifr1e22KxsPl8nrvcrVbLGo1/T9aazebea/M9B8Pwuu/v7xKtKFA70fKe\nttPpFCzq/f299Xq9fPnHLFoOILGQzfYjvFir1cpms1kuWFhYM8tF6d8Utttt/ryPXiOqja8rhFlN\nResFe39/vyfQXq9n/X6/VLR+HYr+rlarQsDLr+VyGYqW3yQ4gIV7XK9JvKmjtZSXk4ru1zEOUDvR\nwspCsL1ezwaDgfX7fev3+7mAeXW7Xbu7uwvdY74vOwaCaOGWY//c7XZzlxkR5DLhRtHnaxJtWczA\n7/2j31PZEVpdqJ1oeS/b7XZz0T48PFiv19vb12KxaKMgFB/DpP6w1ut1YR/d7XYLQSmI1u+X+Xw3\nOi66FtEeOr9+f39PeiHwUq7h2Kx2omVL60Xb7/fz/a2PHCN6HB3B8NGMWXzcg3NaCNYf+axWq0L0\nOApIRWe7/IfJkeW6kjoSQ3Q++rniikAfL1Ana3tVoh0MBnZ7e7t3BITnWLTevUplJ/Fz2+3Wbm9v\nQ+Gxaxwd9ZSJFusaLG3Zkdh6vc69lsVikd83m808nrHZbAo55XUSK6idaMvc48FgUEio8MkWyIgq\nWyC6h/B8CiNHocv+KCPRwkJfi2jLjsSWy6XN5/PCkRoEi/NsL1i8Rp2SU2on2lQg6vHx0QaDQSF1\n0d9DtGblRQMgepwSJj+fsijXLtoo+MdruVzadDq1m5ubPcFyQYZZcctSt3PuWokWZ7QpSzscDsN8\nZNzjTDV13HCIsmMIH1DxwsWezQuV7+u+p8XPIHVWvlgsrNPp5AUgLFgEofi1ooSVOlAr0R4LCwjv\nxNgT4eP+3zP+j4Af+9fyH/d7Zl6HCg5SUdBD31/quVNT9saGj5V9bxxFj4R7c3Oz5/ryFoSzx/Ax\nJLjUSbi1Ei1+UavVyt7e3mw2m9l4PLb7+3u7u7uz9XqdW1a2uLgvK80DURG9X/gjiZbHCxrPwWvA\nc8iawr+L3Hh+zVTh/6lICTS1peD/m/9+8Bx+f+12O0xuybKscLzjYwa73a7wxovXW6/XJ/t/XwK1\nEi3cq9VqZYvFwmazmU0mE7u7u7N2u51nLHkX2Ys2Om6A6wVBcuG8fy61vKj9c5Fo8TXLMrLYApe9\nYZxKtGWCjd5QsKJuIbz4Zx8FpBAdTi0OOHGMQJb2guGzPFjayWRinU7Hms3mnmj9nhbv9mWJFakO\nGHiM58ruo+fwR232IVp8vff3d2u1Wsk3k9SbSnQ9xc849VxZ1B2CirwR/t7KoutZlu1ZWba2vM2A\nYOUeXzh4d2VLC8FmWWbL5XJPqLyyLCuNXppZoSMGL+6YccwVC4JF5wyz/f5VLIqyP2qz4ptKZOn/\nlLL9furNhK2v90j4MX6Hqdc5xtJyOuhyuSxUWdWFWokW77K8p+U2Mog+Rsn8nU7HzGzvHZyvZnZQ\njLxPjvbNvN7f3/M3C+9CR3u/MveRvz9v0XE9tWijI5bUkRa77ylvhV8nstRetKnfFQTLlVaytBeK\n39PiDxV/1Le3t6XtZswsfAdH8APJG9FKWe+oxc3NzU3hDzkKXkXWCG9KXM7ny/pSXgCXCf7pz5iv\nfF/mBbAnkNoi8OtFgTZ0/4iCUPy7wps21zNLtBcKi/bt7a1wjrdarQqijZIszCw8asFzZlbqXqde\nmxcHlMw+EkIia+SFZxZ7AmyRU83svkq0XlypggiI9tCbSpn7DUsbZZrhMX73UU1zXaidaJGjulwu\nC8ENBKhSVhaiTZ2RbjYbM7OjLCovfu7m5saWy2V+Rc4zv6GUud4cZIncRDPb2y/z4z/94z10nFOW\n0YRAEnslZh9vNN7aRmA74YOI/P/l9RWR80uglqLl8zkfmdxsNtZut229Xu+Jz8wK54CRe4zPS+1b\nU3tm37MqWtEfIT9OfX9YZhZ+T6cSLX7GuJaJNlqNRqNQsIErJ0VEZ8pRcgp/Lf8GlqqvrQu1Eq1Z\nMavGBzi22621Wq1QeBBFWcK/mR0VES5L4EChgu8QyT2lUsIzK3oC/uotWSpN879SJlifoRSlIbZa\nrUJvrvv7+4IF9me2/j6KoqeE66PPdaJWouVIIyyj2UeAZLPZ7O33+LGZhRbCR2ejM1j/Wt5aopAh\n5ZqzaFNXMytNc8yyrNR9/wpL64VUFtVtt9s2GAzyhgAsWD6a8+6st7T8BpHaLnhLWydqJ1rOJ/bP\nHUp4MLPCO7hPtDD7OFI5NrmCH6eizIceY+12u9IqIDML3whwfyrRprKeINooqrvZbOzm5iYp2G63\nmz/nv8/IPfaWPVUCWTfBmtVMtGYfbjDu8Yvls8qyxIPorDGVcfTZNEbvRkd749RREYsWQvVXM0tG\nxmHJ/vRnW5b95EXr99/I/44Ei+QIZIBBuN49jiztMcKtk3hrJVoWF6fOpRIXojS61OE+H9Ecu6Kz\n15RrXuY+85EUWq345UUbrT8V7aGfDyL3kWjX63XBmrJgl8tlLlozKwgXgvW/Y/aArkmwZjUTrVmx\niVeUVZTaK/mUwdR99DmHXhNX71b766FjIzPbGznC/ZKyLCsEtjjohXY6f0KZawwPJ9VJcr1eW6/X\ny494UO/c7/dttVrlQgMQrBddytL66h/taStGlLVzCRxyoeEip9xbMyuI1AsXovWC/QrRHuMeeyvY\naDTyUSnwDsqql9hzMrOCK8x7+uVymS+8Lou3bsKtpWgvlbI/SDPL996IfGMfxxlFvIfl4w24nWx5\n/Nf96kCU/15hVXG9vb0tvBFxMonPEYblRhpqlmUFgaK5G3pKY6Hhm39TqBMS7TcQHZWAKCjG0evo\njJYzjjg7CuA1vuqcNuXZcFqlmRUa6nnRRkUDHngU3Esaop3NZjabzUoteV2QaM8MR0LNrJANZGa5\n4KK9G6xPlPjB1i067uJjsFP8H7xQ8f8qi6SzaNHriUXrEyj8iiwtW9n5fJ6LebVa5T8fiVb8MdEf\nkP/DgnXEHzTEnEr8gGj9cRdHWL+qNA9XPtaCCDlCHrnH/jyb97f+vNyLll3k2Wxm8/l8b28r91ic\nBP+H77OAsP/EHtcXiZeVvuHzOaHkq4rg/X2z+W8nTL+XhUC9e+wtLYs2ymPmIJzf00K0/uxa7rH4\nY3yiACcP4ONwmaMqFR/0iQJD2L9mWZZb2K8oT/NCQP4wunHA8rJgU3taTmHk/4cvu4usLLvHUVKH\nLK34Y8rOf/0e15/1RntJDwJS0ed/FbvdLnd3+WuzaNk95j0t3OkoCMdHPJF7DLHC0qZyx2VpxclJ\nuZ1/+lrnwL+h4MwZIz9R0YNB3nd3d4VyRM795mCbTx7BkQ7P8cGCmKNtg85pxdWTyjLzQsXg7n6/\nX5jygDnBsLzcxwuC5X0rrq+vrzaZTGw6neYCRpQ4Sl2MMqrqgEQrPoVPDeWFQJSfozQcDnPBYuQo\ni9a300HnEY4Kz+dze319tfF4nD8XJVFw1LyOgjWTaMV/IFUgEbnEsLCPj4+5YPv9vnW73aSlZdHO\n5/PcunpLC7c4Em1Z0kfVkWjFp/BWlgNIyJ2OZgM/PT3ZcDi0+/v7fN3d3RVECwuJoBNPiRiPx7lo\nD1naOrvGZhKt+A94wXLBg59Y2O/3C6KF64xh3uwec940u8ewsuPxOLe02OuirC+q6pFoxdUT1QtH\nok1ZWswH9hVIKfeYR7u8vr4ebWnNThuNvzQkWvFpohxjJEikAlEQbaprh+/rzJaWRQvLe0i0dUai\nFZ/Ct87xucUQbLfbza/Yw3a73bDHFpcgcjIFzl4562mxWBRqia9NsGYSrfgEsKypPlb9ft96vV4e\nGcaelRMofFqmWdxCJur95LtT1PlYpwyJVhwNV+1w43Xc4/wVVhYfi/KLfboiB6IisabaydQtr/gY\nJFpxNJFoEQnGHjYSrbe0DNcVH7Kyvo1MVDBxDUi04mi4oyQHnLrdbt6kDaJl9zgaS+LdY1//m2pE\nzpU7de5tXIZEK44mqo+9u7vLiwF4TxtZWpTsmcXVSoesbJmlvSYkWnE0kaVFdBiCZUsbucdR8kPU\nWudQEKruUwTKqNfgTvGl+D0t3GO2tGWBqKhmFtey6LG3tOwea08rhKUbrvPkPy/YwWAQusfcnYJb\nvLIocc8VPVwzy/OKUuMsrwmJVhTgs1R/z4EniBWCxeJzWi5yN/uYD5wabTIej+3Xr1/28vKSpyz6\nah6d00q0gmCBRhMBIVpYWOxl+/2+DYdDGwwGhT0tXGPOeuIURe7ztFgsctGORqO8OGA2m+XF7lHv\np2sUrkQrCnCDcT8oDNFitrToSoHFKYxsaVF6x6KdTqd5LvFsNrPX11d7eXnJLW2qblbusRD/jy8A\n8GM4/bksu8doIwO32EeOvaWdz+eFkjvUy/KKLK3v9SzRiquH3WM/Mzfa07J73Ov19gZZcxAKBe4Q\nLap32LpOJpPCYtGuVquwnYzcY3G1pASLuldkOXHlDrvHvV5vr/KnbE8LS/v792/7+fNnXnrHs3m8\nexx1prgmwZpJtMLhhcujMn0gCu4xLO39/X3hLNZfvXsMSwvRjkajveAUFg/OrnOB+zFItCKHBcvW\nFoLletnU4wi4stxwHDWy0+k0F+94PC40I/fzZq+xoidCohUFuGY2srBcI4vECZ/llFq+wTg3GfeF\n7X4otPhAohU50Z6WS/Ag2ig9MRrn4Tv9+8kAbFFTk9yv8Rz2EBKtyEmJll1hP/UuNcE9SlVMCZaF\nGw3LlltcRKIVBbxoceYaWdqUewxLyxlM6/W6MOYjcpF5ELQsbRqJVuTweA+OHPOeNrK0PNYDlpHH\nU3Lz8dR+Fo+V8XQYiVYUiCwtu8ef2dOiMsePpywLSKUGZku4H0i0IufYQBRHj8v2tDwIOiVaH4Qq\nG5gt/kWiFTmRaPnIh/OKfSDKrGhleWo7n8kiLdHvZbH/jdrQSLRFJNorhwveIyvrz2oh2MjKcr2s\nr+RBFQ/P4uHKnWjKncQaI9FeKanuFOgBFSVXpI58vGu8Wq3s7e0tzyFG8n9qFg9nO0mwh5For5DU\nJPcohTGytKkgFCytn3oXDdA6xtL6e/EvEu2V4ufMmsXusbe0HIjyZ7TsHsPSTqdTG4/HNhqNQtGm\nLK2/Fx9ItFeGd4tZtGXR4ygjikd9IAh1yNJijwv3OLK0ZhJsGRLtFeLnzOK5Q5bWu8dRNlRqTwvR\n4jl2j2FpJdTjkGjFXkO3lJWFa5yaz5OKHnMrGc6KgnusSp7PIdFeEX56u3+MM9howTX2FtZnQKG8\njovZsbedTqeFZAvez0q0xyPRXhF8pMML1pUbjvOkAN/DmC0rrKSv4uEKHraqUemdRPs5JNorgwdo\nsavb6XSSouXgE++DIbYsy2yz2RzMK8YeNqqXFccj0V4RXMHj3d7b29tQtBw1brfbe/N3OD/Y71dZ\nsLhX6d2fI9FeETyqstPp5BFhtEX1ovU1tO12e68rBfchPuQeR6V3+HxxPBLtFcGihXXldqh+iBbv\naeFKo40pxOaL3A9V8USCl5X9HBLtFZGaL8vjPVJ7WnaPt9utmVmhogcR40M1s+wOq/zuvyHRXhEp\nS+tHe0R7WiRUbLfbQpcKzoA6ZGmXy6WZxVPgxfFItDUkyivOsiw5W5anBPBgaN7LtlqtZI4xzmKR\nnujrZTlaLP4cibZmpJInsizL96lerMPh0B4eHvbmy/rhWdxwHGL1U+98QQDEKmt6OiTaGuEnBHDT\ntWazWQg68fAsCHcwGOTC9vNluSAArnBZvayfdCfRng6Jtkb4/GG//IhKFiz2s74ED24xAlBsaf1Y\nj9T0don2tEi0NSM19a7dboeWFntZiJZbzHQ6nb20RV8vy1U84/G40AdKov0aJNoakepbzB0VeU/L\nUeOHhwe7v78vDJHm4oDI0qKKJ6qXlWi/Dom2Rhwa6xENg2YXORpVydU83BaVLS2X3s3n8zyCLNF+\nDRJtzShrNn7IPe52u/nrRCJLtZNh9xjnsxLt1yHR1gS2sj55AguJE0iewHkspyumOvy/v78Xehmz\nOHlyuy/BU27x6ZFoK0iq/Sn2sdi/QqA8rR0BJ3+s4wvao8l3KGovy3zSqMqvR6KtGD7biR+zlYUr\nDLHy0Q4sLUTrpwRApDx2EqKNJt95wWoo9Nci0VYQ35iNo8Y85Q6VO4gOH7K0ZsWpd4gU89S7ssl3\nqXpZVfKcFom2YrBIEdnF4uATV+8Mh0N7enrK0xQ5VZGznsyKlhZihSAh2pSbjJ5PGlX5tUi0FcRb\nWBzNwD1mS9vv9+3h4cEeHx8LaYoIPvn8YuxpvWi5UVtZJY+vlZVoT49EWyGiRuNcIBAFouAePz4+\n2nA43Gsx4wNRhybfeeFyK5nVapWLNKqbFadBoq0YkYX1ljYKREG0PrWR+xdz6xhvacvcYtXLnheJ\ntkL4Fqi+MICPeKLV7Xb3PtdnPCFVkS0syu+m02kh44mLAlQvez4k2grRaDRyS8pJ/bgOh0N7fn7O\n96++NpZFyu4wxMZi9WV3nPWk3OLvRaKtENx9Asn/fB0Oh/bjx4/cFYZouWLHz97hvScPzuLhWZxb\nXDYUWpwHibYC8CjKVqtlt7e3hUodziN+fHzMRcvnsbC0/Hpm/57LZllm7+/ve7WycItfX1+Toyol\n2vMj0V44LDB2j5FLjIQJv9g95j5Pfngz36fcY7a0yDGez+cS7Tch0V4wLFizD9HC0uIM9unpyZ6e\nnnKXmAsDfL+n1JGMLwjw7vFoNLLxeFw49pFovweJ9kLxgkXkmC3tYDCwh4cH+/Hjh/311195eZ1f\n7B4jQgy4QMBbWkSMIdrJZLJ3NivRnh+JtgJEe1q4x160PHsHVwSifAKFr+qJ9rTsHo/H4718ZIn2\n/Ei0F4iv3sE9z+HxqYrYy3KLGT8AGnBBAIsP+1XsWblOFs9xQYAm330PEu2FkSoG4IIAHlMZCRTJ\nE9wuxszC9ER2d9miotdTlEDBxQAS7PmRaC8M35uJ7yFYXl60fli0P5flYVmwplgQLc5ifa8nFi0L\nVqI9LxLtBXEor5g7JaasLf4dNyrnDChYWh9swnksW1r0L0b7mM1mE1bwSLTnRaK9MNg19tMCyiwt\nzmKj6QJsab1oEWiClUWUOGVpo95REu55kWgvDLa0LL6UlfUW11tn3wbVu8c40nl5ecmtrLe0XrQq\nufteJNoLgiPGZW5x2Z42CmCVWVq4xRAtXGVEjL2l5XmyKr37HiTaC8MLNprLEwmWz2LxOvyaZsW+\nxcvlsmBpR6ORjUajwsjKyNJ6gUqw50ei/UaiNEWeDOAX0hO5TzGfwzabzeSU9d1uV2gZwzWy2NdO\nJpNCsTu6UfB5rPh+JNozkxr4jCMdZDTx9Do8Rp7xw8NDXsFzc3OzN2/Hn6fiuel0mlfrQKRwhbmw\nPdW7WFwGEu03EO07fV4xTwDAPeplWbS3t7eFHk9RphOuk8nkoGjhOsvCXi4S7ZmJzmJZtMgrRlM2\nHpSFXk/e0rbbbWs0GmHGE1+5xC4SLVtZtrQ60rksJNozkkqewNEMixZF7Q8PD4VG4xiY5d1jP/iZ\nOyhyxhNbWo4Uw9LyRAHN4rlMJNoz489hvWi50TgsK9fLwlWG6+xFG0WHfYldmXvMe2A1HL9MJNpv\nIDrW4Ubj3JXi8fHRnp+f7fn52QaDwV6QigNREFqUPMEZT2XucWpinkR7OUi0ZyaVpsiNxtnSPj09\n2fPzs/399982GAz2ehZjleUWc8YTN2jzol0ul3utaCTWy0OiPSNsYX2iBBe3p4Y/9/v9Qs/iVO9i\n7EfRWRFnsmjK5qe1c/BJXD4S7RnhZuPcvxgLIyg5eYKbsvnqHT46Akis4AkBPETLH+doz1o9JNoz\nwqKFK8x7UxYtnvfjO1iwXrS+fQxbXJ63Ex3piOog0Z4Zv3+9u7vLl7e03N8JljYlWN//ySdZQLhR\ntpMsbbWQaM9I5B5jH4teT949jlrI+MQM33ycLa13jdnaytJWE4n2jGBaOywtN2dDn+JoT8vCjdIf\n2T1OWVrvHsvSVheJ9oxEe1rM4vGijQSL4c9R0QEos7RlwpVoq4NEe0bY0vIgrUOWlqPH/vWY/2pp\nJdhqIdF+AVEBOrdAxV4WVhb7WYg2Eiz2s4fwoz7Y6qqbYj2QaE8MB4f8PYTKYuWpdwhEcRCK3WIh\nzCTakxIVA/A9zl591hOX3/GeFvvZYyysuB4k2hODfavvpNhsNguWltMUuXYWH+fzWVlawUi0JyTq\noshBJN7HsnsMK9vr9QrDs/C5fKwjhER7YnwCBS/u95Ta06bm8ki0Aki0JyQqt/NTALylxZ4Wk9t9\nu1RZWuGRaE8MC5dFy8c8KUvb6/X2glfHHvWI60GiPSGRYL1oubLHr5ubm70ieU5ZPNQoPJqv4ycB\n+KkAOqOtHhLtifHC9YJFwgTvV30hgE9P9PWyUXcJ9IfySRRRIkXUlULirQ4S7QlJRY8hXC5694Xt\nqUHSqVRFP0UAqYspwaZEq4yo6iHRnhjOL472tFERgC+zKxMsrlHztUOC5ceiuki0J8RbWh85ZsFG\n7jECTmWVPD6vmO+5Z3FZnrGsbLWRaE8MZ0RxgQAv7x6zpcVr8JVJFQRERQEQbJlwRfWQaE8I5x5H\n7nG0p/WBqFQACrBofWPxskCUglD1QaI9MZxcwUc+HD2OXGR2j1N4KxuV3kVusd/XytpWG4n2hESz\neqLCgVTwiYksIfat0US81WqVz+fBBHc/EFpucj2QaC+MyI2FyDabjb29vSXXZDKxX79+2a9fv2w0\nGtl4PLbZbBZ2YYzcZlENJNoLIzrOwXOr1aowBY+nBcxms3z+7Gg0speXl1y0mDuLKXheuKJaSLQX\nRCoyjHuMr8SID8ziwf1kMskHa+HKoi3LlBLVQaK9MKIAE+55fCUPiMYkPIjUL97b+jNeucbVQ6K9\nIPxxDkeEN5tNPlBrOp3a6+urvby82O/fv/PrdDq1xWKRB6H4Hu4x75HV3K2aSLQXhneP0QoVovXj\nK3/9+mU/f/60nz9/2nQ6LQzcwj0WpgkoK6raSLQXBkTkz1/9dHcW7f/+9z/7559/bDqdFhItfJYU\ngk5KrKg2Eu0JSVlJtn6LxSKZythut8PzV9zzPhbT3Pl+Op0mJ7lvt9vv/vGIEyHRnhBYSB7oDEGa\nWUHMPPB5MpnY6+urNZvNwmQAHkm5Xq/zc9jfv3/baDSyyWRSiAwfUzcrqo9Ee0JgaSHazwi21+vl\nok2t2WyWn8G+vr7adDq1+Xxuy+WyEA1WgKneSLQnhC3tarUqCJYHPEOwmNmDK0QbHfcgG8qfwy4W\nC1sul7ml9fnF+L5EfZBoTwiL9pBguYAA941GI0yqwBWBKM6Ems/nBfdYrnH9kWhPCES72WzMbF+w\nvg8yV/tg/EcqhZHdbiRL4Ord46iRm6gPEu0JgWjNPvawsLpRpY+vAMqyrLTDBHen8Fe2tPhe+PsS\n9SEr+4VmWabf9ieJ+jylVqo0LxJcVPXjLbH/XFFtdrtd2KFelvbEyCUVX41a1wtRMSRaISqGRCtE\nxZBohagYpdFjIcTlIUsrRMWQaIWoGBKtEBVDohWiYki0QlSM/wO3Ns7mrbgJAgAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f546e13f780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: \n"
     ]
    }
   ],
   "source": [
    "view_image(X_test[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "similarities = similarity(X_test[0], X_train[small_train])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADtCAYAAABTTfKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGlRJREFUeJztnWlv28gSRUuyNmtNAgSY///3XhJbsvbFfh8Gl74sVVOU\nx6FF6h6AEOWM19FRVXdXdbfe3t5MCFEf2l/9AwghrkPSClEzJK0QNUPSClEzJK0QNUPSClEzOkX/\n2Gq1tB4kxBfx9vbWij6uSCtEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0\nQtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQM\nSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStE\nzZC0QtQMSStEzZC0QtQMSStEzZC0QtQMSStEzZC0QtSMzlf/AFXTbret1WolH80se/T3l3h7e7PX\n11d7fX3N3eMS4jO4K2nb7bZ1u13rdrvW6XSyezx/eHiwVquVXRAZV4q3tzczMzudTrbf7+1wONh+\nvz+7l7jiM7hLaQeDQXh1u91MVn95cSEqPz8cDrbZbGy9Xtt6vc7uzcyOx6OkFZ/CXUnbarUyaUej\nkY3HYxuNRtnV7/ft4eHB2u12+AhpWVi+32639vLyYovFIns0+1fYdlvTB+JzuCtpOdKOx2ObTqc2\nm82yx8FgkKXJDw8PuXvIC0kjcdfrtT09PWVfx+xfYbfbraQVn8bdSjsajWw2m9mPHz/sx48f9v37\ndxuNRtbpdJKXl9YLvFwuQ2FXq9VVE1pCFHGX0j4+Ptp4PM6k/fnzp/38+dPG43HhRBWiZUrcxWIR\nCtvtdhVpxadxV9L6Me10Os2k/eeff2w2m1m327Verxc+FkXat7c3m8/nZvYu7HK5tOfnZ+v1epJW\nfBqNlTZaa0X07Pf7NhgMbDgc2mg0sslkYrPZzGazWU5Ufx9J61Pk6XRq4/HYhsOhPT4+Wr/ft263\nm42Jm46fVb/0cXE9jZOW11X9FaW+GK/yZBMeU+uz0XIPf39eKsKEFq6m49/EUo/+vxfladyryFc5\n8TorIqcXNhL3UmFF6sXGhRk884zv2WQ4+4gykre3N2u1Wrl/4+eiHI2SloXhSIf7lLAsF0dZLm00\nK44SeAHi57jHSOsFRTlnJDN/jsS9jsa9iqJIBzFZ2lS0jaqgisSNvj8Ly9LeQ6TlumvIyB9LfZ4o\nTyOl5QjH0nB6nEqNfbS9lBr7F5yP9l7cJsNNEvg7QNTX19dsIq9IYHGZRr2KImEQ4fy6a5kxbZG4\nZX6Ge4u03NF0Op3O/nYs6jXZi8jTKGnN3qWJhLkmyl47nuXv78fUSM3vIdKeTqdM2NPpdPbfQFyu\n49aY9joa9yrid3ffZnfp+mg/LV50kBRrwY+Pj9k68HQ6bfw6LSIsruPxmHvOl/9v8fniMo2T1s9e\ncrrGL5jUBQFT64oA/x0/RmWS+/0+e1GiTa+pvL6+2vF4TF6Hw6HwAkVzBqKB0pqlxfUC+4/z5/j1\nxAgvbrvdtl6vl0nLje+dTsc2m01lf4Ov4PX1NZQRwu52u+zabrfZIz43Wss1k7iexknLovFSQ5ko\ny7OeqXVFM8u9qFjch4eHTNrJZJJ9vU6nY4PBwHa7XaV/i6rBzh3Rrh2Hw8G22222QQBKO83eIzSL\na5Zf+5a47zROWrPiSMvi8v2lIgBPJC6nx0iJO52O9ft9Gw6HuRSwiRyPR9vv97bb7cLH9Xqdq8U2\nexd2v9/b8Xg8+5qS9ZxGSRtV5EQpsY+4/N/yWuIleb24SI85JUZjwnQ6DV+UTeJ4POZSX3+PNkUW\nFmkzbzLg13Elbp5GSQvwP73dbufS3kjYSN6obzYFzy4jPeaUGKnh4XBo/OwoWhI3m41tt9uzazAY\nZH9bRNjdbmebzSabwdeS0GUaJ62PtLxeWFbYa9JkBukxUmL/tZv+wsPGdpAW97h6vZ6ZnQvb6/Vy\nrYv4/2ZmEjagcdKaxeJCXqwfppYiOFV+eHjIpdZ+DRhEa7mIHEzR51x6nvqY/70vfSw1K1tmDO/v\n/ePxeMx6j6PLzLIxLnaq7Pf7WXnp4XDI/Z35XvK+0zhpvbBm7y8qCMuzmhh3bTabrGHA99b6Kinf\nTOCrp1IU9fr65gRfIPKRvwHu/d8ldZX9+aO/Bd6gfPno6XTKvvZ+v7fHx8fswta1/X7f+v3+WTEG\n/z7incZJC7y4KLFjaSEspI3qjstUUnGDQRGpF33UVeRlLSvuJTFTE3BlxtuprWUfHh5yBSaoCuMh\nQavVyqSN9pzu9/vZmi6PZ9VccE6jpI2iLE9uQNhIXEyGRJE0agJIReMi+GtEj6nohc/9yN8gWq/2\nS154TK1Jg3a7fdY5xb8X1313Op1cpvPw8GCHwyEZZSEtv/nxPIPS43caJa1ZPjXkF00UaX20LZKE\nhYsaDni3xhSR9Bg3dzqdbK2Xv3/Z1Nv/DaKImqoDRlrKUkTfk3uS/Ro1nuN35Of4GKQdDAahuPv9\nPvwdosaDe6Zx0prlx0AsLkvLZXWItP7z/dgw1eqH+zLSRrLje+DFju/lU9syE1GXCkuimmB83M8B\nADzHG4xfm8a6a9FzfA8faVlcSOvfZD4yrm8yjZPWl8DxhA7PFkcTUWb5GlifXqJ4IrXN6iVp/QQN\nRyxPNEtd9vdPpcR+ttwPF3ipxf8cZpZ7g4GQHM15xhz/jjQZ398Ly+nxbrfLvdngOBUJm6dx0pqd\niwsuTURFVVT8HNJimYLvj8djFmFS8ASNL5CPJp586nnN7x/9Ln55iws/eLmFfwZ+9GNYnymY2dl4\nnH/+VKSFtKgmQ4Tlsb54p5HSMj5V5o4TFLBzepaapDmdTtZut0Nh8VhW2lS0RokfX9ykXyY9TvWx\nIspyIb9/9JHWS4s3J57kw+/Eb5CpnzO1fa0fMlzqcb53Gi8tgxf1fr/PHdkB2fBOXxRpU9KVTY/x\ngmVRcR/touFnqosommw6nU5nLXOpSJsqnuj3+zaZTHIlmSjd1MxuddyVtNxRst1uc8KeTqczaf3s\na6uV3vC8zERUakdIf7B16roUbaLGCB9ti5rQeSLKy9tqtWwwGOTGvhB2MBhoLbVC7kpaH2khGaIQ\nxmfR+ibSv2j2l7dfLQKR+tIhX6l14LLpcWpbFz/x5CekuOTTC9tqtWw4HOaExblIx+NRkbZC7k5a\nRNqoPQwznalqIrxYP1pcwQ0FXlwWP/W1y0pbFHFTM8cYqxaVV47HYzOzXIRFH6ykrY67kxaR1uw9\nwnJ7WFGtLpcd+gmTMlKxtF5WnoRKVU2VSY/95FlRtOUlIIiXqgRrtVq23W5zwo5GI9vtdpK2Yu5K\nWoxp+R4RlpuwzeKDo6IXMj+/RFRNVbTvsr+/NtL6ybSiTddY2lRDBOYBIOx6vc6k1Zi2Ou5KWqTH\nPg0sipI+gkRLEGWXJaLa3aLljtTyR9Hvl6ovjiqi/HMUSHhxcY+C//F4bNPp1DabjdLjL+CupDV7\nj0ZfASayeP216HSDa9PvSFq/hMXruLyem5KWH3u9XtbgjuUi3pBNVMPdSfvVcLUS6qH531Ite9dI\nm2q980LzeF3UB0lbMb6g37e+paJcmaL5qO44kjcSV9QHSfsFcL8vnqfGk9dMdEVdPkVdP76LSdQD\nSVshXiD/8UvrpNd+j5TEUSQW9UHSVgxHN660KqpGwn3Zr8/fp2jdWelxPZG0XwALahbLGS0plYUl\njNabo3tRHyRtxaR6fc2K5bw20pb5uIStJ5L2C/HS1F2iuv/8dUFbAogkf6sBvYzcegNII2nFGZGk\nXyGuiJG0opBo36qPLEVFNG14UBWSVoRcM3v9meKKy0haUYq/ubGaxL0OSSuSREtSKXn/q9QqpyyP\npBWFRKJ+VNBrhZTAMVqnFRloTPD7U+His2R5IzovcWr3j6KjSbgxnw8EU/vgOZJWZPAui9FOkbPZ\nzCaTiY1GIxsMBpnAvBVOqsPo7e0tO9UBj3yWEi78Oxrs/cFgQtIKgqXFyQl8isJsNrPxeGzD4dAe\nHx+t3+/npE21AOLi85O8sLwjhpdWnUh5JK3IYGn7/X7u3J3Hx0ebTqdZpI2kNSveEdJLyxEWl9+P\nWcKeI2lFRiTtaDSy4XBoo9Eolx6npOVI6/egitJjH2n92FaR9hxJKzIiaYfDoU0mE5tMJhel5fTY\nH/wFab2wPuJGezVL2DySVmRA2k6nk4u0EHY6nWZjWkxEYSfJKNL6zdEhbVGKnNrfSuK+I2lFRio9\nnkwmNp1Ok5GW969CtI2O2PSTUFF6nGrWF+9IWpHhpR0MBjYcDrPNyX2kTaXH/twglpXHtf6oTZz+\nIIqRtCIDxRVYo4W0iLbj8TiLslF6zONZPrwbETXa6BzjV1EeSSsyonVaTpEnk4kNh8Ms0kZVUV5a\nHCu62WwKpVUKXB5JKzI40uJkPEg7Ho9tMpnYYDDIUmMfaXkCCsIiHYa4nCYr0n4MSSsyePYY0vr0\nmCukUMbIE1FResypcRRpVVt8HZJWZHhpo/Q4OhQb6TFPRKXSY0RajWk/jqQVGUiPMab16fF4PM6d\n8MfXpYkojWk/D0krMqJIy+nxZDJJHgwWTURhTOsnoriTR9Jej6S9Q1InGiDt7fV6uXErun36/f7Z\n5/CjL2OMiioUZf87kvbOSB1W3W63c+WJmBn2SzqX9kL2tcdcZMGy+uM2RXkk7R3BO1NEF4omUOmE\naqei83GjXSt8tMX4FtL6KKvZ4+uQtHcES4uZX1zdbjcnLa/B+i1liqJsqsunKNpK2OuQtHcGpPXj\n10haH2nNLm/q5rt8LqXHirLXI2nvCK544hliSOobAfyY9tKujP7s2zLpscS9Hkl7R/j0GNKiNLFM\npOWvFeGjrNLjz0fS3hlRegxhU2NabnK/Nj0uI6ykvQ5tVn5HIMWNxOU0GakxC3vNYVvRroycDmvW\n+L8hae8MLy4unklGSly01CO+Dkl7Z0BYXJGwuFhccTtI2juCx6VFkdZXSyna3haS9s64NtJK2NtD\n0t4ZPKblaJsa0yo9vj0k7R3BqW4qPY4iLT5X3AaS9s7w/bCp9FiR9naRtHeKH9tG7XrXRllfxlh0\nip7Onv04qogSH8bL5sWMzvPhc3q4hFHilkfSiqthwfwxHpGwKXFVxvgxJK34ED5CRmlwKtJGJY2i\nPJJWXIWPsv6gLL/VjBeXP6Yx7ceQtOJq/CSSn3xKRVpOl32Ulbjl0eyx+DCRtGWEVXr835C04mp8\nShyJW5Qe+9ljSXsdkvbO8NVQ0V5RvpyR12yvkZV3reCzaFMpsiiHpL0jogO2/LEf/pR3dP0wZcew\nfHIeNiyHuNq94uNoIuqO8Gf1RAdsXTo0umgc67eYYXH5dIEo2orySNo7osxRlqPRyIbD4dlRlkWb\nkpeNtDiTNrVhuSiHpL0jio6yxKHRw+EwGWnN4uWelLiIqnyWj5+g0pj2eiTtHZE66X04HGaRFh/j\nHRl9pC1KjSNhOT1ObfImyiNp7whE2tT5s5PJJLczI+/KyETRFiJy+huJm+oCkrjlkbR3BEfabrd7\ndv7seDzODt6KTno3SzcIRMs9Pj3e7/fJZgNRHknbMLgH1vfEchTFqQJ+o3JuiOe1WrP32uJoA/LD\n4WCr1Sp5eDQmnsR/R9I2CL/3k9/ADTPD0fEfkah+xhiHae12u0xKvn96erI/f/7YfD635XJpm83G\ndrudHY9HRdNPRNI2CH/AFkvY6XRySzopcVNbzSDKHg4H2263ttlsbL1e567n52d7enqyxWIhaf8i\nkrZh+NJEfiwjbWrLGTPLUuPdbmfr9dpeXl5suVxmj5CWIy3WZlVA8XlI2gbhl3QwfsUVSevTY96t\n0dccI9JC2uVyafP5PLuen5/t+fnZFouFrVYrW6/XirR/AUnbIKLZYT7CkqVFlPXHWvIElh/Temlf\nXl5sPp/bnz9/7OnpyZ6fn+3l5SWXHiPSStrPQ9I2iFTF03A4tOFwaOPxOIuy0bGWnU4n+zoRPj1G\npH16erJfv37Z8/NzFmExk6xI+/lI2gaRirQQdjQaJdNjpMhm8fppNBEFaf/8+WP/+9//bD6f23a7\nzV0a034+krZBRJEWxROoePLpMdcXt9vtpLCoesKaLM8gYzLq5eUlV0iBNVpVPH0ukrZhRLXF3HqH\nFDnVEGBW3BSQ6uJhUaMuHvF5SNoGwbXFPJ4dj8c2nU5tOp2e9cvyBJRZXKbIuyxeqi3mXlr1y/4d\nJG2DSPXLIjVORdqinSm8tJe6eKL9jdUv+7lI2gYRdfFgTDudTs/GtD7SprZG5c3YfHrsG93VL/v3\nkbQNgmeP/cwxIq2vhioa06a2kklF2t1ulzxsS3wekrZBpCItpJ1Op5msPj1O7UxxTaM7Wu/UL/t3\nkbQNokx6jK1SectUH2nN0gdq+ckoP6ZVv+zfR9LWiKhHlq9UjyzuHx8fs8IL7urx9cU+DcY9CiaQ\nCnOvrPplq0PS3jAc/SAmNhn3V7vdttlsZrPZrHArVG69828CmHSKdlHc7XY2n8/t5eUlV6KIPYwV\nTatD0t4g/gR2PCL1RZT0W8NgLZal9VVPReIiynKjO19ouVutVlnEVYStHkl7o/gU2MzCtju+n06n\nNpvNcicF+EibirJm+Ui72+3OGt0Xi4W9vLzYer1WpP1CJO2NUTRmjXZS5LHrbDYL12N5/Fo2PYa0\nq9XKlstl1hyASIu2O0lbPZL2RvHC+jVYlCiicwczxH5MG6XH0XYyZsXSLhaLLNJyeixpq0fS3iAp\nYX2k5a1PsQ57aSKKN327FGm3222u2R0TUT491jpstUjaGyOaMYZgqCv2zQBIiyEsp8dR+120DxTK\nGHn2GJGWpeX0mCeiJG11SNobgiWNLn+UB0fYb9++ZV08vmc2tek4NwO0Wq3cxuKYhEJ6jH7Z5XJp\n6/U6a3BXelw9kvbG4AOfEVlx78exXFOMCSj8e+o8Hn+KOz+uVqvswuQTLoxlISxSY+xKIWmrQ9Le\nEDxLjHVYfvTC4uL1WV8N5SufonpiXIisXl5EWGzWhtRYW8l8DZL2hvATTph0Qo0wNmjzURaRdjwe\n5479SEVaLlXkR6zHRtEW4qKEkWeOlR5Xi6S9MSAtF1H4/Z58eoxIiwO0eJNynoQCqfN4okjL8q5W\nq1xpo9Ljr0HS3hAcafmUADQB+Ejrx7Sj0ejsSBBc2LSNI60/0Y63P43Gtcvl8qyJQNJWj6S9MaJI\nyzPGqUkoSOsP3+K1WdQIc6TlQ7RSY1qeiOI2Pb4X1SFpb4hoTOulLYq0w+EwPCGAGwKiSAtheUwb\nRdn1ep1rbveXqAZJ+wWkuniQDvNWMViPxSwxKqCwFuu3j2G8SFEXDwooUESBsSuqnvis2d1uV/j1\nRTVI2orhSiR/H+3pxBf3y3IzQNEWqHzPkdU3A6DqCYdnQdZU8YSE/TokbYVw+uuLKNrtdraxOKRF\neSJfXFuc2gY1SmH58Cwev+LArMVikWtyx4l3kJbXYiXs1yJpK8R36/iLx6ss6bdv3zKBITRvg+oj\nbWrfYuzpFHXwcEMAHwiNmWLt93Q7SNqK8Us6fPlIi5rib9++2ffv3zNZcXGk9fsW+x0UEWl9eoxI\n+/z8HDYEYC9jTTbdDpK2QqJ1WBRP9Hq9LNLyGBbC/vjxw8bjcW7zttSYlmeIU9LyLDGkXSwWuVnk\nojGt+DokbcVExRMoPfQlikiPWVpf8YTaYt/Mntr61M8ac6RdLBa5PaHU5H6bSNoK8Q0BiLDoj+X0\nmMezkHY0GoXdP9E2qKnzd3yk5V7ZxWJxtgtjNBElvhZJWyFliyf8xT2yvoGd7zm6QlCuL+Ya4qjy\nab1e52qR+fQ7cTtI2oqJyhS5RJELJfgEgNRuE5wWY4YYKTBvg4p9i5+enrJUOGpoT514p/T4dpC0\nFcLLPT7KIpJG0vJWMakyRbN/U2NIy9ufYt11sVjYnz9/QmkRYblVT2nxbSJpK4RPCPBboZaNtNGm\nbwDSYn8nv03MfD635+fnM2m5od0fU6kOnttD0laIj7ScHnMtsW9g550nUsKa5dNjSMtrsJhs4q1Q\nUa6I9Niv7So1vj0kbcVEW6HyJBRHWkxW+UhrFp9A4CPtarXKpP39+3cWXX3nDkvrZ58l7u0haSsk\nGtNGE1G8KRvvpIj6Yr/NKvBjWqTET09P9vv3b5vP57niCS6iQHqstrvbR9JWSGrT8aKJKE6PuSkg\nguuL1+t1lh4/PT3Zr1+/bD6fnx2qhQvSmuXriyXs7SFpK4bXVn0NMkdWLp7wKXAqEqLSyTezYzLq\n5eUlm3TyZ8xqprg+SNoagRMA/Awv7nmSyTezs6C8IZvGrPVD0tYIPrbDVy4dDoezmWEIyylwqoBC\n1AdJWyP8AVmc5u52u0xaLlf028XwLopa1qknkrZGRKfaYR+nzWZzlh7z7DD3xnLVE1JkUR8kbY1I\nnR+LKxrT+mM8WFSlx/VE0taIoqMol8tlVp6IMa0/MGu/359tQaO12PohaWtEKtKikR1lipwe85h2\nv9+HS0UStl5I2orhw6/8nk29Xi9ck+V0FjXEXEvMzyNhedYYXxdI2PohaSsE4nEXTrfbzSqdcJ4O\nyhr94+l0yqTkggm+/MZsfrsYSVp/JG2F+IJ+NAOY/Sv0drvNNmxDOSNfr6+v4a4TPH7le78FqjYc\nbwaStkK4oJ+FxTh1vV6fHW/J96+vr7klHj62I/p4arNx/CyinkjaCuGC/s1mY2aWm1jidjxfj9zt\ndu319fWsoKLo4rpijWObg6StEE6PzfLCrtfr3NYy3DSAe5zHc82FIgot6zQHSVshkNYsL6zfDpU3\nceOzf6KGgbKXqp6aQ6vo3bfVaumt+ZPhfZ6iezM7207mUmte0ZmxKp6oL29vb63o44q0FaOIJ/4r\nxVshCCFuDkkrRM2QtELUDEkrRM2QtELUDEkrRM2QtELUDEkrRM2QtELUDEkrRM2QtELUDEkrRM0o\n7PIRQtweirRC1AxJK0TNkLRC1AxJK0TNkLRC1Iz/A3nZFjoFTCiwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f546e05bd68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: \n"
     ]
    }
   ],
   "source": [
    "view_image(X_train[small_train[np.argmax(similarities)]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets try an other example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADtCAYAAABTTfKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztXWl3qtqyLVtUbJLsff///7vj3J3OBlHj+3DeXHtS1gJN\nbEBrjrEGaBIwyKT6qtZ+vxeHw9EctG/9ARwOx2lw0jocDYOT1uFoGJy0DkfD4KR1OBoGJ63D0TB0\ny37YarU8HuRw3Aj7/b5lve+S1uFoGJy0DkfD4KR1OBoGJ63D0TA4aR2OhsFJ63A0DE5ah6NhcNI6\nHA2Dk9bhaBictA5Hw+CkdTgaBietw9EwOGkdjobBSetwNAxOWoejYXDSOhwNg5PW4WgYnLQOR8Pg\npHU4GgYnrcPRMDhpHY6GwUnrcDQMTlqHo2Fw0jocDYOT1uFoGJy0DkfDUDoWxGGj1WpJq9UK+7zV\n+/p1u90Of6/XPWC/3x9s9X7V69ixHP/CSXsi2u32wWq1WgdkFJGD1+12W7rdrnQ6Hel0Ogf7TSfu\nfr+Xr6+vg4X3d7tdYWu9h98Fgfk9x79w0p4AkBNE04tJrBf+rt/vS6/Xk36/f7DfbjfbWgH5ttut\n7Ha7wsJ72+1WttutbDabsI+F32US47itVssl7v/DSXsCmHy9Xk+63W5YvV5POp1OQepqSdzr9WQw\nGBysJElkMBhIt3v4dVzqRj1Wqp9y/q+vr0BGa7vZbCTP87DlhZ+D2O12W7bbbeHYjn/hpD0RIC2I\nyhKz2+2a6jNWkiQyGo3CStO08LrX64XzXEuqxMj7nfPvdrsDIvLr9XodVpZlB1sQtd1uy2azCZ/D\nJW0RTtoTAIkJ6drv9yVJEkmSJBCX1WfsYzsYDGQ8HstkMilssd/v90UkTpif3rQxgur3v3v+3W5X\nIKJeWZbJarWS1Woly+VSVqtVsOtxbfM8D5+HbWHHXzhpT4BWj0FaqLlQkWNrOBzKbDYrrOl0GvaT\nJDGJcW4JY5GXifLd82+3W1mtVoGcWZYV9pfLpSwWC1kul+EhB8JanwOEbbqtf244aU+EpR4PBgMZ\nDofS7/eD5LC2o9FInp+fC+vl5SXsJ0lycL4m2bSbzUaWy2VYkKjYXywWwYZnCSsiwZuMc4Kw+J2m\ne9bPCSetghWuwXuQrMPh0LRL+WbkLfbTNJWnpyd5enoKRMXrp6cnGQwGhc+iCXNu9biKCKeef7PZ\nFEwGmA3YZ5ufryvbrZCqeG+73R5I40eHk5bA6i9uLrZJkySRNE3DGo/HhddJkpjxV+yPRiOZzWYy\nmUxkOBxKkiSmiihyfsLiGHwe/fqn5+frB5t/t9sVkiQ4BsvHbLVa4Tqx9N3tdrLZbALRT/1M9wgn\nLQE3DodyeGlHkl4gbWwNBoPwt6PRSAaDgfT7/QPSXvNmtIj7k/OzzQ+Cikjw/jJpGRwqw89BWEho\n64HziMR10hLYO2wlQYxGI5lMJtE1GAxM7zH2EfKBSs3Oq3M4go5BjKTnOL+WtLBR8TBkQjJYZRYp\nStj1el1Qq/H3ONYjEtdJq8BOJjhN4CFO01Qmk4lMp1NzMWm1et1ut4PTipMqmLTX8BzjmOcK8zC0\nd11ECteB/0edLcZSFskY6/U6XB9LPeb/5ZGI66QlaEkLpxPWeDyOhmtms1kgbWzhuHpZmVAil1WT\ny+zZn5wf/6dI0U4F+QAmLX5HRAJh8zyXLMvC9bHUYxznkQgr4qQtgG1aDuVAnWWSwuPL+0xaliB4\nHQsFHeOIugROcUQdA/b+4v/9+voK9i2IqVM9Qdr9fh8Iu1qtCppImaR9NDhpCZwjzJIWnmKoxlbY\n5vn5OZBW35SsBpbti1z/JmSJdY7z8/+tS+5QycQPNPayf319BQm7XC4DaS1Jqz//I8FJS7AkLRxH\nICw7nthznKapDIdDs7rn2JrZW96A5zg3x7hjP+cSPH5gIO94tVqFcBjndEPacplelQPtXuGkJZSp\nx0zU4XBYcCSxJLCSMhx/oTPKOCyU53nB+adJC3Vb5C9J+e8fBU5aglaPtaSdTCbBvtXJEU7aaljJ\nFyxtQVoQF6Rl4uK67na7g2v9KOR10hK0pEVcFfasTowok7R8TMe/YG9xr9cLJIN9m+d5eBiCtJzP\nzd7nR25F46Ql8A0VU4+RZ4xsJkvS4lhVNt4jApKWCQtCbjabIGl1nTKIC3CSxaNdXyctgeO0SKhg\nSYsECpAZ0oAzdvhYvHUU1WP9utfryWazOZC0UI85LMT9ox7x+jppFWLeY3iOdRVLWUjiEW+oMsTi\nuHAmxWxalrRM2Ha7/ZDEddISYuox27R48mspEIsjOorgRBNtl5aRlr3H0GpA2Ee77k5aAsdWWeLq\nmlB9I2l71mGjymTQD0N97fv9fuE6s4rs3mOH4wbQNi638xmNRqGIgIsLHrGHlJPWURuwI1CbKKPR\nSPI8N0v4Hs00cdI6agVtmrCkRadGbkXT7XZls9k4aR2OWyCmHqM0Ms9zEZHQ1QJlfI/WrdFJ66gN\nLPVY27Tc1WKz2Txkt0YnraM20J57dA+BTctjQzCx4BHDbU5aR60QU49h06JIHq1okNjySHDSKug0\nOZ76tt1uCwXcVitQx/dRFiMfDoeyXq9DkbyeneSS9kHBrTt5JCPPo9G9ejmX1vFzcM8obdvGsqQe\nibAiTtoD8LBjnvqWZZkMBoNC/SfS6lzangeckcZDzjRpuVzPJe2Dg9ViS9JmWSYixXIy7u/r+Dm4\nd1SZpH3knG8nrQLbsSxpQVyLsC5pzwPdUpVzkC31mKXtI8FJSyiTtFCPWQqA3C5pzwdL0iZJIpvN\nJtrszeO0Dw7tMWZJm2VZoWB7u926pD0jLJu2yhHl6vGDQ0taZN2wI4pVNkhZl7TngzY/+v1+0Hpi\n3uNHI66TlqBVY0jY1Wolq9UqTLhjewsEPjZpnWO7evHvfBdWR0ird5X1fp1Q1T/60VRihpOWwDFa\nJiy63XNMUBfMdzqdyrpOVr1ZBT+XbcxOHGtyH4dI9HCwukgrndjCvgXs8/XyvseOgmoM0qIXFPc3\n0mSAY6oM7JG2tj8t5kbPZqiPrA2wSskzd/G6Lh5YnYnG1TyauLEB1fcOJy2B1WOWtDomaM2hOYa0\n2+1WsiyTLMuCY4tfb7fb0s9WBXSR1M3n8JpjnLrv8DFT9C4NlrJYLGlBWhCWiftIcNIStHqsc1x1\nHFGTtkq9zfNclstldGnSnnoz9nq9MLIENajY54Zpu92u0DOYm6zdGlbeNwhbJmkfCU5agvYcYwo5\n27GWhIXkqiLter2W+Xwu8/lcPj8/D/ZR5P1d9Pv90PIV40vSNJU8z03bGf9LnbzfsUINLWm1TftI\nxHXSErSk5Wwb3RGf7UKQtko9Xq/X8vn5KR8fH/L+/n6wXa/XP/r8SZKEdq/YgrDaacP/R11UTKvC\nCqTVDimXtA4RObRpQVju+BcjbL/fr5RYq9VK5vO5vL+/y+vra1hvb2/y+voacpu/i8FgINPpVFar\nlWRZViCsSLGFKf4PEKAuNz7btDFHlHuPHQXwE557EsEGZK+rTqNLkqT02KvVSt7e3uTt7U3e39/D\nwntZlpU6g8rirUCSJKHuVN/kt/a46ubkOjatc72zLAsxctj9cNqxx/3RiOukJcSK30GKTqcjq9Wq\nECJhydzv90uPn2VZQSVeLBayWq0kz/Nw8+lkBz2Bj+OrHGftdDoyGAwKQ64x6Jrn6fIoE+1guwZY\nOur91Woli8Ui2PlsSuBB9/n5Ga7ber0OfaOctA8MVs3Ym7vf7wM50MYTnk0ODZUhz/PgfJrP57JY\nLA7UWO6WbxFYq+e8MMKEHVHsOeZJfxzGulaM1orB8uvlchlIC+KCtFh83UDaOqn314CTlqDjhExa\nEMqSsFmWyXK5lG63/HJuNpug6kHtsyStJfk4PIOcXB6d0ev1woQ/Ji0TNzbz9VqStszBtN1ugwrM\nhGVJCymL6+ekdYhI8cbi1zyxTUtYnvNThu12W2hdA9vNkrScfQVwBYxOoOCxnBh8DcJq0mr1+JrZ\nUNpnwEUZi8UiLC1p397eZD6fByebS1pHAEiKfUxnY2+lTr7Aqrr5OQYcC2Fw0zh8Bq0eW428Mfya\nlyVpYdPyzNdr2bP8QARZ+eHF6rG2aaEa4/fxN07aB4cmLKvErVYrkExnRLEn+Zjj8+L3NLjjo641\n5Z7AnEiRpmlQkS2bFoS/RRG5bpzH/bdgYrBNC/X44+ND3t7eZLFYRB94jwQnrQITV6ToBKpaVdDH\n0vusElshGW3TJkkSyKpt2Zik5VGdt1aPuSsIbH2ox5b3eLlcFjSeR80/dtIaOFdtqwbivDFJXVVW\nxyEdznziMA9IGlOL2dvMJXnXdERp4nIPLi6i4Dgt9nUN8iNmRTlprwh2IllLl8vpLUtWLVF1TBaq\nM5xPVlLIrforaUkZqy22TIlY44BHgpP2ioCkhbeXnUisvsaWdjzpxVJVl+bpjvy8rgmdwKJDQLpJ\nQIysTlrHVQBJC4kJlRZbOIo4p1nnN4PwvMU+S22O4WrHk3ZAXdN7bBUFsJQtk7bnbs3TVDhprwhI\nWpTQjcdjmU6nMp1OZTabyXA4DGTjrX7PIqXuUGFtrUL+WxFX27YWYa3yOw6FWTnMjwAn7RXBkhYp\nh7PZTJ6fn+X5+VnSNI3au6ziaoeSbh/D9b5WfyirWdo1oSWozpAqs2uBRyWsiJP2qmi1WqH59mg0\nkslkIrPZTF5eXuT3798ymUwKrWF0uxhLxbWKB2KrDl0YLUmrpWysyF0T9BEJK+KkvSrYEQX1+Onp\nKZB2NpsVbFSdpog0yWNapJbt4zVvrwFt05Z1pnz07hRlcNKeiJh6qXOFLWkGddhaT09PMp1Oo6Q9\nJrf5J8BnvAYxLIeUh3eOh5P2BPC4Cu5cwfakdvLwfpqm8vv3b/n165f8/v1bXl5eZDabyWQyCSGb\na49ytKqJLkWQWK3wrbtANg1O2hPA6q21OAnfahI+Go2C0+np6Umen59lNpvJeDwOiRBWn+JrEVa/\n79KtnnDSngA0A4dNqhdnHlle3eFwGEI8k8kk7KOUTqcaXnJWzbG50k7c+sFJewIgaVG7OplMwoK0\n1F39eSVJclA6x21hMHqEwzWXCMtYx4pJVydu/eCkPQE6o2kymcjT01NwJEHa6qVDONw4nMvmtB3L\noZxzoYyw2L8kScvsWrdtj4OT9gTE4qy/fv2Sl5cXGY/HpteXu0boGTv8mlu/xOKql/ifrPeYuNeU\ntk7cajhpTwBL2jRNg6R9eXmR//znPzKbzQoVNnq/1+sVPMqxxAfGJdXismM7cesLJ+0R4CJ1nk7O\nnSNg2+oZOvxad2usujnvLQOItQiecFDmB9CedJHy/smPACetgZjdpafl6X0dn9WSk/s9Wa8ZlyDs\nrc8vUnzwJUkS8o3RPytWCD8cDkVEoj2T6zSP6NJw0ipYqX9YmqB6/zvF5RZxrik5rnl+PMy4OR0P\nBONRoNy1YjQaha4Vulcyd818FInrpI3AcgRZ0tVK1I/lBYvESVKW0HDOm/GW5wdpWdJy47rdbhdt\nNzMcDgvzannyw6OQFXDSKmiyWYOk9b6lIpclRJwi3S5xQ97y/CxpcWw8EHe7XaEfFKTscrmU4XAY\nWq/yteWumY9CXietgSrClklaK1xjocye5N+5FG5xfpa0TFi8x5KWSQvibrdbk7DeQvWBYam1Wnpa\nhC1r4/Jd4l6r2uaa54dE1RIWIze/vr5C/2NOD4UHHqTF54O6/GiJGU5aA1Zo4ljCxhxRVeS4lX12\n7fPjWmkJCw8wzzpi8g6HQ9lsNiJSHJJ27oyxJsBJqxBLr/vO4uNU4db22DXOb2V7YU7S19dXSEYB\nUdM0DTbuer0WESmM52T1eLPZmA3fbn1dLwEnrePq4Adau90OM4w6nU5hcsJ4PC7M6oHHmeuNQd7d\nbhf2dYH9vYWDnLSOq4LVcE5a2e/3wauMbLLxeBwIi7JI7t/M0hZjSa1GcPc2NsRJ67gJLLMBpIWk\nBWFbrVahSwhLWJ5iaPWdAu5J2jppHVeHdtCBUFCPB4NBQcLyEG1LwmLynuVNvieyAk5ax1UBwmlJ\ni/APZ0pBwoLI/X5fRCRIVBAWw71h1+o47r0lXzhpHVeHRVgRKWRKsYSFR7nf75sSdrVaSZIkISQk\nUpyFe28xXCet42qIkRUAaTVhkWuMAgMtYZfLpSRJInmeh2OB3JfsZHkrOGkVrFrN7y7reLeEFcfU\n+0DsRi8jwE/JAbJygUa32y3M9eEMKe6CCScVJ1ycWnHVFDhpDXCMj6HHM+ptrCN+HQgL6NilFccs\nSxSxsrzOTQidRorPXZWVdkyV1T3ASUuISUyQt4ywx4xlvDXYMaM/M8IjuughVtTP22OKD46FJiuj\nKo3UIu89wklrQJMVX741ZyY2lrGumThWLBMzdESkcNMzKUBMi1DnJCznQutznTJ8zCXtA0GrtBwu\niKnDVfNU60JcLWk3m02hqByxUl7IDdZN0zWhzi1pRaSQ4ojzHUPceyasiJM2CpCVV5kdWzVPtS7g\nGOd2u5XNZhMWSIusI26mBujUQ37vXODj4Tz4bMdK2nt0QAFOWgUtFdlei0lVTdiYJ/nW0JIWpF2v\n1yFcAm8tCtX1Z9eqsSb1TxHLlsK5v0PYeyOuk9aAVpHZpi2TtGWDkOtGXBA2z/MQ80QlDf8PIn9V\nYe180uucKrLeL5O02hH1kxLJJqDWpI0F460tf7ll27L9GPgYlqPpHNJVp9lZ+zp2zPvHxJXRohRt\nSrGPtd/vQ44vuiXyPg8Gw/5+vw9VNz+B/v6s7z7W0qcqzHNPhBWpIWk1EfVrqx8TlohUkipGsEtL\nwlOSFcqIFwvX8L61vr6+gmSFOsxrvV5Lq9UKBD1lIRHiHNfl1BzheyPkMagdaUUO25dq72FsiUjU\n1rTsTv36FJwjyaCMsNaDhkM0WPo1O5bgHdbv5Xluvm61WoWBYXqfR5ygpanI3/TDU1H1IIt9J6dc\n53skde1IaxGV7RZr9itS2ESkEHfkfTQOY8kDwB47h7Q99iYp+z0mq5aoMcKxFIUk1eovk5hJzn2E\nrWHZIC8K09M0Ddev3W6HRP5zX6dTvpN7JGcMtSKttkm0Z5DrKi2bS0SiN2Wn0ym04OQC6XOHZqpu\nIOvn2rljqb/acaQJyq1Hrf08zw8eZPxw63Q6B5P+eApgmqaFRAxIWMR4f3J9YtK1jLjfuc73gFqR\nVqS8EyIPZ2YJgK2IHEigbrcreZ6bcTuQ4xxfbsxpVvV7+j3LfmVioRwNRORu/MvlUhaLRdjq/TzP\nS9MwO51OYV6unqGLQnORYpeJU0hb9f9XSddjJfS9ElakhqQVOUylg8cSEjU2mFlEgiTq9XqyXq/N\nMADApD2Xesw49gbTsCQtMpi4hnS5XBbWYrGQz89Pmc/nhS324SHWzjpsMeU+Nqozz/PwQEUvJ0jv\n7167Mq0j9jp2jHsnK1A70lqSlm1ZJi2PksRUNaiMCENYieNakn03XHEqKY+RxjFJy3FV1JFCgi4W\ni0DQj48PeX9/l/f397CPLUiL8+htr9eLjupE32Em7Gg0kjzPj5a0x2oj+Jkm7rF/e++oFWlBVj2z\nFLYrGn4Nh8OD7XA4lFarVYg/JkkSjUmu1+tC2RdPX2PoeKlOTmDplyTJgeMMC97VMvWQyWmtLMtk\nPp/LfD4PROXXHx8fJmmxuEjc+izsUIJPAd9FWSWTdd0cl0OtSKs7FmgVWD/59Wq1WlHPKYiF7vVY\nnBiA/NuyBQ/uer2W5XIp8/k8DIze7/fBlmYnD25oDotY5OVjW1stXbXdCmm7WCzCg4r7BvN5rTg4\nptxDiqZpGrzF4/FYZrOZTKdTSdNURqNRKEBHd8Qq6Kypsiyqsoen9fqRUCvS8o0DKYobJE3Tg8nq\nlnqskwbY2wpJhRschGVJa8VxRf5KWUhC2JXz+Vz6/X6YUcOeax3/hYdb/8+Ans+qnU2WHattWpB4\ntVoVSMuIxcH1hPvxeBwm3E8mE5nNZjKZTGQ8HofOEaeQ1oJF3FMJaWWK3TNqSdp+v2/eNGmaHjhH\neNtqtYKks7bL5TJ09dOEhZRlNRDg8jwtaXGsVqsVwjI4Bt9E7Xa7MFPVwmazKRDT2mctQb/WC6TV\necSxODhMEC1pp9OpzGazgqQFafHAOpa0MZKWJVRoE+WYc9wzeWtFWqjHHMifTqfy9PQks9lMxuPx\nQUiCSdxqtUqzgRaLhSRJYhI2y7LQuY87OYgUn+T8+0xYK8UQNyPsW5a01k2e53nBsaQdTTruaoV9\ntFkANR2fpSwODknLWs5kMgnfwXQ6DQ/P0WgUzIJTJe0p0rWMfFXEvFfi1oq0lqSdzWby/PwsLy8v\nB6TlcM9gMJBWq3WQ0sevoRKz1ERp2mq1CskXGDGh47hcPA5Jy8fS2VbaC142RxX2OIdpdNhmuVwG\nYjJJmazWw0rbtGVxcPYfgLSz2SyQNk3TgqT9rnpcZs/y75xyvEdBLUkLL3GapjKdTuX5+Vl+/fol\n0+n0IEuHyYtMJ53xg+1oNApOJ02+fr8f4pAifxuJoXsC3kOCgx7+VEYOEKIq8wo29+fnpxmyWS6X\n0TRFmAGxggG+qavi4Kwea0nLvoSf2rSnOKKO/dlPfrcpqBVp2Xus1ePfv38H0mrCYrFqa1W5wFml\nCTscDkNYSeSwBI+TL/C3XFWEY3G8UodM0LNXg29aOLZA0re3t8JCVpNesNlx/tjSzicdB9c2LXwK\nkLSTyeQgGw2k/S60V/u7TqhHQq1Iy3FaDv2wjRVLZkeMVOfrasdSmSNHRA5K1/TNxOSFYwnkRvI8\n50RzrFbHSTWyLCuoxnotl8vSih1W5bXDSUQK5OQt9sfjsbmgEqdpepDzrXtHfRfHOpgcNSOthe+q\nSXzTAjzgCao3S6gkSQ5CLlh4IPDNrm9WlrpZlslisSio0VbIh7Fer+Xj46Ngv+LzsWqtVW+Wplbr\nFZaqVsEFtuPxWH79+iXPz88hvIPkFW4Kzg+icxDWcRpqT1rAykzS0N5RnZ7Inuk0TQuEbbX+LUvj\nuCfba5CusP909z/OlEIiBH4GiV9VcwrvMTzHTFr9PzMR+Wf8+bg5G9ussZrZ8XgsT09PgbSQskxa\n7l5x7w3U6orGkFbk+PgbCAuVFe9pexmqM+zPwWAQ0gI5lisigZBacrH6DJUZKZLaDkbNbwwcp4XK\nzpI25uTi9yz1F1toGdrEwHvQPhCTRXgHNj9istZDy3E91JK0saT2Y8A3EBN3v98XSAsiMJlx87L6\nKyIFCRprHAZJCyeVJmyWZZWkRUaUDuOUqcf8HmzqWMsYHSLTiSqj0ShkPGGLhBZcFx0mcil7fdSK\ntOd0NOBGAmFF/qrHFmFx4+psKSZs7ObUCf8gLKc8HuNl5fI7djRZklYTFvWw7JjTUlWnfnKxBRan\njXL6KA91jjVYc1wHtSJtGY4ltL55QCARCQSF9GXCQqIg+UKkKGGzLCs4fPCZeEHS6iQMVifLoEvx\ndMiKrwETBg4oTvi30jxB1Niyamixz5lkWtNwwl4XtSftqXmnIoeN1vB3kERaJYZUGwwGpkqMeK4u\nTbN6HLNKfOrNXRZj1YTlhxH+Vif8c/kiFodv9IKmoR1VeM2agr7GjuuhdqRlKaU7NaDeU6uJ8JJa\nxOD34GkVkcLfsVprdSuEmqvrSvX+reOIvV6vVJJqtRcLr6FpcC0z7HvueHlpWA9qndtd1mnz3mt9\na0VaHTKZz+fBnup2u7LdbsPNBzJxV0ArKV5LOR0S4hsRGUFWOKjT6chqtYpK2++0YT030FFCq8RW\nGaNWh3GNmaSxNj2XhGVyYL+qFayuY7YIfA+oFWnhvEECP9eqQgqOx+OCc4azqHQWEBwmIodeZeuL\n5DYq2lmFxAtda6v7LN0SbNPG0j0t7zFUYS1VbxHS4Wurr3MZWWOkvTfCitSUtNylgT25XLnDEha2\nqo6hAuxBjiVetFqtYA+yhOWEDHQz1BLgpzeHJsV3j4PrULasLpb4mU7MuHZIh6+nlY5aRlZd0aUl\n7T2hVqSFegxJy2V0McLCoYTKG8v2slRjJi4nYwwGg4I3lr3Lx7SjORUxQnznWJxbbCVYlC1oNByH\nvZWkZftVN7arWhZh703a1oq0WtIyYfGUtQgLm80az6i9rExYtoHhfWUJC8KigECnE1ohoFNwjDf5\nFLBzzUpj1MOz9D7bsbeKw2rCsuS0OpLEpK01SeJeUEvSYhgUwieQvKiS0YRFK0/dGQJJB1oVZpsW\nUhVk1d0zWOViVesnWVv4fMdek1OOWVYwoKWoNds15ri7BrR6XDWn6BjCunp8YbDtwgSGkwQpglrC\nIrF+s9mEY7E0xdI3o/4yO52OfH19hdCSZb9aOKda/JNjW0SLec6ryHmr+KulHnPP5yoPcswRdU/E\nrRVpIVlBKI7ZwiE1Go1CQj3yc7ltKW4+EFDn7DKsG/NasUgNnQhyj+e3js3v6Ybseov2O7rjJH//\nrBW59/hK0DE67vbAX4oe9Ygt1L6mPGXL0i7v8fyW447jsNYAMbz++PiQ//3vf/Lnzx95f3+X+Xwe\n6o4tJ5Srx1cEt3hhWCTVBG6SPRNTQa8ldW9xfm238n5ZQ3ZMUEDrHW4WwP2ddV+suj+0v4PakZa/\nVP1+lZSNZcLU8Us7xma8pNS71fm1o4m3nFSjW+2ApBh9AlVZS9p7l7IiNSOtVp1AXOxb6rEmMNuy\ndbVpLMLEpNsliHOr88disFg8WOzz81Pe3t5Cgzt0pdS9oCFpOSQXK+a4F9SKtCLFLxY3DDy/mqDW\nft1T2MoIg/1LfuZbn18Tl787Ji1U4T9//sjr66v8+fNHPj8/o0OzrckO90hYkRqSVqRo0yLWyqQt\nIy76CzflSRsjUVmCSJPPH0ue4Ja2aCML0v7zzz/yzz//hBm7PJ+J28hypVXdv/efoHak5Yuubyjt\nfIpJ2bpK2mNCTvyzcxPn1ucXObRpOWmCHVFQj0Ha//73v/L5+Wl+9xzm0ee6R9SOtAx90XWwHT2U\nuN6Ws5pbAZDSAAAH80lEQVSQc3uPqWxNBB6iOmGCGw3w9D89LHs+n5fW0t4rSTVqTVoNfOE8fwdP\nZUwYgIqEBAsUrtfhC9Xag6VN8M/KXjf1/CAsmhvwQuIET/2Dg6ksKlA3jerSaDRp0RAcnQKhwjFh\nMY6jrl+qRZxrftZrnl+TFg9etI3lbCed7cax13sP6VSh0aRdrVYHQ6CgHvNcmjqRNkaSsoSGc372\nW56fQ3fcRgiqMKbYM2l1phOOcc+Opio0nrR6ngyPbBwOhwdexTrgFOl2qaykW5yfvz/LxImpx1Y4\n55GJ23jS8tiOVqtVkLB6qHKdUGZP8u/c0/ljkhaqsVaPeSwKvkcc51EJK9Jw0uo5O61WqzCrh3NS\n6/jlnuIIuofzx2xa9hLHKni4oJ0dUJf6rHVGo0nLhEWfYXQaxJdeZ9KKFEdo8ut7PL+WtJq0rB7H\nbNpLf8YmoHGkxZfOg53xfq/Xi1aILBYLETlM2dPbqn39t1XvnfK/3RLHnJ9/x9q30gd5aTUY+5Cy\nyCXmMZ9cCHDra1QXNIq0InbSOaSszqjRs3nSNA3e5di2rEWLJjK2t+rycAtYZGSvrlUIgLVYLOTt\n7U1eX1/D4mIAHvHJMVonbBGNIq0mLIreQRztjdTjKheLRbRPkjXTVTdAi03ME3mc8Rg6TsqEtfo4\ncTuYxWJRqNrB9v39PUha9hzr7puOf9Eo0orIQU/c7XYbfsaktebLjkaj0m6FZS1GRSQ4vlgyA49A\nWn5oslTltkCcWop9ToRBPSykK+8vFovC73M2lEvav2gcaVnSamkH0upxlbihRqNRISdZ7yNcxE28\nIUlwHj2x4NGGKmvCcrGGznDSA7Kt4nZey+WykI/s6rGNRpHWetLj/a+vrzBvJzYQejgcHjTo5n10\neEQWFdQyJG1wfS+GefG6d/Ja5gkv7jwBRxMn/lsOKF5ZlpkNDpy0RTSKtCLFShF+jf7GFmFxM/Hc\nGmucI0aCWITl4+J93j4K9EOTJz9on4JuEQPvMIisK3owdkW3oeHmfo6GkZYJitcY5wFC6QnufCPx\nwClrOFWe5wXCwlkF1VnPZ2XJ+yjghyZLWx13ha2KRmxwNlmqM7bW2BU+p+NfNIq0IoftaPAeXud5\nHt5njyacG1Wkjaln6BbI3mTtYa6a9K49z+zUuobEroqjWr/PQCN5q3NEnucFRxNIi/X29hZUYG4X\nw6V5LlGPQ+NIK1LM4uEbywoF4ffZFuNwxHq9DmqyLqrnouzRaCRJkhw4rnhbRVo9a0d7sS9NXE4j\n1DFUNEXTv8+As8lqJM6khTcY0tXKJ+ZyOw/pnIZGkhbQNxWTlqUXq8scS9QOKZYAy+VShsOhLBaL\nMIgZZYDswOJjVJEWoyjZ8cUT665BWj1ig7dMHkvywtyIreVyGTzB7Cnm+Cv3dGInk6u/x6ORpI2p\ncuhcwYTV4QmQdb1eH4R9+v2+rFarwoR03teOK72qRop0u93CMGeQBJ7oSwPXgrtGQLNg9TT2WUBa\n/jveWq1i2OmEronWWErH8WgcaTm0wo4KdG0EaUXkwMsZG/vIc12tyelsA/OCBxqrirS9Xk/SNC1k\n+aBg/1pVPVwWB/UfiweYWdhutwctYnhZMVret3wGXCfrOA6NIy1gfcncapUJq9MVrS3Io0NBeh8k\nZumLbbdbfjmTJDEJa9mTlwBsWiYtJ+tjlGgM2+32wInEziTtWGIpDlvWyk92wp6GRpLWckSxBEYI\nCITVntrYltVkKwEDtbpabcZ+FWlZJeaWONe6cdmmBWkRR31/f5f1el369xiQpYdjcdNwy7NsJf/r\nvGUn7vFoJGkBy9up60Oryu94yyEc7RnGoOnRaBQcU1h4r4q0w+FQRIqEHQ6HVyUtS1rYne/v7/L6\n+ipZlpX+PUjLqYla/dU2K79mcnoM9vtoNGkt6JviFEDaxqp8+v3+QW4sS5Mq0qJwX8dpsQaDwUVv\n4NVqVYidcvLD29vb0aSNLd0aRjcVd3KeB3dH2p9C14lywL/dbkue50Gd1nm4VaSFo0f3ScIYjCRJ\nzM9zLmDGKwir949Rj7Wdqoc6c3/iR25zekk4aQ3wLCEGumUwYTnrqsp7vF6vA8E5uR7kQQngpbBe\nrwuOJ72tckRZyRW6JYx2MLn6e344aRV0fjO/LyIH4SROVqgiLeKU3GFjNBpJmqYyGo0OSHvum32z\n2YRYqg73LBaLQm2yBdjDurid1WKXspeHk5ZgtVHB+9qJAonZ6/VCTnJVRlS32y1UHOlVpV7/FFVx\n1irS8kOKY60saXVnC5e254eTVkEXJHCtLFe4YIA1JOwxBQOo9+UsKk5pvHRmFPKu2YHGr6syk6p6\nQFlEdcKeH05aA2zTItMK+4j98uJYbxliBQPHVgmd4/9i21N7eatIyxqHVoN1DrET9nJw0ipYyRpA\nVTfGYxL+reZw12xZY0nCY21PK7aq963fd5wXTtoS+E3oqCMuq485HI6zw0nrcDQMTlqHo2Fw0joc\nDYOT1uFoGJy0DkfD4KR1OBoGJ63D0TA4aR2OhsFJ63A0DE5ah6NhcNI6HA2Dk9bhaBictA5Hw+Ck\ndTgaBietw9EwOGkdjobBSetwNAxOWoejYXDSOhwNg5PW4WgYnLQOR8PgpHU4GgYnrcPRMDhpHY6G\nwUnrcDQMTlqHo2Fw0jocDUPLh0o5HM2CS1qHo2Fw0jocDYOT1uFoGJy0DkfD4KR1OBqG/wOyyZjE\n0ZO8SQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f546e05b9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: \n"
     ]
    }
   ],
   "source": [
    "view_image(X_test[200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADtCAYAAABTTfKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfWt34srOtLgbAkxm9jn///edZ08mCfeLgffDu6qnLNQG\nggluUK3Vy4bJcHUhtVSSGofDQRwORzpo3vsFOByOy+CkdTgSg5PW4UgMTlqHIzE4aR2OxOCkdTgS\nQ7vsHxuNhueDHI474XA4NKz73dI6HInBSetwJAYnrcORGJy0DkdicNI6HInBSetwJAYnrcORGJy0\nDkdicNI6HInBSetwJAYnrcORGJy0DkdicNI6HInBSetwJAYnrcORGJy0DkdicNI6HInBSetwJAYn\nrcORGJy0DkdicNI6HInBSetwJAYnrcORGJy0DkdicNI6HInBSetwJIbSsSCOx0Oj0QhHLH2bV7PZ\nLPzdtTgcDkdHfb7f78O5Xg4n7VMBJGw2m9JqtcI5bmO1223z2Gxe55iBkGUrz3PJ81x2u13hmOf5\nEXGflcRO2icCSNtut83V6XSk0+lIt9s1V6vVuur59/u97Ha7AiH59na7lc1mc7QajYYcDgfZ7Xbm\n4z4beZ20TwZYzm63WyBpp9ORXq8n/X5fsiyTLMuOztvt6y6X/X4v2+1WttttICmfr9drWa1Wslwu\nw7HZbAbC7vf7QODD4RDOnw1O2icCW1qQlFe/35eXlxd5eXmRwWAQznG70+mc/TwWdrudaUmxlsul\nzOdz6Xa70m63AynzPJfNZhMIDIC4OH8WOGmfCI1Go2Bp2bKCsKPRqLDG43E473a7Fz8fI8/zYE3X\n63XhfLVayWKxCIRlC7vZbKTVakme50fP8UxkBZy0TwTL0mZZFizpaDSSHz9+yOvr69Hx9fVVer3e\nl59XRGS73RZcX30+m80KhIWFxX662WzKfr8vPLaT1vHQYNKypYUrPBqN5PX1VX79+mWufr9/1XNv\nNhtZLBbRlWVZIei02WyCBUbqiYnr7rHjIRDLtzYajUBSJurLy4sMh0MZDocyHo9lPB7Ljx8/woKV\n/fnzp2RZdvK5y7Ddbo/20RydbjQaIYIMsvZ6Pel0OgULLCIhKCXyXIQVcdI+DEBM5Fp17rXVakmv\n15PhcCij0SgQVa+Xl5ewz8X+8txUD8hTRl68LrjoWZaF/7fdbsNzM6FB2na7feQeg7zPFEl20iYM\nTQ7OwSKdw+dZlh0RlG+PRqNAWpAGFu4SRVSMvHBvW62WdDod2e12hfQNkxbE5ffRarWOHvNZiMpw\n0iYKvnhxDkLAQrELCteYycruMfa0MUv7FRmjRV4dweb7QFr8aOC1s6WFwILJXpXEMhU4aROERVhc\nvOx2skBC72M1cfn2YDC4ytJqMHnZ0vJ97XZbttutDAaD8Nq1eww3/XA4SKvVCrJIJ60jGTBhReQo\nMjwYDAJRLbEE38Z9g8HgyNJeqzlm4DXideP2brc7y9KK/LWy0E3H5I2PCidtotCEZSvW7XaDddWB\nJovEOI/tJ7/qHluvGT8A7CZDomjtaVluydaVq488EOWoNcpcYy2c4PzreDwuuL9MXrawvB+uyj1m\ncL5Vl+TleX4yesyVQlWXDaYCJ20NYVlRHHWNK5Mgy7JARkSDOe86HA7D3hZHPmfrytatKvdYvx8N\n/FjwHpzfz263k/V6HWSNeF0gMvDo5XtO2pqBSaiPXAerc7HNZlP6/X5BGKGFEi8vL2YFj7V3xGN+\npyWDpwAvYTQayWKxkPV6LXmeS7fbDdJHyB85QMU1t49cQO+krRFYIGERVNe/Yk+INRgMAlFZ3YTz\nl5eXI0VSLOADt/g7idtoNAqu/XA4DIQ9HA7S7XZlPp8H2SMTFlFkdL2AC82dMB4FTtqaAUTRpLTq\nYLkWttPpBNLGFsrrWDrIt/E8vKrcz57z3mFp+/2+jEYjyfM87F+73a70+32ZTqeFqDY6XjBZd7ud\nNBqNcHykQJWTtkZgS8vKJlY1sVhCCyiYtNjP8jkK2fkx+aglkGxpv+v9s3sMwuJ+vF8OjoGgm82m\n0BkDeDTXWMRJWzvwPtXqLIF9KPaifJvrYbkOFrezLDuypNq68v5Zp1S+470zaZGL5YL9TqdTCECh\nfG+5XIa2NfxadZDqEeCkrRGYMLC0nIJBAIkXor+xInZevV7vKLClV6xC6LvePwgKwrK73Ov1CntY\nLqpfLBaFInktc3T32HET6Eoddom1FDGmdkIFD4jKVT0oYmci6nM+6vNbg0nK52hHg3pbEJZL+Lrd\nrmw2m/BYOpf7SHDSfhNiogi+D64w171a2uEyVRNX8GDhcS5tF3MP4MfKEouISEj3YKFtzXq9FhGR\nzWYj6/W6QFaQ/FHgpL0BylRLZQIJBJNiiiUthrAEEhDcc/F4VTLEW0NLMrG/ByDPRDpos9mEVqwi\nIr1eT5bLpSwWi5DDRaeLR+qb7KStGBZJcdvaQ3KUFkXq2mKy/JAtL+9x9X2IKFepaPoO8Oemi+85\nh4v8LYJMKPKfzWaFbo7sSj9K32Qn7Q0QC+Zw1wYdxW21WkcRYL3YisaOvLhNSwqWVuTvj5smErYP\nsLTb7TYQlqub8EOFwBO3X+WAVMp9k520FSPmAuv8q7Xg9qEroqVo4hytzteyWALHlEirPzu+f7/f\nB/eYLawuR9T9ktfrtSyXy+BxPELfZCftDWBZWN0byWoHw6Qdj8ehqRoWK5p0DhcEtVYqe1oRuygC\n6Z/9fi+9Xi+4uZzDRaCNXWIQtqy8MCWyAk7aCqEDUHofq5VOWkqI4BMsLLog/vz5U379+iWDwaCg\nYNKqJkvNlNqeVkSCRdQu7OFwCB0hWdaJHHW32z0SXGBiwSP1TXbSVgzLwurJdExYLUPUlvbnz5/y\nzz//yD///BNIG1M0xVRM3ymQuBb8mmORXlhYEBZtV7vdbsHCzmYzybIsqKjwnaTeN9lJWzFYHKFJ\nxcom66j3sKwdRtcJqySPLWvK0D8ssR8aDup1Oh3J81x6vZ7s93uZTCaFgJ0uM2Tr7YEoRyHYpKtx\ntLLJOg6Hw7B/ReBJd0W05IbPglN5XD1L1yotTJ2wIk7aSgHSsrKJF9IVlm6Y97NI8XAPYq5vvZeg\nvw4oy+PqrUOMsKnDSVshmLQsRdTtU2JqJ90dEaTV/Ye/uzi9TijL4+quG7HPicmbIpGdtBVCW1ru\n2cTaYIuc+BtL5cT51lglzjPgVB7XsrQ6R50iSTWctBWC97Q8ke7U/Bws7obIiwMqz0ZUjbI8ro6k\n6z5bmrCpEthJWyG0ewxZIibSldW6jkajUKQe6wEF68J7ND4+C2J5XP7crKAdI1XCijhpK4V2j9nS\ncjqHW8FwWoeL1C0Z5LOSlFGWx30Gwoo4aY9g7RdjQgV9Hzr6awtaZmXZPf7qpPVnwak8bqwbBy+R\n4njMFLcaTlopkk8HMqzARmxlWRbtOcwCCaRxUqzCqTu0Go1FLhhBwp91ilHkpyatjipypwRrxivv\nM60GaVmWmZZU510hmGC9sJO2GpwibaPROJqLm9pn/7SktfaHur2J1gYj9WKRGbWesXSOLmLXltZx\nPU4RFmKM1EdlPiVpY9FXfNG6T5PuBqEbfnNqhlVOWvnErU/dPb4dysgrcjwqM7UWq09JWpH4qEjo\nhiGMYLWSJh3LE63bfD8PSeaVWr1r3XHK2oKwHIzC//M9bU2h97E46j0trCu3JY31aGJLrAvTdd1s\nrAbWUQ3OIS1aq6YqBX060orYxera0vIQKJ7vanVFxOLuEVoYUdYbyt3janDOnna/34cjz/lJCU9J\nWhE7H6vdY7a0SNvE2pmCtGX1rrG0UYoXTl2h2/rwd4meyPisefZPSp//05FWq42YNGxlLTUTpIZl\nASqLnLGqk1ST+3WFlpGyp7RYLERECv2QRaRgcVPB05FWxO7fBNJyWR1PVAdpYwEopIH0Y1pW1S3s\nbcDdLNhTGo/HYQIBR+sPh4PsdjvZbrdJfQ9PRVrLHebFrjGX1YG0w+HQjABzJNiy4pqkVnG243po\nSwtZ6Xq9DnN+rJ7IqUXvn4q0IrZ7DPdVW1odiBqNRlG1FFtZ/cNQpmdO6WKpO0BaDiSu12vZbreF\n6QJ6gJeTNgFYlpZ7O3FrGO0e6yjwqW6I2qJatx3VwLK0GAeCHKwmbFlP5Lri6UgbIyxHHNnSch/i\n0WhUWjBgubynjvrc8XVwXCLLsmBhue42z3PZbrdhRKaTtuawxBSXWlr+/3xufenn3ueoBtrSMmEh\nrOCZtrPZLJA2JTwcaS3XE+fWWA7el0L5FJtUh+729wJ32o8d9bm+zzri3Pohslx6QP8Alf3NOT9W\n1/6gWd9zt9uV3W4n+/2+kJ7TI1NS+jF9KNJa1pNv687+epjV6+ur/Pe//5Vfv34V+g4jyFQH7Pf7\nIMXTRyxcpBAO8H2azHwfLnhdRxzrt1y2rICcyPkNyR1xPBRpRf6W11lSQk7nWGs8Hst//vMf+fnz\nZyAtT2K7N0AuTUQc8zwPa7vdHp3j72Kr0WiYUXGtk9ZprFhOmpdIsVG4ddtxHh6KtJYLzJ3+df9h\nfc7zc8bjcWhpWjdLC4LudrvCETNtYguBGbbCbI2bzabphUBAwm1c2frqc71E4nt/J+7leDjS6nwr\nLyhksLgHMfo0ceM1do/rcFGxpYWShy0pgiyxxdZXEz7Pc2m1WoX9u65kgrY61o5HezdwvUWk4CJr\n1OGzTQkPSVpOsPNFx2V2VpM17jDBzcPr1F2CLS2ICgu7Xq9lsVjIYrGQ5XJ5dL5erwsk18dWq1WY\nfKDPe72eSU69DcFr7HQ6ImIHBevyeaaIhyNtLN/a7/eDBdVT1nHEhamL2utsabfbbZDpYR7rfD6X\n2Wx2dL5arQLJsfh2u90OHsdoNJLlcinD4VA2m43keR68DiaonpXb6XQKEWu91+X3AtThs00JD0da\n7R7D0nK+VU9Yxz623+8fXZS6Ufi9YVlauMWwqNPp9GhNJhNZLpele952ux36MC+Xy0DyPM9lv9/L\ndrs9Spfxwt4YYMKihhWfo+6C6MQ9Hw9HWi2S4H0sWxFuGA5L2+/3o0GW74CVZ+VzWNXlchlIxefz\n+Vwmk0mBqDHSwjrz6nQ6pdFl/I2OLvMoz16vJ9vtVrIsC3tafC8gKD7PZrPphP0Ckiat/rK5NIut\nLJRNCDxxR8RzZr9+V/I9lm/F4j2rtdgdhkuM42KxMN1jTgc1Go0jTS5e1263C7Wo2iXGue6rxUJ9\nEBVdEPk75Dyx4zSSJG1MccOusba0TFoEmLhvEyKgeg/23RcSu74c2c3zPFhTJiMfY2TmQFQsCJXn\nuYhIsMKasNvtthCI0i10uO/zcDg8IizSZmhfyt+fk/UyJElakXg3RZ6lY1labnHKHSe0cKBMU3wr\ngCAgkw4ULZfLo73qbDYL5yCole5ZLpdmygfnIBjqS/n14HXgc4qtfr8ffhiwtwWhQXj+/vBZp9IF\nsS5IjrTayjKxtKXVpOVBzbphuC6tuwdx4R6DtOv1urAWi4VMJhP5/PyUyWQSFm4vFovCfpX3rUj3\naFEFLwjq4bIyYZfLZaFm2Nr3wyUGYfF9YJ+LFBB/d2i05jgfyZFWJC5oxy8+B0W0e4z9lrWntdQ7\n321psZcF0WAlV6uVzGYz+fz8lM/PT/n4+JCPj49w/vn5KYvForBP1ftWRIFjiwNfFmH1vl+fv7y8\nFAjLvZo2m410u93Cj6LWQjvOQ5KkBSyReszSgrjclM1yj/G41vHW0O4xIsVwexEF/vj4kPf396O1\nXC5LrakmSew2CBtr9Rpbw+FQRIqE7ff7BTUWf1cISjlhL0OSpLUiu2V7Wra0uvs/Rz9jlSjfBcs9\nhqXF3pVJ++fPH3l7ewvH5XJpEpLP8Tz8nDg2Go1CL2DL47CCgDhfLBZHhB0Oh4V8r87bsoV3nIfk\nSMsE1W4a8rMscmdry8EnHTmuQlqny95iK/a3m80mGhmez+fBNeb9LAeklstleGx+TZe8/msA4nJ6\nCWTl9JW7xNchKdJqhY0Wrp8zZ0eTVbeJuQYsMWRBvnZRY8dTedjpdCofHx8ynU5lPp8HsQT2qngN\njsdGUqQVKQ5+1vnCcwlr5WarAEgb0/fG6lyx2B1mxROO8/k8WFbWEud5blouJ/BjIinSaldYD7Ni\nslpWFzWhTPYqRRTcOAwERLqG3cXY0ikeXkxcrOVyGVI5elyjE/ZxkRRpRYoiCl3sXkZWK71T9fAr\nuLro9seRX7aeIDOLH0BOLrXjonY8prbEukWoiBP20ZEUaa18LEsWz1mWOOAW7jH2p7pMjomsj7Ca\nMfeZy/BYPAFL62R9DiRFWhE5anl6CWm73a4pVazSPdaWdjabFaK87N5qQq/X62hTNg5uWcsJ+zxI\nirRaRKFJy03aYsTF41jHa6H3tOiti6gvk9da6/X6pPBBd1/kc8dzIDnSMmG16kn3KNYpnnb7urd7\nypqxDJEbrfH+FtYVQgnOt4K0/FwxIcS5r+kesH4E7yFYedRoenKkteplUQyAPk9cesfiie/CuSKL\nsqJ3PsbuqxtiJZP3gP7srM9Z/10qSJK0bGV5HCVqObl7ICpTvvviuZS4p2SG+ty6fS/EpI1l990C\n1udRRta6fH6XIgnScumdtrQ8rf3eltaS510qaeT/px+77Pa9UGZd721lyySjKaP2pNUXBRcEcAUP\nej6h/I4LAr6jx5NFqrKLpozEZY8bu+8eOIewVmHHd6Bs25E6ak1a/QWze2xNtoOl1YOWbm1pY1bw\nFHH1/dZjlT3PPXEJYcv+7y0RiwmkbnVrS1qLsNrS1sE9LiMsjl+1tHW9oC4l7Hfuc2Mez6XeTJ1R\nW9IyTu1pORDFnSluHYiqmrCpXTxfJeytEftMY0Gp1FA70pZ1RrDqZPV8HhZW6IZtVcFyZw+HQ0Gx\npKt8WHaoG6yldAFZIyz5++FKKt343WpXU/U+N5Yn56Z33JUSn39K30GtSMvFAFoj3Gq1zBkz3F3R\nElRU3Ww81hECzby5RYwWUnAdLOuGUyEufz/W4C29RbG+E90p5BYFGzwiBZ87On7E6pFT+PyB2pLW\nGu7EA7KYrExa/NrrX/eqtMUgqLW4csciLvcm5lrYlEjLhRp6ygDy5Lq/NHcL4e/lFk0IeHogSAuy\nogHeZDJx0lYJbs6mlyatZWn572/lGlvT1q0Oipq00+nUHD2ZSoWOFrfoGbZIubGltfpL36JriEiR\ntNxX6+PjQ/78+ROsrG4ikMqPJlAr0lrN2fiiKCMsljWGseryO2uws+48EXOPY4OeU7ho8P3oxnnQ\nerMqTevAQXK95anaPcb3wO7xx8eHvL29yefnZ6iowh5Xt+tJAbUjLe+TdPUOB5ws8mZZZg47rpKw\nbGl5sDN+4U9ZWqtONpULxpKRcrEGW1rdFJ6FLlbf5CpLI9k91pZWT15w97gCxEZVIkocs7IgdqyR\ndtXEtSLFFmE1aWO1silcNDHtt55GqOMO7B5bUedbuceWpWUPB+epfP5ArUgbc4+ZsKeIG5PPVQEd\niNKpBd3PietpJ5OJzGazUpVU3WG5x1bBhnaPORDFj8XHKgD3uIy0ujtmSoFAoHak1WkF3QdK5wJ1\nXvDWKCMr94LS/aCQ4qk7yvLkPMqStyogrLWf1em3a3CKWOwB8Swk/IAuFoto9N9J+6Dgi4KL27m1\nTOp5WI7u6jxsv98PhRk8lNsirB5uVrUyylI66e4eKUlDL4GT9gLggmALy7/iMfFEKr/kjUaj0MJH\nezKDwaBgVfXiCivWflddZRVTpOkJBvpvHwVO2gsQs7R6+jqsL0vlUgC2JTpqzzpv3rfi3NrLsqW9\nRSCQzy/VcqdOYiftBcCvOad3NGm5HWrKlpb3r3ofC4IivaNVUN9RGmkRVO9RUwv0nQsn7QWIWVq4\nxuwepyxTtNI5TExefB9c41taWhE5Iuqp89hjpAon7QVgS8vzYy33OFVLa5H2x48fRxJF69jv9wuV\nWLfY08Zc45iV5f/zKHDSXgiekK4HY2n3mLXFKcByjxElfn19LdQrW0tX89wqehwjrBWMejTCijhp\nv4xLgiF1unC0CknnYVmayHlYpHZ41q8+1wPO2DU+h7TW58T36QINvXhromtmLXe5Tt/LJXDSXgge\nXs1qLcjhtPtchaigKlg1yixk4ZSODi5Zdctl40OvmZUUS+noMaJ6WNnn56e8vb0Vama5kuecPG4K\ncNJegJhgnqt1OLJ8z77L+nXz6+dCdD5/eXkJFlUHnbiNj57goFVPmrCXvPcyb4W3JXqt1+ujEjxY\nXq6keoQ8rpP2AmjtLSwtT3znVNAtFUGXvGY+Z/GEXkxanXvlVA5bWT12xZpI+FXSasuIsjuIWbBQ\naseWFoXulqVN2cqKOGkvgmVpec/EtZxaEXQP0mrC6mIMtpoofbTcYyv/GnOPQVZeXyGujgZDiYbP\nF9F6pNpms5l8fn4WWsqgQwgsLUfxUyauk/YCaNL2er1wMTQaDdntdqGOU4sL7vFa9Tm/fngJHEzi\nVrSWYAIpHWvpTiFfKYksS+PECtw/Pz8DWfVAM6s7heUWp0ZcJ+0FwIXIaRF84c1mU/I8l/l8Xqhw\nuYeljRG2bKyKrtZBXaze0+rqKr2f1ZVB+vWcguUa44dRjxD9/PyU9/d3eX9/lz9//gSXmOf+akuL\n5+DnSw1O2gugLS0Ttt1uS57nMpvNbq4IuuT16qN2j6FqssT/VvTYauiG29b7/Mr7tiwtgnzsHk8m\nE3l/f5e3tzf5/fu3TCaTQtUVK9OYtKnDSXshmLggLazYer0+6qTBkdbtdmumMvh47WuLdYZoNpuF\nlj1MVC6x05JFXcyu87A6xXMtmKzcHcRqnAeLCjd5MpkU6puhSktFRnounLQXQBfp73a7QOD9fn/U\nIocJMh6PpdlsRtU7VUgdrZ7EvLIsKxAU5zrNwyonHmRWVR42Bv5MuJWPLmrXzQWYoNwMHtY1FRnp\nuXDSXgjuP9VqtQppBARltKpoNBrJcrmURqNhqnp433YNOAdr9Sa2itj5CP2wXlXmYU+B97Hcf4vz\ns5qsvPD3Wgn1SHDSXghtbfmC4DpUWKrhcBj2V41Go9CFkV2/KjTKXAvLR5zrInbOyY7H46Af5tEr\n3EmxijxsGfDjx/2bdEsfazGZ9ed6qtonRThpLwAT1iKt5R7zDJlms1mY8cNWBFb4GrTb7aO9tNYT\nW4TFYqvKqRy2tFXkYcvA+1lrFlKMrFiWJvlRAlCAk/ZC4ALl3CvuY0sLkgyHw7DfajabhUFcm82m\nkCa5tsOFLl7nnsTIw+oIMd/Osuwo0MRyR03SW9XJljXPO7Wn5X2su8cOEfm7p7Xu05Z2MBiElMN2\nu5Vmsymr1Uo6nY6sVqujiGue51e9tlgQjKt1mKj6HH2jY0UFVodGvP+qwI0GNGnP2dPGCuEfibhO\n2gugUyi4DxeHDkSh+geBkUajcTTNr9FohIvq2pRJrBexlYO1jt1uN1q2FyPoLYJQpyxtmZuMx7GO\nj4JakfZUYbOOvFrd+i01UOz2VwDC7vd7aTabcjgcAvF0yZ6e1dNqtWS5XEqv1ysc8X+qsLSxQJOV\ng9W5WG4mfg/gu+YUD2bL6hk8sbrZZ0CtSCtyvK/RwQiup9SLx05Y1qJKWI/HtbZZlhUIiz0vu3c4\n58jnNeh2u0eEjPVxunVf4q8ArvF2uy30koaWGPri2WxW6A7yaOKJU6gladmC6lyddeQkvA6SsBtb\nNfRjsq6XG7ohcJVlWSEIxecIolwD5GKtQBQrm7hap+qh29cA5XdaYwzFE5NWV/A4ae8E7RrrYIQe\nnmRZWwRSeI9Y9UUJsoKQAFtaXEh8H0gaS/vcMuXDOVcr91onSwshBZP2/f09SBXZ0j6iTPEUakVa\nkbilbbVaJ13kPM9DpBOIEeyrwP7VelwmKFtYWF90Z4ytKkirS+ZYYKHzrveu99Vg99iaMfv5+Rmq\ndxaLRWE/66S9E/R+1honGVtMboaVoqkKOugFgurbKBZglY4lAqhCe2y1kuH7rFWnPS33lGZLC9Lq\nQJS7xzVALOzfaDSOpqhri6sDOSBs1V/oKfcYhIWbrCPesc4MVZCWiwascy1DrNueli2tRVoO3Ll7\nXBNoRQyrcCyi6tsA0jK3TK5r64TXCXljbOF94lhVPtESP5TlXfX99wa+9zJLa33vTto7wupYgIiw\niBzVU6Kx12w2kyzLRETC3m232wWxQFWW5NSFXYcL/54oqxXm89isHfR84mkNvBaLxVFBwKNKFctQ\nK9KKFC2tFqNzcIKjnwgOrVaro7Iykb9dCOvSf/iRERPF6Ood63wymcj//vc/+f37t7y/vxc6KsKi\nxvoXPxNqS1qtcDocDsFtms/noZEYCIumalD8IKKIfeazfbH3gA4i6oAbgkyctuPz6XQq//777xFp\nterpUUvuzkWtSKvztBAb4H7uJ6wJy2IFJiyit8/45X439LZGr1ijcazZbCZ//vwJzdosS6s7ND4j\ncWtFWpFiITRu4wtqNpuyXC6PCItoo0VY9CZ+tJrKuoKrdHRaDl6S1WicOyxyK9Qy0lYZxEsJtSIt\nW1rcRvQYvYXRAcJKxG82myPCDgaDp0u+3wva0nKFzmazCcREsIkbjaNPsRWAWq1WZs+nR+z/dA5q\nRVqRYnQRZW8IRnFFjc7nZVkWCs3hEg8GgyDEf7Yv9l5gCapuxobh22gwjgVt8Ww2i7rOrHyKpdGe\nBbUkLYsWOI3CUUi2sJDkodIHhB2Px4G07h7fHtrSItCk5/hyk3E0Gn9/f5f5fF6qeuN87LMSVqSG\npAViexX0DubSvU6nI+v1WlqtVkGbClE5d+kTkaiw4NnzrNcC3wtPDuR8OtxhPc4Dx8ViUZoS0hMC\nnnE/K1Jj0sagI8wcZebgB4swcOGwfM9ajuugI/lsXaFs4tI6lNfFUjqWou1ZicpIirRaVaNzubqM\nj5VT8/m80Gibe/hCoOGW9jpYWxcEmXgKAM+O5XJFbGO4KVsZcfX5syAp0ooct6ThLoZWTyEmLle/\n7Pd7abeKS7yYAAAHG0lEQVT//9t3S1sNUKXDAUJYWhSww+qCtGh8p6PDsTzssxNWJGHSgrB8v9W9\nj/dUqDflNIGuv3V8HZal1SMpOa1jucdaohgr+nhWwookSFqR43wu7uMKEG1p5/O5tNvtQkADxQQu\nc6wGTFru8cRpHhZWsKUtS+noXOyzf1dJkTZGVo5aWpYW1hZuschflxjDs579QqgCMUsL9/jz8zN8\nJ5yD5Y4eeByd2sG5IzHSivz94thaIq9rBaKQ1AdpRf66xNryOq4D18PynpbdY6tvsW41y3CiHiM5\n0gK8r+WxGiAtu8XokyQihS4Xui6TByNzR8dYV8dbNu6+B2JpFR21j7mviBKzPJHXfD43K3w4COU4\njSRJy4op/iXe7/eFyCWGIINwm80mtBJFe1E+WkOT9Syb7+qrfC/EtL1W3y79wzedTuX//u//5N9/\n/5W3tzf5+PiQ6XR61ITtkSfafQeSI60mLNxjEQn7WpY3grD7/V7W63Xo+6uPIG1sHQ6HQj73O/oq\nfzd0/lvXxeoCAN3DeTqdyp8/f8JCmgek5b2r18V+HcmRFmDC4siWFiV8In/JjMIC9ADmI5NXN/u2\nqo5wLlJ9X+V7AeSxrOlutwufa2xZxQDa0vJjP2uVzrVIkrQWYUWKlhZE4mhmlmWhFzAvHrqsh1Jh\n36w7GAJV91W+N3Rpna6H5f2pda6XRVrdNtZJexmSJK3IsWvMgSh2idny6kbeuon3y8uLvL6+hgsM\nhIUoA7JHxi37Kn83dJWOnuyA9A00xFywDuEEW17uUcyktVRPjvORLGlFpEBYztWKFAmLQNKphQHQ\nFmH7/X5IGQG36qt8T7B7rHPeqNT5+PiQ9/f3cMRCdNiaU8RdRXgf63vay5E0aQF2j3nfpKt4kJuN\ndd0fjUZRwm42m8IoyO/oq/zd4EAUk5bz3di3/vnzR97e3uT379/haJXWWY3a8VyP8rl9Nx6CtAD/\ncqP6hxcT16r0Qa5Wz8JBoEpEglW2rLdu+WrV6sbu4+M1799SE52qkMFxu90GtZIVaGJXWNfCoh7W\nsqJl/ZyctJfjoUgLxPK4seogQHe253GQjUZDVqtVKWl5BEfZwg+IPq/ifVstTPkYE0ZAu80k1eSd\nTqehSyIm1+mma6fcXifp9Xg40pblcZmw/Pc6XQRhhm7VCimkXghkaQtuWXS9OM9bhaXl/SgPL4uV\nvvFiJZkm7Wq1Cq4xF7LzPB3rR8Gy9vx6HZfj4UgLWGkhJij/HXKubGk1YfM8l8FgYJI1tixrzHto\nCDaqij6DtDrqy5JBLZxgS7zZbEyy4sjR49hg53PIitfq+BoekrRleVz+GxAY7ilIazVD3263IYIc\nI601D9aaD8vVRlVGoDmCrhuDsyKJBQ4cOOLKKOuIAgAsK5Vj7alxzq/T8XU8JGlF7DwuW1kmNVzm\nZrMpq9Uq2gwdBNSk5aOevI5zrbcVEVOsce17toomUOXEAn3ryJa2jLz6yKTF6yg7Oq7Dw5JW5DiP\nCzBxdVTXIiysDJPWIi9LIbF4P8mWB8/FYzGreL/cvpStI3oKx0aEcj5W17zinEvpOBerScufvT53\nXI+HJi2gL6BYSR1bZ7aw7OZySsiysoPBQF5eXswWKvx8KMCvugifXzdHwqfTaZjCEBNAaPdYk5YF\n/7ws0jpuh6cgrUZZFBNT5/X+EOS19rNsacuabYNMEGvw+WazCY3mvor1el2oZ+U1mUyOSGtV6sCi\n6v2wNW5Sa4gd34OnJG0ZONepxRIiUkiRsKXpdDrBDda9l+GiovwvVmV07d4W+mAW8fOR+zFZbjL3\n1mLXN5YyckXTfeCkVWApn56Pq/+t3W7LdrsNbq4V1NGBKR1V5vuuJS3EEdw4jc/ZZbeOTGKtGdZN\nxGM9iR23h5OWYBFT349AjyWYwAWPqiKLmGUpo2tJm+d56fxXiCB0uoc9BCYwn7uVrQ+ctAq4GK35\nuLvdLqiYUJCA2yDuer0+KbiIrWtFFhBIxKpsyvakenaORexYL2In7/fCSUvQiikopXg+bkxDzMQ9\nR87If68ljV+FljBqKeMpGaMmc5l2mT8vx/fCSavAlkTPx9UdGq0qImtxEYH1d/y41752S56o5+Jo\nSxmrzIn9PZ7LRRP3gZNWARejVS536rysS+M5JXtVvv6Y/rdMpVRGSBf71wdO2gjcijjqisdobuRw\nPBGctA5HYnDSOhyJwUnrcCQGJ63DkRictA5HYnDSOhyJwUnrcCQGJ63DkRictA5HYnDSOhyJwUnr\ncCQGJ63DkRictA5HYnDSOhyJwUnrcCQGJ63DkRictA5HYnDSOhyJwUnrcCQGJ63DkRictA5HYnDS\nOhyJwUnrcCQGJ63DkRictA5HYnDSOhyJwUnrcCSGhg+YcjjSgltahyMxOGkdjsTgpHU4EoOT1uFI\nDE5ahyMx/D+MzoMhc4sPbQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f546e014898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label: \n"
     ]
    }
   ],
   "source": [
    "similarities = similarity(X_test[200], X_train[small_train])\n",
    "view_image(X_train[small_train[np.argmax(similarities)]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "\n",
    "Logistic regression is a probabilistic, linear classifier. It is parametrized\n",
    "by a weight matrix $W$ and a bias vector $b$ Classification is\n",
    "done by projecting data points onto a set of hyperplanes, the distance to\n",
    "which is used to determine a class membership probability.\n",
    "\n",
    "Mathematically, this can be written as:\n",
    "\n",
    "$$\n",
    "  P(Y=i\\vert x, W,b) = softmax_i(W x + b) \n",
    "$$\n",
    "$$\n",
    "   P(Y=i|x, W,b) =  \\frac {e^{W_i x + b_i}} {\\sum_j e^{W_j x + b_j}}\n",
    "$$\n",
    "\n",
    "The output of the model or prediction is then done by taking the argmax of\n",
    "the vector whose i'th element is $P(Y=i|x)$.\n",
    "\n",
    "$$\n",
    "  y_{pred} = argmax_i P(Y=i|x,W,b)\n",
    "$$\n",
    "\n",
    "![a](d2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import theano\n",
    "from theano import tensor as T\n",
    "import numpy as np\n",
    "import datetime as dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "theano.config.floatX = 'float32'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Theano is a Python library that lets you to define, optimize, and evaluate mathematical expressions, especially ones with multi-dimensional arrays (numpy.ndarray). Using Theano it is possible to attain speeds rivaling hand-crafted C implementations for problems involving large amounts of data. It can also surpass C on a CPU by many orders of magnitude by taking advantage of recent GPUs.\n",
    "\n",
    "Theano combines aspects of a computer algebra system (CAS) with aspects of an optimizing compiler. It can also generate customized C code for many mathematical operations. This combination of CAS with optimizing compilation is particularly useful for tasks in which complicated mathematical expressions are evaluated repeatedly and evaluation speed is critical. For situations where many different expressions are each evaluated once Theano can minimize the amount of compilation/analysis overhead, but still provide symbolic features such as automatic differentiation.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def floatX(X):\n",
    "#     return np.asarray(X, dtype='float32')\n",
    "    return np.asarray(X, dtype=theano.config.floatX)\n",
    "\n",
    "def init_weights(shape):\n",
    "    return theano.shared(floatX(np.random.randn(*shape) * 0.01))\n",
    "\n",
    "def model(X, w):\n",
    "    return T.nnet.softmax(T.dot(X, w))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = T.fmatrix()\n",
    "Y = T.fmatrix()\n",
    "\n",
    "w = init_weights((784, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.00503957, -0.0227594 , -0.00668998, ..., -0.00092346,\n",
       "         0.00191202,  0.00108656],\n",
       "       [-0.00323021,  0.00703294, -0.00368078, ...,  0.0102045 ,\n",
       "        -0.01121792, -0.00557184],\n",
       "       [ 0.00743922, -0.00108072, -0.00547558, ...,  0.00349877,\n",
       "         0.0148891 ,  0.00193652],\n",
       "       ..., \n",
       "       [ 0.01167904,  0.00657063, -0.00629851, ..., -0.00184553,\n",
       "        -0.020172  ,  0.00010953],\n",
       "       [ 0.00517218, -0.00536702,  0.00352126, ...,  0.00743843,\n",
       "        -0.01571855, -0.00655702],\n",
       "       [-0.00206186, -0.00010561, -0.01261193, ...,  0.00719551,\n",
       "        -0.0019429 ,  0.0134165 ]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w.get_value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initialize model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "py_x = model(X, w)\n",
    "y_pred = T.argmax(py_x, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))\n",
    "gradient = T.grad(cost=cost, wrt=w)\n",
    "update = [[w, w - gradient * 0.05]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train = theano.function(inputs=[X, Y], outputs=cost, updates=update, allow_input_downcast=True)\n",
    "predict = theano.function(inputs=[X], outputs=y_pred, allow_input_downcast=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for start, end in zip(range(0, X_train.shape[0], 128), range(128, X_train.shape[0], 128)):\n",
    "    cost = train(X_train[start:end], y_train2[start:end]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0.12248333333333333, 0.1145)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "errors = [(np.mean(y_train != predict(X_train)), \n",
    "           np.mean(y_test != predict(X_test)))]\n",
    "errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for 100 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 (0.10956666666666667, 0.10150000000000001)\n",
      "1 (0.10246666666666666, 0.095399999999999999)\n",
      "2 (0.098183333333333331, 0.091899999999999996)\n",
      "3 (0.095533333333333331, 0.089999999999999997)\n",
      "4 (0.093450000000000005, 0.088900000000000007)\n",
      "5 (0.091366666666666665, 0.087900000000000006)\n",
      "6 (0.089766666666666661, 0.086599999999999996)\n",
      "7 (0.088883333333333328, 0.084900000000000003)\n",
      "8 (0.087650000000000006, 0.0843)\n",
      "9 (0.086683333333333334, 0.084400000000000003)\n",
      "10 (0.085800000000000001, 0.083500000000000005)\n",
      "11 (0.085166666666666668, 0.083299999999999999)\n",
      "12 (0.084449999999999997, 0.083099999999999993)\n",
      "13 (0.083883333333333338, 0.082199999999999995)\n",
      "14 (0.083366666666666672, 0.081600000000000006)\n",
      "15 (0.083016666666666669, 0.081600000000000006)\n",
      "16 (0.082483333333333339, 0.081100000000000005)\n",
      "17 (0.082116666666666671, 0.080799999999999997)\n",
      "18 (0.081833333333333327, 0.080500000000000002)\n",
      "19 (0.081549999999999997, 0.080699999999999994)\n",
      "20 (0.08118333333333333, 0.080399999999999999)\n",
      "21 (0.080816666666666662, 0.080100000000000005)\n",
      "22 (0.080483333333333337, 0.079799999999999996)\n",
      "23 (0.080183333333333329, 0.079699999999999993)\n",
      "24 (0.079750000000000001, 0.079699999999999993)\n",
      "25 (0.079483333333333336, 0.079100000000000004)\n",
      "26 (0.079216666666666671, 0.078899999999999998)\n",
      "27 (0.07906666666666666, 0.079000000000000001)\n",
      "28 (0.078799999999999995, 0.079000000000000001)\n",
      "29 (0.078483333333333336, 0.079000000000000001)\n",
      "30 (0.078100000000000003, 0.078899999999999998)\n",
      "31 (0.077833333333333338, 0.079000000000000001)\n",
      "32 (0.077666666666666662, 0.078600000000000003)\n",
      "33 (0.077566666666666673, 0.078100000000000003)\n",
      "34 (0.077366666666666667, 0.078)\n",
      "35 (0.077166666666666661, 0.078200000000000006)\n",
      "36 (0.077066666666666672, 0.078200000000000006)\n",
      "37 (0.076916666666666661, 0.078200000000000006)\n",
      "38 (0.076700000000000004, 0.078)\n",
      "39 (0.076550000000000007, 0.077700000000000005)\n",
      "40 (0.076466666666666669, 0.077700000000000005)\n",
      "41 (0.076316666666666672, 0.077499999999999999)\n",
      "42 (0.076200000000000004, 0.077399999999999997)\n",
      "43 (0.076050000000000006, 0.077499999999999999)\n",
      "44 (0.075933333333333339, 0.077700000000000005)\n",
      "45 (0.075783333333333328, 0.077700000000000005)\n",
      "46 (0.075566666666666671, 0.077600000000000002)\n",
      "47 (0.07538333333333333, 0.077299999999999994)\n",
      "48 (0.075283333333333327, 0.077100000000000002)\n",
      "49 (0.075083333333333335, 0.076999999999999999)\n",
      "50 (0.075033333333333327, 0.077100000000000002)\n",
      "51 (0.074800000000000005, 0.076799999999999993)\n",
      "52 (0.074766666666666662, 0.076799999999999993)\n",
      "53 (0.074483333333333332, 0.076899999999999996)\n",
      "54 (0.074416666666666673, 0.076499999999999999)\n",
      "55 (0.074300000000000005, 0.076499999999999999)\n",
      "56 (0.074083333333333334, 0.076200000000000004)\n",
      "57 (0.073950000000000002, 0.075899999999999995)\n",
      "58 (0.07378333333333334, 0.075800000000000006)\n",
      "59 (0.073649999999999993, 0.075800000000000006)\n",
      "60 (0.073566666666666669, 0.075700000000000003)\n",
      "61 (0.073516666666666661, 0.075700000000000003)\n",
      "62 (0.073450000000000001, 0.075700000000000003)\n",
      "63 (0.073266666666666661, 0.075700000000000003)\n",
      "64 (0.073266666666666661, 0.075700000000000003)\n",
      "65 (0.073133333333333328, 0.075700000000000003)\n",
      "66 (0.073133333333333328, 0.075800000000000006)\n",
      "67 (0.073083333333333333, 0.075899999999999995)\n",
      "68 (0.072916666666666671, 0.075899999999999995)\n",
      "69 (0.072833333333333333, 0.075800000000000006)\n",
      "70 (0.072849999999999998, 0.075800000000000006)\n",
      "71 (0.07276666666666666, 0.075800000000000006)\n",
      "72 (0.07273333333333333, 0.076100000000000001)\n",
      "73 (0.072583333333333333, 0.075899999999999995)\n",
      "74 (0.072433333333333336, 0.075800000000000006)\n",
      "75 (0.072366666666666662, 0.075800000000000006)\n",
      "76 (0.072366666666666662, 0.075800000000000006)\n",
      "77 (0.072249999999999995, 0.075899999999999995)\n",
      "78 (0.072266666666666673, 0.075899999999999995)\n",
      "79 (0.072249999999999995, 0.075899999999999995)\n",
      "80 (0.07223333333333333, 0.075700000000000003)\n",
      "81 (0.072133333333333327, 0.075800000000000006)\n",
      "82 (0.072016666666666673, 0.075700000000000003)\n",
      "83 (0.071966666666666665, 0.075700000000000003)\n",
      "84 (0.071900000000000006, 0.075600000000000001)\n",
      "85 (0.071883333333333327, 0.075499999999999998)\n",
      "86 (0.071866666666666662, 0.075499999999999998)\n",
      "87 (0.071766666666666673, 0.075499999999999998)\n",
      "88 (0.071766666666666673, 0.075499999999999998)\n",
      "89 (0.071800000000000003, 0.075399999999999995)\n",
      "90 (0.071749999999999994, 0.075300000000000006)\n",
      "91 (0.071633333333333327, 0.075200000000000003)\n",
      "92 (0.071583333333333332, 0.074999999999999997)\n",
      "93 (0.071566666666666667, 0.075200000000000003)\n",
      "94 (0.071516666666666673, 0.075300000000000006)\n",
      "95 (0.071483333333333329, 0.075300000000000006)\n",
      "96 (0.07145, 0.075300000000000006)\n",
      "97 (0.071433333333333335, 0.075200000000000003)\n",
      "98 (0.071400000000000005, 0.075300000000000006)\n",
      "99 (0.071300000000000002, 0.075300000000000006)\n",
      "Total time:  0.4\n"
     ]
    }
   ],
   "source": [
    "t0 = dt.datetime.now()\n",
    "\n",
    "for i in range(100):\n",
    "    \n",
    "    for start, end in zip(range(0, X_train.shape[0], 128), \n",
    "                          range(128, X_train.shape[0], 128)):\n",
    "        cost = train(X_train[start:end], y_train2[start:end])\n",
    "        \n",
    "    errors.append((np.mean(y_train != predict(X_train)), \n",
    "                   np.mean(y_test != predict(X_test))))\n",
    "    print(i, errors[-1])\n",
    "\n",
    "print('Total time: ', (dt.datetime.now()-t0).seconds / 60.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f54617b3080>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXJyEQkAQSlgDZ2NeKiEgNVowgSluXtorF\nFtzqUivqdfm5VK3g0mvV2mq1WuterxvaKtheRcXAVQFRoUUggEBCQggYAlkge76/P042QkIGkswM\nmffz8ZgHOWfOmfnOMc473/WYcw4REQlNYYEugIiIBI5CQEQkhCkERERCmEJARCSEKQREREKYQkBE\nJIT5FAJmNt3M0s1so5nd2sTzI8zsMzMrNbMbG+zvYmYrzGyVma0xs7vbsvAiItI61tI8ATMLAzYC\nU4EcYCUw0zmX3uCY3kAy8CNgj3PukQbPdXPO7TezcOBT4Drn3Odt/klEROSw+VITmAhscs5lOucq\ngNeAcxse4JzLc859CVQ2Ptk5t7/mxy5AJ0Cz00REgoQvIRAPZDXYzq7Z5xMzCzOzVUAu8IFzbuXh\nFVFERNpLu3cMO+eqnXPHAwnAd81sdHu/p4iI+KaTD8dsB5IabCfU7DsszrlCM/sYmA6sa/y8mamZ\nSETkMDnnrDXn+1ITWAkMNbNkM+sMzAQWHOL4ugKZWW8z61Hzc1dgGpDe3InOOT2c4+677w54GYLh\noeuga6FrcehHW2ixJuCcqzKzOcAivNB41jm33syu8p52T5tZHPAFEAVUm9n1wGigP/BizQijMOB1\n59y/2qTkIiLSar40B+Gcew8Y0WjfXxr8vBNIbOLUNcD41hRQRETaj2YMB6HU1NRAFyEo6DrU07Wo\np2vRtlqcLOYvZuaCpSwiIkcDM8O1smPYp+YgEem4Bg4cSGZmZqCLIYeQnJxMRkZGu7y2agIiIa7m\nr8lAF0MOobn/Rm1RE1CfgIhICFMIiIiEMIWAiEgIUwiISId39dVXc//99we6GEFJHcMiIS7YO4YH\nDRrEs88+y5QpUwJdlIBRx7CISDOqqqoCXYQ6TZXlcMvn78+jEBCRoHXRRRexbds2zj77bKKjo3n4\n4YfJzMwkLCyM5557juTkZKZOnQrABRdcQP/+/YmJiSE1NZV16+oXK7700kv5zW9+A8CSJUtITEzk\nkUceIS4ujvj4eF544YVmy1BYWMjll1/OgAEDSExM5K677qr7q/zFF1/ke9/7HjfeeCO9e/dm3rx5\nTe5zznHfffcxcOBA+vXrxyWXXEJhYSFAs5/HXxQCIhK0XnrpJZKSknj33XcpLCzk5ptvrntu6dKl\npKen8/777wPwgx/8gM2bN7Nr1y7Gjx/Pz3/+82ZfNzc3l6KiInJycnjmmWe45pprKCgoaPLYiy++\nmM6dO7NlyxZWrVrFBx98wDPPPFP3/IoVKxg6dCi7du3ijjvuaHLf888/z0svvcSSJUvYsmULRUVF\nzJkz54D3afx5/CbQS6E2WBLViYj/+fL/HrTN40gMHDjQffTRR3XbGRkZLiwszGVkZDR7zp49e5yZ\nucLCQuecc5dccom76667nHPOpaWluW7durmqqqq64/v27etWrFhx0Ovs3LnTdenSxZWWltbte/XV\nV91pp53mnHPuhRdecMnJyQec09S+qVOnuieffLJue8OGDS4iIsJVVVX59Hma+29Us79V371aNkJE\nWhSM/cYJCQl1P1dXV/PrX/+aN998k7y8PMwMMyMvL4+oqKiDzu3VqxdhYfUNId26daO4uPig4zIz\nM6moqKB///5A/R/NSUn199lKTDx4AeXG+3JyckhOTq7bTk5OprKykp07dzb5efxJISAiQc2s6cEv\nDfe/8sorLFy4kMWLF5OUlERBQQExMTGtHvWUmJhIZGQku3fv9qkcze0bMGDAAeszZWZmEhERQVxc\nHFlZWc2+jj+oT0BEglq/fv3YsmXLAfsaf7kXFRXRpUsXYmJi2LdvH7fffnubfKn269ePM844gxtu\nuIGioiKcc2zZsoWlS5ce1utceOGF/OEPfyAjI4Pi4mLuuOMOZs6cWVcbaW1YtYZCQESC2m233ca9\n995LbGwsjzzyCHDwX80XXXQRSUlJxMfH853vfIdJkyYd1nscKjBeeuklysvLGT16NLGxscyYMYPc\n3NzDev3LLruM2bNnM3nyZIYMGUK3bt147LHHfHr/9qbJYiIhLtgni4kmi4mISDtRCIiIhDCFgIhI\nCFMIiIiEMIWAiEgIUwiIiIQwhYCISAhTCIiIhDCFgIhICFMIiEhQGzRoEIsXL27167z44ouccsop\nbVCijsWnEDCz6WaWbmYbzezWJp4fYWafmVmpmd3YYH+CmS02s7VmtsbMrmvLwouI+Mo5165r9ByN\nt5YEH0LAzMKAx4EzgTHAhWY2stFhu4FrgYca7a8EbnTOjQFSgGuaOFdEpElN3V4SYPny5Zx88snE\nxMRw/PHHs2TJkrpzXnjhBYYMGUJ0dDRDhgzh1VdfJT09nauvvpply5YRFRVFbGysT++/Y8cOzj//\nfPr27cuQIUP405/+VPfcvHnzmDFjBrNnz6Znz568+OKLTe4rLy/nv/7rv4iPjychIYEbbriBiooK\noP5Wlw8++CD9+/fnsssua8Or56OW7joDnAT8b4Pt24Bbmzn2brwv/eZe621gajPPNXtXHRFpP8H+\n/97AgQPd4sWL67a3b9/uevXq5d577z3nnHMffvih69Wrl8vLy3P79u1z0dHRbtOmTc4553Jzc926\ndeucc94dv0455RSf37e6utqdcMIJ7r777nOVlZVu69atbsiQIW7RokXOOefmzp3rOnfu7BYsWOCc\nc660tPSgfSUlJe6uu+5yKSkpLi8vz+Xl5blJkya53/zmN8457y5nnTp1crfffrsrLy8/4A5mDTX3\n3wg/3VksHshqsJ0NTDzcsDGzgcA4YMXhnisigWXz2qYZxd19ZKuVugYraL788sv88Ic/5MwzzwRg\n6tSpTJgwgX/961+cd955hIeHs2bNGhISEoiLiyMuLu6I3nPlypXk5eXV3Td44MCBXH755bz22mtM\nmzYNgJSUFM4++2wAunTpctC+yMhIXnnlFZ544gl69eoFwN13380vf/lL5s2bB0B4eDjz5s0jIiLi\niMrZWn65s5iZdQfeBK53zh18DzcRCWpH+uXdHjIzM3njjTdYuHAh4AVEZWUlU6ZMoVu3brz++us8\n9NBDXHbZZXzve9/j4YcfZsSIEUf0Ptu3b69rOnLOUV1dzeTJk+uO8fXWkg1vR5mcnExOTk7ddp8+\nfQIWAOBbCGwHkhpsJ9Ts84mZdcILgL8559451LFz586t+zk1NZXU1FRf30ZEOqjGnbmJiYlcdNFF\n/OUvf2ny+GnTpjFt2jTKysq44447uPLKK1myZMlhdwonJiYyePBgNmzY4HPZmtoXHx9PZmYmo0aN\nArxwGTBgwCFfozlpaWmkpaX5fLwvfAmBlcBQM0sGdgAzgQsPcXzjT/QcsM4592hLb9QwBEREoP72\nklOmTAFg1qxZTJw4kfPOO4/TTz+d8vJyVqxYwbBhw+jUqRPLly/n9NNPJzIyku7du9fdwjEuLo7s\n7GwqKip8+st74sSJREVF8eCDD3LdddcRERFBeno6JSUlTJgwwefyz5w5k/vuu6/unHvvvZfZs2cf\nwZU4+I/j2ial1mhxdJBzrgqYAywC1gKvOefWm9lVZnYlgJnFmVkWcANwh5ltM7PuZnYy8HNgipmt\nMrOvzGx6q0stIiGj8e0lExISeOedd/jtb39Lnz59SE5O5uGHH6a6uprq6moeeeQR4uPj6d27N0uX\nLuXJJ58EYMqUKYwZM4Z+/frRt2/fFt83LCyMd999l9WrVzNo0CD69u3LFVdcQWFh4WGV/84772TC\nhAmMHTuW4447jgkTJtT1MwSDoLq9ZHW1I4C32hQJSbq9ZPALmdtL7tsX6BKIiISWoAqB/PxAl0BE\nJLQEVQjs2RPoEoiIhJagCgHVBERE/EshICISwhQCIiIhzC/LRvhKISDif8nJye26xLK0XnJycru9\ntkJAJMRlZGQEuggSQGoOEhEJYUEVAhoiKiLiX0EVAqoJiIj4l0JARCSEKQREREKYQkBEJIQFVQiU\nlUF5eaBLISISOoIqBGJjNUJIRMSfgioEYmLUJCQi4k9BFQKxsQoBERF/UgiIiIQwhYCISAhTCIiI\nhDCFgIhICAuqEIiJ0RBRERF/CqoQUE1ARMS/FAIiIiFMISAiEsIUAiIiIUwhICISwsw5F+gyAGBm\nrrLS0aWLt5JoWFDFk4hI8DEznHPWmtfw6avWzKabWbqZbTSzW5t4foSZfWZmpWZ2Y6PnnjWznWb2\nnxbfJ6ya7t2hoMD3DyAiIkeuxRAwszDgceBMYAxwoZmNbHTYbuBa4KEmXuL5mnNbVFhWqCYhERE/\n8qUmMBHY5JzLdM5VAK8B5zY8wDmX55z7EqhsfLJz7hPApylge0r2KARERPzIlxCIB7IabGfX7Gtz\n+SX5CgERET/qFOgCNPTHB/5ITs4Qnn4aunRJJTU1NdBFEhEJGmlpaaSlpbXpa/oSAtuBpAbbCTX7\n2txZV55FVOVPGT0a9P0vInKg1NQD/zieN29eq1/Tl+aglcBQM0s2s87ATGDBIY5variSNbP/AHtK\n95CUBNu2+VAqERFptRZDwDlXBcwBFgFrgdecc+vN7CozuxLAzOLMLAu4AbjDzLaZWfea514BPgOG\n1+y/tLn3yi/JZ+BA2Lq11Z9LRER84FOfgHPuPWBEo31/afDzTiCxmXN/5mth8kvymTIQMjJ8PUNE\nRFojqObl7inZw6BBCgEREX8JqhDIL82nb1/Ytw+KiwNdGhGRji+4QqAkHzMYOFC1ARERfwi6EADU\nOSwi4idBFQJ7SrzVJdQvICLiH0EVAqoJiIj4V1CFQJWroqSiRDUBERE/CaoQiO0ay57SPaoJiIj4\nSVCFQExkDHtK9mh0kIiInwRVCMR2jSW/JJ9evaCyEvbuDXSJREQ6tqAMAc0VEBHxj6AKgZiuMewp\n1TBRERF/CaoQiI2M1TBRERE/Cq4Q6FofAqoJiIi0v6ANAdUERETaX1CFgPoERET8K6hCoKmagHOB\nLZOISEcWtCHQsyeEh0N+foALJSLSgQVVCMRExtSFAHhNQuoXEBFpP0EVArFdY+uWkwZNGBMRaW9B\nFQI9I3tSWFZIVXUVoJqAiEh7C6oQCA8LJ6pLFAVlBYBqAiIi7S2oQgAO7BdQTUBEpH0FXQg07BcY\nNQq+/jrABRIR6cCCMgRqawJDhkBxMeTmBrhQIiIdVFCHgBmceCKsXBngQomIdFBBGQJ5+/PqtidO\nhM8/D2CBREQ6sKALgcExg9myZ0vdtkJARKT9+BQCZjbdzNLNbKOZ3drE8yPM7DMzKzWzGw/n3MaG\nxQ5jY/7Guu3a5iCtISQi0vZaDAEzCwMeB84ExgAXmtnIRoftBq4FHjqCcw8wvNdwNu3eVLfdrx90\n7w6bN7f8YURE5PD4UhOYCGxyzmU65yqA14BzGx7gnMtzzn0JVB7uuY0NjhnMtoJtVFRV1L+ImoRE\nRNqFLyEQD2Q12M6u2eeLwz63S6cuxEfHs3Vv/SwxhYCISPvoFOgCNDR37lwA7D/GWzFvcfvs2wEv\nBH796wAWTEQkCKSlpZGWltamr+lLCGwHkhpsJ9Ts88VhnVsbAvn/m09kz8i6/SecAP/+N1RUQESE\nj+8sItLBpKamkpqaWrc9b968Vr+mL81BK4GhZpZsZp2BmcCCQxxvrTgX8DqHN+6uHyEUFeUtJqcl\nJERE2laLIeCcqwLmAIuAtcBrzrn1ZnaVmV0JYGZxZpYF3ADcYWbbzKx7c+e29J7Dew0/YJgoqF9A\nRKQ9mAuSAfhm5mrLkrE3g1OeP4WsG+r7lJ98Er74Ap59NlAlFBEJLmaGc85aPrJ5QTdjGCAxOpG8\n/Xnsr9hft09rCImItL2gDIHwsHAGxwzmm/xv6vaNHetNGCsqCmDBREQ6mKAMATi4c7hzZzj5ZHjv\nvQAWSkSkgwneEIg9MAQALrgAXn89QAUSEemAgjYEhvUaxqb8TQfs+8lP4IMP1CQkItJWgjYEGjcH\nAcTGek1CCxcGqFAiIh3MURUCAD/9KbzxRgAKJCLSAQVtCMQdE0dZZVndTedr/ehH8PHHUFAQoIKJ\niHQgQRsCZubdW6BRv0CPHpCaCu+8E5hyiYh0JEEbAuB1DjfXJKRRQiIirRfUIdDUMFGAs8+GTz6B\n/PwAFEpEpAMJ7hBopnM4KgqmTYO33gpAoUREOpCgDoHRfUbz753/bvK5X/zCW1QuSNa/ExE5KgV1\nCIyNG0tucS65xbkHPXfmmbBvn9csJCIiRyaoQyA8LJxTk0/l460fH/RcWBhcey089lgACiYi0kEE\ndQgATBk0hcVbFzf53MUXw+LFsG2bnwslItJBHB0hkNF0CERFwUUXwRNP+LlQIiIdRNCHwJg+Yygu\nLyZzb2aTz8+Z491tbP/+Jp8WEZFDCPoQMDNSB6byccbB/QIAQ4bApEnw8st+LpiISAcQ9CEAMGVg\n8/0CANdf73UQa7ioiMjhOTpCoKZz2DXzLT9livfv4uZzQkREmnBUhMDQ2KGY2UGLydUyg+uug0cf\n9XPBRESOckdFCJjZIYeKAsyaBcuWeTejFxER3xwVIQBev0BzncMA3brBZZfB44/7sVAiIkc5a66d\n3d/MzB2qLNsKtjHh6Qnk3pxLmDWdXdu2wbhxkJnpzSEQEenIzAznnLXmNY6amkBSjyT6HNOHJRlL\nmj8myeskfvFFPxZMROQodtSEAMBNKTfxwKcPHPKY2uGi1dV+KpSIyFHsqAqBWWNnse7bdXyZ82Wz\nx3zve94tKHUzehGRlvkUAmY23czSzWyjmd3azDGPmdkmM1ttZuMa7L/ezNbUPK5rTWE7h3fmppSb\n+O9P/vsQZYWHHoLbboPS0ta8m4hIx9diCJhZGPA4cCYwBrjQzEY2Oub7wBDn3DDgKuCpmv1jgF8A\nE4BxwFlmNrg1Bb5i/BUszVxKel56s8ekpsLxx2vegIhIS3ypCUwENjnnMp1zFcBrwLmNjjkXeAnA\nObcC6GFmccAoYIVzrsw5VwUsBX7SmgIf0/kYrp14Lb/79HeHPO7BB70awa5drXk3EZGOzZcQiAey\nGmxn1+w71DHba/Z9DZxiZjFm1g34AZB45MX1zJk4hwUbFrCtoPkbCQwbBrNnw9y5rX03EZGOq1N7\nvrhzLt3Mfgd8ABQDq4Cq5o6f2+AbOzU1ldTU1CaPi+kaw1UnXMWcf83h7ZlvNztv4K67YORIuOYa\nGDPmiD+GiEhQSEtLIy0trU1fs8XJYmZ2EjDXOTe9Zvs2wDnnftfgmKeAj51zr9dspwOnOud2Nnqt\n+4Es59xTTbzPISeLNVZeVc7Ul6ZyavKp3DflvmaP++tfvWahxYshIcHnlxcRCXr+miy2EhhqZslm\n1hmYCSxodMwC4KKaQp0E7K0NADPrU/NvEvBj4JXWFLhW5/DOvHXBW7z8n5d5Y23z40GvuAKuvBJO\nPdWbSSwiIvVabA5yzlWZ2RxgEV5oPOucW29mV3lPu6edc/8ysx+Y2TfAPuDSBi/xlpnFAhXAr5xz\nhW1V+L7H9OXtmW8z7W/TGBY7jOP7H9/kcTffDJ06eaOGFi+GQYPaqgQiIke3o2btoEOZv3Y+Ny26\niU8v+5TEHs33Oz/xBPz+9/DllxATc6QlFREJDm3RHNQhQgDg4c8e5oXVL/B/l/4fMV2b/4a/9lrI\nzfVmFFurLp2ISGCF1AJyLbkp5SbOGHIG57x2DiUVJc0e99BDsGmT12EsIhLqOkxNAKDaVTPr77Mo\nqSzhzRlvEh4W3uRx6elwyimQlqahoyJy9FJNoJEwC+OFH71Afkk+D376YLPHjRwJv/sdzJwJ+/f7\nsYAiIkGmQ9UEam0r2MYJT5/AB7M/YFy/cU0e4xxceink5cE//gEREW3y1iIifqOaQDOSeiTx+zN+\nz6y/z6K0sumlRM28fgEzuPhi3X9AREJThwwBgNljZzOy90juXHxns8dERHijhLZv90YNBUmlSETE\nbzpsCJgZT531FK9+/eohb0nZtSssXAjLl8PttysIRCS0dNgQAOjdrTePTn+U69+7nmrXfHtPdDS8\n/773uOUWBYGIhI4OHQIA5406j87hnQ+5vhBA797w0Ufw8cdwww0KAhEJDR0+BMyMB05/gDsX30lF\nVcUhj42NhQ8/9JqGfvUrKC/3UyFFRAKkw4cAwJRBUxgUM4jnVj3X4rE9e8KiRZCVBZMmeRPLREQ6\nqpAIAYDfTvkt9yy9h/0VLc8Oi472Oosvv9ybWfzkk2oeEpGOqUNOFmvO+W+cz8T4idxy8i0+n5Oe\nDj//OQwdCs8+C927t2MBRUQOgyaLHab7p9zPQ589xAOfPEBldaVP54wcCZ9+CsccAykp8M037VxI\nERE/CqkQGNF7BF9c8QUfbf2ISc9OYt2366iqriKnKIfPt3/OzuKdTZ4XGenVAn71K6+f4J13/Fxw\nEZF2ElLNQbWcczz95dPc/tHtFJcXE9s1lgFRA8jbn8fiixczNHZos+d++qm3zMT48fDoo9C/v1+K\nLCJyEN1UppX2le+jU1gnunTqAsAzXz3DPUvuaTEISkrgvvvg6afh3nu9+xiHN71qtYhIu1EItANf\ngwDg66/h6quhuBj++EfvZvYiIv6iEGgnz3z1DLd8cAunJJ9CSkIKKQkpTBgwgWM6H3PQsc7B/Pne\nchMTJsCdd8K4plevFhFpUwqBdpRdmM1nWZ+xLGsZy7KXsWbXGkb0GkFKQgqzxs4iJTHlgONLSuCx\nx+DPf/aWoPjFL7yhpT16BOgDiEiHpxDwo9LKUlbtWMUn2z7hoc8e4uWfvMwZQ8446LiqKm8Nor/+\nFZYuhXvu8Sadqc9ARNqaQiBAPt32KT9+/cfNBkGtf/8b5szxbmH5pz95w0tFRNqKQiCAfA0C5+CV\nV+C22yApCX75S5gxw5t7ICLSGgqBAKsNguu+ex23nnwrEeHN36i4shLefReeegq+/BLOOw9+/GM4\n7TTo3NmPhRaRDkMhEASyCrK4YuEVfLv/W1449wWOjTu2xXO2boU33/RucL9+PXz/+14gTJ8OUVF+\nKLSIdAgKgSDhnOO5Vc9x20e3ERMZU7d/eK/h3D/lfo7rd1yz5+7Y4S1D8fbb8Nln3lyDyy6Ds8+G\nTp38UXoROVopBILM3tK97Nq3q2570eZF3Lv0Xn447Ifce9q9xEfHH/L8ggIvEJ5+GjIyvFFFl18O\nCQntXHAROSr5LQTMbDrwR7wF5551zv2uiWMeA74P7AMucc6trtl/A/ALoBpYA1zqnDvonl0dIQSa\nUlBawAOfPMBTXz7FqcmncsGYCzh7+NlEdTl0u8+aNV7/wauverWDX/4Spk2DsJBa8k9EDsUvIWBm\nYcBGYCqQA6wEZjrn0hsc831gjnPuh2b2XeBR59xJZjYA+AQY6ZwrN7PXgX86515q4n06ZAjU2lu6\nl3fS32H+uvkszVzK1MFTmTF6Rl0g5Jfkszx7Oeu+Xcfk5MmcOOBEzIziYi8InnwS9uyBiy7yHkOG\nBPoTiUig+SsETgLuds59v2b7NsA1rA2Y2VPAx86512u21wOpQDiwDBgHFAH/wAuID5t4nw4dAg01\nDoS47nHsLN7JifEnMqLXCD7a+hFllWWcP/p8ZoyewcT4iYCxahW89JIXCsOHw8yZ3iijfv0C/YlE\nJBD8FQLnAWc6566s2Z4FTHTOXdfgmIXAfzvnPqvZ/hC4xTn3lZldB9wP7AcWOedmN/M+IRMCDe0p\n2cP2ou2M6j2K8DBvWrFzjjW71jB/7Xzmr5tPaWUp548+nwvGXMCJA06kstJ47z144w1v2OnYsV7f\nwYUXqjNZJJQE/Z3FzKwncC6QDAwAupvZz9rzPY82MV1j+E7f79QFAHj/YcfGjeXeKfey/pr1LLxw\nIV07deWif1zEoEcHcfvHN9Nr3Gf86a97yclx3Hijd9Ob4cO9TuWysgB+IBE5qvjyd+N2IKnBdkLN\nvsbHJDZxzOnAFudcPoCZ/R2YBLzS1BvNnTu37ufU1FRSU1N9KF7HZmYcG3csx8Ydyz2n3cPXu75m\n/rr5/Oqfv2Lzns0AJEQnED0rmsgiuGUTXHt5Emd1fpiZ05M54wwtYifSUaSlpZGWltamr+lLc1A4\nsAGvY3gH8DlwoXNufYNjfgBcU9MxfBLwx5qO4YnAs8CJQBnwPLDSOfdEE+8Tks1BreGco6CsgKyC\nLPZX7K/b//fVi/nzV48wcMt9ZPz9Sk6cYJx1Fpx1lldbEJGOwd9DRB+lfojoA2Z2FV4H8dM1xzwO\nTMcbInqpc+6rmv13AzOBCmAVcLlzrqKJ91AItKF1367jkrcvoWt4d44Pu5gdn6fwf+8Mo7rKGDW6\nmkHf2UXy6G+ZNMkRF3fw+VGdoxgQNaDurmsARWVF5BbnMjhm8AHNVyISGJosJodUWV3J86ue56Ot\nH7Esexn7yvfRLTyK3H05dKEH4aV9KS4KIyICoqO9ZqPahe0KywrZUbyDnpE9iYmMYUfxDiqrK4nt\nGktEWATXf/d6Ljv+shbnO4hI+1EIyGHJKcqhpKKE+Oh4Ijt53/aVlbBsGSxYAK+/Dj17ws9+BrNm\nQf8BVezat4s9pXvo370/PSN7YmYsy1rGH5b/gY+2fsTVE67mlpNvIbpLdIA/nUjoUQhIm6quhk8/\nhf/5H2/46ZQpcO21MHkyWBO/Zhl7M5ibNpf3N7/P3afezeXjL6dT2MFjDXbt28W2gm3Nvu+QmCHE\ndI1p9nkRaZpCQNpNUZE3Me3xx72lKn70I69jeeLEg++StmrHKm7+4GY27d7EmL5jSIhKYEDUADbv\n2cyy7GXkl+QzOGYwxsG/q9Wumk35m0iITuCkhJM4beBpnDPiHHpG9vTTJxU5eikEpN05561u+u67\n8M9/equeTpwI48bBccfByJGQmAg9ejjSd68nY28GWQVZbC/aTnKPZFISUxjZeyRh1vyUlMrqSr7e\n9TXLspYZFKGPAAAN10lEQVTx/ub3Wbx1MZOTJ3P28LPpEVk/vnVQz0Ec3/94Ooe3fAOGiqoK1uxa\nw8bdG+v2hVkYccfEkRCdQEJ0wgGd3iJHI4WA+F1WlndTnNWrvdtnbtrk7auqgoED4cQTISXFe4we\nfWT3Vi4sK+Tdje/y/ub3Kav0Zr5Vu2o27t7IpvxNjOs3jvH9xpPYI5HE6ET6HtOXvP15ZBVmkVWQ\nxeqdq/ky50uSeyYzus9ows0rRJWrIrc4l+zCbHKKchjVexQzRs9gxpgZDO+lsbNy9FEISNAoLITN\nm2HFCq+jedky2LmzPhTGjfNqDAkJEBd3ZOEA3jDVlTkr+c/O/5BdmE1WYRY7i3fS55g+JER5f+Ef\nG3csE+MnHrJJqaq6ik+2fcL8dfN5a/1bxHaN5eTEk0lJSOGkhJMYHDP4oJpCaWUpZZVlB9RORAJJ\nISBBLS8Pli/3AmHtWsjO9moNBQUwbBiMGePVFr77XZg0KXB3VauqruKrHV+xPHs5y7KXsWL7CrIK\nsojpGkNidCLVrprswmwKygqICItgwoAJXDDmAn4y6if0616/el9hWaEXTAVZVFRXcMaQM3xquhI5\nUgoBOSrt3w8bNnjB8PXXXlB88YXXv5CSUh8Oo0ZB795Nj0xqb1XV3vDYrMIsDCOxh9fsVF5Vzvvf\nvM/8dfNZsGEBReVFdeccE3EMiT0SSYhOoLSylK17tjJn4hyuPOFKYrvG+v9DSIenEJAOo6wMVq70\nHmvXwrp1kJ4O+/ZB377ectlDhnjhMGaMFxADB0K3boErc1O/r9YgsVbnruYPy//AO+nvHNCElNQj\nifNHnc/5o89v8W5zIoeiEJAOr7TU61vIzfU6oRsGRGamN7lt8GAvFGprEMOHe/0PERGBLr1nb+le\nCkoLAHA41u5ayxvr3mDhhoWM7jOaC8ZcwHmjzusQgVBZXcnSzKWUVpZyUsJJqgG1M4WAhLTqam/I\n6ubNsH69FxBr18I333ih0b8/DB1aX3sYMwZOOAG6dg10yT1llWV8uOXDuqal0X1Gc3y/4+ualGK7\nxtbNrQgPC6d/9/4k9khss9nZ5VXlbC/cXtfBnl2YTXZhNnn78+qOaTystrn3Lqks4X83/S//SP8H\nyT2T6dGlB59v/5z+Uf2ZMGACyT2SSYhOID6qfrb6Qdejqoycopy6IcblVQfdhRaAyE6RjO8/npSE\nFMbGjSXMwti5bydZBVmUV5V77xMdHxL9MQoBkWaUl3ud0Js2eTWHtWu9+zavW+fNb5g82et/OPFE\nLywCrayyjI8zPiY9L73uS3lv6d665yuqKrwvyMIsOoV1oltE69rBKqsrKSgtoF/3fiREJ9QNt02I\nTqBPtz51zVpV1fXDarMKs9hXsa/J1wu3cFIHpjJj9AwGxQyqO3ftt2tZtWNV3fDd7UXbqag+aP1I\nADqFdSI+Kp7E6MQDljZprLi8mJXbV7Isexlb9mypW9MqsUciEWERZBdmk1ucS4/IHgfMYI+JjDng\nc9b+2z+qPxFhLVcbwyyMft37Ed0l+oBmv0BSCIgcpn37vI7opUu94axffAFdunjBkJTkDWFNTITx\n4498nkN7cs6xt3QvpZWlrXqdMAujd7feR/1qsMXlxXQO73zQX/1V1VXk7c+j2lUDXjPc7v2768Ks\nNpCyCrPYUbSDKlfV4ntVVleSW5wLUBcgDcOktgaX1COp2RpTfkk+WQVZZBVmkVOUQ2V1ZZPHRYRF\nMCBqAIk9EunfvX/dkvHZhdkHDEa4ZuI1CgGR1nDO61tYs6Z+CGtGhhcOO3d6s6OHD/dWWY2O9jqo\nJ0/2+iGC5I9B8bOC0oIDms9qv9Rrt7cVbKN/VH9OSjiJlIQUyirLWJa9jGXZyygsKySpRxKJ0YkM\niBrQbJNVWWUZOcVe01hOUQ49I3vWBU105/qayJNnPakQEGkvtfMcMjK8yXCFhbBtG6SleZ3Oqale\nU1K3bl4/w4gRMG1a8PQ5SGBUu2rWf7ve++LPWkaXTl1ISUghJTGFITFD2rQpSc1BIgHgnDfPYckS\n2L0bSkq8uQ+rVnlLapx+Opx5JgwYALGx3lyHhITADmeVjkkhIBJkdu/2Ftv7+GP49ltvOy/Pa2rq\n2RMGDfJGKU2eDKec4s11ULOSHCmFgMhRonY469at3uJ7S5d6j7Awb47D8OHe49hjvU7qPn0CXWI5\nGigERI5itZ3SGzZ4Q1k3bPA6qP/9b6/p6DvfqQ+HoUO95qV+/bzmpWAbtSSBoRAQ6YCc8zqgv/7a\nC4eNG+snwOXmwp493mJ7tSOWevSAXr28R9++Xk3ixBM1gikUKAREQlBlpTdSqaDA+3fvXq/vIT/f\na3Javdpbg6mo6MBmpW7dvNVbhw/31mHq0mCl7MjI+lCJi4PkZAXI0UAhICLN2rXLC4hahYVejWLj\nRtiyBSpqJu465y3gVxsqOTlegIwd6z0SErxmqH79vLCoqPCCKCysfnG/Pn2CZ62mUKIQEJF2sXu3\n1zexZo0XCrVNUWVl3pd9p07e3eR27fL25+V5+7t18x79+9ev2VT7b1KSFxzSdhQCIhIUqqu9FV/3\n7/fmTWRn1y/oV7vy69693j0jBgyo78Po1q2+2SkiwmuGGjTIGzobEeHVOCoqvNpKRER9ANX+XLsd\nqhQCInLU2LvXWwJ8506vprF7txcatcrKvNFSW7Z4/1ZW1n/Jm3lhUPuoDYeKCq9jPCHBeyQleSEy\neLAXKD17es9HR3tNWR2tn0MhICIhzTmvQzwry6t9ZGZ6czG2bvV+ru3nKPBu50BsrFcDiY4+sDbR\n8NGrV/1Cgg3vbFdb2ykp8R4REfWd6b16ebWX6LZZ5dtnCgERER/t3+8Fxu7dXjA0rE00fOTl1YfK\n7t3155t5tYmuXb1HZWV9yHz7rbfGVGSkFwa1QRMd7R1b24TVpUv9sN7o6Pp1pxr/27XrgU1fkZFN\n96e0RQiEcGuaiISS2k7rhIT2eX3n6sNg7976GkhJSX3glJZ6wbJ1q/f8/v31j9oaRu3Ptc1e5eXe\no3Nnr/yxsfXNX23Bp5qAmU0H/giEAc86537XxDGPAd8H9gGXOOdWm9lw4HXAAQYMBu5yzj3WxPmq\nCYiINMG5+o732uavrCy45JLW1wRaHLBlZmHA48CZwBjgQjMb2eiY7wNDnHPDgKuAp7yCu43OueOd\nc+OBE/AC4h+tKXAoSEtLC3QRgoKuQz1di3qheC3MvCaiXr28CX9TpsDFF7fNa/syancisMk5l+mc\nqwBeA85tdMy5wEsAzrkVQA8zi2t0zOnAZudcVivL3OGF4i95U3Qd6ula1NO1aFu+hEA80PCLO7tm\n36GO2d7EMT8FXj3cAoqISPvxy/w9M4sAzgHm++P9RETENy12DJvZScBc59z0mu3bANewc9jMngI+\nds69XrOdDpzqnNtZs30O8Kva12jmfdQrLCJymPwxRHQlMNTMkoEdwEzgwkbHLACuAV6vCY29tQFQ\n40JaaApq7QcREZHD12IIOOeqzGwOsIj6IaLrzewq72n3tHPuX2b2AzP7Bm8E0KW155tZN7xO4Svb\n5yOIiMiRCpoZwyIi4n8BX9jVzKabWbqZbTSzWwNdHn8yswQzW2xma81sjZldV7M/xswWmdkGM3vf\nzHoEuqz+YmZhZvaVmS2o2Q7Ja2FmPcxsvpmtr/n9+G4IX4sbzOxrM/uPmf2PmXUOlWthZs+a2U4z\n+0+Dfc1+djO73cw21fzenOHLewQ0BHyZiNbBVQI3OufGACnANTWf/zbgQ+fcCGAxcHsAy+hv1wPr\nGmyH6rV4FPiXc24UcByQTgheCzMbAFwLjHfOjcVrwr6Q0LkWz+N9PzbU5Gc3s9HABcAovNUb/mzW\n8rqpga4J+DIRrcNyzuU651bX/FwMrAcS8K7BizWHvQj8KDAl9C8zSwB+ADzTYHfIXQsziwZOcc49\nD+Ccq3TOFRCC16JGOHCMmXUCuuLNQwqJa+Gc+wTY02h3c5/9HOC1mt+XDGAT3nfsIQU6BHyZiBYS\nzGwgMA5YDsTVjq5yzuUCfQNXMr/6A/D/8NaaqhWK12IQkGdmz9c0jT1dM8Ai5K6Fcy4H+D2wDe/L\nv8A59yEheC0a6NvMZ/dl0u5BAh0CAphZd+BN4PqaGkHj3voO33tvZj8EdtbUjA5Vhe3w1wKvyWM8\n8ETNulv78JoAQvH3oifeX77JwAC8GsHPCcFrcQit+uyBDoHtQFKD7YSafSGjpor7JvA359w7Nbt3\n1q69ZGb9gF2BKp8fnQycY2Zb8OaUTDGzvwG5IXgtsoEs59wXNdtv4YVCKP5enA5scc7lO+eq8Bag\nnERoXotazX327UBig+N8+j4NdAjUTUQzs854E9EWBLhM/vYcsM4592iDfQuAS2p+vhh4p/FJHY1z\n7tfOuSTn3GC834PFzrnZwEJC71rsBLJqlmIHmAqsJQR/L/CagU4ys8iaTs6peAMHQulaGAfWjpv7\n7AuAmTWjpwYBQ4HPW3zxQM8TqLlXwaPUT0R7IKAF8iMzOxlYCqzBq9I54Nd4/+HewEv1TOAC59ze\nQJXT38zsVOAm59w5ZhZLCF4LMzsOr4M8AtiCNwEznNC8Fnfj/WFQAawCLgeiCIFrYWavAKlAL2An\ncDfwNt46bAd9djO7HfgF3rW63jm3qMX3CHQIiIhI4AS6OUhERAJIISAiEsIUAiIiIUwhICISwhQC\nIiIhTCEgIhLCFAIiIiFMISAiEsL+PzvJydKziraLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5462a08390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = np.array(errors)\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 0], label='train error')\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 1], label='test  error')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACFhJREFUeJztm99v21QbgJ9jO07spk1WuyFp09ES0m5rN+iYVsG4QeIG\n7WL8s1xxw0VBMFUUEUZW6Ba6sjC5+aUljpM4tr8LFNOuZAh9OG0hjxRVsePk5Mk57znv61MRBAFT\nziKddwMuKlMxY5iKGcNUzBimYsYwFTMG5XUnhRD/+rk8CALxZ8enPWYMUzFjmIoZw1TMGKZixjAV\nM4apmDFMxYxhKmYMr135ThpZllEUBSEEkiSRTCaZnZ1lOBwyGAywbZt2u80kimsXSszs7CzpdJp4\nPI6qqmxvb/P+++9Tr9epVqt8++23PHz4kF6vF3lbzlWMoigoikIsFkNVVfL5PMvLyyQSCRKJBB9/\n/DEPHjygWq3y5MkTHMdhb2/v3y1GCEE6ncYwDHK5HLlcjo2NDTY2NvB9H8/zWF9fR5Zlrly5QrFY\nJJPJIMvyRNo3UTGj3qFpGslkkqWlJfL5PIVCgUKhwNbWFu+++y6O4+A4DpqmIUkSc3NzzM3Nkc/n\nWVxcBMC2bYbDYWTxRrzujf/JsoMsyywsLLC4uMjdu3fZ2toKg2sqlSKdTpPJZMhkMgyHQ1zXDUUC\nBEHAd999xzfffMPOzg47OzvU63Ucx/m/5IwrO0TeY4QQxGIxkskkV69eZXNzk/v37/PJJ58gxB9t\nCoIgfK4oCrIsnzoPcP36dVZXVxFCsL+/T7fbpd/v43neP97uyMWoqkqhUKBYLHLr1i1u3bpFoVAA\n/pAx+sX/6rmiKCQSCXRdR9M0VFU9I++fIjIxQggURWFubo5r165x7949bt++zebmJpqmEQTBqQfA\nYDDAdV1838f3fWRZRpblcNaSJIl4PE4ikUDTNGKx2OUTo6oqKysrrK2t8eGHH/LBBx+QzWbRdR1F\n+f1j2+027XYbAN/3KZVKlEolbNum0+lgGAamabKxscHNmzfRdR1VVaNq8ikiExOPx1lZWeHu3bts\nb29z586d8NxwOKTb7WJZFi9evADA8zy++OILPvvsM+r1OvV6ndXVVQqFAsPhkLfeegtVVS+/GEVR\nyGazFAoF0uk0QRDgui69Xo+nT59SqVQol8vs7+8Dv8eTg4MDarUajuMA4Lou3W6XIAhIJBLhDDUJ\nIhMTi8XOiBkMBrRaLR49esTOzg67u7vs7e2duVYIEQZdz/OQJOmMmNFroiLSHjM/P8/S0hLJZBLf\n93n58iXVapVyuczu7i7VavXUNZIkIcsyMzMz6LrOe++9x/b2Npubm6EUIQSapmEYBslkEkmKpkAQ\nmZjRUj6XywHQ7/d5+fIlz58/5/Hjx+zt7YVZ9OjXV1WVWCyGYRgYhsGdO3e4f/8+2Ww2zLoBEokE\n6XSamZmZyyfmJEEQ4Ps+nU6HRqOB4zhIksTi4iK5XA7TNDFNM+wpo9lobW2NTCaDruvh0BJC4DgO\njUaDTqeD7/uRtDlSMSfXKL7vY9s2jUaDXq+HoigsLS1x+/Zt1tfXWV9fJ5VKkUqlwh7zZzEkCAJ6\nvd7lFTMYDHj27BmlUolcLkcqleLNN99ElmWy2SwfffRRmFWbpolhGGG5QdO0UMro76jX+b6P67q4\nrovneZElkZGL+eGHH4jFYpimycrKCisrK9y7dy8cFq/2ipMiTp4TQpySMhITFZGJ8X2fdrsdZsDA\nmS86Lon8s2NBENBsNnnx4gWHh4ccHx9fzqHkeR62bdNsNsOK26vd/q+SxlevaTabHBwccHh4iGVZ\ndDqdqJofnRjXdXn+/DmPHj1icXGR+fl50uk0qVQq/LKxWAxFUXAch263S6vVotVqhcdN02RhYQFJ\nkgiCgHa7za+//kqj0Yh0GMEExHieRzabxTAMVldXz6xeFUWh2+1Sq9WoVCpUKhU0TUPTNG7cuIFp\nmuFrR2KazeblFeP7Pr1ej3q9TqlUYjAYkMlkMAwjHDK6rpNIJKjVatRqNY6Pj7Esi3w+Tz6fJ5fL\nheWH0dQ/mpmiJjIxQRCE1bVSqUSlUglviwBnxNTrdfr9Pv1+n3feeQff97l27dqptdAkN2tHvvL1\nPI9Op0O/3w8LTyNUVUVRFGzbxrZtPM/D931UVWVhYYFkMnlqpmo0GpTLZX777TeGw2Gk7Y5czCir\ndl33tWuWk8MjlUqxvLxMOp0OcykAy7L4/vvvsSzr8osZcXJIjCOdTnPlyhUKhQJra2uYpokQgl6v\nh+M4tNttHMeJXApcoFu0QghM0+Ttt98Oc6fRUOp2u5HnRq9yYcQAZDIZNjc3WV5eZnZ2FlmW8TyP\nWq3GwcEBlmVNTMyF2QYihOCNN97g5s2b5HK58NaI67pYlsWPP/44kaA74kL0mHg8jq7r5PN5rl+/\nTiaTQQhBv9+n3W7z5MkTHj58yC+//PLfESOEYGZmBsMwuHr1Kuvr6+HqeBRbfv75Z77++msajcZ/\nR4wkSayurrK1tUWxWCSZTAK/z2KtVotKpUK1Wg0rf5PiQoi5ceMGn376KcVi8VR23Wg0+OmnnyYa\ndEecq5h4PB5uBykWi8zPzwOEqUGlUmF3d5ejo6PIk8ZXOdeNQ7quh2XNUWkzCIIwtuzv7/Pll19O\nJJt+lXMVo6oquq6Hj1EeZVkWjx8/5unTp1iWxWAwmHj7znUoybIc3ksa7VwQQnB0dMRXX33F4eHh\nxGPLiHMTM0oubdum2+1i23a49+7o6IhyuXwuQXfEuYqxbTssUB0fH4dB9/DwkGfPntFqtSZagznJ\nuQ6lwWBAu92mXC7z+eef0+/3GQwGlEolLMvCtu1zEzOxzYlj3h9JkshkMiwsLIRly2azGa5yo56N\nxm1OPFcxF4HpP4v+TaZixjAVM4bXxpj/MtMeM4apmDFMxYxhKmYMUzFj+B9Lwz6foyNbdwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5462a15ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  9 Predicted:  9\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACa1JREFUeJztm9lvGtcegL9hgAGMA2UxxthsoXip69qNU1VVF7Wq+tC3\n/qV9rPpQ5aGqEiWK7TgYYwdLwTiAzeKwmWVmmPvQO9NmIbntBZzbyydZRpiBMx/n/M7v/M6xoGka\nU17FdN0NeFeZihnCVMwQpmKGMBUzhKmYIZjf9EdBEP7xc7mmacLrnp/2mCFMxQxhKmYIUzFDmIoZ\nwlTMEKZihjAVM4SpmCG8MfP9bxAEAZvNhs1mw2w2YzabsVqtWCyWt17b7/fp9XooioKiKPR6Pbrd\n7ria+lrGJkaSJOLxONFoFK/Xi8/nIxgMEggE3nptPp/n9PSUarVKrVYjl8vx9OlTVFUdV3NfYWRi\nTCYTkiQhSRJ2ux2v18vm5ibr6+sEg0FCoRCJRIJwOPzW98pkMmQyGQqFAsVikdnZWTRNo16v02q1\nUBRl7JKEN9V8/8oi0uFwEIvFSCQSrK2tkUwmmZubw+/3MzMzg9PpxOVycePGjbe+V61W4/Lyklar\nRbvdplAoUCgUePDgAXfv3qVardJsNhkMBv9p84YybBE5sh5jt9tJJBJ88cUXfPnll2xubmIymTCZ\nXozvmqbR7Xbp9/vGc4IgIAh/tM/hcHDjxg0EQcBkMnF1dUW73cbhcFAsFlFVlXa7PRIxwxipmJWV\nFT755BOCwSAmk+mFm9WRZZlsNks2m0XvrZIkYbPZjNcEAgHm5+ex2+3Y7XasViuCILC0tMTW1hay\nLHNxcYGiKKNq/iuMTIzNZiMcDrOyssLMzAzwu4Q//2iaRqfTYX9/n/v37zMYDNA0DafTaVwDkEwm\nURSFQCCA3W7HYrFgsVgIhUJsbGxQKBTY29sbVdNfy8jEKIpCvV6nUqkYQ+jZs2fk83lOTk7I5XIo\nikKn0yGXy5HP5w0x+o3rZLNZMpkMn3/+OT6fzxiObrebWCyG3+/HbB7bhAqMUIzevXO5HJqmMTs7\ny/HxMfv7+9y9e5e9vT263S69Xo9er0e/3zfEvEyxWKRYLLK0tISqqoa02dlZlpaW8Hg8r8SuUTMy\nMY1Gg3v37lGpVPB4PEiSZMwmZ2dnXF5eoqqqMdUOkwKwsLDA7du3WVxcHLuAYYxMTKvVYmdnh1Qq\nhdVqRRRF2u023W4XTdOGStARBMEYUpFIhFu3bhli9GsVRaHb7SLL8qiaPZSRidE0DVVVjVTeZDIZ\nAfdtUqxWKzabjfX1dT788ENu377N+++/j9frxWQyGT3t5OSEBw8ekE6n6fV6o2r6axlpBFNVFVVV\n/9I3KggCdrsdl8vF9vY2P/zwA6FQiIWFBURRZDAY0O12ubq64ujoiDt37nB4ePhCHjQOxhva34Ke\nv8RiMeLxOGtra0SjUWZnZ7FYLNRqNSqVCsfHx2QyGR4/fszh4SHlcnnsS4JrE2MymbDb7bjdbm7e\nvMnHH3/M6uoq4XAYTdMYDAZUKhUODw+5c+cOv/zyC+VymXq9PtaMV2fiYkwmE263G6/Xy8bGBuvr\n68RiMWKxmLHAzGazHB8fk06nSafTHB8fU61W6XQ6b41Xo+JaxHg8HhKJBN988w3ff/89LpcLt9tt\nvCabzfLTTz+RSqVIpVJcXV2NPdi+0s6Jfhq/i5mbmyOZTBIKhXC5XC+sk+D3gKwHXn2WmzQT7zGi\nKBIIBFheXjbEvIy+qh4MBvT7/YkWqHQmLkZVVc7Pzzk6OiISidDv9zGbzYiiaLwmGo3y1Vdf4fP5\nWFpaIp/Pc3Z2RqPRoNFoTCTOTFzMYDCgVCpxeHjIRx99ZOQjL4uZn58nEomwsrLC/fv3uXfvHqen\np7Tb7YkMrWsRc3l5ydOnT/n1119RFAWr1YokSQSDQYLBoFHpW1hYMEqlS0tLpFIp9vf3OT8/p1ar\nGcuDcfSgaxHz/Plzo3abzWaNJcH29jbb29skEgm8Xi9+vx+fz0ckEuHTTz/lt99+w263c3BwgKIo\naJpm/B4115Lg6TdUrVaRZRlRFLFYLMiyzPn5OfF4nFgshs1mQ5IkFhcXCYfDJJNJBEEgEomwvLxM\nOp3m8ePHNJvNkfeckRXD/9aH/3v20R/rlbxQKMTi4iIulwuXy8XXX3/Nt99+i9lsRpZlisUi+Xye\nn3/+mR9//JFCocDV1dXfyojHXgz/O+grcp1Wq0Wv10NVVRqNBk6nE6fTafSmUCjE/Py8UZrY3t6m\n0+nw6NEjUqkUjUZjZEPrWsW8jF7E6vf7VKtVYyi1223y+TxbW1vcunWLcDjMwsIC29vbeL1eHA4H\npVKJbreLqqr/PDHwRy/Sq3z9fp+zszOurq5wuVxEo1Hm5uYQRZH33nsPQRBIJpMkk0lUVeXs7Gwk\n27nvnJg/o8spl8vUajVisRj1et2oCs7MzBhli7W1NZrNJhcXF/98MfDHDKYoCv1+34ghmqYhiiKi\nKBq5jr7/NAqmx0CG8E73GL3C5/P58Pv9fPDBBwSDQZxO5wtF8sFgMPKN/ndajN1ux+PxsLGxwdbW\nFslkkkgkgtvtNtZWerVPluWRZsETFaNv3ouiiCAIDAYD41vWkz1RFHG73bjdbqLRKNFolNXVVdbW\n1ggEAvh8PmZmZhAEgXq9TrVa5cmTJzx58oTz8/ORba1MVIx+41arFbPZTK/XQ9M0Q5h+4ioajbK8\nvMzm5iZbW1vMz8/j9/uRJAmLxWJky+VymYODA3Z2dtjd3aVUKo2s0jcxMXqt1+PxEIvFCIVCVCoV\narWasdft8XjweDyEw2HC4TA3b94kkUjgdDpxOBw0m00qlQrtdpt2u006nWZnZ4dHjx5RLpfpdDoj\na+/ExJjNZubm5lhZWeG7777js88+I5vNcnJygiiKmM1m4vE48XjcOLunHwPR40m5XObo6IhCoUCp\nVCKVSrG7uztyKTBBMYIgIEkSLpeLxcVFVldXcTqd+P1+Y4iFQiFCoRCKoiDLMs+fP+fZs2fIsoyq\nqqTTafb29jg/P6dSqXB6eko+nx9LoXziwVdHFEXm5+dxu91GjJEkCYBut0ur1SKTyZBOp2m32/R6\nPQ4ODtjd3aXVaiHL8lj3sSc+Xf9ZjslkwmKx0Ol0XhgK+XyefD5viOl0OvT7fXK5HLlc7n9rU/+v\nop+uqtfrlEolLi4ujBxkZ2eHhw8fUiwWKZVKKIrCYDCg0+lMbCtlYmIGgwGNRoOzszMePnyIIAg0\nm00ajQblcplKpWKISafTHBwc0Gw2aTabk2riC0ysgqefapiZmcHr9eJ2u40gq58E19GF6IvHcTKs\ngnetpc13gek/i/5FpmKGMBUzhDfGmP9npj1mCFMxQ5iKGcJUzBCmYobwLxQF5FVmBc7bAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54616d9f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  3 Predicted:  3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAABx1JREFUeJztm11PGt0WgJ89wwyfDjN8yVcpaQQ0tqm2id70plftn+0f\naNLExLt+pFFrTLRtaotVEFBgAIUBzsUJ5G3r9PQkHapv57kThT15XHutvdYwYjwe4/Ij0p++gOuK\nK8YGV4wNrhgbXDE2uGJs8Pzsl0KIf30tH4/H4qrX3YixwRVjgyvGhp/mmOuGqqqoqkooFCIUCtFu\nt6nX61iW9dvXulFiNE0jmUyyuLjI8vIy29vbvHjxglar9dvXulFbKRKJUCwWefDgAY8ePaJQKOD1\neh1Z60aJSSQSrKyskMvl8HicDfYbs5WEEMTjcZaWlohGo64YAEmS8Hg86LpONpvF5/PR6/WcXdPR\nT/9NeDwevF4vmqYxPz9PMBjEsiwsy8KpQduNEDMp0YFAAFVV6ff71Ot12u02o9HIkTVvhBhFUQgE\nAvh8PhRF4fLykmq1SrPZZDgcOrLmjRAjyzJerxdFUZBlmU6nQ7lcptFo/N1iFEUhGAzi8/mmYr5+\n/eqKCYVCpFIpNE1DCEGv16PRaGCa5t+dY+bm5shms+i6jhCCbrdLvV7HNM2/O2JisRhLS0vE43EG\ngwHNZpNqtUqr1XIsYv7IAU+WZWRZZjgc/tJ/PBaLsbi4SDwex7KsaVdtmqZj1zhTMZIkoSgKiUSC\nSCRCpVKhWq0yHo+vPKhJkoQsy+i6TiaTQdM0RqMRo9HIsYPddG1HP/0fCCFQFIVQKEQ+n+f+/fuk\nUikURUGSrr4MWZZRVRVN00ilUgSDQfr9PoPBwHExM4sYj8dDPp+nUCiwtrbG0tISkiRxfHyMaZpX\nbovJwU5VVYQQNBoNPnz4QKVScSzpTphZxCiKQi6XY21tjcePH/PkyROWl5fRdR2fz3fle1RVJRgM\n4vV6EUJwdnbGwcEBJycnjkzt/slMc4zH40FRFEajERcXF3S7XTqdDv1+/8q/j0QiFAoFEokEsizT\naDTY39//d0WMEAKPx4OqqozH42/EDAaDK98TjUYplUokEgkkSeLs7Iz3799TqVQcj5iZiZFlGcMw\nyGazzM3NTV+3q0gA4XCYW7duYRgGkiTR6/WmZdqp88uEmYqJRCLcunULTdN+6T2appHL5TAMY9oK\nzEqM4zlGCIHf7ycSiZBMJkmn0wghME2TTqdDr9f7YVtMclEsFiOXyyHLMkdHR5yenmKaJv1+/+af\nYyRJwu/3YxgG8/PzpNNpFEXBNE263S4XFxc/iJmU6e/FVKvVqRjHr9vpBTweD6lUikKhQCQSYTgc\ncnh4yKtXrzg+PkaSJIT49r66ruvk83ni8Th+v59Go8HW1hZfvnxxPOlOmImYZDJJsVjEMAwsy+Lw\n8JCXL1/+TzGxWAyfz0e9Xmdra4tyuTwzMY7lGCEEqqqi6zoLCwusrKwwPz+PoigUi0VGoxGlUon1\n9XVOTk6oVCr0ej263S75fJ5MJkM4HJ6KE0JMeychxM1tCSRJwufzYRgGhUKB1dVVdF1HURSWl5cp\nFoucnZ1Rq9XY2dlhZ2eHWq1GrVYjl8uRzWYJh8NTIZOx5kTUjRQjhCAYDHLv3j0ePnzI4uIi4XCY\n8XhMs9mk3W5jmiaKoqBpGqVSiWg0SqfTwTRNdF3HMAwymQySJJFOp1lbW6PZbLK/v0+/37+Z5VqS\nJEKhEKurqzx9+pRCoUAgEKDdbtNsNimXy1SrVTKZDOl0mjt37nD37t3pYW8SJR6PB0mSyGazBINB\nPn36xMbGBqZpOh41jkWMLMuEQiE0TaPdbrO3t8fe3h4HBwfUajWazSaGYWAYBrFYjGg0Ov05lUqR\nTqenXffu7i7b29u8ffuWTqfDcDi8eVtpUmEmOUZVVarVKkdHRzx//pzNzU0uLy+5vLycTvJSqdS0\npC8sLLC+vk4ymaTVanF0dMTGxgbPnj2bjjOdbiDBATHj8ZjRaIRpmrx584Z+v0+r1aJer7O7u8vp\n6SmWZX2TI4bDIe12G0mSUFWVUqkEQK1WY29vj48fP3JycjKNllngyFYajUY0m002Nzd5/fo1w+EQ\ny7LodrtXTt9arRbdbhdd19F1HdM0GY/HnJ6e8u7dO8rl8pUnZCdxrFxbljWtQJOkatdJj0YjBoMB\nXq+XWCxGIBBACEGz2eTw8NDRG2t2ONpETgbXv4IQAl3XuX37NrquA/+NpHK5zPn5uePl+Xuu1X0l\nn8+HruuoqoplWfR6PVqtFhcXF45Xoe+5VmImXbUkSQwGg+mUz27C5yTXSszk1mu3251uw+FwOPNt\nBNdIzKRd+Pz5M+fn5zPfOt9zrb6DV6lU2NnZYTweT0cN3W7X0a+U2SF+tuCsnz7x+/34/X6CwSCh\nUIhGo0Gj0XBUjN3TJ9dKzJ/AfSzn/8QVY4MrxgZXjA2uGBtcMTa4YmxwxdjgirHhpyffvxk3Ymxw\nxdjgirHBFWODK8aG/wBF4F07H52FPQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54616b2fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  4 Predicted:  4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAABzdJREFUeJztm9tPE9sXgL+Z6bSdtlwKFCoUUokQJBWpMZIYEx9IfPDN\nf9XEF1/0PPimgEbDtYiFpmWGsdALvczt92A6yJFyNOfXETzzJX0p7XT369prr702IziOg8+PiL97\nAFcVX0wXfDFd8MV0wRfTBV9MFwKX/VEQhD9+LXccR7joeT9iuuCL6YIvpgu+mC74Yrrgi+mCL6YL\nl9YxvUAURQYHB4nFYtRqNWq1GpZlYVmW10O5FE/FCIKAoig8ePCAbDbL2toa79+/p1qtUq1WvRzK\nP+KZGEEQiEQijIyMkM1mefr0KaFQiHq9zt7e3n9XTCAQYGJigrm5OWZnZ0mn0ywtLaEoCi9fviSf\nz3s1lJ/CUzHJZJK5uTmmpqYYHR1lZmYGSZL48OGDV8P4aTxblURRJBaLMTIygqIoOI5DpVKhWCxS\nq9W8GsZP41nEiKJIX18fiUSCSCSCbduUy2W+fPnCycnJxYMLBAgGg3Qa9qZpYpomXjTwPRUzMDBA\nMpkkFoth2zb7+/u8e/eOYrF44XvGxsZIpVI4joNpmhwdHaGqKoZh9Hx591RMNBplaGiIcDiMbduo\nqsr6+jq6rv/wWlEUGR8fJ5vNAt+iJZfLYZomlUqFer3e08jxTIzjONi2jWVZ//iFYrEYAwMDLCws\nsLy8TDgcRpIkPn78yNjYGOvr62xsbNBqtXo2Xk/ECIKAJEmIooggnDXMHMdxH9/T399PKpVifn6e\nhw8f0t/fj6IojI6OEovFaLfb5HK56y1GFEUikQhDQ0MkEgkSiQTRaBQAwzA4PT3FMIxz70kkEszP\nz5NKpYhGowSDQQRBYGJigqWlJT5//kwwGOztuHt6dc4q3ng8zvDwMCMjI4TDYSzLot1unxMjiiKy\nLHPjxg0ymQwTExMoikIg8O33Gx0dZXFxkampKfe5XtHziBEEgWAwSDQaRVEUZFmm1WpRLpep1+uY\npolt2wAkk0k34d6/f59UKoUo/p4GgCdiQqEQkUiEcDiMLMscHx+jaRrVatUVIwgCyWSSxcVFstks\n2WyWQCDgSuvkpotyUi/wvO1gmib5fJ7V1VUKhQKGYbhi0uk0jx8/Jp1OIwgCqqqiaRqJRIJkMolh\nGDSbTRqNRs/leJJjBEFwVyTLstjf32dlZYVCoeBWsoFAgHQ6zaNHj5icnMS2bQ4PD/n06RPFYhHL\nsmi1WpycnNBoNNxI6hU9j5hAIMD4+Dhzc3MMDQ1hWRa1Wg1d12k0GoiiyK1bt5iZmWF+fp6BgQF2\nd3fZ2dmhVCpRKpWIx+PYtk2lUiGfz6Pr+vWvfGVZdsXE43FXzNHREaenp4iiyMzMDE+ePHHF5PN5\nXrx4QbPZpN1uMzMz47kYT1K+KIpIkoRlWZyenqKqKvv7+wQCAaanp5mbm+POnTtIksT6+rpb2aqq\nem7KNJtNdzW79jkGzvY+HTGapnFwcIAkSUxPT3P79m0WFhaQJImNjQ1XjKZp5wR8L+ba5xhZlkml\nUmQyGfr7+7FtG1mWicViZLNZlpeXmZ2dxbIsdnd3+euvv9xyv1M1h0IhBEGgVqtRKBQol8t/hpjJ\nyUkymQyNRgNVVZFlmb6+PrLZLM+ePcNxHAzDIJfL8erVKzRNwzAMJEk6tyWoVquumGuffOEsx0iS\nhKIo3L17l3a7TSaTIRKJkMvlyOVybG1toWmam0MGBwe5efMm0WiUarWKruuUSiUqlcr1jxg4q1ZF\nUURRFO7du8fU1BTpdBpZlikUCrx584bNzU10XXe/dDweZ3p6mlgsxsnJCZqm/TliHMeh2WxSq9UI\nBAKEw2FGR0eJRqPIsoyu6+zv77Ozs3NOCnyLtM5m0bIsDMOg3W570t7s+apk2zaNRsPdF4VCIYaG\nhhgfH0cURVRV5eDggN3dXcrl8oXXcBzHPa38ftPZS3oeMZZloaoqu7u7pNNpotEo9Xqd4+NjVlZW\nWF1d5e3bt6iqSr1e73qdbk2tXtFzMaZpUiqV2N7edpvhx8fH7O3t8fr1a54/f87JyQnVatWTSPhZ\nei6m04ZUFAVd19nc3KRYLHJwcMDa2ppnXf9fpediWq0W29vbaJrGxsYGw8PDaJrG0dER5XKZZrPp\n2fT4FXouxrZtt33ZbDY5PDykVqtRr9cxDONCKZ02RSgUIhqNupWvl3hSx3SmSrvddsv5zuMiOgWh\noigMDAwQDod/OGHoNZ518C4T8Xc6Z1CdKOv0ir2cclfyX806B3ONRoNKpeJJK/PvXEkxcBY1nYLO\nF3NFuNJiOvuj31HjXHkxrVYL0zQ9/2zPz5V+hUqlwsHBgXsaWa1W/5y90r/h69evbG1t0Ww23V6x\nV9PqSovRdZ2trS3K5TKFQoG9vT3PppVwWWj+7jvcQqEQ4XCYYDBIMBikXq//37t33e5wu9JivMC/\n9e8X8cV0wRfTBV9MF3wxXbh0Vfov40dMF3wxXfDFdMEX0wVfTBf+B3Rd+iPQLjWMAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5461684518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  4 Predicted:  9\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACQFJREFUeJztm1tXIlcWgL8CiluBKAg20toiCohGQ2PSMUsfkpfOf81L\n/kDyljQmmI6daLdyvwg2KFe5C8xDxlqdpOmZzIjiDN8La8FaVNVX++yzzz5VwmAwYMJfUdz3CYwr\nEzFDmIgZwkTMECZihjARMwTVh34UBOF/fi4fDAbC+76fRMwQPhgx44okSRiNRtrtNvV6nevra/r9\n/q0e40FGjN1u59mzZ3i9XqamphBFEUF474j4j3lQEaNSqRBFkYWFBQKBAMlkkqurKwDK5TLX19e3\nd6xb+6c7wGAwYDabcblcbGxsIEkSlUqFwWAgD6nb4kGIUSgUKBQKbDYbHo8Hj8fD0tIS9Xodg8GA\nWq3+/xxKarUanU6Hz+fjq6++YmVlBUmSqFarnJ6ecnZ2RqfTudVjPggxJpMJu93ORx99xM7ODkql\nkkqlQjabJZ1OUywW6fV6t3rMsRYjCAKCIOByudjd3cXv92O1Wnnz5g3BYJDDw0NqtRq9Xo/b7iuN\ntRidToder8fj8bC3t8fjx48ZDAakUim+//57Tk5OaDQat17DwJiLcTgceL1e/H4/Xq+XUqnE/v4+\nBwcHhMNhCoXCrc5E7zKWBZ5KpUKn0+F0OtnZ2WFtbQ2LxUKpVCIYDPLq1SvS6TSVSmUk0QJjGjFm\nsxmbzcbW1ha7u7totVri8TiHh4cEg0Hi8Tjdbnek5zBWEaNUKlGr1Tx69Ij19XU2NzfZ2tpCo9EQ\nDoc5Ojri119/JZvNjmwI3TBWETM1NcXMzAx+v5+9vT3cbjcqlYp8Ps8vv/xCKpWi3W6PbPi8y9iI\nEQQBs9mM0+mUh5AkSTSbTTKZjJxXOp3OrU/N72MsxAiCgEqlYnFxkZ2dHTweD7Ozs5ydnZFMJnn5\n8iWRSITLy8s7iRYYEzFarRa9Xs/y8jLPnj1jaWkJnU7H+fk5P/zwA4eHhyQSiVsv+z/EvYoRBAFR\nFPF6vWxsbMjRcnV1xcHBAfv7+7x48YJEInHrJf+/4l5nJVEUkSQJn8/H8+fP2d7exul00mg0CIVC\n/PTTT4RCITKZzJ2LubeIUSqVPHnyhJWVFba3t9nc3MRoNFIqlYhGo/z4448kk0m63e6d5ZV3ubeI\nEUWRJ0+e8Nlnn+H3+/H5fHJeOT09JRQKkU6n6Xa7dzIL/Zk7jxhBEORO3MbGBru7u1itVorFIvv7\n+wSDQUKhEJeXlzSbzXuRAvcgRqlUMjU1hcPhYH19nc8//5xyuUyhUCAUCvH1119TKpXkluW7nbm7\nlHRnYhQKBSaTidnZWZ4+fUogEMDr9dLv90mlUvz222+02218Ph/lcplKpUK9XqfRaMj/Ua/Xqdfr\nI+m//Jk7E6NUKpmZmcHpdLK3t8fz588xmUz0ej2SySTBYBC9Xs/6+jqVSoVqtcrFxcUfirqLiwu5\n8h31LHUnYgRBQKPR4HQ6+eSTT3C5XExNTXF+fk4ulyObzaJSqXC5XKytrdFut2m1WjQajT/kmVwu\nRzqdlheTo8xBdyJGoVCg1WplMcvLy5hMJl69esWLFy9oNBqIoojL5eKLL76Q88qfLzqVShGLxfjm\nm2+IRqO0Wq2HK0atVrOwsIDb7WZra4vV1VW0Wi0XFxeEw2FCoRDNZpNWq0Wr1SIajaLX65EkSW5t\nGo1GTCYTCoWCxcVFAoEAxWKRo6MjwuHwSCJn5GJEUcTpdPLpp5/KYkqlErlcjkgkQigUkqWEw2G+\n/fZbrFYrVqsVi8WCxWLh8ePHLCwssLq6itvtJhAIoFAoUCqVnJ2djSRyRibmJq9YLBbcbjd+vx+b\nzUa32yUcDnNwcMDp6Sm1Wo3r62u63S61Wk3OL8ViEUmSkCSJk5MTzGYzgUCAVquFKIqsra2Rz+dJ\nJBLEYjHy+fytdvVGJkahUKDX65mdncXj8eD3+zEYDLRaLd68ecN3331HOBym0WjId/tmSNVqNXnr\n5KYloVKpKJfLqNVqPv74Y9bX1ymVSsRiMdrtNqVS6WGI0Wg0LC8v8/TpU5xOJwaDgWKxSDab5ejo\niJOTEy4uLv4yBAaDwV++02g0GI1GpqensVgs6PV6BoMBrVaLSqUyki2UkYpZXV2V+ysajUZuUR4f\nHxOJRP7ti5Ekibm5Oex2O3a7HUmS6Ha7VKtVCoUC1Wp1/MXcTM1Wq5XV1VU2NjawWCx0u11SqRSH\nh4fkcrn3NrNvEqrVasVms2E2mzGbzbIQh8NBp9Ph6OhIXlul02mq1er4b9EqlUqMRiNzc3OsrKzg\n8/nkveYbMfl8/v0n88/nX5aWltjc3MTj8eB2u7Hb7czNzfH27VvS6bTcxIrH42QyGVqt1vhHzE3p\nPz8/z8zMDKIoks/nicVixONxstksV1dX8irbYDAgSRIGgwGHw4HD4cDlcrG8vIxOp0Oj0ZDJZDg5\nOSGVSpFIJIhGo4TDYUql0kikwAjEqFQqLBYLDocDo9HI9fW13OWPxWIUCgX6/T6CIDA9Pc38/Dx2\nu535+XkCgQDb29uYTCamp6dJJpMkEgkODg7kTt75+bk8e/V6vYezE9nr9SiXy7x9+5ZGo4FKpZKf\ngvryyy+Zn58Hfs8nNwWcyWTCZDJhNBopl8tks1mazSbxeJxYLEY4HCYcDlMul0eST97HrYvpdrvk\ncjn5IkVRZHFxEZvNxubmJs1mE/i9AFSr1ajVavr9Pr1ej9evX/Py5Uui0SiRSIRcLkcul/tDhNxV\n73ckEdNoNCgUCrx+/ZpQKMSjR4+wWq3o9XoEQZCr21qtxuXlJZeXl1xcXHB8fMzx8THZbJZsNkul\nUpEbVnfNrYsZDAZ0Oh2KxSI///wzWq2WnZ0djEYjWq0WURTlSjUSiRCJROShks1m5cfGOp3OnTSk\nhjGSAq/f79NsNuUirlQqcXp6ilqtRqVScXV1Ra1WI5PJcHZ2xtnZmRwh1Wp1FKf0txE+dEf+m3cJ\nBEFAq9XKu4w6nU5e+9zkik6nQ7vdlj/vMofcMOxdgpGJeShMXrL4m0zEDGEiZggTMUOYiBnCRMwQ\nJmKGMBEzhImYIUzEDGEiZggfXCv9PzOJmCFMxAxhImYIEzFDmIgZwj8A+CCTX5voLbUAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5461632c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  6 Predicted:  6\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACOpJREFUeJztm1tvGtcaQNcMYO4wJQbG4GLAcewU27GqNm2jpFKlPvSt\nj/2b/QWt1KgXNW1wm1ZgY8Bg7OCAuc9wGS7noWKatME5R/JQu4cl+cGwZ9iz9M3ee779jTCZTFjw\nd8R/ugPXlYWYGSzEzGAhZgYLMTNYiJmB+bIvBUH418/lk8lEeN3ni4iZwULMDBZiZrAQM4OFmBks\nxMxgIWYGl65jrgJRFFlaWsJqtWKxWLBYLNjtdqxW68xjer0e/X6fwWCApmn0+336/T7zTJEYJkYQ\nBF1KKBRClmX8fj/BYJBYLEYkEgF47cUWCgUKhQKVSoVKpUKpVOLk5IThcGhUd/+GIWJEUcRqteLz\n+QgEAsRiMaLRKKurq7z99ttsb2+zubkJ/CnmZUHpdJp0Ok2pVKJUKmG321FVlXa7TbfbnUvkXLkY\nURSx2WzIssxnn33Gw4cP8Xg8eDwenE4nbrebW7duvXJxo9HolWhYXl5md3eXeDyOoigEg0HsdjuH\nh4dks1kGg8FVd/tvXLkYQRCwWCz4fD4++ugjvvjiC13CaDRiNBoxGAy4uLjQj1FVlW63q/9vt9uR\nJIlbt25hMpnQNI3xeMxoNOLs7Ew/j5EYPvi+LEVRFFqtFoVCgVKpBMB4PObs7IyzszP9mFAoRDgc\n1iPNYrHw8ccf02w2SafTjMdjFEVhPB4b1m/DxYzHYwaDAaqqUi6XOT095dmzZ6TTaf37bDZLNpvV\nj1lfX+f27dvIsowsy7z77rvs7e2RSqWQJIlGo4Gqqob22xAxk8mE8XjMcDhEVVWKxSL5fJ6nT5/y\n7NkzqtUqtVpNb9toNGg2m/rxuVyOWq1GNBql3W4TiUQMjY7XceViJpMJk8kETdNotVqUy2XS6TTJ\nZJKvv/6aH3744Y3nUFWV58+fM5lMcDgcdDqdua5hwCAx/X6fSqXCt99+S7PZ5PT0lGKxyIsXL/6r\nc0wXhKFQiM3NTQKBAKI430W6IWIGgwG1Wo0nT56Qz+dRFIVOp0Oj0XjtMYIgIAgCZrMZk8mE1+vF\n6/USi8XY2trC7/cjCIJ+/nlg2OA7Go2o1Wp0u100TUPTNHq93ittRFHEZDJhs9mw2+2EQiFWVlaI\nx+PE43HW19dZX1/nrbfeYjgcomkao9FIv12NxFAxzWbzlUF1iiiK+kLQZrPh9XqRJIlEIkEikWBv\nb4979+7hcDiw2Wx0u106nQ6qqjIcDg1fw8AcpuuXEUURs9mMJEn4fD7i8TixWAxZlllZWWF5eVl/\nnnI6nQyHQxqNBoeHh6RSKX766Seq1Sq9Xu/mRsxfma6I7Xa7frt88MEH3L9/n2g0ytramt52NBox\nHo9pt9tcXFywv7/PV199RTqd5uLiAk3TDO/v3MRYLBbi8TgbGxvs7u6ys7NDOBwmFAohSdIrbU9P\nTzk5OSGdTpNKpTg6OiKTyVCtVudyG8GcxUSjUR48eMCjR4/48MMPZ7Ytl8skk0keP37M48eP5/pU\nPWWuY4zZbGZpaQmTyXRpu0AgQCKR0GUcHx9TLBZRFOXmph0uw2QyYTabEUXx0ovz+/04nU4EQcBm\ns+FwOHQh8xh4YY5iNE0jk8kgiiLn5+ekUingj0E5EAgQCAT06Xu68l1dXcVmsxEMBnnnnXfY39/n\n559/plqt0mq1bvbT9RRN0zg6OuLs7IxMJsPKygrwxxS+vb1NIpFAkiQkSUKWZRwOB6urq8TjcRKJ\nBIqi4PP5qNVqegrjXyFmMpkwHA7pdrucn5/raQNBEGi32+RyOX0FHAwGkWWZSCRCJBLB6/Xidru5\ne/cun3/+Od988w2KotBoNNA0zZBba65jzDTz1u/3qVar+uf5fF5/XgL0PMx7773H/fv32draQpIk\ntra2iEaj9Ho9fv31V7rdLsPh8OaLmfKmZ516vc5wOEQQBCqVCrlcjnw+ryfVw+Ewu7u7iKJINpul\n3+9feR//ETFvQlEUVFWlUqnwyy+/kMlkyOVyfPrpp0SjUVZWVrh37x6dToeTk5P/HzHTiJpmAadJ\nq42NDVqtFoIg4PF4sNvthuVprqWYv1KtVqnX6xSLRZrNJqPRCKfTic1m08elq+ZGiBmPx3r0TKfo\nlwdrI1hs6s/gRkRMIBBAlmVisRiSJDEejw2NFrghYsLhMO+//z5bW1v4fD5UVTU8OX6txbjdbjwe\nD4lEgocPH3Lnzh0cDscbn86vgmstRpIkotEoe3t7PHr0CJ/Ph9VqnctWyrUSMy0s8vv9+P1+Njc3\n2dzcZHd3l+XlZQRBoNPpUCqV+P333ymVSoalOa+VGKvVisvl4u7du+zt7bGzs8POzg5+vx+r1Uqn\n06Fer3N0dMT3339PLpczrCTEUDHTPaO1tTXC4TA2mw2z2Uyz2aTVajEYDBgMBlgsFr3yKhwOc+fO\nHTY2NlhbWyMSiSAIAoqicHBwwG+//caTJ084Pj6mXq8blgM2tNRsulXy4MEDPvnkE3w+Hy6Xi6Oj\nI3K5HI1Gg1arhdvtxu12k0gk2N7exuPx4Ha7sVqt2Gw22u02zWaTp0+f8uWXX5LL5ahUKvT7fcNy\nMoZGzLQGLxgMcvv2bfx+P5Ik4fV6kWWZTqeDoig4HA4cDgfRaJRoNIrZbMZsNlOpVMjn8xSLRQqF\nAt999x2pVIp6vW54itNQMS8XKNrtdux2O06nk7W1NWRZ1vMzJpNJv+2Wlpb0xVuhUODHH38kmUyS\nTCZ58eKFvq90ozfcRqMRvV6PcrlMNptlMpnopawulwtALxmZlo2cnJzQbDZpNBrs7++TTCY5ODgg\nk8kwGAxu/r7StOqh1WqRSqX0LVeXy4UkSSwtLelt+/0+7XabTCbD4eEhBwcHHBwc8Pz5c8rlMp1O\nx9Dx5HUYGjHT7Y5isYggCPR6Pc7Pz/F6vXrEwJ+JqePjY/L5vP6nKAqKosy9aAhAuOxHr+INN0EQ\n9PHF7Xbjcrn0wXXKtJy12+2iqqpexTmPyoZZb7gZLua6s3j1739kIWYGCzEzWIiZwULMDC6dlf6f\nWUTMDBZiZrAQM4OFmBksxMzgP1iUcbxYM5a5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54615b27f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  3 Predicted:  3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACdpJREFUeJztm+lT29Yahx9ZlhdsI9ssBoONARPANF0YQsjSzDSdTvul\n/26/tJMvmWk6LW0SnBQMoQHM5hUjL7K83g9cqVmushVD26vfjMdjYUnnPLznXc4rC91uF0uvy3bZ\nA/i7ygJjIguMiSwwJrLAmMgCYyL7m/4oCMK/PpZ3u13hfx23LMZEFhgTWWBMZIExkQXGRBYYE1lg\nTGSBMZEFxkQWGBNZYExkgTGRBcZEb6yueyVBEBAEAZvNhiD8Wdy22206nc5bz7fZbIiiaHy22+3Y\n7fbXjne7XdrtNq1Wi2az+c7Xh0sC43A4cDqduN1u3G43nU6HTqeDoigoivLW8z0eD7IsI4oigiAQ\nDAYJhUL09/fj9Xqx28+mpaoqiqKQyWRIp9OUSiVUVaXdbr/1HhcKRhRF7HY7/f39yLLMwMAAgUCA\nZrNJo9Fgd3f3NTCCICCKIjabDUmSkCSJUChEOBzG4XAgCAJjY2NMTU0RCoUIBoNIkgRAuVwml8ux\ntbWFJEns7u6SzWZRVfWtY70wMDabDb/fz8DAANFolGg0yszMDFNTUxSLRQqFAvfu3WNvb884x+Fw\n4HA48Pl8yLLM+Pg4ExMTxGIxYrGYAcbr9eL3++nr68PtdmOznbnORqNBrVZjYmKCaDTKr7/+yv37\n9/8eYGw2Gw6Hg76+PmNis7OzzM7OsrCwwJUrVzg4OCCdTpNKpQx/Ybfb8Xq9eDweQqEQIyMjfPzx\nx3z22WfE43EmJydxOp3AmS/Rl2O73TZeejPR7/fT399PpVLh4cOH7zTunoERBAFJkvD5fFy9epVE\nIsHExASRSASPx4PT6WRzc5P79++Ty+XI5XKk02nGxsaIxWJMTk4yMDBAMBhkcHCQgYEBQqEQoVAI\nv99v+BGAarXKyckJpVKJk5MTstksR0dH1Ot1ACqVCvl8nmfPnlEsFt9p/D0BY7PZsNvt+Hw+RkdH\nWVlZ4csvvyQWizE6OoqiKOTzeVZXV/nuu+8ol8tUKhUCgQDT09MsLy+zvLzMyMgIIyMjyLKMLMtG\nlNHfG40G7XabbDbL/v6+8dra2mJ9fZ1yuQyApmnUajWq1SrVavVywNjtdiNKXLlyhfn5eZaWlpie\nnkaWZSRJwu1209/fz8zMDLdu3cLlcuFyufB6vfh8PsbHx4lEIsZScrlcCIJANpvl8PAQTdNotVoc\nHR2xu7tLLpejWCwaVlMsFsnn8zQaDeAsDWg2m9TrdZrN5uWAkSSJ4eFh5ubm+Pzzz7l+/TqRSIRQ\nKGT4AUmS8Hq9TE1NIUmSYRk6ILvdbuQjuv9QVZWDgwPW1tYol8tomsaTJ0/4+eefKRQKqKpKs9mk\n2WxyHg8qCG+6yPu2T0RRZHBwkLt373L79m0SiQQzMzP09/fjdDo5ODggm80azlVP7vT/tn68UCgY\nk63VasZEdd+haZqxhA4PDw0o75PA6TJrn5ybxeiZrNfrJZFIcPv2bUZHRxkcHATOkq39/X2ePn2K\n1+tFlmVisRjhcJh0Ov1StEilUqRSKQOYPlk9g72IR1fOfSm9GDpfnIDuE5LJJKqqomkasizj8/kM\np6mrUCiQz+dRVZVGo2Fc59Vr9lI9A6ObtT6RVqvF8fExyWSS3d1dDg4OLnSi76tzA6OH0Gq1ysbG\nBkNDQywuLhoO1eVysby8jNPp5NGjRySTSfL5PIVCgUajYTjOVqt1XkP6SzpXi+l0OlQqFdbX15Ek\nCb/fz9jYmOF7rl27xkcffcTo6Cgej4fNzU22trZeyi/+lWAAms0mmUyGJ0+e0Ol02NnZYWFhgdnZ\nWYLBIF6vl3g8jiRJfPLJJ+TzeUqlEqVSiUqlQrlcJpPJkMlkjOh0GcvtXMP1C+chiqJR53z99dd8\n8cUXzMzMEIvFEEURURQNf6SXBDqgtbU11tbWSKVSbG5uvtM2wYeq5+H6lZvRbrep1Wrkcjl++eUX\nTk9Pjco4EokwPj5u1EJ+vx9JkggEAtTrdbxeL5FIhHg8zvT0NM+fP+ePP/4wSoCLUE8s5pVrIIoi\nTqeT0dFRxsfHWV5e5tq1a8zPzzM7O/vS97vdLvV6HVVV2dzcZGNjg3v37vH9998bGe956kIt5pUb\n02q1jCXTaDTodDpGTjM5OfmaBblcLiRJIhqN4nK5OD4+Znt7m3Q6TTabvRCrubCNKn3rslKpkMlk\nePjwIcPDw4yMjHDz5k1u3LjB3NwcwWDQKA3C4TChUIjd3V3W19dRVZVCofDvAgN/+h59ywDOSgVR\nFCkUCjx9+pRYLMbY2BjhcJhgMEgwGCQej/PVV18hCAKHh4coivLOVfKH6lI2w+EMUqVSoVarUSqV\nSCaTDA0NMTw8zMrKCisrK8zNzeH3+4nH44TDYQqFAj/99BP1ep1Wq9XTMH5pYODl6KVPtlwuG9W4\nz+djZmYGp9OJJEn09fUZnYFe61LBvKhOp0O1WkVVVba3twGIx+N0Op2XekZ6T0oQhH+GxQiCgMPh\nwGazfXDNI0kSLpeLaDTK4uKiUU6oqkq1WkVRFCOX6XU2fG5gbDYbLpcLh8NBpVL5IDB9fX0EAgE+\n/fRTvv32W8LhMKIoUiwWyWQyFItFY8n1WucGxufzsbi4SDQaJZPJkM1mjb1YTdNeiiJ6ONYbaPoy\nSSQSJBIJlpaWmJiYwO1202q1ODw85NGjR+zs7FCv1/9ZFiPLMnfu3OHOnTtsb2+zubnJ48ePWV9f\np1QqGWD0Jed2u/F4PHg8HiRJwul0cuvWLb755hvC4TB+v59Op0Oj0WBnZ4cHDx6wtbX1zi3Wv6pz\nA9NoNMhkMhwcHNDX18fCwgKyLDM1NcX+/j5HR0ecnp6iqirRaJRIJEIwGCQQCCCKIpIksbS0RDwe\nx+VyAZBOp3n+/Dmrq6skk0mOjo56nr/oOjcwiqLw448/cnp6yo0bN7h69Srz8/N0u102NjZIpVJs\nb29zfHzMzZs3uX79OiMjI4RCISPK6O2TVquFpmn8/vvv/PDDDzx+/JhUKkWtVruwIvLcwGiaxv7+\nvtEeUVWVoaEhPB6PsePf6XSMdq3X68Xlcr302EYmk+HZs2coisLp6SkPHjxgdXWV/f19FEV57w7A\nX9G5gWm32yiKgqZpnJyc8NtvvzE8PIwsy2iaRr1eR9M0BEFAURRyuRzVapVsNgucJXt7e3vs7e1x\ndHTE8fGx8bler18oFOjRtoMoijgcDuN5lRclSRLT09PEYjEkSTJ60N1u1wCidwnK5TLlcrmnEchs\n26Fn+zF6//rFpaIf1x8Y0ntR/x0gjUYDTdOMzfFWq9Vzn3LhYP4psn7I9Z6ywJjIAmMiC4yJLDAm\nssCYyAJjojfmMf/PsizGRBYYE1lgTGSBMZEFxkT/AS2dPH38A732AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5461519da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  5 Predicted:  5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAABwBJREFUeJztm1lTGtsWgL9umjCLgoATxAHFMRK1UokYk7wkfyA/M5WH\n5CkvyYtmMFUOoEGDWkZRcWSyZTDS5yEFJ56T9lbuBcUbvipeaKrWrq/WXr3WpltQFIUa/0a87gVU\nKzUxKtTEqFATo0JNjAo1MSpIl10UBOH//l6uKIrwq+9rGaNCTYwKNTEq1MSoUBOjwqV3pWqjrq6O\n+vp69Ho9t27dIpfLkclkSKfTnJyccH5+XrZYN0qMz+djYmICt9tNU1MTu7u7RCIRQqEQCwsLnJyc\nlC3WjREjCAJtbW0EAgH6+/vp7OxkbW0Nh8OBLMuEw+GyxrsRYjQaDZIkYTAYMJvN6HQ6AHQ6HTab\nDbPZjEajKWvMGyHGZDJhtVppbGykvr4eg8GAIPyyYS0bN0KMx+PB7/czPDxMS0sLFosFURQ5Ojpi\naWmJzc1N8vl8WWNWtRhBEBBFEbfbzfj4OP39/djt9tJWisfjrK6uEovFODs7K2vsqhaj0+kwGAx4\nPB6GhoZobW1FkiQURaFQKJDL5UilUpyenlIoFMoau6rFGAwG7HY7bW1tdHd3Y7Va0Wg0nJ2dlaTE\n43FkWf6zxNTX19PR0YHD4UCn05XuPKlUir29Pb59+0Y0GiUej/P9+/eyxq5KMYIgIAgCDocDn89H\nU1MTt27dKolJJpNsbGywtbVFLBYjnU6XfQ1VKaZYW3p6enj8+DFer/dCn5JIJFhbW2Nvb6/smVKk\nasVYLBa6urq4f/8+Vqu1VHThbzGxWKxiYqpyui52ucWPVqstXSsUCsTjcdbX19nf3//zMsZqtWIy\nmdDr9SUx+XyefD7P4eEhGxsbHBwc/FliPB4PDx48oL29HVH8O6k3NzdZXl4mFAoRj8fJ5XJU6r/3\nqhJT7HTVxESjUaampvjy5QuJRKLsY8DPVJUYh8OB0+mkr6+Pvr4+HA7HBTHZbLZiDd0/qRoxxb6l\nr6+P3t5eenp6kKQfy1MUBUVRyGQyHB8fI8tyxbZQkaoQI4oiWq2W5uZmBgcHcTqdF44VTk5OSKfT\nbG1tsb6+zuHhYVmPMX9FVYiRJAmdTkdzczMDAwO4XK4LYmRZJhaLsbm5yfr6OqlUqvJrqniE/4Ao\nity+fZvu7m5GRkbo7e2lsbHxgphIJMLbt28JBoMVLbg/UxVi2tvbefjwISMjI/h8vn+dzkUiEV6/\nfk00Gv0zxBgMBurq6nC73XR3d2O320sDJPxo/ZPJJLu7u8TjcTKZTMWLbpFrGwkEQcBoNF44b7HZ\nbBd+U5yiY7EYyWSSbDZ7ZWKuLWO0Wi2Dg4MEAgFGR0dxOByYTCYAcrkcuVyOUCjEu3fvCAaDZLPZ\nivcuP3NtYiRJYnh4mOfPn+N0OnE4HKUtlM1mSaVSzM/P8/LlS46Pj680W+AaxEiSRGtrKx0dHaVD\nKLPZfKHD3d7eJhwOs7q6SiKRuHIpcMViBEFAq9XS1dXF+Pg4Pp8Pm812QQr8mInev3/P2toamUym\nYhP0ZVyZGI1Gg91up7W1lbGxMQKBAB6PB41GgyAIKIrCwcEB+/v7LCwsMDs7y/b2dsU7XDWuTIwk\nSbS0tDA0NMS9e/eYmJhAq9WW6oqiKOzs7BAMBpmdnWVubu7KC+6F9VY6gCAIpb9BBgYGCAQCdHZ2\nYjQagR8ncoeHhxwcHDAzM8P09DQrKytXMkFfRsXFaDQaLBYLzc3N+P1+JicnaWxsLF1XFIXd3V0W\nFxeZnp7mzZs31y4FKihGFEX0ej0NDQ3cvXuXkZER7ty5g8vlQq/XoygK+Xye09NTVldX+fDhA1+/\nfiUej19bXfmZioqpq6vD4/Hw6NEjnj17hsPhwGw2l4ptNpslkUiwsrLC1NRURQ+3f5eyixFFsfTc\nit/vZ3R0FL/fj9vtRqfTIYpiKVOWlpYIhULMz8+zs7PD6elpuZfzX1MRMRaLBbfbzeTkJE+fPsXl\ncmEymUqZkslkODo64uPHj7x69YpoNEoymbz2uvIzZRej1+vxer2MjY0xMDCAx+PBYDAgimLpCYVw\nOMzc3BwzMzNEIhFkWa6aLVSk7GKMRiNDQ0M8efIEr9d7od0vFArIssynT5948eIF29vbVVNs/0nZ\nxBTb/eL5itfrxWazlR7byGQyxGIxNjY2WFxcLGVKuR/4KRdlEyOKIkajkYaGBpxOJy6XC6PRSKFQ\nKNWUYDDI58+fWV5eJplMVmWmFCnrVjo/PyedTrO4uIjRaESSJARBIJPJkEgkCIfDhMNhotFo1WZK\nEeGycf5331fSaDTo9XrsdjtWq7U0BxUKBc7OzpBlGVmWyWQyZLPZ/23lZULtfaWyirmJ1F7k+k1q\nYlSoiVGhJkaFmhgVamJUqIlR4dI+5k+mljEq1MSoUBOjQk2MCjUxKvwFFnoAtO+9sscAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54614fad30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  1 Predicted:  1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACzhJREFUeJztm9tTGmcfxz/LArucFhAERImNaVGbaE3HNE7SZDrT/zeX\nvelNp9NM02li67FJPdQTioqcyQILu+9Fuk81CWlFtL7vy/fGmWXd5fnsb3/HB8myLPp6V45/+wtc\nV/XBdFAfTAf1wXRQH0wH9cF0kPNDH0qS9D8fyy3Lkt53vG8xHdQH00F9MB3UB9NBfTAd1AfTQX0w\nHfTBPOY6SZIkFEXB4/HgcDhwOP56pk6nE5fLhSS9SUkajQavX7+m0WjQbDbpprVy7cFIkoQkSTgc\nDpLJJGNjY3i9XtxutwARDAaJRqO4XC4Adnd3efnyJZlMhqOjIxqNxrnv+6+DsRctyzKKouB2u8Vn\nTqcTWZZRVRVFUZicnOTOnTsEAgFUVRVgotEoyWRS/O/Lly/F5+Vy+b8PjCzLyLKMz+fD5/ORSqVI\npVJiwT6fj0AgQCKRIJlMEolEiEQiuN1uZFkW56mqis/nQ5ZlcV2Hw0Gr1WJ7e5tSqXTu73blYBwO\nBy6XC7fbjd/vx+/3Ew6HCYfDTExMMDExceYVCYVC3Lx5k7GxMZzOf/Z1nU4niqKQyWTwer1dfc8r\nBSPLMm63m+HhYVKpFOl0mnQ6TSgUIhQKMTAwwMDAgDhfURQURUHTtDPO9u+kKAqhUIhAICCs6Lzq\nGRj7KcMbq7B9hyRJWJaFJEnCOiYmJvj000+5f/8+9+7dQ9M0NE079z0tyxLXPn2s1Wqh6zqNRgPT\nNLtaz4XBSJIknCS8gaIoCqqq4vV6URQFXdeRJImZmRmmp6eJx+MkEgnGxsYIBoNnHO55ZJom7XYb\nSZKQZRnLsmi326ysrPDNN9/w7NkzcrlcV9e+EBhJknC5XASDQbxer4AUCAQIBALieLlcBuDLL7/k\n8ePHeDwevF6vMPe/k2maYtHtdlscr9fr6LqOw+HA7XbTbrdpNBosLCzw5MkT1tfXz5x/HnUNxnai\n8Xicubk50um08CHhcJhgMIiiKMiyTK1WwzAMUqmUyEGcTueZV+BDajabNBoN8vk8+XxeLPbw8JDd\n3V0APB4P9XqdcrnML7/8QqFQ6Po1gguAkWWZQCDAjRs3ePToEY8ePRI5RzQaJRAI0Gq1aDabVCoV\narUajUaDer1Oq9XCMAxcLheqqoprmqYpFmNnq5ZlUS6XKZVK7O7usrOzg2EYAGxubrK6uoplWXi9\nXmq1Gvl8nqOjI0qlUlcZ74XB+P1+bt++zRdffEE6nSaRSJxJ1Or1Ojs7O2xtbbG5ucnOzg7VahVd\n11EUBZ/Px+eff85nn30mIk6xWBRP2jRNGo0GjUaD/f19MpkMJycnZyymWCySy+WwLAuXy4VhGNTr\ndWq1Gq1Wq2soFwZjR5Zbt24Ri8WAN0/YNE1KpRKbm5s8e/aM+fl5lpeXKRaLVCoVVFUlFApRKpVw\nuVyi9tnb22Nvb0/4kmq1SrVaZXNzk42NDWF1VzFW7hqM1+vl1q1b3L59+0zuYVkWhmFQqVTY2tpi\nZWWFbDYrXqHTnz9//px8Pi98TaVSoVKpCIsxDINms0mxWOT169fi/69CXYPxeDyMjo7yySefvJNH\nGIZBrVbj4OCAra0tyuUylmUJ5wxvKuClpSUWFhaubLHnUddg6vU6u7u7bGxsEIvFCAaDwF/tgcHB\nQb7++mtGRkbI5XLk83lqtZpoB+i6zu7uLplMhlqthq7rPVtUL9Q1GHth6+vrKIoiwNg5xeDgIF99\n9RUPHz4km81yeHhIuVymUqlQrVYpFAr8/PPPGIbByckJhmGcsRw7q/23rKlrMLVajdXVVcLhMLFY\njBs3brz3PIfDgaZpyLJMJBIRzaN6vU48Hmd6elqA03Wder1OvV4/E41arVbXiVq36hpMtVpldXUV\nVVWZnZ3t+GRtMLZF2bIsi6mpKXRdF9HIDtelUolyucyLFy84OTlB13WR/V6VugbTarUol8tkMhmW\nlpYIh8NomnamzHc4HDidTjRNw+fzvXtzpxOPx0MsFkNVVWEttsWkUikmJiao1+s0m002NjZ49eoV\n1Wq165blP1VPwCwuLqKqKsPDw0SjUeCv4tLj8Yhm1Ds3dzpxOp2oqko8Hn9noZOTk8zOzgow3377\nLblcTmTO1xIMvHkdKpUKS0tLFAoFUTTCWTDJZJKhoSHRgfN6vXg8HgHG7/cTCATeqZ00TUOSJAzD\noN1uMzc3hyzLLC8v89tvv5HP5ymXyxeqiTrpwm2HarXK8vIyq6uronEtLv4nmKGhIVKpFOPj46TT\naaLRKAMDA6LrPzQ09N4qOxAI4Pf7gTcPQVVVRkZGGBgYoF6vY5omtVrteoKBs8XfadnRxM58S6US\nf/zxBz6fT/RqVFUlkUgQj8fF/8ViMWKxGNFolEgkgsvlQpZlYUGzs7OoqsrTp0/57rvvKBQKPc+K\nL7W1aReCzWaTQqHA9vb2mXGI2+1GVVUGBweFbwKYmppienqayclJfD6faETZQP1+P+l0mna7zeLi\nIrVajXa7ff3BnI5GqqpSLBapVqvv5CJ2PmM7clt2DrO1tcX29jajo6OMjo6KV8t26JFIhMHBQYrF\n4oXamO9Tz8HY1mBHmlAohGmaVKvVd861q2jDMCgWi+J4NptlYWGB9fV11tbWuH//Pg6Hg+HhYbxe\nr2il2oO2bDYrXqde6VLAuFwuQqEQMzMzfPzxx3z//fdi8GU3mU7r7SfdbrdpNpvs7++LZlc2m2Vu\nbo6HDx+iaRput5tYLMbdu3fRdZ3j4+Oe5jaXBiYcDjM7O8uDBw/I5XK8evWKcrn8XjCdlM/nKZVK\n7O/v8/z5c0zTJJ1O43K5cLlcJBIJ7t27Ry6XY3FxkUql0rPS4VJ8jJ3DaJrGyMgIDx48QJIk9vb2\n2N/fF4Wknfp3kv2q+f1+0Su2RzLwZiA3NjbG0NAQXq8XWZZ7VjpcWlSSJAm3283AwACPHz8mnU6L\nxCyTybC3tyd6NX8nTdNIpVIidNutULsEicfjImHsVdjuORi7tWkYhqiW/X4/H330EbIsE41GOTk5\n4fj4WEQdu73w9lDfVjKZZGRkhKmpKTRNE1s+7B6zDeufTh3+iS7FYmznWalUKBaLBINBgsEgHo+H\nmzdvijCdyWQ4ODgQCWKniaSmaYRCITH8Pz3QvyxdisW0Wi1KpRK//vormqYxPT3N+Pi4KAHsCaI9\narHB2Anc27IzZNsqTm8QOj0VuNZ5jN3zzefzPH36lOPjY0zTFDnNaQdqTyJtn/D2TilbnY7rui4i\nV7PZ7GnP5lJeJcuyaDabHB4eYpomP/74I06nkzt37jAxMSF8iT3NPK9sS/n9999ZWFjoeaiGS4xK\npmmi6zpHR0f88MMPZLNZDMMgFosJy+lWtqXMz8/z5MkT1tbWKBQKPa2XLr2IrNfrHBwc0Gw2iUQi\ntFotRkZGSCaTDA4Oih1S7wNlz6ztv06nE7fbLZrwy8vLrK2tcXh42POOnvShi/Xq1yd2SI1EIsRi\nMcbHx5mcnOTu3bvMzMx0jEalUolSqUSxWKRYLOL1etE0jcXFRX766SdevHjB/Pw81Wq1a8fb6dcn\nV7Kjys5gs9ks+Xye169fi50LmUyGcDhMKBQS20PsIX61WhXTyXK5jMfjwefzsbGxwcrKCtvb26Jh\n1WtdicWcll15ezwesY/G3mqWSCRIJBKsr6+zvr6OYRiiv9tqtZBlGafTSa1Wo1KpiATyIq9QJ4u5\ncjBvXR+HwyESN7thtb29fSYjhr+2hdhb13rlT64lmD/vIba12k5Y13V0Xb+SOdK1BfNvq//Tv3Oq\nD6aD+mA6qA+mg/pgOuiDUen/WX2L6aA+mA7qg+mgPpgO6oPpoP8AByoj9QoNN38AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54614cb8d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  5 Predicted:  5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEYAAABGCAYAAABxLuKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACphJREFUeJztm9lvWtf2xz/nMM+jHRswQzxESTylbpN0sKqoD3nsY//R\nPlSq+tCqamW1MTG1GmFsxwOjjQ0YOMyc+xCd/Wt/Kb4pxcT3Xr4SshjOPpsPa6+91l7LkqqqTPS2\n5Pc9gduqCZgBmoAZoAmYAZqAGaAJmAHSX/emJEn/9Xu5qqrSX70+sZgBmoAZoAmYAbrWx4xSkiRh\ntVqxWq0YjUaMRiM6nQ6dTofRaMRgMLx1Tb1ep1ar0W636Xa7NJtNms0m40hjxgbGYDAQDoeJxWJM\nT0/j9/sFKJ/Ph9frRZIkJElCVVVUVWVvb49Xr15xcXFBqVQim82STqfpdrs3Pt+xgdHr9czOzrK6\nukosFiMSiWCz2XA4HExPTzM9Pf0WmEQiwZ07dygUCpyfn3NwcIDVaqVSqaAoyo1a0NjAyLKM3+9n\nfn6ehYUFYrEYBoMBg8GAxWIBeOsLBoNBzGYziqKgKAq5XI50Os3e3h7JZJKTkxPS6TTtdnvk8x0b\nGIB+v0+v16PT6dBut9HpdMiyTKvVotVqYTabMZvN4vM+nw+fz0e/36ff71Mul7m8vCQUCuHxeHC7\n3ZjNZorFItVqlXa7Ta/XG8lcpevMcJQBntFoZH5+nlgsxtTUFNPT0ywtLTE/P6/di0gkQjgcfuta\nbWlpAIvFImdnZ2QyGdLpNC9fvmRra4uzszPq9frfgjMowBubxXQ6HV6/fk0ul8NiseB0Onn06BHF\nYhF4A6ZarV7rL6xWK3a7nXA4TDgcZmlpiVKphNlsJpPJ0Gg0aDabI7GasYFRVZVOpyP+NhoNXrx4\nwfHxMfAGTDweZ25ubuAYa2trfPTRR3i9XhwOBzabTTj1UCgkdq9R+Jyx+pher0ev16PVagFQLpdJ\npVLifc1vDFK1WhXbut1ux2KxYLFY8Pl8+P1+nE4nOp1uJHMdK5h/J0VRrl0GBwcH/Prrr+j1emZm\nZjAajTc2l1sFRnOug5TJZHj16hWRSOTGg7xJrjRA/zFgJEnCYDBgtVoxmUxI0l/usiPTrVpKg6TT\n6dDr9YRCITY2NohGo3+ZdI5StxqMLMvIsiySzWg0yvr6OnNzcxgMBvr9Pqqq0u126Xa79Hq9keVN\ntxqMw+HA4/GwuLjIvXv3ePr0KcFgELvdjizLNJtN6vU6hUKBdDrNxcXFyJzyrQSjWYrX6yUSifDk\nyRM2NzeJxWLcuXMHvV5Pv9+nVquJxDKTyVAqlf47wUiShCzLeDwePB4PDx8+ZHl5mbW1NWKxGB6P\nR1iKoigkEgl++uknXrx4QTabpV6v0+/3RzKXWwXGaDRiMpmYmZkhEomwvr7OkydPiMVihMNhZFmm\n3+9TrVYpFou8fPmSr7/+mqOjIyqVysgya7hFYCRJYmVlhY2NDSKRCJFIhEAgQDAYxO12I8uy8CWp\nVIpkMsnOzg75fJ5GozEyS9H03sFoy0ev17O6uspXX31FJBJhbm5OxCraeUwulyMej7O1tSWOGUZt\nKZreGxhJkvB6vXi9XsLhMNFolM3NTaLRKC6XC0mSUBSFer1OOp3m9PSURCJBPB7n+PiYYrF4I5ai\n6b2B0el0+Hw+FhYW+PTTT/nkk0+IRCJEo1FxMKUoCsVikZ2dHX7++Wd2dnb47bffbuQo8/9rrGD0\nej1Wq5UHDx5w//594Uu0kzuXy4WqquRyObLZLPl8nmw2SzweJ5FIkMvlbmTZ/OVcx3IX/i/XcTqd\nPH36lC+//JJQKEQgEBD1Jc2naL7k9PSU09NTdnd3x2Ypmm4MjCRJGI1GzGYzbrcbr9dLMBgUAZtm\nITqdTviSXC5HPp8nHo8Tj8cpl8tUKhUKhcJYakl/1I2CMZlMuN1uotEoCwsLrKyssLy8TDQaFbuO\nqqoigo3H42xvb7Ozs8POzg7dbhdVVUVONE6NDIwsyxiNRtxuN4uLi0QiERwOB263m9nZWfGYmZnB\n6XTSbrd5/fo1h4eHnJyccHx8zNHREUdHR+RyOdrt9o3tOO+ikYKxWq3Mzs7y7NkzNjc3cblcOJ1O\n3G43drtd+JFer0ej0WB3d5fvvvuOVCrF/v4+tVpNhPXvEwqMAIxmKT6fj7W1NR49esSHH37I3bt3\nRQHNbDaj1+upVCqUSiWOjo44ODhge3ubeDzO+fk5pVKJTqczlC8xmUxYLBZMJhMGg0E0AGhHpcNA\nHgkYi8VCIBDgs88+44svviAYDOL3+0UtWpIkOp0Ol5eX7O/v8/333/Pjjz+SzWbFFvxP/IjZbMbn\n84mSSrPZFI672+2OF4wsyxgMBvx+P6urq2xsbLC+vk4oFMLhcKCqKqVSiVKpRKFQIJ/Pk06nOTk5\nYXd3l6OjI1FWHebesiyLpoClpSWWl5fx+/3Y7XYajQalUonff/+dra0tLi4u/vY9/hEYi8XC7Ows\nn3/+Oc+ePSMYDOLz+UQ9ulAosLe3x/b2Nru7u6JroVKpcHV1NbQf+ePyDYVCfPzxxzx//pxAIIDD\n4RBgvvnmG/b29sYLxuVy8cEHH/D48WNx3Gi320WOc35+LmrKqVRKWIjWCHRdBKsFfFo3hCafz8fU\n1BR2ux2bzUYgECAcDvPw4UMikQgejweTyYTNZsNqteL3+4c+Gx4ajMfjYXNzk+fPnxMMBvF6vciy\nTLfbpVqtkslk+OWXX/j2228pl8vCQt7lXFav12MymbBardhsNvH6gwcPWFtbw+/343a7CQaDovPB\n6XSi1+tFYGkwGLDZbENXJocGYzQa8fv9wnz1+jdDybIsot1YLMbq6ir1ep1Go/HOYzscDux2uzjJ\n01IFrahvMplEoqnT6UQnhOZwa7UaiqKQTCZRFGWo7zc0GJ1OJ37RP5qrTqfD4XAQCoV4/PgxXq9X\ndCG8iyRJEkBmZmaYnZ1Flt+Uv/b390kmk9Trdcrlssi9+v0+7Xabs7MzDg8PyWQyFAoFdnd3ubq6\nGur7DQ2mWq2SSCREZDs9Pf2nQM5qtRIOh7HZbLTbbTqdzr8dU7MMm82G3W7H5XLhcrmAN4dVmUyG\nbrdLo9GgXq+TSqXIZDJiuVxeXpLP57m8vKRUKpHP58dvMZeXl/zwww+cn5+ztrbGysoKi4uL2Gw2\nJElCr9czNTWFx+MRZv8u0k70tIeqqvR6PbrdLu12m3a7LcAcHh6SzWapVCpUq1VarZYIErvdrujc\nGkZDg2k2m5ycnNBqtbi6uuLs7Ixiscjdu3dF5VDLrj0ej/jltaSw0+lQLBZF45AGRZOqqjQaDRqN\nBpVKhXK5zPHxMYeHh9RqNRqNBufn5xQKBarVKoqijDSNGBpMp9OhVCpRq9XIZDIkk0kymQz37t3D\nYDBgMplwuVz4/X7u378vwABiOSSTSRKJxJ/G1TJueGOVFxcXIsHUOjW1SLnT6dDpdMTzUWpoMFpn\nVKfTQVEUWq0WBoOBYrEorMVms+F2u0mlUgQCAXGdlsskk0n29/f/NO4fwVxdXXF1dUWhUCCXy401\nsRxZc6IWCWudCLIsiyVlNpsxmUzis1r2rCjKtdu4Vo9ut9vX9s38Ew1qThxb1+Zt1eTfcv6mJmAG\naAJmgCZgBmgCZoAmYAZoAmaAJmAGaAJmgK6NfP+XNbGYAZqAGaAJmAGagBmgCZgB+hfqTWbd85v6\nmgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54614cb7b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  2 Predicted:  2\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(2)\n",
    "small_test = np.random.choice(X_test.shape[0], 10)\n",
    "\n",
    "for i in small_test:\n",
    "    view_image(X_test[i], label=y_test[i], predicted=y_pred[i], size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Neural Net\n",
    "\n",
    "Add a hidden layer with a sigmoid activation function\n",
    "\n",
    "![a](d3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sgd(cost, params, lr=0.05):\n",
    "    grads = T.grad(cost=cost, wrt=params)\n",
    "    updates = []\n",
    "    for p, g in zip(params, grads):\n",
    "        updates.append([p, p - g * lr])\n",
    "    return updates\n",
    "\n",
    "def model(X, w_h, w_o):\n",
    "    h = T.nnet.sigmoid(T.dot(X, w_h))\n",
    "    pyx = T.nnet.softmax(T.dot(h, w_o))\n",
    "    return pyx\n",
    "\n",
    "w_h = init_weights((784, 625))\n",
    "w_o = init_weights((625, 10))\n",
    "\n",
    "py_x = model(X, w_h, w_o)\n",
    "y_x = T.argmax(py_x, axis=1)\n",
    "\n",
    "cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))\n",
    "params = [w_h, w_o]\n",
    "updates = sgd(cost, params)\n",
    "\n",
    "train = theano.function(inputs=[X, Y], outputs=cost, updates=updates, allow_input_downcast=True)\n",
    "predict = theano.function(inputs=[X], outputs=y_x, allow_input_downcast=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 (0.30408333333333332, 0.29780000000000001)\n",
      "1 (0.17413333333333333, 0.1714)\n",
      "2 (0.13850000000000001, 0.1328)\n",
      "3 (0.12273333333333333, 0.1168)\n",
      "4 (0.11371666666666666, 0.1101)\n",
      "5 (0.1079, 0.1051)\n",
      "6 (0.10331666666666667, 0.10150000000000001)\n",
      "7 (0.10036666666666667, 0.098000000000000004)\n",
      "8 (0.097699999999999995, 0.095100000000000004)\n",
      "9 (0.095750000000000002, 0.092799999999999994)\n",
      "10 (0.093700000000000006, 0.0906)\n",
      "11 (0.092133333333333331, 0.088300000000000003)\n",
      "12 (0.090683333333333338, 0.0872)\n",
      "13 (0.089316666666666669, 0.085699999999999998)\n",
      "14 (0.088016666666666674, 0.085000000000000006)\n",
      "15 (0.086966666666666664, 0.084099999999999994)\n",
      "16 (0.086099999999999996, 0.083500000000000005)\n",
      "17 (0.085316666666666666, 0.082900000000000001)\n",
      "18 (0.084400000000000003, 0.082000000000000003)\n",
      "19 (0.083783333333333335, 0.081699999999999995)\n",
      "20 (0.082799999999999999, 0.081199999999999994)\n",
      "21 (0.082166666666666666, 0.080600000000000005)\n",
      "22 (0.081416666666666665, 0.080000000000000002)\n",
      "23 (0.080766666666666667, 0.079200000000000007)\n",
      "24 (0.079916666666666664, 0.078700000000000006)\n",
      "25 (0.079200000000000007, 0.078600000000000003)\n",
      "26 (0.078750000000000001, 0.078200000000000006)\n",
      "27 (0.07825, 0.077899999999999997)\n",
      "28 (0.077799999999999994, 0.077299999999999994)\n",
      "29 (0.07721666666666667, 0.076499999999999999)\n",
      "30 (0.076666666666666661, 0.075899999999999995)\n",
      "31 (0.076050000000000006, 0.075999999999999998)\n",
      "32 (0.075266666666666662, 0.0751)\n",
      "33 (0.07456666666666667, 0.074399999999999994)\n",
      "34 (0.073666666666666672, 0.073899999999999993)\n",
      "35 (0.072999999999999995, 0.073300000000000004)\n",
      "36 (0.072383333333333327, 0.073200000000000001)\n",
      "37 (0.071633333333333327, 0.072700000000000001)\n",
      "38 (0.070866666666666661, 0.072599999999999998)\n",
      "39 (0.070316666666666666, 0.072099999999999997)\n",
      "40 (0.069449999999999998, 0.071099999999999997)\n",
      "41 (0.06876666666666667, 0.070499999999999993)\n",
      "42 (0.068166666666666667, 0.069699999999999998)\n",
      "43 (0.067549999999999999, 0.069199999999999998)\n",
      "44 (0.066733333333333339, 0.068900000000000003)\n",
      "45 (0.066133333333333336, 0.0688)\n",
      "46 (0.065299999999999997, 0.068199999999999997)\n",
      "47 (0.064466666666666672, 0.066799999999999998)\n",
      "48 (0.063783333333333331, 0.066199999999999995)\n",
      "49 (0.063233333333333336, 0.066100000000000006)\n",
      "50 (0.062449999999999999, 0.0654)\n",
      "51 (0.061866666666666667, 0.064500000000000002)\n",
      "52 (0.061366666666666667, 0.063799999999999996)\n",
      "53 (0.060633333333333331, 0.062700000000000006)\n",
      "54 (0.06001666666666667, 0.061899999999999997)\n",
      "55 (0.059366666666666665, 0.061100000000000002)\n",
      "56 (0.058716666666666667, 0.0608)\n",
      "57 (0.058083333333333334, 0.060299999999999999)\n",
      "58 (0.057283333333333332, 0.059900000000000002)\n",
      "59 (0.056583333333333333, 0.059299999999999999)\n",
      "60 (0.055933333333333335, 0.058799999999999998)\n",
      "61 (0.055316666666666667, 0.058099999999999999)\n",
      "62 (0.054649999999999997, 0.057700000000000001)\n",
      "63 (0.054100000000000002, 0.056899999999999999)\n",
      "64 (0.05358333333333333, 0.056300000000000003)\n",
      "65 (0.053066666666666665, 0.055800000000000002)\n",
      "66 (0.052650000000000002, 0.055199999999999999)\n",
      "67 (0.052033333333333334, 0.054899999999999997)\n",
      "68 (0.051650000000000001, 0.054600000000000003)\n",
      "69 (0.051283333333333334, 0.054100000000000002)\n",
      "70 (0.050766666666666668, 0.053699999999999998)\n",
      "71 (0.050316666666666669, 0.053100000000000001)\n",
      "72 (0.049799999999999997, 0.052699999999999997)\n",
      "73 (0.049299999999999997, 0.052600000000000001)\n",
      "74 (0.048916666666666664, 0.051499999999999997)\n",
      "75 (0.048550000000000003, 0.051499999999999997)\n",
      "76 (0.047966666666666664, 0.051299999999999998)\n",
      "77 (0.047600000000000003, 0.050700000000000002)\n",
      "78 (0.047133333333333333, 0.0504)\n",
      "79 (0.046716666666666663, 0.050200000000000002)\n",
      "80 (0.046183333333333333, 0.049399999999999999)\n",
      "81 (0.0458, 0.049200000000000001)\n",
      "82 (0.045350000000000001, 0.048899999999999999)\n",
      "83 (0.044883333333333331, 0.048599999999999997)\n",
      "84 (0.044416666666666667, 0.048000000000000001)\n",
      "85 (0.043900000000000002, 0.046800000000000001)\n",
      "86 (0.043483333333333332, 0.046800000000000001)\n",
      "87 (0.042900000000000001, 0.046600000000000003)\n",
      "88 (0.042516666666666668, 0.046199999999999998)\n",
      "89 (0.042133333333333335, 0.0458)\n",
      "90 (0.041633333333333335, 0.045499999999999999)\n",
      "91 (0.04123333333333333, 0.0453)\n",
      "92 (0.040733333333333337, 0.044999999999999998)\n",
      "93 (0.040349999999999997, 0.044400000000000002)\n",
      "94 (0.040083333333333332, 0.044200000000000003)\n",
      "95 (0.039866666666666668, 0.0441)\n",
      "96 (0.039449999999999999, 0.043799999999999999)\n",
      "97 (0.039166666666666669, 0.043499999999999997)\n",
      "98 (0.038883333333333332, 0.043299999999999998)\n",
      "99 (0.038516666666666664, 0.042799999999999998)\n",
      "Total time:  7.066666666666666\n"
     ]
    }
   ],
   "source": [
    "t0 = dt.datetime.now()\n",
    "\n",
    "errors = []\n",
    "for i in range(100):\n",
    "    \n",
    "    for start, end in zip(range(0, X_train.shape[0], 128), \n",
    "                          range(128, X_train.shape[0], 128)):\n",
    "        cost = train(X_train[start:end], y_train2[start:end])\n",
    "        \n",
    "    errors.append((np.mean(y_train != predict(X_train)), \n",
    "                   np.mean(y_test != predict(X_test))))\n",
    "    print(i, errors[-1])\n",
    "\n",
    "print('Total time: ', (dt.datetime.now()-t0).seconds / 60.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f546160d208>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl0XeV97//3V7M1D9ZkTbbl2BjMTF0IhAooxAm/hLQk\n/EwbxktKsyBN4dd7A6EEc6FdDeVHQ5qhUEiANkAguYAhk5NSmdIGMFMK2LLBg6zBkpFsa57P9/6x\nj6Vj4eHYls859vm81tpLZ8/P3pbPR8/z7MHcHRERSU4p8S6AiIjEj0JARCSJKQRERJKYQkBEJIkp\nBEREkphCQEQkiUUVAma2zMyazGyDmX1tL/M/a2a/M7O3zOw1Mzs7Yt6WyHkzWXgRETk8dqD7BMws\nBdgAXAC0A2uA5e7eFLFMtrsPhj+fCDzl7ovD45uA091955E5BBEROVTR1ASWAu+7e7O7jwFPApdE\nLrA7AMJygVDEuEW5HxERibFovpyrgJaI8dbwtD2Y2efMbB3wPHBtxCwHfm1ma8zsS4dTWBERmVkz\n9he6uz8bbgL6HHB3xKyz3f004NPADWZ2zkztU0REDk9aFMu0AbUR49XhaXvl7i+b2XwzK3b3He6+\nLTz9QzN7hqB56eXp65mZHmIkInKQ3N0OZ/1oagJrgAVmVmdmGcByYGXkAmZWH/H5NCDD3XeYWbaZ\n5Yan5wAXAe/ua0fursGdO+64I+5lSIRB50HnQudi/8NMOGBNwN0nzOxGYBVBaDzs7uvM7Ppgtj8I\nXGpmVwKjwBBwWXj1cuCZ8F/5acCP3H3VjJRcREQOWzTNQbj7L4FF06Y9EPH5HuCevay3GTjlMMso\nIiJHiC7dTEANDQ3xLkJC0HmYonMxRediZh3wZrFYMTNPlLKIiBwNzAw/zI7hqJqDROTYNXfuXJqb\nm+NdDNmPuro6tmzZckS2rZqASJIL/zUZ72LIfuzr32gmagLqExARSWIKARGRJKYQEBFJYgoBETnm\nffnLX+Zv/uZv4l2MhKSOYZEkl+gdw/PmzePhhx/m/PPPj3dR4kYdwyIi+zAxMRHvIkzaW1kOtnyx\nPh6FgIgkrCuvvJKtW7fymc98hvz8fO69916am5tJSUnhBz/4AXV1dVxwwQUAXHbZZVRWVlJUVERD\nQwNr166d3M4111zDN77xDQBWr15NTU0N9913H+Xl5VRVVfHII4/sswy9vb1cd911zJkzh5qaGm6/\n/fbJv8offfRRzjnnHG6++WZmz57NnXfeuddp7s7dd9/N3Llzqaio4Oqrr6a3txdgn8cTKwoBEUlY\njz32GLW1tbzwwgv09vbyV3/1V5PzXnrpJZqamvjVr34FwKc//Wk2btzI9u3bOe200/jTP/3TfW63\no6ODvr4+2tvbeeihh7jhhhvo6enZ67JXXXUVGRkZbNq0ibfeeotf//rXPPTQQ5PzX331VRYsWMD2\n7du57bbb9jrthz/8IY899hirV69m06ZN9PX1ceONN+6xn+nHEzPxfhRqxCNRXURiL5r/ezAzw6GY\nO3eu/9u//dvk+JYtWzwlJcW3bNmyz3V27tzpZua9vb3u7n711Vf77bff7u7ujY2Nnp2d7RMTE5PL\nl5WV+auvvvqR7XR2dnpmZqYPDw9PTnviiSf8vPPOc3f3Rx55xOvq6vZYZ2/TLrjgAv/+978/Ob5+\n/XpPT0/3iYmJqI5nX/9G4emH9d2rx0aIyAElYr9xdXX15OdQKMTXv/51fvKTn9DV1YWZYWZ0dXWR\nl5f3kXVLSkpISZlqCMnOzqa/v/8jyzU3NzM2NkZlZSUw9Udzbe3Ue7Zqamo+st70ae3t7dTV1U2O\n19XVMT4+Tmdn516PJ5YUAiKS0Mz2fvFL5PTHH3+c559/nhdffJHa2lp6enooKio67KueampqyMrK\noru7O6py7GvanDlz9ng+U3NzM+np6ZSXl9PS0rLP7cSC+gREJKFVVFSwadOmPaZN/3Lv6+sjMzOT\noqIiBgYGuPXWW2fkS7WiooKLLrqIm266ib6+PtydTZs28dJLLx3Udi6//HL+4R/+gS1bttDf389t\nt93G8uXLJ2sjhxtWh0MhICIJ7ZZbbuGuu+6iuLiY++67D/joX81XXnkltbW1VFVVsWTJEj7+8Y8f\n1D72FxiPPfYYo6OjHH/88RQXF/OFL3yBjo6Og9r+tddeyxVXXMG5555LfX092dnZfPvb345q/0ea\nbhYTSXKJfrOY6GYxERE5QhQCIiJJTCEgIpLEFAIiIklMISAiksSiCgEzW2ZmTWa2wcy+tpf5nzWz\n35nZW2b2mpmdHe26IiISPwe8RNTMUoANwAVAO7AGWO7uTRHLZLv7YPjzicBT7r44mnUjtqFLREXi\nQJeIJr54XyK6FHjf3ZvdfQx4ErgkcoHdARCWC4SiXVdEROInmhCoAloixlvD0/ZgZp8zs3XA88C1\nB7OuiIjEx4w9QM7dnwWeNbNzgLuBCw92GytWrJj83NDQQENDw0wVT0SOUjP1eslHH32Uhx56iP/4\nj/+YoZLFXmNjI42NjTO6zWhCoA2ojRivDk/bK3d/2czmm1nxwa4bGQIiIjPJ3Y/oM3omJiZITU09\n4LSD3Uak6X8c33nnnQddzumiaQ5aAywwszozywCWAysjFzCz+ojPpwEZ7r4jmnVFRPZlb6+XBHjl\nlVc4++yzKSoq4tRTT2X16tWT6zzyyCPU19eTn59PfX09TzzxBE1NTXz5y1/mt7/9LXl5eRQXF0e1\n/23btvH5z3+esrIy6uvr+cd//MfJeXfeeSdf+MIXuOKKKygsLOTRRx/d67TR0VH+8i//kqqqKqqr\nq7npppsYGxsDpl51ec8991BZWcm11167r6IcOdG8eQZYBqwH3gduCU+7Hviz8Of/BbwLvAn8J3DW\n/tbdxz72+VYdETlyEv3/3ty5c/3FF1+cHG9ra/OSkhL/5S9/6e7uv/nNb7ykpMS7urp8YGDA8/Pz\n/f3333d3946ODl+7dq27B2/8+sQnPhH1fkOhkJ9++ul+9913+/j4uG/evNnr6+t91apV7u6+YsUK\nz8jI8JUrV7q7+/Dw8EemDQ0N+e233+5nnXWWd3V1eVdXl3/84x/3b3zjG+4evOUsLS3Nb731Vh8d\nHd3jDWaR9vVvRKzeLObuvwQWTZv2QMTne4B7ol1XRI4udufMNKP4HYd2KapHXB75r//6r1x88cV8\n8pOfBOCCCy7gjDPO4Oc//zmXXnopqampvPPOO1RXV1NeXk55efkh7XPNmjV0dXVNvjd47ty5XHfd\ndTz55JNceGHQ5XnWWWfxmc98BoDMzMyPTMvKyuLxxx/nu9/9LiUlJQDccccd/Pmf//lkU05qaip3\n3nkn6enph1TOw6U3i4nIAR3ql/eR0NzczFNPPcXzzz8PBAExPj7O+eefT3Z2Nj/+8Y/5+7//e669\n9lrOOecc7r33XhYtOvi/Q5ubm2lra5tsOnJ3QqEQ55577uQy0b5aMvJ1lHV1dbS3t0+Ol5aWxi0A\nQCEgIgluemduTU0NV155JQ888MBel7/wwgu58MILGRkZ4bbbbuPP/uzPWL169UF3CtfU1DB//nzW\nr18fddn2Nq2qqorm5mYWL14MBOEyZ86c/W4jlvTsIBFJaNNfL/nFL36R559/nlWrVhEKhRgeHmb1\n6tW0t7ezfft2Vq5cyeDgIOnp6eTm5k6+wrG8vJzW1tbJTtkDWbp0KXl5edxzzz0MDw8zMTHBe++9\nx+uvv35Q5V++fDl33303XV1ddHV1cdddd3HFFVcc1DaOJIWAiCS06a+XrK6u5rnnnuNv//ZvKS0t\npa6ujnvvvZdQKEQoFOK+++6jqqqK2bNn89JLL/H9738fgPPPP58TTjiBiooKysrKDrjflJQUXnjh\nBd5++23mzZtHWVkZX/rSl+jt7T2o8v/1X/81Z5xxBieddBInn3wyZ5xxxmQ/QyLQ6yVFkpyeHZT4\n4v3sIBEROUYpBEREkphCQEQkiSkERESSmEJARCSJKQRERJKY7hgWSXJ1dXVxv2tV9q+uru6IbVv3\nCYiIHKV0n4CIiBwWhYCISBJLqBBQa5CISGwlVAiMjsa7BCIiySWhQmBgIN4lEBFJLgkVAoOD8S6B\niEhySagQ6O0fj3cRRESSSkKFQHevqgIiIrGUUCGwo1+dAiIisZRQIbBTISAiElNRhYCZLTOzJjPb\nYGZf28v8PzGz34WHl83spIh5W8LT3zKz1/a3n10Dag4SEYmlAz5AzsxSgO8AFwDtwBoze87dmyIW\n2wSc6+49ZrYMeBA4MzwvBDS4+84D7atnUDUBEZFYiqYmsBR4392b3X0MeBK4JHIBd3/F3XvCo68A\nVRGzLcr90DOkEBARiaVovpyrgJaI8Vb2/JKf7jrgFxHjDvzazNaY2Zf2t6O+YTUHiYjE0oy+T8DM\nzgOuAc6JmHy2u28zs1KCMFjn7i/vbf3f/vxRVvS/CUBDQwMNDQ0zWTwRkaNaY2MjjY2NM7rNA75P\nwMzOBFa4+7Lw+C2Au/s3py13EvBTYJm7b9zHtu4A+tz9vr3M84tvf4gX/vf/OLQjERFJMrF6n8Aa\nYIGZ1ZlZBrAcWDmtILUEAXBFZACYWbaZ5YY/5wAXAe/ua0cDo+oTEBGJpQM2B7n7hJndCKwiCI2H\n3X2dmV0fzPYHgduBYuB7FrynbszdlwLlwDNm5uF9/cjdV+1rX4NjCgERkViKqk/A3X8JLJo27YGI\nz18CPtLp6+6bgVOiLczQuDqGRURiKaHuGB6aUE1ARCSWEioEhkMKARGRWEqoEBh1NQeJiMRSgoWA\nagIiIrGUUCEwhmoCIiKxlFAhMG6qCYiIxFJihUCKQkBEJJYSKgRCqWoOEhGJpYQKAU8bIBSKdylE\nRJJHQoUAGQMMD8e7ECIiySOxQiB9kEG1CImIxExihUDqKL19E/EuhYhI0kioELDxbLr7dIWQiEis\nJFQIpE7ksKNP7UEiIrGSWCEQymFHv2oCIiKxklAhkEY2uwZUExARiZWECoEMz6FnUDUBEZFYSawQ\nsGx2KQRERGImsUIgJYe+ITUHiYjESkKFQFZKDn0jqgmIiMRKQoXArNRs+oYVAiIisZJYIZCWw8Co\nmoNERGIloUIgOz2HgTHVBEREYiWqEDCzZWbWZGYbzOxre5n/J2b2u/DwspmdFO26kXIyshkaV01A\nRCRWDhgCZpYCfAf4JHACcLmZHTdtsU3Aue5+MnA38OBBrDspNzOHoXHVBEREYiWamsBS4H13b3b3\nMeBJ4JLIBdz9FXfvCY++AlRFu26kvMwchkMKARGRWIkmBKqAlojxVqa+5PfmOuAXh7Ju/qxsRkJq\nDhIRiZW0mdyYmZ0HXAOccyjrv7HyZ+za/A4rVqygoaGBhoaGmSyeiMhRrbGxkcbGxhndprn7/hcw\nOxNY4e7LwuO3AO7u35y23EnAT4Fl7r7xYNYNz/Nv/exn3P7Ct+n93i9n4NBERI5tZoa72+FsI5rm\noDXAAjOrM7MMYDmwclpBagkC4IrdARDtupGKcnMYNzUHiYjEygGbg9x9wsxuBFYRhMbD7r7OzK4P\nZvuDwO1AMfA9MzNgzN2X7mvdfe2rODeH8RR1DIuIxMoBm4Nixcz85fXvce53L2Xi/n3mhIiIhMWq\nOShmZufnEEpVTUBEJFYSKgSK83IgfYCJiXiXREQkOSRUCORkZEP6IEND8S6JiEhySKgQmJU2C9JG\n6O1TVUBEJBYSKgTMDBufRXefLhMVEYmFhAoBgJSJHHYoBEREYiLhQiA1lMOOPl0hJCISCwkXAmmh\nbHb2KwRERGIh4UIgnRx2Dag5SEQkFhIuBDLIoWdINQERkVhIvBCwbHqHVBMQEYmFhAuBzBTVBERE\nYiXhQqAwJ5ttXQoBEZFYSLgQKC/Mp7Wr58ALiojIYUu4EFg8p472geZ4F0NEJCkkXAgsXVDPrpSN\nepKoiEgMJFwILK6YT0rJRlpa4l0SEZFjX8KFwPyi+YTyN9O0PhTvooiIHPMSLgRyM3LJpIA1Te3x\nLoqIyDEv4UIAoDyjnre3box3MUREjnkJGQLzCuvZ8OGmeBdDROSYl5AhsKSyntYB1QRERI60hAyB\n0+bX05u2kdHReJdEROTYFlUImNkyM2sysw1m9rW9zF9kZv9lZsNmdvO0eVvM7Hdm9paZvRbN/hbO\nnk962UY2qUVIROSISjvQAmaWAnwHuABoB9aY2XPu3hSxWDfwFeBze9lECGhw953RFqq+qJ5Q4UY2\nbIDjjot2LREROVjR1ASWAu+7e7O7jwFPApdELuDuXe7+BjC+l/Utyv1MKsspg9QRfrd+18GsJiIi\nBymaL+cqIPL+3dbwtGg58GszW2NmX4pmBTOjLL2et5vVHiQiciQdsDloBpzt7tvMrJQgDNa5+8t7\nW3DFihWTn4t25NO0fSNwWgyKKCKS+BobG2lsbJzRbZq7738BszOBFe6+LDx+C+Du/s29LHsH0Ofu\n9+1jW/ucb2YeWZbrf/o/eeLhEnp/fsvBHI+ISNIwM9zdDmcb0TQHrQEWmFmdmWUAy4GV+ytXRAGz\nzSw3/DkHuAh4N5qCnVxbz2DWRvr7o1laREQOxQGbg9x9wsxuBFYRhMbD7r7OzK4PZvuDZlYOvA7k\nASEz+ypwPFAKPGNmHt7Xj9x9VTQFW1A8n6zKp/jgAzjllEM7OBER2b+o+gTc/ZfAomnTHoj43AnU\n7GXVfuCQvsIjLxNVCIiIHBkJeccwQG1BLaPpHaxdr9uGRUSOlIQNgfTUdIrTq/ld85Z4F0VE5JiV\nsCEAMK9Aj5QWETmSEjoETq2rpzu0kY3KARGRIyKhQ2BBST3zTtvI00/HuyQiIsemhA6B+UXzyanZ\nyFNPxbskIiLHpoQOgdMqT+OD4Vdo2zahJiERkSMgoUNgbuFcagpqOPOy/1CTkIjIEZDQIQDw+cWf\nx5b8RE1CIiJHQMKHwKXHX8prvf+H1raQmoRERGZYwofAwpKFlOaUcvby/1KTkIjIDEv4EICgSSjl\nRDUJiYjMtKMjBI7/PK/1/pRtHSHefDPepREROXYcFSGwuHQx+Vn5XPX11/jqV+EA78EREZEoHRUh\nAEGT0OjHfsLAAPz4x/EujYjIseGAr5eMlemvl5zunc53+MwTn+Gx0zbzxS8a69ZBTk4MCygikmBi\n9XrJhLCkbAkl2SWsy36Qs8+Gb37kDcciInKwjpqaAMCG7g2c84NzePiCF7j6wqW88QbMnRub8omI\nJJqZqAkcVSEA8GzTs/zFL/6Ca8Ze5xc/KePFFyE3NwYFFBFJMEkZAgBf/7ev80rrK8x7eRWtW9N4\n/nnIyDjCBRQRSTBJ1ScQ6a7z7iItJQ3+ny+TmRXi2mshFIp3qUREjj5HZQikpqTy08t+ygc7N5Dz\nxavY3DzOzTfr/gERkYN1VDYH7TY4NsilT11KmmfT+b3HWVSfycMPq2lIRJJDzJqDzGyZmTWZ2QYz\n+9pe5i8ys/8ys2Ezu/lg1j0c2enZPPv/Pkt6upPzpYvpGt7GsmWwa9dM7kVE5Nh1wBAwsxTgO8An\ngROAy83suGmLdQNfAf7+ENY9LJlpmTz1hac4p+4s3vz9U5h1+tOcfTa88cZM7kVE5NgUTU1gKfC+\nuze7+xjwJHBJ5ALu3uXubwDjB7vuTEhLSeOu8+/i+cufZ2Pd7eRf8ydc/PkdXHUVtLbO9N5ERI4d\n0YRAFdASMd4anhaNw1n3oC2tWsqb17/J7y8pI+WGExmsfY6TT4bbboMdO47UXkVEjl5p8S5ApBUr\nVkx+bmhooKGh4aC3kZ2ezbeWfYs/XvzHXPvctZz7radp+c/7WbiwhBtugJtugsLCmSuziEisNDY2\n0tjYOKPbPODVQWZ2JrDC3ZeFx28B3N0/8vQeM7sD6HP3+w5h3YO+OuhABkYHuO3F23j8nce5bvHX\naPs/N7LymUz+6I/gqqvgE5+AlKPyIlkRkdhdHbQGWGBmdWaWASwHVu6vXIex7ozKycjhW8u+RePV\njbzT18hLJx/H3c/9iEWLx7nxRliwAO66S/0GIpK8orpPwMyWAfcThMbD7v53ZnY9wV/1D5pZOfA6\nkAeEgH7geHfv39u6+9jHjNcEpmvc0sjt/347zbuauXHpV/i9lOt4+l+KePJJOOss+OIXYdkyKCo6\nosUQEZkRSfvsoMP1evvr3P/q/fxsw8+47ITLuPbEG2hafSJPPw2rV8Ppp8PFFweBcMIJYId1ikVE\njgyFwGHa1reNf37zn3ngjQdYULyAG3/vRj5Z90e81JjGz34Gv/oVjIzARRfBhRfCeedBZWVMiygi\nsk8KgRkyNjHGM03P8O1Xv01LbwtfWfoVrjvtOgoyC/nggyAMXnwRGhuhogLOPx8aGuDcc6GsLC5F\nFhFRCBwJa9rWcP+r9/Ns07MsLl3M0jlLWVq1lPPmnUdVbi1vvRWEQWMjvPxyUDP4vd8LmpBOPx1O\nPBEKCuJ9FCKSDBQCR9Dg2CBvbXuLNe1reKX1FV7c/CKlOaV8asGn+ETtJ6gvrqc2bx4bm3J44w0m\nh/feg+JiOP54WLwYFi6ERYuCn9XV6l8QkZmjEIihidAEb2x7g1+8/wtea3+NTTs3sWXXFoqyiji+\n9HiWlC3hhNITmFdYT+pADbuaa9i4PosNG2DDBmhqgqEhWLIkqC0sWAC1tcFQXw+lpfE+QhE52igE\n4izkIVp7W1n74Vre3f4u725/l827NtPS00JbXxsFmQXUFNRQk19DXUEdtbOOJ6NnCYNbTqBjSyEt\nLbB1K7z/PmRnwymnwMknw/z5QTjU1QXvUM7MjPeRikgiUggksJCH2D6wnZaeFrb2bGXLri1BWHz4\nLms/XEvIQ+Rn5lOQWUBRVhH5qRWkDFQy0l3JWFc1/e01dG+uoXNDDZWzs1mwIKg9zJ07NSxapHsa\nRJKZQuAo5e4MjA3QM9xD70gv3UPddPZ3sq1/G9v6ttHS20JLbxAebb1tZKflUpxaS+74XDL7FxL6\ncBF9WxbS8l4VuVRwwqIsFi4MmpXq64OAKC8Pmpj0gh2RY5dCIAmEPMSHAx/S0tvCll1bWN+1nvXd\n69nQvYH2vnY6+zvJsGzSPZ/QWBoTY+mMD88i1F/CWM9s0sdLyEsrpCArn+KcAsrzSqktqKG+tIaP\nVZZTW5NCdTWUlKjTWuRooxAQ3J0dQzvoG+1jbGKM8dA4Q+NDdA92s32gi9bubtp39NCxs4cP+3rY\nPthJ12gLPd7CqPWR3reA0PZF+IeLKLK5lM8K+jBqyvOpmDNG+ZxRaqrSOPekuRTkJdRDZ0WSnkJA\nDsvA6AAbujewvns9727bQFNHM1t2tNA+sJWBsX58Ip3QWDqjE6OMZWwnvWcRxRNLqJ71MeqL53Ni\n9XyWLqrlrBPLyctJj/fhiCQdhYDETM9QP43vreOlde+ytmMjW3o20Tm6ib6UrYxnfEjqWCE5XklJ\nejVzcmuZX1zDnNm5lM5OJz87ndyMXCpyK6jMq6Q0u5TMtEzSU9JJT00nxfQ8b5FDoRCQhDA0PMGr\n73bx6tp21ra2sqmrhbb+FnYNDNA3MI6ljZFd1Et6UQehWR0Mp24nZKNMEDRfFWQVUJFbQUVuBXML\n53JK+SmcWnkqJ5efTEGWbr8W2ReFgCQ8d9i5EzZtgg8+CO6JiBxGRp2q+p2U1HWQN2cbGeUbGSl6\nm057iw/63iE9NT3ooyioob6onkUlizhu9nHUF9dTkVtBVlpWvA9RJG4UAnLU27EDWlqgrQ3a24Ow\nWLs2GJq3OhXzuin/WAuFdS2kln7AcM56dqauZ/vYJrpHOslOz6Y8p5yiWUWT913MSp8VNDWlpJOT\nkUN1fjW1BbXUFdRxUvlJpKeq/0KODQoBOaaNjAR3VG/eHITD1q1BYLS0QHMztLU7hRU7qVjQwZz5\nPZTX9lA8p4fC2cPkF46RmT1G/2gfLT3BfRcbd25ka89Wzq07lz+c94ecWnkqNfk1VOVXkZGqGyrk\n6KMQkKQ2MQGdnbBly9TzmZqagtBoaYHh4eDRG8cdFzzMb/FiqJjfRXvmi/zntt/w3ofv0dLTQkd/\nB/mZ+WSkZpCemk5maibV+dXML5rP/KL5LCxZyJKyJSwoXkBaii6TlcShEBDZj/7+IBCammDdumBo\nagoCo7gYPvYxmDcPaueOM7tmB6UVY8wuG6OodITusRY27dzExh0bg0tot79Le187C4oXUF9cz/zC\n+dQX13NKxSmcWnEqs9JnxftwJQkpBEQOQSgUNCdt3Bg0M23eHNQm2tuDvom2NigsnHoE+Lx5UFMD\nJZUDjOSuZyB9M21DG/lgxwe8ue1N1n64luNmH8fJFSezqGRRMMxeRH1RPZlpevqfHDkKAZEjIBQK\nAmHDBli/PgiI1tagiam1FTo6ICUleKHQvHkwd8Ews+a9zVjRO/Rlrmf7xHo2961na89WqvOrWViy\nkLKcMgoyC8jPzKckuyS4ZyK3korcCmZnz6Ywq5DUlNR4H7ocZRQCInHgDn19QVBs3hxc+vrBB0FY\nbN0aDMPDUDN3lNKFm8it20BOaRfZRb1k5vcwkdVFX6iTjoFtdPR30D3YTe9IL4VZhVTmVVJbUBt0\nWOdVUZlXSWVuJZV5lcwrnEfRLD02VqYoBEQSVF9fEAZbtgRDc/Oen3t6giamuXODJqcFC8epnL+D\n7PJ2JnJbaO/fSltfGx39HWzr30Zbbxubd20mLSWN+UXzqcytZHb2bEpmlTAnbw4LSxayaPYi5hfN\nV+d1ElEIiBylhoamLn/d3ezU1BT0UbS3w5w5QVPT7hcLzZ0LNTVOXnk3ozmb2DHSSddgF91D3bT2\ntrK+ez3ru9bT1tdGRW7F5A12VXlVk01Pc/LmUFNQQ3V+tW6yO0bELATMbBnwLSAFeNjdv7mXZb4N\nfAoYAK5x97fC07cAPUAIGHP3pfvYh0JABBgbCwJi06ag1rC7FrG7qam9HWbPDvdHzA2CoqoqCI6y\nylHIb2MofSvtA1vZ1h80OXX0d9DW1zb51rv8zHyKsoooyCqYvMlu8iVHs4r26LMozSmlZFYJ+Zn5\nmJ43nlDqGk1YAAAKhElEQVRiEgJmlgJsAC4A2oE1wHJ3b4pY5lPAje5+sZn9PnC/u58ZnrcJON3d\ndx5gPwoBkShMTEz1R+xuXtp9x3VbG2zbBl1dwVvnKiuDYc6cqRcNFZeEyCj6kLzSXeQU9zCeGrzc\nqGekh57hHnYM7aBzoHOyKap7sJuuwS6Gx4cpzy3fo89idvZsSrJLKJ5VTFZaFukp6WSkZlCVX8WC\n4gV6OOARFqsQOBO4w90/FR6/BfDI2oCZ/RPw7+7+4/D4OqDB3TvNbDNwhrt3H2A/CgGRGTIxAdu3\nB4Gwe+joCMKhqyu4ya61NahZZGRARUVw70RJSfAzcqioCIKkuGwEcjv4cKRlskaxOyC6h7oZmRhh\nbGKMsdAYm3dupmekhzPmnMGS0iV79FNkpmWSnZ5Ndno2xbOK92i60v0WB2cmQiCaHqQqoCVivBWY\n3qQzfZm28LROwIFfm9kE8KC7//OhF1dEopGaOlUL2B936O4OAmPHjuBzd3fw0L8dO+C//xtWrdod\nJJl0dtaRmVlHWVlQs9gdEGfMmfpcUQFlZTCRtZ13utfQ1NVEyEPB/nBGxkfoG+mjo7+DNe1rJh/r\n0d7XDkBBZgEFWQXkpOcwK30W2enZlGaX7nH/RV5m3mSQ5GXkkZWWpaaqQxSLywjOdvdtZlZKEAbr\n3P3lvS24YsWKyc8NDQ00NDTEoHgiycss6F+YPTu65d2htzcIjY6OPWsaGzZMfe7qgu7uMlJTL6ak\n5OI9ahZz5kB1NZxYE66BzA+arkpKIDVzmN6RHnpGehgcG5wcOvo7WN+1nmebnmXzrs0MjA4wODbI\nwNgAfSN9TPjEVL9GVgEFmQUUZhVSlVdFTUENtQW1nFx+MotLFx/VTVSNjY00NjbO6DajbQ5a4e7L\nwuPRNAc1AX/g7p3TtnUH0Ofu9+1lP2oOEjmGuMPAwJ41i+7uoO9i9813nZ1T87q6gvXKy6f6L0pL\ng4AqLw/CY064xjF7dnBXd2r4/rqR8RH6RvvoGQ4CpHeklx1DO2jtbaWlp4Xmnmbe3PYm3UPdnFV9\nFsfNPm6PMJiVNmuyZlGYVUhJdsnkJbizs2dTkFWQkOERqz6BVGA9QcfwNuA14HJ3XxexzKeBG8Id\nw2cC33L3M80sG0hx934zywFWAXe6+6q97EchIJLk+vuDYOjsDELhww+Dn7trHe3twc/u7qBGkp8f\nBEVZ2dRQXj71M/Jzfj50DnTw25bfsnHnxsl9hjzE8PgwQ2NDDIwNsGt4F91DQV9H12AX3YPdDIwN\nUJhVSGFW4eRVVLkZuZPNVTnpOZPTC7MKqcqvmuzrKMoqOmJNVbG+RPR+pi4R/Tszu56gRvBgeJnv\nAMuYukT0TTObBzxD0C+QBvzI3f9uH/tQCIhI1CYmYNeuICi2bw+Gzs6pn7uH3ePj41P9JBUVU7WM\n0tLgEtuammAoLYX0aa+cGJsYY8fQjskrqHpGehgYHWBofCholhodmKyB7BzaGVyO2xt0oPeO9E6G\nxe4rqNJTg/dd5GXmfeQS3YKsAvIy8sjJyJmsnZRml06+fS8vM2+yXLpZTEQkSv39U30Wu2saXV1B\nSLS2TjVRdXVBTs7UlVIlJVNDeflHQ2T2bMjLC/pX9mYiNMHw+DCDY4MMjQ9NXkE1OjFK/2j/Hk1Y\nvSO99AwHn3cHTP9oP12DXWzr38a2vm3cfNbN3H3+3YBCQERkxoVCQVNT5NVS3d1Tl9bubpraHSJd\nXTA6GgTD7iEyOAoLg5DIy4Pc3CBgdg/FxUEzVbStRe7OhE9MXnKrEBARSQBDQ1MBsft+jN0hsmtX\n8Cyp3cPgYDD09wfLDA8HtYmioiAQ8vODwNj9OT8/mFdcHPw8/nhYsCDYr0JAROQoNzIShMauXUEN\nZPfQ1xf87OkJrqDafRXVJZfAddcF6yoERESS2EyEQOJd+CoiIjGjEBARSWIKARGRJKYQEBFJYgoB\nEZEkphAQEUliCgERkSSmEBARSWIKARGRJKYQEBFJYgoBEZEkphAQEUliCgERkSSmEBARSWIKARGR\nJKYQEBFJYgoBEZEkphAQEUliUYWAmS0zsyYz22BmX9vHMt82s/fN7G0zO+Vg1hURkfg4YAiYWQrw\nHeCTwAnA5WZ23LRlPgXUu/vHgOuBf4p2XfmoxsbGeBchIeg8TNG5mKJzMbOiqQksBd5392Z3HwOe\nBC6ZtswlwGMA7v4qUGBm5VGuK9Polzyg8zBF52KKzsXMiiYEqoCWiPHW8LRololmXRERiZMj1TFs\nR2i7IiIyg8zd97+A2ZnACndfFh6/BXB3/2bEMv8E/Lu7/zg83gT8ATDvQOtGbGP/BRERkY9w98P6\nozstimXWAAvMrA7YBiwHLp+2zErgBuDH4dDY5e6dZtYVxbrA4R+IiIgcvAOGgLtPmNmNwCqC5qOH\n3X2dmV0fzPYH3f3nZvZpM/sAGACu2d+6R+xoRETkoBywOUhERI5dcb9jOJlvJjOzajN70czeM7N3\nzOwvwtOLzGyVma03s1+ZWUG8yxorZpZiZm+a2crweFKeCzMrMLOnzWxd+Pfj95P4XNxkZu+a2X+b\n2Y/MLCNZzoWZPWxmnWb23xHT9nnsZnZr+KbddWZ2UTT7iGsI6GYyxoGb3f0E4CzghvDx3wL8xt0X\nAS8Ct8axjLH2VWBtxHiynov7gZ+7+2LgZKCJJDwXZjYH+ApwmrufRNCEfTnJcy5+SPD9GGmvx25m\nxwOXAYuBTwHfM7MD9rXGuyaQ1DeTuXuHu78d/twPrAOqCc7Bo+HFHgU+F58SxpaZVQOfBh6KmJx0\n58LM8oFPuPsPAdx93N17SMJzEZYK5JhZGjALaCNJzoW7vwzsnDZ5X8f+WeDJ8O/LFuB9gu/Y/Yp3\nCOhmsjAzmwucArwClLt7JwRBAZTFr2Qx9Q/A/wQiO6qS8VzMA7rM7IfhprEHzSybJDwX7t4O/P/A\nVoIv/x53/w1JeC4ilO3j2Kd/n7YRxfdpvENAADPLBX4CfDVcI5jeW3/M996b2cVAZ7hmtL8q7DF/\nLgiaPE4DvuvupxFccXcLyfl7UUjwl28dMIegRvCnJOG52I/DOvZ4h0AbUBsxXh2eljTCVdyfAP/i\n7s+FJ3eGn72EmVUA2+NVvhg6G/ismW0CngDON7N/ATqS8Fy0Ai3u/np4/KcEoZCMvxd/CGxy9x3u\nPgE8A3yc5DwXu+3r2NuAmojlovo+jXcITN6IZmYZBDeTrYxzmWLtB8Bad78/YtpK4Orw56uA56av\ndKxx96+7e627zyf4PXjR3a8Anif5zkUn0GJmC8OTLgDeIwl/Lwiagc40s6xwJ+cFBBcOJNO5MPas\nHe/r2FcCy8NXT80DFgCvHXDj8b5PwMyWEVwJsftmsr+La4FiyMzOBl4C3iGo0jnwdYJ/uKcIUr0Z\nuMzdd8WrnLFmZn8A/H/u/lkzKyYJz4WZnUzQQZ4ObCK4ATOV5DwXdxD8YTAGvAVcB+SRBOfCzB4H\nGoASoBO4A3gWeJq9HLuZ3Qr8D4Jz9VV3X3XAfcQ7BEREJH7i3RwkIiJxpBAQEUliCgERkSSmEBAR\nSWIKARGRJKYQEBFJYgoBEZEkphAQEUli/xdX8B1eojxV7wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f54615a9ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = np.array(errors)\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 0], label='train error')\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 1], label='test  error')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complex Neural Net\n",
    "\n",
    "Two hidden layers with dropout\n",
    "\n",
    "![a](d4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams\n",
    "\n",
    "srng = RandomStreams()\n",
    "\n",
    "def rectify(X):\n",
    "    return T.maximum(X, 0.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding rectifier units\n",
    "![A](d5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def RMSprop(cost, params, lr=0.001, rho=0.9, epsilon=1e-6):\n",
    "    grads = T.grad(cost=cost, wrt=params)\n",
    "    updates = []\n",
    "    for p, g in zip(params, grads):\n",
    "        acc = theano.shared(p.get_value() * 0.)\n",
    "        acc_new = rho * acc + (1 - rho) * g ** 2\n",
    "        gradient_scaling = T.sqrt(acc_new + epsilon)\n",
    "        g = g / gradient_scaling\n",
    "        updates.append((acc, acc_new))\n",
    "        updates.append((p, p - lr * g))\n",
    "    return updates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RMSprop\n",
    "\n",
    "\n",
    "RMSprop is an unpublished, adaptive learning rate method proposed by Geoff Hinton in \n",
    "[Lecture 6e of his Coursera Class](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)\n",
    "\n",
    "RMSprop and Adadelta have both been developed independently around the same time stemming from the need to resolve Adagrad's radically diminishing learning rates. RMSprop in fact is identical to the first update vector of Adadelta that we derived above:\n",
    "\n",
    "$$ E[g^2]_t = 0.9 E[g^2]_{t-1} + 0.1 g^2_t. $$\n",
    "\n",
    "$$\\theta_{t+1} = \\theta_{t} - \\frac{\\eta}{\\sqrt{E[g^2]_t + \\epsilon}} g_{t}.$$\n",
    "\n",
    "RMSprop as well divides the learning rate by an exponentially decaying average of squared gradients. Hinton suggests $\\gamma$ to be set to 0.9, while a good default value for the learning rate $\\eta$ is 0.001."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dropout(X, p=0.):\n",
    "    if p > 0:\n",
    "        retain_prob = 1 - p\n",
    "        X *= srng.binomial(X.shape, p=retain_prob, dtype=theano.config.floatX)\n",
    "        X /= retain_prob\n",
    "    return X\n",
    "\n",
    "def model(X, w_h, w_h2, w_o, p_drop_input, p_drop_hidden):\n",
    "    X = dropout(X, p_drop_input)\n",
    "    h = rectify(T.dot(X, w_h))\n",
    "\n",
    "    h = dropout(h, p_drop_hidden)\n",
    "    h2 = rectify(T.dot(h, w_h2))\n",
    "\n",
    "    h2 = dropout(h2, p_drop_hidden)\n",
    "    py_x = softmax(T.dot(h2, w_o))\n",
    "    return h, h2, py_x\n",
    "\n",
    "def softmax(X):\n",
    "    e_x = T.exp(X - X.max(axis=1).dimshuffle(0, 'x'))\n",
    "    return e_x / e_x.sum(axis=1).dimshuffle(0, 'x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "w_h = init_weights((784, 625))\n",
    "w_h2 = init_weights((625, 625))\n",
    "w_o = init_weights((625, 10))\n",
    "\n",
    "noise_h, noise_h2, noise_py_x = model(X, w_h, w_h2, w_o, 0.2, 0.5)\n",
    "h, h2, py_x = model(X, w_h, w_h2, w_o, 0., 0.)\n",
    "y_x = T.argmax(py_x, axis=1)\n",
    "\n",
    "cost = T.mean(T.nnet.categorical_crossentropy(noise_py_x, Y))\n",
    "params = [w_h, w_h2, w_o]\n",
    "updates = RMSprop(cost, params, lr=0.001)\n",
    "\n",
    "train = theano.function(inputs=[X, Y], outputs=cost, updates=updates, allow_input_downcast=True)\n",
    "predict = theano.function(inputs=[X], outputs=y_x, allow_input_downcast=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 (0.063916666666666663, 0.062399999999999997)\n",
      "1 (0.034116666666666663, 0.035400000000000001)\n",
      "2 (0.025133333333333334, 0.027799999999999998)\n",
      "3 (0.020866666666666665, 0.025600000000000001)\n",
      "4 (0.017466666666666665, 0.023900000000000001)\n",
      "5 (0.015233333333333333, 0.0206)\n",
      "6 (0.015366666666666667, 0.021700000000000001)\n",
      "7 (0.012366666666666666, 0.020199999999999999)\n",
      "8 (0.011950000000000001, 0.019800000000000002)\n",
      "9 (0.011983333333333334, 0.020199999999999999)\n",
      "10 (0.0091000000000000004, 0.0178)\n",
      "11 (0.0084666666666666675, 0.017000000000000001)\n",
      "12 (0.0086999999999999994, 0.017500000000000002)\n",
      "13 (0.0074333333333333335, 0.016500000000000001)\n",
      "14 (0.0067999999999999996, 0.017100000000000001)\n",
      "15 (0.0066166666666666665, 0.016299999999999999)\n",
      "16 (0.005783333333333333, 0.015699999999999999)\n",
      "17 (0.0061833333333333332, 0.0161)\n",
      "18 (0.0058666666666666667, 0.015100000000000001)\n",
      "19 (0.0061500000000000001, 0.017100000000000001)\n",
      "20 (0.0050499999999999998, 0.016199999999999999)\n",
      "21 (0.004816666666666667, 0.016500000000000001)\n",
      "22 (0.0045833333333333334, 0.0161)\n",
      "23 (0.0041000000000000003, 0.015699999999999999)\n",
      "24 (0.0040666666666666663, 0.016)\n",
      "25 (0.0035666666666666668, 0.0149)\n",
      "26 (0.0043333333333333331, 0.0161)\n",
      "27 (0.0032666666666666669, 0.013899999999999999)\n",
      "28 (0.0033333333333333335, 0.0143)\n",
      "29 (0.0029166666666666668, 0.013100000000000001)\n",
      "30 (0.0032166666666666667, 0.014500000000000001)\n",
      "31 (0.0033333333333333335, 0.014999999999999999)\n",
      "32 (0.0028333333333333335, 0.015100000000000001)\n",
      "33 (0.0029666666666666665, 0.0146)\n",
      "34 (0.0030500000000000002, 0.0137)\n",
      "35 (0.0029166666666666668, 0.0147)\n",
      "36 (0.0023666666666666667, 0.014200000000000001)\n",
      "37 (0.0030000000000000001, 0.0149)\n",
      "38 (0.0028333333333333335, 0.0137)\n",
      "39 (0.0020666666666666667, 0.012699999999999999)\n",
      "40 (0.0021666666666666666, 0.0134)\n",
      "41 (0.0023666666666666667, 0.0129)\n",
      "42 (0.0023500000000000001, 0.0141)\n",
      "43 (0.0021666666666666666, 0.013899999999999999)\n",
      "44 (0.0021833333333333331, 0.0154)\n",
      "45 (0.0020500000000000002, 0.0146)\n",
      "46 (0.0021833333333333331, 0.014800000000000001)\n",
      "47 (0.0016166666666666666, 0.014)\n",
      "48 (0.00175, 0.012800000000000001)\n",
      "49 (0.0017333333333333333, 0.012999999999999999)\n",
      "50 (0.0016833333333333333, 0.012999999999999999)\n",
      "51 (0.0013833333333333334, 0.0118)\n",
      "52 (0.0019333333333333333, 0.014200000000000001)\n",
      "53 (0.0013166666666666667, 0.012500000000000001)\n",
      "54 (0.0015, 0.013299999999999999)\n",
      "55 (0.0012166666666666667, 0.0132)\n",
      "56 (0.0012999999999999999, 0.012)\n",
      "57 (0.00115, 0.011900000000000001)\n",
      "58 (0.0014333333333333333, 0.012999999999999999)\n",
      "59 (0.0014666666666666667, 0.012500000000000001)\n",
      "60 (0.0010833333333333333, 0.0124)\n",
      "61 (0.00125, 0.0126)\n",
      "62 (0.0013333333333333333, 0.013299999999999999)\n",
      "63 (0.0012166666666666667, 0.0126)\n",
      "64 (0.0011000000000000001, 0.0129)\n",
      "65 (0.0010833333333333333, 0.012800000000000001)\n",
      "66 (0.0010166666666666666, 0.012999999999999999)\n",
      "67 (0.0010666666666666667, 0.013299999999999999)\n",
      "68 (0.001, 0.013100000000000001)\n",
      "69 (0.0010499999999999999, 0.013100000000000001)\n",
      "70 (0.00089999999999999998, 0.0121)\n",
      "71 (0.00075000000000000002, 0.012699999999999999)\n",
      "72 (0.00080000000000000004, 0.012999999999999999)\n",
      "73 (0.00084999999999999995, 0.0129)\n",
      "74 (0.00083333333333333339, 0.012200000000000001)\n",
      "75 (0.00083333333333333339, 0.012200000000000001)\n",
      "76 (0.00081666666666666671, 0.0121)\n",
      "77 (0.00075000000000000002, 0.013299999999999999)\n",
      "78 (0.00061666666666666662, 0.0124)\n",
      "79 (0.00066666666666666664, 0.0134)\n",
      "80 (0.0006333333333333333, 0.012800000000000001)\n",
      "81 (0.00061666666666666662, 0.013100000000000001)\n",
      "82 (0.0006333333333333333, 0.0134)\n",
      "83 (0.00080000000000000004, 0.012200000000000001)\n",
      "84 (0.00061666666666666662, 0.0124)\n",
      "85 (0.00069999999999999999, 0.0129)\n",
      "86 (0.00075000000000000002, 0.0126)\n",
      "87 (0.00046666666666666666, 0.011900000000000001)\n",
      "88 (0.00056666666666666671, 0.0118)\n",
      "89 (0.00051666666666666668, 0.0121)\n",
      "90 (0.00078333333333333336, 0.012)\n",
      "91 (0.00051666666666666668, 0.012)\n",
      "92 (0.00050000000000000001, 0.0118)\n",
      "93 (0.00056666666666666671, 0.012200000000000001)\n",
      "94 (0.00055000000000000003, 0.0124)\n",
      "95 (0.00069999999999999999, 0.012800000000000001)\n",
      "96 (0.00058333333333333338, 0.0135)\n",
      "97 (0.00055000000000000003, 0.012699999999999999)\n",
      "98 (0.00050000000000000001, 0.012999999999999999)\n",
      "99 (0.00048333333333333334, 0.0117)\n",
      "Total time:  17.85\n"
     ]
    }
   ],
   "source": [
    "t0 = dt.datetime.now()\n",
    "\n",
    "errors = []\n",
    "for i in range(100):\n",
    "    \n",
    "    for start, end in zip(range(0, X_train.shape[0], 128), \n",
    "                          range(128, X_train.shape[0], 128)):\n",
    "        cost = train(X_train[start:end], y_train2[start:end])\n",
    "        \n",
    "    errors.append((np.mean(y_train != predict(X_train)), \n",
    "                   np.mean(y_test != predict(X_test))))\n",
    "    print(i, errors[-1])\n",
    "\n",
    "print('Total time: ', (dt.datetime.now()-t0).seconds / 60.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f545e83d4a8>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FdX9//HXJyQsISwJSwLZgIBsFkQtsmpkUQJV3AUX\nKqhQlVZs9StqFazW/aeCKGhVFleq1RIsIFgMuCLIImUPS0yARCKQPZDl8/tjbkIICbmB5N4L+Twf\nj/sgM/fMzLmTMO975sycEVXFGGNM3eTn7QoYY4zxHgsBY4ypwywEjDGmDrMQMMaYOsxCwBhj6jAL\nAWOMqcPcCgERGSYiW0Vku4g8WEmZ6SKyQ0TWi8h5rnnniMg6EVnr+jdDRP5Ukx/AGGPMqZOq7hMQ\nET9gOzAY2AesBkap6tYyZeKAiao6QkQuAqapap8K1pMCXKSqyTX7MYwxxpwKd1oCvYEdqpqkqgXA\nh8DIcmVGAvMAVHUV0ExEQsuVGQLstAAwxhjf4U4IhANlD9wprnknK7O3gjI3Ah9Ut4LGGGNqj0c6\nhkUkALgS+MgT2zPGGOMefzfK7AWiykxHuOaVLxN5kjJxwI+qeqCyjYiIDWJkjDHVpKpyOsu70xJY\nDXQUkWgRqQ+MAuLLlYkHxgCISB/gsKqmlXl/NG6cClJVe6kyZcoUr9fBF162H2xf2L44+asmVNkS\nUNUiEZkILMUJjbdUdYuITHDe1jdUdZGIDBeRRCAHGFuyvIgE4nQKj6+RGhtjjKkx7pwOQlWXAJ3L\nzXu93PTESpbNBVqdagWNMcbUHrtj2AfFxsZ6uwo+wfbDMbYvjrF9UbOqvFnMU0REfaUuxhhzJhAR\n9DQ7ht06HWSMOXu1a9eOpKQkb1fDnER0dDR79uyplXVbS8CYOs71bdLb1TAnUdnvqCZaAtYnYIwx\ndZiFgDHG1GEWAsYYU4dZCBhjznp33XUXf//7371dDZ9kHcPG1HG+3jHcvn173nrrLQYNGuTtqniN\ndQwbY0wlioqKvF2FUhXVpbr18/TnsRAwxvisMWPG8PPPP3PFFVfQtGlTXnjhBZKSkvDz8+Ptt98m\nOjqawYMHA3DDDTfQpk0bgoODiY2NZfPmzaXrGTt2LI899hgAK1asIDIykhdffJHQ0FDCw8OZM2dO\npXXIzMzkjjvuoG3btkRGRvLoo4+WfiufO3cuAwYM4M9//jMtW7bk8ccfr3CeqvLkk0/Srl07wsLC\nuO2228jMzASo9PN4ioWAMcZnzZs3j6ioKD777DMyMzO5//77S99buXIlW7du5fPPPwdg+PDh7Ny5\nk19++YXzzz+fm2++udL1pqamkpWVxb59+3jzzTe55557yMjIqLDs73//e+rXr8+uXbtYt24dy5Yt\n48033yx9f9WqVXTs2JFffvmFRx55pMJ5s2fPZt68eaxYsYJdu3aRlZXFxInHD7dW/vN4jLeHQi0z\nJKoaYzzPnf97UDOvU9GuXTv973//Wzq9Z88e9fPz0z179lS6zKFDh1RENDMzU1VVb7vtNn300UdV\nVTUhIUEDAwO1qKiotHzr1q111apVJ6wnLS1NGzRooPn5+aXzPvjgA7300ktVVXXOnDkaHR193DIV\nzRs8eLDOnDmzdHrbtm0aEBCgRUVFbn2eyn5Hrvmndey1YSOMMVXyxX7jiIiI0p+Li4t5+OGH+fjj\nj0lPT0dEEBHS09Np0qTJCcu2aNECP79jJ0ICAwPJzs4+oVxSUhIFBQW0adMGOPalOSrq2HO2IiMj\nT1iu/Lx9+/YRHR1dOh0dHU1hYSFpacceu1L283iShYAxxqeJVHzxS9n577//PgsXLmT58uVERUWR\nkZFBcHDwaV/1FBkZScOGDfn111/dqkdl89q2bXvc+ExJSUkEBAQQGhpKcnJypevxBOsTMMb4tLCw\nMHbt2nXcvPIH96ysLBo0aEBwcDA5OTk89NBDNXJQDQsL47LLLuO+++4jKysLVWXXrl2sXLmyWusZ\nPXo0L730Env27CE7O5tHHnmEUaNGlbZGTjesToeFgDHGp02ePJknnniCkJAQXnzxReDEb81jxowh\nKiqK8PBwzj33XPr161etbZwsMObNm8fRo0fp1q0bISEhXH/99aSmplZr/ePGjePWW2/l4osvJiYm\nhsDAQKZPn+7W9mub3SxmTB3n6zeLGbtZzBhjTC2xEDDGmDrMQsAYY+owCwFjjKnDLASMMaYOcysE\nRGSYiGwVke0i8mAlZaaLyA4RWS8i55WZ30xEPhKRLSKySUQuqqnKG2OMOT1VhoCI+AEzgMuB7sBo\nEelSrkwcEKOqnYAJwKwyb08DFqlqV6AnsKWG6m6MMeY0udMS6A3sUNUkVS0APgRGliszEpgHoKqr\ngGYiEioiTYGBqjrb9V6hqmbWXPWNMcacDndCIBxILjOd4pp3sjJ7XfPaA+kiMltE1orIGyLS6HQq\nbIwxpubU9gBy/sD5wD2qukZEXgYmA1MqKjx16tTSn2NjY4mNja3l6hljfF1NPV5y7ty5vPnmm3z1\n1Vc1VDPPS0hIICEhoUbX6U4I7AWiykxHuOaVLxNZSZlkVV3j+vljoMKOZTg+BIwxpiapaq2O0VNU\nVES9evWqnFfddZRV/svx448/Xu16lufO6aDVQEcRiRaR+sAoIL5cmXhgDICI9AEOq2qaqqYBySJy\njqvcYGAzxhjjhooeLwnw/fff079/f4KDg+nVqxcrVqwoXWbOnDnExMTQtGlTYmJi+OCDD9i6dSt3\n3XUX3333HU2aNCEkJMSt7e/fv5/rrruO1q1bExMTwyuvvFL63uOPP87111/PrbfeSvPmzZk7d26F\n844ePcqkSZMIDw8nIiKC++67j4KCAuDYoy6fe+452rRpw7hx42pw77nJnSfPAMOAbcAOYLJr3gRg\nfJkyM4BEYANwfpn5PXGCZD3wCdCskm1U+lQdY0zt8fX/e+3atdPly5eXTu/du1dbtGihS5YsUVXV\nL774Qlu0aKHp6emak5OjTZs21R07dqiqampqqm7evFlVnSd+DRw40O3tFhcX6wUXXKBPPvmkFhYW\n6u7duzUmJkaXLl2qqqpTp07V+vXra3x8vKqq5ufnnzAvLy9PH330Ue3bt6+mp6drenq69uvXTx97\n7DFVdZ5y5u/vrw899JAePXr0uCeYlVXZ7whPPVlMVZcAncvNe73c9PEPzDw2fwPwW3e2Y4zxTfJ4\nzZxG0SmnNlqplhlB891332XEiBFcfvnlAAwePJgLL7yQRYsWce2111KvXj02btxIREQEoaGhhIaG\nntI2V69eTXp6eulzg9u1a8cdd9zBhx9+yNChQwHo27cvV1xxBQANGjQ4YV7Dhg15//33efXVV2nR\nogUAU6ZM4Q9/+EPpqZx69erx+OOPExAQcEr1PF32ZDFjTJVO9eBdG5KSkvjnP//JwoULAScgCgsL\nGTRoEIGBgcyfP5/nn3+ecePGMWDAAF544QU6d+5cxVor3s7evXtLTx2pKsXFxVx88cWlZdx9tGTZ\nx1FGR0ezb9++0ulWrVp5LQDAQsAY4+PKd+ZGRkYyZswYXn/99QrLDx06lKFDh3LkyBEeeeQRxo8f\nz4oVK6rdKRwZGUmHDh3Ytm2b23WraF54eDhJSUl07doVcMKlbdu2J12HJ9nYQcYYn1b+8ZK33HIL\nCxcuZOnSpRQXF5Ofn8+KFSvYt28fv/zyC/Hx8eTm5hIQEEBQUFDpIxxDQ0NJSUkp7ZStSu/evWnS\npAnPPfcc+fn5FBUVsWnTJtasWVP1wmWMGjWKJ598kvT0dNLT03niiSe49dZbq7WO2mQhYIzxaeUf\nLxkREcGCBQt46qmnaNWqFdHR0bzwwgsUFxdTXFzMiy++SHh4OC1btmTlypXMnDkTgEGDBtG9e3fC\nwsJo3bp1ldv18/Pjs88+Y/369bRv357WrVtz5513kplZvUEP/vrXv3LhhRfSo0cPevbsyYUXXlja\nz+AL7PGSxtRx9nhJ32ePlzTGGFMrLASMMaYOsxAwxpg6zELAGGPqMJ8KgaIib9fAGGPqFp8Kgbw8\nb9fAGGPqFp+6Yzg3VwkK8u7dc8bUNdHR0V6/a9WcXHR0dK2t26fuE9i2M49zOjT0dlWMMeaMcNbd\nJ3AoO9fbVTDGmDrFp0LgcI6FgDHGeJKFgDHG1GE+FQIZuRYCxhjjST4VApl5FgLGGONJFgLGGFOH\n+VQIZOVbCBhjjCf5VAhkH7EQMMYYT7IQMMaYOsynQiCnwELAGGM8ya0QEJFhIrJVRLaLyIOVlJku\nIjtEZL2I9Cozf4+IbBCRdSLyw8m2k3vUQsAYYzypygHkRMQPmAEMBvYBq0VkgapuLVMmDohR1U4i\nchEwE+jjersYiFXVQ1VtK6/QQsAYYzzJnZZAb2CHqiapagHwITCyXJmRwDwAVV0FNBORUNd74uZ2\nLASMMcbD3Dk4hwPJZaZTXPNOVmZvmTIKLBOR1SJy58k2lF9kIWCMMZ7kiecJ9FfV/SLSCicMtqjq\n1xUV3P/NN0ydOhWA2NhYYmNjPVA9Y4w5MyQkJJCQkFCj66zyeQIi0geYqqrDXNOTAVXVZ8uUmQV8\nqarzXdNbgUtUNa3cuqYAWar6YgXb0fB7byTl5Q9P9zMZY0yd4KnnCawGOopItIjUB0YB8eXKxANj\nXJXqAxxW1TQRCRSRINf8xsBlwP8q29BRtdNBxhjjSVWeDlLVIhGZCCzFCY23VHWLiExw3tY3VHWR\niAwXkUQgBxjrWjwU+FRE1LWt91R1aWXbKsBCwBhjPMmnHi8ZdG9fsl7+1ttVMcaYM8JZ93jJQsnz\ndhWMMaZO8akQKPKz00HGGONJPhUCxRYCxhjjUb4VAvUsBIwxxpN8KgTU30LAGGM8yadCAP8jFBUX\ne7sWxhhTZ/hWCBQ04nC2XSFkjDGe4lMhIIWBHMyyU0LGGOMpvhUCRYEcyrYQMMYYT/GpEKhXFMih\nHAsBY4zxFN8KgeJADlsIGGOMx/hUCPhrIBm5FgLGGOMpvhUCBJJpIWCMMR7jUyEQQCMy8y0EjDHG\nU3wqBOpLIFkWAsYY4zG+FQJ+gWRbCBhjjMf4VAg09Ask+6iFgDHGeIpvhUC9QHIsBIwxxmN8LgRy\nCywEjDHGU3wqBAIDAsmzEDDGGI/xvRAoshAwxhhP8a0QqB9IvoWAMcZ4jE+FQFD9QI5YCBhjjMe4\nFQIiMkxEtorIdhF5sJIy00Vkh4isF5Hzyr3nJyJrRST+ZNsJahDIEbUQMMYYT6kyBETED5gBXA50\nB0aLSJdyZeKAGFXtBEwAZpVbzb3A5qq21aRhIEeL7clixhjjKe60BHoDO1Q1SVULgA+BkeXKjATm\nAajqKqCZiIQCiEgEMBx4s6oNNW0USAHWEjDGGE9xJwTCgeQy0ymueScrs7dMmZeABwCtakPNAgMp\nFAsBY4zxFP/aXLmIjADSVHW9iMQCcrLySz9+k7xdKUydOpXY2FhiY2Nrs3rGGHNGSUhIICEhoUbX\nKaon/4IuIn2Aqao6zDU9GVBVfbZMmVnAl6o63zW9FbgEpy/gFqAQaAQ0AT5R1TEVbEeXfLOX3y28\nkIKn99XIhzPGmLOZiKCqJ/1yXRV3TgetBjqKSLSI1AdGAeWv8okHxrgq1Qc4rKppqvqwqkapagfX\ncssrCoASzRsHUuxnp4OMMcZTqjwdpKpFIjIRWIoTGm+p6hYRmeC8rW+o6iIRGS4iiUAOMPZUKhMc\nFEhxPQsBY4zxlCpPB3mKiOj+/cW0mRnA0cfyCKgX4O0qGWOMT/PU6SCPCQwUKAgkr9DuFTDGGE/w\nqRBo1AgoaGTDSRtjjIf4VAgEBAAFgWTkWggYY4wn+FQIAPgVBXIo20LAGGM8wULAGGPqMJ8LgXrF\ndjrIGGM8xedCwF8DOWwhYIwxHuF7IUAgmRYCxhjjET4XAgEEkplnIWCMMZ7gcyFQXwLJys/xdjWM\nMaZO8LkQCCwOJS13v7erYYwxdYLPhUCwdiQld6e3q2GMMXWCz4VAiMSwP99CwBhjPMHnQqBVvRjS\nChK9XQ1jjKkTfC4EIoJDKdB8Ducf9nZVjDHmrOdzIRAdJQQVxLDzoJ0SMsaY2uZzIRAVBQGZHdl5\nyELAGGNqm8+FQGQkFB6wloAxxniCz4VAVBRk/xzDjoPWOWyMMbXN50KgWTOol9GR7QesJWCMMbXN\n50IAIKJxDIl2OsgYY2qdT4ZATMtIDuYfIK/AHjhvjDG1ySdDIDqqHsF+0ew+vNvbVTHGmLOaWyEg\nIsNEZKuIbBeRByspM11EdojIehE5zzWvgYisEpF1IrJRRKa4s72oKAg6alcIGWNMbasyBETED5gB\nXA50B0aLSJdyZeKAGFXtBEwAZgGo6hHgUlXtBZwHxIlI76q2GRUF/pkdSbQrhIwxpla50xLoDexQ\n1SRVLQA+BEaWKzMSmAegqquAZiIS6poueUJMA8Af0Ko2GBUFBWkxdsOYMcbUMndCIBxILjOd4pp3\nsjJ7S8qIiJ+IrANSgWWqurqqDUZFQdbPdtewMcbUNv/a3oCqFgO9RKQp8G8R6aaqmysqO3XqVACK\niuDglg4k/mqng4wxpkRCQgIJCQk1uk5RPfnZGRHpA0xV1WGu6cmAquqzZcrMAr5U1fmu6a3AJaqa\nVm5djwI5qvpiBdvRsnUJjzrCr+Obkf1wNv5+tZ5VxhhzxhERVFVOZx3unA5aDXQUkWgRqQ+MAuLL\nlYkHxrgq1Qc4rKppItJSRJq55jcChgJb3alYdEQDmgeEkpyRXHVhY4wxp6TKr9iqWiQiE4GlOKHx\nlqpuEZEJztv6hqouEpHhIpII5ABjXYu3Aea6rjDyA+ar6iJ3KhYVBRkSQ+LBRNoHtz+Vz2aMMaYK\nbp1nUdUlQOdy814vNz2xguU2AuefSsWiomDXEadzeChDT2UVxhhjquCTdwyDEwL1MuyGMWOMqU0+\nHQJH02JIPGRXCBljTG3x6RDI2fFbvkr6iswjmd6ujjHGnJV8OgRSt0ZzecfLmbl6prerY4wxZyWf\nDYHgYCgshHt6Tual71+yYaWNMaYW+GwIiDjPG26a9xsuiriIt9e97e0qGWPMWcdnQwCcU0I//wwP\nD3iY5759joKiAm9XyRhjzipnRAhcFHERnUI68f7G971dJWOMOaucESEA8PDAh3n666cpKi7ybqWM\nMeYs4vMhkOwaOujSdpcSGhTKP9b+w7uVMsaYs4hPh0B0NOza5fwsIrw6/FUe+/Ix0rLTTr6gMcYY\nt1Q5lLSnlB9KGiAzE9q2hfR0aNjQmTf5i8mkZKbw7jXveqGWxhjjOzw1lLTXNG0KXbrA6jLPInv0\n4kf5+uev+e+u/3qvYsYYc5bw6RAAuOQSWLHi2HTj+o15Je4V7vrPXeQX5nuvYsYYcxY440IA4IrO\nV9C1VVcbTsIYY06Tz4fAwIHw/fdQUO4+sXt+ew/zN833TqWMMeYs4fMhEBwMMTGwZs3x82PbxbL9\n1+3szdzrnYoZY8xZwOdDACo+JVS/Xn1GnDOCf2/9t3cqZYwxZ4EzNgQArulyDZ9u/dTzFTLGmLOE\nT98nUCI93Tkl9Ouv4F/mqci5BbmEvRDG7nt30yKwhYdqaowxvuGsv0+gRMuWzrDS69YdPz8wIJAh\nHYbw2fbPvFMxY4w5w50RIQCVnxK6usvVx50SSstO49K5l7IqZZUHa2eMMWemMz4EfnfO71i+ezk5\nR3M4mHeQy969jKD6Qdzy6S1kH832fEWNMeYM4lYIiMgwEdkqIttF5MFKykwXkR0isl5EznPNixCR\n5SKySUQ2isifTrWiF18MX38NReVGkg5uFEyfiD58tPkj4t6LY2iHocSPiqd/ZH/+8vlfTnVzxhhT\nJ1QZAiLiB8wALge6A6NFpEu5MnFAjKp2AiYAs1xvFQJ/VtXuQF/gnvLLuisszBlMrvz9AgDXdL2G\nO+Lv4Pyw83l+6POICNPjprN011IWblt4Kpszxpg6wZ2WQG9gh6omqWoB8CEwslyZkcA8AFVdBTQT\nkVBVTVXV9a752cAWIPxUK3vllbBgwYnzb+x+I88OeZZXR7yKiNNR3rRBU+ZdNY/xn42vdOjp3ILc\nU62KMcacFdwJgXAgucx0CiceyMuX2Vu+jIi0A84DTrnHduTIikMguFEwf+n3F/zk+I8zMHogd194\nN91e68bERRNZu38tBUUFLNi6gKs+vIpmzzRj0Y5Fp1odY4w54/lXXeT0iUgQ8DFwr6tFUKGpU6eW\n/hwbG0tsbOxx7/fuDQcPQmIidOzo3rYfveRRxvQcw9wNc7n2n9dyIOcA54Wdx7he44jrGMeL373I\n8E7Dq/+hjDHGwxISEkhISKjRdVZ5s5iI9AGmquow1/RkQFX12TJlZgFfqup81/RW4BJVTRMRf+Az\nYLGqTjvJdiq9Ways8eOhc2f4yyn0+RZrMQdyDhAaFArA0aKjtJ/WniU3L+E3ob+p/gqNMcaLPHWz\n2Gqgo4hEi0h9YBQQX65MPDDGVak+wGFVLTkR/zaw+WQBUB2VnRJyh5/4lQYAOOMP3X3h3UxbVSNV\nM8aYM45bw0aIyDBgGk5ovKWqz4jIBJwWwRuuMjOAYUAOcJuqrhOR/sBKYCOgrtfDqrqkgm241RLI\nz4fQUNi507mT+HQdyDnAOTPOYfvE7bRq3Or0V2iMMR5SEy2BM2LsoPKuvRauuAJuu61mtn1n/J1E\nN4/mrxf/tWZWaIwxHlBnxg4q73ROCVXk3j738trq1zhSeKTmVmqMMWeAMzIERoyA5cshL69m1ndu\n63Pp3ro7czfMrZkVGmPMGeKMDIEWLaBXL1i2rObW+czgZ5iSMIUnVjxBsRaXzs85msPc9XPtCWbG\nmLPSGRkCADfdBG+8UXPru6DtBfw4/keW7VrG8PeGk3gwkSdXPkn7ae15a91bDJw9kJ0Hdx63jKqS\nkplyWtstLC4kNTv1tNZhjDGn6owNgTFjYO1a+Omnmltn2yZtWf775fQM7Um3V7uReDCRFbetYOXY\nlfxf//8jdm4sWw5sAeDHfT9y8ZyLiZkew18+/wsFRQWntM2/r/w757xyDp8nfl5zH8QYY9x0Rl4d\nVOK552DDBnjvvZqvT2FxIf5+x99QPW/DPB784kEGtx/Mf3f/lycufYKRnUcydsFYDuUfYv5184lo\nGuH2NtJz0+kyowsvD3uZB5Y9wNRLpjLhwgkAFBUXsWbfGsKCwohuHl2jn80Yc3aos5eIlsjMhA4d\n4IcfnH89YeG2haxLXcekPpNo2qAp4NyJ/Nw3zzFt1TRevOxFRp07qnQguxKqesK8+5feT15BHq+O\neJXEg4mMeH8EAyIHkFOQw7Jdy2jbpC37s/aXDnNxdZeraRTQqFr1rWi7xpizQ50PAYCHH4aMDHj1\n1VqoVDV9m/wtf1z8RxrUa8DLw17mvLDz+M/2//D2+rf55udvePead0vHKUrJTKHHzB5sunsTbZq0\nAeBg3kGe+uopurbsyrCOwwhvGk5+YT4Lti5g9vrZrN63mhu63cC4XuO4sO2FZBzJYNnOZXy+83OG\ndRzGdd2uO64+29K3MeSdIcSPiqdXm14e3x/GmNplIQCkpUHXrrBli3MnsbcVazHzNszjkeWPkF+Y\nT/dW3RnXaxxRzaIY/a/RvDr8Va7rdh3jF44nuGEwzw59tuqVuiRnJDNvwzxmr59NQXEBh/IOMSBq\nALHtYnnp+5d4dfirXNP1GsB5zGbft/oSExJD84bN+ej6j2rrIxtjvMRCwOWeeyArCy68EHbvhtRU\nmDYNWreu4UpWQ/bRbH7N/fW48/kbUjcQ914cf7jwD0xfNZ3tf9xOSKOQaq9bVdl0YBMxwTGlp4fW\n7l/LsHeHMe/qeQyMGsilcy8lrmMc/9f//+gwvQMrbltBl5an9DyfM8qUL6fQuH5j7u93/wlDixtz\ntrEQcElOhvvvh1atoF075/6BuDiYNKlm61gTtqZvZfC8wdx94d08cvEjNbru75K/Y+SHI+ncsjMd\ngjswZ+QcRIQnVz5J4sFE5lw1p9rr/CntJ9776T32Zu0lNTuVA7kHGN5xOJP6TDpuMD53qSp7s/ZW\nqwPdXf/Z/h/uXnQ34U3CaRHYgrlXzT2lkDXmTGEhUImlS+Gvf3U6jH1R9tFsAgMCa+Wb6pe7v+Td\nn95l5u9mUr9efQAO5R2i4ysd+XH8j7Rr3s6t9XyX/B1Pff0Ua/at4c7z76RTSCfCgsJo0qAJ72x4\nhw/+9wE3/+ZmJg+YTHjT458xpKrMXj+biKYRDO0wtLRjOi07jbv+cxcLti3gkxs+YWSX8g+oO3UH\ncg7Qc1ZPPrj2A/pF9uPBLx7k062f8tjFj7H78G7Wpa5j96HdPB77ONd2u7bGtmuMN1kIVKKwEMLD\n4Ztv3H/4zNlu8heTyT6azYzhMwA4nH+YtOw0OrfsfELZB5Y+wEebP+LB/g9y23m3VXhFUmp2Kv/v\n2//Hexvf49MbP+WiiIsAp09k0pJJJOxJoEiLaOTfiIcHPsyRwiNM+nwSt/e6nRGdRnDV/Kv4bPRn\npcudDlXl6vlX07lF5+P6WD7Z8gnv/PQO3Vt1p1dYL5o0aMKEzyYwsvNInhv6XGlIGnOmqokQQFV9\n4uVUpeZMnKj6t7/V6CrPaKlZqRr8TLB+vOljvelfN2mzp5tp8DPBumznsuPKfbL5E233cjs9mHvQ\nrfUu3LZQWz3XSj/a9JEWFhXq2H+P1X5v9dPDeYe1qLhIP93yqfb+R2/tMbOH/pDyw3HLhb0Qpom/\nJlb7s6zeu1r/sPAP+sqqV/TrpK/1lVWvaM+ZPTW/IL/KZQ/mHtQr3r9C+7zZR1MyUqq97RKFRYXa\n/63+mrA74ZTXYczpch03T+vYe1a2BAC+/RZuvx02bwa7TN7x4LIHWbprKWPPG8tNv7mJzQc2c90/\nr+OLMV/QI7QHuw/t5qI3L+Kzmz6jd3hvt9e7bv86rvzwSsKCwmjaoCkLRi0gqH5QlcvNWjOLF797\nkW/GfeP2sxx2HtzJgNkDuPP8O0nLTmNd6jr2Z+9nyc1L6N66u1vrKNZinlz5JPM3zeebcd/QvGFz\nt5YrK36Q1TAkAAAXJElEQVRbPLfH306LRi3Y8IcNNPBvUO11GHO6rCVwEsXFqu3aqa5bV6OrPet8\nsPEDjXwxUnce3Km/feO3+tJ3L53SepIzknXKl1M0ryCvWss9uvxR7fZqN92Xue+4+XkFefrRpo/0\nUN6h0nkHcg5op+mddNbqWadUx/LuXXyvxs6JdasFUd6lcy7V9356T3/3/u/0yRVPnvD+wdyDWlBU\nUBPVPKni4mItLCo8rXXkHM05pX1Q0yr729l5cKd++/O3VS77eeLnunbf2tqoWq3KL8jXqV9O1ee/\neV6LiouqtSzWEji5hx+GoiJ41v1L8eukZ79+lr+t/BtDOwzl0xs/9fgdxk999RSz18/mi1u/ILp5\nNKtSVjF2wVgCAwLZfXg3488fzx8u/AM3fXITF0ddzNNDnq6R7RYVF3H9R9cTGBDIO1e/g4hQrMUk\nZySTX5hfWq59cPvj+g82pG5gxPsj2H3vbvZm7eWCNy7ghzt+ICYkhiOFR/jz538uvZcjpFEIEU0j\neCXuFfpF9jtpffIL8wnwC6CeXz236r/91+2MXTCWnQd3Mut3s7iqy1Vuf+4t6Vv4PPFzFicu5vuU\n7ynWYs5pcQ692vSiU0in0osWGvo3ZPwF4wkMCDzpOvdl7aNBvQaENAo5pb+fN9e+yb1L7uW9a947\n7nOs3b+WEe+PIMAvgAvaXsDzQ5+nY4jT0bfr0C4W71jM4sTFrExaybmtz2XHwR0suXkJF7S9oNp1\n8IbVe1dz24Lb6NyiMwdyDxBUP4h3rn6HloHHHpuYkZ9Bs4bNKlzeOoarsHGj8+yBPXvAzy4Zr5Sq\nMnfDXEZ2Hklwo2Cv1GHa99N48fsXuarzVczfNJ9pw6ZxQ/cbSMpI4vlvnmf2+tlc1eUq3r3m3Rq9\nqiq3IJfB8wYT0iiEzCOZbEjdQNMGTWlcvzHgHJjbN2/P57d8XnrKZ9yCcXQK6cRDAx8CnBBNSErg\nteGvccPHNxDZNJLZI2fTuH5jDuQcYEXSCiYtmcTKsSs5p8U5FdZj84HNxM6JJaRRCA/2f5Cbe9xc\nacd1UXER01ZN46mvnmLKJVPoGdaT2+Nvp3d4b6YNm0ZKZgqLdyxm6a6l5BfmExYURpugNqgqG9I2\nsPGXjYQ2DmVIhyHEdYxjUPtB+Pv5879f/sf61PXsOrQLxfm/uPGXjWQdyeKzmz4rHSalpA5f7PqC\nRTsWsThxMQfzDlKkReQW5BLaOJQrO1/J04OfpkmDJlX+DhbvWMzYBWOZNmwaf176Z+7vez+T+kzi\n+5TvuWr+VcwaMYu4TnG8/P3LvPDtCwzuMJj1qevJyM9gWMdhxHWMY2jMUEIahfDplk+ZuHgi34z7\npsIr4ZIOJ/H8t8+z+cBmLou5jLiOcfQI7eHxLz57Du/hlVWv8O7Gd5k2bBo3dr+RwuJCHln+CPM3\nzeeJS59gfep6FicuZvuv23lh6Avc1/e+E9ZjIeCG3/wGXnsNBg6s8VWbGjZ73Wy+3PMlL1z2Aq0b\nH3+n36G8QzRp0OSEQf1qQnpuOvP/N58uLbtwXth5tAhsUfpesRZz/UfX09C/Ie9e/S4Hcg/QeUZn\nEv+YWFquoKiAXq/3IiUzhSmXTGFSn0knHFTeXPsmz3z9DN/e/u0Jn23nwZ1cMucSnhnyDOFNwnnq\n66fYlr6N0eeOJrxpOG2C2hBUP4jNBzazLnUdq/auom2Ttrx95dvEhMQATpg98t9HeHX1q0Q1iyKu\nYxzDOg6jecPm7M/eT2p2KsVaTM/QnvQM6+l2P0ixFnP3f+5m7f61LLllCSGNQli+ezn3fX4fAX4B\nXNP1GuI6xtEzrCd+4kd+YT57M/fy96/+zpd7vuStK99iUPtBla6/5CbHBaMW0DeyLz9n/MyI90fQ\ntWVXEvYkMO/qeQzrOKy0fGp2Kv/a/C/6RvblvLDzKvxCMO37abz+4+t8M+4bghsFc6TwCBvSNvDa\n6tdYuH0hd55/J/0j+7Ns1zIWJy7maNFRXr78Za7uerVb++RUqCr7s/ezfPdyZq+fzYbUDdz0m5t4\nZOAjJ9xvE78tnhk/zGBA1ACGdxpOi0Yt6Pd2P965+h2GdBhyXFkLATe88QY89BCMHg133gk9e9b4\nJsxZLq8gj0HzBjGo3SAa+Ddgb+ZeXr/i9ePKbEvfRvbR7JOehnh0+aMs27WM5b9fXnp6JTkjmYvn\nXMzk/pNLR5AF+GHvDyzduZS07DT2Z+8n40gGXVt2pVdYL3q16UWP0B4VHgCzjmS59e27OlSVB5Y9\nwNKdS4kJiWF96nqeH/o813a99qTfoBftWMSEzybw27a/LT29IQghjUIICwqjRWALJn8xmWnDph13\n70bmkUzuW3IfY3qO4ZJ2l5xSne9bch9Ldy0lwC+Abb9uIyY4hhu738jE3hNPaO1+lfQV4+LH8du2\nv+WVuFeO+xJQIrcglzX71hBUP4iwoDBaN25d5ReSfVn7eGfDO3y550vWpa6jqLiIvpF9ubXHrVzZ\n+Uoa+jd0+/Mk7Elg1Mej+Pb2b+kQfGy0TAsBNyUnw9tvw1tvQcOGUK/esUdTLlzotBaMOZkDOQfo\n93Y/9mXtY/Wdq+nWqlu116GqjF0wln9v/XfpN/zEg4nce9G9FTb1fYmqMuOHGeQV5vGni/7k9gHs\ncP5hPtnyCYXFhYDTsjiYd5D9WfvZn72f4Z2GM67XuBqvb1FxEQu2LSCiaQS/af2bKkffLWlJzd80\nn6u6XFV6Ci2nIIcliUv4JvkburXqxpHCI6Rmp/Jr3q8E+AWULh9UP4geoT04L+w8OgR34LPtn/F9\nyvdc1+06rjjnCnq16UV4k/DTOu0044cZvP7j63x3+3elV99ZCFRTUZFzyWhAgBMGM2c6YTB9eq1u\n1pwlEg8m8vHmj5k8YPIpr0NVSc9NJzU7lf3Z+/H38z/p6RLjWWv2rWFVyqrSU2j1pB6XxVzGkA5D\njuucLSwu5GjR0dLpw/mHWZ+6nvWp69n+63aGdBjCNV2vqbJDvTpUlTvi76B9cHv+evFfAQuB07Zz\nJ/TtC3v3OsFgjDG+7EjhEer51Ss9FVUTIeDWZRYiMkxEtorIdhF5sJIy00Vkh4isF5FeZea/JSJp\nIlKDD4KsGTEx0KkTfG5PdjTGnAEa+Deo8YsjqgwBEfEDZgCXA92B0SLSpVyZOCBGVTsBE4CZZd6e\n7VrWJ916K8yb5+1aGGOMd7jTEugN7FDVJFUtAD4Eyg//OBKYB6Cqq4BmIhLqmv4aOFRzVa5ZN9zg\ntAQOH/Z2TYwxxvPcCYFwILnMdIpr3snK7K2gjE8KCYEhQ+Djj71dE2OM8byav/PmNEydOrX059jY\nWGJjYz2y3VtvhZdegjvu8MjmjDHmlCQkJJCQkFCj66zy6iAR6QNMVdVhrunJOIMWPVumzCzgS1Wd\n75reClyiqmmu6Whgoar2OMl2PH51UImjR6FtW1izxnkymTHGnAk8dXXQaqCjiESLSH1gFBBfrkw8\nMMZVqT7A4ZIAKKmr6+WT6teHG290nkaWne3t2hhjjOdUGQKqWgRMBJYCm4APVXWLiEwQkfGuMouA\n3SKSCLwO3F2yvIi8D3wLnCMiP4vI2Fr4HKftb39znjvQrRv861+g6txc9tNP8M47cMhnu7aNMebU\n1embxSqyciXc7Yqw5GQIC4PQUGeoiaVL7aYyY4zvsDuGa0lBAXz3ndMqaNnSaRFcfbXTbzBzpj2p\nzBjjGywEPCgrC/r1gwkTYOJEb9fGGGNqJgR86hJRX9akCcTHO0EQFQVXXuntGhljzOmz521VQ/v2\nTqfxxIlwyy2wf7+3a2SMMafHQqCa+vWDLVsgIsJ5DsEzz8APP0BOjrdrZowx1WchcAoaN3YO/l99\nBdu3O/0ErVo5I5LOmePt2hljjPusY7iGFBbC2rUwZgwMGgQvv+zchAaQmgrLl8NFFznDV5dQhcRE\n52qjjh29U29jzJnLrg7yQRkZThCkpzt9Bx984LQYBgyA1audAet+9zvn5rNly5zLUY8edTqd+/b1\ndu2NMWcSCwEfVVwMTz8NCQlw001w/fUQFOTMX70aFi1ywmDoUOja1RnK+ve/h8WL4fzzvV17Y8yZ\nwkLgLPLpp3DXXfDf/0L37t6ujTHmTGD3CZxFrr7aucJo0CDnHoS+fZ1X587gZ933xphaYi0BH7Nx\no3Ma6bvvnFdGhtOh3Lev8/Cbvn1t2ApjjMNOB9UBqalOGHz/PXzyCbRoAQ88AFdd5QxqdzJHj8Lb\nb8OoUdC8uWfqa4zxHAuBOqaoCBYsgOeeg7Q05wE4DRs6rwEDnPsVgoKcsjt2wOjRkJnpdEIvXQpN\nm3q1+saYGmYhUEepwv/+51yGmp/v9CV89JFzL8I990BkJEyeDFOmOMNiT5wIGzbAkiXOGEjGmLOD\nhYA5zvbt8OyzsGkTvP469OzpzC8udq482rwZZs1ynp526JBzM9sll1R8WunoUdi3z3mmQmGhU846\nqI3xLRYCxm3FxXDvvc49CcHBTh/BwYNOn8O4cXDrrZCU5NyrsHixczqpTRtnjKScHCcUJk92TjEF\nBDitkbQ02LvXaZGkpzvhEhHhnKZq184ZXqOievz4o/NshrZtrZPbmNNhIWBO208/wT/+AR9+6Axp\nERfnvC644FgLQdW5f+Gpp5xwaNzYCYwmTSA83Bk3qWVLCAyElBTYswd+/tlZzxNPOA/nAVi3zjld\n9csvTmDk5kKXLvCXvzjPeDbGVI+FgPG4jRud00LR0cc6oSuSmwuvveZ0YsfFOYHx0Ufw9787LQ8/\nP6clsmaN02cxcCBMn15x6+HIEWfk1qws8Pd3Xs2bQ4cOVV8hZczZzELA+LyMDGcwvUOH4NFHnUtc\ny8vKcoLghx/g4YedZfbvd/ojfvrJ6eto3965yqmw0Hmlpzstii5dnKE3/P2PvRcc7JyOat/eCZ+f\nf4bdu53yY8c6gWPM2cBCwJxV3nvPGT4jNBTCwpw+gx494NxzoVGjE8tnZTmd3du2OX0N/v5Oy+Dg\nQeegv2ePc4lsdLQTCoGBMG2as86nn3aG5zh8+FhA1Kvn9Hf4+zstkqAg59Wq1fEtDlWnntOmwYgR\nMGnSsRFjjfEkCwFjqik/H2bOdEKgoMBpObRv7wRPcbEzXVDgnM7KynJeRUXOYH9xcU5r5PHHnXn3\n3+/0pWzf7rR2hgxxft640QmV/v2hVy/3rqoqGYr8l1+cOubnOx3zsbF2ystUzmMhICLDgJdxHkLz\nlqo+W0GZ6UAckAPcpqrr3V3WVc5CwHhMTo5zoA0JqfoKpZQU5x6LxYudU0sPPADXXXfs4L5okdMa\n+Plnp9Vx7rlOR/nKlXDgAFx6qdPCKLmKyt/feX5Ep05O38bKlc5QIZGRzvOrS24A3LLFuXrr5pvh\niiucq7G2bIGtW53Ld0vWV1joXJUVGXn8vxERzg2CJf0oDRs622vWzJlWhbw8p5M+IMApWzZwioud\nq8IaNqy1X4M5TR4JARHxA7YDg4F9wGpglKpuLVMmDpioqiNE5CJgmqr2cWfZMuuwEHBJSEggNjbW\n29XwujNpP5S0IMqftkpOhhUrnKBp2dJ5HT3qPExoxw7nIN6/v9OKCA09cb2bN8M778C//pVAt26x\ndO3q9INERBy7KsvPz7lUNznZeaWkOK/kZCfsSvpK8vKc01+ZmdCggROC9es7p7wKCpxWT1CQc9DP\nyXFaQwEBzryS0Cr5t1Mnp6WSm+uESMkrK8v5t6joWPjUq3d80Nar57z8/Z1TdM2bO68WLZxXVaFc\n1d+FqrM/UlKc+rZsWXG54mKn/oWF1R9WJSXFqXtISPWWq2meGkW0N7BDVZNcG/0QGAmUPZCPBOYB\nqOoqEWkmIqFAezeWNeWcSQe/2nQm7YeSA155kZFwyy0nznf3AULdujmnrho0SGDq1NhKy4WFOZf1\nuqPk4New4fF1LipyDuJ5ec6Bv3Fj54B84IATWDt2OOG1cKHzc1rasb6Txo2dTviSfpSyHfWFhce2\noXrstFthoVOPw4ed14EDThi1a+e0qOBYuBQVHWshJScn0LVrbGnAqB5b34EDTkupcWPn8uWdO51g\n7t7dKXvggBO8hw45227UyPmM4eHOKb+hQ53PsX+/0worCcagICdsV62CL790gvTIEeey6thYZ983\naOCEpp+fs9zhw852CguP/X0cOeLUaccOp8+qWbNjLbcWLZzP16iR829Q0LF9Wr/+sUAtueihprgT\nAuFAcpnpFJxgqKpMuJvLGmM8yM+v4st769U79q28rNatnVf//rVft4wM5x6UpKRj9Sw5AB854rRe\n3nzTGRSx5MDv53fsIBsc7LSUgoOd9ak639o3bTq+NRYcfGy9xcWwfr3zpL8ZM5yWWliY8woKcloV\n2dnO/AsucE79de/ubHv1aicU/vOfY31MRUXOwbtkX9av79S7JAwuvRTGj3cO5pmZx1pwhw87AZyb\n6wRVTs6x1lXJugsLnRs2J0youX1eW88TsPtAjTHV1qyZc/VWjx6Vl0lIcB7R6g4R55t2ZGTlZfz8\nnCf6nX8+PPig+3UNCIB+/ZzX6Si5mdJb3OkT6ANMVdVhrunJgJbt4BWRWcCXqjrfNb0VuATndNBJ\nly2zDusQMMaYavJEn8BqoKOIRAP7gVHA6HJl4oF7gPmu0Disqmkiku7GssDpfxBjjDHVV2UIqGqR\niEwElnLsMs8tIjLBeVvfUNVFIjJcRBJxLhEde7Jla+3TGGOMqRafuVnMGGOM53l9hHgRGSYiW0Vk\nu4hUo1vmzCciESKyXEQ2ichGEfmTa36wiCwVkW0i8rmINPN2XT1FRPxEZK2IxLum6+S+cF1m/ZGI\nbHH9fVxUh/fFfSLyPxH5SUTeE5H6dWVfiMhbIpImIj+VmVfpZxeRh0Rkh+vv5jJ3tuHVEHDdTDYD\nuBzoDowWkS7erJOHFQJ/VtXuQF/gHtfnnwx8oaqdgeXAQ16so6fdC2wuM11X98U0YJGqdgV64txb\nU+f2hYi0Bf4InK+qPXBOYY+m7uyL2TjHx7Iq/Owi0g24AeiKM3rDayJVP7HD2y2B0hvRVLUAKLmZ\nrE5Q1dSS4TVUNRvYAkTg7IO5rmJzgau8U0PPEpEIYDjwZpnZdW5fiEhTYKCqzgZQ1UJVzaAO7guX\nekBjEfEHGgF7qSP7QlW/Bg6Vm13ZZ78S+ND197IH2IEb92V5OwQqu8mszhGRdsB5wPdAqKqmgRMU\nQGvv1cyjXgIeAMp2VNXFfdEeSBeR2a5TY2+ISCB1cF+o6j7g/wE/4xz8M1T1C+rgviijdSWfvfzx\ndC9uHE+9HQIGEJEg4GPgXleLoHxv/Vnfey8iI4A0V8voZE3Ys35f4JzyOB94VVXPx7nibjJ18++i\nOc4332igLU6L4Gbq4L44idP67N4Ogb1AVJnpCNe8OsPVxP0YeEdVF7hmp7nGXkJEwoBfvFU/D+oP\nXCkiu4APgEEi8g6QWgf3RQqQrKprXNP/wgmFuvh3MQTYpaoHVbUI+BToR93cFyUq++x7gbL3Rrt1\nPPV2CJTeiCYi9XFuJov3cp087W1gs6pOKzMvHrjN9fPvgQXlFzrbqOrDqhqlqh1w/g6Wq+qtwELq\n3r5IA5JF5BzXrMHAJurg3wXOaaA+ItLQ1ck5GOfCgbq0L4TjW8eVffZ4YJTr6qn2QEfghypX7u37\nBFzPG5jGsZvJnvFqhTxIRPoDK4GNOE06BR7G+cX9EyfVk4AbVPWwt+rpaSJyCfAXVb1SREKog/tC\nRHridJAHALtwbsCsR93cF1NwvhgUAOuAO4Am1IF9ISLvA7FACyANmAL8G/iICj67iDwE3I6zr+5V\n1aVVbsPbIWCMMcZ7vH06yBhjjBdZCBhjTB1mIWCMMXWYhYAxxtRhFgLGGFOHWQgYY0wdZiFgjDF1\nmIWAMcbUYf8fNq5YLeNas8MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f545e874ef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = np.array(errors)\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 0], label='train error')\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 1], label='test  error')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network\n",
    "\n",
    "\n",
    "In machine learning, a convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing. (Wikipedia)\n",
    "\n",
    "\n",
    "![a](d7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Motivation\n",
    "\n",
    "Convolutional Neural Networks (CNN) are biologically-inspired variants of MLPs.\n",
    "From Hubel and Wiesel's early work on the cat's visual cortex, we\n",
    "know the visual cortex contains a complex arrangement of cells. These cells are\n",
    "sensitive to small sub-regions of the visual field, called a *receptive\n",
    "field*. The sub-regions are tiled to cover the entire visual field. These\n",
    "cells act as local filters over the input space and are well-suited to exploit\n",
    "the strong spatially local correlation present in natural images.\n",
    "\n",
    "Additionally, two basic cell types have been identified: Simple cells respond\n",
    "maximally to specific edge-like patterns within their receptive field. Complex\n",
    "cells have larger receptive fields and are locally invariant to the exact\n",
    "position of the pattern.\n",
    "\n",
    "The animal visual cortex being the most powerful visual processing system in\n",
    "existence, it seems natural to emulate its behavior. Hence, many\n",
    "neurally-inspired models can be found in the literature. \n",
    "\n",
    "### Sparse Connectivity\n",
    "\n",
    "\n",
    "CNNs exploit spatially-local correlation by enforcing a local connectivity\n",
    "pattern between neurons of adjacent layers. In other words, the inputs of\n",
    "hidden units in layer **m** are from a subset of units in layer **m-1**, units\n",
    "that have spatially contiguous receptive fields. We can illustrate this\n",
    "graphically as follows:\n",
    "\n",
    "![A](sparse_1D_nn.png)\n",
    "\n",
    "Imagine that layer **m-1** is the input retina. In the above figure, units in\n",
    "layer **m** have receptive fields of width 3 in the input retina and are thus\n",
    "only connected to 3 adjacent neurons in the retina layer. Units in layer\n",
    "**m+1** have a similar connectivity with the layer below. We say that their\n",
    "receptive field with respect to the layer below is also 3, but their receptive\n",
    "field with respect to the input is larger (5). Each unit is unresponsive to\n",
    "variations outside of its receptive field with respect to the retina. The\n",
    "architecture thus ensures that the learnt \"filters\" produce the strongest\n",
    "response to a spatially local input pattern.\n",
    "\n",
    "However, as shown above, stacking many such layers leads to (non-linear)\n",
    "\"filters\" that become increasingly \"global\" (i.e. responsive to a larger region\n",
    "of pixel space). For example, the unit in hidden layer **m+1** can encode a\n",
    "non-linear feature of width 5 (in terms of pixel space)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shared Weights\n",
    "\n",
    "\n",
    "In addition, in CNNs, each filter $h_i$ is replicated across the entire\n",
    "visual field. These replicated units share the same parameterization (weight\n",
    "vector and bias) and form a *feature map*.\n",
    "\n",
    "![](conv_1D_nn.png)\n",
    "\n",
    "In the above figure, we show 3 hidden units belonging to the same feature map.\n",
    "Weights of the same color are shared---constrained to be identical. Gradient\n",
    "descent can still be used to learn such shared parameters, with only a small\n",
    "change to the original algorithm. The gradient of a shared weight is simply the\n",
    "sum of the gradients of the parameters being shared.\n",
    "\n",
    "Replicating units in this way allows for features to be detected *regardless\n",
    "of their position in the visual field.* Additionally, weight sharing increases\n",
    "learning efficiency by greatly reducing the number of free parameters being\n",
    "learnt. The constraints on the model enable CNNs to achieve better\n",
    "generalization on vision problems.\n",
    "\n",
    "\n",
    "### Details and Notation\n",
    "\n",
    "\n",
    "A feature map is obtained by repeated application of a function across\n",
    "sub-regions of the entire image, in other words, by *convolution* of the\n",
    "input image with a linear filter, adding a bias term and then applying a\n",
    "non-linear function. If we denote the k-th feature map at a given layer as\n",
    "$h^k$, whose filters are determined by the weights $W^k$ and bias\n",
    "$b_k$, then the feature map $h^k$ is obtained as follows (for\n",
    "$tanh$ non-linearities):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "    h^k_{ij} = \\tanh ( (W^k * x)_{ij} + b_k ).\n",
    "$$\n",
    "\n",
    "Note\n",
    "\n",
    "* Recall the following definition of convolution for a 1D signal.\n",
    "$$  o[n] = f[n]*g[n] = \\sum_{u=-\\infty}^{\\infty} f[u] g[n-u] = \\sum_{u=-\\infty}^{\\infty} f[n-u] g[u]`.\n",
    "$$\n",
    "\n",
    "* This can be extended to 2D as follows:\n",
    "\n",
    "$$o[m,n] = f[m,n]*g[m,n] = \\sum_{u=-\\infty}^{\\infty} \\sum_{v=-\\infty}^{\\infty} f[u,v] g[m-u,n-v]`.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To form a richer representation of the data, each hidden layer is composed of\n",
    "*multiple* feature maps, $\\{h^{(k)}, k=0..K\\}$. The weights $W$ of\n",
    "a hidden layer can be represented in a 4D tensor containing elements for every\n",
    "combination of destination feature map, source feature map, source vertical\n",
    "position, and source horizontal position. The biases $b$ can be\n",
    "represented as a vector containing one element for every destination feature\n",
    "map. We illustrate this graphically as follows:\n",
    "\n",
    "**Figure 1**: example of a convolutional layer\n",
    "![](cnn_explained.png)\n",
    "\n",
    "\n",
    "The figure shows two layers of a CNN. **Layer m-1** contains four feature maps.\n",
    "**Hidden layer m** contains two feature maps ($h^0$ and $h^1$).\n",
    "Pixels (neuron outputs) in $h^0$ and $h^1$ (outlined as blue and\n",
    "red squares) are computed from pixels of layer (m-1) which fall within their\n",
    "2x2 receptive field in the layer below (shown as colored rectangles). Notice\n",
    "how the receptive field spans all four input feature maps. The weights\n",
    "$W^0$ and $W^1$ of $h^0$ and $h^1$ are thus 3D weight\n",
    "tensors. The leading dimension indexes the input feature maps, while the other\n",
    "two refer to the pixel coordinates.\n",
    "\n",
    "Putting it all together, $W^{kl}_{ij}$ denotes the weight connecting\n",
    "each pixel of the k-th feature map at layer m, with the pixel at coordinates\n",
    "(i,j) of the l-th feature map of layer (m-1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Convolution Operator\n",
    "\n",
    "\n",
    "ConvOp is the main workhorse for implementing a convolutional layer in Theano.\n",
    "ConvOp is used by ``theano.tensor.signal.conv2d``, which takes two symbolic inputs:\n",
    "\n",
    "\n",
    "* a 4D tensor corresponding to a mini-batch of input images. The shape of the\n",
    "  tensor is as follows: [mini-batch size, number of input feature maps, image\n",
    "  height, image width].\n",
    "\n",
    "* a 4D tensor corresponding to the weight matrix $W$. The shape of the\n",
    "  tensor is: [number of feature maps at layer m, number of feature maps at\n",
    "  layer m-1, filter height, filter width]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MaxPooling\n",
    "\n",
    "\n",
    "Another important concept of CNNs is *max-pooling,* which is a form of\n",
    "non-linear down-sampling. Max-pooling partitions the input image into\n",
    "a set of non-overlapping rectangles and, for each such sub-region, outputs the\n",
    "maximum value.\n",
    "\n",
    "Max-pooling is useful in vision for two reasons: \n",
    "* By eliminating non-maximal values, it reduces computation for upper layers.\n",
    "\n",
    "* It provides a form of translation invariance. Imagine\n",
    "     cascading a max-pooling layer with a convolutional layer. There are 8\n",
    "     directions in which one can translate the input image by a single pixel.\n",
    "     If max-pooling is done over a 2x2 region, 3 out of these 8 possible\n",
    "     configurations will produce exactly the same output at the convolutional\n",
    "     layer. For max-pooling over a 3x3 window, this jumps to 5/8.\n",
    "\n",
    "     Since it provides additional robustness to position, max-pooling is a\n",
    "     \"smart\" way of reducing the dimensionality of intermediate representations.\n",
    "\n",
    "Max-pooling is done in Theano by way of\n",
    "``theano.tensor.signal.downsample.max_pool_2d``. This function takes as input\n",
    "an N dimensional tensor (where N >= 2) and a downscaling factor and performs\n",
    "max-pooling over the 2 trailing dimensions of the tensor.\n",
    "\n",
    "\n",
    "### The Full Model: CovNet\n",
    "\n",
    "\n",
    "Sparse, convolutional layers and max-pooling are at the heart of the LeNet\n",
    "family of models. While the exact details of the model will vary greatly,\n",
    "the figure below shows a graphical depiction of a LeNet model.\n",
    "\n",
    "![](mylenet.png)\n",
    "\n",
    "\n",
    "The lower-layers are composed to alternating convolution and max-pooling\n",
    "layers. The upper-layers however are fully-connected and correspond to a\n",
    "traditional MLP (hidden layer + logistic regression). The input to the\n",
    "first fully-connected layer is the set of all features maps at the layer\n",
    "below.\n",
    "\n",
    "From an implementation point of view, this means lower-layers operate on 4D\n",
    "tensors. These are then flattened to a 2D matrix of rasterized feature maps,\n",
    "to be compatible with our previous MLP implementation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/al/anaconda3/lib/python3.5/site-packages/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.\n",
      "  \"downsample module has been moved to the theano.tensor.signal.pool module.\")\n"
     ]
    }
   ],
   "source": [
    "# from theano.tensor.nnet.conv import conv2d\n",
    "from theano.tensor.nnet import conv2d\n",
    "from theano.tensor.signal.downsample import max_pool_2d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify dropout function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def model(X, w, w2, w3, w4, w_o, p_drop_conv, p_drop_hidden):\n",
    "    l1a = rectify(conv2d(X, w, border_mode='full'))\n",
    "    l1 = max_pool_2d(l1a, (2, 2))\n",
    "    l1 = dropout(l1, p_drop_conv)\n",
    "\n",
    "    l2a = rectify(conv2d(l1, w2))\n",
    "    l2 = max_pool_2d(l2a, (2, 2))\n",
    "    l2 = dropout(l2, p_drop_conv)\n",
    "\n",
    "    l3a = rectify(conv2d(l2, w3))\n",
    "    l3b = max_pool_2d(l3a, (2, 2))\n",
    "    # convert from 4tensor to normal matrix\n",
    "    l3 = T.flatten(l3b, outdim=2)\n",
    "    l3 = dropout(l3, p_drop_conv)\n",
    "\n",
    "    l4 = rectify(T.dot(l3, w4))\n",
    "    l4 = dropout(l4, p_drop_hidden)\n",
    "\n",
    "    pyx = softmax(T.dot(l4, w_o))\n",
    "    return l1, l2, l3, l4, pyx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "reshape into conv 4tensor (b, c, 0, 1) format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train2 = X_train.reshape(-1, 1, 28, 28)\n",
    "X_test2 = X_test.reshape(-1, 1, 28, 28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# now 4tensor for conv instead of matrix\n",
    "X = T.ftensor4()\n",
    "Y = T.fmatrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "w = init_weights((32, 1, 3, 3))\n",
    "w2 = init_weights((64, 32, 3, 3))\n",
    "w3 = init_weights((128, 64, 3, 3))\n",
    "w4 = init_weights((128 * 3 * 3, 625))\n",
    "w_o = init_weights((625, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/al/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:3: UserWarning: pool_2d() will have the parameter ignore_border default value changed to True (currently False). To have consistent behavior with all Theano version, explicitly add the parameter ignore_border=True. On the GPU, using ignore_border=True is needed to use cuDNN. When using ignore_border=False and not using cuDNN, the only GPU combination supported is when `ds == st and padding == (0, 0) and mode == 'max'`. Otherwise, the convolution will be executed on CPU.\n",
      "  app.launch_new_instance()\n",
      "/home/al/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:7: UserWarning: pool_2d() will have the parameter ignore_border default value changed to True (currently False). To have consistent behavior with all Theano version, explicitly add the parameter ignore_border=True. On the GPU, using ignore_border=True is needed to use cuDNN. When using ignore_border=False and not using cuDNN, the only GPU combination supported is when `ds == st and padding == (0, 0) and mode == 'max'`. Otherwise, the convolution will be executed on CPU.\n",
      "/home/al/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:11: UserWarning: pool_2d() will have the parameter ignore_border default value changed to True (currently False). To have consistent behavior with all Theano version, explicitly add the parameter ignore_border=True. On the GPU, using ignore_border=True is needed to use cuDNN. When using ignore_border=False and not using cuDNN, the only GPU combination supported is when `ds == st and padding == (0, 0) and mode == 'max'`. Otherwise, the convolution will be executed on CPU.\n"
     ]
    }
   ],
   "source": [
    "noise_l1, noise_l2, noise_l3, noise_l4, noise_py_x = model(X, w, w2, w3, w4, w_o, 0.2, 0.5)\n",
    "l1, l2, l3, l4, py_x = model(X, w, w2, w3, w4, w_o, 0., 0.)\n",
    "y_x = T.argmax(py_x, axis=1)\n",
    "\n",
    "cost = T.mean(T.nnet.categorical_crossentropy(noise_py_x, Y))\n",
    "params = [w, w2, w3, w4, w_o]\n",
    "updates = RMSprop(cost, params, lr=0.001)\n",
    "\n",
    "train = theano.function(inputs=[X, Y], outputs=cost, updates=updates, allow_input_downcast=True)\n",
    "predict = theano.function(inputs=[X], outputs=y_x, allow_input_downcast=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 (0.066083333333333327, 0.058799999999999998)\n",
      "Current iter time:  3.966666666666667\n",
      "1 (0.029049999999999999, 0.025100000000000001)\n",
      "Current iter time:  2.3666666666666667\n",
      "2 (0.021183333333333332, 0.019400000000000001)\n",
      "Current iter time:  2.466666666666667\n",
      "3 (0.012483333333333334, 0.0132)\n",
      "Current iter time:  2.466666666666667\n",
      "4 (0.010183333333333334, 0.0117)\n",
      "Current iter time:  2.45\n",
      "5 (0.0080333333333333333, 0.010500000000000001)\n",
      "Current iter time:  2.4833333333333334\n",
      "6 (0.0078499999999999993, 0.010500000000000001)\n",
      "Current iter time:  2.3666666666666667\n",
      "7 (0.0064999999999999997, 0.0085000000000000006)\n",
      "Current iter time:  2.4833333333333334\n",
      "8 (0.0053166666666666666, 0.0088000000000000005)\n",
      "Current iter time:  2.466666666666667\n",
      "9 (0.0067166666666666668, 0.0092999999999999992)\n",
      "Current iter time:  2.5166666666666666\n",
      "10 (0.003966666666666667, 0.0080999999999999996)\n",
      "Current iter time:  2.5166666666666666\n",
      "11 (0.0043666666666666663, 0.0077999999999999996)\n",
      "Current iter time:  2.55\n",
      "12 (0.0029499999999999999, 0.0080000000000000002)\n",
      "Current iter time:  2.5833333333333335\n",
      "13 (0.0028166666666666665, 0.0077000000000000002)\n",
      "Current iter time:  2.45\n",
      "14 (0.0028500000000000001, 0.0071999999999999998)\n",
      "Current iter time:  2.466666666666667\n",
      "15 (0.0021666666666666666, 0.0067000000000000002)\n",
      "Current iter time:  2.4833333333333334\n",
      "16 (0.0020833333333333333, 0.0073000000000000001)\n",
      "Current iter time:  2.4\n",
      "17 (0.0017333333333333333, 0.0071999999999999998)\n",
      "Current iter time:  2.433333333333333\n",
      "18 (0.0020333333333333332, 0.0067999999999999996)\n",
      "Current iter time:  2.4166666666666665\n",
      "19 (0.0014833333333333332, 0.0068999999999999999)\n",
      "Current iter time:  2.35\n",
      "20 (0.0015, 0.0070000000000000001)\n",
      "Current iter time:  2.45\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-45-254bc5cccb43>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     17\u001b[0m     for start, end in zip(range(0, X_train.shape[0], 128), \n\u001b[0;32m     18\u001b[0m                           range(128, X_train.shape[0], 128)):\n\u001b[1;32m---> 19\u001b[1;33m         \u001b[0mcost\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train2\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_train2\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     20\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     21\u001b[0m     errors.append((np.mean(y_train != predict(X_train2)), \n",
      "\u001b[1;32m/home/al/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    857\u001b[0m         \u001b[0mt0_fn\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    858\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 859\u001b[1;33m             \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    860\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    861\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'position_of_error'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "t0 = dt.datetime.now()\n",
    "\n",
    "errors = []\n",
    "for i in range(100):\n",
    "    t1 = dt.datetime.now()\n",
    "    \n",
    "    for start, end in zip(range(0, X_train.shape[0], 128), \n",
    "                          range(128, X_train.shape[0], 128)):\n",
    "        cost = train(X_train2[start:end], y_train2[start:end])\n",
    "        \n",
    "    errors.append((np.mean(y_train != predict(X_train2)), \n",
    "                   np.mean(y_test != predict(X_test2))))\n",
    "    print(i, errors[-1])\n",
    "    print('Current iter time: ', (dt.datetime.now()-t1).seconds / 60.)\n",
    "\n",
    "print('Total time: ', (dt.datetime.now()-t0).seconds / 60.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time:  53.483333333333334\n"
     ]
    }
   ],
   "source": [
    "print('Total time: ', (dt.datetime.now()-t0).seconds / 60.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f543264a940>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VOW97/HPLyRcwiWEcIcQkFBQW4vWIgJqhKqAbT2n\nVounlQJVrJXTlr66W7Ve4KjdrXVTtbq9bBDhWG91d2/R4wUtRi1Uihasyi2IhHAJmAAJd3L5nT/W\nJJmEBCbJhJlJvu/Xa14z61nPWus3Q/ittZ71rGeZuyMiIm1DUqwDEBGRU0dJX0SkDVHSFxFpQ5T0\nRUTaECV9EZE2RElfRKQNiSjpm9lEM1tvZhvN7JcN1HnQzPLMbI2ZjQyVfcHMVpvZP0LvJWb242h+\nARERiZydrJ++mSUBG4EJwA5gFTDF3deH1ZkEzHL3y83sPOABdx9dz3q2Aee5e0F0v4aIiEQikiP9\nUUCeu+e7exnwLHBFnTpXAIsB3H0lkGZmferU+RrwqRK+iEjsRJL0BwDhiXpbqOxEdbbXU+c7wDON\nDVBERKLnlFzINbMU4JvAn07F9kREpH7JEdTZDgwKmx4YKqtbJ/MEdSYBH7j75w1txMw0CJCISCO5\nuzWmfiRH+quAbDPLMrP2wBRgSZ06S4CpAGY2Gtjn7rvC5l9DBE077q5XFF533nlnzGNoTS/9nvo9\n4/XVFCc90nf3CjObBSwl2EkscPd1ZnZDMNsfd/dXzGyymW0CDgLTq5Y3s1SCi7gzmxShiIhETSTN\nO7j7a8DwOmWP1Zme1cCyh4BeTQ1QRESiR3fktkI5OTmxDqFV0e8ZXfo9Y+ukN2edKmbm8RKLiEgi\nMDO8kRdyI2reEZHWa/DgweTn58c6DDmBrKwstmzZEpV16UhfpI0LHS3GOgw5gYb+jZpypK82fRGR\nNkRJX0SkDVHSFxFpQ5T0RaTVu/HGG7nnnntiHUZc0IVckTYu3i/kDhkyhAULFjB+/PhYhxIzupAr\nIhJSUVER6xCq1RdLY+Nr6e+jpC8icWvq1Kls3bqVb3zjG3Tr1o377ruP/Px8kpKSeOKJJ8jKymLC\nhAkAXH311fTr14/09HRycnJYu3Zt9XqmT5/OHXfcAcDbb79NZmYm8+bNo0+fPgwYMIAnn3yywRhK\nS0u57rrr6N+/P5mZmdx+++3VR92LFi1i3Lhx/OxnP6Nnz57MnTu33jJ35+6772bw4MH07duXadOm\nUVpaCtDg92kpSvoiErcWL17MoEGDePnllyktLeXnP/959bx33nmH9evX8/rrrwMwefJkPv30U3bv\n3s0555zDd7/73QbXW1hYyP79+9mxYwfz58/npptuoqSkpN663//+92nfvj2bN29m9erVvPHGG8yf\nP796/sqVK8nOzmb37t386le/qrds4cKFLF68mLfffpvNmzezf/9+Zs2qPVxZ3e/TYmI9NGjYEKEu\nIqdeJP/3IDqvphg8eLD/5S9/qZ7esmWLJyUl+ZYtWxpcZu/evW5mXlpa6u7u06ZN89tvv93d3XNz\ncz01NdUrKiqq6/fu3dtXrlx53Hp27drlHTp08CNHjlSXPfPMM37xxRe7u/uTTz7pWVlZtZapr2zC\nhAn+yCOPVE9v2LDBU1JSvKKiIqLv09C/Uai8UblWwzCIyEnF43XegQMHVn+urKzk1ltv5YUXXqCo\nqAgzw8woKiqia9euxy2bkZFBUlJNQ0dqaioHDhw4rl5+fj5lZWX069cPqDlIHjSo5rlSmZmZxy1X\nt2zHjh1kZWVVT2dlZVFeXs6uXTWPHQn/Pi0prpJ+eTkkx1VEIhJrZvV3Tgkvf/rpp3nppZdYtmwZ\ngwYNoqSkhPT09Gb3SsrMzKRjx44UFxdHFEdDZf379681vlF+fj4pKSn06dOHgoKCBtfTEuKqTb+g\n4OR1RKRt6du3L5s3b65VVjeZ79+/nw4dOpCens7Bgwe55ZZbopJE+/bty6WXXsrs2bPZv38/7s7m\nzZt55513GrWea665ht///vds2bKFAwcO8Ktf/YopU6ZUn200d+fUGHGV9DdtinUEIhJvbr75Zu66\n6y569OjBvHnzgOOPiqdOncqgQYMYMGAAX/ziFxkzZkyjtnGiHcTixYs5duwYZ5xxBj169OCqq66i\nsLCwUeufMWMG1157LRdeeCFDhw4lNTWVBx98MKLtR1tc3Zz18MPOj34U60hE2pZ4vzlLWvHNWTrS\nFxFpWUr6IiJtSFwl/by8WEcgItK6xVWbfocOzsGD0K5drKMRaTvUph//Wm2bfs+esG1brKMQEWm9\nIkr6ZjbRzNab2UYz+2UDdR40szwzW2NmI8PK08zsT2a2zsw+MbPzGtrOsGFq1xcRaUknTfpmlgQ8\nBFwGnAlcY2Yj6tSZBAx192HADcCjYbMfAF5x99OBLwPrGtpWdrba9UVEWlIkR/qjgDx3z3f3MuBZ\n4Io6da4AFgO4+0ogzcz6mFk34AJ3XxiaV+7upQ1tKDtbR/oiIi0pkqQ/AAgfIGFbqOxEdbaHyoYA\nRWa20Mz+YWaPm1mnhjakpC8i0rJaenizZOAc4CZ3f9/M7gduBu6sr/Kbb85h+XKYMwdycnLIyclp\n4fBEJN5F63GJixYtYv78+bz77rtRiuzUy83NJTc3t1nriCTpbwcGhU0PDJXVrZPZQJ0Cd38/9PkF\noN4LwQD33TeHJ5+EO+6ApLjqVyQiic7dW3SMm4qKCtrV6W9eX1lj1xGu7sHw3LlzGx1nJKl1FZBt\nZllm1h6YAiypU2cJMBXAzEYD+9x9l7vvAgrM7AuhehOAtTSgc2dIT4ftdXcpItIm1fe4RID33nuP\nsWPHkp6eztlnn83bb79dvcyTTz7J0KFD6datG0OHDuWZZ55h/fr13Hjjjfztb3+ja9eu9OjRI6Lt\n79y5k29/+9v07t2boUOH8oc//KF63ty5c7nqqqu49tpr6d69O4sWLaq37NixY/z0pz9lwIABDBw4\nkNmzZ1NWVgbUPLrx3nvvpV+/fsyYMSOKv14DInnSCjAR2ADkATeHym4AZobVeQjYBHwInBNW/mWC\nHcca4M9AWgPbcHf3Cy90X7aswQfIiEiUEedPrRs8eLAvC0sK27dv94yMDH/ttdfc3f3NN9/0jIwM\nLyoq8oMHD3q3bt08Ly/P3d0LCwt97dq17h480eqCCy6IeLuVlZX+la98xe+++24vLy/3zz77zIcO\nHepLly51d/c5c+Z4+/btfcmSJe7ufuTIkePKDh8+7Lfffruff/75XlRU5EVFRT5mzBi/44473D14\nildycrLfcsstfuzYsVpP6ArX0L8RLfXkLHd/DRhep+yxOtO1H/hYU/4h8NVItgM13TYvvjjSJUSk\npdnc6DSL+J1Nu/PXw+5Gfeqpp7j88su57LLLAJgwYQLnnnsur7zyCldeeSXt2rXjo48+YuDAgfTp\n04c+ffo0aZurVq2iqKio+rm3gwcP5rrrruPZZ5/lkksuAeD888/nG9/4BgAdOnQ4rqxjx448/fTT\nPPzww2RkZABw55138sMf/rC6aaZdu3bMnTuXlJSUJsXZWHH3nCr14BGJP01N1i0hPz+f559/npde\negkIdgjl5eWMHz+e1NRUnnvuOX73u98xY8YMxo0bx3333cfw4cNPstb6t7N9+/bqpiB3p7Kykgsv\nvLC6TqSPSgx/vGJWVhY7duyonu7Vq9cpS/gQZ8MwgO7KFZHa6l58zczMZOrUqezZs4c9e/awd+9e\n9u/fzy9+8QsALrnkEpYuXUphYSHDhw9n5syZ9a7nZDIzMznttNNqbaekpKR6Z9PQOuuWDRgw4LhH\nJfbv3/+E62hJcZf0dVeuiISr+7jE733ve7z00kssXbqUyspKjhw5wttvv82OHTvYvXs3S5Ys4dCh\nQ6SkpNClS5fqRxL26dOHbdu2VV9EPZlRo0bRtWtX7r33Xo4cOUJFRQWffPIJ77///skXDjNlyhTu\nvvtuioqKKCoq4q677uLaa69t1DqiKe6S/tCh8OmnUFkZ60hEJB7UfVziwIEDefHFF/n1r39Nr169\nyMrK4r777qOyspLKykrmzZvHgAED6NmzJ++88w6PPPIIAOPHj+fMM8+kb9++9O7d+6TbTUpK4uWX\nX2bNmjUMGTKE3r17c/3111Na2uCgAvW67bbbOPfccznrrLP48pe/zLnnnlt9nSAW4mpo5apY+vaF\nDz6AAXXv+xWRqNPQyvGv1Q6tvP/ofkDt+iIiLSWukv7G4o2A2vVFRFpKXCX99UXrAXXbFBFpKXGZ\n9NW8IyLSMuIr6RfXHOmreUdEJPriKulvKNoABN02N20CdSgQEYmuuBqGIW9PHhWVFaSltaNzZygs\nhH79Yh2VSOuWlZV1yu8KlcbJysqK2rriKun37tyb/JJ8Tks/rbpdX0lfpGVt2bIl1iHIKRRXzTsj\neo6o1YNH7foiItEVX0k/Y4S6bYqItKD4SvphR/rqtikiEn1xm/TVvCMiEn1xlfSH9xxenfTVbVNE\nJPriKun369KPI+VH2HN4D+np0KED7N4d66hERFqPuEr6ZsaIniOqb9IaNkxNPCIi0RRXSR+Ob9fX\nxVwRkehR0hcRaUPiM+kXq9umiEhLiCjpm9lEM1tvZhvN7JcN1HnQzPLMbI2ZnR1WvsXMPjSz1Wb2\n95NtS902RURazknH3jGzJOAhYAKwA1hlZi+6+/qwOpOAoe4+zMzOAx4BRodmVwI57r43koCGpg8l\nf18+xyqOkZ3dvrrbpsaDEhFpvkiO9EcBee6e7+5lwLPAFXXqXAEsBnD3lUCamfUJzbMItwNAh+QO\nZKZl8umeT+nRA5KToago0qVFROREIknGA4CCsOltobIT1dkeVseBN8xslZldH0lQauIREWkZp2Jo\n5bHuvtPMehEk/3Xu/tf6Ks6ZMweA4k+LebXoVf7n6f+zugfPmDGnIFIRkTiWm5tLbm5us9YRSdLf\nDgwKmx4YKqtbJ7O+Ou6+M/T+uZn9F0Fz0QmTfuY/Mnl367uAum2KiFTJyckhJyenenru3LmNXkck\nzTurgGwzyzKz9sAUYEmdOkuAqQBmNhrY5+67zCzVzLqEyjsDlwIfn2yDGm1TRKRlnPRI390rzGwW\nsJRgJ7HA3deZ2Q3BbH/c3V8xs8lmtgk4CEwPLd4H+C8z89C2/ujuS0+2zaqk7+5kZ5va9EVEosQ8\nToaxNDMPj6XX73rx0Y0fkXykL9nZsHevum2KiIQzM9y9UZkx7u7IrVJ1tJ+REST7PXtiHZGISOKL\n26Q/PCMYW99M3TZFRKIlbpO+Bl4TEYm+uE76G4qDcfWV9EVEoiOuk766bYqIRFfcJv3B3QdTeKCQ\nQ2WH1KYvIhIlcZv0k5OSGZo+lLziPDXviIhESdwmfahp4unVC8rL1W1TRKS5EiLpV3Xb1NG+iEjz\nxH/SL1a3TRGRaInrpF91gxaoB4+ISDTEd9LvOZyNxRup9Eod6YuIREFcJ/1uHbrRvWN3tpVuU7dN\nEZEoiOukDzUXc3WkLyLSfPGf9DOCpN+nDxw5Avv2xToiEZHEFf9JX902RUSiJmGSPijpi4g0V0Il\nfXXbFBFpnrhP+gO6DaDkaAklR0p0pC8i0kxxn/STLInhGcPZULxB3TZFRJop7pM+hB6oUrRBR/oi\nIs2UMEl/fdF6+vWDgwehtDTWEYmIJKbESfrFQbfNoUN1tC8i0lQRJX0zm2hm681so5n9soE6D5pZ\nnpmtMbORdeYlmdk/zGxJU4JUt00Rkeg4adI3syTgIeAy4EzgGjMbUafOJGCouw8DbgAerbOanwBr\nmxrksB7D2Lx3M+WV5eq2KSLSDJEc6Y8C8tw9393LgGeBK+rUuQJYDODuK4E0M+sDYGYDgcnA/KYG\n2SmlE/269OOzvZ+pB4+ISDNEkvQHAAVh09tCZSeqsz2szu+BfwG8iTECwTDLGnhNRKR5klty5WZ2\nObDL3deYWQ5gJ6o/Z86c6s85OTnk5ORUT1cNvHZN9jeU9EWkTcrNzSU3N7dZ6zD3Ex+Am9loYI67\nTwxN3wy4u/82rM6jwFvu/lxoej1wEUFb/veAcqAT0BX4s7tPrWc7fqJYHnv/MVbtWMXjX59Ply6w\nezd06dK4Lysi0pqYGe5+woPpuiJp3lkFZJtZlpm1B6YAdXvhLAGmhoIYDexz913ufqu7D3L300LL\nLasv4UeiqgdPUpK6bYqINNVJk767VwCzgKXAJ8Cz7r7OzG4ws5mhOq8An5nZJuAx4EfRDlTdNkVE\nmi+iNn13fw0YXqfssTrTs06yjreBtxsbYJXenXtT4RUUHSpi2LCeSvoiIk2QEHfkQtB2Ff7oRHXb\nFBFpvIRJ+qDn5YqINFdiJf1Qt03dlSsi0jQJlfSrbtAaMAD27AlG3BQRkcglVNIP77Z52mnw6aex\njkhEJLEkVNIfmj6UbaXbOFp+VO36IiJNkFBJP6VdCoO7D2bTnk1q1xcRaYKESvqAum2KiDRDQid9\nHemLiDROYib9YnXbFBFpisRM+kXrGTgQiorg8OFYRyQikjgSLukPzxge6rbpDB6sbpsiIo2RcEk/\nvVM6qSmp7Ni/Q+36IiKNlHBJH4Imng3FG9SuLyLSSImZ9DPUbVNEpCkSM+mr26aISJMkdNJX846I\nSOMkdNLPzIRdu+DIkVhHJCKSGBIy6Q9KG0TRoSKOVB4gKws2b451RCIiiSEhk367pHZk98hmY/FG\nteuLiDRCQiZ9ULu+iEhTJHzSV7dNEZHIJXTS31C8Qc07IiKNEFHSN7OJZrbezDaa2S8bqPOgmeWZ\n2RozGxkq62BmK81stZl9ZGZ3RitwNe+IiDTeSZO+mSUBDwGXAWcC15jZiDp1JgFD3X0YcAPwKIC7\nHwUudvezgZHAJDMbFY3Av5DxBfKK8xiYWcHOnXD0aDTWKiLSukVypD8KyHP3fHcvA54FrqhT5wpg\nMYC7rwTSzKxPaPpQqE4HIBnwaATepX0Xeqb2ZMfBrWRmwmefRWOtIiKtWyRJfwBQEDa9LVR2ojrb\nq+qYWZKZrQYKgTfcfVXTw61NTTwiIo2T3NIbcPdK4Gwz6wb8t5md4e5r66s7Z86c6s85OTnk5OSc\ncN01PXgmKemLSKuXm5tLbm5us9YRSdLfDgwKmx4YKqtbJ/NEddy91MzeAiYCJ036kRieMZx/7von\nZ2bDhg2NWlREJOHUPRieO3duo9cRSfPOKiDbzLLMrD0wBVhSp84SYCqAmY0G9rn7LjPraWZpofJO\nwCXA+kZH2YCq5+Wqr76ISGROeqTv7hVmNgtYSrCTWODu68zshmC2P+7ur5jZZDPbBBwEpocW7wcs\nCvUASgKec/dXohX8iJ4j2FC0gXMnw+rVsHYtnHFGtNYuItL6mHtUOtM0m5l5Y2Nxd7r9phtbf7qV\n5xens2ABrFgByS1+pUJEJPbMDHe3xiyTsHfkQvCFq+7MnTkTunWD++6LdVQiIvEroZM+1PTgMYMF\nC+Df/g0++STWUYmIxKfET/qh5+UCZGXB3XfDtGlQXh7buERE4lHiJ/2eNUkfYOZM6N4dfve7GAYl\nIhKnEj7pD+85vFbSN4P582HePDXziIjUlfBJP7tHNlv2baGsoqy6LCsL7rlHzTwiInUlfNLvmNyR\ngd0Gsnlv7QflXn+9mnlEROpK+KQPx7frQ+1mno8/jlFgIiJxptUmfahp5pk+Xc08IiLQmpJ+cf1D\n+lx/PaSnw733nuKgRETiUKtI+qf3PJ2V21ZytPz4x2dVNfP8/vdq5hERaRVJf/TA0QzvOZyp/z2V\nisqK4+YPGgS//rWaeUREWkXSb5fUjmeufIbdB3dz0ys3Ud/AbdddBz16qJlHRNq2hB5ls67So6Vc\nvOhiJmdP5q7xdx03f+tW+MpX4K234ItfbNamRERirs2NsllXtw7dePW7r/L82ud54L0Hjptf1cwz\nbRqUlR2/vIhIa9eqkj5A7869Wfq9pfzb3/6Np/751HHzr7sOMjLUzCMibVOrat4Jt/bztYxfNJ4F\n31zA5V+4vNY8NfOISGvQ5pt3wp3R6wxenPIi016cxrv579aaN2gQ/Ou/qplHRNqeVpv0Ac4beB5/\n/NYfufL5K/mw8MNa837wA+jZU808ItK2tNrmnXDPf/I8s1+fzTvT3mFoj6HV5QUFcM45sGwZfOlL\nLbJpEZEWo+adBlx95tXcfuHtXPrUpezcv7O6PDMTfvMbNfOISNvRJpI+wA/P/SHTR07nsqcuY+/h\nvdXlM2ZAr17w29/GMDgRkVOkTTTvVHF3Zr8+m/d3vM/Sa5eSmpIK1DTz/OUvcNZZLRqCiEjUtFjz\njplNNLP1ZrbRzH7ZQJ0HzSzPzNaY2chQ2UAzW2Zmn5jZR2b248YEF21mxrzL5jEkfQhX/emq6qdt\nZWYGN23NnAmVlbGMUESkZZ006ZtZEvAQcBlwJnCNmY2oU2cSMNTdhwE3AI+GZpUDP3P3M4HzgZvq\nLnuqJVkST3zzCQxj+ovTqfQgy//gB5CUBAsXxjI6EZGWFcmR/iggz93z3b0MeBa4ok6dK4DFAO6+\nEkgzsz7uXujua0LlB4B1wICoRd9EKe1SeP6q58kvyWf2a7Nxd5KS4OGH4Ve/guLiWEcoItIyIkn6\nA4CCsOltHJ+469bZXreOmQ0GRgIrGxtkS0hNSeWla14iNz+Xe969B4Czz4arrgoSv4hIa5R8KjZi\nZl2AF4CfhI746zVnzpzqzzk5OeTk5LRoXN07due1777GuIXjyOiUwY1fvZG77oLTTw+ae7761Rbd\nvIhIo+Tm5pKbm9usdZy0946ZjQbmuPvE0PTNgLv7b8PqPAq85e7PhabXAxe5+y4zSwZeBl519+OH\nvqxZR4v33mnI5r2buWDhBfz+st9z9ZlXs3gx/OEP8N570K5dTEISETmpluq9swrINrMsM2sPTAGW\n1KmzBJgaCmI0sM/dd4XmPQGsPVHCj7XT0k/jlf/1Cje9chPvbXuPa6+Fjh3hP/4j1pGJiERXRP30\nzWwi8ADBTmKBu//GzG4gOOJ/PFTnIWAicBCY5u6rzWws8A7wEeCh163u/lo924jZkX6Vlze+zMyX\nZrLiByvYXzCYCRPgk0+Cm7dEROJNU47029TNWZG4/737WbB6ActnLGfOLd0oKYEFC2IdlYjI8ZT0\no8Dd+dH/+xH5Jfk8dfkSvnRmMi+8AOefH+vIRERqU9KPkrKKMi5/+nJG9BzB+fse5N57YdUqSD4l\nfZ1ERCKjUTajpOrmrTc3v0nx0Ifp3h0effTky4mIxDsd6Z/A5r2bGfvEWOacvZDbvjORjz+GPn1i\nHZWISEDNOy3gr1v/yree+xZfL3qLisIzWbQo1hGJiATUvNMCxg0ax7zL5rGs79d5Y8Vu3n335MuI\niMQrHelH6PZlt/P8+38h+Y/LWPN+R1JSYh2RiLR1OtJvQXMvnsvIIZkUj5vBH/4QvzsnEZETUdKP\nUJIl8eT/eJK+p2/mtjf/Dzt2xDoiEZHGU/NOIxUeKGTEfaM5Y+e/suKxa2Idjoi0YWreOQX6dunL\n69OWsDL9x/zhv1fEOhwRkUZR0m+C8wafxS2nL2L2e1eycfeWWIcjIhIxNe80kTucOeNB9gx5nA2/\nWE5ax7RYhyQibYyad04hM3jx1v/Nvg8v4oqnvkN5ZXmsQxIROSkl/WYYNsz4ly8+wPoNzuzXZsc6\nHBGRk1LzTjMdPgynjyzBfzCGf8m5kVmjZsU6JBFpIzT2Toy8/DL8+I7POPy/xjB52GQmD5vM1077\nmtr5RaRFKenH0De/CSPOK2DAhD/z6qZXWV6wnHP6ncOk7ElMyp7EWX3OwqxR/zYiIiekpB9Dn30G\n554LL7wAOTlwuPwQuVtyeTXvVV7d9CpHyo8wMXsik7In6SxARKJCST/G/vxnuOUW6NwZfvpT+M53\noEOHYF5ecR6v5L2iswARiRol/ThQWQmvvQb33w8ffww33gg33AC9e9fUOVRW+yzgcPlhJg6dqGsB\nItIoSvpx5uOP4cEH4U9/gm99C37yEzjrrOPr5RXn8eqmYAfwbv67fCHjC4zNHMvYQWMZkzmGQWmD\nTn3wIhL3lPTjVFERPP44PPwwjBgBs2fD5MmQVM9dEkfLj/LBzg9YUbCC5QXLWb51OR2SOzAmcwxj\nM4OdwJf7fJmUdhrQX6Sta7Gkb2YTgfsJbuZa4O6/rafOg8Ak4CAw3d1Xh8oXAF8Hdrl7Pce51cu3\n2qRf5dix4Kj//vth377gyH/aNOjSpeFl3J1P934a7AS2Lmd5wXLyS/I5t/+51TuB8weeT3qn9FP2\nPUQkPrRI0jezJGAjMAHYAawCprj7+rA6k4BZ7n65mZ0HPODuo0PzxgEHgMVtPelXcYcVK4Lkv2wZ\nTJ8Os2bB4MGRLb/38F7e2/Ze9dnAqh2ryErLqj4b+FKfL5GclNysGAelDaJ7x+7NWoeItKyWSvqj\ngTvdfVJo+mbAw4/2zexR4C13fy40vQ7Icfddoeks4CUl/ePl58NDD8ETT8DFFwe9fsaODcb2iVR5\nZTkfFn5YvRNYV7SO5vyWjlNQUsCkYZOYMXIG44eMp11SuyavT0RaRksl/SuBy9x9Zmj6e8Aod/9x\nWJ2XgH919xWh6TeBX7j7P0LTSvonsX8/LFoUXPjt1Cno9fPd70LXrrGJZ8/hPTz90dMsXLOQzw9+\nzrSR05g2chqnpZ8Wm4BE5DhNSfrNawOIsjlz5lR/zsnJIScnJ2axnGpduwZNPD/6UdDk88gjcOut\nQV//G2+sv9dPS+rRqQezRs1i1qhZfFj4IQvXLOS8+efxxd5fZPrI6Vx5+pV0bt/51AYl0sbl5uaS\nm5vbrHVE2rwzx90nhqYjad5ZD1yk5p3m2b4dFiwIev5kZQXJ/9vfho4dYxPP0fKjvLzxZZ5Y8wR/\nK/gbV55+JTPOnsHogaPj5uay8spyPj/4OTsP7KTwQCGVXsl5A86jV+desQ5NJOpaqnmnHbCB4ELu\nTuDvwDXuvi6szmTgptCF3NHA/VUXckPzBxMk/S+dYDtK+g0oLw8GdXvkEVi9Gr7//eCGr+zs2MW0\nY/8OFn9gk8dhAAAL9ElEQVS4mIVrFpJkSUwfOZ1rz7qWfl37RX1b7s6BYwcoPFBYncwLDxSyc/9O\nCg+G3kNlxYeLyeiUQd8ufenbpS+VXsnft/+d3p17B/c9DBzD2EFjGdFzBEmmkcUlsbV0l80HqOmy\n+Rszu4HgiP/xUJ2HgInUdNmsas9/GsgBMoBdBBeFF9azDSX9CGzaBI89Bk8+CeecExz9f/3rkByj\nhjp3Z0XBChauWch/rvtPxg0ax4yRM7j8C5fTvl376nplFWWUHC2h5EgJpUdLj/tcerSUkiMlNZ+P\nlrDvyD52HdjFzgM7AejXpR99u/SlX9d+9O3ct+ZzKMH369KPXp17HddzqaKygrWfr2V5wfLqi917\nD+/l/Mzzq3cCX+3/VTVXScLRzVltyJEjweBujzwS9AC6/nq47joYMCB2MR04doD/XPufPLHmCdZ+\nvpb0junVCbysooy0jmmkdUijW4dutT93SCOtY83n8PlVCb1rh+he0S48UMiKghXVO4F/7vonZ/Q6\no3onMCZzDAO7DYzqNkWiTUm/jfrwQ3j0UXj2WRg/Hq6+Grp3D3oBNfTq0KFx3UIbq6CkgENlh6oT\neKfkTnHT7l+fw2WH+WDnByzfupwV24KdQWpKKmMyxzA8Y3i9O6fwz6kpqVH7fmUVZRwqO1T9Olx+\nmKPlR+nSvkv1NuP995RTQ0m/jdu/H556Cl5/HQ4dCp7q1dCrrCy4INypU817+GvIkGAHMn48DGyD\nB7zuTt6ePJZvXc6WfVtqNTsd1xR1pIRjFceqd3B1dwrd2nejwitqJfFaSb2s9jRA5/adSU1JpVNy\nJ1JTUklpl8LBYwert1leWV7rrKjWNtt3O+6sqluHbvTo1KP6zCk1JTXGv7BEg5K+RKyiAo4ebXin\nsH590HX0rbcgPT1I/hdfHLzCRwyVQFlFWYPXKkqPlpKclFwriVe9OqXUmU7uFNG4Sscqjp34mkj4\ntZNQWfGhYnYd3MXO/TvpkNyh+jpI+DWR6s+hayU9U3vqgnccU9KXqKusDEYLXbYseL3zDmRm1pwF\nXHRR0JQkicPdKTlaUqvXU1WvqLq9o0qPltKrc68GdxDhF9Kjcfbg7hQfLqagpICtJVspKC2goKSA\ngtKa6V0HdpFkSaS0SyElKaXe9+Sk5AbnpSSlkJqSSkanDDJSM+iZ2pOMTqH3sOkOyR2i8Gu3HHcn\nKSlJSV9aVnl50G20aiewYkUwcmjVmcC4cSceQO5UqqgILnh3VqecJjtWcYxdB3Ydt0OotaMI7Tyq\nzh5q7RTqnD30Su3F3iN7ayf1UGLfWrKVbaXb6Jjckcy0TDK7ZTIobRCZ3TJrTffp0gd3p6yyjLKK\nsnrfyyvLG5xXdc2k+HAxRYeKKD5UTNHh0Puhouryjskd690ZVL2ntEuhrCK0rQa2c9y8hsojWbbO\ne6VXwhyU9OXUOnYMVq4MmoGWLYP334eRI4OdwAUXwHnnQbdupyYW96BL65tvBq+33grOVG66CX7+\n86CZSlrGic4ewj/vPrib9I7p1Um8OrFXTadl0qV97I8a3J39x/bX7BTCdgZV0+WV5cedQSQnJZ/w\nDCSSM5GG5tVddztrpyN9ib1Dh4Kj/6qzgA8+CC4KjxkD558fvGdnR6/n0Oefw1/+UpPojx2DSy6B\nr30NJkwIzkzmzoUlS+BnP4Mf/1hH/tJ6qE1f4k5ZWdCldMUK+NvfgvdDh2p2AGPGBA+UT42wOfjQ\nIfjrX+GNN4Ikv3lzcF3ha18LXqefXv8OZcMGuOOO4JrErbfCzJk1zy8WSVRK+pIQtm0LdgBVO4GP\nPoIzzqh9NpCZGSTvigr4xz9qkvzf/w5nn12T5EeNgpRGPERs9Wq47Tb45BOYMwe+973Y3c0s0lxK\n+pKQDh8OmoGqdgIrVgSJfMSIIOH371+T5C+6KDrDTf/1r8ER/+efw113Bc8wru/xlSLxTElfWgX3\noNlm7Vr4yleCpN9S21m6NEj+APfcA5dd1rJ3KotEk5K+SBO4w5//HDT79OoVJP8LLoh1VCInp6Qv\n0gwVFcEwFnfeGVxjuOee4PpBJMvt3QtFRVBcXPNe9XnfvmB9l14aNFnpTEKiRUlfJAqOHoX584Ok\nP25c0AU0PJnXfS8pCe5F6NkTMjKOf09LCy4gL10a7CAuuaSmW6mGtJDmUNIXiaKDB+Hf/x3WrQsS\neENJPT09sh5A7pCXF/REeuMNyM2FwYNrdgIXXBAMdicSKSV9kQRSXh50Qa3aCXz4YXAHc9VOYORI\n9SiSE1PSF0lgpaXB0X/VTqC4OLir+JJL4MILg4Ht2rcPbipr3147BFHSF2lVtm4Nbkh7443g3oWD\nB4NhJo4eDd6Tk2vvBMLf6ytLTg7GIqqsDJqamvI5PT0YVuO004L3qs/p6bpAHQtK+iJthHswxEX4\nTuBk7+XlwdmBWfDe2M9mwdnHZ58Fr82ba96h/p3BkCHBdQtdq2gZSvoicsq5B11Ww3cE4Z+3boUe\nPWp2AF26BGcdjXmlpNR8bt8+6BGVnh68evQImr7atWv573ngQNBbq7Q0GC+qf/8gnlhR0heRuFNR\nATt2BDuALVuCYTfKy0/8KitreN7Ro8G9D3v31rxKS4PRU6t2BHVfPXrUnm7fviZ5l5Q0/Dm8bP/+\n4IwlLS3oonvwIBQWBjuc/v1hwIDar/CyjIyWaf5S0heRNqmyMkjQe/fCnj21dwj1vY4dCxJ3WlrN\nK3y6vs9dux7fNbeyMhi/afv2mteOHbWnt28PRoft37/2jqBfv5rBAt1rvyIpA7jtthZK+mY2Ebgf\nSAIWuPtv66nzIDAJOAhMc/c1kS4bqqekLyKt0qFDsHNn7R3Bzp3BmUvVGYBZ7VckZb/+dQskfTNL\nAjYCE4AdwCpgiruvD6szCZjl7peb2XnAA+4+OpJlw9ahpB8lubm55OTkxDqMVkO/Z3Tp94yepjTv\nRNLTdxSQ5+757l4GPAtcUafOFcBiAHdfCaSZWZ8Il5Uoy83NjXUIrYp+z+jS7xlbkST9AUBB2PS2\nUFkkdSJZVkRETpGWuqdPt2mIiMShSNr0RwNz3H1iaPpmwMMvyJrZo8Bb7v5caHo9cBEw5GTLhq1D\nDfoiIo3U2Db9SJ4OugrINrMsYCcwBbimTp0lwE3Ac6GdxD5332VmRREs26TARUSk8U6a9N29wsxm\nAUup6Xa5zsxuCGb74+7+iplNNrNNBF02p59o2Rb7NiIickJxc3OWiIi0vJgPzmpmE81svZltNLNf\nxjqeRGdmW8zsQzNbbWZ/j3U8icbMFpjZLjP7Z1hZupktNbMNZva6maXFMsZE0cBveaeZbTOzf4Re\nE2MZYyIxs4FmtszMPjGzj8zsx6HyRv19xjTph27eegi4DDgTuMbMRsQyplagEshx97PdfVSsg0lA\nCwn+HsPdDLzp7sOBZcAtpzyqxFTfbwkwz93PCb1eO9VBJbBy4GfufiZwPnBTKF826u8z1kf6unkr\n+ozY/7smLHf/K7C3TvEVwKLQ50XA/zilQSWoBn5LUJfuJnH3wqrhbdz9ALAOGEgj/z5jnRx081b0\nOfCGma0ys+tjHUwr0dvdd0HwHw/Q48ybZ5aZrTGz+WoqaxozGwyMBN4D+jTm7zPWSV+ib6y7nwNM\nJjj9GxfrgFoh9X5oun8HTnP3kUAhMC/G8SQcM+sCvAD8JHTEX/fv8YR/n7FO+tuBQWHTA0Nl0kTu\nvjP0/jnwXwRNaNI8u0JjSWFmfYHdMY4nYbn752EjK/4H8NVYxpNozCyZIOH/X3d/MVTcqL/PWCf9\n6hu/zKw9wc1bS2IcU8Iys9TQUQBm1hm4FPg4tlElJKN2u/MSYFro8/eBF+suIA2q9VuGklKVb6G/\nz8Z6Aljr7g+ElTXq7zPm/fRDXbYeoObmrd/ENKAEZmZDCI7uneDGuz/q92wcM3sayAEygF3AncB/\nA38CMoF84Gp33xerGBNFA7/lxQRt0ZXAFuCGqvZoOTEzGwu8A3xE8H/cgVuBvwPPE+HfZ8yTvoiI\nnDqxbt4REZFTSElfRKQNUdIXEWlDlPRFRNoQJX0RkTZESV9EpA1R0hcRaUOU9EVE2pD/DyYeYBY7\nS5onAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5432689198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res = np.array(errors)\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 0], label='train error')\n",
    "plt.plot(np.arange(res.shape[0]), res[:, 1], label='test  error')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Even more complex networks\n",
    "\n",
    "## GoogLeNet\n",
    "![a](googlenet2.png)\n",
    "\n",
    "[examples](http://www.csc.kth.se/~roelof/deepdream/bvlc_googlenet.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}