6.4-sequence-processing-with-convnets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.0.8'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence processing with convnets\n",
    "\n",
    "This notebook contains the code samples found in Chapter 6, Section 4 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "\n",
    "## Implementing a 1D convnet\n",
    "\n",
    "In Keras, you would use a 1D convnet via the `Conv1D` layer, which has a very similar interface to `Conv2D`. It takes as input 3D tensors \n",
    "with shape `(samples, time, features)` and also returns similarly-shaped 3D tensors. The convolution window is a 1D window on the temporal \n",
    "axis, axis 1 in the input tensor.\n",
    "\n",
    "Let's build a simple 2-layer 1D convnet and apply it to the IMDB sentiment classification task that you are already familiar with.\n",
    "\n",
    "As a reminder, this is the code for obtaining and preprocessing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "25000 train sequences\n",
      "25000 test sequences\n",
      "Pad sequences (samples x time)\n",
      "x_train shape: (25000, 500)\n",
      "x_test shape: (25000, 500)\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "max_features = 10000  # number of words to consider as features\n",
    "max_len = 500  # cut texts after this number of words (among top max_features most common words)\n",
    "\n",
    "print('Loading data...')\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')\n",
    "\n",
    "print('Pad sequences (samples x time)')\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=max_len)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=max_len)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1D convnets are structured in the same way as their 2D counter-parts that you have used in Chapter 5: they consist of a stack of `Conv1D` \n",
    "and `MaxPooling1D` layers, eventually ending in either a global pooling layer or a `Flatten` layer, turning the 3D outputs into 2D outputs, \n",
    "allowing to add one or more `Dense` layers to the model, for classification or regression.\n",
    "\n",
    "One difference, though, is the fact that we can afford to use larger convolution windows with 1D convnets. Indeed, with a 2D convolution \n",
    "layer, a 3x3 convolution window contains 3*3 = 9 feature vectors, but with a 1D convolution layer, a convolution window of size 3 would \n",
    "only contain 3 feature vectors. We can thus easily afford 1D convolution windows of size 7 or 9.\n",
    "\n",
    "This is our example 1D convnet for the IMDB dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 500, 128)          1280000   \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 494, 32)           28704     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 98, 32)            0         \n",
      "_________________________________________________________________\n",
      "conv1d_2 (Conv1D)            (None, 92, 32)            7200      \n",
      "_________________________________________________________________\n",
      "global_max_pooling1d_1 (Glob (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,315,937\n",
      "Trainable params: 1,315,937\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 4s - loss: 0.7713 - acc: 0.5287 - val_loss: 0.6818 - val_acc: 0.5970\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.6631 - acc: 0.6775 - val_loss: 0.6582 - val_acc: 0.6646\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.6142 - acc: 0.7580 - val_loss: 0.5987 - val_acc: 0.7118\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.5156 - acc: 0.8124 - val_loss: 0.4936 - val_acc: 0.7736\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.4029 - acc: 0.8469 - val_loss: 0.4123 - val_acc: 0.8358\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.3455 - acc: 0.8653 - val_loss: 0.4040 - val_acc: 0.8382\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.3078 - acc: 0.8634 - val_loss: 0.4059 - val_acc: 0.8240\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.2812 - acc: 0.8535 - val_loss: 0.4147 - val_acc: 0.8098\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.2554 - acc: 0.8334 - val_loss: 0.4296 - val_acc: 0.7878\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 3s - loss: 0.2356 - acc: 0.8052 - val_loss: 0.4296 - val_acc: 0.7600\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.Embedding(max_features, 128, input_length=max_len))\n",
    "model.add(layers.Conv1D(32, 7, activation='relu'))\n",
    "model.add(layers.MaxPooling1D(5))\n",
    "model.add(layers.Conv1D(32, 7, activation='relu'))\n",
    "model.add(layers.GlobalMaxPooling1D())\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer=RMSprop(lr=1e-4),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=128,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are our training and validation results: validation accuracy is somewhat lower than that of the LSTM we used two sections ago, but \n",
    "runtime is faster, both on CPU and GPU (albeit the exact speedup will vary greatly depending on your exact configuration). At that point, \n",
    "we could re-train this model for the right number of epochs (8), and run it on the test set. This is a convincing demonstration that a 1D \n",
    "convnet can offer a fast, cheap alternative to a recurrent network on a word-level sentiment classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4FFXW+PHvIYAhCLI6IAiJirIoa0QZQGAQxQ1cEEFw\nBBcUBdRx5hUEFVEcRx1f1JdRo4OjEkB+MiKoiCIouCAJQkSCILIZQIyIbIlC4Pz+uJWkE7I00El1\nus/nefrprqpb1acr6VPVt27dK6qKMcaY6FDJ7wCMMcaUH0v6xhgTRSzpG2NMFLGkb4wxUcSSvjHG\nRBFL+sYYE0Us6UchEYkRkX0i0iSUZf0kImeISMjbH4vIhSKyKWB6rYh0DabsMbzXyyJy/7Gub0ww\nKvsdgCmdiOwLmIwDfgcOedO3qWry0WxPVQ8BJ4a6bDRQ1bNCsR0RuQUYrKrdA7Z9Syi2bUxJLOlX\nAKqal3S9M8lbVHVBceVFpLKq5pRHbMaUxv4fw4tV70QAEXlURN4QkekishcYLCKdRGSpiPwqIttF\n5FkRqeKVrywiKiLx3vRUb/k8EdkrIl+ISMLRlvWWXyIi60Rkt4g8JyKficiQYuIOJsbbRGS9iOwS\nkWcD1o0Rkf8VkZ0isgHoXcL+GSsiMwrNmywiT3uvbxGRNd7n+d47Cy9uWxki0t17HScir3uxrQY6\nFCo7TkQ2eNtdLSJ9vPnnAP8HdPWqzn4O2LfjA9a/3fvsO0Vktog0DGbfHM1+zo1HRBaIyC8i8qOI\n/E/A+zzg7ZM9IpIqIqcUVZUmIp/m/p29/bnYe59fgHEi0kxEFnnv8bO3304KWL+p9xkzveXPiEis\nF3OLgHINRSRLROoW93lNKVTVHhXoAWwCLiw071HgAHAF7kBeDTgXOA/3a+40YB0wwitfGVAg3pue\nCvwMJAJVgDeAqcdQ9mRgL9DXW/YX4CAwpJjPEkyMbwMnAfHAL7mfHRgBrAYaA3WBxe7fucj3OQ3Y\nB1QP2PZPQKI3fYVXRoA/AdlAa2/ZhcCmgG1lAN29108BHwO1gaZAeqGy/YGG3t/kei+GP3jLbgE+\nLhTnVGC89/oiL8a2QCzwL2BhMPvmKPfzScAO4C7gBKAm0NFbNgZIA5p5n6EtUAc4o/C+Bj7N/Tt7\nny0HGA7E4P4fzwR6AlW9/5PPgKcCPs833v6s7pXv7C1LAiYGvM+9wFt+fw8r8sP3AOxxlH+w4pP+\nwlLW+yvw/7zXRSXyFwLK9gG+OYayNwFLApYJsJ1ikn6QMZ4fsPy/wF+914tx1Vy5yy4tnIgKbXsp\ncL33+hJgbQll3wHu9F6XlPS3BP4tgDsCyxax3W+Ay7zXpSX9V4HHApbVxF3HaVzavjnK/XwDkFJM\nue9z4y00P5ikv6GUGPrlvi/QFfgRiCmiXGdgIyDe9Erg6lB/r6LpYdU7keOHwAkRaS4i73o/1/cA\nE4B6Jaz/Y8DrLEq+eFtc2VMC41D3Lc0obiNBxhjUewGbS4gXYBow0Ht9vTedG8flIvKlV/XwK+4s\nu6R9lathSTGIyBARSfOqKH4Fmge5XXCfL297qroH2AU0CigT1N+slP18Ki65F6WkZaUp/P/YQERm\nishWL4b/FIphk7pGAwWo6me4Xw1dRORsoAnw7jHGZLA6/UhSuLnii7gzyzNUtSbwIO7Muyxtx52J\nAiAiQsEkVdjxxLgdlyxyldakdCZwoYg0wlU/TfNirAa8CfwdV/VSC/ggyDh+LC4GETkNeB5XxVHX\n2+63AdstrXnpNlyVUe72auCqkbYGEVdhJe3nH4DTi1mvuGX7vZjiAuY1KFSm8Of7B67V2TleDEMK\nxdBURGKKieM1YDDuV8lMVf29mHImCJb0I1cNYDew37sQdls5vOc7QHsRuUJEKuPqieuXUYwzgbtF\npJF3Ue++kgqr6o+4Koj/4Kp2vvMWnYCrZ84EDonI5bi652BjuF9Eaom7j2FEwLITcYkvE3f8uxV3\npp9rB9A48IJqIdOBm0WktYicgDsoLVHVYn85laCk/TwHaCIiI0TkBBGpKSIdvWUvA4+KyOnitBWR\nOriD3Y+4BgMxIjKMgANUCTHsB3aLyKm4KqZcXwA7gcfEXRyvJiKdA5a/jqsOuh53ADDHwZJ+5LoX\nuBF3YfVF3AXXMqWqO4DrgKdxX+LTgRW4M7xQx/g88BGwCkjBna2XZhqujj6vakdVfwXuAd7CXQzt\nhzt4BeMh3C+OTcA8AhKSqn4NPAcs88qcBXwZsO6HwHfADhEJrKbJXf99XDXMW976TYBBQcZVWLH7\nWVV3A72Aa3AHonVAN2/xk8Bs3H7eg7uoGutV290K3I+7qH9Goc9WlIeAjriDzxxgVkAMOcDlQAvc\nWf8W3N8hd/km3N/5d1X9/Cg/uykk9+KIMSHn/VzfBvRT1SV+x2MqLhF5DXdxeLzfsVR0dnOWCSkR\n6Y1rKZONa/J3EHe2a8wx8a6P9AXO8TuWSGDVOybUugAbcHXZFwNX2YU3c6xE5O+4ewUeU9UtfscT\nCax6xxhjooid6RtjTBQJuzr9evXqaXx8vN9hGGNMhbJ8+fKfVbWkJtJAGCb9+Ph4UlNT/Q7DGGMq\nFBEp7a50wKp3jDEmqljSN8aYKGJJ3xhjokjY1ekX5eDBg2RkZPDbb7/5HYopQWxsLI0bN6ZKleK6\nkzHG+K1CJP2MjAxq1KhBfHw8ruNGE25UlZ07d5KRkUFCQkLpKxhjfFEhqnd+++036tatawk/jIkI\ndevWtV9jJUhOhvh4qFTJPScf1XD2xoRGhUj6gCX8CsD+RsVLToZhw2DzZlB1z8OG+ZP47eAT3SpM\n0jemIhs7FrKyCs7LynLzy1M4HXyMPyzpB2Hnzp20bduWtm3b0qBBAxo1apQ3feDAgaC2MXToUNau\nXVtimcmTJ5Ns376ItKWYrsKKm19WwuXgY/xTIS7kHq3kZPdPvGULNGkCEyfCoGMdfgKoW7cuK1eu\nBGD8+PGceOKJ/PWvfy1QJm/Q4UpFH0dfeeWVUt/nzjvvPPYgTVhr0sSdVRc1vzyFy8HH+CfizvTL\n8+fr+vXradmyJYMGDaJVq1Zs376dYcOGkZiYSKtWrZgwYUJe2S5durBy5UpycnKoVasWo0ePpk2b\nNnTq1ImffvoJgHHjxjFp0qS88qNHj6Zjx46cddZZfP65GzBo//79XHPNNbRs2ZJ+/fqRmJiYd0AK\n9NBDD3Huuedy9tlnc/vtt5Pbm+q6dev405/+RJs2bWjfvj2bNm0C4LHHHuOcc86hTZs2jLXTvpCb\nOBHi4grOi4tz88tTcQeZ8j74GP9EXNIv75+v3377Lffccw/p6ek0atSIxx9/nNTUVNLS0vjwww9J\nT08/Yp3du3fTrVs30tLS6NSpE1OmTCly26rKsmXLePLJJ/MOIM899xwNGjQgPT2dBx54gBUrVhS5\n7l133UVKSgqrVq1i9+7dvP/++wAMHDiQe+65h7S0ND7//HNOPvlk5s6dy7x581i2bBlpaWnce++9\nIdo7JtegQZCUBE2bgoh7Tko6vl+gxyJcDj7GPxGX9Mv75+vpp59OYmJi3vT06dNp37497du3Z82a\nNUUm/WrVqnHJJZcA0KFDh7yz7cKuvvrqI8p8+umnDBgwAIA2bdrQqlWrItf96KOP6NixI23atOGT\nTz5h9erV7Nq1i59//pkrrrgCcDdTxcXFsWDBAm666SaqVasGQJ06dY5+R5hSDRoEmzbB4cPuubwT\nfm4M4XDwsRZE/om4Ov3yrjutXr163uvvvvuOZ555hmXLllGrVi0GDx5cZLv1qlWr5r2OiYkhJyen\nyG2fcMIJpZYpSlZWFiNGjOCrr76iUaNGjBs3ztrPmzyDBvlzwMmVWwWb+4s8two2NzZTtoI60xeR\n3iKyVkTWi8joIpY3EZFFIrJCRL4WkUu9+fEiki0iK73HC6H+AIX5+fN1z5491KhRg5o1a7J9+3bm\nz58f8vfo3LkzM2fOBGDVqlVF/pLIzs6mUqVK1KtXj7179zJr1iwAateuTf369Zk7dy7gbnrLysqi\nV69eTJkyhezsbAB++eWXkMdtTC5rQeSvUs/0RSQGmAz0AjKAFBGZo6qB2WYcMFNVnxeRlsB7QLy3\n7HtVbRvasIuXe6YQytY7wWrfvj0tW7akefPmNG3alM6dO4f8PUaOHMmf//xnWrZsmfc46aSTCpSp\nW7cuN954Iy1btqRhw4acd955ecuSk5O57bbbGDt2LFWrVmXWrFlcfvnlpKWlkZiYSJUqVbjiiit4\n5JFHQh67MWAtiHyX29SwuAfQCZgfMD0GGFOozIvAfQHlP/dexwPflPYegY8OHTpoYenp6UfMi1YH\nDx7U7OxsVVVdt26dxsfH68GDB32OKl84/q2mTlVt2lRVxD1Pnep3RNGtaVNV17au4KNpU78jq9iA\nVA0ixwZTp98I+CFgOgM4r1CZ8cAHIjISqA5cGLAsQURWAHuAcaq6pPAbiMgwYBhAE2s7VqJ9+/bR\ns2dPcnJyUFVefPFFKleOuEszIWP1x+Fn4sSCfxOwFkTlKVTZYiDwH1X9p4h0Al4XkbOB7UATVd0p\nIh2A2SLSSlX3BK6sqklAEkBiYqKGKKaIVKtWLZYvX+53GBVGSfXHlvT94WcVrAku6W8FTg2YbuzN\nC3Qz0BtAVb8QkVignqr+BPzuzV8uIt8DZwI2CK4pF1Z/HJ78bkEUzYJpvZMCNBORBBGpCgwA5hQq\nswXoCSAiLYBYIFNE6nsXghGR04BmwIZQBW9MaewOVGMKKjXpq2oOMAKYD6zBtdJZLSITRKSPV+xe\n4FYRSQOmA0O8CwsXAF+LyErgTeB2VbX2gKbc2B2oxhQUVJ2+qr6Ha4YZOO/BgNfpwBHtE1V1FjDr\nOGM05phZ/bExBVmzjyD06NGD0aNHc/HFF+fNmzRpEmvXruX5558vdr0TTzyRffv2sW3bNkaNGsWb\nb755RJnu3bvz1FNPFejKobBJkyYxbNgw4rxT1ksvvZRp06ZRq1at4/hU0aMs6o8PH4bsbHdReP9+\n9wjmdXY2NGsG3bpB69auGwJjypMl/SAMHDiQGTNmFEj6M2bM4Iknnghq/VNOOaXIhB+sSZMmMXjw\n4Lyk/95775Wyhilszx5Yu/bIJBxssi78unCLoGBUqwZVq8Lu3W66Vi3o2tUdALp1g7ZtwVrflq9Q\nd8NeEdi/WBD69evHuHHjOHDgAFWrVmXTpk1s27aNrl27sm/fPvr27cuuXbs4ePAgjz76KH379i2w\n/qZNm7j88sv55ptvyM7OZujQoaSlpdG8efO8rg8Ahg8fTkpKCtnZ2fTr14+HH36YZ599lm3bttGj\nRw/q1avHokWLiI+PJzU1lXr16vH000/n9dJ5yy23cPfdd7Np0yYuueQSunTpwueff06jRo14++23\n8zpUyzV37lweffRRDhw4QN26dUlOTuYPf/gD+/btY+TIkaSmpiIiPPTQQ1xzzTW8//773H///Rw6\ndIh69erx0Ucflf3OD4GNG6FLF9i2reRy1aq5+v7q1d0j93X9+kXPP5rXcXH5Z/VbtsAnn+Q/vF4x\nqFGj4EGgfXuoUqVs9000i9Z7OCpc0r/7biii+/jj0rYteN3YF6lOnTp07NiRefPm0bdvX2bMmEH/\n/v0REWJjY3nrrbeoWbMmP//8M+effz59+vQpdrzY559/nri4ONasWcPXX39N+/bt85ZNnDiROnXq\ncOjQIXr27MnXX3/NqFGjePrpp1m0aBH16tUrsK3ly5fzyiuv8OWXX6KqnHfeeXTr1o3atWvz3Xff\nMX36dF566SX69+/PrFmzGDx4cIH1u3TpwtKlSxERXn75ZZ544gn++c9/8sgjj3DSSSexatUqAHbt\n2kVmZia33norixcvJiEhocL0z/Pjj9Crl6tWeeMNl8BLS8plrUkTuOEG9wDYuhUWL84/COT+kKte\nHTp3zj8InHuu+6VgQiNa7+GocEnfL7lVPLlJ/9///jfgurG4//77Wbx4MZUqVWLr1q3s2LGDBg0a\nFLmdxYsXM2rUKABat25N69at85bNnDmTpKQkcnJy2L59O+np6QWWF/bpp59y1VVX5fX0efXVV7Nk\nyRL69OlDQkICbdu6Lo+K6745IyOD6667ju3bt3PgwAESEhIAWLBgATNmzMgrV7t2bebOncsFF1yQ\nV6YidL/866/Quzds3w4LFkCnTn5HVLRGjWDgQPcAd6AKPAjkdkRWrRr88Y/5B4GOHSE21r+4K7po\nvYejwiX9ks7Iy1Lfvn255557+Oqrr8jKyqJDhw6A68AsMzOT5cuXU6VKFeLj44+pG+ONGzfy1FNP\nkZKSQu3atRkyZMhxdYec2y0zuK6ZA6uRco0cOZK//OUv9OnTh48//pjx48cf8/uFm6wsuPxySE+H\nd94J34RflAYNoH9/9wDIzIQlS/IPAg895HqrOeEEOP/8/INAp07uwGCCEy5DWJY3azsQpBNPPJEe\nPXpw0003MTD3lAw3CtbJJ59MlSpVWLRoEZuL+i8KcMEFFzBt2jQAvvnmG77++mvAdctcvXp1Tjrp\nJHbs2MG8efPy1qlRowZ79+49Yltdu3Zl9uzZZGVlsX//ft566y26du0a9GfavXs3jRo1AuDVV1/N\nm9+rVy8mT56cN71r1y7OP/98Fi9ezMaNG4Hw7n754EG49lr4/HOYOhUuusjviI5P/fpw9dXwzDOu\navPnn2H2bLjjDti7Fx59FHr2zL8wPG4cfPihu+hsihet93BY0j8KAwcOJC0trUDSHzRoEKmpqZxz\nzjm89tprNG/evMRtDB8+nH379tGiRQsefPDBvF8Mbdq0oV27djRv3pzrr7++QLfMw4YNo3fv3vTo\n0aPAttq3b8+QIUPo2LEj5513Hrfccgvt2rUL+vOMHz+ea6+9lg4dOhS4XjBu3Dh27drF2WefTZs2\nbVi0aBH169cnKSmJq6++mjZt2nDdddcF/T7l6fBhGDLE1Yu/8EL+2XIkqVMH+vaFp5+G5cvhl1/c\nr5m77oIDB+Dxx92BrlYtd/Y/ejTMm+daMJl84TKKWHkT1fDq3ywxMVFTUwt2zbNmzRpatGjhU0Tm\naPj5t1KFkSNh8mR47DEYM8aXMHy3d6/7lfPxx646KCUFcnLcheoOHfKrg7p0cQcGExlEZLmqFn/D\nj6fC1ekbU5zx413Cv/ded3YbrWrUgIsvdg9w1TxffJF/TeDZZ+Gpp9zZbdu2+QeBCy5wvyJMZLOk\nbyLCs8/ChAkwdCg8+aRLaMapXh0uvNA9wDVfXbo0/yDwwguugYQInHNOwYNA/fr+xm5Cr8IkfVUt\ntu27CQ9+VRW+/rqrz77ySlcna/8mJatWDXr0cA+A33+HZcvyDwL//jc895xb1rJl/kGgWzfXsshU\nbBWiTn/jxo3UqFGDunXrWuIPU6rKzp072bt3b15b/vIwdy5cdZVLSO++a+3WQ+HAAXeBOPeawGef\nwb59btlZZxU8CHiNv0wYCLZOv0Ik/YMHD5KRkXFc7dZN2YuNjaVx48ZU8foOKOt+TRYvdvXWZ58N\nCxe6umwTejk58NVX+b8ElizJbwl0+uku+Xfv7p4jvY17OIuopG8qnsL9moBrAx2qJnErVrhEc8op\nLgkV6qHClKFDhyAtLf+XwJIlsGuXWxYfX/CXQEKCVbeVF0v6xlfx8UXf7di0KRTRI8RRWbfONTes\nVs1VPTRufHzbM8fn8GFYtSr/l8Dixe4GMnB/m9xfAd26wRln2EGgrFjSN76qVMm1my9MxCWJY5WR\n4Tohy86GTz+FM8889m2ZsnH4sOv+IrAn0Z9+cssaNiz4S6B5czsIhIq10ze+Kot+TX7+2d1pumuX\nq1qwhB+eKlVy11nOPhvuvNMd/NeuLXgQyO3Pr2FDuP56uOkm11LIlD3rhsGUiVD3a7J3L1x6qesb\nf+5c19e8qRhE3Bn9bbfBtGnu19p338FLL8F557k+hVq1cp3HJSXlDzITTZKTXZVopUruOTm5DN9M\nVcPq0aFDBzWRYepU1aZNVUXc89Spx7ad335T7dlTNSZGdc6cUEZowsGOHapPP63aqpUqqFarpjp4\nsOrChaqHDvkdXdmbOlU1Ls599txHXNzRf1+AVA0ixwZVpy8ivYFngBjgZVV9vNDyJsCrQC2vzGh1\ng6kjImOAm4FDwChVnV/Se1mdvgmUkwPXXQf//S+89lr+wCMm8qhCaipMmeJ+EezZ41r/DB0KN94Y\nuc1BQ9XoIdg6/VKPCrgk/j1wGlAVSANaFiqTBAz3XrcENgW8TgNOABK87cSU9H52pm9yHT6sevPN\n7sxn0iS/ozHlaf9+d6bbs6f7+4uo9uqlOn26ana239GFlkjBs/zch8jRbYcgz/SDqdPvCKxX1Q2q\negCYAfQtVEaBmt7rk4Dc0Uj7AjNU9XdV3Qis97ZnTKnuu891CfDAA66bBRM94uLc/RwLFsCGDfDg\ng+5i8MCB7uLviBHuruEwa3x4TIr7BVNWv2yCSfqNgB8CpjO8eYHGA4NFJAN4Dxh5FOsac4R//MN1\nnHbnnfDww35HY/yUkOB6UN240R0ELr0UXn4ZEhNdL6HPPJN/X0BFVN6DuYSq9c5A4D+q2hi4FHhd\nRILetogME5FUEUnNzMwMUUimonrpJdc18sCBrvdMa8dtwLVs6dnTtWzZvh3+9S83ZOTdd7s7s6+9\n1g2ek5Pjd6RHp7wHcwkmMW8FTg2YbuzNC3QzMBNAVb8AYoF6Qa6LqiapaqKqJta3vlyj2ptvwu23\nwyWXwKuvui+6MYXVrg3Dh7veQb/+2lX3fPwxXHaZS5r33++ahVYUgwa5i7aHD7vnshy9K5ivVArQ\nTEQSRKQqMACYU6jMFqAngIi0wCX9TK/cABE5QUQSgGbAslAFbyLLhx+6G3U6dXLJ3+u3zZgSnXOO\nGzpy61aYNQvatXPVg2ee6cYEeOWV/F5CTRBJX1VzgBHAfGANMFNVV4vIBBHp4xW7F7hVRNKA6cAQ\n74LyatwvgHTgfeBOVT1UFh/EVGxffum6SG7Rwo33WriO05jSVK3qBpB/5x344Qc3VvCOHe5u3wYN\n4OabXV9NkXDx93hY3zvGd6tXQ9eubqi+Tz+1gTpM6Ki6oSKnTIE33nBn/Gee6dr+//nP7lpApAi2\nnb7VmBpfbdrk+tOJjXXVO5bwTSiJwB//6Fr7bN/uqnr+8AcYMwZOPRUuv9zd+HfggN+Rlh9L+sY3\nO3ZAr16ux8wPPnBN84wpKyeeCEOGuK6f161zLcRWrIBrrnFt4p94wvXxFOks6Rtf/PqrG/Vq2zY3\nzOHZZ/sdkYkmzZq5dvBbtrj/v9at3c2ATZvChAn5g8JEIkv6ptxlZcEVV7g+1996y7XWMcYPMTHu\nZq8PPnCNCbp2hYcecsl/zJj8cQAiiSV9U64OHoT+/V0riqlTXX2+MeGgY0d4+203FOSll7pmn/Hx\n7uavrUfcXVRxWdI35ebwYVen+u678MILLvkbE25at3aDvKxZ43p4/b//g9NOczcNbtzod3THz5K+\nKReqrtO0adPgscfcoOnGhLOzznKtfdavd238X3nFXQu48Ub49lu/ozt2lvRNuXj4YXfGdO+9rtWE\nMRVFfLzr52fjRhg1yt0t3rKl+6W6cqXf0R09S/qmzD37rEv6Q4e6njOtAzVTEZ1yiuvuYdMmd5F3\n/nzX5cMVV8DSpX5HFzxL+qZMJSe7ap0rr3Q9B1rCNxVd/fquuefmzfDII+6O306d4MILXadvYdbJ\nwREs6ZsyM3euq//s0QOmT4fKlf2OyJjQqVULxo1zZ/5PPeW6E+nRA7p0gXnzwjf5W9I3ZeLjj13/\n5u3bu2ZwsbF+R2RM2TjxRHetauNGmDwZMjJck8/ERNfFw+HDfkdYkCV9E3KpqdCnD5x+ujvjqVHD\n74iMKXuxsXDHHa4f/ylT3MDu11zjun5OTg6fwV0s6ZuQWrMGeveGunXdXY516/odkTHlq2pV12jh\n229dE+VKlWDwYGje3I357Hfnbpb0Tchs3uw6UKtc2fWY2chGQzZRLCbGDfmZlua6G6ldG265Bc44\nwzVfzs72Jy5L+iYkduxwrRf273dn+Gec4XdExoSHSpVc67Vly+D9912/PiNHul5ln3yy/Hv2tKQf\ngZKT3Q0llSq55+Tksn2/wB4z33vP3cZujClIxH1PliyBTz5x35P/+R/3HS3Pnj0t6UeY5GTXxcHm\nza7J2ObNbrqsEn9WlhuIIj0dZs+2HjONCcYFF+T37NmlS37PnmPHln1TT0v6EWbsWJeIA2Vlufmh\nduCAa53wxRfuglWvXqF/D2MiWeGePb//vuxvYLTbZSLMli1HN/9YHToEN9zg6ihffhn69Qvt9o2J\nJrk9ex46VPbvZWf6EaZJk6ObfyxUYfhwmDnT3Yl4882h27Yx0SwmpuzfI6ikLyK9RWStiKwXkSP6\nSBSR/xWRld5jnYj8GrDsUMCyOaEM3hxp4kSIiys4Ly7OzQ+VMWPgpZfg/vvdnYjGmIqj1OodEYkB\nJgO9gAwgRUTmqGp6bhlVvSeg/EigXcAmslW1behCNiUZNMg9jx3rqnSaNHEJP3f+8frHP9xj+HB4\n9NHQbNMYU36CqdPvCKxX1Q0AIjID6AukF1N+IPBQaMIzx2LQoNAl+UBJSa4v/IED3c0l1mOmMRVP\nMNU7jYAfAqYzvHlHEJGmQAKwMGB2rIikishSEbmymPWGeWVSMzMzgwzdlKc33nDDxV12Gbz6qrsH\nwBhT8YT6qzsAeFNVA69BN1XVROB6YJKInF54JVVNUtVEVU2sX79+iEMyx+v9913fIV26uIu3Var4\nHZEx5lgFk/S3AqcGTDf25hVlADA9cIaqbvWeNwAfU7C+34S5Tz+Fq692Tcrmzj3yIrExpmIJJumn\nAM1EJEFEquIS+xGtcESkOVAb+CJgXm0ROcF7XQ/oTPHXAkyYWbnS3W3bpIk72z/pJL8jMsYcr1Iv\n5KpqjoiMAOYDMcAUVV0tIhOAVFXNPQAMAGaoFriJuAXwoogcxh1gHg9s9WPC17p1rp+QmjXd7eJW\n62ZMZBD5+ZqSAAARC0lEQVQNszG9EhMTNTU11e8wotoPP7j6++xs1znUWWf5HZExpjQisty7floi\n64bBFJCZCRdd5HrO/PhjS/jGRBpL+ibPnj1wySVuoOcPPoB2dsndmIhjSd8AriqnTx/X29/bb0PX\nrn5HZIwpC5b0DQcPQv/+sHix63f/0kv9jsgYU1Ys6Ue5w4fdIM7vvAPPP++6WDDGRC67mT6KqcKo\nUe7s/rHHXDcLxpjIZkk/ij34IEyeDH/7m+tIzRgT+SzpR6mnn3ZdI99yi+sq2XrMNCY6WNKPQlOm\nuMFPrr0WXnjBEr4x0cSSfpT573/h1lvdDVivv14+w7MZY8KHJf0osmCBa51z3nku+Z9wgt8RGWPK\nmyX9KLF0KVx5JTRvDu++C9Wr+x2RMcYPlvSjwKpV7oarBg1g/nyoXdvviIwxfrGkH+G+/97V31er\n5qp3GjTwOyJjjJ/sjtwItm0b9OrlullYvBji4/2OyBjjN0v6EWrnTneGn5kJCxdCy5Z+R2SMCQdW\nvROB0tNdC53162HOHDj3XL8jMsaEC0v6Eeadd+D882HfPneG36OH3xEZY8KJJf0Ioeq6U+jTB5o1\ng5QU+OMf/Y7KGBNurE4/AmRnuz50pk2D665z3SzExfkdlTEmHAV1pi8ivUVkrYisF5Ej+mMUkf8V\nkZXeY52I/Bqw7EYR+c573BjK4A1kZLhRrqZPh4kT3bMlfGNMcUo90xeRGGAy0AvIAFJEZI6qpueW\nUdV7AsqPBNp5r+sADwGJgALLvXV3hfRTRKmlS+Gqq1z9/ezZrmrHGGNKEsyZfkdgvapuUNUDwAyg\nbwnlBwLTvdcXAx+q6i9eov8Q6H08ARvn1VehWzd3Vr90qSV8Y0xwgkn6jYAfAqYzvHlHEJGmQAKw\n8GjWFZFhIpIqIqmZmZnBxB21cnJct8hDhkCXLrBsGbRq5XdUxpiKItStdwYAb6rqoaNZSVWTVDVR\nVRPr168f4pAix6+/wuWXuwFQRo6E99+HunX9jsoYU5EEk/S3AqcGTDf25hVlAPlVO0e7rinB2rXu\nhquFCyEpCZ59FqpU8TsqY0xFE0zSTwGaiUiCiFTFJfY5hQuJSHOgNvBFwOz5wEUiUltEagMXefPM\nUZg3zyX8Xbvgo4/cICjGGHMsSk36qpoDjMAl6zXATFVdLSITRCTw8uEAYIaqasC6vwCP4A4cKcAE\nb54Jgir885+uSic+3t1w1bWr31EZYyoyCcjRYSExMVFTU1P9DsN3v/0Gw4a5IQ2vuca11rGBT4wx\nxRGR5aqaWFo564YhDG3b5ppjvv46PPwwzJxpCd8YExrWDUOYSUlxwxru3u3Gsb3qKr8jMsZEEjvT\nDyPJya7OvkoV+PxzS/jGmNCzpB8GDh2C++6DwYNdt8gpKdC6td9RGWMikVXv+Gz3brj+enjvPbj9\ndmt/b4wpW3amH0LJya5pZaVK7jk5ueTy333nzuw/+AD+9S94/nlL+MaYsmVn+iGSnOyaWGZluenN\nm900wKBBR5b/8EPo3x9iYtzr7t3LLVRjTBSzM/0QGTs2P+Hnyspy8wOpwjPPQO/e0Lixq7+3hG+M\nKS+W9ENky5bS5//+uxvh6u67XVfIn38OCQnlE58xxoAl/ZBp0qTk+T/+CH/6kxvK8IEHYNYsqFGj\n/OIzxhiwpB8yEyceOUxhXJyb/9VXcO65sGKFu7t2wgR3sdcYY8qbpZ4QGTTIdXnctCmIuOekJKhc\n2Q12IgKffQbXXut3pMaYaGZJP4QGDYJNm+DwYdiwAdasgQEDoH17d8G2XTu/IzTGRDtrslkG9u6F\nG26At9+Gm292bfCrVvU7KmOMsaQfctu2wUUXwbffurtrR4xwVTvGGBMOLOmH0P79rinmpk1u/NoL\nL/Q7ImOMKciSfogcPuw6TFuxwlXrWMI3xoQjS/ohMno0zJ4Nkya54Q2NMSYcWeudEHjpJXjySbjj\nDhg1yu9ojDGmeJb0j9OCBTB8uOtL55ln7KKtMSa8WdI/Dunp0K8ftGgBb7zhbsQyxphwFlTSF5He\nIrJWRNaLyOhiyvQXkXQRWS0i0wLmHxKRld5jTqgC99tPP7m6+9hYeOcdqFnT74iMMaZ0pZ6bikgM\nMBnoBWQAKSIyR1XTA8o0A8YAnVV1l4icHLCJbFVtG+K4ffXbb27w8u3b4ZNPXJcLxhhTEQRzpt8R\nWK+qG1T1ADAD6FuozK3AZFXdBaCqP4U2zPChCkOHwhdfwOuvQ8eOfkdkjDHBCybpNwJ+CJjO8OYF\nOhM4U0Q+E5GlItI7YFmsiKR6868s6g1EZJhXJjUzM/OoPkB5Gz8eZsyAv//d1ecbY0xFEqpLj5WB\nZkB3oDGwWETOUdVfgaaqulVETgMWisgqVf0+cGVVTQKSABITEzVEMYXc1KmuW+ShQ+G++/yOxhhj\njl4wZ/pbgVMDpht78wJlAHNU9aCqbgTW4Q4CqOpW73kD8DFQIfuaXLLEdZ7WvTu88II1zTTGVEzB\nJP0UoJmIJIhIVWAAULgVzmzcWT4iUg9X3bNBRGqLyAkB8zsD6VQw69fDVVdBfLwb8cp6zDTGVFSl\nVu+oao6IjADmAzHAFFVdLSITgFRVneMtu0hE0oFDwN9UdaeI/BF4UUQO4w4wjwe2+qkIdu2Cyy5z\nF3DffRfq1PE7ImOMOXaiGl5V6ImJiZqamup3GAAcOODutP30U/joI+ja1e+IjDGmaCKyXFUTSytn\n95AWQ9V1r7BoEbz2miV8Y0xksG4YivGPf8CUKfDAA24ULGOMiQSW9Ivw5pswZowb3/bhh/2Oxhhj\nQseSfiHLlrkz+06d4JVXrGmmMSayWNIPsHmzG+6wYUM3+lVsrN8RGWNMaNmFXM+ePa7XzN9+g4UL\noX59vyMyxpjQs6QP5OTAddfBmjVuQPOWLf2OyBhjykbUJ31VuOsul+yTkmxAc2NMZIv6Ov3nnoN/\n/Qv++le49Va/ozHGmLIV1Un/nXfgnnvcgCiPP+53NMYYU/aiNumnpbl2+G3bui6TY2L8jsgYY8pe\nVCb9bdtcS51atWDuXKhe3e+IjDGmfETdhdz9+11b/F27XEdqp5zid0TGGFN+oirpHz4MgwfDihXu\n5qu2ETVcuzHGlC6qkv5998Hs2TBpkqveMcaYaBM1dfpJSfDUU3DHHTBqlN/RGGOMP6Ii6S9Y4JJ9\n797wzDPWiZoxJnpFfNJPT4d+/aBFC3jjDagcVRVaxhhTUEQn/Z9+cnX3sbHuRqyaNf2OyBhj/BWx\n572//ebutN2+HT75BJo29TsiY4zxX1Bn+iLSW0TWish6ERldTJn+IpIuIqtFZFrA/BtF5DvvcWOo\nAi+JKgwdCl98Aa+/Dh07lse7GmNM+Cv1TF9EYoDJQC8gA0gRkTmqmh5QphkwBuisqrtE5GRvfh3g\nISARUGC5t+6u0H+UfOPHw4wZ8Pe/u/p8Y4wxTjBn+h2B9aq6QVUPADOAvoXK3ApMzk3mqvqTN/9i\n4ENV/cVb9iHQOzShF23qVJgwwZ3p33dfWb6TMcZUPMEk/UbADwHTGd68QGcCZ4rIZyKyVER6H8W6\niMgwEUkVkdTMzMzgoy9kyRK4+Wbo3h1eeMGaZhpjTGGhar1TGWgGdAcGAi+JSK1gV1bVJFVNVNXE\n+sc4TuHGjXDVVRAfD7NmQdWqx7QZY4yJaMEk/a3AqQHTjb15gTKAOap6UFU3AutwB4Fg1g2Jhg2h\nf394912oU6cs3sEYYyq+YJJ+CtBMRBJEpCowAJhTqMxs3Fk+IlIPV92zAZgPXCQitUWkNnCRNy/k\nYmPdCFhnnFEWWzfGmMhQausdVc0RkRG4ZB0DTFHV1SIyAUhV1TnkJ/d04BDwN1XdCSAij+AOHAAT\nVPWXsvggxhhjSieq6ncMBSQmJmpqaqrfYRhjTIUiIstVNbG0chHdDYMxxpiCLOkbY0wUsaRvjDFR\nxJK+McZEEUv6xhgTRSzpG2NMFLGkb4wxUcSSvjHGRBFL+sYYE0Us6RtjTBSxpG+MMVHEkr4xxkQR\nS/rGGBNFLOkbY0wUsaRvjDFRxJK+McZEEUv6xhgTRSzpG2NMFLGkb4wxUcSSvjHGRJGgkr6I9BaR\ntSKyXkRGF7F8iIhkishK73FLwLJDAfPnhDJ4Y4wxR6dyaQVEJAaYDPQCMoAUEZmjqumFir6hqiOK\n2ES2qrY9/lCNMcYcr2DO9DsC61V1g6oeAGYAfcs2LGOMMWUhmKTfCPghYDrDm1fYNSLytYi8KSKn\nBsyPFZFUEVkqIlcW9QYiMswrk5qZmRl89MYYY45KqC7kzgXiVbU18CHwasCypqqaCFwPTBKR0wuv\nrKpJqpqoqon169cPUUjGGGMKCybpbwUCz9wbe/PyqOpOVf3dm3wZ6BCwbKv3vAH4GGh3HPEaY4w5\nDsEk/RSgmYgkiEhVYABQoBWOiDQMmOwDrPHm1xaRE7zX9YDOQOELwMYYY8pJqa13VDVHREYA84EY\nYIqqrhaRCUCqqs4BRolIHyAH+AUY4q3eAnhRRA7jDjCPF9HqxxhjTDkRVfU7hgISExM1NTXV7zCM\nMaZCEZHl3vXTEtkducYYE0Us6RtjTBSxpG+MMVHEkr4xxkSRiEn6yckQHw+VKrnn5GS/IzLGmPBT\napPNiiA5GYYNg6wsN715s5sGGDTIv7iMMSbcRMSZ/tix+Qk/V1aWm2+MMSZfRCT9LVuObr4xxkSr\niEj6TZoc3XxjjIlWEZH0J06EuLiC8+Li3HxjjDH5IiLpDxoESUnQtCmIuOekJLuIa4wxhUVE6x1w\nCd6SvDHGlCwizvSNMcYEx5K+McZEEUv6xhgTRSzpG2NMFLGkb4wxUSTsRs4SkUxg83Fsoh7wc4jC\nqehsXxRk+6Mg2x/5ImFfNFXV+qUVCrukf7xEJDWYIcOige2Lgmx/FGT7I1807Qur3jHGmChiSd8Y\nY6JIJCb9JL8DCCO2Lwqy/VGQ7Y98UbMvIq5O3xhjTPEi8UzfGGNMMSzpG2NMFImYpC8ivUVkrYis\nF5HRfsfjJxE5VUQWiUi6iKwWkbv8jslvIhIjIitE5B2/Y/GbiNQSkTdF5FsRWSMinfyOyU8ico/3\nPflGRKaLSKzfMZWliEj6IhIDTAYuAVoCA0Wkpb9R+SoHuFdVWwLnA3dG+f4AuAtY43cQYeIZ4H1V\nbQ60IYr3i4g0AkYBiap6NhADDPA3qrIVEUkf6AisV9UNqnoAmAH09Tkm36jqdlX9ynu9F/elbuRv\nVP4RkcbAZcDLfsfiNxE5CbgA+DeAqh5Q1V/9jcp3lYFqIlIZiAO2+RxPmYqUpN8I+CFgOoMoTnKB\nRCQeaAd86W8kvpoE/A9w2O9AwkACkAm84lV3vSwi1f0Oyi+quhV4CtgCbAd2q+oH/kZVtiIl6Zsi\niMiJwCzgblXd43c8fhCRy4GfVHW537GEicpAe+B5VW0H7Aei9hqYiNTG1QokAKcA1UVksL9Rla1I\nSfpbgVMDpht786KWiFTBJfxkVf2v3/H4qDPQR0Q24ar9/iQiU/0NyVcZQIaq5v7yexN3EIhWFwIb\nVTVTVQ8C/wX+6HNMZSpSkn4K0ExEEkSkKu5CzByfY/KNiAiuznaNqj7tdzx+UtUxqtpYVeNx/xcL\nVTWiz+RKoqo/Aj+IyFnerJ5Auo8h+W0LcL6IxHnfm55E+IXtiBgYXVVzRGQEMB939X2Kqq72OSw/\ndQZuAFaJyEpv3v2q+p6PMZnwMRJI9k6QNgBDfY7HN6r6pYi8CXyFa/W2ggjvksG6YTDGmCgSKdU7\nxhhjgmBJ3xhjooglfWOMiSKW9I0xJopY0jfGmChiSd8YY6KIJX1jjIki/x9rbcADHh06twAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f708b9c0048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VNW5//HPE+7hfrMqQcBqhXARMEU9SCGIFm/wQ6nl\nZsVqUVul9dKWg5dalFP1UEQ9HFv0V0+rEcoLj3cs7a9iUWuRQBUKSKHcDKACCoioGHh+f6xJMgm5\nDGQyO5l836/XvDJ7z9p7P7OTPLNm7bXXMndHRETSS0bUAYiISPIpuYuIpCEldxGRNKTkLiKShpTc\nRUTSkJK7iEgaUnKXcplZAzPbb2YnJbNslMzsFDNLet9fMxtmZpvjlteZ2aBEyh7DsR4zs6nHun0l\n+73HzP4n2fuV6DSMOgBJDjPbH7eYCXwBHIotX+vueUezP3c/BLRIdtn6wN1PS8Z+zOwaYIK7D4nb\n9zXJ2LekPyX3NOHuxck1VjO8xt3/X0XlzayhuxemIjYRST01y9QTsa/dvzezuWb2CTDBzM42s7+Z\n2R4z22FmD5lZo1j5hmbmZtY1tvxk7PWXzewTM3vTzLodbdnY6xeY2T/NbK+ZPWxmb5jZxAriTiTG\na81sg5l9bGYPxW3bwMweMLPdZrYRGF7J+bnNzOaVWTfbzGbGnl9jZmtj7+dfsVp1RfsqMLMhseeZ\nZvZELLbVwBllyt5uZhtj+11tZiNi63sD/wUMijV57Yo7t3fFbX9d7L3vNrNnzeyERM5NVcxsVCye\nPWb2ipmdFvfaVDPbbmb7zOzduPd6lpmtiK3/wMz+M9HjSQ1wdz3S7AFsBoaVWXcPcBC4hPCh3gz4\nOnAm4RvcycA/gRti5RsCDnSNLT8J7AJygEbA74Enj6HsccAnwMjYazcDXwITK3gvicT4HNAa6Ap8\nVPTegRuA1UAW0B5YEv7kyz3OycB+oHncvj8EcmLLl8TKGDAU+AzoE3ttGLA5bl8FwJDY8xnAq0Bb\noAuwpkzZy4ETYr+TcbEYvhJ77Rrg1TJxPgncFXt+fizGvkBT4L+BVxI5N+W8/3uA/4k97xGLY2js\ndzQVWBd73hPYAhwfK9sNODn2fBkwNva8JXBm1P8L9fmhmnv98rq7v+Duh939M3df5u5L3b3Q3TcC\nc4DBlWy/wN3z3f1LII+QVI627MXA2+7+XOy1BwgfBOVKMMZfuPted99MSKRFx7oceMDdC9x9N3Bv\nJcfZCPyD8KEDcB7wsbvnx15/wd03evAK8Geg3IumZVwO3OPuH7v7FkJtPP648919R+x38hThgzkn\ngf0CjAcec/e33f1zYAow2Myy4spUdG4qMwZ43t1fif2O7iV8QJwJFBI+SHrGmvY2xc4dhA/pU82s\nvbt/4u5LE3wfUgOU3OuX9+IXzKy7mb1kZu+b2T5gGtChku3fj3t+gMovolZU9sT4ONzdCTXdciUY\nY0LHItQ4K/MUMDb2fFxsuSiOi81sqZl9ZGZ7CLXmys5VkRMqi8HMJprZO7Hmjz1A9wT3C+H9Fe/P\n3fcBHwOd4socze+sov0eJvyOOrn7OuAWwu/hw1gz3/GxolcB2cA6M3vLzC5M8H1IDVByr1/KdgP8\nNaG2eoq7twLuJDQ71KQdhGYSAMzMKJ2MyqpOjDuAznHLVXXVnA8MM7NOhBr8U7EYmwELgF8Qmkza\nAH9MMI73K4rBzE4GHgGuB9rH9vtu3H6r6ra5ndDUU7S/loTmn20JxHU0+80g/M62Abj7k+4+kNAk\n04BwXnD3de4+htD09kvgaTNrWs1Y5BgpuddvLYG9wKdm1gO4NgXHfBHob2aXmFlD4IdAxxqKcT7w\nIzPrZGbtgZ9WVtjd3wdeB/4HWOfu62MvNQEaAzuBQ2Z2MXDuUcQw1czaWLgP4Ia411oQEvhOwufc\n9wg19yIfAFlFF5DLMRe42sz6mFkTQpJ9zd0r/CZ0FDGPMLMhsWP/mHCdZKmZ9TCz3NjxPos9DhPe\nwBVm1iFW098be2+HqxmLHCMl9/rtFuBKwj/urwkXPmuUu38AfBuYCewGvgr8ndAvP9kxPkJoG19F\nuNi3IIFtniJcIC1uknH3PcBNwDOEi5KjCR9SifgZ4RvEZuBl4Hdx+10JPAy8FStzGhDfTv0nYD3w\ngZnFN68Ubf8HQvPIM7HtTyK0w1eLu68mnPNHCB88w4ERsfb3JsD9hOsk7xO+KdwW2/RCYK2F3lgz\ngG+7+8HqxiPHxkKTp0g0zKwBoRlgtLu/FnU8IulCNXdJOTMbHmumaALcQehl8VbEYYmkFSV3icI5\nwEbCV/5vAqPcvaJmGRE5BmqWERFJQ6q5i4ikocgGDuvQoYN37do1qsOLiNRJy5cv3+XulXUfBiJM\n7l27diU/Pz+qw4uI1ElmVtWd1oCaZURE0pKSu4hIGlJyFxFJQ5qJSaSe+PLLLykoKODzzz+POhRJ\nQNOmTcnKyqJRo4qGFqqckrtIPVFQUEDLli3p2rUrYTBOqa3cnd27d1NQUEC3bt2q3qAcdapZJi8P\nunaFjIzwM++opnwWqd8+//xz2rdvr8ReB5gZ7du3r9a3rDpTc8/Lg0mT4MCBsLxlS1gGGF/tcfBE\n6gcl9rqjur+rOlNzv+22ksRe5MCBsF5EREqrM8l969ajWy8itcvu3bvp27cvffv25fjjj6dTp07F\nywcPJjbs+1VXXcW6desqLTN79mzyktRme8455/D2228nZV+pVmeaZU46KTTFlLdeRJIvLy98M966\nNfyfTZ9evSbQ9u3bFyfKu+66ixYtWnDrrbeWKuPuuDsZGeXXOx9//PEqj/ODH/zg2INMI3Wm5j59\nOmRmll6XmRnWi0hyFV3j2rIF3EuucdVEJ4YNGzaQnZ3N+PHj6dmzJzt27GDSpEnk5OTQs2dPpk2b\nVly2qCZdWFhImzZtmDJlCqeffjpnn302H374IQC33347s2bNKi4/ZcoUBgwYwGmnncZf//pXAD79\n9FMuu+wysrOzGT16NDk5OVXW0J988kl69+5Nr169mDp1KgCFhYVcccUVxesfeughAB544AGys7Pp\n06cPEyZMSPo5S0SdqbkX1RiSWZMQkfJVdo2rJv7n3n33XX73u9+Rk5MDwL333ku7du0oLCwkNzeX\n0aNHk52dXWqbvXv3MnjwYO69915uvvlmfvOb3zBlypQj9u3uvPXWWzz//PNMmzaNP/zhDzz88MMc\nf/zxPP3007zzzjv079+/0vgKCgq4/fbbyc/Pp3Xr1gwbNowXX3yRjh07smvXLlatWgXAnj17ALj/\n/vvZsmULjRs3Ll6XanWm5g7hj2rzZjh8OPxUYhepGam+xvXVr361OLEDzJ07l/79+9O/f3/Wrl3L\nmjVrjtimWbNmXHDBBQCcccYZbN68udx9X3rppUeUef311xkzZgwAp59+Oj179qw0vqVLlzJ06FA6\ndOhAo0aNGDduHEuWLOGUU05h3bp1TJ48mUWLFtG6dWsAevbsyYQJE8jLyzvmm5Cqq04ldxFJjYqu\nZdXUNa7mzZsXP1+/fj0PPvggr7zyCitXrmT48OHl9vdu3Lhx8fMGDRpQWFhY7r6bNGlSZZlj1b59\ne1auXMmgQYOYPXs21157LQCLFi3iuuuuY9myZQwYMIBDhw4l9biJUHIXkSNEeY1r3759tGzZklat\nWrFjxw4WLVqU9GMMHDiQ+fPnA7Bq1apyvxnEO/PMM1m8eDG7d++msLCQefPmMXjwYHbu3Im7861v\nfYtp06axYsUKDh06REFBAUOHDuX+++9n165dHCjbxpUCdabNXURSJ8prXP379yc7O5vu3bvTpUsX\nBg4cmPRj3HjjjXznO98hOzu7+FHUpFKerKws7r77boYMGYK7c8kll3DRRRexYsUKrr76atwdM+O+\n++6jsLCQcePG8cknn3D48GFuvfVWWrZsmfT3UJXI5lDNyclxTdYhkjpr166lR48eUYdRKxQWFlJY\nWEjTpk1Zv349559/PuvXr6dhw9pV3y3vd2Zmy909p4JNitWudyIikgL79+/n3HPPpbCwEHfn17/+\nda1L7NWVXu9GRCQBbdq0Yfny5VGHUaN0QVVEJA0puYuIpCEldxGRNKTkLiKShpTcRSQlcnNzj7gh\nadasWVx//fWVbteiRQsAtm/fzujRo8stM2TIEKrqWj1r1qxSNxNdeOGFSRn35a677mLGjBnV3k+y\nKbmLSEqMHTuWefPmlVo3b948xo4dm9D2J554IgsWLDjm45dN7gsXLqRNmzbHvL/aTsldRFJi9OjR\nvPTSS8UTc2zevJnt27czaNCg4n7n/fv3p3fv3jz33HNHbL9582Z69eoFwGeffcaYMWPo0aMHo0aN\n4rPPPisud/311xcPF/yzn/0MgIceeojt27eTm5tLbm4uAF27dmXXrl0AzJw5k169etGrV6/i4YI3\nb95Mjx49+N73vkfPnj05//zzSx2nPG+//TZnnXUWffr0YdSoUXz88cfFxy8aArhowLK//OUvxZOV\n9OvXj08++eSYz2151M9dpB760Y8g2RMM9e0LsbxYrnbt2jFgwABefvllRo4cybx587j88ssxM5o2\nbcozzzxDq1at2LVrF2eddRYjRoyocB7RRx55hMzMTNauXcvKlStLDdk7ffp02rVrx6FDhzj33HNZ\nuXIlkydPZubMmSxevJgOHTqU2tfy5ct5/PHHWbp0Ke7OmWeeyeDBg2nbti3r169n7ty5PProo1x+\n+eU8/fTTlY7P/p3vfIeHH36YwYMHc+edd/Lzn/+cWbNmce+997Jp0yaaNGlS3BQ0Y8YMZs+ezcCB\nA9m/fz9NmzY9irNdNdXcRSRl4ptm4ptk3J2pU6fSp08fhg0bxrZt2/jggw8q3M+SJUuKk2yfPn3o\n06dP8Wvz58+nf//+9OvXj9WrV1c5KNjrr7/OqFGjaN68OS1atODSSy/ltddeA6Bbt2707dsXqHxY\nYQjjy+/Zs4fBgwcDcOWVV7JkyZLiGMePH8+TTz5ZfCfswIEDufnmm3nooYfYs2dP0u+QVc1dpB6q\nrIZdk0aOHMlNN93EihUrOHDgAGeccQYAeXl57Ny5k+XLl9OoUSO6du1a7jC/Vdm0aRMzZsxg2bJl\ntG3blokTJx7TfooUDRcMYcjgqpplKvLSSy+xZMkSXnjhBaZPn86qVauYMmUKF110EQsXLmTgwIEs\nWrSI7t27H3OsZanmLiIp06JFC3Jzc/nud79b6kLq3r17Oe6442jUqBGLFy9mS3kTJsf5xje+wVNP\nPQXAP/7xD1auXAmE4YKbN29O69at+eCDD3j55ZeLt2nZsmW57dqDBg3i2Wef5cCBA3z66ac888wz\nDBo06KjfW+vWrWnbtm1xrf+JJ55g8ODBHD58mPfee4/c3Fzuu+8+9u7dy/79+/nXv/5F7969+elP\nf8rXv/513n333aM+ZmVUcxeRlBo7diyjRo0q1XNm/PjxXHLJJfTu3ZucnJwqa7DXX389V111FT16\n9KBHjx7F3wBOP/10+vXrR/fu3encuXOp4YInTZrE8OHDOfHEE1m8eHHx+v79+zNx4kQGDBgAwDXX\nXEO/fv0qbYKpyG9/+1uuu+46Dhw4wMknn8zjjz/OoUOHmDBhAnv37sXdmTx5Mm3atOGOO+5g8eLF\nZGRk0LNnz+JZpZIloSF/zWw48CDQAHjM3e8t8/oDQG5sMRM4zt0r7WOkIX9FUktD/tY9NTrkr5k1\nAGYD5wEFwDIze97di69SuPtNceVvBPolHr6IiCRbIm3uA4AN7r7R3Q8C84CRlZQfC8xNRnAiInJs\nEknunYD34pYLYuuOYGZdgG7AK9UPTUSSLaqZ1+ToVfd3lezeMmOABe5e7lTfZjbJzPLNLH/nzp1J\nPrSIVKZp06bs3r1bCb4OcHd2795drRubEuktsw3oHLecFVtXnjHADyrakbvPAeZAuKCaYIwikgRZ\nWVkUFBSgilXd0LRpU7Kyso55+0SS+zLgVDPrRkjqY4BxZQuZWXegLfDmMUcjIjWmUaNGdOvWLeow\nJEWqbJZx90LgBmARsBaY7+6rzWyamY2IKzoGmOf6ziciErmEbmJy94XAwjLr7iyzfFfywhIRkeqo\nc8MP5OfDL38Jf/87HD4cdTQiIrVTnRt+4E9/gqlTw/N27WDIEMjNhaFDoUcPqGCEUBGReiWh4Qdq\nQnWGH9i2DRYvhldeCY+iMYa+8pWSRD90KJx8spK9iKSXRIcfqJPJvaxNm0oS/eLFsGNHWN+5c0mi\nz80NyyIidVm9Su7x3GHdupJk/+qrsHt3eO2UU0oSfW5uqOmLiNQliSb3OndBtSpm0L07fP/7sGAB\nfPhhmE5s5szQJj9vHowdC8cfD716weTJ8Mwz8NFHiR8jLw+6doWMjPAzL6+m3o2IyLFJu5p7VQoL\nYcWKkjb7116Dzz4LHwr9+pW02Q8aBC1bHrl9Xh5MmgRxk6iTmQlz5sD48al7HyJSP9XbZpmjdfAg\nvPVWSTPOm2+GdQ0awNe/XtJm/2//Bs2ahZp6eZPEdOkCxzC2v4jIUVFyP0YHDoQEX5Tsly2DQ4eg\ncWM4+2z4y1/K385M/e5FpOYlbbKO+iYzE849NzwA9u2D118vSfYVOemk1MQnIpKItLugmmytWsGF\nF8KMGaGt/le/CrX4eI0bwz33RBOfiEh5lNyP0rXXwm9+E9rYAZo0CW30jz8OGzdGG5uISBEl92Mw\nfny4eOoe2uh/9avQNt+rFzzwQGijFxGJkpJ7NWVkhNr8mjWhV83NN8PAgbB6ddSRiUh9puSeJFlZ\n8MILoR/8hg2hz/y0aaHJRkQk1ZTck8gMxo2DtWvhssvgZz+DnJzQZCMikkpK7jWgY0eYOxeeey6M\na3PWWfDjH5e+q1VEpCYpudegESNCW/w114SulH36hIHMRERqmpJ7DWvdGn7965IboHJzwwXYvXuj\njUtE0puSe4rk5sLKlXDLLfDYY9CzJ7z4YtRRiUi6UnJPoczM0Dzz5pvQti1cckm4ALtzZ9SRiUi6\nUXKPwIABsHw53HVXGHM+OztcgI1oDDcRSUNK7hFp3Dh0lVyxIsz1Om5cuABbUBB1ZCKSDpTcI9ar\nF/z1r2GmqD//ObTFz5mT2PDBmhFKRCqi5F4LNGgAN90Eq1bBGWeE3jTnnhvudK1I0YxQW7aE5pwt\nW8KyEryIgJJ7rfLVr4ba+6OPhuaaPn3gl78MUwOWddttR94UdeBAWC8iouRey5iFm57WrIHzzoNb\nbw1T/K1aVbrc1q3lb1/RehGpX5Tca6lOneDZZ2HevDC8cP/+4QLsF1+E1yua+UkzQokIJJjczWy4\nma0zsw1mNqWCMpeb2RozW21mTyU3zPrJDL797VCLHzMmjDJ5xhmwdClMnx76zcfLzAzrRUSqTO5m\n1gCYDVwAZANjzSy7TJlTgX8HBrp7T+BHNRBrvdWhAzzxRLijde/eMFH38uXw0ENhRiiz8HPOnDCR\niIhIIhNkDwA2uPtGADObB4wE1sSV+R4w290/BnD3D5MdqMBFF4VJQKZMCTM+nXxymPJv6NCoIxOR\n2iaRZplOwHtxywWxdfG+BnzNzN4ws7+Z2fDydmRmk8ws38zyd+qe+2PSqhX893+H0SUzMkKXyalT\no45KRGqbZF1QbQicCgwBxgKPmlmbsoXcfY6757h7TseOHZN06Ppp8OAwENnVV8MvfgEPPhh1RCJS\nmyTSLLMN6By3nBVbF68AWOruXwKbzOyfhGSvOYhqULNmYTjhjz4KN0GddBKMGhV1VCJSGyRSc18G\nnGpm3cysMTAGeL5MmWcJtXbMrAOhmWZjEuOUCjRoAE8+GQYjGzcu9KQREakyubt7IXADsAhYC8x3\n99VmNs3MRsSKLQJ2m9kaYDHwY3ffXVNBS2mZmfD883DiiWEY4X/9K+qIRCRq5hGNM5uTk+P5+fmR\nHDtd/fOfoZtk+/ZhzPj27aOOSESSzcyWu3tOVeV0h2oa+drXwqTcW7fCyJHw+edRRyQiUVFyTzPn\nnAO//S288QZceWViQweLSPpJpLeM1DHf/naovf/kJ2Gc9/vuizoiEUk1Jfc0deutsGkT3H9/SPDX\nXx91RCKSSkruacosjD2zdSvccAN07gwXXxx1VCKSKmpzT2MNG4Yhg/v2DU01y5dHHZGIpIqSe5pr\n0SKMJtmhQ6i5b9kSdUQikgpK7vXACSfAyy/DZ5/BBRfAxx9HHZGI1DQl93oiOxueeSZMun3ppSUz\nOolIelJyr0dyc8P476++GuZpjejmZBFJAfWWqWcmTAhzst5xR+gieffdUUckIjVByb0euu22kODv\nuQe6dYPvfjfqiEQk2ZTc6yEzeOQRKCiASZOgUyf45jejjkpEkklt7vVUo0Ywfz707AmjR8M770Qd\nkYgkk5J7PdaqFbz0ErRuHSbfLiiIOiIRSRYl93ouKwsWLoR9+0KC37cv6ohEJBmU3IU+feDpp2HN\nmtBE8+WXUUckItWl5C4AnHcezJkDf/oTXHut+sCL1HXqLSPFrroqDBN8992hi+Qdd0QdkYgcKyV3\nKeXnPw994O+8M9zkdMUVUUckIsdCyV1KMYPHHoNt2+Dqq0Mf+KFDo45KRI6W2tzlCI0bhwusX/ta\nGGRs9eqoIxKRo6XkLuVq0yb0gW/WDC68EHbsiDoiETkaSu5SoS5dQoLfvTtM9LF/f/nl8vJC+3xG\nRviZl5fKKEWkPEruUqn+/cMwBe+8E6bqKyws/XpeXhifZsuW0H1yy5awrAQvEi0ld6nShRfC7Nnh\nTtYbbyzdB/622+DAgdLlDxwI60UkOuotIwm59trQB/6++0LTy09/GtZv3Vp++YrWi0hqqOYuCfuP\n/4AxY2DKFJg3L6w76aTyy1a0XkRSI6HkbmbDzWydmW0wsynlvD7RzHaa2duxxzXJD1WilpEBjz8O\ngwbBlVfCa6/B9OmQmVm6XGZmWC8i0akyuZtZA2A2cAGQDYw1s+xyiv7e3fvGHo8lOU6pJZo2hWef\nDcMTjBwJOTlhTJouXcINUF26hOXx46OOVKR+S6TNfQCwwd03ApjZPGAksKYmA5Paq127cHH1rLPg\nggvgzTeVzEVqm0SaZToB78UtF8TWlXWZma00swVm1rm8HZnZJDPLN7P8nTt3HkO4UlucfDK8+CK8\n/z6MGHFkjxkRiVayLqi+AHR19z7An4DfllfI3ee4e46753Ts2DFJh5aoDBgAc+fCsmUwbhwcOhR1\nRCJSJJHkvg2Ir4lnxdYVc/fd7v5FbPEx4IzkhCe13ciR8OCD8NxzMHEiHDwYdUQiAom1uS8DTjWz\nboSkPgYYF1/AzE5w96LRR0YAa5MapdRqN94Ypue7/fbQTPP002F+VhGJTpU1d3cvBG4AFhGS9nx3\nX21m08xsRKzYZDNbbWbvAJOBiTUVsNROt90WukkuXgyDB8P27VFHJFK/mUc0n1pOTo7n5+dHcmyp\nOYsWwWWXQfv28Ic/QI8eUUckkl7MbLm751RVTneoSlJ985uwZAl88QUMHAivvx51RCL1k5K7JF3/\n/qHve8eOMGxYaIMXkdRScpca0a0bvPFGSPTf+hY8/HDUEYnUL0ruUmM6dIA//zl0l5w8GX7yEzh8\nOOqoROoHJXepUc2awYIF8P3vw3/+J0yYENrjRaRmaTx3qXENGsB//VcYBnjKlNAX/plnoHXrqCMT\nSV+quUtKmIUJPp54IgwVPGgQFBREHZVI+lJyl5SaMAFefhk2b4azz4Z//CPqiETSk5K7pNywYaEv\n/KFDcM458Je/RB2RSPpRcpdI9O0b+sKfeCKcfz78/vdRRySSXpTcJTJduoQ7WM88M8zNOnNm1BGJ\npA8ld4lUu3bwxz/C6NFwyy1w003qCy+SDEruErmmTUOzzA9/CLNmhVr8559HHZVI3aZ+7lIrZGTA\nAw9A585w663wwQdhIu62baOOTKRuUs1dag2z0DQzdy787W+hJ83WrVFHJVI3KblLrTNmTBgXftu2\n0Bf+nXeijkik7lFyl1ppyJBwJ6tZuJv1z3+OOiKRukXJXWqt3r1D80yXLnDBBZCXF3VEInWHkrvU\nallZoQY/cGAYuuC++yCimSFF6hT1lpFar02bMB/rxIlhVMn33oMHHwyjTYpUZe9eWLs2PPbvr7q8\nWfXLVPX64MHQs2fVx6kOJXepE5o0Cc0yWVkwYwZs3x6WmzWLOjKpDdxh586QwNesKf1z+/aoozvS\nI48ouYsUy8gIE35kZYU7WYcNg+efh/bto45MUsU9DBVdXhLfvbukXIsWkJ0N550XfvboER5V3TeR\nSJNfVWUS2UeLFlWXqS4ld6lzfvhD6NQptMEPHBiGEO7WLeqoJJkOHYJNm8pP4vFNK+3bh+R92WUl\nSTw7O/x9JNK8ks6U3KVOGj0avvIVGDEi9IVfuDBMxi11y8GDsH59SeIuSuLr1pWejvHEE0PSvuqq\n0km8Y8foYq/tlNylzho0CN54I3STHDw4zNX6zW9GHZWU59NPQ8IuWwvfsCHU0iHUtLt2DUn7/PNL\nN6doSsajp+QudVp2dhgX/sIL4eKL4dFH4cor9ZU8Vb74AnbsCPPi7thx5CN+fZGGDeGUU8IFxdGj\nS5L4aadBZmZ07yXdmEfUaTgnJ8fz8/MjObakn3374NJLw52sjRqFJpv4x/HHl7/ctq0+CMpyh08+\nqTpZ79gBH3985PYZGXDccXDCCSWPLl1Kkvgpp0Djxql/X+nCzJa7e05V5VRzl7TQqhVccQX8/e/w\n0Uch6bRrFxLQ22/Dhx9CYeGR2zVqFBJR2eRf3gdCu3Z1+4Pg8GHYtaviRB2//sCBI7dv0iQk6uOP\nD7XsIUNKJ/Ci1447Tvcg1AYJJXczGw48CDQAHnP3eysodxmwAPi6u6taLimTlwff/35JUvr009Ce\nO2cOjB8fEttHH4WhhIse779fejnRD4Kqvg0UfRAcOgRffpnY4+DB5JYrKvvppyVJ+4MPyn9frVqV\nJOczzyxJ0mUTd5s2dfvDrb6pslnGzBoA/wTOAwqAZcBYd19TplxL4CWgMXBDVcldzTKSTF27wpYt\nR67v0gU2UiEYAAAHaElEQVQ2bz66fR0+HGr+ZZN/2eWiR3kJM5UaNw4fPOU9mjcvP1HHJ3C1c9ct\nyWyWGQBscPeNsR3PA0YCa8qUuxu4D/jxUcYqUm0Vjft+LOPBZ2SE/tPt21d9F2HRB0HZD4CPPgoX\nDitKuo0aVZ6UEy3ToIFq01K+RJJ7J+C9uOUC4Mz4AmbWH+js7i+ZWYXJ3cwmAZMATjrppKOPVqQC\nJ51Ufs29pv/M4j8IsrNr9lgiR6Pao0KaWQYwE7ilqrLuPsfdc9w9p6PuPpAkmj79yOaFzMywXqQ+\nSiS5bwM6xy1nxdYVaQn0Al41s83AWcDzZlZlm5BIsowfHy6edukSmim6dCm5mCpSHyXSLLMMONXM\nuhGS+hhgXNGL7r4X6FC0bGavAreqt4yk2vjxSuYiRaqsubt7IXADsAhYC8x399VmNs3MRtR0gCIi\ncvQS6ufu7guBhWXW3VlB2SHVD0tERKpD0+yJiKQhJXcRkTSk5C4ikoaU3EVE0pCSu4hIGlJyFxFJ\nQ0ruIiJpSMldRCQNKbmLiKQhJXcRkTSk5C4ikoaU3EWSKC8vTPmXkRF+5uVFHZHUVwkNHCYiVcvL\ng0mTSibp3rIlLIOGIpbUU81dJEluu60ksRc5cCCsF0k1JXeRJEnmJN0i1aXkLpIkFU3GrbngJQpK\n7iJJokm6pTZRchdJEk3SLbWJesuIJJEm6ZbaQjV3EZE0pOQuIpKGlNxFRNKQkruISBpSchcRSUNK\n7iIiaUjJXSQNaXRKUT93kTSj0SkFVHMXSTsanVIgweRuZsPNbJ2ZbTCzKeW8fp2ZrTKzt83sdTPL\nTn6oIpIIjU4pkEByN7MGwGzgAiAbGFtO8n7K3Xu7e1/gfmBm0iMVkYRodEqBxGruA4AN7r7R3Q8C\n84CR8QXcfV/cYnPAkxeiiBwNjU4pkFhy7wS8F7dcEFtXipn9wMz+Rai5Ty5vR2Y2yczyzSx/586d\nxxKviFRBo1MKJPGCqrvPdvevAj8Fbq+gzBx3z3H3nI4dOybr0CJSxvjxsHkzHD4cfiqx1z+JJPdt\nQOe45azYuorMA/5PdYISEZHqSSS5LwNONbNuZtYYGAM8H1/AzE6NW7wIWJ+8EEVE5GhVmdzdvRC4\nAVgErAXmu/tqM5tmZiNixW4ws9Vm9jZwM3BljUUsInWG7pSNjrlH07ElJyfH8/PzIzm2iNS8snfK\nQui1o4u71WNmy909p6pyukNVRGqE7pSNlpK7iNQI3SkbLSV3EakRulM2WkruIlIjdKdstJTcRaRG\n6E7ZaGk8dxGpMePHK5lHRTV3EZE0pOQuIpKGlNxFJO3Vxztl1eYuImmtvs4pq5q7iKS1+nqnrJK7\niKS1+nqnrJK7iKS1+nqnrJK7iKS1+nqnrJK7iKS1+nqnrJK7iKS92jKnbCq7ZKorpIhICqS6S6Zq\n7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLpnqLSMikiKpnLxENXcR\nkTSk5C4ikoaU3EVE0pCSu4hIGlJyFxFJQ+bu0RzYbCew5Rg37wDsSmI4dZ3OR2k6HyV0LkpLh/PR\nxd07VlUosuReHWaW7+45UcdRW+h8lKbzUULnorT6dD7ULCMikoaU3EVE0lBdTe5zog6gltH5KE3n\no4TORWn15nzUyTZ3ERGpXF2tuYuISCWU3EVE0lCdS+5mNtzM1pnZBjObEnU8UTGzzma22MzWmNlq\nM/th1DHVBmbWwMz+bmYvRh1L1MysjZktMLN3zWytmZ0ddUxRMbObYv8n/zCzuWbWNOqYalqdSu5m\n1gCYDVwAZANjzSw72qgiUwjc4u7ZwFnAD+rxuYj3Q2Bt1EHUEg8Cf3D37sDp1NPzYmadgMlAjrv3\nAhoAY6KNqubVqeQODAA2uPtGdz8IzANGRhxTJNx9h7uviD3/hPCP2ynaqKJlZlnARcBjUccSNTNr\nDXwD+L8A7n7Q3fdEG1WkGgLNzKwhkAlsjzieGlfXknsn4L245QLqeUIDMLOuQD9gabSRRG4W8BPg\ncNSB1ALdgJ3A47FmqsfMrHnUQUXB3bcBM4CtwA5gr7v/Mdqoal5dS+5Shpm1AJ4GfuTu+6KOJypm\ndjHwobsvjzqWWqIh0B94xN37AZ8C9fIalZm1JXzD7wacCDQ3swnRRlXz6lpy3wZ0jlvOiq2rl8ys\nESGx57n7/0YdT8QGAiPMbDOhuW6omT0ZbUiRKgAK3L3o29wCQrKvj4YBm9x9p7t/Cfwv8G8Rx1Tj\n6lpyXwacambdzKwx4aLI8xHHFAkzM0J76lp3nxl1PFFz93939yx370r4u3jF3dO+dlYRd38feM/M\nToutOhdYE2FIUdoKnGVmmbH/m3OpBxeX69QE2e5eaGY3AIsIV7x/4+6rIw4rKgOBK4BVZvZ2bN1U\nd18YYUxSu9wI5MUqQhuBqyKOJxLuvtTMFgArCL3M/k49GIZAww+IiKShutYsIyIiCVByFxFJQ0ru\nIiJpSMldRCQNKbmLiKQhJXcRkTSk5C4ikob+PyYufzJSbaZ7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f707af3d5c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining CNNs and RNNs to process long sequences\n",
    "\n",
    "\n",
    "Because 1D convnets process input patches independently, they are not sensitive to the order of the timesteps (beyond a local scale, the \n",
    "size of the convolution windows), unlike RNNs. Of course, in order to be able to recognize longer-term patterns, one could stack many \n",
    "convolution layers and pooling layers, resulting in upper layers that would \"see\" long chunks of the original inputs -- but that's still a \n",
    "fairly weak way to induce order-sensitivity. One way to evidence this weakness is to try 1D convnets on the temperature forecasting problem \n",
    "from the previous section, where order-sensitivity was key to produce good predictions. Let's see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We reuse the following variables defined in the last section:\n",
    "# float_data, train_gen, val_gen, val_steps\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "data_dir = '/home/ubuntu/data/'\n",
    "fname = os.path.join(data_dir, 'jena_climate_2009_2016.csv')\n",
    "\n",
    "f = open(fname)\n",
    "data = f.read()\n",
    "f.close()\n",
    "\n",
    "lines = data.split('\\n')\n",
    "header = lines[0].split(',')\n",
    "lines = lines[1:]\n",
    "\n",
    "float_data = np.zeros((len(lines), len(header) - 1))\n",
    "for i, line in enumerate(lines):\n",
    "    values = [float(x) for x in line.split(',')[1:]]\n",
    "    float_data[i, :] = values\n",
    "    \n",
    "mean = float_data[:200000].mean(axis=0)\n",
    "float_data -= mean\n",
    "std = float_data[:200000].std(axis=0)\n",
    "float_data /= std\n",
    "\n",
    "def generator(data, lookback, delay, min_index, max_index,\n",
    "              shuffle=False, batch_size=128, step=6):\n",
    "    if max_index is None:\n",
    "        max_index = len(data) - delay - 1\n",
    "    i = min_index + lookback\n",
    "    while 1:\n",
    "        if shuffle:\n",
    "            rows = np.random.randint(\n",
    "                min_index + lookback, max_index, size=batch_size)\n",
    "        else:\n",
    "            if i + batch_size >= max_index:\n",
    "                i = min_index + lookback\n",
    "            rows = np.arange(i, min(i + batch_size, max_index))\n",
    "            i += len(rows)\n",
    "\n",
    "        samples = np.zeros((len(rows),\n",
    "                           lookback // step,\n",
    "                           data.shape[-1]))\n",
    "        targets = np.zeros((len(rows),))\n",
    "        for j, row in enumerate(rows):\n",
    "            indices = range(rows[j] - lookback, rows[j], step)\n",
    "            samples[j] = data[indices]\n",
    "            targets[j] = data[rows[j] + delay][1]\n",
    "        yield samples, targets\n",
    "        \n",
    "lookback = 1440\n",
    "step = 6\n",
    "delay = 144\n",
    "batch_size = 128\n",
    "\n",
    "train_gen = generator(float_data,\n",
    "                      lookback=lookback,\n",
    "                      delay=delay,\n",
    "                      min_index=0,\n",
    "                      max_index=200000,\n",
    "                      shuffle=True,\n",
    "                      step=step, \n",
    "                      batch_size=batch_size)\n",
    "val_gen = generator(float_data,\n",
    "                    lookback=lookback,\n",
    "                    delay=delay,\n",
    "                    min_index=200001,\n",
    "                    max_index=300000,\n",
    "                    step=step,\n",
    "                    batch_size=batch_size)\n",
    "test_gen = generator(float_data,\n",
    "                     lookback=lookback,\n",
    "                     delay=delay,\n",
    "                     min_index=300001,\n",
    "                     max_index=None,\n",
    "                     step=step,\n",
    "                     batch_size=batch_size)\n",
    "\n",
    "# This is how many steps to draw from `val_gen`\n",
    "# in order to see the whole validation set:\n",
    "val_steps = (300000 - 200001 - lookback) // batch_size\n",
    "\n",
    "# This is how many steps to draw from `test_gen`\n",
    "# in order to see the whole test set:\n",
    "test_steps = (len(float_data) - 300001 - lookback) // batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "500/500 [==============================] - 11s - loss: 0.4106 - val_loss: 0.4331\n",
      "Epoch 2/20\n",
      "500/500 [==============================] - 10s - loss: 0.3604 - val_loss: 0.4252\n",
      "Epoch 3/20\n",
      "500/500 [==============================] - 10s - loss: 0.3364 - val_loss: 0.4371\n",
      "Epoch 4/20\n",
      "500/500 [==============================] - 10s - loss: 0.3224 - val_loss: 0.4271\n",
      "Epoch 5/20\n",
      "500/500 [==============================] - 10s - loss: 0.3085 - val_loss: 0.4420\n",
      "Epoch 6/20\n",
      "500/500 [==============================] - 10s - loss: 0.2988 - val_loss: 0.4774\n",
      "Epoch 7/20\n",
      "500/500 [==============================] - 10s - loss: 0.2916 - val_loss: 0.4484\n",
      "Epoch 8/20\n",
      "500/500 [==============================] - 10s - loss: 0.2853 - val_loss: 0.4446\n",
      "Epoch 9/20\n",
      "500/500 [==============================] - 10s - loss: 0.2788 - val_loss: 0.4728\n",
      "Epoch 10/20\n",
      "500/500 [==============================] - 10s - loss: 0.2709 - val_loss: 0.4370\n",
      "Epoch 11/20\n",
      "500/500 [==============================] - 10s - loss: 0.2694 - val_loss: 0.4472\n",
      "Epoch 12/20\n",
      "500/500 [==============================] - 10s - loss: 0.2633 - val_loss: 0.4499\n",
      "Epoch 13/20\n",
      "500/500 [==============================] - 10s - loss: 0.2598 - val_loss: 0.4428\n",
      "Epoch 14/20\n",
      "500/500 [==============================] - 10s - loss: 0.2572 - val_loss: 0.4471\n",
      "Epoch 15/20\n",
      "500/500 [==============================] - 10s - loss: 0.2543 - val_loss: 0.4597\n",
      "Epoch 16/20\n",
      "500/500 [==============================] - 10s - loss: 0.2512 - val_loss: 0.4420\n",
      "Epoch 17/20\n",
      "500/500 [==============================] - 10s - loss: 0.2481 - val_loss: 0.4462\n",
      "Epoch 18/20\n",
      "500/500 [==============================] - 10s - loss: 0.2450 - val_loss: 0.4486\n",
      "Epoch 19/20\n",
      "500/500 [==============================] - 10s - loss: 0.2417 - val_loss: 0.4558\n",
      "Epoch 20/20\n",
      "500/500 [==============================] - 10s - loss: 0.2399 - val_loss: 0.4512\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.Conv1D(32, 5, activation='relu',\n",
    "                        input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.MaxPooling1D(3))\n",
    "model.add(layers.Conv1D(32, 5, activation='relu'))\n",
    "model.add(layers.MaxPooling1D(3))\n",
    "model.add(layers.Conv1D(32, 5, activation='relu'))\n",
    "model.add(layers.GlobalMaxPooling1D())\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "cnn_history = model.fit_generator(train_gen,\n",
    "                                  steps_per_epoch=500,\n",
    "                                  epochs=20,\n",
    "                                  validation_data=val_gen,\n",
    "                                  validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are our training and validation Mean Absolute Errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f708b9c0080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VNW5//HPE+7hfrMqQcBqhXARMEU9SCGIFm/wQ6nl\nZsVqUVul9dKWg5dalFP1UEQ9HFv0V0+rEcoLj3cs7a9iUWuRQBUKSKHcDKACCoioGHh+f6xJMgm5\nDGQyO5l836/XvDJ7z9p7P7OTPLNm7bXXMndHRETSS0bUAYiISPIpuYuIpCEldxGRNKTkLiKShpTc\nRUTSkJK7iEgaUnKXcplZAzPbb2YnJbNslMzsFDNLet9fMxtmZpvjlteZ2aBEyh7DsR4zs6nHun0l\n+73HzP4n2fuV6DSMOgBJDjPbH7eYCXwBHIotX+vueUezP3c/BLRIdtn6wN1PS8Z+zOwaYIK7D4nb\n9zXJ2LekPyX3NOHuxck1VjO8xt3/X0XlzayhuxemIjYRST01y9QTsa/dvzezuWb2CTDBzM42s7+Z\n2R4z22FmD5lZo1j5hmbmZtY1tvxk7PWXzewTM3vTzLodbdnY6xeY2T/NbK+ZPWxmb5jZxAriTiTG\na81sg5l9bGYPxW3bwMweMLPdZrYRGF7J+bnNzOaVWTfbzGbGnl9jZmtj7+dfsVp1RfsqMLMhseeZ\nZvZELLbVwBllyt5uZhtj+11tZiNi63sD/wUMijV57Yo7t3fFbX9d7L3vNrNnzeyERM5NVcxsVCye\nPWb2ipmdFvfaVDPbbmb7zOzduPd6lpmtiK3/wMz+M9HjSQ1wdz3S7AFsBoaVWXcPcBC4hPCh3gz4\nOnAm4RvcycA/gRti5RsCDnSNLT8J7AJygEbA74Enj6HsccAnwMjYazcDXwITK3gvicT4HNAa6Ap8\nVPTegRuA1UAW0B5YEv7kyz3OycB+oHncvj8EcmLLl8TKGDAU+AzoE3ttGLA5bl8FwJDY8xnAq0Bb\noAuwpkzZy4ETYr+TcbEYvhJ77Rrg1TJxPgncFXt+fizGvkBT4L+BVxI5N+W8/3uA/4k97xGLY2js\ndzQVWBd73hPYAhwfK9sNODn2fBkwNva8JXBm1P8L9fmhmnv98rq7v+Duh939M3df5u5L3b3Q3TcC\nc4DBlWy/wN3z3f1LII+QVI627MXA2+7+XOy1BwgfBOVKMMZfuPted99MSKRFx7oceMDdC9x9N3Bv\nJcfZCPyD8KEDcB7wsbvnx15/wd03evAK8Geg3IumZVwO3OPuH7v7FkJtPP648919R+x38hThgzkn\ngf0CjAcec/e33f1zYAow2Myy4spUdG4qMwZ43t1fif2O7iV8QJwJFBI+SHrGmvY2xc4dhA/pU82s\nvbt/4u5LE3wfUgOU3OuX9+IXzKy7mb1kZu+b2T5gGtChku3fj3t+gMovolZU9sT4ONzdCTXdciUY\nY0LHItQ4K/MUMDb2fFxsuSiOi81sqZl9ZGZ7CLXmys5VkRMqi8HMJprZO7Hmjz1A9wT3C+H9Fe/P\n3fcBHwOd4socze+sov0eJvyOOrn7OuAWwu/hw1gz3/GxolcB2cA6M3vLzC5M8H1IDVByr1/KdgP8\nNaG2eoq7twLuJDQ71KQdhGYSAMzMKJ2MyqpOjDuAznHLVXXVnA8MM7NOhBr8U7EYmwELgF8Qmkza\nAH9MMI73K4rBzE4GHgGuB9rH9vtu3H6r6ra5ndDUU7S/loTmn20JxHU0+80g/M62Abj7k+4+kNAk\n04BwXnD3de4+htD09kvgaTNrWs1Y5BgpuddvLYG9wKdm1gO4NgXHfBHob2aXmFlD4IdAxxqKcT7w\nIzPrZGbtgZ9WVtjd3wdeB/4HWOfu62MvNQEaAzuBQ2Z2MXDuUcQw1czaWLgP4Ia411oQEvhOwufc\n9wg19yIfAFlFF5DLMRe42sz6mFkTQpJ9zd0r/CZ0FDGPMLMhsWP/mHCdZKmZ9TCz3NjxPos9DhPe\nwBVm1iFW098be2+HqxmLHCMl9/rtFuBKwj/urwkXPmuUu38AfBuYCewGvgr8ndAvP9kxPkJoG19F\nuNi3IIFtniJcIC1uknH3PcBNwDOEi5KjCR9SifgZ4RvEZuBl4Hdx+10JPAy8FStzGhDfTv0nYD3w\ngZnFN68Ubf8HQvPIM7HtTyK0w1eLu68mnPNHCB88w4ERsfb3JsD9hOsk7xO+KdwW2/RCYK2F3lgz\ngG+7+8HqxiPHxkKTp0g0zKwBoRlgtLu/FnU8IulCNXdJOTMbHmumaALcQehl8VbEYYmkFSV3icI5\nwEbCV/5vAqPcvaJmGRE5BmqWERFJQ6q5i4ikocgGDuvQoYN37do1qsOLiNRJy5cv3+XulXUfBiJM\n7l27diU/Pz+qw4uI1ElmVtWd1oCaZURE0pKSu4hIGlJyFxFJQ5qJSaSe+PLLLykoKODzzz+POhRJ\nQNOmTcnKyqJRo4qGFqqckrtIPVFQUEDLli3p2rUrYTBOqa3cnd27d1NQUEC3bt2q3qAcdapZJi8P\nunaFjIzwM++opnwWqd8+//xz2rdvr8ReB5gZ7du3r9a3rDpTc8/Lg0mT4MCBsLxlS1gGGF/tcfBE\n6gcl9rqjur+rOlNzv+22ksRe5MCBsF5EREqrM8l969ajWy8itcvu3bvp27cvffv25fjjj6dTp07F\nywcPJjbs+1VXXcW6desqLTN79mzyktRme8455/D2228nZV+pVmeaZU46KTTFlLdeRJIvLy98M966\nNfyfTZ9evSbQ9u3bFyfKu+66ixYtWnDrrbeWKuPuuDsZGeXXOx9//PEqj/ODH/zg2INMI3Wm5j59\nOmRmll6XmRnWi0hyFV3j2rIF3EuucdVEJ4YNGzaQnZ3N+PHj6dmzJzt27GDSpEnk5OTQs2dPpk2b\nVly2qCZdWFhImzZtmDJlCqeffjpnn302H374IQC33347s2bNKi4/ZcoUBgwYwGmnncZf//pXAD79\n9FMuu+wysrOzGT16NDk5OVXW0J988kl69+5Nr169mDp1KgCFhYVcccUVxesfeughAB544AGys7Pp\n06cPEyZMSPo5S0SdqbkX1RiSWZMQkfJVdo2rJv7n3n33XX73u9+Rk5MDwL333ku7du0oLCwkNzeX\n0aNHk52dXWqbvXv3MnjwYO69915uvvlmfvOb3zBlypQj9u3uvPXWWzz//PNMmzaNP/zhDzz88MMc\nf/zxPP3007zzzjv079+/0vgKCgq4/fbbyc/Pp3Xr1gwbNowXX3yRjh07smvXLlatWgXAnj17ALj/\n/vvZsmULjRs3Ll6XanWm5g7hj2rzZjh8OPxUYhepGam+xvXVr361OLEDzJ07l/79+9O/f3/Wrl3L\nmjVrjtimWbNmXHDBBQCcccYZbN68udx9X3rppUeUef311xkzZgwAp59+Oj179qw0vqVLlzJ06FA6\ndOhAo0aNGDduHEuWLOGUU05h3bp1TJ48mUWLFtG6dWsAevbsyYQJE8jLyzvmm5Cqq04ldxFJjYqu\nZdXUNa7mzZsXP1+/fj0PPvggr7zyCitXrmT48OHl9vdu3Lhx8fMGDRpQWFhY7r6bNGlSZZlj1b59\ne1auXMmgQYOYPXs21157LQCLFi3iuuuuY9myZQwYMIBDhw4l9biJUHIXkSNEeY1r3759tGzZklat\nWrFjxw4WLVqU9GMMHDiQ+fPnA7Bq1apyvxnEO/PMM1m8eDG7d++msLCQefPmMXjwYHbu3Im7861v\nfYtp06axYsUKDh06REFBAUOHDuX+++9n165dHCjbxpUCdabNXURSJ8prXP379yc7O5vu3bvTpUsX\nBg4cmPRj3HjjjXznO98hOzu7+FHUpFKerKws7r77boYMGYK7c8kll3DRRRexYsUKrr76atwdM+O+\n++6jsLCQcePG8cknn3D48GFuvfVWWrZsmfT3UJXI5lDNyclxTdYhkjpr166lR48eUYdRKxQWFlJY\nWEjTpk1Zv349559/PuvXr6dhw9pV3y3vd2Zmy909p4JNitWudyIikgL79+/n3HPPpbCwEHfn17/+\nda1L7NWVXu9GRCQBbdq0Yfny5VGHUaN0QVVEJA0puYuIpCEldxGRNKTkLiKShpTcRSQlcnNzj7gh\nadasWVx//fWVbteiRQsAtm/fzujRo8stM2TIEKrqWj1r1qxSNxNdeOGFSRn35a677mLGjBnV3k+y\nKbmLSEqMHTuWefPmlVo3b948xo4dm9D2J554IgsWLDjm45dN7gsXLqRNmzbHvL/aTsldRFJi9OjR\nvPTSS8UTc2zevJnt27czaNCg4n7n/fv3p3fv3jz33HNHbL9582Z69eoFwGeffcaYMWPo0aMHo0aN\n4rPPPisud/311xcPF/yzn/0MgIceeojt27eTm5tLbm4uAF27dmXXrl0AzJw5k169etGrV6/i4YI3\nb95Mjx49+N73vkfPnj05//zzSx2nPG+//TZnnXUWffr0YdSoUXz88cfFxy8aArhowLK//OUvxZOV\n9OvXj08++eSYz2151M9dpB760Y8g2RMM9e0LsbxYrnbt2jFgwABefvllRo4cybx587j88ssxM5o2\nbcozzzxDq1at2LVrF2eddRYjRoyocB7RRx55hMzMTNauXcvKlStLDdk7ffp02rVrx6FDhzj33HNZ\nuXIlkydPZubMmSxevJgOHTqU2tfy5ct5/PHHWbp0Ke7OmWeeyeDBg2nbti3r169n7ty5PProo1x+\n+eU8/fTTlY7P/p3vfIeHH36YwYMHc+edd/Lzn/+cWbNmce+997Jp0yaaNGlS3BQ0Y8YMZs+ezcCB\nA9m/fz9NmzY9irNdNdXcRSRl4ptm4ptk3J2pU6fSp08fhg0bxrZt2/jggw8q3M+SJUuKk2yfPn3o\n06dP8Wvz58+nf//+9OvXj9WrV1c5KNjrr7/OqFGjaN68OS1atODSSy/ltddeA6Bbt2707dsXqHxY\nYQjjy+/Zs4fBgwcDcOWVV7JkyZLiGMePH8+TTz5ZfCfswIEDufnmm3nooYfYs2dP0u+QVc1dpB6q\nrIZdk0aOHMlNN93EihUrOHDgAGeccQYAeXl57Ny5k+XLl9OoUSO6du1a7jC/Vdm0aRMzZsxg2bJl\ntG3blokTJx7TfooUDRcMYcjgqpplKvLSSy+xZMkSXnjhBaZPn86qVauYMmUKF110EQsXLmTgwIEs\nWrSI7t27H3OsZanmLiIp06JFC3Jzc/nud79b6kLq3r17Oe6442jUqBGLFy9mS3kTJsf5xje+wVNP\nPQXAP/7xD1auXAmE4YKbN29O69at+eCDD3j55ZeLt2nZsmW57dqDBg3i2Wef5cCBA3z66ac888wz\nDBo06KjfW+vWrWnbtm1xrf+JJ55g8ODBHD58mPfee4/c3Fzuu+8+9u7dy/79+/nXv/5F7969+elP\nf8rXv/513n333aM+ZmVUcxeRlBo7diyjRo0q1XNm/PjxXHLJJfTu3ZucnJwqa7DXX389V111FT16\n9KBHjx7F3wBOP/10+vXrR/fu3encuXOp4YInTZrE8OHDOfHEE1m8eHHx+v79+zNx4kQGDBgAwDXX\nXEO/fv0qbYKpyG9/+1uuu+46Dhw4wMknn8zjjz/OoUOHmDBhAnv37sXdmTx5Mm3atOGOO+5g8eLF\nZGRk0LNnz+JZpZIloSF/zWw48CDQAHjM3e8t8/oDQG5sMRM4zt0r7WOkIX9FUktD/tY9NTrkr5k1\nAGYD5wEFwDIze97di69SuPtNceVvBPolHr6IiCRbIm3uA4AN7r7R3Q8C84CRlZQfC8xNRnAiInJs\nEknunYD34pYLYuuOYGZdgG7AK9UPTUSSLaqZ1+ToVfd3lezeMmOABe5e7lTfZjbJzPLNLH/nzp1J\nPrSIVKZp06bs3r1bCb4OcHd2795drRubEuktsw3oHLecFVtXnjHADyrakbvPAeZAuKCaYIwikgRZ\nWVkUFBSgilXd0LRpU7Kyso55+0SS+zLgVDPrRkjqY4BxZQuZWXegLfDmMUcjIjWmUaNGdOvWLeow\nJEWqbJZx90LgBmARsBaY7+6rzWyamY2IKzoGmOf6ziciErmEbmJy94XAwjLr7iyzfFfywhIRkeqo\nc8MP5OfDL38Jf/87HD4cdTQiIrVTnRt+4E9/gqlTw/N27WDIEMjNhaFDoUcPqGCEUBGReiWh4Qdq\nQnWGH9i2DRYvhldeCY+iMYa+8pWSRD90KJx8spK9iKSXRIcfqJPJvaxNm0oS/eLFsGNHWN+5c0mi\nz80NyyIidVm9Su7x3GHdupJk/+qrsHt3eO2UU0oSfW5uqOmLiNQliSb3OndBtSpm0L07fP/7sGAB\nfPhhmE5s5szQJj9vHowdC8cfD716weTJ8Mwz8NFHiR8jLw+6doWMjPAzL6+m3o2IyLFJu5p7VQoL\nYcWKkjb7116Dzz4LHwr9+pW02Q8aBC1bHrl9Xh5MmgRxk6iTmQlz5sD48al7HyJSP9XbZpmjdfAg\nvPVWSTPOm2+GdQ0awNe/XtJm/2//Bs2ahZp6eZPEdOkCxzC2v4jIUVFyP0YHDoQEX5Tsly2DQ4eg\ncWM4+2z4y1/K385M/e5FpOYlbbKO+iYzE849NzwA9u2D118vSfYVOemk1MQnIpKItLugmmytWsGF\nF8KMGaGt/le/CrX4eI0bwz33RBOfiEh5lNyP0rXXwm9+E9rYAZo0CW30jz8OGzdGG5uISBEl92Mw\nfny4eOoe2uh/9avQNt+rFzzwQGijFxGJkpJ7NWVkhNr8mjWhV83NN8PAgbB6ddSRiUh9puSeJFlZ\n8MILoR/8hg2hz/y0aaHJRkQk1ZTck8gMxo2DtWvhssvgZz+DnJzQZCMikkpK7jWgY0eYOxeeey6M\na3PWWfDjH5e+q1VEpCYpudegESNCW/w114SulH36hIHMRERqmpJ7DWvdGn7965IboHJzwwXYvXuj\njUtE0puSe4rk5sLKlXDLLfDYY9CzJ7z4YtRRiUi6UnJPoczM0Dzz5pvQti1cckm4ALtzZ9SRiUi6\nUXKPwIABsHw53HVXGHM+OztcgI1oDDcRSUNK7hFp3Dh0lVyxIsz1Om5cuABbUBB1ZCKSDpTcI9ar\nF/z1r2GmqD//ObTFz5mT2PDBmhFKRCqi5F4LNGgAN90Eq1bBGWeE3jTnnhvudK1I0YxQW7aE5pwt\nW8KyEryIgJJ7rfLVr4ba+6OPhuaaPn3gl78MUwOWddttR94UdeBAWC8iouRey5iFm57WrIHzzoNb\nbw1T/K1aVbrc1q3lb1/RehGpX5Tca6lOneDZZ2HevDC8cP/+4QLsF1+E1yua+UkzQokIJJjczWy4\nma0zsw1mNqWCMpeb2RozW21mTyU3zPrJDL797VCLHzMmjDJ5xhmwdClMnx76zcfLzAzrRUSqTO5m\n1gCYDVwAZANjzSy7TJlTgX8HBrp7T+BHNRBrvdWhAzzxRLijde/eMFH38uXw0ENhRiiz8HPOnDCR\niIhIIhNkDwA2uPtGADObB4wE1sSV+R4w290/BnD3D5MdqMBFF4VJQKZMCTM+nXxymPJv6NCoIxOR\n2iaRZplOwHtxywWxdfG+BnzNzN4ws7+Z2fDydmRmk8ws38zyd+qe+2PSqhX893+H0SUzMkKXyalT\no45KRGqbZF1QbQicCgwBxgKPmlmbsoXcfY6757h7TseOHZN06Ppp8OAwENnVV8MvfgEPPhh1RCJS\nmyTSLLMN6By3nBVbF68AWOruXwKbzOyfhGSvOYhqULNmYTjhjz4KN0GddBKMGhV1VCJSGyRSc18G\nnGpm3cysMTAGeL5MmWcJtXbMrAOhmWZjEuOUCjRoAE8+GQYjGzcu9KQREakyubt7IXADsAhYC8x3\n99VmNs3MRsSKLQJ2m9kaYDHwY3ffXVNBS2mZmfD883DiiWEY4X/9K+qIRCRq5hGNM5uTk+P5+fmR\nHDtd/fOfoZtk+/ZhzPj27aOOSESSzcyWu3tOVeV0h2oa+drXwqTcW7fCyJHw+edRRyQiUVFyTzPn\nnAO//S288QZceWViQweLSPpJpLeM1DHf/naovf/kJ2Gc9/vuizoiEUk1Jfc0deutsGkT3H9/SPDX\nXx91RCKSSkruacosjD2zdSvccAN07gwXXxx1VCKSKmpzT2MNG4Yhg/v2DU01y5dHHZGIpIqSe5pr\n0SKMJtmhQ6i5b9kSdUQikgpK7vXACSfAyy/DZ5/BBRfAxx9HHZGI1DQl93oiOxueeSZMun3ppSUz\nOolIelJyr0dyc8P476++GuZpjejmZBFJAfWWqWcmTAhzst5xR+gieffdUUckIjVByb0euu22kODv\nuQe6dYPvfjfqiEQk2ZTc6yEzeOQRKCiASZOgUyf45jejjkpEkklt7vVUo0Ywfz707AmjR8M770Qd\nkYgkk5J7PdaqFbz0ErRuHSbfLiiIOiIRSRYl93ouKwsWLoR9+0KC37cv6ohEJBmU3IU+feDpp2HN\nmtBE8+WXUUckItWl5C4AnHcezJkDf/oTXHut+sCL1HXqLSPFrroqDBN8992hi+Qdd0QdkYgcKyV3\nKeXnPw994O+8M9zkdMUVUUckIsdCyV1KMYPHHoNt2+Dqq0Mf+KFDo45KRI6W2tzlCI0bhwusX/ta\nGGRs9eqoIxKRo6XkLuVq0yb0gW/WDC68EHbsiDoiETkaSu5SoS5dQoLfvTtM9LF/f/nl8vJC+3xG\nRviZl5fKKEWkPEruUqn+/cMwBe+8E6bqKyws/XpeXhifZsuW0H1yy5awrAQvEi0ld6nShRfC7Nnh\nTtYbbyzdB/622+DAgdLlDxwI60UkOuotIwm59trQB/6++0LTy09/GtZv3Vp++YrWi0hqqOYuCfuP\n/4AxY2DKFJg3L6w76aTyy1a0XkRSI6HkbmbDzWydmW0wsynlvD7RzHaa2duxxzXJD1WilpEBjz8O\ngwbBlVfCa6/B9OmQmVm6XGZmWC8i0akyuZtZA2A2cAGQDYw1s+xyiv7e3fvGHo8lOU6pJZo2hWef\nDcMTjBwJOTlhTJouXcINUF26hOXx46OOVKR+S6TNfQCwwd03ApjZPGAksKYmA5Paq127cHH1rLPg\nggvgzTeVzEVqm0SaZToB78UtF8TWlXWZma00swVm1rm8HZnZJDPLN7P8nTt3HkO4UlucfDK8+CK8\n/z6MGHFkjxkRiVayLqi+AHR19z7An4DfllfI3ee4e46753Ts2DFJh5aoDBgAc+fCsmUwbhwcOhR1\nRCJSJJHkvg2Ir4lnxdYVc/fd7v5FbPEx4IzkhCe13ciR8OCD8NxzMHEiHDwYdUQiAom1uS8DTjWz\nboSkPgYYF1/AzE5w96LRR0YAa5MapdRqN94Ypue7/fbQTPP002F+VhGJTpU1d3cvBG4AFhGS9nx3\nX21m08xsRKzYZDNbbWbvAJOBiTUVsNROt90WukkuXgyDB8P27VFHJFK/mUc0n1pOTo7n5+dHcmyp\nOYsWwWWXQfv28Ic/QI8eUUckkl7MbLm751RVTneoSlJ985uwZAl88QUMHAivvx51RCL1k5K7JF3/\n/qHve8eOMGxYaIMXkdRScpca0a0bvPFGSPTf+hY8/HDUEYnUL0ruUmM6dIA//zl0l5w8GX7yEzh8\nOOqoROoHJXepUc2awYIF8P3vw3/+J0yYENrjRaRmaTx3qXENGsB//VcYBnjKlNAX/plnoHXrqCMT\nSV+quUtKmIUJPp54IgwVPGgQFBREHZVI+lJyl5SaMAFefhk2b4azz4Z//CPqiETSk5K7pNywYaEv\n/KFDcM458Je/RB2RSPpRcpdI9O0b+sKfeCKcfz78/vdRRySSXpTcJTJduoQ7WM88M8zNOnNm1BGJ\npA8ld4lUu3bwxz/C6NFwyy1w003qCy+SDEruErmmTUOzzA9/CLNmhVr8559HHZVI3aZ+7lIrZGTA\nAw9A585w663wwQdhIu62baOOTKRuUs1dag2z0DQzdy787W+hJ83WrVFHJVI3KblLrTNmTBgXftu2\n0Bf+nXeijkik7lFyl1ppyJBwJ6tZuJv1z3+OOiKRukXJXWqt3r1D80yXLnDBBZCXF3VEInWHkrvU\nallZoQY/cGAYuuC++yCimSFF6hT1lpFar02bMB/rxIlhVMn33oMHHwyjTYpUZe9eWLs2PPbvr7q8\nWfXLVPX64MHQs2fVx6kOJXepE5o0Cc0yWVkwYwZs3x6WmzWLOjKpDdxh586QwNesKf1z+/aoozvS\nI48ouYsUy8gIE35kZYU7WYcNg+efh/bto45MUsU9DBVdXhLfvbukXIsWkJ0N550XfvboER5V3TeR\nSJNfVWUS2UeLFlWXqS4ld6lzfvhD6NQptMEPHBiGEO7WLeqoJJkOHYJNm8pP4vFNK+3bh+R92WUl\nSTw7O/x9JNK8ks6U3KVOGj0avvIVGDEi9IVfuDBMxi11y8GDsH59SeIuSuLr1pWejvHEE0PSvuqq\n0km8Y8foYq/tlNylzho0CN54I3STHDw4zNX6zW9GHZWU59NPQ8IuWwvfsCHU0iHUtLt2DUn7/PNL\nN6doSsajp+QudVp2dhgX/sIL4eKL4dFH4cor9ZU8Vb74AnbsCPPi7thx5CN+fZGGDeGUU8IFxdGj\nS5L4aadBZmZ07yXdmEfUaTgnJ8fz8/MjObakn3374NJLw52sjRqFJpv4x/HHl7/ctq0+CMpyh08+\nqTpZ79gBH3985PYZGXDccXDCCSWPLl1Kkvgpp0Djxql/X+nCzJa7e05V5VRzl7TQqhVccQX8/e/w\n0Uch6bRrFxLQ22/Dhx9CYeGR2zVqFBJR2eRf3gdCu3Z1+4Pg8GHYtaviRB2//sCBI7dv0iQk6uOP\nD7XsIUNKJ/Ci1447Tvcg1AYJJXczGw48CDQAHnP3eysodxmwAPi6u6taLimTlwff/35JUvr009Ce\nO2cOjB8fEttHH4WhhIse779fejnRD4Kqvg0UfRAcOgRffpnY4+DB5JYrKvvppyVJ+4MPyn9frVqV\nJOczzyxJ0mUTd5s2dfvDrb6pslnGzBoA/wTOAwqAZcBYd19TplxL4CWgMXBDVcldzTKSTF27wpYt\nR67v0gU2UiEYAAAHaElEQVQ2bz66fR0+HGr+ZZN/2eWiR3kJM5UaNw4fPOU9mjcvP1HHJ3C1c9ct\nyWyWGQBscPeNsR3PA0YCa8qUuxu4D/jxUcYqUm0Vjft+LOPBZ2SE/tPt21d9F2HRB0HZD4CPPgoX\nDitKuo0aVZ6UEy3ToIFq01K+RJJ7J+C9uOUC4Mz4AmbWH+js7i+ZWYXJ3cwmAZMATjrppKOPVqQC\nJ51Ufs29pv/M4j8IsrNr9lgiR6Pao0KaWQYwE7ilqrLuPsfdc9w9p6PuPpAkmj79yOaFzMywXqQ+\nSiS5bwM6xy1nxdYVaQn0Al41s83AWcDzZlZlm5BIsowfHy6edukSmim6dCm5mCpSHyXSLLMMONXM\nuhGS+hhgXNGL7r4X6FC0bGavAreqt4yk2vjxSuYiRaqsubt7IXADsAhYC8x399VmNs3MRtR0gCIi\ncvQS6ufu7guBhWXW3VlB2SHVD0tERKpD0+yJiKQhJXcRkTSk5C4ikoaU3EVE0pCSu4hIGlJyFxFJ\nQ0ruIiJpSMldRCQNKbmLiKQhJXcRkTSk5C4ikoaU3EWSKC8vTPmXkRF+5uVFHZHUVwkNHCYiVcvL\ng0mTSibp3rIlLIOGIpbUU81dJEluu60ksRc5cCCsF0k1JXeRJEnmJN0i1aXkLpIkFU3GrbngJQpK\n7iJJokm6pTZRchdJEk3SLbWJesuIJJEm6ZbaQjV3EZE0pOQuIpKGlNxFRNKQkruISBpSchcRSUNK\n7iIiaUjJXSQNaXRKUT93kTSj0SkFVHMXSTsanVIgweRuZsPNbJ2ZbTCzKeW8fp2ZrTKzt83sdTPL\nTn6oIpIIjU4pkEByN7MGwGzgAiAbGFtO8n7K3Xu7e1/gfmBm0iMVkYRodEqBxGruA4AN7r7R3Q8C\n84CR8QXcfV/cYnPAkxeiiBwNjU4pkFhy7wS8F7dcEFtXipn9wMz+Rai5Ty5vR2Y2yczyzSx/586d\nxxKviFRBo1MKJPGCqrvPdvevAj8Fbq+gzBx3z3H3nI4dOybr0CJSxvjxsHkzHD4cfiqx1z+JJPdt\nQOe45azYuorMA/5PdYISEZHqSSS5LwNONbNuZtYYGAM8H1/AzE6NW7wIWJ+8EEVE5GhVmdzdvRC4\nAVgErAXmu/tqM5tmZiNixW4ws9Vm9jZwM3BljUUsInWG7pSNjrlH07ElJyfH8/PzIzm2iNS8snfK\nQui1o4u71WNmy909p6pyukNVRGqE7pSNlpK7iNQI3SkbLSV3EakRulM2WkruIlIjdKdstJTcRaRG\n6E7ZaGk8dxGpMePHK5lHRTV3EZE0pOQuIpKGlNxFJO3Vxztl1eYuImmtvs4pq5q7iKS1+nqnrJK7\niKS1+nqnrJK7iKS1+nqnrJK7iKS1+nqnrJK7iKS1+nqnrJK7iKS92jKnbCq7ZKorpIhICqS6S6Zq\n7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLpnqLSMikiKpnLxENXcR\nkTSk5C4ikoaU3EVE0pCSu4hIGlJyFxFJQ+bu0RzYbCew5Rg37wDsSmI4dZ3OR2k6HyV0LkpLh/PR\nxd07VlUosuReHWaW7+45UcdRW+h8lKbzUULnorT6dD7ULCMikoaU3EVE0lBdTe5zog6gltH5KE3n\no4TORWn15nzUyTZ3ERGpXF2tuYuISCWU3EVE0lCdS+5mNtzM1pnZBjObEnU8UTGzzma22MzWmNlq\nM/th1DHVBmbWwMz+bmYvRh1L1MysjZktMLN3zWytmZ0ddUxRMbObYv8n/zCzuWbWNOqYalqdSu5m\n1gCYDVwAZANjzSw72qgiUwjc4u7ZwFnAD+rxuYj3Q2Bt1EHUEg8Cf3D37sDp1NPzYmadgMlAjrv3\nAhoAY6KNqubVqeQODAA2uPtGdz8IzANGRhxTJNx9h7uviD3/hPCP2ynaqKJlZlnARcBjUccSNTNr\nDXwD+L8A7n7Q3fdEG1WkGgLNzKwhkAlsjzieGlfXknsn4L245QLqeUIDMLOuQD9gabSRRG4W8BPg\ncNSB1ALdgJ3A47FmqsfMrHnUQUXB3bcBM4CtwA5gr7v/Mdqoal5dS+5Shpm1AJ4GfuTu+6KOJypm\ndjHwobsvjzqWWqIh0B94xN37AZ8C9fIalZm1JXzD7wacCDQ3swnRRlXz6lpy3wZ0jlvOiq2rl8ys\nESGx57n7/0YdT8QGAiPMbDOhuW6omT0ZbUiRKgAK3L3o29wCQrKvj4YBm9x9p7t/Cfwv8G8Rx1Tj\n6lpyXwacambdzKwx4aLI8xHHFAkzM0J76lp3nxl1PFFz93939yx370r4u3jF3dO+dlYRd38feM/M\nToutOhdYE2FIUdoKnGVmmbH/m3OpBxeX69QE2e5eaGY3AIsIV7x/4+6rIw4rKgOBK4BVZvZ2bN1U\nd18YYUxSu9wI5MUqQhuBqyKOJxLuvtTMFgArCL3M/k49GIZAww+IiKShutYsIyIiCVByFxFJQ0ru\nIiJpSMldRCQNKbmLiKQhJXcRkTSk5C4ikob+PyYufzJSbaZ7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f707adbb208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The validation MAE stays in the low 0.40s: we cannot even beat our common-sense baseline using the small convnet. Again, this is because \n",
    "our convnet looks for patterns anywhere in the input timeseries, and has no knowledge of the temporal position of a pattern it sees (e.g. \n",
    "towards the beginning, towards the end, etc.). Since more recent datapoints should be interpreted differently from older datapoints in the \n",
    "case of this specific forecasting problem, the convnet fails at producing meaningful results here. This limitation of convnets was not an \n",
    "issue on IMDB, because patterns of keywords that are associated with a positive or a negative sentiment will be informative independently \n",
    "of where they are found in the input sentences.\n",
    "\n",
    "One strategy to combine the speed and lightness of convnets with the order-sensitivity of RNNs is to use a 1D convnet as a preprocessing \n",
    "step before a RNN. This is especially beneficial when dealing with sequences that are so long that they couldn't realistically be processed \n",
    "with RNNs, e.g. sequences with thousands of steps. The convnet will turn the long input sequence into much shorter (downsampled) sequences \n",
    "of higher-level features. This sequence of extracted features then becomes the input to the RNN part of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This technique is not seen very often in research papers and practical applications, possibly because it is not very well known. It is very \n",
    "effective and ought to be more common. Let's try this out on the temperature forecasting dataset. Because this strategy allows us to \n",
    "manipulate much longer sequences, we could either look at data from further back (by increasing the `lookback` parameter of the data \n",
    "generator), or look at high-resolution timeseries (by decreasing the `step` parameter of the generator). Here, we will chose (somewhat \n",
    "arbitrarily) to use a `step` twice smaller, resulting in twice longer timeseries, where the weather data is being sampled at a rate of one \n",
    "point per 30 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This was previously set to 6 (one point per hour).\n",
    "# Now 3 (one point per 30 min).\n",
    "step = 3\n",
    "lookback = 720  # Unchanged\n",
    "delay = 144 # Unchanged\n",
    "\n",
    "train_gen = generator(float_data,\n",
    "                      lookback=lookback,\n",
    "                      delay=delay,\n",
    "                      min_index=0,\n",
    "                      max_index=200000,\n",
    "                      shuffle=True,\n",
    "                      step=step)\n",
    "val_gen = generator(float_data,\n",
    "                    lookback=lookback,\n",
    "                    delay=delay,\n",
    "                    min_index=200001,\n",
    "                    max_index=300000,\n",
    "                    step=step)\n",
    "test_gen = generator(float_data,\n",
    "                     lookback=lookback,\n",
    "                     delay=delay,\n",
    "                     min_index=300001,\n",
    "                     max_index=None,\n",
    "                     step=step)\n",
    "val_steps = (300000 - 200001 - lookback) // 128\n",
    "test_steps = (len(float_data) - 300001 - lookback) // 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is our model, starting with two `Conv1D` layers and following-up with a `GRU` layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv1d_6 (Conv1D)            (None, None, 32)          2272      \n",
      "_________________________________________________________________\n",
      "max_pooling1d_4 (MaxPooling1 (None, None, 32)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_7 (Conv1D)            (None, None, 32)          5152      \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, 32)                6240      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 13,697\n",
      "Trainable params: 13,697\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "500/500 [==============================] - 60s - loss: 0.3387 - val_loss: 0.3030\n",
      "Epoch 2/20\n",
      "500/500 [==============================] - 58s - loss: 0.3055 - val_loss: 0.2864\n",
      "Epoch 3/20\n",
      "500/500 [==============================] - 58s - loss: 0.2904 - val_loss: 0.2841\n",
      "Epoch 4/20\n",
      "500/500 [==============================] - 58s - loss: 0.2830 - val_loss: 0.2730\n",
      "Epoch 5/20\n",
      "500/500 [==============================] - 58s - loss: 0.2767 - val_loss: 0.2757\n",
      "Epoch 6/20\n",
      "500/500 [==============================] - 58s - loss: 0.2696 - val_loss: 0.2819\n",
      "Epoch 7/20\n",
      "500/500 [==============================] - 57s - loss: 0.2642 - val_loss: 0.2787\n",
      "Epoch 8/20\n",
      "500/500 [==============================] - 57s - loss: 0.2595 - val_loss: 0.2920\n",
      "Epoch 9/20\n",
      "500/500 [==============================] - 58s - loss: 0.2546 - val_loss: 0.2919\n",
      "Epoch 10/20\n",
      "500/500 [==============================] - 57s - loss: 0.2506 - val_loss: 0.2772\n",
      "Epoch 11/20\n",
      "500/500 [==============================] - 58s - loss: 0.2459 - val_loss: 0.2801\n",
      "Epoch 12/20\n",
      "500/500 [==============================] - 57s - loss: 0.2433 - val_loss: 0.2807\n",
      "Epoch 13/20\n",
      "500/500 [==============================] - 58s - loss: 0.2411 - val_loss: 0.2855\n",
      "Epoch 14/20\n",
      "500/500 [==============================] - 58s - loss: 0.2360 - val_loss: 0.2858\n",
      "Epoch 15/20\n",
      "500/500 [==============================] - 57s - loss: 0.2350 - val_loss: 0.2834\n",
      "Epoch 16/20\n",
      "500/500 [==============================] - 58s - loss: 0.2315 - val_loss: 0.2917\n",
      "Epoch 17/20\n",
      "500/500 [==============================] - 60s - loss: 0.2285 - val_loss: 0.2944\n",
      "Epoch 18/20\n",
      "500/500 [==============================] - 57s - loss: 0.2280 - val_loss: 0.2923\n",
      "Epoch 19/20\n",
      "500/500 [==============================] - 57s - loss: 0.2249 - val_loss: 0.2910\n",
      "Epoch 20/20\n",
      "500/500 [==============================] - 57s - loss: 0.2215 - val_loss: 0.2952\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Conv1D(32, 5, activation='relu',\n",
    "                        input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.MaxPooling1D(3))\n",
    "model.add(layers.Conv1D(32, 5, activation='relu'))\n",
    "model.add(layers.GRU(32, dropout=0.1, recurrent_dropout=0.5))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=20,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f707ade8128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNW5x/Hvyy6LgIhxQRhwBWQfQYMEUKO4gRiugkPc\ngxKJGq4mRNQYlBvFDTFc16txQdFoVFSUGCUxJnEBRBSQgAg6grIoII6Kw7z3j1MzNMMsPdM93dPT\nv8/z9DNd1aeq3q7ueav61KlzzN0REZHsUC/dAYiISOoo6YuIZBElfRGRLKKkLyKSRZT0RUSyiJK+\niEgWUdKXKjGz+ma21czaJ7NsOpnZgWaW9LbLZnasma2KmV5mZgPiKVuNbd1nZldWd/kK1nu9mf0x\n2euV9GmQ7gCkZpnZ1pjJpsB3wPZo+kJ3n1GV9bn7dqB5sstmA3c/JBnrMbMLgNHuPihm3RckY91S\n9ynp13HuXpJ0ozPJC9z9r+WVN7MG7l6YithEJPVUvZPlop/vj5vZY2b2FTDazI40szfMbJOZrTWz\naWbWMCrfwMzczHKi6Uei1180s6/M7N9m1rGqZaPXTzCz/5jZZjO7w8z+aWbnlBN3PDFeaGYrzOxL\nM5sWs2x9M7vNzDaa2UpgSAX7Z6KZzSw1b7qZ3Ro9v8DMlkbv58PoLLy8deWb2aDoeVMzeziKbTHQ\np1TZq8xsZbTexWY2NJrfDfgDMCCqOtsQs2+vjVn+oui9bzSzZ8xsn3j2TWXMbHgUzyYze9XMDol5\n7UozW2NmW8zsg5j3eoSZLYjmf25mN8W7PakB7q5HljyAVcCxpeZdD2wDTiGcBOwGHA70I/wS7AT8\nBxgXlW8AOJATTT8CbABygYbA48Aj1Si7F/AVMCx6bTzwPXBOOe8lnhifBVoCOcAXxe8dGAcsBtoB\nbYDXwr9CmdvpBGwFmsWsex2QG02fEpUx4GjgG6B79NqxwKqYdeUDg6LnNwN/A1oDHYAlpcqeDuwT\nfSZnRjH8IHrtAuBvpeJ8BLg2en5cFGNPoAnwv8Cr8eybMt7/9cAfo+edoziOjj6jK4Fl0fOuwGpg\n76hsR6BT9PxtYFT0vAXQL93/C9n80Jm+ALzu7s+5e5G7f+Pub7v7m+5e6O4rgXuAgRUs/6S7z3P3\n74EZhGRT1bInAwvd/dnotdsIB4gyxRnj7919s7uvIiTY4m2dDtzm7vnuvhG4oYLtrATeJxyMAH4M\nfOnu86LXn3P3lR68CrwClHmxtpTTgevd/Ut3X004e4/d7hPuvjb6TB4lHLBz41gvQB5wn7svdPdv\ngQnAQDNrF1OmvH1TkZHALHd/NfqMbiAcOPoBhYQDTNeoivCjaN9BOHgfZGZt3P0rd38zzvchNUBJ\nXwA+iZ0ws0PN7AUz+8zMtgCTgD0rWP6zmOcFVHzxtryy+8bG4e5OODMuU5wxxrUtwhlqRR4FRkXP\nz4ymi+M42czeNLMvzGwT4Sy7on1VbJ+KYjCzc8zs3agaZRNwaJzrhfD+Stbn7luAL4H9YspU5TMr\nb71FhM9oP3dfBvw34XNYF1UX7h0VPRfoAiwzs7fM7MQ434fUACV9gfBzP9bdhLPbA919d+AaQvVF\nTVpLqG4BwMyMnZNUaYnEuBbYP2a6sialTwDHmtl+hDP+R6MYdwOeBH5PqHppBfwlzjg+Ky8GM+sE\n3AmMBdpE6/0gZr2VNS9dQ6gyKl5fC0I10qdxxFWV9dYjfGafArj7I+7en1C1U5+wX3D3Ze4+klCF\ndwvwlJk1STAWqSYlfSlLC2Az8LWZdQYuTME2nwd6m9kpZtYAuBRoW0MxPgFcZmb7mVkb4NcVFXb3\nz4DXgT8Cy9x9efRSY6ARsB7YbmYnA8dUIYYrzayVhfsYxsW81pyQ2NcTjn8/I5zpF/scaFd84boM\njwHnm1l3M2tMSL7/cPdyfzlVIeahZjYo2vYVhOswb5pZZzMbHG3vm+hRRHgDPzWzPaNfBpuj91aU\nYCxSTUr6Upb/Bs4m/EPfTbjgWqPc/XPgDOBWYCNwAPAO4b6CZMd4J6Hu/T3CRcYn41jmUcKF2ZKq\nHXffBPwSeJpwMXQE4eAVj98SfnGsAl4EHopZ7yLgDuCtqMwhQGw9+MvAcuBzM4utpile/iVCNcvT\n0fLtCfX8CXH3xYR9fifhgDQEGBrV7zcGphCuw3xG+GUxMVr0RGCphdZhNwNnuPu2ROOR6rFQdSpS\nu5hZfUJ1wgh3/0e64xGpK3SmL7WGmQ2JqjsaA1cTWn28leawROoUJX2pTY4CVhKqDo4Hhrt7edU7\nIlINqt4REckiOtMXEckita7DtT333NNzcnLSHYaISEaZP3/+BnevqJkzUAuTfk5ODvPmzUt3GCIi\nGcXMKruzHFD1johIVlHSFxHJIkr6IiJZJK46fTMbAtxO6ETpPne/odTrFwEXE4bh2wqMcfclMa+3\nJ/QXfq2735yk2EUkCb7//nvy8/P59ttv0x2KxKFJkya0a9eOhg3L63qpYpUm/eh2+OmEfsTzgbfN\nbFZsUgcedfe7ovJDCf2nxI5GdCuhfxERqWXy8/Np0aIFOTk5hM5NpbZydzZu3Eh+fj4dO3asfIEy\nxFO90xdYEQ0UsQ2YyY4BJYoD2RIz2YyYrl/N7FTgI8JIRTVmxgzIyYF69cLfGVUa7lske3377be0\nadNGCT8DmBlt2rRJ6FdZPNU7+7HzYA/5hJFySgdzMWGIu0aE4dQws+aEbmt/DFxe3gbMbAwwBqB9\n+8q6Nt/VjBkwZgwUFITp1avDNEBewn0LitR9SviZI9HPKmkXct19ursfQEjyV0WzryUMS7e1kmXv\ncfdcd89t27bSewt2MXHijoRfrKAgzBcRkR3iSfqfsvMIPyUj5ZRjJnBq9LwfMMXMVgGXEQaNGFfe\ngtX18cdVmy8itcfGjRvp2bMnPXv2ZO+992a//fYrmd62Lb5u988991yWLVtWYZnp06czI0n1vkcd\ndRQLFy5MyrpSLZ7qnbcJgxp3JCT7kYRxQkuY2UExowmdRBjgAXcfEFPmWmCru+80AHQytG8fqnTK\nmi8iyTVjRvgV/fHH4X9s8uTEqlHbtGlTkkCvvfZamjdvzuWX71wb7O64O/XqlX2e+sADD1S6nYsv\nvrj6QdYhlZ7pu3shYSi3OcBS4Al3X2xmk6KWOgDjzGyxmS0k1OufXWMRl2HyZGjadOd5TZuG+SKS\nPMXXz1avBvcd189qouHEihUr6NKlC3l5eXTt2pW1a9cyZswYcnNz6dq1K5MmTSopW3zmXVhYSKtW\nrZgwYQI9evTgyCOPZN26dQBcddVVTJ06taT8hAkT6Nu3L4cccgj/+te/APj666/5yU9+QpcuXRgx\nYgS5ubmVntE/8sgjdOvWjcMOO4wrr7wSgMLCQn7605+WzJ82bRoAt912G126dKF79+6MHj066fss\nHnG103f32cDsUvOuiXl+aRzruLaqwcWr+CwjmWcfIrKriq6f1cT/2wcffMBDDz1Ebm4uADfccAN7\n7LEHhYWFDB48mBEjRtClS5edltm8eTMDBw7khhtuYPz48dx///1MmDBhl3W7O2+99RazZs1i0qRJ\nvPTSS9xxxx3svffePPXUU7z77rv07t27wvjy8/O56qqrmDdvHi1btuTYY4/l+eefp23btmzYsIH3\n3nsPgE2bNgEwZcoUVq9eTaNGjUrmpVqduSM3Lw9WrYKiovBXCV8k+VJ9/eyAAw4oSfgAjz32GL17\n96Z3794sXbqUJUuW7LLMbrvtxgknnABAnz59WLVqVZnrPu2003Yp8/rrrzNy5EgAevToQdeuXSuM\n78033+Too49mzz33pGHDhpx55pm89tprHHjggSxbtoxLLrmEOXPm0LJlSwC6du3K6NGjmTFjRrVv\nrkpUnUn6IlLzyrtOVlPXz5o1a1byfPny5dx+++28+uqrLFq0iCFDhpTZXr1Ro0Ylz+vXr09hYWGZ\n627cuHGlZaqrTZs2LFq0iAEDBjB9+nQuvPBCAObMmcNFF13E22+/Td++fdm+fXtStxsPJX0RiVs6\nr59t2bKFFi1asPvuu7N27VrmzJmT9G3079+fJ554AoD33nuvzF8Ssfr168fcuXPZuHEjhYWFzJw5\nk4EDB7J+/Xrcnf/6r/9i0qRJLFiwgO3bt5Ofn8/RRx/NlClT2LBhAwWl68pSoNb1py8itVc6r5/1\n7t2bLl26cOihh9KhQwf69++f9G384he/4KyzzqJLly4lj+KqmbK0a9eO6667jkGDBuHunHLKKZx0\n0kksWLCA888/H3fHzLjxxhspLCzkzDPP5KuvvqKoqIjLL7+cFi1aJP09VKbWjZGbm5vrGkRFJHWW\nLl1K586d0x1GrVBYWEhhYSFNmjRh+fLlHHfccSxfvpwGDWrX+XFZn5mZzXf33HIWKVG73omISBpt\n3bqVY445hsLCQtydu+++u9Yl/ETVrXcjIpKAVq1aMX/+/HSHUaN0IVdEJIso6YuIZBElfRGRLKKk\nLyKSRZT0RSStBg8evMuNVlOnTmXs2LEVLte8eXMA1qxZw4gRI8osM2jQICprAj516tSdbpI68cQT\nk9IvzrXXXsvNN9e+IcGV9EUkrUaNGsXMmTN3mjdz5kxGjRoV1/L77rsvTz75ZLW3Xzrpz549m1at\nWlV7fbWdkr6IpNWIESN44YUXSgZMWbVqFWvWrGHAgAEl7eZ79+5Nt27dePbZZ3dZftWqVRx22GEA\nfPPNN4wcOZLOnTszfPhwvvnmm5JyY8eOLemW+be//S0A06ZNY82aNQwePJjBgwcDkJOTw4YNGwC4\n9dZbOeywwzjssMNKumVetWoVnTt35mc/+xldu3bluOOO22k7ZVm4cCFHHHEE3bt3Z/jw4Xz55Zcl\n2y/uarm4o7e///3vJYPI9OrVi6+++qra+7YsaqcvIiUuuwySPSBUz54Q5csy7bHHHvTt25cXX3yR\nYcOGMXPmTE4//XTMjCZNmvD000+z++67s2HDBo444giGDh1a7jixd955J02bNmXp0qUsWrRop66R\nJ0+ezB577MH27ds55phjWLRoEZdccgm33norc+fOZc8999xpXfPnz+eBBx7gzTffxN3p168fAwcO\npHXr1ixfvpzHHnuMe++9l9NPP52nnnqqwv7xzzrrLO644w4GDhzINddcw+9+9zumTp3KDTfcwEcf\nfUTjxo1LqpRuvvlmpk+fTv/+/dm6dStNmjSpwt6unM70RSTtYqt4Yqt23J0rr7yS7t27c+yxx/Lp\np5/y+eefl7ue1157rST5du/ene7du5e89sQTT9C7d2969erF4sWLK+1M7fXXX2f48OE0a9aM5s2b\nc9ppp/GPf/wDgI4dO9KzZ0+g4u6bIfTvv2nTJgYOHAjA2WefzWuvvVYSY15eHo888kjJnb/9+/dn\n/PjxTJs2jU2bNiX9jmCd6YtIiYrOyGvSsGHD+OUvf8mCBQsoKCigT58+AMyYMYP169czf/58GjZs\nSE5OTpndKVfmo48+4uabb+btt9+mdevWnHPOOdVaT7HibpkhdM1cWfVOeV544QVee+01nnvuOSZP\nnsx7773HhAkTOOmkk5g9ezb9+/dnzpw5HHroodWOtTSd6YtI2jVv3pzBgwdz3nnn7XQBd/Pmzey1\n1140bNiQuXPnsrqswbBj/OhHP+LRRx8F4P3332fRokVA6Ja5WbNmtGzZks8//5wXX3yxZJkWLVqU\nWW8+YMAAnnnmGQoKCvj66695+umnGTBgwC7lKtOyZUtat25d8ivh4YcfZuDAgRQVFfHJJ58wePBg\nbrzxRjZv3szWrVv58MMP6datG7/+9a85/PDD+eCDD6q8zYroTF9EaoVRo0YxfPjwnVry5OXlccop\np9CtWzdyc3MrPeMdO3Ys5557Lp07d6Zz584lvxh69OhBr169OPTQQ9l///136pZ5zJgxDBkyhH33\n3Ze5c+eWzO/duzfnnHMOffv2BeCCCy6gV69eFVbllOfBBx/koosuoqCggE6dOvHAAw+wfft2Ro8e\nzebNm3F3LrnkElq1asXVV1/N3LlzqVevHl27di0ZBSxZ1LWySJZT18qZJ5GulVW9IyKSRZT0RUSy\niJK+iFDbqnmlfIl+Vkr6IlmuSZMmbNy4UYk/A7g7GzduTOiGLbXeEcly7dq1Iz8/n/Xr16c7FIlD\nkyZNaNeuXbWXjyvpm9kQ4HagPnCfu99Q6vWLgIuB7cBWYIy7LzGzHwM3AI2AbcAV7v5qtaMVkaRr\n2LAhHTt2THcYkiKVVu+YWX1gOnAC0AUYZWZdShV71N27uXtPYApwazR/A3CKu3cDzgYeTlrkIiJS\nZfHU6fcFVrj7SnffBswEhsUWcPctMZPNAI/mv+Pua6L5i4HdzKwxIiKSFvFU7+wHfBIznQ/0K13I\nzC4GxhOqco4uYz0/ARa4+3dlLDsGGAPQvn37OEISEZHqSFrrHXef7u4HAL8Grop9zcy6AjcCF5az\n7D3unuvuuW3btk1WSCIiUko8Sf9TYP+Y6XbRvPLMBE4tnjCzdsDTwFnu/mF1ghQRkeSIJ+m/DRxk\nZh3NrBEwEpgVW8DMDoqZPAlYHs1vBbwATHD3fyYnZBERqa5Kk767FwLjgDnAUuAJd19sZpPMbGhU\nbJyZLTazhYR6/bOL5wMHAteY2cLosVfy34aIiMRDvWyKiNQB6mVTRER2oaQvIpJFlPRFRLKIkr6I\nSBZR0hcRySJK+iIiWURJX0Qkiyjpi4hkESV9EZEsoqQvIpJFlPRFRLKIkr6ISBZR0hcRySJK+iIi\nWURJX0QkiyjpR2bMgJwcqFcv/J0xI90RiYgkX4N0B1AbzJgBY8ZAQUGYXr06TAPk5aUvLhGRZNOZ\nPjBx4o6EX6ygIMwXEalLlPSBjz+u2nwRkUylpA+0b1+1+SIimUpJH5g8GZo23Xle06ZhvohIXVJn\nkr47PP88bNpU9WXz8uCee6BDBzALf++5RxdxRaTuqTNJf/lyGDoUrr++esvn5cGqVVBUFP4q4YtI\nXVRnkv7BB8N558G0aeEAICIiu4or6ZvZEDNbZmYrzGxCGa9fZGbvmdlCM3vdzLrEvPabaLllZnZ8\nMoMvbfJkaNIE/vu/a3IrIiKZq9Kkb2b1genACUAXYFRsUo886u7d3L0nMAW4NVq2CzAS6AoMAf43\nWl+N+MEP4Kqr4Lnn4OWXa2orIiKZK54z/b7ACndf6e7bgJnAsNgC7r4lZrIZ4NHzYcBMd//O3T8C\nVkTrqzGXXgoHHAC//CUUFtbklkREMk88SX8/4JOY6fxo3k7M7GIz+5Bwpn9JFZcdY2bzzGze+vXr\n4429TI0bw803w+LFcPfdCa1KRKTOSdqFXHef7u4HAL8Grqrisve4e66757Zt2zbhWIYNg8GD4Zpr\n4IsvEl6diEidEU/S/xTYP2a6XTSvPDOBU6u5bFKYwdSpoc3+735X01sTEckc8ST9t4GDzKyjmTUi\nXJidFVvAzA6KmTwJKG40OQsYaWaNzawjcBDwVuJhV657d/jZz2D6dFi6NBVbFBGp/SpN+u5eCIwD\n5gBLgSfcfbGZTTKzoVGxcWa22MwWAuOBs6NlFwNPAEuAl4CL3X17DbyPMl13HTRvDuPHp2qLIiK1\nm7l75aVSKDc31+fNm5e09d16a2i3/8ILcOKJSVutiEitYmbz3T23snJ15o7c8owbF+7WHT8evv8+\n3dGIiKRXnU/6jRrBLbfAsmWhfl9EpDb66qvUXH+s80kf4KST4LjjQkueDRvSHY2IyA4rV4abSdu1\ngzPPDD0G16SsSPpmcNtt4Uh6zTXpjkZEkmX7dnjzTfjtb+GII6Bv33BX/uOPwyefVL58urjD3/4G\np54KBx4If/gDnHxyuKHUrGa3Xecv5Mb6xS/gf/8XFi6Ebt1qZBMiUsM2boQ5c2D27PB3w4aQKPv1\nCx0uvvXWjjGv27WDH/5wx6NnT2jYMH2xf/stPPYY3H47vPsutGkDF10EY8fCfrv0VVA18V7Izaqk\nv3EjHHQQ9O4dOmSr6SOqiCSuqAjeeSck+dmzQ1IvKoI994QhQ0KrvOOOCwkUQoONRYvgX//a8Sge\n73q33eDww3ccBI48Mqynpn32Gdx5Z3isXw+HHRZ+keTlhZiSQUm/HHfcAZdcAs88E7prEJHa58sv\nw4nZ7Nnw4ouwbl04ScvNDUn+xBOhTx+oH2efvfn58O9/7zgILFiwo0PGgw8Oyf+HP4RevaBjx3AA\nScZJ4fz54ax+5sywvZNPDsn+6KOTf9KppF+O77+HHj1g27bQKVvjxjW2KamCoqLQLfaiRWG6+B8i\n9m9Z82L/1q8fqvAGDEhNzJJcK1eGuvjZs0OC3r4dWreG448PSf7442GvvZKzrW++gXnzdv41ENvI\no0WLkPw7dQqP4ucdO0JOTsVn54WF8OyzoSuY118PN4iee274bh50UPnLJUpJvwJz5oSfhVOmwBVX\nJGedM2bAxInhZ2T79mFAFw25GL/im+i6ddtR51r81XTf+Xl5r332GdSrF5q9Ff/Ul8wwZw6MGAFb\nt4bq1xNPhBNOCBdmGzSo+e27w4oVsGQJfPRROADF/v3mm53L77PPrgeDjh3DgeSOO2D16nBwuOSS\nMKJfy5Y1/x7iTfq4e6169OnTx1PhpJPcW7Rw/+yzxNf1yCPuTZsWp5/waNo0zJfKvfOOe6NG7sOH\nuxcVVX89Cxe6N2jgftZZyYtNat5997nXr+/eo4f7ypXpjmZXRUXua9e6/+tf4X/6uuvczz3XfdAg\n9/bt3evV2/l/f+BA96efdi8sTG2cwDyPI8emPcmXfqQq6X/wQUgQF1yQ+Lo6dNj5Qy9+dOiQ+Lrr\nuq+/dj/0UPd993XfsCHx9U2cGPb9Sy8lvi6pWUVF7ldfHT6v449337Il3RFVz3ffuS9f7v6Xv7gv\nWpS+OOJN+llZvVNs/PhQ7zZ/friAU1316pV9Q4VZqKuW8o0dG9omv/wyHHNM4uv79tvQLO/bb+H9\n90N9qtQ+27bBBRfAww+H6o+77kpvU8q6QH3vxOGaa0Ld72WXJXYXXPv2VZsvwbPPhn/2yy9PTsKH\n0E77vvtCnepVVRrKR1Jl8+ZQX//wwzBpUvi8lPBTJ6uTfqtWofvl116Dp56q/nomT4amTXee17Rp\nmC9lW7MGzj8/XLS7/vrkrvuoo+DnP4dp00IrEKk9PvkkfD6vvQYPPghXX637ZVItq6t3IDSv6tUr\ntBpYujScKVaHWu/Er6goNL/75z9De+lDD03+NrZsga5dYffdwzbUNDf9Fi4M/WBt3Qp//nPyft1J\noOqdODVoEOr1V60KzQarKy8vrKOoKPxVwi/fbbfBX/8a9ntNJHwIyf6uu0ITvN//vma2IfGbMyfc\nP1GvXmi7roSfPlmf9CF8AYcNg//5n3CTRvGdepJ877wDv/kNDB8ehrOsSSedBKNGhc918eKa3ZaU\n7/77w2fRqRO88Yb6vUq3rK/eKbZixY5qnhYtQr3jwIHh0aePLjQlQ0FB2JdbtoQ7b1NxA9X69dC5\nc+jJ8J//jP+2/bqkqCjcXFRQEB7Fz8uat//+0L//rteoqsM99H553XWhb5w//Sn8ApOaEW/1Tgru\ndcsMBx4IH34Ir74Kf/97eLz4YnitWbPQL0fxQeDww1VHXB3jx4fBbF5+OXV3zLZtG/o+GT06dF97\n6aWp2W6qvftu+EWzdOmuyfy776q2roYNQ180Rx8dHv36hcGIqmLbtvBL7qGHQhcEd9+tE6faQmf6\nFVi3LrQyKD4IvPdemN+kSfinKD4I9OuXvJ7y6qpnnglVOldcEbq/SCX3UL3w97+Hap6cnNRuvya9\n/34YHOjJJ8Ot/gMHhpOUpk3Dd7Jp0/Kfl57XpEk4KL/6angsWBD2XdOmoT6++CDQq1fFv5g2b4af\n/AReeSXEphY6qaFuGGrAhg3h9urLLnPv1cvdLNxN2KiR+4AB4e7Czz+v3rofeSTcwWsW/talLhw+\n/dS9TRv33r3D3YvpsHq1e/Pm7scdl1hXD7XFkiXuZ5wRvi8tWoTv3hdfJHcbX3wRvu+/+IV71647\n7jRv2dJ92DD32293f++9nffnxx+7H3ZYuNv9j39MbjxSMdQNQ8378kv3555zv/xy98MPD31w9Orl\n/tVXVVtPXe67Z/t292OOCe/ngw/SG8sdd4R9++CD6Y0jEcuWueflhWTfrJn7lVcmp/uKeKxd6/7Y\nY6Hrkk6ddnxX99rLfeRI96lT3ffbLxyEXn45NTHJDkr6aTB7dkj8J59ctc6W6nLfPTfdFN7LPfek\nO5JwAPrhD91bt05OR3uptHx56EiuXr1wAP3Vr9zXrUtvTB995H7//e6jR4e+kyAk/XffTW9c2UpJ\nP02mTw979ZJL4l+muJqo9MOs5uJMhfnz3Rs2TLz3zGRasiRUx51+erojic/Kle7nnRd6oWzSxH38\n+Np5wCoqCgemTO00rS6IN+mrnX6S/fznYWT7adNCa5F41Ia+ez75BC68MLShX7gwsb6IAL7+Gs48\nMwx6ce+9tedCXufO4cLiE0/ArFnpjqZ8q1fDmDFhVKcZM2DcuNC3+y23wA9+kO7odmUWWsC1aJHu\nSKRS8RwZgCHAMmAFMKGM18cDS4BFwCtAh5jXpgCLgaXANKIWQ+U9Mv1M3z1U7QwdGn6KP/985eXT\nWae/fbv7XXeFetgmTcIZJbgffHC4OPj++9Vb75gx4ZfKK68kN95k+O47927dQpXEpk3pjmZnn3zi\nPnZs+IXUqJH7xRe75+enOyrJBCSregeoD3wIdAIaAe8CXUqVGQw0jZ6PBR6Pnv8Q+Ge0jvrAv4FB\nFW2vLiR9d/etW0NrlWbNwiAhlUlH650VK8JAEBAutn74Yagnvusu98GDdwwO0bWr+6RJ4SJiPP78\n57Dcr35Vs/En4q23wvu78MKa20ZRkXtBQaiOWbbM/e233f/617B//vjH0Prluuvcr7giHCRPOy0k\n+oYN3S+6KLSEEYlXMpP+kcCcmOnfAL+poHwv4J8xy84HdgOaAvOAzhVtr64kfffQVLFdu3Bxqzad\nrRUWut99tOiNAAARBUlEQVRyi/tuu7nvvrv7vfeWXee+dm1o8XLUUTt+gfTs6f7734cDRFny8933\n2MO9T5/0Nc+M1/jx4T397W+Jreezz8LoT6eeGkZ/6tgxNFFt0KDsazWlH02ahBYwBx7o/rOfhQuk\nIlUVb9Kv9OYsMxsBDHH3C6LpnwL93H1cOeX/AHzm7tdH0zcDFwAG/MHdJ5axzBhgDED79u37rF69\nusKYMsmiReG29oMOCjd6pXtQj8WLQ5fGb74Jp5wCd94J++1X+XKffhpuo3/88dB/CoQ7k884A04/\nPdy+X1QUbrf/97/DjT2HHFKz7yVRX38d+oFp0CDc0VqVG+w++CCMBzBrVni/7tChA/ToEboa2H33\ncLNUWc9LT1f1bleRsiTt5ixgBHBfzPRPCcm7rLKjgTeAxtH0gcALQPPo8W9gQEXbq0tn+sVeeCFU\nJZxySurHzSy2bVuoomnY0H3PPd0ffbT6LWpWrXKfMiWczRefrR55pPuZZ4bn996b3Nhr0ssvh5gn\nTKi4XGGh++uvh6qYgw/e8b779HH/3e/C+Ly1pYWSZCdSXb0DHEu4WLtXzLwrgKtjpq8BflXR9upi\n0nd3/8Mfwt6+9NLUb3vePPfu3cP2R45Mbvvu5cvdJ08O1Rrg/pOfZF7yK24SuWDBzvO//tr92WfD\n623bhvfXsKH7j38cPk/VuUttksyk3wBYCXRkx4XcrqXK9CJc7D2o1PwzgL9G62hIaNlzSkXbq6tJ\n3z103wAhYSRbWReCCwrcf/3rkND22ScksJq0alXtr8cvyxdfuO+9d7ibes0a9//7v9DNwG67hc9r\n993DwfKxx2pfax+RYklL+mFdnAj8J0rsE6N5k4Ch0fO/Ap8DC6PHrGh+feDu6BfAEuDWyrZVl5N+\nbFPOF15I3nrLavLZuHFIZOB+/vmhywgp35NP7rz/9t/ffdy4UP2TiQcyyT7xJn31spliW7fCj34E\ny5eHEYR69Eh8nTk54Wae0urXh5degmOPTXwb2eB//id0CTxsGPTsWXtuKBOJR7wXcpX002DNmtAd\nM4RWNPvum9j66tUr/w7aWvbxikgN0Ri5tdi++8Lzz8OmTXDyyeHsvzry8+G++8pvatihQ/VjFJG6\nSUk/TXr0CG3e33039FGzfXvly2zbFga3+NWvQvvy/fcPoxM1abLroBZNm8LkyTUTu4hkLiX9NDrx\nxDCU33PPweWXl11m9Wq46y449dQwxOAxx8DUqaEjs5tuCiMnbdgADz4YzuzNwt977oG8vNS+HxGp\n/TRGbpqNGxcGZZ86NfRSeMEF4c7dF18MF2GXLg3lOnQI47yecEIYsq70nb15eUryIlI5XcitBbZv\nD+PHvvBCqKopKAgDrw8cCEOGhER/yCFqTSIi5Yv3Qq7O9GuB+vXh0UdDf/Z77BES/aBBYYBrEZFk\nUtKvJZo3D4NliIjUJF3IFRHJIkr6AoRfGTk54UavnBz96hCpq1S9I8yYEcZjLSgI08Xjs4JaBInU\nNTrTFyZO3JHwixUUhPkiUrco6Qsff1y1+SKSuZT0hfbtqzZfRDKXkr4weXLoqyeW+u4RqZuU9IW8\nvNBXj/ruEan71HpHAPXdI5ItdKYvIpJFlPQlKXRzl0hmUPWOJEw3d4lkDp3pS8J0c5dI5lDSl4Tp\n5i6RzKGkLwnTzV0imUNJXxKmm7tEMoeSviRMN3eJZI64kr6ZDTGzZWa2wswmlPH6eDNbYmaLzOwV\nM+sQ81p7M/uLmS2NyuQkL3ypLfLyYNUqKCoKf5XwRWqnSpO+mdUHpgMnAF2AUWbWpVSxd4Bcd+8O\nPAlMiXntIeAmd+8M9AXWJSNwqVvUzl8kNeI50+8LrHD3le6+DZgJDIst4O5z3b240d4bQDuA6ODQ\nwN1fjsptjSknAuxo5796NbjvaOevxC+SfPEk/f2AT2Km86N55TkfeDF6fjCwycz+bGbvmNlN0S+H\nnZjZGDObZ2bz1q9fH2/sUkeonb9I6iT1Qq6ZjQZygZuiWQ2AAcDlwOFAJ+Cc0su5+z3unuvuuW3b\ntk1mSJIB1M5fJHXiSfqfAvvHTLeL5u3EzI4FJgJD3f27aHY+sDCqGioEngF6Jxay1DVq5y+SOvEk\n/beBg8yso5k1AkYCs2ILmFkv4G5Cwl9XatlWZlZ8+n40sCTxsKUuUTt/kdSpNOlHZ+jjgDnAUuAJ\nd19sZpPMbGhU7CagOfAnM1toZrOiZbcTqnZeMbP3AAPurYH3IRlM7fxFUsfcPd0x7CQ3N9fnzZuX\n7jBERDKKmc1399zKyumOXKkT1M5fJD7qT18ynvrzF4mfzvQl46mdv0j8lPQl46mdv0j8lPQl46md\nv0j8lPQl46mdv0j8lPQl46mdv0j8lPSlTki0P381+ZRsoSabkvXU5FOyic70JeupyadkEyV9yXpq\n8inZRElfsp6afEo2UdKXrKcmn5JNlPQl66nJp2QTtd4RISR4JXnJBjrTF0kCtfOXTKEzfZEEqZ2/\nZBKd6YskSO38JZMo6YskSO38JZMo6YskKBnt/HVNQFJFSV8kQYm28y++JrB6NbjvuCagxC81QUlf\nJEGJtvPXNQFJJXP3dMewk9zcXJ83b166wxBJmXr1whl+aWahq2iReJjZfHfPrayczvRF0kzXBCSV\nlPRF0kzXBCSV4kr6ZjbEzJaZ2Qozm1DG6+PNbImZLTKzV8ysQ6nXdzezfDP7Q7ICF6krdE1AUqnS\nOn0zqw/8B/gxkA+8DYxy9yUxZQYDb7p7gZmNBQa5+xkxr98OtAW+cPdxFW1PdfoiVaNrAgLJrdPv\nC6xw95Xuvg2YCQyLLeDuc929+FzjDaBdTCB9gB8Af4k3eBGJn8YDkKqIJ+nvB3wSM50fzSvP+cCL\nAGZWD7gFuLyiDZjZGDObZ2bz1q9fH0dIIlJM4wFIVST1Qq6ZjQZygZuiWT8HZrt7fkXLufs97p7r\n7rlt27ZNZkgidZ7GA5CqiKeXzU+B/WOm20XzdmJmxwITgYHu/l00+0hggJn9HGgONDKzre6+y8Vg\nEak+jQcg8YrnTP9t4CAz62hmjYCRwKzYAmbWC7gbGOru64rnu3ueu7d39xxCFc9DSvgitY/a+WeP\nSs/03b3QzMYBc4D6wP3uvtjMJgHz3H0WoTqnOfAnMwP42N2H1mDcIpIkGg8gu6gbBpEsl5MTEn1p\nHTrAqlXxrWPGjHBfwMcfh1ZDkyfrgJFq8TbZ1MhZIlku0fEA9Eshs6gbBpEsl2g7f90RnFmU9EWy\nXKLt/DVyWGZR0hfJcom289cdwZlFSV9EyMsLF22LisLfqtTF647gzKKkLyIJScYdwbpPIHXUekdE\nEpbIHcFq/ZNaOtMXkbRS65/UUtIXkbRS65/UUtIXkbRS65/UUtIXkbRKRusfXQiOn5K+iKRVoq1/\nkjEwfDYdNNThmohktEQ7jCvdegjCL41MG4gmmWPkiojUWoleCM621kNK+iKS0RK9EJxtrYeU9EUk\noyV6ITjbWg8p6YtIRkv0QnC2tR5SNwwikvES6QaieLnqjvyVad1IqPWOiEgCkjHcZDKo9Y6ISApk\n2oVgJX0RkQRk2oVgJX0RkQRk2iAySvoiIgnItEFk1HpHRCRBmTSITFxn+mY2xMyWmdkKM5tQxuvj\nzWyJmS0ys1fMrEM0v6eZ/dvMFkevnZHsNyAikslS3Q1EpUnfzOoD04ETgC7AKDPrUqrYO0Cuu3cH\nngSmRPMLgLPcvSswBJhqZq2SFbyISKZLdeufeM70+wIr3H2lu28DZgLDYgu4+1x3Lz5WvQG0i+b/\nx92XR8/XAOuAtskKXkQk06W69U88SX8/4JOY6fxoXnnOB14sPdPM+gKNgA/LeG2Mmc0zs3nr16+P\nIyQRkboh1a1/ktp6x8xGA7nATaXm7wM8DJzr7kWll3P3e9w9191z27bVDwERyR7JaP1TFfG03vkU\n2D9mul00bydmdiwwERjo7t/FzN8deAGY6O5vJBauiEjdk0jrn6qK50z/beAgM+toZo2AkcCs2AJm\n1gu4Gxjq7uti5jcCngYecvcnkxe2iIhUR6VJ390LgXHAHGAp8IS7LzazSWY2NCp2E9Ac+JOZLTSz\n4oPC6cCPgHOi+QvNrGfy34aIiMRDvWyKiNQB6mVTRER2oaQvIpJFal31jpmtB8oYkiBuewIbkhRO\nTVB8iVF8iVF8ianN8XVw90rbvNe6pJ8oM5sXT71Wuii+xCi+xCi+xNT2+OKh6h0RkSyipC8ikkXq\nYtK/J90BVELxJUbxJUbxJaa2x1epOlenLyIi5auLZ/oiIlIOJX0RkSySkUk/juEbG5vZ49Hrb5pZ\nTgpj29/M5kbDRy42s0vLKDPIzDbH9Ed0Tarii4lhlZm9F21/l34vLJgW7cNFZtY7hbEdErNvFprZ\nFjO7rFSZlO5DM7vfzNaZ2fsx8/Yws5fNbHn0t3U5y54dlVluZmenML6bzOyD6PN7urxR6yr7LtRg\nfNea2acxn+GJ5Sxb4f97Dcb3eExsq8xsYTnL1vj+Syp3z6gHUJ8wEEsnwqAs7wJdSpX5OXBX9Hwk\n8HgK49sH6B09bwH8p4z4BgHPp3k/rgL2rOD1EwmD4RhwBPBmGj/vzwg3nqRtHxI6DuwNvB8zbwow\nIXo+AbixjOX2AFZGf1tHz1unKL7jgAbR8xvLii+e70INxnctcHkcn3+F/+81FV+p128BrknX/kvm\nIxPP9CsdvjGafjB6/iRwjJlZKoJz97XuviB6/hWhZ9KKRhqrrYYRusR2D+MgtIoGw0m1Y4AP3T2R\nu7QT5u6vAV+Umh37PXsQOLWMRY8HXnb3L9z9S+BlwnjRNR6fu//FQy+5EDOMaTqUs//iEc//e8Iq\nii/KHacDjyV7u+mQiUk/nuEbS8pEX/rNQJuURBcjqlbqBbxZxstHmtm7ZvaimXVNaWCBA38xs/lm\nNqaM16s6TGZNGUn5/2zp3oc/cPe10fPPgB+UUaa27MfzKGMY00hl34WaNC6qfrq/nOqx2rD/BgCf\nezTedxnSuf+qLBOTfkYws+bAU8Bl7r6l1MsLCNUVPYA7gGdSHR9wlLv3Bk4ALjazH6UhhgpZGIRn\nKPCnMl6uDfuwhIff+bWy/bOZTQQKgRnlFEnXd+FO4ACgJ7CWUIVSG42i4rP8Wv+/FCsTk348wzeW\nlDGzBkBLYGNKogvbbEhI+DPc/c+lX3f3Le6+NXo+G2hoZnumKr5ou59Gf9cRRjfrW6pIXMNk1rAT\ngAXu/nnpF2rDPgQ+L67yiv6uK6NMWvejmZ0DnAzkRQemXcTxXagR7v65u2/3MG72veVsN937rwFw\nGvB4eWXStf+qKxOTfqXDN0bTxa0kRgCvlveFT7ao/u//gKXufms5ZfYuvsZgZn0Jn0MqD0rNzKxF\n8XPCBb/3SxWbBZwVteI5AtgcU5WRKuWeYaV7H0Ziv2dnA8+WUWYOcJyZtY6qL46L5tU4MxsC/Iow\njGlBOWXi+S7UVHyx14iGl7PdeP7fa9KxwAfunl/Wi+ncf9WW7ivJ1XkQWpb8h3BVf2I0bxLhyw3Q\nhFAlsAJ4C+iUwtiOIvzMXwQsjB4nAhcBF0VlxgGLCS0R3gB+mOL91yna9rtRHMX7MDZGA6ZH+/g9\nIDfFMTYjJPGWMfPStg8JB5+1wPeEeuXzCdeJXgGWA38F9ojK5gL3xSx7XvRdXAGcm8L4VhDqw4u/\nh8Ut2vYFZlf0XUhRfA9H361FhES+T+n4ould/t9TEV80/4/F37mYsinff8l8qBsGEZEskonVOyIi\nUk1K+iIiWURJX0Qkiyjpi4hkESV9EZEsoqQvIpJFlPRFRLLI/wO0gNF2vd8PIQAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f703c7fe940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Judging from the validation loss, this setup is not quite as good as the regularized GRU alone, but it's significantly faster. It is \n",
    "looking at twice more data, which in this case doesn't appear to be hugely helpful, but may be important for other datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "Here's what you should take away from this section:\n",
    "\n",
    "* In the same way that 2D convnets perform well for processing visual patterns in 2D space, 1D convnets perform well for processing \n",
    "temporal patterns. They offer a faster alternative to RNNs on some problems, in particular NLP tasks.\n",
    "* Typically 1D convnets are structured much like their 2D equivalents from the world of computer vision: they consist of stacks of `Conv1D` \n",
    "layers and `MaxPooling1D` layers, eventually ending in a global pooling operation or flattening operation.\n",
    "* Because RNNs are extremely expensive for processing very long sequences, but 1D convnets are cheap, it can be a good idea to use a 1D \n",
    "convnet as a preprocessing step before a RNN, shortening the sequence and extracting useful representations for the RNN to process.\n",
    "\n",
    "One useful and important concept that we will not cover in these pages is that of 1D convolution with dilated kernels."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}