t81_558_class6_backpropagation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 6: Backpropagation.**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 6.1: Backpropagation Introduction](https://www.youtube.com/watch?v=Oe0-hX4KSZQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=18)\n",
    "* [Part 6.2: Inside Backpropagation](https://www.youtube.com/watch?v=GDlI-3O5r6I&index=19&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 6.3: ADAM Update Rule](https://www.youtube.com/watch?v=zUZRUTJbYm8&index=20&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "\n",
    "Weekly video update:\n",
    "\n",
    "* [Weekly Update #6, Feb 18, 2018](https://www.youtube.com/watch?v=vD2BrT4v_G0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import os\n",
    "import requests\n",
    "import base64\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name, x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = \"{}-{}\".format(name, tv)\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df.as_matrix(result).astype(np.float32), dummies.as_matrix().astype(np.float32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32), df.as_matrix([target]).astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred,y,sort=True):\n",
    "    t = pd.DataFrame({'pred' : pred, 'y' : y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'],inplace=True)\n",
    "    a = plt.plot(t['y'].tolist(),label='expected')\n",
    "    b = plt.plot(t['pred'].tolist(),label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean()) >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "               * (normalized_high - normalized_low) + normalized_low\n",
    "        \n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Classic Backpropagation\n",
    "Backpropagation is the primary means by which a neural network's weights are determined during training. Backpropagation works by calculating a weight change amount ($v_t$) for every weight($\\theta$, theata) in the neural network.  This value is subtracted from every weight by the following equation: \n",
    "\n",
    "$ \\theta_t = \\theta_{t-1} - v_t $\n",
    "\n",
    "This process is repeated for every iteration($t$).  How the weight change is calculated depends on the training algorithm.  Classic backpropagation simply calculates a gradient ($\\nabla$, nabla) for every weight in the neural network with respect to the error function ($J$) of the neural network.  The gradient is scaled by a learning rate ($\\eta$, eta).\n",
    "\n",
    "$ v_t = \\eta \\nabla_{\\theta_{t-1}} J(\\theta_{t-1}) $\n",
    "\n",
    "The learning rate is an important concept for backpropagation training.  Setting the learning rate can be complex:\n",
    "\n",
    "* Too low of a learning rate will usually converge to a good solution; however, the process will be very slow.\n",
    "* Too high of a learning rate will either fail outright, or converge to a higher error than a better learning rate.\n",
    "\n",
    "Common values for learning rate are: 0.1, 0.01, 0.001, etc.\n",
    "\n",
    "Gradients:\n",
    "\n",
    "![Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv.png \"Derivative\")\n",
    "\n",
    "The following link, from the book, shows how a simple [neural network is trained with backpropagation](http://www.heatonresearch.com/aifh/vol3/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Momentum Backpropagation\n",
    "\n",
    "Momentum adds another term to the calculation of $v_t$:\n",
    "\n",
    "$ v_t = \\eta \\nabla_{\\theta_{t-1}} J(\\theta_{t-1}) + \\lambda v_{t-1} $\n",
    "\n",
    "Like the learning rate, momentum adds another training parameter that scales the effect of momentum.  Momentum backpropagation has two training parameters: learning rate ($\\eta$, eta) and momentum ($\\lambda$, lambda).  Momentum simply adds the scaled value of the previous weight change amount ($v_{t-1}$) to the current weight change amount($v_t$).\n",
    "\n",
    "This has the effect of adding additional force behind a direction a weight was moving.  This might allow the weight to escape a local minima:\n",
    "\n",
    "![Momentum](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_5_momentum.png \"Momentum\")\n",
    "\n",
    "A very common value for momentum is 0.9.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch and Online Backpropagation\n",
    "\n",
    "How often should the weights of a neural network be updated?  Gradients can be calculated for a training set element.  These gradients can also be summed together into batches and the weights updated once per batch.\n",
    "\n",
    "* **Online Training** - Update the weights based on gradients calculated from a single training set element.\n",
    "* **Batch Training** - Update the weights based on the sum of the gradients over all training set elements.\n",
    "* **Batch Size** - Update the weights based on the sum of some batch size of training set elements.\n",
    "* **Mini-Batch Training** - The same as batch size, but with a very small batch size.  Mini-batches are very popular and they are often in the 32-64 element range.\n",
    "\n",
    "Because the batch size is smaller than the complete training set size, it may take several batches to make it completely through the training set.  \n",
    "\n",
    "* **Step/Iteration** - The number of batches that were processed.\n",
    "* **Epoch** - The number of times the complete training set was processed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stochastic Gradient Descent\n",
    "\n",
    "Stochastic gradient descent (SGD) is currently one of the most popular neural network training algorithms.  It works very similarly to Batch/Mini-Batch training, except that the batches are made up of a random set of training elements.\n",
    "\n",
    "This leads to a very irregular convergence in error during training:\n",
    "\n",
    "![SGD Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_5_sgd_error.png \"SGD Error\")\n",
    "[Image from Wikipedia](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)\n",
    "\n",
    "Because the neural network is trained on a random sample of the complete training set each time, the error does not make a smooth transition downward.  However, the error usually does go down.\n",
    "\n",
    "Advantages to SGD include:\n",
    "\n",
    "* Computationally efficient.  Even with a very large training set, each training step can be relatively fast.\n",
    "* Decreases overfitting by focusing on only a portion of the training set each step.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Techniques\n",
    "\n",
    "One problem with simple backpropagation training algorithms is that they are highly sensative to learning rate and momentum.  This is difficult because:\n",
    "\n",
    "* Learning rate must be adjusted to a small enough level to train an accurate neural network.\n",
    "* Momentum must be large enough to overcome local minima, yet small enough to not destabilize the training.\n",
    "* A single learning rate/momentum is often not good enough for the entire training process. It is often useful to automatically decrease learning rate as the training progresses.\n",
    "* All weights share a single learning rate/momentum.\n",
    "\n",
    "Other training techniques:\n",
    "\n",
    "* **Resilient Propagation** - Use only the magnitude of the gradient and allow each neuron to learn at its own rate.  No need for learning rate/momentum; however, only works in full batch mode.\n",
    "* **Nesterov accelerated gradient** - Helps mitigate the risk of choosing a bad mini-batch.\n",
    "* **Adagrad** - Allows an automatically decaying per-weight learning rate and momentum concept.\n",
    "* **Adadelta** - Extension of Adagrad that seeks to reduce its aggressive, monotonically decreasing learning rate.\n",
    "* **Non-Gradient Methods** - Non-gradient methods can *sometimes* be useful, though rarely outperform gradient-based backpropagation methods.  These include: [simulated annealing](https://en.wikipedia.org/wiki/Simulated_annealing), [genetic algorithms](https://en.wikipedia.org/wiki/Genetic_algorithm), [particle swarm optimization](https://en.wikipedia.org/wiki/Particle_swarm_optimization), [Nelder Mead](https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method), and [many more](https://en.wikipedia.org/wiki/Category:Optimization_algorithms_and_methods)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ADAM Update\n",
    "\n",
    "ADAM is the first training algorithm you should try.  It is very effective.  Kingma and Ba (2014) introduced the Adam update rule that derives its name from the adaptive moment estimates that it uses.  Adam estimates the first (mean) and second (variance) moments to determine the weight corrections.  Adam begins with an exponentially decaying average of past gradients (m):\n",
    "\n",
    "$ m_t = \\beta_1 m_{t-1} + (1-\\beta_1) g_t $\n",
    "\n",
    "This average accomplishes a similar goal as classic momentum update; however, its value is calculated automatically based on the current gradient ($g_t$).  The update rule then calculates the second moment ($v_t$):\n",
    "\n",
    "$ v_t = \\beta_2 v_{t-1} + (1-\\beta_2) g_t^2 $\n",
    "\n",
    "The values $m_t$ and $v_t$ are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively.  However, they will have a strong bias towards zero in the initial training cycles.  The first moment’s bias is corrected as follows.\n",
    "\n",
    "$ \\hat{m}_t = \\frac{m_t}{1-\\beta^t_1} $\n",
    "\n",
    "Similarly, the second moment is also corrected:\n",
    "\n",
    "$ \\hat{v}_t = \\frac{v_t}{1-\\beta_2^t} $\n",
    "\n",
    "These bias-corrected first and second moment estimates are applied to the ultimate Adam update rule, as follows:\n",
    "\n",
    "$ \\theta_t = \\theta_{t-1} - \\frac{\\alpha \\cdot \\hat{m}_t}{\\sqrt{\\hat{v}_t}+\\eta} \\hat{m}_t $\n",
    "\n",
    "Adam is very tolerant to initial learning rate (\\alpha) and other training parameters. Kingma and Ba (2014)  propose default values of 0.9 for $\\beta_1$, 0.999 for $\\beta_2$, and 10-8 for $\\eta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Methods Compared\n",
    "\n",
    "The following image shows how each of these algorithms train (image credits: [author](Alec Radford), [where I found it](http://sebastianruder.com/optimizing-gradient-descent/index.html#visualizationofalgorithms) ):\n",
    "\n",
    "![Training Techniques](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/contours_evaluation_optimizers.gif \"Training Techniques\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specifying the Update Rule in Tensorflow\n",
    "\n",
    "TensorFlow allows the update rule to be set to one of:\n",
    "\n",
    "* Adagrad\n",
    "* **Adam**\n",
    "* Ftrl\n",
    "* Momentum\n",
    "* RMSProp\n",
    "* **SGD**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 00255: early stopping\n",
      "Score (RMSE): 3.910839796066284\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD8CAYAAAB6paOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsnXd8VeX9x99PdkgC2WEkIewdVgIi\nIoIKuKq4rQsXjtraLXb82tpabWvVal04ceFCwboVQWTIkr0JBDJIyB5kkPH8/njuyb1JbpKbkHuT\nkO/79crr5p57xnOjnM/5bqW1RhAEQei+eHX0AgRBEISORYRAEAShmyNCIAiC0M0RIRAEQejmiBAI\ngiB0c0QIBEEQujkiBIIgCN0cEQJBEIRujgiBIAhCN8enoxfgCpGRkTohIaGjlyEIgtCl2Lx5c67W\nOqql/dwuBEopb2ATkKG1vlgp9SowHSiy7TJPa721uXMkJCSwadMm9y5UEAThNEMpdcSV/TxhEdwH\n7AF6Omz7jdb6fQ9cWxAEQWgBt8YIlFKxwEXAi+68jiAIgtB23B0sfgL4LVDbYPtDSqntSqnHlVL+\nbl6DIAiC0Axucw0ppS4GjmutNyulznH46AEgC/ADFgL3Aw86OX4+MB8gPj6+0fmrqqpIT0+noqKi\n/RffTQkICCA2NhZfX9+OXoogCB7EnTGCqcCPlFIXAgFAT6XUG1rrG2yfVyqlXgF+7exgrfVCjFCQ\nlJTUaGhCeno6ISEhJCQkoJRyzzfoRmitycvLIz09nQEDBnT0cgRB8CBucw1prR/QWsdqrROAa4Fv\ntNY3KKX6AChz974M2NmW81dUVBARESEi0E4opYiIiBALSxC6IR1RR/CmUioKUMBW4K62nkhEoH2R\nv6cgdE88IgRa65XAStvvMz1xTUEQhK7M8ZIKFq1N5YoJsQyMCnbrtaTFRBciNTWVt956q9XHzZs3\nj/ffl7INQehKHDxeytMrUsgqdr+7VoSgC9FWIRAEoeuRWWgEoF9ooNuvJUJwCrzxxhtMmjSJcePG\nceedd3LkyBGGDBlCbm4utbW1TJs2jS+//JLU1FSGDx/OzTffTGJiIldeeSVlZWUAbN68menTpzNx\n4kRmz57NsWPHADh48CDnnXceY8eOZcKECaSkpLBgwQK+++47xo0bx+OPP05NTQ2/+c1vSE5OJjEx\nkeeffx4wGUD33nsvI0eO5KKLLuL48eMd9jcSBKFtZBaWA9C7V4Dbr9Ulms61xF/+t4vdmcXtes6R\nfXvyp0tGNfn5nj17eOedd1izZg2+vr7cc889fPvtt9x///3cddddTJ48mZEjRzJr1ixSU1PZt28f\nL730ElOnTuXWW2/lmWee4b777uOnP/0py5YtIyoqinfeeYff//73vPzyy1x//fUsWLCAuXPnUlFR\nQW1tLY888giPPvooH3/8MQALFy6kV69ebNy4kcrKSqZOncqsWbPYsmUL+/btY8eOHWRnZzNy5Ehu\nvfXWdv37CILgXjILy4kK8cffx9vt1zothKAjWL58OZs3byY5ORmA8vJyoqOj+fOf/8x7773Hc889\nx9at9l56cXFxTJ06FYAbbriBJ598kjlz5rBz507OP/98AGpqaujTpw8lJSVkZGQwd+5cwBR6OePL\nL79k+/btdf7/oqIiDhw4wKpVq7juuuvw9vamb9++zJwp8XlB6GpkFJbT1wNuIThNhKC5J3d3obXm\n5ptv5uGHH663vaysjPT0dABKS0sJCQkBGqdmKqXQWjNq1CjWrVtX77PiYtesG601Tz31FLNnz663\n/dNPP5VUUEHo4mQWljOsd4hHriUxgjZy7rnn8v7779f53/Pz8zly5Aj3338/119/PQ8++CB33HFH\n3f5Hjx6tu+EvXryYs846i2HDhpGTk1O3vaqqil27dtGzZ09iY2NZunQpAJWVlZSVlRESEkJJSUnd\nOWfPns2zzz5LVVUVAPv37+fEiROcffbZvP3229TU1HDs2DFWrFjhkb+JIAjtg9aazMIK+vbyjEUg\nQtBGRo4cyd/+9jdmzZpFYmIi559/PqmpqWzcuLFODPz8/HjllVcAGDFiBIsWLSIxMZH8/Hzuvvtu\n/Pz8eP/997n//vsZO3Ys48aNY+3atQC8/vrrPPnkkyQmJnLmmWeSlZVFYmIiPj4+jB07lscff5zb\nb7+dkSNHMmHCBEaPHs2dd95JdXU1c+fOZciQIYwZM4a7776b6dOnd+SfShCEVlJYVkV5VY3HXENK\n60ZtfDodSUlJuuFgmj179jBixIgOWlHrSE1N5eKLL2bnzjZ10/AoXenvKginKzszirj4qdU8d8NE\n5ozu3ebzKKU2a62TWtpPLAJBEIROhpU62jfU/amjIELgERISErqENSAIQufALgQSIxAEQeiWZBZV\n4OfjRUSQn0euJ0IgCILQycgoLKdfaKDH0sBFCARBEDoZmYXlHosPgAiBIAhCpyOzsNxjNQQgQtCp\nCA42PcczMzO58sorm933iSeeqGtcB3DhhRdSWFjo1vUJguB+TlbXcryk0mOBYhAhcDs1NTWtPqZv\n374tzg9oKASffvopoaGhrb6WIAidi+ziCrT2TPtpCxGCU6Cp9tIJCQk8+OCDnHXWWbz33nukpKQw\nZ84cJk6cyLRp09i7dy8Ahw8fZsqUKSQnJ/PHP/6x3nlHjx4NGCH59a9/zZgxY0hMTOSpp57iySef\nJDMzkxkzZjBjxgzApKjm5uYC8NhjjzF69GhGjx7NE088UXfOESNGcMcddzBq1ChmzZpFeXm5J/9c\ngiC4QIaHU0fhNGk6x2cLIGtH+56z9xi44JEWd3PWXhpMx9DVq1cDpi/Rc889x5AhQ1i/fj333HMP\n33zzDffddx933303N910E08//bTT8y9cuJDDhw+zZcsWfHx8yM/PJzw8nMcee4wVK1YQGRlZb//N\nmzfzyiuvsH79erTWTJ48menTpxMWFsaBAwdYvHgxL7zwAldffTVLlizhhhtuOMU/lCAI7Ymni8nA\nAxaBUspbKbVFKfWx7f0ApdR6pdQBpdQ7SinPJMq6iYbtpa2b/zXXXAOYDqRr167lqquuqhtgYw2f\nWbNmDddddx0AN954o9Pzf/3119x11134+BjNDg8Pb3Y9q1evZu7cuQQFBREcHMzll1/Od999B8CA\nAQMYN24cABMnTiQ1NfUUvrkgCO7A08Vk4BmL4D5gD9DT9v4fwONa67eVUs8BtwHPntIVXHhydxfO\n2ksDBAUFAVBbW0toaGi92QTNHd8QrXWrcomb6x3l7+9f97u3t7e4hgShE5JRWEFEkB8Bvu4fSGPh\nVotAKRULXAS8aHuvgJmAFQldBFzmzjW4G2ftpR3p2bMnAwYM4L333gPMjXrbtm0ATJ06lbfffhuA\nN9980+n5Z82axXPPPUd1dTVg2l0DjVpSW5x99tksXbqUsrIyTpw4wYcffsi0adPa4ZsKguAJMj04\nkMbC3a6hJ4DfArW29xFAoda62vY+Hejn7ECl1Hyl1Cal1KacnBw3L7PtOGsv3ZA333yTl156ibFj\nxzJq1CiWLVsGwH/+8x+efvppkpOTKSoqcnr+22+/nfj4eBITExk7dmzd8Pr58+dzwQUX1AWLLSZM\nmMC8efOYNGkSkydP5vbbb2f8+PHt/K0FQXAXni4mAze2oVZKXQxcqLW+Ryl1DvBr4BZgndZ6sG2f\nOOBTrfWY5s7VWdtQd6X20q7SGf6ugtBd0Voz+k9fcHVyXLtMXnS1DbU7YwRTgR8ppS4EAjAxgieA\nUKWUj80qiAUy3bgGQRCELkNxeTUnTtZ4tIYA3Oga0lo/oLWO1VonANcC32itrwdWAFbZ7M3AMnet\nwd1Ie2lBENqTjqghgI4pKLsf+KVS6iAmZvBSW0/UFaardSXk7ykIHUtHpI6ChwrKtNYrgZW23w8B\nk071nAEBAeTl5REREeGxVq2nM1pr8vLyCAjwbJBKEAQ7mUWeLyaDLlxZHBsbS3p6Op05o6irERAQ\nQGxsbEcvQxC6LRmF5fh5exEZ5N/yzu1IlxUCX19fBgwY0NHLEARBaDcyCyvoExqAl5dnvRzSdE4Q\nBKGT4Ok5BBYiBIIgCJ2EjqgqBhECQRCETkFldQ3ZxRUeDxSDCIEgCEKHo7XmgSU7qNWQnNB8h2F3\nIEIgCILQwTz21X4+2JLBL88fytlDozx+fRECQRCEDmTxhqM89c1Brk2O46czB3fIGkQIBEEQOogV\n+47zh6U7mT40ir9eNrrDimNFCARBEDqAnJJKfrZ4C8N7h/D09RPw9e6427EIgSAIQgfwry/2Un6y\nhievG0+wf8fW9ooQCIIgeJhtaYW8tzmdW88awKCoYOc75eyD58+GI+vcvh4RAkEQBA9SW6v58/92\nERHk33xwuPAoHNsGyv23aRECQRAED/Lhlgy2HC1kwQXDCQnwbXrHYtvMrp593L4mEQJBEAQPUVJR\nxSOf72VcXCiXj3c6rt1h52PmNbi329fVZbuPCoIgdHY2puazeP1RrJFPafll5JRU8uJNSS13GC3O\nhB6R4OPn9nWKEAiCILiBspPV3PPmD1RU1RDWw34z/+X5QxkbF9ryCUqOecQtBCIEgiAIbuGl7w6T\nU1LJkrunMLF/G/oHFR+Dnn3bf2FOcFuMQCkVoJTaoJTappTapZT6i237q0qpw0qprbafce5agyAI\nQkeQV1rJ86sOMXtUTNtEAKAk87SwCCqBmVrrUqWUL7BaKfWZ7bPfaK3fd+O1BUEQOoz/rjhI2clq\nfjN7eNtOUF0JZXkQ4hmLwG1CoLXWQKntra/tRzd9hCAIQtcnLb+MN74/wjXJcQyObqJYrCWsjCEP\nWQRuTR9VSnkrpbYCx4GvtNbrbR89pJTarpR6XCnl2SnNgiAIbuTRL/fh7aX4+XlD236SYpsQeMgi\ncKsQaK1rtNbjgFhgklJqNPAAMBxIBsKB+50dq5Sar5TapJTalJOT485lCoIgnDJaa5bvyWbZ1kxu\nO2sAMT1PYdJYieeKycBDWUNa60Kl1Epgjtb6UdvmSqXUK8CvmzhmIbAQICkpSVxKgiB0SspP1vDh\nlgxeW5fK3qwS+oUGcuf0Qad20jqLoIsLgVIqCqiyiUAgcB7wD6VUH631MWUab18G7HTXGgRBENxF\nWn4Zr39/hHc2plFUXsWIPj355xWJ/GhcXwJ8vU/t5CXHwCcAAsPaZ7Et4E6LoA+wSCnljXFBvau1\n/lgp9Y1NJBSwFbjLjWsQBEFoV9al5PHymsMs35ONUorZo2KYd+YAkhPC2m+wTHGmsQY8NKjGnVlD\n24HxTrbPdNc1BUEQ3EVNrebvn+7hpdWHCQ/y4+5zBnH95P70DQ1s/4uVeK6YDKSyWBAEoUXKTlbz\ns8Vb+XpPNvPOTGDBBcNP3f3THCXHoF+S+87fABECQRCEZsguruC2RRvZnVnMny8ZybypA9x7Qa1N\nsHiEZwLFIEIgCILQJOtS8vjFO1spqajixZuTmDk8xv0XLS+AmkqP1RCACIEgCEIjTlbX8thX+3l+\nVQoJEUG8PC+ZkX17eubiHhxIYyFCIAjC6YXWkL0Teo9xaffSymqO5J2wv6+o5m+f7GFHRhHXTYrj\njxePpIefB2+VJZ6tKgYRAkEQTjdSlsMbV8DdayFmVLO77swo4rZFG8kurqy3PbSHL8/dMJE5o90/\nHawRYhEIgiCcIrkHba/7mxWCr3dn87O3txAa6Mt/rh1XLwtoQnwYUSEd1AbNgyMqLUQIBKEzUVsD\nm16G8TeC7yn0qmkPtr1tfm5a2rHraC3F6ea18KjTj7XWvLImlb99spvR/Xrx4k1JRJ9KX6D2pjgT\ngqI8MqLSQobXC0Jn4uj38Omv4eDXHb0S2PM/OLQCqso7eiWtw3KtFKY5/fj5VYd48OPdnD8yhrfn\nn9G5RACMReChHkMWIgSC0Jmw3AInOkHH3azt5rX0eMeuo7UUZZhXJxZBwYmT/Pebg5w3Ippnr5/o\n2SCwq3hwRKWFCIEgdCbqhCC3Y9dRXmi/kXYGUWoNxU0LwYurD3HCNjnMy8szfXxaTUmmWASC0K0p\nyTKvZR0sBNkOTYFLsztuHa2ltsbuGipKM6mkNgpOnOTVNalcOKYPw3qHdNACW8AaUSkWgSB0YzqL\nRZC1w/57VxKC0mzQNRAxBE6WmipdGy+uPkRZVQ33nTukAxfYAiWenUNgIUIgCJ2JzmIRHNsOPSLM\n710pRmDFB+LPMK+FRwDIt1kDF43pw9CYTmoNgH0gjQdrCEDSRwWhc1FnEeR17DqydkDf8ZDxQ9uE\noLoSltwOtdUQHAMhvY2w1FZDdQVUVUBQJCTf3r49963U0f5nwpbXTeZQ3/G8+F0XsAbAPqLSg1XF\nIEIgCJ0HrTuHRVB9EnL2wpDzzBN2W1xDufthz0fQKw6qNjT9fYbOhtD4U1uvIzaL4HBQIgOA9NR9\npAfksWhtKhcn9mVIZ7YGQCwCQej2VBZDVRl4+5sYgdYem1BVj5y9UFtlevW01SKwbmhXvgJxyUZc\nyvPB28+MYExbD69fBgWp7SsExZlUeQcy46VUdvgH8tXajfxl1Ui8FPxs5mDXzlFbA15unDXQHCXH\nwCcQAkI9elkRAkHoLFjWQPRwOLYNKoog0LM3BMAeKO6daNw6aetbf466oKetTYKPn/13gPCB5rUg\nFQac3ealNuRkwVEyasJITghHlcdzSXAVA85OJirEv2lrIHUN7PoA8g+Zn8I0mDQfLnik3dbllMI0\nePs6iJ0EF/7LiE9xprEGPPwA4LZgsVIqQCm1QSm1TSm1Syn1F9v2AUqp9UqpA0qpd5RSnqujFoTO\njHXztLpmlnVQnCBrB/j2MDfr4GhTR+CQhukSlqgFN9G/v2c/8PIxQtCOZKelkFEbzl8vG01wzEAi\nq49zzrBoRvXt5fyAjB/g9bmmlUZZPvSdYOILG1+AvJR2XVs9ClLh1QshZz9segmW3g011baqYs/G\nB8C9WUOVwEyt9VhgHDBHKXUG8A/gca31EKAAuM2NaxCEroN184yxCUFHpZBm7TDN2ry8jRBUlZlU\nzNZQ0kK/HG8fEz9oRyHYm1WM74lj9Ijsz/DePSE0zhSVNSViJdnw9vXmO963De78Fq56Ba54ybiw\nVvy93dZWj7wUeOUiqCiGWz+HmX+E7e/AktugKN3j8QFwoxBog/V/j6/tRwMzgfdt2xcBl7lrDYLQ\npbCEoPdo89qWgPHxvZB/uO1r0NoIQe9E8956om9tnKAkq74ryBlhCe0mBFpr/rpsG9GqkJHDR5iN\nofEm7lJR2PiA6kp490bz2bVvmQwmi5AYmHwX7Hy/fj3FqS/SpOW+ehFUl8PN/4N+E+DsX8Osv8Hu\npaYIzsM1BODmOgKllLdSaitwHPgKSAEKtdbVtl3SgX7uXIMgdBlKssAvBEL7m/dtsQjevQkWX9d6\nV45F4VGoLLK7p4KjzWtrM4dcaZzWjkLw6Y4sUg+n4IUmIMIWfLaC0A2bz2ltGvulrYfLnoE+iY1P\nOPVnENALvvnbqS2sNAfWL4T3boHHRsDz00wK7c0f17/umT+FC/5lfg9380xkJ7g1WKy1rgHGKaVC\ngQ+BEc52c3asUmo+MB8gPr4dswoEobNScsw8RVtPp621CMoLIHef+f3QShg0o/VrcAwUAwRZQtAG\ni6DP2Ob3CUswcZCKYgho+xjIkooqHvpkN2dGVUAJ0Mv2bNkrzrwWHq1/0936JvzwGkz7NYya6/yk\ngWEw9T5Y/qDpCGsVqLWWL/8A2982MZH+U815hl/kvIXE5Pnmv5n1IOBBPFJZrLUuBFYCZwChSilL\ngGKBzCaOWai1TtJaJ0VFRXlimYLQsVjuFN9A8AtuvUWQ8YN5VV7w/bNtW0PWDnN8tO2ZrS2uoZoq\ns39LQc+wBPNqq/5tluzddteZA1U1tdzz5g8cL6nkngmBZmPPWPNq3VAbNp/b/KqJw8z4ffPXnHyX\nEcLlDxorQmvzvbJ3u25xZe+CQefCL3fDlS/BpDua7yMUOcSjcwgs3Jk1FGWzBFBKBQLnAXuAFcCV\ntt1uBpa5aw2C0KVwdKf0iGijECiYfDcc+KJtWS9ZO0yfHr8etnWEg/JunWuo9DigXYsRQMvuoZx9\n8OK5sOzeepu11ixYsoPvDuTy8OVjGOhniwVYN9oe4eAbZPzuFmX5kLEZhl8IXi3c/vyC4OzfwJE1\n8N9k+HtfeHQIPDsF1v23+WPB1CPkHbCLaifGnRZBH2CFUmo7sBH4Smv9MXA/8Eul1EEgAnjJjWsQ\nhM7Hsnth+3v1t1lVxdbNMyiy9a6hjE0QORTO+rnJeln/XOvXlrW9/tB3L2+T/dMaIbCe3F2JEUDz\nQlBVAe/fZjKXDq2o13rj8a8PsOSHdH5x3lCuSoozVcX+Pe1uJqXsmUMWKd+AroXB57v2XSbOg5GX\nmb/rxHlwwT9h8HnGSsje1fyxRWmmnUbkUNeu1YG4LUagtd4OjHey/RAwyV3XFYROTVU5bHnDPP0n\nXmXfXl4ANZUOFkGkva7AFbQ2T7qDzzcB3tFXwpY3jfvD1aK0snxz80q+vf724KjWzSSo65fTgkUQ\nGGoqaJsTgq//BNk7ODH1AYLWPEzKqsUcH3od29MLeXL5Aa5OiuVn59oqhoszjC/ekdD4+q6ng8uN\n/7/fBNe+i48fXL2o/rbRV8AzZ8AH8+GOb8CnidnGOfvNa9Qw167VgUj3UUHwJHkHAd04LbHuKdrR\nImhFQVnhUXOzjp1o3p9xF1SdMI3XXMWaQeBoEYCJE7TFInClp35zmUP7PoP1z3Fi/B1MXTuelNo+\nZK19i+te+J6HP9vLtCGRPDR3DMqqwi1KtweKLULj7VlDtbVmBOigmafWQiIoEn70X/P3aq7WwArc\ndwGLQIRAEDxJju3mUJptCposGvaht2IErgYlMzab1342IegzFvqfZVIXa6qbPs6RjS8al1LfBoZ8\ncEzrgsUlx0xcoUdky/s2JQTFx2DpPejeY/jJ8UuprNao0Zdzps8e3r9hEO/eOYWX5yXj6+1wC3Nm\nEfSKM7UCFUWQvQNOHHfdLdQcw+bAhJthzX/gyFrn++TsM3+DHuGnfj03I0IgCJ4k94D9d0eroJFF\nEGVcRZUlrp03Y7NpVhcz2r7tjLuh6Cjs+6Tl43d/BLuXwTkLGt+4gqONELgqSlaso6VgLBghKDxq\nAquOfPl7qK5gyYAHWZlSzB8vHsnAc25C6VqSTqxi0oDw+iJQXWksol6x9c/jWEtw4Cvz++BzXfse\nLTH77xDWHz68E06WNf4890CXcAuBi0KglLrPlW2CILRA7j77k3LWNvv2hk3aWltLkLHZWAHevvZt\nwy4w19r/RfPHlheYAqveY+DMnzX+PCjadCN1mPbVLMWtmLkblgA1J+vHQ2qq4cBXFA66lN99V8l5\nI6K5blKcacYXPdI0iHN2TXASI7ClkBalmfhAn7H2IrlTxT8Y5vzDCNnRBlaB1ua/dRdwC4HrFsHN\nTrbNa8d1CEL3IPeACVSG9m9sEQSEmhoCsIuFKwNqaqogcyvEJtXf7uVt2lW0lN3yxR+MG+rSp+sL\niUVwK4vKXGkvYeEscyhzC1QW8/TROHoG+PDIFYn2OMDoy+HoOhMPcMQaWN8wLmFZBFk7TCVxe7iF\nHEk4y7jBjn5ff/uJXCOcXUQIms0aUkpdB/wYGKCU+sjhoxCgg0coCUIXo7bGCMHAc0ymybHt9s8a\ntmQIso2JdMUiOL7H9K6x4gOORI8y3S2b6rF/cDlsfQPO+mXTlcB1RWXZ5qm8JUqOmRtkM2w+UsCj\nX+wjrDKfZ4CnP/iKbwLNc+kVpYv5MfB+/kD+PS+RyGCHrJxRl5u2D7uWwpkOdQXWiMqGrqGgSNPf\nf8vrZpbx4PNaXn9r8A82VcsNhcAKFEedBkIArAWOAZHAvx22lwDbnR4hCIJzCo8av3/kUNPHZs/H\nJgbgH9L4KbrOInBBCDI2mVdnQhAz0uSy5x+GyAaDWU6egP/93Kxn+v1Nn7811cVV5SY424xFUFRW\nxU/e/IFarRkR3ZsavOinswn0NUI1rnobqb6Dufe8M5g5vEEb64hBRrB2fVBfCKwRlQ1dQ1YtQe5+\n8O8Fscktf4fWEneGqVauqbJbVLm21NHI0yBGoLU+orVeqbWeorX+1uHnB4fGcYIguEKuQ15570RA\n2902JVkNLIJWxAgyNpssI8vN4kj0SPN63Il76OByE0ye8wj4BjR9/mBbi5cTLgiB5etvInVUa83v\nlu4gt7SSl25OZtEdU/EOjeOy/lW8cftk3rhpNCOr95CQfCG3ndVE87XRV5jv7NhltSjD1AdYFdGO\nWO6hQeeY9tftTfwZxiJztPBy9puq5obC1ElxNVhcopQqtv1UKKVqlFLF7l6cIJxW1D0lDrXn6mft\nMPntpVmm/bGFX5AZDuOKRZC+2VgDzqZaRQ0HlOmP05CMTSZdtAU3DgGhZj9XagkaZj81YOnWDD7Z\nfoxfnD+UMbG2YTGOKaRH15ng8cBzmr6G1Shu5SPmbwe21NFY5/tbzefaOz5gYTWkS3NwD+XuMxaY\nK5lTnQCXVqm1DtFa97T9BABXAC402xCEduSH19w3NSp7V/28fnfgmFfes695ij+2zRSO1VY3zrTp\nEdmyEFSWmBnDztxCYJ6Qwwc6twjSNxtBaqoy1kIp12sJGtZDOF6uoIz/W7qLpP5h3DV9kP0DRyE4\ntNKITvyUpq8RGg/TF5iunp/+ymToFGU0LiaziBxiArrtHR+wCOltvsPRdfZtOfu7jFsI2thiQmu9\nVCm1oL0XIwhNUlUOH/3U/GO+YUn7n//Nq8xT6GXPtP+5LRzzypUyN+GsHY1TRy2CIlp2DWVuATT0\nS2p6n5iRjS2Cmmpz7IQbXVt7cHSzFkFVTS3b04volXKAwcB32T5UF9QXjmdXpKCBx68Zh7eXg/US\nlmC+Z2WJEYK4yc5dPI6cs8DEW1Y/bkZeFqdDXBOda5JuhQHT3Tv5K36KqVrW2sReitO7TKAYXBQC\npdTlDm+9gCSamCMgCG7Bcjkc/Nrc1GJGtt+5q8qNa8Fy3bgDK6985KX2bb0TTWM4qzumU4ughR4/\n1mD55nrnRI8ygemTZfYbbM4xrLh8AAAgAElEQVQe04KiOQFxJCi6Ucpm/omTfLU7i5X7clh9IJeS\nymp+57Odft5+3PjmPqCxq+rfV40lLrzBTd6KbWRsNsI4848tr0cpOPdPJkBrdQJtqqWFb6B96pu7\niJsM2xZD/iEzFQ1OS4vgEoffq4FU4FLnuwqCG3B8Gl333/Z9crdSD5sa8VheCO/fCnMebnulaFme\nLa/c4fjeicYffniVed/IIog0bp+GFBwxYw13LYXMH0xv/ebaGMSMBLTNhWQTjHRbplFsEy6lhgRH\n29tYAGn5ZVz+7FpySirp3TOAixL7MH1oFJO3vI1XTh8+vGpqo1P0CvRlYFRw43NbQvDDa+Z1oIsD\ndZQyIx5rqmDD886D5Z7CcmUd/d5YKNBlagjARSHQWt/i7oUIQrNYQhB/Jmx/1zw1tsbU3/6e6St/\nyRONP7OeyC33hH9I/c8zNkHKcvju33D5wratP8dJAzJrapZV+RvcIFXS2UyCbe/Ah/PN733Hw/kP\nQuK1zV87epR5Pb7bLgQZm2yZRi6ORQyOMX+f2hoKK2qY98oGKqtqWHL3FCbEh9kLvjblQVg/xseH\nuXZesN/A9/zPpHj2Hef6sUrBBf8wVdT9z3T9uPYmcqgJqh9dZ0RTeZvYTBfB1ayhgUqp/ymlcpRS\nx5VSy5RSXedbCl0fK5B7/oOmMGjD8607fsPzpqjIWQM2x8ElzhqgWQHqnR+0PaBclzrqIAQRg02x\nU8Fhc1NuGLQNijJpiSdP2LdtfcMcd982mL/SjFMMaSAgDQkfYK7jGCdoLtPIGcHRoGupKMpm/mub\nScsv54WbkpjYP9wuAmAft9kaAsOMANSchAHTWt8ZVCkz4rGloLc78fIy2UNp643ohw/skEljbcXV\n3Ka3gHcxw2b6Au8Bi921KEFoRGmWecrqNxFGXAKbXna9IVtFkXFr1FbbC48ccRxu7sw9lHvANHSr\nrTLXbQu5+006qGOKo5c3xNie1p315rFqCaw4wckTxvUw7ILWuUG8vI1Ly8ocqii2uYlcjA9AXZuJ\nx5auZkNqPo9ePZbJAyPq72MN13Gl/bQjSpnmbdB82mhnJ/4M8985fWOXaTZn4WqMQGmtHRubv6GU\nurfJvQWhvSnJNjcjLy/TGG33MjPg5Yy7Wz42dbWZSgXmRt/wJlqUbip9K4rM03lD8g6aG3ZQpBGC\nab9s/dNn7n7zJN8wr7z3GOOmcfYU7dhvKCwBUteYp+ZBM1t3bTDrt7pvZv4A6Ebxgc1HCvj7p3uo\nrK5pdPiIqiz+Bew9cJAHLriGH411crOvLDaTxFprEYD5flnbXY8PdEbibPUEpdkmZbUL4apFsEIp\ntUAplaCU6q+U+i3wiVIqXCnV+ZttC12f0my7Dz02yQTn1j3jWq/9Q9+agezg3PVTlAZRIyAw3LlF\nkJdibuKT7zLVtbs+bP36c/Y7Dx5acQJnN8+G1cUp34BPgImTtJbokWbtJ3LtgWKH2oOisirufesH\n0gvKiAkJaPTjbVvfLYk9mH92E17h4qZrCFpk0AxImGZaSHRV+o43NRDQpTKGwHWL4Brb650Ntt+K\nSSOVeIHgXkqz6pfrn/lTePvHsP9zGHFx88ceWmnyyI+scS4EhUdND5qak40tgqpyIxQRN5gn8cih\n8P2zkHiN6/71k2WmlYOznP3elhA4uXn2sLlerIBxynLoP7X5dhBNYaXbZu8ybrKIwcY3b+OPy3aS\nU1LJB/ecSWKsk9GWlaXwMMyI1U1/72aKyVok6Vbz05XxDYC+E0yFcReqIQDXLYIRWusBjj8O25yK\ngFIqTim1Qim1Rym1y5pfoJT6s1IqQym11fZzYXt9GeE0xnINWQyZbbI09n3W/HHFmSZ/f/C5piK1\n4Y2+tsbsExpngqoNLYL8Q4A2T6pKweQ74dhWSNvg+trzbMNonLkLYkZBn3HOK2kdLYLCNONeaotb\nCOpnDqVvqtd8bdnWDD7alsl95w5xLgJgumz6BUNpM3UNLbSX6Bb0P9Okj3ah1FFwXQiczWJrYj5b\nHdXAr7TWI4AzgJ8opawqoMe11uNsP5+6uAahI8hLsVWvdiC1NeZmGOxwg/H2MTf3g1/Z+80449C3\n5nXgOc7HIpZmmyBwr1iTSlmUbvLSLfIOmlfrJp54rclwWf+c6+u3ppI5cxf4BsKd3zqfmuUXbFxB\nJ3Lh0Aqzra1CEBxtLIz9nxsXkc0tlF5Qxh+W7mRi/zDuPqcFt0xQVPP9hlwdWn86c9bP4ZbPGqcg\nd3JamkfQG+gHBCqlxmMvFewJNFsDrrU+hmlhjda6RCm1x3YuoTOhtfnH3dQ/3i//YMTg3lY8Abc3\nJ3JMsLdhmuSQWbBziZn01XDOrsXhb80NMHqUudGnbTTf2XJvWBlDveLNTVfXGFeQlQNuCUG47Sbp\nH2xcPN8/a3rvuDLtKmefiVG01v+tlL3fUOFRCOlDUfBg1u/KorYNdf3JQYOJOLQSgLWVAyjemcXL\nqw9TW6t5/Opx+Hi38FwYHGMqsHUT7qGSLCOSfkGtX9zpQkCvpltddGJaihHMxkwiiwUec9heAvzO\n1YsopRKA8cB6YCpwr1LqJmATxmpoNANPKTUfmA8QHx/v6qWE1rLnI1hyO/x8p/N89PzD9ulPHYXl\ncmhYcDX4PECZbBhnQqC1PT7g5WUsgsoiU+FrVeJaNQShcfYbWP5huxDkHjQ+b3+HithBM011c95B\n14Qgd5+5dlvy3IMijFBnbOZ433P50X++I6u4ovXnAf7kE8otPlChfbnpkxNUYyqFH71qLPERLfT2\nATMHYMPzsOgSOO8vjauSS465t5+P4DaaFQKt9SJgkVLqCq11mzp9KaWCgSXAz7XWxUqpZ4G/YoLM\nf8UMvGkUJdJaLwQWAiQlJUlfI3eRvcsESfMONBYCrc2N8mSp84pbT2F1vQx20oKh30Q48CVM/23j\n43L3m5vTwHPM+3BbFW1BamMh6BVrnuagfhwh76AJrDrSmkEtGT+YOMboK1ve1xk9ItFH16Gqyvjb\nvj70CPfm9dsmERHUelEJ3ZsOq76gtvdYPrrUpGmGBPg07v3TFLMfMi6yb/8BL840fZPOvM+IsJdX\n60ZUCp0KV7OGRiulRjXcqLV+sLmDlFK+GBF4U2v9ge2YbIfPXwA+dn25QrtjNRIrPNr4s/ICIwJg\nM/s7SgisIKQTi2XILFj5sHGfWMFVC8f4ADjMxz1sb7VQlG6Czv4hZpCIT0D9gHHewfqN4sB1ISgv\ngPduNg3bZj/U/L42jpdU8MqaVMoqTVrs5fk+jK0qo1YrosfN4eNLp9DDr43DVWonwiroMWAyI/v2\nbP3x3r4w6Q4Yey2s/S+sfcrUc/SIgEHnmr/V0DltW5vQobj6f1Spw+8BwMXAnuYOUKbu/CVgj9b6\nMYftfWzxA4C5wE7Xlyu0O9YTsTMhcGy9UHKs44pkrLYOQU7cMENnwcq/m2lbY6+p/9mhlebmb1Wt\nhtpeHQPGhWnGLQTmqTa0v/3zsnwoz29sEfQINz7/5gKntbXw4d0mt/6Wz5pvCmfj693Z/HbJdorL\nqwgOMP80B9f6MBYoCR/JH66a1uI5miVmtBHOMVec2nn8Q2DGAyaD6uDXtp/lRvjCu3AdQDfG1aZz\njvOKUUo9CnzUxO4WU4EbgR1Kqa22bb8DrlNKjcO4hlJpXJsgeJLCZoTAsfWC5afvCEqzzVO7s/z5\n3mONQBz4or4Q1FRD6ncw2qGDun+w2bfeiMO0+pXG4QPsQmAFihsKgZe3yaBpbnTj2idh/2cw5x8Q\n1/yc3PKTNTz06W7e+P4oI/r05J35ZzAkxmZ9fbcTln9Cr1Gzmz2HS/gGwPXvnfp5LHqEQ+LV5qe2\n1rgXLbEVuhRtHeDZgxaKyLTWq3HWkBwkXbSzUFtrDwS7YhF0FKXN+J69vGDI+bD3E3Pzt2bSZm4x\nLQ8GnlN//4YppEXppqK17vMBcPg7Ex9pmDrqSHB0066hI+tg+YMw8jLz1AzU1mp2ZhbV9e4vKDtZ\nt3tBWRW5pZXMP3sgv5o1FH8fh6ZrVjC6rWmjnsLLq8v11xHsuDqYZgf2QTReQDQm0Ct0ZU7kmEAx\nyvS4b0hhmula6eXdsRZBw2KyhgyZBVvfNM2++k8xvXk+v982j/fs+vuGDzA3ajBzBiqLTaDY8fOq\nE+Zvk3fQFAeFOslaC2pmYtfGF83T8o+eYm92Ca+tO8KXu7LILTU3/8TYXgyJsWcheXt5cW1yHFMH\nRzY+14hLTF1D/8b9/QWhvXDVIrgYCAOmAaHAp1rrzc0fInR6rCf+mNGm4tTxiRpMW4TQOFtXyQ62\nCKyGXs4YNMN0Jj3wpel8+cbl5kn/qldN+qUjYQmw4z2oPlk/dbTuc1tmUf5hIwRhCSZI2pDgGPuM\ngQboonTyewzgJ6/t4vtD+fj7eDFrVG9mDIvi7KFRRAa3IuMnoBckyTgQwb24KgSXAncAH2DcPa8o\npV7QWj/ltpUJ7se6ESZMhewdxk0U5uDjLUyDXnHGaijuICHQ2rhgmuu5H9DLtGjY+T5sfcv08L9p\nmWkL3JCwBFOcVpRmz5jq5fDEX5dietjUEDSMD1gER5sYQYPiqpScUoIzDrOmaghpQeUsuGA41yTF\nERbUdXrTC90PV1tM3A6cobX+k9b6/4ApGGEQujLWjdByOzSMExTZMmpC+nScRVBRBNUVjYvJGjJ0\nllm/8oJbv3AuAlD/ib/QoYbAIjQeUKaaOj+leSGoOQkVhQBorVm84SiXPLmK8NpcxgwfzqrfzuCu\n6YNEBIROj8vzCADHJuU1OA8EC12JwjTw72kf7O0oBCfLzJzdXnHmZlyS1XRrAXdi+eEbFpM1ZNz1\nZvbwmT+t7+ppiGMtQdFRM3AmKMr+uY8/9OxHyf6VhFRXsLU8kiNbG1dW98vzJQlYvmkHpSGD+GT7\nMb7cnc2FA7zxPVbDkMHDwEv+iQhdA1eF4BVgvVLKasR+GaZGQOjKFKWbp+GesYCqLwSWtRAab4q1\nairrt2bwFJYQtDSOMSgSLvxny+cL6W2KxgpSjSusV2yjYTGFAf0IOrYJFDyyoZrvv9/a6DRTvApZ\n7AcvfvY962pL8PVW/P7CEdw2sAhepPVTugShA3G1juAxpdRK4CyMJXCL1rqDW1IKp0yRLQbg42du\nXIUOmUNFNlHoFWcyZ8BYBZ4WAquYrCXXkKsoZU8hLcmq7xYC8k+cZFVOMJcpYwD/Y/5caoIaWyO+\n+X1g8UM8cXE/TgyZTmgPP8KD/EwaK4gQCF0Kl+sItNY/AD+4cS2CpylKs/elD+1f3yIodJJRU3LM\nPuDEU5Q20XDuVLCE4EQuDDmvbrPWmt99sIPB1ZHgDfgF07//IOfusKAEAGJUEUQ5NKQrtrVi7imN\ndoWug6vBYuF0o7LUuHqsJ+LQ+AauoTSTkhnSx17M1RG1BKXZxpVjNYRrD8IGmGBwaZaxeGy8tzmd\nz3dlMTbR1sk0ogkRADPdy8u3cS1BcYaxoBzjDoLQyREh6K44xgCs1+IM+1CWwjTzVOvl7SAEHZA5\nZBWTtWeQOizBpJhCnRAcyTvBXz7axRkDw5k5xZZx1FTGEJj1BEebwjNHijMhpG/jIfWC0Ilpa4sJ\noatTl0PvYBFoW8uJsAR76iiYKVoBoR1kEWS1nDHkhOV7snno0z1U1TSeXjalphgrrPyzz3PZ8tU3\nFJVV4eWl+PfV4/D2rzRpqC0NIA92Ul1cnCnxAaHLIULQXSlqkENvWQYFR4wQFKZBwln2/TuqlqD0\neKu7nmYWlvOLd7YSGeJPcv/Gwe3wyqFwyPweEzuYZL9wUHB1Uhz9QgOBQFOQZg2WbwprYpcjxZlm\ngIsgdCFECLorjjEAsAtB4VHjHirJrB8oDundMRZBSVZ9QWqB2lrNr97dRnWt5pV5yfSPcDI2sWoY\nPASg+P1155usqYYMOLvxtoYERUGmQ2qp1kYIhl3g8noFoTMgQtBdKUq3xwDAWAbKywhBcaZxE/Vy\nFII+ZuKXJ6mqMJW7rXANvbT6MOsO5fHPKxKdiwAYV1dIX0A7FwFXCY4xMYLaGvN3LC8wsQfJGBK6\nGCIE3RWrmMzC29fcwAqPOm/GZlkEtbWeC4S6WkxmY8+xYv71xT5mj4rhqqTY5neOGga11ae2vuAY\nM+y+LB+CoxxSRyVGIHQtRAi6K4VppmWzI1YKaV0PHodmbD372m56ua4NbG8P6mYVGyHYlJrPoZwT\nTe7+4upD9Orhy8OXJ6JayjK67Flj9ZwKwbYU0RPHGwiBWARC10KEoDtSW2Nvr+BIaDwcXtU4kAz1\nU0g9JgT2YrKNqflc9dy6Znf39Va8cFOSqfBtiZ59Tn19dbOLsyFmlD1wLBaB0MUQIeiOlGSZp/te\nDZqzhcabp9q8FDN4xXE0pBVULsnyXFaMLThdGRjFgsXbiQ0L5I3bJuPr49w1FeznQ68eTmYHuIuG\nQ+yLM02cpT2roAXBA7hNCJRSccBrQG+gFliotf6PUioceAdIwMwsvlprXeCudQhOqHvibygE/QEN\nR9c17uDZEUVlpccBxTMbi0jJOcGiWyeRENlEALgjsKqHHYUguHf94T6C0AVwZ9SvGviV1noEcAbw\nE6XUSGABsFxrPQRYbnsveJKGxWQWdSmkRxqLhPWU68kU0tIsqgMjeebbVOaO78f0oZ2sbYN/iBnl\naQW1izPELSR0SdwmBFrrY7ZGdWitS4A9QD/MtLNFtt0WYVpaC57EWQwA6s/mbWgRePuaJ2APWgS6\nJJujVSEE+/vwh4tGeOy6LmO1mXC0CEQIhC6IR2xYpVQCMB5YD8RorY+BEQullIcij0IdhWmmaZp/\ncP3tPfuZIjNdUz9jyMKVorICmzXRlhTTgiMmWF1RCOWFlB/9gSOVffm/y0cS0Zo5v57EGlkJRggG\nzezY9QhCG3C7ECilgoElwM+11sUtpvXZj5sPzAeIj3dyUxLaTsMaAgtvHyMG1tD6hrTUZqIkC56a\nAHOfhzFXtm5N2bvg1YtMURaglRcVtT1Ij5zKDeM6cTpmcAzkH4KKYjhZIhaB0CVxa2WQUsoXIwJv\naq0/sG3OVkr1sX3eBzju7Fit9UKtdZLWOikqqpP5hrs6RenOn/jB7h5yJhQhvZsfYp+73xRpHWs8\n0QswlcKHvzPpq47k7INFPzL+9vnfsmLuDww9+Qbzot7hR/MfbLkmoCOxGs9JMZnQhXGbECjzr/cl\nYI/W+jGHjz4Cbrb9fjOwzF1rEJqgKM35jR4grL95bRgsBmMRnMixt6puSIFtwlnuQeef//AaLLoY\nnp4MO943Vcp5KUYEvLzh5o/4qrAP89/dx8g+vXj9tsn0CvRgOmhbCIo2lcXWdDcRAqEL4k7X0FTg\nRmCHUsp6RPwd8AjwrlLqNuAocJUb1yA0pKIIKoubFoIh55ubfWBo489CegPaBEd7OXHXWDfDvAPO\nz529A/xCzOCWJbfBqkfNWmqryblyCf/b68PDn21mZN9evHbrpM4vAmArrtNwbLt5L0IgdEHcJgRa\n69WY+cbOONdd1xVaoCDVvDqLAQCMmmt+nOFYVOZMCCyLoCDVWA3e9W/kVVl7yA8expfjXyAh6wsS\nU57Dt7KUXwX+lc8WZgAZJPUP46V5yV1DBMCeVptpG+Ed0g4Vy4LgYaTypbvx3WPg7Q/9klp/bEtF\nZZZFUFttxMBhjsCKvdlMzNzNl9VT+ONHe4B4vPgbQd7VjI7uwwNJUZwzLJqhMcGdOybQEKvdxrGt\nJr3Wp5NmNwlCM4gQdCcOfAW7l8KMPzRtETRHnUXQhBAUHIGoEZCzB3IPQOQQKqpqeOSzvXyydgsb\nA04wY9o0Nk21D4wP8vMh0M+7DV+mk2AJQXGGDKQRuiynvxBo3b7zbrsqJ8vgk19BxBCY+rO2nSMo\nytQZOKslqCo3TeLGXGmEIO8AG1Pz+f2HO9ifXcqDY2rgAMQOGQ+dtSagLQQ5lMFI11Ghi3J6T9je\n+ha8Nw9ONt26uNvw3aPGdXPx4213X3h5G5+4MyEoPGpe+4ylJiCcNevXc9Vz6ygur+bVW5K5aXCF\n+TxqeNuu3Vnx62EC4CCBYqHLcnoLQXkh7PkIXpplD5J2R47vhTVPwtgfw4Bpp3au0HjIT6m3qbSy\nmiMpuwF4dXctW8uj8CtK4efnDeGbX0/nnGHRkLMXAkI918Lak1jfSYRA6KKc3q6hKfdA1FB4/1ZY\nOAOuehUGTu/oVXkWreGTX5p2ErP+2qpD80or+cPSnaw5mFu37ff05AJWc9afPwcUWkNJZTU3eq/k\nr77wzLZqnoweRHLVBpLPG2o/Wc4+Yw2cjm664BgjjuIaEroop7cQAAw+D+5YAW//GF6fC5c+DeOu\n6+hVeY6Cw3BkDcx6CIIiXT7s2/05/OrdbRSXV3H5hH4E+JqAblDOWHqmfckto3wo8jfB46gQfy7I\n/IraQ/5886drCN6UC19/aiwyqx4hZy8Mv7jdv16nwJpUJhaB0EU5/YUAIGIQ3P41vDwH1j/bvYQg\n11bcFetaumhFVQ3//HwfL685zNCYYF6/bRIj+vS075B2Al76F78YcxKGj7JvfycXwvoTHOBrTxvN\nO2iueyIXyvJOv/iAhVVLIBaB0EU5vWMEjviHQMxoKOtmM3AsIYgY0vx+wL6sEi57eg0vrznMvDMT\n+Ojes+qLAEC0rR109q762wuO2AbbOFzLunbOXvMaNawNX6ALEJZg+iRJMZnQRekeFoFFj3Aoz+/o\nVXiWvAMQGA5BEU3uorVm0dpU/v7ZXnoG+PDKvGRmDG8iqOsfAmEDTLsIRwqPQNwk83tYgkkzzWso\nBKepRZB0GwyZbTKIBKEL0r2EIDAcTpZC9UnwcWHA+WmAzj1AQWB/lqw61OQ+3x3MZdX+HGYOj+af\nVyYS2VKef8yo+hZBeaHpYRSWYN77+Jnf6yyCfSbF8nT1ofsGQOTgjl6FILSZ7iUEPcLMa3m+vV1C\nR1BbCysegjFXQbR7n5Irju3l67LRPPTpnib3CfD14sFLR3HjGf1da+/Qewzs/cTUZ/gF2VtLWK4h\nMHGCPFsX0py9xi10OmYMCcJpQPcSgsBw81qW17FCcPBrU+B1ZC3c8ump3yA/f8C4vc7+Tb3NqRnH\nSDiZR03EYHbcPavJm7yftxd+Pq0IF8WMArSpT4idaG82F+YgBBGDIWWFmT2Qsw8Gn9/KLyUIgqfo\nPsFigB42P3lZB8cJNjwPKDi61ojCqVBdCZtegS1v1ttcVVPLf9/7DIDZ06cREuBLsL+P059WiQCY\noDvY4wRNWQQ1lZC1wwxuOV0DxYJwGtDNhMBmEbQlYFyUAR/eDZUlp7aGvBRz8z/71+bG+fVfjKuo\nrWT8ANXlpl7AQeD++81Bao/vAyA8blRTR7eN0P7gF2yPExQcgYBe9WcYWJlDez8xr6droFgQTgO6\nlxDUuYbaIAS7PoBtb8H+L05tDRtfBC9fSL4DZv7BPFXv+qDl45oidbX998wfANh8pID/rjjIBX1L\nTfaOFcRtL7y8jHsoa6d5X3ikvjUA9lqCvR+bV7EIBKHT0r1iBKdiEaStN68Hl7d+MLtFZalx4Yy6\nDEJiYPSVsOY/8M3fYOSljQa5NMXfP93DorWpALzitZRYFU28Os4Ti97hWV1FVU0tfUMDOSeiCPQA\n92RIxYyCHUtMC4uCI6aVhyNBUeDfC47vBt8ezkdfCoLQKeheFoFvoCn8aa1FoDWkbTC/p3xj3reF\n7e9AZRFMmm/ee3nBuf9n3Do/vObSKT7fmcXCVYeYNiSK26b0Jdn7ANl9ZpAb0J8LI44xb2oCd04f\nxKu3TMI3/6BLhWRtIma0+S6FR51bBErZUyojh5rvKghCp6T7/evsEd56ISg8agKefcebnvsNq2pd\nQWvY8IIZXhKbbN8+ZBbEnQHf/tPMDGiG48UVPPDBdsb068Uz10/gt6NP4KsrST7nUiKHTmFo9X4e\nmDOc++cMZ3BkIOQfcl9+uxUwTvkGqiucu58sEZL4gCB0atwmBEqpl5VSx5VSOx22/VkplaGU2mr7\nudBd12+SwDZUF6dvNK/TF5jXlOWtv27qajOwZdL8+umiSsGMB4zAHPyqycO11vz6/e2UV9Xw+DXj\nTKZPqm0sdPwU6DfBiFVxpjmg8KjJ2okc2uQ5T4mYkeZ136fmtaFFAHYRkviAIHRq3GkRvArMcbL9\nca31ONvPp268vnPaYhGkrQffINPJNGqEiRO0lg0LjQiNvqLxZ7GTAGXy7ZvgtXVHWLU/h99fNJLB\n0cFm4+FV0Hu0+U79JppttoBxXTGXu1xD/iHGCji00rwPcyIEdRaBCIEgdGbcFizWWq9SSiW46/xt\npke4yW1vDWkbzBO3tw8MPtfc1E+Wud5bRmuTMjruxyZO0YCTXgGcDOxD5s5NfFlzoNHn1bWaZ1em\nMGNYFDdMjjcbqyqMpZJ0q3kfMxq8fCBjM4y4BHL3m+2RbhIC65rWwJ/Q+MafDzkfznkABp3rvjUI\ngnDKdETW0L1KqZuATcCvtNZO24EqpeYD8wHi453cZNpKYCstgpMnjHCc9QvzftBMWPdf0+N/iIvV\nsiXHoKqsSV/5v7/axxmlUcSc2Mujafud7hMf3oN/XJlorw7O2Gx88wm2iWO+ASaTJ8NmEeQeMBPB\nejTdbO6UiRlt0kODY5wKHH5BcM4C911fEIR2wdNC8CzwV0DbXv8N3OpsR631QmAhQFJSUhvTdJzQ\nIxwqCk3rAy/vlvfP3AK6xt5Zs/+Z4BNg3EOuCkGebbRjxKBGH61LyWPhqkOc1W84IwqXceCPs5yu\ny1spvLwcYgtWfKD/FPu2fhNNSmdtrXENRQ51b3+f3raAsbP4gCAIXQaPZg1prbO11jVa61rgBWCS\nJ68PGItA15puma5gpY1amT6+gUYMWhMwzrd1/gyvLwRF5VX86t2tJEQEMWnSFFR1Bb4l6fh6ezX6\nqScCAKnfmeZvgWH2bYBcSzEAAA4CSURBVH0nmJTO/BTjGnKnWwhsPYdwHh8QBKHL4FEhUEo5Tu6Y\nC+xsal+30aOV1cVpG0zQ0zoOjM87dz8Uprl2jvwU8PaDXrH1Nv9p2U6ySyp5/Jpx+Pe2ZeHkOncN\n1aOqwqxrwNn1t1sB40MrTQZRhJtbI4cmGGvAMR1WEIQuhzvTRxcD64BhSql0pdRtwD+VUjuUUtuB\nGcAv3HX9JrF85q6kkGoN6RsgbnL97YNtwU9XrYK8FDPMxcHls2xrBku3ZnLfuUMYFxdqr8xtJnOo\njoxNJjU04az626OGmeym7e+a9+5KHbXw8oL7tsHkO917HUEQ3Io7s4acDQZ+yV3Xc5nW9BvKP2Ra\nVsc1eOKNGg4hfU2cYOK8RofV1GrufmMzX+7OBuBzv22k6WjuWPBJvf3Gx4dyzzk2d1FgGARFuyYE\nh1eB8jL1A454eZuCtaNrzXt3u4ZAZgwIwmlA9+o1BPWH07SEFR9oaBEoBYNnwu7/OZ129uJ3h/hy\ndzbXJscRHeLH4O+PU9r7bH42wH5j9vNWXJ0ch4+3g1EWNQxyWxCC7F2w7mmTLeTY7dOi3wQjBMrb\nWCGCIAgt0P2EoDUWQdp60zgt0klB1IhLYcsbJn1y9OV1m3dlFvHol/uYM6o3D18+BlWcAWsrSRo/\nkaTkFlw1kUNhx/vGJeXsSftEHiy+zrSAnvuc83P0m2Bew/p3m3GcgiCcGt2v11BAL/O07IpFkL7R\nTOBy1jBt8LkmULrxxbpNFVU1/PztrYT18OPvl48xOf/NpI42ImqYyfopzW78WU0VvHczlGTBtW82\nPf+3r00I3B0fEAThtKH7CYFSxh9fltf8fhXFxg3T0C1k4eVtqnqPrIHs3QD84/O9HDheyr+uGkt4\nkO1pPN8mBOEuCgE4jxN8/oBJGb3kPxCb1PQ5whJMtlD8GS1fTxAEge7oGgKX+g1VZe3GF80HWZEc\n+sK53z6wagp3Kj92fPBvPor9Ja+sSWXemQlMHxpl3ykvxRSg9ezX8roiHYRg4HT79p1LYOMLMOVe\nGOcsBu+AUvCTjdL2WRAEl+mmQhAB5U47W9Tx+XfruAR4drvmEClN7tfHezKzsj5mSdqFJPXvw4IL\nGrSRyD9kSx114cYc0hv8ezYOGH//nBGJ8x9s+RwgIiAIQqvonkIQGG5vluaE9YfyOLBvJ7U+iq/+\ncqPp49MU6dHw4rlsn1sAyU46i+aluJ7GqZTx7Tu6hvJSTC3DeX9xrSWGIAhCK+mej449wpoMFhdX\nVPHLd7cxIiAPQvo0LwJgqnn7jIWNLzWeXFZbY6aPhQ90fW1Rw+pXF29bbGoGEq9x/RyCIAitoHsK\ngdWB1MnIyT8v20VWcQXTIkrxCnchD18pSL7dzOY9uq7+Z0XpUHPStYwhi6hhJmuovNA0j9v2Ngyc\nAT37tHysIAhCG+ieQtAjHGoqqa0spaKqpu5n2dYMPtiSwb0zBhNclu56QdboK029wYYX6m9votlc\ns1gB49z9cGQ1FKXB2BYCxIIgCKdA940RALc/9xXfZPnX+2hcXCj3TusHa445n8PrDL8eZujMxhfh\nRC4ERZrt+a2oIbCo6zm0F45+D34hMPwi148XBEFoJd1TCGyN544fz+QnM84jyN/8GXy9vLhsfD98\niw+b/VwVAjA9h9Y/C1vfgqk/M9vyDoFPIAT3dv08of3B29/MQdi9DEbNdX0SmiAIQhvolkKwOQcm\nAtcnBnPdbCdTwzJTzasrMQKL6OEQdwb8sAjO/KmJHeSnmEBxa9I5vbxNltHWt8wEMnELCYLgZrpd\njCCnpJKHVx4H4MoRTTxpW6mlrbEIACbebCaDHVlj3uelQEQrMoYsIocaEQjt37jDqCAIQjvTrYRA\na82CJdvJPGnm6/pWNjGlrOCwaezW2nm/Iy8zQePNi6Cm2ghKawLFFtZs47HXSnGYIAhup1vdZd7a\ncJTle49z52zbJK+m+g0VpBproLW99v16QOLVxreftR1qq1oXKLbof6YRlHE/bv2xgiAIraTbCEFK\nTil//Xg304ZEcuPUIaaVQ1P9hiwhaAsTbzbTw1Y+bN63xSIYMA0WHGn7GgRBEFpBtxCCqppafvHO\nVgJ8vXn0qrFmEHxgE9XFtbWnJgS9x5hq4wNfmvdtsQhAJn8JguAx3Dmz+GWl1HGl1E6HbeFKqa+U\nUgdsr2Huur4jTy4/wPb0Ih65fAwxPW0tI3pEOLcISrNNoPZUnsYn3Gxe/YIhOKbt5xEEQfAA7rQI\nXgXmNNi2AFiutR4CLLe9dyubj+Tz9IqDXDUxljmjHdo09Ah3bhFYGUOtSR1tyOgrjAiED5Ane0EQ\nOj3uHF6/SimV0GDzpcA5tt8XASuB+921htLKan7+zlb6hQXypx+Nqv9hYDjkHmh8UIFVTHYKQuAf\nDBf+C3z8W95XEAShg/F0QVmM1voYgNb6mFIq2p0Xe+iT3WQUlPPunVMI9m/wVZsaTlOQarp99oo7\ntYtLxo8gCF2ETltZrJSaD8wHiI+Pb9M5bpk6gNH9epGUEN74w8BwOFkC1SfrD3kvSIWesTL4XRCE\nboOns4aylVJ9AGyvx5vaUWu9UGudpLVOioqKamq3ZhkaE8L1k/s7/7CHTRwaTirLPwxhTRwjCIJw\nGuJpIfgIsKXUcDOwzMPXt1MnBA3cQ6eSOioIgtAFcWf66GJgHTBMKZWulLoNeAQ4Xyl1ADjf9r5j\nsLWirhcnOHkCThw/tYwhQRCELoY7s4aaapt5rruu2SqcWQQFR8yrWASCIHQjukVlsVOcWQQFbZhD\nIAiC0MXpvkJgWQSOjefq2k+La0gQhO5D9xUC3x5mElg911Cq6foZ6JHOF4IgCJ2C7isESkFIb9i1\nFA58bbZZqaPSFkIQhG5E9xUCgMueBW8/ePMKWPxjOL5H4gOCIHQ7urcQJEyFe9bBuX+CQyugOF1S\nRwVB6HZ02hYTHsPHH6b9EhKvgQ3Pw7jrO3pFgiAIHkWEwKJXPzj/wY5ehSAIgsfp3q4hQRAEQYRA\nEAShuyNCIAiC0M0RIRAEQejmiBAIgiB0c0QIBEEQujkiBIIgCN0cEQJBEIRujtJad/QaWkQplQMc\naePhkUBuOy6nPemsa+us64LOu7bOui7ovGvrrOuCzru21q6rv9a6xaHvXUIITgWl1CatdVJHr8MZ\nnXVtnXVd0HnX1lnXBZ13bZ11XdB51+audYlrSBAEoZsjQiAIgtDN6Q5CsLCjF9AMnXVtnXVd0HnX\n1lnXBZ13bZ11XdB51+aWdZ32MQJBEAShebqDRSAIgiA0w2ktBEqpOUqpfUqpg0qpBR28lpeVUseV\nUjsdtoUrpb5SSh2wvYZ1wLrilFIrlFJ7lFK7lFL3dYa1KaUClFIblFLbbOv6i237AKXUetu63lFK\n+XlyXQ7r81ZKbVFKfdzJ1pWqlNqhlNqqlNpk29bh/5/Z1hGqlHpfKbXX9v/blI5em1JqmO1vZf0U\nK6V+3tHrcljfL2z//+9USi22/bto9//XTlshUEp5A08DFwAjgeuUUiM7cEmvAv/f3t2EaFWFARz/\nPWRJ2ofZF+YEJkgJkaOFWUaUfYmEbVoYLVwIbYQKgkiCoGUQZas2RYsIg8pKXPSB1aaFlWZhqX2g\n6KQ5EllQEFpPi3tefJlGCGrmXuY9f7jce85cmD/nPPd95jznvszKMX2PYVtmLsC20p5sTuKRzFyI\nZVhfxqlttz+wIjMXYRgrI2IZnsKzxetnrJtkrx4PYU9fuytecGtmDve9Ztj2XPZ4Du9k5lVYpBm/\nVt0yc18Zq2Fci9/xZtteEBFz8SCuy8yrcQbWmIhYy8wpeeAGvNvX3oANLTvNw+6+9j7MKddzsK8D\n4/Y27uiSG2ZgJ67XfJlm2nhzPIk+Q5oPhxXYiuiCV/ndB3DRmL7W5xLnYb+yL9kltz6XO/FxV7ww\nF4cwW/PfJLfiromItSm7InBqEHuMlL4ucWlmHoFyvqRNmYiYh8XYrgNupfyyC6N4H9/jeGaeLLe0\nNacb8Sj+Ku0LO+IFifciYkdEPFD6Wp9LzMcxvFRKai9ExMyOuPVYg03lunWvzPwBT+MgjuAX7DAB\nsTaVE0GM01dfkToNEXEO3sDDmflr2z6QmX9ms2QfwlIsHO+2yXSKiLsxmpk7+rvHubWtWFuemUs0\nJdH1EXFzSx5jmYYleD4zF+M37ZWo/kGps6/Ga2279Cj7EvfgClyGmZp5Hct/jrWpnAhGcHlfewiH\nW3I5HUcjYg6U82gbEhFxpiYJvJKZm7vkBpl5HB9p9jBmRcS08qM25nQ5VkfEAbyqKQ9t7IAXyMzD\n5TyqqXUv1Y25HMFIZm4v7dc1iaELbjQfsDsz82hpd8HrduzPzGOZeQKbcaMJiLWpnAg+xYKyw36W\nZtm3pWWnsWzB2nK9VlOfn1QiIvAi9mTmM11xi4iLI2JWuT5b81DswYe4ty2vzNyQmUOZOU8TUx9k\n5v1te0FEzIyIc3vXmpr3bh2Is8z8EYci4srSdRu+7oJb4T6nykJ0w+sglkXEjPKc9sbs/4+1tjZm\nJmmzZRW+0dSWH2/ZZZOmzndC89fROk1teRu+LefZLXjdpFlafold5VjVthuuwefFazeeKP3z8Qm+\n0yzjp7c4p7dga1e8isMX5fiqF/Ntz2Wf3zA+K3P6Fi7ogpvmZYSfcH5fX+texeNJ7C3PwMuYPhGx\nVr9ZXKlUKgPOVC4NVSqVSuVfUBNBpVKpDDg1EVQqlcqAUxNBpVKpDDg1EVQqlcqAUxNBpVKpDDg1\nEVQqlcqAUxNBpVKpDDh/AyaV5WF40nNDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1f3e4cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "import tensorflow as tf\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "from keras.callbacks import EarlyStopping\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "\n",
    "path = \"./data/\"\n",
    "preprocess = False\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# create feature vector\n",
    "missing_median(df, 'horsepower')\n",
    "encode_text_dummy(df, 'origin')\n",
    "df.drop('name',1,inplace=True)\n",
    "if preprocess:\n",
    "    encode_numeric_zscore(df, 'horsepower')\n",
    "    encode_numeric_zscore(df, 'weight')\n",
    "    encode_numeric_zscore(df, 'cylinders')\n",
    "    encode_numeric_zscore(df, 'displacement')\n",
    "    encode_numeric_zscore(df, 'acceleration')\n",
    "\n",
    "# Encode to a 2D matrix for training\n",
    "x,y = to_xy(df,'mpg')\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    x, y, test_size=0.20, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(1))\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto')\n",
    "checkpointer = ModelCheckpoint(filepath=\"best_weights.hdf5\", verbose=0, save_best_only=True) # save best model\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor,checkpointer],verbose=0,epochs=1000)\n",
    "model.load_weights('best_weights.hdf5') # load weights from best model\n",
    "\n",
    "# Predict and measure RMSE\n",
    "pred = model.predict(x_test)\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "print(\"Score (RMSE): {}\".format(score))\n",
    "\n",
    "# Plot the chart\n",
    "chart_regression(pred.flatten(),y_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Important Parameters\n",
    "\n",
    "* **learning_rate** - How quickly should the optimizer attempt to train the neural network.  Too high will fail to train.  Too low will train too slowly.\n",
    "* **momentum** - Only used with the momentum optimizer.  How much of the previous weight change direction should be used in the current step.\n",
    "* **every_n_steps** - How often should the validation set be evaluated.\n",
    "* **early_stopping_rounds** - How many rounds (steps) of non-improvement should be tolerated.\n",
    "\n",
    "# Some results\n",
    "* **ADAM**, learning_rate = 1e-1, early_stopping = 100\n",
    "    * Score (RMSE): 3.5879969596862793\n",
    "    * Best step: 1505, Last successful step: 1640\n",
    "* **ADAM**, learning_rate = 1e-3, early_stopping = 100\n",
    "    * Score (RMSE): 2.4205124378204346\n",
    "    * Best step: 4650, Last successful step: 4770\n",
    "* **ADAM**, default learning rate, early_stopping = 1000\n",
    "    * Score (RMSE): 2.410527467727661\n",
    "    * Best step: 6630, Last successful step: 7720\n",
    "* **Momentum**, learning_rate=1e-5,momentum=0.9, early_stopping = 1000\n",
    "    * Score (RMSE): 2.4050464630126953\n",
    "    * Best step: 41270, Last successful step: 42280\n",
    "\n",
    "ADAM will converge quicker (fewer steps) than momentum and get a good RMSE.  For all, it is important to allow enough early_stopping steps to find a good solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Module 6 Assignment\n",
    "\n",
    "You can find the first assignmeht here: [assignment 6](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class1.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}