proj2B/proj2_part4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you turn in the homework, make sure everything runs as expected. To do so, select **Kernel**$\\rightarrow$**Restart & Run All** in the toolbar above.  Remember to submit both on **DataHub** and **Gradescope**.\n",
    "\n",
    "Please fill in your name and include a list of your collaborators below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"Xiong Qiao\"\n",
    "COLLABORATORS = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "60616857f10e17fe3a3659b334a00c81",
     "grade": false,
     "grade_id": "cell-776bdcdaa31aa0dd",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Project 2: NYC Taxi Rides\n",
    "# Part 4:  Feature Engineering and Model Fitting\n",
    "\n",
    "In this final part of the project, you will finally build a regression model that attempts to predict the duration of a taxi ride from all other available information.\n",
    "\n",
    "You will build this model using a processing pipeline and submit your results to Kaggle. We will first walk you through a generic example using the data we saved from Part 1. Please carefully follow these steps as you will need to repeat this for your final model. After, we give you free reign and let you decide how you want to define your final model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-20T22:53:05.666930Z",
     "start_time": "2018-03-20T22:53:03.475900Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5c9f6b0f41bd824fb5842cbcc5026fc3",
     "grade": false,
     "grade_id": "cell-91e2a5cbad6e67c3",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sklearn.linear_model as lm\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from pathlib import Path\n",
    "from sqlalchemy import create_engine\n",
    "from sklearn.model_selection import cross_val_score, train_test_split, GridSearchCV\n",
    "\n",
    "sns.set(style=\"whitegrid\", palette=\"muted\")\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (12, 9)\n",
    "plt.rcParams['font.size'] = 12\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "dbd23a86e04a0d95febfc23f65e93a23",
     "grade": false,
     "grade_id": "cell-f4b545c2be61fe3c",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Training and Validation\n",
    "\n",
    "The following code loads the training and validation data from part 1 into a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fb8ffaea62c10a43b26ec8116c8c9983",
     "grade": false,
     "grade_id": "cell-ec99abaff54cdd7c",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Run this cell to load the data. \n",
    "data_file = Path(\"./\", \"cleaned_data.hdf\")\n",
    "train_df = pd.read_hdf(data_file, \"train\")\n",
    "val_df = pd.read_hdf(data_file, \"val\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "6a6fd19f3ad15b924d20ac1ece2b355f",
     "grade": false,
     "grade_id": "cell-28aff643fb4e8794",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Testing\n",
    "Here we load our testing data on which we will evaluate your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cc32dd0cfdf4a2d4d6708353e06becf3",
     "grade": false,
     "grade_id": "cell-44d123e5bf4c04a4",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>record_id</th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10000</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-01-02 01:45:37</td>\n",
       "      <td>1</td>\n",
       "      <td>1.20</td>\n",
       "      <td>-73.982224</td>\n",
       "      <td>40.768620</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.983765</td>\n",
       "      <td>40.779598</td>\n",
       "      <td>1</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>9.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19000</td>\n",
       "      <td>2</td>\n",
       "      <td>2016-01-02 03:05:16</td>\n",
       "      <td>1</td>\n",
       "      <td>10.90</td>\n",
       "      <td>-73.999977</td>\n",
       "      <td>40.738121</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.888657</td>\n",
       "      <td>40.824364</td>\n",
       "      <td>1</td>\n",
       "      <td>31.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>6.56</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>39.36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21000</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-01-02 03:24:36</td>\n",
       "      <td>1</td>\n",
       "      <td>1.80</td>\n",
       "      <td>-73.986618</td>\n",
       "      <td>40.747379</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.978508</td>\n",
       "      <td>40.729622</td>\n",
       "      <td>1</td>\n",
       "      <td>8.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.96</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>11.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23000</td>\n",
       "      <td>2</td>\n",
       "      <td>2016-01-02 03:47:38</td>\n",
       "      <td>1</td>\n",
       "      <td>5.95</td>\n",
       "      <td>-74.002922</td>\n",
       "      <td>40.744572</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.942413</td>\n",
       "      <td>40.786419</td>\n",
       "      <td>1</td>\n",
       "      <td>20.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>4.36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>26.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27000</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-01-02 04:36:44</td>\n",
       "      <td>1</td>\n",
       "      <td>1.60</td>\n",
       "      <td>-73.986366</td>\n",
       "      <td>40.759464</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.963081</td>\n",
       "      <td>40.760353</td>\n",
       "      <td>2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>9.30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   record_id  VendorID tpep_pickup_datetime  passenger_count  trip_distance  \\\n",
       "0      10000         1  2016-01-02 01:45:37                1           1.20   \n",
       "1      19000         2  2016-01-02 03:05:16                1          10.90   \n",
       "2      21000         1  2016-01-02 03:24:36                1           1.80   \n",
       "3      23000         2  2016-01-02 03:47:38                1           5.95   \n",
       "4      27000         1  2016-01-02 04:36:44                1           1.60   \n",
       "\n",
       "   pickup_longitude  pickup_latitude  RatecodeID store_and_fwd_flag  \\\n",
       "0        -73.982224        40.768620           1                  N   \n",
       "1        -73.999977        40.738121           1                  N   \n",
       "2        -73.986618        40.747379           1                  N   \n",
       "3        -74.002922        40.744572           1                  N   \n",
       "4        -73.986366        40.759464           1                  N   \n",
       "\n",
       "   dropoff_longitude  dropoff_latitude  payment_type  fare_amount  extra  \\\n",
       "0         -73.983765         40.779598             1          6.0    0.5   \n",
       "1         -73.888657         40.824364             1         31.5    0.5   \n",
       "2         -73.978508         40.729622             1          8.5    0.5   \n",
       "3         -73.942413         40.786419             1         20.5    0.5   \n",
       "4         -73.963081         40.760353             2          8.0    0.5   \n",
       "\n",
       "   mta_tax  tip_amount  tolls_amount  improvement_surcharge  total_amount  \n",
       "0      0.5        2.00           0.0                    0.3          9.30  \n",
       "1      0.5        6.56           0.0                    0.3         39.36  \n",
       "2      0.5        1.96           0.0                    0.3         11.76  \n",
       "3      0.5        4.36           0.0                    0.3         26.16  \n",
       "4      0.5        0.00           0.0                    0.3          9.30  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.read_csv(\"./proj2_test_data.csv\")\n",
    "test_df['tpep_pickup_datetime'] = pd.to_datetime(test_df['tpep_pickup_datetime'])\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a3dfe22f0f6736f67aea35e8f97fa73b",
     "grade": false,
     "grade_id": "cell-46e155fe7ae16e6c",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>record_id</th>\n",
       "      <th>VendorID</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.377400e+04</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "      <td>13774.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.465950e+07</td>\n",
       "      <td>1.536082</td>\n",
       "      <td>1.663642</td>\n",
       "      <td>2.954688</td>\n",
       "      <td>-72.953619</td>\n",
       "      <td>40.187999</td>\n",
       "      <td>1.043778</td>\n",
       "      <td>-73.055577</td>\n",
       "      <td>40.245056</td>\n",
       "      <td>1.340061</td>\n",
       "      <td>12.836930</td>\n",
       "      <td>0.333091</td>\n",
       "      <td>0.497985</td>\n",
       "      <td>1.805420</td>\n",
       "      <td>0.304031</td>\n",
       "      <td>0.299652</td>\n",
       "      <td>16.077391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.015133e+07</td>\n",
       "      <td>0.498714</td>\n",
       "      <td>1.311739</td>\n",
       "      <td>3.704427</td>\n",
       "      <td>8.628431</td>\n",
       "      <td>4.753186</td>\n",
       "      <td>0.877637</td>\n",
       "      <td>8.191366</td>\n",
       "      <td>4.512564</td>\n",
       "      <td>0.490019</td>\n",
       "      <td>10.707619</td>\n",
       "      <td>0.429590</td>\n",
       "      <td>0.036632</td>\n",
       "      <td>2.416784</td>\n",
       "      <td>1.363097</td>\n",
       "      <td>0.013522</td>\n",
       "      <td>13.206231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000e+04</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-77.039436</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-77.039436</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-93.300000</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.300000</td>\n",
       "      <td>-93.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.719975e+07</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-73.992058</td>\n",
       "      <td>40.735166</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-73.991318</td>\n",
       "      <td>40.734002</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>8.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.457400e+07</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.700000</td>\n",
       "      <td>-73.981846</td>\n",
       "      <td>40.752432</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-73.979897</td>\n",
       "      <td>40.753263</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>9.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.350000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>11.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.216875e+07</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.157500</td>\n",
       "      <td>-73.967119</td>\n",
       "      <td>40.767264</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-73.962749</td>\n",
       "      <td>40.768455</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>14.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>2.360000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>17.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.940400e+07</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>104.800000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>40.868210</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>41.540859</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>156.040000</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>1.740000</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>187.590000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          record_id      VendorID  passenger_count  trip_distance  \\\n",
       "count  1.377400e+04  13774.000000     13774.000000   13774.000000   \n",
       "mean   3.465950e+07      1.536082         1.663642       2.954688   \n",
       "std    2.015133e+07      0.498714         1.311739       3.704427   \n",
       "min    1.000000e+04      1.000000         0.000000       0.000000   \n",
       "25%    1.719975e+07      1.000000         1.000000       1.000000   \n",
       "50%    3.457400e+07      2.000000         1.000000       1.700000   \n",
       "75%    5.216875e+07      2.000000         2.000000       3.157500   \n",
       "max    6.940400e+07      2.000000         6.000000     104.800000   \n",
       "\n",
       "       pickup_longitude  pickup_latitude    RatecodeID  dropoff_longitude  \\\n",
       "count      13774.000000     13774.000000  13774.000000       13774.000000   \n",
       "mean         -72.953619        40.187999      1.043778         -73.055577   \n",
       "std            8.628431         4.753186      0.877637           8.191366   \n",
       "min          -77.039436         0.000000      1.000000         -77.039436   \n",
       "25%          -73.992058        40.735166      1.000000         -73.991318   \n",
       "50%          -73.981846        40.752432      1.000000         -73.979897   \n",
       "75%          -73.967119        40.767264      1.000000         -73.962749   \n",
       "max            0.000000        40.868210     99.000000           0.000000   \n",
       "\n",
       "       dropoff_latitude  payment_type   fare_amount         extra  \\\n",
       "count      13774.000000  13774.000000  13774.000000  13774.000000   \n",
       "mean          40.245056      1.340061     12.836930      0.333091   \n",
       "std            4.512564      0.490019     10.707619      0.429590   \n",
       "min            0.000000      1.000000    -93.300000     -0.500000   \n",
       "25%           40.734002      1.000000      6.500000      0.000000   \n",
       "50%           40.753263      1.000000      9.500000      0.000000   \n",
       "75%           40.768455      2.000000     14.500000      0.500000   \n",
       "max           41.540859      4.000000    156.040000      4.500000   \n",
       "\n",
       "            mta_tax    tip_amount  tolls_amount  improvement_surcharge  \\\n",
       "count  13774.000000  13774.000000  13774.000000           13774.000000   \n",
       "mean       0.497985      1.805420      0.304031               0.299652   \n",
       "std        0.036632      2.416784      1.363097               0.013522   \n",
       "min       -0.500000      0.000000      0.000000              -0.300000   \n",
       "25%        0.500000      0.000000      0.000000               0.300000   \n",
       "50%        0.500000      1.350000      0.000000               0.300000   \n",
       "75%        0.500000      2.360000      0.000000               0.300000   \n",
       "max        1.740000     40.000000     20.000000               0.300000   \n",
       "\n",
       "       total_amount  \n",
       "count  13774.000000  \n",
       "mean      16.077391  \n",
       "std       13.206231  \n",
       "min      -93.600000  \n",
       "25%        8.750000  \n",
       "50%       11.800000  \n",
       "75%       17.800000  \n",
       "max      187.590000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2357c24398c4ef5cc3899fd598cc5fc6",
     "grade": false,
     "grade_id": "cell-0d441ceef9b7f4e0",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Modeling\n",
    "\n",
    "We've finally gotten to a point where we can specify a simple model. Remember that we will be fitting our model on the training set we created in part 1. We will use our validation set to evaluate how well our model might perform on future data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "712b62308e2058b1acc48ac3e5ce33e0",
     "grade": false,
     "grade_id": "cell-3725d1930850468b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Reusable Pipeline\n",
    "\n",
    "Throughout this assignment, you should notice that your data flows through a single processing pipeline several times.  From a software engineering perspective, this should be sufficient motivation to abstract parts of our code into reusable functions/methods.  We will now encapsulate our entire pipeline into a single function `process_data_gm`.  gm is shorthand for \"guided model\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d5c20c7e375489c94431f21fd5416ddf",
     "grade": false,
     "grade_id": "cell-0627b259f25d7b70",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Copied from part 2\n",
    "def haversine(lat1, lng1, lat2, lng2):\n",
    "    \"\"\"\n",
    "    Compute haversine distance\n",
    "    \"\"\"\n",
    "    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))\n",
    "    average_earth_radius = 6371\n",
    "    lat = lat2 - lat1\n",
    "    lng = lng2 - lng1\n",
    "    d = np.sin(lat * 0.5) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin(lng * 0.5) ** 2\n",
    "    h = 2 * average_earth_radius * np.arcsin(np.sqrt(d))\n",
    "    return h\n",
    "\n",
    "# Copied from part 2\n",
    "def manhattan_distance(lat1, lng1, lat2, lng2):\n",
    "    \"\"\"\n",
    "    Compute Manhattan distance\n",
    "    \"\"\"\n",
    "    a = haversine(lat1, lng1, lat1, lng2)\n",
    "    b = haversine(lat1, lng1, lat2, lng1)\n",
    "    return a + b\n",
    "\n",
    "# Copied from part 2\n",
    "def bearing(lat1, lng1, lat2, lng2):\n",
    "    \"\"\"\n",
    "    Compute the bearing, or angle, from (lat1, lng1) to (lat2, lng2).\n",
    "    A bearing of 0 refers to a NORTH orientation.\n",
    "    \"\"\"\n",
    "    lng_delta_rad = np.radians(lng2 - lng1)\n",
    "    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))\n",
    "    y = np.sin(lng_delta_rad) * np.cos(lat2)\n",
    "    x = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(lng_delta_rad)\n",
    "    return np.degrees(np.arctan2(y, x))\n",
    "\n",
    "# Copied from part 2\n",
    "def add_time_columns(df):\n",
    "    \"\"\"\n",
    "    Add temporal features to df\n",
    "    \"\"\"\n",
    "    df.is_copy = False # propogate write to original dataframe\n",
    "    df.loc[:, 'month'] = df['tpep_pickup_datetime'].dt.month\n",
    "    df.loc[:, 'week_of_year'] = df['tpep_pickup_datetime'].dt.weekofyear\n",
    "    df.loc[:, 'day_of_month'] = df['tpep_pickup_datetime'].dt.day\n",
    "    df.loc[:, 'day_of_week'] = df['tpep_pickup_datetime'].dt.dayofweek\n",
    "    df.loc[:, 'hour'] = df['tpep_pickup_datetime'].dt.hour\n",
    "    df.loc[:, 'week_hour'] = df['tpep_pickup_datetime'].dt.weekday * 24 + df['hour']\n",
    "    return df\n",
    "\n",
    "# Copied from part 2\n",
    "def add_distance_columns(df):\n",
    "    \"\"\"\n",
    "    Add distance features to df\n",
    "    \"\"\"\n",
    "    df.is_copy = False # propogate write to original dataframe\n",
    "    df.loc[:, 'manhattan'] = manhattan_distance(lat1=df['pickup_latitude'],\n",
    "                                                lng1=df['pickup_longitude'],\n",
    "                                                lat2=df['dropoff_latitude'],\n",
    "                                                lng2=df['dropoff_longitude'])\n",
    "\n",
    "    df.loc[:, 'bearing'] = bearing(lat1=df['pickup_latitude'],\n",
    "                                   lng1=df['pickup_longitude'],\n",
    "                                   lat2=df['dropoff_latitude'],\n",
    "                                   lng2=df['dropoff_longitude'])\n",
    "    df.loc[:, 'haversine'] = haversine(lat1=df['pickup_latitude'],\n",
    "                                   lng1=df['pickup_longitude'],\n",
    "                                   lat2=df['dropoff_latitude'],\n",
    "                                   lng2=df['dropoff_longitude'])\n",
    "    return df\n",
    "\n",
    "def select_columns(data, *columns):\n",
    "    return data.loc[:, columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5ede4e2e0cbdd1affacb0273089e4fe3",
     "grade": false,
     "grade_id": "cell-507f8020b5b9a835",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def process_data_gm1(data, test=False):\n",
    "    X = (\n",
    "        data\n",
    "        \n",
    "        # Transform data\n",
    "        .pipe(add_time_columns)\n",
    "        .pipe(add_distance_columns)\n",
    "        \n",
    "        .pipe(select_columns,        \n",
    "              'pickup_longitude',  \n",
    "              'pickup_latitude',   \n",
    "              'dropoff_longitude', \n",
    "              'dropoff_latitude',\n",
    "              'manhattan',\n",
    "             )\n",
    "    )\n",
    "    if test:\n",
    "        y = None\n",
    "    else:\n",
    "        y = data['duration']\n",
    "        \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e1d8e5da0b5acf865ee193e1399c0ef1",
     "grade": false,
     "grade_id": "cell-4a3e4e0d97a10e09",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "We will use our pipeline defined above to pre-process our training and test data in exactly the same way. Our functions make this relatively easy to do!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "328e13b72e8702b7e4c81747075b8c67",
     "grade": false,
     "grade_id": "cell-a9a66cd4aafc3d03",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4388: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  object.__getattribute__(self, name)\n",
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4389: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  return object.__setattr__(self, name, value)\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "X_train, y_train = process_data_gm1(train_df)\n",
    "X_val, y_val = process_data_gm1(val_df)\n",
    "guided_model_1 = lm.LinearRegression(fit_intercept=True)\n",
    "guided_model_1.fit(X_train, y_train)\n",
    "\n",
    "# Predict\n",
    "y_train_pred = guided_model_1.predict(X_train)\n",
    "y_val_pred = guided_model_1.predict(X_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "3d0851ad6ebfe354bc264fccdd8e4e9c",
     "grade": false,
     "grade_id": "cell-8ce124a1c3d16212",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "Here, `y_val` are the correct durations for each ride, and `y_val_pred` are the predicted durations based on the 7 features above (`vendorID`, `passenger_count`, `pickup_longitude`, `pickup_latitude`, `dropoff_longitude`, `dropoff_latitude`, `manhattan`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "59ab7ee7adf0293484323272fb8e17b1",
     "grade": false,
     "grade_id": "cell-b31246d46b057685",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert 600 <= np.median(y_train_pred) <= 700\n",
    "assert 600 <= np.median(y_val_pred) <= 700"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0ebbb5c8e162267992b70f5ef9062c8f",
     "grade": false,
     "grade_id": "cell-5ce43378a0326e03",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "The resulting model really is a linear model just like we saw in class, i.e. the predictions are simply generated by the product $\\Phi\\theta$. For example, the line of code below generates a prediction for $x_1$ by computing $\\phi_1^T\\theta$. Here `guided_model_1.coef_` is $\\theta$ and `X_train.iloc[0, :]` is $\\phi_1$.\n",
    "\n",
    "Note that unlike in class, here the dummy intercept term is not included in $\\Phi$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2fcabc64c7168c803a07cb4cf666932c",
     "grade": false,
     "grade_id": "cell-a486bf8f36567ef8",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "558.751330511368"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.iloc[0, :].dot(guided_model_1.coef_) + guided_model_1.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a97501529bf35475094c5e3fbc949ab6",
     "grade": false,
     "grade_id": "cell-49c8840a2ad44c8d",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "We see that this prediction is exactly the same (except for possible floating point error) as generated by the `predict` function, which simply computes the product $\\Phi\\theta$, yielding predictions for every input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "51bd36405dbe260228829a3f7c5b7eb1",
     "grade": false,
     "grade_id": "cell-8de12c4814aefa25",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "558.75133051135344"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_pred[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "374f92feb628336ad31eba0c2b96892f",
     "grade": false,
     "grade_id": "cell-fd7b951877e62489",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "In this assignment, we will use Mean Absolute Error (MAE), a.k.a. mean L1 loss, to measure the quality of our models.  As a reminder, this quantity is defined as:\n",
    "\n",
    "$$\n",
    "MAE =\\frac{1}{n}\\sum_{i}  | y_i -\\hat{y_i}|\n",
    "$$\n",
    "\n",
    "Why may we want to use the MAE as a metric, as opposed to Mean Squared Error (MSE)? Using our domain knowledge that most rides are short in duration (median is roughly 600 seconds), we know that MSE is susceptible to outliers. Given that some of the outliers in our dataset are quite extreme, it is probably better to optimize for the majority of rides rather than for the outliers. You may want to remove some of these outliers later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "16c1cda3d31d8c0545ba5620a9b335f1",
     "grade": false,
     "grade_id": "cell-a710fd055e77566a",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def mae(actual, predicted):\n",
    "    \"\"\"\n",
    "    Calculates MAE from actual and predicted values\n",
    "    Input:\n",
    "      actual (1D array-like): vector of actual values\n",
    "      predicted (1D array-like): vector of predicted/fitted values\n",
    "    Output:\n",
    "      a float, the MAE\n",
    "    \"\"\"\n",
    "    \n",
    "    mae = np.mean(np.abs(actual - predicted))\n",
    "    return mae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "214be12071732b41221bb4c61bcddf23",
     "grade": false,
     "grade_id": "cell-a00a472b680ffd30",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Error:  266.136130855\n"
     ]
    }
   ],
   "source": [
    "assert 200 <= mae(y_val_pred, y_val) <= 300\n",
    "print(\"Validation Error: \", mae(y_val_pred, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fb5134f9979778939d8b7e9a82bb041f",
     "grade": false,
     "grade_id": "cell-9b97bc51faaf9d24",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "Side note: scikit-learn also has tools to compute mean absolute error (`sklearn.metrics.mean_absolute_error`). In fact, most metrics that we have discussed in this class can be found as part of the [`sklearn.metrics` module](https://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics). Some of these may come in handy as part of your feature engineering!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "aa1513d6f59c9048988eef956733ff01",
     "grade": false,
     "grade_id": "cell-02af4613f224d0b8",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Visualizing Error\n",
    "You should be getting between 200 and 300 MAE, which means your model was off by roughly 3-5 minutes on trips of average length 12 minutes. This is fairly decent performance given that our basic model uses only using the pickup/dropoff latitude and manhattan distance of the trip. 3-5 minutes may seem like a lot for a trip of 12 minutes, but keep in mind that this is the *average* error. This metric is susceptible to extreme outliers, which exist in our dataset. \n",
    "\n",
    "Now we will visualize the residual for the validation set. We will plot the following:\n",
    "\n",
    "1. Distribution of residuals\n",
    "2. Average residual grouping by ride duration\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "77fbfc262742ead4fe263e61455005e0",
     "grade": false,
     "grade_id": "cell-f6e3b30f7693966b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgIAAAEcCAYAAAC1cdy6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmcXHWd7/9Xrb13eklnT4clyZclCBIRQVB/qOM2jIyigCg6i3dQ7/XnODzGx/X3k2GYn17u6L0zojgwMHdEUETFYVxQHBx3UaBZJGA+BELSnb3T6aT36q7l98c51Sk6Xd3V3bXX+/l45JHq8/2ec77frlTOp75rIJVKISIiIrUpWOoCiIiISOkoEBAREalhCgRERERqmAIBERGRGqZAQEREpIYpEBAREalhCgRE8sg592Xn3P/nv77YOWd5vPYPnHPv919/wDn3yzxe+2rn3I/ydb0F3PfVzrkdzrkR59xlBbxPt3+PUJb0G5xzd+fpXinn3MZ8XEukGMKlLoBItTKzXwBuvnzOuRuAjWb23nmu95Z8lMs5dxLwIhAxs7h/7a8CX83H9RfoRuCLZvb5Qt7EzHqB5kLeQ6RSqUVApMw55wLOuWr9rG4Ansklo3NOX1xECkAfLJElcM69HPgXYBPwAJDKSHsdcLeZrfN//gTwUaAV2Ad8GIgAnwQCftP4C2Z2tnPup8CvgNcB5wJnOefu8K93h3+LgHPuC8A1wH7gI2b2Y/9eu4A/N7OH/J9v4Hirw8/984865wDeiNdy8edmdpGf/0Lg88Bm4Dng/zazX/tpPwV+AVwCvAx4GHiPmR3O8jv6IPAJoAP4JXCtme1zzr0AnAx81zmXADrNLDbj3F3APwFXez+6JmAF8AXgNcAI8A9mdrOf/5XAl/xyjwNfNbOPz2wFcc6dDHzZ/93+BrCMe77kfZv5+/Tv8XngdP8e9wEfN7PJWer+VuBzwHpgyC/r52b7PYmUSrV+yxApOOdcFLgfuAvvIfdN4J1Z8jrgvwLnmVkL8CZgl5n9EPgMcK+ZNZvZ2RmnvQ/4L0ALsHuWy54P7ASWA38DfNs515FD0V/j/93m3/PhGWXtAL4P3Ax0Av8b+L5zrjMj23uAP8F7KEeB67LU+xLgfwDvBlb79fg6gJmdCvQCl/rliM12DeAq4G1AG5AEvgs8BawFXg98zDn3Jj/v54HPm1krcCrwjSzX/BrQg/e7+zvg/VnyzSYB/KV/7gV+GT6cJe+/AH/hv+dbgP9cwH1EikItAiKL9yq8b/T/aGYp4FvOuY9nyZsA6oAznHP9ZrYrh+t/2cymm839b++ZDmXc+17n3F/hPTDvWlg1TvA2YIeZpa9zj3Puo8CleN+iAf7VzJ7zy/UN4I+yXOtq4P+Y2eN+3v8ODDrnTsrxdwBws5n1+eefD3SZ2Y1+2k7n3O3AlcCDwBSw0Tm33G+h+M3MiznnuoHzgDf4wcfPnXPfzbEsmFlPxo+7nHO3Aa8F/nGW7FN47/lTZjYIDOZ6H5FiUYuAyOKtAfb6D+K02b65Y2bPAx8DbgAOOee+7pxbM8/1++ZJn+3e810zF2s4sR678b6Bpx3IeD1G9oF4L7mWmY0AAzOuNZ/M38MGYI1z7mj6D17Xyko//c/wugW2O+cedc79YZYyDZrZaMaxWd+32TjnNjvnvuecO+CcG8Jr0VmeJfs7gbcCu51zP3POXZDrfUSKRYGAyOLtB9Y65wIZx7qzZTazr/l98BvwxhL8Tz8p2xag820NOtu99/mvR4HGjLRVC7juPr+MmbqBvfOcN++1/D7+zgVeK7O8fcCLZtaW8afFzN4KYGY7zOwqvC6L/4nXStM043r7gfYZxzPft5f87vwph10Z6f8EbAc2+V0QnwQy34dpZvaomb3dL8/9ZO+qECkZdQ2ILN7DQBz4qHPuFrzm8VcCP5mZ0R8jsBZvAOAE3iCzdCB+EHijcy5oZskF3H+Ff+8vAZfhDV57wE97ErjSOfcD4GzgcuCHflo/Xl/7KXgDAWd6APiCc+49eA+udwJnAN9bQNnSvgZ83Tn3NeD3eN+ef7uAboGZHgGG/IGXNwOTePVuMLNHnXPvBR40s36/tQC8bplpZrbbOfcY8LfOuU/ivWeXAt/xszwH1Dvn3gb8CO9BX5dxiRa8gX8jzrnTgA/h/U5fwh9D8i7ge2Z2zG89SMzMJ1JqahEQWSR/lPg7gA/g9f1eAXw7S/Y64CbgMF6z+gq8Bwx4gwwBBpxzjy+gCL/Fm61wGPg0cLmZDfhpn8IbLDcI/C3eAzld7jE//6/85vVXzajXAPCHwF/hNeP/NfCH2WYFzMWfxfApvJH1+/0yXbnQ62RcL4H30D4HbxbAYeAOYJmf5c3AM865EbyBg1ea2cQsl3oP3mDLI3gDLb+ScY9jeIP/7sBruRgF9mSce51//jBwO3DvHEV+H944giHgWmDOtSJESiGQSs3XSigiIiLVSi0CIiIiNUyBgIiISA1TICAiIlLDFAiIiIjUsKqdPtjT01OHt3rYfjRlR0REql8IbynvR7du3Zptye4TVG0ggBcE/KLUhRARESmyi/E2+MpJNQcC+wE2b95MNBot+s23bdvGli1bin7fYqr2OlZ7/aD661jt9YPqr2O11w/yV8fJyUmee+458J9/uarmQCABEI1Gqaurmy9vQZTqvsVU7XWs9vpB9dex2usH1V/Haq8f5L2OC+oO12BBERGRGqZAQEREpIYpEBAREalhCgRERERqmAIBERGRGqZAQEREpIblNH3QObcZuBPoxNuf/Boz2zEjTwi4GW8/8BRwk5ndkUPap/D2J4/7fz5pZg/6aV8G3oC35zjAN83s04utrIiIiLxUrusI3ArcYmZ3O+feC9wGXDIjz9XARmATXsDwhHPuITPbNU/aI8D/MrMx59zZwM+cc6vNbNy/7k1m9sXFV7E8DY/FGYvNPtWzsS5ES2M1L/EgIiLlYt6njXNuBXAu8Eb/0D3AF51zXWbWn5H1CuB2M0sC/c65+4F3AZ+dKy397d/3OyCAFyzsWVrVyttYLEHPjpFZ07ZualYgICIiRZHL02Y9sNfMEgBmlnDO7fOPZwYC3cDujJ97/TzzpWW6BnjBzDKDgI875/4CeAH472b2+xzKPG3btm0LyZ5XPT09WdPCjcvp3T0wa9ralk727Dw8a1q5mauO1aDa6wfVX8dqrx9Ufx2rvX5Q2jqWzddO59xrgb/jeMsDwP8D7DezpHPuGuCHzrlT0kFJLrZs2VKS5Sl7enrYunVr1vSDgzG6h5tJpVI8sn2YwZEp4vEU8WSK5Svrufw12c8tF/PVsdJVe/2g+utY7fWD6q9jtdcP8lfHWCy2qC+/ucwa6APW+gP+0gP/1vjHM/UCGzJ+7s7IM1cazrkLgLuBy8zM0sfNbK/fnYCZfQVoBtblUOaKceDIJI9sH2L/wCRDYwkGh+N89aFDHBuNl7poIiJSA+YNBMzsEPAkcJV/6CrgiRnjAwC+CXzQORd0znUBlwH3zZfmnDsPuBe43Mwez7ygc25txus34W2ksHdhVSxv214cJRIOcPUbVvKe16/k7RcuJzaV5Fs/P1TqoomISA3ItWvgWuBO59z1wCBeXz7OuQeA683sMeAu4HwgPa3wRjPb6b+eK+1LQANwm3Mufb/3mdnT/j1XAklgCPgjM6uar8oTk0l27B3jjA1NRMNeTNbRGuFVZ7Ty3YcP88cXddHREilxKUVEpJrlFAiY2Xa8B/nM42/NeJ0APpTl/LnSzpvjvm/IpXyVanvvKIkknHly00uOX/bq5fx2+xDf+Okhrr10bZazRURElk4rC5ZIKpVi265RVrZH6VoWfUnayvYobzy3g+//doD+o5MlKqGIiNSCspk1UG3qm9s5OBjLmv7srjEGh+O8/tz2WdOvumQlP35ikDse2MdH37GepvoQ4K0/8JUfHeAnTw3y0cvWsXFtwwnnakEiERHJlZ4WBRJPhrIuGATw8LPHiEYCbJrlQQ5eq8A7L+7i3p8e4lEb5o1bO+heWcdXf3yQweE4kVCAf/7+Pt79uhUEAoGXnKsFiUREJFd6WpTAeCzBkztGOOOkRiLh7L0zH3jTai48cxnf+fVhHnhkgHgihVvfyH+7bC0PPzvEf/QMYn1jnNbdlPUaIiIic1EgUAJ9/THiyRRu/fwP8M3rGrnu3d386VtWs+9wjDM2NNF/bJIjw3GeemGEXz8zxKlrGuYMKERERLLR06ME9h2OURcJsKIt96mBHS0RtpzcTDDodQMEAgEuPquN0YkEj8/RBSEiIjIXBQIlsG8gximrG6Yf6ou1ZnkdG9c28PiOYUbGc151WUREZJoCgSIbjyUYGIrPOtp/MS48cxnxRIpnd4/m5XoiIlJbFAgU2f4j3roAG9fkJxBY1hSmvTnMIa03ICIii6BAoMj2HY4RCsKGlfnbEbGrLUL/0am8XU9ERGqHZg0U2d6BGCvbo3OO8k8kUnMuRjQ5lXrJzyvaojy3Z5zxWIKGulDeyioiItVPgUARTcaT9B+dYuvmljnzTUwl2b5zPGv6aetf2q3Q5c8+OHR0ig0rFQiIiEju1DVQRAeOTJJKwdrO/HULANN7FWhfAhERWSgFAkW073CMALCqIzpv3oWoiwZZ1hTSOAEREVkwBQJFtG8gRldbhGgk/7/2rraoZg6IiMiCKRAokkQixYEjk6xZnt9ugbQVbRGGxhJMTCYLcn0REalOCgSK5ODgJIlk/scHpHW1+eMEjqlVQEREcqdAoEj6j3n99yvzPD4grWuZN3Ogf1DjBEREJHcKBIpkPJYgADTWFeZX3lAXoqUxxCG1CIiIyAIoECiSsViS+miQQGBpGw3NpWuZVhgUEZGFUSBQJN6qf4X9da9oi3J0JM54TDsRiohIbhQIFMl4LFnw5X/TKwz2Hsq+PLGIiEgmBQJFMj6ZLHiLQHrmwK6DEwW9j4iIVA8FAkVSjK6BpvoQTfVBdisQEBGRHCkQKIJEMkVsKkVjEXYGbG+OcEhTCEVEJEcKBIogvdpfQ7Twv+7WprCWGhYRkZwpECiCMX8Uf6G7BgCWNYUYGktM31NERGQuCgSKYDzmtwgUoWtgWVMY8LY8FhERmY8CgSI4HggUo0XACwT2KxAQEZEcKBAogvQCP8UYIzAdCAxoLQEREZmfAoEiGI8lCQSgvgiBQF00SFN9UC0CIiKSEwUCRTA+maShwPsMZFrRFlWLgIiI5CScSybn3GbgTqATGACuMbMdM/KEgJuBNwMp4CYzuyOHtE8BVwJx/88nzexBP60R+Fdgq592nZl9bykVLoWxIiwmlGlFW4TdWmZYRERykOvT6VbgFjPbDNwC3DZLnquBjcAm4ALgBufcSTmkPQKcZ2ZnA38K3Ouca/DTrgOGzWwjcClwh3OuOffqlYeJIuwzkKmrLcqho5MkEqmi3VNERCrTvIGAc24FcC5wj3/oHuBc51zXjKxXALebWdLM+oH7gXfNl2ZmD5rZmJ/vd0AAr+Uhfd6tfr4dwGPAWxZcyxIbjyWLMlAwbUVbhGQSDh3TOAEREZlbLl0D64G9ZpYAMLOEc26ff7w/I183sDvj514/z3xpma4BXjCzPQs8L6tt27YtJHvehBuX07vbK/rIRJD2yUl6d49Mp69rX0fv7j2znjtXWi7pK5cvA+Dnv3mGjSsL2yrQ09NT0OuXWrXXD6q/jtVeP6j+OlZ7/aC0dcxpjEAxOOdeC/wd8MZ8XnfLli3U1dXl85I5eer3u+nesIFEMkX88b2sWL6M7g2t0+nNzQ10b9gw67lzpeWSfvKqOmCE5s5utm5dvug6zKenp4etW7cW7PqlVu31g+qvY7XXD6q/jtVeP8hfHWOx2KK+/ObSXt0HrPUH/KUH/q3xj2fqBTKfTt0ZeeZKwzl3AXA3cJmZWY7XrAjpxYQaizhYsK05TCQc0OqCIiIyr3mfTmZ2CHgSuMo/dBXwhN/Xn+mbwAedc0F//MBlwH3zpTnnzgPuBS43s8dnueZf+Pk2AecBP1xYFUsrvZhQfbR4gwWDgQCr2qPsH1AgICIic8u1a+Ba4E7n3PXAIF5fPs65B4Drzewx4C7gfCA9rfBGM9vpv54r7UtAA3Cbcy59v/eZ2dPAZ4EvO+eeBxLAfzGz4YVXs3RK0SIAsLojyv4jmkIoIiJzyykQMLPteA/ymcffmvE6AXwoy/lzpZ03x31HOT7zoCKNTxZvn4FMqzvrePrFUVKpVNEWMhIRkcqjlQUL7PgWxMXrGgCvRWB8Msmx0XhR7ysiIpVFgUCBjceSBANQFynut/LVnd5MCe05ICIic1EgUGDjsSQNdcXbZyBtdUcUQAMGRURkTgoECmw8lijqjIG0le1RAgE0YFBEROakQKDAxieTRR8oCBCNBOlsjahFQERE5qRAoMDGY4miTx1M0xRCERGZjwKBAhsv8s6DmVZ1RDk4qBYBERHJToFAASUSKSbjqZJ0DQCsbItyZDjOVDxZkvuLiEj5UyBQQOOT/hoCRdyCONOK9iipFPQfmyrJ/UVEpPyVze6D1Wgsll5VsLhdA4lEioODMSL+ba1vlJAfizTWhWhp1NsuIiIePREKaDxWmuWFJ6aSbN85Pr2qYM+OEcYnUwBs3dSsQEBERKapa6CA0jsPlmrWQHOD1yQwPKZlhkVEZHYKBApoukWgBAsKAYSCAZrqQwyPJUpyfxERKX8KBApofNLbZyBa5H0GMrU2KhAQEZHsFAgUUKn2GcjU0hhiaFxdAyIiMjsFAgU0FU8SCZf2V9zSGGZkLEEqlSppOUREpDwpECigqXiKaLh0rQHgtQgkUzA6oUWFRETkRAoECmgyniRS6kBAMwdERGQOCgQKaCqRIhIq7a+41V8zQAMGRURkNgoECmgqniJSwhkD4HUNAAypRUBERGahQKCApuLJkrcIRMJB6qNBhsfVIiAiIidSIFBAk2UwWBC8VgF1DYiIyGwUCBRIKpXyugbKIRBoCGmwoIiIzEqBQIFM+V/AS72OAHgDBoe1loCIiMyi9E+pKhWb8h66ZdEi0BhiKpFiYlJrCYiIyEspECiQyXg5BQL+FEINGBQRkRkUCBRIbMr7O1oGXQPpKYQaMCgiIjOV/ilVpcqqRUCrC4qISBYKBApkeoxAqPSBQH00SCQUUIuAiIicQIFAgRxvESj9rzgQCHjbEatFQEREZij9U6pKpVsEymFBIfAGDGqwoIiIzKRAoEAm/S/f5TBGALS6oIiIzC6cSybn3GbgTqATGACuMbMdM/KEgJuBNwMp4CYzuyOHtD8APgOcBXzBzK7LuOYNwIeBff6hX5nZRxZV0yI7vo5AecRaLQ0hJiaTxLSWgIiIZMj1KXUrcIuZbQZuAW6bJc/VwEZgE3ABcINz7qQc0nYCHwQ+m+XeXzGzc/w/FREEgDdGIBiAEu85NC29HfHhoakSl0RERMrJvI8p59wK4FzgHv/QPcC5zrmuGVmvAG43s6SZ9QP3A++aL83MnjezJ4CqGskWm/L2GQgEyqdrAGBAgYCIiGTIpWtgPbDXzBIAZpZwzu3zj/dn5OsGdmf83OvnmS9tPlf63QcHgL8xs4dzPA+Abdu2LSR73sSmGgmQpHf37lnT17Wvo3f3ngWnLTZ9fBIgxM6+AXrYNV/xc9bT05O3a5Wjaq8fVH8dq71+UP11rPb6QWnrmNMYgRK6Ffi0mU05594I/Ltz7nQzG8j1Alu2bKGurq5wJczi3kefpaEuQveGVbOmNzc30L1hw4LTFpueSqV46Jm9TNHA1q2bc6jB/Hp6eti6dWterlWOqr1+UP11rPb6QfXXsdrrB/mrYywWW9SX31x6sPuAtf6Av/TAvzX+8Uy9QObTpzsjz1xpWZnZATOb8l//h3/OlhzKXHKT8RSRSHl0C4C3lkBzY4jDx9Q1ICIix80bCJjZIeBJ4Cr/0FXAE35ff6ZvAh90zgX98QOXAfflkJaVc25txutzgJMAm++8chCbShEtl5GCvtaGMANDVTUUQ0RElijXroFrgTudc9cDg8A1AM65B4Drzewx4C7gfCA9rfBGM9vpv86a5py7CPg60AoEnHNXAn9mZg8Cn3HObQUSwCTwPjM7sOjaFlFsCurryqdFALwBg/uPTJa6GCIiUkZyCgTMbDveg3zm8bdmvE4AH8py/lxpvwTWZUl7fy7lK0eT8RQtTeUXCGzvjTMVT5bN+gYiIlJaehoUyGQ8VXYP25bGMCmgX+MERETEV15PqiqSXkegnKTXEjh0VN0DIiLiUSBQAMlkisl4+ewzkNba4PUEHRpUi4CIiHgUCBRAbMpbzz9aZl0DzY0hAqhFQEREjiuvJ1WVGPc39im3FoFQMMCy5rACARERmaZAoADGY34gECqvQABgeWtEXQMiIjJNgUABjE8mgPLZgjhTZ2uYg2oREBERX/k9qarARCw9RqD8WgQ6l0U4fGyKZDJV6qKIiEgZUCBQAOU6RgC8roF4IsWRYS01LCIiCgQKYnqMQFl2DUQAzRwQERFP+T2pqkB6jEA5dg0sX+YHAoMKBERERIFAQUyoRUBERCpE+T2pqsBYGY8RqI8GaWkIcfCophCKiIgCgYKYiCUJBb0FfMrRivaougZERARQIFAQ45MJ6sqwNSBtRVtEXQMiIgIoECiIickk0UipS5HdirYoh45OkUppLQERkVqnQKAAxmLJspwxkLayPcrEZJLhsUSpiyIiIiWmQKAAJmJJ6iLlGwisaNPMARER8SgQKIByHyOwsiMKwAENGBQRqXkKBAqg3McIrO6oA2D/gAIBEZFap0CgAMp9jEBTfYjWxhAHjigQEBGpdQoECmBisrzHCIDXKrD/SKzUxRARkRJTIFAA47FkWY8RAFjVEWW/WgRERGqeAoE8SyRTxKaSRMu9RaAzyqGjk8QTWktARKSWKRDIswl/n4FouMQFmceqjijJJPQfU6uAiEgtUyCQZ+lAoBLGCIBmDoiI1DoFAnk27m9BXO5jBFan1xLQOAERkZqmQCDPxie9ZXvLfYxAZ2uEcCigmQMiIjVOgUCeTbcIlPGCQgDBYEAzB0RERIFAvh0fLFjeLQLgdQ/sH1CLgIhILVMgkGfproFyHyMA3syBA0cmtR2xiEgNUyCQZ+mugXIfIwDezIGxWJIhbUcsIlKzcprt7pzbDNwJdAIDwDVmtmNGnhBwM/BmIAXcZGZ35JD2B8BngLOAL5jZdblcs1xljhEYLfNW98yZA8uaynzhAxERKYhcWwRuBW4xs83ALcBts+S5GtgIbAIuAG5wzp2UQ9pO4IPAZxd4zbJUUWMEOtNrCZR5xCIiIgUzbyDgnFsBnAvc4x+6BzjXOdc1I+sVwO1mljSzfuB+4F3zpZnZ82b2BBCf5fZzXbMsjcUSRMIBQsHyDAQSiRQHB2McHIwRCHhjA3bsG5s+Njw229sgIiLVKpf24PXAXjNLAJhZwjm3zz/en5GvG9id8XOvn2e+tLks9rxp27ZtW0j2JevbGyTiBwG9u3dnzbeufR29u/csOG2p6eva1/HrJ4+n1UWCPPbsIRKjBwG4YEsn8bHDWa89U09PT855K1G11w+qv47VXj+o/jpWe/2gtHWs+o7hLVu2UFdXV7T7/eSFXlqaRgHo3rAha77m5oas6XOlLTV9ZlrHrkMkAtC9YQUAq1Y1s7I9+7Uz9fT0sHXr1pzyVqJqrx9Ufx2rvX5Q/XWs9vpB/uoYi8UW9eU3lzECfcBaf+BeegDfGv94pl4g8wnSnZFnrrS5LPa8kpmYTNIQrZzJGMuawhwb1awBEZFaNe8Ty8wOAU8CV/mHrgKe8PvsM30T+KBzLuiPH7gMuC+HtLks9rySGYslqK+gQKC1KczIeELbEYuI1KhcuwauBe50zl0PDALXADjnHgCuN7PHgLuA84H0tMIbzWyn/zprmnPuIuDrQCsQcM5dCfyZmT04zzXL0sRkkoa6ygkEljWFABgai9PRUubrIouISN7lFAiY2Xa8B/LM42/NeJ0APpTl/LnSfgmsW+h55WoslqyoB2p6/YChUQUCIiK1qHK+ulaI0YkETQ2hUhcjZ+lA4Niopg2KiNQiBQJ5NjqeoKmCugYa6oJEIwEGhxUIiIjUosp5YlWARCLF+GSyoloEAoEA7c0RBQIiIjVKgUAejcW8aXhN9ZUTCAB0tIQ5MjxV6mKIiEgJKBDIo9GJygwE2lsijMWSxPx9EkREpHYoEMijkYoNBLwBg4MjahUQEak1CgTyaGzC+0bdXEFjBMDrGgA4onECIiI1R4FAHo2Mey0CjfWV9WttbQwTDMKgxgmIiNScynpilbn0GIHmCusaCAYDtDeHNXNARKQGKRDIo0odLAjQ3hxR14CISA1SIJBH6UCgsa4CA4GWMEOjcabimjkgIlJLFAjk0eh4goZokFAoUOqiLFhHS4QUcHBQ4wRERGqJAoE8Go1V1qqCmdJTCPcNxEpcEhERKSYFAnk0Op6oyPEBAG3NXiCwf2CyxCUREZFiUiCQR6MTCZoqbOpgWiQcpLUxxP4jCgRERGpJZT61ypQXCFRmiwB4Sw2ra0BEpLYoEMijig8EmsMcODJJMpkqdVFERKRIFAjk0UilBwItYSbjKfqPaeaAiEitUCCQJ6lUirGJZMXtM5CpoyUCQO+hiRKXREREikWBQJ7EplLEEykaK7xFAKDvkMYJiIjUCgUCeTJWofsMZGqoC9HSEFKLgIhIDVEgkCcjE5W58+BMG1bVY3vGSl0MEREpksp+apWRSt15cKZNaxrYfXBiuj4iIlLdFAjkyeh45e48mGnj2gZSKbA+tQqIiNQCBQJ5MhrzA4EKnjUAcMrqegIB+P3u0VIXRUREikCBQJ6Mjnvb91Z6i0BDXYiTVtbzbK8CARGRWqBAIE/SfeqVHggAnN7dxPbeMa0wKCJSAxQI5MnoRIJQEOoigVIXZclO39DIWCypaYQiIjVAgUCejE4kaGoIEQhUfiBwRncTAM/2asCgiEi1UyCQJyPjCZrqKr9bAGB1Z5TWphDbNU5ARKTqKRDIk7GJREXvM5ApEAhwRncTz+5Wi4CISLVTIJAnIxOJit5nYKbTNzSx93CMY6PIIRQcAAAV0ElEQVTxUhdFREQKKJxLJufcZuBOoBMYAK4xsx0z8oSAm4E3AyngJjO7Y4lpNwAfBvb5t/mVmX1ksZUtpLGJJGuXR0pdjLw5vbsRgO29Y5x/emuJSyMiIoWSa4vArcAtZrYZuAW4bZY8VwMbgU3ABcANzrmTlpgG8BUzO8f/U5ZBAKRbBKqngWXT2kZCQfi9xgmIiFS1eZ9czrkVwLnAPf6he4BznXNdM7JeAdxuZkkz6wfuB961xLSKMTqRqPh9BjLVR4OcuqaBZ7TCoIhIVcula2A9sNfMEgBmlnDO7fOP92fk6wZ2Z/zc6+dZShrAlc65PwAOAH9jZg/nUOZp27ZtW0j2RUmmYDwW5tjgQXp6DgAQblxO7+7dWc9Z176O3t17Fpy21PT5zj1l+ToOHBgHYG17kp9tG+eXj79IS4MXM4aDCSZGBqfz9/T0ZL1WNaj2+kH117Ha6wfVX8dqrx+Uto45jREooVuBT5vZlHPujcC/O+dON7OBXC+wZcsW6urqCldCYHg8Dvc9w6ZT1rN1q9dQ8tTvd9O9YUPWc5qbG7Kmz5W21PT5zo02NLDzsNeysXpFHcnUQX5hQc4+tRmArZuaWelOAbx/uFu3bs16rUpX7fWD6q9jtdcPqr+O1V4/yF8dY7HYor785tKp3Qes9Qf1pQf3rfGPZ+oFMp8w3Rl5FpVmZgfMbMp//R/+8S05lLmoqmXnwZk6WyN0tkZ4bo+mEYqIVKt5AwEzOwQ8CVzlH7oKeMLvz8/0TeCDzrmgP37gMuC+paQ559amL+6cOwc4CbAF17LARieqY8Oh2bj1DRw4MqlphCIiVSrXroFrgTudc9cDg8A1AM65B4Drzewx4C7gfCA9rfBGM9vpv15s2mecc1uBBDAJvM/MDiywjgVXTRsOzbRpXSO/fmaI5/aMcZ7TNEIRkWqTUyBgZtvxHtYzj78143UC+FCW8xeb9v5cyldqI+mugYbqmT6Y1toYZk1nFOsb4xWbW0pdHBERybPqe3KVQLpFoJqmD2bavL6RweE4h4emSl0UERHJMwUCeVDNXQMAG9c0EAzAc30aNCgiUm0UCORBOhBorJLdB2dqqAvRvaKe5/aME0+kSl0cERHJIwUCeTA6nqAhGiQUCpS6KAWz5eQmRsYT/OTJwfkzi4hIxVAgkAejsSRNVbIFcTYnrapnfVcd//bLwxwd0VRCEZFqoUAgD0bHE1U7PiAtEAjwmpe1MTGZ5K7/KLsZnCIiskgKBPJgdCJBUxXtPJhNR2uE15/bzg8eHeD5fRo4KCJSDar/6VUEXiBQ3S0CaZe9ejktjSFu/e4+Uho3KCJS8RQI5EEtBQJN9SE+8AereWbXKD/bXr2DI0VEaoUCgTwYqaFAAODN53VwyTntPPRMiB8/fqTUxRERkSVQILBEqVSKsYlkTQUCgUCAj71zHad0JfmH+/p44vnhUhdJREQWSYHAEh0+NkU8kWJFe6TURSmKRCLFwcEYR4anuOb17azuqOPGu3bxyPZjHByMMTymqYUiIpVEgcAS9fXHAFjfVV/ikhTHxFSSnh0j9OwY4YnnjvCGre2EQwFu+novDz0+yFgsUeoiiojIAigQWKK+QxMArO+qK3FJSqOlMczbX70cUnD/rw4zoI2JREQqigKBJerrj9HcEKKtOacdnatSR0uEt796OZPxJH9/by+DwwoGREQqhQKBJerrn2B9Vx2BQG1Ppetqi3LpBcsZHInzV7c+T6/fUiIiIuVNgcAS9fXHWL+iNsYHzGdNZx1/fUU3E5NJ/vJLO3jUhkpdJBERmYcCgSUYGU8wOByv2fEBs9m4poF//MgmVnVEueHOF/najw8Sm0qWulgiIpKFAoEl6OtPDxRUi0BaIpEilUrxiSu6ecXmFu566AB/8ve/596fHNTYARGRMlS7I9zyoO9QeuqgWgTSJqaSbN85DsD5ZyxjbVcdDz8zxJd/dIAfPDLAlZes5JKXtxMNKwYVESkH+t94Cfr6JwiHAqzsiJa6KGVrXVc9l7+2i7e9qpP6uiCf//Ye/vSzv+dbPz/EvoFYqYsnIlLz1CKwBH39MdYtryMUrO0ZA/MJBAKcsrqByy9ezt7DU3zjZwf5lx/s519+sJ9V7VFevqmZs05u5owNTaxoi9T8DAwRkWJSILAEfYcmOHVNQ6mLUTGSSVjXFeXjl69n/0CMZ3aP8cyuUX7y5FF+8Ii3eVFna5izT2nhFa6FrZtaaG06/k90eCxOX/8ET+8c5YX94ySTKQKBAOFQgEte3s55rrVUVRMRqVgKBBZpMp7kwJFJXnt2e6mLUjEyxw8AtLdEuOisNi48cxkDQ1OEAvD8vnEetSH+88lBAgHoWhahPhqkPhpkeCzB/iOTADTUBQmHApCC2FSSnz51lNe+rI0/fctqVrSpq0ZEJFcKBBZp3+EYyRR0a6DgkgWDAbraopy2voGOZVFecVorhwan2H1wnKMjcabiKUYnkrQ1hzllTQMnraynvSU83YUwFU+yfyDGDx45wsPPHuOPLlzOH1/URUdLbWwEJSKyFAoEFim92dC6FQoE8i0YCLCqI8qqGYMwT1vfwPa+8RPyR8JB/viiLt5x8Qq+/OB+vv2Lfr7z68O86RUdvO6cdpY1hWltDNFUHyKo8RwiIi+hQGCR0lMH1y3XGgLlYkVblL++YgNXv34V3/jZIR54ZIDv/mZgOj0YgOaGEC2NYTpawpxzagvnndbCqasb5g0QUqkUyRQaGCoiVUeBwCL19U+wos3rv5bSSyRSHBz0grNwCN5zyQrecl47vYdijEwkGJ1IMjwWZ3Q8wchEgv6jU9z90AHueugArU0hzt3YwoVnLuPcTS3URYIMDE3xYj/0/aqfZ3eP8uzuUY4Mx2lrDrO8NcKqjiivfVkbrzytlYjWRBCRCqZAYJH6+mN0a4+BsjFzIGKm+miIc05tPqFbYSyWoPfgBLsOTPDI9iF++tRRgkFIpbw/3sdjH8uXRdi0tpHlyyIMjcY5Ohrn6Z2j/OLpY7Q2hnjd2W28+3Ur6WzVmAQRqTwKBBYhmUyxpz/GWSc3lboosgSNdSFO627itO4mzjqpkV0HJ9j24iiBAHS2RgjEhzlz0yqa68M8vWv0JecmUyl6D07wzK5Rvv/bAR58bJB3XLScd75mBU31oRLVSERk4RQILMK+gRixqaR2HawiU4kUw+NJNqw6vi5E7+4hXjzQzmnrT2z6DwYCnLSqgZNWNbC+K8r3f3OEe35yiAceOcI7Lu7iLa/soKWh8B+vw8cmedSGePGA17LR1x+juT7E6s4oazqjvHxjC688rVWLNIlIVgoEFmhyKsnnvtFHfTTIyzc2l7o4UgY6WyL8yZtX8bqzl/Gtn/fzrz/cz1d/fICLzlzGq85oZfPaRlZ25G92yXgsMb0Q06+eOUpsKkUgAB0tEVa2R5mcSvLcnjEetWH+/dcDbFhZz9vO7+SSl7ertUJETpBTIOCc2wzcCXQCA8A1ZrZjRp4QcDPwZiAF3GRmdxQqrRRSqRQ3/9sebM8Y/+97T2J1Hv9zl8qVOT7hknM7OOvUZp56foSf/u4o//nkUcBbGGlle5RQKEAo6M0+OP7HW0shFAwQnJGWSqVIJCGeTHF0JM7ugxMcHPQWVWquD3H+6a10tEZY3VHnLbCUYSqeZCqe5Oe/O8aXvrOXf/7+Ps46uYnzXCtnn9pMW1OYqcTx/OmZEYlkimTSu280HNBgSJEql2uLwK3ALWZ2t3PuvcBtwCUz8lwNbAQ24QUMTzjnHjKzXQVKK7p/++VhfvzEIO99w0pefeayUhRBKkDXsihv2NrBq7cs48CRSerCAfYdmeTI8BSxKe9Bm0o/cFPejIdkMkUi5Y0/SSYhkUqRSKQIBr0llEPBAC0NIdz6Rt70ig5OXdPAORubGRyeomfHyKzliISDnLe5hYvPamPn/nEe2T7M73aO8M/f35eRK0z4O7+bDjhm01QfZFlTmPaWCF3LIqxoi9LW7P3Xka7H0FicY6NxhsYSTMWTXh2SXvkjoQCRcPrvIJGwV6f0z9FIgM7WCF3LonS1efdoqFPLhUixzBsIOOdWAOcCb/QP3QN80TnXZWb9GVmvAG43syTQ75y7H3gX8NkCpc0nBDA5OZlD1vn19U9w38/28oaXt/COC9uIxebeOS+VTBBkKmt6Ih7Kmj5X2lLT83ntaDh1Qt5qqFdaun6LvXZTHZy6Osypq+t5YX8IOHFfCi9tIuu1z+huzLp2Qf+RUabiJ74HmUbHJ6av79ZFcOvaGR5LcPjYFJ0tIQaOjhAKe+soBAJ4rRMBv4UiFCA2lWR0PMnIeJyhsTi9ByZ4+oX4CUFDKAhNDSGa6kJEwgEvgPFnYExMpBhNpognUiT8v+MJLwCKJ5OzBiCNdSE6W8M01IUIBJj+EwxAkMDxY8EAAbxrTSVTxOMp4um/EymmJoN8+wmjLhIgGglSFw4QjQaJBIMkU6npgGt8Msl4LMnEZHK6nIlkimDAa6EJhwLTQU0wGCAYwLt+wq+X/zqRSDKV8NLiyRRBAkQjx4OhaDhI2P87EvaCtUgo/fPxgCkaCU7faz7790Pf2L558xVTMnW8hcmbgeMFv+nj6Zk5ma1QiYT/t/+eJJLe+z08FODRPbu8gDgEYf/fZjpA9lrOvPum0gVIpaZfn5iW8ZrUCcdekt//YWbaS/KlvPc64b/viYx/60F/L5RgKEA4ECAYgnC61S8UIBwMTO9XM98zJRcZz7sFRdK5tAisB/aaWQLAzBLOuX3+8cxAoBvYnfFzr5+nUGnzWQ3w3HPP5Zh9fh9/C8BRnn32aE75u0KDWdOGDkFXlrdqrrSlpufz2l3dAHtLcu9iXDtdv1KWu39v9rTpci60Xi1wSov3+tRlAGPz3yQnifmzLOha+QniYfYWk2qx/lSA/aUuRoENzJ+lUvkTkrZt25bPq64GXsg1czUPFnwUuBjvE5LP/6FERETKUQgvCHh0ISflEgj0AWudcyG/NSAErPGPZ+oFNmQUIPPbfCHS5rR169YY8Mtc8oqIiFSJnFsC0uYNBMzskHPuSeAq4G7/7ydmjA8A+CbwQefct/EG9l0GvKaAaSIiIrJEuXYNXAvc6Zy7HhgErgFwzj0AXG9mjwF3AecD6WmFN5rZTv91IdJERERkiQKpzGGQIiIiUlO0UoiIiEgNUyAgIiJSwxQIiIiI1DAFAiIiIjWsmhcUKolcNmiqBM65XcCE/wfgE2b2oHPuVXh7TTQAu4D3mtkh/5ysaaXmnPsc8E7gJOAsM9vmH8/6fi02rVTmqOMuZnkv/bSKeT+dc514M4lOBWLA88BfmFn/YutRYXVMAU8D6UWZ32dmT/vnXYq39HoY6AH+xMzG5ksrBX+p+JPx6jEC/Dcze7LKPovZ6riLMvwsqkUg/9IbNG0GbsF78yrV5WZ2jv/nQedcAG8tiY/49fs5cBPAXGll4n68NShmLkg11/u12LRSyVZHmPFewtzvWZm+nyng783MmdnL8BZOuWmx9aikOmakX5jxPqaDgGbgduBSM9sIDAPXzZdWQu83s7PN7OXA54D/4x+vps9itjpCGX4WFQjkUcYGTff4h+4BznXOdZWuVHn1CmDCzNIrNt4KvDuHtJIzs1+a2UtWw5zr/VpsWqHrMZfZ6jiPino/zeyImf0049Bv8FYeXWw9KqmOc3kL8FjGt+Bb8TZsmy+tJMzsWMaPy4BkFX4WT6jjPKeU9N+pAoH8OmGDJiC9QVMl+qpz7nfOuS8559qYscSzmR0Ggs65jnnSytVc79di08rVzPcSKvj9dM4FgQ8B32Hx9aikOqb91Dn3pHPufzjn6vxjhdq4rWCcc3c453qBTwPvpwo/i7PUMa3sPosKBCSbi83sbOA8IAB8scTlkcWrxvfyC3h9r9VQl2xm1rHbzF6B1/1zBvCpUhVsqczsz82sG/gkuW0rX3Gy1LEsP4sKBPJreoMmgDk2aCp76SZmM4sBXwJezfFNoABwzi0HUmZ2ZJ60cjXX+7XYtLKT5b2ECn0//UGRm4ArzCzJ4utRSXXMfB+HgDvI8j7ifYPsyyGt5MzsLuD/AvZQpZ/FdB2dc53l+llUIJBH/ijO9AZNkH2DprLmnGtyzi3zXweAK/Hq1QM0OOcu8rNeC3zDfz1XWlma6/1abFrxSp+bOd5LqMD30zn3aWArcJn/nyksvh4VU0fnXLtzrsF/HQYu5/j7+EPgPOfcJv/nzHrMlVZ0zrlm59z6jJ8vBY4AVfNZnKOOE+X6WdReA3nmnDsNbypLO/4GTWZmpS3VwjjnTgHuw9vbOgQ8C3zUzPY75y7EG5Vbz/FpLAf987KmlZpz7mbgHcAq4DAwYGZnzvV+LTatVGarI3ApWd5L/5yKeT+dc2cC24DngHH/8Itm9seLrUel1BH4e7xypoAI8GvgY2Y24p/3dj9PCHgC+ICZjc6XVmzOuZXAvwNNQALvAXmdmT1eLZ/FbHUEjlKmn0UFAiIiIjVMXQMiIiI1TIGAiIhIDVMgICIiUsMUCIiIiNQwBQIiIiI1TIGAiCyac67bOTeSXtBllvQbnHN35+leKefcxnxcS0SO0zbEIrJoZtYLNJe6HCKyeGoREJH0anUiUoP04RepUc65XcA/AVd7P7pNwD/gbWozAvyDmd3s530l3trom/FWvPuqmX3cOXcS3sp3ETOLO+dOBr6MtzXsbwDLuN/rgLvNbN2MMvy5mT3k3+PzwOn+Pe4DPm5mkwX5BYgIoBYBkVp3FfA2oAP4N+ApYC3weuBjzrk3+fk+D3zezFqBU8m+1vnX8NZGXw78HS/dfnU+CeAv/XMv8Mvw4YVURkQWTi0CIrXtZjPrc86dD3SZ2Y3+8Z3OudvxNkZ5EJgCNjrnlvv7of9m5oWcc91426u+wd8s5+fOue/mWhAz68n4cZdz7jbgtcA/LqpmIpITBQIitS29XesGYI1z7mhGWgj4hf/6z4Abge3OuReBvzWz78241hpgcMaGNruB9eTAObcZ+N/AK4BGvP+feuY8SUSWTIGASG1L7zrWh7eT36bZMpnZDuAq51wQb4fDbznnOmdk2w+0O+eaMoKB7ox7jOI94IHpveO7Ms7/J7zd8a4ys2Hn3MfwttsVkQJSICAiAI8AQ865TwA3A5N4g/YazOxR59x7gQfNrD+j1SCReQEz2+2cewz4W+fcJ4FX4m2D/B0/y3NAvXPubcCPgE8CdRmXaAGGgBF/a9kPASXbV16kVmiwoIhgZgm8h/Y5eLMADgN3AMv8LG8GnnHOjeANHLzSzCZmudR7gPPx9mD/G+ArGfc4hjf47w5gL14LwZ6Mc6/zzx8GbgfuzVP1RGQOgVQqNX8uERERqUpqERAREalhCgRERERqmAIBERGRGqZAQEREpIYpEBAREalhCgRERERqmAIBERGRGqZAQEREpIYpEBAREalh/z8q+BQVhM+9DgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f98af1c60f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Distribution of residuals\n",
    "plt.figure(figsize=(8,4))\n",
    "sns.distplot(np.abs(y_val - y_val_pred))\n",
    "plt.xlabel('residual')\n",
    "plt.title('distribution of residuals');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f4e55b8df4118d91fa9936ea851e7394",
     "grade": false,
     "grade_id": "cell-69e4e7c6638615ac",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAEcCAYAAABNkSdNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzs3Xd4XNWZ+PHvFI1673K3sQ8YmyaDjcGUUI1NQoAQUiAEUkjPJtmwv5RNJZvshk1IIRASEuomELpppoMBYyxsjLF93C1ZvY/azGjK7487I4/k0eiONCONpPfzPHos3Ttz75mrse4757znPZZAIIAQQgghxFDWiW6AEEIIIZKTBAlCCCGEiEiCBCGEEEJEJEGCEEIIISKSIEEIIYQQEUmQIIQQQoiIJEgQYppTSj2jlPrMMPvmKqUCSil7HM7zd6XUz4fZd51SasNYzxHl3OP1Gs9RSh0e63FiPGe3Umr+eJ5TTB9j/k8hhJjctNarJ7oNiTZVXqNS6hXgPq31X0LbtNZZE9ciMdVJT4IQcRCPT6GT+fzJSillUUpNir9z8jsUyUjelGLKU0r9B/B5oASoAb6vtX5UKZUKNAJnaq23Bx9bDFQDc7TWTUqptcDPgbnADuBGrfW24GMPAn8CPmX8qDKB70Q6V/DxNuC/gc8AXcAtwO+BFK21VymVC/wvcAngB/4G/Ehr7Yvwmn4MLAFcwIeBbyml7gK+Gzx/HvBisL1tSqk04C/AasAG7AHWaq0bwz+dBtv4K+A6wBlsY/h5DwKf01q/ENaOY7TWnw7+/BCwCkgH3gO+pLX+YIRfUYhFKfV74FqgHviK1vpFpdTHgP/QWleGtePbwCqt9WURrs0rwBvAOcApwFKl1F9ieI2x/B7SMd4DHwm2+W9D9geAhVrrvcGf/w4c1lr/QCl1DnAfxnvg34DnlVJfB+4FlmP8fX4D43d4WCl1M8a1XaGU+i3wd631V8PPEWz77zF+z73AncAvtNZ+pdR1wOeAjcANQAfwZa31M5F+GUKA9CSI6WEfxh/XXOAnwH1KqXKttRt4BPhE2GOvAl4NBginAHcBXwQKgTuAJ4LBRcgngDVAntbaO9y5go/9PMYf75Mwbl5Db3B3A17gGOBk4EKMP+rD+QjwL4yA4H7g68Fjng1UAO3AH4OP/UywTbOCr+VGoC/CMT8PrA2efxlwZZTzR/IMsBAjSHo32C6zlgP7gSLgR8AjSqkC4AlgnlLquLDHfhrjZjqca4AvANnAoSH7RnqNsfwefgQsCH5dhHGdY1EGFABzgu21YgQac4DZGL+jPwBorb8PvA58VWudpbX+aoTj/R7j9zwf431wLfDZsP3LAY1xjf8b+KtSyhJjm8U0Ij0JYsrTWj8U9uM/lVL/DzgNeBx4APgz8P3g/k9iBANg3Ezu0Fq/Hfz5bqXU94AVwKvBbb/TWteYPNdVwK1a68MASqlfAucFvy/FCCDytNZ9QI9S6jcYN447iOwtrfVjwe/7lFJfxLiBhI7/Y6BaKXUN0I8RHBwT7AmpGuaYVwG/Db0mpdR/YXwiN0VrfVfo++D525VSuVrrThNPbwqeO4Bx7b4NrNFa36uU+idGYPB9pdTxGD0766Ic6+/hPRhKqfB9w77GUfwersL4NN4GtCmlfgf8p4nXGuLH6KVwB3/uAx4Oa/fNwMtmDhTsIfk4cLLWugvoUkrdghEw/TX4sENa6zuDj78buA0oBRpiaLOYRiRIEFOeUupa4FsYNxaALIxPUgAvAelKqeUYfyhPAh4N7psDfEYp9bWwwzkwPqWH1IR9P9K5KoY8Pvz7OUAKUB92Q7MOPf4QQ/fNAR5VSvnDtvkwbgL3YvQi/EMplYfRzf19rXX/kGMMbePQT+HDCt6kbgY+BhRj3ADBeP1mgoTaYIAQfu7Qtb4b+D+l1A8wbnoPht1YI4l23aK9xlh/D6O+XkHNWmtX6AelVAbwG+BiID+4OVspZYs03DFEEcb7M7wNh4AZYT8PBANa697ga5TERzEsCRLElKaUmoMxLnsexidvn1JqK2ABCI7VPogxbNAIrAt+CgPjj//NWuubo5xi4KY20rkwxqxnhj13Vtj3NYAbKAoOW5gxdAnXGuB6rfUbwzz+J8BPlFJzgacxup3/OuQx9UPaNXvI/h4gI+znsrDvP4kxBHI+cBCj27udI69/JDOUUpawQGE2xlADWuuNSikPxlDOJ4Nf0URb3jbaa4z19xA6VqjXYuj16uXo6xU+RXJoO78NKGC51rpBKXUSsIUj1zDa62rB6DGag5E/E2pP7cgvQ4jIJEgQU10mxh/WZgCl1GcxEv7CPQA8BrRyZNgBjBv+o0qpF4BNGH/szwFeCwskYjnXg8A3lFJPYdxsbwrt0FrXK6XWA7copX4IdAPzgJla61cx53bgZqXUZ7TWh4JJmCu11o8rpc7FuInswEjW68foZRjqQeDrSql1wTb+x5D9W4GrlVLPACdijOc/G9yXjXGDbcW4Vr8w2e6QkuC5b8PIrTgOI5gJuQdjfN6rtR5LTYVhX+Mofg8PAv9PKfU2xu//a0P2bwU+qZT6ALgAI09gc5S2ZWMMOXQE8zF+NGR/I0a+wVGCQemDGO+BazFyHb4F/DrK+YSIShIXxZSmtd6Bkb3+FsYf2KUYGePhj3kb42ZRgZF4F9q+GSMv4Q8Yn4j3YmTEj/ZcdwLrgW0Ynw6fxkiQC92sr8XoLt4RPN+/gHLMuxXjk/d6pVQXRhb78uC+suDxnMBOjJyK+yIc407gOYyZCe9iJHaG+yFGkl47Rs/EA2H77sHo3q4NvoaNMbQd4G2MpMcWjGGLK7XWrWH778UIuqIlLJox0muM5ffwE4zXfADjdzu0bd8ALsWYSfApjGA0mt9izAxpwbh+zw7ZfytwpVKqPZj/MNTXMN7L+4ENGL+fuyI8TghTLIFAtN4rIUSiKKVWA7drredMdFsmg+B0wybgFK31nolujxDTgQw3CDFOgje5czE+cZZidCU/GvVJItyXgHckQBBi/EiQIMT4sWB0T/8TY9z5KWKbLjdtBYs4WTi6toQQIoFkuEEIIYQQEY1LT4JSqhAjoWcBRvbzXuCLWuvmYEnR9zkyp/oarfX7weddCvxPsJ1VwGe11r0j7RNCCCHE2I3XcEMA+G+t9SsASqn/AX6JUT8cjGla3eFPUEplYWQhr9Ja7wnWXv8O8NNo+8w0pqqqKhU4FWOO80gFSoQQQojJzoYxS+edysrKaIXIBhmXICFYsvSVsE0bMZKQolkNbA5LUrodo+raT0fYZ8apGDXQhRBCiOlkFcb0WFPGPXFRGcu2folgJbWgV5SxTOozwI+D5VZnM7i8aDVHqqRF22dGfaztFkIIIaaAmO5/EzG74fcYVcz+EPx5tta6RimVg5G38EPgBwlugw9gyZIlpKamjvRYU6qqqqisrBz5gdOUXJ+RyTWKTq7PyOQaRTedr4/b7Wb79u0Q4xD7uFZcVEr9GqOi2se11n6A0EpsWmsnxnr3ZwQfXo1RgzxkNkcWUom2TwghhBBxMG5BQnDJ00rgstDqbUqp/GCBGYLDDVdi1DoHoxzpqUqphcGfb8Sokz7SPiGEEELEwbgECcH137+HURv/TaXUVqXUo8CxwNtKqfcw6tn3Yww3EFxA5wvAOqXUXowV5X490j4hhBBCxMd4zW74gOGXiz0hyvMeBx6PdZ8QQgghxk5WgRRCCCFERBIkCCGEECIiCRKEEELELBAI8J079nL/iw0T3RSRQBIkCCGEiFlNs5sPDvbw6IZmXB6pbj9VSZAghBAiZpt2OQHocfl59b2OCW6NSBQJEoQQQsTsHe1kblkac0vTWLexlUAgMNFNEgkgQYIQQoiY9Lh8fHCwh9NUDmuWF7K3ro/dh/smulkiASRIEEIIEZN393Th88Opx2Zz7sn5pDusrNvYMtHNEgkgQYIQQoiYbNrlJCvdxnGzMslMs/Ghk/N5dVsHzh7vRDdNxJkECUIIIUzz+wO8o7tYtigbm80opLtmRSH93gDrq9omuHUi3iRIEEIIYdruw7109ng57dicgW3zytJZMjeTp99uxe+XBMapRIIEIYQQpm3SXVgtsGxR9qDta5YXUt/mYcverglqmUgECRKEEEKY9s4uJ8fOziA7Y/D6gCuX5JKXZWfdxtYJaplIBAkShBBCmNLm7GdvXd+goYYQh93KRcsK2LTLSWO7ZwJaJxJBggQhhBCmvKONKounqaODBIDVpxUC8Mwm6U2YKiRIEEIIYcom3UVRbgpzy9Ii7i/Nd3DasTk8t7mNfq9/nFsnEkGCBCGEECPyeP28u6eL047NwWKxDPu4NcsL6ej28sYHnePYOpEoEiQIIYQY0fYDPbg8/mGHGkJOWZhNWYGDp6ZQAqPXF6DH5cPl8eHp9+PzBabNWhX2kR8ihBBiutu0y4nDbuHEBVlRH2e1WlizvJC/PlPPgYY+5pWlj1MLE8Pl8XPjb3XEZEyrxXi9NitYLRZsVgtWa+Rtxr/mH2u3WbjqnBIWzsiYgFd9hAQJQgghRvSOdnLC/CzSHCN3QF9YWcA9zzfw1MZWvnrZzHFoXeI89kYzje0ePnFuCRlpNnz+AH6/UXnS5w/gDzCwzfg3gM8P/kAgyvaw5/vBFwjg9fkHbQfo6vVN8KuXIEEIIcQIDje7qWv1cNkZxaYen5Np56yleby0pZ3rV5eTkWpLcAsTo6Pby4OvNnH64hyuvbB8opszISQnQQghRFQDUx8j1EcYztoVhfR5/Ly0pT1RzUq4/3upEXe/n89eND0DBJAgQQghxAg27XIyuySV0nyH6eeoWRkcU5HOUxtbJ2WSX12Lm6febuHiZYXMKok85XM6kCBBCCHEsHpcPt4/0B1TLwKAxWJh7YpCDja6+OBgT4Jalzh3r6/HbrPyqfNKJ7opE0qCBCGEEMPasrcLnz+2oYaQs0/MJzPNylNvT67pkLuqe3jt/U6uWFVMQU7KRDdnQkmQIIQQYlibdjnJSrOxeHZmzM9Nc1g5/5QCNmzvpL2rPwGti79AIMBfn60nL8vOFWeZS9ScyiRIEEIIEZHfH2Cz7uKURdnYbMNXWYxm7YpCvL4Az21ui3PrEmPTri62H+jhUx8qnbSzMuJJggQhhBAR1bV6aO/2csrC6AWUoplZnMZJC7J4+u3Wgfn/ycrnC3DXs3XMKHRwcXCxqulOggQhhBARdXQbQwRFOeZnNUSydkUhzZ39vLPLGY9mJczz77ZR3eTmsxeXYx9lz8lUI0GCEEKIiJzBin+5mWPrdl9xXC6FOfakTmB0eXzc90IDx83OYOXxuRPdnKQhQYIQQoiInL1eALIzxlac12azsPrUQjbv7qKu1R2PpsXdo2+00Or0csPqiqirXE43EiQIIYSIKF49CQAXn1aI1QpPJ2FvQke3l4eC5ZePnxv7LI6pTIIEIYQQETl7vKTYLaSmjP1WUZiTwsrFuazf3Ia73x+H1sWPlF8engQJQgghInL2esnJsMet+33tiiK6+ny8tq0jLseLBym/HN24rAKplCoE7gUWAG5gL/BFrXWzUmoFcAeQDhwEPq21bgo+b1T7hBBCjF1Xr4+cjPjVCjhhfiazilN56u1WLqgsiNtxx+Lv6+tJsVv51PnTu/zycMarJyEA/LfWWmmtTwD2Ab9USlmA+4CvaK0XAa8BvwQY7T4hhBDx4ez1kpMZv8+SFouFNcsL0TW97KntjdtxR2tXdQ+vh8ovZ0/v8svDGZcgQWvdprV+JWzTRmAOsAxwaa03BLffDlwV/H60+4QQQsSBM849CQDnnVJAaoqVpzZObAJjePnly1dJ+eXhWMZ7CU+llBVYDzwB1ALXa63XhO3vBWYC545mn9Z6xNqfVVVVc4EDcXlBQggxRf3iCRtLZgb48CnxTTR8rMrKe9UWvrvGR/rY6jSN2q46C/e9aePSk30sX5DclSDjbF5lZeVBsw8el5yEIX4PdAN/AD46AecfsGTJElJTU+NyrKqqKiorK+NyrKlIrs/I5BpFJ9dnZPG8Rj5/ANfD21gwp5TKyvhm/eeV9/HV3++mxT+byyrH71N86Pr4fAHueE0zowi+cIWaFtUV3W4327dvj/l54zq7QSn1a2Ah8HGttR+oxhh2CO0vAgLB3oDR7hNCCDFGPX0+/AHimpMQsqAinWNnZbBuYyvj3ZsNRvnlGim/bMq4BQlKqZuBSuAyrXWo5FYVkK6UOjP4843Ag2PcJ4QQYoxChZRyxlhtcThrVxRS2+Jm677uhBx/OC6Pj3ufb2DxnAxWLs4Z13NPRuMSJCiljge+B1QAbyqltiqlHg32JlwD/EkptQc4G/gPgNHuE0IIMXahkszxTlwMWbU0j5wM27gnMD66oYW2Li/XXyzll80Yl5wErfUHQMTfhtb6TWBpPPcJIYQYm3it2zAcR4qVC5cV8MiGZlo6+ynKTfwUxG4XPPRaEyuPz5XyyyZJxUUhhBBHcfbEb92G4VyyvJBAAJ59Z3x6E17ZaQ2WXy4bl/NNBRIkCCGEOEqiexIAygtSqVyYzTObWvH6EpvAWNviZtN+CxefWsjMYim/bJYECUIIIY7i7PVit1nISE3sbWLtiiLaury8taMzoee5e309dit86jwpvxwLCRKEEEIcpavXR3aGLeHJfctUNiV5KQlNYAyVXz5T+aX8cowkSBBCCHGUzh5vwqY/hrNZjfUc3tvfTXWTK+7HDwQC/PWZevKz7JyxaFpVVowLCRKEEEIcJd4rQEZz4bIC7DYLT78d/96Et3c52X6wh0+dX0rqRNQYnuQkSBBCCHEUZ+/49CQA5GWlcOaSXJ6vasPl8cXtuD5fgLuerWdmcSoXLSuM23GnEwkShBBCHMXZ6yMngdMfh1q7oohet5+Xt3bE7Zjrq4zyy9ddJOWXR0uCBCGEEIMEAoFx7UkAWDwng3llaazb2BKX9RxcHh/3viDll8dKggQhhBCD9Lj8+P2JW7chEovFwtoVReyvd7GrpnfMx3t0QwvtXV5uWC3ll8di2HeAUuqnZg6gtf7P+DVHCCHERDtSSGn8hhsAzj0pj788U8dTG1s5bvboyyZ3dPfz0KtG+eXFc6T88lhECxNnjVsrhBBCJA1njxEk5CZgmeho0lNtnH9yPs+808bn11SM+vwPvNiI2yvll+Nh2N+A1vqz49kQIYQQySG0TPR49yQArFlRxJMbW1m/uY2PnV0S8/NrW9w8vamV1VJ+OS5iCtOUUtlAEWErOmqt98e7UUIIISZOaLghdxxzEkLmlKaxdF4mT7/dyhWrirFaY8sn+Ptz9aTYrVJ+OU5MJS4qpRYrpbYAncDe4Nee4JcQQogppCvUkzCOUyDDrV1RREO7h6o9XTE9b2d1Dxu2d3LlqmLypfxyXJid3XAb8DJQADiBfOAO4DMJapcQQogJ0tnjxWqFzNSJCRJOX5xDfpaddTGu5/D354zyy5evKk5Qy6Yfs0HCicBNWusOwKK17gT+HfhZwlomhBBiQnT1+shOt8fc1R8vKXYrF59awDvaSWO7x9Rz3tvXzbb9PXz8nBLSJyi4mYrMBgkuINR306KUmh18rtS5FEKIGDzyejO/+sehiW5GVJ293nGtthjJ6tMKsYCp9RwCgQD3vdBAYY6d1afJbSmezAYJrwNXBb//F/AM8CrwUiIaJYQQU9WmXU42745trH28dY1ztcVIivMcLD8uh2c3t+Lx+qM+duu+brYf7OGqs0txpEiNwHgy9S7QWl8V9uP3gA+ALOCeRDRKCCGmqtpWN919Pvq9flLsyXlDc/b4KC90THQzWLO8iLd2OHljeyfnnpQf8TFHehFSuPjUgnFu4dQXc6iotfYD9yagLUIIMaW5PH5aOvsBoxZBYU6SBgm9XtSsjIluBicfk0VFoYN1G1uGDRK27O1mx6FevvKRGdKLkACmggSl1L1AxBU3tNbXxrVFQggxRdW3uge+7+zxUpiTfNP0jMWdxncFyOFYrRbWLC/izqfr2F/fx/zy9EH7A4EA9z7fQHFuChcuk16ERDAbdu0F9oV99QCrgbYEtUsIIaac2rAgoaPbO4EtGV6fx4/XF5jwnISQCyrzcdgtPBVhOmTV7i521fRy9bmlOJJ06GayM5uT8JOh25RSfwV+FPcWCSHEFFXXkvxBQtcElmSOJDvDztkn5vHS1nauX11OZprRLiMXoZGSvBQuqIw8FCHGbiyh11bg7Hg1RAghprraVg+pwXHzzp7kDBJC7ZqIkszDWbO8CJfHz0tb2ge2vaO70IeNXoRkTQCdCszmJHxoyKYM4GpgR9xbJIQQU1Rdi5sFFWnomt6kDRK6BpaJTp4gQc3KYOGMdNZtbGHtCqMOwn0vNFCa7+CCSslFSCSz74K/Dvm5B6Mn4RPxbY4QQkxdta1uli3KpqHNk7TDDZ09xnBDbhIkLoZbu6KI3zxcw/sHeuh1+9hT28c3r5iJ3TYxVSGnC7M5CfMS3RAhhJjKet0+2ru8zChKZV9dn/QkxOisE/K48+k61m1sob7VQ1mBg/NOll6ERBv2XaCUMjXIE6ybIIQQIoq64MyGisJUcjPtyduT0OvDYoGs9OTqSUhzWLmwsoBHNjQD8K0rZ0kvwjiIFgh4gX4TX0IIIUZQ12IsVDSjKBgkJHFPQlaaDdsELe4UzSXLjXyEikIHHxqmuJKIr2j9SeFDDGuAK4H/Ag4Bc4CbgIcT1zQhhJg6agd6EhzkZdmTdrjB2eMlJzO5hhpCZhSlcuOlFSwoT8cmvQjjYth3gtZ6YJkypdS3gGXBpaIBdiulNgObgT8ltolCCDH51bW4Kcyxk+awkZdlp8/tx93vH5gSmSycvT5ykqRGQiQfWVk80U2YVsy+O3Mxpj2GywhuF0IIMYK6VjcVhakA5AY/qSdjb4KzN3l7EsT4M/tOuBt4QSn1W6AGmAV8PbhdCCHECGpbPJy+OAeAvLAgoSRv4ldbDOfs9bGgInl7EsT4MhskfBdj/YaPAxVAPfAH4E6zJ1JK/Rq4ApgLLNVabw9uPwi4gl8AN2mtnwvuWwHcAaQDB4FPa62bRtonhBDJpMflo7PHe6QnIcv405uMMxycPd6kWbdBTDyzdRL8wO3Br9F6DLgVeD3CvitDQUOIUsoC3Adcp7XeoJT6AfBL4Ppo+8bQPiGESIja4JoNFUWDhxuSLUhwefx4vIGkzkkQ4ytanYRrtNb3Br8f9uartb7LzIm01huCxzLbtmWAK/Q8jADlIEYgEG2fEEIkldDCTjMKjaGFvKzkzEkIFVKSnAQREu2d8Ang3uD31wzzmABgKkgYwf3B3oENwPeCsyhmY0y3BEBr3aKUsiqlCqLt01rL8tVCiKQSmv5YHhxuSHdYcdgtSRckdCZptUUxcaJNgbwk7PtzE9iGVVrrGqVUKvBbjFyHTyfwfAO2b98+8oNiUFVVFdfjTTVyfUYm1yi6yXp93t9tJTfdwvZtWwa2pafY2Heokaqq+rieayzXaG+jBbDRULOXKteID5+UJut7aKKYXQWyGOjTWncrpWzAtRgVGe8fa1lmrXVN8F+3Uuo24IngrmqMok2hNhQBAa11m1Jq2H2xnHvJkiWkpqaOpfkDqqqqqKysjMuxpiK5PiOTaxTdZL4+927cw9wKK5WVCwa2lby1G3uancrK+XE7z1ivUdd77UA1y04+njmlaXFrV7KYzO+hsXK73aP6YGy2TsI6YGHw+18A3wG+DdwS8xnDKKUylVK5we8tGMtPbw3urgLSlVJnBn++EXjQxD4hhEgqta1uZhQN/kCSjOs3dPUm5wqQYuKYHXhaxJGb96eAlUA38AHwb2YOoJT6HXA5UIZRc6EVuBR4ONg7YQN2AF8GY0aFUuoa4A6lVBrBaY4j7RNCiGTi7PHS3edjRuHRQUJ1U3L16TuDORLZ6ZKTIAxm3wk+wKGUWgR0aq2rg6tEZpk9kdb66xgFmIY6Ocpz3gSWxrpPCCGSxcCaDUWDiyaF1m8IBAJYLMmxDoGz12cs7iTrIoggs0HCMxjd+YXAP4LbFgO1iWiUEEJMFUemPw7uScjLsuPuD+Dy+ElPTY7ufWevl2ypkSDCmA0SPgd8BmNp6NC0yCLgxwlokxBCTBm1rR6sFigrGNyTMFBQqcebVEGCVFsU4cxWXHQDfw4OMZQC9VrrVxLZMCGEmArqWtyU5DlIsQ/OEx9Y5KnbS3lBfGZZjZWzx0d+tgQJ4ghTsxuUUnlKqQcw1lfYG9z2YaXUzxPZOCGEmOzqWt1H5SPA4J6EZCE9CWIos1Mgbwc6MWoTeILb3sJY8EkIIUQEgUCA2hb3UfkIEFaaOYmmQTp7fbJugxjEbJBwHvB1rXU9RilmtNbNQEmiGiaEEJNdZ4+XXrd/YGGncLmZybV+g6ffj8vjJ1vWbRBhzAYJnRiJigOUUrMxlowWQggRQW2L0fFaEaEnIc1hJc1hTZqCSs5QISUZbhBhzAYJf8EoenQuYFVKnQ7czdiWjhZCiCmtLlgjYWi1xZC8TDudPb7xbNKwugYWd5LhBnGE2ZDxVxhJi38EUjBWfrwDuDVB7RJCiEmvtsWN1Qql+UcnLgLkZtnp6Okf51ZFNtCTIMMNIsyI74ZgyeTPAH/SWv828U0SQoipoa7VTVm+A/swFQxzM+20dHoi7htvTulJEBGMONygtfYB/xuslSCEEMKk2hZ3xHyEEKM0c3IMN4SCBMlJEOHM5iQ8qZS6NKEtEUKIKSQQCFDX6hk2HwFCOQnG+g0TzRkMVqQnQYQzGzKmAf9SSr0F1BCcBgmgtb42EQ0TQojJrL3Li8vjj9qTkJtpx+sL0OPyk5U+sTdnZ6+X9FTrUZUhxfRmNkjYHvwSQghhwuHQwk4Rqi2GDBRU6vFOfJDQ45Nqi+IoZtdu+EmiGyKEEFNJ3cAS0dF7EgA6ur1RhyXGg1GSWYYaxGDSrySEEAlQ2+LGbrNQkjt8T0JulnFTTob1G2TdBhGJBAlCCJEAda0eygsc2IaZ/giQl5kCmCvNHAgEeP39DlyexMyGcPb6yMmUngQxmAQJQgiRAHUt7qhDDQC5wZuymUWeDja6+MUDh/jHy01xad9Qzh7pSRBHkyBBCCHjROoAAAAgAElEQVTizO8PUNcaefXHcCl2K5lp5tZvONjgAuCZTa24+/1xaWeI1xeg1+2XIEEcxVSQoJSyKKU+r5R6SSm1LbjtLKXUVYltnhBCjM36zW1s2989rudsdfbj8QaoiDKzISRUK2EkhxqNIMHZ6+Plre1jbmO4UCElSVwUQ5ntSfgpcAPwZ2B2cNth4KZENEoIIeLl78/V8/DrzeN6ztrQzIYRehIgtH7DyEFCdZOLWSWpzCtL4/E3WuJagKkruG5DjqzbIIYwGyRcB6zVWv+DI4WUDgDzE9EoIYSIB58/QGePl8b28V0foS64RLSZaY25mXZTOQmHGl3MKUnjsjOKOdjo4r049o6EejKk2qIYymyQYANC78hQkJAVtk0IIZKOs8eLPwANbZ5xLX1c2+rGYbdQlJMy4mON9RuiBwnufj/1bR7mlKZxzol55GbaeeyNlng1d2CZaMlJEEOZDRKeBv5XKZUKRo4C8DPgyUQ1TAghxqo9+And3e83Ne4fL3UtbsoLU7Fah5/+GBLKSfD7hw9iappdBAIwpzQNR4qVS5YXsmmXk7qW+Ky75xwYbpCeBDGY2SDhW0AF0AnkYvQgzEFyEoQQSay9q3/g+8b2/iiPjK/aVjcVhSMnLYIx3OAPQFff8PUPqhuNYGB2SRoAa5cXYrNaeOKt+PQmOKUnQQzDVJCgtXZqrS/DCAxWAAu01h/VWncltHVCCDEG7WFj/Q3t47Pavc8XoL7Vw8xic2WWc8PWbxjOoUYXdptlIMehICeFs5bmsb6qjR7X2IsrOXt9pKZYSU2RWfFiMLNTIK1KKSvQDFQBTcGfhRAiabV3HbnxNraNT/JifZsHry/ArOI0U48PLfIUrVbCoSYXM4pSsYdVb/zIGUX0uf08X9U2tgYTKqQkQw3iaGb7lryELQ8dopTyAnXAI8CPtNaSyCiESBod3V7jE7LDQsM4zXA43GzUM5hltich01xPwqIZ6YO2LZqZweI5GTz+ZguXnl6EzUT+w3CMkswy1CCOZrY34GvAS8CFwHHARcCLwHeBLwErgd8mooFCCDFabV395GfbKct30DBOPQnVzcawxqwSkz0JoSBhmJ4El8dPY7uH2aVHH++yM4ppaPPwzi7nKFtrkJ4EMRyzoeO3gFO01p3Bn3crpTYDVVrrBUqp9zGGIYQQImm0d3spyLZTlJPC3rq+cTnn4WYXBdl2MtPM3XRzMuxYLMOvBBk+s2GolYtzKc5N4bE3W1ixOHfUbe7q9VKanzHq54upy2xPQg4w9B2UgTHTAaABSEcIIZJIR7eXvKwUSgscNHX044syzTBeaprcpvMRAGw2C9nptmFzEkLlmOdE6Jmw2SxcenoR7+3r5kD96IOgzl6f9CSIiMwGCfcAzwfXb7hYKfU54Dng7uD+CwGdiAYKIcRotXf1k59lpyw/Fa8vQJszsdMgA4EANc0u0zMbQnKjFFQKzWwYrsTzxacWkJpi4fE3Rzcd0ucL0OOSnAQRmdl3xb8De4CrMeol1AN/BO4M7n8ZeCXejRNCiNHy+gI4e33kZ9kpzTdqFjS0eyjOM1e/YDTau730uPzMKoktSMjLHH79hupGI+iw2SInJmZn2Dnv5AKef7eNU8pjbjJdfT4CASnJLCIzFSRorf3A7cGvSPtd8WyUEEKMVaj7Pj87hbKCYJDQ5mHpvMSds6YpmLQYw3ADGDMcQsMKQx1qcnPsrOj5Ah9eWcTTm1p5Z7+Fs1bGdOqBksy5UkhJRGD6XaGUKgVOA4qAgZBWa32Xief+GrgCmAss1VpvD25fhDFkUQi0AtdqrfeMZZ8QQgC0dxtDC/nZdorzUrBYSPhCTzWh6Y+x9iRk2dm2/+iehD63j8Z2DxctK4j6/DmlaSyek8HOup6YzgvQ2Rta3EmCBHE0s8WULgP2YSwZfQfGlMg7gGtMnucx4Czg0JDttwN/1Fovwhi+uCMO+4QQYqCQUl6WHYfdSmFOSsKnQR5udpPusJpa2ClcbqYdZ68Pn29wYmV1sGci0vTHoZYtyqG23UJHd2x5F6FlonNl3QYRgdnExZ8Dn9Vanwz0BP/9AianPWqtN2ita8K3KaVKgFOA/wtu+j/gFKVU8Wj3mXwtQohpINSTUJBt3LDL8h0J70mobjLyByyW2AobDdRK6B3cm1DdNPzMhqEqF2UD8O6e2GratQaTOSVxUURiNkiYrbV+aMi2u4Frx3DuWUCt1toHEPy3Lrh9tPuEEAIY3JMAUJrvSHjVxcPNsU1/DBlu/YZDjS5S7BbKTSwWdUxFOhmOAFW7Y1tS5909XRTlplCcG1vvh5gezIaOTUqpUq11I3BQKXU60AJM6v6p7du3x/V4VVVSTyoauT4jk2sUXSzXR++3kmq3sH3bFgD8LiutnRbe3lSFPQF/udxeaO60Y+1voaqqOabnNjYD2Hnn3Z20lR4Zcnh/j5XCTAtbt7xr6jgLy6y8vbONdzY3Y6ZKs8cLm7WNyrkB3n3X3DkmO/k/FhuzQcKdwJnAw8BvMKY8+oFbxnDuGmCGUsqmtfYppWwY0ytrMBIjR7MvJkuWLCE1NbYEo+FUVVVRWVkZl2NNRXJ9RibXKLpYr896fZCiPBeVlccC0EYbL++sYeb8JQOrKcbTntpeYA8rTppP5ZK8mJ5b3OTir69qSirmUXlS/sD2372wg8VzM6msnGPqOFsPvct71Vbyyo9j4YyRKyi+8UEn/b6DfPicBZx8THZMbZ6MpvP/MbfbPaoPxmaHG/5Ha/0wgNb6HmARUKm1/mHMZwzSWjcBW4FPBDd9AtiitW4e7b7RtkUIMVh7Vz97a3snuhlj0tbtJT/7yOeg8GmQiRCa/jhzFMMNeREWeep1+2jq6I9Yjnk4xwR7ITabHHJ464NOstJtLJ2XFUNrxXQyYpAQ/KTeo5QaCL211tVa651mT6KU+p1S6jAwE3hBKfVBcNeNwNeUUrsxZkzcGPa00e4TQoxBv9fP9+7az7du3xtzpnwyae/ykp8VFiQECyolKnmxptmF1QoVJvIHhspKt2G1Dl6/oTpUjjmGICErDRbOSDeVl+D1BXh7l5Plx+YMWoJaiHAjDjcEu/R3Y9QkqBvNSbTWXwe+HmH7LmD5MM8Z1T4hxBGHGl1UFDpIsZvtNIR/vdbMwQbjBvXkW61cc0FZopqXUB3d/eRnHelCL8hJwW6z0NDuTsj5aprdlBekxnStQ6xWC7kZ9kHrNxyKYWZDuMpF2Tz4ahPdfT6y0odPvnj/QDfdfT5OP370C0OJqc9sTsL9wDql1K3AYWAgs0Zr/VIiGiaEGJuNOzr5yb0HOXNJLt/75BxT0/Kqm1w88FIjZ52Qh6ffz7q3W7jqnBJSU2K/8U0kd7+fHpd/0HCDzWqhJC+FxoQNN7iYFeOaDeGGrt9Q3ejGYbdQWhBbz0Tlomz+8XITW/d1cWaU3Ii3dnSSmmKhcuHUz0UQo2f2f/6XgHzgx8BfgL8Gv/6SmGYJIcai1dnPbx6uISvdxobtnTz02sgpO35/gFsfqSHdYeVLl1Zw+apinD0+Xny3fRxaHF8DJZmzBn8OStQ0SJ8vQF2rJ+ZKi+FyMwcHCYcaXcwqScNmZppCmONmZZKZZo065BAIBHhrh5NTFmaT5phcAaAYX2bXbkhgtXMhRDz5/QFueagad7+f3311Efe/2Mjdz9WzoDx9oOBOJOs2trLjUC/f/tgs8rJSyM20s3BGOo9saObiUwuwxnizmkjtXaGSzIPn/pcVOHjjg864n6+h3YPXFxhVjYSQvEx7cIaE4VCTixNGkVBos1k46ZhsNu/uIhAIROxB2n24j5bOfq6dpENJYvyYDiGVUilKqVVKqY8Hf85USmUmrmlCiNF49I1mtuzt5gtrZzC7JI1/u2Ims0vT+OU/DlHfFnk8vrHdw9+eq6dyYTbnnWxMwbNYLFy+qpjaFjebtHM8X8KYtQ8s7jT4c1BZvgNnj48+ty+u56sJ5g+MZbghL2y4ocflo6WznzmlozveskXZtHT2D5R1HuqtHZ1YrbDiuJxRt1dMD2bXblgK7Maol/DX4OazgREXdxJCjJ+9tb38/bkGVh6fy+pTjUWB0hw2fvjpuRCAn917EJdn8A0yEAjwh8cOA/C1j84c9Mlz1ZI8SvJSePj1yTXDeGi1xZDQ+H68hxxqmkc//TEkN9NOj8uPx+sfmNlgZs2GSEJ5Bpt3Rw7u3tzRydJ5WbKokxiR2Z6EPwH/qbU+FgjNiXoVo8CSECIJuDw+fvXPanIz7XxjyM2+ojCV7149m4ONLm595DCBwJGqfi9taWfz7i6uu6iM0vzBSXI2m4XLzihm+4Eedh+ePHUTQus25GUO7UkwPpnHO3mxptlFfrY96myCkYRKMzt7fEdmNowySCjOczCnNC1iXsLhZhc1TW5OXyy9CGJkZoOE44H7gt8HALTWPUB6IholhIjdHevqqG1x852rZkVcrOdUlcM1F5TxynsdPPZGC2BME7xjXR3Hzc5g7YqiiMe9cFkBGalWHplEvQntXV5yMmxHTUcMBUFx70locjN7DL0IcCSg6ejp51Cji9QUK6V5sddcCFm2KJv3D/Qc1XP05g6jd2HlYpn6KEZmNkg4CAyqZamUOg3YG+8GCSFi98YHnTz7ThtXnlXMSQuGT078+NklrDw+l788U8fWfV3c/mQdfR4/37h81rBZ9JlpNlafVsjr2zto6kjsAknx0tHtPWqoAYzlkNMc1rhWXQwEAhxudjNzDPkIYAw3AHR2eznU6GJ2SeqYkkUrF2Xj9QXYtr9n0PY3P+hk4Yx0iscQgIjpw2yQ8EPgKaXUTwCHUur/AQ8BP0hYy4QQpjR3erj14RoWzkjnmvOjZ6tbrRa+deUsZhSm8tN7DvLqtg4+cW7piN3aH1lZhAV4PNgDkezau/uPmtkARjJmvJeMbu/20u3yjWn6IxzJn+js8VHd5GJ2jEWUhjp+biapKdZBeQktnf3oml4poCRMMxUkaK3XAauBYoxchDnA5Vrr9QlsmxBiGD5/gK5eL3Wtbm55sIZ+X4Cbrp5jqtpfZpqN/7xmLlhgblkaHzu7eMTnFOc5WLU0j2feaaXHFd+ZAYnQNqQkc7jSgvjWSjgcTFocy/RHOBIkHG5x0+r0jjofIcRht3LigqxBeQkbdxrTP2WoQZhlKrVVKVWktX4X+HKC2yOEGOKVre08t7mNrj4f3X0+uvu89Lj8gx7zzStmxrSy4cziNP70DUVGmtV0GeErVhXzynsdPLOplSvPKonpNYy3ju7hg4SyfAfb9nUPW0MgVtVxmP4IkJFqxW6z8P7+bmD0SYvhli3KZtMuJ3UtbiqKUnnzg05mFDqYPcZeDzF9mJ3/Uq2UegWjPPNjwaRFIUQCBQIB/vFKE/esb2BmcSoVBQ7mlKSRlW4b+MrOsFFW4OD4ObGXLBk6k2Ekx8zI4MT5WTz+ZguXnVGctIsC9bl9uDz+iMMNYLzuPo8fZ69vIA9gLA43u0lzWCnKjXw+sywWC7mZdnbVGLNI4hEkhIpnbd7dxbmZNrbt7+ajZxbHJTgS04PZ/yGzgaswyjPfrpRaBzwAPKO19kZ9phAiZj5fgNueqOXpTa186KR8vnnFzFEtHBRvl68q5kd3H+D19zs496T8iW5ORKEaCUMLKYWELxkdjyChptlYsyEeN968LDutzn7SHFaKxxh0gDH1taLQQdVuJ1npNnx+WCn5CCIGZnMSWrTWt2mtz8SYDvkecDNQn8jGCTEduTx+fn7/QZ7e1MpVZ5fwnatmJUWAAEb39aziVP71WhPNHZ5B9RaSRfsw6zaExHvJ6Jom95jzEUJyM406C2Od2RCuclE27+3v4dVtHRRk21EzM+JyXDE9jCaMLg1+FQEd8W2OENNbZ4+XH999AH24ly9/eAaXnh65dsFEsVotXHVOCbc8VMO1v9pJToaNBRXpxle58W9FUWrMixLFU6iQ0rDDDXGsutjn9tHc2c/MOI3x52UabY7HUEPIskU5PPlWK5t2OblkeeGkWoNDTDyziYuLgU8AnwTSgAeBy7TWmxLYNiGmlbZu+Paf9tDc2c/3PzWXM5K0W/j8UwqYXZLG7ppe9tX3sbe2j8feaMHrM3oV0h1WPnZ2CVedUzIhwcLAcMMwPQkZqTZyMmxxqbp4uCU0syFOQUKwzWOd/hjuhPmZ2G0WvL6AzGoQMTPbk/AG8DDwBeAlrXXy9TEKMYntqe3lzy/bwOrjFzcs4Pi5yb122qKZGSwK67b2+gLUNLnYW9fHxp1O7nm+gS17u/j3j8+mOHfsRXt8/gA9kdcqOkp7txerhYhVJ0OMJaNNHjCKmqb4TH8MCQ03xLMnIc1hY+m8THYf7uWE+cn9vhLJx2yQUKq1nhyl1oSYRAKBAE++1cpfnqkj0wG/+uIxcf0UOV7sNgvzytOZV57O+afk88K77dz2RC1fvnU337xi1qh6RQKBALsP9/HKe+28tq2Dzm4bxy72UDJCpcD2rn5yM+1RezHKChzsr3fF3KahDje7sFqhojA+1QtnFKVit1lYUB7fivdf/vBMOnu8SZPbIiYPU0GC1tqjlCoFTsPIRbCE7ZOVIIUYhY7ufn7zrxo26S5OVdmct7B9UgYIQ1ksFi6oLGDxnEx+9Y9D/Py+g1yyvJAvrKkgNWXkm9ShRhevvNfOq+91UN/mwW6zcOzsDNq6vOia3pGDhG7vsDMbQkrzHby1w4nfHxjTGH11s5vygtS43XxPX5zL3TcdR8Ew+RSjNbM4dcxlo8X0ZDYn4TKMBZ72YMxu+ABYAmxAlosWImZVu53c8lAN3S4fN15awYdPL+Ldd9snullxNaMolVtuPIa71zfw8OvNfHCwh5uuns28MuNTsrvfT0Obh/o2Nw1tHhraPGzb382BBhdWC5y4IIuPn1vCGcfn4Uix8NH/3Ma+uj5WLc2Let6OrsjrNoQrK3Dg9QVo6/KOqb7B4WZ33PIRwEgMjXeAIMRYmB1u+DnwWa31Q0qpdq31yUqpz2IEDEIIkzxeP39/tp5H32hhbmkaN98wf+CmORWl2K187pIKTj4mm1sequYbf9zDwhnpNLR5aOsaXGIl3WFlXnkaN15awaqleUfdLEtyYF9d34jnbO/uZ2ZxVtTHlA2sBukedZDg8wWobXFzqhp+QS0hJjvTxZS01g8N2XY30AB8J75NEiJ5tXT2s7++j1MWZsdccbC6ycWv/nGI/fUuLj29kBtWm+t+nwoqF2Vz2zcWcce6Olqc/VQuyqa8IJXyAgdlBQ7KClLJzbRFLUhUnhdgX330ICEQMHoHRh5uMD79N7Z5WDI35pcDGFMovb7AlBgiEmI4ZoOEJqVUqda6ETiolDodaAFsiWuaEMnF7w9w8/0H2VXTS1FuCpeeXsTqUwvIzhj+v5HPH2Dr3i6efaeNjTudZKRZ+fG181h+XM44tjw55GWlcNPVc0b9/PK8AFsOeWnr6h+2S77H5cfrC5CfFb13oCQ/BYtlbLUSapqNxEcZ6xdTmdkg4U7gTIxpkL8BXgb8wC0JapcQSefFLe3squnlo2cUsb/Bxd+ereeBFxs475QCLltZxKywT5TNHR7Wb25jfVUbTR395GTYuHRFIVeeXSJjzqNUnmfMvN5X10eBinwN27tChZSi/2lz2K0UZKeMqepivKc/CpGMzM5u+FXY9/cEF3vK1FrvTFTDhEgmPS4fdz1bz7GzMvjcJRVYrRYO1Pfx+JstPF/VxtNvt7JsUTanL87lrR2dVO3pIhCAk4/J4nOXVLD8uBwcMv1sTMqD+Yr76vo4VUXuiRkoyTxCkABG8mLDGAoq1TS7yM+2k5UuHapi6hrV6iZa6+p4N0SIZHbfCw109nj56XXzBqbMzStP55tXzOK6i8p5ZlMrT25sYfPuLgpzUrj63FIurMynrEC6ouMlLQXKCxxRkxcHSjKPMNwARvLi+we6R90eY80G+f2KqW3sS6AJMcUdanTxxFstXHxqAQtnHL04Tl6WnU98qJQrzyqmusnN3LK0CV27YCqbX5EePUgYoSRzuNJ8By9v7cfrC8SchBoIBDjc7ObsE6NPxxRispP+TyGiCAQC/OmJWjJSbXzmwvKoj02xW1lQkS4BQgItqEinvs1Dj8sXcX9Htxe7zWJqCKCswIE/AE0d5occAoEAuqaX3z16mG6XT3oSxJQnPQlCRLFheyfv7e/mKx+eQW6UtQDE+AiVK95f38fSeUfXQmgLlmQ2U0WxNGzJ6IrC6Df7ju5+XtzSzvrNbVQ3uUlNsXDeyflcsKxgFK9CiMlD/uoJMQyXx8efn6pjfnkaq5cXTnRzBHBMhREk7KuLHCS0d3kpMJG0CEZPAjBs8qLPF+Cd3U7Wb25j0y4nPj8cOyuDr390JmedkEdmmiQsiqlPggQhhvHPV5po6eznpqtnyxBCkijISSE/yz5sXkJHt/kgoTAnBbvNctQ0yJomF+ur2nhxSzvtwRLPl51RzAWVBXFdnVGIyUCCBCEiqGtx86/Xmjn3pDyWzI1e4leMrwVRkhfbu/uZX2GuzLXNaqE4N4WGNg+9bh+vb+tgfVUbOw71YrXCaSqHC5cVcKrKiTmxUYipQoIEMS119/mwWBi2y/iOp+pIsVm4YXXFOLdMjGRBRTpb9nbh8foH1Z7w+wPGCpAmZjaElBU42LzbySdv3oG738+s4lRuWF3Oh07Ol6JXQiBBgpiG3j/QzQ/u2o/HGyAnw0Z5obGGQOjfPrefTbuc3LC6nMIcuVEkmwUV6fj8UN3o4piwKaldfT78fnOFlEKOnZ3Jrupezj0pjwsrCzh2dkbU9SOEmG6SIkhQSh0EXMEvgJu01s8ppVYAdwDpwEHg01rrpuBzht0nxHDq29z8/L6DFOc5uGhZAfVtbupbPeys7uW1bR34jcq/zCxO5SMriya2sSKi0AyHfXV9g4KEgZLMJgophVx7QRmfPq/U1GwIIaajpAgSgq7UWm8P/aCUsgD3AddprTcopX4A/BK4Ptq+iWi4mBx6XD5+dPcB/AH4yWfmMaNo8LS3fq+fxvZ+GtrczC5NI0XKKCelsgIH6anWo/ISYinJHE4CBCGGl8x/BZcBLq31huDPtwNXmdgnxFF8vgD/9cAh6lrc/OBTc48KEMAohjSzOJVlKoeSPMcEtFKYYbVamF+eftSy0UeqLcoQkRDxkkxBwv1KqW1KqduUUnnAbOBQaKfWugWwKqUKRtgnxFHufLqOqj1dfOUjMzlxgcxWmOwWlKezv96FLzQ+RNi6DTH2JAghhpcs/5tWaa1rlFKpwG+BPwCPJvqk27dvH/lBMaiqqorr8aaaibo+m/ZbeOJdGyuP8VNiO0hV1cEJaYcZ8h6KLnR9bP0WXB4bz7/2LsXZxr5de63YrRZ2bt/KdM49lPdQdHJ9YpMUQYLWuib4r1spdRvwBHArMCf0GKVUERDQWrcppaqH2xfLeZcsWUJqanxqr1dVVVFZWRmXY01FE3V9tu7rYt3W/ZyqsvnetfOSuiiSvIeiC78+eeV9PLp5N2n586k8MR+Al/dVU5jbw7Jlx01kMyeUvIeim87Xx+12j+qD8YQPNyilMpVSucHvLcDVwFagCkhXSp0ZfOiNwIPB76PtEwKAw81ubr7vEDOLUrnp6jlJHSCI2MwuScVuswxKXmzr6icvhhoJQoiRJcP/qFLgYaWUDbABO4Ava639SqlrgDuUUmkEpzkCRNsnBMDe2l5++X+HsNqMmQxSZ39qSbFbmVOaNihIaO/2Ul4gCadCxNOEBwla6/3AycPsexNYGus+MT31uHy8vLWdZ99pY19dH2kOKz/77DzKCmQ536loQUU6G3d2EggEsFgsdHR7WTw7c6KbJcSUMuFBghBjEQgE2HGol+c2t/Latk7c/X7ml6fx5Q/P4JyT8shOl7f4VDW/PI31m9tocfZTkJVCZ49XhhuEiDP5HyUmrardTv78VB3VTW7SHVbOPSmPi08tZNHMdCmtOw0sCFs22jbDQiAg0x+FiDf5HyUmpY07O7n5/kOUFzr45uUzOeuEPNJTJe9gOplfno7FAvvrXBTnGrkI+bIokxBxJUGCmHRCAcL88jRuvn4BWekSHExHGak2Kgoc7KvvY9FMo1chlhUghRAjm/ApkELEQgIEEW5BRTr76vpGvW6DECI6CRLEpLFxhwQIYrAFFek0tnuoaTYWkJWeBCHiS4KEOOj3+unoZVAdeRFfG3d0cvMDh1hQni4BghgQSl6s2t1FusNKmkPeF0LEk4TdcXDrI4d5cYud3z73PsW5KZQVOCjNd1BWYHwtKE9nZnHaRDdz0goPEH5+/XwJEMSAUJCwv95FRaEUUhIi3iRIiIPrLiony9JCanY5De0eGts8vLmjE2ePb+AxFy0r4LqLysiTZWxN63H5eP39Dv74eK0ECCKivKwUCnPstDqlRoIQiSD/q+KgKDeFU+cHqKwsH7S9z+2jod3DS1vaeXRDMxu2d3DN+WWsXVGEzSbz+Ify+wPsb+ijancXm3UXO6p78PtBzcrg55+VAEFENr88nVZnl0x/FCIBJEhIoPRUG/PK0rlhdToXVhbwpydruX1dHc++08aXPjyDE+ZnTXQTJ1wgEODNHU427uikanfXQJb6gop0PnZWCcsWZXPc7EwJqsSwFlSk847uokB6EoSIO/lfNU5mlaRx8/XzeXOHkz+vq+WmO/dx1gl5fO6S8oFCMNPNwYY+/vB4LR8c7CE73cYpC7NZprI5ZWE2BfKpUJgUykuQoTwh4k+ChHFksVg44/hcli3K5qFXm3jo1Sbe3unkc5eUs2Z54bQpJezy+Lj/xUYe3dBMRqqNb14+k/MrC2QpZzEqi2ZmYLVCuSQuChF3EiRMgNQUK58+v4wLKgv43aM1/PHxWjbtcvLNK2ZN+U/QG3d08qcna2nq6OfCZQVcf3E5uZnyNhSjV976wEsAAAx1SURBVJLn4M//dixl+RIkCBFv8td5ApXmO/jZdfN5cmMLdz1Tz5du1Xzjo7NYeXzuRDct7tp74Cf3HGDjTidzStP4ny/OZslcyckQ8TGjSJYDFyIRJEiYYFarhY+sLObkY7L5739W87P7DnLhsgK+uLaCjCmwYNHhZhdPvNXCs5tsWK3d3LC6nMvOKMYuiYhCCJH0JEhIErNL0vjNl47h/hcbeejVJrbt6+Y7V83m+LmZE920mPn9Aar2dPHEmy1s3t2F3WZh6cwA37xaUZInXcJCCDFZSJCQRFLsVq67qJxlKptbHqzhu3/ey0fPLObT55eR5kj+Cto9Lh8vvNvGk2+2UNvqoSDbzrUXlLH6tAL26W0SIAghxCQjQUISWjI3iz98fRF/ebqOh19v5o3tnXz1splULsqe6KZF5O738+ArTTz6RjN9bj/Hzc7g0xeUccbxuaTYkz+4EUIIEZkECUkqM83GNy6fxbkn5fO7Rw/zg7/t50Mn5fP5NRVJVX727Z1Obn+yloZ2D6uW5nLFqhLUrIyJbpYQQog4SJ67jYjohPlZ3Pb1RfzjFaOuwubdTj6/poLzTs6f0LoKDW1ubl9Xx9s7ncwuSeVXn18gFSSFEGKKkSBhEnCkWLn2gjLOPiGPWx+p4ZaHanhpSztfvWwmFYXjO/XL0+/nX681889XGrFaLTJbQQghpjAJEiaROaVp/PqLx/D0plbueraer/xuN19YU8HFpxYkvFchEAiwaVcXf36qlrpWD2ctzeVzayqmbUlpIYSYDiRImGSsVgtrVxSx/Ngc/vdfNfzu0cNs3OnkG5fPTFi1xm37u7l7fT07DvUyoyiVm6+fzykLkzOJUgghRPxIkDBJFec5uPn6+TzxVgt/ezYx1Rp3Vfdwz/MNbNnbTWFOCl+7bCYXLiuQoQUhhJgmJEiYxKxWC5edYVRr/PWDRrXGCyrz+eLaGWSmjb5a44H6Pu5+voG3dzrJzbTzhTUVXLK8kNQUmc4ohBDTiQQJU8Cc0jT+90vH8MBLjTz4ShPb9vdw46UVLJyRQX6WHesIqyu2dfVzoL6P/fUudhzqYeNOJ5lpVq69sIzLVhaRPgXKQwshhIidBAlTRIrdymcuLOc0lcP/PFjNT+45GNxuoSTPQWl+CmX5qZTmp5Cbaedwi5t9dX0caHDR0e0dOE5xbgofP6eEK84qJjtd3h5CCDGdyV1gijluTia3fUPx/oFuGts9NLZ7aGgz/t1b24Gz1weA3WZhTmkap6ps5penM788nXllaWRnyFtCCCGEQe4IU1Caw8r/b+/uY+SqyjiOf9sNFNpKi7RIC30BgQeCWqhFi7xJTDQV8AVJtdgACTUqClEkvjQhBQ2xKhVFeTFulAqFQAQR+afEKFYSUSwCaQy/CilQoGKhpdpCd9N2/eOcbafTO7Ozs7PdvTu/TzLpzD1zzpz75E7n2XvPzHNqHFLY9mbXTrZs28HkCQd6AaKZmdXlJKHNjB3TMSJKUJuZ2eDzcnUzMzMr5CTBzMzMCjlJMDMzs0KlXpMQEccDy4HDgNeBiyX9a2hnZWZmNjKU/UzCbcDNko4HbgZ+NsTzMTMzGzFKmyRExOHAbODuvOluYHZETB66WZmZmY0cZb7cMA14WdJOAEk7I+KVvH1jH307ALq7u1s6oa6urpaON9I4Pn1zjOpzfPrmGNXXrvGp+Lzr13fgy5wkDMQUgLVr17Z00DVr1rR0vJHG8embY1Sf49M3x6g+x4cpwHONPrnMScJ64MiI6MhnETqAqXl7Xx4HzgQ2ADsHcY5mZmbDQQcpQXi8P51G9fT0DM509oOIeATolHRnRCwELpN0zhBPy8zMbEQoe5JwAukrkIcCm0lfgdTQzsrMzGxkKHWSYGZmZoOntF+BNDMzs8HlJMHMzMwKOUkwMzOzQk4SzMzMrJCTBDMzMytU5h9TGhbatRJlRNwAfAqYCbxb0pq8vWY8mm0ro4g4DLgDeCfQBTwLfF7SxoiYSypGdjDwPLBQ0n9yv6bayigiHgCOBnYBW4ErJD3pY2hvEbEEuJb8PvPxs0dEPA9szzeAb0ha6Ri1js8kDFy7VqJ8ADgLeKFqe714NNtWRj3A9yWFpPeQfgZ1aUSMAu4EvpT3dRWwFKDZthK7RNIsSacANwC/yNt9DGURMRuYC7yYH/v42deFkk7Ot5WOUWs5SRiAdq5EKelRSXv9BHa9eDTbNtj7MVgkbZL0SMWmx4AZwBxgu6RH8/bbgPn5frNtpSRpS8XDCcAuH0N7RMQYUrJzOSnpBB8/jXCMWshJwsDsU4kS6K1E2Y7qxaPZttKLiNHAF4EHgelUnH2R9BowOiLePoC20oqIzoh4EbgeuAQfQ5W+DdwpaV3FNh8/+1oREU9HxC0RMRHHqKWcJJgNvp+Qrrn/dKgnMtxIWiRpOrAY+MFQz2e4iIjTgFOBW4Z6LsPcmZJmkWI1Cr/HWs5JwsDsrkQJ0M9KlCNRvXg021ZqeYHnccCnJe0iXVueUdE+CeiRtGkAbaUn6Q7gHOAlfAwBnA2cAKzLi/OOAlYCx+LjZ7feS56SukgJ1en4PdZSThIGIK96fRJYkDctAP4haePQzWro1ItHs237b/atFxHXA+8FPpH/EwNYDRwcEWfkx18A7h1gW+lExPiImFbx+HxgE+BjCJC0VNJUSTMlzSQlTx8hnW1p++MHICLGRcSEfH8U8BnSMeD3WAu5wNMAtWslyoi4CbgAOAJ4DXhd0kn14tFsWxlFxEnAGmAt8FbevE7SJyPiA6SV9wex52tWr+Z+TbWVTUS8A/gtMA7YSUoQrpb0hI+hfeWzCeflr0C2/fEDEBHHAPcBHfn2T+BKSRsco9ZxkmBmZmaFfLnBzMzMCjlJMDMzs0JOEszMzKyQkwQzMzMr5CTBzMzMCjlJMDMzs0IuFW1mhSJiJrAOOEDSjv3RNyKuBY6VtLA/r9esiDgT6JQU++P1zMrGSYKZtY2I6AGOk/QsgKQ/A04QzGrw5QazEokIJ/Y1ODZmrec3ldkwl3+S91bgs+lhzCFVljwZeBn4lqQH83MfIZUX7syPLwUWSTojP+4hla3+GjAJuAv4sqSeXBTpe8ClwH+BZVXzmAD8EPgosAv4JbBE0s6++tbZt6OB24HZwGOAKto+mPflqKpYLJL0+3xp4l3AduBjwFUR8TTwY+BE0s9h3wdcJak7IlblYZ7KcbgMeLXyNSLixBzrotjeDmwDZgJnkX4G+CJJzzWyr2Zl5DMJZuWwADiX9MH+G+Bh4HDgCmBFRPTnlPl5pNK6s4D5pMJBAJ/LbacAc4ALq/otB3aQKhGeAnwYWNRg31ruIhXWmQR8B7ikH/sB8HHg18BEYAWpDsRX83inAR8CLgeQdFbuM0vSeEn3VA4UEQcAv6N+bBcA15HqQzwLXN/P+ZqVipMEs3K4KZfFPRkYDyyV1C3pD8BD7Kl+2Iilkt6Q9CLwxzwmpIThR5LW5/K43+3tkAsyzQO+Imlbrrp4I6nyXt2+tUTEdFKyco2kLkmrSB/S/fEXSQ9I2iXpLUmrJT0maYek50nFes5ucKy59B3b+yX9LS/GXMGe2JmNSL7cYFYO6/O/U4H1knZVtL0AHNmPsf5dcf9N0gfj7rGrxu01AzgA2FDxh/Xo6nnV6FvLVGCzpG1V/abVeH6RytckIo4nXRKZA4wl/R+3usGxGoltrdiZjUg+k2BWDr3lWl8BpkVE5Xt3Oun6OaRr5mMr2o7ox2tsYO8P6OkV99cDXcAkSRPz7RBJJzXQt97rHRoR42r022tf8rqHyVVjVJexvRV4hvQNhkOAxcCoBuYCfcfWrO34TIJZufyV9OH59YhYBpwOnE86bQ/wJHBBRHSS/jLuXZzXiHuBKyPiofwa3+xtkLQhIh4GlkXENcBW4GjgKEl/qte3FkkvRMTfgesiYjHwvrwvD+anrAUOiohzSesEFgNj+hj2baSFk1sj4gTSIs2NFe2vAseQ1hNU6yu2Zm3HZxLMSkRSN2kl/zzgNeAW4GJJz+Sn3Ah0kz4Ml5Oumzfq58BK4CngCeD+qvaLgQNJq/o3kxYMTmmwby0XAe8HNgFLgF/1NkjaQlp02En6a34b8FIf412dx/xfntM9Ve3XAssj4o2ImF/Z0EBszdrOqJ6e6rN1ZmZmZj6TYGZmZjV4TYKZDaqI2FqjaV7+WWQzG6Z8ucHMzMwK+XKDmZmZFXKSYGZmZoWcJJiZmVkhJwlmZmZWyEmCmZmZFfo/F0MR3meP0wcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f98ad044b00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Average residual grouping by ride duration\n",
    "val_residual = X_val.copy()\n",
    "val_residual['duration'] = y_val\n",
    "val_residual['rounded_duration'] = np.around(y_val, -2)\n",
    "val_residual['residual'] = np.abs(y_val - y_val_pred)\n",
    "tmp = val_residual.groupby('rounded_duration').mean()\n",
    "plt.figure(figsize=(8,4))\n",
    "tmp['residual'].plot()\n",
    "plt.ylabel('average residual')\n",
    "plt.title('average residual by ride duration');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fd9ba9658ec2a2c66f73bd8431da529d",
     "grade": false,
     "grade_id": "cell-db93e20cec0d6aff",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "In the first visualization, we see that most of the residuals are centered around 250 seconds ~ 4 minutes. There is a minor right tail, suggesting that we are still unable to accurately fit some outliers in our data. The second visualization also suggests this, as we see the average residual increasing as a somewhat linear function of duration. But given that our average ride duration is roughly 600-700 seconds, it seems that we are indeed optimizing for the average ride because the residuals are smallest around 600-700. \n",
    "\n",
    "Keep this in mind when creating your final model! Visualizing the error is a powerful tool and may help diagnose shortcomings of your model. Let's go ahead and submit to kaggle, although your error on the test set may be higher than 300. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1d0011fc6013000b34a6389a6da0a7e7",
     "grade": false,
     "grade_id": "cell-0398ecf597b3cf59",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Submission to Kaggle\n",
    "\n",
    "The following code will write your predictions on the test dataset to a CSV, which you can submit to Kaggle. You may need to modify it to suit your needs, but we recommend you make a copy and preserve the original function.\n",
    "\n",
    "Remember that if you've performed transformations or featurization on the training data, you must also perform the same transformations on the test data in order to make predictions. For example, if you've created features for the columns `pickup_datetime` or `pickup_latitude` on the training data, you must also extract the same features in order to use scikit-learn's `.predict(...)` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9949c57697286ad5f7b580802a414f7a",
     "grade": false,
     "grade_id": "cell-46e29e97c3760a86",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "def generate_submission(test, predictions, force=False):\n",
    "    if force:\n",
    "        if not os.path.isdir(\"submissions\"):\n",
    "            os.mkdir(\"submissions\")\n",
    "        submission_df = pd.DataFrame({\n",
    "            \"id\": test_df.index.values, \n",
    "            \"duration\": predictions,\n",
    "        },\n",
    "            columns=['id', 'duration'])\n",
    "\n",
    "        timestamp = datetime.isoformat(datetime.now()).split(\".\")[0]\n",
    "\n",
    "        submission_df.to_csv(f'submissions/submission_{timestamp}.csv', index=False)\n",
    "\n",
    "        print(f'Created a CSV file: submission_{timestamp}.csv')\n",
    "        print('You may now upload this CSV file to Kaggle for scoring.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "13d22ae35c56a9b1c0e4d11043809f48",
     "grade": false,
     "grade_id": "cell-1c77a057c8897ad5",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4388: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  object.__getattribute__(self, name)\n",
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4389: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  return object.__setattr__(self, name, value)\n"
     ]
    }
   ],
   "source": [
    "X_test, _ = process_data_gm1(test_df, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-18T23:29:22.604349Z",
     "start_time": "2018-03-18T23:29:22.265985Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "54cecf7cad6b2c9ed04fe0b5be3d8352",
     "grade": false,
     "grade_id": "cell-e4d56cb38e5d6ad5",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created a CSV file: submission_2018-12-05T01:39:56.csv\n",
      "You may now upload this CSV file to Kaggle for scoring.\n"
     ]
    }
   ],
   "source": [
    "assert list(X_train.columns) == list(X_test.columns), \"Different columns or different column ordering\"\n",
    "submission_predictions = (guided_model_1\n",
    "                          .fit(X_train, y_train)\n",
    "                          .predict(X_test))\n",
    "submission_predictions = submission_predictions.astype(int)\n",
    "submission_predictions[submission_predictions < 0] = 0\n",
    "generate_submission(test_df, submission_predictions, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f98988d31cf004d076a6f8bb0f4500b1",
     "grade": false,
     "grade_id": "cell-7e4fef935fcd3d1f",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Check your submission\n",
    "assert isinstance(submission_predictions, np.ndarray), \"Submission not an array\"\n",
    "assert all(submission_predictions >= 0), \"Duration must be non-negative\"\n",
    "assert issubclass(submission_predictions.dtype.type, np.integer), \"Seconds must be integers\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7b706de962b9eb645e07d5f45859da6d",
     "grade": false,
     "grade_id": "cell-841ea29e54b5d9b2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Your Turn!\n",
    "\n",
    "Now it's your turn! Draw upon everything you have learned this semester to find the best features to help your model accurately predict the duration of a taxi ride.\n",
    "\n",
    "You may use whatever method you prefer in order to create features. You may use features that we created and features that you discovered yourself from any of the 2 datasets. However, we want to make it fair to students who are seeing these techniques for the first time. As such, you are only allowed regression models and their regularized forms. This means no random forest, k-nearest-neighbors, neural nets, etc.\n",
    "\n",
    "**Here are some ideas to improve your model:**\n",
    "- **Data selection**: January 2016 was an odd month for taxi rides due to the blizzard. Would it help to select training data differently?\n",
    "- **Data cleaning**: Try cleaning your data in different ways. In particular, consider how to handle outliers. \n",
    "- **Better features**: Explore the 2 datasets and find what features are most helpful. Utilize external datasets to improve your accuracy. \n",
    "- **Regularization**: Try different forms of regularization to avoid fitting to the training set. Recall that `Ridge` and `Lasso` are the names of the classes in `sklearn.linear_model` that combine `LinearRegression` with regularization techniques.\n",
    "- **Model selection**: You can adjust parameters of your model (e.g., the regularization parameter) to achieve higher accuracy. [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) may be helpful.\n",
    "- **Validation**: Recall that you should use cross-validation to do feature and model selection properly! Otherwise, you will likely overfit to your training data.\n",
    "\n",
    "There's many things you could try that could help your model. We have only suggested a few. Be creative and innovative! Please use `proj2_extras.ipynb` for all of your extraneous work. Note that you will be submitting `proj2_extras.ipynb` and we will be grading it. Please properly comment and format this notebook!\n",
    "\n",
    "Once you are satisfied with your results, answer the questions in the Deliverables section. You may want to read this section in advance so you have an idea of what we're looking for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "090b00051aeacddcf11d16d2ba092461",
     "grade": false,
     "grade_id": "cell-926dd79601c9e5bc",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Deliverables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "3f4b4ba48b89bfe6f79cb734e9af2d10",
     "grade": false,
     "grade_id": "cell-28fcb99d211b6d14",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## Feature/Model Selection Process\n",
    "\n",
    "Let's first look at selection of better features. In this following cell, describe the process of choosing good features to improve your model. You should use at least 3-4 sentences each to address the follow questions. Backup your responses with graphs supporting your claim (you can save figures and load them, no need to add the plotting code here). Use these questions to concisely summarize all of your extra work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "48a09f77cc60e445e2669a5658cfa095",
     "grade": false,
     "grade_id": "q1a",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 1a\n",
    "How did you find better features for your model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "712e610263e57c35f71044e0e912da5c",
     "grade": true,
     "grade_id": "q1a-answer",
     "locked": false,
     "points": 5,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1. By reasoning and adding featues accumulately and using cross validation to check whether the features are good.\n",
      "\n",
      "2. By making visualizations of the feature to explore its relationship with our prediction variable, i.e. duration.\n",
      "\n",
      "3. By cleaning features to make them \"real\" good features. This step is extremely important in features which have\n",
      "some extreme outliers, like pickup and dropoff location. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "q1a_answer = r\"\"\"\n",
    "1. By reasoning and adding featues accumulately and using cross validation to check whether the features are good.\n",
    "\n",
    "2. By making visualizations of the feature to explore its relationship with our prediction variable, i.e. duration.\n",
    "\n",
    "3. By cleaning features to make them \"real\" good features. This step is extremely important in features which have\n",
    "some extreme outliers, like pickup and dropoff location. \n",
    "\"\"\"\n",
    "# YOUR CODE HERE\n",
    "print(q1a_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "463df11915ead370c4422b93d45468f1",
     "grade": false,
     "grade_id": "q1b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 1b\n",
    "What did you try that worked / didn't work?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fcb1a0cda1cd4569c4ed1fef85728082",
     "grade": true,
     "grade_id": "q1b-answer",
     "locked": false,
     "points": 5,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1. Good featurtes to use besides what are provided by starter code: total_amount, day_of_week, hour, trip_distance, \n",
      "passenger_count, tolls_amount, tip_amount, extra, fare_amount.\n",
      "\n",
      "2. Features didn't work: improvement_surcharge, mta_tax, payment_type. Some of them are nearly constants for all\n",
      "trips so it's useless to use them. \n",
      "\n",
      "3. Speed could be a good feature. However, we couldn't compute it directly as we have no access to duration in \n",
      "test_df. Though we could follow the same path as part 2 using clustering approach, it essentially uses the \n",
      "pickup, dropoff location and manhattan distance which we've already included in our features. Hence, I \n",
      "think it's not necessary to add speed at this stage.\n",
      "\n",
      "4. Num_accidents could also be a good feature. But due to memory limitation, we can only load the nearby accidents \n",
      "for a centrain week, rather than load all date in January. This result in many NaNs in the final num_accidents \n",
      "feature. Though we could use mean to fill the NaN values, this will decreases the meaning of this feature a lot.\n",
      "Another disadvantage is that it can need to use real time traffic information, i.e. accidents with spatial/temporal \n",
      "proximity to during the inference time which is quite time and memory consuming. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "q1b_answer = r\"\"\"\n",
    "1. Good featurtes to use besides what are provided by starter code: total_amount, day_of_week, hour, trip_distance, \n",
    "passenger_count, tolls_amount, tip_amount, extra, fare_amount.\n",
    "\n",
    "2. Features didn't work: improvement_surcharge, mta_tax, payment_type. Some of them are nearly constants for all\n",
    "trips so it's useless to use them. \n",
    "\n",
    "3. Speed could be a good feature. However, we couldn't compute it directly as we have no access to duration in \n",
    "test_df. Though we could follow the same path as part 2 using clustering approach, it essentially uses the \n",
    "pickup, dropoff location and manhattan distance which we've already included in our features. Hence, I \n",
    "think it's not necessary to add speed at this stage.\n",
    "\n",
    "4. Num_accidents could also be a good feature. But due to memory limitation, we can only load the nearby accidents \n",
    "for a centrain week, rather than load all date in January. This result in many NaNs in the final num_accidents \n",
    "feature. Though we could use mean to fill the NaN values, this will decreases the meaning of this feature a lot.\n",
    "Another disadvantage is that it can need to use real time traffic information, i.e. accidents with spatial/temporal \n",
    "proximity to during the inference time which is quite time and memory consuming. \n",
    "\"\"\"\n",
    "# YOUR CODE HERE\n",
    "print(q1b_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "6a071fab3049c2dcab9592a614b87439",
     "grade": false,
     "grade_id": "q1c",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 1c\n",
    "\n",
    "What was surprising in your search for good features?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bf93f09066047a3039e2f0d144720d19",
     "grade": true,
     "grade_id": "q1c-answer",
     "locked": false,
     "points": 5,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total_amount hour, day_of_week are good features. \n",
      "\n",
      "Actually, in this problem, the most important work is to clean features. One thing that suprises me most is that \n",
      "after I cleaned features(for training data, I just threw away these data points) I chose, the error dropped really\n",
      "a lot. \n",
      "\n",
      "To be more specific, we could discard outliers from out training data. Outliers could be amount less than 0, the day \n",
      "corresponds to the bllizard, etc. Also, we need to properly deal with outliers or dirty data in our test set. There\n",
      "are some data whose longitude and latitude are 0. Instead of making predictions using our model, it's more reasonable \n",
      "to use the average duration of all other data points. After I did these steps, the final mae on test set dropped \n",
      "dramatically, from thousands to around 240.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "q1c_answer = r\"\"\"\n",
    "Total_amount hour, day_of_week are good features. \n",
    "\n",
    "Actually, in this problem, the most important work is to clean features. One thing that suprises me most is that \n",
    "after I cleaned features(for training data, I just threw away these data points) I chose, the error dropped really\n",
    "a lot. \n",
    "\n",
    "To be more specific, we could discard outliers from out training data. Outliers could be amount less than 0, the day \n",
    "corresponds to the bllizard, etc. Also, we need to properly deal with outliers or dirty data in our test set. There\n",
    "are some data whose longitude and latitude are 0. Instead of making predictions using our model, it's more reasonable \n",
    "to use the average duration of all other data points. After I did these steps, the final mae on test set dropped \n",
    "dramatically, from thousands to around 240.\n",
    "\n",
    "\"\"\"\n",
    "# YOUR CODE HERE\n",
    "print(q1c_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "172960adce9522c39f3b2ddfd0dc0fa8",
     "grade": false,
     "grade_id": "q2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 2\n",
    "\n",
    "Just as in the guided model above, you should encapsulate as much of your workflow into functions as possible.  Define `process_data_fm` and `final model` in the cell below.  In order to calculate your final model's MAE, we will run the code in the cell after that.\n",
    "\n",
    "**Note:** You *MUST* name the model you wish to be evaluated on `final_model`. This is what we will be using to generate your predictions. We will take the state of `final_model` right after executing the cell below and run the following code:\n",
    "\n",
    "```\n",
    "# Load in test_df, solutions\n",
    "X_test, _ = process_data_fm(test_df, True)\n",
    "submission_predictions = final_model.predict(X_test)\n",
    "# Generate score for autograding\n",
    "```\n",
    "\n",
    "We encourage you to conduct all of your exploratory work in `proj2_extras.ipynb`, which will be graded for 10 points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make all the data cleaning work into one function\n",
    "def clean_data(df):\n",
    "    index = (df['tpep_pickup_datetime'].dt.day != 23) & (df['tolls_amount'] >= 0) \\\n",
    "                & (df['improvement_surcharge'] >= 0) & (df['mta_tax'] >= 0) & (df['extra'] >= 0) \\\n",
    "                & (df['fare_amount'] >= 0) & (df['duration'] <= 4000) & (df['duration'] >= 150)\n",
    "    return df.loc[index, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(model, test_df):\n",
    "    \"\"\"\n",
    "    Make predictions using model on test_df data with properly\n",
    "    handle of outliers and dirty data, like longitude/latitude = 0.0\n",
    "    \"\"\"\n",
    "    clean_index = (test_df['pickup_latitude'] <= 40.85) & (test_df['pickup_latitude'] >= 40.63) & \\\n",
    "                    (test_df['dropoff_latitude'] <= 40.85) & (test_df['dropoff_latitude'] >= 40.63) & \\\n",
    "                    (test_df['pickup_longitude'] <= -73.65) & (test_df['pickup_longitude'] >= -74.03) & \\\n",
    "                    (test_df['dropoff_longitude'] <= -73.65) & (test_df['dropoff_longitude'] >= -74.03)\n",
    "    \n",
    "    dirty_index = ~ clean_index\n",
    "    if sum(dirty_index) == 0:\n",
    "        return model.predict(test_df)\n",
    "        \n",
    "    clean_pred = model.predict(test_df.loc[clean_index])\n",
    "    avg_duration = np.mean(clean_pred)\n",
    "    \n",
    "    pred = pd.DataFrame({\n",
    "        \"id\": test_df.index.values,\n",
    "        \"duration\": model.predict(test_df),\n",
    "    },\n",
    "        columns=['id', 'duration'])\n",
    "        \n",
    "    # Replace the predictions for outliers\n",
    "    \n",
    "    pred.loc[dirty_index, 'duration'] = avg_duration\n",
    "    assert sum(clean_index) + sum(dirty_index) == len(test_df), \"error in spliting data into clean and dirty!\"\n",
    "    \n",
    "    return np.array(pred['duration'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f3ae9325e11ddba52eb687f02c23916f",
     "grade": true,
     "grade_id": "q2-answer",
     "locked": false,
     "points": 5,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alpha: 1.0, error: 91.79498835118243\n",
      "alpha: 0.5, error: 91.94374277161074\n",
      "alpha: 0.1, error: 92.05542702449938\n",
      "alpha: 0.05, error: 92.06406569056693\n",
      "Best model: alpha 1.0, error 91.79498835118243\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4388: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  object.__getattribute__(self, name)\n",
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4389: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  return object.__setattr__(self, name, value)\n"
     ]
    }
   ],
   "source": [
    "def process_data_fm(data, test=False):\n",
    "    if not test:\n",
    "        data = clean_data(data)\n",
    "    # Put your final pipeline here\n",
    "    X = (\n",
    "        data\n",
    "        \n",
    "        # Transform data\n",
    "        .pipe(add_time_columns)\n",
    "        .pipe(add_distance_columns)\n",
    "        \n",
    "        .pipe(select_columns,        \n",
    "              'pickup_longitude',  \n",
    "              'pickup_latitude',   \n",
    "              'dropoff_longitude', \n",
    "              'dropoff_latitude',\n",
    "              'manhattan',\n",
    "              'total_amount',\n",
    "              'day_of_week',\n",
    "              'hour',\n",
    "              'trip_distance',\n",
    "              'passenger_count',\n",
    "              'tolls_amount',\n",
    "              'tip_amount',\n",
    "              'extra',\n",
    "              'fare_amount'\n",
    "             )\n",
    "    )\n",
    "    if test:\n",
    "        y = None\n",
    "    else:\n",
    "        y = data['duration']\n",
    "        \n",
    "    return X, y\n",
    "    \n",
    "# final_model = ... # Define your final model here, feel free to try other forms of regression\n",
    "# final_model.fit(...)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "\n",
    "# Use grid search to find the best regularzation hyperparameters by cross validataion \n",
    "alphas = [1.0, 0.5, 0.1, 0.05]\n",
    "\n",
    "# Train\n",
    "X_train, y_train = process_data_fm(train_df)\n",
    "X_val, y_val = process_data_fm(val_df)\n",
    "\n",
    "errors = {}\n",
    "min_error = float('inf')\n",
    "for alpha in alphas:\n",
    "    my_model = lm.Ridge(alpha=alpha, fit_intercept=True)\n",
    "    my_model.fit(X_train, y_train)\n",
    "    \n",
    "    # Predict\n",
    "    y_val_pred = predict(my_model, X_val)\n",
    "    error = mae(y_val_pred, y_val)\n",
    "    errors[alpha] = error\n",
    "    print(f\"alpha: {alpha}, error: {error}\")\n",
    "    if error < min_error:\n",
    "        min_error = error\n",
    "        final_model = my_model\n",
    "\n",
    "print(f'Best model: alpha {min(errors, key=errors.get)}, error {min_error}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4388: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  object.__getattribute__(self, name)\n",
      "/srv/conda/envs/data100/lib/python3.6/site-packages/pandas/core/generic.py:4389: FutureWarning: Attribute 'is_copy' is deprecated and will be removed in a future version.\n",
      "  return object.__setattr__(self, name, value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created a CSV file: submission_2018-12-05T01:42:23.csv\n",
      "You may now upload this CSV file to Kaggle for scoring.\n"
     ]
    }
   ],
   "source": [
    "# Feel free to change this cell\n",
    "X_test, _ = process_data_fm(test_df, True)\n",
    "# final_predictions = final_model.predict(X_test)\n",
    "final_predictions = predict(final_model, X_test)\n",
    "final_predictions = final_predictions.astype(int)\n",
    "generate_submission(test_df, final_predictions, True) # Change to true to generate prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "861b71e95b1f972cb10456cf2b12b9ec",
     "grade": false,
     "grade_id": "cell-adae1031103d7053",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 3\n",
    "The following hidden cells will test your model on the test set. Please do not delete any of them if you want credit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1911acd5120ab37471993a87d23c7db5",
     "grade": true,
     "grade_id": "cell-49787d08aa5b0d72",
     "locked": true,
     "points": 0,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NO TOUCH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c5d45274bd4034d9af482544717ec729",
     "grade": true,
     "grade_id": "cell-fd20db43e21281c2",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NOH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ffb5644a5eab9e393947932e6982f0f9",
     "grade": true,
     "grade_id": "cell-dde1980abb5265eb",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# STAHP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bc0c573ae7e3228d15bc11f4033a6c9b",
     "grade": true,
     "grade_id": "cell-4720d6f913f2e2a8",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NO MOLESTE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7b13769e4516b387c2385d4052a75d4b",
     "grade": true,
     "grade_id": "cell-becfc4e39f8d613d",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# VA-T'EN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6d640bfc58c223abbae088de63df3ecd",
     "grade": true,
     "grade_id": "cell-66c75927b3162384",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# NEIN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "64954a2f86231a4b4c0d75fcb8d662fd",
     "grade": true,
     "grade_id": "cell-480ee1c73e5aeed2",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# PLSNO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "651163ce28a1f5235e83f005a03de616",
     "grade": true,
     "grade_id": "cell-5c13060e51df5fbf",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# THIS SPACE IS NOT YOURS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "29a3b381460853a91d8fe9f220324b71",
     "grade": true,
     "grade_id": "cell-9594161a0c8ada1b",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TAWDEETAW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cc76405d6e229a5c8cce770d2f56aa4a",
     "grade": true,
     "grade_id": "cell-7ff15da448905075",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# MAU LEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9193b6943369510c02fd042f8009e567",
     "grade": true,
     "grade_id": "cell-5799ae0e2e02e4ea",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# ALMOST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b71274e2a3a0876812af46e7fadda22a",
     "grade": true,
     "grade_id": "cell-4e8f93c8d961fdc3",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2017f2b6d777ef593da976d61ccfa819",
     "grade": true,
     "grade_id": "cell-cb843d9e9defdb3f",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# THE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f4cd698d64eab1bfb873811b9d7cf3ea",
     "grade": true,
     "grade_id": "cell-a83a8dbeab2377b8",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# END"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f1df2786662d79bb122a671d432f6287",
     "grade": true,
     "grade_id": "cell-c11ba085e96a4d43",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Hmph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9dcf92d155de3c51788e9f94c9b684d2",
     "grade": true,
     "grade_id": "cell-d83490b23ffd0d29",
     "locked": true,
     "points": 1,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Good riddance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cb9bd7842363ea079393c77707824a6c",
     "grade": false,
     "grade_id": "cell-c37f9560ddca0dd2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created a CSV file: submission_2018-12-05T01:39:59.csv\n",
      "You may now upload this CSV file to Kaggle for scoring.\n"
     ]
    }
   ],
   "source": [
    "generate_submission(test_df, submission_predictions, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7bb9c8347d379dfe40c3951647be36ce",
     "grade": false,
     "grade_id": "cell-ac585a9fd1711d23",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "This should be the format of your CSV file.  \n",
    "Unix-users can verify it running `!head submission_{datetime}.csv` in a jupyter notebook cell.\n",
    "![sample_submission_format](figs/sample_submission_format.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ac53667491c5aca21fd2226a4039dffc",
     "grade": false,
     "grade_id": "cell-0f5d963bfa6acfdf",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "Kaggle link: https://www.kaggle.com/t/f8b3c6acc3a045cab152060a5bc79670"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission\n",
    "\n",
    "You're almost done!\n",
    "\n",
    "Before submitting this assignment, ensure that you have:\n",
    "\n",
    "1. Restarted the Kernel (in the menubar, select Kernel$\\rightarrow$Restart & Run All)\n",
    "2. Validated the notebook by clicking the \"Validate\" button.\n",
    "\n",
    "Then,\n",
    "\n",
    "1. **Submit** the assignment via the Assignments tab in **Datahub** \n",
    "1. **Upload and tag** the manually reviewed portions of the assignment on **Gradescope**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}