05_model_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Machine Learning models in scikit-learn ([video #5](https://www.youtube.com/watch?v=0pP4EwWJgIU&list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A&index=5))\n",
    "\n",
    "Created by [Data School](https://www.dataschool.io). Watch all 10 videos on [YouTube](https://www.youtube.com/playlist?list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A). Download the notebooks from [GitHub](https://github.com/justmarkham/scikit-learn-videos).\n",
    "\n",
    "**Note:** This notebook uses Python 3.9.1 and scikit-learn 0.23.2. The original notebook (shown in the video) used Python 2.7 and scikit-learn 0.16."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "- How do I choose **which model to use** for my supervised learning task?\n",
    "- How do I choose the **best tuning parameters** for that model?\n",
    "- How do I estimate the **likely performance of my model** on out-of-sample data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review\n",
    "\n",
    "- Classification task: Predicting the species of an unknown iris\n",
    "- Used three classification models: KNN (K=1), KNN (K=5), logistic regression\n",
    "- Need a way to choose between the models\n",
    "\n",
    "**Solution:** Model evaluation procedures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation procedure #1: Train and test on the entire dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Train the model on the **entire dataset**.\n",
    "2. Test the model on the **same dataset**, and evaluate how well we did by comparing the **predicted** response values with the **true** response values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# added empty cell so that the cell numbering matches the video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in the iris data\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "\n",
    "# create X (features) and y (response)\n",
    "X = iris.data\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       2, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
       "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2,\n",
       "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import the class\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# instantiate the model\n",
    "logreg = LogisticRegression(solver='liblinear')\n",
    "\n",
    "# fit the model with data\n",
    "logreg.fit(X, y)\n",
    "\n",
    "# predict the response values for the observations in X\n",
    "logreg.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# store the predicted response values\n",
    "y_pred = logreg.predict(X)\n",
    "\n",
    "# check how many predictions were generated\n",
    "len(y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification accuracy:\n",
    "\n",
    "- **Proportion** of correct predictions\n",
    "- Common **evaluation metric** for classification problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.96\n"
     ]
    }
   ],
   "source": [
    "# compute classification accuracy for the logistic regression model\n",
    "from sklearn import metrics\n",
    "print(metrics.accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Known as **training accuracy** when you train and test the model on the same data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN (K=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9666666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "knn.fit(X, y)\n",
    "y_pred = knn.predict(X)\n",
    "print(metrics.accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN (K=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X, y)\n",
    "y_pred = knn.predict(X)\n",
    "print(metrics.accuracy_score(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problems with training and testing on the same data\n",
    "\n",
    "- Goal is to estimate likely performance of a model on **out-of-sample data**\n",
    "- But, maximizing training accuracy rewards **overly complex models** that won't necessarily generalize\n",
    "- Unnecessarily complex models **overfit** the training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Overfitting](images/05_overfitting.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Image Credit: [Overfitting](https://commons.wikimedia.org/wiki/File:Overfitting.svg#/media/File:Overfitting.svg) by Chabacano. Licensed under GFDL via Wikimedia Commons.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation procedure #2: Train/test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Split the dataset into two pieces: a **training set** and a **testing set**.\n",
    "2. Train the model on the **training set**.\n",
    "3. Test the model on the **testing set**, and evaluate how well we did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 4)\n",
      "(150,)\n"
     ]
    }
   ],
   "source": [
    "# print the shapes of X and y\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# STEP 1: split X and y into training and testing sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Train/test split](images/05_train_test_split.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What did this accomplish?\n",
    "\n",
    "- Model can be trained and tested on **different data**\n",
    "- Response values are known for the testing set, and thus **predictions can be evaluated**\n",
    "- **Testing accuracy** is a better estimate than training accuracy of out-of-sample performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# added empty cell so that the cell numbering matches the video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(90, 4)\n",
      "(60, 4)\n"
     ]
    }
   ],
   "source": [
    "# print the shapes of the new X objects\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(90,)\n",
      "(60,)\n"
     ]
    }
   ],
   "source": [
    "# print the shapes of the new y objects\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(solver='liblinear')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# STEP 2: train the model on the training set\n",
    "logreg = LogisticRegression(solver='liblinear')\n",
    "logreg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9333333333333333\n"
     ]
    }
   ],
   "source": [
    "# STEP 3: make predictions on the testing set\n",
    "y_pred = logreg.predict(X_test)\n",
    "\n",
    "# compare actual response values (y_test) with predicted response values (y_pred)\n",
    "print(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat for KNN with K=5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9666666666666667\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)\n",
    "print(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat for KNN with K=1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.95\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)\n",
    "print(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we locate an even better value for K?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# try K=1 through K=25 and record testing accuracy\n",
    "k_range = list(range(1, 26))\n",
    "scores = []\n",
    "for k in k_range:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    scores.append(metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Testing Accuracy')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAw1ElEQVR4nO3de5xc9Xnf8c93Z6+j285gSUia4WKiGBSKBV7LsZ3QOIQUHDfYNE6hSYxdXEwTbOPEbSm92EnahNpglyQEIsck0NgmDjFBTYnBJRfq1AWEEXcplrnNSGtJMKPrzN6f/nHOzA6j2d2Z2Tk7uzvP+/Xa1+65zfkdrXae+V2e309mhnPOOVevrnYXwDnn3NLigcM551xDPHA455xriAcO55xzDfHA4ZxzriEeOJxzzjUk0sAh6RJJeyTtlXRDjeMJSfdJelrSY5LOrTj2KUnPSXpW0tck9Yf7Pytpn6Rd4dd7o3wG55xzbxRZ4JAUA24DLgW2AFdK2lJ12o3ALjM7D/gQcGt47SbgE8CQmZ0LxIArKq77opltDb8eiOoZnHPOnSzKGsc2YK+ZvWhmY8A9wGVV52wBHgYws93AGZLWh8e6gQFJ3UAc2B9hWZ1zztWpO8LX3gRkKrazwDuqznkKuBz4tqRtwOlAysyekHQz8CpQBB4ys4cqrrtO0oeAncCvmVm++uaSrgGuAVixYsXbzj777BY9lnPOdYYnnnjiNTNbW70/ysChGvuq5ze5CbhV0i7gGeBJYEJSgqB2ciZwGPgzSb9oZn8C3A78ZvhavwncAvzLk25kth3YDjA0NGQ7d+5swSM551znkPRKrf1RBo4skK7YTlHV3GRmR4GPAEgS8FL49U+Al8zsUHjsG8C7gD8xswOl6yV9CfjLCJ/BOedclSj7OB4HNks6U1IvQef2jsoTJA2GxwA+CjwSBpNXgR+VFA8DykXAC+E1Gype4gPAsxE+g3POuSqR1TjMbELSdcCDBKOi7jSz5yRdGx6/AzgHuFvSJPA8cHV47FFJ9wLfBSYImrC2hy/9OUlbCZqqXgY+FtUzOOecO5k6YVp17+NwzrnGSXrCzIaq93vmuHPOuYZ44HDOOdcQDxzOOecaEuVwXLeETU0Zd/79Sxwtjre7KG6JGDojyYU/fFKuWMuMjE9y/659fPBtabq6aqWJuYXigcPV9PS+I/yX//UCAPK/UTcHMzgtGeeRf/ueyO7x0PMH+Hd//gw/tG4Vbzs9Edl93Nw8cLiaXs0VAHjw+gt5y6mr2lwat9h9/sHd/MHfvcjklBGLqDbw6usnAMjkCh442sz7OFxNmTBwpBIDbS6JWwrSiTgTU8bwkWJk98jkiuH3QmT3cPXxwOFqyuYLJFf0sqLPK6VubqlEHJh+c49CJl94w3fXPh44XE2ZXJG01zZcndLJ4P9KlG/q5cARYXBy9fHA4WrK5AukkvF2F8MtERsHB+gSZCNqRpqYnGL/4RHAaxyLgQcOd5LJKWP/4SLphAcOV5+eWBcb1gyQyUdTGxg+MsLklPGmlX0MHxlhYnIqkvu4+njgcCc5cHSE8UkrNz84V49UYoBsRLWBbBiQ3nXWKUxOGcNHRiK5j6uPBw53ktKoFa9xuEakk/HI+h9KzVPvOuuUN2y79vDA4U5Sam5Iex+Ha0A6EefAsRFGJyZb/trZXIEuwdvPTIbb3kHeTh443EkyuQISbBzsb3dR3BKSSgxgBvsi6OfI5Iucurqf05JxuuQ1jnbzwOFOkskXWL+qn77uWLuL4paQUg01ig7yTC4Y5VfuhPckwLbywOFOks0VvWPcNaycyxHBm3omXyj3uaWT0Y3ecvXxwOFOkq34I3WuXutX9dMb6yqPgGqV0YlJDhwdLQemdCIe2egtVx8PHO4NxiamGD464sl/rmFdXWJTYqDl/Q+lPpPpGkecA0dHGRlvfSe8q48HDvcG+w8XMcOnG3FNSSUGWp49Xj3Kr1Tz2HfYm6vaxQOHe4PSp8WUN1W5JqQS8Zb3P1TP1Dw9oaI3V7VLpIFD0iWS9kjaK+mGGscTku6T9LSkxySdW3HsU5Kek/SspK9J6g/3JyV9S9L3wu8+MX8LlRK4vHPcNSOdHCB3YowToxMte81MvkBPTKxfHQwPLzVZeQd5+0QWOCTFgNuAS4EtwJWStlSddiOwy8zOAz4E3Bpeuwn4BDBkZucCMeCK8JobgIfNbDPwcLjtWiSTL9DdJTas8cDhGjf9pt662kA2V2TT4EB5gah1q/ro7e6KbEJFN7coaxzbgL1m9qKZjQH3AJdVnbOF4M0fM9sNnCFpfXisGxiQ1A3Egf3h/suAu8Kf7wLeH9kTdKBMrsDGij9S5xpRzuVoYWZ3Jl94wywGXV0iNdj6TnhXvygDxyYgU7GdDfdVegq4HEDSNuB0IGVm+4CbgVeBYeCImT0UXrPezIYBwu/rat1c0jWSdkraeejQoRY90vKXzXsOh2teaVBFK4fLZvPFk/rcUsl4y4f9uvpFGThqfWS1qu2bgISkXcDHgSeBibDf4jLgTGAjsELSLzZyczPbbmZDZja0du3ahgvfqTyHw81HckUv8d5Yy2ocJ0YnyJ0YO+nDTDrh2ePtFGXgyALpiu0U081NAJjZUTP7iJltJejjWAu8BPwU8JKZHTKzceAbwLvCyw5I2gAQfj8Y4TN0lMLYBK8dH/PJDV3TJJFOxFvWjFR6neoPM+lknHxhnOMt7IR39YsycDwObJZ0pqRegs7tHZUnSBoMjwF8FHjEzI4SNFH9qKS4JAEXAS+E5+0Argp/vgq4P8Jn6Cilqn/KczjcPKRaWBso1Vyq/0+Wtr3W0R6RBQ4zmwCuAx4keNP/upk9J+laSdeGp50DPCdpN8Hoq0+G1z4K3At8F3gmLOf28JqbgIslfQ+4ONx2LTA9Xt5rHK556bD/way6Zbpx5bVhqmrBac/laKvuKF/czB4AHqjad0fFz98BNs9w7WeAz9TY/zpBDcS12PQfqdc4XPNSiQGOj05wuDBOYkXv3BfMIpMvMNAT45Sq14lyJl43N88cd2WZfJH+ni7Wruxrd1HcElZ6U2/FqKfSKL+gxXpaIt7Dit6Y1zjaxAOHK8vmC6QS8ZP+SJ1rRCuTADO52qP8JJWbxNzC88DhyjK5ok9u6OYt1aJ1OcwsrHHU7nNL+fTqbeOBw5VlwhqHc/Oxur+HNQM9865xHA6H2840yq80eqsVnfCuMR44HABHCuMcG5nwjnHXEunkwLyTAOeaqTmdjHNibJJ8YXxe93GN88DhgJkTrZxrRiuSAOeaqTntuRxt44HDATOPl3euGaWO66mp5puRyh9mZvg/OT0k1wPHQvPA4YDpoZNe43CtkE4MMDYxxWvHR5t+jWy+wJqBHlb399S+RwuH/brGeOBwQPCpbVV/N2vitf9InWtEqgW1gUxu9pmaV/Z1k4j3eFNVG3jgcMDM4+Wda8b0lCDN1wYydczUnE62fqlaNzcPHA4IssZ9ckPXKvOdhHBqysJ1OGb/P5lKDPhKgG3ggcOFiVYF7xh3LdPfE2Ptqr6mm6oOHR9lbGJqzv+T6cT8O+Fd4zxwOA4dH2VkfMqzxl1LBYstNdeMVB7lN0dTVSoZZ2xyioPHmu+Ed43zwOEqxst7jcO1TjoZJ3u4uRpHeZTfHAmp5VwOH5K7oDxwuPJ8Px44XCulE3H2Hx5hYnKq4WvrXRtmekiuB46F5IHD+cp/LhLp5ACTU8bwkZGGr83kC6xd1Ud/T2zW8zYNljrhfWTVQvLA4cjkCpyyopd4b6TrerkOk5rH9OqZXH2j/Pp7Yqxb1ee5HAvMA4cLZsX1ZirXYqWO7WwTtYF6cjjK90nOf14s1xgPHM7X4XCR2DDYT5car3FMTE4xfGSk7pma5zN6yzXHA0eHm5wy9h+eebEc55rVE+tiw5qBhpuRho+MMDllDdU4ho8UGW+iE941xwNHh/vB0REmGvgjda4R6eRAw5MQzjUr7kn3SMSZMvhBE53wrjmRBg5Jl0jaI2mvpBtqHE9Iuk/S05Iek3RuuP8tknZVfB2VdH147LOS9lUce2+Uz7DcTU+n7k1VrvWaWZej1CdS74eZVi1V6+oX2TAaSTHgNuBiIAs8LmmHmT1fcdqNwC4z+4Cks8PzLzKzPcDWitfZB9xXcd0XzezmqMreSerN0HWuGelknANHRxkZn5xzaG1JJl+gS0EfSV33mMfoLdecKGsc24C9ZvaimY0B9wCXVZ2zBXgYwMx2A2dIWl91zkXA983slQjL2rEy+SJq4I/UuUaUhtTuO1x/c1UmV2DDmgF6YvW9PW1Y00+sS95BvoCiDBybgEzFdjbcV+kp4HIASduA04FU1TlXAF+r2ndd2Lx1p6RErZtLukbSTkk7Dx061OwzLHvZXIFTV/fT113fp0HnGlFepa+BZqRGZ2rujnWxYU2/1zgWUJSBQzX2VU9heROQkLQL+DjwJDBRfgGpF/hZ4M8qrrkdOIugKWsYuKXWzc1su5kNmdnQ2rVrm3yE5a+R8fLONWq6GamxGkejo/zSibj3cSygKANHFkhXbKeA/ZUnmNlRM/uImW0FPgSsBV6qOOVS4LtmdqDimgNmNmlmU8CXCJrEXJMyuWK5c9G5Vlu3qo/e7q6655IaGZ/k4LHRhj/MpJMDvqDTAooycDwObJZ0ZlhzuALYUXmCpMHwGMBHgUfM7GjFKVdS1UwlaUPF5geAZ1te8g4xOjHJgWMjXuNwkenqEqnBgbqzx0t9IY2O8ksn4hw6FnTCu+hFNqrKzCYkXQc8CMSAO83sOUnXhsfvAM4B7pY0CTwPXF26XlKcYETWx6pe+nOSthI0e71c47ir0/7DI5j5rLguWqkGpgSZHh7eaI2jNEtukR9at7KxArqGRTqrnZk9ADxQte+Oip+/A2ye4doCcEqN/b/U4mJ2rOmpq72pykUnlRjgmezhus7NNDlTc6piXQ4PHNHzzPEO1miGrnPNSCfi5AvjHB+dmPPcbK5Ab6yL9asaGx5ernF4B/mC8MDRwTK5Ij0xcepqz+Fw0Uk3kNmdyRfYlBigq6vWoMyZrV0ZdMJ7B/nC8MDRwTL5AhsHB4g1+EfqXCPKQ3LrCRx1rsNRratLpBKNT6jomjNn4JB0s6QfWYjCuIWVzXkOh4teZcf1XLL5xnM4yvdpYl4s15x6ahy7ge2SHpV0raQ1URfKLYxsvuiTG7rIJeI9rOiNzfmmfnx0gnxhvOkPM83MxOuaM2fgMLM/NLN3EyTonQE8Lemrkt4TdeFcdE6MTvD6ibHy8p7ORUVSsErfHLkc852pOZ2Ic7gwzrGR8aaud/Wrq48jnKH27PDrNYI5pn5V0j0Rls1FKNvksEfnmpFKDMyZPT49PLy5DzPlNc59ssPI1dPH8QVgD/Be4LfM7G1m9t/M7J8C50ddQBeNZhOtnGtGKpxLyqx6urpppRFRzS5jXB695f0ckasnAfBZ4D+GCXnVfJ6oJaqcw+FNVW4BpJNxToxNki+Mk1zRW/OcTK5AvDc24/E579HA6C03P/U0VeWBntJGOL/U+wHM7EhE5XIRy+SKDPTEeNPK5v5InWtEqRYxW3NVNl8knYgjNTc8fDDew8q+bu8gXwD1BI7PVAYIMzsMfCayErkFkc0XSCUGmv4jda4R0+tyzPymHgzFbb7PTVJdfSlu/uoJHLXOiXSOKxe9TL7o/RtuwVTOJVWLmZHJFeY9yq+e0Vtu/uoJHDslfUHSWZLeLOmLwBNRF8xFx8zI5go+osotmFX9PQzGe2bsf8gXxjkxNjnv/5OpxACZ/Oyd8G7+6gkcHwfGgD8lWIlvBPiVKAvlonWkOM6x0QnvGHcLKsjsrl0baNUov3QiTmFsktyJsXm9jpvdnE1OZnYCuGEByuIWSKkq71njbiGlkwPsHj5W81irRvmV+1LyRU5Z2Tev13IzqyePY62kz0t6QNJfl74WonAuGqU/Us8adwspnYiTzReZmjq5GalVH2YamYnXNa+epqqvEMxXdSbw6wSr7j0eYZlcxDz5z7VDKhlnbHKKQ8dHTzqWzRcYjPewqr+nxpX1K+dy+MiqSNUTOE4xsy8D42b2d2b2L4EfjbhcLkLZfJHV/d2sGZjfH6lzjSjlctSqDWTCHI75WtHXTXJFr+dyRKyewFGaMWxY0s9IOh9IRVgmF7HMPKaudq5Z0/0PJweObG5+ORxvuI+vyxG5egLHfwmnUv814NPAHwKfirRULlIZH4rr2mDTYKnG8cbawNSUkc0XW9bnlgr7Ulx0Zg0c4ay4m83siJk9a2bvCSc53FHPi0u6RNIeSXslnTQyS1JC0n2Snpb0mKRzw/1vkbSr4uuopOvDY0lJ35L0vfB7ovHH7lxmVp7awbmF1N8TY92qvpNqAwePjTI2OdX05IbVUskB9s3QCe9aY9bAYWaTwM8288Jh0LkNuBTYAlwpaUvVaTcCu8zsPIL1Pm4N77vHzLaa2VbgbUABuC+85gbgYTPbDDyMDxVuyKFjo4xOTHlTlWuLdPLkVfrKo/xa9H8ynQg64Q8cG2nJ67mT1dNU9X8l/Z6kH5d0Qemrjuu2AXvN7EUzGwPuAS6rOmcLwZs/ZrYbOEPS+qpzLgK+b2avhNuXAXeFP98FvL+OsrhQeby853C4NkgnTl6lL9vimZrrmRfLzU89geNdwI8AvwHcEn7dXMd1m4BMxXY23FfpKeByAEnbgNM5ueP9CuBrFdvrzWwYIPy+rtbNJV0jaaeknYcOHaqjuJ0hW17zwGscbuGlk3GGj4wwMTlV3ld6g29Vv1s9M/G6+aknc7zZJWJrTbta3eh4E3CrpF3AM8CTwET5BaRegqayf9/ozc1sO7AdYGhoyBs7Q/NdZc25+Ugn4kxOGcNHRipqBgXWreqjvyfWkntsStTuhHetM2fgkPSfa+03s9+Y49IskK7YTgH7q17jKPCR8D4CXgq/Si4FvmtmByr2HZC0wcyGJW0ADs71DG5aJlfkTSt7GehtzR+pc41IVeRyVA7PbeUov77uGOtX93kSYITqaao6UfE1SfBmfkYd1z0ObJZ0ZlhzuAJ4w2iscFGo0kpCHwUeCYNJyZW8sZmK8DWuCn++Cri/jrK4UPBH6rUN1x61cjkyudZP8Z8Ol6p10ainqeqWym1JN1MVAGa4bkLSdcCDQAy408yek3RtePwO4BzgbkmTwPPA1RX3iQMXAx+reumbgK9Luhp4FfjgXGVx0zL5AlvTPoLZtceGNf3EulRuRhqfnGL4SJF0orr7c37SyTiPvZRr6Wu6ac0syBQH3lzPiWb2APBA1b47Kn7+DrB5hmsLwCk19r9OMNLKNWhicor9h0f4p+f5iCrXHt2xLjas6S/XOIYPjzBlrR/ll04McP+uIuOTU/TE6mlYcY2op4/jGaY7tWPAWoIRVm6JGT4ywuSUeQ6Ha6t0RWZ3q4filqSScaYM9h8ucvopK1r62q6+Gsf7Kn6eAA6Y2cRMJ7vFy4fiusUgnRzgb/cEQ+Sn84pa38cBwf95DxytV08dbgOQM7NXzGwf0C/pHRGXy0XAk//cYpBOxDl4bJSR8UkyuSKxLrFhTX9r7+HrckSqnsBxO3C8YrsQ7nNLTDZXQIINazxwuPZJJUsJekUy+QKnru6nu8X9EKeuDjvhfUhuJOr5bckqVn43syma61R3bZbJF9mwup/ebu8sdO1TudhSpoXTqVfqjnWxcbDfkwAjUs87yIuSPiGpJ/z6JPBi1AVzrZfJFVo2kZxzzSr1Z2RzhZYt4FTzPomTJ1R0rVFP4LiWYL6qfQTZ4O8AromyUC4amXzBO8Zd261d2Udvdxd7Dx7n0LHRyEb5BUmAXuOIQj0JgAcJsr7dEjY6McmBo6PeMe7arqtLpBIDPBom6EX1fzKdHOC140EnfKvmwXKBOWscku6SNFixnZB0Z6Slci23z4fiukUknYiz+wfHyj9Hco9Sk5g3V7VcPU1V55nZ4dKGmeWB8yMrkYtEJt/aqaudm4/K/4dRzZ2W8llyI1NP4OiqXJ5VUhIfVbXklMaze9a4WwxK/w97u7tYt6ovmnskTp5Q0bVGPQHgFoJVAO8Ntz8I/FZ0RXJRyOQL9MTE+tWtTbRyrhmlN/XU4ABdXbWW7pm/tav66Ovu8iTACNTTOX63pJ3ATxIsznS5mT0feclcS2VzRTYNDhCL6I/UuUaUOsSjHB4uBZ3w3lTVenU1OYWB4nlJZwFXSvq6mZ0bbdE6w4nRCT6z4zlOjEY7/dejL+X4kY2rI72Hc/Uq1TjSEfe5pZNxHn3pdf71nzwR6X2acV5qkH/9E2e1uxhNqWd23A3APwf+BXAe8NsECyy5Fnj85Rz3PpHl9FPi9EWY0f2mlb2877wNkb2+c40YjPfwzy5Icem50f6ffN95G9l/uMj3Dx2f++QF9PrxMR5+4SAfu/DNkTXVRWnGwCHpXxEEiBTwdYIV+u43s19foLJ1hNJopz+95p2c2uKJ3pxbrCRxy8+/NfL7/NzbUvzc21KR36dR/+P/vcJ/+otnOXhsdEn+3c9W47gN+A7wL8xsJ4Akm+V814RsrhDpyBLn3OJTaqLL5AtLMnDM1jayEbgH+IKkPZJ+E+hZmGJ1jmy+GOnIEufc4rPUkxNnDBxm9pqZ3W5mFxIs1XoEOCjpBUk+HLdFMnmfeNC5TrNpcGknJ9bVG2tmWTO72czeBrwfGI20VB0kkytEPrLEObe49PfEWL+6b8nmmDScAW5mewDvIG+B46MT5AvjkU254JxbvFJLeNr3SFf0kXRJ2D+yV9INNY4nJN0n6WlJj0k6t+LYoKR7Je0Om8feGe7/rKR9knaFX++N8hmiND0NiNc4nOs06SWcnBhZ4JAUIxiZdSmwhSBxcEvVaTcCu8zsPOBDwK0Vx24FvmlmZwNvBV6oOPZFM9safj0Q1TNErRw4vMbhXMdJJ+MMHykyPjnV7qI0rJ4EwAtq7D4CvGJms6U7bwP2mtmL4evcA1wGVE5XsoUgoRAz2y3pDEnrgSJwIfDh8NgYMDbn0ywx2dJU59457lzHSSfiTBkMHx7htFOW1ntAPTWO3wf+H7Ad+BJBbsc9wD9I+ulZrtsEZCq2s+G+Sk8BlwNI2gacTpBw+GbgEPBHkp6U9IeSVlRcd13YvHVn5cy9lSRdI2mnpJ2HDh2q4zEXXiZfYEVvjETcRzk712lSYRP1UhySW0/geBk438yGwlFV5wPPAj8FfG6W62olJlQnEN4EJCTtAj4OPAlMENSELgBuN7PzgRNAqY/kduAsYCswTDB778k3Mtselnlo7dq1czxie2RyRdLJOJLncDjXaZbytO/1jKo628yeK22Y2fOSzjezF+d4w8sC6YrtFLC/8gQzOwp8BEDBi70UfsWBrJk9Gp56L2HgMLMDpeslfQn4yzqeYVHK5gu+sJJzHWrDmn5iXVqSHeT11Dj2SLpd0j8Ov36foJmqDxif5brHgc2SzpTUS7Bu+Y7KE8KRU73h5keBR8zsqJn9AMhIekt47CLCvpFw0sWSDxDUfpYcMyOTK/hQXOc6VHesiw1r+pdtjePDwC8D1xM0P30b+DRB0HjPTBeZ2YSk64AHgRhwp5k9J+na8PgdwDnA3ZImCQLD1RUv8XHgK2FgeZGwZgJ8TtJWgmavl4GP1fEMi06+MM6JsUnvGHeug6UT8SWZBFjPQk5Fgn6EWn0Js85VHA6VfaBq3x0VP38H2DzDtbuAoRr7f2muMi8F00NxvanKuU6VTg7wN3sW5+Cd2dQzHPfdwGcJRjyVzzezN0dXrOXPh+I659KJOIeOjTIyPkl/T6zdxalbPU1VXwY+BTwBTEZbnM5Ratf0wOFc55qeJbfID61b2ebS1K+ewHHEzP4q8pJ0mEyuQCLew8q+hqcLc84tE6XphjL5wrILHH8j6fPAN6iYFdfMvhtZqTpAJl/0EVXOdbjSe0B2iXWQ1xM43hF+r+yoNuAnW1+czpHNFTh7w6p2F8M510ZrV/bR291VXkJ6qahnVNWMQ25dc6amjGy+yMVb1re7KM65NurqEqnEwJIbkjtj4JD0i2b2J5J+tdZxM/tCdMVa3g4dH2VscspX/nPOBbkcSywJcLYaR2lSwVrtKdVzTrkGeA6Hc64knRxgV+Zwu4vRkBkDh5n9Qfjj/zazv688FuZ2uCb5UFznXEk6EedIcZyjI+Os7l8aM2XXM1fV79a5z9WpNKlZacF651znmh5ZtXQ6yGfr43gn8C5gbVU/x2qCuadckzK5AutW9S2pTFHnXDQqczm2bFzd5tLUZ7Y+jl5gZXhOZT/HUeDnoizUcpfJF7yZyjkHVKzLsYRGVs3Wx/F3wN9J+mMzewVAUhewMlxHwzUpkyvy9jNqLlzonOswg+EMEtkllMtRTx/Hb0taHS7d+jzB+hz/JuJyLVsTk1P84OiI1ziccwBISy+Xo57AsSWsYbyfYIr004BlMbV5OwwfGWFyysrVU+ecSyeXVi5HPYGjR1IPQeC438zG8TyOppU+VZQWqnfOuXQiTjZfxGxpvLXWEzj+gGClvRXAI5JOJ+ggd00o53B4jcM5F0olBiiMTZI7MdbuotRlzsBhZr9jZpvM7L0WeIVZlox1s8vkisS6xIY1/e0uinNukSj1eS6VyQ7nDByS1kv6sqS/Cre3AFdFXrJlKpMvsGFNP92xeip7zrlOUM7lWCId5PW8e/0x8CCwMdz+B+D6iMqz7GXzRW+mcs69QTmXY4l0kM8YOCSVcjzeZGZfB6YAzGyCOpeQlXSJpD2S9kq6ocbxhKT7JD0t6TFJ51YcG5R0r6Tdkl4IM9mRlJT0LUnfC78vqYSITK5Q/nThnHMAK/q6Sa7oLU9HtNjNVuN4LPx+QtIphCOpJP0ocGSuF5YUA24DLgW2AFeGzVyVbgR2mdl5wIeAWyuO3Qp808zOBt4KvBDuvwF42Mw2Aw+H20vCyPgkB4+Neo3DOXeSdGKA7FKvcQAKv/8qsAM4S9LfA3cDH6/jtbcBe83sRTMbA+4BLqs6ZwvBmz9mths4I+xTWQ1cCHw5PDZmZofDay4D7gp/votgmPCSUMoM9aG4zrlqqXBI7lIw21xVlZMb3keQ/CeCdcd/Cnh6jtfeBGQqtrNML0Nb8hRwOfBtSduA04EUQVPYIeCPJL0VeAL4pJmdANab2TCAmQ1LWjdHORYNH4rrnJtJKjnAt54/wNSU0dWluS9oo9lqHDGCSQ5XEeRwdIf74tRe3KlarSevzm65CUhI2kVQi3kSmAjvdQFwu5mdD5ygwSYpSddI2ilp56FDhxq5NDKlBel9uhHnXLV0Is7Y5BQHjo20uyhzmq3GMWxmvzGP184C6YrtFLC/8oRwKpOPAEgS8FL4FQeyZvZoeOq9TAeOA5I2hLWNDcDBWjc3s+3AdoChoaFFkY6ZyRfp7e5i7cq+dhfFObfIlHM5ckU2rFnczdn19HE063Fgs6QzJfUCVxD0lUzfIBg51RtufhR4xMyOmtkPgIykt4THLiKYYJHwNUp5JFcB98+znAsmmy+QSgws+mqoc27hlZaSXgq5HLPVOC6azwub2YSk6whyQGLAnWb2nKRrw+N3AOcAd0uaJAgMV1e8xMeBr4SB5UXCmglB89bXJV0NvAp8cD7lXEiZnOdwOOdq25QYQFoauRyzrceRm++Lm9kDBJ3qlfvuqPj5O8DmGa7dBQzV2P868wxq7ZLJF3hrek27i+GcW4T6umOsX9W/JHI5fN6LBXJsZJzDhfHy+sLOOVcttURyOTxwLJDSpwhvqnLOzSSdXBq5HB44Fkg5h8OT/5xzM0gnBhg+UmR8cqrdRZmVB44FUvoU4TUO59xMUsk4Uwb7Dy/uWocHjgWSyRVY2dfNYLyn3UVxzi1S5VlyF3kHuQeOBVLK4QjyHJ1z7mTldTkWeQe5B44FkskVfUSVc25Wp67uJ9alRZ8E6IFjAZgZmbyvw+Gcm113rIuNg/2LfmSVB44FkDsxRmFs0jvGnXNzSifi3lTlpheg91lxnXNzSSfi3jnuKGeCelOVc24u6eQArx0fpThW1wrdbeGBYwF41rhzrl6llonFPPWIB44FkMkXSK7oZUXfbJMRO+cc5dGXi7mfwwPHAsjkghwO55yby/S6HIu3n8MDxwLI5n0dDudcfdau6qOvu8ubqjrZ1JSxL18k5R3jzrk6SCKVGPAaRyc7cGyEsckpr3E45+qWTi7uXA4PHBHLeg6Hc65BQS6HB46OVfrlp71z3DlXp3RygKMjExwpjre7KDV54IhYqZ1y46AHDudcfcpDchdprcMDR8Qy+QLrV/fR3xNrd1Gcc0tEqU90sY6sijRwSLpE0h5JeyXdUON4QtJ9kp6W9JikcyuOvSzpGUm7JO2s2P9ZSfvC/bskvTfKZ5ivTK7gHePOuYaUpidarLPkRpbKLCkG3AZcDGSBxyXtMLPnK067EdhlZh+QdHZ4/kUVx99jZq/VePkvmtnNUZW9lbL5ItvOTLa7GM65JWTNQA+r+ro7sqlqG7DXzF40szHgHuCyqnO2AA8DmNlu4AxJ6yMs04Ian5xi+EjRO8adcw2RRCoZL8+svdhEGTg2AZmK7Wy4r9JTwOUAkrYBpwOp8JgBD0l6QtI1VdddFzZv3SkpUevmkq6RtFPSzkOHDs33WZoyfHiEKQsWoHfOuUakEwMdWeOotbi2VW3fBCQk7QI+DjwJTITH3m1mFwCXAr8i6cJw/+3AWcBWYBi4pdbNzWy7mQ2Z2dDatWvn8xxNKyXweB+Hc65R6WScbL6IWfXbZvtFOV1rFkhXbKeA/ZUnmNlR4CMAkgS8FH5hZvvD7wcl3UfQ9PWImR0oXS/pS8BfRvgM81L6tOATHDrnGpVKDFAcn+S142OsXdXX7uK8QZQ1jseBzZLOlNQLXAHsqDxB0mB4DOCjBIHhqKQVklaF56wAfhp4NtzeUPESHyjtX4wy+QKxLrFhTX+7i+KcW2LSi3h69chqHGY2Iek64EEgBtxpZs9JujY8fgdwDnC3pEngeeDq8PL1wH1BJYRu4Ktm9s3w2OckbSVo9noZ+FhUzzBfmVyRjYP9dMc8XcY515jpBZ2KXHBaza7ctol0ZSEzewB4oGrfHRU/fwfYXOO6F4G3zvCav9TiYkYmk/ccDudcc1LldTkWX43DPwpHyNfhcM41a0VfN6es6F2U2eMeOCIyMj7JoWOj5QxQ55xrVCoZX5TrcnjgiEjpU0LKaxzOuSalEgOLsnPcA0dESp8SvMbhnGtWOhFn/+Eik1OLK5fDA0dEPPnPOTdf6eQA45PGD46OtLsob+CBIyKZXIG+7q5Fl7jjnFs6ytOrL7KRVR44IpLNF0klBghzUZxzrmGlXI7FNtmhB46IZPIFX2fcOTcvGwf7kRZfLocHjohkcp7D4Zybn77uGKeu7l90I6s8cETg6Mg4R4rjPrmhc27eUokBsossl8MDRwRK1UpvqnLOzVc6EfcaRyco53B4U5Vzbp5SyTg/ODrC6MRku4tS5oEjAqWscU/+c87NVzoxgFmwouhi4YEjAtl8kVV93awZ6Gl3UZxzS9z0kNzF01zlgSMCmVyBVDLuORzOuXkrB45F1EHugSMCmXzBR1Q551ri1NX9dHfJaxzLmZl5DodzrmViXWLj4MCiSgL0wNFir58Yozg+6R3jzrmWSScHFtW0Ix44Wqycw+E1Dudci6QT8UU10aEHjhbL5kvrcHjgcM61RjoZ5/UTYxTGJtpdFMADR8tlyiv/eVOVc641Su8n2UXSXBVp4JB0iaQ9kvZKuqHG8YSk+yQ9LekxSedWHHtZ0jOSdknaWbE/Kelbkr4Xfk9E+QyNyuSKnLKilxV93e0uinNumZgekrs4mqsiCxySYsBtwKXAFuBKSVuqTrsR2GVm5wEfAm6tOv4eM9tqZkMV+24AHjazzcDD4faikfWhuM65Fiu9pyyWwBHlx+JtwF4zexFA0j3AZcDzFedsAX4bwMx2SzpD0nozOzDL614G/ET4813A3wL/rrVFD/zuw99jx1P7G7rmlVyBi7esj6I4zrkOtXZlH/09XfzOX+/lK4++2tC1v3X5P+LtZyRbWp4oA8cmIFOxnQXeUXXOU8DlwLclbQNOB1LAAcCAhyQZ8Admtj28Zr2ZDQOY2bCkdbVuLuka4BqA0047rakHWLuqj83rVzZ0zQ+vX8UvvuP0pu7nnHO1SOLXLn4LT2byDV870BNreXmiDBy15tuwqu2bgFsl7QKeAZ4ESsMG3m1m+8PA8C1Ju83skXpvHgaa7QBDQ0PV963LFdtO44ptzQUd55xrpX914ZvbXYSyKANHFkhXbKeAN7T7mNlR4CMACiZ2ein8wsz2h98PSrqPoOnrEeCApA1hbWMDcDDCZ3DOOVclylFVjwObJZ0pqRe4AthReYKkwfAYwEeBR8zsqKQVklaF56wAfhp4NjxvB3BV+PNVwP0RPoNzzrkqkdU4zGxC0nXAg0AMuNPMnpN0bXj8DuAc4G5JkwSd5leHl68H7gtnl+0Gvmpm3wyP3QR8XdLVwKvAB6N6BueccyeTWVPN/0vK0NCQ7dy5c+4TnXPOlUl6oiodAvDMceeccw3ywOGcc64hHjicc841xAOHc865hnRE57ikQ8ArwJuA19pcnHbq5Ofv5GeHzn7+Tn52mN/zn25ma6t3dkTgKJG0s9YIgU7Ryc/fyc8Onf38nfzsEM3ze1OVc865hnjgcM4515BOCxzb5z5lWevk5+/kZ4fOfv5OfnaI4Pk7qo/DOefc/HVajcM559w8eeBwzjnXkI4JHJIukbRH0l5Ji2qd8oUg6WVJz0jaJWlZz/go6U5JByU9W7EvKelbkr4Xfk+0s4xRmeHZPytpX/i73yXpve0sY1QkpSX9jaQXJD0n6ZPh/k753c/0/C3//XdEH4ekGPAPwMUEC0w9DlxpZs/PeuEyIullYMjMln0ilKQLgePA3WZ2brjvc0DOzG4KPzgkzCySterbaYZn/yxw3MxubmfZohYu7LbBzL4brufzBPB+4MN0xu9+puf/eVr8+++UGsc2YK+ZvWhmY8A9wGVtLpOLSLjEcK5q92XAXeHPdxH8QS07Mzx7RzCzYTP7bvjzMeAFYBOd87uf6flbrlMCxyYgU7GdJaJ/0EXMgIckPSHpmnYXpg3Wm9kwBH9gwLo2l2ehXSfp6bApa1k21VSSdAZwPvAoHfi7r3p+aPHvv1MCh2rsW/5tdG/0bjO7ALgU+JWwScN1htuBs4CtwDBwS1tLEzFJK4E/B643s6PtLs9Cq/H8Lf/9d0rgyALpiu0UsL9NZWkLM9sffj8I3EfQfNdJDoRtwKW24INtLs+CMbMDZjZpZlPAl1jGv3tJPQRvml8xs2+Euzvmd1/r+aP4/XdK4Hgc2CzpTEm9wBXAjjaXacFIWhF2liFpBfDTwLOzX7Xs7ACuCn++Cri/jWVZUKU3zdAHWKa/e0kCvgy8YGZfqDjUEb/7mZ4/it9/R4yqAgiHoP13IAbcaWb/tb0lWjiS3kxQywDoBr66nJ9f0teAnyCYTvoA8BngL4CvA6cBrwIfNLNl14k8w7P/BEEzhQEvAx8rtfkvJ5J+DPg/wDPAVLj7RoJ2/k743c/0/FfS4t9/xwQO55xzrdEpTVXOOedaxAOHc865hnjgcM451xAPHM455xrigcM551xDPHC4ZUPS30r6J1X7rpf0+3NcMxRxub4WTvfwqar9n5X06fDn/nDm1s/UuP6D4YynfzOPMhyv+Pm94Uyxp4VlKEhaN8O5JumWiu1Ph5Mmug7mgcMtJ18jSO6sdEW4vy0knQq8y8zOM7MvznBOL0G27xNm9us1Trka+GUze0+d9+ye5dhFwO8Cl5jZq+Hu14Bfm+GSUeBySW+q596uM3jgcMvJvcD7JPVBeaK3jcC3Jd0uaWe4TkGtN+fqT9o/J+mPw5/XSvpzSY+HX++ucW2/pD9SsObJk5JKb/IPAevCdRB+vMZtuwlma/6emZ20Toyk/wz8GHCHpM/PdB9JH5b0Z5L+Z3jPWs/34wRTTvyMmX2/4tCdwD+XlKxx2QTBmtWfqnHMdSgPHG7ZMLPXgceAS8JdVwB/akGW638wsyHgPOAfSzqvgZe+Ffiimb0d+GfAH9Y451fCMvwjgkzduyT1Az8LfN/MtprZ/6lx3b8FJszs+hme6TeAncAvmNm/meU+AO8ErjKzn6zxUn0EU22838x2Vx07ThA8PlmrDMBtwC9IWjPDcddhPHC45aayuaqymernJX0XeBL4EWBLA6/5U8DvSdpFMO/R6tLcXxV+DPgfAOEb8yvAD9fx2t8G3impnnPnus+3ZplKYxz4vwTNXrX8DnCVpNXVB8IZVu8GPlFnGd0y54HDLTd/AVwk6QJgIFwN7Uzg08BFZnYe8L+A/hrXVs6/U3m8C3hnWGvYamabwoVyKtWaur8ejwDXA38laWMd5892nxOzHJsiWAnu7ZJurD5oZoeBrwK/PMP1/50g6Kyoo4xumfPA4ZYVMzsO/C1B00uptrGa4E31iKT1BGuS1HJA0jmSughmES15CLiutCFpa41rHwF+ITz+wwQT6u2ps8x/Dnwe+KakwTlOn899CsD7CJqdatU8vgB8jKDfpfraHMFEgTPVWFwH8cDhlqOvAW8l6HTGzJ4iaKJ6jiCg/P0M190A/CXw1wQL3pR8AhgKh9Q+D1xb49rfB2KSngH+FPiwmY3WW2AzuwP4BrCjos+ilvneJ0fQB/QfJV1Wdew1glmU+2a4/BaCWXddh/PZcZ1zzjXEaxzOOeca4oHDOedcQzxwOOeca4gHDueccw3xwOGcc64hHjicc841xAOHc865hvx/9DC05L98ys4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# import Matplotlib (scientific plotting library)\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# allow plots to appear within the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# plot the relationship between K and testing accuracy\n",
    "plt.plot(k_range, scores)\n",
    "plt.xlabel('Value of K for KNN')\n",
    "plt.ylabel('Testing Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Training accuracy** rises as model complexity increases\n",
    "- **Testing accuracy** penalizes models that are too complex or not complex enough\n",
    "- For KNN models, complexity is determined by the **value of K** (lower value = more complex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions on out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate the model with the best known parameters\n",
    "knn = KNeighborsClassifier(n_neighbors=11)\n",
    "\n",
    "# train the model with X and y (not X_train and y_train)\n",
    "knn.fit(X, y)\n",
    "\n",
    "# make a prediction for an out-of-sample observation\n",
    "knn.predict([[3, 5, 4, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downsides of train/test split?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Provides a **high-variance estimate** of out-of-sample accuracy\n",
    "- **K-fold cross-validation** overcomes this limitation\n",
    "- But, train/test split is still useful because of its **flexibility and speed**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "- Quora: [What is an intuitive explanation of overfitting?](https://www.quora.com/What-is-an-intuitive-explanation-of-over-fitting-particularly-with-a-small-sample-set-What-are-you-essentially-doing-by-over-fitting-How-does-the-over-promise-of-a-high-R%C2%B2-low-standard-error-occur/answer/Jessica-Su)\n",
    "- Video: [Estimating prediction error](https://www.youtube.com/watch?v=_2ij6eaaSl0&list=PL5-da3qGB5IA6E6ZNXu7dp89_uv8yocmf&index=1&t=2m34s) (12 minutes, starting at 2:34) by Hastie and Tibshirani\n",
    "- [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html)\n",
    "    - [Guiding questions](https://github.com/justmarkham/DAT8/blob/master/homework/09_bias_variance.md) when reading this article\n",
    "- Video: [Visualizing bias and variance](http://work.caltech.edu/library/081.html) (15 minutes) by Abu-Mostafa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comments or Questions?\n",
    "\n",
    "- Email: <kevin@dataschool.io>\n",
    "- Website: https://www.dataschool.io\n",
    "- Twitter: [@justmarkham](https://twitter.com/justmarkham)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}