notebooks/13_decision_trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 13 - Decision Trees\n",
    "\n",
    "by [Alejandro Correa Bahnsen](albahnsen.com/)\n",
    "\n",
    "version 0.1, Apr 2016\n",
    "\n",
    "## Part of the class [Practical Machine Learning](https://github.com/albahnsen/PracticalMachineLearningClass)\n",
    "\n",
    "\n",
    "\n",
    "This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). Special thanks goes to [Kevin Markham](https://github.com/justmarkham)\n",
    "\n",
    "*Adapted from Chapter 8 of [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why are we learning about decision trees?\n",
    "\n",
    "- Can be applied to both regression and classification problems\n",
    "- Many useful properties\n",
    "- Very popular\n",
    "- Basis for more sophisticated models\n",
    "- Have a different way of \"thinking\" than the other models we have studied"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson objectives\n",
    "\n",
    "Students will be able to:\n",
    "\n",
    "- Explain how a decision tree is created\n",
    "- Build a decision tree model in scikit-learn\n",
    "- Tune a decision tree model and explain how tuning impacts the model\n",
    "- Interpret a tree diagram\n",
    "- Describe the key differences between regression and classification trees\n",
    "- Decide whether a decision tree is an appropriate model for a given problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Regression trees\n",
    "\n",
    "Major League Baseball player data from 1986-87:\n",
    "\n",
    "- **Years** (x-axis): number of years playing in the major leagues\n",
    "- **Hits** (y-axis): number of hits in the previous year\n",
    "- **Salary** (color): low salary is blue/green, high salary is red/yellow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Salary data](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/salary_color.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Group exercise:\n",
    "\n",
    "- The data above is our **training data**.\n",
    "- We want to build a model that predicts the Salary of **future players** based on Years and Hits.\n",
    "- We are going to \"segment\" the feature space into regions, and then use the **mean Salary in each region** as the predicted Salary for future players.\n",
    "- Intuitively, you want to **maximize** the similarity (or \"homogeneity\") within a given region, and **minimize** the similarity between different regions.\n",
    "\n",
    "Rules for segmenting:\n",
    "\n",
    "- You can only use **straight lines**, drawn one at a time.\n",
    "- Your line must either be **vertical or horizontal**.\n",
    "- Your line **stops** when it hits an existing line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Salary regions](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/salary_regions.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above are the regions created by a computer:\n",
    "\n",
    "- $R_1$: players with **less than 5 years** of experience, mean Salary of **\\$166,000 **\n",
    "- $R_2$: players with **5 or more years** of experience and **less than 118 hits**, mean Salary of **\\$403,000 **\n",
    "- $R_3$: players with **5 or more years** of experience and **118 hits or more**, mean Salary of **\\$846,000 **\n",
    "\n",
    "**Note:** Years and Hits are both integers, but the convention is to use the **midpoint** between adjacent values to label a split.\n",
    "\n",
    "These regions are used to make predictions on **out-of-sample data**. Thus, there are only three possible predictions! (Is this different from how **linear regression** makes predictions?)\n",
    "\n",
    "Below is the equivalent regression tree:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Salary tree](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/salary_tree.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first split is **Years < 4.5**, thus that split goes at the top of the tree. When a splitting rule is **True**, you follow the left branch. When a splitting rule is **False**, you follow the right branch.\n",
    "\n",
    "For players in the **left branch**, the mean Salary is \\$166,000, thus you label it with that value. (Salary has been divided by 1000 and log-transformed to 5.11.)\n",
    "\n",
    "For players in the **right branch**, there is a further split on **Hits < 117.5**, dividing players into two more Salary regions: \\$403,000 (transformed to 6.00), and \\$846,000 (transformed to 6.74)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Salary tree annotated](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/salary_tree_annotated.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What does this tree tell you about your data?**\n",
    "\n",
    "- Years is the most important factor determining Salary, with a lower number of Years corresponding to a lower Salary.\n",
    "- For a player with a lower number of Years, Hits is not an important factor determining Salary.\n",
    "- For a player with a higher number of Years, Hits is an important factor determining Salary, with a greater number of Hits corresponding to a higher Salary.\n",
    "\n",
    "**Question:** What do you like and dislike about decision trees so far?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a regression tree by hand\n",
    "\n",
    "Your **training data** is a tiny dataset of [used vehicle sale prices](https://raw.githubusercontent.com/justmarkham/DAT8/master/data/vehicles_train.csv). Your goal is to **predict price** for testing data.\n",
    "\n",
    "1. Read the data into a Pandas DataFrame.\n",
    "2. Explore the data by sorting, plotting, or split-apply-combine (aka `group_by`).\n",
    "3. Decide which feature is the most important predictor, and use that to create your first splitting rule.\n",
    "    - Only binary splits are allowed.\n",
    "4. After making your first split, split your DataFrame into two parts, and then explore each part to figure out what other splits to make.\n",
    "5. Stop making splits once you are convinced that it strikes a good balance between underfitting and overfitting.\n",
    "    - Your goal is to build a model that generalizes well.\n",
    "    - You are allowed to split on the same variable multiple times!\n",
    "6. Draw your tree, labeling the leaves with the mean price for the observations in that region.\n",
    "    - Make sure nothing is backwards: You follow the **left branch** if the rule is true, and the **right branch** if the rule is false."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does a computer build a regression tree?\n",
    "\n",
    "**Ideal approach:** Consider every possible partition of the feature space (computationally infeasible)\n",
    "\n",
    "**\"Good enough\" approach:** recursive binary splitting\n",
    "\n",
    "1. Begin at the top of the tree.\n",
    "2. For **every feature**, examine **every possible cutpoint**, and choose the feature and cutpoint such that the resulting tree has the lowest possible mean squared error (MSE). Make that split.\n",
    "3. Examine the two resulting regions, and again make a **single split** (in one of the regions) to minimize the MSE.\n",
    "4. Keep repeating step 3 until a **stopping criterion** is met:\n",
    "    - maximum tree depth (maximum number of splits required to arrive at a leaf)\n",
    "    - minimum number of observations in a leaf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Demo: Choosing the ideal cutpoint for a given feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# vehicle data\n",
    "import pandas as pd\n",
    "url = 'https://raw.githubusercontent.com/justmarkham/DAT8/master/data/vehicles_train.csv'\n",
    "train = pd.read_csv(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>year</th>\n",
       "      <th>miles</th>\n",
       "      <th>doors</th>\n",
       "      <th>vtype</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22000</td>\n",
       "      <td>2012</td>\n",
       "      <td>13000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14000</td>\n",
       "      <td>2010</td>\n",
       "      <td>30000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13000</td>\n",
       "      <td>2010</td>\n",
       "      <td>73500</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9500</td>\n",
       "      <td>2009</td>\n",
       "      <td>78000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9000</td>\n",
       "      <td>2007</td>\n",
       "      <td>47000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>2006</td>\n",
       "      <td>124000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3000</td>\n",
       "      <td>2004</td>\n",
       "      <td>177000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2000</td>\n",
       "      <td>2004</td>\n",
       "      <td>209000</td>\n",
       "      <td>4</td>\n",
       "      <td>truck</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3000</td>\n",
       "      <td>2003</td>\n",
       "      <td>138000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1900</td>\n",
       "      <td>2003</td>\n",
       "      <td>160000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2500</td>\n",
       "      <td>2003</td>\n",
       "      <td>190000</td>\n",
       "      <td>2</td>\n",
       "      <td>truck</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>5000</td>\n",
       "      <td>2001</td>\n",
       "      <td>62000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1800</td>\n",
       "      <td>1999</td>\n",
       "      <td>163000</td>\n",
       "      <td>2</td>\n",
       "      <td>truck</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1300</td>\n",
       "      <td>1997</td>\n",
       "      <td>138000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>6571.428571</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    price  year   miles  doors  vtype   prediction\n",
       "0   22000  2012   13000      2    car  6571.428571\n",
       "1   14000  2010   30000      2    car  6571.428571\n",
       "2   13000  2010   73500      4    car  6571.428571\n",
       "3    9500  2009   78000      4    car  6571.428571\n",
       "4    9000  2007   47000      4    car  6571.428571\n",
       "5    4000  2006  124000      2    car  6571.428571\n",
       "6    3000  2004  177000      4    car  6571.428571\n",
       "7    2000  2004  209000      4  truck  6571.428571\n",
       "8    3000  2003  138000      2    car  6571.428571\n",
       "9    1900  2003  160000      4    car  6571.428571\n",
       "10   2500  2003  190000      2  truck  6571.428571\n",
       "11   5000  2001   62000      4    car  6571.428571\n",
       "12   1800  1999  163000      2  truck  6571.428571\n",
       "13   1300  1997  138000      4    car  6571.428571"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# before splitting anything, just predict the mean of the entire dataset\n",
    "train['prediction'] = train.price.mean()\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5936.9819859959835"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate RMSE for those predictions\n",
    "from sklearn import metrics\n",
    "import numpy as np\n",
    "np.sqrt(metrics.mean_squared_error(train.price, train.prediction))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define a function that calculates the RMSE for a given split of miles\n",
    "def mileage_split(miles):\n",
    "    lower_mileage_price = train[train.miles < miles].price.mean()\n",
    "    higher_mileage_price = train[train.miles >= miles].price.mean()\n",
    "    train['prediction'] = np.where(train.miles < miles, lower_mileage_price, higher_mileage_price)\n",
    "    return np.sqrt(metrics.mean_squared_error(train.price, train.prediction))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3984.09174254\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>year</th>\n",
       "      <th>miles</th>\n",
       "      <th>doors</th>\n",
       "      <th>vtype</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22000</td>\n",
       "      <td>2012</td>\n",
       "      <td>13000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>15000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14000</td>\n",
       "      <td>2010</td>\n",
       "      <td>30000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>15000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13000</td>\n",
       "      <td>2010</td>\n",
       "      <td>73500</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9500</td>\n",
       "      <td>2009</td>\n",
       "      <td>78000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9000</td>\n",
       "      <td>2007</td>\n",
       "      <td>47000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>15000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>2006</td>\n",
       "      <td>124000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3000</td>\n",
       "      <td>2004</td>\n",
       "      <td>177000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2000</td>\n",
       "      <td>2004</td>\n",
       "      <td>209000</td>\n",
       "      <td>4</td>\n",
       "      <td>truck</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3000</td>\n",
       "      <td>2003</td>\n",
       "      <td>138000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1900</td>\n",
       "      <td>2003</td>\n",
       "      <td>160000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2500</td>\n",
       "      <td>2003</td>\n",
       "      <td>190000</td>\n",
       "      <td>2</td>\n",
       "      <td>truck</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>5000</td>\n",
       "      <td>2001</td>\n",
       "      <td>62000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1800</td>\n",
       "      <td>1999</td>\n",
       "      <td>163000</td>\n",
       "      <td>2</td>\n",
       "      <td>truck</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1300</td>\n",
       "      <td>1997</td>\n",
       "      <td>138000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>4272.727273</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    price  year   miles  doors  vtype    prediction\n",
       "0   22000  2012   13000      2    car  15000.000000\n",
       "1   14000  2010   30000      2    car  15000.000000\n",
       "2   13000  2010   73500      4    car   4272.727273\n",
       "3    9500  2009   78000      4    car   4272.727273\n",
       "4    9000  2007   47000      4    car  15000.000000\n",
       "5    4000  2006  124000      2    car   4272.727273\n",
       "6    3000  2004  177000      4    car   4272.727273\n",
       "7    2000  2004  209000      4  truck   4272.727273\n",
       "8    3000  2003  138000      2    car   4272.727273\n",
       "9    1900  2003  160000      4    car   4272.727273\n",
       "10   2500  2003  190000      2  truck   4272.727273\n",
       "11   5000  2001   62000      4    car   4272.727273\n",
       "12   1800  1999  163000      2  truck   4272.727273\n",
       "13   1300  1997  138000      4    car   4272.727273"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate RMSE for tree which splits on miles < 50000\n",
    "print('RMSE:', mileage_split(50000))\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3530.14653008\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>year</th>\n",
       "      <th>miles</th>\n",
       "      <th>doors</th>\n",
       "      <th>vtype</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22000</td>\n",
       "      <td>2012</td>\n",
       "      <td>13000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>12083.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14000</td>\n",
       "      <td>2010</td>\n",
       "      <td>30000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>12083.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13000</td>\n",
       "      <td>2010</td>\n",
       "      <td>73500</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>12083.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9500</td>\n",
       "      <td>2009</td>\n",
       "      <td>78000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>12083.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9000</td>\n",
       "      <td>2007</td>\n",
       "      <td>47000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>12083.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>2006</td>\n",
       "      <td>124000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3000</td>\n",
       "      <td>2004</td>\n",
       "      <td>177000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2000</td>\n",
       "      <td>2004</td>\n",
       "      <td>209000</td>\n",
       "      <td>4</td>\n",
       "      <td>truck</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3000</td>\n",
       "      <td>2003</td>\n",
       "      <td>138000</td>\n",
       "      <td>2</td>\n",
       "      <td>car</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1900</td>\n",
       "      <td>2003</td>\n",
       "      <td>160000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2500</td>\n",
       "      <td>2003</td>\n",
       "      <td>190000</td>\n",
       "      <td>2</td>\n",
       "      <td>truck</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>5000</td>\n",
       "      <td>2001</td>\n",
       "      <td>62000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>12083.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1800</td>\n",
       "      <td>1999</td>\n",
       "      <td>163000</td>\n",
       "      <td>2</td>\n",
       "      <td>truck</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1300</td>\n",
       "      <td>1997</td>\n",
       "      <td>138000</td>\n",
       "      <td>4</td>\n",
       "      <td>car</td>\n",
       "      <td>2437.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    price  year   miles  doors  vtype    prediction\n",
       "0   22000  2012   13000      2    car  12083.333333\n",
       "1   14000  2010   30000      2    car  12083.333333\n",
       "2   13000  2010   73500      4    car  12083.333333\n",
       "3    9500  2009   78000      4    car  12083.333333\n",
       "4    9000  2007   47000      4    car  12083.333333\n",
       "5    4000  2006  124000      2    car   2437.500000\n",
       "6    3000  2004  177000      4    car   2437.500000\n",
       "7    2000  2004  209000      4  truck   2437.500000\n",
       "8    3000  2003  138000      2    car   2437.500000\n",
       "9    1900  2003  160000      4    car   2437.500000\n",
       "10   2500  2003  190000      2  truck   2437.500000\n",
       "11   5000  2001   62000      4    car  12083.333333\n",
       "12   1800  1999  163000      2  truck   2437.500000\n",
       "13   1300  1997  138000      4    car   2437.500000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate RMSE for tree which splits on miles < 100000\n",
    "print('RMSE:', mileage_split(100000))\n",
    "train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# check all possible mileage splits\n",
    "mileage_range = range(train.miles.min(), train.miles.max(), 1000)\n",
    "RMSE = [mileage_split(miles) for miles in mileage_range]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# allow plots to appear in the notebook\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = (6, 4)\n",
    "plt.rcParams['font.size'] = 14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7fca44ca7390>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbIAAAEZCAYAAADyqKAxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcXFWd/vHPk5aYBZMAQjICssygLOICKCGAtiigMurI\nOCqjwIjDD5TRgDogA04SRpQBHVkcBsQMAo4wCkrEhRCWBgwEFBSRJTIsgbAlLIEsJHSS7++Pc9vc\nLrq7qrpru1XP+/WqV7ruPXXrVHHpp8+5556jiMDMzKyoRjW7AmZmZiPhIDMzs0JzkJmZWaE5yMzM\nrNAcZGZmVmgOMjMzKzQHmZmZFVrDg0zSFEnfl7RE0kuS/ihp35IyMyU9LmmVpBsk7Vyyf7SkcyQt\nlbRC0hxJW5aUmSTpEknLssfFkiY24jOamVnjNDTIsiCZDwTwfmBH4PPAklyZE4DjgGOAPbJ98ySN\nzx3qLOAjwMeBfYAJwM8lKVfmUuCtwAHAgcBuwMV1+WBmZtY0auTMHpK+DuwbEfsOUeYJ4OyIOC17\nPoYUZl+KiAskTQCWAodHxGVZma2ARcD7ImKepJ2Ae4BpEbEgK7M3cDPwxoh4oH6f0szMGqnRXYsf\nBm6TdJmkpyX9TtIxfTslbQdMAeb1bYuI1cBNwLRs0x7Aq0rKLAbuy5WZCizvC7GszHxgZa6MmZm1\ngUYH2fbA54AHSV1+ZwKnSfpctn8Kqdvx6ZLXPZ3tA5gMrIuIZ4coM4XUaiu1JFfGzMzawKsa/H6j\ngNsj4qTs+V2S3kC6HnZug+tiZmZtoNFB9iSpCzDvPuAL2c9PASK1uhbnykzO9vWV6ZK0WUmrbDKp\nC7KvzOYDvP8WueP0I8nLAJiZDUNEqHyp+ml01+J84I0l295IGqhBRDxMCpr9+3Zmgz32zV4LcAew\ntqTMVsBOuTK3AhtLmporMw0YB9wyWOUiwo8IZsyY0fQ6tMrD34W/C38XQz9aQaNbZN8G5kv6F+B/\nSUPiPw98JVfmTOBESQuBB4CTgeWk4fRExIuSZgOnS1oKPAd8C/g9cF1W5n5Jc4HzJR1FauWdB1wV\nHrFoZtZWGhpkEfFbSX8DfIMUUI8CJ0XEebkyp2etsO8AmwC3AQdExMrcoaYDvcBlwFjgWuDQ6P/n\nwSHAOcDV2fM5pNA0M7M20ugWGRHxK+BXZcqcApwyxP5eUphNH6LMC8Bhw6xm1RYvhhdegF12adQ7\n1k93d3ezq9Ay/F1s4O9iA38XraWhN0S3Mkkxku/ie9+DBQvSv2ZmnUIS0WGDPdrWqFGwfn2za2Fm\n1nkcZDXiIDMzaw4HWY2MGgXr1jW7FmZmncdBViNdXW6RmZk1g4OsRty1aGbWHA6yGnGQmZk1h4Os\nRhxkZmbN4SCrEQeZmVlzOMhqxEFmZtYcDrIacZCZmTVHw+dabFcOMrNiWrMGenvrc+xRo2DcuPoc\n2zZwkNWIb4g2K541a2Czzep3/H32gauvLl/ORsZBViNukZkVz+LFsPnm8PDDza6JjYSvkdWIZ/Yw\nK57HHoOtt252LWykHGQ14haZWfE8+ii8/vXNroWNlIOsRhxkZsXjFll7cJDViIPMrHgcZO3BQVYj\nDjKz4nHXYntwkNWIg8yseNwiaw8OshpxkJkVj4OsPTjIasRBZlYsL76YZvTYZJNm18RGykFWI57Z\nw6xYHnssXR+Tml0TGynP7FEjviHarLGeeQZuuGH4r7/7bncrtgsHWY24a9GssWbPhosugl12Gf4x\nPvWp2tXHmsdBViMOMrPGWrQIjjkmPayz+RpZjTjIzBpr0SLfA2aJg6xGHGRmjbVoEWyzTbNrYa3A\nQVYjDjKzxolwi8w2cJDViIPMrHGWLUv/z02a1OyaWCuoarCHpO2AbYGxwFLg7ohYXYd6FY6DzKxx\n3BqzvLJBJmlb4LPAIcCWQP72wZcl3Qx8F7giIjr2V7lviDZrnEcf9fUx22DIrkVJZwN3AdsDJwE7\nAxOB0cAU4APAr4F/A/4g6e11rW0L8w3RZo3jFpnllWuRrQb+MiKeGWDfEuD67DFL0geAbYDf1LaK\nxeCuRbPGcYvM8oYMsog4HkDSKGBHYFFErByk7C/LvZmkGcCMks1PRcTrsv0XAoeX7F8QEdNyxxgN\nfAv4BOla3XXA5yLi8VyZScA5wAezTT8DPh8RL5Sr43A5yMwq8+STabLekVi4EHbbrTb1seKrdLBH\nAL8ndS3+3wjf837gXWy41lZ6ZWke8Knc/pdL9p9FCqiPA88B3wZ+Lmm3iIiszKXAVsAB2XFmAxcD\nHx5h3QflIDMr7/774c1vhilTRnacri74xjdqUycrvoqCLCJC0kJgc0YeZGsjYukQ+9cMtl/SBOAI\n4PCIuD7bdiiwCHgvME/STsCBwLSIuD0rcxRws6QdIuKBEdZ/QA4ys/KWLIE994Sbb252TaydVHMf\n2fHANyW9VRrRwgfbS3pc0kOSLs2G9OftI+lpSQslfVfS5rl9u5PCd17fhohYDNwH9HU/TgWWR8SC\nXJn5wMpcmZpzkJmVt2wZTJzY7FpYu6nmPrIfAWOAO4C1ktbkd0bEhAqOsQD4B1L34hbAV4FbJO0c\nEc8DvwKuAB4m3a92KnB91m3YSxopuS4ini057tPZPrJ/B2rRLcmVqTkHmVl5L7zgm5it9qoJsn8a\n6ZtFxNz8c0kLSKF1OHBmRPwot/seSXeSug0PAq4c6fuXM3PmzD//3N3dTXd3d8WvdZCZlecWWfH1\n9PTQ09PT7Gr0U3GQRcRFtX7ziFgl6R5gh0H2PylpcW7/U0CXpM1KWmWTgZtyZfLdkX22yPYNKh9k\n1fIN0WbluUVWfKV/5M+aNat5lclUNdeipMmSvizpvyS9Ntu29wDXuSo93hjSsP4nB9m/OWk2kb79\ndwBrgf1zZbYCdgLmZ5tuBTaWNDVXZhowDrhlOPWshG+INivPLTKrh4qDTNLuwELgk8BngL5rYvuT\nrmVVcowzJL1T0raS9gQuJwXMRZLGZ/unStpGUjcwh9SK+ilARLxIGkp/uqT3SHobaVj970n3kxER\n9wNzgfOzY+0FnAdcVa8Ri+CuRbNKuEVm9VBNi+ybwFkR8TYgP9BjLrB3hcfYCvghabDH5cBLwNSI\neIx0P9mupGthC4ELyUYjltyEPZ0UbJcBNwMvAh/K3UMGaV7Iu4CrSQNIfgccVvEnHQYHmVl5L7zg\nFpnVXjWDPXYntcRKPUm6RlVWRBwyxL7VwPsqOEYvKcymD1HmBeocXKUcZGblLVvmFpnVXjUtspeA\nTQbYviNpaHtHc5CZlecWmdVDNUE2B5gh6dXZ88iWePl30r1fHc1BZlaeW2RWD9UE2ZeBTUk3G48j\nLd/yf8Ay4OTaV61YHGRm5blFZvWg/mMkKniBtB+wGykE74yIa+tRsUaTFNV+F688RgqzEU3gZdbG\nxoyB55+HsWObXROrFUlERFN/61UcZJIOA/43ItaUbB8NfCIiLq5D/RqmFkHW1QVr1sCrqhlCY9Yh\nVq+GCRPS/yP+Y699tEKQVdO1eCFpdehSr8n2dTzfFG02uL5uRYeY1Vo1QSbSumSlXg/UbcHKIvF1\nMrPB+WZoq5eynWCS7iYFWAA3Slqb290FbAOUXR26EzjIzAbngR5WL5Vczbk8+/dNwC+AFbl9LwOP\n4OH3gIPMbCgeem/1UjbIImIWgKRHgMtKB3vYBg4ys8G5RWb1Us01shnAxqUbJU2S9FDtqlRcDjKz\nwblFZvVSTZBtS7omVurVpKVWOp6DzGxwbpFZvVQy2OPg3NODJOVHKHYB7yFdJ+t4DjKz/h56CN7x\nDujtTfeRfe1rza6RtaNqBnsEaS2wvF5SiH2phnUqLK8Sbdbf44/DX/0VzJ2bnk+YMHR5s+GoZLDH\nKABJDwNvj4hn6l6rgnKLzKy/FSvSdTF3KVo9VTyZUkRsV8+KtAPP7GHW34oVsPErhoiZ1VY1gz2Q\n9DlJ90haJWn7bNtXJH2sPtUrFrfIzPpbsQJe85pm18LaXcVBJulY0nIt3yVNV9XnceCfalyvQnKQ\nmfW3fLlbZFZ/1bTIjgaOjIizgPw0VXcCu9S0VgXlIDPrz12L1gjVBNk2wB8H2N4LeHUhHGRmpdy1\naI1QTZA9RFpQs9QHgHtrU51ic5CZ9eeuRWuEapaA/CbwHUnjSNfI9pJ0KHA8cEQ9Klc0DjKz/ty1\naI1QzfD7CyW9Cvg6MA64BHgC+EJE/G+d6lcoviHarD8HmTVCNS0yIuIC4AJJrwVGRcSS+lSrmNwi\nM+vP18isEaoKMgBJfwnslP18b0R45vuMb4g268/XyKwRKg4ySZuR5lr8ELB+w2b9HDgiIp6tQ/0K\nxS0ys/7ctWiNUM2oxe8BfwXsC4zJHu8EtgMuqH3VisdBZtafuxatEarpWjwQeE9E3JrbNl/SUcC1\nta1WMTnIzPpz16I1QjUtsqXAygG2rwI6vlsRHGRmpdy1aI1QTZCdApwp6c+rQWc/fyvb1/EcZGYb\nrFsHL70E48Y1uybW7obsWpR0N2lBzT7bAY9Iejx7viWwGtiCdA2toznIzDZYuRLGj0//X5jVU7lr\nZJeX2W85DjKzDdytaI0yZJBFxKxGVaQdeGYPsw0cZNYoDW30S5ohaX3J44mSMjMlPZ4t3nmDpJ1L\n9o+WdI6kpZJWSJqTv26XlZkk6RJJy7LHxZLqvti6b4g228BD761RmtF7fT8wGZiSPXbt2yHpBOA4\n4BhgD2AJME/S+NzrzwI+Anwc2AeYAPxcUn6xz0uBtwIHkG4b2A24uE6f58/ctWi2gYfeW6NUPUVV\nDayNiKWD7JsOfCMirgSQdDgpzP6eNMfjBNJM+4dHxPVZmUOBRcB7SaG3Eym8pkXE7VmZo4CbJe0Q\nEQ/U64M5yMw2cNeiNUozWmTbZ12HD0m6VNJ2ANm/U4B5fQUjYjVwEzAt27QHKXzzZRYD9+XKTAWW\nR8SCXJn5pHvg+srUhYPMbAN3LVqjjCjIJG1U5UsWAP9AajH9Iym45kvaJPs5gKdLXvN0tg9Sl+S6\nAeZ1zJeZQrp5u9SSXJm6cJCZbeCuRWuUioNM0hck/W3u+WzgJUkLJb2xkmNExNyIuDwi/ph1DR4E\ndAGHV1vxVuQgM9vAXYvWKNVcI/sC2UrQkt4JfIx07epvSbN7/HW1bx4RqyTdA+wAzCGtPD0ZWJwr\nNhl4Kvv5KaBL0mYlrbLJpC7IvjKbD/B2W+SOM6CZM2f++efu7m66u7sr/SiAg8xa29q1sGpV497v\n2WcdZO2op6eHnp6eZlejn2qCbEvg4eznDwI/jogfZbN/3DycN5c0BtgRuC4iHpb0FLA/cEdu/77A\nl7KX3AGszcpclpXZirQ+2vyszK3AxpKm9l0nkzSNtKr1LUPVJx9kw+Egs1Z29NHwP/8DG1V7QWAE\nzjyzce9ljVH6R/6sWc2/3biaIHuR1Kp5jBQkZ2Tbe0lLupQl6QzgKuBRUivqq6SA6RsafyZwoqSF\nwAPAycBy0nB6IuLFrEvzdElLgedIrcHfA9dlZe6XNBc4PxutKOA84Kp6jlgE3xBtrW3ZMrjkEvjo\nR5tdE7PaqibIriENgb+TtC7Zr7Ltu7ChpVbOVsAPgdeSBmQsAKZGxGMAEXF61gr7DrAJcBtwQETk\nZ92fTgrPy4CxpCVkDo2I/JyQhwDnAFdnz+cAn6/8ow6Pb4i2VrZ2LbyqGTfcmNVZNaf1McCpwOuB\nj0bEc9n23chaTOVExCEVlDmFIWbTj4heUphNH6LMC8BhldSplty1aK3MQWbtquLTOiJeZIBWTUTM\nqGmNCsxBZq3MQWbtqtwyLpv2tbwkbTpU2VwLrWM5yKyVOcisXZU7rZdK+ouIWAI8Q/+1yfoo295V\n68oVjYPMWpmDzNpVudN6P9LIQIB317kuhecgs1bmILN2VW49shsH+tkG5iCzVuYgs3blRchryEFm\nrcxBZu3KQVZDviHaWpmDzNqVg6yGfEO0tTIHmbUrB1kNuWvRWpmDzNpVRUEmaSNJT0napd4VKjIH\nmbUyB5m1q4qCLJsWqpeB7yOzjIPMWpmDzNpVNV2L55Bmpvf/CoNwkFkrc5BZu6rmtN4XeBfwuKQ/\nAvkZ6YmID9WyYkXkILNW5iCzdlXNaf0McEW9KtIOHGTWynp7HWTWnqqZ/f7T9axIO3CQWStzi8za\nVdXD7yXtIenjksZnz8f7ulniG6Ktla1dCxtt1OxamNVexQEkaTJppeV3kEYv7gA8BPwHsJohFrrs\nFG6RWStzi8zaVTUtsm8DTwObAaty238MHFDLShWVZ/awVuYgs3ZVzWn9HuA9EfG8pPz2B4HX17RW\nBeUWmbWq9evTY5Tn8rE2VM1pPRZ4eYDtm5O6Fjueg8xa1bp1qTXW/29Qs/ZQTZDdBPxD7nlI6gJO\nAK6rZaWKykFmrcrditbOqjm1jwdulPR24NXAt4BdgInA3nWoW+E4yKxVOcisnVXcIouIe4FdgVuA\na4AxpIEeb4uIB+tTvWJxkFmrcpBZO6vq1I6Ip4AZdapL4TnIrFU5yKydVXMf2TVAD3AD8JuIWFuv\nShWVb4i2VuUgs3ZWzWCP24H3k4LseUnXSPoXSdM8s0fiFpm1KgeZtbNq5lo8GUDSWGAa0E0Ktpmk\n4fcTal+9YvEN0daqHGTWzoZze+QE4LXAFsBkYC1wRy0rVVRukVmrcpBZO6vmGtm5pFbYNsBtwI3A\nkcCCiFhTl9oVjIPMWpWDzNpZNaf20cBS4DTgV8AdERF1qVVBOciS1ath4cLqXzdxImy7bc2rYzjI\nrL1Vc2rvQGqRdZNaYq+R9GvS4I+eiLiz5rUrGAdZMns2zJgBW25Z+WvWr4dFi+DFF+tXr07mILN2\nVs1gjwdJEwTPBpC0I2m2j9OAruzR0RxkycqVcMQRcPrplb9m7VoYM6Z+dep0DjJrZ9VcIxsF7AG8\nm9Qq25s0u8cdpPvLOp6DLHn5ZRg9urrXdHWle/A8Q3t9OMisnVVzai8jzbF4Jym4zgR+HREr61Cv\nQvIN0clwgkxKqxf39sKrX12fenUyB5m1s2r+9v07YNOI2CsiToyIuSMNMUknSlov6ezctguzbfnH\nLSWvGy3pHElLJa2QNEfSliVlJkm6RNKy7HGxpIkjqW85bpEla9ZUH2SQXtPbW/v6mIPM2ls1kwbP\njYiVksZIepOkXSQN+6qGpKmkQSN3DbB7HuketSnZ4wMl+88CPgJ8HNiHdG/bz9V/xc9LgbeSVq8+\nENgNuHi49a2Eb4hOhtMigw0tMqu93l4HmbWvioNM0qsknQE8Twqfu0lTVZ0uaaNq3jRrGf0A+DSp\ny7LUmohYGhFLssey3GsnAEcAX46I6yPi98ChwJuB92ZldiKF15ERcXtE3AYcBXxQ0g7V1LUabpEl\nL788vO7BjTZKr7Xac4vM2lk1XYunA58i3U/2BtJw/M+SQuQbVb7vd4EfRcSNg+zfR9LTkhZK+q6k\nzXP7didd25vXtyEiFgP3kabOApgKLI+IBbky84GVuTI15yBLhtsic9di/axdm/5QMGtH1fyN9vfA\nERHxy9y2ByUtBb4HfLmSg0g6EtgeOGSQIr8CrgAeBrYFTgWul7RbRPSSuhrXRcSzJa97OttH9u/S\nAY69JFem5hxkibsWW49bZNbOqjm1J5LuIyv1IDCpkgNIegMpmPaOiAF/5UfEj3JP75F0J7AIOAi4\nsor6NpyDLBnuYA93LdaPg8zaWTWn9l3AF4BjSrZPB35f4TH2AjYD7s2Ny+gC3inpaGB81ur6s4h4\nUtJiUlcmwFNAl6TNSlplk4GbcmXy3ZF9tsj2DWjmzJl//rm7u5vu7u7KPlXGQZYM9xqZuxbrx0Fm\ntdLT00NPT0+zq9FPNaf28cAvJb0X6Lv2NBV4HWk5l0r8FPhNybbvA38CTi0NMYDs+tiWwJPZpjtI\nM+7vD1yWldkK2AmYn5W5FdhY0tS+62SSpgHjgH5D+fPyQTYcDrLEXYutx0FmtVL6R/6sWbOaV5lM\nNVNU3ZR1DR4D7Jht/jFwbkQ8UeExXgTuzW+TtBJ4LiLukzSetL7ZFaTg2g74OqkV9dO+Y0iaDZye\nXZ97DvgWqVV4XVbmfklzgfMlHQUIOA+4KiIeqPQzV8tBlowkyNy1WB8OMmtnVZ3aWWCdVOM65GfQ\nXwfsShoJOYkUZtcDf1dy8/V0oJfUIhsLXAscWjIb/yHAOcDV2fM5wOdrXPd+PLNHMpJrZG6R1YeD\nzNrZkKe2pN0qPdBwZ7+PiP1yP68G3lfBa3pJYTZ9iDIvAIcNp07D5RuiEw+/bz0OMmtn5U7t35Ja\nTCpTLvDs9+5azPiG6NbjILN2Vu7U3q4htWgTDrLEgz1aj4PM2tmQp3ZELGpURdqBgyxx12LrcZBZ\nOxtyiipJFbfIlGw98ioVl4Ms8WCP1uMgs3ZWbq7FWyXNlrTXYAUkbSLps6Rh9R+uae0KxkGWePh9\n63GQWTsrd2rvSBpu/wtJ60k3Iz8BrAY2AXYm3Yh8O3BsRMytY11bnoMs8cwercdBZu1syBZZRCyL\niH8mzaxxNGmG+UmkQSBrgYuAt0XE3p0eYuAg6+PBHq3HQWbtrKJTOyJeAi7PHjYI3xCduGux9TjI\nrJ1Vsx6ZleEboiEiDfYYztpX7lqsHweZtTMHWQ25azH9wuzqSo9quWuxfhxk1s4cZDXkIBv+QA9w\n12I9OcisnTnIashBNvzrY+CuxXpykFk7c5DVkINsZEHmrsX66e11kFn7Khtkkr4uaVzu+Qckjc09\nnyDp4npVsEgcZMOf1QPctVhPbpFZO6ukRXYCsHHu+WXAX+SejwU+WctKFZWDzC2yVrV27fBGkpoV\nQSVBVrqES7klXTqWg2xkgz18jax+3CKzduZrZDXkG6LdImtVDjJrZw6yGnKLbORB5mtk9eEgs3ZW\n6al9tKQVudd8RtKz2fPX1L5axeSZPUY22MNdi/XjILN2Vsmp/Sjw6dzzp4C/H6BMx3OLzF2LrcpB\nZu2s7KkdEds2oB5twUHmmT1alYPM2pmvkdWQg8wze7QqB5m1s0puiH6LpHeXbPukpIckLZF0nqRh\n/upqLw4ydy22KgeZtbNKWmRfA/bpeyJpZ+BC4AHgUtLN0CfUpXYF4yDzzB6tykFm7aySINsNuCb3\n/BPAvRFxYERMB44FPl6PyhWNg8w3RLcqB5m1s0qCbDPgidzzdwJX5Z73AK+vYZ0KyzdEu2uxVTnI\nrJ1VEmRLgS0BJHUBuwO35faPBjq8HZK4ReYboluVg8zaWSVB1gPMkLQ98KVs2w25/TsDj9S2WsXk\nG6JHfo3MLbL6cJBZO6vk1P4qcC3wf8A64AsRsTK3/1DgujrUrXC6umDlSvj70tvFa+DEE2HXXWt/\n3Frz8PvW5CCzdlbJDdGPSNoR2AVYGhFPlBSZASyuR+WK5jWvgSuugBUrypetxuzZcOutxQmykdwQ\n7SCrDweZtbOKTu2IWAvcNci+Abd3qg99qPbHXLAAVq+u/XHr4eWXYcKE4b3W18jqx0Fm7azsqS3p\ni5UcKCL+Y+TVsYGMHQsvvdTsWlTGXYutyUFm7aySU/ubwDPACgZfVDMAB1mdjBlTnCDzYI/W5CCz\ndlbJqf0b0vWxXwCzI+LX9a2SlRo7FpYta3YtKuPh963JQWbtrOzw+4jYE9gTeB74iaSFko6XNHmk\nby7pREnrJZ1dsn2mpMclrZJ0QzYtVn7/aEnnSFoqaYWkOZK2LCkzSdIlkpZlj4slTRxpnZuhaF2L\nntmj9fT2OsisfVU62OMe4IuSTgA+DBwBzJJ0DfCxiFhT7RtLmgocSckgkuw9jgMOB/5EGhU5T9Ib\ncsP+zwI+SJoa6zng28DPJe0WEZGVuRTYCjiA1CU6G7g4q3+hFKlrcSQtsle9KrUcIkCDdWJ3mBUr\n0mCfkXKQWTur6tSOiF7gckkvAuOAg4CxQFVBlrWMfkBasHNmye7pwDci4sqs7OHAEtJinhdImkAK\n0sMj4vqszKHAIuC9pNDbCTgQmBYRt2dljgJulrRDRDxQTX2bbezYYo1aHG6QSemXbW/v8I/Rbn74\nQzjlFNhxx5Ed5+CDHWTWvio+tSVtSxYg2aaLgU9HxHCu3nwX+FFE3Kjcn96StgOmAPP6tkXEakk3\nAdOAC4A9snrnyyyWdF9WZh4wFVgeEQtyZeZLWpmVKVyQFaVFNpLBHrChe9FBlqxaBR/9KJx5ZrNr\nYta6Khl+/0lSgO1Fmiz4KGBurguvKpKOBLYHDhlg9xTSCMinS7Y/Dbwu+3kysC4inh2gzJTccZYO\ncPwluTKFMWZMZ7TIwCMXS61enf77m9ngKmmRXQI8CpxJGoa/M7CzSi5iVHIfmaQ3AKcCe0dEy81K\nOHPmzD//3N3dTXd3d9PqklekFtlIBnuARy6WWr16ZN+nWa319PTQ09PT7Gr0U0mQPUpqJQ3UgupT\n6X1ke5GWhbk3F4RdwDslHQ28iTQwYzL9p72aDDyV/fwU0CVps5JW2WTgplyZzQd4/y1yx3mFfJC1\nknoP9jj3XJgxozbHWrYMJo5gbKhbZP2tWTP8mVLM6qH0j/xZs2Y1rzKZSuZa3LaG7/dT0n1ped8n\njU48NSL+JOkpYH/gDgBJY4B92TDz/h3A2qzMZVmZrYCdgPlZmVuBjSVN7btOJmkaaYDKLTX8PA1R\n78EeCxfCscfC//t/Iz9WVxdsuunwX+8h+P2tXg1bbNHsWpi1tpqMY5K0dUQ8Vq5cRLwI3Fvy2pXA\ncxFxX7bpTOBESQtJgzJOBpaThtMTES9Kmg2cLmkpafj9t4Dfk83CHxH3S5oLnJ+NVhRwHnBV0UYs\nQv27Fpcvhze/GTYfqA3bYO5a7M9di2bljSjIJE0hLfNyBGkY/nD0GzQSEadnrbDvAJuQFvE8oGTp\nmOlAL6lFNpa0zMyhJQNQDgHOAa7Ons8BPj/MOjZVvbsWV6yAjTeu3/Gr4a7F/tas8WAPs3IqGbU4\nCfhP0o3FvcBppID4V+AEUgvriOFWICL2G2DbKcApQ7ymlxRm04co8wJw2HDr1Urq3bXYSkHmrsX+\nPGrRrLw2FUGRAAAQDklEQVRKWmRfJ12jugh4H2kWjf2B8cD7I+LG+lXPoP5diytWpLXUWoFbZP25\na9GsvEqC7CDgiIi4VtK5pJWiH4yIY+tbNetT767F5ctbp0Xma2T9uWvRrLyykwaTbkS+FyAiHgJW\nk2bYsAZ59atTK2V9ne68c9di63LXoll5lQTZKNK1sT7rgFX1qY4NREphVq/rZK0UZO5a7G/NGnct\nmpVTSdeigB9I6psYeAxp8t5+YRYRH6p15WyDvutk48bV/titdI1s9Gj4t3+D2bP7b//rv4ZPfao5\ndWomt8jMyqskyC4qef6DelTEhlavkYvr18PKlfUJyOE47TS4557+2377W7jiis4NMrfIzIZWycwe\nn25ERWxo9RrwsWpVCsmurtofezh23TU98jbdFL75zebUp9k82MOsvEqukVkLqNcQ/FbqVhzM+PGp\n1diJ3LVoVp6DrCDq1bXYSkPvB9PpQeauRbOhOcgKol5di600YnEwnRxk7lo0K89BVhD17Fps9SAb\nN65zg8xdi2blOcgKol6rRPsaWetavx7Wrk331pnZ4BxkBVGvFlmRrpH1W9ugA/TdDF2yGLuZlXCQ\nFUQndy1utBGMGtV5czC6W9GsMg6ygujkrkXozO5FT09lVhkHWUF0ctcidGaQuUVmVhkHWUF0ctci\ndG6QuUVmVp6DrCDq2bXoIGtNvofMrDIOsoLo5CmqoDODzF2LZpVxkBWEr5F1ZpC5a9GsPAdZQbhr\nsfOCzF2LZpVxkBWEuxY7L8jctWhWGQdZQXTypMHQuUHmrkWz8ipZIdpawJQpMGdO7acr2mgj2GKL\n2h6zHjoxyNy1aFYZB1lB7L13mkS2U3VikLlr0awy7lq0QujEIPMUVWaVcZBZIXRikLlFZlYZB5kV\nQqcGmVtkZuU5yKwQOjHIPNjDrDIOMiuEceM6L8jctWhWGQeZFUIntsjctWhWGQeZFUInBpm7Fs0q\n4yCzQujEIHPXolllGhpkkj4n6S5JL2SPWyR9ILf/QknrSx63lBxjtKRzJC2VtELSHElblpSZJOkS\nScuyx8WSJjbqc1rtdWqQuWvRrLxGt8geA44H3gbsDlwPXCnpTbky84DJwJTs8YGSY5wFfAT4OLAP\nMAH4udRv8qZLgbcCBwAHArsBF9f6w1jjdGKQuWvRrDINnaIqIq4q2XSypM8CewF/zLatiYilA71e\n0gTgCODwiLg+23YosAh4LzBP0k6k8JoWEbdnZY4Cbpa0Q0Q8UOvP1W56enro7u5udjX6GT8+rZ32\nve819n0XLuzhjW/sbuybZh58sLWCrBXPi2bxd9FamjbXoqRRwMeA8cD83K59JD0NLANuBE7KBdvu\npDrP6yscEYsl3QdMy7ZPBZZHxIJcmfmSVmZlHGRltOL/pGPGwHHHwYIF5cvW0p139vD8892NfdPM\ntGmw665NeesBteJ50Sz+LlpLw4Ms60a8FRgDLAc+EhH3Zrt/BVwBPAxsC5wKXC9pt4joJXU1rouI\nZ0sO+3S2j+zfgVp0S3JlrGAkOP30xr/vzJnpYWatqxktsvuBtwATgY8CF0t6V0TcGxE/ypW7R9Kd\npG7Dg4ArG19VMzNrdYqI5lZAmgc8EhFHDrL/IeC/IuIMSe8GrgW2yLfKJP0R+HFEzJL0aeDMiJhY\ncpzlwD9FxEWDvE9zvwgzs4KKiBqvlFidVliPbBQw4CBjSZsDWwJPZpvuANYC+wOXZWW2AnZiw3W2\nW4GNJU3tu04maRowDug3lD+v2f8hzMxseBraIpP0DeAXpGH4rwE+CfwzaYj9fGAm6RrZk8B2wNdJ\nQbZzRKzMjnEu8NfAp4HngG+Ruin3iOzDSPpl9rqjAAHnAw9FxN804nOamVnjNLpFNgW4JPv3BeAP\nwPsi4lpJY4BdgUOBSaQwux74u74Qy0wHekktsrGkrsZDo38iHwKcA1ydPZ8DfL5eH8rMzJqn6dfI\nzMzMRqLj51rMps16SNJLkn4raZ9m16kakmYMMK3XEyVlZkp6XNIqSTdI2rlkf02m/ZK0taSrsmMs\nlXSWpLq1+iXtm9V1cfa5DxugTMt8dklvktST1eUxSV9t1HfRyOnfmvldSDpR0u1KU+AtkfQzSbsM\nUK7tz4tKvou2OS8iomMfpGmuXibNFvJG4GzSvW1bNbtuVXyGGcC9wObAFtljs9z+E0jduH8D7Az8\nL/A4MD5X5r+AxcB+pKm9bgB+R9Ziz8r8CrgbeAewJ2kmljm5/aOy/deTbq94T/Y+Z9Xxs78f+Bpw\nMLACOKxkf8t8dtI14SdJ06ftlNX5ReC4Bn0XFwJzS86TSSVlCv9dZPU7LPvvvQvwk+y9JuXKdMR5\nUeF30RbnRcN+4bbiA1gAnFey7U/Aqc2uWxWfYQbwhyH2PwF8Jfd8THZyHJk9nwCsAT6RK7MVsA7Y\nP3u+E7AemJors3e2bYfs+ftJI0pflyvzSWAVsHEDvoflvPKXd8t8duCzpNlqRufKnAQ81qDv4kLg\nZ0O8pl2/i/FZfQ7yeTHgd9EW50XHdi1K2og05dW8kl3XkKayKpLts26ShyRdKmk7gOzfKfSf0ms1\ncBMbPuMeDDDtF3BfrsyA034BK0vK3BcR+W7NuaRfErvX6oNWqgU/+1Tg5oh4uaTM6yRtM/xPWpV9\nJD0taaGk7yrd3tJnwOnfKP53MYHUGngeOv686Pdd5BT+vOjYIANeC3SRprfKy093VQQLgH8gTZT8\nj6S6z5e0SfZzMPRnnExtpv2aUvo+EfEM6S+3ZnyfrfbZX1Emey4a8/30dTPtB3yR1AV0ffYHXV/9\n2vG7OAu4k3R/ad97d+p5UfpdQJucF61wQ7SNQETMzT+XtIA0V+XhwG1NqZS1nOjA6d8k/QepRbB3\nZP1UnWqw76JdzotObpH1/TUwuWT7ZOCpxlenNiJiFXAPsAPpc4ihP+NTQJekzcqU2ZxX2qKkTL/3\nkdTX6m3G99kqn/3Jwcpkz4MmfD8R8STpAv4O2aa2+i4kfZs0mOvdEbEot6vjzoshvotXKOp50bFB\nFmk2/TtI013l7U//ZWUKRenG8h2BJyLiYdIJsH/J/n3Z8Bnz0371lRl02q9cmdJpv24FdpL0ulx1\nDgBWZ+/RUC302e/MldlX0uiSMk+U++VSDxp6+re+MoX8LiSdxYZf3P2Wbeq082Ko72KQ8sU8L2o9\nMqZID9J6aKuBz5B++Z9FGr20dbPrVsVnOAN4J2nZmz2Bn5NG/myd7T+edHH3I8CbSDOiLKb/UONz\ngUdJQ2LfRhoiewf9h9f+EriLdEF2L9KsLFfm9o/K9l9LGqL73ux9zqzjZx9PGsr7VtKF5ZOz5y33\n2UkX2p8AfkgaCn0waQj4sfX+LrJ9Z2T13wboJv2CWdRu3wXwn9mxukl/zfc98p+zI86Lct9FO50X\nTf9F3OwHcDTwEPAS8BtSH3LT61VF/S/NTojVpDksfwzsWFLmX0n3bKwi3QOyc8n+jUghvpR0D9KV\nwJYlZSYCF5NCchlwETChpMxWwM+yYywFvg1sVMfP/i7SEN91JY//bsXPnv3P2ZPV5XHg5EZ8F6SR\nYVeTWiKrSddQZw/wOQv/XQzyHawD/rVV/59o1nfRTueFp6gyM7NC69hrZGZm1h4cZGZmVmgOMjMz\nKzQHmZmZFZqDzMzMCs1BZmZmheYgMzOzQnOQWcdQWk37DyXP725mndqF0irLZze7HtaZHGRWWJK+\nny3NfsEA+/492/ez3OYzSDNg5HXMjAB1DpuPACdWWZ/1kg6uU32sgzjIrMiCNAfcxySN7dsoqQs4\nlDRn3IbCEasionRRQauBiFgWESubXQ/rTA4yK7q7gQdIE0D3OYg0d2ZPvmAlXYmSPi3pHkkvSbpf\n0rEl+4+TdJekFZIWS7pA0sSSMkdIWpSV+Ymkz0paX1Lmg5J+m73Pg5K+llvMcLC6TZV0XXbcZZKu\nlTQl2/eK1pakC/tapJIuJLVGj8laQuskvV7Su7LnB0n6XVaf30rareRYB0v6g6TVkh6V9C8l+/u9\nv6SHJZ0k6TxJL0h6TNKX8/tJf4hcnr3/Q0N9drOhOMis6II00elnctuOAC4covyAJB0JfI00c/yO\nwJeA4yV9LldsHTAd2Bk4BHg7kP8FvhdwAXAOaZbvXwAz8+8r6UDgB9nrdsrq+7fAqUPU7S2kWcf/\nRFog8R2kCaMrXRx3OmmZjAtJM6D/BWmS6T5nAP9MWnb+IeCqbHkTJO0O/Ai4nDRb/AnAiZKOKfOe\nx5JmQX8b8O/A6ZL2zPa9nbQu2GdIq/++vcLPYfZKtZp92w8/Gv0g/VL+GTCJNFv2X5J+Kb5Emmn7\nQuBnufIzgD8M8XwR8MmS95gO3DNEHQ4EXso9/yHwy5Iy55OWi+97fiNwUkmZDwPLh3ifHwDzh9h/\nA3D2QN9PmTJ9s+Z/IrdtPGmZkyNy731tyetmAI8OdmzSTOr/U/KaPwH/knu+Hji42eeRH8V/uEVm\nhRcRy4Cfkv66PwzoiYjF1RwjW612a+B8Scv7HsBpwHa5cvtJuibrKnsR+Akwuq+Lj9SSu73k8LeV\nPN8dOKnkfX4IjJVUukJun751oOohgAV/fpKudd1NanVC/0UU+/wa2FLSxkMc9w8lz58grRpsVlOV\ndkuYtbr/Jq2BtILUNVitvj/qjiJ1wb2CpNeTFi49H/gq8CwplH4IjB7oNUO81yzS2nGlllZxnLz1\npK66vCGvudXIUKM+ewco6z+ereYcZNYWIuI6SS8DmwJzhvH6JZKeAP4qIv5nkGJ7kMLhixERAJI+\nVFLmfl55vWfPkud3khY/rWaAw++A/UgBOpClpOteeW8hdfH1eRnoGuC1Iq3s+wiApPGka2Hfz/bf\nB+xd8pp9gcUxspGKvYPUx6wqDjJrJ7uSll8vbQlUagZwtqQXSEu3bwTsRloN9zTS6MhRwHGSfkJa\n0n16yTHOBm7ORuhdSboG9TclZU4hDaZ4lDSIYi0pON4REScMUrczgFslnU9awn41KUzmZt2o1wPf\nlvRBYCGpZbk1/YPsEeAdkrYhtVyfy+07WdIzwJOk1ZPXkAaTAHwLuF3SDFLr8x3AF4GvDFLXSj0C\nvEfSTcCarIvYrGpu5lvbiIiVEbFiBK+fTRpB+Cng98BNwJGkUXxExN2k4DoOuCcr+6WSYyzIXvN5\n4C7SII7TScHTV+Ya0i0C3aTrZ7eRRgL2u++t5Lh3Ae8F3kjq+lwAfJwN3Xf/nT1mk65f9V2/y/sm\nqVV2L7CEFHSQuvy+Qgqs35IGzRwUES9l7/074O+Ag0nXzr4OfD0izs1XsbTKA32MkudfAt5Nuhfw\nzsE+u1k5ynpIzKxOJH0b2C8i3tLsupSS9C5Sa27ziHiuXHmzVuSuRbMay7oV55G67/YndfONtBuu\nnkoHiZgVioPMrPb2IHWbTSRdozohIs5pbpWG5G4ZKzR3LZqZWaF5sIeZmRWag8zMzArNQWZmZoXm\nIDMzs0JzkJmZWaE5yMzMrND+Pzm+Zm4W78wTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fca4677eac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot mileage cutpoint (x-axis) versus RMSE (y-axis)\n",
    "plt.plot(mileage_range, RMSE)\n",
    "plt.xlabel('Mileage cutpoint')\n",
    "plt.ylabel('RMSE (lower is better)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Recap:** Before every split, this process is repeated for every feature, and the feature and cutpoint that produces the lowest MSE is chosen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a regression tree in scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# encode car as 0 and truck as 1\n",
    "train['vtype'] = train.vtype.map({'car':0, 'truck':1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define X and y\n",
    "feature_cols = ['year', 'miles', 'doors', 'vtype']\n",
    "X = train[feature_cols]\n",
    "y = train.price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=None,\n",
       "           max_leaf_nodes=None, min_samples_leaf=1, min_samples_split=2,\n",
       "           min_weight_fraction_leaf=0.0, presort=False, random_state=1,\n",
       "           splitter='best')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate a DecisionTreeRegressor (with random_state=1)\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "treereg = DecisionTreeRegressor(random_state=1)\n",
    "treereg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3107.1428571428573"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use leave-one-out cross-validation (LOOCV) to estimate the RMSE for this model\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "scores = cross_val_score(treereg, X, y, cv=14, scoring='mean_squared_error')\n",
    "np.mean(np.sqrt(-scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What happens when we grow a tree too deep?\n",
    "\n",
    "- Left: Regression tree for Salary **grown deeper**\n",
    "- Right: Comparison of the **training, testing, and cross-validation errors** for trees with different numbers of leaves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Salary tree grown deep](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/salary_tree_deep.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **training error** continues to go down as the tree size increases (due to overfitting), but the lowest **cross-validation error** occurs for a tree with 3 leaves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning a regression tree\n",
    "\n",
    "Let's try to reduce the RMSE by tuning the **max_depth** parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4050.1443001442999"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# try different values one-by-one\n",
    "treereg = DecisionTreeRegressor(max_depth=1, random_state=1)\n",
    "scores = cross_val_score(treereg, X, y, cv=14, scoring='mean_squared_error')\n",
    "np.mean(np.sqrt(-scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, we could write a loop to try a range of values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# list of values to try\n",
    "max_depth_range = range(1, 8)\n",
    "\n",
    "# list to store the average RMSE for each value of max_depth\n",
    "RMSE_scores = []\n",
    "\n",
    "# use LOOCV with each value of max_depth\n",
    "for depth in max_depth_range:\n",
    "    treereg = DecisionTreeRegressor(max_depth=depth, random_state=1)\n",
    "    MSE_scores = cross_val_score(treereg, X, y, cv=14, scoring='mean_squared_error')\n",
    "    RMSE_scores.append(np.mean(np.sqrt(-MSE_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7fca41b38b00>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEZCAYAAACjPJNSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVNX9//HXGywoChoLRLFXErEnImhcNTYsCUmMLQYx\nP6JGwbSvmuJXNMWI/YuxoV8jNhJLgjUK6pqIJfmqQSwkKhFFo2CjqUj5/P44d3UYd9mZZWbu7O77\n+XjcBzv3nrnzGUn2wzn3c85RRGBmZlZtXfIOwMzMOgcnHDMzqwknHDMzqwknHDMzqwknHDMzqwkn\nHDMzqwknHDMzq4lcE46kn0haIul/stcrSDpH0mRJ8yS9LukGSRsUvW8lSaMlzcrajZe0flGbNSRd\nJ+m97BgrqWctv5+ZmX0it4QjqT8wDJhccHpVYHvgF8AOwCHABsA9kgpjvRgYDBwG7Ab0AO6UpII2\nN2X32hfYD9gRGFuVL2NmZq1SHisNZD2NJ4DvACOBKRExooW2fYFngX4R8aykHsAsYEhEjMva9AGm\nA/tHxISC9wyIiMeyNgOBvwJbRcQLVf2CZmb2KXn1cK4E/hARD5XQticQwLvZ652AFYAJTQ0iYgbw\nPDAgO9UfmNuUbLI2k4D5BW3MzKyGVqj1B0oaBmwKHFFC2xWB84HbI+L17HRvYHFEvF3U/M3sWlOb\nWc3ccmZBGzMzq6GaJhxJWwK/AgZGxJJW2nYFbiA9nzmoBrF5FVMzszaICLXeqvZDarsCawHPSVoo\naSGwB3CipI+yHk1TshkHbAPsFRHvFtzjDaCrpLWK7t0ru9bUZp1mPn/dgjafEhEd9jjjjDNyj8Hf\nzd/P36/jHeWodcL5I9AP2K7g+D9SRdl2EbFQ0grAH0jJpiEiiofGngAWAfs0nciKBvoCk7JTjwKr\nZZVwTW0GkKrgHqnC9zIzs1bUdEgtIuYAzxWekzQfeCcins96NreQCgMOTpfVK2s6OyI+jIg5kq4G\nRkmaBbxDes7zD+D+7HOmSroXuELScYCAy4E7whVqZma5qHnRQDMK+2R9SIkGUk+m0FA+mUdzMrCQ\nNOy2CjARODqW7t8dAYwG/py9Hg8Mr1zY7UtDQ0PeIVRNR/5u4O/X3nX071eOXObh1CNJ4f8WZmbl\nkUTUadGAmZl1Uk44ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZW\nE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044\nZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044ZmZWE044BZ59Nu8IzMw6LiecApdckncEZmYd\nV64JR9JPJC2R9D9F50dKek3S+5IelPS5ousrSRotaZakeZLGS1q/qM0akq6T9F52jJXUc1nxjBsH\n771Xue9nZmafyC3hSOoPDAMmF50/FfgBcCKwMzATmCCpe0Gzi4HBwGHAbkAP4E5JKmhzE7A9sC+w\nH7AjMHZZMR1wAFxzzXJ8KTMza5EiovYfmnoaTwDfAUYCUyJiRHbtdeB/IuI32etupKTzo4gYI6kH\nMAsYEhHjsjZ9gOnA/hExQVJf4FlgQEQ8lrUZCPwV2CoiXmgmpnjkkeDoo+Ff/4IuHmw0M2uVJCJC\nrbfMr4dzJfCHiHio8KSkTYDewISmcxHxIfAXYEB2amdghaI2M4DnC9r0B+Y2JZuszSRgfkGbT+nf\nH9ZYA+65p+1fzMzMmldWwpG0iaQ9JQ2S9IWs91EWScOATYGfN3O5NxDAm0Xn38yuAfQCFkfE28to\n05vUCyo2s6BNM7HB8OEwevQyv4KZmbVBqwlH0saSzpH0CvAicD9wJ/A48J6kCZIOlVTKvbYEfgUc\nGRFLljP2qjjsMHjqKfjnP/OOxMysY1lhWRez6rEhwH3Az4C/Aa8DHwCfAbYBdgd+AZwhaWhE/H0Z\nt9wVWAt4ruD5flfgS5KOz+4nUi9mRsH7egFvZD+/AXSVtFZRL6cXaeitqc06zXz+ugX3+ZSRI0cC\nsOWW8JOfNHDbbQ3L+CpmZp1PY2MjjY2NbXrvMosGJI0CRkXEW63eSBoErBoRtyyjTQ+gT9Hp3wH/\nAn4VEc+3UDTwJqlo4KpWigb2i4iJkrYmFQ0MLCgaGEAqGti6paKBpv8WM2bAttvC9Omw+uqtfXMz\ns86rnKKBkqrUsuGyrYHpETF/OeMrvveDLF2ldgrwE+BY4AXSs57dSNVl87M2lwIHAUOBd4DzgZ7A\nzk1ZQ9LdwPrAcaRe0xXAtIj4agtxROF/i0MPhT32gJNOquS3NTPrWKpRpRbAP4DPtjmqZd/7kxcR\no4ALgUtIQ3i9gH2LEt3JwB+BcaReyxzgkFg6ex5BmuPzZ+Ae4Cng26UGNXx4WnlgSV0+aTIza39K\nnocjaQrw3Yh4tLoh5aO4hxMB228P554L++6bY2BmZnWsWvNwTgHOk7R90Yz+Dskl0mZmlVVOD2cu\n0I2UpBYBCwqvR0SPikdXQ8U9HID334eNNoLHH4dNN80pMDOzOlbxooHspkOWdT0iri3pRnWquYQD\ncMopsHgxnH9+DkGZmdW5qiScjq6lhPPyy7DzzqlEunv3T7/PzKwzq9paapJ6SfqxpMskrZ2dG5it\ngdYhbbwx7LYbXH993pGYmbVvJSccSTsB/wSOIq3y3PTMZh/ScjUdVlOJtDuDZmZtV04P5zzg4ojY\ngaULBu4FBlY0qjqz117pOc5DD7Xe1szMmldOwtkJaK4w4D+kyZkdlpRWHHCJtJlZ25WTcD4A1mzm\n/NakZf87tG9/Gxob4ZVX8o7EzKx9KifhjCetCL1y9jokbQycA9xa4bjqzmqrwdFHw2WX5R2JmVn7\nVM48nB7A3cC2QHfSMv+9gEnAoEov6llrLZVFF3rhBRg4MJVIr7JKjQIzM6tjVZ2HI2kvYEdS7+jJ\niJhYfoj1p5SEAzBoUFpJeujQGgRlZlbnqrXSwLeB30fEgqLzKwGHR8TYsiOtI6UmnHvugZ/9DJ54\nIhUTmJl1ZtWa+HkNac+ZYqtn1zqF/faDefPgkUfyjsTMrH0pJ+GIor1rMhsCsysTTv3r0gVOPNEl\n0mZm5Wp1SC3bByeAz5NWGlhUcLkrsBFwd0R8s1pB1kKpQ2oAs2fDJpvAM8/AeutVOTAzszpWzpDa\nCiW0uSX7cxvgLmBewbWPgJfpBGXRhXr2hCOOgMsvh7POyjsaM7P2odztCcYVFw10FOX0cACefx72\n3DOVSK+8cuvtzcw6omoVDZwBrNbMh60haVoZ9+kQ+vaFfv3g5pvzjsTMrH0oJ+FsTHpmU2xlYP2K\nRNPOeAtqM7PSlVI08LXsx1tI2xIUVqR1BfYG9oyIraoSYY2UO6QGaQXpzTeH3/8evvjFKgVmZlbH\nKjrxU9KS7McglUYXWkgqGvhRRNxZZpx1pS0JB+C882DyZLjuuioEZWZW56q10sC/gS9ExFvLE1y9\namvCeecd2GwzmDoVenXoTRrMzD6tKkUDEbFJR002y+Mzn0lrq115Zd6RmJnVt3KKBpD0PUnPSnpf\n0qbZudMktetJn8vrpJPSnJyFC/OOxMysfpWccCR9H/g5cCVLP8t5DTipwnG1K9tum4oHbrst70jM\nzOpXOT2c44FhEXExSy9v8yRp2ZtOzSXSZmbLVk7C2Qh4ppnzC4GStiPLhuQmS5qdHY9IGlRwvbuk\n0ZJezYbtpmY9q8J7rJS1mSVpnqTxktYvarOGpOskvZcdYyU1t9J1xXz1q2nVgaeequanmJm1X+Uk\nnGmkjdeKDQKeK/EerwKnADsAOwEPAH+StE12/ULgAOAoYGvgl8BvJB1VcI+LgcHAYcBuQA/gTmmp\n3WluArYH9gX2y+Ku6n49K6wAJ5wAl1xSzU8xM2u/yimLHkpKAKcAVwDHAZtnr4+NiN+3KQDpbeC0\niBiTrUx9S0ScWXC9EXg6IkZk21zPAoZExLjseh9gOrB/REyQ1Bd4FhgQEY9lbQYCfwW2iogXWoij\nTWXRhWbNgi23hBdfhLXWWq5bmZm1C9Uqi74GGAn8GlgVuA4YBoxoS7KR1EXS4UB3YFJ2+mHg4CyJ\nIGkAsB1wT3Z9J9IK1xMK4poBPA8MyE71B+Y2JZuszSRgfkGbqlhnHfjKV+Cqq6r5KWZm7VNZZdER\nMSYiNgLWBXpHRJ+IuLqce0jaRtJcYAFwKTA4IpqG5EYATwOvSPoIeBA4NSKaEk5vYHFEvF102zez\na01tZjXz0TML2lTN8OFw6aWwaFHrbc3MOpNS9sNZiqTNgL7Zz89FRLkrRU8l9Vp6At8AxkraI0s6\nI4BdgYOAV4AvAedLejki7is31nKNHDny458bGhpoaGgo+x477QTrrw933AGDB1cuNjOzetDY2Ehj\nY2Ob3lvOM5y1gKuBQ4Cm9dUE3El6hlPc6yj1vhNI67ENJy0M+vXCddkkjQE2ioh9Je0JTATWLfw8\nSc8AN0fEmdmzposiomfR58wFToqIa1uIY7mf4TS56SYYMwYeeKAitzMzq1vV2g/nKlKRwO5At+z4\nErAJMKbcIItiWBlYMTuWFF1fXBDnE6Q5QPs0Xcye9/Tlk+dAjwKrSepf0GYA6bnTI8sRZ8m+/vW0\nttozzRWRm5l1UuX0cN4H9o6IR4vO7wpMjIjuJdzjbNI21a8Cq5PKn/8LGBQR90l6EFiL1NuZDjSQ\nnvP8OCIuze5xKWnIbSjwDnA+aXhu56YuiqS7SXv0HEfqhV0BTIuIry4jtor1cADOPBP+85+05I2Z\nWUdVrdWipwMHR8TTRee3A+6IiA1LuMc1pCTSmzR89jQwKiImZtfXBc4mzZ/5DCnpjImICwvusSJw\nHnAkacLpRODEiHitoE1PYDRp+A9gPDA8IuYsI7aKJpw33ki7gk6bBmuuWbHbmpnVlWolnO+QeiRH\nN/1yz2b4XwuMi4h2XQxc6YQDcNRRqYjghz+s6G3NzOpGxRJONhGzsMEmpGc3Tb2J9YEPgX9HxLZt\nC7c+VCPhPPZYSjr/+hd0bW5zbjOzdq6chNNaWfQtFYin09pll7Rfzj33wEEH5R2NmVm+Sh5S6+iq\n0cMBGDsWbrgB7r234rc2M8tdVZ7hdHTVSjgLFsCGG8Jf/gJbbVXx25uZ5apa83CsDVZeGYYN8yrS\nZmbu4WSq1cMBmDEj7Qr68svQo0dVPsLMLBfu4dSZPn1g773h2mYX1TEz6xyWq4cjacWIWFjBeHJT\nzR4OpGc4w4bB889DF6d5M+sgqtLDkTRC0tcLXl8NfCDpn5L8OLwVu+8O3brBhAmttzUz64jK+bf2\nCLJ9ZiR9CfgmaXmZf5DWM7NlkNJeOaNH5x2JmVk+ylna5gNgy4h4VdK5wFoRcWy2pfNfI2LtagZa\nbdUeUgN4/33YaKO0AsFmm1X1o8zMaqJaRQNzSDt9Qtoe4P7s54Wk5W6sFauuCkOHph1Bzcw6m3J6\nONcBnweeBA4HNoyIdyR9BfhlRPSrXpjVV4seDqTS6J13hunToXurGzqYmdW3avVwTiRtcrYO8I2I\neCc7vyNwU3khdl4bbwy77QbXX593JGZmteWJn5la9XAA7r8fTj4ZpkxJxQRmZu1VxXo4kj5T+POy\njuUNujPZay+IgMbGvCMxM6ud1vbDWQx8NiJmSlrC0nvjfNwMiIho1zu+1LKHA3DZZWlOzm231ewj\nzcwqrpIbsO0BTIqIRdnPLYqIh8oLs77UOuHMm5dKpJ98Mv1pZtYeeXuCNqh1wgH4wQ/SatK/+U1N\nP9bMrGKccNogj4Tz4oswYEAqkV5llZp+tJlZRXi16HZi883hC1+Am1xUbmadgBNOzprWV3NH08w6\nOiecnO27L8yfD5Mm5R2JmVl1lZRwJK0o6Q1Jn692QJ1Nly5w0kleRdrMOr6SEk62ydpCmp+HY8vp\nmGPSnJzXXss7EjOz6ilnSG008BNJK1QrmM6qRw844gi4/PK8IzEzq55yEs7uwFeA1yTdL+n2wqOU\nG0j6nqTJkmZnxyOSBhW12VLSrZLelTRf0v8V7igqaSVJoyXNkjRP0nhJ6xfdYw1J10l6LzvGSupZ\nxnetuZNOgjFjYMGCvCMxM6uOchLOW8CtwN3AK8DbRUcpXgVOAXYAdgIeAP4kaRsASZsADwMvAQ2k\n7RB+DswruMfFwGDgMGA3oAdwp7TUMpg3AdsD+wL7kVa0HlvGd625vn2hXz/4wx/yjsTMrDpyn/gp\n6W3gtIgYI+lGYHFEHN1C2x6kba6HRMS47FwfYDqwf0RMyHYgfRYYEBGPZW0GAn8FtoqIF1q4d80n\nfha7/Xb45S/hb3/LNQwzs5JVdeKnpJ0lHSape/a6e1ue60jqIulwoDswKeuhHAQ8J+keSTMl/U3S\nNwvethOwAjCh6UREzACeBwZkp/oDc5uSTdZmEjC/oE1dOvBAmDULHn8870jMzCqv5IQjqZekx4C/\nATcCvbJLFwDnl3GfbSTNBRYAlwKDI+I50vbVqwE/Bf4MfJk0NHaDpAOyt/cm9YCKh/DezK41tZnV\nzEfPLGhTl7p2hRNPdIm0mXVM5fRMLiT9Yl+L9Aynyc2kCrZSTQW2A3oC3wDGZitRv5td/1NEXJz9\n/LSknYGTgHvK+Iw2GTly5Mc/NzQ00NDQUO2P/JRjj4XNNoM334RevVpvb2ZWS42NjTS2cTOvkp/h\nSHoT2Dsinsl6KNtFxLTsQf8zEdG9TQFIE4CXge+Rhr1GRsSvC67/HDgsIvpJ2hOYCKxb2MuR9Axw\nc0ScKWkocFFE9Cz6nLnASRFxbQtx5P4Mp8l3vwsbbACnn553JGZmy1atZzirAB81c34d4MMy7tNc\nDCtnk0v/DmxVdH1LUlEAwBPAImCfpotZ0UBfoGlxmEeB1ST1L2gzAFgVeGQ54qyZ4cPTnJyFC/OO\nxMyscspJOH8Bjil4HZK6AqcC95dyA0lnS9pN0kbZs5yzgT2A67Mmo4DDJA2TtJmkYaTy50sAImIO\ncDUwStLeknYglTv/oymGiJgK3AtcIam/pF2By4E7WqpQqzf9+sEWW3g3UDPrWMoZUvsc8BDpl/se\nwJ2keTI9gYER8VIJ97iGNL+mNzAbeBoYFRETC9p8G/gZ0Ad4Afh1RPyh4PqKwHnAkaRe10TgxIh4\nraBNT9JzpUOyU+OB4VnCaim2uhlSA7j1VrjwQnj44bwjMTNrWdU2YJPUGziBVJ7cBXgS+G1E/Kct\ngdaTeks4ixbBppvC+PGwww55R2Nm1jzv+NkG9ZZwAM4+G154Af73f/OOxMyseVVJOJLuAxqBB4G/\nR8SiNkdYh+ox4bz1VnqW88ILsPbaeUdjZvZp1apS+xtwACnhvCvpPkk/lTTAK0hXx9prw1e/Cldd\nlXckZmbLr+whNUmrkJaIaciOXYAPI6JHpYOrpXrs4QA8+WRKOtOmwQpO62ZWZ6q6lhppdea1SUvR\n9CLNi3miDfexEuy4Y5oEentJG0CYmdWvctZSu1TSc8A04DjgdWAYsGZE7Fml+Iw0EdTrq5lZe1dO\n0cAS0qKYl5DWNXuiLseg2qheh9QAPvoINt4Y7r03TQo1M6sX1RpS24K0kvOWwG3AO5LukPRDSTu2\nIU4r0UorwXHHwSWX5B2JmVnbtXkejqStSbt3fgvoGhFdKxlYrdVzDwfgjTfSrqDTpsGaa+YdjZlZ\nUpUeTrZh2hclnSrpHlKZ9LdIBQOj2haqlap3bxg0yJNAzaz9KucZzhxgZdJyNo3Z8XBEzK9WcLVU\n7z0cgMcegyOPTBNBu7br/qSZdRTl9HDKmdlxKB0owbRHu+wCa60Fd98NBx+cdzRmZuVpy8TPbsDm\nQAAvRcTy7IVTN9pDDwdg7Fi4/nq47768IzEzq94znBUknUvaCnoyMIW0xM2obMsAq4HDDoPJk2Hq\n1LwjMTMrTzll0aNIRQLHk0qjtyBtVXA0cHblQ7PmrLwyDBvmEmkza3/KKRp4Azg2Iu4uOn8gcFVE\nfLYK8dVMexlSA5gxA7bdFl5+GXq06xXszKy9q9bEz55Ac7t6vgSsUcZ9bDn16QNf/jJce23ekZiZ\nla6chDMZGNHM+ZNJ205bDQ0fnobVlizJOxIzs9KUUxZ9CnC3pC8Dj2Xn+gPrkfbJsRrabTdYZRWY\nMAH22y/vaMzMWldyDyci/kIqFrgFWC07bga2ioiHqxOetUTyKtJm1r60eS21jqY9FQ00+eAD2HDD\ntALBZpvlHY2ZdUblFA0sM+GUswp0RDxZatt61B4TDsCpp8LChXDBBXlHYmadUSUTzhLSigKt3Sy8\nWnQ+pk9Pu4JOnw6rrZZ3NGbW2VQy4WxU6odGxPRS29aj9ppwAAYPToUDxx+fdyRm1tlULOF0Ju05\n4TzwAIwYAVOmpGICM7NaqdjET0mblPGhkrRBqe2tcvbcEyLgwQfzjsTMrGWtlUU/KulqSbu21EDS\nmpJOAJ4DvrKsm0n6nqTJkmZnxyOSBrXQ9gpJSyT9sOj8SpJGS5olaZ6k8ZLWL2qzhqTrJL2XHWMl\n9Wzlu7ZbEpx0kkukzay+tfYMZw3gZ8B3gCWk3T1fBz4E1gQ+B/Ql7f55VkTcu8wPkw4GPgJeICW7\nY0gTSneMiGcK2n0D+CmwDnBhRFxQcO0y4GDg28A7wIWkpXV2bBoTy3Yk7ZPFLeBq0lYKLSbE9jyk\nBjBvHmy0ETz5ZPrTzKwWKv4MR9IqwIHAbsBGwCrAW8BTwL2FyaINwb4NnBYRY7LXGwEPA18G/gyM\nbko4knoAs4AhETEuO9cHmA7sHxETJPUFngUGRMRjWZuBwF9Jk1RfaCGOdp1wAH7wA1hpJTjnnLwj\nMbPOouI7fkbEB6QVBm5ZnsAKSeoCfBPoDjySnesK3Aj8IiL+qU8/Ad8pi3lCQWwzJD0PDMjO9wfm\nNiWbrM0kSfOzNs0mnI7gxBNh111h5Mi07I2ZWT0pZ/HOipC0jaS5wALgUmBwRDybXT4LmBkRV7bw\n9t7A4oh4u+j8m9m1pjazmnnvzII2HdLmm8MXvwg33ph3JGZmn1bO4p2VMhXYjrTdwTeAsZL2ANYF\nhmTXcjFy5MiPf25oaKChoSGvUNps+HA47TQ49liXSJtZ5TU2NtLY2Nim9+Y+D0fSBOBlYAZwOmll\ngyZdScUKr0fEhpL2BCYC6xb2ciQ9A9wcEWdKGgpcFBFLVaVlvaqTIqLZXWQ6wjMcSNsVbL01XH01\n7L573tGYWUdXrQ3YqqULsDLwW2BbUg+n6XgduADYO2v7BLAI2KfpzVnRQF9gUnbqUWA1Sf0L2gwA\nViV7VtSRdeniEmkzq0817eFIOhu4C3gVWB04CvgvYFBE3NdM+39TUKWWnbsUOAgYSiqLPp80PLdz\nQVn03cD6wHGksugrgGkR8dVlxNYhejgAc+bAxhvD00+n3UHNzKqloj0cSb+WtGrB60FZmXTT6x6S\nxpYYW2/gOtJznImkqrP9m0s2meYywMnAH4FxpFLnOcAhRdniCNIOpX8G7iGVb3+7xBjbvR494Mgj\n4fLL847EzOwTrfZwJC0GPhsRM7PXc4DtI2Ja9roX6RmLV4uuI1Onwh57pFWku3XLOxoz66gq/Qyn\n+EaufWoHtt4attsObr4570jMzJJ6KBqwKvEW1GZWT5xwOrBBg+Ctt+Dxx/OOxMys9Imfx0uaV/Ce\n72RroEGqNrM61LVrWu5m9GjYZZe8ozGzzq6UooGXab5abCkRUfLeOfWooxUNNHn3XdhyS7j1VvjS\nl/KOxsw6Gu/42QYdNeEATJwIRx0FjY3Qt2/e0ZhZR9LeVhqwKvvyl2HUqPRM54038o7GzDqrUiZ+\nbpetYVZ47ihJ0yTNlHS5pJWqF6JVwpAhMHQoHHQQzJ+fdzRm1hmV0sP5JWnjNQAkfQ64hrSvzE2k\n5WlOrUp0VlGnnw7bbguHHw6LFuUdjZl1NqUknB2BwqVnDgeei4j9IuJk4PvAYdUIzipLgiuugAUL\nYMQI6KCPrMysTpWScNYirdrc5EvAHQWvG4ENKxiTVdGKK8Itt8CkSXDuuXlHY2adSSkJZxZp5eWm\nLaB3AgqnEq5E2rPG2okePeDuu+GSS2DcuLyjMbPOopSE0wicIWlT4EfZuQcLrn+OtIGatSPrrw93\n3ZWG1v7yl7yjMbPOoJSEczqwBfAi8CvglIgorHM6Gri/CrFZlfXrBzfeCIceCs8/n3c0ZtbRlTTx\nU9IKwOeBWRHxetG17YAZhVs+t0cdeeJna669FkaOhEcfhd69847GzNoTrzTQBp054QCcdRbcfjs8\n9BB07553NGbWXlQ04Uj6YSk3KtwGuj3q7AknAr7zHZg1C/74R1ih1GVdzaxTq3TCWQK8Bcyj5c3X\nIiI2LSvKOtPZEw7AwoVw4IGw+ebw29+meTtmZstS6YTzOOn5zc3A1RHx8PKHWH+ccJI5c2D33dNi\nn6ecknc0ZlbvKrp4Z0TsAuwCvAvcJumfkk6R1Gs547Q65Dk6ZlYtZRUNSFoR+ApwLLAnacmbb0bE\nguqEVzvu4SxtyhTYe++0KoH30TGzllS9Sk3SvsBPSYt6rh0R75V9kzrjhPNp3kfHzFpTlf1wJG0s\n6SxJ04ExwF+BLTpCsrHmeR8dM6ukVotfJR1FGkLblbRo53HAve4OdA5DhsD06WkfHc/RMbPlUWpZ\n9CvAjaTy6GZ5Hk7H5Tk6ZtaSSpdFvwy09pvY83A6OM/RMbPmVLoseuOI2KSVo6RkI+l7kiZLmp0d\nj0galF1bQdI52fV5kl6XdIOkDYrusZKk0ZJmZe3GS1q/qM0akq6T9F52jJXUs5QYrXneR8fMllfJ\nRQPLUpwUluFV4BRgB9K+Og8Af5K0DbAqsD3wi+z6IcAGwD2SCuO8GBhM2mV0N6AHcKe01L+5b8ru\ntS+wH2nX0rFt+nL2Mc/RMbPlsVyLd0rqTdq+4NiIWKWN93gbOC0ixjRzrS/wLNAvIp6V1IO0IdyQ\niBiXtekDTAf2j4gJBe8ZEBGPZW0GkqrqtoqIF1qIw0NqJfIcHTNrUtEhtWx46oZsCOt1SSOUnAFM\nI61CcGzsGuh2AAAQDElEQVQbguwi6XCgO/BIC816kp4fvZu93olUWTehqUFEzACeBwZkp/oDc5uS\nTdZmEjC/oI0tB++jY2ZtUUq90a+B3YFrgf2BC4F9SInigIh4qJwPzIbPHgW6AXOBwRHxbDPtVgTO\nB24v2IOnN7C4mb133syuNbWZ1cxHzyxoY8upcI6O99Exs1KUknAOJA2ZTZR0KWnnz5ci4vtt/Myp\nwHak3ss3gLGS9oiI55oaSOoK3EB6PnNQGz+nbCNHjvz454aGBhoaGmr10e2S5+iYdT6NjY00Nja2\n6b2llEUvBDZq6mVIeh/4QnO9kjYFIE0AXo6IYdnrrsA40grVe0TErIK2ewITgXULezmSngFujogz\nJQ0FLoqInkWfMxc4KSKubSEOP8NpA8/RMevcKr20TRdgYcHrxcD7bQlsGfdfGT7eyvoPwDZAQ2Gy\nyTwBLCIN6ZG9pw/QF5iUnXoUWE1S/4I2A0hVcC09K7I2kuCKK2DBAhgxIiUgM7PmlLrSwASgaUXo\nA4CHKEo6EXFIqx8mnQ3cRSqPXh04CvgvYBBwP3ArqTDgYOA/BW+dHREfZve4lDTMNhR4h/Scpyew\nc1MXRdLdwPqkZXgEXAFMi4ivLiM293CWg/fRMeucyunhlDIAUjwEdX35IX2sN3Bd9uds4GlSOfNE\nSRuREg2knkyhoXwyj+ZkUo9rHLAKaYjt6KJscQQwGvhz9no8MHw54rZWNM3R2XVX2HBDOPzwvCMy\ns3qzXPNwOhL3cCrDc3TMOpeqbE9gVgrP0TGzljjhWMV5Hx0za46LWK0qPEfHzIr5GU7Gz3Aqz3N0\nzDo+P8OxuuA5OmZWyAnHqsr76JhZEw9yWNX16AF33QUDBniOjlln5oRjNdGnD9x5Z6pgW289z9Ep\n1eLFMHt2Stp+BmbtnYsGMi4aqI0JE+Bb34LGRujbN+9o6tcbb8BVV8GVV6aEM38+dOsGa6wBPXsu\nfRSfa+n16qtDFw+iW4WVUzTghJNxwqmd3/0OzjzT++gUi0iJ+LLLUmI+9FA4/njYccd0bd68lHze\ney/92XSU+nr+fFhtteVLWt27p2IQsyZOOG3ghFNbZ54Jd9zhOToA774L114Ll1+ehs1OOCH1Anv2\nbP295Vi8GObObT1BLStpLViQhvdaS1DLSlrdujlpdSROOG3ghFNbEXDssfDWW51zjk4E/P3vqTfz\nxz/CgQemRDNwYH3/Ml64MK0M3tZe1uzZsGTJ0glo5ZXz/la2PB55xAmnbE44tbdwYfpFu/nm8Nvf\n1vcv2kqZPz+tNXfZZemX73HHwdChsM46eUdWOx9+uHQy+uijvCOy5bHbbk44ZXPCyUdn2UfnmWfS\nkNmNN6YKvRNOgH328UN8a/8qvR+OWdV05Dk6CxbArbem3sxLL8GwYTB5MmywQd6RmeXDCcdy19Hm\n6Eyblpb0ueYa2HZb+P734ZBD0qoLZp2ZO/RWF7bdFm64of3uo7NoEYwfDwccALvskl4//DBMnAhf\n/7qTjRn4Gc7H/AynPrS3OTqvv54maI4Zk3pqJ5yQkuYqq+QdmVlt+BmOtVvHHFP/++gsWQIPPJCK\nAO6/Hw47LM0p2n77vCMzq2/u4WTcw6kf9TpH5+23P5mg2a1b6s0cdVQqfDDrrDzxsw2ccOpLvczR\niYDHH0+VZuPHw8EHp0Sz666dY96QWWuccNrACaf+5DlHZ968VMRw2WXp5+OPT8N9a69d2zjM6p2f\n4ViHkMccnSlTUpIZNw4aGmDUqFSu7QmaZsvPCcfqWi3m6Hz4YdqV9LLL4OWX0wTNp59On21mleMh\ntYyH1OpbNfbRefHFNEHzd7+DHXZIz2YOPrh+ihTM2oNyhtQ8UGDtwj77wDnnwKBBaXOytlq0KFW+\n7bdfevAPac7PfffB4MFONmbVVNOEI+l7kiZLmp0dj0gaVNRmpKTXJL0v6UFJnyu6vpKk0ZJmSZon\nabyk9YvarCHpOknvZcdYSRXeXcRq7Zhj0nHQQWnV5XK89hqMHAkbbwznnQdHHw2vvgrnnpsq4cys\n+mrdw3kVOAXYAdgJeAD4k6RtACSdCvwAOBHYGZgJTJBUOP3vYmAwcBiwG9ADuFNaqkj1JmB7YF9g\nP2BHYGz1vpbVyn//N/TrlwoIFi1adtslS1LP5WtfS++ZORPuvhsmTUrDc9261SZmM8tERK4H8DYw\nLPv5deC0gmvdgDkF13sAC4DDC9r0ARYD+2Sv+wJLgP4FbQZm57ZYRhzRkT344IN5h1AxH30Usc8+\nESecELFkyae/26xZEeeeG7H55hHbbRdx+eURc+bkE2sldKS/u+b4+7Vv2e/Okn7f5/YMR1IXSYcD\n3YFJkjYBegMTmtpExIfAX4AB2amdSZV1hW1mAM8XtOkPzI2IxwraTALmF7TpdBobG/MOoWJWXDFV\nlU2alIbEGhsbiUivjz46DZFNmQJjx8JTT6VNzlZfPe+o264j/d01x9+v86j5I9Js+OxRUu9lLjA4\nIp6TtCsQwJtFb3kTWC/7uRewOCLebqZN01KPvYFZzXz0zII21s4VztFZbz247bZU3nz88XDRRbDW\nWnlHaGbF8qjJmQpsB/QEvgGMlbRHDnFYO9enT0o6xxwDF1wAe+3lCZpmda3UsbdqHaThsTHAJqTn\nLDsVXb8TuCb7eU/S85q1ito8A5yR/TwUmN3M58wFhiwjjvDhw4cPH+Ufpf6+r4dZB12AlSPi35Le\nAPYBngCQ1A3YHfhR1vYJYFHWZlzWpg+pUGBS1uZRYDVJ/Zue40gaAKwKPNJSEFHixCUzM2ubmiYc\nSWcDd5HKo1cHjgL2AJrm4lwE/ETSP4EXgJ+TeiY3AUTEHElXA6MkzQLeAc4H/gHcn7WZKule4ApJ\nxwECLgfuiIgXavJFzczsU2rdw+kNXJf9ORt4Gtg/IiYCRMSorFdzCbAm8Diwb0QUTvM7GVhI6uGs\nAkwEjs7K85ocAYwG/py9Hg8Mr9aXMjOz1nktNTMzq4lOXdMjafdsaZwZkpZI+nbeMVWKpJ9I+lu2\nhNBMSbdL+nzecVVKKcskdRTZ3+USSf+TdyyVIumM7DsVHq/nHVelSOot6XfZ//c+kPSMpN3zjqsS\nJP27mb+7JZLuaO29nTrhAKsBU4ARwPs5x1JpXyINTe5Kqu5bBEyUtEauUVXOMpdJ6igk9QeGAZPz\njqUKppLm1vXOjn75hlMZ2bqNk0gVXAcAW5OG9GfmGVcF7cwnf2e9SUuHBfD71t5YD1VquYmIe4B7\nACRdm3M4FRURBxS+lnQ06bnZQFLhRrsWEcX/mvq5pBNICfaZHEKquOwX1/WkUv+R+UZTFYsiorlJ\n2u3dqcDrETG04Nz0vIKptOKJ95KGkX633Nzaezt7D6cz6UH6+34370AqrWiZpBZL39uhK4E/RMRD\neQdSJZtmK8NPk3RTtrxVR/AV4HFJ4yS9KekpSSfmHVQVHQtcFxELWmvohNN5XAw8SZqn1CFI2kbS\nXNKCrpeSlkl6NuewKiL7V+OmpKkBHdFjwDGk1dz/H2lo5hFJa+YZVIVsCnwPeIm0Yv1FwG8kfS/X\nqKpA0r7AxqTJ+63q1ENqnYWkC0gLlw6MjlWW2OwySRHxXL5hLR9JWwK/Iv19Lck7nmqIiHsLX0t6\nDPg3MIT0C7o96wL8LSJ+lr2enP2dnkj6h1FHMgz4e0SUNIztHk4HJ+lC0t5Be0ZEhxlHBoiIRREx\nLSKeyv7P/Q/Sfkrt3a7AWsBzkhZKWkiaIH2ipI8krZhveJUXEe8DzwJb5B1LBfyHtIJ9oeeBDXOI\npWokrQMcQhr6LYl7OB2YpIuBQ4GGTrLKQhdg5byDqIA/An8vOvc74F/AryJiYc0jqrJswvfWpGrD\n9m4SsFXRua3oQIUDmaHAh2TLjJWiUyecbCfRzUnL33QBNpS0HfBORLyaa3DLSdJvgW+RHmDOltQr\nuzSvaOWGdqmEZZLarYiYAyw1LChpPul/l8X/cm6XJJ0L3AG8QiqNPp203mFHqBa9kLTH109JpcI7\nksqiT8s1qsr7DnBT1jstSadeaSDbFuFBUg15oWsj4tgcQqoYSUv49PcCODMizqp1PJUm6RqggaWX\nSRrVtExSRyPpAeCZiBiRdyyVIOkm0sK8a5P2r3oMOD0ipuYaWIVIOgA4G9iSlFRHR8Rv842qciQ1\nkNav/GJEPFHy+zpzwjEzs9px0YCZmdWEE46ZmdWEE46ZmdWEE46ZmdWEE46ZmdWEE46ZmdWEE46Z\nmdWEE46ZmdWEE45ZOyJpp2w736ovBClpSLb9g1lFOOGYtT8VXx4k26f+h7X4LOu8nHDMzKwmnHDM\nMpIelHSppPMkvS1ppqThklaSdImkdyVNl/StgvecLWmqpPezXsI5klYuuH6fpAkFr7tLekHS6BJj\n2l/S85I+kPQQaTHI4jYDJDVKmi9pRvYdVi/6XpdJukjSO9kxqvA6sBFwbjZct7jo/ntJmiJpnqQH\nJG1U4n9Ss6U44Zgt7UhgDvBF0mq/FwN/Av4J7ERaPv+qwu0eSFslbw2cQNrs7qcF9xsCbCfpR9nr\n0aQ9RH7cWiCS+pD2xrmXtLPpaGBUUZt+2fU/Af2AwVnb/23mewnoD3wX+K6k72fXvgbMAM4krb79\n2YL3dSMtq39M9t41gMtbi92sWRHhw4ePCEhbVUwqOjcT+FPB6xWABcDXWrjHccC/is59BfgAOCv7\nc5sS4/k1MLXo3M+AxcCG2etrgTFFbbYHlgBrF3yv5u7zSsHrfwM/LGozJPuszQvOHQl8kPfflY/2\neXTqDdjMmvF00euZwJSmFxGxSNK7wLoAkr4BnEzayG81oCtFIwcRMT7b/+XnwI+jxP3fSb2mx4rO\nPVr0eidgM0mHF5wT6WH/ZsBb2bnm7nOWpNUiYt4yYlgQES8WvH4dWEnSGhHxXilfwqyJE47Z0oq3\nb44WznWRtAtwE3AGaVjrPVJv5tzCxtkznS8Ai4AtKhxvF+Aq4AJSoin0WgXuv6jodVPVmofjrWxO\nOGZtNxCYERG/bjohaeNm2p0HrATsA9wn6a6IuLOE+z9Per5SaNei108Cn4+If7dyr12auc/rBb2b\nj0i9M7Oq8b9SzNruX8D6ko6UtImkE4DCoa2mrYaHAUdFxEPASOBqSeuWcP/LgY2z6rIts+G744ra\nnAN8MatC217SZpIOklT8YH89SRcW3OfHpF5Rk5eB3SWtJ2mtVuIq7kmZlcQJx+wTzU1ybOlcZL2U\nc4ELgcnA3sDpTY0krU2qFvtFRPxfdvo3wHPANa0GE/EqqYezH/AP0rOiU4vaTAG+RCprbsza/Qp4\no+h2N5B6MI8DVwBjgIsKrv83sAHwEum51TJDay12s+Yowv/bMevIsnk2UyJiRN6xWOfmHo6ZmdWE\nE45ZTrLnLnObOeZIurSCH+VhDKsLHlIzy0n2jKdHC5fnRMRbLVwza5eccMzMrCY8pGZmZjXhhGNm\nZjXhhGNmZjXhhGNmZjXx/wGenTmiq3o/agAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fca423d7828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot max_depth (x-axis) versus RMSE (y-axis)\n",
    "plt.plot(max_depth_range, RMSE_scores)\n",
    "plt.xlabel('max_depth')\n",
    "plt.ylabel('RMSE (lower is better)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,\n",
       "           max_leaf_nodes=None, min_samples_leaf=1, min_samples_split=2,\n",
       "           min_weight_fraction_leaf=0.0, presort=False, random_state=1,\n",
       "           splitter='best')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# max_depth=3 was best, so fit a tree using that parameter\n",
    "treereg = DecisionTreeRegressor(max_depth=3, random_state=1)\n",
    "treereg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>year</td>\n",
       "      <td>0.798744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>miles</td>\n",
       "      <td>0.201256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>doors</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>vtype</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  feature  importance\n",
       "0    year    0.798744\n",
       "1   miles    0.201256\n",
       "2   doors    0.000000\n",
       "3   vtype    0.000000"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# \"Gini importance\" of each feature: the (normalized) total reduction of error brought by that feature\n",
    "pd.DataFrame({'feature':feature_cols, 'importance':treereg.feature_importances_})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a tree diagram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a Graphviz file\n",
    "from sklearn.tree import export_graphviz\n",
    "export_graphviz(treereg, out_file='tree_vehicles.dot', feature_names=feature_cols)\n",
    "\n",
    "# At the command line, run this to convert to PNG:\n",
    "#   dot -Tpng tree_vehicles.dot -o tree_vehicles.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Tree for vehicle data](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/tree_vehicles.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading the internal nodes:\n",
    "\n",
    "- **samples:** number of observations in that node before splitting\n",
    "- **mse:** MSE calculated by comparing the actual response values in that node against the mean response value in that node\n",
    "- **rule:** rule used to split that node (go left if true, go right if false)\n",
    "\n",
    "Reading the leaves:\n",
    "\n",
    "- **samples:** number of observations in that node\n",
    "- **value:** mean response value in that node\n",
    "- **mse:** MSE calculated by comparing the actual response values in that node against \"value\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions for the testing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>year</th>\n",
       "      <th>miles</th>\n",
       "      <th>doors</th>\n",
       "      <th>vtype</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3000</td>\n",
       "      <td>2003</td>\n",
       "      <td>130000</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6000</td>\n",
       "      <td>2005</td>\n",
       "      <td>82500</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12000</td>\n",
       "      <td>2010</td>\n",
       "      <td>60000</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   price  year   miles  doors  vtype\n",
       "0   3000  2003  130000      4      1\n",
       "1   6000  2005   82500      4      0\n",
       "2  12000  2010   60000      2      0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read the testing data\n",
    "url = 'https://raw.githubusercontent.com/justmarkham/DAT8/master/data/vehicles_test.csv'\n",
    "test = pd.read_csv(url)\n",
    "test['vtype'] = test.vtype.map({'car':0, 'truck':1})\n",
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Using the tree diagram above, what predictions will the model make for each observation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  4000.,   5000.,  13500.])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use fitted model to make predictions on testing data\n",
    "X_test = test[feature_cols]\n",
    "y_test = test.price\n",
    "y_pred = treereg.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1190.2380714238084"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate RMSE\n",
    "np.sqrt(metrics.mean_squared_error(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Classification trees\n",
    "\n",
    "**Example:** Predict whether Barack Obama or Hillary Clinton will win the Democratic primary in a particular county in 2008:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Obama-Clinton decision tree](https://github.com/justmarkham/DAT8/raw/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/obama_clinton_tree.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Questions:**\n",
    "\n",
    "- What are the observations? How many observations are there?\n",
    "- What is the response variable?\n",
    "- What are the features?\n",
    "- What is the most predictive feature?\n",
    "- Why does the tree split on high school graduation rate twice in a row?\n",
    "- What is the class prediction for the following county: 15% African-American, 90% high school graduation rate, located in the South, high poverty, high population density?\n",
    "- What is the predicted probability for that same county?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing regression trees and classification trees\n",
    "\n",
    "|regression trees|classification trees|\n",
    "|---|---|\n",
    "|predict a continuous response|predict a categorical response|\n",
    "|predict using mean response of each leaf|predict using most commonly occuring class of each leaf|\n",
    "|splits are chosen to minimize MSE|splits are chosen to minimize Gini index (discussed below)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting criteria for classification trees\n",
    "\n",
    "Common options for the splitting criteria:\n",
    "\n",
    "- **classification error rate:** fraction of training observations in a region that don't belong to the most common class\n",
    "- **Gini index:** measure of total variance across classes in a region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example of classification error rate\n",
    "\n",
    "Pretend we are predicting whether someone buys an iPhone or an Android:\n",
    "\n",
    "- At a particular node, there are **25 observations** (phone buyers), of whom **10 bought iPhones and 15 bought Androids**.\n",
    "- Since the majority class is **Android**, that's our prediction for all 25 observations, and thus the classification error rate is **10/25 = 40%**.\n",
    "\n",
    "Our goal in making splits is to **reduce the classification error rate**. Let's try splitting on gender:\n",
    "\n",
    "- **Males:** 2 iPhones and 12 Androids, thus the predicted class is Android\n",
    "- **Females:** 8 iPhones and 3 Androids, thus the predicted class is iPhone\n",
    "- Classification error rate after this split would be **5/25 = 20%**\n",
    "\n",
    "Compare that with a split on age:\n",
    "\n",
    "- **30 or younger:** 4 iPhones and 8 Androids, thus the predicted class is Android\n",
    "- **31 or older:** 6 iPhones and 7 Androids, thus the predicted class is Android\n",
    "- Classification error rate after this split would be **10/25 = 40%**\n",
    "\n",
    "The decision tree algorithm will try **every possible split across all features**, and choose the split that **reduces the error rate the most.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example of Gini index\n",
    "\n",
    "Calculate the Gini index before making a split:\n",
    "\n",
    "$$1 - \\left(\\frac {iPhone} {Total}\\right)^2 - \\left(\\frac {Android} {Total}\\right)^2 = 1 - \\left(\\frac {10} {25}\\right)^2 - \\left(\\frac {15} {25}\\right)^2 = 0.48$$\n",
    "\n",
    "- The **maximum value** of the Gini index is 0.5, and occurs when the classes are perfectly balanced in a node.\n",
    "- The **minimum value** of the Gini index is 0, and occurs when there is only one class represented in a node.\n",
    "- A node with a lower Gini index is said to be more \"pure\".\n",
    "\n",
    "Evaluating the split on **gender** using Gini index:\n",
    "\n",
    "$$\\text{Males: } 1 - \\left(\\frac {2} {14}\\right)^2 - \\left(\\frac {12} {14}\\right)^2 = 0.24$$\n",
    "$$\\text{Females: } 1 - \\left(\\frac {8} {11}\\right)^2 - \\left(\\frac {3} {11}\\right)^2 = 0.40$$\n",
    "$$\\text{Weighted Average: } 0.24 \\left(\\frac {14} {25}\\right) + 0.40 \\left(\\frac {11} {25}\\right) = 0.31$$\n",
    "\n",
    "Evaluating the split on **age** using Gini index:\n",
    "\n",
    "$$\\text{30 or younger: } 1 - \\left(\\frac {4} {12}\\right)^2 - \\left(\\frac {8} {12}\\right)^2 = 0.44$$\n",
    "$$\\text{31 or older: } 1 - \\left(\\frac {6} {13}\\right)^2 - \\left(\\frac {7} {13}\\right)^2 = 0.50$$\n",
    "$$\\text{Weighted Average: } 0.44 \\left(\\frac {12} {25}\\right) + 0.50 \\left(\\frac {13} {25}\\right) = 0.47$$\n",
    "\n",
    "Again, the decision tree algorithm will try **every possible split**, and will choose the split that **reduces the Gini index (and thus increases the \"node purity\") the most.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing classification error rate and Gini index\n",
    "\n",
    "- Gini index is generally preferred because it will make splits that **increase node purity**, even if that split does not change the classification error rate.\n",
    "- Node purity is important because we're interested in the **class proportions** in each region, since that's how we calculate the **predicted probability** of each class.\n",
    "- scikit-learn's default splitting criteria for classification trees is Gini index.\n",
    "\n",
    "Note: There is another common splitting criteria called **cross-entropy**. It's numerically similar to Gini index, but slower to compute, thus it's not as popular as Gini index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a classification tree in scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll build a classification tree using the Titanic data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>0</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name  Sex  Age  SibSp  Parch  \\\n",
       "0                            Braund, Mr. Owen Harris    1   22      1      0   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...    0   38      1      0   \n",
       "2                             Heikkinen, Miss. Laina    0   26      0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)    0   35      1      0   \n",
       "4                           Allen, Mr. William Henry    1   35      0      0   \n",
       "\n",
       "             Ticket     Fare Cabin Embarked  Embarked_Q  Embarked_S  \n",
       "0         A/5 21171   7.2500   NaN        S           0           1  \n",
       "1          PC 17599  71.2833   C85        C           0           0  \n",
       "2  STON/O2. 3101282   7.9250   NaN        S           0           1  \n",
       "3            113803  53.1000  C123        S           0           1  \n",
       "4            373450   8.0500   NaN        S           0           1  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read in the data\n",
    "url = 'https://raw.githubusercontent.com/justmarkham/DAT8/master/data/titanic.csv'\n",
    "titanic = pd.read_csv(url)\n",
    "\n",
    "# encode female as 0 and male as 1\n",
    "titanic['Sex'] = titanic.Sex.map({'female':0, 'male':1})\n",
    "\n",
    "# fill in the missing values for age with the median age\n",
    "titanic.Age.fillna(titanic.Age.median(), inplace=True)\n",
    "\n",
    "# create a DataFrame of dummy variables for Embarked\n",
    "embarked_dummies = pd.get_dummies(titanic.Embarked, prefix='Embarked')\n",
    "embarked_dummies.drop(embarked_dummies.columns[0], axis=1, inplace=True)\n",
    "\n",
    "# concatenate the original DataFrame and the dummy DataFrame\n",
    "titanic = pd.concat([titanic, embarked_dummies], axis=1)\n",
    "\n",
    "# print the updated DataFrame\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Survived:** 0=died, 1=survived (response variable)\n",
    "- **Pclass:** 1=first class, 2=second class, 3=third class\n",
    "    - What will happen if the tree splits on this feature?\n",
    "- **Sex:** 0=female, 1=male\n",
    "- **Age:** numeric value\n",
    "- **Embarked:** C or Q or S"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define X and y\n",
    "feature_cols = ['Pclass', 'Sex', 'Age', 'Embarked_Q', 'Embarked_S']\n",
    "X = titanic[feature_cols]\n",
    "y = titanic.Survived"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,\n",
       "            max_features=None, max_leaf_nodes=None, min_samples_leaf=1,\n",
       "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "            presort=False, random_state=1, splitter='best')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit a classification tree with max_depth=3 on all data\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "treeclf = DecisionTreeClassifier(max_depth=3, random_state=1)\n",
    "treeclf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a Graphviz file\n",
    "export_graphviz(treeclf, out_file='tree_titanic.dot', feature_names=feature_cols)\n",
    "\n",
    "# At the command line, run this to convert to PNG:\n",
    "#   dot -Tpng tree_titanic.dot -o tree_titanic.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Tree for Titanic data](https://raw.githubusercontent.com/justmarkham/DAT8/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/tree_titanic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the split in the bottom right: the **same class** is predicted in both of its leaves. That split didn't affect the **classification error rate**, though it did increase the **node purity**, which is important because it increases the accuracy of our predicted probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pclass</td>\n",
       "      <td>0.242664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sex</td>\n",
       "      <td>0.655584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Age</td>\n",
       "      <td>0.064494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Embarked_Q</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Embarked_S</td>\n",
       "      <td>0.037258</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      feature  importance\n",
       "0      Pclass    0.242664\n",
       "1         Sex    0.655584\n",
       "2         Age    0.064494\n",
       "3  Embarked_Q    0.000000\n",
       "4  Embarked_S    0.037258"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute the feature importances\n",
    "pd.DataFrame({'feature':feature_cols, 'importance':treeclf.feature_importances_})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Comparing decision trees with other models\n",
    "\n",
    "**Advantages of decision trees:**\n",
    "\n",
    "- Can be used for regression or classification\n",
    "- Can be displayed graphically\n",
    "- Highly interpretable\n",
    "- Can be specified as a series of rules, and more closely approximate human decision-making than other models\n",
    "- Prediction is fast\n",
    "- Features don't need scaling\n",
    "- Automatically learns feature interactions\n",
    "- Tends to ignore irrelevant features\n",
    "- Non-parametric (will outperform linear models if relationship between features and response is highly non-linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Trees versus linear models](https://raw.githubusercontent.com/justmarkham/DAT8/226791169b1cc6df8e8845c12e34e748d5ffaa85/notebooks/images/tree_vs_linear.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Disadvantages of decision trees:**\n",
    "\n",
    "- Performance is (generally) not competitive with the best supervised learning methods\n",
    "- Can easily overfit the training data (tuning is required)\n",
    "- Small variations in the data can result in a completely different tree (high variance)\n",
    "- Recursive binary splitting makes \"locally optimal\" decisions that may not result in a globally optimal tree\n",
    "- Doesn't tend to work well if the classes are highly unbalanced\n",
    "- Doesn't tend to work well with very small datasets"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}