cars/Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cars\n",
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "library(ggplot2)\n",
    "library(dplyr)\n",
    "library(tidyr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "options(repr.plot.width = 10, repr.plot.height = 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df <- read.csv(\"data/cars.tidy.csv\", stringsAsFactor = FALSE)\n",
    "df$price_in_1000 <- df$price / 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Correlation of features\n",
    "### Mileage in city vs price\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cor(df$mileage_city, df$price_in_1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ggplot(df) + aes(mileage_city, price_in_1000) + geom_point()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mileage on highway vs Price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cor(df$mileage_highway, df$price_in_1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Engine capacity vs Mileage in city"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cor(df$engine, df$mileage_city)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Price vs Mileage in city"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cor(df$price_in_1000, df$mileage_city)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gears & Engine capacity vs Mileage on Highway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cor(c(df$engine * df$gears), df$mileage_highway)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df$mileage_city <- as.numeric(df$mileage_city)\n",
    "head(df$mileage_city)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Simple Linear Regression\n",
    "\n",
    "Simple linear regression is an approach for predicting a quantitative response using a single feature (or \"predictor\" or \"input variable\"). It takes the following form:\n",
    "\n",
    "    y=β0+β1x\n",
    "\n",
    "What does each term represent?\n",
    "\n",
    "* y is the response\n",
    "* x is the feature\n",
    "* β0 is the intercept\n",
    "* β1 is the coefficient for x\n",
    "\n",
    "Together, β0 and β1 are called the model coefficients. To create your model, you must \"learn\" the values of these coefficients. And once we've learned these coefficients, we can use the model to predict Price!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Estimating (\"Learning\") Model Coefficients\n",
    "Generally speaking, coefficients are estimated using the least squares criterion, which means we are find the line (mathematically) which minimizes the sum of squared residuals (or \"sum of squared errors\"):\n",
    "\n",
    "![](../img/estimating_coefficients.png)\n",
    "\n",
    "**What elements are present in the diagram?**\n",
    "\n",
    "* The black dots are the observed values of x and y.\n",
    "* The blue line is our least squares line.\n",
    "* The red lines are the residuals, which are the distances between the observed values and the least squares line.\n",
    "\n",
    "**How do the model coefficients relate to the least squares line?**\n",
    "* β0 is the intercept (the value of y when x=0)\n",
    "* β1 is the slope (the change in y divided by change in x)\n",
    "Here is a graphical depiction of those calculations:\n",
    "\n",
    "![](../img/slope_intercept.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model <- lm(price_in_1000 ~ mileage_city, data = df)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Model Coefficients\n",
    "\n",
    "How do we interpret the mileage coefficient (β1)?\n",
    "Increase in mileage is associated with a 1153 decrease in price.\n",
    "\n",
    "Note that if an increase in mileage was associated with a positive in price, β1 would be positive.\n",
    "\n",
    "#### Using the Model for Prediction\n",
    "\n",
    "    y=β0+β1x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the Output\n",
    "\n",
    "<table class=\"definitions\">\n",
    "\t<tr>\n",
    "\t\t<th>#</th>\n",
    "\t\t<th>Name</th>\n",
    "\t\t<th>Description</th>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td id=\"residuals\">1</td>\n",
    "\t\t<td>Residuals</td>\n",
    "\t\t<td>The residuals are the difference between the actual values of the variable you're predicting and predicted values from your regression--<code>y - &ycirc;</code>. For most regressions you want your residuals to look like a normal distribution when plotted. If our residuals are normally distributed, this indicates the mean of the difference between our predictions and the actual values is close to 0 (good) and that when we miss, we're missing both short and long of the actual value, and the likelihood of a miss being far from the actual value gets smaller as the distance from the actual value gets larger.<br><br>Think of it like a dartboard. A good model is going to hit the bullseye some of the time (but not everytime). When it doesn't hit the bullseye, it's missing in all of the other buckets evenly (i.e. not just missing in the 16 bin) and it also misses closer to the bullseye as opposed to on the outer edges of the dartboard.</td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>2</td>\n",
    "\t\t<td>Significance Stars</td>\n",
    "\t\t<td>The stars are shorthand for significance levels, with the number of asterisks displayed according to the p-value computed. <code>***</code> for high significance and <code>*</code> for low significance. </td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>3</td>\n",
    "\t\t<td>Estimated Coeffecient</td>\n",
    "\t\t<td>The estimated coefficient is the value of slope calculated by the regression. It might seem a little confusing that the Intercept also has a value, but just think of it as a slope that is always multiplied by 1. This number will obviously vary based on the magnitude of the variable you're inputting into the regression, but it's always good to spot check this number to make sure it seems reasonable.</td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>4</td>\n",
    "\t\t<td>Standard Error of the Coefficient Estimate</td>\n",
    "\t\t<td>Measure of the variability in the estimate for the coefficient. Lower means better but this number is relative to the value of the coefficient. As a rule of thumb, you'd like this value to be at least an order of magnitude less than the coefficient estimate.<br></td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>5</td>\n",
    "\t\t<td>t-value of the Coefficient Estimate</td>\n",
    "\t\t<td>Score that measures whether or not the coefficient for this variable is meaningful for the model. You probably won't use this value itself, but know that it is used to calculate the p-value and the significance levels.</td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>6</td>\n",
    "\t\t<td>Variable p-value</td>\n",
    "\t\t<td>Probability the variable is <em>NOT</em> relevant. You want this number to be as small as possible. If the number is <em>really</em> small, <code>R</code> will display it in scientific notation.</td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>7</td>\n",
    "\t\t<td>Significance Legend</td>\n",
    "\t\t<td>The more punctuation there is next to your variables, the better.<br><br>Blank=bad, Dots=pretty good, Stars=good, More Stars=very good</td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>8</td>\n",
    "\t\t<td>Residual Std Error / Degrees of Freedom</td>\n",
    "\t\t<td>The Residual Std Error is just the standard deviation of your residuals. You'd like this number to be proportional to the quantiles of the residuals in #1. For a normal distribution, the 1st and 3rd quantiles should be 1.5 +/- the std error. <br><br>The Degrees of Freedom is the difference between the number of observations included in your training sample and the number of variables used in your model (intercept counts as a variable).</td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>9</td>\n",
    "\t\t<td>R-squared</td>\n",
    "\t\t<td>Metric for evaluating the goodness of fit of your model. Higher is better with 1 being the best. Corresponds with the amount of variability in what you're predicting that is explained by the model.<br><span><font color=\"red\">WARNING:</font> While a high R-squared indicates good correlation, <a href=\"http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation\" title=\"correlation does not imply causation on wikipedia\" target=\"_blank\">correlation does <em>not</em> always imply causation</a>.</span></td>\n",
    "\t</tr>\n",
    "\t<tr>\n",
    "\t\t<td>10</td>\n",
    "\t\t<td>F-statistic &amp; resulting p-value</td>\n",
    "\t\t<td>Performs an <a href=\"http://en.wikipedia.org/wiki/F-test\">F-test</a> on the model. This takes the parameters of our model (in our case we only have 1) and compares it to a model that has fewer parameters. In theory the model with more parameters should fit better. If the model with more parameters (your model) doesn't perform better than the model with fewer parameters, the F-test will have a high p-value (probability <em>NOT</em> significant boost). If the model with more parameters is better than the model with fewer parameters, you will have a lower p-value.<br><br>\tThe DF, or degrees of freedom, pertains to how many variables are in the model.</td>\n",
    "\t</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Well Does the Model Fit the data?\n",
    "\n",
    "The most common way to evaluate the overall fit of a linear model is by the R-squared value. R-squared is the proportion of variance explained, meaning the proportion of variance in the observed data that is explained by the model, or the reduction in error over the null model. (The null model just predicts the mean of the observed response, and thus it has an intercept and no slope.)\n",
    "\n",
    "R-squared is between 0 and 1, and higher is better because it means that more variance is explained by the model. Here's an example of what R-squared \"looks like\":\n",
    "\n",
    "![](../img/r_squared.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Goodness of fit - R2 score\n",
    "\n",
    "![](../img/r2.gif)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model$fitted.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "resid <- data.frame(model$residuals, model$fitted.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "head(resid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ggplot(resid) + aes(y = model.residuals, x = model.fitted.values) + geom_point() + stat_smooth()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise plot log price vs engine capacity and build a model with it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using multiple variable for regression\n",
    "### Gears & Engine capacity vs Mileage on highway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ggplot(df) + aes(y = mileage_highway, x= engine, color = gears) + geom_point()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gears & Engine vs Mileage on highway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model <- lm(mileage_highway ~ gears + engine, data = df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model <- lm(mileage_highway ~ gears + engine - 1, data = df)\n",
    "summary(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.2.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}