intro_to_sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f441485c",
   "metadata": {},
   "source": [
    "# Intro to Sklearn - Machine Learning in Python\n",
    "\n",
    "## by Corey Wade\n",
    "\n",
    "The following Jupyter Notebook is an introduction to Machine Learning in Python for ODSC West attendees on Nov. 1, 2022. We will be using pandas for data analytics, and sklearn for machine learning. A wide range of models will be covered including Linear and Logistic Regression, Decision Trees, Random Forests, and XGBoost.\n",
    "\n",
    "This presentation includes ML fundamentals covered in Corey Wade's book [Hands-on Gradient Boosting with XGBoost and scikit-learn](https://www.amazon.com/Hands-Gradient-Boosting-XGBoost-scikit-learn/dp/1839218355). Another recommend text is [Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1492032646/). For great web references, check out [Jason Brownlee's Machine Learning Mastery](https://machinelearningmastery.com/about/).\n",
    "\n",
    "Our focus is on tabular data, that is, rows and columns of data sorted in tables; this is contrasted with images and text which are considered unstructured data. When it comes to images and text, neural networks usually perform better. For tabular data, neural networks do not necessarily have an edge. We will focus on XGBoost, one the strongest ML algorithms in the world, that often has an edge in tabular data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ed5173",
   "metadata": {},
   "source": [
    "# Module 1 - Preparing data for ML with pandas\n",
    "\n",
    "The following module provides a brief introduction to pandas. To go more in-depth, try tutorial options from the official documentation: https://pandas.pydata.org/docs/getting_started/tutorials.html."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12ad0d29",
   "metadata": {},
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09f087aa",
   "metadata": {},
   "source": [
    "### Bike Rentals Dataset\n",
    "\n",
    "The [Bike Rentals dataset](https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset) is from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php). It's been modified to include correcting null values for practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bdba1746",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>dteday</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.344167</td>\n",
       "      <td>0.363625</td>\n",
       "      <td>0.805833</td>\n",
       "      <td>0.160446</td>\n",
       "      <td>331</td>\n",
       "      <td>654</td>\n",
       "      <td>985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.363478</td>\n",
       "      <td>0.353739</td>\n",
       "      <td>0.696087</td>\n",
       "      <td>0.248539</td>\n",
       "      <td>131</td>\n",
       "      <td>670</td>\n",
       "      <td>801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.196364</td>\n",
       "      <td>0.189405</td>\n",
       "      <td>0.437273</td>\n",
       "      <td>0.248309</td>\n",
       "      <td>120</td>\n",
       "      <td>1229</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.212122</td>\n",
       "      <td>0.590435</td>\n",
       "      <td>0.160296</td>\n",
       "      <td>108</td>\n",
       "      <td>1454</td>\n",
       "      <td>1562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.226957</td>\n",
       "      <td>0.229270</td>\n",
       "      <td>0.436957</td>\n",
       "      <td>0.186900</td>\n",
       "      <td>82</td>\n",
       "      <td>1518</td>\n",
       "      <td>1600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   instant      dteday  season   yr  mnth  holiday  weekday  workingday  \\\n",
       "0        1  2011-01-01     1.0  0.0   1.0      0.0      6.0         0.0   \n",
       "1        2  2011-01-02     1.0  0.0   1.0      0.0      0.0         0.0   \n",
       "2        3  2011-01-03     1.0  0.0   1.0      0.0      1.0         1.0   \n",
       "3        4  2011-01-04     1.0  0.0   1.0      0.0      2.0         1.0   \n",
       "4        5  2011-01-05     1.0  0.0   1.0      0.0      3.0         1.0   \n",
       "\n",
       "   weathersit      temp     atemp       hum  windspeed  casual  registered  \\\n",
       "0           2  0.344167  0.363625  0.805833   0.160446     331         654   \n",
       "1           2  0.363478  0.353739  0.696087   0.248539     131         670   \n",
       "2           1  0.196364  0.189405  0.437273   0.248309     120        1229   \n",
       "3           1  0.200000  0.212122  0.590435   0.160296     108        1454   \n",
       "4           1  0.226957  0.229270  0.436957   0.186900      82        1518   \n",
       "\n",
       "    cnt  \n",
       "0   985  \n",
       "1   801  \n",
       "2  1349  \n",
       "3  1562  \n",
       "4  1600  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load data into pandas dataframe and show first 5 rows\n",
    "import pandas as pd\n",
    "df = pd.read_csv('bike_rentals.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4450dfe0",
   "metadata": {},
   "source": [
    "## General Data Info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5f32e417",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>731.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>730.000000</td>\n",
       "      <td>730.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>730.000000</td>\n",
       "      <td>730.000000</td>\n",
       "      <td>728.000000</td>\n",
       "      <td>726.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>731.000000</td>\n",
       "      <td>731.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>366.000000</td>\n",
       "      <td>2.496580</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>6.512329</td>\n",
       "      <td>0.028728</td>\n",
       "      <td>2.997264</td>\n",
       "      <td>0.682627</td>\n",
       "      <td>1.395349</td>\n",
       "      <td>0.495587</td>\n",
       "      <td>0.474512</td>\n",
       "      <td>0.627987</td>\n",
       "      <td>0.190476</td>\n",
       "      <td>848.176471</td>\n",
       "      <td>3656.172367</td>\n",
       "      <td>4504.348837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>211.165812</td>\n",
       "      <td>1.110807</td>\n",
       "      <td>0.500343</td>\n",
       "      <td>3.448303</td>\n",
       "      <td>0.167155</td>\n",
       "      <td>2.004787</td>\n",
       "      <td>0.465773</td>\n",
       "      <td>0.544894</td>\n",
       "      <td>0.183094</td>\n",
       "      <td>0.163017</td>\n",
       "      <td>0.142331</td>\n",
       "      <td>0.077725</td>\n",
       "      <td>686.622488</td>\n",
       "      <td>1560.256377</td>\n",
       "      <td>1937.211452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.059130</td>\n",
       "      <td>0.079070</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.022392</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>22.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>183.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.336875</td>\n",
       "      <td>0.337794</td>\n",
       "      <td>0.521562</td>\n",
       "      <td>0.134494</td>\n",
       "      <td>315.500000</td>\n",
       "      <td>2497.000000</td>\n",
       "      <td>3152.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>366.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.499166</td>\n",
       "      <td>0.487364</td>\n",
       "      <td>0.627083</td>\n",
       "      <td>0.180971</td>\n",
       "      <td>713.000000</td>\n",
       "      <td>3662.000000</td>\n",
       "      <td>4548.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>548.500000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>9.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.655625</td>\n",
       "      <td>0.608916</td>\n",
       "      <td>0.730104</td>\n",
       "      <td>0.233218</td>\n",
       "      <td>1096.000000</td>\n",
       "      <td>4776.500000</td>\n",
       "      <td>5956.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>731.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.861667</td>\n",
       "      <td>0.840896</td>\n",
       "      <td>0.972500</td>\n",
       "      <td>0.507463</td>\n",
       "      <td>3410.000000</td>\n",
       "      <td>6946.000000</td>\n",
       "      <td>8714.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          instant      season          yr        mnth     holiday     weekday  \\\n",
       "count  731.000000  731.000000  730.000000  730.000000  731.000000  731.000000   \n",
       "mean   366.000000    2.496580    0.500000    6.512329    0.028728    2.997264   \n",
       "std    211.165812    1.110807    0.500343    3.448303    0.167155    2.004787   \n",
       "min      1.000000    1.000000    0.000000    1.000000    0.000000    0.000000   \n",
       "25%    183.500000    2.000000    0.000000    4.000000    0.000000    1.000000   \n",
       "50%    366.000000    3.000000    0.500000    7.000000    0.000000    3.000000   \n",
       "75%    548.500000    3.000000    1.000000    9.750000    0.000000    5.000000   \n",
       "max    731.000000    4.000000    1.000000   12.000000    1.000000    6.000000   \n",
       "\n",
       "       workingday  weathersit        temp       atemp         hum   windspeed  \\\n",
       "count  731.000000  731.000000  730.000000  730.000000  728.000000  726.000000   \n",
       "mean     0.682627    1.395349    0.495587    0.474512    0.627987    0.190476   \n",
       "std      0.465773    0.544894    0.183094    0.163017    0.142331    0.077725   \n",
       "min      0.000000    1.000000    0.059130    0.079070    0.000000    0.022392   \n",
       "25%      0.000000    1.000000    0.336875    0.337794    0.521562    0.134494   \n",
       "50%      1.000000    1.000000    0.499166    0.487364    0.627083    0.180971   \n",
       "75%      1.000000    2.000000    0.655625    0.608916    0.730104    0.233218   \n",
       "max      1.000000    3.000000    0.861667    0.840896    0.972500    0.507463   \n",
       "\n",
       "            casual   registered          cnt  \n",
       "count   731.000000   731.000000   731.000000  \n",
       "mean    848.176471  3656.172367  4504.348837  \n",
       "std     686.622488  1560.256377  1937.211452  \n",
       "min       2.000000    20.000000    22.000000  \n",
       "25%     315.500000  2497.000000  3152.000000  \n",
       "50%     713.000000  3662.000000  4548.000000  \n",
       "75%    1096.000000  4776.500000  5956.000000  \n",
       "max    3410.000000  6946.000000  8714.000000  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show descriptive statistics\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "01115d05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>instant</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.122242e-01</td>\n",
       "      <td>8.660262e-01</td>\n",
       "      <td>0.494807</td>\n",
       "      <td>0.016145</td>\n",
       "      <td>-0.000016</td>\n",
       "      <td>-0.009415</td>\n",
       "      <td>-0.021477</td>\n",
       "      <td>0.152677</td>\n",
       "      <td>0.154502</td>\n",
       "      <td>0.013773</td>\n",
       "      <td>-0.113047</td>\n",
       "      <td>0.275255</td>\n",
       "      <td>0.659623</td>\n",
       "      <td>0.628830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>season</th>\n",
       "      <td>0.412224</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>-5.428568e-16</td>\n",
       "      <td>0.836863</td>\n",
       "      <td>-0.010537</td>\n",
       "      <td>-0.003080</td>\n",
       "      <td>0.016433</td>\n",
       "      <td>0.019211</td>\n",
       "      <td>0.336388</td>\n",
       "      <td>0.344739</td>\n",
       "      <td>0.209028</td>\n",
       "      <td>-0.228499</td>\n",
       "      <td>0.210399</td>\n",
       "      <td>0.411623</td>\n",
       "      <td>0.406100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yr</th>\n",
       "      <td>0.866026</td>\n",
       "      <td>-5.428568e-16</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>-0.003975</td>\n",
       "      <td>0.008195</td>\n",
       "      <td>-0.004103</td>\n",
       "      <td>-0.002945</td>\n",
       "      <td>-0.050322</td>\n",
       "      <td>0.050979</td>\n",
       "      <td>0.049350</td>\n",
       "      <td>-0.115456</td>\n",
       "      <td>-0.011963</td>\n",
       "      <td>0.249593</td>\n",
       "      <td>0.596168</td>\n",
       "      <td>0.568680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth</th>\n",
       "      <td>0.494807</td>\n",
       "      <td>8.368628e-01</td>\n",
       "      <td>-3.975295e-03</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.019599</td>\n",
       "      <td>0.011707</td>\n",
       "      <td>-0.007395</td>\n",
       "      <td>0.041218</td>\n",
       "      <td>0.226546</td>\n",
       "      <td>0.233626</td>\n",
       "      <td>0.227641</td>\n",
       "      <td>-0.206162</td>\n",
       "      <td>0.124549</td>\n",
       "      <td>0.296062</td>\n",
       "      <td>0.282624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>holiday</th>\n",
       "      <td>0.016145</td>\n",
       "      <td>-1.053666e-02</td>\n",
       "      <td>8.195345e-03</td>\n",
       "      <td>0.019599</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.101960</td>\n",
       "      <td>-0.252224</td>\n",
       "      <td>-0.034627</td>\n",
       "      <td>-0.028759</td>\n",
       "      <td>-0.032685</td>\n",
       "      <td>-0.016095</td>\n",
       "      <td>0.006319</td>\n",
       "      <td>0.054274</td>\n",
       "      <td>-0.108745</td>\n",
       "      <td>-0.068348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weekday</th>\n",
       "      <td>-0.000016</td>\n",
       "      <td>-3.079881e-03</td>\n",
       "      <td>-4.102570e-03</td>\n",
       "      <td>0.011707</td>\n",
       "      <td>-0.101960</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.038678</td>\n",
       "      <td>0.031087</td>\n",
       "      <td>-0.001830</td>\n",
       "      <td>-0.009003</td>\n",
       "      <td>-0.052728</td>\n",
       "      <td>0.014384</td>\n",
       "      <td>0.059923</td>\n",
       "      <td>0.057367</td>\n",
       "      <td>0.067443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workingday</th>\n",
       "      <td>-0.009415</td>\n",
       "      <td>1.643296e-02</td>\n",
       "      <td>-2.945396e-03</td>\n",
       "      <td>-0.007395</td>\n",
       "      <td>-0.252224</td>\n",
       "      <td>0.038678</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.057866</td>\n",
       "      <td>0.055573</td>\n",
       "      <td>0.055329</td>\n",
       "      <td>0.025879</td>\n",
       "      <td>-0.017720</td>\n",
       "      <td>-0.515692</td>\n",
       "      <td>0.306130</td>\n",
       "      <td>0.063781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit</th>\n",
       "      <td>-0.021477</td>\n",
       "      <td>1.921103e-02</td>\n",
       "      <td>-5.032247e-02</td>\n",
       "      <td>0.041218</td>\n",
       "      <td>-0.034627</td>\n",
       "      <td>0.031087</td>\n",
       "      <td>0.057866</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.119527</td>\n",
       "      <td>-0.120651</td>\n",
       "      <td>0.592841</td>\n",
       "      <td>0.038912</td>\n",
       "      <td>-0.247353</td>\n",
       "      <td>-0.260388</td>\n",
       "      <td>-0.297391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp</th>\n",
       "      <td>0.152677</td>\n",
       "      <td>3.363881e-01</td>\n",
       "      <td>5.097873e-02</td>\n",
       "      <td>0.226546</td>\n",
       "      <td>-0.028759</td>\n",
       "      <td>-0.001830</td>\n",
       "      <td>0.055573</td>\n",
       "      <td>-0.119527</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.991702</td>\n",
       "      <td>0.133380</td>\n",
       "      <td>-0.159242</td>\n",
       "      <td>0.543600</td>\n",
       "      <td>0.540327</td>\n",
       "      <td>0.627860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>atemp</th>\n",
       "      <td>0.154502</td>\n",
       "      <td>3.447388e-01</td>\n",
       "      <td>4.934973e-02</td>\n",
       "      <td>0.233626</td>\n",
       "      <td>-0.032685</td>\n",
       "      <td>-0.009003</td>\n",
       "      <td>0.055329</td>\n",
       "      <td>-0.120651</td>\n",
       "      <td>0.991702</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.146036</td>\n",
       "      <td>-0.184754</td>\n",
       "      <td>0.544113</td>\n",
       "      <td>0.544442</td>\n",
       "      <td>0.631357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hum</th>\n",
       "      <td>0.013773</td>\n",
       "      <td>2.090284e-01</td>\n",
       "      <td>-1.154557e-01</td>\n",
       "      <td>0.227641</td>\n",
       "      <td>-0.016095</td>\n",
       "      <td>-0.052728</td>\n",
       "      <td>0.025879</td>\n",
       "      <td>0.592841</td>\n",
       "      <td>0.133380</td>\n",
       "      <td>0.146036</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.251193</td>\n",
       "      <td>-0.076438</td>\n",
       "      <td>-0.090947</td>\n",
       "      <td>-0.100330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>windspeed</th>\n",
       "      <td>-0.113047</td>\n",
       "      <td>-2.284989e-01</td>\n",
       "      <td>-1.196259e-02</td>\n",
       "      <td>-0.206162</td>\n",
       "      <td>0.006319</td>\n",
       "      <td>0.014384</td>\n",
       "      <td>-0.017720</td>\n",
       "      <td>0.038912</td>\n",
       "      <td>-0.159242</td>\n",
       "      <td>-0.184754</td>\n",
       "      <td>-0.251193</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.167568</td>\n",
       "      <td>-0.217596</td>\n",
       "      <td>-0.234704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>casual</th>\n",
       "      <td>0.275255</td>\n",
       "      <td>2.103992e-01</td>\n",
       "      <td>2.495927e-01</td>\n",
       "      <td>0.124549</td>\n",
       "      <td>0.054274</td>\n",
       "      <td>0.059923</td>\n",
       "      <td>-0.515692</td>\n",
       "      <td>-0.247353</td>\n",
       "      <td>0.543600</td>\n",
       "      <td>0.544113</td>\n",
       "      <td>-0.076438</td>\n",
       "      <td>-0.167568</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.395282</td>\n",
       "      <td>0.672804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>registered</th>\n",
       "      <td>0.659623</td>\n",
       "      <td>4.116231e-01</td>\n",
       "      <td>5.961675e-01</td>\n",
       "      <td>0.296062</td>\n",
       "      <td>-0.108745</td>\n",
       "      <td>0.057367</td>\n",
       "      <td>0.306130</td>\n",
       "      <td>-0.260388</td>\n",
       "      <td>0.540327</td>\n",
       "      <td>0.544442</td>\n",
       "      <td>-0.090947</td>\n",
       "      <td>-0.217596</td>\n",
       "      <td>0.395282</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.945517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cnt</th>\n",
       "      <td>0.628830</td>\n",
       "      <td>4.061004e-01</td>\n",
       "      <td>5.686803e-01</td>\n",
       "      <td>0.282624</td>\n",
       "      <td>-0.068348</td>\n",
       "      <td>0.067443</td>\n",
       "      <td>0.063781</td>\n",
       "      <td>-0.297391</td>\n",
       "      <td>0.627860</td>\n",
       "      <td>0.631357</td>\n",
       "      <td>-0.100330</td>\n",
       "      <td>-0.234704</td>\n",
       "      <td>0.672804</td>\n",
       "      <td>0.945517</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             instant        season            yr      mnth   holiday  \\\n",
       "instant     1.000000  4.122242e-01  8.660262e-01  0.494807  0.016145   \n",
       "season      0.412224  1.000000e+00 -5.428568e-16  0.836863 -0.010537   \n",
       "yr          0.866026 -5.428568e-16  1.000000e+00 -0.003975  0.008195   \n",
       "mnth        0.494807  8.368628e-01 -3.975295e-03  1.000000  0.019599   \n",
       "holiday     0.016145 -1.053666e-02  8.195345e-03  0.019599  1.000000   \n",
       "weekday    -0.000016 -3.079881e-03 -4.102570e-03  0.011707 -0.101960   \n",
       "workingday -0.009415  1.643296e-02 -2.945396e-03 -0.007395 -0.252224   \n",
       "weathersit -0.021477  1.921103e-02 -5.032247e-02  0.041218 -0.034627   \n",
       "temp        0.152677  3.363881e-01  5.097873e-02  0.226546 -0.028759   \n",
       "atemp       0.154502  3.447388e-01  4.934973e-02  0.233626 -0.032685   \n",
       "hum         0.013773  2.090284e-01 -1.154557e-01  0.227641 -0.016095   \n",
       "windspeed  -0.113047 -2.284989e-01 -1.196259e-02 -0.206162  0.006319   \n",
       "casual      0.275255  2.103992e-01  2.495927e-01  0.124549  0.054274   \n",
       "registered  0.659623  4.116231e-01  5.961675e-01  0.296062 -0.108745   \n",
       "cnt         0.628830  4.061004e-01  5.686803e-01  0.282624 -0.068348   \n",
       "\n",
       "             weekday  workingday  weathersit      temp     atemp       hum  \\\n",
       "instant    -0.000016   -0.009415   -0.021477  0.152677  0.154502  0.013773   \n",
       "season     -0.003080    0.016433    0.019211  0.336388  0.344739  0.209028   \n",
       "yr         -0.004103   -0.002945   -0.050322  0.050979  0.049350 -0.115456   \n",
       "mnth        0.011707   -0.007395    0.041218  0.226546  0.233626  0.227641   \n",
       "holiday    -0.101960   -0.252224   -0.034627 -0.028759 -0.032685 -0.016095   \n",
       "weekday     1.000000    0.038678    0.031087 -0.001830 -0.009003 -0.052728   \n",
       "workingday  0.038678    1.000000    0.057866  0.055573  0.055329  0.025879   \n",
       "weathersit  0.031087    0.057866    1.000000 -0.119527 -0.120651  0.592841   \n",
       "temp       -0.001830    0.055573   -0.119527  1.000000  0.991702  0.133380   \n",
       "atemp      -0.009003    0.055329   -0.120651  0.991702  1.000000  0.146036   \n",
       "hum        -0.052728    0.025879    0.592841  0.133380  0.146036  1.000000   \n",
       "windspeed   0.014384   -0.017720    0.038912 -0.159242 -0.184754 -0.251193   \n",
       "casual      0.059923   -0.515692   -0.247353  0.543600  0.544113 -0.076438   \n",
       "registered  0.057367    0.306130   -0.260388  0.540327  0.544442 -0.090947   \n",
       "cnt         0.067443    0.063781   -0.297391  0.627860  0.631357 -0.100330   \n",
       "\n",
       "            windspeed    casual  registered       cnt  \n",
       "instant     -0.113047  0.275255    0.659623  0.628830  \n",
       "season      -0.228499  0.210399    0.411623  0.406100  \n",
       "yr          -0.011963  0.249593    0.596168  0.568680  \n",
       "mnth        -0.206162  0.124549    0.296062  0.282624  \n",
       "holiday      0.006319  0.054274   -0.108745 -0.068348  \n",
       "weekday      0.014384  0.059923    0.057367  0.067443  \n",
       "workingday  -0.017720 -0.515692    0.306130  0.063781  \n",
       "weathersit   0.038912 -0.247353   -0.260388 -0.297391  \n",
       "temp        -0.159242  0.543600    0.540327  0.627860  \n",
       "atemp       -0.184754  0.544113    0.544442  0.631357  \n",
       "hum         -0.251193 -0.076438   -0.090947 -0.100330  \n",
       "windspeed    1.000000 -0.167568   -0.217596 -0.234704  \n",
       "casual      -0.167568  1.000000    0.395282  0.672804  \n",
       "registered  -0.217596  0.395282    1.000000  0.945517  \n",
       "cnt         -0.234704  0.672804    0.945517  1.000000  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show correlations between columns\n",
    "df.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "522f45d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# show histograms and scatter plots of all columns\n",
    "import seaborn as sns\n",
    "#sns.pairplot(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5234c5a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 731 entries, 0 to 730\n",
      "Data columns (total 16 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   instant     731 non-null    int64  \n",
      " 1   dteday      731 non-null    object \n",
      " 2   season      731 non-null    float64\n",
      " 3   yr          730 non-null    float64\n",
      " 4   mnth        730 non-null    float64\n",
      " 5   holiday     731 non-null    float64\n",
      " 6   weekday     731 non-null    float64\n",
      " 7   workingday  731 non-null    float64\n",
      " 8   weathersit  731 non-null    int64  \n",
      " 9   temp        730 non-null    float64\n",
      " 10  atemp       730 non-null    float64\n",
      " 11  hum         728 non-null    float64\n",
      " 12  windspeed   726 non-null    float64\n",
      " 13  casual      731 non-null    int64  \n",
      " 14  registered  731 non-null    int64  \n",
      " 15  cnt         731 non-null    int64  \n",
      "dtypes: float64(10), int64(5), object(1)\n",
      "memory usage: 91.5+ KB\n"
     ]
    }
   ],
   "source": [
    "# get info on columns\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a891761b",
   "metadata": {},
   "source": [
    "## Null Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4325a580",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "instant       0\n",
       "dteday        0\n",
       "season        0\n",
       "yr            1\n",
       "mnth          1\n",
       "holiday       0\n",
       "weekday       0\n",
       "workingday    0\n",
       "weathersit    0\n",
       "temp          1\n",
       "atemp         1\n",
       "hum           3\n",
       "windspeed     5\n",
       "casual        0\n",
       "registered    0\n",
       "cnt           0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show total null values per column\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b4b3b069",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sum null values\n",
    "df.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "04c5e869",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>dteday</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>57</td>\n",
       "      <td>2011-02-26</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.282500</td>\n",
       "      <td>0.282192</td>\n",
       "      <td>0.537917</td>\n",
       "      <td>NaN</td>\n",
       "      <td>424</td>\n",
       "      <td>1545</td>\n",
       "      <td>1969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>82</td>\n",
       "      <td>2011-03-23</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.346957</td>\n",
       "      <td>0.337939</td>\n",
       "      <td>0.839565</td>\n",
       "      <td>NaN</td>\n",
       "      <td>203</td>\n",
       "      <td>1918</td>\n",
       "      <td>2121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>129</td>\n",
       "      <td>2011-05-09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.532500</td>\n",
       "      <td>0.525246</td>\n",
       "      <td>0.588750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>664</td>\n",
       "      <td>3698</td>\n",
       "      <td>4362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>130</td>\n",
       "      <td>2011-05-10</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.532500</td>\n",
       "      <td>0.522721</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.115671</td>\n",
       "      <td>694</td>\n",
       "      <td>4109</td>\n",
       "      <td>4803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>213</th>\n",
       "      <td>214</td>\n",
       "      <td>2011-08-02</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.783333</td>\n",
       "      <td>0.707071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.205850</td>\n",
       "      <td>801</td>\n",
       "      <td>4044</td>\n",
       "      <td>4845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>299</td>\n",
       "      <td>2011-10-26</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.484167</td>\n",
       "      <td>0.472846</td>\n",
       "      <td>0.720417</td>\n",
       "      <td>NaN</td>\n",
       "      <td>404</td>\n",
       "      <td>3490</td>\n",
       "      <td>3894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>389</td>\n",
       "      <td>2012-01-24</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.342500</td>\n",
       "      <td>0.349108</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.123767</td>\n",
       "      <td>439</td>\n",
       "      <td>3900</td>\n",
       "      <td>4339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>528</th>\n",
       "      <td>529</td>\n",
       "      <td>2012-06-12</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.653333</td>\n",
       "      <td>0.597875</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>477</td>\n",
       "      <td>4495</td>\n",
       "      <td>4972</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>701</th>\n",
       "      <td>702</td>\n",
       "      <td>2012-12-02</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.823333</td>\n",
       "      <td>0.124379</td>\n",
       "      <td>892</td>\n",
       "      <td>3757</td>\n",
       "      <td>4649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>730</th>\n",
       "      <td>731</td>\n",
       "      <td>2012-12-31</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.215833</td>\n",
       "      <td>0.223487</td>\n",
       "      <td>0.577500</td>\n",
       "      <td>0.154846</td>\n",
       "      <td>439</td>\n",
       "      <td>2290</td>\n",
       "      <td>2729</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     instant      dteday  season   yr  mnth  holiday  weekday  workingday  \\\n",
       "56        57  2011-02-26     1.0  0.0   2.0      0.0      6.0         0.0   \n",
       "81        82  2011-03-23     2.0  0.0   3.0      0.0      3.0         1.0   \n",
       "128      129  2011-05-09     2.0  0.0   5.0      0.0      1.0         1.0   \n",
       "129      130  2011-05-10     2.0  0.0   5.0      0.0      2.0         1.0   \n",
       "213      214  2011-08-02     3.0  0.0   8.0      0.0      2.0         1.0   \n",
       "298      299  2011-10-26     4.0  0.0  10.0      0.0      3.0         1.0   \n",
       "388      389  2012-01-24     1.0  1.0   1.0      0.0      2.0         1.0   \n",
       "528      529  2012-06-12     2.0  1.0   6.0      0.0      2.0         1.0   \n",
       "701      702  2012-12-02     4.0  1.0  12.0      0.0      0.0         0.0   \n",
       "730      731  2012-12-31     1.0  NaN   NaN      0.0      1.0         0.0   \n",
       "\n",
       "     weathersit      temp     atemp       hum  windspeed  casual  registered  \\\n",
       "56            1  0.282500  0.282192  0.537917        NaN     424        1545   \n",
       "81            2  0.346957  0.337939  0.839565        NaN     203        1918   \n",
       "128           1  0.532500  0.525246  0.588750        NaN     664        3698   \n",
       "129           1  0.532500  0.522721       NaN   0.115671     694        4109   \n",
       "213           1  0.783333  0.707071       NaN   0.205850     801        4044   \n",
       "298           2  0.484167  0.472846  0.720417        NaN     404        3490   \n",
       "388           1  0.342500  0.349108       NaN   0.123767     439        3900   \n",
       "528           2  0.653333  0.597875  0.833333        NaN     477        4495   \n",
       "701           2       NaN       NaN  0.823333   0.124379     892        3757   \n",
       "730           2  0.215833  0.223487  0.577500   0.154846     439        2290   \n",
       "\n",
       "      cnt  \n",
       "56   1969  \n",
       "81   2121  \n",
       "128  4362  \n",
       "129  4803  \n",
       "213  4845  \n",
       "298  3894  \n",
       "388  4339  \n",
       "528  4972  \n",
       "701  4649  \n",
       "730  2729  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shows all null values\n",
    "df[df.isna().any(axis=1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c86e37e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# change null values in column\n",
    "df['windspeed'] = df['windspeed'].fillna(df['windspeed'].median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3a0461ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# change null values for entire dataframe\n",
    "df = df.fillna(df.median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4859dd02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>dteday</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>130</td>\n",
       "      <td>2011-05-10</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.532500</td>\n",
       "      <td>0.522721</td>\n",
       "      <td>0.627083</td>\n",
       "      <td>0.115671</td>\n",
       "      <td>694</td>\n",
       "      <td>4109</td>\n",
       "      <td>4803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>213</th>\n",
       "      <td>214</td>\n",
       "      <td>2011-08-02</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.783333</td>\n",
       "      <td>0.707071</td>\n",
       "      <td>0.627083</td>\n",
       "      <td>0.205850</td>\n",
       "      <td>801</td>\n",
       "      <td>4044</td>\n",
       "      <td>4845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>730</th>\n",
       "      <td>731</td>\n",
       "      <td>2012-12-31</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.215833</td>\n",
       "      <td>0.223487</td>\n",
       "      <td>0.577500</td>\n",
       "      <td>0.154846</td>\n",
       "      <td>439</td>\n",
       "      <td>2290</td>\n",
       "      <td>2729</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     instant      dteday  season   yr  mnth  holiday  weekday  workingday  \\\n",
       "129      130  2011-05-10     2.0  0.0   5.0      0.0      2.0         1.0   \n",
       "213      214  2011-08-02     3.0  0.0   8.0      0.0      2.0         1.0   \n",
       "730      731  2012-12-31     1.0  0.5   7.0      0.0      1.0         0.0   \n",
       "\n",
       "     weathersit      temp     atemp       hum  windspeed  casual  registered  \\\n",
       "129           1  0.532500  0.522721  0.627083   0.115671     694        4109   \n",
       "213           1  0.783333  0.707071  0.627083   0.205850     801        4044   \n",
       "730           2  0.215833  0.223487  0.577500   0.154846     439        2290   \n",
       "\n",
       "      cnt  \n",
       "129  4803  \n",
       "213  4845  \n",
       "730  2729  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show rows\n",
    "df.iloc[[129,213,730]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "58a2bce1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>dteday</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>730</th>\n",
       "      <td>731</td>\n",
       "      <td>2012-12-31</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.215833</td>\n",
       "      <td>0.223487</td>\n",
       "      <td>0.5775</td>\n",
       "      <td>0.154846</td>\n",
       "      <td>439</td>\n",
       "      <td>2290</td>\n",
       "      <td>2729</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     instant      dteday  season   yr  mnth  holiday  weekday  workingday  \\\n",
       "730      731  2012-12-31     4.0  1.0   7.0      0.0      1.0         0.0   \n",
       "\n",
       "     weathersit      temp     atemp     hum  windspeed  casual  registered  \\\n",
       "730           2  0.215833  0.223487  0.5775   0.154846     439        2290   \n",
       "\n",
       "      cnt  \n",
       "730  2729  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# change null values by entry\n",
    "df.loc[730,'yr']=1.0\n",
    "df.loc[730, 'season']=4.0\n",
    "df.loc[[730]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d05733e4",
   "metadata": {},
   "source": [
    "## Choose X and y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "107a810d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>instant</th>\n",
       "      <th>dteday</th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.344167</td>\n",
       "      <td>0.363625</td>\n",
       "      <td>0.805833</td>\n",
       "      <td>0.160446</td>\n",
       "      <td>331</td>\n",
       "      <td>654</td>\n",
       "      <td>985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2011-01-02</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.363478</td>\n",
       "      <td>0.353739</td>\n",
       "      <td>0.696087</td>\n",
       "      <td>0.248539</td>\n",
       "      <td>131</td>\n",
       "      <td>670</td>\n",
       "      <td>801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>2011-01-03</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.196364</td>\n",
       "      <td>0.189405</td>\n",
       "      <td>0.437273</td>\n",
       "      <td>0.248309</td>\n",
       "      <td>120</td>\n",
       "      <td>1229</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>2011-01-04</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.212122</td>\n",
       "      <td>0.590435</td>\n",
       "      <td>0.160296</td>\n",
       "      <td>108</td>\n",
       "      <td>1454</td>\n",
       "      <td>1562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2011-01-05</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.226957</td>\n",
       "      <td>0.229270</td>\n",
       "      <td>0.436957</td>\n",
       "      <td>0.186900</td>\n",
       "      <td>82</td>\n",
       "      <td>1518</td>\n",
       "      <td>1600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   instant      dteday  season   yr  mnth  holiday  weekday  workingday  \\\n",
       "0        1  2011-01-01     1.0  0.0   1.0      0.0      6.0         0.0   \n",
       "1        2  2011-01-02     1.0  0.0   1.0      0.0      0.0         0.0   \n",
       "2        3  2011-01-03     1.0  0.0   1.0      0.0      1.0         1.0   \n",
       "3        4  2011-01-04     1.0  0.0   1.0      0.0      2.0         1.0   \n",
       "4        5  2011-01-05     1.0  0.0   1.0      0.0      3.0         1.0   \n",
       "\n",
       "   weathersit      temp     atemp       hum  windspeed  casual  registered  \\\n",
       "0           2  0.344167  0.363625  0.805833   0.160446     331         654   \n",
       "1           2  0.363478  0.353739  0.696087   0.248539     131         670   \n",
       "2           1  0.196364  0.189405  0.437273   0.248309     120        1229   \n",
       "3           1  0.200000  0.212122  0.590435   0.160296     108        1454   \n",
       "4           1  0.226957  0.229270  0.436957   0.186900      82        1518   \n",
       "\n",
       "    cnt  \n",
       "0   985  \n",
       "1   801  \n",
       "2  1349  \n",
       "3  1562  \n",
       "4  1600  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show order of columns\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c96c3555",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.344167</td>\n",
       "      <td>0.363625</td>\n",
       "      <td>0.805833</td>\n",
       "      <td>0.160446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.363478</td>\n",
       "      <td>0.353739</td>\n",
       "      <td>0.696087</td>\n",
       "      <td>0.248539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.196364</td>\n",
       "      <td>0.189405</td>\n",
       "      <td>0.437273</td>\n",
       "      <td>0.248309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.212122</td>\n",
       "      <td>0.590435</td>\n",
       "      <td>0.160296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.226957</td>\n",
       "      <td>0.229270</td>\n",
       "      <td>0.436957</td>\n",
       "      <td>0.186900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   season   yr  mnth  holiday  weekday  workingday  weathersit      temp  \\\n",
       "0     1.0  0.0   1.0      0.0      6.0         0.0           2  0.344167   \n",
       "1     1.0  0.0   1.0      0.0      0.0         0.0           2  0.363478   \n",
       "2     1.0  0.0   1.0      0.0      1.0         1.0           1  0.196364   \n",
       "3     1.0  0.0   1.0      0.0      2.0         1.0           1  0.200000   \n",
       "4     1.0  0.0   1.0      0.0      3.0         1.0           1  0.226957   \n",
       "\n",
       "      atemp       hum  windspeed  \n",
       "0  0.363625  0.805833   0.160446  \n",
       "1  0.353739  0.696087   0.248539  \n",
       "2  0.189405  0.437273   0.248309  \n",
       "3  0.212122  0.590435   0.160296  \n",
       "4  0.229270  0.436957   0.186900  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose X as all rows, and all columns excluding the first 2, and last 3\n",
    "X = df.iloc[:, 2:-3]\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9ddecf0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     985\n",
       "1     801\n",
       "2    1349\n",
       "3    1562\n",
       "4    1600\n",
       "Name: cnt, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose y as the last column\n",
    "y=df.iloc[:, -1]\n",
    "y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d621d92",
   "metadata": {},
   "source": [
    "## The Census Dataset\n",
    "\n",
    "The [Census Dataset](https://archive.ics.uci.edu/ml/datasets/Adult) (also called the Adult Dataset) is also from UCI. We include this dataset to balance regression with classification. Sklearn scoring metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "52b357eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education-num</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          workclass  fnlwgt   education  education-num  \\\n",
       "0   39          State-gov   77516   Bachelors             13   \n",
       "1   50   Self-emp-not-inc   83311   Bachelors             13   \n",
       "2   38            Private  215646     HS-grad              9   \n",
       "3   53            Private  234721        11th              7   \n",
       "4   28            Private  338409   Bachelors             13   \n",
       "\n",
       "        marital-status          occupation    relationship    race      sex  \\\n",
       "0        Never-married        Adm-clerical   Not-in-family   White     Male   \n",
       "1   Married-civ-spouse     Exec-managerial         Husband   White     Male   \n",
       "2             Divorced   Handlers-cleaners   Not-in-family   White     Male   \n",
       "3   Married-civ-spouse   Handlers-cleaners         Husband   Black     Male   \n",
       "4   Married-civ-spouse      Prof-specialty            Wife   Black   Female   \n",
       "\n",
       "   capital-gain  capital-loss  hours-per-week  native-country  income  \n",
       "0          2174             0              40   United-States   <=50K  \n",
       "1             0             0              13   United-States   <=50K  \n",
       "2             0             0              40   United-States   <=50K  \n",
       "3             0             0              40   United-States   <=50K  \n",
       "4             0             0              40            Cuba   <=50K  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# upload Census dataset with no header\n",
    "df2 = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data', header=None)\n",
    "\n",
    "# define columns by name\n",
    "df2.columns = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation',\n",
    "                  'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', \n",
    "                   'income']\n",
    "\n",
    "# show first 5 rows\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6d9036e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32561 entries, 0 to 32560\n",
      "Data columns (total 15 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   age             32561 non-null  int64 \n",
      " 1   workclass       32561 non-null  object\n",
      " 2   fnlwgt          32561 non-null  int64 \n",
      " 3   education       32561 non-null  object\n",
      " 4   education-num   32561 non-null  int64 \n",
      " 5   marital-status  32561 non-null  object\n",
      " 6   occupation      32561 non-null  object\n",
      " 7   relationship    32561 non-null  object\n",
      " 8   race            32561 non-null  object\n",
      " 9   sex             32561 non-null  object\n",
      " 10  capital-gain    32561 non-null  int64 \n",
      " 11  capital-loss    32561 non-null  int64 \n",
      " 12  hours-per-week  32561 non-null  int64 \n",
      " 13  native-country  32561 non-null  object\n",
      " 14  income          32561 non-null  object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 3.7+ MB\n"
     ]
    }
   ],
   "source": [
    "# get column info\n",
    "df2.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18153e6c",
   "metadata": {},
   "source": [
    "## One-hot encoding\n",
    "\n",
    "One-hot encoding means you take each categorical column (say Color), and transform it into new columns for each value (Red, Green, Blue) as the new column header; the new columns values are 1 for presence, and 0 for absence. pd.get_dummies() often works for this purpose. sklearn includes an additional [onehotencoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) that may be useful for pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b214c5bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use pd.get_dummies() to transform categorical into numerical columns\n",
    "df2 = pd.get_dummies(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "920550e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education-num</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>workclass_ ?</th>\n",
       "      <th>workclass_ Federal-gov</th>\n",
       "      <th>workclass_ Local-gov</th>\n",
       "      <th>workclass_ Never-worked</th>\n",
       "      <th>...</th>\n",
       "      <th>native-country_ Scotland</th>\n",
       "      <th>native-country_ South</th>\n",
       "      <th>native-country_ Taiwan</th>\n",
       "      <th>native-country_ Thailand</th>\n",
       "      <th>native-country_ Trinadad&amp;Tobago</th>\n",
       "      <th>native-country_ United-States</th>\n",
       "      <th>native-country_ Vietnam</th>\n",
       "      <th>native-country_ Yugoslavia</th>\n",
       "      <th>income_ &lt;=50K</th>\n",
       "      <th>income_ &gt;50K</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>77516</td>\n",
       "      <td>13</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>83311</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>215646</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>234721</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>338409</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 110 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  fnlwgt  education-num  capital-gain  capital-loss  hours-per-week  \\\n",
       "0   39   77516             13          2174             0              40   \n",
       "1   50   83311             13             0             0              13   \n",
       "2   38  215646              9             0             0              40   \n",
       "3   53  234721              7             0             0              40   \n",
       "4   28  338409             13             0             0              40   \n",
       "\n",
       "   workclass_ ?  workclass_ Federal-gov  workclass_ Local-gov  \\\n",
       "0             0                       0                     0   \n",
       "1             0                       0                     0   \n",
       "2             0                       0                     0   \n",
       "3             0                       0                     0   \n",
       "4             0                       0                     0   \n",
       "\n",
       "   workclass_ Never-worked  ...  native-country_ Scotland  \\\n",
       "0                        0  ...                         0   \n",
       "1                        0  ...                         0   \n",
       "2                        0  ...                         0   \n",
       "3                        0  ...                         0   \n",
       "4                        0  ...                         0   \n",
       "\n",
       "   native-country_ South  native-country_ Taiwan  native-country_ Thailand  \\\n",
       "0                      0                       0                         0   \n",
       "1                      0                       0                         0   \n",
       "2                      0                       0                         0   \n",
       "3                      0                       0                         0   \n",
       "4                      0                       0                         0   \n",
       "\n",
       "   native-country_ Trinadad&Tobago  native-country_ United-States  \\\n",
       "0                                0                              1   \n",
       "1                                0                              1   \n",
       "2                                0                              1   \n",
       "3                                0                              1   \n",
       "4                                0                              0   \n",
       "\n",
       "   native-country_ Vietnam  native-country_ Yugoslavia  income_ <=50K  \\\n",
       "0                        0                           0              1   \n",
       "1                        0                           0              1   \n",
       "2                        0                           0              1   \n",
       "3                        0                           0              1   \n",
       "4                        0                           0              1   \n",
       "\n",
       "   income_ >50K  \n",
       "0             0  \n",
       "1             0  \n",
       "2             0  \n",
       "3             0  \n",
       "4             0  \n",
       "\n",
       "[5 rows x 110 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show df after one-hot-encoding\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "84aa75c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32561 entries, 0 to 32560\n",
      "Columns: 110 entries, age to income_ >50K\n",
      "dtypes: int64(6), uint8(104)\n",
      "memory usage: 4.7 MB\n"
     ]
    }
   ],
   "source": [
    "# get new number of columns\n",
    "df2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d1e862ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# select X as all rows, columns except for last 2\n",
    "X2 = df2.iloc[:, :-2]\n",
    "\n",
    "# select y as last column\n",
    "y2 = df2.iloc[:, -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72d32542",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "21586bc6",
   "metadata": {},
   "source": [
    "# Module 2 - Supervised learning with sklearn\n",
    "\n",
    "In this modules we cover the essentials of the sklearn suite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "dcd088f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data into training and test set\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y2, test_size=0.2, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d45d5561",
   "metadata": {},
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "95388c40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8040621482429495"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import Linear Regression\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# initialize model\n",
    "model = LinearRegression()\n",
    "\n",
    "# fit model to training data\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# score model on test data (uses r2 default metric)\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8d12ef22",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  448.99545626,  1964.15211779,   -26.9008259 ,  -358.1634053 ,\n",
       "          76.05426566,    89.95264279,  -532.8684617 ,  2586.32297494,\n",
       "        3140.02805295, -1297.67659702, -2829.67689761])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show model coefficients\n",
    "model.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a86f2857",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'copy_X': True,\n",
       " 'fit_intercept': True,\n",
       " 'n_jobs': None,\n",
       " 'normalize': False,\n",
       " 'positive': False}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show model params\n",
    "model.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "6095c782",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6772.94745769, 4490.76552698])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show model predictions\n",
    "model.predict(X_test.iloc[-2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e86cd7cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "504    8294\n",
       "239    4334\n",
       "Name: cnt, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compare predictions to actual results\n",
    "y_test[-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7148362f",
   "metadata": {},
   "source": [
    "## Regressors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2c283f0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create function to score regressors\n",
    "def score_reg(model):\n",
    "    model.fit(X_train, y_train)\n",
    "    return model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "2754eb05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8172726869322732"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import score Decision Tree\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "score_reg(DecisionTreeRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e85a76fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8945909330392295"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import and score Random Forest\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "score_reg(RandomForestRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3f10c113",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: xgboost in /Users/coreyjwade/opt/anaconda3/lib/python3.8/site-packages (1.6.2)\n",
      "Requirement already satisfied: scipy in /Users/coreyjwade/opt/anaconda3/lib/python3.8/site-packages (from xgboost) (1.6.2)\n",
      "Requirement already satisfied: numpy in /Users/coreyjwade/opt/anaconda3/lib/python3.8/site-packages (from xgboost) (1.20.1)\n"
     ]
    }
   ],
   "source": [
    "# install XGBoost to your computer\n",
    "import sys\n",
    "!{sys.executable} -m pip install xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b0b4b3b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8911181007632755"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import and score XGBoost\n",
    "from xgboost import XGBRegressor\n",
    "score_reg(XGBRegressor())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6040640e",
   "metadata": {},
   "source": [
    "## Classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "67da76b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# write function to score classifiers\n",
    "def score_clf(model):\n",
    "    model.fit(X2_train, y2_train)\n",
    "    return model.score(X2_test, y2_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b72947fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7930293259634577"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import and score Logistic Regression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "score_clf(LogisticRegression())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f6c30c4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8136035621065562"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import and score Decision Tree for classification\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "score_clf(DecisionTreeClassifier(random_state=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "fbb4f0d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8484569322892677"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import and score Random Forest for classification\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "score_clf(RandomForestClassifier(random_state=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "df9cbf0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.869031168432366"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import and score XGBoost for classification\n",
    "from xgboost import XGBClassifier\n",
    "score_clf(XGBClassifier(random_state=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7afa1bb",
   "metadata": {},
   "source": [
    "## Predictions / Scoring Metrics\n",
    "\n",
    "Making meaningful predictions is arguably the most important part of Machine Learning. You use pandas or numpy arrays to make predictions with sklearn.\n",
    "\n",
    "There are many scoring metrics available in sklearn, especially for classification. See your options here: https://scikit-learn.org/stable/modules/model_evaluation.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fd6e5f3",
   "metadata": {},
   "source": [
    "### Root Mean Squared Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ed310b92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "680.5073611167628\n"
     ]
    }
   ],
   "source": [
    "# build XGBoost model\n",
    "model = XGBRegressor()\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# get predictions for test set\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "# for rmse, import mse first\n",
    "from sklearn.metrics import mean_squared_error\n",
    "# get mse\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "# computer rmse\n",
    "rmse = mse**0.5\n",
    "# show rmse\n",
    "print(rmse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "3299d32c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     731.000000\n",
       "mean     4504.348837\n",
       "std      1937.211452\n",
       "min        22.000000\n",
       "25%      3152.000000\n",
       "50%      4548.000000\n",
       "75%      5956.000000\n",
       "max      8714.000000\n",
       "Name: cnt, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show descriptive stats for y\n",
    "y.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba311464",
   "metadata": {},
   "source": [
    "### Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "9c645bbe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>566</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.665833</td>\n",
       "      <td>0.613025</td>\n",
       "      <td>0.844167</td>\n",
       "      <td>0.208967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>688</th>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.380833</td>\n",
       "      <td>0.375621</td>\n",
       "      <td>0.623333</td>\n",
       "      <td>0.235067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.606667</td>\n",
       "      <td>0.564412</td>\n",
       "      <td>0.862500</td>\n",
       "      <td>0.078383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.566908</td>\n",
       "      <td>0.456250</td>\n",
       "      <td>0.083975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.707059</td>\n",
       "      <td>0.647959</td>\n",
       "      <td>0.561765</td>\n",
       "      <td>0.304659</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     season   yr  mnth  holiday  weekday  workingday  weathersit      temp  \\\n",
       "566     3.0  1.0   7.0      0.0      5.0         1.0           2  0.665833   \n",
       "688     4.0  1.0  11.0      0.0      1.0         1.0           2  0.380833   \n",
       "266     4.0  0.0   9.0      0.0      6.0         0.0           2  0.606667   \n",
       "504     2.0  1.0   5.0      0.0      6.0         0.0           1  0.600000   \n",
       "239     3.0  0.0   8.0      0.0      0.0         0.0           1  0.707059   \n",
       "\n",
       "        atemp       hum  windspeed  \n",
       "566  0.613025  0.844167   0.208967  \n",
       "688  0.375621  0.623333   0.235067  \n",
       "266  0.564412  0.862500   0.078383  \n",
       "504  0.566908  0.456250   0.083975  \n",
       "239  0.647959  0.561765   0.304659  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at last 5 rows\n",
    "X_test.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "29bec669",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7707.2   , 4565.5044], dtype=float32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# predict last 2 rows\n",
    "model.predict(X_test.iloc[-2:,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "1d114c11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "504    8294\n",
       "239    4334\n",
       "Name: cnt, dtype: int64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check last 2 rows actual value\n",
    "y_test.iloc[-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7f3b73f",
   "metadata": {},
   "source": [
    "### NumPy Arrays\n",
    "\n",
    "It's often easier to make predictions from ML models when your inputs are NumPy Arrays. Then you don't have worry about column names. Pandas DataFrames, or NumPy Arrays are okay. NumPy Arrays are better for single rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9a932476",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert data to numpy arrays\n",
    "import numpy as np\n",
    "X_train_np = np.array(X_train)\n",
    "X_test_np = np.array(X_test)\n",
    "y_train_np = np.array(y_train)\n",
    "y_test_np = np.array(y_test)\n",
    "\n",
    "# train model on numpy arrays\n",
    "model = XGBRegressor()\n",
    "model.fit(X_train_np, y_train_np)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "7602bf0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>season</th>\n",
       "      <th>yr</th>\n",
       "      <th>mnth</th>\n",
       "      <th>holiday</th>\n",
       "      <th>weekday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weathersit</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>hum</th>\n",
       "      <th>windspeed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.707059</td>\n",
       "      <td>0.647959</td>\n",
       "      <td>0.561765</td>\n",
       "      <td>0.304659</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     season   yr  mnth  holiday  weekday  workingday  weathersit      temp  \\\n",
       "239     3.0  0.0   8.0      0.0      0.0         0.0           1  0.707059   \n",
       "\n",
       "        atemp       hum  windspeed  \n",
       "239  0.647959  0.561765   0.304659  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select last row to modify\n",
    "X_test.tail(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "48f68967",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3823.1265], dtype=float32)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show predictions\n",
    "model.predict(np.array([[3.0, 0.0, 8.0, 0.0, 0.0, 0.0, 2, 0.676667, 0.624388, 0.817500, 0.222633]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "7f94714e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3823.1265], dtype=float32)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now the prediction works, even though no column headers have been provided\n",
    "model.predict([[3.0, 0.0, 8.0, 0.0, 0.0, 0.0, 2, 0.676667, 0.624388, 0.817500, 0.222633]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba90710e",
   "metadata": {},
   "source": [
    "### Confusion Matrix and Classification Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "cb9f8f84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4587  331]\n",
      " [ 522 1073]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.90      0.93      0.91      4918\n",
      "           1       0.76      0.67      0.72      1595\n",
      "\n",
      "    accuracy                           0.87      6513\n",
      "   macro avg       0.83      0.80      0.82      6513\n",
      "weighted avg       0.87      0.87      0.87      6513\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# show confusion matrix and classification report\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "model = XGBClassifier()\n",
    "model.fit(X2_train, y2_train)\n",
    "y2_pred = model.predict(X2_test)\n",
    "print(confusion_matrix(y2_test, y2_pred))\n",
    "print(classification_report(y2_test, y2_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "70ff08b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7155718572857619"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the f1-score\n",
    "from sklearn.metrics import f1_score\n",
    "f1_score(y2_test, y2_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f35d259",
   "metadata": {},
   "source": [
    "## Your Turn!\n",
    "\n",
    "Try different models to get the best f1-score on the census dataset using random_state=0. You must use default params! (Changing params coming in Module 4 of this notebook.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "6c955cf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# try out different models using f1-scoring metric\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "394f788f",
   "metadata": {},
   "source": [
    "# Module 3 - Cross-validation with sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d9a009b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.41956178, 0.35431328, 0.14316349, 0.19342386, 0.64222721])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import cross_val_score to use cross-validation\n",
    "from sklearn.model_selection import cross_val_score\n",
    "# choose your model\n",
    "model=XGBRegressor()\n",
    "# get scores on five folds of data \n",
    "cross_val_score(model, X, y, scoring='r2', cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "ad60e2ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "934.0368073102878\n"
     ]
    }
   ],
   "source": [
    "# get mean rmse\n",
    "mse = cross_val_score(model, X, y, scoring='neg_mean_squared_error')\n",
    "rmse = (-mse)**0.5\n",
    "print(rmse.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "199edbe3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[680.50736112 674.32594581 660.63198199 813.16389678 654.28787799]\n",
      "696.5834127397749\n"
     ]
    }
   ],
   "source": [
    "# use KFold for shuffled, consistent folds \n",
    "from sklearn.model_selection import KFold\n",
    "kfold = KFold(n_splits=5, shuffle=True, random_state=0)\n",
    "model=XGBRegressor()\n",
    "mse = cross_val_score(model, X, y, scoring='neg_mean_squared_error', cv=kfold)\n",
    "rmse = (-mse)**0.5\n",
    "print(rmse)\n",
    "print(rmse.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "2977fd35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use stratified Kfold for classification to balance all test sets\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "ksfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)\n",
    "#clf=XGBClassifier()\n",
    "#f1 = cross_val_score(clf, X2, y2, scoring='f1', cv=ksfold)\n",
    "#print(f1.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcf9dbd6",
   "metadata": {},
   "source": [
    "# Module 4 - Fine-tuning models with sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e092346c",
   "metadata": {},
   "source": [
    "## GridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "7d598cae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'max_depth': 4}\n",
      "Best score: 660.9391538175233\n"
     ]
    }
   ],
   "source": [
    "# use GridSearchCV to search grid of hyperparameters for best values\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# GridSearch uses a dictionary of parameters to find optimal values\n",
    "params = {'max_depth':[1, 2, 3, 4, 5, 6, 8, 10]}\n",
    "\n",
    "# GridSearchCV takes an ML model, the dictionary of params, etc. as inputs\n",
    "model = XGBRegressor()\n",
    "grid_reg = GridSearchCV(model, params, scoring='neg_mean_squared_error', cv=kfold)\n",
    "\n",
    "# you fit gridsearch on training data just like an ml model\n",
    "grid_reg.fit(X_train, y_train)\n",
    "\n",
    "# now you can access the best parameters, with the best score\n",
    "best_params = grid_reg.best_params_\n",
    "print(\"Best params:\", best_params)\n",
    "best_score = (-grid_reg.best_score_)**0.5\n",
    "print(\"Best score:\", best_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ae8ceff0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function includes all steps in the cell above with XGBoost as the default model\n",
    "def grid_search(params, reg=XGBRegressor()):\n",
    "    grid_reg = GridSearchCV(reg, params, scoring='neg_mean_squared_error', cv=kfold)\n",
    "    grid_reg.fit(X_train, y_train)\n",
    "    best_params = grid_reg.best_params_\n",
    "    print(\"Best params:\", best_params)\n",
    "    best_score = (-grid_reg.best_score_)**0.5\n",
    "    print(\"Best score:\", best_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "4a9d2376",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method XGBModel.get_params of XGBRegressor(base_score=None, booster=None, callbacks=None,\n",
       "             colsample_bylevel=None, colsample_bynode=None,\n",
       "             colsample_bytree=None, early_stopping_rounds=None,\n",
       "             enable_categorical=False, eval_metric=None, gamma=None,\n",
       "             gpu_id=None, grow_policy=None, importance_type=None,\n",
       "             interaction_constraints=None, learning_rate=None, max_bin=None,\n",
       "             max_cat_to_onehot=None, max_delta_step=None, max_depth=None,\n",
       "             max_leaves=None, min_child_weight=None, missing=nan,\n",
       "             monotone_constraints=None, n_estimators=100, n_jobs=None,\n",
       "             num_parallel_tree=None, predictor=None, random_state=None,\n",
       "             reg_alpha=None, reg_lambda=None, ...)>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show params\n",
    "model.get_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "cf564763",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'max_depth': 4, 'n_estimators': 50}\n",
      "Best score: 655.9991890520657\n"
     ]
    }
   ],
   "source": [
    "# search 2 params - 12 models total\n",
    "grid_search({'max_depth':[3, 4, 5],\n",
    "            'n_estimators':[50, 100, 200, 400]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "6b2616ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'colsample_bytree': 1, 'max_depth': 4, 'n_estimators': 25}\n",
      "Best score: 654.3963754643038\n"
     ]
    }
   ],
   "source": [
    "# add additional params\n",
    "grid_search(params={'max_depth':[4],\n",
    "                    'colsample_bytree':[0.4, 0.6, 0.8, 1],\n",
    "                   'n_estimators':[25, 50, 100]})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ccf9829",
   "metadata": {},
   "source": [
    "## RandomSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "5376da92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# RandomizedSearchCV works the same way, but checks n (10 by default) random combinations\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "def random_search(params, reg=XGBRegressor()):\n",
    "    grid_reg = RandomizedSearchCV(reg, params, scoring='neg_mean_squared_error', cv=kfold, n_iter=10, random_state=0)\n",
    "    grid_reg.fit(X_train, y_train)\n",
    "    best_params = grid_reg.best_params_\n",
    "    print(\"Best params:\", best_params)\n",
    "    best_score = (-grid_reg.best_score_)**0.5\n",
    "    print(\"Best score:\", best_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "bcc41eef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'subsample': 0.7, 'n_estimators': 100, 'min_child_weight': 1, 'max_depth': 8, 'learning_rate': 0.1, 'colsample_bytree': 1, 'colsample_bynode': 0.8, 'colsample_bylevel': 0.6}\n",
      "Best score: 625.181686886698\n"
     ]
    }
   ],
   "source": [
    "# the following is a reasonable starting sample of params\n",
    "random_search(params={'subsample':[0.5, 0.6, 0.7, 0.8, 0.9, 1],\n",
    "        'colsample_bynode':[0.5, 0.6, 0.7, 0.8, 0.9, 1],\n",
    "        'colsample_bytree':[0.5, 0.6, 0.7, 0.8, 0.9, 1],\n",
    "        'colsample_bylevel':[0.5, 0.6, 0.7, 0.8, 0.9, 1], \n",
    "        'min_child_weight':[1, 2, 3, 4, 5], \n",
    "        'learning_rate':[0.001, 0.01, 0.1, 0.2, 0.4, 0.6], \n",
    "        'max_depth':[2, 3, 4, 5, 6, 8, 10], \n",
    "        'n_estimators':[25, 50, 100, 200, 400]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "3a933403",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'subsample': 0.6, 'n_estimators': 200, 'min_child_weight': 2, 'max_depth': 6, 'learning_rate': 0.05, 'colsample_bytree': 0.9, 'colsample_bynode': 0.9, 'colsample_bylevel': 0.7}\n",
      "Best score: 615.411076649143\n"
     ]
    }
   ],
   "source": [
    "# adjust based on results\n",
    "random_search(params={'subsample':[0.6, 0.7, 0.8],\n",
    "        'colsample_bynode':[0.7, 0.8, 0.9],\n",
    "        'colsample_bytree':[0.9, 1],\n",
    "        'colsample_bylevel':[0.5, 0.6, 0.7], \n",
    "        'min_child_weight':[1, 2], \n",
    "        'learning_rate':[0.05, 0.1, 0.25], \n",
    "        'max_depth':[6, 8, 10], \n",
    "        'n_estimators':[50, 100, 200]})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a5afe69",
   "metadata": {},
   "source": [
    "## Your turn!\n",
    "\n",
    "Try your own random and grid searches to get the best possible cv score on 5 folds using random_state=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "037b1038",
   "metadata": {},
   "outputs": [],
   "source": [
    "# try your own random searches, and/or grid searches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6eab774",
   "metadata": {},
   "source": [
    "# Module 5 - Feature Importances"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8bd499",
   "metadata": {},
   "source": [
    "## Finalize Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "9f79ec0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "607.1702124988886"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose your best model, fit on your data, then test against unseen data\n",
    "model = XGBRegressor(subsample=0.8, n_estimators=100, max_depth=8,\n",
    "                    learning_rate=0.1, colsample_bytree=1,\n",
    "                    colsample_bynode=0.4, colsample_bylevel=0.6)\n",
    "model.fit(X_train, y_train)\n",
    "y_pred = model.predict(X_test)\n",
    "mse = mean_squared_error(y_pred, y_test)\n",
    "mse**0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "4ed3962f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.19686565, 0.42768875, 0.05280402, 0.01175597, 0.01094317,\n",
       "       0.0068551 , 0.03675866, 0.13856605, 0.05325931, 0.03932855,\n",
       "       0.02517479], dtype=float32)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the influence of each column\n",
    "model.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "6fe5d360",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('yr', 0.42768875),\n",
       " ('season', 0.19686565),\n",
       " ('temp', 0.13856605),\n",
       " ('atemp', 0.053259306),\n",
       " ('mnth', 0.052804016),\n",
       " ('hum', 0.039328545),\n",
       " ('weathersit', 0.036758665),\n",
       " ('windspeed', 0.025174795),\n",
       " ('holiday', 0.011755966),\n",
       " ('weekday', 0.010943173),\n",
       " ('workingday', 0.006855099)]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# zip columns and feature_importances_ into dict\n",
    "feature_dict = dict(zip(X.columns, model.feature_importances_))\n",
    "\n",
    "# import operator\n",
    "import operator\n",
    "\n",
    "# sort dict by values (as list of tuples)\n",
    "sorted(feature_dict.items(), key=operator.itemgetter(1), reverse=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}