dataset_descriptions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Descriptions\n",
    "This notebook contains most of the datasets used in Pandas Cookbook along with the names, types, descriptions and some summary statistics of each column. This is not an exhaustive list as several datasets used in the book are quite small and are explained with enough detail in the book itself. The datasets presented here are the prominent ones that appear most frequently throughout the book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets in order of appearance\n",
    "* [Movie](#Movie-Dataset)\n",
    "* [College](#College-Dataset)\n",
    "* [Employee](#Employee-Dataset)\n",
    "* [Flights](#Flights-Dataset)\n",
    "* [Chinook Database](#Chinook-Database)\n",
    "* [Crime](#Crime-Dataset)\n",
    "* [Meetup Groups](#Meetup-Groups-Dataset)\n",
    "* [Diamonds](#Diamonds-Dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_columns = 80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Movie Dataset\n",
    "\n",
    "### Brief Overview\n",
    "28 columns from 4,916 movies scraped from the popular website IMDB. Each row contains information on a single movie dating back to 1916 to 2015. Actor and director facebook likes should be constant for all instances across all movies. For instance, Johnny Depp should have the same number of facebook likes regardless of which movie he is in. Since each movie was not scraped at the same exact time, there are some inconsistencies in these counts. The dataset **movie_altered.csv** is a much cleaner version of this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>color</th>\n",
       "      <th>director_name</th>\n",
       "      <th>num_critic_for_reviews</th>\n",
       "      <th>duration</th>\n",
       "      <th>director_facebook_likes</th>\n",
       "      <th>actor_3_facebook_likes</th>\n",
       "      <th>actor_2_name</th>\n",
       "      <th>actor_1_facebook_likes</th>\n",
       "      <th>gross</th>\n",
       "      <th>genres</th>\n",
       "      <th>actor_1_name</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>num_voted_users</th>\n",
       "      <th>cast_total_facebook_likes</th>\n",
       "      <th>actor_3_name</th>\n",
       "      <th>facenumber_in_poster</th>\n",
       "      <th>plot_keywords</th>\n",
       "      <th>movie_imdb_link</th>\n",
       "      <th>num_user_for_reviews</th>\n",
       "      <th>language</th>\n",
       "      <th>country</th>\n",
       "      <th>content_rating</th>\n",
       "      <th>budget</th>\n",
       "      <th>title_year</th>\n",
       "      <th>actor_2_facebook_likes</th>\n",
       "      <th>imdb_score</th>\n",
       "      <th>aspect_ratio</th>\n",
       "      <th>movie_facebook_likes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Color</td>\n",
       "      <td>James Cameron</td>\n",
       "      <td>723.0</td>\n",
       "      <td>178.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>855.0</td>\n",
       "      <td>Joel David Moore</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>760505847.0</td>\n",
       "      <td>Action|Adventure|Fantasy|Sci-Fi</td>\n",
       "      <td>CCH Pounder</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>886204</td>\n",
       "      <td>4834</td>\n",
       "      <td>Wes Studi</td>\n",
       "      <td>0.0</td>\n",
       "      <td>avatar|future|marine|native|paraplegic</td>\n",
       "      <td>http://www.imdb.com/title/tt0499549/?ref_=fn_t...</td>\n",
       "      <td>3054.0</td>\n",
       "      <td>English</td>\n",
       "      <td>USA</td>\n",
       "      <td>PG-13</td>\n",
       "      <td>237000000.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>936.0</td>\n",
       "      <td>7.9</td>\n",
       "      <td>1.78</td>\n",
       "      <td>33000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Color</td>\n",
       "      <td>Gore Verbinski</td>\n",
       "      <td>302.0</td>\n",
       "      <td>169.0</td>\n",
       "      <td>563.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>Orlando Bloom</td>\n",
       "      <td>40000.0</td>\n",
       "      <td>309404152.0</td>\n",
       "      <td>Action|Adventure|Fantasy</td>\n",
       "      <td>Johnny Depp</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>471220</td>\n",
       "      <td>48350</td>\n",
       "      <td>Jack Davenport</td>\n",
       "      <td>0.0</td>\n",
       "      <td>goddess|marriage ceremony|marriage proposal|pi...</td>\n",
       "      <td>http://www.imdb.com/title/tt0449088/?ref_=fn_t...</td>\n",
       "      <td>1238.0</td>\n",
       "      <td>English</td>\n",
       "      <td>USA</td>\n",
       "      <td>PG-13</td>\n",
       "      <td>300000000.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>5000.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>2.35</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Color</td>\n",
       "      <td>Sam Mendes</td>\n",
       "      <td>602.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>161.0</td>\n",
       "      <td>Rory Kinnear</td>\n",
       "      <td>11000.0</td>\n",
       "      <td>200074175.0</td>\n",
       "      <td>Action|Adventure|Thriller</td>\n",
       "      <td>Christoph Waltz</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>275868</td>\n",
       "      <td>11700</td>\n",
       "      <td>Stephanie Sigman</td>\n",
       "      <td>1.0</td>\n",
       "      <td>bomb|espionage|sequel|spy|terrorist</td>\n",
       "      <td>http://www.imdb.com/title/tt2379713/?ref_=fn_t...</td>\n",
       "      <td>994.0</td>\n",
       "      <td>English</td>\n",
       "      <td>UK</td>\n",
       "      <td>PG-13</td>\n",
       "      <td>245000000.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>393.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>2.35</td>\n",
       "      <td>85000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Color</td>\n",
       "      <td>Christopher Nolan</td>\n",
       "      <td>813.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>22000.0</td>\n",
       "      <td>23000.0</td>\n",
       "      <td>Christian Bale</td>\n",
       "      <td>27000.0</td>\n",
       "      <td>448130642.0</td>\n",
       "      <td>Action|Thriller</td>\n",
       "      <td>Tom Hardy</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>1144337</td>\n",
       "      <td>106759</td>\n",
       "      <td>Joseph Gordon-Levitt</td>\n",
       "      <td>0.0</td>\n",
       "      <td>deception|imprisonment|lawlessness|police offi...</td>\n",
       "      <td>http://www.imdb.com/title/tt1345836/?ref_=fn_t...</td>\n",
       "      <td>2701.0</td>\n",
       "      <td>English</td>\n",
       "      <td>USA</td>\n",
       "      <td>PG-13</td>\n",
       "      <td>250000000.0</td>\n",
       "      <td>2012.0</td>\n",
       "      <td>23000.0</td>\n",
       "      <td>8.5</td>\n",
       "      <td>2.35</td>\n",
       "      <td>164000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Doug Walker</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>131.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Rob Walker</td>\n",
       "      <td>131.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Documentary</td>\n",
       "      <td>Doug Walker</td>\n",
       "      <td>Star Wars: Episode VII - The Force Awakens</td>\n",
       "      <td>8</td>\n",
       "      <td>143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.imdb.com/title/tt5289954/?ref_=fn_t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   color      director_name  num_critic_for_reviews  duration  \\\n",
       "0  Color      James Cameron                   723.0     178.0   \n",
       "1  Color     Gore Verbinski                   302.0     169.0   \n",
       "2  Color         Sam Mendes                   602.0     148.0   \n",
       "3  Color  Christopher Nolan                   813.0     164.0   \n",
       "4    NaN        Doug Walker                     NaN       NaN   \n",
       "\n",
       "   director_facebook_likes  actor_3_facebook_likes      actor_2_name  \\\n",
       "0                      0.0                   855.0  Joel David Moore   \n",
       "1                    563.0                  1000.0     Orlando Bloom   \n",
       "2                      0.0                   161.0      Rory Kinnear   \n",
       "3                  22000.0                 23000.0    Christian Bale   \n",
       "4                    131.0                     NaN        Rob Walker   \n",
       "\n",
       "   actor_1_facebook_likes        gross                           genres  \\\n",
       "0                  1000.0  760505847.0  Action|Adventure|Fantasy|Sci-Fi   \n",
       "1                 40000.0  309404152.0         Action|Adventure|Fantasy   \n",
       "2                 11000.0  200074175.0        Action|Adventure|Thriller   \n",
       "3                 27000.0  448130642.0                  Action|Thriller   \n",
       "4                   131.0          NaN                      Documentary   \n",
       "\n",
       "      actor_1_name                                 movie_title  \\\n",
       "0      CCH Pounder                                      Avatar   \n",
       "1      Johnny Depp    Pirates of the Caribbean: At World's End   \n",
       "2  Christoph Waltz                                     Spectre   \n",
       "3        Tom Hardy                       The Dark Knight Rises   \n",
       "4      Doug Walker  Star Wars: Episode VII - The Force Awakens   \n",
       "\n",
       "   num_voted_users  cast_total_facebook_likes          actor_3_name  \\\n",
       "0           886204                       4834             Wes Studi   \n",
       "1           471220                      48350        Jack Davenport   \n",
       "2           275868                      11700      Stephanie Sigman   \n",
       "3          1144337                     106759  Joseph Gordon-Levitt   \n",
       "4                8                        143                   NaN   \n",
       "\n",
       "   facenumber_in_poster                                      plot_keywords  \\\n",
       "0                   0.0             avatar|future|marine|native|paraplegic   \n",
       "1                   0.0  goddess|marriage ceremony|marriage proposal|pi...   \n",
       "2                   1.0                bomb|espionage|sequel|spy|terrorist   \n",
       "3                   0.0  deception|imprisonment|lawlessness|police offi...   \n",
       "4                   0.0                                                NaN   \n",
       "\n",
       "                                     movie_imdb_link  num_user_for_reviews  \\\n",
       "0  http://www.imdb.com/title/tt0499549/?ref_=fn_t...                3054.0   \n",
       "1  http://www.imdb.com/title/tt0449088/?ref_=fn_t...                1238.0   \n",
       "2  http://www.imdb.com/title/tt2379713/?ref_=fn_t...                 994.0   \n",
       "3  http://www.imdb.com/title/tt1345836/?ref_=fn_t...                2701.0   \n",
       "4  http://www.imdb.com/title/tt5289954/?ref_=fn_t...                   NaN   \n",
       "\n",
       "  language country content_rating       budget  title_year  \\\n",
       "0  English     USA          PG-13  237000000.0      2009.0   \n",
       "1  English     USA          PG-13  300000000.0      2007.0   \n",
       "2  English      UK          PG-13  245000000.0      2015.0   \n",
       "3  English     USA          PG-13  250000000.0      2012.0   \n",
       "4      NaN     NaN            NaN          NaN         NaN   \n",
       "\n",
       "   actor_2_facebook_likes  imdb_score  aspect_ratio  movie_facebook_likes  \n",
       "0                   936.0         7.9          1.78                 33000  \n",
       "1                  5000.0         7.1          2.35                     0  \n",
       "2                   393.0         6.8          2.35                 85000  \n",
       "3                 23000.0         8.5          2.35                164000  \n",
       "4                    12.0         7.1           NaN                     0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie = pd.read_csv('data/movie.csv')\n",
    "movie.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4916, 28)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Most Common Value</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>color</th>\n",
       "      <td>object</td>\n",
       "      <td>Color or Black and White</td>\n",
       "      <td>19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Color</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>director_name</th>\n",
       "      <td>object</td>\n",
       "      <td>Director Name</td>\n",
       "      <td>102</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Steven Spielberg</td>\n",
       "      <td>2397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_critic_for_reviews</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of critical reviews</td>\n",
       "      <td>49</td>\n",
       "      <td>137.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration</th>\n",
       "      <td>float64</td>\n",
       "      <td>Length in minutes</td>\n",
       "      <td>15</td>\n",
       "      <td>107.09</td>\n",
       "      <td>NaN</td>\n",
       "      <td>191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>director_facebook_likes</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of Facebook likes</td>\n",
       "      <td>102</td>\n",
       "      <td>691.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor_3_facebook_likes</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of Facebook likes</td>\n",
       "      <td>23</td>\n",
       "      <td>631.28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor_2_name</th>\n",
       "      <td>object</td>\n",
       "      <td>Second most prominent actor name</td>\n",
       "      <td>13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Morgan Freeman</td>\n",
       "      <td>3030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor_1_facebook_likes</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of Facebook likes</td>\n",
       "      <td>7</td>\n",
       "      <td>6494.49</td>\n",
       "      <td>NaN</td>\n",
       "      <td>877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gross</th>\n",
       "      <td>float64</td>\n",
       "      <td>Total amount of revenue earned</td>\n",
       "      <td>862</td>\n",
       "      <td>47644514.53</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genres</th>\n",
       "      <td>object</td>\n",
       "      <td>Pipe separated list of all genres in movie</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Drama</td>\n",
       "      <td>914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor_1_name</th>\n",
       "      <td>object</td>\n",
       "      <td>Most prominent actor name</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Robert De Niro</td>\n",
       "      <td>2095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_title</th>\n",
       "      <td>object</td>\n",
       "      <td>Movie title</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Die Hard with a Vengeance</td>\n",
       "      <td>4916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_voted_users</th>\n",
       "      <td>int64</td>\n",
       "      <td>Number of users that scored movie</td>\n",
       "      <td>0</td>\n",
       "      <td>82644.92</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cast_total_facebook_likes</th>\n",
       "      <td>int64</td>\n",
       "      <td>All actor</td>\n",
       "      <td>0</td>\n",
       "      <td>9579.82</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor_3_name</th>\n",
       "      <td>object</td>\n",
       "      <td>Third most prominent actor name</td>\n",
       "      <td>23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Steve Coogan</td>\n",
       "      <td>3519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>facenumber_in_poster</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of faces in movie poster</td>\n",
       "      <td>13</td>\n",
       "      <td>1.38</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>plot_keywords</th>\n",
       "      <td>object</td>\n",
       "      <td>Pipe separated list of plot keywords</td>\n",
       "      <td>152</td>\n",
       "      <td>NaN</td>\n",
       "      <td>based on novel</td>\n",
       "      <td>4756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_imdb_link</th>\n",
       "      <td>object</td>\n",
       "      <td>URL of IMDB</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.imdb.com/title/tt0117333/?ref_=fn_t...</td>\n",
       "      <td>4916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>num_user_for_reviews</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of user reviews</td>\n",
       "      <td>21</td>\n",
       "      <td>267.67</td>\n",
       "      <td>NaN</td>\n",
       "      <td>954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>language</th>\n",
       "      <td>object</td>\n",
       "      <td>Language</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <td>object</td>\n",
       "      <td>Country</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>USA</td>\n",
       "      <td>65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>content_rating</th>\n",
       "      <td>object</td>\n",
       "      <td>G/PG/PG-13/R</td>\n",
       "      <td>300</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>budget</th>\n",
       "      <td>float64</td>\n",
       "      <td>Budget in country currency</td>\n",
       "      <td>484</td>\n",
       "      <td>36547486.03</td>\n",
       "      <td>NaN</td>\n",
       "      <td>438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title_year</th>\n",
       "      <td>float64</td>\n",
       "      <td>Year movie was made 1916-2015</td>\n",
       "      <td>106</td>\n",
       "      <td>2002.45</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actor_2_facebook_likes</th>\n",
       "      <td>float64</td>\n",
       "      <td>Number of Facebook likes</td>\n",
       "      <td>13</td>\n",
       "      <td>1621.92</td>\n",
       "      <td>NaN</td>\n",
       "      <td>917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imdb_score</th>\n",
       "      <td>float64</td>\n",
       "      <td>Score from users</td>\n",
       "      <td>0</td>\n",
       "      <td>6.44</td>\n",
       "      <td>NaN</td>\n",
       "      <td>78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aspect_ratio</th>\n",
       "      <td>float64</td>\n",
       "      <td>Proportion of width to height</td>\n",
       "      <td>326</td>\n",
       "      <td>2.22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_facebook_likes</th>\n",
       "      <td>int64</td>\n",
       "      <td>Number of Facebook likes for movie</td>\n",
       "      <td>0</td>\n",
       "      <td>7348.29</td>\n",
       "      <td>NaN</td>\n",
       "      <td>876</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Data Type  \\\n",
       "Column Name                           \n",
       "color                        object   \n",
       "director_name                object   \n",
       "num_critic_for_reviews      float64   \n",
       "duration                    float64   \n",
       "director_facebook_likes     float64   \n",
       "actor_3_facebook_likes      float64   \n",
       "actor_2_name                 object   \n",
       "actor_1_facebook_likes      float64   \n",
       "gross                       float64   \n",
       "genres                       object   \n",
       "actor_1_name                 object   \n",
       "movie_title                  object   \n",
       "num_voted_users               int64   \n",
       "cast_total_facebook_likes     int64   \n",
       "actor_3_name                 object   \n",
       "facenumber_in_poster        float64   \n",
       "plot_keywords                object   \n",
       "movie_imdb_link              object   \n",
       "num_user_for_reviews        float64   \n",
       "language                     object   \n",
       "country                      object   \n",
       "content_rating               object   \n",
       "budget                      float64   \n",
       "title_year                  float64   \n",
       "actor_2_facebook_likes      float64   \n",
       "imdb_score                  float64   \n",
       "aspect_ratio                float64   \n",
       "movie_facebook_likes          int64   \n",
       "\n",
       "                                                   Column Description  \\\n",
       "Column Name                                                             \n",
       "color                                        Color or Black and White   \n",
       "director_name                                           Director Name   \n",
       "num_critic_for_reviews                     Number of critical reviews   \n",
       "duration                                            Length in minutes   \n",
       "director_facebook_likes                      Number of Facebook likes   \n",
       "actor_3_facebook_likes                       Number of Facebook likes   \n",
       "actor_2_name                         Second most prominent actor name   \n",
       "actor_1_facebook_likes                       Number of Facebook likes   \n",
       "gross                                  Total amount of revenue earned   \n",
       "genres                     Pipe separated list of all genres in movie   \n",
       "actor_1_name                                Most prominent actor name   \n",
       "movie_title                                               Movie title   \n",
       "num_voted_users                     Number of users that scored movie   \n",
       "cast_total_facebook_likes                                   All actor   \n",
       "actor_3_name                          Third most prominent actor name   \n",
       "facenumber_in_poster                  Number of faces in movie poster   \n",
       "plot_keywords                    Pipe separated list of plot keywords   \n",
       "movie_imdb_link                                           URL of IMDB   \n",
       "num_user_for_reviews                           Number of user reviews   \n",
       "language                                                     Language   \n",
       "country                                                       Country   \n",
       "content_rating                                           G/PG/PG-13/R   \n",
       "budget                                     Budget in country currency   \n",
       "title_year                              Year movie was made 1916-2015   \n",
       "actor_2_facebook_likes                       Number of Facebook likes   \n",
       "imdb_score                                           Score from users   \n",
       "aspect_ratio                            Proportion of width to height   \n",
       "movie_facebook_likes               Number of Facebook likes for movie   \n",
       "\n",
       "                           Missing Values         Mean  \\\n",
       "Column Name                                              \n",
       "color                                  19          NaN   \n",
       "director_name                         102          NaN   \n",
       "num_critic_for_reviews                 49       137.99   \n",
       "duration                               15       107.09   \n",
       "director_facebook_likes               102       691.01   \n",
       "actor_3_facebook_likes                 23       631.28   \n",
       "actor_2_name                           13          NaN   \n",
       "actor_1_facebook_likes                  7      6494.49   \n",
       "gross                                 862  47644514.53   \n",
       "genres                                  0          NaN   \n",
       "actor_1_name                            7          NaN   \n",
       "movie_title                             0          NaN   \n",
       "num_voted_users                         0     82644.92   \n",
       "cast_total_facebook_likes               0      9579.82   \n",
       "actor_3_name                           23          NaN   \n",
       "facenumber_in_poster                   13         1.38   \n",
       "plot_keywords                         152          NaN   \n",
       "movie_imdb_link                         0          NaN   \n",
       "num_user_for_reviews                   21       267.67   \n",
       "language                               12          NaN   \n",
       "country                                 5          NaN   \n",
       "content_rating                        300          NaN   \n",
       "budget                                484  36547486.03   \n",
       "title_year                            106      2002.45   \n",
       "actor_2_facebook_likes                 13      1621.92   \n",
       "imdb_score                              0         6.44   \n",
       "aspect_ratio                          326         2.22   \n",
       "movie_facebook_likes                    0      7348.29   \n",
       "\n",
       "                                                           Most Common Value  \\\n",
       "Column Name                                                                    \n",
       "color                                                                  Color   \n",
       "director_name                                               Steven Spielberg   \n",
       "num_critic_for_reviews                                                   NaN   \n",
       "duration                                                                 NaN   \n",
       "director_facebook_likes                                                  NaN   \n",
       "actor_3_facebook_likes                                                   NaN   \n",
       "actor_2_name                                                  Morgan Freeman   \n",
       "actor_1_facebook_likes                                                   NaN   \n",
       "gross                                                                    NaN   \n",
       "genres                                                                 Drama   \n",
       "actor_1_name                                                  Robert De Niro   \n",
       "movie_title                                        Die Hard with a Vengeance   \n",
       "num_voted_users                                                          NaN   \n",
       "cast_total_facebook_likes                                                NaN   \n",
       "actor_3_name                                                    Steve Coogan   \n",
       "facenumber_in_poster                                                     NaN   \n",
       "plot_keywords                                                 based on novel   \n",
       "movie_imdb_link            http://www.imdb.com/title/tt0117333/?ref_=fn_t...   \n",
       "num_user_for_reviews                                                     NaN   \n",
       "language                                                             English   \n",
       "country                                                                  USA   \n",
       "content_rating                                                             R   \n",
       "budget                                                                   NaN   \n",
       "title_year                                                               NaN   \n",
       "actor_2_facebook_likes                                                   NaN   \n",
       "imdb_score                                                               NaN   \n",
       "aspect_ratio                                                             NaN   \n",
       "movie_facebook_likes                                                     NaN   \n",
       "\n",
       "                           Number of Unique Values  \n",
       "Column Name                                         \n",
       "color                                            2  \n",
       "director_name                                 2397  \n",
       "num_critic_for_reviews                         528  \n",
       "duration                                       191  \n",
       "director_facebook_likes                        435  \n",
       "actor_3_facebook_likes                         906  \n",
       "actor_2_name                                  3030  \n",
       "actor_1_facebook_likes                         877  \n",
       "gross                                         4033  \n",
       "genres                                         914  \n",
       "actor_1_name                                  2095  \n",
       "movie_title                                   4916  \n",
       "num_voted_users                               4750  \n",
       "cast_total_facebook_likes                     3960  \n",
       "actor_3_name                                  3519  \n",
       "facenumber_in_poster                            19  \n",
       "plot_keywords                                 4756  \n",
       "movie_imdb_link                               4916  \n",
       "num_user_for_reviews                           954  \n",
       "language                                        47  \n",
       "country                                         65  \n",
       "content_rating                                  18  \n",
       "budget                                         438  \n",
       "title_year                                      91  \n",
       "actor_2_facebook_likes                         917  \n",
       "imdb_score                                      78  \n",
       "aspect_ratio                                    22  \n",
       "movie_facebook_likes                           876  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/movie_decsription.csv', index_col='Column Name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# College Dataset\n",
    "\n",
    "### Brief Overview\n",
    "\n",
    "US department of education data on 7,535 colleges. Only a sample of the total number of columns available were used in this dataset. Visit [the website](https://collegescorecard.ed.gov/data/) for more info. Data was pulled in January, 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INSTNM</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STABBR</th>\n",
       "      <th>HBCU</th>\n",
       "      <th>MENONLY</th>\n",
       "      <th>WOMENONLY</th>\n",
       "      <th>RELAFFIL</th>\n",
       "      <th>SATVRMID</th>\n",
       "      <th>SATMTMID</th>\n",
       "      <th>DISTANCEONLY</th>\n",
       "      <th>UGDS</th>\n",
       "      <th>UGDS_WHITE</th>\n",
       "      <th>UGDS_BLACK</th>\n",
       "      <th>UGDS_HISP</th>\n",
       "      <th>UGDS_ASIAN</th>\n",
       "      <th>UGDS_AIAN</th>\n",
       "      <th>UGDS_NHPI</th>\n",
       "      <th>UGDS_2MOR</th>\n",
       "      <th>UGDS_NRA</th>\n",
       "      <th>UGDS_UNKN</th>\n",
       "      <th>PPTUG_EF</th>\n",
       "      <th>CURROPER</th>\n",
       "      <th>PCTPELL</th>\n",
       "      <th>PCTFLOAN</th>\n",
       "      <th>UG25ABV</th>\n",
       "      <th>MD_EARN_WNE_P10</th>\n",
       "      <th>GRAD_DEBT_MDN_SUPP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama A &amp; M University</td>\n",
       "      <td>Normal</td>\n",
       "      <td>AL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>424.0</td>\n",
       "      <td>420.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4206.0</td>\n",
       "      <td>0.0333</td>\n",
       "      <td>0.9353</td>\n",
       "      <td>0.0055</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>0.0024</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0059</td>\n",
       "      <td>0.0138</td>\n",
       "      <td>0.0656</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7356</td>\n",
       "      <td>0.8284</td>\n",
       "      <td>0.1049</td>\n",
       "      <td>30300</td>\n",
       "      <td>33888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>University of Alabama at Birmingham</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>570.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11383.0</td>\n",
       "      <td>0.5922</td>\n",
       "      <td>0.2600</td>\n",
       "      <td>0.0283</td>\n",
       "      <td>0.0518</td>\n",
       "      <td>0.0022</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0368</td>\n",
       "      <td>0.0179</td>\n",
       "      <td>0.0100</td>\n",
       "      <td>0.2607</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3460</td>\n",
       "      <td>0.5214</td>\n",
       "      <td>0.2422</td>\n",
       "      <td>39700</td>\n",
       "      <td>21941.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Amridge University</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>291.0</td>\n",
       "      <td>0.2990</td>\n",
       "      <td>0.4192</td>\n",
       "      <td>0.0069</td>\n",
       "      <td>0.0034</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.2715</td>\n",
       "      <td>0.4536</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6801</td>\n",
       "      <td>0.7795</td>\n",
       "      <td>0.8540</td>\n",
       "      <td>40100</td>\n",
       "      <td>23370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>University of Alabama in Huntsville</td>\n",
       "      <td>Huntsville</td>\n",
       "      <td>AL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>595.0</td>\n",
       "      <td>590.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5451.0</td>\n",
       "      <td>0.6988</td>\n",
       "      <td>0.1255</td>\n",
       "      <td>0.0382</td>\n",
       "      <td>0.0376</td>\n",
       "      <td>0.0143</td>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.0172</td>\n",
       "      <td>0.0332</td>\n",
       "      <td>0.0350</td>\n",
       "      <td>0.2146</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3072</td>\n",
       "      <td>0.4596</td>\n",
       "      <td>0.2640</td>\n",
       "      <td>45500</td>\n",
       "      <td>24097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Alabama State University</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>AL</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>425.0</td>\n",
       "      <td>430.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4811.0</td>\n",
       "      <td>0.0158</td>\n",
       "      <td>0.9208</td>\n",
       "      <td>0.0121</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>0.0010</td>\n",
       "      <td>0.0006</td>\n",
       "      <td>0.0098</td>\n",
       "      <td>0.0243</td>\n",
       "      <td>0.0137</td>\n",
       "      <td>0.0892</td>\n",
       "      <td>1</td>\n",
       "      <td>0.7347</td>\n",
       "      <td>0.7554</td>\n",
       "      <td>0.1270</td>\n",
       "      <td>26600</td>\n",
       "      <td>33118.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                INSTNM        CITY STABBR  HBCU  MENONLY  \\\n",
       "0             Alabama A & M University      Normal     AL   1.0      0.0   \n",
       "1  University of Alabama at Birmingham  Birmingham     AL   0.0      0.0   \n",
       "2                   Amridge University  Montgomery     AL   0.0      0.0   \n",
       "3  University of Alabama in Huntsville  Huntsville     AL   0.0      0.0   \n",
       "4             Alabama State University  Montgomery     AL   1.0      0.0   \n",
       "\n",
       "   WOMENONLY  RELAFFIL  SATVRMID  SATMTMID  DISTANCEONLY     UGDS  UGDS_WHITE  \\\n",
       "0        0.0         0     424.0     420.0           0.0   4206.0      0.0333   \n",
       "1        0.0         0     570.0     565.0           0.0  11383.0      0.5922   \n",
       "2        0.0         1       NaN       NaN           1.0    291.0      0.2990   \n",
       "3        0.0         0     595.0     590.0           0.0   5451.0      0.6988   \n",
       "4        0.0         0     425.0     430.0           0.0   4811.0      0.0158   \n",
       "\n",
       "   UGDS_BLACK  UGDS_HISP  UGDS_ASIAN  UGDS_AIAN  UGDS_NHPI  UGDS_2MOR  \\\n",
       "0      0.9353     0.0055      0.0019     0.0024     0.0019     0.0000   \n",
       "1      0.2600     0.0283      0.0518     0.0022     0.0007     0.0368   \n",
       "2      0.4192     0.0069      0.0034     0.0000     0.0000     0.0000   \n",
       "3      0.1255     0.0382      0.0376     0.0143     0.0002     0.0172   \n",
       "4      0.9208     0.0121      0.0019     0.0010     0.0006     0.0098   \n",
       "\n",
       "   UGDS_NRA  UGDS_UNKN  PPTUG_EF  CURROPER  PCTPELL  PCTFLOAN  UG25ABV  \\\n",
       "0    0.0059     0.0138    0.0656         1   0.7356    0.8284   0.1049   \n",
       "1    0.0179     0.0100    0.2607         1   0.3460    0.5214   0.2422   \n",
       "2    0.0000     0.2715    0.4536         1   0.6801    0.7795   0.8540   \n",
       "3    0.0332     0.0350    0.2146         1   0.3072    0.4596   0.2640   \n",
       "4    0.0243     0.0137    0.0892         1   0.7347    0.7554   0.1270   \n",
       "\n",
       "  MD_EARN_WNE_P10 GRAD_DEBT_MDN_SUPP  \n",
       "0           30300              33888  \n",
       "1           39700            21941.5  \n",
       "2           40100              23370  \n",
       "3           45500              24097  \n",
       "4           26600            33118.5  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college = pd.read_csv('data/college.csv')\n",
    "college.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7535, 27)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column Name</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Most Common Value</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>INSTNM</td>\n",
       "      <td>object</td>\n",
       "      <td>Institution Name</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Connecticut Center for Massage Therapy-Westport</td>\n",
       "      <td>7535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CITY</td>\n",
       "      <td>object</td>\n",
       "      <td>City Location</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>New York</td>\n",
       "      <td>2514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>STABBR</td>\n",
       "      <td>object</td>\n",
       "      <td>State Abbreviation</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CA</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HBCU</td>\n",
       "      <td>float64</td>\n",
       "      <td>Historically Black College or University</td>\n",
       "      <td>371</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MENONLY</td>\n",
       "      <td>float64</td>\n",
       "      <td>0/1 Men Only</td>\n",
       "      <td>371</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>WOMENONLY</td>\n",
       "      <td>float64</td>\n",
       "      <td>0/1 Women only</td>\n",
       "      <td>371</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RELAFFIL</td>\n",
       "      <td>int64</td>\n",
       "      <td>0/1 Religious Affiliation</td>\n",
       "      <td>0</td>\n",
       "      <td>0.19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SATVRMID</td>\n",
       "      <td>float64</td>\n",
       "      <td>SAT Verbal Median</td>\n",
       "      <td>6350</td>\n",
       "      <td>522.82</td>\n",
       "      <td>NaN</td>\n",
       "      <td>163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>SATMTMID</td>\n",
       "      <td>float64</td>\n",
       "      <td>SAT Math Median</td>\n",
       "      <td>6339</td>\n",
       "      <td>530.77</td>\n",
       "      <td>NaN</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>DISTANCEONLY</td>\n",
       "      <td>float64</td>\n",
       "      <td>Distance Education Only</td>\n",
       "      <td>371</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>UGDS</td>\n",
       "      <td>float64</td>\n",
       "      <td>Undergraduate Enrollment</td>\n",
       "      <td>661</td>\n",
       "      <td>2356.84</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>UGDS_WHITE</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad White</td>\n",
       "      <td>661</td>\n",
       "      <td>0.51</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>UGDS_BLACK</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad Black</td>\n",
       "      <td>661</td>\n",
       "      <td>0.19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>UGDS_HISP</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad Hispanic</td>\n",
       "      <td>661</td>\n",
       "      <td>0.16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>UGDS_ASIAN</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad Asian</td>\n",
       "      <td>661</td>\n",
       "      <td>0.03</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>UGDS_AIAN</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad American Indian/Alaskan Native</td>\n",
       "      <td>661</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>UGDS_NHPI</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad Native Hawaiian/Pacific Isla...</td>\n",
       "      <td>661</td>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>UGDS_2MOR</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad 2 or more races</td>\n",
       "      <td>661</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>UGDS_NRA</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad non-resident aliens</td>\n",
       "      <td>661</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>UGDS_UNKN</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Undergrad race unknown</td>\n",
       "      <td>661</td>\n",
       "      <td>0.05</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>PPTUG_EF</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Students part-time</td>\n",
       "      <td>682</td>\n",
       "      <td>0.23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>CURROPER</td>\n",
       "      <td>int64</td>\n",
       "      <td>0/1 Currently Operating</td>\n",
       "      <td>0</td>\n",
       "      <td>0.92</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>PCTPELL</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Students with Pell grant</td>\n",
       "      <td>686</td>\n",
       "      <td>0.53</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>PCTFLOAN</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Students with federal loan</td>\n",
       "      <td>686</td>\n",
       "      <td>0.52</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>UG25ABV</td>\n",
       "      <td>float64</td>\n",
       "      <td>Percent Students Older than 25</td>\n",
       "      <td>817</td>\n",
       "      <td>0.41</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>MD_EARN_WNE_P10</td>\n",
       "      <td>object</td>\n",
       "      <td>Median Earnings 10 years after enrollment</td>\n",
       "      <td>1122</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "      <td>598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>GRAD_DEBT_MDN_SUPP</td>\n",
       "      <td>object</td>\n",
       "      <td>Median debt of completers</td>\n",
       "      <td>32</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PrivacySuppressed</td>\n",
       "      <td>2038</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Column Name Data Type  \\\n",
       "0               INSTNM    object   \n",
       "1                 CITY    object   \n",
       "2               STABBR    object   \n",
       "3                 HBCU   float64   \n",
       "4              MENONLY   float64   \n",
       "5            WOMENONLY   float64   \n",
       "6             RELAFFIL     int64   \n",
       "7             SATVRMID   float64   \n",
       "8             SATMTMID   float64   \n",
       "9         DISTANCEONLY   float64   \n",
       "10                UGDS   float64   \n",
       "11          UGDS_WHITE   float64   \n",
       "12          UGDS_BLACK   float64   \n",
       "13           UGDS_HISP   float64   \n",
       "14          UGDS_ASIAN   float64   \n",
       "15           UGDS_AIAN   float64   \n",
       "16           UGDS_NHPI   float64   \n",
       "17           UGDS_2MOR   float64   \n",
       "18            UGDS_NRA   float64   \n",
       "19           UGDS_UNKN   float64   \n",
       "20            PPTUG_EF   float64   \n",
       "21            CURROPER     int64   \n",
       "22             PCTPELL   float64   \n",
       "23            PCTFLOAN   float64   \n",
       "24             UG25ABV   float64   \n",
       "25     MD_EARN_WNE_P10    object   \n",
       "26  GRAD_DEBT_MDN_SUPP    object   \n",
       "\n",
       "                                   Column Description  Missing Values  \\\n",
       "0                                    Institution Name               0   \n",
       "1                                       City Location               0   \n",
       "2                                  State Abbreviation               0   \n",
       "3            Historically Black College or University             371   \n",
       "4                                        0/1 Men Only             371   \n",
       "5                                      0/1 Women only             371   \n",
       "6                           0/1 Religious Affiliation               0   \n",
       "7                                   SAT Verbal Median            6350   \n",
       "8                                     SAT Math Median            6339   \n",
       "9                             Distance Education Only             371   \n",
       "10                           Undergraduate Enrollment             661   \n",
       "11                            Percent Undergrad White             661   \n",
       "12                            Percent Undergrad Black             661   \n",
       "13                         Percent Undergrad Hispanic             661   \n",
       "14                            Percent Undergrad Asian             661   \n",
       "15   Percent Undergrad American Indian/Alaskan Native             661   \n",
       "16  Percent Undergrad Native Hawaiian/Pacific Isla...             661   \n",
       "17                  Percent Undergrad 2 or more races             661   \n",
       "18              Percent Undergrad non-resident aliens             661   \n",
       "19                     Percent Undergrad race unknown             661   \n",
       "20                         Percent Students part-time             682   \n",
       "21                            0/1 Currently Operating               0   \n",
       "22                   Percent Students with Pell grant             686   \n",
       "23                 Percent Students with federal loan             686   \n",
       "24                     Percent Students Older than 25             817   \n",
       "25          Median Earnings 10 years after enrollment            1122   \n",
       "26                          Median debt of completers              32   \n",
       "\n",
       "       Mean                                Most Common Value  \\\n",
       "0       NaN  Connecticut Center for Massage Therapy-Westport   \n",
       "1       NaN                                         New York   \n",
       "2       NaN                                               CA   \n",
       "3      0.01                                              NaN   \n",
       "4      0.01                                              NaN   \n",
       "5      0.01                                              NaN   \n",
       "6      0.19                                              NaN   \n",
       "7    522.82                                              NaN   \n",
       "8    530.77                                              NaN   \n",
       "9      0.01                                              NaN   \n",
       "10  2356.84                                              NaN   \n",
       "11     0.51                                              NaN   \n",
       "12     0.19                                              NaN   \n",
       "13     0.16                                              NaN   \n",
       "14     0.03                                              NaN   \n",
       "15     0.01                                              NaN   \n",
       "16     0.00                                              NaN   \n",
       "17     0.02                                              NaN   \n",
       "18     0.02                                              NaN   \n",
       "19     0.05                                              NaN   \n",
       "20     0.23                                              NaN   \n",
       "21     0.92                                              NaN   \n",
       "22     0.53                                              NaN   \n",
       "23     0.52                                              NaN   \n",
       "24     0.41                                              NaN   \n",
       "25      NaN                                PrivacySuppressed   \n",
       "26      NaN                                PrivacySuppressed   \n",
       "\n",
       "    Number of Unique Values  \n",
       "0                      7535  \n",
       "1                      2514  \n",
       "2                        59  \n",
       "3                         2  \n",
       "4                         2  \n",
       "5                         2  \n",
       "6                         2  \n",
       "7                       163  \n",
       "8                       167  \n",
       "9                         2  \n",
       "10                     2932  \n",
       "11                     4397  \n",
       "12                     3242  \n",
       "13                     2809  \n",
       "14                     1254  \n",
       "15                      601  \n",
       "16                      363  \n",
       "17                      957  \n",
       "18                      920  \n",
       "19                     1517  \n",
       "20                     3420  \n",
       "21                        2  \n",
       "22                     4422  \n",
       "23                     4155  \n",
       "24                     4285  \n",
       "25                      598  \n",
       "26                     2038  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/college_decsription.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Employee Dataset\n",
    "\n",
    "### Brief Overview\n",
    "The city of Houston provides information on all its employees to the public. This is a random sample of 2,000 employees with a few of the more interesting columns. For more on [open Houston data visit their website](http://data.houstontx.gov/). Data was pulled in December, 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UNIQUE_ID</th>\n",
       "      <th>POSITION_TITLE</th>\n",
       "      <th>DEPARTMENT</th>\n",
       "      <th>BASE_SALARY</th>\n",
       "      <th>RACE</th>\n",
       "      <th>EMPLOYMENT_TYPE</th>\n",
       "      <th>GENDER</th>\n",
       "      <th>EMPLOYMENT_STATUS</th>\n",
       "      <th>HIRE_DATE</th>\n",
       "      <th>JOB_DATE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>ASSISTANT DIRECTOR (EX LVL)</td>\n",
       "      <td>Municipal Courts Department</td>\n",
       "      <td>121862.0</td>\n",
       "      <td>Hispanic/Latino</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>Female</td>\n",
       "      <td>Active</td>\n",
       "      <td>2006-06-12</td>\n",
       "      <td>2012-10-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>LIBRARY ASSISTANT</td>\n",
       "      <td>Library</td>\n",
       "      <td>26125.0</td>\n",
       "      <td>Hispanic/Latino</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>Female</td>\n",
       "      <td>Active</td>\n",
       "      <td>2000-07-19</td>\n",
       "      <td>2010-09-18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>POLICE OFFICER</td>\n",
       "      <td>Houston Police Department-HPD</td>\n",
       "      <td>45279.0</td>\n",
       "      <td>White</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>Male</td>\n",
       "      <td>Active</td>\n",
       "      <td>2015-02-03</td>\n",
       "      <td>2015-02-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>ENGINEER/OPERATOR</td>\n",
       "      <td>Houston Fire Department (HFD)</td>\n",
       "      <td>63166.0</td>\n",
       "      <td>White</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>Male</td>\n",
       "      <td>Active</td>\n",
       "      <td>1982-02-08</td>\n",
       "      <td>1991-05-25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>ELECTRICIAN</td>\n",
       "      <td>General Services Department</td>\n",
       "      <td>56347.0</td>\n",
       "      <td>White</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>Male</td>\n",
       "      <td>Active</td>\n",
       "      <td>1989-06-19</td>\n",
       "      <td>1994-10-22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UNIQUE_ID               POSITION_TITLE                     DEPARTMENT  \\\n",
       "0          0  ASSISTANT DIRECTOR (EX LVL)    Municipal Courts Department   \n",
       "1          1            LIBRARY ASSISTANT                        Library   \n",
       "2          2               POLICE OFFICER  Houston Police Department-HPD   \n",
       "3          3            ENGINEER/OPERATOR  Houston Fire Department (HFD)   \n",
       "4          4                  ELECTRICIAN    General Services Department   \n",
       "\n",
       "   BASE_SALARY             RACE EMPLOYMENT_TYPE  GENDER EMPLOYMENT_STATUS  \\\n",
       "0     121862.0  Hispanic/Latino       Full Time  Female            Active   \n",
       "1      26125.0  Hispanic/Latino       Full Time  Female            Active   \n",
       "2      45279.0            White       Full Time    Male            Active   \n",
       "3      63166.0            White       Full Time    Male            Active   \n",
       "4      56347.0            White       Full Time    Male            Active   \n",
       "\n",
       "    HIRE_DATE    JOB_DATE  \n",
       "0  2006-06-12  2012-10-13  \n",
       "1  2000-07-19  2010-09-18  \n",
       "2  2015-02-03  2015-02-03  \n",
       "3  1982-02-08  1991-05-25  \n",
       "4  1989-06-19  1994-10-22  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "employee = pd.read_csv('data/employee.csv')\n",
    "employee.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2000, 10)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "employee.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column Name</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Most Common Value</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UNIQUE_ID</td>\n",
       "      <td>int64</td>\n",
       "      <td>Uniquely identifies each employeee</td>\n",
       "      <td>0</td>\n",
       "      <td>999.50</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>POSITION_TITLE</td>\n",
       "      <td>object</td>\n",
       "      <td>Specific Position</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>SENIOR POLICE OFFICER</td>\n",
       "      <td>330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DEPARTMENT</td>\n",
       "      <td>object</td>\n",
       "      <td>Department</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Houston Police Department-HPD</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BASE_SALARY</td>\n",
       "      <td>float64</td>\n",
       "      <td>Base salary</td>\n",
       "      <td>114</td>\n",
       "      <td>55767.93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RACE</td>\n",
       "      <td>object</td>\n",
       "      <td>Race</td>\n",
       "      <td>35</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Black or African American</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>EMPLOYMENT_TYPE</td>\n",
       "      <td>object</td>\n",
       "      <td>Full Time/Part time/Temporary, etc...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>GENDER</td>\n",
       "      <td>object</td>\n",
       "      <td>Male/Female</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>EMPLOYMENT_STATUS</td>\n",
       "      <td>object</td>\n",
       "      <td>Active or Inactive</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Active</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>HIRE_DATE</td>\n",
       "      <td>object</td>\n",
       "      <td>Date Hired</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-03-28</td>\n",
       "      <td>999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>JOB_DATE</td>\n",
       "      <td>object</td>\n",
       "      <td>Date began latest position</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2002-01-05</td>\n",
       "      <td>947</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Column Name Data Type                     Column Description  \\\n",
       "0          UNIQUE_ID     int64     Uniquely identifies each employeee   \n",
       "1     POSITION_TITLE    object                      Specific Position   \n",
       "2         DEPARTMENT    object                             Department   \n",
       "3        BASE_SALARY   float64                            Base salary   \n",
       "4               RACE    object                                   Race   \n",
       "5    EMPLOYMENT_TYPE    object  Full Time/Part time/Temporary, etc...   \n",
       "6             GENDER    object                            Male/Female   \n",
       "7  EMPLOYMENT_STATUS    object                     Active or Inactive   \n",
       "8          HIRE_DATE    object                             Date Hired   \n",
       "9           JOB_DATE    object             Date began latest position   \n",
       "\n",
       "   Missing Values      Mean              Most Common Value  \\\n",
       "0               0    999.50                            NaN   \n",
       "1               0       NaN          SENIOR POLICE OFFICER   \n",
       "2               0       NaN  Houston Police Department-HPD   \n",
       "3             114  55767.93                            NaN   \n",
       "4              35       NaN      Black or African American   \n",
       "5               0       NaN                      Full Time   \n",
       "6               0       NaN                           Male   \n",
       "7               0       NaN                         Active   \n",
       "8               0       NaN                     2016-03-28   \n",
       "9               3       NaN                     2002-01-05   \n",
       "\n",
       "   Number of Unique Values  \n",
       "0                     2000  \n",
       "1                      330  \n",
       "2                       24  \n",
       "3                      791  \n",
       "4                        6  \n",
       "5                        5  \n",
       "6                        2  \n",
       "7                        2  \n",
       "8                      999  \n",
       "9                      947  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/employee_description.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Flights Dataset\n",
    "\n",
    "### Brief Overview\n",
    "A random sample of three percent of the US domestic flights originating from the ten busiest airports. Data is from the U.S. Department of Transportation's (DOT) Bureau of Transportation Statistics. [See here for more info](https://www.kaggle.com/usdot/flight-delays)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MONTH</th>\n",
       "      <th>DAY</th>\n",
       "      <th>WEEKDAY</th>\n",
       "      <th>AIRLINE</th>\n",
       "      <th>ORG_AIR</th>\n",
       "      <th>DEST_AIR</th>\n",
       "      <th>SCHED_DEP</th>\n",
       "      <th>DEP_DELAY</th>\n",
       "      <th>AIR_TIME</th>\n",
       "      <th>DIST</th>\n",
       "      <th>SCHED_ARR</th>\n",
       "      <th>ARR_DELAY</th>\n",
       "      <th>DIVERTED</th>\n",
       "      <th>CANCELLED</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>SLC</td>\n",
       "      <td>1625</td>\n",
       "      <td>58.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>590</td>\n",
       "      <td>1905</td>\n",
       "      <td>65.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>IAD</td>\n",
       "      <td>823</td>\n",
       "      <td>7.0</td>\n",
       "      <td>154.0</td>\n",
       "      <td>1452</td>\n",
       "      <td>1333</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>VPS</td>\n",
       "      <td>1305</td>\n",
       "      <td>36.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>641</td>\n",
       "      <td>1453</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>AA</td>\n",
       "      <td>DFW</td>\n",
       "      <td>DCA</td>\n",
       "      <td>1555</td>\n",
       "      <td>7.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>1192</td>\n",
       "      <td>1935</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>WN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>MCI</td>\n",
       "      <td>1720</td>\n",
       "      <td>48.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1363</td>\n",
       "      <td>2225</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MONTH  DAY  WEEKDAY AIRLINE ORG_AIR DEST_AIR  SCHED_DEP  DEP_DELAY  \\\n",
       "0      1    1        4      WN     LAX      SLC       1625       58.0   \n",
       "1      1    1        4      UA     DEN      IAD        823        7.0   \n",
       "2      1    1        4      MQ     DFW      VPS       1305       36.0   \n",
       "3      1    1        4      AA     DFW      DCA       1555        7.0   \n",
       "4      1    1        4      WN     LAX      MCI       1720       48.0   \n",
       "\n",
       "   AIR_TIME  DIST  SCHED_ARR  ARR_DELAY  DIVERTED  CANCELLED  \n",
       "0      94.0   590       1905       65.0         0          0  \n",
       "1     154.0  1452       1333      -13.0         0          0  \n",
       "2      85.0   641       1453       35.0         0          0  \n",
       "3     126.0  1192       1935       -7.0         0          0  \n",
       "4     166.0  1363       2225       39.0         0          0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights = pd.read_csv('data/flights.csv')\n",
    "flights.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(58492, 14)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column Name</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Most Common Value</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MONTH</td>\n",
       "      <td>int64</td>\n",
       "      <td>Month (1-12)</td>\n",
       "      <td>0</td>\n",
       "      <td>6.22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>DAY</td>\n",
       "      <td>int64</td>\n",
       "      <td>Day of the month</td>\n",
       "      <td>0</td>\n",
       "      <td>15.70</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>WEEKDAY</td>\n",
       "      <td>int64</td>\n",
       "      <td>Weekday 1 (Monday) - 7 (Sunday)</td>\n",
       "      <td>0</td>\n",
       "      <td>3.93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AIRLINE</td>\n",
       "      <td>object</td>\n",
       "      <td>One of 14 major air carriers</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DL</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ORG_AIR</td>\n",
       "      <td>object</td>\n",
       "      <td>Origin aiport code</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ATL</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>DEST_AIR</td>\n",
       "      <td>object</td>\n",
       "      <td>Destination airport code</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>LAX</td>\n",
       "      <td>271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SCHED_DEP</td>\n",
       "      <td>int64</td>\n",
       "      <td>Scheduled department time</td>\n",
       "      <td>0</td>\n",
       "      <td>1387.98</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>DEP_DELAY</td>\n",
       "      <td>float64</td>\n",
       "      <td>Minutes delayed past scheduled departure time</td>\n",
       "      <td>833</td>\n",
       "      <td>10.92</td>\n",
       "      <td>NaN</td>\n",
       "      <td>389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>AIR_TIME</td>\n",
       "      <td>float64</td>\n",
       "      <td>Minutes in air</td>\n",
       "      <td>1018</td>\n",
       "      <td>115.93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>DIST</td>\n",
       "      <td>int64</td>\n",
       "      <td>Flight distance in miles</td>\n",
       "      <td>0</td>\n",
       "      <td>872.90</td>\n",
       "      <td>NaN</td>\n",
       "      <td>850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SCHED_ARR</td>\n",
       "      <td>int64</td>\n",
       "      <td>Scheduled arrival time</td>\n",
       "      <td>0</td>\n",
       "      <td>1549.40</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>ARR_DELAY</td>\n",
       "      <td>float64</td>\n",
       "      <td>Minutes delayed past scheduled arrival time</td>\n",
       "      <td>1018</td>\n",
       "      <td>5.81</td>\n",
       "      <td>NaN</td>\n",
       "      <td>420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>DIVERTED</td>\n",
       "      <td>int64</td>\n",
       "      <td>Is the flight diverted (0/1)</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>CANCELLED</td>\n",
       "      <td>int64</td>\n",
       "      <td>Is the flight cancelled? (0/1)</td>\n",
       "      <td>0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Column Name Data Type                             Column Description  \\\n",
       "0        MONTH     int64                                   Month (1-12)   \n",
       "1          DAY     int64                               Day of the month   \n",
       "2      WEEKDAY     int64                Weekday 1 (Monday) - 7 (Sunday)   \n",
       "3      AIRLINE    object                   One of 14 major air carriers   \n",
       "4      ORG_AIR    object                             Origin aiport code   \n",
       "5     DEST_AIR    object                       Destination airport code   \n",
       "6    SCHED_DEP     int64                      Scheduled department time   \n",
       "7    DEP_DELAY   float64  Minutes delayed past scheduled departure time   \n",
       "8     AIR_TIME   float64                                 Minutes in air   \n",
       "9         DIST     int64                       Flight distance in miles   \n",
       "10   SCHED_ARR     int64                         Scheduled arrival time   \n",
       "11   ARR_DELAY   float64    Minutes delayed past scheduled arrival time   \n",
       "12    DIVERTED     int64                   Is the flight diverted (0/1)   \n",
       "13   CANCELLED     int64                 Is the flight cancelled? (0/1)   \n",
       "\n",
       "    Missing Values     Mean Most Common Value  Number of Unique Values  \n",
       "0                0     6.22               NaN                       11  \n",
       "1                0    15.70               NaN                       31  \n",
       "2                0     3.93               NaN                        7  \n",
       "3                0      NaN                DL                       14  \n",
       "4                0      NaN               ATL                       10  \n",
       "5                0      NaN               LAX                      271  \n",
       "6                0  1387.98               NaN                     1162  \n",
       "7              833    10.92               NaN                      389  \n",
       "8             1018   115.93               NaN                      453  \n",
       "9                0   872.90               NaN                      850  \n",
       "10               0  1549.40               NaN                     1295  \n",
       "11            1018     5.81               NaN                      420  \n",
       "12               0     0.00               NaN                        2  \n",
       "13               0     0.02               NaN                        2  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/flights_description.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Airline Codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IATA_CODE</th>\n",
       "      <th>AIRLINE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UA</td>\n",
       "      <td>United Air Lines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AA</td>\n",
       "      <td>American Airlines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>US</td>\n",
       "      <td>US Airways Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F9</td>\n",
       "      <td>Frontier Airlines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B6</td>\n",
       "      <td>JetBlue Airways</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>OO</td>\n",
       "      <td>Skywest Airlines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AS</td>\n",
       "      <td>Alaska Airlines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NK</td>\n",
       "      <td>Spirit Air Lines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>WN</td>\n",
       "      <td>Southwest Airlines Co.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>DL</td>\n",
       "      <td>Delta Air Lines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>EV</td>\n",
       "      <td>Atlantic Southeast Airlines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>HA</td>\n",
       "      <td>Hawaiian Airlines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>MQ</td>\n",
       "      <td>American Eagle Airlines Inc.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>VX</td>\n",
       "      <td>Virgin America</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   IATA_CODE                       AIRLINE\n",
       "0         UA         United Air Lines Inc.\n",
       "1         AA        American Airlines Inc.\n",
       "2         US               US Airways Inc.\n",
       "3         F9        Frontier Airlines Inc.\n",
       "4         B6               JetBlue Airways\n",
       "5         OO         Skywest Airlines Inc.\n",
       "6         AS          Alaska Airlines Inc.\n",
       "7         NK              Spirit Air Lines\n",
       "8         WN        Southwest Airlines Co.\n",
       "9         DL          Delta Air Lines Inc.\n",
       "10        EV   Atlantic Southeast Airlines\n",
       "11        HA        Hawaiian Airlines Inc.\n",
       "12        MQ  American Eagle Airlines Inc.\n",
       "13        VX                Virgin America"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/airlines.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Airport codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IATA_CODE</th>\n",
       "      <th>AIRPORT</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STATE</th>\n",
       "      <th>COUNTRY</th>\n",
       "      <th>LATITUDE</th>\n",
       "      <th>LONGITUDE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ABE</td>\n",
       "      <td>Lehigh Valley International Airport</td>\n",
       "      <td>Allentown</td>\n",
       "      <td>PA</td>\n",
       "      <td>USA</td>\n",
       "      <td>40.65236</td>\n",
       "      <td>-75.44040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ABI</td>\n",
       "      <td>Abilene Regional Airport</td>\n",
       "      <td>Abilene</td>\n",
       "      <td>TX</td>\n",
       "      <td>USA</td>\n",
       "      <td>32.41132</td>\n",
       "      <td>-99.68190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ABQ</td>\n",
       "      <td>Albuquerque International Sunport</td>\n",
       "      <td>Albuquerque</td>\n",
       "      <td>NM</td>\n",
       "      <td>USA</td>\n",
       "      <td>35.04022</td>\n",
       "      <td>-106.60919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ABR</td>\n",
       "      <td>Aberdeen Regional Airport</td>\n",
       "      <td>Aberdeen</td>\n",
       "      <td>SD</td>\n",
       "      <td>USA</td>\n",
       "      <td>45.44906</td>\n",
       "      <td>-98.42183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ABY</td>\n",
       "      <td>Southwest Georgia Regional Airport</td>\n",
       "      <td>Albany</td>\n",
       "      <td>GA</td>\n",
       "      <td>USA</td>\n",
       "      <td>31.53552</td>\n",
       "      <td>-84.19447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  IATA_CODE                              AIRPORT         CITY STATE COUNTRY  \\\n",
       "0       ABE  Lehigh Valley International Airport    Allentown    PA     USA   \n",
       "1       ABI             Abilene Regional Airport      Abilene    TX     USA   \n",
       "2       ABQ    Albuquerque International Sunport  Albuquerque    NM     USA   \n",
       "3       ABR            Aberdeen Regional Airport     Aberdeen    SD     USA   \n",
       "4       ABY   Southwest Georgia Regional Airport       Albany    GA     USA   \n",
       "\n",
       "   LATITUDE  LONGITUDE  \n",
       "0  40.65236  -75.44040  \n",
       "1  32.41132  -99.68190  \n",
       "2  35.04022 -106.60919  \n",
       "3  45.44906  -98.42183  \n",
       "4  31.53552  -84.19447  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/airports.csv').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chinook Database\n",
    "\n",
    "### Brief Overview\n",
    "This is a sample database of a music store provided by SQLite with 11 tables. The table description image is an excellent way to get familiar with the database. [Visit the sqlite website](http://www.sqlitetutorial.net/sqlite-sample-database/) for more detail.\n",
    "![data/descriptions/data/descriptions/ch09_05_erd.png](data/descriptions/ch09_05_erd.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Crime Dataset\n",
    "\n",
    "### Brief Overview\n",
    "All crime and traffic accidents for the city of Denver from January to September of 2017. This dataset is stored in special binary form called *hdf5*. Pandas uses the PyTables library to help read the data into a DataFrame. [Read the documentation](http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5) for more info on hdf5 formatted data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OFFENSE_TYPE_ID</th>\n",
       "      <th>OFFENSE_CATEGORY_ID</th>\n",
       "      <th>REPORTED_DATE</th>\n",
       "      <th>GEO_LON</th>\n",
       "      <th>GEO_LAT</th>\n",
       "      <th>NEIGHBORHOOD_ID</th>\n",
       "      <th>IS_CRIME</th>\n",
       "      <th>IS_TRAFFIC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>traffic-accident-dui-duid</td>\n",
       "      <td>traffic-accident</td>\n",
       "      <td>2014-06-29 02:01:00</td>\n",
       "      <td>-105.000149</td>\n",
       "      <td>39.745753</td>\n",
       "      <td>cbd</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>vehicular-eluding-no-chase</td>\n",
       "      <td>all-other-crimes</td>\n",
       "      <td>2014-06-29 01:54:00</td>\n",
       "      <td>-104.884660</td>\n",
       "      <td>39.738702</td>\n",
       "      <td>east-colfax</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>disturbing-the-peace</td>\n",
       "      <td>public-disorder</td>\n",
       "      <td>2014-06-29 02:00:00</td>\n",
       "      <td>-105.020719</td>\n",
       "      <td>39.706674</td>\n",
       "      <td>athmar-park</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>curfew</td>\n",
       "      <td>public-disorder</td>\n",
       "      <td>2014-06-29 02:18:00</td>\n",
       "      <td>-105.001552</td>\n",
       "      <td>39.769505</td>\n",
       "      <td>sunnyside</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>aggravated-assault</td>\n",
       "      <td>aggravated-assault</td>\n",
       "      <td>2014-06-29 04:17:00</td>\n",
       "      <td>-105.018557</td>\n",
       "      <td>39.679229</td>\n",
       "      <td>college-view-south-platte</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              OFFENSE_TYPE_ID OFFENSE_CATEGORY_ID       REPORTED_DATE  \\\n",
       "0   traffic-accident-dui-duid    traffic-accident 2014-06-29 02:01:00   \n",
       "1  vehicular-eluding-no-chase    all-other-crimes 2014-06-29 01:54:00   \n",
       "2        disturbing-the-peace     public-disorder 2014-06-29 02:00:00   \n",
       "3                      curfew     public-disorder 2014-06-29 02:18:00   \n",
       "4          aggravated-assault  aggravated-assault 2014-06-29 04:17:00   \n",
       "\n",
       "      GEO_LON    GEO_LAT            NEIGHBORHOOD_ID  IS_CRIME  IS_TRAFFIC  \n",
       "0 -105.000149  39.745753                        cbd         0           1  \n",
       "1 -104.884660  39.738702                east-colfax         1           0  \n",
       "2 -105.020719  39.706674                athmar-park         1           0  \n",
       "3 -105.001552  39.769505                  sunnyside         1           0  \n",
       "4 -105.018557  39.679229  college-view-south-platte         1           0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crime = pd.read_hdf('data/crime.h5')\n",
    "crime.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(460911, 8)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crime.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column Name</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OFFENSE_TYPE_ID</td>\n",
       "      <td>category</td>\n",
       "      <td>Offenes Type</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OFFENSE_CATEGORY_ID</td>\n",
       "      <td>category</td>\n",
       "      <td>Offense Category Name</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>REPORTED_DATE</td>\n",
       "      <td>datetime64[ns]</td>\n",
       "      <td>Reported Date</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>390969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GEO_LON</td>\n",
       "      <td>float64</td>\n",
       "      <td>Longitude</td>\n",
       "      <td>3615</td>\n",
       "      <td>-104.95</td>\n",
       "      <td>89196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GEO_LAT</td>\n",
       "      <td>float64</td>\n",
       "      <td>Latitude</td>\n",
       "      <td>3615</td>\n",
       "      <td>39.73</td>\n",
       "      <td>89012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NEIGHBORHOOD_ID</td>\n",
       "      <td>category</td>\n",
       "      <td>Neighborhood Name</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>IS_CRIME</td>\n",
       "      <td>int64</td>\n",
       "      <td>Is it a crime? (0/1)</td>\n",
       "      <td>0</td>\n",
       "      <td>0.73</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>IS_TRAFFIC</td>\n",
       "      <td>int64</td>\n",
       "      <td>Is it a traffic accident (0/1)</td>\n",
       "      <td>0</td>\n",
       "      <td>0.27</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Column Name       Data Type              Column Description  \\\n",
       "0      OFFENSE_TYPE_ID        category                    Offenes Type   \n",
       "1  OFFENSE_CATEGORY_ID        category           Offense Category Name   \n",
       "2        REPORTED_DATE  datetime64[ns]                   Reported Date   \n",
       "3              GEO_LON         float64                       Longitude   \n",
       "4              GEO_LAT         float64                        Latitude   \n",
       "5      NEIGHBORHOOD_ID        category               Neighborhood Name   \n",
       "6             IS_CRIME           int64            Is it a crime? (0/1)   \n",
       "7           IS_TRAFFIC           int64  Is it a traffic accident (0/1)   \n",
       "\n",
       "   Missing Values    Mean  Number of Unique Values  \n",
       "0               0     NaN                      196  \n",
       "1               0     NaN                       15  \n",
       "2               0     NaN                   390969  \n",
       "3            3615 -104.95                    89196  \n",
       "4            3615   39.73                    89012  \n",
       "5               0     NaN                       78  \n",
       "6               0    0.73                        2  \n",
       "7               0    0.27                        2  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/crime_description.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Meetup Groups Dataset\n",
    "\n",
    "### Brief Overview\n",
    "Data was collected through the [meetup.com API](https://www.meetup.com/meetup_api/) on five Houston-area data science meetup groups. Each row represents a member joining a particular group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>join_date</th>\n",
       "      <th>group</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-11-18 02:41:29</td>\n",
       "      <td>houston machine learning</td>\n",
       "      <td>Houston</td>\n",
       "      <td>TX</td>\n",
       "      <td>us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-05-09 14:16:37</td>\n",
       "      <td>houston machine learning</td>\n",
       "      <td>Houston</td>\n",
       "      <td>TX</td>\n",
       "      <td>us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-12-30 02:34:16</td>\n",
       "      <td>houston machine learning</td>\n",
       "      <td>Houston</td>\n",
       "      <td>TX</td>\n",
       "      <td>us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-07-18 00:48:17</td>\n",
       "      <td>houston machine learning</td>\n",
       "      <td>Houston</td>\n",
       "      <td>TX</td>\n",
       "      <td>us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-05-25 12:58:16</td>\n",
       "      <td>houston machine learning</td>\n",
       "      <td>Houston</td>\n",
       "      <td>TX</td>\n",
       "      <td>us</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             join_date                     group     city state country\n",
       "0  2016-11-18 02:41:29  houston machine learning  Houston    TX      us\n",
       "1  2017-05-09 14:16:37  houston machine learning  Houston    TX      us\n",
       "2  2016-12-30 02:34:16  houston machine learning  Houston    TX      us\n",
       "3  2016-07-18 00:48:17  houston machine learning  Houston    TX      us\n",
       "4  2017-05-25 12:58:16  houston machine learning  Houston    TX      us"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meetup = pd.read_csv('data/meetup_groups.csv')\n",
    "meetup.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7671, 5)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meetup.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column Name</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Most Common Value</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>join_date</td>\n",
       "      <td>object</td>\n",
       "      <td>Date Joined</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-07-10 16:33:53</td>\n",
       "      <td>7631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>group</td>\n",
       "      <td>object</td>\n",
       "      <td>Group Name</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>houston data science</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>city</td>\n",
       "      <td>object</td>\n",
       "      <td>City</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Houston</td>\n",
       "      <td>316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>state</td>\n",
       "      <td>object</td>\n",
       "      <td>State</td>\n",
       "      <td>144</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TX</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>country</td>\n",
       "      <td>object</td>\n",
       "      <td>Country</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>us</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Column Name Data Type Column Description  Missing Values  Mean  \\\n",
       "0   join_date    object        Date Joined               0   NaN   \n",
       "1       group    object         Group Name               0   NaN   \n",
       "2        city    object               City               0   NaN   \n",
       "3       state    object              State             144   NaN   \n",
       "4     country    object            Country               0   NaN   \n",
       "\n",
       "      Most Common Value  Number of Unique Values  \n",
       "0   2016-07-10 16:33:53                     7631  \n",
       "1  houston data science                        5  \n",
       "2               Houston                      316  \n",
       "3                    TX                       46  \n",
       "4                    us                       43  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/meetup_description.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diamonds Dataset\n",
    "\n",
    "### Brief Overview\n",
    "Quality, size and price of nearly 54,000 diamonds scraped from the [Diamond Search Engine](http://www.diamondse.info/) by Hadley Wickham. [Visit blue nile](https://www.bluenile.com/ca/education/diamonds?track=SideNav) for a beginners guide to diamonds. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat      cut color clarity  depth  table  price     x     y     z\n",
       "0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43\n",
       "1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31\n",
       "2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31\n",
       "3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63\n",
       "4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds = pd.read_csv('data/diamonds.csv')\n",
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(53940, 10)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column Name</th>\n",
       "      <th>Data Type</th>\n",
       "      <th>Column Description</th>\n",
       "      <th>Missing Values</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Most Common Value</th>\n",
       "      <th>Number of Unique Values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>carat</td>\n",
       "      <td>float64</td>\n",
       "      <td>Size of diamond in carats</td>\n",
       "      <td>0</td>\n",
       "      <td>0.80</td>\n",
       "      <td>NaN</td>\n",
       "      <td>273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cut</td>\n",
       "      <td>object</td>\n",
       "      <td>Quality of the cut (Fair, Good, Very Good, Pre...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>color</td>\n",
       "      <td>object</td>\n",
       "      <td>Color ranging from worst to best - J through D</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>G</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>clarity</td>\n",
       "      <td>object</td>\n",
       "      <td>Measurement of imperfections ranging from wors...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>SI1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>depth</td>\n",
       "      <td>float64</td>\n",
       "      <td>Total depth percentage = z / mean(x, y) = 2 * ...</td>\n",
       "      <td>0</td>\n",
       "      <td>61.75</td>\n",
       "      <td>NaN</td>\n",
       "      <td>184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>table</td>\n",
       "      <td>float64</td>\n",
       "      <td>Qidth of top of diamond relative to widest point</td>\n",
       "      <td>0</td>\n",
       "      <td>57.46</td>\n",
       "      <td>NaN</td>\n",
       "      <td>127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>price</td>\n",
       "      <td>int64</td>\n",
       "      <td>Price</td>\n",
       "      <td>0</td>\n",
       "      <td>3932.80</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>x</td>\n",
       "      <td>float64</td>\n",
       "      <td>Length in mm</td>\n",
       "      <td>0</td>\n",
       "      <td>5.73</td>\n",
       "      <td>NaN</td>\n",
       "      <td>554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>y</td>\n",
       "      <td>float64</td>\n",
       "      <td>Width in mm</td>\n",
       "      <td>0</td>\n",
       "      <td>5.73</td>\n",
       "      <td>NaN</td>\n",
       "      <td>552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>z</td>\n",
       "      <td>float64</td>\n",
       "      <td>Depth in mm</td>\n",
       "      <td>0</td>\n",
       "      <td>3.54</td>\n",
       "      <td>NaN</td>\n",
       "      <td>375</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Column Name Data Type                                 Column Description  \\\n",
       "0       carat   float64                          Size of diamond in carats   \n",
       "1         cut    object  Quality of the cut (Fair, Good, Very Good, Pre...   \n",
       "2       color    object     Color ranging from worst to best - J through D   \n",
       "3     clarity    object  Measurement of imperfections ranging from wors...   \n",
       "4       depth   float64  Total depth percentage = z / mean(x, y) = 2 * ...   \n",
       "5       table   float64  Qidth of top of diamond relative to widest point    \n",
       "6       price     int64                                              Price   \n",
       "7           x   float64                                       Length in mm   \n",
       "8           y   float64                                        Width in mm   \n",
       "9           z   float64                                        Depth in mm   \n",
       "\n",
       "   Missing Values     Mean Most Common Value  Number of Unique Values  \n",
       "0               0     0.80               NaN                      273  \n",
       "1               0      NaN             Ideal                        5  \n",
       "2               0      NaN                 G                        7  \n",
       "3               0      NaN               SI1                        8  \n",
       "4               0    61.75               NaN                      184  \n",
       "5               0    57.46               NaN                      127  \n",
       "6               0  3932.80               NaN                    11602  \n",
       "7               0     5.73               NaN                      554  \n",
       "8               0     5.73               NaN                      552  \n",
       "9               0     3.54               NaN                      375  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('data/descriptions/diamonds_description.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}