milestone2/Milestone2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Milestone 2 - CS109B Project Group 29\n",
    "**\"Predicting Movie Genres\"**<br>\n",
    "**CS109B (Spring 2017) Final Project Group 29**<br>\n",
    "Calvin J Chiew, Tim Hagmann, Ji Hua<br>\n",
    "TF: Rashmi Banthia<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deliverables\n",
    "\n",
    "- Discussion about the imbalanced nature of the data and how you want to address it\n",
    "- Description of your data\n",
    "- What does your choice of Y look like?\n",
    "- Which features do you choose for X and why?\n",
    "- How do you sample your data, how many samples, and why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import random\n",
    "from bs4 import BeautifulSoup\n",
    "import urllib\n",
    "from urllib import urlopen\n",
    "from urllib import urlretrieve\n",
    "import threading\n",
    "from threading import Timer\n",
    "from time import time, sleep\n",
    "import io\n",
    "import imdb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scraping from TMDb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we used the TMDb API to scrape information on movies that have `tmdb_id` of 1 through 160,000 (see Milestone 1 for code). About 60,000 of these `tmdb_id`'s were empty, returning about 100,000 actual records.\n",
    "\n",
    "Given that `tmdb_id`'s are not assigned in chronological order or in any particular pattern, this 100,000 movies represent a **random** sample of all movies listed on TMDb (estimated to be around 300,000). We chose 100,000 as we believe it is a sufficiently large sample size to develop our traditional ML and deep learning models.\n",
    "\n",
    "We dropped about 30,000 records that were missing poster paths and `imdb_id`'s, leaving us with about 70,000 movies.\n",
    "\n",
    "You can download the raw data file we gathered on our GitHub repository here: https://github.com/greenore/deep-learning-project/blob/master/milestone2/tmdb_movie_info_100k.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>title</th>\n",
       "      <th>plot</th>\n",
       "      <th>poster_path</th>\n",
       "      <th>release_date</th>\n",
       "      <th>budget</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>popularity</th>\n",
       "      <th>user_rating</th>\n",
       "      <th>production_company</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "      <td>Ariel</td>\n",
       "      <td>Taisto Kasurinen is a Finnish coal miner whose...</td>\n",
       "      <td>/gZCJZOn4l0Zj5hAxsMbxoS6CL0u.jpg</td>\n",
       "      <td>1988-10-21</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>69</td>\n",
       "      <td>0.623849</td>\n",
       "      <td>6.8</td>\n",
       "      <td>Villealfa Filmproduction Oy,Finnish Film Found...</td>\n",
       "      <td>Drama,Crime</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "      <td>Shadows in Paradise</td>\n",
       "      <td>An episode in the life of Nikander, a garbage ...</td>\n",
       "      <td>/7ad4iku8cYBuB08g9yAU7tHJik5.jpg</td>\n",
       "      <td>1986-10-16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76</td>\n",
       "      <td>0.288572</td>\n",
       "      <td>6.8</td>\n",
       "      <td>Villealfa Filmproduction Oy</td>\n",
       "      <td>Drama,Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "      <td>Four Rooms</td>\n",
       "      <td>It's Ted the Bellhop's first night on the job....</td>\n",
       "      <td>/eQs5hh9rxrk1m4xHsIz1w11Ngqb.jpg</td>\n",
       "      <td>1995-12-25</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>4300000.0</td>\n",
       "      <td>98</td>\n",
       "      <td>1.768986</td>\n",
       "      <td>6.4</td>\n",
       "      <td>Miramax Films,A Band Apart</td>\n",
       "      <td>Crime,Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "      <td>Judgment Night</td>\n",
       "      <td>While racing to a boxing match, Frank, Mike, J...</td>\n",
       "      <td>/lNXmgUrP6h1nD53gkFh4WDzT6RZ.jpg</td>\n",
       "      <td>1993-10-15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12136938.0</td>\n",
       "      <td>110</td>\n",
       "      <td>0.676269</td>\n",
       "      <td>6.6</td>\n",
       "      <td>Largo Entertainment,JVC Entertainment Networks</td>\n",
       "      <td>Action,Thriller,Crime</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "      <td>Life in Loops (A Megacities RMX)</td>\n",
       "      <td>Timo Novotny labels his new project an experim...</td>\n",
       "      <td>/8YyIjOAxwzD3fZMdmJrfiApod4l.jpg</td>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80</td>\n",
       "      <td>0.125477</td>\n",
       "      <td>5.2</td>\n",
       "      <td>inLoops</td>\n",
       "      <td>Documentary</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id                             title  \\\n",
       "0       2  0094675                             Ariel   \n",
       "1       3  0092149               Shadows in Paradise   \n",
       "2       5  0113101                        Four Rooms   \n",
       "3       6  0107286                    Judgment Night   \n",
       "4       8  0825671  Life in Loops (A Megacities RMX)   \n",
       "\n",
       "                                                plot  \\\n",
       "0  Taisto Kasurinen is a Finnish coal miner whose...   \n",
       "1  An episode in the life of Nikander, a garbage ...   \n",
       "2  It's Ted the Bellhop's first night on the job....   \n",
       "3  While racing to a boxing match, Frank, Mike, J...   \n",
       "4  Timo Novotny labels his new project an experim...   \n",
       "\n",
       "                        poster_path release_date     budget     revenue  \\\n",
       "0  /gZCJZOn4l0Zj5hAxsMbxoS6CL0u.jpg   1988-10-21        0.0         0.0   \n",
       "1  /7ad4iku8cYBuB08g9yAU7tHJik5.jpg   1986-10-16        0.0         0.0   \n",
       "2  /eQs5hh9rxrk1m4xHsIz1w11Ngqb.jpg   1995-12-25  4000000.0   4300000.0   \n",
       "3  /lNXmgUrP6h1nD53gkFh4WDzT6RZ.jpg   1993-10-15        0.0  12136938.0   \n",
       "4  /8YyIjOAxwzD3fZMdmJrfiApod4l.jpg   2006-01-01    42000.0         0.0   \n",
       "\n",
       "  runtime popularity  user_rating  \\\n",
       "0      69   0.623849          6.8   \n",
       "1      76   0.288572          6.8   \n",
       "2      98   1.768986          6.4   \n",
       "3     110   0.676269          6.6   \n",
       "4      80   0.125477          5.2   \n",
       "\n",
       "                                  production_company                 genres  \n",
       "0  Villealfa Filmproduction Oy,Finnish Film Found...            Drama,Crime  \n",
       "1                        Villealfa Filmproduction Oy           Drama,Comedy  \n",
       "2                         Miramax Films,A Band Apart           Crime,Comedy  \n",
       "3     Largo Entertainment,JVC Entertainment Networks  Action,Thriller,Crime  \n",
       "4                                            inLoops            Documentary  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view example\n",
    "tmdb = pd.read_table('tmdb_movie_info_100k.txt')\n",
    "tmdb.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scraping from IMDb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also extracted additional metadata from IMDb for this list of 70,000 movies and matched them to the TMDb records based on `imdb_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id\n",
       "0       2  0094675\n",
       "1       3  0092149\n",
       "2       5  0113101\n",
       "3       6  0107286\n",
       "4       8  0825671"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_df = pd.read_table('imdb_id_100k.txt', sep='\\t')\n",
    "imdb_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69970, 2)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "movie = IMDb()\n",
    "target = io.open(\"imdb_movie_info_100k.txt\", 'w',encoding=\"utf8\")\n",
    "line=u\"tmdb_id\\timdb_id\\tcast\\tdirector\\twriter\\tUSA_rating\"\n",
    "target.write(line+'\\n')\n",
    "\n",
    "for i in range(1,imdb_df.shape[0]):\n",
    "    tmdb_id=imdb_df.iloc[i,0]\n",
    "    imdb_id=imdb_df.iloc[i,1]\n",
    "    data=movie.get_movie(imdb_id)\n",
    "    line=u\"\"\n",
    "    line=line+str(tmdb_id).decode('utf8') #tmdb_id\n",
    "    line=line+u\"\\t\"+str(imdb_id).decode('utf8')#imdb_id\n",
    "\n",
    "    ## get cast\n",
    "    cast_string=u\"\"\n",
    "    try:\n",
    "        cast_num=min(len(data['cast']),5)\n",
    "        for c in range(cast_num):\n",
    "            if c==0:\n",
    "                cast_string=str(data['cast'][c].personID).decode('utf8')\n",
    "            else:\n",
    "                cast_string=cast_string+\",\"+str(data['cast'][c].personID).decode('utf8')\n",
    "    except KeyError:\n",
    "         cast_string=u\"\"\n",
    "    line=line+u\"\\t\"+cast_string\n",
    "\n",
    "    #get director\n",
    "    director_string=u\"\"\n",
    "    try:\n",
    "        director_num=min(len(data['director']),2)\n",
    "        for c in range(director_num):\n",
    "            if c==0:\n",
    "                director_string=str(data['director'][c].personID).decode('utf8')\n",
    "            else:\n",
    "                director_string=director_string+\",\"+str(data['director'][c].personID).decode('utf8')\n",
    "    except KeyError:\n",
    "        director_string=u\"\"\n",
    "    line=line+u\"\\t\"+director_string\n",
    "\n",
    "    # get writer\n",
    "    writer_string=u\"\"\n",
    "    try:\n",
    "        writer_num=min(len(data['writer']),2)\n",
    "        for c in range(writer_num):\n",
    "            if c==0:\n",
    "                writer_string=str(data['writer'][c].personID).decode('utf8')\n",
    "            else:\n",
    "                writer_string=writer_string+\",\"+str(data['writer'][c].personID).decode('utf8')\n",
    "    except KeyError:\n",
    "        writer_string=u\"\"\n",
    "    line=line+u\"\\t\"+writer_string\n",
    "\n",
    "    ## get US certificate\n",
    "    try:\n",
    "        matching = [s for s in data['certificates'] if \"USA\" in s]\n",
    "        if matching!=[]:\n",
    "            usa_rating=matching[0].split(\":\")[1]\n",
    "        else:\n",
    "            usa_rating=u\"\"\n",
    "    except KeyError:\n",
    "        usa_rating=u\"\"\n",
    "    line=line+u\"\\t\"+usa_rating\n",
    "    target.write(line+'\\n')\n",
    "    target.flush()\n",
    "    print i\n",
    "        \n",
    "        \n",
    "target.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the code above, we were able to gather additional metadata (cast, director, writer, picture rating) from IMDb. Information on cast, directors and writers are in the format of personID (7-digit numbers) used by IMDb.\n",
    "\n",
    "You can download the raw data file we gathered on our GitHub repository here: https://github.com/greenore/deep-learning-project/blob/master/milestone2/imdb_movie_info_100k.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>cast</th>\n",
       "      <th>director</th>\n",
       "      <th>writer</th>\n",
       "      <th>USA_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>94675</td>\n",
       "      <td>0656999,0352099,0671231,0383990,0656998</td>\n",
       "      <td>0442454</td>\n",
       "      <td>0442454</td>\n",
       "      <td>Not Rated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>92149</td>\n",
       "      <td>0671231,0653669,0475411,0631741,0477871</td>\n",
       "      <td>0442454</td>\n",
       "      <td>0442454</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>113101</td>\n",
       "      <td>0205423,0004866,0000420,0000187,0001746</td>\n",
       "      <td>0025978,0734319</td>\n",
       "      <td>0025978,0734319</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>107286</td>\n",
       "      <td>0000389,0000421,0001459,0001151,0005315</td>\n",
       "      <td>0394280</td>\n",
       "      <td>0171474,0192342</td>\n",
       "      <td>R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>825671</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1119183</td>\n",
       "      <td>0322198,1119183</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id  imdb_id                                     cast         director  \\\n",
       "0        2    94675  0656999,0352099,0671231,0383990,0656998          0442454   \n",
       "1        3    92149  0671231,0653669,0475411,0631741,0477871          0442454   \n",
       "2        5   113101  0205423,0004866,0000420,0000187,0001746  0025978,0734319   \n",
       "3        6   107286  0000389,0000421,0001459,0001151,0005315          0394280   \n",
       "4        8   825671                                      NaN          1119183   \n",
       "\n",
       "            writer USA_rating  \n",
       "0          0442454  Not Rated  \n",
       "1          0442454        NaN  \n",
       "2  0025978,0734319          R  \n",
       "3  0171474,0192342          R  \n",
       "4  0322198,1119183        NaN  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view example\n",
    "imdb = pd.read_table('imdb_movie_info_100k.txt')\n",
    "imdb.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>USA_rating</th>\n",
       "      <th>cast1</th>\n",
       "      <th>cast2</th>\n",
       "      <th>cast3</th>\n",
       "      <th>cast4</th>\n",
       "      <th>cast5</th>\n",
       "      <th>director1</th>\n",
       "      <th>director2</th>\n",
       "      <th>writer1</th>\n",
       "      <th>writer2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Not Rated</td>\n",
       "      <td>0656999</td>\n",
       "      <td>0352099</td>\n",
       "      <td>0671231</td>\n",
       "      <td>0383990</td>\n",
       "      <td>0656998</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0671231</td>\n",
       "      <td>0653669</td>\n",
       "      <td>0475411</td>\n",
       "      <td>0631741</td>\n",
       "      <td>0477871</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>R</td>\n",
       "      <td>0205423</td>\n",
       "      <td>0004866</td>\n",
       "      <td>0000420</td>\n",
       "      <td>0000187</td>\n",
       "      <td>0001746</td>\n",
       "      <td>0025978</td>\n",
       "      <td>0734319</td>\n",
       "      <td>0025978</td>\n",
       "      <td>0734319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>R</td>\n",
       "      <td>0000389</td>\n",
       "      <td>0000421</td>\n",
       "      <td>0001459</td>\n",
       "      <td>0001151</td>\n",
       "      <td>0005315</td>\n",
       "      <td>0394280</td>\n",
       "      <td>nan</td>\n",
       "      <td>0171474</td>\n",
       "      <td>0192342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>1119183</td>\n",
       "      <td>nan</td>\n",
       "      <td>0322198</td>\n",
       "      <td>1119183</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id USA_rating    cast1    cast2    cast3    cast4    cast5 director1  \\\n",
       "0        2  Not Rated  0656999  0352099  0671231  0383990  0656998   0442454   \n",
       "1        3        NaN  0671231  0653669  0475411  0631741  0477871   0442454   \n",
       "2        5          R  0205423  0004866  0000420  0000187  0001746   0025978   \n",
       "3        6          R  0000389  0000421  0001459  0001151  0005315   0394280   \n",
       "4        8        NaN      nan      nan      nan      nan      nan   1119183   \n",
       "\n",
       "  director2  writer1  writer2  \n",
       "0       nan  0442454      nan  \n",
       "1       nan  0442454      nan  \n",
       "2   0734319  0025978  0734319  \n",
       "3       nan  0171474  0192342  \n",
       "4       nan  0322198  1119183  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split cast into columns\n",
    "cast = imdb['cast'].str.split(',').apply(pd.Series).astype(str)\n",
    "cast.rename(columns={0:'cast1', 1:'cast2', 2:'cast3', 3:'cast4', 4:'cast5'}, inplace=True)\n",
    "\n",
    "# split director into columns\n",
    "directors = imdb['director'].str.split(',').apply(pd.Series).astype(str)\n",
    "directors.rename(columns={0:'director1', 1:'director2'}, inplace=True)\n",
    "\n",
    "# split writers into columns\n",
    "writers = imdb['writer'].str.split(',').apply(pd.Series).astype(str)\n",
    "writers.rename(columns={0:'writer1', 1:'writer2'}, inplace=True)\n",
    "\n",
    "# concatenate relevant columns\n",
    "imdb_info = pd.concat([imdb[['tmdb_id', 'USA_rating']], cast, directors, writers], axis=1)\n",
    "\n",
    "# show head\n",
    "imdb_info.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outcome Variables (Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The outcome variable is the movie genre, which is classified into 19 categories by TMDb (namely Adventure, Fantasy, Animation, Drama, Horror, Action, Comedy, History, Western, Thriller, Crime, Documentary, Science Fiction, Mystery, Music, Romance, Family, War and Foreign). We decided against regrouping the genres because even if we had chosen to do so, we would have only collapsed a few groups (eg. Action-Adventure, Thriller-Crime-Mystery) based on our genre pair analysis in earlier EDA (see Milestone 1). We did not feel that this small reduction in genre categories would make a significant difference in the larger picture. To simplify the problem, we also did not use the genre classifications on IMDb.\n",
    "\n",
    "In addition, each movie may fall into multiple genres, ie. this is a multi-label problem. To address this issue, we coded the outcome into 19 columns of binary (1/0) flags, one for each of the 19 TMDb genres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>Animation</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Horror</th>\n",
       "      <th>Action</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>History</th>\n",
       "      <th>Western</th>\n",
       "      <th>Thriller</th>\n",
       "      <th>Crime</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>Science Fiction</th>\n",
       "      <th>Mystery</th>\n",
       "      <th>Music</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Family</th>\n",
       "      <th>War</th>\n",
       "      <th>Foreign</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id  Adventure  Fantasy  Animation  Drama  Horror  Action  Comedy  \\\n",
       "0        2          0        0          0      1       0       0       0   \n",
       "1        3          0        0          0      1       0       0       1   \n",
       "2        5          0        0          0      0       0       0       1   \n",
       "3        6          0        0          0      0       0       1       0   \n",
       "4        8          0        0          0      0       0       0       0   \n",
       "\n",
       "   History  Western  Thriller  Crime  Documentary  Science Fiction  Mystery  \\\n",
       "0        0        0         0      1            0                0        0   \n",
       "1        0        0         0      0            0                0        0   \n",
       "2        0        0         0      1            0                0        0   \n",
       "3        0        0         1      1            0                0        0   \n",
       "4        0        0         0      0            1                0        0   \n",
       "\n",
       "   Music  Romance  Family  War  Foreign  \n",
       "0      0        0       0    0        0  \n",
       "1      0        0       0    0        0  \n",
       "2      0        0       0    0        0  \n",
       "3      0        0       0    0        0  \n",
       "4      0        0       0    0        0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load pre-processed file\n",
    "genres = pd.read_csv('genres_70k_new.txt')\n",
    "\n",
    "# drop TVMovie as 0 observations\n",
    "genres = genres.drop('TVMovie', 1)\n",
    "\n",
    "# show example\n",
    "genres.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the 19 outcome variables (one for each genre), we created 1 more outcome variable -- the **number of genre tags**. As we will be training separate classification models for each genre, we would like another model that can predict the number of genres to be assigned to each movie. Using this predicted number, we can then assign tags to each test movie with predicted probabilities of every genre. Here, we generate this 20th outcome variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>Animation</th>\n",
       "      <th>Drama</th>\n",
       "      <th>Horror</th>\n",
       "      <th>Action</th>\n",
       "      <th>Comedy</th>\n",
       "      <th>History</th>\n",
       "      <th>Western</th>\n",
       "      <th>...</th>\n",
       "      <th>Crime</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>Science Fiction</th>\n",
       "      <th>Mystery</th>\n",
       "      <th>Music</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Family</th>\n",
       "      <th>War</th>\n",
       "      <th>Foreign</th>\n",
       "      <th>num_genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id  Adventure  Fantasy  Animation  Drama  Horror  Action  Comedy  \\\n",
       "0        2          0        0          0      1       0       0       0   \n",
       "1        3          0        0          0      1       0       0       1   \n",
       "2        5          0        0          0      0       0       0       1   \n",
       "3        6          0        0          0      0       0       1       0   \n",
       "4        8          0        0          0      0       0       0       0   \n",
       "\n",
       "   History  Western     ...      Crime  Documentary  Science Fiction  Mystery  \\\n",
       "0        0        0     ...          1            0                0        0   \n",
       "1        0        0     ...          0            0                0        0   \n",
       "2        0        0     ...          1            0                0        0   \n",
       "3        0        0     ...          1            0                0        0   \n",
       "4        0        0     ...          0            1                0        0   \n",
       "\n",
       "   Music  Romance  Family  War  Foreign  num_genres  \n",
       "0      0        0       0    0        0           2  \n",
       "1      0        0       0    0        0           2  \n",
       "2      0        0       0    0        0           2  \n",
       "3      0        0       0    0        0           3  \n",
       "4      0        0       0    0        0           1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generate new Y variable -- number of genres\n",
    "genres['num_genres'] = genres.ix[:, 1:19].sum(axis=1)\n",
    "genres.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69969, 21)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genres.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "genres.to_csv('outcome_vars.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictor Variables (X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two sets of data, one with movie posters as features, and the other with movie metadata. For the former, we downloaded w500 resolution movie posters from TMDb (see below). For the latter, we have chosen to include the following features:\n",
    "\n",
    "*Extracted from TMDb*\n",
    "- Title\n",
    "- Plot summary\n",
    "- Production company\n",
    "- Release date\n",
    "- Runtime (mins)\n",
    "- Budget (USD)\n",
    "- Revenue (USD)\n",
    "- Popularity\n",
    "- Average user rating (on scale of 1-10)\n",
    "\n",
    "*Extracted from IMDb*\n",
    "- Director (first 2 names listed)\n",
    "- Writer (first 2 names listed)\n",
    "- Cast (first 5 names listed)\n",
    "- Picture rating (US MPAA categories eg. PG-13, R-21)\n",
    "\n",
    "These features were chosen based on our prior beliefs about what are potential predictors of movie genre (see Milestone 1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading Posters from TMDb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the code below, we downloaded w500 resolution movie posters in `.jpg` format from TMDb. Since the total file size is >1GB, we have not added them to GitHub repository. We have included a few posters as examples below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>poster_path</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>/gZCJZOn4l0Zj5hAxsMbxoS6CL0u.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>/7ad4iku8cYBuB08g9yAU7tHJik5.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>/eQs5hh9rxrk1m4xHsIz1w11Ngqb.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>/lNXmgUrP6h1nD53gkFh4WDzT6RZ.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>/8YyIjOAxwzD3fZMdmJrfiApod4l.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id                       poster_path\n",
       "0        2  /gZCJZOn4l0Zj5hAxsMbxoS6CL0u.jpg\n",
       "1        3  /7ad4iku8cYBuB08g9yAU7tHJik5.jpg\n",
       "2        5  /eQs5hh9rxrk1m4xHsIz1w11Ngqb.jpg\n",
       "3        6  /lNXmgUrP6h1nD53gkFh4WDzT6RZ.jpg\n",
       "4        8  /8YyIjOAxwzD3fZMdmJrfiApod4l.jpg"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poster_df = pd.read_table('poster_path_100k.txt', sep='\\t')\n",
    "poster_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "base=\"https://image.tmdb.org/t/p/w500\"\n",
    "filebase=\"G:/data/clean_data/poster/\"\n",
    "count=0\n",
    "for i in range(64208,poster_df.shape[0]):\n",
    "    poster_path=poster_df.iloc[i,1]\n",
    "    filename=str(poster_df.iloc[i,0])+\".jpg\"\n",
    "    urlretrieve(base+poster_path, filebase+filename)\n",
    "    count+=1\n",
    "    if count%100==0:\n",
    "        print count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Poster Examples |\n",
    "|---|---|---|---|---|---|---|\n",
    "| <img src=\"26.jpg\" width=\"150\"> | <img src=\"33.jpg\" width=\"150\"> | <img src=\"35.jpg\" width=\"150\"> | <img src=\"38.jpg\" width=\"150\"> | <img src=\"58.jpg\" width=\"150\"> | <img src=\"59.jpg\" width=\"150\"> | <img src=\"62.jpg\" width=\"150\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting Posters to Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we convert the poster `.jpg` images to arrays of pixel features. All the posters had a width of 500 pixels. Most of them had a height of 750 pixels. To ensure a standard array size for each poster, we only considered the first 750 pixels (and ignored the rest) if the poster was taller than 750 pixels. If the poster was shorter than 750 pixels, we filled it with black pixels at the bottom until a height of 750 pixels.\n",
    "\n",
    "Each poster had 500 x 750 = 375,000 pixels, and for each pixel, we extracted 3 colour (RGB) values. Therefore, each poster had a total of 375,000 x 3 = 1,125,000 features at the end.\n",
    "\n",
    "To limit output file sizes, we processed the posters in batches of 500 posters (total 140 batches).\n",
    "\n",
    "Since the file size of each output matrix is very large, we did not add them to GitHub repository. We show an example of one matrix (first 500 posters) below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy import misc\n",
    "labels=['tmdb_id']\n",
    "color=['r','g','b']\n",
    "for c in color:\n",
    "    for i in range(500*750): ## resolution maxium is 500*750\n",
    "        label=c+str(i)\n",
    "        labels.append(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "target = open(\"G:/data/clean_data/poster_matrix.txt\", 'w')\n",
    "for label in labels:\n",
    "    if label!=labels[-1]:\n",
    "        target.write(label+\"\\t\")\n",
    "    else:\n",
    "        target.write(label+\"\\n\")\n",
    "target.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_group=100 # 250 poster as a file\n",
    "total_groups=poster_df.shape[0]/n_group+1 ## total group number\n",
    "directory=\"G:/data/clean_data/poster/\"\n",
    "n_color=3\n",
    "width=500\n",
    "height=750\n",
    "for i in range(total_groups):\n",
    "    matrix=np.zeros((n_group,width*height*n_color+1),dtype=int)\n",
    "    for j in range(n_group):\n",
    "        tmdb_id=poster_df.iloc[i*500+250+j,0]        \n",
    "        filename=directory+str(tmdb_id)+\".jpg\"        \n",
    "        try: \n",
    "            arr = misc.imread(filename)\n",
    "        except IOError:\n",
    "            continue\n",
    "        matrix[j,0]=tmdb_id \n",
    "        ## check if oversize and set maximum height as 750\n",
    "        trans_height=min(height,arr.shape[0])\n",
    "        ## check if image is colorful\n",
    "        if len(arr.shape)==3:\n",
    "            for c in range(n_color):\n",
    "                for h in range(trans_height):\n",
    "                    ind_start=c*height*width+1+h*width\n",
    "                    ind_end=ind_start+width\n",
    "                    matrix[j,ind_start:ind_end]=arr[h,:,c]\n",
    "        else:\n",
    "            for c in range(n_color):\n",
    "                for h in range(trans_height):\n",
    "                    ind_start=c*height*width+1+h*width\n",
    "                    ind_end=ind_start+width\n",
    "                    matrix[j,ind_start:ind_end]=arr[h,:]\n",
    "    target_name=\"G:/data/clean_data/poster_matrices/poster_matrix_\"+str(i)+\".csv\"\n",
    "    with open(target_name, 'wb') as f:\n",
    "        np.savetxt(f,matrix,fmt='%d',delimiter=\",\")\n",
    "    print i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# view example matrix\n",
    "matrix = np.loadtxt('C:/Users/Calvin/Desktop/Shared/milestone2b/poster_matrix_0.csv', dtype=int, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>1124991</th>\n",
       "      <th>1124992</th>\n",
       "      <th>1124993</th>\n",
       "      <th>1124994</th>\n",
       "      <th>1124995</th>\n",
       "      <th>1124996</th>\n",
       "      <th>1124997</th>\n",
       "      <th>1124998</th>\n",
       "      <th>1124999</th>\n",
       "      <th>1125000</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>306</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>307</td>\n",
       "      <td>230</td>\n",
       "      <td>181</td>\n",
       "      <td>235</td>\n",
       "      <td>251</td>\n",
       "      <td>187</td>\n",
       "      <td>222</td>\n",
       "      <td>248</td>\n",
       "      <td>241</td>\n",
       "      <td>218</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>44</td>\n",
       "      <td>12</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>308</td>\n",
       "      <td>46</td>\n",
       "      <td>58</td>\n",
       "      <td>85</td>\n",
       "      <td>97</td>\n",
       "      <td>87</td>\n",
       "      <td>65</td>\n",
       "      <td>32</td>\n",
       "      <td>129</td>\n",
       "      <td>254</td>\n",
       "      <td>...</td>\n",
       "      <td>253</td>\n",
       "      <td>252</td>\n",
       "      <td>252</td>\n",
       "      <td>252</td>\n",
       "      <td>252</td>\n",
       "      <td>252</td>\n",
       "      <td>254</td>\n",
       "      <td>254</td>\n",
       "      <td>254</td>\n",
       "      <td>254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>309</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>...</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "      <td>255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>310</td>\n",
       "      <td>78</td>\n",
       "      <td>78</td>\n",
       "      <td>78</td>\n",
       "      <td>78</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>80</td>\n",
       "      <td>82</td>\n",
       "      <td>76</td>\n",
       "      <td>...</td>\n",
       "      <td>147</td>\n",
       "      <td>147</td>\n",
       "      <td>147</td>\n",
       "      <td>147</td>\n",
       "      <td>148</td>\n",
       "      <td>149</td>\n",
       "      <td>151</td>\n",
       "      <td>150</td>\n",
       "      <td>150</td>\n",
       "      <td>151</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1125001 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id    1    2    3    4    5    6    7    8    9   ...     1124991  \\\n",
       "0      306    0    0    0    0    0    0    0    0    0   ...           0   \n",
       "1      307  230  181  235  251  187  222  248  241  218   ...           5   \n",
       "2      308   46   58   85   97   87   65   32  129  254   ...         253   \n",
       "3      309  255  255  255  255  255  255  255  255  255   ...         255   \n",
       "4      310   78   78   78   78   77   78   80   82   76   ...         147   \n",
       "\n",
       "   1124992  1124993  1124994  1124995  1124996  1124997  1124998  1124999  \\\n",
       "0        0        0        0        0        0        0        0        0   \n",
       "1       19       44       12       16       16        4        4       25   \n",
       "2      252      252      252      252      252      254      254      254   \n",
       "3      255      255      255      255      255      255      255      255   \n",
       "4      147      147      147      148      149      151      150      150   \n",
       "\n",
       "   1125000  \n",
       "0        0  \n",
       "1       15  \n",
       "2      254  \n",
       "3      255  \n",
       "4      151  \n",
       "\n",
       "[5 rows x 1125001 columns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix_df = pd.DataFrame(matrix)\n",
    "matrix_df.rename(columns={0:'tmdb_id'}, inplace=True)\n",
    "matrix_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Column 0 is the `tmdb_id`. Columns 1 through 375,000 are values for red channel, the next 375,000 columns are values for green channel, and the last 375,000 values are for blue channel.\n",
    "\n",
    "If the poster is grayscale (non-colour), we set the three RGB channel values to be equal.\n",
    "\n",
    "Prior to modelling in milestone 3, we will apply PCA to the matrix for dimensionality reduction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag-of-Words Analysis (for Title & Plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we use bag-of-words analysis to convert the title and plot summaries into features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>title_plot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "      <td>Ariel Taisto Kasurinen is a Finnish coal miner...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "      <td>Shadows in Paradise An episode in the life of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "      <td>Four Rooms It's Ted the Bellhop's first night ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "      <td>Judgment Night While racing to a boxing match,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "      <td>Life in Loops (A Megacities RMX) Timo Novotny ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id                                         title_plot\n",
       "0       2  0094675  Ariel Taisto Kasurinen is a Finnish coal miner...\n",
       "1       3  0092149  Shadows in Paradise An episode in the life of ...\n",
       "2       5  0113101  Four Rooms It's Ted the Bellhop's first night ...\n",
       "3       6  0107286  Judgment Night While racing to a boxing match,...\n",
       "4       8  0825671  Life in Loops (A Megacities RMX) Timo Novotny ..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load csv file\n",
    "raw = pd.read_table('tmdb_movie_info_100k.txt')\n",
    "\n",
    "# concatenate title and plot together \n",
    "raw['title_plot'] = raw['title'].astype(str) + ' ' + raw['plot'].astype(str)\n",
    "\n",
    "# drop all columns except title and plot\n",
    "raw = raw[['tmdb_id', 'imdb_id', 'title_plot']]\n",
    "\n",
    "# view first few rows\n",
    "raw.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "389"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create bag-of-words matrix\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "# ignore stop words and only consider words that make up at least 1% of the corpus\n",
    "vectorizer = TfidfVectorizer(stop_words='english', min_df=0.01)\n",
    "corpus = raw['title_plot'].values\n",
    "X = vectorizer.fit_transform(corpus)\n",
    "vocab = vectorizer.get_feature_names()\n",
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'accident', u'action', u'adventure', u'affair', u'age', u'agent', u'america', u'american', u'amp', u'army', u'arrives', u'art', u'artist', u'attempt', u'attempts', u'away', u'baby', u'bad', u'band', u'based', u'battle', u'beautiful', u'begin', u'begins', u'best', u'big', u'black', u'blood', u'body', u'book', u'boss', u'boy', u'boyfriend', u'boys', u'break', u'bring', u'brings', u'british', u'brother', u'brothers', u'business', u'called', u'car', u'career', u'case', u'caught', u'century', u'chance', u'change', u'characters', u'child', u'childhood', u'children', u'city', u'class', u'classic', u'close', u'college', u'come', u'comedy', u'comes', u'coming', u'company', u'concert', u'control', u'cop', u'country', u'couple', u'course', u'crew', u'crime', u'criminal', u'dangerous', u'dark', u'daughter', u'david', u'day', u'days', u'dead', u'deadly', u'deal', u'death', u'decide', u'decides', u'despite', u'detective', u'die', u'dies', u'different', u'directed', u'director', u'discover', u'discovers', u'doctor', u'documentary', u'does', u'doesn', u'don', u'dr', u'drama', u'dream', u'dreams', u'drug', u'early', u'earth', u'end', u'ends', u'escape', u'events', u'eventually', u'evil', u'ex', u'face', u'fall', u'falls', u'family', u'famous', u'far', u'father', u'female', u'fight', u'film', u'films', u'final', u'finally', u'finds', u'following', u'follows', u'footage', u'force', u'forced', u'forces', u'form', u'free', u'french', u'friend', u'friends', u'friendship', u'future', u'game', u'gang', u'german', u'gets', u'getting', u'girl', u'girlfriend', u'girls', u'gives', u'goes', u'going', u'good', u'government', u'great', u'group', u'guy', u'hand', u'hands', u'happy', u'hard', u'having', u'head', u'heart', u'help', u'hero', u'high', u'history', u'hit', u'home', u'horror', u'house', u'human', u'husband', u'ii', u'including', u'inside', u'instead', u'involved', u'island', u'japanese', u'job', u'john', u'join', u'journey', u'just', u'kids', u'kill', u'killed', u'killer', u'killing', u'king', u'know', u'known', u'la', u'lady', u'land', u'late', u'later', u'law', u'lead', u'leader', u'leads', u'learn', u'learns', u'leave', u'leaves', u'left', u'life', u'like', u'little', u'live', u'lives', u'living', u'local', u'london', u'long', u'look', u'looking', u'los', u'lost', u'love', u'lover', u'make', u'makes', u'making', u'man', u'marriage', u'married', u'marry', u'master', u'meet', u'meets', u'members', u'men', u'middle', u'military', u'mind', u'missing', u'mission', u'modern', u'money', u'mother', u'moves', u'movie', u'mr', u'murder', u'murdered', u'music', u'mysterious', u'named', u'nan', u'new', u'night', u'novel', u'officer', u'old', u'order', u'overview', u'owner', u'parents', u'paris', u'party', u'past', u'people', u'perfect', u'personal', u'place', u'plan', u'plans', u'play', u'plays', u'plot', u'police', u'political', u'popular', u'power', u'powerful', u'prison', u'private', u'problems', u'quickly', u'race', u'real', u'really', u'red', u'relationship', u'released', u'rescue', u'return', u'returns', u'revenge', u'rich', u'right', u'road', u'rock', u'romance', u'romantic', u'run', u'runs', u'save', u'school', u'search', u'second', u'secret', u'seen', u'sees', u'self', u'sent', u'series', u'set', u'sets', u'sex', u'sexual', u'short', u'shot', u'shows', u'singer', u'sister', u'small', u'society', u'son', u'soon', u'south', u'special', u'stage', u'stand', u'star', u'stars', u'start', u'starts', u'state', u'stay', u'stop', u'stories', u'story', u'strange', u'street', u'student', u'students', u'successful', u'suddenly', u'summer', u'taken', u'takes', u'taking', u'tale', u'teacher', u'team', u'tells', u'things', u'time', u'tour', u'town', u'tries', u'trip', u'trouble', u'true', u'truth', u'try', u'trying', u'turn', u'turns', u'tv', u'use', u'using', u'video', u'village', u'visit', u'want', u'wants', u'war', u'way', u'wealthy', u'west', u'white', u'wife', u'wild', u'win', u'woman', u'women', u'work', u'working', u'works', u'world', u'writer', u'written', u'year', u'years', u'york', u'young']\n"
     ]
    }
   ],
   "source": [
    "print(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 389 words in the vocabulary, as shown above. We view the final weighted frequency matrix which was generated using Term Frequency times Inverse Document Frequency (tf-idf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>word_accident</th>\n",
       "      <th>word_action</th>\n",
       "      <th>word_adventure</th>\n",
       "      <th>word_affair</th>\n",
       "      <th>word_age</th>\n",
       "      <th>word_agent</th>\n",
       "      <th>word_america</th>\n",
       "      <th>word_american</th>\n",
       "      <th>word_amp</th>\n",
       "      <th>...</th>\n",
       "      <th>word_work</th>\n",
       "      <th>word_working</th>\n",
       "      <th>word_works</th>\n",
       "      <th>word_world</th>\n",
       "      <th>word_writer</th>\n",
       "      <th>word_written</th>\n",
       "      <th>word_year</th>\n",
       "      <th>word_years</th>\n",
       "      <th>word_york</th>\n",
       "      <th>word_young</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.665059</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.394097</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.184482</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.119018</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.180166</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.165786</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 390 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  word_accident  word_action  word_adventure  word_affair  word_age  \\\n",
       "0       2            0.0          0.0        0.000000     0.000000       0.0   \n",
       "1       3            0.0          0.0        0.000000     0.665059       0.0   \n",
       "2       5            0.0          0.0        0.000000     0.000000       0.0   \n",
       "3       6            0.0          0.0        0.000000     0.000000       0.0   \n",
       "4       8            0.0          0.0        0.184482     0.000000       0.0   \n",
       "\n",
       "   word_agent  word_america  word_american  word_amp     ...      word_work  \\\n",
       "0         0.0           0.0            0.0  0.000000     ...            0.0   \n",
       "1         0.0           0.0            0.0  0.000000     ...            0.0   \n",
       "2         0.0           0.0            0.0  0.000000     ...            0.0   \n",
       "3         0.0           0.0            0.0  0.394097     ...            0.0   \n",
       "4         0.0           0.0            0.0  0.000000     ...            0.0   \n",
       "\n",
       "   word_working  word_works  word_world  word_writer  word_written  word_year  \\\n",
       "0           0.0         0.0    0.000000          0.0      0.000000        0.0   \n",
       "1           0.0         0.0    0.000000          0.0      0.000000        0.0   \n",
       "2           0.0         0.0    0.000000          0.0      0.000000        0.0   \n",
       "3           0.0         0.0    0.000000          0.0      0.000000        0.0   \n",
       "4           0.0         0.0    0.119018          0.0      0.180166        0.0   \n",
       "\n",
       "   word_years  word_york  word_young  \n",
       "0         0.0   0.000000         0.0  \n",
       "1         0.0   0.000000         0.0  \n",
       "2         0.0   0.000000         0.0  \n",
       "3         0.0   0.000000         0.0  \n",
       "4         0.0   0.165786         0.0  \n",
       "\n",
       "[5 rows x 390 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bagofwords = pd.DataFrame(X.toarray(), columns=vocab)\n",
    "bagofwords = bagofwords.add_prefix('word_')\n",
    "bagofwords = pd.concat([raw[['tmdb_id']], bagofwords], axis=1)\n",
    "bagofwords.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bagofwords.to_csv('bagofwords_vars.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Data Cleanup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will clean up the remaining predictor variables, such as production company. As there are too many different production companies (which will result in too many dummy variables), we will consider only the top 20 production companies with the most films, and regroup the other companies into \"Others\".\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"tmdb_movie_info_100k.txt\", sep=\"\\t\")\n",
    "\n",
    "company_dict={}\n",
    "for i in range(data.shape[0]):\n",
    "    com_line=str(data.iloc[i,-2])\n",
    "    words=com_line.split(\",\")\n",
    "    for word in words:\n",
    "        if company_dict.get(word,None)!=None:\n",
    "            company_dict[word]+=1\n",
    "        else:\n",
    "            company_dict[word]=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('nan', 26979)\n",
      "('Warner Bros.', 1342)\n",
      "('Metro-Goldwyn-Mayer (MGM)', 1307)\n",
      "('Paramount Pictures', 1012)\n",
      "('Universal Pictures', 868)\n",
      "('Twentieth Century Fox Film Corporation', 798)\n",
      "('Columbia Pictures Corporation', 667)\n",
      "('RKO Radio Pictures', 540)\n",
      "('Columbia Pictures', 479)\n",
      "('Canal+', 431)\n",
      "('Walt Disney Productions', 423)\n",
      "('United Artists', 308)\n",
      "('Mosfilm', 280)\n",
      "('Walt Disney Pictures', 272)\n",
      "('Shaw Brothers', 264)\n",
      "('New Line Cinema', 257)\n",
      "(' The', 226)\n",
      "('Touchstone Pictures', 215)\n",
      "('Toho Company', 211)\n",
      "(' Inc.', 201)\n",
      "('TriStar Pictures', 192)\n",
      "('British Broadcasting Corporation (BBC)', 188)\n",
      "('Miramax Films', 173)\n",
      "('Universal International Pictures (UI)', 165)\n",
      "('20th Century Fox', 153)\n"
     ]
    }
   ],
   "source": [
    "import operator\n",
    "sorted_x = sorted(company_dict.items(), key=operator.itemgetter(1))\n",
    "sorted_x.reverse()\n",
    "count=0\n",
    "for key in sorted_x:\n",
    "    print key\n",
    "    count+=1\n",
    "    if count>=25:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pre-process the production company variable into 20 columns of binary (1/0) flags, one for each of the top 20 production companies (see code in R below). Movies that are produced by companies other than these top 20 will have \"0\" for all indicator variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "total_companies<-c(\"Warner Bros\",'Metro-Goldwyn-Mayer MGM','Paramount Pictures','Universal Pictures','Twentieth Century Fox Film Corporation','Columbia Pictures Corporation','RKO Radio Pictures','Columbia Pictures','Canal+','Walt Disney Productions','United Artists','Mosfilm','Walt Disney Pictures','Shaw Brothers','New Line Cinema','Touchstone Pictures','Toho Company','TriStar Pictures','British Broadcasting Corporation BBC','Miramax Films')\n",
    "company_ids<-seq(2,length(total_companies)+1)\n",
    "names(company_ids)<-total_companies\n",
    "company<-as.data.frame(setNames(replicate(length(total_companies)+1,numeric(0), simplify = F), seq(1,length(total_companies))))\n",
    "for (i in 1:dim(data)[1]){\n",
    "  print(i)\n",
    "  line<-rep(0,length(total_companies)+1)\n",
    "  line[1]<-as.numeric(data[i,1])\n",
    "  company_line<-as.character(data[i,12])\n",
    "  if (company_line==\"\"){\n",
    "    next\n",
    "  }\n",
    "    \n",
    "  company_words<-strsplit(company_line,\",\",useBytes=T)[[1]]\n",
    "  for (j in 1:length(company_words)){\n",
    "    w<-company_words[j]\n",
    "    if (w==\"\"){\n",
    "      next\n",
    "    }\n",
    "    if (w %in% total_companies){\n",
    "      line[company_ids[[w]]]<-1\n",
    "    }\n",
    "  }\n",
    "  company<-rbind(company,line)\n",
    "}\n",
    "\n",
    "colnames(company)<-c('tmdb_id',total_companies)\n",
    "write.csv(company,\"G:/data/clean_data/companies_100k.txt\",row.names = F,quote = F)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>Warner Bros</th>\n",
       "      <th>Metro-Goldwyn-Mayer MGM</th>\n",
       "      <th>Paramount Pictures</th>\n",
       "      <th>Universal Pictures</th>\n",
       "      <th>Twentieth Century Fox Film Corporation</th>\n",
       "      <th>Columbia Pictures Corporation</th>\n",
       "      <th>RKO Radio Pictures</th>\n",
       "      <th>Columbia Pictures</th>\n",
       "      <th>Canal+</th>\n",
       "      <th>...</th>\n",
       "      <th>United Artists</th>\n",
       "      <th>Mosfilm</th>\n",
       "      <th>Walt Disney Pictures</th>\n",
       "      <th>Shaw Brothers</th>\n",
       "      <th>New Line Cinema</th>\n",
       "      <th>Touchstone Pictures</th>\n",
       "      <th>Toho Company</th>\n",
       "      <th>TriStar Pictures</th>\n",
       "      <th>British Broadcasting Corporation BBC</th>\n",
       "      <th>Miramax Films</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   tmdb_id  Warner Bros  Metro-Goldwyn-Mayer MGM  Paramount Pictures  \\\n",
       "0        2            0                        0                   0   \n",
       "1        3            0                        0                   0   \n",
       "2        5            0                        0                   0   \n",
       "3        6            0                        0                   0   \n",
       "4        8            0                        0                   0   \n",
       "\n",
       "   Universal Pictures  Twentieth Century Fox Film Corporation  \\\n",
       "0                   0                                       0   \n",
       "1                   0                                       0   \n",
       "2                   0                                       0   \n",
       "3                   0                                       0   \n",
       "4                   0                                       0   \n",
       "\n",
       "   Columbia Pictures Corporation  RKO Radio Pictures  Columbia Pictures  \\\n",
       "0                              0                   0                  0   \n",
       "1                              0                   0                  0   \n",
       "2                              0                   0                  0   \n",
       "3                              0                   0                  0   \n",
       "4                              0                   0                  0   \n",
       "\n",
       "   Canal+      ...        United Artists  Mosfilm  Walt Disney Pictures  \\\n",
       "0       0      ...                     0        0                     0   \n",
       "1       0      ...                     0        0                     0   \n",
       "2       0      ...                     0        0                     0   \n",
       "3       0      ...                     0        0                     0   \n",
       "4       0      ...                     0        0                     0   \n",
       "\n",
       "   Shaw Brothers  New Line Cinema  Touchstone Pictures  Toho Company  \\\n",
       "0              0                0                    0             0   \n",
       "1              0                0                    0             0   \n",
       "2              0                0                    0             0   \n",
       "3              0                0                    0             0   \n",
       "4              0                0                    0             0   \n",
       "\n",
       "   TriStar Pictures  British Broadcasting Corporation BBC  Miramax Films  \n",
       "0                 0                                     0              0  \n",
       "1                 0                                     0              0  \n",
       "2                 0                                     0              1  \n",
       "3                 0                                     0              0  \n",
       "4                 0                                     0              0  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "companies = pd.read_csv('companies_100k.txt')\n",
    "companies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Final Dataframe\n",
    "\n",
    "Finally, we take a look at the final dataframe, which has a total of 447 variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# prepare tmdb dataframe\n",
    "tmdb = tmdb.drop(['title', 'plot', 'poster_path', 'production_company', 'genres'], axis=1)\n",
    "tmdb['tmdb_id'] = tmdb['tmdb_id'].apply(pd.to_numeric, errors='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>release_date</th>\n",
       "      <th>budget</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>popularity</th>\n",
       "      <th>user_rating</th>\n",
       "      <th>USA_rating</th>\n",
       "      <th>cast1</th>\n",
       "      <th>cast2</th>\n",
       "      <th>cast3</th>\n",
       "      <th>cast4</th>\n",
       "      <th>cast5</th>\n",
       "      <th>director1</th>\n",
       "      <th>director2</th>\n",
       "      <th>writer1</th>\n",
       "      <th>writer2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "      <td>1988-10-21</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>69</td>\n",
       "      <td>0.623849</td>\n",
       "      <td>6.8</td>\n",
       "      <td>Not Rated</td>\n",
       "      <td>0656999</td>\n",
       "      <td>0352099</td>\n",
       "      <td>0671231</td>\n",
       "      <td>0383990</td>\n",
       "      <td>0656998</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "      <td>1986-10-16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76</td>\n",
       "      <td>0.288572</td>\n",
       "      <td>6.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0671231</td>\n",
       "      <td>0653669</td>\n",
       "      <td>0475411</td>\n",
       "      <td>0631741</td>\n",
       "      <td>0477871</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "      <td>0442454</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "      <td>1995-12-25</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>4300000.0</td>\n",
       "      <td>98</td>\n",
       "      <td>1.768986</td>\n",
       "      <td>6.4</td>\n",
       "      <td>R</td>\n",
       "      <td>0205423</td>\n",
       "      <td>0004866</td>\n",
       "      <td>0000420</td>\n",
       "      <td>0000187</td>\n",
       "      <td>0001746</td>\n",
       "      <td>0025978</td>\n",
       "      <td>0734319</td>\n",
       "      <td>0025978</td>\n",
       "      <td>0734319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "      <td>1993-10-15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12136938.0</td>\n",
       "      <td>110</td>\n",
       "      <td>0.676269</td>\n",
       "      <td>6.6</td>\n",
       "      <td>R</td>\n",
       "      <td>0000389</td>\n",
       "      <td>0000421</td>\n",
       "      <td>0001459</td>\n",
       "      <td>0001151</td>\n",
       "      <td>0005315</td>\n",
       "      <td>0394280</td>\n",
       "      <td>nan</td>\n",
       "      <td>0171474</td>\n",
       "      <td>0192342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80</td>\n",
       "      <td>0.125477</td>\n",
       "      <td>5.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>1119183</td>\n",
       "      <td>nan</td>\n",
       "      <td>0322198</td>\n",
       "      <td>1119183</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id release_date     budget     revenue runtime popularity  \\\n",
       "0       2  0094675   1988-10-21        0.0         0.0      69   0.623849   \n",
       "1       3  0092149   1986-10-16        0.0         0.0      76   0.288572   \n",
       "2       5  0113101   1995-12-25  4000000.0   4300000.0      98   1.768986   \n",
       "3       6  0107286   1993-10-15        0.0  12136938.0     110   0.676269   \n",
       "4       8  0825671   2006-01-01    42000.0         0.0      80   0.125477   \n",
       "\n",
       "   user_rating USA_rating    cast1    cast2    cast3    cast4    cast5  \\\n",
       "0          6.8  Not Rated  0656999  0352099  0671231  0383990  0656998   \n",
       "1          6.8        NaN  0671231  0653669  0475411  0631741  0477871   \n",
       "2          6.4          R  0205423  0004866  0000420  0000187  0001746   \n",
       "3          6.6          R  0000389  0000421  0001459  0001151  0005315   \n",
       "4          5.2        NaN      nan      nan      nan      nan      nan   \n",
       "\n",
       "  director1 director2  writer1  writer2  \n",
       "0   0442454       nan  0442454      nan  \n",
       "1   0442454       nan  0442454      nan  \n",
       "2   0025978   0734319  0025978  0734319  \n",
       "3   0394280       nan  0171474  0192342  \n",
       "4   1119183       nan  0322198  1119183  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge imdb info\n",
    "merged1 = pd.merge(tmdb, imdb_info, how='left', on='tmdb_id')\n",
    "merged1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>release_date</th>\n",
       "      <th>budget</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>popularity</th>\n",
       "      <th>user_rating</th>\n",
       "      <th>USA_rating</th>\n",
       "      <th>cast1</th>\n",
       "      <th>...</th>\n",
       "      <th>United Artists</th>\n",
       "      <th>Mosfilm</th>\n",
       "      <th>Walt Disney Pictures</th>\n",
       "      <th>Shaw Brothers</th>\n",
       "      <th>New Line Cinema</th>\n",
       "      <th>Touchstone Pictures</th>\n",
       "      <th>Toho Company</th>\n",
       "      <th>TriStar Pictures</th>\n",
       "      <th>British Broadcasting Corporation BBC</th>\n",
       "      <th>Miramax Films</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "      <td>1988-10-21</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>69</td>\n",
       "      <td>0.623849</td>\n",
       "      <td>6.8</td>\n",
       "      <td>Not Rated</td>\n",
       "      <td>0656999</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "      <td>1986-10-16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76</td>\n",
       "      <td>0.288572</td>\n",
       "      <td>6.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0671231</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "      <td>1995-12-25</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>4300000.0</td>\n",
       "      <td>98</td>\n",
       "      <td>1.768986</td>\n",
       "      <td>6.4</td>\n",
       "      <td>R</td>\n",
       "      <td>0205423</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "      <td>1993-10-15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12136938.0</td>\n",
       "      <td>110</td>\n",
       "      <td>0.676269</td>\n",
       "      <td>6.6</td>\n",
       "      <td>R</td>\n",
       "      <td>0000389</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80</td>\n",
       "      <td>0.125477</td>\n",
       "      <td>5.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id release_date     budget     revenue runtime popularity  \\\n",
       "0       2  0094675   1988-10-21        0.0         0.0      69   0.623849   \n",
       "1       3  0092149   1986-10-16        0.0         0.0      76   0.288572   \n",
       "2       5  0113101   1995-12-25  4000000.0   4300000.0      98   1.768986   \n",
       "3       6  0107286   1993-10-15        0.0  12136938.0     110   0.676269   \n",
       "4       8  0825671   2006-01-01    42000.0         0.0      80   0.125477   \n",
       "\n",
       "   user_rating USA_rating    cast1      ...       United Artists Mosfilm  \\\n",
       "0          6.8  Not Rated  0656999      ...                  0.0     0.0   \n",
       "1          6.8        NaN  0671231      ...                  0.0     0.0   \n",
       "2          6.4          R  0205423      ...                  0.0     0.0   \n",
       "3          6.6          R  0000389      ...                  0.0     0.0   \n",
       "4          5.2        NaN      nan      ...                  0.0     0.0   \n",
       "\n",
       "  Walt Disney Pictures Shaw Brothers New Line Cinema Touchstone Pictures  \\\n",
       "0                  0.0           0.0             0.0                 0.0   \n",
       "1                  0.0           0.0             0.0                 0.0   \n",
       "2                  0.0           0.0             0.0                 0.0   \n",
       "3                  0.0           0.0             0.0                 0.0   \n",
       "4                  0.0           0.0             0.0                 0.0   \n",
       "\n",
       "  Toho Company TriStar Pictures  British Broadcasting Corporation BBC  \\\n",
       "0          0.0              0.0                                   0.0   \n",
       "1          0.0              0.0                                   0.0   \n",
       "2          0.0              0.0                                   0.0   \n",
       "3          0.0              0.0                                   0.0   \n",
       "4          0.0              0.0                                   0.0   \n",
       "\n",
       "   Miramax Films  \n",
       "0            0.0  \n",
       "1            0.0  \n",
       "2            1.0  \n",
       "3            0.0  \n",
       "4            0.0  \n",
       "\n",
       "[5 rows x 38 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge production companies\n",
    "companies['tmdb_id'] = companies['tmdb_id'].apply(pd.to_numeric, errors='ignore')\n",
    "merged2 = pd.merge(merged1, companies, how='left', on='tmdb_id')\n",
    "merged2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>release_date</th>\n",
       "      <th>budget</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>popularity</th>\n",
       "      <th>user_rating</th>\n",
       "      <th>USA_rating</th>\n",
       "      <th>cast1</th>\n",
       "      <th>...</th>\n",
       "      <th>word_work</th>\n",
       "      <th>word_working</th>\n",
       "      <th>word_works</th>\n",
       "      <th>word_world</th>\n",
       "      <th>word_writer</th>\n",
       "      <th>word_written</th>\n",
       "      <th>word_year</th>\n",
       "      <th>word_years</th>\n",
       "      <th>word_york</th>\n",
       "      <th>word_young</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "      <td>1988-10-21</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>69</td>\n",
       "      <td>0.623849</td>\n",
       "      <td>6.8</td>\n",
       "      <td>Not Rated</td>\n",
       "      <td>0656999</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "      <td>1986-10-16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76</td>\n",
       "      <td>0.288572</td>\n",
       "      <td>6.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0671231</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "      <td>1995-12-25</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>4300000.0</td>\n",
       "      <td>98</td>\n",
       "      <td>1.768986</td>\n",
       "      <td>6.4</td>\n",
       "      <td>R</td>\n",
       "      <td>0205423</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "      <td>1993-10-15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12136938.0</td>\n",
       "      <td>110</td>\n",
       "      <td>0.676269</td>\n",
       "      <td>6.6</td>\n",
       "      <td>R</td>\n",
       "      <td>0000389</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80</td>\n",
       "      <td>0.125477</td>\n",
       "      <td>5.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.119018</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.180166</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.165786</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 427 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id release_date     budget     revenue runtime popularity  \\\n",
       "0       2  0094675   1988-10-21        0.0         0.0      69   0.623849   \n",
       "1       3  0092149   1986-10-16        0.0         0.0      76   0.288572   \n",
       "2       5  0113101   1995-12-25  4000000.0   4300000.0      98   1.768986   \n",
       "3       6  0107286   1993-10-15        0.0  12136938.0     110   0.676269   \n",
       "4       8  0825671   2006-01-01    42000.0         0.0      80   0.125477   \n",
       "\n",
       "   user_rating USA_rating    cast1     ...     word_work word_working  \\\n",
       "0          6.8  Not Rated  0656999     ...           0.0          0.0   \n",
       "1          6.8        NaN  0671231     ...           0.0          0.0   \n",
       "2          6.4          R  0205423     ...           0.0          0.0   \n",
       "3          6.6          R  0000389     ...           0.0          0.0   \n",
       "4          5.2        NaN      nan     ...           0.0          0.0   \n",
       "\n",
       "  word_works word_world word_writer word_written word_year word_years  \\\n",
       "0        0.0   0.000000         0.0     0.000000       0.0        0.0   \n",
       "1        0.0   0.000000         0.0     0.000000       0.0        0.0   \n",
       "2        0.0   0.000000         0.0     0.000000       0.0        0.0   \n",
       "3        0.0   0.000000         0.0     0.000000       0.0        0.0   \n",
       "4        0.0   0.119018         0.0     0.180166       0.0        0.0   \n",
       "\n",
       "   word_york  word_young  \n",
       "0   0.000000         0.0  \n",
       "1   0.000000         0.0  \n",
       "2   0.000000         0.0  \n",
       "3   0.000000         0.0  \n",
       "4   0.165786         0.0  \n",
       "\n",
       "[5 rows x 427 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge bag-of-words\n",
    "bagofwords['tmdb_id'] = bagofwords['tmdb_id'].apply(pd.to_numeric, errors='ignore')\n",
    "merged3 = pd.merge(merged2, bagofwords, how='left', on='tmdb_id')\n",
    "merged3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>release_date</th>\n",
       "      <th>budget</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>popularity</th>\n",
       "      <th>user_rating</th>\n",
       "      <th>USA_rating</th>\n",
       "      <th>cast1</th>\n",
       "      <th>...</th>\n",
       "      <th>Crime</th>\n",
       "      <th>Documentary</th>\n",
       "      <th>Science Fiction</th>\n",
       "      <th>Mystery</th>\n",
       "      <th>Music</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Family</th>\n",
       "      <th>War</th>\n",
       "      <th>Foreign</th>\n",
       "      <th>num_genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0094675</td>\n",
       "      <td>1988-10-21</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>69</td>\n",
       "      <td>0.623849</td>\n",
       "      <td>6.8</td>\n",
       "      <td>Not Rated</td>\n",
       "      <td>0656999</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0092149</td>\n",
       "      <td>1986-10-16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76</td>\n",
       "      <td>0.288572</td>\n",
       "      <td>6.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0671231</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0113101</td>\n",
       "      <td>1995-12-25</td>\n",
       "      <td>4000000.0</td>\n",
       "      <td>4300000.0</td>\n",
       "      <td>98</td>\n",
       "      <td>1.768986</td>\n",
       "      <td>6.4</td>\n",
       "      <td>R</td>\n",
       "      <td>0205423</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0107286</td>\n",
       "      <td>1993-10-15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12136938.0</td>\n",
       "      <td>110</td>\n",
       "      <td>0.676269</td>\n",
       "      <td>6.6</td>\n",
       "      <td>R</td>\n",
       "      <td>0000389</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>0825671</td>\n",
       "      <td>2006-01-01</td>\n",
       "      <td>42000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80</td>\n",
       "      <td>0.125477</td>\n",
       "      <td>5.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>nan</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 447 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id  imdb_id release_date     budget     revenue runtime popularity  \\\n",
       "0       2  0094675   1988-10-21        0.0         0.0      69   0.623849   \n",
       "1       3  0092149   1986-10-16        0.0         0.0      76   0.288572   \n",
       "2       5  0113101   1995-12-25  4000000.0   4300000.0      98   1.768986   \n",
       "3       6  0107286   1993-10-15        0.0  12136938.0     110   0.676269   \n",
       "4       8  0825671   2006-01-01    42000.0         0.0      80   0.125477   \n",
       "\n",
       "   user_rating USA_rating    cast1     ...     Crime Documentary  \\\n",
       "0          6.8  Not Rated  0656999     ...       1.0         0.0   \n",
       "1          6.8        NaN  0671231     ...       0.0         0.0   \n",
       "2          6.4          R  0205423     ...       1.0         0.0   \n",
       "3          6.6          R  0000389     ...       1.0         0.0   \n",
       "4          5.2        NaN      nan     ...       0.0         1.0   \n",
       "\n",
       "  Science Fiction Mystery Music Romance Family  War  Foreign  num_genres  \n",
       "0             0.0     0.0   0.0     0.0    0.0  0.0      0.0         2.0  \n",
       "1             0.0     0.0   0.0     0.0    0.0  0.0      0.0         2.0  \n",
       "2             0.0     0.0   0.0     0.0    0.0  0.0      0.0         2.0  \n",
       "3             0.0     0.0   0.0     0.0    0.0  0.0      0.0         3.0  \n",
       "4             0.0     0.0   0.0     0.0    0.0  0.0      0.0         1.0  \n",
       "\n",
       "[5 rows x 447 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge outcome variables\n",
    "genres['tmdb_id'] = genres['tmdb_id'].apply(pd.to_numeric, errors='ignore')\n",
    "merged4 = pd.merge(merged3, genres, how='left', on='tmdb_id')\n",
    "merged4.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(71458, 447)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged4.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prior to modelling in milestone 3, we will get dummy variables for categorical variables (that have not already been transformed into indicator variables), such as cast, director and writer. The same approach for production companies will be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imbalanced Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we visualize the imbalanced nature of our data by plotting the number of movies classified 1 or 0 in each genre, and the proportion of movies with different number of genres assigned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1695c358>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAFBCAYAAACRsROQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8XHV9//HXGwgCEhCEJCUQQCEsgmLEiEvbS1EQZbPK\nEpCl0KIFBZRqSX8WEmtFsEpAClVANtEQUQRkCetFERQIq4QlLUtIAgEJCTRQZPn8/vieSc69mZt7\nzplz13k/H4/7uDPfOed7vzN3Zj7nuysiMDOz9rTKQBfAzMwGjoOAmVkbcxAwM2tjDgJmZm3MQcDM\nrI05CJiZtbFeg4Ck8ZLulXRP9nuJpGMkrSfpekmPSpopad3cOZMlzZH0sKRdc+kTJD0g6TFJ03Lp\nq0uanp1zh6Rx9T9VMzPrrtcgEBGPRcT7I2IC8AFgKXA5cAJwY0RsBdwMTAaQtC2wH7ANsDtwliRl\n2Z0NHBER44HxknbL0o8AFkXElsA04NS6nqCZmfWsbHPQx4H/iYingb2BC7P0C4F9stt7AdMj4o2I\neBKYA0yUNAYYGRF3ZcddlDsnn9dlwC5ln4iZmZVXNgjsD/w0uz06IhYCRMSzwKgsfSzwdO6c+Vna\nWGBeLn1eltblnIh4E1gsaf2SZTMzs5JWK3qgpBGkq/x/zpK6rzdR5/oTapooeY0LM7MKIqLp92rh\nIEBq358VEX/K7i+UNDoiFmZNPc9l6fOBTXLnbZyl9ZSeP2eBpFWBdSJiUQ9PZNnt3Xbbh7lz53d5\n/Pnn57PhhmOX3R83biwzZ/5qpU+sr/LpnkeRfAZTWYrk49e3elnqymeoPSe/Z6qXpUg+zfJY3i27\nojJBYBLws9z9K4HDgFOAQ4ErcumXSDqN1MyzBXBnREQ2smgicBdwCHBG7pxDgT8A+5I6mns1d+58\nxoz5Upe0V1+9nDFjPpM75swBy6d7HkXyGUxlKZKPX9/qZakrn6H2nPyeqV6WIvkUySOvUBCQtBap\nU/jIXPIpwAxJhwNPkUYEERGzJc0AZgOvA0fF8sv3o4ELgDWAayLiuiz9POBiSXOAF4ADSj0LMzOr\npFAQiIhXgA27pS0iBYZmx58MnNwkfRawfZP018iCSKve8Y6t68imlnxclr7Nx2Xp23wGU1nqysdl\nWdGwmzG83nrbDJp8XJa+zcdl6dt8BlNZ6srHZVlRmT4BM7NBbbPNNuOpp57qkvbII4f1et7KOk7r\nyqM/8hk5chRjx5ZbcMFBwMyGjaeeeop23i0xBZByQWDYNQeZmVlxDgJmZm3MQcDMrI05CJiZtTEH\nATOzNubRQWY2bB2w224snDu3z/IfPW4c02fO7PW4gw8+mBEjRvDjH/94Wdqtt97KZz/7WR566CFG\njx7dZ2XsjYOAmQ1bC+fO5ZYxY/os/50LBpjTTz+d7bbbjptuuolddtmF1157jSOPPJLTTjttQAMA\nuDnIzKzPrb/++pxxxhkceeSRvPLKK0yZMoUtttiCgw8+mIjg29/+NltssQWjRo3iwAMPZMmSJQC8\n+uqrHHTQQWywwQast9567LTTTixa1HSB5cocBMzM+sHnPvc5JkyYwKRJkzj33HM555xzAPj+97/P\ntddey2233ca8efNYe+21OeaYYwA4//zzefXVV1mwYAGLFi3irLPOYo011qi1XA4CZmb95D//8z+5\n+eabOemkk9hoo40A+OEPf8i3v/1txowZw+qrr86//uu/MmPGDABGjBjBn/70Jx577DEkMWHCBNZa\na61ay+Q+ATOzfjJq1Cg22GADtt1222Vpc+fOZc8992SVVdI1eUSwyiqr8Nxzz3HYYYfxzDPPsN9+\n+/Hyyy9z8MEH861vfWvZsXVwTcDMbABtsskm3HDDDSxatIhFixbx4osvsnTpUkaNGsWIESM48cQT\nmT17Nrfddhu//OUvueSSS2r9+w4CZmYD6Atf+AKTJ0/m6aefBuC5557jqquuAuCWW27hoYceIiJY\ne+21GTFiRK21AHBzkJkNY6PHjSs8jLNq/mV1Xyr6+OOPRxK77LILCxcuZPTo0UyaNIk999yTBQsW\n8MUvfpFnnnmGtddem0mTJnHggQfWVXzAQcDMhrEiE7n62+OPP97lviSOP/54jj/++BWOPeiggzjo\noIP6tDxuDjIza2MOAmZmbcxBwMysjTkImJm1MQcBM7M2VigISFpX0s8lPSzpIUkfkrSepOslPSpp\npqR1c8dPljQnO37XXPoESQ9IekzStFz66pKmZ+fcIan8uCszMyutaE3gdOCaiNgGeB/wCHACcGNE\nbAXcDEwGkLQtsB+wDbA7cJaWD4w9GzgiIsYD4yXtlqUfASyKiC2BacCpLT8zMzPrVa9BQNI6wF9G\nxPkAEfFGRCwB9gYuzA67ENgnu70XMD077klgDjBR0hhgZETclR13Ue6cfF6XAbu09KzMzKyQIjWB\nzYE/STpf0j2SfiRpLWB0RCwEiIhngVHZ8WOBp3Pnz8/SxgLzcunzsrQu50TEm8BiSetXfE5mZlZQ\nkRnDqwETgKMj4m5Jp5GagqLbcd3vt0I9PTBlypRlt5cufanGP2lmw81uu+3D3Lnz+yz/cePGMnPm\nrwodu9lmm/Hqq6/y5JNPsuaaawJw3nnn8ZOf/IRbbrml1nJ1dnbS2dlZ6NgiQWAe8HRE3J3d/wUp\nCCyUNDoiFmZNPc9lj88HNsmdv3GW1lN6/pwFklYF1omIptvn5IPApZdeXaD4Ztau5s6dz5gxX+rD\n/M8sfKwk3nrrLaZNm8bkyZO7pNeto6ODjo6OZfenTp3a47G9NgdlTT5PSxqfJe0CPARcCRyWpR0K\nXJHdvhI4IBvxszmwBXBn1mS0RNLErKP4kG7nHJrd3pfU0WxmNqx87Wtf43vf+x4vvbRiK8btt9/O\nxIkTWW+99fjQhz7EHXfc0S9lKjo66BjgEkn3kUYHfRs4BfiEpEdJgeE7ABExG5gBzAauAY6KiEZT\n0dHAecBjwJyIuC5LPw/YQNIc4DhSTcPMbFjZcccd6ejo4Lvf/W6X9BdffJE99tiD4447jhdeeIGv\nfOUrfPrTn+bFF1/s8zIVWkU0Iu4HPtjkoY/3cPzJwMlN0mcB2zdJf400rNTMbFibOnUqH/vYxzju\nuOOWpV199dWMHz9+2TLRBxxwAGeccQZXXXUVhxxySJ+WxzOGzcz60Xve8x722GMPTj45XSdHBAsW\nLGDTTTftctymm27K/Pl916nd4CBgZtbPpkyZwjnnnMP8+fORxNixY3nyySe7HDN37lzGjh3bPIMa\nOQiYmfWzd7/73ey///6cccYZAOy+++7MmTOH6dOn8+abb3LppZfy8MMPs8cee/R5WbyzmJkNW+PG\njS01jLNK/kV1Hwp64okn8pOf/ARJrL/++vz617/mmGOO4R//8R/ZYostuPrqq1l//b6fM+sgYGbD\nVtGJXP2h+7aSG2+8Ma+88sqy+x/5yEe4++67u5/W59wcZGbWxhwEzMzamIOAmVkbcxAwM2tjDgJm\nZm3MQcDMrI15iKiZDRubbrppnyzNPFSMHDmq94O6cRAws2Gj+9IL22zzwV73E3j22TN5+OG7eny8\njjz6M59nny03Oc7NQWZmbcxBwMysjTkImJm1MQcBM7M25iBgZtbGHATMzNqYg4CZWRtzEDAza2MO\nAmZmbcxBwMysjRUKApKelHS/pHsl3ZmlrSfpekmPSpopad3c8ZMlzZH0sKRdc+kTJD0g6TFJ03Lp\nq0uanp1zh6RxdT5JMzNrrmhN4C2gIyLeHxETs7QTgBsjYivgZmAygKRtgf2AbYDdgbO0fEWns4Ej\nImI8MF7Sbln6EcCiiNgSmAac2uLzMjOzAooGATU5dm/gwuz2hcA+2e29gOkR8UZEPAnMASZKGgOM\njIjG6kgX5c7J53UZsEuZJ2FmZtUUDQIB3CDpLkl/n6WNjoiFABHxLNBYw3Qs8HTu3PlZ2lhgXi59\nXpbW5ZyIeBNYLGn9ks/FzMxKKrqU9Ecj4hlJGwLXS3qUFBjyut9vRY8Lgk+ZMmXZ7aVLX6rxT5qZ\nDQ+dnZ10dnYWOrZQEIiIZ7Lfz0v6FTARWChpdEQszJp6nssOnw9skjt94yytp/T8OQskrQqsExGL\nmpUlHwQuvfTqIsU3M2srHR0ddHR0LLs/derUHo/ttTlI0lqS1s5uvx3YFXgQuBI4LDvsUOCK7PaV\nwAHZiJ/NgS2AO7MmoyWSJmYdxYd0O+fQ7Pa+pI5mMzPrY0VqAqOByyVFdvwlEXG9pLuBGZIOB54i\njQgiImZLmgHMBl4HjoqIRlPR0cAFwBrANRFxXZZ+HnCxpDnAC8ABtTw7MzNbqV6DQEQ8AezQJH0R\n8PEezjkZOLlJ+ixg+ybpr5EFETMz6z+eMWxm1sYcBMzM2piDgJlZG3MQMDNrYw4CZmZtzEHAzKyN\nOQiYmbUxBwEzszbmIGBm1sYcBMzM2piDgJlZG3MQMDNrYw4CZmZtzEHAzKyNOQiYmbUxBwEzszbm\nIGBm1sYcBMzM2piDgJlZG3MQMDNrYw4CZmZtzEHAzKyNOQiYmbWxwkFA0iqS7pF0ZXZ/PUnXS3pU\n0kxJ6+aOnSxpjqSHJe2aS58g6QFJj0malktfXdL07Jw7JI2r6wmamVnPytQEjgVm5+6fANwYEVsB\nNwOTASRtC+wHbAPsDpwlSdk5ZwNHRMR4YLyk3bL0I4BFEbElMA04teLzMTOzEgoFAUkbA58Czs0l\n7w1cmN2+ENgnu70XMD0i3oiIJ4E5wERJY4CREXFXdtxFuXPyeV0G7FL+qZiZWVlFawKnAV8DIpc2\nOiIWAkTEs8CoLH0s8HTuuPlZ2lhgXi59XpbW5ZyIeBNYLGn94k/DzMyqWK23AyR9GlgYEfdJ6ljJ\nobGSx8pSTw9MmTJl2e2lS1+q8U+amQ0PnZ2ddHZ2Fjq21yAAfBTYS9KngDWBkZIuBp6VNDoiFmZN\nPc9lx88HNsmdv3GW1lN6/pwFklYF1omIRc0Kkw8Cl156dYHim5m1l46ODjo6Opbdnzp1ao/H9toc\nFBH/EhHjIuJdwAHAzRFxMHAVcFh22KHAFdntK4EDshE/mwNbAHdmTUZLJE3MOooP6XbOodntfUkd\nzWZm1seK1AR68h1ghqTDgadII4KIiNmSZpBGEr0OHBURjaaio4ELgDWAayLiuiz9POBiSXOAF0jB\nxszM+lipIBARtwK3ZrcXAR/v4biTgZObpM8Ctm+S/hpZEDEzs/7jGcNmZm3MQcDMrI05CJiZtTEH\nATOzNuYgYGbWxhwEzMzamIOAmVkbcxAwM2tjDgJmZm3MQcDMrI05CJiZtTEHATOzNuYgYGbWxhwE\nzMzamIOAmVkbcxAwM2tjDgJmZm3MQcDMrI05CJiZtTEHATOzNuYgYGbWxhwEzMzamIOAmVkb6zUI\nSHqbpD9IulfSg5JOytLXk3S9pEclzZS0bu6cyZLmSHpY0q659AmSHpD0mKRpufTVJU3PzrlD0ri6\nn6iZma2o1yAQEa8BO0fE+4EdgN0lTQROAG6MiK2Am4HJAJK2BfYDtgF2B86SpCy7s4EjImI8MF7S\nbln6EcCiiNgSmAacWtcTNDOznhVqDoqIV7KbbwNWAwLYG7gwS78Q2Ce7vRcwPSLeiIgngTnARElj\ngJERcVd23EW5c/J5XQbsUunZmJlZKYWCgKRVJN0LPAvckH2Rj46IhQAR8SwwKjt8LPB07vT5WdpY\nYF4ufV6W1uWciHgTWCxp/UrPyMzMClutyEER8RbwfknrAJdLeg+pNtDlsBrLpZ4emDJlyrLbS5e+\nVOOfNDMbHjo7O+ns7Cx0bKEg0BARL0nqBD4JLJQ0OiIWZk09z2WHzQc2yZ22cZbWU3r+nAWSVgXW\niYhFzcqQDwKXXnp1meKbmbWFjo4OOjo6lt2fOnVqj8f2GgQkbQC8HhFLJK0JfAL4DnAlcBhwCnAo\ncEV2ypXAJZJOIzXzbAHcGREhaUnWqXwXcAhwRu6cQ4E/APuSOprb0uLnn+Gpx7+60mPWHblmP5XG\nzIa7IjWBvwAulLQKqQ/h0oi4RtLvgRmSDgeeIo0IIiJmS5oBzAZeB46KiEZT0dHABcAawDURcV2W\nfh5wsaQ5wAvAAbU8uyFo6w1Hcsv2W670mJ2ffbafSmNmw12vQSAiHgQmNElfBHy8h3NOBk5ukj4L\n2L5J+mtkQcTMzPqPZwybmbUxBwEzszbmIGBm1sYcBMzM2piDgJlZG3MQMDNrYw4CZmZtzEHAzKyN\nOQiYmbUxBwEzszbmIGBm1sYcBMzM2piDgJlZG3MQMDNrYw4CZmZtzEHAzKyNOQiYmbUxBwEzszbm\nIGBm1sYcBMzM2piDgJlZG3MQMDNrYw4CZmZtrNcgIGljSTdLekjSg5KOydLXk3S9pEclzZS0bu6c\nyZLmSHpY0q659AmSHpD0mKRpufTVJU3PzrlD0ri6n6iZma1otQLHvAF8NSLuk7Q2MEvS9cDfATdG\nxKmS/hmYDJwgaVtgP2AbYGPgRklbRkQAZwNHRMRdkq6RtFtEzASOABZFxJaS9gdOBQ6o/dnagFj8\n/DM89fhXV3rMuiPX7KfSmFler0EgIp4Fns1u/6+kh0lf7nsDf50ddiHQCZwA7AVMj4g3gCclzQEm\nSnoKGBkRd2XnXATsA8zM8jopS78MOLP1p2aDxdYbjuSW7bdc6TE7P/tsP5XGzPJK9QlI2gzYAfg9\nMDoiFsKyQDEqO2ws8HTutPlZ2lhgXi59XpbW5ZyIeBNYLGn9MmUzM7PyijQHAZA1BV0GHJvVCKLb\nId3vt0I9PTBlypRlt5cufanGP2lmNjx0dnbS2dlZ6NhCQUDSaqQAcHFEXJElL5Q0OiIWShoDPJel\nzwc2yZ2+cZbWU3r+nAWSVgXWiYhFzcqSDwKXXnp1keKbmbWVjo4OOjo6lt2fOnVqj8cWbQ76MTA7\nIk7PpV0JHJbdPhS4Ipd+QDbiZ3NgC+DOrMloiaSJkgQc0u2cQ7Pb+wI3FyyXmZm1oNeagKSPAgcB\nD0q6l9Ts8y/AKcAMSYcDT5FGBBERsyXNAGYDrwNHZSODAI4GLgDWAK6JiOuy9POAi7NO5BfwyCAz\ns35RZHTQ74BVe3j44z2cczJwcpP0WcD2TdJfIwsiZmbWfzxj2MysjTkImJm1MQcBM7M25iBgZtbG\nHATMzNqYg4CZWRtzEDAza2MOAmZmbcxBwMysjTkImJm1MQcBM7M25iBgZtbGHATMzNqYg4CZWRtz\nEDAza2MOAmZmbcxBwMysjTkImJm1MQcBM7M25iBgZtbGHATMzNqYg4CZWRtzEDAza2O9BgFJ50la\nKOmBXNp6kq6X9KikmZLWzT02WdIcSQ9L2jWXPkHSA5IekzQtl766pOnZOXdIGlfnEzQzs54VqQmc\nD+zWLe0E4MaI2Aq4GZgMIGlbYD9gG2B34CxJys45GzgiIsYD4yU18jwCWBQRWwLTgFNbeD5mZlbC\nar0dEBG3Sdq0W/LewF9nty8EOkmBYS9gekS8ATwpaQ4wUdJTwMiIuCs75yJgH2BmltdJWfplwJnV\nn47VbfHzz/DU41/t8fF1R67Zj6UZPHp7XaB9XxsbWnoNAj0YFRELASLiWUmjsvSxwB254+ZnaW8A\n83Lp87L0xjlPZ3m9KWmxpPUjYlHFslmNtt5wJLdsv2WPj+/87LP9WJrBo7fXBdr3tbGhpWoQ6C5q\nygdAK3twypQpy24vXfpSjX/WzGx46OzspLOzs9CxVYPAQkmjI2KhpDHAc1n6fGCT3HEbZ2k9pefP\nWSBpVWCdldUC8kHg0kuvrlh8M2sH7dps19HRQUdHx7L7U6dO7fHYokFAdL1CvxI4DDgFOBS4Ipd+\niaTTSM08WwB3RkRIWiJpInAXcAhwRu6cQ4E/APuSOprNzFrmZrve9RoEJP0U6ADeKWkuqRP3O8DP\nJR0OPEUaEUREzJY0A5gNvA4cFRGNpqKjgQuANYBrIuK6LP084OKsE/kF4IB6npqZmfWmyOigA3t4\n6OM9HH8ycHKT9FnA9k3SXyMLImZm1r88Y9jMrI05CJiZtTEHATOzNlbXPAEbZNp1aJyZleMgMEx5\naJyZFeHmIDOzNuYgYGbWxtwcZDaA6ui7cf+PtcJBwGwA1dF34/4fa4Wbg8zM2piDgJlZGxvSzUFu\nCzWzdlP3bn9DOgi4LdTM+tpgu9ise7e/IR0EzKw+g+3LbrAY7hebDgI18ofIhrLh/mVnzTkI1Mgf\nIjMbahwE8BW8WZ3q6Lj0Z7L/OAjgK3izOtXRcenPZP/xPAEzszbmIGBm1sYcBMzM2pj7BGzI8Iqb\nNpQN1vfeoAkCkj4JTCPVTs6LiFMGuEg2yHjFTRvKBut7b1A0B0laBTgT2A14DzBJ0tZV8up88cVa\nylRHPi5L3+bjsvRtPoOpLHXl47KsaFAEAWAiMCcinoqI14HpwN5VMupcvLiWAtWRj8vSt/m4LH2b\nz2AqS135uCwrGixBYCzwdO7+vCzNzMz60GAJAmZmNgAUEQNdBiTtBEyJiE9m908AonvnsKSBL6yZ\n2RAUEWqWPliCwKrAo8AuwDPAncCkiHh4QAtmZjbMDYohohHxpqQvAdezfIioA4CZWR8bFDUBMzMb\nGO4YNjNrYw4CZiVJWkXSfgNdDrM6OAjkZB3ULechaSNJ4xo/dZStYjluGYi/3YykL0tar4Z83llH\neVoREW8BX687X0lrtXj+ntns+1byGCvpI5L+qvEzUGWpW6uvbw1/X5I2GcgyNDOo/klVZC/s5yWd\nmN0fJ2lixezmSPqupG0rluXLwELgBuDq7OfXFfPaUtJlkmZLerzxU/T8iHgTeEvSulX+fq4cG0r6\nF0k/kvTjxk+FrEYDd0maIemTkpoOVyvg95J+LulTVfOQ9FFJN0h6LHtdnyjz2mZulPRPkjaRtH7j\np2J5PiJpNvBIdv99ks6qkNX+pPfwqVWWXZF0CvA74BvA17Kff6pQjpbLkivTDZLekbu/nqSZJfOo\n5fWV9EtJn64a3CJ1wF5T5dwmZRkv6RxJ10u6ufFTKa+h3jEs6WzgLeBvImKb7Grz+oj4YIW8RgIH\nAH9HCpA/BqZHxEsFz/9v4EMR8ULZv90kr9uAk4DTgD0bZYqIE0vkcQXwflJQWtpIj4hjSuRxO/Bb\nYBbwZi6PXxTNI5eXgF1Jz2VHYAZpJNj/lMzj48DhwAezPC6IiMdK5PEI8BVWfE6F/2+SnmiSHBHx\nrqJ55PL6A/A54MqIeH+W9seI2K5CXusAk0ivcQDnAz+LiJcLnPso8N6IeK3s3627LLk87m28JitL\n6yWPWl5fSR8nPZedgJ8D50fEoyXzuBA4MyLuKnNek3zuB/6LFd/Ds0pnFhFD+ge4J/t9by7t/hry\n/WtgPunL80JgiwLn3AKsVtPzmpX9frB7Wok8Dm32UzKP+2r+f72PtFrsI8DZwL3AqRXz2jn7Hy0G\nbgU+XPC8P9T5nGp4Tf6Q/a7lPQy8EzgOeBK4FpgDfLnAedcCa9f83CqVJXf+LGBc7v6mjc/8AL6+\n6wJfJC11czspMIwoeO4jwBvA/wAPAA8CD1QoQ6nvgpX9DIp5Ai16PWvLD0jNF6SaQWlZPp8m/VM3\nA74HXAL8JakaN76XLB4HOiVdDSy7moqI71cozmtZtXOO0hyK+cDaZTKIiAslrc7ycj8aaYG+Mn4t\n6VMR0VI1VtKxwCHAn4Bzga9FxOuN50jBNvasT+DzwMGkprcvA1cCO5CuzjYvkM0tkr4L/JKu/6d7\nSjyftYCvkr6gjpS0JbBVRFRp/nta0keAkDQCOBYoPU9G0t7AYcAWwEXAxIh4LivrbOAHvWTxCnCf\npJvo+roUrjnWWJaG/wfcJulWQKTP4pEli1PL6wsrvP/uJX0/fIx0gdVRIIvdqvzdJq6SdBRwOV3/\nV4vKZjQcmoMOIrU/TiBdsX8O+EZE/LxCXo+TrubPi4jbuz12Rm8fBkknNUuPiKkVyvJB0hv1HcC/\nka4+To2I35fIo4P0mjxJ+gBtQqoJ/KZEHi8Dbwf+DDQCSETEOkXzyPKZCvw4Ip5q8tg2UXByoKTH\ngItJVfF53R775yiwD4Wad5hHRPxNkTJkeVxKuko9JCK2y77cbo+IHYrmkctrA+B0UjOXSJMmj42S\nzYqSLiC9xiv8fyXtEhE39XL+oc3SI+LCMuWooyzdjt+A1AQD8PuI+FPJstT1+l4ObEV6/10QEc/k\nHrs7InYskdcoYI3G/YiYW7Is9TVHDvUgAJB1PO1C+gffVPQLpUk+a0fE/9ZQnrUB6sirxXLMAg6M\nrN1S0nhSm+wH+rEMK+0sLXPlktXUTo2I41sozyrA5yJiRtU8snzujogd8+3Tku6PiPe1km8L5VkV\nuDEidm4xn1ZrjrWURdLWEfGIpAnNHi9Ta6uLpJ0joqURd5L2IrUwbAQ8R2reejgi3lNDESsZ0s1B\n2ZvtoYjYmqznv0UnSvoW8CpwHfBe4CsR8ZOC5dmOdJWwfnb/T6QrxYfKFkTSjqSq8Kbk/k8R8d4S\n2YyIXMdVRDyWVYfLlmUvoDFUsLNkk8csUlOdgHHAi9ntdwBzKdZ8AyxbXuQjJf52szzekvR1Uody\nK/4saU2WN0O+m1y1vAxJm5OatTaj6/96r6J5ZK/NW5LWjYglFcvRQbeao6RSNce6ykJqajuS9IW5\nwp8AytTaLiRd+S/O7q8HfC8iDi94/t82u72sMBG/LFoWUq1+J1KQfL+knUnNS6U0KwewhNSH+FyZ\nvIZ0EMjebI9KGle2OtWDXSPi65I+Q/og/C3wG6BQEAB+BHy1cbWQfajOAap8cV1CGqL3IBX7OIC7\nJZ3L8vIfBNxdJgNJ3yGNwrkkSzpW0kcjYnKR8yNi8yyfc4DLG30LknYH9ilTlsx9kq4ktf/nRzyV\n+SDeKOmfgEu75VGmPfUk0oXCJpIuAT5KagOv4lfAecBVVP9fA/wv8KCkqqPBvkf6DHSpOQJVao4t\nlSUijsx+t1Szyby3EQCyPF+UVHh0EWl0Xk+C1LdU1OsR8YLShMNVIuIWSdNKnN9wBPBhUvM1pP6I\nWcDmkr7xZ8paAAAZRUlEQVQZERcXzWhIB4HMesBDku6k65ut8FVUTuMq+dPAzyNiicoNRX97vroY\nEZ2S3l6hHADPR8SVFc9t+EfgaKDxwfstUHZ89KeAHSJNkGpcVd0LFAoCOTtFxD807kTEtZJOLZkH\npHbUF+h6JVj2g7h/9vvobnkUbk+NiBsk3UO6qhPpSrNUW3XO/0XEGRXPzfsl5V6H7mqpOdZUFgAk\n7QtcFxEvS/oGqe/v3yLi3hLZrCJpvYh4MctzfUp890XE35Uq9MotzpqLfwtcIuk5ct9bJawGbBMR\nCwEkjSZ1wH+IdOFaOAgM+T4BSX/dLD0ibq2Q13dIV6evkra8fAfw64j4UMHzLwfuYfk/4PPAByLi\nMxXKsgtpjHX3kRqFPlhZU9lFEXFQ2b/dLZ8HgI7GVXL2Aeos2SyF0gSf39K1VvJXEVHXaIl+0VMb\ndUOVtmpJBwJbkjosK41WqoPSJMC36Po/WrVos0mT/NYkjZ4qNZa+Wx4PRMR7JX0M+BbwXeDEop/J\nLI9DgH8h1R5FGjzy70WvliV9PiJ+IumrzR6PAqP/JB1HGk46mzQKaxXS67sucEmFTurZEbFt7r5I\nTePbquQ8iiFfE6jyZb+SvE7Irk6XZE1NSym31/HhwFSWXwH9Nkur4u+ArUm1k0YTQeEr3qz8m0pa\nPSL+XLEMACcD92YjakTqGzihQj6TSE0ol5Oex2+ytFIkbUwaXvjRLOm3pKvweT2ftUIerQzvbNZG\n3VCqrTpne9KQw7+h6/+6VF7Z8zgZ2JauI0+K1nDqqDk2yrIn8B/A6qQmih2Ab1aooTcmQn0a+FFE\nXJ312xUWERdlgyQaTUt/GxGzS2TRqM2PLPN3u9mYND9ma1IT7+9IQeGqks2QDZ2Sfk0KbACfzdLe\nTpo3U9hwqAm8TNY5R3rDjQCWRskhjLn8tmPFD9FFrZazQjkejYitWszjImAb0jj6fFNZqXkLkv6C\n1C8AcGdEPNtCmd4eEVWqv43zbwB+Stfa1kER8YkSedQ2vLMOSjPNt20xWKMaZpnXJfvS/RtSrbGV\nWbq/Js2R+QSpKehV0nuw1CisrGY8mq4d73X0I5aSjb7akdRP+OHsZ3H+qr5gPiJ98Tcuhn4H/CIq\nfKEPh5rAsuicvTB7s3xMcSlK4/w7SEHgGmB34DZSW9vKzpsWEcdJuorlASlfxir9E7dL2rbkFUt3\n/5P9rELJq5gmQ/QaV9obSdqobFNFNqrnXNKEt3GS3gd8ISKOKpMPsGFEnJ+7f0FW1S7j3RGxv6RJ\nABHxigp2/vQwKmOZkh3UDX8kNT2WGtXRxJoRcZMkRZqPMSX7Ml5pEJA0IyL2k/Qgzd+/pZr+Mq83\n6VOr0um9H/BJ4D8iYnF2QfK1Mhkorel1Emly4ZukGm2QRv+VyaflUVzAmsA6pGagdYEFpJpBKdmX\n/WXZT0uGfBDIy16YX2Vf5lWaLD5HWtbg3oj4u6yzpcjIoMZV6X9U+Js92Yk0EuYJUjuxSE+x0Bs3\nu/IZGRFVFwCrbYhe5jTSbMkrASLiflVbofIFSZ8njVqB1KRUdq2mVoZ31jlSpOEdwCOS7qJrn0DZ\ni4eqs8yPzX7vUfLvrcxDWV/Hqlkz1TGk5o+yNiAb0ablK/KWHQ5+LKm5r9U1vSqP4pL0I+A9wMvA\nH0ivxfcbndUl8rktIj7WrQUEln8/lG4BGfJBoNuV2Sqkqtb/Vczu1UjjyN9QWvzqOdIs25WK5Ys2\n7RARp3cr37GkdW3K+mSFc/JlelPSR3s/ssfzG1Pzd4+ILq+npDWanFIkz6e7XRm+2dOxK3E4qU/g\nNNKH4HbKD82cworDOwuNAKl5pEhD05nmFRwLrEX6wv03Uhv4Ib2dFMtnvh4VEf+cf0xpZdF/XvGs\nXn2ZNM/lNVLz3cysTGVdzfJ5JmuQ5pU8SvpCLepp0hj6VrUyimsc8DbSEinzSTXrUm33ABHxsex3\nK/0TXQyHPoF808AbpPH950TJCRNZXmeRRhEcABxPGut8X9EPvqR7ImJCt7RSPfVN8qw8vVxphdWx\ntDCmvofntEJagXwuA74PnEkaxnYssGNEHFAyn49GxO96SyuQzztZPryz8FIEdYwU6SuS9o1uy6U0\nS1vJ+c3+1w9UaQ5qtSwryXcCKVj9fYlzziMt99DSml6tjuLKmhzfQ+oP+AiwHbAIuCMiSl8IZCOm\ntoyI85WWxhgZEc2Wk1ipIV8TAM5t9qVAhfbVXPv0f0m6DlgnIh7o7bysbflA0iiI/Nj+kaR/cmnq\nYXo55a6AKo+plzSGFEDWVJpY07iEX4d0tVnWF0nrt4wlXQldT9dx+kX9gNRB2FtajyTdFBG7kL4U\nuqf1po6RIo2/WXfVfjLLR4usLK17Of4ROAp4t9KQ4IaRVGvCqVyW3kTEPZIKDw/NzM1+Vs9+qmpp\nFFfWXP1HSYtJNZMlpCa4iZSsDWZN3juSgtv5pOf1E5Z3FBc2HIJAy18KDfkvgoh4snvaStwOPENq\nv8y3ob9MWi62ipanl7fYdLEbqZllY9JzagSBl0i1pVKyK+3KcxYkfZh09bRht6vwdYBCO8JlzVhr\nARsoLR2QD2xji+QRET/M+lteiojTipa/h7xqqdorzb7+FDBWUr65Yh1S7bg3PyUt83wyXfvSXi47\nfLGGsnTPL/+/XoX0uV5QJo+osIBjD/YF3lVlFJekY1heA3id9J1xO2nPktIdw8BnSHuF3AMQEQuU\n9kMpbcgGgTq+FHJ5tfTlkI3EeIo03KsuLU8vVwtj6iOtHHmhpM9GhQ1kmpSl1ZEVq5M6OVej61X4\nS6QO/SK+QFrbfiPSENF8YDuzYB6N/pZJpH6Jlqjr+ldVLSB1nu5Fel4NL5M2z1mpSOv7LJF0OrAo\nsk1fJK0j6UMR8Yf+KksT+f/1G6TaW6n3o9Ly8l8n1aLzTatlBze0MoprM1It6Cu5PphW/DkiQlJj\ncEPVlQmGbhCgni+Fhlq+HCTtRPrS3SYr36pUn7PQmF7+G6pPLz+fdJW3b3b/81la4TH1wAey2lB+\n8a3jI+IbJcvS0vo4kSYF3irpgizoko2EWTsK7vyWddqfLunLEVF0Pfue/E7Smay4/lCpobNRw/pX\nEXE/cL+kn0aFVT9zzqZrDfp/m6SVLkv2ntmk7EiYLL86ruIvIf2f9iA1Sx4KPF8hn8qjuCKiaR9S\nC2ZI+iHwDkn/QBowcU6VjIZDx/Cm0WSN+op5tfTlIOluUqfyz0ntdYcA46PgYmvd8no7aWJM5enl\nku6LbhOgmqX1kkez7f2qdAz/IUpM9V9JPj8lfZDfBO4i1dZOj4jvlsij2Xo03yrzBa4a9iTI5fUb\nUtW+pfWvsr6wKSxfebbRt1BoxnAP75eqHcOdpNrAaqQLq+dIE/IK1Qa69a2toMxrI2lWRHwg/1wk\n3RUlt6BVjUvU1EHSJ0jbtQqYGRE3VMlnKNcEGt6mNAZ3M7o2M5T+MEbED5QmNXXPq/CM4Yj4b0mr\nRtro/XxJpRdby5oIfh1pBcW3SMv7VlHHmPpVJb0tsn1nlcbXv61CWU7POrNaXR9n24h4SWkzoWtJ\nbdizSGvKFPWvEfHzbHTFx7NzzyaNWiokaljdUtIWpFms/9rtob8k9TGVdR5N9k4u4fGs7frs7P5R\npN3yqlg3+z/9PWkNq5O6dTr35sOkoZ0/I42rL7WSYzeN2tEzkj5NarJa6T4XzQzUl3136rpfQ6Uv\n/rzhEAR+Ttpw+VyqvfGXkXQx8G7gvlxeQS8zhnNeUZoWfp/SGkTPkK7kS4l61mOH5mPqy3YWXwLc\npDQUV6TO4ipBqZb1cYARSitb7kPasPv1RrtoCS2vRyPpbaRp+5vR9YLhmyWymQZMjoguHYOSFgHf\nJn2pl7EkIq4teU7eF4EzgG+Q/jc3UX4rx4bVlGb37keaL1DWGFKzZWPk3dWkDZFK780BfEvSuqRh\n3z8g1R5L90/U3NxbWY3fD8DwCAJvRMTZvR9WyI6kK82qbWQHk94YXyK9yTYhfVFU0era8I0O6ypL\nVuTzOEXS/aQr5iBN+tm0QlaVR1Z080PSXJD7gd9I2pTUd1PG/Kw99RPAKdkXetlgfQVpiN8sKm4m\nA4zuHgAAIuJBSZtVyK+lvZMjza0pNW9jJb5Jeq/cFhF3SXoXaaJUIVlN+jrguuz/M4m0QNrUiCjc\nT5fl1VgYcAnLF5Gr4kyaNPe2kF8rWv5+aBgOfQJTSO2NrW+4LP0cOKam3vuWqIX9XiX9gCZrwOTy\nKPVGUZoncCDpi/wJ0kJVpT6Ikn4FHBkVJvEVyHu1iCg8/FBpwbhPknZhmpNdsW4fEdeXyKP0YmhN\n8pgTEVv28Nh/R8QWJfOr1E8h6esRcWpP75tKXyzSO8v0X/WQx9tItbVJpBrXlaR9i+eXzKeONX/Q\n8i1F830LLU0GraqV74fuhkNNoPFi5BeVKrVBSM4GwGylDWpKr+EiaQ/S+P7uHXOlq4sRcWE2tI2I\nKDuSIb972FQqLEugtKvUpOznT6TRFWqhLbyl9XHUy0xd0mzkQiItGPcc8DHS1ekblLhKzdwuaftm\nV/Il3C3pHyKiy6iOrB19Vg/n9KiF/01jT+5Su8714veS7iONRru2bO1aaQXc7UgLOU6NiD+2UJa6\ndm6rpbm3Dtn3Q8v7QcMwqAnUqdXef6Ulgf+WdIVZ6YWVJNKX9pdIbzCRvqR+ULK9uZFfpSsVSW+R\n5hUcERH/naU9XnSkSZP8Wn1tvxBpolbTgFZmKKFysy0jYrykjUg7yfU621LSH0lfJKuRlhB4nAoL\n/GV5jSbVYP/M8i/9HUntzZ+Jkkt2Z/l9G9goInaXtC3w4YhYad9C2ZpUwbKI1IR4OGkZ8hnABRHx\nWMHz32J5M0dLs6lrHJm2KWkl0tVJzb3rAmc1Ph/9SU32gwYOjZL7QQMQEUP6hzTJ6xukDj5IH8w9\nBqgst5DWb28lj6+Sevw3z6W9i9S++pUK+d1TsRz7ANNJIzTOAXYBnmjxuY0mjdXeAxg1gO+Z+7IP\nzr25tAcKnvsiqabX9KdieXYmNVd8GfibFp7XtaSO2Puz+6uRLkgKv0dIFxt1v947k5YKWUxaTPHD\n/fz/PpB0YfVh0nDgCcCEEueP68/yFizTLNJFTOP+eGBWlbyGQ3PQ+aQXpLGZ+3xSx02RXaK6qKH3\n/+vANZJupfpCVQcDn4jcgmYR8Xg21PN6apilWkRE/Iq0LPfbSXs0HAeMUlqU7vIo0X4OIGk/0lDM\nTtIX8A8kfS0iCq2Hrq5LEDQrb5l261ZmWz4RNc1LaYi0L3Wz9vyyNoiIGZImZ/m+IanIiLn88MvK\nK892yTAt0Pd50vt5ISnAXQnsQPp8bl7H3ymo1ZFpvyKbMCfpFxFRdbBHnWrbD3o4BIHKG4Q00Wrv\n/7+Teu3XoPpCVSOiyYqWEfF80X+yui5ItpakxuiZ0lXpSLuA/RT4qdLMz31JSwuXCgKkYYIfjKxj\nOOvvuJHim2Lk28gr9XPktDLbctRK+iXKBvy6Lc2+fBvBbSeKLaHcF23Cd5D22dgnui5Tcrek/+qD\nv7cyrY5My3+fVGoO7QN3SzqXrvtBV+rTGQ5BoJUNQlYQrU322ihaHDFCah+u8tgyUeNa493yfRH4\nUfZT1irRdWTQC5ToVIvcqAdJx0WFURBavtn3NFITxUukVRhPjOKzLVclLVfSyuSlvvJV0tX2uyX9\nDtiQYkuobK00kUt0XUm0dD9HzlaRtVN0FxGnVMivFa3u3BY93B5Ite0HPRyCwBRW3CDksIp5tdr7\nf42kXcs2lXTzvtyVe15jU42h6jpJM1k+e3l/0siPKqp+EHva7LvMSJxnokIHfX+ItMzyX5MCmyg+\nYmSbusqg3HIPzSrkUW2r1Va1unNb4zMp0tLqlWvWrVK2xlSkGfzfp8SouB7z7CFYDymquEFIk3xa\n6v3PmmHeTnqjvc4AvEkGG2VLI0TE75R2gftY9tBi0lpI/1Mhz9JrF3U7v/Jm3wM1LrwIpeUEPs2K\n4+H7rYlK0vOsZLmHGIClF1odmTaY5N/7dfVPDPmagNLm7j8Frszar6vk0YiujQ6//yO1O5fSV80w\nQ9w0sua0SDua/RJA0vbZYyvbs3eZOvs5aG2z7yIbzwyUq0jv3QdpbTx8K+pc7qEWEXFrNny2sWDc\nndEHkxb7Se39E0O+JpBF+f1JV0B3kYY1/jq67YvbSx4tRVdJW0fEI0pb360gyi+SNmxoJas1Snow\nIrbvx7J03+z796SaY+kljgcjVVzxs69o+XIP3yVN+Co1y7zGcnQfmfaXQOGRaYNJt++qlmrEDUO+\nJhDL15lflTTk6x9Iu/WUuTJsNbp+lbTQ1veaPFZlkbTh5B0reWzNfitFUstm34PYta32SWWDLMbl\nhx9WyKP7cg9nkCbFDZRWR6YNJrX3Twz5IADL3rh7kmoEEyi/ymVLvf8RcWT2u+XlhYehWpdGaEVE\nfDIbPtzY7Pt4YDulVTsrbfY9yPweuFxps53SfVKS9gT+g9QntrmkHYBvlunMVb3LPdSlpZFpg0lE\nlNo1sYjh0Bw0g7RR83Wk9W1ujYhS7aHZhJqlZNEVeKXxEOWnqLe0H8FwU/fSCDWWa2PSSLKPkGYw\nvzMiVlZrGfQkPUGa2Fdp2RJJs0i11s5G53fZJrs6l3uoi9LKqu+l68i0ByPi6/1dlsFoONQEzgMm\nZeP6K6kruqr1/QiGnYhYCHxE0s6kK0SAqyPi5v4ui+rf7HuweRr4Y5UAkHk9IpZ0G9pZKq+IGHRX\n2BHxtW4j034UEQPZPDWoDNmaQPZP7VE2EqVfSXqY1vYjsD4k6ftkcwNiECwXXjdJF5D6tK6lwrIl\nks4jbSRzAmkfjGNIM9i/WHthB1DWXDYpIi4Z6LIMBkO5JtAYWjiKdGXXuLLcmXR11+9BgDQzcQzV\ntga0Phb1b/Y92DyR/axOtWVLvkzqRH2NNOx6JlBqx7XBRNI6pFm1Y0kzqW/I7v8TaVMiBwGGcE2g\nQWlnnUMaV3ZKG4RcEBG7DUBZbiEtkJXfjyAiYu/+LotZu5N0BWnV1ztI8ztGkfomjo2I+waybIPJ\ncAgCD0fENrn7qwCzI2LrAShLfmZiYzzyARHxnv4ui7Wf7CKk2c5ghYYoZxdU+0bE4uz+esD0gbig\nqkO+UzsbQv4Mafhr4TlE7WAoNwc13NRkTZpW1u6pLJuZ2H0rxv5eMdHa1z/lbq9Batcvs1nMBo0A\nAGnBQEmj6ircAFi2blKkzdnnOQCsaMgHgYj4kqTPAH+VJd1OapfvN6p/K0az0iKi+7yL3yltlVrU\nW40lVGDZWlpDuakgvxhjfnJV26/plTfkg0DmSVLn8LKN0Pv57z9CWsp1j1i+FeNX+rkM1uYkrZ+7\nuwrwAdLaSEX9P+A2pU2RGs2ZR9ZXwv7VFxOrhqMhGwQG2dX335I2o7lF0nWk9YsG43rzNrzNIl25\nN/alfgI4oujJEXFdtv7VTlnScVVX5LWhY8h2DKvmjdBrKlNjK8ZJpJmXF1FhK0azgSJpLGm/5PyM\n9/Kbl9uQMZSDwD6kq++PkpaMmA6cGxH9uXdpj7R8K8b9I2IwLz9sw4Sko0l7NORH90yKiEI7Tkk6\nhTSw4iFye/GWWTvIhp4hGwQafPVtlki6LyJ26JZWeBMcSY8C7420a5W1iUG3zkdZEbE0In4aEXuS\ntg+8l7QRulm7WVW5hX+ysfFlZg4/DoyovVQ2qA35moCZJdlqmZsCP8ySvgA8HRHHFzz/F8D7SOsH\n5dceOqbHk2zIcxAwGyay2fJfYPkWmDeQ+skKrbAr6dBm6RFRdn8OG0IcBMyGEUmrA1uRhoo+GhGv\n93JK9/Nb3lnMhpYh3ydgZomkDtLWmWcCZwGPSfqrlZ7U9fw9SXthXJfd30HSlX1QVBtEXBMwGyay\nncEObFzFZxMqfxYRHyhxfvedxf4YEdut/EwbylwTMBs+RuSbcSLiMcqN9nk9IpZ0Syu1VasNPUN2\n2QgzW8Hdks4FfpLdPwi4u8T5D0k6kDTUdEvSzmK311xGG2TcHGQ2TEh6G2nnrMZeur8Fzio6+UvS\nWqRF5HYlrT80E/g3L788vDkImA0jkjYEiIjnB7osNjQ4CJgNcdks4ZOAL7G8n+9N4AcR8c0C50+L\niOMkXUXzncm8dtAw5j4Bs6HvK6SFFD8YEU8ASHoXcLakr0TEab2cf3H2+z/6sIw2SLkmYDbESboX\n+ET3tf+zpqHrSywg93bg1Yh4K7u/KvC2iHil7jLb4OEhomZD34hmm79k/QJlhojeBKyVu78mcGOL\nZbNBzkHAbOj7c8XHulsjIv63cSe7vdZKjrdhwH0CZkNffkP1PAFrlMhnqaQJEXEPgKQPAK/WUUAb\nvNwnYGYASPogaYe+BaQAMoa0M96sAS2Y9SkHATNbRtII0iqkUGEVUht63Cdg1uYkfVDSGIDsS38C\n8O/A9yStP6CFsz7nIGBmPyTrQM6Wnv4Oaa/uJcCPBrBc1g/cMWxmq0bEouz2/sCPIuIXwC8k3TeA\n5bJ+4JqAma0qqXFBuAtwc+4xXygOc/4Hm9nPgFsl/Yk0JPS3AJK2IDUJ2TDm0UFmhqSdgL8gLTOx\nNEsbD6zdmDdgw5ODgJlZG3OfgJlZG3MQMDNrYw4CZmZtzEHAzKyN/X/2sm3XqOC+ZwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12f2f8d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# generate proportions\n",
    "genre_prop = pd.DataFrame(genres.sum(axis=0), columns=['Yes'])\n",
    "genre_prop['No'] = genres.shape[0] - genre_prop['Yes']\n",
    "genre_prop = genre_prop.iloc[1:20]\n",
    "\n",
    "# generate plot\n",
    "genre_prop.plot(kind='bar', stacked=True, color=['red', 'navy'], alpha=0.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x16b05cf8>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEPCAYAAAC3NDh4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH/pJREFUeJzt3X20HXV97/H3J/KgPHqxXEyCSYUg3FoUjQRoKGxFIfgA\nFq0leqtLW25ulcAqXa7Y9lqMtVVpbVG4ItGAeEWjoIEoCEFla1GUEJ5EE4jylEBELJQnRfPwuX/M\nnDA5njlnTnIme5/wea211575ze83+7v3SfZ3z/xmfj/ZJiIiYigTeh1ARET0rySJiIiolSQRERG1\nkiQiIqJWkkRERNRKkoiIiFqtJwlJsyStlHSnpHnD1DtU0jpJJ1XK7pF0q6SbJd3QdqwREbG5Hdrc\nuaQJwLnAMcADwDJJl9teOUS9jwBXD9rFRqBj+5E244yIiKG1fSQxA1hl+17b64BFwIlD1JsLXAr8\nYlC5yCmxiIieafsLeDKwurK+pizbRNIk4I22z6NIClUGrpG0TNIprUYaERG/o9XTTQ2dDVT7KqqJ\nYqbttZL2pkgWK2xft23Di4h45mo7SdwPTKms71uWVb0CWCRJwO8Bx0taZ3uJ7bUAth+StJji9NXv\nJAlJGYAqImKUbA8+e/M72j7dtAyYJmmqpJ2Ak4El1Qq29ysfL6Tol3i37SWSdpG0G4CkXYFjgdvr\nXsj2uHyceeaZPY8h8fc+jsQ/Ph/jOf6mWj2SsL1B0qnAUoqEtND2Cklzis1eMLhJZXkfYHF5lLAD\ncLHtpW3GGxERm2u9T8L2VcCBg8rOr6n7rsry3cAh7UYXERHDyeWlPdbpdHodwlZJ/L2V+HtrvMff\nhEZzbqpfSfL28D4iIrYVSbgPOq4jImIcS5KIiIhaSRIREVErSSIiImolSURERK0kiYiIqJUkERER\ntfphFNgxcfCRB/c6hC0yZeIUrrjkil6HERExpO0mSRxxwRG9DmGLXP+u63sdQkRErZxuioiIWkkS\nERFRK0kiIiJqJUlEREStJImIiKjVepKQNEvSSkl3Spo3TL1DJa2TdNJo20ZERDtaTRKSJgDnAscB\nLwZmSzqopt5HgKtH2zYiItrT9pHEDGCV7XttrwMWAScOUW8ucCnwiy1oGxERLWk7SUwGVlfW15Rl\nm0iaBLzR9nmARtM2IiLa1Q93XJ8NbHV/w/Jzlm9anjhjIpMOm7S1u4yI2G50u1263e6o27WdJO4H\nplTW9y3Lql4BLJIk4PeA4yWtb9h2k+lzp49JwBER26NOp0On09m0Pn/+/Ebt2k4Sy4BpkqYCa4GT\ngdnVCrb3G1iWdCHwNdtLJD1rpLYREdGuVpOE7Q2STgWWUvR/LLS9QtKcYrMXDG4yUts2442IiM21\n3idh+yrgwEFl59fUfddIbSMiYtvJHdcREVErSSIiImolSURERK0kiYiIqJUkERERtZIkIiKiVpJE\nRETUSpKIiIhaSRIREVErSSIiImolSURERK0kiYiIqJUkERERtZIkIiKiVpJERETUSpKIiIharScJ\nSbMkrZR0p6R5Q2w/QdKtkm6WdIOkmZVt91S3tR1rRERsrtWZ6SRNAM4FjgEeAJZJutz2ykq1b9pe\nUtY/GPgy8D/KbRuBju1H2owzIiKG1vaRxAxgle17ba8DFgEnVivY/lVldTeKxDBA2yDGiIio0fYX\n8GRgdWV9TVm2GUlvlLQC+BpQnefawDWSlkk6pdVIIyLid7R6uqkp25cBl0k6EvgQ8Jpy00zbayXt\nTZEsVti+bqh9LD9n+abliTMmMumwSW2HHRExbnS7Xbrd7qjbtZ0k7gemVNb3LcuGZPs6SftJ2sv2\nw7bXluUPSVpMcfpqyCQxfe70MQw7ImL70ul06HQ6m9bnz5/fqF3bp5uWAdMkTZW0E3AysKRaQdL+\nleWXAzvZfljSLpJ2K8t3BY4Fbm853oiIqGj1SML2BkmnAkspEtJC2yskzSk2ewHwJklvB34L/Bp4\nS9l8H2CxJJdxXmx7aZvxRkTE5lrvk7B9FXDgoLLzK8tnAWcN0e5u4JC244uIiHq5vDQiImolSURE\nRK0kiYiIqJUkERERtZIkIiKiVpJERETU6othOZ7JVt+3moOPPLjXYWyxKROncMUlV/Q6jIhoSZJE\nj633eo644Iheh7HFrn/X9b0OISJaNOLpJkl/Kmn3cvn/SPpqOXxGRERs55r0Sbzf9uPlCK2vBhYC\n57UbVkRE9IMmSWJD+fw6YIHtK4Cd2gspIiL6RZMkcb+k84E/A66UtHPDdhERMc41+bJ/C3A1cJzt\n/wL2At7balQREdEXRkwS5RzUvwCOLIvWA6vaDCoiIvpDk6ubzgTmAX9bFu0IfL7NoCIioj80Od30\nJ8AJwJMAth8Adm8zqIiI6A9NksRvbRswbJpKtDFJsyStlHSnpHlDbD9B0q2SbpZ0g6SZTdtGRES7\nmiSJL5dXNz1X0inAN4FPN9m5pAnAucBxwIuB2ZIOGlTtm7ZfavtlwF8AnxlF24iIaNGIw3LY/ldJ\nrwEeo5iG9B9sX9Nw/zOAVbbvBZC0CDgRWFnZ/68q9XcDNjZtGxER7Wo0dlOZFJomhqrJwOrK+hqK\nL//NSHoj8GFgb4qb9hq3jYiI9tQmCUnX2T5S0uOU/REDmwDb3mOsgrB9GXBZOfTHh4DXjHYfy89Z\nvml54oyJTDps0liFFxEx7nW7Xbrd7qjb1SYJ20eWz1tzJdP9wJTK+r5lWd1rXidpP0l7jbbt9LnT\ntyLMiIjtW6fTodPpbFqfP39+o3ZN7pP4hKQtHct6GTBN0lRJOwEnA0sG7X//yvLLgZ1sP9ykbURE\ntKtJn8Ry4P2SDgQWA4ts39hk57Y3SDoVWEqRkBbaXiFpTrHZC4A3SXo78Fvg1xTDgNS2HeX7i4iI\nrdDk6qaLgIvKU0BvAj4qaYrtA5q8gO2rKK6KqpadX1k+CziraduIiNh2RjOa6zTgIGAquQw1IuIZ\noUmfxFmSVgEfBG4HXmH7Da1HFhERPdekT+JnwBG2f9l2MBER0V+a9EmcX46vdFRZ9B3bX2s5roiI\n6ANNTjd9GDgd+En5OE3SP7cdWERE9F6T002vAw6xvRFA0kXAzcDftRlYRET0XtOrm55bWd6zjUAi\nIqL/NDmS+DBws6RrKcZtOgp4X6tRRUREX2jScf1FSV3g0LJonu2ftxpVRET0heFGgX35oKI15fMk\nSZNs39ReWBER0Q+GO5K4keLmuYH7I1TZZuBVbQUVERH9YbgkcQbwZopB9xYBi20/sU2iioiIvlB7\ndZPts8s5JeYCLwC+JenLkg7ZZtFFRERPjXgJrO27gMsphuyeAbyo7aAiIqI/DNdxvR/FRD8nUsw1\nvQj4Z9u/3kaxRUREjw3XJ/FT4DaKo4jHKKYS/Sup6L+2/W9NXkDSLOBsnp446KODtr8VmFeuPg68\n2/Zt5bZ7gEeBjcA62zMavauIiBgTwyWJD1JcxQSw25bsXNIE4FzgGOABYJmky21X56O4CzjK9qNl\nQlkAHF5u2wh0bD+yJa8fERFbpzZJ2P7AGOx/BrDK9r0AkhZRnL7alCRs/6BS/wfA5Mq6GN3ESBER\nMYba/gKeTNGfMWANmyeBwf4S+EZl3cA1kpZJOqWF+CIiYhhNxm7aJiS9EngncGSleKbttZL2pkgW\nK2xf15sIIyKeeYa7uul02x+XNNP297Zw//dTdHgP2LcsG/xaL6Hoi5hV7X+wvbZ8fkjSYorTV0Mm\nieXnLN+0PHHGRCYdNmkLQ46I2P50u1263e6o2w13JPFO4OPAOcDgcZyaWgZMkzQVWEtxSe3sagVJ\nU4CvAH9u+2eV8l2ACbafkLQrcCwwv+6Fps+dvoUhRkRs/zqdDp1OZ9P6/Pm1X6ebGS5JrJC0imJA\nv9sq5QJs+yUj7dz2BkmnUtyIN3AJ7ApJc8p9LADeD+wFfFLF9bUDl7ruAyyW5DLOi20vbfSuIiJi\nTAx3ddNsSc8HrgZO2NIXsH0VcOCgsvMry6cAv9MpbftuIEOARET00LAd1+W8ES+VtBNPD8dxh+11\nrUcWERE9N+LVTZKOBj4H3ENxqukFkt5h+7stxxYRET3W5BLYfwOOtX0HgKQXAV8E0lMcEbGda3Iz\n3Y4DCQLA9p3Aju2FFBER/aLJkcSNkj4DfL5cfxvFrHUREbGda5Ik/gp4D3Bauf4fwCdbiygiIvrG\niEnC9m8o+iUaDQ0eERHbj4ywGhERtZIkIiKiVpJERETUanIz3YuA9wJTq/Vtv6rFuCIiog80ubrp\nEuBTwKeBDe2GExER/aRJklhv+7zWI4mIiL7TpE/ia5LeLWmipL0GHq1HFhERPdfkSOId5fN7K2UG\n9hv7cCIiop80uZnuhdsikIiI6D8jnm6StKOk0yRdWj5OldR4gD9JsyStlHSnpHlDbH+rpFvLx3Xl\nfNeN2kZERLua9EmcRzEs+CfLx/SybESSJgDnAscBLwZmSzpoULW7gKNsvxT4ELBgFG0jIqJFTfok\nDi2/wAd8W9KtDfc/A1hl+14ASYuAE4GVAxVs/6BS/wfA5KZtIyKiXU2OJDZI2n9gRdJ+NL9fYjKw\nurK+hqeTwFD+EvjGFraNiIgx1uRI4r3AtZLuopi+dCrwzrEORNIry/0eOdb7joiILdPk6qZvSToA\nOLAsuqMcPryJ+4EplfV9y7LNlJ3VC4BZth8ZTdsBy89Zvml54oyJTDpsUsMQIyK2f91ul263O+p2\ntUlC0qtsf1vSSYM2TZOE7a822P+ysv5UYC1wMjB70OtMAb4C/Lntn42mbdX0uZlyOyKiTqfTodPp\nbFqfP39+o3bDHUkcDXwbeMMQ2wyMmCRsb5B0KrCUov9joe0VkuYUm70AeD+wF/BJSQLW2Z5R17bR\nu4qIiDFRmyRsn1kuftD23dVtkhrfYGf7Kp4+VTVQdn5l+RTglKZtIyJi22lyddNXhii7dKwDiYiI\n/jNcn8RBFDex7TmoX2IP4NltBxYREb03XJ/EgcDrgeeyeb/E49ScHoqIiO3LcH0Sl0v6OjDP9j9v\nw5giIqJPDNsnYXsD8MZtFEtERPSZJndcf0/SucCXgCcHCm3f1FpUERHRF5okiUPK5w9Wygy8auzD\niYiIftJkWI5XbotAYnxafd9qDj7y4F6HscWmTJzCFZdc0eswIvrWiElC0p7AmcBRZdF3KG6we7TN\nwGJ8WO/1HHHBEb0OY4td/67rex1CRF9rcjPdBRSXvb6lfDwGXNhmUBER0R+a9Ensb/tNlfX5km5p\nK6CIiOgfTY4kfi1p0xwPkmYCv24vpIiI6BdNjiT+Crio7JsQ8DDwjlajioiIvtDk6qZbgJdK2qNc\nf6z1qCIioi+MeLpJ0vMkfQLoUkxj+nFJz2s9soiI6LkmfRKLgIeANwFvLpe/1GZQERHRH5okiYm2\n/9H23eXjQ8A+TV9A0ixJKyXdKWneENsPlPR9SU9JOmPQtnsk3SrpZkk3NH3NiIgYG006rpdKOhn4\ncrn+ZuDqJjuXNAE4FzgGeABYJuly2ysr1f4TmMvQAwluBDq2H2nyehERMbaaHEmcAnwB+G35WATM\nkfS4pJE6sWcAq2zfa3td2fbEagXbv7S9HFg/RHs1jDEiIlow4hew7d1tT7C9Q/mYUJbtbnuPEZpP\nBlZX1teUZU0ZuEbSMkmZ6CgiYhtrcroJSSfw9NhNXdtfby+kzcy0vVbS3hTJYoXt64aquPyc5ZuW\nJ86YyKTDJm2jECMi+l+326Xb7Y66XZMB/j4CHApcXBadLmmm7b9tsP/7gSmV9X3LskZsry2fH5K0\nmOL01ZBJYvrc6U13GxHxjNPpdOh0OpvW58+f36hdk/P9rwVeY/sC2xcAs4DXNYxrGTBN0lRJOwEn\nA0uGqa9NC9IuknYrl3cFjgVub/i6ERExBhqdbgKeSzEcB8CeTXdue4OkU4GlFAlpoe0VkuYUm71A\n0j7AjcDuwEZJpwN/AOwNLJbkMs6LbS9t+toREbH1miSJDwM3S7qW4pf+UcD7mr6A7auAAweVnV9Z\nfhB4wRBNn+DpWfEiIqIHhk0SkkTRB3A4Rb8EwDzbP287sIiI6L1hk4RtS7rS9sEM35cQERHboSYd\n1zdJOnTkahERsb1p0idxGPA/Jd0DPEnRL2HbL2kzsIiI6L0mSeK41qOIiIi+VJskJD0b+N/ANOBH\nFJevDjW+UkREbKeG65O4CHgFRYI4HvjYNokoIiL6xnCnm/6gvKoJSQuBzOcQEfEMM9yRxLqBhZxm\nioh4ZhruSOKllfkiBDynXB+4ummkYcIjImKcq00Stp+1LQOJiIj+k1nfIiKiVpJERETUSpKIiIha\nSRIREVErSSIiImq1niQkzZK0UtKdkuYNsf1ASd+X9JSkM0bTNiIi2tVqkpA0ATiXYpDAFwOzJR00\nqNp/AnOBf9mCthER0aK2jyRmAKts32t7HbAIOLFawfYvbS8HBt/VPWLbiIhoV9tJYjKwurK+pixr\nu21ERIyBJvNJjAvLz1m+aXnijIlMOmxSD6OJiOgv3W6Xbrc76nZtJ4n7gSmV9X3LsjFvO33u9FEH\nFxHxTNHpdOh0OpvW58+f36hd26eblgHTJE2VtBNwMrBkmPrairYRETHGWj2SsL1B0qnAUoqEtND2\nCklzis1eIGkf4EZgd2CjpNMp5rJ4Yqi2bcYbERGba71PwvZVwIGDys6vLD8IvKBp24iI2HZyx3VE\nRNRKkoiIiFpJEhERUStJIiIiaiVJRERErSSJiIiolSQRERG1kiQiIqJWkkRERNRKkoiIiFpJEhER\nUStJIiIiaiVJRERErSSJiIiolSQRERG1kiQiIqJW60lC0ixJKyXdKWleTZ1PSFol6RZJL6uU3yPp\nVkk3S7qh7VgjImJzrc5MJ2kCcC5wDPAAsEzS5bZXVuocD+xv+wBJhwHnAYeXmzcCHduPtBlnREQM\nre3pS2cAq2zfCyBpEXAisLJS50TgcwC2fyhpT0n7lNOaipwSixatvm81Bx95cK/D2GJTJk7hikuu\n6HUYsR1rO0lMBlZX1tdQJI7h6txflj0IGLhG0gZgge1PtxhrPAOt93qOuOCIXoexxa5/1/W9DiG2\nc20nia010/ZaSXtTJIsVtq8bquLyc5ZvWp44YyKTDpu0rWKMiOh73W6Xbrc76nZtJ4n7gSmV9X3L\nssF1XjBUHdtry+eHJC2mOAoZMklMnzt9jEKOiNj+dDodOp3OpvX58+c3atf2+f5lwDRJUyXtBJwM\nLBlUZwnwdgBJhwP/ZftBSbtI2q0s3xU4Fri95XgjIqKi1SMJ2xsknQospUhIC22vkDSn2OwFtq+U\n9FpJPwWeBN5ZNt8HWCzJZZwX217aZrwREbG51vskbF8FHDio7PxB66cO0e5u4JB2o4uIiOHk8tKI\niKiVJBEREbWSJCIiolaSRERE1EqSiIiIWkkSERFRK0kiIiJqJUlEREStJImIiKiVJBEREbWSJCIi\nolaSRERE1EqSiIiIWv0+M11EDCNzdEfbkiQixrHM0R1ta/10k6RZklZKulPSvJo6n5C0StItkg4Z\nTduIiGhPq0lC0gTgXOA44MXAbEkHDapzPLC/7QOAOcCnmrbdHmx4akOvQ9gqib+3xnv8Tzz6RK9D\n2CrdbrfXIbSu7SOJGcAq2/faXgcsAk4cVOdE4HMAtn8I7Clpn4Ztx70Nvxnf/8kTf2+N9/iffPTJ\nXoewVZ4JSaLtPonJwOrK+hqKL/+R6kxu2DYixrHHHn1sXHe8P/XwU3zgAx/odRit6seOa/U6gIjY\nNjZ647jueL/s1Zf1OoTWyXZ7O5cOBz5ge1a5/j7Atj9aqfMp4FrbXyrXVwJHAy8cqW1lH+29iYiI\n7ZTtEX+Ut30ksQyYJmkqsBY4GZg9qM4S4D3Al8qk8l+2H5T0ywZtgWZvNCIiRq/VJGF7g6RTgaUU\nneQLba+QNKfY7AW2r5T0Wkk/BZ4E3jlc2zbjjYiIzbV6uikiIsa3cT12k6SFkh6UdFuvYxktSftK\n+rakH0v6kaTTeh3TaEjaWdIPJd1cxn9mr2MaLUkTJN0kaUmvY9kSku6RdGv5N7ih1/GMhqQ9JV0i\naUX5f+CwXsfUlKQXlZ/5TeXzo+Pw/+9fS7pd0m2SLpa0U23d8XwkIelI4Angc7Zf0ut4RkPS84Hn\n275F0m7AcuBE2yt7HFpjknax/StJzwK+B5xme9x8WUn6a2A6sIftE3odz2hJuguYbvuRXscyWpI+\nC3zH9oWSdgB2sf1Yj8MatfKm3zXAYbZXj1S/H0iaBFwHHGT7t5K+BFxh+3ND1R/XRxK2rwPG3X8Q\nANs/t31LufwEsILi3pBxw/avysWdKfq3xs0vDkn7Aq8FPtPrWLaCGIf/hyXtAfyx7QsBbK8fjwmi\n9GrgZ+MlQVQ8C9h1IEEDD9RVHHf/wLZHkn4fOAT4YW8jGZ3ydM3NwM+Ba2wv63VMo/DvwHsZR4lt\nCAaukbRM0im9DmYUXgj8UtKF5SmbBZKe0+ugttCfAV/sdRCjYfsB4GPAfcD9FFeUfrOufpJEj5Wn\nmi4FTi+PKMYN2xttvwzYFzhM0h/0OqYmJL0OeLA8khPj9wbOmbZfTnFE9J7y9Ot4sAPwcuD/lvH/\nCnhfb0MaPUk7AicAl/Q6ltGQ9FyKIY6mApOA3SS9ta5+kkQPlYd6lwL/z/blvY5nS5WnCq4FZvU6\nloZmAieU5/S/CLxS0pDnY/uZ7bXl80PAYsbPsDVrgNW2byzXL6VIGuPN8cDy8vMfT14N3GX7Ydsb\ngK8Cf1RXeXtIEuP5l+AFwE9sf7zXgYyWpN+TtGe5/BzgNcC46HS3/Xe2p9jej+ImzW/bfnuv4xoN\nSbuUR6FI2hU4Fri9t1E1Y/tBYLWkF5VFxwA/6WFIW2o24+xUU+k+4HBJz5Ykis+/9h60fhy7qTFJ\nXwA6wPMk3QecOdAZ1u8kzQTeBvyoPK9v4O9sX9XbyBqbCFxUXt0xAfiS7St7HNMzyT7A4nJImh2A\ni20v7XFMo3EacHF5yuYuyptoxwtJu1D8Iv9fvY5ltGzfIOlS4GZgXfm8oK7+uL4ENiIi2rU9nG6K\niIiWJElEREStJImIiKiVJBEREbWSJCIiolaSRERE1EqSiDEjaaOkf6ms/42kfxijfV8o6aSx2NcI\nr/NmST+R9K0htk2T9DVJq8rxkr7VT0NhSDpb0pqtaP/1cvC91myrv2OMnSSJGEu/AU6StFevA6kq\nhzJv6i+Av7R9zKB97AxcAXzK9gG2DwXmAvuNUYxb9X+xvHP2jcB9ko7ekn3Yfv04Ho01WpIkEWNp\nPcWdm2cM3jD4F6Skx8vnoyV1JV0m6aeSPizpreWERrdKemFlN68pf8GvLAfpGxiJ9qyy/i0Do6GW\n+/2upMuBHw8Rz+xywpXbJH24LHs/cCSwUNJHBzV5G/B921cMFNj+ycAY/OUwGQsl/UDScklvKMvf\nIekrkr4h6Y7qfiU9LulfyzvuD5f08vKzWFbW36esd5qKiXluKUcZGEqHYliO84BNg7VJOkpPT5Cz\nXNKukp4v6Ttl2W3l3f9IunsgwUt6f/k5f1fSFySdUZZfK+kj5ee9stJ2yL9Due1cFZMLLQX+e038\n0a9s55HHmDyAx4DdgLuB3YG/Af6h3HYhcFK1bvl8NPAwxZfHThSDv51ZbjsN+LdK+yvL5WnA6rL+\nKRTDmVCuL6MY3fJo4HFgyhBxTgTuBfai+KH0LeCEctu1wMuGaPMxYO4w7/2fgLeWy3sCdwDPAd4B\n/LT8XHYG7gEml/U2Am8ql3egmLjpeeX6WyjmdYdiOOcdy+U9al5/AUVy2L38bJ5Vli8BjiiXd6GY\nR+AM4G/LMgG7lst3lZ/JK4CbgB3LuO8Ezqh8Pv9SLh9PMUQ8w/wd/gS4uvK5P1L9d5BH/z9yJBFj\nysVw5xcBp4+i2TLbv7D9W+BnwMAYRD8Cfr9S78vla/y0rHcQxcB2by9/jf+Q4kvugLL+DbbvG+L1\nDgWudTEK5kbgYuCoyvYRB4yU9FUV07ZeWhYdC7yvjKNL8UU5pdz2LdtP2P4NxUB2U8vy9RQjcAIc\nCPwhxfwQNwN/TzGMM8CtwBckvQ3YMEQsO1IMF3657ceBG4Djys3fA/5d0lzgv7kY9XMZ8M6yv+gl\ntp8ctMuZ5b7WlX/Prw3aPhDz8sp7qfs7HEU5CJ6LUWu/PTj+6G/jeoC/6Fsfp/glWh1scT3l6c3y\n/Hl1Tt3fVJY3VtY3svm/0epAYyrXRfEL/5pqAOV5+cFffptVGfFdbO7HVBKJ7ZMkTQcGOupFcVSw\nalAch7P5+9vA0+/pKZc/scv2t9ueOcRrv6587ROAv5f0h2VyG3AcxdHLj8rP9jkUczRcafujkr5e\n7uN7ko61/R+SjirLPivpY7Y/P4rPYuD9VN9L3d/hdaPYb/ShHEnEWBKAizmXv0zRCTzgHorTGFBM\neLLjFuz/T1XYn2J2szuAq4F3q5ibA0kHqBihczg3AEdJ2qvs1J5N8et/OF8A/kjS6ytlu1aWr6Y4\nPUYZxyEN3k81Ud0B7F0mFSTtoKcncZpi+zsUE/PsQXEKqGo28Be297P9QorO9GNVDAW9n+0f2z6L\n4gjiIElTgF/YXkgxfevAXA4D8XwPeIOknVUMR/566g20qfs7fBf4s7LPYiLwygafS/SRHEnEWKr+\n0v8Y8J5K2aeBy8vTEVdT/yt/uGGJ76P4gt8dmONiEvfPUJySuqn8Ff0Liqt86oO0fy7pfTydGL5u\n++vDvb7tp8oE8e+SzgYepOjz+FBZ5R+BsyXdRvHj6y6KX/7Dvb9Ny7bXSXozcI6KeTqeVe7vTuDz\nKi5NFfBxV65AUjGXx3HAnMq+fiXpP4A3AH8s6ZUUR2W3A9+gSCrvlbSufA9/Xo3H9o2SllCc5noQ\nuA14tObzGVgf8u9ge7GkV1Ecid0HfH+IzyT6WIYKj4jfIWlX20+WSei7wCkupnuNZ5gcSUTEUBaU\np7t2Bj6bBPHMlSOJiIiolY7riIiolSQRERG1kiQiIqJWkkRERNRKkoiIiFpJEhERUev/AwzFe7wY\nFt/AAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12fbddd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(genres['num_genres'], normed=1, facecolor='limegreen', alpha=0.75)\n",
    "plt.xlim([1,8])\n",
    "plt.xlabel('Number of Genres Assigned')\n",
    "plt.ylabel('Proportion of Movies')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plots above, we can see that there is very imbalanced data, since for all genres, there are a lot more movies classified \"No\" than \"Yes\". In addition, there are many more movies tagged with 1-3 genres than >=4 genres. To address this issue, we will employ **class weighting** in our models.\n",
    "\n",
    "Alternatively, we can downsample the \"No\" class, for example, if there are only 5000 Adventure movies, we will randomly select an equal number of non-Adventure movies, to train the classifer for the Adventure genre. We can also upsample the \"Yes\" class, for example, by applying bootstrapping on the 5000 Adventure movies so that the sample size matches the number of non-Adventure movies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Moving forward, we will split the dataset into training and testing sets. We will train 19 separate classification models for the 19 genres, with traditional machine learning methods, such as random forest and support vector machine. We will also train a regression model for the number of genres assigned. We will evaluate their performance on the test set and explain what we want to use as our performance metrics."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}