Week6_Assignment/Problem_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "c91f96b4a4071e98255e6da0915b874c",
     "grade": false,
     "grade_id": "title",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "# Week 6 Problem 3\n",
    "\n",
    "If you are not using the `Assignments` tab on the course JupyterHub server to read this notebook, read [Activating the assignments tab](https://github.com/UI-DataScience/info490-fa16/blob/master/Week2/assignments/README.md).\n",
    "\n",
    "A few things you should keep in mind when working on assignments:\n",
    "\n",
    "1. Make sure you fill in any place that says `YOUR CODE HERE`. Do **not** write your answer in anywhere else other than where it says `YOUR CODE HERE`. Anything you write anywhere else will be removed or overwritten by the autograder.\n",
    "\n",
    "2. Before you submit your assignment, make sure everything runs as expected. Go to menubar, select _Kernel_, and restart the kernel and run all cells (_Restart & Run all_).\n",
    "\n",
    "3. Do not change the title (i.e. file name) of this notebook.\n",
    "\n",
    "4. Make sure that you save your work (in the menubar, select _File_ → _Save and CheckPoint_)\n",
    "\n",
    "5. You are allowed to submit an assignment multiple times, but only the most recent submission will be graded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "0d27ce963d5e0654f1991d291ec7cb97",
     "grade": false,
     "grade_id": "import",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from nose.tools import assert_equal\n",
    "!mkdir -p ~/w6p3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "d88401e7818c54030a834d64d5305686",
     "grade": false,
     "grade_id": "p1_desc",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "# Problem 1. \n",
    "\n",
    "In this problem, we are going to do basic text processing on two csv files: \n",
    "\n",
    "`~/data/ml-latest-small/movies.csv`\n",
    "\n",
    "`~/data/ml-latest-small/ratings.csv`\n",
    "\n",
    "These two files include 105339 ratings for 10329 movies from [MovieLens](http://movielens.org), a movie recommendation service.\n",
    "\n",
    "First, let's have a look at the data structure in `movies.csv` and `ratings.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "0bcd5dd63a9b8119070d90e5e41b1b27",
     "grade": false,
     "grade_id": "head",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movieId,title,genres\r",
      "\r\n",
      "1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy\r",
      "\r\n",
      "2,Jumanji (1995),Adventure|Children|Fantasy\r",
      "\r\n",
      "3,Grumpier Old Men (1995),Comedy|Romance\r",
      "\r\n",
      "4,Waiting to Exhale (1995),Comedy|Drama|Romance\r",
      "\r\n",
      "5,Father of the Bride Part II (1995),Comedy\r",
      "\r\n",
      "6,Heat (1995),Action|Crime|Thriller\r",
      "\r\n",
      "7,Sabrina (1995),Comedy|Romance\r",
      "\r\n",
      "8,Tom and Huck (1995),Adventure|Children\r",
      "\r\n",
      "9,Sudden Death (1995),Action\r",
      "\r\n",
      "10,GoldenEye (1995),Action|Adventure|Thriller\r",
      "\r\n",
      "11,\"American President, The (1995)\",Comedy|Drama|Romance\r",
      "\r\n",
      "12,Dracula: Dead and Loving It (1995),Comedy|Horror\r",
      "\r\n",
      "13,Balto (1995),Adventure|Animation|Children\r",
      "\r\n",
      "14,Nixon (1995),Drama\r",
      "\r\n",
      "15,Cutthroat Island (1995),Action|Adventure|Romance\r",
      "\r\n",
      "16,Casino (1995),Crime|Drama\r",
      "\r\n",
      "17,Sense and Sensibility (1995),Drama|Romance\r",
      "\r\n",
      "18,Four Rooms (1995),Comedy\r",
      "\r\n",
      "19,Ace Ventura: When Nature Calls (1995),Comedy\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -20 ~/data/ml-latest-small/movies.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "97f8f746d8b5d29e6808a24f93fd2a5d",
     "grade": false,
     "grade_id": "tail",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "userId,movieId,rating,timestamp\r",
      "\r\n",
      "1,16,4.0,1217897793\r",
      "\r\n",
      "1,24,1.5,1217895807\r",
      "\r\n",
      "1,32,4.0,1217896246\r",
      "\r\n",
      "1,47,4.0,1217896556\r",
      "\r\n",
      "1,50,4.0,1217896523\r",
      "\r\n",
      "1,110,4.0,1217896150\r",
      "\r\n",
      "1,150,3.0,1217895940\r",
      "\r\n",
      "1,161,4.0,1217897864\r",
      "\r\n",
      "1,165,3.0,1217897135\r",
      "\r\n",
      "1,204,0.5,1217895786\r",
      "\r\n",
      "1,223,4.0,1217897795\r",
      "\r\n",
      "1,256,0.5,1217895764\r",
      "\r\n",
      "1,260,4.5,1217895864\r",
      "\r\n",
      "1,261,1.5,1217895750\r",
      "\r\n",
      "1,277,0.5,1217895772\r",
      "\r\n",
      "1,296,4.0,1217896125\r",
      "\r\n",
      "1,318,4.0,1217895860\r",
      "\r\n",
      "1,349,4.5,1217897058\r",
      "\r\n",
      "1,356,3.0,1217896231\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -20  ~/data/ml-latest-small/ratings.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "5258427f3bd181304ea1d5dd80bae872",
     "grade": false,
     "grade_id": "sort",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "The header in each csv file already tells us a lot. In `movies.csv`, each line represents a movie with its ID, title, and genres. In `ratings.csv`, each line represents a rating record from a user for a movie. The last column, `timestamp`, is related to the time that the user created the movie rating. For further information, you can have a look at [README.txt](https://info490rb.studentspace.cs.illinois.edu/user/xlu28/edit/data/ml-latest-small/README.txt).\n",
    "\n",
    "Our goal for this problem is to match the rating records from selected users (`userId 22` and `66`) in `ratings.csv` to the movies in `movies.csv`. Note that both files have one column in common: `movieId`. For example, if we look at the first data line (second line in the file, excluding the header) in `ratings.csv`: \n",
    "\n",
    "`1,16,4.0,1217897793`\n",
    "\n",
    "The `movieId` is `16`, which corresponds to the 16th data line in `movies.csv`:\n",
    "\n",
    "`16,Casino (1995),Crime|Drama`\n",
    "\n",
    "Thus, the rating from user `1` for the movie `Casino (1995)` is `4.0`. You could see that the two files are linked by `movieId`. In real life, we often use different IDs to relate data from different files or different databases using a similar process.\n",
    "\n",
    "If you have done the reading for Lesson 3, you'll notice that `join` is a good command to accomplish our goal. Before we use `join`, however, the files must be sorted on the join fields (i.e. `movieId`). Also, note that **`join` assumes the joining column will be of string type, so any number must also be sorted with a regular sort (meaning 10 will come before 2), NOT a numeric sort (option -n).**\n",
    "\n",
    "### 1. Sort the `movies_sorted.csv`\n",
    "\n",
    "In the following code cell: \n",
    "\n",
    "- Remove the first line (the header, since it messes the sorting up).\n",
    "- Sort the `~/data/ml-latest-small/movies.csv` file by the `movieId` column in regular order (default). Again, do NOT do a numeric sort.\n",
    "- Save your output to `~/w6p3/` as `movies_sorted.csv`. \n",
    "- The `~/w6p3/` directory is already created by the first cell in this notebook, and you do not need to write a line to create it. \n",
    "\n",
    "You may do some research on the assertion test codes to get a sample answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "1b7bf951ac46713edc03fe6b65391e7a",
     "grade": false,
     "grade_id": "sort_a",
     "locked": false,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# YOUR CODE HERE\n",
    "# Remove the first line, sort the file by the first column, save in doc\n",
    "sed 1d ~/data/ml-latest-small/movies.csv | sort -t',' -k1,1 > ~/w6p3/movies_sorted.csv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "7f854be978370581737c3374104c3b14",
     "grade": true,
     "grade_id": "sort_t",
     "locked": true,
     "points": 5,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# assertions\n",
    "answer = !cat ~/w6p3/movies_sorted.csv\n",
    "movies = !cat ~/data/ml-latest-small/movies.csv\n",
    "assert_equal(len(answer)+1, len(movies))\n",
    "answer_stripped = [m.strip('\"') for m in answer]\n",
    "movies_stripped = [m.strip('\"') for m in movies][1:]\n",
    "movies_stripped.sort()\n",
    "assert_equal(answer_stripped, movies_stripped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "cf93c2899eabfc60a3e16f3132d71ead",
     "grade": false,
     "grade_id": "users",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "### 2. Generate movie rating csv files for `userId 22` and `66`\n",
    "\n",
    "In the following code cell:\n",
    "\n",
    "- Extract all rating records (lines) from `~/data/ml-latest-small/ratings.csv` with `userId` equals `22`. \n",
    "- Remove the `userId` and `timestamp` columns (column 1 and 4). \n",
    "- Sort the lines by the `movieId` column; again, remember NOT to use the numerical order option. \n",
    "- Save your output to `~/w6p3/` as `ratings_user22.csv`. \n",
    "\n",
    "The above order of steps is recommended; if you switch the order, you may still get the same output, but the field numbers to use can be different. \n",
    "\n",
    "- Do the same for `userId 66` (i.e. extract records, remove columns, sort), and save the output to `~/w6p3/` as `ratings_user66.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "d29f233e097432b70bc649cdc73137cd",
     "grade": false,
     "grade_id": "users_a",
     "locked": false,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# YOUR CODE HERE\n",
    "# Extract all lines with userId = 22, extract column 2 and 3, sort by movieId, save output to the doc.\n",
    "grep -E '^22,.*' ~/data/ml-latest-small/ratings.csv | cut -d',' -f 2,3 | sort -t',' -k1,1 > ~/w6p3/ratings_user22.csv\n",
    "grep -E '^66,.*' ~/data/ml-latest-small/ratings.csv | cut -d',' -f 2,3 | sort -t',' -k1,1 > ~/w6p3/ratings_user66.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "dd6ed960cf4f13580aa4f3f1d15f704d",
     "grade": true,
     "grade_id": "users_t",
     "locked": true,
     "points": 5,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# assertions\n",
    "u22 = !cat ~/w6p3/ratings_user22.csv\n",
    "u66 = !cat ~/w6p3/ratings_user66.csv\n",
    "assert_equal(len(u22), 201)\n",
    "assert_equal(len(u66), 353)\n",
    "assert_equal(str(u22), \"['1049,3.5', '1084,3.5', '1090,1.5', '1120,4.0', '1196,2.5', '1214,1.5', '1215,5.0', '1261,2.5', '1262,2.0', '1263,4.5', '1274,4.5', '1380,1.0', '1485,2.5', '1500,2.5', '153,2.5', '1544,3.0', '1580,3.0', '1591,1.5', '1597,3.5', '161,2.5', '1617,3.5', '163,1.5', '1676,3.0', '1681,1.0', '1682,3.5', '1704,5.0', '1707,2.5', '1729,4.5', '1732,2.5', '186,3.0', '1884,5.0', '1909,2.5', '1923,3.5', '1954,1.5', '2005,3.5', '2012,2.5', '2018,2.5', '2028,2.5', '2054,0.5', '2078,3.0', '2082,3.5', '2140,3.0', '2150,3.5', '2154,3.0', '2167,2.5', '2194,4.0', '2291,3.0', '231,3.5', '2355,3.5', '2363,2.0', '2384,2.5', '2396,4.0', '2407,3.0', '2470,1.0', '25,3.5', '2539,4.0', '2541,2.5', '2640,1.5', '2671,3.5', '2700,2.0', '2706,3.5', '2707,2.5', '2712,3.0', '2716,3.0', '2735,2.0', '2762,4.0', '27728,4.5', '27846,2.5', '2805,3.5', '2822,3.0', '2908,3.5', '2916,2.5', '2950,1.5', '2953,3.5', '296,4.0', '2985,2.0', '3,2.5', '3108,3.5', '3156,4.5', '3173,2.5', '32,2.5', '325,3.0', '3253,3.5', '3258,3.0', '3301,3.5', '337,2.5', '3396,2.5', '342,2.5', '3421,3.0', '3481,4.5', '355,1.0', '356,5.0', '357,3.5', '36,4.5', '3618,3.0', '362,2.5', '3623,2.5', '3624,1.5', '364,4.5', '370,1.5', '3702,2.5', '3703,3.0', '377,4.5', '3793,1.5', '3798,5.0', '3821,2.5', '3827,3.5', '3861,3.5', '3897,3.0', '3901,2.5', '3916,4.5', '3948,4.5', '3967,4.5', '3977,2.5', '3978,4.0', '3980,3.0', '3986,2.5', '3996,2.5', '4011,5.0', '4025,3.5', '4034,3.0', '4085,2.0', '4124,2.5', '4161,3.5', '4226,3.0', '4246,4.0', '4306,4.0', '4321,2.5', '4370,3.5', '4489,3.5', '453,2.0', '4558,2.5', '4638,2.5', '47,4.5', '4718,4.0', '4734,1.5', '4776,3.0', '48,2.5', '480,3.0', '4814,4.5', '4874,5.0', '4878,3.5', '4886,4.5', '4951,3.0', '4979,5.0', '4995,5.0', '5,2.5', '50,3.5', '500,3.5', '5060,1.5', '508,3.5', '5093,4.0', '5134,4.5', '520,2.5', '5218,5.0', '527,3.0', '5309,2.0', '5349,3.5', '541,2.5', '5443,3.0', '5464,5.0', '548,2.5', '5481,2.5', '5502,3.5', '552,0.5', '569,2.0', '5792,3.5', '587,4.5', '5874,2.5', '588,4.0', '5900,4.0', '592,2.5', '5952,4.0', '5956,3.5', '597,2.5', '5989,5.0', '6193,4.0', '6250,2.5', '6266,3.5', '6287,3.0', '6357,3.0', '6377,4.5', '6378,4.5', '6567,2.0', '667,2.5', '6711,4.0', '6870,3.5', '6874,4.0', '6893,4.0', '7143,1.0', '7153,4.0', '7173,3.0', '736,2.5', '7367,4.0', '7437,3.5', '7438,4.5', '761,2.0', '8010,4.0', '833,3.0', '8531,0.5', '919,2.0']\")\n",
    "assert_equal(str(u66), \"['1020,3.0', '104,3.0', '1042,3.0', '1059,3.0', '1060,3.0', '1097,3.0', '1120,3.0', '1167,2.0', '1177,4.0', '1191,3.0', '1196,3.0', '1198,3.0', '1210,3.0', '1213,4.0', '1215,3.0', '123,4.0', '1242,3.0', '125,4.0', '1259,4.0', '1265,4.0', '1270,4.0', '1285,4.0', '1288,5.0', '1291,4.0', '1391,4.0', '1392,3.0', '1393,3.0', '1394,3.0', '1407,3.0', '1409,3.0', '141,2.0', '1441,2.0', '1442,3.0', '1449,5.0', '1457,2.0', '1466,3.0', '1473,1.0', '1476,3.0', '1485,2.0', '1488,2.0', '150,3.0', '1513,3.0', '1517,4.0', '1527,3.0', '1535,2.0', '1542,3.0', '1569,2.0', '1573,2.0', '1580,3.0', '1584,2.0', '1600,3.0', '1602,2.0', '1614,2.0', '1625,4.0', '1639,4.0', '1648,4.0', '166,1.0', '1673,5.0', '1682,3.0', '1693,3.0', '17,4.0', '1704,4.0', '1721,3.0', '1729,3.0', '1732,2.0', '1747,3.0', '1760,3.0', '1777,3.0', '1784,3.0', '1793,2.0', '180,4.0', '1814,3.0', '1858,2.0', '1876,2.0', '1884,4.0', '1885,4.0', '19,1.0', '1904,3.0', '1916,4.0', '1921,4.0', '1923,4.0', '1997,4.0', '2012,2.0', '2021,2.0', '2028,3.0', '203,3.0', '2042,3.0', '2053,2.0', '2059,2.0', '2064,5.0', '2076,4.0', '2082,2.0', '2093,1.0', '2124,2.0', '2142,2.0', '223,4.0', '2231,3.0', '2243,3.0', '2253,3.0', '2271,4.0', '2282,4.0', '2291,3.0', '2294,3.0', '2302,3.0', '231,1.0', '2321,4.0', '2324,5.0', '2333,4.0', '2348,3.0', '235,4.0', '2355,4.0', '2356,2.0', '237,2.0', '2387,2.0', '2396,5.0', '2431,3.0', '2433,3.0', '2435,3.0', '2443,3.0', '248,1.0', '2491,2.0', '2492,4.0', '2496,1.0', '2502,5.0', '2504,4.0', '2507,3.0', '253,2.0', '2541,3.0', '2542,3.0', '2552,1.0', '2558,2.0', '2571,2.0', '2579,5.0', '2580,4.0', '2581,2.0', '2586,3.0', '2589,3.0', '2599,5.0', '260,3.0', '2618,3.0', '2622,2.0', '265,4.0', '2683,4.0', '2692,4.0', '2693,5.0', '2699,4.0', '2700,4.0', '2706,4.0', '2707,4.0', '2710,3.0', '2712,3.0', '2716,4.0', '2718,1.0', '2723,2.0', '275,1.0', '2762,4.0', '2797,3.0', '2798,1.0', '2799,1.0', '2804,4.0', '2809,1.0', '2827,2.0', '2858,5.0', '2889,4.0', '29,2.0', '2918,3.0', '2959,3.0', '296,4.0', '2966,4.0', '2975,2.0', '2987,4.0', '2997,5.0', '3007,3.0', '3020,2.0', '3052,4.0', '3108,3.0', '3115,3.0', '3160,4.0', '317,2.0', '3174,4.0', '3175,5.0', '32,4.0', '3243,1.0', '3248,1.0', '3253,2.0', '3254,2.0', '3255,2.0', '3257,1.0', '3261,4.0', '3273,3.0', '3301,3.0', '3320,3.0', '333,3.0', '3358,5.0', '337,4.0', '339,2.0', '34,3.0', '3401,2.0', '3437,1.0', '344,1.0', '3450,3.0', '3477,4.0', '3481,5.0', '3502,3.0', '351,2.0', '355,2.0', '3556,3.0', '356,3.0', '357,3.0', '3578,4.0', '361,2.0', '3614,1.0', '3616,3.0', '3623,2.0', '364,3.0', '367,2.0', '372,4.0', '3751,5.0', '377,2.0', '3785,1.0', '3793,5.0', '380,2.0', '3852,3.0', '3858,3.0', '3865,3.0', '3893,5.0', '3897,4.0', '39,4.0', '3910,3.0', '3913,5.0', '3943,3.0', '3948,4.0', '3949,2.0', '3975,1.0', '3977,3.0', '3996,4.0', '4014,4.0', '4015,1.0', '4018,1.0', '4019,4.0', '4020,4.0', '4034,4.0', '4068,1.0', '413,3.0', '415,2.0', '4226,5.0', '4233,1.0', '4239,4.0', '4246,4.0', '429,3.0', '4304,4.0', '4306,5.0', '4308,4.0', '4310,3.0', '432,2.0', '435,1.0', '4370,2.0', '440,3.0', '4447,3.0', '45,4.0', '454,2.0', '4641,4.0', '4643,3.0', '4649,4.0', '4701,2.0', '471,2.0', '4718,4.0', '472,2.0', '4738,4.0', '477,3.0', '4773,4.0', '4848,4.0', '485,1.0', '4881,3.0', '4886,5.0', '4890,1.0', '4896,4.0', '497,3.0', '4973,5.0', '4979,5.0', '4993,4.0', '500,2.0', '5013,4.0', '5026,1.0', '505,2.0', '508,3.0', '5222,4.0', '5225,4.0', '5265,3.0', '527,5.0', '529,2.0', '5299,3.0', '532,2.0', '5324,1.0', '5349,5.0', '5377,4.0', '5378,3.0', '538,4.0', '5388,5.0', '5400,1.0', '5415,4.0', '543,2.0', '5445,4.0', '5451,2.0', '5459,3.0', '5460,3.0', '5477,4.0', '5481,3.0', '5502,4.0', '5505,4.0', '551,4.0', '5525,4.0', '5528,4.0', '553,1.0', '5532,3.0', '5618,5.0', '562,2.0', '5620,3.0', '5669,5.0', '5673,5.0', '574,4.0', '5816,4.0', '5829,4.0', '585,3.0', '586,1.0', '588,3.0', '589,1.0', '5902,5.0', '595,3.0', '5952,3.0', '5956,4.0', '5971,5.0', '5989,5.0', '603,2.0', '608,4.0', '627,3.0', '63,2.0', '661,3.0', '671,3.0', '69,2.0', '691,2.0', '700,3.0', '708,3.0', '724,4.0', '766,3.0', '784,3.0', '785,2.0', '805,3.0', '829,3.0', '832,3.0', '851,3.0', '892,3.0', '913,4.0']\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "969a09c774a90ea77e0c0d3457d2e86c",
     "grade": false,
     "grade_id": "join",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "The first 10 lines and the last 10 lines in `ratings_user22.csv` should look like this:\n",
    "\n",
    "```python\n",
    "!head ~/w6p3/ratings_user22.csv\n",
    "```\n",
    "`1049,3.5\n",
    "1084,3.5\n",
    "1090,1.5\n",
    "1120,4.0\n",
    "1196,2.5\n",
    "1214,1.5\n",
    "1215,5.0\n",
    "1261,2.5\n",
    "1262,2.0\n",
    "1263,4.5`\n",
    "\n",
    "```python\n",
    "!tail ~/w6p3/ratings_user22.csv\n",
    "```\n",
    "`7173,3.0\n",
    "736,2.5\n",
    "7367,4.0\n",
    "7437,3.5\n",
    "7438,4.5\n",
    "761,2.0\n",
    "8010,4.0\n",
    "833,3.0\n",
    "8531,0.5\n",
    "919,2.0`\n",
    "\n",
    "\n",
    "### 3. Combine ratings and movies via `movieId` using Unix `join`\n",
    "\n",
    "Now we have the sorted movie information in `~/w6p3/movies_sorted.csv`, movie ratings from `userId 22` in `~/w6p3/movies_rated_by_user22.csv`, and movie ratings from `userId 66` in `~/w6p3/movies_rated_by_user66.csv`. \n",
    "\n",
    "In the following code cell:\n",
    "\n",
    "- For `userId 22`, do a Unix `join` to combine `ratings_user22.csv` with `movies_sorted.csv` using the common key `movieId`. \n",
    "- Save the output to `~/w6p3/` as `movies_rated_by_user22.csv`. \n",
    "- The output column order is: `movieId`, `rating`, `title`, `genres`.\n",
    "- Do the same for `userId 66`, and save the output to `~/w6p3/` as `movies_rated_by_user66.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "1a4280cce583b139d3daa09cdb037841",
     "grade": false,
     "grade_id": "join_a",
     "locked": false,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# YOUR CODE HERE\n",
    "join -t',' ~/w6p3/ratings_user22.csv ~/w6p3/movies_sorted.csv > ~/w6p3/movies_rated_by_user22.csv\n",
    "join -t',' ~/w6p3/ratings_user66.csv ~/w6p3/movies_sorted.csv > ~/w6p3/movies_rated_by_user66.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "81f0722e7bf01e32ac0efe2a47524d20",
     "grade": true,
     "grade_id": "join_p",
     "locked": true,
     "points": 5,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# assertions\n",
    "mr22 = !cat ~/w6p3/movies_rated_by_user22.csv\n",
    "mr66 = !cat ~/w6p3/movies_rated_by_user66.csv\n",
    "assert_equal(len(mr22), len(u22))\n",
    "assert_equal(len(mr66), len(u66))\n",
    "assert_equal(str(mr22), '[\\'1049,3.5,\"Ghost and the Darkness, The (1996)\",Action|Adventure\\', \\'1084,3.5,Bonnie and Clyde (1967),Crime|Drama\\', \\'1090,1.5,Platoon (1986),Drama|War\\', \\'1120,4.0,\"People vs. Larry Flynt, The (1996)\",Comedy|Drama\\', \\'1196,2.5,Star Wars: Episode V - The Empire Strikes Back (1980),Action|Adventure|Sci-Fi\\', \\'1214,1.5,Alien (1979),Horror|Sci-Fi\\', \\'1215,5.0,Army of Darkness (1993),Action|Adventure|Comedy|Fantasy|Horror\\', \\'1261,2.5,Evil Dead II (Dead by Dawn) (1987),Action|Comedy|Fantasy|Horror\\', \\'1262,2.0,\"Great Escape, The (1963)\",Action|Adventure|Drama|War\\', \\'1263,4.5,\"Deer Hunter, The (1978)\",Drama|War\\', \\'1274,4.5,Akira (1988),Action|Adventure|Animation|Sci-Fi\\', \\'1380,1.0,Grease (1978),Comedy|Musical|Romance\\', \\'1485,2.5,Liar Liar (1997),Comedy\\', \\'1500,2.5,Grosse Pointe Blank (1997),Comedy|Crime|Romance\\', \\'153,2.5,Batman Forever (1995),Action|Adventure|Comedy|Crime\\', \\'1544,3.0,\"Lost World: Jurassic Park, The (1997)\",Action|Adventure|Sci-Fi|Thriller\\', \\'1580,3.0,Men in Black (a.k.a. MIB) (1997),Action|Comedy|Sci-Fi\\', \\'1591,1.5,Spawn (1997),Action|Adventure|Sci-Fi|Thriller\\', \\'1597,3.5,Conspiracy Theory (1997),Drama|Mystery|Romance|Thriller\\', \\'161,2.5,Crimson Tide (1995),Drama|Thriller|War\\', \\'1617,3.5,L.A. Confidential (1997),Crime|Film-Noir|Mystery|Thriller\\', \\'163,1.5,Desperado (1995),Action|Romance|Western\\', \\'1676,3.0,Starship Troopers (1997),Action|Sci-Fi\\', \\'1681,1.0,Mortal Kombat: Annihilation (1997),Action|Adventure|Fantasy\\', \\'1682,3.5,\"Truman Show, The (1998)\",Comedy|Drama|Sci-Fi\\', \\'1704,5.0,Good Will Hunting (1997),Drama|Romance\\', \\'1707,2.5,Home Alone 3 (1997),Children|Comedy\\', \\'1729,4.5,Jackie Brown (1997),Crime|Drama|Thriller\\', \\'1732,2.5,\"Big Lebowski, The (1998)\",Comedy|Crime\\', \\'186,3.0,Nine Months (1995),Comedy|Romance\\', \\'1884,5.0,Fear and Loathing in Las Vegas (1998),Adventure|Comedy|Drama\\', \\'1909,2.5,\"X-Files: Fight the Future, The (1998)\",Action|Crime|Mystery|Sci-Fi|Thriller\\', \"1923,3.5,There\\'s Something About Mary (1998),Comedy|Romance\", \\'1954,1.5,Rocky (1976),Drama\\', \\'2005,3.5,\"Goonies, The (1985)\",Action|Adventure|Children|Comedy|Fantasy\\', \\'2012,2.5,Back to the Future Part III (1990),Adventure|Comedy|Sci-Fi|Western\\', \\'2018,2.5,Bambi (1942),Animation|Children|Drama\\', \\'2028,2.5,Saving Private Ryan (1998),Action|Drama|War\\', \\'2054,0.5,\"Honey, I Shrunk the Kids (1989)\",Adventure|Children|Comedy|Fantasy|Sci-Fi\\', \\'2078,3.0,\"Jungle Book, The (1967)\",Animation|Children|Comedy|Musical\\', \\'2082,3.5,\"Mighty Ducks, The (1992)\",Children|Comedy\\', \\'2140,3.0,\"Dark Crystal, The (1982)\",Adventure|Fantasy\\', \\'2150,3.5,\"Gods Must Be Crazy, The (1980)\",Adventure|Comedy\\', \\'2154,3.0,How Stella Got Her Groove Back (1998),Drama|Romance\\', \\'2167,2.5,Blade (1998),Action|Horror|Thriller\\', \\'2194,4.0,\"Untouchables, The (1987)\",Action|Crime|Drama\\', \\'2291,3.0,Edward Scissorhands (1990),Drama|Fantasy|Romance\\', \\'231,3.5,Dumb & Dumber (Dumb and Dumber) (1994),Adventure|Comedy\\', \\'2355,3.5,\"Bug\\\\\\'s Life, A (1998)\",Adventure|Animation|Children|Comedy\\', \\'2363,2.0,Godzilla (Gojira) (1954),Drama|Horror|Sci-Fi\\', \\'2384,2.5,Babe: Pig in the City (1998),Adventure|Children|Drama\\', \\'2396,4.0,Shakespeare in Love (1998),Comedy|Drama|Romance\\', \\'2407,3.0,Cocoon (1985),Comedy|Sci-Fi\\', \\'2470,1.0,Crocodile Dundee (1986),Adventure|Comedy\\', \\'25,3.5,Leaving Las Vegas (1995),Drama|Romance\\', \\'2539,4.0,Analyze This (1999),Comedy\\', \\'2541,2.5,Cruel Intentions (1999),Drama\\', \\'2640,1.5,Superman (1978),Action|Adventure|Sci-Fi\\', \\'2671,3.5,Notting Hill (1999),Comedy|Romance\\', \\'2700,2.0,\"South Park: Bigger, Longer and Uncut (1999)\",Animation|Comedy|Musical\\', \\'2706,3.5,American Pie (1999),Comedy|Romance\\', \\'2707,2.5,Arlington Road (1999),Thriller\\', \\'2712,3.0,Eyes Wide Shut (1999),Drama|Mystery|Thriller\\', \\'2716,3.0,Ghostbusters (a.k.a. Ghost Busters) (1984),Action|Comedy|Sci-Fi\\', \\'2735,2.0,\"Golden Child, The (1986)\",Action|Adventure|Comedy|Fantasy|Mystery\\', \\'2762,4.0,\"Sixth Sense, The (1999)\",Drama|Horror|Mystery\\', \\'27728,4.5,Ghost in the Shell 2: Innocence (a.k.a. Innocence) (Inosensu) (2004),Action|Animation|Drama|Sci-Fi|Thriller\\', \\'27846,2.5,\"Corporation, The (2003)\",Documentary\\', \\'2805,3.5,Mickey Blue Eyes (1999),Comedy|Romance\\', \\'2822,3.0,Medicine Man (1992),Adventure|Romance\\', \"2908,3.5,Boys Don\\'t Cry (1999),Drama\", \\'2916,2.5,Total Recall (1990),Action|Adventure|Sci-Fi|Thriller\\', \\'2950,1.5,\"Blue Lagoon, The (1980)\",Adventure|Drama|Romance\\', \\'2953,3.5,Home Alone 2: Lost in New York (1992),Children|Comedy\\', \\'296,4.0,Pulp Fiction (1994),Comedy|Crime|Drama|Thriller\\', \\'2985,2.0,RoboCop (1987),Action|Crime|Drama|Sci-Fi|Thriller\\', \\'3,2.5,Grumpier Old Men (1995),Comedy|Romance\\', \\'3108,3.5,\"Fisher King, The (1991)\",Comedy|Drama|Fantasy|Romance\\', \\'3156,4.5,Bicentennial Man (1999),Drama|Romance|Sci-Fi\\', \\'3173,2.5,Any Given Sunday (1999),Drama\\', \\'32,2.5,Twelve Monkeys (a.k.a. 12 Monkeys) (1995),Mystery|Sci-Fi|Thriller\\', \"325,3.0,National Lampoon\\'s Senior Trip (1995),Comedy\", \"3253,3.5,Wayne\\'s World (1992),Comedy\", \\'3258,3.0,Death Becomes Her (1992),Comedy|Fantasy\\', \\'3301,3.5,\"Whole Nine Yards, The (2000)\",Comedy|Crime\\', \"337,2.5,What\\'s Eating Gilbert Grape (1993),Drama\", \\'3396,2.5,\"Muppet Movie, The (1979)\",Adventure|Children|Comedy|Musical\\', \"342,2.5,Muriel\\'s Wedding (1994),Comedy\", \\'3421,3.0,Animal House (1978),Comedy\\', \\'3481,4.5,High Fidelity (2000),Comedy|Drama|Romance\\', \\'355,1.0,\"Flintstones, The (1994)\",Children|Comedy|Fantasy\\', \\'356,5.0,Forrest Gump (1994),Comedy|Drama|Romance|War\\', \\'357,3.5,Four Weddings and a Funeral (1994),Comedy|Romance\\', \\'36,4.5,Dead Man Walking (1995),Crime|Drama\\', \\'3618,3.0,Small Time Crooks (2000),Comedy|Crime\\', \\'362,2.5,\"Jungle Book, The (1994)\",Adventure|Children|Romance\\', \\'3623,2.5,Mission: Impossible II (2000),Action|Adventure|Thriller\\', \\'3624,1.5,Shanghai Noon (2000),Action|Adventure|Comedy|Western\\', \\'364,4.5,\"Lion King, The (1994)\",Adventure|Animation|Children|Drama|Musical|IMAX\\', \\'370,1.5,Naked Gun 33 1/3: The Final Insult (1994),Action|Comedy\\', \\'3702,2.5,Mad Max (1979),Action|Adventure|Sci-Fi\\', \\'3703,3.0,\"Road Warrior, The (Mad Max 2) (1981)\",Action|Adventure|Sci-Fi\\', \\'377,4.5,Speed (1994),Action|Romance|Thriller\\', \\'3793,1.5,X-Men (2000),Action|Adventure|Sci-Fi\\', \\'3798,5.0,What Lies Beneath (2000),Drama|Horror|Mystery\\', \\'3821,2.5,Nutty Professor II: The Klumps (2000),Comedy\\', \\'3827,3.5,Space Cowboys (2000),Action|Adventure|Comedy|Sci-Fi\\', \\'3861,3.5,\"Replacements, The (2000)\",Comedy\\', \\'3897,3.0,Almost Famous (2000),Drama\\', \\'3901,2.5,Duets (2000),Comedy|Drama\\', \\'3916,4.5,Remember the Titans (2000),Drama\\', \\'3948,4.5,Meet the Parents (2000),Comedy\\', \\'3967,4.5,Billy Elliot (2000),Drama\\', \"3977,2.5,Charlie\\'s Angels (2000),Action|Comedy\", \\'3978,4.0,\"Legend of Bagger Vance, The (2000)\",Drama|Romance\\', \\'3980,3.0,Men of Honor (2000),Drama\\', \\'3986,2.5,\"6th Day, The (2000)\",Action|Sci-Fi|Thriller\\', \\'3996,2.5,\"Crouching Tiger, Hidden Dragon (Wo hu cang long) (2000)\",Action|Drama|Romance\\', \\'4011,5.0,Snatch (2000),Comedy|Crime|Thriller\\', \\'4025,3.5,Miss Congeniality (2000),Comedy|Crime\\', \\'4034,3.0,Traffic (2000),Crime|Drama|Thriller\\', \\'4085,2.0,Beverly Hills Cop (1984),Action|Comedy|Crime|Drama\\', \\'4124,2.5,Jaws: The Revenge (1987),Horror|Thriller\\', \\'4161,3.5,\"Mexican, The (2001)\",Action|Comedy\\', \\'4226,3.0,Memento (2000),Mystery|Thriller\\', \"4246,4.0,Bridget Jones\\'s Diary (2001),Comedy|Drama|Romance\", \\'4306,4.0,Shrek (2001),Adventure|Animation|Children|Comedy|Fantasy|Romance\\', \\'4321,2.5,City Slickers (1991),Comedy|Western\\', \\'4370,3.5,A.I. Artificial Intelligence (2001),Adventure|Drama|Sci-Fi\\', \\'4489,3.5,Coming to America (1988),Comedy|Romance\\', \\'453,2.0,For Love or Money (1993),Comedy|Romance\\', \\'4558,2.5,Twins (1988),Comedy\\', \\'4638,2.5,Jurassic Park III (2001),Action|Adventure|Sci-Fi|Thriller\\', \\'47,4.5,Seven (a.k.a. Se7en) (1995),Mystery|Thriller\\', \\'4718,4.0,American Pie 2 (2001),Comedy\\', \\'4734,1.5,Jay and Silent Bob Strike Back (2001),Adventure|Comedy\\', \\'4776,3.0,Training Day (2001),Crime|Drama|Thriller\\', \\'48,2.5,Pocahontas (1995),Animation|Children|Drama|Musical|Romance\\', \\'480,3.0,Jurassic Park (1993),Action|Adventure|Sci-Fi|Thriller\\', \"4814,4.5,Don\\'t Say a Word (2001),Thriller\", \\'4874,5.0,K-PAX (2001),Drama|Fantasy|Mystery|Sci-Fi\\', \\'4878,3.5,Donnie Darko (2001),Drama|Mystery|Sci-Fi|Thriller\\', \\'4886,4.5,\"Monsters, Inc. (2001)\",Adventure|Animation|Children|Comedy|Fantasy\\', \\'4951,3.0,Lord of the Flies (1990),Adventure|Drama|Thriller\\', \\'4979,5.0,\"Royal Tenenbaums, The (2001)\",Comedy|Drama\\', \\'4995,5.0,\"Beautiful Mind, A (2001)\",Drama|Romance\\', \\'5,2.5,Father of the Bride Part II (1995),Comedy\\', \\'50,3.5,\"Usual Suspects, The (1995)\",Crime|Mystery|Thriller\\', \\'500,3.5,Mrs. Doubtfire (1993),Comedy|Drama\\', \\'5060,1.5,M*A*S*H (a.k.a. MASH) (1970),Comedy|Drama|War\\', \\'508,3.5,Philadelphia (1993),Drama\\', \\'5093,4.0,Collateral Damage (2002),Action|Thriller\\', \\'5134,4.5,Mean Machine (2001),Comedy|Drama\\', \\'520,2.5,Robin Hood: Men in Tights (1993),Comedy\\', \\'5218,5.0,Ice Age (2002),Adventure|Animation|Children|Comedy\\', \"527,3.0,Schindler\\'s List (1993),Drama|War\", \\'5309,2.0,Three Men and a Little Lady (1990),Comedy|Romance\\', \\'5349,3.5,Spider-Man (2002),Action|Adventure|Sci-Fi|Thriller\\', \\'541,2.5,Blade Runner (1982),Action|Sci-Fi|Thriller\\', \\'5443,3.0,Juwanna Mann (2002),Comedy\\', \\'5464,5.0,Road to Perdition (2002),Crime|Drama\\', \\'548,2.5,Terminal Velocity (1994),Action|Mystery|Thriller\\', \\'5481,2.5,Austin Powers in Goldmember (2002),Comedy\\', \\'5502,3.5,Signs (2002),Horror|Sci-Fi|Thriller\\', \\'552,0.5,\"Three Musketeers, The (1993)\",Action|Adventure|Comedy|Romance\\', \\'569,2.0,Little Big League (1994),Comedy|Drama\\', \\'5792,3.5,Roger Dodger (2002),Comedy|Drama\\', \\'587,4.5,Ghost (1990),Comedy|Drama|Fantasy|Romance|Thriller\\', \\'5874,2.5,Friday After Next (2002),Comedy\\', \\'588,4.0,Aladdin (1992),Adventure|Animation|Children|Comedy|Musical\\', \\'5900,4.0,Analyze That (2002),Comedy|Crime\\', \\'592,2.5,Batman (1989),Action|Crime|Thriller\\', \\'5952,4.0,\"Lord of the Rings: The Two Towers, The (2002)\",Adventure|Fantasy\\', \\'5956,3.5,Gangs of New York (2002),Crime|Drama\\', \\'597,2.5,Pretty Woman (1990),Comedy|Romance\\', \\'5989,5.0,Catch Me If You Can (2002),Crime|Drama\\', \\'6193,4.0,Poolhall Junkies (2002),Comedy|Drama|Thriller\\', \\'6250,2.5,Dreamcatcher (2003),Drama|Horror|Sci-Fi|Thriller\\', \\'6266,3.5,What a Girl Wants (2003),Comedy|Drama|Romance\\', \\'6287,3.0,Anger Management (2003),Comedy\\', \\'6357,3.0,High Society (1956),Comedy|Musical|Romance\\', \\'6377,4.5,Finding Nemo (2003),Adventure|Animation|Children|Comedy\\', \\'6378,4.5,\"Italian Job, The (2003)\",Action|Crime\\', \\'6567,2.0,Buffalo Soldiers (2001),Comedy|Crime|Drama|War\\', \\'667,2.5,Bloodsport 2 (a.k.a. Bloodsport II: The Next Kumite) (1996),Action\\', \\'6711,4.0,Lost in Translation (2003),Comedy|Drama|Romance\\', \\'6870,3.5,Mystic River (2003),Crime|Drama|Mystery\\', \\'6874,4.0,Kill Bill: Vol. 1 (2003),Action|Crime|Thriller\\', \\'6893,4.0,\"Italian Job, The (1969)\",Action|Comedy|Crime\\', \\'7143,1.0,\"Last Samurai, The (2003)\",Action|Adventure|Drama|War\\', \\'7153,4.0,\"Lord of the Rings: The Return of the King, The (2003)\",Action|Adventure|Drama|Fantasy\\', \\'7173,3.0,Along Came Polly (2004),Comedy|Romance\\', \\'736,2.5,Twister (1996),Action|Adventure|Romance|Thriller\\', \\'7367,4.0,\"Ladykillers, The (2004)\",Comedy|Crime\\', \\'7437,3.5,Connie and Carla (2004),Comedy\\', \\'7438,4.5,Kill Bill: Vol. 2 (2004),Action|Drama|Thriller\\', \\'761,2.0,\"Phantom, The (1996)\",Action|Adventure\\', \\'8010,4.0,\"Power of One, The (1992)\",Drama\\', \\'833,3.0,High School High (1996),Comedy\\', \\'8531,0.5,White Chicks (2004),Action|Comedy|Crime\\', \\'919,2.0,\"Wizard of Oz, The (1939)\",Adventure|Children|Fantasy|Musical\\']')\n",
    "assert_equal(str(mr66), '[\\'1020,3.0,Cool Runnings (1993),Comedy\\', \\'104,3.0,Happy Gilmore (1996),Comedy\\', \\'1042,3.0,That Thing You Do! (1996),Comedy|Drama\\', \"1059,3.0,William Shakespeare\\'s Romeo + Juliet (1996),Drama|Romance\", \\'1060,3.0,Swingers (1996),Comedy|Drama\\', \\'1097,3.0,E.T. the Extra-Terrestrial (1982),Children|Drama|Sci-Fi\\', \\'1120,3.0,\"People vs. Larry Flynt, The (1996)\",Comedy|Drama\\', \\'1167,2.0,Dear God (1996),Comedy\\', \\'1177,4.0,Enchanted April (1992),Drama|Romance\\', \\'1191,3.0,Madonna: Truth or Dare (1991),Documentary|Musical\\', \\'1196,3.0,Star Wars: Episode V - The Empire Strikes Back (1980),Action|Adventure|Sci-Fi\\', \\'1198,3.0,Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981),Action|Adventure\\', \\'1210,3.0,Star Wars: Episode VI - Return of the Jedi (1983),Action|Adventure|Sci-Fi\\', \\'1213,4.0,Goodfellas (1990),Crime|Drama\\', \\'1215,3.0,Army of Darkness (1993),Action|Adventure|Comedy|Fantasy|Horror\\', \\'123,4.0,Chungking Express (Chung Hing sam lam) (1994),Drama|Mystery|Romance\\', \\'1242,3.0,Glory (1989),Drama|War\\', \\'125,4.0,Flirting With Disaster (1996),Comedy\\', \\'1259,4.0,Stand by Me (1986),Adventure|Drama\\', \\'1265,4.0,Groundhog Day (1993),Comedy|Fantasy|Romance\\', \\'1270,4.0,Back to the Future (1985),Adventure|Comedy|Sci-Fi\\', \\'1285,4.0,Heathers (1989),Comedy\\', \\'1288,5.0,This Is Spinal Tap (1984),Comedy\\', \\'1291,4.0,Indiana Jones and the Last Crusade (1989),Action|Adventure\\', \\'1391,4.0,Mars Attacks! (1996),Action|Comedy|Sci-Fi\\', \\'1392,3.0,Citizen Ruth (1996),Comedy|Drama\\', \\'1393,3.0,Jerry Maguire (1996),Drama|Romance\\', \\'1394,3.0,Raising Arizona (1987),Comedy\\', \\'1407,3.0,Scream (1996),Comedy|Horror|Mystery|Thriller\\', \\'1409,3.0,Michael (1996),Comedy|Drama|Fantasy|Romance\\', \\'141,2.0,\"Birdcage, The (1996)\",Comedy\\', \\'1441,2.0,Benny & Joon (1993),Comedy|Romance\\', \\'1442,3.0,Prefontaine (1997),Drama\\', \\'1449,5.0,Waiting for Guffman (1996),Comedy\\', \\'1457,2.0,Fools Rush In (1997),Comedy|Drama|Romance\\', \\'1466,3.0,Donnie Brasco (1997),Crime|Drama\\', \\'1473,1.0,Best Men (1997),Action|Comedy|Crime|Drama\\', \\'1476,3.0,Private Parts (1997),Comedy|Drama\\', \\'1485,2.0,Liar Liar (1997),Comedy\\', \\'1488,2.0,\"Devil\\\\\\'s Own, The (1997)\",Action|Drama|Thriller\\', \\'150,3.0,Apollo 13 (1995),Adventure|Drama|IMAX\\', \"1513,3.0,Romy and Michele\\'s High School Reunion (1997),Comedy\", \\'1517,4.0,Austin Powers: International Man of Mystery (1997),Action|Adventure|Comedy\\', \\'1527,3.0,\"Fifth Element, The (1997)\",Action|Adventure|Comedy|Sci-Fi\\', \\'1535,2.0,Love! Valour! Compassion! (1997),Drama|Romance\\', \\'1542,3.0,Brassed Off (1996),Comedy|Drama|Romance\\', \"1569,2.0,My Best Friend\\'s Wedding (1997),Comedy|Romance\", \\'1573,2.0,Face/Off (1997),Action|Crime|Drama|Thriller\\', \\'1580,3.0,Men in Black (a.k.a. MIB) (1997),Action|Comedy|Sci-Fi\\', \\'1584,2.0,Contact (1997),Drama|Sci-Fi\\', \"1600,3.0,She\\'s So Lovely (1997),Drama|Romance\", \\'1602,2.0,Leave It to Beaver (1997),Comedy\\', \\'1614,2.0,In & Out (1997),Comedy\\', \\'1625,4.0,\"Game, The (1997)\",Drama|Mystery|Thriller\\', \\'1639,4.0,Chasing Amy (1997),Comedy|Drama|Romance\\', \\'1648,4.0,\"House of Yes, The (1997)\",Comedy|Drama\\', \\'166,1.0,\"Doom Generation, The (1995)\",Comedy|Crime|Drama\\', \\'1673,5.0,Boogie Nights (1997),Drama\\', \\'1682,3.0,\"Truman Show, The (1998)\",Comedy|Drama|Sci-Fi\\', \\'1693,3.0,Amistad (1997),Drama|Mystery\\', \\'17,4.0,Sense and Sensibility (1995),Drama|Romance\\', \\'1704,4.0,Good Will Hunting (1997),Drama|Romance\\', \\'1721,3.0,Titanic (1997),Drama|Romance\\', \\'1729,3.0,Jackie Brown (1997),Crime|Drama|Thriller\\', \\'1732,2.0,\"Big Lebowski, The (1998)\",Comedy|Crime\\', \\'1747,3.0,Wag the Dog (1997),Comedy\\', \\'1760,3.0,Spice World (1997),Comedy\\', \\'1777,3.0,\"Wedding Singer, The (1998)\",Comedy|Romance\\', \\'1784,3.0,As Good as It Gets (1997),Comedy|Drama|Romance\\', \\'1793,2.0,Welcome to Woop-Woop (1997),Comedy\\', \\'180,4.0,Mallrats (1995),Comedy|Romance\\', \\'1814,3.0,\"Price Above Rubies, A (1998)\",Drama\\', \\'1858,2.0,Mr. Nice Guy (Yat goh ho yan) (1997),Action|Comedy\\', \\'1876,2.0,Deep Impact (1998),Drama|Sci-Fi|Thriller\\', \\'1884,4.0,Fear and Loathing in Las Vegas (1998),Adventure|Comedy|Drama\\', \\'1885,4.0,\"Opposite of Sex, The (1998)\",Comedy|Drama|Romance\\', \\'19,1.0,Ace Ventura: When Nature Calls (1995),Comedy\\', \\'1904,3.0,Henry Fool (1997),Comedy|Drama\\', \"1916,4.0,Buffalo \\'66 (a.k.a. Buffalo 66) (1998),Drama|Romance\", \\'1921,4.0,Pi (1998),Drama|Sci-Fi|Thriller\\', \"1923,4.0,There\\'s Something About Mary (1998),Comedy|Romance\", \\'1997,4.0,\"Exorcist, The (1973)\",Horror|Mystery\\', \\'2012,2.0,Back to the Future Part III (1990),Adventure|Comedy|Sci-Fi|Western\\', \\'2021,2.0,Dune (1984),Adventure|Sci-Fi\\', \\'2028,3.0,Saving Private Ryan (1998),Action|Drama|War\\', \\'203,3.0,\"To Wong Foo, Thanks for Everything! Julie Newmar (1995)\",Comedy\\', \\'2042,3.0,D2: The Mighty Ducks (1994),Children|Comedy\\', \\'2053,2.0,\"Honey, I Blew Up the Kid (1992)\",Children|Comedy|Sci-Fi\\', \\'2059,2.0,\"Parent Trap, The (1998)\",Children|Comedy|Romance\\', \\'2064,5.0,Roger & Me (1989),Documentary\\', \\'2076,4.0,Blue Velvet (1986),Drama|Mystery|Thriller\\', \\'2082,2.0,\"Mighty Ducks, The (1992)\",Children|Comedy\\', \\'2093,1.0,Return to Oz (1985),Adventure|Children|Fantasy\\', \\'2124,2.0,\"Addams Family, The (1991)\",Children|Comedy|Fantasy\\', \\'2142,2.0,\"American Tail: Fievel Goes West, An (1991)\",Adventure|Animation|Children|Musical|Western\\', \\'223,4.0,Clerks (1994),Comedy\\', \\'2231,3.0,Rounders (1998),Drama\\', \\'2243,3.0,Broadcast News (1987),Comedy|Drama|Romance\\', \\'2253,3.0,Toys (1992),Comedy|Fantasy\\', \\'2271,4.0,Permanent Midnight (1998),Drama\\', \\'2282,4.0,Pecker (1998),Comedy|Drama\\', \\'2291,3.0,Edward Scissorhands (1990),Drama|Fantasy|Romance\\', \\'2294,3.0,Antz (1998),Adventure|Animation|Children|Comedy|Fantasy\\', \\'2302,3.0,My Cousin Vinny (1992),Comedy\\', \\'231,1.0,Dumb & Dumber (Dumb and Dumber) (1994),Adventure|Comedy\\', \\'2321,4.0,Pleasantville (1998),Comedy|Drama|Fantasy\\', \\'2324,5.0,Life Is Beautiful (La Vita è bella) (1997),Comedy|Drama|Romance|War\\', \\'2333,4.0,Gods and Monsters (1998),Drama\\', \\'2348,3.0,Sid and Nancy (1986),Drama\\', \\'235,4.0,Ed Wood (1994),Comedy|Drama\\', \\'2355,4.0,\"Bug\\\\\\'s Life, A (1998)\",Adventure|Animation|Children|Comedy\\', \\'2356,2.0,Celebrity (1998),Comedy\\', \\'237,2.0,Forget Paris (1995),Comedy|Romance\\', \\'2387,2.0,Very Bad Things (1998),Comedy|Crime\\', \\'2396,5.0,Shakespeare in Love (1998),Comedy|Drama|Romance\\', \\'2431,3.0,Patch Adams (1998),Comedy|Drama\\', \\'2433,3.0,\"Civil Action, A (1998)\",Drama\\', \\'2435,3.0,Hurlyburly (1998),Drama\\', \\'2443,3.0,Playing by Heart (1998),Drama|Romance\\', \\'248,1.0,Houseguest (1994),Comedy\\', \\'2491,2.0,Simply Irresistible (1999),Comedy|Romance\\', \\'2492,4.0,20 Dates (1998),Comedy|Romance\\', \\'2496,1.0,Blast from the Past (1999),Comedy|Romance\\', \\'2502,5.0,Office Space (1999),Comedy|Crime\\', \\'2504,4.0,200 Cigarettes (1999),Comedy|Drama\\', \\'2507,3.0,Breakfast of Champions (1999),Comedy|Sci-Fi\\', \\'253,2.0,Interview with the Vampire: The Vampire Chronicles (1994),Drama|Horror\\', \\'2541,3.0,Cruel Intentions (1999),Drama\\', \\'2542,3.0,\"Lock, Stock & Two Smoking Barrels (1998)\",Comedy|Crime|Thriller\\', \"2552,1.0,My Boyfriend\\'s Back (1993),Comedy\", \\'2558,2.0,Forces of Nature (1999),Comedy|Romance\\', \\'2571,2.0,\"Matrix, The (1999)\",Action|Sci-Fi|Thriller\\', \\'2579,5.0,Following (1998),Crime|Mystery|Thriller\\', \\'2580,4.0,Go (1999),Comedy|Crime\\', \\'2581,2.0,Never Been Kissed (1999),Comedy|Romance\\', \\'2586,3.0,Goodbye Lover (1999),Comedy|Crime|Thriller\\', \\'2589,3.0,Friends & Lovers (1999),Comedy|Drama|Romance\\', \\'2599,5.0,Election (1999),Comedy\\', \\'260,3.0,Star Wars: Episode IV - A New Hope (1977),Action|Adventure|Sci-Fi\\', \\'2618,3.0,\"Castle, The (1997)\",Comedy\\', \"2622,2.0,William Shakespeare\\'s A Midsummer Night\\'s Dream (1999),Comedy|Fantasy\", \\'265,4.0,Like Water for Chocolate (Como agua para chocolate) (1992),Drama|Fantasy|Romance\\', \\'2683,4.0,Austin Powers: The Spy Who Shagged Me (1999),Action|Adventure|Comedy\\', \\'2692,4.0,Run Lola Run (Lola rennt) (1998),Action|Crime\\', \\'2693,5.0,Trekkies (1997),Documentary\\', \\'2699,4.0,Arachnophobia (1990),Comedy|Horror\\', \\'2700,4.0,\"South Park: Bigger, Longer and Uncut (1999)\",Animation|Comedy|Musical\\', \\'2706,4.0,American Pie (1999),Comedy|Romance\\', \\'2707,4.0,Arlington Road (1999),Thriller\\', \\'2710,3.0,\"Blair Witch Project, The (1999)\",Drama|Horror|Thriller\\', \\'2712,3.0,Eyes Wide Shut (1999),Drama|Mystery|Thriller\\', \\'2716,4.0,Ghostbusters (a.k.a. Ghost Busters) (1984),Action|Comedy|Sci-Fi\\', \\'2718,1.0,Drop Dead Gorgeous (1999),Comedy\\', \\'2723,2.0,Mystery Men (1999),Action|Comedy|Fantasy\\', \\'275,1.0,Mixed Nuts (1994),Comedy\\', \\'2762,4.0,\"Sixth Sense, The (1999)\",Drama|Horror|Mystery\\', \\'2797,3.0,Big (1988),Comedy|Drama|Fantasy|Romance\\', \\'2798,1.0,Problem Child (1990),Children|Comedy\\', \\'2799,1.0,Problem Child 2 (1991),Comedy\\', \\'2804,4.0,\"Christmas Story, A (1983)\",Children|Comedy\\', \\'2809,1.0,Love Stinks (1999),Comedy\\', \\'2827,2.0,\"Astronaut\\\\\\'s Wife, The (1999)\",Horror|Sci-Fi|Thriller\\', \\'2858,5.0,American Beauty (1999),Drama|Romance\\', \\'2889,4.0,\"Mystery, Alaska (1999)\",Comedy|Drama\\', \\'29,2.0,\"City of Lost Children, The (Cité des enfants perdus, La) (1995)\",Adventure|Drama|Fantasy|Mystery|Sci-Fi\\', \"2918,3.0,Ferris Bueller\\'s Day Off (1986),Comedy\", \\'2959,3.0,Fight Club (1999),Action|Crime|Drama|Thriller\\', \\'296,4.0,Pulp Fiction (1994),Comedy|Crime|Drama|Thriller\\', \\'2966,4.0,\"Straight Story, The (1999)\",Adventure|Drama\\', \\'2975,2.0,\"Best Man, The (1999)\",Comedy|Drama\\', \\'2987,4.0,Who Framed Roger Rabbit? (1988),Adventure|Animation|Children|Comedy|Crime|Fantasy|Mystery\\', \\'2997,5.0,Being John Malkovich (1999),Comedy|Drama|Fantasy\\', \\'3007,3.0,American Movie (1999),Documentary\\', \\'3020,2.0,Falling Down (1993),Action|Drama\\', \\'3052,4.0,Dogma (1999),Adventure|Comedy|Fantasy\\', \\'3108,3.0,\"Fisher King, The (1991)\",Comedy|Drama|Fantasy|Romance\\', \\'3115,3.0,Flawless (1999),Drama\\', \\'3160,4.0,Magnolia (1999),Drama\\', \\'317,2.0,\"Santa Clause, The (1994)\",Comedy|Drama|Fantasy\\', \\'3174,4.0,Man on the Moon (1999),Comedy|Drama\\', \\'3175,5.0,Galaxy Quest (1999),Adventure|Comedy|Sci-Fi\\', \\'32,4.0,Twelve Monkeys (a.k.a. 12 Monkeys) (1995),Mystery|Sci-Fi|Thriller\\', \\'3243,1.0,Encino Man (1992),Comedy\\', \\'3248,1.0,Sister Act 2: Back in the Habit (1993),Comedy\\', \"3253,2.0,Wayne\\'s World (1992),Comedy\", \"3254,2.0,Wayne\\'s World 2 (1993),Comedy\", \\'3255,2.0,\"League of Their Own, A (1992)\",Comedy|Drama\\', \\'3257,1.0,\"Bodyguard, The (1992)\",Drama|Romance|Thriller\\', \\'3261,4.0,Singles (1992),Comedy|Drama|Romance\\', \\'3273,3.0,Scream 3 (2000),Comedy|Horror|Mystery|Thriller\\', \\'3301,3.0,\"Whole Nine Yards, The (2000)\",Comedy|Crime\\', \"3320,3.0,Mifune\\'s Last Song (Mifunes sidste sang) (1999),Comedy|Drama|Romance\", \\'333,3.0,Tommy Boy (1995),Comedy\\', \\'3358,5.0,Defending Your Life (1991),Comedy|Drama|Fantasy|Romance\\', \"337,4.0,What\\'s Eating Gilbert Grape (1993),Drama\", \\'339,2.0,While You Were Sleeping (1995),Comedy|Romance\\', \\'34,3.0,Babe (1995),Children|Drama\\', \\'3401,2.0,Baby... Secret of the Lost Legend (1985),Adventure|Sci-Fi\\', \\'3437,1.0,Cool as Ice (1991),Drama\\', \\'344,1.0,Ace Ventura: Pet Detective (1994),Comedy\\', \\'3450,3.0,Grumpy Old Men (1993),Comedy\\', \\'3477,4.0,Empire Records (1995),Comedy|Drama\\', \\'3481,5.0,High Fidelity (2000),Comedy|Drama|Romance\\', \\'3502,3.0,My Life (1993),Drama\\', \\'351,2.0,\"Corrina, Corrina (1994)\",Comedy|Drama|Romance\\', \\'355,2.0,\"Flintstones, The (1994)\",Children|Comedy|Fantasy\\', \\'3556,3.0,\"Virgin Suicides, The (1999)\",Drama|Romance\\', \\'356,3.0,Forrest Gump (1994),Comedy|Drama|Romance|War\\', \\'357,3.0,Four Weddings and a Funeral (1994),Comedy|Romance\\', \\'3578,4.0,Gladiator (2000),Action|Adventure|Drama\\', \\'361,2.0,It Could Happen to You (1994),Comedy|Drama|Romance\\', \\'3614,1.0,Honeymoon in Vegas (1992),Comedy|Romance\\', \\'3616,3.0,Loser (2000),Comedy|Romance\\', \\'3623,2.0,Mission: Impossible II (2000),Action|Adventure|Thriller\\', \\'364,3.0,\"Lion King, The (1994)\",Adventure|Animation|Children|Drama|Musical|IMAX\\', \\'367,2.0,\"Mask, The (1994)\",Action|Comedy|Crime|Fantasy\\', \\'372,4.0,Reality Bites (1994),Comedy|Drama|Romance\\', \\'3751,5.0,Chicken Run (2000),Animation|Children|Comedy\\', \\'377,2.0,Speed (1994),Action|Romance|Thriller\\', \\'3785,1.0,Scary Movie (2000),Comedy|Horror\\', \\'3793,5.0,X-Men (2000),Action|Adventure|Sci-Fi\\', \\'380,2.0,True Lies (1994),Action|Adventure|Comedy|Romance|Thriller\\', \\'3852,3.0,\"Tao of Steve, The (2000)\",Comedy\\', \\'3858,3.0,Cecil B. DeMented (2000),Comedy\\', \\'3865,3.0,\"Original Kings of Comedy, The (2000)\",Comedy|Documentary\\', \\'3893,5.0,Nurse Betty (2000),Comedy|Crime|Drama|Romance|Thriller\\', \\'3897,4.0,Almost Famous (2000),Drama\\', \\'39,4.0,Clueless (1995),Comedy|Romance\\', \\'3910,3.0,Dancer in the Dark (2000),Drama|Musical\\', \\'3913,5.0,Barenaked in America (1999),Documentary\\', \\'3943,3.0,Bamboozled (2000),Comedy\\', \\'3948,4.0,Meet the Parents (2000),Comedy\\', \\'3949,2.0,Requiem for a Dream (2000),Drama\\', \\'3975,1.0,Lucky Numbers (2000),Comedy|Drama\\', \"3977,3.0,Charlie\\'s Angels (2000),Action|Comedy\", \\'3996,4.0,\"Crouching Tiger, Hidden Dragon (Wo hu cang long) (2000)\",Action|Drama|Romance\\', \\'4014,4.0,Chocolat (2000),Drama|Romance\\', \\'4015,1.0,\"Dude, Where\\\\\\'s My Car? (2000)\",Comedy|Sci-Fi\\', \\'4018,1.0,What Women Want (2000),Comedy|Romance\\', \\'4019,4.0,Finding Forrester (2000),Drama\\', \\'4020,4.0,\"Gift, The (2000)\",Thriller\\', \\'4034,4.0,Traffic (2000),Crime|Drama|Thriller\\', \\'4068,1.0,Sugar & Spice (2001),Comedy\\', \\'413,3.0,Airheads (1994),Comedy\\', \\'415,2.0,Another Stakeout (1993),Comedy|Thriller\\', \\'4226,5.0,Memento (2000),Mystery|Thriller\\', \\'4233,1.0,Tomcats (2001),Comedy\\', \\'4239,4.0,Blow (2001),Crime|Drama\\', \"4246,4.0,Bridget Jones\\'s Diary (2001),Comedy|Drama|Romance\", \\'429,3.0,Cabin Boy (1994),Comedy\\', \\'4304,4.0,Startup.com (2001),Documentary\\', \\'4306,5.0,Shrek (2001),Adventure|Animation|Children|Comedy|Fantasy|Romance\\', \\'4308,4.0,Moulin Rouge (2001),Drama|Musical|Romance\\', \\'4310,3.0,Pearl Harbor (2001),Action|Drama|Romance|War\\', \"432,2.0,City Slickers II: The Legend of Curly\\'s Gold (1994),Adventure|Comedy|Western\", \\'435,1.0,Coneheads (1993),Comedy|Sci-Fi\\', \\'4370,2.0,A.I. Artificial Intelligence (2001),Adventure|Drama|Sci-Fi\\', \\'440,3.0,Dave (1993),Comedy|Romance\\', \\'4447,3.0,Legally Blonde (2001),Comedy|Romance\\', \\'45,4.0,To Die For (1995),Comedy|Drama|Thriller\\', \\'454,2.0,\"Firm, The (1993)\",Drama|Thriller\\', \\'4641,4.0,Ghost World (2001),Comedy|Drama\\', \\'4643,3.0,Planet of the Apes (2001),Action|Adventure|Drama|Sci-Fi\\', \\'4649,4.0,Wet Hot American Summer (2001),Comedy\\', \\'4701,2.0,Rush Hour 2 (2001),Action|Comedy\\', \\'471,2.0,\"Hudsucker Proxy, The (1994)\",Comedy\\', \\'4718,4.0,American Pie 2 (2001),Comedy\\', \"472,2.0,I\\'ll Do Anything (1994),Comedy|Drama\", \\'4738,4.0,Happy Accidents (2000),Romance|Sci-Fi\\', \"477,3.0,What\\'s Love Got to Do with It? (1993),Drama|Musical\", \\'4773,4.0,Haiku Tunnel (2001),Comedy\\', \\'4848,4.0,Mulholland Drive (2001),Crime|Drama|Film-Noir|Mystery|Thriller\\', \\'485,1.0,Last Action Hero (1993),Action|Adventure|Comedy|Fantasy\\', \\'4881,3.0,\"Man Who Wasn\\\\\\'t There, The (2001)\",Crime|Drama\\', \\'4886,5.0,\"Monsters, Inc. (2001)\",Adventure|Animation|Children|Comedy|Fantasy\\', \\'4890,1.0,Shallow Hal (2001),Comedy|Fantasy|Romance\\', \"4896,4.0,Harry Potter and the Sorcerer\\'s Stone (a.k.a. Harry Potter and the Philosopher\\'s Stone) (2001),Adventure|Children|Fantasy\", \\'497,3.0,Much Ado About Nothing (1993),Comedy|Romance\\', \\'4973,5.0,\"Amelie (Fabuleux destin d\\\\\\'Amélie Poulain, Le) (2001)\",Comedy|Romance\\', \\'4979,5.0,\"Royal Tenenbaums, The (2001)\",Comedy|Drama\\', \\'4993,4.0,\"Lord of the Rings: The Fellowship of the Ring, The (2001)\",Adventure|Fantasy\\', \\'500,2.0,Mrs. Doubtfire (1993),Comedy|Drama\\', \\'5013,4.0,Gosford Park (2001),Comedy|Drama|Mystery\\', \\'5026,1.0,\"Brotherhood of the Wolf (Pacte des loups, Le) (2001)\",Action|Mystery|Thriller\\', \\'505,2.0,North (1994),Comedy\\', \\'508,3.0,Philadelphia (1993),Drama\\', \\'5222,4.0,Kissing Jessica Stein (2001),Comedy|Romance\\', \\'5225,4.0,And Your Mother Too (Y tu mamá también) (2001),Drama|Romance\\', \\'5265,3.0,Death to Smoochy (2002),Comedy|Crime|Drama\\', \"527,5.0,Schindler\\'s List (1993),Drama|War\", \\'529,2.0,Searching for Bobby Fischer (1993),Drama\\', \\'5299,3.0,My Big Fat Greek Wedding (2002),Comedy|Romance\\', \\'532,2.0,Serial Mom (1994),Comedy|Crime|Horror\\', \\'5324,1.0,Life or Something Like It (2002),Comedy|Romance\\', \\'5349,5.0,Spider-Man (2002),Action|Adventure|Sci-Fi|Thriller\\', \\'5377,4.0,About a Boy (2002),Comedy|Drama|Romance\\', \\'5378,3.0,Star Wars: Episode II - Attack of the Clones (2002),Action|Adventure|Sci-Fi|IMAX\\', \\'538,4.0,Six Degrees of Separation (1993),Drama\\', \\'5388,5.0,Insomnia (2002),Action|Crime|Drama|Mystery|Thriller\\', \\'5400,1.0,\"Sum of All Fears, The (2002)\",Drama|Thriller\\', \\'5415,4.0,Divine Secrets of the Ya-Ya Sisterhood (2002),Comedy|Drama\\', \\'543,2.0,So I Married an Axe Murderer (1993),Comedy|Romance|Thriller\\', \\'5445,4.0,Minority Report (2002),Action|Crime|Mystery|Sci-Fi|Thriller\\', \\'5451,2.0,Pumpkin (2002),Comedy|Drama|Romance\\', \\'5459,3.0,Men in Black II (a.k.a. MIIB) (a.k.a. MIB 2) (2002),Action|Comedy|Sci-Fi\\', \\'5460,3.0,\"Powerpuff Girls, The (2002)\",Action|Animation|Children|Comedy\\', \\'5477,4.0,Sex and Lucia (Lucía y el sexo) (2001),Drama|Romance\\', \\'5481,3.0,Austin Powers in Goldmember (2002),Comedy\\', \\'5502,4.0,Signs (2002),Horror|Sci-Fi|Thriller\\', \\'5505,4.0,\"Good Girl, The (2002)\",Comedy|Drama\\', \\'551,4.0,\"Nightmare Before Christmas, The (1993)\",Animation|Children|Fantasy|Musical\\', \\'5525,4.0,Mostly Martha (Bella Martha) (2001),Comedy|Drama|Romance\\', \\'5528,4.0,One Hour Photo (2002),Drama|Thriller\\', \\'553,1.0,Tombstone (1993),Action|Drama|Western\\', \"5532,3.0,Amy\\'s O (a.k.a. Amy\\'s Orgasm) (2001),Comedy|Romance\", \\'5618,5.0,Spirited Away (Sen to Chihiro no kamikakushi) (2001),Adventure|Animation|Fantasy\\', \\'562,2.0,Welcome to the Dollhouse (1995),Comedy|Drama\\', \\'5620,3.0,Sweet Home Alabama (2002),Comedy|Romance\\', \\'5669,5.0,Bowling for Columbine (2002),Documentary\\', \\'5673,5.0,Punch-Drunk Love (2002),Comedy|Drama|Romance\\', \\'574,4.0,Spanking the Monkey (1994),Comedy|Drama\\', \\'5816,4.0,Harry Potter and the Chamber of Secrets (2002),Adventure|Fantasy\\', \\'5829,4.0,Men with Brooms (2002),Comedy|Drama|Romance\\', \\'585,3.0,\"Brady Bunch Movie, The (1995)\",Comedy\\', \\'586,1.0,Home Alone (1990),Children|Comedy\\', \\'588,3.0,Aladdin (1992),Adventure|Animation|Children|Comedy|Musical\\', \\'589,1.0,Terminator 2: Judgment Day (1991),Action|Sci-Fi\\', \\'5902,5.0,Adaptation (2002),Comedy|Drama|Romance\\', \\'595,3.0,Beauty and the Beast (1991),Animation|Children|Fantasy|Musical|Romance|IMAX\\', \\'5952,3.0,\"Lord of the Rings: The Two Towers, The (2002)\",Adventure|Fantasy\\', \\'5956,4.0,Gangs of New York (2002),Crime|Drama\\', \\'5971,5.0,My Neighbor Totoro (Tonari no Totoro) (1988),Animation|Children|Drama|Fantasy\\', \\'5989,5.0,Catch Me If You Can (2002),Crime|Drama\\', \\'603,2.0,\"Bye Bye, Love (1995)\",Comedy\\', \\'608,4.0,Fargo (1996),Comedy|Crime|Drama|Thriller\\', \\'627,3.0,\"Last Supper, The (1995)\",Drama|Thriller\\', \"63,2.0,Don\\'t Be a Menace to South Central While Drinking Your Juice in the Hood (1996),Comedy|Crime\", \\'661,3.0,James and the Giant Peach (1996),Adventure|Animation|Children|Fantasy|Musical\\', \\'671,3.0,Mystery Science Theater 3000: The Movie (1996),Comedy|Sci-Fi\\', \\'69,2.0,Friday (1995),Comedy\\', \\'691,2.0,Mrs. Winterbourne (1996),Comedy|Romance\\', \\'700,3.0,Angus (1995),Comedy\\', \\'708,3.0,\"Truth About Cats & Dogs, The (1996)\",Comedy|Romance\\', \\'724,4.0,\"Craft, The (1996)\",Drama|Fantasy|Horror|Thriller\\', \\'766,3.0,I Shot Andy Warhol (1996),Drama\\', \\'784,3.0,\"Cable Guy, The (1996)\",Comedy|Thriller\\', \\'785,2.0,Kingpin (1996),Comedy\\', \\'805,3.0,\"Time to Kill, A (1996)\",Drama|Thriller\\', \"829,3.0,Joe\\'s Apartment (1996),Comedy|Fantasy|Musical\", \\'832,3.0,Ransom (1996),Crime|Thriller\\', \\'851,3.0,Basquiat (1996),Drama\\', \\'892,3.0,Twelfth Night (1996),Comedy|Drama|Romance\\', \\'913,4.0,\"Maltese Falcon, The (1941)\",Film-Noir|Mystery\\']')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "92fb48d43aaae2f26302e5276816a92e",
     "grade": false,
     "grade_id": "comm",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "The first 10 lines and the last 10 lines in `movies_rated_by_user22.csv` should look like this:\n",
    "\n",
    "```python\n",
    "!head ~/w6p3/movies_rated_by_user22.csv\n",
    "```\n",
    "`1049,3.5,\"Ghost and the Darkness, The (1996)\",Action|Adventure\n",
    "1084,3.5,Bonnie and Clyde (1967),Crime|Drama\n",
    "1090,1.5,Platoon (1986),Drama|War\n",
    "1120,4.0,\"People vs. Larry Flynt, The (1996)\",Comedy|Drama\n",
    "1196,2.5,Star Wars: Episode V - The Empire Strikes Back (1980),Action|Adventure|Sci-Fi\n",
    "1214,1.5,Alien (1979),Horror|Sci-Fi\n",
    "1215,5.0,Army of Darkness (1993),Action|Adventure|Comedy|Fantasy|Horror\n",
    "1261,2.5,Evil Dead II (Dead by Dawn) (1987),Action|Comedy|Fantasy|Horror\n",
    "1262,2.0,\"Great Escape, The (1963)\",Action|Adventure|Drama|War\n",
    "1263,4.5,\"Deer Hunter, The (1978)\",Drama|War`\n",
    "\n",
    "```python\n",
    "!tail ~/w6p3/movies_rated_by_user22.csv\n",
    "```\n",
    "`7173,3.0,Along Came Polly (2004),Comedy|Romance\n",
    "736,2.5,Twister (1996),Action|Adventure|Romance|Thriller\n",
    "7367,4.0,\"Ladykillers, The (2004)\",Comedy|Crime\n",
    "7437,3.5,Connie and Carla (2004),Comedy\n",
    "7438,4.5,Kill Bill: Vol. 2 (2004),Action|Drama|Thriller\n",
    "761,2.0,\"Phantom, The (1996)\",Action|Adventure\n",
    "8010,4.0,\"Power of One, The (1992)\",Drama\n",
    "833,3.0,High School High (1996),Comedy\n",
    "8531,0.5,White Chicks (2004),Action|Comedy|Crime\n",
    "919,2.0,\"Wizard of Oz, The (1939)\",Adventure|Children|Fantasy|Musical`\n",
    "\n",
    "### 4. Find same Comedy ratings from the two users\n",
    "\n",
    "In the following code cell: \n",
    "\n",
    "- Do a comparision between `movies_rated_by_user22.csv` and `movies_rated_by_user66.csv` to find the movies that both users have given the same rating scores. In other words, find the lines shared by both files. Discard the lines that are unique to either file. \n",
    "- Remove the `movieId` column (the first column). Be careful about lines like `5.0,\"Royal Tenenbaums, The (2001)\",Comedy|Drama`, since Unix will treat the string `\"Royal Tenenbaums, The (2001)\"` as two fields separated by a comma.\n",
    "- Extract the movie lines whose `genres` column contain the string `Comedy`. For example, a line whose `genres` is `Action|Comedy|Sci-Fi` or `Comedy|Drama` should be kept; a line whose `genres` is `Crime|Drama` should be removed.\n",
    "- Sort the lines by `rating` in a descending numeric order (from 5.0 to 0.0) and then sort the rest columns in the default order (ascending string order).\n",
    "- Save your output to `~/w6p3/` as `common_ratings.csv`.\n",
    "- It is recommended to use `sed` and simple regular expressions, but many other Unix commands can solve this problem as well. \n",
    "\n",
    "A sample output is provided in the assertion test cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "3d43c940868b6f4099135d245fdb94e4",
     "grade": false,
     "grade_id": "comm_a",
     "locked": false,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# YOUR CODE HERE\n",
    "# We need to sort both the docs again before use comm, use cut to extract columns from the second one\n",
    "#use grep to extract lines contain the string \"Comedy\".\n",
    "#use sort with option --key=1,1 to sort only the first column, and by default the rest columns will be sorted in the default order.\n",
    "#save the output to ~/w6p3/ as common_ratings.csv.\n",
    "comm -12 <(sort ~/w6p3/movies_rated_by_user22.csv) <(sort ~/w6p3/movies_rated_by_user66.csv) | cut -d',' -f 2- | grep 'Comedy' | sort -t',' --key=1,1nr > ~/w6p3/common_ratings.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "e61ca4c9943ddcafa164fc2f57811b8a",
     "grade": true,
     "grade_id": "comm_t",
     "locked": true,
     "points": 10,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# assertions\n",
    "common = !cat ~/w6p3/common_ratings.csv\n",
    "assert_equal(len(common), 5)\n",
    "common_stripped = [m.strip('\"') for m in common]\n",
    "assert_equal(common_stripped, \n",
    "             ['5.0,\"Royal Tenenbaums, The (2001)\",Comedy|Drama',\n",
    "              '4.0,American Pie 2 (2001),Comedy',\n",
    "              \"4.0,Bridget Jones's Diary (2001),Comedy|Drama|Romance\",\n",
    "              '4.0,Pulp Fiction (1994),Comedy|Crime|Drama|Thriller',\n",
    "              '3.0,Men in Black (a.k.a. MIB) (1997),Action|Comedy|Sci-Fi'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "03beb0e258a6952f45cd1b597cc4e51d",
     "grade": false,
     "grade_id": "bonus",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "### Bonus (for fun):\n",
    "\n",
    "The code cell below converts the `common_ratings.csv` file to [ROT13 encoding](https://en.wikipedia.org/wiki/ROT13), which is mentioned in Chapter 20 of [The Linux Command Line(PDF)](https://sourceforge.net/projects/linuxcommand/?source=dlp) as our Lesson 3 readings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "f1b2da19ef06cf930615d4f06fc80855",
     "grade": false,
     "grade_id": "rot13",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0,\"Eblny Graraonhzf, Gur (2001)\",Pbzrql|Qenzn\r",
      "\r\n",
      "4.0,Nzrevpna Cvr 2 (2001),Pbzrql\r",
      "\r\n",
      "4.0,Oevqtrg Wbarf'f Qvnel (2001),Pbzrql|Qenzn|Ebznapr\r",
      "\r\n",
      "4.0,Chyc Svpgvba (1994),Pbzrql|Pevzr|Qenzn|Guevyyre\r",
      "\r\n",
      "3.0,Zra va Oynpx (n.x.n. ZVO) (1997),Npgvba|Pbzrql|Fpv-Sv\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat ~/w6p3/common_ratings.csv| tr a-zA-Z n-za-mN-ZA-M"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": false,
    "nbgrader": {
     "checksum": "0f883bb2bd7ab721c31808b88fcb20ae",
     "grade": false,
     "grade_id": "cleanup",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "## Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": false,
    "nbgrader": {
     "checksum": "ff4d5e091157ff895061e224a3bcd90c",
     "grade": false,
     "grade_id": "rm",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "!rm -rf ~/w6p3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}