Week2_Assignment/Problem_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "e47273d4bd70fdd02407683134b2fdda",
     "grade": false,
     "grade_id": "intro",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "# Week 2 Problem 1\n",
    "\n",
    "If you are not using the `Assignments` tab on the course JupyterHub server to read this notebook, read [Activating the assignments tab](https://github.com/UI-DataScience/info490-fa16/blob/master/Week2/assignments/README.md).\n",
    "\n",
    "A few things you should keep in mind when working on assignments:\n",
    "\n",
    "1. Make sure you fill in any place that says `YOUR CODE HERE`. Do **not** write your answer in anywhere else other than where it says `YOUR CODE HERE`. Anything you write anywhere else will be removed or overwritten by the autograder.\n",
    "\n",
    "2. Before you submit your assignment, make sure everything runs as expected. Go to menubar, select _Kernel_, and restart the kernel and run all cells (_Restart & Run all_).\n",
    "\n",
    "3. Do not change the title (i.e. file name) of this notebook.\n",
    "\n",
    "4. Make sure that you save your work (in the menubar, select _File_ → _Save and CheckPoint_)\n",
    "\n",
    "5. You are allowed to submit an assignment multiple times, but only the most recent submission will be graded.\n",
    "\n",
    "***\n",
    "\n",
    "The purpose of this problem is twofold:\n",
    "\n",
    "1. We want you to gain experience executing Unix commands\n",
    "2. We want to demonstrate how that can be done from a Jupyter Notebook\n",
    "\n",
    "Hopefully you've already gained some experience executing unix commands from the [Lesson 1 Readings](https://github.com/UI-DataScience/info490-fa16/blob/master/Week2/lesson1.md). There are several ways to execute the same Unix commands from a Jupyter notebook, many of which we will not cover.\n",
    "\n",
    "#### Method 1: Cell Magics\n",
    "Cell [magics](https://ipython.org/ipython-doc/3/interactive/magics.html) start with a double percent (`%%`) and affect the entire cell. One we might be interested in is the `%%bash` cell magic which allows us to effectively transform a code cell into a bash script. Don't worry if you don't know what all of the following code does, it is simply for demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "ef3cd2085b66aa6887c1584da85bb774",
     "grade": false,
     "grade_id": "example_bash",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_subdir01\n",
      "test_subdir02\n",
      "test_subdir03\n",
      "test_subdir04\n",
      "test_subdir05\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# set a name for the test directory\n",
    "dir='testdir'\n",
    "\n",
    "# go to the home directory\n",
    "cd ~ \n",
    "\n",
    "# make a directory for testing\n",
    "# remove it an remake if it already exists\n",
    "if [[ -e $dir ]]; then\n",
    "    rm -rf $dir \n",
    "fi\n",
    "mkdir $dir\n",
    "\n",
    "# go into that directory\n",
    "cd $dir\n",
    "\n",
    "# make 5 subdirectories if they don't already exist\n",
    "mkdir -p test_subdir{01..5}\n",
    "\n",
    "# list the contents of `dir`\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "d3727c753d6f2d2e0a31411050ea4b8f",
     "grade": false,
     "grade_id": "intro_exclamation",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "#### Method 2: Exclamation Points\n",
    "\n",
    "You can also precede a single shell command with an exclamation mark (`!`) to execute it from Jupyter (or IPython in general)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "396d790de3472e1cb701f7219dc9434a",
     "grade": false,
     "grade_id": "sample_ls",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2001\t      2002.csv.bz2  email\t     nltk_data\r\n",
      "2001.csv      airports.csv  enron-spam\t     plane-data.csv\r\n",
      "2001.csv.bz2  carriers.csv  misc\t     textdm\r\n",
      "2002.csv      delta.csv     ml-latest-small  weather\r\n"
     ]
    }
   ],
   "source": [
    "!ls ~/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "31b36b34a3fc819e65a45210ce6000d4",
     "grade": false,
     "grade_id": "into_storing_variable",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "We can also store the output of a Unix command in a variable. This is how we'll grade this assignment. In a moment, we'll ask you to write a Unix command using the exclamation mark method and store the results in a variable. For now, just follow along."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "eb79f3f88aeb1e29628193c8da42bf03",
     "grade": false,
     "grade_id": "sample_variable",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2001', '2001.csv', '2001.csv.bz2', '2002.csv', '2002.csv.bz2', 'airports.csv', 'carriers.csv', 'delta.csv', 'email', 'enron-spam', 'misc', 'ml-latest-small', 'nltk_data', 'plane-data.csv', 'textdm', 'weather']\n"
     ]
    }
   ],
   "source": [
    "out = !ls ~/data\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "bfa905a8ee6b357253bc31fd72836454",
     "grade": false,
     "grade_id": "clarification",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "To be clear, the output of the Unix command `ls ~/data`, which lists the contents of the `~/data` directory, is now stored in a Python variable called `out`. And if we want to know the *number* of files in the `~/data` directory, we can use Python's [len](http://stackoverflow.com/questions/20860430/what-is-a-len-function-in-python-and-how-would-you-use-it) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "8de5326567aa755cd30b5130bb5731e6",
     "grade": false,
     "grade_id": "length",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "eb22f3e66149b76046c5d736a1b611f3",
     "grade": false,
     "grade_id": "intro_to_wc",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "Equivalently, we could have used unix to accomplish the same task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "deletable": false,
    "nbgrader": {
     "checksum": "a61f64c3d6c5eab15b5cf3105b51eae1",
     "grade": false,
     "grade_id": "example_wc",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "num_files = !ls ~/data | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "18022ccb858fb1863b2767f3335416a1",
     "grade": false,
     "grade_id": "print_example_wc",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['16']\n"
     ]
    }
   ],
   "source": [
    "print(num_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "286292a1ec2fb3827df9181bc9d5f8aa",
     "grade": false,
     "grade_id": "hint",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "You might want to remember the above method of counting the files in a directory, as it may come in handy later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "4d8239e50633f4e63cd9402e4d6743fb",
     "grade": false,
     "grade_id": "problem_1_description",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "# Problem 1\n",
    "\n",
    "Your task is to count the number of files in the `~/data/weather` directory. (Hint: similar setup to how I counted the number of files in `~/data` above.) In the following code cell, write a unix command that lists the files in `~/data/weather` and pipe that command into one that will count the files. Use the exclamation mark method and store your result in a variable called `weather_file_count`. Replace the comment `#your code here` with your unix command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "1e9a81b85441dc9bd7c047b20574dfb2",
     "grade": false,
     "grade_id": "problem_1",
     "locked": false,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "weather_file_count = !ls ~/data/weather | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "617bdc7d194a2dbd13a9e0275669ba39",
     "grade": false,
     "grade_id": "view_p1_answer",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['366']\n"
     ]
    }
   ],
   "source": [
    "print(weather_file_count) # run this cell to view your result!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "c6381749f07e0062b40bdbee3c080678",
     "grade": false,
     "grade_id": "intro_autograding",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "The following cells demonstrate how we will grade these assignments. The first cell makes sure your result is in the proper format, and attempts to fix some common errors in formatting. In the future, when you've learned more Python, you'll be expected to ensure your results are in the proper format yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": false,
    "nbgrader": {
     "checksum": "d5fd83c055bdd5965715cf1d60637461",
     "grade": false,
     "grade_id": "parse_function",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import IPython\n",
    "def parse_result(res):\n",
    "    if type(res) == IPython.utils.text.SList:\n",
    "        res = res[0]\n",
    "    if type(res) == str:\n",
    "        try:\n",
    "            res = int(res)\n",
    "        except ValueError:\n",
    "            print(\"Your code doesn't produce something we can convert to an integer.\\nPlease check your result and try again\")\n",
    "    return(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "9c91657d030eb73d0e82ae93626d34bd",
     "grade": false,
     "grade_id": "intro_unit_tests",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "In the next cell are items called [*unit tests*](https://en.wikipedia.org/wiki/Unit_testing), and are a common way to ensure software is working as expected. We will use them to ensure your result is what we expect it to be, which makes it easier for us to grade but also gives you as the student immediate feedback. **If your code does not pass the unit tests, it will not pass the autograder.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "5e2fb01c8f6b4a5e7b9182594c82c08f",
     "grade": true,
     "grade_id": "answer_p1",
     "locked": true,
     "points": 10,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from nose.tools import assert_equal\n",
    "weather_file_count_parsed = parse_result(weather_file_count)\n",
    "assert_equal(type(weather_file_count_parsed), int)\n",
    "assert_equal(weather_file_count_parsed, 366)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "00948240d854ceea33205501c677279d",
     "grade": false,
     "grade_id": "problem_2_description",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "# Problem 2\n",
    "\n",
    "Now you want to know the total number of *lines* in **all files** in the `~/data/email/spam` directory. (Hint: there are two ways to do this. You could combine the files first and then pipe that result into a command to count the lines, or you could count each file individually and grab the total from the result. **The first approach will require no extra parsing if done correctly, so I recommend that approach.**) You may wish to use the `cat` command.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": false,
    "nbgrader": {
     "checksum": "5b8c2d924720f539c2801f4b168f6c50",
     "grade": false,
     "grade_id": "problem_2",
     "locked": false,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "lines_of_spam = !cat ~/data/email/spam/* | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "78944508f9834fcd775c24c3d350176b",
     "grade": false,
     "grade_id": "view_p2",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['195563']\n"
     ]
    }
   ],
   "source": [
    "print(lines_of_spam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": false,
    "nbgrader": {
     "checksum": "81ebd5f8ef67b69b562bfc01bdc64da2",
     "grade": true,
     "grade_id": "answer_p2",
     "locked": true,
     "points": 10,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "lines_of_spam_parsed = parse_result(lines_of_spam)\n",
    "assert_equal(type(lines_of_spam_parsed), int)\n",
    "assert_equal(lines_of_spam_parsed, 195563)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "checksum": "6cdee3860bfa4567ccd3c384f592a681",
     "grade": false,
     "grade_id": "cleanup",
     "locked": true,
     "solution": false
    }
   },
   "source": [
    "## Cleaning up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": false,
    "nbgrader": {
     "checksum": "dd7faef4e8eca12abab2c7c32b9d9d29",
     "grade": false,
     "grade_id": "clean_testdir",
     "locked": true,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "!rm -rf ~/testdir/ # remove that test directory we made before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}