unit08_svm/lab_emnist_partial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab: SVMs on Extended MNIST\n",
    "\n",
    "In the [MNIST demo](demo_mnist_svm.ipynb), we saw how SVMs can be used for the classic MNIST problem of digit recognition. In this lab, we are going to extend the MNIST dataset by adding a number of non-digit letters and see if the classifier can distinguish the digits from the non-digits. All non-digits will be lumped as a single 11-th class. This is a highly simplified version of 'detection' problem (as opposed to 'classification' problem). Detection is vital in OCR and related problems since the non useful characters must be rejected. \n",
    "\n",
    "In addition to the concepts in the demo, you will learn:\n",
    "* Combine multiple datasets\n",
    "* Select the SVM parameters (`C` and `gamma`) via cross-validation.\n",
    "* Use the `GridSearchCV` method to search for parameters with cross-validation.\n",
    "\n",
    "Note:  An [earlier version](lab_emnist_2017_partial.ipynb) of this lab made you manually create the combined letter and digit data.  In this lab, we will download the data from NIST website.  But, the old lab is still useful to look at if you want to see how to use `skimage` package for a number of image pre-processing tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we download the standard packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from sklearn import linear_model, preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the EMNIST Dataset\n",
    "\n",
    "After creating the highly popular MNIST dataset, NIST created an extended version of the dataset to include letters and digits.     The extended datase (called EMNIST) also has many more examples per class.  \n",
    "\n",
    "To download the data, first go to the [EMNIST webpage](https://www.nist.gov/itl/iad/image-group/emnist-dataset).  Near the bottom, you will see a link for `MATLAB format dataset`.  If you click on this link, you will download a `zip` file with several datasets in it.  The total file is 726M, so it may take some time and diskspace to download.  Extract two files:\n",
    "* `emnist-digits.mat`:  This is a file of digits `0` to `9`, but with more examples per class.\n",
    "* `emnist-letters.mat`:  This is a file of letters `a/A` to `z/Z`.  The lower and upper case letters are grouped into the same class.\n",
    "\n",
    "Once you get these two files, you can save yourself the diskspace and remove all the other files.\n",
    "\n",
    "You can download the files manually, or you can run the following commands which will download the files automatically.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "import requests\n",
    "import os\n",
    "import zipfile\n",
    "\n",
    "def download_file(src_url, dst_fn):\n",
    "    \n",
    "    if os.path.exists(dst_fn):\n",
    "        print('File %s already exists' % dst_fn)\n",
    "        return\n",
    "    \n",
    "    print('Downloading %s' % dst_fn)\n",
    "    \n",
    "    # Streaming, so we can iterate over the response.\n",
    "    r = requests.get(src_url, stream=True)\n",
    "\n",
    "    # Total size in MB.\n",
    "    total_size = int(r.headers.get('content-length', 0)); \n",
    "    block_size = 1024\n",
    "    wrote = 0 \n",
    "    with open(dst_fn, 'wb') as f:\n",
    "        with tqdm(total=total_size//block_size, unit='kB', \n",
    "                           unit_scale=True, unit_divisor=1024) as pbar:\n",
    "            for data in r.iter_content(block_size):\n",
    "                wrote = wrote + len(data)\n",
    "                pbar.update(1)\n",
    "                f.write(data)\n",
    "    if total_size != 0 and wrote != total_size:\n",
    "        print(\"ERROR, something went wrong\") \n",
    "\n",
    "# Get file names\n",
    "matlab_dir = 'matlab'\n",
    "digits_fn =  os.path.join(matlab_dir,'emnist-digits.mat')\n",
    "letters_fn = os.path.join(matlab_dir,'emnist-letters.mat')\n",
    "\n",
    "# Check if files exists\n",
    "if os.path.exists(matlab_dir):\n",
    "    if os.path.exists(digits_fn) and os.path.exists(letters_fn):\n",
    "        print('Files already downloaded')\n",
    "        files_exists = True\n",
    "else:\n",
    "    files_exists = False\n",
    "    \n",
    "if not files_exists:\n",
    "    # First download the zip file if needed\n",
    "    src_url = \"http://www.itl.nist.gov/iaui/vip/cs_links/EMNIST/matlab.zip\"\n",
    "    dst_fn = 'matlab.zip'\n",
    "    download_file(src_url, dst_fn)\n",
    "    \n",
    "    # Then, unzip the file\n",
    "    print('Unzipping %s...' % dst_fn)\n",
    "    zip_ref = zipfile.ZipFile(dst_fn, 'r')\n",
    "    zip_ref.extractall('.')\n",
    "    zip_ref.close()\n",
    "    print('Unzip completed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since MATLAB files are still widely-used, Python has excellent routines for loading MATLAB files.  The function below uses the `scipy.io` package to extract the relevant fields from the MATLAB file.  Specifically, the function extracts the training and test data from MATLAB file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import scipy.io\n",
    "def load_emnist(file_path='emnist-digits.mat'):\n",
    "    \"\"\"\n",
    "    Loads training and test data with ntr and nts training and test samples\n",
    "    The `file_path` is the location of the `eminst-balanced.mat`.\n",
    "    \"\"\"    \n",
    "    \n",
    "    # Load the MATLAB file\n",
    "    mat = scipy.io.loadmat(file_path)\n",
    "    \n",
    "    # Get the training data\n",
    "    Xtr = mat['dataset'][0][0][0][0][0][0][:]\n",
    "    ntr = Xtr.shape[0]\n",
    "    ytr = mat['dataset'][0][0][0][0][0][1][:].reshape(ntr).astype(int)\n",
    "    \n",
    "    # Get the test data\n",
    "    Xts = mat['dataset'][0][0][1][0][0][0][:]\n",
    "    nts = Xts.shape[0]\n",
    "    yts = mat['dataset'][0][0][1][0][0][1][:].reshape(nts).astype(int)\n",
    "    \n",
    "    print(\"%d training samples, %d test samples loaded\" % (ntr, nts))\n",
    "\n",
    "    return [Xtr, Xts, ytr, yts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the function above to get all the digit images from the `emnist-digits.mat` file.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Load the digit data from emnist-digits.mat\n",
    "# Xtr_dig, Xts_dig, ytr_dig, yts_dig = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, use the function above to get all the letter characters from the `emnist-letters.mat` file.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Load the digit data from emnist-letters.mat\n",
    "# Xtr_let, Xts_let, ytr_let, yts_let = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the function from the demo to plot the digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plt_digit(x,y=None):\n",
    "    nrow = 28\n",
    "    ncol = 28\n",
    "    xsq = x.reshape((nrow,ncol))\n",
    "    plt.imshow(xsq.T,  cmap='Greys_r')\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])    \n",
    "    if y != None:\n",
    "        plt.title('%d' % y)        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot 8 random samples from the digit training data.  You can use the `plt_digit` function above with `subplot` to create a nice display.  You may want to size your plot with the `plt.figure(figsize=(10,20))` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Plot 8 random samples from the training data of the digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, plot 8 samples from the letters training data.  You should see that the labels go from 0 to 25 corresponding to `a` to `z`.  Upper and lower case letters belong to the same class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Plot 8 random samples from the training data of the letters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Non-Digit Class\n",
    "\n",
    "SVM classifiers are VERY SLOW to train.  The training is particularly slow when there are a large number of classes, since the one classifier must be trained for each pair of labels.  To make the problem easier, we are going to lump all of the letters in one class and add that class to the digits.  \n",
    "\n",
    "Before we begin, we first need to remove all the letters corresponding to `i/I`, `l/L` and `o/O`.  The reason is that these letters would get confused with the digits `0` and `1`.  Create arrays `Xtr_let_rem` and `ytr_let_rem` from the data `Xtr_let` and `ytr_let`, where the samples `i` with `ytr_let[i] == 9, 12` or `15` are removed.   Create `Xts_let_rem` and `yts_let_rem` similarly.\n",
    "\n",
    "If you are clever, you can do this without a for-loop via python broadcasting and `np.all(..., axis=1)` command.  But, you will receive full marks if you use a `for-loop`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "remove_list = np.array([9,12,15])\n",
    "\n",
    "# TODO:  Create arrays with labels 9, 12 and 15 removed\n",
    "# Xtr_let_rem, ytr_let_rem = ...\n",
    "# Xts_let_rem, yts_let_rem = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since training and testing an SVM is VERY SLOW, we will use only a small subset of the training and test data.  Of course, you will not get great results with this small dataset.  But, we can at least illustrate the basic concepts.  \n",
    "\n",
    "Create arrays `Xtr1_dig` and `ytr1_dig` by selecting 5000 random training digit samples from `Xtr_dig` and `ytr_dig`.  Create arrays `Xtr1_let` and `ytr1_let` by selecting 1000 random training letter samples from `Xtr_let_rem` and `ytr_let_rem`.  Similarly, create test arrays `Xts1_dig,Xts1_let,yts1_dig,yts1_let` with 5000 digits and 1000 letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of training and test digits and letters\n",
    "ntr_dig = 5000\n",
    "ntr_let = 1000\n",
    "nts_dig = 5000\n",
    "nts_let = 1000\n",
    "\n",
    "# TODO Create sub-sampled training and test data\n",
    "# Xtr1_dig, ytr1_dig = ...\n",
    "# Xts1_dig, yts1_dig = ...\n",
    "# Xtr1_let, ytr1_let = ...\n",
    "# Xts1_let, yts1_let = ... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create data by combining the digit and letter arrays.\n",
    "* Create an array `Xtr` by stacking `Xtr1_dig`, `Xtr1_let`.  This should result in 6000 total samples.\n",
    "* Create a new label vector `ytr` where `ytr[i] = ytr1_dig[i]` for any digit sample and `ytr[i]=10` for any letter sample.  Thus, all the letters are lumped into a single class with label 11.\n",
    "\n",
    "Create test arrays `Xts` and `yts` similarly. \n",
    "\n",
    "You may wish to use the `np.hstack` and `np.vstack` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Create combined letter and digit training and test data\n",
    "# Xtr, ytr = ..\n",
    "# Xts, yts = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data above takes values from 0 to 255.  Rescale the data from -1 to 1.  This will get slightly better performance on the SVM.  Save the scaled data into arrays `Xtr1` and `Xts1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Rescale the data from -1 to 1\n",
    "# Xtr1 = ...\n",
    "# Xts1 = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the SVM classifier\n",
    "\n",
    "First create the SVM classifer. Use an `rbf` classifier with `C=2.8` and `gamma=.0073`. We will look at how to select these parameters laters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "# TODO:  Create a classifier: a support vector classifier\n",
    "# svc = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the classifier using the scaled training data.  SVMs are insanely slow to train.  But, in this lab, we have kept the training size very small. So, the fitting should take about a minute or two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Fit the classifier on the training data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measure the accuracy on the test data.  This too will take another huge amount of time.  Print the accuracy.  If you did everything right, you should get an accuracy of around 89%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Measure error on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error rate is quite a bit higher than what we got in the digits only case.  Actually, had we done a classifier using all 36 labels instead of collapsing the letters to a single class, the SVM classifier would have done much better.  The reason is that the \"letters\" class is now extremely complex.  \n",
    "\n",
    "Print a confusion matrix.  You should see that the error rate on the \"letters\" class is much higher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Print a confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print:\n",
    "* What fraction of digits are mislabeled as letters?  \n",
    "* What fraction of letters are mislabeled as digits?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Print above two error rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting gamma and C via Cross-Validation (Using For-Loops)\n",
    "\n",
    "In the above example, and in the demo, we used a given `gamma` and `C` value.  The selection of the parameters depend on the problem and decent performance of the SVM requires that you select these parameters carefully.  The best way to select the parameters is via cross validation.  Specifically, generally, one tries different values of `gamma` and `C` and selects the pair of values the lowest test error rate.\n",
    "\n",
    "In the code below, we will try to use 3 values for `C` and `gamma` as specified in the arrays `C_test` and `gam_test`.  For each `C` and `gamma` in these arrays, fit a model on the training data and measure the accuracy on the test data.  Then, print the `C` and `gamma` that result in the best accuracy.   \n",
    "\n",
    "Normally, you would try a large number of values for each of the parameters, but an SVM is very slow to train -- even with this small data set.  So, we will just do 3 values of each.  Even then, this could take 30 minutes or so to complete. \n",
    "\n",
    "In this lab, you may do the parameter search over `C` and `gamma` in one of two ways:\n",
    "* This section:  Use for loops and manually search over the parameters.  This is more direct and you will see and control exactly what is happening.\n",
    "* Next section:  Use the `GridSearchCV` method in the `sklearn` package.  This takes a little reading, but once you learn this method, you can more easily use this for complex parameter searches.\n",
    "\n",
    "**You only need to submit the solutions to one of the two sections.**  Pick whichever one you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "C_test = [0.1,1,10]\n",
    "gam_test = [0.001,0.01,0.1]\n",
    "\n",
    "nC = len(C_test)\n",
    "ngam = len(gam_test)\n",
    "acc = np.zeros((nC,ngam))\n",
    "\n",
    "# TODO:  Measure and print the accuracy for each C and gamma value.  Store the results in acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Print the accuracy matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Print the maximum accuracy and the corresponding best C and gamma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Using `GridSearchCV` (Optional Section)\n",
    "\n",
    "\n",
    "In the previous section, you would have likely used `for-loops` to search over the different `C` and `gamma` values.  Since this type of parameter search is so commonly used, `sklearn` has an excellent method `GridSearchCV` that can perform all the operations for you.  In this lab, `GridSearchCV` is not that useful.  But, once you get to more complex parameter searches, the `GridSearchCV` method can save you writing a lot of code.  Importantly, `GridSearchCV` supports parallelization so that fits with different parameters can be fit at the same time.  In this optional section, we will show how to use this method.  \n",
    "\n",
    "**You do not have to do this section, if you did the previous section**.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `GridSearchCV` method does the train-test split in addition to the parameter search.  In this case, you have already a fixed train-test split.  So, you first need to combine the train and test data back into a single dataset.\n",
    "\n",
    "Create arrays `X` and `y` from `Xtr1`, `Xts1`, `ytr` and `yts`.  Use `np.vstack` and `np.hstack`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Create combined trained and test data X and y.\n",
    "# X = ...\n",
    "# y = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, `GridSearchCV` will do $K$-fold validation and automatically split the data into training and test in each fold.  But, in this case, we want it to perform only one fold with a specific train-test split.  To do this, we need to do the following:\n",
    "* Create a vector `test_fold` where `test_fold[i] = -1` for the samples `i` in the training data (this indicates that they should not be used as test data in any fold) and `test_fold[i] = 0` for the samples `i` in the test data (this indicates that they should be as test data in fold 0).\n",
    "* Call the method  `ps = sklearn.model_selection.PredefinedSplit(test_fold)` to create a predefined test split object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Create a pre-defined test split object \n",
    "# import sklearn.model_selection\n",
    "# test_fold = ...\n",
    "# ps = sklearn.model_selection.PredefinedSplit(test_fold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, read about the `GridSearchCV` method to set up a classifier that includes searching over the parameter grid.  \n",
    "* For the `param_grid` parameter, you will want to create a dictionary to search over `C` and `gamma`.  You will also need to select the `kernel` parameter.\n",
    "* Set `cv = ps` to use the fixed train-test split.\n",
    "* Set `verbose=10` to monitor the progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Create a GridSearchCV classifier\n",
    "# clf = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the classifier using the `fit` method.  The fit method will now search over all the parameters. This will take about 30 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Fit the classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the `best_score_` and `best_params_` attributes of the classifier to find the best score and parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Print the best parameter and score of the classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can print the test and train score from the `cv_results_['mean_test_score']` and `cv_results_['mean_train_score']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO:  Print the mean test score for each parameter value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}