t81_558_class3_tensor_flow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow.**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 3.1: Neural Network Introduction](https://www.youtube.com/watch?v=W50SiRuu-cs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 3.2: Introduction to Keras and Tensorflow](https://www.youtube.com/watch?v=PpH4KAEyhIM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 3.3: Saving and Loading a Keras Neural Network](https://www.youtube.com/watch?v=dBpUg6JMQpk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "\n",
    "Weekly video update:\n",
    "\n",
    "* [Weekly Update #3, January 29, 2018](https://www.youtube.com/watch?v=OaTNmazref0&t=11s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import os\n",
    "import requests\n",
    "import base64\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name, x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = \"{}-{}\".format(name, tv)\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df.as_matrix(result).astype(np.float32), dummies.as_matrix().astype(np.float32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32), df.as_matrix([target]).astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred,y,sort=True):\n",
    "    t = pd.DataFrame({'pred' : pred, 'y' : y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'],inplace=True)\n",
    "    a = plt.plot(t['y'].tolist(),label='expected')\n",
    "    b = plt.plot(t['pred'].tolist(),label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean()) >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "               * (normalized_high - normalized_low) + normalized_low\n",
    "        \n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Basics\n",
    "\n",
    "Neural networks were one of the first machine learning models.  Their popularity has fallen twice and is now on its third rise.  Deep learning implies the use of neural networks.  The \"deep\" in deep learning refers to a neural network with many hidden layers.  Because neural networks have been around for so long, they have quite a bit of baggage.  Many different training algorithms, activation/transfer functions, and structures have been added over the years.  This course is only concerned with the latest, most current state of the art techniques for deep neural networks.  I am not going to spend any time discussing the history of neural networks.  If you would like to learn about some of the more classic structures of neural networks, there are several chapters dedicated to this in your course book.  For the latest technology, I wrote an article for the Society of Actuaries on deep learning as the [third generation of neural networks](https://www.soa.org/Library/Newsletters/Predictive-Analytics-and-Futurism/2015/december/paf-iss12.pdf).\n",
    "\n",
    "Neural networks accept input and produce output.  The input to a neural network is called the feature vector.  The size of this vector is always a fixed length.  Changing the size of the feature vector means recreating the entire neural network.  Though the feature vector is called a \"vector,\" this is not always the case.  A vector implies a 1D array.  Historically the input to a neural network was always 1D.  However, with modern neural networks you might see inputs, such as:\n",
    "\n",
    "* **1D Vector** - Classic input to a neural network, similar to rows in a spreadsheet.  Common in predictive modeling.\n",
    "* **2D Matrix** - Grayscale image input to a convolutional neural network (CNN).\n",
    "* **3D Matrix** - Color image input to a convolutional neural network (CNN).\n",
    "* **nD Matrix** - Higher order input to a CNN.\n",
    "\n",
    "Prior to CNN's, the image input was sent to a neural network simply by squashing the image matrix into a long array by placing the image's rows side-by-side.  CNNs are different, as the nD matrix literally passes through the neural network layers.\n",
    "\n",
    "Initially this course will focus upon 1D input to neural networks.  However, later sessions will focus more heavily upon higher dimension input.\n",
    "\n",
    "**Dimensions** The term dimension can be confusing in neural networks.  In the sense of a 1D input vector, dimension refers to how many elements are in that 1D array.  For example a neural network with 10 input neurons has 10 dimensions.  However, now that we have CNN's, the input has dimensions too.  The input to the neural network will *usually* have 1, 2 or 3 dimensions.  4 or more dimensions is unusual.  You might have a 2D input to a neural network that has 64x64 pixels.  This would result in 4,096 input neurons.  This network is either 2D or 4,096D, depending on which set of dimensions you are talking about!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification or Regression\n",
    "\n",
    "Like many models, neural networks can function in classification or regression:\n",
    "\n",
    "* **Regression** - You expect a number as your neural network's prediction.\n",
    "* **Classification** - You expect a class/category as your neural network's prediction.\n",
    "\n",
    "The following shows a classification and regression neural network:\n",
    "\n",
    "![Neural Network Classification and Regression](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann_class_reg.png \"Neural Network Classification and Regression\")\n",
    "\n",
    "Notice that the output of the regression neural network is numeric and the output of the classification is a class.  Regression, or two-class classification, networks always have a single output.  Classification neural networks have an output neuron for each class. \n",
    "\n",
    "The following diagram shows a typical neural network:\n",
    "\n",
    "![Feedforward Neural Networks](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann.png \"Feedforward Neural Networks\")\n",
    "\n",
    "There are usually four types of neurons in a neural network:\n",
    "\n",
    "* **Input Neurons** - Each input neuron is mapped to one element in the feature vector.\n",
    "* **Hidden Neurons** - Hidden neurons allow the neural network to abstract and process the input into the output.\n",
    "* **Output Neurons** - Each output neuron calculates one part of the output.\n",
    "* **Context Neurons** - Holds state between calls to the neural network to predict.\n",
    "* **Bias Neurons** - Work similar to the y-intercept of a linear equation.  \n",
    "\n",
    "These neurons are grouped into layers:\n",
    "\n",
    "* **Input Layer** - The input layer accepts feature vectors from the dataset.  Input layers usually have a bias neuron.\n",
    "* **Output Layer** - The output from the neural network.  The output layer does not have a bias neuron.\n",
    "* **Hidden Layers** - Layers that occur between the input and output layers.  Each hidden layer will usually have a bias neuron.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neuron Calculation\n",
    "\n",
    "The output from a single neuron is calculated according to the following formula:\n",
    "\n",
    "$ f(x,\\theta) = \\phi(\\sum_i(\\theta_i \\cdot x_i)) $\n",
    "\n",
    "The input vector ($x$) represents the feature vector and the vector $\\theta$ (theta) represents the weights. To account for the bias neuron, a value of 1 is always appended to the end of the input feature vector.  This causes the last weight to be interpreted as a bias value that is simply added to the summation. The $\\phi$ (phi) is the transfer/activation function. \n",
    "\n",
    "Consider using the above equation to calculate the output from the following neuron:\n",
    "\n",
    "![Single Neuron](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_abstract_nn.png \"Single Neuron\")\n",
    "\n",
    "The above neuron has 2 inputs plus the bias as a third.  This neuron might accept the following input feature vector:\n",
    "\n",
    "```\n",
    "[1,2]\n",
    "```\n",
    "\n",
    "To account for the bias neuron, a 1 is appended, as follows:\n",
    "\n",
    "```\n",
    "[1,2,1]\n",
    "```\n",
    "\n",
    "The weights for a 3-input layer (2 real inputs + bias) will always have an additional weight, for the bias.  A weight vector might be:\n",
    "\n",
    "```\n",
    "[ 0.1, 0.2, 0.3]\n",
    "```\n",
    "\n",
    "To calculate the summation, perform the following:\n",
    "\n",
    "```\n",
    "0.1*1 + 0.2*2 + 0.3*3 = 1.4\n",
    "```\n",
    "\n",
    "The value of 1.4 is passed to the $\\phi$ (phi) function, which represents the activation function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activation Functions\n",
    "\n",
    "Activation functions, also known as transfer functions, are used to calculate the output of each layer of a neural network.  Historically neural networks have used a hyperbolic tangent, sigmoid/logistic, or linear activation function.  However, modern deep neural networks primarily make use of the following activation functions:\n",
    "\n",
    "* **Rectified Linear Unit (ReLU)** - Used for the output of hidden layers.\n",
    "* **Softmax** - Used for the output of classification neural networks. [Softmax Example](http://www.heatonresearch.com/aifh/vol3/softmax.html)\n",
    "* **Linear** - Used for the output of regression neural networks (or 2-class classification).\n",
    "\n",
    "The ReLU function is calculated as follows:\n",
    "\n",
    "$ \\phi(x) = \\max(0, x) $\n",
    "\n",
    "The Softmax is calculated as follows:\n",
    "\n",
    "$ \\phi_i(z) = \\frac{e^{z_i}}{\\sum\\limits_{j \\in group}e^{z_j}} $\n",
    "\n",
    "The Softmax activation function is only useful with more than one output neuron.  It ensures that all of the output neurons sum to 1.0.  This makes it very useful for classification where it shows the probability of each of the classes as being the correct choice.\n",
    "\n",
    "To experiment with the Softmax, click [here](http://www.heatonresearch.com/aifh/vol3/softmax.html).\n",
    "\n",
    "The linear activation function is essentially no activation function:\n",
    "\n",
    "$ \\phi(x) = x $\n",
    "\n",
    "For regression problems, this is the activation function of choice.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why ReLU?\n",
    "\n",
    "Why is the ReLU activation function so popular?  It was one of the key improvements to neural networks that makes deep learning work. Prior to deep learning, the sigmoid activation function was very common:\n",
    "\n",
    "$ \\phi(x) = \\frac{1}{1 + e^{-x}} $\n",
    "\n",
    "The graph of the sigmoid function is shown here:\n",
    "\n",
    "![Sigmoid Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_sigmoid.png \"Sigmoid Activation Function\")\n",
    "\n",
    "Neural networks are often trained using gradient descent.  To make use of gradient descent, it is necessary to take the derivative of the activation function.  This allows the partial derivatives of each of the weights to be calculated with respect to the error function.  A derivative is the instantaneous rate of change:\n",
    "\n",
    "![Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv.png \"Derivative\")\n",
    "\n",
    "The derivative of the sigmoid function is given here:\n",
    "\n",
    "$ \\phi'(x)=\\phi(x)(1-\\phi(x)) $\n",
    "\n",
    "This derivative is often given in other forms.  The above form is used for computational efficiency. To see how this derivative was taken, see [this](http://www.heatonresearch.com/aifh/vol3/deriv_sigmoid.html).\n",
    "\n",
    "The graph of the sigmoid derivative is given here:\n",
    "\n",
    "![Sigmoid Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv_sigmoid.png \"Sigmoid Derivative\")\n",
    "\n",
    "The derivative quickly saturates to zero as *x* moves from zero.  This is not a problem for the derivative of the ReLU, which is given here:\n",
    "\n",
    "$ \\phi'(x) = \\begin{cases} 1 & x > 0 \\\\ 0 & x \\leq 0 \\end{cases} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why are Bias Neurons Needed?\n",
    "\n",
    "The activation functions seen in the previous section specifies the output of a single neuron.  Together, the weight and bias of a neuron shape the output of the activation to produce the desired output.  To see how this process occurs, consider the following equation. It represents a single-input sigmoid activation neural network.\n",
    "\n",
    "$ f(x,w,b) = \\frac{1}{1 + e^{-(wx+b)}} $ \n",
    "\n",
    "The *x* variable represents the single input to the neural network.  The *w* and *b* variables specify the weight and bias of the neural network.  The above equation is a combination of the weighted sum of the inputs and the sigmoid activation function.  For this section, we will consider the sigmoid function because it clearly demonstrates the effect that a bias neuron has.\n",
    "\n",
    "The weights of the neuron allow you to adjust the slope or shape of the activation function.  The following figure shows the effect on the output of the sigmoid activation function if the weight is varied:\n",
    "\n",
    "![Adjusting Weight](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_weight.png \"Bias 1\")\n",
    "\n",
    "The above diagram shows several sigmoid curves using the following parameters:\n",
    "\n",
    "```\n",
    "f(x,0.5,0.0)\n",
    "f(x,1.0,0.0)\n",
    "f(x,1.5,0.0)\n",
    "f(x,2.0,0.0)\n",
    "```\n",
    "\n",
    "To produce the curves, we did not use bias, which is evident in the third parameter of 0 in each case. Using four weight values yields four different sigmoid curves in the above figure. No matter the weight, we always get the same value of 0.5 when x is 0 because all of the curves hit the same point when x is 0.  We might need the neural network to produce other values when the input is near 0.5.  \n",
    "\n",
    "Bias does shift the sigmoid curve, which allows values other than 0.5 when x is near 0. The following figure shows the effect of using a weight of 1.0 with several different biases:\n",
    "\n",
    "\n",
    "![Adjusting Bias](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_value.png \"Bias 1\")\n",
    "\n",
    "The above diagram shows several sigmoid curves with the following parameters:\n",
    "\n",
    "```\n",
    "f(x,1.0,1.0)\n",
    "f(x,1.0,0.5)\n",
    "f(x,1.0,1.5)\n",
    "f(x,1.0,2.0)\n",
    "```\n",
    "\n",
    "We used a weight of 1.0 for these curves in all cases.  When we utilized several different biases, sigmoid curves shifted to the left or right.  Because all the curves merge together at the top right or bottom left, it is not a complete shift.\n",
    "\n",
    "When we put bias and weights together, they produced a curve that created the necessary output from a neuron.  The above curves are the output from only one neuron.  In a complete network, the output from many different neurons will combine to produce complex output patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow - Google\n",
    "\n",
    "![TensorFlow](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_tensorflow.png \"TensorFlow\")\n",
    "\n",
    "TensorFlow is an open source software library for machine learning in various kinds of perceptual and language understanding tasks. It is currently used for both research and production by different teams in many commercial Google products, such as speech recognition, Gmail, Google Photos, and search, many of which had previously used its predecessor DistBelief. TensorFlow was originally developed by the Google Brain team for Google's research and production purposes and later released under the Apache 2.0 open source license on November 9, 2015.\n",
    "\n",
    "* [TensorFlow Homepage](https://www.tensorflow.org/)\n",
    "* [TensorFlow GitHib](https://github.com/tensorflow/tensorflow)\n",
    "* [TensorFlow Google Groups Support](https://groups.google.com/forum/#!forum/tensorflow)\n",
    "* [TensorFlow Google Groups Developer Discussion](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss)\n",
    "* [TensorFlow FAQ](https://www.tensorflow.org/resources/faq)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What version of TensorFlow do you have?\n",
    "\n",
    "TensorFlow is very new and changing rapidly.  It is very important that you run the same version of it that I am using.  For this semester we will use a specific version of TensorFlow (mentioned in the last class notes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Self Driving Car](http://imgc-cn.artprintimages.com/images/P-473-488-90/94/9475/CFB6500Z/posters/paul-noth-does-your-car-have-any-idea-why-my-car-pulled-it-over-new-yorker-cartoon.jpg)\n",
    "\n",
    "[Wrong version of TensorFlow?](https://twitter.com/reza_zadeh/status/849160032608440320)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor Flow Version: 1.4.1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"Tensor Flow Version: {}\".format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing TensorFlow\n",
    "\n",
    "* [IBM Data Scientist Workbench](datascientistworkbench.com) - All platforms, use your browser.\n",
    "* Windows - Supported platform.\n",
    "* Mac - Supported platform.\n",
    "* Linux - Supported platform.\n",
    "\n",
    "[TensorFlow Setup Instructions](https://www.tensorflow.org/versions/r0.10/get_started/os_setup.html#docker-based_installation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why TensorFlow\n",
    "\n",
    "* Supported by Google\n",
    "* Works well on Linux/Mac\n",
    "* Excellent GPU support\n",
    "* Python is an easy to learn programming language\n",
    "* Python is [extremely popular](http://www.kdnuggets.com/2014/08/four-main-languages-analytics-data-mining-data-science.html) in the data science community\n",
    "\n",
    "# Other Deep Learning Tools\n",
    "TensorFlow is not the only only game in town.  These are some of the best supported alternatives.  Most of these are written in C++. In order of my own preference (ordered in my own estimation of approximate importance):\n",
    "\n",
    "* [TensorFlow](https://www.tensorflow.org/) Google's deep learning API.  The focus of this class, along with Keras.\n",
    "* [MXNet](https://mxnet.incubator.apache.org/) Apache foundation's deep learning API. Can be used through Keras.\n",
    "* [Theano](http://deeplearning.net/software/theano/) - Python, from the academics that created deep learning.\n",
    "* [Keras](https://keras.io/) - Also by Google, higher level framework that allows the use of TensorFlow, MXNet and Theano interchangeably.\n",
    "[Torch](http://torch.ch/) is used by Google DeepMind, the Facebook AI Research Group, IBM, Yandex and the Idiap Research Institute.  It has been used for some of the most advanced deep learning projects in the world.  However, it requires the [LUA](https://en.wikipedia.org/wiki/Lua_(programming_language)) programming language.  It is very advanced, but it is not mainstream.  I have not worked with Torch (yet!).\n",
    "* [PaddlePaddle](https://github.com/baidu/Paddle) - [Baidu](http://www.baidu.com/)'s deep learning API.\n",
    "* [Deeplearning4J](http://deeplearning4j.org/) - Java based. Supports all major platforms. GPU support in Java!\n",
    "* [Computational Network Toolkit (CNTK)](https://github.com/Microsoft/CNTK) - Microsoft.  Support for Windows/Linux, command line only.  Bindings for predictions for C#/Python. GPU support.\n",
    "* [H2O](http://www.h2o.ai/) - Java based.  Supports all major platforms.  Limited support for computer vision. No GPU support.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using TensorFlow\n",
    "\n",
    "TensorFlow is a low-level mathematics API, similar to [Numpy](http://www.numpy.org/).  However, unlike Numpy, TensorFlow is built for deep learning.  TensorFlow works by allowing you to define compute graphs with Python.  In this regard, it is similar to [Spark](http://spark.apache.org/).  TensorFlow compiles these compute graphs into highly efficient C++/[CUDA](https://en.wikipedia.org/wiki/CUDA) code.\n",
    "\n",
    "The [TensorBoard](https://www.tensorflow.org/versions/r0.10/how_tos/summaries_and_tensorboard/index.html) command line utility can be used to view these graphs.  The iris neural network's graph used in this class is shown here:\n",
    "\n",
    "![Iris Graph](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_graph_tf.png \"Iris Graph\")\n",
    "\n",
    "Expanding the DNN gives:\n",
    "\n",
    "\n",
    "![Iris DNN Graph](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_graph_dnn.png \"Iris DNN Graph\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Graphs\n",
    "\n",
    "So far we've communicated with TensorFlow using TensorFlow Learn, which allows you to specify the number of hidden layers and simply create the neural network.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a\nHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIy\nMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAIIAlgDASIA\nAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA\nAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3\nODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWm\np6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEA\nAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSEx\nBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElK\nU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3\nuLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDyuiii\nvROUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACii\nlAJOAKAEoxnpUixf3j+VSABegp2Jc0RrET14p4RV7U6ighybCiiimSFFFFABRRRQAUUUUAFFFFAB\nRRRQAUUUUAFFFFABRRRQAUUUUAIQCMEVG0X938qlopDTaK5BBwRSVYIDdRUTRkcjkUrGikmMoooo\nKCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA\nKKKKACiiigAooooAKKKKACiiigAooooAKKKKACjGelPWMnrwKlVQo4osS5JEaxH+LipAABgClopm\nbbYUUUUxBRRRQAUUUUAFFFFABRRRQAUUoBJwKeEA6800rickhgBPQU4R+pqSiqUUZubG7Fpdo9BT\ngpNLsPtVqD7EOfmM2j0FIUWnkEdaSk0NSZGYz25phBHWp6CAetS4lKb6kFFPZO4/KmVDVjRNPYKK\nKKBhRRRQAUUUUAMaMNyODURBU4NWKQgMMGlYpSsV6KcyFfpTaRqncKKKKACiiigAooooAKKKKACi\niigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKK\nkWLu35UCbSGKpbpUyoF9zTsY6UU7Gbk2FFFFMkKKKKACiiigAooooAKKKKACiigDJwKACnKmeT0p\nypjk9afVKPczlPsAAHSiiirMwp6r3NIoyafWkI31IlLoRzS+SudhYdzkYH1qQFWGQRilYBgQRkGs\nq4jFvMiRsyRyHaQOle/h8LRrU7Ws1+JjFcztfU01Kuu5CGHqKQqD7UseAgA6CnEV5+Kw6hNqOwRm\nRFSKSpaQqDXA6fY2U+5HTWUN9aeQRSVm10ZafVEJBBwaSpiARg1GylfpWbjY2jK42iiikUFFFFAB\nRRRQAEAjBqF028jpU1FIaditRT3TbyOlMpGydwooooAKKKKACiiigAooooAKKKKACiiigAooooAK\nKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoooxnpQAU5ULfT1p6x45b8qkp2Ic+wiqF6\nUtFFMzCiiigAooooAKKKKACiiigAooooAKKKeqZ5NCVxNpDVUsalVQopelFaJWMpSbCiiimSFAGT\nRT1GOe9VGN2JuwoGBSjrRSiumEbsxkxay9Rj3Oj5fjrjoPc9q1Kq3FtHMdzKCcY5r2sFs1e1+vYi\nnNQmmylY3jea/muyqBwsi7SPqK1gciufmh2XKxoM91QnAzWnb3QeJS3ykjoa66mHdRckneS38/M2\nr01pOGzL1RySLGVBB+ZtowKpy6kkblCCeM5zgVCss11dqUUIYgThj+HI7VzRwMOZxm/lfUzjSna7\n0RqEVGRj6VIm/wAtfMxvwN23pn2oIyMV4demlJoqEiKgjIwaCMHFFchsRMmOR0ptT1Gydx+VQ49j\nSM+jGUUUVJoFFFFABRRRQAVC6beR0qaggEYNIaditRTnXafam0jbcKKKKACiiigAooooAKKKKACi\niigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW+lTKoXpS0UzNy\nbCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUoBPSnKnc/lUgGBgVSiQ522GqgHuadRRV2M27hRRRQ\nIKKAM9KkC4+tVGLYm7CKvc06iitkraIybuAFOoorrpU3sjKUgqNuhpxaq886xjlgK9jC0XfQhJt6\nFK+jDgnJB5H15qiJN4LOWCk4YL379T+P6VLc3DSKAFPzjK85yM4/pTJkRLdHAwH6JnJVu+fwx2Fb\n4qtBSgoPXun0vb+ux6tGLjG0iW3tfPbc6qF/ugYrWtreOFQEXGKoWVwpAXOSOuB0rUQ9K0xUfZw5\nYbHDiJzbs9h9Np1Ia+brx1IgxjDIplS0xhjntXDOPU3i+g2iiisyxrJu571ERg4NT0jKGFS43LjK\n25DRSlSp5pKg1CiiigAooooARgGGDUBBBwasUyRdwyOopMqLsQ0UUUjUKKKKACiiigAooooAKKKK\nACiiigAooooAKKKKACiiigAooooAKKKKACiiigApyoW6dPWnLF3b8qlp2Ic+w1UC9OvrTqKKZmFF\nFFABRRRQAUUUUAFFFFABRRRQAUU4IT9KeEA+tNRbJckhioT7CpAoXpS0VaVjNybCiiimSFFFKFJp\npXASnBfWnBQKWtFDuQ5dgAx0ooorRLsQFOFAGKK6KdPUzlICcVVurjyYiwP/ANao9QEzKPLfag+9\ngc1ktKSxDozliBmRto7f/X7969ulRVOHPZvTp/XQ1o0Oe0mzQjviw2yYDgc81UvJ9zZVuen4f5xT\nZbeRpEKeWF2hQynIJHXn1ppWG3hDXeVxwVUc5zkZ+o3fgKcsfThRXNG769Leep1wowjLmQkk6WkE\nU7xgyZzgALnnOf1I4B/SqsGpmWaRJtxSRQirnOPTJP8Ah19KbeX9tcWyQIsiqhJHHfHXGee/0B71\nWsJIobgSysRtzgBc54P5H096+Uq4xvEQUZrl0v8Arfbodah7rbWpr/LFcqke7KNg5OR79h05rZjl\nBUc1kCOG7lM6XICOGI45BHbA/OmC6LfIHIB43Hj/AD+dfW4OtRq0HzO1tXf9O/4nFXouo1Y3TcRq\nyqzcscCpqw4sNe7Zc5HTcAMkHqBjgcf55rbByK5cVSi6anBbnFUp+zaQlHWnEU2vFnGzKTuRkY+l\nJUvWmFcdK55QtqjVS7jaKKKgoCAetRMu0+1S0UmrlRlYgopzLjkdKbWbVjVO4UUUUDCiiigCGRcH\nI6GmVYIyMGoCNpIpM1i76CUUUUigooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi\n7t+VAm0hqoW6dPWpVQL06+tOopmbk2FFFFMkKKKKACiiigAooooAKKKKACinBCfpTwgH1pqLZLkk\nMCE/SnhAPc06irUUjNybCiiimSFFFFABQBnpShSfpUgGOlXGDZLlYaFx1p1FLitox7GbfcSilxS4\nrRU31J5kJiloorWMLbEN3CiikJrrpU38yGyN+RWNqG0Y5GeuK2JDgVj+ar3ZSRRIHwFH93J56jrj\n0r1vaeyoNtXvp/XY6sJF81+xWv8Az5IYp4nWOMkFmGRg9CemcA46Z7deAMm5uZbuQSSnLAYq7qFx\nGqNbwjYeFkGAQ3f7319u3WsyvhsyqL2rhFvpdX0uuny/O569KOl/uCiiivNNixZ3RtJ/MC5yMH1x\n/L862LW7h1B9ph8sJ0bOQOn5cKeua5+tzTN0+nsglCsHIJLc7cDP4YHTPY+lenltV+0VOT01e39e\nvyMKySXN1LFtFJPIZCwJPGTzjGOQfWtyJdkYUdAMVk2IaOUoG3KADnHXPpWsp6V9pVhBUUqa0/y0\n2/yPHxUm52Y+kIpaK8WpT6GMWNopSKSuWUWjVO4hUGmFSKkorNwTKUmiKipCoNMKkVk4NGikmJUT\nLjkdKlo61DVy4uxBRSsu00lZmydwooooAKZIuRn0p9FA07MrUU512tim1JsFFFFABRRRQAUUUUAF\nFFFABRRRQAUUUUAFFFFABTlQt06etOWPu35VLTsQ59hqoF6dfWnUUUzMKKKKACiiigAooooAKKKK\nACinBCfapAoFNRbJc0iMIT7U8IB9adRVpJGbk2FFFFMkKKKKACiilAJoASnhfWlC4pa1jC25m5dg\noxRTq6KcOZmcpWDFFZl3eTQTPGxXBGVK8Ef5/wA+lC6hlFUDMpAwp4yT0r1qeDi7rm1Q3RqNKS6m\nizqgyzAD1JpQQayJbl7rEYVgQRu25OORnBFSafNJsIkYEDgd/wBe9axwsHLkjq7Xv0CVCUYczevY\n1KTIqPcKa8gUZqo4V3sc9myUtTS1Zn26RkZwvyrwSeBnOMZ/WhL7LFXI3DuMgH8DXRSo03LlT1/y\nN/q00rsuyNxj86yLzbuDEcZ6VYkuSUyFyuM5B7c9fyP1qhNN5hrtXs3CUeZdVudFCnKLuRaoPOt4\nZIgSsa7XIPT0Pr3+lZNbkT+cotpfmj2kKOmCTnnHUZArMvLOS0cblYIxO0sACcYz396/P81oP2rq\nLXv5dF8n/wAA9Sm+X3WVqKKK8g1CtmwQRWnmECObja7rj5c54HftyR/KqVnp01yUdlZLdicy44GP\n84q88wZEiVAsacAde3rXt5PheaspTVl/Wv8Al5+RhUfMrR6bmjYZWNQSSBzjPStNWGKwoLkIBnir\n8VydgcqQh/i7dcc+lfdVoQaUU0r7f8A8mtSk22aIalyDVI3SquefwqvDqGPnfcY2cjd2XivPq4WM\nfidr6GMaM5JtI1qKjjlWRdwOR/KpK86ph3FkXtowxSYpaKx9g29h8wmBRioLvYYTuAwOearabdtK\nFhIUbE/P6V0Sy9ez5lvr+BpFScXJdC4y9xTalIqNhg14tSFmaxlcaRkYqEjBxU9Mde4rGSNoO2hH\nRRRUGoUUUUAMkXK59KhqzUDrtbFJmkH0G0UUUiwooooAKKKKACiiigAooooAKKKekeeT0oBuw1VL\nHiplQL9fWnAADAop2MnK4UUUUyQooooAKKKKACiiigAopQpPSpAgHuaaTZLkkMCE/SnqoX606irS\nSM3JsKKKKZIUUUUAFFFFABRQAT0p4UDrVKLYm0hoUn6VIBjpRRWsYpGbdwoopQKtK7sS3YBS0UV2\n0Y66GUmUb941hPmdDxVC3VyUMgUwryjE/dPUDI/D/PFX7uEyZIbjaVx25rFuF2TfOF652gYH0r16\n9OpOj7qVl167f16ndhOVx5b6ly7ibCiBAgTDOfulTz6/09B7VXgacHZGvzDHB46/5/Wm6vLcx7Hj\ndfLxkP8ALlu2c9ScdcVVfV3zPsXO9Qqs3Xjufr/XvXjxzNYSUoOVr2e1/u/4NkdUYOUFpf8ArqbE\nd8CSPmAJO3PcUy6ufkIBGfT1qrLMlzAs4neV+h3YGPwH4f5xTHCR2vnzM43Z24GcYI7d85r24ZjR\ndHnlv5drb/dqYrDpNSsWPPjtbAzKZFyTwOC46Dntzjv68VHuW9tmmgEgEZw24/KBjrnA9PwGKrX8\nll5KxKzPjLIyjBII43ZHr+n14zY55YlZY3KhiCceoOQfY18rWzOdGr7r038767/h8ux0xp395b+Z\nrSXIgkiaJvMDhd4yFTJyRkjHHse2emBTJ9QgjRI7di0LcuhHIwfoOT9T/KsiiuL+1Kyvy9bfpf77\na3+Rr7FXu/6/4boaR1iTyyscMatnhiAcLjpjH4/XtTJNXupgRL5cgOcBl4HQ9O/TvmqFFcv1uvr7\n2+ny+4fsodhzOrKAI1UjuM8/mackgQf6pCw6M2T+mcfpUdFYqck7/wCX+Rduhbi1K6hi8pHATB+X\naO4xn/PpUzaruLf6NF8zZ5HIX0B/rWdRW6xddO/MyeSPY24tQtbrMM2YlyBGxGSPqeOOg5p1vdrd\nvIAobZHhVbpgc9RyBwPb6dDhU+OWSI5jkZD6qcV1wzaun735f0vnvt2IdGPQ29kuws7iMxuoyWA6\n9859qk85bxjtZvKhUjzNuQxOM/Qc/hx61jS3089uIZZGcBs5J6/X1q3pUcfk3DtKPM2jYnmbecnn\n8MD/ACa9CWaSxVeOl1b/AD0/4bV9DN0+VXf9bG3p8mxPL3KyrjDKODxWhvrBhMtuPmBDDHyYOef6\n+3t9M2vtoKjB69hX1cI0q0bxle255tag3O66mgbmIMV8wZHUZ6UomVhlWBHrmsZDvdjNGBCcyEsS\nCcDtz1xk/TNE8rpM0sAYwuxCtnIY5yT/AJ9K5fa0IVXTnpbrdW7j+qX0TLOoS5XYSQh6kf4VJpZK\nq4cAMG5A7e3/AOuqEq3PmAFMlfmyBkcdalkBt5h5TuA+Cq9W98k9Dn61c6tGVblg73VtO9/+H62N\nXS/dchucGmsMrVawhmTfJM+5nx+GKtHrXiY6jGnPlickdHZO5FRQRg0V5R0ELDBxSVJIOM1HWbVm\nbxd0FFFFIYUyRcjPpT6DyMUhp2ZWopSMEikpGwUUUUAFFFFABRRRQAUUoBY4FTIgUe9AnKwiR45P\nWn0UVRk3cKKKKBBRRRQAUUUUAFFFKqlqAbsJ1p6x+v5U8KF6UtWo9zJz7B0oooqiAopQpPal2e9N\nRbFdDaKfsHrRtFVyMXMhlFSbR6UtP2YucjCk04IO9OoqlBITkwoooqyQoopQKpRb2E3YAKWiiuiE\nLaIzbuFITxQTio3kC9TivRw9Bt6EbjZOh+lZTCN7mVZHVOmN/Tr/AJ/Wr7zKR1rJvJFZuvNetLD8\n1Fweh14ZNSKWryTMUBSRLcnKDJ2njsPWsyt+KA6hbtAYxkYPnEZPHQZ7elY9zHBGxEMxfJyBjgL2\n59fw/wAK+BzLDyjUdXpt8/JdvS/metTkl7nUsafepaRSeapkRuAm7oeOf0/lUE95JM8pG1FkOWVR\nwen+A/yar0VxvFVORU1ol+O/X57bF8kb3CiiiucsKKKKACiiigAooooAKKKKACiiigApVYo6sMZB\nyMjNJRQBet74CSR7lpGLdPLCjnuc9u3Tr3q2imVwbY5XG9SWGQB6+nSsarVldNBJsZyIZPlcHkAe\nuPUV62BzGpSfs3Kye79XrcylBJXQt3eyyyyJnEZblAQRnGCf/r0ltf3FqzGOQ/MMc81Vorg+s1eb\nn5ncvkja1jcspZktmeZ1Jb54yArHIB6n8enUfznht7gx5TDc7treuOv1qK2gZrBYc7PLPzBv4m5O\neCcDHfvitexQhG5JG4gZxkduxr7PLORUH7S7ku/yf5+X3nnYipyK8S1CpWMbic+9OPWnU09a4sXP\nmlc8+G9xjjmm09xxTK8yaszpjsIRkYqGp6icYY1nI1g+g2iiioNAooooAilHINR1O4yh9uagqWax\negUUUUFBRRRQAUoBY4FABY4FTqoUYFBMpWBVCjApaKKoyCiiigAooooAKKKKACigAnpUqoF5700r\nickhqp3P5VJRRVpWMW2woopQM/SqSuIACaeFApabNJ5MRkKlgOTjriuqjQdSXLHcylMdRSqwZQQe\ntLitp4acNGZ86G0UuKMVn7Nj5kJRS4FLij2TDmQ2inUU/ZeYuYTFGKWirUIoXMwooorRRb0RNwpC\ncClprGuujS1sS2ULnUBFKYwhJAyeO3tVVnkuHy7IqEbSM5Iwc9Mj0H50/UARMj4GF7nt71BLI8Vn\nOYCwkQHIVSAgB6evTOT7CuvFTdKLjsmtNN/Lfy8n2PQowjyqUVqQK0/lk7DtxngY49cenHWkMssV\nk08aK2xsPlVPB/X+f+FX+1i14JWjHlY27eCQMY6496df3qqXgiiEZ2gHD525wSOnXt1x145rx8Rn\nMa1Fx5vh8tb200/z+7Y7fZtPbcgvLwTKiRqqqEAbA4J4J689c96pUUV8xUqzqu82dMYpKyCiiisx\nhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAOjCFwJCQp7jt6Vr/ZbCAb4/MuCSdpY4VT\nn26n9Kxqu2NysbeW6ocjCu+SF5z0APv2713YKrShNKpG+u/X07fgZzi3qbMCi4dXdVyBj1J+tbEQ\nCgADgViwxXED4bAUZ+btxn8unf1rWt5N8YJBB7g9Qa+9cqM6VqO3X/gnjYpSve+hZpDQDmlrw69N\n6nPFjSMioqlqNvvGvNqLqdMGJUcg6GpKa/3axexrHciooorM2CiiigAquwwxFWKilHzA+tJlwepH\nRRRSNAooooAnRdo96dRRVGG4UUUUAFFFFABRRRQAUqqWNKq7j7VKAB0pqNyJSsIqhRS0UVoZBRRR\nQAoGTUgGBikUYHvS1tCNjOTuKOtJIiuhVgCD1BpRTJpPLiLbS2Ow616WCg3NW3MJ6vQyLjfaz7Ip\nWCsMMCc7Rnr+ta8R+Qc9qw53a5uAfLbaDkrg5I55/wA+tasEqmMYIxivcnD2kZRXRr8jevF8kW9y\n3RUYf3oEqsSAwOOtedLCPsct2SUE4pu6qt1cGFNwGeQDjrVUsJzMaTk7InjuY5JHRTyhwalrAjbO\noMwcHnkqTj8+/wCVbSOCOtaywylDnitDStS9m1ZktFN3UbvauZ4fyMbjqKbu9qCSatUmFxSajY8Y\npScVTvJmjQbcZyM5OK7KFK2pUIuTsiG8mQAqepHSs63w0xTa53jHyPtI5zU6zx+SzTc5yJDtzgcg\nYP8AnqOtVDcWv2aXyGmLKck4Xp0HUepA/I9sVzZhmEFD2TjbS+tk7d7eh61GnyJopXtvb20zJFMZ\ncccdjxnP6/561KdJI0rl3OWNNr4WtKEptwVl0/rU7oppahRRRWRQUUUUAFFFFABRRRQAUUUUAFFF\nFABRRRQAUUUUAFFFFABRRRQAUUUUAalkBJbySMS7sdoDnGMcnHPPGO3cfjq2d3HHGFLAY9awNOkE\nd6gIJWTMZwOfm4qydRhiZhbpJjGBITgnpnI6Y6/pX1OV5rToUHGpv3f3+bOStR9o+VnSi6QpkMMe\ntRWt6z3HluDtc/uztxnuay1zdEtDIrRu2WCjPl59R7Zq7aSrHeODCVYsEGFxgY4/QZr23WpV0o07\naq79ey/z8jhlh1CL6s1jUb9RUhpj9K+drqzZjTYykb7p+lLRXKbEFFFFZHQFFFFABTJBlPpT6QjI\nIoGtGV6KKKk2CiiigCzRRRVGAUUUUAFFFFABTlTPJ6Uqp3P5VJVKPczlPogoooqzMKKKKACnKvc0\nKueTT60hHqyJS6IKKKUCt4x5mZt2FpGGRS0HpXoYfR3RjIyNRtmdwyqCo64OCaqCSSI7Q4UHOCew\nz36fXpW44zWXeRqUPY4zXvU4OonKMmm15fLc66Fa6UJK6IjdSxxAt3xggHGPXJoguWglJlG3cAaZ\naTMJz8v7tVJKqTwPbn/635Upnsrtpg8gikV+WbaN3pjPP+cn286WYSpVFGr0uui6d9v+Bc6nTjqr\naM0Uu1fjoR1yKqXsm9ODgg5B71TkDwPnejBv4k+7nuP8+opDHMyBjjDDIy4yfw/A13RxmH5Fz6Nr\nVfmRHDcsuaJND++aZo4SHA3ZB4B/oO/4VZhuZIwhk6MOo5HUjH6VVeRNPgS48tiz9MnOPYj0OOv1\n9KbaTi8sfKWPfOg/vHPUc49BnA988da8qlmUYVeS+nZ9v62XkzSpTUltoaL33zBE27iCck4HAzUl\nrdGVTuPIOMjHP5ZrKgEscx3KuzgPuwVIz1+nTmp3/wBFcSRj903Tacgfj3PBr0IYqnVrOH2ej8zG\neGio2W5sB/xpS1Zkd78yqwKluRkY49atCbI+9XQqMZax1Rxyoyi9SZmwMmqF467CG/Gpnk9Dk1Ql\nlQTHzMEY4JGRnP6961f7mDna9jWhTvIYk7PEY58+RsI3FfujGOP1/wDr1hShFkZUO5Rxn1961725\nu7dVBaOSFl5HKhvwzk+vHrz04xa/P82rqpU219OnS2uzPXoxsrhRRRXkmwUUUUAFFFFABRRRQAUU\nUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFACo7RuroSGU5BHY1YFjcOu+KJ2j3YBIAPqMjP\nHBH51WrVsTFPaiLzER05OVI4J6579QOcdPxrrwVGNeqqU3ZMzqS5VdFyGERmJbSMYAUuxADMccnn\nJH6itWztnjbzZWG8jGFAAx26Co7FUEQIAGQDitAdK+1q4eGEhyw3tY8avXcnygaa33acaa33a8Gr\nq2RDoR0UUVym5C33j9aSlbhjSVkzdbBRRRQMKKKKAK7DDEUlPlGH+tMqTZbBRRRQMs0UUVRgFFFO\nVCfpRa4m7DQCelSqmOT1pQABxS1ajYzlO+wUUUVRAUUAE9KeE9aai2JtIYBnpTwuOtOAxRWsYJbk\nOVwoopQK1jFyIbsAFLRRXVCk+hk5BSMe1BOKguJTHGWAyRXo4eg20kTq3Yc7AVk3jAttLFRjGew/\nD8/89XvfjcQRgj3z+oqnNI0gDlSIycbuv+TXqr2dOldysn1X6HbQoyUtUPjJNmyW5USNwVPVsg9/\nw6f41z5JYknqeTXR2kq2QeSSfMSnDKoJHPGTxxWJezRTTOY4wMtkvk89f/rdeeM96+Iza0knzbdO\n/n+mu/Q9Kk/eaS/rsaFtfQJZRi5+dsHBB3N1Ixjt1B544PrVSXVbmQLtbYy45XA6Yx/n6elUaK4J\n4+tKKinayS030t136GipRTuyzPf3NwhSWXcpOcbQBn/P9fWmRXUsCYibyznJZeGPTjPpxUNFc/t6\nvNz8zv3vqVyRta2heTVJkikTJDOSxZTjJOcn689sdKsWFxbvbG2ncq5ffvI9jxn9fxxWTRW0MZVi\n43d0tPk3fp+ZLpx6aHTTiG4VntnBkg+VxjB6kc9sdfoP0Z5rwv5coAYDkdcVjWd/NZMTEcA9QO/B\nxn16/wD6q0zKt5arct8smNp245xj8+v+R0+oyrN1ZQfq1287v+m2csqDWj2Jnn+XqPoKjt/3s/Rm\nl6oq8ZP17VW3VLFIscMshLIRx5gG7bn24/D3r2sfioewai9/6/roKNPlTKOpz3Ul0UuWyydAOmDy\nD+tUqdJI0sjSOcsxLE+pNNr82rVHUm5vqdsI8sUgooorMoKKKKACiiigAooooAKKKKACiiigAooo\noAKKKKACiiigAooooAKKKKACiiigAq/YTiK3nVpxGpKgAk8568AdOOfoOvFUKtWlotyGZ5hEqsAS\nVJ6g+n0/WujCSqRrJ0leXQidram1b3D/ACxw5bcNyFgFyPxrWsJHltEeT73esyGLMawwRyMg48wv\nyw9DwDj29q17aLyIFj3FiO5r7arVqzo/vlZ2XTr1PHxPJ03JD1pj9KcetNftXhVXuZw6DKKKK5jY\nif75ptOk+9Taye5vHYKKKKBhRRRQBFKOhqOppfu/jUNSzWOwUUUUFFmlClulOVM8npUnStFE5ZTt\nsNVAPc06iirsZt3CinBSaUKB71Sg2S5JDQCacFA606itFBIhybCiijFWlfYkKMUuKWtY0+5Dl2DF\nFFITgV10qV2Ztlae9WCULIMKRw3v6VKkodQwPBGao6lEXUN1VeqjvVeC5MI2O2R1U56ivbhh4Oyt\nutP1NlRUqalHc1i3pVeZgVI7YqB7olgqYZm4AzjNVTeB1zXTSoqMrdQhRluQSMY5Nh2lC2TuUHr1\n5xmn3l59ijzDFmKQ4BI+Xp7r7tjn147VGixzOoyxkJHyEYDc9M544qDUr7dbxwJDLEMdHJ6eoPcH\nkV87nNWEXJx06bddOnn8vzPThG9kynf3Ru7pmGNgOF4xkdP6VVoor5CrUlVm5yer/r8DsjFRVkFF\nFFZjCiiigAooooAKntrya0LGIgbhg8f16jt+VQUVUJyhLmg7MTSaszdnhZ4Y7pFISQDg9jj+VV7s\nvFpwQyMBI2THg4Ppz07fpUmiXSZNtOWKn7gLHA/DPr6D1o8Q4S4hiX7qpnrx1r2p4mpPDOotrWfk\n3pb9fmc6l76pNefyRjUUUV4Z0hRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRR\nRQAUUUUAFFFFABV+waLyXUyFZC2QoB5wOO/uaoVu6YvlmFVSHzNufMxknJPQjHrg/SvTyilUqYle\nztda6/IyrS5YXNbTpN4dGwHQ4IBzgfX860Kr2lstuhA6sdx+tWD0r6XG1XJ+89TwpuMptx2G0x+t\nPpjferxamxrHcbRRRWJoRyfe/CmU+T734Uys3ubx2CiiikMKKKKAGyfcNQVYb7p+lV6TNIbBRRRS\nLNCinBc08LjoK6owbPOckhgT1p4AHSlxRiuiNF9EZOYlLgDqaWqt7IFt2PUrhgPoc13YbB+0kkyO\nZt2RawKKghuEkAwwPGetTZFaVMFKDtYlya3FopCwFQpco8jxg/MhwacMHOS0FdvYnopu8e1IX960\njhX2FdjicU0tVS8uTDCWQjIIqFb0MmScEdcGvQo4T7y1Sk1zE1w4wc1lxFWeVC21uNn1z1Hv/nBo\nnmaWXajctgDnvT4LfZumnT5IzkPuADc+/b06VOPrQjT9h9q6/q/3/cehRp+zjdjI7UoyyPJ8ik7i\nuVIx6ZHrTRbwu0gjmBKjhW+Ug8cH+X1rLnvXe4aSFpEBHUsc9c59uf8APWq3mPvL723k5LZ5z618\n1Wzm07JuVtL3tp5ad9dfQ7I0pPVux0EMEtuJZWUBoxlTjPJ447HrWDcSCa4eQdGOe/8AUn+daEOo\n79PktnC8RnJc53HtjPQjjv249Dl1wZjXVWMHGV76v8rNd1+uhVNSu3JBRRRXlmwUUUUAFFFFABRR\nRQAUUUUAOjdo3DKSCPQkfyq3qQfzYndifMjDjPoc4/lVKppgRHASPvR5Bz1+Zh/SumnWkqE6V9HZ\n/c/+CS17yZDRRRXMUFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQA\nUUVJFE00gUcDPLdhVRi5NRitQJ7CAyXCuYiyLk5x8uccAk8YzXSaXaiOCNipXAIC5Pr/AJ/IVV0+\nwXamVIjHIVjnJP8An07CtxBgV9pgsAsFR5qi99/gePjMTze5EdSGlpD1rkryvc44ISo2+8akqM9T\nXDU2N4biUUUVkaEcn3vwplPk+9+FMrN7m8dgooopDCiiigBG+6fpVerNVqTNIBRRRSLL2+87W9v/\nAN/z/wDEUnmX2f8Aj3gx/wBdj/8AE1KH9aeDXq0q0Uv6/wAjypRfb+vvK5kvv+feD/v8f/iaYZb/\nAHgeRDjH/PU//E1czS5rsp4mC/pf5Ge3Rf18yiZL/wD54Q/9/T/8TWTeyXReTzI4w2zgeYfQ+3Nd\nJVO+gV4HIUbjwCfXtXbSnGtF01Jpv0/yX5mlGqoTV4owtOlvS7tsDOSdxc7T/KtQTajzmCLrx+8P\n+FXILRIiWA+Y9T61Y2irp8tCmqTm5W66f5MdbERnK6ijGE2pFW3QR9T1bH9KrRvcCbdGimU53jee\nPT+H+proSgNV47NY5pJO7nP0raNWDt77VvTX/wAlCNeKT91GS8+obk3RqDngBzyfypZJ9R8klolH\nuGPH6VueWPQUhQdxTjVi38b/AA/+RF9Yjp7iObeS6dlEygIcZ+Y9PypsoIn2lI1TJztkJ7nvj+lb\nN9bF4SEXJyKgWwUR/Moz/KqnhnW2qyWndfdt8zqp4mCje3yM+2G4kssZcfcDSEZP/fPNQ3sl8086\n+XlAuXAXcq5XJ5I45yfr9KszRCCYEAHaQQD3/wA4qeKVbtXhnEe1jhARluuTyOf5ZryMbg6kW488\nrX+L5PR6barbudHPf30v68jmqKmuLcwysisJAoyWXt9ahr46pTlSk4TVmjsTTV0FFaFvpzNZvcso\nZfLYjGflIz16enr/AIHPq6tCdJRc1bmV16CjNSbt0CiiisSgooooAKKKKACiiigAooooAKM0VLKc\nxwcKMIenf5m61SV02IioooqRhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRR\nQAUUUUATQLbtnzpHU84AXjp6/wD1q1rNEG0AAKeRjo3Xn371h1Nb3DQOOTsJBZQeuP68mvWynMKe\nDrc04Jrv1RlVpucbJnaQ42jFWB0rH0++D7ULh89GXoT/ADrXU5FfX4lxqRVSDumeBVhKErMdTT1p\n1IeteFV2HESoz1NSVGeprkqbG8BKKKKyLI5PvfhTKfJ978KZWb3N47BRRRSGFFFFABVarNVqTNIB\nRRRSLNCgHFRq/Y/nUlbJnC1bceH9adUVAJFaqb6kOPYlzQwVxhhkZzzTA/rTgwNb067i7xZm4D6K\nbRWjrN7kcg6im0uaarCcBaKTNLW8arJcRCAaYV/OpKQjiu2jXZDVijcRgg8VlxIgklkcZKAMo/H/\nAD/ga0dRl8tQo4LfxdhVOCEzfPyqqfl5/X6/59K9SrF16SpJ76+i8/U78O+WHM9hiXMbuN4KFiSZ\nGPK8k9gD1/maYDaRyyyKpdz825uNze2PxP6Vca08t1eNV3KcgHpmqpsyq4PWud5XGUrKWnfRu+35\nG8a0HqOSdrqCaExjDAbFUZ5HbHfgVz80flTOmc7TjOR/QkVtxbIZUfe3mgjAIwoOe5z0xVPUrOSN\nEnaaOXdwWUBeO3ux6+/TNfM5vhJRTS15W/W3V/f6/Lr005RUtNL/AJmbRRRXzp0hRRRQAUUUUAFF\nFFABRRRQAoG4gDGTxycVavyPMiQIF2RBDgY3YzzVrRLZJZ2lkcAJ2zyfXv8Ah+PtT/EKqLuJ0I2s\nmcD69f8APpXoQpQjg5Sk/ebVvRP/AD/IwdS9VRsY9FFFeebhRRRQAUUUUAFFFFABRRRQAUUUUAFF\nFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAF3T5m85YCVKMSQrLnLY49xkgV0unXYliRWdWcjPBH\nIrja6DTfM2QyIhEe3hQ3A5wep7kV9JkeKnJvDSa5d9b/AIHFjKUZQudHSGoredJ0yhBwcH2NSnpX\nZiqfK2eRG6dmJUbfeNSVG33q82psbw3EooorE0I5PvfhTKfJ978KZWb3N47BRRRSGFFFFACNwpPt\nVerDfdP0qvSZpAKKKKRZZpyuV+lNoqjBq5MGDdKWoKer9j+dWpdzNw7ElFAORkUVRmKGIp4YGo6K\npTaE4pktFRhiKcGB9q0U0yHFodS5pKK0Ta2IaHUU2lBrop1dSJRKdzZG5lG9sRqOAOpNTRwiNAoz\nwMZqeivUjjG1YlylZR6IiK1XmQBT6VcIqvcRs8TBeprtw1e8rNig9TDkjaWTKIxTPLAcY+tPv7a5\nmhEVs5MSdRz6DPO3/e7/AIVNJYMWYsepzgdP1qs4liQR7tqZ529T1/xPtWGPwVXFJz022T1S338/\nI9WnUi7KL2Mm8tWtLh4yG25+UkdR/jVeuhigh1FnidXDsRhsgnA7ZI449P16VjXdt9nkYK6sueBu\nBI9Pr9enIr4vG4J0m5w+H8n2/wAvI7IVLvlluV6KKK881CiiigAooooAKkiglm3eWhYKMsew+pp0\nFrLcMFjQknOODzwT/SteKL7DaqrEx3BBbK9cEjHP4f57d+DwM68lzJpem/p/VjOc7aLcdJO0NpFZ\noQdgG4gY59OOO9VL5zNYpnfuRuem3/PWnEFjkkk+9SoPMs5oWkfYfmKJyTjknnj/APVX0WKyx06N\n2ulvw/S3n8zJSilp3/4cxKKVlZGKsCrA4IIwQaSvjjpCiiigAooooAKKKKACiiigAooooAKKKKAC\niiigAooooAKKKKACiiigAooooAKKKKACr+nrGqmfzHWZW2rtIGMj8/WqFXLG6hgV0mWQq7KfkxnA\nDev1FdWClThXjKr8JE78uh02moV8x24LtnaOn+etaNYsNw0cayxTxvAxwpbOfoemOhrWgmWeFZEP\nBr7HE+yqR5qOy/X7zw60JKXM+o6mP1p5601+1eHUWjHDcZRRRWBqRyfe/CmU6T71NrN7m8dgooop\nDCiiigBsn3DUFTS/c/GoaTNYbBRRRSKLNFFFUYBRRRQAoJHSpFYN9aiopp2JcUyeimK+eD1p9Wnc\nyasFFFFMQoJFPDA1HRVRk0JxTJaKardjTq2TT2M2rCg0tNpw5rohO+jM5IKawp1Ield1CbuZNELp\nn+lZN4oD5Ynb1wOp/p3/AM99hzWXeMoQ554r3aF5wavY3w7akV42e3smkhLMSc43fKDg9f04+lc8\nxZnJcksTkk9c1v2iP57oGCqylWYMCOf0P0+tJ/Z1nA0z3JLlnwq4IK/XBHXn6j07fI5lhHWmlB7N\n97Pz9ejt+B60ZqDd92VLXTY7qySRmEZ5G5eSTnuPoOg9RVOWyuIFUvGwz2we/T+v5GtSdzK4VY2Q\nJxtJyQfT/P8AWlF1KqqNo3KMBsnP8/8AOBWksl9pCLUem662trbbX+rjU5LW/wAjGe3miz5kMiY6\n7lIx/nI/OkEUhjEgUlCduR68f4it64RtQtUhSVS4OSB1Y8449Mnk/wCFR6fC1laG5V4xKwIALc9R\nxjp7/Q15ay29Xljfl76J37W/4b8CvatLXcyhYz+U0jLtC5yCDnjOfp071bsLGB7f7VOzeXu2bcd8\nH8+349qu20k0k5RE4dtzBcjHqc/160+WNZGEMKqEBySvIJ5Oc9e9ejQyde2jBL3lvfVeT+XZ9e5E\nqrWj/AeUi05JBblt85BIzjGGPGByO4/ycRMj3EpldQGPXb0NTraOzo0jZKjGenHarYhAHINfT4HB\nww0feWupxTrJbO77ma8GB0xTbY+VOW3BXH3VYfK31/Q1ovEMfL+VUWhzccMVIwVwcEnPb3/wroxq\njOhK/QqlU5tGY+oCX7a7TQ+Uzc7cY/H8aq1saj9pujHH5JCgAgyOMjPPXOOR69wfSsevzXG0vZ1X\na9n5deq6Xt3/AAR6NJ3igooorkNAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigA\nooooAKKKKACrdnBFNDMZFY7SuCrYI68dMc/0+tVK19PAgtPMczEScBAwCkZ5/wDQRnpXbl9FVsRG\nDV0Z1XaOhbt45FIaLLqE2YcY3DGOcGtfTw4s08zO73qGxC+Uo4JwKvjpX2+LpU6MOSC6WPFr1nJ8\nrQHrTH6U801vu185V6ih0I6KKK5jYif75ptOf7xptZPc3WwUUUUDCiiigCOXoBUVPl+/+FMqWbR2\nCiiigZZoooqjAKKKKACiiigAp6v2NMooTsJq5PRUSvjg9KlrRO5i1YKKKKYgp6tng0yinF2Ymrkt\nApAcilroT6oya6DqQ8CgUyZXaIhG2sehxnFelhLSmkzGS6Gff3TRMFGAG43HtVBt0pztZlGRwfvc\n/hjj2/CnTiSO5CysGLHAYgHj6EVqW8CrGMDtXuJX5uZtRWmjt6v/AC+Z280aME0rszAlwYQE3IFG\nAAR069R3/wA9qbHA91MfMGCAMnue9bvlr6fpSJbpGWKrgtyayi6FNppf8Ht936mf1t2ehRjsQmSR\nnPsBVe8iEcZOK2dvtVS8t2kQbMBgwOcciuiliU/dj8jOnWbmnJmXAGtnm+eMuF27e+e4/Q9Papo4\nXnSNGG1VHKgdwTj9DUcaqt8VC7Ru6Fs/l61tJGAK5qUKcUqsr81/TX0OnEVnDbqZr2JU7kUEYIKH\ngN6frzUtnamNSWXGTwMdB+ZrQ2+1Lt+lN4mKk5LdnI68nHlZGEx0GKCpqTaaTGKhV7vcxuyFlDfW\nqF5EpUk1qEZ+tUb2JnQbQDk8g+ldlGqrG1GXvIz4/PeF0meQW+wlj/s9c/z+vNYMoQSN5Zyh5Ht7\nfhXQiEzWxSSSQKAd46hAAenP9fXrVBrC3W2laKZnYnbgxE7R19Pbr6Z+lfH5th3Od4JJat73T7f1\n1/H2KU7Xv+BlUUrKyMVYEEdiKSvnWmnZnUFFFFIAooooAKKKKACiiigAooooAKKKKACiiigAoooo\nAKKKKACiiigAooooAACTgDJNW1vniTy40jKBshmjGSMAc+nQdKTT0L30RDFQh3lsdAOf6VK+m/MT\nDOrp2yPmPrx+f5V2YejXa56O9/63Ik43szWido/La2/eRuAS2D8pI6HHAPPua1LW6MvySIySYztK\nnp61hwrHZPtjhOVbEjH5icdcccVoWkMhu3PmbSrDICgcY4HHsa+woe2qQUK3a/8Awf6+Xc83EU4O\nLl+Jqmmt92nGmP0rxq2jZyQ6DKKKK5Dchb7x+tJRRWR0BRRRQAUUUE4BNAEDnLmm0UVJuFFFFAFh\nWDDNLVdWKnIqcEMMimZSjYWiiimSFFFFABRRRQAU5W2n2ptFANXJgQRkUtQgkdKlVg1WncxlGwtF\nFFUSKDg1J1qKnK2PpVwlbRkyVyQdaGOF5pKZcRmeFowQN3BJGcCvRwcoqaUnZGElcx7x1nuP3QLq\nMF8cjH+c/ma2IVHljHTFOSJEUAKOlP6V6dfF07NQ3fX00CpPnSilogoozRmvPdZt7kcoUhGaWinG\ntJMXKVorNIppJRyX9e1WQMUUVrLEyluN3k7thRRRWXtWKwUUUVcaqe4mhpFMdcipaYwruoVGmIyr\ny3B3HOCetZ8G1ZSWeNUUZO/ODWjfs3mLGBkN1GcE+w96qzKYrSd4ciRhzwMfe4wfTqfwrfH1E6Tj\nGN5eS23t/SPUw7airvcyb97WWdntywHX5uh6cD0x/T86dXhpVx9rEDYx1MgBIxjPHr3+tP1CwaIv\nMhiKgAsI84B4Bx2xk9OvPpXwdajXm5VZxt3O+M4q0UzOooorjNQooooAKKKKACiiigAooooAKKKK\nACiiigAooooAKKKKACiiigAooq3ZWhuHLYDKvJUOAx+nB9fxq6cHOXKhN2V2WNP2CCXG3zk/vYXA\nPBA5ye3GP586lnaB13hmG4Y4OKijfz5MG3QKc8BR745/Lp+Va1rF5cQBxnvgYr73LsM8LQcKkd+6\n31/q55mKrWWm5H9iTy9uMj3qOztJFut5yqR8LwBuHTB79u9aQFLRiMWmrNarY4Y1Z2a7iGo36in1\nGxyxr56rK5pBCUh+6fpS01/u1zvY2W5FRRRWZuFFFFABTJDhPrT6ilPIFIqKuyOiiikahRRRQAUq\nsVORSUUAWAQwyKWq6sVORU6sGGRTMpRsLRRRTJCiiigAooooAKOlFFAEivng0+oKcrkdeRVKXczl\nDsS0UgIIyKWrMxQ2PpTwQelR0VcZtEuKZLRUYY07f7Vp7RMjlY6ik3CjcPWndBZi0UZzRTEFGaKK\nd2gFzRmkoqlUkieVDqKbSg1pGaZLi0LSEZFLRXTSqcr1IaMy7sZJZvMV+2AD/DVdC9u+xkKgDJZD\n0BOOBkD0/GtojNQSwJIMMufavT/d1k1LRvrr/VjeniWkoy2MENOQdw+YDbll5A/yfrSGKd7ExwxF\nnc43bDwvQgH/AD1/LXa2RVwoAHtWXcxiNiQcE8HBratl9OpR5aej/S1vl+XkdlPEc7sZd3YvbbD1\nUoGOOcdM8jjrkfhVSt+ArBbNPPHHJB02sAxz/Tmse5FtuP2cvwcYI4I9c9fwx/hXw2NwkKLvGXy6\n/Ly9bHdCbbs/vIKKKK841CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooqe0tzcThcMUHzSF\neoXuaunCVSShFXbE2krsijCFx5hIXqcdfpWx/oUihLW4Ubc4SUbc89ie3PANYtAODkda6MLi3h5X\nST1+f3kyjfW509p+6ZVbGSMjB7f0/GtiPkcVz8MqtaieVRIZOhHG3qNp6ntweta9jJuQgEkA8HGM\n/TnpX3NLFSxdG9tuvf8Ar17nj4qlb3i7SGlpD1rxsQ2mzmgIeBUVPc8UyuCo9bHTBaBTJD0FPqJz\nlqxlsaQWo2iiioNgooooAKrscsTUznCmoKTNILqFFFFIsKKKKACiiigApQSDkGkooAnVw31p1Vqk\nSTs3507mco9iWiiimQFFFFABRRRQAUUUUAKCQcinq4PXio6KadhOKZPRUSsV+lSBgatO5k4tC0UU\nUyQooooAKXJHekooAcHNODA1HRVKbRLiiWiowxFPDA1qppkuLQtFFFUSKDS02lBrWE+jIkuotIRS\n0V3UZ6mTRBIOD9KymWN7mXzlZ1GMKCe5x/OtC6mEeVwSdpb8BWLcszzFWBU5wc+vT6V7E2vq8lzW\nf9fodmEg2ypqxlWVY2ZCq8AqOTgDqcc/0rNrc1gzGGOJIWNvgFGwePf6ketUG0yceaU+ZYwGBxgk\nHp+PP/66+HxuHq1KzcE2rL5abLul5f8AD+pSmuRXEsrL7asiqyoyfNubPT0qGa2mgdkdD8vUjkdu\n/wCI/Otby47a0EEbSbycsHUqefbPsP0pHEUto0bMUckliFyXyR9M+vNdn9kylQU2rNLddX2+92uL\n2rvdbf1qYlFat9ZWscSyxyMiHKqWXJfA+g6nj9fWssIzKzBSQvLEDp9a8WtQlSlyyNozUlcSiiis\nSgooooAKKKKACiiigAooooAKVVZ2CqCWJwAByTTzC6wiVlwhOAT3/wA/1q9pi2rRT+aqecigpvYg\nHk5/HpXRTw8pTUJaX11JlKyuQW9kZWkWVmhKYJ3Rkjn19KvRuLUGOIB49uDnOH68n061PGXmbeFx\nJn7wAxnOc/4+vHpzYFntAK5DKcg19fgMjUIt1NH08+39dTkniEnaRz9zbSQzuuzgc/KCQB1qOG3l\nnLCJC20ZOB0rfVfMLQzlDEcgksBhsfe9zg/16ikl8yGdobZvlQkgr976E9e1eVLI/wB+6cXe3T8d\n+nTu+5oq+livYrJNZFTAyrH93arEO2OvXrwfb29bdveyqgWNNzY/DgVHJLOJcrGqljwoUHJ9fr6V\nJKyPIBD8yqQdyjKOSM5OOh/P8K9zCQrYRfV5a3Xrptey+7btv0wmozV2jbifegJ/lilPWqtjdGdW\nRoyjp1Bq0etcWYQ5KljzVFxk0xjnnFNpSck0lePJ3Z0LRAeBmoOtSSHAx61HWcmbQWlwoooqSwoo\nooAilPQfjUdKxyxNJUmyVkFFFFAwooooAKKKKACiiigAooooAcrlfcelTKwYcVXoBIORRclxuWaK\njWQHg8GpKozasFFFFAgooooAKKKKACiiigB4cjrzTwQelQ0dKakS4Jk9FRiT1p4IPQ1aaZk4tC0U\nUUxBRRRQAUUUUAODetPqKlBIrSM7bkuPYkopAQaWtTMcKKaKdXVRnqjKSKN/EskR3MVx3BxWdbhZ\nCEeKPy+gY4G5uwz/AJ/pVq9hnmlc7WWNRj72d3vgf5/lSLYyeXG+cumCqtwB07fhXtVKXtqfLZad\ne+my/wCDsdlGap01eW/4EV7I0eHAWTf95iR8p5HGOnT1PSq8UtxuaQDeWAB3dCP8/wCTVh4XtcSN\ntC7txAAIXkdM1JpyMyZMe0dRjpj2qMPhqcZyhVWujT/rqauoo0uaOq2GfZXkbMshfbkLu9M1HdW2\nEyBz9eK2AmBTZIg6ketejCtCK5I6I41iHzXZiS2kU2nldw3Bjtk2YAxzyfyHXvTY410+2eJWSR3Y\nMRtPA7EHAIP8u3eriW1xCriPjdzkcHOfUc4xSLaMxJk3AZ4Vmzj+n+RXkxypTq3n5631tr+d/uO1\n1463d0Zk1nGZYYoUUMdu5XBGSOMZODnPp79MZpk+mIPL+zuzr/G56DJwOMD+Z/StVoHjQrHhVYFW\nx/F1x/P/ABqnIjxoU3NsJyVzxms/7AjJS5kvL5W+69tfXuaQr3as/wCv62KLaTdiNnEe/a2MLknp\nkH6Y/Gozp12CQYHUjPDcE/T1/CtKEMpEkjFY1UldxwGx2GeOv5VR1DUpbs7PMJjXODjbkccEZ6cV\n85isHQw9+Zvys9381t/wx0KU27KxTaNlUElcH0YH+VOSF5B8gBJ6LuGT9B1NR0V5acb7fj/wDXUm\nW0uHXcsLlcE5xxwM/wBKsHSbobjtUhW2k54+v06/lUljqr26pbyn/R8ndtX5uff2qeSFlIckMrYI\nZTkcjIr18HgaGKmowb+emvy6ef6mTnJXvoNh0mOPdLdSExKRtKDh/Uc4PoeO2aksrR7WSVlbJK/L\nzgdxyT9R09e3QyxwNKo3EkDgc9KupbM6Kj4IUbc4ySM5xnsOK+iWQ06TTjb73pp+Pp/wTkniLXuy\nhJOZoXW5UururAHOABnPf6U6KBLJ3UE+VIMrgDcpGMgn/OPzrSNoMZHDDkEdRVeOzllxHI7bUY9e\nuMdj6Vricvp+0U6SXb779unl5kQxEGnfRD7CMMpYKVTjaCcnp/jmtDZxjiiCBYYwi9BU2K2qYnls\nk9jz6k+eTaKZsYWLEr945Izxmnrbog+Ufj3qzRiso413sJzm92Y1/FtAJyqeoGTmpdMBkEjuBuLc\n49e/+RVu7RTAwbGMdziqumQSJiVs7XXI+bpXTOpde1v0aOhT5qDXU0QiqSwAyeppGOBTiajY5OK+\naxFZzd2RTiNoopjtgY9a4m7HQld2GMctmkoorI3WgUUUUAFMkbC49afUDtuakyoq7G0UUUjUKKKK\nACiiigAooooAKKKKACiiigAooooAKeshXg8imUUA1csBgwyKWq4JByDUqyA8Hg07mbjbYfRRRTIC\niiigAooooAKKKKACjpRRQA8SetPDA9KhopqTIcEyeiow5HXmnhgelWmmQ4tC0UUUyQooooAAcU8N\nmmUVUZNCauS0oNRhvWn1tGfVGTj3HdaMD0ptLmuqGIa6mbgNkiSRdrDIyDSqiqMAUuaWuhYqTVrk\nuL2DA9KTApaKqNcnlGFaaV9KlpCO9dlKu+4tirIgx0+orKvAAQuQMnGa2pBkGsYQGS7OSMR4Zg3I\nIzz9PpXfPEclByfXT0udmF1d30Kepu1vaRQq2VlXJzk4GQePxGelY9aep27Y89XZ0ODtXlUHTrnj\nnOOKzK+AzOTliHfayt6d/n9/fU9elblCiiivPNQrbsz51gokzIykBIwNpx0yvr0649c9BWJW1pfl\nQWTTvGGYybd2TnHH+ev8q9HK5zhXvHt/l/SMa3wmhYkOox1rTVRisyxYtIAoYRgAKD29a1lHSvva\n1ZuCk1Y8XEaTsKFpQoFLRXk1K7WhgkFFBOKbmuGdXuWojqTIpKKy9qy+UHVHGHUMM5wRQMKoVQAA\nMADtQSB1phYnpSniJcvK2NQFZuwplFFcjdzZKwhOBk1ETk5pXbceOlNrKTubxjYKKKKRQUUUUANd\ntq+5qCnO25vYU2pZrFWQUUUUFBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAPWQrweRU\noYMMiq9KCQcg0XJcbliimLIDweDT6ozasFFFFAgooooAKKKKACiiigAooooAeHI6808MDUNFNSZL\ngmT0VGHI6808MD0q00zNxaFooopkhSgkUlFAEgYGlqKnBiOtaxn3Icew+ikBBpa0IFzS02itI1Gt\nyXHsOooorrpT1sZNEb8CsXUFBx1J+taGoTvCoARtp6sO1ZLK8jHc6tgjIf5fT/H1r241I06Lcu3a\n/odeFpu6mRag32a2SEwZRsCRgefUjOPUcdeh68isiWKSFtsilWxnBroJZzFLHiIbQA64bJ5x3/Dp\n7UyVIr+MG5by5DzvA64OB9Byc/QGvlsdgZ1kqvNpsu3muu7v5fej0ac3Hdepz9FaF1pqWsSym4yj\nZ24UZPTpz6n9D1qCygjuZvLdmBIOMD2JJP09O9eI8NUVRU2tXtquuxupprmGW1u91MI0xnqfp9Op\nrbs7WWzdhcTh0YY2A5JGFGPyPb0HpRElnZSmLy33KrAyDgkn8+MdKZ5LcFizRryAfT8+OnrX0OW5\nVNx9rHddddfJHPUqc2j0RPZzmIldhwPmOPQ//rrbiYMgYdCMisaIB78tLknrhlIIyeh49+v/ANat\ntQAoA6V7dao3QTm9f+B17s8zFKPMrC0hNKabXiVJ/eZRQUUUwv6VztpbmqVxxIFNLE9KbRWTm2Wo\npBRRRUFBUbvngUO+eBTKiUjSMerCiiipNAooooAKZI21eOpp54Gars25iaTKirsSiiikahRRRQAU\nUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAU9ZCvB5FMooBq5YDBhkUtVwSDkGpV\nkB4PBp3M3G2w+iiimQFFFFABRRRQAUUUUAFFFFABRRRQA9Xx15p4IIyKhoBI6VSkQ4J7E9FMV+x/\nOn1SdzNprcKKKKYgp4f1plFNSa2E0mS0VGCRTwQa2jJMhxsLTqbRWsJ8pm1cVlDDBFU7y1EkR2rz\n7DmrtFerh8VKGlzNNxd0Y8Vi333UBiOg7VVu4BGcKOTzgV0BWqtxbrJzzn1FetRrRlDkil5XN4Yl\n895GPcWz3VnFDnY2eA3Qc7ecd857HjPpVG0sZBcszPt8obwwzhu/Xr3H/wBatCaB4NrrtDJ0Kjrz\n1pbiTfAqeZlgBuYNw/PT14xnn3rwMTlajWhOej9fOy+7fT0Z6UKl17uzGsFe6V0bdvOcYPTvzgVs\nRwrtGRmqNlbDhjn1wa1kFe9Lmw9Pkvc87E1E2kuhAbKJ2RiuNhyMVaopDXjYrEyn8XQwV5biGiim\nMe1eVKVtWbJX0Bmz9KbRRWDdzVKwUUUhIAyaQC1Ez54HShmLfSm1DkaxjbVhRRRUlhRRRQAUUU12\n2r70AtRkjZ+UfjUdFFSbJWQUUUUDCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKA\nCiiigAooooAKKKKAHrIV4PIqUEEZFV6UEg5FFyXG5YopiyBuDwafVGbVgooooEFFFFABRRRQAUUU\nUAFFFFABTlYr9KbRQDVyZWDUtQVIr9jVqXcylDsPoooqiAo6UUUAPVs/WnVFUinI961hK+jM5Rtq\nOBpabSiumnLoZSXUWo26VJUEsqpkEjPXFexhG5NWMra6GffSKoIJ5OeKoBMLskJUZ3BsZA7f5+lT\n3Ewe6G0huMbW6Hn/ACfwq9BaZi/e/Mx+9716VZOcuVuyj163/wCAehGaowTfUq2tyYm2SjHvnita\nGRZFDKwIPcVny6WHclSoXH3cf4U2FZbO6CFQ5cHG3j36dqid6qcZWfn/AMAxqRp1FeD17GvTaELG\nNS67WIGVznB9KOlfNV371jKCEY4FR0pOTmkrhlK7OiKsFFFMZ8cDrUt2KSuOZgtREknmk60Vm3c1\njGwUUUUigooooAKKKKAEJwMmoGbcc0533HjpTKlmsY2CiiigoKKKKACiiigAooooAKKKKACiiigA\nooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAp6yFeDyKZRQDVywCCMilquCQcg1K\nsgPB4NO5m42H0UUUyAooooAKKKKACiiigAooooAKKKKAHKxX6VICCMioaUEg5FNSsTKNyaimqwb6\n06tDJqwUoODSUUCJQcigdaYp7U+uiEr6mMlbQdWVqTRh0ViNx7FcnH8/5Vqk4FZN7IklxGAcqh3O\nR2Fe/gablCS6CofxLkNla+bIyTBmUDIJXYD9AOnb8q2woApsYAUYpxNZYmuqcVShsiak3Vldi1FJ\nCJGjbO0o2en6U/NISBXAsZOGsQUBxNRs2fpSFiaSuCpUcmbRjYKKCcDJqJn3cDpWTdjVRuKz9hTK\nKKzbuapWCiiigYUUUUAFFFFABUUj54HSh3zwOlR0maRj1YUUUUiwooooAKKKKACiiigAooooAKKK\nKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHK5X3HpUys\nG6VXoouS4plmioll/vfnUgIIyDTM2mhaKKKYgooooAKKKKACiiigAooooAKer9j+dMooTsJpMnoq\nEMV6VIHB+taKVzJxaHU9Wz9aZRVxlYhq4s6PKm1XUA9QRyfx7U5IIo02KihfQCkDmjf7V6CzCago\nLSxk6b2HIqxoFXOB6mgsBTCxNJXJVryqS5nuUoWHFiabRRWLbe5olYKQsFHNNZ+w/OoycnJqHIuM\nL7isxY0lFFQahRRRQAUUUUAFFFBIAyaAConfPA6Ujvu4HSmUrmkY9WFFFFIsKKKKACiiigAooooA\nKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAo\noooAKKKKAClBK9DSUUASrKP4uKkqtSqxXoadyHDsWKKjWUHrxUmc9KCGmgooopiCiiigAooooAKK\nKKACiiigBwcjvThIO/FR0U02iXFMm3Ke9LUFFPmJ9mT9KQso71DRRzD9mSGT0FMLE9aSik22UopB\nRRRSGFFFFABRRRQAUUZx1qJpey/nSGk2PZwvXr6VEzFj7elNopGiikFFFFBQUUUUAFFFFABRRRQA\nUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABR\nRRQAUUUUAFFFFABRRRQAUUUUAFKCV6GiigCRZf7w/KnghuhoopkSirXFooopmYUUUUAFFFFABRRR\nQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUxpAOnNFFJlxSZEzFjzSUUUjQKKKKACiiigAooooAK\nKKKACiiigAooooAKKKKACiiigAooooA//9k=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import libraries for simulation\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "# Imports for visualization\n",
    "import PIL.Image\n",
    "from io import BytesIO\n",
    "from IPython.display import Image, display\n",
    "\n",
    "def DisplayFractal(a, fmt='jpeg'):\n",
    "  \"\"\"Display an array of iteration counts as a\n",
    "     colorful picture of a fractal.\"\"\"\n",
    "  a_cyclic = (6.28*a/20.0).reshape(list(a.shape)+[1])\n",
    "  img = np.concatenate([10+20*np.cos(a_cyclic),\n",
    "                        30+50*np.sin(a_cyclic),\n",
    "                        155-80*np.cos(a_cyclic)], 2)\n",
    "  img[a==a.max()] = 0\n",
    "  a = img\n",
    "  a = np.uint8(np.clip(a, 0, 255))\n",
    "  f = BytesIO()\n",
    "  PIL.Image.fromarray(a).save(f, fmt)\n",
    "  display(Image(data=f.getvalue()))\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "\n",
    "# Use NumPy to create a 2D array of complex numbers\n",
    "\n",
    "Y, X = np.mgrid[-1.3:1.3:0.005, -2:1:0.005]\n",
    "Z = X+1j*Y\n",
    "\n",
    "xs = tf.constant(Z.astype(np.complex64))\n",
    "zs = tf.Variable(xs)\n",
    "ns = tf.Variable(tf.zeros_like(xs, tf.float32))\n",
    "\n",
    "tf.global_variables_initializer().run()\n",
    "\n",
    "# Compute the new values of z: z^2 + x\n",
    "zs_ = zs*zs + xs\n",
    "\n",
    "# Have we diverged with this new value?\n",
    "not_diverged = tf.abs(zs_) < 4\n",
    "\n",
    "# Operation to update the zs and the iteration count.\n",
    "#\n",
    "# Note: We keep computing zs after they diverge! This\n",
    "#       is very wasteful! There are better, if a little\n",
    "#       less simple, ways to do this.\n",
    "#\n",
    "step = tf.group(\n",
    "  zs.assign(zs_),\n",
    "  ns.assign_add(tf.cast(not_diverged, tf.float32))\n",
    "  )\n",
    "\n",
    "for i in range(200): step.run()\n",
    "    \n",
    "DisplayFractal(ns.eval())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[ 12.]], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Create a Constant op that produces a 1x2 matrix.  The op is\n",
    "# added as a node to the default graph.\n",
    "#\n",
    "# The value returned by the constructor represents the output\n",
    "# of the Constant op.\n",
    "matrix1 = tf.constant([[3., 3.]])\n",
    "\n",
    "# Create another Constant that produces a 2x1 matrix.\n",
    "matrix2 = tf.constant([[2.],[2.]])\n",
    "\n",
    "# Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.\n",
    "# The returned value, 'product', represents the result of the matrix\n",
    "# multiplication.\n",
    "product = tf.matmul(matrix1, matrix2)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "  result = sess.run([product])\n",
    "  print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-2. -1.]\n"
     ]
    }
   ],
   "source": [
    "# Enter an interactive TensorFlow Session.\n",
    "import tensorflow as tf\n",
    "sess = tf.InteractiveSession()\n",
    "\n",
    "x = tf.Variable([1.0, 2.0])\n",
    "a = tf.constant([3.0, 3.0])\n",
    "\n",
    "# Initialize 'x' using the run() method of its initializer op.\n",
    "x.initializer.run()\n",
    "\n",
    "# Add an op to subtract 'a' from 'x'.  Run it and print the result\n",
    "sub = tf.subtract(x, a)\n",
    "print(sub.eval())\n",
    "# ==> [-2. -1.]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.,  6.], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess.run(x.assign([4.0, 6.0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.  3.]\n"
     ]
    }
   ],
   "source": [
    "print(sub.eval())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Close the Session when we're done.\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Keras\n",
    "\n",
    "[Keras](https://keras.io/) is a layer on top of Tensorflow that makes it much easier to create neural networks.  Rather than define the graphs, like you see above, you define the individual layers of the network with a much more high level API.  Unless you are performing research into entirely new structures of deep neural networks it is unlikely that you need to program TensorFlow directly.  \n",
    "\n",
    "**For this class, we will use usually use TensorFlow through Keras, rather than direct TensorFlow**\n",
    "\n",
    "Keras is a separate install from TensorFlow.  To install Keras, use **pip install keras** after **pip install tensorflow**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions for Tensorflow\n",
    "\n",
    "The following functions will be used in conjunction with TensorFlow to help preprocess the data.  It is okay to just use them. For better understanding, try to understand how they work.\n",
    "\n",
    "These functions allow you to build the feature vector for a neural network. Consider the following:\n",
    "\n",
    "* Predictors/Inputs \n",
    "    * Fill any missing inputs with the median for that column.  Use **missing_median**.\n",
    "    * Encode textual/categorical values with **encode_text_dummy**.\n",
    "    * Encode numeric values with **encode_numeric_zscore**.\n",
    "* Output\n",
    "    * Discard rows with missing outputs.\n",
    "    * Encode textual/categorical values with **encode_text_index**.\n",
    "    * Do not encode output numeric values.\n",
    "* Produce final feature vectors (x) and expected output (y) with **to_xy**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import os\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name, x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = \"{}-{}\".format(name, tv)\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df.as_matrix(result).astype(np.float32), dummies.as_matrix().astype(np.float32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32), df.as_matrix([target]).astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred,y,sort=True):\n",
    "    t = pd.DataFrame({'pred' : pred, 'y' : y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'],inplace=True)\n",
    "    a = plt.plot(t['y'].tolist(),label='expected')\n",
    "    b = plt.plot(t['pred'].tolist(),label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean()) >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "               * (normalized_high - normalized_low) + normalized_low"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple TensorFlow Regression: MPG\n",
    "\n",
    "This example shows how to encode the MPG dataset for regression.  This is slightly more complex than Iris, because:\n",
    "\n",
    "* Input has both numeric and categorical\n",
    "* Input has missing values\n",
    "\n",
    "To encode categorical values that are part of the feature vector, use the functions from above.  If the categorical value is the target (as was the case with Iris, use the same technique as Iris). The iris technique allows you to decode back to Iris text strings from the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      " - 0s - loss: 71556.1182\n",
      "Epoch 2/100\n",
      " - 0s - loss: 2613.1372\n",
      "Epoch 3/100\n",
      " - 0s - loss: 3612.5451\n",
      "Epoch 4/100\n",
      " - 0s - loss: 527.8907\n",
      "Epoch 5/100\n",
      " - 0s - loss: 508.8241\n",
      "Epoch 6/100\n",
      " - 0s - loss: 298.2748\n",
      "Epoch 7/100\n",
      " - 0s - loss: 297.0491\n",
      "Epoch 8/100\n",
      " - 0s - loss: 281.9724\n",
      "Epoch 9/100\n",
      " - 0s - loss: 277.2532\n",
      "Epoch 10/100\n",
      " - 0s - loss: 272.9845\n",
      "Epoch 11/100\n",
      " - 0s - loss: 269.6429\n",
      "Epoch 12/100\n",
      " - 0s - loss: 265.0733\n",
      "Epoch 13/100\n",
      " - 0s - loss: 261.2955\n",
      "Epoch 14/100\n",
      " - 0s - loss: 257.5943\n",
      "Epoch 15/100\n",
      " - 0s - loss: 259.9363\n",
      "Epoch 16/100\n",
      " - 0s - loss: 252.5989\n",
      "Epoch 17/100\n",
      " - 0s - loss: 244.4611\n",
      "Epoch 18/100\n",
      " - 0s - loss: 242.5242\n",
      "Epoch 19/100\n",
      " - 0s - loss: 239.3464\n",
      "Epoch 20/100\n",
      " - 0s - loss: 233.8800\n",
      "Epoch 21/100\n",
      " - 0s - loss: 228.8684\n",
      "Epoch 22/100\n",
      " - 0s - loss: 224.4594\n",
      "Epoch 23/100\n",
      " - 0s - loss: 220.7069\n",
      "Epoch 24/100\n",
      " - 0s - loss: 215.8769\n",
      "Epoch 25/100\n",
      " - 0s - loss: 214.3565\n",
      "Epoch 26/100\n",
      " - 0s - loss: 208.6877\n",
      "Epoch 27/100\n",
      " - 0s - loss: 205.2195\n",
      "Epoch 28/100\n",
      " - 0s - loss: 202.4255\n",
      "Epoch 29/100\n",
      " - 0s - loss: 196.3724\n",
      "Epoch 30/100\n",
      " - 0s - loss: 190.7239\n",
      "Epoch 31/100\n",
      " - 0s - loss: 189.3009\n",
      "Epoch 32/100\n",
      " - 0s - loss: 183.2447\n",
      "Epoch 33/100\n",
      " - 0s - loss: 181.0352\n",
      "Epoch 34/100\n",
      " - 0s - loss: 176.0184\n",
      "Epoch 35/100\n",
      " - 0s - loss: 173.1029\n",
      "Epoch 36/100\n",
      " - 0s - loss: 168.1229\n",
      "Epoch 37/100\n",
      " - 0s - loss: 167.7445\n",
      "Epoch 38/100\n",
      " - 0s - loss: 164.4952\n",
      "Epoch 39/100\n",
      " - 0s - loss: 160.0360\n",
      "Epoch 40/100\n",
      " - 0s - loss: 156.5678\n",
      "Epoch 41/100\n",
      " - 0s - loss: 153.2082\n",
      "Epoch 42/100\n",
      " - 0s - loss: 149.6373\n",
      "Epoch 43/100\n",
      " - 0s - loss: 146.3159\n",
      "Epoch 44/100\n",
      " - 0s - loss: 147.7797\n",
      "Epoch 45/100\n",
      " - 0s - loss: 142.1970\n",
      "Epoch 46/100\n",
      " - 0s - loss: 135.4717\n",
      "Epoch 47/100\n",
      " - 0s - loss: 133.1635\n",
      "Epoch 48/100\n",
      " - 0s - loss: 132.1582\n",
      "Epoch 49/100\n",
      " - 0s - loss: 129.2102\n",
      "Epoch 50/100\n",
      " - 0s - loss: 128.5930\n",
      "Epoch 51/100\n",
      " - 0s - loss: 125.9701\n",
      "Epoch 52/100\n",
      " - 0s - loss: 119.8784\n",
      "Epoch 53/100\n",
      " - 0s - loss: 118.8305\n",
      "Epoch 54/100\n",
      " - 0s - loss: 116.3244\n",
      "Epoch 55/100\n",
      " - 0s - loss: 112.5092\n",
      "Epoch 56/100\n",
      " - 0s - loss: 110.4144\n",
      "Epoch 57/100\n",
      " - 0s - loss: 107.8703\n",
      "Epoch 58/100\n",
      " - 0s - loss: 105.1340\n",
      "Epoch 59/100\n",
      " - 0s - loss: 103.9498\n",
      "Epoch 60/100\n",
      " - 0s - loss: 101.4870\n",
      "Epoch 61/100\n",
      " - 0s - loss: 100.5480\n",
      "Epoch 62/100\n",
      " - 0s - loss: 100.9421\n",
      "Epoch 63/100\n",
      " - 0s - loss: 100.9219\n",
      "Epoch 64/100\n",
      " - 0s - loss: 95.7873\n",
      "Epoch 65/100\n",
      " - 0s - loss: 95.7065\n",
      "Epoch 66/100\n",
      " - 0s - loss: 93.1833\n",
      "Epoch 67/100\n",
      " - 0s - loss: 89.3494\n",
      "Epoch 68/100\n",
      " - 0s - loss: 90.1269\n",
      "Epoch 69/100\n",
      " - 0s - loss: 86.9266\n",
      "Epoch 70/100\n",
      " - 0s - loss: 85.3947\n",
      "Epoch 71/100\n",
      " - 0s - loss: 83.2898\n",
      "Epoch 72/100\n",
      " - 0s - loss: 81.2991\n",
      "Epoch 73/100\n",
      " - 0s - loss: 80.4928\n",
      "Epoch 74/100\n",
      " - 0s - loss: 80.6459\n",
      "Epoch 75/100\n",
      " - 0s - loss: 77.3044\n",
      "Epoch 76/100\n",
      " - 0s - loss: 77.4645\n",
      "Epoch 77/100\n",
      " - 0s - loss: 81.6769\n",
      "Epoch 78/100\n",
      " - 0s - loss: 75.2157\n",
      "Epoch 79/100\n",
      " - 0s - loss: 72.4649\n",
      "Epoch 80/100\n",
      " - 0s - loss: 71.8997\n",
      "Epoch 81/100\n",
      " - 0s - loss: 71.0201\n",
      "Epoch 82/100\n",
      " - 0s - loss: 71.0562\n",
      "Epoch 83/100\n",
      " - 0s - loss: 73.2754\n",
      "Epoch 84/100\n",
      " - 0s - loss: 72.3099\n",
      "Epoch 85/100\n",
      " - 0s - loss: 68.5493\n",
      "Epoch 86/100\n",
      " - 0s - loss: 67.4236\n",
      "Epoch 87/100\n",
      " - 0s - loss: 64.7394\n",
      "Epoch 88/100\n",
      " - 0s - loss: 64.7737\n",
      "Epoch 89/100\n",
      " - 0s - loss: 66.3231\n",
      "Epoch 90/100\n",
      " - 0s - loss: 65.1089\n",
      "Epoch 91/100\n",
      " - 0s - loss: 65.0595\n",
      "Epoch 92/100\n",
      " - 0s - loss: 62.0333\n",
      "Epoch 93/100\n",
      " - 0s - loss: 61.5400\n",
      "Epoch 94/100\n",
      " - 0s - loss: 59.8940\n",
      "Epoch 95/100\n",
      " - 0s - loss: 58.7464\n",
      "Epoch 96/100\n",
      " - 0s - loss: 58.8625\n",
      "Epoch 97/100\n",
      " - 0s - loss: 58.3078\n",
      "Epoch 98/100\n",
      " - 0s - loss: 56.8759\n",
      "Epoch 99/100\n",
      " - 0s - loss: 56.4023\n",
      "Epoch 100/100\n",
      " - 0s - loss: 56.5878\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1a23bf2dd8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "cars = df['name']\n",
    "df.drop('name',1,inplace=True)\n",
    "missing_median(df, 'horsepower')\n",
    "x,y = to_xy(df,\"mpg\")\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Controling the Amount of Output\n",
    "\n",
    "One line is produced for each training epoch.  You can eliminate this output by setting the verbose setting of the fit command:\n",
    "\n",
    "* **verbose=0** - No progress output (use with Juputer if you do not want output)\n",
    "* **verbose=1** - Display progress bar, does not work well with Jupyter\n",
    "* **verbose=2** - Summary progress output (use with Jupyter if you want to know the loss at each epoch)\n",
    "\n",
    "## Regression Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will perform actual predictions.  These predictions are assigned to the **pred** variable. These are all MPG predictions from the neural network.  Notice that this is a 2D array?  You can always see the dimensions of what is returned by printing out **pred.shape**.  Neural networks can return multiple values, so the result is always an array.  Here the neural network only returns 1 value per prediction (there are 398 cars, so 398 predictions).  However, a 2D array is needed because the neural network has the potential of returning more than one value.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (398, 1)\n",
      "[[ 11.24502563]\n",
      " [  4.01193285]\n",
      " [  6.25534105]\n",
      " [  8.11264038]\n",
      " [  9.82144165]\n",
      " [ -0.88497895]\n",
      " [ -6.7752986 ]\n",
      " [ -4.92581129]\n",
      " [ -6.42831373]\n",
      " [ -2.12612891]\n",
      " [ -3.03775001]\n",
      " [  4.62356615]\n",
      " [  0.12539701]\n",
      " [-26.69992065]\n",
      " [ 25.27157593]\n",
      " [ 20.29391479]\n",
      " [ 19.02035522]\n",
      " [ 17.54202271]\n",
      " [ 24.60809326]\n",
      " [ 25.25540161]\n",
      " [ 31.07150269]\n",
      " [ 27.37719727]\n",
      " [ 26.39709473]\n",
      " [ 19.65597534]\n",
      " [ 17.96322632]\n",
      " [ 10.52926636]\n",
      " [ 15.89938354]\n",
      " [ 13.2494812 ]\n",
      " [ 22.62887573]\n",
      " [ 24.83493042]\n",
      " [ 20.49523926]\n",
      " [ 23.29684448]\n",
      " [ 22.48306274]\n",
      " [ 12.27069092]\n",
      " [ 24.83712769]\n",
      " [ 20.42788696]\n",
      " [ 21.49545288]\n",
      " [ 22.22567749]\n",
      " [ 12.06143188]\n",
      " [  7.96261644]\n",
      " [ 12.60299683]\n",
      " [ 16.5173645 ]\n",
      " [ 17.06509399]\n",
      " [ 12.8578186 ]\n",
      " [ 18.20748901]\n",
      " [ 12.52334595]\n",
      " [ 24.96511841]\n",
      " [ 19.70547485]\n",
      " [ 19.006073  ]\n",
      " [ 22.70736694]\n",
      " [ 21.73614502]\n",
      " [ 28.54495239]\n",
      " [ 26.35336304]\n",
      " [ 25.85037231]\n",
      " [ 22.78005981]\n",
      " [ 23.71243286]\n",
      " [ 24.9694519 ]\n",
      " [ 24.31298828]\n",
      " [ 25.60595703]\n",
      " [ 31.13302612]\n",
      " [ 22.98843384]\n",
      " [ 23.07907104]\n",
      " [ 13.27230835]\n",
      " [  7.00424242]\n",
      " [ 17.3453064 ]\n",
      " [ 12.44070435]\n",
      " [ 12.17385864]\n",
      " [  2.78073144]\n",
      " [ 17.98666382]\n",
      " [ 16.67526245]\n",
      " [  5.72976732]\n",
      " [ 30.67892456]\n",
      " [ 15.52609253]\n",
      " [ 20.73440552]\n",
      " [ 23.18643188]\n",
      " [ 16.4781189 ]\n",
      " [ 30.85818481]\n",
      " [ 28.9737854 ]\n",
      " [ 34.87240601]\n",
      " [ 28.31924438]\n",
      " [ 25.62683105]\n",
      " [ 27.01571655]\n",
      " [ 26.55657959]\n",
      " [ 26.07098389]\n",
      " [ 24.65081787]\n",
      " [  9.65621948]\n",
      " [ 12.40078735]\n",
      " [ 11.65145874]\n",
      " [ 19.93508911]\n",
      " [ 12.13088989]\n",
      " [  9.07815552]\n",
      " [ 11.50271606]\n",
      " [ 15.59487915]\n",
      " [ 19.13803101]\n",
      " [  2.21279931]\n",
      " [  2.23703027]\n",
      " [  4.04556322]\n",
      " [ 20.49813843]\n",
      " [ 20.16342163]\n",
      " [ 17.49728394]\n",
      " [ 17.71652222]\n",
      " [ 22.06015015]\n",
      " [ 27.68780518]\n",
      " [ 19.61532593]\n",
      " [ 16.11380005]\n",
      " [ 17.31491089]\n",
      " [ 15.07113647]\n",
      " [ 15.11422729]\n",
      " [ 27.64151001]\n",
      " [ 25.3237915 ]\n",
      " [ 26.88525391]\n",
      " [ 28.64822388]\n",
      " [ 24.74368286]\n",
      " [ 19.77828979]\n",
      " [ 26.79052734]\n",
      " [ 13.07443237]\n",
      " [ -1.20895362]\n",
      " [ 29.9263916 ]\n",
      " [ 24.59741211]\n",
      " [ 29.28445435]\n",
      " [ 30.12258911]\n",
      " [  6.37570238]\n",
      " [ 27.18722534]\n",
      " [ 23.22573853]\n",
      " [  2.39718652]\n",
      " [ 25.29537964]\n",
      " [ 21.78536987]\n",
      " [ 17.05813599]\n",
      " [ 21.24667358]\n",
      " [ 27.88955688]\n",
      " [ 27.67724609]\n",
      " [ 27.52783203]\n",
      " [ 27.26223755]\n",
      " [ 27.9836731 ]\n",
      " [ 23.44436646]\n",
      " [ 28.17367554]\n",
      " [ 21.28079224]\n",
      " [ 22.05947876]\n",
      " [ 22.67391968]\n",
      " [ 28.84823608]\n",
      " [ 21.57022095]\n",
      " [ 27.20373535]\n",
      " [ 27.82778931]\n",
      " [ 29.1362915 ]\n",
      " [ 25.5272522 ]\n",
      " [ 28.89263916]\n",
      " [ 27.61343384]\n",
      " [ 27.56082153]\n",
      " [ 26.12414551]\n",
      " [ 26.76373291]\n",
      " [ 27.39520264]\n",
      " [ 28.39874268]\n",
      " [ 24.33621216]\n",
      " [ 22.69900513]\n",
      " [ 26.45785522]\n",
      " [ 22.2774353 ]\n",
      " [ 12.57369995]\n",
      " [ 18.96969604]\n",
      " [ 23.5430603 ]\n",
      " [ 21.74057007]\n",
      " [ 31.52719116]\n",
      " [ 29.38394165]\n",
      " [ 25.17660522]\n",
      " [ 32.28848267]\n",
      " [ 18.25680542]\n",
      " [ 16.63406372]\n",
      " [  8.10232544]\n",
      " [ 27.99472046]\n",
      " [ 27.97338867]\n",
      " [ 17.48178101]\n",
      " [ 27.90939331]\n",
      " [ 28.42471313]\n",
      " [ 30.04232788]\n",
      " [ 28.01577759]\n",
      " [ 27.07455444]\n",
      " [ 25.78158569]\n",
      " [ 23.23007202]\n",
      " [ 30.82888794]\n",
      " [ 35.0428772 ]\n",
      " [ 33.44271851]\n",
      " [ 27.18161011]\n",
      " [ 25.84747314]\n",
      " [ 29.76660156]\n",
      " [ 25.50863647]\n",
      " [ 26.0333252 ]\n",
      " [ 28.53726196]\n",
      " [ 26.97103882]\n",
      " [ 22.43026733]\n",
      " [ 19.07485962]\n",
      " [ 21.21426392]\n",
      " [ 14.76864624]\n",
      " [ 23.46560669]\n",
      " [ 21.28878784]\n",
      " [ 25.86380005]\n",
      " [ 21.56216431]\n",
      " [ 30.37210083]\n",
      " [ 29.59204102]\n",
      " [ 26.01272583]\n",
      " [ 26.07214355]\n",
      " [ 29.84359741]\n",
      " [ 28.09646606]\n",
      " [ 25.13101196]\n",
      " [ 19.09298706]\n",
      " [ 23.21038818]\n",
      " [ 27.73043823]\n",
      " [ 27.98800659]\n",
      " [ 28.3598938 ]\n",
      " [ 35.0977478 ]\n",
      " [ 15.29434204]\n",
      " [ 40.11416626]\n",
      " [ 27.53384399]\n",
      " [ 37.76199341]\n",
      " [ 13.94125366]\n",
      " [ 13.32479858]\n",
      " [ 18.88369751]\n",
      " [ 12.51071167]\n",
      " [ 27.32165527]\n",
      " [ 25.31787109]\n",
      " [ 27.59320068]\n",
      " [ 24.08178711]\n",
      " [ 27.27182007]\n",
      " [ 16.95950317]\n",
      " [ 30.17697144]\n",
      " [ 19.17471313]\n",
      " [ 25.54251099]\n",
      " [ 23.46969604]\n",
      " [ 25.2102356 ]\n",
      " [ 29.75241089]\n",
      " [ 25.07785034]\n",
      " [  5.00613451]\n",
      " [ 12.12826538]\n",
      " [  5.38925219]\n",
      " [ 17.21817017]\n",
      " [ 24.36672974]\n",
      " [ 27.84146118]\n",
      " [ 29.91903687]\n",
      " [ 29.41915894]\n",
      " [ 27.62902832]\n",
      " [ 25.50817871]\n",
      " [ 26.62542725]\n",
      " [ 28.14300537]\n",
      " [ 28.69503784]\n",
      " [ 27.16043091]\n",
      " [ 34.77450562]\n",
      " [ 30.05725098]\n",
      " [ 23.63076782]\n",
      " [ 31.3152771 ]\n",
      " [ 29.42980957]\n",
      " [ 25.71890259]\n",
      " [ 19.80661011]\n",
      " [ 13.87454224]\n",
      " [ 13.64089966]\n",
      " [ 27.15206909]\n",
      " [ 26.77316284]\n",
      " [ 25.07516479]\n",
      " [ 29.23049927]\n",
      " [ 26.94058228]\n",
      " [ 23.89962769]\n",
      " [ 24.73214722]\n",
      " [ 26.68429565]\n",
      " [ 28.61935425]\n",
      " [ 19.6980896 ]\n",
      " [ 10.31295776]\n",
      " [ 18.28305054]\n",
      " [  8.08041382]\n",
      " [ 19.10830688]\n",
      " [ 28.80447388]\n",
      " [ 27.09365845]\n",
      " [ 24.93768311]\n",
      " [ 28.09503174]\n",
      " [ 26.42529297]\n",
      " [ 25.39584351]\n",
      " [ 30.21286011]\n",
      " [ 26.53164673]\n",
      " [ 30.36355591]\n",
      " [ 27.91073608]\n",
      " [ 29.78677368]\n",
      " [ 31.06155396]\n",
      " [ 27.2953186 ]\n",
      " [ 28.87002563]\n",
      " [ 21.67630005]\n",
      " [ 25.73226929]\n",
      " [ 32.07199097]\n",
      " [ 24.98080444]\n",
      " [ 24.86453247]\n",
      " [ 18.71548462]\n",
      " [ 17.45748901]\n",
      " [ 13.243927  ]\n",
      " [ 16.19796753]\n",
      " [ 17.45516968]\n",
      " [ 14.27444458]\n",
      " [ 20.87020874]\n",
      " [ 10.32705688]\n",
      " [ 26.51861572]\n",
      " [ 28.62594604]\n",
      " [ 23.87625122]\n",
      " [ 32.22897339]\n",
      " [ 37.49020386]\n",
      " [ 14.23574829]\n",
      " [ 38.91586304]\n",
      " [ 23.47109985]\n",
      " [ 28.45480347]\n",
      " [ 27.73446655]\n",
      " [ 29.52780151]\n",
      " [ 29.61468506]\n",
      " [ 26.98931885]\n",
      " [ 19.54049683]\n",
      " [ 21.1645813 ]\n",
      " [ 25.20303345]\n",
      " [ 28.2517395 ]\n",
      " [ 29.03646851]\n",
      " [ 28.46063232]\n",
      " [ 29.54473877]\n",
      " [ 27.34811401]\n",
      " [ 32.01327515]\n",
      " [ 32.34597778]\n",
      " [ 27.91513062]\n",
      " [ 28.8298645 ]\n",
      " [ 30.47653198]\n",
      " [ 31.68771362]\n",
      " [ 28.04165649]\n",
      " [ 29.05685425]\n",
      " [ 30.95465088]\n",
      " [ 26.63259888]\n",
      " [ 31.11721802]\n",
      " [ 32.02914429]\n",
      " [ 35.85705566]\n",
      " [ 38.49276733]\n",
      " [ 39.80661011]\n",
      " [ 26.01226807]\n",
      " [ 23.22360229]\n",
      " [ 29.76272583]\n",
      " [ 26.6696167 ]\n",
      " [ 23.4848938 ]\n",
      " [ 33.46566772]\n",
      " [ 28.94882202]\n",
      " [ 31.78463745]\n",
      " [ 29.97756958]\n",
      " [ 27.59957886]\n",
      " [ 27.65994263]\n",
      " [ 25.7336731 ]\n",
      " [ 22.6055603 ]\n",
      " [ 25.94946289]\n",
      " [ 27.58728027]\n",
      " [ 27.34829712]\n",
      " [ 27.13098145]\n",
      " [ 28.77416992]\n",
      " [ 29.30450439]\n",
      " [ 30.22625732]\n",
      " [ 28.20730591]\n",
      " [ 30.03338623]\n",
      " [ 28.18222046]\n",
      " [ 33.3510437 ]\n",
      " [ 28.46963501]\n",
      " [ 28.74508667]\n",
      " [ 28.56799316]\n",
      " [ 30.60501099]\n",
      " [ 30.02017212]\n",
      " [ 33.46261597]\n",
      " [ 38.77304077]\n",
      " [ 37.4703064 ]\n",
      " [ 25.55477905]\n",
      " [ 28.49862671]\n",
      " [ 25.32754517]\n",
      " [ 14.53512573]\n",
      " [ 26.85293579]\n",
      " [ 29.98355103]\n",
      " [ 32.24978638]\n",
      " [ 32.75802612]\n",
      " [ 29.03601074]\n",
      " [ 32.09133911]\n",
      " [ 28.3626709 ]\n",
      " [ 28.697052  ]\n",
      " [ 31.86264038]\n",
      " [ 32.86221313]\n",
      " [ 25.54098511]\n",
      " [ 29.0871582 ]\n",
      " [ 28.24154663]\n",
      " [ 28.89343262]\n",
      " [ 29.02883911]\n",
      " [ 24.69412231]\n",
      " [ 28.72140503]\n",
      " [ 29.85958862]\n",
      " [ 28.23272705]\n",
      " [ 28.24786377]\n",
      " [ 28.71282959]\n",
      " [ 25.16995239]\n",
      " [ 18.43466187]\n",
      " [ 25.43283081]\n",
      " [ 16.36935425]\n",
      " [ 28.11752319]\n",
      " [ 25.95141602]\n",
      " [ 31.9367981 ]\n",
      " [ 31.44802856]\n",
      " [ 31.79394531]\n",
      " [ 24.78573608]\n",
      " [ 32.56332397]\n",
      " [ 33.7840271 ]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(\"Shape: {}\".format(pred.shape))\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to see how good these predictions are.  We know what the correct MPG is for each car, so we can measure how close the neural network was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 7.4123663902282715\n"
     ]
    }
   ],
   "source": [
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(\"Final score (RMSE): {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that, on average the predictions were within +/- 5.89 values of the correct value.  This is not very good, but we will soon see how to improve it.\n",
    "\n",
    "We can also print out the first 10 cars, with predictions and actual MPG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Car name: chevrolet chevelle malibu, MPG: [ 18.], predicted MPG: [ 11.24502563]\n",
      "2. Car name: buick skylark 320, MPG: [ 15.], predicted MPG: [ 4.01193285]\n",
      "3. Car name: plymouth satellite, MPG: [ 18.], predicted MPG: [ 6.25534105]\n",
      "4. Car name: amc rebel sst, MPG: [ 16.], predicted MPG: [ 8.11264038]\n",
      "5. Car name: ford torino, MPG: [ 17.], predicted MPG: [ 9.82144165]\n",
      "6. Car name: ford galaxie 500, MPG: [ 15.], predicted MPG: [-0.88497895]\n",
      "7. Car name: chevrolet impala, MPG: [ 14.], predicted MPG: [-6.7752986]\n",
      "8. Car name: plymouth fury iii, MPG: [ 14.], predicted MPG: [-4.92581129]\n",
      "9. Car name: pontiac catalina, MPG: [ 14.], predicted MPG: [-6.42831373]\n",
      "10. Car name: amc ambassador dpl, MPG: [ 15.], predicted MPG: [-2.12612891]\n"
     ]
    }
   ],
   "source": [
    "# Sample predictions\n",
    "for i in range(10):\n",
    "    print(\"{}. Car name: {}, MPG: {}, predicted MPG: {}\".format(i+1,cars[i],y[i],pred[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple TensorFlow Classification: Iris\n",
    "\n",
    "This is a very simple example of how to perform the Iris classification using TensorFlow.  The iris.csv file is used, rather than using the built-in files that many of the Google examples require.  \n",
    "\n",
    "**Make sure that you always run previous code blocks.  If you run the code block below, without the codeblock above, you will get errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      " - 0s - loss: 1.4264\n",
      "Epoch 2/100\n",
      " - 0s - loss: 1.2595\n",
      "Epoch 3/100\n",
      " - 0s - loss: 1.1274\n",
      "Epoch 4/100\n",
      " - 0s - loss: 1.0116\n",
      "Epoch 5/100\n",
      " - 0s - loss: 0.9432\n",
      "Epoch 6/100\n",
      " - 0s - loss: 0.8849\n",
      "Epoch 7/100\n",
      " - 0s - loss: 0.8209\n",
      "Epoch 8/100\n",
      " - 0s - loss: 0.7531\n",
      "Epoch 9/100\n",
      " - 0s - loss: 0.7023\n",
      "Epoch 10/100\n",
      " - 0s - loss: 0.6625\n",
      "Epoch 11/100\n",
      " - 0s - loss: 0.6231\n",
      "Epoch 12/100\n",
      " - 0s - loss: 0.5887\n",
      "Epoch 13/100\n",
      " - 0s - loss: 0.5583\n",
      "Epoch 14/100\n",
      " - 0s - loss: 0.5327\n",
      "Epoch 15/100\n",
      " - 0s - loss: 0.5109\n",
      "Epoch 16/100\n",
      " - 0s - loss: 0.4893\n",
      "Epoch 17/100\n",
      " - 0s - loss: 0.4722\n",
      "Epoch 18/100\n",
      " - 0s - loss: 0.4560\n",
      "Epoch 19/100\n",
      " - 0s - loss: 0.4409\n",
      "Epoch 20/100\n",
      " - 0s - loss: 0.4268\n",
      "Epoch 21/100\n",
      " - 0s - loss: 0.4139\n",
      "Epoch 22/100\n",
      " - 0s - loss: 0.4024\n",
      "Epoch 23/100\n",
      " - 0s - loss: 0.3912\n",
      "Epoch 24/100\n",
      " - 0s - loss: 0.3800\n",
      "Epoch 25/100\n",
      " - 0s - loss: 0.3699\n",
      "Epoch 26/100\n",
      " - 0s - loss: 0.3604\n",
      "Epoch 27/100\n",
      " - 0s - loss: 0.3503\n",
      "Epoch 28/100\n",
      " - 0s - loss: 0.3413\n",
      "Epoch 29/100\n",
      " - 0s - loss: 0.3332\n",
      "Epoch 30/100\n",
      " - 0s - loss: 0.3223\n",
      "Epoch 31/100\n",
      " - 0s - loss: 0.3142\n",
      "Epoch 32/100\n",
      " - 0s - loss: 0.3055\n",
      "Epoch 33/100\n",
      " - 0s - loss: 0.2952\n",
      "Epoch 34/100\n",
      " - 0s - loss: 0.2882\n",
      "Epoch 35/100\n",
      " - 0s - loss: 0.2809\n",
      "Epoch 36/100\n",
      " - 0s - loss: 0.2720\n",
      "Epoch 37/100\n",
      " - 0s - loss: 0.2674\n",
      "Epoch 38/100\n",
      " - 0s - loss: 0.2581\n",
      "Epoch 39/100\n",
      " - 0s - loss: 0.2522\n",
      "Epoch 40/100\n",
      " - 0s - loss: 0.2419\n",
      "Epoch 41/100\n",
      " - 0s - loss: 0.2393\n",
      "Epoch 42/100\n",
      " - 0s - loss: 0.2308\n",
      "Epoch 43/100\n",
      " - 0s - loss: 0.2230\n",
      "Epoch 44/100\n",
      " - 0s - loss: 0.2196\n",
      "Epoch 45/100\n",
      " - 0s - loss: 0.2105\n",
      "Epoch 46/100\n",
      " - 0s - loss: 0.2077\n",
      "Epoch 47/100\n",
      " - 0s - loss: 0.2027\n",
      "Epoch 48/100\n",
      " - 0s - loss: 0.1954\n",
      "Epoch 49/100\n",
      " - 0s - loss: 0.1900\n",
      "Epoch 50/100\n",
      " - 0s - loss: 0.1841\n",
      "Epoch 51/100\n",
      " - 0s - loss: 0.1814\n",
      "Epoch 52/100\n",
      " - 0s - loss: 0.1749\n",
      "Epoch 53/100\n",
      " - 0s - loss: 0.1712\n",
      "Epoch 54/100\n",
      " - 0s - loss: 0.1664\n",
      "Epoch 55/100\n",
      " - 0s - loss: 0.1630\n",
      "Epoch 56/100\n",
      " - 0s - loss: 0.1587\n",
      "Epoch 57/100\n",
      " - 0s - loss: 0.1542\n",
      "Epoch 58/100\n",
      " - 0s - loss: 0.1508\n",
      "Epoch 59/100\n",
      " - 0s - loss: 0.1482\n",
      "Epoch 60/100\n",
      " - 0s - loss: 0.1439\n",
      "Epoch 61/100\n",
      " - 0s - loss: 0.1424\n",
      "Epoch 62/100\n",
      " - 0s - loss: 0.1399\n",
      "Epoch 63/100\n",
      " - 0s - loss: 0.1376\n",
      "Epoch 64/100\n",
      " - 0s - loss: 0.1338\n",
      "Epoch 65/100\n",
      " - 0s - loss: 0.1299\n",
      "Epoch 66/100\n",
      " - 0s - loss: 0.1293\n",
      "Epoch 67/100\n",
      " - 0s - loss: 0.1271\n",
      "Epoch 68/100\n",
      " - 0s - loss: 0.1254\n",
      "Epoch 69/100\n",
      " - 0s - loss: 0.1219\n",
      "Epoch 70/100\n",
      " - 0s - loss: 0.1184\n",
      "Epoch 71/100\n",
      " - 0s - loss: 0.1207\n",
      "Epoch 72/100\n",
      " - 0s - loss: 0.1157\n",
      "Epoch 73/100\n",
      " - 0s - loss: 0.1154\n",
      "Epoch 74/100\n",
      " - 0s - loss: 0.1121\n",
      "Epoch 75/100\n",
      " - 0s - loss: 0.1108\n",
      "Epoch 76/100\n",
      " - 0s - loss: 0.1096\n",
      "Epoch 77/100\n",
      " - 0s - loss: 0.1081\n",
      "Epoch 78/100\n",
      " - 0s - loss: 0.1065\n",
      "Epoch 79/100\n",
      " - 0s - loss: 0.1078\n",
      "Epoch 80/100\n",
      " - 0s - loss: 0.1044\n",
      "Epoch 81/100\n",
      " - 0s - loss: 0.1039\n",
      "Epoch 82/100\n",
      " - 0s - loss: 0.1024\n",
      "Epoch 83/100\n",
      " - 0s - loss: 0.1044\n",
      "Epoch 84/100\n",
      " - 0s - loss: 0.0998\n",
      "Epoch 85/100\n",
      " - 0s - loss: 0.0983\n",
      "Epoch 86/100\n",
      " - 0s - loss: 0.0977\n",
      "Epoch 87/100\n",
      " - 0s - loss: 0.0967\n",
      "Epoch 88/100\n",
      " - 0s - loss: 0.0962\n",
      "Epoch 89/100\n",
      " - 0s - loss: 0.0938\n",
      "Epoch 90/100\n",
      " - 0s - loss: 0.0933\n",
      "Epoch 91/100\n",
      " - 0s - loss: 0.0930\n",
      "Epoch 92/100\n",
      " - 0s - loss: 0.0965\n",
      "Epoch 93/100\n",
      " - 0s - loss: 0.0946\n",
      "Epoch 94/100\n",
      " - 0s - loss: 0.0941\n",
      "Epoch 95/100\n",
      " - 0s - loss: 0.0897\n",
      "Epoch 96/100\n",
      " - 0s - loss: 0.0887\n",
      "Epoch 97/100\n",
      " - 0s - loss: 0.0909\n",
      "Epoch 98/100\n",
      " - 0s - loss: 0.0921\n",
      "Epoch 99/100\n",
      " - 0s - loss: 0.0861\n",
      "Epoch 100/100\n",
      " - 0s - loss: 0.0881\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1a244b8b38>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "url=\"https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/data/iris.csv\"\n",
    "df=pd.read_csv(io.StringIO(requests.get(url).content.decode('utf-8')),na_values=['NA','?'])\n",
    "\n",
    "species = encode_text_index(df,\"species\")\n",
    "x,y = to_xy(df,\"species\")\n",
    "\n",
    "# Split into train/test\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Iris-setosa' 'Iris-versicolor' 'Iris-virginica']\n"
     ]
    }
   ],
   "source": [
    "# Print out number of species found:\n",
    "\n",
    "print(species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a neural network trained, we would like to be able to use it. The following code makes use of our neural network. Exactly like before, we will generate preditions.  Notice that 3 values come back for each of the 150 iris flowers.  There were 3 types of iris (Iris-setosa, Iris-versicolor, and Iris-virginica).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (150, 3)\n",
      "[[  9.98801351e-01   1.19864324e-03   4.02921474e-09]\n",
      " [  9.96937037e-01   3.06302961e-03   2.74731224e-08]\n",
      " [  9.97945249e-01   2.05469760e-03   1.74359194e-08]\n",
      " [  9.95330334e-01   4.66955686e-03   7.24215781e-08]\n",
      " [  9.98846173e-01   1.15384313e-03   4.07176382e-09]\n",
      " [  9.98296678e-01   1.70333043e-03   4.78728479e-09]\n",
      " [  9.97264504e-01   2.73554865e-03   3.10557553e-08]\n",
      " [  9.98084068e-01   1.91597582e-03   1.01389412e-08]\n",
      " [  9.94175494e-01   5.82446670e-03   1.41353695e-07]\n",
      " [  9.97229397e-01   2.77063227e-03   2.19415206e-08]\n",
      " [  9.99134123e-01   8.65905080e-04   1.56024105e-09]\n",
      " [  9.96804476e-01   3.19550256e-03   2.87309820e-08]\n",
      " [  9.97362792e-01   2.63724825e-03   2.41258320e-08]\n",
      " [  9.97874260e-01   2.12569628e-03   3.02767837e-08]\n",
      " [  9.99765098e-01   2.34858424e-04   9.26183644e-11]\n",
      " [  9.99592125e-01   4.07889340e-04   2.70956202e-10]\n",
      " [  9.99331892e-01   6.68106077e-04   9.40612921e-10]\n",
      " [  9.98548925e-01   1.45109848e-03   5.71691716e-09]\n",
      " [  9.98831570e-01   1.16844242e-03   1.74242221e-09]\n",
      " [  9.98786390e-01   1.21358898e-03   4.07123268e-09]\n",
      " [  9.97508645e-01   2.49138731e-03   9.05332609e-09]\n",
      " [  9.98213172e-01   1.78689882e-03   7.91140842e-09]\n",
      " [  9.99235153e-01   7.64906988e-04   2.99160718e-09]\n",
      " [  9.91322398e-01   8.67749006e-03   9.99730574e-08]\n",
      " [  9.91483986e-01   8.51584971e-03   1.31941022e-07]\n",
      " [  9.94506419e-01   5.49351890e-03   5.81933506e-08]\n",
      " [  9.95649397e-01   4.35059052e-03   3.77118248e-08]\n",
      " [  9.98653173e-01   1.34682073e-03   4.43619763e-09]\n",
      " [  9.98754859e-01   1.24518422e-03   3.98710354e-09]\n",
      " [  9.95099723e-01   4.90014302e-03   6.58099850e-08]\n",
      " [  9.94573295e-01   5.42655634e-03   7.05153340e-08]\n",
      " [  9.97759819e-01   2.24015815e-03   8.06209144e-09]\n",
      " [  9.99476492e-01   5.23586350e-04   7.32732874e-10]\n",
      " [  9.99649763e-01   3.50215472e-04   2.45434728e-10]\n",
      " [  9.96458471e-01   3.54146468e-03   3.27484742e-08]\n",
      " [  9.98696268e-01   1.30373961e-03   5.63481883e-09]\n",
      " [  9.99303699e-01   6.96260773e-04   1.00032893e-09]\n",
      " [  9.98955131e-01   1.04490551e-03   3.74791798e-09]\n",
      " [  9.96338606e-01   3.66132613e-03   6.72787266e-08]\n",
      " [  9.98264611e-01   1.73546420e-03   7.70425501e-09]\n",
      " [  9.98696148e-01   1.30390422e-03   5.23620391e-09]\n",
      " [  9.84499156e-01   1.55001953e-02   7.19365971e-07]\n",
      " [  9.97316658e-01   2.68336572e-03   3.89882793e-08]\n",
      " [  9.93772209e-01   6.22763298e-03   7.03919127e-08]\n",
      " [  9.94755864e-01   5.24417683e-03   4.22068354e-08]\n",
      " [  9.95640159e-01   4.35977522e-03   5.44124532e-08]\n",
      " [  9.98766065e-01   1.23400486e-03   4.10993373e-09]\n",
      " [  9.97091532e-01   2.90846988e-03   3.41638149e-08]\n",
      " [  9.99054492e-01   9.45555570e-04   2.03068540e-09]\n",
      " [  9.98294890e-01   1.70512416e-03   8.79214213e-09]\n",
      " [  1.29870663e-03   9.93850350e-01   4.85091796e-03]\n",
      " [  2.16765981e-03   9.78850722e-01   1.89815871e-02]\n",
      " [  1.72100833e-03   9.64290917e-01   3.39881107e-02]\n",
      " [  5.75574953e-03   8.64833474e-01   1.29410774e-01]\n",
      " [  2.50282837e-03   9.34854865e-01   6.26423731e-02]\n",
      " [  4.89645265e-03   7.89505124e-01   2.05598384e-01]\n",
      " [  2.76314816e-03   9.12130415e-01   8.51064771e-02]\n",
      " [  1.14018368e-02   9.78868544e-01   9.72959865e-03]\n",
      " [  1.97158568e-03   9.83697712e-01   1.43306833e-02]\n",
      " [  6.56568864e-03   8.73764575e-01   1.19669817e-01]\n",
      " [  7.71828135e-03   9.57310557e-01   3.49710919e-02]\n",
      " [  3.42564657e-03   9.58283067e-01   3.82913314e-02]\n",
      " [  3.34996474e-03   9.87264752e-01   9.38525423e-03]\n",
      " [  3.62327346e-03   8.16873610e-01   1.79503158e-01]\n",
      " [  8.54976568e-03   9.86167610e-01   5.28254034e-03]\n",
      " [  1.67567504e-03   9.94095564e-01   4.22874698e-03]\n",
      " [  4.53208853e-03   6.47802711e-01   3.47665220e-01]\n",
      " [  3.82150034e-03   9.87297177e-01   8.88124760e-03]\n",
      " [  3.21094296e-03   6.36821747e-01   3.59967321e-01]\n",
      " [  4.34362236e-03   9.80559349e-01   1.50970677e-02]\n",
      " [  2.14864733e-03   3.27507496e-01   6.70343876e-01]\n",
      " [  2.81962892e-03   9.91083145e-01   6.09711977e-03]\n",
      " [  2.24865694e-03   4.12626535e-01   5.85124791e-01]\n",
      " [  3.77866183e-03   8.93014848e-01   1.03206560e-01]\n",
      " [  2.17805337e-03   9.90840375e-01   6.98153535e-03]\n",
      " [  1.79737143e-03   9.91334260e-01   6.86845463e-03]\n",
      " [  1.99152553e-03   9.57668781e-01   4.03396003e-02]\n",
      " [  2.14788737e-03   7.20279813e-01   2.77572215e-01]\n",
      " [  3.73785803e-03   8.48442495e-01   1.47819713e-01]\n",
      " [  1.84218809e-02   9.79403079e-01   2.17498722e-03]\n",
      " [  4.71990416e-03   9.78789330e-01   1.64909028e-02]\n",
      " [  5.22765610e-03   9.86812055e-01   7.96029065e-03]\n",
      " [  3.65384505e-03   9.88149285e-01   8.19686614e-03]\n",
      " [  7.95271306e-04   8.70728195e-02   9.12131906e-01]\n",
      " [  4.34741378e-03   4.79417861e-01   5.16234756e-01]\n",
      " [  3.25459335e-03   9.29754555e-01   6.69909567e-02]\n",
      " [  1.86599838e-03   9.73683298e-01   2.44507566e-02]\n",
      " [  3.19101429e-03   9.28792298e-01   6.80166781e-02]\n",
      " [  4.19700146e-03   9.69402373e-01   2.64006276e-02]\n",
      " [  5.43737598e-03   9.14440811e-01   8.01217780e-02]\n",
      " [  5.78136276e-03   7.43438780e-01   2.50779837e-01]\n",
      " [  3.45645822e-03   9.05784309e-01   9.07591954e-02]\n",
      " [  3.58628388e-03   9.81553555e-01   1.48601634e-02]\n",
      " [  1.03831412e-02   9.80637908e-01   8.97892844e-03]\n",
      " [  5.08527411e-03   9.02825713e-01   9.20889899e-02]\n",
      " [  3.87278851e-03   9.76146877e-01   1.99803319e-02]\n",
      " [  4.22346732e-03   9.55488265e-01   4.02882844e-02]\n",
      " [  2.50551663e-03   9.85246241e-01   1.22482106e-02]\n",
      " [  3.78434137e-02   9.59057331e-01   3.09919938e-03]\n",
      " [  4.10414860e-03   9.64234531e-01   3.16613056e-02]\n",
      " [  2.11693641e-05   1.58046244e-03   9.98398364e-01]\n",
      " [  1.86402554e-04   1.30200973e-02   9.86793458e-01]\n",
      " [  8.07668766e-05   1.57871675e-02   9.84132051e-01]\n",
      " [  1.38121512e-04   1.38507877e-02   9.86011147e-01]\n",
      " [  4.42517048e-05   3.88480094e-03   9.96070981e-01]\n",
      " [  2.29285615e-05   4.03601211e-03   9.95941043e-01]\n",
      " [  4.98023757e-04   1.97201036e-02   9.79781866e-01]\n",
      " [  6.78506985e-05   1.27816629e-02   9.87150431e-01]\n",
      " [  9.87806707e-05   1.03943925e-02   9.89506781e-01]\n",
      " [  3.06020993e-05   6.46216236e-03   9.93507266e-01]\n",
      " [  1.11325725e-03   2.51239091e-01   7.47647643e-01]\n",
      " [  2.58563639e-04   3.36682536e-02   9.66073215e-01]\n",
      " [  2.20755552e-04   4.28061746e-02   9.56973076e-01]\n",
      " [  1.56217022e-04   8.39126017e-03   9.91452456e-01]\n",
      " [  7.88086109e-05   4.73901071e-03   9.95182216e-01]\n",
      " [  1.49715139e-04   2.10514702e-02   9.78798807e-01]\n",
      " [  3.78241530e-04   5.80461286e-02   9.41575587e-01]\n",
      " [  3.65966371e-05   1.09857526e-02   9.88977611e-01]\n",
      " [  8.80891093e-06   8.72088072e-04   9.99119103e-01]\n",
      " [  6.61699392e-04   6.27872348e-02   9.36551034e-01]\n",
      " [  8.24944436e-05   1.52267795e-02   9.84690726e-01]\n",
      " [  2.15351422e-04   1.43169612e-02   9.85467613e-01]\n",
      " [  2.38468001e-05   4.07296792e-03   9.95903194e-01]\n",
      " [  1.33546779e-03   2.34855846e-01   7.63808668e-01]\n",
      " [  1.46110004e-04   2.44120806e-02   9.75441754e-01]\n",
      " [  3.11892130e-04   8.29663351e-02   9.16721761e-01]\n",
      " [  1.79308129e-03   3.21941197e-01   6.76265717e-01]\n",
      " [  1.58381579e-03   2.54457265e-01   7.43958890e-01]\n",
      " [  6.78361830e-05   5.77576412e-03   9.94156361e-01]\n",
      " [  8.49531847e-04   2.70282537e-01   7.28867948e-01]\n",
      " [  9.80289333e-05   2.31614430e-02   9.76740539e-01]\n",
      " [  3.62340914e-04   2.07355887e-01   7.92281747e-01]\n",
      " [  5.57348503e-05   4.44794539e-03   9.95496273e-01]\n",
      " [  1.98767078e-03   3.57798755e-01   6.40213609e-01]\n",
      " [  2.43352202e-04   1.94730330e-02   9.80283558e-01]\n",
      " [  5.80725609e-05   1.84796453e-02   9.81462240e-01]\n",
      " [  4.93837360e-05   5.24047157e-03   9.94710088e-01]\n",
      " [  3.77226737e-04   5.37952445e-02   9.45827544e-01]\n",
      " [  1.85646280e-03   2.87830710e-01   7.10312784e-01]\n",
      " [  4.75211971e-04   1.21941745e-01   8.77583027e-01]\n",
      " [  5.29171375e-05   7.15238648e-03   9.92794693e-01]\n",
      " [  6.80737779e-04   2.08893463e-01   7.90425777e-01]\n",
      " [  1.86402554e-04   1.30200973e-02   9.86793458e-01]\n",
      " [  3.86577340e-05   4.91102459e-03   9.95050371e-01]\n",
      " [  3.80400379e-05   4.98855440e-03   9.94973421e-01]\n",
      " [  2.81508488e-04   5.74348532e-02   9.42283690e-01]\n",
      " [  4.69035702e-04   6.36941791e-02   9.35836792e-01]\n",
      " [  5.71524200e-04   1.04886301e-01   8.94542217e-01]\n",
      " [  1.01385747e-04   1.15668131e-02   9.88331735e-01]\n",
      " [  5.55547711e-04   5.70462607e-02   9.42398250e-01]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(\"Shape: {}\".format(pred.shape))\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# If you would like to turn of scientific notation, the following line can be used:\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]\n",
      " [ 1.  0.  0.]]\n"
     ]
    }
   ],
   "source": [
    "# The to_xy function represented the input in the same way.  Each row has only 1.0 value because each row is only one type\n",
    "# of iris.  This is the training data, we KNOW what type of iris it is.  This is called one-hot encoding.  Only one value\n",
    "# is 1.0 (hot)\n",
    "print(y[0:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1\n",
      " 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n",
      "Expected: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "# Usually the column (pred) with the highest prediction is considered to be the prediction of the neural network.  It is easy\n",
    "# to convert the predictions to the expected iris species.  The argmax function finds the index of the maximum prediction\n",
    "# for each row.\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "print(\"Predictions: {}\".format(predict_classes))\n",
    "print(\"Expected: {}\".format(expected_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Iris-setosa' 'Iris-setosa' 'Iris-setosa' 'Iris-setosa' 'Iris-setosa'\n",
      " 'Iris-setosa' 'Iris-setosa' 'Iris-setosa' 'Iris-setosa']\n"
     ]
    }
   ],
   "source": [
    "# Of course it is very easy to turn these indexes back into iris species.  We just use the species list that we created earlier.\n",
    "print(species[predict_classes[1:10]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9733333333333334\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "# Accuracy might be a more easily understood error metric.  It is essentially a test score.  For all of the iris predictions,\n",
    "# what percent were correct?  The downside is it does not consider how confident the neural network was in each prediction.\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(\"Accuracy: {}\".format(correct))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below performs two ad hoc predictions.  The first prediction is simply a single iris flower.  The second predits two iris flowers.  Notice that the argmax in the second prediction requires **axis=1**?  Since we have a 2D arraw now, we must specify which axis to take the argmax over.  The value **axis=1** specifies we want the max column index for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.00294724  0.23910996  0.7579428 ]]\n",
      "Predict that [[ 5.  3.  4.  2.]] is: Iris-virginica\n",
      "[[ 0.00294724  0.23910996  0.7579428 ]\n",
      " [ 0.9932161   0.00678376  0.00000006]]\n",
      "Predict that [[ 5.   3.   4.   2. ]\n",
      " [ 5.2  3.5  1.5  0.8]] is: ['Iris-virginica' 'Iris-setosa']\n"
     ]
    }
   ],
   "source": [
    "# ad hoc prediction\n",
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0]], dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred)\n",
    "print(\"Predict that {} is: {}\".format(sample_flower,species[pred]))\n",
    "\n",
    "# predict two sample flowers\n",
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0],[5.2,3.5,1.5,0.8]], dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(\"Predict that {} is: {}\".format(sample_flower,species[pred]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Load/Save Trained Network\n",
    "\n",
    "Complex neural networks will take a long time to fit/train.  It is helpful to be able to save these neural networks so that they can be reloaded later.  A reloaded neural network will not require retraining.  Keras provides three formats for neural network saving.\n",
    "\n",
    "* **YAML** - Stores the neural network structure (no weights) in the [YAML file format](https://en.wikipedia.org/wiki/YAML).\n",
    "* **JSON** - Stores the neural network structure (no weights) in the [JSON file format](https://en.wikipedia.org/wiki/JSON).\n",
    "* **HDF5** - Stores the complete neural network (with weights) in the [HDF5 file format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format). Do not confuse HDF5 with [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop).  They are different.  We do not use HDFS in this class.\n",
    "\n",
    "Usually you will want to save in HDF5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before save score (RMSE): 4.152003765106201\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "path = \"./data/\"\n",
    "save_path = \"./dnn/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "cars = df['name']\n",
    "df.drop('name',1,inplace=True)\n",
    "missing_median(df, 'horsepower')\n",
    "x,y = to_xy(df,\"mpg\")\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=0,epochs=100)\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x)\n",
    "\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(\"Before save score (RMSE): {}\".format(score))\n",
    "\n",
    "# save neural network structure to JSON (no weights)\n",
    "model_json = model.to_json()\n",
    "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "\n",
    "# save neural network structure to YAML (no weights)\n",
    "model_yaml = model.to_yaml()\n",
    "with open(os.path.join(save_path,\"network.yaml\"), \"w\") as yaml_file:\n",
    "    yaml_file.write(model_yaml)\n",
    "\n",
    "# save entire network to HDF5 (save everything, suggested)\n",
    "model.save(os.path.join(save_path,\"network.h5\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network.  The weights from the previous fit are used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we reload the network and perform another prediction.  The RMSE should match the previous one exactly if the neural network was really saved and reloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After load score (RMSE): 4.152003765106201\n"
     ]
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "model2 = load_model(os.path.join(save_path,\"network.h5\"))\n",
    "pred = model2.predict(x)\n",
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(\"After load score (RMSE): {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Module 3 Assignment\n",
    "\n",
    "You can find the first assignmeht here: [assignment 3](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class3.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}