cats/step_5_to_8_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feline Neural Network - Part 1\n",
    "\n",
    "**Author(s):** bfoo@google.com, kozyr@google.com\n",
    "\n",
    "Let's train a basic convolutional neural network to recognize cats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "Ensure that you have downloaded the data into your VM, i.e. you've already run these commands in your shell:\n",
    "\n",
    "```\n",
    "mkdir -p ~/data/training_small\n",
    "gsutil -m cp gs://$BUCKET/catimages/training_images/000*.png ~/data/training_small/\n",
    "gsutil -m cp gs://$BUCKET/catimages/training_images/001*.png ~/data/training_small/\n",
    "mkdir -p ~/data/debugging_small\n",
    "gsutil -m cp gs://$BUCKET/catimages/training_images/002*.png ~/data/debugging_small\n",
    "mkdir -p ~/data/training_images\n",
    "gsutil -m cp gs://$BUCKET/catimages/training_images/*.png ~/data/training_images/\n",
    "mkdir -p ~/data/validation_images\n",
    "gsutil -m cp gs://$BUCKET/catimages/validation_images/*.png ~/data/validation_images/\n",
    "```\n",
    "\n",
    "If you are going through this notebook more than once, please delete the output folder in your VM to start afresh each time by running the following so that you can start over:\n",
    "\n",
    "```rm -r ~/data/output_cnn_small```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your username:\n",
    "YOUR_GMAIL_ACCOUNT = '******' # Whatever is before @gmail.com in your email address"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Libraries for this section:\n",
    "import os\n",
    "import datetime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import tensorflow as tf\n",
    "from tensorflow.contrib.learn import RunConfig, Experiment\n",
    "from tensorflow.contrib.learn.python.learn import learn_runner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Directory settings:\n",
    "TRAIN_DIR = os.path.join('../..', YOUR_GMAIL_ACCOUNT, 'data/training_small/')  # Where the subset training dataset lives.\n",
    "DEBUG_DIR = os.path.join('../..', YOUR_GMAIL_ACCOUNT, 'data/debugging_small/')  # Where the debugging dataset lives.\n",
    "VALID_DIR = os.path.join('../..', YOUR_GMAIL_ACCOUNT, 'data/validation_images/')  # Where the full validation dataset lives.\n",
    "OUTPUT_DIR = os.path.join('../..', YOUR_GMAIL_ACCOUNT, 'data/output_cnn_small/')  # Where we store our logging and models.\n",
    "\n",
    "# TensorFlow setup:\n",
    "NUM_CLASSES = 2  # This code can be generalized beyond 2 classes (binary classification).\n",
    "QUEUE_CAP = 5000  # Number of images the TensorFlow queue can store during training.\n",
    "# For debugging, QUEUE_CAP is ignored in favor of using all images available.\n",
    "TRAIN_BATCH_SIZE = 128  # Number of images processed every training iteration.\n",
    "DEBUG_BATCH_SIZE = 64  # Number of images processed every debugging iteration.\n",
    "TRAIN_STEPS = 100 # Number of batches to use for training.\n",
    "DEBUG_STEPS = 2 # Number of batches to use for debugging.\n",
    "# Example: If dataset is 5 batches ABCDE, train_steps = 2 uses AB, train_steps = 7 uses ABCDEAB).\n",
    "\n",
    "# Monitoring setup:\n",
    "TRAINING_LOG_PERIOD_SECS = 60  # How often we want to log training metrics (from training hook in our model_fn).\n",
    "CHECKPOINT_PERIOD_SECS = 60  # How often we want to save a checkpoint.\n",
    "\n",
    "# Hyperparameters we'll tune in the tutorial:\n",
    "DROPOUT = 0.8  # Regularization parameter for neural networks - must be between 0 and 1.\n",
    "\n",
    "# Additional hyperparameters:\n",
    "LEARNING_RATE = 0.001  # Rate at which weights update.\n",
    "CNN_KERNEL_SIZE = 3  # Receptive field will be square window with this many pixels per side.\n",
    "CNN_STRIDES = 2  # Distance between consecutive receptive fields.\n",
    "CNN_FILTERS = 16  # Number of filters (new receptive fields to train, i.e. new channels) in first convolutional layer.\n",
    "FC_HIDDEN_UNITS = 512  # Number of hidden units in the fully connected layer of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize what we're working with and get the pixel count for our images.  They should be square for this to work, but luckily we padded them with black pixels where needed back in Step 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_inputs(dir, filelist=None, img_rows=1, img_cols=3, figsize=(20, 10)):\n",
    "  \"\"\"Display the first few images.\n",
    "  \n",
    "  Args:\n",
    "    dir: directory where the files are stored\n",
    "    filelist: list of filenames to pull from, if left as default, all files will be used\n",
    "    img_rows: number of rows of images to display\n",
    "    img_cols: number of columns of images to display\n",
    "    figsize: sizing for inline plots\n",
    "    \n",
    "  Returns:\n",
    "    pixel_dims: pixel dimensions (height and width) of the image\n",
    "  \"\"\"\n",
    "  if filelist is None:\n",
    "    filelist = os.listdir(dir)  # Grab all the files in the directory\n",
    "  filelist = np.array(filelist)\n",
    "  plt.close('all')\n",
    "  fig = plt.figure(figsize=figsize)\n",
    "  print('File names:')\n",
    "\n",
    "  for i in range(img_rows * img_cols):\n",
    "    print(str(filelist[i]))\n",
    "    a=fig.add_subplot(img_rows, img_cols,i + 1)\n",
    "    img = mpimg.imread(os.path.join(dir, str(filelist[i])))\n",
    "    plt.imshow(img)\n",
    "  plt.show()\n",
    "  return np.shape(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "pixel_dim = show_inputs(TRAIN_DIR)\n",
    "print('Images have ' + str(pixel_dim[0]) + 'x' + str(pixel_dim[1]) + ' pixels.')\n",
    "pixels = pixel_dim[0] * pixel_dim[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5 - Get tooling for training convolutional neural networks\n",
    "\n",
    "Here is where we enable training convolutional neural networks on data inputs like ours.  We'll build it using a TensorFlow estimator. TensorFlow (TF) is designed for scale, which means it doesn't pull all our data into memory all at once, but instead it's all about lazy execution.  We'll write functions which it will run when it's efficient to do so.  TF will pull in batches of our image data and run the functions we wrote.  \n",
    "\n",
    "In order to make this work, we need to write code for the following:\n",
    "\n",
    "* Input function: generate_input_fn()\n",
    "* Neural network architecture: cnn()\n",
    "* Model function: generate_model_fn()\n",
    "* Estimator: tf.estimator.Estimator()\n",
    "* Experiment: generate_experiment_fn()\n",
    "* Prediction generator: cat_finder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input function\n",
    "\n",
    "The input function tells TensorFlow what format of feature and label data to expect.  We'll set ours up to pull in all images in a directory we point it at.  It expects images with filenames in the following format: number_number_label.extension, so if your file naming scheme is different, please edit the input function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input function:\n",
    "def generate_input_fn(dir, batch_size, queue_capacity):\n",
    "  \"\"\"Return _input_fn for use with TF Experiment.\n",
    "  \n",
    "  Will be called in the Experiment section below (see _experiment_fn).\n",
    "  \n",
    "  Args:\n",
    "    dir: directory we're taking our files from, code is written to collect all files in this dir.\n",
    "    batch_size: number of rows ingested in each training iteration.\n",
    "    queue_capacity: number of images the TF queue can store.\n",
    "    \n",
    "  Returns:\n",
    "    _input_fn: a function that returns a batch of images and labels.\n",
    "  \"\"\"\n",
    "\n",
    "  file_pattern = os.path.join(dir, '*')  # We're pulling in all files in the directory.\n",
    "\n",
    "  def _input_fn():\n",
    "    \"\"\"A function that returns a batch of images and labels.\n",
    "\n",
    "    Args:\n",
    "      None\n",
    "\n",
    "    Returns:\n",
    "      image_batch: 4-d tensor collection of images.\n",
    "      label_batch: 1-d tensor of corresponding labels.\n",
    "    \"\"\"\n",
    "\n",
    "    height, width, channels = [pixel_dim[0], pixel_dim[1], 3]  # [height, width, 3] because there are 3 channels per image.\n",
    "    filenames_tensor = tf.train.match_filenames_once(file_pattern)  # Collect the filenames\n",
    "    # Queue that periodically reads in images from disk:\n",
    "    # When ready to run iteration, TF will take batch_size number of images out of filename_queue.\n",
    "    filename_queue = tf.train.string_input_producer(\n",
    "      filenames_tensor,\n",
    "      shuffle=False)  # Do not shuffle order of the images ingested.\n",
    "    # Convert filenames from queue into contents (png images pulled into memory):\n",
    "    reader = tf.WholeFileReader()\n",
    "    filename, contents = reader.read(filename_queue)\n",
    "    # Decodes contents pulled in into 3-d tensor per image:\n",
    "    image = tf.image.decode_png(contents, channels=channels)\n",
    "    # If dimensions mismatch, pad with zeros (black pixels) or crop to make it fit:\n",
    "    image = tf.image.resize_image_with_crop_or_pad(image, height, width)\n",
    "    # Parse out label from filename:\n",
    "    label = tf.string_to_number(tf.string_split([tf.string_split([filename], '_').values[-1]], '.').values[0])\n",
    "    # All your filenames should be in this format number_number_label.extension where label is 0 or 1.\n",
    "    # Execute above in a batch of batch_size to create a 4-d tensor of collection of images:\n",
    "    image_batch, label_batch = tf.train.batch(\n",
    "      [image, label],\n",
    "      batch_size,\n",
    "      num_threads=1,  # We'll decline the multithreading option so that everything stays in filename order.\n",
    "      capacity=queue_capacity)\n",
    "    # Normalization for better training:\n",
    "    # Change scale from pixel uint8 values between 0 and 255 into normalized float32 values between 0 and 1:\n",
    "    image_batch = tf.to_float(image_batch) / 255\n",
    "    # Rescale from (0,1) to (-1,1) so that the \"center\" of the image range is 0:\n",
    "    image_batch = (image_batch * 2) - 1\n",
    "    return image_batch, label_batch\n",
    "  return _input_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural network architecture\n",
    "\n",
    "This is where we define the architecture of the neural network we're using, such are the number of hidden layers and units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CNN architecture:\n",
    "def cnn(features, dropout, reuse, is_training):\n",
    "  \"\"\"Defines the architecture of the neural network.\n",
    "  \n",
    "  Will be called within generate_model_fn() below.\n",
    "  \n",
    "  Args: \n",
    "    features: feature data as 4-d tensor (of batch_size) pulled in when_input_fn() is executed.\n",
    "    dropout: regularization parameter in last layer (between 0 and 1, exclusive).\n",
    "    reuse: a scoping safeguard. First time training: set to False, after that, set to True.\n",
    "    is_training: if True then fits model and uses dropout, if False then doesn't consider the dropout\n",
    "    \n",
    "  Returns:\n",
    "    2-d tensor: each image's \"logit\", [logit(1-p), logit(p)]\n",
    "                where p=Pr(1) i.e. probability that class is 1 (cat in our case).\n",
    "                In CNN terminology, \"logit\" doesn't always mean the logit function you might\n",
    "                have encountered studying statistics: logit(p) = logodds(p) = log(p / (1-p))\n",
    "                Instead of converting using the inverse of the logodds function, use softmax.\n",
    "  \"\"\"\n",
    "\n",
    "  # Next, we define a scope for reusing our variables, choosing our network architecture and naming our layers.\n",
    "  with tf.variable_scope('cnn', reuse=reuse):\n",
    "    layer_1 = tf.layers.conv2d(  # 2-d convolutional layer; size of output image is (pixels/stride) a side with channels = filters.\n",
    "      inputs=features,  # previous layer (inputs) is features argument to the main function\n",
    "      kernel_size=CNN_KERNEL_SIZE,  # 3x3(x3 because we have 3 channels) receptive field (only square ones allowed)\n",
    "      strides=CNN_STRIDES,  # distance between consecutive receptive fields\n",
    "      filters=CNN_FILTERS,  # number of receptive fields to train; think of this as a CNN_FILTERS-channel image which is input to next layer)\n",
    "      padding='SAME',  # SAME uses zero padding if not all CNN_KERNEL_SIZE x CNN_KERNEL_SIZE positions are filled, VALID will ignore missing\n",
    "      activation=tf.nn.relu)  # activation function is ReLU which is f(x) = max(x, 0) \n",
    "    \n",
    "    # For simplicity, this neural network doubles the number of receptive fields (filters) with each layer.\n",
    "    # By using more filters, we are able to preserve the spatial dimensions better by storing more information.\n",
    "    #\n",
    "    # To determine how much information is preserved by each layer, consider that with each layer,\n",
    "    # the output width and height is divided by the `strides` value.\n",
    "    # When strides=2 for example, the input width W and height H is reduced by 2x, resulting in\n",
    "    # an \"image\" (formally, an activation field) for each filter output with dimensions W/2 x H/2.\n",
    "    # By doubling the number of filters compared to the input number of filters, the total output\n",
    "    # dimension becomes W/2 x H/2 x CNN_FILTERS*2, essentially compressing the input of the layer\n",
    "    # (W x H x CNN_FILTERS) to half as many total \"pixels\" (hidden units) at the output.\n",
    "    #\n",
    "    # On the other hand, increasing the number of filters will also increase the training time proportionally,\n",
    "    # as there are more weights and biases to train and convolutions to perform.\n",
    "    #\n",
    "    # As an exercise, you can play around with different numbers of filters, strides, and kernel_sizes.\n",
    "    # To avoid very long training time, make sure to keep kernel sizes small (under 5),\n",
    "    # strides at least 2 but no larger than kernel sizes (or you will skip pixels),\n",
    "    # and cap the number of filters at each level (no more than 512).\n",
    "    #\n",
    "    # When modifying these values, it is VERY important to keep track of the size of your layer outputs,\n",
    "    # i.e. the number of hidden units, since the final layer will need to be flattened into a 1D vector with size\n",
    "    # equal to the total number of hidden units. For this reason, using strides that are divisible by the width\n",
    "    # and height of the input may be the easiest way to avoid miscalculations from rounding.\n",
    "    layer_2 = tf.layers.conv2d(\n",
    "      inputs=layer_1,\n",
    "      kernel_size=CNN_KERNEL_SIZE,\n",
    "      strides=CNN_STRIDES,\n",
    "      filters=CNN_FILTERS * (2 ** 1),  # Double the number of filters from previous layer\n",
    "      padding='SAME',\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    layer_3 = tf.layers.conv2d(\n",
    "      inputs=layer_2,\n",
    "      kernel_size=CNN_KERNEL_SIZE,\n",
    "      strides=CNN_STRIDES,\n",
    "      filters=CNN_FILTERS * (2 ** 2),  # Double the number of filters from previous layer\n",
    "      padding='SAME',\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    layer_4 = tf.layers.conv2d(\n",
    "      inputs=layer_3,\n",
    "      kernel_size=CNN_KERNEL_SIZE,\n",
    "      strides=CNN_STRIDES,\n",
    "      filters=CNN_FILTERS * (2 ** 3),  # Double the number of filters from previous layer\n",
    "      padding='SAME',\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    layer_5 = tf.layers.conv2d(\n",
    "      inputs=layer_4,\n",
    "      kernel_size=CNN_KERNEL_SIZE,\n",
    "      strides=CNN_STRIDES,\n",
    "      filters=CNN_FILTERS * (2 ** 4),  # Double the number of filters from previous layer\n",
    "      padding='SAME',\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    layer_5_flat = tf.reshape(  # Flattening to 2-d tensor (1-d per image row for feedforward fully-connected layer)\n",
    "      layer_5, \n",
    "      shape=[-1, # Reshape final layer to 1-d tensor per image.\n",
    "             CNN_FILTERS * (2 ** 4) *  # Number of filters (depth), times...\n",
    "             pixels  / (CNN_STRIDES ** 5) / (CNN_STRIDES ** 5)])  # ... number of hidden units per filter (input pixels / width decimation / height decimation)\n",
    "    \n",
    "    dense_layer= tf.layers.dense(  # fully connected layer\n",
    "      inputs=layer_5_flat,\n",
    "      units=FC_HIDDEN_UNITS,  # number of hidden units\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    dropout_layer = tf.layers.dropout(  # Dropout layer randomly keeps only dropout*100% of the dense layer's hidden units in training and autonormalizes during prediction.\n",
    "      inputs=dense_layer, \n",
    "      rate=dropout,\n",
    "      training=is_training)  \n",
    "\n",
    "    return tf.layers.dense(inputs=dropout_layer, units=NUM_CLASSES)  # 2-d tensor: of \"logits\" for each image in batch. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model function\n",
    "\n",
    "The model function tells TensorFlow how to call the model we designed above and what to do when we're in training vs evaluation vs prediction mode.  This is where we define the loss function, the optimizer, and the performance metric (which we picked in Step 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model function:\n",
    "def generate_model_fn(dropout):\n",
    "  \"\"\"Return a function that determines how TF estimator operates.\n",
    "\n",
    "  The estimator has 3 modes of operation:\n",
    "  * train (fitting and updating the model)\n",
    "  * eval (collecting and returning validation metrics)\n",
    "  * predict (using the model to label unlabeled images)\n",
    "\n",
    "  The returned function _cnn_model_fn below determines what to do depending\n",
    "  on the mode of operation, and returns specs telling the estimator what to\n",
    "  execute for that mode.\n",
    "\n",
    "  Args:\n",
    "    dropout: regularization parameter in last layer (between 0 and 1, exclusive)\n",
    "\n",
    "  Returns:\n",
    "    _cnn_model_fn: a function that returns specs for use with TF estimator\n",
    "  \"\"\"\n",
    "\n",
    "  def _cnn_model_fn(features, labels, mode):\n",
    "    \"\"\"A function that determines specs for the TF estimator based on mode of operation.\n",
    "    \n",
    "    Args: \n",
    "      features: actual data (which goes into scope within estimator function) as 4-d tensor (of batch_size),\n",
    "                pulled in via tf executing _input_fn(), which is the output to generate_input_fn() and is in memory\n",
    "      labels: 1-d tensor of 0s and 1s\n",
    "      mode: TF object indicating whether we're in train, eval, or predict mode.\n",
    "      \n",
    "    Returns:\n",
    "           estim_specs: collections of metrics and tensors that are required for training (e.g. prediction values, loss value, train_op tells model weights how to update)\n",
    "    \"\"\"\n",
    "\n",
    "    # Use the cnn() to compute logits:\n",
    "    logits_train = cnn(features, dropout, reuse=False, is_training=True)\n",
    "    logits_eval = cnn(features, dropout, reuse=True, is_training=False)\n",
    "    # We'll be evaluating these later.\n",
    "\n",
    "    # Transform logits into predictions:\n",
    "    pred_classes = tf.argmax(logits_eval, axis=1)  # Returns 0 or 1, whichever has larger logit.\n",
    "    pred_prob = tf.nn.softmax(logits=logits_eval)[:, 1]  # Applies softmax function to return 2-d probability vector.\n",
    "    # Note: we're not outputting pred_prob in this tutorial, that line just shows you\n",
    "    # how to get it if you want it. Softmax[i] = exp(logit[i]) / sum(exp((logit[:]))\n",
    "\n",
    "    # If we're in prediction mode, early return predicted class (0 or 1):\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "      return tf.estimator.EstimatorSpec(mode, predictions=pred_classes)\n",
    "\n",
    "    # If we're not in prediction mode, define loss function and optimizer.\n",
    "\n",
    "    # Loss function:\n",
    "    # This is what the algorithm minimizes to learn the weights.\n",
    "    # tf.reduce_mean() just takes the mean over a batch, giving back a scalar.\n",
    "    # Inside tf.reduce_mean() we'll select any valid binary loss function we want to use.\n",
    "    loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(\n",
    "      logits=logits_train, labels=tf.cast(labels, dtype=tf.int32)))\n",
    "\n",
    "    # Optimizer:\n",
    "    # This is the scheme the algorithm uses to update the weights.\n",
    "    # AdamOptimizer is adaptive moving average, feel free to replace with one you prefer.\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE)\n",
    "\n",
    "    # The minimize method below doesn't minimize anything, it just takes a step.\n",
    "    train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())\n",
    "\n",
    "    # Performance metric:\n",
    "    # Should be whatever we chose as we defined in Step 1.  This is what you said you care about!\n",
    "    # This output is for reporting only, it is not optimized directly.\n",
    "    acc = tf.metrics.accuracy(labels=labels, predictions=pred_classes)\n",
    "\n",
    "    # Hooks - pick what to log and show:\n",
    "    # Hooks are designed for monitoring; every time TF writes a summary, it'll append these.\n",
    "    logging_hook = tf.train.LoggingTensorHook({\n",
    "      'x-entropy loss': loss,\n",
    "      'training accuracy': acc[0],\n",
    "    }, every_n_secs=TRAINING_LOG_PERIOD_SECS)\n",
    "\n",
    "    # Stitch everything together into the estimator specs, which we'll output here so it can\n",
    "    # later be passed to tf.estimator.Estimator()\n",
    "    estim_specs = tf.estimator.EstimatorSpec(\n",
    "      mode=mode,\n",
    "      predictions=pred_classes,\n",
    "      loss=loss,\n",
    "      train_op=train_op,\n",
    "      training_hooks=[logging_hook],\n",
    "      eval_metric_ops={  # This bit is Step 7!\n",
    "        'accuracy': acc,\n",
    "      }\n",
    "    )\n",
    "\n",
    "    # TF estim_specs defines a huge dict that stores different metrics and operations for useby TF Estimator.\n",
    "    # This gives you the interaction between your architecture in cnn() and the weights, etc. in the current iteration which\n",
    "    # will be used as input in the next iteration.\n",
    "    return estim_specs\n",
    "\n",
    "  return _cnn_model_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF Estimator\n",
    "\n",
    "This is where it all comes together: TF Estimator takes in as input everything we've created thus far and when executed it will output everything that is necessary for training (fits a model), evaluation (outputs metrics), or prediction (outputs predictions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TF Estimator:\n",
    "# WARNING: Don't run this block of code more than once without first changing OUTPUT_DIR.\n",
    "estimator = tf.estimator.Estimator(\n",
    "  model_fn=generate_model_fn(DROPOUT),  # Call our generate_model_fn to create model function\n",
    "  model_dir=OUTPUT_DIR,  # Where to look for data and also to paste output.\n",
    "  config=RunConfig(\n",
    "    save_checkpoints_secs=CHECKPOINT_PERIOD_SECS,\n",
    "    keep_checkpoint_max=20,\n",
    "    save_summary_steps=100,\n",
    "    log_step_count_steps=100)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF Experiment\n",
    "\n",
    "A TF Experiment defines **how** to run your TF estimator during training and debugging only.  TF Experiments are not necessary for prediction once training is complete.\n",
    "\n",
    "*TERMINOLOGY WARNING:* The word \"experiment\" here is not used the way it is used by typical scientists and statisticians.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TF Experiment:\n",
    "def experiment_fn(output_dir):\n",
    "    \"\"\"Create _experiment_fn which returns a TF experiment\n",
    "\n",
    "    To be used with learn_runner, which we imported from tf.\n",
    "\n",
    "    Args: \n",
    "      output_dir: which is where we write our models to.\n",
    "    Returns: \n",
    "      a TF Experiment\n",
    "    \"\"\"\n",
    "\n",
    "    return Experiment(\n",
    "      estimator=estimator,\n",
    "      train_input_fn=generate_input_fn(TRAIN_DIR, TRAIN_BATCH_SIZE, QUEUE_CAP),  # Generate input function above.\n",
    "      eval_input_fn=generate_input_fn(DEBUG_DIR, DEBUG_BATCH_SIZE, QUEUE_CAP),\n",
    "      train_steps=TRAIN_STEPS,  # Number of batches to use for training.\n",
    "      eval_steps=DEBUG_STEPS, # Number of batches to use for eval.\n",
    "      min_eval_frequency=1 # Run eval once every min_eval_frequency number of checkpoints.\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 6 - Train a model!\n",
    "\n",
    "Let's run our lovely creation on our training data.  In order to train, we need learn_runner(), which we imported from TensorFlow above.  For prediction, we will only need estimator.predict()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enable TF verbose output:\n",
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "start_time = datetime.datetime.now()\n",
    "print('It\\'s {:%H:%M} in London'.format(start_time) + ' --- Let\\'s get started!')\n",
    "# Let the learning commence!  Run the TF Experiment here.\n",
    "learn_runner.run(experiment_fn, OUTPUT_DIR)\n",
    "# Output lines using the word \"Validation\" are giving our metric on the non-training dataset (from DEBUG_DIR).\n",
    "end_time = datetime.datetime.now()\n",
    "print('\\nIt was {:%H:%M} in London when we started.'.format(start_time))\n",
    "print('\\nWe\\'re finished and it\\'s {:%H:%M} in London'.format(end_time))\n",
    "print('\\nCongratulations!  Training is complete!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get predictions and performance metrics\n",
    "\n",
    "Create functions for outputting observed labels, predicted labels, and accuracy.  Filenames must be in the following format: number_number_label.extension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Observed labels from filenames:\n",
    "def get_labels(dir):\n",
    "  \"\"\"Get labels from filenames.\n",
    "  \n",
    "  Filenames must be in the following format: number_number_label.png\n",
    "  \n",
    "  Args:\n",
    "    dir: directory containing image files\n",
    "    \n",
    "  Returns:\n",
    "    labels: 1-d np.array of binary labels\n",
    "  \"\"\"\n",
    "  filelist = os.listdir(dir)  # Use all the files in the directory\n",
    "  labels = np.array([])\n",
    "  for f in filelist:\n",
    "    split_filename = f.split('_')\n",
    "    label = int(split_filename[-1].split('.')[0])\n",
    "    labels = np.append(labels, label)\n",
    "  return labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cat_finder function for getting predictions:\n",
    "def cat_finder(dir, model_version):\n",
    "  \"\"\"Get labels from model.\n",
    "\n",
    "  Args:\n",
    "    dir: directory containing image files\n",
    "\n",
    "  Returns:\n",
    "    predictions: 1-d np array of binary labels\n",
    "  \"\"\"\n",
    "\n",
    "  num_predictions = len(os.listdir(dir))\n",
    "  predictions = []  # Initialize array.\n",
    "\n",
    "  # Estimator.predict() returns a generator g. Call next(g) to retrieve the next value.\n",
    "  prediction_gen = estimator.predict(\n",
    "    input_fn=generate_input_fn(dir=dir,\n",
    "                               batch_size=TRAIN_STEPS,\n",
    "                               queue_capacity=QUEUE_CAP\n",
    "                              ),\n",
    "    checkpoint_path=model_version\n",
    "  )\n",
    "\n",
    "  # Use generator to ensure ordering is preserved and predictions match order of validation_labels:\n",
    "  i = 1\n",
    "  for pred in range(0, num_predictions):\n",
    "    predictions.append(next(prediction_gen)) #Append the next value of the generator to the prediction array\n",
    "    i += 1\n",
    "    if i % 1000 == 0:\n",
    "      print('{:d} predictions completed (out of {:d})...'.format(i, len(os.listdir(dir))))\n",
    "  print('{:d} predictions completed (out of {:d})...'.format(len(os.listdir(dir)), len(os.listdir(dir))))\n",
    "  return np.array(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get training accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_accuracy(truth, predictions, threshold=0.5, roundoff = 2):\n",
    "  \"\"\"Compares labels with model predictions and returns accuracy.\n",
    "  \n",
    "  Args:\n",
    "    truth: can be bool (False, True), int (0, 1), or float (0, 1)\n",
    "    predictions: number between 0 and 1, inclusive\n",
    "    threshold: we convert the predictions to 1s if they're above this value\n",
    "    roundoff: report accuracy to how many decimal places?\n",
    "    \n",
    "  Returns:\n",
    "    accuracy: number correct divided by total predictions\n",
    "  \"\"\"\n",
    "  truth = np.array(truth) == (1|True)\n",
    "  predicted = np.array(predictions) >= threshold\n",
    "  matches = sum(predicted == truth)\n",
    "  accuracy = float(matches) / len(truth)\n",
    "  return round(accuracy, roundoff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = os.listdir(TRAIN_DIR)\n",
    "model_version = os.path.join(OUTPUT_DIR, 'model.ckpt-' + str(TRAIN_STEPS))\n",
    "predicted = cat_finder(TRAIN_DIR, model_version)\n",
    "observed = get_labels(TRAIN_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy is ' + str(get_accuracy(observed, predicted)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 7 - Debugging and Tuning\n",
    "\n",
    "## Debugging\n",
    "\n",
    "It's worth taking a look to see if there's something special about the images we misclassified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = os.listdir(DEBUG_DIR)\n",
    "model_version = os.path.join(OUTPUT_DIR + 'model.ckpt-' + str(TRAIN_STEPS))\n",
    "predicted = cat_finder(DEBUG_DIR, model_version)\n",
    "observed = get_labels(DEBUG_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Debugging accuracy is ' + str(get_accuracy(observed, predicted)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'files': files, 'predicted': predicted, 'observed': observed})\n",
    "hit = df.files[df.observed == df.predicted]\n",
    "miss = df.files[df.observed != df.predicted]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show successful classifications:\n",
    "show_inputs(DEBUG_DIR, hit, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show unsuccessful classifications:\n",
    "show_inputs(DEBUG_DIR, miss, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning\n",
    "\n",
    "In this demo, we'll see extremely basic tuning of the dropout rate.  There are plenty of more sophisticated options (check out Cloud ML Engine!) but I'd like to show you that the core principles are simple.  We'll just step through an array of options and see which dropout setting gets you the best accuracy, then we'll select that one when we train with more data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Disable TF verbose output:\n",
    "tf.logging.set_verbosity(tf.logging.FATAL)\n",
    "\n",
    "# Get output:\n",
    "dropouts = np.array([])\n",
    "accuracies = np.array([])\n",
    "for i in range(9):\n",
    "  tune_output_dir = os.path.join(OUTPUT_DIR, 'dropout0.' + str(i + 1) + '/')\n",
    "  tune_dropout = (float(i) + 1) / 10\n",
    "  print('It\\'s {:%H:%M} in London'.format(datetime.datetime.now()) + ' --- Dropout setting is ' + str(tune_dropout))\n",
    "  # Try a new dropout setting for TF Estimator:\n",
    "  estimator = tf.estimator.Estimator(\n",
    "    model_fn=generate_model_fn(tune_dropout),\n",
    "    model_dir=tune_output_dir,\n",
    "    config=RunConfig(\n",
    "      save_checkpoints_secs=CHECKPOINT_PERIOD_SECS,\n",
    "      keep_checkpoint_max=20,\n",
    "      save_summary_steps=100,\n",
    "      log_step_count_steps=100\n",
    "    )\n",
    "  )\n",
    "  # Train it!\n",
    "  learn_runner.run(experiment_fn, tune_output_dir)\n",
    "  # Identify the model version:\n",
    "  tuned_model = os.path.join(tune_output_dir, 'model.ckpt-' + str(TRAIN_STEPS))\n",
    "  # Output predicted and observed labels:\n",
    "  predicted = cat_finder(DEBUG_DIR, model_version=tuned_model)\n",
    "  observed = get_labels(DEBUG_DIR)\n",
    "  # Compute performance metric:\n",
    "  accuracy = get_accuracy(truth=observed, predictions=predicted)\n",
    "  print('Accuracy is: ' + str(accuracy))\n",
    "  # Append to array:\n",
    "  dropouts = np.append(dropouts, tune_dropout)\n",
    "  accuracies = np.append(accuracies, accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_dropout = dropouts[np.argmax(accuracies)]\n",
    "print(\"Dropout tuning complete! Set hyperparameter to \" + str(best_dropout) + \".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 8 - Validation\n",
    "\n",
    "Imagine we had skipped Step 7 (don't worry, we'll use the tuned hyperparameter when we train again in the next notebook) and we considered launching the CNN model we just trained in Step 6.  The training accuracy looked good, right?  Never fear, validation will keep you safe!  It's important to validate before trusting a model, so let's do that now and apply cat_finder() to our validation dataset.  Since this is validation, we'll only look at the final performance metric (accuracy) and nothing else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = os.listdir(VALID_DIR)\n",
    "predicted = cat_finder(VALID_DIR, model_version)\n",
    "observed = get_labels(VALID_DIR)\n",
    "print('\\nValidation accuracy is ' + str(get_accuracy(observed, predicted)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}