batch-norm/Batch_Normalization_Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization – Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch normalization is most useful when building deep neural networks. To demonstrate this, we'll create a convolutional neural network with 20 convolutional layers, followed by a fully connected layer. We'll use it to classify handwritten digits in the MNIST dataset, which should be familiar to you by now.\n",
    "\n",
    "This is **not** a good network for classfying MNIST digits. You could create a _much_ simpler network and get _better_ results. However, to give you hands-on experience with batch normalization, we had to make an example that was:\n",
    "1. Complicated enough that training would benefit from batch normalization.\n",
    "2. Simple enough that it would train quickly, since this is meant to be a short exercise just to give you some practice adding batch normalization.\n",
    "3. Simple enough that the architecture would be easy to understand without additional resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook includes two versions of the network that you can edit. The first uses higher level functions from the `tf.layers` package. The second is the same network, but uses only lower level functions in the `tf.nn` package.\n",
    "\n",
    "1. [Batch Normalization with `tf.layers.batch_normalization`](#example_1)\n",
    "2. [Batch Normalization with `tf.nn.batch_normalization`](#example_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell loads TensorFlow, downloads the MNIST dataset if necessary, and loads it into an object named `mnist`. You'll need to run this cell before running anything else in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True, reshape=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization using `tf.layers.batch_normalization`<a id=\"example_1\"></a>\n",
    "\n",
    "This version of the network uses `tf.layers` for almost everything, and expects you to implement batch normalization using [`tf.layers.batch_normalization`](https://www.tensorflow.org/api_docs/python/tf/layers/batch_normalization) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the following function to create fully connected layers in our network. We'll create them with the specified number of neurons and a ReLU activation function.\n",
    "\n",
    "This version of the function does not include batch normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DO NOT MODIFY THIS CELL\n",
    "\"\"\"\n",
    "def fully_connected(prev_layer, num_units):\n",
    "    \"\"\"\n",
    "    Create a fully connectd layer with the given layer as input and the given number of neurons.\n",
    "    \n",
    "    :param prev_layer: Tensor\n",
    "        The Tensor that acts as input into this layer\n",
    "    :param num_units: int\n",
    "        The size of the layer. That is, the number of units, nodes, or neurons.\n",
    "    :returns Tensor\n",
    "        A new fully connected layer\n",
    "    \"\"\"\n",
    "    layer = tf.layers.dense(prev_layer, num_units, activation=tf.nn.relu)\n",
    "    return layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the following function to create convolutional layers in our network. They are very basic: we're always using a 3x3 kernel, ReLU activation functions, strides of 1x1 on layers with odd depths, and strides of 2x2 on layers with even depths. We aren't bothering with pooling layers at all in this network.\n",
    "\n",
    "This version of the function does not include batch normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DO NOT MODIFY THIS CELL\n",
    "\"\"\"\n",
    "def conv_layer(prev_layer, layer_depth):\n",
    "    \"\"\"\n",
    "    Create a convolutional layer with the given layer as input.\n",
    "    \n",
    "    :param prev_layer: Tensor\n",
    "        The Tensor that acts as input into this layer\n",
    "    :param layer_depth: int\n",
    "        We'll set the strides and number of feature maps based on the layer's depth in the network.\n",
    "        This is *not* a good way to make a CNN, but it helps us create this example with very little code.\n",
    "    :returns Tensor\n",
    "        A new convolutional layer\n",
    "    \"\"\"\n",
    "    strides = 2 if layer_depth % 3 == 0 else 1\n",
    "    conv_layer = tf.layers.conv2d(prev_layer, layer_depth*4, 3, strides, 'same', activation=tf.nn.relu)\n",
    "    return conv_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Run the following cell**, along with the earlier cells (to load the dataset and define the necessary functions). \n",
    "\n",
    "This cell builds the network **without** batch normalization, then trains it on the MNIST dataset. It displays loss and accuracy data periodically while training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch:  0: Validation loss: 0.69066, Validation accuracy: 0.09580\n",
      "Batch: 25: Training loss: 0.33632, Training accuracy: 0.07812\n",
      "Batch: 50: Training loss: 0.32540, Training accuracy: 0.10938\n",
      "Batch: 75: Training loss: 0.32689, Training accuracy: 0.09375\n",
      "Batch: 100: Validation loss: 0.32562, Validation accuracy: 0.10700\n",
      "Batch: 125: Training loss: 0.32580, Training accuracy: 0.03125\n",
      "Batch: 150: Training loss: 0.32605, Training accuracy: 0.10938\n",
      "Batch: 175: Training loss: 0.32327, Training accuracy: 0.09375\n",
      "Batch: 200: Validation loss: 0.32601, Validation accuracy: 0.09860\n",
      "Batch: 225: Training loss: 0.32541, Training accuracy: 0.07812\n",
      "Batch: 250: Training loss: 0.32626, Training accuracy: 0.07812\n",
      "Batch: 275: Training loss: 0.32360, Training accuracy: 0.12500\n",
      "Batch: 300: Validation loss: 0.32589, Validation accuracy: 0.09240\n",
      "Batch: 325: Training loss: 0.32271, Training accuracy: 0.15625\n",
      "Batch: 350: Training loss: 0.32303, Training accuracy: 0.12500\n",
      "Batch: 375: Training loss: 0.32643, Training accuracy: 0.10938\n",
      "Batch: 400: Validation loss: 0.32498, Validation accuracy: 0.11260\n",
      "Batch: 425: Training loss: 0.32552, Training accuracy: 0.09375\n",
      "Batch: 450: Training loss: 0.32669, Training accuracy: 0.10938\n",
      "Batch: 475: Training loss: 0.32649, Training accuracy: 0.04688\n",
      "Batch: 500: Validation loss: 0.32516, Validation accuracy: 0.11000\n",
      "Batch: 525: Training loss: 0.32924, Training accuracy: 0.01562\n",
      "Batch: 550: Training loss: 0.32491, Training accuracy: 0.20312\n",
      "Batch: 575: Training loss: 0.32237, Training accuracy: 0.28125\n",
      "Batch: 600: Validation loss: 0.32532, Validation accuracy: 0.09240\n",
      "Batch: 625: Training loss: 0.32465, Training accuracy: 0.14062\n",
      "Batch: 650: Training loss: 0.32736, Training accuracy: 0.07812\n",
      "Batch: 675: Training loss: 0.32522, Training accuracy: 0.12500\n",
      "Batch: 700: Validation loss: 0.32503, Validation accuracy: 0.09860\n",
      "Batch: 725: Training loss: 0.32523, Training accuracy: 0.09375\n",
      "Batch: 750: Training loss: 0.32533, Training accuracy: 0.14062\n",
      "Batch: 775: Training loss: 0.32533, Training accuracy: 0.12500\n",
      "Final validation accuracy: 0.11420\n",
      "Final test accuracy: 0.12190\n",
      "Accuracy on 100 samples: 0.13\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DO NOT MODIFY THIS CELL\n",
    "\"\"\"\n",
    "def train(num_batches, batch_size, learning_rate):\n",
    "    # Build placeholders for the input samples and labels \n",
    "    inputs = tf.placeholder(tf.float32, [None, 28, 28, 1])\n",
    "    labels = tf.placeholder(tf.float32, [None, 10])\n",
    "    \n",
    "    # Feed the inputs into a series of 20 convolutional layers \n",
    "    layer = inputs\n",
    "    for layer_i in range(1, 20):\n",
    "        layer = conv_layer(layer, layer_i)\n",
    "\n",
    "    # Flatten the output from the convolutional layers \n",
    "    orig_shape = layer.get_shape().as_list()\n",
    "    layer = tf.reshape(layer, shape=[-1, orig_shape[1] * orig_shape[2] * orig_shape[3]])\n",
    "\n",
    "    # Add one fully connected layer\n",
    "    layer = fully_connected(layer, 100)\n",
    "\n",
    "    # Create the output layer with 1 node for each \n",
    "    logits = tf.layers.dense(layer, 10)\n",
    "    \n",
    "    # Define \n",
    "    model_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))\n",
    "    \n",
    "    train_opt = tf.train.AdamOptimizer(learning_rate).minimize(model_loss)\n",
    "    \n",
    "    correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(labels,1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    \n",
    "    # Train and test the network\n",
    "    with tf.Session() as sess:\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        for batch_i in range(num_batches):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "\n",
    "            # train this batch\n",
    "            sess.run(train_opt, {inputs: batch_xs, \n",
    "                                 labels: batch_ys})\n",
    "            \n",
    "            # Periodically check the validation or training loss and accuracy\n",
    "            if batch_i % 100 == 0:\n",
    "                loss, acc = sess.run([model_loss, accuracy], {inputs: mnist.validation.images,\n",
    "                                                              labels: mnist.validation.labels})\n",
    "                print('Batch: {:>2}: Validation loss: {:>3.5f}, Validation accuracy: {:>3.5f}'.format(batch_i, loss, acc))\n",
    "            elif batch_i % 25 == 0:\n",
    "                loss, acc = sess.run([model_loss, accuracy], {inputs: batch_xs, labels: batch_ys})\n",
    "                print('Batch: {:>2}: Training loss: {:>3.5f}, Training accuracy: {:>3.5f}'.format(batch_i, loss, acc))\n",
    "\n",
    "        # At the end, score the final accuracy for both the validation and test sets\n",
    "        acc = sess.run(accuracy, {inputs: mnist.validation.images,\n",
    "                                  labels: mnist.validation.labels})\n",
    "        print('Final validation accuracy: {:>3.5f}'.format(acc))\n",
    "        acc = sess.run(accuracy, {inputs: mnist.test.images,\n",
    "                                  labels: mnist.test.labels})\n",
    "        print('Final test accuracy: {:>3.5f}'.format(acc))\n",
    "        \n",
    "        # Score the first 100 test images individually, just to make sure batch normalization really worked\n",
    "        correct = 0\n",
    "        for i in range(100):\n",
    "            correct += sess.run(accuracy,feed_dict={inputs: [mnist.test.images[i]],\n",
    "                                                    labels: [mnist.test.labels[i]]})\n",
    "\n",
    "        print(\"Accuracy on 100 samples:\", correct/100)\n",
    "\n",
    "\n",
    "num_batches = 800\n",
    "batch_size = 64\n",
    "learning_rate = 0.002\n",
    "\n",
    "tf.reset_default_graph()\n",
    "with tf.Graph().as_default():\n",
    "    train(num_batches, batch_size, learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this many layers, it's going to take a lot of iterations for this network to learn. By the time you're done training these 800 batches, your final test and validation accuracies probably won't be much better than 10%. (It will be different each time, but will most likely be less than 15%.)\n",
    "\n",
    "Using batch normalization, you'll be able to train this same network to over 90% in that same number of batches.\n",
    "\n",
    "# Add batch normalization\n",
    "\n",
    "To add batch normalization to the layers created by `fully_connected`, we did the following:\n",
    "1. Added the `is_training` parameter to the function signature so we can pass that information to the batch normalization layer.\n",
    "2. Removed the bias and activation function from the `dense` layer.\n",
    "3. Used `tf.layers.batch_normalization` to normalize the layer's output. Notice we pass `is_training` to this layer to ensure the network updates its population statistics appropriately.\n",
    "4. Passed the normalized values into a ReLU activation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fully_connected(prev_layer, num_units, is_training):\n",
    "    \"\"\"\n",
    "    Create a fully connectd layer with the given layer as input and the given number of neurons.\n",
    "    \n",
    "    :param prev_layer: Tensor\n",
    "        The Tensor that acts as input into this layer\n",
    "    :param num_units: int\n",
    "        The size of the layer. That is, the number of units, nodes, or neurons.\n",
    "    :param is_training: bool or Tensor\n",
    "        Indicates whether or not the network is currently training, which tells the batch normalization\n",
    "        layer whether or not it should update or use its population statistics.\n",
    "    :returns Tensor\n",
    "        A new fully connected layer\n",
    "    \"\"\"\n",
    "    layer = tf.layers.dense(prev_layer, num_units, use_bias=False, activation=None)\n",
    "    layer = tf.layers.batch_normalization(layer, training=is_training)\n",
    "    layer = tf.nn.relu(layer)\n",
    "    return layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add batch normalization to the layers created by `conv_layer`, we did the following:\n",
    "1. Added the `is_training` parameter to the function signature so we can pass that information to the batch normalization layer.\n",
    "2. Removed the bias and activation function from the `conv2d` layer.\n",
    "3. Used `tf.layers.batch_normalization` to normalize the convolutional layer's output. Notice we pass `is_training` to this layer to ensure the network updates its population statistics appropriately.\n",
    "4. Passed the normalized values into a ReLU activation function.\n",
    "\n",
    "If you compare this function to `fully_connected`, you'll see that – when using `tf.layers` – there really isn't any difference between normalizing a fully connected layer and a convolutional layer. However, if you look at the second example in this notebook, where we restrict ourselves to the `tf.nn` package, you'll see a small difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_layer(prev_layer, layer_depth, is_training):\n",
    "    \"\"\"\n",
    "    Create a convolutional layer with the given layer as input.\n",
    "    \n",
    "    :param prev_layer: Tensor\n",
    "        The Tensor that acts as input into this layer\n",
    "    :param layer_depth: int\n",
    "        We'll set the strides and number of feature maps based on the layer's depth in the network.\n",
    "        This is *not* a good way to make a CNN, but it helps us create this example with very little code.\n",
    "    :param is_training: bool or Tensor\n",
    "        Indicates whether or not the network is currently training, which tells the batch normalization\n",
    "        layer whether or not it should update or use its population statistics.\n",
    "    :returns Tensor\n",
    "        A new convolutional layer\n",
    "    \"\"\"\n",
    "    strides = 2 if layer_depth % 3 == 0 else 1\n",
    "    conv_layer = tf.layers.conv2d(prev_layer, layer_depth*4, 3, strides, 'same', use_bias=False, activation=None)\n",
    "    conv_layer = tf.layers.batch_normalization(conv_layer, training=is_training)\n",
    "    conv_layer = tf.nn.relu(conv_layer)\n",
    "\n",
    "    return conv_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch normalization is still a new enough idea that researchers are still discovering how best to use it. In general, people seem to agree to remove the layer's bias (because the batch normalization already has terms for scaling and shifting) and add batch normalization _before_ the layer's non-linear activation function. However, for some networks it will work well in other ways, too. \n",
    "\n",
    "Just to demonstrate this point, the following three versions of `conv_layer` show other ways to implement batch normalization. If you try running with any of these versions of the function, they should all still work fine (although some versions may still work better than others). \n",
    "\n",
    "**Alternate solution that uses bias in the convolutional layer but still adds batch normalization before the ReLU activation function.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_layer(prev_layer, layer_num, is_training):\n",
    "    strides = 2 if layer_num % 3 == 0 else 1\n",
    "    conv_layer = tf.layers.conv2d(prev_layer, layer_num*4, 3, strides, 'same', use_bias=True, activation=None)\n",
    "    conv_layer = tf.layers.batch_normalization(conv_layer, training=is_training)\n",
    "    conv_layer = tf.nn.relu(conv_layer)\n",
    "    return conv_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Alternate solution that uses a bias and ReLU activation function _before_ batch normalization.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_layer(prev_layer, layer_num, is_training):\n",
    "    strides = 2 if layer_num % 3 == 0 else 1\n",
    "    conv_layer = tf.layers.conv2d(prev_layer, layer_num*4, 3, strides, 'same', use_bias=True, activation=tf.nn.relu)\n",
    "    conv_layer = tf.layers.batch_normalization(conv_layer, training=is_training)\n",
    "    return conv_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Alternate solution that uses a ReLU activation function _before_ normalization, but no bias.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_layer(prev_layer, layer_num, is_training):\n",
    "    strides = 2 if layer_num % 3 == 0 else 1\n",
    "    conv_layer = tf.layers.conv2d(prev_layer, layer_num*4, 3, strides, 'same', use_bias=False, activation=tf.nn.relu)\n",
    "    conv_layer = tf.layers.batch_normalization(conv_layer, training=is_training)\n",
    "    return conv_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To modify `train`, we did the following:\n",
    "1. Added `is_training`, a placeholder to store a boolean value indicating whether or not the network is training.\n",
    "2. Passed `is_training` to the `conv_layer` and `fully_connected` functions.\n",
    "3. Each time we call `run` on the session, we added to `feed_dict` the appropriate value for `is_training`.\n",
    "4. Moved the creation of `train_opt` inside a `with tf.control_dependencies...` statement. This is necessary to get the normalization layers created with `tf.layers.batch_normalization` to update their population statistics, which we need when performing inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch:  0: Validation loss: 0.69135, Validation accuracy: 0.09860\n",
      "Batch: 25: Training loss: 0.58727, Training accuracy: 0.10938\n",
      "Batch: 50: Training loss: 0.47633, Training accuracy: 0.09375\n",
      "Batch: 75: Training loss: 0.40283, Training accuracy: 0.17188\n",
      "Batch: 100: Validation loss: 0.36413, Validation accuracy: 0.11260\n",
      "Batch: 125: Training loss: 0.34614, Training accuracy: 0.09375\n",
      "Batch: 150: Training loss: 0.34372, Training accuracy: 0.04688\n",
      "Batch: 175: Training loss: 0.31415, Training accuracy: 0.25000\n",
      "Batch: 200: Validation loss: 0.33029, Validation accuracy: 0.20880\n",
      "Batch: 225: Training loss: 0.37454, Training accuracy: 0.20312\n",
      "Batch: 250: Training loss: 0.35721, Training accuracy: 0.42188\n",
      "Batch: 275: Training loss: 0.30789, Training accuracy: 0.46875\n",
      "Batch: 300: Validation loss: 0.18574, Validation accuracy: 0.71100\n",
      "Batch: 325: Training loss: 0.22868, Training accuracy: 0.64062\n",
      "Batch: 350: Training loss: 0.27518, Training accuracy: 0.65625\n",
      "Batch: 375: Training loss: 0.05569, Training accuracy: 0.90625\n",
      "Batch: 400: Validation loss: 0.09854, Validation accuracy: 0.85200\n",
      "Batch: 425: Training loss: 0.04723, Training accuracy: 0.90625\n",
      "Batch: 450: Training loss: 0.02127, Training accuracy: 0.98438\n",
      "Batch: 475: Training loss: 0.00786, Training accuracy: 0.98438\n",
      "Batch: 500: Validation loss: 0.04686, Validation accuracy: 0.93780\n",
      "Batch: 525: Training loss: 0.02769, Training accuracy: 0.95312\n",
      "Batch: 550: Training loss: 0.00446, Training accuracy: 1.00000\n",
      "Batch: 575: Training loss: 0.05585, Training accuracy: 0.92188\n",
      "Batch: 600: Validation loss: 0.03143, Validation accuracy: 0.95780\n",
      "Batch: 625: Training loss: 0.00918, Training accuracy: 0.98438\n",
      "Batch: 650: Training loss: 0.03437, Training accuracy: 0.95312\n",
      "Batch: 675: Training loss: 0.03991, Training accuracy: 0.96875\n",
      "Batch: 700: Validation loss: 0.03455, Validation accuracy: 0.95300\n",
      "Batch: 725: Training loss: 0.03672, Training accuracy: 0.93750\n",
      "Batch: 750: Training loss: 0.03293, Training accuracy: 0.95312\n",
      "Batch: 775: Training loss: 0.02039, Training accuracy: 0.96875\n",
      "Final validation accuracy: 0.97460\n",
      "Final test accuracy: 0.97450\n",
      "Accuracy on 100 samples: 0.98\n"
     ]
    }
   ],
   "source": [
    "def train(num_batches, batch_size, learning_rate):\n",
    "    # Build placeholders for the input samples and labels \n",
    "    inputs = tf.placeholder(tf.float32, [None, 28, 28, 1])\n",
    "    labels = tf.placeholder(tf.float32, [None, 10])\n",
    "\n",
    "    # Add placeholder to indicate whether or not we're training the model\n",
    "    is_training = tf.placeholder(tf.bool)\n",
    "\n",
    "    # Feed the inputs into a series of 20 convolutional layers \n",
    "    layer = inputs\n",
    "    for layer_i in range(1, 20):\n",
    "        layer = conv_layer(layer, layer_i, is_training)\n",
    "\n",
    "    # Flatten the output from the convolutional layers \n",
    "    orig_shape = layer.get_shape().as_list()\n",
    "    layer = tf.reshape(layer, shape=[-1, orig_shape[1] * orig_shape[2] * orig_shape[3]])\n",
    "\n",
    "    # Add one fully connected layer\n",
    "    layer = fully_connected(layer, 100, is_training)\n",
    "\n",
    "    # Create the output layer with 1 node for each \n",
    "    logits = tf.layers.dense(layer, 10)\n",
    "    \n",
    "    # Define loss and training operations\n",
    "    model_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))\n",
    "    \n",
    "    # Tell TensorFlow to update the population statistics while training\n",
    "    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):\n",
    "        train_opt = tf.train.AdamOptimizer(learning_rate).minimize(model_loss)\n",
    "    \n",
    "    # Create operations to test accuracy\n",
    "    correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(labels,1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    \n",
    "    # Train and test the network\n",
    "    with tf.Session() as sess:\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        for batch_i in range(num_batches):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "\n",
    "            # train this batch\n",
    "            sess.run(train_opt, {inputs: batch_xs, labels: batch_ys, is_training: True})\n",
    "            \n",
    "            # Periodically check the validation or training loss and accuracy\n",
    "            if batch_i % 100 == 0:\n",
    "                loss, acc = sess.run([model_loss, accuracy], {inputs: mnist.validation.images,\n",
    "                                                              labels: mnist.validation.labels,\n",
    "                                                              is_training: False})\n",
    "                print('Batch: {:>2}: Validation loss: {:>3.5f}, Validation accuracy: {:>3.5f}'.format(batch_i, loss, acc))\n",
    "            elif batch_i % 25 == 0:\n",
    "                loss, acc = sess.run([model_loss, accuracy], {inputs: batch_xs, labels: batch_ys, is_training: False})\n",
    "                print('Batch: {:>2}: Training loss: {:>3.5f}, Training accuracy: {:>3.5f}'.format(batch_i, loss, acc))\n",
    "\n",
    "        # At the end, score the final accuracy for both the validation and test sets\n",
    "        acc = sess.run(accuracy, {inputs: mnist.validation.images,\n",
    "                                  labels: mnist.validation.labels, \n",
    "                                  is_training: False})\n",
    "        print('Final validation accuracy: {:>3.5f}'.format(acc))\n",
    "        acc = sess.run(accuracy, {inputs: mnist.test.images,\n",
    "                                  labels: mnist.test.labels,\n",
    "                                  is_training: False})\n",
    "        print('Final test accuracy: {:>3.5f}'.format(acc))\n",
    "        \n",
    "        # Score the first 100 test images individually, just to make sure batch normalization really worked\n",
    "        correct = 0\n",
    "        for i in range(100):\n",
    "            correct += sess.run(accuracy,feed_dict={inputs: [mnist.test.images[i]],\n",
    "                                                    labels: [mnist.test.labels[i]],\n",
    "                                                    is_training: False})\n",
    "\n",
    "        print(\"Accuracy on 100 samples:\", correct/100)\n",
    "\n",
    "\n",
    "num_batches = 800\n",
    "batch_size = 64\n",
    "learning_rate = 0.002\n",
    "\n",
    "tf.reset_default_graph()\n",
    "with tf.Graph().as_default():\n",
    "    train(num_batches, batch_size, learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With batch normalization, we now get excellent performance. In fact, validation accuracy is almost 94% after only 500 batches. Notice also the last line of the output: `Accuracy on 100 samples`. If this value is low while everything else looks good, that means you did not implement batch normalization correctly. Specifically, it means you either did not calculate the population mean and variance while training, or you are not using those values during inference.\n",
    "\n",
    "# Batch Normalization using `tf.nn.batch_normalization`<a id=\"example_2\"></a>\n",
    "\n",
    "Most of the time you will be able to use higher level functions exclusively, but sometimes you may want to work at a lower level. For example, if you ever want to implement a new feature – something new enough that TensorFlow does not already include a high-level implementation of it, like batch normalization in an LSTM – then you may need to know these sorts of things.\n",
    "\n",
    "This version of the network uses `tf.nn` for almost everything, and expects you to implement batch normalization using [`tf.nn.batch_normalization`](https://www.tensorflow.org/api_docs/python/tf/nn/batch_normalization).\n",
    "\n",
    "This implementation of `fully_connected` is much more involved than the one that uses `tf.layers`. However, if you went through the `Batch_Normalization_Lesson` notebook, things should look pretty familiar. To add batch normalization, we did the following:\n",
    "1. Added the `is_training` parameter to the function signature so we can pass that information to the batch normalization layer.\n",
    "2. Removed the bias and activation function from the `dense` layer.\n",
    "3. Added `gamma`, `beta`, `pop_mean`, and `pop_variance` variables.\n",
    "4. Used `tf.cond` to make handle training and inference differently.\n",
    "5. When training, we use `tf.nn.moments` to calculate the batch mean and variance. Then we update the population statistics and use `tf.nn.batch_normalization` to normalize the layer's output using the batch statistics. Notice the `with tf.control_dependencies...` statement - this is required to force TensorFlow to run the operations that update the population statistics.\n",
    "6. During inference (i.e. when not training), we use `tf.nn.batch_normalization` to normalize the layer's output using the population statistics we calculated during training.\n",
    "7. Passed the normalized values into a ReLU activation function.\n",
    "\n",
    "If any of thise code is unclear, it is almost identical to what we showed in the `fully_connected` function in the `Batch_Normalization_Lesson` notebook. Please see that for extensive comments. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fully_connected(prev_layer, num_units, is_training):\n",
    "    \"\"\"\n",
    "    Create a fully connectd layer with the given layer as input and the given number of neurons.\n",
    "    \n",
    "    :param prev_layer: Tensor\n",
    "        The Tensor that acts as input into this layer\n",
    "    :param num_units: int\n",
    "        The size of the layer. That is, the number of units, nodes, or neurons.\n",
    "    :param is_training: bool or Tensor\n",
    "        Indicates whether or not the network is currently training, which tells the batch normalization\n",
    "        layer whether or not it should update or use its population statistics.\n",
    "    :returns Tensor\n",
    "        A new fully connected layer\n",
    "    \"\"\"\n",
    "\n",
    "    layer = tf.layers.dense(prev_layer, num_units, use_bias=False, activation=None)\n",
    "\n",
    "    gamma = tf.Variable(tf.ones([num_units]))\n",
    "    beta = tf.Variable(tf.zeros([num_units]))\n",
    "\n",
    "    pop_mean = tf.Variable(tf.zeros([num_units]), trainable=False)\n",
    "    pop_variance = tf.Variable(tf.ones([num_units]), trainable=False)\n",
    "\n",
    "    epsilon = 1e-3\n",
    "    \n",
    "    def batch_norm_training():\n",
    "        batch_mean, batch_variance = tf.nn.moments(layer, [0])\n",
    "\n",
    "        decay = 0.99\n",
    "        train_mean = tf.assign(pop_mean, pop_mean * decay + batch_mean * (1 - decay))\n",
    "        train_variance = tf.assign(pop_variance, pop_variance * decay + batch_variance * (1 - decay))\n",
    "\n",
    "        with tf.control_dependencies([train_mean, train_variance]):\n",
    "            return tf.nn.batch_normalization(layer, batch_mean, batch_variance, beta, gamma, epsilon)\n",
    " \n",
    "    def batch_norm_inference():\n",
    "        return tf.nn.batch_normalization(layer, pop_mean, pop_variance, beta, gamma, epsilon)\n",
    "\n",
    "    batch_normalized_output = tf.cond(is_training, batch_norm_training, batch_norm_inference)\n",
    "    return tf.nn.relu(batch_normalized_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The changes we made to `conv_layer` are _almost_ exactly the same as the ones we made to `fully_connected`. However, there is an important difference. Convolutional layers have multiple feature maps, and each feature map uses shared weights. So we need to make sure we calculate our batch and population statistics **per feature map** instead of per node in the layer.\n",
    "\n",
    "To accomplish this, we do **the same things** that we did in `fully_connected`, with two exceptions:\n",
    "1. The sizes of `gamma`, `beta`, `pop_mean` and `pop_variance` are set to the number of feature maps (output channels) instead of the number of output nodes.\n",
    "2. We change the parameters we pass to `tf.nn.moments` to make sure it calculates the mean and variance for the correct dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_layer(prev_layer, layer_depth, is_training):\n",
    "    \"\"\"\n",
    "    Create a convolutional layer with the given layer as input.\n",
    "    \n",
    "    :param prev_layer: Tensor\n",
    "        The Tensor that acts as input into this layer\n",
    "    :param layer_depth: int\n",
    "        We'll set the strides and number of feature maps based on the layer's depth in the network.\n",
    "        This is *not* a good way to make a CNN, but it helps us create this example with very little code.\n",
    "    :param is_training: bool or Tensor\n",
    "        Indicates whether or not the network is currently training, which tells the batch normalization\n",
    "        layer whether or not it should update or use its population statistics.\n",
    "    :returns Tensor\n",
    "        A new convolutional layer\n",
    "    \"\"\"\n",
    "    strides = 2 if layer_depth % 3 == 0 else 1\n",
    "    \n",
    "    in_channels = prev_layer.get_shape().as_list()[3]\n",
    "    out_channels = layer_depth*4\n",
    "    \n",
    "    weights = tf.Variable(\n",
    "        tf.truncated_normal([3, 3, in_channels, out_channels], stddev=0.05))\n",
    "    \n",
    "    layer = tf.nn.conv2d(prev_layer, weights, strides=[1,strides, strides, 1], padding='SAME')\n",
    "\n",
    "    gamma = tf.Variable(tf.ones([out_channels]))\n",
    "    beta = tf.Variable(tf.zeros([out_channels]))\n",
    "\n",
    "    pop_mean = tf.Variable(tf.zeros([out_channels]), trainable=False)\n",
    "    pop_variance = tf.Variable(tf.ones([out_channels]), trainable=False)\n",
    "\n",
    "    epsilon = 1e-3\n",
    "    \n",
    "    def batch_norm_training():\n",
    "        # Important to use the correct dimensions here to ensure the mean and variance are calculated \n",
    "        # per feature map instead of for the entire layer\n",
    "        batch_mean, batch_variance = tf.nn.moments(layer, [0,1,2], keep_dims=False)\n",
    "\n",
    "        decay = 0.99\n",
    "        train_mean = tf.assign(pop_mean, pop_mean * decay + batch_mean * (1 - decay))\n",
    "        train_variance = tf.assign(pop_variance, pop_variance * decay + batch_variance * (1 - decay))\n",
    "\n",
    "        with tf.control_dependencies([train_mean, train_variance]):\n",
    "            return tf.nn.batch_normalization(layer, batch_mean, batch_variance, beta, gamma, epsilon)\n",
    " \n",
    "    def batch_norm_inference():\n",
    "        return tf.nn.batch_normalization(layer, pop_mean, pop_variance, beta, gamma, epsilon)\n",
    "\n",
    "    batch_normalized_output = tf.cond(is_training, batch_norm_training, batch_norm_inference)\n",
    "    return tf.nn.relu(batch_normalized_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To modify `train`, we did the following:\n",
    "1. Added `is_training`, a placeholder to store a boolean value indicating whether or not the network is training.\n",
    "2. Each time we call `run` on the session, we added to `feed_dict` the appropriate value for `is_training`.\n",
    "3. We did **not** need to add the `with tf.control_dependencies...` statement that we added in the network that used `tf.layers.batch_normalization` because we handled updating the population statistics ourselves in `conv_layer` and `fully_connected`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch:  0: Validation loss: 0.68918, Validation accuracy: 0.09860\n",
      "Batch: 25: Training loss: 0.53546, Training accuracy: 0.07812\n",
      "Batch: 50: Training loss: 0.41245, Training accuracy: 0.09375\n",
      "Batch: 75: Training loss: 0.36121, Training accuracy: 0.07812\n",
      "Batch: 100: Validation loss: 0.33833, Validation accuracy: 0.09900\n",
      "Batch: 125: Training loss: 0.33615, Training accuracy: 0.06250\n",
      "Batch: 150: Training loss: 0.36635, Training accuracy: 0.04688\n",
      "Batch: 175: Training loss: 0.34496, Training accuracy: 0.07812\n",
      "Batch: 200: Validation loss: 0.34961, Validation accuracy: 0.09900\n",
      "Batch: 225: Training loss: 0.34695, Training accuracy: 0.17188\n",
      "Batch: 250: Training loss: 0.50702, Training accuracy: 0.04688\n",
      "Batch: 275: Training loss: 0.34169, Training accuracy: 0.20312\n",
      "Batch: 300: Validation loss: 0.36515, Validation accuracy: 0.16300\n",
      "Batch: 325: Training loss: 0.32103, Training accuracy: 0.29688\n",
      "Batch: 350: Training loss: 0.31099, Training accuracy: 0.31250\n",
      "Batch: 375: Training loss: 0.27836, Training accuracy: 0.43750\n",
      "Batch: 400: Validation loss: 0.32132, Validation accuracy: 0.39140\n",
      "Batch: 425: Training loss: 0.29945, Training accuracy: 0.45312\n",
      "Batch: 450: Training loss: 0.28822, Training accuracy: 0.43750\n",
      "Batch: 475: Training loss: 0.16682, Training accuracy: 0.70312\n",
      "Batch: 500: Validation loss: 0.14634, Validation accuracy: 0.75320\n",
      "Batch: 525: Training loss: 0.34097, Training accuracy: 0.45312\n",
      "Batch: 550: Training loss: 0.15460, Training accuracy: 0.78125\n",
      "Batch: 575: Training loss: 0.02774, Training accuracy: 0.95312\n",
      "Batch: 600: Validation loss: 0.03633, Validation accuracy: 0.94900\n",
      "Batch: 625: Training loss: 0.07376, Training accuracy: 0.90625\n",
      "Batch: 650: Training loss: 0.05857, Training accuracy: 0.93750\n",
      "Batch: 675: Training loss: 0.06562, Training accuracy: 0.92188\n",
      "Batch: 700: Validation loss: 0.05356, Validation accuracy: 0.92300\n",
      "Batch: 725: Training loss: 0.01616, Training accuracy: 0.98438\n",
      "Batch: 750: Training loss: 0.02600, Training accuracy: 0.96875\n",
      "Batch: 775: Training loss: 0.01844, Training accuracy: 0.96875\n",
      "Final validation accuracy: 0.95780\n",
      "Final test accuracy: 0.96050\n",
      "Accuracy on 100 samples: 0.98\n"
     ]
    }
   ],
   "source": [
    "def train(num_batches, batch_size, learning_rate):\n",
    "    # Build placeholders for the input samples and labels \n",
    "    inputs = tf.placeholder(tf.float32, [None, 28, 28, 1])\n",
    "    labels = tf.placeholder(tf.float32, [None, 10])\n",
    "\n",
    "    # Add placeholder to indicate whether or not we're training the model\n",
    "    is_training = tf.placeholder(tf.bool)\n",
    "\n",
    "    # Feed the inputs into a series of 20 convolutional layers \n",
    "    layer = inputs\n",
    "    for layer_i in range(1, 20):\n",
    "        layer = conv_layer(layer, layer_i, is_training)\n",
    "\n",
    "    # Flatten the output from the convolutional layers \n",
    "    orig_shape = layer.get_shape().as_list()\n",
    "    layer = tf.reshape(layer, shape=[-1, orig_shape[1] * orig_shape[2] * orig_shape[3]])\n",
    "\n",
    "    # Add one fully connected layer\n",
    "    layer = fully_connected(layer, 100, is_training)\n",
    "\n",
    "    # Create the output layer with 1 node for each \n",
    "    logits = tf.layers.dense(layer, 10)\n",
    "    \n",
    "    # Define loss and training operations\n",
    "    model_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))\n",
    "    train_opt = tf.train.AdamOptimizer(learning_rate).minimize(model_loss)\n",
    "    \n",
    "    # Create operations to test accuracy\n",
    "    correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(labels,1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    \n",
    "    # Train and test the network\n",
    "    with tf.Session() as sess:\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        for batch_i in range(num_batches):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(batch_size)\n",
    "\n",
    "            # train this batch\n",
    "            sess.run(train_opt, {inputs: batch_xs, labels: batch_ys, is_training: True})\n",
    "            \n",
    "            # Periodically check the validation or training loss and accuracy\n",
    "            if batch_i % 100 == 0:\n",
    "                loss, acc = sess.run([model_loss, accuracy], {inputs: mnist.validation.images,\n",
    "                                                              labels: mnist.validation.labels,\n",
    "                                                              is_training: False})\n",
    "                print('Batch: {:>2}: Validation loss: {:>3.5f}, Validation accuracy: {:>3.5f}'.format(batch_i, loss, acc))\n",
    "            elif batch_i % 25 == 0:\n",
    "                loss, acc = sess.run([model_loss, accuracy], {inputs: batch_xs, labels: batch_ys, is_training: False})\n",
    "                print('Batch: {:>2}: Training loss: {:>3.5f}, Training accuracy: {:>3.5f}'.format(batch_i, loss, acc))\n",
    "\n",
    "        # At the end, score the final accuracy for both the validation and test sets\n",
    "        acc = sess.run(accuracy, {inputs: mnist.validation.images,\n",
    "                                  labels: mnist.validation.labels, \n",
    "                                  is_training: False})\n",
    "        print('Final validation accuracy: {:>3.5f}'.format(acc))\n",
    "        acc = sess.run(accuracy, {inputs: mnist.test.images,\n",
    "                                  labels: mnist.test.labels,\n",
    "                                  is_training: False})\n",
    "        print('Final test accuracy: {:>3.5f}'.format(acc))\n",
    "        \n",
    "        # Score the first 100 test images individually, just to make sure batch normalization really worked\n",
    "        correct = 0\n",
    "        for i in range(100):\n",
    "            correct += sess.run(accuracy,feed_dict={inputs: [mnist.test.images[i]],\n",
    "                                                    labels: [mnist.test.labels[i]],\n",
    "                                                    is_training: False})\n",
    "\n",
    "        print(\"Accuracy on 100 samples:\", correct/100)\n",
    "\n",
    "\n",
    "num_batches = 800\n",
    "batch_size = 64\n",
    "learning_rate = 0.002\n",
    "\n",
    "tf.reset_default_graph()\n",
    "with tf.Graph().as_default():\n",
    "    train(num_batches, batch_size, learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, the model with batch normalization quickly reaches a high accuracy. But in our run, notice that it doesn't seem to learn anything for the first 250 batches, then the accuracy starts to climb. That just goes to show - even with batch normalization, it's important to give your network a bit of time to learn before you decide it isn't working."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}