docs/notebooks/Neural_Network_and_Data_Loading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "18AF5Ab4p6VL"
   },
   "source": [
    "# Training a Simple Neural Network, with PyTorch Data Loading\n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.sandbox.google.com/github/google/jax/blob/master/docs/notebooks/Neural_Network_and_Data_Loading.ipynb)\n",
    "\n",
    "**Copyright 2018 Google LLC.**\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "https://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B_XlLLpcWjkA"
   },
   "source": [
    "![JAX](https://raw.githubusercontent.com/google/jax/master/images/jax_logo_250px.png)\n",
    "\n",
    "Let's combine everything we showed in the [quickstart notebook](https://colab.research.google.com/github/google/jax/blob/master/docs/notebooks/quickstart.ipynb) to train a simple neural network. We will first specify and train a simple MLP on MNIST using JAX for the computation. We will use PyTorch's data loading API to load images and labels (because it's pretty great, and the world doesn't need yet another data loading library).\n",
    "\n",
    "Of course, you can use JAX with any API that is compatible with NumPy to make specifying the model a bit more plug-and-play. Here, just for explanatory purposes, we won't use any neural network libraries or special APIs for builidng our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "OksHydJDtbbI"
   },
   "outputs": [],
   "source": [
    "import jax.numpy as jnp\n",
    "from jax import grad, jit, vmap\n",
    "from jax import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MTVcKi-ZYB3R"
   },
   "source": [
    "## Hyperparameters\n",
    "Let's get a few bookkeeping items out of the way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "-fmWA06xYE7d"
   },
   "outputs": [],
   "source": [
    "# A helper function to randomly initialize weights and biases\n",
    "# for a dense neural network layer\n",
    "def random_layer_params(m, n, key, scale=1e-2):\n",
    "  w_key, b_key = random.split(key)\n",
    "  return scale * random.normal(w_key, (n, m)), scale * random.normal(b_key, (n,))\n",
    "\n",
    "# Initialize all layers for a fully-connected neural network with sizes \"sizes\"\n",
    "def init_network_params(sizes, key):\n",
    "  keys = random.split(key, len(sizes))\n",
    "  return [random_layer_params(m, n, k) for m, n, k in zip(sizes[:-1], sizes[1:], keys)]\n",
    "\n",
    "layer_sizes = [784, 512, 512, 10]\n",
    "param_scale = 0.1\n",
    "step_size = 0.01\n",
    "num_epochs = 8\n",
    "batch_size = 128\n",
    "n_targets = 10\n",
    "params = init_network_params(layer_sizes, random.PRNGKey(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BtoNk_yxWtIw"
   },
   "source": [
    "## Auto-batching predictions\n",
    "\n",
    "Let us first define our prediction function. Note that we're defining this for a _single_ image example. We're going to use JAX's `vmap` function to automatically handle mini-batches, with no performance penalty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "7APc6tD7TiuZ"
   },
   "outputs": [],
   "source": [
    "from jax.scipy.special import logsumexp\n",
    "\n",
    "def relu(x):\n",
    "  return jnp.maximum(0, x)\n",
    "\n",
    "def predict(params, image):\n",
    "  # per-example predictions\n",
    "  activations = image\n",
    "  for w, b in params[:-1]:\n",
    "    outputs = jnp.dot(w, activations) + b\n",
    "    activations = relu(outputs)\n",
    "  \n",
    "  final_w, final_b = params[-1]\n",
    "  logits = jnp.dot(final_w, activations) + final_b\n",
    "  return logits - logsumexp(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dRW_TvCTWgaP"
   },
   "source": [
    "Let's check that our prediction function only works on single images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "4sW2A5mnXHc5",
    "outputId": "9d3b29e8-fab3-4ecb-9f63-bc8c092f9006"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10,)\n"
     ]
    }
   ],
   "source": [
    "# This works on single examples\n",
    "random_flattened_image = random.normal(random.PRNGKey(1), (28 * 28,))\n",
    "preds = predict(params, random_flattened_image)\n",
    "print(preds.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "PpyQxuedXfhp",
    "outputId": "d5d20211-b6da-44e9-f71e-946f2a9d0fc4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Invalid shapes!\n"
     ]
    }
   ],
   "source": [
    "# Doesn't work with a batch\n",
    "random_flattened_images = random.normal(random.PRNGKey(1), (10, 28 * 28))\n",
    "try:\n",
    "  preds = predict(params, random_flattened_images)\n",
    "except TypeError:\n",
    "  print('Invalid shapes!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "oJOOncKMXbwK",
    "outputId": "31285fab-7667-4871-fcba-28e86adc3fc6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 10)\n"
     ]
    }
   ],
   "source": [
    "# Let's upgrade it to handle batches using `vmap`\n",
    "\n",
    "# Make a batched version of the `predict` function\n",
    "batched_predict = vmap(predict, in_axes=(None, 0))\n",
    "\n",
    "# `batched_predict` has the same call signature as `predict`\n",
    "batched_preds = batched_predict(params, random_flattened_images)\n",
    "print(batched_preds.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "elsG6nX03BvW"
   },
   "source": [
    "At this point, we have all the ingredients we need to define our neural network and train it. We've built an auto-batched version of `predict`, which we should be able to use in a loss function. We should be able to use `grad` to take the derivative of the loss with respect to the neural network parameters. Last, we should be able to use `jit` to speed up everything."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NwDuFqc9X7ER"
   },
   "source": [
    "## Utility and loss functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "6lTI6I4lWdh5"
   },
   "outputs": [],
   "source": [
    "def one_hot(x, k, dtype=jnp.float32):\n",
    "  \"\"\"Create a one-hot encoding of x of size k.\"\"\"\n",
    "  return jnp.array(x[:, None] == jnp.arange(k), dtype)\n",
    "  \n",
    "def accuracy(params, images, targets):\n",
    "  target_class = jnp.argmax(targets, axis=1)\n",
    "  predicted_class = jnp.argmax(batched_predict(params, images), axis=1)\n",
    "  return jnp.mean(predicted_class == target_class)\n",
    "\n",
    "def loss(params, images, targets):\n",
    "  preds = batched_predict(params, images)\n",
    "  return -jnp.mean(preds * targets)\n",
    "\n",
    "@jit\n",
    "def update(params, x, y):\n",
    "  grads = grad(loss)(params, x, y)\n",
    "  return [(w - step_size * dw, b - step_size * db)\n",
    "          for (w, b), (dw, db) in zip(params, grads)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "umJJGZCC2oKl"
   },
   "source": [
    "## Data Loading with PyTorch\n",
    "\n",
    "JAX is laser-focused on program transformations and accelerator-backed NumPy, so we don't include data loading or munging in the JAX library. There are already a lot of great data loaders out there, so let's just use them instead of reinventing anything. We'll grab PyTorch's data loader, and make a tiny shim to make it work with NumPy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "gEvWt8_u2pqG",
    "outputId": "2c83a679-9ce5-4c67-bccb-9ea835a8eaf6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: torch in /opt/anaconda3/lib/python3.7/site-packages (1.4.0)\n",
      "Requirement already satisfied: torchvision in /opt/anaconda3/lib/python3.7/site-packages (0.5.0)\n",
      "Requirement already satisfied: numpy in /opt/anaconda3/lib/python3.7/site-packages (from torchvision) (1.17.2)\n",
      "Requirement already satisfied: six in /opt/anaconda3/lib/python3.7/site-packages (from torchvision) (1.12.0)\n",
      "Requirement already satisfied: pillow>=4.1.1 in /opt/anaconda3/lib/python3.7/site-packages (from torchvision) (6.2.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install torch torchvision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "cellView": "both",
    "id": "94PjXZ8y3dVF"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from torch.utils import data\n",
    "from torchvision.datasets import MNIST\n",
    "\n",
    "def numpy_collate(batch):\n",
    "  if isinstance(batch[0], np.ndarray):\n",
    "    return np.stack(batch)\n",
    "  elif isinstance(batch[0], (tuple,list)):\n",
    "    transposed = zip(*batch)\n",
    "    return [numpy_collate(samples) for samples in transposed]\n",
    "  else:\n",
    "    return np.array(batch)\n",
    "\n",
    "class NumpyLoader(data.DataLoader):\n",
    "  def __init__(self, dataset, batch_size=1,\n",
    "                shuffle=False, sampler=None,\n",
    "                batch_sampler=None, num_workers=0,\n",
    "                pin_memory=False, drop_last=False,\n",
    "                timeout=0, worker_init_fn=None):\n",
    "    super(self.__class__, self).__init__(dataset,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=shuffle,\n",
    "        sampler=sampler,\n",
    "        batch_sampler=batch_sampler,\n",
    "        num_workers=num_workers,\n",
    "        collate_fn=numpy_collate,\n",
    "        pin_memory=pin_memory,\n",
    "        drop_last=drop_last,\n",
    "        timeout=timeout,\n",
    "        worker_init_fn=worker_init_fn)\n",
    "\n",
    "class FlattenAndCast(object):\n",
    "  def __call__(self, pic):\n",
    "    return np.ravel(np.array(pic, dtype=jnp.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "l314jsfP4TN4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /tmp/mnist/MNIST/raw/train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75806ce83ace4f69b81bbc4251c5573f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/mnist/MNIST/raw/train-images-idx3-ubyte.gz to /tmp/mnist/MNIST/raw\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw/train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "274ed4ab05f34f70b7a5bb6cf427ffd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/mnist/MNIST/raw/train-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to /tmp/mnist/MNIST/raw/t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d38fa4eabf3c4d4494eb59e078ac94e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/mnist/MNIST/raw/t10k-images-idx3-ubyte.gz to /tmp/mnist/MNIST/raw\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "523ac9565c5f4509a1ee8fdbb1e6d66d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/mnist/MNIST/raw/t10k-labels-idx1-ubyte.gz to /tmp/mnist/MNIST/raw\n",
      "Processing...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "# Define our dataset, using torch datasets\n",
    "mnist_dataset = MNIST('/tmp/mnist/', download=True, transform=FlattenAndCast())\n",
    "training_generator = NumpyLoader(mnist_dataset, batch_size=batch_size, num_workers=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "FTNo4beUvb6t",
    "outputId": "65a9087c-c326-49e5-cbfc-e0839212fa31"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/torchvision/datasets/mnist.py:55: UserWarning: train_data has been renamed data\n",
      "  warnings.warn(\"train_data has been renamed data\")\n",
      "/opt/anaconda3/lib/python3.7/site-packages/torchvision/datasets/mnist.py:45: UserWarning: train_labels has been renamed targets\n",
      "  warnings.warn(\"train_labels has been renamed targets\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/torchvision/datasets/mnist.py:60: UserWarning: test_data has been renamed data\n",
      "  warnings.warn(\"test_data has been renamed data\")\n",
      "/opt/anaconda3/lib/python3.7/site-packages/torchvision/datasets/mnist.py:50: UserWarning: test_labels has been renamed targets\n",
      "  warnings.warn(\"test_labels has been renamed targets\")\n"
     ]
    }
   ],
   "source": [
    "# Get the full train dataset (for checking accuracy while training)\n",
    "train_images = np.array(mnist_dataset.train_data).reshape(len(mnist_dataset.train_data), -1)\n",
    "train_labels = one_hot(np.array(mnist_dataset.train_labels), n_targets)\n",
    "\n",
    "# Get full test dataset\n",
    "mnist_dataset_test = MNIST('/tmp/mnist/', download=True, train=False)\n",
    "test_images = jnp.array(mnist_dataset_test.test_data.numpy().reshape(len(mnist_dataset_test.test_data), -1), dtype=jnp.float32)\n",
    "test_labels = one_hot(np.array(mnist_dataset_test.test_labels), n_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xxPd6Qw3Z98v"
   },
   "source": [
    "## Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "X2DnZo3iYj18",
    "outputId": "0eba3ca2-24a1-4cba-aaf4-3ac61d0c650e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 in 55.15 sec\n",
      "Training set accuracy 0.9157500267028809\n",
      "Test set accuracy 0.9195000529289246\n",
      "Epoch 1 in 42.26 sec\n",
      "Training set accuracy 0.9372166991233826\n",
      "Test set accuracy 0.9384000301361084\n",
      "Epoch 2 in 44.37 sec\n",
      "Training set accuracy 0.9491666555404663\n",
      "Test set accuracy 0.9469000697135925\n",
      "Epoch 3 in 41.75 sec\n",
      "Training set accuracy 0.9568166732788086\n",
      "Test set accuracy 0.9534000158309937\n",
      "Epoch 4 in 41.16 sec\n",
      "Training set accuracy 0.9631333351135254\n",
      "Test set accuracy 0.9577000737190247\n",
      "Epoch 5 in 38.89 sec\n",
      "Training set accuracy 0.9675000309944153\n",
      "Test set accuracy 0.9616000652313232\n",
      "Epoch 6 in 40.68 sec\n",
      "Training set accuracy 0.9708333611488342\n",
      "Test set accuracy 0.9650000333786011\n",
      "Epoch 7 in 41.50 sec\n",
      "Training set accuracy 0.973716676235199\n",
      "Test set accuracy 0.9672000408172607\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "  start_time = time.time()\n",
    "  for x, y in training_generator:\n",
    "    y = one_hot(y, n_targets)\n",
    "    params = update(params, x, y)\n",
    "  epoch_time = time.time() - start_time\n",
    "\n",
    "  train_acc = accuracy(params, train_images, train_labels)\n",
    "  test_acc = accuracy(params, test_images, test_labels)\n",
    "  print(\"Epoch {} in {:0.2f} sec\".format(epoch, epoch_time))\n",
    "  print(\"Training set accuracy {}\".format(train_acc))\n",
    "  print(\"Test set accuracy {}\".format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xC1CMcVNYwxm"
   },
   "source": [
    "We've now used the whole of the JAX API: `grad` for derivatives, `jit` for speedups and `vmap` for auto-vectorization.\n",
    "We used NumPy to specify all of our computation, and borrowed the great data loaders from PyTorch, and ran the whole thing on the GPU."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Neural Network and Data Loading.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "jupytext": {
   "formats": "ipynb,md:myst"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}