pytorch/nn_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "nn package\n",
    "==========\n",
    "\n",
    "我们重新设计了 nn package, 以便与 autograd 完全集成.\n",
    "让我们来回顾一下这些变化.\n",
    "\n",
    "**用 autograd 替换 containers:**\n",
    "\n",
    "    你不再需要使用像 ``ConcatTable`` 这样的 Containers, 或者像\n",
    "    ``CAddTable`` 这样的模块, 或者使用 nngraph 并且 debug. 我们将无缝地使用\n",
    "    autograd 来定义我们的神经网络. 例如,\n",
    "\n",
    "    * ``output = nn.CAddTable():forward({input1, input2})`` 简化为\n",
    "      ``output = input1 + input2``\n",
    "    * ``output = nn.MulConstant(0.5):forward(input)`` 简化为\n",
    "      ``output = input * 0.5``\n",
    "\n",
    "**中间状态不再存放在上述提到的那些模块中, 而是存放在计算图中:**\n",
    "\n",
    "    因为这个原因, 所以使用循环网络变得更加简单. 如果你想创建一个循环网络,\n",
    "    只需多次使用相同的 Linear 层, 而不必考虑共享权重.\n",
    "\n",
    "    .. figure:: /_static/img/torch-nn-vs-pytorch-nn.png\n",
    "       :alt: torch-nn-vs-pytorch-nn\n",
    "\n",
    "       torch-nn-vs-pytorch-nn\n",
    "\n",
    "**Simplified debugging:**\n",
    "\n",
    "    使用Python的pdb调试器进行调试是直观的, 调试器和堆栈跟踪在发生错误的地方停止.\n",
    "    What you see is what you get(所见即所得, 译者注:应该是说可视化吧).\n",
    "\n",
    "Example 1: ConvNet\n",
    "------------------\n",
    "\n",
    "让我们来创建一个小的 ConvNet.\n",
    "\n",
    "你所有的网络都来自 ``nn.Module`` 基类:\n",
    "\n",
    "-  在构造函数中, 声明你想要使用的所有层.\n",
    "-  在 forward 函数中, 你可以定义模型从输入到输出将如何运行\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class MNISTConvNet(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        # 这是你实例化所有模块的地方\n",
    "        # 你可以稍后使用你在此给出的相同名称访问它们\n",
    "        super(MNISTConvNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 10, 5)\n",
    "        self.pool1 = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(10, 20, 5)\n",
    "        self.pool2 = nn.MaxPool2d(2, 2)\n",
    "        #\n",
    "        self.fc1 = nn.Linear(320, 50)\n",
    "        self.fc2 = nn.Linear(50, 10)\n",
    "\n",
    "    # 这是 forward 函数, 它定义了只接受一个输入的网络结构,\n",
    "    # 如果你愿意, 可以随意定义支持使用更多输入的网络结构.\n",
    "    def forward(self, input):\n",
    "        x = self.pool1(F.relu(self.conv1(input)))\n",
    "        x = self.pool2(F.relu(self.conv2(x)))\n",
    "\n",
    "        # 在你的创建模型的过程中, 你可以疯狂地使用任意的python代码创建你的模型结构,\n",
    "        # 这些操作都是完全合法的, 并且会被autograd正确处理:\n",
    "        # if x.gt(0) > x.numel() / 2:\n",
    "        #      ...\n",
    "        #\n",
    "        # 你甚至可以做一个循环来重复使用相同的模块, 模块内部的模块不再\n",
    "        # 处于临时状态, 所以你可以在 forward 时多次使用它们.\n",
    "        # while x.norm(2) < 10:\n",
    "        #    x = self.conv1(x)\n",
    "\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们来使用定义好的 ConvNet.\n",
    "你应该先创建一个类的实例.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MNISTConvNet(\n",
      "  (conv1): Conv2d(1, 10, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (fc1): Linear(in_features=320, out_features=50, bias=True)\n",
      "  (fc2): Linear(in_features=50, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "net = MNISTConvNet()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><h4>Note</h4><p>``torch.nn`` 只支持 mini-batches , 整个 ``torch.nn`` package\n",
    "    只支持输入 mini-batch 格式的样本, 而不支持输入单个样本.\n",
    "\n",
    "    例如, ``nn.Conv2d`` 将采用 ``nSamples x nChannels x Height x Width`` \n",
    "    的 4D Tensor.\n",
    "\n",
    "    如果你有一个单个的样本, 只需使用 ``input.unsqueeze(0)`` 添加一个\n",
    "    虚假的 batch 维度.</p></div>\n",
    "\n",
    "创建一个包含随机数据的单个样本的 mini-batch,\n",
    "并将该样本传入到 ConvNet .\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "input = Variable(torch.randn(1, 1, 28, 28))\n",
    "out = net(input)\n",
    "print(out.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0593,  0.0660,  0.0110,  0.1343,  0.0000,  0.0633,  0.0833,\n",
       "          0.1700,  0.0000,  0.0543]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一个虚拟目标标签, 并使用损失函数来计算 error.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.2339)\n"
     ]
    }
   ],
   "source": [
    "target = Variable(torch.LongTensor([3]))\n",
    "loss_fn = nn.CrossEntropyLoss()  # LogSoftmax + ClassNLL Loss\n",
    "err = loss_fn(out, target)\n",
    "err.backward()\n",
    "\n",
    "print(err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_fn?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ConvNet 的 ``out`` 是一个 ``Variable``. 我们使用它来计算损失,\n",
    "计算结果 ``err`` 也是一个 ``Variable``.\n",
    "调用 ``err`` 的 ``.backward`` 方法将会通过 ConvNet 将梯度传播到它的权重.\n",
    "\n",
    "让我们来访问单个层的权重和梯度:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10, 1, 5, 5])\n"
     ]
    }
   ],
   "source": [
    "print(net.conv1.weight.grad.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.7718)\n",
      "tensor(0.6119)\n"
     ]
    }
   ],
   "source": [
    "print(net.conv1.weight.data.norm())  # norm of the weight\n",
    "print(net.conv1.weight.grad.data.norm())  # norm of the gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forward and Backward Function Hooks\n",
    "-----------------------------------\n",
    "\n",
    "我们已经检查了权重和梯度. 但是如何检查\n",
    "/ 修改一个层的输出和 grad\\_output?\n",
    "\n",
    "我们为此引出了 **hooks**.\n",
    "\n",
    "你可以在一个 ``Module`` 或一个 ``Variable`` 上注册一个函数.\n",
    "hook 可以是 forward hook 也可以是一个 backward hook.\n",
    "当 forward 被执行后 forward hook 将会被执行.\n",
    "backward hook 将在执行 backward 阶段被执行.\n",
    "让我们来看一个例子.\n",
    "\n",
    "我们在 conv2 注册一个 forward hook 来打印一些信息\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inside Conv2d forward\n",
      "\n",
      "input:  <class 'tuple'>\n",
      "input[0]:  <class 'torch.Tensor'>\n",
      "output:  <class 'torch.Tensor'>\n",
      "\n",
      "input size: torch.Size([1, 10, 12, 12])\n",
      "output size: torch.Size([1, 20, 8, 8])\n",
      "output norm: tensor(15.5888)\n"
     ]
    }
   ],
   "source": [
    "def printnorm(self, input, output):\n",
    "    # input是将输入打包成的 tuple 的input\n",
    "    # 输出是一个 Variable. output.data 是我们感兴趣的 Tensor\n",
    "    print('Inside ' + self.__class__.__name__ + ' forward')\n",
    "    print('input: ', type(input))\n",
    "    print('input[0]: ', type(input[0]))\n",
    "    print('output: ', type(output))\n",
    "    print('')\n",
    "    print('input size:', input[0].size())\n",
    "    print('output size:', output.data.size())\n",
    "    print('output norm:', output.data.norm())\n",
    "\n",
    "\n",
    "net.conv2.register_forward_hook(printnorm)\n",
    "\n",
    "out = net(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们在 conv2 注册一个 backward hook 来打印一些信息\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inside Conv2d forward\n",
      "\n",
      "input:  <class 'tuple'>\n",
      "input[0]:  <class 'torch.Tensor'>\n",
      "output:  <class 'torch.Tensor'>\n",
      "\n",
      "input size: torch.Size([1, 10, 12, 12])\n",
      "output size: torch.Size([1, 20, 8, 8])\n",
      "output norm: tensor(15.5888)\n",
      "Inside Conv2d backward\n",
      "Inside class:Conv2d\n",
      "\n",
      "grad_input:  <class 'tuple'>\n",
      "grad_input[0]:  <class 'torch.Tensor'>\n",
      "grad_output:  <class 'tuple'>\n",
      "grad_output[0]:  <class 'torch.Tensor'>\n",
      "\n",
      "grad_input size: torch.Size([1, 10, 12, 12])\n",
      "grad_output size: torch.Size([1, 20, 8, 8])\n",
      "grad_input norm: tensor(0.1159)\n",
      "Inside Conv2d backward\n",
      "Inside class:Conv2d\n",
      "\n",
      "grad_input:  <class 'tuple'>\n",
      "grad_input[0]:  <class 'torch.Tensor'>\n",
      "grad_output:  <class 'tuple'>\n",
      "grad_output[0]:  <class 'torch.Tensor'>\n",
      "\n",
      "grad_input size: torch.Size([1, 10, 12, 12])\n",
      "grad_output size: torch.Size([1, 20, 8, 8])\n",
      "grad_input norm: tensor(0.1159)\n"
     ]
    }
   ],
   "source": [
    "def printgradnorm(self, grad_input, grad_output):\n",
    "    print('Inside ' + self.__class__.__name__ + ' backward')\n",
    "    print('Inside class:' + self.__class__.__name__)\n",
    "    print('')\n",
    "    print('grad_input: ', type(grad_input))\n",
    "    print('grad_input[0]: ', type(grad_input[0]))\n",
    "    print('grad_output: ', type(grad_output))\n",
    "    print('grad_output[0]: ', type(grad_output[0]))\n",
    "    print('')\n",
    "    print('grad_input size:', grad_input[0].size())\n",
    "    print('grad_output size:', grad_output[0].size())\n",
    "    print('grad_input norm:', grad_input[0].data.norm())\n",
    "\n",
    "\n",
    "net.conv2.register_backward_hook(printgradnorm)\n",
    "\n",
    "out = net(input)\n",
    "err = loss_fn(out, target)\n",
    "err.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.2339)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个完整的可以运行的 MNIST 例子在此链接中\n",
    "https://github.com/pytorch/examples/tree/master/mnist\n",
    "\n",
    "Example 2: Recurrent Net\n",
    "------------------------\n",
    "\n",
    "接下来, 让我们看一下用 PyTorch 创建 recurrent nets.\n",
    "\n",
    "由于网络的状态是保存在图中, 而不是在 layer 中, 所以您可以简单地\n",
    "创建一个 nn.Linear 并重复使用它.\n",
    "\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "cat(seq, dim=0, out=None) -> Tensor\n",
    "\n",
    "Concatenates the given sequence of :attr:`seq` tensors in the given dimension.\n",
    "All tensors must either have the same shape (except in the concatenating\n",
    "dimension) or be empty.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNN(nn.Module):\n",
    "\n",
    "    # 你也可以在你模型的构造函数中传入参数\n",
    "    def __init__(self, data_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "\n",
    "        self.hidden_size = hidden_size\n",
    "        input_size = data_size + hidden_size\n",
    "\n",
    "        self.i2h = nn.Linear(input_size, hidden_size)\n",
    "        self.h2o = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "    def forward(self, data, last_hidden):\n",
    "        input = torch.cat((data, last_hidden), 1)\n",
    "        hidden = self.i2h(input)\n",
    "        output = self.h2o(hidden)\n",
    "        return hidden, output\n",
    "\n",
    "\n",
    "rnn = RNN(50, 20, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "更完整的使用 LSTMs 和 Penn Tree-bank 的语言模型位于\n",
    "`here <https://github.com/pytorch/examples/tree/master/word\\_language\\_model>`_\n",
    "\n",
    "PyTorch 默认已经为 ConvNets 和 Recurrent Nets 提供了无缝的 CuDNN 集成.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_fn = nn.MSELoss()\n",
    "\n",
    "batch_size = 10\n",
    "TIMESTEPS = 100\n",
    "\n",
    "# 创建一些假数据\n",
    "batch = Variable(torch.randn(batch_size, 50))\n",
    "hidden = Variable(torch.zeros(batch_size, 20))\n",
    "target = Variable(torch.zeros(batch_size, 10))\n",
    "\n",
    "loss = 0\n",
    "for t in range(TIMESTEPS):\n",
    "    # 是的! 你可以多次使用同一个网络,\n",
    "    # 将损失相加, 并且调用 call backward!\n",
    "    hidden, output = rnn(batch, hidden)\n",
    "    loss += loss_fn(output, target)\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(11.6475)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}