pytorch/NLP/char_rnn_generation_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "基与字符级RNN（Char-RNN）的人名生成\n",
    "*******************************************\n",
    "**作者**: `Sean Robertson <https://github.com/spro/practical-pytorch>`_\n",
    "\n",
    "在 :doc:`上一个教程 </intermediate/char_rnn_classification_tutorial>` \n",
    "里我们使用RNN把名字分类到它所属的语言中, 这次我们改变一下来学习从语言中生成名字. \n",
    "\n",
    "::\n",
    "\n",
    "    > python sample.py Russian RUS\n",
    "    Rovakov\n",
    "    Uantov\n",
    "    Shavakov\n",
    "\n",
    "    > python sample.py German GER\n",
    "    Gerren\n",
    "    Ereng\n",
    "    Rosher\n",
    "\n",
    "    > python sample.py Spanish SPA\n",
    "    Salla\n",
    "    Parer\n",
    "    Allan\n",
    "\n",
    "    > python sample.py Chinese CHI\n",
    "    Chan\n",
    "    Hang\n",
    "    Iun\n",
    "\n",
    "我们仍然手工搭建一个包含几个线性层的小的RNN. 这次的最大的不同是输入一个类别, 每次输出一个字母, \n",
    "而不是读入所有名字的字母来预测一个类别. 循环的预测每一个字母来构成语言（也可以用文\n",
    "字或者其他更高级的结构完成）, 通常被称为“语言模型”. \n",
    "\n",
    "**推荐阅读: **\n",
    "\n",
    "假设你至少安装了PyTorch, 熟悉Python, 理解Tensors: \n",
    "\n",
    "-  http://pytorch.org/ : 安装说明\n",
    "-  :doc:`/beginner/deep_learning_60min_blitz` 获取一般的 PyTorch 入门\n",
    "-  :doc:`/beginner/pytorch_with_examples` 广泛且深入的概述\n",
    "-  :doc:`/beginner/former_torchies_tutorial` 如果曾经是 Lua Torch 的用户\n",
    "\n",
    "下面这些对了解 RNNs 和其工作原理也是很有用的: \n",
    "\n",
    "-  `The Unreasonable Effectiveness of Recurrent Neural\n",
    "   Networks <http://karpathy.github.io/2015/05/21/rnn-effectiveness/>`__\n",
    "   展示了一系列真实生活中的例子\n",
    "-  `Understanding LSTM\n",
    "   Networks <http://colah.github.io/posts/2015-08-Understanding-LSTMs/>`__\n",
    "   是一篇特别关于LSTMs的文章, 但是对于一般的RNNs也很有益的\n",
    "\n",
    "还建议上一个教程:  :doc:`/intermediate/char_rnn_classification_tutorial`\n",
    "\n",
    "\n",
    "数据准备\n",
    "==================\n",
    "\n",
    ".. Note::\n",
    "   从 `这里 <https://download.pytorch.org/tutorial/data.zip>`_\n",
    "   下载数据, 并解压到当前目录. \n",
    "\n",
    "更多的细节参考上一个教程, 总之, 数据含有一批纯文本文件:  ``data/names/[Language].txt`` \n",
    "每一行一个人名. 将行分割成数组, 并把 Unicode 转换成 ASCII 编码, 最后放进一个字典里 ``{language: [names ...]}``. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# categories: 0 []\n",
      "O'Neal\n"
     ]
    }
   ],
   "source": [
    "from __future__ import unicode_literals, print_function, division\n",
    "from io import open\n",
    "import glob\n",
    "import unicodedata\n",
    "import string\n",
    "\n",
    "all_letters = string.ascii_letters + \" .,;'-\"\n",
    "n_letters = len(all_letters) + 1 # 添加 EOS 标记\n",
    "\n",
    "def findFiles(path): return glob.glob(path)\n",
    "\n",
    "# 将 Unicode 字符串转换为纯 ASCII 编码, 感谢 http://stackoverflow.com/a/518232/2809427\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "        and c in all_letters\n",
    "    )\n",
    "\n",
    "# 读取文件并分割成行\n",
    "def readLines(filename):\n",
    "    lines = open(filename, encoding='utf-8').read().strip().split('\\n')\n",
    "    return [unicodeToAscii(line) for line in lines]\n",
    "\n",
    "# 构建映射字典 category_lines , 每个类别是由很多个行组成的list\n",
    "category_lines = {}\n",
    "all_categories = []\n",
    "for filename in findFiles('data/names/*.txt'):\n",
    "    category = filename.split('/')[-1].split('.')[0]\n",
    "    all_categories.append(category)\n",
    "    lines = readLines(filename)\n",
    "    category_lines[category] = lines\n",
    "\n",
    "n_categories = len(all_categories)\n",
    "\n",
    "print('# categories:', n_categories, all_categories)\n",
    "print(unicodeToAscii(\"O'Néàl\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建网络\n",
    "====================\n",
    "\n",
    " 这个网络扩展了 `上一个教程的RNN <http://pytorch.apachecn.org/cn/tutorials/intermediate/char_rnn_classification_tutorial.html>`__ , 为类别张量添加了一个额外的参数, 并和其他的参数串联在一起. 类别张量\n",
    "和字母的输入一样是 one-hot 向量. \n",
    "\n",
    "我们将输出解释成为下一个字母的概率, 采样的时候, 最有可能的输出被当做下一个输入. \n",
    "\n",
    "为了让网络更加有效工作, 我添加了第二个线性层 ``o2o`` （在合并了隐藏层和输出层的后面）. \n",
    "还有一个 Dropout 层,  `使输入的部分值以给定的概率值随机的变成 0 <https://arxiv.org/abs/1207.0580>`__ \n",
    "（这里概率取0.1）, 这样做通常是为了模糊输入以防止过拟合. 这里我们在网络的最末端使用它, 从而故意添加一些混乱和增加采样的多样化. \n",
    "\n",
    ".. figure:: https://i.imgur.com/jzVrf7f.png\n",
    "   :alt:\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "        self.i2h = nn.Linear(n_categories + input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(n_categories + input_size + hidden_size, output_size)\n",
    "        self.o2o = nn.Linear(hidden_size + output_size, output_size)\n",
    "        self.dropout = nn.Dropout(0.1)\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, category, input, hidden):\n",
    "        input_combined = torch.cat((category, input, hidden), 1)\n",
    "        hidden = self.i2h(input_combined)\n",
    "        output = self.i2o(input_combined)\n",
    "        output_combined = torch.cat((hidden, output), 1)\n",
    "        output = self.o2o(output_combined)\n",
    "        output = self.dropout(output)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "\n",
    "    def initHidden(self):\n",
    "        return Variable(torch.zeros(1, self.hidden_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练\n",
    "=========\n",
    "训练前的准备\n",
    "----------------------\n",
    "\n",
    "首先, 利用辅助函数产生随机的（category, line）对: \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# 从list中随机选取项\n",
    "def randomChoice(l):\n",
    "    return l[random.randint(0, len(l) - 1)]\n",
    "\n",
    "# 获取随机的类别和该类别中随机的行\n",
    "def randomTrainingPair():\n",
    "    category = randomChoice(all_categories)\n",
    "    line = randomChoice(category_lines[category])\n",
    "    return category, line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对每一个时间点（也就是说在训练集中词的每个字母）网络的输入是 ``(类别, 当前字母, 隐藏层状态)`` , \n",
    "输出是 ``(下一个字母, 下一个隐藏层状态)`` . 对于每一个训练集, 我们需要的是类别、输入的字母集、输出/目标字母集. \n",
    "\n",
    "因为在每一步, 我们从当前的字母预测下一个字母, 这样的字母对是在原有行中连续字母的集合, \n",
    "例如, 对于 ``\"ABCD<EOS>\"`` 将会产生 (\"A\", \"B\"), (\"B\", \"C\"),\n",
    "(\"C\", \"D\"), (\"D\", \"EOS\"). \n",
    "\n",
    ".. figure:: https://i.imgur.com/JH58tXY.png\n",
    "   :alt:\n",
    "\n",
    "类别张量是一个大小为 ``<1 x n_categories>`` 的 `one-hot\n",
    "tensor <https://en.wikipedia.org/wiki/One-hot>`__ 张量, \n",
    "在训练的每一个时间点把它喂给网络 —— 这是一个设计的选择, 它可以被当作为初始隐藏状或其他策略的一部分. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 类别的 one-hot 向量\n",
    "def categoryTensor(category):\n",
    "    li = all_categories.index(category)\n",
    "    tensor = torch.zeros(1, n_categories)\n",
    "    tensor[0][li] = 1\n",
    "    return tensor\n",
    "\n",
    "# 输入串从第一个字母到最后一个字母（不包括 EOS ）的 one-hot 矩阵\n",
    "def inputTensor(line):\n",
    "    tensor = torch.zeros(len(line), 1, n_letters)\n",
    "    for li in range(len(line)):\n",
    "        letter = line[li]\n",
    "        tensor[li][0][all_letters.find(letter)] = 1\n",
    "    return tensor\n",
    "\n",
    "# 目标的第二个字母到结尾（EOS）的 LongTensor \n",
    "def targetTensor(line):\n",
    "    letter_indexes = [all_letters.find(line[li]) for li in range(1, len(line))]\n",
    "    letter_indexes.append(n_letters - 1) # EOS\n",
    "    return torch.LongTensor(letter_indexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了训练过程的便利, 添加一个 ``randomTrainingExample`` 函数, 获取随机的 (category, line) 对, \n",
    "并把他们转换成需要的 (category, input, target) 张量. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从随机的（category, line）对中生成 category, input, and target 张量 \n",
    "def randomTrainingExample():\n",
    "    category, line = randomTrainingPair()\n",
    "    category_tensor = Variable(categoryTensor(category))\n",
    "    input_line_tensor = Variable(inputTensor(line))\n",
    "    target_line_tensor = Variable(targetTensor(line))\n",
    "    return category_tensor, input_line_tensor, target_line_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "网络的训练\n",
    "--------------------\n",
    "\n",
    "与分类相比, 分类只用到了最后的输出, 而这里每个步都会产生一个预测, 所以我们需要计算每一步的损失. \n",
    "\n",
    "自动求导（autograd）的魔力就在于, 它允许将每一步的损失简单的加和, 并在最后调用 backward \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.NLLLoss()\n",
    "\n",
    "learning_rate = 0.0005\n",
    "\n",
    "def train(category_tensor, input_line_tensor, target_line_tensor):\n",
    "    hidden = rnn.initHidden()\n",
    "\n",
    "    rnn.zero_grad()\n",
    "\n",
    "    loss = 0\n",
    "\n",
    "    for i in range(input_line_tensor.size()[0]):\n",
    "        output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)\n",
    "        loss += criterion(output, target_line_tensor[i])\n",
    "\n",
    "    loss.backward()\n",
    "\n",
    "    for p in rnn.parameters():\n",
    "        p.data.add_(-learning_rate, p.grad.data)\n",
    "\n",
    "    return output, loss.data[0] / input_line_tensor.size()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了跟踪训练花费了多长时间, 这里添加一个 ``timeSince(timestamp)`` 函数, 返回一个人们易读的字符串: \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "def timeSince(since):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练和往常一样, 不停的调用 train 并等待一会, 打印当前时间, 每隔 ``print_every`` \n",
    "个例子打印 loss, 将每 ``plot_every`` 个例子的平均损失保存在 ``all_losses`` 中以便后面画图. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "empty range for randrange() (0,0, 0)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-8-03063af0800e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0miter\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mn_iters\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 12\u001b[1;33m     \u001b[0moutput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mloss\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mrandomTrainingExample\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     13\u001b[0m     \u001b[0mtotal_loss\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[0mloss\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-5-329adcb105d8>\u001b[0m in \u001b[0;36mrandomTrainingExample\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# 从随机的（category, line）对中生成 category, input, and target 张量\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mrandomTrainingExample\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m     \u001b[0mcategory\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mline\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrandomTrainingPair\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m     \u001b[0mcategory_tensor\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mVariable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcategoryTensor\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcategory\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m     \u001b[0minput_line_tensor\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mVariable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputTensor\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mline\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-3-8c56d37bc5fe>\u001b[0m in \u001b[0;36mrandomTrainingPair\u001b[1;34m()\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;31m# 获取随机的类别和该类别中随机的行\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mrandomTrainingPair\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m     \u001b[0mcategory\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrandomChoice\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mall_categories\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m     \u001b[0mline\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrandomChoice\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcategory_lines\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mcategory\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mcategory\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mline\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-3-8c56d37bc5fe>\u001b[0m in \u001b[0;36mrandomChoice\u001b[1;34m(l)\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;31m# 从list中随机选取项\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mrandomChoice\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ml\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0ml\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mrandom\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ml\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m-\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;31m# 获取随机的类别和该类别中随机的行\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\random.py\u001b[0m in \u001b[0;36mrandint\u001b[1;34m(self, a, b)\u001b[0m\n\u001b[0;32m    219\u001b[0m         \"\"\"\n\u001b[0;32m    220\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 221\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ma\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mb\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    222\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    223\u001b[0m     def _randbelow(self, n, int=int, maxsize=1<<BPF, type=type,\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\random.py\u001b[0m in \u001b[0;36mrandrange\u001b[1;34m(self, start, stop, step, _int)\u001b[0m\n\u001b[0;32m    197\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mistart\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_randbelow\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mwidth\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    198\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mstep\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 199\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"empty range for randrange() (%d,%d, %d)\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mistart\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mistop\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwidth\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    200\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    201\u001b[0m         \u001b[1;31m# Non-unit step argument supplied.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: empty range for randrange() (0,0, 0)"
     ]
    }
   ],
   "source": [
    "rnn = RNN(n_letters, 128, n_letters)\n",
    "\n",
    "n_iters = 100000\n",
    "print_every = 5000\n",
    "plot_every = 500\n",
    "all_losses = []\n",
    "total_loss = 0 # 每 plot_every 次迭代需要重置\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "for iter in range(1, n_iters + 1):\n",
    "    output, loss = train(*randomTrainingExample())\n",
    "    total_loss += loss\n",
    "\n",
    "    if iter % print_every == 0:\n",
    "        print('%s (%d %d%%) %.4f' % (timeSince(start), iter, iter / n_iters * 100, loss))\n",
    "\n",
    "    if iter % plot_every == 0:\n",
    "        all_losses.append(total_loss / plot_every)\n",
    "        total_loss = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制损失\n",
    "-------------------\n",
    "\n",
    "从 all\\_losses 中绘制历史损失, 以展现网络的学习过程\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x194f75e8048>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "网络采样\n",
    "====================\n",
    "\n",
    "为了采样, 我们给网络一个字母并问下一个字母是什么, 重复这个过程直到 EOS 标记. \n",
    "\n",
    "-  创建输入类别、起始字母和隐藏层状态的张量\n",
    "-  创建一个带有起始字母的 ``output_name`` 串\n",
    "-  直到最大的输出长度, \n",
    "\n",
    "   -  当前字母喂给网络\n",
    "   -  从最高的输出获取下一个字母和下一个隐藏层状态\n",
    "   -  如果输出字母是 EOS, 算法结束\n",
    "   -  如果输出是常规字母, 将其加入到 ``output_name`` 并继续\n",
    "\n",
    "-  返回最终的名字\n",
    "\n",
    ".. Note::\n",
    "   与给定起始字母不同的是, 有其他的策略是在训练的时候包含一个“串起始”标记, 让网络选择属于自己的起始字母. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_length = 20\n",
    "\n",
    "# 从类别和起始字母采样\n",
    "def sample(category, start_letter='A'):\n",
    "    category_tensor = Variable(categoryTensor(category))\n",
    "    input = Variable(inputTensor(start_letter))\n",
    "    hidden = rnn.initHidden()\n",
    "\n",
    "    output_name = start_letter\n",
    "\n",
    "    for i in range(max_length):\n",
    "        output, hidden = rnn(category_tensor, input[0], hidden)\n",
    "        topv, topi = output.data.topk(1)\n",
    "        topi = topi[0][0]\n",
    "        if topi == n_letters - 1:\n",
    "            break\n",
    "        else:\n",
    "            letter = all_letters[topi]\n",
    "            output_name += letter\n",
    "        input = Variable(inputTensor(letter))\n",
    "\n",
    "    return output_name\n",
    "\n",
    "# 给定一个类别和多个起始字母 获取个采样结果\n",
    "def samples(category, start_letters='ABC'):\n",
    "    for start_letter in start_letters:\n",
    "        print(sample(category, start_letter))\n",
    "\n",
    "samples('Russian', 'RUS')\n",
    "\n",
    "samples('German', 'GER')\n",
    "\n",
    "samples('Spanish', 'SPA')\n",
    "\n",
    "samples('Chinese', 'CHI')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "练习\n",
    "=========\n",
    "\n",
    "-  尝试使用不同 类别->行 数据集, 例如: \n",
    "\n",
    "   -  小说系列 -> 角色名字\n",
    "   -  词性 -> 词语\n",
    "   -  国家 -> 城市\n",
    "\n",
    "-  使用“串起始”标记, 使采样的时候不用给定起始字母\n",
    "-  使用更大和/或更好的网络结构获取更好的结果\n",
    "\n",
    "   -  尝试一下 nn.LSTM 和 nn.GRU 层\n",
    "   -  将这些 RNNs 组合成更高级的网络\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}