pytorch/NLP/05-advanced_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "高级教程: 作出动态决策和 Bi-LSTM CRF\n",
    "======================================================\n",
    "\n",
    "动态 VS 静态深度学习工具集\n",
    "--------------------------------------------\n",
    "\n",
    "Pytorch 是一个 *动态* 神经网络工具包. 另一个动态工具包的例子是 `Dynet <https://github.com/clab/dynet>`__ \n",
    "(我之所以提这个是因为使用 Pytorch 和 Dynet 是十分类似的. \n",
    "如果你看过 Dynet 中的例子, 那么它将有可能对你在 Pytorch 下实现它有帮助). 与动态相反的是 *静态* 工具包, \n",
    "包括了 Theano, Keras, TensorFlow 等等. \n",
    "下面是这两者核心的一些区别: \n",
    "\n",
    "* 在一个静态工具包中, 你一次性定义好一个计算图, 接着编译它, 然后把数据流输实例送进去. \n",
    "* 在一个动态工具包中, 你 *为每一个实例* 定义一个计算图, 它完全不需要被编译并且是在运行中实时执行的. \n",
    "\n",
    "若没有丰富的经验, 你很难体会出其中的差别. 举一个例子, 假设我们想要构建一个深度句法分析器. \n",
    "那么我们的模型需要下列的一些步骤: \n",
    "\n",
    "* 我们从下往上构建树\n",
    "* 标注根节点(句子中的词语)\n",
    "* 从那儿开始, 使用一个神经网络和词向量来找到组成句法的不同组合. 一旦当你形成了一个新的句法, \n",
    "  使用某种方式得到句法的嵌入表示 (embedding). 在这个例子里, 我们的网络架构将会\n",
    "  完全的依赖于输入的句子. 来看这个句子: \"绿色猫抓了墙\", 在这个模型的某一节点, 我们想要把范围\n",
    "  $(i,j,r) = (1, 3, \\text{NP})$ 合并起来(即, 一个 NP 句法范围跨越词1到词3, \n",
    "  在这个例子中是\"绿色猫\"). \n",
    "\n",
    "然而, 另一个句子可能是\"某处, 那个大肥猫抓了墙.\" 在这个句子中, \n",
    "我们想要在某点形成句法 $(2, 4, NP)$ . 我们想要形成的句法将会依赖于这个实例. 如果仅仅编译这个计算图一次, \n",
    "就像在静态工具包中那样, 那么我们给这个逻辑编程将会变得十分困难或者根本不可能. 然而, 在一个动态工具包中, \n",
    "并不仅仅只有一个预定义的计算图. 对于每一个实例, 都能够有一个新的计算图, 所以上面的问题就不复存在了. \n",
    "\n",
    "动态工具包也具有更容易调试和更接近所使用的编程语言的特点(我的意思是 Pytorch 和 Dynet 看上去\n",
    "比 Keras 和 Theano 更像 Python). \n",
    "\n",
    "\n",
    "Bi-LSTM CRF (条件随机场) 讨论\n",
    "-------------------------------------------\n",
    "\n",
    "在这一部分, 我们将会看到一个完整且复杂的 Bi-LSTM CRF (条件随机场)用来命名实体识别 (NER) 的例子. \n",
    "上面的 LSTM 标注工具通常情况下对词性标注已经足够用了, 但一个序列模型比如 CRF 对于在 NER 下取得\n",
    "强劲的表现是至关重要的. 假设熟悉 CRF. 尽管这个名字听上去吓人, 但所有的模型只是一个由 LSTM 提供\n",
    "特征的 CRF. 但这是一个高级的模型, 远比这个教程中的其它早期的模型更加复杂. 如果你要跳过这一部分, \n",
    "没有关系. 想要确定你是否准备好, 那看看你是不是能够: \n",
    "\n",
    "-  复现标签 k 的第 i 步维特比变量的算法. \n",
    "-  修改上述循环来计算正向变量. \n",
    "-  再一次修改上述复现来在对数空间中计算正向变量. (提示: 对数-求和-指数)\n",
    "\n",
    "如果你能够完成以上三件事, 那么你就不难理解下面的代码了. 回想一下, CRF 计算的是一个条件概率. \n",
    "让 $y$ 作为一个标注序列, $x$ 作为某个词的输入序列. 接下来我们计算: \n",
    "\n",
    "\\begin{align}P(y|x) = \\frac{\\exp{(\\text{Score}(x, y)})}{\\sum_{y'} \\exp{(\\text{Score}(x, y')})}\\end{align}\n",
    "\n",
    "上面的分数 Score 是由定义一些对数势能 \n",
    "$\\log \\psi_i(x,y)$ 而决定的. 进而\n",
    "\n",
    "\\begin{align}\\text{Score}(x,y) = \\sum_i \\log \\psi_i(x,y)\\end{align}\n",
    "\n",
    "要使分割函数易于掌控, 势能必须只能集中于局部的特征. \n",
    "\n",
    "\n",
    "在 Bi-LSTM CRF 中, 我们定义两种势能 (potential): 释放 (emission) 和过渡 (transition). \n",
    "索引 $i$ 处字的释放势能来自于 $i$ 时间处的 Bi-LSTM 的隐藏状态. \n",
    "过渡势能的分数储存在 $|T|x|T|$ 矩阵 $\\textbf{P}$ , 其中 \n",
    "$T$ 是标注集合. 在我的实现中, $\\textbf{P}_{j,k}$ 是从标注 $k$ 过渡到\n",
    "标注 $j$ 的得分. 因此: \n",
    "\n",
    "\\begin{align}\\text{Score}(x,y) = \\sum_i \\log \\psi_\\text{EMIT}(y_i \\rightarrow x_i) + \\log \\psi_\\text{TRANS}(y_{i-1} \\rightarrow y_i)\\end{align}\n",
    "\n",
    "\\begin{align}= \\sum_i h_i[y_i] + \\textbf{P}_{y_i, y_{i-1}}\\end{align}\n",
    "\n",
    "在上面第二个表达式中, 我们认为标签被分配了独一无二的非负索引. \n",
    "\n",
    "如果上面的讨论太简短了, 你还可以看看 `这个 <http://www.cs.columbia.edu/%7Emcollins/crf.pdf>`__ \n",
    "由 Michael Collins 写的关于 CRFs 的文章. \n",
    "\n",
    "具体实现笔记\n",
    "--------------------\n",
    "\n",
    "下面的例子实现了在对数空间中的前向算法来计算出分割函数和维特比算法来进行译码. \n",
    "反向传播将会为我们自动计算出梯度. 我们不需要手动去实现这个. \n",
    "\n",
    "这个代码中的实现并没有优化过. 如果你理解下面的过程, 也许你会觉得下面的代码中, 前向算法中\n",
    "的迭代下一次标注可以在一次大的运算中完成. 虽然有简化的余地, 但我想的是让代码可读性更好. \n",
    "如果你想进行相关的修改, 也许你可以在一些真实的任务中使用这个标注器. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 如何轻松愉快地理解条件随机场\n",
    "https://www.jianshu.com/p/55755fc649b1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x16d9b5edad0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 作者: Robert Guthrie\n",
    "\n",
    "import torch\n",
    "import torch.autograd as autograd\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一些帮助函数, 使代码可读性更好\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_scalar(var):\n",
    "    # 返回 python 浮点数 (float)\n",
    "    return var.view(-1).data.tolist()[0]\n",
    "\n",
    "\n",
    "def argmax(vec):\n",
    "    # 以 python 整数的形式返回 argmax\n",
    "    _, idx = torch.max(vec, 1)\n",
    "    return idx\n",
    "\n",
    "\n",
    "def prepare_sequence(seq, to_ix):\n",
    "    idxs = [to_ix[w] for w in seq]\n",
    "    tensor = torch.LongTensor(idxs)\n",
    "    return autograd.Variable(tensor)\n",
    "\n",
    "\n",
    "# 使用数值上稳定的方法为前向算法计算指数和的对数\n",
    "def log_sum_exp(vec):\n",
    "    max_score = vec[0, argmax(vec)]\n",
    "    max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1])\n",
    "    return max_score + \\\n",
    "        torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "t=np.arange(12).reshape(3,4)\n",
    "argmax(torch.from_numpy(t))\n",
    "t=torch.from_numpy(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "t0=t[0].view(1,-1)\n",
    "t0=t0.double()\n",
    "log_sum_exp(t0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建模型\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.LSTM?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.randn(1,10).expand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.randn(1,10).expand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  微比特算法\n",
    "已知初始概率，状态转移概率，状态到观测的概率，并且已知观测序列，求序列对应最可能的隐藏状态\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 初始化最开始N个 状态概率\n",
    "2. 上一步每个状态到下一步每个状态再到观察层的概率，计算出上一个状态对应的不同最大值。\n",
    "3. 把上一步的状态结果替换上上一步的状态概率结果。\n",
    "4. 保存路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiLSTM_CRF(nn.Module):\n",
    "\n",
    "    def __init__(self, vocab_size, tag_to_ix, embedding_dim, hidden_dim):\n",
    "        super(BiLSTM_CRF, self).__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.vocab_size = vocab_size\n",
    "        self.tag_to_ix = tag_to_ix#标签。比如BIO启动，停止\n",
    "        self.tagset_size = len(tag_to_ix)\n",
    "\n",
    "        self.word_embeds = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim // 2,\n",
    "                            num_layers=1, bidirectional=True)#会自动填充吗？\n",
    "\n",
    "        # 将LSTM的输出映射到标记空间cong hidden 到标签个数\n",
    "        self.hidden2tag = nn.Linear(hidden_dim, self.tagset_size)\n",
    "\n",
    "        # 过渡参数矩阵. 条目 i,j 是,一种被认为是模块参数的矩阵\n",
    "        # *从* j *到* i 的过渡的分数\n",
    "        \n",
    "        self.transitions = nn.Parameter(\n",
    "            torch.randn(self.tagset_size, self.tagset_size))\n",
    "        '''\n",
    "        从行向列转换，为转换概率\n",
    "            a b c st en\n",
    "        a            -\n",
    "        b            -\n",
    "        c            -\n",
    "        st - - - - - - \n",
    "        en           -\n",
    "        任何状态不能像start转换。\n",
    "        end不能像任何状态转换。\n",
    "        '''\n",
    "        # 这两句声明强制约束了我们不能\n",
    "        # 向开始标记标注传递和从结束标注传递        \n",
    "        self.transitions.data[tag_to_ix[START_TAG], :] = -10000\n",
    "        self.transitions.data[:, tag_to_ix[STOP_TAG]] = -10000\n",
    "\n",
    "        self.hidden = self.init_hidden()\n",
    "\n",
    "    def init_hidden(self):\n",
    "        '''\n",
    "        num_layer*number_direction??,batch_size，hidden\n",
    "        '''\n",
    "        return (autograd.Variable(torch.randn(2, 1, self.hidden_dim // 2)),\n",
    "                autograd.Variable(torch.randn(2, 1, self.hidden_dim // 2)))\n",
    "\n",
    "    def _forward_alg(self, feats):\n",
    "        '''\n",
    "        feats:lstm features\n",
    "        len_senten,target_size\n",
    "        序列长度，目标长度的大小\n",
    "        '''\n",
    "        # 执行前向算法来计算分割函数\n",
    "        init_alphas = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
    "        # START_TAG 包含所有的分数\n",
    "        init_alphas[0][self.tag_to_ix[START_TAG]] = 0.\n",
    "\n",
    "        # 将其包在一个变量类型中继而得到自动的反向传播\n",
    "        forward_var = autograd.Variable(init_alphas)\n",
    "\n",
    "        # 在句子中迭代\n",
    "        for feat in feats:#句子序列的特征。\n",
    "            alphas_t = []  # 在这个时间步的前向变量\n",
    "            for next_tag in range(self.tagset_size):#5\n",
    "                # 对 emission 得分执行广播机制: 它总是相同的, # 不论前一个标注如何                \n",
    "                #expand 延伸。特征其实就是x---->到所有的情况\n",
    "                emit_score = feat[next_tag].view(\n",
    "                    1, -1).expand(1, self.tagset_size)\n",
    "                # trans_score 第 i 个条目是 从i过渡到 next_tag 的分数\n",
    "                trans_score = self.transitions[next_tag].view(1, -1)\n",
    "                # next_tag_var 第 i 个条目是在我们执行 对数-求和-指数 前边缘的值 (i -> next_tag)\n",
    "                next_tag_var = forward_var + trans_score + emit_score\n",
    "                # 这个标注的前向变量是\n",
    "                # 对所有的分数执行 对数-求和-指数\n",
    "                alphas_t.append(log_sum_exp(next_tag_var))\n",
    "            forward_var = torch.cat(alphas_t).view(1, -1)\n",
    "        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]\n",
    "        alpha = log_sum_exp(terminal_var)\n",
    "        return alpha\n",
    "\n",
    "    def _get_lstm_features(self, sentence):\n",
    "        '''利用embed，然后输入到lstm 利用lstM的输出在做一个线性函数当做特征。\n",
    "        OUTPUT:seq_len, batch, hidden_size * num_directions\n",
    "        是不是要保证这里的bathc为1\n",
    "        '''\n",
    "        self.hidden = self.init_hidden()\n",
    "        embeds = self.word_embeds(sentence).view(len(sentence), 1, -1)\n",
    "        lstm_out, self.hidden = self.lstm(embeds, self.hidden)\n",
    "        lstm_out = lstm_out.view(len(sentence), self.hidden_dim)\n",
    "        lstm_feats = self.hidden2tag(lstm_out)\n",
    "        return lstm_feats\n",
    "\n",
    "    def _score_sentence(self, feats, tags):\n",
    "        '''\n",
    "        senten_len*dim\n",
    "        senten_len\n",
    "        \n",
    "        '''\n",
    "        # 给出标记序列的分数\n",
    "        score = autograd.Variable(torch.Tensor([0]))\n",
    "        #都要添加开始标记\n",
    "        tags = torch.cat([torch.LongTensor([self.tag_to_ix[START_TAG]]), tags])\n",
    "        \n",
    "        for i, feat in enumerate(feats):\n",
    "            '''\n",
    "            feta,每个单词对应的特征\n",
    "            i,单词所在的位置\n",
    "            score+第i转移到i+1的概率？+i+1的特征值。\n",
    "            '''\n",
    "            score = score +self.transitions[tags[i + 1], tags[i]] + feat[tags[i + 1]]\n",
    "            \n",
    "        score = score + self.transitions[self.tag_to_ix[STOP_TAG], tags[-1]]\n",
    "        return score\n",
    "\n",
    "    def _viterbi_decode(self, feats):\n",
    "        '''\n",
    "        \n",
    "        '''\n",
    "        backpointers = []\n",
    "\n",
    "        # 在对数空间中初始化维特比变量\n",
    "        #\n",
    "        init_vvars = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
    "        init_vvars[0][self.tag_to_ix[START_TAG]] = 0\n",
    "\n",
    "        # 在第 i 步的 forward_var 存放第 i-1 步的维特比变量\n",
    "        forward_var = autograd.Variable(init_vvars)\n",
    "        #\n",
    "        for feat in feats:\n",
    "            bptrs_t = []  # 存放这一步的后指针\n",
    "            viterbivars_t = []  # 存放这一步的维特比变量\n",
    "\n",
    "            for next_tag in range(self.tagset_size):\n",
    "                # next_tag_var[i] 存放先前一步标注i的\n",
    "                # 维特比变量, 加上了从标注 i 到 next_tag 的过渡\n",
    "                # 的分数\n",
    "                # 我们在这里并没有将 emission 分数包含进来, 因为\n",
    "                # 最大值并不依赖于它们(我们在下面对它们进行的是相加)\n",
    "                #第next_tag行的转移到各个状态的概率\n",
    "                next_tag_var = forward_var + self.transitions[next_tag]\n",
    "                best_tag_id = argmax(next_tag_var)#求出那个状态最有可能。\n",
    "                bptrs_t.append(best_tag_id)#添加最好的状态id.\n",
    "                viterbivars_t.append(next_tag_var[0][best_tag_id])\n",
    "            # 现在将所有 emission 得分相加, 将 forward_var \n",
    "            # 赋值到我们刚刚计算出来的维特比变量集合\n",
    "            forward_var = (torch.cat(viterbivars_t) + feat).view(1, -1)\n",
    "            backpointers.append(bptrs_t)\n",
    "\n",
    "        # 过渡到 STOP_TAG\n",
    "        terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]\n",
    "        best_tag_id = argmax(terminal_var)\n",
    "        path_score = terminal_var[0][best_tag_id]\n",
    "\n",
    "        # 跟着后指针去解码最佳路径\n",
    "        best_path = [best_tag_id]\n",
    "        for bptrs_t in reversed(backpointers):\n",
    "            best_tag_id = bptrs_t[best_tag_id]\n",
    "            best_path.append(best_tag_id)\n",
    "        # 弹出开始的标签 (我们并不希望把这个返回到调用函数)\n",
    "        start = best_path.pop()\n",
    "        assert start == self.tag_to_ix[START_TAG]  # 健全性检查\n",
    "        best_path.reverse()\n",
    "        return path_score, best_path\n",
    "\n",
    "    def neg_log_likelihood(self, sentence, tags):\n",
    "        feats = self._get_lstm_features(sentence)\n",
    "        forward_score = self._forward_alg(feats)\n",
    "        gold_score = self._score_sentence(feats, tags)\n",
    "        return forward_score - gold_score\n",
    "\n",
    "    def forward(self, sentence):  # 不要把这和上面的 _forward_alg 混淆\n",
    "        # 得到 BiLSTM 输出分数\n",
    "        lstm_feats = self._get_lstm_features(sentence)\n",
    "\n",
    "        # 给定特征, 找到最好的路径\n",
    "        score, tag_seq = self._viterbi_decode(lstm_feats)\n",
    "        return score, tag_seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行训练\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([ 2.6907]), [tensor([ 1]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 1])])\n",
      "(tensor([ 20.4906]), [tensor([ 0]), tensor([ 1]), tensor([ 1]), tensor([ 1]), tensor([ 2]), tensor([ 2]), tensor([ 2]), tensor([ 0]), tensor([ 1]), tensor([ 2]), tensor([ 2])])\n"
     ]
    }
   ],
   "source": [
    "START_TAG = \"<START>\"\n",
    "STOP_TAG = \"<STOP>\"\n",
    "EMBEDDING_DIM = 5\n",
    "HIDDEN_DIM = 4\n",
    "\n",
    "# 制造一些训练数据\n",
    "training_data = [(\n",
    "    \"the wall street journal reported today that apple corporation made money\".split(),\n",
    "    \"B I I I O O O B I O O\".split()\n",
    "), (\n",
    "    \"georgia tech is a university in georgia\".split(),\n",
    "    \"B I O O O O B\".split()\n",
    ")]\n",
    "\n",
    "word_to_ix = {}\n",
    "for sentence, tags in training_data:\n",
    "    for word in sentence:\n",
    "        if word not in word_to_ix:\n",
    "            word_to_ix[word] = len(word_to_ix)\n",
    "\n",
    "tag_to_ix = {\"B\": 0, \"I\": 1, \"O\": 2, START_TAG: 3, STOP_TAG: 4}\n",
    "\n",
    "model = BiLSTM_CRF(len(word_to_ix), tag_to_ix, EMBEDDING_DIM, HIDDEN_DIM)\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)\n",
    "\n",
    "# 在训练之前检查预测结果\n",
    "precheck_sent = prepare_sequence(training_data[0][0], word_to_ix)\n",
    "precheck_tags = torch.LongTensor([tag_to_ix[t] for t in training_data[0][1]])\n",
    "print(model(precheck_sent))\n",
    "\n",
    "# 确认从之前的 LSTM 部分的 prepare_sequence 被加载了\n",
    "for epoch in range(\n",
    "        300):  # 又一次, 正常情况下你不会训练300个 epoch, 这只是示例数据\n",
    "    for sentence, tags in training_data:\n",
    "        # 第一步: 需要记住的是Pytorch会累积梯度\n",
    "        # 我们需要在每次实例之前把它们清除\n",
    "        model.zero_grad()\n",
    "\n",
    "        # 第二步: 为我们的网络准备好输入, 即\n",
    "        # 把它们转变成单词索引变量 (Variables)\n",
    "        sentence_in = prepare_sequence(sentence, word_to_ix)\n",
    "        targets = torch.LongTensor([tag_to_ix[t] for t in tags])\n",
    "\n",
    "        # 第三步: 运行前向传递. \n",
    "        neg_log_likelihood = model.neg_log_likelihood(sentence_in, targets)\n",
    "\n",
    "        # 第四步: 计算损失, 梯度以及\n",
    "        # 使用 optimizer.step() 来更新参数\n",
    "        neg_log_likelihood.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "# 在训练之后检查预测结果\n",
    "precheck_sent = prepare_sequence(training_data[0][0], word_to_ix)\n",
    "print(model(precheck_sent))\n",
    "# 我们完成了!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 11,  12,  13,  14,  15,  16,  11])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_in"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}