pytorch/NLP/03-word_embeddings_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "词汇嵌入:编码词汇语义\n",
    "===========================================\n",
    "\n",
    "单词嵌入是真实数字的密集向量,在你的词汇表中每一个单词都是. 在NLP中, 通常情况下,\n",
    "您的特性就是单词!但是你应该怎么在你的电脑中表示一个单词?可以存储它的ascii字符表示,\n",
    "但那仅仅告诉你单词 *是* 什么,没有说太多它 *意味* 着什么 (你也许可以从它的词缀中派生出它的词性,\n",
    "或者从它的大小写中得到它的属性,但并不多.). 更重要的是, 在什么意义上你能把这些表象结合起来? \n",
    "我们经常需要神经网络的密集输出, 输入为 $|V|$ 维, 其中 $V$ 是我们的词汇表, \n",
    "但经常输出是更小维度的 (如果我们只预测少量的标签的话).\n",
    "我们如何从一个大的维度空间得到一个更小的维度空间?\n",
    "\n",
    "如果我们不用 ascii 字符表示, 而使用one-hot encoding呢?\n",
    "那就是, 我们用如下所示表示 $w$ 字符\n",
    "\n",
    "\\begin{align}\\overbrace{\\left[ 0, 0, \\dots, 1, \\dots, 0, 0 \\right]}^\\text{|V| elements}\\end{align}\n",
    "\n",
    "其中 1 是 $w$ 的特征位置.其他的单词也是在其他位置有一个1, 在另外的位置都是0. \n",
    "\n",
    "除了巨大的占用空间外,这种表述有一个巨大的缺陷. 它仅仅简单的把所有的单词都看作独立实体认\n",
    "为它们彼此之间毫无关联.我们真正想要的是单词之间有一些 *相似* .为什么? 让我们来看一下例子.\n",
    "\n",
    "假设我们正在搭建一个语言模型. 假设我们在训练集中看到了如下语句.\n",
    "\n",
    "* The mathematician ran to the store.\n",
    "* The physicist ran to the store.\n",
    "* The mathematician solved the open problem.\n",
    "\n",
    "现在假设我们得到了一个在训练集从未看到过的新句子:\n",
    "\n",
    "* The physicist solved the open problem.\n",
    "\n",
    "我们的语言模型对此句可能运行的不错, 但如果我们能使用以下两个事实,情况会好得多吗:\n",
    "\n",
    "* 我们看到数学家和物理学家在句子中有着相同的作用.它们之间有一个语义关系.\n",
    "* 我们已经看到数学家在这个新的没看过的的句子中扮演着同样的角色, 就像我们现在看到的物理学家一样.\n",
    "\n",
    "然后我们就推断物理学家在这个句子里是很合适的?这就是我们指的相似的意思:我们指的是\n",
    "*语义相似度*, 不仅仅是拼字一样的表示.\n",
    "它是一种通过连接我们所看到的和我们没有看到的东西之间的点来对抗语言数据稀疏性的技术.\n",
    "这个例子当然要依赖于一个基本的语言假设:在相似的语境中出现的单词在语义上是相互关联的. \n",
    "这被叫做 `distributional\n",
    "hypothesis <https://en.wikipedia.org/wiki/Distributional_semantics>`__.\n",
    "\n",
    "\n",
    "Getting Dense Word Embeddings(密集字嵌入)\n",
    "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
    "\n",
    "我们如何来解决那个问题?我们怎么能在单词中编码语义相似呢? 也许我们会想出一些语义属性.\n",
    "举个例子, 我们看到了, 数学家和物理学家都会跑, 也许我们可以把\"能跑\"这个语义属性给一个高分.\n",
    "考虑一下其他的属性, 想象一下, 你可能会在这些属性上给一些普通的单词得分.\n",
    "\n",
    "如果每一个属性都是一维, 那我们可以给一个向量代表一个单词, 像这样:\n",
    "\n",
    "\\begin{align}q_\\text{mathematician} = \\left[ \\overbrace{2.3}^\\text{can run},\n",
    "   \\overbrace{9.4}^\\text{likes coffee}, \\overbrace{-5.5}^\\text{majored in Physics}, \\dots \\right]\\end{align}\n",
    "\n",
    "\\begin{align}q_\\text{physicist} = \\left[ \\overbrace{2.5}^\\text{can run},\n",
    "   \\overbrace{9.1}^\\text{likes coffee}, \\overbrace{6.4}^\\text{majored in Physics}, \\dots \\right]\\end{align}\n",
    "\n",
    "这样我们就可以通过如下来得到这些单词之间的相似度:\n",
    "\n",
    "\\begin{align}\\text{Similarity}(\\text{physicist}, \\text{mathematician}) = q_\\text{physicist} \\cdot q_\\text{mathematician}\\end{align}\n",
    "\n",
    "尽管通常情况下需要归一化:\n",
    "\n",
    "\\begin{align}\\text{Similarity}(\\text{physicist}, \\text{mathematician}) = \\frac{q_\\text{physicist} \\cdot q_\\text{mathematician}}\n",
    "   {\\| q_\\text{\\physicist} \\| \\| q_\\text{mathematician} \\|} = \\cos (\\phi)\\end{align}\n",
    "\n",
    "其中 $\\phi$ 是两个向量的角度. 这就意味着,极端相似的单词(嵌入方向是同一个)\n",
    "会得到相似度为1.反之为 -1.\n",
    "\n",
    "\n",
    "你可以认为本章刚开始的稀疏one-hot 向量是我们刚定义向量的特殊形式,其中单词的相似度为 0, \n",
    "然后我们可以给每一个单词一些独特的语义属性.这些向量是 *密集的* , 也就是说他们是非零的.\n",
    "\n",
    "但是这些新的向量是一种巨大的痛苦:你可以想到数千种不同的语义属性,它们可能与决定相似性有关,\n",
    "而且究竟你怎样把它们设置成不同的属性? 深度学习的中心思想是比起需要程序员去自己设计特征,\n",
    "神经网络学习特征的表示. 所以为什么不在我们的模型中让单词嵌入到系数中,然后让它们在训练中更新呢? \n",
    "这就是我们要做的. 我们会有一些 *潜在的语义属性* 网络可以, 严肃来讲, 学习. 注意, 嵌入词可能无法解释.\n",
    "那就是尽管如上所示我们手工制作的矢量图,我们可以看到数学家和物理学家的相似之处是他们都喜欢咖啡.\n",
    "如果我们允许神经网络学习嵌入, 并看到数学家和物理学家在第二个维度中有很大的价值,\n",
    "但是它意味着什么很不清晰. 在潜在语义来讲它们是相似的, 但是对我们来说是无法解释的. \n",
    "\n",
    "\n",
    "总结一下, **单词嵌入是一个单词 *语义* 的表示,语义信息的有效编码可能与手头任务相关**. \n",
    "你也可以嵌入其他的东西: 部分的语音标签, 解析树, 其他任何东西! 特征嵌入是这个领域的核心思想.\n",
    "\n",
    "\n",
    "Word Embeddings in Pytorch（Pytorch中的单词嵌入）\n",
    "~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
    "\n",
    "在我们举例或练习之前, 关于如何在Pytorch中使用嵌入以及在一般的深度学习编程中,有一些快速\n",
    "的说明.与制作one-hot向量时我们对每一个单词定义一个特别的索引相似,单词嵌入时同样需要对每\n",
    "一个单词定义一个特别的索引. 这些将是查找表中的键. 意思是,嵌入被储存为一个 $|V| \\times D$ 矩阵, \n",
    "其中 $D$ 是嵌入的维度, 这样的词被赋予了索引 $i$ 它的嵌入被储存在矩阵的\n",
    "第 $i$ 行. 在所有的代码中, 从单词到索引的映射是一个命名的字典 word\\_to\\_ix.\n",
    "\n",
    "允许你使用嵌入的模块式 torch.nn.Embedding,这需要两个参数:词汇量和嵌入的维度.\n",
    "\n",
    "为了索引到这个表中,你需要使用 torch.LongTensor (索引为整数,不能为浮点数).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1a4510547f0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 作者: Robert Guthrie\n",
    "\n",
    "import torch\n",
    "import torch.autograd as autograd\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.Embedding?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.8330,  0.3568,  1.3209, -0.5273, -0.9138]])\n"
     ]
    }
   ],
   "source": [
    "word_to_ix = {\"hello\": 0, \"world\": 1,'me':2}\n",
    "embeds = nn.Embedding(2, 5)  # 2 单词, 5 维嵌入\n",
    "lookup_tensor = torch.LongTensor([word_to_ix[\"hello\"]])\n",
    "hello_embed = embeds(autograd.Variable(lookup_tensor))\n",
    "print(hello_embed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "例子: N-Gram 语言模型\n",
    "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
    "\n",
    "回想一下 在n-gram语言模型中,给定一系列单词 $w$ , 我们需要计算\n",
    "\n",
    "\\begin{align}P(w_i | w_{i-1}, w_{i-2}, \\dots, w_{i-n+1} )\\end{align}\n",
    "\n",
    "$w_i$ 是句子中第i个单词.\n",
    "\n",
    "本例中, 我们将计算一些训练集的损失函数并且用反向传播更新系数.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['When', 'forty'], 'winters'), (['forty', 'winters'], 'shall'), (['winters', 'shall'], 'besiege')]\n"
     ]
    }
   ],
   "source": [
    "CONTEXT_SIZE = 2\n",
    "EMBEDDING_DIM = 10\n",
    "# 我们将使用 Shakespeare Sonnet 2\n",
    "test_sentence = \"\"\"When forty winters shall besiege thy brow,\n",
    "And dig deep trenches in thy beauty's field,\n",
    "Thy youth's proud livery so gazed on now,\n",
    "Will be a totter'd weed of small worth held:\n",
    "Then being asked, where all thy beauty lies,\n",
    "Where all the treasure of thy lusty days;\n",
    "To say, within thine own deep sunken eyes,\n",
    "Were an all-eating shame, and thriftless praise.\n",
    "How much more praise deserv'd thy beauty's use,\n",
    "If thou couldst answer 'This fair child of mine\n",
    "Shall sum my count, and make my old excuse,'\n",
    "Proving his beauty by succession thine!\n",
    "This were to be new made when thou art old,\n",
    "And see thy blood warm when thou feel'st it cold.\"\"\".split()\n",
    "# 我们应该对输入进行标记,但是我们将忽略它\n",
    "# 建造一系列元组.  每个元组 ([ word_i-2, word_i-1 ], 都是目标单词)\n",
    "trigrams = [(test_sentence[i:i+CONTEXT_SIZE], test_sentence[i +CONTEXT_SIZE])\n",
    "            for i in range(len(test_sentence) - CONTEXT_SIZE)]\n",
    "# 输出前 3, 为了你看到他的各式\n",
    "print(trigrams[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = set(test_sentence)\n",
    "word_to_ix = {word: i for i, word in enumerate(vocab)}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb=nn.Embedding(2,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.4629,  0.5493, -1.4471,  0.2785, -1.0307, -1.1284,  0.2359,\n",
       "          0.1504, -0.9224,  0.5913],\n",
       "        [ 0.0732, -1.4927,  1.1664,  0.2566,  0.7201, -0.4593,  0.2768,\n",
       "          0.1331,  1.7475,  0.5126]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb(autograd.Variable(torch.LongTensor([0,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "        vocab_size=1200 单词本\n",
    "        embedding_dim=10，嵌入维度\n",
    "        context_size=2， 单词长度\n",
    "* 使用EMbedding(单词本个数，嵌入维度)，单词长度*嵌入维度\n",
    "* linear1，\n",
    "* relu\n",
    "* Linear2, 单词本，预测下一个对应的单词位置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 loss: tensor(4.5298)\n",
      "10 loss: tensor(4.2976)\n",
      "20 loss: tensor(4.0705)\n",
      "30 loss: tensor(3.8427)\n",
      "40 loss: tensor(3.6113)\n",
      "50 loss: tensor(3.3692)\n",
      "60 loss: tensor(3.1166)\n",
      "70 loss: tensor(2.8520)\n",
      "80 loss: tensor(2.5769)\n",
      "90 loss: tensor(2.2951)\n",
      "100 loss: tensor(2.0130)\n",
      "110 loss: tensor(1.7378)\n",
      "120 loss: tensor(1.4801)\n",
      "130 loss: tensor(1.2483)\n",
      "140 loss: tensor(1.0506)\n",
      "150 loss: tensor(0.8859)\n",
      "160 loss: tensor(0.7513)\n",
      "170 loss: tensor(0.6427)\n",
      "180 loss: tensor(0.5540)\n",
      "190 loss: tensor(0.4818)\n",
      "200 loss: tensor(0.4231)\n",
      "210 loss: tensor(0.3747)\n",
      "220 loss: tensor(0.3343)\n",
      "230 loss: tensor(0.3005)\n",
      "240 loss: tensor(0.2718)\n",
      "250 loss: tensor(0.2471)\n",
      "260 loss: tensor(0.2256)\n",
      "270 loss: tensor(0.2068)\n",
      "280 loss: tensor(0.1903)\n",
      "290 loss: tensor(0.1757)\n",
      "300 loss: tensor(0.1627)\n",
      "310 loss: tensor(0.1512)\n",
      "320 loss: tensor(0.1409)\n",
      "330 loss: tensor(0.1318)\n",
      "340 loss: tensor(0.1235)\n",
      "350 loss: tensor(0.1161)\n",
      "360 loss: tensor(0.1095)\n",
      "370 loss: tensor(0.1034)\n",
      "380 loss: tensor(1.00000e-02 *\n",
      "       9.7952)\n",
      "390 loss: tensor(1.00000e-02 *\n",
      "       9.2963)\n",
      "400 loss: tensor(1.00000e-02 *\n",
      "       8.8396)\n",
      "410 loss: tensor(1.00000e-02 *\n",
      "       8.4212)\n",
      "420 loss: tensor(1.00000e-02 *\n",
      "       8.0366)\n",
      "430 loss: tensor(1.00000e-02 *\n",
      "       7.6808)\n",
      "440 loss: tensor(1.00000e-02 *\n",
      "       7.3526)\n",
      "450 loss: tensor(1.00000e-02 *\n",
      "       7.0492)\n",
      "460 loss: tensor(1.00000e-02 *\n",
      "       6.7669)\n",
      "470 loss: tensor(1.00000e-02 *\n",
      "       6.5043)\n",
      "480 loss: tensor(1.00000e-02 *\n",
      "       6.2593)\n",
      "490 loss: tensor(1.00000e-02 *\n",
      "       6.0296)\n",
      "500 loss: tensor(1.00000e-02 *\n",
      "       5.8150)\n",
      "510 loss: tensor(1.00000e-02 *\n",
      "       5.6141)\n",
      "520 loss: tensor(1.00000e-02 *\n",
      "       5.4251)\n",
      "530 loss: tensor(1.00000e-02 *\n",
      "       5.2474)\n",
      "540 loss: tensor(1.00000e-02 *\n",
      "       5.0800)\n",
      "550 loss: tensor(1.00000e-02 *\n",
      "       4.9219)\n",
      "560 loss: tensor(1.00000e-02 *\n",
      "       4.7726)\n",
      "570 loss: tensor(1.00000e-02 *\n",
      "       4.6315)\n",
      "580 loss: tensor(1.00000e-02 *\n",
      "       4.4979)\n",
      "590 loss: tensor(1.00000e-02 *\n",
      "       4.3712)\n",
      "600 loss: tensor(1.00000e-02 *\n",
      "       4.2509)\n",
      "610 loss: tensor(1.00000e-02 *\n",
      "       4.1364)\n",
      "620 loss: tensor(1.00000e-02 *\n",
      "       4.0276)\n",
      "630 loss: tensor(1.00000e-02 *\n",
      "       3.9239)\n",
      "640 loss: tensor(1.00000e-02 *\n",
      "       3.8250)\n",
      "650 loss: tensor(1.00000e-02 *\n",
      "       3.7305)\n",
      "660 loss: tensor(1.00000e-02 *\n",
      "       3.6402)\n",
      "670 loss: tensor(1.00000e-02 *\n",
      "       3.5538)\n",
      "680 loss: tensor(1.00000e-02 *\n",
      "       3.4710)\n",
      "690 loss: tensor(1.00000e-02 *\n",
      "       3.3918)\n",
      "700 loss: tensor(1.00000e-02 *\n",
      "       3.3156)\n",
      "710 loss: tensor(1.00000e-02 *\n",
      "       3.2425)\n",
      "720 loss: tensor(1.00000e-02 *\n",
      "       3.1726)\n",
      "730 loss: tensor(1.00000e-02 *\n",
      "       3.1051)\n",
      "740 loss: tensor(1.00000e-02 *\n",
      "       3.0405)\n",
      "750 loss: tensor(1.00000e-02 *\n",
      "       2.9782)\n",
      "760 loss: tensor(1.00000e-02 *\n",
      "       2.9182)\n",
      "770 loss: tensor(1.00000e-02 *\n",
      "       2.8606)\n",
      "780 loss: tensor(1.00000e-02 *\n",
      "       2.8049)\n",
      "790 loss: tensor(1.00000e-02 *\n",
      "       2.7515)\n",
      "800 loss: tensor(1.00000e-02 *\n",
      "       2.6999)\n",
      "810 loss: tensor(1.00000e-02 *\n",
      "       2.6501)\n",
      "820 loss: tensor(1.00000e-02 *\n",
      "       2.6019)\n",
      "830 loss: tensor(1.00000e-02 *\n",
      "       2.5552)\n",
      "840 loss: tensor(1.00000e-02 *\n",
      "       2.5102)\n",
      "850 loss: tensor(1.00000e-02 *\n",
      "       2.4667)\n",
      "860 loss: tensor(1.00000e-02 *\n",
      "       2.4245)\n",
      "870 loss: tensor(1.00000e-02 *\n",
      "       2.3836)\n",
      "880 loss: tensor(1.00000e-02 *\n",
      "       2.3439)\n",
      "890 loss: tensor(1.00000e-02 *\n",
      "       2.3055)\n",
      "900 loss: tensor(1.00000e-02 *\n",
      "       2.2683)\n",
      "910 loss: tensor(1.00000e-02 *\n",
      "       2.2321)\n",
      "920 loss: tensor(1.00000e-02 *\n",
      "       2.1970)\n",
      "930 loss: tensor(1.00000e-02 *\n",
      "       2.1629)\n",
      "940 loss: tensor(1.00000e-02 *\n",
      "       2.1298)\n",
      "950 loss: tensor(1.00000e-02 *\n",
      "       2.0976)\n",
      "960 loss: tensor(1.00000e-02 *\n",
      "       2.0663)\n",
      "970 loss: tensor(1.00000e-02 *\n",
      "       2.0359)\n",
      "980 loss: tensor(1.00000e-02 *\n",
      "       2.0062)\n",
      "990 loss: tensor(1.00000e-02 *\n",
      "       1.9774)\n"
     ]
    }
   ],
   "source": [
    "class NGramLanguageModeler(nn.Module):\n",
    "\n",
    "    def __init__(self, vocab_size, embedding_dim, context_size):\n",
    "        super(NGramLanguageModeler, self).__init__()\n",
    "        self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        #embeds 输出维度context_size * embedding_dim,每个单词对应一行。\n",
    "        self.linear1 = nn.Linear(context_size * embedding_dim, 128)\n",
    "        #输入单词个数\n",
    "        self.linear2 = nn.Linear(128, vocab_size)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        embeds = self.embeddings(inputs).view((1, -1))\n",
    "        #self.Liner \n",
    "        out = F.relu(self.linear1(embeds))\n",
    "        out = self.linear2(out)\n",
    "        log_probs = F.log_softmax(out, dim=1)\n",
    "        return log_probs\n",
    "\n",
    "\n",
    "losses = []\n",
    "loss_function = nn.NLLLoss()\n",
    "#单词本个数，单词嵌入维度，2\n",
    "model = NGramLanguageModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.001)\n",
    "\n",
    "for epoch in range(1000):\n",
    "    total_loss = torch.Tensor([0])\n",
    "    for context, target in trigrams:\n",
    "\n",
    "        # 步骤 1. 准备好进入模型的数据 (例如将单词转换成整数索引,并将其封装在变量中)\n",
    "        context_idxs = [word_to_ix[w] for w in context]\n",
    "        context_var = autograd.Variable(torch.LongTensor(context_idxs))\n",
    "\n",
    "        # 步骤 2. 回调 *积累* 梯度. 在进入一个实例前,需要将之前的实力梯度置零\n",
    "        model.zero_grad()\n",
    "\n",
    "        # 步骤 3. 运行反向传播,得到单词的概率分布\n",
    "        log_probs = model(context_var)\n",
    "\n",
    "        # 步骤 4. 计算损失函数. (再次注意, Torch需要将目标单词封装在变量中)\n",
    "        loss = loss_function(log_probs, autograd.Variable(\n",
    "            torch.LongTensor([word_to_ix[target]])))\n",
    "\n",
    "        # 步骤 5. 反向传播并更新梯度\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        total_loss += loss.data\n",
    "    if epoch%10==0:\n",
    "        print(epoch,'loss:',loss.data)\n",
    "    losses.append(total_loss)\n",
    "#print(losses)  # 在训练集中每次迭代损失都会减小!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise: Computing Word Embeddings: Continuous Bag-of-Words(练习: 计算单词嵌入: 连续单词包)\n",
    "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
    "\n",
    "连续单词包模型 (CBOW) 在NLP深度学习中使用的很频繁. 这个模型尝试去预测文中目标单词的\n",
    "前后一些单词. 它有别于语言建模, 因为CBOW不是顺序的, 也不需要是概率性的.CBOW被用来快\n",
    "速训练单词嵌入,而这些嵌入被用来初始化一些复杂模型的嵌入.通常情况下, 这被称为 *预训练嵌入* .\n",
    "它几乎总是能帮助提升百分之几的性能.\n",
    "\n",
    "CBOW模型如下所示.给定一个目标单词 $w_i$ 和 $N$ \n",
    "代表单词每一遍的滑窗距, $w_{i-1}, \\dots, w_{i-N}$\n",
    "和 $w_{i+1}, \\dots, w_{i+N}$ , 将所有上下文词统称为 $C$ ,CBOW试图去最小化如下\n",
    "\n",
    "\\begin{align}-\\log p(w_i | C) = -\\log \\text{Softmax}(A(\\sum_{w \\in C} q_w) + b)\\end{align}\n",
    "\n",
    "\n",
    "其中 $q_w$ 是单词 $w$ 的嵌入.\n",
    "\n",
    "在Pytorch中通过填充下面的类来实现这个模型. 一些建议:\n",
    "\n",
    "* 想好你需要定义的系数.\n",
    "* 确保你知道每一步操作后的构造. 如果想要重构请使用 .view().\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONTEXT_SIZE = 2  # 左右各2个单词\n",
    "raw_text = \"\"\"We are about to study the idea of a computational process.\n",
    "Computational processes are abstract beings that inhabit computers.\n",
    "As they evolve, processes manipulate other abstract things called data.\n",
    "The evolution of a process is directed by a pattern of rules\n",
    "called a program. People create programs to direct processes. In effect,\n",
    "we conjure the spirits of the computer with our spells.\"\"\".split()\n",
    "\n",
    "# 通过从 `raw_text` 得到一组单词, 进行去重操作\n",
    "vocab = set(raw_text)\n",
    "vocab_size = len(vocab)\n",
    "\n",
    "word_to_ix = {word: i for i, word in enumerate(vocab)}\n",
    "data = []\n",
    "for i in range(2, len(raw_text) - 2):\n",
    "    context = [raw_text[i - 2], raw_text[i - 1],\n",
    "               raw_text[i + 1], raw_text[i + 2]]\n",
    "    target = raw_text[i]\n",
    "    data.append((context, target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOW(nn.Module):\n",
    "\n",
    "    def __init__(self,vocab_size,context_size,embedding_dim):\n",
    "        super(CBOW,self).__init__()\n",
    "        self.embeding=nn.Embedding(vocab_size,embedding_dim)\n",
    "        self.Linear1=nn.Linear(context_size*embedding_dim,128)\n",
    "        self.Linear2=nn.Linear(128,vocab_size)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        x=self.embeding(inputs).view((1,-1))\n",
    "        x=self.Linear1(x)\n",
    "        x=F.relu(x)\n",
    "        x=self.Linear2(x)\n",
    "        x=F.log_softmax(x,dim=1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 16])"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autograd.Variable(torch.LongTensor([y]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 loss: tensor(3.6178)\n",
      "100 loss: tensor(3.4572)\n",
      "200 loss: tensor(3.2895)\n",
      "300 loss: tensor(3.1178)\n",
      "400 loss: tensor(2.9454)\n",
      "500 loss: tensor(2.7732)\n",
      "600 loss: tensor(2.5967)\n",
      "700 loss: tensor(2.4123)\n",
      "800 loss: tensor(2.2151)\n",
      "900 loss: tensor(2.0030)\n",
      "1000 loss: tensor(1.7807)\n",
      "1100 loss: tensor(1.5551)\n",
      "1200 loss: tensor(1.3372)\n",
      "1300 loss: tensor(1.1379)\n",
      "1400 loss: tensor(0.9608)\n",
      "1500 loss: tensor(0.8096)\n",
      "1600 loss: tensor(0.6849)\n",
      "1700 loss: tensor(0.5848)\n",
      "1800 loss: tensor(0.5036)\n",
      "1900 loss: tensor(0.4377)\n",
      "2000 loss: tensor(0.3838)\n",
      "2100 loss: tensor(0.3397)\n",
      "2200 loss: tensor(0.3030)\n",
      "2300 loss: tensor(0.2723)\n",
      "2400 loss: tensor(0.2465)\n",
      "2500 loss: tensor(0.2245)\n",
      "2600 loss: tensor(0.2056)\n",
      "2700 loss: tensor(0.1893)\n",
      "2800 loss: tensor(0.1751)\n",
      "2900 loss: tensor(0.1627)\n",
      "3000 loss: tensor(0.1519)\n",
      "3100 loss: tensor(0.1422)\n",
      "3200 loss: tensor(0.1336)\n",
      "3300 loss: tensor(0.1259)\n",
      "3400 loss: tensor(0.1189)\n",
      "3500 loss: tensor(0.1125)\n",
      "3600 loss: tensor(0.1068)\n",
      "3700 loss: tensor(0.1015)\n",
      "3800 loss: tensor(1.00000e-02 *\n",
      "       9.6650)\n",
      "3900 loss: tensor(1.00000e-02 *\n",
      "       9.2204)\n",
      "4000 loss: tensor(1.00000e-02 *\n",
      "       8.8108)\n",
      "4100 loss: tensor(1.00000e-02 *\n",
      "       8.4322)\n",
      "4200 loss: tensor(1.00000e-02 *\n",
      "       8.0815)\n",
      "4300 loss: tensor(1.00000e-02 *\n",
      "       7.7559)\n",
      "4400 loss: tensor(1.00000e-02 *\n",
      "       7.4530)\n",
      "4500 loss: tensor(1.00000e-02 *\n",
      "       7.1705)\n",
      "4600 loss: tensor(1.00000e-02 *\n",
      "       6.9066)\n",
      "4700 loss: tensor(1.00000e-02 *\n",
      "       6.6594)\n",
      "4800 loss: tensor(1.00000e-02 *\n",
      "       6.4276)\n",
      "4900 loss: tensor(1.00000e-02 *\n",
      "       6.2098)\n",
      "5000 loss: tensor(1.00000e-02 *\n",
      "       6.0048)\n",
      "5100 loss: tensor(1.00000e-02 *\n",
      "       5.8114)\n",
      "5200 loss: tensor(1.00000e-02 *\n",
      "       5.6292)\n",
      "5300 loss: tensor(1.00000e-02 *\n",
      "       5.4568)\n",
      "5400 loss: tensor(1.00000e-02 *\n",
      "       5.2935)\n",
      "5500 loss: tensor(1.00000e-02 *\n",
      "       5.1385)\n",
      "5600 loss: tensor(1.00000e-02 *\n",
      "       4.9913)\n",
      "5700 loss: tensor(1.00000e-02 *\n",
      "       4.8515)\n",
      "5800 loss: tensor(1.00000e-02 *\n",
      "       4.7187)\n",
      "5900 loss: tensor(1.00000e-02 *\n",
      "       4.5924)\n",
      "6000 loss: tensor(1.00000e-02 *\n",
      "       4.4720)\n",
      "6100 loss: tensor(1.00000e-02 *\n",
      "       4.3571)\n",
      "6200 loss: tensor(1.00000e-02 *\n",
      "       4.2474)\n",
      "6300 loss: tensor(1.00000e-02 *\n",
      "       4.1426)\n",
      "6400 loss: tensor(1.00000e-02 *\n",
      "       4.0425)\n",
      "6500 loss: tensor(1.00000e-02 *\n",
      "       3.9467)\n",
      "6600 loss: tensor(1.00000e-02 *\n",
      "       3.8548)\n",
      "6700 loss: tensor(1.00000e-02 *\n",
      "       3.7669)\n",
      "6800 loss: tensor(1.00000e-02 *\n",
      "       3.6824)\n",
      "6900 loss: tensor(1.00000e-02 *\n",
      "       3.6017)\n",
      "7000 loss: tensor(1.00000e-02 *\n",
      "       3.5242)\n",
      "7100 loss: tensor(1.00000e-02 *\n",
      "       3.4496)\n",
      "7200 loss: tensor(1.00000e-02 *\n",
      "       3.3778)\n",
      "7300 loss: tensor(1.00000e-02 *\n",
      "       3.3089)\n",
      "7400 loss: tensor(1.00000e-02 *\n",
      "       3.2426)\n",
      "7500 loss: tensor(1.00000e-02 *\n",
      "       3.1787)\n",
      "7600 loss: tensor(1.00000e-02 *\n",
      "       3.1170)\n",
      "7700 loss: tensor(1.00000e-02 *\n",
      "       3.0574)\n",
      "7800 loss: tensor(1.00000e-02 *\n",
      "       2.9998)\n",
      "7900 loss: tensor(1.00000e-02 *\n",
      "       2.9441)\n",
      "8000 loss: tensor(1.00000e-02 *\n",
      "       2.8901)\n",
      "8100 loss: tensor(1.00000e-02 *\n",
      "       2.8380)\n",
      "8200 loss: tensor(1.00000e-02 *\n",
      "       2.7874)\n",
      "8300 loss: tensor(1.00000e-02 *\n",
      "       2.7385)\n",
      "8400 loss: tensor(1.00000e-02 *\n",
      "       2.6911)\n",
      "8500 loss: tensor(1.00000e-02 *\n",
      "       2.6453)\n",
      "8600 loss: tensor(1.00000e-02 *\n",
      "       2.6009)\n",
      "8700 loss: tensor(1.00000e-02 *\n",
      "       2.5579)\n",
      "8800 loss: tensor(1.00000e-02 *\n",
      "       2.5161)\n",
      "8900 loss: tensor(1.00000e-02 *\n",
      "       2.4754)\n",
      "9000 loss: tensor(1.00000e-02 *\n",
      "       2.4359)\n",
      "9100 loss: tensor(1.00000e-02 *\n",
      "       2.3976)\n",
      "9200 loss: tensor(1.00000e-02 *\n",
      "       2.3605)\n",
      "9300 loss: tensor(1.00000e-02 *\n",
      "       2.3242)\n",
      "9400 loss: tensor(1.00000e-02 *\n",
      "       2.2891)\n",
      "9500 loss: tensor(1.00000e-02 *\n",
      "       2.2549)\n",
      "9600 loss: tensor(1.00000e-02 *\n",
      "       2.2217)\n",
      "9700 loss: tensor(1.00000e-02 *\n",
      "       2.1893)\n",
      "9800 loss: tensor(1.00000e-02 *\n",
      "       2.1577)\n",
      "9900 loss: tensor(1.00000e-02 *\n",
      "       2.1270)\n"
     ]
    }
   ],
   "source": [
    "# 创建模型并且训练. 这里有一些函数可以在使用模型之前帮助你准备数据\n",
    "def make_context_vector(context, word_to_ix):\n",
    "    idxs = [word_to_ix[w] for w in context]\n",
    "    tensor = torch.LongTensor(idxs)\n",
    "    return autograd.Variable(tensor)\n",
    "\n",
    "model=CBOW(vocab_size,4,embedding_dim)\n",
    "loss_f=nn.NLLLoss()\n",
    "optimizer=optim.SGD(model.parameters(),lr=1e-4)\n",
    "CONTEXT_SIZE=4\n",
    "\n",
    "for epoch in range(10000):\n",
    "    loss=[]\n",
    "    for words,target in data:        \n",
    "        x=make_context_vector(words,word_to_ix)\n",
    "        y=word_to_ix[target]\n",
    "        model.zero_grad()\n",
    "        x_=model(x)\n",
    "        loss=loss_f(x_,autograd.Variable(torch.LongTensor([y])))\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "    if epoch%100==0:\n",
    "        print(epoch,'loss:',loss.data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}