zh-nbs/Lesson3_imdb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical Deep Learning for Coders, v3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson3_imdb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDB影评数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data 准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's download the dataset we are going to study. The [dataset](http://ai.stanford.edu/~amaas/data/sentiment/) has been curated by Andrew Maas et al. and contains a total of 100,000 reviews on IMDB. 25,000 of them are labelled as positive and negative for training, another 25,000 are labelled for testing (in both cases they are highly polarized). The remaning 50,000 is an additional unlabelled data (but we will find a use for it nonetheless).\n",
    "\n",
    "首先，让我们先下载需要使用的数据。 [IMDB](http://ai.stanford.edu/~amaas/data/sentiment/) 数据集由Andrew Maas等人收集，里面有10万条IMDB网站上的影评。其中2.5万条是积极的评论, 2.5万条是消极的评论, 另外2.5万条是用作测试的评论(这些数据两极分化得很厉害)，剩余的5万条是额外的未标记的数据（以后我们会将这些数据用做其他用途）。\n",
    "\n",
    "We'll begin with a sample we've prepared for you, so that things run quickly before going over the full dataset.\n",
    "\n",
    "我们一起来看一下提前准备好的样本，这样会比跑遍整个数据集快一些。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/ubuntu/notebooks/data/imdb_sample/data_clas_export.pkl'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/export_lm.pkl'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/export.pkl'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/texts.csv'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/data_lm_export.pkl'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/export_clas.pkl'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/models'),\n",
       " PosixPath('/home/ubuntu/notebooks/data/imdb_sample/save_data_clas.pkl')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It only contains one csv file, let's have a look at it.\n",
    "\n",
    "例子里只包含了一个csv文档，我们一起来看一下里面的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>negative</td>\n",
       "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>This is a extremely well-made film. The acting...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>negative</td>\n",
       "      <td>Every once in a long while a movie will come a...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>Name just says it all. I watched this movie wi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>This movie succeeds at being one of the most u...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                               text  is_valid\n",
       "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
       "1  positive  This is a extremely well-made film. The acting...     False\n",
       "2  negative  Every once in a long while a movie will come a...     False\n",
       "3  positive  Name just says it all. I watched this movie wi...     False\n",
       "4  negative  This movie succeeds at being one of the most u...     False"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(path/'texts.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is a extremely well-made film. The acting, script and camera-work are all first-rate. The music is good, too, though it is mostly early in the film, when things are still relatively cheery. There are no really superstars in the cast, though several faces will be familiar. The entire cast does an excellent job with the script.<br /><br />But it is hard to watch, because there is no good end to a situation like the one presented. It is now fashionable to blame the British for setting Hindus and Muslims against each other, and then cruelly separating them into two countries. There is some merit in this view, but it\\'s also true that no one forced Hindus and Muslims in the region to mistreat each other as they did around the time of partition. It seems more likely that the British simply saw the tensions between the religions and were clever enough to exploit them to their own ends.<br /><br />The result is that there is much cruelty and inhumanity in the situation and this is very unpleasant to remember and to see on the screen. But it is never painted as a black-and-white case. There is baseness and nobility on both sides, and also the hope for change in the younger generation.<br /><br />There is redemption of a sort, in the end, when Puro has to make a hard choice between a man who has ruined her life, but also truly loved her, and her family which has disowned her, then later come looking for her. But by that point, she has no option that is without great pain for her.<br /><br />This film carries the message that both Muslims and Hindus have their grave faults, and also that both can be dignified and caring people. The reality of partition makes that realisation all the more wrenching, since there can never be real reconciliation across the India/Pakistan border. In that sense, it is similar to \"Mr & Mrs Iyer\".<br /><br />In the end, we were glad to have seen the film, even though the resolution was heartbreaking. If the UK and US could deal with their own histories of racism with this kind of frankness, they would certainly be better off.'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It contains one line per review, with the label ('negative' or 'positive'), the text and a flag to determine if it should be part of the validation set or the training set. If we ignore this flag, we can create a DataBunch containing this data in one line of code:\n",
    "\n",
    "文档里的每一行都是一个影评，影评附有标签（“负面”或是“正面”）、评论文字以及一个标明是属于训练集还是验证集的标签，如果我们忽略这个（标明所属数据集的）标签，我们可以有下面这行代码来产生一个 DataBunch（数据堆）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_lm = TextDataBunch.from_csv(path, 'texts.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By executing this line a process was launched that took a bit of time. Let's dig a bit into it. Images could be fed (almost) directly into a model because they're just a big array of pixel values that are floats between 0 and 1. A text is composed of words, and we can't apply mathematical functions to them directly. We first have to convert them to numbers. This is done in two differents steps: tokenization and numericalization. A `TextDataBunch` does all of that behind the scenes for you.\n",
    "\n",
    "运行这行代码会启动一个需要稍微花点时间的程序，让我们来更深入地了解一下。图像本质上是一个巨大的像素值数列，这个数列由0到1 之间的数字组成，因此图像数据基本上可以直接输入到模型中。但是，一段文字是由词组成的，而我们不能直接对词运用数学函数。那么我们首先需要将这些信息转化为数字。这一过程需要通过两部完成：分词和数值化。`TextDataBunch`在幕后为您完成所有这些工作。\n",
    "\n",
    "Before we delve into the explanations, let's take the time to save the things that were calculated.\n",
    "\n",
    "在我们开始讲解内容之前，让我们先花点时间将计算好的数据存档。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_lm.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next time we launch this notebook, we can skip the cell above that took a bit of time (and that will take a lot more when you get to the full dataset) and load those results like this:\n",
    "\n",
    "下次我们启动这个notebook, 可以直接跳过之前稍费时间的单元格，直接用下面的代码载入之前保存的结果（如果你载入的是全部数据，之前这些步骤会花费更多时间）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = load_data(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization 分词"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step of processing we make the texts go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
    "\n",
    "处理数据的第一步是将文字分拆成单词, 或者更确切地说, 标准词(tokens)。最简单的方式是基于空格对句子进行分拆, 但我们能更智能地分词：\n",
    "\n",
    "- we need to take care of punctuation\n",
    "<br>我们需要考虑标点\n",
    "\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "<br>有些词是由两个不同的词缩写的，比如isn't或don't\n",
    "\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "<br>我们可能需要清理文本的某些部分，比如文字中可能会有HTML代码\n",
    "\n",
    "To see what the tokenizer had done behind the scenes, let's have a look at a few texts in a batch.<br>\n",
    "为了明白分词器幕后是如何工作的，让我们来看一下数据堆中的一些文本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj raising xxmaj victor xxmaj vargas : a xxmaj review \\n \\n  xxmaj you know , xxmaj raising xxmaj victor xxmaj vargas is like sticking your hands into a big , steaming bowl of xxunk . xxmaj it 's warm and gooey , but you 're not sure if it feels right . xxmaj try as i might , no matter how warm and gooey xxmaj raising xxmaj</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxup the xxup shop xxup around xxup the xxup corner is one of the sweetest and most feel - good romantic comedies ever made . xxmaj there 's just no getting around that , and it 's hard to actually put one 's feeling for this film into words . xxmaj it 's not one of those films that tries too hard , nor does it come up with</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj now that xxmaj che(2008 ) has finished its relatively short xxmaj australian cinema run ( extremely limited xxunk screen in xxmaj sydney , after xxunk ) , i can xxunk join both xxunk of \" xxmaj at xxmaj the xxmaj movies \" in taking xxmaj steven xxmaj soderbergh to task . \\n \\n  xxmaj it 's usually satisfying to watch a film director change his style /</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj this film sat on my xxmaj tivo for weeks before i watched it . i dreaded a self - indulgent xxunk flick about relationships gone bad . i was wrong ; this was an xxunk xxunk into the screwed - up xxunk of xxmaj new xxmaj yorkers . \\n \\n  xxmaj the format is the same as xxmaj max xxmaj xxunk ' \" xxmaj la xxmaj ronde</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj many neglect that this is n't just a classic due to the fact that it 's the first xxup 3d game , or even the first xxunk - up . xxmaj it 's also one of the first stealth games , one of the xxunk definitely the first ) truly claustrophobic games , and just a pretty well - xxunk gaming experience in general . xxmaj with graphics</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = TextClasDataBunch.from_csv(path, 'texts.csv')\n",
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols: \n",
    "\n",
    "为了更简洁易读，我们将所有评论删节到100个词。我们可以看到文字标记化算法不仅仅是基于空格和标点进行了分词：\n",
    "\n",
    "- the \"'s\" are grouped together in one token\n",
    "<br>所有“'s”都被合并为一个标准词\n",
    "\n",
    "\n",
    "- the contractions are separated like this: \"did\", \"n't\"\n",
    "<br>词语的缩写被分开，比如“did” 和 “n't”\n",
    "\n",
    "\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "<br>所有包含HTML连接的内容被清理，并且所有文字都采用小写\n",
    "\n",
    "\n",
    "- there are several special tokens (all those that begin by xx), to replace unknown tokens (see below) or to introduce different text fields (here we only have one).\n",
    "<br>为了代替未知的标准词（如下）或者引入不同的文本字段(这里我们只有一个)，（在结果中可以看到）有一些特殊的标准词（它们都以xx开头）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numericalization 数值化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at least twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token `UNK`.\n",
    "\n",
    "一旦我们从文本中完成了标准词提取，就会生成一个包含所有词汇的列表，将标准词转化成整数。这里我们只保留至少出现两次的标准词，并设置词库上限为60,000(默认设置), 同时将所有不能分进行分词的词标记为“未知标准词” `UNK`。\n",
    "\n",
    "The correspondance from ids to tokens is stored in the `vocab` attribute of our datasets, in a dictionary called `itos` (for int to string).\n",
    "\n",
    "id和标准词的关系存储在数据集的`vocab`属性中，在字典 `itos` 中（由int类型转换成string类型）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the',\n",
       " '.']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we look at what a what's in our datasets, we'll see the tokenized text as a representation:\n",
    "\n",
    "如果我们查看数据集里的“what's”的形式，我们会看到如下经过分词后的文本： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos i know that originally , this film was xxup not a box office hit , but in light of recent xxmaj hollywood releases ( most of which have been decidedly formula - ridden , plot less , pointless , \" save - the - blonde - chick - no - matter - what \" xxunk ) , xxmaj xxunk of xxmaj all xxmaj xxunk , certainly in this sorry context deserves a second opinion . xxmaj the film -- like the book -- loses xxunk in some of the historical background , but it xxunk a uniquely xxmaj american dilemma set against the uniquely horrific xxmaj american xxunk of human xxunk , and some of its tragic ( and funny , and touching ) consequences . \n",
       "\n",
       " xxmaj and worthy of xxunk out is the youthful xxmaj robert xxmaj xxunk , cast as the leading figure , xxmaj xxunk , whose xxunk xxunk is truly universal as he sets out in the beginning of his ' coming of age , ' only to be xxunk disappointed at what turns out to become his true education in the ways of the xxmaj southern plantation world of xxmaj xxunk , at the xxunk of the xxunk period . xxmaj when i saw the previews featuring the ( xxunk ) blond - xxunk xxmaj xxunk , i expected a xxunk , a xxunk , a xxunk -- i was pleasantly surprised . \n",
       "\n",
       " xxmaj xxunk xxmaj davis , xxmaj ruby xxmaj dee , the late xxmaj ben xxmaj xxunk , xxmaj xxunk xxmaj xxunk , xxmaj victoria xxmaj xxunk and even xxmaj xxunk xxmaj guy xxunk vivid imagery and formidable skill as actors in the backdrop xxunk of xxunk , voodoo , xxmaj xxunk \" xxunk , \" and xxmaj xxunk revolt woven into this tale of human passion , hate , love , family , and racial xxunk in a society which is supposedly gone and yet somehow is still with us ."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the underlying data is all numbers\n",
    "\n",
    "但实际上，底层的数据形式都是数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,   18,  146,   19, 3788,   10,   20,   31,   25,    5])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds[0][0].data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With the data block API 用data block API处理文字"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the data block API with NLP and have a lot more flexibility than what the default factory methods offer. In the previous example for instance, the data was randomly split between train and validation instead of reading the third column of the csv.\n",
    "\n",
    "活地处理各种情况。比如在之前的例子中，数据随机分为训练集和验证集，而非通过读取csv中第三列的标签来分组。\n",
    "\n",
    "With the data block API though, we have to manually call the tokenize and numericalize steps. This allows more flexibility, and if you're not using the defaults from fastai, the various arguments to pass will appear in the step they're revelant, so it'll be more readable.\n",
    "\n",
    "不过如果要使用数据块API，我们需要手动完成分词和数值化的各个步骤。这样可以更加灵活。如果你没有使用fastai工具包里的默认设置，你也可以像下面的步骤一样进行各种设置，并且代码可读性也更高。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = (TextList.from_csv(path, 'texts.csv', cols='text')\n",
    "                .split_from_df(col=2)\n",
    "                .label_from_df(cols=0)\n",
    "                .databunch())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model 语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that language models can use a lot of GPU, so you may need to decrease batchsize here.\n",
    "\n",
    "需要注意的是语言文字模型会用掉许多GPU，因此你可能会需要减小每个批次的样本容量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bs=48"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's grab the full dataset for what follows.\n",
    "\n",
    "现在我们为接下来的步骤获取完整的数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/ubuntu/.fastai/data/imdb/test'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/tmp_clas'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/README'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/unsup'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/train'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/tmp_lm'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/models'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/imdb.vocab')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB)\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/ubuntu/.fastai/data/imdb/train/neg'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/train/unsupBow.feat'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/train/pos'),\n",
       " PosixPath('/home/ubuntu/.fastai/data/imdb/train/labeledBow.feat')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reviews are in a training and test set following an imagenet structure. The only difference is that there is an `unsup` folder on top of `train` and `test` that contains the unlabelled data.\n",
    "\n",
    "现在影评遵循imagenet的结构分到了训练集和测试集中。唯一的区别是，在`测试集`和`训练集`上会有个包括未标记数据的`unsup`文件夹。\n",
    "\n",
    "We're not going to train a model that classifies the reviews from scratch. Like in computer vision, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipedia called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset)). That model has been trained to guess what the next word is, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "我们不需要从无到有地训练一个影评分类模型。就像计算机视觉模型一样，我们将使用一个在更大训练集上预训练好的模型（在维基上有一个清洗好的子集  [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset) ）。这个模型被训练来猜测下一个词是什么，它的输入数据是之前已有的词汇。该模型采用循环神经网络结构，并且有一个每次看到新词都会更新的隐层状态。 隐层状态里包含的信息，是文本中到截止这个点之前的所有句子。\n",
    "\n",
    "We are going to use that 'knowledge' of the English language to build our classifier, but first, like for computer vision, we need to fine-tune the pretrained model to our particular dataset. Because the English of the reviews left by people on IMDB isn't the same as the English of wikipedia, we'll need to adjust the parameters of our model by a little bit. Plus there might be some words that would be extremely common in the reviews dataset but would be barely present in wikipedia, and therefore might not be part of the vocabulary the model was trained on.\n",
    "\n",
    "我们用这样的预训练模型信息来创建我们的分类器。但首先，正如计算机视觉一样，我们需要对预训练的模型进行调参来适应我们的这个数据集。由于IMDB上影评的英语语言和维基百科上的英语语言风格不尽相同，我们需要将参数进行一定的调整。另外，可能会有些词在影评数据中出现的频率较高，但在维基百科上基本没出现过，因此可能和模型预训练时用的词库不太一样。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is where the unlabelled data is going to be useful to us, as we can use it to fine-tune our model. Let's create our data object with the data block API (next line takes a few minutes).\n",
    "\n",
    "我们可以用未标记的数据进行模型微调，这就是未标记数据具有价值的地方。让我们通过数据块API来建立一个数据对象。（下行会花费数分钟的时间）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train', 'test', 'unsup']) \n",
    "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "            .split_by_rand_pct(0.1)\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            .label_for_lm()           \n",
    "           #We want to do a language model so we label accordingly\n",
    "            .databunch(bs=bs))\n",
    "data_lm.save('data_lm.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to use a special kind of `TextDataBunch` for the language model, that ignores the labels (that's why we put 0 everywhere), will shuffle the texts at each epoch before concatenating them all together (only for training, we don't shuffle for the validation set) and will send batches that read that text in order with targets that are the next word in the sentence.\n",
    "\n",
    "对于语言模型，我们需要用一个特殊的`TextDataBunch`，它会忽略标签（这就是为什么我们给所有地方都设置为0的原因），在将每个轮次的文字合并在一起之前打乱所有的文字（仅限于模型训练，我们不会对验证集进行混洗），并会分批次按顺序读取文字和接下来对应的单词。\n",
    "\n",
    "The line before being a bit long, we want to load quickly the final ids by using the following cell.\n",
    "\n",
    "之前的代码会有点长，我们可以用下面的代码用id快速导入对应的文字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'data_lm.pkl', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='5%'>  <col width='95%'>  <tr>\n",
       "    <th>idx</th>\n",
       "    <th>text</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>original script that xxmaj david xxmaj dhawan has worked on . xxmaj this one was a complete bit y bit rip off xxmaj hitch . i have nothing against remakes as such , but this one is just so lousy that it makes you even hate the original one ( which was pretty decent ) . i fail to understand what actors like xxmaj salman and xxmaj govinda saw in</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>' classic ' xxmaj the xxmaj big xxmaj doll xxmaj house ' , which takes xxmaj awful to a whole new level . i can heartily recommend these two xxunk as a double - bill . xxmaj you 'll laugh yourself silly . xxbos xxmaj this movie is a pure disaster , the story is stupid and the editing is the worst i have seen , it confuses you incredibly</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>of xxmaj european cinema 's most quietly disturbing sociopaths and one of the most memorable finales of all time ( shamelessly stolen by xxmaj tarantino for xxmaj kill xxmaj bill xxmaj volume xxmaj two ) , but it has plenty more to offer than that . xxmaj playing around with chronology and inverting the usual clichés of standard ' lady vanishes ' plots , it also offers superb characterisation and</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>but even xxmaj martin xxmaj short managed a distinct , supporting character . ) \\n\\n i can understand the attraction of an imaginary world created in a good romantic comedy . xxmaj but this film is the prozac version of an imaginary world . i 'm frightened to consider that anyone could enjoy it even as pure fantasy . xxbos movie i have ever seen . xxmaj actually i find</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>xxmaj pre - xxmaj code film . xxbos xxmaj here 's a decidedly average xxmaj italian post apocalyptic take on the hunting / killing humans for sport theme ala xxmaj the xxmaj most xxmaj dangerous xxmaj game , xxmaj turkey xxmaj shoot , xxmaj gymkata and xxmaj the xxmaj running xxmaj man . \\n\\n xxmaj certainly the film reviewed here is nowhere near as much fun as the other listed</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file).\n",
    "\n",
    "我们可以很轻易地将模型和预训练的权重结合为一个学习器对象。在你第一次运行下面的代码时，所有模型的信息会下载并存储到`~/.fastai/models/` 或者其他由你的config文件指定的地方。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8lfXZ+PHPlU0WARJGCBtZsglLFPdCRdFaFxYcRa1tqdZRH/1Vq4+j1T51VS21qIibqlWLoCJoRRESRth7hQAJCQkkZOf6/XHuYIoJCXDus3K9X6/z8j73OPf19YRc+c5bVBVjjDHG28L8HYAxxpjQZAnGGGOMKyzBGGOMcYUlGGOMMa6wBGOMMcYVlmCMMca4whKMMcYYV1iCMcYY4wpLMMYYY1wR4e8AvCU5OVm7du3q7zCMMSaoZGZm7lPVFDc+29UEIyLbgINANVClqulHHBfgGWAccAiYrKpLnWOTgAecU/9XVV872r26du1KRkaGdwtgjDEhTkS2u/XZvqjBnKmq+xo4diFwkvMaCbwIjBSR1sCDQDqgQKaIfKSq+30QrzHGGC/wdx/MpcAM9VgEJIlIB+B84HNVLXCSyufABf4M1BhjzLFxO8Eo8JmIZIrIlHqOdwR21nmf7exraP9/EZEpIpIhIhl5eXleDNsYY8yJcjvBjFHVoXiawm4XkbFHHJd6rtGj7P/vHarTVDVdVdNTUlzpozLGGHOcXE0wqprj/DcX+AAYccQp2UCnOu/TgJyj7DfGGBMkXEswIhInIgm128B5wKojTvsI+Jl4jAKKVHU3MBc4T0RaiUgr59q5bsVqjDHG+9wcRdYO+MAzEpkI4E1VnSMitwKo6kvAbDxDlDfhGaZ8g3OsQEQeAZY4n/Wwqha4GKsxxhgvk1B5ZHJ6erraPBhjTHMzKzObyuoarhnR+biuF5HMI+coeou/hykbY4w5AbMyd/L+0mx/h1EvSzDGGBPECkoqaB0X5e8w6mUJxhhjglhBSQVt4qP9HUa9LMEYY0yQqqlRT4KxGowxxhhvKiytpEaxJjJjjDHeVVBSDmBNZMYYY7wrv7gCwJrIjDHGeFd+iSfBWBOZMcYYr6pNMFaDMcYY41UFThNZK0swxhhjvKmgpJyWLSKJDA/MX+WBGZUxxphG7QvgOTBgCcYYY4JWQXEFbeItwRhjjPGyQF6HDCzBGGNM0MovKad1XGBOsgRLMMYYE5RqapT9hypJtiaywHagrJKKqhp/h2GMMU1WVFpJdY1aE1kg27qvhDGPf8mHy3f5OxRjjGmyfGcdMkswAaxrm1jSWsfy0lebqakJjcdHG2NC3w/rkDXjPhgRCReRZSLyST3HuojIPBHJEpEFIpJW59ifRGS1iKwVkWdFRFyKj9vO6MGWvBI+W7PHjVsYY4zXFdQuE9PM+2CmAmsbOPYUMENVBwIPA48DiMgpwBhgINAfGA6c7laA4/q3p0ubWF5YsBlVq8UYYwJfoK9DBi4nGKdGchHwcgOn9APmOdvzgUudbQVigCggGogE9roVZ0R4GLeM7UFWdhELN+W7dRtjjPGa/ABfhwzcr8E8DdwDNDREawVwhbM9AUgQkTaq+h2ehLPbec1V1R/VgkRkiohkiEhGXl7eCQV6+dCOpCRE88KCTSf0OcYY4wuBvg4ZuJhgRORiIFdVM49y2l3A6SKyDE8T2C6gSkR6An2BNKAjcJaIjD3yYlWdpqrpqpqekpJyQvHGRIZz86nd+HZzPst3Fp7QZxljjNvyA3wdMnC3BjMGGC8i24C38SSJmXVPUNUcVb1cVYcA9zv7ivDUZhaparGqFgOfAqNcjBWA60Z1ITEmghetFmOMCXD5xYG9TAy4mGBU9T5VTVPVrsDVwJeqOrHuOSKSLCK1MdwHTHe2d+Cp2USISCSe2k1DAwW8Jj46gkmndGXu6r0s2pJPtQ1bNsYEqEBfhwwgwtc3FJGHgQxV/Qg4A3hcRBT4GrjdOW0WcBawEk+H/xxV/dgX8U0+pSuvLtzG1dMWERMZRu92CfRLTeTk1JYMSkuid/sEoiICt83TGNM85JdUMLRLK3+HcVQ+STCqugBY4Gz/vs7+WXiSyZHnVwO3+CK2I7WJj2bOHWNZtDmfNbsPsHb3AT5dtYe3Fu8EICoijH4dEpl0ShcmDElr5NOMMcb7POuQBX4fjM9rMMGgY1ILrhiWdnh4m6qSvb+UFdmFrNhZyFcb8rhnVhYD05LokRLv11iNMc1PMKxDBrZUTJOICJ1ax3LxwFTuv6gfb9w8ipjIcB7812qbmGmM8bn8IJjFD5ZgjktKQjR3n9+bbzbt498rd/s7HGNMM3N4mZgAXocMLMEct+tGduHk1EQe+WQNxeVV/g7HGNOM5BcH/krKYAnmuIWHCY9c1p+9B8p5dt5Gf4djjGlGrImsGRjauRVXpXdi+jdb2bD3oL/DMcY0E7VNZK1iLcGEtHsv7ENcdAT3/jOLQxXWVGaMcV9BSQWJMREBPycvsKMLAq3jonj88gGs2FnIz/6xmANllf4OyRgT4vYVl9MmPrA7+MESjFeMG9CB568dyvKdhUx8+XsKD1X4OyRjTAgrCIKFLsESjNeMG9CBv10/jHW7D3L1tEXsc0Z5GGOMtwXDOmRgCcarzu7bjn9MTmdbfgnXTFtkzWXGGFfsK64I+BFkYAnG6047KYXpk4azZV8Jv313BTW2IrMxxotq1yGzGkwzdUrPZO4f15fP1+y1J2QaY7zqQJlnHbJAn8UPlmBcc8OYrlw6OJU/f76BBetz/R2OMSZE7CsOjkmWYAnGNSLCE5cPpHe7BKa+vZwd+Yf8HZIxJgTUTrK0JrJmrkVUONOuT0dVmfJ6BvtLbPiyMebEFJR4RqhaE5mhc5tYnrt2KFvySrjipW/ZWWA1GWPM8QuWdcjAEoxPnN4rhddvGsG+g+VMeOFbVmYX+TskY0yQyi8OjnXIwAcJRkTCRWSZiHxSz7EuIjJPRLJEZIGIpNU51llEPhORtSKyRkS6uh2rm0Z2b8P7vziF6Igwrpr2HfOt498YcxwKSipICIJ1yMA3NZipwNoGjj0FzFDVgcDDwON1js0AnlTVvsAIIOh/I/dsm8AHvziF7ilx3PjqEn760nf845utZO+3ZjNjTNPkl1SQHATrkIHLCcapkVwEvNzAKf2Aec72fOBS57p+QISqfg6gqsWqGhK/hdsmxvD2lNH85uxeHCir5JFP1nDqH+cz/vlvWLfngL/DM8YEuPzi8qAYQQbu12CeBu4Baho4vgK4wtmeACSISBugF1AoIu87zWtPiki4y7H6THx0BFPPOYk5vxnLgrvO4L4L+7CnqIzJ05eQU1jq7/CMMQEsWNYhAxcTjIhcDOSqauZRTrsLOF1ElgGnA7uAKiACOM05PhzoDkyu5x5TRCRDRDLy8vK8XALf6Jocxy2n92DGTSMoKa9i8iuLKSq1NcyMMfXLO1hOSoI1kY0BxovINuBt4CwRmVn3BFXNUdXLVXUIcL+zrwjIBpap6hZVrQI+BIYeeQNVnaaq6aqanpKS4mJR3NenfSJ/u34YW/eVMGVGBuVV1f4OyRgTYCqra8gvqaBtc08wqnqfqqapalfgauBLVZ1Y9xwRSRaR2hjuA6Y720uAViJSmzXOAta4FWugOKVnMk9dOYjvtxbYQpnGmB+pHaJsNZgGiMjDIjLeeXsGsF5ENgDtgEcBVLUaT/PYPBFZCQjwd1/H6g+XDu7I7y7swydZu/n7f7b4OxxjTADJPVgGQNuEGD9H0jQRvriJqi4AFjjbv6+zfxYwq4FrPgcG+iC8gHPL2O5kbCvg2XkbuWxIR9olBscPkzHGXXkHPcvENPsmMnP8RIQHLupHZbXyx0/X+TscY0yAyHUSjDWRmRPSNTmOm0/rxvvLdpG5vcDf4RhjAkDuAU+CsYmW5oTdfmZP2iVG89BHa6i2Dn9jmr284jJaxUYGxTIxYAkmoMVFR/A/4/qyclcR72Xs9Hc4xhg/yz1QHjQd/GAJJuCNH5RKepdW/GnuepuAaUwzl3uwnLaJwdE8BpZgAp6I8ND4k9l/qILn5m30dzjGGD/KO1hOSpD0v4AlmKDQv2NLfjI0jRnfbbeVl41pplTVk2CsBmO87Y5ze4HA019YLcaY5qiotJKK6hqrwRjvS01qwaTRXXh/aTYb9h70dzjGGB87PMkyiCZeW4IJIr84oydxURH8ac56f4dijPGx3CCbxQ+WYIJKq7gobj2jB1+s3UvGNpt8aUxzkhdks/jBEkzQuWFMV1ISovnjnHWo2uRLY5qLHxa6tARjXBIbFcGvzz6JJdv2M29trr/DMcb4SO6BclpEhhMf7ZM1ir3CEkwQunp4J7qnxHHHO8v5ZuM+f4djjPGBvGLPkyxFxN+hNJklmCAUGR7GzJtG0rFVCya/sph3l9gyMsaEOs8yMcHTPAaWYIJWalIL3rt1NKN7tOGef2bx1Nz11idjTAjLPVgWVB38YAkmqCXERDJ98nCuHt6J5+dv4pdvLuNgma1XZkwoyjtoNRjjY5HhYTx++QDuu7APc1bv4eLnviEru9DfYRljvKisspoDZVVBNckSLMGEBBHhltN78PaUUVRW1XDFi9/yj2+2WpOZMSHi8ByYIFomBnyQYEQkXESWicgn9RzrIiLzRCRLRBaISNoRxxNFZJeIPO92nKFgeNfWzJ56Gqf3assjn6zhptcyyD1Q5u+wjDEn6PCjkoNooUvwTQ1mKrC2gWNPATNUdSDwMPD4EccfAb5yMbaQkxQbxd9/NoyHLunHwk37OPcvX/Phsl1WmzEmiOU5kyytBlOHUyO5CHi5gVP6AfOc7fnApXWuHQa0Az5zM8ZQJCJMHtON2VNPo0dKHL95ZzlTXs88PBPYGBNcfljo0hJMXU8D9wA1DRxfAVzhbE8AEkSkjYiEAX8G7nY5vpDWIyWe9249hfvH9eXrDXlc9vxCqqob+iqMMYEq92A5YQJt4izBACAiFwO5qpp5lNPuAk4XkWXA6cAuoAr4BTBbVY86g1BEpohIhohk5OXleSv0kBIeJvx8bHcenTCAnKIyNuwt9ndIxphjlHugnDbx0YSHBc8sfgA3F7UZA4wXkXFADJAoIjNVdWLtCaqaA1wOICLxwBWqWiQio4HTROQXQDwQJSLFqvq7ujdQ1WnANID09HTrZDiK4V1bAbAiu5B+qYl+jsYYcyzyioNvDgy4WINR1ftUNU1VuwJXA1/WTS4AIpLsNIcB3AdMd669TlU7O9fehWcgwH8lF3NsOreOJSk2kuU7bI6MMcEmGGfxgx/mwYjIwyIy3nl7BrBeRDbg6dB/1NfxNBciwqC0JFbYJExjgk4wzuIHd5vIDlPVBcACZ/v3dfbPAmY1cu2rwKuuBdeMDO6UxHNfbqSkvIq4IFry25jmrLpG2VdcQduE4JrFDzaTv1kZ3CmJGoWVu4r8HYoxpokKSiqorlFrIjOBbVCnJABW7LRmMmOCxeE5MJZgTCBrHRdF59axLLcEY0zQOPyo5CCbZAmWYJqdQZ2SrAZjTBA5vA5ZvPXBmAA3uFMSOUVltgimMUHi8ErK1kRmAt3gTi0BrJnMmCCRd7CchOgIWkSF+zuUY2YJppk5ObUlEWFiCcaYIJF3sDzolumv1aQEIyI9RCTa2T5DRH4tIknuhmbcEBMZTp8OCTbh0pggkXuwLChHkEHTazD/BKpFpCfwD6Ab8KZrURlXDUpLImtnETU1tnybMYEu92A5KUE4yRKanmBqVLUKz5L6T6vqHUAH98IybhrcKYmD5VVs2XdsKyuv3X2AQxVVLkVljDmSqpJ7IDiXiYGmJ5hKEbkGmATUPvo40p2QjNsGOxMul+9s2oz+quoa/veTNVz4zH849/++5vM1e90MzxjjKCqtpLSymg4tQ7sGcwMwGnhUVbeKSDdgpnthGTd1T4knPjqC5Tv3N3ruvuJyJv7je17+Zis/GZZGfHQEP5+RwZQZGeQUlvogWmOar5xCz3SC1KQWfo7k+DRpxUNVXQP8GkBEWgEJqvqEm4EZ94SHCQPTWrLC6YfZsq+YFTuL2JxXTJv4aNJataBjUgsOllVxxzvL2X+ogv/76SAuH5pGZXUNL/9nK8/M28C5//cVT1wxkEsGpfq7SMaEpNo/4oK1BtOkBCMiC4DxzvnLgTwR+UpV73QxNuOiQZ2S+NtXmxn0h884WO7pVwkTOLLfP61VC/552yn07+iZPxMZHsZtZ/Tg4oEduPPd5dzxznLioyM4s0/bo96vpkb598rd5B4s54ZTuhIWZE/mM8Yfdhd5EkzHUK7BAC1V9YCI3Ay8oqoPikiWm4EZd108sANLt++nZ9t4BnVKYnCnJHqkxFNUWsmu/aVk7z9EYWklF/ZvT1Js1I+u79Q6llduGME10xZx2xuZvHHzSIZ1aV3vvb7fks9js9eyItvT55OxrYC/XDWYmMjgmzhmjC/lFJURGS4kxwdnJ39TE0yEiHQAfgrc72I8xkdOTm3JO7eM/tH+1nFRtI6LYkBay0Y/Iz46glduGM6VL33HDa8s4b1bT6F3+wTA8wyLrOxCXliwmc/X7KVDyxj+fOUg9h+q4NHZa9k9bREvT0oP2n84xvhCTmEp7RJjgrbG39QE8zAwF1ioqktEpDuw0b2wTLBIjo9mxo0juOLFb/nZ9O+5ZWwPFm8t4NvN+zhQVkV8dAR3n9+bm07tdrjGktYqlt+8s4wJLyzklckj6Nk23s+lMCYw7S4sI7VlcDaPAYhqaEy2S09P14yMDH+H0Wyt23OAn770HQfKquiY1IJTeyYz5qRkxp6UXG8T2/Kdhdz82hKqa5Qv7jydNlaTMeZHTv3jl6R3acXTVw9x7R4ikqmq6W58dlM7+dOA54AxgALfAFNVNduNoEzw6dM+kTm/GUtFVQ1d2sQicvQq/eBOSbz581GMe+Y/PDl3PU9cMdBHkRoTHKprlL0HyugQpB380PR5MK8AHwGpQEfgY2dfo0QkXESWicgn9RzrIiLzRCRLRBY4iQwRGSwi34nIaufYVU2M0/hRalILuibHNZpcavVql8DkU7ryTsZOsmxtNGP+y77iciqrldQgHaIMTU8wKar6iqpWOa9XgZQmXjsVWNvAsaeAGao6EE8/z+PO/kPAz1T1ZOAC4GlbXDM0TT3nJNrERfPgR6ttbTRj6qidAxOskyyh6Qlmn4hMdGoj4SIyEchv7CKnRnIR8HIDp/QD5jnb84FLAVR1g6pudLZzgFyantBMEEmIieR3F/Zh2Y5C3l+2y9/hGBMwdhd5ZvF3COJO/qYmmBvxDFHeA+wGfoJn+ZjGPA3cA9Q0cHwFcIWzPQFIEJE2dU8QkRFAFLC5ibGaIHP5kI4M6ZzEE5+u40BZpb/DMSYg/FCDCfEmMlXdoarjVTVFVduq6mXA5Ue7RkQuBnJVNfMop90FnC4iy4DTgV3A4eV6nbk3rwM3qOqPkpSITBGRDBHJyMvLa0pRTAAKCxP+MP5k8kvK+cvnG8g7WE5OYSnb9pWwYe9BVmYXkbGtgIWb9vHtpn1UVjf090poKi6vYsm2Asoqq/0divGhnMIyWkSG07JF8K4r3NR5MPW5E08NpSFjgPEiMg6IARJFZKaqTqw9wWn+uhxAROKBK1S1yHmfCPwbeEBVF9V3A1WdBkwDzzDlEyiL8bOBaUlcld6JVxZu45WF24567jl92/HCdUOJigj9B7LuyD/Eja8tYVNuMQkxEYzr34FLB6cysnsbBNh/qIJ9xRWUVFQxKC2J8CCdkGd+bHdRKalJMU0eNBOITiTBHLXUqnofcB94noIJ3FU3uTj7k4ECp3ZyHzDd2R8FfIBnAMB7JxCjCSK/v6QfA9OSqK6pITI8jKgIzys6IpzoiDCiI8JYtrOQJz5dxy/eyOSv1w0lOiJ0l5tZsq2AW17PpLpGeXRCfzK37+eTrBzeydhJXFQ4ZVU1VNcZGHFS23juPr835/ZrF9S/lIxHTlFZUHfww4klmOOqMYjIw0CGqn4EnAE8LiIKfA3c7pz2U2As0EZEJjv7Jqvq8hOI1wS42KgIrh3Z+ajnjOzehrjoCP7fh6u4beZSXpwYmknmg2XZ3DtrJR1bteAfk9LpnhLPdSO7UHpZNfPW7eX7LQW0bBFJcnwUyQnRlFXW8ML8TUx5PZOhnZO494I+jOzepvEbmYCVU1hK797BPbbpqDP5ReQg9ScSAVqo6okkKK+ymfzNy8xF23ngw1Wc2TuFFycOC5mFM1WVZ+dt4i9fbGBkt9b87fph9a6EUJ+q6hpmZWbz9Bcb2XOgjKeuHMRPhqW5HLFxQ0VVDb3/36f8+qyTuOPcXq7ey82Z/EdtxFbVBFVNrOeVEEjJxTQ/E0d14bEJA5i/Po8Lnv6aOat2E+zLHtXUKA99tJq/fLGBy4d25PWbRjY5uQBEhIdx9YjOLLj7DIZ2TuJPc9bZI66D1N4DZagG7zL9tUK/l9SErGtHdmbGjSOIDA/j1plLuepvi1ixMzhXBKisruGOd5fz2nfbuenUbjz1k0HHPYghJjKc/xnXl9yD5Y0OmDCB6fCDxoJ4iDJYgjFBbmyvFD6dehqPTujPln3FXPrXhby+aLu/wzompRXVTJmRwb+W53D3+b154KK+J7w8e3rX1pzbrx0vLdhMQUmFlyI1vpJTVPsky+CuwVgzlwl6EeFhXDeyC+MHpTJp+mL+/vUWJo7sHLAjqT5akcPc1XvYXVjK7qIycg+WU6OekWLXjezitfvcc35vzn/6a577ciMPXnKy1z7XuC+n0DOLP5gnWYLVYEwISYiJ5NqRXdhRcIilOwKzqeyjFTn8+q1lLN2+n+iIcEb3aMNtp/fgjZtHejW5AJzULoGfpndi5qLt7Mg/5NXPNu7aXVRKUmwksVHBXQcI7uiNOcL5J7fjgQ/D+HDZLoZ1aeXvcP7Loi353PXuCkZ0bc2Mm0b4ZOTbb87pxYfLd/Hnz9fzjIvPFDHelVNYFvTNY2A1GBNiEmIiOadvOz7JyvHKkjIl5VXMWbWbiqoT+6yNew8yZUYGnVq3YNrPfDesun3LGG4c041/Lc8hY1uBT+5pTlxOYWlQL9NfyxKMCTkThnRk/6FKvt5wYuvTVdcov3xzKbfOXMqlf13I6pyi4/qc3ANlTH5lCVER4bx6w4hjGnrsDbee0YN2idFcPW0Rf5yzjtKKH9Y0U1W+35LPlBkZ3PnucnIPlPk0NlO/3SEwix+sicyEoLG9UmgVG8kHy3Zxdt92x/05j89ey/z1eVw/qgufrtrDpc8v5PYze3L7mT2POoS4rLKa1TlFrNhZxIrsQr7bnE9xeRXvTBlNp9axxx3P8UqMieTTqWN5bPZaXlywmY9X5PDwpSdTXQMvLtjE0h2FtImL4mB5FZ+v2cvvLuzDNcM7n/BINnN8SsqrKCqtDPohymAJxoSgyPAwLhmUyjtLdnKwrJKEmGNfjfbdJTt5+ZutTBrdhT9c2p/fnteLP3y8hmfmbWTu6j08f+1QeraN/9F1S7YVMGVGBvsPeR470D4xhsGdkrjp1G4MSGt5wmU7Xq3jog7P7H/gw1Xc+Kpn1YtOrVvwyGX9uXJYGruLyvif91dy/wer+GDpLp64YgA92yb4LebmarczRDk1BPpgjrpUTDCxpWJMXUt37OfyF77lyZ8M5Mr0Tsd07eKtBVz38iJGdmvDqzcMJyL8h9rKF2v2cu8/s6ioquGZawZzVp8fakhzV+/h128tIzWpBfde0IfBnZJoH4Dt6BVVNbyXuZOEmEjG9W//X+VTVWZlZvPo7LVUVtXw4sRhjO0V3OthBZuvN+Txs+mLefeW0Yzo1tr1+/ltqRhjgtWQTkl0aRPLh8uP7SmZy3bs59aZmXRqFctfrx36X798Ac7p146PfnUqXZJjuem1DP46fxOqysxF27ltZiZ9OiQy69bRXNC/fUAmF4CoiB/mDR1ZPhHhyvROzP3NWDq3iePGV5fw/tJsP0XaPO0+PMkyMH9+joU1kZmQJCJcOrgjz325kT1FZUf9ZV9eVc3slbt57dvtLN9ZSKvYSF6elE7L2Pqb1jomteC9W07hd+9n8eTc9cxeuZvVOQc4s3cKf71uaNDPXQBolxjDO7eM4tbXM7nz3RXsPVDOrad3D9jJq6FkV2EZIgTsHyjHIvj/JRjTgMsGp/LsvI08++VGfpreiR4pcSTERKKqbMs/ROb2/WRuL+Cz1XvJL6mge3IcD13Sj8uHpZHYSL9Ni6hwnr5qMCenJvLEp+u4clgaj10+gMjw0GkUSIyJ5JUbhnPXe1n8cc46CkrKuf+ifv4OK+TtLiwlJT46JH6WLMGYkNU9JZ6z+rTlze938Ob3OwBolxhNdY2yr9izPldCTARjeiRz3ajOjOmRfEwjp0SEKWN7cNXwziTGRITkX/fREeE8c9VgEmMi+Pt/tnLeye0Z3tX9foHmLFSGKIMlGBPi/v6zdLbnl7Apt5hNecVsyi1GEIZ1acWwLq04qW38CQ/HDeZnpjdFWJjwwEX9+GLtXh6bvZb3bzslJJNpoMgpKqVP+9AYvWcJxoS08DChe0o83VPiOc/fwQSxFlHh3HluL+7950rmrt7DBf07+DukkKSq5BSWcmbvtv4OxSuCv5HPGOMTVwxN46S28fxpznqvLMNjfiy/pIKyypqQaSKzBGOMaZKI8DDuvaAPW/aV8M6Snf4OJyRtySsBoEdKnJ8j8Q7XE4yIhIvIMhH5pJ5jXURknohkicgCEUmrc2ySiGx0XpPcjtMY07iz+7ZlRNfWPP3FRkrK7XHM3rYptxiAHik/XiUiGPmiBjMVWNvAsaeAGao6EHgYeBxARFoDDwIjgRHAgyISWGuvG9MMiQi/G9eHfcXl/P0/W/wdTsjZnFdMTGQYHUOkiczVTn6nRnIR8ChwZz2n9APucLbnAx862+cDn6tqgfM5nwMXAG+5Ga8xpnFDO7fiwv7tee7LTcxeuZuT2ibQo208Azu25Oy+bW2E2QnYlFtM9+QTH9kYKNweRfY0cA/Q0Ji7FcAVwDPABCBBRNoAHYG6jbzZzj5jTAB4dMIAuiXHsWHvQVblFDF71W5U4Zax3fndhX0syRwKMbnDAAAVH0lEQVSnzXnFDO0cOo01riUYEbkYyFXVTBE5o4HT7gKeF5HJwNfALqAKqO+n80ercorIFGAKQOfOnb0QtTGmKVrHRXHPBX0Ovy+rrObRf6/lb19vIbFFJLef2dOP0QWn0opqdhWWcuWwY1ucNZC5WYMZA4wXkXFADJAoIjNVdWLtCaqaA1wOICLxwBWqWiQi2cAZdT4rDVhw5A1UdRowDTyrKbtUDmNMI2Iiw/nD+JM5WFbJk3PXk9gikutHdfF3WEFly75iVKn3MRDByrVOflW9T1XTVLUrcDXwZd3kAiAiySJSG8N9wHRney5wnoi0cjr3z3P2GWMCVFiY8OSVgzinb1t+/69VfLjs2Faybu4OjyBrGxpDlMEP82BE5GERGe+8PQNYLyIbgHZ4BgPgdO4/AixxXg/XdvgbYwJXZHgYz187lJHdWvPb91Ywf12uv0MKGpvzSggT6NomdBKMPXDMGON1xeVVXD3tOzbnlvDOLaMYmJbk75AC3u1vLmXVriK+uvtMn97XHjhmjAkq8dERTJ88nDbxUdz46hK255f4O6SAtzm3mJ4hMsGyliUYY4wr2ibE8NqNI6iqUSZNX0x+cbm/QwpY1TXKln0l9AihDn6wBGOMcVGPlHj+MSmd3UVl3PRaBocqbHmZ+mTvP0RFVU3IrEFWyxKMMcZVw7q05tlrhpCVXcjNlmTqtTnPM4IslIYogyUYY4wPnH9ye/7800Es2pLPja8usSRzhFBb5LKWJRhjjE9MGJLGX64azOKtBUyevsRWY65jc24JyfFRJMVG+TsUr7IEY4zxmUsHd+SZq4eQuWM/k6YvptiSDACb8orpHmK1F7AEY4zxsUsGpfLcNUNYtrOQJ+es83c4fqeqbMotDrn+F7AEY4zxg3EDOnDpoFT+uXRXs6/F5JdUUFRaGXL9L2AJxhjjJ9eP7kJxeRUfNPM1yzbnhuYIMrAEY4zxk8GdkujfMZHXv9tGqCxZdTw25dWOIAutOTBgCcYY4yciws9GdWXD3mIWb22+a9luzi2hRWQ4qS1D4zHJdVmCMcb4zSWDUmnZIpIZi7b7OxS/2ZxXTPeUuJB5THJdlmCMMX7TIiqcK4elMXfVHnIPlPk7HL8I1RFkYAnGGONn143qQlWN8tbinf4OxedqH5MciiPIwBKMMcbPuiXHMbZXCm8u3k5ldY2/w/GpTSE8ggwswRhjAsD1o7qw90A5n6/Z6+9QfGrlriIA+qe29HMk7rAEY4zxu7P6tKVLm1junZXFpyt3+zscn1m5q5CWLSLp1Dr0RpCBJRhjTAAIDxPeuHkk3dvGc9sbS/nDx6upqAr95rKs7CIGprVEJPRGkIEPEoyIhIvIMhH5pJ5jnUVkvnM8S0TGOfsjReQ1EVkpImtF5D634zTG+Fdaq1jeu2U0k0/pyisLt/HTv33HrsJSf4flmrLKatbvOciAjqHZPAa+qcFMBdY2cOwB4F1VHQJcDbzg7L8SiFbVAcAw4BYR6epynMYYP4uKCOOh8SfzwnVD2ZRbzC9mZvo7JNes23OQqhplYJolmOMiImnARcDLDZyiQKKz3RLIqbM/TkQigBZABXDAxVCNMQFk3IAO3H1+b1ZkF7Eyu8jf4bjicAe/1WCO29PAPUBDjakPARNFJBuYDfzK2T8LKAF2AzuAp1S1+a4lYUwzdNmQjsREhvHWkh3+DsUVK7MLaR0XRcek0OzgBxcTjIhcDOSq6tHquNcAr6pqGjAOeF1EwoARQDWQCnQDfisi3eu5xxQRyRCRjLy8PO8XwhjjNy1bRHLxwFT+tSw0l/TPyi5iQMfQ7eAHd2swY4DxIrINeBs4S0RmHnHOTcC7AKr6HRADJAPXAnNUtVJVc4GFQPqRN1DVaaqarqrpKSkp7pXEGOMX14zoTElFNR+vyGn85CBSVlnNxtzikO5/ARcTjKrep6ppqtoVTwf+l6o68YjTdgBnA4hIXzwJJs/Zf5Z4xAGjAHv0nTHNzNDOSfRpn8Bbi0OrmWzN7gNU12hI97+AH+bBiMjDIjLeeftb4OcisgJ4C5isngdD/BWIB1YBS4BXVDXL17EaY/xLRLhmRGeysotYtSt0OvtrBy6Eeg0mwhc3UdUFwAJn+/d19q/B05R25PnFeIYqG2OaucuGdOSx2Wt5c/EOHpswwN/heEVWdhHJ8dG0T4zxdyiuspn8xpiAVrezvyREOvtX7ioM6Rn8tSzBGGMC3rUjQ6ez/1BFFZtyi0O+/wUswRhjgsDQzkn0bpfA9IVbKaus9nc4J2RNzgFqFAZagjHGGP8TEe4+vzcb9hbz0Eer/R3OCclyOvgHhHgHP1iCMcYEiXP6teOXZ/bk7SU7g3rY8spdRbRLjKZdiHfwgyUYY0wQuePcXoztlcKD/1rN8p2F/g7nuGRlF4b0Csp1WYIxxgSN8DDhmasG0zYxmttmZrKvuNzfIR2T4vIqtuwrYUDHJH+H4hOWYIwxQaVVXBQvTRxGQUkFv5i5lKJDlf4OqclW7SpCFQakJTZ+cgiwBGOMCTr9O7bkySsHsWznfi55/htW5wTHLP8lWwsQgaGdW/k7FJ+wBGOMCUrjB6Xy9pTRVFTVcPkL3/Jexk5/h9SoRVvz6d0ugaTYKH+H4hOWYIwxQWtYl1Z88utTGdalFXfPyuJ/PlhJTY36O6x6VVTVkLl9P6O6t/F3KD5jCcYYE9SS46N5/aaR/Py0brz5/Q4+zgrM2f5Z2YWUVdYwqntrf4fiM5ZgjDFBLzxMuO/Cvpycmsif5qwPyNn+32/1PJR3RDerwRhjTFAJCxPuv6gvuwpLmb5wq7/D+ZFFWzz9L63jmkf/C1iCMcaEkFN6JHNO37a8MH9zQM2RqayuIWPb/mbVPAaWYIwxIea+cX0pq6zmL59v8Hcoh2VlF1FaWc3IZtTBD5ZgjDEhpkdKPNeN7Mxbi3ewce9Bf4cDwPdb8wEY0c1qMMYYE9SmntOLuOgIHpu91t+hALBoSwEntY0nOT7a36H4lCUYY0zIaR0XxS/P7Mn89XlMfXuZX/tjKqtryNxW0Kzmv9RyPcGISLiILBORT+o51llE5jvHs0RkXJ1jA0XkOxFZLSIrRST017Y2xnjNTad2Y+rZJ/Hpyj2c/eeveDdjJ6q+n4S5alcRJRXVjGxmHfzgmxrMVKCheuoDwLuqOgS4GngBQEQigJnArap6MnAGEDwr2hlj/C4iPIw7zu3F7Kmn0qtdPPfMyuKavy9iZ8Ehn8ZRO/9lZDOa/1LL1QQjImnARcDLDZyiQO2yoi2B2im45wFZqroCQFXzVTXwZk4ZYwJez7YJvDNlNI9fPoDVuw4w7pn/8PEK3832X7Qlnx4pcaQkNK/+F3C/BvM0cA9Q08Dxh4CJIpINzAZ+5ezvBaiIzBWRpSJyj8txGmNCWFiYcM2Izsyeeho928Xzq7eWce+sLA5VVLl636rD81+aX+0FXEwwInIxkKuqmUc57RrgVVVNA8YBr4tIGBABnApc5/x3goicXc89pohIhohk5OXleb8QxpiQ0ql1LO/eMprbz+zBu5k7ueS5b9iSV+za/VbnHKC4vKrZzX+p5WYNZgwwXkS2AW8DZ4nIzCPOuQl4F0BVvwNigGQgG/hKVfep6iE8tZuhR95AVaeparqqpqekpLhXEmNMyIgMD+Pu8/sw86aRFB6q5Pp/LGZPUZnX71NTozwzbyMRYdLsZvDXci3BqOp9qpqmql3xdOB/qaoTjzhtB3A2gIj0xZNg8oC5wEARiXU6/E8H1rgVqzGm+RnTM5nXbhxB4aEKJk1fTFGpd8cRPf3FBr5cl8vvL+lH24TmOQjW5/NgRORhERnvvP0t8HMRWQG8BUxWj/3A/wFLgOXAUlX9t69jNcaEtv4dW/K369PZsq+Yn7+W4bVVmOeu3sOzX27iymFpXD+qi1c+MxiJP8aFuyE9PV0zMjL8HYYxJgh9vCKHX7+9jHP7tuPFicMID5Pj/qxNuQe57K/f0j0ljndvGU1MZLgXI/U+EclU1XQ3PjvCjQ81xphgcsmgVPYVl/OHj9dw1p8XcHafdpzVpy0jurUmKqLpDT2FhyqY8nom0RFhvDRxWMAnF7dZgjHGGOCGMd1oFRvFB8t2MfP77UxfuJW4qHC6pcQRGxlBi6hwYqPCad8yhv6pLenfsSU9UuIor6rhy3W5/DtrN/PX51JVo7xx80hSk1r4u0h+Z01kxhhzhEMVVSzclM+C9bnkFJZyqKKasspqSiqq2bW/lFKnryYm0lO7KausISUhmosGdOCKoWkMSGvpz/CPiTWRGWOMD8VGRXBuv3ac26/dj45V1yhb9xWzclcRK7MPUKPKBf3bM7xr6xPquwlFlmCMMeYYhIcJPdsm0LNtAhOG+DuawGbL9RtjjHGFJRhjjDGusARjjDHGFZZgjDHGuMISjDHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMK0JmqRgRyQO2H7G7JVB0jPsa204G9h1nmPXd+1jOaUp5fFWWxmJt7JxjLcuR72u36+6z76ZpsTZ2jn03/v0dcLTz3ChLnKq688RGVQ3ZFzDtWPc1tg1keDOeYzmnKeXxVVlOtDzHWpajlKHuPvtu7LsJ6O+mKWXx5nfj9s9ZY69QbyL7+Dj2NWXbm/EcyzlNKY+vytLUz2nonGMty5HvP27gnONl383R99t347vfAUc7L5DK0qiQaSLzFRHJUJdWHvW1UCoLhFZ5QqksEFrlsbI0XajXYNwwzd8BeFEolQVCqzyhVBYIrfJYWZrIajDGGGNcYTUYY4wxrmjWCUZEpotIroisOo5rh4nIShHZJCLPiojUOfYrEVkvIqtF5E/ejbrBeLxeFhF5SER2ichy5zXO+5E3GJMr341z/C4RURFJ9l7ER43Hje/mERHJcr6Xz0Qk1fuR1xuPG2V5UkTWOeX5QESSvB95gzG5UZ4rnX/7NSLiel/NiZShgc+bJCIbndekOvuP+u+qXm4OUQv0FzAWGAqsOo5rFwOjAQE+BS509p8JfAFEO+/bBnFZHgLuCpXvxjnWCZiLZ85UcrCWBUisc86vgZeCuCznARHO9h+BPwbzzxnQF+gNLADSA7UMTnxdj9jXGtji/LeVs93qaOU92qtZ12BU9WugoO4+EekhInNEJFNE/iMifY68TkQ64PkH/p16/s/PAC5zDt8GPKGq5c49ct0thYdLZfEbF8vzF+AewGedj26URVUP1Dk1Dh+Vx6WyfKaqVc6pi4A0d0vxA5fKs1ZV1/sifud+x1WGBpwPfK6qBaq6H/gcuOB4f0806wTTgGnAr1R1GHAX8EI953QEsuu8z3b2AfQCThOR70XkKxEZ7mq0R3eiZQH4pdN0MV1EWrkXapOcUHlEZDywS1VXuB1oE5zwdyMij4rITuA64PcuxtoYb/yc1boRz1/H/uTN8vhLU8pQn47Azjrva8t1XOWNaOJNmwURiQdOAd6r07wYXd+p9eyr/QsyAk/VchQwHHhXRLo7Wd9nvFSWF4FHnPePAH/G8wvA5060PCISC9yPpznGr7z03aCq9wP3i8h9wC+BB70caqO8VRbns+4HqoA3vBnjsfBmefzlaGUQkRuAqc6+nsBsEakAtqrqBBou13GV1xLMfwsDClV1cN2dIhIOZDpvP8Lzi7duNT4NyHG2s4H3nYSyWERq8Kz3k+dm4PU44bKo6t461/0d+MTNgBtxouXpAXQDVjj/6NKApSIyQlX3uBz7kbzxc1bXm8C/8UOCwUtlcTqTLwbO9vUfY0fw9nfjD/WWAUBVXwFeARCRBcBkVd1W55Rs4Iw679Pw9NVkczzldbsDKtBfQFfqdI4B3wJXOtsCDGrguiV4aim1HV7jnP23Ag87273wVDclSMvSoc45dwBvB/N3c8Q52/BRJ79L381Jdc75FTAriMtyAbAGSPHlz5fbP2f4qJP/eMtAw538W/G0wrRytls3pbz1xuWPLzRQXsBbwG6gEk+GvgnPX7lzgBXOD/3vG7g2HVgFbAae54dJq1HATOfYUuCsIC7L68BKIAvPX20dfFEWt8pzxDnb8N0oMje+m386+7PwrCvVMYjLsgnPH2LLnZdPRsS5WJ4JzmeVA3uBuYFYBupJMM7+G53vZBNwQ2PlPdrLZvIbY4xxhY0iM8YY4wpLMMYYY1xhCcYYY4wrLMEYY4xxhSUYY4wxrrAEY0KaiBT7+H4vi0g/L31WtXhWS14lIh83tsqwiCSJyC+8cW9jvMGGKZuQJiLFqhrvxc+L0B8WZnRV3dhF5DVgg6o+epTzuwKfqGp/X8RnTGOsBmOaHRFJEZF/isgS5zXG2T9CRL4VkWXOf3s7+yeLyHsi8jHwmYicISILRGSWeJ5j8kbtszGc/enOdrGzIOUKEVkkIu2c/T2c90tE5OEm1rK+44dFO+NFZJ6ILBXP8zkudc55Aujh1HqedM6927lPloj8wYv/G41plCUY0xw9A/xFVYcDVwAvO/vXAWNVdQie1Ykfq3PNaGCSqp7lvB8C/AboB3QHxtRznzhgkaoOAr4Gfl7n/s849290PSdnHayz8aymAFAGTFDVoXieP/RnJ8H9DtisqoNV9W4ROQ84CRgBDAaGicjYxu5njLfYYpemOToH6FdnpdlEEUkAWgKvichJeFaKjaxzzeeqWveZG4tVNRtARJbjWQvqmyPuU8EPC4RmAuc626P54VkabwJPNRBnizqfnYnn2RzgWQvqMSdZ1OCp2bSr5/rznNcy5308noTzdQP3M8arLMGY5igMGK2qpXV3ishzwHxVneD0Zyyoc7jkiM8or7NdTf3/lir1h07Ohs45mlJVHSwiLfEkqtuBZ/E8/yUFGKaqlSKyDYip53oBHlfVvx3jfY3xCmsiM83RZ3ienwKAiNQua94S2OVsT3bx/ovwNM0BXN3YyapahOexyHeJSCSeOHOd5HIm0MU59SCQUOfSucCNzvNBEJGOItLWS2UwplGWYEyoixWR7DqvO/H8sk53Or7X4HnEAsCfgMdFZCEQ7mJMvwHuFJHFQAegqLELVHUZnpVxr8bzQK50EcnAU5tZ55yTDyx0hjU/qaqf4WmC+05EVgKz+O8EZIyrbJiyMT7mPF2zVFVVRK4GrlHVSxu7zphgY30wxvjeMOB5Z+RXIX56DLUxbrMajDHGGFdYH4wxxhhXWIIxxhjjCkswxhhjXGEJxhhjjCsswRhjjHGFJRhjjDGu+P++g7XHqdtVDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 16:47 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>4.232034</th>\n",
       "    <th>4.060273</th>\n",
       "    <th>0.292894</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.save('fit_head')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.load('fit_head');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the fine-tuning, we can then unfeeze and launch a new training.\n",
    "\n",
    "要完成微调，我们可以解冻模型并开启新的训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 3:08:33 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>3.958489</th>\n",
       "    <th>3.885153</th>\n",
       "    <th>0.310139</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>3.871605</th>\n",
       "    <th>3.814774</th>\n",
       "    <th>0.319821</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>3.804589</th>\n",
       "    <th>3.767966</th>\n",
       "    <th>0.325793</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>3.771248</th>\n",
       "    <th>3.729666</th>\n",
       "    <th>0.330175</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>3.677534</th>\n",
       "    <th>3.699244</th>\n",
       "    <th>0.333532</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>3.644140</th>\n",
       "    <th>3.674071</th>\n",
       "    <th>0.336564</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>3.603597</th>\n",
       "    <th>3.655099</th>\n",
       "    <th>0.338747</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>3.524271</th>\n",
       "    <th>3.641979</th>\n",
       "    <th>0.340568</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>3.505476</th>\n",
       "    <th>3.636194</th>\n",
       "    <th>0.341246</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>3.461232</th>\n",
       "    <th>3.635963</th>\n",
       "    <th>0.341371</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How good is our model? Well let's try to see what it predicts after a few given words.\n",
    "\n",
    "我们的模型表现怎么样呢？ 嗯，让我们来看看在几个词过后模型预测出的词是怎样的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.load('fine_tuned');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "TEXT = \"I liked this movie because\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I liked this movie because of the cool scenery and the high level of xxmaj british hunting . xxmaj the only thing this movie has going for it is the horrible acting and no script . xxmaj the movie was a big disappointment . xxmaj\n",
      "I liked this movie because it was one of the few movies that made me laugh so hard i did n't like it . xxmaj it was a hilarious film and it was very entertaining . \n",
      "\n",
      " xxmaj the acting was great , i 'm\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to save not only the model, but also its encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word.\n",
    "\n",
    "我们不但保存了模型，而且保存了它的编码器，（也就是）负责创建和更新隐层状态（的部分）。剩下的负责猜词的部分，我们就不管了。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier 分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll create a new data object that only grabs the labelled data and keeps those labels. Again, this line takes a bit of time.\n",
    "\n",
    "现在我们要创建一个新的数据对象，仅抓取有标签的数据并且保留标签。这个步骤可能会需要一点时间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
    "             #grab all the text files in path\n",
    "             .split_by_folder(valid='test')\n",
    "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
    "             .label_from_folder(classes=['neg', 'pos'])\n",
    "             #label them all with their folders\n",
    "             .databunch(bs=bs))\n",
    "\n",
    "data_clas.save('data_clas.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_clas = load_data(path, 'data_clas.pkl', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='90%'>  <col width='10%'>  <tr>\n",
       "    <th>text</th>\n",
       "    <th>target</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj titanic directed by xxmaj james xxmaj cameron presents a fictional love story on the historical setting of the xxmaj titanic . xxmaj the plot is simple , xxunk , or not for those who love plots that twist and turn and keep you in suspense . xxmaj the end of the movie can be figured out within minutes of the start of the film , but the love</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj here are the matches . . . ( adv . = advantage ) \\n\\n xxmaj the xxmaj warriors ( xxmaj ultimate xxmaj warrior , xxmaj texas xxmaj tornado and xxmaj legion of xxmaj doom ) v xxmaj the xxmaj perfect xxmaj team ( xxmaj mr xxmaj perfect , xxmaj ax , xxmaj smash and xxmaj crush of xxmaj demolition ) : xxmaj ax is the first to go</th>\n",
       "    <th>neg</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos i felt duty bound to watch the 1983 xxmaj timothy xxmaj dalton / xxmaj zelah xxmaj clarke adaptation of \" xxmaj jane xxmaj eyre , \" because i 'd just written an article about the 2006 xxup bbc \" xxmaj jane xxmaj eyre \" for xxunk . \\n\\n xxmaj so , i approached watching this the way i 'd approach doing homework . \\n\\n i was irritated at first</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>xxbos xxmaj no , this is n't a sequel to the fabulous xxup ova series , but rather a remake of the events that occurred after the death of xxmaj xxunk ( and the disappearance of xxmaj woodchuck ) . xxmaj it is also more accurate to the novels that inspired this wonderful series , which is why characters ( namely xxmaj orson and xxmaj xxunk ) are xxunk ,</th>\n",
       "    <th>pos</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then create a model to classify those reviews and load the encoder we saved before.\n",
    "\n",
    "我们可以建立一个模型来对影评进行分类，并且导入之前存储好的编码器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)\n",
    "learn.load_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 03:40 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.310078</th>\n",
       "    <th>0.197204</th>\n",
       "    <th>0.926960</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.save('first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.load('first');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 04:03 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.255913</th>\n",
       "    <th>0.169186</th>\n",
       "    <th>0.937800</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.save('second')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.load('second');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 05:42 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.223174</th>\n",
       "    <th>0.165679</th>\n",
       "    <th>0.939600</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.save('third')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learn.load('third');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 15:17 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.240424</th>\n",
       "    <th>0.155204</th>\n",
       "    <th>0.943160</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.217462</th>\n",
       "    <th>0.153421</th>\n",
       "    <th>0.943960</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category pos, tensor(1), tensor([7.5928e-04, 9.9924e-01]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"I really loved that movie, it was awesome!\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}