pytorch/NLP/char_rnn_classification_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "用字符级RNN分类名称\n",
    "*********************************************\n",
    "**作者**: `Sean Robertson <https://github.com/spro/practical-pytorch>`_\n",
    "\n",
    "我们将建立和训练一个基本的字符级RNN进行分类单词. 字符级别的RNN将单词读为一系列字符 - 在每个步骤输出一个预测和“隐藏状态”,\n",
    "将先前的隐藏状态作为下一步的输入. 我们采取最后的预测作为输出,即该单词属于哪一类.\n",
    "\n",
    "具体来说,我们将用18种语言的几千个姓氏作为训练集并根据拼写预测名称来自哪种语言:\n",
    "\n",
    "::\n",
    "\n",
    "    $ python predict.py Hinton\n",
    "    (-0.47) Scottish\n",
    "    (-1.52) English\n",
    "    (-3.57) Irish\n",
    "\n",
    "    $ python predict.py Schmidhuber\n",
    "    (-0.19) German\n",
    "    (-2.48) Czech\n",
    "    (-2.68) Dutch\n",
    "\n",
    "\n",
    "**推荐阅读:**\n",
    "\n",
    "假设你至少已经安装了PyTorch,知道Python和了解张量:\n",
    "\n",
    "-  http://pytorch.org/ 安装步骤\n",
    "-  :doc:`/beginner/deep_learning_60min_blitz` 大体了解PyTorch\n",
    "-  :doc:`/beginner/pytorch_with_examples` 深入概括\n",
    "-  :doc:`/beginner/former_torchies_tutorial` 假设你是前Lua Torch用户\n",
    "\n",
    "了解RNN及其工作方式也很有用:\n",
    "\n",
    "-  `递归神经网络的不合理有效性 <http://karpathy.github.io/2015/05/21/rnn-effectiveness/>`__\n",
    "   展示了一堆真实生活的例子\n",
    "-  `理解LSTM网络 <http://colah.github.io/posts/2015-08-Understanding-LSTMs/>`__\n",
    "   是关于LSTM的具体内容,但也包含有关RNN的一般信息\n",
    "\n",
    "准备数据\n",
    "==================\n",
    "\n",
    ".. 注意::\n",
    "   从这里下载数据\n",
    "   `here <https://download.pytorch.org/tutorial/data.zip>`_\n",
    "   并将其解压到当前目录.\n",
    "\n",
    "在 ``data/names`` 目录中包含18个名为as的文本文件 \"[Language].txt\" .\n",
    "每个文件都包含一堆名称,每个名称一行大多是罗马化（但我们仍然需要从Unicode转换为ASCII）.\n",
    "\n",
    "我们最终会得到每种语言的名称列表字典 ``{language: [names ...]}``  通用变量“类别”和“行”\n",
    "（在我们的例子中用于语言和名称）用于以后的扩展性.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import unicode_literals, print_function, division\n",
    "from io import open\n",
    "import glob\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "dirs=r'G:\\git\\data\\pytorch\\data\\names\\*.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Arabic.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Chinese.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Czech.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Dutch.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\English.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\French.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\German.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Greek.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Irish.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Italian.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Japanese.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Korean.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Polish.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Portuguese.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Russian.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Scottish.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Spanish.txt', 'G:\\\\git\\\\data\\\\pytorch\\\\data\\\\names\\\\Vietnamese.txt']\n"
     ]
    }
   ],
   "source": [
    "def findFiles(path): return glob.glob(path)\n",
    "\n",
    "print(findFiles(r'G:\\git\\data\\pytorch\\data\\names\\*.txt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.ascii_letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unicodedata.normalize('NFD', 'a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unicodedata.category('a')!='Mn'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slusarski\n"
     ]
    }
   ],
   "source": [
    "\n",
    "all_letters = string.ascii_letters + \" .,;'\"\n",
    "n_letters = len(all_letters)\n",
    "\n",
    "# 将 Unicode 字符串转换为纯 ASCII 编码, 这里感谢 http://stackoverflow.com/a/518232/2809427\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "        and c in all_letters\n",
    "    )\n",
    "\n",
    "print(unicodeToAscii('Ślusàrski'))\n",
    "\n",
    "# 构建category_lines字典, 每种语言的名称列表\n",
    "category_lines = {}\n",
    "all_categories = []\n",
    "\n",
    "# 读取一个文件并分成几行\n",
    "def readLines(filename):\n",
    "    lines = open(filename, encoding='utf-8').read().strip().split('\\n')\n",
    "    return [unicodeToAscii(line) for line in lines]\n",
    "\n",
    "for filename in findFiles(dirs):\n",
    "    category = filename.split(os.sep)[-1].split('.')[0]\n",
    "    all_categories.append(category)\n",
    "    lines = readLines(filename)\n",
    "    category_lines[category] = lines\n",
    "\n",
    "n_categories = len(all_categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们有 ``category_lines``, 这是一个映射每个类别的字典\n",
    "(语言) 到行列表 (名称). 我们也跟踪\n",
    "``all_categories`` (只是一个语言列表) 和 ``n_categories`` 为以后做参考.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Abandonato', 'Abatangelo', 'Abatantuono', 'Abate', 'Abategiovanni']\n"
     ]
    }
   ],
   "source": [
    "print(category_lines['Italian'][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将名字转化为张量\n",
    "--------------------------\n",
    "\n",
    "现在我们已经组织了所有的名字,我们需要将它们变成张量以便使用它们.\n",
    "\n",
    "为了表示单个字母,我们使用大小为 ``<1 x n_letters>`` 的\"单热矢量\".\n",
    "除了当前字母的索引处的1以外,单热矢量剩余填充0, e.g. ``\"b\" = <0 1 0 0 0 ...>``.\n",
    "\n",
    "为了说出一个词,我们将其中的一部分加入到二维矩阵中\n",
    "``<line_length x 1 x n_letters>``.\n",
    "\n",
    "额外的1维度是因为PyTorch假定所有内容都是批量的 - 我们在这里只使用1的批量大小.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
      "          0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
      "          0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,\n",
      "          0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
      "          0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])\n",
      "torch.Size([5, 1, 57])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# 从all_letters中查找字母索引, e.g. \"a\" = 0\n",
    "def letterToIndex(letter):\n",
    "    return all_letters.find(letter)\n",
    "\n",
    "# 只是为了演示, 把一个字母变成一个 <1 x n_letters> 张量\n",
    "def letterToTensor(letter):\n",
    "    tensor = torch.zeros(1, n_letters)\n",
    "    tensor[0][letterToIndex(letter)] = 1\n",
    "    return tensor\n",
    "\n",
    "# 把一行变成一个 <line_length x 1 x n_letters>,\n",
    "# 或一批单热字符向量\n",
    "def lineToTensor(line):\n",
    "    tensor = torch.zeros(len(line), 1, n_letters)\n",
    "    for li, letter in enumerate(line):\n",
    "        tensor[li][0][letterToIndex(letter)] = 1\n",
    "    return tensor\n",
    "\n",
    "print(letterToTensor('J'))\n",
    "\n",
    "print(lineToTensor('Jones').size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 5,1,57\n",
    "       \n",
    "    分别代表什么？Jones对应一个y，但是确实有五个字母，每个字母有57个维度\n",
    "    \n",
    "    所以分别对应，时间序列的长度，样本，输入的维度？\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建网络\n",
    "====================\n",
    "\n",
    "在autograd之前, 在Torch中创建一个循环神经网络涉及到克隆几个步骤一个图层的参数.\n",
    "图层保持隐藏状态和渐变, 现在完全由图形本身处理.\n",
    "这意味着您可以以非常“纯粹”的方式实现RNN, 作为常规的前馈层.\n",
    "\n",
    "这个RNN模块 (大部分都是复制 `the PyTorch for Torch users tutorial <http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#example-2-recurrent-net>`__)\n",
    "只有2个线性层可以在输入和隐藏状态下运行, 在输出之后有一个LogSoftmax层.\n",
    "\n",
    ".. figure:: https://i.imgur.com/Z2xbySO.png\n",
    "   :alt:\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(input_size + hidden_size, output_size)\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, input, hidden):\n",
    "        combined = torch.cat((input, hidden), 1)\n",
    "        hidden = self.i2h(combined)\n",
    "        output = self.i2o(combined)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "\n",
    "    def initHidden(self):\n",
    "        return Variable(torch.zeros(1, self.hidden_size))\n",
    "\n",
    "n_hidden = 128\n",
    "#57,128,18\n",
    "rnn = RNN(n_letters, n_hidden, n_categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了运行这个网络的一个步骤, 我们需要传递一个输入 (在我们的例子中是当前字母的张量) 和一个先前的隐藏状态 (我们首先初始化为零) .\n",
    "我们将返回输出 (每种语言的概率) 和下一个隐藏状态 (我们为下一步保留).\n",
    "请记住, PyTorch模块对变量进行操作, 而不是直接对张量进行操作."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "input = Variable(letterToTensor('A'))\n",
    "hidden = Variable(torch.zeros(1, n_hidden))\n",
    "output, next_hidden = rnn(input, hidden)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 18])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了提高效率我们不希望为每一步创建一个新的张量,\n",
    "所以我们使用 ``lineToTensor`` 而不是 ``letterToTensor`` 并使用切片.\n",
    "这可以通过预先计算批次的张量进一步优化.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-2.8423, -2.9474, -2.8411, -2.8580, -2.9305, -2.9061, -2.8690,\n",
      "         -2.9037, -2.7865, -2.9326, -2.9553, -2.8911, -2.8138, -2.8958,\n",
      "         -2.9437, -2.9259, -2.9467, -2.8587]])\n"
     ]
    }
   ],
   "source": [
    "input = Variable(lineToTensor('Albert'))\n",
    "hidden = Variable(torch.zeros(1, n_hidden))\n",
    "\n",
    "output, next_hidden = rnn(input[0], hidden)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "正如你所看到的输出是一个 ``<1 x n_categories>`` 张量,\n",
    "每个项目都是该类别的可能性 (越高越有可能).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练\n",
    "========\n",
    "准备训练\n",
    "----------------------\n",
    "\n",
    "在训练之前,我们应该做一些辅助功能.\n",
    "首先是解释网络的输出, 我们知道这是每个类别的可能性.\n",
    "我么可以使用 ``Tensor.topk`` 得到最大价值的指数:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Irish', tensor(8))\n"
     ]
    }
   ],
   "source": [
    "def categoryFromOutput(output):\n",
    "    top_n, top_i = output.data.topk(1) # Tensor out of Variable with .data\n",
    "    category_i = top_i[0][0]\n",
    "    return all_categories[category_i], category_i\n",
    "\n",
    "print(categoryFromOutput(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也希望能够快速获得训练示例 (名称及其语言):\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Khoury',\n",
       " 'Nahas',\n",
       " 'Daher',\n",
       " 'Gerges',\n",
       " 'Nazari',\n",
       " 'Maalouf',\n",
       " 'Gerges',\n",
       " 'Naifeh',\n",
       " 'Guirguis',\n",
       " 'Baba',\n",
       " 'Sabbagh',\n",
       " 'Attia',\n",
       " 'Tahan',\n",
       " 'Haddad',\n",
       " 'Aswad',\n",
       " 'Najjar',\n",
       " 'Dagher',\n",
       " 'Maloof',\n",
       " 'Isa',\n",
       " 'Asghar',\n",
       " 'Nader',\n",
       " 'Gaber',\n",
       " 'Abboud',\n",
       " 'Maalouf',\n",
       " 'Zogby',\n",
       " 'Srour',\n",
       " 'Bahar',\n",
       " 'Mustafa',\n",
       " 'Hanania',\n",
       " 'Daher',\n",
       " 'Tuma',\n",
       " 'Nahas',\n",
       " 'Saliba',\n",
       " 'Shamoon',\n",
       " 'Handal',\n",
       " 'Baba',\n",
       " 'Amari',\n",
       " 'Bahar',\n",
       " 'Atiyeh',\n",
       " 'Said',\n",
       " 'Khouri',\n",
       " 'Tahan',\n",
       " 'Baba',\n",
       " 'Mustafa',\n",
       " 'Guirguis',\n",
       " 'Sleiman',\n",
       " 'Seif',\n",
       " 'Dagher',\n",
       " 'Bahar',\n",
       " 'Gaber',\n",
       " 'Harb',\n",
       " 'Seif',\n",
       " 'Asker',\n",
       " 'Nader',\n",
       " 'Antar',\n",
       " 'Awad',\n",
       " 'Srour',\n",
       " 'Shadid',\n",
       " 'Hajjar',\n",
       " 'Hanania',\n",
       " 'Kalb',\n",
       " 'Shadid',\n",
       " 'Bazzi',\n",
       " 'Mustafa',\n",
       " 'Masih',\n",
       " 'Ghanem',\n",
       " 'Haddad',\n",
       " 'Isa',\n",
       " 'Antoun',\n",
       " 'Sarraf',\n",
       " 'Sleiman',\n",
       " 'Dagher',\n",
       " 'Najjar',\n",
       " 'Malouf',\n",
       " 'Nahas',\n",
       " 'Naser',\n",
       " 'Saliba',\n",
       " 'Shamon',\n",
       " 'Malouf',\n",
       " 'Kalb',\n",
       " 'Daher',\n",
       " 'Maalouf',\n",
       " 'Wasem',\n",
       " 'Kanaan',\n",
       " 'Naifeh',\n",
       " 'Boutros',\n",
       " 'Moghadam',\n",
       " 'Masih',\n",
       " 'Sleiman',\n",
       " 'Aswad',\n",
       " 'Cham',\n",
       " 'Assaf',\n",
       " 'Quraishi',\n",
       " 'Shalhoub',\n",
       " 'Sabbag',\n",
       " 'Mifsud',\n",
       " 'Gaber',\n",
       " 'Shammas',\n",
       " 'Tannous',\n",
       " 'Sleiman',\n",
       " 'Bazzi',\n",
       " 'Quraishi',\n",
       " 'Rahal',\n",
       " 'Cham',\n",
       " 'Ghanem',\n",
       " 'Ghanem',\n",
       " 'Naser',\n",
       " 'Baba',\n",
       " 'Shamon',\n",
       " 'Almasi',\n",
       " 'Basara',\n",
       " 'Quraishi',\n",
       " 'Bata',\n",
       " 'Wasem',\n",
       " 'Shamoun',\n",
       " 'Deeb',\n",
       " 'Touma',\n",
       " 'Asfour',\n",
       " 'Deeb',\n",
       " 'Hadad',\n",
       " 'Naifeh',\n",
       " 'Touma',\n",
       " 'Bazzi',\n",
       " 'Shamoun',\n",
       " 'Nahas',\n",
       " 'Haddad',\n",
       " 'Arian',\n",
       " 'Kouri',\n",
       " 'Deeb',\n",
       " 'Toma',\n",
       " 'Halabi',\n",
       " 'Nazari',\n",
       " 'Saliba',\n",
       " 'Fakhoury',\n",
       " 'Hadad',\n",
       " 'Baba',\n",
       " 'Mansour',\n",
       " 'Sayegh',\n",
       " 'Antar',\n",
       " 'Deeb',\n",
       " 'Morcos',\n",
       " 'Shalhoub',\n",
       " 'Sarraf',\n",
       " 'Amari',\n",
       " 'Wasem',\n",
       " 'Ganim',\n",
       " 'Tuma',\n",
       " 'Fakhoury',\n",
       " 'Hadad',\n",
       " 'Hakimi',\n",
       " 'Nader',\n",
       " 'Said',\n",
       " 'Ganim',\n",
       " 'Daher',\n",
       " 'Ganem',\n",
       " 'Tuma',\n",
       " 'Boutros',\n",
       " 'Aswad',\n",
       " 'Sarkis',\n",
       " 'Daher',\n",
       " 'Toma',\n",
       " 'Boutros',\n",
       " 'Kanaan',\n",
       " 'Antar',\n",
       " 'Gerges',\n",
       " 'Kouri',\n",
       " 'Maroun',\n",
       " 'Wasem',\n",
       " 'Dagher',\n",
       " 'Naifeh',\n",
       " 'Bishara',\n",
       " 'Ba',\n",
       " 'Cham',\n",
       " 'Kalb',\n",
       " 'Bazzi',\n",
       " 'Bitar',\n",
       " 'Hadad',\n",
       " 'Moghadam',\n",
       " 'Sleiman',\n",
       " 'Shamoun',\n",
       " 'Antar',\n",
       " 'Atiyeh',\n",
       " 'Koury',\n",
       " 'Nahas',\n",
       " 'Kouri',\n",
       " 'Maroun',\n",
       " 'Nassar',\n",
       " 'Sayegh',\n",
       " 'Haik',\n",
       " 'Ghanem',\n",
       " 'Sayegh',\n",
       " 'Salib',\n",
       " 'Cham',\n",
       " 'Bata',\n",
       " 'Touma',\n",
       " 'Antoun',\n",
       " 'Antar',\n",
       " 'Bata',\n",
       " 'Botros',\n",
       " 'Shammas',\n",
       " 'Ganim',\n",
       " 'Sleiman',\n",
       " 'Seif',\n",
       " 'Moghadam',\n",
       " 'Ba',\n",
       " 'Tannous',\n",
       " 'Bazzi',\n",
       " 'Seif',\n",
       " 'Salib',\n",
       " 'Hadad',\n",
       " 'Quraishi',\n",
       " 'Halabi',\n",
       " 'Essa',\n",
       " 'Bahar',\n",
       " 'Kattan',\n",
       " 'Boutros',\n",
       " 'Nahas',\n",
       " 'Sabbagh',\n",
       " 'Kanaan',\n",
       " 'Sayegh',\n",
       " 'Said',\n",
       " 'Botros',\n",
       " 'Najjar',\n",
       " 'Toma',\n",
       " 'Bata',\n",
       " 'Atiyeh',\n",
       " 'Halabi',\n",
       " 'Tannous',\n",
       " 'Kouri',\n",
       " 'Shamoon',\n",
       " 'Kassis',\n",
       " 'Haddad',\n",
       " 'Tuma',\n",
       " 'Mansour',\n",
       " 'Antar',\n",
       " 'Kassis',\n",
       " 'Kalb',\n",
       " 'Basara',\n",
       " 'Rahal',\n",
       " 'Mansour',\n",
       " 'Handal',\n",
       " 'Morcos',\n",
       " 'Fakhoury',\n",
       " 'Hadad',\n",
       " 'Morcos',\n",
       " 'Kouri',\n",
       " 'Quraishi',\n",
       " 'Almasi',\n",
       " 'Awad',\n",
       " 'Naifeh',\n",
       " 'Koury',\n",
       " 'Asker',\n",
       " 'Maroun',\n",
       " 'Fakhoury',\n",
       " 'Sabbag',\n",
       " 'Sarraf',\n",
       " 'Shamon',\n",
       " 'Assaf',\n",
       " 'Boutros',\n",
       " 'Malouf',\n",
       " 'Nassar',\n",
       " 'Qureshi',\n",
       " 'Ghanem',\n",
       " 'Srour',\n",
       " 'Almasi',\n",
       " 'Qureshi',\n",
       " 'Ghannam',\n",
       " 'Mustafa',\n",
       " 'Najjar',\n",
       " 'Kassab',\n",
       " 'Shadid',\n",
       " 'Shamoon',\n",
       " 'Morcos',\n",
       " 'Atiyeh',\n",
       " 'Isa',\n",
       " 'Ba',\n",
       " 'Baz',\n",
       " 'Asker',\n",
       " 'Seif',\n",
       " 'Asghar',\n",
       " 'Hajjar',\n",
       " 'Deeb',\n",
       " 'Essa',\n",
       " 'Qureshi',\n",
       " 'Abboud',\n",
       " 'Ganem',\n",
       " 'Haddad',\n",
       " 'Koury',\n",
       " 'Nassar',\n",
       " 'Abadi',\n",
       " 'Toma',\n",
       " 'Tannous',\n",
       " 'Harb',\n",
       " 'Issa',\n",
       " 'Khouri',\n",
       " 'Mifsud',\n",
       " 'Kalb',\n",
       " 'Gaber',\n",
       " 'Ganim',\n",
       " 'Boulos',\n",
       " 'Samaha',\n",
       " 'Haddad',\n",
       " 'Sabbag',\n",
       " 'Wasem',\n",
       " 'Dagher',\n",
       " 'Rahal',\n",
       " 'Atiyeh',\n",
       " 'Antar',\n",
       " 'Asghar',\n",
       " 'Mansour',\n",
       " 'Awad',\n",
       " 'Boulos',\n",
       " 'Sarraf',\n",
       " 'Deeb',\n",
       " 'Abadi',\n",
       " 'Nazari',\n",
       " 'Daher',\n",
       " 'Gerges',\n",
       " 'Shamoon',\n",
       " 'Gaber',\n",
       " 'Amari',\n",
       " 'Sarraf',\n",
       " 'Nazari',\n",
       " 'Saliba',\n",
       " 'Naifeh',\n",
       " 'Nazari',\n",
       " 'Hakimi',\n",
       " 'Shamon',\n",
       " 'Abboud',\n",
       " 'Quraishi',\n",
       " 'Tahan',\n",
       " 'Safar',\n",
       " 'Hajjar',\n",
       " 'Srour',\n",
       " 'Gaber',\n",
       " 'Shalhoub',\n",
       " 'Attia',\n",
       " 'Safar',\n",
       " 'Said',\n",
       " 'Ganem',\n",
       " 'Nader',\n",
       " 'Asghar',\n",
       " 'Mustafa',\n",
       " 'Said',\n",
       " 'Antar',\n",
       " 'Botros',\n",
       " 'Nader',\n",
       " 'Ghannam',\n",
       " 'Asfour',\n",
       " 'Tahan',\n",
       " 'Mansour',\n",
       " 'Attia',\n",
       " 'Touma',\n",
       " 'Najjar',\n",
       " 'Kassis',\n",
       " 'Abboud',\n",
       " 'Bishara',\n",
       " 'Bazzi',\n",
       " 'Shalhoub',\n",
       " 'Shalhoub',\n",
       " 'Safar',\n",
       " 'Khoury',\n",
       " 'Nazari',\n",
       " 'Sabbag',\n",
       " 'Sleiman',\n",
       " 'Atiyeh',\n",
       " 'Kouri',\n",
       " 'Bitar',\n",
       " 'Zogby',\n",
       " 'Ghanem',\n",
       " 'Assaf',\n",
       " 'Abadi',\n",
       " 'Arian',\n",
       " 'Shalhoub',\n",
       " 'Khoury',\n",
       " 'Morcos',\n",
       " 'Shamon',\n",
       " 'Wasem',\n",
       " 'Abadi',\n",
       " 'Antoun',\n",
       " 'Baz',\n",
       " 'Naser',\n",
       " 'Assaf',\n",
       " 'Saliba',\n",
       " 'Nader',\n",
       " 'Mikhail',\n",
       " 'Naser',\n",
       " 'Daher',\n",
       " 'Morcos',\n",
       " 'Awad',\n",
       " 'Nahas',\n",
       " 'Sarkis',\n",
       " 'Malouf',\n",
       " 'Mustafa',\n",
       " 'Fakhoury',\n",
       " 'Ghannam',\n",
       " 'Shadid',\n",
       " 'Gaber',\n",
       " 'Koury',\n",
       " 'Atiyeh',\n",
       " 'Shamon',\n",
       " 'Boutros',\n",
       " 'Sarraf',\n",
       " 'Arian',\n",
       " 'Fakhoury',\n",
       " 'Abadi',\n",
       " 'Kassab',\n",
       " 'Nahas',\n",
       " 'Quraishi',\n",
       " 'Mansour',\n",
       " 'Samaha',\n",
       " 'Wasem',\n",
       " 'Seif',\n",
       " 'Fakhoury',\n",
       " 'Saliba',\n",
       " 'Cham',\n",
       " 'Bahar',\n",
       " 'Shamoun',\n",
       " 'Essa',\n",
       " 'Shamon',\n",
       " 'Asfour',\n",
       " 'Bitar',\n",
       " 'Cham',\n",
       " 'Tahan',\n",
       " 'Tannous',\n",
       " 'Daher',\n",
       " 'Khoury',\n",
       " 'Shamon',\n",
       " 'Bahar',\n",
       " 'Quraishi',\n",
       " 'Ghannam',\n",
       " 'Kassab',\n",
       " 'Zogby',\n",
       " 'Basara',\n",
       " 'Shammas',\n",
       " 'Arian',\n",
       " 'Sayegh',\n",
       " 'Naifeh',\n",
       " 'Mifsud',\n",
       " 'Sleiman',\n",
       " 'Arian',\n",
       " 'Kassis',\n",
       " 'Shamoun',\n",
       " 'Kassis',\n",
       " 'Harb',\n",
       " 'Mustafa',\n",
       " 'Boulos',\n",
       " 'Asghar',\n",
       " 'Shamon',\n",
       " 'Kanaan',\n",
       " 'Atiyeh',\n",
       " 'Kassab',\n",
       " 'Tahan',\n",
       " 'Bazzi',\n",
       " 'Kassis',\n",
       " 'Qureshi',\n",
       " 'Basara',\n",
       " 'Shalhoub',\n",
       " 'Sayegh',\n",
       " 'Haik',\n",
       " 'Attia',\n",
       " 'Maroun',\n",
       " 'Kassis',\n",
       " 'Sarkis',\n",
       " 'Harb',\n",
       " 'Assaf',\n",
       " 'Kattan',\n",
       " 'Antar',\n",
       " 'Sleiman',\n",
       " 'Touma',\n",
       " 'Sarraf',\n",
       " 'Bazzi',\n",
       " 'Boulos',\n",
       " 'Baz',\n",
       " 'Issa',\n",
       " 'Shamon',\n",
       " 'Shadid',\n",
       " 'Deeb',\n",
       " 'Sabbag',\n",
       " 'Wasem',\n",
       " 'Awad',\n",
       " 'Mansour',\n",
       " 'Saliba',\n",
       " 'Fakhoury',\n",
       " 'Arian',\n",
       " 'Bishara',\n",
       " 'Dagher',\n",
       " 'Bishara',\n",
       " 'Koury',\n",
       " 'Fakhoury',\n",
       " 'Naser',\n",
       " 'Nader',\n",
       " 'Antar',\n",
       " 'Gerges',\n",
       " 'Handal',\n",
       " 'Hanania',\n",
       " 'Shadid',\n",
       " 'Gerges',\n",
       " 'Kassis',\n",
       " 'Essa',\n",
       " 'Assaf',\n",
       " 'Shadid',\n",
       " 'Seif',\n",
       " 'Shalhoub',\n",
       " 'Shamoun',\n",
       " 'Hajjar',\n",
       " 'Baba',\n",
       " 'Sayegh',\n",
       " 'Mustafa',\n",
       " 'Sabbagh',\n",
       " 'Isa',\n",
       " 'Najjar',\n",
       " 'Tannous',\n",
       " 'Hanania',\n",
       " 'Ganem',\n",
       " 'Gerges',\n",
       " 'Fakhoury',\n",
       " 'Mifsud',\n",
       " 'Nahas',\n",
       " 'Bishara',\n",
       " 'Bishara',\n",
       " 'Abadi',\n",
       " 'Sarkis',\n",
       " 'Masih',\n",
       " 'Isa',\n",
       " 'Attia',\n",
       " 'Kalb',\n",
       " 'Essa',\n",
       " 'Boulos',\n",
       " 'Basara',\n",
       " 'Halabi',\n",
       " 'Halabi',\n",
       " 'Dagher',\n",
       " 'Attia',\n",
       " 'Kassis',\n",
       " 'Tuma',\n",
       " 'Gerges',\n",
       " 'Ghannam',\n",
       " 'Toma',\n",
       " 'Baz',\n",
       " 'Asghar',\n",
       " 'Zogby',\n",
       " 'Aswad',\n",
       " 'Hadad',\n",
       " 'Dagher',\n",
       " 'Naser',\n",
       " 'Shadid',\n",
       " 'Atiyeh',\n",
       " 'Zogby',\n",
       " 'Abboud',\n",
       " 'Tannous',\n",
       " 'Khouri',\n",
       " 'Atiyeh',\n",
       " 'Ganem',\n",
       " 'Maalouf',\n",
       " 'Isa',\n",
       " 'Maroun',\n",
       " 'Issa',\n",
       " 'Khouri',\n",
       " 'Harb',\n",
       " 'Nader',\n",
       " 'Awad',\n",
       " 'Nahas',\n",
       " 'Said',\n",
       " 'Baba',\n",
       " 'Totah',\n",
       " 'Ganim',\n",
       " 'Handal',\n",
       " 'Mansour',\n",
       " 'Basara',\n",
       " 'Malouf',\n",
       " 'Said',\n",
       " 'Botros',\n",
       " 'Samaha',\n",
       " 'Safar',\n",
       " 'Tahan',\n",
       " 'Botros',\n",
       " 'Shamoun',\n",
       " 'Handal',\n",
       " 'Sarraf',\n",
       " 'Malouf',\n",
       " 'Bishara',\n",
       " 'Aswad',\n",
       " 'Khouri',\n",
       " 'Baz',\n",
       " 'Asker',\n",
       " 'Toma',\n",
       " 'Koury',\n",
       " 'Gerges',\n",
       " 'Bishara',\n",
       " 'Boulos',\n",
       " 'Najjar',\n",
       " 'Aswad',\n",
       " 'Shamon',\n",
       " 'Kouri',\n",
       " 'Srour',\n",
       " 'Assaf',\n",
       " 'Tannous',\n",
       " 'Attia',\n",
       " 'Mustafa',\n",
       " 'Kattan',\n",
       " 'Asghar',\n",
       " 'Amari',\n",
       " 'Shadid',\n",
       " 'Said',\n",
       " 'Bazzi',\n",
       " 'Masih',\n",
       " 'Antar',\n",
       " 'Fakhoury',\n",
       " 'Shadid',\n",
       " 'Masih',\n",
       " 'Handal',\n",
       " 'Sarraf',\n",
       " 'Kassis',\n",
       " 'Salib',\n",
       " 'Hajjar',\n",
       " 'Totah',\n",
       " 'Koury',\n",
       " 'Totah',\n",
       " 'Mustafa',\n",
       " 'Sabbagh',\n",
       " 'Moghadam',\n",
       " 'Toma',\n",
       " 'Srour',\n",
       " 'Almasi',\n",
       " 'Totah',\n",
       " 'Maroun',\n",
       " 'Kattan',\n",
       " 'Naifeh',\n",
       " 'Sarkis',\n",
       " 'Mikhail',\n",
       " 'Nazari',\n",
       " 'Boutros',\n",
       " 'Guirguis',\n",
       " 'Gaber',\n",
       " 'Kassis',\n",
       " 'Masih',\n",
       " 'Hanania',\n",
       " 'Maloof',\n",
       " 'Quraishi',\n",
       " 'Cham',\n",
       " 'Hadad',\n",
       " 'Tahan',\n",
       " 'Bitar',\n",
       " 'Arian',\n",
       " 'Gaber',\n",
       " 'Baz',\n",
       " 'Mansour',\n",
       " 'Kalb',\n",
       " 'Sarkis',\n",
       " 'Attia',\n",
       " 'Antar',\n",
       " 'Asfour',\n",
       " 'Said',\n",
       " 'Essa',\n",
       " 'Koury',\n",
       " 'Hadad',\n",
       " 'Tuma',\n",
       " 'Moghadam',\n",
       " 'Sabbagh',\n",
       " 'Amari',\n",
       " 'Dagher',\n",
       " 'Srour',\n",
       " 'Antoun',\n",
       " 'Sleiman',\n",
       " 'Maroun',\n",
       " 'Tuma',\n",
       " 'Nahas',\n",
       " 'Hanania',\n",
       " 'Sayegh',\n",
       " 'Amari',\n",
       " 'Sabbagh',\n",
       " 'Said',\n",
       " 'Cham',\n",
       " 'Asker',\n",
       " 'Nassar',\n",
       " 'Bitar',\n",
       " 'Said',\n",
       " 'Dagher',\n",
       " 'Safar',\n",
       " 'Khouri',\n",
       " 'Totah',\n",
       " 'Khoury',\n",
       " 'Salib',\n",
       " 'Basara',\n",
       " 'Abboud',\n",
       " 'Baz',\n",
       " 'Isa',\n",
       " 'Cham',\n",
       " 'Amari',\n",
       " 'Mifsud',\n",
       " 'Hadad',\n",
       " 'Rahal',\n",
       " 'Khoury',\n",
       " 'Bazzi',\n",
       " 'Basara',\n",
       " 'Totah',\n",
       " 'Ghannam',\n",
       " 'Koury',\n",
       " 'Malouf',\n",
       " 'Zogby',\n",
       " 'Zogby',\n",
       " 'Boutros',\n",
       " 'Nassar',\n",
       " 'Handal',\n",
       " 'Hajjar',\n",
       " 'Maloof',\n",
       " 'Abadi',\n",
       " 'Maroun',\n",
       " 'Mifsud',\n",
       " 'Kalb',\n",
       " 'Amari',\n",
       " 'Hakimi',\n",
       " 'Boutros',\n",
       " 'Masih',\n",
       " 'Kattan',\n",
       " 'Haddad',\n",
       " 'Arian',\n",
       " 'Nazari',\n",
       " 'Assaf',\n",
       " 'Attia',\n",
       " 'Wasem',\n",
       " 'Gerges',\n",
       " 'Asker',\n",
       " 'Tahan',\n",
       " 'Fakhoury',\n",
       " 'Shadid',\n",
       " 'Sarraf',\n",
       " 'Attia',\n",
       " 'Naifeh',\n",
       " 'Aswad',\n",
       " 'Deeb',\n",
       " 'Tannous',\n",
       " 'Totah',\n",
       " 'Cham',\n",
       " 'Baba',\n",
       " 'Najjar',\n",
       " 'Hajjar',\n",
       " 'Shamoon',\n",
       " 'Handal',\n",
       " 'Awad',\n",
       " 'Guirguis',\n",
       " 'Awad',\n",
       " 'Ganem',\n",
       " 'Naifeh',\n",
       " 'Khoury',\n",
       " 'Hajjar',\n",
       " 'Moghadam',\n",
       " 'Mikhail',\n",
       " 'Ghannam',\n",
       " 'Guirguis',\n",
       " 'Tannous',\n",
       " 'Kanaan',\n",
       " 'Handal',\n",
       " 'Khoury',\n",
       " 'Kalb',\n",
       " 'Qureshi',\n",
       " 'Najjar',\n",
       " 'Atiyeh',\n",
       " 'Gerges',\n",
       " 'Nassar',\n",
       " 'Tahan',\n",
       " 'Hadad',\n",
       " 'Fakhoury',\n",
       " 'Salib',\n",
       " 'Wasem',\n",
       " 'Bitar',\n",
       " 'Fakhoury',\n",
       " 'Attia',\n",
       " 'Awad',\n",
       " 'Totah',\n",
       " 'Deeb',\n",
       " 'Touma',\n",
       " 'Botros',\n",
       " 'Nazari',\n",
       " 'Nahas',\n",
       " 'Kouri',\n",
       " 'Ghannam',\n",
       " 'Assaf',\n",
       " 'Asfour',\n",
       " 'Sarraf',\n",
       " 'Naifeh',\n",
       " 'Toma',\n",
       " 'Asghar',\n",
       " 'Abboud',\n",
       " 'Issa',\n",
       " 'Sabbag',\n",
       " 'Sabbagh',\n",
       " 'Isa',\n",
       " 'Koury',\n",
       " 'Kattan',\n",
       " 'Shamoon',\n",
       " 'Rahal',\n",
       " 'Kalb',\n",
       " 'Naser',\n",
       " 'Masih',\n",
       " 'Sayegh',\n",
       " 'Dagher',\n",
       " 'Asker',\n",
       " 'Maroun',\n",
       " 'Dagher',\n",
       " 'Sleiman',\n",
       " 'Botros',\n",
       " 'Sleiman',\n",
       " 'Harb',\n",
       " 'Tahan',\n",
       " 'Tuma',\n",
       " 'Said',\n",
       " 'Hadad',\n",
       " 'Samaha',\n",
       " 'Harb',\n",
       " 'Cham',\n",
       " 'Atiyeh',\n",
       " 'Haik',\n",
       " 'Malouf',\n",
       " 'Bazzi',\n",
       " 'Harb',\n",
       " 'Malouf',\n",
       " 'Ghanem',\n",
       " 'Cham',\n",
       " 'Asghar',\n",
       " 'Samaha',\n",
       " 'Khouri',\n",
       " 'Nassar',\n",
       " 'Rahal',\n",
       " 'Baz',\n",
       " 'Kalb',\n",
       " 'Rahal',\n",
       " 'Gerges',\n",
       " 'Cham',\n",
       " 'Sayegh',\n",
       " 'Shadid',\n",
       " 'Morcos',\n",
       " 'Shamoon',\n",
       " 'Hakimi',\n",
       " 'Shamoon',\n",
       " 'Qureshi',\n",
       " 'Ganim',\n",
       " 'Shadid',\n",
       " 'Khoury',\n",
       " 'Boutros',\n",
       " 'Hanania',\n",
       " 'Antoun',\n",
       " 'Naifeh',\n",
       " 'Deeb',\n",
       " 'Samaha',\n",
       " 'Awad',\n",
       " 'Asghar',\n",
       " 'Awad',\n",
       " 'Saliba',\n",
       " 'Shamoun',\n",
       " 'Mikhail',\n",
       " 'Hakimi',\n",
       " 'Mikhail',\n",
       " 'Cham',\n",
       " 'Halabi',\n",
       " 'Sarkis',\n",
       " 'Kattan',\n",
       " 'Nazari',\n",
       " 'Safar',\n",
       " 'Morcos',\n",
       " 'Khoury',\n",
       " 'Essa',\n",
       " 'Nassar',\n",
       " 'Haik',\n",
       " 'Shadid',\n",
       " 'Fakhoury',\n",
       " 'Najjar',\n",
       " 'Arian',\n",
       " 'Botros',\n",
       " 'Daher',\n",
       " 'Saliba',\n",
       " 'Saliba',\n",
       " 'Kattan',\n",
       " 'Hajjar',\n",
       " 'Nader',\n",
       " 'Daher',\n",
       " 'Nassar',\n",
       " 'Maroun',\n",
       " 'Harb',\n",
       " 'Nassar',\n",
       " 'Antar',\n",
       " 'Shammas',\n",
       " 'Toma',\n",
       " 'Antar',\n",
       " 'Koury',\n",
       " 'Nader',\n",
       " 'Botros',\n",
       " 'Bahar',\n",
       " 'Najjar',\n",
       " 'Maloof',\n",
       " 'Salib',\n",
       " 'Malouf',\n",
       " 'Mansour',\n",
       " 'Bazzi',\n",
       " 'Atiyeh',\n",
       " 'Kanaan',\n",
       " 'Bishara',\n",
       " 'Hakimi',\n",
       " 'Saliba',\n",
       " 'Tuma',\n",
       " 'Mifsud',\n",
       " 'Hakimi',\n",
       " 'Assaf',\n",
       " 'Nassar',\n",
       " 'Sarkis',\n",
       " 'Bitar',\n",
       " 'Isa',\n",
       " 'Halabi',\n",
       " 'Shamon',\n",
       " 'Qureshi',\n",
       " 'Bishara',\n",
       " 'Maalouf',\n",
       " 'Srour',\n",
       " 'Boulos',\n",
       " 'Safar',\n",
       " 'Shamoun',\n",
       " 'Ganim',\n",
       " 'Abadi',\n",
       " 'Koury',\n",
       " 'Shadid',\n",
       " 'Zogby',\n",
       " 'Boutros',\n",
       " 'Shadid',\n",
       " 'Hakimi',\n",
       " 'Bazzi',\n",
       " 'Isa',\n",
       " 'Totah',\n",
       " 'Salib',\n",
       " 'Shamoon',\n",
       " 'Gaber',\n",
       " 'Antar',\n",
       " 'Antar',\n",
       " 'Najjar',\n",
       " 'Fakhoury',\n",
       " 'Malouf',\n",
       " 'Salib',\n",
       " 'Rahal',\n",
       " 'Boulos',\n",
       " 'Attia',\n",
       " 'Said',\n",
       " 'Kassis',\n",
       " 'Bahar',\n",
       " 'Bazzi',\n",
       " 'Srour',\n",
       " 'Antar',\n",
       " 'Nahas',\n",
       " 'Kassis',\n",
       " 'Samaha',\n",
       " 'Quraishi',\n",
       " 'Asghar',\n",
       " 'Asker',\n",
       " 'Antar',\n",
       " 'Totah',\n",
       " 'Haddad',\n",
       " 'Maloof',\n",
       " 'Kouri',\n",
       " 'Basara',\n",
       " 'Bata',\n",
       " 'Antar',\n",
       " 'Shammas',\n",
       " 'Arian',\n",
       " 'Gerges',\n",
       " 'Seif',\n",
       " 'Almasi',\n",
       " 'Tuma',\n",
       " 'Shamoon',\n",
       " 'Khoury',\n",
       " 'Hakimi',\n",
       " 'Abboud',\n",
       " 'Baz',\n",
       " 'Seif',\n",
       " 'Issa',\n",
       " 'Nazari',\n",
       " 'Harb',\n",
       " 'Shammas',\n",
       " 'Amari',\n",
       " 'Totah',\n",
       " 'Malouf',\n",
       " 'Sarkis',\n",
       " 'Naser',\n",
       " 'Zogby',\n",
       " 'Handal',\n",
       " 'Naifeh',\n",
       " 'Cham',\n",
       " 'Hadad',\n",
       " 'Gerges',\n",
       " 'Kalb',\n",
       " 'Shalhoub',\n",
       " 'Saliba',\n",
       " 'Tannous',\n",
       " 'Tahan',\n",
       " 'Tannous',\n",
       " 'Kassis',\n",
       " 'Shadid',\n",
       " 'Sabbag',\n",
       " 'Tahan',\n",
       " 'Abboud',\n",
       " 'Nahas',\n",
       " 'Shamoun',\n",
       " ...]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_lines['Arabic']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "category = Russian / line = Rokhmanov\n",
      "category = Czech / line = Morek\n",
      "category = Spanish / line = Fuentes\n",
      "category = Dutch / line = Oomen\n",
      "category = Japanese / line = Tamura\n",
      "category = Irish / line = Quirke\n",
      "category = Italian / line = Albricci\n",
      "category = Greek / line = Gavrilopoulos\n",
      "category = Japanese / line = Ninomiya\n",
      "category = Spanish / line = Sala\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "def randomChoice(l):\n",
    "    return l[random.randint(0, len(l) - 1)]\n",
    "\n",
    "def randomTrainingExample():\n",
    "    #lang\n",
    "    category = randomChoice(all_categories)\n",
    "    #name\n",
    "    line = randomChoice(category_lines[category])\n",
    "    #\n",
    "    category_tensor = Variable(torch.LongTensor([all_categories.index(category)]))\n",
    "    line_tensor = Variable(lineToTensor(line))\n",
    "    return category, line, category_tensor, line_tensor\n",
    "\n",
    "for i in range(10):\n",
    "    category, line, category_tensor, line_tensor = randomTrainingExample()\n",
    "    print('category =', category, '/ line =', line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 1, 57])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 16])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练网络\n",
    "--------------------\n",
    "\n",
    "现在训练这个网络所需要的就是向大家展示一些例子, 让它猜测, 并告诉它是否是错误的.\n",
    "\n",
    "对于损失函数 ``nn.NLLLoss`` 是适当的, 因为RNN的最后一层是 ``nn.LogSoftmax``.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.NLLLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每个训练循环都会:\n",
    "\n",
    "-  创建输入和目标张量\n",
    "-  创建一个归零的初始隐藏状态\n",
    "-  读入每个字母\n",
    "\n",
    "   -  为下一个字母保持隐藏状态\n",
    "\n",
    "-  比较最终输出与目标\n",
    "-  反向传播\n",
    "-  返回输出和损失\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 1, 57])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.005 # 如果设置得太高, 可能会爆炸. 如果太低, 可能无法学习.\n",
    "\n",
    "def train(category_tensor, line_tensor):\n",
    "    hidden = rnn.initHidden()\n",
    "\n",
    "    rnn.zero_grad()\n",
    "\n",
    "    for i in range(line_tensor.size()[0]):\n",
    "        output, hidden = rnn(line_tensor[i], hidden)\n",
    "\n",
    "    loss = criterion(output, category_tensor)\n",
    "    loss.backward()\n",
    "\n",
    "    # 将参数梯度添加到它们的值,再乘以学习速率\n",
    "    for p in rnn.parameters():\n",
    "        p.data.add_(-learning_rate, p.grad.data)\n",
    "\n",
    "    return output, loss.data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们只需要运行一些例子. 由于 ``train`` 函数返回输出和损失,我们可以打印它的猜测,并记录绘图的损失\n",
    "既然有1000个例子, 我们只打印每个 ``print_every`` 的例子, 并取平均的损失.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F:\\app\\anaconda\\lib\\site-packages\\ipykernel_launcher.py:18: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000 5% (0m 4s) 2.9634 Pae / Chinese ✗ (Korean)\n",
      "10000 10% (0m 9s) 1.2760 Cao / Chinese ✗ (Vietnamese)\n",
      "15000 15% (0m 14s) 0.9743 Bahar / Arabic ✓\n",
      "20000 20% (0m 18s) 1.0636 Shriver / German ✓\n",
      "25000 25% (0m 23s) 1.7162 Rhee / Korean ✓\n",
      "30000 30% (0m 27s) 1.3450 Sokolik / Polish ✗ (Czech)\n",
      "35000 35% (0m 32s) 0.5193 Dang / Vietnamese ✓\n",
      "40000 40% (0m 36s) 1.7789 Elstone / Scottish ✗ (English)\n",
      "45000 45% (0m 41s) 0.9809 Watt / Scottish ✓\n",
      "50000 50% (0m 46s) 1.2113 Pagano / Italian ✓\n",
      "55000 55% (0m 50s) 0.4146 Gorka / Polish ✓\n",
      "60000 60% (0m 55s) 2.8518 Ba / Korean ✗ (Arabic)\n",
      "65000 65% (1m 0s) 1.5045 Wasem / Arabic ✓\n",
      "70000 70% (1m 5s) 0.0015 Zdunowski / Polish ✓\n",
      "75000 75% (1m 9s) 1.1991 Salomon / French ✓\n",
      "80000 80% (1m 14s) 1.2694 Maria / Spanish ✗ (Portuguese)\n",
      "85000 85% (1m 18s) 0.3671 Shimaoka / Japanese ✓\n",
      "90000 90% (1m 23s) 3.6131 Duncan / Irish ✗ (Scottish)\n",
      "95000 95% (1m 28s) 2.2451 Kerr / German ✗ (Scottish)\n",
      "100000 100% (1m 32s) 2.5636 Girard / Scottish ✗ (French)\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "n_iters = 100000\n",
    "print_every = 5000\n",
    "plot_every = 1000\n",
    "\n",
    "\n",
    "\n",
    "# 跟踪绘图的损失\n",
    "current_loss = 0\n",
    "all_losses = []\n",
    "\n",
    "def timeSince(since):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "for iter in range(1, n_iters + 1):\n",
    "    category, line, category_tensor, line_tensor = randomTrainingExample()\n",
    "    output, loss = train(category_tensor, line_tensor)\n",
    "    current_loss += loss\n",
    "\n",
    "    # 打印循环数,损失,名称和猜测\n",
    "    if iter % print_every == 0:\n",
    "        guess, guess_i = categoryFromOutput(output)\n",
    "        correct = '✓' if guess == category else '✗ (%s)' % category\n",
    "        print('%d %d%% (%s) %.4f %s / %s %s' % (iter, iter / n_iters * 100, timeSince(start), loss, line, guess, correct))\n",
    "\n",
    "    # 将当前损失平均值添加到损失清单\n",
    "    if iter % plot_every == 0:\n",
    "        all_losses.append(current_loss / plot_every)\n",
    "        current_loss = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制结果\n",
    "--------------------\n",
    "\n",
    "从 ``all_losses`` 绘制历史损失显示网络学习:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2b761dc6e48>]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VNX9//HXZzJZyL6SQBayQNiXQNiVRRRxK2KtxVqX1pbWtn7V2s1+f98udrG1rbVarbWKS1VsVVRUUFBBZCesAUJCSFhC9oWEJIRs5/fHDDGBhIRkwiQzn+fjkQeZe8/MfG7HvnPn3HPPEWMMSiml3IfF2QUopZS6tDT4lVLKzWjwK6WUm9HgV0opN6PBr5RSbkaDXyml3IwGv1JKuRkNfqWUcjMa/Eop5Waszi6gPeHh4SY+Pt7ZZSilVL+xY8eOUmNMRFfa9sngj4+PJy0tzdllKKVUvyEiR7vaVrt6lFLKzWjwK6WUm9HgV0opN6PBr5RSbkaDXyml3IwGv1JKuRkNfqWUcjMuE/x1DU08u/4w23LLnV2KUkr1aS4T/ABLNxzhD6sy0HWElVKqYy4T/D6eHtw7byg7j51kbWaxs8tRSqk+y2WCH+CW1FjiQn3500dZNDfrWb9SSrXHpYLf08PCD69KJqOgipX7CpxdjlJK9UkuFfwAN4wfTHKkP4+tyaKxqdnZ5SilVJ/jcsHvYREenD+cnJIalu884exylFKqz3G54AeYPyqSlLhg/vjhQSpq6p1djlJK9SkuGfwiwu8XjaXydAO/W5nh7HKUUqpPccngBxg5KJAlsxJ5c0ceG7NLnV2OUkr1GS4b/AD/M28Y8WG+/PztdE7XNzm7HKWU6hNcOvh9PD34/U1jOVpWy5OfHnJ2OUop1Se4dPADzEgKZ+GEwbyw8QjleqFXKaU6D34RiRWRtSKSISL7ReS+dtr8WER223/2iUiTiITa9x0RkXT7PqesoP6DuUM53dDECxtznfH2SinVp3TljL8ReNAYMxKYBnxfREa1bmCM+ZMxZoIxZgLwEPCZMab1NJlz7ftTHVb5RRgWGcCC0VG8uOkIVXUNzihBKaX6jE6D3xhTYIzZaf/9FJABRF/gKbcCyxxTnuN8f+5QTtU18u/NR51dilJKOdVF9fGLSDyQAmztYL8vsAB4q9VmA6wWkR0isqR7Zfbc2JggZidHsHRDro7wUUq5tS4Hv4j4Ywv0+40xVR00uwHYeE43z0xjzETgGmzdRLM6eP0lIpImImklJSVdLeui/OCKoZTV1LNs27FeeX2llOoPuhT8IuKJLfRfNcYsv0DTxZzTzWOMybf/Wwy8DUxp74nGmGeNManGmNSIiIiulHXRJseHMiU+lH99nkN9o07gppRyT10Z1SPA80CGMeaxC7QLAmYD77ba5iciAWd/B+YD+3padE98b24SBZV1vLNbJ3BTSrknaxfazARuB9JFZLd928+BOABjzDP2bYuA1caYmlbPjQTetv3twAq8Zoz50BGFd9fs5AhGDQrkmc8Oc/PEGCwWcWY5Sil1yXUa/MaYDUCn6WiMeRF48ZxtOcD4btbWK0SEe+Ykce+yXaw+UMiCMYOcXZJSSl1SLn/nbnuuGRPFkDBf/rHusC7MrpRyO24Z/FYPC9+ZlcSevEo2Hy5zdjlKKXVJuWXwA9w0MZqIAG/+uT7H2aUopdQl5bbB7+PpwXVjB5F2pJzmZu3uUUq5D7cNfoDhUQHU1Ddx4uRpZ5eilFKXjNsHP8DBwlNOrkQppS4dtw7+5Ehb8GcVafArpdyHWwe/v7eVmJABesavlHIrbh38ACOiAsgs7GjOOaWUcj1uH/zDowLIKanRSduUUm7D7YM/OTKAxmZDTmm1s0tRSqlLwu2Df0RUIACZ2s+vlHITbh/8CeF+WC2iF3iVUm7D7YPfy2ohKcKfLA1+pZSbcPvgB9sFXj3jV0q5Cw1+bMF/4uRpTtU1OLsUpZTqdRr8wPCWO3h1ZI9SyvVp8PPFnD06skcp5Q40+IHo4AH4eXnoHbxKKbegwQ9YLEJyVACZOlmbUsoNaPDbjYgK5EB+FU26KItSysV1GvwiEisia0UkQ0T2i8h97bSZIyKVIrLb/vOLVvsWiEimiGSLyM8cfQCOMi0xlKq6RvbmnXR2KUop1ausXWjTCDxojNkpIgHADhFZY4w5cE67z40x17feICIewFPAVUAesF1EVrTzXKe7fFgEIrA+q5SUuBBnl6OUUr2m0zN+Y0yBMWan/fdTQAYQ3cXXnwJkG2NyjDH1wOvAwu4W25tC/bwYGx3E+kMlzi5FKaV61UX18YtIPJACbG1n93QR2SMiq0RktH1bNHC8VZs8OvijISJLRCRNRNJKSpwTvrOGRbD7+EkqT+uNXEop19Xl4BcRf+At4H5jzLnjHncCQ4wx44EngXfOPq2dl2r36qkx5lljTKoxJjUiIqKrZTnUrOQImpoNm7JLnfL+Sil1KXQp+EXEE1vov2qMWX7ufmNMlTGm2v77SsBTRMKxneHHtmoaA+T3uOpekhIXjL+3Vbt7lFIurSujegR4HsgwxjzWQZsoeztEZIr9dcuA7cAwEUkQES9gMbDCUcU7mqeHhRlJYazPKsUYHdaplHJNXRnVMxO4HUgXkd32bT8H4gCMMc8ANwP3iEgjcBpYbGzJ2SgiPwA+AjyApcaY/Q4+BoealRzB6gNFHC6pYehAf2eXo5RSDtdp8BtjNtB+X33rNn8H/t7BvpXAym5V5wSzk23XF9ZnlWjwK6Vckt65e47YUF8Swv20n18p5bI0+NsxOzmCLTll1DU0ObsUpZRyOA3+dsxOjqCuoZltueXOLkUppRxOg78d0xLD8LJaWJep3T1KKdejwd+OAV4eTEsMY11WsbNLUUoph9Pg78Cc5AhySmo4Vlbr7FKUUsqhNPg7MGe4bVinnvUrpVyNBn8HEsL9iAv11X5+pZTL0eDvgIgwZ3gEmw6X6rBOpZRL0eC/gDnDdVinUsr1aPBfwPTEcB3WqZRyORr8F6DDOpVSrkiDvxNXjhxITkkN249od49SyjVo8Hfi5kkxDAzw5g+rDuoc/Uopl6DB3wlfLyv3X5nMjqMVrDlQ5OxylFKqxzT4u+CW1BgSI/x49KNMGpuanV2OUkr1iAZ/F1g9LPzk6hFkF1fz5o48Z5ejlFI9osHfRVePjmRiXDB//ThLb+hSSvVrGvxdJCI8cFUyRVVn+Gh/obPLUUqpbtPgvwgzk8IZHOTD8p0nnF2KUkp1W6fBLyKxIrJWRDJEZL+I3NdOm9tEZK/9Z5OIjG+174iIpIvIbhFJc/QBXEoWi7BoYjSfHyqhqKrO2eUopVS3dOWMvxF40BgzEpgGfF9ERp3TJheYbYwZB/wGePac/XONMROMMak9rtjJbpoYQ7OBd3frWb9Sqn/qNPiNMQXGmJ32308BGUD0OW02GWMq7A+3ADGOLrSvSIrwZ0JsMG/tOKE3dCml+qWL6uMXkXggBdh6gWZ3A6taPTbAahHZISJLLrbAvujLk2LILDrF/vwqZ5eilFIXrcvBLyL+wFvA/caYdhNPROZiC/6ftto80xgzEbgGWzfRrA6eu0RE0kQkraSkb8+GecO4QXh5WHhrp47pV0r1P10KfhHxxBb6rxpjlnfQZhzwHLDQGFN2drsxJt/+bzHwNjClvecbY541xqQaY1IjIiIu7igusWBfL+aNHMiK3fk06J28Sql+piujegR4HsgwxjzWQZs4YDlwuzEmq9V2PxEJOPs7MB/Y54jCne3GlGjKaup11k6lVL9j7UKbmcDtQLqI7LZv+zkQB2CMeQb4BRAGPG37O0GjfQRPJPC2fZsVeM0Y86FDj8BJpiWGIQLbcyuYkRTu7HKUUqrLOg1+Y8wGQDpp8y3gW+1szwHGn/+M/i9ogCfDIwP0jF8p1e/onbs9MCUhlJ3HKnTGTqVUv6LB3wOT40OprW/iQIEO61RK9R8a/D0wOT4UgG252t2jlOo/NPh7ICrIh9jQAaQdqei8sVJK9REa/D00OT6U7UfKdfoGpVS/ocHfQ1PiQymrqSentMbZpSilVJdo8PdQqr2fP02HdSql+gkN/h5KivAjzM+Lbbnaz6+U6h80+HtIREiND9EbuZRS/YYGvwNMjg/lWHmtrsqllOoXNPgdYFpiGACfHix2ciVKKdU5DX4HGD04kOGRAby+7ZizS1FKqU5p8DuAiHDrlFj25FWy70Sls8tRSqkL0uB3kEUpMXhbLby+/Yuz/rqGJn61Yj+ZhaecWJlSSrWlwe8gQb6eXDduEO/syqe2vhGAh98/wIubjrB0Q66Tq1NKqS90ZSEW1UW3Tolj+c4TvL+nAE+r8NrWYwR4W/k4o4imZoOH5YLLGiil1CWhZ/wOlDokhKED/Xnms8P8fPk+piSE8ttFYyirqdc7e5VSfYYGvwPZLvLGkVNag5+3B3+/NYV5IyPx8rCw+kCRs8tTSilAg9/hbp4Yw5UjI3n6tkkMDPTB39vKzKFhfLS/UGfwVEr1CRr8Dhbk68lzd6YyJSG0ZdvVo6PIqzhNRoGO7lFKOZ8G/yUwb2QkIvDR/kJnl6KUUp0Hv4jEishaEckQkf0icl87bUREnhCRbBHZKyITW+27U0QO2X/udPQB9AcRAd6kDgnR4FdK9QldOeNvBB40xowEpgHfF5FR57S5Bhhm/1kC/ANAREKBXwJTgSnAL0UkxEG19ytXj47iYOEpjpbVUFnbwNGyGhqamp1dllLKDXUa/MaYAmPMTvvvp4AMIPqcZguBl43NFiBYRAYBVwNrjDHlxpgKYA2wwKFH0E/MHxUFwOw/rWP8w6uZ/ad1/H5lhpOrUkq5o4u6gUtE4oEUYOs5u6KB460e59m3dbTd7cSF+fLwwtEUV50hxM+LVekFrEwv4BfXj0JEb+xSSl06XQ5+EfEH3gLuN8ZUnbu7naeYC2xv7/WXYOsmIi4urqtl9St3TI9v+T14gCcPvrGHfSeqGBsT5LyilFJup0ujekTEE1vov2qMWd5OkzwgttXjGCD/AtvPY4x51hiTaoxJjYiI6EpZ/drcEQOxCHycoTd2KaUura6M6hHgeSDDGPNYB81WAHfYR/dMAyqNMQXAR8B8EQmxX9Sdb9/m9kL9vJg0JOS84P/zR5k8vS7bSVUppdxBV7p6ZgK3A+kistu+7edAHIAx5hlgJXAtkA3UAt+w7ysXkd8A2+3Pe9gYo5PW2F05MpJHVh0k/+RpBgcPYH9+JX9fm02At5VvzkzAx9PD2SUqpVxQp8FvjNlA+331rdsY4Psd7FsKLO1WdS7uylG24P8ko4jbp8fzp48y8bAIp840sj6rhPmjo5xdolLKBemdu06UFOFPYrgfazKK2ZJTxrrMEn54VTIhvp58kF7g7PKUUi5K5+N3snkjB/LSpqNU1NQTGejN3ZclkFdRy4rd+dQ1NGl3j1LK4fSM38muHBlJfVMz6Scquf/KZHw8Pbhu7GBq6ptYl1ni7PKUUi5Ig9/JJg0JIcTXk8RwP74yKQaAaYmhhPp5aXePUqpXaFePk1k9LDx3ZypBAzyxelhati0YE8U7u05wur6JAV7a3aOUchw94+8DJg0JZejAgDbbrh87iNr6JtZlFjupKqWUq9Iz/j5qSkIo4f5e/Oq9/fxlTRal1WcYNtCf5++aTKCPp7PLU0r1Y3rG30dZPSzcN28YcaG+DBvoz4LRUew6dpJ7XtlBfaNO56yU6j494+/Dbp8ez+2tJnZLjQ/lR2/s4X/fTufRm8fprJ5KqW7R4O9Hbp4Uw/HyWv72ySHiQn25d94wZ5eklOqHtKunn7n/ymHcOGEwf/04i+ziameXo5TqhzT4+xkR4f+uH4WPpwd/++SQs8tRSvVDGvz9UJi/N3fNiOf9vflkFp5ydjlKqX5Gg7+fWjIrEX8vK39dk+XsUpRS/YwGfz8V7OvFNy9L4MP9hew7UenscpRS/YgGfz929+UJBA3w5A+rDlJeU+/scpRS/YQGfz8W6OPJvVcMZUN2KZN/9zF3LN3G+3vbXdJYKaVa6Dj+fu5blycyLTGMD9ILeH9vPj94bRcDA3yYkhDq7NKUUn2UnvG7gDHRQfx0wQg+un8WQQM8eX5DTpv9tfWNPPd5DnUNTU6qUCnVl2jwuxBfLytfmxrH6gNFHCurbdn+2OosfvtBBm/tzHNidUqpvkKD38XcOT0eDxFe2JQLQGbhKV7YdASAd3adcGJlSqm+otPgF5GlIlIsIvs62P9jEdlt/9knIk0iEmrfd0RE0u370hxdvDpfVJAP148bxH+3H6fydAP/9+4+Anys3H1ZAtuPVJBXUdv5iyilXFpXzvhfBBZ0tNMY8ydjzARjzATgIeAzY0x5qyZz7ftTe1aq6qq7L0ukpr6JJS+nsS23nJ8uGMFdM+IBeHe3jvpRyt11GvzGmPVAeWft7G4FlvWoItVjY2OCmBIfytbccsbHBvPV1FhiQ32ZNCSEd3efwBjj7BKVUk7ksD5+EfHF9s3grVabDbBaRHaIyBJHvZfq3PfmJjHA04PfLByNxWKbt//GCYPJKqomo6Dz+X0+PlDE8XLtFlLKFTny4u4NwMZzunlmGmMmAtcA3xeRWR09WUSWiEiaiKSVlJQ4sCz3NGf4QPb9+mrGxQS3bLtu3GCsFuHd3Re+yFteU8+Sf6fxuw8yzttX39hMY5OuAKZUf+bI4F/MOd08xph8+7/FwNvAlI6ebIx51hiTaoxJjYiIcGBZ7svD0naFrlA/L2YlR7BiTz7NzR1393ySUUSzgbWZxVSfaWzZbozh1n9t4cdv7u21mpVSvc8hwS8iQcBs4N1W2/xEJODs78B8oN2RQerSuTElmoLKOtZmFnfYZvWBIrysFs40NvNJRlHL9vQTlew4WsH6rBK9TqBUP9aV4ZzLgM3AcBHJE5G7ReS7IvLdVs0WAauNMTWttkUCG0RkD7AN+MAY86Eji1cXb/6oSJIi/Pjft/dR0c7EbrX1jazPKmHx5FgiA715f29By75l244BUFZTz/Hy05esZqWUY3U6V48x5tYutHkR27DP1ttygPHdLUz1Dh9PD/62OIVFT2/k52+n8/RtE9ss2r4+q5Qzjc0sGB2Fh0V4dcsxTtU1ICK8uzuf8TFB7MmrZOexCuLCfJ14JEqp7tI7d93QmOggfjR/OKv2FfJGWttpHFYfKCRogCdTEkK5ftxg6puaWXOgiBW786mtb+L/rh+Fr5cHO49VOKl6pVRP6eycburblyfyWVYJv3pvPylxwQyLDKCxqZlPMoqZN3IgVg8LKbHBDA7y4YO9BZRUn2FEVACThoQwPiZYg1+pfkzP+N2UxSL85Zbx+Hp5sPjZLezPr2RbbjmVpxuYPyqqpc21YwexNrOYvXmVLJ4ci4gwcUgwGQWnqK1v7ORdlFJ9kQa/GxsUNID/fmc63lYLi5/dwlPrsvHxtDA7+YvhtNePH0yzAW+rhUUpMQBMjAuhqdmQnqdLPirVH2nwu7nECH/euGcG4f7ebMwu4/JhEQzw8mjZPz4miORIf748KYYgX08AUuJCANh57KRTalZK9Yz28Suig21n/r9asZ+7Zsa32ScivHfvZXi0GvkT6udFQrhfj/r5D+RX8efVmVw/bhDXjxuMl1XPQZS6VPT/bQqAiABvnrptIpPjz1+y0dvqgdWj7X8qKXHB7DpW0emNXB/tL+RXK/bT0Gqah6Zmw8+W72VtZjE//O8eLvvjpzzz2WG9KUypS0SDX3XLxLgQSqsvfCPXCxtz+e4rO3hx0xEe/fBgy/b/bD/O3rxK/nrLBF765hSGRfrzh1UHWZelczQpdSlo8KtumWjv5991/PzunuZmwyMrM/j1ewe4amQkt06J5V+f57LmQBEVNfU8+tFBpiaEsnDCYGYnR/D8nZPxslrYeKj0Uh+GUm5J+/hVtyRH+ttu5DpawcIJ0S3bm5oNP3lzL2/tzOP2aUP41ZdG09DUzN68Sn70xh6mJ4Zxqq6RhxeOablj2MfTg0lxIWw8XOasw1HKregZv+oWq4eFyfGh/Dctj7d32e7+bWxq5of/3c1bO/N44MpkHl44Gg+L4OPpwdO3TaS52fDh/kLunB7P8KiANq83c2gYGQVVlFWfccbhKOVWNPhVtz168zjGxgTxwH/28NM393L/f3bz7u58fnz1cO67clibOYCGhPnxt1snMHd4BPdfNey815oxNByAzTl61q9Ub9OuHtVtkYE+vPatqTz+8SH+vjYbgIeuGcF3Zie12/6KEZFcMSKy3X3jooMI8LayMbuM68cN7rWaW8streHZ9YeprW/i8a9OaPOHSilXpsGvesTqYeFHVw9n5tBwKmrruXbsoG6/ztTEUDYfbnuBt/J0A0EDPB1Raou8iloeWXmQlfsKODuC9KcLRjA4eIBD30epvkq7epRDTE8K63bonzUjKZwjZbWcOGkbIrr5cBmTfrOGpRtyHVFii9+vzODTg8XcMzuJ5+9MBWCX3oWs3IgGv+ozZtr7+Tdml1J9ppEfvbGHxmbDY2uyKK6qc8h7GGPYklPOdeMG8ZMFI7h8WAReVgu72xmWqpSr0uBXfUZypD/h/t5syi7ldx8coKDyNH/5ynjqG5v5Q6sbwHriUHE15TX1TE2w3aHsZbUwZnCgnvErt6LBr/oMEWFGUhgf7i9k2bbjLJmVxJcnxXD35Qks33mCHUfLe/weW+yjhqYlhrVsmxAbQvqJyjbTSijlyjT4VZ8yc2gYdQ3NJEf684B92OcP5g4lKtCHX67YT1Nz1+fzeeKTQzzz2eE227bmlBMdPICYkC8u5KbEBXOmsZmDBacccxBK9XEa/KpPmTcykumJYTz+1RS8rbbpof28rTx07Qj2najirR15nbyCTUZBFX/9OIvHP86i8nQDYOvf35pbxtSE0DZDNyfEBgNoP79yGxr8qk8J9/dm2ZJpjBoc2Gb7l8YPZlxMEE+ty6axC10yj354EC8PC3UNzazYfQKAwyXVlFbXMzWx7QykMSEDCPf31n5+5TY6DX4RWSoixSKyr4P9c0SkUkR2239+0WrfAhHJFJFsEfmZIwtX7kVE+N6coRwtq+WD9II2++oammhu1QW0+XAZazNLeOCqZEYOCuT17cdt23Ns1wha9++ffe2UuGB2HdfgV+6hK2f8LwILOmnzuTFmgv3nYQAR8QCeAq4BRgG3isionhSr3Nv8UZEkR/rz1NrslqA/Xl7L5Y+u5donPmdv3kmMMfzhw4MMCvLhrhnx3Dollv35VaTnVbI1p4yoQB/iQn3Pe+0JscHkltZQUVPf5XqMMWzMLuV0fZPDjlGpS6HT4DfGrAe6M5xiCpBtjMkxxtQDrwMLu/E6SgG2xd+/N2coWUXVrMkoorK2gbte2EZ9YzMVtfXc+NRGvv1yGnuOn+SBq5Lx8fRg4YRovK0Wlm0/xpaccqYlhrY7NUNKnL2fP8921v/y5iNc8ed1F7x/YMWefG57bit3vbCNmjO68LzqPxzVxz9dRPaIyCoRGW3fFg0cb9Umz76tXSKyRETSRCStpEQX5FDtu37cIOJCffn7p9l855U0jpef5tnbJ7H6gdl8dXIsH2cU29YInmhbGD5ogCfXjR3EG2nHKa0+w9RzunnOGhcTjAjsPnaS5Tvz+MW7+8kpreGlzUfabV/X0MSjH2YyOMiHtKMV3PXCNqo1/FU/4Yjg3wkMMcaMB54E3rFvb2/Gqw7H4hljnjXGpBpjUiMiIhxQlnJFVg8L98xJIv1EJVtyyvnTV8YxNTGMoAGePHLTON6/9zJe+MYUPCxf/Oe3eEocDU22//TO3rh1Ln9vK8MjA3hzRx4/fnMvM5LCmDdiIK9sOUZt/fmB/vyGXE6cPM1fbpnAE4tT2HnsJHc8v5Xj5bUXrL/mTCN5FbWU19RT19Cky00qp+jxJG3GmKpWv68UkadFJBzbGX5sq6YxQH5P30+pmyZG896efK4YMbDNIjAAY6KDzms/OT6EpAg/TtU1khDu1+HrTogN5vXtxxkfG8yzd6SSWVjFl/+xmTd35HHH9PiWdqXVZ/jHusNcOTKS6Um2bxAWgXuX7eLyR9cyJjqQa8YM4uZJMUQG+rQ87+MDRTz4xp6W4aVgu27x9G0Tz1vTWKne1OPgF5EooMgYY0RkCrZvEWXASWCYiCQAJ4DFwNd6+n5KeVs9eO3b07rcXkT42+IUqs80XnDq5UUp0ZTX1PPHL4/D39vKpCGhpMQF8/yGXG6bOqTlW8Rf12RR19DEQ9eOaHnuNWMHsTY6iFX7Cli1r5A/fZTJ3z45xNemxPHtWYm8vOkI/1yfw+jBgfz82hGcrm/iSFktL246wq/fO8DDC0frtNDqkuk0+EVkGTAHCBeRPOCXgCeAMeYZ4GbgHhFpBE4Di43t+2ujiPwA+AjwAJYaY/b3ylEo1Yn2vgmca2pi2HnXAL59eSLfe3Unaw4UccWIgfx7y1GWbTvGHdPjSYrwb9M2NtSXJbOSWDIriaNlNTy99jCvbDnKi5uOAPD1aXH8v+tG4ePp0fIcb6uFf67PITHCj2/MTOj5gSrVBdIX+xhTU1NNWlqas8tQisamZub8eR0DPD1obDbkltYwKzmCJxenEOTb+ToBx8trWboxl9QhoVw37vxpq5ubDd99ZQcfZxTx/J2TmTtiYJv9tfWNVJ5uYFCQrhWgLkxEdhhjUrvSVjsWlboAq4eFuy9L4FBxNRaBF+6azEvfmNyl0Afbt4Bf3jC63dAH2xDVxxdPIDkygF+/t/+8i72/+yCDeX/5rNOLxu1pbjZkFp7SC8jqPBr8SnXijunxvPbtqXx4/yzmjhjo8L54Xy8rX582hCNltWQXV7dsb2o2fLS/kNr6Jn785p42dyc3NjVztKym3VBvbjasTC/g2ic+5+rH1/Pkp9kOrVf1f7r0olKd8LAIM5LCe/U9rhoVyf97Zx+rDxQxLDIAgF3HKiitrmfu8AjWZpbwytaj3DE9noqaer77yg625pYTH+bLTRNjmDM8giNltaTnnWRdZgmHiqtJjPBjRlIYj3+cxZSE0POmqrgYlbW2kUhd/aaj+jY941eqD4gM9GF8TBDjamLSAAARYklEQVRrDhS1bFt9oAhPD+Fvt6YwKzmCR1YeZG1mMTc+vZFdx09y7xVDGRQ0gMfWZPGlv2/kf5bt4qXNRwkc4MnfFk9gzQOzefaOVOLD/Ljv9V2UVZ/pVm11DU0s+sdGfrBsp6MOVzmZnvEr1UdcNSqSP6+2LTMZEeDNR/sLmZEUTqCPJ3/88ljmP7aeb7ywnXB/L5Z9exqThoQAtsXjdxytICnCn+FRAXi2uifA39vKk19LYdHTm3jwjT0svXMyFsvFdVU9+ekhckpqKKuuxxijw05dgJ7xK9VHXDUqCoCPM4o5VFzN0bJa5o+OBGBQ0AAevXkcs5IjePt7M1tCHyAmxJeFE6IZEx3UJvTPGj04iP+7fhTrMkt4Y8fx8/ZfSEZBFf/8LIcQX08qTzdQfKp73xo6U9/YzBOfHOr2t5KqugadL+kiaPAr1UckR/oTF+rL6gOFrN5fCMBVIyNb9l8zdhAvf3MKse3MLtqZr0+NY3xMEE+tPdyl9QzAdnH5Z8vTCRrgye8XjQUgs7B3Vilbc6CIx9Zk8fLmo916/t0vbue+13c7uKqO1Tf272U6NfiV6iNEhKtGRbIpu4wVe/JJiQtmYKspH3r62t+bO5Rj5bW8v/eL9QxO1tZz3ROf89jqzPOe89KmI+w5fpJf3DCq5ca23gr+d+yL5azaV9BJy/NV1jaQdrSCTYdLL8m6yaXVZ5jw8Gqe35Db6+/VWzT4lepDrhoVSX1TM1lF1cy3d/047LVHtl3PwBjDz95KZ39+FU98mt0myN7akcfvVmYwd3gEXxo/mFA/LyICvMkscnzwn6ytZ11mMREB3mQVVZNdfHHvsSW3DGOgtr6J9BOVDq/vXGlHKqitb+KRlRnsOtY/l+vU4FeqD0kdEkKwfcjk2f59R7FYhO/PHcqhYtt6Bq9vP86H+wv5yYLhLBgdxW/eP8B7e/JZuiGXB9/Yw9SEUJ782sSWi7nDIwPIOif4G5uae9zt8UF6AQ1Nhj/cZOtOWpleeFHP33y4DC+rLcq25XZn6ZCLs+t4BZ4eQmSgD/cu29Vm0r3+QoNfqT7E6mFh4fjBjI8NPm8uIEe4buwghoT58scPD/Lr9/Zz2dBwvjsriccXT2BKfCj3/2c3D79/gPmjIll612T8vb8Y+JdsD/6mVjeS/fHDg0x/5JPz/iBcjHd35ZMU4ccVIwaSOiSElekX192z6XApUxNCSYrwY2tOWbfr6Kpdx04yenAQT9yaQkFlHQ8t39vv7o7W4Feqj/nlDaNZfs+MXnltq4eFe2YnkVNSg6+XlcduGY/FIvh4evCvO1KZGBfMbVPjePq2iW0mkwMYERVAXUNzy/QRxhje21NAWU09tz23lSOlNZ2+f1n1GR5+7wD5J08DtqGo246UsyglGhHhmrGDOFh4ipyS6k5eyabk1BmyiqqZkRTO1MQw0o5UtPnD5GiNTc2k51UyITaYSUNC+PHVw1mZXsi7u/vXjPMa/Er1MRaLtFlIxtEWTYzm5kkx/P3WlDYXj4N8PXnjuzP43aKx7a4PkBxlu6P4bD//wcJTFFbVsWRWIk3Nhtue28oJe6B35LWtx1i6MZebnt7EgfyqlsA8u67CNWNs1zVW7etad88W+xn+jKQwpiaEcupMIxkFVZ08C17fdowbntxw0ReDM4tOcbqhqWWpziWXJzIiKoBn1+f0q7N+DX6l3Iy31YM/f2U8M4Ze3DQUwwbaup6y7CN71mYWA3D3ZQm8/M0pVNU18NV/bmbnBS54rtxXyNCB/ojALf/czL83H2XSkJCWIaqDgweQEhfMyvQCjDGszSzm9ue3smJP+2fUmw6XEeBjZfTgQKbYV1fb0kl3z4mTp3n4/QOkn6hk59GLuzi765htTeaJcbb7KCwW4Y7p8RwoqGLHRb6WM2nwK6W6xM/bSmzoAA7az/jXZZYwalAgkYE+jIkO4pW7p2IM3PyPTfxldeZ5Z9O5pTVkFFRx65Q4ln9vBjEhAyisquPGlLarqF07ZhD786v40t838o0XtrMlp4wH/7ubTYdLz6tp8+FSpiaEYfWwMChoAHGhvmy9wAVeYwy/eGcfxoDVIqzL6nh9730nKkn97RoO5H/xDWLXsZOE+3sRE/LFNNk3pgwmwMfa7XsQnEGDXynVZcMjA8kqPEXl6QZ2HK1g7ogv1sceHxvMqvsvZ1FKDE9+ms3Nz2ymrqGpZf/Zi7YLxkQxKGgAb3x3Or9fNJavTIpp8x7XjI3C00Moqz7D7xeNZctD84gP8+M7/97RZqjniZOnOVJWy4ykLyafm5oQyvYj5W1mMm1t1b5CPjlYzIPzk0mND2HtweIOj3XphlxKq+v51+c5Ldt2Ha9gQmxIm2krfL2sfGVSLCvTCyiuqmv3tTZml/boArijafArpbpseJQ/uaU1rD1YTFOzYc7wtgvHBPp48pdbxvPYLePZc/xky+pjYLs5a0JsMNHBtrPlAB9PvjY17ryLyDEhvnz64BzW/ngOX5saR5i/N0vvmoy31YO7Xtjech1h82F7//7QVsGfGMbJ2gayim2jj5ZuyOWB/+zmuc9z2Jhdyi9X7GdMdCB3zYhnzvCBtusUleeH9cnaet5PL2CApwfv782nqKqOytoGckpqWvr3W7t9+hAamw3LtrWdEqOxqZnfvn+A257bykPL0y/if+nepcGvlOqy5MgAGpsNSzfmEuhjJSX2/BAEuGliDPNGDOSpT7MprT7DsbJa9p2o4tqxXbspLTbUF2+rR5vHz9+ZSmn1GS7746fc8s/NvLz5CGF+XiQPDGhpN9Xez/9GWh63/HMzD79/gHWZxfz2gwxue24rZdVneGTROKweFuYMt31b+Szr/LP+N3fkUd/YzF+/OoHGZsMrW46yO8/Wv9/eMSeE+zErOYLXth1t6eIqr6nnzhe28dyGXOJCfdl9/CSn6vrGmH+dnVMp1WUjogIB2JtXyXXjBrU7+uesh64dydWPr+fxj7OIDbFdvL1mTPsrkXXF+NhgVt03i3d2nWDVvgKyiqpZlBLdZrbRmJABDA7y4fkNtj9Mf/3qeG6cEE1J9RnS8yrxtnowNsa2/vLwyACiAn1Yl1nCVyfHtbyGMYbXth0jJS6YBWOiuHJkJK9uPUZjs0EExnXwx+7O6UO4+6U0bn5mM1WnG8irqEVE+NPN44gOGcDX/rWVbbnlzBvp2BvzukODXynVZQnhflgtQmOzYU5yxAXbDh3oz21T43h16zEGBfkwNjqoWxPMnfv+D1yVzANXJXOsrJZQf682+0WEb8xMIP1EJT+/diRRQbbhqgMDfJg30ue8tnOGR/DB3gIamppbZjbdmltOTkkNf/7KeAC+MTOeNQeKeGFjLsMjA9rc1NbanOEDmZ0cQUVtPaMGBTJ/dCRfGj+Y0YODqGtowttqYUN2af8IfhFZClwPFBtjxrSz/zbgp/aH1cA9xpg99n1HgFNAE9DY1YWAlVJ9k5fVQmKEH1lF1cwefuHgB7hv3jDe3nWCvIrTfG1qXKftL0ZcWPt/RL49K7HLrzFn+EBe336cHUcrWlYoe3XrMQJ9rFxvXyd5emIYI6ICOFh4igkdnO2DbaW2l745pd19Pp4eTEkIZVN2799Z3BVd6eN/EVhwgf25wGxjzDjgN8Cz5+yfa4yZoKGvlGu4bGgElw8LZ2BA5zOHhvl7c9+8YVgtwnVju9/N01tmDg2zDevMtA3rPFJaw4f7CvjypJiWi84iwjcvSwBo98JuV81ICiez6BTFp9of+XMpdXrGb4xZLyLxF9i/qdXDLUBMR22VUv3fL24YdVHt774sgYUTookI8O6lirovwMeT1PgQ3t+bT0ZBFesPleDlYeHr04a0abcoJZozDU3cMH5wt9/rsqHh/BHYlF3Wcu+CMYZTZxoJ8LZe0pXNHN3HfzewqtVjA6wWEQP80xhz7rcBpZSLE5E+GfpnXTkykt9+kEFjk+HeuUP5SmrsedciPD0s3D49vkfvM2pwIEEDPNmYXcqNKdE0Nxu+9XIanx4sxttqYWCgN/Fhfvz77qk9ep+ucFjwi8hcbMF/WavNM40x+SIyEFgjIgeNMes7eP4SYAlAXJxj+wKVUqojd86IZ9KQEMZGB11wlFJPeViEGUlhbMwuxRjDs5/n8OnBYr4+LQ5fLyvFVXWX7KzfIcEvIuOA54BrjDEtVy+MMfn2f4tF5G1gCtBu8Nu/DTwLkJqa2n9mO1JK9WueHhZS4kI6b+gAM4aGs2pfISv25PPnjzK5ZkwUv1k45pIvYN/jP28iEgcsB243xmS12u4nIgFnfwfmA/t6+n5KKdVfXWafGO+H/91DRIA3j9w09pKHPnRtOOcyYA4QLiJ5wC8BTwBjzDPAL4Aw4Gn7AZwdthkJvG3fZgVeM8Z82AvHoJRS/UJ8mC+Dg3woqKrj8a9OINjXq/Mn9YKujOq5tZP93wK+1c72HGB890tTSinXIiI8dO1ITjc0tSxg7wx6565SSl1CPRkS6ig6SZtSSrkZDX6llHIzGvxKKeVmNPiVUsrNaPArpZSb0eBXSik3o8GvlFJuRoNfKaXcjBjT9+ZDE5ES4Gg3nx4OlDqwnP7AHY8Z3PO43fGYwT2P+2KPeYgxpvNl0eijwd8TIpLmbqt9ueMxg3setzseM7jncffmMWtXj1JKuRkNfqWUcjOuGPzuuLyjOx4zuOdxu+Mxg3sed68ds8v18SullLowVzzjV0opdQEuE/wiskBEMkUkW0R+5ux6eouIxIrIWhHJEJH9InKffXuoiKwRkUP2fy/NIqKXkIh4iMguEXnf/jhBRLbaj/k/IuKc5Yx6kYgEi8ibInLQ/plPd/XPWkQesP+3vU9ElomIjyt+1iKyVESKRWRfq23tfrZi84Q93/aKyMSevLdLBL+IeABPAdcAo4BbRWSUc6vqNY3Ag8aYkcA04Pv2Y/0Z8IkxZhjwif2xq7kPyGj1+I/AX+3HXAHc7ZSqetffgA+NMSOwrWiXgQt/1iISDfwPkGqMGQN4AItxzc/6RWDBOds6+myvAYbZf5YA/+jJG7tE8ANTgGxjTI4xph54HVjo5Jp6hTGmwBiz0/77KWxBEI3teF+yN3sJuNE5FfYOEYkBrgOesz8W4ArgTXsTVzzmQGAW8DyAMabeGHMSF/+ssa0MOEBErIAvUIALftbGmPVA+TmbO/psFwIvG5stQLCIDOrue7tK8EcDx1s9zrNvc2kiEg+kAFuBSGNMAdj+OAADnVdZr3gc+AnQbH8cBpw0xjTaH7viZ54IlAAv2Lu4nhMRP1z4szbGnAD+DBzDFviVwA5c/7M+q6PP1qEZ5yrBL+1sc+nhSiLiD7wF3G+MqXJ2Pb1JRK4Hio0xO1pvbqepq33mVmAi8A9jTApQgwt167TH3qe9EEgABgN+2Lo5zuVqn3VnHPrfu6sEfx4Q2+pxDJDvpFp6nYh4Ygv9V40xy+2bi85+9bP/W+ys+nrBTOBLInIEWzfeFdi+AQTbuwPANT/zPCDPGLPV/vhNbH8IXPmzvhLINcaUGGMagOXADFz/sz6ro8/WoRnnKsG/HRhmv/Lvhe1i0Aon19Qr7H3bzwMZxpjHWu1aAdxp//1O4N1LXVtvMcY8ZIyJMcbEY/tsPzXG3AasBW62N3OpYwYwxhQCx0VkuH3TPOAALvxZY+vimSYivvb/1s8es0t/1q109NmuAO6wj+6ZBlSe7RLqFmOMS/wA1wJZwGHgf51dTy8e52XYvuLtBXbbf67F1uf9CXDI/m+os2vtpeOfA7xv/z0R2AZkA28A3s6urxeOdwKQZv+83wFCXP2zBn4NHAT2Af8GvF3xswaWYbuO0YDtjP7ujj5bbF09T9nzLR3bqKduv7feuauUUm7GVbp6lFJKdZEGv1JKuRkNfqWUcjMa/Eop5WY0+JVSys1o8CullJvR4FdKKTejwa+UUm7m/wOUOKHgbiqVPAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2b761dd7780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估结果\n",
    "======================\n",
    "\n",
    "要查看网络在不同类别中的表现如何, 我们将创建一个混淆矩阵,\n",
    "为每个实际语言 (行) 指示网络猜测哪种语言 (列).\n",
    "为了计算混淆矩阵,一堆样本通过网络运行 ``evaluate()``,\n",
    "这和 ``train()`` 减去反向传播是一样的.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVwAAAEwCAYAAAD7IMkNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXecXVW5hp93hoQEAgEEkd6LgUsNTZAuYqGoKCBcCSrIlc5VAQsiFhQLl6ZSFBBBmqIBkSJFpKcQEhJAkCIRFUKvKTPv/WOtk+w5s885e0/OzDkzrCe//Zuz1/722uuUrL32t77vXbJNIpFIJPqfjlY3IJFIJN4ppA43kUgkBojU4SYSicQAkTrcRCKRGCBSh5tIJBIDROpwE4lEYoBIHW4ikUgMEKnDTSQSiQEidbiJRCIxQKQON5FI9CuSlpf0C0l/ivtjJH2u1e1qBanDTSQS/c1FwI3AinH/b8AxLWtNC0kdbiIxiJC0eKvb0AeWtX0l0A1gex7Q1domtYbU4SYSgwBJ75M0A3g47m8s6actblZR3pD0LsAAkrYGXmltk1rDIq1uQCKRKMTpwAeB8QC2H5S0fWubVJjjCO1eS9JdwHLAPq1tUmtIHW4iMUiw/YykbNGgeCy3PVnSDsB6gIBHbc9tcbNaQnIpJBKDg2ckvQ+wpOGSvkR0L7Q7kj4JjLQ9HdgbuELSZi1uVktIHW4iMTg4DDgcWAmYCWwS9wcD37D9mqTtCG6Ri4GftbhNLUFpxYdEItGfSHrA9qaSTgWm2b6sUtbqtg00aYSbSAwCJJ0maUlJwyTdImmWpANb3a6C/FPSucCngOslLco7tO95R77pRGIQspvtV4GPElwK6wJfbm2TCvMpQuLD7rZfBpZh8LS9qaQON5EYHAyLfz8M/Mb2i61sTBlsvwk8B2wXi+YBj7WuRa0jdbiJxODgWkmPAGOBWyQtB7zd4jYVQtI3geOBE2PRMODXrWtR60iTZolEBkmLAf8LrGr7EEnrAOvZvq7FTUPS0sCrtrtiiu8Stv/d6nY1QtIUYFNgcmWiTNJU2xu1tmUDTxrhJhI9uRCYDWwT92cC32ldcwLxRnA4C8KpViSMdgcDcxxGdpXU3sGoB9EUUoebSPRkLdunAXMBbL9FyI5qNRcCc4D3xf22uBEU5MoYpbCUpEOAPwPnt7hNLSGl9iYSPZkjaSQLRmNrEUa8rWYt2/tK2h/CjUBVeb7tiu0fSfoA8Cohvfck2ze3uFktIXW4iURPvgncAKwi6VJgW2BcS1sUaNcbQSFs3yzpPmKfI2mZwRRp0SzSpFkiUUWUEtya4Eq41/asFjeJOEL8OjAGuIl4I7B9eyvbVQRJXwBOAd4iaOIKsO01W9qwFpA63AEg5pCvY/vCGM4zyvaTrW5XfyJpJWA1Mk9Rtu9oXYuKIWlbYIrtN2Im12bAGbafbnHT+vVGIKkTWJ6e39c/mlT3Y8A27XDjajWpw+1nYgziWEJo0bqSVgSusr1ti5vWb0j6AbAvMIMFEoK2vWfrWlUMSVOBjYGNgF8BvwQ+bnuHFrcrV/u23k2s6E1P0pEEV8p/iKsyBNPaYVtx4HAIsHpV/Z/Nsb2B8Bm+Wau+dwrJh9v/fIwYgwhg+1lJS7S2Sf3O3oQbzKDxMWaYZ9uS9gLOtP0LSQe1ulH0TIUdAWwJTAJ2zjOuddMD8jroownf1wsl2vMH4K+EiINGurwnAndHH+7834Tto0pcb0iQOtz+Z078D/xOikF8gpBNNBg73NcknQj8N/D++Kg9rME5/Y7tPbL7klYBTqtzSpmb3jOUX/JmMdvHF7Q9F7gVmMaCEfQ7ktTh9j/VMYifpcUxiFGt6RP0fhw8pUmXeBOYIukWBt+IZl/g08Bnbf9b0qrAD1vcpjxmAhvWOd7wpifpuIzt7ZL+SM/v6yd16r9O0odtX1+grfNsH9fYbOiTOtx+pk1jEP9AGNFMon9GoePjNuiInexvgXVi0SzgmkbnxdUYVqfnDexXzWqXpLOIIWGEhKVNgAfrnFLkpldxbf0jbsPjVq8dr8V2CPiqpNmEJJFK5MGSOafdJulQ4NqqtqSwsERziS6Et2P++3qETvdPrVzTSdJDtuuNjvLO6dcOpV2ITyGHAsvYXitqKfzc9i51zrkEWAuYQs9JwqaN6Kv8yPOAp2zfVdB+PrYvbnCdDkIUzat9amh+nXkROSksLNF8JE0C3g8sDdwLTATetH1AC9t0HnCW7WkF7Ut1KLGTOpUQMzqCBSe0/X+wKLSyJXBfRmhlmu3/qnPOw8AYD9L/TJIuIyzh00V46hkN/MR2TVdKjfC5/2tWKNlQJbkUShJHrG/Z7o77HcCIOiEvsv2mpM8ROrnTJD0wUO3t0RBpGuFxcBHgYElPEB7xKo+DtcKAxlKuQ7mQEGZ0OrATcDDtoUdQhNm251SyZiUtwoJH+Vo8BLwH+Fd/NSrz3WV5hXAD/051hEHJm94Y269KOgC4niClOIn6vuufARtL2hj4CvAL4BIgN3xO0oY5bRlyT0iNSB1ueW4BdgVej/uLETJ/3lfDXpK2AQ4APhfLWvW5f7SP55XtUEbavkWSYsLAyZL+SuiEW0KJRIy/SPoqMDL63r9I8D3WY1lghqT76emjbGbc8Z8II9DL4v5+8e+rwEXAHlX2ZW56wyQNI0Q2nG17biWqpg7Z8Lkz6oXPxVj0HQkd7vXAh4A7CXHO7yhSh1ueEbYrnS22X4/SebU4hhCHeI3t6ZLWBG6rd4H+yvqpZEvFPPyZtmdL2pEFQf7V7biWMKpagnIdyttx5P+YpCOAfwLvrtUuSR+y/aeqssNs/7zM+6tTf5mY1BMIN8ZpwBcIHcQFDS5xcjPa2YBtq5Jlpkm6y/a2yl/brMxN71zgKcIk3B2SViN05PWohM8dCGzfIHxuH0IyyQO2D5a0PI0/0yFJ6nDL84akzWxPBpC0OSFHPBfbfyGMmhaP+08ANSdTamX9EDrFWud8HPgBoVMT9WeMAX4LjJW0NuFRcDxh5PThKrsf1bpmA44hjPyPAr5NCM6vlzzwDUmzbd8KIOl4woioKR0uJWJSo6vofEqE7sXvuBR9CM0bJWkr2/fF87cERsVj83LsC9/0bJ8JnJkpelrSTg3eQiV87nMFwufest0taZ6kJQnL7bS9P78/SJNmJZG0BXA58GwsWgHY1/akGvbbEDq1UbZXjT6vL9j+Yg37x4GtymT9xHP2sP1wQfvJtjeT9BXCf4azVGfZaklrAP+y/XbcHwksb/upom1s0J5lgesI2VS7A+sD+zUrkkPSn4BPZp9M6thuSxixVtwPDYVWJG0NnAW8lxBW1Qm8UeeGV0l3rYTmzc/Usv3jGvZbENKMR8U2vQp8HpgOfMT2lTn2DwNLEW56o4HTbN+bsTnQ9q8z8bg9aBCHWxhJPwW+SnCD/C/BHTfF9sHNqH8wkUa4JbE9QdL6hPAuAY806Bj+D/ggMS7V9oOqkRcf6UvWz3+KdraRuQq6qp9hge+vXjbVVfT0UXfFsi3yjCWNBb5Gb59p7ijd9ixJexLSRCcB+zR5xr9MIsYvgGOp6ggbcDahM7mKMMH4GRbE8dZiZdu7F6wf2xOA/5I0mjBQejlz+Moa9hA6t1odWyXrMS/VPPfzl3Sn7e0y8bjzD1HjqSozuPh5vNEsaXtqjTYNaVKHWxBJO9u+NT6+Z1lHErZ/V+tc28+op1Z0r//I6kPWT6YtEyVdAfy+6pxabTqYEAb0XdtPxhFsvUX9FrE9J1PvHEn1AuQvJYxW66Zy5vynHU541NwnuB5rjxBLUiYR45Vqf3IRbD8uqdN2F3ChpLsbnHK3pP8qEZp3UtV+5bqnVJX/n+1jMv736nbumXl9bnz55+qY3jjS74Xt7eLfUnog1ZOWkravMWk5pEkdbnF2IOSDV88GQ/hh1+rcnlFIGnDspI4iPOpVUzrrp6otbwK7FWmT7Rlk/MgOUpHfr3Od5yXtaXs8QJyZrie193zFth5l/9P2FTcI9q/iNkk/JHx22ZvX5DrnvBm/2ymSTiNEc+RqZixEaN4bmdcjCBEneb+jS+LfMv73swhxtI3KelB0crfkpOWQJvlw+5nonzyDEEomQgjZ0WV8tE1sy5W2P1UjprPmI3+MariUsHChCG6Pz9h+vIb9LsD+hBC6hiNuheHaAcAatr+tIMyygu37y7y/WqhETKqkvAgS285V5YrnrEaY5BxOcEeMBn6a9/lE25q4oO5unHQbb/uDBWyXBlapfoyP8wvvI0xynp45tCTwMdsb16mzsKSjpEeBjYpMWg510gi3JAoi0N8EtiN0WncCp9TqQB1ElwtnlUm6mTDB83LcXxq4vN5/LEkXEzrx7Dk/dm9t0qPj31LxuLb/DmwtaRThJv1ag1MOJkx8DaNnpEWtp4CfRrudCRM8rwPnUMNH3AcKx6TabjQ7n3fO03EicQXb32pkC8VD8+qwGHVm+iXdDuxJ+D8+hfCU8hf3FJEZTpiEW4SeftxXCaFc9Sgj6TiY1eOaSupwy3M54VHoE3H/AOAKwgi2Fyoh1BxZLjshYvslSTVjWCMb5ZzTK+LA9r/i31KrF1SHMNXyH2bY2HVSYXPYKkZNPBDrfamBj7gspRIxJH0E2ICeo+GaSmqS9iA8wg8H1pC0CeEmXC/xoWhoXuUa2aeSTmA5ws2pFqMdssc+D1xo+5sK4urz8YKQxbccVirOXu+TwGN16i8zuTuY1eOaSupwy7OM7ewP/TuS9q5jX0aoGaBL0qoVX1h8BG3k9+mQtLTtl+I5y5Dz3eZMUs0/RP243bLqYvdKGhN9xUWYG/2BFc3g5ag/2Vb2JlY4JlXSzwmjx50Iwfn7AI1cGycT9Bduj+2YImn1Bud0254XJz7/zzE0r4599qlkHiEyJS/+tsIiklYAPkWIGKnHfvTW1j2REHVRizKSjnmTlu9IX2bqcMtzm6T9WBCKsw/wxzr2ZYSaIfznuFNSJZh+e4J6VT1+TJj1vprwQ/4U8L1qo4WYpCoVwkRwtxykoBJVZELoTIIE4vKSvkv4TL9ep/6yN7EyiRjvs72RpKm2vyXpx9R2hVSYZ/sVlVu1vGxo3nds/3e2QNIl1WUZTgFuBO6MoYxrUjVilfQhwoh6JUnZxIclyU+myFJmcncp22dUXfvoWsZDmTRpVhD11AFdnAUjsA7g9VqjQ0nfAe52MaHmyjnLsmCxwHtcYPE9SWMIHYmAW+qNLuMIuJrXXCOeWOXVxXInhuq5MhRim3dhQftrxhVLmmJ7kyJtKYuk+2xvJele4OPAC8BDtmvG1Ur6BWGC8ASC6+UoYJjtw+qcM4YQmneP7d/E0Lx9bedGiygmq2T2FwGm2h5T/l3Or2Njgq7uKUA27Ow14LbKE9PCUt32WFYz0WYokzrcfiZ21IsTRnqNhJqzM/Zr2j5FIWXyPfVm7PNGOvVGP5KeAlYBXortWYoQyvQccIirsuYkzQDWBhqOWOOj+1SX19stvLJx0ZuYSsSkZs75BiEkahfCxJ2B822fVG2bOWcxwpNJJSzvRsKI9O167SuCgl7BV4GRBF8ohM9/DnCe7RNrnHca8B1C2vkNBC2DY2z3irdWEK4RsG4serTWzTdzznIElbBqX/fOGZv9Cem/2xGeSCosAXTZzp33GMqkDrcPxCiAdej5Q2tKTKGknxFn7G2/N17rJts1Z+xzRj+dwLRao5/op7zG9o1xfzdCSu2VBOWnrarsS41YJV0KnOiCgjsquLJxzlNG3ZuYpM1tT5KUKxnoBhoIcbJwhO2ak0Pxs/6+7S/XsqmyLx2aF29iF9TxUeddZ4rtTSR9jKAlcSxh1Nor1Ct+Pr8iCNiIcDM+qN5vWtJNhMniLxFG6gcR4q+Pz9isBqxBCMk7IXP6a4SbciO3xdDDdtpKbIT89WmE0eFthBHErTl268e/m+VtdeqfHP8+kCl7sIbtiYQf7zxCKM9rcXsBOLXONSbWKiPkuOedsx1wcHy9HCFmtlb9t8Z23MKCCZPxdeynEP6jZ9/z1BZ9vyOA4wh+298SOqoRDc7p9f3XsV0h/l0tb6tz3qSS72N6/Hs+sHuD39Ekws2usr9uo+tVjme/J+AvdexXA3aNr0cCS7Ti+231libNynM0IT70Xts7Rd9jXuzlcYTJrjwxElNjeWtKzNjbPhU4VdKprvFoWYMXFRS5Lo/7+wIvxev2ulZ2BEqIaR1GSAXOTf8k//OoR6mVjVVwtYFao8gKzp/E+xXhZnFW3N+fkL31yTpNekDSeMKs/vyMMOckeriPoXmEyI8tvEAjoRHXSnqEMCD4Yvwd1XJxDLP9aKaNf4tuhnpUXA7/imF0zwIr5xkqs2wRYeWQlQlKcDWXLRqqJJdCSSRNsL2FwlIsWzkErjdtEkdBdX9fQidyMXHG3nbNEB3VEMNxjUfCOClXSd4QIXnjW4TQr1VdlSEV3+umhNF3ZdmZqTU6rMo5qxF8sn+OPs5O10iYkPQlgovmA4THz88Cl9k+q4b9VIJPciNCZ/gL4OO2d6iyK53VJelBVz1255VVHb8wv/reLoCc0DyxwE1i1/btzyCMPJ8mdOqNIj8qrq9XHdbTW4wgGvPvHLtfxjZU0oIPIOhn1FTzkvRRgl92FcLNaUngW85J6VYfli0aqqQRbnlmSlqKIBRzs6SXWCDVmItKLMBo+1KFddAqM/Z7u7ESWNZ/OILw455EjVG0Q9TDkTXqykvXLTsCrR7RrESdEY3Lr2xcaLWBPowiIYxWt3aUMZS0FVBzscZ4ncIyg+57aN6HyhhL+kzmdfZQ3u/uf4DDCdEVIiT2/LRe/baviy9fIcQs16MvyxYNSdIIdyGIkw2jgRucUdOqsim7AOMvCCFYUzJlJ9s+uUS7ViFon+5f4/i6hMmO1el5E8jtoPswAi08oolujBtdYsZaIUb5BkKK7vbA8wQXQ+6ISSX0ahUWhFyPEGMKsCpBJKab2pEZK8f6t2VBuvfRtmc2eB8bExYYBbjDDSQLq+z/arvmMukKy6pXGEG42U223StlV5mVpeN+J7Coa6/Th0Jc7xnANoTP5h7gWAeB/Wrb04CXCTHHRxKWLZphu1FCxtCj1U7kwbQRYm4fKnnOw8QbW0H7mYTO+TOZssklrylClEKt4w8SRjVbAptXtgZ1foCg6P8j4AMNbO+Lfx+Ifysxo7XsxxNSUYu+v/cQfOTvj/urZj+vHPuJhLC2Bwid7cEEaco829zJLOpMagE3xzoXids44OYG7+Fowlpxp8RtGnBks+xzzh9NjYlLwmrSozL7owhhd/Xquxf478x7PrDyvefYdhAyA68Cro6vC/+fGEpbGuGWpA8hT1cBRzlOlhSwn0xYXuZSwijraGCC6wSJx9FM5YvsIASzP2U7b60rJE2yvXnB9vRlBFpqRCPpSkKix830nHRqSq69pIm2x2b9zpLutl1r4U8U9CuyYX81v+88H34jv370Q29j+424vzghCaKWYlsp+5zzhxFueu9tUvvvc+/wwXttb12kPe9Ukg+3PCsA0xUWVKx0Dra9V9ZIfV+AUbZfBfaQdDIhP390gzZNzLyeB/zGVYLSVVwr6YuEdNpsm16sNnSYcHlT0mjXiUetIrsQ46HAH23XWzTwjyxIj67cOHrlyaoPqw1EyujV7kmILFmRkAiyGuEpZYM67Z8VoyV+E/f3J4Tm1UP0TEvuIuc999VePZM9OgjSlLUmXqvX6RtLnXX6IrdJOoEQ6WLCRO8fFbMYbb+oPsqBDmVSh1uebMiTCDP9eb7S8QRx5r9Wle9AEE/pgYJq1HvIiHzYPjlOVNVbkgfbF8ewH2w/X+A9VCaYspNtprbc39uEVWLrjkDjJNbKts8Bzo+TZ8sBm0t62fbVdeyJN6XlYlt66U+4j6sNEB59O4AjCHG1qxDSdvP4NmG0/WfbmyosppjrC8/wWcIyO6fHtt9N7WVtKlwI3Cfpmri/NyHaoln2WQHyecDTru1TPga4StKzsf0rEjrQelSOV3Q+Kp3/Z1nwW3othvDtwTt0kqwXrfZpDMaN8Mh+GiEz5zZyfGmERRE3yikfC1y7sPbxmAhKVbMII6qXCBNIJzX5/R6Ut+XY3UUQuq7sTyFEKqxK0EdYKPto0xc/+tFFymJ5JQHkQaAjvr6/hu3Kda65R4F2bUaIDDga2LTZ9lXndgIHVJVtQUgbhxBbfQQhaeVsgipeXj3zz8n8NsYTBIiWqbI9mjCZ9hRhVelNmvm7HIxbyxswWDZCDORJhMfLOwm+yafr2NfsFMiZ0CprH8uPJfg918iUrUnI5T82x/4rmdefrDr2vRz7VUt+RhOq9s/OvL53Ye0zxy4t0zZyJh3JZLVVlf+ZMGl0NsFFcAY1JpCAR4HVc8oPBv5e45wRhBHl2cAXCPGu9dpe1n5JQgbi2QRtB8WO9GngD9WfS6WTJDxFPUsQ3/k2cHWtz7IP56xGeGJ5IP7/OQlYt8xva6hsLW/AYNkIoS9/AdbOlD1Rx/7xMsfK2sfyB4Blc8qXy+tQsh1PdSdUo1PK2v+2wGdU7z306oDK2meOFUodJrgCriWM/MdnttsILoO8uhcjjKI7CaO3I6k92vswQfJwnUzZiQTfde7ol6A/8OvYef6ekCFX7zMta/8H4KJofyXhhvwXckaXZFJ9CUI9J2f2a6V4lz6n6vxN4++2q5HtUNySD7c4nyAINd+msNTz5dSf5Jgg6RDb52cLJX2OkJSwsPYQUjJ7STfafr5GaqZqvM7bry6ruZxLhvtqvIcvkC/iXda+QtHU4bsJE2TL0jPF+jWgen2vPHH2yvs/SdLfga/ZvqVy0Pb1kmYDf1IQof884ZF7e9eWNhzjGC8cY64biZuXtV8zY38Bwd20qvOz/DolLeIgIrMLPXWXa/UNpc+Jv8XdCf9/diHcAMqmfw8JUodbENvXANfEcJyK+tLyCupe19i+qeqUY6L9ASzoMMcSAu8/lnOJsvYQJPpqkXfMNV7n7Teyz+NY4PeSPk149IQQ47so4TNbWPvQkAYqXxm7pwmP0tsUsK05ERdD4zYkuDJ6yE46LN0zjhBNcjewi+vLMs6XPXRY8aFR0xbGvkvSkzU6Wwguk79ImkWISvgrzJ/ArRWRUvicmD24P/ARwo3icuBQx9C2dyIpDnchiCEwnyQIR9fK0tqJBf9Jp9u+tUGdhe0lddFz+ez5hwgKV8Nq2Ive+qpl7e3aef87syCMqsh7LmRfYxRasz1l7Rsh6Qu2z82pX4SbxFwWhGvl1l/1nWU/11rvob/ttyaEOt7kBTG+6xISIXKXhi96jsIKyJcR3FG9Qg7fiaQON5FIJAaIjlY3IJFIJN4ppA53IZHUaIHHZJ/sW3qNwW4/lEgd7sJT9seT7JP9QF9jsNsPGVKHm0gkEgNEmjQryGJLL+rRKy7Wq/zNl2az2NKL9ip/4+H8e9lcv80wjcg9Vsq+xvc2l9kMo3d7akUMz/VshinHvsZJ9dqvjt72c7rfZnhHjfeb8xbm+G2G16g/77datz05ZXOYzfC8zwdgkd5RknO632J4x8h8+66uXkVl2w9N/I5r0Er7vDC2Wt/BW36DOX67YdxbPT640+J+4cXe30sek6bOvtH27gtzvbKkONyCjF5xMQ7+TSNh+wVM3Kp359xMPHt2Y6MMyulMGp9U7gGoY2TxGwnU7oBq2r9d8j13lmz/u5YpZd/9clHxtIDnlVuk1nP7sKhtd7HOZqDoGFH8N3Hv23VXvS/ECy92cf+Nqxay7VzhsWUX+oIlSR1uIpEYMhjozl9ztS1oSx+upI9JssKKuGXPfb1G+SmSCotoJxKJwYcxc91VaGsFbdnhEtIB7yTkXvcgplmWxvZJtv+8sA1LJBLtTXfBf62g7TpcSaMIi/F9jtjhStpR0m2SLiMoMSHp95ImSZpeHdcn6ceSJku6pSLMLekiSfvE11tIulvSg5Lul9TXlVQTiUQbYUyXi22toO06XIJoyQ22/wa8KGmzWL4lQa1pTNz/rMO6XGOBoyS9K5YvTpAV3IygSvTNbOUKS61cQRCg3hjYlRrLiUg6VNJESRPffKnchE0ikWgN3bjQ1grascPdn6AqRPxbWd7kfttPZuyOkvQgYfXQVQjLeEPQrb0ivv41YQmcLOsB/7I9AcD2q1Fqrhe2z7M91vbYvNCvRCLRXhjowoW2VtBWUQpxlLozsGFcy6uT8BleT0YVS9KOhJHpNrbflHQ7mRVWq8jTOE3Bx4nEEKVVo9citNsIdx/gV7ZXs7267VWAJ+k9Sh0NvBQ72/UJi/5V6Ij1AHyaMPmW5RFgRUlbAEhaQlJb3XgSiUTfMDDXLrS1gnbraPYHvl9V9lvgf4C/Z8puAA6TNJWwrtS9mWNvABtImkQQRO6x+qjtOZL2Bc6SNJLgv90VyA0nSyQSgwe30F1QhLbqcG3vmFN2JmFF0GzZbOBDNeoYFV9+o6p8XOb1BHqOihOJxFDA0NW+/W17dbjtzBszxIRNiocAj/zLUqXqf2uH/5RtUinKppX2ha659Vb8GXg8t7FNlu5n/1XK/pUDtiplP/rS+0rZ19JSGFSsvXpx28cXfmI6ZJo1B0m7E1Zt7gQusP39quOrAhcDS0WbE2zXzU9uNx9uIpFILASiq+BWt5aQYHUO4Ul6DLC/pDFVZl8HrrS9KSFn4KeNWpdGuIlEYsgQJs0WSnCswpbA47afAJB0ObAXMKPqcpU14kYDzzaqdMBHuJLeI+lySX+XNEPS9THB4Loa9hfk3FkSiUSiFyEOt/AId9lKYlPcshmrKwHPZPZnxrIsJwMHSppJCF09slH7BnSEqyCOeQ1wse1K2u4mwB61zrH9+QFqXiKRGAJ0Fx/hzrI9tsaxvEqqner7AxfZ/rGkbYBLJG1ou6YbeaBHuDsBc23/vFJgewphbftRkq6W9IikS2PnjKTbJY2Nr1+X9N2ogXCvpOVj+XKSfitpQty2jeU7SJoStwcqmgmSvhztpkr61gB/BolEop8oOcKtx0xCBmuFlentMvgccCWA7XsIyVd1NXYHusPdEJhU49imwDEEB/WaBAGbahYH7o0aCHcAh8TyM4DTbW8BfAK4IJbtopTZAAAgAElEQVR/CTjc9ibA+4G3JO1GSAPeEtgE2FzS9nkNymopzCVpKSQS7Y4RXXQU2howAVhH0hpRf2U/YHyVzT+AXQAkvZfQ4T5fr9J2mjS73/ZMAElTgNXpnSU2B6j4eicBH4ivdwXGZJbzWDKOZu8CfiLpUuB3tmfGDnc34IFoO4rQAd9R3SDb5wHnhQqXGQIxOonE0KeES6EmtudJOgK4kRDy9Uvb0yWdAky0PR74X+B8SccSBtfj3GAZk4HucKezIO22muwQsov8ts3NvKGsTQdBV6Fa9ev7kv4IfBi4NwqQCzjV9rl9eQOJRKJ9MWKO+ySZ3buuEFN7fVXZSZnXM8h/Eq/JQLsUbgUWlVRxBRA1DXZYyHpvAo7I1LlJ/LuW7Wm2fwBMBNYn3LE+G3V3kbSSpHcv5PUTiUQbEBIfOgptrWBAR7i2LeljwP9JOgF4G3gK+P1CVn0UcE7UVliE4B44DDhG0k6E0fAM4E+2Z0d/yz3RBfE6cCDw3EK2IZFItAEFJsRaxoD7cG0/C3wq59D5GZsjMq93zLwelXl9NXB1fD2LKpGaWJ4bF2f7DMJEWyKRGELYosvtm0DbTpNmbY06OugYWXzp87d2mlWq/idP3aaU/VrffrCUvRYbWco+nFTyh7tUuZWK/EzDxJwedJdcGr4sHYuVW9p+qcsnlrLXRuXWRPUjT5SyB+hcrtzK390vvlTuAio3evSjxd+D5zbn++1OI9xEIpHof8KkWft2a+3bskQikShJZdKsXWnbltXQXFi3CfWeLOlLzWhjIpFoP7qsQlsraMsRbh3NheWBv7WybYlEon2pZJq1K+3aslqaCx/IaCP8U9KFAJIOlHR/LD83alkiaXdJk6P2wi2Z+sdEjYYnJB01oO8skUj0K93uKLS1gnbtcHM1F2yfFHURdgBeAM6OMbX7AtvGY13AAZKWI4SafSJqL3wyU9X6wAcJegrflDSsX99NIpEYEIJ4TVO0FPqFtnQp1CO6Gy4liNVMivnOmwMTYiLDSEISw9bAHbafBLD9YqaaP8Z10WZLeo7gqpiZc61DgUMBRmjx/ntTiUSiKRgxt0mpvf1Bu3a49TQXTgZm2r4w7ovg6z0xayRpT3rrV1YootvQQ7xmdOeySbwmkWhzbNo68aFdW5aruSDpmwSFsKzf9RZgn4oegqRlJK0G3APsIGmNSvmAtT6RSLQI0V1wawVtOcKto7mwGLAicH90H4y3fZKkrwM3SeoA5hI0cO+NLoHfxfLnWCDnmEgkhiCmvUe4bdnhQl3NhTzbK4Arcsr/BPypquzkqv0N+97KRCLRbrRzWFjbdrhthw3dzVrxvjdr/+TxUvaP/KxcXv76xzxZyh5Ao8rpL8xdrpyWQue/SwaH9LOWQvebb5ayV2fJyZlHy38HZZn37L9L2XeMHFHKvvutt8vVP7z4d6y5parOxagpAuT9RepwE4nEkCEsk96cbk3S7gRVwU7gAtvfrzp+OiFnAIK78922l6pXZ+pwE4nEEKLQApGNawnJU+cQ5n1mEsJOx8dVHgCwfWzG/kjCuox1aVtnh6SumDk2PWaKHRcnvxqd99UCNhdJqhV2lkgkBimmaZlmWwKP237C9hzgcmCvOvb7A79pVGnbdrjAW7Y3sb0B4S7zYeCbBc5r2OEmEomhS5OWSV8JeCazPzOW9SKGoa5BCGetSzt3uPOx/Rwh4+sIBcZJOrtyXNJ1knaU9H1gZBwZXxqPfUbS1DhKviRT7faS7o56Cmm0m0gMAWyVGeEuK2liZjs0U1Vej1wr+Wk/4GrbXY3aN2h8uLafiC6Fmgs+2j5B0hFRUwFJGwBfI+gszKpKflgB2I6gqzCeuFxPIpEYvIRJs8LRI7Nsj61xbCawSmZ/ZaDWEiX7AYcXueCg6XAjZb3hOxPuPLOgl57C7213AzMkLZ97saSlkEgMMpq2ptkEYJ2YqfpPQqf66V5Xk9YDliZktjZkULgUACStSdA9eA6YR8+21womFMX0FHI7ctvn2R5re+xwFi3Z4kQiMdCESTMV2urWY88DjgBuBB4GrrQ9XdIpUaelwv7A5bYLaa0MihFulFr8OXB2TPt9CvhidDGsRJhRrDBX0jDbcwk6C9dIOt32C5KWqRrlJhKJIUazMs1sXw9cX1V2UtX+yWXqbOcOd6SkKcAwwoj2EuAn8dhdwJPANOAhYHLmvPOAqZIm2z5A0neBv0jqAh4Axg1Q+xOJxACTMs36iF3b8x2H7wfUOHY8cHxm/2Lg4iqbcVX7oxamrYlEon1o50Uk27bDbTs6OtCI4n7c0l/53DmlzMtqI7z0ofVK2QMsPaWc96XzjXLvISq+Fa9/9JKl7Ltefb2UPd0No3p6UlZLoaPcr0J90O7oXPE9pey73rN0uQtMfric/QBjw9zu1OEmEolEvxNcCqnDTSQSiQGhGVoK/UVLbwUZvYTKdsJC1PV6/LuipJpJDJJWl/RQX6+TSCTal2aFhfUXrR7hvlXJCmsWUbg8peomEu9I2tul0JYtk/SUpG9JmixpmqT1Y/lykm6O5edKelrSslXnzh/BStpA0v1x9DxV0jrRrFPS+VGJ7CZJ5ZS2E4lE29LOa5q1usMdqZ4uhX0zx2bZ3gz4GfClWPZN4NZYfg2waoP6DwPOiKPosSxYCn0d4JyoRPYy8IkmvZ9EItFCQpRCZ6GtFbSzS+F38e8k4OPx9XbAxwBs3yDppQb13wN8TdLKwO9sPxZDkZ60PSVT/+p5J/fQUuhIWgqJRLvT7okPrR7h1qOiddDFghtDqU/S9mXAnsBbwI2Sdq6qu7r+6vMXaCkkr0MiMShILoXmcSdxJV9JuxFUemoSBW+esH0mQYJxo35vYSKRaBntHqXQ6g632of7/Qb23wJ2kzQZ+BDwL+C1Ovb7Ag9FTYb1gV81pdWJRKJtadISO/1CS324tfQSbK+eeT0R2DHuvgJ80PY8SdsAO9meHe1Gxb9PARvG16cCp1ZV/2LleLT5URPeSiKRaANsMa+Nw8JaPWlWllWBK6Ms4xzgkBa3J5FItBntPGk2qDpc249RYCnifkGCYcMLm3eXFU6ZM7dcczrK/aiWueOZxkZVPPL9mqsZ5bLeyS+XvkZ/UvYzckmtGC1S7r9P9xtvlLLvWGyxUvYAXf95vpR92eCorpKf6UBT8eG2K+079k4kEok+0KxJM0m7S3pU0uO1ZAckfUrSjJhEdVmjOttihBvFwadlivaOvtj+uNaOwJdsf7Q/6k8kEq2jWXG4kjqBc4APEBKmJkgab3tGxmYd4ETCIrUvSWr4SNgWHS4NNBUkLRLXGEokEom6NCnGdkvgcdtPAEi6HNgLmJGxOYSQsfoSgO3nGlXati4FSeMkXSXpWuCmWPZlSROiLsK3Ytnqkh7O00aQtLakP0t6MOovrBWrHyXpakmPSLpUZZWwE4lEW2LDvO6OQlsDVgKyEx8zY1mWdYF1Jd0l6V5JuzeqtF063Gw87jWZ8m2Ag2zvHBMd1iHceTYBNpe0fbSrpY1waSzfGHgfIW4XwsTbMcAYYE1g2358b4lEYgAp4cNdVtLEzHZoppq8QVj1yryLEPqeHQmr914gaal6bWt3l8LNmVV2d4vbA3F/FOHN/oMcbQRJSwAr2b4GwPbbMH9Zl/ttz4z7UwhaCndWX7ynlkJa9iyRaHdK+nBn2R5b49hMYJXM/srAszk298YVwp+U9CihT5pQ64LtMsKtRTaORsCptjeJ29q2fxGP5Wkj1PvUy2spdCQthURiMGCr0NaACcA6ktaQNBzYjyAPkOX3wE4AUSZ2XeCJepW2e4eb5Ubgs5JGAUhaqd6soO1XgZmS9o72i0oqH9iYSCQGFc0Qr4mT9EcQ+p2HgSttT5d0iqQ9o9mNwAuSZgC3AV+2/UK9etvFpdAQ2zdJei9wT3QLvA4cSBih1uK/gXMlnQLMBT7Z7w1NJBItw25e4oPt64Hrq8pOyrw2cFzcCtEWHW5FB6Gq7CLgoqqyM4AzcqrI1UaImWk7V9k+AdyesTmiD01OJBJtiehKy6QnEonEwFDAP9syUodblO5u/Pbbhc3L5vF3LF1X2rd3/SMWLWXvV14tZQ+w/vHVUTD1+dsPy2kvrHngP0rZdwwfVsre3eXaT8lwbHWWVCIoG+7dXVLcAdCwcv+l/Wo9ddOc+ocX1xMJ9iW+szkLPzJtdy2F1OEmEomhg4Mft10ZcGeHpOUlXSbpCUmTJN0j6WMD3Y5EIjE0aecldgZ0hBtTaH8PXGz707FsNcK6Y0XO77RdLyohkUi8g3GbT5oNdMt2BubY/nmlwPbTts+S1CnphxmthC9AUPeSdFuUPpsWtRMekXSBpIeiFsKuMZ/5MUlbxvO2lHS3pAfi3/Vi+ThJv5N0Q7Q/bYA/g0Qi0Y/YxbZWMNA+3A2AyTWOfQ54xfYWkhYF7pJ0Uzy2JbCh7SclrQ6sTYipPZSQEfJpwhLqewJfBfYGHgG2j8vx7Ap8jwUaC5sQ9BRmA49KOst2eYXuRCLRdqQohRpIOofQUc4BngY2krRPPDyakJc8h6B98GTm1CdtT4t1TAdusW1J0wi6CJXzL46alQay06W32H4lnj8DWI2eykCV9i3QUtDiC/+GE4lEvxJGr+3b4Q60S2E6sFllx/bhwC7AcgTtgyMzWglr2K6McKvXJslqIXRn9rtZcBP5NnCb7Q2BPYARNc4vpqWgEXkmiUSizUjLpC/gVmCEpP/JlFX0DW4E/kfSMABJ60oLNawcDfwzvh63EPUkEolBRPLhRuJj/97A6ZK+AjxPGL0eD1xFcAdMjtEMzxN8sX3lNIJL4ThCR59IJIY4RnS3cZTCgPtwbf+LIHWWx1fjluV2emofPEVP7YRxecds30OQS6vwjVh+ERmNhrS2WSIxtGjjvIeUaZZIJIYQbT5pljrcgtjGc+YUt59Xbs3LrlmzStmXzWn33PJrcOqt4toRAGt+uloQvz43PjulsVGGD65Yc53RfDrKaR2U1UboerW8PkV/0/327MZGWfxWKfOyn1Gp/zPdTcppauMhbupwE4nEkKKdR7ht613uT80FSa83o55EItFeGOjuVqGtEZJ2l/SopMclnZBzfJyk5zML4H6+UZ1tOcItqrkgaZG4FEYikUiEHrcJI1xJncA5wAcIi0VOkDTe9owq0yvKLGLQriPcepoL4yRdJela4CYASV/OaDB8q3KOpAMl3R/vPufGD5HM8WXjyPkjA/XGEolE/9KkONwtgcdtP2F7DnA5sNfCtq1dO9x6mgsA2wAH2d5Z0m6EFOAtCRoJm0vaPq5/ti+wbVyCvQs4oFKBpOWBPwIn2f5jP72PRCIx0LjgVp+V6JnuPzOWVfOJONC7WtIqOcd70JYuhWqqNBfOAW62/WI8vFvcHoj7owgd8EbA5oRHAYCRwHPRZhhwC3C47b/Uue4CLQXSgr+JRPtTaAn0CstKmpjZP8/2efMr6k11N30t8BvbsyUdBlxM7zUUe9CuHe50Fih7YfvwuO575cPJaisIONX2udkKJB1J8AGfmFP/PGAS8EGgZocbP/zzAJbseFcbB5skEon5FP+fOsv22BrHZgLZEevKQI+4x6ol0c8HftDogu3qUqinuVDNjcBnJY0CkLSSpHcTRrD7xNdIWiZOvEH4Sj4LrJ83+5hIJAYpBner0NaACcA6ktaQNJyQHTs+ayBphczunsDDjSptyxFuA82FkVW2N0V/7T3RdfA6cKDtGZK+DtwkqQOYCxxOkIHEdpek/YBrJb1q+6cD9f4SiUR/svBRClFH+wjCgK4T+KXt6ZJOASbaHg8cJWlPwhPzixQQyWrLDhcaai5cVGV7BnBGTh1XAFfklI+Kf+cQ3AqJRGKo0CTnn+3rgeuryk7KvD4RyHNZ1qRtO9xEIpHoE20825I63IKos5OOJZcsbN/90kul6nd3//5KOt+7dvmTnvlXOfvRS5QyL6uN8NiZW5WyX+eo+0rZd4wq/v0CdM8uqVvQVVIrYNiwxjZVdC5R7jvwisuWsu+e9rdS9ri7hG25qmvW0capvanDTSQSQ4pWiYsXoV2jFBpSTw9B0t19PTeRSAxyulVsawFDaoQrqdN2l+33tbotiUSiNSiNcPsPSTtKuk3SZUBlJd/X498VJN0RtRQekvT+zHnflfSgpHtjmm8ikRjsFE3rbVGnPOg73MiWwNdsj6kq/zRwY9RS2BioKF4vDtxre2PgDuCQAWtpIpHoRxQmzYpsLWCouBTut/1kTvkE4JdxJeDf2650uHOA6+LrSQQJtl700FLoGNXcFicSif4huRT6nTfyCm3fAWxPWC79EkmfiYfm2vPnMruoceOxfZ7tsbbHDu8YmWeSSCTaje6CWwsYKiPcXKJ2wj9tny9pcWAz4FctblYikegvUhxuS9kR+LKkuQSNhc/UN08kEoOddo5SGLQdbkYP4Xbg9hrHLiZoVOaeG19fDVzdj01NJBIDSRt3uEPFh5tIJBJtz6Ad4Q40njePrlmzCttr+PBS9XcsWu6r6H4jd56wJl0zSubAA53LLF3KvvuV18pdoKOzsU2GstoImz9QbmZk8tbltBFcVkuh5PvtWLF8eHjXo4+XstfLr5Sy73zXMqXsu55/vpR9M0guhUQikRgITMvSdovQdi6FTJbY6pI+XcB+dUkPxddjJZ3Z321MJBJtTMo06xOrEzLFCmN7ou2j+qc5iURiMCAX21pBO3e43wfeH3UQjo0j2b9Kmhy3XgI1UVfhuvh6S0l3S3og/l0vlo+T9DtJN0h6TNJpA/y+EolEf9KkEa6k3SU9KunxemsfStpHkiXVWpByPu3swz0B+JLtjwJIWgz4gO23Ja0D/Aao9wYfAbaPaxPtCnyPBSsBbwJsCswGHpV0lu1natSTSCQGE00YvUrqBM4hpP3PBCZIGm97RpXdEsBRQKEZ3XbucKsZBpwtaRNCOu66DexHAxfHztnx/Aq32H4FQNIMYDWgV4fbQ0uh5qLBiUSiXWiiu2BL4HHbTwBIuhzYC5hRZfdt4DTgS0UqbWeXQjXHAv8hqH6NBRrFXX0buM32hsAewIjMsWw8TyEthWEs2ueGJxKJAaQ5AuQr0XMQNjOWzUfSpsAqtq+jIO08wn0NyC7QNBqYabtb0kGEpYvrMZogWgMFli9OJBJDgxIj3GUlTczsn2f7vEo1Ofbza5bUAZxOyb6lnTvcqcA8SQ8SlkX/KfBbSZ8EbqOGQliG0wguheOAW/uzoYlEoo0o3uHOsl1rHmgmsEpmf2Xg2cz+EsCGwO2SAN4DjJe0p+1sJ96DtutwMzoIc4Fdqg5vlHl9YrR7ivDGe+gq2L6Hnn7eb8TyiwgdeOV6H21a4xOJRGtpng93ArCOpDUIT8r7kQlTjXNA85c8lnQ7YZK/ZmcLg8uHm0gkEo1pQliY7XnAEcCNwMPAlbanSzpF0p59bVrbjXDbFXV00LFY8UiFsloHWqSfv4o+rB3d9cKL/dCQhUDlUjan7P6eUvZHTr+jlP05Y7cqZV+W7r8/VfocDSun4eG5c0rZl9ETAcrpR3SVq7oWapK4uO3rgeuryk6qYbtjkTrTCDeRSCQGiH7tcCu6CIlEIjFgtLGWQnIpJBKJoUMLdRKK0O8uBUmjJN0S9Q+mSdorlq8u6RFJF0uaKunqmL6LpJMkTZD0kKTzFOMuJN0u6QeS7pf0N0nvj+Wdkn4Yz5kq6QuxfAVJd0Q9hocy9rtJuie26SpJaUneRGKo0MYj3IHw4b4NfMz2ZsBOwI8rHSiwHiHYeCPgVeCLsfxs21vELLGRQDZ0axHbWwLHAN+MZZ8DXrG9BbAFcEgM5/g0cKPtTQgZalMkLQt8Hdg1tmkicFy/vPNEIjHwtHGHOxAuBQHfk7Q9YXHilYCKlP0ztu+Kr39NEIH4EbCTpK8AiwHLANOBa6Pd7+LfSQQJR4DdgI0k7RP3RwPrEGLpfilpGPB721Mk7QCMAe6K/f5w4J7chme1FLR4X99/IpEYIETzohT6g4HocA8AlgM2tz1X0lMs0DWovs9Y0ghCVtlY289IOpl8HYSsBoKAI23fWH3x2NF/BLhE0g+Bl4Cbbe/fqOExze88gNGdy7axZyiRSADJh0sYbT4XO9udCMpcFVaVtE18vT9wJws611nRt7oPjbkR+J84kkXSupIWl7RavPb5wC+AzYB7gW0lrR1tF5PUSHkskUgMFt6JLgVJixBGo5cC10aRiCkEndoKDwMHSToXeAz4me03JZ0PTAOeIrgFGnEBwb0wOfqHnwf2BnYEvixpLvA68Bnbz0saB/xGUkUC7OtA+VUWE4lE+9HGI9z+dClsAPzd9ixgm+qDklYHum0fVn3M9tcJnWB1+Y6Z17OIPlzb3cBX45bl4rhV13MrYXItkUgMMd5xLgVJhxFWZOjVaSYSiUS/8k5zKdj+OfDzBjZPEVW+BgPu7qb7zTcL23dstH6p+runPtLY6J1OST2IrudfKGV/5trlvrOfPv3HUvaHr7VjKXvPm1fKfkAo+R10LjO6sK1eLqG7UAunKIVEIpEYON5pLoUyZPUWJH04rqS7aivblEgkBi/tvEx624xwJe0CnAXsZvsfBc9ZJOpWJhKJRCCNcOsTNQ7OBz5i+++xbLWowTA1/l01ll8k6SeSbgN+EONtfxl1FB6o0mr4a9RLmCzpfbF8x6jJcHXUcrg0k2qcSCQGM0UnzIawlkIjFgX+AOxtOztzdDbwq6izcClwZubYugQthP8FvgbcGnUUdgJ+KGlx4DngA1EvYd+q8zclaDGMAdYEtu2Xd5ZIJAYU0TyXgqTdJT0q6XFJJ+QcPywKck2RdKekMY3qbIcOdy5wN0GAJss2wGXx9SXAdpljV9mu6MPvBpwgaQphPbMRwKrAMOB8SdOAqwida4X7bc+M8btTWKDJ0ANJh0qaKGni3B4rqycSiXalGR2upE7gHOBDhL5j/5wO9TLb/xXFsU4DftKobe3Q4XYDnwK2kFSduJAl+xFl168R8Anbm8RtVdsPA8cC/yGohI0liNRUyPaeWU2Gnhe0z7M91vbYYSyaZ5JIJNqN5rgUtgQet/2E7TnA5cBePS5jv5rZXbxIre3Q4WL7TYIE4wGSKiPduwkrZUIQwLmzxuk3AkdmNHM3jeWjgX/FUex/A00I8kskEm1PczrclYBnMvszY1kPJB0u6e+EEe5RjSptiw4XwPaLwO7A1+PE11HAwZKmEjrMo2uc+m2C+2CqpIfiPgTFsYMk3Uvw+ZZb1TGRSAw+CroTokth2YrLMG6HZmrKm0jv1U3bPsf2WsDxFMisbXlYmO1RmdfPAGtkDu+cYz+uav8t4As5do8BG2WKTozltxN8vRW7I/rU8EQi0Z4Uj0CYZXtsjWMzgVUy+ysDz9ap63LgZ40u2DYj3EQikWgG6i62NWACsI6kNSQNJ7g3x/e4jrROZvcjBMXDurR8hDuoKJFH3v1Qw8++B53Lv7uUfdd/nitlr0X68FV39q/b23PL5ax0DB9Wyr777bfL1T9iRGOjDF9c/f2l7L/294ml7L+75ial7KH8e3BXOeEBd3U1NsrQ9dIr/VZ3LZqRRWZ7nqQjCHNEncAvbU+XdAow0fZ44AhJuxIirV4CDmpUb+pwE4nE0KGJSQ22rweuryo7KfO61rxSTVKHm0gkhhYptbc8kroyy5tfpbiEeh371+PfFSVdXcdu9RjNkEgkhhjNzDTrD9q2wwXeiokMGwJzgF4rQ+Rh+1nbRdZBSyQSQxB1u9DWCtq5w83yV6Cy6ONxcdT7kKRjqg2zI1hJG0i6P46Up2ZmFTslnS9puqSbJI0cuLeSSCT6jSRes3DExSg/BEyTtDlwMLAVsDVwSCazLI/DgDNirvNYQmwdwDrAObY3AF4GPlHj2klLIZEYZCSXQt8YGQVpJgL/ICxzvh1wje03bL8O/A6oF5tzD/BVSccDq8UkCYAnbU+JrydRQ7wmaSkkEoOQNh7htnOUwltxZDqfsrq1ti+TdB8hKPlGSZ8HnqC3eE1yKSQSQ4R33Kq9/cgdwN6SFouatx8j+HdzkbQm8ITtMwlZIhvVsk0kEkOENMJtDrYnS7oIuD8WXWD7gTqn7AscKGku8G/gFGDJ/m1lIpFoGU6r9vaJrKhNVflPyBH6rdhnl1+3fSpwapXpi2SWZ7f9o+a0OJFItJpKHG670rYdbruhjg46Ri1R2L77tddK1d/94svl2lNSG8Hzyq+12dHP19CwcvWX1UZY5D3Ll7Kf99ysUvZltDUAvrv25qXsP/PoU6XsAS7ZYI3GRhnKfmelP9N//6eUfVMo+b0MJKnDTSQSQ4o0wk0kEomBoIUTYkUoFaVQVt8g5/x6a5YlEonEQtMkPdx+oWxYWJ/0DRToAFKHm0gk+pWh1OFmqatvEDUNHpb0U2AyIVNsZBwhX1qt2iXpS5JOjq+3iNoH90j6YUYbYZykszPnXCdpx/h6t2g/OY6+R8Xy70uaEev7USxbTtJvJU2I27YL8TkkEol2wYRJsyJbC+hTh1tC32A94Fe2N7V9MAtGyAc0uMSFwGG2tyFkgjVqz7KEBdx2tb0ZIR34OEnLEJIjNrC9EfCdeMoZwOm2tyDoKFxQo975WgpzXG6GPJFItIZmaSlI2l3So5Iel3RCzvHjMoO5WySt1qjOsh1uWX2Dp23fW+YCkpYClrB9dyy6rMBpWwNjgLti+w4CVgNeBd4GLpD0ceDNaL8rcHa0HQ8sKalXzFdWS2G4yi1dkkgkWkQTMs0kdQLnEAaWY4D9JY2pMnsAGBsHc1cTlkqvS9kohbL6BvWWJp9Hzw6/0qPVq6/eOTfb3r/6BElbArsQFoE7grAScAewTUbMJpFIDAGamPiwJfC47ScAJF0O7AXMqBjYvi1jfy9wYKNKm6GlUEbfYK6kykqA/wHeLeldkhYFPgpg+yXgNUlbR83pPyAAAA8BSURBVLv9Muc/BWwiqUPSKoQPBcKb3VZSxae8mKR1ox93dFyb6BigcrO4idD5Eu3Lr9aXSCTaDxcTHy8gQL4S8Exmf2Ysq8XngD81qnSh43Br6RtIWj3H/DxgqqTJtg9QWAHzPuBJ4JGM3eeA8yW9AdwOVJb+vCvaTgMeIkzGYft5SeOA38TOG4JP9zXgD5JGEG5+x8ZjRwHnSJpK+AzuoGDERSKRaHOKj3CXlZRdSvk82+fF13lP2rk1SzqQoLe9Q6MLlupwy+gbZDUNMmXHA8dn9s8Ezsypcnr0ixCd1ROjvYHcCTfbtwJb5BzaMsd2FkHYJpFIDDFKuBRm2R5b49hMYJXM/srAs72uFZZJ/xqwg+2GqxS0a6bZRySdSGjf08C41jYnkUgMCgw0Z72yCcA6ktYA/klwbX46axCjsc4Fdrf9XJFK27LDtX0FcEWr29GL7v6LltbwYY2NMnS/+WZjox4XKKXdHhhWrk1lYxs9Z04pey1abtWNsmI0ZcV0PKfc76FjRLn2/+q9q5eyB/jfvz1Yyv4nG2xWyr60GE2Z312zQmObUI/teZKOAG4EOoFf2p4e3aATbY8HfgiMAq6KsQP/sL1nvXrbssNNJBKJvtIs8Zo42X59VdlJmde7lq1zwFd8qNJjuDbG3Tar7rGS8nzCiUTiHUJaJr0nWT2GF4HDm1Wx7Ym2j2pWfYlEYpCRlkmvyz3E2DZJO0q6rnJA0tkx1KuWHsIn4yj5QUl3VNchaUtJd0t6IP5dL5aPk/Q7STdIekxSw+yQRCIxOAiJDy60tYKW+XBj6twuhPTgenYVPYT1bTvjgjgJ+KDtf9ZwSzwCbB+d37sC3yPoJkBIgNiUsHrvo5LOsv1MTh2JRGKw0cZrmrVihFvRY3gBWAa4uYF9LT2Eu4CLJB1CmEWsZjRh9vAh4HRgg8yxW2y/YvttQqperuhEEq9JJAYf7TzCbZkPl9DJDWeBDzdXJ8H2PELywm+BvYEbYvlhhGyyVYApkt5VdZ1vA7dFX/EeLNBdgDCyrdBFjZF+Eq9JJAYZbe7DbZlLwfYrko4ipN7+jJDgMCam5o4guBvujHoIi9m+XtK9wOMAktayfR9wn6Q96JkVAmGE+8/4elz/v6NEItF6WheBUISWTprZfgB4ENgv+lCvBKYClxKkzwCWAK6Lugd/YYEewg8lTYsugztiPVlOA06VdBf5LodEIjEUaWMB8gEf4VbrMdjeI/P6K8BXck7L00P4eI7d7XHD9j3Auplj34jlFwEXZer5aMGmJxKJdsetWz6nCCnTLJFIDC1aNHotQupwC2Ibz5lb2L5z+XeXqr/rP4W0L+bTMaLcJF7Hiu8pZQ/Q/fwLpew7371cKfuuf/27lL2GDy9l77nzytnPbij21JOS+hRapOR/t+6SehnAj9fdqJS9/rxsuQvs/M/GNlla0fm1b3+bOtxEIjG0UD+KTC0srdBS+Jqk6TFrbIqkrUqev4mkD2f2d5T0vsz+YZI+U+f8kyV9qW+tTyQSbY0JiQ9FthYwoCNcSdsQltLZzPbsuNpuuefEkCU2lgUqPjsCrwN3A9j+eXNam0gkBhuidUkNRRjoEe4KBJX12RBWXrD9rKQtot7Bg5Lul7SEpBGSLoyhXw9I2knScOAUYN84Oj6esDTOsXH//dkRrKSjMhoMl2faMUbS7ZKeiLHAiURiqJDCwuZzE3CSpL8BfyaIjN8T/+5re4KkJYG3gKMBbP+XpPXjuesSNBTG2j4CQNJI4HXbFVGbXTLXOwFYI46ms3oL6wM7EWJ8H5X0M9vFZ8QSiUT7kka4AduvA5sDhwLPEzraLwD/sj0h2rwa03m3Ay6JZY8QMtHWzau3DlOBS+Mib9kp6z/anh3XNnsOWD7v5KyWwtykpZBItD9t7sMd8Ekz2122b7f9TcJS5R8nP5CjD2vC9OIjwDmETn6SpMqIvrSWwrCkpZBIDArU3V1oa1iPtLukRyU9HhezrT6+vaTJkuZJ2qdI2wa0w5W0nqR1MkWbAA8DK0raItosETvGO4gr9EpaF1gVeJSw9PkSmTqq9yvX6gBWsX0bIXttKcL6Q4lEYshS0H/bwO0Q5WPPAT4EjAH2lzSmyuwfBJ2Wy4q2bqB9uKOAs6I/dR5BiOZQ4MJYPpLgv90V+Cnwc0nTou246Iu9DTghSjyeClwLXC1pL+DIzLU6gV9LGk0YLZ9u+2X1ZTHFRCIxODDN8uFuCTxu+wmAOOm+F0HONVzKfioeK+ygGNAO1/Yk4H05h2YBW+eUj8up40Vgi6ribHrNXzOvt8s5/+Sq/Q3zW5tIJAYlxf2zy0qamNk/z/Z58fVKQHZRgpnw/+2df4xcVRXHP1+20GrrbqhUU1rSNmn9RQUpohAtYoxtrfVHokglJhKIP0gw0QQSjQm0JFgrRIgFDVVLCSa1ukHYQOOW1igUu2VboLhLrKJSu1ShpU3dFrH74/jHvcO+Tt/Mvtud7sxuzieZzJv7zjv3vvnjzJ377vkeknIG8vBMM8dxxhUJ+3APmtn7K7nJaRvx1NkD7mli4JUDSfZNzc1p/nt7k+x5OW08AJo0Mck+9Z7POPvsNP+vHkqy7//o+5Lsz+r4c5L94GtpWgeD/03b6TL44bTxAzR1dKVdsOjlJPO9K/P+oFZm1sodxY0HklxXpjZLCj2cqLE9E9g/UqcecB3HGT+YwUBN9nx1AvMkzSEUMlgOXD1Sp3URIB+pnkIB/38c5vzRWvbnOE4DUYNdCjEX4AagnbCT6ldm1i3pVkmfBogZsj3AlcC9krqHG9qoz3BrpKdQFTNL+9/jOM74oUaZZma2iSHNllLbzZnjTsJSQ2HqMcOtpKfwoqTVUUvhKUlzASR9StKOqKewRdLbY/sKSevyNBFKM1hJ0yU9HmfRXZIWZmxui9oNHSWfjuOMcQwYtGKvOlCPgLsZOE/SXyT9WNJHMuf+Y2YfAO4G7opt24BLzewi4JecWILnXcBiwp65WySdWdbX1UB7rBJ8IfBsbJ8MdJjZhYQEi6/U7vYcx6kfBjZY7FUH6lHT7Kiki4GFBAGZjZm0uQ2Z9zvj8cxoM52w9PCPjLtH40z5f5JKmgg9mfOdwLoYiB8ys1LAPQ48Eo93AR/PG6ukrxISM5jEm0/ldh3HGU2MWj00Oy3U5aFZjp7C50qnsmbxfQ1wt5m9lyB0kxU1qKqJYGaPA5cTnjI+kBEm7zN7Y6HHtRQcZzzRwPKM9aj4kKensDceX5V53x6PWwgBE+DLiX3NAl4xs58CPwcWnNKgHccZOzRwwK3HPtxKegrLgImSdhB+CL4Y7VcAv5b0EtABzEno6wrgJkl9hKoQFUvvOI4zHqhfMC1CPdZwc/UUoqjMPWa2ssz+YeDhHD8ryj7PzxxPie/3A/fnXDslc9wKtCbehuM4jYgBDVxE0jPNHMcZX/gMd3jMbHa9x1AVM6zveGHzpvPfmeR+oHtP6oiSGDx2LP2iU7kmgYED6foOKUzYuivJPnVeNGHmjCT7/p6XhjfKcMa2Z4c3KuN0h5pZt1RN4jyJ9f/cVth26dJEfZBcapbae1pomIDrOI4zYgysTntsi1B1l0LM4lpc1vbNmOFVdd1T0mxJIxZ7cBzHSWIMZ5ptIKjkZFkO3Gdmw9XwmU0N1HUcx3GSaOBtYcMF3FZgmaSJEGatwLlAj6Su2NYk6XZJnVH962vx2u8DC6OOwbckXSPpQUm/lfRXST8odSLpJ7E6breklZn2FyV9T9L2eH6BpHZJf5P09YzdTZn+V8a2yZIejXoJXZKuiu0XS/qDpF3R1/SRfomO4zQIZmGXQpFXHai6hmtmr0p6ClhC2Jq1nFDaPPvzcB1wxMwuiYH5SUmbgW8DN5rZMgBJ1xCSHC4iZIjtkbTGzPYB3zWzQ7Fw21ZJF5jZc9H/PjO7TNKdwHrgQ4Rss25CzbNFwDyCnoKANkmXA9OA/Wb2ydh/S0zxXQN8xswOxCB8G3DtqX19juM0HGN8l0JpWaEUcMuD0yLgAg2VCW4hBMC8R/pbzewIgKTngVmEukFfiLoFEwhqYu8BSgG3Lb7/CZhiZr1Ar6TXY/LEovh6JtpNif0/AdwhaTXwiJk9IWk+MB94LO77bQL+VenGXUvBccYahg3UqnRE7SkScB8CfihpAfAmM3s6Li2UEPANM2vPXiTpihxfJ2kfKCiq3whcYmaHJa0nXy9hsOz6wTh+AavM7N7yzqJIzlJgVZx1/wboNrPLqt5xJBaUWwvQrKmN+7PpOE6gJM/YoAyrpWBmR4HfA+sYUvPK0g5cX5JGlPQOSZOBXuAtBcbQDBwDjkRd2k8UG/oJ/V8raUrsf4akt0k6F3jNzH4B3EHQUdgDTFMQQUfSmZLOT+zPcZxGpkbyjJKWSNoj6YWMomH2/ERJG+P5HWUT0VyK7sPdADzIyTsWAH5G2JHwtML/9APAZwlLAv2SdhPWXg/nOTaz3ZKeIazJ/h14suCYStdvlvRuYHtcJjgKfAmYC9yuUDO+D7jezI7HpY8fSWoh3P9dsW/HccY4BlgNZrjxedI9BOnWHqBTUpuZPZ8xuw44bGZzJS0HVjMkwJXv1xp4gbmRaNZU+6A+Vti+0TLNnNpzujPNxgNpmWYH2f1cX1558sI0a6pdOmFRIdvH+jfuqlQmPf4LXmFmi+Pn7wCY2aqMTXu02S5pAvBvYJpVCaqeaeY4zriiRg/NZhAe6JfoAcqL3b5hY2b9ko4AbwUOVnLqAbcgvRw+uMVa9+acOoe8L7iroqt8+8q4faPa78ttTfVf2zE1mP3M85LsZyX0mUsvh9u3WOs5Bc0nSdqZ+bw2PiiH8DC+nPKZaxGbE/CAWxAzm5bXLmlnpb8lbu/2jdDHWLdPwcyW1MhVD5D9uZgJ7K9g0xOXFFqAQ9Wc1qXEjuM4ToPTCcyTNEfSWYQNA21lNm0MVaH5PPC7auu34DNcx3Gck4hrsjcQtp02AevMrFvSrcBOM2sjlO16QNILhJlt3i6uE/CAO3LWDm/i9m5f1z7Gun1dMLNNwKaytpszx68DV6b49G1hjuM4o4Sv4TqO44wSHnAdx3FGCQ+4juM4o4QHXMdxnFHCA67jOM4o4QHXcRxnlPCA6ziOM0r8H+k+HtNvQPlSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2b761e6f898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 在混淆矩阵中跟踪正确的猜测\n",
    "confusion = torch.zeros(n_categories, n_categories)\n",
    "n_confusion = 10000\n",
    "\n",
    "# 只要返回给定一行的输出即可\n",
    "def evaluate(line_tensor):\n",
    "    hidden = rnn.initHidden()\n",
    "\n",
    "    for i in range(line_tensor.size()[0]):\n",
    "        output, hidden = rnn(line_tensor[i], hidden)\n",
    "\n",
    "    return output\n",
    "\n",
    "# 通过一堆示例并记录哪些是正确的猜测\n",
    "for i in range(n_confusion):\n",
    "    category, line, category_tensor, line_tensor = randomTrainingExample()\n",
    "    output = evaluate(line_tensor)\n",
    "    guess, guess_i = categoryFromOutput(output)\n",
    "    category_i = all_categories.index(category)\n",
    "    confusion[category_i][guess_i] += 1\n",
    "\n",
    "# 通过将每一行除以其总和来标准化\n",
    "for i in range(n_categories):\n",
    "    confusion[i] = confusion[i] / confusion[i].sum()\n",
    "\n",
    "# 设置绘图\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "cax = ax.matshow(confusion.numpy())\n",
    "fig.colorbar(cax)\n",
    "\n",
    "# 设置轴\n",
    "ax.set_xticklabels([''] + all_categories, rotation=90)\n",
    "ax.set_yticklabels([''] + all_categories)\n",
    "\n",
    "# Force label at every tick\n",
    "ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "# sphinx_gallery_thumbnail_number = 2\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "您可以从主轴上选取显示错误猜测哪些语言的亮点,\n",
    "e.g. Chinese for Korean, and Spanish for Italian.\n",
    "它似乎与希腊语很好,英语很差 (可能是因为与其他语言重叠).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在用户输入上运行\n",
    "---------------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "> Dovesky\n",
      "(-0.80) Russian\n",
      "(-0.84) Czech\n",
      "(-2.73) English\n",
      "\n",
      "> Jackson\n",
      "(-0.49) Scottish\n",
      "(-1.40) English\n",
      "(-3.02) Russian\n",
      "\n",
      "> Satoshi\n",
      "(-1.12) Japanese\n",
      "(-1.56) Polish\n",
      "(-1.81) Arabic\n"
     ]
    }
   ],
   "source": [
    "def predict(input_line, n_predictions=3):\n",
    "    print('\\n> %s' % input_line)\n",
    "    output = evaluate(Variable(lineToTensor(input_line)))\n",
    "\n",
    "    # 获取前N个类别\n",
    "    topv, topi = output.data.topk(n_predictions, 1, True)\n",
    "    predictions = []\n",
    "\n",
    "    for i in range(n_predictions):\n",
    "        value = topv[0][i]\n",
    "        category_index = topi[0][i]\n",
    "        print('(%.2f) %s' % (value, all_categories[category_index]))\n",
    "        predictions.append([value, all_categories[category_index]])\n",
    "\n",
    "predict('Dovesky')\n",
    "predict('Jackson')\n",
    "predict('Satoshi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "脚本的最终版本 `in the Practical PyTorch repo <https://github.com/spro/practical-pytorch/tree/master/char-rnn-classification>`__\n",
    "将上面的代码分成几个文件:\n",
    "\n",
    "-  ``data.py`` (加载文件)\n",
    "-  ``model.py`` (定义RNN)\n",
    "-  ``train.py`` (运行训练)\n",
    "-  ``predict.py`` (用命令行参数运行 ``predict()`` )\n",
    "-  ``server.py`` (使用bottle.py将预测用作JSON API)\n",
    "\n",
    "运行 ``train.py`` 来训练和保存网络.\n",
    "\n",
    "运行具有名称的 ``predict.py`` 来查看预测:\n",
    "\n",
    "::\n",
    "\n",
    "    $ python predict.py Hazaki\n",
    "    (-0.42) Japanese\n",
    "    (-1.39) Polish\n",
    "    (-3.51) Czech\n",
    "\n",
    "运行 ``server.py`` 和查看 http://localhost:5533/Yourname 获取预测的JSON输出.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "练习\n",
    "=========\n",
    "\n",
    "-  尝试使用不同的数据集 线条 -> 类别, 例如:\n",
    "\n",
    "   -  任何单词 -> 语言\n",
    "   -  姓 -> 性别\n",
    "   -  角色名字 -> 作家\n",
    "   -  页面标题 -> 博客或subreddit\n",
    "\n",
    "-  通过更大和/或更好的形状网络获得更好的结果\n",
    "\n",
    "   -  添加更多线性图层\n",
    "   -  试试 ``nn.LSTM`` 和 ``nn.GRU`` 图层\n",
    "   -  将多个这些RNN组合为更高级别的网络\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}