pytorch/.ipynb_checkpoints/c-reinforcement_q_learning-checkpoint.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "强化学习（DQN）教程\n",
    "=====================================\n",
    "**作者**: `Adam Paszke <https://github.com/apaszke>`_\n",
    "\n",
    "\n",
    "本教程演示如何使用 PyTorch 对任务 CartPole-v0 训练 Deep Q Learning（DQN）代理（即一个算法黑箱）, \n",
    "该任务来自于 `OpenAI Gym <https://gym.openai.com/>`__.\n",
    "\n",
    "**任务**\n",
    "\n",
    "该代理需要决定将小车往左还是往右推, 因此小车上的杆子始终保持竖直. 你可以在 \n",
    "`Gym website <https://gym.openai.com/envs/CartPole-v0>`__ 找到一个官方的公示榜单, 其罗列了不同的算法和可视化. \n",
    "\n",
    ".. 图:: /_static/img/cartpole.gif\n",
    "   :alt: 小车推杆\n",
    "\n",
    "   小车推杆\n",
    "\n",
    "代理通过观察当前环境（小车和杆子的组合体）下的状态选择一个合适的行为（往左还是往右推）, 随后环境状态得到转变, \n",
    "并返回一个回馈因子来量化该行为所带来的后果（好处或是坏处）. 在这个任务中, 如果杆子移动太远则整个代理环境终止. \n",
    "\n",
    "小车推杆任务被设计为有4个输入参数传给代理, 它们是环境状态, 环境位置, 环境速率等. \n",
    "然而, 神经网络单靠观察这个场景就可以解决该任务, 因此我们用一些以小车为中心的屏幕图作为输入参数就行了. \n",
    "正因为此, 我们并不能仅仅凭借将我们所得结果与公示榜单上的结果对比来得出结论, 我们的任务远比这个难. \n",
    "不幸的是, 这将会导致我们的训练速度变慢, 因为我们必须得渲染屏幕图所有的帧数. \n",
    "\n",
    "严格来说, 我们将状态定义为前一个屏幕图与当前屏幕图之间的差别. 这也会使得代理将图中推杆的速率也考虑进去. \n",
    "\n",
    "**包**\n",
    "\n",
    "\n",
    "首先, 我们导入一些需要用到的包.  第一, 我们需要 `gym包 <https://gym.openai.com/docs>`__ \n",
    "环境需要这个包（使用 'pip install gym' 安装该包). \n",
    "我们也会使用来自于 PyTorch 的以下包:\n",
    "\n",
    "-  神经网络 neural networks (``torch.nn``)\n",
    "-  优化 optimization (``torch.optim``)\n",
    "-  自微分 automatic differentiation (``torch.autograd``)\n",
    "-  视觉任务工具 (``torchvision`` - `一个独立的包 <https://github.com/pytorch/vision>`__).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__class__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " 'action_space',\n",
       " 'close',\n",
       " 'force_mag',\n",
       " 'gravity',\n",
       " 'length',\n",
       " 'masscart',\n",
       " 'masspole',\n",
       " 'metadata',\n",
       " 'np_random',\n",
       " 'observation_space',\n",
       " 'polemass_length',\n",
       " 'render',\n",
       " 'reset',\n",
       " 'reward_range',\n",
       " 'seed',\n",
       " 'spec',\n",
       " 'state',\n",
       " 'step',\n",
       " 'steps_beyond_done',\n",
       " 'tau',\n",
       " 'theta_threshold_radians',\n",
       " 'total_mass',\n",
       " 'unwrapped',\n",
       " 'viewer',\n",
       " 'x_threshold']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import math\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import namedtuple\n",
    "from itertools import count\n",
    "from copy import deepcopy\n",
    "from PIL import Image\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "import torchvision.transforms as T\n",
    "\n",
    "\n",
    "env = gym.make('CartPole-v0').unwrapped\n",
    "\n",
    "# 设置 matplotlib\n",
    "is_ipython = 'inline' in matplotlib.get_backend()\n",
    "if is_ipython:\n",
    "    from IPython import display\n",
    "\n",
    "plt.ion()\n",
    "\n",
    "# 如果要使用 gpu 的话\n",
    "use_cuda = torch.cuda.is_available()\n",
    "FloatTensor = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor\n",
    "LongTensor = torch.cuda.LongTensor if use_cuda else torch.LongTensor\n",
    "ByteTensor = torch.cuda.ByteTensor if use_cuda else torch.ByteTensor\n",
    "Tensor = FloatTensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "重播记忆 (Replay Memory)\n",
    "-------------\n",
    "\n",
    "我们将使用体验重播记忆来训练我们的DQN.  它存储了代理观察到的变化过程, 允许我们之后能够\n",
    "重复使用这些数据. 通过对重播记忆随机取样, 建立了批处理的变化过程将会被解耦合. 这一机制\n",
    "也被证明能够大幅度地提高和优化 DNQ 训练步骤的稳定性. \n",
    "\n",
    "对此, 我们将需要两个类:\n",
    "\n",
    "-  ``Transition`` - 一个命名元祖（tuple）, 代表了环境的单次变化\n",
    "\n",
    "-  ``ReplayMemory`` - 一个有限大小的循环缓冲区, 用于保存最近观察到的转换过程.  \n",
    "                      它也实现了``.sample（）``方法, 用于选择随机批次的转换进行训练\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "Transition = namedtuple('Transition',\n",
    "                        ('state', 'action', 'next_state', 'reward'))\n",
    "\n",
    "\n",
    "class ReplayMemory(object):\n",
    "\n",
    "    def __init__(self, capacity):\n",
    "        self.capacity = capacity\n",
    "        self.memory = []\n",
    "        self.position = 0\n",
    "\n",
    "    def push(self, *args):\n",
    "        \"\"\"Saves a transition.\"\"\"\n",
    "        if len(self.memory) < self.capacity:\n",
    "            self.memory.append(None)\n",
    "        self.memory[self.position] = Transition(*args)\n",
    "        self.position = (self.position + 1) % self.capacity\n",
    "\n",
    "    def sample(self, batch_size):\n",
    "        return random.sample(self.memory, batch_size)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.memory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在, 让我们开始构建我们的模型. 但是在此之前我们首先得重新定义什么是 DNQ. \n",
    "\n",
    "DQN 算法\n",
    "-------------\n",
    "\n",
    "我们的环境是确定性的, 所以这里提出的所有方程也都是为简单起见而确定性地制定的.  \n",
    "在强化学习概念中, 其还会包含有对环境中随机变化过程的期望值. \n",
    "\n",
    "我们的目标是训练出一个机制, 尽可能做到最大化折扣因子和累积回馈因子. \n",
    "$R_{t_0} = \\sum_{t=t_0}^{\\infty} \\gamma^{t - t_0} r_t$, \n",
    "$R_{t_0}$ 也被称为 *回馈因子*. 折扣因子 $\\gamma$ 应该是一个\n",
    "位于 $0$ 和 $1$ 之间的常量, 且确保了其总和是收敛的. 它的存在意义是让\n",
    "回馈的重要程度与时间成正比, 即离现在1分钟的回馈比离现在1小时的回馈要更重要, \n",
    "因为离当前时间越近, 我们的预测值可以更准确, 更可信. \n",
    "\n",
    "Q-learning 的主要原理是, 假如我们有一个函数, \n",
    "$Q^*: State \\times Action \\rightarrow \\mathbb{R}$, 这将会定义返回值, \n",
    "如果我们在给定状态下做出动作, 那么我们将会更容易据此训练出一个机制, 并做到最大化回馈因子. \n",
    "\n",
    "\\begin{align}\\pi^*(s) = \\arg\\!\\max_a \\ Q^*(s, a)\\end{align}\n",
    "\n",
    "由于我们对整个环境一无所知, 我们不需要知道确定的 $Q^*$. 但是, 因为神经网络\n",
    "是一个泛化的逼近函数, 所以我们可以直接构造一个网络并训练它去模拟 $Q^*$即可. \n",
    "\n",
    "对于我们训练的更新规则来说, 我们只需要让每一个 $Q$ 遵从贝尔曼方程 (Bellman equation)\n",
    "就可以了. \n",
    "\n",
    "\\begin{align}Q^{\\pi}(s, a) = r + \\gamma Q^{\\pi}(s', \\pi(s'))\\end{align}\n",
    "\n",
    "方程两边的实际差值即为时间差分误差 (temporal difference error), $\\delta$:\n",
    "\n",
    "\\begin{align}\\delta = Q(s, a) - (r + \\gamma \\max_a Q(s', a))\\end{align}\n",
    "\n",
    "为了使得该误差值最小化, 我们要使用 `Huber loss <https://en.wikipedia.org/wiki/Huber_loss>`__. \n",
    "当时间差分误差较小时, Huber loss 表现地与均方误差 (mean squared error) 一样, 而当时间差分误差较大时, \n",
    "Huber loss 表现地与绝对均差 (mean absolute error) 一样. 这一性质使得它在预测带有较多噪音的\n",
    "$Q$ 值上更具有鲁棒性. 我们通过从重播记忆中取出一批样本来计算 $B$. \n",
    "\n",
    "\\begin{align}\\mathcal{L} = \\frac{1}{|B|}\\sum_{(s, a, s', r) \\ \\in \\ B} \\mathcal{L}(\\delta)\\end{align}\n",
    "\n",
    "\\begin{align}\\text{where} \\quad \\mathcal{L}(\\delta) = \\begin{cases}\n",
    "     \\frac{1}{2}{\\delta^2}  & \\text{for } |\\delta| \\le 1, \\\\\n",
    "     |\\delta| - \\frac{1}{2} & \\text{otherwise.}\n",
    "   \\end{cases}\\end{align}\n",
    "\n",
    "Q-network\n",
    "^^^^^^^^^\n",
    "\n",
    "我们的模型是一个卷积神经网络 (CNN), 将当前屏幕图与之前屏幕图的差值作为唯一输入, 输出值有两个, \n",
    "分别代表 $Q(s, \\mathrm{left})$ 和 $Q(s, \\mathrm{right})$ (其中 $s$ \n",
    "是网络的输入). 从效果上来看, 我们的神经网络模型可以预测在当前输入下采取特定行为带来的 *quality*, \n",
    "即对整个环境的影响. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.BatchNorm2d?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.Conv2d?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DQN(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(DQN, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)\n",
    "        self.bn1 = nn.BatchNorm2d(16)\n",
    "        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)\n",
    "        self.bn2 = nn.BatchNorm2d(32)\n",
    "        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)\n",
    "        self.bn3 = nn.BatchNorm2d(32)\n",
    "        self.head = nn.Linear(448, 2)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.bn1(self.conv1(x)))\n",
    "        x = F.relu(self.bn2(self.conv2(x)))\n",
    "        x = F.relu(self.bn3(self.conv3(x)))\n",
    "        return self.head(x.view(x.size(0), -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获取输入 \n",
    "^^^^^^^^^^^^^^^^\n",
    "\n",
    "以下代码用来获取和处理来自环境中的被渲染的图像. 其中使用了 ``torchvision`` 包, \n",
    "这使得图像变换变得更加简单.  只要运行该代码块就会显示当前所提取图像. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F:\\app\\anaconda\\lib\\site-packages\\torchvision\\transforms\\transforms.py:188: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.\n",
      "  \"please use transforms.Resize instead.\")\n"
     ]
    },
    {
     "ename": "NotImplementedError",
     "evalue": "abstract",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNotImplementedError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-5-dc8561bb2a78>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     38\u001b[0m \u001b[0menv\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     39\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 40\u001b[1;33m plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),\n\u001b[0m\u001b[0;32m     41\u001b[0m            interpolation='none')\n\u001b[0;32m     42\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtitle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Example extracted screen'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-5-dc8561bb2a78>\u001b[0m in \u001b[0;36mget_screen\u001b[1;34m()\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mget_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m     screen = env.render(mode='rgb_array').transpose(\n\u001b[0m\u001b[0;32m     17\u001b[0m         (2, 0, 1))  # transpose into torch order (CHW)\n\u001b[0;32m     18\u001b[0m     \u001b[1;31m# Strip off the top and bottom of the screen\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\gym\\envs\\classic_control\\cartpole.py\u001b[0m in \u001b[0;36mrender\u001b[1;34m(self, mode)\u001b[0m\n\u001b[0;32m    105\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mviewer\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    106\u001b[0m             \u001b[1;32mfrom\u001b[0m \u001b[0mgym\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menvs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclassic_control\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mrendering\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 107\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mviewer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrendering\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mViewer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mscreen_width\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscreen_height\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    108\u001b[0m             \u001b[0ml\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mr\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mt\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m-\u001b[0m\u001b[0mcartwidth\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcartwidth\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcartheight\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m-\u001b[0m\u001b[0mcartheight\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    109\u001b[0m             \u001b[0maxleoffset\u001b[0m \u001b[1;33m=\u001b[0m\u001b[0mcartheight\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m4.0\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\gym\\envs\\classic_control\\rendering.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, width, height, display)\u001b[0m\n\u001b[0;32m     49\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwidth\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mwidth\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     50\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mheight\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mheight\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 51\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpyglet\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mWindow\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mwidth\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mwidth\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheight\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mheight\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdisplay\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     52\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_close\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow_closed_by_user\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     53\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0misopen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\pyglet\\window\\__init__.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, width, height, caption, resizable, style, fullscreen, visible, vsync, display, screen, config, context, mode)\u001b[0m\n\u001b[0;32m    502\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    503\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mscreen\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 504\u001b[1;33m             \u001b[0mscreen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdisplay\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_default_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    505\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    506\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mconfig\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\pyglet\\canvas\\base.py\u001b[0m in \u001b[0;36mget_default_screen\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     71\u001b[0m         \u001b[1;33m:\u001b[0m\u001b[0mrtype\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;33m:\u001b[0m\u001b[1;32mclass\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mScreen\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     72\u001b[0m         '''\n\u001b[1;32m---> 73\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_screens\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     74\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     75\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mget_windows\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\pyglet\\canvas\\base.py\u001b[0m in \u001b[0;36mget_screens\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     63\u001b[0m         \u001b[1;33m:\u001b[0m\u001b[0mrtype\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mlist\u001b[0m \u001b[0mof\u001b[0m \u001b[1;33m:\u001b[0m\u001b[1;32mclass\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mScreen\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     64\u001b[0m         '''\n\u001b[1;32m---> 65\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'abstract'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     66\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     67\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mget_default_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNotImplementedError\u001b[0m: abstract"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25d9d3659b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resize = T.Compose([T.ToPILImage(),\n",
    "                    T.Scale(40, interpolation=Image.CUBIC),\n",
    "                    T.ToTensor()])\n",
    "\n",
    "# This is based on the code from gym.\n",
    "screen_width = 600\n",
    "\n",
    "\n",
    "def get_cart_location():\n",
    "    world_width = env.x_threshold * 2\n",
    "    scale = screen_width / world_width\n",
    "    return int(env.state[0] * scale + screen_width / 2.0)  # MIDDLE OF CART\n",
    "\n",
    "\n",
    "def get_screen():\n",
    "    screen = env.render(mode='rgb_array').transpose(\n",
    "        (2, 0, 1))  # transpose into torch order (CHW)\n",
    "    # Strip off the top and bottom of the screen\n",
    "    screen = screen[:, 160:320]\n",
    "    view_width = 320\n",
    "    cart_location = get_cart_location()\n",
    "    if cart_location < view_width // 2:\n",
    "        slice_range = slice(view_width)\n",
    "    elif cart_location > (screen_width - view_width // 2):\n",
    "        slice_range = slice(-view_width, None)\n",
    "    else:\n",
    "        slice_range = slice(cart_location - view_width // 2,\n",
    "                            cart_location + view_width // 2)\n",
    "    # Strip off the edges, so that we have a square image centered on a cart\n",
    "    screen = screen[:, :, slice_range]\n",
    "    # Convert to float, rescare, convert to torch tensor\n",
    "    # (this doesn't require a copy)\n",
    "    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255\n",
    "    screen = torch.from_numpy(screen)\n",
    "    # 调整大小并添加批量维度 (BCHW)\n",
    "    return resize(screen).unsqueeze(0).type(Tensor)\n",
    "\n",
    "env.reset()\n",
    "plt.figure()\n",
    "plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),\n",
    "           interpolation='none')\n",
    "plt.title('Example extracted screen')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练\n",
    "--------\n",
    "\n",
    "超参数和函数 \n",
    "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
    "这一代码块实例化我们的模型和优化器, 并且定义了一些函数. \n",
    "函数:\n",
    "\n",
    "-  ``Variable`` - 这是一个对 ``torch.autograd.Variable`` 的简单包装器, \n",
    "                  它会在我们每次构建变量时自动将数据发送到GPU. \n",
    "-  ``select_action`` - 根据ε贪婪法则选择后续行动.  简而言之, 我们有时会使用\n",
    "                       我们的模型来选择动作, 有时我们仅均匀采样.  选择随机动作的\n",
    "                       概率大小将从 “EPS_START” 开始, 并沿着到 “EPS_END” 的方向\n",
    "                       呈指数衰减.  ``EPS_DECAY`` 控制衰减速度. \n",
    "-  ``plot_durations`` - 一个协助绘制动态帧持续时间的函数, 以及过去100动态帧（官方\n",
    "                        评估中使用的测量方法）的平均值.  绘制的图像将会显示在包含\n",
    "                        主要训练循环的单元代码块下面, 并且在每节动态帧之后更新. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "GAMMA = 0.999\n",
    "EPS_START = 0.9\n",
    "EPS_END = 0.05\n",
    "EPS_DECAY = 200\n",
    "\n",
    "model = DQN()\n",
    "\n",
    "if use_cuda:\n",
    "    model.cuda()\n",
    "\n",
    "optimizer = optim.RMSprop(model.parameters())\n",
    "memory = ReplayMemory(10000)\n",
    "\n",
    "\n",
    "steps_done = 0\n",
    "\n",
    "\n",
    "def select_action(state):\n",
    "    global steps_done\n",
    "    sample = random.random()\n",
    "    eps_threshold = EPS_END + (EPS_START - EPS_END) * math.exp(-1. * steps_done / EPS_DECAY)\n",
    "    steps_done += 1\n",
    "    if sample > eps_threshold:\n",
    "        return model(\n",
    "            Variable(state, volatile=True).type(FloatTensor)).data.max(1)[1].view(1, 1)\n",
    "    else:\n",
    "        return LongTensor([[random.randrange(2)]])\n",
    "\n",
    "\n",
    "episode_durations = []\n",
    "\n",
    "\n",
    "def plot_durations():\n",
    "    '''\n",
    "    持续时间\n",
    "    '''\n",
    "    plt.figure(2)\n",
    "    plt.clf()\n",
    "    durations_t = torch.FloatTensor(episode_durations)\n",
    "    plt.title('Training...')\n",
    "    plt.xlabel('Episode')\n",
    "    plt.ylabel('Duration')\n",
    "    plt.plot(durations_t.numpy())\n",
    "    # Take 100 episode averages and plot them too\n",
    "    if len(durations_t) >= 100:\n",
    "        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)\n",
    "        means = torch.cat((torch.zeros(99), means))\n",
    "        plt.plot(means.numpy())\n",
    "\n",
    "    plt.pause(0.001)  # pause a bit so that plots are updated\n",
    "    if is_ipython:\n",
    "        display.clear_output(wait=True)\n",
    "        display.display(plt.gcf())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练循环 \n",
    "^^^^^^^^^^^^^\n",
    "\n",
    "最终用于训练模型的代码\n",
    "\n",
    "在下面代码中有一个 ``optimize_model`` 函数, 它用于实现优化过程的其中一步. 它首先\n",
    "取出一个批次的样本, 然后将所有的张量全部合并到一个中, 并计算 $Q(s_t, a_t)$ \n",
    "和 $V(s_{t+1}) = \\max_a Q(s_{t+1}, a)$, 最终将这些结果全都融入到loss中去. \n",
    "假如 $s$ 是一个终止状态, 则 $V(s) = 0$. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_sync = 0\n",
    "\n",
    "\n",
    "def optimize_model():\n",
    "    global last_sync\n",
    "    if len(memory) < BATCH_SIZE:\n",
    "        return\n",
    "    transitions = memory.sample(BATCH_SIZE)\n",
    "    # 将betch转置 (详见 http://stackoverflow.com/a/19343/3343043).\n",
    "    batch = Transition(*zip(*transitions))\n",
    "\n",
    "    # 计算非最终状态的掩码并连接批处理元素s\n",
    "    non_final_mask = ByteTensor(tuple(map(lambda s: s is not None,\n",
    "                                          batch.next_state)))\n",
    "\n",
    "    # 我们不想通过预期的动作值反向传播, volatile 变量会临时将模型参数\n",
    "    # 'requires_grad' 更改为False！\n",
    "    non_final_next_states = Variable(torch.cat([s for s in batch.next_state\n",
    "                                                if s is not None]),\n",
    "                                     volatile=True)\n",
    "    state_batch = Variable(torch.cat(batch.state))\n",
    "    action_batch = Variable(torch.cat(batch.action))\n",
    "    reward_batch = Variable(torch.cat(batch.reward))\n",
    "\n",
    "    # 计算 Q(s_t, a) - 模型计算出 Q(s_t), 然后我们选择某一栏动作执行\n",
    "    state_action_values = model(state_batch).gather(1, action_batch)\n",
    "\n",
    "    # 对所有下一状态计算出 V(s_{t+1})\n",
    "    next_state_values = Variable(torch.zeros(BATCH_SIZE).type(Tensor))\n",
    "    next_state_values[non_final_mask] = model(non_final_next_states).max(1)[0]\n",
    "    # 此时我们不想让 volatile flag 混乱了我们的loss, 因此我们将其置为False\n",
    "    # 在此之后, 我们将会直接丢弃满足该变量, 并设 requires_grad=False\n",
    "    next_state_values.volatile = False\n",
    "    # 计算 Q 的期望值\n",
    "    expected_state_action_values = (next_state_values * GAMMA) + reward_batch\n",
    "\n",
    "    # 计算 Huber 损失\n",
    "    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values)\n",
    "\n",
    "    # 优化模型\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    for param in model.parameters():\n",
    "        param.grad.data.clamp_(-1, 1)\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面代码中包含主要的训练循环. 首先, 我们重新设置环境, 并实例化 ``state`` 变量. \n",
    "然后, 我们对动作取样并执行, 观察下一屏幕图并得到回馈因子 (通常为1), 同时优化一次模型. \n",
    "当动态帧结束时 (即我们的模型fail了), 开始新一轮的循环. \n",
    "\n",
    "下面的 `num_episodes` 变量设置的很小. 你可以把这个 notebook 下载下来然后运行更多帧. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "NotImplementedError",
     "evalue": "abstract",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNotImplementedError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-8-558f33296baf>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      3\u001b[0m     \u001b[1;31m# Initialize the environment and state\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m     \u001b[0menv\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m     \u001b[0mlast_screen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m     \u001b[0mcurrent_screen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m     \u001b[0mstate\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcurrent_screen\u001b[0m \u001b[1;33m-\u001b[0m \u001b[0mlast_screen\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-5-dc8561bb2a78>\u001b[0m in \u001b[0;36mget_screen\u001b[1;34m()\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mget_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m     screen = env.render(mode='rgb_array').transpose(\n\u001b[0m\u001b[0;32m     17\u001b[0m         (2, 0, 1))  # transpose into torch order (CHW)\n\u001b[0;32m     18\u001b[0m     \u001b[1;31m# Strip off the top and bottom of the screen\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\gym\\envs\\classic_control\\cartpole.py\u001b[0m in \u001b[0;36mrender\u001b[1;34m(self, mode)\u001b[0m\n\u001b[0;32m    105\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mviewer\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    106\u001b[0m             \u001b[1;32mfrom\u001b[0m \u001b[0mgym\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0menvs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclassic_control\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mrendering\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 107\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mviewer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrendering\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mViewer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mscreen_width\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscreen_height\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    108\u001b[0m             \u001b[0ml\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mr\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mt\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m-\u001b[0m\u001b[0mcartwidth\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcartwidth\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcartheight\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m-\u001b[0m\u001b[0mcartheight\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    109\u001b[0m             \u001b[0maxleoffset\u001b[0m \u001b[1;33m=\u001b[0m\u001b[0mcartheight\u001b[0m\u001b[1;33m/\u001b[0m\u001b[1;36m4.0\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\gym\\envs\\classic_control\\rendering.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, width, height, display)\u001b[0m\n\u001b[0;32m     49\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwidth\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mwidth\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     50\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mheight\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mheight\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 51\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpyglet\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mWindow\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mwidth\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mwidth\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheight\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mheight\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdisplay\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     52\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_close\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwindow_closed_by_user\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     53\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0misopen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\pyglet\\window\\__init__.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, width, height, caption, resizable, style, fullscreen, visible, vsync, display, screen, config, context, mode)\u001b[0m\n\u001b[0;32m    502\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    503\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mscreen\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 504\u001b[1;33m             \u001b[0mscreen\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdisplay\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_default_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    505\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    506\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mconfig\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\pyglet\\canvas\\base.py\u001b[0m in \u001b[0;36mget_default_screen\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     71\u001b[0m         \u001b[1;33m:\u001b[0m\u001b[0mrtype\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;33m:\u001b[0m\u001b[1;32mclass\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mScreen\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     72\u001b[0m         '''\n\u001b[1;32m---> 73\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_screens\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     74\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     75\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mget_windows\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mF:\\app\\anaconda\\lib\\site-packages\\pyglet\\canvas\\base.py\u001b[0m in \u001b[0;36mget_screens\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     63\u001b[0m         \u001b[1;33m:\u001b[0m\u001b[0mrtype\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mlist\u001b[0m \u001b[0mof\u001b[0m \u001b[1;33m:\u001b[0m\u001b[1;32mclass\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mScreen\u001b[0m\u001b[0;31m`\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     64\u001b[0m         '''\n\u001b[1;32m---> 65\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'abstract'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     66\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     67\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mget_default_screen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNotImplementedError\u001b[0m: abstract"
     ]
    }
   ],
   "source": [
    "num_episodes = 10\n",
    "for i_episode in range(num_episodes):\n",
    "    # Initialize the environment and state\n",
    "    env.reset()\n",
    "    last_screen = get_screen()\n",
    "    current_screen = get_screen()\n",
    "    state = current_screen - last_screen\n",
    "    for t in count():\n",
    "        # Select and perform an action\n",
    "        action = select_action(state)\n",
    "        _, reward, done, _ = env.step(action[0, 0])\n",
    "        reward = Tensor([reward])\n",
    "\n",
    "        # 观察记录新状态\n",
    "        last_screen = current_screen\n",
    "        current_screen = get_screen()\n",
    "        if not done:\n",
    "            next_state = current_screen - last_screen\n",
    "        else:\n",
    "            next_state = None\n",
    "\n",
    "        # 将变化过程存到内存中\n",
    "        memory.push(state, action, next_state, reward)\n",
    "\n",
    "        # 转移到下一状态\n",
    "        state = next_state\n",
    "\n",
    "        # 对目标神经网络执行一步优化\n",
    "        optimize_model()\n",
    "        if done:\n",
    "            episode_durations.append(t + 1)\n",
    "            plot_durations()\n",
    "            break\n",
    "\n",
    "print('Complete')\n",
    "env.render(close=True)\n",
    "env.close()\n",
    "plt.ioff()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}