18_reinforcement_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 18 – Reinforcement Learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code in chapter 18_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ageron/handson-ml2/blob/master/18_reinforcement_learning.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    !apt update && apt install -y libpq-dev libsdl2-dev swig xorg-dev xvfb\n",
    "    !pip install -q -U tf-agents pyvirtualdisplay gym[atari]\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# To get smooth animations\n",
    "import matplotlib.animation as animation\n",
    "mpl.rc('animation', html='jshtml')\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"rl\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to OpenAI gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will be using [OpenAI gym](https://gym.openai.com/), a great toolkit for developing and comparing Reinforcement Learning algorithms. It provides many environments for your learning *agents* to interact with. Let's start by importing `gym`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's list all the available environments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_values([EnvSpec(Copy-v0), EnvSpec(RepeatCopy-v0), EnvSpec(ReversedAddition-v0), EnvSpec(ReversedAddition3-v0), EnvSpec(DuplicatedInput-v0), EnvSpec(Reverse-v0), EnvSpec(CartPole-v0), EnvSpec(CartPole-v1), EnvSpec(MountainCar-v0), EnvSpec(MountainCarContinuous-v0), EnvSpec(Pendulum-v0), EnvSpec(Acrobot-v1), EnvSpec(LunarLander-v2), EnvSpec(LunarLanderContinuous-v2), EnvSpec(BipedalWalker-v3), EnvSpec(BipedalWalkerHardcore-v3), EnvSpec(CarRacing-v0), EnvSpec(Blackjack-v0), EnvSpec(KellyCoinflip-v0), EnvSpec(KellyCoinflipGeneralized-v0), EnvSpec(FrozenLake-v0), EnvSpec(FrozenLake8x8-v0), EnvSpec(CliffWalking-v0), EnvSpec(NChain-v0), EnvSpec(Roulette-v0), EnvSpec(Taxi-v3), EnvSpec(GuessingGame-v0), EnvSpec(HotterColder-v0), EnvSpec(Reacher-v2), EnvSpec(Pusher-v2), EnvSpec(Thrower-v2), EnvSpec(Striker-v2), EnvSpec(InvertedPendulum-v2), EnvSpec(InvertedDoublePendulum-v2), EnvSpec(HalfCheetah-v2), EnvSpec(HalfCheetah-v3), EnvSpec(Hopper-v2), EnvSpec(Hopper-v3), EnvSpec(Swimmer-v2), EnvSpec(Swimmer-v3), EnvSpec(Walker2d-v2), EnvSpec(Walker2d-v3), EnvSpec(Ant-v2), EnvSpec(Ant-v3), EnvSpec(Humanoid-v2), EnvSpec(Humanoid-v3), EnvSpec(HumanoidStandup-v2), EnvSpec(FetchSlide-v1), EnvSpec(FetchPickAndPlace-v1), EnvSpec(FetchReach-v1), EnvSpec(FetchPush-v1), EnvSpec(HandReach-v0), EnvSpec(HandManipulateBlockRotateZ-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v1), EnvSpec(HandManipulateBlockRotateParallel-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v1), EnvSpec(HandManipulateBlockRotateXYZ-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v1), EnvSpec(HandManipulateBlockFull-v0), EnvSpec(HandManipulateBlock-v0), EnvSpec(HandManipulateBlockTouchSensors-v0), EnvSpec(HandManipulateBlockTouchSensors-v1), EnvSpec(HandManipulateEggRotate-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v1), EnvSpec(HandManipulateEggFull-v0), EnvSpec(HandManipulateEgg-v0), EnvSpec(HandManipulateEggTouchSensors-v0), EnvSpec(HandManipulateEggTouchSensors-v1), EnvSpec(HandManipulatePenRotate-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v1), EnvSpec(HandManipulatePenFull-v0), EnvSpec(HandManipulatePen-v0), EnvSpec(HandManipulatePenTouchSensors-v0), EnvSpec(HandManipulatePenTouchSensors-v1), EnvSpec(FetchSlideDense-v1), EnvSpec(FetchPickAndPlaceDense-v1), EnvSpec(FetchReachDense-v1), EnvSpec(FetchPushDense-v1), EnvSpec(HandReachDense-v0), EnvSpec(HandManipulateBlockRotateZDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateParallelDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateXYZDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockFullDense-v0), EnvSpec(HandManipulateBlockDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v1), EnvSpec(HandManipulateEggRotateDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v1), EnvSpec(HandManipulateEggFullDense-v0), EnvSpec(HandManipulateEggDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v1), EnvSpec(HandManipulatePenRotateDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v1), EnvSpec(HandManipulatePenFullDense-v0), EnvSpec(HandManipulatePenDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v1), EnvSpec(Adventure-v0), EnvSpec(Adventure-v4), EnvSpec(AdventureDeterministic-v0), EnvSpec(AdventureDeterministic-v4), EnvSpec(AdventureNoFrameskip-v0), EnvSpec(AdventureNoFrameskip-v4), EnvSpec(Adventure-ram-v0), EnvSpec(Adventure-ram-v4), EnvSpec(Adventure-ramDeterministic-v0), EnvSpec(Adventure-ramDeterministic-v4), EnvSpec(Adventure-ramNoFrameskip-v0), EnvSpec(Adventure-ramNoFrameskip-v4), EnvSpec(AirRaid-v0), EnvSpec(AirRaid-v4), EnvSpec(AirRaidDeterministic-v0), EnvSpec(AirRaidDeterministic-v4), EnvSpec(AirRaidNoFrameskip-v0), EnvSpec(AirRaidNoFrameskip-v4), EnvSpec(AirRaid-ram-v0), EnvSpec(AirRaid-ram-v4), EnvSpec(AirRaid-ramDeterministic-v0), EnvSpec(AirRaid-ramDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v0), EnvSpec(AirRaid-ramNoFrameskip-v4), EnvSpec(Alien-v0), EnvSpec(Alien-v4), EnvSpec(AlienDeterministic-v0), EnvSpec(AlienDeterministic-v4), EnvSpec(AlienNoFrameskip-v0), EnvSpec(AlienNoFrameskip-v4), EnvSpec(Alien-ram-v0), EnvSpec(Alien-ram-v4), EnvSpec(Alien-ramDeterministic-v0), EnvSpec(Alien-ramDeterministic-v4), EnvSpec(Alien-ramNoFrameskip-v0), EnvSpec(Alien-ramNoFrameskip-v4), EnvSpec(Amidar-v0), EnvSpec(Amidar-v4), EnvSpec(AmidarDeterministic-v0), EnvSpec(AmidarDeterministic-v4), EnvSpec(AmidarNoFrameskip-v0), EnvSpec(AmidarNoFrameskip-v4), EnvSpec(Amidar-ram-v0), EnvSpec(Amidar-ram-v4), EnvSpec(Amidar-ramDeterministic-v0), EnvSpec(Amidar-ramDeterministic-v4), EnvSpec(Amidar-ramNoFrameskip-v0), EnvSpec(Amidar-ramNoFrameskip-v4), EnvSpec(Assault-v0), EnvSpec(Assault-v4), EnvSpec(AssaultDeterministic-v0), EnvSpec(AssaultDeterministic-v4), EnvSpec(AssaultNoFrameskip-v0), EnvSpec(AssaultNoFrameskip-v4), EnvSpec(Assault-ram-v0), EnvSpec(Assault-ram-v4), EnvSpec(Assault-ramDeterministic-v0), EnvSpec(Assault-ramDeterministic-v4), EnvSpec(Assault-ramNoFrameskip-v0), EnvSpec(Assault-ramNoFrameskip-v4), EnvSpec(Asterix-v0), EnvSpec(Asterix-v4), EnvSpec(AsterixDeterministic-v0), EnvSpec(AsterixDeterministic-v4), EnvSpec(AsterixNoFrameskip-v0), EnvSpec(AsterixNoFrameskip-v4), EnvSpec(Asterix-ram-v0), EnvSpec(Asterix-ram-v4), EnvSpec(Asterix-ramDeterministic-v0), EnvSpec(Asterix-ramDeterministic-v4), EnvSpec(Asterix-ramNoFrameskip-v0), EnvSpec(Asterix-ramNoFrameskip-v4), EnvSpec(Asteroids-v0), EnvSpec(Asteroids-v4), EnvSpec(AsteroidsDeterministic-v0), EnvSpec(AsteroidsDeterministic-v4), EnvSpec(AsteroidsNoFrameskip-v0), EnvSpec(AsteroidsNoFrameskip-v4), EnvSpec(Asteroids-ram-v0), EnvSpec(Asteroids-ram-v4), EnvSpec(Asteroids-ramDeterministic-v0), EnvSpec(Asteroids-ramDeterministic-v4), EnvSpec(Asteroids-ramNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v4), EnvSpec(Atlantis-v0), EnvSpec(Atlantis-v4), EnvSpec(AtlantisDeterministic-v0), EnvSpec(AtlantisDeterministic-v4), EnvSpec(AtlantisNoFrameskip-v0), EnvSpec(AtlantisNoFrameskip-v4), EnvSpec(Atlantis-ram-v0), EnvSpec(Atlantis-ram-v4), EnvSpec(Atlantis-ramDeterministic-v0), EnvSpec(Atlantis-ramDeterministic-v4), EnvSpec(Atlantis-ramNoFrameskip-v0), EnvSpec(Atlantis-ramNoFrameskip-v4), EnvSpec(BankHeist-v0), EnvSpec(BankHeist-v4), EnvSpec(BankHeistDeterministic-v0), EnvSpec(BankHeistDeterministic-v4), EnvSpec(BankHeistNoFrameskip-v0), EnvSpec(BankHeistNoFrameskip-v4), EnvSpec(BankHeist-ram-v0), EnvSpec(BankHeist-ram-v4), EnvSpec(BankHeist-ramDeterministic-v0), EnvSpec(BankHeist-ramDeterministic-v4), EnvSpec(BankHeist-ramNoFrameskip-v0), EnvSpec(BankHeist-ramNoFrameskip-v4), EnvSpec(BattleZone-v0), EnvSpec(BattleZone-v4), EnvSpec(BattleZoneDeterministic-v0), EnvSpec(BattleZoneDeterministic-v4), EnvSpec(BattleZoneNoFrameskip-v0), EnvSpec(BattleZoneNoFrameskip-v4), EnvSpec(BattleZone-ram-v0), EnvSpec(BattleZone-ram-v4), EnvSpec(BattleZone-ramDeterministic-v0), EnvSpec(BattleZone-ramDeterministic-v4), EnvSpec(BattleZone-ramNoFrameskip-v0), EnvSpec(BattleZone-ramNoFrameskip-v4), EnvSpec(BeamRider-v0), EnvSpec(BeamRider-v4), EnvSpec(BeamRiderDeterministic-v0), EnvSpec(BeamRiderDeterministic-v4), EnvSpec(BeamRiderNoFrameskip-v0), EnvSpec(BeamRiderNoFrameskip-v4), EnvSpec(BeamRider-ram-v0), EnvSpec(BeamRider-ram-v4), EnvSpec(BeamRider-ramDeterministic-v0), EnvSpec(BeamRider-ramDeterministic-v4), EnvSpec(BeamRider-ramNoFrameskip-v0), EnvSpec(BeamRider-ramNoFrameskip-v4), EnvSpec(Berzerk-v0), EnvSpec(Berzerk-v4), EnvSpec(BerzerkDeterministic-v0), EnvSpec(BerzerkDeterministic-v4), EnvSpec(BerzerkNoFrameskip-v0), EnvSpec(BerzerkNoFrameskip-v4), EnvSpec(Berzerk-ram-v0), EnvSpec(Berzerk-ram-v4), EnvSpec(Berzerk-ramDeterministic-v0), EnvSpec(Berzerk-ramDeterministic-v4), EnvSpec(Berzerk-ramNoFrameskip-v0), EnvSpec(Berzerk-ramNoFrameskip-v4), EnvSpec(Bowling-v0), EnvSpec(Bowling-v4), EnvSpec(BowlingDeterministic-v0), EnvSpec(BowlingDeterministic-v4), EnvSpec(BowlingNoFrameskip-v0), EnvSpec(BowlingNoFrameskip-v4), EnvSpec(Bowling-ram-v0), EnvSpec(Bowling-ram-v4), EnvSpec(Bowling-ramDeterministic-v0), EnvSpec(Bowling-ramDeterministic-v4), EnvSpec(Bowling-ramNoFrameskip-v0), EnvSpec(Bowling-ramNoFrameskip-v4), EnvSpec(Boxing-v0), EnvSpec(Boxing-v4), EnvSpec(BoxingDeterministic-v0), EnvSpec(BoxingDeterministic-v4), EnvSpec(BoxingNoFrameskip-v0), EnvSpec(BoxingNoFrameskip-v4), EnvSpec(Boxing-ram-v0), EnvSpec(Boxing-ram-v4), EnvSpec(Boxing-ramDeterministic-v0), EnvSpec(Boxing-ramDeterministic-v4), EnvSpec(Boxing-ramNoFrameskip-v0), EnvSpec(Boxing-ramNoFrameskip-v4), EnvSpec(Breakout-v0), EnvSpec(Breakout-v4), EnvSpec(BreakoutDeterministic-v0), EnvSpec(BreakoutDeterministic-v4), EnvSpec(BreakoutNoFrameskip-v0), EnvSpec(BreakoutNoFrameskip-v4), EnvSpec(Breakout-ram-v0), EnvSpec(Breakout-ram-v4), EnvSpec(Breakout-ramDeterministic-v0), EnvSpec(Breakout-ramDeterministic-v4), EnvSpec(Breakout-ramNoFrameskip-v0), EnvSpec(Breakout-ramNoFrameskip-v4), EnvSpec(Carnival-v0), EnvSpec(Carnival-v4), EnvSpec(CarnivalDeterministic-v0), EnvSpec(CarnivalDeterministic-v4), EnvSpec(CarnivalNoFrameskip-v0), EnvSpec(CarnivalNoFrameskip-v4), EnvSpec(Carnival-ram-v0), EnvSpec(Carnival-ram-v4), EnvSpec(Carnival-ramDeterministic-v0), EnvSpec(Carnival-ramDeterministic-v4), EnvSpec(Carnival-ramNoFrameskip-v0), EnvSpec(Carnival-ramNoFrameskip-v4), EnvSpec(Centipede-v0), EnvSpec(Centipede-v4), EnvSpec(CentipedeDeterministic-v0), EnvSpec(CentipedeDeterministic-v4), EnvSpec(CentipedeNoFrameskip-v0), EnvSpec(CentipedeNoFrameskip-v4), EnvSpec(Centipede-ram-v0), EnvSpec(Centipede-ram-v4), EnvSpec(Centipede-ramDeterministic-v0), EnvSpec(Centipede-ramDeterministic-v4), EnvSpec(Centipede-ramNoFrameskip-v0), EnvSpec(Centipede-ramNoFrameskip-v4), EnvSpec(ChopperCommand-v0), EnvSpec(ChopperCommand-v4), EnvSpec(ChopperCommandDeterministic-v0), EnvSpec(ChopperCommandDeterministic-v4), EnvSpec(ChopperCommandNoFrameskip-v0), EnvSpec(ChopperCommandNoFrameskip-v4), EnvSpec(ChopperCommand-ram-v0), EnvSpec(ChopperCommand-ram-v4), EnvSpec(ChopperCommand-ramDeterministic-v0), EnvSpec(ChopperCommand-ramDeterministic-v4), EnvSpec(ChopperCommand-ramNoFrameskip-v0), EnvSpec(ChopperCommand-ramNoFrameskip-v4), EnvSpec(CrazyClimber-v0), EnvSpec(CrazyClimber-v4), EnvSpec(CrazyClimberDeterministic-v0), EnvSpec(CrazyClimberDeterministic-v4), EnvSpec(CrazyClimberNoFrameskip-v0), EnvSpec(CrazyClimberNoFrameskip-v4), EnvSpec(CrazyClimber-ram-v0), EnvSpec(CrazyClimber-ram-v4), EnvSpec(CrazyClimber-ramDeterministic-v0), EnvSpec(CrazyClimber-ramDeterministic-v4), EnvSpec(CrazyClimber-ramNoFrameskip-v0), EnvSpec(CrazyClimber-ramNoFrameskip-v4), EnvSpec(Defender-v0), EnvSpec(Defender-v4), EnvSpec(DefenderDeterministic-v0), EnvSpec(DefenderDeterministic-v4), EnvSpec(DefenderNoFrameskip-v0), EnvSpec(DefenderNoFrameskip-v4), EnvSpec(Defender-ram-v0), EnvSpec(Defender-ram-v4), EnvSpec(Defender-ramDeterministic-v0), EnvSpec(Defender-ramDeterministic-v4), EnvSpec(Defender-ramNoFrameskip-v0), EnvSpec(Defender-ramNoFrameskip-v4), EnvSpec(DemonAttack-v0), EnvSpec(DemonAttack-v4), EnvSpec(DemonAttackDeterministic-v0), EnvSpec(DemonAttackDeterministic-v4), EnvSpec(DemonAttackNoFrameskip-v0), EnvSpec(DemonAttackNoFrameskip-v4), EnvSpec(DemonAttack-ram-v0), EnvSpec(DemonAttack-ram-v4), EnvSpec(DemonAttack-ramDeterministic-v0), EnvSpec(DemonAttack-ramDeterministic-v4), EnvSpec(DemonAttack-ramNoFrameskip-v0), EnvSpec(DemonAttack-ramNoFrameskip-v4), EnvSpec(DoubleDunk-v0), EnvSpec(DoubleDunk-v4), EnvSpec(DoubleDunkDeterministic-v0), EnvSpec(DoubleDunkDeterministic-v4), EnvSpec(DoubleDunkNoFrameskip-v0), EnvSpec(DoubleDunkNoFrameskip-v4), EnvSpec(DoubleDunk-ram-v0), EnvSpec(DoubleDunk-ram-v4), EnvSpec(DoubleDunk-ramDeterministic-v0), EnvSpec(DoubleDunk-ramDeterministic-v4), EnvSpec(DoubleDunk-ramNoFrameskip-v0), EnvSpec(DoubleDunk-ramNoFrameskip-v4), EnvSpec(ElevatorAction-v0), EnvSpec(ElevatorAction-v4), EnvSpec(ElevatorActionDeterministic-v0), EnvSpec(ElevatorActionDeterministic-v4), EnvSpec(ElevatorActionNoFrameskip-v0), EnvSpec(ElevatorActionNoFrameskip-v4), EnvSpec(ElevatorAction-ram-v0), EnvSpec(ElevatorAction-ram-v4), EnvSpec(ElevatorAction-ramDeterministic-v0), EnvSpec(ElevatorAction-ramDeterministic-v4), EnvSpec(ElevatorAction-ramNoFrameskip-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v4), EnvSpec(Enduro-v0), EnvSpec(Enduro-v4), EnvSpec(EnduroDeterministic-v0), EnvSpec(EnduroDeterministic-v4), EnvSpec(EnduroNoFrameskip-v0), EnvSpec(EnduroNoFrameskip-v4), EnvSpec(Enduro-ram-v0), EnvSpec(Enduro-ram-v4), EnvSpec(Enduro-ramDeterministic-v0), EnvSpec(Enduro-ramDeterministic-v4), EnvSpec(Enduro-ramNoFrameskip-v0), EnvSpec(Enduro-ramNoFrameskip-v4), EnvSpec(FishingDerby-v0), EnvSpec(FishingDerby-v4), EnvSpec(FishingDerbyDeterministic-v0), EnvSpec(FishingDerbyDeterministic-v4), EnvSpec(FishingDerbyNoFrameskip-v0), EnvSpec(FishingDerbyNoFrameskip-v4), EnvSpec(FishingDerby-ram-v0), EnvSpec(FishingDerby-ram-v4), EnvSpec(FishingDerby-ramDeterministic-v0), EnvSpec(FishingDerby-ramDeterministic-v4), EnvSpec(FishingDerby-ramNoFrameskip-v0), EnvSpec(FishingDerby-ramNoFrameskip-v4), EnvSpec(Freeway-v0), EnvSpec(Freeway-v4), EnvSpec(FreewayDeterministic-v0), EnvSpec(FreewayDeterministic-v4), EnvSpec(FreewayNoFrameskip-v0), EnvSpec(FreewayNoFrameskip-v4), EnvSpec(Freeway-ram-v0), EnvSpec(Freeway-ram-v4), EnvSpec(Freeway-ramDeterministic-v0), EnvSpec(Freeway-ramDeterministic-v4), EnvSpec(Freeway-ramNoFrameskip-v0), EnvSpec(Freeway-ramNoFrameskip-v4), EnvSpec(Frostbite-v0), EnvSpec(Frostbite-v4), EnvSpec(FrostbiteDeterministic-v0), EnvSpec(FrostbiteDeterministic-v4), EnvSpec(FrostbiteNoFrameskip-v0), EnvSpec(FrostbiteNoFrameskip-v4), EnvSpec(Frostbite-ram-v0), EnvSpec(Frostbite-ram-v4), EnvSpec(Frostbite-ramDeterministic-v0), EnvSpec(Frostbite-ramDeterministic-v4), EnvSpec(Frostbite-ramNoFrameskip-v0), EnvSpec(Frostbite-ramNoFrameskip-v4), EnvSpec(Gopher-v0), EnvSpec(Gopher-v4), EnvSpec(GopherDeterministic-v0), EnvSpec(GopherDeterministic-v4), EnvSpec(GopherNoFrameskip-v0), EnvSpec(GopherNoFrameskip-v4), EnvSpec(Gopher-ram-v0), EnvSpec(Gopher-ram-v4), EnvSpec(Gopher-ramDeterministic-v0), EnvSpec(Gopher-ramDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v0), EnvSpec(Gopher-ramNoFrameskip-v4), EnvSpec(Gravitar-v0), EnvSpec(Gravitar-v4), EnvSpec(GravitarDeterministic-v0), EnvSpec(GravitarDeterministic-v4), EnvSpec(GravitarNoFrameskip-v0), EnvSpec(GravitarNoFrameskip-v4), EnvSpec(Gravitar-ram-v0), EnvSpec(Gravitar-ram-v4), EnvSpec(Gravitar-ramDeterministic-v0), EnvSpec(Gravitar-ramDeterministic-v4), EnvSpec(Gravitar-ramNoFrameskip-v0), EnvSpec(Gravitar-ramNoFrameskip-v4), EnvSpec(Hero-v0), EnvSpec(Hero-v4), EnvSpec(HeroDeterministic-v0), EnvSpec(HeroDeterministic-v4), EnvSpec(HeroNoFrameskip-v0), EnvSpec(HeroNoFrameskip-v4), EnvSpec(Hero-ram-v0), EnvSpec(Hero-ram-v4), EnvSpec(Hero-ramDeterministic-v0), EnvSpec(Hero-ramDeterministic-v4), EnvSpec(Hero-ramNoFrameskip-v0), EnvSpec(Hero-ramNoFrameskip-v4), EnvSpec(IceHockey-v0), EnvSpec(IceHockey-v4), EnvSpec(IceHockeyDeterministic-v0), EnvSpec(IceHockeyDeterministic-v4), EnvSpec(IceHockeyNoFrameskip-v0), EnvSpec(IceHockeyNoFrameskip-v4), EnvSpec(IceHockey-ram-v0), EnvSpec(IceHockey-ram-v4), EnvSpec(IceHockey-ramDeterministic-v0), EnvSpec(IceHockey-ramDeterministic-v4), EnvSpec(IceHockey-ramNoFrameskip-v0), EnvSpec(IceHockey-ramNoFrameskip-v4), EnvSpec(Jamesbond-v0), EnvSpec(Jamesbond-v4), EnvSpec(JamesbondDeterministic-v0), EnvSpec(JamesbondDeterministic-v4), EnvSpec(JamesbondNoFrameskip-v0), EnvSpec(JamesbondNoFrameskip-v4), EnvSpec(Jamesbond-ram-v0), EnvSpec(Jamesbond-ram-v4), EnvSpec(Jamesbond-ramDeterministic-v0), EnvSpec(Jamesbond-ramDeterministic-v4), EnvSpec(Jamesbond-ramNoFrameskip-v0), EnvSpec(Jamesbond-ramNoFrameskip-v4), EnvSpec(JourneyEscape-v0), EnvSpec(JourneyEscape-v4), EnvSpec(JourneyEscapeDeterministic-v0), EnvSpec(JourneyEscapeDeterministic-v4), EnvSpec(JourneyEscapeNoFrameskip-v0), EnvSpec(JourneyEscapeNoFrameskip-v4), EnvSpec(JourneyEscape-ram-v0), EnvSpec(JourneyEscape-ram-v4), EnvSpec(JourneyEscape-ramDeterministic-v0), EnvSpec(JourneyEscape-ramDeterministic-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v0), EnvSpec(JourneyEscape-ramNoFrameskip-v4), EnvSpec(Kangaroo-v0), EnvSpec(Kangaroo-v4), EnvSpec(KangarooDeterministic-v0), EnvSpec(KangarooDeterministic-v4), EnvSpec(KangarooNoFrameskip-v0), EnvSpec(KangarooNoFrameskip-v4), EnvSpec(Kangaroo-ram-v0), EnvSpec(Kangaroo-ram-v4), EnvSpec(Kangaroo-ramDeterministic-v0), EnvSpec(Kangaroo-ramDeterministic-v4), EnvSpec(Kangaroo-ramNoFrameskip-v0), EnvSpec(Kangaroo-ramNoFrameskip-v4), EnvSpec(Krull-v0), EnvSpec(Krull-v4), EnvSpec(KrullDeterministic-v0), EnvSpec(KrullDeterministic-v4), EnvSpec(KrullNoFrameskip-v0), EnvSpec(KrullNoFrameskip-v4), EnvSpec(Krull-ram-v0), EnvSpec(Krull-ram-v4), EnvSpec(Krull-ramDeterministic-v0), EnvSpec(Krull-ramDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v0), EnvSpec(Krull-ramNoFrameskip-v4), EnvSpec(KungFuMaster-v0), EnvSpec(KungFuMaster-v4), EnvSpec(KungFuMasterDeterministic-v0), EnvSpec(KungFuMasterDeterministic-v4), EnvSpec(KungFuMasterNoFrameskip-v0), EnvSpec(KungFuMasterNoFrameskip-v4), EnvSpec(KungFuMaster-ram-v0), EnvSpec(KungFuMaster-ram-v4), EnvSpec(KungFuMaster-ramDeterministic-v0), EnvSpec(KungFuMaster-ramDeterministic-v4), EnvSpec(KungFuMaster-ramNoFrameskip-v0), EnvSpec(KungFuMaster-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-v0), EnvSpec(MontezumaRevenge-v4), EnvSpec(MontezumaRevengeDeterministic-v0), EnvSpec(MontezumaRevengeDeterministic-v4), EnvSpec(MontezumaRevengeNoFrameskip-v0), EnvSpec(MontezumaRevengeNoFrameskip-v4), EnvSpec(MontezumaRevenge-ram-v0), EnvSpec(MontezumaRevenge-ram-v4), EnvSpec(MontezumaRevenge-ramDeterministic-v0), EnvSpec(MontezumaRevenge-ramDeterministic-v4), EnvSpec(MontezumaRevenge-ramNoFrameskip-v0), EnvSpec(MontezumaRevenge-ramNoFrameskip-v4), EnvSpec(MsPacman-v0), EnvSpec(MsPacman-v4), EnvSpec(MsPacmanDeterministic-v0), EnvSpec(MsPacmanDeterministic-v4), EnvSpec(MsPacmanNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v4), EnvSpec(MsPacman-ram-v0), EnvSpec(MsPacman-ram-v4), EnvSpec(MsPacman-ramDeterministic-v0), EnvSpec(MsPacman-ramDeterministic-v4), EnvSpec(MsPacman-ramNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v4), EnvSpec(NameThisGame-v0), EnvSpec(NameThisGame-v4), EnvSpec(NameThisGameDeterministic-v0), EnvSpec(NameThisGameDeterministic-v4), EnvSpec(NameThisGameNoFrameskip-v0), EnvSpec(NameThisGameNoFrameskip-v4), EnvSpec(NameThisGame-ram-v0), EnvSpec(NameThisGame-ram-v4), EnvSpec(NameThisGame-ramDeterministic-v0), EnvSpec(NameThisGame-ramDeterministic-v4), EnvSpec(NameThisGame-ramNoFrameskip-v0), EnvSpec(NameThisGame-ramNoFrameskip-v4), EnvSpec(Phoenix-v0), EnvSpec(Phoenix-v4), EnvSpec(PhoenixDeterministic-v0), EnvSpec(PhoenixDeterministic-v4), EnvSpec(PhoenixNoFrameskip-v0), EnvSpec(PhoenixNoFrameskip-v4), EnvSpec(Phoenix-ram-v0), EnvSpec(Phoenix-ram-v4), EnvSpec(Phoenix-ramDeterministic-v0), EnvSpec(Phoenix-ramDeterministic-v4), EnvSpec(Phoenix-ramNoFrameskip-v0), EnvSpec(Phoenix-ramNoFrameskip-v4), EnvSpec(Pitfall-v0), EnvSpec(Pitfall-v4), EnvSpec(PitfallDeterministic-v0), EnvSpec(PitfallDeterministic-v4), EnvSpec(PitfallNoFrameskip-v0), EnvSpec(PitfallNoFrameskip-v4), EnvSpec(Pitfall-ram-v0), EnvSpec(Pitfall-ram-v4), EnvSpec(Pitfall-ramDeterministic-v0), EnvSpec(Pitfall-ramDeterministic-v4), EnvSpec(Pitfall-ramNoFrameskip-v0), EnvSpec(Pitfall-ramNoFrameskip-v4), EnvSpec(Pong-v0), EnvSpec(Pong-v4), EnvSpec(PongDeterministic-v0), EnvSpec(PongDeterministic-v4), EnvSpec(PongNoFrameskip-v0), EnvSpec(PongNoFrameskip-v4), EnvSpec(Pong-ram-v0), EnvSpec(Pong-ram-v4), EnvSpec(Pong-ramDeterministic-v0), EnvSpec(Pong-ramDeterministic-v4), EnvSpec(Pong-ramNoFrameskip-v0), EnvSpec(Pong-ramNoFrameskip-v4), EnvSpec(Pooyan-v0), EnvSpec(Pooyan-v4), EnvSpec(PooyanDeterministic-v0), EnvSpec(PooyanDeterministic-v4), EnvSpec(PooyanNoFrameskip-v0), EnvSpec(PooyanNoFrameskip-v4), EnvSpec(Pooyan-ram-v0), EnvSpec(Pooyan-ram-v4), EnvSpec(Pooyan-ramDeterministic-v0), EnvSpec(Pooyan-ramDeterministic-v4), EnvSpec(Pooyan-ramNoFrameskip-v0), EnvSpec(Pooyan-ramNoFrameskip-v4), EnvSpec(PrivateEye-v0), EnvSpec(PrivateEye-v4), EnvSpec(PrivateEyeDeterministic-v0), EnvSpec(PrivateEyeDeterministic-v4), EnvSpec(PrivateEyeNoFrameskip-v0), EnvSpec(PrivateEyeNoFrameskip-v4), EnvSpec(PrivateEye-ram-v0), EnvSpec(PrivateEye-ram-v4), EnvSpec(PrivateEye-ramDeterministic-v0), EnvSpec(PrivateEye-ramDeterministic-v4), EnvSpec(PrivateEye-ramNoFrameskip-v0), EnvSpec(PrivateEye-ramNoFrameskip-v4), EnvSpec(Qbert-v0), EnvSpec(Qbert-v4), EnvSpec(QbertDeterministic-v0), EnvSpec(QbertDeterministic-v4), EnvSpec(QbertNoFrameskip-v0), EnvSpec(QbertNoFrameskip-v4), EnvSpec(Qbert-ram-v0), EnvSpec(Qbert-ram-v4), EnvSpec(Qbert-ramDeterministic-v0), EnvSpec(Qbert-ramDeterministic-v4), EnvSpec(Qbert-ramNoFrameskip-v0), EnvSpec(Qbert-ramNoFrameskip-v4), EnvSpec(Riverraid-v0), EnvSpec(Riverraid-v4), EnvSpec(RiverraidDeterministic-v0), EnvSpec(RiverraidDeterministic-v4), EnvSpec(RiverraidNoFrameskip-v0), EnvSpec(RiverraidNoFrameskip-v4), EnvSpec(Riverraid-ram-v0), EnvSpec(Riverraid-ram-v4), EnvSpec(Riverraid-ramDeterministic-v0), EnvSpec(Riverraid-ramDeterministic-v4), EnvSpec(Riverraid-ramNoFrameskip-v0), EnvSpec(Riverraid-ramNoFrameskip-v4), EnvSpec(RoadRunner-v0), EnvSpec(RoadRunner-v4), EnvSpec(RoadRunnerDeterministic-v0), EnvSpec(RoadRunnerDeterministic-v4), EnvSpec(RoadRunnerNoFrameskip-v0), EnvSpec(RoadRunnerNoFrameskip-v4), EnvSpec(RoadRunner-ram-v0), EnvSpec(RoadRunner-ram-v4), EnvSpec(RoadRunner-ramDeterministic-v0), EnvSpec(RoadRunner-ramDeterministic-v4), EnvSpec(RoadRunner-ramNoFrameskip-v0), EnvSpec(RoadRunner-ramNoFrameskip-v4), EnvSpec(Robotank-v0), EnvSpec(Robotank-v4), EnvSpec(RobotankDeterministic-v0), EnvSpec(RobotankDeterministic-v4), EnvSpec(RobotankNoFrameskip-v0), EnvSpec(RobotankNoFrameskip-v4), EnvSpec(Robotank-ram-v0), EnvSpec(Robotank-ram-v4), EnvSpec(Robotank-ramDeterministic-v0), EnvSpec(Robotank-ramDeterministic-v4), EnvSpec(Robotank-ramNoFrameskip-v0), EnvSpec(Robotank-ramNoFrameskip-v4), EnvSpec(Seaquest-v0), EnvSpec(Seaquest-v4), EnvSpec(SeaquestDeterministic-v0), EnvSpec(SeaquestDeterministic-v4), EnvSpec(SeaquestNoFrameskip-v0), EnvSpec(SeaquestNoFrameskip-v4), EnvSpec(Seaquest-ram-v0), EnvSpec(Seaquest-ram-v4), EnvSpec(Seaquest-ramDeterministic-v0), EnvSpec(Seaquest-ramDeterministic-v4), EnvSpec(Seaquest-ramNoFrameskip-v0), EnvSpec(Seaquest-ramNoFrameskip-v4), EnvSpec(Skiing-v0), EnvSpec(Skiing-v4), EnvSpec(SkiingDeterministic-v0), EnvSpec(SkiingDeterministic-v4), EnvSpec(SkiingNoFrameskip-v0), EnvSpec(SkiingNoFrameskip-v4), EnvSpec(Skiing-ram-v0), EnvSpec(Skiing-ram-v4), EnvSpec(Skiing-ramDeterministic-v0), EnvSpec(Skiing-ramDeterministic-v4), EnvSpec(Skiing-ramNoFrameskip-v0), EnvSpec(Skiing-ramNoFrameskip-v4), EnvSpec(Solaris-v0), EnvSpec(Solaris-v4), EnvSpec(SolarisDeterministic-v0), EnvSpec(SolarisDeterministic-v4), EnvSpec(SolarisNoFrameskip-v0), EnvSpec(SolarisNoFrameskip-v4), EnvSpec(Solaris-ram-v0), EnvSpec(Solaris-ram-v4), EnvSpec(Solaris-ramDeterministic-v0), EnvSpec(Solaris-ramDeterministic-v4), EnvSpec(Solaris-ramNoFrameskip-v0), EnvSpec(Solaris-ramNoFrameskip-v4), EnvSpec(SpaceInvaders-v0), EnvSpec(SpaceInvaders-v4), EnvSpec(SpaceInvadersDeterministic-v0), EnvSpec(SpaceInvadersDeterministic-v4), EnvSpec(SpaceInvadersNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v4), EnvSpec(SpaceInvaders-ram-v0), EnvSpec(SpaceInvaders-ram-v4), EnvSpec(SpaceInvaders-ramDeterministic-v0), EnvSpec(SpaceInvaders-ramDeterministic-v4), EnvSpec(SpaceInvaders-ramNoFrameskip-v0), EnvSpec(SpaceInvaders-ramNoFrameskip-v4), EnvSpec(StarGunner-v0), EnvSpec(StarGunner-v4), EnvSpec(StarGunnerDeterministic-v0), EnvSpec(StarGunnerDeterministic-v4), EnvSpec(StarGunnerNoFrameskip-v0), EnvSpec(StarGunnerNoFrameskip-v4), EnvSpec(StarGunner-ram-v0), EnvSpec(StarGunner-ram-v4), EnvSpec(StarGunner-ramDeterministic-v0), EnvSpec(StarGunner-ramDeterministic-v4), EnvSpec(StarGunner-ramNoFrameskip-v0), EnvSpec(StarGunner-ramNoFrameskip-v4), EnvSpec(Tennis-v0), EnvSpec(Tennis-v4), EnvSpec(TennisDeterministic-v0), EnvSpec(TennisDeterministic-v4), EnvSpec(TennisNoFrameskip-v0), EnvSpec(TennisNoFrameskip-v4), EnvSpec(Tennis-ram-v0), EnvSpec(Tennis-ram-v4), EnvSpec(Tennis-ramDeterministic-v0), EnvSpec(Tennis-ramDeterministic-v4), EnvSpec(Tennis-ramNoFrameskip-v0), EnvSpec(Tennis-ramNoFrameskip-v4), EnvSpec(TimePilot-v0), EnvSpec(TimePilot-v4), EnvSpec(TimePilotDeterministic-v0), EnvSpec(TimePilotDeterministic-v4), EnvSpec(TimePilotNoFrameskip-v0), EnvSpec(TimePilotNoFrameskip-v4), EnvSpec(TimePilot-ram-v0), EnvSpec(TimePilot-ram-v4), EnvSpec(TimePilot-ramDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v4), EnvSpec(TimePilot-ramNoFrameskip-v0), EnvSpec(TimePilot-ramNoFrameskip-v4), EnvSpec(Tutankham-v0), EnvSpec(Tutankham-v4), EnvSpec(TutankhamDeterministic-v0), EnvSpec(TutankhamDeterministic-v4), EnvSpec(TutankhamNoFrameskip-v0), EnvSpec(TutankhamNoFrameskip-v4), EnvSpec(Tutankham-ram-v0), EnvSpec(Tutankham-ram-v4), EnvSpec(Tutankham-ramDeterministic-v0), EnvSpec(Tutankham-ramDeterministic-v4), EnvSpec(Tutankham-ramNoFrameskip-v0), EnvSpec(Tutankham-ramNoFrameskip-v4), EnvSpec(UpNDown-v0), EnvSpec(UpNDown-v4), EnvSpec(UpNDownDeterministic-v0), EnvSpec(UpNDownDeterministic-v4), EnvSpec(UpNDownNoFrameskip-v0), EnvSpec(UpNDownNoFrameskip-v4), EnvSpec(UpNDown-ram-v0), EnvSpec(UpNDown-ram-v4), EnvSpec(UpNDown-ramDeterministic-v0), EnvSpec(UpNDown-ramDeterministic-v4), EnvSpec(UpNDown-ramNoFrameskip-v0), EnvSpec(UpNDown-ramNoFrameskip-v4), EnvSpec(Venture-v0), EnvSpec(Venture-v4), EnvSpec(VentureDeterministic-v0), EnvSpec(VentureDeterministic-v4), EnvSpec(VentureNoFrameskip-v0), EnvSpec(VentureNoFrameskip-v4), EnvSpec(Venture-ram-v0), EnvSpec(Venture-ram-v4), EnvSpec(Venture-ramDeterministic-v0), EnvSpec(Venture-ramDeterministic-v4), EnvSpec(Venture-ramNoFrameskip-v0), EnvSpec(Venture-ramNoFrameskip-v4), EnvSpec(VideoPinball-v0), EnvSpec(VideoPinball-v4), EnvSpec(VideoPinballDeterministic-v0), EnvSpec(VideoPinballDeterministic-v4), EnvSpec(VideoPinballNoFrameskip-v0), EnvSpec(VideoPinballNoFrameskip-v4), EnvSpec(VideoPinball-ram-v0), EnvSpec(VideoPinball-ram-v4), EnvSpec(VideoPinball-ramDeterministic-v0), EnvSpec(VideoPinball-ramDeterministic-v4), EnvSpec(VideoPinball-ramNoFrameskip-v0), EnvSpec(VideoPinball-ramNoFrameskip-v4), EnvSpec(WizardOfWor-v0), EnvSpec(WizardOfWor-v4), EnvSpec(WizardOfWorDeterministic-v0), EnvSpec(WizardOfWorDeterministic-v4), EnvSpec(WizardOfWorNoFrameskip-v0), EnvSpec(WizardOfWorNoFrameskip-v4), EnvSpec(WizardOfWor-ram-v0), EnvSpec(WizardOfWor-ram-v4), EnvSpec(WizardOfWor-ramDeterministic-v0), EnvSpec(WizardOfWor-ramDeterministic-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v0), EnvSpec(WizardOfWor-ramNoFrameskip-v4), EnvSpec(YarsRevenge-v0), EnvSpec(YarsRevenge-v4), EnvSpec(YarsRevengeDeterministic-v0), EnvSpec(YarsRevengeDeterministic-v4), EnvSpec(YarsRevengeNoFrameskip-v0), EnvSpec(YarsRevengeNoFrameskip-v4), EnvSpec(YarsRevenge-ram-v0), EnvSpec(YarsRevenge-ram-v4), EnvSpec(YarsRevenge-ramDeterministic-v0), EnvSpec(YarsRevenge-ramDeterministic-v4), EnvSpec(YarsRevenge-ramNoFrameskip-v0), EnvSpec(YarsRevenge-ramNoFrameskip-v4), EnvSpec(Zaxxon-v0), EnvSpec(Zaxxon-v4), EnvSpec(ZaxxonDeterministic-v0), EnvSpec(ZaxxonDeterministic-v4), EnvSpec(ZaxxonNoFrameskip-v0), EnvSpec(ZaxxonNoFrameskip-v4), EnvSpec(Zaxxon-ram-v0), EnvSpec(Zaxxon-ram-v4), EnvSpec(Zaxxon-ramDeterministic-v0), EnvSpec(Zaxxon-ramDeterministic-v4), EnvSpec(Zaxxon-ramNoFrameskip-v0), EnvSpec(Zaxxon-ramNoFrameskip-v4), EnvSpec(CubeCrash-v0), EnvSpec(CubeCrashSparse-v0), EnvSpec(CubeCrashScreenBecomesBlack-v0), EnvSpec(MemorizeDigits-v0)])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gym.envs.registry.all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Cart-Pole is a very simple environment composed of a cart that can move left or right, and pole placed vertically on top of it. The agent must move the cart left or right to keep the pole upright."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make('CartPole-v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize the environment by calling is `reset()` method. This returns an observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "obs = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observations vary depending on the environment. In this case it is a 1D NumPy array composed of 4 floats: they represent the cart's horizontal position, its velocity, the angle of the pole (0 = vertical), and the angular velocity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.01258566, -0.00156614,  0.04207708, -0.00180545])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An environment can be visualized by calling its `render()` method, and you can pick the rendering mode (the rendering options depend on the environment)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: some environments (including the Cart-Pole) require access to your display, which opens up a separate window, even if you specify `mode=\"rgb_array\"`. In general you can safely ignore that window. However, if Jupyter is running on a headless server (ie. without a screen) it will raise an exception. One way to avoid this is to install a fake X server like [Xvfb](http://en.wikipedia.org/wiki/Xvfb). On Debian or Ubuntu:\n",
    "\n",
    "```bash\n",
    "$ apt update\n",
    "$ apt install -y xvfb\n",
    "```\n",
    "\n",
    "You can then start Jupyter using the `xvfb-run` command:\n",
    "\n",
    "```bash\n",
    "$ xvfb-run -s \"-screen 0 1400x900x24\" jupyter notebook\n",
    "```\n",
    "\n",
    "Alternatively, you can install the [pyvirtualdisplay](https://github.com/ponty/pyvirtualdisplay) Python library which wraps Xvfb:\n",
    "\n",
    "```bash\n",
    "python3 -m pip install -U pyvirtualdisplay\n",
    "```\n",
    "\n",
    "And run the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import pyvirtualdisplay\n",
    "    display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()\n",
    "except ImportError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we will set `mode=\"rgb_array\"` to get an image of the environment as a NumPy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400, 600, 3)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img = env.render(mode=\"rgb_array\")\n",
    "img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_environment(env, figsize=(5,4)):\n",
    "    plt.figure(figsize=figsize)\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    plt.imshow(img)\n",
    "    plt.axis(\"off\")\n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASUAAADICAYAAACuyvefAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAF4klEQVR4nO3dzY8bdx3H8e/Y3sfulhSiKlSoIr2UWy9BKDlHAokD/wF3cuN/yYkLh4gr/wRCWiEFISHoAVVERXS3PKQJWbRee8fDgQjqbHcDnk39sfN63Wb8oK+0o7e8Y89vmq7rCiDFYNkDAHyeKAFRRAmIIkpAFFECoogSEGX0ksf9XgB4FZqLHvBJCYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBElIIooAVFECYgiSkAUUQKiiBIQRZSAKKIERBktewBWX9d1Vd3sc3uaqqappmmWNhOrS5TobfzkqD7+xc+qex6m4eZ23fjgu7X/zvtLnoxVJEr0NpuO6/jTj+Y+Le2/8y1RYiHOKdFbOx2f2zfc2F7CJKwDUaK38WdH8+eUmqa23/r68gZipYkSV6B7YdsJbhYnSkAUUQKiiBK9dF33798pwRURJXo7+fuf5raHG1u1+ca15QzDyhMlepudTee2m+GoBn4SwIJECYgiSkAUUQKiiBL9dF21k5O5Xc1gaIUAFiZK9DJrp3X67K9z+7b2r9dwc2dJE7HqRImr1wzKpSYsSpSAKKIERBElIIoo0cvZ+FmdnRzP7du+dqPKt28sSJToZXY2rVk7f5nJaGvXTwJYmCgBUUQJiCJKQBRRopeundaLa3QPRpvLGYa1IEr0Mn5yVF17Nrdv56vfWNI0rANRopcvXArXN2/0IEpAFFECoogSPbmTCVdLlOjl5PGf57ab4UZt7H5lSdOwDkSJXtrJeG67GQxrtL23pGlYB6IERBElIIooAVFEiYV1XVez6encvqYZWLaEXkSJhXWztsZPDuf2be695UQ3vYgSV6sZPL+bCSzG0QNEESUgiigBUUSJhbWTk5qePJvbt/Xm9WoGDisW5+hhYV07rdl0/jKT0fZeNU5004OjB4giSkAUUQKiiBIL62btuTW6m8FoSdOwLkSJhY2f/uXctW+7199d0jSsC1Ficd2sXlwO1zdv9OUIAqKIEhBFlIAozRfe4fS/3D/nNdS2bT148KCOjo4ufd77107r3cEn/9k+a2f1y0/3a7L99qWvu3v3bt26detKZmVlXbgSoO9vOadt27p//349fPjw0uf96Affrh9+7zv12eTt2hyc1nYd1f2f/LT+ePjk0tft7OyIEhcSJRZ22u7WwePv19Pp12rYtPXNrYPqup8veyxWnHNKLOyT8Xv1dHq9qgbVdhv10T8/qMlsZ9ljseJEiYV1L5wWmHXDarvhkqZhXYgSC7v55se1P3pcVV0N6qzee+M3tT08XvZYrLhLzynNZrMvaw6C/K9/91/99nf14aMf1+PJjdocjGu3Duvwb/946eu6rnNsveYGlywEeGmUDg4OrnwY8k2n0zo+fvknnl//4bCqDqvq9//X+z969Mix9Zq7c+fOhY9dGqXLXsj6mkwmtbf36u7ddvPmTccWF3JOCYgiSkAUUQKiiBIQRZSAKK5945zhcFj37t2rw8PDV/L+t2/ffiXvy3qwdAmwDBcuXeLfNyCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEEWUgCiiBEQRJSCKKAFRRAmIIkpAFFECoogSEGX0ksebL2UKgOd8UgKiiBIQRZSAKKIERBElIIooAVH+Bc6iAHz8aqUYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_environment(env)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how to interact with an environment. Your agent will need to select an action from an \"action space\" (the set of possible actions). Let's see what this environment's action space looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(2)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yep, just two possible actions: accelerate towards the left or towards the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the pole is leaning toward the right (`obs[2] > 0`), let's accelerate the cart toward the right:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.01261699,  0.19292789,  0.04204097, -0.28092127])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "action = 1  # accelerate right\n",
    "obs, reward, done, info = env.step(action)\n",
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the cart is now moving toward the right (`obs[1] > 0`). The pole is still tilted toward the right (`obs[2] > 0`), but its angular velocity is now negative (`obs[3] < 0`), so it will likely be tilted toward the left after the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure cart_pole_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAADvCAYAAADM8A71AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAGHklEQVR4nO3dzWpcBRjH4fdMkklMG2s1+BFF0aIWP6s7wYXQ4kLoBbjpRYilvQGhdyBFvAbrSqwrN5aCGyuIINUK1o9qqmlsYhsnx40Uy8Q2mbbz76HPswrvDHPexeHHMDlzpmnbtgAYv156AYA7lQADhAgwQIgAA4QIMEDI5HUed4kEwI1rNhp6BwwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAymV4ARtW27ZW/m6YJbgKjEWA668fPP6wLP3xVD730RvWmZqqqqun1avsDu6rpTYS3g+sTYDrr8vL5Wvn1+zp9/N0rs97UTD3/5js1ObM9uBlsjs+AAUIEGCBEgAFCBBggRIABQgQYIESAAUIEGCBEgAFCBBggRIABQgSYThpcXq211aWh+fTcfW7EQ2cIMJ3019IvtXz266H5fU+9UhP9uwIbwdYJMECIAAOECDBAiAADhAgwQIgAA4QIMECIAAOECDBAiAADhAgwQIgA00mXls4NzZqJqZratjOwDYxGgOmkX7/6dGg2Nbujdj7+UmAbGI0AA4QIMECIAAOECDBAiAADhAgwQIgAA4QIMECIAAOECDBAiAADhAgwQIgA0zl//ny6Vs//ODSf3/1qVeOUpjucrXTO2uqFGlxeGZrP3PNgNU0T2AhGI8AAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAkyntG1bf69eGJr3Jvs10Z8JbASjE2C6pV2vX059MjSenX+05hZ2BxaC0QkwndO27YZzd0KjawQYIESAAUIEGCBEgAFCBBggRIABQgQYIGQyvQCsrKzUYDDY1HPb9UG16+tD88FgUMvLy5t6jX6/X9PT01vaEW6F5v8uav/XNR+Em2H//v114sSJTT13otfU+2+9Xg/du/2q+Zff/VZvv/fppl7j0KFDdfDgwS3vCTdgw28JeQdM3NLSUi0uLm7quRO9plbWpuvcpUeqqqqptu7t/1Rrf69t+jVWV1dH3hVuJgGmU9pq6os/Xqv++rNXJk9sO1Xr7c/RvWAU/glHp7z89GM1OfPgfyZNffvnC/XByd9jO8GoBJhOuf/+J2t6+urPf9uqOnX6XGYhuAECTKfs7J+rqd6lq2Z3TVwcmkEX+AyYTjl/4WKtLR6vsxefqYfn5+qe7f167u7PatvE8E3a4XYnwHTKRye/qY9OHqmqqhd3PVAL83P1cVWdXdzcNcBwO7nmdcBHjhxxHTC33NGjR+vMmTNjO96+fftq7969YzseHD58eOvXAR84cODWbAP/cezYsbEGeM+ePc5tbgvXDPDCwsK49uAO1u/3x3q8ubk55za3BVdBAIQIMECIAAOECDBAiAADhPgiBnE7duyo+fn5sR1vdnZ2bMeCa3FDduK28osYN4NfxCBgwy9iCDDArbdhgH0GDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIQIMECLAACECDBAiwAAhAgwQIsAAIZPXebwZyxYAdyDvgAFCBBggRIABQgQYIESAAUIEGCDkH6bUxi6unNOlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_environment(env)\n",
    "save_fig(\"cart_pole_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like it's doing what we're telling it to do!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment also tells the agent how much reward it got during the last step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the game is over, the environment returns `done=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, `info` is an environment-specific dictionary that can provide some extra information that you may find useful for debugging or for training. For example, in some games it may indicate how many lives the agent has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequence of steps between the moment the environment is reset until it is done is called an \"episode\". At the end of an episode (i.e., when `step()` returns `done=True`), you should reset the environment before you continue to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "if done:\n",
    "    obs = env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now how can we make the poll remain upright? We will need to define a _policy_ for that. This is the strategy that the agent will use to select an action at each step. It can use all the past actions and observations to decide what to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A simple hard-coded policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's hard code a simple strategy: if the pole is tilting to the left, then push the cart to the left, and _vice versa_. Let's see if that works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "\n",
    "def basic_policy(obs):\n",
    "    angle = obs[2]\n",
    "    return 0 if angle < 0 else 1\n",
    "\n",
    "totals = []\n",
    "for episode in range(500):\n",
    "    episode_rewards = 0\n",
    "    obs = env.reset()\n",
    "    for step in range(200):\n",
    "        action = basic_policy(obs)\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        episode_rewards += reward\n",
    "        if done:\n",
    "            break\n",
    "    totals.append(episode_rewards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41.718, 8.858356280936096, 24.0, 68.0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(totals), np.std(totals), np.min(totals), np.max(totals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, as expected, this strategy is a bit too basic: the best it did was to keep the poll up for only 68 steps. This environment is considered solved when the agent keeps the poll up for 200 steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize one episode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "\n",
    "frames = []\n",
    "\n",
    "obs = env.reset()\n",
    "for step in range(200):\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    action = basic_policy(obs)\n",
    "\n",
    "    obs, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now show the animation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_scene(num, frames, patch):\n",
    "    patch.set_data(frames[num])\n",
    "    return patch,\n",
    "\n",
    "def plot_animation(frames, repeat=False, interval=40):\n",
    "    fig = plt.figure()\n",
    "    patch = plt.imshow(frames[0])\n",
    "    plt.axis('off')\n",
    "    anim = animation.FuncAnimation(\n",
    "        fig, update_scene, fargs=(frames, patch),\n",
    "        frames=len(frames), repeat=repeat, interval=interval)\n",
    "    plt.close()\n",
    "    return anim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly the system is unstable and after just a few wobbles, the pole ends up too tilted: game over. We will need to be smarter than that!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Policies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a neural network that will take observations as inputs, and output the probabilities of actions to take for each observation. To choose an action, the network will estimate a probability for each action, then we will select an action randomly according to the estimated probabilities. In the case of the Cart-Pole environment, there are just two possible actions (left or right), so we only need one output neuron: it will output the probability `p` of the action 0 (left), and of course the probability of action 1 (right) will be `1 - p`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "n_inputs = 4 # == env.observation_space.shape[0]\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(5, activation=\"elu\", input_shape=[n_inputs]),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\"),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular environment, the past actions and observations can safely be ignored, since each observation contains the environment's full state. If there were some hidden state then you may need to consider past actions and observations in order to try to infer the hidden state of the environment. For example, if the environment only revealed the position of the cart but not its velocity, you would have to consider not only the current observation but also the previous observation in order to estimate the current velocity. Another example is if the observations are noisy: you may want to use the past few observations to estimate the most likely current state. Our problem is thus as simple as can be: the current observation is noise-free and contains the environment's full state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may wonder why we plan to pick a random action based on the probability given by the policy network, rather than just picking the action with the highest probability. This approach lets the agent find the right balance between _exploring_ new actions and _exploiting_ the actions that are known to work well. Here's an analogy: suppose you go to a restaurant for the first time, and all the dishes look equally appealing so you randomly pick one. If it turns out to be good, you can increase the probability to order it next time, but you shouldn't increase that probability to 100%, or else you will never try out the other dishes, some of which may be even better than the one you tried."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a small function that will run the model to play one episode, and return the frames so we can display an animation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_policy_net(model, n_max_steps=200, seed=42):\n",
    "    frames = []\n",
    "    env = gym.make(\"CartPole-v1\")\n",
    "    env.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    obs = env.reset()\n",
    "    for step in range(n_max_steps):\n",
    "        frames.append(env.render(mode=\"rgb_array\"))\n",
    "        left_proba = model.predict(obs.reshape(1, -1))\n",
    "        action = int(np.random.rand() > left_proba)\n",
    "        obs, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "    env.close()\n",
    "    return frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at how well this randomly initialized policy network performs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yeah... pretty bad. The neural network will have to learn to do better. First let's see if it is capable of learning the basic policy we used earlier: go left if the pole is tilting left, and go right if it is tilting right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make the same net play in 50 different environments in parallel (this will give us a diverse training batch at each step), and train for 5000 iterations. We also reset environments when they are done. We train the model using a custom training loop so we can easily use the predictions at each training step to advance the environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 4999, Loss: 0.094"
     ]
    }
   ],
   "source": [
    "n_environments = 50\n",
    "n_iterations = 5000\n",
    "\n",
    "envs = [gym.make(\"CartPole-v1\") for _ in range(n_environments)]\n",
    "for index, env in enumerate(envs):\n",
    "    env.seed(index)\n",
    "np.random.seed(42)\n",
    "observations = [env.reset() for env in envs]\n",
    "optimizer = keras.optimizers.RMSprop()\n",
    "loss_fn = keras.losses.binary_crossentropy\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    # if angle < 0, we want proba(left) = 1., or else proba(left) = 0.\n",
    "    target_probas = np.array([([1.] if obs[2] < 0 else [0.])\n",
    "                              for obs in observations])\n",
    "    with tf.GradientTape() as tape:\n",
    "        left_probas = model(np.array(observations))\n",
    "        loss = tf.reduce_mean(loss_fn(target_probas, left_probas))\n",
    "    print(\"\\rIteration: {}, Loss: {:.3f}\".format(iteration, loss.numpy()), end=\"\")\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "    actions = (np.random.rand(n_environments, 1) > left_probas.numpy()).astype(np.int32)\n",
    "    for env_index, env in enumerate(envs):\n",
    "        obs, reward, done, info = env.step(actions[env_index][0])\n",
    "        observations[env_index] = obs if not done else env.reset()\n",
    "\n",
    "for env in envs:\n",
    "    env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like it learned the policy correctly. Now let's see if it can learn a better policy on its own. One that does not wobble as much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train this neural network we will need to define the target probabilities `y`. If an action is good we should increase its probability, and conversely if it is bad we should reduce it. But how do we know whether an action is good or bad? The problem is that most actions have delayed effects, so when you win or lose points in an episode, it is not clear which actions contributed to this result: was it just the last action? Or the last 10? Or just one action 50 steps earlier? This is called the _credit assignment problem_.\n",
    "\n",
    "The _Policy Gradients_ algorithm tackles this problem by first playing multiple episodes, then making the actions in good episodes slightly more likely, while actions in bad episodes are made slightly less likely. First we play, then we go back and think about what we did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating a function to play a single step using the model. We will also pretend for now that whatever action it takes is the right one, so we can compute the loss and its gradients (we will just save these gradients for now, and modify them later depending on how good or bad the action turned out to be):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_one_step(env, obs, model, loss_fn):\n",
    "    with tf.GradientTape() as tape:\n",
    "        left_proba = model(obs[np.newaxis])\n",
    "        action = (tf.random.uniform([1, 1]) > left_proba)\n",
    "        y_target = tf.constant([[1.]]) - tf.cast(action, tf.float32)\n",
    "        loss = tf.reduce_mean(loss_fn(y_target, left_proba))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    obs, reward, done, info = env.step(int(action[0, 0].numpy()))\n",
    "    return obs, reward, done, grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `left_proba` is high, then `action` will most likely be `False` (since a random number uniformally sampled between 0 and 1 will probably not be greater than `left_proba`). And `False` means 0 when you cast it to a number, so `y_target` would be equal to 1 - 0 = 1. In other words, we set the target to 1, meaning we pretend that the probability of going left should have been 100% (so we took the right action)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create another function that will rely on the `play_one_step()` function to play multiple episodes, returning all the rewards and gradients, for each episode and each step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_multiple_episodes(env, n_episodes, n_max_steps, model, loss_fn):\n",
    "    all_rewards = []\n",
    "    all_grads = []\n",
    "    for episode in range(n_episodes):\n",
    "        current_rewards = []\n",
    "        current_grads = []\n",
    "        obs = env.reset()\n",
    "        for step in range(n_max_steps):\n",
    "            obs, reward, done, grads = play_one_step(env, obs, model, loss_fn)\n",
    "            current_rewards.append(reward)\n",
    "            current_grads.append(grads)\n",
    "            if done:\n",
    "                break\n",
    "        all_rewards.append(current_rewards)\n",
    "        all_grads.append(current_grads)\n",
    "    return all_rewards, all_grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Policy Gradients algorithm uses the model to play the episode several times (e.g., 10 times), then it goes back and looks at all the rewards, discounts them and normalizes them. So let's create couple functions for that: the first will compute discounted rewards; the second will normalize the discounted rewards across many episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_rewards(rewards, discount_rate):\n",
    "    discounted = np.array(rewards)\n",
    "    for step in range(len(rewards) - 2, -1, -1):\n",
    "        discounted[step] += discounted[step + 1] * discount_rate\n",
    "    return discounted\n",
    "\n",
    "def discount_and_normalize_rewards(all_rewards, discount_rate):\n",
    "    all_discounted_rewards = [discount_rewards(rewards, discount_rate)\n",
    "                              for rewards in all_rewards]\n",
    "    flat_rewards = np.concatenate(all_discounted_rewards)\n",
    "    reward_mean = flat_rewards.mean()\n",
    "    reward_std = flat_rewards.std()\n",
    "    return [(discounted_rewards - reward_mean) / reward_std\n",
    "            for discounted_rewards in all_discounted_rewards]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say there were 3 actions, and after each action there was a reward: first 10, then 0, then -50. If we use a discount factor of 80%, then the 3rd action will get -50 (full credit for the last reward), but the 2nd action will only get -40 (80% credit for the last reward), and the 1st action will get 80% of -40 (-32) plus full credit for the first reward (+10), which leads to a discounted reward of -22:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-22, -40, -50])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discount_rewards([10, 0, -50], discount_rate=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To normalize all discounted rewards across all episodes, we compute the mean and standard deviation of all the discounted rewards, and we subtract the mean from each discounted reward, and divide by the standard deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([-0.28435071, -0.86597718, -1.18910299]),\n",
       " array([1.26665318, 1.0727777 ])]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discount_and_normalize_rewards([[10, 0, -50], [10, 20]], discount_rate=0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_iterations = 150\n",
    "n_episodes_per_update = 10\n",
    "n_max_steps = 200\n",
    "discount_rate = 0.95"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(lr=0.01)\n",
    "loss_fn = keras.losses.binary_crossentropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(5, activation=\"elu\", input_shape=[4]),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\"),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 149, mean rewards: 199.6"
     ]
    }
   ],
   "source": [
    "env = gym.make(\"CartPole-v1\")\n",
    "env.seed(42);\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    all_rewards, all_grads = play_multiple_episodes(\n",
    "        env, n_episodes_per_update, n_max_steps, model, loss_fn)\n",
    "    total_rewards = sum(map(sum, all_rewards))                     # Not shown in the book\n",
    "    print(\"\\rIteration: {}, mean rewards: {:.1f}\".format(          # Not shown\n",
    "        iteration, total_rewards / n_episodes_per_update), end=\"\") # Not shown\n",
    "    all_final_rewards = discount_and_normalize_rewards(all_rewards,\n",
    "                                                       discount_rate)\n",
    "    all_mean_grads = []\n",
    "    for var_index in range(len(model.trainable_variables)):\n",
    "        mean_grads = tf.reduce_mean(\n",
    "            [final_reward * all_grads[episode_index][step][var_index]\n",
    "             for episode_index, final_rewards in enumerate(all_final_rewards)\n",
    "                 for step, final_reward in enumerate(final_rewards)], axis=0)\n",
    "        all_mean_grads.append(mean_grads)\n",
    "    optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = render_policy_net(model)\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Markov Chains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "States: 0 0 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n",
      "States: 0 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n",
      "States: 0 1 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 ...\n",
      "States: 0 0 3 \n",
      "States: 0 0 0 1 2 1 2 1 3 \n",
      "States: 0 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 3 \n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "transition_probabilities = [ # shape=[s, s']\n",
    "        [0.7, 0.2, 0.0, 0.1],  # from s0 to s0, s1, s2, s3\n",
    "        [0.0, 0.0, 0.9, 0.1],  # from s1 to ...\n",
    "        [0.0, 1.0, 0.0, 0.0],  # from s2 to ...\n",
    "        [0.0, 0.0, 0.0, 1.0]]  # from s3 to ...\n",
    "\n",
    "n_max_steps = 50\n",
    "\n",
    "def print_sequence():\n",
    "    current_state = 0\n",
    "    print(\"States:\", end=\" \")\n",
    "    for step in range(n_max_steps):\n",
    "        print(current_state, end=\" \")\n",
    "        if current_state == 3:\n",
    "            break\n",
    "        current_state = np.random.choice(range(4), p=transition_probabilities[current_state])\n",
    "    else:\n",
    "        print(\"...\", end=\"\")\n",
    "    print()\n",
    "\n",
    "for _ in range(10):\n",
    "    print_sequence()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Markov Decision Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define some transition probabilities, rewards and possible actions. For example, in state s0, if action a0 is chosen then with proba 0.7 we will go to state s0 with reward +10, with probability 0.3 we will go to state s1 with no reward, and with never go to state s2 (so the transition probabilities are `[0.7, 0.3, 0.0]`, and the rewards are `[+10, 0, 0]`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "transition_probabilities = [ # shape=[s, a, s']\n",
    "        [[0.7, 0.3, 0.0], [1.0, 0.0, 0.0], [0.8, 0.2, 0.0]],\n",
    "        [[0.0, 1.0, 0.0], None, [0.0, 0.0, 1.0]],\n",
    "        [None, [0.8, 0.1, 0.1], None]]\n",
    "rewards = [ # shape=[s, a, s']\n",
    "        [[+10, 0, 0], [0, 0, 0], [0, 0, 0]],\n",
    "        [[0, 0, 0], [0, 0, 0], [0, 0, -50]],\n",
    "        [[0, 0, 0], [+40, 0, 0], [0, 0, 0]]]\n",
    "possible_actions = [[0, 1, 2], [0, 2], [1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-Value Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q_values = np.full((3, 3), -np.inf) # -np.inf for impossible actions\n",
    "for state, actions in enumerate(possible_actions):\n",
    "    Q_values[state, actions] = 0.0  # for all possible actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 0.90  # the discount factor\n",
    "\n",
    "history1 = [] # Not shown in the book (for the figure below)\n",
    "for iteration in range(50):\n",
    "    Q_prev = Q_values.copy()\n",
    "    history1.append(Q_prev) # Not shown\n",
    "    for s in range(3):\n",
    "        for a in possible_actions[s]:\n",
    "            Q_values[s, a] = np.sum([\n",
    "                    transition_probabilities[s][a][sp]\n",
    "                    * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp]))\n",
    "                for sp in range(3)])\n",
    "\n",
    "history1 = np.array(history1) # Not shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18.91891892, 17.02702702, 13.62162162],\n",
       "       [ 0.        ,        -inf, -4.87971488],\n",
       "       [       -inf, 50.13365013,        -inf]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(Q_values, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The optimal policy for this MDP, when using a discount factor of 0.90, is to choose action a0 when in state s0, and choose action a0 when in state s1, and finally choose action a1 (the only possible action) when in state s2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try again with a discount factor of 0.95:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q_values = np.full((3, 3), -np.inf) # -np.inf for impossible actions\n",
    "for state, actions in enumerate(possible_actions):\n",
    "    Q_values[state, actions] = 0.0  # for all possible actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 0.95  # the discount factor\n",
    "\n",
    "for iteration in range(50):\n",
    "    Q_prev = Q_values.copy()\n",
    "    for s in range(3):\n",
    "        for a in possible_actions[s]:\n",
    "            Q_values[s, a] = np.sum([\n",
    "                    transition_probabilities[s][a][sp]\n",
    "                    * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp]))\n",
    "                for sp in range(3)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[21.73304188, 20.63807938, 16.70138772],\n",
       "       [ 0.95462106,        -inf,  1.01361207],\n",
       "       [       -inf, 53.70728682,        -inf]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 1])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(Q_values, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the policy has changed! In state s1, we now prefer to go through the fire (choose action a2). This is because the discount factor is larger so the agent values the future more, and it is therefore ready to pay an immediate penalty in order to get more future rewards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q-Learning works by watching an agent play (e.g., randomly) and gradually improving its estimates of the Q-Values. Once it has accurate Q-Value estimates (or close enough), then the optimal policy consists in choosing the action that has the highest Q-Value (i.e., the greedy policy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need to simulate an agent moving around in the environment, so let's define a function to perform some action and get the new state and a reward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def step(state, action):\n",
    "    probas = transition_probabilities[state][action]\n",
    "    next_state = np.random.choice([0, 1, 2], p=probas)\n",
    "    reward = rewards[state][action][next_state]\n",
    "    return next_state, reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need an exploration policy, which can be any policy, as long as it visits every possible state many times. We will just use a random policy, since the state space is very small:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exploration_policy(state):\n",
    "    return np.random.choice(possible_actions[state])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's initialize the Q-Values like earlier, and run the Q-Learning algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "Q_values = np.full((3, 3), -np.inf)\n",
    "for state, actions in enumerate(possible_actions):\n",
    "    Q_values[state][actions] = 0\n",
    "\n",
    "alpha0 = 0.05 # initial learning rate\n",
    "decay = 0.005 # learning rate decay\n",
    "gamma = 0.90 # discount factor\n",
    "state = 0 # initial state\n",
    "history2 = [] # Not shown in the book\n",
    "\n",
    "for iteration in range(10000):\n",
    "    history2.append(Q_values.copy()) # Not shown\n",
    "    action = exploration_policy(state)\n",
    "    next_state, reward = step(state, action)\n",
    "    next_value = np.max(Q_values[next_state]) # greedy policy at the next step\n",
    "    alpha = alpha0 / (1 + iteration * decay)\n",
    "    Q_values[state, action] *= 1 - alpha\n",
    "    Q_values[state, action] += alpha * (reward + gamma * next_value)\n",
    "    state = next_state\n",
    "\n",
    "history2 = np.array(history2) # Not shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18.77621289, 17.2238872 , 13.74543343],\n",
       "       [ 0.        ,        -inf, -8.00485647],\n",
       "       [       -inf, 49.40208921,        -inf]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(Q_values, axis=1) # optimal action for each state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure q_value_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAEYCAYAAABBfQDEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XecXFX5x/HPk82md5JAEpIAMSEISvwRaVKigAqioKGJKIJ0UIoogpRIEWlSBJRqkCJNECMo0gIoLaEkkACBBEgnPdmWtnl+f5w77N2Z2TK7s3tnZ7/v1+u+ZuaeuXees7Nz55lzzznX3B0REREREQk6JB2AiIiIiEghUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIjBJkEREREZEYJcgiIiIiIjFKkEVERKRgmNlWZuZmNjbpWKT9UoIsRcPMLjWzt5KOIylmtm/0pdIn6VhEpHWY2RAzu9XM5pvZejNbYGa3mdmWDWw32cxubK04czQPGAS02+O5JE8JsuRFMw7S083s9jrKDogSvlEtE3XTmNlxZraqrsetFMN8MzsjbfULhC+V1a0Zi4gkw8y2BqYCOwBHA58DjgK2B6aY2VaJBZeFmXVqzPPcvdrdF7v7xpaOSaQuSpCl2Zp5kL4DONzMumcpOxZ40d1n5TXgAmVBaVO3d/f10ZeKLo8p0j7cBGwC9nX3Z9x9rrs/B+wbrb+pqTuOGj3uN7OV0fK4mY2MlY8ws8fMbLGZVZjZG2Z2YNo+PjazCWZ2Z9SIcG+s+8R4M3vKzCrNbKaZ7RfbrlYXCzMbFz3ex8xejbaZamb/l/Z6x5rZ3Kh8kpmdYmY6HkqTKEGWfGjOQfpuoBQ4LL7SzAYA3wFujx6XRgfZj8ysysxmmdnZZmZ17djM7jGzv6ety+iGEbUAv2tma83sfTP7WX37Tdt2X+A2oHd0AHczOz8q62xmV0Wt6RVm9lr0/M+2jZ7/TTObCqwD9jGzkWb2DzP71MzKzex1M9s/tt1/gSHAtdH2G9P21yf23EPM7B0zWxd9cZwbr1vUEn2umd1uZmvMbJ6ZndWYuotIcsysH/BN4CZ3r4yXRY9vBvY3s75N2Hc34DlgLbA3sBuwCHg6KgPoAfwL2A/YEfgb8IiZjU7b3VnAe8BY4LzY+suAG6JtpwD3m1mPBkK7HPgV8H/AckLCbVHMuxG+L24CxgD/AH6TU8VFYpQgS7M09yDt7iuAvxNai+N+CFQBD0ePS4C5hER6O+DCaPlRM+M/GbgYOD/a7y+j+yc2chcvAD8H1hC6NwwCro3K/gJ8BTgC+CJwL/C4me2Qto8rgHOB0YSW+J7A44QfGF8CHgMei7XefIfwZXVh9HpD6qjbzsCD0fIF4NdR3U5Ke+rZwBuEL53fA9dE24pI4RoJGPBuHeUzo/KRdZTX54ho22Pcfbq7v0c4JvYADgRw92nu/id3f9vdP3T3ywjHkUPS9vW8u18ZPeeD2Ppr3X1StO48oB8hsa3PBe7+XBTPxYRjZur49zPgP+5+hbvPcvfbgEebUHcRQAmyNF8+DtK3A3tY7b7GxwL3pZJud1/r7hPcfYq7f+zu9wO3At9vZvznAz9397+5+0fu/hhwJXBKYzZ29/WE5Nij7g2L3b0iqsuhwCHu/qK7z3b364GngBPSdnOhuz/l7nPcfZm7v+Hut0RfPB+4+8XAdGB89JorCC3zZdHrfVpHeD8Hnnb3i6MvjLsJyfs5ac97wt1vjr7ArgU+Br7WmPqLSOLq6kKQOlO0X3QmKrXs2Yh97gRsDZSltiOMbegLjAAws+5mdmXUPWJl9JyxwLC0fU2t4zWmx+4vjG4HNhBXfduMBl5Le/6rDexPpE4dkw5AikZDB+n1ZvYD4JZY2f7u/iLwDPARISn+lZntQui/fHStHZmdChwDDAe6ErpmzG5qwGY2CBgM3GFmt8WKOgLVTd1vZCdC3Wel9dboDPwn7bm1vkCi04wTgG8RWog7Al3IPPg3ZDvCac+4/wK/NrNusRb/6WnPWUjDX1QikqwPCMfd7Qln4dJtB2wEbgQeiK1f0Ih9dyDMIHFElrIV0e3VhLOHZ0exVBLOmqUPxKuo4zU2pO64u0fHyYYa7TbE7qe+c1LbGHV/D4nkTAmyNFdjD9IfEZLZ+C/6BfDZwfHPwMlm9mvgJ8A0d3899cQoub6a0Cr6CqHV9mfAAfXEtomaBD0lPggudWA9nsyWhuYeaDsQkuydyEy2K9Mep3+BXEtowf0F8GH0/HvJ/OJpSF1fGOnrNmQp19klkQLm7ivM7N/AKWZ2bbyLW9RP+FTgUXdfTe4z27xBODu3zN3rmqFnD+Av7v636DW7EFqXkxpU/S6Q3jVMXcWkyZQgS7PkeJAGKKtjV38mtJoeSmi1OC+tfA/gJXe/Obb/zzUQ3lLCabe4eB+3hcCnwDbufm8D+6rPekIf6bg3onUDo1byXOwBTHT3R+Czv+M2wNsNvGa6mdG+0vf9SXp/cRFpk04FXiYMnjuf0GAxgjAAbgOhEaE+/c0svd/vEsIP8rMJYx8uJIz/GAocBPwp6jc8C/iumT0WvdZFhDNdSbkB+K+Z/YLQWLMX8N0E45E2Tq1Ekg+nEpK1p83sa2Y21MzGEfrbNuYgjbvPB54kDOorJRyg42YBY83sG9EsDxMIA+Dq82y0zdFm9jkzOxfYJfaaTkjKzzWz081sWzPbIXp+ej/d+nwM9Ijq3t/Murr7u4TTmn+xMJ3R1mb2ZTP7pZkd3MD+ZgHfM7MvmVlqcF/nLK+5l4WpmDarYz/XAPua2QVmNsrMfgicSehjLSJtnLt/ROj3O4MwI9DHhNknNgFj3H1xA7s4HHgzbTkr+gG9FzAHeIgwC8VdhD7IK6NtzyIk0y8SZrN4JbqfCHd/mXA28GeEbmMHEwZAr00qJmnj3F2LlmYvwJaE6c4WELoUOOFg2TeHfXwv2u7eLGWdCa3MqwgH6NsIU/h8GHvOpcBbadtdAiwmnGK8kXDATH/OUYQvhrWE/nUvAofVE+dxwKrYYyMMGFwWxX9+tL4TYaT1HEKL7yLCjBRfisr3jZ7fJ23/WxGS+0rCFaXOBP4N3B57zlcILcrrgI117Y8wovyd6PXnEmbLsFj5fOCMtNf/L3Bd0v9TWrRoyX0Bfhp93g9KOpakF0J3tbeTjkNL21zMXX3aJf/M7KeEFsxDPcwMISIircDMDiN0tbjO3auSjqe1RN0rngLKCQ0G1wLneZidRyQnSpClxbTXg7SIiLQ+M3sAGAf0JgwMvwW43pXoSBMoQRYRERERidEgPRERERGRmKKa5q1///6+1VZbJR2GiEizvP7668vcfUC+96tjpIgUg5Y6RsYVVYK81VZbMXVqXVe1FBFpG8zsk5bYr46RIlIMWuoYGacuFiIiIiIiMUqQRURERERilCCLiIiIiMQoQRYRERERiVGCLCIiIiISowRZRERERCRGCbKIiIiISIwSZBERERGRGCXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIjBJkEREREZEYJcgiIiIiIjFKkEVEREREYpQgi4iIiIjEKEEWEREREYlRgiwiIiIiEqMEWUREREQkRgmyiIiIiEiMEmQRERERkRglyCIiIiIiMUqQRUREErZpE6xdG+5v3AgrVoT7ZWXw4YfJxSXSXnVMOgAREZH2zB0GDYIlS+p+zpQpMHYszJ0LCxbANtvA5pu3Xowi7Y1akEVERFrJ7ruDWVieeCKsmzSp/uQY4MtfDtsMHx72scUW4fH3vw+VlS0ft0h7owRZRESkFbzzDrz8cs3jb30rJLkHHdT0fd5/P3TvDvPnNz8+EamhBFlERKSFrVkD559f/3MeeCD0RXYPS1kZzJkT1vfqVf+2Q4eGbUQkPxJJkM2ss5ndYWafmFmZmb1pZvvHyvcxs/fMrNLMnjOz4UnEKSIi0hzXXx9aiXv3hsceC+tuvBFeeqn28x55BA47LDw3pUcP2HrrsH716pAAL1oUkui5c+G552r3Q+7QAfbdV10uRPIhqUF6HYF5wN7AXOAA4EEz+wJQDjwCHAdMAi4BHgB2TSZUERGRxquqgj/8ARYvhmuvzSw/8kjo2xeWL4fTT4cDD4Tvfrdx+95ii3A7dGhYFi0KiXHKM8+ELhdlZSHBFpGmSSRBdvcKYEJs1T/N7CNgJ2AzYIa7PwRgZhOAZWY22t3fa+1YRUREGuuJJ0Lf4rpMmBCSY4B+/eDuu5v3emawfj106lR7fc+eIVneYw+48EI4/ngYMaLu/TzzTOgf/ctfZu5LpD0qiD7IZrY5MAqYAWwPTEuVRcn07Gi9iIhIXk2aBIMHh2Tz1VfDujPOCInlpk2N20dlJfziF9mT48suCy297nDRRfmLO6W0NMT57rvwwx/WrN9nH+jcGa64Aj73OXj22TBbxqOPwimnQEVFeN7f/x66ZlxwAXTrBg8/3Ph6ixQr84R79ZtZKfAvYLa7n2hmdwBL3f1Xsef8D7jN3Sdm2f4E4ASAYcOG7fTJJ5+0TuAiIi3EzF5397F52peOkfV4+mnYb7/a626/HY47rubxW2/BjjvWvY+NG8Mguqqq2uunTYOPP4bvfCdv4TZow4bcWoBLS8M26UaNgvffz19cIvmUz2NkXRJtQTazDsDdwHrgtGh1OZA+XrcXUJZtH+5+q7uPdfexAwYMaLFYRUTaIh0jM334ITz4YGjRPeGEzPJ4cgwwZgy89lrd+9t779rJsVlogf3iF1s3OYaQ8LqHxDylvguKZEuOAWbNCvU46aT8xPXqq3DvvfnZl0hrSCxBNjMD7gA2B8a7e+pjOgPYMfa87sCIaL2IiEizjBwJhx8eBrd99FFYt+229W+zyy4hYUy1qq5bF/r+nnpqzYwU220X+iBXVdWejSIJw4eH5HfTpjBY8LXXQqKfzU47hVkyKitDd5O4W24JdVm4sPb6l18OXTXiVq+GPfesuRBKfNl1VzjqqJrHxx0HH3wAP/lJmMv52mtDH+n07b7+9drdPaqrwwDEs88OyXt6q71IviTWxcLM/gSMAfZ19/LY+gHAh8CxwOPAb4C93b3BWSzGjh3rU6dObaGIRURaR0udPmyPx0j3kGi5w8UXw4oVcMMNtZ8zcGDoI7xuXeiDC3DmmXD11VBS0vjXmj07XAK6kC1YEOpbWhpadJcsCf2t4wn94sXh0tfZDBwYBhmmd7/45jfh3/9uubjrs912cNNNob/1NtvAJZfAzJmhL/VmmyUTk7Ss1uhikcgsFtG8xicC64DFVvPJPNHd7zWz8cCNwD3Aq8ARScQpIiJtV6p/cefOYcaIRYtql5eWwsknwzXXhNbkrl1DIl1dXZMYu4dp2B5/vP7Xuuuuwk+OAYYMqbn/gx9kf84WW4R6z50bWqLjlizJflnshpLj00+HPn3gN7/JLd7GePdd+NrXMtf371/78XvvNXymoDnKy2HGDOjSpf4+69I2JNLFwt0/cXdz9y7u3iO23BuVP+3uo929q7uPc/ePk4hTRETartTgu3XrMpPj558PXSSuvx46pjUVpbca//OfMG9e3a9TUQE/+lHz4y00w4aFRPmf/8wsiyfacb/6Vc2VAOPLddeFKe7cw6DGo4+GceNg2bLQLaW8PHObbBc86dQJ9t8/zDP91FO51Wf06JquGy+8kNu2ELpzLF6cuf6++2D8+DC13q67hq4sqdd5991Q3zff1JUO25rEZ7HIp/Z4+lBEio+6WDTfzJmwfR2Tg26xRehq0CHHJqKXXw6ty+vXh32bhZbp9AS7WLmHvsiDBtX+2338cbji3157hR8eLWHTprrfrzlzwnv697+HH0KnnRZauQ84AN55p+F9/+53Yf7n1Mns+fPhz38OU/L94Q9hf3XZe+/c6/zCC6Hf94YN4SzG3/8eYu3TJ7f9tIS1a0O//Ntvh3POCd1pSkpy/6y0tNboYqEEWUSkwChBbr7DDoOHHqq9bocdQjLTq1f7SWoleP11GNvAJ6pDh+bN/7zHHmHg4pQpIeHNVdeuobV5xowwkLQhS5eGbbp3r0nun3gidAk67riQ4NfV0p/NJ5+EVv34DCjpzML84IceWvcP0NagBDlHPXv29J122qnWusMOO4xTTjmFyspKDsjyH/vjH/+YH//4xyxbtoxDDjkko/zkk0/m8MMPZ968efwwPgN75Oc//znf/va3ef/99znxxBMzys8//3z23Xdf3nrrLc4444yM8t/+9rfsvvvuvPTSS5x33nkZ5ddddx1jxozh6aef5tJLL80ov+WWW9h2222ZNGkS11xzTUb53XffzRZbDGXixEe5/faH2bixO9XVndm0qQubNnXmtNPOoaSkJ8899yqvvPIm7qVs2lSKe0c2bSrlW986GPdS3n77XT76aC7uJbh3iG5L2G23PaiuhtmzP2LJkuVAB9zDT80OHTry+c/vwKZNMG/efFavXgMYYLgbHTt2ZOutR0StEgupqEidTwuf9I4dOzF06FAAFi1aRFXV2lp169SpM4MHDwbC9uvXr69V3qVLFzbffIto/wvYuHFjrfKuXbsyYMBAABYsmE91dfVnZe5G9+7d2Gyz0Ilt/vx5bNpU+7PSo0d3+vULI0Dmzp2b8bfv1asnffr0xX0T8+bNzyjv3bs3vXv3prp6IwsWLMwo79u3Dz179mLjxg0sXLgoo7xfv3706NGD9evXsXjxpxnl/ftvRrdu3Vm3bi2ffprZaXDAgP507dqNqqpKli5dllG++eYD6dy5C5WVFSxbtjyjfIstNqdTp86Ul5ezYsWKjPLBgwfRsWMpZWVrWLlyVUb5kCGDKSnpyOrVq1m9enVG+dChW2LWgVWrVrJmTeYsj8OGDQNgxYrllJdX1Crr0MHYcsvwv7N8+bLY/1ZQUlLCkOibY+nSJRn/W6WlHRk0KPxvLVnyKWvXrqtV3qlTJ7aIrvm7ePHiLP97nRk4MMyttWjRQjZsSP/f68KAAQM54gh4++3xLF9e++/7/PPPt8jBvz0cI8Op+a2YOnUiABdc8ClHH705l1wykw8//DkdO9ae9uDuu+9m6NChPPDAA/zxj3/M2P/DDz9M//79mThxIhMnTswof+KJJ+jWrRs333wzDz74YEb55MmTAbj66qv5Z1pfha5du/Kvf/0LgEsuuYRnnnmmVvlmm23G3/72NwDOPfdcXn755VrlW265Jffccw8AZ5xxBm+99Vat8lGjRnHrrbcCcMIJJzBr1qxa5WPGjOG6664D4KijjmL+/NrHqd12243LL78cgPHjM/9P99lnHy644AIA9t9/f6rSppQ48MADOfvsswEYN24c6ZL636usHMqUKY2/hOHWW1dSXf0wm232CiUlFXzwwVmsXh06Ge+4I2yzzVyWLj2OkpLax4FrrrmNSZNGsnr1m7z88m9ZsOB7zJ///Ua95oABzzFz5hfo378/d9xxF3fe+RClpeWUl4/g9dfvaHTsAIcf/gSvvTaf+fMPYYst/s2gQf+ke3fnmWf+ihmccspt3HrrD6iu7pbTflN69lzB2rVQXd2VTZu6AtCnz0K+/vXBVFXB9OlvsnHjHDp3XsLatYNYsuSrdOmyiS9/uS/dusHChffRs+cdVFVtQXn5KHr3ns64caO4/PLLqaqCww47mrKy2vO3t9QxMk6/odsod2PdugG89lpX3ngDnn56G2bPPpn16/uxfv1mrF/fl40bezB69GZRP67vRkttJ5+curdLtNQWHVuB7aKltprj/dbRUtv//pe6t2XWerz5Zure4KzlNXlT5pDqykpY9Vnelbl9ZWUYsR5k/oyurISa431mfFVVoX9cMDRr+dKlqUfDspZ/+imErv7Zy0N/to51lgelWcsXLEjd65y1vKbPZJes5TU5fbes5TWtCN2jpbbU9FjQI1pqmz07da8XmVObh7log97RUlvNd3nfaKmtZhT9ZtFSV3n/jLLa5QMzyqqqYM2a1KPMSWSrquL/m1tkLV+5MvUo83+zqir8b36a+btGmqGsbCRvvHFbrXX77VfGiBGbM378bK65RnOCCXTrNo+99x732Y+zq676iEsuWUtZWfiOGzHiRnr3nsbvf38se+21Gy+//BbnnXfnZ9uPGXM6EP9xNotLL12f8To9emxiwgSYNGk+b765lBEjbmHLLf/GRx8dz8aN3fn977dl6NABvPzyU1x00S5s3FhznFy69KsMGBDOerzzztHA0U2u7wMP1Pz4mDfvSObNOxKId5s4/rPyXr3eYYcdzqVfvz785jd3M2UK/OMfD/P22+uZP/9Q+vV7jbKybVm/vua4WlbWL+M1V60aTM3vxS9FS43ycnjuudSjI6OlxvLlM5g5Mwy0ray8i9LSFWza1ImuXRfQtesCYJ9c/ww5K6oW5GI8fege5oqcNi109n/vvbC8/372AQzZdOgQ+jb16QO9e4fTMd261V66dg0jvTt3DoMgOnUK90tLw/2SknBKMrWkHqf6JqXfdugQTsXE7zdmgcz7KXXdj8t1fX2SnsdUil+fPqHvZDp1sWi81ICuRYtgyyy/w4voK07agYoK6JHZ3pDhq18N3Tj23LPm/q9/HfqC9+kTGjB22in+Q71hF14YBlI25rtv6dJwhcmpUyH9xM4pp4R5twcNCvNql5SEvvsQBli++mrILZrXQFCk07xJ3dxD8jt5cugrN3ly9lGzEK6ONGJE6GM0aFDNMnhwKOvbN3xQevRQsicixSd+WeUsvTe47LLWjUekubp3D41fRx4ZBu9B+BF9xx3hO71Xr8y+v9kay7beOpyl2rQJHnkE9t035AOhK0QYiHjggeFM33/+A/vsk1ueMGBAmCVmv/3g3HMbfn59gyznzw/9rvfaKzTWPfAA3HhjeP6pp4aBsHfdFR6/+27o490a1IJcANxDIvznP4d/1PRfVQMHhqs4bbddWEaPDnM59s086ywiRUAtyI1zxhlhmrZsNm7M7SIfItJ2FO2FQiT49FOYODFMp1LTHzO0/u69dxhNOm5czdyNIiISuGdPjk8+OcxgoeRYRJpDCXIrcw+dzm+5BR57LLRyQOg7d+yxcMQRSohFRLKpqAhdzrbZJvS7zObmm1s3JhEpTkqQW9HHH4f+NE88ER6XlMB3vgMnnBCuY68WDxGR7MrKQv/LbP70JzjpJLjpptaNSUSKlxLkVrBxY7jM5kUXhc70vXvD2WeHFuPB2Wc3ExGRmN13z77+6afDAKNsg/RERJpKCXILmzIFjj8+TNMGcPjhIVnONrWTiIjUVl0d5uzOdsng884LybGISL4V2NW1i0dlJfzsZ2H2iWnTYPjw0LXi/vuVHIuINEZZWZhvfZttatbFL8iW5cJ6IiJ5oRbkFlBRAd/+drhKTEkJnHVW6F7RPfNiZCIiUodrrslc9+yzYfaf3XfXMVVEWo4S5DyrqAiTb0+eHFqKH38c/u//ko5KRKTtefHF2o+/9a0ww88xxyQTj4i0HzknyGbWGRgMdAWWuvvSvEfVRsWT40GDQgvyttsmHZWISNvz5JOhtRjCVbSGDAnzw4uItIZG9UE2s55mdrKZvQCsBj4E3gEWm9k8M7vNzL7ckoEWuoqK0Lqh5FhEpPH+8Y8wRuPZZ2HmTNhhh3CV0G9+s+Y5u+4aBuN11DlPEWklDR5uzOxM4HxgDvAP4DJgIVAF9AN2APYEnjKzV4CfuvsHLRZxASovD8nxCy+Eadueew5GjUo6KhGRwrZmDRx0ULi/zz7wxS/CjBmZzxs6tHXjEhFpzO/x3YG93T3LJDsAvAbcaWYnAT8B9gbaTYKs5FhEJDfV1WEA82mn1V4/fXrmc885B7p2bZ24RERSGkyQ3f3QxuzI3dcB7e4in6ecUpMcT54MI0cmHZGISOG68074yU8a99x774Ujj2zZeEREslGPrmZ46im4+27o0gWeeUbJsYhIfRYubFxy/Lvfwfe/D8OGtXxMIiLZNDlBNrPNgQOBIYAT+iU/7u6L8xRbQaushJNOCvcnTIDRoxMNR0Sk4A0Zkn39kUfCffeF+zfdFM7MiYgkqUlX0jOzk4Fnga2AJcBSYGvgGTNrF4e2iy+GOXPCoJKzzko6GhGRtqlbtzCNm4hIIWlqC/IZwBh3r4qvNLPLgGkUeV/kadPg6qvDhPW33QalpUlHJCJS2Natq7l/9tnhGArwm9+E6dseeADuuQeOPz6Z+ERE4pqaIDvQlzDVW1y/qKxoVVeHA3h1Nfz0p7DzzklHJCJSeGbPhgEDoFev8HjZspqyq66CQw+F+fNrpnk77LCwiIgUgqYmyGcBk81sBjAvWjcM+DxwZj4CK1Q33QRTpsCWW8JllyUdjYhI4Zk5E7bfvubxmWfWJMJf+EK43XlnNTCISOFqUoLs7k+Y2ZPAzoTLThuwAHjN3avzGF9BmTsXzjsv3L/pJujZM9l4REQKzYYNNd0nUq69NiwQWpVFRApdTgmymXUm9D8eDcwH3gLecvfZLRBbQXEPk9pXVMD48fCd7yQdkYhIYXnqKfj61+t/zqBBrROLiEhz5NqC/CdgP+Bx4BxgLdDdzMqA6e6+V57jKxiPPAKTJoX+dDfckHQ0IiKFo7IS3nyz4eQYas7CiYgUslynefsW8CN3PxFYB3wZOBaoBF7Jc2wFwx0uvDDcv/zycNU8EREJjjgC9tij7vLjjqu5rznjRaQtyLUFuSvwQXR/PdDB3e8ys57A5/IaWQF56aUw6GTzzTUFkYhIukmTsq/32JxGv/sd9OkDHZo0+76ISOvK9VA1h3DlPAiD8raM7v8LOCJfQRWa224Lt8ccozmPRUTibq5j1vsuXWo/3mwzKClp+XhERPIh1wT5QSDVy2wy8JPo/heALtk2aOtWrgwT2EPt04QiIu3dI4/AqafWXjd2bJgG8403kolJRCQfcupi4e7xmX+vBKaY2QqgB3BLPgMrFPfeC2vXwr77wogRSUcjIpK8f/4TRo0KM/rE/ec/sN9+ycQkIpJPTe4N5u7zge2B04GD3f2nuWxvZqeZ2VQzW2dmE2PrtzIzN7Py2HJBU+NsDne49dZw/4QTkohARKSwTJ0K3/42bLvOwCVNAAAe70lEQVRt7fX9+ys5FpHi0dQr6QHg7iuAu5u4+ULgUuAbhMF/6fq4+8amxpYPr74Kb78dJrZPXQVKRKQ9mzEjc90uu4RWZRGRYtGsBLk53P0RADMbS81gv4KSaj0+5hjo1CnZWERECkF5eea6V4p2kk8Raa8KecKdT8xsvpn92cz6t/aLr14N998f7mtwnohIsDHtvF56P2QRkWJQiAnyMsIFSIYDOwE9gXvrerKZnRD1ZZ66dOnSvAVx771QVQVf+xqMHJm33YqItKp8HyPLymrun3gi/PGPzd6liEjByVuCbGbDzKzZ+3P3cnef6u4b3f1T4DTg62bWq47n3+ruY9197IABA5r78tE+4ZZoTg4NzhORtizfx8gLoiHTV14Jf/pTGKMhIlJs8tmC/DEwzcz2yuM+AVLXYrI877dOU6bA9OlhVPbBB7fWq4qIFJ5Vq2DJknB/ypSa9W+/nUw8IiKtIZ+D9I4FtgauAnZp6Mlm1jF6/RKgxMy6ABsJ3SpWES5p3Re4AZjs7qvzGGu9UoPzfvxj6Ny5tV5VRKSwuEPfvuF+RQU8/3xN2TbbJBOTiEhryFuC7O4To7sXNXKT89OeexTwG+B94LfAQGAN8BTw/fxE2bA1a+Cvfw33jz++tV5VRKSwuMP++9c8vvhi6BA753j66a0fk4hIa2lygmxmmwMHAkMI3SAWAo+7++LGbO/uE4AJdRT/talxNdd990FlJYwbF64UJSLSHk2fDk8+WfP4iitq+h8ffHBNy7KISDFqUh9kMzsZeBbYClgCLCV0r3jGzE7JW3QJuPPOcKvBeSLSno0Zk7nupZfC7bhxrRqKiEira2oL8hnAGHeviq80s8uAacDNzQ0sCStWhMuoduqkK+eJSPvlnrnu6KNDqzJAly6tG4+ISGtraoLshAF0VWnr+1Ez60Sb8+KL4Yth112hW7ekoxERScb69Znr7rqr5v4uDQ7DFhFp25qaIJ8FTDazGcC8aN0w4PPAmfkILAmTJ4dbnT4UkfasKtb08cc/wskn1y4fPrx14xERaW1NSpDd/QkzexLYGRhMmKN4AfCau1fnMb5WlUqQv/rVRMMQEUlUKkEeOBBOOikzQdYAPREpdjklyGb2EPBUdGWmajNbAWxJmKc4f9d5TsCKFTBtWpj3eNddk45GRCQZBx8Mc+aE+127JhuLiEhScm1B3otwIRDMbDPgVULr8Toz28fd2+y1lV54oab/sQagiEh7VF0Njz1W87ikJLlYRESSlOs0bz2BRdH98YTLS28G3AZclr+wWp/6H4tIe1dRUftxqiX5179u/VhERJKUa4I8FxgR3T8E+Iu7bwQmAm26Y4L6H4tIe5eeIKdccknN/TfeaJ1YRESSlGsXizuBm8zsCeCrwEmx/bTZidFWrAjze3burOmLRKT9Wb8eysvh8MOzl5tlnxtZRKRY5ZQgu/uVZgbwDeBsd49OwLEz8EmeY2s1zz8fDv677ab+xyLS/my9NSxcmLk+PvexiEh7kvM0b+5+JXBl2urNgfvzElEC1P9YRNqzbMkxwI9+1LpxiIgUiqZeKKSWKGlus9T/WETaqxUrsq9Xciwi7VmDg/TMbOvG7syCoc0LqXUtXx76H3fpAjvvnHQ0IiKta/787Ouvuqp14xARKSSNmcXiZTO7w8x2q+sJZtbXzE4GZgIH5S26VvD88+FW/Y9FpD2aPbv240suCbNZDByYTDwiIoWgMV0sRgO/Bh43s2rgdcJcyGuBvsDnge2A14Az3P3JFoq1Raj/sYi0Z++9V3N/p53g/POTi0VEpFA02ILs7qvc/RfAEOBk4D2gD7A1sBG4C/iSu3+lrSXHoP7HItK+bdoUbseNgylTEg1FRKRgNHqQnrtXAQ9HS1FYtgzeflv9j0Wk/Vq/PtzuuWeY71hERHK/kl5RSfU/3n33cJEQEZH2Zu3acNutzV7qSUQk/3JOkM1sfzN73MxmpmasMLPjzGyf/IfXstT/WETau3Xrwq0aCUREauSUIJvZD4AHgVmEPsilUVEJ8Mv8htbylCCLSHt3/fXhVgmyiEiNXFuQfwkc7+5nEgbopbwCjMlbVK1g6VJ45x3o2lX9j0VEVq1KOgIRkcKRa4I8Eng5y/pyoFfzw2k96n8sIiIiItnkmiAvBEZlWb8XMDvL+oKl7hUi0t5VV9fc798/uThERApNrgnyrcANZvaV6PFQMzsauBL4Y14ja2FKkEWkvduwoeb+kUcmF4eISKFp9DzIAO5+pZn1Bp4CugDPAeuAq939phaIr0WUl8OMGdCpk/ofi0j7lUqQe/QIi4iIBDklyADu/mszu4xwiekOwEx3L897ZC0odWnVUaNCkiwi0h6lLhJSWlr/80RE2pucEmQz+0cd6wFw9+/kIaYWl0qQR49ONg4RkSSlWpCVIIuI1JZrC/LytMelwI7AUOCRvETUCt59N9xut12ycYiIJCnVgqwzaSIiteXaB/mYbOvN7BqgLC8RtQK1IIuIwJo14Xb+/GTjEBEpNDlfaroOtwCn5GlfLU4tyCIicPHFSUcgIlKY8pUgb5un/bS4DRvgww/D/VHZZnQWEWknXnst6QhERApTroP0bkhfBQwC9gfuzFdQLWnOnJAkDx8O3bsnHY2ISHJ6tanrn4qItJ5cB+l9Ie3xJmApcCZtJEFW/2MRkeCQQ+Dtt6FLl6QjEREpLLkO0vtqSwXSWlIJsvofi0h7V1ISbs88M9k4REQKTb76IOfMzE4zs6lmts7MJqaV7WNm75lZpZk9Z2bD8/W6qQF6akEWkfZu+vRwqxZkEZHaGmxBruviINnkeKGQhcClwDeArrHX60+YU/k4YBJwCfAAsGsO+66TWpBFRIKnngq3qZZkEREJGtPFIv3iIHnh7o8AmNlYYMtY0feAGe7+UFQ+AVhmZqPd/b3mvaZakEVEUnr1gpUrYf/9k45ERKSwNJgg13VxkBa0PTAt9voVZjY7Wp+RIJvZCcAJAMOGDat3x4sXh4nx+/aFAQPyGrOISEGq7xiZupLeFlu0dlQiIoUtsT7I9egBrE5btxrome3J7n6ru49197EDGsh64xcIMWt+oCIiha6+Y+S6deFWl5oWEakt12neMLOOwM7AMKDWYdXd/5KHmMqB9Nk5e5GHS1lrijcRkRqpFmQlyCIiteV6oZDRhIFzWxMuElId7WMDsA7IR4I8Azg69prdgRHR+mbRJaZFRGqkEuTOnZONQ0Sk0OTaxeI64HWgN1AJbAeMBd4CxueyIzPraGZdgBKgxMy6RK3TjwI7mNn4qPxCYHpzB+iBWpBFRFLcaxLk0tJkYxERKTS5JshfBi519wrCVfQ6uvsbwC+Ba3Lc1/lAFfAr4Kjo/vnuvpSQbF8GrAR2AY7Icd9ZqQVZRCTYsCHclpZCh0IcjSIikqBc+yAboeUYwiWmhwDvA/OBz+WyI3efAEyoo+xpIK/tvGVlsGBBOJW41Vb53LOISNuzOhoKrf7HIiKZck2Q3wF2BOYArwHnmFk1cDzwYZ5jy6tU94pRozQpvojIrFnhtqIi2ThERApRrgnyZUD36P75wD+B54BlwGF5jCvv1P9YRKRGqovFbrslG4eISCFqVIJsZvu4+zPu/mRqnbvPAT5vZv2Ale7uLRVkPugS0yIiNVID9Hr0SDYOEZFC1NihGU+Z2Rwz+7WZDY4XuPuKQk+OQZeYFhGJ0xzIIiJ1a2yCvD3wCPBT4BMze9zMDjazNtObVy3IIiI1lCCLiNStUQmyu7/r7mcDWwKHAw48BCwwsyvMbNsWjLHZNmyADz4Il5ceNSrpaEREkqcEWUSkbjnNfunuG939EXc/EBgO3AB8D5hpZi+0RID5MGcObNwIw4dDt25JRyMikrzHHw+377yTbBwiIoWoydPDu/tC4GZCkrwK+Eq+gso39T8WEantnnvC7YwZycYhIlKIcp3mDQAz2xc4FjgYWAv8Fbg9j3HllaZ4ExEREZHGanSCbGbDgGOAHxO6V7wAnAA87O5rWyS6PNElpkVEauvWDSor4XM5XQNVRKR9aFQXCzN7inD1vBOB+4FR7j7O3e8p9OQY1IIsIpLu4IPD7UUXJRuHiEghamwLchVhMN7j7l7dgvHknbtakEVE0q1bF247d042DhGRQtSoBNndv5O+zsy+Akx193V5jyqPFi2CsjLo1w/69086GhGRwqAEWUSkbk2exQL4FzAkX4G0lHjrsVmysYiIFIr//S/cKkEWEcnUnAS5TaSb6n8sIpJp5Mhwq4YDEZFMzUmQ2wRdYlpEJNPaaHj15psnG4eISCFqToJ8IvBpvgJpKbpIiIhIplSC3KVLsnGIiBSinC8UYma9gZHAO7SBFmi1IIuIZFqzJtwqQRYRydToBNfMhpnZJGA58CrwJrDMzP5qZgNjzyuYIR9r1sCCBWEQyvDhSUcjIlI4Fi8OtxqkJyKSqVEtyGY2BHgF2ARcCMwkDNL7PHAK8IqZfQnYK1p3RYtEm6OPPw6322wDJSWJhiIiUlA6dw5TvfXpk3QkIiKFp7FdLC4CPgL2dfeq2PpHzexa4D/AP4BdgKPyG2LTLVwYbocU/GR0IiKtx71mHuROnZKNRUSkEDU2QT4A+EFacgyAu1ea2fnAs8DP3P3hfAbYHIsWhdtBg5KNQ0SkkGzcGG47doQOBT+SRESk9TX20DgAmF1P+YdAtbvf2PyQ8ifVgjx4cLJxiIgUErUei4jUr7EJ8hLgc/WUjwQWNz+c/FILsohIpk+jCTorK5ONQ0SkUDU2Qf4XcGm2GSrMrAtwCfBEPgPLB7Ugi4hk+uEPk45ARKSwNbYP8gRgKvChmd0IvAc4sD1hFosS4LCWCLA51IIsIpLp5ZeTjkBEpLA1KkF294VmtjtwM/BbwhRvEJLkfwOnuvvClgmx6VItyEqQRURERKSxGn0lPXf/GDjAzPoS+hwDfODuK1sisOZyr5kIXwmyiEimX/0q6QhERApTzpeajhLi11oglrxasQLWr4fevaFbt6SjEREpDKtW1dwfMya5OEREClnRzoCpAXoiIpkmTKi5379/YmGIiBS0ok2QNUBPRCTT9dfX3N977+TiEBEpZEWbIKsFWUSkfh1z7mQnItI+FG2CrBZkEREREWmKok2Q1YIsIlK3v/wl6QhERApXwSbIZjbZzNaaWXm0vJ/L9mpBFhHJNGpUuN1552TjEBEpZAWbIEdOc/ce0bJtLhuqBVlEJNPGjeG2pCTZOEREClmhJ8hNphZkEZFMc+aEWw3QExGpW6EnyJeb2TIz+5+Zjcv2BDM7wcymmtnUpUuXAuEqerrMtIhI7WPkp58u/2y95kAWEalbISfI5wDbAEOAW4FJZjYi/Unufqu7j3X3sQMGDABg5cpwFb1evaB791aNWUSkoMSPkf36bQaE7hU9eiQcmIhIASvYBNndX3X3Mndf5+53Af8DDmjMtup/LCKSadOmcLvllsnGISJS6Ao2Qc7CAWvME9X/WEQkk3u47dQp2ThERApdQSbIZtbHzL5hZl3MrKOZ/QDYC3iyMdurBVlEJFMqQe7cOdk4REQKXaGOYy4FLgVGA9XAe8DB7t6ouZDVgiwikkktyCIijVOQCbK7LwW+3NTt1YIsIpJpw4Zwu3ZtsnGIiBS6guxi0VxqQRYRyWTRKI6qqmTjEBEpdEWZIKsFWUQkU6qLxfbbJxuHiEihK8oEWS3IIiKZVq4Mt+qDLCJSv6JLkHUVPRGR7JYvr30rIiLZFV2CvGoVrFsHPXvqSlEiItk8/3zSEYiIFLaiS5DV/1hEREREmqPoEmT1PxYRqd/NNycdgYhIYSu6BFktyCIi2aWmeTv22GTjEBEpdEWXIKsFWUQku9Q0byUlycYhIlLoii5B1gwWIiL1U4IsIlK/okuQUy3I6mIhIpJdqquFiIhkV3QJslqQRUTqptZjEZGGFV2CrBZkEZG6KUEWEWlY0SXIakEWEalbh6I76ouI5F9RHSqrq2Ht2nAFvZ49k45GRKTwqAVZRKRhRZUgb9gQbtV6LCKSXUVF0hGIiBS+okyQ1f9YRERERJqqKBNktSCLiIiISFMVVYK8fn24VQuyiEh2AwcmHYGISOErqgRZLcgiIvU788ykIxARKXxFmSCrBVlEJDsdH0VEGlaUCbJakEVEshszJukIREQKX1EmyGohERHJrmPHpCMQESl8RZkgqwVZRCS70tKkIxARKXxFlSBv2gTdu+sqeiIidVGCLCLSsKJKkCG0HpslHYWISGFSgiwi0rCiS5DV/1hEpG7qgywi0rCiS5DV/1hEpG5qQRYRaVjRJchqQRYRqVuvXklHICJS+IouQVYLsohI3dTFQkSkYUWXIKsFWUQkuw5Fd8QXEWkZRXe4VAuyiEh2muFHRKRxii5BVguyiEh2SpBFRBqn6BJktSCLiIiISHMUbIJsZv3M7FEzqzCzT8zsyIa26dBBI7RFROqiFmQRkcYp5PHMNwHrgc2BMcDjZjbN3WfUtUFpqb4ARETqouOjiEjjFGQLspl1B8YDF7h7ubv/F/gH8MP6ttME+CIidVOCLCLSOIXagjwKqHb3WbF104C9059oZicAJ0QP15nZO60QXyHpDyxLOogEtMd6q87tx7b52pGOke32f6g91lt1bj/ydoysS6EmyD2A1WnrVgM905/o7rcCtwKY2VR3H9vy4RWO9lhnaJ/1Vp3bDzObmq996RjZ/uoM7bPeqnP7kc9jZF0KsosFUA6kD7frBZQlEIuIiIiItCOFmiDPAjqa2cjYuh2BOgfoiYiIiIjkQ0EmyO5eATwCXGxm3c3sK8BBwN0NbHpriwdXeNpjnaF91lt1bj9aqt7t8e/ZHusM7bPeqnP70eL1Nndv6ddoEjPrB9wJ7AcsB37l7vclG5WIiIiIFLuCTZBFRERERJJQkF0sRERERESSogRZRERERCSmKBJkM+tnZo+aWYWZfWJmRyYdU76Z2WlmNtXM1pnZxLSyfczsPTOrNLPnzGx4QmHmlZl1NrM7ove0zMzeNLP9Y+XFWu97zGyRma0xs1lmdlysrCjrnGJmI81srZndE1t3ZPQ/UGFmf4/GJxQFM5sc1bc8Wt6PleWt3sVwjGzO8SDa9s7oM7XYzM5K23fBf65y/Ww09J63hc+VmR1hZu9GMc42sz2j9UX5XpvZVmb2hJmtjGK/0cw6RmVjzOz1KO7XzWxMbDszsyvMbHm0XGlWc93M+rZtbdbEXKY572tD29bJ3dv8AvwVeIBwgZE9CBcV2T7puPJcx+8BBwN/BCbG1veP6nso0AW4Cngl6XjzVOfuwARgK8KPuQMJc2FvVeT13h7oHN0fDSwGdirmOsfq/h/gReCe2N+iDNgr+nzfB9yfdJx5rO9k4Lg6/gfyVu9iOEY253gAXB79X/UFtos+U9+MytrE5yrXz0Z973lb+FwRBuh/Auwavd9DoqVo32vgCWBiFNsWwNvAz4BO0d/iTKBztO4ToFO03YnA+8CW0d9oJnBSVFbvtgnUsUm5THPe1/q2rTfWpP8h8vDH7g6sB0bF1t0N/C7p2Fqovpem/VOdALyU9veoAkYnHWsL1X86ML691JtwOc1FwGHFXmfgCOBBQhKUSgJ+C9wXe86I6PPeM+l481TnyWRPkPNW72I+Rjb2eAAsAL4eK7+EKCFsC5+rXD8bDb3nbeFzBbwE/CTL+qJ9r4F3gQNij68CbgG+HtXLYmVzqUkQXwJOiJX9hChBbGjbBOuaUy7TnPe1vm3rW4qhi8UooNrdZ8XWTSP8Qm4PtifUF/hsDunZFGH9zWxzwvs9gyKvt5ndbGaVwHuEBPkJirjOZtYLuBj4eVpRep1nE33xt150Le5yM1tmZv8zs3HRunzWuyiPkY09HphZX2BwvJza9S/oz1UTPxsNvecF/bkysxJgLDDAzD40s/lRd4OuFPF7DVwPHGFm3cxsCLA/8G9CfNM9yu4i06mjXmTWub5tC0WLvK+N2LZOxZAg9yA0rcetJvyKbg/aRf3NrBS4F7jL3d+jyOvt7qcQ6rIn4aI56yjuOl8C3OHu89LWF3OdAc4BtiGcFr0VmGRmI8hvvYvub5jj8aBH7HF6GQ1sWwia8tloqE6FXufNgVLgEMIxcAzwJeB8ivu9fp6QuK0B5gNTgb+T+/u5GugR9UMu9DqntNT72tC2dSqGBLkc6JW2rhehf1V7UPT1N7MOhNOD64HTotVFX293r3b3/xL6lZ1MkdY5GjCyL3BtluKirHOKu7/q7mXuvs7d7wL+BxxAfutdVH/DJhwPymOP08sa2jZRzfhsNFSngq1zpCq6/YO7L3L3ZcDvafiz0Zbf6w7Ak4QGke6EfrV9gSvI/f3sBZRHrcYFW+c0LfW+NrRtnYohQZ4FdDSzkbF1OxJOu7UHMwj1BcDMuhP6kxVF/aNfwHcQWhTGu/uGqKio652mIzV1K8Y6jyMMtJprZouBs4HxZvYGmXXehjDQZFbmboqCA0Z+6100x8imHA/cfSWhm9KOsV3F61/In6txNO2z0dB7XtCfq+g9m0/4PKQr1ve6HzAUuDH6wbwc+DPhR8EM4IvxmSmAL1JHvcisc33bFooWeV8bsW3dkuykncfO3vcTRux2B75CGxyh3Yg6diSMzryc0HrSJVo3IKrv+GjdFRTQqNw81PtPwCtAj7T1RVlvYCBhQE4PoAT4BlABHFTEde5GGLGdWq4GHo7qmzrduGf0+b6HAhtt34x694ne39Rn+QfRe71tvutdLMfIph4PgN8RTl/3JcwMs4iaAU4F+7lqzmejvve8LXyuCP2up0THxL6EWQguKdb3OopvDvCr6HjQB3iU0JUoNRPF6YQfMqdRexaLkwgD/IYQ+tvOIHMWi6zbJlDHJuUyzXlf69u23liT/ofI0x+8H6GfTgVhdOaRScfUAnWcQPg1HV8mRGX7EgZzVRFGxW+VdLx5qvPwqJ5rCadJUssPirXe0Qf9eWAV4QvsbeD4WHnR1TnL32AC0Uj96PGR0ee6AngM6Jd0jHl8r6cQTvWtIiR++7VEvYvhGNmc4wEhMbgz+kx9CpyVtu828bnK5bPR0Hte6J8rQh/km6PPxmLgBqBLMb/XhL7Wk4GVwDLgIWBgVPYl4PUo7jeAL8W2M+BKYEW0XEntWSvq3Dah/+Gcc5nmvK8NbVvXYtHGIiIiIiJCcfRBFhERERHJGyXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFmkmcxsgpm9k3QcIiKFSMdIaYs0D7K0KWY2Eejv7gfG77fSa28FfAR82d2nxtb3ADp7uDSoiEhidIwUyY+OSQcgkjQz6whUexN/Lbp76opeIiJFR8dIaY/UxULaJDObABwNfMvMPFrGRWVDzOx+M1sZLY+b2cj4tmb2jpn92MxmA+uA7mb2TTN7MdpmhZk9aWbbxV72o+h2SvR6k+P7i+2/g5ldYGbzzGydmb1tZgfFyreKth9vZk+ZWaWZzTSz/WLPKTWzG8xsYbSPeWb2u7z/IUWkKOkYKdI8SpClrboaeBB4GhgULS+ZWTfgOWAtsDewG7AIeDoqS9kaOBI4FNgxen534DpgZ2AcsBqYZGadom12jm6/Gb3e9+qI7XTgF8A5wBeAR4FHzGxM2vMuA26IXn8KcH90KhLgZ8B3gSOAkcDhwPsN/1lERAAdI0WaRV0spE1y93IzqwLWufvi1HozOwow4JjU6UAzOxFYAhxI+MIA6AT80N0/je32b/HXMLNjgDWEg/5/gaVR0fL4a2ZxNnC1u98XPb7QzPaK1h8Ve9617j4peq3zgB8BY6LXGg7MAl6M6jEXeKn+v4qISKBjpEjzqAVZis1OhJaPMjMrN7NyQitHX2BE7Hnz0w78mNkIM7vPzGab2RrgU8JnZFhjX9zMegGDgf+lFf0X+Hzauumx+wuj24HR7UTCF8EsM7vJzL5lZvq8ikhz6Rgp0ghqQZZi0wF4i3DaLd2K2P2KLOWTgAXAidHtRmAmoSUlV9kGs6Sv2/BZgbubGUQ/Wt39DQsjwr8JfA24C5hmZvu5+6YmxCMiAjpGijSKEmRpy9YDJWnr3gC+Dyxz91WN3ZGZbQZsB5zq7s9F6/6P2p+R9dFt+mt+xt3XmNlCYA/g2VjRHoQvkkZz9zLgIeAhC9M1vQJ8jnBaUUSkITpGijSREmRpyz4G9jezbYHlhNOE9xL6sT1mZhcS+qUNBQ4C/uTuH9Sxr5XAMuB4M5sHDAGuIrSQpCwBqoBvmNnHwFp3X51lX1cBF5vZB8DrhD51exJObTaKmZ1FGDjzFqEV5UhCX7/5jd2HiLR7H6NjpEiTqL+OtGW3Ae8CUwmDQ77i7pXAXsAcQsvCe4RTb30JB/isolNyhwNfBN4BbgIuIExvlHrORsLI6eMI/eEeq2N3NxC+AK6M9vVdYLy7v5VD3coIo7xfI7T4jAH2j+onItIYOkaKNJGupCciIiIiEqMWZBERERGRGCXIIiIiIiIxSpBFRERERGKUIIuIiIiIxChBFhERERGJUYIsIiIiIhKjBFlEREREJEYJsoiIiIhIzP8D281N+Dou0q8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "true_Q_value = history1[-1, 0, 0]\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharey=True)\n",
    "axes[0].set_ylabel(\"Q-Value$(s_0, a_0)$\", fontsize=14)\n",
    "axes[0].set_title(\"Q-Value Iteration\", fontsize=14)\n",
    "axes[1].set_title(\"Q-Learning\", fontsize=14)\n",
    "for ax, width, history in zip(axes, (50, 10000), (history1, history2)):\n",
    "    ax.plot([0, width], [true_Q_value, true_Q_value], \"k--\")\n",
    "    ax.plot(np.arange(width), history[:, 0, 0], \"b-\", linewidth=2)\n",
    "    ax.set_xlabel(\"Iterations\", fontsize=14)\n",
    "    ax.axis([0, width, 0, 24])\n",
    "\n",
    "save_fig(\"q_value_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build the DQN. Given a state, it will estimate, for each possible action, the sum of discounted future rewards it can expect after it plays that action (but before it sees its outcome):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "input_shape = [4] # == env.observation_space.shape\n",
    "n_outputs = 2 # == env.action_space.n\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(32, activation=\"elu\", input_shape=input_shape),\n",
    "    keras.layers.Dense(32, activation=\"elu\"),\n",
    "    keras.layers.Dense(n_outputs)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select an action using this DQN, we just pick the action with the largest predicted Q-value. However, to ensure that the agent explores the environment, we choose a random action with probability `epsilon`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epsilon_greedy_policy(state, epsilon=0):\n",
    "    if np.random.rand() < epsilon:\n",
    "        return np.random.randint(n_outputs)\n",
    "    else:\n",
    "        Q_values = model.predict(state[np.newaxis])\n",
    "        return np.argmax(Q_values[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also need a replay memory. It will contain the agent's experiences, in the form of tuples: `(obs, action, reward, next_obs, done)`. We can use the `deque` class for that (but make sure to check out DeepMind's excellent [Reverb library](https://github.com/deepmind/reverb) for a much more robust implementation of experience replay):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "replay_memory = deque(maxlen=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's create a function to sample experiences from the replay memory. It will return 5 NumPy arrays: `[obs, actions, rewards, next_obs, dones]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_experiences(batch_size):\n",
    "    indices = np.random.randint(len(replay_memory), size=batch_size)\n",
    "    batch = [replay_memory[index] for index in indices]\n",
    "    states, actions, rewards, next_states, dones = [\n",
    "        np.array([experience[field_index] for experience in batch])\n",
    "        for field_index in range(5)]\n",
    "    return states, actions, rewards, next_states, dones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create a function that will use the DQN to play one step, and record its experience in the replay memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_one_step(env, state, epsilon):\n",
    "    action = epsilon_greedy_policy(state, epsilon)\n",
    "    next_state, reward, done, info = env.step(action)\n",
    "    replay_memory.append((state, action, reward, next_state, done))\n",
    "    return next_state, reward, done, info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's create a function that will sample some experiences from the replay memory and perform a training step:\n",
    "\n",
    "**Notes**:\n",
    "* The first 3 releases of the 2nd edition were missing the `reshape()` operation which converts `target_Q_values` to a column vector (this is required by the `loss_fn()`).\n",
    "* The book uses a learning rate of 1e-3, but in the code below I use 1e-2, as it significantly improves training. I also tuned the learning rates of the DQN variants below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "discount_rate = 0.95\n",
    "optimizer = keras.optimizers.Adam(lr=1e-2)\n",
    "loss_fn = keras.losses.mean_squared_error\n",
    "\n",
    "def training_step(batch_size):\n",
    "    experiences = sample_experiences(batch_size)\n",
    "    states, actions, rewards, next_states, dones = experiences\n",
    "    next_Q_values = model.predict(next_states)\n",
    "    max_next_Q_values = np.max(next_Q_values, axis=1)\n",
    "    target_Q_values = (rewards +\n",
    "                       (1 - dones) * discount_rate * max_next_Q_values)\n",
    "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
    "    mask = tf.one_hot(actions, n_outputs)\n",
    "    with tf.GradientTape() as tape:\n",
    "        all_Q_values = model(states)\n",
    "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
    "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's train the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "rewards = [] \n",
    "best_score = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 599, Steps: 200, eps: 0.010"
     ]
    }
   ],
   "source": [
    "for episode in range(600):\n",
    "    obs = env.reset()    \n",
    "    for step in range(200):\n",
    "        epsilon = max(1 - episode / 500, 0.01)\n",
    "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
    "        if done:\n",
    "            break\n",
    "    rewards.append(step) # Not shown in the book\n",
    "    if step >= best_score: # Not shown\n",
    "        best_weights = model.get_weights() # Not shown\n",
    "        best_score = step # Not shown\n",
    "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\") # Not shown\n",
    "    if episode > 50:\n",
    "        training_step(batch_size)\n",
    "\n",
    "model.set_weights(best_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure dqn_rewards_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACJEElEQVR4nO2dd5wkVbn3f09Vp4mb88IG0sKSWSSDCIiIXr2CGbOiXrnGq+JrQuUqGK8YL4pg4KKgmEBQQBABQZbswhJ3Fzbn2Ykdqs77R9WpOnXqVHV1z/TM7Ozz/Xxmt7viqTPh/PqJJIQAwzAMwzDMRMIa6wEwDMMwDMOMNCxwGIZhGIaZcLDAYRiGYRhmwsECh2EYhmGYCQcLHIZhGIZhJhy5sR5AK5k+fbpYuHDhWA+DYRiGYZgW8cADD2wVQszQt09ogbNw4UIsX758rIfBMAzDMEyLIKI1pu3somIYhmEYZsLBAodhGIZhmAkHCxyGYRiGYSYcLHAYhmEYhplwsMBhGIZhGGbCMWoCh4iKRHQFEa0hol4ieoiIzlL2n0ZEK4logIhuJ6IFyj4iokuJaJv/9VUiotEaO8MwDMMwuxejacHJAXgBwCkAJgH4LIBriWghEU0HcL2/bSqA5QB+pZx7PoBXAzgMwKEAXgHgvaM2coZhGIZhditGrQ6OEKIfwEXKphuIaBWAowBMA7BCCHEdABDRRQC2EtESIcRKAG8D8A0hxFp//zcAvAfAD0dr/AzDMAzD7D6MWaE/IpoFYH8AKwC8H8Ajcp8Qop+IngWwFMBK//9HlNMf8beZrns+PIsP9t5775aMnWGY4XPej+/DlI4CvvPGI8ZsDP9x9QPoKzv42TtfZNz/wvYBvO3Kf+KX7zkWM7tLAID/99vHcP+q7bAtwiXnHIrD95psPPey257GN295CmcunYU/r9iEf3zqJTj3B//AhWctwYPP78DV9z0fOb5oWzhywRRs2jWEmz98clPP8+sH1uKTv3kU9/2/0/CKy+7C9oEKAOCDL9kXcya14WPXPYJ3n7gIn3nFQQCAEy/9K1571F7oKNq4+MYn8IpD5+DOp7ZgqObCJsLXXnsoVm7oxXdvfwZf/vdD8OyWPvz96S34y0dOAQC89Sf/RHcph1ceNhfv+8UDeO1R8/HVcw8DALz/Fw/gtpWbAQCHzJuE37z/+GCcr/3hPXhkbQ9mdRfxxwtOxCu/exc27Srj8PmTce37jgMA/Oc1D6FnsBr73lzzz+dx0R9WQCjbXnvUfPz3vx+C79/xDL5685N41eFz8deVm1GuuTjjoFm455mt6K84kescufdkrNk2gG39FRy9cAq++KqD8Zrv34MZXUW0F2ys3Nibac7b8jZO2m86/vL4pti+rmION3zwRLzrquV4fvsAXnzAjNhxOYtwxkGz8OcVG+GK2CVSmTuphMntBTy+YVfmc2Z0FrHvzE7847ltxv0E4KT9puPWJzZj35mdOHHf6ThwThc+/4cVmcd3xoGzcOfTWzCjq4hpHQU8+PxOfO9NR+JlB8/OPM6RYEwEDhHlAVwN4KdCiJVE1Algi3ZYD4Au/3Wn/17d10lEJISITLkQ4nIAlwPAsmXLGvxxYRhmtLjrma0AMKYC50+PbUzd//TmXjy3pR/Pbx8IBM7/KcJk5YZdiQLn4Rd2AgD+vMJb0K5bvhbrdg7iw796GEctmILpHQX82+HzAADlmoMr716Nvz3l/Rl0XAHbajzM8Mt/egKOK/D0pj5s3DWEMw6ahQfW7MC/1u3CjoEqAGDF+nAxXLtjEN+69Sm85ghvHDc8ugEA8MYX7YVfP7AWj67twVObvIX+sXU9uOafUVF2pz/e/Wd1QQjg0bXhn+nH1vVg0bQOzOgq4q5ntqJ3qIquUh6DFQf3r96Bye15vLB9EM9u6cML2wcxpT2Pf67ejqGqg1Lexh8fWW98xhXre0AEvOP4RQCAGx9bj3/5z7Rinff/Hx5ZDyE88XGj/0znHDkfM7qKAICnN/UG4mv/WZ149IUerN7aj57BKnoGvXk69YAZOGB2d+p8b+8v49rla3HDoxswvbOIc4+aH+zb0DOI3z+8Hs9u7g8EiH6cEAL/e+dz+P3D61HIWXjnCYtS76eyc6CCX97/ArBtAGcfMgd7TW2ve86qrX3484pNWLdzEAfP68aJ+8a6G+Bn/1iNW5/w5uaZzX14ZnMfzjlyPgq2hTcdsyB2vM7vH16HGx/z5rx3qIbntvTDtggLp9cf30gz6gKHiCwAPwdQAXCBv7kPgP6T1A2gN2F/N4A+XdwwDMOMJJWa9yfGSfjoWkv5SNtfrkXe/2tdT3gtAew9rR0XnrUEAFB1XFx59+rg2M29Q5gzqa3h8br+n0T5/zlHzsfaHYOouQJVx/XuL49Rxq4/xwdP2w+3PbEZPQPV4NmHqlELiErNv7aK4wocOn8STjtwJu56ZivWbBvAwfMmYX3PIABg6dxu3P3MNgxVXf/9JNz1zFZs6BnCoukdKfcSmNSWD+buyY27sKWvDADBM8qVYVZ3Eau3DQAA3nfKYuw3y/vMvLFnCH+95DYcMKsLpxwwA1fevRrlWvQZ3v/iffGiRVMTxwEAq7f249rlawEAC5XvJwA8unYnfv/wemzrL0fO2XtqW+S4n9y9ClVHYGp7IbK9Hq4r8LentmB7fwWXnHMIukr5uufc8+zWQHC/eP+Z+K8zD4gdc+Nj6zGwfTCybXPvEPaZ2ZlpfI+t24kNPUORbR8/8wAsqSMWW8Gopon7mU9XAJgF4BwhRNXftQJeALE8rgPAPv722H7/9QowDMO0kJrrLXpJOiZJ+ADAgOYSUS0n63YOwlISQfO2hc5i+Hlz3Y7oApMVubBX/IU+bxPyNqHmuqHA8cc8qAgW/TnaCzlMastj11A1uOZgJUXguHEh6LgCOZuw0Bcrq7b2AwDW7/SebW/f4iCvu/c07/2TG3fhU9c/mniviuMiZ4VLl21ZkPpKF2rqol/K28Hr2ZNK+NRZS3DhWUtQzNmo1FyUa9HnO2BWF+rRVrCNrwGgw/9+bu2rRLar41CP6yhGt9fDsgiffcVB+OwrDsokbgBgakcheD1FeR0ZXy4+jo09Q5jVVcp0D9P56s/2aDLadXB+AOBAAK8UQqi/wb8FcDARnUNEJQCfA/CoH2AMAD8D8FEimkdEcwF8DMBVozhuhmH2QEKLQBMWnErUgrNuZ/gnb+OuIeiFLia15Y3HNoK0ypSrUuBYyFmEmiNQdaIiRBVgUshJOgo2utvy6BmsGgWRTiBwlHlyhYBFFAiZNds8gSPF295TOyLXXegLnC/d8ASu+ecLyfdyBAo5VeCEz61/P7rbwoVVFyDnn7wPXnzATBT9a/UNed+vgm1h7qQSJrXXFw0RgaMJl85A4EQtOMVcdNntKOQixzfCyw+Zg/OOre82kkxtD0XN1A7z8+nzBACbdg1hVncx0z2K+bis6CqNjcAZtbv6dW3eC6AMYKNSxua9QoiriegcAN8F8AsA9wF4g3L6/wJYDOAx//2P/W0MwzAto1qLL9wqJteMZKCcLAhcIUCIKpwpHflA2NQTOE9t6sWabQM446BZAID7ntsGy6LANSWtETmbkLMtVJ3QgiOPGVAEmGp5KeYs5GwLk9ry2LRrKLAwpFpwnLjbS8YRtRdymNVdxKqtnqto/c5BWATMn+K54KTA2WtKO4jqP3vVcZFT4pNsi4Lvj/796FYsG7oAkRRsb0Hu9V2Kbzt+QeS8NNRrJlpwenWBox9nR45vJarVZmqHWbCYLDC7hmpBDFo99OcDxs6CM5pp4msAJEbNCSFuBWB08PmxNp/wvxiGYUaFah0XVSMWHMCLCdm0qwwhELPgTG4LF5/1dRb5l37rTgDA6kvOBgC8/vJ7AYQLrm7BKdfiLippwbEo+hxyoe0u5fD05ira/YU73YITje/xtoWB0rO7S0GczLqdQ5jVXQpcNWX/uqWCjVldJWzcFY3f0Kk6Anlbd1FJgaNZcLIIHN+i0utbcD542n6ZXT6BhcwVseu3+++39esuKs2CE7ioWr8c520LXaUceodqEWtOZHwGCw4AzMoocPTnA/YcFxXDMMxuQ9UPPHUThExSDI4QIhZkDABHLwyDVvVi7KpLZJsWt5EVaZ0ZkhYcy7Pg1Bw3FjAtLTiFnBV5DiloJrXlI0HGWVxUqqfLdQVs/xnz/hgAYEtfGTO7ioEVRl43b1lGN8hgxcF9Skpz1XGRtxULDoXPpLvapGukmLNgJWSlFQOB44WEFnKNLYtS2OixNZZF6CjYBheV2ZU1WiJAxuFMSXBRlRKeP7OLymTBGSMXFQschmGYBGTcittgDE655hqtPscoWTn6cjtZicFJC15OQ56lWnDyFqHqiEQLTjFnRywfMiZkUlseveVa8IzpLirv2qrAcERowcnZFNxjsFJDeyGHnC9SZBZVziYUDVaWT13/KF5/+b14YftAcA/VgmNZpAic8DksCq0iprgSiW7BKdiNLYvS4mG6R0cxFxOreoyKnO9Gg4ybZYpvuZmaEGScNFdzJmV1UbEFh2EYZtxT10WVEINjst4AwKLpncFr3aAwWbHgJAmqegRBxr4FJ29bnrhQs6hEVOAUclZEmLT7C213Wx5CALv8ujDpaeLSOhRuc1wRWE3ythXM5WDVQVvBDsRPYMGxrZgVBACe2OBVC5Euv2pNBOII8KxUUuBUFaGWs61gsU5yT8nnBzyBU8hZMctaPeS1TffoLOYC15wkFmQ8ii4qAJjWUUApb6G9YL5f0lwtmJacuq8in0+Nk2KBwzAMM86o1qmDk7Rdigf5h32/mZ149KKXQsluji2kU5SYiGYtOIGLSrGKeC6q0ILjai6qYs6KxuD4C1+3b1GSBQIzuagUYea4IljkbEu14HgCR1phpGUob1OiewQIU+CrmgVHDTJ2FKFWsK1U8SGRLpXeoarR+lCPtHt0FHOoaPV1dBHX6QvKzgTBMdLsNbUd86ckF90ziUwAkTlPQ1rh2gt24O4cKxfVmLVqYBiGGe80myYurQ1TOwroK9fQWcqhu5SPZE7pdgIZxDmzqwinOX0TWJoCC47lu6hcN0wTN1hwTDE4MkBXVvbNFGTsX0cIAVcgqPWTs0IRNVR10ZYPLTjSMpSzLKOLSurAQOA4mouKKEwTj1hwKDE+RkW14DQjcKSLyhScq7qdbN/SNNYWnP868wD8x4v3SdxvmqtG3Hby+Yp5GyV43xNTXM5owBYchmGYBOq5qJIsLf1+iriMc5Cpt6rRRrfgnHXwbPzuAydg3pS2REGVFd2C40QsON4xMo294Ft4pEchyKJqiy64+pDUMepp4nJe7MBFRYE7b7DqoC1vI++bs4aq2Sw4kmpNRIKMZRYTEH6/vO1WanyMJEgTH6o1tRDLbKn2BBeVRNY5iqeJj26QcWcxl5rybcqCOuWAeEuHJKRILeYsdJZyY2a9AdiCwzAMk0i9OjjVhBicAcWCA4SBpWr1Yj3UI2dbOHyvybCJGnJRmcSQWgcnbxOqSqsGaW0Jg4wtDFVddLflsXOgGlgd0qwZrisizS5rWpCvnK8wyNiKBCu3FewgjkZahnIJMTjBcyIUMTktyDgQVooFp6BYcLLF4FQxvTNbppBKW50gY8nktjy291diQcado2zBqYc+Vz8870icvH8DAkdacHJedW49dX80YQsOwzBMAvVcVMkWnKjAMVpwEu6pZgXVQwgR66EEINhWsC3PPeQoLippwamGgdA11w0sDDIGx0oJtnWFiIg7XdhIQ0pgwbEIVceF6woM+s00wzTxsK1EmqhSXVSqy8SmMAan6qouKiuTiypIEy/XGk4RB+rH4EhkGQC9kF6zrRpahSrUiIAzl85ODEg2UVLmvKuUH7MAY4AtOAzDMInosSXx/ekuqmlS4Pif2lXJkKQfbKJYPZckHDdB4AQuKitI0ZbBrkElY3+MNVfAcUUQcyMXs7Ru5o4QkWevaQHMcvyyDo4MMpZjbcvbgRVmqJKeRaU3EK05IlrJ2CYMVBy89of3YItSNThvE9oK3j3SXFRS4AiRbrVKopQiouTiXsiFYituwbEjx441qgDrKOQazipTLTj/+ZJ9MWT4+RwtxseMMgzDjENkcbxGY3Cki0qWxi8GFpxwsUiykFgW4CbH80aouSLWJBKIFvqTKdpJdXAc1+tTJWNupCVBHV9nMYc+JfXddaPuoHoWHOmiku6otrwVK/SXs8wWHN39VXVc5HNRCw4A3L96R+Q8VTC1GeJKJKrVpikLToqImtnlubwqNTey8Kscv+90vOekRThk/qSG790K1GDp9hRhmET4nDaOWTxtxMbVDCxwGIZhElD7N/3krlU4ZvHUyP4kC44M8pVun8CCEwkyNt/TaiAGxxUisNZE76/UwQmabUatLIO+i0pacOZMasN5x+6NF/sBpWqtmY6iHRU4QkQCemuBK8+7vh6DIzuaSzETLfSXHoMj50KtdZPXelGZyDeYJq6/zkraPRYqtWNKgQUnelx3KY9Pn31Qw/dtFTLQ+4i9J+PMpbMbPz+waI19BAwLHIZhmASkq8V1Bb54w+Px/QlBxnLxl24H+Uc/4qJKiMKxlaaZ9XASLDjlmgsi71oyy0jG4Kjp2vIaNdfr0H3xqw8Jx0GqwMnB65Ps31eISPCoKvQ895V3bVnoz4sDEkHNm5Kx0B8ZF0VHsdzI//U0cRM5mwJ3W1tKDIlqtRnpOjgLp4f1ZpIsOOMNaYk6+5A5ePdJixs+X7XgjDXje6YZhmHGEOmiSsqWSrLgyMW/U+mFBKRnUUkayaJyXBEIFZVyNRQBMtZFCokgIFfJqnLcaIduAJHeTXp8iHCjbRlUseO4InRRUWjBqTpuYK0xp4lbxkVRTz2vOSKSRaWPO7QaZbPgDN9FldwOQi2oJ59tPCz8aWQJzE4jeM5xYMEZ+xEwDMOMU6QIMIkIICX42IlacKRbIq0OjsSyKDHmx3R/cxaVE7hxpCtIjlUXDI4jIp2/JaoFRxcIcQtOOAZXhC6qSCXjSAyODVumiVeUGBzDoqjG4AghUHFcFBT3md5EU441bxM6ijaWzu3GQXO7Y9eVFIdpwVk6txtLZncF7kgVVSTIZxvvFpy0oOksjKfnZBcVwzBMAnrNGJ1tfWU8t6UPi2d0RrZXXeEXrosuFmmVjCUWJXcv13FcgbKhwnC5FtaKyVvRhcZRspGAMAYnyRKijl/iKm4oeQ11THL8lhJk7LgimMe2ghVacGqe9YiI0mNwHBG8zmmtGlRKeS9eyOvDZeHGD54Uu6aKd28vfqgZC87J+89IrROzeHoH2ov2sIXDaDFnUgmdxRwWz8jWe0pnPD3n2EsshmGYcYos9JfUpuCRtT14yTf+Fttec1y/9UD006xqtEnKwlZ7K9XDSaiDM1R1g2q/arBwMWcFgcBVNxqDY2tCSBUO+qdx19XTxIWyLxQ8UodIa1LvkNf2oaS0anDcsHmm6VN/IMiUOKJ8iotKZjXlrGzLGxEFdXVa4T667WOn4Ib/PGm3icGZ1lnEv75wJo7ce0pT54+n5xz7ETAMw4wBWawklcBFlTFv26fqeIv21I4CchYFfaYyuaiU3kr1SAwyrjrBAq9aO2SciCNCa0gWC46e+RNzUUWK/rlKq4boGHqHvEysNqXQHxBamYwWHCd0qUlRprZq0IOMpYuqkMtev6XQwkVZfp/DHk0Te9ndY4OMiegCIlpORGUiukrZ/mYi6lO+BohIENFR/v6LiKiqHdN4eDfDMIxPFiuJdMMMJrio0s7L2xamdxbxt0+citOWzASQzUXVeBaVuZJxPhdWEZZIl5mjWEMcKXDsFIGjW3BENPBaHYMjRDB+NcgYCC047YUcLIsCK5asa2MSOGEMjotqTQqcZBeVDPrNasHxnk+KotYtiVM6CijY1rgp6Ncq2vI2SnkrqAE1loz2TK8HcDGAMwG0yY1CiKsBXC3fE9HbAXwWwIPKub8SQpw3OsNkGGaikyVTSXdRyViNeqjVdudNDv7UQV1z0yw4mV1UrjBalyqOG1hFVAuOTMN2hQisLmWlKKCKGmSsp2+7rojMX7nmBt2yXTd0Wclb24GLKrTgyLFVamEGV6qLygndYqkCxx+rekw98oGLrHVWh3OOnI9lC6Y21PZgdyRnW7jhP0/EvMnt9Q9uMaNqwRFCXC+E+B2AbXUOfRuAn4nhttRlGGZcMh5+tTMJHCcaZJzWnyl6njAusBELTkqhv4ydGrxCfwml8KVFRnXnSAuJowiUqhN1JwXjiFhw4kHGVcVFpVbqjVhwElxU0k0jrUtyrlQLzntP9oz0ataXbDehWpviAifMosrK1r6yf//WLYmlvI0DZne17PrjiX1ndqW2xxgtxp0zkIgWADgZwM+0Xa8kou1EtIKI3p9y/vm+G2z5li1bWjpWhmGaYxzom0xWEhnzIa0kdkaBU3PdmMsHiAYWJ7uosokv7z7mSsZA6KJRXTVShLguIpWIveOSXVS668YLTA7PrzhucIxq3dGDjGULCzmOsJWD978UGF3FHF60aGrwjPJ/+VpvtqkiF9ZGLDhSrJ118JzM5zDjn/FoK3srgL8LIVYp264FcDmATQCOAfAbItophLhGP1kIcbl/LJYtWzYO/owyDKOTNcakpWNowkVlWQAyhOPoDSElWVo1NBqDkxQAbcqikhaSmutGgoTlfVWiQcDRfa7WbBMIRYsqRKTFS1pwBioO8jYF7rmgGKEVdREJw3gcpZ9WmgWnFLi/sltwrnjbMrQVbOw9bezdKszIMV4FzpfVDUIItUb6PUT0bQDnAogJHIZhxj9ZC9m1kta6qNwEC0L9ZptEww8yBuLiAVBcVAaBogsCdXwyIFie4grEBFJRCWB2g0J/Mh4mLOqnWpRszUUlj/NaTETnz0sTzxBkLLOoGrDgnHbgrMzHMrsP40rgENEJAOYC+HWdQwWSLbwMw4xzxoMFJ4uLSk8Tz+6iimclAZqLaoRaNeiuJknOjlpPACXI2I330YpVMlYbWhJ5sUEijIfRzw/cX0oKutQoduCiciKxMaGw8f7vLObwvlP2wb8dNhc9g9XI9X9+75pAzOVTKhnLDtiNWHCYiclop4nniKgEwAZgE1GJiFSR9TYAvxFC9GrnvYqIppDHiwB8EMDvR2/kDMOMJONB4GQJ5JVWCpkmri+mSVT9Qn860cwp87VkNlIWXCG8CrwGa0UgHpIsOJoFRq94rD6qZ8EJNxhdVL54UgOYpSCUczFYdSLxPHoMDhHhwrOW4KC53TGB8tyWflxy08rIs3nXTgoyHnchpswoM9o/AZ8BMAjgQgDn+a8/AwC+8HkdgJ8aznsDgGcA9MILPr5UCGE6jmGY3YDx4KKqZVA4oYvKC47NaMDxXVSGGBzldZJW8iwl2e5Tc7y2CKZ75SyDBcd3I8lKxKo40C04qhiziCLP7rrx+ZMi6//uex595VrkmnJ8Q1UnKk5kppdBDOrjiT5beLzu6is1EWTMTExG1UUlhLgIwEUJ+4YATE7Y98aWDYphmFFnd7DgCMVKIZttZnV61BLSxLN0E/diXbK3anCEgEWEtrwdaSkhhY1qCZEZRjc+tgEVx0Wb37dJP07HtqJjd7Q0cSCMwfn5vWvw15Wb/fNMQcZx64vp3qYgbYlapXgk0sSZiQlLXIZhRh2Rsc5LK6kXgxOp8+JbcrLKsmpCDE4ki2okXFSul3JvWYSZ3cXIvkLgogr/zHcUvcX/kptWwnFFpO5LmsXEIopYnNQ4G4nagmDdzsHINfMJMThhEHLzFpwkF1UjlYyZiQn/BDAMM+qMBwtOPRFRdeIqLGuPqJrj1s3iSXRRNZAmXnNduELAImBGZ1Tg5Axp4vOnRNOg1QJ+aRYTi7QYnJQgY/08IBQrg5Wa2UVltOAkz596DT0uKqiDMw6aPTJjC/8EMAwz6owHgVNvDCaBo7tlJHpl5pqTkEWlLMZJrRoayaKSlhSLwoaeEj1FGwAmt+UjxxQjFpzk5cC2tBgcEZ+LgqHNgZ7JNajH4EgXVYMxOBEXlTaP0zqKKOSsmOBj9jzGVZo4wzB7BuMhyLi+BSe+f8jQuRvwnkfVM1XXjQT3SrJEhaj1ZsKxmOvq1BwBV3hiaUZXdEEPCv1ZydaOUmYLTvRcNVNKYrLg6M02XQGji6rRGBy1n5MuhKZ1FnDXJ0/F9A4WOHs6bMFhGGbU2R16UZksOEnD1q1BNUfEqv8C0RicpEJ/UkhId9i/1vVgv0/fhDue3Gy8rxACtgUcvtfkyD6Ti0oXDaoFJy3IWE8TFyJef8foojJYaEwuKpM7L82C01FMFjg5izCzq5Q5pZ+ZuLDAYRhm1MnaLXskuOy2p/Heny+Pj6EJgZOE/jg1x2zByZJFJa0eco4eemEnAOAvj2+KHeu48GNwCK86fC6ufvcxsc7YUlDYFqVacNIEhU3RkGhTHR1TJ24pqFSrjd7XCgDmKh3Xg3NTBFdHQR23fh4va4wH/yQwDDPqjKaL6slNvVi5sTe2vZEsqnroFpxqQm2a+mX+QquHFGDSMmJqqllzXTiurFNDOGHf6YFQkRlSUkgdPG9SLF5FzaJKC+q1LIrEDDlucqG/yHlakDEQteBs3DUEAFhg6AGVJLhKeSsiYmJd0Nlww/hwDA7DMKNO1mykkUAIYQworjeGYVtwTIJBqw5sQi7s8pqy+rAp/ke6qEy3khaVmV0lfPXcQ3HakpkxoVfMaMHR08SFiLd6SHMzqaJGFX5rd3jp5Iumd8TOTRJccj6Ce2iiLWuANjPxYYHDMMyoM5ohOI4rjEX9RtJFZYrBMfeiUlxUCdeSQkK3MJX9In5q/JLqolLvDURjYl63bC//2n2Ra0YtOA0U+stowdFbMXivw+Mqfm+pBdPiAidJcOnj1HUQ6xtGwi4qhmFGndFME3eF+X4j6aLSj6y65qwnSnwTIoWEFGBS2MhGk6owc1wXroiKj5obFzgSXXSp1pB6Fhx1vytEIE4kphgc2xBkbLL0TO8sxMeaMB49OFsd13tPXox9ZsTFErNnwhYchmFGnVEVOIaUZm97+nnDtuAYs6hUC066i0q60KSwkTE4jmbBcYQwBizrrhwgLg5UEZSaRaX3ohIiqO4saTPcT08T11//4M1H4slNvcaaQFktOOr7T738wMRnYPY8WOAwDDPqjKYbwRXCeL/6FpwGYnCUQ2UPK3MWlfm1SiBwhOyDJS043v+qMHNc10sTNwiENJeRRBVB5u7nnjvR1tLEf3n/CzGh0VmKLycyzigpyPisQ+bgrEPmxM7zxpOeRh+8z9oBldnjYIHDMMyoM5p1cJwkF1UdE46eBp2GUJxU0kVkrIOjWG2S1mXS0sQDC47/vxux4HjxRaZF3ugySrPg1HFRqafe8eSW2DFdxfhykjMGGWeLjEiy4Ojb01xrzJ4Nx+AwDDPqqNaTVoudpCyqegYa3QWjctkbj8CP3roseK9aiKQwMtZjIfVlwgJO0kXlvZfCRnYKj7ioRLKLylx4TzsmYwyObsExYbLgmLKoTHVwTBCRUXSxwGGywgKHYZhRx424dFp7L1NbAbk9Dd1Fpa7vNhHOOGgWvviqpQCiVhVZ4ddUByebi8ofnxDY3DuEtTsGAAC7BqvevbQgY6FlUUnMhfcs7ZjwfafBAiOvapHZ4qSKC9P5ah0cWZzPNC9JmFLpdSsUCxwmCXZRMQwz6qiCwBUCVqYuTc3fyySi6gU6S0tMIWehUnPRlrcxUPGsKFKESHeSeqmqb3GpF2Sc5KOyAguOwIsu+WuwfddQDUIILYvK74NluFfJGIOjHxOKoCkd8Uwm8oNw9FYNkra8jb5yDYBZ4Khz0FnKob/ipBYUNJ1fiT2DJnA4BodJgC04DMOMOqogaHU0jusmxeCk31m6qKSVQ80SkkJFrrWqm03G4NRrtplcBycaZKyOd7DqRFxUru9+MxkxTBYcXaR0+93F33XiooTReNhaoT+JKpDaCob7KSfJ+cvqogKAdsM12UXFZKUpgUNEbUR0OhEtaPC8C4hoORGViegqZftCIhJE1Kd8fVbZT0R0KRFt87++Sqa8QoZhdgt0C06r72VME8+YRSWFgrqAS6uBjKNRLy/PM7uoyPhaRS7YSW41ocX7OK4wpllnyaJaNK0Dt3zkZHzmbHN6deiiSrDgFJSWCXX+JEsx1IiLqsNgFdKbirLAYZLI5KLyxcg/hRDfJ6ICgH8CWAqgQkT/LoS4KeP91gO4GMCZAOLd1YDJQoiaYfv5AF4N4DB4H/huAfAcgB9mvC/DMOMINxJk3Np7OQkuqnoWHL0icJshIDew4KhZVP55xkJ/9T1UgdXjJd/4W2yfK7RCf/6zmRZ5Y5CxdtOcTdhvVpd5IJExwSii1Dmp1707FDiNWHCiS9TvPnACDpwTHS93DWeSyPqTdiaAe/3X/wagC8BsABf5X5kQQlwvhPgdgG2ZR+jxNgDfEEKsFUKsA/ANAG9v8BoMw4wTVG3RaoHjusJY80ZvNaATWHB8S4hqwZFrfehOUq/rx+DUWciTluVUS4gucFw30UVlKvSnF/OrZ02RQ0lyUbUpAiQtzdwbj+XfM7vA0VPPD99rcsz1Vu++zJ5L1p+0KQA2+69fBuA3QojNAH4J4KARHM8aIlpLRFcS0XRl+1IAjyjvH/G3xSCi83032PItW+K1GhiGGXvEqLqozPeo12yzormojG0NKH4t2eLBVAcn4qJKbEWQPCYBodXBQbKLymDBiWcg1RNhFIzVHGSsdvXOZsExtWpIoqMYF2k6XOiPSSLrT9pGAAcTkQ3PmnOrv70TQHUExrEVwNEAFgA4Cp6F6GplfyeAHuV9D4BOUxyOEOJyIcQyIcSyGTNmjMDQGIYZaSIWnBbfS8at6PV26lUy1l1UqjXBDoKM44trWh2cLGtxmstFd1HJDLHMQcYpbQ5Sx0RkbMUQcVHVi8Hxx5PWEkKn3RCDo8MxOEwSWQXOTwD8CsC/ADgAbvO3HwNg5XAHIYToE0IsF0LUhBCbAFwA4KVE1O0f0gegWzmlG0CfGM1yqAzDjBj6It1K5PV1g009C07VcWFR6MbpUgrZWVoMjvoMFcdLJTdZUCJZVEl1cFKEgtDaTtQcz6KTNQZHv3YjVYX/5w2H47xj945sjwReZ3RRNdKmQ4rKsw+dg0vPOcQ8NrbgMAlk+ukWQnwRwDsBXA7gRCGELE1QA3BpC8YlfwXkT+4KeAHGksP8bQzD7Iaon01E9pZPTSHFhx5UnCVNXLXCyJRqILRWmGJwZK0cY9p0A802TQjEM9DchEJ/JkuQvq2u9UOJwZk7uQ0XnhXNtipFLDjpl5LzIXtrZUFmUR0+fzJef/TexmM4yJhJInOhPyHEbwzbftrIzYgo59/TBmATUQmeSDoKwE4AT8OL97kMwB1CCOmW+hmAjxLRn+D9jn8MwHcauTfDMOOHqIuq9TE43v+6iyr9vGpNoGBbwfmqBScs9Be/9qAUOAaXjqpDktblNIOEnvLuuAKOMGc4mdBdUnWDjLUx6dYSU22gJKQYakjg+KKov2JKrmWYdBIFDhG9NetFhBA/y3joZwB8Xnl/HoAvAHgSwJcBzASwC14a+BuV4/4XwGIAj/nvf+xvYxhmFHFdgd89vA6vOnzesGIfolaIkRhZyr1c6aISxu1J1FwXeZuC8zqLcQuOqZKx7BllymJSRUCiiyo9yjgicGqu8LuJpz5K4rWTrEhJ5+ljNhXiS0LOh+ytlQVpwekvs8BhGifNgvM97X0BQB6A/Om04AUYl+FZWOoihLgIyWnl16ScJwB8wv9iGGaMuHb5C7jw+sewvb+Cd5+0uOnrROvgjE0MjpomLkQ8E6nqu6ikEFIr8IYuqvB8ibRQmFxUgCcShKjfbNOEQFRMuW6yi8qEflzSGNWxAqHA0QWSyUoFAK9bNh93PxOtBvLKQ+fiB3c8i5csmZlprEAocPrqCBzbIrz/lH0yX5fZM0gUOEKIoJoSEZ0NT5h8GMB9/uZjAHwTwJdaNzyGYcYT2we88LstfeVhXSeySI9CoT8gHnOjFxvUNULFd1HJ81V3TmDRMFQyli6q9oTFn+AJlXqF/ky4IlrTp+YKuG52F5UqUFZfcnbd44M0ca3uj6SUIJC+eu5hsW0Hze3OdE+VYxdPAwCcfuCs1OOe/fLLG7ous2eQNQbn6wDeKYT4h7LtbiL6MICrANwwwuNiGGYcElgdhilKIuIi5WJDVQe9QzXM6Co2fy/f5hxLE1ctOIbzAheVf77aJDJmwVGuMFj1Tki24HgmnCRRkhpkbEgT1wv97TW1DS9sHzSeL48796j5ifcwnxd9XokUcW980V7BtlMPGLnyHPvO7MSqr7w8s4BjGJWsAmchgH7D9gEA5tB2hmEmHHKBK9dcDFUdY5xJFrJWMn7LFffh/tU7Gv7kH71X/SwqT/wkuKj889UA3TAmxbfgKGElMgbHlKYNeHPoxO4W3a/y1XMPheMKfOr6x2JZVI4bTxO/9aOnJM4pEeFfXzgz0bWUhP68klLejlxvxRfOTHzuZmFxwzRL1p/E+wBcRkTz5Ab/9bcQtnBgGGaCIz/JX3XPaiz57M1NXydrHZz7V+9o+h769XVXmGzFAJgtOJWaQF4VOIqLKihkbMiiGqo6aMvbiQuz7vbR0d1AU9sLgbhy3WgWVdVx4YroOcWcnSo8O4u5zAHieksKnULOilyvo5ir26KCYUaLrD+J7wYwDcBqIlpNRKsBrIaX9fSe1gyNYZjxxkh9mBZa/EsrkTpGF1IRgWMYQ811UbApEEbqwh0W+otPyGDFSQ/eDcRRNhdVMW9FjlWfY1tfBa4rRuz7oiMvmxQXxFWEmfFMVoGzHsCRAM6GF1j8LQAvB3CIEOKZFo2NYZhxxki5C1RrSqsrGYvAgqMLnPQ4ID2LSu0tZWsxKa/4zl34v/ueB+C5qNJcQLr1J75fEzg5W8nWCt1hcyaVsKl3qKEsqmbhasHM7khdgeP3n+oBsJ8Q4i9CiMuEEN8WQtzCrRIYZs9ipJY5PYOplSRlUVXqWHCqNRGpg6NacEx1Yb55y1MAPIFTyif/aZUuqiSxqIuVQs6KuMLk88yZVMKmXUNwElo1jCR1enIyzLik7o+tEMIBsAZeHRyGYfZgRuqDfKMCp15Rvizn6vep1ik4V3VdPwbHe68GGZPRzeQdWM9FFZybsD/mospZgSgSCJ9nzuQ2DFVd9AxUW+eiSmkqCrRenDLMcMiqy78E4BIimt7KwTAMM74ZqXVUNOiiqtf5Ow0pUHQLTr0YnKrjCRx5Xs6mQOTYhhgceY3BSj0XlbkqsESP0VUtOEJp1TB3UgkAsGuo1jIXlbxqkouq1W02GGY4ZE0T/y8AiwCsI6K10FLGhRCHjvTAGIYZf4xUY8NoHZz6OK5AkxnpShZVYzE4Q1UXxZwVxPDkLAvFnIVaxQkWfHU25PUHq06kMadOELib0UVVzIVBxq4I7zN7UltwTKtjZNiCw+yOZBU4v27pKBiGaZgNPYO47YnNOO/YBaN2z5GLwVFf118lhxOILC0e+jXqxeBIV5NaybiUt9FfcYKJMC38Q1UHs7pTChPWcVGZgowVO1HwHNKCA7QwRoa0/xWOXTwVpx+UXmGYYcaSTAJHCPGFVg+EYZjGeMeV92Plxl68dOkszOwq1T9hBBi5LKrGYnBqw4jBkddvtA7OYNVBe8EO0sxzthUUsZPWH9WgJZTzhueiigcZW4oFR45nulLdueXF8AwT9Mvzj2vtPRlmmHBsPMPspmzv9/pCudmbMw+bEQsyjlURzn58oziJLirVghO//kClhra8rbioCP915gEAgGkdXs5FpD6N22iQcYKLyhRkrKSJy+cpqHV5WqRvLjxrCQCgvdikf5BhxpBMFhwiKgD4NIA3wmvNEHEwCyH4p59hxojRLFGStCg3SqRVQ4bj9QDhxu5lThOv1pLjgFxXYKjqoq2Qi1Qyfs2R8/GaI8M+TpRgwUmrJFzXgqOnidtWMOtPb+7Fdr/RqdrdvFUxOG8+ZgHefMzouUAZZiTJGoPzJQCvB/AVeEX+Pg6vP9UbAHy2JSNjGCaVsYjvbEWaeKYsqiYFjhAicFHpt0mLwRmq+R3BC3aYRWUIdLE0hVNzXPSXa+gsJv9pDcJaEoOMtfcWBcde8H8PBdtVgcP9mhgmTlYX1esAvE8I8b/w+sT9XgjxQQCfB3BGqwbHMEx9RnNpGylXiOoSyuJiazZNXNVFsUJ/ah0c7fIDlVDgyFvn7fjD6zE463YOwhXAXlPbE8dUrw6OFCuT2vJY+aWXRc5RUeN8Wl3JWGfe5Lb6BzHMGJPVgjMLwOP+6z4Ak/3XNwO4dITHxDBMBsYiRbc1LqrWWXDSmnpGg4yj+wZ9gVPKh1lUpiaSEQOOEFi11augsWh6R+KYguJ5CR8v5TgLOStwdemz3l6wMbu7hLxNqDoiVjunlTz8uTMi1iOGGa9k/Sl9HsBc//UzAM70Xx8HYDDrzYjoAiJaTkRlIrpK2X4sEd1CRNuJaAsRXUdEc5T9FxFRlYj6lK/FWe/LMMzIMFaVjJsVOGmusLRCf4PV0ILjKkHGOqprSABYs20AALBgWooFJ/jfPJl5X60cNn9SsE230Ow9tR2WRcaCg61mcnsB7YWsn40ZZuzIKnB+C+A0//W3AXyBiFYBuArAjxu433oAFwP4ibZ9CoDL4cX1LADQC+BK7ZhfCSE6la/nGrgvwzAJnH3Z3/HRax/OdGwrmm2OnsCJ7osW+ouiuqhMrRokeiXjVVv70VGwMaMzuQ5O2ObBvH9qRwG/ft9x+PYbjoidI5GWnbxvBuIYHIaJk7UOzqeU178mohcAnADgKSHEDVlvJoS4HgCIaBmA+cr2m9TjiOi7AP6W9boMs2cilH+bY9OuIaxYvwsr1u/CN193eN3js8TgvO/nD6CzlMPXX3tY4jGiwSDjZgv9qcIovdlmdN9ApQbAExJu0KrB4KJSXgsIrNnWj72ndaQKjjCLKvmYZQunRu+jHSpdYLYd7WrOMExIU45UIcR9QohvNiJuGuRkACu0ba/0XVgriOj9SScS0fm+G2z5li1bWjQ8hhk/DCcW58ZHNwAAZnalVN5VyGIouHnFRvz6gbWpx6TFxpiPr39fE2kVk6uOG9aX0c4bClxUYZq4Ocg4asHpGawGNXKSSCkOnHyOcp+8Tbjo35YCCDO7Wt1NnGF2RzIJHCJ6ioj+l4jeqMbGtAIiOhTA5+ClokuuBXAggBkA3gPgc0T0RtP5QojLhRDLhBDLZsyY0cqhMsyYElbobV7hbNw1BABYmBIUa7rncGm0Dk6tyWqGaoFA/RLVmhsUy9Ofy+yiqhNk7J+XVgPHOye9Do7xHOX1O05YhEl+ryvpNmMXFcPEyWrB+RqADgBfBbCWiJ5sheAhon0B3ATgQ0KIv8vtQojHhRDrhRCOEOIeeHFA547UfRlmd2Y4mkMKgKyVgodRb0+7TqOVjId/H1OzTZkNpGdRSYGjpmKbg4yVN8Kz/LSnVDFWz2kkMFgVMOpZYZBx5ksxzB5D1hicHwH4ERCIkBfDq3/zU3giadgh9US0AMCtAL4khPh5vSFhdMt/MMy4Qy7JrWhjkHjPETLhiEazqJqNwVHOczRRVXFcdLfl0AvEVKJME28r2Pi/9xyD3zywzthJXRUprhB1+1AB9evgmFBvHRE7TYglhtlTyCxMiMgCcDQ8cfMSeEHG6wDc0cA1cv49bQA2EZUA1ODV2fkrgO8JIX5oOO9VAO4EsNMfwwcB/L+s92UYxkxSI8p6xw+XaGxM/eOdJk046nhVUSWbdwYuKu08NU38+H2m4/h9phuvrwuLgTp9qIAwPbwxF1V4sCp25P05Bodh4mTtRXUjgBMBbIOX3XQNgPOFEGsavN9n4FU/lpwH4Avw/r4sBvB5Igr2CyE6/ZdvgJdaXgSwFsClQoifNnhvhplQiAatLyZksG9Wy0yWonxZyNKqQR1Ts0HG0SyqcLusgRO4qBJicEq5bNYYwPsjNlStL3BCLdKIi8r82rYaF0sMs6eQ1YJzBjzryU0AbgdwhxBia6M3E0JcBOCihN1fSDnPGFDMMMzwrCpuIJKyHt/8vZKukzR+tU7NSBf6k402k2JwBis1lPKW0S2lou52XAEHyOCiajxuhgxWG3U7u6gYJk7WIONJAN4EYAeAD8MLNH6MiC4jon9v1eAYhqnPcCw4Ujdkj8Fp+lbadeoHGauZU00LHFd9HV6jUseCM1R16woVwJy9VDfIOOXc5HNUUaO6q3wXFQschomRNch4EF4A8K1AEGj8aQDvB/ABeDE1DMOMASOSRZXZgjPyLqqkK0YsOE032zRXMg5cVAkxOFXHDVompGGSFfXTxJPPrXeOfp60ArG+YZg4WWNwZsILLj7V/39/AJsB/Aaey4phmBHEcQWEEMbquRK5KA8ns0kKgOwxOOlEOnSnoMbDJImmmnJQs5liSVlUcpyhBSeeQp5F4JhcQ5ldVA2UWbUMVhv1NbuoGCZO1hicjf7XnfBq0NwhhFjZslExzB7O2Zf9HSs39mL1JWfXPXZ4MTje/1ldQLoQEEIEC/ZNj23A+69+sOHrJI2/poyp1qTASXKFhUHGtnEMNddFzlC5WMckLDK7qEYgyNhqIp6HYfYUsgqcg1jQMMzosXJjb91jGk3xNhEGGTcXg+MKQOqAW57Y1PB90+6tdvtuNgZHtRQ5hhicYs5sRqk5wljYT8dkOCnVzaJq3EdlcksBoRWI08QZJk4mI6kUN0S0jIheT0Qd/vsOv7YNwzBjxHBSt0MXVWPHm97r1ow0UZIli6rm1BdB9UiOwdGyqLTLZ47BMeiK9owxOE1XMja4qLhVA8PEyRqDMwvAH+AV2RMA9gPwHIBvAhgC8KFWDZBhGDNBHZwma8QAw8+iUs/TM3mqjgvbMi/2qvhJEmhqFlWzLqpIU0/ltbQOFW1zmnjNFU27qOrVwZE0HWRsqGrMMTgMEydrmNu34MXgTAMwoGy/DsBLR3pQDMNkZ1gWnGFmUcm3X//zk/j9I+si+yop1fkilpWEw9QsqmaDjNXhRuvgpKeJVx3X2FxTp5kg49DqUvfyxvtEX3v/ZzA2McweR1b30mkAThNC7NBMoc8C2HvER8UwTF3CLKrmr9FoDE7S+d+9/ZnYvmpKRlXUglP/mKZjcBJcVNIiJK00+tVrjkA+gwXHJFLqtmpoxkWlvDa1amAXFcPEyar72wBUDNtnwHNRMQwz2vir8kgInKzXMAUZJ6FaYJLuq7+Onq8EGY9ADI5juKd0qwkhcN9z2/CHR9YD8LOoMlhwjAKn5XVwwjc2u6gYJpGsAudOAG9X3gsisgF8EsBtIz0ohmGyMxwXldQQWS04aUHGOtUUF1XEgpNYyXj4Fhw34T6BwLHCQn+vv/xefPCahwB44qxVMTjNZFGZ2jOorzmJimHiZHVRfQLA34joaHgNL78BYCm8Fg4ntGhsDMNkYDhp4o027NSPEikBzukCRx1D/fOb70Wl3jMe9yNjV0x1cJquZFyvQaf/f7NWF1M8DqeJM0ycrGnijwM4BMA9AP4CoAQvwPgIIcSzrRsewzD1GE4lY+m2ydqtuzELTlYXlfmYkUgTj4gaQ8CxbKZ519NbYvfOUgdHFylZGnRKs8tIZFFJLxrH4DBMnLoWHCLKA7gLwFuFEJ9v/ZAYhsmCXK+HV+jPv9YIpInrZHZRZUkTTxFLaUTcUm5cMEkRc9EfH4+cl7UOji5w2gv1jeJh/6jsoqR+q4bMl2KYPYa6v8FCiCqARRheTz+GYVpG87+aDbuoYhac5GPT0sSTsptUqiNhwUkIZpb3TOrCnTUGRzfDZOpALv9vpJIx18FhmIbJGmT8UwDvaeVAGIZpjFCcNH8Np8E6OLrOSLP8pKWJu64I4kYSg4zVbuIjEYMj4tezEzKlapnr4ETfl/JZMq8at7qomVPRSsbyfxY4DKOTNci4A8CbiegMAA8A6Fd3CiE+ONIDYxgmG6NZB0fXGc2miTuuF+PidU03H1NxnOB1s5WMo1lUyvYgi8p8XtXNVgdnOC6qRqJworVvwtfSAsX6hmHiZLXgHAjgQQA7ACyGF3Asvw7OejMiuoCIlhNRmYiu0vadRkQriWiAiG4nogXKPiKiS4lom//1VeKoOmYPJ4zBGUYl4wZr6ejxMmkp6mkxOK4QQYxL0vjL1fD8ZisZuwarDRA+b1JAcK3JXlTZXFSNi5KkOjj8Z5BhkslkwRFCnDpC91sP4GIAZ8IrHggAIKLpAK4H8G4AfwTwJQC/AnCsf8j5AF4N4DB4f9dvgdcL64cjNC6G2W0ZlgXHbZ0FJzUGJ+KiMh9Trg2/0J9jCCxWtydlStWarIOTqQ9VU26luFtKfT2cnwGGmaiMagcTIcT1QojfAdim7XoNgBVCiOuEEEMALgJwGBEt8fe/DcA3hBBrhRDr4NXhefvojJphxicisL4Mx4LTmMDRV9I0y0pa5pMjELiAEi04tdBF1XwMTnoWVZLIqGatg9OEBSfIoqp7ZPwc73U8i2o4VjyGmaiMlxZtSwE8It8IIfrh9blaatrvv14KA0R0vu8GW75lyxbTIQwzoRjO0hZ0E89cB0e7d2oMTsYg44RjVBfVSAQZq6/luJMK5DVbByeLBac5F1XERxUgx88Ch2HijBeB0wmgR9vWA6ArYX8PgE5THI4Q4nIhxDIhxLIZM2a0ZLAMM54YXgxOo5WMsxf6q+eikllKSRaocs2FbREKOWtEXFSRLCqtDo6KEAI1VyDXRCXjTAJn2M02420bhpNJxzATlaxZVK2mD0C3tq0bQG/C/m4AfWI4tnmGmSCMbRZV84X+coGLynxMueagmPNEhtNkob+kpp56JWMVmbGVb8aCk8lF1XhgcETUGLbzn0KGiZP4EYWI/kpEk/3XbyWiYgvHsQJeALG8dweAffztsf3+6xVgmD0YaU0ZlgUnaLaZ8Z6NpImn1MFxROgCSgsyLuYs2EQj0k08EoMj6+AYxIaMHcpkwdFOb2/AgtNsFpVankdqsGZdeAwzkUn7DT4BQLv/+kp4jTWHBRHliKgEwAZgE1GJiHIAfgvgYCI6x9//OQCPCiFW+qf+DMBHiWgeEc0F8DEAVw13PAwz3kn7ZB4EGQ/j+qoAyGIF0I9JLfRXpw5OljTxQs7r7dR0mriisUzxOKYYnKp/UpY6OLqXvJTBgiMZyWabrG8YJk6ai2olgC8T0e3wrKKvI6JdpgOFED/LeL/PAPi88v48AF8QQlxEROcA+C6AXwC4D8AblOP+F179ncf89z/2tzHMhMYVQL11diSyqDLfK3Z+8rFpMTiuCF1UiUHGNQfFnI2aI5qPwUmogxMW+kux4DTR4KkRF1Uj+iapXg+xi4phEkkTOO8H8G0Ar4L3N+gSmP8WCXgWlroIIS6ClwJu2ncrgCUJ+wSAT/hfDDOh0dOZ7YSEYnnU8GJw1NfJ9wru2VA38fQYHGnBSQsyLuYsDFapaReMSLBQha0awuc9ab/puPe5baj5487iotJpyEXVQKJ4UpCxHD5nUTFMnESBI4S4B8DRAEBELoDFQojNozUwhtlTSWoQmcRIFPrLei9TkHGSQElzKzmuQFs+QwxO3oI91LzAUTWWExE73v+qwLH91hFVGWScpdmmRrY0cf//ZmNwKDpmgF1UDGMi60eURQC4qAzDjAIRV0pafRpZw2aEXFRZLhNvtpl8XppbSXVRpRX6K+ZsX3jUH1vSfcLX8e1qkHHOsuAKhBacDM02dRpxUTXiATOlhgPAUQumAAD2ndmZ/WIMs4eQtVXDGiKaRUQfAHAQvD+tjwP4vhBiUysHyDB7GrUGrSrD+fCeFKOShD4eV4jEMaaJkmiQsfmYisyisghO1kqEhmsAnjVGtSjJ51YtONJiUw1cVC2y4DTRbDPqogpfn3vUfBy7eBr2mtoeO4dh9nQyfUQhohMAPAPgTQAGAQwBeDOAp4nouNYNj2H2PNSaL9lcVMNPE896r9j5wixQiNKv5woovajSY3Bsi9BkGRys2zmIUt7C1I6C0Vqlu6jkfQFkatWgM6ktn+GoJjqAk/kNEbG4YZgEshb6+zqAawC8TwjhAgARWfCaXX4DwPGtGR7D7Hk4CW6VJIYTgyMavJcuWkSCBaeYs9J7UUUqGZuPKVfdwEXVbJr46q39WDitA71DtaiLyo0X+stpAqfRLKor33E0DplXv5qGvGwjaeKmwGKGYdLJ+hHlcHjNLoPPe/7rbwI4ogXjYpg9lppiVkmtgxMU+mv+XtHA28YDmt2EGJyCbeG3D63Fi/77VqPryxM4Mk08JQYn7xX6qzXpolq9zRM4RDC6qHIRC47357DSpAXn1ANmxurimAizqLKTlEXFMEwyWX+De+AFGussArBzxEbDMHsQ1y1/Af913SOx7dF6LfWvkyQQspDUjDL5+GwWHNsi7BioYnNvGUNVJ7Y/GmRsvpd0UVlNBhk7rsAL2wexYHq7ZwUyWKssiltwmhU4WRlus03WNwyTjay/wb8EcAURvZmIFhHRQiI6D8CP4LmuGIZpkI//+lH8+oG1se2RBpEpqkOu18Ox4ERdVI0HNLvCHORs18lAylTJuOa5qHJNBhmv3zmIiuNi0bQOWBSN43FdAYuilhHbjgqcZoKMsyCnpjEXlfqaFQ7DZCFrDM4n4P0t+IlyThXADwBc2IJxMcweiypqmmmf0Oi9ZP2XZgKaTVlUHz1jf1x935rwHMN1XCGUIGPzvcpVr9lm3qZIZllWegarAICpHQVYWtCzK4QnFBStEFhwnOytGpqhkQJ/pnNY3zBMNjJZcIQQFSHEhwBMgRePcwSAqUKIjwghKi0cH8PscdQyuqhGqpJxPaEhuf3Jzbj7mW3a+QJCMa4cPK8bHzxtv0h9maQYHCkg0rKoCjkLxZyNcrVxC07VCV1NFpFW1NALMFaFg625qJqpg5OFZpptqpooS5wPwzDZLTgAACHEAMJ+UAzDtABTz6Q0hheD4wX7VlC/Ds47rrw/fm9hHqOanWQSMGoWlSm+pua4qLkCxZyNYt7Cjv7GP0dJoZizyRCD47uoTBacFruoKCj016yLaqRHxDATk9Z8RGEYpmmyChwpHJpMMPLPDbOZUuvWJIgf3UVlqi9jEk6uCIN4TfE10k1UzFso5iwMDcOCk7MsEEUDlV1XwKaos0jGDZWdVgcZ+/83G2TchIuLYfZEWOAwzDgjGoNT//jhVDJWhUbavTbsGko836R9VIFj2u/F/iAxvka6pIo5C6W8jXItnolVj6AruE2wrXjNH4soIhxiFpwWmUrkZZtvtjmy42GYiQoLHIYZA9ICg9UFv5n2CY2gBvumXWfN1v7E803PosbgmK7rCAHL8lxHJoEjLTheDI4VFN9rBDl3OYv8LKqoZYw0F5WcB5nWXsi1KgZnuL2oWOEwTBZY4DDMGDBQCS0SukBQXTapLqrYi8ZxRf2eUACwetuAeQxCRM4zuahMzyBdRHnLXPFYrUVTzNlNCZxYkHGk5o8n7FSpEFQylgKnxS6qRjxNqqZhfcMw2cgcZExEBQAHA5gJTRgJIf40wuNimAlN71AteO0KQI1njcSKjEKhvywWnLU7QoEzb3IbLn71wXjHVffDdROCjOtlUfkCw7bNVYqlOCnYMganCReVEmRsaZWMZZp4xILjfxOGaqH1qBVIC0xDLiqug8MwDZNJ4BDRGQB+Dk/c6AgA9VvoMgwT0DtUDV7LWjSSzK0aRqDQn1pROO1eqgUlbxNmdBWD800CR81A0ncLISD8GJhcgotKbsvbMgZneEHGehaV40qhoXQTl0HGvphqWZBxE2niqhjiGByGyUbW3+DvAbgBXmuGdgBtyteItLIloj7tyyGi7/j7FhKR0PZ/diTuyzBjwa6IBUd3UTXYqqFJgSOFRphFlXysOkYiCqwIei8q+TLNgiPf2xYhZ1mR7ukSNVW7mLPguAK1Bvs1BEHGFvlZVNECiraVFIPTYguO/38jlhh2UTFM42R1Uc0B8GUhxJq6RzaJEKJTviaiDgCbAFynHTZZCFEDw+zm6BYclUbr4DQbZCxvI9Ojs6aJE4XtBoDkXlRJ45PBvrYfZFw1uKikBadgWyjmfctKzUWuAauKo7iobE3gOK7volKOl1YnmbHVuiwq6aJq/ByAg4wZJitZ/1rcAOD4Vg5E41wAmwH8fRTvyTCjhhqD46RYcLJkUTXroXICNxDVvZc6RgISLTiSaBZVdJ/UMxYR8jYZ7xu4l2xCKe95wBuNw5HCKW9bsCxoWVTxNHHVglPIWS0TEs25qMyvGYZJJqsF530AriaiowD8C14fqgAhxM9GeFxvA/AzEQ8KWENEAsAtAD4uhNiqn0hE5wM4HwD23nvvER4Ww4wMakyJXkSvFnGl1L9Ws72opGVFWirSLqN6hyyiIA4kKQbHZMH584qNeHpTL95+wiL/GO84UxZVNZJFFVpwGkF1UXlZVFEXlWXBnEVVc1qWQQWoAoeDjBmmlWQVOGcCOA3AywEMIPqhUQAYMYFDRHsDOAXAu5TNWwEcDeBhANPgxQRd7Y8rghDicgCXA8CyZcuGUwONYVqGmgo+XBdV8zE43v+5DC4qEYnBCRfnbIX+vANuemwD7l+9A285biEAacGxzFlUSpBxMedZcBoVOKEVKN6LyjFlUfnzIC04rSLIomqykjELHIbJRtbf4q8D+C6ALiFEpxCiS/nqHuExvRXAXUKIVXKDEKJPCLFcCFETQmwCcAGAlxLRSN+bYUYF1SKS5qJqZQyOvK+MPUnzhqljspQgY6EV+pOv1V5U8lxHeK9dJci4vgWHFAtOYy6qIE3c8tPEI3Vw4LdqiFcyLteclnUSB5RWDc2ez/qGYTKRVeBMBvBDIYS5nOnI8lYAP61zjPxTxb/qzG5JpJifq+8bnSwqVwn2Vd+b0EVY1EUVP17VB2E6u4Djf8n75mwrIU08dFGFMTiNWXAiQcYWxYRjciXjVltwvP+btcSwwGGYbGR1Uf0GwOkAnm3hWEBExwOYBy17ioiOAbATwNMApgC4DMAdQoieVo6HYVpFTXOXJO3LEl/TrAVH+HohjMHJlkWlWnCSCv3ZYZpVICxc33ojrxXWwTE12wwDoAMLTqNBxrKSsd9sM9JN3M+iUolacLILnOMWT8PC6dmrZVhNuKhM5zMMk05WgfMcgP8mopMBPIp4kPE3R2g8bwNwvRCiV9u+GMCX4RUa3AUvyPiNI3RPhhl1ItYEzYIRiRUZThW/emMIXFT1WzWo+1TLR3KQsXqu76Jy4xacTEHG+eaDjC3y3GW2LnBkqwY1eFex4HSVMhd5xzXnH9vQuEIXFVtwGKaVZP0tfieAXnip4nq6uAAwIgJHCPHehO3XALhmJO7BMOOBtFTwWoMuqubr4ESzqHShpaJamdT0apGUJm4IMnaF8ESOjMHx08TLBteTtOrkhhNk7IZ1cywrOpeyknG0QrC04LiYNs6CjFXYgsMw2cgkcIQQi1o9EIYZr/QMVFHMh7EgI4EqGOJBxtlaNYTHNDcGV7Pg6OOIHKsX+qtjwVEXYXmqE7io/GMsgm1ZqLpx15Pqoir5Fpw12xoLAaw5IhBvehaVEJ51J5p+7f1fro5WmniT54/cUBhmQsPdxBmmDod98S949ffuHtFrqu0J0urgZLPgNDcGNxaDk3ysExE40VYN9bqJR7KoIi4qIG9RRNBJapFmm56wvPjGJ3Dfc9syP5/jagKnnouKZLPNxmJwGkVajZp3UbHEYZgsZG22eVnafiHEB0dmOAwzPlm5UQ8LGx5pIqbhOjhN1jKOuajSLDhqDA6yxOAYXFS+9UY+n0UpMTiO6qIKxcbWvkqGJwuvIYWK12wz3OeIuItKPlPVEaOSRdW8i2rkxsIwE5msMTiHaO/zAJb45z84oiNimD2AaGfr6AKvtiRoZaE/3UWVvdkmEuvgLJ7RAUBr1eAbaGSQsZqenk9IE6+qWVSKa9BuYHWvOWGndKLoPAshYCekiXv3bZ3AkbdpPk2cFQ7DZCFrDM6p+jYiKgG4AtwvimEappZipRnMIHBMxfUaRXdRpdbBSUoTV4KM33XiInzkjP0BmC04UtyoQcaeBcdQyVhJ8UY+3N5Isb+q6wZVmm2iyDy5It5sUxUcxdEIMm7yfLbgMEw2mv4tFkIMAfhvAJ8eueEwzJ5BWhbVYCW5CKBE1SJf/8tTuPLuVQ2PQS/0l1oHJ9ZsU44jLPT3kiUz0Vn0PjNZCS4qIRQXlSXr4JhdVLZFsCxCMWfjwrOWAGis4abjhhYciyga2O3GWzWor0elkjFnUTFMSxnux5QZADpHYiB7Aq4r8P5fPIAH1mwf66HsUWzuHcLbr/wndg5kj9+QtKoOTUTgNGPB0d5/4Y+P44+PrMelN6/MPgYRuoEA4P7VO/CBqx80pour41DTxL0gY2+fuu7mEiw4APCeny0H4FlVcrY5BqfmiIjIeO1R8wE0Vs04kkWlxeC4wksdhyFNHMAo9aJiocIwrSTTbzERfVT7+hgRfR3ALwH8qbVDnDhs7S/jpn9txHt/zmFLI0HvUBUX/uZR9A5VU4/74R3P4Y4nt+C65WsbvkfV4D4ZCVSBs6O/ggt/8ygGK56wicbgmM83CZ//vOYh/OCO7MXGRWDB8f4M/OCOZ3HjYxuw3SAEHS3K2JQmrgoEyxCDI4XThp4h/75emrjJglNxXM895SNT9AcbsOCoQcYWwZAmbs6iAlobg6O3iGgUi31UDJOJrEHG/6m9dwFsAXAlgK+M6IgmMM2mhTJmfnrPavzy/hcwe1IJHz59/8TjdvkCaFJbPvGYJCqjIHC+9ucnsXJjL5bOm4S3HLsgEDpAstuo2eJ+0Wt4/+vuGJOo011lYZBx6C5TBUIkTVyz4ATXsLxCf+Y0cYF8Li5wGnFR1RQXlZdFFXVRFXN6DE74uqUWHAyvlg3/FWGYbHChvzGhNW6PPQ35SbZehdueQU/gdLdlL78vqTZYPTcrqtVCWiWkO2Ww6qC9YGOg4iRacEZA34TBvppFoGJ45og4EVCCjFULTniIek0p0nTdFAYZm2NwVDeX7YuhRlxU3jWkBUdvtun9/KhuIholF9Vx+0zD5t5y0+dzDA7DZKPxv/gAiCgHoCSE6Bvh8Uxo5B/6FrYX2qOQ1WZNC7LKLl/g5KzGF62qYfEdCVSrhWxVIN0insDJYaDiJFYXHhkLjozBic6LSUToRfLCOjjhzzMlWXA0F5XEspCYJl5R3EuSUt5uPMhYKfSnTpkrKxmr41EtOC10UZ203wyctN+Mps9nDxXDZCP1t5iITiOi12nbLgTQB2AnEd1MRJNbOL4JRVopfKZxZCpvPYHTO1QDAONCWo/WxeCEr4f81GfpKhqqOugsei6ZZBfV8McgL53TVkxTnEskfgUZLDiRVg1mF1VgwUlwUelWlFLebihNXK2DY1H0/m6dGJxWCpxhwwKHYTJR77f4QgDz5RsiehG8rt4/B/AJAIeB08RTeX7bQJC9I03xzdYtYaLks1pw/BicZjKiGm3wmBXVgiOtEoEFp+JZcIBkS02aBSfrz1eSi8pkJVHFgezjJF+LQOAoQcYJzTZVbIuQT0kT14VXKW815qJy9UrGisBx/WywhCyqfAtdVMOFXVQMk416v8WHAPib8v61AO4RQrxHCPFNAB8E8G+tGtxE4OSv3Y6zL7sLQPQP/K6hahAbwjSH/IRfLxBYuqhMloJ6tMqCoy7qQwYXVYdvwUmsg5MyrKxCTm/VIDFZcNRpiFpwwjFGgowNFpy4i8rLolJr40iqjoi7qHKhi2rtjoG6Qk5NEyeiyFwGLqqEOjjj2YLDAodhslHvt3gygM3K+xMA3Ky8vx/AvBEe04Rj3c5BAOGiJgB88teP4mPXPjx2g5oAyDWxrsCRLqom4mlaJXBMFhi5YA9VHXQUm7fgZHXFycNy2mJeruOicoW5F1WkDo6SmSWnUHdRCREep4tPL8Vbt+B4Aueh53fgxEtvx7XLX0h9vqrjBs9mW/E4It1ytbtYcFjeMEw26v0WbwCwDwAQURHAEQD+oezvAtB8OsAER18c5SLhugJbesvY0kDjQCaOLPFfz0UlacZFlfXajWISW3J8AxUHHb6LKknHpAmcrKKsEQtO5H5CaIX+vM1J8SyhBSd6zd6hanBvfT6qxiBjz0X12LoeAMCja3vSHi8WZByNwUE8Bke5XaGFlYyHC1twGCYb9QTOTQC+SkQvAXApgH5Ee08dCuCZFo1tt6fPtxxI1E/WVcdtWQrynkLVn880EaIu9s0EGbeyDo4eRFtzvXgWmSYOJAempz1JZhdVQgyO2ioiuKYmDgAvcNezOsVjcEz30cdVytvBvfXvTc3kosrbGKw6QdB4Vym9rpFXBydME/dq9oQfMryCewkxOOPYRcUmHIbJRr3f4s8BGAJwK4B3AniPEEI1O7wTwC0jNRgiuoOIhoioz/96Utl3GhGtJKIBIrqdiBaM1H1bRa8mcBzFRVWuuS1bPPcUHH/+0iwWapCwqaBcPVqWJi5ELM7DcQXKNRdCYFguqqxjDgv96Wni9bKoQkHjKr2oEgWOv18VSe85aRGOXTwtuLcufiqOG3FzAUDRj8HpK3u/VzLTLImq4yKvWHDUsUgXVVKzzfEscDhNnGGykfpbLITYKoQ4GcAUAFOEEL/VDnktgC+O8JguEEJ0+l8HAAARTQdwPYDPApgKYDmAX43wfUecXVoLgWBREr4FhwXOsKhlsOConaqbESutsrKZLDiOKwJxEQQZN1HoL2swdeCisuu7qCIWHCWoWO1FlbTwmoKMTz9wFoDQeqR3FK+5bkwAlvIWyjUX/YHASS/jpaaJy0uFGV0GF9UoNdscLuyiYphsZK1kbHR2CyFGq2vkawCsEEJcBwBEdBGArUS0RAiRvbvgKCMFTptfZl4NMq46IvZHnWkMKVjSLGHqvqZicFrpojJYcKS4aA9icJoIMs5swZExOFmCjMPX8upEntAyFfoTihPNVAdHCo+8bXZRVWsiJrxkkLF0/erB0To1NwwylmNzXIG87f1PhOQ08XFswWF9wzDZGI+/xV8hoq1EdDcRvdjfthTAI/IAIUQ/gGf97RGI6HwiWk5Ey7ds2TIa401EuqhkPIX6CbbquKi0yP2xpyBdTukxOOEcD7fQ3+1PbsY3b3mq4WuYqLkC+RzFtsk+VB2GnxmVtEdpOIsqY5CxFGRq3ZtoHZyE+xhicGSDT/l/liDjNl/g9PoWnHqCtaYFGXtjR/AMdkqhv3riaSxhCw7DZGO8/RZ/EsBieKnnlwP4IxHtA6ATgG5F6oGXxRVBCHG5EGKZEGLZjBnNl0MfCQKBU9QsOEKgUnNRaaAq6+7GP1dtx6/uf76hcx5b24Or7l6V+fjAgpPRRdVMDI567XdceT8uu+3phq9hwmzBcUMLThCDYz4/SfgAcXdPEkGQcRYXlSsCa0vYXDO5m7hqYJLaRR2zrJOTT0oTd5OzqPoyVqb26uCEaeLeWEJrklfoL0TVDePZRcX6hmGyMa4EjhDiPiFErxCiLIT4KYC7AbwcXmuIbu3wbgC9oz3GRuj1XVQy5ddVXFQVx21ZAOt44Jf/fB7/c2tjYuCV370LF/3x8czHy0VxoJIsFIebRdW6XlQChVw0SLbmiqDoX2D1S8qiGhELjndc3tKDjM29qKRVIxpkbC70Fx1r3EUlY2+SsqiqNWGug1Nz0FuWlanThZxaS0dtLeH976eFR9LEdxMXFadRMUwmxu9vsYeA9ydoBby2EAAAIuqAV59nxRiNKxPSglPKR1N+xR4QZFx1RdPiIGurAbkoDlRqicdUaoqLqonxNGpl+91D64IsnzQ8Cw7FtsmfiVIuXeCMRAyOvJce7Gx2UYVWDbXuTVKhPxXpSlL1iIyvSaqDUzNYcIo5r+rxjv6qP/7051TdXIHAUayo8VYN4bnjWeBwFhXDZGPc/BYT0WQiOpOISkSUI6I3AzgZwJ8B/BbAwUR0DhGV4KWvPzqeA4yBsEWA/MMfBhl7LqqaK1JdDbszNcdtqjUCkN1qIhfFNAuOOoaRsuAkCbB1Owfx4V89jJse21D3urWELCrpEpOiOGnI/1yVHN9fzTjv/WVv3vRsJFOQseeiisbgFHIWegaruONJL9bNSlh5TWni0nIjXUj6z0qlZhI43pxs768EY0rCcYUvyqTA0cbipjfb1OOSxhPEPiqGyUSmLKpRIg/gYgBLADgAVgJ4tRDiSQAgonMAfBfALwDcB+ANYzTOzEgLTvgJNvwkq7YZKFnp9Tx2R2qugNOkBadcc2KLv/EevgWi5ls+TJ+6q8ONwTFY2Wpu3H0CAIO+JclUR0bHdeOF7GqKBaeY9/YltXT4xG8eTbx21mwxafnqKkX/DCR1Ew8Ejr9tUlse1z+4LjimXpp4JMjYX6TthCwq0xzLnwk5vjTBKucxTBOPu6j0OjiRXlTjuFXDONZeDDOuGDcCRwixBcDRKftvhSd+dhv6KtEeSEHdlkhtFjf4tD6RqDluoiWhZ7CKmuNiWmfRuL9cc+PR46Z7aFlpJoETcVGNUKsGr8pu/FhZVDBLB/Ka66ZbcHxrhclYlMU1k4V+3/KlCxxTDI4jFMHh335yewFecXOPaJCxWjcnbqmUgiOfkkWlZzLFKj+nPKf8Xhe0NPHgQ4ZIr2TMFhyG2f0ZNwJnIlL2FwrdgqMyUQONa65IjAU54ot/gSuA1Zecbdyftf+Tev2keYwEGY9Qs82K46INcYUjx52ldo4r4h2ra64Izg0sOIafmXrXz/qc/eUaLALaCtFnGTS4/FwRWnCkFWRyW7RVQtKy6wgRazmh18FRv+dCCGM38aJBECYhCzTGLTj+87jxLKrdpQ4OwzDZ4N/iFlKuSVN66ErRSVvMswbbjkeqjhv0VtKpZ0jJYgEBorEmSVaL4cfgmCw4rvG5AoGT0YKjd6x2DRYcUy+qpOvL9Tmri6q/7DX1tDWLQNkQWO26oVgIXFTtmsBJadWgu9rkPT0rELBzMOwAE1pfzC4qSdqHA/mzocfgOLqLKrGSMf9pZJjdHf4tbiFyoXYUs7hO0sK8cuMuHPCZm/HC9oHWDbCF6G65RjAtsCYczUVlQnVRjVQvqsc3eN+b57dFvzfSspJF4DiOQNEYg+PdL4zBiZ+bdH25KGd1UQ1Uamgv2rH0bpPAdIwWnELkGFUgqD/qnosqej1pUZnS4YmkHf2hwAnjZzQXlaFuUBJyHmNp4rLooIhXMqaIBYfdQAyzu8MCp4XIhUIu8qZP1knuhhe2D6LiuNjQM9S6AbaQtGeuR9kQA2K8h1qlOIuLqimxFR/LM5v7UHFcvPjrt+M137872N6IBUcVDME2Nyz+WLAtpVu3PiazAJQCILMFp+JZcHTDi1HgRLKovG1TNAtOJAZH2e6aXFR+7M0U34KzvT/s2xaKkzoxOBmKHcbSxINSDZ6LCgkWnPFcyZhhmGzwb3ELkem2csFpxEUlt++utXKka6iZ8Wd2USnXVoXi6/73H/j6n5+MjANoTmyZxi+z41wBPPj8znAMDcTgOG6815JqwcnnrKBbt06yBce7XjVrFlXZs+AQUUQ86NeXIkuvZDw5ReCouCI+93YQg2Ohu5TDjoG4BUe3ohT1wohpLirNChSLwRGItWqwFYWjW4sYhtn94N/iFlLRLDjmIGPzYlVtwN2Rhee3DeDsy/6ObX3lEblePQIXVROBvc24qN7z0+X4/cNeyvI/V23Hd29/BoBXERfw+hjpY7n4hsfxu4fWIQ2zwKkajmzMRaX2SZJcefdqfPmmJwB4C2zNFfje7c/iiQ27IsclCUC5mGdt1dBfqQVNPduUtDB9/uU863VwJrVHXVSk/DVRA4JdYciiUpTF1I4CthlcVM1acG57YhPe+KP7vHN8kaTHJ3l1cJAYZKyLT4Zhdj9Y4LQQPQbH9Ac5OXYkuzUgCz+5exVWrN+F3z+8fkSuVw/5XFmLzqlkdVGplorntvbjQ798OHaMnL+2gh0rJvfHR9fjtpWbU+9hqmmza9BcqbjciIvKFcbCeNI6olov9AafST8T0uqQ1RXXX3aCIn9RgRMNopbuJelWknv0rCZVILzt+IX4wKn7wLYIrht3UanWkikdhUgMTi3BRRXPojLPw7t+uhxbesuRMcv7RZqDWhSJu1GtOeM5TZxhmGywwGkhQRaVvyCZ3Q3mxUguYiPlopK9jdLaGowkcpFtzoKTNQYng6VECpy8HXOTlGsu+uu0VRg0iC3ZC0m/hxQ25YRxrdraj6/9eSWEEJ6LKmERLdhWZOGdqllKpABcNL0jel7OXFMmCc+C4/1cqKniXisRtY6NvH68I7eK+jilvI2Pn7kE7XkbjsGCoz771PZCUJ0YCH/24y4qLYsqg5CTmWpSfDlCBG6qtDRxrjXDMLs/LHBaiF4Hp5Eg45GOwZELmSzutnbHQEOduxtFLrJpcS8RK4FyXFYXlclSoS+6cqEu5a3Y8ZWaW7dv1JChJoyMwZHs8t/XCzL+wNUP4nu3P4vV2wbguAK2Zf710xf2KR1RgSN/Zr7+2kMj24O+ThmtZgN+mjgQtoaQ91a/B65mwZHvT9l/JpbMDksymmJwLIsgRDzdXbVeTe0oRGJwkiw4scKIGYRc3opmUa1YtwvXLX/B34bEVg0Mw+z+sMBpIVmyqKoJi2HWGJzfP7wOWzPE1chYiwF/QX/nVffjoj8+3rKYnCxBxqq4UxfU7IX+4tWAdcEYcVE5AjXHxc/vXYOBSg2VTBYcg4tKFzh+z7F6MThSGPSXa77ACfeV8uEbvT5Od1u0Hqe8fsGOBt3m7SYsOEXvGlIAy58T1YrmiKjgkD/GbQUbP37bsuA4kz6wyPu5TxO6Uzs8C44Up0GAsGbhaiSLShJacLz3H7vuEVx4/WPeNiu52SbDMLs/LHBahKtUpU2z4NSPwUn+I94zWMWHfvlw3UBZIAyalI0pt/b5DQtbVEwwSx0c1Q2itgfI7KJyRbAwJ50rx1HKeTE4V9y1Cp/93b/wy3++gJorMgmcLq0ZpR5k3CMFTh0LjnQD7RqsoqZZcLpKYUaSnsGjf4ukGCzmo8fJxTyLBUcIgYGKasHxzpUxOeo8SveSdFGpSeCq6DBZQGyL/CDj5LHMmVRCuRaWRAhcVJqgUeelkLMyPacUSbZBvRRzdkSUsVuKYSYWLHCGwQNrtieneTvyU7YVVPQ1ponXyaJKsvAAYQBsveaOz23pw9odgwBCgSOtCVkDehulmiGGSJ071YKT2UXliEhwLADc++y22Dgs8hZExxX4rS8G5YLXV06/12DFifVqiruoNIGT8MxyrFt8q5maSaSKKN01o/+MhRYcXQD4aeIZLDjlmgvHFYEFR7qfpGBUO4rHs6jUe6YLHPJT3dOE9FELpgIA7l+9HeWaE3RK15+vqHyvOwrxmCoAeOSFnZH3eh0clVnd5l5oDMNMDLgXVZM8t6UP5/zgHzjv2L1x8asPie2XwqG9aKMy4Hrl6huogyMDVdXFsmeginLNwczuEgBV4LiBm2q6oYHlS77xt+B1vx9kLMeSVUw0SprVSqKKn4gFJ2uhP9eNCZzzf/5A7B5524JtEQarAis39gIIrTBJFpzntw1gelcBg1UH0zsLQE+4L9GCU8dFFQgcmeGjxNpELCHaxw5dJAYCR7NwSJGSpd6PjD2SFhspBNr99wMVB89s7sO+MzsTs6jU8wBzLyqbCK6bPqYD53Sho2Bj+eoduOvprbjugbX+/TQXlXKvjmIu5oobqjp41ffujmwLBI7BgjOru2R0qzEMMzFggdMkMg7j0bU9xv1SOHQUctg5UEXNdRPSxBMq8PrZVaoF5/hLbkN/xcHqS85GueYEC2W55mDZxbcCSG5gKRnwLRbyA7Wpc/RIIDNc9OdTRZ4qBFQrVPYsKhFrFBnd76LiuCjYFvK2FbnfjgFPlAxWHT8eJrrSnfy123HIvEkYrISp1OFYo+N7elMfBisOtvluv75yDT0D1VivJulSCiw4yj3V17rnJSZwZENOXeDYXuG6LNllO/3nn+Q3zJTBxR3+fH75T0/gnme34c6PnxqMWx6jZgOqIispBmf7QCXVypizLbxo0VT86bENke+97qJSg687CrmYi2qXoT5R2Kohft9ZXSVQYotQhmF2d9hF1STy02WiBcbfLk3+jisS0sQd4zUqjrcgqItbv5LR86Yf3Ydzf/iPyL2yIBcBOZZ67q1mkYusvtiqdXGiQcZmd1XqPdy4i0ploOqg5ngVg22LIgugmrXTr6XOyzE/tq7Hi8Hx42OSPu1/+7anceDnbsZvHvQsD6u29uOwL/4ldpwsCSCFqeqiUgWOtHY88rmXopS3YiJRWrh0Cw4A5C0rU/p0j9/cUja7lJaODl/M3eO7+rb2l1NdVKqVxRTDYlmEWx7fhLdfeX/qeD551hL0DFYjWW15zZSlXr+9aAcfGIaqDoQQ6BuKW+PkmPWGogAws7vIFhyGmcCwwGmCquMGcRhJpvdA4PgLRi0hk+S7tz+L/T9zU+DmCO7hL4ZJQcYPrNkR3kuxKNQLmpX3kUNpRBxlxXXDWiO61UpdrIdvwXFTLTiDFSdwUeUsQs9AOMfqa33OBjTR11mKF8PLgp6yLp8xEDiq1UY5Vs7ZpPY8Oov5xMwwXeAcOKcbtkWZXFTSgjPZt+DkNAtO+AzxGBx1rPUCc2XsS71MvyWzu7FQq+uTzyVfu7OYg+MK9AxWcdSXbsGfV2wypvzLMZvGWcrbbL9hmAkMC5wm+PCvHsYbf3QvgOQsIWmFaPcXRccxCxz5h3/9zsHI9rR4jtjCqVg81mgdrvW4H5mOm2TBueeZrVh44Y1YtbXf+FxZUK000gLlugILL7wR3771qdg+fRyNVDJOEx395RoqSgyOmt4dseDoAkcLPO7UasWonHfs3on311PM5XuZwabG4KgaJuICsikWaF42BBl/701H4uNnHoCcTYmB3TsHKlh28a148PkdgYtONruU19IF40ClFtbBkeNtIPGukf5fcye3Rd7nEuoEAZ57ruoIrN0xgP6Kg/tWbTNacOSYTVlUAGdOMcxEZtwIHCIqEtEVRLSGiHqJ6CEiOsvft5CIBBH1KV+fHauxqrEPSYuJXIQ6/CyVJAuORC1VD6RXMtYtHKogWL2tP/3YmoutfZXEGJxf3u8VQXvo+dBCdOFvHsVbrrgvcew66nPK19Iq8qO/h8UFVfG2aVfYNT1rewrHTY/BGah4LqpCzooFrO5ULDh6VpTuspKWkpLBJXT6gbMS769b5QZ9F+Nm/1nVzB617YDq1svnLGOQcSEXrXZ87OKpyPuxRkl1cNbtHMTWvjKe2tiLnb7Ak3FCSdlG/eVaYI0rGCw49WjEBTpvcinyPq3hZc6y4LhuIBZXbuhFb4oFJ6nGDcsbhpm4jKcg4xyAFwCcAuB5AC8HcC0RqSlKk4UQo9NrIAX1k3yii0pmUfmf/h03niauuhM29Q5F9qVVMtZN8eWag1LewlDVjQkcU6G61dv6g8wYPd5FtnJQLSNS9GRFdUPJ1wOGxafiuLjq7lW497ntWL5mO5bM7kKl5hpjcK64axVWrOvBN19/uHLteBaVymDVc1HlLAoaUUqiFhxtDrT3cpE0WXBMWWuSXYM1zJkUHQ+AoLFkzhB3A4TuQ3nvWAxOzUFRex5bqfeSZFWUJQL6Kw56BquwKExPD2JVNCXQV3aCsUlrSCOVkxoTOFELTpqLKmd7z7nVd/et3LgrJlSBMMhYFYMvP2Q23nfKPv72zMNjGGY3Y9xYcIQQ/UKIi4QQq4UQrhDiBgCrABw11mPTKeXChS4pCyrIogosOG7sk6+anbOxJxqjkFZTRXeplGtuEJC5tTdqCTIKnK39iosqen252A+nyadqgZCZLv2GlgeVmouL/vg4bl6xEVv7KnjXiYvQUcwZM7u+dMPjuF4raFgvi6q/XIvE4EgsAnYq1pW+cg2uK/DRax/G8tXbYxYcmUWk18MBvO+hns0kSbLgBOOwCD887yj86vxjI6JGzQ7K21Y8BqcWr+AshUneosQsKvlzM1CuYedAFZPa8kH6dJhtZLLgJAcZ12OogRivyVrPrTQXVc4i1BwRuHh3DFSxamsfAOAVh84JjjMJtzMOmoVD508GwC4qhpnIjBuBo0NEswDsD2CFsnkNEa0loiuJaHrCeecT0XIiWr5ly5aWjE2tIJtUTTXMovIWRdeNxut0FGxMVtKIVRcNkN6qQbfgDFWdoG7OzgFN4CiL6uLpHchZhNXb+hUXldmC0ztUQ1+5hk/5Ze3r8eTGXlx688pYQUPpLjEFP+vicGpHwbdEZfvU77gCxVy9IGOBfM6KLHAzuoqRRbq/XMOuoSquf3Adzv3hP2INSY9dPA3vOWkRvvumI2P3aCvYQeaRzi5d4FQd7D21PXg/b3IbXnbwbByzeFrk50i15hQMMTWVmhsTVfL5cn5hyQef3xE09pSoFpydg9WIoMgFlo7oM/T5bSW8scg6ONkVTiMxOLo1Ls1FZVte4cZtimt3+WrPrXrxqw8OtklhqxqmOovR9H2GYSYm41LgEFEewNUAfiqEWAlgK4CjASyAZ9Hp8vfHEEJcLoRYJoRYNmPGjJaMT7XgJMU7BDE4BcWCo/yxn9pZiCwwm3uHsLWvjG/d8hT++Mj6IE3X5KLSXSpDVTcQQju1RVUVC6W8jflT2rB6axiIrMfoSEtLX7mGK/6+Ctf883nj8wHAExt24ef/WA0AeOOP7sUP7ngWPYPVyJjl64EEC47K5PY8SnnbeKxEtU5UXTfWmFJlQGZRWRSx4MzoirqV+iu1iHtDnR/A+x5++uyDsNfU9lgsRylvx9pFSFQLzs/+sRo9g1UcOCdsTnnE3pOD12rjSFUgei6qeByVbsGRlpec76L6+HWP4Hu3P4snNvSGz+mLzDuf2oI/PrI+IrDTY3CiLqoGNEtDnHXIbJytWF9yKd/bvE2ouS629pYDy9oDa3agYFsRy6i00KjPpdc1YhhmYjLuBA4RWQB+DqAC4AIAEEL0CSGWCyFqQohN/vaXElH3WIxRbYyYGGTsCwuZJq43HJzWUQxSdAFg064yPnX9Y/j2bU/jP695SLl+fDXRrSG95XAhjVlwFIGTz1nYb1YXHl23M9imW0uk1aFvqIaNmlVJz95644/uxWd/vwL95VoQ09JfcYxBxrrbx3s2XeAU0F6wUy04Utw5roAQ6W6MHQOV0EWlWANmaHEzfb4FR/LFGx6P7Fdjb3SXRlveTlwwf/T357CxZwg1x8Xnfu8ZIvebGQocad0Doj3B1GnO21ZQMmCo6uCKu1ZhsOoYqhh74+os5dAzWA0sRb9/OHTrSeH4+IZdAIBnNvdF7gPEY3D6y7Wg8GBQtbhFAqe9kMP3FCuZ3rJCxfZdVFv6ylg8vQMzu4qouQKdpVws3gqIChyTq5FhmInHuBI45K0eVwCYBeAcIUS8NKmH/BM7Jg501T2QnCYeLfSnZ1FN7yxEPkGv2zGIdTuiqeJAaOVQz9VdVLsGw/e6BUeP+zh64RS8sD28j5pi7rhhTENfuRbrNK6LLWl1eHJTb7Aoe3EvSpCxzKIy9HyKWXDa8mgv5NBXruH3D6/zRYzAbx9aGxwjxZyMcTIVuwM8V8uW3nLgolItOHpgcH+5FszhofMnQUeN85HfhyWzuzCzq4i8TYkWnJUbe/Gx6x6OfE+6Sjm8ZMlMfOJlB0SOTQp5yueswP34nb8+jS/d8DhueXxTYgzO3Elt2LBzMGjkedvKzeFzaiLzA6fuG7yWcSvnHjU/InL6yo6hm3iLFI6GyTr3umXz8dKDZgWWqq19FUzvLGLJHO+zTpLYVJ9pUhu7qBhmT2C8fZT5AYADAZwuhAhWYSI6BsBOAE8DmALgMgB3CCHMfRJaTJYsKuny6Par4DputOHgtI5ixBK0cdeQMbZBBpiqVg3dgiNdIUTR9OdyzcFflQUOQmDZwqmRc9UU8+39lcD98NALO7FGy8gaqkUtB3Mml9C7qQ9PbgzdIL1DtSCwGghdSqYYHD14dlKb56Jau2MQH/rlw9jRX8GSOd34yK8eCY6R4k7Ob9Kn8XmT27Bp1xAGKw5mdhWDBa5gW+jWFrj+shP0lzpm0dSg/ca0jgK29VeMsSAfPWN/vHTpbABIjMEBgB391UgJgPaCjZ+8/ejYcU5CLJdaB0cVpnrskbQszZ3chjuf3hLUt3lmcx+291cwtaMQEZkn7Tc9yCQCgAXTOoI2H6qA6VdicPJNZFENB1MA8FfPPQyAF3TuuAI7+is4eG43ZnWXcOdTWxItqqphavakkvEYAOgu5YLvK8MwuzfjxoJDRAsAvBfA4QA2KvVu3gxgMYCbAfQC+BeAMoA3jtVY9XThoNZLpYZnt3hm/w09g+gu5YJPizUtTXxqZwGT/EVojv8Hd9OueLVX+QdbjZUxVWwFvN46OwcqQazP125+ElfdszrY7wrg4LlRC4Wakr1dWYgfeWFnRCwBwJBmDZItDFb6Lg85NjUuKQgyNrioVAtOV9FzLajWkOVrdsRElizmJgWJKlZWfeXlwevZ3SVs2lXG1r4ypnUWAwtOe9GOVOvtKNjoK4cxOEtmh17P956yGABiPaWAaFG6JAsO4MWRqPNqavoIRC2Bbzl2QfBajcFRU9uThN3cySUMVBw8v30AU/xxL1/tdedWf270jCUVNdi3v1ILXJPS9fP24xcmnjta5Cwv+Lpn0MsGk7E7G3qGjMer857m+nr0ojPx9dceNrKDZRhmTBg3FhwhxBqku5yuGa2x1EPPYBmo1NBVyuOD1zyEW5/YjKf/+yys2zGIuZPbAsuBowUZn7TfdDzmWwr2ntqe+Ie5arDgmOp9AMCs7iI27hpCX6WG7lIeTykxFoD3ybyQs3DgnG484YuSIc2Ck4aevi1FxjNbwvv0DdXQrSy+VdfFUNWJVVhWnw0IRYQqFm54dANueHSDds8a1u8cRM9g3IKjfuKf1V3CivU92D5QwYzOQmBlGSg7aFNiX6Z0FNBfrgXPcsDsMEbmPSctxrtOXGysgqvWbOnwr3flO47Gi/efAVcASz57E6qOwJptAxFhoqfxS+TPxs0fPgkHzArHoAqcpzeF83z4XpON15Hj2rhrCP922Fzc9K8NePD5nXjp0tmR7LDJKW6aS885NIgDU7OochbhuS+/fFzUjrEtCkT/pLY8DvRdVIclzIsePM0wzMRn3Fhwdid0C44M3rzzqa0AgI09Q1i3cxDzJrcFloOa36phyewuPPjZM3D8PtODhXP+lHYkIa0cqsBJEiIzujxLkOyzpAsxqa/OP3lRsK1c8xoVqoHC0iVzgRKjAcRr6si4lVVbQitLX7kaDTJ2BC74v4ciliTT9aRLxVRMT+XHdz2H4y/5K+59zssy606wZMzsLmL1tgEIAUzrLOK0A2cC8Nxinco5U9oLEQvOvjM7g31ElFjiX42favddct2lXHDOw597Kf7rpfujZ7CKZ5X5WTDN/L2W7stSzo4ItYLfkmBbXzkS9P0izdUomTclFF4zuorYZ0YnVm70xKxai2iKwSoleeVhc7H8M6fjzKWzPBeVPzaLCJZF46J2jBpILK14//rCmfjV+ccajzc129zdSIo3YxjGDP/GNIEuHGQMjIypeWH7gCdwpqgWHE/g2BZhaoe3mEvxo8asqJ/egTCwV3VRyWaNOrO6veBZ6VrS2xNIV8O/HzEft370ZBw8rxtDVRc/v3cNln7+z4FVR8ZgHLH35Ejsgi5w5HOvV6xPvUPxIONbn9hkHK/qAptssOCYuPsZT9hIt4t0k0lknMjs7jDOYnpnEfsq2UvzFRHQ3ZYL6uCU8lZdgSVRF3lpwVGzojqKORzgu7sefmEnAOCmD52EVx0+13g9KQrVGkve83iF/pb7zVXl/ByupJirqK6z7lIeS2Z3BTFSajXpSSkuKsCbsyntBazfORRY35LE3liQMwQNdxZzid8/+e2a1mF+7iWzu4zbxxP3f/p0LP/M6WM9DIbZbRg3LqrdiaL2R/Sl37oTZy6dhbaCjV1DNbzpx17fprmT24JaHjU/yFj9wzx7Ultw3E0fOgkzuor4+HWP4MlNYdCuyYLz9OZwv4pc1Lf7lpgdWsq4Gjy678wulHJeSva1y71WDP9a57nM5Cf2hdM7YBHFGnMe9aVbsGzhFAxWHXQU7IhloK9cixStS6qqC0TrxMiFJ0ng/PxdL8Jbrvhn8H6jH6+kx6Lcc+Fp6CvX8IgvKgAvY83b95JYy4yOQg5beyvoHarFxFJWpPtLz+CRgvPJjb3oKNiBG8WEXkxPIgv93b9qOwo5C3d+4lRs3lWOiCkVdQGf1JbDkjnd+N3D69EzUI18n9JcVJK3Hb8Qv3lwbVDsMSV0ZdRRxVZ3W/0/Y/JH/9jF02L7/vKRkzGrOznweLzA2V8M0xgscJqglI//pf/zik2RKrWAt8DJdF1pwVGDHc9cOguXv+UovGTJzMDkPrMr+oe26rh4YfsA/v379wTXVF0eKnv77o+NPV62zbY+XeBEj3eECAoKAsDz2wfQVcxhSkcBz28fwPwpbd54/RMHqw7KNQfb+iv48wrPKrPvrK6ImHhmcx/+59ang/dJafQA8Fu/9cI7T1iEd5ywEECyi+qEfaKFq5/03S66KJnRVcSMrmJEWE3z08KldUON/eks5gIXlXR33frRUxKz4/70wZNiIkyKLD2bSt73+e0D2GtqtM+STtDQMhe34FRrLu5fswOHz5+M6Z3F1P5XRIS5k0pY3zOE7rY8FviC54mNuyIxOFlqwRw4pxu/eu9xeI3/s5fkmrr7wpfESgoMh7s+eSp29CdViPBQU8izLPx7T2vHlW8/GsftExc4+88a/9YbhmEahwVOEyS1B1AXkPedsg/OXDobz/liRNbBUWMBiCiWkvq+F++DxTM68JWbVgLwYkbufDpsOXHA7G5s2mVuQbFgWgeIgHU7PZfRVm3R0Qv1rdXq7jy7pR8LprXj6ncfgyc29KKYsyPjLVcdPLs5Kq72n9kZCJwZXUXc/K+Nkf2m9HAA+NgZ++MbtzwFADhm8VTs5YvDJMuEZVEgRgBPEFiESEaUihpLoxf2U7NoOoo59Fc8F5UUS+q5OgfNjVthXn34PEztKASuR4lqTZlSxyUk0X+28jkLQzUXKzfsimRXpTFnchvW9wyhq5THYX7PpQfW7IikietWyCSO3HsKijkL5ZqbGMcyb3JbrFHmcJg/pR3zp6QfYysFHrsNlrc/XnBirAzBqUtmjsj4GIbZPRhHRufdB5MFBwC2KhaTT77sALQXcpEsqpofg5PGoukdePkhYbn6as0FKcllabECnUUbs7pKWLdjEFXHxY6BKk4/cBb+48VevRO9Ptt33nhEzGIwpb2A+VPaccZBswAgFoPz5KZdkeP3mxWKge5SLmaxufo+c6uHtyqpxqpFRH39ny+JBjnrLqCuUj7RqqBuT3NhdBS9GJwtSsn/RpnSUcCrDp8X217K28E1505KFwBBs0ytuF3e9noulWtuJMMrjUXTOwB47s2pHQXsO7MT9z63LVJwMKlBqAlp9UgSq2lc+97jcOU7jk4Nam4GUwyOyiHzJ+GoBXVUEsMwExq24DSB6ka57n3H4bLbnsbfn/YyqI5eOAXnHbsgWGCDLCpXwHVFpkyI+VPa8KHT9sOTG3tx28pN2NwbBvHqQcgqxZyNeVPasH7nID79Wy9u4pQDZuC4xdPw/TuejVWgPXbxNHzizANw8Y1PBNt0K4TqUvvIrx7Badqn4P2U8XSmxLC88UV7Y0tvOQg4VmNNVKuNOrcfOm0/fOevzwTvD5zTFckkkuLhirctM/av+s37j8NDz+80iqBfvOsYrN85iM29Q6g6Ais39uK1y/ZKHH+zTO0ooHeohoW+6EjijxeciLue2RIba0ERPGkxPCqffvmB6CrlgsyxoxdODXqKXfTKg7BxVxlHJ2RhmfjGaw/D925/NlYkMgsvWuSdM7WjgB0DVZx/8mK8+IDh94iLBopzbArDMHFY4DSB+un36IVT8d+vPgQnf+12AN5Crn6al5/M73xqC5av2YGT9jM2QY9ARPjIGfvj2uUv4OYVGyNZSAfPCwv1nX3oHMyb3IbL73wuGNfcyW244dH1+MdznkXlhH2mBYumKaxEbzw5d3I0Bki3OKml/wFg6ZxuvOOEhXjFoXPwrVuehm1RLH7lpP2m4zNnH4jbVm4OnqWUt1DIWajU3EQLTs62cMGp++J434JwxkGzcfuToXtusx9ofNqBs+IPBuCoBVNx1ALzonyi/3248u5VALzYplYUsJNViBcmpIdLDprbbXR/qe60NNeZypSOAj7/yqXB+9ctm49VW/tw3OLpePsJi1LONDOts4jPvfKghs9T+dFbl+HyO5/DJ1+2ZESysVTrTNbMN4Zh9izYRdUE+h9UtfaIHmshm0Feu9zrp7TZUK04iTOXzkbBtvCvdaFbaEpH+Gn1/afsgw+dtl/wvpizMXdyKXBF3fThk7F4RmcQO2FqBXH6gbMin6jVSr4A8IM3H4UT9g0DMw+ZNwnffsPhwfuZ3SV8/pVLcdSCqThz6Sy875TFsXiMn7/rGHQUcyj5wnB2dwlEhJm+uFLdDXoA73+deQCO39cTI68+Yi5OP3AmrvRbHegxFs0g2yicdfCclqRB9/punQXT0i04SUiBM6u72PRCfsTeU/DL84/Dh07fr/7BLWLxjE5ccs6hIzbHaZWYGYZhABY4TaGXerctwrXvPQ4HzOqKuRGmd3kxEHIRV1PA6zGpLY9Tl4TiY/GMDkxpLwRpzxZRJHOnmLdw/D7TMb2zgFcfPjcQGrMmFTGru4jPnh3/FN5RzOGqd7woeH/gnKgL7Lh9puEX7zomeP+HC07AQf4z6u62txy3EB8/cwm+8ppDMG9yG15+yOwg/gcI2zVIK9RXXnMI9praFil02JayiLcXcvjx247GqUtm4t+PmIePnbF/4rFZOeuQOdhnRkfQlmGkkUHRi+q4qJKQgiCpcvGezIdO2w//fkQ89olhGAZgF9WI8aJFU/Hnj5wc295eyOHWj54CAHjZ/9yJE/et76JSefXh8/DnFZvw+mV74dJzDwUAXPzqQ/C+XzwQs5QUbAun7D8Dyz9zRmR7MWfjvv+XrUCYKWWWtMyv+VPaUchZuPScQ4zXOHn/Gbj7wpfEth+xl+dW+MCpnug5ab8Z+PsnosfJzt16kUKdb73+8NT9WTlwTjdu+9iLR+RaJj73ioPwlZtWBtaqRpG9zQ7fyxwwe/ahc/DUxuyieSLxkREQuAzDTFxY4IwiN384LoDqceqSmZg7qRSJz3jZwbODzs+A1/zwqntWJzZyzMI5R87Hbx5cm1rs7tV+Fd62go2nLj6r4XssnN4RGbcJGXD8fsXyszvzjhMW4R1NxL1ITth3On72jzV4+SHmDtffe9ORTV+bYRhmIkN6bZSJxLJly8Ty5ctbcu2FF96IZQum4NfvP74l11dx6qSXy+/hcHsECSESrzFS92Aax9UKRDIMwzAhRPSAEGKZvp0tOE3yxBdfFrRhaDX1AjNHSnSkXYeFzdjB4oZhGKZxWOA0SVudppAMwzAMw4wdnEXFMAzDMMyEgwUOwzAMwzATjt1G4BDRVCL6LRH1E9EaInrTWI+JYRiGYZjxye4Ug/M9ABUAswAcDuBGInpECLFiTEfFMAzDMMy4Y7ew4BBRB4BzAHxWCNEnhLgLwB8AvGVsR8YwDMMwzHhktxA4APYH4AghnlK2PQJgqX4gEZ1PRMuJaPmWLVv03QzDMAzD7AHsLgKnE0CPtq0HQKyvgBDiciHEMiHEshkzZui7GYZhGIbZA9hdBE4fgG5tWzeAPbMJD8MwDMMwqewuQcZPAcgR0X5CiKf9bYcBSA0wfuCBB7YS0ZoWjWk6gK0tuvZEheescXjOGofnrHF4zhqD56txWjlnC0wbd5teVET0SwACwLvhZVH9CcDxY5VFRUTLTb0vmGR4zhqH56xxeM4ah+esMXi+Gmcs5mx3cVEBwH8AaAOwGcA1AN7PKeIMwzAMw5jYXVxUEEJsB/DqsR4HwzAMwzDjn93JgjPeuHysB7AbwnPWODxnjcNz1jg8Z43B89U4oz5nu00MDsMwDMMwTFbYgsMwDMMwzISDBQ7DMAzDMBMOFjgMwzAMw0w4WOA0CBFNJaLfElE/Ea0hojeN9ZjGGiK6wO//VSaiq7R9pxHRSiIaIKLbiWiBso+I6FIi2uZ/fZWIaNQfYJQhoiIRXeH//PQS0UNEdJayn+fMABH9gog2ENEuInqKiN6t7OM5S4GI9iOiISL6hbKN58wAEd3hz1Wf//Wkso/nLAEiegMRPeGvjc8S0Un+9rGbMyEEfzXwBa8Gz6/g9cc6EV5PrKVjPa4xnpPXwEvh/wGAq5Tt0/35eS2AEoCvAbhX2f9eAE8CmA9gHoDHAbxvrJ9nFOarA8BFABbC+5DxCnhtRxbynKXO21IARf/1EgAbARzFc5Zp7v4C4O8AfuG/5zlLnqs7ALzbsJ3nLHnOzgCwBsCx/t+0ef7XmM7ZmE/M7vTlL0wVAPsr234O4JKxHtt4+AJwsSZwzgdwjzZ/gwCW+O/vAXC+sv9d6g//nvQF4FEA5/CcZZ6vAwBsAPA6nrO6c/UGANfCE9VS4PCcJc9XksDhOUues3sAvGu8zRm7qBpjfwCOEOIpZdsj8D5ZMnGWwpsfAIAQoh/AswjnK7Ife+hcEtEseD9bK8BzlgoRfZ+IBgCshCdw/gSes0SIqBvAFwF8TNvFc5bOV4hoKxHdTUQv9rfxnBkgIhvAMgAziOgZIlpLRN8lojaM8ZyxwGmMTnjmNpUeAF1jMJbdgXrzpe/vAdC5h/mt8wCuBvBTIcRK8JylIoT4D3hzcRKA6wGUwXOWxpcAXCGEeEHbznOWzCcBLIbnMrkcwB+JaB/wnCUxC0AewLnwfi8PB3AEgM9gjOeMBU5j9AHo1rZ1w4ufYOLUmy99fzeAPuHbKic6RGTBc3FWAFzgb+Y5q4MQwhFC3AXPb/9+8JwZIaLDAZwO4FuG3TxnCQgh7hNC9AohykKInwK4G8DLwXOWxKD//3eEEBuEEFsBfBPjYM5Y4DTGUwByRLSfsu0weK4FJs4KePMDACCiDgD7IJyvyH7sQXPpf0K5At6nn3OEEFV/F89ZdnII54bnLM6L4QWuP09EGwH8F4BziOhB8Jw1ggBA4DkzIoTYAWAtvHnSGds5G+vgpN3tC8Av4WVSdQA4AZxFBXgLTQnAV+BZJEr+thn+/Jzjb7sU0Qj69wF4Ap4peK7/g72nZB38EMC9ADq17Txn5vmaCS9YthOADeBMAP0AXsVzljhn7QBmK19fB/Brf754zsxzNtn/2ZJ/w97s/5wdwHOWOm9fBHC//3s6BV7G3pfGes7GfGJ2ty8AUwH8zv+hfx7Am8Z6TGP9BS87Q2hfF/n7TocXEDoILzthoXIeAfgqgO3+11fh90ebyF8AFvhzNATPRCu/3sxzljhnMwD8DcBOALsAPAbgPcp+nrP6c3gR/CwqnrPUn7P74blQdsL7EHIGz1ndecsD+L4/ZxsBXAagNNZzxs02GYZhGIaZcHAMDsMwDMMwEw4WOAzDMAzDTDhY4DAMwzAMM+FggcMwDMMwzISDBQ7DMAzDMBMOFjgMwzAMw0w4WOAwDLNbQESCiM5t4fWX+fdY2Kp7MAwzerDAYRim5RDRVb540L/ubeAycwD8sVVjZBhmYpEb6wEwDLPHcCuAt2jbKllPFkJsHNnhMAwzkWELDsMwo0VZCLFR+9oOBO6nC4joRiIaIKI1RHSeerLuoiKiz/nHlYloIxH9TNlXJKL/IaJNRDRERPcS0Yna9V5GRCv9/X8HsL8+YCI6noj+5o9pHRH9gIj07sgMw4xDWOAwDDNe+AKAPwA4HMDlAH5GRMtMBxLROfC6Y/8HgP0AvALAP5VDvgrg9QDeCeAIeL2rbiaiOf75e8HrKXeLf7/v+Oeo9zgEwF/8MR0G4DX+sT8Z3mMyDDMacC8qhmFaDhFdBeA8eA1GVb4nhPgkEQkAPxZCvEc551YAG4UQ5/nvBYDXCiF+TUQfBfBeAAcLIaravToA7ADwbiHEz/xtNoCnAFwjhPgMEX0ZwLkADhCy6x/RZ+B1QF4khFjtW4SqQoh3Kdc+HMBDAGYJITaPyOQwDNMSOAaHYZjR4k4A52vbdiqv/6Ht+weAsxOudR2ADwFYRUR/BnAzgD8IIcoA9oHX3fhuebAQwiGifwA4yN90IIB7RfQTnn7/owDsS0SvV7aR//8+AFjgMMw4hgUOwzCjxYAQ4pmRuJAQ4gUiOgDAaQBOB/ANAJ8nomMQihCTeVpuI8M+HQvAjwF8y7BvXWMjZhhmtOEYHIZhxgvHGt4/kXSwEGJICHGjEOIjAI4GsBTACQCegZedFQQV+y6q4wA87m96HMAxRKQKHf3+DwJYKoR4xvA12MTzMQwzirAFh2GY0aJIRLO1bY4QYov/+jVEdD+AO+DFx5wG4BjThYjo7fD+ft0HoA9eQHEVwNNCiH4i+gGAS4hoK4BVAD4CYBaA7/uX+CGAjwH4HyL6PoBDALxPu82lAO4loh8C+F8AvQCWAHilEOK9jT8+wzCjCQschmFGi9MBbNC2rQMw3399EYBzAFwGYAuAdwgh7k+41k4AnwTwdXjxNo8DeI0QYpW//5P+/1cCmAwvMPhlQogNACCEeJ6IXgPgm/CClR8AcCGAX8gbCCEeJaKTAVwM4G8AbADPAfhtY4/NMMxYwFlUDMOMOWqG1FiPhWGYiQHH4DAMwzAMM+FggcMwDMMwzISDXVQMwzAMw0w42ILDMAzDMMyEgwUOwzAMwzATDhY4DMMwDMNMOFjgMAzDMAwz4WCBwzAMwzDMhOP/AzU9PXQwEZfwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot(rewards)\n",
    "plt.xlabel(\"Episode\", fontsize=14)\n",
    "plt.ylabel(\"Sum of rewards\", fontsize=14)\n",
    "save_fig(\"dqn_rewards_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "state = env.reset()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for step in range(200):\n",
    "    action = epsilon_greedy_policy(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    \n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad at all! 😀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Double DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(32, activation=\"elu\", input_shape=[4]),\n",
    "    keras.layers.Dense(32, activation=\"elu\"),\n",
    "    keras.layers.Dense(n_outputs)\n",
    "])\n",
    "\n",
    "target = keras.models.clone_model(model)\n",
    "target.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "discount_rate = 0.95\n",
    "optimizer = keras.optimizers.Adam(lr=6e-3)\n",
    "loss_fn = keras.losses.Huber()\n",
    "\n",
    "def training_step(batch_size):\n",
    "    experiences = sample_experiences(batch_size)\n",
    "    states, actions, rewards, next_states, dones = experiences\n",
    "    next_Q_values = model.predict(next_states)\n",
    "    best_next_actions = np.argmax(next_Q_values, axis=1)\n",
    "    next_mask = tf.one_hot(best_next_actions, n_outputs).numpy()\n",
    "    next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1)\n",
    "    target_Q_values = (rewards + \n",
    "                       (1 - dones) * discount_rate * next_best_Q_values)\n",
    "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
    "    mask = tf.one_hot(actions, n_outputs)\n",
    "    with tf.GradientTape() as tape:\n",
    "        all_Q_values = model(states)\n",
    "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
    "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "replay_memory = deque(maxlen=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 599, Steps: 55, eps: 0.0100"
     ]
    }
   ],
   "source": [
    "env.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "rewards = []\n",
    "best_score = 0\n",
    "\n",
    "for episode in range(600):\n",
    "    obs = env.reset()    \n",
    "    for step in range(200):\n",
    "        epsilon = max(1 - episode / 500, 0.01)\n",
    "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
    "        if done:\n",
    "            break\n",
    "    rewards.append(step)\n",
    "    if step >= best_score:\n",
    "        best_weights = model.get_weights()\n",
    "        best_score = step\n",
    "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\")\n",
    "    if episode >= 50:\n",
    "        training_step(batch_size)\n",
    "        if episode % 50 == 0:\n",
    "            target.set_weights(model.get_weights())\n",
    "    # Alternatively, you can do soft updates at each step:\n",
    "    #if episode >= 50:\n",
    "        #target_weights = target.get_weights()\n",
    "        #online_weights = model.get_weights()\n",
    "        #for index in range(len(target_weights)):\n",
    "        #    target_weights[index] = 0.99 * target_weights[index] + 0.01 * online_weights[index]\n",
    "        #target.set_weights(target_weights)\n",
    "\n",
    "model.set_weights(best_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure double_dqn_rewards_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAEYCAYAAABRMYxdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACiO0lEQVR4nO2dd5wkVdX3f6e6e8JO2JyB3SXsLixRlhwkgyLKI+ojYg6YMD8qvgZQMafnMYsJMYtiRFAJoiBpySwsS9hdNufJoUPd94+qW3XvrVuhZ7qnZ3bPl88w0xVvVff2PXXO75xDQggwDMMwDMPsSTiNHgDDMAzDMEytYQOHYRiGYZg9DjZwGIZhGIbZ42ADh2EYhmGYPQ42cBiGYRiG2ePIN3oA9WTGjBli4cKFjR4GwzAMwzB14v77798hhJhpLt+jDZyFCxdixYoVjR4GwzAMwzB1gojW2ZZziIphGIZhmD0ONnAYhmEYhtnjYAOHYRiGYZg9DjZwGIZhGIbZ42ADh2EYhmGYPY4xM3CIqJmIfkhE64iol4geJKIXKOvPJKJVRDRARLcR0QJlHRHRF4hop//zRSKisRo7wzAMwzATi7H04OQBrAfwfACTAXwcwG+IaCERzQBwvb9sGoAVAH6t7HspgAsBHAHgcAAvAvDWMRs5wzAMwzATijGrgyOE6AdwpbLoL0S0BsDRAKYDWCmEuA4AiOhKADuIaKkQYhWA1wH4ihBig7/+KwDeAuC7YzV+hmEYhmEmDg0r9EdEswEsBrASwNsBPCzXCSH6iegZAMsArPJ/P6zs/rC/zHbcS+F5fLDffvvVZewMwzDMyPj8javw3dufwRcuOgz/fUz6d/SNj27G23/+AFZ+8ly0NXtT1hV/fAw7+oq44dHNOOvg2bj5ia340ssOx8uX76vtW6q4WPKxG3HIvE48va0Pqz79Am39w+u78Oof3oPhsou3Pf8AvP/sxfjS31ahVBH4fy88GN+89Sl8+e+r8ZIj5+HWVdswXHaDfZfM7sCf33Vy6vjf/JMV2NozhHU7+zF5UgFzJ7fi4fVd+MVbjsfRC6Zq2573v//C+l0DWDKnAw4Rfvv2E/HjO9fgk39+PLjOOz58Os7+6r/wofOWYPdACV+/5SlcdeGhKJZdfP6mVSg4hO++5micclCksC/uXbMLb7zmPhQrbmS7257chm/e+jSue+sJcJxkBciX/rYK3//3GrTkHZy+dBb++NAmXP6CpXjb8w+A6wos/fhNuOLFh2BbzzC6B0u48sXW6bruNMTAIaICgJ8D+IkQYhURtQPYbmzWDaDD/7vdf62uayciEkIIdSchxNUArgaA5cuXa+sYhmGYxvLAc7sBAKu29Gba/v9ueQoA8NyuARw8txMAsGLdbqzc1AMAuPmJrQCAJy3H6x8uwxXAYxu9bYUQUOWba3b0o3eojLxDeNw/3gPrulCseIbM45u9ZTc+ugXFiouLj90Pk1sLuG/truA60pDjA4CeoTLW7xoEAKzd0R8xcOQ9eeC5rmDZE5v167zjqR0YLFV8o2eWv18PhksuhBDoL7p4eluf1cB5Znsf+obLePnR++C6+zfgGWW7xzf14P51u1GsuGhxconXtHJTD1xXoGeojBsf3aKNs1hxUay4uOovT+CYRdPQNVDMdJ/qwZhnURGRA+CnAIoALvMX9wHoNDbtBNAbs74TQJ9p3DAMwzDjm57BEgCg2m9vV9nBtq/tcATdE2HuJ/y9Wgs5yOlEQIR/+9tX/D/e/vwDcPkLluKkA6ZXPX4T1zhA3HRmLla9K3KdK7zrbynkgtdJx3rTKYsi25nXnIQQ3j0Dwntju7dCiFHfp9EwpgaOn/n0QwCzAVwkhCj5q1bCExDL7doAHOAvj6z3/14JhmEYZkLRO1QGED+hm0iPi7q5aRyY68Od9Zfmfq4fcXIcCgwkIaD9DQAV3xKQzp9wTCOfvc094w5lGis5xQPlBsaFgCsEck7yuKRBl/e3U++HPI/t3kbHJAJDS96buP2E1fQcG8bag/MdAAcDuEAIMags/z2AQ4noIiJqAfAJAI/4AmMAuBbA+4loPhHNA/ABANeM4bgZhmGYGtA75HtwMm6fIgcJyDYx2/fJOaQYC/GTfWjgINh2pJhGSNyhTAPBcdR14TiECA2XNGPJsVRZkftkvaScY/eOmYao66JhjGUdnAXwUruPBLCFiPr8n0uEENsBXATgMwB2AzgOwCuV3b8H4M8AHgXwGIAb/GUMwzDMBMF1BfqGPQ9OFoMECI0J3duQcRqOCUmZq3MOKRO8CGZp8yzScyNDX6PxTZiXEOsBMUNUmgcn3FcoHpzY+6MYdOZ28u/MHhzDSJL3Vr3HqjesEYxlmvg6RByG2vqbASyNWScAfMj/YRiGYSYg/cVyMClntVECY0LTi0S3s4VlojoX+z45sntwosaF/tvbf2Q1Z+O8SSbmdUnjhChc5/pjluGrWA2O/9uxbGeG5ZIQAsgZ7hHpqTHfp0ZKZblVA8MwDDMmSP0NUH2ISt3eqsGx7BvR3ERee79VD44rROiNMENUCI0Ldf+RkGZ8mWOUqBocdcyqLibNG2Tz4IQi42wenFyMB0c7JvYikTHDMAyz96IZOJldOJbJ2LJZFqMn6sHxfuccUkIs8XoUJ9DgyBBV7UTG8SJdnTBMpoSF/FBQPkVkrGqO5H7BearwrAkB5HK6geMa90x6wvYmkTHDMAyzlyIFxkD2EFUYDgqX2UNU0WXpHpxwwldDLGGIyjhoHUXGcd6guHRyonDMUoMTenDizun9DkJUyobVaHCEQNSDYzOQxOju0WhhA4dhGIYZE3oUAyezyNj/rRoEWUNU5kJz4g81KUrKtXIuc3vHFBmPJkTl2o2tCMbiskxZh+7Bcd30LCq5OGcxhITxO3HsijEV7G8JcQmI7ILwOsAGDsMwDDMm6CGqbPtQQkqzviy6MOLJiISoQg+OMJZZNg+MrcCDU8MQlYhJpzYNBLUmj6kbcizhPO0cZojKyHhK2tcce1SDox/HO1Zjs6jYwGEYhmHGhB7fwGnKOZkFuoEHR1mWtdBfaojKH4RDpFXydS3eCLmd9zv+nFmJepOShcGSwMABaWEhV3jGoEN2Y089lhy/rZJxFotELSqoLlN3DzxhHKJiGIZh9nSkBqezNZ/Z+xFkLCmzcVYNTooDJ3idz5Eiko3P/AkK/SGl3kwGsmpwzPskDRyQnrkkhGe4EFHqsWyGkPwri+HpncsuMjbF4ByiYhiGYfZ4SmVvsmvO5zI/2YcZSyE2D4XNYErTuQRp4ooHx3XVyV7fPshgsqSuV4t5CXGGgGlwlDVhcLiv10hU1xPFnZMgDSHlWK70wGQIUQmBfC5Og6Ofj0NUDMMwzB5PxU/7UVsjpCGnUc2DY9kui+chLiPJcVQPjhqiMsYS6UWVfs6sY8layVjeQ4IuhpbVhZM9OB6O78GxiYwzeXAQ9eCEES7V08Z1cBiGYZi9gIqv3XAo+5O9nEcrqrdhhBqcqMjY+53TNDih28HcPcyiQrjtCImEy+KMEmNFWREZq+nsAukaHHk/iCwenBjdUdxxIr2ogpumLuMQFcMwDLMXUHE9YyLJy2AijYpKmgbHFqKKhIHM194CNYvKO5a+XhLJohoLD47xWhMZy22ELzKGL5iOOad6CocQCSeZ28Rhq4MT1uTRt2MPDsMwDLPH49VP0fsopWFvthndLkvqeCRE5f9WQ2ay7YG6XhJmUdWj2aZ9u+Q08XDMQgg45I3N1B6ZEOmZY9547GE5+5j0ruaAUpNHDVE1VIHDBg7DMAwzRpQrAnnH8fUj2fYJPTjqUluIKt2DY24RV8k4OFSsBkfffySY401L7ZZohf4CkbH3N5GX3xVfydjXHBF5hpBFg5NVZBxNE4+O13U5RMUwDMPsBbi+l4GIqn66r1iyh1SyBHhMz0agwVEL/UFPv1YJ08T1/UdCxPjK6MGJa68Q3tv0jCwvi8r0igltmyRsaeKqNylYBg5RMQzDMHsBFTcUGbsxlXtNZMaSrfO1il1knHzsoLKvmiYuROjNMccSdBMffbPNrBock1IQotIL/UkPjuNQqjdIenrsGpwMHhxYRMamB0dI8TN7cBiGYZg9HJlF5Qlks018ch5N8+DYDIT0Zpv+OdS0dW3S17d3jBDVaLwTI9fguMEYTA+OFBmnFvqDcc3K+bOl24d9r+LGCYTi50bBBg7DMAwzJri+B0fto5RGUAcnzYNj2TfNiJDr847iDVGOH/Hg1LDZZpoA2hyjRGqRyNhGho0civcshR4caQip28UIj2LGHglRWa4jqSr0WDCmBg4RXUZEK4homIiuUZZfQkR9ys8AEQkiOtpffyURlYxt9h/LsTMMwzCjo+yKqtPEKRAZqwaOZcMMXp04o8KJZFHJ9frxIh6cUYWo9NdJtWtUW6KixPZCkbHfbNNJadUgwvBWXJp4Vg1OFpGx9/feE6LaBOAqAD9SFwohfi6EaJc/AN4B4FkADyib/VrdRgjx7NgNm2EYhhktrivg+BqcrBOfLURlt2/SdTlxRoVj6lliKv1JY6sWzTbN8cYbJUBzPpyqAw+O4oGRoSACaRWOo+fUqzGr54yr3hx3HMcwcOSOoR9I+IX+0o9XL/JjeTIhxPUAQETLAeyTsOnrAFwrRlMmkmEYhhlXBBocqmbii4qMrXobi2jZ3CySmg34tWOMiTkqx9G8KLVotpk1i8ozcHIYKnkXqGpw1LCQ9PR4dXDijyUvI9JsU/EGpY9dRAv9WfaX9XkaxbjT4BDRAgCnArjWWHUBEe0iopVE9PaE/S/1w2Artm/fXtexMgzDMNkJs6jiM31Mwpoz4TLbrvZKxlGDxlwf1o7RvSHm/tp0PqaVjAWaFA+OWgdHDasFWVRJzTYhtGKFNpFxJg+OJUQVFPozZD2N9FKMOwMHwGsB/FsIsUZZ9hsABwOYCeAtAD5BRBfbdhZCXC2EWC6EWD5z5sz6j5ZhGIbJREVqcJDdgyOnUT2LKj0cBcQX9lP3Car/aiGa6GStimpNgW1WtN0i4bN4YbAeogp1NPIYXqG/sMZQfEaWXstHL/QXGktpSG+Rej2h10gPJaZVVa4n49XA+Ym6QAjxuBBikxCiIoT4D4D/A/CyhoyOYRiGGRHSg4OEfkkmtqrBtn2zpI7bNDkEAhTBrdoBW91eD1HZj59GIRdOueq+967Zhc/c8IR1H1cIzcCxeXAEZLiN4DjJWVRqLR89My37dciMLdVes2l4ZBPQRjGmGpw0iOgkAPMA/DZlUwHDY8gwDMOMb1w/vTip47WJvdmmXWYcWZISBhKKbkXV3aihHwkpFs5I6uDkHELeIRSDc4frXvG9u6z7SAOhOZ8Lr8EithZCr4MTq+dBmJHlOFFjBMhmtAmh1twx77HxdwMtnLFOE88TUQuAHIAcEbUQkWpkvQ7A74QQvcZ+LyGiqeRxLIB3A/jj2I2cYRiGGS3Sg1NNLyo5Iad2E8+wLPIaYe0YTc9iG4fy90iabTqk61aypmO7Apje3oQT9p8OQPHgmIX+XET0RNEDhvfT1EFVpcFB2M8q3F96k0JPjoCo2stVS8Y6RPUxAIMALgfwav/vjwGAb/i8AkZ4yueVAJ4G0AtPfPwFIYRtO4ZhGGacUlZFxhnNA1urBqsGx7JvWqaS6/paEkTbHpjbOxYPTjWTt0MUG6KKw/UHk3MI//vKIwGohl44ZtcNU8CTKhl7Xh5VZByuE8o2WcZFBM3qs90z0WCR8ViniV8J4MqYdUMApsSsswqKGYZhmImDq6aJZ+1F5f9Wu4nbjZkRhKgQ6lZUPYvpjQAMgXBw/LTRhzhEgQfHzECKwxVSJxSev6ymiQciY+GfI7nZplBFxjHNNrNckpqxZe5v6noa6MAZXxochmEYZs9FZlEhoZ2Aib3ZZnQ72zLTk2FLzSYj80gouhF1f3sWVfbZW2pwAKCQi2pXbAj/P0/Qq2uR1IJ+Mhwkw0ZJhf7UNHEzpRuIGoU2ZDaWKjMO9UDKIcXeFaJiGIZh9lJc1xO3VpMmblYyTpq8I8ssHht9ve8dUdaqmT/q/qrPxVabJw1Vg1PIORnFvPC1NdH7oBZLlMUJTcG0iRQiy+uxenAyXZMICiSqY9W3CH8aBRs4DMMwzJhQdl3kHcczELKKjP3fcmJP67OkLUvZRgjZOkLx4MAerrFVMq4qROWEGpxCzslcUM/T1oSC3jBNPNQxeaEsv2hhSohK3lAzAyoMd6WPS6bX20TGkWM20MJhA4dhGIYZEyoCgUGRNURlil9jPTjWEJVp0Jjr4ac7q+GeuDo4aohKjq2KEJWvwZGenGyGhAhSspOyyWTml5OhkWngwSH9+gSS76+KLCqYJjKWLSQaBRs4DMMwzJjgugI50sMrqfv426V6cKytGtJei4hR4Ar7ZK+GY4IQVUahtLePp8HJO05iQ0wVAaWonlEPSAtRiTDcllRjSHqsvOsxm23qv5Nwhe5V8pZFvV6NDlGxyJhhGIYZE4JeVG72XlRyu4olBKJvF7+vxJZFZYZ1PGFsuF6ienBC1U41ImNfaJyjRJ2MOV6zLYIqMgb0exK2nYg7ntJs09HvT+CByXBNYWPPcJmZ0SWXcbNNhmEYZtzw2b8+ga/+Y3XNjyvTxL2/s+0T1npJ3iFLf6ro67CSMVSjxjJZq5N5EKKqYu7OESGfc5Ru6hk8OG60qF5Y6E/1OmVvtikNtagGpwqRsa/BUY0++/4i8/tcD9iDwzAMw2jcs2YXOpprPz3ohf6yIT0Ksg5OVR4c4yxRkbFfO0Y9rhaiUrdWC/1VLzIOQ1SU6GUxx+/6sSczi0q9Hi/TKpryHjme4sExM9mEsk0arrBkURl/qZ6wRsEeHIZhGEbDdesjDnVdqXnJHrpwDcPGtpvigLHuG7y2aHAIBMfRvSG2tgWaBsf/XV2Iivwfxx9vllAQAKnBgenBMQwUEbadSEqll8aZ2WzTlgUVh/DPr3pwwv3VMUW1TGMJGzgMwzCMRrlOBk5FVN+LSo4jFBlHdyx4LbRj95WYRoUrPTiKweEJe6MHUyU4jj9zpl2DKVKWHhwzgylpf1m7JhQ2hwaO2iAzzLZKaLYplGabZhaVRXcURyDO1o6t/1aP1ShPDhs4DMMwjIbnwan9cStu9b2owklSaK9VvLRriwbHeG3rTUW+d0T12qjeHInWiwrR6srWsaseIMfT4IQi42wenNBb4i0rK24pVdjrKh6cxFYNyvWo70FVHhwBP2ymenDkOuWYKcUZ6w0bOAzDMIxGxfcI1Py4fquGanpRCWPiFZb98jm7psdW2M98LbOBzPN468NtVW+FfJF2h9Rj5TJqcPJKLEzW5FFFxtKTpeqYZGq740RDT/r16t3Ere9BJs9SaEypYzV3F8bvsYYNHIZhGEaj3h4cqkZkLMJ9Abt2xasMbPHgWDw22muExoMZovFaNoQ76IX+somM1dWy2aYM0cUZIWojToGwvUK0krEttV16cOLGI4LjxDfbzKLBCcNh5rVq12XxhI0lbOAwDMMwGvXS4LjCKzSXtdCd3AdQs6ii28RVBs7ebFPPopL7as02ldkyEBmnXIMW4nIIB8/pwMFzO2NF0YDuwVHTv82WFd75w7HK6sJJ4S+tDo6h1QmOlcGzJsNhqi7JvH/qnw2ybzhNnGEYhtGp1NGDI0M02UXG8ne8niPv2D1CWVo1RCsZh+fRQ1Rqmrh/vJSxm1lY7z9nCQDg7K/eHmuE5HMOgEqwvxAxhf6UUJTnbaLUZpvSWJLXYPfgpGOOSb1WNSzVyDYNABs4DMMwjIHMyqk1ZSVNPOvkJ7dKatWQz9nTkswlUYPHn6iV17puRPHAaGniGUNUyno19JSkQYp4cCAzvYxWDcrovEXp91Zo1xPXbDP9fUlq1TDSY9YDNnAYhmEYjUod6+BUXehPhqgSNCIFx8kkMrZmUUHX1KhiY3V7a7PNtCwqw6BQ/47TuuRzqsg4rNUjz1tRz6kZEJTebFOEobZIHR3lnInXJEIDy9FcONH97QUTx44RaXCIqJWIziKiBVXudxkRrSCiYSK6Rlm+kIgEEfUpPx9X1hMRfYGIdvo/XyS9MQjDMAxTIyquqKqRZObjCr9VQzUenEAbEj9ZxqaJRxYZHhzf6yENFukxkfuqBoylFVVqGE/T8KgeoMQsqnBa9sYQGiUOEcoVGaLSix/KsSYV+tONJd3IzFqUT6426+BYm21GZTljSiYPjm+M3CuE+DYRNQG4F8AyAEUi+i8hxI0Zz7cJwFUAzgXQalk/RQhRtiy/FMCFAI6Ad6/+AeBZAN/NeF6GYRgmIxVRLw+OMjFm1uD4HpwUA8e2PCoqjhmPoqmxOEgA6GniQYgq5SJUY0ELUSHekFA9OKEXKdTNyDo4aiZaUJGZklPQBdQ0cX0MWY0Rud7U4NhqBzU6RJXVg3MugLv9v18MoAPAHABX+j+ZEEJcL4T4A4CdmUfo8ToAXxFCbBBCbATwFQCvr/IYDMMwTAYqrqhLWKHsuqHIOOM+prbDWsk458QYOPrrqMg47N8E6BlKMoNJ4lhCVGkXERvicuLDNrlI6rUIzkdK7Ro1TVx6nmRNn6yF/kbSqsHsXB6ONWohTZQQ1VQA2/y/zwPwOyHENgC/AnBIDcezjog2ENGPiWiGsnwZgIeV1w/7yyIQ0aV+GGzF9u3bazg0hmGYvYN69KKStVqcKrppe/t5v8M6OFGyFvqLtm7QxbKm90F9rXorKNg+bfDhn6rh4lVOtu+cc6IeHLkrwV7JWI5VipFjs6gQGiWqsaQONatwmiJp4vI4atjMOPgYk9XA2QLgUCLKwfPm3OwvbwdQqsE4dgA4BsACAEfD8xD9XFnfDqBbed0NoN2mwxFCXC2EWC6EWD5z5swaDI1hGGbvoh51cOQEmKPq0sQDA8fQ4qjkHXvtF5vHRl+v93nSaswE//PQWjUEIa1s3g5AN1ySivF5aeLh+L36NqFuxg0MDGjbqf2h4t472Zlc7q97cKJjTromIrvIWDWaqmn/UA+yZlH9CMCv4WloKgBu8ZcfB2DVaAchhOgDsMJ/uZWILgOwmYg6hRA9APoAdCq7dALoE41qcMEwDLMHY4ZnaoE0HnIOEidh21gAu2EjyarBia73wkWByNiYnOP2DrOoEg+va3gMD1DoMdEPYrZq0IvzKcdWdnOFAAkK+kPFjss4lrZdlW84gazdxLVDGr/HmkwGjhDiU0S0EsB+AK4TQhT9VWUAX6jDuAIdk/97JTyB8b3+6yP8ZQzDMEyNqUeauDxeznESwygmcrPEbuI5x+pNMZfYPDhOxhCV5q3I2GwzzoNDisC3YhhuushY1uoJw0q2a3OFgCN8kbGTXAeHFG+Q3mwzOuaka3KU+kHqeKzC5XHuwYEQ4neWZT+p5mRElPfPmQOQI6IWeEbS0QC6ADwFT+/zdQD/FELIsNS1AN5PRH+Fdx8/AOAb1ZybYRiGSUd6DWpdybisenASUplNgiwqJSXaJB/bqsHQ4Ljmen2SNmvMqC91D0ywSSLxIuXQwCubBo7Wi0q2aoiOQZJzKDBIpQYnSWQsD29mW2UVBIcaHL19RVIl5HpUxc5CrIFDRK/NehAhxLUZN/0YgCuU168G8EkATwL4LIBZAHrgpYFfrGz3PQD7A3jUf/0DfxnDMAxTQ5I8JbU4rtSJZD16kGqcMK6ck7HZprkeusg4kkWVUKjPesDI+eP2D68jauCEVoPr+mOUaeKWY+eIUIHQUsXj3jq1Dk5Eg5PQ68s8hrweLYsqwVuTpYFnPUjy4HzLeN0EoABA2sAOPIHxMDwPSypCiCsRn1b+y4T9BIAP+T8MwzBMnUjylIwGN/DgVCsy1g0b225xvahSs6iECFKrzfUCZpq38nfM8SLnV/7Oac06Qy9LuaK7lbRKxgg9M4CXgWYe2/FbV7lCpm7H212qNyjSbDPYJvs1ydFIL5LZv8u60xgSm0UlhOiQPwBeCeARAKcAaPF/TgHwEIBXjcE4GYZhmDEgfJKvsQfHP16+yjRxaWTILCpzAp4/pdXvRRW/b4Aw18v+TVEPjhBC217VvwQOnIzeDiCahRUXoso5ulfENErUdYCefo6ULCp1qVkvR2Q0bIUbjkXeEzkGV9htmUaFqLKmiX8ZwLuFEHcKIcr+z50A3guv6B7DMAyzB1AxPCY1O64MUTmkZRGlIcMbtlYNczpbcOflZ0SK1gX7pnpwQq+HOkZvWz20ooaHgt5VaWNXNThamriiwanoR8mbBg7CNHF1DEGoyEg/j7sX8nhaHRybIDitOrO/Xq1kLI0yT781fkJUWQ2chQD6LcsH4GVWMQzDMHsAlYo0cGp8XBmi8gvEZRYZu/r+trBRXFgm0prB4sFR051N0a0uErYdP/kakjw4oQbHCFGpGhxf8B3WrgmPIe9H3jCczAJ+KjIkF4xBK/SX7X2X61UNjjRwXGH3ADUoiSqzgXMPgK8T0Xy5wP/7awhbODAMwzATnFCDUz8PTqQGSwJys1CDEzUa4tLO5aIfvX55ZF8gDP/YCv1BmJWMqw9RqesVaY3WqsL04OSMXlTQvC7hdqqmKRgXkpttCtjDXd65sl5T6MFxFAPTO77dVzPee1G9GcB0AGuJaC0RrQWwFl7W01vqMzSGYRhmrLF5SmqBnOTyDiW2KjAx68XYvBNxuhO576IZ7f4YzPW6J8Ks7GtqVsK/5YuUcE5MiCrJg1MwhMSelyk6BmmIatlZTkqzTaF2E7drcLIUR/SugYJjhSGqmCyqBnlwstbB2QTgeQBOB7AU3ufpcQA3czVhhmGYPYd6ldcvKx4HSsj0iRuPTRsURHNiPEJBcUGpmbFocvJEgQGht2rQQ1QE3UDx9k8ee1yaOSlGSFRkrLZqEFr/KPUYMvnKLCCYXOjPrINj0+CkXJP04MCmwWmcMWMj1cDx+091AzhCCPF3AH+v+6gYhmGYhlAOPCW1nank8aTHJHuauL6/SuiNsAhklH2lzWCri6N6cEyRsXYuLU08nNCTUI+hZjs5FO6cJjKWHc+986rHtoWoCEC8gNtVYlREZqE/ec40kbG8BpsGxx6kGrchKiFEBcA6eHVwGIZhmD0YW7ZSLagYE3K1vaisHhxl4rdNzEGDz5hzSuMh9MgoBo4bak3U3+p5UzOObN4myJCa93fEgxNp1RDqf2y9n8wmnokaHCXcZQq9A41TqtEW3pfAg6OE+GwhxPEuMv40gM8T0Yx6DoZhGIZpLGNRydipIkYlhyGzu7SwkSK+tTmcpAEShqiix6aYOjhmeMvWTTxrxpG5v9oHKlLoz4kaMYEGR5mxK27UwKEgTTx+TGrmmebBkZ6yNA+O9IqREjqTISrY39ZGRa2yanD+B8AiABuJaAOMlHEhxOG1HhjDMAwz9oSekuTtHt3QjRXrduENJy3KdlwlrbmaQn9yq7LF8ArFt2T1pgTF8GI8OF6zTXslYzXrC67QPDgIQlSpipXgL90QCVO5k1s16EJiVQcUZFEZhpMpHtZGI3Q9jzsaDw4oqsFx4zxpjTFxsho4v63rKBiGYZhxQVYPzgXfvAMAqjZwcjJNPON4ghCVDJ0p69TwkVVkbHg5zG1ks81Qg6OvA5I9OOljD/+OrYNjanCUEJU0NlWvi7nONJw848l+d/WMLKNVQ5UeHC9E5WtwgsKH9lYN4zqLSgjxyXoPhGEYhmk8lTppcNTKu0kdr03kZuVgXIoHJzAaYnpR+b/DEIrhwYFs1eC9jrRqgL3Any2tPGns6j7mMc00cT1TSh+DrsHxfkfr4MQLuGXdH+9Y9jTxNMIQFUV6ZJnVn5W9Mh271mTV4DAMwzB7AfXT4Hi/w0rG2faLeHA0o8H7HVcZWa2947021ru6J0ILUSkGmXcu3ZAwx5I0dnWs8lhxHhx1u1DQG2qNwrHr4Su5b5K8SUAEx4pvtpl8TWqrBnnufOAhs3twxnUvKiJqIqJPEtFqIhoioor6U+9BMgzDMGNDverghJoWT79RbZq4tVWDUrQuaWKN87h4WVShJ8KmwZEeEi1NXGZRpRkDynpTDCzHZo5J3U4ahbbqw7YQlawSHffeyZCcdyyzsGG29129p2q4S66z+m/Gs4EDL4vqdfAaa7oAPgjgWwB2AnhHfYbGMAzDjDVliyFRC9SsJM/LkDUkYnpw1BCV/zuu9ouRBm6baB0KDSU9RIVgvN52upgXSA+86EUJda1MXOVgvd+UZ+HI8anRMlNALdcnNduEFqIaZaE/VYOjhAAnYrPNVwB4mxDiewAqAP4ohHg3gCsAnF2vwTEMwzBji2vRoWTdPnE7xeNQTS+qsF6Mq70GzDRxW4hKFxHbKhkTQg+OauBoWVTQjQvzmrIwvS0sJaeGh4wscS0rKmyLEO5nYhUZx9o3YasGM7Ve/llVqwYzi0rEib0TD1k3sho4s+G1ZgCAPgBT/L9vAnBOjcfEMAzDNIikar42sk7y8riyh1H2NPHQo+S6eqVcte2ANXtHiojla2MbIfyQmRJiCcZr1MGxNdtMc0zIa/zCRYfhNScsCPdHfEhIy5SKiIyj58gZ40ou9BfW0jFrEcl90t8WEVyDYxg4XiVj2x7j24PzHIB5/t9PAzjX//sEAINZT0ZElxHRCiIaJqJrlOXHE9E/iGgXEW0nouuIaK6y/koiKhFRn/Kzf9bzMgzDMNmoCNXASZ+YKlkNFSWkUl2auH4u9XTqhB9XyVhvpmmuF5onQjPuEoyLMESVfBVySDM7mtGczyn7x3tM1FCWWnPGG0OyB0dea5xhKj1WcgzWEFUVGpygVYNSSHE8NdvMauD8HsCZ/t//B+CTRLQGwDUAflDF+TYBuArAj4zlUwFcDWAhgAUAegH82Njm10KIduXn2SrOyzAMw2TAVs03iayTl+aNSEhljh5fDxtpBo4SbrEdzhVejCqsPKxvJYwQVlKISvWsZK9krBso4f6hB6tiHMSWJm6rgxNsb7RqiMsoA7x7pIa7RlLoT6+DI8cQHmMi1sH5iPL3b4loPYCTAKwWQvwl68mEENcDABEtB7CPsvxGdTsi+iaA27Mel2EYhqkNNqFt1u2TCOrgKFlLQgirV0JFHUPZ1UWsYS+qGINJyEnfLgoWxnj0Cd/7HWRRqd3EMzbblKvNSyQKdSnm/Ssohf7KSljPdhxAb+3gVRdO0OAY+iW7Bse+L4L1oaEaERnHeXDGeYhKQwhxjxDiq9UYN1VyKoCVxrIL/BDWSiJ6e9yORHSpHwZbsX379joNj2EYZs/E9lSfRNYQlZ5ebA8ZpY2nUjGmyiDDKb4Ojp5KbhEZx3h4Ir2olNmy2mabphGnemnMYb/4iHl408mLvDEYYTKbyNiskKwaj5HxKNub2wVNVjOG3QBSjhUaONaeYOM5ROXXv/keEV2samPqAREdDuAT8FLRJb8BcDCAmQDeAuATRHSxbX8hxNVCiOVCiOUzZ86s51AZhmH2OGztCpIQGTNkwkrGatp2hhCYOjYR48GJCVEJoRsHthCVQ8nNNtVU9ICMISp5OjO0pIqMTQNxVmcLXr58H20dKfuZaB6cBL2RNx4RuR653Yg8OP4yNU08xpHWELJ6cL4EoA3AFwFsIKIn62HwENGBAG4E8B4hxL/lciHE40KITUKIihDiP/B0QC+r1XkZhmEYj4qS01tbD07oEQk9IBn2cwWact5UVXZdbadg4o+p/SJFxt42UU+C9OAkaXCshf4QWGjJY5d6FcM00dPEo8cIjA8zVT1VZGxvHCqRmiO5rbpdsHnG91MtkKhWip5wzTaFEN8H8H0gMEJOg1f/5ifwjKSsTTtjIaIFAG4G8GkhxE/ThgS7McswDMOMAtWDk8U7M6I08ZjKwjaE8BpQFiveMXQPTrzxEhw/8PJEtSne6tD8sIWownPohgSQvSie6cFxnOTKwWZdHtMo0Y+lZ1El3VutVYPRgDQsPJh8TaoHxwxRTUiRMQAQkQPgGHjGzRnwRMYbAfyzimPk/XPmAOSIqAVAGV6dnVsBfEsI8V3Lfi8B8C8AXf4Y3g3g/2U9L8MwDJONatPEsxb6U9PEs7Y6ADwjopBzAFRQrtjTxGMrGUMxUBDVlwghtIlaC8/J3lnWLCrdwxJHsDpimITGlu0YQdq69ABZjCxJThtXctVmNWRn6o7CUFU2DQ4RguvSRMbjKEiVVYNzA4DdAH4JYKn/e5kQYpEQ4g1VnO9j8OrmXA7g1f7fHwPwZgD7A7hCrXWj7PdKePV3egFcC+ALQoifVHFehmEYJgOuRYeSuL2xSf9wGbeu2hrZTk0Tz5qFJMcgM4tMD46WXRTjwVHToqMhKt0oUI07s9eTalrIv1M9ONC9QBLvkFKDE93PNKCypomrGhy7RyscvbmdWlAxCbUBqBPxBo2vZptZPThnw/Oe3AjgNgD/FELsqPZkQogrAVwZs/qTCftZBcUMwzBMbSlrBk769qYG58O/ewR/eWQzbvnA83HAzHblWOFknzULSe5XCDQ4+h6q+NauwRFa1pDpLZHrA2+GtdCfbhCof2euGWMsV4vx2Tw4gUfJDJNZlBn5qjQ4ImIsRTw4adfk/yblGEE3cdiNvnGdRQVgMoBXwfPivBee0PhRIvo6Ef1XvQbHMAzDjC2j7UX17PZ+AMBgsaJvF2RRKQZFlhCVQGDgeIX+VA+O99she4hKD8lEtxHQPRG2NhWOakVB/zvNwxVkURmuF6JkDY7c2tTg2Org6M021Swq+9hCPY+xnaHFiUMoRpc0uHLKseIMq0aQVWQ8CE8AfDMQCI0/CuDtAN4JT1PDMAzDTHB0DU769pHUa/+3ORkHhfOUiTFTmrgvMga8LCp1F11kbPPgAHJKV42K8Nh+HRzLtZghKrPeTBbCSsY6WhaVVWSsh6hsXiSJmUVl66slkWnx+hj9df47lxp2UzQ4QV+rtGab4zlERUSz4ImLT/d/LwawDcDv4IWsGIZhmD2Aals1mGnOQXE7Y1rXNDgZs5C8bQQK/kzquvpkGYqM444llGrH0clXenjkBG3vRRXV4GQOUQXjTPDgJIqM9RiX1YNTRaE/U5OkbpcUMtOPoe4f9eDYaFQl46wanC3+z7/g1aD5pxBiVd1GxTAMwzSEag2cuPnQnIyD0IYTpjJnS0MHCnnFg6N1E5feGXurBtdVvB8O2SsZI9QE2TLIrB4cY5v4sesi4XB/tQ5OdD9TE5TkwTFFxuHYoscVyrFDQ0j+zurBCb1SQRFFxYNjNZDGswcHwCFs0DAMw+z5VNuLypzk4yb1ijJZVysyzjuhBsc2f6qVkVVviemxiHYTl5O0HhKS6+R+5vVk9kAZx5CobSPsdXCkQZdBg2OIn9WMpshwhNK6IijOZ3hwUo02OZZQ75NXQ1QJ+4w1mUTG0rghouVE9N9E1Oa/bvNr2zAMw4xbeoZKWHj5DbjliWj6MqNTbS8qW/sDIBqiUkW7SV4GEyGgVDLWRcZmdlEkBAU9xGTX4CgeHK0OTuhxAsxCf9lCVHEaHC9EFW4T8fAYIuwgFGfT4CiFcEjZJs2Do26n3tP0zLDQgJXHyinGkrX+zjivgzObiO4BcC+AX8ArzAcAXwXwlTqNjWEYpiY8tbUXAPDN254e0/P2DZfx/C/dhgee2z2m5x0NtkyirNsD8V4NNYtK9TJs6xnCy7/7H9y/LnqP5GSaV+rgWAv9BQaB3ZvkbWPJohJSt+If3yYyTtC/ZM6iinhwwtYSFVdoXhh1e7Oasq0OjubBcVKabYrQsFG3UzdNzaJSxhhptgn7PRnvaeJfg6fBmQ5gQFl+HYBzaj0ohmGYWmKGG8aKh57rwrqdA/jy354c0/OOhnKVGhxzk7j0Z3WyVj04H/7dI7hv7W78beUWbfuf3r0O3/7nMwCg1cHRjBb/d2zrBKFk+lB08pYeHmsdnIgGJ9yv6iyqiIcmNLZcYUkj93+H7S305SpVN9v0/1a3UzdNe8fVawo9OOE6e4hqfIuMzwRwphBit+EiewbAfjUfFcMwTA2JK7hW9/PCPsGNZ6qtgxNpFhmj5dDSxIMnfoHVW72i9b1DJW37j//hseDvsA5OXJq4PWSkFvrzUslt6+0enGgWlRoKite5qMi1thCUqsGJdBs36vKQ4SlR0UTGSCn0h6ix5AqBr/w9NMCze6UQ0fMIAau7pkEOnMwenFYARcvymQCGajcchmGY2iNinqTrf17vt60C7Xil2jo4Zh2XwDNhZAep3ohQFOx3CAewsSt+KpGtGsoVs1WDvp3pP3CFXtjO5lWiGA+O2U1cNUJCIyJ2yN54YlLmHcXYsoWogjTxEYiMg07kMaEiM6xUrojAUya3SSI8bqhdygUGpl0E3igLJ6uB8y8Ar1deCyLKAfgwgFtqPSiGYZhaEj5Jj62hEfcEP56pthdVJOwTE6KSy3MOaaLgst+MaVPXYOw58r4HxwyBaL2okEVkbI7dN7gsRkE0i0oPBdnOZxJ4O4yZ1tTgmCGq2FYNqWni8ffCuyYRMZaKFb3idNo7HmpwwnPrrRqiRxjvIaoPAbidiI4B0AxPWLwMXguHk+o0NoZhmJoQl80yZuedQBaOpsHJUKfGrOMidzc9O3I7M01cnm/j7sFImrek4O9gZlGZehKrHkjJGooYQMJvb+BEr0UVRXv7h+vkn2nZQUEWlPHJU40tIYRWjdi7Hn//DM029UrGyc02hQgHL7cbKrnGNmkhqvAzHdx/mUXlxmRRjWcPjhDicQCHAfgPgL8DaIEnMD5KCPFM0r4MwzB7LQ3S/owGW7G7JKKtGrzXtqJ6gF7J2BWeB4MIGCxV0DWg63Akei+qcLlZZThicKghGccmMtY1OFqrBhmiCjQruqfEu0brcLXjy2tWCT1AAhVFJxSsh67BcSxjkOQND46jCH5thLoZ73WxYho4SVdkaHD8ccsQ1c7+Ih7Z0B3dJ/mQdSPVg0NEBQB3AHitEOKK+g+JYRimxhjhhrE77UQXGVe3vbpP1LPj931SRcZCoFRxcdCsdqze2odHN3bj1MUzI+cIKhlXdI1HNE08ek7VOLAJn9WsLmuzTWsWVTaRsVnHRqJ6WSpu9HNJ0qOk6F1UCjlCqaKHr+Q1JjXbFMr9kNsVy/oblbnQn6rB8a3Ad/z8gZh9GmPipHpwhBAlAIvQOCOMYRhmVMRNNPWmUdlbo8EWpknCpmux7euKUExLyrYVV+DUg2aiozmPPz28yXoOtZKxrVWD2VdJPWfgsaDoJKYaXYDde2WrZCxfp+pVYkKUat0eL0RlHFuOwfDgSEF2k7JDXin0l9Zs0xXR6zANnHQPTmi0B2niKf+wxnWICsBPALylngNhGIapF43ypAQGzgRy4VQU4U2mZptxImNjhlU9FaqXoewKTGrO46xDZuO2Vdus52jKq3VwlBXGbTVHK5QJ3daqwVsfeiKsWVQxKdpqR/A44gxctW5PxY2GqEKRsb+/zHjyx9RcyEW2lduFxqPFgwPV4PN+lyqmByf5mtSHBXkMU0MUZXyLjNsAXEJEZwO4H0C/ulII8e5aD4xhGKZWhLqBsQ5ReUwc86b6NPFIiCpmXyFEoPugwCPhbZR3CDPamzBQ1DN6JIWgkrE+GYcZQXEiY8W4JLMlQSg8N+vOePvqImPzTSSkG4ChBsf04IQGnk2DExiApgfHt3hUD44uMtarCkfGYxh8gMWDkzGPSg3tpf27Gte9qAAcDOABALsB7A9PcCx/Ds16MiK6jIhWENEwEV1jrDuTiFYR0QAR3UZEC5R1RERfIKKd/s8XaSI9EjEM01AapQFoVP2d0aA+0Gcp9BfXiyqaRSUi4R7pPcjnCPmcE4RgTGSIyvTgRETGFm+SqjlRV6tp4NZu4kEdHP1ckiwhKnk5tvCWNz7vPKYHJFIHx7hfzQXFwDE8OOmF/nRNUbUiY9WDQxk9OI0KUWXy4AghTq/R+TYBuArAufCKBwIAiGgGgOsBvBnAnwF8GsCvARzvb3IpgAsBHAHvPfoHgGcBfLdG42IYZg8m8KSMsaURTsYTx8KxddROItqLyg9RxQh6Ab3IHOB5cPIOaSnqKgWlFxUpj+Vm+nR6HZyoB0etyKuFqGTdHsOICs+dIUQFOT67h0YI777EGjiGhkfe6+a83YOjGh02W1GIsLGn3G646jTx8BrCDLUUA2ech6hqghDiegAgouUA9lFWvRTASiHEdf76KwHsIKKlfifz1wH4ihBig7/+K/A0QWzgMAyTijqZjSVxvYjGM9X2olJtkpd95z/Y2jPsLXdNAyfakkB6D3KOg7zjBKJjc8LPK2niqrEQadUQGZvQQjI2D44qlq1Y1odZVIYRgixZVPb1pKyv+GP84ztPUtoo6AaX3L7kv26KMXCcNA+OCI/V4nuBeofLxpgTL0mrKZVVZNyoENWYGjgJLAPwsHwhhOgnomf85avM9f7fy2wHIqJL4Xl8sN9+3CaLYZgw7DLWdkbZmKAmAno13+pCVCuUjuDWlG3DWLjk+/cA8Dw0Mhuo7LrIOTltX7XZZl5ZFWpwvN/REJVe7Vjz4ATCc9IK1QXjNXtRWcJMqXfHMJIkWvdtv1XDEftOUdZ7v00PTlmGqJSb0NqU0/ZTO6/bhiOvo63Zm/67B0uRbTJckhYOy6eGqBpj4WTV4NSbdgDdxrJuAB0x67sBtNt0OEKIq4UQy4UQy2fOjNZTYBhm78NsWjh253X9847paUeFOjFmqoPjbxRX2E99nTOMhcGSJyrO+SEqIAxbqTQpk7ZWydjw4CTWwTGyqNRQi3x71Gs3e1GZnx3PI5TNgxMNb4XrXWHT4OhGSmDwWEJUHS2hn4JAgTFoZkcBUpPkHWySbxj1GAZOerPN0CsZ6HnG2jWakfFi4PQB6DSWdQLojVnfCaBPNMosZBhmQtGoVg2h52h8TgA2qq1kLCddU6xqSxOP64qddyiY5G06nJwiMlaHpCRIAYhqPdSQjHf46LWp6c7qtf/gjjXaWCNGCjI02wzObTdghKvfFwTbG2P0zy6NFjVE1dlSUI4bervM9wPQm4+2x3hw0lw4mmEoQ1QplsS4K/RHRLcS0RT/79cSUXMdx7ESnoBYnrsNwAH+8sh6/++VYBiGyYCZjTJ2552AHpyKagSkby8nL7OnkS1NXE6E5u3IO04YhrJMzA55RlDFdfVu4vJ3aOHo54SeuWX34ER7P6nEZ1Gli4zjNFhqLy7XVugvxoMjjb+4NHHVwClZPGFqr69JTXYDJ72ScfiwYIrG4xiPhf5OAjDJ//vH8BprjgoiyhNRC4AcgBwRtRBRHsDvARxKRBf56z8B4BFfYAwA1wJ4PxHNJ6J5AD4A4JrRjodhmL2DMPQwtpZGuUGG1WiouheVb48Ml/UaNlnSxCX5XLIHh8ibyMuu7qMJvSsxISpXr/siLNdGoOBjYbteM6ymjiktOygs9KgvDwsd+qJqyweESMmEkgaOb7Sohf7UXR2iwPgplS0hKuXvtmbvGBENThUenFCDk+zCGY9p4qsAfJaIboN3e19BRD22DYUQ12Y838cAXKG8fjWATwohriSiiwB8E8DPANwD4JXKdt+DV3/nUf/1D/xlDMMwqTTOg6OHGCYCei8q+8ykLpeGTFq6sZomboZkcg4FqeB2A8czgCoVoXtwAuPFP6cZokLosYhocJRjmJWDVWy9qACZRRXdXju/sL//pgbHpmFxiIJ7G6TVu1Jk7GjbqX8X/FndrsEJt2/J50AEdA0UtW0ye3AovK7xGqJKMnDeDuD/ALwE3mfh84gpjgjPw5KKEOJKAFfGrLsZwNKYdQLAh/wfhmGYqggnirE9byCYnTj2jV9Z1zNIYuruGaEeGaIyPDi2NPGEEFWgs7GGqHwDR5gaHN27YqtkrBYi1rKo3PDYgYFjueA4o4wyiIxDDY6+XK287FoqGcvxhgYy/PH5ISrFwFH3JAJyFK/BUevgOA5hUiGH7kE9TTzNFFENQ9U7lmWfsSbW7hJC/EcIcYwQYiq8e7i/EKLD8mOKgxmGYcYVjfKkmBNUVoQQeG7nQE3HMlSqYGvPUOp2FVcEdWfinrz1bCN5/GQNjppFZRMZJ3pwIDU4ZhaVXO8bDMZ+QujNNs3xyGPIVWu298MkzKLSl9uad5oEdWxiRMRCiNgQldc7yzfMHSkyjmZRkebBCfU5Vg0O9M/ipOY8ugd1D056ob/wmuT7qDb8tO+UvLpeZM2iWgRgez0HwjAMUy8aFaIqx0xwafzugY049Uu34e5nd9ZsLG/6yX047rO3pG5XcQUKsi5MzMRkq5VjanCiWVTxGpxcjsJ2DJaJ2XG8TCoziypMAbefUwgEXiPVYAD0DCc5rk3dUQPQ1PlIyDiejVgPjqIZct1wjPrxw3shd7fVwVHvJRGhkJfGkD1EpW7f1pSLGELZNTjZPTjjLotKRQixDsAsIvoUEf2WiK4jok8S0ew6j49hGGbUBE/CDaqDU21o7MHndgMAntrWV7Ox3Pl0NmOp4goUfA9B3NO8zcCJenBsxobdg1NwHEVkHJ2YCV6dHE+Doy8H4g1IV/HgaKJdGB6chJkwzKIyx1RF3ybDODKzqKwhKorWb7JVMnYorCFEQGI2mnkumUllbpPlmlTDMLUXVeLa+pHJwCGikwA8DeBVAAYBDAG4BMBTRHRC/YbHMAwzehqlhRmPlYyzFKeT3pQ4D47WdduN0eBYs6j8F6YHRw1RWTw4Mouq5LqakNhs9G1emoASxiLS9g0znJIDl/GVjCldryKNqEgauPfbFd59shkIQyUXT27t1bYPPTi6BkdWJXaIlDo49hCVerGyFk5km6RrQviZDurgjNM08aytGr4M4JcA3iaEJ80iIgdeL6ivADixPsNjGIYZPY0r9Cf881e3Xz3nA1cASZKJsisCj0Dc07zqCZHz6HA5XYMTVzQvLU3cm7gJZdODYxgfSQ0+yRiT2p8sKc05rtYLUfbGlOb1hg0xBVzX7sGxjUGO38yiamvKoXuwBNI0OLZum7o3aVJzLrJJei+q8BqyVjIe1yEqAEfCa3YZ3DH/768COKoO42IYhqkZjWrVICdr05vRSGwhIBXXFYFoNNbAUUNUMR6caLPNeGPBExmHoRXTcCDywjKlimt1B6i9nVS0rCHSVcFq+Ki1KYfLTj/Qeq1x4ZcsISrp7ai2m7jtXCq6yDj04AyX3cATZq+Dozc8bbOEqFIbBCihPWkIF1JExuM6RAWv99Miy/JFALpqNhqGYZg60KhQUeDBaVQ7ZQu2JozaeiECL0DcXGcrBjhkiowj3hQlTdx4I/KKBqfiisgYZeilVHE1D4MZPrI12wyMKkcfd2h8eK8vOGKe9Vpju4lTFRoc43pb/UJ9l1//CB7d2J2q0TLP3WQYODPavUYDQ6UKco7XQsHmwXENkfGkpqgHJ+s1OUQ4d9kcfPrCQzGroyV5p3FYB0flVwB+SEQfAvAfeAbZyfBq4/yyTmNjGIapCWG67tieV+pJ0oyKOOoxXFsISKXiIt2Do6aJxxT6qyZNXNXglFwR8XhJ8exzuwZw8xPbwuXGDTJH64mMPfKOg7JbUdbpx4hLdY6rZGxmZdlQq/6qyAaZ/3lmp3W9ie3c4TrC/118JH5293M4bP5kL5PKcewaHEV0DYSeH22bFH+LGu6d2taE1xy/ABt2J5c0aJR9n9XA+RC86/mRsk8JwHcAXF6HcTEMw9SMivKlPKbn9cNBjdIg2LCJeFUqrpsqMrb1dIp4cCxp4oFmxjheIUcoVpzg/GYUra05j6acg4e26lll0vMRhnz0c5YrYbitOe9gWAmjyZ5bssBgU0w53theVMhQByfmfVc7gHvnyGbgXP+OE3H7k9sjtW9mdbTg/WcvDpYVcmRPE4duLLXZNDjJEUyr0ZYW+r3iTytx6PxOHL1gWvLBa0wmA0cIUQTwHiL6CLwmmATgaSFEbStRMQzD1IFGhYgCDc44ClGlaXAqrkBrIdmDo3pYKoEGJz1NPCyaF/XgSD1HqRL14MyZ3BLUd1FJq2Q8XK6gxQ8HNRdyKCq6lJJ/H6TnKM6DY55DXZ7Vbo16cAqJ6yNj8E3C5+03Fc/bbyp+d/+GyDqVgtQrGXh1cMLtzXEA2T046mkrKUYzAOzsK6ZuU2uyanAAAEKIASHEo0KIR9i4YRhmoiANjbG2M+Tkn+H7f8xIM7ZcEdZSia2Do6aJxxT6S0oTt2lwpIFh0+DM7mgJxqSSVsl4qOQGgtzmvKNlekkDQB43LpMqDFFZNDhpxkBMaLSzVfct2LKQzj4kLDMXNa7sf0s8vZJpYEa9mJ02Ayez0Rb+bXrvbKi6obFi7M/IMAwzxsjJdqxDRdKwSs1MGUPSQ1RqFlX8NhI5iUebbYZ/r9rSg6e398VnUSmVjEsVN+Jx62zN2w0c2D04w+UKbn58K4bLlaDqr2fghBOxvA/yuPEhKjlm49wZRMZytXm9pmFhcx59/7XLg7/jsrDkOEyaclEPTlj3J1xmhsrU7eIICySGB5o3pRUA8N6zDordT62+PFawgcMwzLhie+8wnt1euwq+QDgJj7WdURlhiKqe40zNonJFMOlnSROXxmNSs83z/vff2N47HFs0L28U+ouIjImsBkiYAq6P6yt/X403X7sCuwdKaClID05OM8JkM8p8SohKelcirRqQ3mwzrv5Ss+HNSKsjk+jBsYWoLBocYdneauCkeKXUVg2S9uY81n7+/NhMNABoLoy9uZFVZMwwDDMmnPT5W1GsuFj7+fNrdswwRNUYD8740uBk8OBU1YvK+x0t9BfdOdDgWJaraeI2zZSt1ko4yUqRsfdKbVQaeHAKeohKenCk4RRr4BhGlLo87V21eU281/EemaQxZF1XsHpwRGR7mwYnc6E/i2GVTxioadSNBezBYRhmXFG0VWAdJW6DDJzxmEWVpQ5OmgZHfYuyFvoD9LYJKvmcE5yz5LrWwoiZQlSQmVHh8eXE2pRzUFTCX9IACIrVpWpwzGuhVGNAWMI5SecwCY26pBCVzYPjoFjWB2erydNpDVGleHBkqwbLkJOywRoRosrswSGiJgCHApgFwzASQvy1xuNiGIapGaEGZ2zPO+o6OHXIa7eW8Fdw1RBVzLhtzTYHTQPHsmuYRaUvzxseHPV+HTK3EwCCBqAq0TRx/7Vq4MgQVUH2aHLR4uRCkbF/3LgwURCisqWJZ+gmnqXRaty5Wws5lCplq/4n2NfqwbGFqKLGVmfrCDw4rvQE2Tw48T6TRnhwMhk4RHQ2gJ/CM25MBICxN80YhmEy0qiKwhMxi6pcpchYzqPdgyV9m4SWCrYQVSEQGYugFsu7zzwIl566PwAllOQQ/vDOk/Cib9wRzPTyeDZ9SJAm7nsQhksuWgq5IMsoznMjkdV+ZfVhidH5wYorRKr3Bghr7Zi0NuXQM1S2GFdk/VtiD1FFj2/X4CQzXHbhkD1kmOjBaYAGJ+sZvwXgL/BaM0wC0Kr8TKrFQIioz/ipENE3/HULiUgY6z9ei/MyDLPnU2lQiKrcIMMqiTQNjuuq3cSze3C6B0qx20jk/GcaWYWcg1yQJh6GqBbNmBR0vJYT6tS2Juw3fZJ2vEiISjEI1DRxIExnLwcenGQDZHZnC37z1hNw7rI52nKvDo7Az+5eh4WX34DBYjRVulwRqX2agHhR+SS/V5R5L1U7wuxUDtgNnHDfcGfTaPPGkvz5GCx6tYVshluyBmf8ZlHNBfBZIcQ6IcSQEGJY/anFQIQQ7fIHwGwAgwCuMzabomz36Vqcd29k+VX/wE/vWtvoYTDMmCG/6xtWB6fqE9dvoJl6UeX1kE/SMeTk2+V3sw6Wx3QFB6I6K6+jt7euVAlDVOpkrKZzh54ge5q4rsEJ08SBUAwdZFGleHAIwLGLpkXquJB/vu/88xkAwI6+6FRYrLhW7ZBJXOq+9D6ZxpOeRRXFK/RnanCi2hmbkZL2EDBYqlgNIwCBkWpjPIuM/wLgxHoOxOBlALYB+PcYnnOvQAiBHX1FfPyPKxs9FIYZM2ot9r3sFw/gF/c8l7jND/79LO54ekdNz1sLslQyTvfg6Nu7rkDXQBHT25qs20ikYWJO6ERhJeOKK4LzqoZKYODknWBSDz04MqQW3S9IE/cnZWngmFlUccRFmBy/knFSBKpUcWOP/4/3nYpTDpoRbGdjZofXRDPqdUsWGTfZNDhB9lMyaXWSBkthdWiTJA9O2n2uB1lFxm8D8HMiOhrAY/D6UAUIIa6t8bheB+BaEfWVrSMiAeAfAD4ohNhh7khElwK4FAD222+/Gg9r4jOe0lUZZqyQ39m1sjP+8shm/OWRzXjVcfHfMVfd8ETw93gycLLUwUnT4Jhp4n3FMlwBTJ3UhB1+SX65jfo1ngu8NNEJXa4rV9xgjGqoSXpQiKJdxAMNjv+bKMmDU9HGEJceHmJfTyQ1Nv65LfeqVBaxHpyDZnfg/MPm4t9P7YjNHPzaK47AL+99DkfsMzlybonNM2LV4Fj2VXn3mQfh8U3dWLOj376Bz3DJRaulCzmQrMFJq/VTD7IaOOcCOBPACwEMQPefCgA1M3CIaD8AzwfwJmXxDgDHAHgIwHR4mqCf++PSEEJcDeBqAFi+fPn4+VYZJ6TF3xlmT6SWaeJmS4IsjKcHi9Q6OCK90J9Zybir33vmnaZ6cIJ07HBbObma4RNvnefFKStZVI7mwQkFxWa6ufwtAkM2PH5Ug+NN/CX/HGkhpKR5WSAMk9k8Y6WKm6jxkcLbOK/J9PZmXHZGtDqwLXSnktSqwcx++uorjkBLIYcXHjYX7/v1Q9b3RiUpRJUW7htrsho4XwbwTQBXCiGSzbvR81oAdwgh1sgFQog+ACv8l1uJ6DIAm4moUwjRU+fx7FGMpydJhhkryjUMUXUZYtosjNS+qYddlBSCEEJAiNCYiO1FZYiMuwY9r8309miISvUkSEMkLkyWz3kGThBqskzkRITmvINXHbcfTj7QC/E4gRfF209tqtmsVDIGwpYSpbLebDOOuCwoGaKS2AyDNA2OHFNa2DAyppT1Xh0cs/CifduXPm8fZT97F3KVwWK8gdMAJ00iWc2tKQC+OwbGDeAZOD9J2SbwttV5LHsc7MFhJgq1zDyqpch4V3/1XZFH6sGpR/ZVJWEyleMMNThx41L2ESIw+qZOatKWA7qxIc99yoEzsWhGW+S4BcfxWjXIEJVFg+N5cAif/a/DcMS+U/xlvkHmb6tWLFYrGXvr/CyqoJt4usjYupw8gyr0Stk9OEnak5agNk9173Na5rnVUJEanISdk7KvJIOlSmzKd5aU+LEkq4HzOwBn1XMgAEBEJwKYDyN7ioiOI6IlROQQ0XQAXwfwTyFEd73HtKcxntJVGSYJWx2VkWLTg4wUaeDEPcUKIfCXRzZpy0Zs4NTB45r0kCPXSb1L3LbqeyOEl0EFQBMZy3utTpjSyzF5UgG3/c9pkePmcoRipYI/PezdPzVE1ZQLNTgm0Wab4TnDXlRGiMofS5oGJ66NgqyDI9fadDSlSrwGBwCact5nqFSuzoOT1trBrsHxs6hS9jM9PyZDCSGq8UbWENWzAD5DRKcCeARRkfFXazSe1wG4XgjRayzfH8Bn4RUa7IEnMr64Rufcq2APDjNR8Jo+1u5YQG09OLYiaQBw02NbcNkvHtSWjdRQqYd2J+mYrqLTKOQoqBUT2U4r9CfQPeDdk9OXzsLXb31aO8+w5sFJvp684+Bnd4fZaVqIKkHLEhb6871GFYsHJ69nUcmJPK3QX3IWVVjIzxb6K5bdxBBY0GC0yhBVWuzCrsHxfieFkZos6eUmg6VKrMh4vJHVwHkjgF54qeJmurgAUBMDRwjx1pjlvwTwy1qcY29nPIkdmfHP5u5BtBZymKKEHsaKWn5Wa1nob/dAsoFjVvRVz5+VUCxb3diykKTBCUNUhLwTH64wu4n3DJUBAAfP7cTaz5+Pkz5/q1WDYx7v2EXTcO+aXcFrM81YtT3CEFV0hg7TxL3XRUUIHhEZl8IQVc6hEWf3kH8+ubftXqVpcGSbiDSjwnZuID71upAniwbH9+AkhqhGp8EZb2QycIQQi+o9EGZsYAOHqYYTPncr2ppyWPmp88b83LX0NtayVcNOPw26rdn+9Zm3TDoj9uDUwcJJ9OD4c5vjkD/ZpWdRCSGCybRZSeWW91r1ppjG1a/ecryWkmuGi+wi4+h4zErGmganoKeJD5Uq2NYz5IeP0o2bWAOIyDubv9oeonKDSsw2pIGSZlRExuRfcEuMFqYln0Ox4moeprQ0ccC7x2W/rlHcdQ8l1MEZb4yvnC6m7rCBw1RLv6UE/VhQUw9ODZttSg9OnNFiK3Y2UTQ48j7lSIYrMnhwXOGlQ+comExzDgXblJSu1mYoxlGabALRe5ezaHBsBFsFHhw1RKVnUX3yz4/j2M/egh29w6nhKe3YBo4UGfuvbTqaUooHRxp0acX1ImPyTxoXKgoF1eGYshT6k2ONq8sDAEMJdXDGG1mbbX49ab0Q4t21GQ5Tb1iDw4xnTG1HrahtiMoLQakTt4qt2Nl4yKLysn6S9R5yXS4lRBU0qswRXBGdyB2ioLhisVKJ7BeH6f1ybFlUFheE3E4e3WrgyJoz/j3d3jds7VBuEufxkK0aJLZr8wr9JWcteftWmybuHTMuVGQ2FgXs3cRNVI+SzUtTrrgoVtw9K0QF4DDjdQHAUn//B2o6IqausAeHGc+U3OyC1GqopchY9gUqxRgKqoFz/mFzkc8Rbn1i24jOVct/rgTPAEjyFshLyjkOCvn0EFVTzoHriki2kKzyCwDFBA+OScSDY6tkbNlPLpPnVD0XckI3PUA7+4qJrQXCY8dlUREEwhBQbJp4QpPJDj98tdCSMp84Jn9IcaEivWpzAYDiwUkMUclrsb/vQ/59jQuNjTeyanBON5cRUQuAH4L7RU0o2MBh6sWjG7pxwTfvwL8/dDr2nTZpRMdQv1jHa5q4dN/HPXWrXtJD5nVid3+x6muRm9fy3yv5LpykYwYhKie5Jopc3pR34Arhi2nDmTNHlEmDYxLR4NgqGWdIE7elOjsOoSkfpkHv7B/O1AgzVoLjny8pTdy8LyazOltw7RuPxZH7TUkdh3bulBBVi9F3C1BDVAkeHN8Yi3vfpXE/UTw4IzbDhBBDAD4D4KO1Gw5Tb+QT1Dirx8QA+NZtT+Obtz7V6GGMGFn75Q8PbhzxMVQdQ6VKXUIScmKtRYhKZujEhahKRngk51DEqPjMDY/jyj/FN7yV4YRaGGQS+U8+KUwtjRKHCIWEEJU8RnM+h4rwrtkMUYUaHDeyXxyml0Wt9yKNHft3lxGiqrg45aAZ+NHrl2tbHTZ/cvD3zr5ipEO45Fuvep556AjyGkdT6A8ATl08E50thcRtTOTnKT5EpRc1BNQQVfxxpTEWVwtnyM9A21tExjMBtNdiIHsDrivw9p/dj/vX7UrfuG5jaNipmRRueWIr/vnk9kYPY8TsN93z2jyb0qwvCXWSqIcHpxYOEfnlHxduUa+huZCDowhuJd//9xpc85+1seeQh6hPFlWSBkd6cCgxRCXr40xqzqFYrkQ1OA6FKduaByf5C8j0SKgenCQPhNwsDIu5WDy7A2csna1tJ1s7AN61xoWozlk2OyhaGOvxIF2DY/NOpRX6GynSkzIpTmQcZIx59/sX9zyHr9/iPTyl1cEB4j040sDZ00TG7zcXAZgL4BIAf631oPZUdvQP48bHtuC+tbux4mN1LwxtJfDgNOTs44tv3fY0DpnbidOXzmr0UAAAgyXX2hl4oiCftkdj4KiTYdJEXC0jERn/4p7nMHVSAS84bK623CwUZ1JSrKjmnIMcRT04adTSIJPIJ/fELCrVwEkMUXnbdTTnMVhyIynXjpImbqtkHEdL3jRwwr+T0pwD4ay/0XC5YvXOXHrq/tjUNYjr7t8AIL5NAyEULsd9ZsIQVYIGp5ycRTVSBn1DoznOgxOEqLzt/t/vHw3WJYWoQtGz/ZrleasJUX3g7MU4ZfHMzNvXkqwi43cZr10A2wH8GMDnajqiPZikD9ZYwRocjy3dQ/jS354EAKz9/PkNHo3HcKmCOnwXjhlywn92e9+Ij6FpcGrobZSf+2ocInJSMD8fRaPUv4kWoio4gTdDrUmSRmDg1PDfqzxUlkrGaQaOPEZ7Sx47+4oRD46aJj6shaiS31Qz9KGGqGS4LklkLCAC0bMtNNTWnMcXX3Y4/vDQxsQ6OI7f2Vy9VttYuwbCvmSxGpyECswjRTYMjTM0WoKihpb7nVIHB6itBudNpyzCpKaspkZt4UJ/DaFxRgYbOB5/9nvdzOpobvBIQgZL9qfOiYKc+Hv9qrYjQf1irbp8fQJhHZzaiYwzhajyTpAJ5AqvvozKQLFs/fKvZVo74BkHlcCjksGD47dqGLJNkAgzyNqb89iwe9DPFlKzqMI08VIVImOziaMaojp4bieO2m8KrrhgWWQ/VWQs35+4f0tEhM6WAnb2F+M9OBSeO+47s7O1gHU7+wOjzKbJyqLBGQkDRe/fWGyIyiIyliTXwfE1OHEGTornCADecsoiTG4t4Mt/X+2fr3EP9iO680SUJyLW3lSJqIPbuVrkP9bx1vV1rBBC4A8PbsQDz+0GgMQqo2PNUKky4QzQvuEybnhkM4Dk4mBZKVbRt6ga5LFGckxzH/lUXKoIqwhYDQE153OBV8527rjO5GEWVdXDtaKeOks3ccf34MRpZqSh0tFSwFCpEtGa5EjpC1VWQ1Qj9+C0FHL4/TtOwpF+B3Hbdq4IJ/WkcO/kVk/UG9dok4jwsfMPwZRJBcyMeQia3JpHz1BZMRz1aytXXLgivVv5SDjrkNloyju45LgF1vVq1WaTpEadQR2cFJFxkgfno+cfgsvOOCh43cipJvHOE9GZRPQKY9nlAPoAdBHRTUQ0pY7j26Ooh2CwWvb2Qn+3PLEN7/31Q7jxsS0AgB19ww0eUchgqTKu3p8sGTwfuf5RvPMXD2DVlh4jJXVk16Fef60MnBVrd2FT16A/rur3396rf0aKKZoSs8hckpZjd3+0bxUweg/OYxu7sa13KHI8IJsGJ+8bOMUUkXF7cx6DxUokHdpRdEeacDzlPY1qcKqbHW1tI2x0+AaOaXx88Nwlwd/nHToHD33inNiMoc6WAroHS8H9NA2csBhi7Q2cfaZOwuqrXoAlczqs621p4pLELKqU3liDIxAZj1sDB8DlAPaRL4joWHhdvX8K4EMAjgCniSfy3M6BIE4rn3pqmfpZLZWEOPbewK4B/Ym5Z6ispVI2CiEEhkpuTcMyoyWLgbFx9wAAoG+oXBPvSzWTYVZ+dvc6TG4t4IWHzYkYDNt6hrCleyhmT4+NvnEkKaZoSvQsKkfxLlg8OAN2D85oQ2ov+sYdOOsrt4fHy2g4yvOGvajihdSFHKGlkMNQ2Y3Joop6cF5/0sLEcZsF5LIaOLLx6ebuoeDfc3NCgb3JMQbOO08/MLMmb3Jrweuk7jdXNT2Y8nWWfle1xpYmLknrRQUkaXCStT82xnOI6jAAtyuvXw7gP0KItwghvgrg3QBeXK/B7Qmc+qXbcP7X7wCgf1n1DJWsXYfrjawtspdGqKzyJ9k8sZHIJ61qe9LUE9XjGDcpqjqFYhX1TuIo1SFENVCsYFZHCzpbChGD4aN/eAwf/O3DifvbDBw5gdh0F5EQFcVrOXb12z2IogaaoR5FC6W+l1nq4HganGSRcc4htBS8wnlDJbMOTliSQnqB1nzuhfjguUsTx5wUokpiwfQ2LJndgT89vCkUwiZ4GWb4KeCjMT46fSNJhhmjHpxkLVA9CQ0cmwYnKYsqWYOTJUQlufjY/QAkp6XXm7Q7PwWAWmP8JAA3Ka/vAzC/xmPa45BfkPKLRQD48G8fwQd+89CYj2U8hMkaibBYOOPBwJFfyuMpRKVOyHEp0XICqghh9BwamSeqWAcPTtEXwBJRRP+2u79o1cGoXtatiofHdb2qvVK7Zc2ciQtRudFj70oJUdVKg6N5cBKMaK0OTs6JNbhLFRcFxwkmur7hkl7J2PDgNOWcTLq/kXpwAOCCI+bi/nW7scH/vo0T4ALAEb6OZ1vvyEPU0gskMe9VKfDgNMDA8d8XmwYn6W2QGpzdMdowGaJqaUq/pqsuPBSPXnlOpL/YWJJ25s0ADgAAImoGcBSAu5T1HQDGj4hhnGF+ycunI9cV2N47jO0NmFgDkfFeGqRS7Tv53TkedDhDvis5rRDaWJLFwMkpk7fmwRmhJ6oerRqKZc/Akd2fVYbLrvUpVzVcVDe/XN7mGzhx1WslTXknyJyS16Ne47Yee3hMblKrcHZWDY6rGTgU+yRfrgjk/RAVAPQMli3NNkMDJ6unxPQM5KpwNS+e7elRVm/pBYDE1OTlC6cCAB5a35X5+CZm9WHzXknvXiMMHDVN3PwMJXnF5Fgvv/5RXP/Ahsj6oVIFRMmd3SU5h9BRZYXmWpM2yhsBfJGIzgDwBQD90HtPHQ7g6TqNbcLTZ6TLql8spYobq1SvJ+PJQ9AI1Kuf3dkCYHwYOOPdgzNcseuUghCVqFGIqjL6Y5hID4Jj8eAMlytWnYJ6LarQ1jRwTEPunK/djp/f81zw2iGKpBur53vwuS7rmOWkNBIvlq12jm7gJGRRZayDU3Zd5HOhB6d3qKRNeqq3bKBYxqSM2Ypm+rFThW0wf2orAOCpbV4dpiQPztI5nQCAeZNbs5/AwPTgXP/ARrzv1w8FrxupwcnnvBYhNgNe6pVsqN3VbVXVB4sVtBZyEyYLN+3j8wkAQwBuBvBGAG8RQqhuhzcC+EetBkNE/ySiISLq83+eVNadSUSriGiAiG4jInt+3DjCrAdSUUJUw2W3Jmm11VLLwmETEfVhRoYZbE/wY42sNzKuNDhVhKjKFf3zPFKxtDqh1uqzKkNUDkU1LUMl11rrRfdGKd6cksweygXHVlm9VS9y2GTJopLHbso5eGhDl9XAGk33c1uXczejBidIE6e0EJVAwSG0+EaEmQ6dUyoZ9w2XM5djMDU41YSo5k/RDZy25ngDJ+cQ/nTZSbjubSdkPr5JZ2v0mn6v9GGTn+VGVSdvzjsYLlciYaqpk5pi91GNMVvH8MFSZcI02gRSDBwhxA4hxKkApgKYKoT4vbHJywF8qsZjukwI0e7/LAEAIpoB4HoAHwcwDcAKAL+u8XlrTs+QHl8PvmSE78FpgIETfLlNDAO85qhf9PLLtFZhoZ6hEpZf9Q/c/ezOqveVsW05wXz+xlW49NoVNRnXSMlm4ITraxGiqoUXyHbMQINj1rQpVzBs0SkMx4wjS4gKAKa1NeHblzwPsztbIiJjeYyTDpyOYtnFYxu7I/u7oxAZq/deeoK09Psshf4cQj4hRFVxBXI5CkIhALSKvWqzzf7hcqKxodJiGANZRcaA51GZ1JTDM76B05pSPffwfaZg3pTaeXBMGqnBAbzvt+Fy1ICfMil+3KoXzpYeP1iqTJhGm0DGQn9CiG4hRORbQAixy/Do1IuXAlgphLjO72J+JYAjiChZkt9gpIEjLV5VZFyqiIaEqCrciypAPlnVaiJ9dEM3dvQV8b83r656X/mUJT0f3739Gfz98a01GddIUTUwcRNdzo8hFCu6K3ykxrveqqF2Bk5z3gtRmfZCrAZHDVFZ/o4LUUlmtjfjhX4PK8cIUcljLJrh1Uq1idylE2a0Bs6AH/p0tRBV/DHl/c8RoSmxF5UvMlbCQHHNNvuHK2jLWKp/pFlUgBcWmz+lFX3DfpXfOk/EafqSRhs4zXkHQ6VK8PAkmdIa78FRM77ylvjgUKkyYRptAqPvJl4PPkdEO4joTiI6zV+2DECQyymE6AfwjL9cg4guJaIVRLRi+/bGdmaWISoZC1a/ZEoVN7aIVj0ZRxrWhqDe8RbD8Bwt8jgj+UKTBo4rxk8YUZ0o40XG3u/hklsT78sGv64OUDsDZ7gcH6IaLnkGjinEjAu3yWtsb4rPogJ0b0bOqIMjDSoZ4hiy3NvKKDQ4aohKlqLQCyjGfwnc/exONOUcLJgxCYWcAyHsYzBFxgAsaeKjC1F95AVLq06xVj0ykzJ6jUaKTJOPo9hAkTEgQ1RuJESVqMFRxirbQagMldw9J0TVAD4MYH94qedXA/gzER0AoB2A6cfthpfFpSGEuFoIsVwIsXzmzMZ0MJUEBk6z4cHxBZnFBhSYCzw4e6sLRwtR+R6cGll98jjVVl8FoD1ljRehsWoMpGlwhssV3SgYofH+5JbeQAdQyzRxma6teqWEEEH2munFiQu3Sb1MWohKffoNq8O62rFlFo4tRCbv/UiSqNTxSi9yliyqiivwl0c24cyDZ6GzpZBY9K3susgraeIAYtPE+4vl4H6lIf9NTm9rwluff0CmfVTm+IkDOYfq0gPKxMykUpGfraY6NNvMQnM+h+FS1MBxEr6f8sq63uGogSNFxhOFcWXgCCHuEUL0CiGGhRA/AXAngBfCaw3RaWzeCaB3rMdYDb3+l4t0z7pKiKpYcROb3tWL8TJ5Ngr18mXGRq3eB3lvba7dNNQ4+XipZqxpT+IMHEcaOKYHZ2TXsGpLL5bNmwygtiEqrw6L/v57vaS8v00Dx5YaLo8FhCLjOENONQ5bg5okrnYu+SRtC5HJ74oReXCU8XYPlCLjiRPVb9g9gB19RZy2xHswTCr6Jjtxqx6MgpFFVdE0ONV5cEZa4HBGhxd+GatMH5s3pFTxPIJX3/4s2ppyWDC9re7jsNFSkCLj7P8W1XtmZgH/88ltuOvZnZGGqOOZ8T5SAU8ushJeWwgAABG1wavPs7JB48qE9ODIf7QV5amsUSLjidbMsdaok3bB8VJ4a2VQyMluJGmh49KDo6WJx4SoSDdw2ppGbjT2DJWwsWsQy+Z5zzI1NXD8rt5Cm+gr1r+913YPjqnBUf8NixiPlzQC5HsszyU1HDaDYzStGtQxyWrGmhfK4jECwoKk+06dBCA0WGxGXFjJ2B6iyil6Jy9ElVVkHGZljYTpbV5jzDiDvNbYdDhdAyXs6i/irmd34m3PPwAz2u3NOutNcyGHgWIl8CRVS5/hwXn9j+8DUF2bhkYzbgwcIppCROcSUYvfrfwSAKcC+BuA3wM4lIguIqIWeOnrjwghVjVyzGn0+PFvaRSHImMvRFV2xZjrLcZrob9SxcWtq+ovqlXDUTnHqxVRq9RsOXGNpHLnUFExcBqUKt49WMJdz4QZYJk8OGoWVcUNMlfMsN+tq7amGvQbd3sT7P4zPfFtzQr9BWnieh0c9cl22HjKjfNGDRsGzk1+01ZAN+pUr0erUVW2aGpwbCGq0YiMlYuU30HyOIUcRUSnEnn/ZT2ZpBBVqeLVwVENnCZDg1NxBcoVL4un2hDVSAsczvA7f49VCQ6bB6d7sBhon/aZNvIsrdHS3pxHf7Eca9DGIcN8qgdHfT9YZDwyCgCuArAdwA4A7wJwoRDiSSHEdgAXAfgMgN0AjgPwykYNNCvSgxPUtAh+h08oY10LZ7x4B0y+cctTeOM1K/Cv1fUVhqvXn3cIBYdqdk9kxkp+BBqcIc2D05gQ1Tt+fj8u/v7dQWg1S5q4nNMDD05zVLh91zM78cZrVuArf0/OLhs2wz81eF8qrkDFFWjK5SIi46wenGJZRJZL0eyvV6zHmh39AKA9KatGgTQCZDFHeS87muM9OGGaeLbrVFHP3V/Uv4PamvOxBs6mLq+q8pzJ3gSXlyEqy/jKfrPNOA1OU97rUdXvX3NWkbFskDlS23ZGe3yGUD2Q/ai++LLDce0bjwXgeXCk5ywtlbyetDXn0T9cXYgKAO7+f2fiZUfvo3lwegbDv8dTra40sn3qxgDfiDkmYf3NAMZ1WrhJX1F3D8svbNWoKVXcMa0rII2s8SYyXrfLy57ZGdN8sFaoX/65HCGfc2omMpZGykgMHHXSaVQYceWmHgChJyKLgSPv3XC5gmLZxRS/iJhqnOz2O2av9Q2BOOQ5Ah1GDe5DUFQv72Co5GluhBAgIs2wMCcBuV9rIadnURl1cIDQOFKNVLUJZ2DgBCEqvwBcwQmKsZlIA2cknwV1ApKTlHw/2pvzQUdok01dg5jZ0RwYGU1JIuOKi3xzXg9RKRlPUyc1YfdAEf3++bN6cKR34OXL9820vcnMMQ4HdfoeHO+z7xkzXQOl4F4kiZDrTXtzDn3DZe1zedbBszLumw8edAC94ezWmPYi45FxY+DsiUi3t+nBURlrofF49eBIk6DevUDVL+u832+nVGsPzgg0OJrI2CjUNtZl0eU9ylIHR35+i4YGRzUag/fW1spdQRoVsodQLT6rqoEj/xbCM/DVsFRcFlVbc96qwZmueArkMvV4ugfH0dbL7ZvzvoFjecIOKxmPToMjDQx5nPbmPLoGBq37beoeDKoBA4oGx5Ym7grkHdIyBlUNztS2JgyXXWz3m1lmNXCa8g6e+NR5I67+O9Z6F6nB6R0qBxWCd/UXA0OtoR6cpjz6h8tBde27PnJGZgOwoyWPvuFy8P2jGjhbJpCBM55CVHsc8slMPgHaviiSxHC1arSnMpoKqfVETuL1HpY6WeUcQt6pnQdHGjgjuYY4kfFYenOCjCLDMAfsOhFvmzAzaLjiBj2HVMNd2mdp90V2I5dhj6R6LVmRPbRkHRwg/OyrISVTpxBqbXLWxpvzp7Ticy89DEBoUKj3SP133Wp4cOTxmvIOmgu5GA+OPtZqUO99/7BeIVuGqGzfLZu6BjFvSkvwOjFE5dfBAbyUbkDX4EzzJ/v1fl2jrCJjwPPiJKUyJzHWBkWH/3nvHSphVmcziDxDUWpwOhscohooVvCjO9cA8LxqWfWB7c15uCL8zG5SDBzZiX0iwAZOHZFfkklPY3HCy1VberDkYzdh/a4B6/qRUg68STU97KgJn/Lri/rlL59AaxVTHjImsGpQJ1jV4GqEx01OuKqBE9eQVI5PanAmBcUT1XvgG68p5w3CQk3SwKl25PHHbM6pPaG8dckeHL/eTVNeez/UPlKLZrRp+2qiZVVk3GQXGTflHC+V1+LBCR5ERnAP1HsvQ1QVJURVcYX1M9o9WMZkpcptUoiq5DfbBICZvrBX9VxO9Y2e9bu8iTFrJePRIt/jtjESwh402xPEy9DezPZmbOoaDOoPNdKDo+qerrrw0KqkENIztdsvM/Ds9j405x38432n4ksvO7y2A60jbODUEfnFJycB29N43GS4ftcgihUXm7tr6w6UYxgvtVYCgqf8+k7o6nXnHAeFXC1Fxt5kMpKwY5wHpxZju/nxrTj7q7enZjHJe28a5gCwtcdu4MhjylDIJEt9GHnctLd2WNG9eOcf/WdUDVGR4cFJEhmHWpuc9h5IL11rUy6osivPob6HWogqb2pwQq9Scz5nTxOX3xmjbNXQbxo4vmZkyKLD6R0qBZoSICVE5TfbBBQDR/G6TGvzJsjndnm6q6whqlrw13efglv/57QxOde5y+bg2jcei9eesBCAV0l5Y9cgegbLaMo5DWu0Cej3XNY2ysrBc70aug+v70LFFbhp5RacctAMHDS7IwghTwTYwKkj8qk8NCqyh6jk8lrXylG7FNfbmKgGyviUP1pMD04+59TM2JOT30h6jGkGjjLGWoTPntzai6e29QXji0Oe1W7g2A1tuY1MKW2z6GdCIz6rBie7B2eoVMF/fftOPPjcbvsxlXCQY7RMUD0ucSLjtua89m90oFhG3iE0553Aw1EMPDjh/VX/aTkOoSnvhCGqQIOTixcZK1XPqyXsgUS48bEteOXVd4UGjv/+mJlUw+UKhsuulvacHKIKPTizOrywlgzLAGHH6ic2e7VYZ3e2YKw4ZF7nmJ2PiHDq4pmBFmn+1FZs6hpC92AJna2FMdfPqagNTmV9oKwcOn8yWgs53LtmF+5ftxtbe4bxkiPn13qIdYcNnDpSNDw4dpFxnHhTFyRWw7V3rcWHf/uI9ctRnbTinAO3rdqGD173sH1lnaAxilFpWVQOIe9QzYTecoIbiVEalyZei7HJ8aSOS5YukAaO//mZ1dGMbbEeHG8b6ZKfZBEZS4MpXYOjZ1Fl8eDs6BvGg891WTtyq9fSlFM1OHJc6Wnik5p0D07/cAWTmrwquU1BCwZpMIXHMNsEtOQdfO/2Z/HDO9YEYyrkyG+I6OK9v3oQ161YH2wvTzmiLCp/HxluuvvZXZHsL9PAkSUtVM2IWb/HPIf02Ozr13qReh8gnFBXbupGU94JdDp7OvMDD04pqHPUKNQQVbW1awo5B0ftNwUr1u3C45u8f1vH7z+9puMbC9jAqSPmk7DNgxM36cgvwZHoOT771yfw6xXr8RvlC1OiTp5xX553PL0D1z+4serzjoaxes5RJ14vi6r2IuORvGeDJTd4CtRDVKMfW1YDJ/Tg+J5Hf+KeN6UVW3vtHhw5vi4/Vj/ZT5VVM9MCAydlnGo4KW/0jYojyOKKMQTVY5oeHDU0ZMuias47ERG62nYg8OD4QmbZNPN1JyzAn991snY8OcF8+i+PY3tfEZ0teRB5lYCHyxX84aFN+OBvHwm2DysZp96CCPJ9lmnLAHDrE9vQlHeC0MNg0W7gqB4ceZ39Fs9f2Q1Fxm97/gF4x2kH4OJj9wvWd7TkkfMfHuZPaR2xaHiiMW9yC4plF09s6Wmo/gYYfVhw4Yw2bOkewtqdA2hryo15jaFawAZOHQmyqPwvHJvIWC0ipi3P+tRt4ZC5nQCAFWujbnv1cHEZGsWyG1QhHWvqnd2lTry5nGzVUNs08ZG8Z8OlipJirYaoauHB8Y5RivmsSSIaHCENnBZ0DZTsT/L+saUIWYYmNA9OSWaXZQtRNfmNMbO8L2nGm83AEf6mmoFjhKhkB/JCztG8aAPFSuClMjU48v68+ZT9sWSO3gdYFXj+4cGNQcXd5rxjDR0GRUFHkUU1VTFwfr1iPc5cOguz/NBN1IPjGaiy+CAQhjgGLE0XSxU36LnWUsjhQ+ct1bwEjkPB+dXMrD2dkw+agUKO8Oz2/obWwAGyF1eMo6M5j96hMtbu7MeC6W0NDbeNFDZw6oiZbluNyHg0GhzpKu63tLuvZPDgDFs6LLuuwLduezpo3lcvalWTJo6oB6d2WVRmpdqq9i1VgswF9X35+i1PYXd/cVTjqtYbWDQ8j/MmeyEIWdNEpWx4J0MDJ6rByerBafY9OFkK/QX/TmLuucxmsqWJq5lrpvG2tWcIM9qbvTpJRmVg+WRcMDQ48ni2bBX1fgyWKkG9luZ8Drss7+9oyjnIz7g5wZ6+dFZs2MnmwZFiUqsHx2+2mcRBszwjT35+9gYOnNWBL1x0OA6d34kXHDqnoWMZrQenvTmP4bKLp7b2BRmDEw02cOpIliyquC/mrBqcPz60MZLCK1ND+4btrmVJXAhAjls997+f3oEv/e1JXPGnxxLHM1Lkw8FIBLrVoNfB8UMQNRIZDwYanOonpaFSJXjiKinjue7+DfjI9Y+OalyZNTg+ZmhVlu63pYqbXr5pbdFKxra6OjaKFRdEfvo+1diDo6WJp4eoVm3pxZLZHcgbWXYDwxYPTqDBkRqi6Neq2bhQuvtbCo7VcAwfiqyXlYh8SDD/fe8/oy2syaMYLd0DJfzwjjUA9OaR8jr7LR6cspImHscLD/Mm+N11figab7z0efvgL+86Ba9UQnaNoK2K2kM2ZMbdxq5BLJg+qRZDGnPYwKkTrlJrIsmDk67Bif+S7x4s4T2/egh/MPQy0t1s+2JSn4rjnpDlhKR+6UvDQ82UqCUyi6reHdaLhgcnn6uNyLhYdrHN16mM5BoGi5XgC8UMl3QNjs6DE7QKSblOudb04EivTM+QLVShH1OmB2u1YzIaWMWyi6acAyJCLkeZBLZZNTjNeSdwscvDDhYrcChsSigZLFawdmc/ls7t8EJUyr+D/mI5yBRrjglR2Tw45r9F1YNjM+SkbTKSLCp5700vzcIZbWht0jubA8Dnb1qFW1dtA6B7cAo5B015x+oJVkXGcbzo8HmYO7kFbzxpYdXXwIye0Yao1P33mcoGzl7H/et2xad5V8Inx7IrIISwp4mnZFEleTTkF5ieniqCp0X7k5ew/q0iQ1TqtfnhdqvoUQiBu5/dOaq088CDU2cDx6xkXKhRmvjKTd0Y8oXCIxEZD5Xc4AvFzOgZbdQu8Aam1sGBdn5pYEivjM24NY0Q2YuqZPHgpDX9k7oXAMhRVgOnGg2Ot0x+TvuGy2hvzmNya0FrJrh6ay+EAJbO8Q0c5fMxUKzEhqjW7OjH5NaC1rJAIv+tySfhwMCxeHuA0PsykiwqeS9Mr9T0tqZIXyxAN6LMsFZbUw4Dhie44no9vaQGJ46pbU246yNn4sQDZ1R9Dczokd6695510Ij2V43d6RNQYAywgTNint3eh4u+cxc+9ZeV1vXyS10WPnOF3WMSZyANWyal7oEStin1SEIDx8WOvmHs6BvGYKkSTIimWxww0sRjDRz5BRl+scmn3+7BEjZ3671svFobd+OX90aztp7a2luVWDnJY1ULVGOmFpWM1+8awECxjPvW7gIAnHjA9KqNNFlZVnpwHt1gpDyP8pZk9aDIXlERD06CgWMah50tBeQdMjw4UUM8bpzSK5Jzshk4addmr4PjresdKqOjpYCOlnyQ5g4Aa3d6xekOmNnuX4uaJl4OXP85//NTrFQwWKzgbyu34LxlybqLfaZ6ehQ5YcQVghtdLypvn5ccMU9bTkRWDc5cRSPT3qI/9U9qykc8OPJej6TnGjN2EBHWfv58vPesxSPav10RnE+boGn+bOCMEOmuf8ScjHyGlVLvgDcR2NPE7V9gMuNF9eCc+PlbcOxnbwmOL2P3w+UKll91M5ZfdXNQcI3I7sFRJ400DY76BCgnvYfWd+GEz92qbb/Fr7b85JYebfmu/iLO/tq/8KHfPoK+4XJQ6dd6vTLTp84enFLEgzO6LKpTvngb3vDj+/Doxh7sM7UV8ya3pmYrmcjJRva1+YGvh5CMpJqtSuY08cCDo2dRyb5CPRYDp1QRWvpoU96JZKaZTSbjkCEqwDM+M3lwUsT4UvjbnM9FRMZ9w6XAg6Mab/LftvTGSA8s4P2bUiu5NvlZVnev2Yn+YgUvOmJu4njNAnRx5fPle5F2C3b2DUfukzTIXnPCQqz69HkAgIW+50iOXdXgVLTq3rrR0tYc9eDI9zZNZMxMbFRjV4apJxps4IwQGX+O9cBEqrKKmDTxivUY8qlXz+AIv2he9f178LLv3qWdCwB6faNmTmeLVnhLohk4VWRRJT19twRPhfp1yAZt1z+4Ee/6xQP48O/ixbJBQ9IqDZxqM5Y0D05udM025f27Z80u9AyWML2tCYU8VW2kBQZOiz1mPtrU+azGY9hXqqK9bm3yKu7aDJyKKyKTtldbSDFwLJV+bRSVEFVBqfybRNjN3H6Puv3Qk6w7A6gGThntLXl0tha0a5MPBm3N+WASL1UEXFdgQEnnB8Iu5Wu2e14fWaLBZKmfNn7comkAgAXTvKwU1YOjGgyVDFlU3QMlHH3Vzfji31Zpy8uuJ9bOOV6dnYevOAc3vufU4Hx5h7TWG0n3ua056sGR/17SQlTMxEbV4LAHZy+iVHGDtMp4I0GGqMLS9bZtv3nbM1j8sRsj7n/pBYgL2dy/bnd4LsWwkCX1Z3e2oFhxI3oO1RsQJz2xPXGbwld1spJf0kPGubYpxeHufGYnnt3eZz8hwqfOagS/q7f24pBP3ISnt/Vm3kf1ruQcZ1QiY/XeygJwhZxTtQZHTjCqS1ilZhqcBM9SueIGn0/5vssQZt4hzwgYsnlwXMydrBs4XuZR1Ps3lMWD43+WFs1owzO+0ZC4j+VBQKVrsIiOljzyOSVE5W/aN6RqcHQDh8h7OMkH/ZhcDJUrECL8Nw14xtxw2cXanf3oaMnHTgTXve0E3Hn5GXjF8n1x03tPwckHeboU1YOjVj9OSkwAPN3M7gFPfH7DI5u1daWKQEExPia3FoIaNY5DOOWgGbjpsc3B+ztYqqCQI9x5+RmR87Q15SN1euS/F/bg7NmoD1yNLlo4UtjAGQHv/fVDuPj7dwNIF+rK7sqVit3Akam3ajt6INQO2DwUpphXNSwe3+SFieb4T9WmFycuRHXln1bi4I/f5I89qsExjZe7nt2J5336H9jWMxRMZtFaIuFTYrHsWut9SLIKYVWe3taHsivw9LboRHjbqm1YePkNkW7sJUODUxhFmrjq5u/zDRwvZFGtB8fb3tQ/SEbbMyxLTSVbyrT8bDsORcI4krLFg5N39OJ4gUcwgwZHGjhL5nTg6W29qfcyCOXGbNc9UAoq+pohql7fg2NeW9+wlylFRIFguFQWwb8l1YPT7Htw1u4cwMKEYmgdLQXMn9IKIsLSOaGXR/X4yH+a6vsd99Z/4LqHcdqX/2ndxusTFW98XHjUfGzqHsJDG7oAeJ+/WR0tmD8lWq9mUlMuEuruUzxczJ6L6sExQ5cThXFj4BBRMxH9kIjWEVEvET1IRC/w1y0kIkFEfcrPxxs1VtWtHBv7D5r1+dVpYzw4ErOYW5J40syOUL0rj/o9eWTtEvPLSQ0dqOO55j9rMViqoG+4HAlRfeT6R/GJP+pi6vvW7MKu/iKe2zUQTNCDhpfHbNC4s78YO1nLybSaOjjSYLIZTr++zxM8mxopM4sqN4pCf6prv7/oeQPMyrdZGAo8OHY9xuhDVOkGjmqcXnvXOnznn8/oHpyWfMTAEcL7TJt9hrziidE08UweHN+gOHhOJ0oVgTU7kr04aSLj3QNFTPF7MuWMOjh9Q2V0NOfR2VJAf7ESjFkVEgchKtcNNGSaBifvGbRrd/Rj4QiKoR2535Tg7+FyJbinkrjSEtc/EN9KpVRxrZlckuftNxUAsHqL5/kcLFWstXsAz4gxPThB1eMGV+pl6sukKvtXjUfGjYEDIA9gPYDnA5gM4OMAfkNEC5Vtpggh2v2fTzdgjAB0t3JsiEpmUflfhhU3miauWsVmr5+kp24zO2q4HH5ByaaD8qna3Fb12qhjl2NZvbU3Uujvl/c+FxmD9Dj1FyvB5Gg+oW81GjQWy641swuovhjdV/7+JL54k6c9kK56FZkRZD5QqxNvIUcojKJVg6o56h+uoK05h0LOQSXFmDVJDVGNUncdVBtOMLxMo/kLN60K9sv5Hhw1lVo9rjmZxomM1fYfv71/A17/43u1/dQQlWx18MRmXbhuEnr+7NfWNRh6cKR35eN/fAzfvPUpJU3c+zcqxcXee+ktkzqTckVgmy/qVwuoFXKEPz28Cc/tGsCCadXXClGNJVf4Wh/lUmzG7X+e2am9Nh8aSm5yleF5U1rRlHOwxs8WGy5VYsXOk5pykeQAW9VjZs9jIrZmMBk3n1AhRD+AK5VFfyGiNQCOBnB/QwYVQ0s+/DKIe1oPsqgCD44b+bJqbw6fird0R40BwB6yMb0yw2UXBcfBENzgiXfO5GbrtuqTujoJL5g2Cc/u6Meqzb3Bubf2DOGtP11hvb6NvoEzMFwOJnrzCX1bzxCmTCoEjRgBz9tie/KrVoPz+wc3BhOS6sH508ObsGZ7f+C2N/+Jar2oHAf5UTTbVO+lDFEV8mHBwpyT7Qko8ODUSWScpVWDaeCo5835GhxTEyPfs3zOwTcuPiroOSQzj2zHHiq7aM85+B9Lt/rhiovJTd5n48BZ7ZjUlMODz3Vhe+8wOlry+O9j9ovsk1YvqnugFIReZJbanU/vxJ1Pe0aCFBkDXpbYtLamwPDxriV8P6+64QlMmVTA8oXTguM3Kd7ci47exzqGNH7xluPwlb+vxv3rdmO4XNEMRtt7b3ZON7coK32ibOQcwr7TWrFuhxe+HSxVgvRxk7bmfOShJPTgjJvpg2GsjCcPjgYRzQawGIAaG1lHRBuI6MdEZK0eRUSXEtEKIlqxffv2uoxNLc4Vp98Is6i8LwHX1fU6bU05rduvGc5JatVgfuEMlSpB3Rx5Clnbwty2f7gchAHUL08pjly1pScY+3dvfxZ/W7nVen2burzx9hcrgT6n1whhbO0dwv6G235Hn12HI+9jFg1OxRVBajqgh/fe/csH8bWbVwdf+uZDiOohCioZj9iDExo4xbKL9qZ8cG+r0eFs2O0ZizYNBDCy7uQq1YaoJFIfliPPg7N7oKgZxfI9yzuEC46Yh6MXTAteayEq5TO8u7+Ij/7+Ue31R65/BF0DRS1EVcg5OGq/Kbh3zS5cdcMTsRl4cRlif3lkE357/wZ0DZaCFFdbsTIpMgaA9/z6Ibiu8EJUTXoxv/vX7cbD67vwwXOXBEX6gFAY/NZT9x9xv54TD5iBC4+aD8DzCmq1qiwfzXU7dUMzqsERqTVqFk5vC+r9DBYrWqNMlUlNuciY5INFo5tJMvXnM/91KH7yxmMbPYwRMy4NHCIqAPg5gJ8IIVYB2AHgGAAL4Hl0Ovz1EYQQVwshlgshls+cObMu41M9OHFu/0CD06R4cJQviWntTUHVV8DLONrRN4yv/WM1/vzwpsANbZuUTOHwUMnVJpHOlnxgPJnbDhQrwZNXxfKUvX7XQLDc1ntIIj04f3xoI/6+cgsALxzwq3ufw6MbuiGEwNodA1hqpM3GCY3lRJXFm7Ktd0gzFnf6x9TFmTLlWT9epJJxhjTx392/AQ8852Wt/WbFety+2jOczfRamUWlXk8WVm3uQXtzPqhVYjJkaXZYDVnSxG0enJWbekDkiYyXL5yG3qEyfnznmmB96MHRJ9N8LioyltHYt/70fvz8njDkeeczO/DLe9fjbyu3oFiuaPq2YxZOwxNbkkNUcaHcy37xIP7nuofRNVAM/i2ohomkoyUfeBQfXt+Frb1DgTdOvbbf3r8BTTkHLzpcL54njc/RVnoNMhFLFe3BwxbqXLtjILJMxQtRJX+1L5jehnU7ByCEwFDJRXPebuDIe9OntOnoZQNnr+GS4xbg+YvrM4+OBePOx0hEDoCfAigCuAwAhBB9AGSsZCsRXQZgMxF1CiGSvwHrgCrISyswJlNKTV3G9LZmLfVua88wPnL9o/jH47rHxDZRmmGn3mHdczKtrSn48jFTe/uHy5jW1oSd/UVNjyPj7Nssjf+S+PdTO4K/uwaKuNxvDPnvD52OvuEyDp03Wdt+Z4zRJL0BWQwDM+NManC2KF4weatVAbYQQssMk5WMZZVpJyZT4AN+OOVnbzoOH/rtIzh+/2l4/uKZWhYV4HkD5D2txoOzaksvFs9uj21eOGAYUnc/uxOuEDjxgGwl8EMPTvy9HbBoox58bjdyvgvsgsPn4vv/ehZ/f3wr3nzK/t7xpAfHGHfBkia+eHYHVm3pxeOGpkYWq7xv7W4tiwoADt9nsuadEEJEdAFpGhxXhCmudg9OAUtmdwSve4fKvmBcioy98axYtwvPWzAlki4rdUk246kapAZmuOwGDyVNeccaolq7sx8z2puDBxBhBKlKZTe1T9T+M9swWKpgY9cghkrxHpwp/vV2DRYx2TcUZYgqLqTKMOOFceXBIe/b64cAZgO4SAgR19kxiECMycAM1KfM+DRxGaKyZ1HNaG/SQlQbdw9i42594gaiZfOBaNjJFH9ObWsKQk6qx0QI4XtwvPOqHiWpo7F1Ns6KeitW+RkaS+d2aNvsjPHgSG+AfCLe2TeMW56wh8c2GPdJXqM8JxDeI9XLMlx2rZWMgej7ePvq7ZGWFB+47iEAwGY/PGZqjlQPTtYChEIIPLm1N/B02br2mlksr7z6brzq+/dkFjJn6UxvGsL7z2hDqSIC8TkRYXZnc/D0DoSfSXMybSmEqcUrN3VjU/cQFs/WPwcSOUnft3aXFqICgPlT9HvRZelKHV5bvJdLekqb87mIbqSjJY/Jkwq45g3HAPAmb1VkLD8fQyU3kg4PhPdt1AaO4sGR4ep5k1si7VT6h8vY1juMw+ZH08slqgcqjoP9f5dPbunFUKkSnN9Efkep9753qIy2ptyETR1m9h7GlYED4DsADgZwgRAimF2I6DgiWkJEDhFNB/B1AP8UQtj7JNSZLFlUphu34grNYzK9rTl4OgI878PO/qhxEaTYqinJhoEjhcry4XbaJK+p3qSmnKZPKVa8dhGdrdEQlTQEqvXgxLHKf1I3J7b4EJUearjkB/fgTT9ZEfGSAKH+RyKvcZ2SUiwLt5lCYCA0OvO+yBjQtVTD5Qpe96N78V/f+o+2/9aeYRy3aBo2dw3BdUUkdNTWnMM8Pz3/Tw9vsl6nSc9gGV0DpUCrdPsHT49sUyy71s/ZPc/ujCx7amtvtKZShjo48jN0xD6TccDMNhzjC2kdxWPS0VIInt4BJURlTHSzO1uCz9Hn/upluh2lpEOr7Oj137udA9jaM6x5cKRoWWL7bJYyiNNlh3MAmGkYInIC7wg8nmVNZKxmOdmMGHnfRh2iCjw4oYEzZ3JLxHiRSQSHzQ89o+bncFPXYKyeSyL/Xf7y3vXY1D0U78HxjUM1U7FnsMQp4syEYNwYOES0AMBbARwJYItS7+YSAPsDuAlAL4DHAAwDuLhRYzVTKuXkM1As4xm/Wu/m7kF0toQCxrKRJj6tvQmT/S8PWQnWTKsG7J2B41KtZ3d4x5FfSlMnNWGX8sUke8pIo+tRJRvDZkjEkSV7cM2Ofsyb3KIViwLiQ1SmTkR6Y7oGowbRpq7B4LgtBQf9xQp6hkrYbTxlAp7hJoTAYxu7g2XSsMw5FEzO6gT59DbvPdzSM6R5LIiAc5bNQbHiNTc1ix+2N+dxwgHTcdbBs3D1v56NvzkK8vrSer1IA1Q1dP799I7Idmd/7V848fO3asuyaHCkF/Bnbz4Ot3zgNBy9cKp2XsDTdqn3Qx7P1HvM7mjG1p4hPLqhG+t3D+DcZbPxhpMWBesPmtUe/L3d+DyoXk1zEjWF+EC2RqLzlMneNFJky4RO37Ozu98TO0sPyLzJ8fsCocDXNJyqRXpQhksutvqG3LzJrZE+ZDJt/mglk6u/WA40Z0IIbOwajBiHJvLe3ux7SeOyqOT7odZA8pqUcniKGf+MGwNHCLFOCEFCiBal1k27EOLnQohfCiEWCSHahBBzhRCvFUJsadRYzQ7AUr/y7l8+iDO/cjtKFRcbdw9i3pTWwI1bMUTGpxw0IxAg75dQP6Nk8eCok4zK7E7vS1Y+BU9ra8Lu/iJ29xfRP1wODCP55XTVDU+gd6gEIUSmvj+So/adEllmFoXa0V/EzA79S7+jOW8NUe3uLwaeAbNRZddAKZI1talrEPtOm4SnP/MCfOtVzwPgFS1Tv4Rl6GOo5OLHd67Fi75xB/7xuPeRkYZlPhdWqt3ZNxyMYdXmMNSlFpqb3tYUCIE3dg1GjMK2Zq/67bGLpqF7sGRtbWAiDYvOlFLo8jOmevm2dkcnfMlwuRK0ysiib+oeLMGhsDmsradSR0sBfcPliIC7yfj3MLuzBUMlFxd88w6s8yv8AsCx/qQ8R2ntsKNvWDOYzzkkvhu3NHB29A0H/x5szTbN9iSqN8P0tEw2PDhSxyUNnNmTm2P3VZk6yl49QT+3cgXbeobgkHefyhX9e+PJLb1ozjs4dJ4eopL/fnf1FzFcdlM9OABw1sGzg7+b4wycVj1ENVyu4LldA6mfV4YZD4wbA2ciYXpwpEbiX6u9J+ot3UPY6LuJpYeg7LdqWDqnAw98/GyceMCMwPjZZ2q8gRP08VEMkLgwjxT9yQljalsTdvUXcdSn/4ELvnFHME5VS7Bh92AwUc2xaAxMHvz42Th8nymR5WZGxYZdA5Ev/X2nTcJOS5r4xd+/O/C+lCouuhVPTNdACT/497M4/nO3YO2OfvQNl8N7m3MC7cqqLb3oUrxVaoPHH/mZP1K7o3pw5HtwxlduxzGfuRkA8OTW0MD51+qw1EBLIRd4AzZ2DUaMQulVktuYoSIhRBBe7PcNBWmUpfV6kYbQNsXLZxaHVLPI3vnzB3DsZ27BQLGcyYPTPVhCZ2shEFofqHhZJB0teVRcEXyOpF7LNGRndeqv5f249k3H4tErz9FCWtt7h9HWlA8+e4fOjxpWkk1dQxgolrH8qpvxxmvu065J1ReZmYOqJ2jBdHsqtwzbys+IfAhQs4tsXprD9/FCRWlZS2nI0hObu4ewtWcIMzuaMbvTC1GpXtgnt/Zi8eyOiIEhH15kduO8DAbONy4+Cqf6GTJxHpzJioFTrri45Pv34PHNPZEwOcOMR9jAGQGmB0dOUjK7av2uAW8Snqp6cDwDJ+dQIACWX/RqZdQlhmZFTk5qiCpOCCy/ZOX8Mb2tCZv8p/xnd/QHXYGft99UfPFlhwPwJmHpiZg9Od3AmdrWFJnQAGCS0Wbg2R39mGaEXfabNgm7+vVaKsWyi9WKQVGsuHjY75EDAN2DxeD1Xx/bjEOv+BtWbenFfN8FP29yCzpa8li1pQddg6XIe9MzWAomLandmeprMvKKyBjwvD1rd/TjofVdgdhXHUtT3sH8qd7EsX7XYKR7epqB853bn8GyK/6GlZu6seyKv+FX960PvDyqgWPTbsqGovKY8ya3REKa6nhufmIbAOCQT/wtWFaq2LU8gCeWVY1UW2XbDiMzL2js2qF/bkwxrvQmtBRy6GgpIKcUodveO4xJTTnc+J5TcN9Hz4pkSUlDa05nC7528+rgesIyClENTl+MhxMA3nXGgfjt206ILG8teKLZlX4vt30tDx22ENUv33I87v7ImbHny4q89x/9/WP4zYoNmN3ZEnhk1dDcat/AMQ0qadRtqsLAaW3KaeFCG/mcg47mPLoGi/jjQ5uwwm/yO1G7SzN7F2zgjADTnXvO1/6Ft/50RSDUe9UP7kHvUBnzprQGdTTKvshYfXqd48f3501pxY3vOQUrPnZWJHZu8+A8FdM9W34py5DX1ElNmjEkn7ramvM4bYn35LZJ8UTM6cymI5hhcdXbJs6oB6cVW3qGcMgnbgqaYG7YPaAJKcsVgb88Egp0dw+UMKfTu0+/8ftLAeEXuNe8sAOrt/Rh90AJ+0zVv9jXKc02N+z2/p6s9CYyK76e9uV/4t41u3DhkfMxra1J0yk153PobClgdmczntraGymOJ/UK8wMvj+5hkf2x/vSQd303PbYlMI6lBwFAZExEnofqsY3duPSnXlHvw/aZHNGk2JphqtyzZhcWf+xG/PupaAHM7sFSqhdJejVO+NytuG/trkD0a3psTAPHnGzVubnsCrQ152MN59++7QT87b2nWrPLvLT/aIjKLJug0tacx8G+1+8YX2cEeJ+jjpY8HvUN2oUzLAZOR/Rz39ac10JuI2XelFb88HXLA6NmTmcLZvn3UXrtShUX23qHI59xIEzdTisaaSI9Z9t648OdU9q8auR3PrMD09ua8Pf3nYr//e8js10YwzQQNnBGgK0x3d9Wbo0Uy5rd2Rw8rUoPjlpr5dxls3H1a47Gm09ehIPndmJGezNmGU/DpYqL9bsG8F/f/k9wTLNkvuTCo+bhe685Gm862atTomaPAMBrfuj1/pnUlMOMtmY05Rxs6Bq0hq4kqrZGGmfT26IT0XApGv6QT3lyYpTXNlx28dE/PAYAQTVVyWCpghsf24IXHuZpMboGSsGX99qdobEyX/mS32fqJGzuGUT3QDES7lOrvgYhqknSg+OgoHh8jlWEmxceNR8Lpk/S0mP3n+mFN5bO6cSqLbqBQxT2bpnZ3oxCjrBx9yCuf2ADXvzNO1BxRVAg8t61uwB4qegf8esGqcaFWThv4fQ2rNrcqzUOXTZvMnqHylqfoDTNz7qdXhHHvz66ObKuZ7CkGVkAcOflZ+AP7zwpeK2GRX55z3PY2jOEya2FiLdntmHwmJOtmV6c1NRvyqQmLJnTEeh4VLb3DQeGTdkVgVZF9eDYxLBtzXn87E3H4QevO0Zb3tGShys8D63qldp3mjd+2+e+lpx58Gxc/46TcMUFh+DDL1ga/Hvc2jOEPz60EQd99EYIYf93KktMrN7ai+ltTZk1QfIz3ZQQYpvS2oSugSJWrN2N5QunYvHsjsD4YpjxDEvhR0Bc1U91snnb8w/Aucvm4FnfGJF1cHKKC56IcM4yXVT5ttMOwP4z2/C5G7302mLFxb+UJ+4lczqxtcfegqI5n8O5yvHivuTamvNwHMLcKS3Y1DUUTNS2L0452jefvAivPn5B7HFt7QSkgfPXd5+C1Vt7tSynf63ejk1dg5GqrNILccbS2bj5iW3oGiyiy+KZUA2CWZ3N2NozjNZCDiceqE+mamsIqVOYKg2cHGGW4jW45Pj98JVXHIH71u7CohltWDS9DQ8+1wUAuOKCQ4JeQ0vndOCuZ3Zi0cxw0lU7ajsOYe7kVqzfNYAnt/TgkQ3dXr0RX/wqj6miaiDMtOslszuwcnM3lvnC0i+//IjgfdnWM4yFM/LavUvjl/euR89gGR84ZzHe9+uH8MPXH4PuwVLEEzF/SqtmnKjGwubuIXQW8xFjBvBSq799yfOw79RJWLerPxDySszwRltT+tfQXEtW0JNbejXPTcl10ezkgvf53WcehJfH9Ic6+aBokcSO5gKAQSyc3qY9iPzmrSfg4fVdETF1PZg/pTXIOAt7wg3j+/8Os/LkPf/VpcejkHNw0Xf+EzTOfHJLb6T2VBJnLJ2FL77scLzo8Lmx20yZVMATm3uxpWcIrz1hQdXXxDCNgj04I8DmwQH0yfTD5y3BpKa8lkVVdkVqcaxFM9rwwsPCL5tS2QUp9QyXzon/8jLHdfYhs/H6ExfinacfoC2Xmp/5U1rx54e9nj1AaOA05R1c+8Zj8fM3Hxd4Jc47dA4W+rVabE/Fw6UKfvPWE/Cj1y8PlsnU532nTcKZB88OnrBleOzPD2/SPCzq+JfO6cCU1gK6B0roHihpWoEPnrtEq+I7u6MFxbKL7sESZrQ1Wd8fGVbLO4QLj5qPKy44BDPam7X+QTPbm7HvtEl46fO8SVEVpL72hIWBTmLp3A4UKy5ueCT0hJiT9vIFU3H76u2BVuSuZ3dGChSqqNoTqa+44oJDcMO7T8ZJB07H+l2D+NdT2zFlUgEvO3qfwBj5v1uewv/evBpAWPsnCfne3fDoZnz/38/i4Q3duG/NLvQMlVNL73cq7/ujG7uxZke/1SgGgBceNheH7TM50toAAD583lJ86Lwlwftk6rds2B4qnts1oBk4l/v9qqSBc+GR87BvFR2+pYbKDIfNndyK8w6NNwDqRVPewfS2JmztHdK8hfKeH7//dBy9YCpmtDdj3Q7PO/fk1l4snRMv1DYhIrxi+b5avR+T8w+biy09Q+hsyVvfT4YZr7CBMwJUl/x1bzsBpyhPg8csnIr/e+WRwYQVZFH5LvQs1T/3mdqK95x5EM7za66o8XFThKxiTgKzOlpw5YuX4QNnL8HbTwuNHPnE/PoTFwIAfuH3Blo4fRLeftoBuOk9p+DUxTNx0oEzAtNKveYDZ7bjXWcciG9f8jx84OzFwfUdu2gazlg6O/BGmJP++YfPxRtOWoj/e+VRWDqnA3c8vQNrlLCT+sV84Kx2dLYW8Kv71uPetbtwwMx2fPLFy3DtG4/FO08/ULuP6iQ7eVITCoaGpZCjIMTR3pLHrI6W4ClZ9eDMMDQgRy8INRrq+Z6/eBbOXDpL29YMX7zkqPnoGy4HGpHfP7gBFVcEhQCTkAbOCw6di2XzJuP8w+ch7xDuW7s7CJ1IndXvH9yI/735KQDZPDj/74UH44IjvEnqNys8w/aJzT0ZNTjh+v5iGau39kVCqlnoaCngHacdiPZm73hpVXcB4NXH7xeELSWbuga1Fg2/f3AjtvWGtYuqbSXwrF8S4AXGeRrJrM4WrN81oNXQMjVPC6dPwpqd/Vi3sx9DJRdLEh6CRsJ/H7Mv/uecxfjua46uid6IYcYKNnBGgNkQ8DMXHha8vvjY/fCSI+cHr+XE+K/V27Fi3e5MBg4R4X1nL8YZB89CqSKCYlwAcKhSwfT8w+fi0lP3t45LxXEIHz5vafBaGiDnLJuDFx8xLwgvtTbl8OHzlmL/mUpmhT9ctdKp4xA+cM4SvPCwubjsjAPx5pMX4VeXHh+sl1/AbcaTeVtzHldcsAyTWwtYvnAqHli3G89s6wvS2lWBdUshFxTcA7zMp9eduDBIa1VRwyRTJxVw2D56/6tZHS1B8UOz8KDqOZluGGQnHDA9ci7AM9x++Ppj8LHzD8Zy3wiaZgivTzpgOl58xDyctmQmjl00DY9t9LJz3n3mQdZjqvzkjcfitScsCIyvaW1NOHaRpw+S93afqa3atfQOlQID55xDZuMv7zrZeuzjFk3DNy4+Cs9fPDMQhv96xXoUyy4OSjCeAd1zd9WFh+LYhdPwoiNG7tmQ2pu2BA1OeO4CvqYIW6dMKuA3Kzbg4fVd2nb3r90dGDgdzdXVavnki5fhnacfgAuVf7+N5pSDZmj93oCoMb1wRhvW7ujHrau8zLnlimFeC4gIl51xUObeZwwzXmADZwSYokpV8GpWpJUZMfJpeZulWnEc5y6bg6acE0yOQJjiDABvf/4BeI8yYcZpgyRXv+ZovOzofTR9gfq0Z6uFIUv1x9XJICJ87EWH4Kj9wi/Vb73qeThz6SxrLRXJMQunob/oNfuT2pK+4Qred9ZifPxFhwAAPuH/BsLMJxuqB2fRjDZ83a/vsXh2u7++ORAWmwaOivne5RzCpy88FG97/gHW7d98yv740RuOwXGLpuFD5y7R1uVzDr5+8VG45g3HaoURzzx4Ni45br/YMQDee/KplxyqvU/SwJFGAREF1wd4AmJZK+fblzxPM4RVZEbThUeFoYatPcNozjs4d9ls6z6S1kIOLz5iHq55wzG45LgF+M3bTsDpS2Yl7pOEvJak8IiK+vlePKvDWi7h3rW78OBzuzGrozk2lBzH+YfPxQfPXRpJVW8k7z97Mc45ZHbwbwSIirQXz27Htt5hXHvXOhy+z2T9AYVh9mLYwBkBZg2KnEP4zVtPwJLZHUEKqmRGRxMOnNUehKrUInJpTG4t4PSlocdi/5ltmDqpKdCTOESae7855Qv9nGVz8OWXH6EtUzU9tswI+V0f5x2ycej8yfjh649JNLhO2D/0jsi+Or1DJbznrIPwppO98NEbT14U/B1nYAF6obkj952CGe3NuPaNxwYGy2HzJwehIpv7/soLDsFJB063dhN/zfELcPkLlkaWSzpbCvj1W0+ILSAHINAutTXlMKO9CZ/5r8Nw9iGhMfGeDF4dGS5TdTxLlc/a2p396B4sob05H/TXepXFkJLG+TmHzMFR+03BK5bvg2ltTXj18QtS+wsREb5+8VE4bRRGjYr0GMVV5k7CllIOANc/sBG3PbkNFxwxb1wZKiOlpZDD1a9djhvefQreefoBeOlRUe/Scj/777ldA5r3mGH2dtjAqRHHLpqGv73v1EiMelJTHje///l4+rMvxNI5HXjzyYtijmBHusv/e/m+uPUDp6GQc3CVHxIz02+TUj3jOHLfKZjV0YyvvuIIq3fjg75notbN9VRjSk72b7LcG1mnY0dMDyvAmwSWzunAu844UJvUXnTEPMzsaMYHzl2Csw6ehamTCvjo+QdH9n/9SYvw8zcfH1leK6RodeGMtmB8Uhz7/dcux/t8HVMSR+03FTmHNG/SaYtnBpqeh9d34cbHNmui6c/+12H4x/tOBeBpt1Rjtq05j9+/4yR88WVH4IGPnx14zcaSj59/MCa3FgLReTWonwepgXvPmQehe7CEUkXgoufZs6cmMh88dym+aqk/c+i8yWgpOHAIuGAUIUOG2dPgNPEx5Kb3nlr1PqcvnYV5k1twiOKiPu/QOVj7+fOD168/cSGu+c9aqwcijentzbj3o2fFrr/kuAW45Lj6pIZ++sJD8fE/PIbn7TdVux6V05fOwmf++gTOODjZa2C7t685fgFe46e2v+TI+Q17upUCZ7WWy8uO3gf/fHK79r4m0d6cxzOffaG27Jxlc3DOsjk448v/xA/uWIMcEb73mqO1bQ6a3RF7bxvNrM4WPHzFOVXtc+j8TuzqK+Lly/fFPWt2YcXHzgoqDPcOlfCTu9bi4mP3y3xf9wSa8g5OW+z9+xiJ6Jth9lRICHvp9j2B5cuXixUrVtTl2AsvvwHLF0zFb99+Yl2Or1JJyb6S7+FEdMmnXRvgXd9EvDaJ6woc97lb8KaTF8XqeUbDnU/vwOt+dC/+59wldTn+eMU1CmcCXu2YsahXM94QQkAIjOghh2EmOkR0vxBieWQ5GzgjY7BY0bpRM0wSvUMltBZygT6mHsevdRiRYRhmIhBn4HCIaoS0ZkhtZRhJvY0PNm4YhmF02P3AMAzDMMweBxs4DMMwDMPscUwYA4eIphHR74mon4jWEdGrGj0mhmEYhmHGJxNJg/MtAEUAswEcCeAGInpYCLGyoaNiGIZhGGbcMSE8OETUBuAiAB8XQvQJIe4A8CcAr2nsyBiGYRiGGY9MCAMHwGIAFSHEamXZwwCWmRsS0aVEtIKIVmzfvn3MBsgwDMMwzPhhohg47QC6jWXdACKNhYQQVwshlgshls+cWX0JeIZhGIZhJj4TxcDpA2DWXu8EkL1zJcMwDMMwew0TRWS8GkCeiA4SQjzlLzsCQKLA+P77799BROvqNKYZAHbU6dh7KnzPqofvWfXwPasevmfVwfereup5z6wNEydMqwYi+hUAAeDN8LKo/grgxEZlURHRCltpaCYevmfVw/esevieVQ/fs+rg+1U9jbhnEyVEBQDvANAKYBuAXwJ4O6eIMwzDMAxjY6KEqCCE2AXgwkaPg2EYhmGY8c9E8uCMN65u9AAmIHzPqofvWfXwPasevmfVwferesb8nk0YDQ7DMAzDMExW2IPDMAzDMMweBxs4DMMwDMPscbCBwzAMwzDMHgcbOFVCRNOI6PdE1E9E64joVY0eU6Mhosv8/l/DRHSNse5MIlpFRANEdBsRLVDWERF9gYh2+j9fJCIa8wsYY4iomYh+6H9+eonoQSJ6gbKe75kFIvoZEW0moh4iWk1Eb1bW8T1LgIgOIqIhIvqZsozvmQUi+qd/r/r8nyeVdXzPYiCiVxLRE/7c+AwRneIvb9w9E0LwTxU/8Grw/Bpef6yT4fXEWtbocTX4nrwUXgr/dwBcoyyf4d+flwNoAfAlAHcr698K4EkA+wCYD+BxAG9r9PWMwf1qA3AlgIXwHjJeBK/tyEK+Z4n3bRmAZv/vpQC2ADia71mme/d3AP8G8DP/Nd+z+Hv1TwBvtiznexZ/z84GsA7A8f532nz/p6H3rOE3ZiL9+BNTEcBiZdlPAXy+0WMbDz8ArjIMnEsB/Me4f4MAlvqv/wPgUmX9m9QP/970A+ARABfxPct8v5YA2AzgFXzPUu/VKwH8Bp5RLQ0cvmfx9yvOwOF7Fn/P/gPgTePtnnGIqjoWA6gIIVYryx6G92TJRFkG7/4AAIQQ/QCeQXi/tPXYS+8lEc2G99laCb5niRDRt4loAMAqeAbOX8H3LBYi6gTwKQAfMFbxPUvmc0S0g4juJKLT/GV8zywQUQ7AcgAziehpItpARN8kolY0+J6xgVMd7fDcbSrdADoaMJaJQNr9Mtd3A2jfy+LWBQA/B/ATIcQq8D1LRAjxDnj34hQA1wMYBt+zJD4N4IdCiPXGcr5n8XwYwP7wQiZXA/gzER0AvmdxzAZQAPAyeP8ujwRwFICPocH3jA2c6ugD0Gks64Snn2CipN0vc30ngD7h+yr3dIjIgRfiLAK4zF/M9ywFIURFCHEHvLj928H3zAoRHQngLABfs6zmexaDEOIeIUSvEGJYCPETAHcCeCH4nsUx6P/+hhBisxBiB4CvYhzcMzZwqmM1gDwRHaQsOwJeaIGJshLe/QEAEFEbgAMQ3i9tPfaie+k/ofwQ3tPPRUKIkr+K71l28gjvDd+zKKfBE64/R0RbAPwPgIuI6AHwPasGAYDA98yKEGI3gA3w7pNJY+9Zo8VJE+0HwK/gZVK1ATgJnEUFeBNNC4DPwfNItPjLZvr35yJ/2RegK+jfBuAJeK7gef4He2/JOvgugLsBtBvL+Z7Z79cseGLZdgA5AOcC6AfwEr5nsfdsEoA5ys+XAfzWv198z+z3bIr/2ZLfYZf4n7MlfM8S79unANzn/zudCi9j79ONvmcNvzET7QfANAB/8D/0zwF4VaPH1OgfeNkZwvi50l93FjxB6CC87ISFyn4E4IsAdvk/X4TfH21P/gGwwL9HQ/BctPLnEr5nsfdsJoDbAXQB6AHwKIC3KOv5nqXfwyvhZ1HxPUv8nN0HL4TSBe8h5Gy+Z6n3rQDg2/492wLg6wBaGn3PuNkmwzAMwzB7HKzBYRiGYRhmj4MNHIZhGIZh9jjYwGEYhmEYZo+DDRyGYRiGYfY42MBhGIZhGGaPgw0chmEYhmH2ONjAYRhmQkBEgoheVsfjL/fPsbBe52AYZuxgA4dhmLpDRNf4xoP5c3cVh5kL4M/1GiPDMHsW+UYPgGGYvYabAbzGWFbMurMQYktth8MwzJ4Me3AYhhkrhoUQW4yfXUAQfrqMiG4gogEiWkdEr1Z3NkNURPQJf7thItpCRNcq65qJ6H+JaCsRDRHR3UR0snG884holb/+3wAWmwMmohOJ6HZ/TBuJ6DtEZHZHZhhmHMIGDsMw44VPAvgTgCMBXA3gWiJabtuQiC6C1x37HQAOAvAiAPcqm3wRwH8DeCOAo+D1rrqJiOb6++8Lr6fcP/zzfcPfRz3HYQD+7o/pCAAv9bf90eguk2GYsYB7UTEMU3eI6BoAr4bXYFTlW0KIDxORAPADIcRblH1uBrBFCPFq/7UA8HIhxG+J6P0A3grgUCFEyThXG4DdAN4shLjWX5YDsBrAL4UQHyOizwJ4GYAlQnb9I/oYvA7Ii4QQa32PUEkI8Sbl2EcCeBDAbCHEtprcHIZh6gJrcBiGGSv+BeBSY1mX8vddxrq7AJwfc6zrALwHwBoi+huAmwD8SQgxDOAAeN2N75QbCyEqRHQXgEP8RQcDuFvoT3jm+Y8GcCAR/beyjPzfBwBgA4dhxjFs4DAMM1YMCCGersWBhBDriWgJgDMBnAXgKwCuIKLjEBohNve0XEaWdSYOgB8A+Jpl3cbqRswwzFjDGhyGYcYLx1tePxG3sRBiSAhxgxDifQCOAbAMwEkAnoaXnRWIiv0Q1QkAHvcXPQ7gOCJSDR3z/A8AWCaEeNryMziC62MYZgxhDw7DMGNFMxHNMZZVhBDb/b9fSkT3AfgnPH3MmQCOsx2IiF4P7/vrHgB98ATFJQBPCSH6ieg7AD5PRDsArAHwPgCzAXzbP8R3AXwAwP8S0bcBHAbgbcZpvgDgbiL6LoDvAegFsBTABUKIt1Z/+QzDjCVs4DAMM1acBWCzsWwjgH38v68EcBGArwPYDuANQoj7Yo7VBeDDAL4MT2/zOICXCiHW+Os/7P/+MYAp8ITB5wkhNgOAEOI5InopgK/CEyvfD+ByAD+TJxBCPEJEpwK4CsDtAHIAngXw++oum2GYRsBZVAzDNBw1Q6rRY2EYZs+ANTgMwzAMw+xxsIHDMAzDMMweB4eoGIZhGIbZ42APDsMwDMMwexxs4DAMwzAMs8fBBg7DMAzDMHscbOAwDMMwDLPHwQYOwzAMwzB7HP8fzvECH//G7xMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot(rewards)\n",
    "plt.xlabel(\"Episode\", fontsize=14)\n",
    "plt.ylabel(\"Sum of rewards\", fontsize=14)\n",
    "save_fig(\"double_dqn_rewards_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "env.seed(43)\n",
    "state = env.reset()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for step in range(200):\n",
    "    action = epsilon_greedy_policy(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "   \n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dueling Double DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "K = keras.backend\n",
    "input_states = keras.layers.Input(shape=[4])\n",
    "hidden1 = keras.layers.Dense(32, activation=\"elu\")(input_states)\n",
    "hidden2 = keras.layers.Dense(32, activation=\"elu\")(hidden1)\n",
    "state_values = keras.layers.Dense(1)(hidden2)\n",
    "raw_advantages = keras.layers.Dense(n_outputs)(hidden2)\n",
    "advantages = raw_advantages - K.max(raw_advantages, axis=1, keepdims=True)\n",
    "Q_values = state_values + advantages\n",
    "model = keras.models.Model(inputs=[input_states], outputs=[Q_values])\n",
    "\n",
    "target = keras.models.clone_model(model)\n",
    "target.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "discount_rate = 0.95\n",
    "optimizer = keras.optimizers.Adam(lr=7.5e-3)\n",
    "loss_fn = keras.losses.Huber()\n",
    "\n",
    "def training_step(batch_size):\n",
    "    experiences = sample_experiences(batch_size)\n",
    "    states, actions, rewards, next_states, dones = experiences\n",
    "    next_Q_values = model.predict(next_states)\n",
    "    best_next_actions = np.argmax(next_Q_values, axis=1)\n",
    "    next_mask = tf.one_hot(best_next_actions, n_outputs).numpy()\n",
    "    next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1)\n",
    "    target_Q_values = (rewards + \n",
    "                       (1 - dones) * discount_rate * next_best_Q_values)\n",
    "    target_Q_values = target_Q_values.reshape(-1, 1)\n",
    "    mask = tf.one_hot(actions, n_outputs)\n",
    "    with tf.GradientTape() as tape:\n",
    "        all_Q_values = model(states)\n",
    "        Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)\n",
    "        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))\n",
    "    grads = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "replay_memory = deque(maxlen=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 599, Steps: 200, eps: 0.010"
     ]
    }
   ],
   "source": [
    "env.seed(42)\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "rewards = []\n",
    "best_score = 0\n",
    "\n",
    "for episode in range(600):\n",
    "    obs = env.reset()    \n",
    "    for step in range(200):\n",
    "        epsilon = max(1 - episode / 500, 0.01)\n",
    "        obs, reward, done, info = play_one_step(env, obs, epsilon)\n",
    "        if done:\n",
    "            break\n",
    "    rewards.append(step)\n",
    "    if step >= best_score:\n",
    "        best_weights = model.get_weights()\n",
    "        best_score = step\n",
    "    print(\"\\rEpisode: {}, Steps: {}, eps: {:.3f}\".format(episode, step + 1, epsilon), end=\"\")\n",
    "    if episode >= 50:\n",
    "        training_step(batch_size)\n",
    "        if episode % 50 == 0:\n",
    "            target.set_weights(model.get_weights())\n",
    "\n",
    "model.set_weights(best_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAENCAYAAADzFzkJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABz0klEQVR4nO1dd7wdVbX+1pxzbk3CTbkJkEBCgBBqQIJBmgUUewELKnbBhuXpe4pP1ChYQEXFZ8OHIog8C+B70os0RcEA0kMJEEhIb7fl3nvOmfX+mNkze/bsPWfm9Jvs7/dL7pwpe6+ZM2evvda31trEzLCwsLCwsEgDp9UCWFhYWFhMHFilYWFhYWGRGlZpWFhYWFikhlUaFhYWFhapYZWGhYWFhUVq5FstQCMxY8YMnjdvXqvFsLCwsJhQuPfeezcyc7/u2A6tNObNm4dly5a1WgwLCwuLCQUiWmk6Zt1TFhYWFhapYZWGhYWFhUVqWKVhYWFhYZEaVmlYWFhYWKRG05QGEXUS0UVEtJKIBonofiJ6jXT8eCJaTkQjRHQrEc2VjhERnUtEm/x/5xERNUt2CwsLCwsPzbQ08gCeB/BSALsA+DKA3xPRPCKaAeBKf980AMsA/E669nQAbwawCMAhAF4P4CNNk9zCwsLCAkATQ26ZeRjAUmnX1UT0DIDDAUwH8Agz/wEAiGgpgI1EtJCZlwN4H4DvMfMq//j3AJwG4GfNkt/CwsLCooWcBhHNArAAwCMADgTwgDjmK5gV/n6ox/3tA6EBEZ1ORMuIaNmGDRsaIbrFBMeqLSO4dfn6VosBAFg/OIrL73kO1z+8tiV93/BI9n7veWYzfn77CoyX3Irnrtw0jDufzPY7HBor4X//tRrbx8u48r5VEMs3FMsufr/sebhu9uUcrntoDTYPj2e+TqCavi+7e6Xxe904NKY9dteKjXh6w1Bs//3PbcEjL2wDALgu4/fLnkexrH/+T60fxPk3PYEHnt+aWtYsaElyHxEVAFwG4NfMvJyIJgFQ36xtACb725P8z/KxSURErCwIwswXArgQABYvXmwXC7GI4c0/vgsbh8bw7Ldf12pRcOp/340n1nmDxENLX4XJXYWm9f3OC/+BFRuG8eQ3XoNCLv388SOXLsOWkSJeNHcqjpg3LfHcl37nNgDI9Ky/dNVD+N9/vYBFc3bBA6u2YdcpXThqnxm48I6n8Z0bHgcBeNviPVK3t2loDB+77D68aM8+XPnxo1NfJ+Ont63A+Tc9gUKO8JbD5qTq80tXPQxAf+9X3rcK37puOR77+qvRVcgF+9/1i7u117zlJ3cF+6+4bxU+/8cHsWFwDJ94+T6xti/667O4/J7nMHNyJxbt0Zf6HtOi6ZYGETkALgUwDuAMf/cQgCnKqVMADBqOTwEwpCoMC4s02Dg0BgAoGWZqzcRzm0eC7Som0DXh2U0jlU/SYMtIEQCMM91asWbbKADg6Y3DAIDBsRIAYNOQZyls217M1F7Jf7CrtmyvWqZN/juzbSRd38Vy8pdZLDOYAbeKIWyrL4PJchorlTG7rxunHjlXe7xWNFVp+BFPFwGYBeBkZhbfwCPwSG5xXi+Avf39seP+9iOwsKgCjh93N1Ist1YQBU6T4wGrGbAi1zdI5waPYQJPCSs9WzHfreUrML0uZZdRyDXuZWq2pfFTAPsDeAMzy2r/KgAHEdHJRNQF4CsAHvRJcAC4BMBniWg2Ee0O4HMALm6i3BY7EHo6PK/syFjrlQZJP/1mR5GLAavagavcIEO/UY+hmTqoXMFsFI+uFsVturJUZuQaOANpZp7GXHhhsocCWEtEQ/6/dzPzBgAnA/gGgC0AlgA4Rbr85wD+DOAhAA8DuMbfZ2GRGcKHPDJearEk0QGyVYlHXOVwWg0hXQ1q1U3iuTbTmV2pL/HoqhGpklItlt1MHFVWNDPkdiUSfhfMfDOAhYZjDODz/j8Li5rQ0yGURjtYGiEmmjemVvdW09ACf1clK0wo6moeYaVryi4jvwO5pywsWo52UhrtgKrdUw2yNKjONlc92st6p5UUamBpNEDxFl1Gzmnc0G6VhsVOh25faQy3hXsqHNAmWjBg8yyN+vTTzMdb0XVXAxFeyT1VKrso7AichoVFu6Db5zS2t4GlMZHdU42KWBaDYr2eRz2I9axNVNIZ4ngjFG+pbN1TFhZ1hXBPDY+13tJoGfstodpxq1GWhmmQr5awD69vHmTXnc6CDDiNBvRdcl3krXvKwqJ+6BYht21gacholXeq6uipCeZOq8X9V+lKNelPfjY6q6MeIbcmlCwRbmFRX3QXvNe+HZRG5KfdKqXRZkS4CiFftYR2o3Xb8rUDWPT1G/H7Zc8H+2RloHtOwa4GyFYss7U0LCzqCcf3f7TDTDlChE8wVqPR0VNB1nSN7dXDFZSkrh5f61U7uuOJsHye/Gh075mQqZZHaHp9y66LvCXCLSzqh3ZavqsdZKl23GqUzq37M2mSLpYnAFFOI35ukI3fAOEsEW5h0SC0W4hryziNDB3L5zaqjEi9UQ8pk9oI3WfyvuTnJI7XYmmYlGvRbWxGuFUaFjst2mHMm2ght/Ig1yxOo16ox/etqw8mrAX5kPxsdO6peiT3Gd1TO0rtKQuL9oHvM2+xFEDzixTqkOU5lKTStrVyQj+/fQXmnXmNscR6wBXX6MrhOgzQYVs6q8H7K3+Tsj5lze3VWiwyCcUdrMqthUXboO0sjRYJlKXbyAy6RkvjglueBACMKSsAqoq0VuXUyJwIQFIaktxuJfdUDbWnKqFUtnkaFhYNQbtFK7WXNHqUXHkwrK2ttJeLAbgdQm717in/mLQvmqdhtk4aQoS71j1lYdEQtIOl0RbIYmmU62dpCJiGN/H91MqdcGyj8ahkkdWDCDehVLbuKQuLHRbyxHUiZISXKhC89YB4JG6dB9a6RFElRELB8F3q5G9klduS6yJvo6csLOqJxvq4s2FiJfeVI+6pxsoblNqo1dJotJz+X9l9VtE9VYfkPhNKLu84VW6J6AwiWkZEY0R0sbT/3dJKfkNENEJETESH+8eXElFROWd+M2W32AHRbv6pCVBGJBI91aiMcH+8E4NtrcqpntFT2mi3gAgPd0WUq9Y9pVxcJ5RdBjN2qPU0XgBwDoBfyjuZ+TJmniT+Afg4gKcB3Ced9jv5HGZ+unliW+xICEnI1qMNIm4zPYfoYNiY/lX3lOizXS2xIE9D2hcJuU1wT9Vb74rw5UZmhDdtuVcAYOYrAYCIFgOYk3Dq+wBcwu2WsmuxQ6CRMfJZMdGS++rJaYRrd+vbqZffv56ThMQ8DenLrOSeClykNQilU6JCwe5URDgRzQVwHIBLlENvIKLNRPQIEX0s4frTfRfYsg0bNphOs7Boi5lrWxDhGTqulOmcqV/DftUFJPqsddnWRj3fSpyGzr0mvHz1DiYo+dFtO5J7Kg3eC+BOZn5G2vd7APsD6AdwGoCvENE7dRcz84XMvJiZF/f39zdeWosJh0YmVk1EZHkMcvZ2vcqImNxTQT8154PU74vW5mn4zcvjdOpFmGoQTadEi7422qksDXhK49fyDmZ+lJlfYOYyM98F4IcA3toS6SwmPNqK07DRUxUHTrdGTqORiXSAbC3I672Hx3XcTz2We01yT+00GeFEdDSA3QH8scKpjLZYKNNiIqMdLI32cE+lP7cUmUHXS4Dkw41yg9ULgcrIULCwJgsjIXpCfD8NTNNoeshtnoi6AOQA5Iioi4hkMv59AK5g5kHlujcR0VTy8GIAnwLwv82T3GJHAgd/W6812mHmk+U5mEJJ//bURlzz4JqG9C8smurLiDTYHRnIFyJtnkY1MiVxUMIqcxoYltdsS+MsANsBnAngVH/7LADwlcnbobimfJwC4CkAg/AI8nOZWXeehUVFcKg12gptJo4WpbJeabz7v+/GJ357n+6SilDHQHW8q9k9VdVV2ds3Rk8lVLmtNxEe8CsNVBrNDrldCmCp4dgogD7DMS3pbWFRDRpd9TQLIsu9tq6OSGpUGz3FzEa3SrwVNXoqdTfJMtSnmXi7IuQ2Ej0FaVvnnqr+HUxyT4m+GkhptBenYWHRTLRbGtAE0BkR8rtmriHl5XWrcltLTkTCteI9MmaEJyzCVBMRrrlU9LUjuacsLFoPEU3TBjqjLTLCMzyHSP5BBgsgzaArEHNP1fxFNfaLjsdORe9JH3IrjtVZFqs0LCzqjzalNCYGZLdLhjyNpDMrtVJrGZF6hNwmjcHiMchuo6gbT3eNIMLrK5PoyyoNC4sGoN0sjYlQGl0+t1l5GjUXLKzp6srQKc+Ka6lz5E9V0Ne0EpZGDQ1XgFUaFjsdQhKy9Vqj1tIY9UCWMTm6TkQ2Itx4TPke1CeiI5qrQU11nhKuDUKCM9SeCkqjV5FVn/QUgpIr1tKwsKgfGuVPrgYRS6MNlFglRKKC6uSeUg+q41393FONga42lhvhNOLXiDDcamRKdPX5B+1yrxYWOwEmQvSUbDFkqQmVSIRXuLb25V5rf7CJnIYbtzTkIAHtehrBIkz1/dKte8rCogGo56I8taIdSqNneQ7ymTWvqGfIilbdUPUKuW3U9x1mrIeo5J4KHl0VIiU9BUuEW1g0AK1XFSEa6XtuBLjCYGi8LsNTr3fIbT10RVIbWiI8Ze2pei/CVNZYPfWGVRoWOy3aQXmYYvubiWqJ8Cxuo2T3VHI7ZTfdeZXaT3t12WU8tX6w8onifBaupnBflPvRyNSgYAzRruU0LCzqiIYXsKsSbSaOFkLGvEN188enLY3e6H4EfnjzEzjh/Dvw5LpQcSTN3MuaBZUqZc43KhjDuqcsLBqA0J3cBsP0xPJORaJzGrYIkxo9Va8yIilx73NbAADrBsZStRFOQji2D0iuPVWN4k26wrqnLCwagYAYba0YgOqeao0M1ZQR8SyN+vShuuXUc90GuXLqBTFQmxL69Bnh3t+qQm6Tcl5sGRELi8ahHYagKBHeIk4jU0a4h1xG91RSHyYlEXxWRt2swQNZlXHS+bquyxqlVikjPHRP6a2TaiG6ajtOg4i6iegEIppbb4EsLBoNu0Z49RADWz7n1M09pUJtVs0HyTq4ZrVQggx0zbirT9SLWxpp3VMcuSadfEnKum3yNIjoYiL6uL/dAeAeADcCeJyIXtM48Sws6o/wN9d6rTHR3FMCWTmNaiO0ACl5rkkEkFAyaXsLLI1IBWDZgtD0UUPIbbqSJq23NE4E8A9/+40AJgPYFd6CSkvTdkZEZxDRMiIaI6KLpf3ziIiJaEj692XpOBHRuUS0yf93Hk20AHeLtkM7WBrRMiLNQ8QtkuE6MZPNEWXLJE+URf2scBxByGyVIbd1fLBa95SInpJCayu5p3RVbtOKmfwsG89ppF25byqA9f72q+Gt472eiP4HwJcy9PcCgHPgKaFuzfE+Zi5p9p8O4M0AFsF7ZjcBeBrAzzL0bWEBQM4Qbq0cQHT23Ex5qu1LXOcQMmmbLAUL45xG+n707Wc8X1yQctwN3VN6l1Ta5L60brfkREPvb64NLI21AA4iohy8Af9mf/8kAMW0nTHzlcz8JwCbsggJ4H0AvsfMq5h5NYDvAXh/xjYsLADUPnPd0ZCpjEjg76e6Pb+YpaEcr72MSFYOBMb+dE3pk/squKdqWNUlDafRDiG3vwTwOwAPAygDuMXfvwTA8jrKs5KIVhHRr4hohrT/QAAPSJ8f8PfFQESn+y6wZRs2bKijaBY7GtrC0mhRlVs2bFdC4J5yKBtPkeGY6s0Jw1OrdE9VdVX6gVdYGiZOI3m513CfSc4sSs9tgnsqldJg5q8D+CCACwEcw8zj/qESgHPrIMdGAEcAmAvgcHicyWXS8UkAtkmftwGYpOM1mPlCZl7MzIv7+/vrIJrFjobAPdVaMWJornsqeSZsvM7/m3VMypankfy5XnTmXl+8Bhfc8qRGIPM1SSG3sgVQlMrcai0DjYvU9IwqcT4ygozwBiZTpG6ama9g5u8z8ypp36+Z+X9rFYKZh5h5GTOXmHkdgDMAvIqIpvinDAGYIl0yBcAQt0OZUosJh8Ax0AZvT6viOaq+9YDTyEaEJ51sckeZjmd2NyUMxuff9IRGnozRU5qQ24dXD2C/WZO9/QlEeJpcl/j9m8+VAxUaBSMRTkTvTdsIM19SH3HCJoUY/t9H4JHg9/ifF/n7LCyqRrtxGq0jwjNwGmJApdqT0dLMrLsKTs2cRlYVKfM2SbIJqApgcLSIx9YO4H0vmYfH1w0mJ/dF9unlVJ9zUpiubr3yeiMpeurHyucOAAUAwu5y4JHgYwBSKQ0iyvt95gDkiKgLnovrcABbATwJL1LrAgC3MbNwSV0C4LNEdC285/w5AD9K06eFhQrW/WJbhGg+eDM5jercU0HGceaQW/PZSRnhC3edUjunUeVjTTvuBisL+v1s214EMzBnqhcgql0iXBdya1Kisc8J7ilXcBopBK8SRvcUM08W/wCcAuBBAMcC6PL/HQvgXwDelaG/swBsB3AmgFP97bMAzAdwPYBBeGT7GIB3Stf9HMCfATzkH7/G32dhUQVY+r99MLFCbjMS4YnnmmfSTj0smjqer1MkIantbQgl0pl3Ip9116S5tTinIf6a3V7tkKfxXQAfZOa/S/v+RkSfAXAxgKvTNMLMS2FOBrw84ToG8Hn/n4VFTUj60TUb7ZCiWo3FQFQ/pRv7GiJKg2p2T+kzsitLn7Y3tWJtSSiNQi7yOXJN8DcNp6HneHRXtlPtqXkAhjX7RwDsWTdpLCyaiNarjPZANRaD45Ea6a/LcEx2TxGFCWv1dN8l8QJCCaQOueXo31JZtTTi2YmBonHlfSZ59Ncm8SvtkKdxN4ALiGi22OFvfx9heRELiwmBdlIWkTyNCeGe8qNzHD2nYZrBJ5fzjn4WA9/lpx0Jong13exVbs1unHpALQlS8jWBUBpaSyPgaaR9Kd/M8FrNfQWcRustjQ8DmA7gWSJ6loieBfAsgJkATmuMaBYWjUE7rdzXrCJ8KiJEeBWl0R2DoZGWzDXJIs49dt8ZeMne071+Yn1k++L0bpwEJZap9fCeVUujkHPgUHLtqTTKKx6CLK7Vnev9baR7Ki2n8QKAFwF4OYCF8Nx9jwK42eZKWExUtMOL27KMcNZvp72OiMAcd7u4zHBSKkJTty6HM2WHKHDv1JfTqHx+2ueiKgBhWeQcQt7Rl5DXRfCld0+Zz2+Ge6qi0vDrTW0DsIiZb4RXEt3CYsIi+L222Xynqe6pKq+Ty4gUy+nbTTNIh585CBn1iHDRtrmRW5evx4ahMbx98R4amRIGbZ08yt9KfauDuFASeccxlpAXeyI1qirIIxA2p7uvxrunKioNZi4T0Up4eRoWFhMeOn/yzoaqy4gERHjyTDd2XcY8DcFbEKVz4Xzg4n8CgFZp6JAqE7taS8MvIZLPEfIOGTgN30Waon1TqXhd9d9yG3EaZwP4tlJE0MJiYqMNtEbETdTMfiPb2TkNMiT3VWMtxTgNhtbSqBoZlFsggHJOkmssJMK9z6XA0iA4JkuDo9d6+0wKV79DS4QLTqMN8jT+HcBeAFYT0Soo4bfMfEi9BbOwaBRC90PrtUY0M7s1nEa268RMNmO7GdxTLiNiaTQiuS8x5NYglwmm5L58zvEtDT33o/aR1rWXJF/AaTSwYGFapfHHxolgYdFctFP0VCROv5kd10iEexnhupluytlyYh8czOt1mefZQ271fVQ6P22EmZrcJyrc5h2qyGmkUoiqUtUUSFRlb3lGODN/rWESWFi0CO2gNBhAR87BeLnG5emaBDF4mgYlo9LIRITL0VOakNPMIbdmN06FC1NBTe4rR6KnKAjBjV4T5zTM4coqp6HfD4Rl2ltSe8rCYkdHq91T1z20BusGRoPwyOZGT1XXWRhya5jBZxNCC5c5WA+CJE6jWWVEdHkQ6TgNnwj3LyzkCLkcaRdhCngJju+Ly6r/nOSeajkRTkQdRPQ1InqCiEaJqCz/a5h0FhYNQNY4/Eag7DI+dtl92Dw8Lv3AW8NpZHJP+X8dw3KvmtQN/7rKg7SAHD0lFyxsXhkRvVyV2gqS+3yfY85xzHkawbWV+1DPCJYr1lzbDPdUluip98Fbm9sF8B/wSqdvAvDxxohmYdEYhGuEtw7yQNJIV4IJ0Qlu+idRabnXTO4pg4XFjAinUXMZEa08KQbrCEmdntMQ7qi8Q3BIX0ZET4Sbnp3qnhPnx9HS0ugK3g7go8z8c3hrhP8vM38KwFcBvLJRwllYNBKttDTkgVDMCpvqnqqys0iehu646bqkNjWfnUiehipDRk5DS9hXvi5tfaowfNb7G0ZP+RnhGk5DH3JrkkO91kyEh5xG6y2NWfDKhgDe0qt9/vb1AF5VZ5ksLBqK8MfZOq2hVnIFmitNGgIWAJavHcDqrdtj+ylj9FSiLMo1LocZ4aZ+MrWv2ZdYe0ozkxecRpJ1JeQsSkR4zpDcl+U5GYnwBGXotEFp9OcA7O5vPwXgRH/7JfAWUkoFIjqDiJYR0RgRXSztP5KIbiKizUS0gYj+QES7SceXElGRiIakf/PT9mthIaNdOA0B8QNvrqUhbSec9+of3Imjv/2X4LNwf5hW7jNGACUSz1FEOY3aQ26zJvfp5hSJq+UpVkM5CLl1kM+RoTR6XA5jDyYiXHeqpHAbhbRK4yoAx/vbPwTwNSJ6Bt4CTP+dob8XAJwD4JfK/qkALoS3bsdceCv4/Uo553fMPEn693SGfi0sYmglpyGPI410JZhQbVKhONNxoH2A5tLoCW3qOI0gI7z2kNus8gTnJNR20u0T32lJck+ZLA1drlDajHDVslGPNfp9Spun8UVp+49E9DyAowE8wcypVu3zr70SAIhoMYA50v7r5POI6L8A3J62XQuLLEiKPmkW5DBMJyCEW6nG0iEMudVbGplKfhitkmiV25pDbrWDf8L5ihKIthWHqcpt3s/T0Fk1HFxrlsMkKxv2A0DZbfwkJG1GeATMfDe8hZkaheMAPKLsewMRbQawBsB/MfNPdRcS0ekATgeAPfe0iwpaxJFk3jcLsntKuFuaKk9K91T8spBo1c66q7qLOKchhj25YGGtuSVqH1mkSuY0ov3IVW4d0if36cJ6TRLpanOZZGEpx6VRSJun8QQR/ZyI3ilzDY0AER0C4CvwwnoFfg9gfwD98BZ9+goRvVN3PTNfyMyLmXlxf39/I0W1mOBon+ip5veflghXERCthugp08w5q3vKqSOnkVVphIOyPKAncRpRS0MuI+JxGmYiPE2+THx/tD+13UZbGml10ncA9AI4D8AqInq8EUqEiPYBcB2ATzPznWI/Mz/KzC8wc5mZ74LHq7y1Xv1a7Fxg5W8rEM3TaC0RXs2FpmHJNWiN5OQ+pQ1ptkyoRxmROFK5hdLuExaGIMJdBpEX4JBzHD2nEVxbWTHFno+r3w9EF7BqFNJyGr8A8AsgGNhfBi8/49fwFE9Vbi4ZRDQXwM0AzmbmSyuJBPN7a2GRjDagDrR5Gs3MCFfS+9JfJ3MwdZJFaUeucquzNOrTZ5ISM3NeSWHGYiJQchkFX+vljaXR45ZGWllD+eLnCoXVSKQe7InIAXAEPIXxCnhE+GoAt2VoI+/3mQOQI6IuACV4eSB/AfBjZv6Z5ro3AbgDwFZfhk8B+M+0/VpYyGgHIlxHsjYTadwiOohwWDKUEamuYGHckggywp3qcj+S2geS1WTWkGw1E7xUdoM1us3RU+LaykLp3HfeX70yauT64EBKpUFE1wA4Bl7ZkNsBXA7gdGZembG/s+BlkQucCuBr8B7XfABfJaLgODNP8jdPgRem2wlgFYBzmfnXGfu2sGgbRKKnhJO4me6paq9jz8QnmIjY7P2pxzxrRmSExxdhyqpD9G6cyu6ytN2IpgSXUXIZeaE0SJ+noSP3U/eXcH7buKfguaK2wuMbbgVwGzNvzNoZMy8FsNRw2Fh+nZm1pLeFRTVot+S+XAuip6K+9AzXwc+hyLrca4ZIKzkjXC5YGEQwZXxSWjlTWHpplaLYJyyKssvI53ylYSDCxZ5qQm4r52lUbrMWpCXCdwHwLgBbAHwGHhn+EBFdQERvaZRwFhaNQDhTa6F7SsdptIgIz9Iv+3yDKWeimqVZY4OiG80IrzXkVqcWk6On2HhOkktOWBrFMiNXkdNQN8zfQ0wOnWtLOjdzxnxGpCXCt8MjqW8GAjL8SwA+BuAT8DgKC4sJhXaxNCZSSIfgG8zLsBosjaQ2NbWVxLjnRU+pMqQUNuH8RI5F+VsJKqdRLLso5JI5jTBMV+433bNLdE+5jV0fHEjPacyER4C/3P+7AMB6AFfAc1dZWEwYtMNyr9qQ2xZZPlnLiBD5nIbmeDV5GrraSjKnEY8eqh2pkvu0RLOuLe+vWEdj8/A4pvZ0ADBbGuIm0nznukABk3zlJrin0nIaa/1/d8DLkbiNmZc3TCoLiwai3dxTuVa7pwznyDkX7Ls9PEuDjCv3VVXlVtNGyGnUHnKb3h6qsq3APcVgZqwfHMXMKZ0AYMzT0FoaJoUbu9Z8/tBoCZO6as6ASETa1g+wSsJiR0MrLY2Id6olGeGVb16O8Cq5jEKOfCvAI6Wz1XRK4hCin9WV++LJfRVFr3h+onJj8zlJlgbgWZDrBsZwwG5TAAA5B4lEeBrlbaA0tM9/6/Zx9PlWTqOQiggXCoOIFhPRO4io1//c6+deWFhMGATRUy2UodW1p9IQ4bKM4yXP9SIS77JaGsljdFwpBFVuHU3IbdboKa1yS1BikhxpIN/zaMnFpqExzJrSBcCrP1Uqm0ujp3MNmtxT8TO3jhTR111IJ3iVSFt7ahYR3Q3gHgC/hZeMBwDnw1sC1sJiwoBjG82HrvZUM5MNObKt71eWUUQGMSQiXNduShdL4rkRTqNRlkZ112mjp6TG1g2MwmVg5mTPPVXIEYqagoVhRJjcX2WFyxxKoFPQW0eK6OtpA6UB4PvwOI3pAEak/X+AXbnPYoKilZxGJE/DaYWlkcI9pbE0QitAzzVUo/e07il/mwz91ApTjSxPHsE3pHNPMYff4daRcQDA5C5v4O7IOxjXWRoaOdIq3ERLownuqbSupeMBHM/MW5QY4BUAbP1xi4mFNoiekgeLVkTcsvFDCDkBbixQGsl5ANW5p+KfnQROIyt0VydZGqz8rQSXGR05B9vdMgZHSwCAroI3H+/I5VB2GWU3Wt6DNZaGUR7FlWhyr44WyxgtutilHdxTALoBjGv29wMYrZ84FhaNR9ZBoRGQSeZgEG6RQKZuZRnHA/eUH3JL+iuNZG4iER49FoueqnB+JWRd94MzviAMz6IAgKExT2l0FrzUtULeu5GiYm2EnEaa9iVrJHJt9OJt24sA0DZK4w4A75c+MxHlAHwBwC31FsrCopHIRkI2BtE8De9vU6vcZiTCxTZzcu2pLJaGSVcyhwfrYWmklSd2TkoC3WUOlYawNPKe0ujIefuFpSZfI/9NkinOaegt5bXbvPm7IOEbhbTuqc8DuJ2IjoBXNPB7AA6EV17k6AbJZmHRULTS0pB/8K1YIzzN3csDWqA04C3yYybCqyE14tcLRUp+nganGFyNzWdQbp44YkBP21aoHISlIdxTnb4yGVeUhvbZpQhIYEku9exVW7YDAGb3dWvbqRfShtw+CuBgAHcBuBFAFzwS/DBmXtE48Sws6o+k9Qia0j8zvn71o8HnlteeMgxWOkvDZZERbljuNcVsOXZM6j9cGZAif9PkM6RpX+1He36C68ikKFX3VJfvnhL7VTJctJ0uMz26bbp29VYvRmn21MYqjYqWBhEVAPwVwHuZ+asNlcbCoglodZ7GWMnFMxuHg8+Bm6aZSkPezuieAsyWhrGMSMpEQFexNMTfNG6cNHBd9nM/KnMaES4hoc+IpTFqUBqSpWGymtLfl37Ss2rLdkzuyree02DmIoC90Fpr3sKi/mhl+JSEVrin0ty6PLCGpTA4rD2V0e2TRhZxPUl5Gt7+zM0mtp9mNEsbUhzhNBT3VCGnUxoaeRJEUq1C8VnNNB8cLTU8RwNIT4T/GsBpjRTEwqJZaLWloQ48YhGmZsqTZvEfeVAK13Dwy4hoCgmK49r+UriD5G2SOI24vNmelNy+iAjLWrCQDMEKzN4gLpTG4JieCJejp9IoikgfisWjlmKX223GBCSt0ugFcDoR/YuILvLX0Qj+pe2MiM4gomVENEZEFyvHjiei5UQ0QkS3+muGi2NEROcS0Sb/33nU6KLxFjssWm1fqANW6LdvnmRqRI4OOiLcS7wzJyNWcw86V1kip5GVCJe2Re5JqoxwjVyxc/z9ldxTcvRUtF3Wbuv6UK9XCyE2Y9U+IH301P4A7vO35yvHsnyFLwA4B8CJ8HI/AABENAPAlQA+DODPAM4G8DsAR/qnnA7gzQAW+f3dBOBpALH1xC0s0qJV3ilVabS69pQJ8kQ2EnIr8jRSEsWV+pMHy8A95X/WcRpZIbefxtJIyrhW94l2Ynka/mcdp2HiZ4zPTulfXBPP/eCmFL9MuwjTy+vRGTNfCXiFDwHMkQ6dBOARZv6Df3wpgI1EtNAvlvg+AN9j5lX+8e/Bc5dZpWGRGWE2bmu0hjrLbfT6B5WQxj0Vhtx6s1mvym0ctZZGF9uqpeFKg2Ut31rSUqkxuXTut1h73l85T6Mj78Dxv9ROTfSUidNIK0dgaSg1rZibU10grXuq0TgQwAPiAzMPwytRcqDuuL99IDQgotN9F9iyDRs2NEhcix0BLePBY0oj7oLZNlLEb+9+rmEuq4jCNHRhck8B5pX7skRPBcc0g2jIacRlyfrFRd1T4j4qn+/qHpHSt87S6MqHw2pHznNTqXkauuaMLrDINgfPPRbGi/biNBqNSQC2Kfu2AZhsOL4NwCQdr8HMFzLzYmZe3N/f3xBhLXYMtIt7KrQ0wv2fv+IB/OdVD+Hh1QMNkSGdeyquNCDcUwCGx8u477ktSrvp/PJ+U/F9/jgor9wn7zddl9i/tCsaOmxqI95PpefV6XMag6PFgM8A9GVE5O8/qrwqPzvZPaWWXHfd5qzN0i5KYwjAFGXfFACDhuNTAAxxK+tAWExYtDp6ykSEy9g45JV6GyuVGy6PMblPwwUwogPTST+5K9pWitly0lHV0pBLrKRzKelaz8hpBHkQccdZ3D3l7emf0om8QxgYLUXyJDoqhNymc5FyZEtco5Zc39ksjUfgkdwAvMWdAOzt748d97cfgYVFFdAPCs2D6hrp7vBmps0UJ41bxNUm94VlRLTX1JinkchpVJBXvt7YPlcvp65v8Yhm9HbidYfsBgDYu39ScLwSEY4030PkHA4iwOIhtyluoA4wKg0i+gsR9fnb7yWizlo7I6I8EXUByAHIEVGXv/LfVQAOIqKT/eNfAfCgtMTsJQA+S0SziWh3AJ8DcHGt8ljsnGi1faoqq27fndFMsVIt92oqIwKpMq/arsnSSJzZh0jKCA8tRHNblZRBFveULspJ7Vu2jI6cP93rQ7ouCLkt60Nusyb3ydfoKue22tI4GkCPv/0reMUJa8VZALYDOBPAqf72Wcy8AcDJAL4BYAuAJQBOka77ObxQ3IcAPAzgGn+fhUXVaJXyULsVPvB2szQi7ikpeoqIjFE6xiq3KWUJrlc4jbSWQSXuRCX0tW1o2jIqQ4mDOXbfGQCAty/eIzjeqSHC0/IzYd+KeypQalHF3g4ht8sBfJOIboU3uXg7EWlZOWa+JE1nzLwUwFLDsZsBLDQcY3iVdj+fph8LiyQEg0LLQm6j/crEabMQ99bH4WrzNPxV9QyDUzVPVBfJJa+n4fUruxVTtiX2aZRS5uQ+Q9+yZTRnag+e+dZrI1aYtvZUhMPRy2mUg+PL8Oac0FJtdXLfxwD8EMCbfHm+DYPLEJ77yMJiQiApeasZiHEagXtKM+A1SIZUy73qLA0Oq9wKyGR9lugp3bF4lVuxX3ZPpWsr2KcZpBPdZQm8h7onUBpO1DISKOTi0VNRC0a2IlI8O47KIGeFu622NJj5LgBHAAARuQDmM/P6xotkYdEctIraUNen7u7wZqMtc5cZOo4Q4UH0lLfcqzw4jYyFSsM8g9dZAPF9aka4rmBh1uckn5+G04DGqqhEUps4nnzOgUMpM8KNlkZUsUQsjZLrrXAEUba+faKn9gJgM+UsdgiEPuvWjNJqt0nuqUYNAWncU5VW7hMQpTPEcW1/iS6l+HYsT4NTBqdW4DR0K+ZlaUM9pi4apUNH3olmhEfazeYn89xT4eeiK1sw3JTqAmnLiKwkollE9AkAB8C7jUcB/ISZ1zVSQAuLuiOFm6Ox3es5jWbKk0ZfmtxTasjt8HioNKoLuZVcLH4/pOM0gu8tYcCv8BTlKLAscpmjp6Jy6tCRc4yWhswbpc1xkeWSczWEQm80UlkaRHQ0gKcAvAtexNMogHcDeJKIXtI48SwsGoh24zSaavnoXSQydHkawm8ucxrDEfeUwS+fVqoETiONttd2Lw/SKSyN8FypCUPfaoiwDh35HMZKLtYNjHrchq5dmBWemtYRWeekHCXYWx1yK+O7AC4HsICZ38PM7wGwAMD/wFsv3MJiwoCVv81GW0RPRUTQPwlTyC0QzQofkSyNdP2Zj8UzwsPkvjTQ6gxpW8zs0+SNpNHh4fNIsjQIKzYMYck3b8FPb1sRuZd0UVyygueIdRIpT9KkMiJpS6MfCuD9zGGEMTO7RHQ+gPsbIZiFRaMQRk+1itOI9tvd6pBbw2Mo64hwjudpmOoqRfpLHKTl2baHkNMI200TPaWNeJJ2yeVQjPJIpL8qmHqdsMYS3VN5B/c8sxkA8PzmEUUJ6OU0ya9aGhH3lB+k0GiktTS2wSPDVewFYGvdpLGwaAJab2lEP7cieirVLFoebKXlXh0vJTw4VirrB8FKIImvEIhXuZWIcDFwZyWxNRaTGsEWOV/TVrgvep34mDRUd0hVb6f2dqSy8mTooq3yjq4QYhtxGvDcUBcR0buJaC8imkdEpwL4BTy3lYXFhEPrQlyjn7sS8jQaJ0N8dq9CXxo97gKJ5goY+kuQQZe3QAqnkWZGbuxI02diaXSNcjLl9oR5Gub2ZKUxViwnVLk1yKN8dpmDdToiRQu59cl9Mj4PT4n9UrqmCOCn8EqCWFhMGKSJwmkk2oHTkFEpBwFQM8KrcU9lk0XNCPcKFlZupFKCZJoyIsFCTSnkDYnw5OgpgbGSa07uM3UY0QsMlz1FNDxejj37fLsoDWYeB/BpIvoivOqzBOApZh5ppHAWFo1Aq5SFgBhovvPWQ7DvrMnIadw0jUaWARFQV+6LWhvlaJhRTbI0JCNcx2mksjTiMsZn/d7fRCJcsjRGi2VtBrzXtkHhKtyKZ2nkABSjrkG0l6UBAPCVxEMNksXCoqlotXuqr6cDh+7Rh5WbhpsuD6cZrGSlIbt1KGpryAOXOeQ2iYeIX+8onIbLYQuJA36FfeLadJZG3ApQL0uT3FdQLQ0N8Z8EVXkxh4pItTR2pkWYLCyaBtMAILDgrOtwztWPNqz/WKmMptCXUZgieCLnyDNipWChPDjJWclGH33KgT4czKPraaR3b2ncUwmhw7prdEUNQ0tDPRcROXUYLZYj21ndU2oJFZc5IMJVkrydoqcsLHYYVBqAxksu/vuvzzS8f5U8barhk6IzebCSyW6x3KtApDy30cWSThaV0xD9pF1Po9Jt6Vbui5Pb+v06pEnuG9ge5rGMlVxz7akKfXjneLWnck7I9YRtcVtFT1lY7HDQzUrlWWGjEIaVRnMRtLPkBsnAhm0ZWk6D42VEihH3VDWyxAfRgNNwwv1pSqNXyvR23Xgb6jW6PI1K0VNJM/yB0SIAT7F47qn08sp9h58hKY3os29G7SmrNCx2Wuh+ruIH3kiYXBpaf3yDtIbJRRI9J6403MA9FcoeLfttaiudLKY8DdfTGpWhOUdHPMvWkaropJSUpGYjbSe5pwa2e+/UnKk9vnvKYGkYHpLqnmIG8n7JdVex8tqKCCeiDgAHAZgJRdkw87W1CkJEQ8qubngFET9JRPMAPANgWDp+LjOfXWu/FjsfktbTEK6ERmZppyFPBapdy7qiDKnCVz04pGaER88rlvRrRVTqT6skgz6jnEZtZUTiyi/JpaatT2W6L8WdpsPwuGe9zu7rxoahMaOVk849JTgNxz8mnddOZUSI6JUALoWnMFQwvDW/awIzB6uxE1EvgHUA/qCc1sfMlQvdWFikgG4gG/QtjUldmQILMyEI00TUPVVpllxPpPLX+4Lmcw7K5dBlQ1DcU8pCQJllkftULA1Hct0FBkBCH5VCbnUKIVppVs48l2Xk4LiMNHkahRyhWGbMmtKJ57eMaAl2k+xeH1H5jEQ42osI/zGAq+GVDemBZwWIfz0J11WLtwJYD+DOBrRtsZMjHHzixwZGvTnJpM7GKQ3V0ghKZWhn441iNWR59PvFYNWRc2KWhhzxZVqVLk0f6rF4RriU3BdwDQltVZHcJxdcrGRcqPtUJafDnz95DL510sHo7sjFQ24zchpe9JSe02g3Inw3AN9k5pXMPMrMY/K/Bsj1PgCXcPyJriSiVUT0KyKaobuQiE4nomVEtGzDBrtulEUcYRROHML/3EiloSaEJf3QG2ZppDhHDEj5HEUIZFKS+4qGtSLS9qdbjjVUqMntxtqqcJpoR9Jz2Lo95LF0Vo/crinSKmmGv3DXKXjni/dEZz6nCbmNSJ8oc3hNGD2lttVOpdGvBnBUIwURIKI9AbwUwK+l3RvhLT07F8DhACYDuEx3PTNfyMyLmXlxf39/o8W1mIBIisIRRHhvZ/M5jUqulUbIAFS2ZvIOBSG3gXtKOi6H45otjXQuJZVYFj3VlNxXwT21daQYO26SK962/rvUobPgYKzoxlxKlfqQ3WeejKboqRavEa7gowAuI6LDATwMr+5UAGa+pI4yvRfAX5n5Gan9IQDL/I/riOgMAGuIaAozD9Sxb4udCvFf6VjR+4V6ZRoag2A27Sght9pzG6M10vnSvQMFxT2l5peMKwsB1SKLidMAyxZikgJKdvEJUWUifNv28Vj/pjaqSe4T6Mg5KLmuVoklQVVkHOE0ZBnbq4zIiQCOB/BaACNQ3jkA9VYa365wjui/+am0FhMeJlcD0LhBWtdHmozwhkmTomExIOVzFBDhXshtlAgvlStnhCd2p8yWgfCZONLgmMrSqGCtuZroKdnSqBQCG3dPVeY0BIjI52b07ZluSyXkPUtDRE8pz66N8jS+C+C/AExm5knMPFn6N6VewhDRUQBmQ4maIqIlRLQfETlENB3ABQBuY+Zt9erbYucBK38jxxqvM8IZj5qnUScldueTG/A/9zxXQYbKLqXA0nCcyOJFKhGepvZU6sgwhdNwGsRpyAphi8E9peZHSOLFzk8zwxeFKUuuPq+l0vcgzpGjp1jROu3EafQB+BkzD1c6sUa8D8CVzDyo7J8P4HoAg/DcY2MA3tlgWSx2cOhmkpXKYg+MFvGt6x7DeMk1nFEZ8aJ8vjzakTV7+++56B6ceWVyXdEsM1xBhG8eHsf9z20FEJ1Zj6eJnkpZ+iPuupM4jSqjp2QI5VdmRl9PAUTAthHZPaVvq9IzSqU0/NE2miNSGar1xgzk/OQ+mdAXiZeNRlr31BUATgCwooGygJk/Yth/OexiTxZ1QlL0VKUksvNvfAIX3/Us5s/oxTuO2LPK/qOz08Bt30R3WZpmhZyFnIOSy7j/uS0AvMxmGWkywtPKorru5IKF1bunJOshcE8BecdBTyEXJN/J/attmaOnohOAJAgFKJSGQ/GQWR30yX36PI1mlBFJqzSeBvANIjoOwIOIE+Hn11swC4vGQbgo4kcqDdJjvoVRqqRdEiC8E8HsNOGH3hR3mXGw8v7mcw5c5qDG1KlH7ol/PL05OC/Ncq/JA3185q2GI6d2T1XoW17uNed49yZzMiwn+qXoQbSdJqlORDyJdyfnUEqLL6pY5IKFaln5ZiT3pVUaH4TnGjoK8dBbBmCVhsWEQ6Uy2o2CiTzVWz4NsjQq9Cv3XXAIpTIHA24h50T03HgaIjxJaWj6VFfuY5ZPTHB1aaOn5Pa9v2Vm5Ii8bG1DRnu0Lf1EI4ulEXIa4vunVOVR1Agpl6GNnmqrMiLMvFejBbGwaBZqcU+pbVTVv/83NDSCWhnGc+sNlUDVn+P9zTmEMnNA4OYcNXpKP+imlyUuV+C6k4jwNFVuK1kaQZ6Gy3AcQp4dJfor2T1lajsNpyFOKYvnSFQVEQ7WR0+llaNW2Cq3Fjsdkoa1SoNePX6TpoEx6dx6I02rzF6yWD5HKLuhpZF31OQ+OU8je38RS8BvihRLQw1VNcusaz/cKZcRyTmEnG9FBf0bFIWJT8kUPSXcU2WTeyrZTSjOinIaUVnahggnoguSjjPzp+ojjoVFE6HlNPxDhhGqHmN425VGTxisHCLkHAdltxzxxcu+8/EIp2F6btk4ifh6GpVn5NEW9BD3UPbvrZBTM9plfkVn+UX3hWVEErsFEOc0HKoc7aXK5PWp5zSY2yu572DlcwHAQv/6++oq0QTCw6u3oauQwz4zJ1U+2aJtkBS6mVQ2vV6IhdwGfevObZQU6dwiBCDnR/mIGXLecepbe0ohc4FkSyNpoNU9L1kkEenlugyHAHKciKUUnbnrZYz2F5U3CaRwGjlH4TRSuqeinEb02bUTp/FydR8RdQG4CDtxJdrX/+ivAIBnv/26FktikQWhqyH+K9WupSChHj9KdXYaVLnVchrVaw3ht9chlasHoaXhEeHe4JrPJbinalRyQZVbUTbe35+a09C6pzzkHQqURtn1ZusOkbLyoIHTMLSvuhqTIIhw1w2v0UWOqdApMpGnoZLk7VQaPQZmHgXwDQBfqp84FhaNRxIRLjjRRpYTUct/J6EWS6OoVrqTZTBsR/v2Zq45x7c0JE5D1p5plntNO9AHxLI/MkWT+8xtBNcnpJ4Xck4ga5nZd08pIbeGtkzvTJbaUyK5Tygux6GUyX1xTiZHcUuDm2Rp1EqE9wOwvhmLCQntrDSwNCpcW4d+Y8l92nOr70kmeE0yqNvqOZ6l4VW5LUtuFRlJq+BJrZllkY6pg7DOUFJbqsR3eDNwbzEkkckvyovncxThNFx16q5s1hJyqyb3edFT8n3or9NFfwXuKaXCcNsk9xHRZ9Vd8NbYeDeAmpd6tbBoJkJOw+yeamS+hrGMSJ27TFIaaeC6wtJw4Lphcl/eieZplN3kQTtpv3rMlBGeVBo9zcBLADryTpBTItxTBccxLiKVrgKt336m2lOh8k1Tol5VDICXlCj3L+RNKn5ZL6Qlwj+pfHYBbADwKwDfqqtEFhYNxOBoMVidL4l4rmRp1PLTVGfTST/0rG4y2dVSSnRPVR6sAk6DPHdO2ZSnIRPJVfjT5CtCYllYGmJGnaCQItsaXsjf1ZFzAtK+zF4fnhWlJ/J1RLjafpb1NISFFpQRcRJCfJkl11y4X9TO0q7cl1KOWmGT+zLCdRlPbxy2EVMTFG/72d+Dbd0YVIkIT7o2LbJkhGe1PsYjSqM291TIaXhEuMxpyIpOLpqX3TmlR9LKfaaB29vW9e0NwAXJ0nBdRs7PQdlelBWFftt0I6qrMQmOEnKbIwqUQKwb1t+76iKMZoQ3p4xIVZwGEeWJaKccNX96+wqccP7teGxN+6/9NDRWwubh8con7kRYvlYqoKzlNLy/pvG2Hj/JGBFO0f0ysk7c5eq7stslJkOKtkJOwxu4yn6YqqNYGmXDTF1tK6kf9fowT0MqIyKkVgfuFPdBEES44p7KJYXcxpWR2leW5D6hCMXzchyKrMpnVLgaReaQn+ehyNhyIpyIjieityv7zgQwBGArEV1PRH0NlK/t8I+nNwEA1g2MtliSyjj23L/gRWff1Gox2ha6Aa4Sp1EP2mHjkKfI1dm0tr+MpsaYpDSSifCoW0OHiKXhepZG3g9riobcVuY0khAhwpWM8EjIrWHgrmQ1sd9eR87BeMk7QURP5ZWMcNbM6pOQKblP5TRi66nov5OIe0oK13WINO6p1lsaZwKYIz4Q0YsBfBPApQA+D2ARdrKQW/GjbORyoPWCvLiMRRwll2ODclr3VLUYLZbxnRseB5DuB57ZPVVKx2mk6SRiabiMUtkNM5Gl86JErUnZpnSV+X8dhdNgJMzEI9FXek6DEHVPiegp2frwrg+vi7rd9JOJYOafgkwQ55SlMiJmy8zgMpMEdJSCh80qI1JJaRwM4Hbp89sA3MXMp/nl0D8F4I31EoaIbiOiUSIa8v89Lh07noiWE9EIEd1KRHPr1W8WBEqjYMt27QhQ/f4BEW4Yb2v9UYo1yIH4bNrkj8/UfinOaQyNlWKDXRpl5LLnjhK+d8/SkN1F0X68a/RtpVV+pjXCk9ad0CkdHTpyFBLhriHk1jBYm62c0F1UCY5iaThqyK3hOl1yn0Oei1CVMY3yqhWVRr4+AOulz0fDW0FP4J/wlmetJ87wl5SdxMz7AQARzQBwJYAvA5gGYBmA39W531QYK3oLthQcqzR2BKgunDBPozGWhtxuWLBQzKZr5zRkl0qpzNg4NIaDvnoDfnJbdP20NKvSuSwijByU/dLo+Vx8UCobBt1of2ao60UA0noakTIinKmtsG+P1OjIS5yGf295n+TXyZ8mlDhwT6WYTojkPrlasGvQGnJ3Zk4jqnTaxdJYA2BvACCiTgCHAfi7dHwyvKVXG42TADzCzH/wM9GXAlhERAub0HcEQZx3A+P4LZqHcYUsFhZGxa+3yu9fntWS8rce3ciulmLZxfoB7+f55wdeyNwH+4NQzkFgaYiS3LLSiXAa2cSN9R/OpKN/mRPSBivN1jkkwmPRU0rIbaU8DXVXluS+mKWRkBFukkOMO45DcCi+dGw7RE9dB+A8InoFgHMBDCNaa+oQAE/VWaZvEdFGIvobEb3M33cggAfECf5a5Sv8/U2FcC80ssyEReMhflslVWk02NKQB6hYlVtNl1nliFgaLhvdW2maDTkNjwgvl0P3lPmayj76JKghpWGehnnWryv3ET0uMsKdgPOJuKfSWBqC01DazpLcp95LTo1+MiX3yQrEDS2xGBHO7VFG5CsARgHcDG/1vtOYWY7h/CCAeobnfAHAfHgurwsB/JmI9oZXqmSbcu42eJZOBER0OhEtI6JlGzZsqKNoHsZKnnuqmiSmRuPVP7gDR3/7L60WY0JARK4UFfdUmNxX4fut8tcpD1Dqehq6HrO+ZSXFPWVCNOLINFj5nIZPhBddiQhPwV0kKQrXYJ2Ea3b4UVoBp5HEK0Q+aWTyMqVl95TLYe0pMxGu4TRifEp6S0M8u6JpPY2IxZSsyETos3p9y5P7mHkjgOOIaBcAQ8xcVk55G7zw27qAme+WPv6aiN4J4LV+H1OU06fAW4JWbeNCeAoHixcvrvvIPibNVNoNkRwEi0SIekpqLkPq0uj1cE+JonwJDqqsIbdRS6PWgoUhpyFqTwlOw3xNOsvGFKZbUiyNkO8J+4wN3An9CAQht5LSyDl+yG0aIlzfbKRibSWIc8qStZDK5aZ133nuqVhp9CawGqnYXGbeplEYYObNiuVRbzA8d+Qj8MJ7AQBE1AuPa3mkgX1rESgN656a0AhnfdncU7Wa/7I7LMk9Ve26HrKikC0N1X2SZlEj4e4IrTI3dE8ZrRP5erOcpgzvoPx64J4KZam0poXXlu4+vL+FHKEo8jT8svH5nKPkaYTX6SaGJvdUluQ+OU/DVIo92kdckREQd0+hOZZG24QAEVEfEZ1IRF1+xvm7ARwH4AYAVwE4iIhO9tfx+AqAB5l5ebPlDKtkNrtni3pCTbQSEONHo6rcaonwwD0Vb7UWTiNaiM88OzfBG4QoiPoZL7mB28h4TcTFIm9He0xracgFC02yR2bihoFezQh3WazcR5ES8rK7qcxKI5rOxUdKMZKGtafC6Kl0IbdxpeGF3FJcSbcBEd5MFACcA68Q4kZ4RRLfzMyPM/MGACfDW79jC4AlAE5plaBAe7qn6oE/3b8a+3/5+kiS2I4IEc+u3mdaIrxaTkvHaQjIXYYLM2VsXyXCjf4P7WYEIafhDRNjJX1yX6TZlO4pNeonkNl/PgVfU8kr9xkFrnAv7LvZYlVuyeNOmOW1w71r8jlHS1KbCxamT+4T9+gkJveF2zqehZQyIlm4lVqRtsptw+ErhiMSjt8Mb4nZtsCO6p4655pHsb1Yxtbt45g5uavV4jQM6nrNAmndQtXOGWT3UZpJYWZLQ1JK8hoYMfdUJOLI7GqSLY2xoqvN0zDJm8THlA1+LHn9bE/usF2jG61i9JQXOtyhRE85fvSU16+LnJML5C84lDJPI/1grYbc5ghRhZci8ky2NBwKa1dlyRepFe1kaUwoyDPNI75xM9Zua/9aVGkgXuhmvHythPiRxzgN/2MlS7LakFxZSanRU/VANHrKDT5XmxGOiKVR1maER9o1bSdYGtH9Hm+ilkZnTprty33G2y27jFwuGinFzMj5taeAcPYfLHKUc7TWkHofWTgN4RKVy5ub3G66elxAGHnlOIgQ4VmUV62wSqNKyO/8hsExXPPQmtYJU0eIH8qO6n4TcCRyV0Zq91S1SiPinvL+CgWtzWbO2I2aEV7T9ygsDV/OsRSchokIV6Uwuqf8/AmBwNJIzNMwffBQLDMKOQcdeQeu74oqc1h7CohzlYWc3nUUVxqhu6gSxKMTwRCOEj1ljpgKPwg5VU5DnNIOZUQsDCgr4YwVrPaWIkvYpjg1qaz2jgA1Zl4gzNNIvr4+7qnKlkbmRZik9ouum+CekrYTXC8OATkxsMoFC1O4UpL2y+7dSMSSkkAYKVjI8fPVtnW9F8suOnJOREGUXW+A7ciH9wZIloajWBrCbWmwctKtER4NuVWjn0yQ3zXxuyQifxGndJOcesIqjSqhDjaN0vDnXr8cNz26rqY2ssw2xbk7utII/MvGPI3kZ1btDD7qnooe0/vjs0GWq+yaLY207inP0vAElTkN0+WVXEWBbHKYq1KSJBdRGqEsqVxiWkvDk7vgyz5edgOFGCiNUhhVBXiLM0WsJsN9BBnahuMyYpyGEj0l9xK9p7ilQYiH7Mp9NBJtQ4RPNKiJU/Wu+fLYmgH88q/P4LqH12LT0BheecCsqtsqM6f+osUMMGnVtx0B1eZpCFS7hrgud0K8OboWs1sa4fnFMhsT/NIs9yqaCohwKXrKBFPeQcw9ZbI0XA7WvwaiBQtN8lZSVMWyi0LOQaevIIpl14+eomDfWClqaRRUTsNg5WTL01AsjaSMcA35LWQXbcml0bO4yWqFVRpVImZp1PnL+sil9+K5zSMAgFG/3tVFf30Gc6f14IQKCkReXxgwl/nWwd2BLY27VmwMtqt1Twn+oVqdqro1geQJR22chpsqpNM4gxechu+MHy2W0eEP6EnX6A+ocrraQ6qlAYShpeGaFmrTZr4DCDkN2T3lshc91RHjNIR7irRh52rzWQbrnEK65yjZGgv7CLdDTgOR0uihm6yyHLXCuqcyQNb+al0fdRWuWiE3J+pdnX31o/jwJcsqXqu6JCqFB59/4+N4ePW2yLnqYLoj4F2/CKvUGKOnUloa1bqnkp6r1j2V1dJQ1gg31Z9K0yozw3FCS2NkvITOQvLiY+bS6NH9puQ+ET0lQ/j+0ygkXf8epxGS3sWy61e51XAa/uPL55xUZURE1ny6goXeX6Olob+lKBFeFkqK/NLo0ffVhty2GaKJU9HBpt6+RLm10WK2Wb+6Ip14Se95ZjMu+fuzkXPHSy4u+MtTOOkndwEIf8Cqr39HgzrrEzC5IQQqLQdbCTp3UeieireZtZuoe8o1K7cUA2LAafiWhstAV96pcI22ixhMcuksDW9GLYW9KteYBlsBj9NwAgVRLLtB9JRYgXO8pBLh0TwNBN97tG2Ry5IGIafhZ4QnlBFRt1WLSKynEZTy98+17qk2gzwrjZmuGb8sZsZPbluB1x+yG+ZO7008d7QYK/uViPFydLYmXE5v/7m3FErOIbx7yVwA4QusriuxI1oaMsRAGFtPo4KlkTa6Sgdmxg9vfhIAcNTe04P9QRkRTZtZ+4kR4caZvyyXvi2XQ8JVQKxYGS6WpF5f2R0Wl9ON7C/konNZUle4U91TFbRGscQoSJbGWMmF64ZZ4oCJCNcpVmWSAU7tElKjp3IJ62nIcJmDbPbxsjcWqCv3cbC2urU02gryQBpzAWX8da8fHMN3bngcH7j4n9rj8uxlNGNJj1KZIzNOdeD40lUPB9sm5ZB6fekJipwSMy9QSWmIwbKaigAbBsfw7CaPp/re2xcF+xM5jYzxUyoRXkueBsOTracjdEl15aPuKdWVZCKsVSnKipwCSZxG2Eh84NZtB+27rp+nEZaO8SwNKXrKH4zF91tQQm6D9jWWRtqBWrsIk+F5qS63wEoqiWuja4SLay2n0WYoKv5iGVndOWJms33cYEXInEZGS0POBAaS6ySZ5Jbv9eoHX8CDq7ZmkqHdESb3RZ9NJUuilrj4otSoLkkuOvsX/WTrI1yPgrz3QPjA1b5Mg5Uig0PA9EkdwT5haQjE62fp3S0mOYHou6Zb6CngNFJYMaaQ246cg46cp/Buf2IDyi5jz2k9cSLcvyafo5jVpoMI3U2DMLnPtzSCbPe46yu6ngYi4cKAx104Tvw9aUa6mFUaBqwfGMXl9zwX2Sf7v1X3VFZ3jnA5rdk2ijXbtieeO6b09d93Ph1TBJFY7rIbUQZJs2JTaK18P2f89n688b/+lijjRIMY7J7dNIw/3rsq2C+eq4mzEI+1GkpDniBoV8DTuUMyE+He+V2FXKT2VLyryoO74DRmTOoM9gkOQFyjWgUpKBQA8cz1YNuNh/XKM2pdW5VcbcUSR/I07nxyI4iANx06O5jBqyG3HhEetiFkVJsXEWZpkFM5DccciaeG36qEPfmcRlmZxNiM8BbitEvvxRevfChSUypqaUQH8qzunBFpAPnAr6IuqusfXoNVm0NFonIa51zzGB5QZv7qC16qMEsSg4apmm1Qi2cHzdco5LxFbC67+zn8+x8eiKzo5v3VXxe4p6p4LhGloZQQUMedarmTsut6Cw7lHZRcNxWnYYLn949aGl2KpaFGDaappeTJqbc0Sm7c0hC++yBIQZEzTUZ4Ieeg4A+827YX0VPIoauQC/I0VE5DLVgo16yK3K/LmYlwOSNcbtP0nbgcEuHFCmVErKXRQmwaGgMQHVQjL7diWcgzc9dl/O6fzwWhsjoMj5eC7Y1+XwIf/c19EYJ2rOTGXlbVvRFZfMd1I/Lp9Jl4cU2Whmhv6/ai8R4mMpiBqT3hYCivswB4z+dP96+OXVeLe2p7UbY0kt1TQbROFZxGXqxIp0weNM0nguHl+3RKPEZgafhy5RLq5yRZM7Iyk991sXa3DPKvN4mszspVFF3hngqVRrfP06gzeHF9zqGIjGLNDbV1j9MwCKYgKI0eEOFhG6rscj8ep5GLyClqgqkVDCwR3kIEkQ7yi1PWbwPAH+9dFczKb3x0Lb5wxUP4gR8po4M865R/JLqXfrRYjv34hQvqqvu9flViUVYiZWas3DQcuV7Ib+I0hLLcPDymPT7R4TJjWm+oNNTkLgD4zO/+hceVJXTVmV0WjEgThZilobQpNqvhNEQhvmKZUTZ8v1GXjslfHxKr3X5+RhZLIwlqaHCwv8wxhSrWnUhT4kX3vIR7qiOvURo51dLwGigo62kIAjreH6ee3etqTwGmUGtp0sdAh/++iHEjJMKjVkozyohYpaHBv57fipV+lIv8Qie5p57ZOIw/3Ps8gHBG+cJWM1chu6fkH54aAgr4SiNm2bi48M6n8W+/ewB/fvCF2I+wpER6vfQ7t0WvN4TahvfnXb9pyFvNt0MKg3x49TY8vaFuS8O3BGVGRGmMKe4Jge2Ka1D8SKtxT40WzZyGt160PFCIATJbH56l4fhroLswUW1p8kyYQ9fL5C4vOr8ipyG9TlH3FJTz9JyGztJw1JDbmJyRT7H7CNxTkoIQSjAWchsk90XdU+L3Hg/35dQ8grrcq7iu0lfBfk6JQ8DQmDfx6OnIK+tpCEsjlSg1oW2UBhF1EtFFRLSSiAaJ6H4ieo1/bB4RMRENSf++3ChZ7nlmU7A9VpQVhX52JLBlxHPliCiNpNXv5FmnbOKrpDfgDWTq4FUsu1jn8y1bhscjP7yi4pbQZsn6/URIyMiMT1gantIQgwYAvP5Hf8Urvne78d50eGHrdty7cnOma+oJdb0T1+WIr16daZpQi3tKnihUciME/vuM/YhBN58jvzR6Za4tMU/DF3OS//0LS0Ncoio/Y0kPpRPxfsqr6Xn74ws9iXUjRAtxItzsBgO8iZGnNMJ2uzvyQf+ALrnPI8JV7k/nnqqW0wijp+Kyq+4ph7z1zAdHhdLI2TIi8BINnwfwUgC7APgygN8T0TzpnD5mnuT/O7tRgsjJRaMSLxG1OuJvp3gJ1BdRh5FxvX97zJD9PTRainwult1gVvzc5u2RmX+p7EaUkm5WvG5gDA+u2hqxmGTFJMoVbPM5jZ5OffmIZzYOx1xfOnzy8vtx8k//jnUDrVms6uXfvS3yWXVPmSyNOPEJ7XlpoCp+Gap7Koy/z4aSX4aj4HhEeLColjKYJM3cQxlkS6MAABF+A4hH60SeS0IfwtLoyjvRkFttRjj5A7hezkr3Uip70UfidwkA3b7yE0rv9ic2AAgtbxFaLO7HxA1lCblVi2SKZxe6mPTKz3U9hZN3CAOjRV/+XMQC2ynLiDDzMDMvZeZnmdll5qsBPAPg8GbLIisNeRAvambiMkJz05+dJORuyEpDfulM5LkwSwXGS2Hc+i//9gze+rO/h7K5HHGN6ZTGOy78O974X3+LWDaDkmIS9yeOi/UF/vls1Fp4+Xdvi7m+dBD3laXM+5PrBoOAhFoRdzNFiXAdpwHEn11gadQYPaWCSM8z1GxpiFDRhGaSBmNhEU3xLQ01e10d4NPKKwbhrkIultwX4zQo2m7MPaXIrKLoV0iQXaw9HeJ+PPmXrdyCzcPj2OJb1iLMWF0qILYCItKTz3mHkHMoeA8KGt5UB5e9+lZ5h4LfaHdHLrKeRhA9tZNZGhEQ0SwACwA8Iu1eSUSriOhXRDTDcN3pRLSMiJZt2LChqr7ll0v2Q0eIcM2gIaxfoSxMVgOgkKKypWGwTlSlIVsaKsZLLlZtCZWGzpUiXj7ZbSNmMUD4oxaDvUPABbc8ibdJyml1AmejYlpvZ6yPSnjl9+/AK79/R+rzs8B1ORg4gPgiPMF+5fsIQ2GrVxpPnPOa2DF1hhjOILP1USqH0VNFN6mMSFpOw9ue1Ok9q+GxqOJTifDorF/PWwDh8+vuyMUsDXNyXyiXri0gzjUye65amdMQ/apYs207Ng6NY3JnPnDDCQJe/PbV711+RpVARNhtl67gtyVkEGOM6dmx7wKTy7X3dOSiRHigNHYiS0MGERUAXAbg18y8HMBGAEcAmAvP8pjsH4+BmS9k5sXMvLi/v7+q/gt5Pceg8/krcgMIB5qxsostw+NaQlzwH0DUxDe5tIY1SsM0qxsZL+PqB8PlZ5NI21uWrw+2H3h+a6R9IFR8DhEeWzMQufbob//F2K6Mx9cOBs9LdbNVguBU6g2XOXBRAPE4fYExwyJNWQfzgdEinlg/hIIUxaNCN2hk1U0PrNoK8v3fJalgYXywi/elwuUw7v81B+8GAJjf3xu5RrY08uqa11KzqqUXJCHm45aGGsZLviwm607uR81pEm135MP1NIAwGgwAfvvhJQC80j6bh8cxbVJHuJ63G+UH1Sgq4TpKi9l93aEMQmmMa0j2iKvSqzYscz1d+Vw0T6OJZUTarmAhETkALgUwDuAMAGDmIQCiJvg6IjoDwBoimsLMA/qWqkfBaGmY8zTk4+LveMnFUd/+C7YXy3j226+LnLtifchByPXZTJbGoFZp6OVftnIz/iUpgKRZ8TWScvnCFQ/F7mVMIgCTBsqhsVIwGxVYsWEIrss48Qd3RM6rJ4RSkfkJGQOjRQxock3KrFgaBiJctRbVDFwdtm0vYqxUxszJXcG+U37+Dzy6ZgC7dBf0F1F08K7GolmxYQhPrPPeqzlTuyN5GurEoZJLR/QtBsQ3LtodL9uvH1N8bkPnnprUlTfaL6prTjzHroKDYemYztIQBQvNhTvDXkfG4wEjgJfMmc85mNpTwJaRYqSe1h7TegAAGwZ8pdHbEQm5l3/rqvtYfkZpMGdqD+5+xnPxChlGE/K55D6ER6Kr4MDxo6nE+yGUslrssRFoK6VB3lT9IgCzALyWmU2+DPEtNkSvRjiNSHJfsntKnBtYGqWykfx8SlIasnVhqjOlWhrjZTYOKPeu3AIA+I8T98N3bngcOmqlr6eArSNmV5H4oYgfiWcpmAewx9cO4PC50zAwWkS5zFi9dTte/6O/Yv6MaAXftJZG2pDWF519EwDElLLAa35wp9aN5rqMLmngCAvWRc+LV8EVf83yHXferdi2vRiR6VHfSpvSrf/JxWtDZXd/yVZZIedgZLwUzM5jSiNF8zKnASBQGDIiSqMzrxQsDCEns8rydBZykQRSbRkRv8aSUAhJ96IqDfGbFL/pWVO6sGWkGLE0+id7rtO7VmzEpuFxzO7rChSB6zLG2Y21J5AluQ8A5k3vCba7C967IBSqwdAIiiIKS0NMdvKOE4wdT2/wglH2mpFcMbseaDf31E8B7A/gDcwc/NKJaAkR7UdEDhFNB3ABgNuYeVsjhOiIKI2ytC0pEI1FIAZ8VXmoeGLdIDYNj+M/TtwPxy3oj6yXYeQ01OipkpnTeHDVNnTkHCya0wcA2DIyHgk3BCoPGtuVeymWOdHSOOeaxwAAS75xCw47+yas8KO5nt4YjaxSLSYTspaDN8HEu7gM9EgDh9nSiMohu6dMkWDbErLoE6Nb5AG3CktDvCO/+sARfp6GZGnECFwzsSyQJjJIHuAndxWULPBwO2ZpCL9+IRf5LSUVLBTv5LDSliz/iKKcxHchLLxe3xqWreKuQg6TOvP4079ewGNrBjBjUmckEU92Rau/T85oabx18RwAnuXT60ckbg84Db1rT/Am4rkIhTdjUgc2+nlUT64bAhGwd/+k1LJUi7ZRGkQ0F8BHABwKYK2Uj/FuAPMBXA9gEMDDAMYAvLNRskTdUx53sEWKrAD0taZGpQEW0Ptgl68dwKt8cnfu9B7svktXMECWXTYOROpgu3V7MTEaZ96MnkBRfOTSezFjUif27g9nIarlomKNT5AL90wShyJkB8IfgOk+0loaaZSG6f7T8CAuc4QMVQvWCQhLY3C0iNFiOfiR3vDwWiz55i3465MbUQ+IQV4gJH3TXT9aLAfPfI+p3SjkKFIafbRYjgQhJOVQyLtN46FcbkNgcmfeKK9qAQi5ugpOxGof9xdMkiEKForve0gJppD7VIn6rSPe9yUi5Z73l1BetEdf5Lzvvm1RsH3Ynn0Bz1iWSHAgPonIEnILALvt0o2/fO6luPHfXhoosErvesn1VhkU7inx3s6c3IlNw2MolV2s2DCE2X3dWoK/3mgbpcHMK5mZmLlLysWYxMyXMfPlzLwXM/cy827M/F5mXtsoWQqRZLsyLrzjaRx29k145IVtKOQIkzvz2jwN8UKJWWtZmXVtHh7Hq39wZ7Bv5uQudBVyGC2WMTxWwpeuegj/8ccHI21e8bGjAMQH+QtueRLXPLQGJuw5rTfygx4vuThy/vTgx2GKOwe8cEMRfRW4p9xkS0N98b957XLteWk5jTRriBzw1etj+5avHcCLzr4JF//tmcR1zl03qjTUjGABoTQPXnojFn75+iAYQCiTh18wG7uDo8VgcO2t8GPu6466C7MmEb78u7fhzCs9TmpSZwF5xyPCRXTcuoExHLL0xlRtyTJUmkXL0VO9nTljwULVAgiVRi4Szrp1pIipPVE3GJF3vpiQqO+QbDWpEwnh+trFb/OIedMAAIfPnRo575UHzAoU5Evmzwjuy3VDXqSr4GjdU1lLd8zvn4S9ZvQGFkMQPWW4p4HRIqZ0FwL3lLhu5pQuMAObhsexdmAUu+/SjWagrTiNdkEhH7U07vRnkw+s2oZpvR0YLbqBpXHmaxbi29ctD84FQv94hKsoubEZ8MzJnegq5DA0VsKBX71BK8t0n+DNGnU0Z2p3JCprrOSiI+9g35mVzde9+3vx6JoBDI2VJPeUm5iclnRMxqA0S3R994kumkhWQuMlFy77oaTSLFQ3nj7ru8OW/vnRSN6JCo8IlzmN0NLozIeDw3jZxfqEhMSkWebBS2/EN99yMN61ZE/0dOYxPF7Gkr2mac+dNqkjUudL3Fta59QaKXR6Ulce+Rxh0/A4nlxfeW5l6oNhHhB1RHjOiZYTT+IaAqWRD91TA9tLKLkcyZ8BPL5ny0j421F/CxFLQ3VP+Yq4z3dPnffWQ/DZVy0IZvmh7IQbP3McBkaL2HN6D3JP+3IyB0pjUmc+tvSyy1w1s9pViLqndO5JANg6UkRfTyF493skSwPwrPoNg2M4YPcp1QmSEW1jabQTZE5jeKwUfF65aRjTej1/pyCKD9p9F9x15ivQkXcwWiqj7HK4wJI08A2PlWLWwswpnZjSnTfO4BfuOjl4sYbGsysNeRY4NFZCZz4XK9Ggw76zJmFwtISDvnpDQNgPjpZwzzPmMiDbxytbBoC3ep2YDX71/x7BgrOu05LestJYcNZ1WPjl6xPX9CiWXbiKAvrJbSuM5x8ypy9ChsqcxnSlkOF9z201tlNplnn9I96g3duRw/z+Xpz95oO0503r7YxMKuS1yLMkEhJ5XE1nPqd10+mSGHXKV/RrdE/5f2WloSbhyVB5CGGFe3ka3vZGX2nK5V0AYMWGYdy1wivtM623A8Pj5cg7I1s34t1yXU9+4Z7q8xVRb2fe6Pffd9ZkHD53mn8vIREurPLeznwseoqrsDQExPuX9NsplV0MjpbQ192BTn8cmucvDz1zihedt35gDOsHRgMl0mhYpaGBzGlc+o+V+PvT3gvrslcy3SEKSxQ7wO593Zg7rQf/+68XsPd/Xotf3PkMAERmJYefc3Ns5tvTkUdftz5U9LenLcHlpx0ZJBlVY2moUSgdeSeiEE046UVzgu0n11cuTJhzKNXqgsct6MfweBlXP/gCAO/ZAsDe/3ltZLAZGC1qyeRH1wwEC1apCviYc/+CI75xMz548bJgn2r9/PL9i/G704/E704/Ej885VC9e4qBqZHyImVsSqj0WymZSpCXI+NlvHjetGASoGJ6b0hqAuFAfvUDazD/P68NfPGV0JXPwXEIrztk12DfGxftHmwHkToV9NBeX7wWT28cNt6fyDeYL/Fk6nrhUbeRKXrKQdF1A/ctECaC6iAGRtmikHkMsf+Tl9+P+f95beCemtKVzaki14kS70ZvRz62hG5WTkNG3D0Vd+0N+L/7vp4Cdt3FUxLz/Agp8Sye3TSM4fEyZk0JQ7wbCas0NFAjjWRM7srDoXh5Y9NgIOPUi+4GAFzywRfjz2ccA8B7GXQ4dI8+TO3tCGr9VCKuVRziR07J6Mw7MZJRh72m9+KKjx1lzinQtJvGPXXcvjPQkXOwwg8PlKNkBrZ793fpP1bikKU34l2/uFvbxppto3jg+a0xd966gTFsqkCAH7NPP5bMn44l86ejpyOvtTSYozWpfnzrisDlpYNa7kO1moTi3j5eTiQpp/V2YP1g3A0mAiAeXBVyJ+Oa9VUExAx+yV7Tg33H7z8z2B4peu2Z/OcqTAPiSS+ajUs++GJ89Q0HBvvUNSjkZ6ES1AFPkM+B2TtXVFSeruTc/OidhwXbIjxWnkSJPJxpvR3YMlyE63LA9z27cRhTuvKp3nsZwiKXI9BE0UbZ7VyLpSHqW4nfTtS1530ILaUCDtuzDwCwm688xLMQOVnW0mghkhJkLv7AiyPuqVBppH+Ue83oxcFzdgFgVhoiFltksWZNitu9rzumaDrz0UqfrzxglvbarkIOh8+dioNmp/ORduYdlFxOJJ4B75527+vCqi0jQT8Cm4bHwMz4ya1PGTOmAeDGR9bhTT9Ov/TsF169MNhW25WT++QfrupTvz/BPTVaLOMLf3wQr73gr2DmmE+d/TIUw+Ml9HaYZ7v9kztRLDN+frvnUlOj88Rs9JEXtmHBWddFAiZ0a1nLPvvF86bhZN96FIO3GNu9BDFg3pnX4NzrlwcyC5gGRCLCcQv6Ucg5uPmzL8U1nzoGfT0dEZfYc/7yAnOn98Sey7qBMUzv7QizoksuNgzp3VNH7R0qwEBpSO+2iAqb3deNx9cNRt6PP/3rBbzmoN2095CEKf6Eadv2YoTTAKJKw6sLVaXSyDsgCr9bnXW9VQoZfu9L5uFnpx4eWI5inBLVHw7xx5RGwyoNDZIGrT2m9UTcU0K/qANNEuQYcZN7SsBxvLITwrX1Hyful3j+1Z88JrBiVPeUpzTCe5vd143dd4mbtGFIXzpzVwz+J/3krsTzZk7uRHdHHlc/uAbzzrwm8sN/xfdux/rBMazZNorXHWz+kV/mu7TS4kPH7GU8lnMIv/nQEvR25PCc7/5xmWMWlroQk4ztxTJ+t+x5PLZmAMeed2ssQun5zdux1xevhcv6ekcCbzvcG9S/dd1yvP5Hd2LdQNQl9rk/PIDrH16L113wVwDAzY+FhR/lWffuffHvrK+7gFcf5LmrwkQyf/bcmQ9CdX/qc0CyW9U0qZGxz8xJOHD3XTBzssfLiEFVuDaP2nsGto4UcdFfn8Hic27CPv95LS6/5znMmdoduLme2zSC+5/bgmm9HdhVcbP0Sb8toTTkAVb8Nr78+gPwniPn4qHV0Yi2k140u+I9qBDWzubh8ZjSGFMqX1frniIidBdygfyywhV6W1gau3QXkHMIrz5oV+P6Hc3I0QCs0tBCHVgBz5Vy67+/DIAfUx8s8O59gersKAnyLFD3o1T9r115J1gS9vWHhAPqx1+2d+zag2bvElgxh8+dik8dv29wrDOfi9TfcZlx1SeOxg9POVQr58wpUXO3U1Gm4t0VSkP+sc6aEjeV50zrTnT9ifIKL9vPXDMsKVRYh468g1s+91Jc+qEXa48fs+8MHDl/eqDIto4UY5ZmUkLij28NyXa5SKTA4+tChZMUdjt9UmcwU3x4tRfWu8e0aAjlN699LNiWgxzk/Itj9ok/u56OXBBxMzhaxNHf/kuwquTU3g48+kK0Es/gWNhe2omDfK54V59YN4jejhyOmOeFt5599aPYODQefIezp3Zj31neQLd87QDuemoTluw1LTZzlyc/+82aDAA47ZJl+NJVXoixcE8dtmcfvvbGA3H2mw7EO1+8R3CNmpORBtMCpTEWEPW9gdIIXZkPrR5IFZFowoJZk4PfzaaheCDEhkHvWfZXcD1dftqRTSlWCFiloYU8sH3g6HkAPNNPpOg7FBYcFD9edZ0BFbKfXLZkVKXxw1MOxbWfPjayr7OQw1jJxV4zejFnaliG4Jh9tIV+AxARPi0pjZfsPR19PR04dl/vuo1DY5g1pQsnHhiSprKbbZYyYLz50OiMTZSPnqFRmMfuGx+8Zvd14wfvONRoLX3q8vsBRP3xKtKG9l7/mWPx29O8QnR790/SyiOwz6zoj/7UI/fExR84An/53Etj575sv3586hX7xPZ/9pULYvvU53LsguQCmnLE2CsPmIV3vXhu5Ljs/hPPYfXW7Tj2vFsBAC9d0I+vvuGAWLtEFCiNFRuGIlnyfT0dWO5bUuK9lC0XnfI3QZwrLJd/PrsFi/boi7yzMuZM7cFeM7x8os//8UGsHRjFWw5LtgpOPHBXdOQcbB0p4k/3r0ap7GJg1CsLUsh5NZne85J5+OZbDsbSNxyAn7z7Ran4RhXTfTJ+0/B48DzEQmTC0li+dhAbh8awZL75fa2EI+dPx4OrtmK0WI6EXK/2JyDrB5KVxtWfPAaXn3YkXrJ39TJkhc3T0ECeab5sv36cc81jEXNYTvBxAqXhXXPU3tOD8EDAG4RHiy6O3mcG/vzAC7G+ejry+P47FmHPaT1YtWU73nRo/EezcNfJ2DA4hk+8fB/kHMKhe/ThX89vTZX9mXMI//uJo7F66/agMNvHX7YP7nxyYzCLkSOqbvxMOFC+ftFu2Dg0hn89vxV3rdiEuTN6cOmHXozv3vA4Hli1DR956d7IO4TJXXn889ktwXV79/fi6286EH+8d1VElsldBUzuKuATL98Hq7aM4PJ7nscpR+yB//nn88E5B+4+Bbvu0oUfnnIofnjzk0EZkrPffBC+/KeHg/OO3XdGkD/z2oN3xcJdp+D8m56Qnln6mPVTl8zF4GgJv737OUzuzGN+/yTM9039b510ML54ZVjI8TUH7YpXHbArLvjLU5E29tt1cuTzv79qAd531Dwc7LurvvDqhRXdBzMmdQYFBw+ZvUvMRy2+L8CLxpp35jWR40vfeGBkgLzhM8cF/JHgb0T7h+3Zh1OXzMX/Se+kcL/IbsO0wRBAaGmsGxjFtu1FLF87gM8cvwBzpoYW0+sO3i0gqY9fOBOd+RzOPfkQPL52AP2TO40826UfejG6Cjn0duYxv78Xy9cOYni8jH2+dB2AuHIjIrz/aLNrshLEb+u86x/Hwl0nY8akThw02/s+Tjg/LMDZXcjhVQaZ0+Dg2bugWGYcvPQGzJnag4W7TsbytYO44r7V+PrVj2LfWZPR11MwTkqFTM2EVRoayEpjn5mT8f13LMKCWeGg8OXXH4CPX3YfgDDET1gSx+7bj2P2nYEZkzrx+T8+iGP37ceSvabh7UfsgQ8fs1fgO5fxlsM8f/bhc2OHAADfe9siXHHfarz5UI8Au/gDR+Dye57XRkjpsGiPvoiJLkL3hDnrOITz374Ii+dOw55SQbWZk7vw+VcvxBevfAh3rdiEfWdOxrH79uPah9bggVXb0Jl3cOqRc3HZ3VGe4cPHzo+QzNd86hg8vznqujlh/1m4/J7nMTJexqeP3xc/vMVzl3z8Zd4s/k2HzsbMyV145y/+AQB4z5FzI0rjF+9djF/c8TQch3D6cfPhMmO0WE7MzTBhj2k9+OZbDsaL9pwaG6hPOWKPiNKYMakTU3s78Mv3L46E97543jT84aMvCdYbOeMV+0baERZrEr7/jkOx5Ju3APByZWS3R0feSVwJEogWwwM8RSaUmbA0RJjz999+KObN6MUdT4ZrzmweHsdR37oFL8iJgp3ph4i9Z3qW+Ed/cx/OfM1CMHsW+m4Sb3b6cfPxuVctwF0rNgUz9LcePkfbngzZUpwztTuwjgQaWd11+dpBnPW6/bXh6v9x4n6YPqn6qCXhniuWGc9sHMabD90dq7duxxX3rQr6XjCrOVxFWliloYFKIItBXeC1B++GB5e+Cr/+27NY7JcjeN9R8zBWcvGBo+ehq5DD9Q97s6npvR348LHzAcQH77SYOaULH5P4i76ejuDzRe9bDIcIH7j4n6nbmze9B199wwERt5Scm6HiP07cD3tO68HxC73QTTHrESGravKZWj33wN13wYG7RwfjVyycibNetz9O2H8WduvrCpSGPCsVIYYCP37Xi/C3FRvxhkN2R1chh08eHx2YP//qhXjpgv4gtj0rdIMXEeG/3nUYlj27BRff9WzgknvFwnB2eeF7DsfU3g4c0TsNv//ISyJuBoE0LpJZU7oCC2rO1J6IS6J/UmfgVnrNQbviuoejmd4XvufwRJ+2mgG9m0+YT1b4M5kzesOi3YOyG2kgTxRElYR9Zk4CEeH/zjga/3p+a/D+z6+BtN1v18m4+bFwHZgPHbNX7F2pB354yqH49P/8CwDw7iVzwWD82wkL8P2bQ4s2zWQgCSJRT+CYffuxaI8+fO3Pjwb7svBKzYBVGlViSlchMmh1FXIR0vmE/WfhU6/YBx/yFUajcPz+3uD1y/cvTl3cjojwgQym+7TejojS+tyrFmBKVx5v9C2fty3eA+sGxtDdkcN3bng88Lf/4aMviZS3UGX4sObZzJaURlchh/NOPiRIIHvdIbvhdYckh0/W4l824fWH7I5j9+nHlO4CDpRKNVz0vsUgiiqQFytlQq742FGR9dsr4dyTD8GV963CAbtNARHh++9YhDlTe7B87WBgaX33bYuCRK4XzZ2KjpyDV0kTAB2m9hTwbycswPK1A1i0R1+g+LcMh8lvA6Ml/Pur9sMXr3oIZZfx769aYIzUMeGyDy/Bb+95LlinRQSSHDKnL7VlXAkff9k+YAYO2H0KHCK8NiHarha86dDZcJnR1xOGBn/6hH2x/26Tcctj6/Gy/fprJp878g7OefNB2KW7gOVrB/CmQ3dHIeegtyOPz1/hhVUftU/z+Io0oGrq9k8ULF68mJctW1b5RA1+fdezfq5C832GExWjxTLOv+kJnPGKfbRrLyThsTUDuO7htfi3E/ZtWhTIRMPv/vkc+id3RpRUrXhy3SD+/MALeNNhs3Hlfavw2Vfuh9VbtuOye1biCycuzKw0BO54YgNWbhrGe14yr26y7my47O6VePD5bfjKGw6IWYqNBhHdy8yLtces0rCwsLCwkJGkNGzIrYWFhYVFalilYWFhYWGRGhNGaRDRNCK6ioiGiWglEb2r1TJZWFhY7GyYSNFTPwYwDmAWvCVhryGiB5j5kZZKZWFhYbETYUJYGkTUC+BkAF9m5iFm/iuA/wPwntZKZmFhYbFzYUIoDQALAJSZ+Qlp3wMADlRPJKLTiWgZES3bsGGDetjCwsLCogZMFKUxCcA2Zd82AJPVE5n5QmZezMyL+/uTC8RZWFhYWGTDRFEaQwDUCnRTAJgXOrCwsLCwqDsmChH+BIA8Ee3LzE/6+xYBSCTB77333o1ElG3VnhAzAGys8tp2g72X9oS9l/bDjnIfQG33YiifOoEywonof+AtbfxheNFT1wI4qlHRU0S0zJQROdFg76U9Ye+l/bCj3AfQuHuZKO4pAPg4gG4A6wFcDuBjNtzWwsLCormYKO4pMPNmAG9utRwWFhYWOzMmkqXRbFzYagHqCHsv7Ql7L+2HHeU+gAbdy4ThNCwsLCwsWg9raVhYWFhYpIZVGhYWFhYWqWGVhoWFhYVFaliloWAilWAnojP8OltjRHSxcux4IlpORCNEdCsRzZWOERGdS0Sb/H/nUQvXWCWiTiK6yH/eg0R0PxG9Rjo+Ye7Fl+k3RLSGiAaI6Aki+rB0bELdiwAR7UtEo0T0G2nfhLoXIrrNv4ch/9/j0rEJdS++XKcQ0WP+WLWCiI719zf2XpjZ/pP+wcsB+R28elfHwKtxdWCr5TLIehK8MOSfArhY2j/Dl/ttALoAfAfAP6TjHwHwOIA5AGYDeBTAR1t4H70AlgKYB28i83p4JWLmTbR78WU6EECnv70QwFoAh0/Ee5FkuxHAnQB+MxHfMV+m2wB8WLN/It7LKwGsBHCk/5uZ7f9r+L20/GVsp3/+4DUOYIG071IA3261bBXkPkdRGqcDuEu5r+0AFvqf7wJwunT8Q/KL1Q7/ADwIrxz+hL4XAPsBWAPg7RP1XgCcAuD38BS7UBoT7l4SlMZEvJe7AHyoFfdi3VNRpC7B3uY4EJ7cAABmHgawAuF9RI6jze6RiGbB+y4ewQS9FyL6CRGNAFgOT2lciwl4L0Q0BcDXAXxOOTTh7sXHt4hoIxH9jYhe5u+bUPdCRDkAiwH0E9FTRLSKiP6LiLrRhHuxSiOK1CXY2xyV7kM9vg3ApDbx0xYAXAbg18y8HBP0Xpj54/BkPBbAlQDGMDHv5WwAFzHz88r+iXgvXwAwH55b5kIAfyaivTHx7mUWgAKAt8J7vw4FcBiAs9CEe7FKI4odpQR7pftQj08BMMS+vdoqEJEDzx04DuAMf/eEvBcAYOYye6tMzgHwMUyweyGiQwGcAOD7msMT6l4AgJnvZuZBZh5j5l8D+BuA12Li3ct2/++PmHkNM28EcD6adC9WaUQRlGCX9lUswd6GeASe3ACC5XL3RngfkeNog3v0ZzoXwZtFnczMRf/QhLsXDfIIZZ5I9/IyeMEIzxHRWgD/DuBkIroPE+9edGAAhAl2L8y8BcAqePKraPy9tJLMacd/AP4HXgRVL4Cj0d7RU3l4ERLfgjdD7/L39ftyn+zvOxfRCIqPAngMnpm+u//StDoa5GcA/gFgkrJ/Qt0LgJnwiONJAHIATgQwDOBNE/BeegDsKv37LoA/+vcx0e6lz/8uxG/k3f73st9Euxdfpq8D+Kf/vk2FF9l2djPupWU33a7/AEwD8Cf/hXoOwLtaLVOCrEvhzTbkf0v9YyfAI2G3w4samSddRwDOA7DZ/3ce/DpkLbqPub7so/DMZ/Hv3RPwXvoB3A5gK4ABAA8BOE06PmHuxfC+/WYi3ov/vfwTnptmK7wJyisn4r34MhUA/MS/l7UALgDQ1Yx7sQULLSwsLCxSw3IaFhYWFhapYZWGhYWFhUVqWKVhYWFhYZEaVmlYWFhYWKSGVRoWFhYWFqlhlYaFhYWFRWpYpWFhUWcQERPRWxvY/mK/j3mN6sPCwgSrNCwsJBDRxf6ArP77R4ZmdgPw50bJaGHRSuRbLYCFRRviZgDvUfaNp72YmdfWVxwLi/aBtTQsLOIYY+a1yr/NQOB6OoOIrvGX01xJRKfKF6vuKSL6in/eGBGtJaJLpGOdRPQDIlrnL0X6DyI6Rmnv1f7ynaNEdCe8tUagnHMUEd3uy7SaiH7qr4VhYVFXWKVhYZEdXwPwf/DWMbgQwCVEtFh3IhGdDK867McB7AtvKdt7pFPOA/AOAB+EtybCQwCuJ6Ld/Ov3gFcL7Sa/vx/518h9HAxvOdb/g1e19CT/3F/WdpsWFnHY2lMWFhKI6GIAp8Irnijjx8z8BSJiAP/NzKdJ19wMYC0zn+p/ZgBvY+Y/EtFn4a3LfBCH5d7Fdb0AtsBbgvQSf18OXon+y5n5LCL6JrzFdvZjUXGO6Cx4FU33YuZnfculyMwfkto+FMD9AGYx8/q6PBwLC1hOw8JChzvgrbUsY6u0/Xfl2N8BvM7Q1h8AfBrAM0R0A4DrAfwfM4/BW+egAG8xIADewk1E9HcAB/i79odX2lqe3an9Hw5gHyJ6h7RPrMS2NwCrNCzqBqs0LCziGGHmp+rREDM/T0T7ATgeXsnq7wH4KhEtQTiw68x9sS/NMpwOgP+GfoW91dkktrBIhuU0LCyy40jN58dMJzPzKDNfw8z/BuAIAAfCW+DrKXhRWQHx7bunXgLgUX/XowCWKGs4q/3fB2+hsKc0/7bDwqKOsJaGhUUcnUS0q7KvzMwb/O2TiOif8Ba4eSs8K2KJriEiej+839nd8BaWegeAIoAnmXmYiH4K4NtEtBHAMwD+Dd6Stz/xm/gZgM8B+AER/QTAwfBWX5NxLoB/ENHPAPwc3kJDCwG8gZk/kv32LSzMsErDwiKOEwCsUfatBjDH314KbznNCwBsAPABZv6noa2tAL4Ab6nUAjzL4SRmfsY//gX/76/gLUl6P4BXM/MaAGDm54joJADnwyPU7wVwJoDfiA6Y+UEiOg7AOfBWDcwBeBrAVdlu28KiMmz0lIVFBsiRUa2WxcKiFbCchoWFhYVFalilYWFhYWGRGtY9ZWFhYWGRGtbSsLCwsLBIDas0LCwsLCxSwyoNCwsLC4vUsErDwsLCwiI1rNKwsLCwsEiN/wcSr9d7zWy/1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(rewards)\n",
    "plt.xlabel(\"Episode\")\n",
    "plt.ylabel(\"Sum of rewards\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "state = env.reset()\n",
    "\n",
    "frames = []\n",
    "\n",
    "for step in range(200):\n",
    "    action = epsilon_greedy_policy(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    if done:\n",
    "        break\n",
    "    img = env.render(mode=\"rgb_array\")\n",
    "    frames.append(img)\n",
    "    \n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like a pretty robust agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using TF-Agents to Beat Breakout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use TF-Agents to create an agent that will learn to play Breakout. We will use the Deep Q-Learning algorithm, so you can easily compare the components with the previous implementation, but TF-Agents implements many other (and more sophisticated) algorithms!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-Agents Environments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.wrappers.TimeLimit at 0x7fe46bf261d0>"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tf_agents.environments import suite_gym\n",
    "\n",
    "env = suite_gym.load(\"Breakout-v4\")\n",
    "env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gym.envs.atari.atari_env.AtariEnv at 0x7fe46bdbba50>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(0, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]]], dtype=uint8))"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.seed(42)\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]]], dtype=uint8))"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.step(1) # Fire"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure breakout_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAIpCAYAAAD3tqwgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAASjElEQVR4nO3dW29c533v8f+aGZ5EKhEpKrLlpLHRGE2yN2IlKbCDHApHCFDkRfZl9K5RbV0IAWoDzcbeiHNAvRNBkWmLlCNKlKg5PPvCjRK3NknHwzU/Sp8P4JvRwjyPxuL6ch3mWV1rrQAgzWDREwCATyJQAEQSKAAiCRQAkQQKgEgCBUCk0VF/2HWde9ABOFWtte6TXncEBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARDpyLX46E/XdXXp0qXa3t6urvvEZalOzWQyqZ2dnfrwww+P3XZ7e7suXbpUg0G/v9vMZrPa2dmpvb29Y7fd3Nysy5cv13A47GFmf9Zaqw8++KDu3r1brZ39ZSw3NjbqxRdfrOXl5d7HvnfvXu3s7NR0Ou197EUZDod1+fLl2tzcPHbb3d3dev/992s2m/Uws8URqBDD4bC++c1v1ve+973ed6wPHjyo69evHxuowWBQX/va1+pHP/pR7zutx48f140bN+revXvH7vxffvnl+vGPf1xra2s9ze4j4/G4bt68WXt7e8/EjvXy5cv1k5/8pLa2tnodt7VWv/jFL+r69ev16NGjXsdepOXl5Xrttdfq29/+9pG/pM5ms3rrrbfqzTffrCdPnvQ4w/4JVIiu6+rcuXO1tbVVo1G//1uWl5drZWXlRNuura3V1tbWibefl0ePHtXq6uqJtl1dXa3Nzc1aX18/5Vl93Hg8rnPnzvU65mlaWlqqCxcu1MWLF3sdt7VWGxsbvR+lL1rXdbW+vl4XL148NlDr6+u9n2lZhOfrXwAAZ4ZAARDJKb4zprVWDx8+rL29vZpMJkduu7KyUhcvXjzxqbF5Ojg4qN3d3RqPx0dut7y8XBcvXuz9elHVR6cN9/b26vDw8MjtRqNRXbx4sfdThmfFeDyu3d3dOjg4mNt73r1795m4jsfnI1Bn0K1bt+rNN9+shw8fHrndCy+8UNeuXasXX3yxp5n92Z07d+qNN9449saLra2tunbtWn31q1/tZ2J/YXd3t65fv14ffPDBkdudP3++Xn/99Xr11Vefi/P+n9X+/n7dvHmz3n333bm958OHD5/5GwA4nkCdQQcHB3Xnzp3a398/crvBYLCwH/LHjx/Xe++9V7u7u0duNx6P6/Hjxz3N6uMODw9rZ2en7ty5c+R2Fy5ceK7uJvusJpNJ3b17t27fvr3oqfCMcQ0KgEgCBUAkp/iAz2U0Gj1dvWNeHj16VPv7+8/Eihz89QQK+Fw2Njbq+9//fl29enUu79daq3feead+/vOfL+z6JBkECvhclpeX66WXXprb+7XW6t69e70v+UUe16AAiCRQAERyig/4RJPJpO7fvz+3xYu7rqvV1dVaWVnxhWdORKCAT7Szs1M/+9nP5rZy/Wg0qqtXr9Y3vvENgeJEBAr4RPv7+/XOO+/M7f1WVlbqpZdeqq9//etze0+eba5BARBJoACI5BTfGTQajWptbe3Yx22srq4u7Kmkw+GwVldXj32Mxurq6sK+7zIYDE40x7W1tefyOzmDwaCWl5fndr1oZWWllpaWXH/ixATqDLpy5Updu3bt2GctbWxs1IULF/qZ1H/xpS99qV5//fVjVwJYW1ur7e3tnmb1cVtbW/XDH/7w2OcYrays1AsvvPDc7VgvXbpUr732Wp0/f34u7zccDusrX/nKc/c58tcTqDOm67ra3t6u7e3tY9cpW+SOYHNzszY3N6Pn+IUvfKGuXr0aPcdF2tzcrO985ztz+wXief0c+esJ1Bn0px/05B94c3w2dF23sNPE4F8eAJEcQQWZzWY1mUx6f8TAZxnzT3Ps+7fqyWRSs9nsRNv+aY7HXaObt/F4XNPptNcxT1NrbSGfY1U9U5/jZzGbzWo8Hh95VN9aO/HPwlknUCGm02n99re/XcjO//Dw8NjHnld99IPxu9/9rt54443e72obj8d169atE4X09u3bdePGjVpaWuphZn82nU7r3XfffWZ2Hru7u3Xz5s1aX1/vddzWWv3hD3+oJ0+e9Druoo3H4/rlL39Z+/v7xwbq97///bF38T4LuqN+4Luu87SwHg0Gg4VdD5nNZifa+S/ymsSzNMezYJGf4/N0lPCXTroPeNY+n9baJ/6lBQqAhfq0QLlJAoBIR16DcvstAItyZKB+8IMf9DUPAPiYIwP105/+tK95AMDHHBmo4xbRBIDT4iYJACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEijvgZqrdXBwUE9fPiwWmt9DQvAnHRdV+vr63Xu3Lnquu7Ux+stULPZrH7961/X22+/XdPptK9hAZiT0WhU3/3ud+tb3/rWsxWo1lrt7u7Wb37zmxqPx30NC8CcLC8v1yuvvNLbWTDXoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEijvgbqqmp1OKzN5eUad11fwwIwJ8vLy7U6HPY2Xm+BGnRd/f3WVv3PV1+tNp32NSwAc9KNRrW6tVWDng4yeg3U354/X1++cqUGAgVw5sxGo7q1sVF3qqr1MF5vgfpLnVN8ABzDTRIARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkRayFl/rZZlBAM6yfgO1Mqs6P66aWc0c4Kxpw1a1POttvP4C1VW1tUm1i4fVmkABnDWtm1StTT56Am0P+j2C6v6zwM0pPoAzp2vVerxzwU0SAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIPS4W26p1raaDWbXW33LtAMzHrBtU6/p7ol9vgWpVNR5N69HquLo26WtYAOZk1s1qPOrvcUm9Pm5jNmg1Hk5rUI6gAM6aWc1q1vX3uCTXoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkXpdLLZ1VdVV9bdYOwDz8qd9eF/6C1RXNVmb1aO1aXXV33LtAMxHq0FNejzA6PUIarLa6vG5aXUetwFw5rSa1uRgVvW4n/F6DVR1f/EfAGdLq173326SACCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABE6m2x2FZVD2pU99tqlcdtAJw9bVTDGvW25nePgerqvbZat9tmzQQK4MwZtGF9ua3WC/WMBaqq6rAGdb9GNXNmEeDMGdawDnvcfysFAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiBSr4vF1mS56vF6tW7W67AAfH6tDaomS72N11+g2qBm779c071XazrrY6F2AOZq0Gq2Na7anPTyvI0eA9VV+/ByTf/fl2s2c2YR4KzphrNqw1tVF+5Ude3Ux+v3FN9TjqAAOJpDGQAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIC1mLr7XTX2QQgDnred/dW6Bam9WD/d/We3/43zWZ9DUqAPOyNKq68qVzVbVafSz63WugPtx7u/7jNzdq/EShAM6a5ZWleuVv/qFa+15VDU99vB5P8bWaTh/X+Mkfazwe9zcsAHPRdcs1mx32Np6bJACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRRoueAKQbVNX6aFQrw+HT11pr9XA6rcfT6eImBs84gYJjnF9aqn+8cqX+xxe/+PS1R9Np/cudO/XW3t4CZwbPNoGCY6wOh3V1c7N+fPny09f2J5P61f379fbeXrUFzg2eZQIFn0HXdYueAjw33CQBQCSBAiCSQAEQSaAAiCRQAEQSKAAiuc0cjtGqatpajVurah9962k8m9Ws+QYUnCaBgmMcTCZ1Y2enbj18+PS1w9msfr2/70u6cIoECo7xYDKpf93ZqRvvv//0tVZVE0dQcKoECk5g0pogQc/cJAFAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQKTRUX+4P5jObaDpcFaHXas2t3cE5m3QdXVxebk2lpYWPZVPNWut7j15UvfH40VP5fnTWi0dHta5Bw9qMBye+nBHBur/rD2Z20Cz6aR2liYCBcHWh8P6xytX6n9tb1e36Ml8ikfTaf3z7dv15s6O/UnPBq3V1gcf1Mu/+lUNB6d/Au7IQO0uze8IatZN62DQqnX+SUGq0WBQr2xs1N9vbVXXZSbqwXhc/7a7W12VQPWsa61WDw7qi3t7NeohUK5BARBJoACIJFAARBIoACIJFACRBAqASAIFQKQjvwcFPF9aa/VoOq3743Hs96AeTib1ZDZb9DTogUABTz2aTuuN996r/9jfX/RUPtWktfq/H37oS7rPAYECnjqczeqtvb16e29v0VM5kjg9HwQK+G8EgARukgAgkiMoAE6kVdW0tXoym9W0nf5xtkABcCKT2aze3turx9NpDeZ4l+c/fMrrAgXAiUxaq7f39urf53wTzT99yutHP1H31ntzm0CbTuvJH/erfH0B4MyatdbbbrxrR5xH3Py7l+d2krG1Vof37tfj3T9W9XDuEoCzobX2iecLjwxU13n8LQCn69MC5TZzACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkQQKgEgCBUAkgQIgkkABEEmgAIgkUABEEigAIgkUAJEECoBIAgVAJIECIJJAARBJoACIJFAARBIoACIJFACRBAqASAIFQCSBAiCSQAEQSaAAiCRQAEQSKAAiCRQAkbrW2qLnAAD/jSMoACIJFACRBAqASAIFQCSBAiCSQAEQ6f8DiyfmfwULrTYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "img = env.render(mode=\"rgb_array\")\n",
    "\n",
    "plt.figure(figsize=(6, 8))\n",
    "plt.imshow(img)\n",
    "plt.axis(\"off\")\n",
    "save_fig(\"breakout_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]],\n",
       "\n",
       "       [[0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        ...,\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0],\n",
       "        [0, 0, 0]]], dtype=uint8))"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.current_time_step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Specifications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BoundedArraySpec(shape=(210, 160, 3), dtype=dtype('uint8'), name='observation', minimum=0, maximum=255)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_spec()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BoundedArraySpec(shape=(), dtype=dtype('int64'), name='action', minimum=0, maximum=3)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.action_spec()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=ArraySpec(shape=(), dtype=dtype('int32'), name='step_type'), reward=ArraySpec(shape=(), dtype=dtype('float32'), name='reward'), discount=BoundedArraySpec(shape=(), dtype=dtype('float32'), name='discount', minimum=0.0, maximum=1.0), observation=BoundedArraySpec(shape=(210, 160, 3), dtype=dtype('uint8'), name='observation', minimum=0, maximum=255))"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.time_step_spec()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Wrappers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can wrap a TF-Agents environments in a TF-Agents wrapper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.wrappers.ActionRepeat at 0x7fe46872cad0>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tf_agents.environments.wrappers import ActionRepeat\n",
    "\n",
    "repeating_env = ActionRepeat(env, times=4)\n",
    "repeating_env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gym.envs.atari.atari_env.AtariEnv at 0x7fe46bdbba50>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repeating_env.unwrapped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the list of available wrappers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ActionClipWrapper           Wraps an environment and clips actions to spec before applying.\n",
      "ActionDiscretizeWrapper     Wraps an environment with continuous actions and discretizes them.\n",
      "ActionOffsetWrapper         Offsets actions to be zero-based.\n",
      "ActionRepeat                Repeates actions over n-steps while acummulating the received reward.\n",
      "FlattenObservationsWrapper  Wraps an environment and flattens nested multi-dimensional observations.\n",
      "GoalReplayEnvWrapper        Adds a goal to the observation, used for HER (Hindsight Experience Replay).\n",
      "HistoryWrapper              Adds observation and action history to the environment's observations.\n",
      "ObservationFilterWrapper    Filters observations based on an array of indexes.\n",
      "OneHotActionWrapper         Converts discrete action to one_hot format.\n",
      "PerformanceProfiler         End episodes after specified number of steps.\n",
      "PyEnvironmentBaseWrapper    PyEnvironment wrapper forwards calls to the given environment.\n",
      "RunStats                    Wrapper that accumulates run statistics as the environment iterates.\n",
      "TimeLimit                   End episodes after specified number of steps.\n"
     ]
    }
   ],
   "source": [
    "import tf_agents.environments.wrappers\n",
    "\n",
    "for name in dir(tf_agents.environments.wrappers):\n",
    "    obj = getattr(tf_agents.environments.wrappers, name)\n",
    "    if hasattr(obj, \"__base__\") and issubclass(obj, tf_agents.environments.wrappers.PyEnvironmentBaseWrapper):\n",
    "        print(\"{:27s} {}\".format(name, obj.__doc__.split(\"\\n\")[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `suite_gym.load()` function can create an env and wrap it for you, both with TF-Agents environment wrappers and Gym environment wrappers (the latter are applied first)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "from gym.wrappers import TimeLimit\n",
    "\n",
    "limited_repeating_env = suite_gym.load(\n",
    "    \"Breakout-v4\",\n",
    "    gym_env_wrappers=[partial(TimeLimit, max_episode_steps=10000)],\n",
    "    env_wrappers=[partial(ActionRepeat, times=4)],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.wrappers.ActionRepeat at 0x7fe4686ff550>"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "limited_repeating_env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gym.envs.atari.atari_env.AtariEnv at 0x7fe3de8b6c90>"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "limited_repeating_env.unwrapped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an Atari Breakout environment, and wrap it to apply the default Atari preprocessing steps:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: Breakout requires the player to press the FIRE button at the start of the game and after each life lost. The agent may take a very long time learning this because at first it seems that pressing FIRE just means losing faster. To speed up training considerably, we create and use a subclass of the `AtariPreprocessing` wrapper class called `AtariPreprocessingWithAutoFire` which presses FIRE (i.e., plays action 1) automatically at the start of the game and after each life lost. This is different from the book which uses the regular `AtariPreprocessing` wrapper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.environments import suite_atari\n",
    "from tf_agents.environments.atari_preprocessing import AtariPreprocessing\n",
    "from tf_agents.environments.atari_wrappers import FrameStack4\n",
    "\n",
    "max_episode_steps = 27000 # <=> 108k ALE frames since 1 step = 4 frames\n",
    "environment_name = \"BreakoutNoFrameskip-v4\"\n",
    "\n",
    "class AtariPreprocessingWithAutoFire(AtariPreprocessing):\n",
    "    def reset(self, **kwargs):\n",
    "        obs = super().reset(**kwargs)\n",
    "        super().step(1) # FIRE to start\n",
    "        return obs\n",
    "    def step(self, action):\n",
    "        lives_before_action = self.ale.lives()\n",
    "        obs, rewards, done, info = super().step(action)\n",
    "        if self.ale.lives() < lives_before_action and not done:\n",
    "            super().step(1) # FIRE to start after life lost\n",
    "        return obs, rewards, done, info\n",
    "\n",
    "env = suite_atari.load(\n",
    "    environment_name,\n",
    "    max_episode_steps=max_episode_steps,\n",
    "    gym_env_wrappers=[AtariPreprocessingWithAutoFire, FrameStack4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf_agents.environments.atari_wrappers.AtariTimeLimit at 0x7fe46bf46510>"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play a few steps just to see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.seed(42)\n",
    "env.reset()\n",
    "for _ in range(4):\n",
    "    time_step = env.step(3) # LEFT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_observation(obs):\n",
    "    # Since there are only 3 color channels, you cannot display 4 frames\n",
    "    # with one primary color per frame. So this code computes the delta between\n",
    "    # the current frame and the mean of the other frames, and it adds this delta\n",
    "    # to the red and blue channels to get a pink color for the current frame.\n",
    "    obs = obs.astype(np.float32)\n",
    "    img = obs[..., :3]\n",
    "    current_frame_delta = np.maximum(obs[..., 3] - obs[..., :3].mean(axis=-1), 0.)\n",
    "    img[..., 0] += current_frame_delta\n",
    "    img[..., 2] += current_frame_delta\n",
    "    img = np.clip(img / 150, 0, 1)\n",
    "    plt.imshow(img)\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure preprocessed_breakout_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAGoCAYAAADiuSpNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADJ9JREFUeJzt3U+InOUBx/F3Jlmwi+AW/6QExMtiKggGeuplCaUiueToXQQFoaBIiUJ7KgQ8CEK9tFBE8KCeRZHmENJCT5LsQU08KVpipVFZZDAsO28vPWTmeZcdJ+87729mPp/b+/Ds+z7ZnZ1vXt533h3UdV0BQJJh3wsAgGniBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHGO93HQwWDgk78AVHVdD5rGnTkBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECIE4vH8KlPSdOnCjGPvvssyO/7sqVK0fOefTRR4uxjY2Nie2dnZ1izu7ubjH29ttvT2yfPXu2mPPFF18UY99+++3EdtO/9+TJkxPbb7zxRjHn+eefL8aeeOKJie133323mDMajYqx69evT2xPf0+qqvl7N+2ee+45ck6il156qRg7f/78xPb0z62qqurLL7+c63jT3++qqqpnn312rn0tg9dee60Ye+qppya2L1y4UMx55ZVXOltTH5w5ARBHnACII04AxBEnAOK4IWJNnTlz5sg5n376aTE2ffNBm1599dVi7M0335zYbroY//LLL3e2pqaL8dPfu3lvSlll165dK8aabjiZxc2bN+90OSwhZ04AxBEnAOKIEwBxXHMCfpIPPvigGLtx48Zc+5r+sPIzzzxTzGn6wPh777031/FYHs6cAIgjTgDEEScA4ogTAHHcELGmLl26dOSc+++/v/uF3ObFF18sxqafxtz0gdcunTp1qhib/t41PZV8lZ0+fboYm/45zWpZn8xO95w5ARBHnACII04AxBEnAOIM6rpe+EFPnz69+IMCEOfq1auDpnFnTgDEEScA4ogTAHF6uea0t7e30IOOx+OJ7eGwbPL0nMPmtXH8Wffd5Zq61Pe6+z7+Oprld2yVzfqaa+u9qK05CWva2tpyzQmA5SBOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQJzjfS/gTly/fr0YG41GPawEYHVtbm4WY6dOner0mM6cAIgjTgDEEScA4ogTAHEGdV0v/KB7e3utHHRnZ6cY293dbWPXAPzfY489Voxdvnx5Yns8HhdzhsPhkXO2trYGTcd05gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBECc430v4E489NBDxdhoNCrGxuPxxPZwWDZ5es5h8+Yx7767XFOX+l5338dfR7P8jq2yWV9zbb0XtTVn1jU9+OCDxVjX1usVBMBSECcA4ogTAHGW+prT+fPnizHXnPrX97r7Pv46cs1pta85bW5uFmNdW69XEABLQZwAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgzlJ/CPe+++4rxvb393tYCcDq2tjYWPgxnTkBEEecAIgjTgDEEScA4iz1DRFNF+m6fBLwvDyV3FPJV52nkq/2U8mPHTtWjHVtvV5BACwFcQIgjjgBEEecAIiz1DdENF24Ozg4mGnePHPmNe++l/Wict/r7vv462jdv+ez/vvbei9q8z2t7/fHQ4+58CMCwBHECYA44gRAnJW75tT0YbG6rie2B4PBkXMOmzePeffd5Zq61Pe6+z7+Oprld2yVzfqaa+u9qK05s67JNScAqMQJgEDiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHGW+gkRDzzwQDHW9IQIf6Z9sfped9/HX0f+TPtq/5n2pr/2cOvWrWKsTev1CgJgKYgTAHHECYA44gRAHHECII44ARBHnACII04AxFnqD+HevHmzGGv6sBgA82t6uMHdd9/d6TGdOQEQR5wAiCNOAMRZ6mtOe3t7xdj+/n4xVtf1xPZgMDhyzmHz5jHvvrtcU5f6Xnffx19Hs/yOrbJZX3NtvRe1NWfWNW1sbBRjrjkBsHbECYA44gRAHHECIM5S3xAxGo2KsR9//LGHlQCsrrvuumvhx3TmBEAccQIgjjgBEEecAIiz1DdEfPLJJ8VY05PKx+PxxPZwWDZ5es5h8+Yx7767XFOX+l5338dfR7P8jq2yWV9zbb0XtTVn1jXde++9xdj29nYx1qb1egUBsBTECYA44gRAHHECII44ARBHnACII04AxBEnAOIs9Ydw33rrrWLs448/Lsb8mfbF6nvdfR9/Hfkz7av9Z9ofeeSRYuzcuXPFWJucOQEQR5wAiCNOAMQRJwDiLPUNEV9//XUx9tVXX/WwEoDV1fRU8q45cwIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACIc7zvBUCy3zaM/Xpq+0+LWAisGWdOAMQRJwDiiBMAcVxzYn1tTm3/vJzym3+XY8c6WQxwO2dOAMQRJwDiiBMAccQJgDhuiGB9/WFqe6Oc8uffl2N7nSwGuJ0zJwDiiBMAccQJgDiuObGCdhrGflYO/fDh5PZH5ZQbrawH+KmcOQEQR5wAiCNOAMQRJwDiuCGCFfS7hrGGux0ufFiOARGcOQEQR5wAiCNOAMQRJwDiuCGCaINqMLH9evV6MWe32p3Y/mv1l4Y9/bPNZQEdc+YEQBxxAiCOOAEQxzUnog2n/v/0cPVwMeed6p2pkcsdrghYBGdOAMQRJwDiiBMAccQJgDhuiCDaQXUwsf149XhPKwEWyZkTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA44gRAHHECII44ARDneB8HHY1GrexnPB63sp/DDKrBxPbZ6uxc+6mr+sh9t/l1s3i/utYw+stW9l01rLtqad2z+Xs5dGa/HNvsfiVdOHFjcvtXV/pZx+2mf+Kz/rT/M7X9UQtroX37++XvzzfffNPKvre2thrHnTkBEEecAIgjTgDEEScA4vRyQ8T333/fyn4ODg5a2c9hhlPtfqF6Ya79jKvyxo3pfbf5dbN4v/pbw+jTrey7alj3Yv8f9I9y6OmGGyJ+0f1KurB9eXL7hYAbIqZ/4rP+tP81te2GiEy3bt0qxj7//POJ7aYb1IbD4ZFztre3G4/pzAmAOOIEQBxxAiCOOAEQp5cbIuq64ckHgxmemNDwdV06qCZvuHiuem6u/STeEFFV/20Ya+tydN83RPxQDv2xYdpG5wvpxNW9ye35XpXtmveGiIafFFRV5cwJgEDiBEAccQIgzmDR13GqqqqefPLJVg568eLFYuy7775rY9cALEBd1403HDhzAiCOOAEQR5wAiCNOAMTp5YaIwWCw+IMCEMcNEQAsDXECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4gzquu57DQAwwZkTAHHECYA44gRAHHECII44ARBHnACII04AxBEnAOKIEwBxxAmAOOIEQBxxAiCOOAEQR5wAiCNOAMQRJwDiiBMAccQJgDjiBEAccQIgjjgBEEecAIgjTgDEEScA4ogTAHHECYA4/wPd8aiZkkg/HgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 6))\n",
    "plot_observation(time_step.observation)\n",
    "save_fig(\"preprocessed_breakout_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the Python environment to a TF environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.environments.tf_py_environment import TFPyEnvironment\n",
    "\n",
    "tf_env = TFPyEnvironment(env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a small class to normalize the observations. Images are stored using bytes from 0 to 255 to use less RAM, but we want to pass floats from 0.0 to 1.0 to the neural network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Q-Network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.networks.q_network import QNetwork\n",
    "\n",
    "preprocessing_layer = keras.layers.Lambda(\n",
    "                          lambda obs: tf.cast(obs, np.float32) / 255.)\n",
    "conv_layer_params=[(32, (8, 8), 4), (64, (4, 4), 2), (64, (3, 3), 1)]\n",
    "fc_layer_params=[512]\n",
    "\n",
    "q_net = QNetwork(\n",
    "    tf_env.observation_spec(),\n",
    "    tf_env.action_spec(),\n",
    "    preprocessing_layers=preprocessing_layer,\n",
    "    conv_layer_params=conv_layer_params,\n",
    "    fc_layer_params=fc_layer_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the DQN Agent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.agents.dqn.dqn_agent import DqnAgent\n",
    "\n",
    "train_step = tf.Variable(0)\n",
    "update_period = 4 # run a training step every 4 collect steps\n",
    "optimizer = keras.optimizers.RMSprop(lr=2.5e-4, rho=0.95, momentum=0.0,\n",
    "                                     epsilon=0.00001, centered=True)\n",
    "epsilon_fn = keras.optimizers.schedules.PolynomialDecay(\n",
    "    initial_learning_rate=1.0, # initial ε\n",
    "    decay_steps=250000 // update_period, # <=> 1,000,000 ALE frames\n",
    "    end_learning_rate=0.01) # final ε\n",
    "agent = DqnAgent(tf_env.time_step_spec(),\n",
    "                 tf_env.action_spec(),\n",
    "                 q_network=q_net,\n",
    "                 optimizer=optimizer,\n",
    "                 target_update_period=2000, # <=> 32,000 ALE frames\n",
    "                 td_errors_loss_fn=keras.losses.Huber(reduction=\"none\"),\n",
    "                 gamma=0.99, # discount factor\n",
    "                 train_step_counter=train_step,\n",
    "                 epsilon_greedy=lambda: epsilon_fn(train_step))\n",
    "agent.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the replay buffer (this will use a lot of RAM, so please reduce the buffer size if you get an out-of-memory error):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**: we use a replay buffer of size 100,000 instead of 1,000,000 (as used in the book) since many people were getting OOM (Out-Of-Memory) errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.replay_buffers import tf_uniform_replay_buffer\n",
    "\n",
    "replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(\n",
    "    data_spec=agent.collect_data_spec,\n",
    "    batch_size=tf_env.batch_size,\n",
    "    max_length=100000) # reduce if OOM error\n",
    "\n",
    "replay_buffer_observer = replay_buffer.add_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a simple custom observer that counts and displays the number of times it is called (except when it is passed a trajectory that represents the boundary between two episodes, as this does not count as a step):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ShowProgress:\n",
    "    def __init__(self, total):\n",
    "        self.counter = 0\n",
    "        self.total = total\n",
    "    def __call__(self, trajectory):\n",
    "        if not trajectory.is_boundary():\n",
    "            self.counter += 1\n",
    "        if self.counter % 100 == 0:\n",
    "            print(\"\\r{}/{}\".format(self.counter, self.total), end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add some training metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.metrics import tf_metrics\n",
    "\n",
    "train_metrics = [\n",
    "    tf_metrics.NumberOfEpisodes(),\n",
    "    tf_metrics.EnvironmentSteps(),\n",
    "    tf_metrics.AverageReturnMetric(),\n",
    "    tf_metrics.AverageEpisodeLengthMetric(),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int64, numpy=0>"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_metrics[0].result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 0\n",
      "\t\t EnvironmentSteps = 0\n",
      "\t\t AverageReturn = 0.0\n",
      "\t\t AverageEpisodeLength = 0.0\n"
     ]
    }
   ],
   "source": [
    "from tf_agents.eval.metric_utils import log_metrics\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "log_metrics(train_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the collect driver:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.drivers.dynamic_step_driver import DynamicStepDriver\n",
    "\n",
    "collect_driver = DynamicStepDriver(\n",
    "    tf_env,\n",
    "    agent.collect_policy,\n",
    "    observers=[replay_buffer_observer] + train_metrics,\n",
    "    num_steps=update_period) # collect 4 steps for each training iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the initial experiences, before training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20000/20000"
     ]
    }
   ],
   "source": [
    "from tf_agents.policies.random_tf_policy import RandomTFPolicy\n",
    "\n",
    "initial_collect_policy = RandomTFPolicy(tf_env.time_step_spec(),\n",
    "                                        tf_env.action_spec())\n",
    "init_driver = DynamicStepDriver(\n",
    "    tf_env,\n",
    "    initial_collect_policy,\n",
    "    observers=[replay_buffer.add_batch, ShowProgress(20000)],\n",
    "    num_steps=20000) # <=> 80,000 ALE frames\n",
    "final_time_step, final_policy_state = init_driver.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's sample 2 sub-episodes, with 3 time steps each and display them:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: `replay_buffer.get_next()` is deprecated. We must use `replay_buffer.as_dataset(..., single_deterministic_pass=False)` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(9) # chosen to show an example of trajectory at the end of an episode\n",
    "\n",
    "#trajectories, buffer_info = replay_buffer.get_next( # get_next() is deprecated\n",
    "#    sample_batch_size=2, num_steps=3)\n",
    "\n",
    "trajectories, buffer_info = next(iter(replay_buffer.as_dataset(\n",
    "    sample_batch_size=2,\n",
    "    num_steps=3,\n",
    "    single_deterministic_pass=False)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('step_type',\n",
       " 'observation',\n",
       " 'action',\n",
       " 'policy_info',\n",
       " 'next_step_type',\n",
       " 'reward',\n",
       " 'discount')"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectories._fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 3, 84, 84, 4])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectories.observation.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 2, 84, 84, 4])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tf_agents.trajectories.trajectory import to_transition\n",
    "\n",
    "time_steps, action_steps, next_time_steps = to_transition(trajectories)\n",
    "time_steps.observation.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 1, 1],\n",
       "       [1, 1, 1]], dtype=int32)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trajectories.step_type.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sub_episodes_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAHbCAYAAADWAWzeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABGMUlEQVR4nO3dW3Mb94Hn/W+jcSRAUiRF0QfJji0/WY+ddRzP7sU8NamaC9fu1lzM9da8hXlj8wIy11tPTbm2JpmKk3V2HNmOEh1sWaJI8YBzo7ufiybApkSCIAUSoPT9sGjJEgg12fihf939738HaZoiSZIkKVOY9QJIkiRJ88SCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUk5x3F8GQeAccNKMpWkaTPpYMyvN3qSZNa/S7J2UV48gS5IkSTkWZEmSJCln7BCLV0EQBIRhSK1Wo1g8+u2maUqSJC88vlAovPC4brdLFEVHvqZUKlEulykUCqPP4eOHdygc/hoEAUEQjH4PkCQJg8GANE2JoogoigBGzxWGIaVS6YXlOU6hUBg979BgMKDdbhPH8ek/qAtSLBYJw5BCoTD6WSVJQhzHpGlKv98ffd/zIr+sYRiOXjdRFBHHMUmSEEXRC68dTYeZNbNnZWZnx7ya17O6Knl9pQtypVKhXC5z48YNPv/8c27fvj36uzRN2d/fp9lskr/ddqVSYXFxkWKxOAry/v4+X3zxBXfu3KHX67G7u8tgMOD27dv84he/oF6vs76+zsrKCoPBgE6nw2AwYDAYjF6Y5XKZcrlMGIZUKhWKxSJbW1vcv3+fVqvFnTt3+PbbbwnDkNXVVRYWFtjY2ODjjz+m0Wic+D0O3xTq9TqNRuNI0P/yl7/wq1/9ivv371/AT/d0pVKJW7dusb6+zvXr1/nkk0+4du0aOzs7bG5u0mw2+fLLL/nmm2+Yp1ueLy8v88knn3D9+nXW19e5efMmQRBw9+5dHjx4wN7eHt988w3b29uzXtRXjpk1s+dhZmfDvJrX87gqeX1lC3IQBJRKJWq1Gm+++SZ///d/z9/+7d+O/j5JEp48ecLm5uaRvZRGo8HGxgaVSmW0d/PkyRNarRZbW1vs7+/TarWI45hbt27xd3/3d6ytrfHBBx9w8+ZNoihiZ2eHXq9Hr9ej0+kAUK/XWVhYoFQqsbi4SLlc5s9//jO//e1v2draot1u86c//YkwDLl27RorKyv81V/9Ff/tv/031tfXT/weh3u1a2trrK+vE4bh6O9//etf82//9m8zDe+bb77J7du3uX37Nv/wD//ArVu3ePjwId9++y1Pnz5lc3OTb7/9dq7Cu7i4yCeffMJ7773HBx98wGeffUYQBHzxxRd8+eWX/Pjjjzx69Gjm4X3VmFkze15m9vKZV/N6Xlclr69sQYbDPb/hKaD86Z/BYMDOzg537949cnpkY2ODlZWVI+Ednr7In8IZPv/wVE2xWKRYLNLv99nb26PZbLK/vz9awevr66yurlKr1Wg0GqPHD79++LxpmtLr9Wi322xvb3Pv3j329vaOnGKqVquj00LDZWs0Gi98j2EYvnBK6LINf0bDZRueDhp+znr5jpN/zeTXf/51oIthZmefCTOrSZnX2efBvF6cV7ogjzMYDPjNb37DP//zP9Pr9UZ//tlnn7G8vMytW7eOHVN1mr29Pb788ku+//57Hjx4wB//+EfSNOXTTz/lww8/ZG1tjVqtRr1eP/broyjiyZMnbG9vs7m5yXfffTc6dVSpVKhWq7z77rtcv36dMAwpl8sUi0XSNOXtt98+8/JKV4WZla4O86qr7rVd00mS8OjRI37/+9+PTtEALC0t0Ww2GQwG5xp43+/3efLkCQ8ePODbb7/l97//PWma0mg0WF5eHg2aH7dc7XYbyN4IHj16BECtVqNarY5CH0URxWJxNAZsf39/rk6hSNNmZqWrw7zqqnOaN0mSJCnHgixJkiTlvLZDLCC7AnRhYeHIgPBqtToa2H6egeLDK3vL5TLVapWFhQXSNB0N+h9ObTPOcJD68MKEIAioVqvUajVqtRrlcnn0XMNf81fWSq8qMytdHeZVV9lrW5CLxSI///nP+cd//Mcjk2h/8MEHbGxsUKvVKJVKZ37eRqPBxx9/zMbGBrdv3+ajjz4iTVPef/99bt26xdLSEouLi2OXa2VlhYWFBW7cuMF/+k//6cgVuaVSibW1tSNX1IZhyBtvvGGA9Uozs9LVYV511b3WBfnjjz9mbW3tyByNS0tLrK+vU6lUzjVFSr1e56c//Snvvvsu3W6XVqsFZPP+1et1SqXSiVfXDpdrdXWVlZUVPvzwQ/77f//vozka83cJGn4O94SHV9xKryozK10d5lVX3diCXKlULms5pq5QKIxOjQwnLP/LX/4y+vskSdje3mZ7e/tIeLvdLnEcH7m95dbW1iiEw1skQjaNzbNnz0jTlHq9Prqzz/7+Pv1+n36/T7fbBaDdbo+mtNnd3aVUKvH999+zvb09umvQ8O4/wzv19Pt9dnd3T7wNZj68vV6Pbrd75LGPHz8mTdOZrcdyuUySJHS7Xfb29vj++++J45gff/yRzc1Nnj17RhzHVCqVmd9SMi8MQ9rtNjs7Ozx58oT79+8TBAGbm5vs7+/T7XZHd2uaN/O4TJMys2b2vK5qZudtec7CvJrX87oqeQ3GTVvy/vvvX9k5TfKTUNfrdW7fvs3q6uro74dTweTnZwRGKyU/aXmv1+P+/fs8ffqUOI7pdrskScL169d58803R3us1Wp1dM/3JElG90OHw/ulB0EwCmi73WZ3d5d+v8/W1hZbW1sEQTA69dRoNLhx48bozeKk7xOysV6VSuXI3vjOzg537txhZ2dnij/ZyYVhyPLyMgsLC9TrdTY2NqhWq6O9/n6/z6NHj9ja2pqr6XNqtRpvvPEGCwsLLCwssLy8DMCzZ8/Y3d2l1+uN7sx0Ge7evTvxIRYza2Zfhpmdjkkza17N68swr9NxUl7HFuRPP/10fn6i0mvqd7/73cQF2cxKszdpZs2rNHsn5XXsEItf/vKXF7M0ki6EmZWuDvMqza+xR5D/4z/+w71bacY++uijiY8gm1lp9ibNrHmVZu+kvI49gnzz5s2LWRpJF8LMSleHeZXml3fSkyRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUs7YW01PKkkSkiQBIE29tbx0kiDIbvleKBQoFGa3f2pmpcnMQ2bNqzSZaeZ1KgW51+uxv79PHMckSUIcx9N4WumVEoYhhUKBMAxZXFykVqvNbFnMrHS6ecmseZVON+28TqUgR1FEs9kkiiKiKDK80jGKxSLFYpFSqTTTcgxmVprEvGTWvEqnm3Zep1KQ9/f3uXv3Lu12m263S7fb9TSQlBMEAdVqlWq1Sr1ep1qtsri4OLPlMbPSePOUWfMqjXcReX3pgpymKffu3eNf/uVf2NzcZGtri2fPnhleKScIAlZXV1ldXeXGjRvU63XeeuutmSyLmZVONy+ZNa/S6S4ir1M5gtxqtXj48CGPHj3i8ePHbG5uGl4pJwgC1tfX2djYoN/v02q1Zro8ZlYab54ya16l8S4ir07zJkmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVJOcRpPUigUKBaLo89SqUSaptN4aumVEAQBpVJplJFCYbb7pmZWGm+eMmtepfEuIq9TKcjlcpmlpSW63S6DwQDA8Eo5QRCwtrbGysoKy8vLlMvlmS6PmZXGm6fMmldpvIvI61QKchiG1Go1FhYWqNfr9Ho9wyvlBEFAvV5nYWGBarVKGIYzXR4zK403T5k1r9J4F5HXly7IQRDQaDR49913WVpaYm1tjb29vZdeMOlVEgQBS0tLLC8vs7q6SqPRmOmymFlpvHnJrHmVTncReZ3KEeSNjQ3+5m/+hk6nQ6/Xo9frTeNppVdKpVKhUqlQq9W4cePGTJfFzEqnm5fMmlfpdNPO61QKcqVSYWVlhXq9ThRFozFSkg4NL64pl8tUKpWZLouZlU43L5k1r9Lppp1Xp3mTJEmScizIkiRJUs5UhlgEQUChUCAMQ5IkIU1Tr7CVcoIgIAzD0WcQBDNfHjMrnWyeMmtepfEuIq9TKcjFYpGFhQXiOGYwGDg+SjpGfqL/YnEq0XupZTGz0njzklnzKp1u2nmd2jzIlUqFOI4pFovEcTyNp5VeKcM921nflWu4LGZWGm9eMmtepdNNO69THWIxPO3jqR/pRYVCgUKhQBAEczPEwsxKJ5uXzJpX6XTTzutUC3L+/w2wdCg/hnAY4HlYnvz/m1np0Dxl1rxK411EXqc2qGrY2POfkg7NWzbMrDTePGXDvErjTTsbU7/qwOBKx5vXXJhZ6XjzmAvzKh1v2rlwHmRJkiQp50LmrXHvVrpazKx0dZhX6eJdyBhkSS+at4zM2/JI82aeMjJPyyLNo2lnZOpDLAyvdLxhNuZtIzdPyyLNk3nM7LwshzRvpp1XxyBLkiRJORZkSZIkKWcqBdmxUdJk5iUjZlaazDxkxLxKk5lmRqZykd7wPvGAt8GUTpDfyIVhONNlMbPS6eYls+ZVOt208zqVglwsFqlUKu7hSqcYbtziOCZJkpkth5mVJjMPmTWv0mSmmdepFOThPbA9DSSdbBjcNE1nWo7BzEqTmJfMmlfpdNPOqxfpSZIkSTkWZEmSJClnKkMshuM98pM0SzpqeGHNPFxkY2al081LZs2rdLpp53UqBXkwGDAYDGa+0ZeugiAIKBaLFAqzO4FjZqXJzTqz5lWa3LTyOpWCnCQJURQdGSAt6aj8BTaFQmGmBdnMSqebl8yaV+l0087r1ArycEqNJEkMsPSc50M761kszKw03jxl1rxK411EXqdWkPv9PnEcj0JseKVDw+CGYUgYhpTL5Zkuj5mVxpunzJpXabyLyOtUL9Ibfg4GgxMf+/zFBV5sMB/yb7anvfG6Dg89/7Ma97MrFrO4BUEw842bmb36zOz5XMXMmterz7yezyzzOpWC3O/32dvbI4oi+v3+iRcTDAdOF4vF0a0AZzkOU4cGgwH9fn80wfZJpydch0cNT32maTr2QpogCCiVSpRKJcrlMtVqlVqtNoMlzpjZq8/Mns9VzKx5vfrM6/nMMq9TKchRFLG/v0+326XX641eBM8LgoBKpUK1WqVQKFAul0eNX7PV6/VoNptHjlIcx3V41PBNL0mS0ev/pNf+MLSVSoVr165d/sLmmNmrz8yez1XMrHm9+szr+cwyr1P5qXe7XZ4+fUq73abdbtPpdI59XBAENBoNFhcXKRaLLCwsUKlUprEIeglpmtJsNnn69ClRFBFF0Ymn8FyHh9I0pd/v0263GQwG7O/v02w2Tzy1U6vVWFhYoF6vs7GxcclLe5SZvdrM7Plc1cya16vNvJ7PrPP60gU5TVN++OEH/tf/+l88ffqU7e1tdnZ2jj19EIYh77zzDu+88w61Wo2NjQ2Wl5dfdhE0Bffu3eP//J//Q6vVotVq0W63j30Rug6P2t3d5fHjx3Q6He7du8eDBw+OPTJQKBRYWVlhZWWF9fV1VlZWePvtt2ewxGb2VWFmz+eqZda8vhrM6/nMMq9TOYK8u7vLd999x6NHj3j8+DGbm5snrvhWq0WapjQaDcIwnPnFSsr88MMPfP311+zu7rK3t8f+/v6xj3MdHvX06VMePHhAq9Xi66+/5s6dO8duuIIgYH19nY2NDXZ3d9nd3Z3B0h4ys1efmT2fq5hZ83r1mdfzmWVeX9+R35IkSdIxLMiSJElSzqVfGpmf4uS0+Rx1eYbTqExyKsd1eOj5u1u9ilzf88nMns+rnlnX9Xwyr+czy7xeakFO05Td3V0ePHhApVJhd3eXRqNxmYugEzx69Ijd3V3a7TZRFJ34ONfhUc1mk62tLbrdLru7u6/cBtf1Pb/M7Pm8ypl1Xc8v83o+s8zrpRbkJEnY2dmh1+sRhiEPHjygVCpd5iLoBO12m93dXQaDwYnzM4Lr8HlRFNHr9Yjj+MSrkq8y1/f8MrPn8ypn1nU9v8zr+cwyr5c+xCKOY6IoOnJnFM3e8AU4yYvPdXhoMBgQRdHYOyNdda7v+WRmz+dVz6zrej6Z1/OZZV4vvSAPv9EgCCgUCq/1PcbnSZIko1s4nhZg1+Gh4VixNE3HHhW4ylzf88nMns+rnlnX9Xwyr+czy7xeekF+VffaXyeuw9eL6/vqcx2+PlzXV5/rcD44zZskSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJOcVZL4CmIyRbmSkwAJLZLo4kSdKVZUG+wgpkKzAEbgHvAzHwR+DBDJdLkiTpKrMgX2EBUAZKwLvA/wtEwA7wkOxosiRJks7GgnyFlYAloAo0gApZaXZguSRJ0vlZkK+wZeA/AyvATbKS3MKVKkmS9DLsUldYBXgD2CAryWWgR3YUWZIkSedjQb4qCrnPKlCCOIJmCypxVoz3gDawj+OPpVk7JrJEZGd54hkulyTpdBbkq6JIdoi4DKwDS9Dbg80H0GtnG97ewefjGS6mpMwxkWWPbIaZ9gyXS5J0OgvyVVEgOwRVBurAYnYEuVvIVmKHbKPbJyvJkmbrmMgS4UW0knQVWJCvggBYBd4CFsgmPV6D7gP44SFUmtmGt0926rY7uyWVxImR5QHZFIzN2S2aJGkCFuSrYLi1/YDsMNR7wDp0ivDDvx99aIrjj6VZOyGyFIF/H/N1kqT5YEG+CgKyc7U1sqkrErLDxREk3lNamjtjIutt4CXpCrAgXxULwHWy+0o/BX4gO1freAppLhlZSbq6LMhXwfCe0g2y8RObwBbZVncww+WSdCwjK0lXmwV57oRk17xXGJ2kTQuwvQ3fbQMpPCGbL2qX7JytpNkxstKVcUxcKQDbB59ew6MhC/LcKQPvkF3zvgzcgrgI3/07PPo1EGVb2AFOWSHNAyMrXRnHxHV08exBXCXAgjyHQrLRi8tk18HfAErQakDLm0hLc8fISlfGCXGlQTY0ShqyIM+dErBBNjFUSHYLkCbZcSdP/khzx8hKV4Zx1aS8qdPcKQNvAz8lmzm1TTYyqo3xleaQkZWuDOOqSXkEeYYCAkJCAAoUCAhIKTMgJSEiu2l06+CzP8MllZQJ4CCz2fGFANIUBgOIEiMrzRPjqpdgQZ6hKlWWWaZIkRo1qlTpUeURj9mnR3bN+w9kJ4Ge4C0GpFmrko1eLJJd/16Fbg/uPoLBvpGV5olx1UuwIM9QmTJLLFGhwuLBRxvYYYd9toB94Eey0VE9jK80a2VgiWySqMXsM2rDjzsQ7RtZaZ4YV70EC/IM1aixwQYLLFChQpkyEBGyT3bZQJvsxM9wgihJs1Uju8RngWyrW862qu0wezc1stL8MK56CRbkGbrOdf4r/5VllunQoUuXbXao8i1wnyyyEdmlA+7bSrN3HfivZOdtO0AX+tvwYzW7TZ6RleaHcdVLsCDPUJUqa6yxyiq77LLHHm3ahPTJjiBLmi9VslsMrJKNYNyDtA3dcPyXSbp8xlUvwYI8QwkJERFduvzIj9znPrvs0rQcSzNXpDiaZSY4uIVATMiAASldstGL98m2vGZWmq0ih1NWHNzyI4khGkA3Na46MwvyDMXE9OjRps097vFbfkuXLrvsznrRpNdaQECJEhUqBAcfAH2KxPRJaQP3gN+SXeJjZqXZCchuAVI5+P1BQY770IuhnRpXnZkFeYYGDGjRAqBJkxYt+vSJvVxAmqmAgDJl6tRHf5YCKQUCOmQjFpsczqBqZqXZCcimrKjn/iyFQXp4v3fjqjOyIM/Qj/zIv/KvFCnyhCe0aBEffEianSJF3uM9bnOblJQ2bQYMeMyAb/nfxAzIZk5tkW1tzaw0O0Wym0ffJtuVbQMD+PEx/Ou3UIyNq87MgjxDW2yxzTYAqTe5lOZCQECRIm/yJj/jZ8TE7LBDly4Jd/kz/5dsBlUzK81eQFZl3gR+RtZ+d4AubCWw/efsYcZVZ2RBnjGLsTRfhmOOq1Rp0KBHj4iIFi169EiJcWsrzYvhmOMq0CC75UdEdri4l91bWjqHwqwXQJLmxbAcFymyyirv8i7Xuc4uu9zjHptsMmAw68WUBByW4yLZXG7vkk1+vEt2Vd4mmFedkwVZknICAgoUqFChQYMqVXr02GWXNm0SbykgzYlhQS6QzWDRIDuS3CMryW28BYjOyyEWkpSTkhIRcY97/IbfsMMOP/CDBVmaAxUqrLJKhSoxCQMSEsrs85Q2/w48A37AgqyXZUGWpAMpKQkJPXr8gT/wmMf06PE939OkSULiLDPSDC2wwAd8wBprdA/uI9Aj4QEP6PDng5v4fE82r1uCU1bovCzIkvSchIQWLUJC+vTp0iUimvViSa+14fUBDRoss0xxdEYnIqQJ7JENr+iCedVLsiBLUs7wKHKLFgMGxMRERM44I83I8MY9JUqsscZP+Sm3uMX3fM8dviGgRzacYpvsorwIZ5rRy7IgS9JzUlI6Bx+SZisgoEKFGjVWWOE93uMDPiAm5i53ycpwh2zcsWOONR3OYiFJkuba8MxOTEyHDk2atGiNfo3o41FjTZNHkCVJ0txKSRkwICJijz3+xJ9o0+ZrvuYbvqFJkw4dh0FpqizIkiRprqWkxMT06LHNNgUKbLLJFlu0ac968fQKsiBLkqS5NSzHAQEtWnzP9+yx550tdaEsyJIkaa4NDj4iIlq0KFAgOviQLoIFWZIkzbXh+OJhUZYumrNYSJIkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJElXWkAw1ecrjvvLwWAw0ZMkSUKapqNPaRIBsABUyfbUwoM/CwkpUaRAgSJFwtHfTCYup/QXU5IiJEGJOCiRf1XGFOgRElMgPfj/lJQBA2ImeM0PBrC3B73e5N/sgTRNSZKEKIrodrsTfc3S0tLEz29mdVlCQurUKVIkoEKW5gIpIekZj72EhZhKqU+hkJBUEpJqDMFpr8sU6AA9IAEGB7++6CUie6GZNa+6PCFQB4pQDqEWQhhkfzy2Cb6oAJQOfq2QbcOPbqFTsjymBHSBDkEKlU6ZUq9MkEBhUIDk+O16TECHEoODbfQkAgLWWGMtXeNaco1CVHjpvI79sURRdOoTp2nKYDAYBViaVAisAtfJXojV0a8lGjQoUmSBBWrUzrRf2Kmn7L6TEC1AL6zTCxtHNthdimxTo0fIgJA+BWJSWrSI6Zz+DzSb8Kc/Qb8PZ3jNDze0cRzT7XZpNpsEwXT3eM2sLkuZMje4QZ06AasUeBMoM6BMTJmz7NTWih1WG7uUShHRWo/oeg8Kp702B8AT4BkQAU1OKsjnjOyFZ9a86vKUgRtAHRYqsFGDciHrzC823LGKQIOsJK+RbcOP7hIPgC4wIGCLgB8JBymrT5ZYfLZIISpQbBYpJMfvSHcp8iMNWpSAYKKSHBCwlC6xnCyzEq8QdsOXzuvYgpwkx7/Z5A3fQKSzCsj2PhtkL8QFssAtUGCZIkVK1KmwQI2zpLdTTCnWY6JF6IQNOuEyaXAYxBIlOgd70gVCUkIKJBQokb2JTKB4EJ0gOHNJTtOUOI4nPnp0FmZWlyUkpEr1oCAvUWANqDCgymDSHB2oFzosl4qUyxFRrUN/qQPhabmKgH2yYpxy2nvEOSN7oZk1r7o8IVkTrkOxCrU6VMNsA1znTAV5eAS5DNSApYNnPxSRbdUHBDQJCChGsLZfYrlZpZCGlIISwQlnmjqU6LFIkQqQHYc+TUBAgwb1tE4trUE8+Rmak4wtyL0JzkelaUoURSRJ4h6uziQE3gA+JIvttdGvZd7gGmWqVFinxOqZxhZ1lxJ2PxwQXYe98gY75TdJckHcoca3rLJPhQ5FmpSJiOnxmDbP4LT91ceP4eFD2NrKtrQTvuaHR6KiKKLVarG7uzvx9zQpM6vLUqXKLW6xwRuEvEvIz4EFeoT0znjOdq3c5f2VXRYWIrof7NH5cIekeFop7JFtphOgdfDZP/aR54zshWfWvOryVIFbwAY0FuEn16ERwjrZqdwzFOQysEJ2UOsDsm340cT3gGcE9A6GTf5AqZfyVmmNteQtCq0ihVaNoB8+/9QAtKhyjzfYpU56cAT5tFd9QEA5KVOJK1SiCsVW8aXzOtWC7PgonUUIbAA/JduB3SAL3AZlfsIyFRYIeevg1O3kYxp7izHN/yciuglb1Xd5UnufJDh8qT+mTp832WKBJmXKVOkSscM94MfT/4E//xn+9//ODkWdQf4o1HBjO+0hFmZWl6VChbd5m/d4j5CPKfFLUhbpwCQDlY54s9Tj02tNFpciWu9tsf/pE9LKaQW5BewCO2TvJpsnPvKckb3wzJpXXZ4K8DbwHiyuwjtvwUo568xn28RSIjugtZQ9G58ePPuhFvAYaFPgGSElyq2En+yusLFzkyAsweZwkMaLmtS5wU94xjVSApJJ23sKQRzAAAqtAnu7e2cq/s8bW5DPEkRDq7MaXmKzx+EJmQ4QkFChT5mQkDYF9k88FXOcXhTT2hsQbcOz6g5blS3i4HBP9RkdmhToUKVDmR4V+gyIeQJsceq+6s5ONpjxJVzUhs7M6rLExLRps8ceIVsUeUTKHl0CumfcKtWSHpu9Fp1uRHv/Ga3tLZJSfMpXdcg2xD2yd5CTX89TiOyFZNa86vLEQBvYg6gAzSqExWwI/9kuGSAhS12XbJDTNs9X3Q7ZjmuHkA4FEsqdlGqrR9xrEUQlSFOCEwpyiwGbPGOH+ODo8WTjkENCCgcfC+kC5bT8UjNbnPHaRWl6IuBbsk1ckdH1tdRps8yPhBQpsUuRe5wlvdGThPa/JsQN6BS/pV1cPnKRXpsSj2nQpUif8GBGi4QWewdLc4q9veycbZKcbTDjGaVpOvUjzNK0NGnyFV9xn/sE3KHA78gu0gsZnHEWi3on4v8+alPejok6HfoP2qQTjUH+nuzIce/g83iXFFkzqznWBL4C7sPTKnzZgGoBFsnGIZ9BB3hEVow7wAOOG4PcBiICHhHQJYxg5fuHLG62CHoFgl6JgOOHWPQp8ZQ/0KYycTkOCKhRo37w8TN+xk1ujv2a0/JqQdbMxMBTsoDlp40p0afCHgUKlGhTPmEv8yRRE7p3U+IyDIIyUaFCvmBHFGhTIqZATHAwlUxKnz5ZsE/R62WXxV/wER03tJpnPXo84hHbbAOPCQ42kwlFUooTT88EUI4iftjrUghjku6AeDuaYJ84Idvot8neTU4+4nxJkTWzmmM9RrW2FcL3ZSgG2dHjs21iicjO/IZkR5G3eT6uycGjhhkdUEhgoblDtd0liAPCuHDi0d2IkH3K9A+meJ14FouDj2tc4yf85PSvOSWvYwvynTt3JlgsePjwIa1Wi263SxzHngrSRFKyS2oKB5/D34ekdIkJSCgCxROmbjpJHKf0O5BEEAcxcTA4MvhwQECPIsnB2KbsM5sJedxGdiSKsolVYeItbpqm9Pt9Wq0WxWKRr7/+eqIpngA+//zziR4HZlaXJyGhT59kNEowGc2BnJ5wZOgkcRoTD/oUkoS0F5ME8YQFucfh/Mcnv4bPEdlLyax51eVJyLayCcQh9CIYZON1z3irgdEsx8MEBi98eUK2LR0W5ZQggag3oDPoQwKF9OSCnM2DPCAanYmaYOFSSPoJ/VafTrHDV19/xU60M9H3c1Jeg3FB+6d/+qeJUnj//n2++uorms0m7XabTuesl2jodTUsx3AYgSxsw/+efQRRGkB6EPjs9MzzzxAcbE4P/zwb5zTJtbJkW9g4PvPhqEKhQBiGhGHI4uIilUploiNO9+/fn/hHYGZ1mULCUVazJGe/P2t9C0gpBOnBl6ekp86BPDQsxuOze87IXnhmzasu18GGMQAKwdGN7hkFuc/jB1QNX9oJHFxmFyQBQVrI7gE05pU/3OU+6/tIWAgphAUKYYGFxQXKlfJL5XXsEeTHjx9PtFA7Ozv0er3RZObSpBJOmtp/wrJ60pdOf4rhlzacpuksd+Q6KzOryxRPcsZlUvnIT/FpX8ZFZ9a86nIdBCvlpTN2vrhO4R8eI0qi0UHrZrf50s83tiB/8803Ez3J/v4+rVaLKIqI4zl5Z5NeQ2ZWujrMqzS/xg6xKJVKEx3Cc35G6eKkaTrxCTAzK83epJk1r9LsnZTXsQU5CALTKM3YWQqymZVmb9LMmldp9k7K69kmq5QkSZJecRZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5QRp6q3gJUmSpCGPIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTnFcX8ZBIG32ZNmLE3TYNLHmllp9ibNrHmVZu+kvHoEWZIkScqxIEuSJEk5FmRJkiQpZ+wY5FdBEASEYUitVqNYPPrtpmlKkiQvPL5QKLzwuG63SxRFR76mVCpRLpcpFAqjz+Hj0zQd/X74vEEQjH4PkCQJg8GANE2JoogoigAIw5BCoUCxWKRSqYwen3/u4fMOFQqFI48DGAwGtNtt4jg+40/t7AqFwmi5wzAkDMMXlicv/7PI/zyGP4tut/vC93hZCoXCaL2GYTh63URRRBzHJElCFEUvvHY0HWbWzJ6VmZ0d82pez+qq5PWVLsiVSoVyucyNGzf4/PPPuX379ujv0jRlf3+fZrN55EVSqVRYXFykWCyOgry/v88XX3zBnTt36PV67O7uMhgMuH37Nr/4xS+o1+usr6+zsrLCYDCg0+kwGAwYDAajQJbLZcrlMmEYUqlUKBaLbG1tcf/+fVqtFnfu3OHbb7+lWCyysbFBo9HgnXfe4b/8l//C8vIycRyPPlutFr1eDzh88dfrdRqNxpE3nr/85S/86le/4v79+xf+s15cXOSdd96hXq/zxhtv8Pbbb7/wZjk0DEShUGBhYYFqtUq/32d3d5der8cf//hHvvjiC1qt1oUv93GWl5f55JNPuH79Ouvr69y8eZMgCLh79y4PHjxgb2+Pb775hu3t7Zks36vMzJrZ8zCzs2Fezet5XJW8vrIFOQgCSqUStVqNN998k7//+7/nb//2b0d/nyQJT548YXNz88heSqPRYGNjg0qlMtq7efLkCa1Wi62tLfb392m1WsRxzK1bt/i7v/s71tbW+OCDD7h58yZRFLGzs0Ov16PX69HpdACo1+ssLCxQKpVYXFykXC7z5z//md/+9rdsbW3Rbrf505/+RKlUYn19nfX1df76r/+a//k//ydvvfXWaO83iqLRcgzfXIIgYG1tjfX1dcIwHH0vv/71r/m3f/u3SwlvvV7nvffeY21tjb/6q7/ik08+oVwuH/vY4RtZsVhkZWWFpaUl2u02P/zwA81mk4WFBX7729/OLLyLi4t88sknvPfee3zwwQd89tlnBEHAF198wZdffsmPP/7Io0ePZh7eV42ZNbPnZWYvn3k1r+d1VfL6yhZkONzzG54Cyu9tDQYDdnZ2uHv37pHTIxsbG6ysrBwJ7/C0Rv5UxfD5h6d9isUixWKRfr/P3t4ezWaT/f390QpeX19ndXWVWq1Go9EYPX749c+fKsk/bxiGo1MPvV6PR48e8fjx4yN/H4YhGxsbR77H007BTFMURezt7REEAT/88AONRmMU0OHyDU+pVKvV0RvZtWvXRutm+NjnT79dtvxrJr/+868DXQwza2bPw8zOhnk1r+dxVfL6ShfkcQaDAb/5zW/453/+59GpFIDPPvuM5eVlbt26deyYqtPs7e3x5Zdf8v333/PgwQP++Mc/kqYpn376KR9++CFra2vUajXq9fqZnrfT6bC9vc2TJ0/41a9+xW9+8xtKpRILCwtUKhX+x//4H/zkJz+hVCqd6XmnZXd3lz/84Q+USiX+8Ic/0Gg0KJVKLC8vj/bs33jjDWq1GktLS6yurrKwsECtVmN1dXUmy6yrxcxOl5nVRTKv02VeL99rW5CTJOHRo0f8/ve/H52iAVhaWqLZbDIYDM418L7f7/PkyRMePHjAt99+y+9//3vSNKXRaLC8vEyapvT7/TM953BQfafTYW9vj++++44vv/yScrnM4uIitVqNTz/99FIuFDhJv9/n6dOnR/6sWCxy/fp1lpaWWF5eZjAYsLi4OHqz7Pf7R944pXHM7HSZWV0k8zpd5vXyOc2bJEmSlGNBliRJknJe2yEWwGh8UX5AeLVaHQ28P89A8eGVveVyeTRQPk1TqtUqpVJpNLXNeZ53OKi9UqmwsLBAuVymVqtRq9UolUozHdg+XLbhRRXDiwKGyzecDmj4M8hfQCFNysxOj5nVRTOv02NeL99rW5CLxSI///nP+cd//MfRPIoAH3zwARsbG6NAnFWj0eDjjz9mY2OD27dv89FHH5GmKe+//z63bt1iaWmJxcXFMz1nEASjgfZhGPL555/z7rvvjoJcKpX4+c9/PrOLByCbtuXmzZvU63Vu3LjBm2++eeQCh3K5zPLyMuVymYWFBZaWlqhWq2f+Wej1ZWany8zqIpnX6TKvl++1Lsgff/wxa2trR+ZoXFpaYn19nUqlcq4pXOr1Oj/96U9599136Xa7o3kGFxcXqdfrlEqlM19dGwTBaEqcWq3GL3/5S37+858fmQLn+vXrJ86JeBkajQa3b99mbW2NDz/8cDRH4/NT9gRBQLlcHk3k3mg0ZrbMulrM7HSZWV0k8zpd5vXyjS3IlUrlspZj6gqFwuhUw3DC8r/85S+jv0+ShO3tbba3t4+Et9vtEsfxkdtbbm1tjUI4vEUiZNPYPHv2jDRNqdfrozv77O/v0+/36ff7dLtdANrt9mhKm93dXUqlEt9//z3b29ujuwYN9wIhm/Nwf3+fhw8f0u/3R889/DdbrdaRScyjKKLf7x85nfL48WPSNL2U9VgsFonjmH6/T6vVYnt7+8S97eHpsTAMabVaNBoNut0uT548od1u02w2R7cAnYUwDGm32+zs7PDkyRPu379PEARsbm6yv79Pt9sdHVmYN/O4TJMys2b2vK5qZudtec7CvJrX87oqeQ3G3Yv7/fffn82NuqcgPwl1vV7n9u3bR+YCHE4F8/wUKMOVkp+0vNfrcf/+fZ4+fUocx3S7XZIk4fr166PTHPV6nWq1OpouJkkSkiQZTQsznKA7CILRuKB2u83u7i79fp+trS22trYoFArU63UqlQpLS0u8/fbbVKvV0T3Uh/coHwwGo+8TskA8f0/5nZ0d7ty5w87OzgX/tLNxZdeuXaNUKtFoNFhaWjpx7FN+r3z4Bjv8ucZxzObmJvfv3z/zVD3TUqvVeOONN1hYWGBhYYHl5WUAnj17NrpV5/DOTJfh7t27Ex9iMbNmdlJm9uJMmlnzal4nZV4vzkl5HVuQP/300ysbXulV8bvf/W7igmxmpdmbNLPmVZq9k/I6dojFL3/5y4tZGkkXwsxKV4d5lebX2CPI//Ef/+HerTRjH3300cRHkM2sNHuTZta8SrN3Ul7HHkG+efPmxSyNpAthZqWrw7xK88sZpCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpQz9lbTk0qShCRJAEhTby0vnSQIslu+FwoFCoXZ7Z+aWWky85BZ8ypNZpp5nUpB7vV67O/vE8cxSZIQx/E0nlZ6pYRhSKFQIAxDFhcXqdVqM1sWMyudbl4ya16l0007r1MpyFEU0Ww2iaKIKIoMr3SMYrFIsVikVCrNtByDmZUmMS+ZNa/S6aad16kU5P39fe7evUu73abb7dLtdj0NJOUEQUC1WqVarVKv16lWqywuLs5secysNN48Zda8SuNdRF5fuiCnacq9e/f4l3/5FzY3N9na2uLZs2eGV8oJgoDV1VVWV1e5ceMG9Xqdt956aybLYmal081LZs2rdLqLyOtUjiC3Wi0ePnzIo0ePePz4MZubm4ZXygmCgPX1dTY2Nuj3+7RarZkuj5mVxpunzJpXabyLyKvTvEmSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUU5zGkxQKBYrF4uizVCqRpuk0nlp6JQRBQKlUGmWkUJjtvqmZlcabp8yaV2m8i8jrVApyuVxmaWmJbrfLYDAAMLxSThAErK2tsbKywvLyMuVyeabLY2al8eYps+ZVGu8i8jqVghyGIbVajYWFBer1Or1ez/BKOUEQUK/XWVhYoFqtEobhTJfHzErjzVNmzas03kXk9aULchAENBoN3n33XZaWllhbW2Nvb++lF0x6lQRBwNLSEsvLy6yurtJoNGa6LGZWGm9eMmtepdNdRF6ncgR5Y2ODv/mbv6HT6dDr9ej1etN4WumVUqlUqFQq1Go1bty4MdNlMbPS6eYls+ZVOt208zqVglypVFhZWaFerxNF0WiMlKRDw4tryuUylUplpstiZqXTzUtmzat0umnn1WneJEmSpBwLsiRJkpQzlSEWQRBQKBQIw5AkSUjT1CtspZwgCAjDcPQZBMHMl8fMSiebp8yaV2m8i8jrVApysVhkYWGBOI4ZDAaOj5KOkZ/ov1icSvRealnMrDTevGTWvEqnm3ZepzYPcqVSIY5jisUicRxP42mlV8pwz3bWd+UaLouZlcabl8yaV+l0087rVIdYDE/7eOpHelGhUKBQKBAEwdwMsTCz0snmJbPmVTrdtPM61YKc/38DLB3KjyEcBngelif//2ZWOjRPmTWv0ngXkdepDaoaNvb8p6RD85YNMyuNN0/ZMK/SeNPOxtSvOjC40vHmNRdmVjrePObCvErHm3YunAdZkiRJyrmQeWvcu5WuFjMrXR3mVbp4FzIGWdKL5i0j87Y80ryZp4zM07JI82jaGZn6EAvDKx1vmI1528jN07JI82QeMzsvyyHNm2nn1THIkiRJUo4FWZIkScqZSkF2bJQ0mXnJiJmVJjMPGTGv0mSmmZGpXKQ3vE884G0wpRPkN3JhGM50WcysdLp5yax5lU437bxOpSAXi0UqlYp7uNIphhu3OI5JkmRmy2FmpcnMQ2bNqzSZaeZ1KgV5eA9sTwNJJxsGN03TmZZjMLPSJOYls+ZVOt208+pFepIkSVKOBVmSJEnKmcoQi+F4j/wkzZKOGl5YMw8X2ZhZ6XTzklnzKp1u2nmdSkEeDAYMBoOZb/SlqyAIAorFIoXC7E7gmFlpcrPOrHmVJjetvE6lICdJQhRFRwZISzoqf4FNoVCYaUE2s9Lp5iWz5lU63bTzOrWCPJxSI0kSAyw95/nQznoWCzMrjTdPmTWv0ngXkdepFeR+v08cx6MQG17p0DC4YRgShiHlcnmmy2NmpfHmKbPmVRrvIvI6tYv0hnu2wz1dSS8ahnjWzKw0mXnIrHmVJjPNvE51DHIcx6OLCSQdVSwWKRaLM7/pAJhZaRLzklnzKp1u2nmdWkEeDAZEUWR4pRPkxw/OQ0E2s9J485JZ8yqdbtp5neoY5H6/PwqwpKOKxSKlUmk0p+ksmVnpdPOSWfMqnW7aeZ1KQe52u2xubtLtdul2u/R6vWk8rfRKqVQqVKtVarUaS0tLLC8vz2xZzKx0unnJrHmVTjftvE6lIEdRRKvVot1u0+l06Ha7XmEr5QRBQLVaJYqi0enSWTKz0njzlFnzKo13EXmd/eX0kiRJ0hyxIEuSJEk5U5sHeTiBufeMl14UBMEoI3EczzwfZlYab54ya16l8S4ir1MpyHt7e3z33Xfs7Oywt7fH/v6+4ZVygiBgcXGR5eVlrl27xq1bt2a6PGZWGm+eMmtepfEuIq8vXZDTNGVnZ4dvvvmGJ0+e8PTpU7a3tw2vlBMEAWtra6ytrbGxscFnn302s2Uxs9Lp5iWz5lU63UXkdWrzIPd6PXq9Ht1ul06nY3ilnCAIRtMz9fv9ubhRiJmVTjZPmTWv0ngXkdepTvO2v7/P3t4eu7u7My8A0jwpFAqUSiUqlQqLi4tEUTTT5TGz0njzlFnzKo13EXmdSkGO4/jInm273XbvVsoJgmA0f2m32535nfTMrDTePGXWvErjXUReneZNkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKUY0GWJEmScizIkiRJUo4FWZIkScqxIEuSJEk5FmRJkiQpx4IsSZIk5ViQJUmSpBwLsiRJkpRjQZYkSZJyLMiSJElSjgVZkiRJyimO+8vBYDDRkyRJQpqmpGlKIU0pAWXgOlAHusAeMDj4feelFvnVExCwcPAREFCgQEAw0VcCpATEVBhQGf3ZJAZBTKfUYVCIshVW47ldphRIDj77QPPg9y8KgcrBr8P1f+KSxGQvhoTsxdA7fVkTstfO4OCzf7B0V02apiRJQhRFdLvdib5maWlp4uc/T2bT9Cr+JF9OQECNGmXKFChQokThTMcLAgZUiA9e6elzX5sC3YOPF16pwcHnAtDgLJElYECJJgUiyiTUSM4b2TM5R2RfGReZ2Vc9r2XKlCgRElKhQnF85RgjIKZMQpGUAgkhJwUnJqFPn5iYhISYGII0y9vCwZcVjvvyzsFnyuGr/aXimj1NE4g4zGXOBcX1tTbNvI59tUZRNNHCDAYDkiSBg3JcA5aBvwZuApvAn4DWwe+P2WS81gICVlnlTd4kJBy9oZwmHW2YC3S4RpvrpBN83VA37PBj7UcGpRZcA94ga7cjCdmmMAaeHfz++E1jGVg5+HWJ7OlOrBtdoE32pvHjiU95RA94evBlbbL3nfj0L5srw+DGcUy326XZbBIEZ3q7PdVZM3uVNrbTVKDAtYOPEiUWWaR09MU/VkqBDit0WSElJCE8UpJTEp6yRZ9NkvwmLyDbiwzJjiDc4pR34aNCWtS4T4k9rjHgDSJK+XfTySN7JueI7CvhojP7Kud1uBO6zDJlyqyySp36uZ4rJaTLEhF1EooMqJKcEJyIPs/YoUuXARFdeqRBAqvAmxwexTmyqUzI2skTsld5B+i/bFyz0nOfwyOEEVxCXF9b087r2HWdJKfvy6RpSpImo92sIMheTMOitEa2Z1Qle20M9/uuTswvXkBAmTKLLFKkSInSRHvaWUHONsxF1ghYP9iznkwatCkUe1AqZitokeww8EjM4XHbLuNG5BTI1nmV7KzBEhy/JCnZm1NA9k4wYSfpkr3XDN9jplsrL8/wCFAcxxMfPTqLSTObJsnoyMhV/Vm+jICAEiUWWBhlr3L0xT/WMHMF1o4tyAnpQXHt88IxoeEPvUIWlMl7OQHBwTtEgSqFl4nsxM4Z2VfGRWZ24m3sBI+bR0WKlClTpcriwcd5JISUWKXHIglFImonbut69CkREtEhoU9AJyvIZbJtXJHsRXxkE5uQbWGKZK/4w3fFl4hr9oUlODiO9YILiOtrb5p5HdvCer3T92XSNCUiIqknpIsJ5b2URgmWEliP4Y00W/HnPbHyOihQ4A3e4Gf8jAoVatQmOpqVbZiLJITssM5T3iQ+w0/6Wdjk6eKf2a/vwrvAz8nOI43EwD7ZZvEu2a7w8QNkKsA62RvITeAnjFnnO2SHg1sHT/+EU/eY9sne354A2wdPMf16ebGGe7ZRFNFqtdjd3Z36vzFpZoki6knCYpKwd3DmJyFb46/DzmtIyDrrvM/71KnzJm+ycPTFP1ZKyBZv8Yy3SCgQUyDJbVSz07zf85iHJPlzHQWyDWYJeAv4Bdle5cTL/YRFtqnTPohsn4X8Gps8smeyw5kj+0q46MxOvI2NIpIkuVJHkQMCGjR4gzdo0OCn/JQb3DjXcyWU2OUtWlxnQIEuJQYn1MkWfRKeUaBLmzZddqEwyM6Q/oxsY1XjuaYbkwWxyeEgot7LxvVwg9U++P/nxgZeUFxfW9PO6/QK8kJCWk8pV1NKJViK4XqSFeQdTjiaKCAryBts8DEfU6M28dGshJCYCgkhj9ngIW+fqSD/GO7zdeM6XHuWtdpPyMbGjERk6c5ON2UV9XjDgrwKfAD855MenZK9aTwEdoE/T7as22RvXZAV46v4esrv1Q7DO+0hFmcpyAtJQj1NqR4U5JisJF+Nze/LCQm5znV+wk9YZpnb3Gb56It/rJiQR/yEH3mHmEJuxGJmQMIT7hOwwZHBQAHZu26Z7HTvp3CWs84h92jwB67xhJskfEJw3shO7JyRfSVcdGbPWpCv0hjkYUHeYIMVVviIj3iXd8/1XAPKPOV9dniLiIAW2Uv9ODtEPGOHPj1i9gh4CoUINoCPycrxC6deBmQt9j7Z1mXv4Ht4qbjCPeAPZAFKeOF03QXE9bU27byObVMTB7EM6UoKCcTPIKlAN4KdATxJsrE1bQ5PJVyNeF+elJQuXXYOxk316VOeICop4cGFCyHPgH0KZyrIrbRJHG1Cbyc7NLTNc+86A7Ldm+EAh5NP8w1PFbXJ3lq2GHMq6tnBg/bJ3hUmsHOwBMM3kqv+GrqoDd2kz1kGVtKUhGx1VMhW/fNF71U1zFyTJgEB22wTnbjJfVF21qbGHkWSUUE+fCMekNDjMdlx1+dGyw+vzOmQBeUMh4xSdojo0SOmRcI26XkjeybniOwr5yIye5bnuyrFOG/AgA4dypTZYefcY5BjyjxjgT2y96k2AYMTBoe1iOizy4AeMU1gG9Ioy8QO2a8vtNHhsdw+z78LvkRcs39vOMj4mKMPFxRXMZ28TmfkwwbwN5DuQbsI7SZ0OvD/xfD7Qda7/szhiYurF/OLFRPzLd8SEY3GbE1ykR4HV/OmBLRZYp9rL1xNP04r6rG3tQnNdrZidnjuTSM/d8QTDs8THfNcZOu4CjwGvmHMeKo2h+eVHjPRC6IL/EC2kR539OBVkqbp1I8wDx1Elj2yN4HhicXhbAWvuoiIv/AXWrQoU+YP/OHMY5BbrNDiGikBCQFpboOdkPKIZ8Rsc2SzN7wqZwB8ffDrGQY1RuyxxX2aNOkxYIf0vJE9k3NE9rV0kZm9ahISnvKUhIQKFTbZZInJZ+TJSwlpc40+i8QERBSP7JDm9RiwSYs2EX36JLQhTuBbsg3H8JDwCxfpfQ88IivJ3dGfvkRcszfY+2RvsMccHbyguGpCp+V1OgV5EXgX0jZEf4J24+BFFWavwzZXc8zoZUlI2GSTPn0KuY/TDTfKARE1+tTPVJAHcUSvuQedfrbCmjz3ppGf7mZ4Dfvx+mR710Wy07A/Mubir+jgC4Y77ZMs68Hi9TmcMedVd5Eb2oPI0iabYaZx8OdXcejKeSQkbLNNhw4hIVWqE+6UZlICIhYYsDCaTSY98opPadIipckLl60PIzW8YP4MV+bE9GmyQ4c+PRKaz89bM3lkz+QckX0tWY4PpaS0aBETExKyx96ZdkKPPlfAgCoxFVIKxJRO3NbFxHRoEzEgISZhAEmaTVLR5/CCuRfmR2ySvboPDxO8ZFyzf2+Hw2t1nyvIFxRXTei0vI4tyHfu3JnoH3nYe0ir3aLX6RJtxsSdlLQH3SR7qb0Kp8QvWkREm/ZoDuRJCnL2M81W8IA+A3qcZU6CQRqTJAeTpvXJDs2+sLUdXrY1fi0mHBbX4SmpY5dk+JTDd4UJ52qLOdyTv4oXkqVpSr/fp9VqUSwW+frrryea4gng888/n/jfmTSzvYcPabdadLpdNuOYTprS4/XY8YBs4z1gQI8eBQqjjfjkXw8xEQO65HdU888f0yM97oTsMCDDzJ1hi5sSZRt8EvqkLxPZM/yb54rslXcZmZ14G/vwIa1Wi263SxzHV2a4RUxMRERMTIECg3MeJsvm+u+TUDoye9NxEhIG9A7Kce4Vm43NOJwD+YWC3OW4sRAvEdej49aOWWUXENfX1kXkNRgXtH/6p3+aaH3dH9znq95XNKMmnbttOvc60IfCPgT9wxeBTjY8ajzZDULyDm8Wcpajxxx8VRwcJPeEaWiGj5tkLRZyv576XQyf8gxXhQ0fOvy8agqFAmEYEoYhi4uLVCqViY443b9/f+IXxaSZHdy/T++rr4iaTe6229zrdOjzeo0xHeZt+HFWaS40x/3QU5KjcyC/uADnOGSfDejg4AYhLxnZM/7LZ47slXfRmZ14G3v/Pl999RXNZpN2u02nczXmOgiO+TjvMx3uiHLw+5MNc5fmtxbDwJy4CPk7ebz47OeM67E3CDnuYfaklzftvI49gvz48eOJFmon3qEX9RgMBsTNhPRgnHv8uryLTkFy2sb0ogzXUcxLpzN57lcdNZym6Sx3+DmrSTMb7+wQ9bLMNpNkdGnK6xTZmeTt6AKcKyxTjKxOcdGZnXgbu7ND7yCvV2lO5PTgYy6cM29T+nJdgmnndWxB/uabbyZ6kv1kn1bcIkoi4t04u9rndToXJ82JSTOb7O8Tt1okUcRuHBtZaQYm3sbu79NqtYiiiDg2pdJlGDvEolQqTbTrl6TJ4Z7i8JSCpKlI03Ti85KTZjY9uDU8aWpkpSmbNLMTb2Ov2BzI0lVyUl7HFuQgCEyjNGNnKchmVpq9STNrXqXZOymv3vpbkiRJyrEgS5IkSTkWZEmSJCnHgixJkiTlWJAlSZKkHAuyJEmSlGNBliRJknIsyJIkSVKOBVmSJEnKsSBLkiRJORZkSZIkKSdIU28FL0mSJA15BFmSJEnKsSBLkiRJORZkSZIkKceCLEmSJOVYkCVJkqQcC7IkSZKU8/8DJK0ydWj7XvQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x489.6 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6.8))\n",
    "for row in range(2):\n",
    "    for col in range(3):\n",
    "        plt.subplot(2, 3, row * 3 + col + 1)\n",
    "        plot_observation(trajectories.observation[row, col].numpy())\n",
    "plt.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0, wspace=0.02)\n",
    "save_fig(\"sub_episodes_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = replay_buffer.as_dataset(\n",
    "    sample_batch_size=64,\n",
    "    num_steps=2,\n",
    "    num_parallel_calls=3).prefetch(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the main functions to TF Functions for better performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf_agents.utils.common import function\n",
    "\n",
    "collect_driver.run = function(collect_driver.run)\n",
    "agent.train = function(agent.train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we are ready to run the main loop!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_agent(n_iterations):\n",
    "    time_step = None\n",
    "    policy_state = agent.collect_policy.get_initial_state(tf_env.batch_size)\n",
    "    iterator = iter(dataset)\n",
    "    for iteration in range(n_iterations):\n",
    "        time_step, policy_state = collect_driver.run(time_step, policy_state)\n",
    "        trajectories, buffer_info = next(iterator)\n",
    "        train_loss = agent.train(trajectories)\n",
    "        print(\"\\r{} loss:{:.5f}\".format(\n",
    "            iteration, train_loss.loss.numpy()), end=\"\")\n",
    "        if iteration % 1000 == 0:\n",
    "            log_metrics(train_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next cell to train the agent for 50,000 steps. Then look at its behavior by running the following cell. You can run these two cells as many times as you wish. The agent will keep improving! It will likely take over 200,000 iterations for the agent to become reasonably good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: calling foldr_v2 (from tensorflow.python.ops.functional_ops) with back_prop=False is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "back_prop=False is deprecated. Consider using tf.stop_gradient instead.\n",
      "Instead of:\n",
      "results = tf.foldr(fn, elems, back_prop=False)\n",
      "Use:\n",
      "results = tf.nest.map_structure(tf.stop_gradient, tf.foldr(fn, elems))\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201: calling foldr_v2 (from tensorflow.python.ops.functional_ops) with back_prop=False is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "back_prop=False is deprecated. Consider using tf.stop_gradient instead.\n",
      "Instead of:\n",
      "results = tf.foldr(fn, elems, back_prop=False)\n",
      "Use:\n",
      "results = tf.nest.map_structure(tf.stop_gradient, tf.foldr(fn, elems))\n",
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 0\n",
      "\t\t EnvironmentSteps = 4\n",
      "\t\t AverageReturn = 0.0\n",
      "\t\t AverageEpisodeLength = 0.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "998 loss:0.00008"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 24\n",
      "\t\t EnvironmentSteps = 4004\n",
      "\t\t AverageReturn = 1.7000000476837158\n",
      "\t\t AverageEpisodeLength = 184.1999969482422\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1998 loss:0.00181"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 48\n",
      "\t\t EnvironmentSteps = 8004\n",
      "\t\t AverageReturn = 1.7000000476837158\n",
      "\t\t AverageEpisodeLength = 182.39999389648438\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2998 loss:0.00005"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 73\n",
      "\t\t EnvironmentSteps = 12004\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<<244 more lines>>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t\t NumberOfEpisodes = 1003\n",
      "\t\t EnvironmentSteps = 176004\n",
      "\t\t AverageReturn = 5.099999904632568\n",
      "\t\t AverageEpisodeLength = 246.5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44998 loss:0.00165"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 1019\n",
      "\t\t EnvironmentSteps = 180004\n",
      "\t\t AverageReturn = 5.199999809265137\n",
      "\t\t AverageEpisodeLength = 256.6000061035156\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45998 loss:0.00136"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 1035\n",
      "\t\t EnvironmentSteps = 184004\n",
      "\t\t AverageReturn = 4.599999904632568\n",
      "\t\t AverageEpisodeLength = 252.1999969482422\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46998 loss:0.00100"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 1050\n",
      "\t\t EnvironmentSteps = 188004\n",
      "\t\t AverageReturn = 5.699999809265137\n",
      "\t\t AverageEpisodeLength = 276.5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47998 loss:0.00116"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 1063\n",
      "\t\t EnvironmentSteps = 192004\n",
      "\t\t AverageReturn = 5.900000095367432\n",
      "\t\t AverageEpisodeLength = 296.3999938964844\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48998 loss:0.00049"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl: \n",
      "\t\t NumberOfEpisodes = 1077\n",
      "\t\t EnvironmentSteps = 196004\n",
      "\t\t AverageReturn = 7.800000190734863\n",
      "\t\t AverageEpisodeLength = 308.29998779296875\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49999 loss:0.00073"
     ]
    }
   ],
   "source": [
    "train_agent(n_iterations=50000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = []\n",
    "def save_frames(trajectory):\n",
    "    global frames\n",
    "    frames.append(tf_env.pyenv.envs[0].render(mode=\"rgb_array\"))\n",
    "\n",
    "watch_driver = DynamicStepDriver(\n",
    "    tf_env,\n",
    "    agent.policy,\n",
    "    observers=[save_frames, ShowProgress(1000)],\n",
    "    num_steps=1000)\n",
    "final_time_step, final_policy_state = watch_driver.run()\n",
    "\n",
    "plot_animation(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to save an animated GIF to show off your agent to your friends, here's one way to do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "import PIL\n",
    "\n",
    "image_path = os.path.join(\"images\", \"rl\", \"breakout.gif\")\n",
    "frame_images = [PIL.Image.fromarray(frame) for frame in frames[:150]]\n",
    "frame_images[0].save(image_path, format='GIF',\n",
    "                     append_images=frame_images[1:],\n",
    "                     save_all=True,\n",
    "                     duration=30,\n",
    "                     loop=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"images/rl/breakout.gif\" />\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<img src=\"images/rl/breakout.gif\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra material"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deque vs Rotating List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `deque` class offers fast append, but fairly slow random access (for large replay memories):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[121958, 671155, 131932, 365838, 259178]"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import deque\n",
    "np.random.seed(42)\n",
    "\n",
    "mem = deque(maxlen=1000000)\n",
    "for i in range(1000000):\n",
    "    mem.append(i)\n",
    "[mem[i] for i in np.random.randint(1000000, size=5)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47.4 ns ± 3.02 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mem.append(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "182 µs ± 6.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit [mem[i] for i in np.random.randint(1000000, size=5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you could use a rotating list like this `ReplayMemory` class. This would make random access faster for large replay memories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReplayMemory:\n",
    "    def __init__(self, max_size):\n",
    "        self.buffer = np.empty(max_size, dtype=np.object)\n",
    "        self.max_size = max_size\n",
    "        self.index = 0\n",
    "        self.size = 0\n",
    "\n",
    "    def append(self, obj):\n",
    "        self.buffer[self.index] = obj\n",
    "        self.size = min(self.size + 1, self.max_size)\n",
    "        self.index = (self.index + 1) % self.max_size\n",
    "\n",
    "    def sample(self, batch_size):\n",
    "        indices = np.random.randint(self.size, size=batch_size)\n",
    "        return self.buffer[indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([757386, 904203, 190588, 595754, 865356], dtype=object)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mem = ReplayMemory(max_size=1000000)\n",
    "for i in range(1000000):\n",
    "    mem.append(i)\n",
    "mem.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "519 ns ± 17.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mem.append(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.24 µs ± 227 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mem.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Custom TF-Agents Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a custom TF-Agent environment, you just need to write a class that inherits from the `PyEnvironment` class and implements a few methods. For example, the following minimal environment represents a simple 4x4 grid. The agent starts in one corner (0,0) and must move to the opposite corner (3,3). The episode is done if the agent reaches the goal (it gets a +10 reward) or if the agent goes out of bounds (-1 reward). The actions are up (0), down (1), left (2) and right (3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyEnvironment(tf_agents.environments.py_environment.PyEnvironment):\n",
    "    def __init__(self, discount=1.0):\n",
    "        super().__init__()\n",
    "        self._action_spec = tf_agents.specs.BoundedArraySpec(\n",
    "            shape=(), dtype=np.int32, name=\"action\", minimum=0, maximum=3)\n",
    "        self._observation_spec = tf_agents.specs.BoundedArraySpec(\n",
    "            shape=(4, 4), dtype=np.int32, name=\"observation\", minimum=0, maximum=1)\n",
    "        self.discount = discount\n",
    "\n",
    "    def action_spec(self):\n",
    "        return self._action_spec\n",
    "\n",
    "    def observation_spec(self):\n",
    "        return self._observation_spec\n",
    "\n",
    "    def _reset(self):\n",
    "        self._state = np.zeros(2, dtype=np.int32)\n",
    "        obs = np.zeros((4, 4), dtype=np.int32)\n",
    "        obs[self._state[0], self._state[1]] = 1\n",
    "        return tf_agents.trajectories.time_step.restart(obs)\n",
    "\n",
    "    def _step(self, action):\n",
    "        self._state += [(-1, 0), (+1, 0), (0, -1), (0, +1)][action]\n",
    "        reward = 0\n",
    "        obs = np.zeros((4, 4), dtype=np.int32)\n",
    "        done = (self._state.min() < 0 or self._state.max() > 3)\n",
    "        if not done:\n",
    "            obs[self._state[0], self._state[1]] = 1\n",
    "        if done or np.all(self._state == np.array([3, 3])):\n",
    "            reward = -1 if done else +10\n",
    "            return tf_agents.trajectories.time_step.termination(obs, reward)\n",
    "        else:\n",
    "            return tf_agents.trajectories.time_step.transition(obs, reward,\n",
    "                                                               self.discount)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The action and observation specs will generally be instances of the `ArraySpec` or `BoundedArraySpec` classes from the `tf_agents.specs` package (check out the other specs in this package as well). Optionally, you can also define a `render()` method, a `close()` method to free resources, as well as a `time_step_spec()` method if you don't want the `reward` and `discount` to be 32-bit float scalars. Note that the base class takes care of keeping track of the current time step, which is why we must implement `_reset()` and `_step()` rather than `reset()` and `step()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(0, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[1, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 0, 0]], dtype=int32))"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_env = MyEnvironment()\n",
    "time_step = my_env.reset()\n",
    "time_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TimeStep(step_type=array(1, dtype=int32), reward=array(0., dtype=float32), discount=array(1., dtype=float32), observation=array([[0, 0, 0, 0],\n",
       "       [1, 0, 0, 0],\n",
       "       [0, 0, 0, 0],\n",
       "       [0, 0, 0, 0]], dtype=int32))"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_step = my_env.step(1)\n",
    "time_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 7.\n",
    "\n",
    "See Appendix A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.\n",
    "_Exercise: Use policy gradients to solve OpenAI Gym's LunarLander-v2 environment. You will need to install the Box2D dependencies (`python3 -m pip install -U gym[box2d]`)._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.\n",
    "_Exercise: Use TF-Agents to train an agent that can achieve a superhuman level at SpaceInvaders-v4 using any of the available algorithms._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please follow the steps in the [Using TF-Agents to Beat Breakout](http://localhost:8888/notebooks/18_reinforcement_learning.ipynb#Using-TF-Agents-to-Beat-Breakout) section above, replacing `\"Breakout-v4\"` with `\"SpaceInvaders-v4\"`. There will be a few things to tweak, however. For example, the Space Invaders game does not require the user to press FIRE to begin the game. Instead, the player's laser cannon blinks for a few seconds then the game starts automatically. For better performance, you may want to skip this blinking phase (which lasts about 40 steps) at the beginning of each episode and after each life lost. Indeed, it's impossible to do anything at all during this phase, and nothing moves. One way to do this is to use the following custom environment wrapper, instead of the `AtariPreprocessingWithAutoFire` wrapper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AtariPreprocessingWithSkipStart(AtariPreprocessing):\n",
    "    def skip_frames(self, num_skip):\n",
    "        for _ in range(num_skip):\n",
    "          super().step(0) # NOOP for num_skip steps\n",
    "    def reset(self, **kwargs):\n",
    "        obs = super().reset(**kwargs)\n",
    "        self.skip_frames(40)\n",
    "        return obs\n",
    "    def step(self, action):\n",
    "        lives_before_action = self.ale.lives()\n",
    "        obs, rewards, done, info = super().step(action)\n",
    "        if self.ale.lives() < lives_before_action and not done:\n",
    "            self.skip_frames(40)\n",
    "        return obs, rewards, done, info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moreover, you should always ensure that the preprocessed images contain enough information to play the game. For example, the blasts from the laser cannon and from the aliens should still be visible despite the limited resolution. In this particular case, the preprocessing we did for Breakout still works fine for Space Invaders, but that's something you should always check if you want try other games. To do this, you can let the agent play randomly for a while, and record the preprocessed frames, then play the animation and ensure the game still looks playable.\n",
    "\n",
    "You will also need to let the agent train for quite a long time to get good performance. Sadly, the DQN algorithm is not able to reach superhuman level on Space Invaders, likely because humans are able to learn efficient long term strategies in this game, whereas DQN can only master fairly short strategies. But there has been a lot of progress over the past few years, and now many other RL algorithms are able to surpass human experts at this game. Check out the [State-of-the-Art for Space Invaders on paperswithcode.com](https://paperswithcode.com/sota/atari-games-on-atari-2600-space-invaders)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.\n",
    "_Exercise: If you have about $100 to spare, you can purchase a Raspberry Pi 3 plus some cheap robotics components, install TensorFlow on the Pi, and go wild! For an example, check out this [fun post](https://homl.info/2) by Lukas Biewald, or take a look at GoPiGo or BrickPi. Start with simple goals, like making the robot turn around to find the brightest angle (if it has a light sensor) or the closest object (if it has a sonar sensor), and move in that direction. Then you can start using Deep Learning: for example, if the robot has a camera, you can try to implement an object detection algorithm so it detects people and moves toward them. You can also try to use RL to make the agent learn on its own how to use the motors to achieve that goal. Have fun!_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's your turn now: go crazy, be creative, but most of all, be patient and move forward step by step, you can do it!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}