ch22ScalingUpDoubleDeepQ.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chapter 22: Scaling Up Double Deep Q-Learning\n",
    "\n",
    "\n",
    "New Skills in This Chapter:\n",
    "\n",
    "• Playing an Atari game with and without the Baselines game wrapper\n",
    "\n",
    "• Creating a Q-network to train all Atari games\n",
    "\n",
    "• Defining A function to test any Atari game\n",
    "\n",
    "• Capturing a game episode with high scores in an Atari game"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3b25bf6",
   "metadata": {},
   "source": [
    "***\n",
    "*At DeepMind we have pioneered the combination of these approaches - deep\n",
    "reinforcement learning - to create the first artificial agents to achieve human-level\n",
    "performance across many challenging domains.*<br>\n",
    "***\n",
    "--DeepMind, 2016\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "117477d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.makedirs(\"files/ch22\", exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f7dbb5c",
   "metadata": {},
   "source": [
    "# 22.1. Get Started with the Seaquest Game"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b7aaab",
   "metadata": {},
   "source": [
    "## 22.1.1. The Seaquest Game in OpenAI Gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3667df7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "\n",
    "env1 = gym.make(\"Seaquest-v0\")\n",
    "env1.reset()\n",
    "env1.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "48ee5e38",
   "metadata": {},
   "outputs": [],
   "source": [
    "actions = env1.action_space\n",
    "print(f\"The action space for the Seaquest game is {actions}\")\n",
    "meanings = env1.env.get_action_meanings()\n",
    "print(f'''The meanings of the actions for the Seaquest game are\n",
    "      \\n {meanings}''')\n",
    "# Print out the observation space in this game\n",
    "obs_space = env1.observation_space\n",
    "print(f\"The observation space for Seaquest game {obs_space}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1401e695",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "num_actions1 = env1.action_space.n\n",
    "env1.reset()\n",
    "for _ in range(20):\n",
    "    action = np.random.choice(num_actions1)\n",
    "    obs1, reward, done, info = env1.step(action)\n",
    "plt.imshow(obs1)\n",
    "plt.show()\n",
    "env1.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "91bebcfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "env1.reset()\n",
    "env1.render()\n",
    "history = []\n",
    "while True:\n",
    "    action = np.random.choice(num_actions1)\n",
    "    obs1, reward, done, info = env1.step(action)\n",
    "    env1.render()\n",
    "    history.append([reward, done, info])\n",
    "    if len(history)>1:\n",
    "        if info[\"ale.lives\"]<history[-2][2][\"ale.lives\"]:\n",
    "            pprint(history[-10:])\n",
    "            break\n",
    "env1.close()        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "894f24ec",
   "metadata": {},
   "source": [
    "## 22.1.2. Seaquest with the Baselines Game Wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "adfa3a2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from baselines.common.atari_wrappers import make_atari\n",
    "from baselines.common.atari_wrappers import wrap_deepmind\n",
    "\n",
    "env1 = make_atari(\"SeaquestNoFrameskip-v4\")\n",
    "env1 = wrap_deepmind(env1, frame_stack=True, scale=True)\n",
    "obs1 = env1.reset()\n",
    "history = []\n",
    "while True:\n",
    "    action = env1.action_space.sample()\n",
    "    obs1, reward, done, info = env1.step(action)\n",
    "    history.append([reward, done, info])\n",
    "    env1.render()\n",
    "    if done:\n",
    "        pprint(history[-10:])\n",
    "        break\n",
    "env1.close()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e481351d",
   "metadata": {},
   "outputs": [],
   "source": [
    "env1.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20db9147",
   "metadata": {},
   "source": [
    "## 22.1.3. Preprocessed Seaquest Game Windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "02285e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "npobs1=np.array(obs1)\n",
    "for i in range(4):\n",
    "    plt.imshow(npobs1[:,:,i])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b349838",
   "metadata": {},
   "source": [
    "## 22.1.4. Subplots of Seaquest Game Windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e8ddf785",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch22util import seaquest_pixels\n",
    "\n",
    "seaquest_pixels()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3608b0ab",
   "metadata": {},
   "source": [
    "# 22.2. Get Started with Beam Rider"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f811cd",
   "metadata": {},
   "source": [
    "## 22.2.1. Beam Rider without the Game Wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2eaf5ed0",
   "metadata": {},
   "outputs": [],
   "source": [
    "env2 = gym.make(\"BeamRider-v0\")\n",
    "env2.reset()\n",
    "env2.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3e8bdd6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "actions = env2.action_space\n",
    "print(f\"The action space for Beam Rider is {actions}\")\n",
    "meanings = env2.env.get_action_meanings()\n",
    "print(f'''The meanings of the actions for Beam Rider are\n",
    "      \\n {meanings}''')\n",
    "obs_space = env2.observation_space\n",
    "print(f\"The observation space for Beam Rider is {obs_space}\")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f9ea5edc",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_actions2 = env2.action_space.n\n",
    "env2.reset()\n",
    "for _ in range(20):\n",
    "    action = np.random.choice(num_actions2)\n",
    "    obs2, reward, done, info = env2.step(action)\n",
    "plt.imshow(obs2)\n",
    "plt.show()\n",
    "env2.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b5540783",
   "metadata": {},
   "outputs": [],
   "source": [
    "env2.reset()\n",
    "env2.render()\n",
    "history = []\n",
    "while True:\n",
    "    action = np.random.choice(num_actions2)\n",
    "    obs2, reward, done, info = env2.step(action)\n",
    "    env2.render()\n",
    "    history.append([reward, done, info])\n",
    "    if len(history)>1:\n",
    "        if info[\"ale.lives\"]<history[-2][2][\"ale.lives\"]:\n",
    "            pprint(history[-10:])\n",
    "            break\n",
    "env2.close()        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead4e520",
   "metadata": {},
   "source": [
    "## 22.2.2. Beam Rider with the Baselines Game Wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f9370fe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "env2 = make_atari(\"BeamRiderNoFrameskip-v4\")\n",
    "env2 = wrap_deepmind(env2, frame_stack=True, scale=True)\n",
    "obs2 = env2.reset()\n",
    "history = []\n",
    "while True:\n",
    "    action = env2.action_space.sample()\n",
    "    obs2, reward, done, info = env2.step(action)\n",
    "    history.append([reward, done, info])\n",
    "    env2.render()\n",
    "    if done:\n",
    "        pprint(history[-10:])\n",
    "        break\n",
    "env2.close()        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcb36cda",
   "metadata": {},
   "source": [
    "As you can see, when the number of lives changes from 3 to 2, the variable done becomes True and the episode ends. Note that the reward is 7, not -1, but we can code it as -1 by using this line of code; you'll see it in the script for training later:\n",
    "\n",
    "```python\n",
    "    # Each time the agent loses a life, set Q to -1; important\n",
    "    new_Qs = Qs * (1 - dones) - dones\n",
    "```\n",
    "\n",
    "Run the following to close the game window:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3d9ac4ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "env2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51b2dd98",
   "metadata": {},
   "source": [
    "## 22.2.3. Preprocessed Beam Rider Game Windows\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "db9f0c8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "npobs2=np.array(obs2)\n",
    "for i in range(4):\n",
    "    plt.imshow(npobs2[:,:,i])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e873408",
   "metadata": {},
   "source": [
    "## 22.2.4. Subplots of Beam Rider Game Windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9e2950df",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch22util import beamrider_pixels\n",
    "\n",
    "beamrider_pixels() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7daa4284",
   "metadata": {},
   "source": [
    "# 22.3. Scaling Up the Double Deep Q-Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331792c8",
   "metadata": {},
   "source": [
    "## 22.3.1. Differences Among Atari Games\n",
    "\n",
    "To create a function that can be applied to any Atari game, we first need to understand the differences among Atari games. \n",
    "\n",
    "Obviously, the name of the game is different. But there is a pattern. For the four games we have seen so far, Breakout, Space Invaders, Seaquest, and Beam Rider, their environment names are the following:\n",
    "* BreakoutNoFrameskip-v4\n",
    "* SpaceInvadersNoFrameskip-v4\n",
    "* SeaquestNoFrameskip-v4\n",
    "* BeamRiderNoFrameskip-v4\n",
    "Therefore we can use this line of code \n",
    "\n",
    "```python\n",
    "f\"{name}NoFrameskip-v4\"\n",
    "```\n",
    "\n",
    "in the function to scale up the game environment. \n",
    "\n",
    "The number of actions is different in different games. For the four games Breakout, Space Invaders, Seaquest, and Beam Rider, the numbers of actions are 4, 6, 18, and 9, respectively. However, we can use the code:\n",
    "\n",
    "```python\n",
    "num_actions = env.action_space.n\n",
    "```\n",
    "\n",
    "in the function to retrieve the number of actions for each game automatically.  \n",
    "\n",
    "We can leave everything else in the training program the same."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0cd039c",
   "metadata": {},
   "source": [
    "## 22.3.2. A Generic Double Deep Q-Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "80bb9bf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_shape = (84, 84, 4,)\n",
    "def create_model(num_actions):\n",
    "    model=keras.models.Sequential()\n",
    "    model.add(keras.layers.Conv2D(filters=32,kernel_size=8,\n",
    "     strides=(4,4),activation=\"relu\",input_shape=input_shape))\n",
    "    model.add(keras.layers.Conv2D(filters=64,kernel_size=4,\n",
    "     strides=(2,2),activation=\"relu\"))\n",
    "    model.add(keras.layers.Conv2D(filters=64,kernel_size=3,\n",
    "     strides=(1,1),activation=\"relu\"))\n",
    "    model.add(keras.layers.Flatten())\n",
    "    model.add(keras.layers.Dense(512,activation=\"relu\"))\n",
    "    model.add(keras.layers.Dense(num_actions))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1bf59717",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=0.00025\n",
    "optimizer=keras.optimizers.Adam(learning_rate=lr,clipnorm=1)\n",
    "loss_function=keras.losses.Huber()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dc9ad6c",
   "metadata": {},
   "source": [
    "## 22.3.3. The Training Process for Any Atari Game "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "42041881",
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma=0.99 \n",
    "batch_size=32  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f6496cbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a replay buffer \n",
    "memory=deque(maxlen=50000)\n",
    "# Create a running rewards list \n",
    "running_rewards=deque(maxlen=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d209b751",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replay and update model parameters\n",
    "def update_Q(num_actions):\n",
    "    global dnn,target_dnn\n",
    "    dones,frames,new_frames,rewards,actions=gen_batch()\n",
    "    # update the Q table\n",
    "    preds = target_dnn.predict(new_frames, verbose=0)\n",
    "    Qs = rewards + gamma * tf.reduce_max(preds, axis=1)\n",
    "    # if done=1  reset Q to  -1; important\n",
    "    new_Qs = Qs * (1 - dones) - dones\n",
    "    # update model parameters\n",
    "    onehot = tf.one_hot(actions, num_actions)\n",
    "    with tf.GradientTape() as t:\n",
    "        Q_preds=dnn(frames)\n",
    "        # Calculate old Qs for the action taken\n",
    "        old_Qs=tf.reduce_sum(tf.multiply(Q_preds,onehot),axis=1)\n",
    "        # Calculate loss between new Qs and old Qs\n",
    "        loss=loss_function(new_Qs, old_Qs)\n",
    "    # Update using backpropagation\n",
    "    gs=t.gradient(loss,dnn.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(gs,dnn.trainable_variables)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "9520d717",
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_episode(num_actions,name):\n",
    "    global frame_count,env,dnn,target_dnn\n",
    "    # reset state and episode reward before each episode\n",
    "    state = np.array(env.reset())\n",
    "    episode_reward = 0    \n",
    "    # Allow 10,000 steps per episode\n",
    "    for timestep in range(1, 10001):\n",
    "        frame_count += 1\n",
    "        # Calculate current epsilon based on frame count\n",
    "        epsilon = max(0.1, 1 - frame_count * (1-0.1) /1000000)\n",
    "        # Use epsilon-greedy for exploration\n",
    "        if frame_count < epsilon_random_frames or \\\n",
    "            epsilon > np.random.rand(1)[0]:\n",
    "            # Take random action\n",
    "            action = np.random.choice(num_actions)\n",
    "        # Use exploitation\n",
    "        else:\n",
    "            state_tensor = tf.convert_to_tensor(state)\n",
    "            state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "            action_probs = dnn(state_tensor, training=False)\n",
    "            action = tf.argmax(action_probs[0]).numpy()\n",
    "        # Apply the sampled action in our environment\n",
    "        state_next, reward, done, _ = env.step(action)\n",
    "        state_next = np.array(state_next)\n",
    "        episode_reward += reward\n",
    "        # Change done to 1.0 or 0.0 to prevent error\n",
    "        if done==True:\n",
    "            done=1.0\n",
    "        else:\n",
    "            done=0.0\n",
    "        # Save actions and states in replay buffer\n",
    "        memory.append([state, state_next, action, reward, done])\n",
    "        # current state becomes the next state in next round\n",
    "        state = state_next\n",
    "        # Update Q once batch size is over 32\n",
    "        if len(memory) > batch_size and \\\n",
    "            frame_count % update_after_actions == 0:\n",
    "            update_Q(num_actions)\n",
    "        if frame_count % update_target_network == 0:\n",
    "            # update the the target network with new weights\n",
    "            target_dnn.set_weights(dnn.get_weights())\n",
    "            # Periodically save the model\n",
    "            dnn.save(f\"files/ch22/{name}.h5\")         \n",
    "        if done:\n",
    "            running_rewards.append(episode_reward)\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8590cd4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_atari(name):\n",
    "    global frame_count,env,num_actions,dnn,target_dnn\n",
    "    # Use the Baseline Atari environment\n",
    "    env = make_atari(f\"{name}NoFrameskip-v4\")\n",
    "    # Process and stack the frames\n",
    "    env = wrap_deepmind(env, frame_stack=True, scale=True)\n",
    "    num_actions = env.action_space.n\n",
    "    \n",
    "    # Network for training\n",
    "    dnn=create_model(num_actions)\n",
    "    # Network for predicting (target network)\n",
    "    target_dnn=create_model(num_actions)     \n",
    "    episode=0\n",
    "    frame_count=0\n",
    "    while True: \n",
    "        episode += 1\n",
    "        play_episode(num_actions,name)\n",
    "        running_reward = np.mean(np.array(running_rewards))\n",
    "        if episode%20==0:\n",
    "            # Log details\n",
    "            m=\"running reward: {:.2f} at episode {} and frame {}\"\n",
    "            print(m.format(running_reward,episode,frame_count))\n",
    "        if running_reward>20:\n",
    "            dnn.save(f\"files/ch22/{name}.h5\")\n",
    "            print(f\"solved at episode {episode}\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2558834a",
   "metadata": {},
   "source": [
    "# 22.4. Try It on Seaquest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "343be396",
   "metadata": {},
   "source": [
    "## 22.4.1. Train the Model in Seaquest\n",
    "The following line of code will train the agent in the Seaquest game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "82cb90d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch22util import train_atari\n",
    "\n",
    "train_atari(\"Seaquest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c854b27e",
   "metadata": {},
   "source": [
    "We first import the function *train_atari()* from the local model *ch22util* and call the function. We put the name of the game, *Seaquest*, as the argument to the function. The training takes a couple of days. But you can use a pre-trained model that I put on the book's GitHub repository, saved as *Seaquest.h5*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "619c3cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "reload1 = tf.keras.models.load_model(\"files/ch22/Seaquest.h5\")\n",
    "state = env1.reset()\n",
    "for i in range(4):\n",
    "    score = 0\n",
    "    for j in range(10000):\n",
    "        if np.random.rand(1)[0]<0.01:\n",
    "            action = np.random.choice(num_actions1)\n",
    "        else:\n",
    "            state_tensor = tf.convert_to_tensor(state)\n",
    "            state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "            action_probs = reload1(state_tensor, training=False)\n",
    "            action = tf.argmax(action_probs[0]).numpy()    \n",
    "        state, reward, done, info = env1.step(action)\n",
    "        score += reward\n",
    "        env1.render()\n",
    "        if done:\n",
    "            print(\"the score is\", score)\n",
    "            break\n",
    "env1.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3058e09d",
   "metadata": {},
   "source": [
    "## 22.4.2. Test the Average Score in Seaquest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c25703d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_atari(name):\n",
    "    reload = tf.keras.models.load_model(f\"files/ch22/{name}.h5\")\n",
    "    env = make_atari(f\"{name}NoFrameskip-v4\")\n",
    "    env = wrap_deepmind(env, frame_stack=True, scale=True)\n",
    "    scores = []\n",
    "    num_actions = env.action_space.n\n",
    "    for i in range(100):\n",
    "        state = env.reset()\n",
    "        score = 0\n",
    "        for j in range(10000):\n",
    "            if np.random.rand(1)[0]<0.01:\n",
    "                action = np.random.choice(num_actions)\n",
    "            else:\n",
    "                state_tensor = tf.convert_to_tensor(state)\n",
    "                state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "                action_probs = reload(state_tensor, training=False)\n",
    "                action = tf.argmax(action_probs[0]).numpy()    \n",
    "            state, reward, done, info = env.step(action)\n",
    "            score += reward\n",
    "            if done:\n",
    "                print(f\"the score in episode {i+1} is {score}\")\n",
    "                scores.append(score)\n",
    "                break\n",
    "    env.close()\n",
    "    print(f\"the average score is {np.array(scores).mean()}\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13b5fa7b",
   "metadata": {},
   "source": [
    "To test the trained model in Seaquest, we import the function from the local module and call it to test 100 episodes of the game, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2401436e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch22util import test_atari\n",
    "\n",
    "test_atari(\"Seaquest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2afdb60c",
   "metadata": {},
   "source": [
    "## 22.4.3. Animate Successful Episodes\n",
    "We'll highlight episodes where the agent performs well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c60f2169",
   "metadata": {},
   "outputs": [],
   "source": [
    "import imageio\n",
    "import pickle \n",
    "\n",
    "for i in range(20):\n",
    "    state = env1.reset()\n",
    "    frames = []\n",
    "    for j in range(10000):\n",
    "        if np.random.rand(1)[0]<0.01:\n",
    "            action = np.random.choice(num_actions1)\n",
    "        else:\n",
    "            state_tensor = tf.convert_to_tensor(state)\n",
    "            state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "            action_Qs = reload1(state_tensor, training=False)\n",
    "            action = tf.argmax(action_Qs[0]).numpy()    \n",
    "        state, reward, done, info = env1.step(action)\n",
    "        frames.append(env1.render(mode='rgb_array'))\n",
    "        if done:\n",
    "            pickle.dump(frames,\\\n",
    "                open(f'files/ch22/{name}{i+1}.p', 'wb'))\n",
    "            imageio.mimsave(f\"files/ch22/{name}{i+1}.gif\",\\\n",
    "                            frames, fps=240)\n",
    "            break\n",
    "env1.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d2b5e83",
   "metadata": {},
   "source": [
    "Go to the book's GitHub repository and download the file *Seaquest2.zip*. Unzip the file and save the unzipped file *Seaquest2.p* in the folder /Desktop/mla/files/ch22/. The convert it into an animation as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "bf77395c",
   "metadata": {},
   "outputs": [],
   "source": [
    "Seaquest2=pickle.load(open(\"files\\ch22\\Seaquest2.p\",\"rb\"))\n",
    "imageio.mimsave(\"files\\ch22\\seaqueste2.gif\",Seaquest2[::5],fps=24)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f1f0b10",
   "metadata": {},
   "source": [
    "<img src=\"https://gattonweb.uky.edu/faculty/lium/ml/Seaquest_episode2.gif\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "54e4afb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "plots=Seaquest2[::22]\n",
    "last=Seaquest2[-1].reshape(1,210,160,3)\n",
    "Seaquest_plots=np.concatenate([plots,last],axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5f4807fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,16),dpi=100)\n",
    "for i in range(25):\n",
    "    plt.subplot(5,5,i+1)\n",
    "    plt.imshow(Seaquest_plots[i])\n",
    "    plt.axis('off')\n",
    "plt.subplots_adjust(bottom=0.001,right=0.999,top=0.999,\n",
    "left=0.001, hspace=-0.1,wspace=0.1)\n",
    "plt.savefig(\"files/ch22/Seaquest_plots.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64a6ac9f",
   "metadata": {},
   "source": [
    "<img src=\"https://gattonweb.uky.edu/faculty/lium/ml/Seaquest_plots.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e948c524",
   "metadata": {},
   "source": [
    "# 22.5. Try It on Beam Rider"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de826de5",
   "metadata": {},
   "source": [
    "## 22.5.1. Train the Model in Beam Rider"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5af420a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch22util import train_atari\n",
    "\n",
    "train_atari(\"BeamRider\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3ec7816",
   "metadata": {},
   "source": [
    "The training takes a couple of days. You can use a pre-trained model, *BeamRider.h5*, that I put on the book's GitHub repository. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f9376630",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception ignored in: <function SimpleImageViewer.__del__ at 0x0000020A0BEDC5E0>\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\hlliu2\\Anaconda3\\envs\\MLA\\lib\\site-packages\\gym\\envs\\classic_control\\rendering.py\", line 369, in __del__\n",
      "    self.close()\n",
      "  File \"C:\\Users\\hlliu2\\Anaconda3\\envs\\MLA\\lib\\site-packages\\gym\\envs\\classic_control\\rendering.py\", line 365, in close\n",
      "    self.window.close()\n",
      "  File \"C:\\Users\\hlliu2\\Anaconda3\\envs\\MLA\\lib\\site-packages\\pyglet\\window\\win32\\__init__.py\", line 299, in close\n",
      "    super(Win32Window, self).close()\n",
      "  File \"C:\\Users\\hlliu2\\Anaconda3\\envs\\MLA\\lib\\site-packages\\pyglet\\window\\__init__.py\", line 823, in close\n",
      "    app.windows.remove(self)\n",
      "  File \"C:\\Users\\hlliu2\\Anaconda3\\envs\\MLA\\lib\\_weakrefset.py\", line 114, in remove\n",
      "    self.data.remove(ref(item))\n",
      "KeyError: <weakref at 0x0000020A0BE80400; to 'Win32Window' at 0x0000020A0B9F7610>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the score is 23.0\n",
      "the score is 7.0\n",
      "the score is 30.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import gym\n",
    "import tensorflow as  tf\n",
    "from baselines.common.atari_wrappers import make_atari\n",
    "from baselines.common.atari_wrappers import wrap_deepmind\n",
    "env2 = make_atari(\"BeamRiderNoFrameskip-v4\")\n",
    "env2 = wrap_deepmind(env2, frame_stack=True, scale=True)\n",
    "reload2 = tf.keras.models.load_model(\"files/ch22/BeamRider.h5\")\n",
    "num_actions2 = num_actions2 = env2.action_space.n\n",
    "state = env2.reset()\n",
    "for i in range(3):\n",
    "    score = 0\n",
    "    for j in range(10000):\n",
    "        if np.random.rand(1)[0]<0.01:\n",
    "            action = np.random.choice(num_actions2)\n",
    "        else:\n",
    "            state_tensor = tf.convert_to_tensor(state)\n",
    "            state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "            action_Qs = reload2(state_tensor, training=False)\n",
    "            action = tf.argmax(action_Qs[0]).numpy()    \n",
    "        state, reward, done, info = env2.step(action)\n",
    "        score += reward\n",
    "        env2.render()\n",
    "        if done:\n",
    "            print(\"the score is\", score)\n",
    "            break\n",
    "env2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45be5c50",
   "metadata": {},
   "source": [
    "## 22.5.2. The Average Score in Beam Rider"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "50beb801",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_atari(\"BeamRider\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "757fa83f",
   "metadata": {},
   "source": [
    "## 22.5.3. A Successful Episode in Beam Rider"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "89d209b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(15):\n",
    "    state = env2.reset()\n",
    "    frames = []\n",
    "    for j in range(10000):\n",
    "        if np.random.rand(1)[0]<0.01:\n",
    "            action = np.random.choice(num_actions2)\n",
    "        else:\n",
    "            state_tensor = tf.convert_to_tensor(state)\n",
    "            state_tensor = tf.expand_dims(state_tensor, 0)\n",
    "            action_Qs = reload2(state_tensor, training=False)\n",
    "            action = tf.argmax(action_Qs[0]).numpy()    \n",
    "        state, reward, done, info = env2.step(action)\n",
    "        frames.append(env2.render(mode='rgb_array'))\n",
    "        if done:\n",
    "            pickle.dump(frames,\\\n",
    "                open(f'files/ch22/{name}{i+1}.p', 'wb'))\n",
    "            imageio.mimsave(f\"files/ch22/{name}{i+1}.gif\",\\\n",
    "                            frames, fps=240)\n",
    "            break\n",
    "env2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d84c12",
   "metadata": {},
   "source": [
    "Go to the book's GitHub repository and download the file *BeamRider4.zip*. Unzip the file and save the unzipped file *BeamRider4.p* in the folder /Desktop/mla/files/ch22/. The convert it into an animation as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3676be4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "BeamRider4=pickle.load(open(\"files\\ch22\\BeamRider4.p\",\"rb\"))\n",
    "imageio.mimsave(\"files\\ch22\\BeamRider4.gif\",\\\n",
    "                BeamRider4[::5],fps=24)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a67a9da",
   "metadata": {},
   "source": [
    "<img src=\"https://gattonweb.uky.edu/faculty/lium/ml/BeamRider_episode4.gif\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7873bd9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "plots=BeamRider4[::73]\n",
    "last=BeamRider4[-1].reshape(1,210,160,3)\n",
    "BeamRider_plots=np.concatenate([plots,last],axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "de2066d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,16),dpi=100)\n",
    "for i in range(25):\n",
    "    plt.subplot(5,5,i+1)\n",
    "    plt.imshow(BeamRider_plots[i])\n",
    "    plt.axis('off')\n",
    "plt.subplots_adjust(bottom=0.001,right=0.999,top=0.999,\n",
    "left=0.001, hspace=-0.1,wspace=0.1)\n",
    "plt.savefig(\"files/ch22/BeamRider_plots.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d87e25",
   "metadata": {},
   "source": [
    "<img src=\"https://gattonweb.uky.edu/faculty/lium/ml/BeamRider_plots.jpg\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "6eeb5490",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}