Frozen_Lake_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands On #1 - Skating the Frozen Lake\n",
    "\n",
    "## Goal:\n",
    "* Get familiar with programming OpenAI Gym & it’s interfaces\n",
    "* Solve OpenAI Gym : Frozen Lake, given an optimum deterministic policy \n",
    "\n",
    "## Steps:\n",
    "1. Understand OpenAI Setup & interface\n",
    "2. Examine Random Policy Run\n",
    "3. Run the cells upto _5. Hands-on ToDo_\n",
    "4. Create aPolicy which captures the deterministic policy\n",
    "5. Use deterministic policy to skate to the goal\n",
    "6. Metrics that can be programmed:\n",
    " * Episodes before solve\n",
    " * Solved 0.78 over 100 consecutive runs\n",
    " * Max score over 1000 episodes\n",
    " * Avg over 1000 episodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Install the required packages\n",
    "\n",
    "* No esoteric requirements\n",
    "* You can run them without docker\n",
    "* pip install -r requirements.txt\n",
    "* Requirements\n",
    " * python 3.6, pytorch, openAI gym, numpy, matplotlib\n",
    " * anaconda is easier but not needed\n",
    " * Miniconda works fine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.0. Define imports\n",
    "\n",
    "python 3, numpy, matplotlib, torch, gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# General imports\n",
    "import gym\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "from collections import namedtuple, deque\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# torch imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Global Constants and other variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants Definitions\n",
    "BUFFER_SIZE = int(1e5)  # replay buffer size\n",
    "BATCH_SIZE = 64         # minibatch size\n",
    "GAMMA = 0.99            # discount factor\n",
    "TAU = 1e-3              # for soft update of target parameters\n",
    "LR = 5e-4               # learning rate \n",
    "UPDATE_EVERY = 4        # how often to update the network\n",
    "# Number of neurons in the layers of the Q Network\n",
    "FC1_UNITS = 16\n",
    "FC2_UNITS = 8\n",
    "FC3_UNITS = 4\n",
    "# Store models flag. Store during calibration runs and do not store during hyperparameter search\n",
    "STORE_MODELS = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Work Area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed : 0:00:10.002450\n"
     ]
    }
   ],
   "source": [
    "# Work area to quickly test utility functions\n",
    "import time\n",
    "from datetime import datetime, timedelta\n",
    "start_time = time.time()\n",
    "time.sleep(10)\n",
    "print('Elapsed : {}'.format(timedelta(seconds=time.time() - start_time)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/miniconda3/lib/python3.7/site-packages/gym/envs/registration.py:14: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.\n",
      "  result = entry_point.load(False)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('FrozenLake-v0')\n",
    "env.reset()\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State :\n",
      " 0  1  2  3\n",
      " 4  5  6  7\n",
      " 8  9 10 11\n",
      "12 13 14 15\n",
      "\n",
      " S  F  F  F\n",
      " F  H  F  H\n",
      " F  F  F  H\n",
      " H  F  F  G\n"
     ]
    }
   ],
   "source": [
    "print(\"State :\")\n",
    "print(\" 0  1  2  3\")\n",
    "print(\" 4  5  6  7\")\n",
    "print(\" 8  9 10 11\")\n",
    "print(\"12 13 14 15\")\n",
    "print()\n",
    "print(\" S  F  F  F\")\n",
    "print(\" F  H  F  H\")\n",
    "print(\" F  F  F  H\")\n",
    "print(\" H  F  F  G\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0 = Left, 1 = Down, 2 = Right, 3 = Up ]\n",
      "{0: [(0.3333333333333333, 0, 0.0, False), (0.3333333333333333, 0, 0.0, False), (0.3333333333333333, 4, 0.0, False)], 1: [(0.3333333333333333, 0, 0.0, False), (0.3333333333333333, 4, 0.0, False), (0.3333333333333333, 1, 0.0, False)], 2: [(0.3333333333333333, 4, 0.0, False), (0.3333333333333333, 1, 0.0, False), (0.3333333333333333, 0, 0.0, False)], 3: [(0.3333333333333333, 1, 0.0, False), (0.3333333333333333, 0, 0.0, False), (0.3333333333333333, 0, 0.0, False)]}\n",
      "\n",
      "{0: [(0.3333333333333333, 10, 0.0, False), (0.3333333333333333, 13, 0.0, False), (0.3333333333333333, 14, 0.0, False)], 1: [(0.3333333333333333, 13, 0.0, False), (0.3333333333333333, 14, 0.0, False), (0.3333333333333333, 15, 1.0, True)], 2: [(0.3333333333333333, 14, 0.0, False), (0.3333333333333333, 15, 1.0, True), (0.3333333333333333, 10, 0.0, False)], 3: [(0.3333333333333333, 15, 1.0, True), (0.3333333333333333, 10, 0.0, False), (0.3333333333333333, 13, 0.0, False)]}\n",
      "\n",
      "{0: [(1.0, 15, 0, True)], 1: [(1.0, 15, 0, True)], 2: [(1.0, 15, 0, True)], 3: [(1.0, 15, 0, True)]}\n"
     ]
    }
   ],
   "source": [
    "print('[ 0 = Left, 1 = Down, 2 = Right, 3 = Up ]')\n",
    "print(env.unwrapped.P[0])\n",
    "print()\n",
    "print(env.unwrapped.P[14])\n",
    "print()\n",
    "print(env.unwrapped.P[15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slippery Slope !\n",
    "#### It is very slippery. To start with, let us create an environment that is not slippery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gym.envs.registration import register\n",
    "register(\n",
    "    id='FrozenLakeNotSlippery-v0',\n",
    "    entry_point='gym.envs.toy_text:FrozenLakeEnv',\n",
    "    kwargs={'map_name' : '4x4', 'is_slippery': False},\n",
    "    max_episode_steps=100,\n",
    "    reward_threshold=0.78, # optimum = .8196\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/miniconda3/lib/python3.7/site-packages/gym/envs/registration.py:14: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.\n",
      "  result = entry_point.load(False)\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "env = gym.make('FrozenLakeNotSlippery-v0')\n",
    "env.reset()\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Examine the State and Action Spaces\n",
    "\n",
    "The state space is 16, with four actions [ 0 = Left, 1 = Down, 2 = Right, 3 = Up ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Discrete(16)\n",
      "Discrete(4)\n",
      "[0, 1, 2, 3]\n",
      "[ 0 = Left, 1 = Down, 2 = Right, 3 = Up ]\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space)\n",
    "print(env.action_space)\n",
    "act_space = [i for i in range(0,env.action_space.n)]\n",
    "print(act_space)\n",
    "# env.unwrapped.get_action_meanings() # AttributeError: 'FrozenLakeEnv' object has no attribute 'get_action_meanings'\n",
    "print('[ 0 = Left, 1 = Down, 2 = Right, 3 = Up ]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "States = 16\n",
      "Actions = 4\n",
      "{0: [(1.0, 0, 0.0, False)], 1: [(1.0, 4, 0.0, False)], 2: [(1.0, 1, 0.0, False)], 3: [(1.0, 0, 0.0, False)]}\n"
     ]
    }
   ],
   "source": [
    "#print(dir(env))\n",
    "#print(dir(env.unwrapped))\n",
    "print('States = {:d}'.format(env.unwrapped.nS))\n",
    "print('Actions = {:d}'.format(env.unwrapped.nA))\n",
    "print(env.unwrapped.P[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frozen Lake, but slippery no more !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Test the environment with Random Action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0 ]  ->  0  = [ 0 ] R =  0.0\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  1  = [ 4 ] R =  0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 4 ]  ->  0  = [ 4 ] R =  0.0\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 4 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  1  = [ 4 ] R =  0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 4 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  1  = [ 4 ] R =  0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 4 ]  ->  2  = [ 5 ] R =  0.0\n",
      "  (Right)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "End game! Reward:  0.0\n",
      "You lost :(\n",
      "\n",
      "[ 0 ]  ->  0  = [ 0 ] R =  0.0\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  3  = [ 0 ] R =  0.0\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  2  = [ 1 ] R =  0.0\n",
      "  (Right)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 1 ]  ->  0  = [ 0 ] R =  0.0\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  0  = [ 0 ] R =  0.0\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  0  = [ 0 ] R =  0.0\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 0 ]  ->  2  = [ 1 ] R =  0.0\n",
      "  (Right)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 1 ]  ->  1  = [ 5 ] R =  0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "End game! Reward:  0.0\n",
      "You lost :(\n",
      "\n",
      "[ 0 ]  ->  2  = [ 1 ] R =  0.0\n",
      "  (Right)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 1 ]  ->  3  = [ 1 ] R =  0.0\n",
      "  (Up)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 1 ]  ->  3  = [ 1 ] R =  0.0\n",
      "  (Up)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 1 ]  ->  2  = [ 2 ] R =  0.0\n",
      "  (Right)\n",
      "SF\u001b[41mF\u001b[0mF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 2 ]  ->  0  = [ 1 ] R =  0.0\n",
      "  (Left)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 1 ]  ->  1  = [ 5 ] R =  0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n",
      "End game! Reward:  0.0\n",
      "You lost :(\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i_episode in range(3):\n",
    "    state = env.reset()\n",
    "    while True:\n",
    "        action = env.action_space.sample()\n",
    "        next_state, reward, done, info = env.step(action)\n",
    "        print('[',state,']',' -> ', action,' = [',next_state,']', 'R = ',reward)\n",
    "        env.render()\n",
    "        if done:\n",
    "            print('End game! Reward: ', reward)\n",
    "            print('You won :)\\n') if reward > 0 else print('You lost :(\\n')\n",
    "            break\n",
    "        else:\n",
    "            state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Hands-On ToDo\n",
    "----\n",
    "#### Create a dictionary aPolicy that captures the deterministic policy from the picture below\n",
    "##### e.g. To start from State S (which is 0) and go down, the aPilocy={0:1,}\n",
    "#### Add more elements to the dictionary that captures the good path from Start to Goal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# A Deterministic Optimal Policy\n",
    "aPolicy = {}\n",
    "\n",
    "# [ 0 = Left, 1 = Down, 2 = Right, 3 = Up ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='Frozen_Lake_Policy.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Run the deterministic policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0 ]  ->  1  = [ 4 ] 0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 4 ]  ->  1  = [ 8 ] 0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "[ 8 ]  ->  2  = [ 9 ] 0.0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "[ 9 ]  ->  1  = [ 13 ] 0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "[ 13 ]  ->  2  = [ 14 ] 0.0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "[ 14 ]  ->  2  = [ 15 ] 1.0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "End game! Reward:  1.0\n",
      "You won :)\n",
      "\n",
      "[ 0 ]  ->  1  = [ 4 ] 0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "[ 4 ]  ->  1  = [ 8 ] 0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "[ 8 ]  ->  2  = [ 9 ] 0.0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "[ 9 ]  ->  1  = [ 13 ] 0.0\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "[ 13 ]  ->  2  = [ 14 ] 0.0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "[ 14 ]  ->  2  = [ 15 ] 1.0\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "End game! Reward:  1.0\n",
      "You won :)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i_episode in range(2): # Should be over in 6 steps, try for 2 episodes\n",
    "    state = env.reset()\n",
    "    while True:\n",
    "        policy_action = aPolicy.get(state,-1)\n",
    "        if policy_action == -1 :\n",
    "            action = env.action_space.sample()\n",
    "        else:\n",
    "            action = policy_action\n",
    "        next_state, reward, done, info = env.step(action)\n",
    "        print('[',state,']',' -> ', action,' = [',next_state,']', reward)\n",
    "        env.render()\n",
    "        if done:\n",
    "            print('End game! Reward: ', reward)\n",
    "            print('You won :)\\n') if reward > 0 else print('You lost :(\\n')\n",
    "            break\n",
    "        else:\n",
    "            state = next_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### That's All _Folks_ !"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}