docs_src/callbacks.tracker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tracking Callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "from fastai.gen_doc.nbdoc import *\n",
    "from fastai.vision import *\n",
    "from fastai.callbacks import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This module regroups the callbacks that track one of the metrics computed at the end of each epoch to take some decision about training. To show examples of use, we'll use our sample of MNIST and a simple cnn model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.MNIST_SAMPLE)\n",
    "data = ImageDataBunch.from_folder(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"TerminateOnNaNCallback\" class=\"doc_header\"><code>class</code> <code>TerminateOnNaNCallback</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L10\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#TerminateOnNaNCallback-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h2>\n",
       "\n",
       "> <code>TerminateOnNaNCallback</code>() :: [`Callback`](/callback.html#Callback)\n",
       "\n",
       "<div class=\"collapse\" id=\"TerminateOnNaNCallback-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#TerminateOnNaNCallback-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>TerminateOnNaNCallback</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "A [`Callback`](/callback.html#Callback) that terminates training if loss is NaN.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TerminateOnNaNCallback)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, training diverges and the loss goes to nan. In that case, there's no point continuing, so this callback stops the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>0.495584</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = simple_cnn((3,16,16,2))\n",
    "learn = Learner(data, model, metrics=[accuracy])\n",
    "learn.fit_one_cycle(1,1e4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using it prevents that situation to happen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='0' class='' max='2', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      0.00% [0/2 00:00<00:00]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='0' class='progress-bar-interrupted' max='32', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      Interrupted\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch/Batch (0/5): Invalid loss, terminating training.\n",
      "Epoch/Batch (0/6): Invalid loss, terminating training.\n"
     ]
    }
   ],
   "source": [
    "model = simple_cnn((3,16,16,2))\n",
    "learn = Learner(data, model, metrics=[accuracy], callbacks=[TerminateOnNaNCallback()])\n",
    "learn.fit(2,1e4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TerminateOnNaNCallback.on_batch_end\" class=\"doc_header\"><code>on_batch_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L16\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#TerminateOnNaNCallback-on_batch_end-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_batch_end</code>(**`last_loss`**, **`epoch`**, **`num_batch`**, **\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"TerminateOnNaNCallback-on_batch_end-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#TerminateOnNaNCallback-on_batch_end-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_batch_end</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Test if `last_loss` is NaN and interrupts training.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TerminateOnNaNCallback.on_batch_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"Callback.on_epoch_end\" class=\"doc_header\"><code>on_epoch_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callback.py#L186\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#Callback-on_epoch_end-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_epoch_end</code>(**\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"Callback-on_epoch_end-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#Callback-on_epoch_end-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_epoch_end</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Called at the end of an epoch.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TerminateOnNaNCallback.on_epoch_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"EarlyStoppingCallback\" class=\"doc_header\"><code>class</code> <code>EarlyStoppingCallback</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L53\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#EarlyStoppingCallback-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h2>\n",
       "\n",
       "> <code>EarlyStoppingCallback</code>(**`learn`**:[`Learner`](/basic_train.html#Learner), **`monitor`**:`str`=***`'valid_loss'`***, **`mode`**:`str`=***`'auto'`***, **`min_delta`**:`int`=***`0`***, **`patience`**:`int`=***`0`***) :: [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback)\n",
       "\n",
       "<div class=\"collapse\" id=\"EarlyStoppingCallback-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#EarlyStoppingCallback-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>EarlyStoppingCallback</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "A [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback) that terminates training when monitored quantity stops improving.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(EarlyStoppingCallback)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This callback tracks the quantity in `monitor` during the training of `learn`. `mode` can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will stop training after `patience` epochs if the quantity hasn't improved by `min_delta`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='4' class='' max='50', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      8.00% [4/50 00:09<01:51]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.677729</td>\n",
       "      <td>0.677514</td>\n",
       "      <td>0.560353</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.679820</td>\n",
       "      <td>0.677514</td>\n",
       "      <td>0.560353</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.679434</td>\n",
       "      <td>0.677514</td>\n",
       "      <td>0.560353</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.679893</td>\n",
       "      <td>0.677514</td>\n",
       "      <td>0.560353</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='32' class='' max='32', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [32/32 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5: early stopping\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'bool' object has no attribute 'items'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-4384b4e1abc4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m learn = Learner(data, model, metrics=[accuracy], \n\u001b[1;32m      3\u001b[0m                 callback_fns=[partial(EarlyStoppingCallback, monitor='accuracy', min_delta=0.01, patience=3)])\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1e-42\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/fastai/fastai/basic_train.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, epochs, lr, wd, callbacks)\u001b[0m\n\u001b[1;32m    180\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mdefaults\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextra_callbacks\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mcallbacks\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mdefaults\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextra_callbacks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    181\u001b[0m         fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,\n\u001b[0;32m--> 182\u001b[0;31m             callbacks=self.callbacks+callbacks)\n\u001b[0m\u001b[1;32m    183\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    184\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcreate_opt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mFloats\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwd\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mFloats\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m->\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/fastai/fastai/basic_train.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(epochs, model, loss_func, opt, data, callbacks, metrics)\u001b[0m\n\u001b[1;32m     97\u001b[0m                                        cb_handler=cb_handler, pbar=pbar)\n\u001b[1;32m     98\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mval_loss\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m             \u001b[0;32mif\u001b[0m \u001b[0mcb_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_epoch_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    100\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m         \u001b[0mexception\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/fastai/fastai/callback.py\u001b[0m in \u001b[0;36mon_epoch_end\u001b[0;34m(self, val_loss)\u001b[0m\n\u001b[1;32m    287\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'last_metrics'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mval_loss\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mval_loss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    288\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'epoch'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 289\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'epoch_end'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcall_mets\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mval_loss\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    290\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'stop_training'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    291\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/fastai/fastai/callback.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, cb_name, call_mets, **kwargs)\u001b[0m\n\u001b[1;32m    222\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcall_mets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    223\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mmet\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmetrics\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_and_update\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmet\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcb_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 224\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mcb\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_and_update\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcb_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    225\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    226\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mset_dl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mDataLoader\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/fastai/fastai/callback.py\u001b[0m in \u001b[0;36m_call_and_update\u001b[0;34m(self, cb, cb_name, **kwargs)\u001b[0m\n\u001b[1;32m    213\u001b[0m         \u001b[0;34m\"Call `cb_name` on `cb` and update the inner state.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    214\u001b[0m         \u001b[0mnew\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mifnone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34mf'on_{cb_name}'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 215\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnew\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    216\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    217\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{k} isn't a valid key in the state of the callbacks.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'bool' object has no attribute 'items'"
     ]
    }
   ],
   "source": [
    "model = simple_cnn((3,16,16,2))\n",
    "learn = Learner(data, model, metrics=[accuracy], \n",
    "                callback_fns=[partial(EarlyStoppingCallback, monitor='accuracy', min_delta=0.01, patience=3)])\n",
    "learn.fit(50,1e-42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"EarlyStoppingCallback.on_train_begin\" class=\"doc_header\"><code>on_train_begin</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L60\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#EarlyStoppingCallback-on_train_begin-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_train_begin</code>(**\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"EarlyStoppingCallback-on_train_begin-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#EarlyStoppingCallback-on_train_begin-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_train_begin</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Initialize inner arguments.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(EarlyStoppingCallback.on_train_begin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"EarlyStoppingCallback.on_epoch_end\" class=\"doc_header\"><code>on_epoch_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L65\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#EarlyStoppingCallback-on_epoch_end-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_epoch_end</code>(**`epoch`**, **\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"EarlyStoppingCallback-on_epoch_end-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#EarlyStoppingCallback-on_epoch_end-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_epoch_end</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Compare the value monitored to its best score and maybe stop training.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(EarlyStoppingCallback.on_epoch_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"SaveModelCallback\" class=\"doc_header\"><code>class</code> <code>SaveModelCallback</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L77\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#SaveModelCallback-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h2>\n",
       "\n",
       "> <code>SaveModelCallback</code>(**`learn`**:[`Learner`](/basic_train.html#Learner), **`monitor`**:`str`=***`'valid_loss'`***, **`mode`**:`str`=***`'auto'`***, **`every`**:`str`=***`'improvement'`***, **`name`**:`str`=***`'bestmodel'`***) :: [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback)\n",
       "\n",
       "<div class=\"collapse\" id=\"SaveModelCallback-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#SaveModelCallback-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>SaveModelCallback</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "A [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback) that saves the model when monitored quantity is best.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(SaveModelCallback)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This callback tracks the quantity in `monitor` during the training of `learn`. `mode` can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will save the model in `name` whenever determined by `every` ('improvement' or 'epoch'). Loads the best model at the end of training is `every='improvement'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = simple_cnn((3,16,16,2))\n",
    "learn = Learner(data, model, metrics=[accuracy])\n",
    "learn.fit_one_cycle(5,1e-4, callbacks=[SaveModelCallback(learn, every='epoch', monitor='accuracy', name='model')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choosing `every='epoch'` saves an individual model at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ~/.fastai/data/mnist_sample/models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit_one_cycle(5,1e-4, callbacks=[SaveModelCallback(learn, every='improvement', monitor='accuracy', name='best')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choosing `every='improvement'` saves the single best model out of all epochs during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ~/.fastai/data/mnist_sample/models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"SaveModelCallback.on_epoch_end\" class=\"doc_header\"><code>on_epoch_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L92\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#SaveModelCallback-on_epoch_end-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_epoch_end</code>(**`epoch`**:`int`, **\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"SaveModelCallback-on_epoch_end-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#SaveModelCallback-on_epoch_end-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_epoch_end</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Compare the value monitored to its best score and maybe save the model.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(SaveModelCallback.on_epoch_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"SaveModelCallback.on_train_end\" class=\"doc_header\"><code>on_train_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L102\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#SaveModelCallback-on_train_end-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_train_end</code>(**\\*\\*`kwargs`**)\n",
       "\n",
       "<div class=\"collapse\" id=\"SaveModelCallback-on_train_end-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#SaveModelCallback-on_train_end-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_train_end</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Load the best model.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(SaveModelCallback.on_train_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"ReduceLROnPlateauCallback\" class=\"doc_header\"><code>class</code> <code>ReduceLROnPlateauCallback</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L107\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#ReduceLROnPlateauCallback-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h2>\n",
       "\n",
       "> <code>ReduceLROnPlateauCallback</code>(**`learn`**:[`Learner`](/basic_train.html#Learner), **`monitor`**:`str`=***`'valid_loss'`***, **`mode`**:`str`=***`'auto'`***, **`patience`**:`int`=***`0`***, **`factor`**:`float`=***`0.2`***, **`min_delta`**:`int`=***`0`***) :: [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback)\n",
       "\n",
       "<div class=\"collapse\" id=\"ReduceLROnPlateauCallback-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#ReduceLROnPlateauCallback-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>ReduceLROnPlateauCallback</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "A [`TrackerCallback`](/callbacks.tracker.html#TrackerCallback) that reduces learning rate when a metric has stopped improving.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(ReduceLROnPlateauCallback)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This callback tracks the quantity in `monitor` during the training of `learn`. `mode` can be forced to 'min' or 'max' but will automatically try to determine if the quantity should be the lowest possible (validation loss) or the highest possible (accuracy). Will reduce the learning rate by `factor` after `patience` epochs if the quantity hasn't improved by `min_delta`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"ReduceLROnPlateauCallback.on_train_begin\" class=\"doc_header\"><code>on_train_begin</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L115\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#ReduceLROnPlateauCallback-on_train_begin-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_train_begin</code>(**\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"ReduceLROnPlateauCallback-on_train_begin-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#ReduceLROnPlateauCallback-on_train_begin-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_train_begin</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Initialize inner arguments.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(ReduceLROnPlateauCallback.on_train_begin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"ReduceLROnPlateauCallback.on_epoch_end\" class=\"doc_header\"><code>on_epoch_end</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L120\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#ReduceLROnPlateauCallback-on_epoch_end-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_epoch_end</code>(**`epoch`**, **\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"ReduceLROnPlateauCallback-on_epoch_end-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#ReduceLROnPlateauCallback-on_epoch_end-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_epoch_end</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Compare the value monitored to its best and maybe reduce lr.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(ReduceLROnPlateauCallback.on_epoch_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h2 id=\"TrackerCallback\" class=\"doc_header\"><code>class</code> <code>TrackerCallback</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L23\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#TrackerCallback-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h2>\n",
       "\n",
       "> <code>TrackerCallback</code>(**`learn`**:[`Learner`](/basic_train.html#Learner), **`monitor`**:`str`=***`'valid_loss'`***, **`mode`**:`str`=***`'auto'`***) :: [`LearnerCallback`](/basic_train.html#LearnerCallback)\n",
       "\n",
       "<div class=\"collapse\" id=\"TrackerCallback-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#TrackerCallback-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>TrackerCallback</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "A [`LearnerCallback`](/basic_train.html#LearnerCallback) that keeps track of the best value in `monitor`.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TrackerCallback)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TrackerCallback.get_monitor_value\" class=\"doc_header\"><code>get_monitor_value</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L39\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#TrackerCallback-get_monitor_value-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>get_monitor_value</code>()\n",
       "\n",
       "<div class=\"collapse\" id=\"TrackerCallback-get_monitor_value-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#TrackerCallback-get_monitor_value-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>get_monitor_value</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Pick the monitored value.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TrackerCallback.get_monitor_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Callback methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't call these yourself - they're called by fastai's [`Callback`](/callback.html#Callback) system automatically to enable the class's functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"TrackerCallback.on_train_begin\" class=\"doc_header\"><code>on_train_begin</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/callbacks/tracker.py#L35\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#TrackerCallback-on_train_begin-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4>\n",
       "\n",
       "> <code>on_train_begin</code>(**\\*\\*`kwargs`**:`Any`)\n",
       "\n",
       "<div class=\"collapse\" id=\"TrackerCallback-on_train_begin-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#TrackerCallback-on_train_begin-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>on_train_begin</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div>\n",
       "\n",
       "Initializes the best value.  "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(TrackerCallback.on_train_begin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Undocumented Methods - Methods moved below this line will intentionally be hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Methods - Please document or move to the undocumented section"
   ]
  }
 ],
 "metadata": {
  "jekyll": {
   "keywords": "fastai",
   "summary": "Callbacks that take decisions depending on the evolution of metrics during training",
   "title": "callbacks.tracker"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}