notebooks/hugging_face_text_classification_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "# Please don't change the text below, it's for formatting the documentation.\n",
    "title: \"Text Classification with Hugging Face Transformers Tutorial\"\n",
    "description: \"Learn how to create a custom text classification model with Hugging Face Transformers.\"\n",
    "image: \"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/00-project-food-not-food-overview.png\"\n",
    "format: \n",
    "  html:\n",
    "    code-fold: false\n",
    "    page-layout: full\n",
    "jupyter: python3\n",
    "number-sections: true\n",
    "toc: true\n",
    "toc-depth: 3\n",
    "toc-expand: 2 # expand toc to multiple levels\n",
    "code-block-border-left: true\n",
    "code-block-bg: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/mrdbourke/learn-huggingface/blob/main/notebooks/hugging_face_text_classification_tutorial.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "[Source code on GitHub](https://github.com/mrdbourke/learn-huggingface/blob/main/notebooks/hugging_face_text_classification_tutorial.ipynb) | [Online book version](https://www.learnhuggingface.com/notebooks/hugging_face_text_classification_tutorial.html) | [Setup guide](https://www.learnhuggingface.com/extras/setup) | [Video Course](https://dbourke.link/ZTM-HF-Text-Classification) (step by step walkthrough)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Welcome to the Learn Hugging Face Text Classificaiton project!\n",
    "\n",
    "This tutorial is hands-on and focused on writing resuable code.\n",
    "\n",
    "We'll start with a text dataset, build a model to classify text samples and then share our model as a demo others can use.\n",
    "\n",
    "To do so, we'll be using a handful of helpful open-source tools from the Hugging Face ecosystem.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/00-project-food-not-food-overview.png\"\n",
    "     alt=\"\n",
    "Project overview image for 'Food Not Food' classification at Nutrify, a food app. The project involves building and deploying a binary text classification model to identify food-related text using Hugging Face Datasets, Transformers, and deploying with Hugging Face Hub/Spaces and Gradio. Examples include labels for 'A photo of sushi rolls on a white plate' (food), 'A serving of chicken curry in a blue bowl' (food), and 'A yellow tractor driving over a grassy hill' (not food). The process is visually depicted from data collection to model training and demo deployment.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>We're going to put on our internship hats and build a food not food text classification model using tools from the Hugging Face ecosystem.</figcaption>\n",
    "</figure>\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Feel to keep reading through the notebook but if you'd like to run the code yourself, be sure to go through the [setup guide](https://www.learnhuggingface.com/extras/setup) first.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What we're going to build\n",
    "\n",
    "We're going to be bulding a `food`/`not_food` **text classification model**. \n",
    "\n",
    "Given a piece of a text (such as an image caption), our model will be able to predict if it's about food or not.\n",
    "\n",
    "This is the same kind of model I use in my own work on [Nutrify](https://www.nutrify.app) (an app to help people learn about food).\n",
    "\n",
    "More specifically, we're going to follow the following steps:\n",
    "\n",
    "1. **[Data](https://huggingface.co/datasets/mrdbourke/learn_hf_food_not_food_image_captions): Problem defintion and dataset preparation** - Getting a dataset/setting up the problem space.\n",
    "2. **[Model](https://huggingface.co/mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased): Finding, training and evaluating a model** - Finding a text classification model suitable for our problem on Hugging Face and customizing it to our own dataset.\n",
    "3. **[Demo](https://huggingface.co/spaces/mrdbourke/learn_hf_food_not_food_text_classifier_demo): Creating a demo and put our model into the real world** - Sharing our trained model in a way others can access and use.\n",
    "\n",
    "By the end of this project, you'll have a trained model and [demo on Hugging Face](https://huggingface.co/spaces/mrdbourke/learn_hf_food_not_food_text_classifier_demo) you can share with others:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<iframe\n",
       "\tsrc=\"https://mrdbourke-learn-hf-food-not-food-text-classifier-demo.hf.space\"\n",
       "\tframeborder=\"0\"\n",
       "\twidth=\"850\"\n",
       "\theight=\"650\"\n",
       "></iframe>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML \n",
    "\n",
    "HTML(\"\"\"\n",
    "<iframe\n",
    "\tsrc=\"https://mrdbourke-learn-hf-food-not-food-text-classifier-demo.hf.space\"\n",
    "\tframeborder=\"0\"\n",
    "\twidth=\"850\"\n",
    "\theight=\"650\"\n",
    "></iframe>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Note this is a hands-on project, so we'll be focused on writing reusable code and building a model that can be used in the real world. If you are looking for explainers to the theory of what we're doing, I'll leave links in the extra-curriculum section.\n",
    "\n",
    "::: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is text classification?\n",
    "\n",
    "Text classification is the process of assigning a category to a piece of text.\n",
    "\n",
    "Where a category can be almost anything and a piece of text can be a word, phrase, sentence, paragraph or entire document.\n",
    "\n",
    "Example text classification problems include:\n",
    "\n",
    "| **Problem** | **Description** | **Problem Type** |\n",
    "|-----|-----|-----|\n",
    "| Spam/phishing email detection | Is an email spam or not spam? Or is it a phishing email or not? | Binary classification (one thing or another) |\n",
    "| Sentiment analysis | Is a piece of text positive, negative or neutral? Such as classifying product reviews into good/bad/neutral. | Multi-class classification (one thing from many) |\n",
    "| Language detection | What language is a piece of text written in? | Multi-class classification (one thing from many) |\n",
    "| Topic classification | What topic(s) does a news article belong to? | Multi-label classification (one or more things from many) |\n",
    "| Hate speech detection | Is a comment hateful or not hateful? | Binary classification (one thing or another) |\n",
    "| Product categorization | What categories does a product belong to? | Multi-label classification (one or more things from many) |\n",
    "| Business email classification | Which category should this email go to? | Multi-class classification (one thing from many) | \n",
    "\n",
    "Text classification is a very common problem in many business settings.\n",
    "\n",
    "For example, a project I've worked on previously as a machine learning engineer was building a text classification model to classify different insurance claims into `claimant_at_fault`/`claimant_not_at_fault` for a large insurance company. \n",
    "\n",
    "It turns out the deep learning-based model we built was very good (98%+ accuracy on the test dataset).\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/02-example-text-classification-workflow.png\"\n",
    "     alt=\"Diagram showing an example text classification workflow for insurance claims. It includes customer-submitted text such as 'Someone crashed into my car' and 'I accidentally crashed into someone's letterbox'. These texts are processed through an insurance app by Lime Insurance Co. and input into a model that determines fault based on the text. The classification outputs are 'Not at Fault' and 'At Fault', allowing claims to be forwarded to the appropriate department. The workflow is broken down into steps: Insurance claims, Insurance app, Model, and Classification outputs.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>An example text classification problem I once worked on to classify insurance claim texts into at fault or not fault. This result of the model would send the claim to a different department in the insurance company.</figcaption>\n",
    "</figure>\n",
    "\n",
    "Speaking of models, there are several different kinds of models you can use for text classification.\n",
    "\n",
    "And each will have its pros and cons depending on the problem you're working on.\n",
    "\n",
    "Example text classification models include:\n",
    "\n",
    "| **Model** | **Description** | **Pros** | **Cons** |\n",
    "|-----|-----|-----|-----|\n",
    "| Rule-based | Uses a set of rules to classify text (e.g. if text contains \"sad\" -> sentiment = low) | Simple, easy to understand | Requires manual creation of rules |\n",
    "| [Bag of Words](https://en.wikipedia.org/wiki/Bag-of-words_model) | Counts the frequency of words in a piece of text | Simple, easy to understand | Doesn't capture word order |\n",
    "| [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) | Weighs the importance of words in a piece of text | Simple, easy to understand | Doesn't capture word order |\n",
    "| Deep learning-based models | Uses neural networks to learn patterns in text | Can learn complex patterns at scale | Can require large amounts of data/compute power to run, not as easy to understand (can be hard to debug) |\n",
    "\n",
    "For our project, we're going to go with a deep learning model.\n",
    "\n",
    "Why?\n",
    "\n",
    "Because Hugging Face helps us do so.\n",
    "\n",
    "And in most cases, with a quality dataset, a deep learning model will often perform better than a rule-based or other model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why train your own text classification models?\n",
    "\n",
    "You can customize **pre-trained models** for text classification as well as API-powered models and LLMs such as GPT, Gemini, Claude or Mistral.\n",
    "\n",
    "Depending on your requirements, there are several pros and cons for using your own model versus using an API.\n",
    "\n",
    "Training/fine-tuning your own model:\n",
    "\n",
    "| Pros | Cons |\n",
    "| :----- | :----- |\n",
    "| **Control:** Full control over model lifecycle. | Can be complex to get setup. |\n",
    "| No usage limits (aside from compute constraints). | Requires dedicated compute resources for training/inference. |\n",
    "| Can train once and deploy everywhere/whenever you want (for example, Tesla deploying a model to all self-driving cars). | Requires maintenance over time to ensure performance remains up to par. |\n",
    "| **Privacy:** Data can be kept in-house/app and doesn’t need to go to a third party. | Can require longer development cycles compared to using existing APIs. |\n",
    "| **Speed:** Customizing a small model for a specific use case often means it runs much faster. | |\n",
    "\n",
    "Using a pre-built model API (e.g. GPT, Gemini, Claude, Mistral):\n",
    "\n",
    "| Pros | Cons |\n",
    "| :----- | :----- |\n",
    "| **Ease of use:** often can be setup within a few lines of code. | If the model API goes down, your service goes down. |\n",
    "| No maintenance of compute resources. | Data is required to be sent to a third-party for processing. |\n",
    "| Access to the most advanced models. | The API may have usage limits per day/time period. |\n",
    "| Can scale if usage increases. | Can be much slower than using dedicated models due to requiring an API call. |\n",
    "\n",
    "For this project, we're going to focus on fine-tuning our own model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow we're going to follow\n",
    "\n",
    "Our motto is *data, model, demo!*\n",
    "\n",
    "So we're going to follow the rough workflow of:\n",
    "\n",
    "1. Create and preprocess data.\n",
    "2. Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "I say rough because machine learning projects are often non-linear in nature.\n",
    "\n",
    "As in, because machine learning projects involve many experiments, they can kind of be all over the place.\n",
    "\n",
    "But this worfklow will give us some good guidelines to follow.\n",
    "\n",
    "<figure style=\"text-align: center; display: inline-block;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/01-hugging-face-workflow.png\"\n",
    "     alt=\"The diagram shows the Hugging Face model development workflow, which includes the following steps: start with an idea or problem, get data ready (turn into tensors/create data splits), pick a pretrained model (to suit your problem), train/fine-tune the model on your custom data, evaluate the model, improve through experimentation, save and upload the fine-tuned model to the Hugging Face Hub, and turn your model into a shareable demo. Tools used in this workflow are Datasets/Tokenizers, Transformers/PEFT/Accelerate/timm, Hub/Spaces/Gradio, and Evaluate.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption style=\"width: 100%; box-sizing: border-box;\">A general Hugging Face workflow from idea to shared model and demo using tools from the Hugging Face ecosystem. You'll notice some of the steps don't match with our workflow outline above. This is because the text-based workflow outline above breaks some of the steps down for educational purposes. These kind of workflows are not set in stone and are more of guide than specific directions. See information on each of the tools in the <a href=\"https://huggingface.co\">Hugging Face documentation</a>.</figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing necessary libraries\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "First, we'll import the required libraries.\n",
    "\n",
    "If you're running on your local computer, be sure to check out the getting [setup guide](https://www.learnhuggingface.com/extras/setup) to make sure you have everything you need.\n",
    "\n",
    "If you're using Google Colab, many of them the following libraries will be installed by default.\n",
    "\n",
    "However, we'll have to install a few extras to get everything working.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "If you're running on Google Colab, this notebook will work best with access to a GPU. To enable a GPU, go to `Runtime` ➡️ `Change runtime type` ➡️ `Hardware accelerator` ➡️ `GPU`.\n",
    "\n",
    "::: \n",
    "\n",
    "We'll need to install the following libraries from the Hugging Face ecosystem:\n",
    "\n",
    "* [`transformers`](https://huggingface.co/docs/transformers/en/installation) - comes pre-installed on Google Colab but if you're running on your local machine, you can install it via `pip install transformers`.\n",
    "* [`datasets`](https://huggingface.co/docs/datasets/installation) - a library for accessing and manipulating datasets on and off the Hugging Face Hub, you can install it via `pip install datasets`.\n",
    "* [`evaluate`](https://huggingface.co/docs/evaluate/installation) - a library for evaluating machine learning model performance with various metrics, you can install it via `pip install evaluate`.\n",
    "* [`accelerate`](https://huggingface.co/docs/accelerate/basic_tutorials/install) - a library for training machine learning models faster, you can install it via `pip install accelerate`.\n",
    "* [`gradio`](https://www.gradio.app/guides/quickstart#installation) - a library for creating interactive demos of machine learning models, you can install it via `pip install gradio`.\n",
    "\n",
    "We can also check the versions of our software with `package_name.__version__`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "collapsed": true,
    "id": "qJWCmI1rzOR6",
    "jupyter": {
     "outputs_hidden": true
    },
    "outputId": "ccd72531-249d-402f-91b0-7b701b2dc7eb",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using transformers version: 4.43.2\n",
      "Using datasets version: 2.20.0\n",
      "Using torch version: 2.4.0+cu121\n"
     ]
    }
   ],
   "source": [
    "# Install dependencies (this is mostly for Google Colab, as the other dependences are available by default in Colab)\n",
    "try:\n",
    "  import datasets, evaluate, accelerate\n",
    "  import gradio as gr\n",
    "except ModuleNotFoundError:\n",
    "  !pip install -U datasets evaluate accelerate gradio # -U stands for \"upgrade\" so we'll get the latest version by default\n",
    "  import datasets, evaluate, accelerate\n",
    "  import gradio as gr\n",
    "\n",
    "import random\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import transformers\n",
    "\n",
    "print(f\"Using transformers version: {transformers.__version__}\")\n",
    "print(f\"Using datasets version: {datasets.__version__}\")\n",
    "print(f\"Using torch version: {torch.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful, as long as your versions are the same or higher to the versions above, you should be able to run the code below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting a dataset\n",
    "\n",
    "Okay, now we're got the required libraries, let's get a dataset.\n",
    "\n",
    "Getting a dataset is one of the most important things a machine learning project.\n",
    "\n",
    "The dataset you often determines the type of model you use as well as the quality of the outputs of that model.\n",
    "\n",
    "Meaning, if you have a high quality dataset, chances are, your future model could also have high quality outputs.\n",
    "\n",
    "It also means if your dataset is of poor quality, your model will likely also have poor quality outputs.\n",
    "\n",
    "For a text classificaiton problem, your dataset will likely come in the form of text (e.g. a paragraph, sentence or phrase) and a label (e.g. what category the text belongs to).\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/learn-hf-food-not-food-text-classification-dataset.png\"\n",
    "     alt=\"A screenshot of a webpage from Hugging Face's datasets section, specifically showing the 'learn_hf_food_not_food_image_captions' dataset by user mrdourke. The page displays a dataset card with a dataset viewer for a text classification task. The viewer shows examples of captions with their corresponding labels, distinguishing between 'food' and 'not_food.' The screenshot includes interface elements like navigation menus, a search bar, and options to use or edit the dataset.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Our <a href=\"https://huggingface.co/datasets/mrdbourke/learn_hf_food_not_food_image_captions\"> food not food image caption dataset</a> on the Hugging Face Hub.</figcaption>\n",
    "</figure>\n",
    "\n",
    "In our case, our dataset comes in the form of a collection of synthetic image captions and their corresponding labels (food or not food).\n",
    "\n",
    "This is a dataset I've created earlier to help us practice building a text classification model.\n",
    "\n",
    "You can find it on Hugging Face under the name [`mrdbourke/learn_hf_food_not_food_image_captions`](https://huggingface.co/datasets/mrdbourke/learn_hf_food_not_food_image_captions).\n",
    "\n",
    "::: {.callout-tip}\n",
    "\n",
    "## Food Not Food Image Caption Dataset Creation\n",
    "\n",
    "You can see how the Food Not Food image caption dataset was created in the example [Google Colab notebook](https://colab.research.google.com/drive/14xr3KN_HINY5LjV0s2E-4i7v0o_XI3U8?usp=sharing).\n",
    "\n",
    "A Large Language Model (LLM) was asked to generate various image caption texts about food and not food.\n",
    "\n",
    "Getting another model to create data for a problem is known as **synthetic data generation** and is a very good way of bootstrapping towards creating a model.\n",
    "\n",
    "One workflow would be to use real data wherever possible and use synthetic data to boost when needed.\n",
    "\n",
    "Note that it's always advised to evaluate/test models on real-life data as opposed to synthetic data.\n",
    "   \n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where can you get more datasets?\n",
    "\n",
    "The are many different places you can get datasets for text-based problems.\n",
    "\n",
    "One of the best places is on the Hugging Face Hub, specifically [huggingface.co/datasets](https://huggingface.co/datasets).\n",
    "\n",
    "Here you can find many different kinds of problem specific data such as [text classification](https://huggingface.co/datasets?task_categories=task_categories:text-classification&sort=trending).\n",
    "\n",
    "There are also many more datasets available on [Kaggle Datasets](https://www.kaggle.com/datasets).\n",
    "\n",
    "And thanks to the power of LLMs (Large Language Models), you can also now create your own text classifications by generating samples (this is how I created the dataset for this project). \n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/03-places-to-get-text-datasets.png\"\n",
    "     alt=\"\n",
    "Side-by-side comparison of text classification datasets available on Hugging Face and Kaggle. The Hugging Face interface shows a variety of datasets categorized under 'Text Classification', while the Kaggle interface displays a search for text classification datasets, listing several options. Both platforms provide extensive datasets for machine learning and natural language processing projects. The image emphasizes that Hugging Face and Kaggle are valuable resources for obtaining datasets, depicted with respective platform screenshots and highlighted sections for text classification.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Hugging Face Datasets and Kaggle Datasets are two of the best places on the internet to find all kinds of different datasets. If you can't find an existing dataset related to your problem you can either use your own data or potentially generate synthetic data samples with an LLM. For more on synthetic data generation, see the <a href=\"https://developer.nvidia.com/blog/creating-synthetic-data-using-llama-3-1-405b\">Creating Synthetic Data</a> article by NVIDIA.</figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the dataset\n",
    "\n",
    "Once we've found/prepared a dataset on the Hugging Face Hub, we can use the Hugging Face [`datasets`](https://huggingface.co/docs/datasets/en/index) library to load it.\n",
    "\n",
    "To load a dataset we can use the [`datasets.load_dataset(path=NAME_OR_PATH_OF_DATASET)`](https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_dataset) function and pass it the name/path of the dataset we want to load.\n",
    "\n",
    "In our case, our dataset name is `mrdbourke/learn_hf_food_not_food_image_captions` (you can also change this for your own dataset).\n",
    "\n",
    "And since our dataset is hosted on Hugging Face, when we run the following code for the first time, it will download it.\n",
    "\n",
    "If your target dataset is quite large, this download may take a while.\n",
    "\n",
    "However, once the dataset is downloaded, subsequent reloads will be mush faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 533
    },
    "id": "PGt72DrtqRfM",
    "outputId": "2f45489c-02e8-4c03-bb7c-d73faa46c5b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 250\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the dataset from Hugging Face Hub\n",
    "dataset = datasets.load_dataset(path=\"mrdbourke/learn_hf_food_not_food_image_captions\")\n",
    "\n",
    "# Inspect the dataset\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset loaded!\n",
    "\n",
    "Looks like our dataset has two features, `text` and `label`.\n",
    "\n",
    "And 250 total rows (the number of examples in our dataset).\n",
    "\n",
    "We can check the column names with `dataset.column_names`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': ['text', 'label']}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What features are there?\n",
    "dataset.column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like our dataset comes with a `train` split already (the whole dataset).\n",
    "\n",
    "We can access the `train` split with `dataset[\"train\"]` (some datasets also come with built-in `\"test\"` splits too)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text', 'label'],\n",
       "    num_rows: 250\n",
       "})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access the training split\n",
    "dataset[\"train\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How about we check out a single sample?\n",
    "\n",
    "We can do so with indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'Creamy cauliflower curry with garlic naan, featuring tender cauliflower in a rich sauce with cream and spices, served with garlic naan bread.',\n",
       " 'label': 'food'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! We get back a dictionary with the keys `text` and `label`.\n",
    "\n",
    "The `text` key contains the text of the image caption and the `label` key contains the label (food or not food)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect random examples from the dataset\n",
    "\n",
    "At 250 total samples, our dataset isn't too large.\n",
    "\n",
    "So we could sit here and explore the samples one by one.\n",
    "\n",
    "But whenever I interact with a new dataset, I like to view a bunch of random examples and get a *feel* of the data.\n",
    "\n",
    "Doing so is inline with the data explorer's motto: *visualize, visualize, visualize!* \n",
    "\n",
    "As a rule of thumb, I like to view at least 20-100 random examples when interacting with a new dataset.\n",
    "\n",
    "Let's write some code to view 5 random indexes of our data and their corresponding text and labels at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Random samples from dataset:\n",
      "\n",
      "Text: Set of spatulas kept in a holder | Label: not_food\n",
      "Text: Mouthwatering paneer tikka masala, featuring juicy paneer in a rich tomato-based sauce, garnished with fresh coriander leaves. | Label: food\n",
      "Text: Pair of reading glasses left open on a book | Label: not_food\n",
      "Text: Set of board games stacked on a shelf | Label: not_food\n",
      "Text: Two handfuls of bananas in a fruit bowl with grapes on the side, the fruit bowl is blue | Label: food\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "random_indexs = random.sample(range(len(dataset[\"train\"])), 5)\n",
    "random_samples = dataset[\"train\"][random_indexs]\n",
    "\n",
    "print(f\"[INFO] Random samples from dataset:\\n\")\n",
    "for item in zip(random_samples[\"text\"], random_samples[\"label\"]):\n",
    "    print(f\"Text: {item[0]} | Label: {item[1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful! Looks like our data contains a mix of shorter and longer sentences (between 5 and 20 words) of texts about food and not food.\n",
    "\n",
    "We can get the unique labels in our dataset with [`dataset[\"train\"].unique(\"label\")`](https://huggingface.co/docs/datasets/v2.20.0/en/package_reference/main_classes#datasets.Dataset.unique)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['food', 'not_food']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get unique label values\n",
    "dataset[\"train\"].unique(\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If our dataset is small enough to fit into memory, we can count the number of different labels with Python's [`collections.Counter`](https://docs.python.org/3/library/collections.html#counter-objects) (a method for counting objects in an iterable or mapping)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'food': 125, 'not_food': 125})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check number of each label\n",
    "from collections import Counter\n",
    "\n",
    "Counter(dataset[\"train\"][\"label\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent, looks like our dataset is well balanced with 125 samples of food and 125 samples of not food.\n",
    "\n",
    "In a binary classification case, this is ideal.\n",
    "\n",
    "If the classes were dramatically unbalanced (e.g. 90% food and 10% not food) we might have to consider collecting/creating more data.\n",
    "\n",
    "But best to train a model and see how it goes before making any drastic dataset changes.\n",
    "\n",
    "Because our dataset is small, we could also inspect it via a pandas DataFrame (however, this may not be possible for extremely large datasets)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>A slice of pizza with a generous amount of shr...</td>\n",
       "      <td>food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pair of reading glasses left open on a book</td>\n",
       "      <td>not_food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>Telescope positioned on a balcony</td>\n",
       "      <td>not_food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>A close-up of a family playing a board game wi...</td>\n",
       "      <td>not_food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>Rich and spicy lamb rogan josh with yogurt gar...</td>\n",
       "      <td>food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>181</th>\n",
       "      <td>A steaming bowl of fiery chicken curry, infuse...</td>\n",
       "      <td>food</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>Pizza with a stuffed crust, oozing with cheese</td>\n",
       "      <td>food</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  text     label\n",
       "142  A slice of pizza with a generous amount of shr...      food\n",
       "6          Pair of reading glasses left open on a book  not_food\n",
       "97                   Telescope positioned on a balcony  not_food\n",
       "60   A close-up of a family playing a board game wi...  not_food\n",
       "112  Rich and spicy lamb rogan josh with yogurt gar...      food\n",
       "181  A steaming bowl of fiery chicken curry, infuse...      food\n",
       "197     Pizza with a stuffed crust, oozing with cheese      food"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Turn our dataset into a DataFrame and get a random sample\n",
    "food_not_food_df = pd.DataFrame(dataset[\"train\"])\n",
    "food_not_food_df.sample(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label\n",
       "food        125\n",
       "not_food    125\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the value counts of the label column\n",
    "food_not_food_df[\"label\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wXTxiYra4ahX"
   },
   "source": [
    "## Preparing data for text classification\n",
    "\n",
    "We've got our data ready but there are a few steps we'll need to take before we can model it.\n",
    "\n",
    "The main two being:\n",
    "\n",
    "1. **Tokenization** - turning our text into a numerical representation (machines prefer numbers rather than words), for example, `{\"a\": 0, \"b\": 1, \"c\": 2...}`.\n",
    "2. **Creating a train/test split** - right now our data is in a training split only but we'll create a test set to evaluate our model's performance.\n",
    "\n",
    "These don't necessarily have to be in order either.\n",
    "\n",
    "Before we get to them, let's create a small mapping from our labels to numbers.\n",
    "\n",
    "In the same way we need to tokenize our text into numerical representation, we also need to do the same for our labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a mapping from labels to numbers\n",
    "\n",
    "Our machine learning model will want to see all numbers (people do well with text, computers do well with numbers).\n",
    "\n",
    "This goes for text as well as label input.\n",
    "\n",
    "So let's create a mapping from our labels to numbers.\n",
    "\n",
    "Since we've only got a couple of labels (`\"food\"` and `\"not_food\"`), we can create a dictionary to map them to numbers, however, if you've got a fair few labels, you may want to make this mapping programmatically.\n",
    "\n",
    "We can use these dictionaries later on for our model training as well as evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label to ID mapping: {'not_food': 0, 'food': 1}\n",
      "ID to Label mapping: {0: 'not_food', 1: 'food'}\n"
     ]
    }
   ],
   "source": [
    "# Create mapping from id2label and label2id\n",
    "id2label = {0: \"not_food\", 1: \"food\"}\n",
    "label2id = {\"not_food\": 0, \"food\": 1}\n",
    "\n",
    "print(f\"Label to ID mapping: {label2id}\")\n",
    "print(f\"ID to Label mapping: {id2label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.callout-note}\n",
    "\n",
    "In a binary classification task (such as what we're working on), the positive class, in our case `\"food\"`, is usually given the label `1` and the negative class (`\"not_food\"`) is given the label `0`. \n",
    "\n",
    ":::\n",
    "\n",
    "Rather than hard-coding our label to ID maps, we can also create them programmatically from the dataset (this is helpful if you have many classes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label to ID mapping: {'not_food': 0, 'food': 1}\n",
      "ID to Label mapping: {0: 'not_food', 1: 'food'}\n"
     ]
    }
   ],
   "source": [
    "# Create mappings programmatically from dataset\n",
    "id2label = {idx: label for idx, label in enumerate(dataset[\"train\"].unique(\"label\")[::-1])} # reverse sort list to have \"not_food\" first\n",
    "label2id = {label: idx for idx, label in id2label.items()}\n",
    "\n",
    "print(f\"Label to ID mapping: {label2id}\")\n",
    "print(f\"ID to Label mapping: {id2label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our dictionary mappings created, we can update the labels of our dataset to be numeric.\n",
    "\n",
    "We can do this using the [`datasets.Dataset.map`](https://huggingface.co/docs/datasets/en/process#map) method and passing it a function to apply to each example.\n",
    "\n",
    "Let's create a small function which turns an example label into a number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156,
     "referenced_widgets": [
      "cf51cae23d2b4026a4f890069983da4c",
      "ca79aac952ca41c096241ad371c055fe",
      "0233456210f8469f85e648ba1c1f83ff",
      "c0028b925edf40ecb94ee219a26dfe36",
      "d4a7a1f7a10040ccb07704d845de6cd6",
      "1b61ce44fb554db2bb906fb39b41b7e0",
      "dcc6949983954dfb9b7f8bd0ad6cd6d8",
      "2afc2c8167ad4d15ab6a35c1d1f1f3cb",
      "070785d29788406db9369ce39e24b7cf",
      "b6cb26e5df334ac980005863a38581a1",
      "ae6d2d013c03448887d53f8a0760e77d"
     ]
    },
    "id": "sCX-iCcW2riv",
    "outputId": "eba840c3-7652-41a8-a121-546cb9a8147f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'This is a sentence about my favourite food: honey.', 'label': 1}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Turn labels into 0 or 1 (e.g. 0 for \"not_food\", 1 for \"food\")\n",
    "def map_labels_to_number(example):\n",
    "  example[\"label\"] = label2id[example[\"label\"]]\n",
    "  return example\n",
    "\n",
    "example_sample = {\"text\": \"This is a sentence about my favourite food: honey.\", \"label\": \"food\"}\n",
    "\n",
    "# Test the function\n",
    "map_labels_to_number(example_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like our function works!\n",
    "\n",
    "How about we map it to the whole dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': ['Creamy cauliflower curry with garlic naan, featuring tender cauliflower in a rich sauce with cream and spices, served with garlic naan bread.',\n",
       "  'Set of books stacked on a desk',\n",
       "  'Watching TV together, a family has their dog stretched out on the floor',\n",
       "  'Wooden dresser with a mirror reflecting the room',\n",
       "  'Lawn mower stored in a shed'],\n",
       " 'label': [1, 0, 0, 0, 0]}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Map our dataset labels to numbers\n",
    "dataset = dataset[\"train\"].map(map_labels_to_number)\n",
    "dataset[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Looks like our labels are all numerical now.\n",
    "\n",
    "We can check a few random samples using [`dataset.shuffle()`](https://huggingface.co/docs/datasets/en/process#shuffle) and indexing for the first few."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f5XEUYH_31IS",
    "outputId": "8089bfe3-2322-4bac-d2fd-89f626d0a1e6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': ['Set of oven mitts hanging on a hook',\n",
       "  'Set of cookie cutters collected in a jar',\n",
       "  'Pizza with a dessert twist, featuring a sweet Nutella base and fresh strawberries on top',\n",
       "  'Set of binoculars placed on a table',\n",
       "  'Two handfuls of bananas in a fruit bowl with grapes on the side, the fruit bowl is blue'],\n",
       " 'label': [0, 0, 1, 0, 1]}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shuffle the dataset and view the first 5 samples (will return different results each time) \n",
    "dataset.shuffle()[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the dataset into training and test sets\n",
    "\n",
    "Right now our dataset only has a training split.\n",
    "\n",
    "However, we'd like to create a test split so we can evaluate our model.\n",
    "\n",
    "In essence, our model will learn patterns (the relationship between text captions and their labels of food/not_food) on the training data.\n",
    "\n",
    "And we will evaluate those learned patterns on the test data.\n",
    "\n",
    "We can split our data using the [`datasets.Dataset.train_test_split`](https://huggingface.co/docs/datasets/v2.20.0/en/package_reference/main_classes#datasets.Dataset.train_test_split) method.\n",
    "\n",
    "We can use the `test_size` parameter to define the percentage of data we'd like to use in our test set (e.g. `test_size=0.2` would mean 20% of the data goes to the test set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "OO53Xx9l3c_l"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 200\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 50\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create train/test splits\n",
    "dataset = dataset.train_test_split(test_size=0.2, seed=42) # note: seed isn't needed, just here for reproducibility, without it you will get different splits each time you run the cell\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect!\n",
    "\n",
    "Our dataset has been split into 200 training examples and 50 testing examples.\n",
    "\n",
    "Let's visualize a few random examples to make sure they still look okay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lIVwtdEw4VHF",
    "outputId": "35f0da00-ab15-42e0-9d81-757ad3da1618"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Random sample from training dataset:\n",
      "Text: Set of dumbbells stacked in a gym\n",
      "Label: 0 (not_food)\n",
      "\n",
      "[INFO] Random sample from testing dataset:\n",
      "Text: Two handfuls of bananas in a fruit bowl with grapes on the side, the fruit bowl is blue\n",
      "Label: 1 (food)\n"
     ]
    }
   ],
   "source": [
    "random_idx_train = random.randint(0, len(dataset[\"train\"]))\n",
    "random_sample_train = dataset[\"train\"][random_idx_train]\n",
    "\n",
    "random_idx_test = random.randint(0, len(dataset[\"test\"]))\n",
    "random_sample_test = dataset[\"test\"][random_idx_test]\n",
    "\n",
    "print(f\"[INFO] Random sample from training dataset:\")\n",
    "print(f\"Text: {random_sample_train['text']}\\nLabel: {random_sample_train['label']} ({id2label[random_sample_train['label']]})\\n\")\n",
    "print(f\"[INFO] Random sample from testing dataset:\")\n",
    "print(f\"Text: {random_sample_test['text']}\\nLabel: {random_sample_test['label']} ({id2label[random_sample_test['label']]})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing text data\n",
    "\n",
    "Labels numericalized, dataset split, time to turn our text into numbers.\n",
    "\n",
    "How?\n",
    "\n",
    "**Tokenization**.\n",
    "\n",
    "What's tokenization?\n",
    "\n",
    "Tokenization is the process of converting a non-numerical data source into numbers.\n",
    "\n",
    "Why?\n",
    "\n",
    "Because machines (especially machine learning models) prefer numbers to human-style data.\n",
    "\n",
    "In the case of the text `\"I love pizza\"` a very simple method of tokenization might be to convert each word to a number.\n",
    "\n",
    "For example, `{\"I\": 0, \"love\": 1, \"pizza\": 2}`.\n",
    "\n",
    "However, for most modern machine learning models, the tokenization process is a bit more nuanced.\n",
    "\n",
    "For example, the text `\"I love pizza\"` might be tokenized into something more like `[101, 1045, 2293, 10733, 102]`.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/05-example-of-tokenization-with-openai-tokenizer.png\"\n",
    "     alt=\"Comparison showing tokenization of the text 'I love learning about machine learning and building my own models' using the OpenAI Tokenizer. The left image displays the text segmented into tokens with colors indicating different words, illustrating 'People like words.' The right image shows the same text converted into numerical token IDs, demonstrating 'Machines prefer numbers.' The text is split into 11 tokens, translating to numerical values that the machine can process, highlighting how natural language is transformed for machine understanding.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Although it may seem like you can type text directly to machine learning models, behind the scenes they are converting it to numbers first. This happens for all kinds of data being passed to machine learning models. It goes from its raw form (e.g. text, image, audio) and gets turned into a numerical representation (often called tokenization) before it is processed by the model. Exactly how data gets turned into numbers will often be different depending on the model. This example shows the use of <a href=\"https://platform.openai.com/tokenizer\">OpenAI's GPT-3.5 & GPT-4 tokenizer</a>.</figcaption>\n",
    "</figure>\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Depending on the model you use, the tokenization process could be different. \n",
    "\n",
    "For example, one model might turn `\"I love pizza\"` into `[40, 3021, 23317]`, where as another model might turn it into `[101, 1045, 2293, 10733, 102]`. \n",
    "\n",
    "To deal with this, Hugging Face models often pair models and tokenizers together by name. \n",
    "\n",
    "Such is the case with [`distilbert/distilbert-base-uncased`](https://huggingface.co/distilbert/distilbert-base-uncased) (there is a `tokenizer.json` file as well as a `tokenizer_config.json` file which contains all of the tokenizer implementation details). \n",
    "\n",
    "For more examples of tokenization, you can see OpenAI's [tokenization visualizer tool](https://platform.openai.com/tokenizer) as well as their open-source library [`tiktoken`](https://github.com/openai/tiktoken), Google also have an open-source tokenization library called [`sentencepiece`](https://github.com/google/sentencepiece), finally Hugging Face's [`tokenizers`](https://github.com/huggingface/tokenizers) library is also a great resource (this is what we'll be using behind the scenes).\n",
    "\n",
    ":::\n",
    "\n",
    "Many of the text-based models on Hugging Face come paired with their own tokenizer.\n",
    "\n",
    "For example, the [`distilbert/distilbert-base-uncased`](https://huggingface.co/distilbert/distilbert-base-uncased) model is paired with the `distilbert/distilbert-base-uncased` tokenizer.\n",
    "\n",
    "We can load the tokenizer for a given model using the [`transformers.AutoTokenizer.from_pretrained`](https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes) method and passing it the name of the model we'd like to use.\n",
    "\n",
    "The `transformers.AutoTokenizer` class is part of a series of Auto Classes (such as `AutoConfig`, `AutoModel`, `AutoProcessor`) which automatically loads the correct configuration settings for a given model ID.\n",
    "\n",
    "Let's load the tokenizer for the `distilbert/distilbert-base-uncased` model and see how it works.\n",
    "\n",
    ":::{.callout-note}\n",
    "\n",
    "Why use the `distilbert/distilbert-base-uncased` model?\n",
    "\n",
    "The short answer is that I've used it before and it works well (and fast) on various text classification tasks.\n",
    "\n",
    "It also performed well in the [original research paper](https://arxiv.org/abs/1910.01108) which introduced it.\n",
    "\n",
    "The longer answer is that Hugging Face has many available open-source models for many different problems available at [https://huggingface.co/models](https://huggingface.co/models).\n",
    "\n",
    "Navigating these models can take some practice.\n",
    "\n",
    "And several models may be suited for the same task (though with various tradeoffs such as size and speed).\n",
    "\n",
    "However, overtime and with adequate experimentation, you'll start to build an intuition on which models are good for which problems.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 201,
     "referenced_widgets": [
      "6cdf8319c7184ee5b107cea1718b03c6",
      "4ef091cf8c084af1acb95d62ffe1f0f2",
      "35205e647f814b97a2e977af48cc7a17",
      "0eaf7b7abd71413597e0533efa3feda2",
      "34edb9945bb54a69868d62ecd5b1ae56",
      "495cd338f7614f7a9a68fc3dc3916dcf",
      "dc17c88c62244e2f9a574ef895132a83",
      "8580f622bac8422a992b56b5f39b3693",
      "ca74fdefa7844f11b1d9052269efd585",
      "c56c3c1ff2a040b0952b5bac655b71dc",
      "3d5b841a59a3489ea5b4112182d4ae70",
      "f1fa4e09b42c4904b9be033d33943e05",
      "470f2248b2444869914197c7481e10ae",
      "c6cbefbc4f4d4cfe98400aa05370e3e7",
      "6a7b459551104d47a90527c885647830",
      "d0080133b35c40578eba5db18dc82db6",
      "54e1a02cf71743ec9da128ff6c882f34",
      "6429d7801ba14ed4861726bd7832ecee",
      "d0c86013f8194633be0ac27ca9b8876c",
      "7b91cb6b9b5e4ba891a6cfaab2d5948b",
      "4f8713c60a0947ae86ee9d1d4de8ab3a",
      "4b6f81607c31496a83d5964983fed168",
      "c4055bbdeb6c4833b4fb50970248a154",
      "31cf1f106af54f7986b4d2f462125cca",
      "1c5d45fa6fb14097bf36584ec7ef651c",
      "1aa8fdd087ce42bdb16ae19a38e79b27",
      "ef3b72dea05d404db610152cb17ba8f9",
      "506baa608274455eae12e54e1d7b7dd5",
      "103ff0dd04da40a69d85763bbf571490",
      "9ccbc146f4ad4382879b5d8b526cac4e",
      "fda99413adc8497cba92a9c6416f77ad",
      "80ef815be4ac4b64b3b3d99a6c43fadb",
      "d1cc110825034545a2532fc567b5325e",
      "ec74087b5c9d4e538675956af72ae138",
      "fe87ce116db84d5ba3d00e4a8c249dc1",
      "c45933bed8084b53b562621699870695",
      "72df691ec7784ae7b0542171a7789555",
      "4c92f17cff2944a9a2a49be56649d81e",
      "20a883d10ea945a9889429c5efb76a43",
      "2f44328dc8a04f84a59e26d08d343c77",
      "37cede645a5b48a9b89df987b9ef91e2",
      "13b1494b02194773a5fafe39b211c88a",
      "39f3c0ec5b4446cda93df7a999d4fd41",
      "bb7f6dabc3ad4779a87b211227040606"
     ]
    },
    "id": "EfFLh-Sw4ifl",
    "outputId": "f277b622-267d-4d06-b2cb-6bc204b2f62d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DistilBertTokenizerFast(name_or_path='distilbert/distilbert-base-uncased', vocab_size=30522, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={\n",
       "\t0: AddedToken(\"[PAD]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t100: AddedToken(\"[UNK]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t101: AddedToken(\"[CLS]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t102: AddedToken(\"[SEP]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "\t103: AddedToken(\"[MASK]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
       "}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=\"distilbert/distilbert-base-uncased\",\n",
    "                                          use_fast=True) # uses fast tokenization (backed by tokenziers library and implemented in Rust) by default, if not available will default to Python implementation\n",
    "\n",
    "tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "There's our tokenizer!\n",
    "\n",
    "It's an instance of the [`transformers.DistilBertTokenizerFast`](https://huggingface.co/docs/transformers/v4.41.3/en/model_doc/distilbert#transformers.DistilBertTokenizerFast) class.\n",
    "\n",
    "You can read more about it in the documentation.\n",
    "\n",
    "For now, let's try it out by passing it a string of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 1045, 2293, 10733, 102], 'attention_mask': [1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test out tokenizer\n",
    "tokenizer(\"I love pizza\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 1045, 2293, 10733, 999, 102], 'attention_mask': [1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try adding a \"!\" at the end\n",
    "tokenizer(\"I love pizza!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo!\n",
    "\n",
    "Our text gets turned into numbers (or tokens).\n",
    "\n",
    "Notice how with even a slight change in the text, the tokenizer produces different results?\n",
    "\n",
    "The `input_ids` are our tokens.\n",
    "\n",
    "And the `attention_mask` (in our case, all `[1, 1, 1, 1, 1, 1]`) is a mask which tells the model which tokens to use or not. \n",
    "\n",
    "Tokens with a mask value of `1` get used and tokens with a mask value of `0` get ignored.\n",
    "\n",
    "There are several attributes of the `tokenizer` we can explore.\n",
    "\n",
    "* `tokenizer.vocab` will return the vocabulary of the tokenizer or in other words, the unique words/word pieces the tokenizer is capable of converting into numbers.\n",
    "* `tokenizer.model_max_length` will return the maximum length of a sequence the tokenizer can process, pass anything longer than this and the sequence will be truncated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of tokenizer vocabulary: 30522\n",
      "Max tokenizer input sequence length: 512\n"
     ]
    }
   ],
   "source": [
    "# Get the length of the vocabulary \n",
    "length_of_tokenizer_vocab = len(tokenizer.vocab)\n",
    "print(f\"Length of tokenizer vocabulary: {length_of_tokenizer_vocab}\")\n",
    "\n",
    "# Get the maximum sequence length the tokenizer can handle\n",
    "max_tokenizer_input_sequence_length = tokenizer.model_max_length\n",
    "print(f\"Max tokenizer input sequence length: {max_tokenizer_input_sequence_length}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah, looks like our tokenizer has a vocabulary of `30,522` different words and word pieces.\n",
    "\n",
    "And it can handle a sequence length of up to `512` (any sequence longer than this will be automatically truncated from the end).\n",
    "\n",
    "Let's check out some of the vocab.\n",
    "\n",
    "Can I find my own name?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3817"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Does \"daniel\" occur in the vocab?\n",
    "tokenizer.vocab[\"daniel\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oooh, looks like my name is `3817` in the tokenizer's vocab.\n",
    "\n",
    "Can you find your own name? (note: there may be an error if the token doesn't exist, we'll get to this)\n",
    "\n",
    "How about \"pizza\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10733"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.vocab[\"pizza\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if a word doesn't exist in the vocab?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'akash'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[26], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mtokenizer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvocab\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43makash\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'akash'"
     ]
    }
   ],
   "source": [
    "tokenizer.vocab[\"akash\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dam, we get a `KeyError`.\n",
    "\n",
    "Not to worry, this is okay, since when calling the `tokenizer` on the word, it will automatically split the word into word pieces or subwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 9875, 4095, 102], 'attention_mask': [1, 1, 1, 1]}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(\"akash\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It works!\n",
    "\n",
    "We can check what word pieces `\"akash\"` got broken into with [`tokenizer.convert_ids_to_tokens(input_ids)`](https://huggingface.co/docs/transformers/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast.convert_ids_to_tokens)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]', 'aka', '##sh', '[SEP]']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens(tokenizer(\"akash\").input_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahhh, it seems `\"akash\"` was split into two tokens, `[\"aka\", \"##sh\"]`.\n",
    "\n",
    "The `\"##\"` at the start of `\"##sh\"` means that the sequence is part of a larger sequence. \n",
    "\n",
    "And the `\"[CLS]\"` and `\"[SEP]\"` tokens are special tokens indicating the start and end of a sequence.\n",
    "\n",
    "Now, since tokenizers can deal with any text, what if there was an unknown token?\n",
    "\n",
    "For example, rather than `\"pizza\"` someone used the pizza emoji 🍕?\n",
    "\n",
    "Let's try!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]', '[UNK]', '[SEP]']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try to tokenize an emoji\n",
    "tokenizer.convert_ids_to_tokens(tokenizer(\"🍕\").input_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahh, we get the special `\"[UNK]\"` token.\n",
    "\n",
    "This stands for \"unknown\".\n",
    "\n",
    "The combination of word pieces and `\"[UNK]\"` special token means that our `tokenizer` will be able to turn almost any text into numbers for our model.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Keep in mind that just because one tokenizer uses an unknown special token for a particular word or emoji (🍕) doesn't mean another will.\n",
    "\n",
    ":::\n",
    "\n",
    "Since the `tokenizer.vocab` is a Python dictionary, we can get a sample of the vocabulary using `tokenizer.vocab.items()`.\n",
    "\n",
    "How about we get the first 5?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('!', 999), ('\"', 1000), ('#', 1001), ('##!', 29612), ('##\"', 29613)]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the first 5 items in the tokenizer vocab\n",
    "sorted(tokenizer.vocab.items())[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's our `'!'` from before! Looks like the first five items are all related to punctuation points.\n",
    "\n",
    "How about a random sample of tokens?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('##vies', 25929),\n",
       " ('responsibility', 5368),\n",
       " ('##pm', 9737),\n",
       " ('persona', 16115),\n",
       " ('rhythm', 6348)]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "random.sample(sorted(tokenizer.vocab.items()), k=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a preprocessing function to tokenize text\n",
    "\n",
    "Rather than tokenizing our texts one by one, it's best practice to define a preprocessing function which does it for us.\n",
    "\n",
    "This process works regardless of whether you're working with text data or other kinds of data such as images or audio.\n",
    "\n",
    "::: {.callout-tip}\n",
    "## Turning data into numbers\n",
    "\n",
    "For any kind of machine learning workflow, an important first step is turning your input data into numbers.\n",
    "\n",
    "As machine learning models are algorithms which find patterns in numbers, before they can find patterns in your data (text, images, audio, tables) it must be numerically encoded first (e.g. tokenizing text).\n",
    "\n",
    "To help with this, `transformers` has an [`AutoProcessor`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoProcessor) class which can preprocess data in a specific format required for a paired model.\n",
    "\n",
    ":::\n",
    "\n",
    "To prepare our text data, let's create a preprocessing function to take in a dictionary which contains the key `\"text\"` which has the value of a target string (our data samples come in the form of dictionaries) and then returns the tokenized `\"text\"`.\n",
    "\n",
    "We'll set the following parameters in our `tokenizer`:\n",
    "\n",
    "* `padding=True` - This will make all the sequences in a batch the same length by padding shorter sequences with 0's until they equal the longest size in the batch. Why? If there are different size sequences in a batch, you can sometimes run into dimensionality issues.\n",
    "* `truncation=True` - This will shorten sequences longer than the model can handle to the model's max input size (e.g. if a sequence is 1000 long and the model can handle 512, it will be shortened to 512 via removing all tokens after 512).\n",
    "\n",
    "You can see more parameters available for the `tokenizer` in the [`transformers.PreTrainedTokenizer` documentation](https://huggingface.co/docs/transformers/en/main_classes/tokenizer#transformers.PreTrainedTokenizer).\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "For more on padding and truncation (two important concepts in sequence processing), I'd recommend reading the Hugging Face documentation on [Padding and Truncation](https://huggingface.co/docs/transformers/en/pad_truncation).\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_text(examples):\n",
    "    \"\"\"\n",
    "    Tokenize given example text and return the tokenized text.\n",
    "    \"\"\"\n",
    "    return tokenizer(examples[\"text\"],\n",
    "                     padding=True, # pad short sequences to longest sequence in the batch\n",
    "                     truncation=True) # truncate long sequences to the maximum length the model can handle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful!\n",
    "\n",
    "Now let's try it out on an example sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 1045, 2293, 10733, 102], 'attention_mask': [1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_sample_2 = {\"text\": \"I love pizza\", \"label\": 1}\n",
    "\n",
    "# Test the function\n",
    "tokenize_text(example_sample_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking good!\n",
    "\n",
    "How about we map our `tokenize_text` function to our whole `dataset`?\n",
    "\n",
    "We can do so with the [`datasets.Dataset.map` method](https://huggingface.co/docs/datasets/v2.20.0/en/package_reference/main_classes#datasets.Dataset.map).\n",
    "\n",
    "The `map` method allows us to apply a given function to all examples in a dataset.\n",
    "\n",
    "By setting `batched=True` we can apply the given function to batches of examples (many at a time) to speed up computation time.\n",
    "\n",
    "Let's create a `tokenized_dataset` object by calling `map` on our `dataset` and passing it our `tokenize_text` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 260,
     "referenced_widgets": [
      "985174675f354a9bb2580a70ad2b7625",
      "b9e83e2c1d2f4970a0905efde0055e3d",
      "8c3f85da75764955a572261460db619b",
      "80995fb6ed6f4557b1037dea2798b2fb",
      "016181d9f51a4abea666046fb95f2aee",
      "5b1fd51b695b4c81ab351ed5b1f92966",
      "a5076ac1d4ad4c09aa1e6f12449915af",
      "e2f32fe9433d458dbc278da438f27449",
      "64513b9e43254d5a9bbbffc9cb9a054c",
      "eb256ba48acd446bbe6b832c9558f7c6",
      "4ef76858d67f4ee6a99b305edb682a43",
      "9455512d944c418b87e4d1fe324619e6",
      "b93a51136d1146e08914db125d0259fb",
      "c74948f6d29e4e0e868a3ccf836d2377",
      "522f6c1e5e7340039b7bbf700704699f",
      "0673f25c77b04212bc2b63f56454a5d2",
      "054e759a27a442afa8c666fc3d4c3f2e",
      "e4633e912a57492497bc2fbfad02351c",
      "79144b8d1ec44b6bb3dcc381f7e829ce",
      "60e84d7134b646f6b291b0439d9ec890",
      "6294e85e97a8495bb18d11dff1991a79",
      "91ee20aae0474a4c90a26463f72bacf1"
     ]
    },
    "id": "KorM6An04xXH",
    "outputId": "d72b4325-b4de-4f65-f930-50007d45c8d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
       "        num_rows: 200\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'label', 'input_ids', 'attention_mask'],\n",
       "        num_rows: 50\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Map our tokenize_text function to the dataset\n",
    "tokenized_dataset = dataset.map(function=tokenize_text, \n",
    "                                batched=True, # set batched=True to operate across batches of examples rather than only single examples\n",
    "                                batch_size=1000) # defaults to 1000, can be increased if you have a large dataset\n",
    "\n",
    "tokenized_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset tokenized!\n",
    "\n",
    "Let's inspect a pair of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6GihQUpk5C2w",
    "outputId": "c1b0b958-b9a5-4274-e8c1-eaf0ac4cdf3a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Key: text\n",
      "Train sample: Set of headphones placed on a desk\n",
      "Test sample: A slice of pepperoni pizza with a layer of melted cheese\n",
      "\n",
      "[INFO] Key: label\n",
      "Train sample: 0\n",
      "Test sample: 1\n",
      "\n",
      "[INFO] Key: input_ids\n",
      "Train sample: [101, 2275, 1997, 2132, 19093, 2872, 2006, 1037, 4624, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Test sample: [101, 1037, 14704, 1997, 11565, 10698, 10733, 2007, 1037, 6741, 1997, 12501, 8808, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "\n",
      "[INFO] Key: attention_mask\n",
      "Train sample: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Test sample: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get two samples from the tokenized dataset\n",
    "train_tokenized_sample = tokenized_dataset[\"train\"][0]\n",
    "test_tokenized_sample = tokenized_dataset[\"test\"][0]\n",
    "\n",
    "for key in train_tokenized_sample.keys():\n",
    "    print(f\"[INFO] Key: {key}\")\n",
    "    print(f\"Train sample: {train_tokenized_sample[key]}\")\n",
    "    print(f\"Test sample: {test_tokenized_sample[key]}\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful! Our samples have been tokenized.\n",
    "\n",
    "Notice the zeroes on the end of the `inpud_ids` and `attention_mask` values.\n",
    "\n",
    "These are padding tokens to ensure that each sample has the same length as the longest sequence in a given batch.\n",
    "\n",
    "We can now use these tokenized samples later on in our model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization takeaways\n",
    "\n",
    "We've now seen and used tokenizers in practice.\n",
    "\n",
    "A few takeaways before we start to build a model:\n",
    "\n",
    "* Tokenizers are used to turn text (or other forms of data such as images and audio) into a numerical representation ready to be used with a machine learning model.\n",
    "* Many models reuse existing tokenizers and many models have their own specific tokenizer paired with them. Hugging Face's `transformers.AutoTokenizer`, `transformers.AutoProcessor` and `transformers.AutoModel` classes make it easy to pair tokenizers and models based on their name (e.g. `distilbert/distilbert-base-uncased`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EVb02xAu5I0x"
   },
   "source": [
    "## Setting up an evaluation metric\n",
    "\n",
    "Aside from training a model, one of the most important steps in machine learning is evaluating a model.\n",
    "\n",
    "To do, we can use evaluation metrics.\n",
    "\n",
    "An evaluation metric attempts to represent a model's performance in a single (or series) of numbers (note, I say \"attempts\" here because evaluation metrics are useful to guage performance but the real test of a machine learning model is in the real world).\n",
    "\n",
    "There are many different kinds of evaluation metrics for various problems.\n",
    "\n",
    "But since we're focused on text classification, we'll use [accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision#In_binary_classification) as our evaluation metric.\n",
    "\n",
    "A model which gets 99/100 predictions correct has an accuracy of 99%.\n",
    "\n",
    "$$\n",
    "\\text{Accuracy} = \\frac{\\text{correct classifications}}{\\text{all classifications}}\n",
    "$$\n",
    "\n",
    "For some projects, you may have a minimum standard of a metric.\n",
    "\n",
    "For example, when I worked on an insurance claim classification model, the clients required over 98% accuracy on the test dataset for it to be viable to use in production. \n",
    "\n",
    "If needed, we can craft these evaluation metrics ourselves.\n",
    "\n",
    "However, Hugging Face has a library called [`evaluate`](https://huggingface.co/docs/evaluate/en/index) which has various metrics built in ready to use.\n",
    "\n",
    "We can load a metric using `evaluate.load(\"METRIC_NAME\")`.\n",
    "\n",
    "Let's load in `\"accuracy\"` and build a function to measure accuracy by comparing arrays of predictions and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "90455578408b4f33bc9a9c732d431236",
      "a2e056a63fd34329a21330a4fc69429d",
      "347ce846f3564c58ac1eb26ae2f3adc4",
      "9a01bb7c8f5e4940ba2444fa63a3db6f",
      "b8beefe086284c7297004fcad9303617",
      "593b4aaa5c9e4e1898cc4d5064940770",
      "4df9f85ae9e943ceb3ed1a5d9e143f5c",
      "a20bdb762700449097d5010ce7d29921",
      "2cb043cfb0fc448e81a34710d65bb691",
      "b8d8eac2274a4040a4163663896753f7",
      "01f82250d2e14f198dd0f561969d54ff"
     ]
    },
    "id": "D06NRS1-5S_P",
    "outputId": "4a2b0f85-db47-4f0f-cbc3-fa148c19a646"
   },
   "outputs": [],
   "source": [
    "import evaluate\n",
    "import numpy as np\n",
    "from typing import Tuple\n",
    "\n",
    "accuracy_metric = evaluate.load(\"accuracy\")\n",
    "\n",
    "def compute_accuracy(predictions_and_labels: Tuple[np.array, np.array]):\n",
    "  \"\"\"\n",
    "  Computes the accuracy of a model by comparing the predictions and labels.\n",
    "  \"\"\"\n",
    "  predictions, labels = predictions_and_labels\n",
    "\n",
    "  # Get highest prediction probability of each prediction if predictions are probabilities\n",
    "  if len(predictions.shape) >= 2:\n",
    "    predictions = np.argmax(predictions, axis=1)\n",
    "\n",
    "  return accuracy_metric.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy function created!\n",
    "\n",
    "Now let's test it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy when all predictions are correct: {'accuracy': 1.0}\n",
      "Accuracy when one prediction is wrong: {'accuracy': 0.9}\n"
     ]
    }
   ],
   "source": [
    "# Create example list of predictions and labels\n",
    "example_predictions_all_correct = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n",
    "example_predictions_one_wrong = np.array([0, 0, 0, 0, 1, 0, 0, 0, 0, 0])\n",
    "example_labels = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n",
    "\n",
    "# Test the function\n",
    "print(f\"Accuracy when all predictions are correct: {compute_accuracy((example_predictions_all_correct, example_labels))}\")\n",
    "print(f\"Accuracy when one prediction is wrong: {compute_accuracy((example_predictions_one_wrong, example_labels))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent, our function works just as we'd like.\n",
    "\n",
    "When all predictions are correct, it scores 1.0 (or 100% accuracy) and when 9/10 predictions are correct, it returns 0.9 (or 90% accuracy).\n",
    "\n",
    "We can use this function during training and evaluation of our model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "38IFAZTl5-iI"
   },
   "source": [
    "## Setting up a model for training\n",
    "\n",
    "We've gone through the important steps of setting data up for training (and evaluation).\n",
    "\n",
    "Now let's prepare a model.\n",
    "\n",
    "We'll keep going through the following steps:\n",
    "\n",
    "1. ✅ Create and preprocess data.\n",
    "2. Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "Let's start by creating an instance of a model.\n",
    "\n",
    "Since we're working on text classification, we'll do so with `transformers.AutoModelForSequenceClassification` (where sequence classification means a sequence of something, e.g. our sequences of text).\n",
    "\n",
    "We can use the `from_pretrained()` method to instatiate a pretrained model from the Hugging Face Hub.\n",
    "\n",
    ":::{.callout-note}\n",
    "\n",
    "The \"pretrained\" in [`transformers.AutoModelForSequenceClassification.from_pretrained`](https://huggingface.co/docs/transformers/v4.42.0/en/model_doc/auto#transformers.AutoConfig.from_pretrained) means acquiring a model which has already been trained on a certain dataset.\n",
    "\n",
    "This is common practice in many machine learning projects and is known as **transfer learning**.\n",
    "\n",
    "The idea is to take an existing model which works well on a task similar to your target task and then **fine-tune** it to work even better on your target task.\n",
    "\n",
    "In our case, we're going to use the pretrained DistilBERT base model ([`distilbert/distilbert-base-uncased`](https://huggingface.co/distilbert/distilbert-base-uncased)) which has been trained on [many thousands of books](https://huggingface.co/datasets/bookcorpus/bookcorpus) as well as a version of the [English Wikipedia](https://huggingface.co/datasets/legacy-datasets/wikipedia) (millions of words).\n",
    "\n",
    "This training gives it a very good baseline representation of the patterns in language.\n",
    "\n",
    "We'll take this baseline representation of the patterns in language and adjust it slightly to focus specifically on predicting whether an image caption is about food or not (based on the words it contains).\n",
    "\n",
    "The main two benefits of using transfer learning are:\n",
    "\n",
    "1. Ability to get good results with smaller amounts of data (since the main representations are learned on a larger dataset, we only have to show the model a few examples of our specific problem).\n",
    "2. This process can be repeated acorss various domains and tasks. For example, you can take a computer vision model trained on millions of images and customize it to your own use case. Or an audio model trained on many different nature sounds and customize it specifically for birds.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/09-transfer-learning-example.png\"\n",
    "     alt=\"Diagram illustrating the concept of transfer learning in two domains: Computer Vision and Natural Language Processing (NLP). In the Computer Vision example, the model is pre-trained on the ImageNet dataset. The model learns patterns and weights from this dataset and is then applied to specific tasks such as identifying images of steak, sushi, and pizza. In the NLP example, the model is pre-trained on large text corpora, such as Wikipedia articles about transfer learning. The learned patterns and weights are then applied to specific tasks, such as classifying text about 'A yellow tractor driving over a grassy hill' as 'Not Food' and 'A photo of sushi rolls on a white plate' as 'Food'. The model learns from a similar problem space and applies the learned patterns to specific problems, improving accuracy and performance.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Transfer learning is the process of taking what one model has learned from a (typically large) dataset and applying them to your own custom dataset. This process can be replicated across domains such as computer vision, natural language processing and more.</figcaption>\n",
    "</figure>\n",
    "\n",
    "So when starting a new machine learning project, one of the first questions you should ask is: does an existing pretrained model similar to my task exist and can I fine-tune it for my own task?\n",
    "\n",
    "For an end-to-end example of transfer learning in PyTorch (another popular deep learning framework), see [PyTorch Transfer Learning](https://www.learnpytorch.io/06_pytorch_transfer_learning/).\n",
    "\n",
    ":::\n",
    "\n",
    "Time to setup our `model` instance.\n",
    "\n",
    "A few things to note:\n",
    "\n",
    "* We'll use [`transformers.AutoModelForSequenceClassification.from_pretrained`](https://huggingface.co/docs/transformers/v4.42.0/en/model_doc/auto#transformers.AutoConfig.from_pretrained), this will create the model architecture we specify with the `pretrained_model_name_or_path` parameter. \n",
    "* The `AutoModelForSequenceClassification` class comes with a classification head on top of our mdoel (so we can customize this to the number of classes we have with the `num_labels` parameter).\n",
    "* Using `from_pretrained` will also call the [`transformers.PretrainedConfig`](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/configuration#transformers.PretrainedConfig) class which will enable us to set `id2label` and `label2id` parameters for our fine-tuning task.\n",
    "\n",
    "Let's refresh what our `id2label` and `label2id` objects look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id2label: {0: 'not_food', 1: 'food'}\n",
      "label2id: {'not_food': 0, 'food': 1}\n"
     ]
    }
   ],
   "source": [
    "# Get id and label mappings\n",
    "print(f\"id2label: {id2label}\")\n",
    "print(f\"label2id: {label2id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful, we can pass these mappings to `transformers.AutoModelForSequenceClassification.from_pretrained`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105,
     "referenced_widgets": [
      "c3faf065f2d2435697fb730bf04ff1cb",
      "8fc36b3359d24eb987c4be9eaf5d4e95",
      "beb76be168f840d79c950efc4bdfc8d1",
      "63e64e93abca400293b7b7742bd94aaf",
      "d41ef095cc5b4c23b31236b501dcf8c1",
      "d6b2772488464e80b333332d6cd46eeb",
      "5616046a1d1347cfbbe82fa67b79af1a",
      "f8d378a142934482aa04a608ada9acdb",
      "b6c20dee590f40e5b2c96beefc1643e1",
      "b791095b95214d82af1e344a2b5898a2",
      "a3fa870efa0f4463b1e60fa106f782f1"
     ]
    },
    "id": "c5c6bxLV53D5",
    "outputId": "ed01beb4-3cf7-4209-eaea-444e5aa51c5b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    "\n",
    "# Setup model for fine-tuning with classification head (top layers of network)\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\n",
    "    pretrained_model_name_or_path=\"distilbert/distilbert-base-uncased\",\n",
    "    num_labels=2, # can customize this to the number of classes in your dataset\n",
    "    id2label=id2label, # mappings from class IDs to the class labels (for classification tasks)\n",
    "    label2id=label2id\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model created!\n",
    "\n",
    "You'll notice that a warning message gets displayed: \n",
    "\n",
    "> Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
    "> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
    "\n",
    "This is essentially saying \"hey, some of the layers in this model are newly initialized (with random patterns) and you should probably customize them to your own dataset\".\n",
    "\n",
    "This happens because we used the `AutoModelForSequenceClassification` class.\n",
    "\n",
    "Whilst the majority of the layers in our model have already learned patterns from a large corpus of text, the top layers (classifier layers) have been randomly setup so we can customize them on our own.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/07-learning-and-transfer-learning-paradigms.png\"\n",
    "     alt=\"The diagram illustrates learning and transfer learning paradigms, showing three stages: Pretraining, Feature Extraction, and Full Fine-tuning. In the Pretraining stage, a base model is trained on a large dataset, requiring most compute resources. In the Feature Extraction stage, the base model's parameters are frozen (indicated by a snowflake), and a custom dataset is used to produce custom outputs. In the Full Fine-tuning stage, trainable layers (indicated by flames) are added to the base model using a custom dataset to create customized outputs for specific tasks. The key indicates that flames represent trainable components and snowflakes represent frozen components. This method involves adding layers to a base model to customize it for specific tasks.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Various forms of training paradigms. Typically you'll start with a model that has been pretrained on a large dataset. For example, a base model could be one that has read all of Wikipedia + 1000's of books (like our DistilBert model) and has a good general representation of language data. This representation can then be tailored to a specific use case by customizing the outputs and adjusting the representation slightly by feeding it custom data. This process is often referred to as fine-tuning.</figcaption>\n",
    "</figure>\n",
    "\n",
    "Let's try and make a prediction with our model and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "DistilBertForSequenceClassification.forward() got an unexpected keyword argument 'text'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[40], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# Try and make a prediction with the loaded model (this will error)\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[43mmodel\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mtokenized_dataset\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtrain\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/learn_hf/lib/python3.11/site-packages/torch/nn/modules/module.py:1553\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1551\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1552\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1553\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/learn_hf/lib/python3.11/site-packages/torch/nn/modules/module.py:1562\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1557\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1558\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1559\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1560\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1561\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1562\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1564\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1565\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "\u001b[0;31mTypeError\u001b[0m: DistilBertForSequenceClassification.forward() got an unexpected keyword argument 'text'"
     ]
    }
   ],
   "source": [
    "# Try and make a prediction with the loaded model (this will error)\n",
    "model(**tokenized_dataset[\"train\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! We get an error.\n",
    "\n",
    "Not to worry, this is only because our model hasn't been trained on our own dataset yet.\n",
    "\n",
    "Let's take a look at the layers in our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DistilBertForSequenceClassification(\n",
       "  (distilbert): DistilBertModel(\n",
       "    (embeddings): Embeddings(\n",
       "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (transformer): Transformer(\n",
       "      (layer): ModuleList(\n",
       "        (0-5): 6 x TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (activation): GELUActivation()\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n",
       "  (classifier): Linear(in_features=768, out_features=2, bias=True)\n",
       "  (dropout): Dropout(p=0.2, inplace=False)\n",
       ")"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the model \n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that the model comes in 3 main parts (data flows through these sequentially):\n",
    "\n",
    "1. `embeddings` - This part of the model turns the input tokens into a learned representation. So rather than just a list of integers, the values become a learned representation. This learned representation comes from the base model learning how different words and word pieces relate to eachother thanks to its training data. The size of `(30522, 768)` means the `30,522` words in the vocabulary are all represented by vectors of size `768` (one word gets represented by 768 numbers, these are often not human interpretable).\n",
    "2. `transformer` - This is the main body of the model. There are several `TransformerBlock` layers stacked on top of each other. These layers attempt to learn a deeper representation of the data going through the model. A thorough breakdown of these layers is beyond the scope of this tutorial, however, for and in-depth guide on Transformer-based models, I'd recommend reading [*Transformers from scratch*](https://peterbloem.nl/blog/transformers) by Peter Bloem, going through [Andrej Karpathy's lecture on Transformers and their history](https://www.youtube.com/watch?v=XfpMkf4rD6E) or reading the original [*Attention is all you need*](https://arxiv.org/abs/1706.03762) paper (this is the paper that introduced the Transformer architecture).\n",
    "3. `classifier` - This is what is going to take the representation of the data and compress it into our number of target classes (notice `out_features=2`, this means that we'll get two output numbers, one for each of our classes).\n",
    "\n",
    "For more on the entire DistilBert architecture and its training setup, I'd recommend reading the [*DistilBert paper*](https://arxiv.org/abs/1910.01108) from the Hugging Face team.\n",
    "\n",
    "Rather than breakdown the model itself, we're focused on using it for a particular task (classifying text)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counting the parameters of our model\n",
    "\n",
    "Before we move into training, we can get another insight into our model by counting its number of parameters.\n",
    "\n",
    "Let's create a small function to count the number of trainable (these will update during training) and total parameters in our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'trainable_parameters': 66955010, 'total_parameters': 66955010}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def count_params(model):\n",
    "    \"\"\"\n",
    "    Count the parameters of a PyTorch model.\n",
    "    \"\"\"\n",
    "    trainable_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "    total_parameters = sum(p.numel() for p in model.parameters())\n",
    "\n",
    "    return {\"trainable_parameters\": trainable_parameters, \"total_parameters\": total_parameters}\n",
    "\n",
    "# Count the parameters of the model\n",
    "count_params(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Looks like our model has a total of 66,955,010 parameters and all of them are trainable.\n",
    "\n",
    "A parameter is a numerical value in a model which is capable of being updated to better represent the input data.\n",
    "\n",
    "I like to think of them as a small opportunity to learn patterns in the data.\n",
    "\n",
    "If a model has three parameters, it has three small opportunities to learn patterns in the data.\n",
    "\n",
    "Whereas, if a model has 60,000,000+ (60M) parameters (like our `model`), it has 60,000,000+ small opportunities to learn patterns in the data. \n",
    "\n",
    "Some models such as Large Language Models (LLMs) like [Llama 3 70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) have 70,000,000,000+ (70B) parameters (over 1000x our model).\n",
    "\n",
    "In essence, the more parameters a model has, the more opportunities it has to learn (generally).\n",
    "\n",
    "More parameters often results in more capabilities.\n",
    "\n",
    "However, more parameters also often results in a much larger model size (e.g. many gigabytes versus hundreds of megabytes) as well as a much longer compute time (fewer samples per second).\n",
    "\n",
    "For our use case, a binary text classification task, 60M parameters is more than enough.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Why count the parameters in a model?\n",
    "\n",
    "While it may be tempting to always go with a model that has the most parameters, there are many considerations to take into account before doing so.\n",
    "\n",
    "> What hardware is the model going to run on?\n",
    "\n",
    "If you need the model to run on cheap hardware, you'll likely want a smaller model.\n",
    "\n",
    "> How fast do you need the model to be?\n",
    "\n",
    "If you need 100-1000s of predictions per second, you'll likely want a smaller model.\n",
    "\n",
    "> \"I don't mind about speed or cost, I just want quality.\"\n",
    "\n",
    "Go with the biggest model you can.\n",
    "\n",
    "However, often times you can get really good results by training a small model to do a specific task using quality data than by just always using a large model.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a directory for saving models\n",
    "\n",
    "Training a model can take a while.\n",
    "\n",
    "So we'll want a place to save our models.\n",
    "\n",
    "Let's create a directory called `\"learn_hf_food_not_food_text_classifier-distilbert-base-uncased\"` (it's a bit verbose and you can change this if you like but I like to be specific). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create model output directory\n",
    "from pathlib import Path\n",
    "\n",
    "# Create models directory\n",
    "models_dir = Path(\"models\")\n",
    "models_dir.mkdir(exist_ok=True)\n",
    "\n",
    "# Create model save name\n",
    "model_save_name = \"learn_hf_food_not_food_text_classifier-distilbert-base-uncased\"\n",
    "\n",
    "# Create model save path\n",
    "model_save_dir = Path(models_dir, model_save_name)\n",
    "\n",
    "model_save_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up training arguments with TrainingArguments\n",
    "\n",
    "Time to get our model ready for training!\n",
    "\n",
    "We're up to step 3 of our process:\n",
    "\n",
    "1. ✅ Create and preprocess data.\n",
    "2. ✅ Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "The `transformers.TrainingArguments` class contains a series of helpful items, including hyperparameter settings and model saving strategies to use throughout training.\n",
    "\n",
    "It has many parameters, too many to explain here.\n",
    "\n",
    "However, the following table breaks down a helpful handful.\n",
    "\n",
    "Some of the parameters we'll set are the same as the defaults (this is on purpose as the defaults are often pretty good), some such as `learning_rate` are different.\n",
    "\n",
    "| Parameter | Explanation |\n",
    "|:-----|:-----|\n",
    "| `output_dir` | Name of output directory to save the model and checkpoints to. For example, `learn_hf_food_not_food_text_classifier_model`.  |\n",
    "| `learning_rate` | Value of the initial learning rate to use during training. Passed to [`transformers.AdamW`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments). Initial learning rate because the learning rate can be dynamic during training. The ideal learning is experimental in nature. Defaults to `5e-5` or `0.00001` but we'll use `0.0001`.  |\n",
    "| `per_device_train_batch_size`| Size of batches to place on target device during training. For example, a batch size of `32` means the model will look at 32 samples at a time. A batch size too large will result in out of memory issues (e.g. your GPU can't handle holding a large number of samples in memory at a time). |\n",
    "| `per_device_eval_batch_size` | Size of batches to place on target device during evaluation. Can often be larger than during training because no gradients are being calculated. For example, training batch size could be 32 where as evaluation batch size may be able to be 128 (4x larger). Though these are only esitmates. |\n",
    "| `num_train_epochs` | Number of times to pass over the data to try and learn patterns. For example, if `num_train_epochs=10`, the model will do 10 full passes of the training data. Because we're working with a small dataset, 10 epochs should be fine to begin with. However, if you had a larger dataset, you may want to do a few experiments using less data (e.g. 10% of the data) for a smaller number of epochs to make sure things work. |\n",
    "| `eval_strategy` | When to evaluate the model on the evaluation data. If `eval_strategy=\"epoch\"`, the model will be evaluated every epoch. See the documentation for more options. **Note:** This was previously called `evaluation_strategy` but was shortened in `transformers==4.46`. |\n",
    "| `save_strategy` | When to save a model checkpoint. If `save_strategy=\"epoch\"`, a checkpoint will be saved every epoch. See the documentation for more save options. |\n",
    "| `save_total_limit` | Number of total amount of checkpoints to save (so we don't save `num_train_epochs` checkpoints). For example, can limit to 3 saves so the total number of saves are the 3 most recent as well as the best performing checkpoint (as per `load_best_model_at_end`). |\n",
    "| `use_cpu` | Set to `False` by default, will use CUDA GPU (`torch.device(\"cuda\")`) or MPS device (`torch.device(\"mps\")`, for Mac) if available. This is because training is generally faster on an accelerator device. |\n",
    "| `seed` | Set to `42` by default for reproducibility. Meaning that subsequent runs with the same setup should achieve the same results. |\n",
    "| `load_best_model_at_end` | When set to `True`, makes sure that the best model found during training is loaded when training finishes. This will mean the best model checkpoint gets saved regardless of what epoch it happened on. This is set to `False` by default. |\n",
    "| `logging_strategy` | When to log the training results and metrics. For example, if `logging_strategy=\"epoch\"`, results will be logged as outputs every epoch. See the documentation for more logging options. |\n",
    "| `report_to` | Log experiments to various experiment tracking services. For example, you can log to Weights & Biases using `report_to=\"wandb\"`. We'll turn this off for now and keep logging to a local directory by setting `report_to=\"none\"`. |\n",
    "| `push_to_hub` | Automatically upload the model to the Hugging Face Hub every time the model is saved. We'll set `push_to_hub=False` as we'll see how to do this manually later on. See the documentation for more options on saving models to the Hugging Face Hub. |\n",
    "| `hub_token` | Add your Hugging Face Hub token to push a model to the Hugging Face Hub with `push_to_hub` (will default to [`huggingface-cli login`](https://huggingface.co/docs/huggingface_hub/en/guides/cli) details). |\n",
    "| `hub_private_repo` | Whether or not to make the Hugging Face Hub repository private or public, defaults to `False` (e.g. set to `True` if you want the repository to be private). |\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "To get more familiar with the `transformers.TrainingArguments` class, I'd highly recommend [reading the documentation](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments) for 15-20 minutes. Perhaps over a couple of sessions. There are quite a large number of parameters which will be helpful to be aware of.\n",
    "\n",
    ":::\n",
    "\n",
    "Phew!\n",
    "\n",
    "That was a lot to take in.\n",
    "\n",
    "But let's now practice setting up our own instance of `transformers.TrainingArguments`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "sqNdIIJd6QQO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Saving model checkpoints to: models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\n"
     ]
    }
   ],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "print(f\"[INFO] Saving model checkpoints to: {model_save_dir}\")\n",
    "\n",
    "# Create training arguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=model_save_dir,\n",
    "    learning_rate=0.0001,\n",
    "    per_device_train_batch_size=32,\n",
    "    per_device_eval_batch_size=32,\n",
    "    num_train_epochs=10,\n",
    "    eval_strategy=\"epoch\", # was previously \"evaluation_strategy\"\n",
    "    save_strategy=\"epoch\",\n",
    "    save_total_limit=3, # limit the total amount of save checkpoints (so we don't save num_epochs checkpoints)\n",
    "    use_cpu=False, # set to False by default, will use CUDA GPU or MPS device if available\n",
    "    seed=42, # set to 42 by default for reproducibility\n",
    "    load_best_model_at_end=True, # load the best model when finished training\n",
    "    logging_strategy=\"epoch\", # log training results every epoch\n",
    "    report_to=\"none\", # optional: log experiments to Weights & Biases/other similar experimenting tracking services (we'll turn this off for now) \n",
    "    # push_to_hub=True # optional: automatically upload the model to the Hub (we'll do this manually later on)\n",
    "    # hub_token=\"your_token_here\" # optional: add your Hugging Face Hub token to push to the Hub (will default to huggingface-cli login)\n",
    "    hub_private_repo=False # optional: make the uploaded model private (defaults to False)\n",
    ")\n",
    "\n",
    "# Optional: Print out training_args to inspect (warning, it is quite a long output)\n",
    "# training_args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training arguments created!\n",
    "\n",
    "Let's put them to work in an instance of `transformers.Trainer`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up an instance of Trainer\n",
    "\n",
    "Time for step 4!\n",
    "\n",
    "1. ✅ Create and preprocess data.\n",
    "2. ✅ Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. ✅ Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "The [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer) class allows you to train models.\n",
    "\n",
    "It's built on PyTorch so it gets to leverage all of the powerful PyTorch toolkit.\n",
    "\n",
    "But since it also works closely with the `transformers.TrainingArguments` class, it offers many helpful features.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "`transformers.Trainer` can work with `torch.nn.Module` models, however, it is designed to work best with [`transformers.PreTrainedModel`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel)'s from the `transformers` library.\n",
    "\n",
    "This is not a problem for us as we're using `transformers.AutoModelForSequenceClassification.from_pretrained` which loads a `transformers.PreTrainedModel`.\n",
    "\n",
    "See the [`transformers.Trainer` documentation](https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer) for tips on how to make sure your model is compatible.\n",
    "\n",
    ":::\n",
    "\n",
    "| Parameter | Explanation |\n",
    "|:-----|:-----|\n",
    "| `model` | The model we'd like to train. Works best with an instance of `transformers.PreTrainedModel`. Most models loaded using `from_pretrained` will be of this type. |\n",
    "| `args` | Instance of `transformers.TrainingArguments`. We'll use the `training_args` object we defined earlier. But if this is not set, it will default to the default settings for `transformers.TrainingArguments`. |\n",
    "| `train_dataset`| Dataset to use during training. We can use our `tokenized_dataset[\"train\"]` as it has already been preprocessed. |\n",
    "| `eval_dataset` | Dataset to use during evaluation (our model will not see this data during training). We can use our `tokenized_dataset[\"test\"]` as it has already been preprocessed.  |\n",
    "| `tokenizer` | The `tokenizer` which was used to preprocess the data. Passing a tokenizer will also pad the inputs to maximum length when batching them. It will also be saved with the model so future re-runs are easier. |\n",
    "| `compute_metrics` | An evaluation function to evaluate a model during training and evaluation steps. In our case, we'll use the `compute_accuracy` function we defined earlier. |\n",
    "\n",
    "With all this being said, let's build our `Trainer`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "7vXcfL6q7Fqs"
   },
   "outputs": [],
   "source": [
    "from transformers import Trainer\n",
    "\n",
    "# Setup Trainer\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_dataset[\"train\"],\n",
    "    eval_dataset=tokenized_dataset[\"test\"],\n",
    "    # Note: the 'tokenizer' parameter will be changed to 'processing_class' in Transformers v5.0.0\n",
    "    tokenizer=tokenizer, # Pass tokenizer to the Trainer for dynamic padding (padding as the training happens) (see \"data_collator\" in the Trainer docs)\n",
    "    compute_metrics=compute_accuracy\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo! We've created our own `trainer`.\n",
    "\n",
    "We're one step closer to training!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training our text classification model\n",
    "\n",
    "We've done most of the hard word setting up our `transformers.TrainingArguments` as well as our `transformers.Trainer`.\n",
    "\n",
    "Now how about we train a model?\n",
    "\n",
    "Following our steps:\n",
    "\n",
    "1. ✅ Create and preprocess data.\n",
    "2. ✅ Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. ✅ Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. ✅ Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "Looks like all we have to do is call `transformers.Trainer.train()`.\n",
    "\n",
    "We'll be sure to save the results of the training to a variable `results` so we can inspect them later.\n",
    "\n",
    "Let's try!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 174
    },
    "id": "MEzkvmO6-Whg",
    "outputId": "22937d9f-a794-4579-bc67-70ba6cb05dcd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='70' max='70' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [70/70 00:07, Epoch 10/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.328100</td>\n",
       "      <td>0.039627</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.019200</td>\n",
       "      <td>0.005586</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.003700</td>\n",
       "      <td>0.002026</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.001700</td>\n",
       "      <td>0.001186</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.001100</td>\n",
       "      <td>0.000858</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.000800</td>\n",
       "      <td>0.000704</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.000800</td>\n",
       "      <td>0.000619</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.000700</td>\n",
       "      <td>0.000571</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>0.000547</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>0.000539</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Train a text classification model\n",
    "results = trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woahhhh!!!\n",
    "\n",
    "How cool is that!\n",
    "\n",
    "We just trained a text classification model!\n",
    "\n",
    "And it looks like the training went pretty quick (thanks to our smaller dataset and relatively small model, for larger datasets, training would likely take longer).\n",
    "\n",
    "How about we check some of the metrics?\n",
    "\n",
    "We can do so using the `results.metrics` attribute (this returns a Python dictionary with stats from our training run)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_runtime: 7.5421\n",
      "train_samples_per_second: 265.177\n",
      "train_steps_per_second: 9.281\n",
      "total_flos: 18110777160000.0\n",
      "train_loss: 0.03574410408868321\n",
      "epoch: 10.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect training metrics\n",
    "for key, value in results.metrics.items():\n",
    "    print(f\"{key}: {value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Looks like our overall training runtime is low because of our small dataset.\n",
    "\n",
    "And looks like our `trainer` was able to process a fair few samples per second.\n",
    "\n",
    "If we were to 1000x the size of our dataset (e.g. ~250 samples -> ~250,000 samples which is quite a substantial dataset), it seems our training time still wouldn't take too long.\n",
    "\n",
    "The `total_flos` stands for \"[floating point operations](https://en.wikipedia.org/wiki/FLOPS)\" (also referred to as FLOPS), this is the total number of calculations our model has performed to find patterns in the data. And as you can see, it's quite a large number! \n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Depending on the hardware you're using, the results with respect to `train_runtime`, `train_samples_per_second` and `train_steps_per_second` will likely be different.\n",
    "\n",
    "The faster your accelerator hardware (e.g. NVIDIA GPU or Mac GPU), the lower your runtime and higher your samples/steps per second will be.\n",
    "\n",
    "For reference, on my local NVIDIA RTX 4090, I get a `train_runtime` of 8-9 seconds, `train_samples_per_second` of 230-250 and `train_steps_per_second` of 8.565.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the model for later use\n",
    "\n",
    "Now our model has been trained, let's save it for later use.\n",
    "\n",
    "We'll save it locally first and push it to the Hugging Face Hub later.\n",
    "\n",
    "We can save our model using the [`transformers.Trainer.save_model`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Trainer.save_model) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Saving model to models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\n"
     ]
    }
   ],
   "source": [
    "# Save model\n",
    "print(f\"[INFO] Saving model to {model_save_dir}\")\n",
    "trainer.save_model(output_dir=model_save_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model saved locally! Before we save it to the Hugging Face Hub, let's check out its metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting the model training metrics\n",
    "\n",
    "We can get a log of our model's training state using `trainer.state.log_history`.\n",
    "\n",
    "This will give us a collection of metrics per epoch (as long as we set `logging_strategy=\"epoch\"` in `transformers.TrainingArguments`), in particular, it will give us a loss value per epoch.\n",
    "\n",
    "We can extract these values and inspect them visually for a better understanding our model training.\n",
    "\n",
    "Let's get the training history and inspect it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'loss': 0.3281,\n",
       "  'grad_norm': 0.6938912272453308,\n",
       "  'learning_rate': 9e-05,\n",
       "  'epoch': 1.0,\n",
       "  'step': 7},\n",
       " {'eval_loss': 0.03962664306163788,\n",
       "  'eval_accuracy': 1.0,\n",
       "  'eval_runtime': 0.0135,\n",
       "  'eval_samples_per_second': 3707.312,\n",
       "  'eval_steps_per_second': 148.292,\n",
       "  'epoch': 1.0,\n",
       "  'step': 7},\n",
       " {'loss': 0.0192,\n",
       "  'grad_norm': 0.14873287081718445,\n",
       "  'learning_rate': 8e-05,\n",
       "  'epoch': 2.0,\n",
       "  'step': 14},\n",
       " {'eval_loss': 0.005585948005318642,\n",
       "  'eval_accuracy': 1.0,\n",
       "  'eval_runtime': 0.0147,\n",
       "  'eval_samples_per_second': 3399.06,\n",
       "  'eval_steps_per_second': 135.962,\n",
       "  'epoch': 2.0,\n",
       "  'step': 14}]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get training history \n",
    "trainer_history_all = trainer.state.log_history \n",
    "trainer_history_metrics = trainer_history_all[:-1] # get everything except the training time metrics (we've seen these already)\n",
    "trainer_history_training_time = trainer_history_all[-1] # this is the same value as results.metrics from above\n",
    "\n",
    "# View the first 4 metrics from the training history\n",
    "trainer_history_metrics[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, looks like the metrics are logged every epochs in a list Python dictionaries with interleaving `loss` (this is the training set loss) and `eval_loss` values.\n",
    "\n",
    "How about we write some code to separate the training set metrics and the evaluation set metrics?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] First two items in training set:\n",
      "[{'epoch': 1.0,\n",
      "  'grad_norm': 0.6938912272453308,\n",
      "  'learning_rate': 9e-05,\n",
      "  'loss': 0.3281,\n",
      "  'step': 7},\n",
      " {'epoch': 2.0,\n",
      "  'grad_norm': 0.14873287081718445,\n",
      "  'learning_rate': 8e-05,\n",
      "  'loss': 0.0192,\n",
      "  'step': 14}]\n",
      "\n",
      "[INFO] First two items in evaluation set:\n",
      "[{'epoch': 1.0,\n",
      "  'eval_accuracy': 1.0,\n",
      "  'eval_loss': 0.03962664306163788,\n",
      "  'eval_runtime': 0.0135,\n",
      "  'eval_samples_per_second': 3707.312,\n",
      "  'eval_steps_per_second': 148.292,\n",
      "  'step': 7},\n",
      " {'epoch': 2.0,\n",
      "  'eval_accuracy': 1.0,\n",
      "  'eval_loss': 0.005585948005318642,\n",
      "  'eval_runtime': 0.0147,\n",
      "  'eval_samples_per_second': 3399.06,\n",
      "  'eval_steps_per_second': 135.962,\n",
      "  'step': 14}]\n"
     ]
    }
   ],
   "source": [
    "import pprint # import pretty print for nice printing of lists\n",
    "\n",
    "# Extract training and evaluation metrics\n",
    "trainer_history_training_set = []\n",
    "trainer_history_eval_set = []\n",
    "\n",
    "# Loop through metrics and filter for training and eval metrics\n",
    "for item in trainer_history_metrics:\n",
    "    item_keys = list(item.keys())\n",
    "    # Check to see if \"eval\" is in the keys of the item\n",
    "    if any(\"eval\" in item for item in item_keys):\n",
    "        trainer_history_eval_set.append(item)\n",
    "    else:\n",
    "        trainer_history_training_set.append(item)\n",
    "\n",
    "# Show the first two items in each metric set\n",
    "print(f\"[INFO] First two items in training set:\")\n",
    "pprint.pprint(trainer_history_training_set[:2])\n",
    "\n",
    "print(f\"\\n[INFO] First two items in evaluation set:\")\n",
    "pprint.pprint(trainer_history_eval_set[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful!\n",
    "\n",
    "How about we take it a step further and turn our metrics into pandas DataFrames so we can view them easier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loss</th>\n",
       "      <th>grad_norm</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>epoch</th>\n",
       "      <th>step</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.3281</td>\n",
       "      <td>0.693891</td>\n",
       "      <td>0.00009</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0192</td>\n",
       "      <td>0.148733</td>\n",
       "      <td>0.00008</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0037</td>\n",
       "      <td>0.037808</td>\n",
       "      <td>0.00007</td>\n",
       "      <td>3.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0017</td>\n",
       "      <td>0.022227</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>4.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.018665</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>5.0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     loss  grad_norm  learning_rate  epoch  step\n",
       "0  0.3281   0.693891        0.00009    1.0     7\n",
       "1  0.0192   0.148733        0.00008    2.0    14\n",
       "2  0.0037   0.037808        0.00007    3.0    21\n",
       "3  0.0017   0.022227        0.00006    4.0    28\n",
       "4  0.0011   0.018665        0.00005    5.0    35"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create pandas DataFrames for the training and evaluation metrics\n",
    "trainer_history_training_df = pd.DataFrame(trainer_history_training_set)\n",
    "trainer_history_eval_df = pd.DataFrame(trainer_history_eval_set)\n",
    "\n",
    "trainer_history_training_df.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "And the evaluation DataFrame?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eval_loss</th>\n",
       "      <th>eval_accuracy</th>\n",
       "      <th>eval_runtime</th>\n",
       "      <th>eval_samples_per_second</th>\n",
       "      <th>eval_steps_per_second</th>\n",
       "      <th>epoch</th>\n",
       "      <th>step</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.039627</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0135</td>\n",
       "      <td>3707.312</td>\n",
       "      <td>148.292</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.005586</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0147</td>\n",
       "      <td>3399.060</td>\n",
       "      <td>135.962</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002026</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0136</td>\n",
       "      <td>3680.635</td>\n",
       "      <td>147.225</td>\n",
       "      <td>3.0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.001186</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0151</td>\n",
       "      <td>3303.902</td>\n",
       "      <td>132.156</td>\n",
       "      <td>4.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000858</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0159</td>\n",
       "      <td>3146.137</td>\n",
       "      <td>125.845</td>\n",
       "      <td>5.0</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   eval_loss  eval_accuracy  eval_runtime  eval_samples_per_second  \\\n",
       "0   0.039627            1.0        0.0135                 3707.312   \n",
       "1   0.005586            1.0        0.0147                 3399.060   \n",
       "2   0.002026            1.0        0.0136                 3680.635   \n",
       "3   0.001186            1.0        0.0151                 3303.902   \n",
       "4   0.000858            1.0        0.0159                 3146.137   \n",
       "\n",
       "   eval_steps_per_second  epoch  step  \n",
       "0                148.292    1.0     7  \n",
       "1                135.962    2.0    14  \n",
       "2                147.225    3.0    21  \n",
       "3                132.156    4.0    28  \n",
       "4                125.845    5.0    35  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer_history_eval_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And of course, we'll have follow the data explorer's motto of *visualize, visualize, visualize!* and inspect our loss curves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAIjCAYAAADvBuGTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB85klEQVR4nO3dd3gU9drG8Xt30yuBQAqEFpGWAEqTJniMBFQUVNrhleJRPIIFIyqoNFGxoKKgoHgExIZ6BBuiiOKxgKKI0kSkQwg9HdJ23j+SXViSQBKSzCb5fq5rL8js7Mwzu5tN7vx+84zFMAxDAAAAAIALYjW7AAAAAACoDghXAAAAAFAOCFcAAAAAUA4IVwAAAABQDghXAAAAAFAOCFcAAAAAUA4IVwAAAABQDghXAAAAAFAOCFcAAAAAUA4IV0A5Wb16tSwWi1avXm1aDRaLRVOnTnVZtm7dOnXt2lX+/v6yWCzasGGDpk6dKovFUun17d69WxaLRQsXLqz0fZdVUc/puda98847K7agM5T3e27hwoWyWCzavXu3c1mvXr3Uq1evctl+dVDUc1RS7vAZURkaN26skSNHml1Gscx+T5fmM6U8mfW5X91VxZ9rqFiEK7gFi8VSolt5/VKSmJioqVOnasOGDeWyPXeVk5OjgQMH6vjx43r++ee1ePFiNWrUqML3+/bbb2vWrFkVvh8z/Pjjj5o6daqSk5PLdbuOH9COm6enp0JDQ9W1a1c99NBD2rt3b7nt64knntCyZcvKpU6LxaKgoCC1a9dOc+bMUV5eXrnV6VDa99OFHB9QHpYvX25KgELFqc4/11C+PMwuAJCkxYsXu3z9xhtvaOXKlYWWt2zZslz2l5iYqGnTpqlx48Zq165duWzTHZw8eVIeHqe/rXfs2KE9e/Zo/vz5uvXWW53LH3nkEU2YMKHC6nj77be1adMmjRs3zmV5o0aNdPLkSXl6elbYvsvb2c/pjz/+qGnTpmnkyJGqVatWue9v6NChuvrqq2W323XixAmtW7dOs2bN0gsvvKD//Oc/GjJkiHPdyy+/XCdPnpSXl1ep9vHEE0/opptuUv/+/V2W33zzzRoyZIi8vb1LXKckpaSkaPny5brrrru0Z88ePfPMM6Wq53yKez8Vp7jjKw+leY7OVtbXC1XP8uXL9dJLLxUZsM7+TEHVUJ1+rqFi8d0Nt/B///d/Ll+vXbtWK1euLLQc5+bj4+Py9eHDhyWpUAjw8PAw5Ye7xWIpVKO7q+x6L7300kLv+z179qh3794aMWKEWrZsqbZt20qSrFZrudZns9lks9nKVOeYMWPUuXNnvf322+UWrjIyMuTv718u2yqvfZTmOTpbeb9eqJp4D7gnwzB06tQp+fr6lupxVfHnGioW0wJRZdjtds2aNUutW7eWj4+PwsLCdPvtt+vEiRPOdaZMmSKr1apVq1a5PHb06NHy8vLS77//rtWrV6tjx46SpFGjRjmnNp1vvvSBAwf0r3/9S5GRkfL29laTJk10xx13KDs7u9jHfPfddxo4cKAaNmwob29vRUVF6d5779XJkydd1ktKStKoUaPUoEEDeXt7KyIiQtdff73LeR2//PKL4uPjFRoaKl9fXzVp0kS33HKLy3bOnMs/cuRI9ezZU5I0cOBAWSwW53kGxc29f/PNN9WpUyf5+fkpJCREl19+ub788kvn/R999JGuueYa53MQHR2t6dOnu0wF69Wrlz777DPt2bPH+dw2btxYUvFz07/++mv16NFD/v7+qlWrlq6//npt3brVZR1HzX///bdz1Cg4OFijRo1SZmZmsa+BJL344ouy2WwuU/meffZZWSwWJSQkOJfl5eUpMDBQDz74YJHP6dSpU3X//fdLkpo0aeI8vrPPv1m2bJliYmLk7e2t1q1ba8WKFees73waNWqkhQsXKjs7W08//bRzeVHn8Gzfvl033nijwsPD5ePjowYNGmjIkCFKSUlxHk9GRoYWLVrkrN9xfsyFnE9ksVgUFhZWZGj//PPPna9vYGCgrrnmGm3evNllnZEjRyogIEA7duzQ1VdfrcDAQA0bNuyc76fi6iju+BzvoS1btuif//ynQkJC1L17d0nSH3/8oZEjR6pp06by8fFReHi4brnlFh07dsxl+0U9R40bN9a1116r77//Xp06dZKPj4+aNm2qN954w+WxRb1evXr1UkxMjLZs2aIrrrhCfn5+ql+/vsvr7LBnzx5dd9118vf3V7169XTvvffqiy++KNGU6T179mjMmDFq3ry5fH19VadOHQ0cOLDQa+04vh9++EEJCQmqW7eu/P39NWDAAB05csRlXcMw9Nhjj6lBgwby8/PTFVdcUeh1PZeSfKZfe+21atq0aZGP79Klizp06OD8esGCBfrHP/6hevXqydvbW61atdLcuXPPW0dx7/uiXq+SfKaPHDlSL730kiTXKe8ORZ1z9dtvv6lv374KCgpSQECArrzySq1du7bIOkvy2pRUbm6upk+frujoaHl7e6tx48Z66KGHlJWV5bJeSX7+vPvuu2rfvr0CAwMVFBSk2NhYvfDCC+etISMjQ/fdd5+ioqLk7e2t5s2ba+bMmTIMw7lOTEyMrrjiikKPtdvtql+/vm666SaXZed7X0mnv2+/+OILdejQQb6+vnrllVeKrLG0P9ccn2d79+7Vtddeq4CAANWvX9/5vti4caP+8Y9/yN/fX40aNdLbb79daJ/JyckaN26c83m56KKL9NRTT8lut5/3OYW5GLlClXH77bdr4cKFGjVqlO6++27t2rVLc+bM0W+//aYffvhBnp6eeuSRR/TJJ5/oX//6lzZu3KjAwEB98cUXmj9/vqZPn662bdvq0KFDevTRRzV58mSNHj1aPXr0kCR17dq12H0nJiaqU6dOSk5O1ujRo9WiRQsdOHBAH3zwgTIzM4ud5vP+++8rMzNTd9xxh+rUqaOff/5Zs2fP1v79+/X+++8717vxxhu1efNm3XXXXWrcuLEOHz6slStXau/evc6ve/furbp162rChAmqVauWdu/erQ8//PCcz1f9+vX1xBNP6O6771bHjh0VFhZW7PrTpk3T1KlT1bVrVz366KPy8vLSTz/9pK+//lq9e/eWlP/DPSAgQAkJCQoICNDXX3+tyZMnKzU11Tla8fDDDyslJUX79+/X888/L0kKCAgodr9fffWV+vbtq6ZNm2rq1Kk6efKkZs+erW7dumn9+vWFfpEeNGiQmjRpohkzZmj9+vV67bXXVK9ePT311FPF7qNHjx6y2+36/vvvde2110rK/yXJarXqu+++c67322+/KT09XZdffnmR27nhhhv0119/6Z133tHzzz+v0NBQSVLdunWd63z//ff68MMPNWbMGAUGBurFF1/UjTfeqL1796pOnTrF1ng+Xbp0UXR0tFauXFnsOtnZ2YqPj1dWVpbuuusuhYeH68CBA/r000+VnJys4OBgLV68WLfeeqs6deqk0aNHS5Kio6NLXU9mZqaOHj0qSUpNTdXnn3+uFStWaOLEiS7rLV68WCNGjFB8fLyeeuopZWZmau7cuerevbt+++03l9c3NzdX8fHx6t69u2bOnCk/Pz+Fh4eX6v1UkuMbOHCgmjVrpieeeML5C9zKlSu1c+dOjRo1SuHh4dq8ebNeffVVbd68WWvXrj1vI4C///5bN910k/71r39pxIgRev311zVy5Ei1b99erVu3PudjT5w4oT59+uiGG27QoEGD9MEHH+jBBx9UbGys+vbtKyn/F9B//OMfOnjwoO655x6Fh4fr7bff1jfffHPObTusW7dOP/74o4YMGaIGDRpo9+7dmjt3rnr16qUtW7bIz8/PZf277rpLISEhmjJlinbv3q1Zs2bpzjvv1JIlS5zrTJ48WY899piuvvpqXX311Vq/fr169+59zj84nakkn+mDBw/W8OHDtW7dOucfxaT8sLh27VqXUdK5c+eqdevWuu666+Th4aFPPvlEY8aMkd1u19ixY0tU0/mU5DP99ttvV2JiYpFT24uyefNm9ejRQ0FBQXrggQfk6empV155Rb169dK3336rzp07u6xfktempG699VYtWrRIN910k+677z799NNPmjFjhrZu3aqlS5dKUol+/qxcuVJDhw7VlVde6fws3rp1q3744Qfdc889xe7fMAxdd911+uabb/Svf/1L7dq10xdffKH7779fBw4ccH7PDx48WFOnTlVSUpLCw8Odj//++++VmJjoMl26JO8rh23btmno0KG6/fbbddttt6l58+ZF1lnan2tS/h/r+vbtq8svv1xPP/203nrrLd15553y9/fXww8/rGHDhumGG27QvHnzNHz4cHXp0kVNmjSRlP/52rNnTx04cEC33367GjZsqB9//FETJ07UwYMHOffL3RmAGxo7dqxx5tvzu+++MyQZb731lst6K1asKLR848aNhpeXl3HrrbcaJ06cMOrXr2906NDByMnJca6zbt06Q5KxYMGCEtUzfPhww2q1GuvWrSt0n91uNwzDML755htDkvHNN98478vMzCy0/owZMwyLxWLs2bPHMAzDOHHihCHJeOaZZ4rd/9KlSw1JRe7/TJKMKVOmOL921PT++++7rDdlyhSX53f79u2G1Wo1BgwYYOTl5RV5fMUdz+233274+fkZp06dci675pprjEaNGhVad9euXYWe93bt2hn16tUzjh075lz2+++/G1ar1Rg+fHihmm+55RaXbQ4YMMCoU6dOoX2dKS8vzwgKCjIeeOAB5zHVqVPHGDhwoGGz2Yy0tDTDMAzjueeeM6xWq3HixAnnY89+Tp955hlDkrFr165C+5FkeHl5GX///bfLsUgyZs+efc4aHc/Nud4H119/vSHJSElJMQyj8Hvut99+K/L1Ppu/v78xYsSIQssXLFhQ6Nh69uxp9OzZs1CdRd3uuOMOl/dLWlqaUatWLeO2225z2U9SUpIRHBzssnzEiBGGJGPChAmF6iru/VTa43O8h4YOHVrovqLe2++8844hyfjf//7nXFbUc9SoUaNC6x0+fNjw9vY27rvvPueyoj4jevbsaUgy3njjDeeyrKwsIzw83Ljxxhudy5599llDkrFs2TLnspMnTxotWrQotM2iFHV8a9asKbRvx/HFxcW5vJb33nuvYbPZjOTkZOfxeXl5Gddcc43Leg899JAhqcjn/0wl/UxPSUkp9DwahmE8/fTTLp+jxR1jfHy80bRpU5dlZ7+ni3pNDaPsn+mGUfhn2JnO/kzp37+/4eXlZezYscO5LDEx0QgMDDQuv/zyQnWe77Upztmf+xs2bDAkGbfeeqvLeuPHjzckGV9//bVhGCX7+XPPPfcYQUFBRm5u7jlrONuyZcsMScZjjz3msvymm24yLBaL87N027ZtRX6OjhkzxggICHC+LqX5XcHxfbtixYoS1Vqan2uOz7MnnnjCuezEiROGr6+vYbFYjHfffde5/M8//yz0npg+fbrh7+9v/PXXXy77mjBhgmGz2Yy9e/eWqGaYg2mBqBLef/99BQcH66qrrtLRo0edt/bt2ysgIMDlr7cxMTGaNm2aXnvtNcXHx+vo0aNatGhRmc8xstvtWrZsmfr16+cyBcXhXH/RPnPudkZGho4ePaquXbvKMAz99ttvznW8vLy0evXqQtMWHBznTH366afKyckp03Gcy7Jly2S32zV58mRZra4fC2ce35nHk5aWpqNHj6pHjx7KzMzUn3/+Wer9Hjx4UBs2bNDIkSNVu3Zt5/I2bdroqquu0vLlyws95t///rfL1z169NCxY8eUmppa7H6sVqu6du2q//3vf5Ly/6J67NgxTZgwQYZhaM2aNZLyR7NiYmIuqFFFXFycy0hJmzZtFBQUpJ07d5Z5mw6Ov5SmpaUVeX9wcLAk6YsvvjjvVMkLNXr0aK1cuVIrV67Uf//7X40dO1avvPKKyzTLlStXKjk5WUOHDnX5vrXZbOrcuXORoy533HFHhdYtFX4PSa7v7VOnTuno0aO67LLLJEnr168/7zZbtWrlHAWX8kczmzdvXqLXPSAgwOX8NS8vL3Xq1MnlsStWrFD9+vV13XXXOZf5+PjotttuO+/2Jdfjy8nJ0bFjx3TRRRepVq1aRR7f6NGjXb73e/Tooby8PO3Zs0dS/ohzdna27rrrLpf1Stp0pKSf6UFBQerbt6/ee+89l2liS5Ys0WWXXaaGDRsWeYwpKSk6evSoevbsqZ07dzqnxV6oknyml0ZeXp6+/PJL9e/f32X6Y0REhP75z3/q+++/L/TZdr7XpqQcn69nfs9K0n333SdJ+uyzzySV7OdPrVq1lJGRcc6R9eJqsNlsuvvuuwvVYBiGPv/8c0nSxRdfrHbt2rmMzuXl5emDDz5Qv379nK9LaX5XkPKnd8fHx5eq5tI4s5FUrVq11Lx5c/n7+2vQoEHO5c2bN1etWrVcvt/ff/999ejRQyEhIS7HERcXp7y8POfPMrgnwhWqhO3btyslJUX16tVT3bp1XW7p6enOxg0O999/v9q2bauff/5ZU6ZMUatWrcq87yNHjig1NVUxMTGlfuzevXudwSEgIEB169Z1ngfl+GHv7e2tp556Sp9//rnCwsKcUwiSkpKc2+nZs6duvPFGTZs2TaGhobr++uu1YMGCQvPiy2rHjh2yWq3nfZ42b96sAQMGKDg4WEFBQapbt67zl8Ky/PLi+GWgqKkYLVu21NGjR5WRkeGy/MxfpiQpJCREkooNpg49evTQr7/+qpMnT+q7775TRESELr30UrVt29Y5NfD77793+QW5LM6uz1Hj+eorifT0dElSYGBgkfc3adJECQkJeu211xQaGqr4+Hi99NJL5faL5ZmaNWumuLg4xcXF6YYbbtCcOXM0ZswYzZo1Sxs3bpSU/30rSf/4xz8Kfd9++eWXhb5vPTw81KBBg3Kv9WyOqTdnOn78uO655x6FhYXJ19dXdevWda5XkufvQl73Bg0aFPojzdmP3bNnj6Kjowutd9FFF513+1J+h7rJkyc7z98IDQ1V3bp1lZycXOTxne/7zPG926xZM5f16tat61z3XErzmT548GDt27fP+UeQHTt26Ndff9XgwYNdtvnDDz8oLi7Oee5m3bp19dBDD0kq2+dTUUrymV4aR44cUWZmZrGfgXa7Xfv27XNZXtbPwLPt2bNHVqu10HsoPDxctWrVcr7GJfn5M2bMGF188cXq27evGjRooFtuuaVE55ru2bNHkZGRhT7THJ2BzwyMgwcP1g8//KADBw5Iyj8n7vDhwy7vg9L+rlDUZ0F58fHxcZkyLuX/Aayo7/fg4GCX12/79u1asWJFoWOIi4uTpELHAffCOVeoEux2u+rVq6e33nqryPvP/gDbuXOn8xc7xy96lS0vL09XXXWVjh8/rgcffFAtWrSQv7+/Dhw4oJEjR7qclDpu3Dj169dPy5Yt0xdffKFJkyZpxowZ+vrrr3XJJZfIYrHogw8+0Nq1a/XJJ5/oiy++0C233KJnn31Wa9euPe/c7/KQnJysnj17KigoSI8++qiio6Pl4+Oj9evX68EHH6y0k2yL69R25l+1i9K9e3fl5ORozZo1+u6775whqkePHvruu+/0559/6siRIxccrspaX0ls2rRJ9erVU1BQULHrPPvssxo5cqQ++ugjffnll7r77rs1Y8YMrV27tsKDy5VXXqk5c+bof//7n2JjY53vicWLF7ucJ+Fw9miyt7d3oZHTilBUN7BBgwbpxx9/1P3336927dopICBAdrtdffr0KdF7+0Je94p8zzjcddddWrBggcaNG6cuXbooODhYFotFQ4YMKfL4Krqm0nym9+vXT35+fnrvvffUtWtXvffee7JarRo4cKBznR07dujKK69UixYt9NxzzykqKkpeXl5avny5nn/++XO+hsXNPjj7mm2l+UyvSOX92pzvfMKS/PypV6+eNmzYoC+++EKff/65Pv/8cy1YsEDDhw/XokWLylTX2QYPHqyJEyfq/fff17hx4/Tee+8pODhYffr0ca5T2t8VStsZsDSKe51K8vrZ7XZdddVVeuCBB4pc9+KLL77wAlFhCFeoEqKjo/XVV1+pW7du5/0wtNvtGjlypIKCgjRu3DjnNW9uuOEG5zqluUp93bp1FRQUpE2bNpWq5o0bN+qvv/7SokWLNHz4cOfy4qZNREdH67777tN9992n7du3q127dnr22Wf15ptvOte57LLLdNlll+nxxx/X22+/rWHDhundd991mXpQFtHR0bLb7dqyZUux1/1avXq1jh07pg8//NCl4cOuXbsKrVvS59dxQeNt27YVuu/PP/9UaGhoubXi7tSpk7y8vPTdd9/pu+++c3b9u/zyyzV//nxnh8nimlk4lOa9U57WrFmjHTt2lOjyBLGxsYqNjdUjjzyiH3/8Ud26ddO8efP02GOPSaq4Y8jNzZV0eoTNMT2yXr16zr+4lkVp6y3t+idOnNCqVas0bdo0TZ482bnc8Qcad9CoUSNt2bJFhmG4HN/ff/9dosd/8MEHGjFihJ599lnnslOnTpX5YtiO793t27e7TGc7cuRIiUZQSvOZ7u/vr2uvvVbvv/++nnvuOS1ZskQ9evRQZGSkc51PPvlEWVlZ+vjjj11GdkrS8MMx8nP2c3H2NLvSfKaX9D1Yt25d+fn5FfsZaLVaFRUVVaJtlVajRo1kt9u1fft2l2tIHjp0SMnJyYUuOH++nz9eXl7q16+f+vXrJ7vdrjFjxuiVV17RpEmTih1hbdSokb766iulpaW5jF45ppmfWUOTJk3UqVMnLVmyRHfeeac+/PBD9e/f3+Wac6V5X5VWZX72R0dHKz09/YI+N2EepgWiShg0aJDy8vI0ffr0Qvfl5ua6/FB87rnn9OOPP+rVV1/V9OnT1bVrV91xxx3OzmaSnL+wl+QXC6vVqv79++uTTz7RL7/8Uuj+4v5a6Pjr1Jn3G4ZRqDVtZmamTp065bIsOjpagYGBzmkXJ06cKLQfRwgqj6mB/fv3l9Vq1aOPPlror6+O/RZ1PNnZ2Xr55ZcLbc/f379EU2QiIiLUrl07LVq0yOW12LRpk7788kvnRWrLg4+Pjzp27Kh33nlHe/fudRm5OnnypF588UVFR0crIiLinNspzXunvOzZs0cjR46Ul5eXMxQWJTU11RlwHGJjY2W1Wl3eJ/7+/hVS/yeffCJJzutwxcfHKygoSE888USR52qUtH10Sd9PZ65fmuMr6r0tya06csXHx+vAgQP6+OOPnctOnTql+fPnl+jxNput0PHNnj270OhMScXFxcnT01OzZ8922W5Jn7PSfKZL+aMWiYmJeu211/T7778XmhJY1GuYkpKiBQsWnLcWxx8BzjyPJS8vT6+++up591HUZ7pU8s8Jm82m3r1766OPPnJpBX/o0CG9/fbb6t69+zlHqi+E4/P17NfsueeekyRdc801kkr28+fsSxZYrVa1adPGZZ3iasjLy9OcOXNclj///POyWCzObpkOgwcP1tq1a/X666/r6NGjhd4HpX1flUZpP4cuxKBBg7RmzRp98cUXhe5LTk4u9DkP98LIFaqEnj176vbbb9eMGTO0YcMG9e7dW56entq+fbvef/99vfDCC7rpppu0detWTZo0SSNHjlS/fv0k5bcPb9euncaMGaP33ntPUv4P01q1amnevHkKDAyUv7+/OnfuXOz86yeeeEJffvmlevbsqdGjR6tly5Y6ePCg3n//fX3//fdFNkBo0aKFoqOjNX78eB04cEBBQUH673//W+ivun/99ZeuvPJKDRo0SK1atZKHh4eWLl2qQ4cOOdvLLlq0SC+//LIGDBig6OhopaWlaf78+QoKCiqXAHLRRRfp4Ycf1vTp09WjRw/dcMMN8vb21rp16xQZGakZM2aoa9euCgkJ0YgRI3T33XfLYrFo8eLFRYbL9u3ba8mSJUpISFDHjh0VEBDgfD3O9swzz6hv377q0qWL/vWvfzlbsQcHBxe6FsyF6tGjh5588kkFBwcrNjZWUv6oSvPmzbVt2zbn9ZDOpX379pLyW/MOGTJEnp6e6tevX7mNsK1fv15vvvmm7Ha7kpOTtW7dOv33v/91Pt+OX1iK8vXXX+vOO+/UwIEDdfHFFys3N1eLFy+WzWbTjTfe6HIMX331lZ577jlFRkaqSZMmhdo9l7ROKb/BxqpVq/Tf//5XXbt2dbbuDwoK0ty5c3XzzTfr0ksv1ZAhQ1S3bl3t3btXn332mbp161bol6qilOb9VJbjCwoKcp7rmJOTo/r16+vLL78sclTWLLfffrvmzJmjoUOH6p577lFERITeeust58VLz/dX9WuvvVaLFy9WcHCwWrVqpTVr1uirr74q8+UB6tatq/Hjx2vGjBm69tprdfXVV+u3337T559/7rxEwbmU9DPdwXHts/Hjxxd6P0tS7969nSMnt99+u9LT0zV//nzVq1dPBw8ePGctrVu31mWXXaaJEyfq+PHjql27tt59991Cv8CW9DNdOv05cffddys+Pl42m82lXfiZHnvsMa1cuVLdu3fXmDFj5OHhoVdeeUVZWVlFXu+svLRt21YjRozQq6++6pz2/fPPP2vRokXq37+/87pSJfn5c+utt+r48eP6xz/+oQYNGmjPnj2aPXu22rVr5zIqdrZ+/frpiiuu0MMPP6zdu3erbdu2+vLLL/XRRx9p3LhxhS6jMGjQII0fP17jx49X7dq1C43slPZ9VRql/Ry6EPfff78+/vhjXXvttc5LOmRkZGjjxo364IMPtHv37hJ9n8EkldSVECiV4trYvvrqq0b79u0NX19fIzAw0IiNjTUeeOABIzEx0cjNzTU6duxoNGjQoFBL2hdeeMGQZCxZssS57KOPPjJatWpleHh4lKgt+549e4zhw4cbdevWNby9vY2mTZsaY8eONbKysgzDKLpt75YtW4y4uDgjICDACA0NNW677TZna27H/o4ePWqMHTvWaNGiheHv728EBwcbnTt3Nt577z3ndtavX28MHTrUaNiwoeHt7W3Uq1fPuPbaa41ffvnFpUaVsRW7w+uvv25ccsklhre3txESEmL07NnTWLlypfP+H374wbjssssMX19fIzIy0njggQeML774otBxp6enG//85z+NWrVqGZKc7WuLallrGIbx1VdfGd26dTN8fX2NoKAgo1+/fsaWLVuKrPnIkSMuy4tro1yUzz77zJBk9O3b12X5rbfeakgy/vOf/xR6zNnPqWHkt8mtX7++YbVaXfYtyRg7dmyhbTRq1Oi8ranPbnHu4eFh1K5d2+jcubMxceJElzbPDme/53bu3GnccsstRnR0tOHj42PUrl3buOKKK4yvvvrK5XF//vmncfnllxu+vr4ubbPL2ordw8PDaNq0qXH//fc729qfXWd8fLwRHBxs+Pj4GNHR0cbIkSNd3r8jRoww/P39i3xuins/Fae44yvuPWQYhrF//35jwIABRq1atYzg4GBj4MCBRmJiYqHXv7hW7Ndcc02hbZ793BXXir1169aFHjtixIhCx7lz507jmmuuMXx9fY26desa9913n/Hf//7XkGSsXbv2nM/JiRMnjFGjRhmhoaFGQECAER8fb/z555+F3puO4zu77XZRtefl5RnTpk0zIiIiDF9fX6NXr17Gpk2bSvR+dzjXZ/rZhg0b5mxFXpSPP/7YaNOmjeHj42M0btzYeOqpp4zXX3/9vO9pwzCMHTt2GHFxcYa3t7cRFhZmPPTQQ8bKlSvL9JluGIaRm5tr3HXXXUbdunUNi8Xi8nlb1GfK+vXrjfj4eCMgIMDw8/MzrrjiCuPHH390Wac0r01Rivrcz8nJMaZNm2Y0adLE8PT0NKKiooyJEye6XFqjJD9/PvjgA6N3795GvXr1DC8vL6Nhw4bG7bffbhw8ePCcNRlG/iUb7r33XiMyMtLw9PQ0mjVrZjzzzDMu7ebP1K1btyJbyJ+pJO+r4r5vi1Oan2vFfZ4V9/1eVC1paWnGxIkTjYsuusjw8vIyQkNDja5duxozZ840srOzS1w3Kp/FMMrxjFkAAFBpZs2apXvvvVf79+9X/fr1zS4HAGo8whUAAFXAyZMnC12P65JLLlFeXp7++usvEysDADhwzhUAAFXADTfcoIYNG6pdu3ZKSUnRm2++qT///LPYttMAgMpHuAIAoAqIj4/Xa6+9prfeekt5eXlq1aqV3n333UId0wAA5mFaIAAAAACUA65zBQAAAADlgHAFAAAAAOWAc66KYLfblZiYqMDAwPNemBEAAABA9WUYhtLS0hQZGSmr9dxjU4SrIiQmJioqKsrsMgAAAAC4iX379qlBgwbnXIdwVYTAwEBJ+U9gUFCQydUAAAAAMEtqaqqioqKcGeFcCFdFcEwFDAoKIlwBAAAAKNHpQjS0AAAAAIByQLgCAAAAgHJAuAIAAACAcsA5VwAAAHB7eXl5ysnJMbsMVEM2m00eHh7lcgkmwhUAAADcWnp6uvbv3y/DMMwuBdWUn5+fIiIi5OXldUHbIVwBAADAbeXl5Wn//v3y8/NT3bp1y2V0AXAwDEPZ2dk6cuSIdu3apWbNmp33QsHnQrgCAACA28rJyZFhGKpbt658fX3NLgfVkK+vrzw9PbVnzx5lZ2fLx8enzNuioQUAAADcHiNWqEgXMlrlsp1y2QoAAAAA1HCEKwAAAAAoB4QrAAAAoApo3LixZs2aVeL1V69eLYvFouTk5AqrSZIWLlyoWrVqVeg+qgrCFQAAAFCOLBbLOW9Tp04t03bXrVun0aNHl3j9rl276uDBgwoODi7T/lB6dAsEAAAAytHBgwed/1+yZIkmT56sbdu2OZcFBAQ4/28YhvLy8uThcf5fy+vWrVuqOry8vBQeHl6qx+DCMHIFAACAKsMwDGVm55pyK+lFjMPDw5234OBgWSwW59d//vmnAgMD9fnnn6t9+/by9vbW999/rx07duj6669XWFiYAgIC1LFjR3311Vcu2z17WqDFYtFrr72mAQMGyM/PT82aNdPHH3/svP/saYGO6XtffPGFWrZsqYCAAPXp08clDObm5uruu+9WrVq1VKdOHT344IMaMWKE+vfvX6rXae7cuYqOjpaXl5eaN2+uxYsXu7yGU6dOVcOGDeXt7a3IyEjdfffdzvtffvllNWvWTD4+PgoLC9NNN91Uqn2biZErAAAAVBknc/LUavIXpux7y6Px8vMqn1+fJ0yYoJkzZ6pp06YKCQnRvn37dPXVV+vxxx+Xt7e33njjDfXr10/btm1Tw4YNi93OtGnT9PTTT+uZZ57R7NmzNWzYMO3Zs0e1a9cucv3MzEzNnDlTixcvltVq1f/93/9p/PjxeuuttyRJTz31lN566y0tWLBALVu21AsvvKBly5bpiiuuKPGxLV26VPfcc49mzZqluLg4ffrppxo1apQaNGigK664Qv/973/1/PPP691331Xr1q2VlJSk33//XZL0yy+/6O6779bixYvVtWtXHT9+XN99910pnllzEa4AAACASvboo4/qqquucn5du3ZttW3b1vn19OnTtXTpUn388ce68847i93OyJEjNXToUEnSE088oRdffFE///yz+vTpU+T6OTk5mjdvnqKjoyVJd955px599FHn/bNnz9bEiRM1YMAASdKcOXO0fPnyUh3bzJkzNXLkSI0ZM0aSlJCQoLVr12rmzJm64oortHfvXoWHhysuLk6enp5q2LChOnXqJEnau3ev/P39de211yowMFCNGjXSJZdcUqr9m4lw5eY2J6Zo4/4U9Y2NULCvp9nlAAAAmMrX06Ytj8abtu/y0qFDB5ev09PTNXXqVH322Wc6ePCgcnNzdfLkSe3du/ec22nTpo3z//7+/goKCtLhw4eLXd/Pz88ZrCQpIiLCuX5KSooOHTrkDDqSZLPZ1L59e9nt9hIf29atWws13ujWrZteeOEFSdLAgQM1a9YsNW3aVH369NHVV1+tfv36ycPDQ1dddZUaNWrkvK9Pnz7OaY9VAedcubmxb63XhA836o/9yWaXAgAAYDqLxSI/Lw9TbhaLpdyOw9/f3+Xr8ePHa+nSpXriiSf03XffacOGDYqNjVV2dvY5t+Pp6frHd4vFcs4gVNT6JT2XrLxERUVp27Ztevnll+Xr66sxY8bo8ssvV05OjgIDA7V+/Xq98847ioiI0OTJk9W2bdsKbydfXghXbi6mfn7rzI0HUkyuBAAAABXlhx9+0MiRIzVgwADFxsYqPDxcu3fvrtQagoODFRYWpnXr1jmX5eXlaf369aXaTsuWLfXDDz+4LPvhhx/UqlUr59e+vr7q16+fXnzxRa1evVpr1qzRxo0bJUkeHh6Ki4vT008/rT/++EO7d+/W119/fQFHVnmYFujmYusH69M/DmoT4QoAAKDaatasmT788EP169dPFotFkyZNKtVUvPJy1113acaMGbrooovUokULzZ49WydOnCjVqN3999+vQYMG6ZJLLlFcXJw++eQTffjhh87uhwsXLlReXp46d+4sPz8/vfnmm/L19VWjRo306aefaufOnbr88ssVEhKi5cuXy263q3nz5hV1yOWKcOXmHCNXmw6kmlwJAAAAKspzzz2nW265RV27dlVoaKgefPBBpaZW/u9/Dz74oJKSkjR8+HDZbDaNHj1a8fHxstlKfr5Z//799cILL2jmzJm655571KRJEy1YsEC9evWSJNWqVUtPPvmkEhISlJeXp9jYWH3yySeqU6eOatWqpQ8//FBTp07VqVOn1KxZM73zzjtq3bp1BR1x+bIYlT3JsgpITU1VcHCwUlJSFBQUZGotKZk5avvol5Kk3yf3VrAfTS0AAEDNcerUKe3atUtNmjSRj4+P2eXUOHa7XS1bttSgQYM0ffp0s8upMOd6n5UmG3DOlZsL9vNUVG1fSfmdAwEAAICKsmfPHs2fP19//fWXNm7cqDvuuEO7du3SP//5T7NLqxIIV1VALE0tAAAAUAmsVqsWLlyojh07qlu3btq4caO++uortWzZ0uzSqgTOuaoCYuoHa/nGJMIVAAAAKlRUVFShTn8oOUauqoCYyPyRq82JNLUAAAAA3BXhqgpwdAzcdTRDqadyTK4GAAAAQFEIV1VAbX8v1a+V39RiC6NXAAAAgFsiXFURMfXz2z5yMWEAAADAPRGuqgg6BgIAAADujXBVRbQuCFeMXAEAAADuiXBVRTg6Bu48mqH0rFyTqwEAAIDZdu/eLYvFog0bNlT4vhYuXKhatWpV+H4aN26sWbNmVfh+KgrhqoqoG+it8CAfGYa09SBNLQAAANzZyJEjZbFYCt369OljdmnnVVTAGTx4sP766y9zCqpCuIhwFRJTP1hJqae0cX+KOjaubXY5AAAAOIc+ffpowYIFLsu8vb1NqubC+Pr6ytfX1+wy3B4jV1VILOddAQCAms4wpOwMc26GUapSvb29FR4e7nILCQmRJP3zn//U4MGDXdbPyclRaGio3njjDUnSihUr1L17d9WqVUt16tTRtddeqx07dhS7v6Km7i1btkwWi8X59Y4dO3T99dcrLCxMAQEB6tixo7766ivn/b169dKePXt07733Okfbitv23LlzFR0dLS8vLzVv3lyLFy92ud9isei1117TgAED5Ofnp2bNmunjjz8u2ZNXYO/evbr++usVEBCgoKAgDRo0SIcOHXLe//vvv+uKK65QYGCggoKC1L59e/3yyy+SpD179qhfv34KCQmRv7+/WrdureXLl5dq/6XFyFUV4mzHnki4AgAANVROpvREpDn7fihR8vIvl00NGzZMAwcOVHp6ugICAiRJX3zxhTIzMzVgwABJUkZGhhISEtSmTRulp6dr8uTJGjBggDZs2CCrtWxjJOnp6br66qv1+OOPy9vbW2+88Yb69eunbdu2qWHDhvrwww/Vtm1bjR49Wrfddlux21m6dKnuuecezZo1S3Fxcfr00081atQoNWjQQFdccYVzvWnTpunpp5/WM888o9mzZ2vYsGHas2ePatc+/ywsu93uDFbffvutcnNzNXbsWA0ePFirV692Po+XXHKJ5s6dK5vNpg0bNsjT01OSNHbsWGVnZ+t///uf/P39tWXLFudzXVEIV1WIY+Tq78PpyszOlZ8XLx8AAIC7+vTTTwv9Mv/QQw/poYceUnx8vPz9/bV06VLdfPPNkqS3335b1113nQIDAyVJN954o8tjX3/9ddWtW1dbtmxRTExMmWpq27at2rZt6/x6+vTpWrp0qT7++GPdeeedql27tmw2mwIDAxUeHl7sdmbOnKmRI0dqzJgxkqSEhAStXbtWM2fOdAlXI0eO1NChQyVJTzzxhF588UX9/PPPJTr3bNWqVdq4caN27dqlqKgoSdIbb7yh1q1ba926derYsaP27t2r+++/Xy1atJAkNWvWzPn4vXv36sYbb1RsbKwkqWnTpiV9msqM386rkHpBPqob6K0jaVnaejBN7RuFmF0SAABA5fL0yx9BMmvfpXDFFVdo7ty5LsscIzYeHh4aNGiQ3nrrLd18883KyMjQRx99pHfffde57vbt2zV58mT99NNPOnr0qOx2u6T80FDWcJWenq6pU6fqs88+08GDB5Wbm6uTJ09q7969pdrO1q1bNXr0aJdl3bp10wsvvOCyrE2bNs7/+/v7KygoSIcPHy7xPqKiopzBSpJatWqlWrVqaevWrerYsaMSEhJ06623avHixYqLi9PAgQMVHR0tSbr77rt1xx136Msvv1RcXJxuvPFGl3oqAudcVTGcdwUAAGo0iyV/ap4ZtzPOXSoJf39/XXTRRS63M6fDDRs2TKtWrdLhw4e1bNky+fr6uozo9OvXT8ePH9f8+fP1008/6aeffpIkZWdnF7k/q9Uq46zzwnJycly+Hj9+vJYuXaonnnhC3333nTZs2KDY2Nhit3mhHFP0HCwWizMkloepU6dq8+bNuuaaa/T111+rVatWWrp0qSTp1ltv1c6dO3XzzTdr48aN6tChg2bPnl1u+y4K4aqKiSkIVxsJVwAAAFVa165dFRUVpSVLluitt97SwIEDnWHk2LFj2rZtmx555BFdeeWVatmypU6cOHHO7dWtW1dpaWnKyMhwLjv7Glg//PCDRo4cqQEDBig2Nlbh4eHavXu3yzpeXl7Ky8s7575atmypH374odC2W7VqdZ6jLrmWLVtq37592rdvn3PZli1blJyc7LKfiy++WPfee6++/PJL3XDDDS4dGqOiovTvf/9bH374oe677z7Nnz+/3OorCtMCq5iYyIKmFoQrAAAAt5aVlaWkpCSXZR4eHgoNDXV+/c9//lPz5s3TX3/9pW+++ca5PCQkRHXq1NGrr76qiIgI7d27VxMmTDjn/jp37iw/Pz899NBDuvvuu/XTTz9p4cKFLus0a9ZMH374ofr16yeLxaJJkyYVGklq3Lix/ve//2nIkCHy9vZ2qdfh/vvv16BBg3TJJZcoLi5On3zyiT788EOXzoMXKi4uTrGxsRo2bJhmzZql3NxcjRkzRj179lSHDh108uRJ3X///brpppvUpEkT7d+/X+vWrXOeqzZu3Dj17dtXF198sU6cOKFvvvlGLVu2LLf6isLIVRUT2yB/5Gr74XSdyjn3XxQAAABgnhUrVigiIsLl1r17d5d1hg0bpi1btqh+/frq1q2bc7nVatW7776rX3/9VTExMbr33nv1zDPPnHN/tWvX1ptvvqnly5crNjZW77zzjqZOneqyznPPPaeQkBB17dpV/fr1U3x8vC699FKXdR599FHt3r1b0dHRqlu3bpH76t+/v1544QXNnDlTrVu31iuvvKIFCxaoV69eJX+CzsNiseijjz5SSEiILr/8csXFxalp06ZasmSJJMlms+nYsWMaPny4Lr74Yg0aNEh9+/bVtGnTJEl5eXkaO3asWrZsqT59+ujiiy/Wyy+/XG71FVmzcfbETCg1NVXBwcFKSUlRUFCQ2eW4MAxDHR77SscysrVsbDe1i6pldkkAAAAV5tSpU9q1a5eaNGkiHx8fs8tBNXWu91lpsgEjV1WMxWLhvCsAAADADRGuqiBnx8D9hCsAAADAXRCuqqCY+gVNLRIJVwAAAIC7IFxVQY5pgX8dSlNWLk0tAAAAAHdAuKqC6tfyVS0/T+XkGforKd3scgAAACocPdhQkcrr/UW4qoIsFovzvCuaWgAAgOrMZrNJkrKzs02uBNVZZmamJDkv4lxWXES4ioqpH6zvth8lXAEAgGrNw8NDfn5+OnLkiDw9PWW1MjaA8mMYhjIzM3X48GHVqlXLGebLyi3C1UsvvaRnnnlGSUlJatu2rWbPnq1OnToVue6HH36oJ554Qn///bdycnLUrFkz3Xfffbr55pud6xiGoSlTpmj+/PlKTk5Wt27dNHfuXDVr1qyyDqnCxUTmj1xtpqkFAACoxiwWiyIiIrRr1y7t2bPH7HJQTdWqVUvh4eEXvB3Tw9WSJUuUkJCgefPmqXPnzpo1a5bi4+O1bds21atXr9D6tWvX1sMPP6wWLVrIy8tLn376qUaNGqV69eopPj5ekvT000/rxRdf1KJFi9SkSRNNmjRJ8fHx2rJlS7W5+JxjWuCfB9OUnWuXlwd/xQEAANWTl5eXmjVrxtRAVAhPT88LHrFysBgmnx3YuXNndezYUXPmzJEk2e12RUVF6a677tKECRNKtI1LL71U11xzjaZPny7DMBQZGan77rtP48ePlySlpKQoLCxMCxcu1JAhQ867vdJchdkshmGo7bQvlXoqV5/d3V2tC0ayAAAAAJSf0mQDU4c7srOz9euvvyouLs65zGq1Ki4uTmvWrDnv4w3D0KpVq7Rt2zZdfvnlkqRdu3YpKSnJZZvBwcHq3LlzsdvMyspSamqqy83dWSwWZ0v2TZx3BQAAAJjO1HB19OhR5eXlKSwszGV5WFiYkpKSin1cSkqKAgIC5OXlpWuuuUazZ8/WVVddJUnOx5VmmzNmzFBwcLDzFhUVdSGHVWnoGAgAAAC4jyp5ok5gYKA2bNigdevW6fHHH1dCQoJWr15d5u1NnDhRKSkpztu+ffvKr9gK1No5cuX+I20AAABAdWdqQ4vQ0FDZbDYdOnTIZfmhQ4fO2a3DarXqoosukiS1a9dOW7du1YwZM9SrVy/n4w4dOqSIiAiXbbZr167I7Xl7e8vb2/sCj6byOUauth5MVW6eXR62KpmVAQAAgGrB1N/Gvby81L59e61atcq5zG63a9WqVerSpUuJt2O325WVlSVJatKkicLDw122mZqaqp9++qlU26wKGtX2U4C3h7Jy7fr7SLrZ5QAAAAA1mumt2BMSEjRixAh16NBBnTp10qxZs5SRkaFRo0ZJkoYPH6769etrxowZkvLPj+rQoYOio6OVlZWl5cuXa/HixZo7d66k/EYP48aN02OPPaZmzZo5W7FHRkaqf//+Zh1mhbBaLWodGaSfdh3Xxv0pahHunp0NAQAAgJrA9HA1ePBgHTlyRJMnT1ZSUpLatWunFStWOBtS7N271+VK3BkZGRozZoz2798vX19ftWjRQm+++aYGDx7sXOeBBx5QRkaGRo8ereTkZHXv3l0rVqyoNte4OlNs/WD9tOu4Nh1I0cAOVaMRBwAAAFAdmX6dK3dUFa5z5bDstwMat2SD2jcK0X/v6Gp2OQAAAEC1UmWuc4UL57jW1ZbEVOXZyckAAACAWQhXVVyTUH/5edl0MidPO2lqAQAAAJiGcFXF2QqaWkhcTBgAAAAwE+GqGnBMDSRcAQAAAOYhXFUDMZH54WrzgVSTKwEAAABqLsJVNRDboCBcJabITlMLAAAAwBSEq2qgaai/fDytysjO065jGWaXAwAAANRIhKtqwMNmVauI/KYWmzjvCgAAADAF4aqaiHU0tdhPuAIAAADMQLiqJloXhKtNiYQrAAAAwAyEq2rCMXK1+UAqTS0AAAAAExCuqomL6gXIy8OqtKxc7T2eaXY5AAAAQI1DuKomPG1WtSxoasHFhAEAAIDKR7iqRmLr0zEQAAAAMAvhqhqJiaSpBQAAAGAWwlU1EuPoGHggVYZBUwsAAACgMhGuqpGLwwLlZbMq5WSO9p84aXY5AAAAQI1CuKpGvDysah4eKImmFgAAAEBlI1xVMzH16RgIAAAAmIFwVc2cPu+KcAUAAABUJsJVNRN7RriiqQUAAABQeQhX1czFYYHysFp0IjNHiSmnzC4HAAAAqDEIV9WMj6dNF4cVNLXYz9RAAAAAoLIQrqohR1MLzrsCAAAAKg/hqhpynneVSLgCAAAAKgvhqhqKoakFAAAAUOkIV9VQy4gg2awWHU3P1qHULLPLAQAAAGoEwlU15ONpU7N6AZK4mDAAAABQWQhX1VTryPypgYQrAAAAoHIQrqqp2IKOgZsJVwAAAEClIFxVU7ENGLkCAAAAKhPhqppqGREkq0U6nJalw6mnzC4HAAAAqPYIV9WUn5eHouvmN7XgelcAAABAxSNcVWOO611t3J9qciUAAABA9Ue4qsacFxNm5AoAAACocISraizWEa5oagEAAABUOMJVNdYqMkgWi3Qw5ZSOpmeZXQ4AAABQrRGuqrEAbw81CfWXxOgVAAAAUNEIV9VcTCRTAwEAAIDKQLiq5k6fd0XHQAAAAKAiEa6qOWc7dkauAAAAgApFuKrmWtcPkiQdSD6pExnZJlcDAAAAVF+Eq2ouyMdTjev4SeJ6VwAAAEBFIlzVAK2ZGggAAABUOMJVDeBoarGZphYAAABAhSFc1QCxjFwBAAAAFY5wVQO0jsxvarH3eKZSMnNMrgYAAAConghXNUAtPy9F1faVJG2mqQUAAABQIQhXNURMJFMDAQAAgIpEuKohHBcT3pRIUwsAAACgIhCuaghHU4tNjFwBAAAAFYJwVUM4Rq52Hc1Q2imaWgAAAADljXBVQ9T291L9Wo6mFkwNBAAAAMqbW4Srl156SY0bN5aPj486d+6sn3/+udh158+frx49eigkJEQhISGKi4srtP7IkSNlsVhcbn369Know3B7jpbsTA0EAAAAyp/p4WrJkiVKSEjQlClTtH79erVt21bx8fE6fPhwkeuvXr1aQ4cO1TfffKM1a9YoKipKvXv31oEDB1zW69Onjw4ePOi8vfPOO5VxOG6N864AAACAimN6uHruued02223adSoUWrVqpXmzZsnPz8/vf7660Wu/9Zbb2nMmDFq166dWrRooddee012u12rVq1yWc/b21vh4eHOW0hISGUcjluLaUA7dgAAAKCimBqusrOz9euvvyouLs65zGq1Ki4uTmvWrCnRNjIzM5WTk6PatWu7LF+9erXq1aun5s2b64477tCxY8eK3UZWVpZSU1NdbtWR41pXO49mKCMr1+RqAAAAgOrF1HB19OhR5eXlKSwszGV5WFiYkpKSSrSNBx98UJGRkS4BrU+fPnrjjTe0atUqPfXUU/r222/Vt29f5eXlFbmNGTNmKDg42HmLiooq+0G5sbqB3goP8pFhSFsOVs8ACQAAAJjFw+wCLsSTTz6pd999V6tXr5aPj49z+ZAhQ5z/j42NVZs2bRQdHa3Vq1fryiuvLLSdiRMnKiEhwfl1ampqtQ1YMfWDlJR6Shv3p6hj49rnfwAAAACAEjF15Co0NFQ2m02HDh1yWX7o0CGFh4ef87EzZ87Uk08+qS+//FJt2rQ557pNmzZVaGio/v777yLv9/b2VlBQkMutunJc72pTIuddAQAAAOXJ1HDl5eWl9u3buzSjcDSn6NKlS7GPe/rppzV9+nStWLFCHTp0OO9+9u/fr2PHjikiIqJc6q7K6BgIAAAAVAzTuwUmJCRo/vz5WrRokbZu3ao77rhDGRkZGjVqlCRp+PDhmjhxonP9p556SpMmTdLrr7+uxo0bKykpSUlJSUpPT5ckpaen6/7779fatWu1e/durVq1Stdff70uuugixcfHm3KM7sQxcvX34XSdzC76HDQAAAAApWf6OVeDBw/WkSNHNHnyZCUlJaldu3ZasWKFs8nF3r17ZbWezoBz585Vdna2brrpJpftTJkyRVOnTpXNZtMff/yhRYsWKTk5WZGRkerdu7emT58ub2/vSj02dxQW5KO6gd46kpalLQdT1b4RLeoBAACA8mAxDMMwuwh3k5qaquDgYKWkpFTL869GLfhZ32w7omnXtdaIro3NLgcAAABwW6XJBqZPC0Tl47wrAAAAoPwRrmogx3lXGwlXAAAAQLkhXNVAjnC1/XC6TuXQ1AIAAAAoD4SrGigi2Ed1/L2UZzf0Z1Ka2eUAAAAA1QLhqgayWCxqzdRAAAAAoFwRrmqo2Pr5nU42E64AAACAckG4qqFiGbkCAAAAyhXhqoZqHZkfrv46lKasXJpaAAAAABeKcFVDNQjxVS0/T+XkGforKd3scgAAAIAqj3BVQ1ksFsVEMjUQAAAAKC+EqxrMcb2rTYmEKwAAAOBCEa5qMEdTi02MXAEAAAAXjHBVg8UUtGP/82CacvLsJlcDAAAAVG2EqxqsYW0/Bfp4KDvPrr8OpZldDgAAAFClEa5qsDObWjA1EAAAALgwhKsaLraBI1ylmlwJAAAAULURrmo4R8dA2rEDAAAAF4ZwVcPFROY3tdh6MFW5NLUAAAAAyoxwVcM1ruOvAG8PZeXa9feRdLPLAQAAAKoswlUNZ7Va1Kpg9GrjfqYGAgAAAGVFuILzYsKbE2lqAQAAAJQV4QrOcEVTCwAAAKDsCFdQTP38aYFbElOVZzdMrgYAAAComghXUJPQAPl52XQyJ087aWoBAAAAlAnhCrJZLWoVUdDUgqmBAAAAQJkQriDp9MWENx2gqQUAAABQFoQrSDrd1GITI1cAAABAmRCuIOn0yNXmxBTZaWoBAAAAlBrhCpKk6Lr+8vG0KiM7T7uOZZhdDgAAAFDlEK4gSfKwWdWyoKkFUwMBAACA0iNcwYnzrgAAAICyI1zByXHeFe3YAQAAgNIjXMEpJrKgqcWBVJpaAAAAAKVEuIJTs7AAeXlYlZaVq73HM80uBwAAAKhSCFdw8rRZ1TI8UBJTAwEAAIDSIlzBheO8q02JhCsAAACgNAhXcBFDx0AAAACgTAhXcHG6HXuqDIOmFgAAAEBJEa7g4uKwQHnaLEo5maP9J06aXQ4AAABQZRCu4MLLw6rmNLUAAAAASo1whUJiOe8KAAAAKDXCFQppXXAxYUauAAAAgJIjXKEQx8jV5kSaWgAAAAAlRbhCIc3DA+Vhteh4RrYSU06ZXQ4AAABQJRCuUIiPp03NwgqaWuxnaiAAAABQEoQrFCm2fpAkaXMi4QoAAAAoCcIVihRTn6YWAAAAQGkQrlCkmDPasdPUAgAAADg/whWK1CoiSDarRUfTs3UoNcvscgAAAAC3R7hCkXw8bbqoboAkpgYCAAAAJUG4QrHOnBoIAAAA4NwIVyhWTEHHQMIVAAAAcH6EKxQr1jFyRTt2AAAA4LzcIly99NJLaty4sXx8fNS5c2f9/PPPxa47f/589ejRQyEhIQoJCVFcXFyh9Q3D0OTJkxURESFfX1/FxcVp+/btFX0Y1U6ryCBZLNKh1CwdTjtldjkAAACAWzM9XC1ZskQJCQmaMmWK1q9fr7Zt2yo+Pl6HDx8ucv3Vq1dr6NCh+uabb7RmzRpFRUWpd+/eOnDggHOdp59+Wi+++KLmzZunn376Sf7+/oqPj9epUwSE0vDz8lB0QVMLpgYCAAAA52YxTL6IUefOndWxY0fNmTNHkmS32xUVFaW77rpLEyZMOO/j8/LyFBISojlz5mj48OEyDEORkZG67777NH78eElSSkqKwsLCtHDhQg0ZMuS820xNTVVwcLBSUlIUFBR0YQdYxd27ZIOW/nZACVddrLuvbGZ2OQAAAEClKk02MHXkKjs7W7/++qvi4uKcy6xWq+Li4rRmzZoSbSMzM1M5OTmqXbu2JGnXrl1KSkpy2WZwcLA6d+5c7DazsrKUmprqckO+1pH5byDasQMAAADnZmq4Onr0qPLy8hQWFuayPCwsTElJSSXaxoMPPqjIyEhnmHI8rjTbnDFjhoKDg523qKio0h5KteVoarGZcAUAAACck+nnXF2IJ598Uu+++66WLl0qHx+fMm9n4sSJSklJcd727dtXjlVWba0LwlViyikdS88yuRoAAADAfZkarkJDQ2Wz2XTo0CGX5YcOHVJ4ePg5Hztz5kw9+eST+vLLL9WmTRvncsfjSrNNb29vBQUFudyQL8DbQ01D/SUxNRAAAAA4F1PDlZeXl9q3b69Vq1Y5l9ntdq1atUpdunQp9nFPP/20pk+frhUrVqhDhw4u9zVp0kTh4eEu20xNTdVPP/10zm2ieDGOqYGJnIsGAAAAFMf0aYEJCQmaP3++Fi1apK1bt+qOO+5QRkaGRo0aJUkaPny4Jk6c6Fz/qaee0qRJk/T666+rcePGSkpKUlJSktLT0yVJFotF48aN02OPPaaPP/5YGzdu1PDhwxUZGan+/fubcYhVXkz9gqYW+xm5AgAAAIrjYXYBgwcP1pEjRzR58mQlJSWpXbt2WrFihbMhxd69e2W1ns6Ac+fOVXZ2tm666SaX7UyZMkVTp06VJD3wwAPKyMjQ6NGjlZycrO7du2vFihUXdF5WTeYYudqUSLgCAAAAimP6da7cEde5cpVyMkdtp30pSfpt0lUK8fcyuSIAAACgclSZ61yhagj29VSjOn6SGL0CAAAAikO4Qok4pwYeoKkFAAAAUBTCFUokJtIRrhi5AgAAAIpCuEKJxNLUAgAAADgnwhVKxNGOfc+xTKWczDG5GgAAAMD9EK5QIrX8vNQgxFeStJmpgQAAAEAhhCuUGFMDAQAAgOIRrlBijo6BG+kYCAAAABRCuEKJOcIV0wIBAACAwghXKLGYyPymFjuPZijtFE0tAAAAgDMRrlBidQK8FRnsI0nanMjUQAAAAOBMhCuUimNqIBcTBgAAAFwRrlAqhCsAAACgaIQrlMrpduxMCwQAAADORLhCqThGrnYcSVdGVq7J1QAAAADug3CFUqkb6K2wIG8ZhrTlIKNXAAAAgAPhCqUWy3lXAAAAQCGEK5Ra68j8cLWRcAUAAAA4Ea5Qao6Rq80HmBYIAAAAOBCuUGqxDfLD1fbDaTqZnWdyNQAAAIB7IFyh1OoFeis0wFt2mloAAAAAToQrlJrFYlFs/SBJ0uZEzrsCAAAAJMIVyshxvauN+wlXAAAAgES4Qhk5wtWmRKYFAgAAABLhCmXk6Bi4/VCaTuXQ1AIAAAAgXKFMIoJ9VNvfS7l2Q38mpZldDgAAAGA6whXKxGKxnJ4ayMWEAQAAgLKFq3379mn//v3Or3/++WeNGzdOr776arkVBvcXE5nfMZBwBQAAAJQxXP3zn//UN998I0lKSkrSVVddpZ9//lkPP/ywHn300XItEO4r1tnUgnAFAAAAlClcbdq0SZ06dZIkvffee4qJidGPP/6ot956SwsXLizP+uDGHNMCtyWlKSuXphYAAACo2coUrnJycuTt7S1J+uqrr3TddddJklq0aKGDBw+WX3Vwaw1CfBXs66mcPEN/JaWbXQ4AAABgqjKFq9atW2vevHn67rvvtHLlSvXp00eSlJiYqDp16pRrgXBfFouFqYEAAABAgTKFq6eeekqvvPKKevXqpaFDh6pt27aSpI8//tg5XRA1Q+v6+U0tNtLUAgAAADWcR1ke1KtXLx09elSpqakKCQlxLh89erT8/PzKrTi4P8fI1WbCFQAAAGq4Mo1cnTx5UllZWc5gtWfPHs2aNUvbtm1TvXr1yrVAuDdHuNqalKacPLvJ1QAAAADmKVO4uv766/XGG29IkpKTk9W5c2c9++yz6t+/v+bOnVuuBcK9Naztp0AfD2Xn2vXXoTSzywEAAABMU6ZwtX79evXo0UOS9MEHHygsLEx79uzRG2+8oRdffLFcC4R7s1gsiol0TA1MNbkaAAAAwDxlCleZmZkKDAyUJH355Ze64YYbZLVaddlll2nPnj3lWiDcXwxNLQAAAICyhauLLrpIy5Yt0759+/TFF1+od+/ekqTDhw8rKCioXAuE+4uhHTsAAABQtnA1efJkjR8/Xo0bN1anTp3UpUsXSfmjWJdcckm5Fgj352xqcTBVuTS1AAAAQA1VplbsN910k7p3766DBw86r3ElSVdeeaUGDBhQbsWhamhcx18B3h5Kz8rV30fS1SKc0UsAAADUPGUKV5IUHh6u8PBw7d+/X5LUoEEDLiBcQ1mtFrWKDNLPu45r04FUwhUAAABqpDJNC7Tb7Xr00UcVHBysRo0aqVGjRqpVq5amT58uu51pYTWRo2PgJppaAAAAoIYq08jVww8/rP/85z968skn1a1bN0nS999/r6lTp+rUqVN6/PHHy7VIuL/YBvmjVYQrAAAA1FRlCleLFi3Sa6+9puuuu865rE2bNqpfv77GjBlDuKqBHE0tNiemKs9uyGa1mFwRAAAAULnKNC3w+PHjatGiRaHlLVq00PHjxy+4KFQ9TUID5Odl08mcPO08km52OQAAAEClK1O4atu2rebMmVNo+Zw5c9SmTZsLLgpVj81qUauIgqmBXO8KAAAANVCZpgU+/fTTuuaaa/TVV185r3G1Zs0a7du3T8uXLy/XAlF1xNQP1i97Tmjj/lQN4HJnAAAAqGHKNHLVs2dP/fXXXxowYICSk5OVnJysG264QZs3b9bixYvLu0ZUETEF510xcgUAAICayGIYhlFeG/v999916aWXKi8vr7w2aYrU1FQFBwcrJSVFQUFcs6mktiWlKX7W/xTg7aE/pvSWlaYWAAAAqOJKkw3KNHIFFCW6rr98PK1Kz8rVrmMZZpcDAAAAVCrCFcqNh82qlhFc7woAAAA1E+EK5SomsuC8K8IVAAAAaphSdQu84YYbznl/cnJyqQt46aWX9MwzzygpKUlt27bV7Nmz1alTpyLX3bx5syZPnqxff/1Ve/bs0fPPP69x48a5rDN16lRNmzbNZVnz5s31559/lro2lJ7jYsKbDqSaXAkAAABQuUoVroKDg897//Dhw0u8vSVLlighIUHz5s1T586dNWvWLMXHx2vbtm2qV69eofUzMzPVtGlTDRw4UPfee2+x223durW++uor59ceHmXqOI8yOLNjoGEYslhoagEAAICaoVSpY8GCBeW68+eee0633XabRo0aJUmaN2+ePvvsM73++uuaMGFCofU7duyojh07SlKR9zt4eHgoPDy8XGtFyTQLC5CXh1Vpp3K151imGof6m10SAAAAUClMO+cqOztbv/76q+Li4k4XY7UqLi5Oa9asuaBtb9++XZGRkWratKmGDRumvXv3nnP9rKwspaamutxQNp42q1qGB0rielcAAACoWUwLV0ePHlVeXp7CwsJcloeFhSkpKanM2+3cubMWLlyoFStWaO7cudq1a5d69OihtLS0Yh8zY8YMBQcHO29RUVFl3j+k1gVTAzfS1AIAAAA1SLXrFti3b18NHDhQbdq0UXx8vJYvX67k5GS99957xT5m4sSJSklJcd727dtXiRVXP46mFptpagEAAIAaxLROD6GhobLZbDp06JDL8kOHDpXr+VK1atXSxRdfrL///rvYdby9veXt7V1u+6zpHO3YNx6gqQUAAABqDtNGrry8vNS+fXutWrXKucxut2vVqlXq0qVLue0nPT1dO3bsUERERLltE+d2cXiAPG0WpZzM0f4TJ80uBwAAAKgUpk4LTEhI0Pz587Vo0SJt3bpVd9xxhzIyMpzdA4cPH66JEyc618/OztaGDRu0YcMGZWdn68CBA9qwYYPLqNT48eP17bffavfu3frxxx81YMAA2Ww2DR06tNKPr6by9rCpuaOpBeddAQAAoIYw9QJQgwcP1pEjRzR58mQlJSWpXbt2WrFihbPJxd69e2W1ns5/iYmJuuSSS5xfz5w5UzNnzlTPnj21evVqSdL+/fs1dOhQHTt2THXr1lX37t21du1a1a1bt1KPraaLiQzWpgOp2nggRX1jGTUEAABA9WcxDMMwuwh3k5qaquDgYKWkpCgoKMjscqqkN9fu0SPLNunyi+vqjVs6mV0OAAAAUCalyQbVrlsg3ENMQcfATQVNLQAAAIDqjnCFCtEiPFAeVouOZ2QrMeWU2eUAAAAAFY5whQrh42lTszCaWgAAAKDmIFyhwsRE5s9JJVwBAACgJiBcocLENjh93hUAAABQ3RGuUGFaR+aHq40HUmlqAQAAgGqPcIUK0yoiSFaLdDQ9S4dSs8wuBwAAAKhQhCtUGF8vm5rVo6kFAAAAagbCFSpU6/r5TS02Eq4AAABQzRGuUKFiCy4mvDmRcAUAAIDqjXCFChVT39HUgnAFAACA6o1whQrVKiJIFot0KDVLh9NOmV0OAAAAUGEIV6hQ/t4eiq4bIEnafCDV5GoAAACAikO4QoWLiaSpBQAAAKo/whUqnOO8K9qxAwAAoDojXKHCEa4AAABQExCuUOFaF0wLTEw5pWPpWSZXAwAAAFQMwhUqXKCPp5qG+kuSNiXS1AIAAADVE+EKlaI1UwMBAABQzRGuUCli6+dPDSRcAQAAoLoiXKFSxETmj1zRjh0AAADVFeEKlcIxLXD/iZM6kZFtcjUAAABA+SNcoVIE+3qqUR0/SdJmmloAAACgGiJcodIwNRAAAADVGeEKlcZ5MeFEwhUAAACqH8IVKk0MHQMBAABQjRGuUGkc0wL3HMtUyskck6sBAAAAyhfhCpUmxN9LDUJ8JUmbmRoIAACAaoZwhUrlGL1iaiAAAACqG8IVKlVsA0e4oh07AAAAqhfCFSpV60iaWgAAAKB6IlyhUsUWtGPfeTRDaadoagEAAIDqg3CFSlUnwFuRwT6SpC2JTA0EAABA9UG4QqVrXTB6tZGpgQAAAKhGCFeodI6pgZsZuQIAAEA1QrhCpYupn9/UgpErAAAAVCeEK1S6mIKRqx1H0pWRlWtyNQAAAED5IFyh0tUL9FFYkLcMQ9p6kKmBAAAAqB4IVzBFTCRNLQAAAFC9EK5gCsfUwE0HGLkCAABA9UC4gilOhytGrgAAAFA9EK5gCkc79u2H03QyO8/kagAAAIALR7iCKcKCvBUa4C27IW1NYmogAAAAqj7CFUxhsVic17tiaiAAAACqA8IVTBPLeVcAAACoRghXME1rZzt2pgUCAACg6iNcwTSxDQqaWhxK06kcmloAAACgaiNcwTSRwT6q7e+lXLuhbUlpZpcDAAAAXBDCFUxjsVjUOjK/qcVGzrsCAABAFUe4gqkcTS02JxKuAAAAULURrmCqmPqOphaEKwAAAFRthCuYyjFytS0pTVm5NLUAAABA1UW4gqkahPgq2NdTOXmGth9KN7scAAAAoMxMD1cvvfSSGjduLB8fH3Xu3Fk///xzsetu3rxZN954oxo3biyLxaJZs2Zd8DZhLovFopj6NLUAAABA1WdquFqyZIkSEhI0ZcoUrV+/Xm3btlV8fLwOHz5c5PqZmZlq2rSpnnzySYWHh5fLNmE+x3lXmwhXAAAAqMJMDVfPPfecbrvtNo0aNUqtWrXSvHnz5Ofnp9dff73I9Tt27KhnnnlGQ4YMkbe3d7lsE+aLiSRcAQAAoOozLVxlZ2fr119/VVxc3OlirFbFxcVpzZo1lbrNrKwspaamutxQeRxNLbYmpSknz25yNQAAAEDZmBaujh49qry8PIWFhbksDwsLU1JSUqVuc8aMGQoODnbeoqKiyrR/lE2jOn4K9PFQdq6dphYAAACoskxvaOEOJk6cqJSUFOdt3759ZpdUo1gsFrWOzG9qwdRAAAAAVFWmhavQ0FDZbDYdOnTIZfmhQ4eKbVZRUdv09vZWUFCQyw2VyzE1cFMi4QoAAABVk2nhysvLS+3bt9eqVaucy+x2u1atWqUuXbq4zTZRORwdA2nHDgAAgKrKw8ydJyQkaMSIEerQoYM6deqkWbNmKSMjQ6NGjZIkDR8+XPXr19eMGTMk5Tes2LJli/P/Bw4c0IYNGxQQEKCLLrqoRNuEe3KEq60HU5WbZ5eHjRmrAAAAqFpMDVeDBw/WkSNHNHnyZCUlJaldu3ZasWKFsyHF3r17ZbWe/iU7MTFRl1xyifPrmTNnaubMmerZs6dWr15dom3CPTWp4y9/L5sysvO040iGmocHml0SAAAAUCoWwzAMs4twN6mpqQoODlZKSgrnX1WiQfPW6OfdxzVzYFvd1L6B2eUAAAAApcoGzL2C23BMDaRjIAAAAKoiwhXcRkx92rEDAACg6iJcwW042rFvOZiqPDuzVQEAAFC1EK7gNprWDZCvp02Z2XnadTTd7HIAAACAUiFcwW3YrBa1isyfGsj1rgAAAFDVEK7gVmKdTS1STa4EAAAAKB3CFdxKa0auAAAAUEURruBWYhsUNLVITJWdphYAAACoQghXcCsX1Q2Qt4dV6Vm52n0sw+xyAAAAgBIjXMGteNisahnB1EAAAABUPYQruB1HU4vNiTS1AAAAQNVBuILbialfMHK1n5ErAAAAVB2EK7idGEc79sQUGQZNLQAAAFA1EK7gdprVC5SXzaq0U7naezzT7HIAAACAEiFcwe14eVjVIiJQEk0tAAAAUHUQruCWnFMDD9DUAgAAAFUD4QpuKSbSEa4YuQIAAEDVQLiCW4qlqQUAAACqGMIV3NLF4QHytFmUnJmj/SdOml0OAAAAcF6EK7glbw+bLg7Lb2rB1EAAAABUBYQruK0zpwYCAAAA7o5wBbfVuiBcbaRjIAAAAKoAwhXclmPkavMBmloAAADA/RGu4LZahAfKZrXoWEa2DqacMrscAAAA4JwIV3BbPp42NasXIEnaSFMLAAAAuDnCFdzamVMDAQAAAHdGuIJbi3E2tSBcAQAAwL0RruDWYpzt2OkYCAAAAPdGuIJbaxURJKtFOpKWpUOpNLUAAACA+yJcwa35etl0kaOpxX6mBgIAAMB9Ea7g9k5PDSRcAQAAwH0RruD2YiILwhVNLQAAAODGCFdwe7ENHOGKphYAAABwX4QruL1WEUGyWKSk1FM6kpZldjkAAABAkQhXcHv+3h5qGuoviamBAAAAcF+EK1QJsfU57woAAADujXCFKsHRMXAj4QoAAABuinCFKsERrjYn0tQCAAAA7olwhSqhVWSQJOlA8kkdz8g2uRoAAACgMMIVqoQgH081KWhqwdRAAAAAuCPCFaqMGJpaAAAAwI0RrlBlxBRMDSRcAQAAwB0RrlBlONuxJxKuAAAA4H4IV6gyWkfmh6t9x08qOZOmFgAAAHAvhCtUGcF+nmpY20+StOkALdkBAADgXghXqFKYGggAAAB3RbhCldK6fn5TC9qxAwAAwN0QrlClOEauNhOuAAAA4GYIV6hSYgqaWuw+lqnUUzkmVwMAAACcRrhClRLi76X6tXwlcb0rAAAAuBfCFaqc01MD6RgIAAAA90G4QpUTQ1MLAAAAuCHCFaqcGNqxAwAAwA25Rbh66aWX1LhxY/n4+Khz5876+eefz7n++++/rxYtWsjHx0exsbFavny5y/0jR46UxWJxufXp06ciDwGVyBGudh3NUHpWrsnVAAAAAPlMD1dLlixRQkKCpkyZovXr16tt27aKj4/X4cOHi1z/xx9/1NChQ/Wvf/1Lv/32m/r376/+/ftr06ZNLuv16dNHBw8edN7eeeedyjgcVILQAG9FBPvIMGjJDgAAAPdherh67rnndNttt2nUqFFq1aqV5s2bJz8/P73++utFrv/CCy+oT58+uv/++9WyZUtNnz5dl156qebMmeOynre3t8LDw523kJCQyjgcVJLTUwNpagEAAAD3YGq4ys7O1q+//qq4uDjnMqvVqri4OK1Zs6bIx6xZs8ZlfUmKj48vtP7q1atVr149NW/eXHfccYeOHTtWbB1ZWVlKTU11ucG9Oa53RTt2AAAAuAtTw9XRo0eVl5ensLAwl+VhYWFKSkoq8jFJSUnnXb9Pnz564403tGrVKj311FP69ttv1bdvX+Xl5RW5zRkzZig4ONh5i4qKusAjQ0WLbZDfMZBwBQAAAHfhYXYBFWHIkCHO/8fGxqpNmzaKjo7W6tWrdeWVVxZaf+LEiUpISHB+nZqaSsByc46Rqx1H0pWZnSs/r2r5VgYAAEAVYurIVWhoqGw2mw4dOuSy/NChQwoPDy/yMeHh4aVaX5KaNm2q0NBQ/f3330Xe7+3traCgIJcb3Fu9IB/VC/SW3ZC2cN4VAAAA3ICp4crLy0vt27fXqlWrnMvsdrtWrVqlLl26FPmYLl26uKwvSStXrix2fUnav3+/jh07poiIiPIpHG4htj7nXQEAAMB9mN4tMCEhQfPnz9eiRYu0detW3XHHHcrIyNCoUaMkScOHD9fEiROd699zzz1asWKFnn32Wf3555+aOnWqfvnlF915552SpPT0dN1///1au3atdu/erVWrVun666/XRRddpPj4eFOOERWjdUG42niAkSsAAACYz/QTVQYPHqwjR45o8uTJSkpKUrt27bRixQpn04q9e/fKaj2dAbt27aq3335bjzzyiB566CE1a9ZMy5YtU0xMjCTJZrPpjz/+0KJFi5ScnKzIyEj17t1b06dPl7e3tynHiIrhGLnanMjIFQAAAMxnMQzDMLsId5Oamqrg4GClpKRw/pUbO5hyUl1mfC2b1aLN0+Ll42kzuyQAAABUM6XJBqZPCwTKKjzIR6EBXsqzG9pykKmBAAAAMBfhClWWxWJRjGNqIE0tAAAAYDLCFao0x/WuNhKuAAAAYDLCFaq0GGc7dqYFAgAAwFyEK1RpMfXzTyr861CaTuXkmVwNAAAAajLClbvb9rn0w4tSbpbZlbil+rV8FeLnqVy7oW1JaWaXAwAAgBqMcOXOcrOlFROllZOkOR2kTf+V6Jzv4symFpu43hUAAABMRLhyZ1abdPn9UmCElLxX+uAW6T9XSXt/Mrsyt3L6vCvCFQAAAMxDuHJnVpt0yTDprl+lXg9Jnv7S/nXS672l90ZIx3eaXaFbiKWpBQAAANwA4aoq8PKXej0o3b1eunS4ZLFKW5ZJczpJXzwsnTxhdoWmcrRj35aUpuxcu8nVAAAAoKYiXFUlgeHSdbOl27+Tov8h2XOkNXOkFy+R1s7NP0erBoqq7atgX09l59n11yGaWgAAAMAchKuqKDxGunmp9H//leq1yh+5WjFBermztOXjGtf0Ir+pRX5Lds67AgAAgFkIV1XZRXH5o1j9XpD86+Wfg/XezdKCvtL+X82urlI5pgZuJFwBAADAJISrqs7mIbUfmX8+1uUPSB6+0t410mv/kD74V36XwRrgdDt2mloAAADAHISr6sI7UPrHw/mdBdv+U5JF2vSBNLuDtHKKdKp6j+g4wtXWg6nKyaOpBQAAACof4aq6Ca4vDZgr3f6t1LiHlJcl/TArv+nFz/OlvByzK6wQjWr7KdDbQ9m5dm0/lG52OQAAAKiBCFfVVURbacQn0tAlUujFUuYxafl46eUu0p/Lq13TC6vVotaOphaJ1XuUDgAAAO6JcFWdWSxS8z7SHT9K1zwr+YVKx7ZL7w6VFvWTEjeYXWG5cjS1oGMgAAAAzEC4qglsnlLHW/ObXnS/V7J5S7u/k17tKX14u5Sy3+wKy0VsA8IVAAAAzEO4qkl8gqW4qdJdv0ixg/KX/fGuNLu9tGq6lFW1L8DbumDkasvBVOXS1AIAAACVjHBVE9VqKN04X7rta6lhVyn3lPTdTOnFS6VfFkh5uWZXWCZNQ/3l72XTqRy7dhzJMLscAAAA1DCEq5qsfntp1HJp8JtS7aZSxmHp03HSvO7S9pVVrumF1Wpxjl4xNRAAAACVjXBV01ksUst+0pifpD5PSb4h0pGt0ls3SYsHSEkbza6wVBwdAzcSrgAAAFDJCFfI5+ElXfZv6e7fpK53STYvaec30rwe0kdjpdSDZldYIrEFFxPeTDt2AAAAVDLCFVz5hki9H5PG/iy1HiDJkH57U5p9qbT6SSnbvc9linGGq1Tl2avWtEYAAABUbYQrFK12E2ngQulfK6UGnaScTGn1jPymF+sXS/Y8syssUnTdAPl62pSZnaddR9PNLgcAAAA1COEK5xbVSfrXl/lBq1YjKT1J+vhO6ZXLpR3fmF1dITarRa0i88+72nQg1eRqAAAAUJMQrnB+Fkv+FME71+VPGfQJlg5tkhb3l968STq81ewKXcRE0tQCAAAAlY9whZLz8M5vdnH3BqnzHZLVQ/p7pTS3q/TJPVL6YbMrlHT6vCvasQMAAKAyEa5Qen61pb5P5je9aNlPMuzSrwulFy+R/veMlJ1panlnNrWw09QCAAAAlYRwhbKrE51/AeJRn0uRl0rZ6dLXj0lzOki/vyvZ7aaU1axegLw9rErPytXuY+7d3RAAAADVB+EKF65RV+nWVdINr0nBUVLqAWnp7dL8XtKu7yq9HA+bVS0jOO8KAAAAlctiGAbzps6Smpqq4OBgpaSkKCgoyOxyqpacU9JPc6XvnpOyCrr1Nb9auupRKbRZpZXxyLKNenPtXlkt+RcWviy6jro0raOOjWvL39uj0uoAAABA1VaabEC4KgLhqhxkHM2/LtYvCyQjT7LYpA63SL0mSP6hFb77TQdSdPc7v2nnUddpgR5Wi9o0CFaX6Drq0jRU7RuFyNfLVuH1AAAAoGoiXF0gwlU5OrJNWjlF+uvz/K+9g6QeCfndBj19Knz3B5JPau2OY1qz85jW7DimA8knXe73slnVrmEtdWlaR12i6+iShrXk7UHYAgAAQD7C1QUiXFWAnd9KXz4iJf2R/3VwQyluihRzY/51tCrJvuOZWnNG2EpKPeVyv7eHVe0bhTjDVpsGteTlwamJAAAANRXh6gIRriqI3S79sURa9aiUlpi/rH57qffjUqMulV6OYRjafcw1bB1Nz3JZx9fTpg6NQ9Qluo4ua1pHbeoHy8NG2AIAAKgpCFcXiHBVwbIzpTUvSd8/L+UUnBPVsp8UNy2/vbtJDMPQjiPpzrC1dudxHc/IdlnH38umjk1qO0e2WkcGy2atvJE3AAAAVC7C1QUiXFWStEPS6iek9W/kX4jY6il1uk26/P78CxWbzG439NfhtPywteOYftp1XCknc1zWCfTxUOcmdQoaZNRRi/BAWQlbAAAA1Qbh6gIRrirZoS3SyknS31/lf+0TLF3+QH7Q8vA2t7Yz5NkNbT2YqrUFUwh/3nVcaVm5LuuE+HmeDlvRddSsXoAslXhOGQAAAMoX4eoCEa5M8vcq6ctJ0uHN+V+HNJbipkqt+ldq04uSys2za3NiqvN8rXW7jyszO89lndAAL3VuWsc5jbBpqD9hCwAAoAohXF0gwpWJ7HnShrekrx+T0g/lL4vqnN/0IqqjubWdR06eXX/sT3GObP2y57hO5dhd1qkX6O2cQtgluo4a1vYjbAEAALgxwtUFIly5gax06cfZ0o8vSjmZ+cta35Dfvj2ksamllVRWbp5+35dS0CDjqNbvTVZ2rmvYigz20WVnhK0GIX4mVQsAAICiEK4uEOHKjaQmSl8/nj+aJUOyeUmdb5d6jJd8a5ldXamcysnT+r0nnBc13rAvWTl5rt9+UbV9nUGrS9NQhQdX/IWWAQAAUDzC1QUiXLmhg3/kX4R417f5X/vWlnpNkDrcItk8za2tjDKzc/XrnhPO1u9/7E9Rnt3127FJqL8uKwhblzWtrXqBhC0AAIDKRLi6QIQrN2UY0vaV+SHr6Lb8ZbWjpaselVpc45ZNL0ojPStX63Yfd45sbTqQorOyli6qF+Ac2bqsaR3V9vcyp1gAAIAagnB1gQhXbi4vV1q/SPrmCSnzaP6yRt2k3o9J9S81t7ZylHIyR+t2HXd2I9yalKqzv1tbhAeeHtlqUkfBflVzFA8AAMBdEa4uEOGqijiVKv0wS1rzkpR7Kn9Z7CDpyslSrShTS6sIyZnZWrvzuLMb4bZDaS73WyxSq4gg58hWxya1FeRD2AIAALgQhKsLRLiqYpL35bdu/+Pd/K89fKTLxkjd75V8qu/rdzQ9Sz/tPK41O49qzY5j2nEkw+V+q0WKrR/s7EbYsXFt+Xt7mFQtAABA1US4ukCEqyoq8Tfpi0ekPd/nf+1bWwprLfkES95B+UHrzH+9Awv+H+x6n6dflTx/63DqKa3Zecw5srX7WKbL/R5Wi9o0CHZ2ImzfKES+XjaTqgUAAKgaCFcXiHBVhRmGtG25tHKydOzvsm3D6pEfvJxBLPgcwSyo6PDmFWB6QEtMPukMWmt2HtP+Eydd7ve0WXRJVIhzZOuShrXk40nYAgAAOBPh6gIRrqqBvBxp9/dS5jHpVIqUlZp/jlax/6ZIWWmSYT//tkvCYi0IaGcHs8AiRtGKCG8+QZJXoGS1lk89kvYdz8wf2SoIWwdTTrnc7+VhVfuGIeoSXUeN6vjJ28Mqb09b/r8eNvl45v/r7WGVj6dN3p5W+XjY5GmzyFIFR/oAAABKgnB1gQhXNZRhSNkZRQevcwazs/5vzy2ngixnhbGiglkxI2dn/mstPBplGIb2HMt0diJcs/OYjqRlla1Ki04HrrOCWKFAdmZgKwhn3p4leczpdc98jM1KqAMAABWLcHWBCFcoM8OQck6eFcCKGTnLSiv+vrzs8qvJK+AcwSx/5MzwDtThbG9tOWHRH0ftOn7KolN50qk8q07mGjrp+DfXolN5hjJyLMqTVXmyFfxrLfS1VPHBx9NmKRTITo+2nQ5nRYW0koZA531nrOtlszJaBwBADVGabOAWrcNeeuklPfPMM0pKSlLbtm01e/ZsderUqdj133//fU2aNEm7d+9Ws2bN9NRTT+nqq6923m8YhqZMmaL58+crOTlZ3bp109y5c9WsWbPKOBzUZBaL5OWXfwsML/t2ck6dY+QsrWSjao729Nnp+be0g8WXLSms4HbF+WqzFdzOw7BYZVhssls8ZFissssm+xn/5skmu6zKdf5rVZ6R/2+uYVWuYVOuYVGOrMoxrMqxW5VjWPLvk/X0Y+1W5Z6yKU+WM8JdEaGv4HF5silLVmUWEQjz1zmzLovL14517BarrDYPWW2esnl4yMPDUx42D1ltHrJY86dJWqxWWSz5N1ksslosslht+fdZLLJabfnLrZb89QrWt1pOf221Wk/fb7HKarNJFslmscpiteXfb5FsBfu0WSyyWiSrNX9/NqtkLdi3y3rWgvUsjvUsshTcb7OctY5jW0Vs+/Q+LbIW7Otc2z5fLVL+t5BF+cssFossjmWEWQBAFWB6uFqyZIkSEhI0b948de7cWbNmzVJ8fLy2bdumevXqFVr/xx9/1NChQzVjxgxde+21evvtt9W/f3+tX79eMTExkqSnn35aL774ohYtWqQmTZpo0qRJio+P15YtW+Tj41PZhwiUnqdP/i2g8PdAieVmu46cZaUVEcTOMXJmzyu45UpGwb92u+vX5zhHzWLYZTHssiqn7MfgskFVxmBY6eUW3EyUZ1hkKP9mL3ii7HIsk+yyKn+KQtHLDee/Z95vkWGcvZ7F5XbmtgxZZZeUe9Y6hfd1xuONMx+f/38Hx/9P/5v/f4skw3L6zeB8jOX016fXyT9mx7GfvfzMx+is7RT1tVEQ9owzHut8Y1rOeIyUf96l8/+Ox1nPWOWMbVjO3r+l4K5i6rLk3+9S95k1SzJcvlfOeq7O+v+ZjCK2Z1jOfFzh7Rb7OJdVi163qDqK2lah5ZaSbav42k4vsxTxXJ35j+M95cJSVI0WFfk3AOe2zqy/mBULHWLhZcYZ+3Hdylnn6BZxXKffN2dUbcl/rc6sz/E/x2tvKVxYkYdiKfg+thS/uuvSs56bol6Lc/5dxXJmzaXbxulvRcvZS1yem+L2WfT7xvW5LXqrbspN/4Dl6Ruky3oPNLuMUjF9WmDnzp3VsWNHzZkzR5Jkt9sVFRWlu+66SxMmTCi0/uDBg5WRkaFPP/3Uueyyyy5Tu3btNG/ePBmGocjISN13330aP368JCklJUVhYWFauHChhgwZUmibWVlZyso6fb5JamqqoqKimBYInI9hFBHACkKZy9cFQczl67xiAtzZy86xTon3Wdo6Cu/TsOfJcPybl+tcdmbotBh5koz856XgX4uMghBqyFKw3GLkRxMAAFC8vdb6ajh5i9llVJ1pgdnZ2fr11181ceJE5zKr1aq4uDitWbOmyMesWbNGCQkJLsvi4+O1bNkySdKuXbuUlJSkuLg45/3BwcHq3Lmz1qxZU2S4mjFjhqZNm1YORwTUMBaLZPPIv1VzFTZwZhSEL2cgs59e5ghqhf6v8ntMsY83ithW+TzGMAzZ7XkF/9plGHYZBcsdT0n+evb80Y+C+4yCbZ9ezyi43/F4OfdvL9jnmevKub7h3J9FRkHpp9fPf4i9YOjFkN1wFlVQQ8E2zqrn7O2f3r/h3JZxxjqnH1OwL0P5/3fZxpn7PutxLn8bPf1/iwovtxT7d9RiHudc/8z7VcT9Re/P5XEuuy6itiL3e/Z2C2/DUszxn2/fZ27X+b8inx/jrJUKfVEw4mMUWt2x7My1LY7X/sxlOmNZ4f+cv7ZSrOf4X9HvheK353IMzrXPfM2K2Fyxr13h7Z937RKPARR+vYp6ZFGvzQXvs5gNOp/zkjwf5aKy9lN5x5TpG6GGlbKn8mPqb0RHjx5VXl6ewsLCXJaHhYXpzz//LPIxSUlJRa6flJTkvN+xrLh1zjZx4kSXwOYYuQKACmexSJaadX0xi0p02h4AAFVO9f9zcwl4e3vL29vb7DIAAAAAVGHld4XSMggNDZXNZtOhQ4dclh86dEjh4UV3WgsPDz/n+o5/S7NNAAAAALhQpoYrLy8vtW/fXqtWrXIus9vtWrVqlbp06VLkY7p06eKyviStXLnSuX6TJk0UHh7usk5qaqp++umnYrcJAAAAABfK9GmBCQkJGjFihDp06KBOnTpp1qxZysjI0KhRoyRJw4cPV/369TVjxgxJ0j333KOePXvq2Wef1TXXXKN3331Xv/zyi1599VVJ+ddCGTdunB577DE1a9bM2Yo9MjJS/fv3N+swAQAAAFRzpoerwYMH68iRI5o8ebKSkpLUrl07rVixwtmQYu/evbJaTw+wde3aVW+//bYeeeQRPfTQQ2rWrJmWLVvmvMaVJD3wwAPKyMjQ6NGjlZycrO7du2vFihVc4woAAABAhTH9OlfuqDS97AEAAABUX6XJBqaecwUAAAAA1QXhCgAAAADKAeEKAAAAAMoB4QoAAAAAygHhCgAAAADKAeEKAAAAAMoB4QoAAAAAygHhCgAAAADKAeEKAAAAAMoB4QoAAAAAygHhCgAAAADKAeEKAAAAAMqBh9kFuCPDMCRJqampJlcCAAAAwEyOTODICOdCuCpCWlqaJCkqKsrkSgAAAAC4g7S0NAUHB59zHYtRkghWw9jtdiUmJiowMFAWi8XsclBGqampioqK0r59+xQUFGR2OajmeL+hsvGeQ2Xi/YbK5k7vOcMwlJaWpsjISFmt5z6ripGrIlitVjVo0MDsMlBOgoKCTP+mRM3B+w2VjfccKhPvN1Q2d3nPnW/EyoGGFgAAAABQDghXAAAAAFAOCFeotry9vTVlyhR5e3ubXQpqAN5vqGy851CZeL+hslXV9xwNLQAAAACgHDByBQAAAADlgHAFAAAAAOWAcAUAAAAA5YBwBQAAAADlgHCFamXGjBnq2LGjAgMDVa9ePfXv31/btm0zuyzUEE8++aQsFovGjRtndimoxg4cOKD/+7//U506deTr66vY2Fj98ssvZpeFaiovL0+TJk1SkyZN5Ovrq+joaE2fPl30Q0N5+d///qd+/fopMjJSFotFy5Ytc7nfMAxNnjxZERER8vX1VVxcnLZv325OsSVAuEK18u2332rs2LFau3atVq5cqZycHPXu3VsZGRlml4Zqbt26dXrllVfUpk0bs0tBNXbixAl169ZNnp6e+vzzz7VlyxY9++yzCgkJMbs0VFNPPfWU5s6dqzlz5mjr1q166qmn9PTTT2v27Nlml4ZqIiMjQ23bttVLL71U5P1PP/20XnzxRc2bN08//fST/P39FR8fr1OnTlVypSVDK3ZUa0eOHFG9evX07bff6vLLLze7HFRT6enpuvTSS/Xyyy/rscceU7t27TRr1iyzy0I1NGHCBP3www/67rvvzC4FNcS1116rsLAw/ec//3Euu/HGG+Xr66s333zTxMpQHVksFi1dulT9+/eXlD9qFRkZqfvuu0/jx4+XJKWkpCgsLEwLFy7UkCFDTKy2aIxcoVpLSUmRJNWuXdvkSlCdjR07Vtdcc43i4uLMLgXV3Mcff6wOHTpo4MCBqlevni655BLNnz/f7LJQjXXt2lWrVq3SX3/9JUn6/fff9f3336tv374mV4aaYNeuXUpKSnL5+RocHKzOnTtrzZo1JlZWPA+zCwAqit1u17hx49StWzfFxMSYXQ6qqXfffVfr16/XunXrzC4FNcDOnTs1d+5cJSQk6KGHHtK6det09913y8vLSyNGjDC7PFRDEyZMUGpqqlq0aCGbzaa8vDw9/vjjGjZsmNmloQZISkqSJIWFhbksDwsLc97nbghXqLbGjh2rTZs26fvvvze7FFRT+/bt0z333KOVK1fKx8fH7HJQA9jtdnXo0EFPPPGEJOmSSy7Rpk2bNG/ePMIVKsR7772nt956S2+//bZat26tDRs2aNy4cYqMjOQ9BxSBaYGolu688059+umn+uabb9SgQQOzy0E19euvv+rw4cO69NJL5eHhIQ8PD3377bd68cUX5eHhoby8PLNLRDUTERGhVq1auSxr2bKl9u7da1JFqO7uv/9+TZgwQUOGDFFsbKxuvvlm3XvvvZoxY4bZpaEGCA8PlyQdOnTIZfmhQ4ec97kbwhWqFcMwdOedd2rp0qX6+uuv1aRJE7NLQjV25ZVXauPGjdqwYYPz1qFDBw0bNkwbNmyQzWYzu0RUM926dSt0eYm//vpLjRo1MqkiVHeZmZmyWl1/XbTZbLLb7SZVhJqkSZMmCg8P16pVq5zLUlNT9dNPP6lLly4mVlY8pgWiWhk7dqzefvttffTRRwoMDHTOxw0ODpavr6/J1aG6CQwMLHQ+n7+/v+rUqcN5fqgQ9957r7p27aonnnhCgwYN0s8//6xXX31Vr776qtmloZrq16+fHn/8cTVs2FCtW7fWb7/9pueee0633HKL2aWhmkhPT9fff//t/HrXrl3asGGDateurYYNG2rcuHF67LHH1KxZMzVp0kSTJk1SZGSks6Ogu6EVO6oVi8VS5PIFCxZo5MiRlVsMaqRevXrRih0V6tNPP9XEiRO1fft2NWnSRAkJCbrtttvMLgvVVFpamiZNmqSlS5fq8OHDioyM1NChQzV58mR5eXmZXR6qgdWrV+uKK64otHzEiBFauHChDMPQlClT9Oqrryo5OVndu3fXyy+/rIsvvtiEas+PcAUAAAAA5YBzrgAAAACgHBCuAAAAAKAcEK4AAAAAoBwQrgAAAACgHBCuAAAAAKAcEK4AAAAAoBwQrgAAAACgHBCuAAAAAKAcEK4AAChnFotFy5YtM7sMAEAlI1wBAKqVkSNHymKxFLr16dPH7NIAANWch9kFAABQ3vr06aMFCxa4LPP29japGgBATcHIFQCg2vH29lZ4eLjLLSQkRFL+lL25c+eqb9++8vX1VdOmTfXBBx+4PH7jxo36xz/+IV9fX9WpU0ejR49Wenq6yzqvv/66WrduLW9vb0VEROjOO+90uf/o0aMaMGCA/Pz81KxZM3388ccVe9AAANMRrgAANc6kSZN044036vfff9ewYcM0ZMgQbd26VZKUkZGh+Ph4hYSEaN26dXr//ff11VdfuYSnuXPnauzYsRo9erQ2btyojz/+WBdddJHLPqZNm6ZBgwbpjz/+0NVXX61hw4bp+PHjlXqcAIDKZTEMwzC7CAAAysvIkSP15ptvysfHx2X5Qw89pIceekgWi0X//ve/NXfuXOd9l112mS699FK9/PLLmj9/vh588EHt27dP/v7+kqTly5erX79+SkxMVFhYmOrXr69Ro0bpscceK7IGi8WiRx55RNOnT5eUH9gCAgL0+eefc+4XAFRjnHMFAKh2rrjiCpfwJEm1a9d2/r9Lly4u93Xp0kUbNmyQJG3dulVt27Z1BitJ6tatm+x2u7Zt2yaLxaLExERdeeWV56yhTZs2zv/7+/srKChIhw8fLushAQCqAMIVAKDa8ff3LzRNr7z4+vqWaD1PT0+Xry0Wi+x2e0WUBABwE5xzBQCocdauXVvo65YtW0qSWrZsqd9//10ZGRnO+3/44QdZrVY1b95cgYGBaty4sVatWlWpNQMA3B8jVwCAaicrK0tJSUkuyzw8PBQaGipJev/999WhQwd1795db731ln7++Wf95z//kSQNGzZMU6ZM0YgRIzR16lQdOXJEd911l26++WaFhYVJkqZOnap///vfqlevnvr27au0tDT98MMPuuuuuyr3QAEAboVwBQCodlasWKGIiAiXZc2bN9eff/4pKb+T37vvvqsxY8YoIiJC77zzjlq1aiVJ8vPz0xdffKF77rlHHTt2lJ+fn2688UY999xzzm2NGDFCp06d0vPPP6/x48crNDRUN910U+UdIADALdEtEABQo1gsFi1dulT9+/c3uxQAQDXDOVcAAAAAUA4IVwAAAABQDjjnCgBQozAbHgBQURi5AgAAAIByQLgCAAAAgHJAuAIAAACAckC4AgAAAIByQLgCAAAAgHJAuAIAAACAckC4AgAAAIByQLgCAAAAgHLw/46zHvtaqbeWAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot training and evaluation loss\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(trainer_history_training_df[\"epoch\"], trainer_history_training_df[\"loss\"], label=\"Training loss\")\n",
    "plt.plot(trainer_history_eval_df[\"epoch\"], trainer_history_eval_df[\"eval_loss\"], label=\"Evaluation loss\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.title(\"Text classification with DistilBert training and evaluation loss over time\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "B-e-a-utiful!\n",
    "\n",
    "That is exactly what we wanted.\n",
    "\n",
    "Training and evaluation loss going down over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pushing our model to the Hugging Face Hub\n",
    "\n",
    "We've saved our model locally and confirmed that it seems to be performing well on our training metrics but how about we push it to the Hugging Face Hub?\n",
    "\n",
    "The Hugging Face Hub is one of the best sources of machine learning models on the internet.\n",
    "\n",
    "And we can add our model there so others can use it or we can access it in the future (we could also keep it private on the Hugging Face Hub so only people from our organization can use it).\n",
    "\n",
    "Sharing models on Hugging Face is also a great way to showcase your skills as a machine learning engineer, it gives you something to show potential employers and say \"here's what I've done\".\n",
    "\n",
    ":::{.callout-note}\n",
    "\n",
    "Before sharing a model to the Hugging Face Hub, be sure to go through the following steps: \n",
    "\n",
    "1. Setup a Hugging Face token using the [`huggingface-cli login` command](https://huggingface.co/docs/huggingface_hub/en/guides/cli).\n",
    "2. Read through the [user access tokens guide](https://huggingface.co/docs/hub/en/security-tokens).\n",
    "3. Set up an access token via [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) (ensure it has \"write\" access).\n",
    "\n",
    "If you are using Google Colab, you can add your token under the \"Secrets\" tab on the left.\n",
    "\n",
    "On my local computer, my token is saved to `/home/daniel/.cache/huggingface/token` (thanks to running `huggingface-cli login` on the command line).\n",
    "\n",
    "And for more on sharing models to the Hugging Face Hub, be sure to check out the [model sharing documentation](https://huggingface.co/docs/transformers/en/model_sharing).\n",
    "\n",
    ":::\n",
    "\n",
    "We can push our model, tokenizer and other assosciated files to the Hugging Face Hub using the [`transformers.Trainer.push_to_hub`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Trainer.push_to_hub) method. \n",
    "\n",
    "We can also optionally do the following:\n",
    "\n",
    "* Add a [model card](https://huggingface.co/docs/hub/en/model-cards) (something that describes how the model was created and what it can be used for) using [`transformers.Trainer.create_model_card`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Trainer.create_model_card).\n",
    "* Add a custom `README.md` file to the model repository to explain more details about the model using [`huggingface_hub.HfApi.upload_file`](https://huggingface.co/docs/huggingface_hub/en/guides/upload#upload-a-file). This method is similar to model card creation method above but with more customization.\n",
    "\n",
    "Let's save our model to the Hub!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "38d516c3db4f4c49ae5a2074d1994611",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload 2 LFS files:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a61fa0525609401c855f9a86feaab961",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad253fc8b1b44b038e9bcfb1b99c4154",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "training_args.bin:   0%|          | 0.00/5.24k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased/commit/8a8a8aff5bdee5bc518e31558447dc684d448b8f', commit_message='Uploading food not food text classifier model', commit_description='', oid='8a8a8aff5bdee5bc518e31558447dc684d448b8f', pr_url=None, pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save our model to the Hugging Face Hub\n",
    "# This will be public, since we set hub_private_repo=False in our TrainingArguments\n",
    "model_upload_url = trainer.push_to_hub(\n",
    "    commit_message=\"Uploading food not food text classifier model\",\n",
    "    # token=\"YOUR_HF_TOKEN_HERE\" # This will default to the token you have saved in your Hugging Face config\n",
    ")\n",
    "print(f\"[INFO] Model successfully uploaded to Hugging Face Hub with at URL: {model_upload_url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model pushed to the Hugging Face Hub!\n",
    "\n",
    "::: {.callout-note} \n",
    "\n",
    "You may see the following error:\n",
    "\n",
    "> 403 Forbidden: You don't have the rights to create a model under the namespace \"mrdbourke\".\n",
    "> Cannot access content at: https://huggingface.co/api/repos/create.\n",
    "> If you are trying to create or update content, make sure you have a token with the `write` role.\n",
    "\n",
    "Or even:\n",
    "\n",
    "> HfHubHTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/api/repos/create (Request ID: Root=1-6699c52XXXXXX)\n",
    ">\n",
    "> Invalid username or password.\n",
    "\n",
    "In this case, be sure to go through the [setup steps above](https://huggingface.co/docs/hub/en/security-tokens) to make sure you have a Hugging Face access token with \"write\" access.\n",
    "\n",
    ":::\n",
    "\n",
    "And since it's public (by default), you can see it at [https://huggingface.co/mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased](https://huggingface.co/mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased) (it gets saved to the same name as our target local directory). \n",
    "\n",
    "You can now share and interact with this model online.\n",
    "\n",
    "As well as download it for use in your own applications.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/04-model-on-hf-hub.png\"\n",
    "     alt=\"\n",
    "Side-by-side images showcasing the process of using a model from the Hugging Face Hub. The first image shows the model card page for 'learn_hf_food_not_food_text_classifier-distilbert-base-uncased' on Hugging Face Hub, highlighting the 'Use this model' button. The second image displays the code snippet provided by the Hugging Face Transformers library for utilizing the model in a text classification pipeline. This demonstrates the ease of accessing and implementing pretrained models from the Hugging Face Hub using the Transformers library.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>The Hugging Face Hub allows us to store and share models, datasets and demos. We can set these to be private or public. Models stored on the Hub can easily be accessed via Hugging Face Transformers.</figcaption>\n",
    "</figure>\n",
    "\n",
    "But before we make an application/demo with our trained model, let's keep evaluating it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making and evaluating predictions on the test data\n",
    "\n",
    "Model trained, let's now evaluate it on the test data.\n",
    "\n",
    "Or step 7 in our workflow:\n",
    "\n",
    "1. ✅ Create and preprocess data.\n",
    "2. ✅ Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. ✅ Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. ✅ Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. ✅ Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. ✅ Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "A reminder that the test data is data that our model has never seen before.\n",
    "\n",
    "So it will be a good estimate of how our model will do in a production setting.\n",
    "\n",
    "We can make predictions on the test dataset using [`transformers.Trainer.predict`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.predict).\n",
    "\n",
    "And then we can get the prediction values with the `predictions` attribute and assosciated metrics with the `metrics` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Prediction metrics on the test data:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'test_loss': 0.0005385442636907101,\n",
       " 'test_accuracy': 1.0,\n",
       " 'test_runtime': 0.0421,\n",
       " 'test_samples_per_second': 1186.857,\n",
       " 'test_steps_per_second': 47.474}"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform predictions on the test set\n",
    "predictions_all = trainer.predict(tokenized_dataset[\"test\"])\n",
    "prediction_values = predictions_all.predictions\n",
    "prediction_metrics = predictions_all.metrics\n",
    "\n",
    "print(f\"[INFO] Prediction metrics on the test data:\")\n",
    "prediction_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah!\n",
    "\n",
    "Looks like our model did an outstanding job!\n",
    "\n",
    "And it was *very* quick too.\n",
    "\n",
    "This is one of the benefits of using a smaller pretrained model and customizing it to your own dataset.\n",
    "\n",
    "You can achieve outstanding results in a very quick time as well as have a model capable of performing thousands of predictions per second.\n",
    "\n",
    "We can also calculate the accuracy by hand by comparing the prediction labels to the test labels.\n",
    "\n",
    "To do so, we'll:\n",
    "\n",
    "1. Calculate the prediction probabilities (though this is optional as we could skip straight to 2 and get the same results) by passing the `prediction_values` to [`torch.softmax`](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html).\n",
    "2. Find the index of the prediction value with the highest value (the index will be equivalent to the predicted label) using [`torch.argmax`](https://pytorch.org/docs/stable/generated/torch.argmax.html) (we could also use [`np.argmax`](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html) here) to find the predicted labels. \n",
    "3. Get the true labels from the test dataset using `dataset[\"test\"][\"label\"]`.\n",
    "4. Compare the predicted labels from 2 to the true labels from 3 using [`sklearn.metrics.accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) to find the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Test accuracy: 100.0%\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# 1. Get prediction probabilities (this is optional, could get the same results with step 2 onwards)\n",
    "pred_probs = torch.softmax(torch.tensor(prediction_values), dim=1)\n",
    "\n",
    "# 2. Get the predicted labels\n",
    "pred_labels = torch.argmax(pred_probs, dim=1)\n",
    "\n",
    "# 3. Get the true labels\n",
    "true_labels = dataset[\"test\"][\"label\"]\n",
    "\n",
    "# 4. Compare predicted labels to true labels to get the test accuracy\n",
    "test_accuracy = accuracy_score(y_true=true_labels, \n",
    "                               y_pred=pred_labels)\n",
    "\n",
    "print(f\"[INFO] Test accuracy: {test_accuracy*100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah!\n",
    "\n",
    "Looks like our model performs really well on our test set.\n",
    "\n",
    "It will be interesting to see how it goes on real world samples.\n",
    "\n",
    "We'll test this later on.\n",
    "\n",
    "How about we make a pandas DataFrame out of our test samples, predicted labels and predicted probabilities to further inspect our results?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>true_label</th>\n",
       "      <th>pred_label</th>\n",
       "      <th>pred_prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A slice of pepperoni pizza with a layer of mel...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Red brick fireplace with a mantel serving as a...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.999662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A bowl of sliced bell peppers with a sprinkle ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Set of mugs hanging on a hook</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.999682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Standing floor lamp providing light next to an...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.999678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  true_label  pred_label  \\\n",
       "0  A slice of pepperoni pizza with a layer of mel...           1           1   \n",
       "1  Red brick fireplace with a mantel serving as a...           0           0   \n",
       "2  A bowl of sliced bell peppers with a sprinkle ...           1           1   \n",
       "3                      Set of mugs hanging on a hook           0           0   \n",
       "4  Standing floor lamp providing light next to an...           0           0   \n",
       "\n",
       "   pred_prob  \n",
       "0   0.999369  \n",
       "1   0.999662  \n",
       "2   0.999365  \n",
       "3   0.999682  \n",
       "4   0.999678  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a DataFrame of test predictions\n",
    "test_predictions_df = pd.DataFrame({\n",
    "    \"text\": dataset[\"test\"][\"text\"],\n",
    "    \"true_label\": true_labels,\n",
    "    \"pred_label\": pred_labels,\n",
    "    \"pred_prob\": torch.max(pred_probs, dim=1).values\n",
    "})\n",
    "\n",
    "test_predictions_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find the examples with the lowest prediction probability to see where the model is unsure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>true_label</th>\n",
       "      <th>pred_label</th>\n",
       "      <th>pred_prob</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>A bowl of cherries with a sprig of mint for ga...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>A close-up shot of a cheesy pizza slice being ...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>A fruit platter with a variety of exotic fruit...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Boxes of apples, pears, pineapple, manadrins a...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>A bowl of sliced kiwi with a sprinkle of sugar...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Close-up of a sushi roll with avocado, cucumbe...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Crunchy sushi roll with tempura flakes or pank...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Cherry tomatoes and mozzarella balls in a bowl...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Two handfuls of bananas in a fruit bowl with g...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Seasonal sushi roll with ingredients like pers...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999361</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text  true_label  pred_label  \\\n",
       "40  A bowl of cherries with a sprig of mint for ga...           1           1   \n",
       "11  A close-up shot of a cheesy pizza slice being ...           1           1   \n",
       "26  A fruit platter with a variety of exotic fruit...           1           1   \n",
       "42  Boxes of apples, pears, pineapple, manadrins a...           1           1   \n",
       "46  A bowl of sliced kiwi with a sprinkle of sugar...           1           1   \n",
       "37  Close-up of a sushi roll with avocado, cucumbe...           1           1   \n",
       "31  Crunchy sushi roll with tempura flakes or pank...           1           1   \n",
       "9   Cherry tomatoes and mozzarella balls in a bowl...           1           1   \n",
       "14  Two handfuls of bananas in a fruit bowl with g...           1           1   \n",
       "44  Seasonal sushi roll with ingredients like pers...           1           1   \n",
       "\n",
       "    pred_prob  \n",
       "40   0.999331  \n",
       "11   0.999348  \n",
       "26   0.999351  \n",
       "42   0.999353  \n",
       "46   0.999360  \n",
       "37   0.999360  \n",
       "31   0.999360  \n",
       "9    0.999360  \n",
       "14   0.999360  \n",
       "44   0.999361  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show 10 examples with low prediction probability\n",
    "test_predictions_df.sort_values(\"pred_prob\", ascending=True).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm, it looks like our model has quite a high prediction probability for almost all samples.\n",
    "\n",
    "We can further evalaute our model by making predictions on new custom data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fGndCPKMAG-p"
   },
   "source": [
    "## Making and inspecting predictions on custom text data\n",
    "\n",
    "We've seen how our model performs on the test dataset (quite well).\n",
    "\n",
    "But how might we check its performance on our own custom data?\n",
    "\n",
    "For example, text-based image captions from the wild.\n",
    "\n",
    "Well, we've got two ways to load our model now too:\n",
    "\n",
    "1. Load model locally from our computer (e.g. via `models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased`).\n",
    "2. Load model from Hugging Face Hub (e.g. via `mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased`).\n",
    "\n",
    "Either way of loading the model results in the same outcome: being able to make predictions on given data.\n",
    "\n",
    "So how about we start by setting up our model paths for both local loading and loading from the Hugging Face Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup local model path\n",
    "local_model_path = \"models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\"\n",
    "\n",
    "# Setup Hugging Face model path (see: https://huggingface.co/mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased)\n",
    "# Note: Be sure to change \"mrdbourke\" to your own Hugging Face username\n",
    "huggingface_model_path = \"mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussing ways to make predictions (inference)\n",
    "\n",
    "When we've loaded our trained model, because of the way we've set it up, there are two main ways to make predictions on custom data:\n",
    "\n",
    "1. **Pipeline mode** using [`transformers.pipeline`](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/pipelines#pipelines) and passing it our target model, this allows us to preprocess custom data and make predictions in one step.\n",
    "2. **PyTorch mode** using a combination of [`transformers.AutoTokenizer`](https://huggingface.co/docs/transformers/v4.42.0/en/model_doc/auto#transformers.AutoTokenizer) and [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/v4.42.0/en/model_doc/auto#transformers.AutoModelForSequenceClassification) and passing each our target model, this requires us to preprocess our data before passing to a model, however, it offers the most customization.\n",
    "\n",
    "Each method supports:\n",
    "\n",
    "1. Predictions one at a time (batch size of 1), for example, one person using the app at a time.\n",
    "2. Batches of predictions at a time (predictions with a batch size of `n` where `n` can be any number, e.g. `8`, `16`, `32`), for example, many people using a service simultaneously such as a voice chat and needing to filter comments (predicting on batches of size `n` is usually much faster than batches of 1).\n",
    "\n",
    "Whichever method we choose, we'll have to set the target device we'd like the operations to happen on.\n",
    "\n",
    "In general, it's best to make predictions on the most powerful accelerator you have available.\n",
    "\n",
    "And in most cases that will be a NVIDIA GPU > Mac GPU > CPU.\n",
    "\n",
    "So let's write a small function to pick the target device for us in that order.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "Making predictions is also referred to as **inference**.\n",
    "\n",
    "Because the model is going to *infer* on some data what the output should be.\n",
    "\n",
    "Inference is often faster than training on a per sample basis as no model weights are updated (less computation).\n",
    "\n",
    "However, inference can use more compute than training over the long run because you could train a model once over a few hours (or days or longer) and then use it for inference for several months (or longer), millions of times (or more).\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Using device: cuda\n"
     ]
    }
   ],
   "source": [
    "def set_device():\n",
    "    \"\"\"\n",
    "    Set device to CUDA if available, else MPS (Mac), else CPU.\n",
    "\n",
    "    This defaults to using the best available device (usually).\n",
    "    \"\"\"\n",
    "    if torch.cuda.is_available():\n",
    "        device = torch.device(\"cuda\")\n",
    "    elif torch.backends.mps.is_available() and torch.backends.mps.is_built():\n",
    "        device = torch.device(\"mps\")\n",
    "    else:\n",
    "        device = torch.device(\"cpu\")\n",
    "    return device\n",
    "\n",
    "DEVICE = set_device()\n",
    "print(f\"[INFO] Using device: {DEVICE}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Target device set!\n",
    "\n",
    "Let's start predicting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zJ8jAbX6Aa7D"
   },
   "source": [
    "### Making predictions with pipeline\n",
    "\n",
    "The [`transformers.pipeline`](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/pipelines#transformers.pipeline) method creates a machine learning pipeline.\n",
    "\n",
    "Data goes in one end and predictions come out the other end.\n",
    "\n",
    "You can create pipelines for many different tasks, such as, text classification, image classification, object detection, text generation and more.\n",
    "\n",
    "Let's see how we can create a pipeline for our text classification model.\n",
    "\n",
    "To do so we'll:\n",
    "\n",
    "1. Instantiate an instance of `transformers.pipeline`.\n",
    "2. Pass in the `task` parameter of `text-classification` (we can do this because our model is already formatted for text classification thanks to using `transformers.AutoModelForSequenceClassification`).\n",
    "3. Setup the `model` parameter to be `local_model_path` (though we could also use `huggingface_model_path`).\n",
    "4. Set the target device using the `device` parameter.\n",
    "5. Set `top_k=1` to get to the top prediction back (e.g. either `\"food\"` or `\"not_food\"`, could set this higher to get more labels back).\n",
    "6. Set the `BATCH_SIZE=32` so we can pass to the `batch_size` parameter. This will allow our model to make predictions on up to `32` samples at a time. Predicting on batches of data is usually much faster than single samples at a time, however, this often saturates at a point (e.g. predicting on batches of size 64 may be the same speed as 32 due to memory contraints).\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "There are many more pipelines available in the [Hugging Face documentation](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/pipelines#transformers.pipeline).\n",
    "\n",
    "As an exericse, I'd spend 10-15 minutes reading through the pipeline documentation to get familiar with what's available.\n",
    "\n",
    ":::\n",
    "\n",
    "Let's setup our pipeline! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "B44pOUBKAcvh",
    "outputId": "fb71c373-8dc5-42c3-9749-c58ab59253c3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<transformers.pipelines.text_classification.TextClassificationPipeline at 0x7f2695245950>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import pipeline\n",
    "\n",
    "# Set the batch size for predictions\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "# Create an instance of transformers.pipeline\n",
    "food_not_food_classifier = pipeline(task=\"text-classification\", # we can use this because our model is an instance of AutoModelForSequenceClassification\n",
    "                                    model=local_model_path, # could also pass in huggingface_model_path\n",
    "                                    device=DEVICE, # set the target device\n",
    "                                    top_k=1, # only return the top predicted value\n",
    "                                    batch_size=BATCH_SIZE) # perform predictions on up to BATCH_SIZE number of samples at a time \n",
    "\n",
    "food_not_food_classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've created an instance of [`transformers.pipelines.text_classification.TextClassificationPipeline`](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/pipelines#transformers.TextClassificationPipeline)!\n",
    "\n",
    "Now let's test it out by passing it a string of text about food."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'label': 'food', 'score': 0.9993335604667664}]]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test our trained model on some example text \n",
    "sample_text_food = \"A delicious photo of a plate of scrambled eggs, bacon and toast\"\n",
    "food_not_food_classifier(sample_text_food)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Our model gets it right.\n",
    "\n",
    "How about a string *not* about food?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8YP2KDxPFQ8C",
    "outputId": "7d9f1336-883e-474a-cc9a-aa605ba2afc1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'label': 'not_food', 'score': 0.9996254444122314}]]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test the model on some more example text\n",
    "sample_text_not_food = \"A yellow tractor driving over the hill\"\n",
    "food_not_food_classifier(sample_text_not_food)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo!\n",
    "\n",
    "Correct again!\n",
    "\n",
    "What if we passed in random text?\n",
    "\n",
    "As in, someone types in something random to the model expecting an output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'label': 'not_food', 'score': 0.9985743761062622}]]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pass in random text to the model\n",
    "food_not_food_classifier(\"cvnhertiejhwgdjshdfgh394587\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nature of machine learning models is that they are a predictive/generative function.\n",
    "\n",
    "If you input data, they will output something.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "When deploying machine learning models, there are many things to take into consideration.\n",
    "\n",
    "One of the main ones being: \"what data is going to go into the model?\"\n",
    "\n",
    "If this was a public facing model and people could enter any kind of text, they could enter random text rather than a sentence about food or not food.\n",
    "\n",
    "Since our main goal of the model is be able to classify image captions into `food`/`not_food`, we'd also have to consider image cpations that are poorly written or contain little text.\n",
    "\n",
    "This is why it's important to continually test your models with as much example test/real-world data as you can.\n",
    "\n",
    ":::\n",
    "\n",
    "Our `pipeline` can also work with the model we saved to the Hugging Face Hub.\n",
    "\n",
    "Let's try out the same pipeline with `model=hugggingface_model_path`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "340b736ae6854b8f8746972458caa65b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/738 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b65c413a92b4c799a34b2907cb4e736",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'label': 'food', 'score': 0.9993208646774292}]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pipeline also works with remote models (will have to laod the model locally first)\n",
    "food_not_food_classifier_remote = pipeline(task=\"text-classification\", \n",
    "                                           model=huggingface_model_path, # load the model from Hugging Face Hub (will download the model if it doesn't already exist)\n",
    "                                           batch_size=BATCH_SIZE,\n",
    "                                           device=DEVICE)\n",
    "\n",
    "food_not_food_classifier_remote(\"This is some new text about bananas and pancakes and ice cream\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful!\n",
    "\n",
    "Our model loaded from Hugging Face gets it right too!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making multiple predictions at the same time with batch prediction\n",
    "\n",
    "We can make predictions with our model one at a time but it's often much faster to do them in batches.\n",
    "\n",
    "To make predictions in batches, we can set up our `transformers.pipeline` instance with the `batch_size` parameter greater than `1`.\n",
    "\n",
    "Then we'll be able to pass multiple samples at once in the form of a Python list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create batch size (we don't need to do this again but we're doing it for clarity)\n",
    "BATCH_SIZE = 32 # this number is experimental and will require testing on your hardware to find the optimal value (e.g. lower if there are memory issues or higher to try speed up inference)\n",
    "\n",
    "# Setup pipeline to handle batches (we don't need to do this again either but we're doing it for clarity)\n",
    "food_not_food_classifier = pipeline(task=\"text-classification\", \n",
    "                                    model=local_model_path,\n",
    "                                    batch_size=BATCH_SIZE,\n",
    "                                    device=DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wonderful, now we've set up a `pipeline` instance capable of handling batches, we can pass it a list of samples and it will make predictions on each.\n",
    "\n",
    "How about we try with a collection of sentences which are a bit tricky?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ypr_Y65CFjFb",
    "outputId": "3a50b522-c155-49fd-8c0e-27541486bca4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'not_food', 'score': 0.9986234903335571},\n",
       " {'label': 'not_food', 'score': 0.9993952512741089},\n",
       " {'label': 'not_food', 'score': 0.9992876648902893},\n",
       " {'label': 'not_food', 'score': 0.9994683861732483},\n",
       " {'label': 'not_food', 'score': 0.9993450045585632},\n",
       " {'label': 'not_food', 'score': 0.9994571805000305},\n",
       " {'label': 'not_food', 'score': 0.9991866946220398},\n",
       " {'label': 'food', 'score': 0.9993101358413696},\n",
       " {'label': 'not_food', 'score': 0.9995250701904297},\n",
       " {'label': 'food', 'score': 0.9966572523117065}]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a list of sentences to make predictions on\n",
    "sentences = [\n",
    "    \"I whipped up a fresh batch of code, but it seems to have a syntax error.\",\n",
    "    \"We need to marinate these ideas overnight before presenting them to the client.\",\n",
    "    \"The new software is definitely a spicy upgrade, taking some time to get used to.\",\n",
    "    \"Her social media post was the perfect recipe for a viral sensation.\",\n",
    "    \"He served up a rebuttal full of facts, leaving his opponent speechless.\",\n",
    "    \"The team needs to simmer down a bit before tackling the next challenge.\",\n",
    "    \"The presentation was a delicious blend of humor and information, keeping the audience engaged.\",\n",
    "    \"A beautiful array of fake wax foods (shokuhin sampuru) in the front of a Japanese restaurant.\",\n",
    "    \"Daniel Bourke is really cool :D\",\n",
    "    \"My favoruite food is biltong!\"\n",
    "]\n",
    "\n",
    "food_not_food_classifier(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah! That was quick!\n",
    "\n",
    "And it looks like our model performed fairly well.\n",
    "\n",
    "Though there was one harder sample which may be deemed as `food`/`not_food`, the sentence containing \"shokuhin sampuru\" (meaning \"[food model](https://en.wikipedia.org/wiki/Food_model)\" in Japanese). \n",
    "\n",
    "Is a sentence about food models (fake foods) still about food?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time our model across larger sample sizes\n",
    "\n",
    "We can *say* that our model is fast or that making predictions in batches is faster than one at a time.\n",
    "\n",
    "But how about we run some tests to confirm this?\n",
    "\n",
    "Let's start by making predictions one at a time across 100 sentences (10x our `sentences` list) and then we'll write some code to make predictions in batches.\n",
    "\n",
    "We'll time each and see how they go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Number of sentences: 1000\n",
      "[INFO] Time taken for one at a time prediction: 2.5376925468444824 seconds\n",
      "[INFO] Avg inference time per sentence: 0.0025376925468444823 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# Create 1000 sentences\n",
    "sentences_1000 = sentences * 100\n",
    "\n",
    "# Time how long it takes to make predictions on all sentences (one at a time)\n",
    "print(f\"[INFO] Number of sentences: {len(sentences_1000)}\")\n",
    "start_time_one_at_a_time = time.time()\n",
    "for sentence in sentences_1000:\n",
    "    # Make a prediction on each sentence one at a time\n",
    "    food_not_food_classifier(sentence)\n",
    "end_time_one_at_a_time = time.time()\n",
    "\n",
    "print(f\"[INFO] Time taken for one at a time prediction: {end_time_one_at_a_time - start_time_one_at_a_time} seconds\")\n",
    "print(f\"[INFO] Avg inference time per sentence: {(end_time_one_at_a_time - start_time_one_at_a_time) / len(sentences_1000)} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, on my local NVIDIA RTX 4090 GPU, it took around 5.5 seconds to make 1000 predictions one at a time.\n",
    "\n",
    "That's pretty good!\n",
    "\n",
    "But let's see if we can make it faster with batching.\n",
    "\n",
    "To do so, we can increase the size of our `sentences_big` list and pass the list directly to the model to enable batched prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Number of sentences: 100\n",
      "[INFO] Inference time for 100 sentences: 0.04512 seconds.\n",
      "[INFO] Avg inference time per sentence: 0.0004512 seconds.\n",
      "\n",
      "[INFO] Number of sentences: 1000\n",
      "[INFO] Inference time for 1000 sentences: 0.31447 seconds.\n",
      "[INFO] Avg inference time per sentence: 0.00031447 seconds.\n",
      "\n",
      "[INFO] Number of sentences: 10000\n",
      "[INFO] Inference time for 10000 sentences: 1.82615 seconds.\n",
      "[INFO] Avg inference time per sentence: 0.00018261 seconds.\n",
      "\n",
      "[INFO] Number of sentences: 100000\n",
      "[INFO] Inference time for 100000 sentences: 18.63373 seconds.\n",
      "[INFO] Avg inference time per sentence: 0.00018634 seconds.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in [10, 100, 1000, 10_000]:\n",
    "    sentences_big = sentences * i\n",
    "    print(f\"[INFO] Number of sentences: {len(sentences_big)}\")\n",
    "\n",
    "    start_time = time.time()\n",
    "    # Predict on all sentences in batches \n",
    "    food_not_food_classifier(sentences_big)\n",
    "    end_time = time.time()\n",
    "\n",
    "    print(f\"[INFO] Inference time for {len(sentences_big)} sentences: {round(end_time - start_time, 5)} seconds.\")\n",
    "    print(f\"[INFO] Avg inference time per sentence: {round((end_time - start_time) / len(sentences_big), 8)} seconds.\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah!\n",
    "\n",
    "It looks like inference/prediction time is ~10-20x faster when using batched prediction versus predicting one at a time (on my local NVIDIA RTX 4090).\n",
    "\n",
    "I ran some more tests with the same model on a different GPU on Google Colab ([NVIDIA L4 GPU](https://www.nvidia.com/en-au/data-center/l4/)) and got similar results.\n",
    "\n",
    "| Number of Sentences | Total Prediction Time | Prediction Type |\n",
    "|:-----|:-----|:-----|\n",
    "| 100           | 0.62                  | one at a time   |\n",
    "| 1000          | 6.19                  | one at a time   |\n",
    "| 10000         | 61.08                 | one at a time   |\n",
    "| 100000        | 605.46                | one at a time   |\n",
    "| 100           | 0.06                  | batch           |\n",
    "| 1000          | 0.51                  | batch           |\n",
    "| 10000         | 4.97                  | batch           |\n",
    "| 100000        | 49.7                  | batch           |\n",
    "\n",
    "*Testing the speed of a custom text classifier model on different numbers of sentences with one at a time or batched prediction. Tests conducted on Google Colab with a NVIDIA L4 GPU. See the [notebook for code to reproduce](https://colab.research.google.com/drive/14oC-UDCIHrvRZOEKnAvr6JIHrvZzXOkB?usp=sharing).*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HBdDyzEvAvwK"
   },
   "source": [
    "### Making predictions with PyTorch\n",
    "\n",
    "We've seen how to make predictions/perform inference with `transformers.pipeline`, now let's see how to do the same with PyTorch.\n",
    "\n",
    "Performing predictions with PyTorch requires an extra step compared to `pipeline`, we have to prepare our inputs first (turn the text into numbers).\n",
    "\n",
    "Good news is, we can prepare our inputs with the tokenizer that got automatically saved with our model.\n",
    "\n",
    "And since we've already trained a model and uploaded it to the Hugging Face Hub, we can load our model and tokenizer with `transformers.AutoTokenizer` and `transformers.AutoModelForSequenceClassification` passing it the saved path we used (mine is `mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased`).\n",
    "\n",
    "Let's start by loading the tokenizer and see what it looks like to tokenize a piece of sample text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "id": "TEl6YIR6Azbq"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  1037, 12090,  6302,  1997,  1037,  5127,  1997, 13501,  6763,\n",
       "          1010, 11611,  1998, 15174,   102]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "# Setup model path (can be local or on Hugging Face)\n",
    "# Note: Be sure to change \"mrdbourke\" to your own username\n",
    "model_path = \"mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\"\n",
    "\n",
    "# Create an example to predict on\n",
    "sample_text_food = \"A delicious photo of a plate of scrambled eggs, bacon and toast\"\n",
    "\n",
    "# Prepare the tokenizer and tokenize the inputs\n",
    "tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model_path)\n",
    "inputs = tokenizer(sample_text_food, \n",
    "                   return_tensors=\"pt\") # return the output as PyTorch tensors \n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Text tokenized!\n",
    "\n",
    "We get a dictionary of `input_ids` (our text in token form) and `attention_mask` (tells the model which tokens to pay attention to, `1` = pay attention, `0` = no attention).\n",
    "\n",
    "Now we can load the model with the same path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    "\n",
    "# Load our text classification model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(pretrained_model_name_or_path=model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model loaded!\n",
    "\n",
    "Let's make a prediction.\n",
    "\n",
    "We can do so using the context manager `torch.no_grad()` (because no gradients/weights get updated during inference) and passing our model out `inputs` dictionary. \n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "A little tidbit about using dictionaries as function inputs in Python is the ability to unpack the keys of the dictionary into function arguments.\n",
    "\n",
    "This is possible using `**TARGET_DICTIONARY` syntax. Where the `**` means \"use all the keys in the dictionary as function parameters\".\n",
    "\n",
    "For example, the following two lines are equivalent:\n",
    "\n",
    "```\n",
    "# Using ** notation\n",
    "outputs = model(**inputs)\n",
    "\n",
    "# Using explicit notation\n",
    "outputs = model(input_ids=inputs[\"input_ids\"],\n",
    "                attention_mask=inputs[\"attention_mask\"])\n",
    "```\n",
    ":::\n",
    "\n",
    "Let's make a prediction with PyTorch!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequenceClassifierOutput(loss=None, logits=tensor([[-3.3686,  3.9443]]), hidden_states=None, attentions=None)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs) # '**' means input all of the dictionary keys as arguments to the function\n",
    "    # outputs = model(input_ids=inputs[\"input_ids\"],\n",
    "    #                 attention_mask=inputs[\"attention_mask\"]) # same as above, but explicitly passing in the keys\n",
    "\n",
    "outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful, we've got some outputs, which contain `logits` with two values (one for each class).\n",
    "\n",
    "The index of the higher value is our model's predicted class.\n",
    "\n",
    "We can find it by taking the `outputs.logits` and calling [`argmax().item()`](https://pytorch.org/docs/stable/generated/torch.argmax.html) on it.\n",
    "\n",
    "We can also find the prediction probability by passing `outputs.logits` to [`torch.softmax`](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OpETXjJHBD8k",
    "outputId": "bfc30a2d-9a47-441f-f2e9-7cee7fd4989d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: A delicious photo of a plate of scrambled eggs, bacon and toast\n",
      "Predicted label: food\n",
      "Prediction probability: 0.9993335604667664\n"
     ]
    }
   ],
   "source": [
    "# Get predicted class and prediction probability\n",
    "predicted_class_id = outputs.logits.argmax().item()\n",
    "prediction_probability = torch.softmax(outputs.logits, dim=1).max().item()\n",
    "\n",
    "print(f\"Text: {sample_text_food}\")\n",
    "print(f\"Predicted label: {model.config.id2label[predicted_class_id]}\")\n",
    "print(f\"Prediction probability: {prediction_probability}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful! A prediction made with pure PyTorch! It looks very much correct too.\n",
    "\n",
    "How about we put it all together?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: A photo of a broccoli, salmon, rice and radish dish\n",
      "Predicted class: food (prob: 99.94%)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "\n",
    "model_path = \"mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\"\n",
    "\n",
    "# Load the model and tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=model_path)\n",
    "model = AutoModelForSequenceClassification.from_pretrained(pretrained_model_name_or_path=model_path)\n",
    "\n",
    "# Make sample text and tokenize it\n",
    "sample_text = \"A photo of a broccoli, salmon, rice and radish dish\"\n",
    "inputs = tokenizer(sample_text, return_tensors=\"pt\")\n",
    "\n",
    "# Make a prediction\n",
    "with torch.no_grad():\n",
    "    outputs = model(**inputs)\n",
    "\n",
    "# Get predicted class and prediction probability\n",
    "output_logits = outputs.logits\n",
    "predicted_class_id = torch.argmax(output_logits, dim=1).item()\n",
    "predicted_class_label = model.config.id2label[predicted_class_id]\n",
    "predicted_probability = torch.softmax(output_logits, dim=1).max().item()\n",
    "\n",
    "# Print outputs\n",
    "print(f\"Text: {sample_text}\")\n",
    "print(f\"Predicted class: {predicted_class_label} (prob: {predicted_probability * 100:.2f}%)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together\n",
    "\n",
    "Ok, ok, we've covered a lot of ground going from dataset to trained model to making predictions on custom samples.\n",
    "\n",
    "How about we put all of the steps we've covered so far together in a single code cell (or two)?\n",
    "\n",
    "To do so, we'll:\n",
    "\n",
    "1. Import necessary packages (e.g. `datasets`, `transformers.pipeline`, `torch` and more).\n",
    "2. Setup variables for model training and saving pipeline such as our model name, save directory and dataset name.\n",
    "3. Create a directory for saving models.\n",
    "4. Load and preprocess the dataset from Hugging Face Hub using `datasets.load_dataset`.\n",
    "5. Import a tokenizer with `transformers.AutoTokenizer` and map it to our dataset with `dataset.map`.\n",
    "6. Set up an evaluation metric with `evaluate` & create a function to evaluate our model's predictions.\n",
    "7. Import a model with `transformers.AutoModelForSequenceClassification` and prepare it for training with `transformers.TrainingArguments` and `transformers.Trainer`.\n",
    "8. Train the model on our text dataset by calling `transformers.Trainer.train`.\n",
    "9. Save the trained model to a local directory.\n",
    "10. Push the model to the Hugging Face Hub.\n",
    "11. Evaluate the model on the test data.\n",
    "12. Test the trained model on a custom sample using `transformers.pipeline` to make sure it works.\n",
    "\n",
    "Phew!\n",
    "\n",
    "A fair few steps but nothing we can't handle!\n",
    "\n",
    "Let's do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Creating directory for saving models: models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\n",
      "[INFO] Downloading dataset from Hugging Face Hub, name: mrdbourke/learn_hf_food_not_food_image_captions\n",
      "[INFO] Tokenizing text for model training with tokenizer: distilbert/distilbert-base-uncased\n",
      "[INFO] Loading model: distilbert/distilbert-base-uncased\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Model loading complete!\n",
      "[INFO] Commencing model training...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='70' max='70' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [70/70 00:07, Epoch 10/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>0.067892</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.028300</td>\n",
       "      <td>0.009194</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>0.004919</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.002000</td>\n",
       "      <td>0.002121</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.001200</td>\n",
       "      <td>0.001302</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.000900</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.000800</td>\n",
       "      <td>0.000839</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.000700</td>\n",
       "      <td>0.000766</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.000700</td>\n",
       "      <td>0.000728</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.000700</td>\n",
       "      <td>0.000715</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Model training complete, saving model to local path: models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\n",
      "[INFO] Uploading model to Hugging Face Hub...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d19ca05e73243aeb31b954bc693755f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "training_args.bin:   0%|          | 0.00/5.24k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9568ab8358243f689b1610c273396a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94bf9f6e2bdf41a7bc5b8ea16e9d4389",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload 2 LFS files:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Model upload complete, model available at: https://huggingface.co/mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased/tree/main/\n",
      "[INFO] Performing evaluation on test dataset...\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Prediction metrics on the test data:\n",
      "{'test_accuracy': 1.0,\n",
      " 'test_loss': 0.0007152689504437149,\n",
      " 'test_runtime': 0.0507,\n",
      " 'test_samples_per_second': 986.278,\n",
      " 'test_steps_per_second': 39.451}\n"
     ]
    }
   ],
   "source": [
    "# 1. Import necessary packages\n",
    "import pprint\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "import datasets\n",
    "import evaluate\n",
    "\n",
    "from transformers import pipeline\n",
    "from transformers import TrainingArguments, Trainer\n",
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "\n",
    "# 2. Setup variables for model training and saving pipeline\n",
    "DATASET_NAME = \"mrdbourke/learn_hf_food_not_food_image_captions\"\n",
    "MODEL_NAME = \"distilbert/distilbert-base-uncased\"\n",
    "MODEL_SAVE_DIR_NAME = \"models/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\"\n",
    "\n",
    "# 3. Create a directory for saving models\n",
    "# Note: This will override our existing saved model (if there is one)\n",
    "print(f\"[INFO] Creating directory for saving models: {MODEL_SAVE_DIR_NAME}\")\n",
    "model_save_dir = Path(MODEL_SAVE_DIR_NAME)\n",
    "model_save_dir.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# 4. Load and preprocess the dataset from Hugging Face Hub\n",
    "print(f\"[INFO] Downloading dataset from Hugging Face Hub, name: {DATASET_NAME}\")\n",
    "dataset = datasets.load_dataset(path=DATASET_NAME)\n",
    "\n",
    "# Create mappings from id2label and label2id (adjust these for your target dataset, can also create these programmatically)\n",
    "id2label = {0: \"not_food\", 1: \"food\"}\n",
    "label2id = {\"not_food\": 0, \"food\": 1}\n",
    "\n",
    "# Create function to map IDs to labels in dataset\n",
    "def map_labels_to_number(example):\n",
    "    example[\"label\"] = label2id[example[\"label\"]]\n",
    "    return example\n",
    "\n",
    "# Map preprocessing function to dataset\n",
    "dataset = dataset[\"train\"].map(map_labels_to_number)\n",
    "\n",
    "# Split the dataset into train/test sets\n",
    "dataset = dataset.train_test_split(test_size=0.2, seed=42)\n",
    "\n",
    "# 5. Import a tokenizer and map it to our dataset\n",
    "print(f\"[INFO] Tokenizing text for model training with tokenizer: {MODEL_NAME}\")\n",
    "tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path=MODEL_NAME,\n",
    "                                          use_fast=True)\n",
    "\n",
    "# Create a preprocessing function to tokenize text\n",
    "def tokenize_text(examples):\n",
    "    return tokenizer(examples[\"text\"],\n",
    "                     padding=True,\n",
    "                     truncation=True)\n",
    "\n",
    "tokenized_dataset = dataset.map(function=tokenize_text,\n",
    "                                batched=True,\n",
    "                                batch_size=1000)\n",
    "\n",
    "# 6. Set up an evaluation metric & function to evaluate our model\n",
    "accuracy_metric = evaluate.load(\"accuracy\")\n",
    "\n",
    "def compute_accuracy(predictions_and_labels):\n",
    "    predictions, labels = predictions_and_labels\n",
    "\n",
    "    if len(predictions.shape) >= 2:\n",
    "        predictions = np.argmax(predictions, axis=1)\n",
    "    \n",
    "    return accuracy_metric.compute(predictions=predictions, references=labels) # note: use \"references\" parameter rather than \"labels\"\n",
    "\n",
    "\n",
    "# 7. Import a model and prepare it for training \n",
    "print(f\"[INFO] Loading model: {MODEL_NAME}\")\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\n",
    "    pretrained_model_name_or_path=MODEL_NAME,\n",
    "    num_labels=2,\n",
    "    id2label=id2label,\n",
    "    label2id=label2id\n",
    ")\n",
    "print(f\"[INFO] Model loading complete!\")\n",
    "\n",
    "# Setup TrainingArguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=model_save_dir,\n",
    "    learning_rate=0.0001,\n",
    "    per_device_train_batch_size=32,\n",
    "    per_device_eval_batch_size=32,\n",
    "    num_train_epochs=10,\n",
    "    eval_strategy=\"epoch\",\n",
    "    save_strategy=\"epoch\",\n",
    "    save_total_limit=3,\n",
    "    use_cpu=False,\n",
    "    seed=42,\n",
    "    load_best_model_at_end=True,\n",
    "    logging_strategy=\"epoch\",\n",
    "    report_to=\"none\",\n",
    "    push_to_hub=False,\n",
    "    hub_private_repo=False # Note: if set to False, your model will be publically available\n",
    ")\n",
    "\n",
    "# Create Trainer instance and train model\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_dataset[\"train\"],\n",
    "    eval_dataset=tokenized_dataset[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    compute_metrics=compute_accuracy\n",
    ")\n",
    "\n",
    "# 8. Train the model on our text dataset\n",
    "print(f\"[INFO] Commencing model training...\")\n",
    "results = trainer.train()\n",
    "\n",
    "# 9. Save the trained model (note: this will overwrite our previous model, this is ok)\n",
    "print(f\"[INFO] Model training complete, saving model to local path: {model_save_dir}\")\n",
    "trainer.save_model(output_dir=model_save_dir)\n",
    "\n",
    "# 10. Push the model to the Hugging Face Hub\n",
    "print(f\"[INFO] Uploading model to Hugging Face Hub...\")\n",
    "model_upload_url = trainer.push_to_hub(\n",
    "    commit_message=\"Uploading food not food text classifier model\",\n",
    "    # token=\"YOUR_HF_TOKEN_HERE\" # requires a \"write\" HF token \n",
    ")\n",
    "print(f\"[INFO] Model upload complete, model available at: {model_upload_url}\")\n",
    "\n",
    "# 11. Evaluate the model on the test data\n",
    "print(f\"[INFO] Performing evaluation on test dataset...\")\n",
    "predictions_all = trainer.predict(tokenized_dataset[\"test\"])\n",
    "prediction_values = predictions_all.predictions\n",
    "prediction_metrics = predictions_all.metrics\n",
    "\n",
    "print(f\"[INFO] Prediction metrics on the test data:\")\n",
    "pprint.pprint(prediction_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo! It all worked!\n",
    "\n",
    "Now let's make it sure works by turing it into a `transformers.pipeline` and passing it a custom sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[{'label': 'food', 'score': 0.9969706535339355}]]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 12. Make sure the model works by testing it on a custom sample\n",
    "food_not_food_classifier = pipeline(task=\"text-classification\",\n",
    "                                    model=model_save_dir, # can also use model on Hugging Face Hub path \n",
    "                                    device=torch.device(\"cuda\") if torch.cuda.is_available() else \"cpu\",\n",
    "                                    top_k=1,\n",
    "                                    batch_size=32)\n",
    "\n",
    "food_not_food_classifier(\"Yo! We just built a food not food sentence classifier model! Good news is, it can be replicated for other kinds of text classification!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice!\n",
    "\n",
    "Looks like putting all of our code in one cell worked.\n",
    "\n",
    "How about we make our model even more accessible by turning it into a demo?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Turning our model into a demo\n",
    "\n",
    "Once you've trained and saved a model, one of the best ways to continue to test it and show/share it with others is to create a demo.\n",
    "\n",
    "Or step number 8 in our workflow:\n",
    "\n",
    "1. ✅ Create and preprocess data.\n",
    "2. ✅ Define the model we'd like use with [`transformers.AutoModelForSequenceClassification`](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification) (or another similar model class).\n",
    "3. ✅ Define training arguments (these are hyperparameters for our model) with [`transformers.TrainingArguments`](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "4. ✅ Pass `TrainingArguments` from 3 and target datasets to an instance of [`transformers.Trainer`](https://huggingface.co/docs/transformers/en/main_classes/trainer).\n",
    "5. ✅ Train the model by calling [`Trainer.train()`](https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/trainer#transformers.Trainer.train).\n",
    "6. ✅ Save the model (to our local machine or to the Hugging Face Hub).\n",
    "7. ✅ Evaluate the trained model by making and inspecting predctions on the test data.\n",
    "8. Turn the model into a shareable demo.\n",
    "\n",
    "A demo is a small application with the focus of showing the workflow of your model from data in to data out.\n",
    "\n",
    "It's also one way to start testing your model in the wild.\n",
    "\n",
    "You may know where it works and where it doesn't but chances are someone out there will find a new bug before you do.\n",
    "\n",
    "To build our demo, we're going to use an open-source library called [Gradio](https://www.gradio.app/guides/quickstart). \n",
    "\n",
    "Gradio allows you to make machine learning demo apps with Python code and best of all, it's part of the Hugging Face ecosystem so you can share your demo to the public directly through Hugging Face.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/06-gradio-workflow-for-ml-demos.png\"\n",
    "     alt=\"Diagram illustrating the workflow for creating machine learning demos using Gradio. The process starts with data from datasets, which is fed into a model built using Transformers. The model is then used to create a demo hosted on Hub/Spaces/Gradio. The Gradio interface helps visualize the workflow, with inputs such as 'A photo of sushi rolls on a white plate' and 'A yellow tractor driving over a grassy hill'. These inputs are processed by the model to produce outputs classified as 'Food' or 'Not Food'. This setup demonstrates how Gradio facilitates the creation of interactive ML demos.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Going on the premise of data, model, demo, Gradio helps to create the demo. Once you've got a trained model on the Hugging Face Hub, you can setup a Gradio interface to import that model and interact it with it. Gradio interfaces can be deployed on Hugging Face Spaces and shared with others so they can try your model too.</figcaption>\n",
    "</figure>\n",
    "\n",
    "Gradio works on the premise of input -> function (this could be a model) -> output.\n",
    "\n",
    "In our case:\n",
    "\n",
    "* Input = A string of text.\n",
    "* Function = Our trained text classification model.\n",
    "* Output = Predicted output of food/not_food with prediction probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a simple function to perform inference\n",
    "\n",
    "Let's create a function to take an input of text, process it with our model and return a dictionary of the predicted labels.\n",
    "\n",
    "Our function will:\n",
    "\n",
    "1. Take an input of a string of text.\n",
    "2. Setup a text classification pipeline using `transformers.pipeline` as well as our trained model (this can be from our local machine or loaded from Hugging Face). We'll return all the probabilities from the output using `top_k=None`.\n",
    "3. Get the outputs of the text classification pipeline from 2 as a list of dictionaries (e.g. `[{'label': 'food', 'score': 0.999105}, {'label': 'not_food', 'score': 0.00089}]`).\n",
    "4. Format and return the list of dictionaries from 3 to be compatible with Gradio's [`gr.Label`](https://www.gradio.app/docs/gradio/label) output (we'll see this later) which requires a dictionary in the form `[{\"label_1\": probability_1, \"label_2\": probability_2}]`.\n",
    "\n",
    "Onward!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'food': 0.9992194175720215, 'not_food': 0.0007805348141118884}"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import Dict\n",
    "\n",
    "# 1. Create a function which takes text as input \n",
    "def food_not_food_classifier(text: str) -> Dict[str, float]:\n",
    "    \"\"\"\n",
    "    Takes an input string of text and classifies it into food/not_food in the form of a dictionary.\n",
    "    \"\"\"\n",
    "\n",
    "    # 2. Setup the pipeline to use the local model (or Hugging Face model path)\n",
    "    food_not_food_classifier = pipeline(task=\"text-classification\", \n",
    "                                        model=local_model_path,\n",
    "                                        batch_size=32,\n",
    "                                        device=\"cuda\" if torch.cuda.is_available() else \"cpu\", # set the device to work in any environment\n",
    "                                        top_k=None) # return all possible scores (not just top-1)\n",
    "    \n",
    "    # 3. Get outputs from pipeline (as a list of dicts)\n",
    "    outputs = food_not_food_classifier(text)[0]\n",
    "    \n",
    "    # 4. Format output for Gradio (e.g. {\"label_1\": probability_1, \"label_2\": probability_2})\n",
    "    output_dict = {}\n",
    "    for item in outputs:\n",
    "        output_dict[item[\"label\"]] = item[\"score\"]\n",
    "\n",
    "    return output_dict\n",
    "\n",
    "# Test out the function\n",
    "food_not_food_classifier(\"My lunch today was chicken and salad\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful!\n",
    "\n",
    "Looks like our function is working."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a small Gradio demo to run locally\n",
    "\n",
    "We've got a working function to go from text to predicted labels and probabilities.\n",
    "\n",
    "Let's now build a Gradio interface to showcase our model.\n",
    "\n",
    "We can do so by:\n",
    "\n",
    "1. Importing Gradio (using `import gradio as gr`).\n",
    "2. Creating an instance of [`gr.Interface`](https://www.gradio.app/docs/gradio/interface) with parameters `inputs=\"text\"` (for our text-based inputs) called `demo` and `outputs=gr.Label(num_top_classes=2)` to display our output dictionary. We can also add some descriptive aspects to our `demo` with the `title`, `description` and `examples` parameters.\n",
    "3. Running/launching the demo with [`gr.Interface.launch()`](https://www.gradio.app/docs/gradio/interface#interface-launch).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  http://127.0.0.1:7860\n",
      "\n",
      "To create a public link, set `share=True` in `launch()`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"http://127.0.0.1:7860/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. Import Gradio as the common alias \"gr\"\n",
    "import gradio as gr\n",
    "\n",
    "# 2. Setup a Gradio interface to accept text and output labels\n",
    "demo = gr.Interface(\n",
    "    fn=food_not_food_classifier, \n",
    "    inputs=\"text\", \n",
    "    outputs=gr.Label(num_top_classes=2), # show top 2 classes (that's all we have)\n",
    "    title=\"Food or Not Food Classifier\",\n",
    "    description=\"A text classifier to determine if a sentence is about food or not food.\",\n",
    "    examples=[[\"I whipped up a fresh batch of code, but it seems to have a syntax error.\"],\n",
    "              [\"A delicious photo of a plate of scrambled eggs, bacon and toast.\"]])\n",
    "\n",
    "# 3. Launch the interface\n",
    "demo.launch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woohoo!\n",
    "\n",
    "We've made a very clean way of interacting with our model.\n",
    "\n",
    "However, our model is still only largely accessible to us (except for the model file we've uploaded to Hugging Face).\n",
    "\n",
    "How about we make our demo publicly available so it's even easier for people to interact with our model?\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "The [`gradio.Interface`](https://www.gradio.app/docs/gradio/interface) class is full of many different options, I'd highly recommend reading through the documentation for 10-15 minutes to get an idea of it.\n",
    "\n",
    "If your workflow requires inputs -> function (e.g. a model making predictions on the input) -> output, chances are, you can build it with Gradio.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making our demo publicly accessible\n",
    "\n",
    "One of the best ways to share your machine learning work is by creating an application.\n",
    "\n",
    "And one of the best places to share your applications is [Hugging Face Spaces](https://huggingface.co/docs/hub/spaces).\n",
    "\n",
    "Hugging Face Spaces allows you to host machine learning (and non-machine learning) applications for free (with [optional paid hardware upgrades](https://huggingface.co/docs/hub/spaces-overview#hardware-resources)).\n",
    "\n",
    "If you're familiar with GitHub, Hugging Face Spaces works similar to a GitHub repository (each Space is a Git repository itself).\n",
    "\n",
    "If not, that's okay, think of Hugging Face Spaces as an online folder where you can upload your files and have them accessed by others.\n",
    "\n",
    "Creating a Hugging Face Space can be done in two main ways:\n",
    "\n",
    "1. **Manually** - By going to the [Hugging Face Spaces](https://huggingface.co/spaces) website and clicking \"Create new space\". Or by going directly to [https://www.huggingface.co/new-space](https://huggingface.co/new-space). Here, you'll be able to setup a few settings for your Space and choose the framework/runtime (e.g. Streamlit, Gradio, Docker and more).\n",
    "2. **Programmatically** - By using the [Hugging Face Hub Python API](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) we can write code to [directly upload files to the Hugging Face Hub](https://www.gradio.app/guides/using-hugging-face-integrations#hosting-your-gradio-demos-on-spaces), including Hugging Face Spaces.\n",
    "\n",
    "Both are great options but we're going to take the second approach.\n",
    "\n",
    "This is so we can create our Hugging Face Space right from this notebook.\n",
    "\n",
    "To do so, we'll create three files:\n",
    "\n",
    "1. `app.py` - This will be the Python file which will be the main running file on our Hugging Face Space. Inside we'll include all the code necessary to run our Gradio demo (as above). Hugging Face Spaces will automatically recoginize the `app.py` file and run it for us.\n",
    "2. `requirements.txt` - This text file will include all of the Python packages we need to run our `app.py` file. Before our Space starts to run, all of the packages in this file will be installed.\n",
    "3. `README.md` - This markdown file will include details about our Space as well as specific Space-related metadata (we'll see this later on).\n",
    "\n",
    "We'll create these files with the following file structure:\n",
    "\n",
    "```\n",
    "demos/\n",
    "└── food_not_food_text_classifier/\n",
    "    ├── app.py\n",
    "    ├── README.md\n",
    "    └── requirements.txt\n",
    "```\n",
    "\n",
    "Why this way?\n",
    "\n",
    "Doing it in the above style means we'll have a directory which contains all of our demos (`demos/`) as well as a dedicated directory which contains our `food`/`not_food` demo application (`food_not_food_text_classifier/`).\n",
    "\n",
    "This way, we'll be able to upload the whole `demos/food_not_food_text_classifier/` folder to Hugging Face Spaces.\n",
    "\n",
    "Let's start by making a directory to store our demo application files.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Make a directory for demos\n",
    "demos_dir = Path(\"../demos\")\n",
    "demos_dir.mkdir(exist_ok=True)\n",
    "\n",
    "# Create a folder for the food_not_food_text_classifer demo\n",
    "food_not_food_text_classifier_demo_dir = Path(demos_dir, \"food_not_food_text_classifier\")\n",
    "food_not_food_text_classifier_demo_dir.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Demo directory created, let's now create our requried files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making an app file\n",
    "\n",
    "Our `app.py` file will be the main part of our Hugging Face Space.\n",
    "\n",
    "The good news is, we've already created most of it when we created our original demo. \n",
    "\n",
    "Inside the `app.py` folder we'll:\n",
    "\n",
    "1. Import the required libraries/packages for running our demo app.\n",
    "2. Setup a function for going from text to our trained model's predicted outputs. And because our model is already hosted on the Hugging Face Hub, we can pass `pipeline` our model's name (e.g. `mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased`) and when we upload our `app.py` file to Hugging Face Spaces, it will load the model directly from the Hub.\n",
    "    * **Note:** Be sure to change \"`mrdbourke`\" to your own Hugging Face username.\n",
    "3. Create a demo just as before with `gr.Interface`.\n",
    "4. Launch our demo with `gr.Interface.launch`.\n",
    "\n",
    "We can write all of the above in a notebook cell.\n",
    "\n",
    "And we can turn it into a file by using the [`%%writefile`](https://ipython.readthedocs.io/en/stable/interactive/magics.html#cellmagic-writefile) magic command and passing it our target filepath.\n",
    "\n",
    "Let's do it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../demos/food_not_food_text_classifier/app.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../demos/food_not_food_text_classifier/app.py\n",
    "# 1. Import the required packages\n",
    "import torch\n",
    "import gradio as gr\n",
    "\n",
    "from typing import Dict\n",
    "from transformers import pipeline\n",
    "\n",
    "# 2. Define function to use our model on given text \n",
    "def food_not_food_classifier(text: str) -> Dict[str, float]:\n",
    "    # Set up text classification pipeline\n",
    "    food_not_food_classifier = pipeline(task=\"text-classification\", \n",
    "                                        # Because our model is on Hugging Face already, we can pass in the model name directly\n",
    "                                        model=\"mrdbourke/learn_hf_food_not_food_text_classifier-distilbert-base-uncased\", # link to model on HF Hub\n",
    "                                        device=\"cuda\" if torch.cuda.is_available() else \"cpu\",\n",
    "                                        top_k=None) # return all possible scores (not just top-1)\n",
    "    \n",
    "    # Get outputs from pipeline (as a list of dicts)\n",
    "    outputs = food_not_food_classifier(text)[0]\n",
    "\n",
    "    # Format output for Gradio (e.g. {\"label_1\": probability_1, \"label_2\": probability_2})\n",
    "    output_dict = {}\n",
    "    for item in outputs:\n",
    "        output_dict[item[\"label\"]] = item[\"score\"]\n",
    "\n",
    "    return output_dict\n",
    "\n",
    "# 3. Create a Gradio interface with details about our app\n",
    "description = \"\"\"\n",
    "A text classifier to determine if a sentence is about food or not food. \n",
    "\n",
    "Fine-tuned from [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on a [small dataset of food and not food text](https://huggingface.co/datasets/mrdbourke/learn_hf_food_not_food_image_captions).\n",
    "\n",
    "See [source code](https://github.com/mrdbourke/learn-huggingface/blob/main/notebooks/hugging_face_text_classification_tutorial.ipynb).\n",
    "\"\"\"\n",
    "\n",
    "demo = gr.Interface(fn=food_not_food_classifier, \n",
    "             inputs=\"text\", \n",
    "             outputs=gr.Label(num_top_classes=2), # show top 2 classes (that's all we have)\n",
    "             title=\"🍗🚫🥑 Food or Not Food Text Classifier\",\n",
    "             description=description,\n",
    "             examples=[[\"I whipped up a fresh batch of code, but it seems to have a syntax error.\"],\n",
    "                       [\"A delicious photo of a plate of scrambled eggs, bacon and toast.\"]])\n",
    "\n",
    "# 4. Launch the interface\n",
    "if __name__ == \"__main__\":\n",
    "    demo.launch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`app.py` file created!\n",
    "\n",
    "Now let's setup the requirements file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a requirements file\n",
    "\n",
    "When you upload an `app.py` file to Hugging Face Spaces, it will attempt to run it automatically.\n",
    "\n",
    "And just like running the file locally, we need to make sure all of the required packages are available.\n",
    "\n",
    "Otherwise our Space will produce an error like the following:\n",
    "\n",
    "```\n",
    "===== Application Startup at 2024-06-13 05:37:21 =====\n",
    "\n",
    "Traceback (most recent call last):\n",
    "  File \"/home/user/app/app.py\", line 1, in <module>\n",
    "    import torch\n",
    "ModuleNotFoundError: No module named 'torch'\n",
    "```\n",
    "\n",
    "Good news is, our demo only has three requirements: `gradio`, `torch`, `transformers`.\n",
    "\n",
    "Let's create a `requirements.txt` file with the packages we need and save it to the same directory as our `app.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../demos/food_not_food_text_classifier/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../demos/food_not_food_text_classifier/requirements.txt\n",
    "gradio\n",
    "torch\n",
    "transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful!\n",
    "\n",
    "Hugging Face Spaces will automatically recognize the `requirements.txt` file and install the listed packages into our Space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a README file\n",
    "\n",
    "Our `app.py` can contain information about our demo, however, we can also use a `README.md` file to further communicate our work.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "It is common practice in Git repositories (including GitHub and Hugging Face Hub) to add a `README.md` file to your project so people can read more (hence \"read me\") about what your project is about.\n",
    "\n",
    ":::\n",
    "\n",
    "We can include anything in [markdown-style text](https://huggingface.co/docs/hub/en/spaces-config-reference) in the `README.md` file.\n",
    "\n",
    "However, Spaces also have a special [YAML block](https://simple.wikipedia.org/wiki/YAML) at the top of the `README.md` file in the root directory with configuration details.\n",
    "\n",
    "Inside the YAML block you can put special metadata details about your Space including:\n",
    "\n",
    "* `title` - The title of your Space (e.g. `title: Food Not Food Text Classifier`).\n",
    "* `emoji` - The emoji to display on your Space (e.g. `emoji: 🍗🚫🥑`).\n",
    "* `app_file` - The target app file for Spaces to run (set to `app_file: app.py` by default).\n",
    "\n",
    "And there are plenty more in the [Spaces Configuration References documentation](https://huggingface.co/docs/hub/en/spaces-config-reference).\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "    <!-- figtemplate -->\n",
    "    <img src=\"https://huggingface.co/datasets/mrdbourke/learn-hf-images/resolve/main/learn-hf-text-classification/08-huggingface-spaces-yaml-front-matter.png\"\n",
    "     alt=\"Screenshot of a README.md file on Hugging Face Spaces. The file includes YAML front matter and Markdown content. The YAML front matter specifies the title as 'Food Not Food Text Classifier', with other attributes such as emoji, colorFrom, colorTo, sdk (Gradio), sdk_version, app_file, pinned, and license. Below the YAML section, the Markdown content includes a header 'Food Not Food Text Classifier' and a description of the project, which is a small demo to showcase a text classifier determining if a sentence is about food or not. The classifier is based on a DistilBERT model fine-tuned on a small synthetic dataset of 250 generated food or not food image captions.\" \n",
    "     style=\"width: 100%; max-width: 900px; height: auto;\"/>\n",
    "     <figcaption>Example of Hugging Face Spaces README.md file with YAML front matter (front matter is another term for \"things at the front/top of the file\") for formatting the Space.</figcaption>\n",
    "</figure>\n",
    "\n",
    "Let's create a `README.md` file with a YAML block at the top detailing some of the metadata about our project.\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "The YAML block at the top of the `README.md` can take some practice. \n",
    "\n",
    "If you want to see a demo of how one gets created, try making a Hugging Face Space with the \"Create new Space\" button on the [https://huggingface.co/spaces](https://huggingface.co/spaces) page and seeing what the `README.md` file starts with (that's how I found out what to do!).\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ../demos/food_not_food_text_classifier/README.md\n"
     ]
    }
   ],
   "source": [
    "%%writefile ../demos/food_not_food_text_classifier/README.md\n",
    "---\n",
    "title: Food Not Food Text Classifier\n",
    "emoji: 🍗🚫🥑\n",
    "colorFrom: blue\n",
    "colorTo: yellow\n",
    "sdk: gradio\n",
    "app_file: app.py\n",
    "pinned: false\n",
    "license: apache-2.0\n",
    "---\n",
    "\n",
    "# 🍗🚫🥑 Food Not Food Text Classifier\n",
    "\n",
    "Small demo to showcase a text classifier to determine if a sentence is about food or not food.\n",
    "\n",
    "DistillBERT model fine-tuned on a small synthetic dataset of 250 generated [Food or Not Food image captions](https://huggingface.co/datasets/mrdbourke/learn_hf_food_not_food_image_captions).\n",
    "\n",
    "[Source code notebook](https://github.com/mrdbourke/learn-huggingface/blob/main/notebooks/hugging_face_text_classification_tutorial.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`README.md` created!\n",
    "\n",
    "Now let's check out the files we have in our `demos/food_not_food_text_classifier/` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "README.md  app.py  requirements.txt\n"
     ]
    }
   ],
   "source": [
    "!ls ../demos/food_not_food_text_classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect!\n",
    "\n",
    "Looks like we've got all the files we need to create our Space.\n",
    "\n",
    "Let's upload them to the Hugging Face Hub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading our demo to Hugging Face Spaces\n",
    "\n",
    "We've created all of the files required for our demo, now for the fun part!\n",
    "\n",
    "Let's upload them to Hugging Face Spaces.\n",
    "\n",
    "To do so programmatically, we can use the [Hugging Face Hub Python API](https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api).\n",
    "\n",
    "::: {.callout-note}\n",
    "\n",
    "The [Hugging Face Hub Python API](https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api) has many different options for interacting with the Hugging Face Hub programmatically.\n",
    "\n",
    "You can create repositories, upload files, upload folders, add comments, change permissions and much much more.\n",
    "\n",
    "Be sure to explore the documentation for at least 10-15 minutes to get an idea of what's possible.\n",
    "\n",
    ":::\n",
    "\n",
    "To get our demo hosted on Hugging Face Spaces we'll go through the following steps:\n",
    "\n",
    "1. Import the required methods from the `huggingface_hub` package, including [`create_repo`](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api#huggingface_hub.HfApi.create_repo), [`get_full_repo_name`](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api#huggingface_hub.HfApi.get_full_repo_name), [`upload_file`](https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api#huggingface_hub.HfApi.upload_file) (optional, we'll be using `upload_folder`) and [`upload_folder`](https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api#huggingface_hub.HfApi.upload_folder).\n",
    "2. Define the demo folder we'd like to upload as well as the different parameters for the Hugging Face Space such as repo type (`\"space\"`), our target Space name, the target Space SDK (`\"gradio\"`), our [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) with write access (optional if it already isn't setup).\n",
    "3. Create a repository on Hugging Face Spaces using the `huggingface_hub.create_repo` method and filling out the appropriate parameters.\n",
    "4. Get the full name of our created repository using the `huggingface_hub.get_full_repo_name` method (we could hard code this but I like to get it programmatically incase things change).\n",
    "5. Upload the contents of our target demo folder (`../demos/food_not_food_text_classifier/`) to Hugging Face Hub with [`huggingface_hub.upload_folder`](https://huggingface.co/docs/huggingface_hub/en/package_reference/hf_api#huggingface_hub.HfApi.upload_folder).\n",
    "6. Hope it all works and inspect the results! 🤞\n",
    "\n",
    "A fair few steps but we've got this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Creating repo on Hugging Face Hub with name: learn_hf_food_not_food_text_classifier_demo\n",
      "[INFO] Full Hugging Face Hub repo name: mrdbourke/learn_hf_food_not_food_text_classifier_demo\n",
      "[INFO] Uploading ../demos/food_not_food_text_classifier to repo: mrdbourke/learn_hf_food_not_food_text_classifier_demo\n",
      "[INFO] Demo folder successfully uploaded with commit URL: https://huggingface.co/spaces/mrdbourke/learn_hf_food_not_food_text_classifier_demo/tree/main/.\n"
     ]
    }
   ],
   "source": [
    "# 1. Import the required methods for uploading to the Hugging Face Hub\n",
    "from huggingface_hub import (\n",
    "    create_repo,\n",
    "    get_full_repo_name,\n",
    "    upload_file, # for uploading a single file (if necessary)\n",
    "    upload_folder # for uploading multiple files (in a folder)\n",
    ")\n",
    "\n",
    "# 2. Define the parameters we'd like to use for the upload\n",
    "LOCAL_DEMO_FOLDER_PATH_TO_UPLOAD = \"../demos/food_not_food_text_classifier\"\n",
    "HF_TARGET_SPACE_NAME = \"learn_hf_food_not_food_text_classifier_demo\"\n",
    "HF_REPO_TYPE = \"space\" # we're creating a Hugging Face Space\n",
    "HF_SPACE_SDK = \"gradio\"\n",
    "HF_TOKEN = \"\" # optional: set to your Hugging Face token (but I'd advise storing this as an environment variable as previously discussed)\n",
    "\n",
    "# 3. Create a Space repository on Hugging Face Hub \n",
    "print(f\"[INFO] Creating repo on Hugging Face Hub with name: {HF_TARGET_SPACE_NAME}\")\n",
    "create_repo(\n",
    "    repo_id=HF_TARGET_SPACE_NAME,\n",
    "    # token=HF_TOKEN, # optional: set token manually (though it will be automatically recognized if it's available as an environment variable)\n",
    "    repo_type=HF_REPO_TYPE,\n",
    "    private=False, # set to True if you don't want your Space to be accessible to others\n",
    "    space_sdk=HF_SPACE_SDK,\n",
    "    exist_ok=True, # set to False if you want an error to raise if the repo_id already exists \n",
    ")\n",
    "\n",
    "# 4. Get the full repository name (e.g. {username}/{model_id} or {username}/{space_name})\n",
    "full_hf_repo_name = get_full_repo_name(model_id=HF_TARGET_SPACE_NAME)\n",
    "print(f\"[INFO] Full Hugging Face Hub repo name: {full_hf_repo_name}\")\n",
    "\n",
    "# 5. Upload our demo folder\n",
    "print(f\"[INFO] Uploading {LOCAL_DEMO_FOLDER_PATH_TO_UPLOAD} to repo: {full_hf_repo_name}\")\n",
    "folder_upload_url = upload_folder(\n",
    "    repo_id=full_hf_repo_name,\n",
    "    folder_path=LOCAL_DEMO_FOLDER_PATH_TO_UPLOAD,\n",
    "    path_in_repo=\".\", # upload our folder to the root directory (\".\" means \"base\" or \"root\", this is the default)\n",
    "    # token=HF_TOKEN, # optional: set token manually\n",
    "    repo_type=HF_REPO_TYPE,\n",
    "    commit_message=\"Uploading food not food text classifier demo app.py\"\n",
    ")\n",
    "print(f\"[INFO] Demo folder successfully uploaded with commit URL: {folder_upload_url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent!\n",
    "\n",
    "Looks like all of the files in our target demo folder were uploaded!\n",
    "\n",
    "Once this happens, Hugging Face Spaces will take a couple of minutes to build our application.\n",
    "\n",
    "If there are any errors, it will let us know.\n",
    "\n",
    "Otherwise, our demo application should be running live and be ready to test at a URL similar to: [https://huggingface.co/spaces/mrdbourke/learn_hf_food_not_food_text_classifier_demo](https://huggingface.co/spaces/mrdbourke/learn_hf_food_not_food_text_classifier_demo) (though you may have to swap my username \"`mrdbourke`\" for your own as well as the name you chose for the Space)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing our hosted demo\n",
    "\n",
    "One of the really cool things about Hugging Face Spaces is that we can share our demo application as a link so others can try it out.\n",
    "\n",
    "We can also embed it right into our notebook.\n",
    "\n",
    "To do so, we can go to the three dots in the top right of our [hosted Space](https://huggingface.co/spaces/mrdbourke/learn_hf_food_not_food_text_classifier_demo) and select \"Embed this Space\".\n",
    "\n",
    "We then have the option to embed our Space using a JavaScript web component, HTML `iframe` or via the direct URL.\n",
    "\n",
    "Since Jupyter notebooks have the ability to render HTML via [`IPython.display.HTML`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.HTML), let's embed our Space with HTML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<iframe\n",
       "\tsrc=\"https://mrdbourke-learn-hf-food-not-food-text-classifier-demo.hf.space\"\n",
       "\tframeborder=\"0\"\n",
       "\twidth=\"850\"\n",
       "\theight=\"450\"\n",
       "></iframe>     \n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "\n",
    "# You can get embeddable HTML code for your demo by clicking the \"Embed\" button on the demo page\n",
    "HTML(data='''\n",
    "<iframe\n",
    "\tsrc=\"https://mrdbourke-learn-hf-food-not-food-text-classifier-demo.hf.space\"\n",
    "\tframeborder=\"0\"\n",
    "\twidth=\"850\"\n",
    "\theight=\"450\"\n",
    "></iframe>     \n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that's cool!\n",
    "\n",
    "We can try out our Food Not Food Text Classifier app from right within our notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "You should be very proud of yourself!\n",
    "\n",
    "We've just gone end-to-end on a machine learning workflow with Hugging Face.\n",
    "\n",
    "From loading a dataset to training a model to deploying that model in the form of a public demo.\n",
    "\n",
    "Here are some of the main takeaways from this project.\n",
    "\n",
    "**The Hugging Face ecosystem is a collection of powerful and open-source tools for machine learning workflows.**\n",
    "\n",
    "* Hugging Face `datasets` helps you to store and preprocess datasets of almost any shape and size.\n",
    "* Hugging Face `transformers` has many built-in pretrained models for many different use cases and components such as `transformers.Trainer` help you to tailor those models to your own custom use cases.\n",
    "* Hugging Face `tokenizers` works closely with `transformers` and allows the efficient conversion of raw text data into numerical representation (which is required for machine learning models). \n",
    "* The Hugging Face Hub is a great place to share your models and machine learning projects. Over time, you can build up a portfolio of machine learning-based projects to show future employers or clients and to help the community grow.\n",
    "* There are many more, but I'll leave these for you to explore as extra-curriculum.\n",
    "\n",
    "**A common machine learning workflow: dataset -> model -> demo.**\n",
    "\n",
    "Before a machine learning model is incorporated into a larger application, a very common workflow is to:\n",
    "\n",
    "1. Find an existing or create a new dataset for your specific problem.\n",
    "2. Train/fine-tune and evaluate an existing model on your dataset.\n",
    "3. Create a small demo application to test your trained model. \n",
    "\n",
    "We've just gone through all of these steps for text classification! \n",
    "\n",
    "Text classification is a very common problem in many business settings. If you have a similar problem but a different dataset, you can replicate this workflow.\n",
    "\n",
    "**Building your own model has several advantages over using APIs.**\n",
    "\n",
    "APIs are very helpful to try something out.\n",
    "\n",
    "However, depending on your use case, you may often want to create your own custom model.\n",
    "\n",
    "Training your own model can often result in faster predictions and far less running costs over time.\n",
    "\n",
    "The Hugging Face ecosystem enables the creation of custom models for almost any kind of machine learning problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises and Extensions\n",
    "\n",
    "There's no better way to improve other than practicing what you've learned.\n",
    "\n",
    "The following exercises and extensions are designed for you to practice the things we've covered in this project.\n",
    "\n",
    "1. Our text classification model works on `food`/`not_food` text samples. How would you create your own binary text classification model on different classes?\n",
    "    * Create ~10 or samples of your own text classes (e.g. 10 samples each of `spam`/`not_spam` emails) and retrain a text classification model.\n",
    "    * Bonus: Share the model you've made in a demo just like we did here. Send it to me, I'd love to see it! My email is on [my website](https://www.mrdbourke.com).\n",
    "2. We've trained our model on two classes (binary classification) but how might we increase that to 3 or more classes (multi-class classification)?\n",
    "    * Hint: see the `num_labels` parameter in `transformers.AutoModelForSequenceClassification`. \n",
    "3. Our model seems to work pretty good on our test data and on the few number of examples we tried manually. Can you find any examples where our model fails? For example, what kind of sentences does it struggle with? How could you fix this?\n",
    "    * Hint: Our model has been trained on examples with at least 5-12 words, does it still work with short sentences? (e.g. \"pie\").\n",
    "    * Bonus: If you find any cases where our model doesn't perform well, make an extra 10-20 examples of these and add them to the dataset and then retrain the model (you'll have to lookup [\"how to add rows to an existing Hugging Face dataset\"](https://discuss.huggingface.co/t/how-do-i-add-things-rows-to-an-already-saved-dataset/27423)). How does the model perform after adding these additional samples?\n",
    "4. Datasets are fundamental to any machine learning project, getting to know how to process and interact with them is a fundamental skill. Spend 1 hour going through the [Hugging Face Datasets tutorial](https://huggingface.co/docs/datasets/en/tutorial).\n",
    "    * Write 5 things you can do with Hugging Face Datasets and where they might come in handy.\n",
    "5. The Hugging Face `transformers` library has many features. The following readings are to help understand a handful of them.\n",
    "    * Spend 10 minutes exploring the [`transformers.TrainingArguments` documentation](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.TrainingArguments).\n",
    "    * Spend 10 minutes reading the [`transformers.Trainer` documentation](https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer).  \n",
    "    * Spend 10 minutes reading the Hugging Face [model sharing documentation](https://huggingface.co/docs/transformers/en/model_sharing).  \n",
    "    * Spend 10 minutes reading the Hugging Face [`transformers.pipeline` documentation](https://huggingface.co/docs/transformers/en/main_classes/pipelines#pipelines).  \n",
    "        * What does a `pipeline` do?\n",
    "        * Name 3 different kinds of pipelines and describe what they do in a sentence\n",
    "6. Gradio is a powerful library for making machine learning demos, learning more about it will help you in future creations. Spend 10-15 minutes reading the [Gradio quickstart documentation](https://www.gradio.app/guides/quickstart). \n",
    "    * What are 3 kinds of demos you can create?\n",
    "    * What are 3 different inputs and outputs you can make?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra resources\n",
    "\n",
    "There are many things we touched over but didn't go into much depth in this notebook.\n",
    "\n",
    "The following resources are for those who'd like to learn a little bit more.\n",
    "\n",
    "* See how the food not food image caption dataset was created with synthetic text data (image captions generated by a Large Language Model) in the example [Google Colab notebook](https://colab.research.google.com/drive/14xr3KN_HINY5LjV0s2E-4i7v0o_XI3U8?usp=sharing).\n",
    "* Hugging Face have a [great guide on sequence classification](https://huggingface.co/docs/transformers/en/tasks/sequence_classification) (it's what this notebook was built on).\n",
    "* For more on the concept of padding and truncation in sequence processing, I'd recommend the [Hugging Face padding and truncation guide](https://huggingface.co/docs/transformers/en/pad_truncation). \n",
    "* For more on Transformers (the architecture) as well as the DistilBert model:\n",
    "    - Read [*Transformers from scratch*](https://peterbloem.nl/blog/transformers) by Peter Bloem.\n",
    "    - Watch [Andrej Karpathy's lecture on Transformers and their history](https://www.youtube.com/watch?v=XfpMkf4rD6E).\n",
    "    - Read the original [*Attention is all you need*](https://arxiv.org/abs/1706.03762) paper (the paper that introduced the Transformer architecture).\n",
    "    - Read the [*DistilBert paper*](https://arxiv.org/abs/1910.01108) from the Hugging Face team (paper that introduced the DistilBert architecture and training setup).\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "L4",
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "016181d9f51a4abea666046fb95f2aee": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "01f82250d2e14f198dd0f561969d54ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0233456210f8469f85e648ba1c1f83ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2afc2c8167ad4d15ab6a35c1d1f1f3cb",
      "max": 152322,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_070785d29788406db9369ce39e24b7cf",
      "value": 152322
     }
    },
    "054e759a27a442afa8c666fc3d4c3f2e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0673f25c77b04212bc2b63f56454a5d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "070785d29788406db9369ce39e24b7cf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "0eaf7b7abd71413597e0533efa3feda2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c56c3c1ff2a040b0952b5bac655b71dc",
      "placeholder": "​",
      "style": "IPY_MODEL_3d5b841a59a3489ea5b4112182d4ae70",
      "value": " 48.0/48.0 [00:00&lt;00:00, 3.86kB/s]"
     }
    },
    "103ff0dd04da40a69d85763bbf571490": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "13b1494b02194773a5fafe39b211c88a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1aa8fdd087ce42bdb16ae19a38e79b27": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_80ef815be4ac4b64b3b3d99a6c43fadb",
      "placeholder": "​",
      "style": "IPY_MODEL_d1cc110825034545a2532fc567b5325e",
      "value": " 232k/232k [00:00&lt;00:00, 3.49MB/s]"
     }
    },
    "1b61ce44fb554db2bb906fb39b41b7e0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1c5d45fa6fb14097bf36584ec7ef651c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9ccbc146f4ad4382879b5d8b526cac4e",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_fda99413adc8497cba92a9c6416f77ad",
      "value": 231508
     }
    },
    "20a883d10ea945a9889429c5efb76a43": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2afc2c8167ad4d15ab6a35c1d1f1f3cb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2cb043cfb0fc448e81a34710d65bb691": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2f44328dc8a04f84a59e26d08d343c77": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "31cf1f106af54f7986b4d2f462125cca": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_506baa608274455eae12e54e1d7b7dd5",
      "placeholder": "​",
      "style": "IPY_MODEL_103ff0dd04da40a69d85763bbf571490",
      "value": "vocab.txt: 100%"
     }
    },
    "347ce846f3564c58ac1eb26ae2f3adc4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a20bdb762700449097d5010ce7d29921",
      "max": 4203,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2cb043cfb0fc448e81a34710d65bb691",
      "value": 4203
     }
    },
    "34edb9945bb54a69868d62ecd5b1ae56": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "35205e647f814b97a2e977af48cc7a17": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8580f622bac8422a992b56b5f39b3693",
      "max": 48,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ca74fdefa7844f11b1d9052269efd585",
      "value": 48
     }
    },
    "37cede645a5b48a9b89df987b9ef91e2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "39f3c0ec5b4446cda93df7a999d4fd41": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3d5b841a59a3489ea5b4112182d4ae70": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "470f2248b2444869914197c7481e10ae": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_54e1a02cf71743ec9da128ff6c882f34",
      "placeholder": "​",
      "style": "IPY_MODEL_6429d7801ba14ed4861726bd7832ecee",
      "value": "config.json: 100%"
     }
    },
    "495cd338f7614f7a9a68fc3dc3916dcf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b6f81607c31496a83d5964983fed168": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4c92f17cff2944a9a2a49be56649d81e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4df9f85ae9e943ceb3ed1a5d9e143f5c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4ef091cf8c084af1acb95d62ffe1f0f2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_495cd338f7614f7a9a68fc3dc3916dcf",
      "placeholder": "​",
      "style": "IPY_MODEL_dc17c88c62244e2f9a574ef895132a83",
      "value": "tokenizer_config.json: 100%"
     }
    },
    "4ef76858d67f4ee6a99b305edb682a43": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4f8713c60a0947ae86ee9d1d4de8ab3a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "506baa608274455eae12e54e1d7b7dd5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "522f6c1e5e7340039b7bbf700704699f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6294e85e97a8495bb18d11dff1991a79",
      "placeholder": "​",
      "style": "IPY_MODEL_91ee20aae0474a4c90a26463f72bacf1",
      "value": " 30465/30465 [00:00&lt;00:00, 31103.17 examples/s]"
     }
    },
    "54e1a02cf71743ec9da128ff6c882f34": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5616046a1d1347cfbbe82fa67b79af1a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "593b4aaa5c9e4e1898cc4d5064940770": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5b1fd51b695b4c81ab351ed5b1f92966": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "60e84d7134b646f6b291b0439d9ec890": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6294e85e97a8495bb18d11dff1991a79": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "63e64e93abca400293b7b7742bd94aaf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b791095b95214d82af1e344a2b5898a2",
      "placeholder": "​",
      "style": "IPY_MODEL_a3fa870efa0f4463b1e60fa106f782f1",
      "value": " 268M/268M [00:01&lt;00:00, 287MB/s]"
     }
    },
    "6429d7801ba14ed4861726bd7832ecee": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "64513b9e43254d5a9bbbffc9cb9a054c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6a7b459551104d47a90527c885647830": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4f8713c60a0947ae86ee9d1d4de8ab3a",
      "placeholder": "​",
      "style": "IPY_MODEL_4b6f81607c31496a83d5964983fed168",
      "value": " 483/483 [00:00&lt;00:00, 39.6kB/s]"
     }
    },
    "6cdf8319c7184ee5b107cea1718b03c6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4ef091cf8c084af1acb95d62ffe1f0f2",
       "IPY_MODEL_35205e647f814b97a2e977af48cc7a17",
       "IPY_MODEL_0eaf7b7abd71413597e0533efa3feda2"
      ],
      "layout": "IPY_MODEL_34edb9945bb54a69868d62ecd5b1ae56"
     }
    },
    "72df691ec7784ae7b0542171a7789555": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_39f3c0ec5b4446cda93df7a999d4fd41",
      "placeholder": "​",
      "style": "IPY_MODEL_bb7f6dabc3ad4779a87b211227040606",
      "value": " 466k/466k [00:00&lt;00:00, 2.36MB/s]"
     }
    },
    "79144b8d1ec44b6bb3dcc381f7e829ce": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7b91cb6b9b5e4ba891a6cfaab2d5948b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "80995fb6ed6f4557b1037dea2798b2fb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_eb256ba48acd446bbe6b832c9558f7c6",
      "placeholder": "​",
      "style": "IPY_MODEL_4ef76858d67f4ee6a99b305edb682a43",
      "value": " 121857/121857 [00:04&lt;00:00, 25895.88 examples/s]"
     }
    },
    "80ef815be4ac4b64b3b3d99a6c43fadb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8580f622bac8422a992b56b5f39b3693": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8c3f85da75764955a572261460db619b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e2f32fe9433d458dbc278da438f27449",
      "max": 121857,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_64513b9e43254d5a9bbbffc9cb9a054c",
      "value": 121857
     }
    },
    "8fc36b3359d24eb987c4be9eaf5d4e95": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d6b2772488464e80b333332d6cd46eeb",
      "placeholder": "​",
      "style": "IPY_MODEL_5616046a1d1347cfbbe82fa67b79af1a",
      "value": "model.safetensors: 100%"
     }
    },
    "90455578408b4f33bc9a9c732d431236": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a2e056a63fd34329a21330a4fc69429d",
       "IPY_MODEL_347ce846f3564c58ac1eb26ae2f3adc4",
       "IPY_MODEL_9a01bb7c8f5e4940ba2444fa63a3db6f"
      ],
      "layout": "IPY_MODEL_b8beefe086284c7297004fcad9303617"
     }
    },
    "91ee20aae0474a4c90a26463f72bacf1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9455512d944c418b87e4d1fe324619e6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b93a51136d1146e08914db125d0259fb",
       "IPY_MODEL_c74948f6d29e4e0e868a3ccf836d2377",
       "IPY_MODEL_522f6c1e5e7340039b7bbf700704699f"
      ],
      "layout": "IPY_MODEL_0673f25c77b04212bc2b63f56454a5d2"
     }
    },
    "985174675f354a9bb2580a70ad2b7625": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b9e83e2c1d2f4970a0905efde0055e3d",
       "IPY_MODEL_8c3f85da75764955a572261460db619b",
       "IPY_MODEL_80995fb6ed6f4557b1037dea2798b2fb"
      ],
      "layout": "IPY_MODEL_016181d9f51a4abea666046fb95f2aee"
     }
    },
    "9a01bb7c8f5e4940ba2444fa63a3db6f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b8d8eac2274a4040a4163663896753f7",
      "placeholder": "​",
      "style": "IPY_MODEL_01f82250d2e14f198dd0f561969d54ff",
      "value": " 4.20k/4.20k [00:00&lt;00:00, 379kB/s]"
     }
    },
    "9ccbc146f4ad4382879b5d8b526cac4e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a20bdb762700449097d5010ce7d29921": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a2e056a63fd34329a21330a4fc69429d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_593b4aaa5c9e4e1898cc4d5064940770",
      "placeholder": "​",
      "style": "IPY_MODEL_4df9f85ae9e943ceb3ed1a5d9e143f5c",
      "value": "Downloading builder script: 100%"
     }
    },
    "a3fa870efa0f4463b1e60fa106f782f1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a5076ac1d4ad4c09aa1e6f12449915af": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ae6d2d013c03448887d53f8a0760e77d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b6c20dee590f40e5b2c96beefc1643e1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b6cb26e5df334ac980005863a38581a1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b791095b95214d82af1e344a2b5898a2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b8beefe086284c7297004fcad9303617": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b8d8eac2274a4040a4163663896753f7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b93a51136d1146e08914db125d0259fb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_054e759a27a442afa8c666fc3d4c3f2e",
      "placeholder": "​",
      "style": "IPY_MODEL_e4633e912a57492497bc2fbfad02351c",
      "value": "Map: 100%"
     }
    },
    "b9e83e2c1d2f4970a0905efde0055e3d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5b1fd51b695b4c81ab351ed5b1f92966",
      "placeholder": "​",
      "style": "IPY_MODEL_a5076ac1d4ad4c09aa1e6f12449915af",
      "value": "Map: 100%"
     }
    },
    "bb7f6dabc3ad4779a87b211227040606": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "beb76be168f840d79c950efc4bdfc8d1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f8d378a142934482aa04a608ada9acdb",
      "max": 267954768,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b6c20dee590f40e5b2c96beefc1643e1",
      "value": 267954768
     }
    },
    "c0028b925edf40ecb94ee219a26dfe36": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b6cb26e5df334ac980005863a38581a1",
      "placeholder": "​",
      "style": "IPY_MODEL_ae6d2d013c03448887d53f8a0760e77d",
      "value": " 152322/152322 [00:06&lt;00:00, 24189.50 examples/s]"
     }
    },
    "c3faf065f2d2435697fb730bf04ff1cb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8fc36b3359d24eb987c4be9eaf5d4e95",
       "IPY_MODEL_beb76be168f840d79c950efc4bdfc8d1",
       "IPY_MODEL_63e64e93abca400293b7b7742bd94aaf"
      ],
      "layout": "IPY_MODEL_d41ef095cc5b4c23b31236b501dcf8c1"
     }
    },
    "c4055bbdeb6c4833b4fb50970248a154": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_31cf1f106af54f7986b4d2f462125cca",
       "IPY_MODEL_1c5d45fa6fb14097bf36584ec7ef651c",
       "IPY_MODEL_1aa8fdd087ce42bdb16ae19a38e79b27"
      ],
      "layout": "IPY_MODEL_ef3b72dea05d404db610152cb17ba8f9"
     }
    },
    "c45933bed8084b53b562621699870695": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_37cede645a5b48a9b89df987b9ef91e2",
      "max": 466062,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_13b1494b02194773a5fafe39b211c88a",
      "value": 466062
     }
    },
    "c56c3c1ff2a040b0952b5bac655b71dc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c6cbefbc4f4d4cfe98400aa05370e3e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d0c86013f8194633be0ac27ca9b8876c",
      "max": 483,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7b91cb6b9b5e4ba891a6cfaab2d5948b",
      "value": 483
     }
    },
    "c74948f6d29e4e0e868a3ccf836d2377": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_79144b8d1ec44b6bb3dcc381f7e829ce",
      "max": 30465,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_60e84d7134b646f6b291b0439d9ec890",
      "value": 30465
     }
    },
    "ca74fdefa7844f11b1d9052269efd585": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ca79aac952ca41c096241ad371c055fe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1b61ce44fb554db2bb906fb39b41b7e0",
      "placeholder": "​",
      "style": "IPY_MODEL_dcc6949983954dfb9b7f8bd0ad6cd6d8",
      "value": "Map: 100%"
     }
    },
    "cf51cae23d2b4026a4f890069983da4c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ca79aac952ca41c096241ad371c055fe",
       "IPY_MODEL_0233456210f8469f85e648ba1c1f83ff",
       "IPY_MODEL_c0028b925edf40ecb94ee219a26dfe36"
      ],
      "layout": "IPY_MODEL_d4a7a1f7a10040ccb07704d845de6cd6"
     }
    },
    "d0080133b35c40578eba5db18dc82db6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d0c86013f8194633be0ac27ca9b8876c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d1cc110825034545a2532fc567b5325e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d41ef095cc5b4c23b31236b501dcf8c1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d4a7a1f7a10040ccb07704d845de6cd6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d6b2772488464e80b333332d6cd46eeb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dc17c88c62244e2f9a574ef895132a83": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "dcc6949983954dfb9b7f8bd0ad6cd6d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e2f32fe9433d458dbc278da438f27449": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e4633e912a57492497bc2fbfad02351c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "eb256ba48acd446bbe6b832c9558f7c6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ec74087b5c9d4e538675956af72ae138": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_fe87ce116db84d5ba3d00e4a8c249dc1",
       "IPY_MODEL_c45933bed8084b53b562621699870695",
       "IPY_MODEL_72df691ec7784ae7b0542171a7789555"
      ],
      "layout": "IPY_MODEL_4c92f17cff2944a9a2a49be56649d81e"
     }
    },
    "ef3b72dea05d404db610152cb17ba8f9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f1fa4e09b42c4904b9be033d33943e05": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_470f2248b2444869914197c7481e10ae",
       "IPY_MODEL_c6cbefbc4f4d4cfe98400aa05370e3e7",
       "IPY_MODEL_6a7b459551104d47a90527c885647830"
      ],
      "layout": "IPY_MODEL_d0080133b35c40578eba5db18dc82db6"
     }
    },
    "f8d378a142934482aa04a608ada9acdb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fda99413adc8497cba92a9c6416f77ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "fe87ce116db84d5ba3d00e4a8c249dc1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_20a883d10ea945a9889429c5efb76a43",
      "placeholder": "​",
      "style": "IPY_MODEL_2f44328dc8a04f84a59e26d08d343c77",
      "value": "tokenizer.json: 100%"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}