assets/hub/huggingface_pytorch-pretrained-bert_gpt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook is optionally accelerated with a GPU runtime.\n",
    "### If you would like to use this acceleration, please select the menu option \"Runtime\" -> \"Change runtime type\", select \"Hardware Accelerator\" -> \"GPU\" and click \"SAVE\"\n",
    "\n",
    "----------------------------------------------------------------------\n",
    "\n",
    "# GPT\n",
    "\n",
    "*Author: HuggingFace Team*\n",
    "\n",
    "**Generative Pre-Training (GPT) models for language understanding**\n",
    "\n",
    "<img src=\"https://pytorch.org/assets/images/GPT1.png\" alt=\"alt\" width=\"50%\"/>\n",
    "\n",
    "\n",
    "### Model Description\n",
    "\n",
    "GPT was released together with the paper [Improving Language Understanding by Generative Pre-Training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) by Alec Radford et al at OpenAI. It's a combination of two ideas: Transformer model and large scale unsupervised pre-training.\n",
    "\n",
    "Here are three models based on [OpenAI's pre-trained weights](https://github.com/openai/finetune-transformer-lm) along with the associated Tokenizer.\n",
    "It includes:\n",
    "- `openAIGPTModel`: raw OpenAI GPT Transformer model (fully pre-trained)\n",
    "- `openAIGPTLMHeadModel`: OpenAI GPT Transformer with the tied language modeling head on top (fully pre-trained)\n",
    "- `openAIGPTDoubleHeadsModel`: OpenAI GPT Transformer with the tied language modeling head and a multiple choice classification head on top (OpenAI GPT Transformer is pre-trained, the multiple choice classification head is only initialized and has to be trained)\n",
    "\n",
    "### Requirements\n",
    "\n",
    "Unlike most other PyTorch Hub models, BERT requires a few additional Python packages to be installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install tqdm boto3 requests regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example\n",
    "\n",
    "Here is an example on how to tokenize the text with `openAIGPTTokenizer`, and then get the hidden states computed by `openAIGPTModel` or predict the next token using `openAIGPTLMHeadModel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### First, tokenize the input\n",
    "import torch\n",
    "tokenizer = torch.hub.load('huggingface/pytorch-pretrained-BERT', 'openAIGPTTokenizer', 'openai-gpt')\n",
    "\n",
    "#  Prepare tokenized input\n",
    "text = \"Who was Jim Henson ? Jim Henson was a puppeteer\"\n",
    "tokenized_text = tokenizer.tokenize(text)\n",
    "indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)\n",
    "tokens_tensor = torch.tensor([indexed_tokens])\n",
    "\n",
    "### Get the hidden states computed by `openAIGPTModel`\n",
    "model = torch.hub.load('huggingface/pytorch-pretrained-BERT', 'openAIGPTModel', 'openai-gpt')\n",
    "model.eval()\n",
    "\n",
    "# Compute hidden states features for each layer\n",
    "with torch.no_grad():\n",
    "\thidden_states = model(tokens_tensor)\n",
    "\n",
    "### Predict the next token using `openAIGPTLMHeadModel`\n",
    "lm_model = torch.hub.load('huggingface/pytorch-pretrained-BERT', 'openAIGPTLMHeadModel', 'openai-gpt')\n",
    "lm_model.eval()\n",
    "\n",
    "# Predict all tokens\n",
    "with torch.no_grad():\n",
    "\tpredictions = lm_model(tokens_tensor)\n",
    "\n",
    "# Get the last predicted token\n",
    "predicted_index = torch.argmax(predictions[0, -1, :]).item()\n",
    "predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]\n",
    "assert predicted_token == '.</w>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirement\n",
    "The model only support python3.\n",
    "\n",
    "### Resources\n",
    "\n",
    " - Paper: [Improving Language Understanding by Generative Pre-Training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)\n",
    " - [Blogpost from OpenAI](https://openai.com/blog/language-unsupervised/)\n",
    " - Initial repository (with detailed examples and documentation): [pytorch-pretrained-BERT](https://github.com/huggingface/pytorch-pretrained-BERT)"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}