assets/hub/pytorch_fairseq_translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook is optionally accelerated with a GPU runtime.\n",
    "### If you would like to use this acceleration, please select the menu option \"Runtime\" -> \"Change runtime type\", select \"Hardware Accelerator\" -> \"GPU\" and click \"SAVE\"\n",
    "\n",
    "----------------------------------------------------------------------\n",
    "\n",
    "# Transformer (NMT)\n",
    "\n",
    "*Author: Facebook AI (fairseq Team)*\n",
    "\n",
    "**Transformer models for English-French and English-German translation.**\n",
    "\n",
    "\n",
    "\n",
    "### Model Description\n",
    "\n",
    "The Transformer, introduced in the paper [Attention Is All You Need][1], is a\n",
    "powerful sequence-to-sequence modeling architecture capable of producing\n",
    "state-of-the-art neural machine translation (NMT) systems.\n",
    "\n",
    "Recently, the fairseq team has explored large-scale semi-supervised training of\n",
    "Transformers using back-translated data, further improving translation quality\n",
    "over the original model. More details can be found in [this blog post][2].\n",
    "\n",
    "\n",
    "### Requirements\n",
    "\n",
    "We require a few additional Python dependencies for preprocessing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install fastBPE regex requests sacremoses subword_nmt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### English-to-French Translation\n",
    "\n",
    "To translate from English to French using the model from the paper [Scaling\n",
    "Neural Machine Translation][3]:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# Load an En-Fr Transformer model trained on WMT'14 data :\n",
    "en2fr = torch.hub.load('pytorch/fairseq', 'transformer.wmt14.en-fr', tokenizer='moses', bpe='subword_nmt')\n",
    "\n",
    "# Use the GPU (optional):\n",
    "en2fr.cuda()\n",
    "\n",
    "# Translate with beam search:\n",
    "fr = en2fr.translate('Hello world!', beam=5)\n",
    "assert fr == 'Bonjour à tous !'\n",
    "\n",
    "# Manually tokenize:\n",
    "en_toks = en2fr.tokenize('Hello world!')\n",
    "assert en_toks == 'Hello world !'\n",
    "\n",
    "# Manually apply BPE:\n",
    "en_bpe = en2fr.apply_bpe(en_toks)\n",
    "assert en_bpe == 'H@@ ello world !'\n",
    "\n",
    "# Manually binarize:\n",
    "en_bin = en2fr.binarize(en_bpe)\n",
    "assert en_bin.tolist() == [329, 14044, 682, 812, 2]\n",
    "\n",
    "# Generate five translations with top-k sampling:\n",
    "fr_bin = en2fr.generate(en_bin, beam=5, sampling=True, sampling_topk=20)\n",
    "assert len(fr_bin) == 5\n",
    "\n",
    "# Convert one of the samples to a string and detokenize\n",
    "fr_sample = fr_bin[0]['tokens']\n",
    "fr_bpe = en2fr.string(fr_sample)\n",
    "fr_toks = en2fr.remove_bpe(fr_bpe)\n",
    "fr = en2fr.detokenize(fr_toks)\n",
    "assert fr == en2fr.decode(fr_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### English-to-German Translation\n",
    "\n",
    "Semi-supervised training with back-translation is an effective way of improving\n",
    "translation systems. In the paper [Understanding Back-Translation at Scale][4],\n",
    "we back-translate over 200 million German sentences to use as additional\n",
    "training data. An ensemble of five of these models was the winning submission to\n",
    "the [WMT'18 English-German news translation competition][5].\n",
    "\n",
    "We can further improved this approach through [noisy-channel reranking][6]. More\n",
    "details can be found in [this blog post][7]. An ensemble of models trained with\n",
    "this technique was the winning submission to the [WMT'19 English-German news\n",
    "translation competition][8].\n",
    "\n",
    "To translate from English to German using one of the models from the winning submission:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# Load an En-De Transformer model trained on WMT'19 data:\n",
    "en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de.single_model', tokenizer='moses', bpe='fastbpe')\n",
    "\n",
    "# Access the underlying TransformerModel\n",
    "assert isinstance(en2de.models[0], torch.nn.Module)\n",
    "\n",
    "# Translate from En-De\n",
    "de = en2de.translate('PyTorch Hub is a pre-trained model repository designed to facilitate research reproducibility.')\n",
    "assert de == 'PyTorch Hub ist ein vorgefertigtes Modell-Repository, das die Reproduzierbarkeit der Forschung erleichtern soll.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do a round-trip translation to create a paraphrase:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Round-trip translations between English and German:\n",
    "en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de.single_model', tokenizer='moses', bpe='fastbpe')\n",
    "de2en = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.de-en.single_model', tokenizer='moses', bpe='fastbpe')\n",
    "\n",
    "paraphrase = de2en.translate(en2de.translate('PyTorch Hub is an awesome interface!'))\n",
    "assert paraphrase == 'PyTorch Hub is a fantastic interface!'\n",
    "\n",
    "# Compare the results with English-Russian round-trip translation:\n",
    "en2ru = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-ru.single_model', tokenizer='moses', bpe='fastbpe')\n",
    "ru2en = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.ru-en.single_model', tokenizer='moses', bpe='fastbpe')\n",
    "\n",
    "paraphrase = ru2en.translate(en2ru.translate('PyTorch Hub is an awesome interface!'))\n",
    "assert paraphrase == 'PyTorch is a great interface!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "- [Attention Is All You Need][1]\n",
    "- [Scaling Neural Machine Translation][3]\n",
    "- [Understanding Back-Translation at Scale][4]\n",
    "- [Facebook FAIR's WMT19 News Translation Task Submission][6]\n",
    "\n",
    "\n",
    "[1]: https://arxiv.org/abs/1706.03762\n",
    "[2]: https://code.fb.com/ai-research/scaling-neural-machine-translation-to-bigger-data-sets-with-faster-training-and-inference/\n",
    "[3]: https://arxiv.org/abs/1806.00187\n",
    "[4]: https://arxiv.org/abs/1808.09381\n",
    "[5]: http://www.statmt.org/wmt18/translation-task.html\n",
    "[6]: https://arxiv.org/abs/1907.06616\n",
    "[7]: https://ai.facebook.com/blog/facebook-leads-wmt-translation-competition/\n",
    "[8]: http://www.statmt.org/wmt19/translation-task.html"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}