assets/hub/nvidia_deeplearningexamples_waveglow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook requires a GPU runtime to run.\n",
    "### Please select the menu option \"Runtime\" -> \"Change runtime type\", select \"Hardware Accelerator\" -> \"GPU\" and click \"SAVE\"\n",
    "\n",
    "----------------------------------------------------------------------\n",
    "\n",
    "# WaveGlow\n",
    "\n",
    "*Author: NVIDIA*\n",
    "\n",
    "**WaveGlow model for generating speech from mel spectrograms (generated by Tacotron2)**\n",
    "\n",
    "<img src=\"https://pytorch.org/assets/images/waveglow_diagram.png\" alt=\"alt\" width=\"50%\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "waveglow = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_waveglow')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "will load the WaveGlow model pre-trained on [LJ Speech dataset](https://keithito.com/LJ-Speech-Dataset/)\n",
    "\n",
    "### Model Description\n",
    "\n",
    "The Tacotron 2 and WaveGlow model form a text-to-speech system that enables user to synthesise a natural sounding speech from raw transcripts without any additional prosody information. The Tacotron 2 model (also available via torch.hub) produces mel spectrograms from input text using encoder-decoder architecture. WaveGlow is a flow-based model that consumes the mel spectrograms to generate speech.\n",
    "\n",
    "### Example\n",
    "\n",
    "In the example below:\n",
    "- pretrained Tacotron2 and Waveglow models are loaded from torch.hub\n",
    "- Tacotron2 generates mel spectrogram given tensor represantation of an input text (\"Hello world, I missed you\")\n",
    "- Waveglow generates sound given the mel spectrogram\n",
    "- the output sound is saved in an 'audio.wav' file\n",
    "\n",
    "To run the example you need some extra python packages installed.\n",
    "These are needed for preprocessing the text and audio, as well as for display and input / output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install numpy scipy librosa unidecode inflect librosa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.io.wavfile import write"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the waveglow model for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "waveglow = waveglow.remove_weightnorm(waveglow)\n",
    "waveglow = waveglow.to('cuda')\n",
    "waveglow.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load tacotron2 from PyTorch Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')\n",
    "tacotron2 = tacotron2.to('cuda')\n",
    "tacotron2.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's make the model say *\"hello world, I missed you\"*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"hello world, I missed you\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now chain pre-processing -> tacotron2 -> waveglow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocessing\n",
    "sequence = np.array(tacotron2.text_to_sequence(text, ['english_cleaners']))[None, :]\n",
    "sequence = torch.from_numpy(sequence).to(device='cuda', dtype=torch.int64)\n",
    "\n",
    "# run the models\n",
    "with torch.no_grad():\n",
    "    _, mel, _, _ = tacotron2.infer(sequence)\n",
    "    audio = waveglow.infer(mel)\n",
    "audio_numpy = audio[0].data.cpu().numpy()\n",
    "rate = 22050"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can write it to a file and listen to it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "write(\"audio.wav\", rate, audio_numpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, play it right away in a notebook with IPython widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Audio\n",
    "Audio(audio_numpy, rate=rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Details\n",
    "For detailed information on model input and output, training recipies, inference and performance visit: [github](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2) and/or [NGC](https://ngc.nvidia.com/catalog/model-scripts/nvidia:tacotron_2_and_waveglow_for_pytorch)\n",
    "\n",
    "### References\n",
    "\n",
    " - [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://arxiv.org/abs/1712.05884)\n",
    " - [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://arxiv.org/abs/1811.00002)\n",
    " - [Tacotron2 and WaveGlow on NGC](https://ngc.nvidia.com/catalog/model-scripts/nvidia:tacotron_2_and_waveglow_for_pytorch)\n",
    " - [Tacotron2 and Waveglow on github](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2)"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}