seq2seq/sequence_to_sequence_implementation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Character Sequence to Sequence \n",
    "In this notebook, we'll build a model that takes in a sequence of letters, and outputs a sorted version of that sequence. We'll do that using what we've learned so far about Sequence to Sequence models. This notebook was updated to work with TensorFlow 1.1 and builds on the work of Dave Currie. Check out Dave's post [Text Summarization with Amazon Reviews](https://medium.com/towards-data-science/text-summarization-with-amazon-reviews-41801c2210b).\n",
    "\n",
    "<img src=\"images/sequence-to-sequence.jpg\"/>\n",
    "\n",
    "\n",
    "## Dataset \n",
    "\n",
    "The dataset lives in the /data/ folder. At the moment, it is made up of the following files:\n",
    " * **letters_source.txt**: The list of input letter sequences. Each sequence is its own line. \n",
    " * **letters_target.txt**: The list of target sequences we'll use in the training process. Each sequence here is a response to the input sequence in letters_source.txt with the same line number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import time\n",
    "\n",
    "import helper\n",
    "\n",
    "source_path = 'data/letters_source.txt'\n",
    "target_path = 'data/letters_target.txt'\n",
    "\n",
    "source_sentences = helper.load_data(source_path)\n",
    "target_sentences = helper.load_data(target_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by examining the current state of the dataset. `source_sentences` contains the entire input sequence file as text delimited by newline symbols."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_sentences[:50].split('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`target_sentences` contains the entire output sequence file as text delimited by newline symbols.  Each line corresponds to the line from `source_sentences`.  `target_sentences` contains a sorted characters of the line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_sentences[:50].split('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess\n",
    "To do anything useful with it, we'll need to turn the each string into a list of characters: \n",
    "\n",
    "<img src=\"images/source_and_target_arrays.png\"/>\n",
    "\n",
    "Then convert the characters to their int values as declared in our vocabulary:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_character_vocab(data):\n",
    "    special_words = ['<PAD>', '<UNK>', '<GO>',  '<EOS>']\n",
    "\n",
    "    set_words = set([character for line in data.split('\\n') for character in line])\n",
    "    int_to_vocab = {word_i: word for word_i, word in enumerate(special_words + list(set_words))}\n",
    "    vocab_to_int = {word: word_i for word_i, word in int_to_vocab.items()}\n",
    "\n",
    "    return int_to_vocab, vocab_to_int\n",
    "\n",
    "# Build int2letter and letter2int dicts\n",
    "source_int_to_letter, source_letter_to_int = extract_character_vocab(source_sentences)\n",
    "target_int_to_letter, target_letter_to_int = extract_character_vocab(target_sentences)\n",
    "\n",
    "# Convert characters to ids\n",
    "source_letter_ids = [[source_letter_to_int.get(letter, source_letter_to_int['<UNK>']) for letter in line] for line in source_sentences.split('\\n')]\n",
    "target_letter_ids = [[target_letter_to_int.get(letter, target_letter_to_int['<UNK>']) for letter in line] + [target_letter_to_int['<EOS>']] for line in target_sentences.split('\\n')] \n",
    "\n",
    "print(\"Example source sequence\")\n",
    "print(source_letter_ids[:3])\n",
    "print(\"\\n\")\n",
    "print(\"Example target sequence\")\n",
    "print(target_letter_ids[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "This is the final shape we need them to be in. We can now proceed to building the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "#### Check the Version of TensorFlow\n",
    "This will check to make sure you have the correct version of TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from distutils.version import LooseVersion\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.layers.core import Dense\n",
    "\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "epochs = 60\n",
    "# Batch Size\n",
    "batch_size = 128\n",
    "# RNN Size\n",
    "rnn_size = 50\n",
    "# Number of Layers\n",
    "num_layers = 2\n",
    "# Embedding Size\n",
    "encoding_embedding_size = 15\n",
    "decoding_embedding_size = 15\n",
    "# Learning Rate\n",
    "learning_rate = 0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_inputs():\n",
    "    input_data = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    lr = tf.placeholder(tf.float32, name='learning_rate')\n",
    "\n",
    "    target_sequence_length = tf.placeholder(tf.int32, (None,), name='target_sequence_length')\n",
    "    max_target_sequence_length = tf.reduce_max(target_sequence_length, name='max_target_len')\n",
    "    source_sequence_length = tf.placeholder(tf.int32, (None,), name='source_sequence_length')\n",
    "    \n",
    "    return input_data, targets, lr, target_sequence_length, max_target_sequence_length, source_sequence_length\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequence to Sequence Model\n",
    "\n",
    "We can now start defining the functions that will build the seq2seq model. We are building it from the bottom up with the following components:\n",
    "\n",
    "    2.1 Encoder\n",
    "        - Embedding\n",
    "        - Encoder cell\n",
    "    2.2 Decoder\n",
    "        1- Process decoder inputs\n",
    "        2- Set up the decoder\n",
    "            - Embedding\n",
    "            - Decoder cell\n",
    "            - Dense output layer\n",
    "            - Training decoder\n",
    "            - Inference decoder\n",
    "    2.3 Seq2seq model connecting the encoder and decoder\n",
    "    2.4 Build the training graph hooking up the model with the \n",
    "        optimizer\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Encoder\n",
    "\n",
    "The first bit of the model we'll build is the encoder. Here, we'll embed the input data, construct our encoder, then pass the embedded data to the encoder.\n",
    "\n",
    "- Embed the input data using [`tf.contrib.layers.embed_sequence`](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence)\n",
    "<img src=\"images/embed_sequence.png\" />\n",
    "\n",
    "- Pass the embedded input into a stack of RNNs.  Save the RNN state and ignore the output.\n",
    "<img src=\"images/encoder.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def encoding_layer(input_data, rnn_size, num_layers,\n",
    "                   source_sequence_length, source_vocab_size, \n",
    "                   encoding_embedding_size):\n",
    "\n",
    "\n",
    "    # Encoder embedding\n",
    "    enc_embed_input = tf.contrib.layers.embed_sequence(input_data, source_vocab_size, encoding_embedding_size)\n",
    "\n",
    "    # RNN cell\n",
    "    def make_cell(rnn_size):\n",
    "        enc_cell = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                           initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "        return enc_cell\n",
    "\n",
    "    enc_cell = tf.contrib.rnn.MultiRNNCell([make_cell(rnn_size) for _ in range(num_layers)])\n",
    "    \n",
    "    enc_output, enc_state = tf.nn.dynamic_rnn(enc_cell, enc_embed_input, sequence_length=source_sequence_length, dtype=tf.float32)\n",
    "    \n",
    "    return enc_output, enc_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Decoder\n",
    "\n",
    "The decoder is probably the most involved part of this model. The following steps are needed to create it:\n",
    "\n",
    "    1- Process decoder inputs\n",
    "    2- Set up the decoder components\n",
    "        - Embedding\n",
    "        - Decoder cell\n",
    "        - Dense output layer\n",
    "        - Training decoder\n",
    "        - Inference decoder\n",
    "\n",
    "\n",
    "### Process Decoder Input\n",
    "\n",
    "\n",
    "In the training process, the target sequences will be used in two different places:\n",
    "\n",
    " 1. Using them to calculate the loss\n",
    " 2. Feeding them to the decoder during training to make the model more robust.\n",
    "\n",
    "Now we need to address the second point. Let's assume our targets look like this in their letter/word form (we're doing this for readibility. At this point in the code, these sequences would be in int form):\n",
    "\n",
    "\n",
    "<img src=\"images/targets_1.png\"/>\n",
    "\n",
    "We need to do a simple transformation on the tensor before feeding it to the decoder:\n",
    "\n",
    "1- We will feed an item of the sequence to the decoder at each time step. Think about the last timestep -- where the decoder outputs the final word in its output. The input to that step is the item before last from the target sequence. The decoder has no use for the last item in the target sequence in this scenario. So we'll need to remove the last item. \n",
    "\n",
    "We do that using tensorflow's tf.strided_slice() method. We hand it the tensor, and the index of where to start and where to end the cutting.\n",
    "\n",
    "<img src=\"images/strided_slice_1.png\"/>\n",
    "\n",
    "2- The first item in each sequence we feed to the decoder has to be GO symbol. So We'll add that to the beginning.\n",
    "\n",
    "\n",
    "<img src=\"images/targets_add_go.png\"/>\n",
    "\n",
    "\n",
    "Now the tensor is ready to be fed to the decoder. It looks like this (if we convert from ints to letters/symbols):\n",
    "\n",
    "<img src=\"images/targets_after_processing_1.png\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Process the input we'll feed to the decoder\n",
    "def process_decoder_input(target_data, vocab_to_int, batch_size):\n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Set up the decoder components\n",
    "\n",
    "        - Embedding\n",
    "        - Decoder cell\n",
    "        - Dense output layer\n",
    "        - Training decoder\n",
    "        - Inference decoder\n",
    "\n",
    "#### 1- Embedding\n",
    "Now that we have prepared the inputs to the training decoder, we need to embed them so they can be ready to be passed to the decoder. \n",
    "\n",
    "We'll create an embedding matrix like the following then have tf.nn.embedding_lookup convert our input to its embedded equivalent:\n",
    "<img src=\"images/embeddings.png\" />\n",
    "\n",
    "#### 2- Decoder Cell\n",
    "Then we declare our decoder cell. Just like the encoder, we'll use an tf.contrib.rnn.LSTMCell here as well.\n",
    "\n",
    "We need to declare a decoder for the training process, and a decoder for the inference/prediction process. These two decoders will share their parameters (so that all the weights and biases that are set during the training phase can be used when we deploy the model).\n",
    "\n",
    "First, we'll need to define the type of cell we'll be using for our decoder RNNs. We opted for LSTM.\n",
    "\n",
    "#### 3- Dense output layer\n",
    "Before we move to declaring our decoders, we'll need to create the output layer, which will be a tensorflow.python.layers.core.Dense layer that translates the outputs of the decoder to logits that tell us which element of the decoder vocabulary the decoder is choosing to output at each time step.\n",
    "\n",
    "#### 4- Training decoder\n",
    "Essentially, we'll be creating two decoders which share their parameters. One for training and one for inference. The two are similar in that both created using tf.contrib.seq2seq.**BasicDecoder** and tf.contrib.seq2seq.**dynamic_decode**. They differ, however, in that we feed the the target sequences as inputs to the training decoder at each time step to make it more robust.\n",
    "\n",
    "We can think of the training decoder as looking like this (except that it works with sequences in batches):\n",
    "<img src=\"images/sequence-to-sequence-training-decoder.png\"/>\n",
    "\n",
    "The training decoder **does not** feed the output of each time step to the next. Rather, the inputs to the decoder time steps are the target sequence from the training dataset (the orange letters).\n",
    "\n",
    "#### 5- Inference decoder\n",
    "The inference decoder is the one we'll use when we deploy our model to the wild.\n",
    "\n",
    "<img src=\"images/sequence-to-sequence-inference-decoder.png\"/>\n",
    "\n",
    "We'll hand our encoder hidden state to both the training and inference decoders and have it process its output. TensorFlow handles most of the logic for us. We just have to use the appropriate methods from tf.contrib.seq2seq and supply them with the appropriate inputs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decoding_layer(target_letter_to_int, decoding_embedding_size, num_layers, rnn_size,\n",
    "                   target_sequence_length, max_target_sequence_length, enc_state, dec_input):\n",
    "    # 1. Decoder Embedding\n",
    "    target_vocab_size = len(target_letter_to_int)\n",
    "    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))\n",
    "    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)\n",
    "\n",
    "    # 2. Construct the decoder cell\n",
    "    def make_cell(rnn_size):\n",
    "        dec_cell = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                           initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "        return dec_cell\n",
    "\n",
    "    dec_cell = tf.contrib.rnn.MultiRNNCell([make_cell(rnn_size) for _ in range(num_layers)])\n",
    "     \n",
    "    # 3. Dense layer to translate the decoder's output at each time \n",
    "    # step into a choice from the target vocabulary\n",
    "    output_layer = Dense(target_vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "\n",
    "\n",
    "    # 4. Set up a training decoder and an inference decoder\n",
    "    # Training Decoder\n",
    "    with tf.variable_scope(\"decode\"):\n",
    "\n",
    "        # Helper for the training process. Used by BasicDecoder to read inputs.\n",
    "        training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                            sequence_length=target_sequence_length,\n",
    "                                                            time_major=False)\n",
    "        \n",
    "        \n",
    "        # Basic decoder\n",
    "        training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                           training_helper,\n",
    "                                                           enc_state,\n",
    "                                                           output_layer) \n",
    "        \n",
    "        # Perform dynamic decoding using the decoder\n",
    "        training_decoder_output = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                                       impute_finished=True,\n",
    "                                                                       maximum_iterations=max_target_sequence_length)[0]\n",
    "    # 5. Inference Decoder\n",
    "    # Reuses the same parameters trained by the training process\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        start_tokens = tf.tile(tf.constant([target_letter_to_int['<GO>']], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "\n",
    "        # Helper for the inference process.\n",
    "        inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(dec_embeddings,\n",
    "                                                                start_tokens,\n",
    "                                                                target_letter_to_int['<EOS>'])\n",
    "\n",
    "        # Basic decoder\n",
    "        inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        enc_state,\n",
    "                                                        output_layer)\n",
    "        \n",
    "        # Perform dynamic decoding using the decoder\n",
    "        inference_decoder_output = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                            impute_finished=True,\n",
    "                                                            maximum_iterations=max_target_sequence_length)[0]\n",
    "         \n",
    "\n",
    "    \n",
    "    return training_decoder_output, inference_decoder_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Seq2seq model \n",
    "Let's now go a step above, and hook up the encoder and decoder using the methods we just declared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def seq2seq_model(input_data, targets, lr, target_sequence_length, \n",
    "                  max_target_sequence_length, source_sequence_length,\n",
    "                  source_vocab_size, target_vocab_size,\n",
    "                  enc_embedding_size, dec_embedding_size, \n",
    "                  rnn_size, num_layers):\n",
    "    \n",
    "    # Pass the input data through the encoder. We'll ignore the encoder output, but use the state\n",
    "    _, enc_state = encoding_layer(input_data, \n",
    "                                  rnn_size, \n",
    "                                  num_layers, \n",
    "                                  source_sequence_length,\n",
    "                                  source_vocab_size, \n",
    "                                  encoding_embedding_size)\n",
    "    \n",
    "    \n",
    "    # Prepare the target sequences we'll feed to the decoder in training mode\n",
    "    dec_input = process_decoder_input(targets, target_letter_to_int, batch_size)\n",
    "    \n",
    "    # Pass encoder state and decoder inputs to the decoders\n",
    "    training_decoder_output, inference_decoder_output = decoding_layer(target_letter_to_int, \n",
    "                                                                       decoding_embedding_size, \n",
    "                                                                       num_layers, \n",
    "                                                                       rnn_size,\n",
    "                                                                       target_sequence_length,\n",
    "                                                                       max_target_sequence_length,\n",
    "                                                                       enc_state, \n",
    "                                                                       dec_input) \n",
    "    \n",
    "    return training_decoder_output, inference_decoder_output\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model outputs *training_decoder_output* and *inference_decoder_output* both contain a 'rnn_output' logits tensor that looks like this:\n",
    "\n",
    "<img src=\"images/logits.png\"/>\n",
    "\n",
    "The logits we get from the training tensor we'll pass to tf.contrib.seq2seq.**sequence_loss()** to calculate the loss and ultimately the gradient.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the graph\n",
    "train_graph = tf.Graph()\n",
    "# Set the graph to default to ensure that it is ready for training\n",
    "with train_graph.as_default():\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    input_data, targets, lr, target_sequence_length, max_target_sequence_length, source_sequence_length = get_model_inputs()\n",
    "    \n",
    "    # Create the training and inference logits\n",
    "    training_decoder_output, inference_decoder_output = seq2seq_model(input_data, \n",
    "                                                                      targets, \n",
    "                                                                      lr, \n",
    "                                                                      target_sequence_length, \n",
    "                                                                      max_target_sequence_length, \n",
    "                                                                      source_sequence_length,\n",
    "                                                                      len(source_letter_to_int),\n",
    "                                                                      len(target_letter_to_int),\n",
    "                                                                      encoding_embedding_size, \n",
    "                                                                      decoding_embedding_size, \n",
    "                                                                      rnn_size, \n",
    "                                                                      num_layers)    \n",
    "    \n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_decoder_output.rnn_output, 'logits')\n",
    "    inference_logits = tf.identity(inference_decoder_output.sample_id, name='predictions')\n",
    "    \n",
    "    # Create the weights for sequence_loss\n",
    "    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        \n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(lr)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Batches\n",
    "\n",
    "There's little processing involved when we retreive the batches. This is a simple example assuming batch_size = 2\n",
    "\n",
    "Source sequences (it's actually in int form, we're showing the characters for clarity):\n",
    "\n",
    "<img src=\"images/source_batch.png\" />\n",
    "\n",
    "Target sequences (also in int, but showing letters for clarity):\n",
    "\n",
    "<img src=\"images/target_batch.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch, pad_int):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_batches(targets, sources, batch_size, source_pad_int, target_pad_int):\n",
    "    \"\"\"Batch targets, sources, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(sources)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        sources_batch = sources[start_i:start_i + batch_size]\n",
    "        targets_batch = targets[start_i:start_i + batch_size]\n",
    "        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))\n",
    "        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_targets_lengths = []\n",
    "        for target in pad_targets_batch:\n",
    "            pad_targets_lengths.append(len(target))\n",
    "        \n",
    "        pad_source_lengths = []\n",
    "        for source in pad_sources_batch:\n",
    "            pad_source_lengths.append(len(source))\n",
    "        \n",
    "        yield pad_targets_batch, pad_sources_batch, pad_targets_lengths, pad_source_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "We're now ready to train our model. If you run into OOM (out of memory) issues during training, try to decrease the batch_size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Split data to training and validation sets\n",
    "train_source = source_letter_ids[batch_size:]\n",
    "train_target = target_letter_ids[batch_size:]\n",
    "valid_source = source_letter_ids[:batch_size]\n",
    "valid_target = target_letter_ids[:batch_size]\n",
    "(valid_targets_batch, valid_sources_batch, valid_targets_lengths, valid_sources_lengths) = next(get_batches(valid_target, valid_source, batch_size,\n",
    "                           source_letter_to_int['<PAD>'],\n",
    "                           target_letter_to_int['<PAD>']))\n",
    "\n",
    "display_step = 20 # Check training loss after every 20 batches\n",
    "\n",
    "checkpoint = \"best_model.ckpt\" \n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "        \n",
    "    for epoch_i in range(1, epochs+1):\n",
    "        for batch_i, (targets_batch, sources_batch, targets_lengths, sources_lengths) in enumerate(\n",
    "                get_batches(train_target, train_source, batch_size,\n",
    "                           source_letter_to_int['<PAD>'],\n",
    "                           target_letter_to_int['<PAD>'])):\n",
    "            \n",
    "            # Training step\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: sources_batch,\n",
    "                 targets: targets_batch,\n",
    "                 lr: learning_rate,\n",
    "                 target_sequence_length: targets_lengths,\n",
    "                 source_sequence_length: sources_lengths})\n",
    "\n",
    "            # Debug message updating us on the status of the training\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                \n",
    "                # Calculate validation cost\n",
    "                validation_loss = sess.run(\n",
    "                [cost],\n",
    "                {input_data: valid_sources_batch,\n",
    "                 targets: valid_targets_batch,\n",
    "                 lr: learning_rate,\n",
    "                 target_sequence_length: valid_targets_lengths,\n",
    "                 source_sequence_length: valid_sources_lengths})\n",
    "                \n",
    "                print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}  - Validation loss: {:>6.3f}'\n",
    "                      .format(epoch_i,\n",
    "                              epochs, \n",
    "                              batch_i, \n",
    "                              len(train_source) // batch_size, \n",
    "                              loss, \n",
    "                              validation_loss[0]))\n",
    "\n",
    "    \n",
    "    \n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, checkpoint)\n",
    "    print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def source_to_seq(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    sequence_length = 7\n",
    "    return [source_letter_to_int.get(word, source_letter_to_int['<UNK>']) for word in text]+ [source_letter_to_int['<PAD>']]*(sequence_length-len(text))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "input_sentence = 'hello'\n",
    "text = source_to_seq(input_sentence)\n",
    "\n",
    "checkpoint = \"./best_model.ckpt\"\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')\n",
    "    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')\n",
    "    \n",
    "    #Multiply by batch_size to match the model's input parameters\n",
    "    answer_logits = sess.run(logits, {input_data: [text]*batch_size, \n",
    "                                      target_sequence_length: [len(text)]*batch_size, \n",
    "                                      source_sequence_length: [len(text)]*batch_size})[0] \n",
    "\n",
    "\n",
    "pad = source_letter_to_int[\"<PAD>\"] \n",
    "\n",
    "print('Original Text:', input_sentence)\n",
    "\n",
    "print('\\nSource')\n",
    "print('  Word Ids:    {}'.format([i for i in text]))\n",
    "print('  Input Words: {}'.format(\" \".join([source_int_to_letter[i] for i in text])))\n",
    "\n",
    "print('\\nTarget')\n",
    "print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))\n",
    "print('  Response Words: {}'.format(\" \".join([target_int_to_letter[i] for i in answer_logits if i != pad])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}