tensorflow/tf-lstm_gru.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. lstm GRU:           http://blog.csdn.net/guotong1988/article/details/75045933\n",
    "2. jupyter notebook:    http://blog.csdn.net/winnerineast/article/details/52274556"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 一，  lstm\n",
    "1. 忘记门\n",
    "2. 新增门\n",
    "3. 更新\n",
    "4. 输出\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 二， GRU\n",
    " \n",
    "\n",
    "r(t) 负责决定h(t−1) 对new memory h^(t) 的重要性有多大， 如果r(t) 约等于0 的话，h(t−1) 就不会传递给new memory h^(t)\n",
    "\n",
    "new memory\n",
    "\n",
    "h^(t) 是对新的输入x(t) 和上一时刻的hidden state h(t−1) 的总结。计算总结出的新的向量h^(t) 包含上文信息和新的输入x(t).\n",
    "\n",
    "Update gate\n",
    "\n",
    "z(t) 负责决定传递多少ht−1给ht 。 如果z(t) 约等于1的话，ht−1 几乎会直接复制给ht ，相反，如果z(t) 约等于0， new memory h^(t) 直接传递给ht.\n",
    "\n",
    "Hidden state:\n",
    "\n",
    "h(t) 由 h(t−1) 和h^(t) 相加得到，两者的权重由update gate z(t) 控制。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "$$ z_t = \\sigma(W_z \\text{x}_t + U_z \\text{h}_{t-1})  $$\n",
    "$$  r_t = \\sigma(W_t \\text{x}_t + U_t \\text{h}_{t-1}) $$\n",
    "$$ \\widetilde{h}_t = \\text{tanh}(W\\text{x}_t + U(r_t \\circ \\text{h}_{t-1})) $$\n",
    "$$ \\text{h}_t = (1 - z_t) \\circ \\text{h}_{t-1} + z_t \\circ \\widetilde{h}_t $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.06551912  0.17653495]]\n",
      "LSTMStateTuple(c=array([[-0.18371592,  0.29048127]], dtype=float32), h=array([[-0.06551912,  0.17653495]], dtype=float32))\n",
      "[[-0.08273311 -0.28885445]]\n",
      "[[-0.08273311 -0.28885445]]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.contrib import rnn\n",
    "\n",
    "x = tf.constant([[1]], dtype = tf.float32)\n",
    "\n",
    "lstm_cell = rnn.BasicLSTMCell(2)\n",
    "gru_cell = rnn.GRUCell(2)\n",
    "\n",
    "state0_lstm = lstm_cell.zero_state(1,dtype=tf.float32)\n",
    "output,state = lstm_cell(x,state0_lstm)\n",
    "\n",
    "state0_gru = gru_cell.zero_state(1,dtype=tf.float32)\n",
    "output2,state2 = gru_cell(x,state0_gru)\n",
    "\n",
    "init = tf.global_variables\n",
    "_initializer()\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    print(sess.run(output))\n",
    "    print(sess.run(state))\n",
    "\n",
    "    print(sess.run(output2))\n",
    "    print(sess.run(state2))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}