Session_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Session 2.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ahmadhajmosa/3d-force-graph/blob/master/Session_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rmSUExzC-ZBV",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wgWZldyi-Zef",
        "colab_type": "text"
      },
      "source": [
        "# Lab on Machine Learning and Applications in Intelligent Vehicles\n",
        "## Session 1: Introduction\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a75NLOuwKXSs",
        "colab_type": "text"
      },
      "source": [
        "#Session 2: 05.06 - 13:00 - 14:30 :\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ciUtGM-W_ehP",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "## Intro:\n",
        "\n",
        "Tensorflow is a powerful framework for implementing and deploying large-scale deep learning models. Recently, it has been widely used in both reasearch and production. TF objective is to combine scale and flexibility.\n",
        "\n",
        "In the past session, we will learning the following:\n",
        "\n",
        "1. TF programming stack\n",
        "2. TF programming concepts including computatoin graphs, operations and sessions. \n",
        "3. Implementation of linear regression\n",
        "4. Implementation of feed-forward neural networks\n",
        "\n",
        "## TF stack:\n",
        "\n",
        "TensorFlow is a framework composed of two core building blocks — a library for defining computational graphs and a runtime for executing such graphs on a variety of different hardware\n",
        "\n",
        "\n",
        "![alt text](https://www.tensorflow.org/images/layers.png)\n",
        "\n",
        "\n",
        "Before goining into details about the stack, let us talk about computational graphs.\n",
        "\n",
        "### Computational Graphs\n",
        "\n",
        "A directed graph is a data structure consisting of nodes (vertices) and edges. It’s a set of vertices connected pairwise by directed edges.\n",
        "\n",
        "Graphs come in many shapes and sizes and are used to solve many real-life problems, such as representing networks including telephone networks, circuit networks, road networks, and even social networks. \n",
        "![alt text](https://cdn-images-1.medium.com/max/800/1*V6aYjD3AxDbEKYahkGqVQw.png)\n",
        "\n",
        "TensorFlow uses directed graphs internally to represent computations, and they call this data flow graphs (or computational graphs).\n",
        "\n",
        "The nodes in TF data flow graph mostly represents operations, variables and placeholders.\n",
        "\n",
        "Take for example the following operation:\n",
        "![alt text](https://cdn-images-1.medium.com/max/800/1*6E3sfit6DCeJ9mOz17g4bA.png)\n",
        "\n",
        "To create a computational graph out of this program, we create nodes for each of the operations in our program, along with the input variables a and b. In fact, a and b could be constants if they don’t change. If one node is used as the input to another operation we draw a directed arrow that goes from one node to another.\n",
        "\n",
        "The computational graph for this program might look like this:\n",
        "![alt text](https://cdn-images-1.medium.com/max/800/1*vPb9E0Yd1QUAD0oFmAgaOw.png)\n",
        "\n",
        "Operations create or manipulate data according to specific rules. In TensorFlow those rules are called Ops, short for operations. Variables on the other hand represent shared, persistent state that can be manipulated by running Ops on those variables.\n",
        "\n",
        "The questions now what are the advantages of representing operations as directed graphs: The main advantage of using directed graphs is the ability to do **parallelism** and what is called **dependency driving scheduling**. \n",
        "For example, consider again the follwoing code:\n",
        "![alt text](https://cdn-images-1.medium.com/max/800/1*6E3sfit6DCeJ9mOz17g4bA.png)\n",
        "At the most fundamental level, most computer programs are mainly composed of two things — primitive operations and an order in which these operations are executed, often sequentially, line by line. This means we would first multiply a and b and only when this expression was evaluated we would take their sum. Computational graphs on the otherhand, exclusively specify the dependencies across the operations.\n",
        "If we look at our computational graph we see that we could execute the multiplication and addition in parallel. That’s because these two operations do not depend on each other.\n",
        " So we can use the topology of the graph to drive the scheduling of operations and execute them in the most efficient manner, e.g. using multiple GPUs on a single machine or even distribute the execution across multiple machines.\n",
        " Another key advantage is portability. The graph is a language-independent representation of our code. So we can build the graph in Python, save the model (TensorFlow uses protocol buffers), and restore the model in a different language, say C++, if you want to go really fast.\n",
        " \n",
        " \n",
        "\n",
        "--------------------------------\n",
        "# References:\n",
        "\n",
        "https://medium.com/@d3lm/understand-tensorflow-by-mimicking-its-api-from-scratch-faa55787170d\n",
        "\n",
        "https://www.tensorflow.org/guide/extend/architecture\n",
        "\n",
        "https://www.tensorflow.org/guide/low_level_intro\n",
        "\n",
        "  \n",
        " \n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v-GFJPVDnEwx",
        "colab_type": "text"
      },
      "source": [
        "# placeholder: tensors are feeded externaly for example inputs tensors + output tensors\n",
        "\n",
        "# variables : tensors represent the parameters of the network/graph ie. nn weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hlmSCbhtoJBs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 623
        },
        "outputId": "3944635e-d3d4-44df-ed0d-9e0bb3decbfd"
      },
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "# Parameters\n",
        "learning_rate = 0.001\n",
        "training_iters = 2000\n",
        "batch_size = 128\n",
        "\n",
        "# Network Parameters\n",
        "\n",
        "num_inputs = 3\n",
        "num_outputs = 4\n",
        "num_samples= 10\n",
        "# Training data\n",
        "x_gr = np.random.rand(num_samples,num_inputs)\n",
        "y_gr = np.random.rand(num_samples,num_outputs)\n",
        "\n",
        "\n",
        "# tf Graph input\n",
        "x = tf.placeholder(tf.float32, [None, num_inputs])\n",
        "y = tf.placeholder(tf.float32, [None, num_outputs])\n",
        "\n",
        "\n",
        "# weights \n",
        "w_1 = tf.Variable(tf.random_normal([num_inputs,num_outputs ]))\n",
        "\n",
        "# model\n",
        "y_p = tf.matmul(x, w_1)\n",
        "\n",
        "\n",
        "# cost\n",
        "\n",
        "cost = tf.reduce_mean(tf.pow(y-y_p,2)) # \n",
        "\n",
        "# optimisation \n",
        "\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
        "\n",
        "# initalizing the graph and the weights\n",
        "\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "# Launch the graph\n",
        "with tf.Session() as sess:\n",
        "    \n",
        "    sess.run(init)\n",
        "    \n",
        "    for i in range(10):\n",
        "    \n",
        "        sess.run(optimizer, feed_dict={x: x_gr, y: y_gr}) \n",
        "\n",
        "        pr_cost = sess.run(cost, feed_dict={x: x_gr,y: y_gr})\n",
        "    \n",
        "        print('iter: ',i, 'cost: ', pr_cost)\n",
        "    \n",
        "    y_p_p = sess.run(y_p, feed_dict={x: x_gr, y: y_gr})\n",
        "    \n",
        "    print('predicted ', y_p_p)\n",
        "    print('real ', y_gr)\n",
        "\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "iter:  0 cost:  0.5653571\n",
            "iter:  1 cost:  0.56433046\n",
            "iter:  2 cost:  0.563306\n",
            "iter:  3 cost:  0.5622834\n",
            "iter:  4 cost:  0.5612631\n",
            "iter:  5 cost:  0.5602449\n",
            "iter:  6 cost:  0.559229\n",
            "iter:  7 cost:  0.55821526\n",
            "iter:  8 cost:  0.55720377\n",
            "iter:  9 cost:  0.5561945\n",
            "predicted  [[ 1.0505906   0.5653913   0.89188135  1.1427511 ]\n",
            " [ 0.35649964  0.3298351   1.9360914  -0.34332806]\n",
            " [ 0.50524384  0.5289509   0.7199453  -0.25971183]\n",
            " [ 1.0613623   0.4766806   0.8471539   1.4413421 ]\n",
            " [-0.06205952 -0.07624078  1.7560362  -0.30563003]\n",
            " [ 0.5944062   0.73459816  0.94493824 -0.6532444 ]\n",
            " [ 0.55748725  0.3385056   0.56963336  0.47462258]\n",
            " [-0.03822562 -0.30034953  2.2770126   0.31180674]\n",
            " [ 0.01269248  0.04701352  1.679703   -0.43875962]\n",
            " [ 0.24362542  0.2605578   0.17971367 -0.10767066]]\n",
            "real  [[0.60593751 0.79581398 0.74329931 0.43528756]\n",
            " [0.66362961 0.74310117 0.6574602  0.81652125]\n",
            " [0.72119844 0.48900856 0.31647815 0.45463157]\n",
            " [0.86174712 0.86786259 0.35128847 0.10135116]\n",
            " [0.20425938 0.19220971 0.93472783 0.68472656]\n",
            " [0.43276727 0.10864394 0.03879083 0.95408809]\n",
            " [0.47258952 0.49114472 0.81064755 0.52228463]\n",
            " [0.9445423  0.26700245 0.29713047 0.53999886]\n",
            " [0.26961495 0.09676023 0.2027186  0.40247876]\n",
            " [0.55639426 0.95486324 0.97060228 0.26508848]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fv2aqi3Fu-AJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 550
        },
        "outputId": "0eb6b662-362f-4a40-d147-d05e08465aab"
      },
      "source": [
        "sess = tf.Session() \n",
        "sess.run(init)\n",
        "    \n",
        "for i in range(10):\n",
        "    \n",
        "        sess.run(optimizer, feed_dict={x: x_gr, y: y_gr}) \n",
        "\n",
        "        pr_cost = sess.run(cost, feed_dict={x: x_gr,y: y_gr})\n",
        "    \n",
        "        print('iter: ',i, 'cost: ', pr_cost)\n",
        "    \n",
        "y_p_p = sess.run(y_p, feed_dict={x: x_gr, y: y_gr})\n",
        "    \n",
        "print('predicted ', y_p_p)\n",
        "print('real ', y_gr)\n",
        "\n",
        "#sess.close()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter:  0 cost:  0.44907537\n",
            "iter:  1 cost:  0.4482487\n",
            "iter:  2 cost:  0.44742402\n",
            "iter:  3 cost:  0.44660124\n",
            "iter:  4 cost:  0.44578046\n",
            "iter:  5 cost:  0.4449617\n",
            "iter:  6 cost:  0.4441449\n",
            "iter:  7 cost:  0.4433302\n",
            "iter:  8 cost:  0.44251767\n",
            "iter:  9 cost:  0.44170707\n",
            "predicted  [[ 0.05161749  0.6016415   1.3809869   1.7861187 ]\n",
            " [ 1.0831431   0.42920434  1.7101262   0.7552119 ]\n",
            " [ 0.6064712   0.3318555   0.7899883   0.63920474]\n",
            " [-0.1346097   0.59945095  1.3910965   1.8948383 ]\n",
            " [ 0.84721744  0.21161869  1.3733904   0.26691765]\n",
            " [ 0.95271885  0.40499678  0.96117437  0.6507136 ]\n",
            " [ 0.1413996   0.3328487   0.7936052   0.92443055]\n",
            " [ 0.7044489   0.29134876  1.8998408   0.6490483 ]\n",
            " [ 0.9195231   0.23615484  1.3265045   0.28074935]\n",
            " [ 0.2163114   0.13715433  0.24560657  0.2721173 ]]\n",
            "real  [[0.60593751 0.79581398 0.74329931 0.43528756]\n",
            " [0.66362961 0.74310117 0.6574602  0.81652125]\n",
            " [0.72119844 0.48900856 0.31647815 0.45463157]\n",
            " [0.86174712 0.86786259 0.35128847 0.10135116]\n",
            " [0.20425938 0.19220971 0.93472783 0.68472656]\n",
            " [0.43276727 0.10864394 0.03879083 0.95408809]\n",
            " [0.47258952 0.49114472 0.81064755 0.52228463]\n",
            " [0.9445423  0.26700245 0.29713047 0.53999886]\n",
            " [0.26961495 0.09676023 0.2027186  0.40247876]\n",
            " [0.55639426 0.95486324 0.97060228 0.26508848]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vUPSS03avw5D",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 550
        },
        "outputId": "facbe06d-d947-404f-9e5b-95d5a8efb538"
      },
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "# Parameters\n",
        "learning_rate = 0.001\n",
        "training_iters = 2000\n",
        "batch_size = 128\n",
        "\n",
        "# Network Parameters\n",
        "\n",
        "num_inputs = 3\n",
        "num_h1_n = 4\n",
        "num_h2_n = 10\n",
        "num_outputs = 4\n",
        "\n",
        "num_samples= 10\n",
        "\n",
        "# Training data\n",
        "x_gr = np.random.rand(num_samples,num_inputs)\n",
        "y_gr = np.random.rand(num_samples,num_outputs)\n",
        "\n",
        "\n",
        "# tf Graph input\n",
        "x = tf.placeholder(tf.float32, [None, num_inputs])\n",
        "y = tf.placeholder(tf.float32, [None, num_outputs])\n",
        "\n",
        "\n",
        "# weights \n",
        "w_1 = tf.Variable(tf.random_normal([num_inputs,num_h1_n ]))\n",
        "w_2 = tf.Variable(tf.random_normal([num_h1_n,num_h2_n ]))\n",
        "w_3 = tf.Variable(tf.random_normal([num_h2_n,num_outputs ]))\n",
        "\n",
        "# bias \n",
        "b_1 = tf.Variable(tf.random_normal([num_h1_n]))\n",
        "b_2 = tf.Variable(tf.random_normal([num_h2_n]))\n",
        "b_3 = tf.Variable(tf.random_normal([num_outputs]))\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# model\n",
        "\n",
        "h1 = tf.nn.sigmoid(tf.add(tf.matmul(x, w_1),b_1)) # model of hidden layer 1\n",
        "h2 = tf.nn.sigmoid(tf.add(tf.matmul(h1, w_2),b_2)) # model of hidden layer 2\n",
        "y_p = tf.add(tf.matmul(h2, w_3),b_3) # model of the output layer\n",
        "\n",
        "\n",
        "\n",
        "# cost\n",
        "\n",
        "cost = tf.reduce_mean(tf.pow(y-y_p,2)) # \n",
        "\n",
        "# optimisation \n",
        "\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
        "\n",
        "# initalizing the graph and the weights\n",
        "\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "# Launch the graph\n",
        "with tf.Session() as sess:\n",
        "    \n",
        "    sess.run(init)\n",
        "    \n",
        "    for i in range(10):\n",
        "    \n",
        "        sess.run(optimizer, feed_dict={x: x_gr, y: y_gr}) \n",
        "\n",
        "        pr_cost = sess.run(cost, feed_dict={x: x_gr,y: y_gr})\n",
        "    \n",
        "        print('iter: ',i, 'cost: ', pr_cost)\n",
        "    \n",
        "    y_p_p = sess.run(y_p, feed_dict={x: x_gr, y: y_gr})\n",
        "    \n",
        "    print('predicted ', y_p_p)\n",
        "    print('real ', y_gr)\n",
        "\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "iter:  0 cost:  3.3546257\n",
            "iter:  1 cost:  3.3263478\n",
            "iter:  2 cost:  3.298264\n",
            "iter:  3 cost:  3.2703712\n",
            "iter:  4 cost:  3.2426636\n",
            "iter:  5 cost:  3.2151332\n",
            "iter:  6 cost:  3.1877713\n",
            "iter:  7 cost:  3.1605754\n",
            "iter:  8 cost:  3.1335444\n",
            "iter:  9 cost:  3.10668\n",
            "predicted  [[0.40542805 2.522129   1.5951774  3.148585  ]\n",
            " [0.3167858  2.1697245  1.6324538  3.2137492 ]\n",
            " [0.26505983 2.0856333  1.5818598  3.1661534 ]\n",
            " [0.5225722  2.6145244  1.7394813  3.2885988 ]\n",
            " [0.27241385 2.2369065  1.5139519  3.0873516 ]\n",
            " [0.48785162 2.4845834  1.7652564  3.2892911 ]\n",
            " [0.3519792  2.264122   1.6373122  3.236201  ]\n",
            " [0.35564208 2.2671547  1.6636946  3.2024086 ]\n",
            " [0.46710527 2.5822039  1.6825379  3.1649113 ]\n",
            " [0.25208998 2.106409   1.5723711  3.090046  ]]\n",
            "real  [[0.36544457 0.39483693 0.37937616 0.79014518]\n",
            " [0.42060702 0.57857147 0.36140349 0.63731132]\n",
            " [0.09807871 0.00988529 0.33781145 0.8633689 ]\n",
            " [0.18984533 0.93611594 0.00126396 0.53592332]\n",
            " [0.36521555 0.34973814 0.60808643 0.30143436]\n",
            " [0.52384758 0.83897256 0.84312165 0.24282307]\n",
            " [0.45749241 0.63146022 0.13439527 0.36707223]\n",
            " [0.3809692  0.72245415 0.34505035 0.15178537]\n",
            " [0.40203668 0.41739336 0.77573583 0.19966756]\n",
            " [0.69439129 0.15010037 0.36116077 0.68111029]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zwhCMk9VyVj-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 822
        },
        "outputId": "8cf2227b-4b58-4112-db71-bd7e6bb758aa"
      },
      "source": [
        ""
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-2-8d45f2bd7c6a>:7: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please write your own downloading logic.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use urllib or similar directly.\n",
            "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.data to implement this functionality.\n",
            "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
            "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.data to implement this functionality.\n",
            "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.one_hot on tensors.\n",
            "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
            "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
            "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
            "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADjlJREFUeJzt3X+MHPV5x/HPgzmfg20wDsnlBCZH\nqJOUoNRODtMCak0dKLFQTZrGtVvQVXK4lEBVlAiFOopK8kdFUUNEQ7B6FCsmDT8iBcemMm2Ikwil\nIuAzcmyDCRBygJ2zD2xHNqSx7+ynf+w4OszNd5fd2Z09P++XdLq9eebHo4GPZ3ZnZ77m7gIQz0ll\nNwCgHIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQJ7dyY1Ot06dpeis3CYTyW72hw37Iapm3\nofCb2RWS7pA0RdJ/uPutqfmnaboutEWNbBJAwhO+seZ56z7tN7Mpkr4h6eOSzpO03MzOq3d9AFqr\nkff8CyS94O4vuvthSQ9IWlJMWwCarZHwnynplXF/78ymvYmZ9ZvZoJkNjupQA5sDUKSmf9rv7gPu\n3uvuvR3qbPbmANSokfDvkjRn3N9nZdMATAKNhH+TpLlmdo6ZTZW0TNL6YtoC0Gx1X+pz9zEzu0HS\n/6hyqW+1uz9dWGcAmqqh6/zuvkHShoJ6AdBCfL0XCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxA\nUIQfCIrwA0ERfiAowg8ERfiBoBoapdfMhiQdlHRE0pi79xbRFIDmayj8mUvd/bUC1gOghTjtB4Jq\nNPwu6ftmttnM+otoCEBrNHraf4m77zKzd0t61MyedffHxs+Q/aPQL0nTdEqDmwNQlIaO/O6+K/s9\nImmtpAUTzDPg7r3u3tuhzkY2B6BAdYffzKab2cxjryVdLml7UY0BaK5GTvu7JK01s2Pruc/d/7uQ\nrgA0Xd3hd/cXJf1Bgb0AaCEu9QFBEX4gKMIPBEX4gaAIPxAU4QeCKuKuPpRs+HMX5dbM08tO25ue\nYf8H08t3P34kvf6Hn0yvAKXhyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQZ0w1/lHrs+/1i1Jv/7w\naLK+9vI7i2ynpX5/6qa6l/2tjyXrp530jmR95Jo3kvVf/Vv+/2K3774suezepacm62Ov7EzWkcaR\nHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCMvcqN3wX6FSb7RfaorqXf+7uC3Jrzy6+K7lsp3XUvV2U\n4+qhhcn6/r+u8j2AoZcL7GZyeMI36oDvs1rm5cgPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0FVvZ/f\nzFZLulLSiLufn02bLelBST2ShiQtdff9zWuzYtWl9+bWql3H/5e9c5P1kcMz6+qpCA9t/miyfvbD\nNV22LcXORenjx22L78utfXLGgeSy/9nz42T96vsWJuv7/+qs3BrPAqjtyP9NSVccN+1mSRvdfa6k\njdnfACaRquF398ck7Ttu8hJJa7LXayRdVXBfAJqs3vf8Xe4+nL3eLamroH4AtEjDH/h55eaA3BsE\nzKzfzAbNbHBUhxrdHICC1Bv+PWbWLUnZ75G8Gd19wN173b23Q511bg5A0eoN/3pJfdnrPknrimkH\nQKtUDb+Z3S/pcUkfMLOdZrZC0q2SLjOz5yV9LPsbwCQyqe7nt49+KLf22rz0vd3v/t7Pk/Uje4+/\noIEinPThD+bWrnzgf5PLXj/rlYa2/YF7rsut9Xzp8YbW3a64nx9AVYQfCIrwA0ERfiAowg8ERfiB\noCbVpT6cWPZe+0fJ+uCXVzW0/s2HDufWVp6zoKF1tysu9QGoivADQRF+ICjCDwRF+IGgCD8QFOEH\ngiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjpEN9CInSsvyq0dnX+wqdvumpJ/P//Yn6aH\nRT/5h5uLbqftcOQHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaCqPrffzFZLulLSiLufn027RdK1kl7N\nZlvp7huqbYzn9jfHye/rya29sKI7uexdywYK7ubNFk4bza1NsfKOPb8YfT1Z/+x7L2lRJ8Uq+rn9\n35R0xQTTv+bu87KfqsEH0F6qht/dH5O0rwW9AGihRs67bjCzrWa22sxOL6wjAC1Rb/hXSTpX0jxJ\nw5K+mjejmfWb2aCZDY7qUJ2bA1C0usLv7nvc/Yi7H5V0t6TcUQ/dfcDde929t0Od9fYJoGB1hd/M\nxn+E/AlJ24tpB0CrVL2l18zul7RQ0hlmtlPSP0laaGbzJLmkIUmfaWKPAJqgavjdffkEk+9pQi9h\nvf6pC5P1Vz+SPkH7yl88kFtbNnN/XT0Vpz2/R/axH9yYrL9fgy3qpDzt+V8GQNMRfiAowg8ERfiB\noAg/EBThB4Li0d0FsPkfStZn3TmcrG/oWZWsN/PW1++9MSNZ3/5/ZzW0/v+6bWFubcqh9O3kfV95\nOFnvP+1X9bQkSZq6u6PuZU8UHPmBoAg/EBThB4Ii/EBQhB8IivADQRF+ICiu89fopS/nDzX9pWUP\nJpf9m5l7k/WXx36TrD97OP2IxL+//9O5tVOG009x7v7xa8n6kWeeS9arOU0/rXvZ5/+xq8rK09f5\nf5l4PHfPuvSjuyPgyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQXGdv0azLhjJrVW7jr/omT9P1ke/\n/p5k/R3rnkzWe/R4sp5ypO4lG3f0T+Yn61fNqvaE+PSxa9/RqfnFJ7dVWfeJjyM/EBThB4Ii/EBQ\nhB8IivADQRF+ICjCDwRV9Tq/mc2RdK+kLkkuacDd7zCz2ZIelNQjaUjSUncvezzopnnnivz7v3/v\nc9cllz33pvR1+JP1cl09TXb73z8tWb94WmPHpv7tV+fWzlBjzyk4EdSyd8ckfd7dz5P0h5KuN7Pz\nJN0saaO7z5W0MfsbwCRRNfzuPuzuT2WvD0raIelMSUskrclmWyPpqmY1CaB4b+u8ysx6JM2X9ISk\nLnc/Ng7VblXeFgCYJGoOv5nNkPRdSTe6+4HxNXd3VT4PmGi5fjMbNLPBUR1qqFkAxakp/GbWoUrw\nv+3uD2WT95hZd1bvljThnS/uPuDuve7e26HOInoGUICq4Tczk3SPpB3ufvu40npJfdnrPknrim8P\nQLPUckvvxZKukbTNzLZk01ZKulXSd8xshaSXJC1tTovtYWx4d27t3Jvya8i394KxhpbfcTj9yPOZ\nd53W0PpPdFXD7+4/kZT38PdFxbYDoFX4hh8QFOEHgiL8QFCEHwiK8ANBEX4gKB7djab6s+0Hcmtr\nZ32jytKJR29L6nu6L1k//ZFNVdYfG0d+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK6/xoqr88dWtu\n7ZSTZiSXfW70jWT9lDtn1dUTKjjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQXOdHQ0Y+e1Gy3jUl\n/576X47mD3suScv/+aZk/YxH0kOfI40jPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfU6v5nNkXSv\npC5JLmnA3e8ws1skXSvp1WzWle6+oVmNohzW2Zmsf/LvfpisHzx6OLe2+Mnrksue/e9cx2+mWr7k\nMybp8+7+lJnNlLTZzB7Nal9z939tXnsAmqVq+N19WNJw9vqgme2QdGazGwPQXG/rPb+Z9UiaL+mJ\nbNINZrbVzFab2ek5y/Sb2aCZDY7qUEPNAihOzeE3sxmSvivpRnc/IGmVpHMlzVPlzOCrEy3n7gPu\n3uvuvR1Kv38E0Do1hd/MOlQJ/rfd/SFJcvc97n7E3Y9KulvSgua1CaBoVcNvZibpHkk73P32cdO7\nx832CUnbi28PQLPU8mn/xZKukbTNzLZk01ZKWm5m81S5/Dck6TNN6RDlOurJ8rcevjRZf+RnC3Nr\nZ3/np/V0hILU8mn/TyTZBCWu6QOTGN/wA4Ii/EBQhB8IivADQRF+ICjCDwTFo7uR5KP5t+RKUs8X\nue12suLIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBmXv6fu1CN2b2qqSXxk06Q9JrLWvg7WnX3tq1\nL4ne6lVkb+9193fVMmNLw/+WjZsNuntvaQ0ktGtv7dqXRG/1Kqs3TvuBoAg/EFTZ4R8oefsp7dpb\nu/Yl0Vu9Sumt1Pf8AMpT9pEfQElKCb+ZXWFmPzezF8zs5jJ6yGNmQ2a2zcy2mNlgyb2sNrMRM9s+\nbtpsM3vUzJ7Pfk84TFpJvd1iZruyfbfFzBaX1NscM/uRmT1jZk+b2T9k00vdd4m+StlvLT/tN7Mp\nkp6TdJmknZI2SVru7s+0tJEcZjYkqdfdS78mbGZ/LOl1Sfe6+/nZtNsk7XP3W7N/OE939y+0SW+3\nSHq97JGbswFlusePLC3pKkl/qxL3XaKvpSphv5Vx5F8g6QV3f9HdD0t6QNKSEvpoe+7+mKR9x01e\nImlN9nqNKv/ztFxOb23B3Yfd/ans9UFJx0aWLnXfJfoqRRnhP1PSK+P+3qn2GvLbJX3fzDabWX/Z\nzUygKxs2XZJ2S+oqs5kJVB25uZWOG1m6bfZdPSNeF40P/N7qEnf/iKSPS7o+O71tS155z9ZOl2tq\nGrm5VSYYWfp3ytx39Y54XbQywr9L0pxxf5+VTWsL7r4r+z0iaa3ab/ThPccGSc1+j5Tcz++008jN\nE40srTbYd+004nUZ4d8kaa6ZnWNmUyUtk7S+hD7ewsymZx/EyMymS7pc7Tf68HpJfdnrPknrSuzl\nTdpl5Oa8kaVV8r5ruxGv3b3lP5IWq/KJ/y8kfbGMHnL6ep+kn2U/T5fdm6T7VTkNHFXls5EVkt4p\naaOk5yX9QNLsNurtW5K2SdqqStC6S+rtElVO6bdK2pL9LC573yX6KmW/8Q0/ICg+8AOCIvxAUIQf\nCIrwA0ERfiAowg8ERfiBoAg/ENT/AyErW1pw/s8cAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UaeRp0T10834",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# training data\n",
        "X_train = mnist.train.images\n",
        "Y_train = mnist.train.labels\n",
        "\n",
        "# training data\n",
        "X_test = mnist.test.images\n",
        "Y_test = mnist.test.labels\n",
        "\n",
        "# training data\n",
        "X_val = mnist.validation.images\n",
        "Y_val = mnist.validation.labels\n",
        "\n",
        "\n",
        "# Parameters\n",
        "learning_rate = 0.001\n",
        "training_iters = 2000\n",
        "batch_size = 128\n",
        "\n",
        "# Network Parameters\n",
        "\n",
        "num_inputs = 784\n",
        "num_h1_n = 100\n",
        "num_h2_n = 100\n",
        "num_outputs = 10\n",
        "\n",
        "\n",
        "\n",
        "# tf Graph input\n",
        "x = tf.placeholder(tf.float32, [None, num_inputs])\n",
        "y = tf.placeholder(tf.float32, [None, num_outputs])\n",
        "\n",
        "\n",
        "# weights \n",
        "w_1 = tf.Variable(tf.random_normal([num_inputs,num_h1_n ]))\n",
        "w_2 = tf.Variable(tf.random_normal([num_h1_n,num_h2_n ]))\n",
        "w_3 = tf.Variable(tf.random_normal([num_h2_n,num_outputs ]))\n",
        "\n",
        "# bias \n",
        "b_1 = tf.Variable(tf.random_normal([num_h1_n]))\n",
        "b_2 = tf.Variable(tf.random_normal([num_h2_n]))\n",
        "b_3 = tf.Variable(tf.random_normal([num_outputs]))\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# model\n",
        "\n",
        "h1 = tf.nn.sigmoid(tf.add(tf.matmul(x, w_1),b_1)) # model of hidden layer 1\n",
        "h2 = tf.nn.sigmoid(tf.add(tf.matmul(h1, w_2),b_2)) # model of hidden layer 2\n",
        "y_p = tf.add(tf.matmul(h2, w_3),b_3) # model of the output layer\n",
        "\n",
        "\n",
        "\n",
        "# cost\n",
        "\n",
        "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_p, labels=y)) # cross entropy cost\n",
        "\n",
        "\n",
        "# Evaluate model\n",
        "correct_pred = tf.equal(tf.argmax(y_p, 1), tf.argmax(y, 1))\n",
        "\n",
        "## 3 images, y_p=[[0.1,0.0,0,0.9],[0.9,0.1,0,0.],[0,0.9,0,0.1]] \n",
        "\n",
        "# tf.argmax(y_p, 1) [3,0,1] \n",
        "\n",
        "# 3 images, y=[[0,0.0,0,1],[0,1,0,0],[0,1,0,0]] \n",
        "\n",
        "# tf.argmax(y, 1) [3,1,1]\n",
        "\n",
        "# tf.equal [True,False,True]--[1,0,1]--- 2/3 \n",
        "\n",
        "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
        "#\n",
        "\n",
        "# optimisation \n",
        "\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
        "\n",
        "# initalizing the graph and the weights\n",
        "\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "# Launch the graph\n",
        "with tf.Session() as sess:\n",
        "    \n",
        "    sess.run(init)\n",
        "    \n",
        "    for i in range(1000):\n",
        "        \n",
        "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
        "        \n",
        "        # Run optimization op (backprop)\n",
        "        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})\n",
        "    \n",
        "\n",
        "        train_cost, train_acc  = sess.run([cost,accuracy], feed_dict={x: batch_x,y: batch_y})\n",
        "    \n",
        "        \n",
        "        test_batch_x, test_batch_y = mnist.test.next_batch(batch_size)\n",
        "\n",
        "        test_cost, test_acc  = sess.run([cost,accuracy], feed_dict={x: test_batch_x,y: test_batch_y})\n",
        "        print('iter: ',i, 'train_cost: ', train_cost, 'train_acc: ', train_acc,'test_cost: ', test_cost, 'test_acc: ', test_acc )\n",
        "\n",
        "    \n",
        "    #y_p_p = sess.run(y_p, feed_dict={x: x_gr, y: y_gr})\n",
        "    \n",
        "    print('predicted ', y_p_p)\n",
        "    print('real ', y_gr)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VDkPoIsB90MW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf \n",
        "\n",
        "\n",
        "def variable_summaries(var):\n",
        "  \"\"\"Attach a lot of summaries to a Tensor (for TensorBoard visualization).\"\"\"\n",
        "  with tf.name_scope('summaries'):\n",
        "    mean = tf.reduce_mean(var)\n",
        "    tf.summary.scalar('mean', mean)\n",
        "    with tf.name_scope('stddev'):\n",
        "      stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n",
        "    tf.summary.scalar('stddev', stddev)\n",
        "    tf.summary.scalar('max', tf.reduce_max(var))\n",
        "    tf.summary.scalar('min', tf.reduce_min(var))\n",
        "    tf.summary.histogram('histogram', var)\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iRJsyYmvBE43",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 822
        },
        "outputId": "609a4648-c8d2-4fb0-a20b-905fad48f9a2"
      },
      "source": [
        "%matplotlib inline\n",
        "\n",
        "from matplotlib.pyplot import imshow\n",
        "\n",
        "# Import MNIST data\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
        "\n",
        "mnist.train.images.shape\n",
        "\n",
        "image =mnist.train.images[0].reshape((28,28))\n",
        "#MNIST data input (img shape: 28*28)\n",
        "imshow(image)\n",
        "\n",
        "mnist.train.labels[0]"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-1-8d45f2bd7c6a>:7: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please write your own downloading logic.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use urllib or similar directly.\n",
            "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.data to implement this functionality.\n",
            "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
            "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.data to implement this functionality.\n",
            "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use tf.one_hot on tensors.\n",
            "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
            "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
            "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
            "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADjlJREFUeJzt3X+MHPV5x/HPgzmfg20wDsnlBCZH\nqJOUoNRODtMCak0dKLFQTZrGtVvQVXK4lEBVlAiFOopK8kdFUUNEQ7B6FCsmDT8iBcemMm2Ikwil\nIuAzcmyDCRBygJ2zD2xHNqSx7+ynf+w4OszNd5fd2Z09P++XdLq9eebHo4GPZ3ZnZ77m7gIQz0ll\nNwCgHIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQJ7dyY1Ot06dpeis3CYTyW72hw37Iapm3\nofCb2RWS7pA0RdJ/uPutqfmnaboutEWNbBJAwhO+seZ56z7tN7Mpkr4h6eOSzpO03MzOq3d9AFqr\nkff8CyS94O4vuvthSQ9IWlJMWwCarZHwnynplXF/78ymvYmZ9ZvZoJkNjupQA5sDUKSmf9rv7gPu\n3uvuvR3qbPbmANSokfDvkjRn3N9nZdMATAKNhH+TpLlmdo6ZTZW0TNL6YtoC0Gx1X+pz9zEzu0HS\n/6hyqW+1uz9dWGcAmqqh6/zuvkHShoJ6AdBCfL0XCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxA\nUIQfCIrwA0ERfiAowg8ERfiBoBoapdfMhiQdlHRE0pi79xbRFIDmayj8mUvd/bUC1gOghTjtB4Jq\nNPwu6ftmttnM+otoCEBrNHraf4m77zKzd0t61MyedffHxs+Q/aPQL0nTdEqDmwNQlIaO/O6+K/s9\nImmtpAUTzDPg7r3u3tuhzkY2B6BAdYffzKab2cxjryVdLml7UY0BaK5GTvu7JK01s2Pruc/d/7uQ\nrgA0Xd3hd/cXJf1Bgb0AaCEu9QFBEX4gKMIPBEX4gaAIPxAU4QeCKuKuPpRs+HMX5dbM08tO25ue\nYf8H08t3P34kvf6Hn0yvAKXhyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQZ0w1/lHrs+/1i1Jv/7w\naLK+9vI7i2ynpX5/6qa6l/2tjyXrp530jmR95Jo3kvVf/Vv+/2K3774suezepacm62Ov7EzWkcaR\nHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCMvcqN3wX6FSb7RfaorqXf+7uC3Jrzy6+K7lsp3XUvV2U\n4+qhhcn6/r+u8j2AoZcL7GZyeMI36oDvs1rm5cgPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0FVvZ/f\nzFZLulLSiLufn02bLelBST2ShiQtdff9zWuzYtWl9+bWql3H/5e9c5P1kcMz6+qpCA9t/miyfvbD\nNV22LcXORenjx22L78utfXLGgeSy/9nz42T96vsWJuv7/+qs3BrPAqjtyP9NSVccN+1mSRvdfa6k\njdnfACaRquF398ck7Ttu8hJJa7LXayRdVXBfAJqs3vf8Xe4+nL3eLamroH4AtEjDH/h55eaA3BsE\nzKzfzAbNbHBUhxrdHICC1Bv+PWbWLUnZ75G8Gd19wN173b23Q511bg5A0eoN/3pJfdnrPknrimkH\nQKtUDb+Z3S/pcUkfMLOdZrZC0q2SLjOz5yV9LPsbwCQyqe7nt49+KLf22rz0vd3v/t7Pk/Uje4+/\noIEinPThD+bWrnzgf5PLXj/rlYa2/YF7rsut9Xzp8YbW3a64nx9AVYQfCIrwA0ERfiAowg8ERfiB\noCbVpT6cWPZe+0fJ+uCXVzW0/s2HDufWVp6zoKF1tysu9QGoivADQRF+ICjCDwRF+IGgCD8QFOEH\ngiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjpEN9CInSsvyq0dnX+wqdvumpJ/P//Yn6aH\nRT/5h5uLbqftcOQHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaCqPrffzFZLulLSiLufn027RdK1kl7N\nZlvp7huqbYzn9jfHye/rya29sKI7uexdywYK7ubNFk4bza1NsfKOPb8YfT1Z/+x7L2lRJ8Uq+rn9\n35R0xQTTv+bu87KfqsEH0F6qht/dH5O0rwW9AGihRs67bjCzrWa22sxOL6wjAC1Rb/hXSTpX0jxJ\nw5K+mjejmfWb2aCZDY7qUJ2bA1C0usLv7nvc/Yi7H5V0t6TcUQ/dfcDde929t0Od9fYJoGB1hd/M\nxn+E/AlJ24tpB0CrVL2l18zul7RQ0hlmtlPSP0laaGbzJLmkIUmfaWKPAJqgavjdffkEk+9pQi9h\nvf6pC5P1Vz+SPkH7yl88kFtbNnN/XT0Vpz2/R/axH9yYrL9fgy3qpDzt+V8GQNMRfiAowg8ERfiB\noAg/EBThB4Li0d0FsPkfStZn3TmcrG/oWZWsN/PW1++9MSNZ3/5/ZzW0/v+6bWFubcqh9O3kfV95\nOFnvP+1X9bQkSZq6u6PuZU8UHPmBoAg/EBThB4Ii/EBQhB8IivADQRF+ICiu89fopS/nDzX9pWUP\nJpf9m5l7k/WXx36TrD97OP2IxL+//9O5tVOG009x7v7xa8n6kWeeS9arOU0/rXvZ5/+xq8rK09f5\nf5l4PHfPuvSjuyPgyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQXGdv0azLhjJrVW7jr/omT9P1ke/\n/p5k/R3rnkzWe/R4sp5ypO4lG3f0T+Yn61fNqvaE+PSxa9/RqfnFJ7dVWfeJjyM/EBThB4Ii/EBQ\nhB8IivADQRF+ICjCDwRV9Tq/mc2RdK+kLkkuacDd7zCz2ZIelNQjaUjSUncvezzopnnnivz7v3/v\nc9cllz33pvR1+JP1cl09TXb73z8tWb94WmPHpv7tV+fWzlBjzyk4EdSyd8ckfd7dz5P0h5KuN7Pz\nJN0saaO7z5W0MfsbwCRRNfzuPuzuT2WvD0raIelMSUskrclmWyPpqmY1CaB4b+u8ysx6JM2X9ISk\nLnc/Ng7VblXeFgCYJGoOv5nNkPRdSTe6+4HxNXd3VT4PmGi5fjMbNLPBUR1qqFkAxakp/GbWoUrw\nv+3uD2WT95hZd1bvljThnS/uPuDuve7e26HOInoGUICq4Tczk3SPpB3ufvu40npJfdnrPknrim8P\nQLPUckvvxZKukbTNzLZk01ZKulXSd8xshaSXJC1tTovtYWx4d27t3Jvya8i394KxhpbfcTj9yPOZ\nd53W0PpPdFXD7+4/kZT38PdFxbYDoFX4hh8QFOEHgiL8QFCEHwiK8ANBEX4gKB7djab6s+0Hcmtr\nZ32jytKJR29L6nu6L1k//ZFNVdYfG0d+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK6/xoqr88dWtu\n7ZSTZiSXfW70jWT9lDtn1dUTKjjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQXOdHQ0Y+e1Gy3jUl\n/576X47mD3suScv/+aZk/YxH0kOfI40jPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfU6v5nNkXSv\npC5JLmnA3e8ws1skXSvp1WzWle6+oVmNohzW2Zmsf/LvfpisHzx6OLe2+Mnrksue/e9cx2+mWr7k\nMybp8+7+lJnNlLTZzB7Nal9z939tXnsAmqVq+N19WNJw9vqgme2QdGazGwPQXG/rPb+Z9UiaL+mJ\nbNINZrbVzFab2ek5y/Sb2aCZDY7qUEPNAihOzeE3sxmSvivpRnc/IGmVpHMlzVPlzOCrEy3n7gPu\n3uvuvR1Kv38E0Do1hd/MOlQJ/rfd/SFJcvc97n7E3Y9KulvSgua1CaBoVcNvZibpHkk73P32cdO7\nx832CUnbi28PQLPU8mn/xZKukbTNzLZk01ZKWm5m81S5/Dck6TNN6RDlOurJ8rcevjRZf+RnC3Nr\nZ3/np/V0hILU8mn/TyTZBCWu6QOTGN/wA4Ii/EBQhB8IivADQRF+ICjCDwTFo7uR5KP5t+RKUs8X\nue12suLIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBmXv6fu1CN2b2qqSXxk06Q9JrLWvg7WnX3tq1\nL4ne6lVkb+9193fVMmNLw/+WjZsNuntvaQ0ktGtv7dqXRG/1Kqs3TvuBoAg/EFTZ4R8oefsp7dpb\nu/Yl0Vu9Sumt1Pf8AMpT9pEfQElKCb+ZXWFmPzezF8zs5jJ6yGNmQ2a2zcy2mNlgyb2sNrMRM9s+\nbtpsM3vUzJ7Pfk84TFpJvd1iZruyfbfFzBaX1NscM/uRmT1jZk+b2T9k00vdd4m+StlvLT/tN7Mp\nkp6TdJmknZI2SVru7s+0tJEcZjYkqdfdS78mbGZ/LOl1Sfe6+/nZtNsk7XP3W7N/OE939y+0SW+3\nSHq97JGbswFlusePLC3pKkl/qxL3XaKvpSphv5Vx5F8g6QV3f9HdD0t6QNKSEvpoe+7+mKR9x01e\nImlN9nqNKv/ztFxOb23B3Yfd/ans9UFJx0aWLnXfJfoqRRnhP1PSK+P+3qn2GvLbJX3fzDabWX/Z\nzUygKxs2XZJ2S+oqs5kJVB25uZWOG1m6bfZdPSNeF40P/N7qEnf/iKSPS7o+O71tS155z9ZOl2tq\nGrm5VSYYWfp3ytx39Y54XbQywr9L0pxxf5+VTWsL7r4r+z0iaa3ab/ThPccGSc1+j5Tcz++008jN\nE40srTbYd+004nUZ4d8kaa6ZnWNmUyUtk7S+hD7ewsymZx/EyMymS7pc7Tf68HpJfdnrPknrSuzl\nTdpl5Oa8kaVV8r5ruxGv3b3lP5IWq/KJ/y8kfbGMHnL6ep+kn2U/T5fdm6T7VTkNHFXls5EVkt4p\naaOk5yX9QNLsNurtW5K2SdqqStC6S+rtElVO6bdK2pL9LC573yX6KmW/8Q0/ICg+8AOCIvxAUIQf\nCIrwA0ERfiAowg8ERfiBoAg/ENT/AyErW1pw/s8cAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tXZtWbWQ92P_",
        "colab_type": "text"
      },
      "source": [
        "# CNN using Tensorflow\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q6k9YPZQ95z5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        },
        "outputId": "088dcb25-3829-4f00-eb1c-f396042d5988"
      },
      "source": [
        "n_classes=10\n",
        "learning_rate=0.002\n",
        "batch_size=64\n",
        "\n",
        "import tensorflow as tf\n",
        "# Create some wrappers for simplicity\n",
        "def conv2d(x, W, b, strides=1):\n",
        "    # Conv2D wrapper, with bias and relu activation\n",
        "    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')\n",
        "    x = tf.nn.bias_add(x, b)\n",
        "    return tf.nn.relu(x)\n",
        "\n",
        "\n",
        "num_inputs = 784\n",
        "num_outputs= 10\n",
        "\n",
        "# tf Graph input\n",
        "x = tf.placeholder(tf.float32, [None, num_inputs],name = 'input')\n",
        "y = tf.placeholder(tf.float32, [None, num_outputs],name =  'output')\n",
        "\n",
        "\n",
        "def maxpool2d(x, k=2):\n",
        "    # MaxPool2D wrapper\n",
        "    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],\n",
        "                          padding='SAME')\n",
        "\n",
        "\n",
        "# Create model\n",
        "def conv_net(x, weights, biases):\n",
        "    # Reshape input picture\n",
        "    \n",
        "    with tf.name_scope('input-reshape'):\n",
        "        x = tf.reshape(x, shape=[-1, 28, 28, 1])\n",
        "\n",
        "    with tf.name_scope('conv-1'):\n",
        "\n",
        "        # Convolution Layer\n",
        "        conv1 = conv2d(x, weights['wc1'], biases['bc1'])\n",
        "        print('con1_before max',conv1.get_shape().as_list())\n",
        "\n",
        "    with tf.name_scope('maxpooling-1'):\n",
        "        \n",
        "        # Max Pooling (down-sampling)\n",
        "        conv1 = maxpool2d(conv1, k=2)\n",
        "        print('con1_after max',conv1.get_shape().as_list())\n",
        "\n",
        "\n",
        "    # Convolution Layer\n",
        "    with tf.name_scope('conv-2'):\n",
        "\n",
        "        conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])\n",
        "        print('con2_before max',conv2.get_shape().as_list())\n",
        "\n",
        "\n",
        "    with tf.name_scope('maxpooling-2'):\n",
        "        \n",
        "     \n",
        "        # Max Pooling (down-sampling)\n",
        "        conv2 = maxpool2d(conv2, k=2)\n",
        "        print('con2_after max', conv2.get_shape().as_list())\n",
        "\n",
        "    # Fully connected layer\n",
        "    # Reshape conv2 output to fit fully connected layer input\n",
        "    #wd1 numx3x3  wd1.get_shape() -> numx9 \n",
        "    with tf.name_scope('flatten'):\n",
        "    \n",
        "        fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])\n",
        "        fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])\n",
        "        fc1 = tf.nn.relu(fc1)\n",
        "    # Apply Dropout\n",
        "    #fc1 = tf.nn.dropout(fc1, dropout)\n",
        "\n",
        "    # Output, class prediction\n",
        "    with tf.name_scope('output'):\n",
        "    \n",
        "        out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])\n",
        "    return out\n",
        "\n",
        "\n",
        "# Store layers weight & bias\n",
        "weights = {\n",
        "    # 5x5 conv, 1 input, 32 outputs\n",
        "    'wc1': tf.Variable(tf.random_normal([5, 5, 1, 32]), name=\"wc1\"),\n",
        "    # 5x5 conv, 32 inputs, 64 outputs\n",
        "    'wc2': tf.Variable(tf.random_normal([5, 5, 32, 64])),\n",
        "    # fully connected, 7*7*64 inputs, 1024 outputs\n",
        "    'wd1': tf.Variable(tf.random_normal([7*7*64, 1024])),\n",
        "    # 1024 inputs, 10 outputs (class prediction)\n",
        "    'out': tf.Variable(tf.random_normal([1024, n_classes]))\n",
        "}\n",
        "\n",
        "biases = {\n",
        "    'bc1': tf.Variable(tf.random_normal([32])),\n",
        "    'bc2': tf.Variable(tf.random_normal([64])),\n",
        "    'bd1': tf.Variable(tf.random_normal([1024])),\n",
        "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
        "}\n",
        "\n",
        "y_p = conv_net(x, weights, biases)\n",
        "\n",
        "#crossentropy cost\n",
        "with tf.name_scope('cross_entropy'):\n",
        "\n",
        "    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_p, labels=y)) # cross entropy cost\n",
        "\n",
        "#tf.summary.scalar('cross_entropy', cost)\n",
        "\n",
        "\n",
        "# Evaluate model\n",
        "with tf.name_scope('operations'):\n",
        "\n",
        "    correct_pred = tf.equal(tf.argmax(y_p, 1), tf.argmax(y, 1))\n",
        "    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
        "#\n",
        "merged = tf.summary.scalar('accuracy', accuracy)\n",
        "\n",
        "\n",
        "\n",
        "# optimisation \n",
        "with tf.name_scope('optimisation'):\n",
        "\n",
        "    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
        "\n",
        "# initalizing the graph and the weights\n",
        "#merged = tf.summary.merge_all()\n",
        "\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "\n",
        "\n",
        "# Launch the graph\n",
        "with tf.Session() as sess:\n",
        "    #graph_writer = tf.summary.FileWriter('./log/train', sess.graph)\n",
        "    #graph_writer.add_summary()\n",
        "    train_writer = tf.summary.FileWriter('./log/train', sess.graph)\n",
        "    test_writer = tf.summary.FileWriter('./log/test')\n",
        "    sess.run(init)\n",
        "    print('started')\n",
        "    \n",
        "    for i in range(10000):\n",
        "        \n",
        "        \n",
        "        \n",
        "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
        "        \n",
        "\n",
        "        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})\n",
        "\n",
        "\n",
        "        summary,train_cost , train_acc = sess.run([merged,cost,accuracy], feed_dict={x: batch_x,y: batch_y})\n",
        "        train_writer.add_summary(summary, i)\n",
        "\n",
        "        #print('started')\n",
        "\n",
        "        test_batch_x, test_batch_y = mnist.test.next_batch(batch_size)\n",
        "\n",
        "        summary,test_cost, test_acc  = sess.run([merged,cost,accuracy], feed_dict={x: test_batch_x,y: test_batch_y})\n",
        "        train_writer.add_summary(summary, i)\n",
        "        if i % 100 ==0:\n",
        "\n",
        "            print('iter: ',i, 'train_cost: ', train_cost, 'train_acc: ', train_acc,'test_cost: ', test_cost, 'test_acc: ', test_acc )\n",
        "\n",
        "\n",
        "    #y_p_p = sess.run(y_p, feed_dict={x: x_gr, y: y_gr})\n",
        "    \n",
        "    print('predicted ', y_p_p)\n",
        "    print('real ', y_gr)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "con1_before max [None, 28, 28, 32]\n",
            "con1_after max [None, 14, 14, 32]\n",
            "con2_before max [None, 14, 14, 64]\n",
            "con2_after max [None, 7, 7, 64]\n",
            "started\n",
            "iter:  0 train_cost:  33300.164 train_acc:  0.15625 test_cost:  35811.4 test_acc:  0.125\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uhbgp1CwE_4t",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 247
        },
        "outputId": "8830d9b8-2e33-4cfc-ab20-aa5d29aca761"
      },
      "source": [
        "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
        "!unzip ngrok-stable-linux-amd64.zip"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-06-05 19:52:13--  https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
            "Resolving bin.equinox.io (bin.equinox.io)... 52.4.95.48, 52.45.111.123, 52.207.111.186, ...\n",
            "Connecting to bin.equinox.io (bin.equinox.io)|52.4.95.48|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 16648024 (16M) [application/octet-stream]\n",
            "Saving to: ‘ngrok-stable-linux-amd64.zip’\n",
            "\n",
            "ngrok-stable-linux- 100%[===================>]  15.88M  39.3MB/s    in 0.4s    \n",
            "\n",
            "2019-06-05 19:52:14 (39.3 MB/s) - ‘ngrok-stable-linux-amd64.zip’ saved [16648024/16648024]\n",
            "\n",
            "Archive:  ngrok-stable-linux-amd64.zip\n",
            "  inflating: ngrok                   \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yipQU39JOccV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!rm -rf log"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "96bJem1FFBXx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LOG_DIR = './log'\n",
        "get_ipython().system_raw(\n",
        "    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'\n",
        "    .format(LOG_DIR)\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "faCuFGDZFGCj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "get_ipython().system_raw('./ngrok http 6006 &')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LdPYibnJFI1H",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2c415d80-5170-44c0-8708-4afdd02662af"
      },
      "source": [
        "! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n",
        "    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\""
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "https://3e0137d5.ngrok.io\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}