edx_tf/ML0120EN-1.4-Review-LogisticRegressionwithTensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.skills.network/\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DL0120ENedX/labs/Template%20for%20Instructional%20Hands-on%20Labs/images/IDSNlogo.png\" width=\"400px\" align=\"center\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "Logistic Regression is one of most important techniques in data science. It is usually used to solve the classic classification problem.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "<font size = 3><strong>This lesson covers the following concepts of Logistics Regression:</strong></font>\n",
    "<br>\n",
    "<h2>Table of Contents</h2>\n",
    "\n",
    "<ol>\n",
    "    <li><a href=\"#ref1\">Linear Regression vs Logistic Regression</a></li>\n",
    "    <li><a href=\"#ref2\">Utilizing Logistic Regression in TensorFlow</a></li>\n",
    "    <li><a href=\"#ref3\">Training</a></li>\n",
    "</ol>    \n",
    "</div>\n",
    "<p></p>\n",
    "<br>\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref1\"></a>\n",
    "<h2>What is different between Linear and Logistic Regression?</h2>\n",
    "\n",
    "While Linear Regression is suited for estimating continuous values (e.g. estimating house price), it is not the best tool for predicting the class in which an observed data point belongs. In order to provide estimate for classification, we need some sort of guidance on what would be the <b>most probable class</b> for that data point. For this, we use <b>Logistic Regression</b>.\n",
    "\n",
    "<div class=\"alert alert-success alertsuccess\" style=\"margin-top: 20px\">\n",
    "<font size=\"3\"><strong>Recall linear regression:</strong></font>\n",
    "<br>\n",
    "<br>\n",
    "Linear regression finds a function that relates a continuous dependent variable, <i>y</i>, to some predictors (independent variables <i>x1</i>, <i>x2</i>, etc.). Simple linear regression assumes a function of the form:\n",
    "<br><br>\n",
    "$$\n",
    "y = w0 + w1 \\times x1 + w2 \\times x2 + \\cdots\n",
    "$$\n",
    "<br>\n",
    "and finds the values of <i>w0</i>, <i>w1</i>, <i>w2</i>, etc. The term <i>w0</i> is the \"intercept\" or \"constant term\" (it's shown as <i>b</i> in the formula below):\n",
    "<br><br>\n",
    "$$\n",
    "Y = W X + b\n",
    "$$\n",
    "<p></p>\n",
    "\n",
    "</div>\n",
    "\n",
    "Logistic Regression is a variation of Linear Regression, useful when the observed dependent variable, <i>y</i>, is categorical. It produces a formula that predicts the probability of the class label as a function of the independent variables.\n",
    "\n",
    "Despite the name logistic <i>regression</i>, it is actually a <b>probabilistic classification</b> model. Logistic regression fits a special s-shaped curve by taking the linear regression and transforming the numeric estimate into a probability with the following function:\n",
    "\n",
    "$$\n",
    "ProbabilityOfaClass = \\theta(y) = \\frac{e^y}{1 + e^y} = exp(y) / (1 + exp(y)) = p \n",
    "$$\n",
    "\n",
    "which produces p-values between 0 (as y approaches minus infinity $-\\infty$) and 1 (as y approaches plus infinity $+\\infty$). This now becomes a special kind of non-linear regression.\n",
    "\n",
    "In this equation, <i>y</i> is the regression result (the sum of the variables weighted by the coefficients), <code>exp</code> is the exponential function and $\\theta(y)$ is the <a href=\"http://en.wikipedia.org/wiki/Logistic_function\">logistic function</a>, also called logistic curve. It is a common \"S\" shape (sigmoid curve), and was first developed for modeling population growth.\n",
    "\n",
    "You might also have seen this function before, in another configuration:\n",
    "\n",
    "$$\n",
    "ProbabilityOfaClass = \\theta(y) = \\frac{1}{1+e^{-y}}\n",
    "$$\n",
    "\n",
    "So, briefly, Logistic Regression passes the input through the logistic/sigmoid function but then treats the result as a probability:\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/kgv9alcghmjcv97op4d6onkyxevk23b1.png\" width=\"400\" align=\"center\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref2\"></a>\n",
    "<h2>Utilizing Logistic Regression in TensorFlow</h2>\n",
    "\n",
    "\n",
    "We begin by installing TensorFlow version 2.2.0 and its required prerequistes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install grpcio==1.24.3\n",
    "!pip install tensorflow==2.2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Restart kernel for latest version of TensorFlow to be activated**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For us to utilize Logistic Regression in TensorFlow, we first need to import the required libraries. To do so, you can run the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that running the following cell outputs version *2.2.0*. If the installation was successful and you still don't see the expected version number, restart the kernel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.0\n"
     ]
    }
   ],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will load the dataset we are going to use. In this case, we are utilizing the <code>iris</code> dataset, which is inbuilt -- so there's no need to do any preprocessing and we can jump right into manipulating it. We separate the dataset into <i>xs</i> and <i>ys</i>, and then into training <i>xs</i> and <i>ys</i> and testing <i>xs</i> and <i>ys</i>, (pseudo)randomly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Understanding the Data</h3>\n",
    "\n",
    "<h4><code>Iris Dataset</code>:</h4>\n",
    "This dataset was introduced by British Statistician and Biologist Ronald Fisher, it consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). In total it has 150 records under five attributes - petal length, petal width, sepal length, sepal width and species. <a href=\"https://archive.ics.uci.edu/ml/datasets/iris\">Dataset source</a>\n",
    "\n",
    "Attributes\n",
    "Independent Variable\n",
    "<ul>\n",
    "    <li>petal length</li>\n",
    "    <li>petal width</li>\n",
    "    <li>sepal length</li>\n",
    "    <li>sepal width</li>\n",
    "</ul>\n",
    "Dependent Variable\n",
    "<ul>    \n",
    "    <li>Species\n",
    "        <ul>\n",
    "            <li>Iris setosa</li>\n",
    "            <li>Iris virginica</li>\n",
    "            <li>Iris versicolor</li>\n",
    "        </ul>\n",
    "    </li>\n",
    "</ul>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = load_iris()\n",
    "iris_X, iris_y = iris.data[:-1,:], iris.target[:-1]\n",
    "iris_y= pd.get_dummies(iris_y).values\n",
    "trainX, testX, trainY, testY = train_test_split(iris_X, iris_y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define x and y. These variables will hold our iris data (both the features and label matrices) We also need to give them shapes which correspond to the shape of our data. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numFeatures is :  4\n",
      "numLabels is :  3\n"
     ]
    }
   ],
   "source": [
    "# numFeatures is the number of features in our input data.\n",
    "# In the iris dataset, this number is '4'.\n",
    "numFeatures = trainX.shape[1]\n",
    "print('numFeatures is : ', numFeatures )\n",
    "# numLabels is the number of classes our data points can be in.\n",
    "# In the iris dataset, this number is '3'.\n",
    "numLabels = trainY.shape[1]\n",
    "print('numLabels is : ', numLabels )\n",
    "\n",
    "X = tf.Variable( np.identity(numFeatures), tf.TensorShape(numFeatures),dtype='float32') # Iris has 4 features, so X is a tensor to hold our data.\n",
    "yGold = tf.Variable(np.array([1,1,1]),shape=tf.TensorShape(numLabels),dtype='float32') # This will be our correct answers matrix for 3 classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Set model weights and bias</h3>\n",
    "\n",
    "Much like Linear Regression, we need a shared variable weight matrix for Logistic Regression. We initialize both <code>W</code> and <code>b</code> as tensors full of zeros. Since we are going to learn <code>W</code> and <code>b</code>, their initial value does not matter too much. These variables are the objects which define the structure of our regression model, and we can save them after they have been trained so we can reuse them later.\n",
    "\n",
    "We define two TensorFlow variables as our parameters. These variables will hold the weights and biases of our logistic regression and they will be continually updated during training. \n",
    "\n",
    "Notice that <code>W</code> has a shape of [4, 3] because we want to multiply the 4-dimensional input vectors by it to produce 3-dimensional vectors of evidence for the difference classes. <code>b</code> has a shape of [3] so we can add it to the output. TensorFlow variables need to be initialized with values, e.g. with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.zeros([4, 3]))  # 4-dimensional input and  3 classes\n",
    "b = tf.Variable(tf.zeros([3])) # 3-dimensional output [0,0,1],[0,1,0],[1,0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Randomly sample from a normal distribution with standard deviation .01\n",
    "\n",
    "weights = tf.Variable(tf.random.normal([numFeatures,numLabels],\n",
    "                                       mean=0.,\n",
    "                                       stddev=0.01,\n",
    "                                       name=\"weights\"),dtype='float32')\n",
    "\n",
    "\n",
    "bias = tf.Variable(tf.random.normal([1,numLabels],\n",
    "                                    mean=0.,\n",
    "                                    stddev=0.01,\n",
    "                                    name=\"bias\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Logistic Regression model</h3>\n",
    "\n",
    "We now define our operations in order to properly run the Logistic Regression. Logistic regression is typically thought of as a single equation:\n",
    "\n",
    "$$\n",
    "ŷ =sigmoid(WX+b)\n",
    "$$\n",
    "\n",
    "However, for the sake of clarity, we can have it broken into its three main components: \n",
    "- a weight times features matrix multiplication operation, \n",
    "- a summation of the weighted features and a bias term, \n",
    "- and finally the application of a sigmoid function. \n",
    "\n",
    "As such, you will find these components defined as three separate operations below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Three-component breakdown of the Logistic Regression equation.\n",
    "# Note that these feed into each other.\n",
    "def logistic_regression(x):\n",
    "    apply_weights_OP = tf.matmul(X, weights, name=\"apply_weights\")\n",
    "    add_bias_OP = tf.add(apply_weights_OP, bias, name=\"add_bias\") \n",
    "    activation_OP = tf.nn.sigmoid(add_bias_OP, name=\"activation\")\n",
    "    return activation_OP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen before, the function we are going to use is the <i>logistic function</i> $(\\frac{1}{1+e^{-Wx}})$, which is fed the input data after applying weights and bias. In TensorFlow, this function is implemented as the <code>nn.sigmoid</code> function. Effectively, this fits the weighted input with bias into a 0-100 percent curve, which is the probability function we want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref3\"></a>\n",
    "<h2>Training</h2>\n",
    "\n",
    "The learning algorithm is how we search for the best weight vector (${\\bf w}$). This search is an optimization problem looking for the hypothesis that optimizes an error/cost measure.\n",
    "\n",
    "<b>What tell us our model is bad?</b>  \n",
    "The Cost or Loss of the model, so what we want is to minimize that. \n",
    "\n",
    "<h3>Cost function</h3>\n",
    "Before defining our cost function, we need to define how long we are going to train and how should we define the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of Epochs in our training\n",
    "numEpochs = 700\n",
    "\n",
    "# Defining our learning rate iterations (decay)\n",
    "learningRate = tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=0.0008,\n",
    "                                          decay_steps=trainX.shape[0],\n",
    "                                          decay_rate= 0.95,\n",
    "                                          staircase=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>What is the cost function in our model?</b>  \n",
    "The cost function we are going to utilize is the Squared Mean Error loss function.\n",
    "\n",
    "<b>How to minimize the cost function?</b>   \n",
    "We can't use <b>least-squares linear regression</b> here, so we will use <a href=\"http://en.wikipedia.org/wiki/Gradient_descent\">gradient descent</a> instead. Specifically, we will use batch gradient descent which calculates the gradient from all data points in the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Defining our cost function - Squared Mean Error\n",
    "loss_object = tf.keras.losses.MeanSquaredLogarithmicError()\n",
    "optimizer = tf.keras.optimizers.SGD(learningRate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want some additional operations to keep track of our model's efficiency over time. We can do this like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Accuracy metric.\n",
    "def accuracy(y_pred, y_true):\n",
    "# Predicted class is the index of the highest score in prediction vector (i.e. argmax).\n",
    "    print('y_pred : ',y_pred)\n",
    "    print('y_true : ',y_true)\n",
    "    correct_prediction = tf.equal(tf.argmax(y_pred, -1), tf.argmax(y_true, -1))\n",
    "\n",
    "    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we first wrap computation inside a GradientTape for automatic differentiation. Then we compute gradients and update W and b."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimization process. \n",
    "\n",
    "def run_optimization(x, y):\n",
    "    with tf.GradientTape() as g:\n",
    "        pred = logistic_regression(x)\n",
    "        loss = loss_object(pred, y)\n",
    "    gradients = g.gradient(loss, [weights, bias])\n",
    "    optimizer.apply_gradients(zip(gradients, [weights, bias]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we move on to actually running our operations. We will start with the operations involved in the prediction phase (i.e. the logistic regression itself).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define and run the actual training loop, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_pred :  tf.Tensor(\n",
      "[[0.4980757  0.50024974 0.50098586]\n",
      " [0.50015104 0.49280357 0.50036424]\n",
      " [0.4993304  0.5014533  0.4978648 ]\n",
      " [0.5003031  0.50101405 0.49749905]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 0, training accuracy 0, loss 0.083081, change in loss 0.083081\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4981558  0.5003298  0.5010659 ]\n",
      " [0.5002311  0.492884   0.5004443 ]\n",
      " [0.49941045 0.5015333  0.49794498]\n",
      " [0.5003831  0.50109416 0.4975792 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 10, training accuracy 0, loss 0.0830502, change in loss 3.08007e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49823588 0.5004099  0.5011459 ]\n",
      " [0.500311   0.49296445 0.5005243 ]\n",
      " [0.49949047 0.5016133  0.49802515]\n",
      " [0.500463   0.50117415 0.49765936]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 20, training accuracy 0, loss 0.0830194, change in loss 3.07783e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49831593 0.50048995 0.5012259 ]\n",
      " [0.500391   0.49304485 0.5006043 ]\n",
      " [0.49957046 0.50169337 0.4981053 ]\n",
      " [0.500543   0.5012542  0.4977395 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 30, training accuracy 0, loss 0.0829886, change in loss 3.07783e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49839598 0.50057    0.5013058 ]\n",
      " [0.50047094 0.49312523 0.50068426]\n",
      " [0.49965045 0.50177336 0.4981854 ]\n",
      " [0.5006229  0.50133425 0.49781966]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 40, training accuracy 0, loss 0.0829579, change in loss 3.07485e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.498476   0.50065005 0.50138575]\n",
      " [0.50055087 0.4932056  0.50076425]\n",
      " [0.4997304  0.5018533  0.49826548]\n",
      " [0.50070286 0.5014142  0.49789974]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 50, training accuracy 0, loss 0.0829272, change in loss 3.07187e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.498556   0.50073004 0.5014657 ]\n",
      " [0.50063074 0.49328595 0.5008442 ]\n",
      " [0.49981034 0.5019333  0.4983455 ]\n",
      " [0.5007828  0.50149417 0.49797985]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 60, training accuracy 0, loss 0.0828964, change in loss 3.07411e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49863598 0.50081    0.50154555]\n",
      " [0.50071067 0.49336627 0.5009241 ]\n",
      " [0.49989027 0.5020132  0.49842557]\n",
      " [0.5008626  0.5015741  0.49805993]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 70, training accuracy 0, loss 0.0828657, change in loss 3.07038e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49871594 0.50089    0.5016254 ]\n",
      " [0.50079054 0.4934466  0.50100404]\n",
      " [0.49997017 0.5020931  0.4985056 ]\n",
      " [0.50094247 0.501654   0.4981399 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 80, training accuracy 0, loss 0.082835, change in loss 3.06889e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49879587 0.50096995 0.5017053 ]\n",
      " [0.50087035 0.49352688 0.5010839 ]\n",
      " [0.50005007 0.502173   0.4985856 ]\n",
      " [0.5010223  0.501734   0.49821994]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 90, training accuracy 0, loss 0.0828044, change in loss 3.0674e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.498875   0.5010491  0.5017843 ]\n",
      " [0.5009494  0.49360636 0.501163  ]\n",
      " [0.5001291  0.50225204 0.4986648 ]\n",
      " [0.5011013  0.50181305 0.49829915]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 100, training accuracy 0, loss 0.082774, change in loss 3.03611e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4989509  0.50112504 0.5018602 ]\n",
      " [0.5010252  0.4936826  0.5012388 ]\n",
      " [0.500205   0.5023279  0.49874076]\n",
      " [0.50117713 0.50188893 0.49837512]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 110, training accuracy 0, loss 0.0827449, change in loss 2.91318e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4990268  0.5012009  0.50193596]\n",
      " [0.501101   0.49375883 0.5013147 ]\n",
      " [0.5002808  0.50240374 0.49881673]\n",
      " [0.50125295 0.5019648  0.49845114]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 120, training accuracy 0, loss 0.0827158, change in loss 2.90945e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49910268 0.5012768  0.5020118 ]\n",
      " [0.5011768  0.49383503 0.5013905 ]\n",
      " [0.5003566  0.5024796  0.4988926 ]\n",
      " [0.5013287  0.5020407  0.4985271 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 130, training accuracy 0, loss 0.0826867, change in loss 2.90871e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49917853 0.50135267 0.50208753]\n",
      " [0.50125253 0.4939112  0.50146633]\n",
      " [0.50043243 0.50255543 0.4989686 ]\n",
      " [0.50140446 0.5021165  0.49860296]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 140, training accuracy 0, loss 0.0826576, change in loss 2.90871e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49925438 0.5014285  0.50216335]\n",
      " [0.5013283  0.49398738 0.5015421 ]\n",
      " [0.50050825 0.5026312  0.49904445]\n",
      " [0.5014802  0.5021923  0.49867895]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 150, training accuracy 0, loss 0.0826285, change in loss 2.90573e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49933022 0.50150436 0.50223905]\n",
      " [0.50140405 0.49406356 0.5016179 ]\n",
      " [0.500584   0.50270694 0.49912035]\n",
      " [0.5015559  0.50226814 0.49875486]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 160, training accuracy 0, loss 0.0825995, change in loss 2.90647e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.499406   0.5015802  0.5023148 ]\n",
      " [0.50147974 0.4941397  0.50169367]\n",
      " [0.50065976 0.5027827  0.49919626]\n",
      " [0.5016316  0.50234383 0.49883068]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 170, training accuracy 0, loss 0.0825704, change in loss 2.90349e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4994818  0.50165594 0.5023905 ]\n",
      " [0.50155544 0.49421582 0.50176936]\n",
      " [0.50073546 0.50285846 0.4992721 ]\n",
      " [0.5017073  0.50241965 0.4989066 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 180, training accuracy 0, loss 0.0825414, change in loss 2.90275e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49955758 0.50173175 0.5024662 ]\n",
      " [0.50163114 0.4942919  0.5018451 ]\n",
      " [0.5008112  0.5029342  0.49934795]\n",
      " [0.501783   0.5024954  0.4989825 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 190, training accuracy 0, loss 0.0825124, change in loss 2.90126e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4996322  0.5018064  0.5025407 ]\n",
      " [0.50170565 0.49436685 0.5019197 ]\n",
      " [0.5008858  0.5030088  0.4994226 ]\n",
      " [0.5018575  0.50257    0.49905717]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 200, training accuracy 0, loss 0.0824838, change in loss 2.85655e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49970415 0.5018783  0.5026126 ]\n",
      " [0.5017775  0.49443913 0.50199157]\n",
      " [0.50095767 0.50308067 0.49949458]\n",
      " [0.50192934 0.5026419  0.49912915]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 210, training accuracy 0, loss 0.0824563, change in loss 2.75224e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4997761  0.50195026 0.5026845 ]\n",
      " [0.50184935 0.4945114  0.50206345]\n",
      " [0.50102955 0.50315255 0.49956658]\n",
      " [0.5020012  0.5027138  0.49920115]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 220, training accuracy 0, loss 0.0824288, change in loss 2.75373e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.499848   0.5020222  0.5027563 ]\n",
      " [0.5019212  0.49458364 0.50213534]\n",
      " [0.50110143 0.50322443 0.4996386 ]\n",
      " [0.50207305 0.5027857  0.49927318]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 230, training accuracy 0, loss 0.0824013, change in loss 2.75075e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.49991992 0.5020941  0.5028281 ]\n",
      " [0.501993   0.49465585 0.50220716]\n",
      " [0.50117326 0.50329626 0.49971056]\n",
      " [0.5021449  0.50285757 0.49934515]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 240, training accuracy 0, loss 0.0823738, change in loss 2.74926e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.4999918  0.502166   0.50289994]\n",
      " [0.5020648  0.49472806 0.50227904]\n",
      " [0.5012451  0.50336814 0.49978253]\n",
      " [0.50221664 0.50292945 0.49941716]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 250, training accuracy 0, loss 0.0823463, change in loss 2.75001e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5000637  0.50223786 0.50297177]\n",
      " [0.5021366  0.49480027 0.50235087]\n",
      " [0.5013169  0.50343996 0.49985448]\n",
      " [0.5022884  0.5030013  0.49948907]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 260, training accuracy 0, loss 0.0823188, change in loss 2.74703e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50013554 0.50230974 0.50304353]\n",
      " [0.50220835 0.49487242 0.5024227 ]\n",
      " [0.5013887  0.5035118  0.49992633]\n",
      " [0.50236017 0.5030731  0.49956098]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 270, training accuracy 0, loss 0.0822913, change in loss 2.74703e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5002074  0.50238156 0.5031153 ]\n",
      " [0.5022801  0.4949446  0.50249445]\n",
      " [0.5014605  0.50358355 0.49999827]\n",
      " [0.5024319  0.5031449  0.49963292]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 280, training accuracy 0, loss 0.0822639, change in loss 2.74628e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50027925 0.50245345 0.50318706]\n",
      " [0.5023519  0.49501675 0.5025662 ]\n",
      " [0.5015323  0.5036553  0.50007015]\n",
      " [0.50250363 0.50321674 0.49970484]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 290, training accuracy 0, loss 0.0822364, change in loss 2.7433e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5003496  0.5025238  0.50325733]\n",
      " [0.50242215 0.49508744 0.50263655]\n",
      " [0.50160265 0.50372565 0.5001406 ]\n",
      " [0.50257397 0.5032871  0.49977526]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 300, training accuracy 0, loss 0.0822096, change in loss 2.68817e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5004178  0.502592   0.50332546]\n",
      " [0.5024903  0.49515593 0.50270474]\n",
      " [0.5016708  0.50379384 0.50020885]\n",
      " [0.50264204 0.50335526 0.49984354]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 310, training accuracy 0, loss 0.0821835, change in loss 2.60621e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.500486   0.5026602  0.5033936 ]\n",
      " [0.50255835 0.49522445 0.50277287]\n",
      " [0.5017389  0.50386196 0.5002771 ]\n",
      " [0.50271016 0.50342345 0.49991181]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 320, training accuracy 0, loss 0.0821575, change in loss 2.60249e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5005542  0.5027284  0.5034617 ]\n",
      " [0.5026265  0.49529293 0.502841  ]\n",
      " [0.50180703 0.5039301  0.50034535]\n",
      " [0.5027783  0.5034916  0.4999801 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 330, training accuracy 0, loss 0.0821315, change in loss 2.60249e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50062233 0.5027966  0.5035298 ]\n",
      " [0.50269455 0.4953614  0.5029091 ]\n",
      " [0.50187516 0.5039982  0.5004136 ]\n",
      " [0.5028463  0.5035597  0.5000483 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 340, training accuracy 0, loss 0.0821054, change in loss 2.60249e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5006905  0.5028647  0.50359786]\n",
      " [0.5027626  0.49542984 0.50297725]\n",
      " [0.5019433  0.50406635 0.5004818 ]\n",
      " [0.50291437 0.50362784 0.5001165 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 350, training accuracy 0, loss 0.0820794, change in loss 2.601e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50075865 0.50293285 0.5036659 ]\n",
      " [0.5028307  0.4954983  0.5030453 ]\n",
      " [0.50201136 0.5041344  0.50055   ]\n",
      " [0.50298244 0.50369596 0.5001847 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 360, training accuracy 0, loss 0.0820534, change in loss 2.59876e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5008268  0.503001   0.503734  ]\n",
      " [0.5028987  0.49556673 0.5031134 ]\n",
      " [0.5020794  0.5042025  0.50061816]\n",
      " [0.50305045 0.50376403 0.5002529 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 370, training accuracy 0, loss 0.0820274, change in loss 2.59951e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5008949  0.5030691  0.503802  ]\n",
      " [0.50296676 0.49563515 0.50318146]\n",
      " [0.5021475  0.50427055 0.50068635]\n",
      " [0.50311846 0.50383216 0.5003211 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 380, training accuracy 0, loss 0.0820015, change in loss 2.59727e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.500963   0.50313723 0.50387007]\n",
      " [0.5030348  0.49570355 0.5032495 ]\n",
      " [0.5022155  0.5043386  0.5007545 ]\n",
      " [0.5031865  0.5039002  0.5003893 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 390, training accuracy 0, loss 0.0819755, change in loss 2.59578e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5010294  0.5032036  0.50393635]\n",
      " [0.50310105 0.49577022 0.5033158 ]\n",
      " [0.50228184 0.50440496 0.50082093]\n",
      " [0.5032528  0.50396657 0.50045574]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 400, training accuracy 0, loss 0.0819502, change in loss 2.53096e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50109404 0.50326824 0.50400096]\n",
      " [0.5031656  0.49583519 0.5033805 ]\n",
      " [0.50234646 0.5044696  0.5008856 ]\n",
      " [0.50331736 0.5040312  0.50052047]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 410, training accuracy 0, loss 0.0819255, change in loss 2.46465e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5011587  0.5033329  0.5040655 ]\n",
      " [0.5032302  0.49590012 0.503445  ]\n",
      " [0.50241107 0.5045342  0.50095034]\n",
      " [0.5033819  0.5040958  0.50058514]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 420, training accuracy 0, loss 0.0819009, change in loss 2.46316e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5012233  0.5033976  0.5041301 ]\n",
      " [0.50329477 0.49596506 0.50350964]\n",
      " [0.5024757  0.5045988  0.50101507]\n",
      " [0.50344646 0.5041604  0.50064987]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 430, training accuracy 0, loss 0.0818763, change in loss 2.46242e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50128794 0.5034622  0.5041947 ]\n",
      " [0.5033593  0.49602997 0.50357425]\n",
      " [0.50254023 0.50466335 0.50107974]\n",
      " [0.503511   0.504225   0.5007146 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 440, training accuracy 0, loss 0.0818517, change in loss 2.46093e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50135255 0.5035268  0.50425917]\n",
      " [0.5034238  0.49609488 0.5036388 ]\n",
      " [0.50260484 0.5047279  0.50114435]\n",
      " [0.50357556 0.5042896  0.5007793 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 450, training accuracy 0, loss 0.0818271, change in loss 2.46093e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50141716 0.5035914  0.5043237 ]\n",
      " [0.50348836 0.49615976 0.50370336]\n",
      " [0.5026694  0.50479245 0.501209  ]\n",
      " [0.50364006 0.5043542  0.50084394]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 460, training accuracy 0, loss 0.0818025, change in loss 2.46018e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5014818  0.503656   0.5043882 ]\n",
      " [0.50355285 0.49622464 0.5037679 ]\n",
      " [0.50273395 0.504857   0.50127363]\n",
      " [0.50370455 0.50441873 0.5009086 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 470, training accuracy 0, loss 0.0817779, change in loss 2.45869e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5015463  0.5037206  0.50445276]\n",
      " [0.50361735 0.49628952 0.50383246]\n",
      " [0.50279844 0.50492156 0.5013383 ]\n",
      " [0.50376904 0.5044833  0.5009733 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 480, training accuracy 0, loss 0.0817533, change in loss 2.4572e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50161093 0.50378513 0.50451726]\n",
      " [0.50368184 0.49635437 0.50389695]\n",
      " [0.502863   0.50498605 0.5014029 ]\n",
      " [0.50383353 0.50454783 0.5010379 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 490, training accuracy 0, loss 0.0817287, change in loss 2.4572e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5016735  0.5038478  0.5045798 ]\n",
      " [0.50374436 0.49641728 0.50395954]\n",
      " [0.5029255  0.50504863 0.50146556]\n",
      " [0.50389606 0.5046104  0.5011006 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 500, training accuracy 0, loss 0.0817049, change in loss 2.38195e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50173485 0.5039091  0.50464106]\n",
      " [0.50380564 0.49647886 0.5040208 ]\n",
      " [0.5029868  0.5051099  0.50152695]\n",
      " [0.5039573  0.50467175 0.50116193]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 510, training accuracy 0, loss 0.0816816, change in loss 2.33352e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5017961  0.5039704  0.50470227]\n",
      " [0.50386685 0.49654046 0.504082  ]\n",
      " [0.50304806 0.5051712  0.5015883 ]\n",
      " [0.50401855 0.504733   0.5012233 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 520, training accuracy 0, loss 0.0816583, change in loss 2.33129e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50185746 0.5040317  0.5047635 ]\n",
      " [0.50392807 0.49660203 0.5041433 ]\n",
      " [0.5031093  0.5052324  0.5016497 ]\n",
      " [0.5040797  0.50479424 0.50128466]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 530, training accuracy 0, loss 0.081635, change in loss 2.32905e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50191873 0.504093   0.5048247 ]\n",
      " [0.5039892  0.49666357 0.5042045 ]\n",
      " [0.50317055 0.50529367 0.50171095]\n",
      " [0.5041409  0.50485545 0.501346  ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 540, training accuracy 0, loss 0.0816117, change in loss 2.32905e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50198    0.50415426 0.5048859 ]\n",
      " [0.50405043 0.4967251  0.5042657 ]\n",
      " [0.50323176 0.5053549  0.5017723 ]\n",
      " [0.5042021  0.5049167  0.5014073 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 550, training accuracy 0, loss 0.0815884, change in loss 2.33054e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5020412  0.5042155  0.50494707]\n",
      " [0.5041116  0.49678665 0.50432694]\n",
      " [0.503293   0.5054161  0.50183356]\n",
      " [0.5042633  0.50497794 0.50146866]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 560, training accuracy 0, loss 0.0815651, change in loss 2.32682e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5021025  0.50427675 0.5050083 ]\n",
      " [0.5041728  0.49684817 0.50438815]\n",
      " [0.50335413 0.5054773  0.5018949 ]\n",
      " [0.50432444 0.5050392  0.50152993]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 570, training accuracy 0, loss 0.0815418, change in loss 2.32831e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5021637  0.50433797 0.50506943]\n",
      " [0.50423396 0.49690968 0.50444937]\n",
      " [0.50341535 0.50553846 0.50195616]\n",
      " [0.5043856  0.50510037 0.50159127]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 580, training accuracy 0, loss 0.0815186, change in loss 2.32607e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5022249  0.5043992  0.5051306 ]\n",
      " [0.50429505 0.4969712  0.5045105 ]\n",
      " [0.5034765  0.5055996  0.50201744]\n",
      " [0.50444674 0.5051616  0.50165254]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 590, training accuracy 0, loss 0.0814953, change in loss 2.32458e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.502284   0.50445825 0.5051896 ]\n",
      " [0.50435406 0.49703053 0.50456953]\n",
      " [0.5035355  0.5056587  0.50207657]\n",
      " [0.5045057  0.50522065 0.50171167]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 600, training accuracy 0, loss 0.0814729, change in loss 2.24411e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50234216 0.5045164  0.50524765]\n",
      " [0.5044121  0.49708894 0.50462765]\n",
      " [0.5035936  0.50571674 0.50213474]\n",
      " [0.50456375 0.5052787  0.5017699 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 610, training accuracy 0, loss 0.0814508, change in loss 2.20761e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5024003  0.50457454 0.5053057 ]\n",
      " [0.50447017 0.49714732 0.5046857 ]\n",
      " [0.50365174 0.50577486 0.502193  ]\n",
      " [0.5046218  0.5053369  0.5018281 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 620, training accuracy 0, loss 0.0814287, change in loss 2.20686e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5024584  0.50463265 0.50536376]\n",
      " [0.5045282  0.4972057  0.5047438 ]\n",
      " [0.5037098  0.5058329  0.5022511 ]\n",
      " [0.50467986 0.50539494 0.50188625]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 630, training accuracy 0, loss 0.0814067, change in loss 2.20537e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5025165  0.50469077 0.5054218 ]\n",
      " [0.5045863  0.4972641  0.50480187]\n",
      " [0.50376785 0.50589097 0.50230926]\n",
      " [0.50473785 0.505453   0.5019444 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 640, training accuracy 0, loss 0.0813846, change in loss 2.20537e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5025746  0.5047489  0.5054798 ]\n",
      " [0.5046443  0.49732244 0.5048599 ]\n",
      " [0.5038259  0.505949   0.5023674 ]\n",
      " [0.50479585 0.5055111  0.5020026 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 650, training accuracy 0, loss 0.0813626, change in loss 2.20314e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50263274 0.504807   0.50553787]\n",
      " [0.5047023  0.4973808  0.504918  ]\n",
      " [0.50388396 0.5060071  0.50242555]\n",
      " [0.5048539  0.50556916 0.5020607 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 660, training accuracy 0, loss 0.0813406, change in loss 2.20537e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5026908  0.50486505 0.50559586]\n",
      " [0.5047603  0.49743915 0.50497603]\n",
      " [0.503942   0.50606513 0.50248367]\n",
      " [0.5049119  0.5056272  0.5021189 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 670, training accuracy 0, loss 0.0813185, change in loss 2.20239e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.50274885 0.5049231  0.50565386]\n",
      " [0.5048183  0.49749747 0.505034  ]\n",
      " [0.504      0.5061231  0.5025418 ]\n",
      " [0.5049699  0.5056852  0.502177  ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 680, training accuracy 0, loss 0.0812965, change in loss 2.20165e-05\n",
      "y_pred :  tf.Tensor(\n",
      "[[0.5028069  0.50498116 0.50571185]\n",
      " [0.50487626 0.4975558  0.505092  ]\n",
      " [0.504058   0.5061812  0.5025999 ]\n",
      " [0.50502783 0.50574327 0.5022351 ]], shape=(4, 3), dtype=float32)\n",
      "y_true :  <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([1., 1., 1.], dtype=float32)>\n",
      "step 690, training accuracy 0, loss 0.0812745, change in loss 2.20165e-05\n",
      "final accuracy on test set: tf.Tensor(0.0, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Initialize reporting variables\n",
    "display_step = 10\n",
    "epoch_values = []\n",
    "accuracy_values = []\n",
    "loss_values = []\n",
    "loss = 0\n",
    "diff = 1\n",
    "# Training epochs\n",
    "for i in range(numEpochs):\n",
    "    if i > 1 and diff < .000001:\n",
    "        print(\"change in loss %g; convergence.\"%diff)\n",
    "        break\n",
    "    else:\n",
    "        # Run training step\n",
    "        run_optimization(X, yGold)\n",
    "        \n",
    "        # Report occasional stats\n",
    "        if i % display_step == 0:\n",
    "            # Add epoch to epoch_values\n",
    "            epoch_values.append(i)\n",
    "            \n",
    "            pred = logistic_regression(X)\n",
    "\n",
    "            newLoss = loss_object(pred, yGold)\n",
    "            # Add loss to live graphing variable\n",
    "            loss_values.append(newLoss)\n",
    "            \n",
    "            # Generate accuracy stats on test data\n",
    "            acc = accuracy(pred, yGold)\n",
    "            accuracy_values.append(acc)\n",
    "            \n",
    "    \n",
    "            # Re-assign values for variables\n",
    "            diff = abs(newLoss - loss)\n",
    "            loss = newLoss\n",
    "\n",
    "            #generate print statements\n",
    "            print(\"step %d, training accuracy %g, loss %g, change in loss %g\"%(i, acc, newLoss, diff))\n",
    "\n",
    "        \n",
    "\n",
    "          \n",
    "\n",
    "# How well do we perform on held-out test data?\n",
    "print(\"final accuracy on test set: %s\" %str(acc))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Why don't we plot the loss to see how it behaves?</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXgUVb7G8e8vCRDCFpAgELaAKLKp0LIJiVd2BkFxQ1EZGURcQMC5I44zc/VxZpwRBUEQxB1HQURQVJSgowmLLEH21bDIrhEEZN/O/aMLjUkgDUnoTuf9PE8/pKvO6f5VU+TlVFXXMeccIiIimUUEuwAREQk9CgcREclG4SAiItkoHEREJBuFg4iIZKNwEBGRbAIKBzPrZGbrzCzdzIbmsN7MbJS3frmZNcm0brCZrTKzlWY20cyiveXDzGyt136amcVmec0aZnbAzP6Y140UEZFzk2s4mFkkMAboDNQHbjez+lmadQbqeo9+wFivbzwwEPA55xoCkUBPr88soKFzrjGwHngsy2uOAD49j20SEZE8CmTk0AxId85tdM4dAyYB3bO06Q5McH7zgVgzq+KtiwJKmlkUEAPsAHDOJTvnTnht5gPVTr+Ymd0AbARWned2iYhIHkQF0CYe2Jrp+TageQBt4p1zaWb2LLAFOAwkO+eSc3iPPsC7AGZWCngUaA8EdEipYsWKrlatWoE0FRERz+LFi390zsXltC6QcLAclmW950aObcysPP5RRQKwF3jPzO50zv3nl45mjwMngLe9RU8CI5xzB8xyetlf+vXDfwiLGjVqkJaWFsCmiIjIaWb23ZnWBRIO24DqmZ5Xwzs0FECbdsAm51yGV8hUoBXwH+95b6Ar0Nb9epOn5sDNZvYMEAucMrMjzrnRmd/QOTceGA/g8/l0gygRkXwUSDgsAuqaWQKwHf8J5TuytJkOPGRmk/D/ct/nnNtpZluAFmYWg/+wUlsgDfxXQOE/fJTknDt0+oWcc21O/2xmTwAHsgaDiIgUrFzDwTl3wsweAmbiv9roNefcKjPr760fB8wAugDpwCHgHm/dAjObAnyD/9DRErz/7QOjgRLALO/w0XznXP983DYRETlPFg637Pb5fE7nHEREzo2ZLXbO+XJap29Ii4hINgoHERHJRuEgIiLZFOlwOHL8JE9MX8UPPx8JdikiIiGlSIfDsq17eWfhFjqOSGXGip3BLkdEJGQU6XBoXvsiZgxsTfUKMTzw9jcMnLiEvYeOBbssEZGgK9LhAHBJpTK8f38rhrS/lBkrdtJhRCpfrvsh2GWJiARVkQ8HgGKREQxsW5cPHryG2Jhi3PP6Ih6bupwDR0/k3llEJAwpHDJpGF+Ojwa05r6k2kxatJXOI1NZsHF3sMsSEbngFA5ZlIiK5LHOl/PefS2JMKPny/N56uPVHDl+MtiliYhcMAqHM/DVqsCMgW24s3lNXp2zid+Nms2yrXuDXZaIyAWhcDiLUiWieOqGhkzo04yDR0/SY+w8hiev49iJU8EuTUSkQCkcApB4aRwzByfS/YqqjPpvOje+OJd1u34OdlkiIgVG4RCgciWLMfy2Kxl3Z1N27TvC9S/MYVzKBk6eKvx3tRURyUrhcI46NazMzMGJ/E+9OP716Vpue+lrNv94MNhliYjkK4XDeahYugTj7mzKiNuuYN33P9N55GwmfL2ZUxpFiEiYUDicJzPjxquqkTw4kasTKvC3D1dx12sL2L73cLBLExHJM4VDHlUpV5I377maf97YiKVb9tJxRCqTF20lHGbYE5GiS+GQD8yMO5rX4LNBiTSoWpY/vb+cP7yZxg/7dStwESmcFA75qHqFGCbe24K/da3P3PQfaT8ilenLdmgUISKFjsIhn0VEGH1aJzDj4TYkVCzFwIlLeOidJew5qFuBi0jhoXAoIHXiSjOlf0v+1OkyklfvosOIFJJX7Qp2WSIiAVE4FKCoyAgeuPYSPhrQmkploun31mKGTF7KvsPHg12aiMhZKRwugHqVy/LBg9cwsG1dPly6g44jUklZnxHsskREzkjhcIEUj4pgSPtLmfZAK0pHR9H7tYX8edoKTSgkIiFJ4XCBNa4Wy8cDWtMvsTYTF26h88hU5mtCIREJMQqHIIguFsmfu2SaUGj8fJ78aBWHj2lCIREJDQqHIPLVqsCnD7ehd8uavD53M11GzWbxd3uCXZaIiMIh2GKKR/Fk94a8c29zjp04xS3jvubpGWs0LamIBJXCIUS0qlORmYMTue3qGryUupHrX5jD8m2allREgkPhEEJKl4ji6R6NeLNPM34+coIbX5zHc5qWVESCIKBwMLNOZrbOzNLNbGgO683MRnnrl5tZk0zrBpvZKjNbaWYTzSzaWz7MzNZ67aeZWay3vJmZLfUey8zsxvza2MIiyZuW9Mar4nnhv+l0HzOX1Tv2B7ssESlCcg0HM4sExgCdgfrA7WZWP0uzzkBd79EPGOv1jQcGAj7nXEMgEujp9ZkFNHTONQbWA495y1d67a8EOgEvmVnUeW9hIVWuZDGeveUKXrnbx48HjtJt9BxGffEtx09qFCEiBS+QkUMzIN05t9E5dwyYBHTP0qY7MMH5zQdizayKty4KKOn9go8BdgA455Kdc6e/ATYfqOYtP5RpeTRQpG9p2q7+xSQPSqRLoyoMn7WeHi/OY/33Pwe7LBEJc4GEQzywNdPzbd6yXNs457YDzwJbgJ3APudccg7v0Qf49PQTM2tuZquAFUD/TGFBpjb9zCzNzNIyMsL7VhTlSxVn1O1XMbZXE7bvPUzXUXMYl7KBk5qWVEQKSCDhYDksy/pbKcc2ZlYe/6giAagKlDKzO3/T0exx4ATw9i8dnVvgnGsAXA08dvo8xW9e3Lnxzjmfc84XFxcXwGYUfp0bVSF5cCLX1avEvz5dy83j5rEx40CwyxKRMBRIOGwDqmd6Xg3v0FAAbdoBm5xzGc6548BUoNXpRmbWG+gK9HI5zIjjnFsDHAQaBlBnkVCxdAnG3tmEkT2vZGPGQTqPnM2rczZxSqMIEclHgYTDIqCumSWYWXH8J5SnZ2kzHbjbu2qpBf7DRzvxH05qYWYxZmZAW2AN+K+AAh4FujnnDp1+Ie99oryfawKXAZvzspHhxszofmU8swYn0vqSijz18Wp6jp/P5h8PBrs0EQkTuYaDd7z/IWAm/l/sk51zq8ysv5n195rNADYC6cDLwANe3wXAFOAb/OcPIoDxXp/RQBlglnfZ6jhveWtgmZktBaYBDzjnfszzloahSmWjeaW3j+duuYI1u/bTeeRs3py3WaMIEckzC4f5jX0+n0tLSwt2GUG1a98Rhk5dzlfrMmhRuwLDbr6C6hVigl2WiIQwM1vsnPPltE7fkA4TlctF8/rvr+aZmxqzavt+Oj6fyn/mf0c4hL+IXHgKhzBiZtx6dXU+G5xI05rl+csHK7nr1YVs++lQ7p1FRDJROISh+NiSTOjTjH/e2IglW36i0/Ozmbhwi0YRIhIwhUOYMjPuaF6DzwYl0rhaOR6buoLery9ix97DwS5NRAoBhUOYq14hhv/8oTlPdW9A2uY9dByRyuS0rRpFiMhZKRyKgIgI466Wtfjs4UTqVy3Ln6Ysp88bi9i170iwSxOREKVwKEJqXBTDxHtb8MT19fl64246jEjh/cXbNIoQkWwUDkVMRITx+2sS+OzhRC69uAyPvLeMeyek8cN+jSJE5FcKhyKqVsVSvHtfS/7yu8uZ/e2PtBuewtRvNIoQET+FQxEWGWH0bVObTx9uQ92LyzBk8jLunbBYowgRUTgI1I4rzeRfRhEZtB+RyrQlGkWIFGUKBwF+HUXMeLgNl1QqzeB3NYoQKcoUDvIbdXIYRXywZLtGESJFjMJBssk8iqgTV4pB7y6l31uL+eFnjSJEigqFg5xRnbjSvNe/FY93uZzU9Rm0H65RhEhRoXCQs4qMMO5N1ChCpKhROEhAso4iOoxI5cOlGkWIhCuFgwQs8ygioWIpHp60lPs0ihAJSwoHOWd14kozpX8r/tylHl95owidixAJLwoHOS+REUa/xDrMGNiG2hX95yL0vQiR8KFwkDy5pJL/XETm70XoHk0ihZ/CQfLsN/doqlSaIZOX0ffNNL7XKEKk0FI4SL6pHVead+9ryV+71mfuhh9pPzyFKZovQqRQUjhIvoqMMP7QOoFPH07ksspl+ON7yzTrnEghpHCQApFQsRTv9mvJ/3mzzrUfkaK5q0UKEYWDFJiICOMeb9a5y6v4567+/euL2LH3cLBLE5FcKBykwNWqWIpJ97bgyW4NWLhpDx1HpPLuoi0aRYiEMIWDXBAREUbvVrWYOSiRBvFlefT9Fdz92kK2axQhEpIUDnJB1bgohnf6tuCp7g1Y/N1PdByRyjsLNIoQCTUKB7ngIiKMu1r6RxGNq5Xjz9NWcOerC9i651CwSxMRj8JBgqZ6hRje7tucf9zYkGVb99Hx+VQmfL2ZU6c0ihAJtoDCwcw6mdk6M0s3s6E5rDczG+WtX25mTTKtG2xmq8xspZlNNLNob/kwM1vrtZ9mZrHe8vZmttjMVnh/XpdfGyuhx8zo1bwmMwcn0rRmef724Spuf3k+3+0+GOzSRIq0XMPBzCKBMUBnoD5wu5nVz9KsM1DXe/QDxnp944GBgM851xCIBHp6fWYBDZ1zjYH1wGPe8h+B651zjYDewFvnvXVSaMTHlmRCn2Y8c1NjVu/cT8fnU3l1ziZOahQhEhSBjByaAenOuY3OuWPAJKB7ljbdgQnObz4Qa2ZVvHVRQEkziwJigB0Azrlk59wJr818oJq3fIlzboe3fBUQbWYlznP7pBAxM269ujqzBifRqk5Fnvp4Nbe+9DUbMg4EuzSRIieQcIgHtmZ6vs1blmsb59x24FlgC7AT2OecS87hPfoAn+aw/CZgiXPuaNYVZtbPzNLMLC0jIyOAzZDConK5aF7t7WP4rVeQ/sMBuoyczfjUDRpFiFxAgYSD5bAs67/SHNuYWXn8o4oEoCpQyszu/E1Hs8eBE8DbWZY3AP4N3JdTUc658c45n3POFxcXF8BmSGFiZvRoUo1ZgxNJvDSOf85YS4+x8/j2+5+DXZpIkRBIOGwDqmd6Xg3v0FAAbdoBm5xzGc6548BUoNXpRmbWG+gK9HKZLnQ3s2rANOBu59yGwDdHwk2lstGMv6spo26/ii27D/K7UXMY82U6J06eCnZpImEtkHBYBNQ1swQzK47/hPL0LG2mA3d7Vy21wH/4aCf+w0ktzCzGzAxoC6wB/xVQwKNAN+fcLxe4e1ctfQI85pybm8ftkzBgZnS7oiqzhiTRvv7FDJu5jhtenMuanfuDXZpI2Mo1HLyTxg8BM/H/Yp/snFtlZv3NrL/XbAawEUgHXgYe8PouAKYA3wArvPcb7/UZDZQBZpnZUjMb5y1/CLgE+Ku3fKmZVcr7pkphV7F0Ccb0asKLvZqwc+8Ruo2ew8jPv+XYCY0iRPKbhcNtC3w+n0tLSwt2GXIB7Tl4jCemr2L6sh1cXqUsw25uTMP4csEuS6RQMbPFzjlfTuv0DWkplCqUKs6o269i/F1N2X3gKN3HzGXYzLUcOX4y2KWJhAWFgxRqHRpUZtbgJHpcFc+YLzfQ9YU5fLPlp2CXJVLoKRyk0CsXU4xht1zBm32acejoCW4aO4+/f7yaw8c0ihA5XwoHCRtJl8Yxc3AivZrX4JU5m+g0MpX5G3cHuyyRQknhIGGlTHQx/n5DI965tznOQc/x8/nrBys5cPRE7p1F5BcKBwlLrepU5LNBbehzTQL/WfAdHUekkrpet1kRCZTCQcJWTPEo/nZ9fab0b0mJYhHc/dpC/jRlGfsOHw92aSIhT+EgYa9pzQrMGNiG+6+tw/vfbKfDiBQ+X/19sMsSCWkKBykSootF8minekx7oBXlY4rTd0IaD09awp6Dx4JdmkhIUjhIkdK4WizTH2rNoHZ1+WT5TtoPT+Hj5TsIhzsFiOQnhYMUOcWjIhjU7lI+GtCaqrEleeidJdz31mJ+2H8k2KWJhAyFgxRZl1cpy7QHWjG0cz2+Wp9Bu+EpvJe2VaMIERQOUsRFRUbQP6kOnz3chssql+F/pyyn9+uL2PbTodw7i4QxhYMIUDuuNO/2a8mT3RqQtnkPHUekMuHrzZzS1KRSRCkcRDwREUbvVrWYOSiRJjXL87cPV9Fz/Hw2ZhwIdmkiF5zCQSSL6hVimNCnGc/c3Ji1u/bTeeRsxqVs0NSkUqQoHERyYGbc6qvO50OSSLo0jn99upYeY+exdpemJpWiQeEgchaVykbz0l1NGX3HVWz/6TDXvzCHEbPWa2pSCXsKB5FcmBldG1dl1pAkfteoCiO/+JbrX5jDsq17g12aSIFROIgEqEKp4jzf8ype7e1j3+Hj3PjiXP45Y40mFZKwpHAQOUdtL7+Y5CGJ3HZ1dcanbqSzJhWSMKRwEDkPZaOL8XSPxrzTtzmnvEmFHp+2gp+P6HbgEh4UDiJ50OqSiswclEjf1glMXLiFDiNS+e9a3Q5cCj+Fg0gelSweyV+61uf9+1tRJjqKPm+kMUi3A5dCTuEgkk+uqlGejwe08d8OfMVO2g1PYfoy3Q5cCieFg0g+On078I8HtKF6hRgGTlzCvRPS2LVPtwOXwkXhIFIALqtchqn3t+Ivv7ucOek/0n54ChMXbtEoQgoNhYNIAYmMMPq2qc3MQYk0jC/HY1NXcMfLC/hu98FglyaSK4WDSAGreVEp3rm3Of/q0YiV2/fR8flUXpm9kZO6HbiEMIWDyAVgZvRsVoNZQ5JofUlF/v7JGnqMnce6XT8HuzSRHAUUDmbWyczWmVm6mQ3NYb2Z2Shv/XIza5Jp3WAzW2VmK81soplFe8uHmdlar/00M4v1ll9kZl+a2QEzG51fGyoSCiqXi+blu328cPtVbNtziK4vzGbErPUcPaFbcEhoyTUczCwSGAN0BuoDt5tZ/SzNOgN1vUc/YKzXNx4YCPiccw2BSKCn12cW0NA51xhYDzzmLT8C/BX44/lvlkjoMjOuv+K3N/LrOmoO32z5KdilifwikJFDMyDdObfROXcMmAR0z9KmOzDB+c0HYs2sircuCihpZlFADLADwDmX7Jw74bWZD1Tzlh90zs3BHxIiYev0jfxev+dqDh49wU1j5/HE9FUcPHoi984iBSyQcIgHtmZ6vs1blmsb59x24FlgC7AT2OecS87hPfoAnwZatEg4+Z/LKpE8JIm7W9Tkza8302FEKinrM4JdlhRxgYSD5bAs62UWObYxs/L4RxUJQFWglJnd+ZuOZo8DJ4C3A6glc79+ZpZmZmkZGfqHJIVb6RJRPNm9IVP6tyS6WAS9X1vIkHeX8pNuwSFBEkg4bAOqZ3peDe/QUABt2gGbnHMZzrnjwFSg1elGZtYb6Ar0cuf47SDn3HjnnM8554uLizuXriIhq2nNCnwysA0DrruE6ct20G54Ch/pFhwSBIGEwyKgrpklmFlx/CeUp2dpMx2427tqqQX+w0c78R9OamFmMWZmQFtgDfivgAIeBbo55w7l0/aIFHrRxSJ5pMNlfDSgNdXKl2SAdwuOnfsOB7s0KUIskP+RmFkX4Hn8Vxu95pz7h5n1B3DOjfN+8Y8GOgGHgHucc2le3yeB2/AfOloC9HXOHTWzdKAEcHqWlPnOuf5en81AWaA4sBfo4Jxbfab6fD6fS0tLO9dtFwl5J085Xp+7iWeT1xEVEcHQzvW4o1kNIiJyOpIrcm7MbLFzzpfjunAYriocJNxt2X2IoVOXM2/DbpolVOBfPRpRO650sMuSQu5s4aBvSIsUAjUuiuHtvs155qbGrN25n04jZzPmy3SOnzwV7NIkTCkcRAoJM+PWq6vz+ZAk2tarxLCZ6+g+ei4rt+8LdmkShhQOIoVMpbLRjL2zKePubErGgaN0HzOXp2es4fAx3YJD8o/CQaSQ6tSwMp8PSeJWXzVeSt1Ip5GpzEv/MdhlSZhQOIgUYuVKFuPpHo15597mGHDHKwv405Rl7Dt0PNilSSGncBAJA63qVOSzQYncf20d3v9mO22Hp/DJ8p368pycN4WDSJiILhbJo53qMf2ha6hcrgQPvvMN/d5arPmr5bwoHETCTIOq5fjggWv4c5d6zP42g/bDU3h7wXec0sxzcg4UDiJhKCoygn6JdZg5KJFG1crx+LSV9Bw/nw0ZB4JdmhQSCgeRMFbzolL+L8/d3Ji1u/bTeeRsRv/3W315TnKlcBAJc2bGrb7qfP5IEu0vv5hnk9dz/QtzWLZ1b7BLkxCmcBApIiqViWZMryaMv6spPx06xo0vzuXvH6/m0DHNPCfZKRxEipgODSoza0gSdzSvwStzNtFhRCqpmnlOslA4iBRBZaOL8fcbGjH5vpaUiIrg7tcWMvjdpezRzHPiUTiIFGHNEvwzzw287hI+Xr6Dts99xbQl2/TlOVE4iBR10cUiGdLhMj4e0IZaFUsx+N1l9H59EVv3aILGokzhICIAXFa5DFP6t+LJbg1YvHkPHUak8srsjZzQZa9FksJBRH4RGWH0blWLWUOSaFXnIv7+yRp6jJ3H6h37g12aXGAKBxHJpmpsSV7p7eOF269ix97DXD96Dv/+bC1HjmvOiKJC4SAiOTIzrr+iKp8PSeKmJvGM/WoDnZ7XnBFFhcJBRM4qNqY4z9x8Be/0bY7j1zkj9h7SZa/hTOEgIgFpdUlFZg5KpH+Sf86IdsNT+Hj5Dl32GqYUDiISsOhikQzt7J8zokq5kjz0zhL6vpnG9r2Hg12a5DOFg4icswZVyzHtgVb85XeXM2/DbjoMT+H1uZs4qTkjwobCQUTOS1RkBH3b1CZ5cCK+WhV48qPV9Bg7jzU7ddlrOFA4iEieVK8Qwxv3XM3Inleybc8hrn9Bl72GA4WDiOSZmdH9yni+eCSJG6/yX/ba8flU5uqy10JL4SAi+SY2pjjDbvFf9mpAr1cW8Mf3lvGT7vZa6CgcRCTftbqkIp8NSuTB/6nDB0v8l71+uHS7LnstRBQOIlIgootF8r8d6/HRgNZUrxDDw5OW8nvd7bXQUDiISIG6vEpZ3r/ff7fXNO9ury+n6m6voS6gcDCzTma2zszSzWxoDuvNzEZ565ebWZNM6wab2SozW2lmE80s2ls+zMzWeu2nmVlspj6Pea+1zsw65seGikjwZL7b6zWXXMQ/ZqzhhhfnsnL7vmCXJmeQaziYWSQwBugM1AduN7P6WZp1Bup6j37AWK9vPDAQ8DnnGgKRQE+vzyygoXOuMbAeeMzrU99r0wDoBLzo1SAihVzV2JK8fLePF3s14fv9R+k2eg7/+GQ1h46dCHZpkkUgI4dmQLpzbqNz7hgwCeiepU13YILzmw/EmlkVb10UUNLMooAYYAeAcy7ZOXd6j5gPVMv0WpOcc0edc5uAdK8GEQkDZkaXRlX4fEgSt11dg5dnb6L98FS+XPdDsEuTTAIJh3hga6bn27xlubZxzm0HngW2ADuBfc655Bzeow/w6Tm8H2bWz8zSzCwtIyMjgM0QkVBSrmQxnu7RiPf6t6Rk8UjueX0RAyYuIePno8EuTQgsHCyHZVmvR8uxjZmVxz8SSACqAqXM7M7fdDR7HDgBvH0O74dzbrxzzuec88XFxeWyCSISqq6uVYFPBrZmSPtLmblyF22f+4pJC7dwSvdpCqpAwmEbUD3T82p4h4YCaNMO2OScy3DOHQemAq1ONzKz3kBXoJf79QLoQN5PRMJIiahIBraty6eD2lCvSlmGTl1Bz5fnk/7DgWCXVmQFEg6LgLpmlmBmxfGfLJ6epc104G7vqqUW+A8f7cR/OKmFmcWYmQFtgTXgvwIKeBTo5pw7lOW1eppZCTNLwH+Se2EetlFECok6caWZdG8L/n1TI9bt+pkuI2fz/OfrOXpC92m60HINB++k8UPATPy/2Cc751aZWX8z6+81mwFsxH/y+GXgAa/vAmAK8A2wwnu/8V6f0UAZYJaZLTWzcV6fVcBkYDXwGfCgc057hkgRERFh3HZ1DT4fkkSnhpV5/vNv6TJyNgs37Ql2aUWKhcPX2X0+n0tLSwt2GSJSAL5a9wN/+WAl2346zO3NqjO00+WUiykW7LLCgpktds75clqnb0iLSEi79rJKJA9OpF9ibSanbaPt8BQ+WqbpSQuawkFEQl5M8Sj+3OVyPnzwGqqUi2bAxCX0eWMR237SfZoKisJBRAqNhvH+6Un/2rU+Czbtof1w3aepoCgcRKRQiYqM4A+tE0genEjLOv77NHUfM5fl2/YGu7SwonAQkUKpWvkYXu3tv09Txs9HuWHMXJ78aBUHjuo+TflB4SAihdYv92l6JIlezWvyxrzNtB+eQvKqXcEurdBTOIhIoVc2uhhP3dCQKf1bUa5kMfq9tZj73kpj174jwS6t0FI4iEjYaFqzPB8NaM2jneqRsj6DdsNTeGPuJk7qPk3nTOEgImGlWGQE919bh+RBSVxVI5YnPlpNj7HzWL1jf7BLK1QUDiISlmpcFMOEPs0Y2fNKtv90iOtHz+HpGWs0sVCAFA4iErbMjO5XxvP5kCRu9VXjpdSNdBihiYUCoXAQkbAXG1Ocp3s0ZvJ9LYku5p9Y6KF3vuGHn3XC+kwUDiJSZDRL+HVioeTV39P2uRTeXvCdJhbKgcJBRIqU0xMLffZwGxrFl+PxaSu5edw81u7SCevMFA4iUiTVjivN232bM/zWK9i8+xBdR83h35+t5fAxTR8DCgcRKcLMjB5NqvHFkCR6NIln7Fcb6PB8Cl/phLXCQUSkfKniPHPzFUzq14LikRH8/vQJ6/1F94S1wkFExNOi9kXMeLgNj2Q6Yf3W/KJ5wlrhICKSSYmoSAa0rcvMQYk0rl6Ov36wkpvGzWPNzqJ1wlrhICKSg4SKpfjPH5rz/G1XsmX3Ibq+MIenPy0637BWOIiInIGZccNV8XzxSBK3NK3GSykbaT88lS/Xhv8Ja4WDiEguYmOK86+b/N+wLlk8knveWMSDb3/D92F8wlrhICISoGYJFZgxsA1/7HAps9Z8T7vnUpjw9eawvCW4wkFE5BwUj4rgoevqkjwokStrxPK3D1fR48W5rPBNvFIAAAiZSURBVNy+L9il5SuFg4jIeahVsdSvtwTfe5huo+fw1MerORgmc1grHEREztPpW4J/MeRaejarwatzNtFueAozw2AOa4WDiEgelYspxj9vbMT79/vnsL7vrcX0fTON7XsPB7u086ZwEBHJJ6fnsH6scz3mpv9I++EpvJy6kRMnTwW7tHOmcBARyUfFIiO4L6kOyYMTaVH7Iv4xYw3Xj57Lki0/Bbu0c6JwEBEpANUrxPBqbx9jezVhz8Gj9Bg7j79+sJL9R44Hu7SAKBxERAqImdG5URU+H5JE75a1eHvBd7R9LoWPlu3AudD+bkRA4WBmncxsnZmlm9nQHNabmY3y1i83syaZ1g02s1VmttLMJppZtLf8Fm/5KTPzZWpf3MxeN7MVZrbMzK7Nh+0UEQmaMtHFeKJbAz58sDWVy0YzYOISer++iC27DwW7tDPKNRzMLBIYA3QG6gO3m1n9LM06A3W9Rz9grNc3HhgI+JxzDYFIoKfXZyXQA0jN8lr3AjjnGgHtgefMTCMcESn0GlUrxwcPXsP/XV+fxZv30H5ECmO+TOfYidA7YR3IL91mQLpzbqNz7hgwCeiepU13YILzmw/EmlkVb10UUNLMooAYYAeAc26Nc25dDu9XH/jCa/MDsBfw5dBORKTQiYww7rkmgS8euZbr6lVi2Mx1dBk1m/kbdwe7tN8IJBziga2Znm/zluXaxjm3HXgW2ALsBPY555Jzeb9lQHczizKzBKApUD1rIzPrZ2ZpZpaWkZERwGaIiISOyuWiGXtnU177vY8jx0/Sc/x8Hpm8jN0Hjga7NCCwcLAclmU9k5JjGzMrj39UkQBUBUqZ2Z25vN9r+MMlDXgemAdk+z66c268c87nnPPFxcXl8pIiIqHpunoXM2twEg9cW4cPl27nuudSmLRwS9BnnwskHLbx2/+5V8M7NBRAm3bAJudchnPuODAVaHW2N3POnXDODXbOXemc6w7EAt8GUKeISKFUsngkf+pUjxkPt+Gyi8swdOoKbn3pa9buCt7sc4GEwyKgrpklmFlx/CeUp2dpMx2427tqqQX+w0c78R9OamFmMWZmQFtgzdnezGtbyvu5PXDCObf63DZLRKTwufTiMrx7XwuG3dyYDRkH6DoqeLPP5RoOzrkTwEPATPy/2Cc751aZWX8z6+81mwFsBNKBl4EHvL4LgCnAN8AK7/3GA5jZjWa2DWgJfGJmM73XqgR8Y2ZrgEeBu/JjQ0VECgMz4xZfdf77yLXc1OTX2edmrf7+wtYR6l/ECITP53NpaWnBLkNEJN8t2ryHx6etYP33B2hf/2Ke6NaA+NiS+fLaZrbYOZfj1aD6/oCISAi7ulYFPhnYhqGd6zH724xfbuZ3vIBv5qdwEBEJccUiI+ifVIdZg5Noefpmfi/MYfF3BXczP4WDiEghUb1CDK/09vHSXU3Zd/g4N42dxz8+KZjrdaIK5FVFRKRAmBkdG1Sm9SUVef7z9dSoEFMg76NwEBEphEqViOLx32W9zV3+0WElERHJRuEgIiLZKBxERCQbhYOIiGSjcBARkWwUDiIiko3CQUREslE4iIhINmFxV1YzywC+y8NLVAR+zKdyCoLqyxvVlzeqL29Cub6azrkcp9IMi3DIKzNLO9Nta0OB6ssb1Zc3qi9vQr2+M9FhJRERyUbhICIi2Sgc/MYHu4BcqL68UX15o/ryJtTry5HOOYiISDYaOYiISDZhHQ5mttnMVpjZUjNL85ZVMLNZZvat92f5M/TtZGbrzCzdzIZewPqGmdlaM1tuZtPMLDbQvheovifMbLu3bKmZdTlD32B9fu9mqm2zmS0NtG8B1BdrZlO8v881ZtYyxPa/nOoLpf0vp/pCaf/Lqb6Q2f/yzDkXtg9gM1Axy7JngKHez0OBf+fQLxLYANQGigPLgPoXqL4OQJT3879zqu9MfS9QfU8Af8ylX9A+vyzrnwP+FsTP702gr/dzcSA2xPa/nOoLpf0vp/pCaf/LVl8o7X95fYT1yOEMuuP/S8X784Yc2jQD0p1zG51zx4BJXr8C55xLds6d8J7OB6pdiPfNZ0H7/E4zMwNuBSZeyPfN9P5lgUTgVQDn3DHn3F5CZP87U32hsv+d5fMLRNA+v0zrg7r/5YdwDwcHJJvZYjPr5y272Dm3E8D7s1IO/eKBrZmeb/OWXYj6MusDfHqefQuyvoe8ww6vneGwSCh8fm2A751z355H3/xQG8gAXjezJWb2ipmVInT2vzPVl1kw97+z1RcK+19un1+w9788C/dwuMY51wToDDxoZokB9rMclhXEZV1nrM/MHgdOAG+fa98Crm8sUAe4EtiJf+icVdA/P+B2zv6/toL+/KKAJsBY59xVwEH8h5ECcSE+v7PWFwL735nqC5X9L7e/32Dvf3kW1uHgnNvh/fkDMA3/cPN7M6sC4P35Qw5dtwHVMz2vBuy4QPVhZr2BrkAv5x2gDLRvQdfnnPveOXfSOXcKePkM7xvszy8K6AG8e65989E2YJtzboH3fAr+Xyahsv+dqb5Q2f9yrC+E9r+zfX6hsP/lWdiGg5mVMrMyp3/Gf6JtJTAd6O016w18mEP3RUBdM0sws+JAT69fgddnZp2AR4FuzrlD59L3AtVXJVOzG8/wvkH7/LzV7YC1zrlt59E3XzjndgFbzewyb1FbYDUhsv+dqb5Q2f/OUl9I7H9n+fuFENj/8kWwz4gX1AP/McFl3mMV8Li3/CLgC+Bb788K3vKqwIxM/bsA6/Ff9fD4BawvHf/x0qXeY1zW+s7U9wLV9xawAliO/x9clVD6/Lx1bwD9s7S/oJ+f9z5XAmneZ/UBUD5U9r+z1BcS+99Z6guJ/e9M9YXS/pfXh74hLSIi2YTtYSURETl/CgcREclG4SAiItkoHEREJBuFg4iIZKNwEBGRbBQOIiKSjcJBRESy+X9bnPX06GIxcQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot([np.mean(loss_values[i-50:i]) for i in range(len(loss_values))])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Try changing the parameters such as the length of training, and maybe some operations to see how the model behaves. Does it take much longer? How is the performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to learn more?\n",
    "\n",
    "Running deep learning programs usually needs a high performance platform. __PowerAI__ speeds up deep learning and AI. Built on IBM’s Power Systems, __PowerAI__ is a scalable software platform that accelerates deep learning and AI with blazing performance for individual users or enterprises. The __PowerAI__ platform supports popular machine learning libraries and dependencies including TensorFlow, Caffe, Torch, and Theano. You can use [PowerAI on IMB Cloud](https://cocl.us/ML0120EN_PAI).\n",
    "\n",
    "Also, you can use __Watson Studio__ to run these notebooks faster with bigger datasets.__Watson Studio__ is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, __Watson Studio__ enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of __Watson Studio__ users today with a free account at [Watson Studio](https://cocl.us/ML0120EN_DSX).This is the end of this lesson. Thank you for reading this notebook, and good luck on your studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thanks for completing this lesson!\n",
    "\n",
    "This is the end of **Logistic Regression with TensorFlow** notebook. Hopefully, now you have a deeper understanding of Logistic Regression and how its structure and flow work. Thank you for reading this notebook and good luck on your studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Created by: <a href=\"https://br.linkedin.com/in/walter-gomes-de-amorim-junior-624726121\">Saeed Aghabozorgi</a> , <a href=\"https://br.linkedin.com/in/walter-gomes-de-amorim-junior-624726121\">Walter Gomes de Amorim Junior</a> ,  Victor Barros Costa\n",
    "\n",
    "Updated to TF 2.X by  <a href=\"https://www.linkedin.com/in/samaya-madhavan\"> Samaya Madhavan </a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}