Weight_Pruning_in_Keras_with_Fashion_MNIST.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Weight Pruning in Keras with Fashion MNIST.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a7k5Nhs1Ixcy"
      },
      "source": [
        "# Weight Pruning\n",
        "\n",
        "**Overview**\n",
        "\n",
        "Magnitude-based weight pruning gradually zeroes out model weights during the training process to achieve model sparsity. Sparse models are easier to compress, and we can skip the zeroes during inference for latency improvements.\n",
        "\n",
        "This technique brings improvements via model compression. In the future, framework support for this technique will provide latency improvements. We've seen up to 6x improvements in model compression with minimal loss of accuracy.\n",
        "\n",
        "The technique is being evaluated in various speech applications, such as speech recognition and text-to-speech, and has been experimented on across various vision and translation models.\n",
        "\n",
        "In this example we will be using Fashion MNIST dataset.\n",
        "This example require Tensorflow 2.4 version or higher."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4rB3VhAMJsgv"
      },
      "source": [
        "## Initial Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dNDzooiWJxjV",
        "outputId": "b6c895a2-0730-4561-fa20-abec18560571"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fri Mar 26 13:11:16 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.56       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   66C    P8    11W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KV21lnJbJ1qH",
        "outputId": "8928d429-37f6-45c9-f1be-eb2b9d1888a7"
      },
      "source": [
        "!pip install -q tensorflow-model-optimization"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25l\r\u001b[K     |██                              | 10kB 26.3MB/s eta 0:00:01\r\u001b[K     |███▉                            | 20kB 32.9MB/s eta 0:00:01\r\u001b[K     |█████▊                          | 30kB 21.9MB/s eta 0:00:01\r\u001b[K     |███████▋                        | 40kB 25.0MB/s eta 0:00:01\r\u001b[K     |█████████▌                      | 51kB 22.1MB/s eta 0:00:01\r\u001b[K     |███████████▍                    | 61kB 24.5MB/s eta 0:00:01\r\u001b[K     |█████████████▎                  | 71kB 19.0MB/s eta 0:00:01\r\u001b[K     |███████████████▏                | 81kB 20.0MB/s eta 0:00:01\r\u001b[K     |█████████████████               | 92kB 19.2MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 102kB 19.2MB/s eta 0:00:01\r\u001b[K     |████████████████████▉           | 112kB 19.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████▊         | 122kB 19.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████▊       | 133kB 19.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▋     | 143kB 19.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▌   | 153kB 19.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▍ | 163kB 19.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 174kB 19.2MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z17MXMWfKA7U"
      },
      "source": [
        "## Helper Functions - To determine the file size"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aQzc0ZcdKF1H"
      },
      "source": [
        "def get_file_size(file_path):\n",
        "    size = os.path.getsize(file_path)\n",
        "    return size\n",
        "\n",
        "def convert_bytes(size, unit=None):\n",
        "    if unit == \"KB\":\n",
        "        return print('File Size: ' + str(round(size/1024, 3)) + 'Kilobytes')\n",
        "    elif unit == 'MB':\n",
        "        return print('File Size: ' + str(round(size/(1024*1024), 3)) + 'Megabytes')\n",
        "    else:\n",
        "        return print('File Size: ' + str(size) + 'bytes')"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "McOFIWu6Jll8"
      },
      "source": [
        "## Import the necessary modules"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gxKwKMCoIA3p"
      },
      "source": [
        "import os\n",
        "import time\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import tempfile\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sys import getsizeof\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_model_optimization as tfmot\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Conv2D, Dense, Flatten, MaxPooling2D, GlobalAvgPool2D, Dropout\n",
        "\n",
        "%load_ext tensorboard"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jDImDEBpKoh_"
      },
      "source": [
        "## Load the Fashion MNIST dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAJHZTedLLgc"
      },
      "source": [
        "The **[Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist)** dataset which contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels), as seen here:\n",
        "\n",
        "![](https://www.tensorflow.org/images/fashion-mnist-sprite.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sguh6K9CWfBO"
      },
      "source": [
        "## Baseline Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6t8m1DhyKpZa",
        "outputId": "95e33ff8-7a78-4fd0-b860-8f8d59ba2310"
      },
      "source": [
        "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
        "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()\n",
        "\n",
        "#Storing test labels\n",
        "test_labels = y_test\n",
        "\n",
        "x_train = x_train.astype(\"float32\") / 255.0\n",
        "x_train = np.reshape(x_train, (-1, 28, 28, 1))\n",
        "y_train = tf.one_hot(y_train, 10)\n",
        "\n",
        "x_test = x_test.astype(\"float32\") / 255.0\n",
        "x_test = np.reshape(x_test, (-1, 28, 28, 1))\n",
        "y_test = tf.one_hot(y_test, 10)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
            "32768/29515 [=================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
            "26427392/26421880 [==============================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
            "8192/5148 [===============================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
            "4423680/4422102 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ypSVOZOoOlCi"
      },
      "source": [
        "class_name = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot']"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "To8FXs70MQAD"
      },
      "source": [
        "#### Display the shape of the training as well testing images and labels"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VhxoNwadKMyq",
        "outputId": "c63bd33f-f40a-4e59-d063-464c1a00f3af"
      },
      "source": [
        "print(\"Training Image Shape: \",x_train.shape)\n",
        "print(\"Training Label Shape\", y_train.shape)\n",
        "print(\"Testing Image Shape: \",x_test.shape)\n",
        "print(\"Testing Label Shape\", y_test.shape)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training Image Shape:  (60000, 28, 28, 1)\n",
            "Training Label Shape (60000, 10)\n",
            "Testing Image Shape:  (10000, 28, 28, 1)\n",
            "Testing Label Shape (10000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtttX65TMblr"
      },
      "source": [
        "### Define the Hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zU2puLX0MHxF"
      },
      "source": [
        "AUTO = tf.data.AUTOTUNE\n",
        "BATCH_SIZE = 64\n",
        "EPOCHS = 10\n",
        "NUM_CLASSES=10"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZHWAHlrMh5z"
      },
      "source": [
        "### Creating the Data Pipeline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LRJJrk5fMfuo"
      },
      "source": [
        "train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "\n",
        "train_ds = (\n",
        "    train_ds\n",
        "    .shuffle(BATCH_SIZE * 100)\n",
        "    .batch(BATCH_SIZE)\n",
        ")\n",
        "\n",
        "test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n",
        "\n",
        "test_ds = (\n",
        "    test_ds\n",
        "    .batch(BATCH_SIZE)\n",
        ")"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FFKt67u-OOuZ"
      },
      "source": [
        "___Pipeline___ is ready!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9c7Zdm8HOWLX"
      },
      "source": [
        "### Visiualise the Training Images"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 591
        },
        "id": "uv3BXZ5HN-eT",
        "outputId": "bdc959c5-9bbc-4caa-94f4-d1882cc5756e"
      },
      "source": [
        "sample_images, sample_labels = next(iter(train_ds))\n",
        "plt.figure(figsize=(10, 10))\n",
        "for i, (image, label) in enumerate(zip(sample_images[:9], sample_labels[:9])):\n",
        "    ax = plt.subplot(3, 3, i + 1)\n",
        "    plt.imshow(image.numpy().squeeze())\n",
        "    plt.title(class_name[np.argmax(label.numpy().tolist())])\n",
        "    plt.axis(\"off\")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAI+CAYAAACxLHDrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xcV53///dnRs0qltx7L+m9QyCBUBJS2UAKgSSwlAAbytKW/X13l76UZWEhCwECG5aFQGghDZJACqQ3p9uOY8ctLpGLZEm22uj8/pjxrvD5XEfjSJZ9/Ho+Hn4kes/RnTuje2c+ujqfORZCEAAAQEpyw70DAAAAg40CBwAAJIcCBwAAJIcCBwAAJIcCBwAAJIcCBwAAJIcC5yWYWTCzueXeBmDgzGy5mb1uuPcD2JMM9D3GzGaWxlbsjv3aW+wzBY6Z3Wlmm82seg/Yl0vNrGBm7aV/y8zs/YO07avN7AuDsS3sm8zsRDO718xazWyTmd1jZscM934BewrOkb3DPlHgmNlMSa+SFCSdNaw783/uCyHUhxDqJZ0r6atmdsRw7xT2bWY2UtKNkr4tabSkKZI+K6lrOPdrIPjtFbvD3nyO7Gv2iQJH0sWS7pd0taRL+t9QuuLxn2Z2k5m1mdkDZjbH20ipal9lZic7t1Wb2b+Z2UozW29mV5rZiIHsXAhhgaSFkg7ot72zzOxpM2spXX3qf9sBpaylNOasUv5eSRdJ+mTpytANA7l/oJ/5khRCuCaEUAghbAsh3BpCeKJ05fHu0nG+2cyeN7PTtn+jmTWa2Q/NbK2ZvWBmXzCzfOm2OWZ2u5ltNLMNZvZTM2vydqB0fD9vZheWvj7DzB4rHe/3mtmh/cYuN7NPmdkTkjoocrAb7Owc2elxXjpeP25mT5Su/vzCzGr63f6J0vmzxsze1f9Ozex0M1tgZltK70Of2W2PeC+1LxU4Py39e6OZTdjh9gtUrMBHSXpO0hd33ICZnSrpGknnhhDudO7jyyoe+IdLmqtiVf/PA9m50qXN+ZIeLn09v3RfH5E0TtLNkm4wsyozq5R0g6RbJY2XdLmkn5rZfiGE75ce41dLV4fOHMj9A/08K6lgZj82s9PMbNQOtx8nabGksZK+KumHZmal266W1Kvi8X+EpDdIenfpNpP0r5Imq1jIT5P0mR3v3MyOlHSLpMtDCNeUrmr+SNL7JI2R9D1J1+/wp+YLJZ0uqSmE0LvrDx0YkJ2dIwM5zs+TdKqkWZIOlXSp9L/vMR+X9HpJ8yTtOCetQ8X3siYVj/f3m9k5g/aoUhRCSPqfpBMl9UgaW/p6kaSP9rv9aklX9fv6TZIW9fs6SPq0pBWSDt5h20HFF3NT8eCb0++2EyQ9n7FPl6r4RtAiqa20nW9LstLt/yTp2n7jc5JekHSyin9qWycp1+/2ayR9pt/j+cJwP+/823v/qfjCfLWk1aXj9HpJE0rH7XP9xtWWjt2Jpdu7JI3od/uFku7IuI9zJC3o9/VyFX/JWC3p5H75dyV9fofvXSzppH7f967hfs74t2/9yzpHnHHecf72fl9/VdKVpf//kaQv97tt/vb3mIx9+Kakb5T+f2ZpbMVwPzd70r994QrOJZJuDSFsKH39M+3wZyoVC4bttkqq3+H2j6hYcDyVcR/jVHyxf6R0Gb1F0h9KeZb7QwhNIYQGFd8gDpL0pdJtk1UsqCRJIYQ+SatUvCo0WdKqUrbditJtwMsWQlgYQrg0hDBV0sEqHnPfLN28rt+4raX/rZc0Q1KlpLX9zoHvqXiVUWY2wcx+XvrT1RZJ/6PiVaD+LpN0b/jrK6QzJH1s+zZL251W2qftVr38Rw0MXNY5MsDjPOv9ZrL++lhe0e//ZWbHmdkdZtZsZq0qni87bhv9JF3glObAnCfpJDNbZ2brJH1U0mFmdlgZm3qrpHPM7MMZt2+QtE3SQaWipSmE0BiKE4hfUghhvaRfS9r+J6U1Kr6wb38cpuKL+gul26aZWf+f3fTSbVKxigcGRQhhkYq/qR78EkNXqXgFZ2y/c2BkCOGg0u1fUvHYPCSEMFLS21W88tnfZZKmm9k3dtjuF/ttsymEUBtCuKb/bu7aowNevh3OkYEc51nWqvg6v930HW7/mYpXiqaFEBolXVnGtvdJSRc4Kl4eLEg6UMW5MYereGnxLyr+LXOg1kg6RdKHzWnnLl1N+YGkb5jZ9t9Yp5jZGweycTMbI+nNkp4uRddKOt3MTinNufmYim8e90p6QMWq/5NmVmnFCc9nSvp56XvXS5pdxmMD/peZ7W9mHzOzqaWvp6n4p6b7d/Z9IYS1Ks4L+7qZjTSzXGnC5UmlIQ2S2iW1mtkUSZ9wNtOm4tyEV5vZl0vZDyRdVvrt1cysrjTZsuFlP1hgF7zEOTKQ4zzLtZIuNbMDzaxW0r/scHuDpE0hhE4zO1bS217uY0ld6gXOJZL+K4SwMoSwbvs/SVdIuqicjosQwkoVi5x/MLN3O0M+peIE5ftLlyb/KGm/nWzyBCt9Do6KHVTNKk4YVghhsYqV/7dVvDp0pqQzQwjdIYTu0tenlW77jqSLS79FSNIPJR1Yupx/3UAfH1DSpuJE4gfMrEPFF+2nVCyyX8rFkqokPSNps6RfSZpUuu2zko6U1CrpJkm/8TYQQmhRcZLlaWb2+RDCw5Leo+I5u1nFc+zSXXlgwCDZ2TkyoOPcE0L4vYp/Cr5dxeP89h2GfEDS58ysTcUGlmtf3sNI3/ZJrQAAAMlI/QoOAADYB1HgAACA5FDgAACA5FDgAACA5FDgAACA5Oy0Tfr1ubfSYoVhc1vfL/e4D7HinMBw4pwA/trOzgmu4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgOTsdKkGAACAHS357yOjbP63ut2x4eGn3Nyqq/3xXV27vmP9cAUHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhy4qAACGmFVWuXno8TuPlMsPfON9hV3Yo7+WHzvGzeff0urm10/8XpR996h57tjfH9Tk32lfGNjO7SKu4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgOTQRTWUzPw8DHzmeM8bjnbzylsf3pU9AgAMg9Db4+b5pkb/G5wOo8KWLYOyL959Xnbffe7Yw6tfdPM1vXF21U/e5I6donv9HcllvEcOEq7gAACA5FDgAACA5FDgAACA5FDgAACA5FDgAACA5NBFNZTK6JbK8sErrnXz02o3uPk5b31PlNl9j7/s/QAAvAwZ7weFFn+tp6H026f/GGVrervcsWt6R7j5RxefF2VTvpzRLZUhdPn3OVi4ggMAAJJDgQMAAJJDgQMAAJJDgQMAAJLDJOOhVMZSDat+dbA79LUj/ElbV7b447/6s+9H2fLeMe7YlkKtmz/cNivKbn7av7+x4/yPDn/d5MVu/rtrT4yyqf9a3sQ0ANgj5PJ+3leIoopZM9yhK8+d4uZdR3VEWX1dpzv27BlPunlP8PdvRW93lD3aOc0dO61yo5tv/cOEKKs+fbQ7tvqmh9w8k/feuQtNO1zBAQAAyaHAAQAAyaHAAQAAyaHAAQAAyaHAAQAAyaGLahBYZZWbh554pnqWuhp/7Nc2HO/mj7VMdfPvPH5SlF1x3M/csZ19lW5+xqjHomzMEe3u2MXt8Ux6SfrHcQ+6+f2PHOvmALC3sbzfpRScLqrmK/z3ifsO+3c3//GWeVF2/drD3LFza9a7+dcXv87Nj61bGmWrevwOqEc6Zrr5+X/7pyj70VMnuGPn3OTGsgq/BAm9vc7gjK7kneAKDgAASA4FDgAASA4FDgAASA4FDgAASA4FDgAASA5dVMOg7YK4M+qhI690x362+UA3P6RpjZt3FeIf6ZOdfsfVuq5GN3+kbWaU/Xn5HHdszza/E+uuCU1uXr0+Xl+lzx0JJKaMtemwdwi9PQMeO+r0JW5+rvxOWU/va/2u1Z/cM9fNey731yE88vAXo2xx1yR37PVLDnHzmec/EWVzFHfg7ozbLSW/M7mcruTtuIIDAACSQ4EDAACSQ4EDAACSQ4EDAACSwyTjQRAK8cdy78zIJW1RdmXLFHdse6HazVds9T9WuzIX70t7ocbfj4pON9/UUxdl9bUZY9v9jx8/PWP8t3PU1Puy5/7dn1C5/9dXRtmmq0a4Y+u/PNLNc3ct2PUd2x0s49j35h47H/WPPdBuniBecfsj/m5kjB+9yJ/EOyEfn1v3bPQnKh81dZWbb8y4T1eZE+x3ZUKxh3cbAACQHAocAACQHAocAACQHAocAACQHAocAACQHLqoBkOZHQ/PXdgQZW+qX+yO/UWf3wF19/rZbj6vqTnK5tesc8d+e+lr3Hx2Yzw/fkbjZndsZb68hRbyrfFSDf48f+wt8gftF2XH/exJd+wx4R43//nEo6Ls0zP/4I5d8G8z3HzJMVl76Mjl/XwIu5cqJoxz8zV/E5/L4//z3iHbDwyTjE4iq/CXu1EY+Gtr1pIHNTf7XVdrC9visRX+shMXjH/Aza+ccGKUFdbHS0AUd7C8brPDnYbI3y4+tKxtSFzBAQAACaLAAQAAyaHAAQAAyaHAAQAAyaHAAQAAyaGLagjlmxrd/OBjno+yy5ae5479hxk3u/mKsWPdfHJ1ywD3Tsrn/Fn6azri/f77Wbe6Y/9jxevc/P5Ovxtl/SmTomzMsuUZe4g9ScdbjnPzu7/1vSg78uHz3bHVlX63x+jGuLvum4te6479yP63u/myUQe4eWGz0wGY1S01CN1VdtRBbl79Db/DJPxqwJvG3mww1l3KWtMpS8Zxe3N73Pl4aMML7th72ue7+YbT5kTZqKv9Y7z1Zn+dq1dNXOrmX5kQt1H98coT3LHy3zolcQUHAAAkiAIHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhy6qcnmz2DNmx6+78EA3f2DuFVH2sbXHu2Of6Zzi5gs2+nnLyBFR1l6odse+YnzczSVJNz8f7/ftW/zHcuToVW5em/PXNZl7abzm1uYfuEP3PVkdEs7xlavx1yjr6+x82bvx7JXHuvmP3xB3S0nSrFv+Ng6D/1hOP8Rfo2pTd22Urd8Wr9kmSV957I1uXndRnZuPv6KMdZ0GYS2q5Z/yf2+8acZv3fysuk++7PvEXmAw1j8rc02nLIfUxK/bPaG8cuBzX3woyqq/5K+r9YetC938+s1HuPkjXXFn2YQ743UWXwpXcAAAQHIocAAAQHIocAAAQHIocAAAQHKYZFyuMiZ5zX7bEjd/5WMXRNlVB/3EHfv+RW8b8P1J0utGPxNlDTl/4un97fFHbUtSoRDXvfevn+mObarZ5uZtTVVufumEe6LsG/I/Yn+v4k0QtvJ+f7BK/3QMXV1RVu5k4opJE6Os48f+ROUjavzJ5++6/1I3nz0tnvx3/pSH3bFV5i/V4BlXscXNm6eOdPO2Q+MJ9pL067PiiYwbb5vs3+fj/sfmW2983i97hztUJ0xZ5ubXtB7t5kecGZ+zzf/qbxt7sUGYwF5OI8LOfH6/uKnlNY86S5pIurAxXjZBks5eHC/H8l9zf+GOvWqtv6TP+Jp2Nz+qOn7/CKvXumN3his4AAAgORQ4AAAgORQ4AAAgORQ4AAAgORQ4AAAgOXRRlfnx2VYRP2WLv+N/3PTzc/w1CK7viD+W/qPPneeObW6pd/Oz5/sfed9SiLe9cJvfMVKd8ztaLtzvkSjb2ON/DP4lY+KuKEm6f5vfoXVZ44oo+8axh7hj9ypeF0Mor2sidA18fH7ebDdf+KnRbn7pMfFyBZOqWtyx3/nOOW4+7k3r3fycSY9H2SNtM9yxWcfcyIq4K+yxwnR37KiKrW6+vtvvrhpd0xFlx73NX75h7CVtbu51Is6s2uCOfXTrTDfP8vFJt0TZp3RcWdvAXiCrAyqr29J7DxqkpRpG3hEvg/KpMQ+6Y0+ddYqbF44dH2WTfuG/XzVUxp2gkjS/dl3WLkb6OuLz+KVwBQcAACSHAgcAACSHAgcAACSHAgcAACSHAgcAACRn8LqoMmaIWz6jSylj5njo7Rn4fZYzo7zMbqksX1wSd1/MrvizO3b+ny9z8zGN8Wzw5k1+B0jjyPJmjs+rjmelL+8c6469eeWBbp6z+Hk9dcZCd+xPNr3CzX/30JFu/sGzvh9lq94Qz+hPQX6k/zMNM/2uts5JfgdC4z+uirJNnf6xf9cB33Tz06/4ZHx/Y/1tnHiR36HX4HQ6SdLirfE6VwfXrXHH9gT/PGwrxOtiVZp/bvbJf605vH6lm08ZtSnKlnTF+yxJfcF/XWrri/fvD61+99+2QqWb1+f9TpJrWpyOqRQ6C/cUGa/93ntTKGS8HwzGOlJZsjp2K531mHr8tdKy/POyR928IRdv542T4/Wpivzjdu0r/bXfytGU9zsiV/f6a1SViys4AAAgORQ4AAAgORQ4AAAgORQ4AAAgOTudZOwtSyBJVhVPfurb6k8WCr3+R7PvdmVOEmu9ea6bX78lngS9sdufHBr6/Pox70zi3X+K/5HVb56wwM2zJme90BN/VP+U6s3u2FG129y8x9nvG5cd5I4d98N4aQhJmn+z/7HfV58cf7x3OMz/ePy9ybNXHhtloya3umPbO6rdPF/hT+ZbsWJKlDXd62/j3J9/ws1zTU7Y5w7VXYvmu/n48f7jmd24McqWWPxzlqRRlf5x600oHl+1xR2blz85emtf/LokSQ92xMuGFDImE2dNgu4L8cRmL5OkEXm/UWJKtb80Rm0u/rn/ZZa/NMo+J6N5Jd8QNyYUtvjHS9ZrfyjnPSFrmYVyGl3KXGahnAnFS37sN3V8ZZU/mb7rpIEvkZCla3T8eNZmTA7OZZyz4yr8n9ktHf77b7m4ggMAAJJDgQMAAJJDgQMAAJJDgQMAAJJDgQMAAJKzS0s1ZHVMefIHzPNvyPm1lW2LOwrCJr8LqNCa0WUxb3aUbTpmnDt2/t894+a3zPiVm5+37JQoe3Grv9TAafOfdnPvI9vn1Lzojt3Q6297ZfcYN/dmqzd3+9vwuqUkaVxtvDzEutVxd5Yk1T6yws2z+hO6Q3zIHT9tecboPU/W8guqjFuSRlT53TQd2/wOqM7N8XIAkqRC3MHRe6rfkbOtxz+le3ri7qCGOn/phQPGrnfz6rzfETmlJt6XuTX+Nhpy/n025Qe+JElWB1RVxtIOOYt/NlUZR2jW8hBe51Z3xu+HWcs9PNI5081fW/dslP2yuYwla1KW0XmU2TFVhq7Tj4my6pseKms/BmMJoFyd3zHX1+Es6fP+E9yxHz36Zje/8aBRA96PchVq4/Mqn9Ft1trjv7bVmH+c393q1Q3ld9tyBQcAACSHAgcAACSHAgcAACSHAgcAACSHAgcAACRnp11UWetIeZ0ks//kr6Nz3phfunlzr9+NsrEQr+u0ocfvAvrI6CfdvDbnr99Ujr957vVuvujFCVH2hlmL3LHbCpVu7nVqbO3zO2tqc/56JK29I/xt5+Jtb+n1tz2l3l9bqLYivs+/f8Wt7ti22/zZ8VcteKWbn1n3rSj7j2de447dE2WdE/P/9uEBb6PxoP3cvOVgf12vzlFxZ0LuUW9xKakio6lDE+JtbGn0f3YPLPO37TQjFfOeeNsZK/coo0nJzXPdGesQ+Y1YqmrzO128/c44rZQr+Nuo7Ijzmg3+Ripa/TXe9NxKN75+a9wRWaFH/G3sibI6iTwh4yDK6FLKH+ivi7b5sLirs7vBP142v8L/OX3omNui7Afv8l+3pp7rd8SWtcZhRoeR1y2V5YMf+q2bX/2PZ7t5rR5wc69zq5z9kKRQ4XQWZvwcOzPeC7PWlbtn+awom6Unyti7Iq7gAACA5FDgAACA5FDgAACA5FDgAACA5FDgAACA5OzSWlQ9v22Mso+Nv8Yd+x8bTnbzmTUb3Hx2Vbwm0/TKje7Y/9oyx82faJ8aZSva/bWU1m7xu7nqa/yusBpnfaFFrXFnlSRNqfW7lMaPiNfUqM3599eUH/i6X5K/7s7Yke3u2GlV/vNaa/G+9GXUwllricw43v/5TqqIu+Q6t1a5Y/dEuZF+R19uQrzWWWHVC+7YwtOL3bwho1HDv0fsqTL6hDLXHMpPGB9lVut3Se6RyukkKtOSi/0195Zc/N0o+3mbv+7SIdVr3LytL37def/xS9yxbzjzg25ec8ODbm4V8VtrVgdmli2/j9/f/mtFvM6iJNX91u+WylJux5QrF3dANeX8kqIm779PZK1NF5b750q5uIIDAACSQ4EDAACSQ4EDAACSQ4EDAACSs0uTjOucj/K/a5s/+SlrqYFvPXSKv+2n42UF6l4TTzyWpKPHrXLzR1+cFmUbVvsfP68qf0pg8Ocka/aoTVG238j17tipVZvdvDEfT/Ba3e1Ppnu8I34sknRwnT+BdXzFlijb2BtP7JWkZV3x5EZJ6uyLP1a7L+PD91d1+hP7XtjqP9+PdayLsmNmr3DH7ol61/k/a09+XDzxWJKsxl86I+vj6t31F3ozJnbmM35n8T4mPuv+chnbyLrPnLPtvoxtZ/H2O2s/snRlrL/gPX8Z2w5Zz5/DClnTiTNkPX898QTMvrUDP86Gm1X7x3Po8hsnyjHy+YGPnVdV3nO2sS+eyHpjh9/w8Olv/NjNv3HDAW5ezoTidR9+hZvfdvDXouy8934kYyvL/DhrGY1BmBheUR8ft7mMayZ9wX//aMj5k49HZjyccnEFBwAAJIcCBwAAJIcCBwAAJIcCBwAAJIcCBwAAJGeXuqi2nRTPVv/WZee6Yw+8eKGbL3jdFW7++nEXR1nz4rHu2PsKGZ0Qzoztyib/I6HHN/nLGBw6xv9477kj4o6u0RX+NtoK/setz3GWo5hS6Xdc5ev8bpQfN7/SzR9ZFy9TccxEv9vs4XV+h1bO4vts2ZTxMfMb444rSXIasSRJSybGnUVzxvvLOuztCs3Nw70LwJBb/ZGj3PypD38nyl7z9Nnu2E03TXHzIy94csD7sa7gL7vz43X+a+W7J/45yvx3CenUWr8j7APfOs7N530oXjohq1vql38fd0tJ0teaT4yy6t8/lLGHu1+hO+7QynvdmpLqKzOWPnLeayRp9MKsn0R5uIIDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSs0tdVJ5xV97n5s1X+uPPPPvDbp57X7zW09fP+B937MR8q5s/1jk9ytr6atyxx9UudfNxua1u3txXG2WbCv5aT89um+jm73zinVHWcEe8XUmaeIO/GEvv2nhNJ0mapLhrretef12oD+53l5uv7IrXxXphgr+NroJ/CM2s3ejm+4+Iu9P+5Z5z3LE6yY8B7DmmfOVeN//pu+LXkVsP/I07dsN+29y8M2NJs69sPCLK5lb7a1G9b9Kdbv7qmnjtspyy1s/yrwUse8v3/OFviaOlPXe7Q8fm/fWirv/98VE2U/777FCuOZWlckS8jlSF/P0YkffXnGou+Gt/VWyOj4cyV36TxBUcAACQIAocAACQHAocAACQHAocAACQnEGbZFyuEb97MCOPs+9qrjs2f9B+br7piFFRtnW8X8v9l7+agir8OW+q2RTPehv9+BZ3bFjwtJvP0uP+xh29Ax6Zbf2nZ7n5L/v83ArxdK6Q8z+CO9/hTx5rbfE/Nn3Biw1RNr/jYXes4rnYAPYSX3rq1Ch77TH+pNx7Oye7+bRKv1nhtIZ4CYee4L/Gj87Fk4kl6fne+DVtXMbrXEufP8W1Keff5yZnfHPG0j03tM9x85n/X8aEYodV+m/loWvoJhnXjYgnZOfNfz7GVfnLGXUEf5JxrqUtyphkDAAAIAocAACQIAocAACQHAocAACQHAocAACQnGHrohoMhacXu3mj07zUOIT7kfFp4nuM3F0LXvY2/N6C7Mc+GN1fAPZek66ojrLGn/hdM2fU+d1SK3r9DqjlPfHSMX0Zv6+39fmdntMq4u5Xf2R2t1Qh4xWw2+noOip+OiRJF//8NDef5SzLYNX+RkJPma+45ryih/LeySrycV9TV/CfwcaKjKWPev1u28I6f9mNcnEFBwAAJIcCBwAAJIcCBwAAJIcCBwAAJIcCBwAAJGev7qICAOyZ8nc8GmXnnP9ed2ztl9a6+XXzbnHz+ZVZ/U4DVwi1UZa1llJP8Nd06stYIWlULh9lLxT8TqJZ/1jGmlNe95OkEMpcqcl7nBmPMcvfzbkjyioUP25J2q/a//meVec/J9/vHZw+XK7gAACA5FDgAACA5FDgAACA5FDgAACA5FDgAACA5NBFBQDYLeyex9x820n++DfqcP+GYw+JouVn17tDRx7mr3P16knPRdlxDUvdsVMqNrv58p6Jbt7c2xBlt70p3ueiVRl5rK+zc8BjJflrTklSuV1Xjn+5681R9rkGv7ut8tm4Y02SPlbrr38121mHa1dwBQcAACSHAgcAACSHAgcAACSHAgcAACSHScYAgL3Lg09G0cwHy9vEU242K2N0Vl6OgU8mHjTBn8Q7GOa/76Eh2/Zg4QoOAABIDgUOAABIDgUOAABIDgUOAABIDgUOAABIjoUhnGUNAAAwHLiCAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkkOBswvM7FIzu7vf18HM5g7nPgF7MzNbbmavG+79APYUnBMv3z5f4JQOom1m1m5m683sajOrH+79AoaLmZ1oZveaWauZbTKze8zsmOHeL2C4cE7snfb5AqfkzBBCvaQjJR0t6f8N8/7slJlVDPc+IE1mNlLSjZK+LWm0pCmSPiupazj3ayA4LzAUOCf2XhQ4/YQQXpD0e0kHl/7s9L8Hh5ndaWbvfqltmFmjmf23mTWb2Qoz+39mljOzajNrMbOD+40dV7p6NL709Rlm9lhp3L1mdmi/scvN7FNm9oSkjn39wMWQmS9JIYRrQgiFEMK2EMKtIYQntv9p1sz+zcw2m9nzZnba9m8sHfs/NLO1ZvaCmX3BzPKl2+aY2f80RewAACAASURBVO1mttHMNpjZT82sydsBMzugtO0LS19zXmA4cU7spShw+jGzaZLeJGnzy9jMtyU1Spot6SRJF0t6ZwihS9JvJF3Yb+x5ku4KIbxoZkdI+pGk90kaI+l7kq43s+p+4y+UdLqkphBC78vYRyDLs5IKZvZjMzvNzEbtcPtxkhZLGivpq5J+aGZWuu1qSb2S5ko6QtIbJG3/pcAk/aukyZIOkDRN0md2vHMzO1LSLZIuDyFcw3mBPQDnxN4qhLBP/5O0XFK7pBZJKyR9R8WDLUiq6DfuTknvLv3/pZLu7ndbUPEAzkvqlnRgv9veJ+nO0v+/TtLSfrfdI+ni0v9/V9Lnd9i3xZJO6ref7xru54t/6f8rHf9XS1qt4ovz9ZImlI775/qNqy0d+xNLt3dJGtHv9gsl3ZFxH+dIWtDv6+UqXvZfLenkfjnnBf+G/R/nxN75b5+9dLWDc0IIf9z+hZnN3MXtjJVUqWKhtN0KFf9mK0l3SKo1s+MkrZd0uKTflm6bIekSM7u83/dWqVjdb7dqF/cLGLAQwkIVX7hlZvtL+h9J31Txt8h1/cZtLf2iWq/i3IRKSWv/75dX5VQ6Zs1sgqT/kPQqSQ2l23a8UnqZilc07+yXcV5g2HFO7J34E5Wvo/Tf2n7ZxAF83wZJPSoegNtNl/SCJIUQCpKuVbGKv1DSjSGEttK4VZK+GEJo6vevNoRwTb9thfIfCrDrQgiLVPzN9eCXGLpKxd9Wx/Y7fkeGEA4q3f4lFY/fQ0IIIyW9XcVL9P1dJmm6mX1jh+1yXmCPwTmx96DAcYQQmlUsSt5uZnkze5ekOQP4vu0FzBfNrMHMZkj6exWr/e1+Jul8SReV/n+7H0i6zMyOs6I6MzvdzBoG6WEBL8nM9jezj5nZ1NLX01Qsxu/f2feFENZKulXS181spBUn1s8xs5NKQxpU/FNwq5lNkfQJZzNtkk6V9Goz+3Ip47zAsOKc2HtR4GR7j4oH3EZJB0m6d4Dfd7mKV4CWSbpbxSLmR9tvDCE8ULp9soodW9vzh0v3eYWKlymfU+mSKLAbtak4afIBM+tQ8UX8KUkfG8D3XqzipfJnVDyGfyVpUum2z6r4MQytkm5SccJ9JITQIun1kk4zs89zXmAPwDmxl7LSpCQAAIBkcAUHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhwIHAAAkZ6efZPz63FuTabFqveh4N990xjY3P3r6SjdvqorHT65ucceOr9zi5ss7x0bZym2j3bEP3nmAm0+/zV/INn/Ho26+N7qt75c7fujVsEvpnMDeh3MC+Gs7Oye4ggMAAJJDgQMAAJJDgQMAAJJDgQMAAJJDgQMAAJKz0y6q4ZCrrY2yRf/ur0p/6Ql3u/m/jHvGSR9zxxZC34D3TZLyFteEs256T8Zgv7ng+VOvGvB+5C+909/2pX7suXVrpZu/7y8Xu/n8dz4y8I0DALAH4goOAABIDgUOAABIDgUOAABIDgUOAABIzrBNMq6YMtnN3/qnh6PspBG3umOb+6rd/PqOxijbmjG2O+TdvC+j9ptY0Rpllx1/pzt2a1+Vm/9ha7wv63rjfZak2py/JEON9bi5Z0y+3c0fet233PxD95wRZRtfuXnA9wcAwHDjCg4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEjOsHVRNVzb6eaHV6+KshvbD3LHjqvY4uY1ubjDKGf+Ugg1GXmWlkK8lMRJdYvcsQWZmzcXRkbZ6IxOp3zG/lWq4OY9irvCVvWMccc+tM3v3PrqtBui7KzLPuGOHXflfW4OAMBw4goOAABIDgUOAABIDgUOAABIDgUOAABIDgUOAABIzrB1UX1x6vVuvqhnbJRNrvTXQeoMlW5eCHHdVmV+11Fnn7+NLF431rPdE8raRl2uOw79hitVZXRLZT12T2XGY8/q3FpTiNfKOviSp92x668c8G4AALDbcAUHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhwIHAAAkZ9i6qGZWxGs6SdJDnSOiLKsLqCfE6y5JUiHELUlZXVRZ285aRyqvEGXe2lflyslfc6o74zFm6QnxjzTrsWQ9J1v74i6qS8bf4479qg4pY+8AANg9uIIDAACSQ4EDAACSQ4EDAACSQ4EDAACSM+STjLvfeLSb5+0xN28pxJOPD6lZ5Y4t9I5y8z5nqYasicDesg6SP5m4XN5+SFI+Y3LvYPAmFNfluvyxGfu3pa8myo6vaX55OwYAwG7EFRwAAJAcChwAAJAcChwAAJAcChwAAJAcChwAAJCcIe+iaplbVdb4glNzZS2nkLXUwJa+yjj0V0IYFp0h3r+sZSeyHnuWcrq/xuTb3Xxdb1OUjc/XlbUfAAAMJ67gAACA5FDgAACA5FDgAACA5FDgAACA5FDgAACA5Az9WlSNfl4IflvT1r6Bd11VWu+Ax3rdWYMlX2aLVjnjvbWlitvwu6W85yRrzams56SjjJ+BHXOIm4eHnhzwNjBMzD+2FF7+OmyZd1k58GMr9HQP2X4MhqzHsqfvN7Cv4AoOAABIDgUOAABIDgUOAABIDgUOAABIzpBPMt463Z8IvC34E/G8ybNV5U7itYFPkix3gvDu1pcxQThf5hIOnjrL+BmU8fy1zfKXcKh/aJd2CbtTuZOJc/5yIvIaBjK2PRgTcHvecLSbj3hmrZv3rn4hDrMeS5a++HwbtMnE3mTvIZzojT1LxYxpbt49fWyU9db5b9kVHf77bO4vC3Z9x0qsYuBlQugdeOPP7sAVHAAAkBwKHAAAkBwKHAAAkBwKHAAAkBwKHAAAkJwh76LK1fe4+dYw8C6gzuDvZkOu081bCnFnT1a3VM6GrosqqwPKk7VsQtZ+5zP2u8rprqrNdWVs2+/UKISMj/D3xlYOeCj2NFmdRE7H0E7zcu6yocHNu4+ZH2WVm/3ze8U7/P2Y/j+T3LzK6aKynH+Mh76Bdy91nHucm/fU+edy03/fN+BtY+9mRxwUZe2z692x64/1j5eG5XHW3egft315/4V45qJxbl5obnZzz2B0RmV1im16xRQ3H/WHxVFW2Ly57PvlCg4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEjOkHdR1dVndDqV0byU1enU1+fPHPe6gGpyfudFQQPvGBpKmZ1L5teg3cHvgKm0eMZ7jfmdbFndaeWsRVXGUOwO3rpGWQahK0qSdPyhUbTiTf4aZSOP2uDmr5n8SJT95pYT/G2M9Lsplr/F79Caf0ucDUZnSO/fbnTzUWevdPPMU4V1p/YY+XF+19HWY2e6eeUW/zhaf8yIeNsZS5eNWO/nmw+Lt33Q/qvcsc+uHe/mi/55tpvP/N30eP+6/PdZK2R0IPfEedeYGnfshun+e017vBuSpO6G/aNs7PfL70LkCg4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEjOkHdRNY3wu6jaMjqgKp21lGqcTJIa8u1uvq63McqyuqWy1osayjWqPFmdS15XlJS9317HVFXGelbVuYzuqoyfjaeMZavS5nUvDVZ3TDnbLuM+c3V+p5Pm+q0Nz13U5ObjD43bQLqW+d0UWx72u1RuWzs2yuykre7Yrh7/Zeu1hyx08zv//fgom/Nr/3Wpcm2Lm6950+Qoa13in1ejDvTPn8Uf8NciOuBTz0ZZoaXVHYtsWesddU+Pjy1Jap9aHWWb9/dfV0ct8n/WK07PeAHMOd2sa/zjtnOy/xqfb4hfnzu+PNUde+oXHnPzT7/yDjd/9LT4Ofm7P17sjq1b7u93X8ZSdp761f7rUvcE/7G31sQb93+KO8cVHAAAkBwKHAAAkBwKHAAAkBwKHAAAkJwhn2ScJWuZAG9SbUPGRNvLnrvAzT898+Yo21jwJ/hpN08mzpLPmAhcl+ty87ZC/FHgkjQmF0/M/OnmeJKlJF006n5/X8p4TgrVzDKW5E/uzWXMwstaIqGM8RWTJrpDV5/nfzT7uCfiSbWrTvQnAodD29y8d5N/Hrb/Pt6XhoxXlrYD/M+rn/mlJ6OsUHOUO7b69c1ufseDB7n5Ga+Ol4FYdqQ/ZXHhYzPcPL8t/vmOnesv1fDhX1/v5q+u8Z/Xc7/tTO7cxyYZ5xr8ZTb62vznLHdo/FH+y84Z7Y7NmuC64ej4vKps9V/PWuZlTD5+0h/ful98n711/n5Yj7+NkXfH52fnqPIaF/5xzWluvq3gTITP+9t+5d8scPMXtsZNBy2d/vvSQaPXuvl9a2a6uRaO8vMycQUHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhwIHAAAkZ8i7qPI5vyOnMwx8qYYszz43yc2nzd0SZd7yDTu7v6ylHcoxGMs9FDKWZMgyOh9/vPefVs93x146+l43zzkdXYXgP5auJrqoMmV1S5U53iqr4qFb/O6SSX/2u29GfntdlO1f7S91ctcv/e6lkX4DlLZNjLsvbHaHO7bieX95iA1vPzLKRj3rLyWy+hi/I3LSfL+76g+3Hh1l+Xn+Y3/LSX5n4bF1y6Js/6p4iQpJeqjT78R6qse/z+fPiztGZjzlDt2r5GriLiCbPsUdu2223wFVvdFfUmP9P8UHY+j0z4mOgt+hNeOm+Lit2Op3rS5/t99h9OozFrn5bQsPiLLeQnmv5YWquKty3Un+a8Rqp6NJkn4z9zY3P/6xt0RZvtbvknx8g/8z29odv4e3bfLP7/WPT3Dzig7//aN3wuAsc8MVHAAAkBwKHAAAkBwKHAAAkBwKHAAAkBwKHAAAkJwh76LKmT8bOi8/99Zeymc06tQv9TuxxuYz1vRxDEa31FDqyVizq5BRm07Kx2uBbH3M71CoOcyfkZ/P+Jl5Kpw1elJWeE3c7SNJz58VdzrNv9rvaLIOvzOksGylm4cep2PEySRJC55249YT42zUg/66Ma857yE3v2v1XDfvWRd3qVRnHENzP/e4m8s5Z3P1fkfGvNtb3Lz7xvFu3lsfdwCeOdt/ni5sesDNt4TqKFve66+XM64i7uKUpIMr/efk8rfcGGXX//MYd+yeyI7w1wBb8aa4c3XqHX53XVa3VKHGf/2r/XFtlE2+3D9/nq33u+7WvD0+h3pa45+zJJ08Z6Gbj6vyO+NG1MfvY1Ob/NeDmQ3+mmbjjoy3/VjLVH9sjb8fc669zM0n7f9ilE0bv8kdu/rRyW4+Yr/4PLS8321bu8Z/n+32m5tVqI23YxXllytcwQEAAMmhwAEAAMmhwAEAAMmhwAEAAMkZtknGWRry26KsM/gTlMY+7k+0rLd4oljWsgl9GUshDMYyC1m8+8y6P2/StSR19MaTWiWp0uLJmhMe9CcTt73Dn6Rdju7GPXuS9mBbd7n/8zhozJooe36mP7lb8if3njLdfy4Xv3P/KOse60/APfPbf3Lz6z7++ihbfqw/mbj6roluPrpuq5uPmB4vqVCRsUTLmvcc7uadY+PXid6Z/sTTcWP8SaMbHxnn5peceleUfWzMo+7Y+ly8vIAk9YT4HMrJfz7u9w8RrS74S098/eH4ZzNP/v7tiVY6k4klqfvA+LV8yRR/Em9lq98YMvpp//1j4yHxudLyh1nu2LnXb3DzRZ/yjyPPnY/FSy9IkvL+/lWviV9bVxb85+n5EdMGvB89o/zX8oUZy4PUTPOXr1i3Md6XQpf/M7CJ/gHd3R2XD7bZf1/qODY+FiSpotJfHuL9B8bLCP3u7Ne5Y3eGKzgAACA5FDgAACA5FDgAACA5FDgAACA5FDgAACA5Q95FVa6mXNyZ0Fzwu06qb3/CzfMW121ZS0Moo3spc/wgyJs/E74cfWXUpnX3LHHzjX3xx51LUkMunvHeK3+fe+r3raUaqm4b6eZT37U0ykZW+l1AGzr9Dqg12/wui+lXrRjg3knPbvU7oD73nz+Issu/9QF3bGGj/5HyB09Y6+Zjq+KP33914yJ37MyP+B0t17UcFW+30v/4+ca8371UN8fv9rigYXOUvWXpGe7Y9tf6yyxkLo0xCPamjinPrB8tc/MlH4q7msYe1uyObaz2z5XKE/zXne4t8XnY3uF3wC2c75+zo+6PO516R/idjFsn++8TVvDHB6e7qmuC/1gqG/3jdvwovwPK01TjdyllWdnSFGWFGv89ZdYYfwkHT9XEdX6e9x97e4/fVXfXhvlRtuHQgS/BtB1XcAAAQHIocAAAQHIocAAAQHIocAAAQHIocAAAQHKGvIuqIaOTpDP46yB1h3im9KaCv2bIYHQ2DGW3VDnyGd1c3vMhSaPzfoeJp7A57iKRpFU9Y9x8ZmXc6XJ3Z8YaPaOGbs2uPdG4797n5vdWviLKDjx/oTu2tct/Lpdv8Neuqrq/Icra9vfXNcrX+/nSLWOj7N3vucnfj07/uBhZ4Z/Li9snRNmXF53qjm1p8TvIqpfFz8nsq1e7Y0Or311SmDfVzf+7Pe5SKTzzrDs2S64u3u/Q7T/XyvmdNWZ+3tfpP697i961fufMrE/HecUkv8vv+XfNdvOKY/zXrpE18c/0TdOfccc2Vvhdd6sOjc+38VX+sdXa63fybuge+HpWrd3+eZ/F6zzK6jrq6PHXgMoyqSF+nN19/ntNfaXf5XVwQ7z+XtbztKXXf+ybO/1O3pkNG6Ns46Ly32u4ggMAAJJDgQMAAJJDgQMAAJJDgQMAAJJDgQMAAJIz5F1UnQW/W2pjRmfUtMp49vRDrfGaJkUDn1Xtra8kSW19/qzvrK4mTyGUVycW5HRTZGwjl7EfVYOwntU9rfPcfP9x8ZpDLRnrVuVH+TPs9zUTvnVvlG38lj/2xX/fz81HrPePgWnXxd0KvcuWD3jfstyoURm3ZKy7Ux13c0lSy1vix1PT6295bMZaP13Ormw8cbI7tvGnD/gbf9DvuHn5Z4rUt83pdOorb8vmdGJJUtfpx0TZtjF73DKBgyKr42raF/28HAtmzHTzvkb/ed94RHzQtU/1j8/CCL/bNqPJNXP8y2UZ51Wux9/vXPfA18rKb/PHrst4jEtXx+d9xTb/cTc+F69XJ0nW67/WLF8Qd1CO1P3+jvzUjyWu4AAAgARR4AAAgORQ4AAAgORQ4AAAgOQM+Uy21a2Nbl45xZ8tVeVMCXy+w//oeKnZTQshnriUNSnXnfArKWNelT82YyJw1uTjcpaHyNx2xn6X44/P7O/mf/+a26LshV5/QurIBv9j0JFt7t9nTJbLsPn846Os8sDx7tgXXuMfcw3L4rzlYP8cbHraf1nojFd7kCR1jYnPrTkHxhOjJWnpM/7E4cnz4nO5udVvRBj7bv/j/pf/wW9G6JgVL6lQucl/jD2N/uvE1Nvi823D2/xjv7Ml4yP5M077UY/G+zLuSn9JEF3tx5B6V6wqa/yoJ5xskPYFf224FkTiCg4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEgOBQ4AAEjOkHdRTWhod/Oe4N91Q647yhYsmumOnZ/RRdUevOUDynuouTKWgcjcRhnLPWSps/j5kKQtfRmdGmUYsbTav89T4v3u6Ktyx05qaHPzuG8Fu6rhFwPvuppz48C36/dhDa15Gnini/8B+9nH1hTFS4wMpem/3a13B6BMXMEBAADJocABAADJocABAADJocABAADJocABAADJGfIuqobKTjfv7Kt08xpnzai6Zf7YLJsK8TZqzO+9yFqjKmsNKHdsmStteOtIVTprcElSZ/Afezn7l6V+lb/fnSHev6yut55COat2AQCwe3AFBwAAJIcCBwAAJIcCBwAAJIcCBwAAJGfIJxnnzJ/ImjVJtsOZzDry+fIm1NbEc2TVEfylBloKtW5emTH5uByV1jvgsX3BrzXL3w9/Urenbv3A968m50/Sfr55tJvP0uoBbxsAgMHGFRwAAJAcChwAAJAcChwAAJAcChwAAJAcChwAAJCcIe+iqq3oLmv8VmcJh8qO8rqovtJ8cpT94/i73LEzK1rdvNrpxJKkLqcprDJjrLfkQZaGzG4zfxv1lrV8hd8t5qls8zujtvbFh0Uh47GMa2of8P0BALC7cAUHAAAkhwIHAAAkhwIHAAAkhwIHAAAkhwIHAAAkZ8i7qJa2jnXzM8Z0uXnOWaMq311eF9XCo+I1ls5/44fdsVtm+N1InWP8rqHeurjbqXaNP7Zhjb/WU8jF47tG+rVm1n7kMprTJv+x2UmXuGOtx1/namNfvD5XXcYddvcO+SEEAEDZuIIDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSM+QtMI3VnW5eZX4HT0/IR1lli99xVY6qWx52c7/Ha/eL+5Z2jf+s+qzgd6cVQlz31uX8n0FXb/zzAgBguHEFBwAAJIcCBwAAJIcCBwAAJIcCBwAAJGfIJxk/u3a8m0+c0ermW/pqoizX2eOOzVrAwSqr4jBkjLYyazxvO1nbyLrPwdgPZ7kHSQo9zvIQff7U49yKdW6+qmdMlM2petEd29dHjQwA2PPw7gQAAJJDgQMAAJJDgQMAAJJDgQMAAJJDgQMAAJJjIYTh3gcAAIBBxRUcAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAACQHAocAHsEMwtmNncA42aWxlbsjv0CXsrOjt2BHtcYfBQ4u8DMlpvZNjNrN7PNZnaTmU0b7v0ChoKZnWhm95pZq5ltMrN7zOyY4d4vYLCZ2Z2l1/TqPWBfLjWzQul9pt3MlpnZ+wdp21eb2RcGY1t7MgqcXXdmCKFe0iRJ6yV9e5j3Bxh0ZjZS0o0qHt+jJU2R9FlJXcO5X8BgM7OZkl4lKUg6a1h35v/cF0KoL73XnCvpq2Z2xHDv1N6CAudlCiF0SvqVpAMlycxON7MFZrbFzFaZ2Wf6jzezi81shZltNLN/Kl0Net0w7DowEPMlKYRwTQihEELYFkK4NYTwhJnNMbPbS8fyBjP7qZk1bf/G0rH9cTN7onT15xdmVtPv9k+Y2VozW2Nm7+p/py91HgFD4GJJ90u6WtIl/W8oXfH4z9LV+jYze8DM5ngbKV3xXGVmJzu3VZvZv5nZSjNbb2ZXmtmIgexcCGGBpIWSDui3vbPM7Gkzayldfep/2wGlrKU05qxS/l5JF0n6ZOnK0A0Duf+9EQXOy2RmtZLOV/HEkKQOFU+UJkmnS3q/mZ1TGnugpO+oeHBNktSo4m/EwJ7qWUkFM/uxmZ1mZqP63WaS/lXSZBVfdKdJ+swO33+epFMlzZJ0qKRLJcnMTpX0cUmvlzRP0o5FfuZ5BAyRiyX9tPTvjWY2YYfbL1Dx6uUoSc9J+uKOGygd19dIOjeEcKdzH19W8ZeGwyXNVfH1/58HsnOlPwvPl/Rw6ev5pfv6iKRxkm6WdIOZVZlZpaQbJN0qabykyyX91Mz2CyF8v/QYv1q6OnTmQO5/b0SBs+uuM7MWSa0qvkh/TZJCCHeGEJ4MIfSFEJ5Q8QA8qfQ9b5F0Qwjh7hBCt4oHdhiGfQcGJISwRdKJKh6nP5DUbGbXm9mEEMJzIYTbQghdIYRmSf+u/zvWt/tWCGFNCGGTii+4h5fy8yT9VwjhqRBCh3YojF7iPAIGlZmdKGmGpGtDCI9IWirpbTsM+20I4cEQQq+KBcLhO9z+Vknfk3RaCOFB5z5M0nslfTSEsCmE0CbpSyoWTlmOL12BaZP0oKSfSFpSuu18STeVzsEeSf8maYSkV0g6XlK9pC+HELpDCLer+KfmCwfyfKSCAmfXnRNCaJJUI+nvJN1lZhPN7Dgzu8PMms2sVdJlksaWvmeypFXbNxBC2Cpp4+7ecaAcIYSFIYRLQwhTJR2s4nH8TTObYGY/N7MXzGyLpP/R/x3r263r9/9bVXzRlXY4FySt6P9NL3EeAYPtEkm3hhA2lL7+mXb4M5Wyj+XtPqJigfRUxn2Mk1Qr6ZFS0dIi6Q+lPMv9IYSmEEKDpImSDlKxKJKK59D/njchhD4Vz6kppdtWlbLtVmgf+4sBBc7LVJqX8BtJBRV/0/2ZpOslTQshNEq6UsVL+ZK0VtLU7d9b+tvrmN27x8CuCyEsUnGOwsEqvtAGSYeEEEZKerv+71h/KWtV/JPWdtN3uH1n5xEwaEqvw+dJOsnM1pnZOkkflXSYmR1WxqbeKukcM/twxu0bJG2TdFCpaGkKITSWJhC/pBDCekm/lrT9T0prVLzqtP1xmIrn1Aul26aZWf/3+Oml26R95C8HFDgvkxWdreLfZRdKapC0KYTQaWbH6q8vc/5K0plm9gozq1Lxsjwv2thjmdn+ZvYxM5ta+nqaipe571fxWG+X1GpmUyR9ooxNXyvpUjM7sDSP7V92uH1n5xEwmM5R8RfUA1X8s9PhKs4p+4uK83IGao2kUyR92Jx27tLVlB9I+oaZjZckM5tiZm8cyMbNbIykN0t6uhRdK+l0MzulNOfmYyp2N94r6QEVrzJ90swqSxOez5T089L3rpc0u4zHtleiwNl1N5hZu6QtKk42uySE8LSkD0j6XOlvpv+s4kEoSSrdfrmKB9laFd8cXhQtt9hztUk6TtIDZtahYmHzlIovpp+VdKSK89BukvSbgW40hPB7Sd+UdLuKEzZv32FI5nkEDLJLVJwPtjKEsG77P0lXSLrIyvhAyRDCShWLnH8ws3c7Qz6l4vF+f+nPun+UtN9ONnlCqdOpXcVfoJtVfA9RCGGxildNv63i1aEzVfz4ku7SHM8zJZ1Wuu07ki4uXYGVpB9KOrD0p7LrBvr49jYWwj5xpWqPZGb1klokzQshPD/c+wMAQCq4grObmdmZZlZrZnUqznp/UtLy4d0rAADSQoGz+52t4t9q16j4+R8XBC6jAQAwqPgTFQAASA5XcAAAQHIocAAAQHJ22v72+txb+fsVhs1tfb/c4z4jiHMCw4lzuEMofwAAHjVJREFUAvhrOzsnuIIDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSQ4EDAACSs9OlGgAA2Gfk8n7eV/Bz81cJmHBvQ5Qt2jTBHdu8YpSbV22O96Vhmb8b429c6uaF9S/637CP4AoOAABIDgUOAABIDgUOAABIDgUOAABIDgUOAABIDl1UAABIspzfFRX6MsZXVLr5pJotUdbb5HdozR/V7OYzazdG2djKNnfs7y44zM1XPXG8mxfq466wysYud2xfn38dZHRjR5Tlc/4T1batxs2zTHv78/F+bN1a1jYkruAAAIAEUeAAAIDkUOAAAIDkUOAAAIDkUOAAAIDk7FoXlbf+RghlbeL5fz3BzWfeuC2+u3seK2vbAAYgYx0dq6py89Dld1kMxuuBuwZQ1vo/ewg75hA/f8ZfMKivI+46KX7DIDx/GBShr7znPTfC7w56viNei2prr39edRX8t+FNXbVR1txR745tGhG/b0rSqa9e4OartzZFWZ/814MXM+6zpX1ElHVv87vKcpV+d1Wh03/su9Ix5d7voGwFAABgD0KBAwAAkkOBAwAAkkOBAwAAkkOBAwAAkjP0a1H9aaobN26L19mQpGVVo6Ps7G/5ddjitglu3tIZz+6eOXKTO/aexXPcfPY0f32QVQ9PibKeRr/bI9fp73fdqjhvn+NvI1RkLIISMjpgeuM81Prbrl7lz+qvPmxzlLW1xDP6JSn/or8N63Vj5bvi/ZvwUI8/GEMqs1uqJ+OHl9V15azHE3q6y9rGyx4rZXceDUaX0rFxx9SSy/2Xz3nfnOlv45Gny7tP7HaW99eLChkdfVbnvy42OmtGVZj/Wr6xUOfmW3vi8zNzraeuajd/ZIP//uutDWXmnxOd2/zXiV6nAypXmfE8ualUtdbvuhosXMEBAADJocABAADJocABAADJocABAADJ2ekk41ytP4GqnI9RPnz0aje/9sFj3Hz04ng60gfe8md37Ls3XeTmHV3xpKj3T7zdHVud8ydUjq7yP1Z93Kvao2zBnfu5Y2sPjifrSlLHpHj/cgV/cluhzZ+E9alX3+TmN754aJQ9u3a8O3bKn/1JZav2i+/z746+wx27tc+fgLZ06zg3/8qUP0TZCfP+zh2LoZW59EK528maUOwOzpjcG4ZwWYYyJhTn5/tNBwvfFTcujPlzxoTUR/yPx8/Esgx7jLKOZUlrz5nl5pP1RJTNrPUba8bXxBOSJWlZ+9go6+nLmNRc3enm0+r896CnNk2KsqyJwOMb4vc8SerqjcuHEZXlNYws7fbfmwYLV3AAAEByKHAAAEByKHAAAEByKHAAAEByKHAAAEBydtpFVU63VOHkI938t4v9LqVXHbrYzf/7zLhj6oB73u+OnT7anyF++oz4I9E/N9vfvxd/l/FR1kua3LxyS1wTLr7sO+7Ys5ac6uatP5geZc2X+s/1dSde4ean3nm5mx85e2WUfe7I692xVxXe7OZzP7styk77/VPu2O9uOMnN73p2npufuNzpOtjgf8w4hpZVZizVUMjoaMr6uPqjDoqyDYePdMdOuGOtm/eObYiyfIff0bL8zWPcfPTCjP1zPt1+5P0r3LHrvu53RjXeGr9UjrnqPndsppy/bQVnB+ms2iu0T/Pzjt743GrujDuXJOnUcf5r6+sb43xjb707dkql/164rtd/HzuqIT7+m/J+5/AjHX6n2PyadVH2oxWvcMdufGCim9e3uvHgLK8iruAAAIAEUeAAAIDkUOAAAIDkUOAAAIDkUOAAAIDk7LSLKpxwmJuvOakuyib/xe8COnraKjefVLPFzY/6bNwxVVnnr5Lxpct/6OY/2RTP5F7z8WPdsVcf+k03//m049z8N08fHmX3dDpdEJJWXDfbzdsviLuUGqv8NTw+tPQ8Nw/b/B/daWPjmffLuvz1Pv74sx+5+ZsWvynKLnrine7YV09Z6uZ1Df7aKL0L4ln9Nf7kfQyxrG4py2essZTRRbXqDY1RNvEUfw26xa/1O6D6er2uCb/La9SYDW7+wiy/c2vuzPVRtvQIvzOkZ6nfqTG2xcnL7erIeP7c7qqhXJsLg6ZnnN8l3FmIX5+fXOAfc0vX+Ouf5ZwmwqpW/5jLWBJQW/y3IPWOi99vKkf470ENd/jrX/3ROWUL1f57df1Wf783vcpfD2/b2fFalSOue9AduzNcwQEAAMmhwAEAAMmhwAEAAMmhwAEAAMnZ6STjii81u/nIK0dEmd3zmDu2udP/LOvPT73BzZ+6aUqUHXzDC+7Yo6r9mVUbGx+Pt/vIoWVt4+AJ/oSmtZ3xhMpX1vh1Yt0af/LxlkPiSYVtC0e7Y7d2+fmowze5+bOd8UdiX/es/9h/+ZdT3HzCDx6Jsknj/Mlj13/qKDdfdu733Hz2mvdF2YwPx/cnSfqaH2OQZC29UOdPKgw9/tIJ26bE26l+a5s7ds7meCmRwTLhAH95kEUfnByHo/1zUxnxi6+OH2PbDP9j6ad94V5/I1myJh9jaA3CcgCV9f454fnG6f/t5uPz/rny8WffGmVrloxzx4Yq/8A9ZH+/yefpVfGyEfV1fmPIBR+62809L3b7E/2Xto918/NHL3Pzqxe9McqmXjfg3fhfXMEBAADJocABAADJocABAADJocABAADJocABAADJ2WkX1crNo9x8yi/uH/AdnDPJ7666pvXojD2KO4x+t+QQd+iFTQ+4+Stq4tngFfc+7Y6d/cd3uflnjr3ezVd9aX6U9XzvT+7Yrrf7nU7fO+C3Ufa1Sy9yx37s6mvc/PPPne7mj783fq4W/PYqd+y57zzZzTUv/kjxkNFdMO0WPz/r0FPdfL9/eCbKFn3nCH8/MKSsutrN+9r8ro7eU/yOuUl/jrtRCps37/qO7aLCwiVu3jR9vyirrvQ/Yn/D4/6yJrM/FL/WWKXfgVnmAg6qmDghynrXv1jmVlC2cpfacJx3wKNuPrYyPoc+css73LFN01vc/GsH/SrKnpzmdyWv7Y6XwJGkmpy//MIv58ZdzJ990T+//7j+ADfv6ImP/5kj/fe86XX+68F1q/3loDonZLQzlokrOAAAIDkUOAAAIDkUOAAAIDkUOAAAIDkUOAAAIDk77aKaP9afyd9Rxh28tWGRm39o5Zlu3vDT9ij79ZRfuGOf7HLWmJF0eHU8Y7vqVr8j7IAL1rj5hQ+vd/NrVsWz49+76mR37Dtm++tZLemO14tae6K//s+vN/rdZlft/z9u/uYzPh5ltTm/22PDBf4M9tE/us/NPdu+6q//03lV3IklSc1XdEXZe474c8bWPzng/UD5Qlf8s5CkfMaaThtn+8fRmB8M/Hgpi7dWkJTZ/dJ96jFu3tsXv6bU5/wujfnfX+tvw9uNjLW5cg0Nbr71ZL8bZc0r487RyXfPcMdiEA3CWlT3bfBf594xNe40XvY3/vp813f4r/0feeL8KBs5wl8vav8m/716ZUfG+96C46MsX+WvifbauYvd/NzRD0fZsm6/C/EnK49z8+aWejdvnB2/h1fMKv+c4AoOAABIDgUOAABIDgUOAABIDgUOAABIDgUOAABIzk67qDZ3+bO7/V4K36vufb+bnzzLXzfm4Lq4q6kn+HXYgdV+x8Nblp4RZS92+LO1m/7/9u40vMryzAP4fU5OtpONhGwgSUiGLbggKJV1FJAKuIGipc5A6SAjTmXaUdt6udQy4zB1Q1SgorXTq5dYLM7UGRVUqMhUFtl3QgIJJIEQSEL2nJOzvPPFfpn7/2TO0VCSx//v49/Hkzc57/ueh3Pd93vXVcLc7+AZHuWz9cyPwZ4TcO17Z66F+aKCLSrLW49neGwbPRDmb+Th92btvJdU9vSFUXDtrmd+CfPBN89X2aCF+P3aOXIdzKcsWwDz/5j0isru+anu/BIReXINjC+vKDt7/uKiOD5PP93NJyJy7q8zYZ65OopuKbfuDBIRkTDu1OgOrf3w7cx/WF+zNbH4NVJC1TD3FA1U2anv4C7OjlzcoeXuxO9NDGhmS959Gh9gb9fTrx8D13VXwnxKtu4kEhHZ2VKkso/qr4Jr5+dshfn20b9W2aYOfG2urJwE87rWJJj/fMx/qWxeah1cu6alL8yfKJmlMl8AX4OOYzj3YyJ/3/0D8XF0hd/gEBERkXW4wSEiIiLrcINDRERE1uEGh4iIiKzTZZHxqcosmA8dqQt2nX1H4Nq5xXhcQUkrLnDc3qiLsxan44K7wvX3wzy5RJdBH3p4FVx7SwAXAo9Y9yOY/+u9a1WW4MYFyRs/wI+On3OVfgz1P/8MvxWJn+DHvstYHD93dprKan+m/6YiImMewY+OL7vpNyqb5scjI9a24EeBFy3FIzpSQPFpw1WGwsOeqIcXQ0ZzfGfvwudFv7ePwdxYHowKR6MtJu6Gx+YbehEksVa/dloFGr4gcuGmATCvG6WPxYunvEjGYXw+e8/jv0nAqw88eA6Piun1etL1E8Wx1N6QBvOCeFyYG3D0fS4QxoX3L57+NsxfEn18Swp1cbCIyLXpuDg+Pasdr0/Q66ccvReubTCMkgiE9O/T6cefY55YfO7HxOCC/A6//gwPxUX/fQy/wSEiIiLrcINDRERE1uEGh4iIiKzDDQ4RERFZhxscIiIisk6XXVQ5m/F/Pv6AfvzzkEX4NZ7MxN00O5Jx19W5kK5Wn3rsdri2eFkzzNdv+r3KljXgjhETJx13RuV6mlS2q6MQrvWejbxK/+i4t2D+3gg8YuK+Cvxo7pX5H6psxqNz4VrfhhyYbx2mK9vdXlxJ//S7c2B+/52fwHxzh+7M6+zfCdfSl9DYAwd3Hxg7Q751tYrimvHa0EXd5Wc8DhHcMWV4JL8rDg96cfxgXkGUUqvwNetP08ftrcT3juo5iTB3OvS9MJiEf8eko7hjpDMF/3vS09GDOots1A3jIfyT8fmyvy0f5s3BBJVVtemRISIiozNwl/CuhgKVPVOpxxCJiCwpwN1V/143EeZLz8xQWVIsvg+npXfA/Gyr/qz2x+Bzv48Xv0ZtE+4STojT17I7GM2QqC//n6j/DyIiIqIejhscIiIisg43OERERGQdbnCIiIjIOtzgEBERkXW67KJKe2sHzLMX6jlS0fZALHhjMcwDqbqyPdgPV3dnjI28qvrhjHKYv/H0P8D81Qm/hvmHTSNU9p9H8TyreT/YAvORu3TnUeLbuMJ+1KP7YL48/wOY/7B6usriX82AaxOD+O8693M948t5GXcifDBpGcxnrn0Y5p52/Tq63+AbytTtgRg6QFyx+Joov0t34xU+tj3yn9eVKI7bCeAZUFExdHO15cTCPAT+JLPf+QyuXfbbu2Be8Jqez9V08xC49twYfHxZe3HnW3wj7v6ibuIy/Dve0R0/njw8i2x8XgXMO9DJJSItAX1XS43zwbVobpUInvXUEcDn+H4f7uYqa8bzJAckNcIcaQ3Ew7xPgu6ManXjtY6D7xHxsfh+kJPSojK3oUuyK/wGh4iIiKzDDQ4RERFZhxscIiIisg43OERERGSdLouMTXyP6yLj+vdT4dodvj0wL3gdj3CoflOPD0gwFCLFtPc1HaJy5au4mNhXhAuXlp7Qj7IWEZmTp3+f2HJcJvubtvEw957ShWIr/m0FXPt8lS4aFhH5qE0/xltEZFzaSZXV1uG1svMQjPPivqWymgm4EO6hMjyqIaEBF5WlT6lRWVVV5O+j1UyPjgfFkCZN94yCedHTe/XLRvyqX0IjGUyiKZg28Ay4AubVd+PzubkY3yey8vToiXXzpsK1A3Ztgzn6zZPXfQHXnr9+LMxb++NrKG3vOZV1Qyk2/VkU5+2FyXkwvz0FN3vUBvS4AhGRxBjdwHGkqR9cW9WRDvOMhDaVpRkKlU3HMTClHuYxLn31n2nHr9HSiQuHk8Foh4zEdri22Y8/I/0BvAUpPayLvQeX4uutK/wGh4iIiKzDDQ4RERFZhxscIiIisg43OERERGQdbnCIiIjIOl+pi8q1db/KOreMg2s35F8D85KfD4b51lEvqqwsqB8zLyKyqAh3RrWGdaX5FVt0RbqIyIpFq2CeE4P3fnNPzlLZzdN1h4qIyEdlxTA/svgNlQ3903y4dmrRcZivODkJ5s17MlVW8t4v4drh2/4W5oWLK1UW/i7uAGhe1x/mHvxEccn26kdwV/uz8eJvGsMIAs9A3dlx9HH9PouIJKTh8/zc5KtVNqiwFq6t2IsfV1/0B9wh4dp2QIfd0BF2ah7ulmovwiNGXLF4FEL6rWX6MCI+iugFM3FnZmcLfqx/+HzdJTwaS4EuPVcMvn6cYOQ9aY3T8Dle1qG7e0VEkj14SFFjwKsyYydRCH8Me1z6fE6P0+MRRESq/bgTq28svh9sqR2ksrwUPL6hILkB5lVt+mdWNOCxQK21+DPc7cOfs2knuue7F36DQ0RERNbhBoeIiIisww0OERERWYcbHCIiIrIONzhERERkna/URYX0fwHPcMn8fivMf3HLWpjfvuTHKotvxt0R/jtxRbkPdGqUP4Rn46xp1HOXREQWpOO5FyXbClV2KAt3nZROXw3zGcMnqyxvJH4rVqzBxzHhF9fCvPMOXdXvd3BXx9Fxb8F82oC5KrttGJ5btXzSbpgXrVsE832ndEdQ7EXus0VEan50A8xbR+iuQO9x3JETSMXta32K9Tym+BjcXTLD8J42jE+C+bH6ISrLeQR3S4VK9aw0EZGaR3QXZns+Pj6PF+dF9+nuTiPTrCxT91cU4s7h9wCMJxIRkXAb7nTp9dDf2IWvdVMHlIkT0udXNN1SIiKd00arbMYQPHNqQ8VwmGen4s83f0j/Pg3N+PrJTMOvEXb038+fiD8nUmLxjKrydtxtOSlXdxZe49XdsyIiT+6ZCfOgXx+Luw6f+3E+fL11ZuL7RFzT159lJ8JvcIiIiMhC3OAQERGRdbjBISIiIutwg0NERETW4QaHiIiIrNNlF5UrPh7mjh/P30A2gw4LEZHOMP7Ru/9Fz036bTOuBK/u7AvzlQ26Ot5Uk72+6kqYP5l5EOal39PHN+HgXXDtrfcugLl/ou6A2bL6dbj2ttLpME/56DDMVz2rO2C2+/D7uKoGz7NyN+qujve3XA/XvvidnTB/7Nv/DfNcj5538pP4u+FaW7lTUmDef+UemKPr7eQLY+DalAr8b5Y20XNjTvjx3JiTju4UFBHx5eE2oLHDdGdU7Sr8Ozb+fizMWwt0N8Wdo/GMt2PXRdctEw1XLO5OcwKGFii4uJsOprdDHWmGWWROOPIZZVEzzHhrfFDPxTvSiGfuhcP4EyQYjvw7goR43M16RXITzOt9uusq3o3P/Y4Q7l66L3sHzNee1x2be8bha3ZwLj6+iuf0fClfK/5cDyfii8Jl6IhMPYX/VtHiNzhERERkHW5wiIiIyDrc4BAREZF1uMEhIiIi63RdZGx4fDYqF4rJyoJr787BhYJPfTwb5uNXPqCyHy9dA9dmePAjrpcdnKKy50e/C9fOTMKvsbQOP5r77TX6tQvWVsG1wRxcOLdk+a9UVhHAx+H8HS4QLltyNczzPJtUdsjfH669P/d/YP7ES7NUNvsKXKw268QMmB/ePxDmG2e+oLJr+p+Fa3u74JTrcO7F11VSOS7mc8Aj70N9cRHevvvehPmwz/X4jURD0WNzSyLMXU24APfAh8UqM9RCigvXMcpt4/R94sgPr4Jr3RLFSAYTw0gGVxwu1oymyNiFJ8tIOLppBL0fKO51Dx8MlwayvPgl/JEXHwdS8flZNQV/zI3LPqKyk024ocU02aPOMH5hYGaDykZmnoFrE9343PKAE6kthH/HoYm1MP9DPb4H1Y/Xo1tMOgtxM0+/PhdUVn4ev48SY7jeDD8z7kSNyr5KawG/wSEiIiLrcINDRERE1uEGh4iIiKzDDQ4RERFZhxscIiIisk6XXVTh9vaIXyj8Dq7unuqthHnebath/pMvFqnsqdXz4Nr3Fj8H86ODdNfQJ424I+O1KlwhXr4zH+ZFm3SnS7hBjx8QESl/cADMb0rU1fGvNQ6Fa4Plp2D+7B2fwjwnRr/27zrwcTQGcMX78uFrVdY/Bp8L333hUZgnzMJdYSUB3aUQzePOeypn/LUqa/hHPfJCRKT9kB6bICISmI07Mjw1upPOW4r7Dwo7/h6/RovuaOlswq8RHmwYxWLohBhzhx5rcvRlfL2l3l8N8w2b9CiQos+34+MwPHpfuuFx/+E2/J5FI8aH/67uSziN4HIK3Iw7dWof9Kms82QqXBtMN/TIuA33Bj/IY/H5mdAH34sqW/SoklbDWBtvAr4mMrwdMA+AljnTOIWcuGaYf9YwSGVPDVsP1264iLtqS17Bo4hSBXfFIqEEw3sA7tsuB5/7jhu/N44fX8vBmnORHdz/o/d/shARERH9H9zgEBERkXW4wSEiIiLrcINDRERE1uEGh4iIiKzTZRdVNGqacXX80vM3wny4F88f2vH8ayqrDuIqeNNol3/K2qyy00F8fG3puPvr1mG6A0BEZM7EySo7/jauYM/ah4fSvH6n7vJafli/rohI9kw8F6gljP9+bzXrqvm1GyfAtYm1uOJ968RClU3PPwrXtg7Ae+SOenzcnzbpGV8lH+P5NIIP+7Jq+P5YmLdM1+dobiI+h9oG4U6d4bl6touISH227na72II74Iqz6mF+cluByvoewZ0rHbn4tuDE4U6IA2/q8//iDXhtuhtfE0WPGTqmkG7olrqk8K9u7SyqmvG48yi/j56PVJqB7wsSxPcRlw/nsc06DyXgP7y/A18rlWd116KTiuezpffFn0FFKXUwz4rT6w834ZmA9X7cPbnwr7aq7PEDM+Ha/GfxvTx1V+TdUiYxPnzNJsbpzjInzjCIzcDT0G1bEIjf4BAREZF1uMEhIiIi63CDQ0RERNbhBoeIiIis43IcQ0WciEz1zMH/sRuK/GKG6sdQi4gEspJVduZGXCQWTDIc3kD9+OxMw+O6B6TgMQunm/RjvEVEMp/QlYLhA8fgWvp6NobX4cq5y+gW71x40tUsHKWyphGd8DUK8nFhYosfF7xfn1OlsrYgLuysbMFjIOI9uqD47B/z4Nri6aUwP9OaBvPa2j4wRwbP3xPx2h4FjYcw3AdLf6XHToiIiOFsHrJgd8SH0ROviWlDfwqvidHvHlcZGmEgIlLSkgPzBh8uwEXXSmMTXuuE8J8sKVU3AfRNwiNpMhJwY0B+0kV8fIEElV3sNDSMgLUiIjdmlansT9fgtZcSGkMjIlLxA50F2/A4CjGMakg7gO95Oa9si+jYRLq+JvgNDhEREVmHGxwiIiKyDjc4REREZB1ucIiIiMg63OAQERGRdbp+TvIlfCR66PgJmLt14b3kfX7JDkNaDHmG4E6X6B5ETbYJ+/D4hZxXddU/7gsx8xbjkRUn+uvxFm25uFshkIwbCuqydB5MxZ0Nx9YPwQdo6FUYul53Iob349EevZYT+ZU/ZCHuFPPk4jMCD8zoPUJl5TDfuHSiylIf0B2BIiL39MN/swQ3Hp1w2p+psqYQ7lIKO/jETY7RowYuBnHHrqlrsSmAf2ZTp+52Cjv4+4SnCt+H+TN/8z2VueQAXOvyGMarBA1nlwv8TQwd1e52/B6IgHuQoWPN1EXlrb20n6j8BoeIiIisww0OERERWYcbHCIiIrIONzhERERkHW5wiIiIyDpdd1ER0V9M6JiePSMi4gGjzvBUqMvjG9FZ2MXMvkjXBmvOddPB9A4p7+xQmfMOXvu7STNgXj4fr3/ous9UNiPlIFzrc/DHXJuj5yB90YZnJJ40zIvKiMUzqmZn7tI/L4w7sZYsXABzz/bI57YZu6UMXHH6d3f8uqtMRCSQgX/39LRmlTW68d1gTMEpmJ9ZgztHEVOnWFf4DQ4RERFZhxscIiIisg43OERERGQdbnCIiIjIOtzgEBERkXXYRUVERJdVzOa9MB+8Ga//WFJV9sfCe+Ha+nH9YF43UmeefNwVleLFHUZ1F1Jg/unuMSrLXqnn1YmIeCTybqnu4gQi77qKP1gJ87pPdAdUdjWeX1kuxTBP3rof5qgP0QlFPxuT3+AQERGRdbjBISIiIutwg0NERETW4QaHiIiIrMMiYyIi6vWCFadhnmbK13z9n5nx9V/i8ghHXrAbunAB5rkv4zwaUQxAiW5cypf4DQ4RERFZhxscIiIisg43OERERGQdbnCIiIjIOtzgEBERkXVczleoTCYiIiLqyfgNDhEREVmHGxwiIiKyDjc4REREZB1ucIiIiMg63OAQERGRdbjBISIiIuv8L/Ain7faBLIBAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x720 with 9 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6x_0GcYQOsy"
      },
      "source": [
        "### Define the Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w0zuHJd8O366"
      },
      "source": [
        "def training_model():\n",
        "    model = tf.keras.Sequential(\n",
        "        [\n",
        "            Conv2D(16, (5, 5), activation=\"relu\", input_shape=(28, 28, 1)),\n",
        "            MaxPooling2D(pool_size=(2, 2)),\n",
        "            Conv2D(32, (5, 5), activation=\"relu\"),\n",
        "            MaxPooling2D(pool_size=(2, 2)),\n",
        "            Dropout(0.2),\n",
        "            GlobalAvgPool2D(),\n",
        "            Flatten(),\n",
        "            Dense(128, activation=\"relu\"),\n",
        "            Dense(NUM_CLASSES, activation=\"softmax\"),\n",
        "        ]\n",
        "    )\n",
        "    return model"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y-doUhJcRniS"
      },
      "source": [
        "initial_model = training_model()\n",
        "initial_model.save_weights(\"initial_weights.h5\")"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QlRuYCWXQnX0",
        "outputId": "63e2c521-258d-47bc-ba23-55d7f4f2c985"
      },
      "source": [
        "model = training_model()\n",
        "model.load_weights(\"initial_weights.h5\")\n",
        "\n",
        "model.summary()\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss=\"categorical_crossentropy\",\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(train_ds, validation_data=test_ds, epochs=EPOCHS)\n",
        "\n",
        "test_loss, test_acc = model.evaluate(test_ds)\n",
        "print(\"Baseline Test accuracy: {:.2f}%\".format(test_acc * 100))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_2 (Conv2D)            (None, 24, 24, 16)        416       \n",
            "_________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2 (None, 12, 12, 16)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 8, 8, 32)          12832     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_3 (MaxPooling2 (None, 4, 4, 32)          0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 4, 4, 32)          0         \n",
            "_________________________________________________________________\n",
            "global_average_pooling2d_1 ( (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "flatten_1 (Flatten)          (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 128)               4224      \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 10)                1290      \n",
            "=================================================================\n",
            "Total params: 18,762\n",
            "Trainable params: 18,762\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/10\n",
            "938/938 [==============================] - 36s 4ms/step - loss: 1.1849 - accuracy: 0.5637 - val_loss: 0.6812 - val_accuracy: 0.7387\n",
            "Epoch 2/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.6577 - accuracy: 0.7471 - val_loss: 0.5951 - val_accuracy: 0.7766\n",
            "Epoch 3/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.5712 - accuracy: 0.7883 - val_loss: 0.5265 - val_accuracy: 0.8058\n",
            "Epoch 4/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.5057 - accuracy: 0.8133 - val_loss: 0.4839 - val_accuracy: 0.8286\n",
            "Epoch 5/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.4640 - accuracy: 0.8312 - val_loss: 0.4451 - val_accuracy: 0.8378\n",
            "Epoch 6/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.4373 - accuracy: 0.8421 - val_loss: 0.4283 - val_accuracy: 0.8503\n",
            "Epoch 7/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.4087 - accuracy: 0.8514 - val_loss: 0.3965 - val_accuracy: 0.8598\n",
            "Epoch 8/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.3986 - accuracy: 0.8545 - val_loss: 0.3911 - val_accuracy: 0.8580\n",
            "Epoch 9/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.3746 - accuracy: 0.8627 - val_loss: 0.3685 - val_accuracy: 0.8712\n",
            "Epoch 10/10\n",
            "938/938 [==============================] - 3s 4ms/step - loss: 0.3668 - accuracy: 0.8661 - val_loss: 0.3648 - val_accuracy: 0.8697\n",
            "157/157 [==============================] - 0s 2ms/step - loss: 0.3648 - accuracy: 0.8697\n",
            "Baseline Test accuracy: 86.97%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Q3xCi53UKI-"
      },
      "source": [
        "### Save the Baseline Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oWI8IMPJRHRf",
        "outputId": "6deb8deb-71be-48af-b1d1-9131ca8c413b"
      },
      "source": [
        "_, keras_file = tempfile.mkstemp('.h5')\n",
        "tf.keras.models.save_model(model, keras_file, include_optimizer=False)\n",
        "\n",
        "print('Saved Baseline Model to:', keras_file)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saved Baseline Model to: /tmp/tmp3nbcp8gi.h5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z-ZRh1WJWVKJ"
      },
      "source": [
        "## Fine-tune Model with Pruning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "11fv5se_UUCn"
      },
      "source": [
        "prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_s8QxLsXMdk"
      },
      "source": [
        "### Define the Hyperparamteres"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TjuK6RlWXG1P"
      },
      "source": [
        "VALIDATION_SPLIT = 0.1 \n",
        "EPOCHS=6"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZcC8Vdf1W3yk"
      },
      "source": [
        "images, labels = next(iter(train_ds))\n",
        "\n",
        "num_images = images.shape[0] * (1 - VALIDATION_SPLIT)\n",
        "end_step = np.ceil(num_images / BATCH_SIZE).astype(np.int32) * EPOCHS"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jRjsO4y_XdgZ"
      },
      "source": [
        "### Define Model for Pruning\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tumeNSuvXZXv"
      },
      "source": [
        "pruning_params = {\n",
        "      'pruning_schedule': tfmot.sparsity.keras.PolynomialDecay(initial_sparsity=0.50,\n",
        "                                                               final_sparsity=0.80,\n",
        "                                                               begin_step=0,\n",
        "                                                               end_step=end_step)\n",
        "}"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0Pa-ODQvXkmr",
        "outputId": "af15185e-e7aa-46e7-c803-9b8490276744"
      },
      "source": [
        "model = training_model()\n",
        "model.load_weights(\"initial_weights.h5\")\n",
        "\n",
        "model_for_pruning = prune_low_magnitude(model, **pruning_params)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py:2281: UserWarning: `layer.add_variable` is deprecated and will be removed in a future version. Please use `layer.add_weight` method instead.\n",
            "  warnings.warn('`layer.add_variable` is deprecated and '\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SfJFfuaCX8iR"
      },
      "source": [
        "### `prune_low_magnitude` requires a recompile."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vUmmm6oqXw8D",
        "outputId": "8abbe484-312f-4d87-f7fc-4c9797a563fb"
      },
      "source": [
        "model_for_pruning.compile(optimizer='adam',\n",
        "              loss=\"categorical_crossentropy\",\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model_for_pruning.summary()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "prune_low_magnitude_conv2d_4 (None, 24, 24, 16)        818       \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_max_pool (None, 12, 12, 16)        1         \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_conv2d_5 (None, 8, 8, 32)          25634     \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_max_pool (None, 4, 4, 32)          1         \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_dropout_ (None, 4, 4, 32)          1         \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_global_a (None, 32)                1         \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_flatten_ (None, 32)                1         \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_dense_4  (None, 128)               8322      \n",
            "_________________________________________________________________\n",
            "prune_low_magnitude_dense_5  (None, 10)                2572      \n",
            "=================================================================\n",
            "Total params: 37,351\n",
            "Trainable params: 18,762\n",
            "Non-trainable params: 18,589\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kK1vAql8X3tj",
        "outputId": "1a07ffbe-27ef-48fe-b9bc-200feb0839a3"
      },
      "source": [
        "logdir = tempfile.mkdtemp()\n",
        "\n",
        "callbacks = [\n",
        "  tfmot.sparsity.keras.UpdatePruningStep(),\n",
        "  tfmot.sparsity.keras.PruningSummaries(log_dir=logdir),\n",
        "]\n",
        "\n",
        "model_for_pruning.fit(train_ds, validation_data=test_ds, epochs=EPOCHS, callbacks=callbacks)\n",
        "_, model_for_pruning_accuracy = model_for_pruning.evaluate(test_ds)\n",
        "print(\"Pruned test accuracy: {:.2f}%\".format(model_for_pruning_accuracy * 100))"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/6\n",
            "  3/938 [..............................] - ETA: 4:22 - loss: 2.2953 - accuracy: 0.1276WARNING:tensorflow:Callback method `on_train_batch_begin` is slow compared to the batch time (batch time: 0.0076s vs `on_train_batch_begin` time: 0.0519s). Check your callbacks.\n",
            "WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0076s vs `on_train_batch_end` time: 0.0414s). Check your callbacks.\n",
            "938/938 [==============================] - 12s 10ms/step - loss: 1.3383 - accuracy: 0.5146 - val_loss: 0.7343 - val_accuracy: 0.7356\n",
            "Epoch 2/6\n",
            "938/938 [==============================] - 9s 10ms/step - loss: 0.7143 - accuracy: 0.7331 - val_loss: 0.6354 - val_accuracy: 0.7685\n",
            "Epoch 3/6\n",
            "938/938 [==============================] - 9s 10ms/step - loss: 0.6257 - accuracy: 0.7637 - val_loss: 0.5806 - val_accuracy: 0.7815\n",
            "Epoch 4/6\n",
            "938/938 [==============================] - 9s 9ms/step - loss: 0.5742 - accuracy: 0.7855 - val_loss: 0.5425 - val_accuracy: 0.8047\n",
            "Epoch 5/6\n",
            "938/938 [==============================] - 9s 9ms/step - loss: 0.5300 - accuracy: 0.8056 - val_loss: 0.5064 - val_accuracy: 0.8163\n",
            "Epoch 6/6\n",
            "938/938 [==============================] - 9s 9ms/step - loss: 0.4993 - accuracy: 0.8198 - val_loss: 0.4850 - val_accuracy: 0.8290\n",
            "157/157 [==============================] - 0s 2ms/step - loss: 0.4850 - accuracy: 0.8290\n",
            "Pruned test accuracy: 82.90%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p3omtvNUbcbw"
      },
      "source": [
        "### Save the Pruning Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1f1Nu3TRbcIH",
        "outputId": "03bad57a-4de1-41b0-d55f-c943f3541d83"
      },
      "source": [
        "model_for_export = tfmot.sparsity.keras.strip_pruning(model_for_pruning)\n",
        "\n",
        "_, pruned_keras_file = tempfile.mkstemp('.h5')\n",
        "tf.keras.models.save_model(model_for_export, pruned_keras_file, include_optimizer=False)\n",
        "print('Saved pruned Keras model to:', pruned_keras_file)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saved pruned Keras model to: /tmp/tmpx45i9kk8.h5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gC0k2_SJeTSO"
      },
      "source": [
        "## Saving the TF-Lite Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kfxqBBflcgkE"
      },
      "source": [
        "TF_LITE_MODEL_FILE_NAME = 'simple_model.tflite'\n",
        "TF_LITE_PRUNED_MODEL_FILE_NAME = 'pruned_model.tflite'"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1n4nP6Ve3Cp"
      },
      "source": [
        "### Convert the Simple Model to TF-Lite"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X2Xi3gcweoXs",
        "outputId": "1ff1d820-3305-4b85-e0a0-313c6a9bf7bd"
      },
      "source": [
        "tf_lite_converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
        "tflite_model = tf_lite_converter.convert()\n",
        "\n",
        "tflite_model_name = TF_LITE_MODEL_FILE_NAME\n",
        "open(tflite_model_name, \"wb\").write(tflite_model)\n",
        "\n",
        "convert_bytes(get_file_size(TF_LITE_MODEL_FILE_NAME), \"KB\")"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: /tmp/tmpzvz7ldko/assets\n",
            "File Size: 76.684Kilobytes\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Y-97M4afEm6"
      },
      "source": [
        "### Checking the Input Tensor Shape"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e818Y-zee9gV",
        "outputId": "514ba293-52d1-4978-9f26-2143cee3bbcd"
      },
      "source": [
        "interpreter = tf.lite.Interpreter(model_path = TF_LITE_MODEL_FILE_NAME)\n",
        "input_details = interpreter.get_input_details()\n",
        "output_details = interpreter.get_output_details()\n",
        "\n",
        "print(\"Input Shape:\", input_details[0]['shape'])\n",
        "print(\"Input Type:\", input_details[0]['dtype'])\n",
        "print(\"Output Shape:\", output_details[0]['shape'])\n",
        "print(\"Output Type:\", output_details[0]['dtype'])"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input Shape: [ 1 28 28  1]\n",
            "Input Type: <class 'numpy.float32'>\n",
            "Output Shape: [ 1 10]\n",
            "Output Type: <class 'numpy.float32'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xRfwtdQNfMMc"
      },
      "source": [
        "### Resize Tensor Shape"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OLiFdfcRfKKT",
        "outputId": "817cb027-547e-4215-b3eb-fa279ba799df"
      },
      "source": [
        "interpreter.resize_tensor_input(input_details[0]['index'], (10000, 28, 28, 1))\n",
        "interpreter.resize_tensor_input(output_details[0]['index'], (10000, 10))\n",
        "interpreter.allocate_tensors()\n",
        "\n",
        "input_details = interpreter.get_input_details()\n",
        "output_details = interpreter.get_output_details()\n",
        "\n",
        "print(\"Input Shape:\", input_details[0]['shape'])\n",
        "print(\"Input Type:\", input_details[0]['dtype'])\n",
        "print(\"Output Shape:\", output_details[0]['shape'])\n",
        "print(\"Output Type:\", output_details[0]['dtype'])"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input Shape: [10000    28    28     1]\n",
            "Input Type: <class 'numpy.float32'>\n",
            "Output Shape: [10000    10]\n",
            "Output Type: <class 'numpy.float32'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l7eJGRHwh-yj"
      },
      "source": [
        "### Make predictions on Baseline TF-Lite Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pNGtR_VMfaI9",
        "outputId": "9206702e-b228-4e11-d34a-1faaff5c1328"
      },
      "source": [
        "test_imgs_numpy = np.array(x_test, dtype=np.float32)\n",
        "\n",
        "interpreter.set_tensor(input_details[0]['index'], test_imgs_numpy)\n",
        "interpreter.invoke()\n",
        "\n",
        "tflite_model_predictions = interpreter.get_tensor(output_details[0]['index'])\n",
        "\n",
        "print(\"Prediction results shape:\", tflite_model_predictions.shape)\n",
        "prediction_classes = np.argmax(tflite_model_predictions, axis=1)\n",
        "\n",
        "acc = accuracy_score(prediction_classes, test_labels)\n",
        "print('Test accuracy TFLITE Baseline Model :', acc)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Prediction results shape: (10000, 10)\n",
            "Test accuracy TFLITE Baseline Model : 0.829\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EbFreY2RhFXW"
      },
      "source": [
        "### Convert the pruned model to tflite"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-2ZyS7I0f8Lr",
        "outputId": "5424f035-7950-46e7-9e5c-70c6c63e0c59"
      },
      "source": [
        "tf_lite_converter = tf.lite.TFLiteConverter.from_keras_model(model_for_export)\n",
        "tflite_model = tf_lite_converter.convert()\n",
        "\n",
        "tflite_model_name = TF_LITE_PRUNED_MODEL_FILE_NAME\n",
        "open(tflite_model_name, \"wb\").write(tflite_model)\n",
        "\n",
        "convert_bytes(get_file_size(TF_LITE_PRUNED_MODEL_FILE_NAME), \"KB\")"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: /tmp/tmpobl8nwfr/assets\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Assets written to: /tmp/tmpobl8nwfr/assets\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "File Size: 76.684Kilobytes\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pIZeH7p5hp75"
      },
      "source": [
        "### Check the Input Tensor Shape"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oPD0YCwhVkd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b0bd0941-a210-4c05-f767-80e3055814e9"
      },
      "source": [
        "interpreter = tf.lite.Interpreter(model_path = TF_LITE_PRUNED_MODEL_FILE_NAME)\n",
        "input_details = interpreter.get_input_details()\n",
        "output_details = interpreter.get_output_details()\n",
        "\n",
        "print(\"Input Shape:\", input_details[0]['shape'])\n",
        "print(\"Input Type:\", input_details[0]['dtype'])\n",
        "print(\"Output Shape:\", output_details[0]['shape'])\n",
        "print(\"Output Type:\", output_details[0]['dtype'])"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input Shape: [ 1 28 28  1]\n",
            "Input Type: <class 'numpy.float32'>\n",
            "Output Shape: [ 1 10]\n",
            "Output Type: <class 'numpy.float32'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "or3W7CnAhwgc"
      },
      "source": [
        "### Resize the Tensor Shape"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E1p5ln9-haG-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ad29401c-3699-424f-f6c5-31bc9ab11d13"
      },
      "source": [
        "interpreter.resize_tensor_input(input_details[0]['index'], (10000, 28, 28, 1))\n",
        "interpreter.resize_tensor_input(output_details[0]['index'], (10000, 10))\n",
        "interpreter.allocate_tensors()\n",
        "\n",
        "input_details = interpreter.get_input_details()\n",
        "output_details = interpreter.get_output_details()\n",
        "\n",
        "print(\"Input Shape:\", input_details[0]['shape'])\n",
        "print(\"Input Type:\", input_details[0]['dtype'])\n",
        "print(\"Output Shape:\", output_details[0]['shape'])\n",
        "print(\"Output Type:\", output_details[0]['dtype'])"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input Shape: [10000    28    28     1]\n",
            "Input Type: <class 'numpy.float32'>\n",
            "Output Shape: [10000    10]\n",
            "Output Type: <class 'numpy.float32'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcKk40nSh1Zp"
      },
      "source": [
        "### Make predictions on Pruned TF-Lite Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oRORnk94he74",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1047e19c-5a3f-49a1-9982-293c61bd313b"
      },
      "source": [
        "test_imgs_numpy = np.array(x_test, dtype=np.float32)\n",
        "\n",
        "interpreter.set_tensor(input_details[0]['index'], test_imgs_numpy)\n",
        "interpreter.invoke()\n",
        "\n",
        "tflite_model_predictions = interpreter.get_tensor(output_details[0]['index'])\n",
        "\n",
        "print(\"Prediction results shape:\", tflite_model_predictions.shape)\n",
        "prediction_classes = np.argmax(tflite_model_predictions, axis=1)\n",
        "\n",
        "acc_pruned = accuracy_score(prediction_classes, test_labels)\n",
        "print('Test accuracy TFLITE Pruned Model :', acc_pruned)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Prediction results shape: (10000, 10)\n",
            "Test accuracy TFLITE Pruned Model : 0.829\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F_VDR3tMqD5z"
      },
      "source": [
        "## **Baseline Model vs Pruned Model**\n",
        "\n",
        "| Configuration          | Baseline Model | Pruned Model |\n",
        "|------------------------|----------------|--------------|\n",
        "| EPOCHS                 | 10             | 6            |\n",
        "| Model Accuracy         | 86.97%         | 82.90%       |\n",
        "| TF-Lite File Size      | 76.684 KB      | 76.684 KB    |\n",
        "| TF-Lite Model Accuracy | 82.90%         | 82.90%       |\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "csqbWIgRqZuI"
      },
      "source": [
        "## Note\n",
        "\n",
        "This notebook shows an example of Pruning with Tensorflow with the help of Fashion MNIST Dataset.\n",
        "\n",
        "[Pruning](https://www.tensorflow.org/model_optimization/guide/pruning) is one of the method to optimise the tensorflow models\n",
        "For a Comprehensive Guide you can click [here](https://www.tensorflow.org/model_optimization/guide/pruning/comprehensive_guide)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uv3AKa6PsYm1"
      },
      "source": [
        "# By [Sayan Nath](https://github.com/sayannath)"
      ]
    }
  ]
}