08a. Practical_NeuralNets with Bias.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see the actual derivation of these equations \n",
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo(\"WRr2e7mKCUY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical Implementation of Neural Networks\n",
    "\n",
    "We define a new objective function as:\n",
    "\n",
    "$$ J(\\mathbf{W})=-[\\mathbf{y}^{(i)}\\ln[\\mathbf{a}^{(L+1)}]^{(i)}+(1-\\mathbf{y}^{(i)})\\ln(1-[\\mathbf{a}^{(L+1)}]^{(i)}) ]$$\n",
    "\n",
    "Notice that this is highly similar to the log-likelihood we derived for logistic regression.\n",
    "\n",
    "Such that the new update for the final layer becomes:\n",
    "\n",
    "$$ \\left[\\frac{\\partial J(\\mathbf{W})}{\\mathbf{z}^{(L)}}\\right]^{(i)} = ([\\mathbf{a}^{(L+1)}]^{(i)}- \\mathbf{y}^{(i)}) $$\n",
    "\n",
    "$$ \\left[\\frac{\\partial J(\\mathbf{W})}{\\mathbf{z}^{(2)}}\\right]^{(i)} = ([\\mathbf{a}^{(3)}]^{(i)}- \\mathbf{y}^{(i)})$$\n",
    "\n",
    "And this is straight forward to convert to vectorized math for the sensitivity:\n",
    "\n",
    "$$ \\mathbf{V}^{(2)} = \\mathbf{A}^{(3)}-\\mathbf{Y} $$\n",
    "\n",
    "Which can be derived as follows\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "J(\\mathbf{W})&=-\\left[\\mathbf{y}^{(i)}\\ln\\phi([\\mathbf{z}^{(L)}]^{(i)})+(1-\\mathbf{y}^{(i)})\\ln(1-[\\phi(\\mathbf{z}^{(L)}]^{(i)})) \\right] \\\\\n",
    "\\frac{\\partial J}{\\partial \\mathbf{z}^{(L)}} &=  -\\left[\\frac{\\mathbf{y}^{(i)}}{\\phi([\\mathbf{z}^{(L)})]^{(i)}}\\left( \\frac{\\partial}{\\partial \\mathbf{z}^{(L)}}\\phi([\\mathbf{z}^{(L)}]^{(i)}) \\right)\n",
    "+\\frac{(1-\\mathbf{y}^{(i)})}{(1-[\\phi(\\mathbf{z}^{(L)})]^{(i)})}\\left( \\frac{\\partial}{\\partial \\mathbf{z}^{(L)}}(1-[\\phi(\\mathbf{z}^{(L)}]^{(i)}))\\right) \\right] \\\\\n",
    " &=  -\\left[\\frac{\\mathbf{y}^{(i)}}{\\phi([\\mathbf{z}^{(L)}]^{(i)})} \\phi([\\mathbf{z}^{(L)}]^{(i)})\\left(1-\\phi([\\mathbf{z}^{(L)}]^{(i)}) \\right)\n",
    "+\\left(\\frac{1-\\mathbf{y}^{(i)}}{1-[\\phi(\\mathbf{z}^{(L)}]^{(i)})}\\right)  \\left(-\\phi([\\mathbf{z}^{(L)}]^{(i)})\\left(1-\\phi([\\mathbf{z}^{(L)}]^{(i)}) \\right)\\right) \\right] \\\\\n",
    " &=  -\\left[\\mathbf{y}^{(i)}\\left(1-\\phi([\\mathbf{z}^{(L)}]^{(i)}) \\right)\n",
    "+\\left(1-\\mathbf{y}^{(i)}\\right)  \\left(-\\phi([\\mathbf{z}^{(L)}]^{(i)})\\right) \\right] \\\\\n",
    "&=  -\\left[\\mathbf{y}^{(i)}-\\mathbf{y}^{(i)}\\phi([\\mathbf{z}^{(L)}]^{(i)}) \n",
    "-\\phi([\\mathbf{z}^{(L)}]^{(i)})+\\phi([\\mathbf{z}^{(L)}]^{(i)})\\mathbf{y}^{(i)}   \\right] \\\\\n",
    "&=  -\\left[\\mathbf{y}^{(i)}-\\phi([\\mathbf{z}^{(L)}]^{(i)})   \\right] \\\\\n",
    "&=  \\phi([\\mathbf{z}^{(L)}]^{(i)}) - \\mathbf{y}^{(i)} \\\\\n",
    "&=  [\\mathbf{a}^{(L+1)}]^{(i)} - \\mathbf{y}^{(i)} \\\\\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option is to choose the softmax, which is defined as:\n",
    "\n",
    "$$  a_j^{(L+1)} = \\frac{\\exp(z_j^{(L)})}{\\sum_i \\exp(z_i^{(L)})}  $$\n",
    "\n",
    "However, it has an identical update equation as the sigmoid final layer using cross entropy. We will stick with cross entropy in order to keep the consistency to previous feedforward code. However, please note that that sigmoid activation functions are more susceptible to gradient instability. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example adapted from https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "# Original Author: Sebastian Raschka\n",
    "\n",
    "# This is the optional book we use in the course, excellent intuitions and straightforward programming examples\n",
    "# please note, however, that this code has been manipulated to reflect our assumptions and notation.\n",
    "import numpy as np\n",
    "from scipy.special import expit\n",
    "import pandas as pd\n",
    "import sys\n",
    "\n",
    "# start with a simple base classifier, which can't be fit or predicted\n",
    "# it only has internal classes to be used by classes that will subclass it\n",
    "class TwoLayerPerceptronBase(object):\n",
    "    def __init__(self, n_hidden=30,\n",
    "                 C=0.0, epochs=500, eta=0.001, random_state=None):\n",
    "        np.random.seed(random_state)\n",
    "        self.n_hidden = n_hidden\n",
    "        self.l2_C = C\n",
    "        self.epochs = epochs\n",
    "        self.eta = eta\n",
    "        \n",
    "    @staticmethod\n",
    "    def _encode_labels(y):\n",
    "        \"\"\"Encode labels into one-hot representation\"\"\"\n",
    "        onehot = pd.get_dummies(y).values.T\n",
    "            \n",
    "        return onehot\n",
    "\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        W1_num_elems = (self.n_features_)*self.n_hidden\n",
    "        W1 = np.random.uniform(-1.0, 1.0, size=W1_num_elems)\n",
    "        W1 = W1.reshape(self.n_hidden, self.n_features_) # reshape to be W\n",
    "        b1 = np.zeros((self.n_hidden, 1))\n",
    "        \n",
    "        W2_num_elems = (self.n_hidden)*self.n_output_\n",
    "        W2 = np.random.uniform(-1.0, 1.0, size=W2_num_elems)\n",
    "        W2 = W2.reshape(self.n_output_, self.n_hidden)\n",
    "        b2 = np.zeros((self.n_output_, 1))\n",
    "        \n",
    "        return W1, W2, b1, b2\n",
    "    \n",
    "    @staticmethod\n",
    "    def _sigmoid(z):\n",
    "        \"\"\"Use scipy.special.expit to avoid overflow\"\"\"\n",
    "        # 1.0 / (1.0 + np.exp(-z))\n",
    "        return expit(z)\n",
    "    \n",
    "    \n",
    "    @staticmethod\n",
    "    def _L2_reg(lambda_, W1, W2):\n",
    "        \"\"\"Compute L2-regularization cost\"\"\"\n",
    "        # only compute for non-bias terms\n",
    "        return (lambda_/2.0) * np.sqrt(np.mean(W1[:, 1:] ** 2) + np.mean(W2[:, 1:] ** 2))\n",
    "    \n",
    "    def _cost(self,A3,Y_enc,W1,W2):\n",
    "        '''Get the objective function value'''\n",
    "        cost = np.mean((Y_enc-A3)**2)\n",
    "        L2_term = self._L2_reg(self.l2_C, W1, W2)\n",
    "        return cost + L2_term\n",
    "    \n",
    "    def _feedforward(self, X, W1, W2, b1, b2):\n",
    "        \"\"\"Compute feedforward step\n",
    "        -----------\n",
    "        X : Input layer with original features.\n",
    "        W1: Weight matrix for input layer -> hidden layer.\n",
    "        W2: Weight matrix for hidden layer -> output layer.\n",
    "        ----------\n",
    "        a1-a3 : activations into layer (or output layer)\n",
    "        z1-z2 : layer inputs \n",
    "\n",
    "        \"\"\"\n",
    "        A1 = X.T\n",
    "        Z1 = W1 @ A1 + b1\n",
    "        A2 = self._sigmoid(Z1)\n",
    "        Z2 = W2 @ A2 + b2\n",
    "        A3 = self._sigmoid(Z2)\n",
    "        return A1, Z1, A2, Z2, A3\n",
    "    \n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        # vectorized backpropagation\n",
    "        V2 = -2*(Y_enc-A3)*A3*(1-A3)\n",
    "        V1 = A2*(1-A2)*(W2.T @ V2)\n",
    "        \n",
    "        gradW2 = V2 @ A2.T\n",
    "        gradW1 = V1 @ A1.T\n",
    "        \n",
    "        gradb2 = np.sum(V2, axis=1).reshape((-1,1))\n",
    "        gradb1 = np.sum(V1, axis=1).reshape((-1,1))\n",
    "        \n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        gradW1 += W1 * self.l2_C\n",
    "        gradW2 += W2 * self.l2_C\n",
    "\n",
    "        return gradW1, gradW2, gradb1, gradb2\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\"\"\"\n",
    "        _, _, _, _, A3 = self._feedforward(X, self.W1, self.W2, self.b1, self.b2)\n",
    "        y_pred = np.argmax(A3, axis=0)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Momentum, Mini-batch and Cooling\n",
    "Like in the previous notebook, we will add in momentum, cooling and min-batching to the implementation to get a better optimizer. \n",
    "\n",
    "As a bonus, let's add support for keeping track of the validation in the training loop. That is, we will add in a calculation to test the validation loss and accuracy at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "# just start with the vectorized version and minibatch\n",
    "class TLPMiniBatch(TwoLayerPerceptronBase):\n",
    "    def __init__(self, alpha=0.0, decrease_const=0.1, \n",
    "                 decrease_iter = 10, shuffle=True, \n",
    "                 minibatches=1, **kwds):        \n",
    "        # need to add to the original initializer \n",
    "        self.alpha = alpha\n",
    "        self.decrease_const = decrease_const\n",
    "        self.decrease_iter = decrease_iter\n",
    "        self.shuffle = shuffle\n",
    "        self.minibatches = minibatches\n",
    "        # but keep other keywords\n",
    "        super().__init__(**kwds)\n",
    "        \n",
    "    \n",
    "    def fit(self, X, y, print_progress=False, XY_test=None):\n",
    "        \"\"\" Learn weights from training data. With mini-batch\"\"\"\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2, self.b1, self.b2 = self._initialize_weights()\n",
    "\n",
    "        # start momentum at zero for previous updates\n",
    "        rho_W1_prev = np.zeros(self.W1.shape) # for momentum\n",
    "        rho_W2_prev = np.zeros(self.W2.shape) # for momentum\n",
    "\n",
    "        self.cost_ = []\n",
    "        self.score_ = []\n",
    "        # get starting acc\n",
    "        self.score_.append(accuracy_score(y_data,self.predict(X_data)))\n",
    "        # keep track of validation, if given\n",
    "        if XY_test is not None:\n",
    "            X_test = XY_test[0].copy()\n",
    "            y_test = XY_test[1].copy()\n",
    "            self.val_score_ = []\n",
    "            self.val_score_.append(accuracy_score(y_test,self.predict(X_test)))\n",
    "            self.val_cost_ = []\n",
    "            \n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            # adaptive learning rate\n",
    "            # decrease at certain epochs\n",
    "            eta = self.eta * self.decrease_const**(np.floor(i/self.decrease_iter))\n",
    "\n",
    "            if print_progress>0 and (i+1)%print_progress==0:\n",
    "                sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n",
    "                sys.stderr.flush()\n",
    "\n",
    "            if self.shuffle:\n",
    "                idx_shuffle = np.random.permutation(y_data.shape[0])\n",
    "                X_data, Y_enc, y_data = X_data[idx_shuffle], Y_enc[:, idx_shuffle], y_data[idx_shuffle]\n",
    "\n",
    "            mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n",
    "            mini_cost = []\n",
    "            for idx in mini:\n",
    "\n",
    "                # feedforward\n",
    "                A1, Z1, A2, Z2, A3 = self._feedforward(X_data[idx],\n",
    "                                                       self.W1,\n",
    "                                                       self.W2,\n",
    "                                                       self.b1,\n",
    "                                                       self.b2\n",
    "                                                      )\n",
    "                \n",
    "                cost = self._cost(A3,Y_enc[:, idx],self.W1,self.W2)\n",
    "                mini_cost.append(cost) # this appends cost of mini-batch only\n",
    "\n",
    "                # compute gradient via backpropagation\n",
    "                gradW1, gradW2, gradb1, gradb2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2, \n",
    "                                                  Y_enc=Y_enc[:, idx],\n",
    "                                                  W1=self.W1,W2=self.W2)\n",
    "\n",
    "                # momentum calculations\n",
    "                rho_W1, rho_W2 = eta * gradW1, eta * gradW2\n",
    "                self.W1 -= (rho_W1 + (self.alpha * rho_W1_prev)) # update with momentum\n",
    "                self.W2 -= (rho_W2 + (self.alpha * rho_W2_prev)) # update with momentum\n",
    "                self.b1 -= eta * gradb1\n",
    "                self.b2 -= eta * gradb2\n",
    "                rho_W1_prev, rho_W2_prev = rho_W1, rho_W2\n",
    "                \n",
    "\n",
    "            self.cost_.append(np.mean(mini_cost))\n",
    "            self.score_.append(accuracy_score(y_data,self.predict(X_data)))\n",
    "            if XY_test is not None:\n",
    "                yhat = self.predict(X_test)\n",
    "                self.val_score_.append(accuracy_score(y_test,yhat))\n",
    "            \n",
    "        return self\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [End review]\n",
    "___\n",
    "\n",
    "## Start of new material"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to implement the new style of objective function, \n",
    "# we just need to update the final layer calculation of the gradient\n",
    "class TLPMiniBatchCrossEntropy(TLPMiniBatch):\n",
    "    def _cost(self,A3,Y_enc,W1,W2):\n",
    "        '''Get the objective function value'''\n",
    "        cost = -np.mean(np.nan_to_num((Y_enc*np.log(A3)+(1-Y_enc)*np.log(1-A3))))\n",
    "        L2_term = self._L2_reg(self.l2_C, W1, W2)\n",
    "        return cost + L2_term\n",
    "    \n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        # vectorized backpropagation\n",
    "        V2 = (A3-Y_enc) # <- this is only line that changed\n",
    "        V1 = A2*(1-A2)*(W2.T @ V2)\n",
    "        \n",
    "        gradW2 = V2 @ A2.T\n",
    "        gradW1 = V1 @ A1.T\n",
    "        \n",
    "        gradb2 = np.sum(V2, axis=1).reshape((-1,1))\n",
    "        gradb1 = np.sum(V1, axis=1).reshape((-1,1))\n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        gradW1 += W1 * self.l2_C\n",
    "        gradW2 += W2 * self.l2_C\n",
    "\n",
    "        return gradW1, gradW2, gradb1, gradb2\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows: 60000, columns: 784\n",
      "Rows: 10000, columns: 784\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# more diverse fashion MNIST data\n",
    "# Let's use Raschka's implementation for using the mnist dataset:\n",
    "# https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "import os\n",
    "import struct\n",
    "import numpy as np\n",
    "# from sklearn.preprocessing import RobustScaler\n",
    " \n",
    "def load_mnist(path, kind='fashion_train'):\n",
    "    \"\"\"Load MNIST data from `path`\"\"\"\n",
    "    labels_path = os.path.join(path, '%s-labels-idx1-ubyte' % kind)\n",
    "    images_path = os.path.join(path, '%s-images-idx3-ubyte' % kind)\n",
    "        \n",
    "    with open(labels_path, 'rb') as lbpath:\n",
    "        magic, n = struct.unpack('>II', lbpath.read(8))\n",
    "        labels = np.fromfile(lbpath, dtype=np.uint8)\n",
    "\n",
    "    with open(images_path, 'rb') as imgpath:\n",
    "        magic, num, rows, cols = struct.unpack(\">IIII\", imgpath.read(16))\n",
    "        images = np.fromfile(imgpath, dtype=np.uint8).reshape(len(labels), 784)\n",
    " \n",
    "    return images, labels\n",
    "\n",
    "X_train, y_train = load_mnist('data/', kind='fashion_train')\n",
    "print('Rows: %d, columns: %d' % (X_train.shape[0], X_train.shape[1]))\n",
    "\n",
    "X_test, y_test = load_mnist('data/', kind='fashion_t10k')\n",
    "print('Rows: %d, columns: %d' % (X_test.shape[0], X_test.shape[1]))\n",
    "\n",
    "X_train = X_train/255.0 - 0.5\n",
    "X_test = X_test/255.0 - 0.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 1/200"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 30.4 s, sys: 2.58 s, total: 33 s\n",
      "Wall time: 8.47 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 20/20"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27.1 s, sys: 2.04 s, total: 29.2 s\n",
      "Wall time: 7.32 s\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "vals = { 'n_hidden':30, \n",
    "         'C':0.1, 'epochs':20, 'eta':0.01, # poor starting learning rate!!\n",
    "         'alpha':0.001, 'decrease_const':0.1, 'decrease_iter':15,\n",
    "         'minibatches':50,\n",
    "         'shuffle':True,'random_state':1}\n",
    "\n",
    "nn_quad = TLPMiniBatch(**vals)\n",
    "nn_cross = TLPMiniBatchCrossEntropy(**vals)\n",
    "\n",
    "%time nn_cross.fit(X_train, y_train, print_progress=1, XY_test=(X_test, y_test))\n",
    "%time nn_quad.fit(X_train, y_train, print_progress=1, XY_test=(X_test, y_test))\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.style.use('ggplot')\n",
    "\n",
    "def print_result(nn,X_train,y_train,X_test,y_test,title=\"\",color=\"red\"):\n",
    "    \n",
    "    print(\"=================\")\n",
    "    print(title,\":\")\n",
    "    yhat = nn.predict(X_train)\n",
    "    print('Resubstitution acc:',accuracy_score(y_train,yhat))\n",
    "    \n",
    "    yhat = nn.predict(X_test)\n",
    "    print('Validation acc:',accuracy_score(y_test,yhat))\n",
    "    \n",
    "    if hasattr(nn,'val_score_'):\n",
    "        plt.plot(range(len(nn.val_score_)), nn.val_score_, color=color,label=title)\n",
    "        plt.ylabel('Validation Accuracy')\n",
    "    else:\n",
    "        plt.plot(range(len(nn.score_)), nn.score_, color=color,label=title)\n",
    "        plt.ylabel('Resub Accuracy')\n",
    "        \n",
    "    plt.xlabel('Epochs')\n",
    "    plt.tight_layout()\n",
    "    plt.legend(loc='best')\n",
    "    plt.grid(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=================\n",
      "Cross Entropy Loss :\n",
      "Resubstitution acc: 0.83525\n",
      "Validation acc: 0.8179\n",
      "=================\n",
      "MSE Loss :\n",
      "Resubstitution acc: 0.11028333333333333\n",
      "Validation acc: 0.1109\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABM00lEQVR4nO3dd3gU5drH8e+WbHrdhYQSLKELghgEEakRj8hRUBEsh5cDiIiFg8oBFFGUJlVFEJEcmg0RsCGKAQQkIAEMUqQEBUFKSEJ63Z15/1hYE1LYkG1J7s915drdmdmZX4Yl987MM8+jUVVVRQghhPAwWncHEEIIIcoiBUoIIYRHkgIlhBDCI0mBEkII4ZGkQAkhhPBIUqCEEEJ4JL27A1TVmTNnqvR+k8lESkqKg9I4j+R0LMnpWNUlJ1SfrLUpZ/369cucLkdQQgghPJIUKCGEEB5JCpQQQgiPVO2vQV1JVVXy8/NRFAWNRnPV5c+fP09BQYELklWN5LSPqqpotVp8fHzs+vcXQniuGleg8vPz8fLyQq+371fT6/XodDonp6o6yWk/s9lMfn4+vr6+bs0hhKiaGneKT1EUu4uTqJn0ej2Korg7hhCiimpcgZLTOgLkcyBETVDjCpQQQoiaQc6FOUFycjKvvvoq+/btw2AwEBkZyWuvvUZUVJRTt/uf//yHnTt3EhgYCICvry9fffVVuctnZGSwdu1aBg8e7NRcV2rSpAnHjh1z6TaFqLYsFjS5uWhycqyPubloc3OhoACNqoKqgqJYH4s/BzRlTb90+lujqlBUhKaoCAoL0RQUlP380k/x58Vfa4cOhX79nPKrS4FyMFVVGTp0KP379+e9994D4MCBA6SkpJQoUBaLxSmNCSZMmECfPn3sWjYzM5Ply5eXWaCclU+I6swrMZHAGTOsf6D1elS9HnS6Uo9lzSs+DVW1FRtNbi7aYsXncjHS5edTLzsbTX6+y39PVa9H9fICb29Ug8H63GBA9fb++7nBgBIUhNaJjZGkQDnY9u3b8fLyYtCgQbZprVq1AiA+Pp45c+YQHh7OwYMH+e677xg/fjy//vorOp2OV199lTvuuIMjR47w/PPPU1hYiKqqLFq0iAYNGjBs2DDOnj2LoiiMGjWK+++/365Ms2fP5q+//uLPP//kr7/+YtiwYQwdOpSpU6dy8uRJ7rrrLrp06ULPnj3tyrdy5Uq+++47CgsL+fPPP+nXrx/PP/88M2bMwGQyMWTIEACmT59OnTp1GDp06FUzHjhwgHHjxpGfn891113H7NmzCQkJITY2lhUrVqDX62nSpAnvvfceO3bsYOLEiYD1WtOaNWsICAio7D+VEJWiP3gQ46OPonp7Y77xRjR5eWgsFjCb0ZjN1iOdS4+YzdZ5xZ8Xe0RVUf38UP39rY9+fij+/ijBwaj16qH6+eFtNJKn1aL4+6P6+v697OVHgwG0WtBoUDUa0Ghsry8/Vy9fi71i+uXnql5vXc+lgqMaDODlZS2idjKZTOCkLplcVqASExNZsmQJiqLQs2dP+vbtW2J+bm4u77zzDqmpqVgsFv75z3/SvXv3Km0zaOJEvA4dqnAZjUZDZUa9L2rZkszXXy93/pEjR2jdunW58xMTE9m0aRONGjVi4cKFAGzcuJGkpCQeeeQRtm3bxooVKxg6dCgPPPAAhYWFWCwWNm/eTEREBCtWrACsRz9lmTx5Mm+//TYAzZo149133wUgKSmJVatWkZOTw5133smgQYN46aWXOHLkCD/88ANgLaD25Lv8e2zcuBFfX1/uvfdeevbsySOPPMITTzzBkCFDUBSFr776im+++cau/fqf//yHN954g9tvv52ZM2cyZ84cXn/9debPn8+OHTvw9vYmIyMDgIULFzJ16lTat29PTk4O3t7edm1DiGulT0rC+MgjqH5+pKxdiyUy0unbNJlMZFaDvvicySWNJBRFITY2lpdeeom5c+eyfft2Tp8+XWKZ7777joYNGzJz5kxee+01li9fjtlsdkU8l2rbti2NGjUCICEhgQcffBCAxo0b07BhQ37//XduvfVW5s2bx/z58zl9+jS+vr60aNGCbdu2MWXKFH7++WeCgoLKXP+ECRP44Ycf+OGHH2zFCaBnz554e3sTFhaGyWTiwoUL15wP4M477yQsLAxfX1/uuecedu3aRWRkJKGhoRw4cIAtW7Zw0003ERYWdtV9kpmZSUZGBrfffjsA/fv35+effwagRYsWPPPMM6xevdp2+0D79u2ZNGkSsbGxZGRkyG0Fwql0f/6JccAA0GhI+fRTlxQnYeWS/9lJSUlEREQQHh4OQKdOnUhISKBhw4a2ZTQaDfn5+baeIAICAtBqq1Y/KzrSuUyv1zu0EDZt2pR169aVO9/Pz8/2vLwjt379+nHLLbewceNGHnvsMWbOnEm3bt1Yv349mzZtYtq0aXTt2pXRo0fbnav4UYZOp8NisVxzPijdjPvy68cee4zPPvuM5ORkBg4caHe+8ixfvpydO3eyYcMG3nrrLTZv3swzzzxDz5492bRpE//85z9ZuXIljRs3rvK2hLiS9uxZjAMHosnPJ2XVKizyOXMplxSotLQ0jEaj7bXRaCzViusf//gHM2bM4MknnyQvL4/Ro0eXWaDi4uKIi4sDrNc4TCZTifnnz5+v9DdqR34D79atG2+++SaffPIJ//rXvwD45ZdfyMvLQ6fTodFobNvr1KkTX3zxBd26deP48eOcOXOGZs2acfr0aaKiomjcuDGnTp3iyJEjNG/enJCQEAYMGEBgYCArV64slVur1aLT6cqcrtVqS0zX6XQEBASQk5Njm25vvkOHDrFt2zaysrLw8fHh+++/56233kKv19O7d29mzJhBUVER77//frkNLYpnCQsLIyQkhN27d9OxY0fWrl1Lp06d0Gq1nDlzhq5du9qyFBQUcO7cOVq3bk3r1q3Zu3cvv//+O82bNy+xfm9v71KfjSu3X9F8TyE5Hc/urBcu4PXYY5CWhvm77wiJjnZ+uGKqyz51Zk6XFKiyvolf+Q183759XHfddUycOJHz58/zxhtv0Lx58xLf6AFiYmKIiYmxvb5yHJKCgoJKtT5z9BEUwOLFi3n11VeZN28e3t7eNGzYkEmTJnHu3DlUVbVt7/HHH2fcuHF07doVnU7HnDlz0Ol0rF27ljVr1qDX66lbty6jRo3iwIEDTJo0CY1Gg5eXF9OmTSuVW1EUJk2axJw5c2zT1q1bh6IoKIpSYnmLxUJQUBDR0dF06dKF7t2707NnT7vyWSwW2rdvz8iRIzlx4gT9+vWjVatWmM1mDAYDt99+O8HBwSXWVVxeXh5t2rSxvR4+fDhz5861NZJo1KgRc+bMoaCggJEjR5KVlYWqqjzxxBP4+/szbdo04uPj0Wq1NG3alK5du5baTkFBQYVj1NSmsXaqKmjCBLy3bSP7qafIe/BB60X0K3hCTnvZk1WTno7p4YdRT54k7aOPKLz+eqc1BChPddmnzhwPSqNWpoXANTp69CirVq3i5ZdfBmDt2rWA9VTWZdOmTaNv3760aNECgEmTJvHYY49d9dTNlQMW5ubmlipqFXFGgXIGT8q5cuVKfv31V6ZMmVJqnlarpWfPnrz//vvceOONbkhndbXPQW36z18VuhMnqHvnnahBQWjT0zFHRpL9zDPk9u8PxU4buztnZVwtqyY7G+Mjj+B14ABpS5ZQ0K2b68IVU132abUfsDAqKoqzZ8+SnJyM2WwmPj6e6CsOl00mE/v37wcgPT2dM2fOULduXVfEEw5y9OhROnToQOfOnd1anITjBMyfD15eJG/aROqKFSgmEyFjx1K3c2f8li4FN9yj41R5eYQNHozXvn1cXLDAbcVJWLnkCApg7969LFu2DEVR6N69Ow888AAbNmwAoFevXqSlpbFgwQIuXrwIwP3330+XLl2uul45gvIsnpJTjqCqTvvXX4TfcQe5jz5KxtSp1omqive2bQTMmYN3QgKWiAiyn3oK31GjSMnJcUvOyip3nxYWEjZ0KN6bN5P+zjvkPfCA68MVU5s+o249xedMUqA8i6fklAJVdUGvvIL/8uUkb9+OpViLWwBUFUN8PIFz5+K9YwdqeDiZw4eTO2gQaiX+/7lDmfvUbCZ05Eh8160jfcYMch97zD3hiqlNn1G3nuITQlQv2gsX8P/4Y/IefLB0cQLQaCi84w5SP/+clNWrUW+6ieA33qBuhw4EzJ+PJjvb9aGvlaIQ8sIL+K5bR8Zrr3lEcRJWUqCEEKX4L1oEhYVkPf30VZct7NgR8/r1XPjiC4ratCFo6lTCO3Qg4O230ZTT44nHUFWCX34Zv88/J3PMGHKeeMLdiUQxUqCE8FSKAm7o9V1z8SL+y5aR989/YqlED/xF7duT9uGHXPjmGwqjowmaMYPwjh0JmDMHTXq68wJfK1UlaPJk/JcvJ2vkSLJHjXJ3InEFKVBO0KBBA5599lnba7PZTOvWrW0dyF64cIFBgwYRExNDt27dbDf0njp1iqioKO666y7bz6pVq0qt/6GHHmLfvn2u+WWE2/h99BGGVq3w3rzZpdsN+N//0ObkkF3sM1wZRbfcQtqyZVxYv56C228naPZswjt2JPSJJwgeP57A2bPxW7oUn6+/xrBjB/pjx9CkpdmGgXCVgLfeImDhQnL+7//IeuklaweqwqNIJ2ZO4Ofnx5EjR8jLy8PX15etW7cSERFhmz9z5ky6dOnCsGHDADhUrEPb6667ztZ5q6jFVBX///0PgOCXXyZ540Zw4rAGl2mysvD/3//Iu/tuzJfuSbxWRTffzMXYWLIOHiRgwQK8Dh5Eu2MHukstda+k6nQoYWEoJhOK0Yjl0qNiMqGYTFhMJiz162OJjEQNDq5SNv/33ydo1ixy+/cnY/JkKU4eSgqUk3Tv3p2NGzfSp08fvvjiC/r27WvrADU5OZmuXbvalm3ZsmWVt3fx4kVeeOEF/vzzT3x8fJgxYwYtW7Ysc2iKnJwcnnrqKbKysrBYLEybNo0OHTpUOYNwHEN8PF5Hj2IZNAj98uUEvv02WePGOX27/suXo01PJ/u55xy2TvNNN5E+f36xCWa0Fy+iTUlBm5KCLjXV+jw11fqTkoIuJQVDYqJ1ehkNLpSQEMyRkVgiI7Fcd531+eXHhg1L3ER8Je3ixQS//jp5ffqQPmuWdfgJ4ZFqdIGaODGIQ4dKd8tSXGWH22jZsojXX7/6hd/777+fuXPnEhMTw2+//cbAgQNtBWrw4ME89dRTLFmyhDvvvJMBAwbYjrAuj8902eTJk+0qHrNnz6ZVq1b873//46effmLUqFH88MMPZQ5N8eGHH9K1a1dGjRqFxWIhLy/P7t9fuIb/0qUoISFY3nmHgtxcAhYuJO+BBzA3beq0bWry8vBftIj8rl0patvWadtBr0epUwelTh37ls/PR5uaii4lBd3p0+hOnUL/55/o/vwT/ZEj+MTFWQcQvETVaFAiIjA3aoQlMhLzdddZC1mjRuiTktCNHUt+z55cnDcPpCd8jyb/Ok7SsmVLTp8+zZdffkmPHj1KzOvWrRvx8fH8+OOPbNq0ibvvvptNmzYB136Kb9euXXzwwQcAdO7cmYsXL5KZmWkbmqJfv37cc8891K9fn7Zt2/LCCy9gNpu5++67bQMqCs+gPXMGn++/J+eJJzD4+pI5cSI+cXEEjx9P6uefO+10lN/HH6NLSeGipzUW8PFBadAApUEDior14WijKGjPn0d/6hS6kydLFDDv7dvxXb3aOrz55cW7dSPt/ffBYHDhLyGuRY0uUPYc6TjzxtJevXrx+uuv8/nnn9t6yLgsNDSUfv360a9fPwYNGsTOnTu5+eabr3lb5XXIW9bQFB07dmT16tVs3LiRUaNGMWLECPr373/N2xaO5f/RR6Ao5AwahAFQjEYyX36ZkDFj8P3sM/IGDHD8RgsKCHjvPQo6dKCwup3u1WpR6tWjsF49uO220vMLCtCdPo3+1Ck0GRkEDBwIctagWpCTr040YMAARo8ebesA97KffvrJdlotOzubkydP0qBBgyptq2PHjqxZswawjowbFhZGYGAgJ06coEWLFjz99NO0adOGpKQkTp8+jclk4rHHHmPgwIG2PhCFBygsxO+jjyjo0QPLddfZJucOHEhB+/YEvfEG2rQ0h2/W7/PP0Z09WzObWnt7Y4mKoqBbN/Lvvx/8/d2dSNipRh9BuVv9+vVtLfWK279/PxMmTECv16MoCo888ght27bl1KlTpa5BDRw4kKFDh5Zax6BBg2xjKt166628+eabPP/888TExODj48Nbb70FWIf+KD40Rffu3fnyyy9ZuHAher0ef39/2xDxwv18v/0W3YULpA8eXHKGVkvG9OnUuftugiZPJr3YkCpVZjYTMH8+hW3aUGBH/5dCuIr0xechfcddjeSsnOraF5+xb190Fy6QvG0baLWlcgZOnUrg/PmkrF5NYceODtmm7+rVhD73HGn/+x/5d999Tevw1P1ZluqStTbllL74hPBw+gMH8E5IIGfQoHKbPmePHo05MpLgceOgWMu1a6YoBMybR1Hz5uQXO3IXwhNIgRLCQ/gvW4bi40NuBY0gVF9fMqZMwevYMQIWLqzyNn3Wr8fr2DGynntO7gcSHqfGfSKr+RlL4SDV7XOgSU/Hd80a8vr1Qw0JqXDZgp49yevdm8C330Z34sS1b1RVCXjnHcw33EB+nz7Xvh4hnKTGFSitVusR10CE+5jNZrTV7GjA77PP0Obnk3Nl44hyZLz+OqpeT/DLL8M1FmPvTZswHDhA1rPPgk53TesQwplqXCs+Hx8f8vPzKSgoQGPHDY3e3t4UFBS4IFnVSE77qKqKVqvFx8fHbRkqTVHwX7aMwuhozHbeNK3Uq0fWf/9L8MSJ+Hz9Nfn33Ve5baoqgW+/jblBA7ePHCtEeWpcgdJoNPhWolPN2tRSxhWqS05P4r1lC/oTJ7j44ouVel/O4MH4rlpF8KuvUtCtG2pQkN3vNcTHY9izh/QpU8Cr4u7AhHCX6nUeRIgayH/pUix16pB3772Ve6NOR8abb6JNSSHozTcr9dbAd97BUrcuuQMHVm6bQriQFCgh3Ej35594b9xI7qOPXlPfcEVt2pAzeDB+y5bhlZho13u89uzB+6efyH7ySahOp0JFreOyU3yJiYksWbIERVHo2bMnffv2LTH/q6++Ytu2bQAoisLp06eJjY0lICDAVRGFcDn/5ctBqyXn8ceveR1Z//0vvuvWETx2LCnr1l21h+7Ad95BCQkh99JAmUJ4KpcUKEVRiI2NZcKECRiNRsaPH090dDQNGza0LXPfffdx36ULvbt372bdunVSnETNlpeH3yefkH/33Sjl3ElvDzUwkIxJkwgbMQL/JUvIeeKJcpfVHziAT1wcmWPGoEqfdMLDueQUX1JSEhEREYSHh6PX6+nUqRMJCQnlLr99+3buuOMOV0QTwm18v/oKbXq63U3LK5Lfpw/5PXoQOHMm2iu6/youcN48lMBAcv797ypvUwhnc8kRVFpaGkaj0fbaaDRy7NixMpctKCggMTGxzA5SAeLi4oiLiwNg+vTpmEymKmXT6/VVXocrSE7HcntOVUX/4YcoLVoQdN995Y7xVKmcCxagaduWulOmYF65svT8337Da906lP/+F2NUVBXCVzGnm1WXrJLTRQWqvLGKyrJnzx6aNWtW7um9mJgYYmJibK+r2qS5ujSLlpyO5e6cXnv3UmfvXtKnTCE3NbXc5SqVMzCQgP/8h6Dp08n65BMKruhbL2TyZPQ+Plx47DEUB//u7t6flVFdstamnG7tLNZoNJJa7D9hamoqoaGhZS67fft2Onfu7IpYQriN/9KlKAEB5D30kEPXm/3kkxQ1bUrwyy+jyc21TdedPInv2rXkPv44SrGzGUJ4MpcUqKioKM6ePUtycjJms5n4+Hiio6NLLZebm8uhQ4fKnCdETaFNScH366/Je+ghVEc3BDIYyJg+Hf1ffxEwd65tcsCCBaDTkT1ihGO3J4QTueQUn06nY8iQIUyZMgVFUejevTuRkZFs2LABsA6NDrBr1y7atGlTvbqpEaKS/D75BE1hoUMaR5SlsEMHcgYOJGDRIvIeeAAlJAS/zz4j9+GHUSIinLJNIZzBZfdBtWvXjnbt2pWYdrkwXdatWze6devmqkhCuJ7Fgt+KFRTccQfmJk2ctpnMl1/GZ8MGQsaOpbBtW7BYyH76aadtTwhnkJ4khLiCJjubkKefxnfVKoev2ycuDv1ffznt6OkyNSyMzFdewbBnDwGxseT164elUSOnblMIR5MCJURxikLIqFH4ffEFof/5D0ETJ4IDh2/xX7IES7165F9x9sAZ8vr3p+D221E1GrKffdbp2xPC0Wpcb+ZCVEXgnDn4fvcdGRMmoDt3joDFi/E6coS0995DDQur0rp1SUl4b9tG5n//e9XuiBxCoyFt8WL0SUmYGzd2/vaEcDApUEJc4rNuHYFz55Lbvz85I0aARkNRy5aEjBtHnT59SFuyBHOzZte8fv/ly1G9vKwdw7qIGhJCkbSKFdWUnOITAtAfPEjIqFEU3nIL6dOn23p2yBswgJTPP0eTl4fpn//E5/vvr2n9mpwc/D77jLw+fVDq1HFkdCFqLClQotbTpqYSNmQIanAwabGxpYagKLr1Vi58+y3mJk0IGzLEen9RJYdZ9129Gm1WltMbRwhRk0iBErVbURGhTz6J7sIF0mJjUcLDy1xMqVePlNWryX3wQYJmzSL0ySdL9NRQIVW1DuneqhVFt97qwPBC1GxSoEStFjxxIt47dpA+cyZFbdtWvLCPD+lvv03GK6/gs349pvvuQ3fq1FW3Yfj5Z7wOHyZ38OByO4UVQpQmBUrUWn4rVuC/fDnZI0aQ9+CD9r1JoyFnxAjSVqxA99dfmHr3xrBjR4Vv8V+yBCUkhLwrBukUQlTMrgL13//+l3Xr1pGenu7kOEK4huannwieMIH87t3JfOmlSr+/oFs3LnzzDUpYGMaBA/FbtqzM5bTnzuHz3XfkDhiA6utb1dhC1Cp2FagHHniA3377jWeffZapU6fy008/UVhY6OxsQjiF7vRp9AMHYmnUiIvz54NOd03rsURFkfL11xR07UrISy8RPHYsXPH/wv+jj8BiIWfQIEdEF6JWses+qI4dO9KxY0eys7OJj4/n+++/Z/Hixdx222106dKFVq1aOTunEA6hyc0lbMgQKCgg9fPPUYODq7Q+NSiItCVLCJwxg8B330V/7BgXFy1CMZmgsBC/Dz+koHt3LNdf75hfQIhapFI36gYEBNC1a1d8fHz46quv+Pnnn/ntt9/QarUMHTqUm2++2Vk5hag6VSVk9Gj0hw5hXrsWi6N6V9DpyBo/3npT7/PPY+rdm7T//Q/98ePokpNJl6blQlwTuwqUoij8+uuvbN26lb1799K0aVP69u3LbbfdhsFgYOfOncybN48PPvjA2XmFuGYB77yD7zffWHv6vucecPBopfn330/qjTcS9u9/Y7r/fpTwcMzXXUdB9+4O3Y4QtYVdBerJJ58kKCiILl268PjjjxN2RZ9kHTt25PtrvMNeCFfw3rCBoBkzyH3gAbKfegpnjThW1Lo1F9avJ/SJJ/BOSCDjlVdAK41lhbgWdhWocePGERUVVeEyr776qkMCCeFo+iNHCH3mGQrbtCF9xgyn34uk1KlD6mef4RMXR35MjFO3JURNZtdXu9OnT3Py5MkS006cOMHWrVudEkoIR9FcvGjtxsjPj7TFi8FVTb0NBvJ79waDwTXbE6IGsqtArVy5EqPRWGKayWTi008/dUooUfto//oLny+/JGjiRILHjsXv44/RHzpUtbGYzGbCnnoK3ZkzpC1ejFK/vuMCCyGczq5TfHl5efj5+ZWY5ufnR05OjlNCiRquqAivQ4cw7N6NYfduvHbvRn/mDACKjw8YDPh/+KH1ta8vRa1aUdS2LUVt21LYti2W666z6zRd0Btv4L1tGxfnzJEhJ4SohuwqUA0bNmTnzp106tTJNm3Xrl00bNjQacFEzaG5eBHDnj1/F6TERLR5eQBY6tWjsH17cqKjKYyOpqhlS9Dp0P3xB4Z9+/BKTMSQmIj/ihVoLrUSVUJCKGzTxlawitq0KdXJq+/KlQQsXkz20KHkDRjg8t9ZCFF1dhWoxx57jGnTphEfH09ERATnzp1j//79jB8/3tn5hCsoCj7r1uH7zTeoBgNqQABKQACqv//fz/38UAMCrK8vTb/8HG/vv49oVBVdUpKtGBl278br2DHrLJ2OoptuIvfRRym89VYKo6NRGjQoM5IlKoq8qCjyHnjAOqGoCP2RIxgSE/Hatw/DL7/g/e67aCwW6/L16lmLVdu2WOrUIWTcOAo6dyZz4kSn7z4hhHNoVNW+gW1SUlL46aefSElJwWQy0blzZ0wmk90bSkxMZMmSJSiKQs+ePelbRseZBw8eZOnSpVgsFgIDA5k0adJV13vm0qmha2UymUhx8P0wzuCUnKqKz/ffEzhrFl6//YYlIgLVYECTnY02JwdNQYF9q9HrbcVKl5eHJi0NuHSk064dhe3bW4+O2rZFveJUcVVo8vLwOnAAr19+sRatxET0J04AYL7uOi588025w7TX6n93J6guOaH6ZK1NOeuXc33Y7p4kTCZTmUXFHoqiEBsby4QJEzAajYwfP57o6OgSpwhzcnJYvHgxL7/8MiaTiYyMjGvalrCDquK9aROBs2Zh+PVXzDfcwMV33yXvvvtK9ktXVPR3scrOLvVck5uL9vLznBy02dl4BwaS1bIlhdHRmKOinHoPkOrray1+7dvbpmkuXsRr/37MTZqUW5yEENWD3QVq9+7dHDp0iMzMzBLTn3nmmau+NykpiYiICMIvXSfo1KkTCQkJJQrUTz/9RIcOHWxHZcFV7CNNlEFVMWzbRtDMmRj27sXcqBEX58yxDjWhL+Oj4OWFGhqKJTTU7k2YTCZy3fitTw0NpbBLF7dtXwjhOHYVqFWrVvHDDz/QqVMndu7cSUxMDNu3b+f222+3ayNpaWklmqkbjUaOXboucdnZs2cxm8289tpr5OXl0bt3b7p27VpqXXFxccTFxQEwffr0Sp1mLIter6/yOlyhqjk1W7eie/11tNu2oUZGYp4/H+X//g9/Ly/8PSinq0hOx6ouOaH6ZJWcdhaozZs3M2HCBBo1asSPP/7I4MGD6dy5M6tXr7ZrI2Vd5tJc0UzYYrHwxx9/8Morr1BYWMiECRNo0qRJqXOTMTExxBS7O7+q5z5r+nler927rUdMP/2EJTyc9ClTyH3kEWvDBiecRq3p+9PVJKfjVZestSlnla5B5eTk0KhRI+sb9HrMZjONGzfm0KFDdm3caDSSmppqe52amkroFaeNjEYjgYGB+Pj44OPjQ4sWLTh58mS5wUXFvPbtI3DWLHw2bcJiMpHx6qvk/OtfrutJQQghqsiuK9gRERGcOnUKgMjISDZs2MDWrVsJCAiwayNRUVGcPXuW5ORkzGYz8fHxRF9x42R0dDSHDx/GYrFQUFBAUlISDcppgizKpz94kNB//5s6vXtj2LuXzJdeInnHDnKGD5fiJISoVuw6ghowYABZWVmA9Z6ot99+m/z8fIYNG2bXRnQ6HUOGDGHKlCkoikL37t1thQ6gV69eNGzYkLZt2/Liiy+i1Wrp0aOH7ahNXJ3+6FECZ83Cd906lKAgMl98kZxhw1ADA90dTQghrslV74NSFIVDhw7RvHlz9GW19HIzuQ8KvDduJOzf/0b18SFn2DCyhw9HDQlxbcBLasL+9CSS0/GqS9balLO8SzlXPcWn1WqZMWOGRxYnAbrffyf0mWcwN29O8s6dZP33v24rTkII4Uh2XYNq0aIFR48edXYWUUma7GzChg5F1elIi41FkRtThRA1iF2HRXXq1GHatGlER0djNBpLNBEfIB1xuoeqEjJ6NPqkJFI/+QRLZKS7EwkhhEPZVaAKCwtpf6k7mbRL/awJ9wp49118v/2WjIkTKezc2d1xhBDC4ewqUCNHjnR2DlEJ3ps2Efjmm+T262dtPi6EEDWQXQXq/Pnz5c4Lv2IcHuFcuj/+sDaKaNGCjJkz7Rq4TwghqiO7CtRzzz1X7ryVK1c6LIyomCYnh7Bhw0CjIS02FlVuvBVC1GB2Fagri1B6ejqrVq2iRYsWTgklynC5UcTRo6R99BEWuYlZCFHDXdNgPSEhIQwePJiPP/7Y0XlEObSzZ+O7bh2ZL71EgQwnIYSoBa55NLkzZ85QYOeIq6JqvH/8Ed2ECeTddx85I0a4O44QQriEXaf4Jk6cWOLep4KCAk6dOsVDDz3ktGDCSnfiBKFPP416002kz54tjSKEELWGXQWqR48eJV77+Phw3XXXUa9ePaeEElaa3FxrowjAvGoVqp+fmxMJIYTr2FWgunXr5uQYohRVJeSFF9AfOULahx8SeOONUA06jhRCCEex6xrUrFmz+O2330pM++2335g9e7ZTQgnwX7gQ36++ImvcOAq6dnV3HCGEcDm7CtShQ4do1qxZiWlNmzbl4MGDTglV23lv3UrQ1Knk9elDtvTiIYSopewqUF5eXuTn55eYlp+fj06nc0qo2kz355+EPvUU5qZNSZ8zRxpFCCFqLbsKVJs2bVi0aBG5ubkA5ObmEhsbS9u2bZ2ZrdbR5OURNnQoqCppixej+vu7O5IQQriNXY0kBg0axLx58xgyZAgBAQFkZ2fTtm1bnn32WWfnqz1UleAXX0T/22+kLV+O5YYb3J1ICCHcyq4CFRAQwPjx40lPTyclJQWTyUSIjNrqUP7vv4/fF1+QOXYsBVc06xdCiNrIrgK1b98+6tSpQ/369W2F6cyZM6SkpHDzzTc7M1+tYNi6laApU8jr3ZtsOSoVQgjAzmtQsbGx+F7Rc7aPjw+xsbF2bygxMZFRo0bx7LPP8sUXX5Saf/DgQf7v//6PMWPGMGbMGD7//HO7112daXJzCX36acyNG5M+d640ihBCiEvsOoLKyMggNDS0xLTQ0FDS09Pt2oiiKMTGxjJhwgSMRiPjx48nOjqahg0blliuRYsWjBs3zr7kNYTP99+jS0vj4vvvowYEuDuOEEJ4DLuOoMLDwzlw4ECJaQcPHqRu3bp2bSQpKYmIiAjCw8PR6/V06tSJhISEyqetgXzXrMFcvz6FHTu6O4oQQngUu46g+vfvz6xZs+jRowfh4eGcP3+ezZs32z0UfFpaGkaj0fbaaDRy7NixUssdPXqUMWPGEBoayr/+9S8iIyNLLRMXF0dcXBwA06dPx2Qy2ZWhPHq9vsrruGbJyXht2YLy/POYrlLs3ZqzEiSnY0lOx6suWSUnaFRVVe1ZMCkpiU2bNpGamorRaKRHjx40btzYro3s2LGDffv2MeLSUBFbt24lKSmJIUOG2JbJzc1Fq9Xi4+PD3r17Wbp0Ke+8885V133mzBm7MpTHZDKR4qY+7vyWLCFkwgSSN27E3Lx5hcu6M2dlSE7HkpyOV12y1qac9evXL3O6XUdQAI0bNy5RkE6dOsWHH37I448/ftX3Go1GUlNTba9TU1NLXdPyK9ZTd7t27YiNjSUzM5OgoCB7I1Y7fqtXU9SixVWLkxBC1EZ2FyiAzMxMfvrpJ7Zu3cqJEyfs7kkiKiqKs2fPkpycTFhYGPHx8Tz33HMllklPTyc4OBiNRkNSUhKKohAYGFiZeNWK7o8/MPzyCxkTJrg7ihBCeKSrFiiz2cyePXvYsmULiYmJGI1GLl68yNSpU7nxxhvt2ohOp2PIkCFMmTIFRVHo3r07kZGRbNiwAYBevXqxc+dONmzYgE6nw2Aw8J///KfEIIk1je8XX6BqNOTdf7+7owghhEeqsEDFxsYSHx+PTqejY8eOvPbaazRt2pThw4eXaPRgj3bt2tGuXbsS03r16mV7/o9//IN//OMflVpntaWq+K1eTWHHjijlnHsVQojarsICtWHDBgICAujfvz933HFHietE4tp57duH/o8/yH76aXdHEUIIj1VhgZo3bx5bt27lq6++YunSpdxyyy107twZOxv+iXL4rlmDajCQ17u3u6MIIYTHqvBG3bp16/LQQw8xb948JkyYQEBAAAsXLiQzM5NPPvmE06dPuypnzWE24/vll+THxKAGB7s7jRBCeCy7epIAazdEI0aMYNGiRTz77LOkpqYyZswYZ2arkbx/+gldSgp5Dzzg7ihCCOHRKtXMHMBgMNC5c2c6d+5MWlqaMzLVaL6rV6MEB5MvQ2oIIUSF7D6CKktYWJijctQKmtxcfL77jrw+fcDb291xhBDCo1WpQInK8dmwAW1uLnn9+rk7ihBCeDwpUC7ku3q1tefyDh3cHUUIITyeFCgX0aam4r1li/XoSSu7XQghrsauRhLZ2dl89dVXnDx5kvz8/BLzJk2a5JRgNY3P11+jsVjk9J4QQtjJrgL19ttvYzabuf322zEYDM7OVCPZei5v0cLdUYQQolqwq0AdPXqUxYsX4+Xl5ew8NZLuxAkMe/eS+fLL7o4ihBDVhl0XQxo1alRiPCdROb5r16JqNORKz+VCCGE3u46gWrVqxdSpU+nWrRshISEl5vWQG04rpqr4rVlj7bm8QQN3pxFCiGrDrgJ1+PBhjEYj+/fvLzVPClTFvH79Ff3vv5P91FPujiKEENWKXQXq1VdfdXaOGst39Wprz+X33uvuKEIIUa3Y3RdfdnY2e/bsIS0tjbCwMG699VYCAgKcma36M5vx/eor6blcCCGugV2NJI4ePcqzzz7LDz/8wMmTJ4mLi+PZZ5/l6NGjzs5XrXlv347uwgW590kIIa6BXUdQS5cuZdiwYdxxxx22afHx8SxZsoRp06Y5LVx1Jz2XCyHEtbPrCOrs2bPcfvvtJaZ17NiRc+fOOSVUTaDJy7P2XH7vveDj4+44QghR7dhVoCIiIoiPjy8xbceOHYSHh9u9ocTEREaNGsWzzz7LF198Ue5ySUlJDBgwgJ07d9q9bk/kvWED2pwcOb0nhBDXyK5TfIMHD2b69OmsX78ek8nEhQsXOHv2LOPGjbNrI4qiEBsby4QJEzAajYwfP57o6GgaNmxYarmPPvqItm3bVvoX8TR+q1djqVePwo4d3R1FCCGqJbsKVLNmzZg3bx579+7l4sWL3HrrrbRr187uVnxJSUlERETYjrg6depEQkJCqQK1fv16OnTowPHjxyv5a3iWyz2XZw8fLj2XCyHENbK7mXlAQABdunS5po2kpaVhNBptr41GI8eOHSu1zK5du3j11Vd57733yl1XXFwccXFxAEyfPh2TyXRNmS7T6/VVXseVtKtWoTGb8Rk6FG8HrdsZOZ1BcjqW5HS86pJVclZQoKZMmcLLlzo3nThxIhqNpszl7BluQ1XVUtOuXN/SpUt57LHH0F7liCMmJoaYmBjb65SUlKtuvyImk6nK6yi1zhUrKGrRggsREeCgdTsjpzNITseSnI5XXbLWppz169cvc3q5Bapr166251XtzshoNJbobDY1NZXQ0NASyxw/fpy3334bgMzMTH755Re0Wi233XZblbbtarqTJzHs2UPmSy+5O4oQQlRr5Raozp072543aNCAJk2alFomKSnJro1ERUVx9uxZkpOTCQsLIz4+nueee67EMvPnzy/x/NZbb612xQnAd80aAHL79nVvECGEqObsuoI/efLkMqdPmTLFro3odDqGDBnClClTGD16NLfffjuRkZFs2LCBDRs22J/W06kqvmvXUnD77dJzuRBCVFGFjSQURQGs15Au/1x2/vx5dDqd3Rtq164d7dq1KzGtV69eZS779NNP271eT+K1fz9ex4+T/uST7o4ihBDVXoUF6pFHHrE9HzhwYIl5Wq2WfnITagnSc7kQQjhOhQXq3XffRVVVXnvttRKt9TQaDUFBQRgMBqcHrDYsFmvP5T17ol4xqKMQQojKq7BA1alTB4AFCxa4JEx15r19O7rkZPIeeMDdUYQQokaw+0bd3bt3c+jQITIzM0tMf+aZZxweqjryXb0aJShIei4XQggHsasV36pVq1i0aBGKorBz504CAgLYt28ffn5+zs5XLWjy8vBZv156LhdCCAey6whq8+bNTJgwgUaNGvHjjz8yePBgOnfuzOrVq52dr1qw9Vwup/eEEMJh7DqCysnJoVGjRoC13yWz2Uzjxo05dOiQU8NVF35r1kjP5UII4WB2HUFFRERw6tQpIiMjbTfYBgQE2N2beU2mTUvD+8cfpedyIYRwMLsK1IABA8jKygLg0Ucf5Z133iE/P59hw4Y5NVx14PPVV2jMZhmYUAghHMyuAlW8B4gmTZowb948pwWqbny/+IKi5s0xt2zp7ihCCFGjlFugzp8/b9cKKjPse02jycnBsHcv2dW0ayYhhPBk5RaoK3sbL8/KlSsdFqa68dq7F43FQmE17HVdCCE8XbkFqnjh2bx5M/v376d///7UqVOHCxcu8Pnnn9O6dWuXhPRUht27UTUaCq/oBFcIIUTV2dXsbOXKlYwYMYJ69eqh1+upV68ew4cP59NPP3V2Po9m2L0bc7NmqMHB7o4ihBA1jl0FSlVVkpOTS0y7cOGCbTiOWsliwbBnD4XR0e5OIoQQNZJdrfjuvfdeXn/9dbp162Ybf37Lli3cW4uHldAfOYI2K4vC9u3dHUUIIWokuwrUfffdR6NGjdixYwcnTpwgJCSEp556irZt2zo5nucyJCQASIESQggnsbs387Zt29bqgnQlw+7dWOrWxXKpCyghhBCOVW6BWrNmDQ9c6vy0oqbkAwYMcHyqasCwe7f1+pNG4+4oQghRI5VboFJTU8t8LkB7/jz6P/8kZ/Bgd0cRQogaq9wC9cQTT9iejxw5ssobSkxMZMmSJSiKQs+ePenbt2+J+QkJCaxcuRKNRoNOp2Pw4ME0b968ytt1Brn+JIQQzueSro4URSE2NpYJEyZgNBoZP3480dHRNGzY0LZM69atiY6ORqPRcPLkSebOnctbb71lVwZXMyQkoPr4UNSqlbujCCFEjeWSro6SkpKIiIiwFbNOnTqRkJBQokD5FBuJtqCgAI0HX9sx7N5NYdu2YDC4O4oQQtRYdnV1VFVpaWkYjUbba6PRyLFjx0ott2vXLj7++GMyMjIYP368w7bvSJq8PLwOHCB7xAh3RxFCiBrN7mbmVaGqaqlpZR0h3Xbbbdx2220cOnSIlStX8sorr5RaJi4ujri4OACmT5+OyWSqUja9Xl+pdWi2bkVjNuPTowfeVdx2ZVQ2p7tITseSnI5XXbJKTjsLlMVi4fvvv+fQoUO2gQsvmzRp0lXfbzQaS7UKDA0NLXf5li1bMn/+fDIzMwkKCioxLyYmhpiYGNvrlJQUe36Fcl3uGcNeAXFxeAEpTZuiVnHblVHZnO4iOR1Lcjpedclam3LWr1+/zOl29cW3bNky4uLiaNmyJb///jsdOnQgIyODm266ya6NR0VFcfbsWZKTkzGbzcTHxxN9RR92586dsx1p/f7775jNZgIDA+1avysZEhIoatIEtYICK4QQoursOoL6+eefmTJlCiaTic8++4zevXvTpk0bFi1aZNdGdDodQ4YMYcqUKSiKQvfu3YmMjGTDhg0A9OrVi507d7J161Z0Oh0Gg4HRo0d7XkMJRcGwZw95tbgPQiGEcBW7ClRhYaGtkYPBYKCgoIAGDRpw4sQJuzfUrl27EkPHg7UwXda3b99S90Z5Gn1SEtqMDOnBXAghXMCuAtWgQQOOHz9O48aNufHGG1m1ahW+vr6EhYU5O59Hsd2gKwVKCCGcrsJrUJfHexo8eDBarXXR//u//+OPP/5gz549DB8+3PkJPYghIQGL0YjlxhvdHUUIIWq8Co+gRowYQZcuXejSpQuNLvXaXa9evTKbf9cGhoQE6SBWCCFcpMIjqCeeeILk5GTGjx/P2LFj+fbbb8nMzHRVNo+ivXAB/YkT0v+eEEK4SIVHUO3bt6d9+/bk5OQQHx/P1q1b+eijj7j55pvp2rUr0dHR6PUuudfX7Qx79gBQJNefhBDCJeyqLv7+/tx1113cddddJCcns3XrVpYtW8YHH3xAbGysszN6BENCAqrBQGHr1u6OIoQQtYJdN+peVlRURFJSEseOHSMjI8N2Xao2MCQkUHTzzVCsU1shhBDOY9cR1OHDh9myZQs7duwgODiYO++8k2HDhlGnTh1n5/MM+fl47d9PztCh7k4ihBC1RoUF6rPPPmPbtm1kZ2fTsWNHxo0b57GDCDqT4ddf0RQWSgMJIYRwoQoL1LFjxxg4cCDt27fHUIvHPjLs3g1A4a23ujmJEELUHhUWqJdfftlVOTyaV0IC5htuQKkGXd8LIURNUalGErWSqlpH0JXTe0II4VJSoK5Cd/w4urQ0KVBCCOFiUqCuwnb9SQqUEEK4lBSoqzDs3o0SEoI5KsrdUYQQolaRAnUVhoQEa+s9rewqIYRwJfmrWwFNWhpeSUlyek8IIdxAClQF5PqTEEK4jxSoChh270b18qKwTRt3RxFCiFpHClQFDLt3U9SqFfj6ujuKEELUOlKgylNYiGHfPusIukIIIVzOZaMNJiYmsmTJEhRFoWfPnvTt27fE/G3btvHll18C4OPjw7Bhw7j++utdFa8Ur/370eTny/UnIYRwE5ccQSmKQmxsLC+99BJz585l+/btnD59usQydevW5bXXXmPWrFk8+OCDLFq0yBXRymVISACkgYQQQriLSwpUUlISERERhIeHo9fr6dSpEwmXCsBlzZo1IyAgAIAmTZqQmprqimjlMuzejfm661Dq1nVrDiGEqK1ccoovLS0No9Foe200Gjl27Fi5y2/atIlbbrmlzHlxcXHExcUBMH36dExV7GFcr9eXXoeq4rV3L0pMTJXX7yhl5vRAktOxJKfjVZesktNFBUpV1VLTNBpNmcseOHCAzZs38/rrr5c5PyYmhpiYGNvrlJSUKmUzmUyl1qE7cYLw8+fJatWK3Cqu31HKyumJJKdjSU7Hqy5Za1PO+vXrlzndJaf4jEZjiVN2qamphIaGllru5MmTvP/++4wZM4bAwEBXRCuTXH8SNcW+fV7Mnx9AGd8RhfB4LilQUVFRnD17luTkZMxmM/Hx8URf0Xw7JSWFWbNm8cwzz5RbTV3FkJCAEhSEuVkzt+YQoirOntUyaFAYU6cGsXKl3Msnqh+XnOLT6XQMGTKEKVOmoCgK3bt3JzIykg0bNgDQq1cvPv/8c7Kzs1m8eLHtPdOnT3dFvFIMu3dLB7GiWisshCefDCM3V0Pr1oVMmhRMt24FREQo7o4mhN1cdh9Uu3btaNeuXYlpvXr1sj0fMWIEI0aMcFWccmkyMvA6coS8++5zdxQhrtnkyUHs2WNgwYI0WrUqolevurz8cjCLF1+knMu/QngcOUS4gmHPHgDpQUJUW19+6UNsbABDh2Zz//35REVZeOGFLL77zpdvvvFxdzwh7CYF6gqGhARUnY6iK472hKgOjh7V8+KLIbRvX8Arr2Tapg8fns3NNxcyYUIwaWlyCCWqBylQVzAkJFB0002ofn7ujiJEpWRna3jiiVD8/FTee+8iXl5/z9PrYdasdNLTtUyaFOy+kEJUghSo4oqK8EpMlOblotpRVXjxxRB+/13PggUXqVevdGOIm24y8/TT2Xz+uR+bN3u7IaUQlSMFqhivQ4fQ5uXJ9SdR7Sxe7M/XX/syfnwWd9xRWO5yo0Zl0aRJEWPHBpOdLaf6hGeTAlWM7QZdKVCiGtm1y8DkyUH84x95PPVUdoXLentbT/WdOaNj2rQgFyUU4tpIgSrGkJCAuUEDFDffKCyEvS5c0DJiRCgNG1qYOzfdribk0dFFDBmSw9Kl/vz8s8H5IYW4RlKgLlNV6w26cv1JVBNmMzz1VCgZGRo++CCNoCD7+zMaOzaLyEgzL74YQn6+E0MKUQVSoC7R/fUXunPnpECJauPNNwPZscObN9/MoGVLc6Xe6++vMmNGOr//rmfuXPf1eylERaRAXSLXn0R1sn69DwsWBPKvf+Xw0EN517SOLl0KGTgwh/feC2D/fq+rv0EIF5MCdYkhIQHF3x9z8+bujiJEhX7/Xcfo0SG0aVPIpEkZVVrXK69kYjQqvPBCCEVFDgoohINIgbrEkJBg7T1C77LuCYWotNxcDcOHh6HTwaJFF/Gu4u1MISEqU6dmcPCgFwsXBjgmpBAOIgUK0GRloT98WK4/CY+mqjB2bDCHD+uZP/8iDRtaHLLee+7Jp0+fPObODSQpSb6gCc8hBQow/PILGkWRAiU82ooVfqxZ48cLL2TRrVuBQ9c9eXIGvr4qL74YjCIjcggPIQWKSx3EarUU3nKLu6MIUaZffvHi1VeD6dEjn1GjKr4Z91rUqaPw2msZJCR4s2yZ9EMpPIMUKC7doNu8Oaobh5kXojxpaVqGDw+lbl0Lb7990WnjaD70UB7du+czdWoQp0/rnLMRISqhVheo48d1pCWb8dq7V07vCY9kscAzz4SQkqLjgw8uEhZm/824laXRwPTpGWg01mtdqvM2JYRdanWBeuWVYCKv9+EfOWtYVDCYCxdq9e4QHmjKFB1btvgweXIGN9/s/HbgDRtaeOmlTH780YdVq3ydvj0hKlKrm+yMHZvFZv1J1m68gec/bYJmpcpttxXSu3c+99yTR4MGcrVYuI7FAn/8oWf/fi/bT3y8jocfzuXRR3NdlmPQoFy++MKXSZOC6datgLp15f+BcA+NqlbvA/kzZ85U6f3hzz+P+uMWflyRyLfr/Vi/3ofDh6131d9ySyH33JNP79553HCDY5r0XiuTyURKSopbM9hDctqnqMg6+u2BA14cOGAtRgcPepGbaz2K9/ZWad68iK5ddTz33Hl8XXwwk5Sko1evusTE5LNo0cWrLu/u/VkZ1SXr1XKqqvVzlJ+vIS+v5I+904qKNJjNYDZffiz+vPxHi8W6bYtFw3PPqQwenFyl37V+OR10u+wIKjExkSVLlqAoCj179qRv374l5v/1118sWLCAP/74g4EDB3Lfffe5JJd2xw7y20fT8iYLLW/K4sUXszh+XMe33/ry7bc+TJ0axNSpQbRoUcS99+Zxzz35NGtmtqvXaCEA8vPh8OG/j4oOHPDi8GEvCgqsHyI/P4Wbbipi4MBcWrUqonXrIpo0MePldfmPlOszN25s4fnns5g2LYhvv82jd+/q26OsokB6upYLF6w/Wq2G8+d9yc3VlPOjtT3Py7M+5uT8PT8/X4tGo6LRWK/babV/P4JaYlrx5wBa7d/vU9XiP9YFLh8uWB81KErEFctZp8PfBaKydDoVX18VHx8Vb28VvR70+suPxZ+reHmBr69SanrxZZs2dV6P+C45glIUhVGjRjFhwgSMRiPjx49n1KhRNGzY0LZMRkYGFy5cICEhAX9/f7sLVFWOoLRnzhDRvj0ZkyaRM2xYmcucPq3j2299+PZbH3bvNqCqGm680Uzv3nnce28+rVsXuaRY1ZRvfZ7CnpwWS8lvp5X9ppqZqeHwYS+OHtXb/pAEByu2ItS6dRGtWhVyww0WdOU0mnPn/iwqgj59TCQn69i8OZmQkPL/VFyZU1Wt38aLijQUFVkfNRrw8rL+0fPyUtHpuOb/O6oKGRkaUlJ0JCdfLj66Kx6tz1NStJjNFW9Ir1fx8yv+o5R47eur4u9vfe7jY90PinJlkbEWmsvTLz/+vaymxPTLhervn7+LF4Cvry/5+XllLGddVqcDX1/V9uPjo5Z4Xd40Lwd3u+iIz6hbj6CSkpKIiIggPDwcgE6dOpGQkFCiQAUHBxMcHMzevXtdEQko1kFsBS34Gja0MHx4DsOH53D+vJbvvvPh2299ee+9AN59N5CGDc00aVK5nqSvhZeXnqKiMKdvp6r0emvO4l97/v5WWPJbYsl5paeVt0zpx9LfPlXVWmAsFusfDOvzv1+rqo6ionDbdEXRlHpeVHRtfz19fRV8fa1/zJo2NXPXXfm2gtSwoaXaHH17ecHs2en07l2Hvn1NmEyKreAUFlofzWYNhYVgsegoKIigsPDytKv/khqNisHw9zd1g0EtUcCKT9PrrdMyM7UkJ2tJSdGVuQ29XsVkUqhb10KdOtYjU+trhTp1LJhMCpGRwRQUpJUoQAYPHBbLZDKQkpLp7hhu5ZIClZaWhtFotL02Go0cO3bsmtYVFxdHXFwcANOnT8dkMl1zLt3Bg6h+fgR36YI9XytMJrjpJnjhBUhJKWLdOi1ffaXl/Hnnf7o1Gg2q6oH/i66g0WgAw6XnxaeX/Xj5efFvjhUtf+Uf9/Lma7Wg0135Y/3Wefn0hFZbfDm11HLWb5zg6wt+ftYfa+H5e9rl15en+fhcmVELeF/6qTy9Xl+lz3hVdesG8+ZZWL5ch5eXDl9fMBgu/1wuItaReosXmsvz9Xrr/Mv/vQoLuXREZS1yhYV/T7M+aoo9lpxfUAB16kCrVirh4Qp166pEREB4uEp4uPUxNJRi94lpKauhsl6vw2wOddEevHbu/re3lzNzuqRAlXUWUXONXyNjYmKIiYmxva7KoWXYb79huO02UjKurUfoe++1/rhCTTp15gkcnVNRICfH+uNInrA/77/f+lMRT8ipqpCWdvXlPCGrPWpTTree4jMajaSmptpep6amEhrq/m8waStWYPLxQYYUFUIIz+OSO1OjoqI4e/YsycnJmM1m4uPjifaUgQEDZIgBIYTwRC45gtLpdAwZMoQpU6agKArdu3cnMjKSDRs2ANCrVy/S09MZN24ceXl5aDQavv32W+bMmYOfn3RcKYQQtZHL7oNq164d7dq1KzGtV69etuchISEsXLjQVXGEEEJ4OOl8TgghhEeSAiWEEMIjSYESQgjhkaRACSGE8EhSoIQQQngkKVBCCCE8UrUfD0oIIUTNVOuPoMaNG+fuCHaRnI4lOR2ruuSE6pNVckqBEkII4aGkQAkhhPBItb5AFR+6w5NJTseSnI5VXXJC9ckqOaWRhBBCCA9V64+ghBBCeCYpUEIIITySy4bbcLfExESWLFmCoij07NmTvn37lpivqipLlizhl19+wdvbm5EjR3LjjTe6NGNKSgrz588nPT0djUZDTEwMvXv3LrHMwYMHmTFjBnXr1gWgQ4cOPPTQQy7NCfD000/j4+ODVqtFp9Mxffr0EvM9YX+eOXOGuXPn2l4nJyfz8MMPc++999qmuWt/LliwgL179xIcHMzs2bMByM7OZu7cuVy4cIE6deowevRoAsoYUPNqn2Vn51yxYgV79uxBr9cTHh7OyJEj8ff3L/Xeq31GXJH1s88+Y+PGjQQFBQHwyCOPlBr2B9y/T+fOncuZM2cAyM3Nxc/Pj5kzZ5Z6r6v2aXl/i1z+GVVrAYvFoj7zzDPquXPn1KKiIvXFF19UT506VWKZPXv2qFOmTFEVRVGPHDmijh8/3uU509LS1OPHj6uqqqq5ubnqc889VyrngQMH1GnTprk825VGjhypZmRklDvfE/ZncRaLRR02bJianJxcYrq79ufBgwfV48ePq88//7xt2ooVK9S1a9eqqqqqa9euVVesWFHqffZ8lp2dMzExUTWbzbbMZeVU1at/RhytrKwrV65Uv/zyywrf5wn7tLhly5apq1atKnOeq/ZpeX+LXP0ZrRWn+JKSkoiIiCA8PBy9Xk+nTp1ISEgosczu3bvp0qULGo2Gpk2bkpOTw8WLF12aMzQ01HaU4evrS4MGDUhLS3NpBkfxhP1Z3P79+4mIiKBOnTpuy1Bcy5YtS33zTEhIoGvXrgB07dq11GcU7PssOztnmzZt0Ol0ADRt2tRjPqNlZbWHJ+zTy1RVZceOHdxxxx1O2749yvtb5OrPaK04xZeWlobRaLS9NhqNHDt2rNQyJpOpxDJpaWmEhoa6LGdxycnJ/PHHHzRu3LjUvKNHjzJmzBhCQ0P517/+RWRkpBsSwpQpUwC46667SjU19bT9uX379nL/03vK/szIyLDtn9DQUDIzM0stY89n2ZU2bdpEp06dyp1f0WfEVb7//nu2bt3KjTfeyKBBg0oVB0/ap7/99hvBwcHUq1ev3GVcvU+L/y1y9We0VhQotYyW9BqNptLLuEp+fj6zZ89m8ODB+Pn5lZh3ww03sGDBAnx8fNi7dy8zZ87knXfecXnGN954g7CwMDIyMpg8eTL169enZcuWtvmetD/NZjN79uzh0UcfLTXPU/anvTxpv65ZswadTsedd95Z5vyrfUZcoVevXrZriitXrmT58uWMHDmyxDKetE8r+iIFrt+nFf0tKo8j92etOMVnNBpJTU21vU5NTS31Td5oNJKSklLhMq5gNpuZPXs2d955Jx06dCg138/PDx8fHwDatWuHxWIp81uMs4WFhQEQHBxM+/btSUpKKjHfU/YnwC+//MINN9xASEhIqXmesj/Bui8vnwa9ePGi7cJ+cfZ8ll3hxx9/ZM+ePTz33HPl/vG52mfEFUJCQtBqtWi1Wnr27Mnx48dLLeMp+9RisbBr164Kj0hduU/L+lvk6s9orShQUVFRnD17luTkZMxmM/Hx8URHR5dYJjo6mq1bt6KqKkePHsXPz8/lH1JVVVm4cCENGjSgT58+ZS6Tnp5u+4aSlJSEoigEBga6Mib5+fnk5eXZnv/66680atSoxDKesD8vq+hbqSfsz8uio6PZsmULAFu2bKF9+/allrHns+xsiYmJfPnll4wdOxZvb+8yl7HnM+IKxa977tq1q8zTt56wT8F6nbR+/folTo8V58p9Wt7fIld/RmtNTxJ79+5l2bJlKIpC9+7deeCBB9iwYQNgPQ2gqiqxsbHs27cPg8HAyJEjiYqKcmnGw4cPM3HiRBo1amT7VvrII4/YjkR69erFd999x4YNG9DpdBgMBgYNGkSzZs1cmvP8+fPMmjULsH7r69y5s0fuT4CCggKeeuop3n33XdspiuI53bU/33rrLQ4dOkRWVhbBwcE8/PDDtG/fnrlz55KSkoLJZOL5558nICCAtLQ03n//fcaPHw+U/Vl2Zc61a9diNptt13KaNGnC8OHDS+Qs7zPiTGVlPXjwICdOnECj0VCnTh2GDx9OaGiox+3THj16MH/+fJo0aUKvXr1sy7prn5b3t6hJkyYu/YzWmgIlhBCieqkVp/iEEEJUP1KghBBCeCQpUEIIITySFCghhBAeSQqUEEIIjyQFSohq7OGHH+bcuXPujiGEU9SKro6EcJWnn36a9PR0tNq/v/t169aNoUOHujGVENWTFCghHGzs2LHcfPPN7o4hRLUnBUoIF/jxxx/ZuHEjN9xwA1u2bCE0NJShQ4fSunVrwNpjwAcffMDhw4cJCAjg/vvvt/VUrSgKX3zxBZs3byYjI4N69eoxZswYW2/xv/76K1OnTiUrK4s77riDoUOHotFoOHfuHO+99x4nTpxAr9fTqlUrRo8e7bZ9IERlSYESwkWOHTtGhw4diI2NZdeuXcyaNYv58+cTEBDA22+/TWRkJO+//z5nzpzhjTfeIDw8nNatW/PNN9+wfft2xo8fT7169Th58mSJPvD27t3LtGnTyMvLY+zYsURHR9O2bVs+/fRT2rRpw6uvvorZbOb33393428vROVJgRLCwWbOnGkb0A/g8ccfR6/XExwczL333otGo6FTp058/fXX7N27l5YtW3L48GHGjRuHwWDg+uuvp2fPnmzdupXWrVuzceNGHn/8cerXrw/A9ddfX2J7ffv2xd/fH39/f2666SZOnDhB27Zt0ev1XLhwgYsXL2I0GmnevLkrd4MQVSYFSggHGzNmTKlrUD/++CNhYWElhqaoU6cOaWlpXLx4kYCAAHx9fW3zTCaTbWiI1NRUwsPDy91e8WFEvL29yc/PB6yF8dNPP+Wll17C39+fPn360KNHD0f8ikK4hBQoIVwkLS0NVVVtRSolJYXo6GhCQ0PJzs4mLy/PVqRSUlJsY/8YjUbOnz9f6aEVQkJCGDFiBGDtnfqNN96gZcuWREREOPC3EsJ55D4oIVwkIyOD9evXYzab2bFjB3/99Re33HILJpOJZs2a8fHHH1NYWMjJkyfZvHmzbaTanj17snLlSs6ePYuqqpw8eZKsrKyrbm/Hjh22geP8/f0BSjR/F8LTyRGUEA725ptvligEN998M+3bt6dJkyacPXuWoUOHEhISwvPPP28bHHHUqFF88MEHPPnkkwQEBNC/f3/bacI+ffpQVFTE5MmTycrKokGDBrz44otXzXH8+HGWLl1Kbm4uISEh/Pvf/6Zu3brO+aWFcAIZD0oIF7jczPyNN95wdxQhqg053hdCCOGRpEAJIYTwSHKKTwghhEeSIyghhBAeSQqUEEIIjyQFSgghhEeSAiWEEMIjSYESQgjhkf4ffbjsbZoPyDcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print_result(nn_cross,X_train,y_train,X_test,y_test,title=\"Cross Entropy Loss\",color=\"red\")\n",
    "print_result(nn_quad,X_train,y_train,X_test,y_test,title=\"MSE Loss\",color=\"blue\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Back to Slides]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TLPBetterInitial(TLPMiniBatchCrossEntropy):             \n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights Glorot and He normalization.\"\"\"\n",
    "        init_bound = 4*np.sqrt(6. / (self.n_hidden + self.n_features_))\n",
    "        W1 = np.random.uniform(-init_bound, init_bound,(self.n_hidden, self.n_features_))\n",
    "\n",
    "        # reduce the final layer magnitude in order to balance the size of the gradients\n",
    "        # between \n",
    "        init_bound = 4*np.sqrt(6 / (self.n_output_ + self.n_hidden))\n",
    "        W2 = np.random.uniform(-init_bound, init_bound,(self.n_output_, self.n_hidden)) \n",
    "        \n",
    "        b1 = np.zeros((self.n_hidden, 1))\n",
    "        b2 = np.zeros((self.n_output_, 1))\n",
    "        \n",
    "        return W1, W2, b1, b2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 1/200"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 50.8 s, sys: 4.82 s, total: 55.6 s\n",
      "Wall time: 14.2 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 20/20"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 51.7 s, sys: 5.01 s, total: 56.8 s\n",
      "Wall time: 14.4 s\n",
      "=================\n",
      "Traditional :\n",
      "Resubstitution acc: 0.8404666666666667\n",
      "Validation acc: 0.8269\n",
      "=================\n",
      "Glorot Initial :\n",
      "Resubstitution acc: 0.8677333333333334\n",
      "Validation acc: 0.848\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABAqElEQVR4nO3deXhU5dn48e8s2ffMZCEQBAm7CMYgiAhCQmrVInW3bohUEZcKlQpI1RYRqnBRtVK3FJfaivbVV/v6UzCKoAQKAkFlKQkCsoQkM5N9nznn98ckQ4YsDMlsydyf65orZ3nmnHsOw7nnPOc5z6NRVVVFCCGE8DNaXwcghBBCtEcSlBBCCL8kCUoIIYRfkgQlhBDCL0mCEkII4ZckQQkhhPBLel8H0F0nT57s1vuNRiMmk8lN0XiOxOleEqd79ZQ4oefEGkhxpqSktLtcrqCEEEL4JUlQQggh/JIkKCGEEH5JEpQQQgi/JAlKCCGEX5IEJYQQwi9JghJCCOGXJEEJIYTwSz3+QV0hApmiQH29hrq6jl8NDRqsVrDZwGbTNP9tO2212rfX3jSAVmt/6XSqY7plPjJSS319pGNeowGdDqd5bfPPYY3mzJd9fcu69svQvA0VnY7m17lPa7VgNkNxsZ6mJg2NjRoaGqCxUUNTU/vT9nloaLBPtz4eLZ9Lq1VbxXj25QBNTWC1atr8tVqhqUmDTqejujrOMX/mX1WFoCD75wsKUtHrQa9v+Xt6uuMyp4+7qoKq2rd5ev70q2XZmX9VVcNVV2lIT/fM91sSlBCtNDVBVZWG2lotTU32k3TLf8SWaUWxvwAUpb3lGse0qye91tONjafLgJ6KCkO7iae+3v7yBL1edTrBazQtCct+LGy20wnstGiPxOIZief8Dp1OJThYJSjIPt/yb9zR90NVXfu3sScM+3Zb/w0J0aLV6tss1+shMlJBozmd0GpqtI7EZbO1/mv/N2qd/Fr+Kkrn8bX8cGh52Ze1nTcabZKghGhNVe3/+VqfqFtedXUaams1VFVpqKnRNv/VUF2tpbpaQ3W1hoYGPWVlBqqqtM3r7OsbGjxzwj+b4GC11ev0fHg4BAVpCAtTiYtTCAtTHa/wcOf5sDCV0FC1zbKQEPvJVa+3/6LX61uubk5Pt77S0J5jxb+qQlyckZISUzsJ7PT8mb/K7b/ENe3+Uu+orKLQzpWeffuuTNtsEB8fRUNDZZtjHRJiTwLt/TsEB9uPzblqSVjOicv+WVqSjaaDr5ynuzqy2U5Pn5l4zoU9TvfF1ZokKOFTjY1w7JiOI0f0HD6s58gRHaWlOkeiOZ142iajs/0CPFNYmEJUlEpEhEpsLISGQt++NqKiFCIiVMffyEiVyEiFoKDWVTNqq2ob15efeZJrLxG17Kc9PaE/No3GXqUUHNyyRD2jxJnzvmU0RmAy1XtlXy1VnW35/ph0JeF6myQo4XGNjfDTT62TkJ7Dh+3zx4/rnKqJIiMVkpJszVcDEBamEh+vEBqqOr1arhY6mm+daCIj7fP6Vt92+4nf7IOjIYRwlSSoAGSzQUWFFotFQ1mZjrIyDWVlWiortc2/+NTmm9unq4Lam26vXFCQhu++i3BcDR0+bE9Cra92oqIUBg60MmZMIzNm2BgwwMrAgVYGDrRhMChdqmYQQvQ+kqB6AUWBH3+0V41ZLFrKyuyvjqYrKjQu38Dtmhiio+1JKD29keuuc05C8fGShHq99pp8Nd+U0rTcSOrCNNB5U79Wy9UOmgZqoqIIKi0FqxWNvWUBmuYWBRp7KwLnv+2sb7mZpGm5sdTqRpNjWfONtw7LdHTzrXlaFxxMbEODfRvtNasD0GpRW34ltqprVs+sc25Z1npep7PH0XLDUFHsn7N5vt1pVbX/m7S6yai94w74+c898jWSBNVDnTih5euvQ9i8OYRvvgnBbG5boRwWphAfrxAXpxAXp5Ka2khcnNo833qdfToqSmn+/3P6hnLr6TPn25tOSoohNtZMXJwkoQ6pKpqaGjQVFWirqtBWVjqmNZWVaJtfmspKdE1NxNXUnD4pNN/11zSfJDWn24+fnrZaHWVbynHGCVHT0gyx9V38VtOa9ta1LGv9F+wnT6D9EX38U4Ibt+VIBi0n/tY3JVsni9ZJoyWBdtJMTqvTEdzy79Re+ZZ/y5bE0SoRahSl7bLW882JX225SdbcOkZtVSWiOlePoLa0oGmZbpmv99z9PElQPURFBaxfH8rmzfak9OOP9n+6xEQbV1zRwGWXNdC3r80p8YSGdnVvXb+BazSqmExKl9/f49TVobNY0JpM9pfZjNZsRtc8fWbCaUlCjgTRATUkBCU6Gm10NHqtFvR6+0nB3hQPtbn5nRoW5jhZqKeb5J1e31z+jIeJTp/wWj+cdEY5xy/ulvXtXZE0C4+IoLa21mlZu1cwrWPqbPp0PbLT9OmNt9vMz/FX097y5ldUXByVdXUQFGTfdlCQfd96vfPfdtY7lrWcnD34K6wnNJABe5yeasYnCcpPNTXB7t3BjoSUnx+EzRZPeLjC+PGN3HlnDZdf3sDQoVa5UnEXmw1NyxVNZaU98ZjNHSYfrcmEtqam3U2poaHY4uNRY2JQoqOxpaSgDBuGEh2NGh19+m9UlL1MVJTTspZfFz3lJBViNFLVA+IEiDQaaeghsQY6SVB+QlWhsFDvSEhbtwZTU6NFq1UZPbqJ3/1OISOjjPT0xlbNeYUTVbVfpZhMaMvLT1+1tFy5tKpG09fXYzCZ7FVsFRX2xFRd3fGm9XoUg8HxajzvPPu00YhiMGBr/tsyr0ZEePTXtRCBwGsJKj8/n7Vr16IoCpmZmcyYMcNpfW1tLS+88AJmsxmbzcYvfvELpkyZ4q3wfKKuTsPmzSGsXx/Kpk0hnDplr8IYMMDK9dfXMWlSAxMmNBATozb/km70ccQ+0Nh4+qql1ZWMzmRCW1rquJJxVKk1dnyMVJ3OcZVCfDyEhWEdOBC19dVLy9VNVJQ98TQnHTUmRhKOEF7mlQSlKAo5OTksWbIEg8HAokWLyMjIoF+/fo4yn332Gf369WPhwoVUVlbym9/8hssvvxy9vndd5JWVacjNDeWzz0L56qsQ6uu1xMQoTJrUwKRJDVx+eQOpqbazb6gX0FRWojt+HN2JE+hOnEDfPK0tLkbXknwqKtp9rxoSYr9qMRpREhOxjhjhfBUTF+dUfabGxKCGhTmSjNFoxCzVPEL4Na+c/QsLC0lOTiYpKQmACRMmsGPHDqcEpdFoqK+vR1VV6uvriYyMRHuufa74qRMndKxfb09K27YFY7Np6NPHxq231vKzn9Uzfnyjo3+vXkNR0JaUOBKQ/sQJp2SkO34cbVWV01vU4GBsKSnY+vShaeTI0wknIQHFaHRczSgJCVKFJkQA8EqCslgsGAwGx7zBYKCgoMCpzJVXXsmzzz7LfffdR11dHfPmzWs3QeXm5pKbmwvAihUr7C1IukGv13d7G2dSVdi/X8NHH2n4+GMtu3bZP8fw4QqPPqpw7bUK6ekqGk0w4NoNJU/E2W2qCidPotm/3/HSHjpEn6NH4fhxNE1NzsXj4lBTU2HQINQrrsDavz/qeedBaipq//6QmAhaLRpA1/zyFL88nu2QON2vp8QqcXopQalq22bLmjN+/e7Zs4fzzjuPJ554guLiYpYuXcqwYcMIDw93KpeVlUVWVpZjvrstnNzVSkpRYOfOID77LIzPPgvlyBH7oU1Pb+Txx+v52c/qGDTodNWd+Rx72fFpay5FsV8FHTyIvqCAoOa/+oICp6sgJTYWddgw6kePxnb11faroX79sPXti61vX9SoqM73Y7F4+IOc1lNax0mc7tdTYg2kOFNS2n+KzisJymAwYG51RjabzcTFxTmV2bhxIzNmzECj0ZCcnExiYiInT54kLS3NGyF22e7dQfzzn+Fs2BBKaamOoCCVyy5r4L77qvnZz+pJSupBzwTZbOh++ul0EmqdiOrqThdLTMSalkbd9dfTNHgw1iFDsA4ZgmIwYExIoLwH/KcSQvg/rySoQYMGUVRURElJCfHx8eTl5fHwww87lTEajXz//fcMHz6c8vJyTp48SWLiuY/Z4m1z5sRhsWiZOrWBn/+8nqlT64mO9n1PxWelquiOHSNo926Cd+4kePdugvbtQ9PqqXBbnz40DRlC7W23OZJQU1oa6hk/LoQQwhO8kqB0Oh2zZs1i2bJlKIrClClTSE1NZcOGDQBkZ2dz/fXXs2bNGn77298CcNtttxEd7d8DoKkqFBfruO++ahYtqjr7G3xIU1lJUH4+wbt3E7xrF0G7d6NrvqpVQkNpGj2amjvvpGnYMKyDB2MdPPjsVXJCCOFBXmvDnZ6eTvoZwy5mZ2c7puPj41myZIm3wnGLigr7KKkGg59V41mt6A8csCej3bsJ2r0bfUGBo8+0prQ0GjIzabzoIhrT07EOHUrva0YohOjpetdDRl5mMtlb5xmNvk1QmooKQvLyCN65k6Dduwnas8dxz8gWH0/TRRdRN306TRdfTOPo0faHToUQws9JguqGlh7EvZ6gFIWgH34gZONGQjZuJHjXLjQ2G2pQEE0XXEDtr35F00UX0XjRRdjOO0+eFxJC9EiSoLqh5QrKYPBCzw8mE2H/+7+EfPklIZs2oWtuKdd44YVUP/AADVOm0Dh6NISEeD4WIYTwAklQ3eDRKj6bjaA9ewhtvkoKys8nTlWxxcXRMHkyDVOm0DB5MkqCO0e2EUII/yEJqhvMZnuCio93T4LSlpYSsmmTvepu0yZ0ZWWoGg1NY8ZgW7KEsksuoWn0aOdxcYQQopeSBNUNJpOO2FilWw3gNJWVROTkELphA8HffQeAzWikITOThilTqJ80CTU+HqPRSJM8ACuECCCSoLqhtFRLQkIX7z9ZrYS/8w5RK1eiLSujMSODyt/9joapU2kaOfL0SKdCCBGgJEF1g9ms7dL9p5CNG4n+4x8JOniQhvHjqXzqKZpGjfJAhEII0XNJguoGk0nL8OFWl8vr//tfopcuJXTjRqwDBmB5/XXqr7xSmoELIUQ7JEF1g9msw2g8+yi3WrOZqJUrCX/nHdSICCqeeIKau+9Gxm4XQoiOSYLqosZGKC/XYjR2cg+qoYGIv/2NqOefR1NbS82dd1I9fz5KfLz3AhVCiB5KElQXWSwtD+m2cw9KVQn95BOily1D/9NP1GdmUvn732MdPNjLUQohRM8lCaqLOnpINyg/n+g//IGQ7dtpGjYM8z//ScOkSb4IUQghejRJUF10Zj982pMniV6xgvD/+R9sRiPlf/oTtbfcAno5xEII0RVy9uyi0lL7FVRCZA1RK58n4q9/RaOqVD34INUPPihjKQkhRDdJguqiliq+wYtnE7Ujl7rp06lcvBhbaqqPIxNCiN5BElQXmc1agoNVjN99Q/Xdd1P59NO+DkkIIXoV6U+ni0wmHYY4K9qGemz9+/s6HCGE6HUkQXWRyaQlIbp51NrkZB9HI4QQvY/Xqvjy8/NZu3YtiqKQmZnJjBkznNZ//PHHfP311wAoisLx48fJyckhMjLSWyGeE7NZS0JYNQCKJCghhHA7ryQoRVHIyclhyZIlGAwGFi1aREZGBv369XOUmT59OtOnTwfg22+/5ZNPPvHb5ATN/fD1KQfAlpTk22CEEKIX8koVX2FhIcnJySQlJaHX65kwYQI7duzosPyWLVu47LLLvBFal6iq/TmoBE0pIAlKCCE8wStXUBaLBYPB4Jg3GAwUFBS0W7ahoYH8/Hzuueeedtfn5uaSm5sLwIoVKzAajd2KTa/Xn/M2Kiuhvl5DX63JPphgqytBT+lKnL4gcbqXxOl+PSVWidNLCUpV1TbLNB0MMbFz506GDh3aYfVeVlYWWVlZjnlTN0eZNRqN57yNw4d1QBKGmh+xJiV1OwZXdCVOX5A43UvidL+eEmsgxZmSktLucq9U8RkMBsxms2PebDYTFxfXbtktW7YwceJEb4TVZS0P6SbVHJbqPSGE8BCvJKhBgwZRVFRESUkJVquVvLw8MjIy2pSrra1l37597a7zJy398CVXFkoTcyGE8BCvVPHpdDpmzZrFsmXLUBSFKVOmkJqayoYNGwDIzs4GYPv27YwePZrQ0FBvhNVlLVdQyZYDKEnX+TgaIYTonbz2HFR6ejrp6elOy1oSU4srrriCK664wlshdVlLgkpUi6mTKj4hhPAI6UmiC8xmLdERTYTQiK1PH1+HI4QQvZIkqC4wmXQkRNYCoMgVlBBCeIRLCep3v/sdn3zyCeXl5R4Op2cwmbQYQysB6YdPCCE8xaV7UNdddx3ffPMN7777LsOHD2fSpElccsklBAcHezo+v2QyaRmmt6DqdCg94EE6IYToiVxKUOPHj2f8+PFUV1eTl5fH+vXref3117nkkkuYNGkSF1xwgafj9Csmk5bEuBKUhATQ6XwdjhBC9Ern1IovMjKSyZMnExoayscff8x//vMf9u/fj1ar5Z577uHCCy/0VJx+w2qFsjItiVEnpHpPCCE8yKUEpSgK3333HZs3b2bXrl0MGTKEGTNmOKr5tm3bxosvvshrr73m6Xh9rqxMi6pqSKo/Jr1ICCGEB7mUoO677z6io6OZNGkSt99+O/Hx8U7rx48fz/r16z0SoL9xdHNUdUjGgRJCCA9yKUEtXLiQQYMGdVrmySefdEtA/s7Ri0TtYWxJ430cjRBC9F4uNTM/fvw4R48edVp25MgRNm/e7JGg/FlLP3yJlMg9KCGE8CCXEtS6deucxnMCexfr7777rkeC8meObo4okSo+IYTwIJcSVF1dHeHh4U7LwsPDqamp8UhQ/qy0VIteayOOMmkkIYQQHuRSgurXrx/btm1zWrZ9+3b6eWEkWX9jNmsxhtegRZUqPiGE8CCXGkncdtttLF++nLy8PJKTkzl16hTff/89ixYt8nR8fsdk0pEQYkG1hqLGxPg6HCGE6LVcSlDDhg1j1apVfPPNN5hMJtLS0pg5c6bHxqH3ZyaTlgSt2X711MGw9UIIIbrP5Z4kjEYjM2bM8GAoPYPZrGWoekruPwkhhIe5nKC+/fZb9u3bR2VlpdPyBx980O1B+TOTSUuS/rjcfxJCCA9zqZHE+++/z6uvvoqiKGzbto3IyEj27NnTpmVfb1dbq6G2VktS7VEZB0oIITzMpSuojRs3smTJEvr3789XX33FzJkzmThxIv/zP//j8o7y8/NZu3YtiqKQmZnZbnXh3r17eeONN7DZbERFRfGHP/zB5e17g9nc3M2R9QS2pMBrwSiEEN7kUoKqqamhf//+9jfo9VitVtLS0ti3b59LO1EUhZycHJYsWYLBYGDRokVkZGQ4NVOvqanh9ddf5/HHH8doNFJRUdGFj+NZpaXNCYpibH3G+jgaIYTo3Vyq4ktOTubYsWMApKamsmHDBjZv3kxkZKRLOyksLCQ5OZmkpCT0ej0TJkxgx44dTmW++eYbxo0b52gZGOOHTbidepGQKj4hhPAol66gbr75ZqqqqgD7M1HPP/889fX1zJ4926WdWCwWp66SDAYDBQUFTmWKioqwWq089dRT1NXVcdVVVzF58mRXP4dXSD98QgjhPWdNUIqiEBwczJAhQwBIS0vjxRdfPKedqKraZpnmjGeIbDYbhw8f5ve//z2NjY0sWbKEwYMHk5KS4lQuNzeX3NxcAFasWNHtZ7H0er3L26irs19BJVCKfuRICAvr1r7PxbnE6UsSp3tJnO7XU2KVOF1IUFqtlmeffZa33nqryzsxGAyYzWbHvNlsJi4urk2ZqKgoQkNDCQ0NZfjw4Rw9erRNgsrKyiIrK8sxbzKZuhwX2J/vcnUbR49GE6nXERoZwqmaGvBiX4TnEqcvSZzuJXG6X0+JNZDiPPM838Kle1DDhw/n4MGDXd75oEGDKCoqoqSkBKvVSl5eHhkZGU5lMjIyOHDgADabjYaGBgoLC+nbt2+X9+kJZrOWhKByeUhXCCG8wKV7UAkJCSxfvpyMjAwMBoNT9dzNN9981vfrdDpmzZrFsmXLUBSFKVOmOBpbAGRnZ9OvXz/GjBnDo48+ilarZerUqY6Wg/7CZNKRqCmVBCWEEF7gUoJqbGxk7Fh7s2qLxdKlHaWnp5Oenu60LDs722l++vTpTJ8+vUvb9waTSUuarUjGgRJCCC9wKUHNnTvX03H0CCaTlssaf5IrKCGE8AKXElRxcXGH65IC5GStKPZ7UElqsTQxF0IIL3ApQT388MMdrlu3bp3bgvFn5eVaFEXTPNT7GF+HI4QQvZ5LCerMJFReXs7777/P8OHDPRKUP2rdi4RcQQkhhOe51Mz8TLGxscycOZN//OMf7o7HbzklqACp1hRCCF/qUoICOHnyJA0NDe6Mxa+1JKgEjQklIcHH0QghRO/nUhXfE0884fTsU0NDA8eOHeOGG27wWGD+pmWoDaNRQdW7PM6jEEKILnLpTDt16lSn+dDQUM477zz69OnjkaD8kcmkQ4uN2OQgynwdjBBCBACXEtQVV1zh4TD8X2mpFqOuDPok+joUIYQICC7dg1q5ciX79+93WrZ//35WrVrlkaD8kdmsJVGVcaCEEMJbXEpQ+/btY+jQoU7LhgwZwt69ez0SlD8ylWpIVIqkBZ8QQniJSwkqKCiI+vp6p2X19fXodDqPBOWPzMXNTcwD6L6bEEL4kksJavTo0bz66qvU1tYCUFtbS05ODmPGjPFkbH7FZNHJUO9CCOFFLjWSuPPOO3nxxReZNWsWkZGRVFdXM2bMGB566CFPx+cX6uuhqjaouReJC3wdjhBCBASXElRkZCSLFi2ivLwck8mE0WgkNjbWw6H5j5ZnoKQXCSGE8B6XEtSePXtISEggJSXFkZhOnjyJyWTiwgsv9GR8fsFstt9rS9CXoZ4xVL0QQgjPcOkeVE5ODmFhYU7LQkNDycnJ8UhQ/qa01H6YDAYbtOpRQwghhOe4lKAqKiqIO+PKIS4ujvLyck/E5Hda+uEzJqo+jkQIIQKHSwkqKSmJH374wWnZ3r17SUwMjF4VWqr4jP2CfRyJEEIEDpfuQd14442sXLmSqVOnkpSURHFxMRs3bjynoeDz8/NZu3YtiqKQmZnJjBkznNbv3buXZ5991pH0xo0b5zed0ZpMGsKoJTQlhiZfByOEEAHCpQQ1duxYlixZwpdffsmuXbswGAw8/vjjpKWlubQTRVHIyclhyZIlGAwGFi1aREZGBv369XMqN3z4cBYuXHjun8LDzKcUEjGjpMhDukII4S0ujxuRlpbmlJCOHTvG3//+d26//fazvrewsJDk5GSSmptoT5gwgR07drRJUP7KfNImD+kKIYSXndPARpWVlXzzzTds3ryZI0eOuNyThMViwWAwOOYNBgMFBQVtyh08eJAFCxYQFxfHHXfcQWpqapsyubm55ObmArBixQqMRuO5fIQ29Hr9WbdRZqmjLyVEDh1KRDf311WuxOkPJE73kjjdr6fEKnG6kKCsVis7d+5k06ZN5OfnYzAYKCsr45lnnuH88893aSeq2rb1m+aM5toDBw5kzZo1hIaGsmvXLp577jleeOGFNu/LysoiKyvLMW8ymVyKoSNGo/Gs2yg2RXERJVhCh2Dr5v66ypU4/YHE6V4Sp/v1lFgDKc6UlJR2l3faii8nJ4f77ruPnJwcjEYjTz31FC+++CLh4eFOV0RnYzAYMJvNjnmz2dym2Xp4eDihoaEApKenY7PZqKysdHkfnqKqYKoKs1fxJSf7OhwhhAgYnSaoDRs2APZWfLfccgtDhgzp0k4GDRpEUVERJSUlWK1W8vLyyMjIcCpTXl7uuNIqLCxEURSioqK6tD93qqjQ0KToSQypQA0P93U4QggRMDqt4nvxxRfZvHkzH3/8MW+88QYXXXQREydObLfKrjM6nY5Zs2axbNkyFEVhypQppKamOhJgdnY227ZtY8OGDeh0OoKDg3nkkUfaVAP6QstDuobYRh9HIoQQgaXTBJWYmMgNN9zADTfcwP79+9m0aRMvv/wydXV1/POf/+Saa65xuSVeeno66enpTsuys7Md01deeSVXXnllFz6CZzke0k2QXiSEEMKbXOpJAuzPKM2ZM4dXX32Vhx56CLPZzIIFCzwZm19wXEElu3yohBBCuME5NTMHCA4OZuLEiUycOBGLxeKJmPyKqdRezWhIDfFxJEIIEVi6dVkQHx/vrjj8luVYAwCxAyJ9HIkQQgQWqbc6C/OJRuIxo0tJ8HUoQggRUCRBnYW9H74SbPIMlBBCeJUkqLMoNelIoliGehdCCC9zqZFEdXU1H3/8MUePHqW+vt5p3R/+8AePBOYvTOUhjKEEJXGcr0MRQoiA4lKCev7557FarVx66aUEBwfWoH2mmnASQqsgKMjXoQghREBxKUEdPHiQ119/naAAO0k3NkJZYyTGxPqzFxZCCOFWLt2D6t+/v1Nnr4HCYrEfHmO81ceRCCFE4HHpCuqCCy7gmWee4YorriA2NtZp3dSpUz0Rl19w9CIh3RwJIYTXuZSgDhw4gMFg4Pvvv2+zrjcnKPMpBQBD38Cq2hRCCH/gUoJ68sknPR2HX7IcrgHA0D/Ux5EIIUTgcbkvvurqanbu3InFYiE+Pp6LL76YyMje3f2P6ai9cYRhUO/+nEII4Y9caiRx8OBBHnroIT7//HOOHj1Kbm4uDz30EAcPHvR0fD5lPtFEMA2ED+j9fQ4KIYS/cekK6o033mD27NlcdtlljmV5eXmsXbuW5cuXeyw4XzOXQiIlqH2kmyMhhPA2l66gioqKuPTSS52WjR8/nlOnTnkkKH9hsuhJpBQlAHptF0IIf+NSgkpOTiYvL89p2datW0nq5f3TmSpCSQipAD8Yel4IIQKNS1V8M2fOZMWKFXz66acYjUZKS0spKipi4cKFLu8oPz+ftWvXoigKmZmZzJgxo91yhYWFPP7448ybN4/x48e7vH1PKK2LZERkjU9jEEKIQOVSgho6dCgvvvgiu3btoqysjIsvvpj09HSXW/EpikJOTg5LlizBYDCwaNEiMjIy6NevX5ty77zzDmPGjDnnD+JuqgqljbEYYxp8HYoQQgQkl5uZR0ZGMmnSpC7tpLCwkOTkZEeV4IQJE9ixY0ebBPXpp58ybtw4Dh061KX9uFNNjYZ6NRSDQfF1KEIIEZA6TFDLli3j8ccfB+CJJ55A08F9GFeG27BYLBgMBse8wWCgoKCgTZnt27fz5JNP8te//tWl4D2p5RmohGS5/ySEEL7QYYKaPHmyY7q73Rmpatu+7M5MeG+88Qa33XYbWm3n7TZyc3PJzc0FYMWKFRiNxm7Fptfr291GwaljAPQbHNPtfbhDR3H6G4nTvSRO9+spsUqcnSSoiRMnOqb79u3L4MGD25QpLCx0aScGg8GpN3Sz2UxcXJxTmUOHDvH8888DUFlZye7du9FqtVxyySVO5bKyssjKynLMm0wml2LoiNFobHcbR787BQwiNLH7+3CHjuL0NxKne0mc7tdTYg2kOFNSUtpd7lIz86effrrd5cuWLXNp54MGDaKoqIiSkhKsVit5eXlkZGQ4lXnppZccr/HjxzN79uw2ycmbzMfsjSPiB0X5LAYhhAhknTaSUBR7AwFVVR2vFsXFxeh0Opd2otPpmDVrFsuWLUNRFKZMmUJqaiobNmwAIDs7u6vxe4yluSfzuGGGs5QUQgjhCZ0mqFtvvdUxfcsttzit02q1/PKXv3R5R+np6aSnpzst6ygxPfDAAy5v11NMpRpiKCc4PsLXoQghREDqNEH95S9/QVVVnnrqKafWehqNhujoaIKDgz0eoK+YyoJICCoDQnwdihBCBKROE1RCQgIAa9as8Uow/sRUHUZCaBWSoIQQwjdcflD322+/Zd++fVRWVjotf/DBB90elD8oqYtmcGKZr8MQQoiA5VIrvvfff59XX30VRVHYtm0bkZGR7Nmzh/DwcE/H5xuqSok1HmNck68jEUKIgOVSgtq4cSNLlixh5syZ6PV6Zs6cyWOPPUZpaamn4/MJpcSCGQOGhLYPGAshhPAOlxJUTU0N/fv3B+xPDVutVtLS0ti3b59Hg/OV8gILKloMfVxrRi+EEML9XLoHlZyczLFjx0hNTXU8vxQZGelyb+Y9TVmh/T6bIbX3tlIUQgh/51KCuvnmm6mqqgLgV7/6FS+88AL19fXMnj3bo8H5iuVoHQDxA3tnAhZCiJ7ApQTV+gHbwYMH8+KLL3osIH9gPmFvHBGfFu3jSIQQInB1mKCKi4td2kBvHPbd1NzNkSFZC0hDCSGE8IUOE9TDDz/s0gbWrVvntmD8hcmsR08TsbGSnIQQwlc6TFCtE8/GjRv5/vvvufHGG0lISKC0tJR//etfjBo1yitBepupIhhjcAVnGZpKCCGEB7l0Cl63bh1z5syhT58+6PV6+vTpw7333su7777r6fh8orQmgoSIGl+HIYQQAc2lBKWqKiUlJU7LSktLHcNx9CpNTZQ2xmKMrvd1JEIIEdBcasV39dVX88c//pErrrjCMXripk2buPrqqz0dn9dpS0ooIZGM+DpfhyKEEAHNpQQ1ffp0+vfvz9atWzly5AixsbHcf//9jBkzxsPheZ+uuJgShmNI/MnXoQghREBzuTfzMWPG9MqEdKb6oyZqiMTQ1+VDI4QQwgM6PAt/8MEHXHfddUDnTclvvvlm90flQ2WHqwEw9A/zcSRCCBHYOkxQZrO53enezvyTvXFE/HlhQKNvgxFCiADWYYL69a9/7ZieO3dut3eUn5/P2rVrURSFzMxMZsyY4bR+x44drFu3Do1Gg06nY+bMmQwbNqzb+z1X5hNWABIS5SFdIYTwJa90daQoCjk5OSxZsgSDwcCiRYvIyMigX79+jjKjRo0iIyMDjUbD0aNHWb16NX/+859disGdTKUaAIzGXtiEXgghehCvdHVUWFhIcnKyI5lNmDCBHTt2OCWo0NBQx3RDQwMajcal/bubyWI/JAaDJCghhPAll7o66i6LxYLBYHDMGwwGCgoK2pTbvn07//jHP6ioqGDRokXtbis3N5fc3FwAVqxYgdFo7FZser3eaRumqjAig+pJTTV08i7vOzNOfyVxupfE6X49JVaJ8xyamXeHqra9n9PeFdIll1zCJZdcwr59+1i3bh2///3v25TJysoiKyvLMW8ymboVW8uDxwCa2lpKG2MwxtZgMjV0a7vu1jpOfyZxupfE6X49JdZAijMlJaXd5S4lKJvNxvr169m3b59j4MIWf/jDH876foPB0KZVYFxcXIflR4wYwUsvvURlZSXR0d4bk0l76hQlJGKMldZ7Qgjhay71xffmm2+Sm5vLiBEj+PHHHxk3bhwVFRWMHDnSpZ0MGjSIoqIiSkpKsFqt5OXlkZGR4VTm1KlTjiutH3/8EavVSlRU1Dl+nO7RtSQouf8khBA+59IV1H/+8x+WLVuG0Wjkvffe46qrrmL06NG8+uqrLu1Ep9Mxa9Ysli1bhqIoTJkyhdTUVDZs2ABAdnY227ZtY/Pmzeh0OoKDg5k3b57XG0rYuzm6jFHJIAMVCiGEb7mUoBobGx2NHIKDg2loaKBv374cOXLE5R2lp6c7DR0P9sTUYsaMGW2ejfK6UyWUkoAxtQLwr3tQQggRaFxKUH379uXQoUOkpaVx/vnn8/777xMWFkZ8fLyn4/OqiiOV2NBj6KPzdShCCBHwOr0H1TLe08yZM9E2Dy971113cfjwYXbu3Mm9997r+Qi9yHLc3jjCmCD3oIQQwtc6vYKaM2cOkyZNYtKkSfTv3x+APn36tNv8uzcwFdkAeUhXCCH8QacJ6te//jVff/01ixYtol+/fkyePJmJEyd6tem3N5lN9qtE6eZICCF8r9MENXbsWMaOHUtNTQ15eXls3ryZd955hwsvvJDJkyeTkZGBXt9Lxk1SVUwVwYAkKCGE8AcuZZeIiAimTZvGtGnTKCkpYfPmzbz55pu89tpr5OTkeDpGr9CUlVFqjUeDSlycJCghhPA1lx7UbdHU1ERhYSEFBQVUVFQ47kv1BvZnoBIxRNWjk0Z8Qgjhcy5dQR04cIBNmzaxdetWYmJiuPzyy5k9ezYJCQmejs9rdKdOUUwSCXFNvg5FCCEEZ0lQ7733Hl9//TXV1dWMHz+ehQsX+mQQQW/QFhdTwjgMMlChEEL4hU4TVEFBAbfccgtjx44lODjYWzH5hK6oiBISGZks9XtCCOEPOk1Qjz/+uLfi8DldcTElmiQmn32AYCGEEF5wTo0kerPGkxYq1Wh5SFcIIfyEJKhmlhPN3RzJM1BCCOEXJEE1M5fY/0qCEkII/yAJCsBqxVQWBIDBYPNxMEIIIUASFADa0lJKVPszXQnSk7kQQvgFSVCcHuodpIpPCCH8hSQoTndzFBZiIzxcHtQVQgh/IAkK0DZfQRnibWg0vo5GCCEEuNgXnzvk5+ezdu1aFEUhMzOTGTNmOK3/+uuv+eijjwAIDQ1l9uzZDBgwwCux2av4LsYo3RwJIYTf8MoVlKIo5OTksHjxYlavXs2WLVs4fvy4U5nExESeeuopVq5cyfXXX8+rr77qjdCA5io+fQoGoyQoIYTwF15JUIWFhSQnJ5OUlIRer2fChAns2LHDqczQoUOJjIwEYPDgwZjNZm+EBjRX8WkSpYGEEEL4Ea9U8VksFgwGg2PeYDBQUFDQYfkvv/ySiy66qN11ubm55ObmArBixQqMRmO3YtPr9ailJkqs8aSmaru9PU/R6/V+G1trEqd7SZzu11NilTi9lKBUtW3VmaaD1gg//PADGzdu5I9//GO767OyssjKynLMm0ymbsVmNBqpOlFNkxpEREQFJlNNt7bnKUajsduf1RskTveSON2vp8QaSHGmpKS0u9wrVXwGg8Gpys5sNhMXF9em3NGjR3nllVdYsGABUVFR3ggNamsxVYYC8gyUEEL4E68kqEGDBlFUVERJSQlWq5W8vDwyMjKcyphMJlauXMmDDz7YYTb1iOZxoEASlBBC+BOvVPHpdDpmzZrFsmXLUBSFKVOmkJqayoYNGwDIzs7mX//6F9XV1bz++uuO96xYscLjsWlOnnQkKOmHTwgh/IfXnoNKT08nPT3daVl2drZjes6cOcyZM8db4Tho5ApKiICgqir19fUUFxfT0NDg63DOqrfFqaoqWq2W0NDQDtsgnMlrCcpvnTjhSFDx8ZKghOit6uvrCQoKIiQkBJ1O5+twzkqv1/e6OK1WK/X19YSFhblUPuC7OtIUFVGsSyE2ViEoyNfRCCE8RVEU9Hr5Te5Ler0eRXH9QiDgExRFRZSEpGI0yv0nIXozV6uVhGedy79DwCcozcmTFOv6yDhQQgjhZyRBnTxJiZqAwSAJSgjhORaLhWnTpjFt2jTGjBnDxRdf7JhvbGw8p23dcMMN7NmzB4A77riDiooKKioqeOONNxxlTp06xa9//Wt3foQ2+/a0wK6QVVU4eZJSNY4J0oJPCOFB8fHxfP755wCsWrWKiIgIp5bLVqu1S/fI3n77bQCOHTvGW2+9xcyZMwFITk7mtdde637gPhTQCUpTUUFTvY0yIjAaK30djhDCS6KfeIKgffvcus2mESOo7KCLto488sgjxMbG8sMPPzBq1CimT5/Ok08+6WjptmrVKtLS0qirq2P+/PkUFBSQlpZGfX29Yxvjxo3j008/5ZlnnuHo0aNMmzaNSZMmMXPmTO666y6+/PJL6uvrWbRoEd999x06nY4nn3ySyy67jHXr1vH5559TV1fHkSNH+PnPf86SJUsAWLhwIXv27KG+vp6rr76aRx991K3HyxUBnaB0p05hwt7JoVTxCSF84ccff2TdunXodDqqqqr44IMP0Ov1bNmyhT/96U+89tprvPXWW4SFhZGbm8u+ffu48sor22xn8eLF/Pe//3VcpR07dsyxrqXq74svvqCwsJBbb72Vr7/+GoC9e/eyfv16goODmTRpEnfffTd9+/blscceIy4uDpvNxs0338y+ffsYMWKE5w9IK4GdoIqLOSkP6QoRcM71SseTrrnmGsdzRJWVlTzyyCMcPnwYjUZDU1MTAP/5z3+YNWsWACNGjGD48OHntI8dO3Zw9913A5CWlka/fv348ccfAZg4cSLR0dEADBkyhBMnTtC3b1/+/e9/884772Cz2SguLqagoMDrCSqgG0m0DPUOkqCEEL4RHh7umH7uueeYMGECX375JW+//bZTDw3daSbf3ogSLYKDgx3TWq0Wq9XKTz/9xCuvvMK6devIzc0lMzPTqVrRWwI6QelaJSjph08I4WtVVVUkJycDsG7dOsfycePG8eGHHwJw4MAB9u/f3+a9ERERVFdXt7vd1u8/dOgQJ06cYNCgQZ3GERYWRnR0NKWlpWzcuLHLn6k7AjtBFRdTHDYAkCsoIYTv3X///Sxfvpxrr70Wm+30j+Y777yTmpoasrKyWLNmDWPGjGnz3vj4eMaOHcvUqVNZunSp07q77roLm81GZmYm999/P6tXryYkJKTDOEaOHMkFF1zAlClTmD9/PmPHjnXbZzwXGrWza78e4OTJk11+b9ysWTz57Y08XzWLH38swp8fNA+kwcu8QeJ0r54QZ21tLeHh4ej1eqxWq6/DOaveGmfLv0NrHQ2xFNCNJBqmTKHYdAmGk4pfJychhAhEAV3FV3vHHZQkjJR++IQQwg8FdIICKC2V+09CCOGPJEGVauQhXSGE8EMBnaBUFUpK5ApKCCH8kdcaSeTn57N27VoURSEzM5MZM2Y4rT9x4gRr1qzh8OHD3HLLLUyfPt3jMdXUaKiv18g9KCGE8ENeuYJSFIWcnBwWL17M6tWr2bJlC8ePH3cqExkZyd13380vfvELb4QEgMlk//hSxSeE8IbS0lIeeOABLr30Uq688kp+8Ytf8OmnnwKQl5fHnXfe6Zb9fPbZZxw8eLDddatWreLll1/u9P179uzh97//vSOuHTt2ONa99dZbvP/++52+35V9uMIrV1CFhYUkJyeTlJQEwIQJE9ixYwf9+vVzlImJiSEmJoZdu3Z5IyQASkvtCUoGKxRCeJqqqsyaNYsbb7yRl156CYDjx4+zYcOGLm2vs+E5PvvsM7KyshgyZEiXtj169GhGjx4NwNatW4mIiHA8rOuuJOoKryQoi8WCwWBwzBsMBgoKCrq0rdzcXHJzcwFYsWIFRqOxy3E1NtoffkpLi8Zo9O/nlfV6fbc+q7dInO4lcbpPcXGx44T+1FOx7N3r3tPfyJFWnn66/a6GADZv3kxISIij01eAAQMGcO+99wKg0+nQaDTo9XrKysp45JFHOHr0KGFhYaxcuZKRI0fy3HPPcerUKY4dO0Z8fDyPP/44jzzyCGazGYPBwPPPP09RURGff/4527Zt44UXXuBvf/sbAwYMcOxTq9Wi1WrR6/X88pe/JD09nW+++YbKykpWr17N+PHj2bJlC2vWrGH58uW8/fbb6HQ6PvjgA5555hm+/vprIiIimDt3Lm+//TZ///vfaWxsZODAgfzlL38hPDzcaR9nCgkJcfm74pUE1V5nFV3t+DArK4usrCzHfHeeXj98OByIRaczYzL591VUT3hSHyROd5M43aehoQGdToder0dRlE47UO0KRVE67VFh//79jBw5ssMyNpsNVVWxWq386U9/YtSoUeTk5PDNN9/w4IMP8vnnn6MoCnv27OHDDz8kLCyMu+66i+uvv56bbrqJd999l8WLF/O3v/2NadOmkZWVxTXXXAPgtE9FURyxqqpKY2Mjn3zyCV988QXPPfcc69atc8SSkpLCHXfc4TS44qZNmxzv/9nPfsYdd9zhiPnvf/87s2bNctrHmRoaGtp8V3zak4TBYMBsNjvmzWYzcXFx3th1p+QelBCB6Y9/9P0ApYsXL2b79u0EBwfz//7f/3Nat337dv72t78B9uEwysrKqKy0x5ydnU1YWBgAO3fu5PXXXwfg+uuv5+mnnz7nOK666ioALrzwwjZtA87mv//9L7Nnz6aiooKamhomT558zvvvjFcaSQwaNIiioiJKSkqwWq3k5eWRkZHhjV13ymzWEhOj0kmfiUII4RZDhgzhhx9+cMw/88wzvPfee04/3lt0Vut0Zj927ZU5Fy3Dbeh0unPu+2/evHksX76cL774gnnz5jkND+IOXklQOp2OWbNmsWzZMubNm8ell15KamoqGzZscNwgLC8vZ86cOXzyySd88MEHzJkzh9raWo/GZTLpSEjw6C6EEAKwXwk1NDTw5ptvOpbV1dW1W3b8+PF88MEHgL0VXXx8PFFRUW3KZWRk8NFHHwHwwQcfcMkllwD2VtE1NTVuibuzYTyqq6tJTEykqanJMZyHO3ntOaj09HTS09OdlmVnZzumY2Nj3dIs8VxUV2tITPTvxhFCiN5Bo9GQk5PDU089xV//+lcMBgNhYWEsXry4Tdn58+fz29/+lqysLEJDQ/nzn//c7jaXLl3K/Pnzefnll4mPj2f16tUAXHvttSxYsICcnBxeffVVp0YS52ratGncd999rF+/vk0V4oIFC/j5z39Ov379GDZsWIeJrKsCergNgNhYI+Xl/n1zF3rGTWiQON1N4nQfGW7DMzw53EZAd3UE0MFjBEIIIXws4BOUEEII/yQJSggREHr43Yxe41z+HSRBCSECglar7RH3dHozq9WKVut62pE7MEKIgBAaGkp9fT0ajcbtz+t4QkhISK+KU1VVtFotoaGhLm9bEpQQIiBoNBrCwsJ6RItD6BktI8GzcUoVnxBCCL8kCUoIIYRfkgQlhBDCL/X4niSEEEL0TgF/BbVw4UJfh+ASidO9JE736ilxQs+JVeKUBCWEEMJPSYISQgjhlwI+QbUePt6fSZzuJXG6V0+JE3pOrBKnNJIQQgjhpwL+CkoIIYR/kgQlhBDCLwVMX3z5+fmsXbsWRVHIzMxkxowZTutVVWXt2rXs3r2bkJAQ5s6dy/nnn+/VGE0mEy+99BLl5eVoNBqysrK46qqrnMrs3buXZ599lsTERADGjRvHDTfc4NU4AR544AFCQ0PRarXodDpWrFjhtN4fjufJkycdQ2ADlJSUcNNNN3H11Vc7lvnqeK5Zs4Zdu3YRExPDqlWrAKiurmb16tWUlpaSkJDAvHnziIyMbPPes32XPR3n22+/zc6dO9Hr9SQlJTF37lwiIiLavPds3xFvxPree+/xxRdfEB0dDcCtt95Kenp6m/f6+piuXr3aMTp4y4izzz33XJv3euuYdnQu8vp3VA0ANptNffDBB9VTp06pTU1N6qOPPqoeO3bMqczOnTvVZcuWqYqiqP/973/VRYsWeT1Oi8WiHjp0SFVVVa2trVUffvjhNnH+8MMP6vLly70e25nmzp2rVlRUdLjeH45nazabTZ09e7ZaUlLitNxXx3Pv3r3qoUOH1Pnz5zuWvf322+qHH36oqqqqfvjhh+rbb7/d5n2ufJc9HWd+fr5qtVodMbcXp6qe/Tvibu3Fum7dOvWjjz7q9H3+cExbe/PNN9X333+/3XXeOqYdnYu8/R0NiCq+wsJCkpOTSUpKQq/XM2HCBHbs2OFU5ttvv2XSpEloNBqGDBlCTU0NZWVlXo0zLi7OcZURFhZG3759sVgsXo3BXfzheLb2/fffk5ycTEJCgs9iaG3EiBFtfnnu2LGDyZMnAzB58uQ231Fw7bvs6ThHjx6NTqcDYMiQIX7zHW0vVlf4wzFtoaoqW7du5bLLLvPY/l3R0bnI29/RgKjis1gsGAwGx7zBYKCgoKBNGaPR6FTGYrEQFxfntThbKykp4fDhw6SlpbVZd/DgQRYsWEBcXBx33HEHqampPogQli1bBsC0adPaNDX1t+O5ZcuWDv/T+8vxrKiocByfuLg4Kisr25Rx5bvsTV9++SUTJkzocH1n3xFvWb9+PZs3b+b888/nzjvvbJMc/OmY7t+/n5iYGPr06dNhGW8f09bnIm9/RwMiQanttKTXaDTnXMZb6uvrWbVqFTNnziQ8PNxp3cCBA1mzZg2hoaHs2rWL5557jhdeeMHrMS5dupT4+HgqKip4+umnSUlJYcSIEY71/nQ8rVYrO3fu5Fe/+lWbdf5yPF3lT8f1gw8+QKfTcfnll7e7/mzfEW/Izs523FNct24db731FnPnznUq40/HtLMfUuD9Y9rZuagj7jyeAVHFZzAYMJvNjnmz2dzml7zBYHAadKu9Mt5gtVpZtWoVl19+OePGjWuzPjw83DEiZXp6Ojabrd1fMZ4WHx8PQExMDGPHjqWwsNBpvb8cT4Ddu3czcOBAYmNj26zzl+MJ9mPZUg1aVlbmuLHfmivfZW/46quv2LlzJw8//HCHJ5+zfUe8ITY2Fq1Wi1arJTMzk0OHDrUp4y/H1GazsX379k6vSL15TNs7F3n7OxoQCWrQoEEUFRVRUlKC1WolLy+PjIwMpzIZGRls3rwZVVU5ePAg4eHhXv+SqqrKyy+/TN++fbnmmmvaLVNeXu74hVJYWIiiKERFRXkzTOrr66mrq3NMf/fdd/Tv39+pjD8czxad/Sr1h+PZIiMjg02bNgGwadMmxo4d26aMK99lT8vPz+ejjz7iscceIyQkpN0yrnxHvKH1fc/t27e3W33rD8cU7PdJU1JSnKrHWvPmMe3oXOTt72jA9CSxa9cu3nzzTRRFYcqUKVx33XVs2LABsFcDqKpKTk4Oe/bsITg4mLlz5zJo0CCvxnjgwAGeeOIJ+vfv7/hVeuuttzquRLKzs/nss8/YsGEDOp2O4OBg7rzzToYOHerVOIuLi1m5ciVg/9U3ceJEvzyeAA0NDdx///385S9/cVRRtI7TV8fzz3/+M/v27aOqqoqYmBhuuukmxo4dy+rVqzGZTBiNRubPn09kZCQWi4VXXnmFRYsWAe1/l70Z54cffojVanXcyxk8eDD33nuvU5wdfUc8qb1Y9+7dy5EjR9BoNCQkJHDvvfcSFxfnd8d06tSpvPTSSwwePJjs7GxHWV8d047ORYMHD/bqdzRgEpQQQoieJSCq+IQQQvQ8kqCEEEL4JUlQQggh/JIkKCGEEH5JEpQQQgi/JAlKiB7spptu4tSpU74OQwiPCIiujoTwlgceeIDy8nK02tO//a644gruueceH0YlRM8kCUoIN3vssce48MILfR2GED2eJCghvOCrr77iiy++YODAgWzatIm4uDjuueceRo0aBdh7DHjttdc4cOAAkZGRXHvttY6eqhVF4X//93/ZuHEjFRUV9OnThwULFjh6i//uu+945plnqKqq4rLLLuOee+5Bo9Fw6tQp/vrXv3LkyBH0ej0XXHAB8+bN89kxEOJcSYISwksKCgoYN24cOTk5bN++nZUrV/LSSy8RGRnJ888/T2pqKq+88gonT55k6dKlJCUlMWrUKP7v//6PLVu2sGjRIvr06cPRo0ed+sDbtWsXy5cvp66ujscee4yMjAzGjBnDu+++y+jRo3nyySexWq38+OOPPvz0Qpw7SVBCuNlzzz3nGNAP4Pbbb0ev1xMTE8PVV1+NRqNhwoQJ/Pvf/2bXrl2MGDGCAwcOsHDhQoKDgxkwYACZmZls3ryZUaNG8cUXX3D77beTkpICwIABA5z2N2PGDCIiIoiIiGDkyJEcOXKEMWPGoNfrKS0tpaysDIPBwLBhw7x5GIToNklQQrjZggUL2tyD+uqrr4iPj3camiIhIQGLxUJZWRmRkZGEhYU51hmNRsfQEGazmaSkpA7313oYkZCQEOrr6wF7Ynz33XdZvHgxERERXHPNNUydOtUdH1EIr5AEJYSXWCwWVFV1JCmTyURGRgZxcXFUV1dTV1fnSFImk8kx9o/BYKC4uPich1aIjY1lzpw5gL136qVLlzJixAiSk5Pd+KmE8Bx5DkoIL6moqODTTz/FarWydetWTpw4wUUXXYTRaGTo0KH84x//oLGxkaNHj7Jx40bHSLWZmZmsW7eOoqIiVFXl6NGjVFVVnXV/W7dudQwcFxERAeDU/F0IfydXUEK42Z/+9CenRHDhhRcyduxYBg8eTFFREffccw+xsbHMnz/fMTjib37zG1577TXuu+8+IiMjufHGGx3VhNdccw1NTU08/fTTVFVV0bdvXx599NGzxnHo0CHeeOMNamtriY2N5e677yYxMdEzH1oID5DxoITwgpZm5kuXLvV1KEL0GHK9L4QQwi9JghJCCOGXpIpPCCGEX5IrKCGEEH5JEpQQQgi/JAlKCCGEX5IEJYQQwi9JghJCCOGX/j8xngC5LSikgQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vals['epochs'] = 20\n",
    "vals['eta'] = 0.001\n",
    "\n",
    "nn_tradit = TLPMiniBatchCrossEntropy(**vals)\n",
    "nn_better = TLPBetterInitial(**vals)\n",
    "\n",
    "%time nn_tradit.fit(X_train, y_train, print_progress=1, XY_test=(X_test, y_test))\n",
    "%time nn_better.fit(X_train, y_train, print_progress=1, XY_test=(X_test, y_test))\n",
    "\n",
    "print_result(nn_tradit,X_train,y_train,X_test,y_test,title=\"Traditional\",color=\"red\")\n",
    "print_result(nn_better,X_train,y_train,X_test,y_test,title=\"Glorot Initial\",color=\"blue\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 75/75"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 3s, sys: 17 s, total: 3min 20s\n",
      "Wall time: 51 s\n",
      "=================\n",
      "Long Run :\n",
      "Resubstitution acc: 0.8976333333333333\n",
      "Validation acc: 0.8641\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1B0lEQVR4nO3de1xUdf4/8NeZGe7DdQZBBSrBa1pGeEnNG8T2q74u31a72uaiW1Z2zw2S7aYkW7qappnFYu1u39xq61vf3TZ3WlMTTZJI00wwJU0EYZT7MJdzfn8MTIxcPAhz5oy8no8HD+acOXPOa0bkzedzPudzBEmSJBAREamMxtsBiIiIOsMCRUREqsQCRUREqsQCRUREqsQCRUREqsQCRUREqqRT6kAlJSUoKCiAKIpITU1FRkaG2/MNDQ149dVXUVlZCT8/P9x3331ISEhQKh4REamMIi0oURSRn5+Pp556CqtWrcLOnTtx4sQJt20++OADXHrppVixYgUWLVqETZs2KRGNiIhUSpECVVZWhtjYWMTExECn02HSpEkoKipy2+bEiRMYM2YMAGDw4ME4ffo0zp49q0Q8IiJSIUUKlNlshsFgcC0bDAaYzWa3bS655BJ8+eWXAJwF7fTp0x22AQCTyYSsrCxkZWV5NjQREXmVIuegOptNSRAEt+WMjAxs2rQJixcvRkJCAi677DJoNB3rZ1paGtLS0lzLJ0+e7FU2o9GI6urqXu1DacysDGZWhq9l9rW8gPozDxo0qNP1ihQog8GAmpoa13JNTQ0iIyPdtgkODsb9998PwFnQFi1ahAEDBigRj4iIVEiRLr7ExERUVFSgqqoKdrsdhYWFSElJcdumsbERdrsdAPDZZ59h5MiRCA4OViIeERGpkCItKK1Wi8zMTOTm5kIURcyYMQPx8fHYsmULACA9PR0//fQTXnnlFWg0GsTFxWHhwoVKRCMiIpVS7Dqo5ORkJCcnu61LT093PR42bBjWrFmjVBwiIlI5ziRBRESqxAJFRESqxAJFRESqpNg5KCIAgCRBU10NTWUlRKMRotEI6M7zYyiK0FRUQHf0KARRhBgUBCkkBFJwsNsXOrluDpIEiCKg1fY6NxyO824m2GwQamuhqatzfq+vhxQUBDEiAmJ4OKTISEiBgUBLC4R2X5JOBykgAAgKguTv33VehwPan36C7uhRaMxmSDod4Ofn/K7VQvD3R1BlJYTGRghNTYAodvicJK0WEATn5yUIP3/19CPR6YCAAEh+fs7Mbe/fagWsVgh2u/NY7bdpO3YbUYRw8iT8T5yA0NAAobHR+e8VEACp7cvfH9DpnMfTaJw/L5IE2GwQ7Hbn987+bQQBUtv2Wq3zsVbrfNz6XbDZnMdtaICm3WcGUXQeQ5Kcx/T3h+TvD8nPD4LRCP+Ghm732+F9tn1mbZ91+89eCXV10NbUuP4/9OaoEvBzfo3G+XMdEdE3Oc/BAkVd0pw8CaGlBY64OMDP78J24nAg6IMPoN23D4b9++F3+DA07aawkjQaiNHRcMTEQAoNdf5CCgyEFBAAwWaD7ocfoD16FJrm5vMeSgwMdBYqSYJgtf78y7L1OJK/v+sXjThgAOzx8XDExcGRkABRr4fu+HFof/wRuvJyaI8fh6apCQNtNucvQFG8sPd/gaSAAIhhYc7//GFhkEJCnEX6xx9d76krkd0+q05GbwfoIV/LCwAxHtpv/cMPo/53v/PIvlmg+iubzfWX6rn89u6FfsMGBH7yCQRJgqTVwhEXB/ull8I2ZgzqH3us09edy3/HDoQ/9xz8vvsOUlQUMHQomm+6CfZhw+CIjYWmpgbaykpoKiuhrax0/hXb0OBsVVgsgCDAftllaJk0CfbERNiHDAECAlytA6Gpyfm4uRlCU9PPfwFrNM6/2AMCnK0LrdZVsGC1QrBYoK2shK68HAE7dkDT1ATAWcQcgwfDkZAAS2oqAqOj0WS3O/8a9vNz/qV8vr94/fychSUsDFJ4OES9HoLFAs3Zs86v2loIzc2uItz2JdjtEFpanC0ri8X5Xurq3Fpi9qFDYUlPh+Oyy2AfMsTZ+nQ4fm5J2O2IiI2FuaXF1cKERvPzZ9X21b6F0Pa9pyTp59ZLS4vzs5Uktz8CJK3W2bJpv03rtY4uggB9bCzqRBFiSAgkvd75ebd+Dm0tTDgczn05HM59tLaMXC3Izlosouh8r62vc73e4XCut9udOVuPK+r1kIKCfm79tP17i+LPrUKrFRHBwag1mzvsE6Lo/ExaH59LaGuRtf/sFRIaGor6xka3lpt0Aa0313to9z5sI0d6ILETC1Q/FPjppwhfvBiaujrYRo6E7corYb3qKkhBQQgpKEDAnj0Qw8PR8MADsA8Z4mxRHDsG3bFjCH3lFWhqalD70ktd/rLWHjmC8KVLEfjvf8OekADzxo3Q//rXbrOJqIYkQXPmDIT6ejgGDXJrKfoZjahX8fQwnZGMRjjOySzp9V5KI0+I0YgWH/qcJaMRVh/KCzg/42YfywywQCnCb98+hD3/PISWFjTeeScsv/yl8y+17kgS/IqLEfDFF3AMHOhsQSQmOvt6rVb4lZTAf+9e+O3dC93Ro7AnJsJ2+eXOr9GjnS2WcwiNjQh79lmEvP02bJdfjqbbboN/SQmCPvwQIX/+MwDAHheH2ueeQ9Ptt0MKCemwj9A//AGha9bANno0mubN65A55I03ELZsGaTAQNQtWYKGzEwgMBB6pfrae0oQIEZFAZ18XkTkXSxQHiScOYOwP/wBwX/5i3NAQGQkIh9/HOLzz6Np9mw03X47HJdc4ixWrb/ANZWVCH7/fQRt3gy/srIO+3RERUHT1IRoi8W5PHAg7ElJ8P/ySwR/8IFrO3tCAqwpKbBefTWsKSkQmpsR+cgj0JaXo37RItQ//jjQemIbogjdDz9AU1kJ64QJ3Q5aqF+8GH7ffYfwZ56BffhwWK+5pvWAdoQ/8wxCNm1C8/XXozYvD2J0dB99kkTUH7FAeYIoInjzZoTm5kJTW4vGzEzUP/EEpNBQZyF56y2EvPUW9Pn5AACp9byFFBoK7fHjEBwOWFNScPall9D8//4fNGYzdEeOQPfDD9D98AMCjUbUjhoFa3IyxHazAAtmM/wOHIDfgQPwLy5GwM6dCP77313P2+PiUPPee7BOnOieV6OBPSkJSEo6/3vTaHBm7VoYb7oJkffcg+pPPoEYEYHI++5D4H/+g4b77kPdU091PqKOiKgHBKmze2H4ELXdbsOvuBjhv/89/EtK0DJ+PGpzc2EfNarDdprTpxHw2WfQms3Ok+CtX/aEBDTNmQNHN8VCdmZJgvbECfh/9RU0lZVouuMOSGFhvXl7LtojRxB9003OEX4AdN9/j9rcXDTddVfvMqsIMyvD1zL7Wl5A/Zm9eruN/kBz+jTCXngBwX/7GxwxMTizZg2ab765y4EEYnQ0mm+7zbOhBAGO+Hg0x8f3+a4diYk4s349ou66C1JICMxvvYWW6dP7/DhE1H+xQPVUSwuCPvwQ2pqan4ft1tcj6OOPIVgsqH/gATQ89JDqR071hZYZM1CzeTMcsbFwJCZ6Ow4RXWRYoHpAqK1F1Pz5CNi1y7VODAqCFByMlmuuQV1OTr/7RW2dPNnbEYjoIsUCJZOmogKGu+6CrqwMZ1avhuWGG5yj7zgYgIjII1igZNB9/z2i5s6Fpq4ONW+9BevUqd6ORER00WOBOg//oiJE3X03pIAAVL//PuyjR3s7EhFRv8AC1Q3NyZOImjcPosGAmrffhsMDo+GIiKhzPIHSFYcDkQ8+CFitqNm0icWJiEhhbEF1Qf/yywjYvRtnVq/udyPziIjUQLECVVJSgoKCAoiiiNTUVGRkZLg939TUhDVr1qCmpgYOhwP/9V//hRkzZigVz43/7t0IXbUKTb/6FZrnzPFKBiKi/k6RAiWKIvLz85GTkwODwYDs7GykpKQgrnWaHAD417/+hbi4OGRlZaGurg4PP/wwrr32WujOd7fVPiaYzYhctAiOhATUvvCCoscmIqKfKXIOqqysDLGxsYiJiYFOp8OkSZNQVFTkto0gCLBYLJAkCRaLBXq9HhqlrzGSJEQ8/jg01dU4s2FDv5gNgohIrRRpnpjNZhgMBteywWBAaWmp2zbXX389XnzxRdx7771obm7Go48+2mmBMplMMJlMAIC8vDwYjb27+bJOp3PtQygpgd+WLbDn5iLcS92LcrTP7CuYWRnM7Hm+lhfwzcyAQgWqswnThXMmUf3mm29wySWX4Omnn0ZlZSWWLl2KESNGIDg42G27tLQ0pKWluZZ7O0Nv+1l+/X/8EUYAZ4cNU/UdM9U+M3FnmFkZzOx5vpYXUH/mrmYzV6QPzWAwuN3uu6amBpGRkW7bbN26FRMmTIAgCIiNjcWAAQN6fSuNnhJsNgDO+zMREZF3KVKgEhMTUVFRgaqqKtjtdhQWFiIlJcVtG6PRiP379wMAzp49i5MnT2LAgAFKxPuZ1er83nanWSIi8hpFuvi0Wi0yMzORm5sLURQxY8YMxMfHY8uWLQCA9PR0/OpXv8L69evx+OOPAwDuvPNOhPXRzfXkYguKiEg9FBvDnZycjOTkZLd16enprsdRUVHIyclRKk7nWgsUW1BERN7HqY7aEVq7+CQWKCIir2OBaoddfERE6sEC1R4HSRARqQYLVDuuLj62oIiIvI4Fqh128RERqQcLVHvs4iMiUg0WqHYEmw2SRgNotd6OQkTU77FAtWezsfVERKQSLFDtCFYrzz8REakEC1Q7gs3GAkVEpBIsUO2xi4+ISDVYoNphFx8RkXqwQLVns3EePiIilWCBakdgFx8RkWqwQLXDLj4iIvVggWrPagVYoIiIVIEFqh2B56CIiFSDBaoddvEREakHC1R7Nhu7+IiIVEKn1IFKSkpQUFAAURSRmpqKjIwMt+c/+ugj7NixAwAgiiJOnDiB/Px86PV6pSKyi4+ISEUUKVCiKCI/Px85OTkwGAzIzs5GSkoK4uLiXNvMmjULs2bNAgB89dVX+Mc//qFocQLAQRJERCqiSBdfWVkZYmNjERMTA51Oh0mTJqGoqKjL7Xfu3InJkycrEc0N5+IjIlIPWS2o3/3ud5g2bRomT56MiIiIHh/EbDbDYDC4lg0GA0pLSzvdtqWlBSUlJZg/f36nz5tMJphMJgBAXl4ejEZjj/O0p9PpXPvQOhwICAvr9T49rX1mX8HMymBmz/O1vIBvZgZkFqibb74ZX3zxBd555x2MHDkSU6dOxfjx4+Ev83yNJEkd1gmC0Om2e/fuxfDhw7vs3ktLS0NaWpprubq6WlaGrhiNRtc+YiwWWBwO1PZyn57WPrOvYGZlMLPn+VpeQP2ZBw0a1Ol6WQVq4sSJmDhxIhoaGlBYWIhPP/0Ub7zxBsaPH4+pU6di9OjR3b7eYDCgpqbGtVxTU4PIyMhOt925cyemTJkiJ1af4yAJIiL16NE5KL1ej2nTpuG6666D0WjEl19+iddeew0PP/ww9u3b1+XrEhMTUVFRgaqqKtjtdhQWFiIlJaXDdk1NTTh48GCnzynCamWBIiJSCVktKFEUsW/fPmzfvh3FxcUYNmwYMjIyXN18u3fvxtq1a/H66693+nqtVovMzEzk5uZCFEXMmDED8fHx2LJlCwAgPT0dALBnzx5ceeWVCAwM7KO31zMCR/EREamGrAJ17733IiwsDFOnTsXcuXMRFRXl9vzEiRPx6aefdruP5ORkJCcnu61rK0xtpk+fjunTp8uJ1PccDgiiyFF8REQqIatAZWVlITExsdttnnnmmT4J5DVWq/M7u/iIiFRB1jmoEydOoLy83G3dsWPHsH37do+E8gbBZgMAtqCIiFRCVoHavHmz23VMgHPY4jvvvOORUN7gKlBsQRERqYKsAtXc3Izg4GC3dcHBwWhsbPRIKK9o6+JjC4qISBVkFai4uDjs3r3bbd2ePXvc5tLzdeziIyJSF1mDJO68804sX74chYWFiI2NxalTp7B//35kZ2d7Op9yOEiCiEhVZBWoESNGYOXKlfjiiy9QXV2NpKQkzJs3zyfnduoKW1BEROoi+3YbRqOxwz2cLiYsUERE6iK7QH311Vc4ePAg6urq3NYvWrSoz0N5Bbv4iIhURdYgiXfffRcbN26EKIrYvXs39Ho9vvnmmw4j+3wZh5kTEamLrBbU1q1bkZOTg4SEBHz++eeYN28epkyZgvfff9/T+RQjtLag2MVHRKQOslpQjY2NSEhIAOC88ZXdbkdSUhIOHjzo0XCKYhcfEZGqyGpBxcbG4vjx44iPj3fNQq7X67u8qaAv4iAJIiJ1kVWgbr31VtTX1wNwXhP18ssvw2KxYMGCBR4Np6jWAsUWFBGROpy3QImiCH9/fwwbNgwAkJSUhLVr13o8mNJ4DoqISF3Oew5Ko9HgxRdfhE4ne0S6T2IXHxGRusgaJDFy5EgcPnzY01m8i4MkiIhURVazKDo6GsuXL0dKSgoMBgMEQXA9d+utt3osnJLYgiIiUhdZBcpqtWLcuHEAALPZ7NFA3iJwkAQRkarIKlD3339/rw9UUlKCgoICiKKI1NTUTuf1O3DgADZt2gSHw4HQ0FA899xzvT6ubBwkQUSkKrIKVGVlZZfPxcTEnPf1oigiPz8fOTk5MBgMyM7ORkpKitv9pBobG/HGG29gyZIlMBqNqK2tlROtzwg2GyRBALRaRY9LRESdk1WgHnrooS6f27x583lfX1ZWhtjYWFcxmzRpEoqKitwK1BdffIEJEya4buERHh4uJ1rfsdmAgACg3fk1IiLyHlkF6twidPbsWbz77rsYOXKkrIOYzWYYDAbXssFgQGlpqds2FRUVsNvtePbZZ9Hc3IwbbrgB06ZNk7X/viC0tLB7j4hIRS7o4qaIiAjMmzcPDz/8MKZMmXLe7SVJ6rBOOKel4nA4cPToUfz+97+H1WpFTk4Ohg4dikGDBrltZzKZYDKZAAB5eXm9vmmiTqeD0WiEVquFEBDgEzdhbMvsS5hZGczseb6WF/DNzMAFFigAOHnyJFpaWmRtazAYUFNT41quqalBZGRkh21CQ0MRGBiIwMBAjBw5EuXl5R0KVFpaGtLS0lzL1dXVF/oWADhvxFhdXY3w+noE6nS93p8S2jL7EmZWBjN7nq/lBdSf+dzf821kFainn37arcXT0tKC48ePY/bs2bIOnpiYiIqKClRVVSEqKgqFhYUdzmulpKTgT3/6ExwOB+x2O8rKynDjjTfK2n9fEKxWdvEREamIrAI1c+ZMt+XAwEBccsklGDhwoKyDaLVaZGZmIjc3F6IoYsaMGa5Z0QEgPT0dcXFxGDt2LJ544gloNBrMnDnTdYsPRdhsAAsUEZFqyCpQ06dP7/WBkpOTkZyc7LYuPT3dbXnWrFmYNWtWr491IQSbjXfTJSJSEVlz8a1YsQLfffed27rvvvsOK1eu9Egob2AXHxGRusgqUAcPHsTw4cPd1g0bNgwHDhzwSCivYBcfEZGqyCpQfn5+sFgsbussFgu0F9GsC4LVyi4+IiIVkVWgrrzySmzcuBFNTU0AgKamJuTn52Ps2LGezKYogS0oIiJVkTVI4te//jXWrl2LzMxM6PV6NDQ0YOzYsXjwwQc9nU85NhuksDBvpyAiolayCpRer0d2djbOnj2L6upqGI1GREREeDiasgSrFVJAgLdjEBFRK1kF6ptvvkF0dDQGDRrkKkwnT55EdXU1rrjiCk/mU47Vyi4+IiIVkXUOKj8/H0FBQW7rAgMDkZ+f75FQ3iDYbBxmTkSkIrIKVG1tbYe58yIjI3H27FlPZPIKjuIjIlIXWQUqJiYG3377rdu6AwcOYMCAAR4J5RUcxUdEpCqyzkHNmTMHK1aswMyZMxETE4PKykps3bq1T24Frxbs4iMiUhdZLahx48YhJycHFosFxcXFsFgsWLJkCcaNG+fpfMqxWgF28RERqYbs+0ElJSUhKSnJtXz8+HH85S9/wdy5cz0STGlsQRERqUuPblhYV1eHL774Atu3b8exY8cunpkkHA4IDgcHSRARqch5C5TdbsfevXuxbds2lJSUwGAw4MyZM3jhhRcwZMgQJTJ6ns3m/M4WFBGRanRboPLz81FYWAitVouJEyfi2WefxbBhw3DPPffAYDAoldHjhNYCxS4+IiL16LZAbdmyBXq9HnPmzMHkyZMRHBysVC5FuQoUu/iIiFSj2wK1du1abN++HR999BE2bdqEq666ClOmTIEkSUrlU4bV6vzOFhQRkWp0O8x8wIABmD17NtauXYucnBzo9Xps2LABdXV1+J//+R+cOHFCqZweJbQWKLagiIjUQ9Z1UAAwcuRILFy4EBs3bsSDDz6ImpoaLF682JPZlNPWgmKBIiJSjR4NMwcAf39/TJkyBVOmTIHZbJb9upKSEhQUFEAURaSmpiIjI8Pt+QMHDuDFF190TZ80YcIEzJ49u6fxLggHSRARqU+PC1R7UVFRsrYTRRH5+fnIycmBwWBAdnY2UlJSEBcX57bdyJEjkZWV1ZtIF4QFiohIfWR38fVGWVkZYmNjERMTA51Oh0mTJqGoqEiJQ8vDLj4iItXpVQtKLrPZ7HbdlMFgQGlpaYftDh8+jMWLFyMyMhJ33XUX4uPjO2xjMplgMpkAAHl5eTAajb3KptPpENE6fD7MaITUy/0pQafT9fp9K42ZlcHMnudreQHfzAwoVKA6G5YuCILb8mWXXYb169cjMDAQxcXFeOmll7BmzZoOr0tLS0NaWpprubq6ulfZjEYj6qurYQBQ29wMay/3pwSj0djr9600ZlYGM3uer+UF1J950KBBna6XVaAaGhrw0Ucfoby8HBaLxe2555577ryvNxgMqKmpcS3X1NR0uAFi+4uAk5OTkZ+fj7q6OoSFhcmJ2Dttw8x5DoqISDVkFaiXX34Zdrsd11xzDfwv4DxNYmIiKioqUFVVhaioKBQWFuKhhx5y2+bs2bMIDw+HIAgoKyuDKIoIDQ3t8bEuBAdJEBGpj6wCdfjwYbzxxhvwu8Bf4FqtFpmZmcjNzYUoipgxYwbi4+OxZcsWAEB6ejp2796NLVu2QKvVwt/fH4888kiHbkCP4SAJIiLVkVWgEhISUFNTg9jY2As+UHJyMpKTk93Wpaenux5ff/31uP766y94/73BFhQRkfrIKlCjR4/GCy+8gOnTpyMiIsLtuZkzZ3oil6JYoIiI1EdWgTp06BAMBgP279/f4bmLoUChpcX5PSDAuzmIiMhFVoF65plnPJ3Dq9iCIiJSH9nXQTU0NGDv3r0wm82IiorC1VdfDb1e78lsimkrUBwkQUSkHrKmOjp8+DAefPBB/Pvf/0Z5eTlMJhMefPBBHD582NP5lMHroIiIVEdWC2rTpk1YsGABJk+e7FpXWFiIgoICLF++3GPhlCLYbJAEAdBqvR2FiIhayWpBVVRU4JprrnFbN3HiRJw6dcojoRRnszm795S67oqIiM5LVoGKjY1FYWGh27pdu3YhJibGI6GUJlit7N4jIlIZWV188+bNQ15eHj755BMYjUacPn0aFRUVXrl3kycINhsLFBGRysgqUMOHD8fatWtRXFyMM2fO4Oqrr0ZycvJFM4rP1cVHRESqIXuYuV6vx9SpUz2ZxWvYxUdEpD5dFqjc3FwsWbIEAPD00093OXGrnNttqJ7NBrBAERGpSpcFatq0aa7HF8V0Rt0QrFZI7OIjIlKVLgvUlClTXI8HDx6MoUOHdtimrKzMM6kUxgJFRKQ+soaZL1u2rNP1ubm5fRrGa9jFR0SkOt0OkhBFEQAgSZLrq01lZSW0F8nMC2xBERGpT7cF6vbbb3c9vu2229ye02g0+O///m/PpFKYYLNBCgrydgwiImqn2wL1yiuvQJIkPPvss26j9QRBQFhYGPwvllaHzQYpLMzbKYiIqJ1uC1R0dDQAYP369YqE8RZ28RERqY/sC3W/+uorHDx4EHV1dW7rFy1aJOv1JSUlKCgogCiKSE1NRUZGRqfblZWVYcmSJXj00UcxceJEufF6h4MkiIhUR9YovnfffRcbN26EKIrYvXs39Ho9vvnmGwQHB8s6iCiKyM/Px1NPPYVVq1Zh586dOHHiRKfb/fWvf8XYsWN79CZ6i3PxERGpj6wCtXXrVuTk5GDevHnQ6XSYN28ennzySZw+fVrWQcrKyhAbG4uYmBjodDpMmjQJRUVFHbb75JNPMGHCBIQpfD6IXXxEROojq4uvsbERCQkJzhfodLDb7UhKSsLBgwdlHcRsNsNgMLiWDQYDSktLO2yzZ88ePPPMM3j11Ve73JfJZILJZAIA5OXlwWg0ysrQFZ1OB8HhQGBoKPx6uS+l6HS6Xr9vpTGzMpjZ83wtL+CbmQGZBSo2NhbHjx9HfHw84uPjsWXLFuj1etmzmbe/fqrNuXP7bdq0CXfeeSc0mu4bdWlpaUhLS3MtV1dXy8rQFaPRCF1LC5pFEXW93JdSjEZjr9+30phZGczseb6WF1B/5kGDBnW6XlaBuvXWW1FfXw8AuOOOO7BmzRpYLBYsWLBA1sENBgNqampcyzU1NYiMjHTb5siRI3j55ZcBAHV1dfj666+h0Wgwfvx4WcfolZYWDpIgIlIZWQUqOTnZ9Xjo0KFYu3Ztjw6SmJiIiooKVFVVISoqCoWFhXjooYfctlm3bp3b46uvvlqZ4oTWQRI8B0VEpCpdFqjKykpZO5Bz23etVovMzEzk5uZCFEXMmDHD1VUIAOnp6TLjeoDDAcHhYIEiIlKZLgvUuS2crmzevFnWdsnJyW4tMaDrwvTAAw/I2mefsNmc39nFR0SkKl0WqPaFZ+vWrdi/fz/mzJmD6OhonD59Gu+99x7GjBmjSEiPsloBgNdBERGpjKzroDZv3oyFCxdi4MCB0Ol0GDhwIO655x688847ns7neW0Fil18RESqIqtASZKEqqoqt3WnT5923Y7Dp7UWKHbxERGpi6xRfDfeeCOef/55TJ8+3TWeftu2bbjxxhs9nc/z2MVHRKRKsgrUrFmzkJCQgF27duHYsWOIiIjAfffdp/iceR7R1oJiFx8RkarIns187NixF0dBOofQOoqPLSgiInXpskD9/e9/x8033wyg+6Hkt956a9+nUhIHSRARqVKXBercqYkuWhwkQUSkSl0WqN/+9reux/fff78iYbyipQUAu/iIiNRGkamOVI2DJIiIVEmxqY5Ui+egiIhUSdZURxc1juIjIlIlWTNJXMwEdvEREamSrOugHA4HPv30Uxw8eNB148I2zz33nEeCKYYzSRARqZKsFtSbb74Jk8mEUaNG4YcffsCECRNQW1uLyy+/3NP5PI8FiohIlWQVqC+//BJPPfUUbrjhBmi1Wtxwww1YvHgxDhw44Ol8nscuPiIiVZJVoKxWKwwGAwDA398fLS0tGDx4MI4dO+bJbMpgC4qISJVknYMaPHgwjhw5gqSkJAwZMgTvvvsugoKCEBUV5el8nscWFBGRKnXbgmq739O8efOg0Tg3vfvuu3H06FHs3bsX99xzj+cTehqHmRMRqVK3LaiFCxdi6tSpmDp1KhISEgAAAwcOxO9//3tFwinBNcxcJ3tidyIiUkC3v5V/+9vfYseOHcjOzkZcXBymTZuGKVOmICwsrMcHKikpQUFBAURRRGpqKjIyMtyeLyoqwubNmyEIArRaLebNm4cRI0b0+Dg91tLinEVCEDx/LCIikq3bAjVu3DiMGzcOjY2NKCwsxPbt2/HXv/4VV1xxBaZNm4aUlBToZLQ8RFFEfn4+cnJyYDAYkJ2djZSUFMTFxbm2GTNmDFJSUiAIAsrLy7Fq1SqsXr2612/wvKxWdu8REamQrFF8ISEhuO6667B06VKsWrUKiYmJePPNN3HvvffKOkhZWRliY2MRExMDnU6HSZMmoaioyG2bwMBACK2tmJaWFtdjj7NaOQ8fEZEK9ejEi81mQ1lZGUpLS1FbW4vhw4fLep3ZbHYNUwcAg8GA0tLSDtvt2bMHb7/9Nmpra5Gdnd3pvkwmE0wmEwAgLy8PRqOxJ2+hA43dDgQG9no/StLpdD6VF2BmpTCz5/laXsA3MwMyC9ShQ4ewbds27Nq1C+Hh4bj22muxYMECREdHyzqIJEkd1nXWQho/fjzGjx+PgwcPYvPmzZ0OxkhLS0NaWpprubq6WlaGrsS0tEDSanu9HyUZjUafygsws1KY2fN8LS+g/syDBg3qdH23Bepvf/sbduzYgYaGBkycOBFZWVkXNHDBYDB0uENvZGRkl9uPGjUK69atQ11d3QUNyOgRq5V30yUiUqFuC1RpaSluu+02jBs3Dv69OE+TmJiIiooKVFVVISoqCoWFhR3uN3Xq1CnExMRAEAT88MMPsNvtCA0NveBjyiVYrRB5DoqISHW6LVBLlizpk4NotVpkZmYiNzcXoihixowZiI+Px5YtWwAA6enp2L17N7Zv3w6tVgt/f388+uijygyU4Cg+IiJVUuzq1OTkZCQnJ7utS09Pdz3OyMjocG2UItjFR0SkSv3+hoWw2TjMnIhIhVig2IIiIlIlFiheqEtEpEosUC0tHCRBRKRC/b5ACeziIyJSpX5foNjFR0SkTixQHMVHRKRKLFDs4iMiUiUWKHbxERGpEgsUW1BERKrEAsW5+IiIVKl/FyhRhGC3s4uPiEiF+neBstmc39mCIiJSnX5doITWAsUuPiIi9enXBQpWq/M7u/iIiFSnXxcoobVAsQVFRKQ+/btAtXXxsQVFRKQ6/bpAubr42IIiIlKdfl2g2IIiIlIvnVIHKikpQUFBAURRRGpqKjIyMtye37FjB/73f/8XABAYGIgFCxbg0ksv9WgmFigiIvVSpAUliiLy8/Px1FNPYdWqVdi5cydOnDjhts2AAQPw7LPPYsWKFfjVr36FjRs3ej4Yu/iIiFRLkQJVVlaG2NhYxMTEQKfTYdKkSSgqKnLbZvjw4dDr9QCAoUOHoqamxuO5eB0UEZF6KdLFZzabYTAYXMsGgwGlpaVdbv+f//wHV111VafPmUwmmEwmAEBeXh6MRuMF5xKCggAA4dHRkHqxH6XpdLpevW9vYGZlMLPn+VpewDczAwoVKEmSOqwTBKHTbb/99lts3boVzz//fKfPp6WlIS0tzbVcXV19wbkCqqthAHC2qQm2XuxHaUajsVfv2xuYWRnM7Hm+lhdQf+ZBgwZ1ul6RLj6DweDWZVdTU4PIyMgO25WXl+O1117D4sWLERoa6vFc7OIjIlIvRQpUYmIiKioqUFVVBbvdjsLCQqSkpLhtU11djRUrVmDRokVdVtM+x6mOiIhUS5EuPq1Wi8zMTOTm5kIURcyYMQPx8fHYsmULACA9PR3vvfceGhoa8MYbb7hek5eX59FcbEEREamXYtdBJScnIzk52W1denq66/HChQuxcOFCpeIA+HkuPragiIjUp1/PJAFOFktEpFqKtaDUiF18RCSHJEmwWCwQRREOhwMtLS3ejtQjlZWVXs8sSRI0Gg0CAwO7HMV9rn5doFx31A0I8G4OIlI1i8UCPz8/6HQ66HQ6aLVab0fqEbVkttvtsFgsCGq9BvV8+nUXH+8HRURyiKIIna5//z3fF3Q6HURRlL19/y5QbS0o/uARUTfkdknR+fXks+zXBQo2m3Mmc/7wERGpTr9uOghWK4eYE5FPGDp0aLdzmPal2bNno7KyEgEBAfD398eLL76I0aNHK3Ls9vp9C4oFioioo1deeQUmkwl33303li1b5pUMbEGxQBFRD+hzcqD99ts+3adt1CjUdTFBdne+/fZbZGVlwWKx4JJLLsHKlSsRERGB2bNn46qrrkJhYSFqa2uxevVqpKSkoLm5GY888giOHDmCpKQknDhxArm5ubjyyiu7PMbVV1+NV199FQCwcuVKhISEuCZVmDlzJt58800AwNy5czF+/Hh89dVXiI2NxZ/+9CfZo/W60q9bUAJbUETkwx555BEsWbIEJpMJI0aMwB//+EfXc3a7Hf/4xz/w3HPPYcWKFQCAN998E+Hh4TCZTHjkkUewb9++8x5j69at+MUvfnHe7Y4ePYq7774bW7duRVhYGP75z39e+Btr1a9bULDZeDddIuqRhmXLYLfbvR0DdXV1qK2txTXXXAMAmDNnDu69917X8zfccAMA4IorrsDx48cBAHv27MH8+fMBACNGjMDIkSO73P+iRYvQ1NQEURTxr3/967x54uPjXeep2h+zN/p3C6qlxTmKj4joIuPf+rtNq9XC4XAA6PzefF155ZVXsHv3bmRkZGDJkiWufbW/jqn97BQB7SY8aH/M3ujfBYpdfETko8LCwhAeHo4vv/wSAPD+++9j4sSJ3b5m/Pjx+PjjjwEAhw8fxqFDh7rd3s/PD7/73e9QXFyM0tJSxMfHY//+/QCA/fv348cff+yDd9I1dvGxQBGRD2hubsbVV1/tWr7nnnuwevVq1yCJhIQEt3NQnbn77rvx8MMPIy0tDZdffjlGjhx53pvDBgUF4Z577sGGDRuwbNkyvPfee7juuuswduxYDBkypE/eW1cEqSdtPhU6efLkBb9WU1mJqNBQVAcH92Eiz1P77Zs7w8zKYGbPaGpqQnDr7wmdTqeKc1A90ZbZ4XDAZrMhMDAQx44dw6233oodO3a4ugOV0P6zbNPVTWr7dQtKjIkBjEZA5f85iIj6QnNzM+bMmQNb6zRvy5cvV7Q49VS/LlBERP2JXq/HJ5984u0YsvXrQRJERHL4+JkQVenJZ8kCRUR0HhqNxufOO6mR3W6HRiO/7CjWxVdSUoKCggKIoojU1FRkZGS4Pf/TTz9h/fr1OHr0KG677TbMmjVLqWhERN0KDAyExWJBS0sLAgMDvX532p4KCAjweub2d9SVS5ECJYoi8vPzkZOTA4PBgOzsbKSkpCAuLs61jV6vx29+8xsUFRUpEYmISDZBEFzzyvnCqMNz+WJmQKEuvrKyMsTGxiImJgY6nQ6TJk3qUIjCw8ORlJSkitsSExGR9ylSoMxmMwwGg2vZYDDAbDYrcWgiIvJRinTxdTZq40JvoWwymWAymQAAeXl5MBqNvcqm0+l6vQ+lMbMymFkZvpbZ1/ICvpkZUKhAGQwG1NTUuJZramoQGRl5QftKS0tDWlqaa7kvLjJT84VqXWFmZTCzMnwts6/lBXwzsyJdfImJiaioqEBVVRXsdjsKCwuRkpKixKHPKysry9sReoyZlcHMyvC1zL6WF/DNzIBCLSitVovMzEzk5uZCFEXMmDED8fHx2LJlCwAgPT0dZ8+eRVZWFpqbmyEIAv75z3/ij3/8Y4c5m4iIqH9Q7Dqo5ORkJCcnu61LT093PY6IiMCGDRuUikNERCrX72eSaH8+y1cwszKYWRm+ltnX8gK+mRm4CG63QUREF6d+34IiIiJ1YoEiIiJV6rf3gzrf5LVqsX79ehQXFyM8PBwrV64EADQ0NGDVqlU4ffo0oqOj8eijj0Kv13s5qVN1dTXWrVuHs2fPQhAEpKWl4YYbblB1ZqvVimeeecZ1x9GJEyfilltuUXXmNqIoIisrC1FRUcjKylJ95gceeACBgYHQaDTQarXIy8tTfebGxkZs2LABx48fhyAIuO+++zBo0CDVZj558iRWrVrlWq6qqsItt9yCadOmqTZzl6R+yOFwSIsWLZJOnTol2Ww26YknnpCOHz/u7VidOnDggHTkyBHpsccec63785//LH3wwQeSJEnSBx98IP35z3/2UrqOzGazdOTIEUmSJKmpqUl66KGHpOPHj6s6syiKUnNzsyRJkmSz2aTs7Gzp+++/V3XmNh9//LG0evVqafny5ZIkqftnQ5Ik6f7775dqa2vd1qk989q1ayWTySRJkvPno6GhQfWZ2zgcDmnBggVSVVWVz2Rur1928cmZvFYtRo0a1eGvnKKiIkybNg0AMG3aNFVlj4yMxJAhQwAAQUFBGDx4MMxms6ozC4LgugWAw+GAw+GAIAiqzgw4Z2QpLi5Gamqqa53aM3dGzZmbmprw3XffYebMmQCcUwaFhISoOnN7+/fvR2xsLKKjo30mc3v9souvs8lrS0tLvZioZ2pra11TRUVGRqKurs7LiTpXVVWFo0ePIikpSfWZRVHEk08+iVOnTuEXv/gFhg4dqvrMmzZtwty5c9Hc3Oxap/bMAJCbmwsAuO6665CWlqbqzFVVVQgLC8P69etRXl6OIUOGYN68earO3N7OnTsxefJkAL7xs3GuflmgpD6cvJY6Z7FYsHLlSsybN88nZgPRaDR46aWX0NjYiBUrVuDHH3/0dqRu7d27F+Hh4RgyZAgOHDjg7TiyLV26FFFRUaitrcWyZcswaNAgb0fqlsPhwNGjR5GZmYmhQ4eioKAAH374obdjyWK327F3717ccccd3o5ywfplgerLyWu9ITw8HGfOnEFkZCTOnDmDsLAwb0dyY7fbsXLlSlx77bWYMGECAPVnbhMSEoJRo0ahpKRE1Zm///57fPXVV/j6669htVrR3NyMNWvWqDozAERFRQFw/jyMGzcOZWVlqs5sMBhgMBgwdOhQAMDEiRPx4Ycfqjpzm6+//hqXXXYZIiIiAPjO/8H2+uU5KDVPXitHSkoKtm3bBgDYtm0bxo0b5+VEP5MkCRs2bMDgwYNx0003udarOXNdXR0aGxsBOEf07d+/H4MHD1Z15jvuuAMbNmzAunXr8Mgjj2D06NF46KGHVJ3ZYrG4uiMtFgv27duHhIQEVWeOiIiAwWDAyZMnATjP6cTFxak6c5v23XuAuv8PdqXfziRRXFyMN9980zV57c033+ztSJ1avXo1Dh48iPr6eoSHh+OWW27BuHHjsGrVKlRXV8NoNOKxxx5TzXDRQ4cO4emnn0ZCQoKr2/T222/H0KFDVZu5vLwc69atgyiKkCQJ11xzDWbPno36+nrVZm7vwIED+Pjjj5GVlaXqzJWVlVixYgUAZ9fZlClTcPPNN6s6MwAcO3YMGzZsgN1ux4ABA3D//fdDkiRVZ25pacF9992HV155xdXFrvbPuTP9tkAREZG69csuPiIiUj8WKCIiUiUWKCIiUiUWKCIiUiUWKCIiUiUWKCIfdsstt+DUqVPejkHkEf1yJgkiT3nggQdw9uxZaDQ//+03ffp0zJ8/34upiHwTCxRRH3vyySdxxRVXeDsGkc9jgSJSwOeff47PPvsMl112GbZt24bIyEjMnz8fY8aMAeCcYf/111/HoUOHoNfr8ctf/hJpaWkAnDOtf/jhh9i6dStqa2sxcOBALF68GEajEQCwb98+vPDCC6ivr8fkyZMxf/58CIKAU6dO4dVXX8WxY8eg0+kwevRoPProo177DIh6igWKSCGlpaWYMGEC8vPzsWfPHqxYsQLr1q2DXq/Hyy+/jPj4eLz22ms4efIkli5dipiYGIwZMwb/93//h507dyI7OxsDBw5EeXk5AgICXPstLi7G8uXL0dzcjCeffBIpKSkYO3Ys3nnnHVx55ZWuuwX/8MMPXnz3RD3HAkXUx1566SVotVrX8ty5c6HT6RAeHo4bb7wRgiBg0qRJ+Pjjj1FcXIxRo0bh0KFDyMrKgr+/Py699FKkpqZi+/btGDNmDD777DPMnTvXdWuKSy+91O14GRkZCAkJQUhICC6//HIcO3YMY8eOhU6nw+nTp3HmzBkYDAaMGDFCyY+BqNdYoIj62OLFizucg/r8888RFRXldt+x6OhomM1mnDlzBnq9HkFBQa7njEYjjhw5AsB5O5iYmJguj9d2OwUACAgIgMViAeAsjO+88w6eeuophISE4KabbnLdGZbIF7BAESnEbDZDkiRXkaqurkZKSgoiIyPR0NCA5uZmV5Gqrq523TvJYDCgsrISCQkJPTpeREQEFi5cCMA5y/zSpUsxatQoxMbG9uG7IvIcXgdFpJDa2lp88sknsNvt2LVrF3766SdcddVVMBqNGD58ON5++21YrVaUl5dj69atuPbaawEAqamp2Lx5MyoqKiBJEsrLy1FfX3/e4+3atct1Y86QkBAAcBv+TqR2bEER9bE//OEPboXgiiuuwLhx4zB06FBUVFRg/vz5iIiIwGOPPYbQ0FAAwMMPP4zXX38d9957L/R6PebMmePqJrzppptgs9mwbNky1NfXY/DgwXjiiSfOm+PIkSPYtGkTmpqaEBERgd/85jcYMGCAZ940kQfwflBECmgbZr506VJvRyHyGWzvExGRKrFAERGRKrGLj4iIVIktKCIiUiUWKCIiUiUWKCIiUiUWKCIiUiUWKCIiUqX/D5RQtwu1iaRhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# just for fun, let's look at the output of what happens when we let our system loose\n",
    "# these values have been hand tuned\n",
    "vals = {'n_hidden':50, \n",
    "         'C':1e-2, 'epochs':75, 'eta':0.005, \n",
    "         'alpha':0.1, 'decrease_const':0.1,\n",
    "         'decrease_iter':20,\n",
    "         'minibatches':len(X_train)/256,\n",
    "         'shuffle':True,'random_state':1}\n",
    "\n",
    "nn_long_sigmoid = TLPBetterInitial(**vals)\n",
    "\n",
    "%time nn_long_sigmoid.fit(X_train, y_train, print_progress=1, XY_test=(X_test,y_test))\n",
    "print_result(nn_long_sigmoid,X_train,y_train,X_test,y_test,title=\"Long Run\",color=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "______\n",
    "**Self Test**: Resubstitution accuracy is more important than validation accuracy when training.\n",
    "- A. True\n",
    "- B. False\n",
    "- C. It depends\n",
    "\n",
    "\n",
    "\n",
    "## [Back To Slides]\n",
    "_____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rectified Linear Units\n",
    "Now lets look at another type of activation function. The rectified linear unit (ReLu) is an interesting choice to use because of its simplicity. Moreover, we will learn that its backpropagated derivative has some characteristics that are advantageous. \n",
    "\n",
    "We define the ReLu neuron as:\n",
    "\n",
    "$$\n",
    "    \\phi(\\mathbf{z}^{(i)})=\\left\\{\n",
    "                \\begin{array}{ll}\n",
    "                \\mathbf{z}^{(i)}  \\text{,  if }\\mathbf{z}^{(i)}>0 \\\\\n",
    "                \\mathbf{0} \\text{,      else } \\\\\n",
    "                \\end{array}\n",
    "              \\right.\n",
    "$$\n",
    "\n",
    "Which has a very simple derivative:\n",
    "$$\n",
    "    \\frac{\\partial\\phi(\\mathbf{z}^{(i)})}{\\partial\\mathbf{z}^{(i)}}=\\left\\{\n",
    "                \\begin{array}{ll}\n",
    "                \\mathbf{1} \\text{,  if }\\mathbf{z}^{(i)}>0 \\\\\n",
    "                \\mathbf{0} \\text{,      else } \\\\\n",
    "                \\end{array}\n",
    "              \\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TLPReLU(TLPBetterInitial):\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        # suggested relu/sigmoid bounds\n",
    "        #Xavier Glorot, Yoshua Bengio ; \n",
    "        #  Understanding the difficulty of training deep feedforward neural networks\n",
    "        #  Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.\n",
    "        init_bound = np.sqrt(2)*np.sqrt(6. / (self.n_hidden + self.n_features_))\n",
    "        W1 = np.random.uniform(-init_bound, init_bound,(self.n_hidden, self.n_features_))\n",
    "        \n",
    "        # reduce the final layer magnitude in order to balance the size of the gradients\n",
    "        # between \n",
    "        init_bound = 4*np.sqrt(6 / (self.n_output_ + self.n_hidden))\n",
    "        W2 = np.random.uniform(-init_bound, init_bound,(self.n_output_, self.n_hidden)) \n",
    "        \n",
    "        b1 = np.zeros((self.n_hidden, 1))\n",
    "        b2 = np.zeros((self.n_output_, 1))\n",
    "        \n",
    "        return W1, W2, b1, b2\n",
    "    \n",
    "    @staticmethod\n",
    "    def _relu(Z):\n",
    "        return np.maximum(0, Z.copy())\n",
    "        \n",
    "    def _feedforward(self, X, W1, W2, b1, b2):\n",
    "        \"\"\"Compute feedforward step\n",
    "        \"\"\"\n",
    "        # A1->W1->ReLu->A2->W2->Sigmoid\n",
    "        A1 = X.T\n",
    "        Z1 = W1 @ A1 + b1\n",
    "        A2 = self._relu(Z1)\n",
    "        Z2 = W2 @ A2 + b2\n",
    "        A3 = self._sigmoid(Z2) # never use relu as last layer for classification (yuck!)\n",
    "        return A1, Z1, A2, Z2, A3\n",
    "    \n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        # vectorized backpropagation\n",
    "        V2 = (A3-Y_enc) \n",
    "        \n",
    "        # V3[Z2<=0] = 0 # could can change to be relu back prop on this layer too!\n",
    "        \n",
    "        # old update: V1 = A2*(1-A2)*(W2.T @ V2)\n",
    "        # the derivative of sigmoid was A2(1-A2), but now that is relu\n",
    "        # so we change it to:\n",
    "        \n",
    "        V1 = (W2.T @ V2) \n",
    "        V1[Z1<=0] = 0\n",
    "        # relu derivative only zeros out certain values! easy!\n",
    "        \n",
    "        gradW2 = V2 @ A2.T\n",
    "        gradW1 = V1 @ A1.T\n",
    "        \n",
    "        gradb2 = np.sum(V2, axis=1).reshape((-1,1))\n",
    "        gradb1 = np.sum(V1, axis=1).reshape((-1,1))\n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        gradW1 += (W1 * self.l2_C)\n",
    "        gradW2 += (W2 * self.l2_C)\n",
    "\n",
    "        return gradW1, gradW2, gradb1, gradb2\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 75/75"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 55s, sys: 16.6 s, total: 3min 11s\n",
      "Wall time: 48.9 s\n",
      "{'n_hidden': 50, 'C': 0.01, 'epochs': 75, 'eta': 0.0001, 'alpha': 0.1, 'decrease_const': 0.1, 'decrease_iter': 20, 'minibatches': 234.375, 'shuffle': True, 'random_state': 1}\n"
     ]
    }
   ],
   "source": [
    "vals['eta'] = 0.0001 # needs much smaller starting learning rate\n",
    "nn_long_relu = TLPReLU(**vals) # same as previous parameter values\n",
    "%time nn_long_relu.fit(X_train, y_train, print_progress=True, XY_test=(X_test,y_test))\n",
    "print(vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=================\n",
      "ReLU :\n",
      "Resubstitution acc: 0.877\n",
      "Validation acc: 0.8551\n",
      "=================\n",
      "Sigmoid :\n",
      "Resubstitution acc: 0.8976333333333333\n",
      "Validation acc: 0.8641\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/FklEQVR4nO3de3wU9b3/8dfszm4298tuSLgEud8UxRjkqgjE1KrlcKy3qj2lYC3etT+tRClibSpVLCJiqZoDPR5bEdta9VSLaRGUACIURZBCuJVLICSBhFz2Mjvz+2PJkiUJbCC7mZXP8/HIg93Z2Zn3bsJ89vud735HMQzDQAghhDAZS2cHEEIIIVojBUoIIYQpSYESQghhSlKghBBCmJIUKCGEEKYkBUoIIYQpqdHa0aZNm1i8eDG6rjNx4kQmT54c8nhdXR2/+c1vOHz4MDabjbvvvpuePXtGK54QQgiTiUoLStd1iouLefzxx5k3bx6rV69m//79Iev8+c9/plevXsydO5f77ruPJUuWRCOaEEIIk4pKgSorKyM7O5usrCxUVWX06NGsX78+ZJ39+/czdOhQALp3786RI0c4duxYNOIJIYQwoagUqOrqapxOZ/C+0+mkuro6ZJ0LLriAdevWAYGCduTIkRbrAJSUlDBjxgxmzJgR2dBCCCE6VVTOQbU2m5KiKCH3J0+ezJIlS3j00Ufp2bMnvXv3xmJpWT/z8/PJz88P3j948OA5ZXO5XFRWVp7TNqJNMkeHZI6OWMsca3nB/Jm7devW6vKoFCin00lVVVXwflVVFenp6SHrJCQkcM899wCBgnbffffRpUuXaMQTQghhQlHp4uvbty/l5eVUVFSgaRqlpaXk5eWFrFNfX4+maQD8/e9/Z/DgwSQkJEQjnhBCCBOKSgvKarUydepUioqK0HWd8ePHk5OTw/LlywEoKCjgwIEDvPTSS1gsFnr06MH06dOjEU0IIYRJRe17ULm5ueTm5oYsKygoCN4eMGAAL774YrTiCCGEMDmZSUIIIYQpSYESQghhSlKghBBCmFLUzkEJAYCmYd2zB2tFBXpmJv6sLIzkZDjle3EhGhtR9+xB3b0b/H6MhASMxMTAvwkJ6PHxwfvY7YHn+P3g9aJ4vYHbdjuG3Q422+n31ZzfDx4PiqaBzwe6fsanKD4fltpalGPHsNTWYjl+HCM+Hj0tLfhjOBwoXi+K243i8YDbDTYbRlwchsMR+DchAeLiWm6/rg7rnj2ou3ZhOXo08DyrNfivxWYjoaICS309SkMD6HrwfTISEtATEsBqDbwHFkvgX0XBCPc9aU5VA+9p03t74vXj8QT+9fkCuWw2iIvDsNsDWZu/HsNAUVXiDhxAqa/HUlcX+B2feB9wOE4+78SPYbWiAPh8J383fn/rGa3WQE6L5eTzLRZQ1cB9rzfwXtXXo9TVoTQ0oOg6GEbg963rgeec+P1gs2FxOnE0NLTYVot9tPaeWizBn7N6z8+SkpaGrbo68HqaXttZb0w5+RosFvTMTPSsrI4L24wUKNEmdccOlPp6tF69MNLSzm4jbjeJv/891i++IPOrr1B37gwcvJrRHQ707Gz05OTAgezED5qGumsX1gMHUFr5sndrDFUFw0Bp64AFGHFx+DMz8efk4O/RA61nT4zERKz796Pu3Yv13/9G3bcPxe2m9a8PRofhcKCnpKCnpATylZdjrag44/PSmm9DUcJ+7zqT88yrmEpGZwc4C5kR2u7xBx/k+E9/GpFtS4E6Tym1taBpGBmn/FczDOI+/pik3/yGuNWrg4v1tDS0Xr3wXXwxtbNmYcTHn34HhoHj/fdJKSpC3bcPo3dvfP364Z44Ea1/f/zZ2Viqq7EeOoT18GEshw9jqasLtCg8nsAnaUXBm5eHdsstaH374u/dG8NuD3zabWhAaWwMthSUhobA8sbGQIug6ZO9zRb4VOvzBVotJz7hWw8dwrp/P3Gffkr8oUMohoGelIT/ggvQ+vfHM2ECjqwsGjyewDZU9WSL43QvW1XRU1MxUlMDhSU5OdBSOnoUy7FjWGpqUBobg62lppYFmobi8QR+3O5Aa+L4cZTa2sBz6urQBg1C690brXdvfL1740vPREULtiIUv5+0rl2p9ngCLSaHI5DX48HS9B41NgZaG02fpA3j7AqYYbR4TzGMwOux2QKvSVUDr+vEeni9LT84KArJ2dkc8/sxkpLQExPBam3RwlR0PZDb7w9sw2I52XJU1cC+Tv3dnPqcU24rfn8g64n9GklJgfft1Bamroe8zoykJI5WVp7cpqaF5mu63dp71tR6MYzAelGSnJxMbX19i5ZzuzW9hhOvQzEMtD59Oj7wCVKgzjeGQfxbb5E6axaWujq0Cy7Ad8kleIcNw0hIIHHJEmzbtuHPzqb2iSfQ+vbFuns36t69qLt3k/D661iOHePoyy+3+Qdu++ILUmbPJu6zz/ANHkzlm29iLfhPduw4ytGjFqqrLTTUKSSm6yT3NEhO1klONjjR+AkeOzUNvF4Fj+fEj1shSdXpcoFORoZO00xY9fUK27er/OtfKjt22NB1SLIZJKo6SYkGCQkGVqtBUw+Mqho4HJCYqJOUZJBsd2PUNfD1IRc7ymxs325j5+cqoGKx+LDZwG43iIsLbKvpJz7eoLFRobbWQm2tQk2NBb8fnE6dzEwdl8uP06mTkGBgtxvY7WDrapzoZTRC6l1lpYVD1VbKywM/Ph907arT7QI/3br5ycz0s2+fytatKls/tvH11zZqaxUyMnS6dNHJzPTjcunExcVRU+PF7Q68ZxYLpKXppKXppKbqpKYGipHPB5qmBHvH/H6l6fiKrisnjl9G8D1uyhro2TGCx7fmxzu/HxobFdxuhcZGBa9XwWYLvG9xcRAXF/g9aJoSrJF+v4LN5qChwR1cDoGeWFUNvFdWq9H0p3vi70NpXl9P9MQpIX87zW83PdZaHbZYwGYL/O2pqtHsc8jJ349hhL4/drsDj8cd8p6c6Vgf2MbJ97zptZ58fmB/zV9jW5nPRlxcHB6P54zrnZonnBr2rW+5ub6/uwNStiQFKgrsn3xC6lNPoXg81N92G4233IJ+asvlVJpG3MqVxH36Kf6uXdH69Am0InJy4OhR4laswL5xI7YNG1B370br2xffhRfiGzIE30UX4e/dm+DR5QSlupq0xx4j/q9/xTNyJJ4JE7Bt2oRtwwbi330XAN/gwRydP5/GSZOornNQU6PAgJPb6PXWS3R/8Rf4LryQuvvuC9m+oRs0PPpr+r/5a2ocmSwYsJDXfFPZN9VOQ4MF6Lh+aqvVwOXSUVWDAwdO/hk3HQQD+zs7WVl++vbVSE6G+nrweqGx0YLbrdDQcPKnsVHB4TBITTVITdVJSdGxWmHHDpU1aywcPWo9885OER+vk52tExdnsHatlZqa0NeRkKAzeLDGpEmNuFw6R45YqKiwcOSIld27VWw2CzabisMRKAyapnDwoEpNjYVjxyxoWugRx2ptOigbwdMyitJUEJRmBaD5qQsl5H7TAdVqDRTt+HgDhyNQlP1+5USxBI9Hwe8PrNd0SsliMbDZLCiKI7gcwOdTCDQMA/82/9DfdPBs+jlZKI0WDYQzHWybioamnSwegcKgBItK035U1TjxAceC3+8Ivv7T9Caf8l4HGnpNBbGp8DYvsND+AhHuvv3+0x/uQxtHStjFcehQ35lXOkuK0dpMrjHEzJPFWg4eJPWpp4h//320Cy7An5VF3GefYcTF0XjddTR873v4L7gg0CWUmAiKglpWRvzSpST88Y9YDx/GUNVAF84Jhs0WPIdjWCxoAwei9e2LumsX6vbtwXX1lBS8l12G97LLaLw0j39v8zF04SPYa6vZ+cMZHLz1bnQl8End61VQKo6g7TvCJ7XD2PyVnc2b7ZSXt3aANXgn4Ta+07CUv937e3KmX8XevVY+fEdh4hv3M7nxTRYzhafSf01qTiI9egRaAL17O4iLqyMjQyc9XSchQae+3sLx4wrHjwdaILoeemCxWjnx6fvkp/DaWuXEQdnKkSMWPB6Fvn01Bg3SGDjQR8+e/hP/GaGhQaGuLlBMdP3kJ2C/P1Bc6uoCP/X1FhQF+vXT6N/fR1qa0SF/Gz4fVFcHCpvXq+D1Bg64Pl/LA7/TqZOd7Sc11Qg5KNXXKxw8aOXwYQvdu/u54AL/qZ87Qpwus2EE3pPAe2u0a7xIJJl9ItNTxVpeMH/mtiaLlQIViV+c10vSq6+S9MILKLrO8fvuo+7uu8HhQN22jcT/+R/i//jHwHmWEwyLBSM5GUtNDYbVimfCBBpuuQX3xIkodXWoO3cGitCuXcRnZnJs0CB8w4ZhJCWd3K/Hg7pjB7avvsK+cSOWdRuIK/sXFgK/4i0M4Xbe4AuGtRldUQz69dMYOtTHhRf6yMwM7Sc/etTChk80Zv+jgD56GSNZyxEy+YsymTHGaj69bhYZv/oxaaFzAZv+P0hrJHN0xFrmWMsL5s8sBaoNHf2Li/vHP0idNQt1924aCwqofeop/K1cul6pqyPu00+xVFcHT4RbamrQevak8YYb0E/M5G4YgaJw6JCFw4cDn6Td7mTKyxuD5z4aGxVSUw2cTh2n009amsEnn9j5v/+LJ1GrYfolq/jWxXspH/efeK3xwW4TRQmMZLbZDGw2g8REg4EDNRITz/wnYew9gOvb13LckoJqMUg5fpBj81/APWlSq+ub/T9IayRzdMRa5ljLC+bP3KmX2zgfWHfvJnX2bBwlJWi9e1P1P/+DZ+LENtf3OZLYP/Q6jh5tOsluoaZG4fBhK/ues7Jvn8q+fVYOHrTi9bbsh7Fak06c9wj0+TcNPmhaNyVF54c/rOcHP/DQu3cekMdAAM58ojQcygXdOb7kVZw334yekkLVsrfwnTJDvRBCnAspUO2k1NWR8PrrWCsrTw5vPn4cx4oVGDYbNTNnUj9tWvALo4YBu3ZZWbMmjm3bVPbsCfzs22dtccK6icvlJyfHz8UX+7j22kays3WysvwnfnQGDkynoaGyxfkDw4C6OoXqagtduujEx0e2cey9/HKOvP8+usuFnp0d0X0JIc4/UqDawXL4MBn/9V/Yv/oq8CXKZjMYNNxwA8d/+lP8XbLYtcvKunVxlJbaWbMmjkOHAoMNkpN1evXSuOgiH9dd10jPnn4yMgKjv5paQy5XYFjy6SQmQmNjy+WKAsnJBsnJYQ4r6gDaRRdFbV9CiPOLFKgwqWVlZNx+O5bqaqpefx3PhAnBx/79bysffeRg3c/sfPaZnSNHAgUpM9PP6NEeRo/2MmqUhz59/KYYNSWEELFAClQYbOvX45wyBUNVqXr7bXyXXAJAVZWFefOSeP31RDRNoUcPjSuu8DBypJcRI7z07atJQRJCiLMkBeoM4j76iIzp0/FnZ1P1xhv4e/WisVHh1VcTWbgwicZGhe99r4F7762jZ8/oda0JIcQ3nRSo07Du3En6PffgGzCA6v/9X46QyZsLE/jv/07k0CErBQWNPP74cfr31868MSGEEO0iBaotHg8Zd9+NERfHqof/h0VP9eW99+LxehVGj/awcOFRRo70dnZKIYT4xpIC1YaUoiJsW7YwZ8zbFP7wEpKSdG67rYEf/KCeAQOkxSSEEJEWtQK1adMmFi9ejK7rTJw4kcmTJ4c83tDQwIsvvkhVVRV+v5/vfOc7jB8/PlrxQsQtX05ScTHbvv1jCj/4LlOn1vHYY8dJSorpSTeEECKmRKVA6bpOcXExM2fOxOl0UlhYSF5eHj169Aiu8+GHH9KjRw9mzJhBbW0tDz74IFdccQWqGt1GnuXAAdIffhjPRUO54V9z6dVLY+bM2tYubiqEECKCzv6aBO1QVlZGdnY2WVlZqKrK6NGjWb9+fcg6iqLgdrsxDAO3201SUhKW003bHAl+P+n33w8+Hy9fuYSvdyXx85/XSHESQohOEJXmSXV1NU7nyYs6O51OduzYEbLONddcw7PPPsuPf/xjGhsbefjhh1stUCUlJZSUlAAwZ84cXC7XOWVTVTW4DWX9emzr1lH19IvM+tWlXHutzi23JAPJ57SPjtY8c6yQzNEhmSMv1vJCbGaGKBWo1iZMV075BusXX3zBBRdcwKxZszh8+DBPP/00gwYNIiEhIWS9/Px88vPzg/fPdYbe5rP82g8exAW8sHwAXi888cQRKivN990ms89M3BrJHB2SOfJiLS+YP3Nbs5lHpQ/N6XRSVVUVvF9VVUV6eugFg1asWMGIESNQFIXs7Gy6dOlyzpfSaK+mCwGWfJLC9Ol19OplvuIkhBDni6gUqL59+1JeXk5FRQWaplFaWkreKZdmcLlcbN68GYBjx45x8OBBupy4JlK06O7A95pSXRbuv7/uDGsLIYSIpKh08VmtVqZOnUpRURG6rjN+/HhycnJYvnw5AAUFBXz3u9/l5Zdf5v/9v/8HwO23305KSko04gX9u0wnE7htinbGGcWFEEJEVtTGcOfm5pKbmxuyrKCgIHg7IyODmTNnRitOq3z1gS/gZuVYOzWHEEKIKHXxxQrDEzgHZY2XCTaEEKKzSYFqRvcEzkFZ4+2dnEQIIYQUqOZOtKDUBGlBCSFEZ5MC1YzhPdHFlyAtKCGE6GxSoJox3E0tKFsnJxFCCCEFqhnFGzgHpcogCSGE6HRSoJrzetGwYo+XYeZCCNHZpEA15/XhxY7dLl/SFUKIziYFqhnFFyhQNpsUKCGE6GxSoJpRfN4TBaqzkwghhJAC1UxTCyra10kUQgjRkhyKm1E0Hz5FvgMlhBBmIAWqGYvPKwVKCCFMQgpUM4rmxafEdXYMIYQQSIEKYdV8+CwyQkIIIcxAClQzVr8HzSJdfEIIYQZSoJqxaj4pUEIIYRJSoJqx+r34pUAJIYQpSIFqxur34bfKOSghhDCDqE3bvWnTJhYvXoyu60ycOJHJkyeHPP7uu+/yySefAKDrOvv376e4uJikpKRoRcSqe/HbpAUlhBBmEJUCpes6xcXFzJw5E6fTSWFhIXl5efTo0SO4zqRJk5g0aRIAn3/+Of/3f/8X1eIEoOpeaUEJIYRJRKWLr6ysjOzsbLKyslBVldGjR7N+/fo211+9ejVjxoyJRrQQqu5FV6UFJYQQZhBWC+qnP/0p48aNY8yYMaSlpbV7J9XV1TidzuB9p9PJjh07Wl3X4/GwadMmpk2b1urjJSUllJSUADBnzhxcLle78zSnqmpwG8d0H4bdcc7bjLTmmWOFZI4OyRx5sZYXYjMzhFmgbrjhBj799FPefPNNBg8ezJVXXsnll1+O3R5ea8MwWl6+QlGUVtfdsGEDAwcObLN7Lz8/n/z8/OD9ysrKsDK0xeVyBbfhMLz4FPWctxlpzTPHCskcHZI58mItL5g/c7du3VpdHlaBGjlyJCNHjqSuro7S0lL+9re/8dprr3H55Zdz5ZVXctFFF532+U6nk6qqquD9qqoq0tPTW1139erVjB07NpxYHU7VvRhyrQ0hhDCFdp2DSkpKYty4cVx99dW4XC7WrVvHb3/7Wx588EG+/PLLNp/Xt29fysvLqaioQNM0SktLycvLa7FeQ0MDW7dubfWxaLAbHjkHJYQQJhFWC0rXdb788ktWrVrFxo0bGTBgAJMnTw52861du5YFCxbw6quvtvp8q9XK1KlTKSoqQtd1xo8fT05ODsuXLwegoKAAgM8++4xLLrkEh8PRQS+vHQwDO9KCEkIIswirQP34xz8mJSWFK6+8kjvuuIOMjIyQx0eOHMnf/va3024jNzeX3NzckGVNhanJVVddxVVXXRVOpI7n92PBwJDvQQkhhCmEVaBmzJhB3759T7vOk08+2SGBOovi8wFIC0oIIUwirHNQ+/fvZ+/evSHL9uzZw6pVqyISqlN4vYF/7VKghBDCDMIqUEuXLg35HhMEhi2++eabEQnVGQxPoAUlBUoIIcwhrALV2NhIQkJCyLKEhATq6+sjEqozaA1NLSg5ByWEEGYQVoHq0aMHa9euDVn22WefhcylF+u0Bi1wQ1pQQghhCmENkrj99tt55plnKC0tJTs7m0OHDrF582YKCwsjnS9qtIZAF58SJwVKCCHMIKwCNWjQIJ5//nk+/fRTKisr6devH1OmTInJuZ3a4m88cQ4qLq5zgwghhADacbkNl8vV4hpO3yT+E+egpAUlhBDmEHaB+vzzz9m6dSu1tbUhy++7774OD9UZmlpQFocUKCGEMIOwBkksW7aMV155BV3XWbt2LUlJSXzxxRctRvbFsqYCJS0oIYQwh7BaUCtWrGDmzJn07NmTjz/+mClTpjB27Fj++Mc/Rjpf1OhuaUEJIYSZhNWCqq+vp2fPnkDgwleaptGvXz+2bt0a0XDRFOzii5cCJYQQZhBWCyo7O5t9+/aRk5MTnIU8KSmpzYsKxiLdIy0oIYQwk7AK1C233MLx48eBwHei5s+fj9vt5s4774xouGgyTnTxqdKCEkIIUzhjgdJ1HbvdzoABAwDo168fCxYsiHiwaGs6B2WVAiWEEKZwxnNQFouFZ599FlUNe0R6TGqaLFYKlBBCmENYgyQGDx7M9u3bI52lUxknLrdhTZDJYoUQwgzCahZlZmbyzDPPkJeXh9PpRFGU4GO33HJLxMJFlTtQoGyJ0oISQggzCKtAeb1ehg8fDkB1dXVEA3UWw9vUxffN7soUQohYEdbR+J577jnnHW3atInFixej6zoTJ05sdV6/LVu2sGTJEvx+P8nJyTz11FPnvN9wKScKlC1BWlBCCGEGYRWow4cPt/lYVlbWGZ+v6zrFxcXMnDkTp9NJYWEheXl5IdeTqq+v57XXXuOJJ57A5XJRU1MTTrSO4/WhYcXmsABGdPcthBCihbAK1AMPPNDmY0uXLj3j88vKysjOzg4Ws9GjR7N+/fqQAvXpp58yYsSI4CU8UlNTw4nWcXxevNix2aQ4CSGEGYRVoE4tQseOHWPZsmUMHjw4rJ1UV1fjdDqD951OJzt27AhZp7y8HE3TmD17No2NjVx77bWMGzeuxbZKSkooKSkBYM6cOed8TSpVVXG5XPzbMPBip1s3F1brOW0y4poyxxLJHB2SOfJiLS/EZmZox+U2mktLS2PKlCk8+OCDjB079ozrG0bLVknzkYAAfr+f3bt387Of/Qyv18vMmTPp378/3bp1C1kvPz+f/Pz84P3KysqzeQlBLpeLyspKtIYGvNhpOHpu24uGpsyxRDJHh2SOvFjLC+bPfOpxvslZD1k7ePAgHo8nrHWdTidVVVXB+1VVVaSnp7dYJzk5GYfDgcPhYPDgwezdu7fN4B1N8fnwYUc586pCCCGiIKwCNWvWrJAWj8fjYd++fdx4441h7aRv376Ul5dTUVFBRkYGpaWlLc5r5eXl8d///d/4/X40TaOsrIzrrruuHS/l3CiaD59iR76mK4QQ5hBWgZowYULIfYfDwQUXXEDXrl3D2onVamXq1KkUFRWh6zrjx48PzooOUFBQQI8ePRg2bBiPPPIIFouFCRMmBC/xEQ0WzSsFSgghTCSsAnXVVVed845yc3PJzc0NWVZQUBByf9KkSUyaNOmc93U2rD4PXksciZ2ydyGEEKcKay6+uXPn8vXXX4cs+/rrr3n++ecjEqozWDQfmkW+pCuEEGYRVoHaunUrAwcODFk2YMAAtmzZEpFQncHq96FZpINPCCHMIqwCZbPZcLvdIcvcbjdWs39hqB2sfq8UKCGEMJGwCtQll1zCK6+8QkNDAwANDQ0UFxczbNiwSGaLKqvfi1+6+IQQwjTCGiTxX//1XyxYsICpU6eSlJREXV0dw4YN4/777490vqix+n34VWlBCSGEWYRVoJKSkigsLOTYsWNUVlbicrlIS0uLcLTosupe/FYpUEIIYRZhFagvvviCzMxMunXrFixMBw8epLKykosvvjiS+aJG1b34VeniE0IIswjrHFRxcTHx8fEhyxwOB8XFxREJ1RlU3YsuLSghhDCNsApUTU1Ni7nz0tPTOXbsWCQydQqb4ZMWlBBCmEhYBSorK4uvvvoqZNmWLVvo0qVLREJ1BlX3otviOjuGEEKIE8I6B3XTTTcxd+5cJkyYQFZWFocPH2bFihUdcil4s7AbHnRpQQkhhGmE1YIaPnw4M2fOxO12s3HjRtxuN0888QTDhw+PdL6osRleDBlmLoQQphH29aD69etHv379gvf37dvH//7v/3LHHXdEJFhUGQZ2vBg2aUEJIYRZtOuChbW1tXz66aesWrWKPXv2fHNmkvD7sWBg2KUFJYQQZnHGAqVpGhs2bGDlypVs2rQJp9PJ0aNH+eUvf0mfPn2ikTHiFJ8PQFpQQghhIqctUMXFxZSWlmK1Whk5ciSzZ89mwIAB3HXXXTidzmhljDjD4w3csEuBEkIIszhtgVq+fDlJSUncdNNNjBkzhoSEhGjliiqtIdCCQlpQQghhGqctUAsWLGDVqlW8++67LFmyhEsvvZSxY8diGEa08kWFv/FEgYqTc1BCCGEWpx1m3qVLF2688UYWLFjAzJkzSUpKYtGiRdTW1vKHP/yB/fv3RytnRHnrAgVKkS4+IYQwjbC+BwUwePBgpk+fziuvvML9999PVVUVjz76aCSzRU1TC0qJkwIlhBBm0a5h5gB2u52xY8cyduxYqqurw37epk2bWLx4MbquM3HiRCZPnhzy+JYtW3j22WeD0yeNGDGCG2+8sb3xzoq/Qbr4hBDCbNpdoJrLyMgIaz1d1ykuLmbmzJk4nU4KCwvJy8ujR48eIesNHjyYGTNmnEuks+JvDIziU6RACSGEaYTdxXcuysrKyM7OJisrC1VVGT16NOvXr4/GrsOiuzUAlLhzqtdCCCE6UFSOyNXV1SHfm3I6nezYsaPFetu3b+fRRx8lPT2d73//++Tk5LRYp6SkhJKSEgDmzJmDy+U6p2yqqhJvDcxinuxMO+ftRYOqqjGRsznJHB2SOfJiLS/EZmaIUoFqbVi6oigh93v37s3LL7+Mw+Fg48aNPPfcc7z44ostnpefn09+fn7wfmVl5Tllc7lc1BwJnEtz675z3l40uFyumMjZnGSODskcebGWF8yfuVu3bq0uD6tA1dXV8e6777J3717cbnfIY0899dQZn+90Oqmqqgrer6qqanEBxOZfAs7NzaW4uJja2lpSUlLCiXhOmrr4LA4ZxSeEEGYRVoGaP38+mqYxatQo7GcxoWrfvn0pLy+noqKCjIwMSktLeeCBB0LWOXbsGKmpqSiKQllZGbquk5yc3O59nQ3dHRgkYY2XAiWEEGYRVoHavn07r732GraznArIarUydepUioqK0HWd8ePHk5OTw/LlywEoKChg7dq1LF++HKvVit1u56GHHmrRDRgphicwzFxaUEIIYR5hFaiePXtSVVVFdnb2We8oNzeX3NzckGUFBQXB29dccw3XXHPNWW//XOjuQIGyxssoPiGEMIuwjsgXXXQRv/zlL7nqqqtIS0sLeWzChAmRyBVVTS0oNUG+ByWEEGYRVoHatm0bTqeTzZs3t3jsm1CgOHG5DWuCdPEJIYRZhFWgnnzyyUjn6Fzepi4+KVBCCGEWYZ90qaurY8OGDVRXV5ORkcFll11GUlJSJLNFTdMFC22JUqCEEMIswprqaPv27dx///189NFH7N27l5KSEu6//362b98e6XzRcaIFZUuQQRJCCGEWYR2RlyxZwp133smYMWOCy0pLS1m8eDHPPPNMxMJFi+LzoWHFHm8FvlkXYxRCiFgVVguqvLycUaNGhSwbOXIkhw4dikioqPN68WLHZpPiJIQQZhFWgcrOzqa0tDRk2Zo1a8jKyopIqGhTfD682LFaOzuJEEKIJmF18U2ZMoU5c+bwwQcf4HK5OHLkCOXl5Z1y7aZIUHyBFlSUJq4QQggRhrAK1MCBA1mwYAEbN27k6NGjXHbZZeTm5n5jRvEpPh8+xY7UJyGEMI+wh60lJSVx5ZVXRjJLp7FogQIl80gIIYR5tFmgioqKeOKJJwCYNWtWmxO3hnO5DbOzah58SpwUKCGEMJE2C9S4ceOCt78R0xmdhkXz4bPIl3SFEMJM2ixQY8eODd7u3r07/fv3b7FOWVlZZFJFmUXzolmk/SSEEGYS1jDzX/ziF60uLyoq6tAwncXq90mBEkIIkzntIAld1wEwDCP40+Tw4cNYvyFfHLL4vXilQAkhhKmctkB973vfC96+9dZbQx6zWCz853/+Z2RSRZnq99JgTensGEIIIZo5bYF66aWXMAyD2bNnh4zWUxSFlJQU7PZvRqvDqvvwf0NeixBCfFOctkBlZmYC8PLLL0clTGdR/V78VhnFJ4QQZhL2F3U///xztm7dSm1tbcjy++67L6znb9q0icWLF6PrOhMnTmTy5MmtrldWVsYTTzzBww8/zMiRI8ONd05U3YtulRaUEEKYSVij+JYtW8Yrr7yCruusXbuWpKQkvvjiCxISEsLaia7rFBcX8/jjjzNv3jxWr17N/v37W13vjTfeYNiwYe16EedKNXz4VSlQQghhJmEVqBUrVjBz5kymTJmCqqpMmTKFxx57jCNHjoS1k7KyMrKzs8nKykJVVUaPHs369etbrPfBBx8wYsQIUlKiO2DBpnvQVeniE0IIMwmri6++vp6ePXsGnqCqaJpGv3792Lp1a1g7qa6uxul0Bu87nU527NjRYp3PPvuMJ598kt/85jdtbqukpISSkhIA5syZg8vlCitDW1RVxWZ4UeISznlb0aKqasxkbSKZo0MyR16s5YXYzAxhFqjs7Gz27dtHTk4OOTk5LF++nKSkpLBnM2/+/akmp87tt2TJEm6//XYsltM36vLz88nPzw/er6ysDCtDW1wuFzbDi2axnPO2osXlcsVM1iaSOTokc+TFWl4wf+Zu3bq1ujysAnXLLbdw/PhxAG677TZefPFF3G43d955Z1g7dzqdVFVVBe9XVVWRnp4ess7OnTuZP38+ALW1tfzzn//EYrFw+eWXh7WPc2EzvOhyDkoIIUwlrAKVm5sbvN2/f38WLFjQrp307duX8vJyKioqyMjIoLS0lAceeCBknYULF4bcvuyyy6JSnDAM7HgxbHIOSgghzKTNAnX48OGwNhDOZd+tVitTp06lqKgIXdcZP358sKsQoKCgIMy4EeD3Y8HAsEuBEkIIM2mzQJ3awmnL0qVLw1ovNzc3pCUGbReme++9N6xtdgTD4w3csEkXnxBCmEmbBap54VmxYgWbN2/mpptuIjMzkyNHjvD2228zdOjQqISMJK3BSxyAtKCEEMJUwvoe1NKlS5k+fTpdu3ZFVVW6du3KXXfdxZtvvhnpfBHnqz/RgpICJYQQphJWgTIMg4qKipBlR44cCV6OI5adLFDSxSeEEGYS1ii+6667jp///OdcddVVwfH0K1eu5Lrrrot0vojz1XkCN6QFJYQQphJWgZo0aRI9e/ZkzZo17Nmzh7S0NO6+++6oz5kXCU0tKMUhBUoIIcwk7NnMhw0b9o0oSKfSGnwAKHHSxSeEEGbSZoH605/+xA033ACcfij5Lbfc0vGposjfeKIFFSctKCGEMJM2C9SpUxN9U/kbAy0oi3TxCSGEqbRZoH70ox8Fb99zzz1RCdMZtAY5ByWEEGYUlamOzKypi88qBUoIIUwlalMdmVWwQMXLIAkhhDCTsKY6+ibT3YECZXGEPaBRCCFEFIQ1k8Q3WVOBssZLF58QQphJWM0Gv9/P3/72N7Zu3Rq8cGGTp556KiLBokVvlAIlhBBmFFYL6ne/+x0lJSUMGTKEXbt2MWLECGpqarjwwgsjnS/imi63oSZIgRJCCDMJq0CtW7eOxx9/nGuvvRar1cq1117Lo48+ypYtWyKdL+KCBSpRBkkIIYSZhFWgvF4vTqcTALvdjsfjoXv37uzZsyeS2aKiqUBJF58QQphLWOegunfvzs6dO+nXrx99+vRh2bJlxMfHk5GREel8EWd4AjNJqAkyik8IIczktC2opus9TZkyBYslsOoPfvADdu/ezYYNG7jrrrsinzDCmlpQNjkHJYQQpnLaZsP06dO58sorufLKK+nZsycAXbt25Wc/+1m7d7Rp0yYWL16MrutMnDiRyZMnhzy+fv16li5diqIoWK1WpkyZwqBBg9q9n/ZSfF40rNjjz/sR90IIYSqnLVA/+tGP+OSTTygsLKRHjx6MGzeOsWPHkpKS0q6d6LpOcXExM2fOxOl0UlhYSF5eHj169AiuM3ToUPLy8lAUhb179zJv3jxeeOGFs3pR7eL14sWOKj18QghhKqc9LA8fPpzhw4dTX19PaWkpq1at4o033uDiiy9m3Lhx5OXloYZxZC8rKyM7Ozs4b9/o0aNZv359SIFyOBzB2x6PB0VRzvY1tY/Phxc70dqdEEKI8ITVbkhMTOTqq6/m6quvpqKiglWrVvG73/2OV199leLi4jM+v7q6OjgKEMDpdLJjx44W63322Wf8/ve/p6amhsLCwla3VVJSQklJCQBz5szB5XKF8xLatMvnxafYz3k70aSqakzlBckcLZI58mItL8RmZmjHFXUBfD4fZWVl7Nixg5qaGgYOHBjW8wzDaLGstRbS5ZdfzuWXX87WrVtZunRpq+e68vPzyc/PD96vrKxsxytoSfF68GE/5+1Ek8vliqm8IJmjRTJHXqzlBfNn7tatW6vLwypQ27ZtY+XKlaxZs4bU1FSuuOIK7rzzTjIzM8PaudPpbHEBxPT09DbXHzJkCAsXLqS2trbd57vay+Lz4rPYkTF8QghhLqctUG+99RaffPIJdXV1jBw5khkzZpzVyLq+fftSXl5ORUUFGRkZlJaWtricx6FDh8jKykJRFHbt2oWmaSQnJ7d7X+1l0bx4lTgpUEIIYTKnLVA7duzg1ltvZfjw4djtZz8VkNVqZerUqRQVFaHrOuPHjycnJ4fly5cDUFBQwNq1a1m1ahVWqxW73c7DDz8clYESiuZFs0h5EkIIs1GM1k4QxZCDBw+e2/Nz7ySupgLnznc7KFHkmb0/uTWSOTokc+TFWl4wf+a2zkGd999OtWpeNItMFCuEEGYjBcrvwy8FSgghTEcKlN+LZpVzUEIIYTZSoHQvfqu0oIQQwmzO+wKl+r34pQUlhBCmIwVK96Cr0oISQgizkQKle9Gli08IIUznvC9QNsOLX1pQQghhOlKgDC+GTc5BCSGE2UiBMrxyDkoIIUxICpS0oIQQwpTO7wJlGNjxYtilQAkhhNmc3wXK78eCgWGTLj4hhDCb87pA+Rt9gRvSxSeEEKZzXhcoreFEgTqHa10JIYSIjLAu+f5N5avzBm7IOSghxFkwDAO3242u61G5wOrZOnz4MB6Pp1MzGIaBxWLB4XCE/V6d1wUq2MUnBUoIcRbcbjc2mw1VNfehVFVVrFZrZ8dA0zTcbjfx8fFhrS9dfIASJ118Qoj203Xd9MXJTFRVRdf1sNeXAgUQJy0oIUT7mblbz6za855FrfRv2rSJxYsXo+s6EydOZPLkySGPf/LJJ/zlL38BwOFwcOedd9KrV6+IZvI3agBYHFKghBDCbKLSgtJ1neLiYh5//HHmzZvH6tWr2b9/f8g6Xbp0Yfbs2cydO5fvfve7vPLKKxHP5W8MDJKwOKSLTwgRm3Jycrj66quZMGECP/jBD6ipqTnt+s8//zyLFi0KWfbQQw/x/vvvhyzr379/h2dtr6gUqLKyMrKzs8nKykJVVUaPHs369etD1hk4cCBJSUlA4I2pqqqKeC7d3XQOSlpQQojY5HA4+Oijj/jHP/5BWloaS5Ys6exIHSYqXXzV1dU4nc7gfafTyY4dO9pc/x//+AeXXnppxHPpbuniE0J0jFmzUti6tWOPJUOG+Pj5z2vDXv+yyy7j66+/BmDPnj088cQTVFVVkZCQwLPPPku/fv06NF+kRaVAGYbRYllbJ8q++uorVqxYwc9//vNWHy8pKaGkpASAOXPm4HK5zjrXQSXwx5TiSj+n7USbqqoxlRckc7RI5shrnvfw4cPBUXwWi6XDB01YLJawRgmqqorf76e0tJTbbrsNVVV57LHHeO655+jTpw8bNmzg8ccf509/+hMWi6XFdi0WC1artcW+IjFCMS4uLuzfd1QKlNPpDOmyq6qqIj09vcV6e/fu5be//S2FhYUkJye3uq38/Hzy8/OD9ysrK886V131MQAa/Z5z2k60uVyumMoLkjlaJHPkNc/r8XiC3y+aPftYRPanaad/3O12M378ePbv38/QoUMZM2YMNTU1fP7550ybNg0INAg8Hg+apqHrOrquozXbsGEY+P3+kGWKooTc7ygeT8vjbbdu3VpdNyrnoPr27Ut5eTkVFRVomkZpaSl5eXkh61RWVjJ37lzuu+++NsN2NMMdGCRhjZcuPiFEbGo6B7Vu3Tp8Ph9LlixB13VSUlL46KOPguenVq5c2eY20tPTQwZXHD16lIyMjGjEP62oFCir1crUqVMpKiri4YcfZtSoUeTk5LB8+XKWL18OwNtvv01dXR2vvfYajz76KDNmzIh4Lt0TGCQhBUoIEetSUlJ4+umnWbRoEQ6Hg5ycHN577z0g0ELasmVLm88dNWoU7777Ll5v4EP7W2+9xejRo6OS+3Si9j2o3NxccnNzQ5YVFBQEb0+fPp3p06dHKw4AxolRfBaHfBNcCBH7LrroIoYMGcJf/vIXXnrpJQoLC5k/fz5+v59JkyZx4YUXAjB//nxeffXV4PM2bNjA5s2b+fa3v43FYqFXr17MmTOns15GkGK0NoIhhhw8ePCsn7v1oaXkL/sJG9/bRHZuZgemiqxY67MHyRwtkjnymudtaGggISGhkxOdmaqqETmfdDZae8869RyUWRknuvjUBGlBCSGE2ZzXBQpvU4GSc1BCCGE2UqAAW6IUKCGEMBspUIAaL118QghhNud3gfJ50bBic3T+hbyEEEKEOq8LlOL14cWO5bx+F4QQwpzO60Oz4gsUKCGEiGXz589n/Pjx5Ofnc/XVV7Nx40YeeeQRtm/fHtH9fv/732/18h6tXdLjbJzXJ18smgefIgVKCBG7Pv/8c0pKSvjwww+Ji4ujuroar9fL3LlzI77v119/PaLbP68LlOLzSYESQnSIlFmzsG3d2qHb9A0ZQm0bV3ZoUlFRQUZGBnFxcQDBOfRuvPFGfvazn3HJJZfwxhtvsGDBArKzs+nduzd2u52ioiIeeughHA4HZWVlHDhwgF//+tcsW7aMDRs2cOmll/LCCy8A8M4777BgwQIMw2DixIk88cQTAIwYMYIPPviAjIwM5s+fz9tvv023bt1wOp1cfPHF5/z6z+suPovmRZMCJYSIYePGjePgwYOMHTuWwsJC1qxZE/L4oUOHmDdvHu+99x5/+MMfKCsrC3m8pqaGZcuWMXv2bKZMmcKPfvQjVqxYwbZt2/jqq684dOgQRUVFvPXWWyxfvpxNmzbx4Ycfhmzjyy+/5N1332X58uW89tprfPHFFx3y2s7rFpRV8+G1xJ3fb4IQokOcqaUTKYmJiXz44YesW7eO0tJS7r77bgoLC4OPb9q0iVGjRgUvcXT99deza9eu4ONXX301iqIwaNAgXC4XgwcPBmDAgAHs37+fAwcOMGrUqOBFZ2+44QbWrl3LNddcE9zGunXruOaaa4iPjw9usyOc18dmi+ZFs0gLSggR26xWK6NHj2b06NEMGjSIZcuWBR8703SrdnvgGGixWILdhE33NU0L+6KFHX2xRjjfu/j8PvxSoIQQMaysrCykRbRlyxZ69OgRvD9s2DDWrFnDsWPH0DSNv/71r+3a/qWXXsratWuprq7G7/fzzjvvMGrUqJB1Ro4cyYcffkhjYyN1dXV89NFH5/aiTjivW1Br0q9BcXi5qbODCCHEWWpoaGDmzJnU1taiqiq9evXi2Wef5a677gKga9euPPDAA1x//fVkZ2fTv3//Nq9Y3pqsrCwKCwu56aabMAyDCRMm8K1vfStknaFDh/Kd73yHgoICevTowYgRIzrktZ3Xl9s4fNhCcnIGCQmxM9U/xN7lCUAyR4tkjrxYvNyGx+MhLi4OTdOYNm0at956K9/+9rc7JUt7LrdxXregsrJ0XC6Iof8bQgjRbs899xwrV67E4/Ewbty4kAEOZnZeFyghhDgfzJ492zQXLGyP83qQhBBCnIsYP0PSKdrznkmBEkKIs9Q0FFuER9M0LO2YnTtqXXybNm1i8eLF6LrOxIkTmTx5csjjBw4c4OWXX2b37t3ceuutTJo0KVrRhBDirDgcDtxuNx6PJyLfA+oocXFxeDyeTs1gGAYWiwWHwxH2c6JSoHRdp7i4mJkzZ+J0OiksLCQvLy9krH5SUhI//OEPWb9+fTQiCSHEOVMUJTh7gpnF2kjJJlHp4isrKyM7O5usrCxUVWX06NEtClFqair9+vXDapWLBwohhIhSC6q6ujo4jxOA0+lkx44dZ7WtkpISSkpKAJgzZw4ul+ucsqmqes7biDbJHB2SOTpiLXOs5YXYzAxRKlCtjdo42/7a/Px88vPzg/fPtdkai01fyRwdkjk6Yi1zrOUF82fu1C/qOp1OqqqqgverqqqCM+ueq7ZeWLS3EW2SOTokc3TEWuZYywuxmTkq56D69u1LeXk5FRUVaJpGaWkpeXl50dj1Gc2YMaOzI7SbZI4OyRwdsZY51vJCbGaGKLWgrFYrU6dOpaioCF3XGT9+PDk5OSxfvhyAgoICjh07xowZM2hsbERRFP7617/y61//OibmuRJCCNHxovY9qNzcXHJzc0OWFRQUBG+npaWxaNGiaMURQghhcuf9TBLNB1zECskcHZI5OmItc6zlhdjMDN+Ay20IIYT4ZjrvW1BCCCHMSQqUEEIIUzpvrwd1pslrzeLll19m48aNpKam8vzzzwNQV1fHvHnzOHLkCJmZmTz88MMkJSV1ctKAyspKFi5cyLFjx1AUhfz8fK699lpTZ/Z6vTz55JNomobf72fkyJHcfPPNps7cRNd1ZsyYQUZGBjNmzDB95nvvvReHw4HFYsFqtTJnzhzTZ66vr2fRokXs27cPRVG4++676datm2kzHzx4kHnz5gXvV1RUcPPNNzNu3DjTZm6TcR7y+/3GfffdZxw6dMjw+XzGI488Yuzbt6+zY7Vqy5Ytxs6dO42f/OQnwWWvv/668ec//9kwDMP485//bLz++uudlK6l6upqY+fOnYZhGEZDQ4PxwAMPGPv27TN1Zl3XjcbGRsMwDMPn8xmFhYXGv/71L1NnbvLee+8ZL7zwgvHMM88YhmHuvw3DMIx77rnHqKmpCVlm9swLFiwwSkpKDMMI/H3U1dWZPnMTv99v3HnnnUZFRUXMZG7uvOziC2fyWrMYMmRIi08569evZ9y4cQCMGzfOVNnT09Pp06cPAPHx8XTv3p3q6mpTZ1YUJXgJAL/fj9/vR1EUU2eGwIwsGzduZOLEicFlZs/cGjNnbmho4Ouvv2bChAlAYE67xMREU2dubvPmzWRnZ5OZmRkzmZs7L7v4OnLy2s5QU1MTnCoqPT2d2traTk7UuoqKCnbv3k2/fv1Mn1nXdR577DEOHTrEt771Lfr372/6zEuWLOGOO+6gsbExuMzsmQGKiooAuPrqq8nPzzd15oqKClJSUnj55ZfZu3cvffr0YcqUKabO3Nzq1asZM2YMEBt/G6c6LwuU0YGT14rWud1unn/+eaZMmRITs4FYLBaee+456uvrmTt3Lv/+9787O9JpbdiwgdTUVPr06cOWLVs6O07Ynn76aTIyMqipqeEXv/iF6eeH8/v97N69m6lTp9K/f38WL17MO++809mxwqJpGhs2bOC2227r7Chn7bwsUJGcvDYaUlNTOXr0KOnp6Rw9epSUlJTOjhRC0zSef/55rrjiCkaMGAGYP3OTxMREhgwZwqZNm0yd+V//+heff/45//znP/F6vTQ2NvLiiy+aOjNARkYGEPh7GD58OGVlZabO7HQ6cTqd9O/fH4CRI0fyzjvvmDpzk3/+85/07t2btLQ0IHb+DzZ3Xp6DMvPkteHIy8tj5cqVAKxcuZLhw4d3cqKTDMNg0aJFdO/eneuvvz643MyZa2trqa+vBwIj+jZv3kz37t1Nnfm2225j0aJFLFy4kIceeoiLLrqIBx54wNSZ3W53sDvS7Xbz5Zdf0rNnT1NnTktLw+l0cvDgQSBwTqdHjx6mztykefcemPv/YFvO25kkNm7cyO9+97vg5LU33HBDZ0dq1QsvvMDWrVs5fvw4qamp3HzzzQwfPpx58+ZRWVmJy+XiJz/5iWmGi27bto1Zs2bRs2fPYLfp9773Pfr372/azHv37mXhwoXouo5hGIwaNYobb7yR48ePmzZzc1u2bOG9995jxowZps58+PBh5s6dCwS6zsaOHcsNN9xg6swAe/bsYdGiRWiaRpcuXbjnnnswDMPUmT0eD3fffTcvvfRSsIvd7O9za87bAiWEEMLczssuPiGEEOYnBUoIIYQpSYESQghhSlKghBBCmJIUKCGEEKYkBUqIGHbzzTdz6NChzo4hRESclzNJCBEp9957L8eOHcNiOfnZ76qrrmLatGmdmEqI2CQFSogO9thjj3HxxRd3dgwhYp4UKCGi4OOPP+bvf/87vXv3ZuXKlaSnpzNt2jSGDh0KBGbYf/XVV9m2bRtJSUn8x3/8B/n5+UBgpvV33nmHFStWUFNTQ9euXXn00UdxuVwAfPnll/zyl7/k+PHjjBkzhmnTpqEoCocOHeI3v/kNe/bsQVVVLrroIh5++OFOew+EaC8pUEJEyY4dOxgxYgTFxcV89tlnzJ07l4ULF5KUlMT8+fPJycnht7/9LQcPHuTpp58mKyuLoUOH8v7777N69WoKCwvp2rUre/fuJS4uLrjdjRs38swzz9DY2Mhjjz1GXl4ew4YN48033+SSSy4JXi14165dnfjqhWg/KVBCdLDnnnsOq9UavH/HHXegqiqpqalcd911KIrC6NGjee+999i4cSNDhgxh27ZtzJgxA7vdTq9evZg4cSKrVq1i6NCh/P3vf+eOO+4IXpqiV69eIfubPHkyiYmJJCYmcuGFF7Jnzx6GDRuGqqocOXKEo0eP4nQ6GTRoUDTfBiHOmRQoITrYo48+2uIc1Mcff0xGRkbIdccyMzOprq7m6NGjJCUlER8fH3zM5XKxc+dOIHA5mKysrDb313Q5BYC4uDjcbjcQKIxvvvkmjz/+OImJiVx//fXBK8MKEQukQAkRJdXV1RiGESxSlZWV5OXlkZ6eTl1dHY2NjcEiVVlZGbx2ktPp5PDhw/Ts2bNd+0tLS2P69OlAYJb5p59+miFDhpCdnd2Br0qIyJHvQQkRJTU1NXzwwQdomsaaNWs4cOAAl156KS6Xi4EDB/L73/8er9fL3r17WbFiBVdccQUAEydOZOnSpZSXl2MYBnv37uX48eNn3N+aNWuCF+ZMTEwECBn+LoTZSQtKiA72q1/9KqQQXHzxxQwfPpz+/ftTXl7OtGnTSEtL4yc/+QnJyckAPPjgg7z66qv8+Mc/JikpiZtuuinYTXj99dfj8/n4xS9+wfHjx+nevTuPPPLIGXPs3LmTJUuW0NDQQFpaGj/84Q/p0qVLZF60EBEg14MSIgqahpk//fTTnR1FiJgh7X0hhBCmJAVKCCGEKUkXnxBCCFOSFpQQQghTkgIlhBDClKRACSGEMCUpUEIIIUxJCpQQQghT+v+r4Ddd8tWP9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print_result(nn_long_relu, X_train, y_train, X_test, y_test, title=\"ReLU\",color=\"blue\")\n",
    "print_result(nn_long_sigmoid, X_train, y_train, X_test, y_test, title=\"Sigmoid\",color=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sklearn Implementation\n",
    "Let's see if our method stacks up to the implementations from `scikit-learn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0.2\n",
      "CPU times: user 3min 20s, sys: 18.8 s, total: 3min 39s\n",
      "Wall time: 55.2 s\n",
      "Validation Acc: 0.8725\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ericlarson/opt/anaconda3/envs/mlenv2022/lib/python3.7/site-packages/sklearn/neural_network/_multilayer_perceptron.py:696: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
      "  ConvergenceWarning,\n"
     ]
    }
   ],
   "source": [
    "from sklearn import __version__ as sklearn_version\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "print(sklearn_version)\n",
    "# these values have been hand tuned\n",
    "# sigmoid with SGD, batching, nesterov's momentum, \n",
    "# L2, adaptive learning rate\n",
    "clf = MLPClassifier(hidden_layer_sizes=(50,), \n",
    "                    activation='logistic', # compare to sigmoid\n",
    "                    solver='sgd', \n",
    "                    alpha=1e-4, # L2 penalty\n",
    "                    batch_size=128, # min of 128, num_samples\n",
    "                    learning_rate='adaptive', # decrease rate if loss goes up\n",
    "                    learning_rate_init=0.1, # only SGD\n",
    "                    power_t=0.5,    # only SGD with inverse scaling\n",
    "                    max_iter=100, \n",
    "                    shuffle=True, \n",
    "                    random_state=1, \n",
    "                    tol=1e-9, # for stopping\n",
    "                    verbose=False, \n",
    "                    warm_start=False, \n",
    "                    momentum=0.9, # only SGD\n",
    "                    nesterovs_momentum=True, # only SGD\n",
    "                    early_stopping=False, \n",
    "                    validation_fraction=0.1, # only if early_stop is true\n",
    "                    beta_1=0.9, # adam decay rate of moment\n",
    "                    beta_2=0.999, # adam decay rate of moment\n",
    "                    epsilon=1e-08) # adam numerical stabilizer\n",
    "\n",
    "%time clf.fit(X_train,y_train)\n",
    "yhat = clf.predict(X_test)\n",
    "print('Validation Acc:',accuracy_score(yhat,y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 12s, sys: 18 s, total: 3min 30s\n",
      "Wall time: 53.1 s\n",
      "Validation Acc: 0.8678\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ericlarson/opt/anaconda3/envs/mlenv2022/lib/python3.7/site-packages/sklearn/neural_network/_multilayer_perceptron.py:696: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
      "  ConvergenceWarning,\n"
     ]
    }
   ],
   "source": [
    "# but with this implementation we can also easily change Neurons\n",
    "# ReLU with SGD, batching, simple momentum, \n",
    "# L2, adaptive learning rate\n",
    "clf = MLPClassifier(hidden_layer_sizes=(50, ), \n",
    "                    activation='relu', # type of non-linearity, every layer except output layer\n",
    "                    solver='sgd', \n",
    "                    alpha=1e-4, # L2 penalty\n",
    "                    batch_size= 128, # min of 128, num_samples\n",
    "                    learning_rate='adaptive', # adapt learning? \n",
    "                    learning_rate_init=0.1, # only SGD\n",
    "                    power_t=0.0,    # only SGD with inverse scaling of learning rate\n",
    "                    max_iter=100, # stopping criteria\n",
    "                    shuffle=True, \n",
    "                    random_state=1, \n",
    "                    tol=0, # for stopping\n",
    "                    verbose=False, \n",
    "                    warm_start=False, \n",
    "                    momentum=0.9, # only SGD\n",
    "                    nesterovs_momentum=False, # only SGD\n",
    "                    early_stopping=False, \n",
    "                    validation_fraction=0.0, # only if early_stop is true\n",
    "                    beta_1=0.9, # adam decay rate of moment\n",
    "                    beta_2=0.999, # adam decay rate of moment\n",
    "                    epsilon=1e-08) # adam numerical stabilizer\n",
    "\n",
    "%time clf.fit(X_train,y_train)\n",
    "yhat = clf.predict(X_test)\n",
    "print('Validation Acc:',accuracy_score(yhat,y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ericlarson/opt/anaconda3/envs/mlenv2022/lib/python3.7/site-packages/sklearn/neural_network/_multilayer_perceptron.py:696: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n",
      "  ConvergenceWarning,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 17min 11s, sys: 47.8 s, total: 17min 58s\n",
      "Wall time: 4min 31s\n",
      "Validation Acc: 0.858\n"
     ]
    }
   ],
   "source": [
    "# but what if we also use AdaM optimization?\n",
    "# ReLU with AdaM, batching, simple momentum, \n",
    "# L2, adaptive learning rate\n",
    "clf = MLPClassifier(hidden_layer_sizes=(50, ), \n",
    "                    activation='relu', # type of non-linearity, every layer except output layer\n",
    "                    solver='adam', \n",
    "                    alpha=1e-4, # L2 penalty\n",
    "                    batch_size= 128, # min of 200, num_samples\n",
    "                    learning_rate='adaptive', # adapt learning? \n",
    "                    learning_rate_init=0.01, # only SGD\n",
    "                    power_t=1,    # only SGD with inverse scaling of learning rate\n",
    "                    max_iter=100, # stopping criteria\n",
    "                    shuffle=True, \n",
    "                    random_state=1, \n",
    "                    tol=0, # for stopping\n",
    "                    verbose=False, \n",
    "                    warm_start=False, \n",
    "                    momentum=0.9, # only SGD\n",
    "                    nesterovs_momentum=False, # only SGD\n",
    "                    early_stopping=False, \n",
    "                    validation_fraction=0.0, # only if early_stop is true\n",
    "                    beta_1=0.9, # adam decay rate of moment\n",
    "                    beta_2=0.999, # adam decay rate of moment\n",
    "                    epsilon=1e-08) # adam numerical stabilizer\n",
    "\n",
    "%time clf.fit(X_train,y_train)\n",
    "yhat = clf.predict(X_test)\n",
    "print('Validation Acc:',accuracy_score(yhat,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## In this notebook, you learned:\n",
    "- Simple optimization methods and tricks for Neural Networks\n",
    "- Cross Entropy Loss\n",
    "- Glorot and He Initialization\n",
    "- Rectified Linear Activation Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}