guide/14-deep-learning/ml-and-dl-on-tabular-data.ipynb

{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# Machine learning and deep learning on tabular data"]}, {"cell_type": "markdown", "metadata": {"toc": true}, "source": ["<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n", "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction-\" data-toc-modified-id=\"Introduction--1\">Introduction <a class=\"anchor\" id=\"1\"></a></a></span></li><li><span><a href=\"#Prepare-tabular-data-\" data-toc-modified-id=\"Prepare-tabular-data--2\">Prepare tabular data <a class=\"anchor\" id=\"2\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#Data-Preprocessing-for-FullyConnectedNetwork\" data-toc-modified-id=\"Data-Preprocessing-for-FullyConnectedNetwork-2.1\">Data Preprocessing for FullyConnectedNetwork<a class=\"anchor\" id=\"3\"></a></a></span></li><li><span><a href=\"#Data-Preprocessing-for-MLModel\" data-toc-modified-id=\"Data-Preprocessing-for-MLModel-2.2\">Data Preprocessing for MLModel<a class=\"anchor\" id=\"4\"></a></a></span></li></ul></li><li><span><a href=\"#Load-model-architecture\" data-toc-modified-id=\"Load-model-architecture-3\">Load model architecture<a class=\"anchor\" id=\"5\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#FullyConnectedNetwork-\" data-toc-modified-id=\"FullyConnectedNetwork--3.1\">FullyConnectedNetwork <a class=\"anchor\" id=\"6\"></a></a></span></li><li><span><a href=\"#MLModel\" data-toc-modified-id=\"MLModel-3.2\">MLModel<a class=\"anchor\" id=\"7\"></a></a></span></li></ul></li><li><span><a href=\"#Model-Training\" data-toc-modified-id=\"Model-Training-4\">Model Training<a class=\"anchor\" id=\"8\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#FullyConnectedNetwork-Model-Training-\" data-toc-modified-id=\"FullyConnectedNetwork-Model-Training--4.1\">FullyConnectedNetwork Model Training <a class=\"anchor\" id=\"9\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#Learning-Rate-Search\" data-toc-modified-id=\"Learning-Rate-Search-4.1.1\">Learning Rate Search</a></span></li></ul></li><li><span><a href=\"#ML-Model-Training\" data-toc-modified-id=\"ML-Model-Training-4.2\">ML Model Training<a class=\"anchor\" id=\"10\"></a></a></span></li></ul></li><li><span><a href=\"#Model-Validation-metrics,-Saving-&amp;-Loading\" data-toc-modified-id=\"Model-Validation-metrics,-Saving-&amp;-Loading-5\">Model Validation metrics, Saving &amp; Loading<a class=\"anchor\" id=\"11\"></a></a></span></li><li><span><a href=\"#Model-Inferencing-\" data-toc-modified-id=\"Model-Inferencing--6\">Model Inferencing <a class=\"anchor\" id=\"12\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#Predicting-to-features\" data-toc-modified-id=\"Predicting-to-features-6.1\">Predicting to features</a></span></li><li><span><a href=\"#Predicting-to-rasters\" data-toc-modified-id=\"Predicting-to-rasters-6.2\">Predicting to rasters</a></span></li></ul></li><li><span><a href=\"#Summary-\" data-toc-modified-id=\"Summary--7\">Summary <a class=\"anchor\" id=\"13\"></a></a></span></li></ul></div>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Introduction <a class=\"anchor\" id=\"1\"></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The `arcgis.learn` module includes [FullyConnectedNetwork](https://developers.arcgis.com/python/api-reference/arcgis.learn.toc.html#fullyconnectednetwork) and [MLModel](https://developers.arcgis.com/python/api-reference/arcgis.learn.toc.html#mlmodel) classes to train machine learning and deep learning models on tabular data. Tabular data can be any vector data in the form of a [feature layer](https://developers.arcgis.com/python/guide/working-with-feature-layers-and-features/) or [spatially enabled dataframe](https://developers.arcgis.com/python/guide/introduction-to-the-spatially-enabled-dataframe/). Explanatory variables can take the form of fields in the attribute table of the feature layer, raster datasets, or distance features used to calculate proximity values for use as additional variables. In addition to validation of model performance based on the training data, predictions can be made to produce either features or a prediction raster.  Predictions can be performed for both categorical variables and continuous variables.\n", "\n", "The `FullyConnectedNetwork` model is based on deep neural networks, whereas the `MLModel` uses machine learning algorithms to train models.\n", "\n", "The `MLModel` allows you to use any [regression](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) or [classification](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) model from scikit-learn, whereas `FullyConnectedNetwork` allows you to train a deep neural network with a specified number of layers."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Prepare tabular data <a class=\"anchor\" id=\"2\"></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Data preparation can be a time consuming process that typically involves splitting the data into training and validation sets, applying various data augmentation techniques, creating the necessary data structures for loading data into the model, memory management by using the appropriately sized mini-batches of data, and so on.\n", "\n", "You can use the `prepare_tabulardata()` method to create a `TabularDataObject` that can be fed into the `MLModel` and the `FullyConnectedNetwork` models. It allows you to use attribute data from feature layers and spatially enabled  `DataFrame`s. Additionally, you can use explanatory variables from rasters. The raster can be imagery layer which can be accessed through ArcGIS Enterprise or local raster files which can be accessed using the `Raster` class from the `arcgis.raster` module. This method performs normalization of the datasets, imputation, and splits it into training and validation datasets before feeding the datasets into the model.\n", "\n", "By default the `prepare_tabulardata()` method treats the fields in the training data as continuous variables. If you have categorical variables, pass that variable as a `(<filed_name>, True)` tuple as shown below:\n", "\n", "`X = [\"Field_1\", (\"Field_2\", True),.......\"Field_n\"]`\n", "\n", "Here **Field_2** is treated as categorical and the others as continuous variables. If variables are stored as multiple rasters, then pass them as a list of `Raster` objects as shown below:\n", "\n", "`explanatory_rasters = [RasterObject1,(RasterObject2, True),.......RasterObjectn]` \n", "\n", "Here, `RasterObject2` is treated as categorical and the others as continuous variables."]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Data Preprocessing for FullyConnectedNetwork<a class=\"anchor\" id=\"3\"></a>  "]}, {"cell_type": "markdown", "metadata": {}, "source": ["Once the explanatory variables and explanatory rasters are defined, we can prepare our TabularDataObject by using `prepare_tabulardata()` function. The following are some of the key arguments for this function:\n", "\n", "- `input_features`: It takes in data in the form of a feature layer or spatially enabled DataFrame. Feature layers can be of point or polygon geometry.\n", "\n", "- `variable_predict`: Field name from the `input_features` layer. It automatically deduces fields as continuous or categorical by using a simple logic. If the field type is float and has more than 20 unique values, the field is considered continuous. Otherwise, the field will be considered categorical.\n", "\n", "- `date_field`: Field name from the `input_features` layer. If specified, the field will be split into year, month, week, day, dayofweek, dayofyear,  is_month_end, is_month_start, is_quarter_end,  is_quarter_start, is_year_end, is_year_start,  hour, minute, second, elapsed.  If specified here, there is no need to specify as part of `explanatory_variables`.\n", "\n", "- `explanatory_variables` : List of the field names containing the explanatory variables from the `input_layer`, where tuples of size 2 have to be passed for categorical variables as discussed above. \n", "\n", "- `explanatory_rasters` : List of the field names containing the explanatory variables in multiple input rasters, where tuples of size 2 have to be passed for categorical variables as discussed above.\n", "\n", "- `distance_features`: Distance features are used to automatically create explanatory variables by calculating a distance from the provided features to the input_features. Distances will be calculated from each of the input explanatory distance features to the nearest Input training features. If the input `distance_features` are polygons or lines, the distance attributes are calculated as the distance between the closest segments of the pair of features. However, distances are calculated differently for polygons and lines. See how [proximity tools calculate distance](https://pro.arcgis.com/en/pro-app/tool-reference/analysis/how-near-analysis-works.htm) for details."]}, {"cell_type": "markdown", "metadata": {}, "source": ["Once the explanatory variables or the explanatory rasters list is ready, these can now be used as input for the `prepare_tabulardata()` function. The function takes the feature layer or a spatial DataFrame containing the dataset along with the y-vairable defined as the variable_predict and prepares it for fitting the `FullyConnectedNetwork` model as follows:"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["data = prepare_tabulardata(input_features,\n", "                            variable_predict, \n", "                            explanatory_variables=X,\n", "                            explanatory_rasters=explanatory_rasters)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Data Preprocessing for MLModel<a class=\"anchor\" id=\"4\"></a>  \n", "Similar to the `FullyConnectedNetwork`, here the first step consists of making a list of the explanatory variables. Continuous variables are listed as-is and categorical variables are listed as a `<variable_name>, True` tuple. Note that most of the components for the implementation of the `MLModel` framework are similar to that of the `FullyConnectedNetwork` other than a few exceptions, which are the `make_pipeline()`, `make_column_transformer` and the `MLModel()` methods. The `make_pipeline` and `make_column_transformer` are provided by sklearn library and these two are used to create the `preprocessors` parameter in the `prepare_tabulardata` method for `MLModel`."]}, {"cell_type": "markdown", "metadata": {}, "source": ["The `make_pipeline` function is used for applying some operations on the full dataset instead of individual columns. Thus instead of different scalers, if the whole dataset is to be scaled with a single scaling function then this can be framed as \n", "\n", "`numerical_transformer = make_pipeline(MinMaxScaler())` followed by using it as an input for the make_column_transformer.\n", " \n", "`preprocessors = make_column_transformer((numerical_transformer, X))` where X is the list of input columns. \n", "\n", "The `preprocessors` created from the above step would then be the input for the `prepare_tabulardata` method. The pipeline method could also be used for applying any number of steps like, step 1 -> step 2 -> step 3 and so on to the full dataset. For example, the below line will first scale a dataset using a scikit-learn scaler followed by applying a scikit-learn model to the scaled dataset:\n", "\n", "`make_pipeline(StandardScaler(), GaussianNB(priors=None))`"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The `make_column_transformer` function is used for applying column-wise transformation steps. With this the user can select any scaler function from scikit-learn to transform the fields using the selected scaler. Suppose we want to use scaler1 on column c1 & c2, and scaler2 on c3 & c4, then we can use it as follows:\n", "\n", "`preprocessors = make_column_transfomer((scaler1, [c1,c2]), (scaler2,[c3,c4]))`\n", "\n", "Thus, if there are both numerical and categorical fields in a dataset, they can be scaled using different scalers. Also, if the user wants to drop a column or to pass them through untransformed, then use strings `drop` and `passthrough` respectively instead of the scaler. It can also be used for applying multiple steps on each column like for column 1 - step 1 -> step 2 -> step 3 and then for column 2 - step 1 -> step 2 -> step 3."]}, {"cell_type": "markdown", "metadata": {}, "source": ["Once the explanatory variables or the explanatory rasters list is ready, along with the `preprocessors` transformation,  these can now be used as input for the `prepare_tabulardata()` method. The method takes the `feature layer` or a spatial enabled `DataFrame` containing the dataset and prepares it for fitting the `MLModel` model as follows:"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["from arcgis.learn import prepare_tabulardata\n", "data = prepare_tabulardata(input_features,\n", "                               variable_predict,\n", "                               explanatory_variables=X,\n", "                               preprocessors=preprocessors)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Here, most of the input parameters are similar to those described for `FullyConnectedNetwork`, other than the `preprocessors`. The `preprocessors`, as explained above, contains information about data transformations.\n", "\n", "Now at this stage the `data.show_batch()` can be used for visualizing the data frame that is being used for the model. "]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Load model architecture<a class=\"anchor\" id=\"5\"></a>  \n", "\n", "### FullyConnectedNetwork <a class=\"anchor\" id=\"6\"></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Once the data has been prepared by the `prepare_tabulardata` method it is ready to be passed to the ANN for training. The ANN is known here as the `FullyConnectedNetwork` and is imported from `arcgis.learn` followed by initialization of the neural network as follows:\n", "\n", "```python\n", "from arcgis.learn import FullyConnectedNetwork\n", "fcn = FullyConnectedNetwork(data)\n", "```\n", "\n", "According to the default setting the initialization stored in the variable named fcn, creates a deep neural network of two hidden layers with the first layer containing 500 nodes and a second layer having 100 nodes. The user can add their own configuration of layers by passing a list in the `layers` parameter, specifying the number of nodes for each layer as follows:"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["fcn = FullyConnectedNetwork(data, layers=[200, 300])"]}, {"cell_type": "markdown", "metadata": {}, "source": ["It will represent two hidden layers with nodes 300 and 200 respectively.\n", "\n", "`Emb_szs` \u2013  Usually categorical fields are transformed into numerical type by one-hot encoding or embedding. Here embedding is used where once the size of the embedding is mentioned in the `Emb_szs` parameter, it will transform the categories into numeric. This is used only for categorical variables."]}, {"cell_type": "markdown", "metadata": {}, "source": ["### MLModel<a class=\"anchor\" id=\"7\"></a> \n", "\n", "In the case of the machine learning model, the data prepared by the `prepare_tabulardata` method is ready to be passed to the `MLModel` method, along with the selected machine learning model for training. Here, for demonstration purpose, the `GradientBoostingRegressor` model from scikit-learn is used, which is passed into the `MLModel` function, along with its parameters.\n", "\n", "First, we import the `MLModel` framework from `arcgis.learn`. Then, we specify the model to be used from scikit-learn and define the necessary parameters as follows:\n", "\n", "```python\n", "from arcgis.learn import MLModel\n", "model = MLModel(data, 'sklearn.ensemble.GradientBoostingRegressor', n_estimators=100, random_state=43)\n", "```"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Model Training<a class=\"anchor\" id=\"8\"></a> \n", "\n", "### FullyConnectedNetwork Model Training <a class=\"anchor\" id=\"9\"></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["#### Learning Rate Search\n", "\n", "The initialized model will need a learning rate for training, and the `lr_find()` method can be used to search for an optimal learning rate to pass into the final model fitting as follows:"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["fcn.lr_find()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The automatic lr_finder will return a value that takes a conservative estimate of the learning rate. While the conservative estimate generated by lr_finder is reliable, an expert can interpret the graph more appropriately and find a better learning rate for the final model training. In the figure below, the suggested learning rate by the `lr_find()` method is around `0.0002754`."]}, {"cell_type": "markdown", "metadata": {}, "source": ["<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXgV5dn48e+dFZKQQCBhSyAIKIYdAiIuFbUKSsEFfMGqoFZaxarVLtrF1/r+bGvVWtcqiBuKuFesC1pRUVnDIjsSIEDYEpKQlez3748z6DGGLJDJWXJ/rutcznnmmZl7jofcZ+ZZRlQVY4wxprFCfB2AMcaYwGKJwxhjTJNY4jDGGNMkljiMMcY0iSUOY4wxTRLm6wBaQqdOnTQlJcXXYRhjTMBYtWrVIVVNqGtdq0gcKSkppKen+zoMY4wJGCKy61jr7FaVMcaYJrHEYYwxpkkscRhjjGkSSxzGGGOaxBKHMcaYJrHEYYwxpkkscRhjjGkSVxOHiIwVka0ikiEid9axPlJEXnXWLxeRFKe8o4h8KiLFIvJ4rW2Gi8h6Z5tHRUTcPAdjjAlEH286yFOfb3dl364lDhEJBZ4AxgGpwFQRSa1V7XogX1X7AA8D9zvlZcCfgF/Xset/ATcAfZ3X2OaP3hhjAtvCjQd4cUmmK/t284pjJJChqjtUtQKYD0ysVWci8IKz/AZwnoiIqpao6pd4Esi3RKQrEKuqy9TzBKoXgUtcPAdjjAlIOUXlJLSLdGXfbiaO7sAer/dZTlmddVS1CigAOjawz6wG9mmMMa1eoCYOnxKRGSKSLiLpOTk5vg7HGGNaVE5xYCaOvUCy1/skp6zOOiISBsQBuQ3sM6mBfQKgqrNUNU1V0xIS6pzg0RhjglJ1jZJbXE5CTOAljpVAXxHpJSIRwBRgQa06C4BpzvIkYJHTdlEnVd0PFIrIKKc31TXAO80fujHGBK68kgpqFNeuOFybVl1Vq0TkZmAhEAo8q6obReReIF1VFwBzgLkikgHk4UkuAIhIJhALRIjIJcAFqroJuAl4HmgLfOC8jDHGOHKKyoEATBwAqvo+8H6tsru9lsuAycfYNuUY5enAgOaL0hhjgktOsbuJI2gbx40xprX69oojpo0r+7fEYYwxQeZo4ujULsKV/VviMMaYIJNTVE5MZBhREe60RljiMMaYIOPmGA6wxGGMMUEnp6jMtTEcYInDGGOCjpvTjYAlDmOMCTqWOIwxxjRaWWU1hWVVljiMMcY0zqGjg/+sjcMYY0xjuD3dCFjiMMaYoGKJwxhjTJO4PU8VWOIwxpigklNUjgjER7sz3QhY4jDGmKCSU1ROfFQE4aHu/Xm3xGGMMUHE7TEcYInDGGOCitvzVIElDmOMCSo5Re49a/woSxzGGBMkVNVuVRljjGm8ovIqyqtqLHEYY4xpnJYY/AeWOIwxJmh896xxSxzGGGMawa44jDHGNIklDmOMMU2SU1xOeKgQ1zbc1eNY4jDGmCBxdAyHiLh6HEscxhgTJFpiDAe4nDhEZKyIbBWRDBG5s471kSLyqrN+uYikeK27yynfKiIXepXfKiIbRGSjiNzmZvzGGBNIAj5xiEgo8AQwDkgFpopIaq1q1wP5qtoHeBi439k2FZgC9AfGAk+KSKiIDABuAEYCg4HxItLHrXMwxphA0hLzVIG7VxwjgQxV3aGqFcB8YGKtOhOBF5zlN4DzxHNzbiIwX1XLVXUnkOHs71RguaqWqmoV8DlwmYvnYIwxAaG6Rsktdn+eKnA3cXQH9ni9z3LK6qzjJIICoGM9224AzhKRjiISBVwEJNd1cBGZISLpIpKek5PTDKdjjDH+K7e4nBp1vysuQJjrR2hGqrpZRO4HPgJKgLVA9THqzgJmAaSlpWmLBWmMMS2otKKK+Sv2MPuLHQCc3Lmd68d0M3Hs5ftXA0lOWV11skQkDIgDcuvbVlXnAHMAROQveK5GjDGm1Xlv3X7++O/15JdWMrJXPH+7fBCnndTR9eO6mThWAn1FpBeeP/pTgCtr1VkATAOWApOARaqqIrIAmCci/wC6AX2BFQAikqiq2SLSA0/7xigXz8EYY/zWY4u20SE6gtnXpJGWEt9ix3UtcahqlYjcDCwEQoFnVXWjiNwLpKvqAjxXDnNFJAPIw5NccOq9BmwCqoCZqnr0ltSbItIRqHTKD7t1DsYY469UlczcEn56Ws8WTRrgchuHqr4PvF+r7G6v5TJg8jG2vQ+4r47ys5o5TGOMCTjZReWUVdaQ0im6xY9tI8eNMSYA7TxUAkBKx6gWP7YlDmOMCUC7co8mDrviMMYY0wiZuaWEhwrd2rdt8WNb4jDGmAC0K7eE5PgoQkPcnQm3LpY4jDEmAO08VOqT21RgicMYYwKOqrIrt4SePmgYB0scxhgTcHKKyymtqKaXD7rigiUOY4wJOJmHSgHoabeqjDHGNEZmru/GcIAlDmOMCTi7cksICxG6+6ArLljiMMaYgJOZW0pyfBRhob75E26JwxhjAkzmId/1qAJLHMYYE1A8XXF9N4YDLHEYY0xAyS2poLi8ymcN42CJwxhjAkqmMytuTx+N4QBLHMYYE1Aycz1jOOxWlTHGmEbZlVtCaIiQ1ME3XXHBEocxxgSUnYdKSOrQlnAfdcUFSxzGGBNQduWW+myqkaMscRhjTIBQVTJzS3zaowoscRhjTMDIL62kqKzKpw3jYInDGGMCgqqyYmceACmdfHvFEebToxtjjKnXJ5sP8s7afSzZnsuh4nJCQ4RTusT6NCZLHMYY46dqapRbXllDZHgoZ/ftxOjenTizbye6+WhW3KMscRhjjJ/aX1hGSUU1v7/4VH56Wk9fh/Mta+Oox97DR6ipUV+HYYxppTKyiwHokxDj40i+z9XEISJjRWSriGSIyJ11rI8UkVed9ctFJMVr3V1O+VYRudCr/FcislFENojIKyLSxo3Yq2uUcf9cTNp9/+XGl1bx4tLMb+eIMcaYlvBt4khsJYlDREKBJ4BxQCowVURSa1W7HshX1T7Aw8D9zrapwBSgPzAWeFJEQkWkO3ALkKaqA4BQp16zq65R7v5Jf8acksi6rALufmcj5z70Ge+s3evG4Ywx5ge25xTTPiqc+OgIX4fyPW62cYwEMlR1B4CIzAcmApu86kwE7nGW3wAeFxFxyuerajmwU0QynP3tdmJuKyKVQBSwz43gI8JCmDQ8iUnDk1BVdueV8rs313H7a18TFhLCxYO6unFYY4z5VkZ2MX0SYvD8WfQfbt6q6g7s8Xqf5ZTVWUdVq4ACoOOxtlXVvcCDeBLIfqBAVT+q6+AiMkNE0kUkPScn54RORETo2TGaOdNGMDS5PbfOX8NHGw+c0D6NMaYh27OL/e42FQRY47iIdMBzNdIL6AZEi8hVddVV1VmqmqaqaQkJCc1y/OjIMJ67dgQDuscxc95qPt2a3eA2xeVV/OrVtd8O3DHGmMbIL6kgt6SC3n7WMA7uJo69QLLX+ySnrM46IhIGxAG59Wx7PrBTVXNUtRJ4CxjtSvTH0K5NOC9cN5KTO7fjllfWkJVfesy6qspvXv+at9fs5bb5aygur2rBSI0xgWx7jn82jIO7iWMl0FdEeolIBJ5G7AW16iwApjnLk4BFqqpO+RSn11UvoC+wAs8tqlEiEuW0hZwHbHbxHOoU1zacf/10OKrwq1fXUn2MLrvPfLGTDzYc4LJh3dlfWMaDC7e2cKTGmEDlrz2qwMXE4bRZ3AwsxPPH/TVV3Sgi94rIBKfaHKCj0/h9O3Cns+1G4DU8DekfAjNVtVpVl+NpRF8NrHfin+XWOdSnR8co7p3Yn5WZ+TzxacYP1i/fkcvfPtzCuAFdeGjyYK4Z1ZMXlmayald+vfu1cSPGGPBccUSGhfh8lHhdxPMDP7ilpaVpenp6s+9XVbnt1bX8Z91+Xvv5KIb3jAdgT14plz65hNi2Ybwz8wzatQmnuLyKC/7xOdGRYbx3y1lEhIVQVlnNF9sOsWZ3Ppv3F7LlQBGHisuZMLg7N55zEn0S2zV7zMaYwHDtcys4UFjOB7ee5ZPji8gqVU2ra51NOXICRIT/u2QAq3blc/O8NZzaNZZN+wo5UFhG2/BQ5t1wGu3ahAMQExnGfZcO5NrnV3LnW+uorFYWbT5ISUU1YSFCn8QYRp3UkTbhIby9Zi9vrcnigtTO3DymLwOT4nx8psaYlpaRU8zgpPa+DqNOljhOUGybcB6ZMpTrnl/J3vwjnN67I6ldYznnlAT6dv7+FcOYfolMHNKNt1bvJT46gglDunPxwK6M6NWByLDQb+v95sJ+PP/VTp5fksnCjQe5eFBXfnPBKaR08szBn5FdxNtr9lJeWcNdF51KaIh/9fE2xpyYsspqsvKPcPmwJF+HUie7VdXCyiqr2bS/kEHd4whr4JnBRWWVzF68g9lf7KSyuoYJQ7qx7WAx6/cWECJQo/CzM3vxx/G1B+QbYwLZpn2FXPToFzx+5VDGD+rmkxjqu1UVUOM4gkGb8FCG9ejQYNIAT9ff2y84hc9/ew5TRiazYO0+FOVP41NZ9vvzmD46hWe+3MkrK3a3QOTGmJaS4XTF9ccxHGC3qgJCYrs2/L9LBvLnCQO+d1vqjxefys5DJfzp3xvoGR/F6D6dfBilMaa5bM8uJkSgVyffPiL2WOyKI4DUbssICw3hsSuHclJCNL94aRU7nF8pjVVZXcOdb65j4uNf8sqK3ZRW2ABFY/xBRk4xyfFRtAkPbbiyD1jiCHCxbcKZM20E4aEhXP9COgWllY3arqyympteXs38lXs4fKSSu95az2n3fcI9CzaSW1zuctTGmPpszy7229tUYIkjKCTHR/HU1cPJyi/lpnmrqKyuqbd+aUUVP3shnY83HeTPE/rz2a/P4Y1fnM55pyby8vJdTH56KQcKylooemOMt+oaZcehEr8cMX6UJY4gMSIlnr9cOpCvMnK5991NddYpraji063ZXPXMcpZsP8SDkwczbXQKIkJaSjz/nDKUeTeMIruwnMlPL2FP3rHn4TLGuCMrv5SKqhq/e+qfN2scDyKT05LJyC7m6cU76BwbSb8usRwoLONAQRmrd+eTnplPRXUNURGhPH7lMC4a+MNnioxIiefln53GNc+u4IqnlzL3+tO+10BnY0aMcdfROap6J/pnwzhY4gg6vx3bj+05xTz40TffloUInNy5HdPPSOGsvp0YkRJfb6Pb4OT2zJ8xiqvnLOf8f3z+vXU94qM4o08nzuzTidG9O9LBz55MZkygS3fms/PnNg4bABiEyquqWb4jj9i24XSJbUOnmIhGjRupbXduKe+u2/ft7L/VNcrGfYUs25FLcXkVoSHC+EFdueGskxjQ3aZFMeZELf4mh+nPreCC1C48dfVwn8ZS3wBASxymyaqqa/g6q4D31u3n1ZW7KamoZtRJ8dx63smc3rujr8MzJiBlZBdz6ZNf0b19W968cTTRkb69IXTCiUNEegNZqlouIucAg4AXVfVws0bqEksc7iksq2T+it0891Um+wvKmD46hTvH9fPb/ufG+KPDpRVc8sRXFJVV8c7NZ5DUIcrXITXLlCNvAtUi0gfPMzR6AfOaKT4TwGLbhDPj7N4suuMcpo9O4fklmYx/7Es27C3wdWjGBARV5eZ5a9h7+AhPXz3cL5JGQxqbOGqcBzNdCvxTVX8F/LBLjmm12kaEcs+E/sy9fiRFZZVc8sRXvLO29pOCjTG1bTlQxJcZh/jd2H6kpcT7OpxGaWziqBSRqXge8/ofpyzcnZBMIDurbwILbzubYT07cPtrX/Phhv2+DskYv5aemQfAhf27+DiSxmts4rgWOB24T1V3Os8Bn+teWCaQtY+K4NnpIxicFMcvX1nDJ5sP+jokY/zWysx8OsdGktTB/x4ReyyNShyquklVb1HVV0SkA9BOVe93OTYTwGIiw3j+upGc2jWWG19azRfbcnwdkjF+KT0zj7SUeEQCZ3BtoxKHiHwmIrEiEg98DTwnIv9wNzQT6GLbhPPidSM5KSGaG15M//aS3BjjsffwEfYVlDGiZwdfh9Ikjb1VFaeqhcBlwHOqOhw4372wTLBoHxXBSz87ja5xbbnu+ZVs3l/o65CM8RtHf0yN6BUYjeJHNTZxhIlIV+AKvmscN6ZROsVEMvf6kURFhHH1nBVkHirxdUjG+IUVO/OIiQyjX5dYX4fSJI1NHPcCC4HtqrpSRE4CtrkXlgk2SR2imHv9SKprarhqznIOFtq07cakZ+YzrGeHgJs8tLGN46+r6iBVvdF5v0NVL3c3NBNs+nZux/PXjiSvpIKbXl7d4HNDjAlmBaWVbD1YFHDtG9D4xvEkEXlbRLKd15sikuR2cCb4DE5uz/2XD2LVrnz+/uEWX4djjM+s2h2Y7RvQ+FtVzwELgG7O612nzJgm+8ngblxzek9mf7GThRsP+DocY3xiZWY+4aHC4KT2vg6lyRqbOBJU9TlVrXJezwMJDW0kImNFZKuIZIjInXWsjxSRV531y0UkxWvdXU75VhG50Ck7RUTWer0KReS2Rp6D8SN/uPhUBiXF8evXv2Z3rj1p0LQ+6Zl5DOgeR9uIwJsQtLGJI1dErhKRUOd1FZBb3wYiEgo8AYwDUoGpIpJaq9r1QL6q9gEeBu53tk0FpgD9gbHAkyISqqpbVXWIqg4BhgOlwNuNPAfjRyLDQnniymEIcOPLqygsq/R1SMa0mLLKar7eU8CIAJmbqrbGJo7r8HTFPQDsByYB0xvYZiSQ4TSkVwDzgYm16kwEXnCW3wDOE8/wyYnAfFUtV9WdQIazP2/n4enltauR52D8THJ8FI9MGcrWA0VcOXsZucXlvg7JmBaxfm8BFdU1wZ04VHWXqk5Q1QRVTVTVS4CGelV1B/Z4vc9yyuqs48y+WwB0bOS2U4BXjnVwEZkhIukikp6TY9Nd+Ksx/RKZfU0a2w4Wc8XTS9lfcMTXIRnjupXOwL/hAdijChp/xVGX25stiiYSkQhgAvD6seqo6ixVTVPVtISEBptjjA+N6ZfIi9eN5GBhOZP+tdQGCJqgt3rXYU5KiCY+OsLXoRyXE0kcDY1Y2Qske71PcsrqrCMiYUAcnraThrYdB6xWVZt2NUicdlJHXrlhFEcqq7ni6aVszyn2dUjGuGbz/kIGdo/zdRjH7UQSR0PPnF0J9BWRXs4VwhQ8XXq9LcDzjA/wtJssUs+zbBcAU5xeV72AvsAKr+2mUs9tKhOYBibF8coNo6iuUabMWkZGtiUPE3wOl1aw9/ARUrsG1jQj3upNHCJS5HR5rf0qwjOe45icNoub8UxVshl4TVU3isi9IjLBqTYH6CgiGXhufd3pbLsReA3YBHwIzFTVaiemaODHwFvHec7Gj53SpR3zZ4xCFSd5FPk6JGOa1SZnos/UboGbOMTzAz+4paWlaXp6uq/DME2QkV3M1NnLUFXevukMkuP9/znMxjTGM1/s4P+9t5n0P55Pp5hIX4dzTCKySlXT6lp3IreqjHFNn8QY5s8YRXllDbfOX0OVzWtlgsSm/YV0jo3066TREEscxm/1TojhvssGsnr3YR5blOHrcIxpFpv2FQZ0+wZY4jB+bsLgblw2rDuPLdpmTxA0Aa+8qpqM7OKAbt8ASxwmANw7cQBJHaK4df5aCo7Y1CQmcG07WExVjZLaNXC74oIlDhMAYiLDeGTKEA4UlnHHa2s5UlHt65CMOS7B0KMKLHGYADG0Rwfu+Ukqn2zJZvLTS9h32KYmMYFn075CoiJC6RngvQQtcZiAcfXpKTxzTRqZh0qZ8PhX1uZhAs6m/YWc2jWWkAB7VGxtljhMQDnv1M78e+ZoYiJDmTp7GZ9/YxNYmsCgqmwOgh5VYInDBKA+ie14Z+aZnNQpht+8/jWHSyt8HZIxDcrKP0JReVXAt2+AJQ4ToOKiwnnoisHklVTwvws2+jocYxq0cZ+nYfxUu+IwxncGdI/jlvP68s7afby/fr+vwzGmXpv2FxIicErndr4O5YRZ4jAB7cZzejMoKY4/vL2enCJ7gqDxX5v2FXJSQkxAPmO8NkscJqCFh4bw0OTBlFRU87s319mcVsZvbd4fHA3jYInDBIG+ndvxh4tOZdGWbG58eTVllTZA0PiXb5/BEQQN42CJwwSJaaNT+POE/ny86SDTnl1BUZlNTWL8x8vLdwMwKICf+ufNEocJGtNGp/DIlCGs2pXP1NnLOFRsbR7G9xZuPMCDH21l/KCunN67o6/DaRaWOExQmTikO7OnpZGRXcyNL62yNg/jUxv3FfCrV9cyqHscD04ejEhgjxg/yhKHCTpjTknkb5cNYmVmPo98ss3X4ZhWKruojBteSCeubTizr0mjTXjg96Y6yhKHCUqXDO3O5OFJPP5pBl9lHPJ1OKYV+vXr68gvrWT2NWkkxrbxdTjNyhKHCVp/ntif3gkx3Dp/rY3xMC3qYGEZi7/J4Rc/6s2AIGkQ92aJwwStqIgwHr9yKEVlldz+2lpqatTXIZlWYuHGAwBcPKiLjyNxhyUOE9T6dYnlngn9+WLbIR5dZO0dpmW8v34/fRNj6JMY+NOL1MUShwl6U0Ykc/mwJB75ZBufbsn2dTgmyB0qLmfFzjzGDQjOqw2wxGFaARHhvksHcGqXWG6dv4bduaW+DskEsY82HqRGYdzArr4OxTWWOEyr0CY8lKeuGo6I8POXVtlzy41rPtiwn16dounXJThvU4HLiUNExorIVhHJEJE761gfKSKvOuuXi0iK17q7nPKtInKhV3l7EXlDRLaIyGYROd3NczDBo0fHKP45ZQhbDhRyy/w1lFZU+TokE2QOl1awdHsuYwd0CZrBfnVxLXGISCjwBDAOSAWmikhqrWrXA/mq2gd4GLjf2TYVmAL0B8YCTzr7A3gE+FBV+wGDgc1unYMJPmNOSeSen/Tnv5sPMulfS8nKt9tWpvl8vOkgVTXKRQOC9zYVuHvFMRLIUNUdqloBzAcm1qozEXjBWX4DOE88aXoiMF9Vy1V1J5ABjBSROOBsYA6Aqlao6mEXz8EEoWmjU3h22gj25JUy8fGvWLEzr2k72L4dbroJYmMhJMTz35tu8pSbVu2DDQdI6tCWAd2DYxbcY3EzcXQH9ni9z3LK6qyjqlVAAdCxnm17ATnAcyKyRkSeEZFod8I3wWxMv0TennkGcW3DuXL2Mv747/XsyWvE1ccHH8CgQfDMM1BUBKqe/z7zjKf8gw/cD974pcKySr7cdohxQX6bCgKvcTwMGAb8S1WHAiXAD9pOAERkhoiki0h6Tk5OS8ZoAkSfxBjennkGk9OSeW1lFuc8+Bm3v7qWzfsL695g+3aYNAlKS6Gy1rTtlZWe8kmT7Mqjlfp8aw4V1TWMDfLbVOBu4tgLJHu9T3LK6qwjImFAHJBbz7ZZQJaqLnfK38CTSH5AVWepapqqpiUkJJzgqZhgFdc2nL9eNpDFvx3D9NEpfLDhAOMe+YILH17ME59mfL/r7kMP/TBh1FZZCQ8/7G7Qxi99c7CI0BBhUFLwTTFSm5uJYyXQV0R6iUgEnsbuBbXqLACmOcuTgEWqqk75FKfXVS+gL7BCVQ8Ae0TkFGeb84BNLp6DaSW6xLXhT+NTWXLnudw7sT/t2oTxwMKtnP3Ap4z952L+/uEWqufObVzimDu3ZYI2fmVXbind2rchPDTQbuQ0XZhbO1bVKhG5GVgIhALPqupGEbkXSFfVBXgaueeKSAaQhye54NR7DU9SqAJmqurRjve/BF52ktEO4Fq3zsG0Ph2iI7jm9BSuOT2FrPxS3l+/n082Z/P04h38urikcTspLnY3SOOXduWV0iM+ytdhtAjx/MAPbmlpaZqenu7rMEwAKzhSSVRCPOElDSeFI22jWbRsG+f2S6RtRPA8g8HUb+i9HzF2QFf+etlAX4fSLERklaqm1bUu+K+pjGkGcW3DCb/maggPr7deVWgYCwaey8x5q7n4sS/ItcfXtgqFZZXkl1bSs2PruOKwxGFMY91xR4OJIywygkkv/YNZVw9nb/4Rpj+3kuJyG6Ee7I52oujZSm5VWeIwprF694Y33oCoqB8mkPBwT/kbbxDatw8X9O/Cv64axqb9hcx4MZ2ySpsbK5jtdsYA9bArDmPMD4wbB+vWwYwZ3x85PmOGp3zcuG+rntuvMw9OHsSS7bncOn8NVdU1PgzcuGmXc8XRWhrHLXEY01S9e8Pjj0NBAVRXe/77+OOe8louHZrE3eNTWbjxIDe+vNpm5Q1Su/NKiI+OoF2b+m9lBgtLHMa47Loze/HnCZ6JFafMXsYhazAPOrtyW09XXLDEYUyLmDY6haevGs7WA4Vc9uQSduTYWI9gsjuvtNX0qAJLHMa0mAv6d+GVG0ZRUl7FpU8u4fNvbA61YFBRVcO+w0daTY8qsMRhTIsa2qMDb990Bl3j2jD9uRU8vmgbNTXBPwg3mO09fIQahR4dW89E3ZY4jGlhPTpG8dZNo5kwuBsPfvQNP39pFQVHGpgDy/itXbmeqWjsVpUxxlVREWH883+GcPf4VBZtyebcBz9j/ordVNvVR8D5dgyH3aoyxrhNRLjuzF68M/MMTkqI5s631vOTx75k2Y5cX4dmmmBXbiltwkNIbBfp61BajCUOY3xsQPc4Xvv56Tw2dSiHSyuYMmsZV89Zzurd+b4OzTTCbmdW3GB/6p83SxzG+AER4SeDu/HJHedw17h+bNzn6bY77dkVlkD83O7cUnrEt56GcbDEYYxfaRsRys9/1JsvfjuGO8f1Y/3eAi57cglXz1nOip15vg7P1KKqrW4MB1jiMMYvRUeG8Qsngfz+on5s3l/IFU8vZcqspd9/nK3xqZyico5UVlviMMb4j+jIMGac3Zsvfnsud49PZfP+Ii598iu7feUndrXCHlVgicOYgNA2IpTrzuzFWzeNJjoyjKmzlvHeuv2+DqvVa22z4h5licOYANI7IYa3bxrNgO5xzJy3mn98/I0968OHdueVEiKQ1MEShzHGj3WMieTln53GpUO78+gn2zj3wc94c1WWDR70gd25JXSNa0tEWOv6U9q6ztaYINEmPJSH/2cI8244jY4xkdzx+teMf+xL1ljbR4va1V7rK1YAABCUSURBVAp7VIElDmMC2ujenXhn5hk8OnUoBaUVTHpqKU98mmFXHy2gpkbZlWuJwxgTgEJChAmDu/HBbWczbkAXHli4lavnLOdgYZmvQwtqz3y5g7ySCk7v3cnXobQ4SxzGBIm4tuE8NnUof798EGt2H+aChxczd2mmXX24YOO+Ah5YuJUL+3fmJ4O6+jqcFmeJw5ggIiJcMSKZ/9xyJv27xfKndzYy/rEvbdR5MyqrrOa2+WvpEBXBXy8b1KrmqDrKEocxQah3Qgwv/+w0nvzpMAqPVHLF00u54cV0GzjYDP72wRa2ZRfzwOTBxEdH+Docn3A1cYjIWBHZKiIZInJnHesjReRVZ/1yEUnxWneXU75VRC70Ks8UkfUislZE0t2M35hAJiJcNLAr/739R/zq/JNZsTOPy55cwhVPL+Wzrdm+Di8gLf4mh+eXZDJ9dAo/OjnB1+H4jGuJQ0RCgSeAcUAqMFVEUmtVux7IV9U+wMPA/c62qcAUoD8wFnjS2d9RY1R1iKqmuRW/McGibUQot57flyV3nsufxqeyJ6+U6c+t5K631lNeZYMHm+Kf//2GHvFR3Dmun69D8Sk3rzhGAhmqukNVK4D5wMRadSYCLzjLbwDnieeG4URgvqqWq+pOIMPZnzHmOEVHhnH9mb34/DdjuPGc3ryyYjdXPL2M/QVHfB1aQFifVcDq3YeZPjqFNuGhDW8QxNxMHN2BPV7vs5yyOuuoahVQAHRsYFsFPhKRVSIy41gHF5EZIpIuIuk5OTkndCLGBJOIsBB+N7YfT101jIyDRYx/9Es+3HCAquoaX4fm115cmklURCiXD0/ydSg+F+brAI7Dmaq6V0QSgY9FZIuqLq5dSVVnAbMA0tLSrD+iMbWMHdCVPokx/HzuKn7x0ioS20Vy2bAkJg1Pok9ijK/D8yv5JRW88/U+Jg9PIq5tuK/D8Tk3rzj2Asle75OcsjrriEgYEAfk1retqh79bzbwNnYLy5jj1iexHR/cejZPXTWMQUlxzP5iB+f/43Pu/3ALNTb+41uvpu+hoqqGa05P8XUofsHNxLES6CsivUQkAk9j94JadRYA05zlScAiVVWnfIrT66oX0BdYISLRItIOQESigQuADS6egzFBLyIshLEDuvLMtBEsvetcpo5M5l+fbWfG3HSKy6sAqKqu4fX0PVz86Bc89sk2PP9MW4fqGuWlZbs4rVc8p3Rp5+tw/IJrt6pUtUpEbgYWAqHAs6q6UUTuBdJVdQEwB5grIhlAHp7kglPvNWATUAXMVNVqEekMvO0MuAkD5qnqh26dgzGtTWK7Nvz1skGkdo3lnnc3cdmTX/HT03oy58ud7M4rpXNsJA99/A1bDxbx4OTBraKR+NMt2WTlH+H3F53q61D8hrSGXw5paWmanm5DPoxpiq8yDnHTy6spOFLJgO6x3HbeyZzbL5GnFm/ngYVbGdg9jtnXpNE5to2vQz0hC77ex33vbSI+OpK+iTH0TYwhOT6KjjERxEdHcN97m9mRU8IXvxtDeGjrGTMtIquONeTBEocx5piy8kvZlVvK6N4dvze1xkcbD3Dbq2sRIDk+ioR2kXSKiWTCkG6MOSXRdwE30X/W7eOWV9aQ2i2WTjGRZGQXk5X/w+7Jt//4ZG45r68PIvQdSxyWOIxpdlsOFDJ36S6yi8rJKSonK7+U3JIK7p04gKtH9fR1eA36cMN+Zs5bw7Ae7Xn+2pFER3ru3JdWVLG/oIy8kgpyi8spLq/m4oFdaRsR/LflvNWXOAKxO64xxg/06xLLfZcO/Pb9kYpqbp63mj/9ewPZhWXc/uOT/XYCwI83HeTmeWsYnBTHc15JAyAqIozeCTH0br0zijSo9dywM8a4qm1EKE9fPZz/SUvmsUUZ/PaNdX75TJDi8ipuf3Ut/bvF8sJ1I4mJtN/PTWWfmDGm2YSFhvC3ywfSOTaSRxdl8PqqLIYkt+fHqZ2ZMLgbyfG+f1reayv3UFRexb0TB9CujQ3mOx52xWGMaVYiwu0XnMLHvzqb31x4CqrKAwu38uOHP+fFpZk+HQNSXaM8t2QnaT07MDi5vc/iCHR2xWGMcUXfzu3o27kdM8f0YU9eKX/89wbufmcji7Zk8/dJg0hs1/LdeD/edJA9eUe4a5yNyTgRdsVhjHFdcnwUz187gnsn9mfp9lwufHgxb67KavFpTZ79aifd27flgtTOLXrcYGOJwxjTIkSEa05P4b1bzqJnx2jueP1rJj21hPVZBS1y/A17C1ixM49rz0ghrBUN5HODfXrGmBbVJzGGt24czQOTBrE7r5QJT3zJ7a+tZdmOXFevQOZ8uZPoiFCuGJHccGVTL2vjMMa0uJAQYXJaMhcO6MIj/93GvOW7eWv1XrrEtmH8oK5cPjyJU7vGNtvxDhaW8e7X+7j69J7EWk+qE2Yjx40xPldaUcV/N2ezYO1ePv8mh8pqZUhye64c2YPxg7sSFXH8v3HLq6r5+dxVLP4mh89+PYYeHX3fJTgQ2JQjljiMCRj5JRW8vWYvr6zYzbbsYtpFhnHFiGSmj05p8jiQ8qpqfjF3FZ9uzeEvlw7kytN6uBR18LHEYYnDmICjqqzalc/cZbt4b91+alT5cWpnrjujFyN7xTc4nYkljRNjicMShzEB7UBBGS8uzWTeit0cLvVM837dGb0YP6gbYSFCdlE5u/NK2V9whEPFnskJV+zMI31XviWN42SJwxKHMUHhSEU1b6/Zy7Nf7SQju5h2bcKoqKqhvKrme/XCQoROMZHc/uOTrRfVcbLZcY0xQaFtRChXntaDqSOTWbztEO+t20f7qAiS46PoER9F9/Zt6BQTSWybcEJC/HNm3mBgicMYE3BEhB+dnMCPTra5z33BBgAaY4xpEkscxhhjmsQShzHGmCaxxGGMMaZJLHEYY4xpEkscxhhjmsQShzHGmCaxxGGMMaZJWsWUIyKSA+xqRNU4oL7HkR1rfe3y+t4fXa6rrBNwqBFxuh1zQ8snEu/xxtxQWSDE3NTP3N++Fw3F6V0WKDH7w/fiWDE2FLvbMfdU1bpHWKqqvZwXMOt41tcur+/90eVjlKX7Q8wNLZ9IvMcbc0NlgRBzUz9zf/teNBRnIMbsD9+LxnwXfB1z7Zfdqvq+d49zfe3y+t6/W0/Z8XAj5oaWTyTexmxf1/qGygIh5qZ+5v72vaj9PhC+y7Xf147ZH74Xtcv88bv8Pa3iVlWgEJF0PcZslP4o0OIFi7mlBFrMgRYv+DZmu+LwL7N8HUATBVq8YDG3lECLOdDiBR/GbFccxhhjmsSuOIwxxjSJJQ5jjDFNYonDBSLyrIhki8iG49h2uIisF5EMEXlURMRr3S9FZIuIbBSRv/t7zCJyj4jsFZG1zusif4/Za/0dIqIi0qn5Inbtc/4/EVnnfMYfiUg3P4/3Aed7vE5E3haR9s0Vr4sxT3b+3dWISLM1SJ9IrMfY3zQR2ea8pnmV1/t9b7Lj6Qdsrwb7SZ8NDAM2HMe2K4BRgAAfAOOc8jHAf4FI531iAMR8D/DrQPqcnXXJwEI8g0Y7+XvMQKxXnVuAp/w83guAMGf5fuD+APiMTwVOAT4D0nwdqxNHSq2yeGCH898OznKH+s7reF92xeECVV0M5HmXiUhvEflQRFaJyBci0q/2diLSFc8fgWXq+b/9InCJs/pG4G+qWu4cIzsAYnaVizE/DPwWaPaeI27ErKqFXlWjmzNul+L9SFWrnKrLgKTmitfFmDer6tbmjPNEYj2GC4GPVTVPVfOBj4GxbvwbtcTRcmYBv1TV4cCvgSfrqNMdyPJ6n+WUAZwMnCUiy0XkcxEZ4Wq0HicaM8DNzi2JZ0Wkg3uhfuuEYhaRicBeVf3a7UC9nPDnLCL3icge4KfA3S7GCs3zvTjqOjy/gN3WnDG7rTGx1qU7sMfr/dH4m/28wk5kY9M4IhIDjAZe97q1GNnE3YThuQQdBYwAXhORk5xfEM2umWL+F/B/eH4B/x/wEJ4/FK440ZhFJAr4PZ5bKS2imT5nVPUPwB9E5C7gZuB/my1IL80Vr7OvPwBVwMvNE90xj9NsMbutvlhF5FrgVqesD/C+iFQAO1X10paM0xJHywgBDqvqEO9CEQkFVjlvF+D5Q+t92Z4E7HWWs4C3nESxQkRq8ExyluOvMavqQa/tZgP/cSnWo0405t5AL+Br5x9tErBaREaq6gE/jbm2l4H3cSlx0Ezxish0YDxwnls/frw092fspjpjBVDV54DnAETkM2C6qmZ6VdkLnOP1PglPW8hemvu8mquRx14/aLxKwavBC1gCTHaWBRh8jO1qN2Jd5JT/ArjXWT4ZzyWp+HnMXb3q/AqY7++fc606mTRz47hLn3Nfrzq/BN7w83jHApuAhOb+bN3+XtDMjePHGyvHbhzfiadhvIOzHN/Y73uTYnbrf1xrfgGvAPuBSjxXCtfj+SX7IfC184/m7mNsmwZsALYDj/Pd6P4I4CVn3Wrg3ACIeS6wHliH5xddV3+PuVadTJq/V5Ubn/ObTvk6PBPUdffzeDPw/PBZ67yarReYizFf6uyrHDgILPRlrNSROJzy65zPNwO4tinf96a8bMoRY4wxTWK9qowxxjSJJQ5jjDFNYonDGGNMk1jiMMYY0ySWOIwxxjSJJQ7TKolIcQsf7xkRSW2mfVWLZybcDSLybkOzy4pIexG5qTmObQzYEwBNKyUixaoa04z7C9PvJu5zlXfsIvIC8I2q3ldP/RTgP6o6oCXiM8HPrjiMcYhIgoi8KSIrndcZTvlIEVkqImtEZImInOKUTxeR10XkXeAjETlHRD4TkTfE87yJl48+98ApT3OWi51JCb8WkWUi0tkp7+28Xyki9zbyqmgp303QGCMin4jIavE8e2GiU+dvQG/nKuUBp+5vnOOsE5E/N+PHaFoBSxzGfOcR4GFVHQFcDjzjlG8BzlLVoXhmnv2L1zanA9NU9Vzn/VDgNiAVOAk4o47jRAPLVHUwsBi4wev4jzjH39dQsM5cS+fhGZUPUAZcqqrD8Dy/5SEncd0JbFfVIar6GxG5AOgLjASGAMNF5OyGjmfMUTbJoTHfOR9I9ZqVNNaZrTQOeEFE+uKZ6Tfca5uPVdX7eQorVDULQETW4pmH6Mtax6nguwkfVwE/dpZP57vnJMwDHjxGnG299r0Kz3MXwDMP0V+cJFCD50qkcx3bX+C81jjvY/AkksXHOJ4x32OJw5jvhACjVLXMu1BEHgc+VdVLnfaCz7xWl9TaR7nXcjV1/xur1O8aF49Vpz5HVHWIiMThSUAzgUfxPIsjARiuqpUikgm0qWN7Af6qqk838bjGAHaryhhvH+GZXRYAETk6tXUc301DPd3F4y/Dc4sMYEpDlVW1AM+jYu8QkTA8cWY7SWMM0NOpWgS089p0IXCdczWFiHQXkcRmOgfTCljiMK1VlIhkeb1ux/NHOM1pMN6EZyp7gL8DfxWRr4BQF2O6DbhdRFYAXYGChjZQ1TV4ZsWdiudZHGkiko7n6mOLUycX+MrpvvuAqn6E51bYUhFZD7zB9xOLMfWy7rjG+AnnCYRHVFVFZAowVVUnNrSdMS3N2jiM8R/DgcednlCHcfExu8acCLviMMYY0yTWxmGMMaZJLHEYY4xpEkscxhhjmsQShzHGmCaxxGGMMaZJ/j/QsBY7s4ZsBwAAAABJRU5ErkJggg==\">"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The model is now ready for training, using the `model.fit()` method. The `model.fit()` is given the number of epochs for training and the estimated learning rate generated by `lr_find()` in the previous step:"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["fcn.fit(100, lr=0.0002754)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["This model will train for 100 epochs using the provided learning rate. If the learning rate is not mentioned, then an optimal learning rate is automatically deduced for training the model.\n", "\n", "Other important inputs are the two optional boolean parameters of `early_stopping` and `checkpoint`. The `early_stopping` parameter by default is set to `False`, but if set to `True`, will stop the training if validation loss stops improving for 5 epochs. The `checkpoint` parameter can be used to save the best model during training. When set to `True`, the best model based on validation loss will be saved during training."]}, {"cell_type": "markdown", "metadata": {}, "source": ["### ML Model Training<a class=\"anchor\" id=\"10\"></a>\n", "Once the machine learning model is initialized with the data and the scikit-learn model parameters mentioned in the previous step, the `model.fit()` method is used for fitting the machine learning model. "]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["model.fit()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The training results are printed using `model.show_results()` to assess the quality of the trained model and also compute model metrics. The `model.show_results()` will  typically output a table where the last column will contain the values predicted by the model after being applied to the test dataset. Comparing the predicted values with the actual target variable will give some indication of the performance of the trained model. Subsequently, the model metrics of the trained model can be estimated using the `model.score()` function which currently returns the r-square, which also reflects model performance."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Model Validation metrics, Saving & Loading<a class=\"anchor\" id=\"11\"></a> "]}, {"cell_type": "markdown", "metadata": {}, "source": ["After training the `FullyConnectedNetwork` model, the train vs valid losses can be plotted using `model.plot_losses()` to check if the model is overfitting. For example, the plot below shows that the model has trained well. While the losses are still gradually decreasing, they are not decreasing at a significant level and the model is therefore ready and can be saved for inferencing."]}, {"cell_type": "markdown", "metadata": {}, "source": ["<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxV5Z348c/3Ltn3BQQCJCyKAVFCBNG6II5C2xHbYgvVGW21trZOf62//mawnVrrTKc69efS0ak/q6J2rEixttSitFVblyoQrCKLSGSRsGUBErLn3vv9/XFOwk1IyA3k5t4k3/frdc25z3nOud9zIveb5zznPI+oKsYYY0ykPLEOwBhjzOBiicMYY0yfWOIwxhjTJ5Y4jDHG9IklDmOMMX3ii3UAAyEvL08LCwtjHYYxxgwqGzZsqFbV/K7lwyJxFBYWUlZWFuswjDFmUBGR3d2V26UqY4wxfRLVxCEi80Vkm4iUi8jSbtYnisiz7vq1IlLolueKyKsiUi8iD3bZZomIvC8iG0XkJRHJi+YxGGOM6SxqiUNEvMBDwAKgGFgiIsVdqt0AHFbVScB9wN1ueTPwfeA7XfbpAx4A5qrqdGAjcEu0jsEYY8zxotnHMQsoV9UdACKyHFgIbAmrsxC4w11eCTwoIqKqDcAbIjKpyz7FfaWKSA2QAZRH7xCMMfGkra2NiooKmpubYx3KkJKUlERBQQF+vz+i+tFMHGOAPWHvK4DZPdVR1YCI1AK5QHV3O1TVNhG5GXgfaAC2A9/orq6I3ATcBDBu3LiTPwpjTNyoqKggPT2dwsJCRCTW4QwJqkpNTQ0VFRUUFRVFtM2g6hwXET9wMzADGI1zqeq27uqq6iOqWqqqpfn5x91NZowZhJqbm8nNzbWk0Y9EhNzc3D614qKZOPYCY8PeF7hl3dZx+y8ygZoT7PMcAFX9SJ1hfVcA5/dXwMaY+GdJo//19ZxGM3GsByaLSJGIJACLgVVd6qwCrnOXFwGv6InHed8LFItIexPi74Ct/RhzJ7uqG3hje7dXzYwxZtiKWh+H22dxC7AG8AKPq+pmEbkTKFPVVcBjwC9EpBw4hJNcABCRXTid3wkichVwuapuEZEfAq+JSBuwG7g+WsdwyT1/BmDXXZ+K1kcYYwaRmpoa5s2bB8CBAwfwer20Xwpft24dCQkJve7jS1/6EkuXLuWMM86IaqzRFNUnx1V1NbC6S9ntYcvNwNU9bFvYQ/nDwMP9F6UxxkQmNzeXd999F4A77riDtLQ0vvOdTk8NoKqoKh5P9xd0li1bFvU4o21QdY4bY0w8Ki8vZ9q0aXzta1+jpKSE/fv3c9NNN1FaWsrUqVO58847O+p+4hOf4N133yUQCJCVlcXSpUs5++yzmTNnDpWVlTE8isgNi7GqjDFDzw9/t5kt++r6dZ/FozP4wd9PPaltt2zZwhNPPMHDDzsXRO666y5ycnIIBALMnTuXRYsWUVzc+Rno2tpaLr74Yu666y5uvfVWHn/8cZYuPW6QjbhjLY4IBEM2L7sx5sQmTpxIaWlpx/tnnnmGkpISSkpK2Lp1K1u2bDlum+TkZBYsWADAzJkz2bVr10CFe0qsxRGB1kCI5ARvrMMwxoQ52ZZBtKSmpnYsb9++nQceeIB169aRlZXFtdde2+1zEuGd6V6vl0AgMCCxniprcUSgNRiKdQjGmEGkrq6O9PR0MjIy2L9/P2vWrIl1SP3KWhwRaA1Y4jDGRK6kpITi4mKmTZvGhAkTuOCCC2IdUr+SEz9vNzSUlpbqyUzkVLj09wD8demljM5K7u+wjDF9tHXrVs4888xYhzEkdXduRWSDqpZ2rWuXqiJgLQ5jjDnGEkcE2qyPwxhjOljiiECLtTiMMaaDJY4IWIvDGGOOscRxAl6PM9Sw9XEYY8wxljhOoD1xtAWH/p1nxhgTKUscJ+Brb3EEgzGOxBgTD+bOnXvcw3z3338/N998c4/bpKWlAbBv3z4WLVrUbZ1LLrmE3h4ZuP/++2lsbOx4/8lPfpIjR45EGnq/ssRxAscuVVmLwxgDS5YsYfny5Z3Kli9fzpIlS3rddvTo0axcufKkP7tr4li9ejVZWVknvb9TYYnjBDoSh3WOG2OARYsW8fvf/57W1lYAdu3axb59+5gxYwbz5s2jpKSEs846i9/+9rfHbbtr1y6mTZsGQFNTE4sXL2b69Ol84QtfoKmpqaPezTff3DEc+w9+8AMAfvrTn7Jv3z7mzp3L3LlzASgsLKS62pmh9N5772XatGlMmzaN+++/v+PzzjzzTL7yla8wdepULr/88k6fcyqiOuSIiMwHHsCZAfBRVb2ry/pE4ClgJs5c419Q1V0ikgusBM4FnlDVW8K2SQAeBC4BQsD3VPW5aMTffqmqzTrHjYk/Ly6FA+/37z5POwsW3NXj6pycHGbNmsWLL77IwoULWb58OZ///OdJTk7m+eefJyMjg+rqas477zyuvPLKHufy/tnPfkZKSgobN25k48aNlJSUdKz70Y9+RE5ODsFgkHnz5rFx40a++c1vcu+99/Lqq6+Sl5fXaV8bNmxg2bJlrF27FlVl9uzZXHzxxWRnZ7N9+3aeeeYZfv7zn/P5z3+e5557jmuvvfaUT1PUWhwi4gUeAhYAxcASESnuUu0G4LCqTgLuA+52y5uB7wPf4XjfAypV9XR3v3+JQvgAeKS9c9wShzHGEX65qv0ylary3e9+l+nTp3PZZZexd+9eDh482OM+XnvttY4v8OnTpzN9+vSOdStWrKCkpIQZM2awefPmbodjD/fGG2/wmc98htTUVNLS0vjsZz/L66+/DkBRURHnnHMO0L/DtkezxTELKFfVHQAishxYCISfhYXAHe7ySuBBERFVbQDeEJFJ3ez3y8AUAFUNAdXRCR/a/1gIDoPxvIwZdE7QMoimhQsX8u1vf5t33nmHxsZGZs6cyRNPPEFVVRUbNmzA7/dTWFjY7TDqvdm5cyf33HMP69evJzs7m+uvv/6k9tMuMTGxY9nr9fbbpapo9nGMAfaEva9wy7qto6oBoBbI7WmHItLeE/RvIvKOiPxKREb2UPcmESkTkbKqqqqTOgDByRwhm8jJGONKS0tj7ty5fPnLX+7oFK+trWXEiBH4/X5effVVdu/efcJ9XHTRRfzyl78EYNOmTWzcuBFwhmNPTU0lMzOTgwcP8uKLL3Zsk56eztGjR4/b14UXXshvfvMbGhsbaWho4Pnnn+fCCy/sr8Pt1mDrHPcBBcBfVbUEeAu4p7uKqvqIqpaqaml+fv5JfVhHi8MShzEmzJIlS3jvvfc6Esc111xDWVkZpaWlPP3000yZMuWE2998883U19czffp0/vM//5NZs2YBcPbZZzNjxgymTp3Kl7/85U7Dsd90003Mnz+/o3O8XUlJCddffz2zZs1i9uzZ3HjjjcyYMaOfj7izqA2rLiJzgDtU9Qr3/W0AqvrjsDpr3DpviYgPOADkqxuUiFwPlLZ3jovT01QPpKtqSETGAi+p6gmnAjvZYdXP//HL7Ktt5l8/dSY3Xjihz9sbY/qXDasePfEyrPp6YLKIFLl3Qi0GVnWpswq4zl1eBLyiJ8hk7rrf4dxRBTCPzn0m/ar9joiQ9XEYY0yHqCUOt8/iFmANsBVYoaqbReROEbnSrfYYkCsi5cCtwNL27UVkF3AvcL2IVITdkfUvwB0ishH4B+B/R+sYvnqx08oI2KUqY4zpENXnOFR1NbC6S9ntYcvNwNU9bFvYQ/lu4KL+i7Jni88dx+2/3Wyd48bEEVXt8fkIc3L62mUx2DrHB1T7A4D2GIcx8SEpKYmampo+f9GZnqkqNTU1JCUlRbxNVFscg52nI3FY5jAmHhQUFFBRUcHJ3mJvupeUlERBQUHE9S1x9MLnEXsA0Jg44ff7KSoqinUYw55dquqFxyPWOW6MMWEscfTCK2Kd48YYE8YSRy98HrHOcWOMCWOJoxcej1jnuDHGhLHE0QvrHDfGmM4scfSipqGVdTsPxToMY4yJG5Y4IvDhwfpYh2CMMXHDEocxxpg+scTRi69eNIFEn50mY4xpZ9+IvUhN9NESCBGwe3KNMQawxNGrlAQvAA2twRhHYowx8cESRy9SEpzhvJoscRhjDBDlxCEi80Vkm4iUi8jSbtYnisiz7vq1IlLolueKyKsiUi8iD/aw71Uisima8QMd/RstAUscxhgDUUwcIuIFHgIWAMXAkrBZ/NrdABxW1UnAfcDdbnkz8H3gOz3s+7M4c49HXaK/PXFYH4cxxkB0WxyzgHJV3aGqrcByYGGXOguBJ93llcA8ERFVbVDVN3ASSCcikoYzzey/Ry/0Y5J8Th/HnkONA/FxxhgT96KZOMYAe8LeV7hl3dZx5yivBXJ72e+/Af8XGJBv8vYWxw1Plg3ExxljTNwbVJ3jInIOMFFVn4+g7k0iUiYiZacyW5jX5jY2xphOopk49gJjw94XuGXd1hERH5AJ1Jxgn3OAUhHZBbwBnC4if+6uoqo+oqqlqlqan59/UgcA0NRmneLGGBMumoljPTBZRIpEJAFYDKzqUmcVcJ27vAh4RU8wC72q/kxVR6tqIfAJ4ENVvaTfIw+TleKP5u6NMWbQidqc46oaEJFbgDWAF3hcVTeLyJ1AmaquAh4DfiEi5cAhnOQCgNuqyAASROQq4HJV3RKteHsyc3wOALMKcwb6o40xJi5FLXEAqOpqYHWXstvDlpuBq3vYtrCXfe8Cpp1ykBG4+PR8jjS2DsRHGWNM3BtUneOxkpLgpdGeHDfGGMASR0SSLXEYY0wHSxwRSPZ7aba7q4wxBrDEERG7VGWMMcdY4ohAcoKPprYgoVCPdwobY8ywYYkjAu1zcjTbCLnGGGOJIxLticPm5DDGGEscEUn2O4nD+jmMMcYSR0SS21scdmeVMcZY4ohE+6Uqa3EYY4wljogk+52RWRpbAzGOxBhjYs8SRwRSE61z3Bhj2lniiEBKgtPiaLDEYYwxljgi0d7iaGyxS1XGGGOJIwLtLY4PDhyNcSTGGBN7ljgikOreVfXEX3fFNhBjjIkDUU0cIjJfRLaJSLmILO1mfaKIPOuuXysihW55roi8KiL1IvJgWP0UEfm9iHwgIptF5K5oxt/O57X8aowx7aL2jSgiXuAhYAFQDCwRkeIu1W4ADqvqJOA+4G63vBn4PvCdbnZ9j6pOAWYAF4jIgmjE39U1s8eRk5owEB9ljDFxLZp/Ss8CylV1h6q2AsuBhV3qLASedJdXAvNERFS1QVXfwEkgHVS1UVVfdZdbgXeAgigeQ4esFD9HGltthFxjzLAXzcQxBtgT9r7CLeu2jqoGgFogN5Kdi0gW8PfAyz2sv0lEykSkrKqqqo+hHy87JYGQwlG7s8oYM8wNyov3IuIDngF+qqo7uqujqo+oaqmqlubn55/yZ2alOJepjjS2nvK+jDFmMItm4tgLjA17X+CWdVvHTQaZQE0E+34E2K6q9/dDnBHJTvEDcLixbaA+0hhj4lI0E8d6YLKIFIlIArAYWNWlzirgOnd5EfCKqp6wE0FE/h0nwXyrn+M9ofaO8cMN1uIwxgxvvmjtWFUDInILsAbwAo+r6mYRuRMoU9VVwGPAL0SkHDiEk1wAEJFdQAaQICJXAZcDdcD3gA+Ad0QE4EFVfTRax9EuLy0RgOr6lmh/lDHGxLWoJQ4AVV0NrO5SdnvYcjNwdQ/bFvawW+mv+PoiN81pcVTXW4vDGDO8DcrO8VhISfCR7PdSYy0OY8wwZ4mjD3LTEqixPg5jzDBniaMPctMSrY/DGDPsWeLog7zUBGqsj8MYM8xZ4ugD51KVtTiMMcObJY4+yE1LpKa+lV4eNTHGmCHNEkcf5KYmEAgpdU02XpUxZviyxNEHHQ8B2uUqY8wwZomjD9ofArQOcmPMcGaJow9yU23YEWOMscTRB3kdw45Y4jDGDF+WOPogNy0Rr0c4WNfce2VjjBmiLHH0gdcjjEhP5ECttTiMMcOXJY4+GpmRZC0OY8ywZomjj0ZlJvHxocZYh2GMMTETUeIQkYkikuguXyIi3xSRrOiGFp+mjcnk40ON7K5piHUoxhgTE5G2OJ4DgiIyCWfWviLgl71tJCLzRWSbiJSLyNJu1ieKyLPu+rUiUuiW54rIqyJSLyIPdtlmpoi8727zU3GnARwoM8dnA7DnUNNAfqwxxsSNSBNHSFUDwGeA+1X128CoE20gIl7gIWABUAwsEZHiLtVuAA6r6iTgPuBut7wZ+D7wnW52/TPgK8Bk9zU/wmPoF6MzkwHYd8QShzFmeIo0cbSJyBLgOuAFt8zfyzazgHJV3aGqrcByYGGXOguBJ93llcA8ERFVbVDVN3ASSAcRGQVkqOrb6ow0+BRwVYTH0C9GZjoPAe6rtcRhjBmeIk0cXwLmAD9S1Z0iUgT8opdtxgB7wt5XuGXd1nFbNLVAbi/7rOhlnwCIyE0iUiYiZVVVVb2EGrlEn5e8tERrcRhjhi1fJJVUdQvwTQARyQbSVfXuE28VW6r6CPAIQGlpab+Ogz4mO5l9R+yWXGPM8BTpXVV/FpEMEckB3gOWici9vWy2Fxgb9r7ALeu2joj4gEygppd9FvSyz6gryEpmr7U4jDHDVKSXqjJVtQ74LLBMVWcCl/WyzXpgsogUiUgCsBhY1aXOKpx+E4BFwCt6glmSVHU/UCci57l3U/0j8NsIj6HfjMlOZu/hJkIhm9DJGDP8RJo4fG7H9Oc51jl+Qm6fxS3AGmArsEJVN4vInSJypVvtMSBXRMqBW4GOW3ZFZBdwL3C9iFSE3ZH1deBRoBz4CHgxwmPoNwXZybQGQzbYoTFmWIqojwO4EycBvKmq60VkArC9t41UdTWwukvZ7WHLzcDVPWxb2EN5GTAtwrijYkyWc0tuxZEmRmQkxTIUY4wZcBG1OFT1V6o6XVVvdt/vUNXPRTe0+DUm20kcew9bP4cxZviJtHO8QESeF5FK9/WciBT0vuXQ1N7isA5yY8xwFGkfxzKcjuzR7ut3btmwlJ7kJyPJZy0OY8ywFGniyFfVZaoacF9PAPlRjCvujclOsRaHMWZYijRx1IjItSLidV/XcuLnLYa8AveWXGOMGW4iTRxfxrkV9wCwH+eZi+ujFNOgMMZ9CPAEj50YY8yQFOldVbtV9UpVzVfVEap6FTBs76oCp8VR3xKgtqkt1qEYY8yAOpUZAG/ttygGoY5nOexylTFmmDmVxDGgEyjFm6L8VAC27K+LcSTGGDOwTiVxDOuL+2eMTCc90cfmvbWxDsUYYwbUCYccEZGjdJ8gBEiOSkSDhIgwLjeF3YcaYx2KMcYMqBMmDlVNH6hABqNxOSlsO3g01mEYY8yAOpVLVcPeuJwUKg7Z8OrGmOHFEscpGJebQmswxIE6mw3QGDN8WOI4BeNyUgD42Po5jDHDiCWOU2CJwxgzHEU1cYjIfBHZJiLlIrK0m/WJIvKsu36tiBSGrbvNLd8mIleElX9bRDaLyCYReUZEYjaT0uisZLwe4eMaSxzGmOEjaolDRLzAQ8ACoBhYEjb9a7sbgMOqOgm4D7jb3bYYZ47yqcB84L/dwRXHAN8ESlV1GuB168WE3+thdFaStTiMMcNKNFscs4Byd7bAVmA5sLBLnYXAk+7ySmCeiIhbvlxVW1R1J8784rPcej4gWUR8QAqwL4rH0KtxOfYshzFmeIlm4hgD7Al7X+GWdVtHVQNALZDb07aquhe4B/gYZ5TeWlX9Q3cfLiI3iUiZiJRVVVX1w+F0b1xOCnsscRhjhpFB1TkuItk4rZEinJkIU925QY6jqo+oaqmqlubnR2/OqXE5qRxqaOVos42Sa4wZHqKZOPYCY8PeF7hl3dZxLz1l4kwQ1dO2lwE7VbVKVduAXwPnRyX6CI3Pde6s2lVtrQ5jzPAQzcSxHpgsIkUikoDTib2qS51VwHXu8iLgFXVmRloFLHbvuioCJgPrcC5RnSciKW5fyDxgaxSPoVdnjsoAYMt+G+zQGDM8RC1xuH0WtwBrcL7cV6jqZhG5U0SudKs9BuSKSDnO/B5L3W03AyuALcBLwDdUNaiqa3E60d8B3nfjfyRaxxCJ8TkppCX6eN9GyTXGDBMyHKY+LS0t1bKysqjtf8kjb1PfEuB3//SJqH2GMcYMNBHZoKqlXcsHVed4vCoencH2yqM2/7gxZliwxNEPCnNTaG4L2TSyxphhwRJHPzhvQi4Ab5RXxzgSY4yJPksc/WDSiDTSk3xssg5yY8wwYImjH4gI00ZnWuIwxgwLljj6ybQxGWw9cJS2YCjWoRhjTFRZ4ugn08Zk0hoIUV5ZH+tQjDEmqixx9JOScdkAvPJBZYwjMcaY6LLE0U/G5qQw5bR01u86FOtQjDEmqixx9KNpY5wOcnsQ0BgzlFni6EfTRmdQXd/KwbqWWIdijDFRY4mjH51VkAlgAx4aY4Y0Sxz9qHhUJh6xxGGMGdoscfSj5AQvk0ak2YOAxpghzRJHP2vvIDfGmKEqqolDROaLyDYRKReRpd2sTxSRZ931a0WkMGzdbW75NhG5Iqw8S0RWisgHIrJVROZE8xj66qwxmVQebaGyrjnWoRhjTFRELXGIiBd4CFgAFANLRKS4S7UbgMOqOgm4D7jb3bYYZ6rZqcB84L/d/QE8ALykqlOAs4nx1LFdTRtjHeTGmKEtmi2OWUC5qu5Q1VZgObCwS52FwJPu8kpgnjuX+EJguaq2qOpOoByYJSKZwEU4U86iqq2qeiSKx9BnxaMyEOsgN8YMYdFMHGOAPWHvK9yybuu4c5TXArkn2LYIqAKWicjfRORREUmNTvgnJzXRx6T8NN7dE1f5zBhj+s1g6xz3ASXAz1R1BtAAHNd3AiAiN4lImYiUVVVVDWSMnD8xl7d31NDcFhzQzzXGmIEQzcSxFxgb9r7ALeu2joj4gEyg5gTbVgAVqrrWLV+Jk0iOo6qPqGqpqpbm5+ef4qH0zSVnjKC5LcS6nTZulTFm6Ilm4lgPTBaRIhFJwOnsXtWlzirgOnd5EfCKOgM9rQIWu3ddFQGTgXWqegDYIyJnuNvMA7ZE8RhOynkTckn2e1mz+UCsQzHGmH4XtcTh9lncAqzBufNphapuFpE7ReRKt9pjQK6IlAO34l52UtXNwAqcpPAS8A1Vbb/u80/A0yKyETgH+I9oHcPJSk7wcumZI3hx0wFaAzaxkzFmaJHhMJJraWmplpWVDehnvrz1IDc8WcYj/zCTy6eeNqCfbYwx/UFENqhqadfywdY5PmhcdHo+eWmJrNxQEetQjDGmX1niiBK/18NnZozmlQ8qqam3YdaNMUOHJY4ourp0LIGQsqLMWh3GmKHDEkcUnT4ynekFmbzywcFYh2KMMf3GEkeUXTApj799fISGlkCsQzHGmH5hiSPKLpyURyCkvPxBZaxDMcaYfmGJI8rOm5BLUV4q96zZRig09G99NsYMfZY4oszjEb528QQ+PtTIsr/uinU4xhhzyixxDICrZowhNcHLb/7WdaguY4wZfCxxDIBEn5dbLp3M+3trbWZAY8ygZ4ljgFw4OQ+AFzbuj3EkxhhzaixxDJDiURkU5aXy4KvlBII28KExZvCyxDFAPB7hW5dN5lBDK+98bLMDGmMGL0scA2julBGkJ/l46q1dsQ7FGGNOmiWOAZSR5GfRzAL+sPmgDXxojBm0LHEMsCWzxtEaDPHcOzbwoTFmcIpq4hCR+SKyTUTKRWRpN+sTReRZd/1aESkMW3ebW75NRK7osp1XRP4mIi9EM/5oOH1kOucWZvPLtR/bk+TGmEEpaolDRLzAQ8ACoBhYIiLFXardABxW1UnAfcDd7rbFOHOUTwXmA//t7q/d/8KZjnZQuu78QnbVNPK7jftiHYoxxvRZNFscs4ByVd2hqq3AcmBhlzoLgSfd5ZXAPBERt3y5qrao6k6g3N0fIlIAfAp4NIqxR9Unp41iymnp/Ncr5QSt1WGMGWSimTjGAHvC3le4Zd3WUdUAUAvk9rLt/cA/Ayd8GEJEbhKRMhEpq6qqOtljiAqPR/j63EmUV9bzyGs7Yh2OMcb0yaDqHBeRTwOVqrqht7qq+oiqlqpqaX5+/gBE1zd/P30UC6adxr1/3MamvbWxDscYYyIWzcSxFxgb9r7ALeu2joj4gEyg5gTbXgBcKSK7cC59XSoi/xON4KNNRPiPz5xFdkoCn/6vN9hzqDHWIRljTESimTjWA5NFpEhEEnA6u1d1qbMKuM5dXgS8oqrqli9277oqAiYD61T1NlUtUNVCd3+vqOq1UTyGqMpOTeC+L5wDwL88txHn0I0xJr5FLXG4fRa3AGtw7oBaoaqbReROEbnSrfYYkCsi5cCtwFJ3283ACmAL8BLwDVUNRivWWLpgUh7/dtU0/vpRDY++vjPW4RhjTK9kOPyVW1paqmVlZbEOo0ehkPKVp8p4+YNK/vVTZ3LjhRNiHZIxxiAiG1S1tGv5oOocH6o8HuH+xedQMi6Lf//9Vn5ud1oZY+KYJY44kZ7k59mvzmFWUQ4/Wr2V17fH1y3ExhjTzhJHHPF7PSy7/lxOH5nGV3+xgQ27D8c6JGOMOY4ljjiTmujjf26YzYj0RK5fto4/bjkY65CMMaYTSxxxaERGEr/8ynmclpHEV54q4/MPv0VtU1uswzLGGMASR9wanZXMc18/n4LsZNbtOsSPV2+luS1oz3oYY2LOEkccy0jy88dvX8z15xeyfP0epnz/JRY9/BYH65pjHZoxZhizxBHnkhO83P7pYr44exwAG3YfZs6PX+beP2yz1ocxJibsAcBBJBRS3t9by4OvlvPHLQcZlZnE0zfOZkJ+WqxDM8YMQfYA4BDg8Qhnj83iZ9eUcPXMAvbXNvOlJ9ZTXnk01qEZY4YRSxyDkM/r4SdXn80vvzKb3TWNXHbva9z4ZBnv7TkS69CMMcOAJY5B7PyJeaz46hyumDqSN8urWfjQm3zx52/zh80Heuz/qG8J0Nx26uNFtgSClFcetX4WY4Yh6+MYIirrmvm/f/iQ323cR2NrEL9XKMxNJT89kSS/l6wUP40tQV7afACAH145lSvPHk12akK3+2toCfCLt3fz149qKBmXRUO2DkAAABVESURBVHqSn9lFOeSmJfDTl7fzzDpngkYRmDMhl+996kzG56YSCIbITPZ3TIn7/t5aHnq1nNMyk/hcSQGb9tZSPDqTcTkp5KUl4MwUbIyJRz31cVjiGGICwRDL3tzFCxv38V5F55kF89ISKB2fw/t7a9l7pAmAguxkJuansWDaaRyoa6Zs12EKspNZUbaH3qZDT0nwMnlE2nGfk5boo74l0Gus6Uk+zpuQyzWzx3HR5Hw8ns5JRFX5254j5KclUpCd3Ock09QaJMnviavkpKo8/JcdbNlfx1XnjGbuGSOOO25j4oUljmGSOMI5l5PqOWNkOuD0jYBzd9bbO2t44E/bOdzYypHGNiqPthy3/d+fPZrbP13Mwbpm3iyvZuv+Ol7afICHvljCpVNGdHwhVx1t4dHXd3CooZWsFD9tQaW6voX0JB/jc1OZXZRDcoKXP205SE1DKxPy0wgEQ7y35wi/eXcfAJNGpFE8KoOahhZ8Hg8h9//L17dXAzAxP5U5E3OZcloGM8ZlMXV0JqGQ8vIHlXxUVc+5hTkEQ0p9SxsfHDjKC+/tZ8v+OgCS/V58XmHuGSO49rzxiEDJuGy8HgFVCLSAPymqv4tNbstr075a9hxq6ii/+PR8vnrRBOZMzI2rBGcMWOIYXIlj91vw/goIBUGDEAqBhtxlt0xDbnl7WdflUNj2JyoLoqEQLa2ttAYCpCV4COBDfckkJqeBP7nTS/3JiD/FfZ8CviTnZ/v7LvWPW+dLBs+xrrXdNQ2s3XmIx9/YybaDRxFgTHZyx5freRNyuGhSNs+++QFNDfWkSDMptDBzVCK7DlSRQgvJtJAiLSTjrEuRFpJoZXRKiGx/G8k009xQjz/U5NSlhVRPC6nSSqI6D1PWSQaVieNozJhIekExbxzJoTppPKPHn86YnHRmjs9mR3U9dU0BWgJBdlY3MDE/jZnjs0lN9HX7a2wJBGlqDbK/tpnrHl9H5dEWpo7OYM6EXK6/oJDV7+/n7pe2EQwpf1c8km/MncTZBZmWQEzciEniEJH5wAOAF3hUVe/qsj4ReAqYiTPX+BdUdZe77jbgBiAIfFNV14jIWLf+SECBR1T1gd7iGHSJ473l8Id/BfGAeMHjdZc97nIPZZ3We7ps3/5TOpd1bBP2M9gKbc3Q1gRtjZ1/Bpo7l50Mb2K3iUY9PqS1AdoaUfentDZCqG/jdAXFjySk4klIgYQU53MSUqkN+qkN+Al6U6hoELbWBGkikZTkZHKDlYwJ7GGi7CNP6jr21ax+duhoPtJRlIfG8JGOplzHsFNPo4UEEn0eFs0s4Iuzx5GbmshTb+1izeYD7DvSTFPYTQgJPg+//cYFnDkqo1OshxpaefT1HTz+5k6a20KcPjKN8yfmcbQ5QH1LG3lpiSgw94wRnD4yjfG5qQC0BUPcuuI9fvfePrwe4awxmWQm+1l87liS/F4O1jVz5qgMikdn4Pd2vgemqTVIWyhEit+LR8QulZkeDXjiEBEv8CHwd0AFzhzkS1R1S1idrwPTVfVrIrIY+IyqfkFEioFngFnAaOBPwOnACGCUqr4jIunABuCq8H12Z9AljsFC1U0kTWGvsKTSKck0dVnu8j7QBME2SEh1v+hTwJ967Kc/uUtZCupPQRJSu2yTAl5/ROEHgiFC6nypA9Q2trH1QB1vbPyQqwrqyWjYCVUfQvWHeGq2k9O2Hw/OvxdFaE4rYFtgNOvr89iuozsSSy1p+DzCJybnkZeWSNXRFm6+ZCLnTcjtMZb9tU2sLKvgmXUfs6/WaQVlJvuPG9wy2e/FI5CVksDeI01kpfhpbgsSUmgNhI7bb0qCl4LsZD5XUsCumgbW7jhE1dEWjrp9UIk+Dxefns8/zz+DRJ+XlRsquHTKCM4em+Wck6Y2fvbnj9hRVc8lZ4xgyayxiAitgRDllfUcrGt2L0v6OX9SLhlJzo0RHx9qpLktyD1rtvHJs0Zx6ZQRZKX4CSkIWLIaJGKROOYAd6jqFe772wBU9cdhdda4dd4SER9wAMjn2NzjP+5ar8tn/BZ4UFX/eKJYLHGYftHWBDXlUP2hm1C2QfV2tHo7EjzWRxRKycOTfwbknQ75Z0DeZMg7AzLGdLpM15PG1gCHGloZlZnMtgNHKcpL5S8fVrJpbx3PvVPBqMwkmttCfG5mAV86vxCPR1BVapva+PhQI02tQTweYUdVPS9tOsDmfXWd+rDG56YweUQ6LYEgyX4vr2+v7tQ6ajc2J5lQiI4bKdqdluH0Bx3oMmZaeqKPkCoNrc6+2q+4hX/FeARCCucWZnPR5Hwm5KcxqyiH/PREVJW1Ow9xuKGV4tEZrNt5iHf3HMHv9ZCW6GNXTQPjc1P44uzxjMlK7vU8mlMXi8SxCJivqje67/8BmK2qt4TV2eTWqXDffwTMBu4A3lbV/3HLHwNeVNWVYdsWAq8B01T12LWFY+tvAm4CGDdu3Mzdu3dH4SiNwekvOvKxm1C2OT/bl5vDHsr0p0LeJCeJ5J0O+adD2kgIBZzWVsfP9uWAs9y+Lnx9+7putw2G1WsjFGyjuaWVBE+Q1pYWkr2KaAgS0yApiwZPOu9VKy3+DA4HU9h+1Mf7NUJyei7NvjS++alzmTF5PE+v38tfP6rmQF0LNfUtXD1zLF4PzJ6Qy3t7jvD2jkN8VFVPTmoCDS0BSsZnc/35hew51MiLmw6ws9r54q862kJ5ZT37a48lnnE5KdS3OAkznNcjJPk8NLQGSfB6CIScVtXU0ZlMGpHGBZPyCIZC1LcEeWf3YZYumELQHZrnnLFZ5KUlsvVAHc9tqGDBtFHkpSeQkeQnNy0Bj8hxl/HaNbcFqTraQpLfS3KCl7SwfqzG1gA+j4cEn4dQSNmyv476lgAH65qZNiaTCXmpQ6afakglDhFJA/4C/EhVf91bLNbiMDGhCg3Vbsvkw47LXlR/CLV7+uczPH7n0pzHD14feHxhy37nffuy133v8Tl9Wa310HTESW5NR3rvS0rMgKQsSMqE5PCf7iu5u5+ZzrKv8/NCqsrO6gYON7Zy5++28F5FLVNOS2fq6EyumDqSndUNAFx73ngSfR721zZTkJ3Mvtpmlr2xkz9tPUjF4SYCvdwzLtK5xdNVXloiaYle5kzMpaa+lU9NH0VeWiLXPLq2U71Lp4xgZEYihxvaeGVbJRlJfqacls7anTW0BY//gCS/h9zURG68sIhzC3NoC4bYVdNAZV0LB+qamTk+m5Jx2VQdbcHnFe5+aRsf7K9j5vhsikdlMHlkOpdOGdFxGbX9nIUnpLU7alj66/cJhpRJI9IozE3lqhmjGZOVTG5a4gnPS6SGzKUqEfEDLwBrVPXeSGKxxGHiTks91GyHxkNhX+j+7r/k2xODxxuWJPzuzQ/99JetqnMprj2JNB+B5trOiaXbslpnubcbJfwpx5KJPwV8ic4xeBNQrx/xJoA3oaPs+Jf/uOVW8bP5QCOZ6WmkJCVT1RiirKKerPRUCkdm81FNCwcalNSUZC6eMoaPDrUQUqHqaDN7jjSTmuBje2UdH1Y2sPewczlOkY6fRXkpXDg5nz9treRgXTNtIRiVkcTsCblUHm3ho8p6xuSkcFpGEpdPPY2MZD9b9tfx5keHONLQyo7qho7LdooQQtz99/w7C38GKsVt6QRDSoLPQ9VR5xb3+hanxdN+eXFifip7jzTR3Hasj2tkRiIzx2dzy9zJFI/O6PazIhGLxOHD6RyfB+zF6Rz/oqpuDqvzDeCssM7xz6rq50VkKvBLjnWOvwxMBkLAk8AhVf1WpLFY4jAmygItbhLpLtkc6VzW1uTcuRdsc3+Gv9o6LwdanNvHhxhFUDyoOIlEPB7EvUsyBARVCIYgiBAMKR6vF494CCpuHcHr8ZKa5MMjXhBnm4bWEEGEtqBytCWIz+tl/Hc3OIn6JPSUOLq/Ab0fqGpARG4B1uDcjvu4qm4WkTuBMlVdBTwG/EJEyoFDwGJ3280isgLYAgSAb6hqUEQ+AfwD8L6IvOt+1HdVdXW0jsMYEwFfIqSNcF79rb3Ppsdk00PC6a4cgLA/ljv94axRKnP/o+o8O4XTxyTt790yZ1nxquINe991fedtju3Xi5IRViczEKS5tc1pmfYzewDQGGNMt2w+DmOMMf3CEocxxpg+scRhjDGmTyxxGGOM6RNLHMYYY/rEEocxxpg+scRhjDGmTyxxGGOM6ZNh8QCgiFQBJzs8bh5Q3Y/hDKTBHDtY/LE0mGOHwR1/PMU+XlXzuxYOi8RxKkSkrLsnJweDwRw7WPyxNJhjh8Ed/2CI3S5VGWOM6RNLHMYYY/rEEkfvHol1AKdgMMcOFn8sDebYYXDHH/exWx+HMcaYPrEWhzHGmD6xxGGMMaZPLHH0QETmi8g2ESkXkaWxjqc7IjJWRF4VkS0isllE/pdbniMifxSR7e7PbLdcROSn7jFtFJGS2B4BiIhXRP4mIi+474tEZK0b47MikuCWJ7rvy931hbGM240pS0RWisgHIrJVROYMlnMvIt92/5/ZJCLPiEhSPJ97EXlcRCpFZFNYWZ/PtYhc59bfLiLXxTj+n7j/72wUkedFJCts3W1u/NtE5Iqw8vj4XlJVe3V54Ux1+xEwAUgA3gOKYx1XN3GOAkrc5XScOd6Lgf8ElrrlS4G73eVPAi8CApwHrI2DY7gVZ375F9z3K4DF7vLDwM3u8teBh93lxcCzcRD7k8CN7nICkDUYzj0wBtgJJIed8+vj+dwDFwElwKawsj6dayAH2OH+zHaXs2MY/+WAz12+Oyz+Yvc7JxEocr+LvPH0vRST/3Hj/QXMAdaEvb8NuC3WcUUQ92+BvwO2AaPcslHANnf5/wFLwup31ItRvAXAy8ClwAvuP/TqsH9MHb8HnLnr57jLPreexDD2TPfLV7qUx/25dxPHHvcL1Oee+yvi/dwDhV2+ePt0roElwP8LK+9Ub6Dj77LuM8DT7nKn75v28x9P30t2qap77f+w2lW4ZXHLvXwwA1gLjFTV/e6qA8BIdznejut+4J+BkPs+FziiqgH3fXh8HbG762vd+rFSBFQBy9xLbY+KSCqD4Nyr6l7gHuBjYD/OudzA4Dn37fp6ruPmd9CNL+O0kmAQxG+JYwgQkTTgOeBbqloXvk6dP03i7p5rEfk0UKmqG2Idy0ny4Vx6+JmqzgAacC6XdIjjc58NLMRJfqOBVGB+TIM6RfF6riMhIt8DAsDTsY4lUpY4urcXGBv2vsAtizsi4sdJGk+r6q/d4oMiMspdPwqodMvj6bguAK4UkV3AcpzLVQ8AWSLic+uEx9cRu7s+E6gZyIC7qAAqVHWt+34lTiIZDOf+MmCnqlapahvwa5zfx2A59+36eq7j6XcAgIhcD3wauMZNfjAI4rfE0b31wGT3LpMEnA7BVTGO6TgiIsBjwFZVvTds1Sqg/Y6R63D6PtrL/9G96+Q8oDasqT+gVPU2VS1Q1UKc8/uKql4DvAoscqt1jb39mBa59WP2F6aqHgD2iMgZbtE8YAuD4NzjXKI6T0RS3P+H2mMfFOc+TF/P9RrgchHJdltdl7tlMSEi83Eu1V6pqo1hq1YBi9272YqAycA64ul7KRYdK4PhhXNnxoc4dzF8L9bx9BDjJ3Ca5xuBd93XJ3GuP78MbAf+BOS49QV4yD2m94HSWB+DG9clHLuragLOP5Jy4FdAolue5L4vd9dPiIO4zwHK3PP/G5w7dQbFuQd+CHwAbAJ+gXMHT9yee+AZnP6YNpzW3g0nc65x+hLK3deXYhx/OU6fRfu/3YfD6n/PjX8bsCCsPC6+l2zIEWOMMX1il6qMMcb0iSUOY4wxfWKJwxhjTJ9Y4jDGGNMnljiMMcb0iSUOMySISFBE3hWR90TkHRE5v5f6WSLy9Qj2+2cRKe2/SAc/EXlCRBb1XtMMVZY4zFDRpKrnqOrZOIO//biX+lk4o77GpbAnuI2JO5Y4zFCUARwGZxwvEXnZbYW8LyIL3Tp3ARPdVspP3Lr/4tZ5T0TuCtvf1SKyTkQ+FJEL3bpedz6F9e58Cl91y0eJyGvufje11w8nIrtE5G53n+tEZJJb/oSI3CsirwJ3izPfxG/c/b8tItPDjmmZG+tGEfmcW365iLzlHuuv3DHMEJG7xJmzZaOI3OOWXe3G956IvNbLMYmIPOju4/fAiP78ZZnBx/6qMUNFsoi8i/OU8yicsa8AmoHPqGqdiOQBb4vIKpwBCaep6jkAIrIAZ+C/2araKCI5Yfv2qeosEfkk8AOcsZ5uwBnK4lwRSQTeFJE/AJ/FGfr6RyLiBVJ6iLfO3ec/4owS/Gm3/HTgMlUNish/AX9T1atE5FLgKZyn1b/vfvZZbuzZ7rH9q7ttg4j8C3CriDyEM2T3FFVVOTZZ0O3AFaq6N6ysp2OaAZwBnIUzAu0W4PGIfitmSLLEYYaKprAkMAd4SkSm4Qw/8R8ichHO8O1jODb8drjLgGXqjhmkqofC1rUPHrkBZ04FcMY5mh52rT8TZ0yh9cDj4gw++RtVfbeHeJ8J+3lfWPmvVDXoLn8C+JwbzysikisiGW6si9s3UNXD4ow2XIzzZQ/ORD9v4QyB3gw8Js4siy+4m70JPCEiK8KOr6djugh4xo1rn4i80sMxmWHCEocZclT1Lfcv8HycsX3ygZmq2ibOaLxJfdxli/szyLF/MwL8k6oeN0iem6Q+BfxCRH6iqk91F2YPyw19jK3jY4E/quqSbuKZhTOQ4WLgFuBSVf2aiMx243xXRM7p6ZjclpYxHayPwww5IjIFZ5rNGpy/mivdpDEXGO9WO4oz3W67PwJfEpEUdx/hl6q6swa42W1ZICKni0iqiIwHDqrqz3FGLu5pbvEvhP18q4c6rwPXuPu/BKhWZ76VPwLfCDvebOBt4IKw/pJUN6Y0IFNVVwPfwrnUhYhMVNW1qno7zox+Y3s6JuA14AtuH8goYG4v58YMcdbiMENFex8HOH85X+f2EzwN/E5EynBGIP0AQFVrRORNEdkEvKiq/8f9q7tMRFqB1cB3T/B5j+JctnpHnGtDVcBVOCP9/h8RaQPqgX/sYftEEVmL88fbca0E1x04l702Ao0cG0L834GH3NiDwA9V9dfizO3wjNs/AU6fx1HgtyKS5J6Xb7vrfiIik92yl3Hmr97YwzE9j9Nn9D7OyKx/OcF5McOAjY5rzABzL5eVqmp1rGMx5mTYpSpjjDF9Yi0OY4wxfWItDmOMMX1iicMYY0yfWOIwxhjTJ5Y4jDHG9IklDmOMMX3y/wEs8dItqFolrgAAAABJRU5ErkJggg==\">"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Finally the `fcn.show_results()` or generally the `model.show_results()` for both the `MLModel` and the `FullyConnectedNetwork` model will print the training results with the prediction saved in the column named `prediction_results` of the dataframe and will let the user assess the prediction on the test set by the trained model. This can also be determined by the model metrics of the trained model which is estimated similarly for the two models by the `model.score()` method and returns R-square score for regression model and accuracy for classification model.\n", "\n", "The trained model can now be saved using the `model.save(name_or_path)` command, which will save the model weights and create an `Esri Model Definition and Deep Learning Package` or `DLPK` zip. For example:"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["fcn.save(r'C:\\2020\\5-May\\solar energy\\saved_model\\trained_model', publish=True)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The `publish` parameter is an optional boolean which publishes the DLPK as an item in Arcgis portal. Another optional parameter is `framework`, which by default saves the model with a PyTorch backend(`framework='PyTorch'`). It can be used to also save the model with a TF-ONNX backend(`framework='TF-ONNX'`).    \n", "\n", "The saved model can be loaded back using `model.from_model(emd_path, data=None)` or here by using `fcn.from_model(emd_path, data=None)` or by `MLModel.from_model(emd_path, data=None)`, which will load the trained model from the provided `emd_path` and would be ready for inferencing. The `data` parameter is optional and requires a fastai Databunch returned  by the `prepare_tabulardata` function, and is only for further training the model on a new dataset."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Model Inferencing <a class=\"anchor\" id=\"12\"></a> "]}, {"cell_type": "markdown", "metadata": {}, "source": ["### Predicting to features\n", "\n", "With a model that has been trained using any combination of `Explanatory Variables`, `distance Features`, and `Explanatory Training Rasters`, it can be used to predict to either points or polygons in either the same or a different study area. Predicting to features requires that every feature receiving a prediction have values for each field, distance feature, and raster provided.\n", "\n", "If the Input Training Features and Input Prediction Features field names do not match, a `match_field_names` parameter is provided. When matching explanatory variables, the Prediction and Training fields must be of the same type (a double field in Training must be matched to a double field in Prediction).\n", "\n", "### Predicting to rasters\n", "\n", "Using a trained model, you can choose to predict to a raster in either the same or a different study area. If you want to use prediction rasters other than those used for training the model because you are predicting either in a different study area or a different time period, add raster names mapping in `match_field_names` parameter."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Summary <a class=\"anchor\" id=\"13\"></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Here in this example we have used only one classifier model which is the gradient boosting classifier. As we know there are several other models available in scikit-learn, and the `MLModel()` method provides the flexibility of choosing any one of them with its respective hyperparameters to initialize the model. Subsequently using model selection, the best model can be selected along with the optimal hyperparameters.  "]}, {"cell_type": "markdown", "metadata": {}, "source": ["Both the `FullyConnectedNetwork` and `MLModel` discussed above has their own set of pro and cons, and a brief comparison is as follows:"]}, {"cell_type": "markdown", "metadata": {}, "source": ["| `FullyConnectedNetwork`                                                                                                             | `MLModel`                                                                                                                                                                                                                    |\n", "|-------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n", "| Usually requires large dataset                                                                                                      | Performs well with comparatively fewer data points                                                                                                                                                                           |\n", "| Returns different result for each run due to random weight initialization                                                           | More stable than neural network, and usually returns the same result except in some cases which depends on the data used for the model.                                                                                      |\n", "| It has some universal parameters for fine-tuning, like the number of hidden layers, learning rate, number of epochs, batch size etc | Several scikit-learn models are available, and each need its own set of unique parameters & hyperparameters setting, followed by fine-tuning. Ideally should be followed by model selection for selecting the optimal model. |"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Implementation of the `FullyConnectedNetwork` and `MLModel` architecture can be better understood in one of the sample notebooks where energy generation from solar photovoltaic power plants using weather variables has been predicted using both models and is available in this [repository for GIS analysts and data scientists](https://developers.arcgis.com/python/sample-notebooks/), along with other interesting notebooks based on the various modules from ArcGIS API for Python.   "]}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.10"}, "toc": {"base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": false, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": {}, "toc_section_display": true, "toc_window_display": false}}, "nbformat": 4, "nbformat_minor": 4}