legal-nlp/04.1.Training_Legal_Binary_Classifier.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "KLqW6FOnEvov",
      "metadata": {
        "id": "KLqW6FOnEvov"
      },
      "source": [
        "# Training Binary Classification Models with Legal NLP"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wxZDXLDCXkk_",
      "metadata": {
        "id": "wxZDXLDCXkk_"
      },
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pZ6sKi8ZX1z4",
      "metadata": {
        "id": "pZ6sKi8ZX1z4"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/legal-nlp/04.1.Training_Legal_Binary_Classifier.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "245d53bb",
      "metadata": {
        "id": "245d53bb"
      },
      "source": [
        "In this notebook, you will learn how to use Spark NLP and Legal NLP to train binary classification models.\n",
        "\n",
        "Let`s dive in!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gk3kZHmNj51v",
      "metadata": {
        "collapsed": false,
        "id": "gk3kZHmNj51v"
      },
      "source": [
        "#🎬 Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_914itZsj51v",
      "metadata": {
        "id": "_914itZsj51v",
        "pycharm": {
          "is_executing": true
        }
      },
      "outputs": [],
      "source": [
        "! pip install -q johnsnowlabs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YPsbAnNoPt0Z",
      "metadata": {
        "id": "YPsbAnNoPt0Z"
      },
      "source": [
        "##🔗 Automatic Installation\n",
        "Using my.johnsnowlabs.com SSO"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fY0lcShkj51w",
      "metadata": {
        "id": "fY0lcShkj51w",
        "pycharm": {
          "is_executing": true
        }
      },
      "outputs": [],
      "source": [
        "from johnsnowlabs import nlp, legal\n",
        "\n",
        "# nlp.install(force_browser=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hsJvn_WWM2GL",
      "metadata": {
        "id": "hsJvn_WWM2GL"
      },
      "source": [
        "##🔗 Manual downloading\n",
        "If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.\n",
        "\n",
        "- Go to my.johnsnowlabs.com\n",
        "- Download your license\n",
        "- Upload it using the following command"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "i57QV3-_P2sQ",
      "metadata": {
        "id": "i57QV3-_P2sQ"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "print('Please Upload your John Snow Labs License using the button below')\n",
        "license_keys = files.upload()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xGgNdFzZP_hQ",
      "metadata": {
        "id": "xGgNdFzZP_hQ"
      },
      "source": [
        "- Install it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "OfmmPqknP4rR",
      "metadata": {
        "id": "OfmmPqknP4rR"
      },
      "outputs": [],
      "source": [
        "nlp.install()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DCl5ErZkNNLk",
      "metadata": {
        "id": "DCl5ErZkNNLk"
      },
      "source": [
        "#📌 Starting"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wRXTnNl3j51w",
      "metadata": {
        "id": "wRXTnNl3j51w"
      },
      "outputs": [],
      "source": [
        "spark = nlp.start()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e9d3bac3",
      "metadata": {
        "id": "e9d3bac3"
      },
      "source": [
        "##📜 Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fd9e23a6",
      "metadata": {
        "id": "fd9e23a6"
      },
      "source": [
        "📚For the text classification tasks, we will use two annotators:\n",
        "\n",
        "- `MultiClassifierDL`: `Multilabel Classification` (can predict more than one class for each text) using a Bidirectional GRU with Convolution architecture built with TensorFlow that supports up to 100 classes. The inputs are Sentence Embeddings such as state-of-the-art UniversalSentenceEncoder, BertSentenceEmbeddings or SentenceEmbeddings.\n",
        "- `ClassifierDL`: uses the state-of-the-art Universal Sentence Encoder as an input for text classifications. Then, a deep learning model (DNNs) built with TensorFlow that supports `Binary Classification` and `Multiclass Classification` (up to 100 classes)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "19d62bec",
      "metadata": {
        "id": "19d62bec"
      },
      "source": [
        "Since this model can takes longer time to train, we will limit (reduce) the size of the training data to avoid having it training for hours. \n",
        "\n",
        "> Please note that this reduction can greatly impact the performance of the model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5qFM3Zueb50G",
      "metadata": {
        "id": "5qFM3Zueb50G"
      },
      "source": [
        "##Loading the data\n",
        "\n",
        "▒▒▒▒▒▒▒▒▒▒ 100% ᴄᴏᴍᴘʟᴇᴛᴇ! "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0db04dd2",
      "metadata": {
        "id": "0db04dd2"
      },
      "source": [
        "Since the deep learning models can take some time to train, we will limit our dataset to a smaller number of observations in order to illustrate how to use Spark NLP and Finance NLP annotators and pipelines to train the model, but without having to wait too much.\n",
        "\n",
        "Please note that the quality and the quantity of training data is very relevant to the obtained trianed model, and the results we obtain here are for illustration purposes only. To obtain a more realistic model, pelase consider using the full dataset or addin extra observations from different sources. \n",
        "\n",
        "Here we will use very small sample. We will train a sample model to classify if a cluse relevant `ti-allowance` or `other` in legal documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "NcP97dzbM7yn",
      "metadata": {
        "id": "NcP97dzbM7yn"
      },
      "outputs": [],
      "source": [
        "! wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/legal-nlp/data/legal_clf.csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "FtXdRXNEb50G",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FtXdRXNEb50G",
        "outputId": "f792952e-0250-4068-c1a5-84e3439b804d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of the full dataset: (254, 2)\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('legal_clf.csv', encoding=\"utf8\")\n",
        "print(f\"Shape of the full dataset: {df.shape}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "SWqOIj_pfEU5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "SWqOIj_pfEU5",
        "outputId": "726e1b2c-e4e1-4f1d-eddb-1e1c85d7b54b"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-74735e02-dd2a-4490-8c4e-cfa03dbdec7c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>meal-allowance Subject to the terms and provis...</td>\n",
              "      <td>ti-allowance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>construction-of-the-tenant-improvements Tenant...</td>\n",
              "      <td>ti-allowance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Tenant Improvements Lessor, at Lessor’s cost, ...</td>\n",
              "      <td>ti-allowance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Provided there shall not be existing a default...</td>\n",
              "      <td>ti-allowance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Landlord shall provide Tenant a tenant improve...</td>\n",
              "      <td>ti-allowance</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-74735e02-dd2a-4490-8c4e-cfa03dbdec7c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-74735e02-dd2a-4490-8c4e-cfa03dbdec7c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-74735e02-dd2a-4490-8c4e-cfa03dbdec7c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                                text      category\n",
              "0  meal-allowance Subject to the terms and provis...  ti-allowance\n",
              "1  construction-of-the-tenant-improvements Tenant...  ti-allowance\n",
              "2  Tenant Improvements Lessor, at Lessor’s cost, ...  ti-allowance\n",
              "3  Provided there shall not be existing a default...  ti-allowance\n",
              "4  Landlord shall provide Tenant a tenant improve...  ti-allowance"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Ntfw_obpuf4A",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ntfw_obpuf4A",
        "outputId": "ea3bc784-2131-491b-b5bb-de90b33ac540"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "other           135\n",
              "ti-allowance    119\n",
              "Name: category, dtype: int64"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df['category'].value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hrFio0xccT7J",
      "metadata": {
        "id": "hrFio0xccT7J"
      },
      "source": [
        " ## 🔎 With Bert Sentence Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "o0muwDJ5cT7K",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o0muwDJ5cT7K",
        "outputId": "9e601a25-96e9-4c97-e5f3-9f2327d27ebf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "sent_bert_base_cased download started this may take some time.\n",
            "Approximate size to download 389.1 MB\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "documentAssembler = nlp.DocumentAssembler() \\\n",
        "     .setInputCol(\"text\") \\\n",
        "     .setOutputCol(\"document\")\n",
        "  \n",
        "embeddings = nlp.BertSentenceEmbeddings.pretrained(\"sent_bert_base_cased\", \"en\") \\\n",
        "    .setInputCols(\"document\") \\\n",
        "    .setOutputCol(\"sentence_embeddings\")\n",
        "\n",
        "docClassifier = legal.ClassifierDLApproach()\\\n",
        "    .setInputCols([\"sentence_embeddings\"])\\\n",
        "    .setOutputCol(\"class\") \\\n",
        "    .setLabelColumn(\"category\") \\\n",
        "    .setBatchSize(64) \\\n",
        "    .setMaxEpochs(32) \\\n",
        "    .setEnableOutputLogs(True)\\\n",
        "    .setOutputLogsPath(\"binary_bert_logs\")\\\n",
        "    .setLr(0.002)\\\n",
        "    .setRandomSeed(0)\\\n",
        "    .setDropout(0.2)\n",
        "\n",
        "nlpPipeline = nlp.Pipeline(stages=[\n",
        "    documentAssembler,\n",
        "    embeddings,\n",
        "    docClassifier\n",
        "    ])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Fa29Qw5cUp8j",
      "metadata": {
        "id": "Fa29Qw5cUp8j"
      },
      "outputs": [],
      "source": [
        "spark_df = spark.createDataFrame(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9Vb2VqzucT7M",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9Vb2VqzucT7M",
        "outputId": "51075280-21ba-4566-fc2c-2dd27da3081c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 1.24 s, sys: 188 ms, total: 1.43 s\n",
            "Wall time: 3min 36s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "# splitting dataset into train and test set\n",
        "train, test = spark_df.randomSplit([0.90, 0.10], seed = 0)\n",
        "\n",
        "clf_model = nlpPipeline.fit(train)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "NdAu3IE1cT7N",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NdAu3IE1cT7N",
        "outputId": "b36a68c4-f0ba-423d-ad34-729b7e9b443b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training started - epochs: 32 - learning_rate: 0.002 - batch_size: 64 - training_examples: 232 - classes: 2\n",
            "Epoch 0/32 - 0.30s - loss: 3.2437663 - acc: 0.61145836 - batches: 4\n",
            "Epoch 1/32 - 0.08s - loss: 2.268199 - acc: 1.0 - batches: 4\n",
            "Epoch 2/32 - 0.09s - loss: 1.4406782 - acc: 1.0 - batches: 4\n",
            "Epoch 3/32 - 0.08s - loss: 1.3018677 - acc: 1.0 - batches: 4\n",
            "Epoch 4/32 - 0.08s - loss: 1.2646831 - acc: 1.0 - batches: 4\n",
            "Epoch 5/32 - 0.08s - loss: 1.2572342 - acc: 1.0 - batches: 4\n",
            "Epoch 6/32 - 0.06s - loss: 1.2554077 - acc: 1.0 - batches: 4\n",
            "Epoch 7/32 - 0.06s - loss: 1.2545748 - acc: 1.0 - batches: 4\n",
            "Epoch 8/32 - 0.06s - loss: 1.2542142 - acc: 1.0 - batches: 4\n",
            "Epoch 9/32 - 0.07s - loss: 1.2540405 - acc: 1.0 - batches: 4\n",
            "Epoch 10/32 - 0.09s - loss: 1.2539155 - acc: 1.0 - batches: 4\n",
            "Epoch 11/32 - 0.08s - loss: 1.2538443 - acc: 1.0 - batches: 4\n",
            "Epoch 12/32 - 0.08s - loss: 1.2537851 - acc: 1.0 - batches: 4\n",
            "Epoch 13/32 - 0.07s - loss: 1.2536979 - acc: 1.0 - batches: 4\n",
            "Epoch 14/32 - 0.08s - loss: 1.2536134 - acc: 1.0 - batches: 4\n",
            "Epoch 15/32 - 0.07s - loss: 1.253551 - acc: 1.0 - batches: 4\n",
            "Epoch 16/32 - 0.07s - loss: 1.2535069 - acc: 1.0 - batches: 4\n",
            "Epoch 17/32 - 0.06s - loss: 1.2534728 - acc: 1.0 - batches: 4\n",
            "Epoch 18/32 - 0.07s - loss: 1.2534441 - acc: 1.0 - batches: 4\n",
            "Epoch 19/32 - 0.07s - loss: 1.2534189 - acc: 1.0 - batches: 4\n",
            "Epoch 20/32 - 0.06s - loss: 1.2533972 - acc: 1.0 - batches: 4\n",
            "Epoch 21/32 - 0.06s - loss: 1.2533789 - acc: 1.0 - batches: 4\n",
            "Epoch 22/32 - 0.06s - loss: 1.2533635 - acc: 1.0 - batches: 4\n",
            "Epoch 23/32 - 0.08s - loss: 1.2533501 - acc: 1.0 - batches: 4\n",
            "Epoch 24/32 - 0.08s - loss: 1.2533383 - acc: 1.0 - batches: 4\n",
            "Epoch 25/32 - 0.06s - loss: 1.2533277 - acc: 1.0 - batches: 4\n",
            "Epoch 26/32 - 0.06s - loss: 1.253318 - acc: 1.0 - batches: 4\n",
            "Epoch 27/32 - 0.06s - loss: 1.2533091 - acc: 1.0 - batches: 4\n",
            "Epoch 28/32 - 0.08s - loss: 1.2533004 - acc: 1.0 - batches: 4\n",
            "Epoch 29/32 - 0.07s - loss: 1.2532928 - acc: 1.0 - batches: 4\n",
            "Epoch 30/32 - 0.06s - loss: 1.2532855 - acc: 1.0 - batches: 4\n",
            "Epoch 31/32 - 0.07s - loss: 1.2532789 - acc: 1.0 - batches: 4\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "log_file_name = os.listdir(\"binary_bert_logs\")[0]\n",
        "\n",
        "with open(\"binary_bert_logs/\"+log_file_name, \"r\") as log_file :\n",
        "    print(log_file.read())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "37AVLj_Keye_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "37AVLj_Keye_",
        "outputId": "762392d5-66bf-49a5-a5cf-c79869261759"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-b16a52e4-2559-4094-bcff-4a4c19f51e3d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>category</th>\n",
              "      <th>text</th>\n",
              "      <th>result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Disbursement of Tenant Improvement Allowance N...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>In consideration of the foregoing, the parties...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Landlord shall make available for use by Tenan...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Landlord shall provide Tenant the Tenant Impro...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Landlord, at Landlord’s sole cost and expense,...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b16a52e4-2559-4094-bcff-4a4c19f51e3d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b16a52e4-2559-4094-bcff-4a4c19f51e3d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b16a52e4-2559-4094-bcff-4a4c19f51e3d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "       category                                               text  \\\n",
              "0  ti-allowance  Disbursement of Tenant Improvement Allowance N...   \n",
              "1  ti-allowance  In consideration of the foregoing, the parties...   \n",
              "2  ti-allowance  Landlord shall make available for use by Tenan...   \n",
              "3  ti-allowance  Landlord shall provide Tenant the Tenant Impro...   \n",
              "4  ti-allowance  Landlord, at Landlord’s sole cost and expense,...   \n",
              "\n",
              "           result  \n",
              "0  [ti-allowance]  \n",
              "1  [ti-allowance]  \n",
              "2  [ti-allowance]  \n",
              "3  [ti-allowance]  \n",
              "4  [ti-allowance]  "
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "preds = clf_model.transform(test)\n",
        "\n",
        "preds_df = preds.select('category','text',\"class.result\").toPandas()\n",
        "\n",
        "preds_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Oo0X1tyacT7P",
      "metadata": {
        "id": "Oo0X1tyacT7P"
      },
      "outputs": [],
      "source": [
        "# The result is an array since in Spark NLP you can have multiple sentences.\n",
        "# Let's explode the array and get the item(s) inside of result column out\n",
        "preds_df['result'] = preds_df['result'].apply(lambda x : x[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VOvxSG6Afz5d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VOvxSG6Afz5d",
        "outputId": "878c8800-4ec5-4857-ebc4-aae9d299117f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       other       1.00      1.00      1.00         9\n",
            "ti-allowance       1.00      1.00      1.00        13\n",
            "\n",
            "    accuracy                           1.00        22\n",
            "   macro avg       1.00      1.00      1.00        22\n",
            "weighted avg       1.00      1.00      1.00        22\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# We are going to use sklearn to evalute the results on test dataset\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "print(classification_report(preds_df[\"category\"], preds_df[\"result\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "IHzIfMSuUgiD",
      "metadata": {
        "id": "IHzIfMSuUgiD"
      },
      "source": [
        "##📌 Saving & loading back the trained model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "r5Tec2WwUhP1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r5Tec2WwUhP1",
        "outputId": "8e3ce118-9aa6-4e7c-d74d-552129f1bedc"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[DocumentAssembler_35f723003301,\n",
              " BERT_SENTENCE_EMBEDDINGS_68370801062d,\n",
              " LegalClassifierDLModel_c296b09c9db1]"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "clf_model.stages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4hjFsxVsUps_",
      "metadata": {
        "id": "4hjFsxVsUps_"
      },
      "outputs": [],
      "source": [
        "clf_model.stages[-1].write().overwrite().save('Clf_Use')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "UUdl5FoBbE5X",
      "metadata": {
        "id": "UUdl5FoBbE5X"
      },
      "outputs": [],
      "source": [
        "# Load back  saved Classifier Model\n",
        "ClfModel = legal.ClassifierDLModel.load('Clf_Use')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "295Ip9xZcjU-",
      "metadata": {
        "id": "295Ip9xZcjU-"
      },
      "outputs": [],
      "source": [
        "ld_pipeline = nlp.Pipeline(stages=[documentAssembler, embeddings,ClfModel])\n",
        "ld_pipeline_model = ld_pipeline.fit(spark.createDataFrame([['']]).toDF(\"text\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Gdm6mxvadFyj",
      "metadata": {
        "id": "Gdm6mxvadFyj"
      },
      "outputs": [],
      "source": [
        "# Apply Model Transform to testData\n",
        "ld_preds = ld_pipeline_model.transform(test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "aQYCSHcIdJbi",
      "metadata": {
        "id": "aQYCSHcIdJbi"
      },
      "outputs": [],
      "source": [
        "ld_preds_df = ld_preds.select('text','category',\"class.result\").toPandas()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VZnsGh6YdVUf",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "VZnsGh6YdVUf",
        "outputId": "85c166a9-a1a5-4dd8-ba8c-c5251263054d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-0b7f8f56-963e-41e2-9e68-9659e7831ce2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>category</th>\n",
              "      <th>result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Disbursement of Tenant Improvement Allowance N...</td>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>In consideration of the foregoing, the parties...</td>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Landlord shall make available for use by Tenan...</td>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Landlord shall provide Tenant the Tenant Impro...</td>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Landlord, at Landlord’s sole cost and expense,...</td>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0b7f8f56-963e-41e2-9e68-9659e7831ce2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0b7f8f56-963e-41e2-9e68-9659e7831ce2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0b7f8f56-963e-41e2-9e68-9659e7831ce2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                                text      category  \\\n",
              "0  Disbursement of Tenant Improvement Allowance N...  ti-allowance   \n",
              "1  In consideration of the foregoing, the parties...  ti-allowance   \n",
              "2  Landlord shall make available for use by Tenan...  ti-allowance   \n",
              "3  Landlord shall provide Tenant the Tenant Impro...  ti-allowance   \n",
              "4  Landlord, at Landlord’s sole cost and expense,...  ti-allowance   \n",
              "\n",
              "           result  \n",
              "0  [ti-allowance]  \n",
              "1  [ti-allowance]  \n",
              "2  [ti-allowance]  \n",
              "3  [ti-allowance]  \n",
              "4  [ti-allowance]  "
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ld_preds_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7kKWRZNu0lCL",
      "metadata": {
        "id": "7kKWRZNu0lCL"
      },
      "source": [
        "##🔎 With RoBerta Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rHw4ikwLISTb",
      "metadata": {
        "id": "rHw4ikwLISTb"
      },
      "source": [
        "We do not have Legal Sentence Embeddings yet, But we can use the Legal RoBerta Embeddings and then average them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "oaiL-Txz0lCM",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oaiL-Txz0lCM",
        "outputId": "5157bd4a-ae5d-4939-de52-1b7b7ecc0e82"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "roberta_embeddings_legal_roberta_base download started this may take some time.\n",
            "Approximate size to download 447.2 MB\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "embeddings = (\n",
        "    nlp.RoBertaEmbeddings.pretrained(\"roberta_embeddings_legal_roberta_base\", \"en\")\n",
        "    .setInputCols([\"document\", \"token\"])\n",
        "    .setOutputCol(\"embeddings\")\n",
        "    .setMaxSentenceLength(512)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "l3jxCehJ0lCO",
      "metadata": {
        "id": "l3jxCehJ0lCO"
      },
      "outputs": [],
      "source": [
        "document_assembler = (\n",
        "    nlp.DocumentAssembler().setInputCol(\"text\").setOutputCol(\"document\")\n",
        ")\n",
        "\n",
        "tokenizer = nlp.Tokenizer().setInputCols([\"document\"]).setOutputCol(\"token\")\n",
        "\n",
        "embeddingsSentence = (\n",
        "    nlp.SentenceEmbeddings()\n",
        "    .setInputCols([\"document\", \"embeddings\"])\n",
        "    .setOutputCol(\"sentence_embeddings\")\n",
        "    .setPoolingStrategy(\"AVERAGE\")\n",
        ")\n",
        "\n",
        "classifierdl = (\n",
        "    legal.ClassifierDLApproach()\n",
        "    .setInputCols([\"sentence_embeddings\"])\n",
        "    .setOutputCol(\"class\")\n",
        "    .setLabelColumn(\"category\")\n",
        "    .setMaxEpochs(3)\n",
        "    .setLr(0.001)\n",
        "    .setEnableOutputLogs(True)\n",
        "    .setOutputLogsPath(\"binary_roberta_logs\")\n",
        "    .setBatchSize(4)\n",
        "    .setDropout(0.2)\n",
        ")\n",
        "\n",
        "clf_pipeline = nlp.Pipeline(\n",
        "    stages=[document_assembler, tokenizer, embeddings, embeddingsSentence, classifierdl]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "W6qwSLYv0lCP",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W6qwSLYv0lCP",
        "outputId": "c592b45f-4840-4a70-e9c1-545fae242f83"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 3.83 s, sys: 523 ms, total: 4.35 s\n",
            "Wall time: 11min 4s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "clf_pipelineModel = clf_pipeline.fit(train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eVFKUw5cZeXB",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eVFKUw5cZeXB",
        "outputId": "1828946c-1ec8-43b8-a931-b593612890ea"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training started - epochs: 3 - learning_rate: 0.001 - batch_size: 4 - training_examples: 232 - classes: 2\n",
            "Epoch 0/3 - 0.60s - loss: 28.548275 - acc: 0.8189655 - batches: 58\n",
            "Epoch 1/3 - 0.40s - loss: 19.857958 - acc: 0.9741379 - batches: 58\n",
            "Epoch 2/3 - 0.38s - loss: 18.892944 - acc: 0.99568963 - batches: 58\n",
            "\n"
          ]
        }
      ],
      "source": [
        "log_files = os.listdir(\"binary_roberta_logs\")\n",
        "\n",
        "with open(\"binary_roberta_logs/\"+log_files[0], \"r\") as log_file :\n",
        "    print(log_file.read())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "IEK22MWg0lCQ",
      "metadata": {
        "id": "IEK22MWg0lCQ"
      },
      "outputs": [],
      "source": [
        "preds = clf_pipelineModel.transform(test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0QMopvFG0lCQ",
      "metadata": {
        "id": "0QMopvFG0lCQ"
      },
      "outputs": [],
      "source": [
        "preds_df = preds.select(\"category\", \"text\", \"class.result\").toPandas()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ADdEhII40lCR",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "ADdEhII40lCR",
        "outputId": "79d753ec-f93c-42cf-c599-e6622008c848"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-5745f63e-7f37-4eff-bf2d-4d580bda9afe\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>category</th>\n",
              "      <th>text</th>\n",
              "      <th>result</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Disbursement of Tenant Improvement Allowance N...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>In consideration of the foregoing, the parties...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Landlord shall make available for use by Tenan...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Landlord shall provide Tenant the Tenant Impro...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ti-allowance</td>\n",
              "      <td>Landlord, at Landlord’s sole cost and expense,...</td>\n",
              "      <td>[ti-allowance]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5745f63e-7f37-4eff-bf2d-4d580bda9afe')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5745f63e-7f37-4eff-bf2d-4d580bda9afe button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5745f63e-7f37-4eff-bf2d-4d580bda9afe');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "       category                                               text  \\\n",
              "0  ti-allowance  Disbursement of Tenant Improvement Allowance N...   \n",
              "1  ti-allowance  In consideration of the foregoing, the parties...   \n",
              "2  ti-allowance  Landlord shall make available for use by Tenan...   \n",
              "3  ti-allowance  Landlord shall provide Tenant the Tenant Impro...   \n",
              "4  ti-allowance  Landlord, at Landlord’s sole cost and expense,...   \n",
              "\n",
              "           result  \n",
              "0  [ti-allowance]  \n",
              "1  [ti-allowance]  \n",
              "2  [ti-allowance]  \n",
              "3  [ti-allowance]  \n",
              "4  [ti-allowance]  "
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "preds_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wvc6vZjH0lCS",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvc6vZjH0lCS",
        "outputId": "de34a727-48f4-44f4-b5ed-b7b7af12f537"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       other       1.00      0.89      0.94         9\n",
            "ti-allowance       0.93      1.00      0.96        13\n",
            "\n",
            "    accuracy                           0.95        22\n",
            "   macro avg       0.96      0.94      0.95        22\n",
            "weighted avg       0.96      0.95      0.95        22\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Let's explode the array and get the item(s) inside of result column out\n",
        "preds_df['result'] = preds_df['result'].apply(lambda x : x[0])\n",
        "\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "print (classification_report(preds_df['category'], preds_df['result']))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rMHZmT9_deAQ",
      "metadata": {
        "id": "rMHZmT9_deAQ"
      },
      "source": [
        "#📌 Save model and Zip it for Modelshub Upload/Downloads"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "QZ79YbdodfFc",
      "metadata": {
        "id": "QZ79YbdodfFc"
      },
      "outputs": [],
      "source": [
        "# Save a Spark NLP model\n",
        "clf_pipelineModel.stages[-1].write().overwrite().save('Clf_RoBerta')\n",
        "\n",
        "# cd into saved dir and zip\n",
        "! cd /content/ClfBert ; zip -r /content/ClfBert.zip *"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "tfinWtXTRynE",
      "metadata": {
        "id": "tfinWtXTRynE"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "eaaf1efa59a88a5a39769af8d2cb09ad2646375b6f26a94367c2f840c23a5e60"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}