open-source-nlp/07.0.Context_Spell_Checker.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"dM_VpZ33OQ2u"},"source":["![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"]},{"cell_type":"markdown","metadata":{"id":"LR3OIVkDOQ2w"},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/open-source-nlp/07.0.Context_Spell_Checker.ipynb)"]},{"cell_type":"markdown","metadata":{"id":"V52RWZgkKskS"},"source":["# Context Spell Checker"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"nGPgQUvGiJkX"},"outputs":[],"source":["! pip install -q pyspark==3.4.1 spark-nlp==5.3.2"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":254},"id":"xZ1DPExi95xa","outputId":"45763f3b-39c6-46f8-fdc6-d21321cc706e","executionInfo":{"status":"ok","timestamp":1711554015347,"user_tz":-60,"elapsed":126476,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Spark NLP version 5.3.2\n","Apache Spark version: 3.4.1\n"]},{"output_type":"execute_result","data":{"text/plain":["<pyspark.sql.session.SparkSession at 0x79f8d94a89d0>"],"text/html":["\n","            <div>\n","                <p><b>SparkSession - in-memory</b></p>\n","                \n","        <div>\n","            <p><b>SparkContext</b></p>\n","\n","            <p><a href=\"http://ac20f1b35af3:4040\">Spark UI</a></p>\n","\n","            <dl>\n","              <dt>Version</dt>\n","                <dd><code>v3.4.1</code></dd>\n","              <dt>Master</dt>\n","                <dd><code>local[*]</code></dd>\n","              <dt>AppName</dt>\n","                <dd><code>Spark NLP</code></dd>\n","            </dl>\n","        </div>\n","        \n","            </div>\n","        "]},"metadata":{},"execution_count":2}],"source":["import sparknlp\n","\n","spark = sparknlp.start() # for GPU training >> sparknlp.start(gpu = True)\n","\n","from sparknlp.base import *\n","from sparknlp.annotator import *\n","from pyspark.ml import Pipeline\n","import pandas as pd\n","\n","print(\"Spark NLP version\", sparknlp.version())\n","print(\"Apache Spark version:\", spark.version)\n","\n","spark"]},{"cell_type":"markdown","metadata":{"id":"TOjVN8NKOQ22"},"source":["<H1> Noisy Channel Model Spell Checker - Introduction </H1>\n","\n","blogpost : https://medium.com/spark-nlp/applying-context-aware-spell-checking-in-spark-nlp-3c29c46963bc\n","\n","<div>\n","<p><br/>\n","The idea for this annotator is to have a flexible, configurable and \"re-usable by parts\" model.<br/>\n","Flexibility is the ability to accommodate different use cases for spell checking like OCR text, keyboard-input text, ASR text, and general spelling problems due to orthographic errors.<br/>\n","We say this is a configurable annotator, as you can adapt it yourself to different use cases avoiding re-training as much as possible.<br/>\n","</p>\n","</div>\n","\n","\n","<b> Spell Checking at three levels: </b>\n","The final ranking of a correction sequence is affected by three things,\n","\n","\n","1. Different correction candidates for each word - __word level__.\n","2. The surrounding text of each word, i.e. it's context - __sentence level__.\n","3. The relative cost of different correction candidates according to the edit operations at the character level it requires - __subword level__.\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"jUCfqQbLOQ23"},"source":["### Initial Setup\n","As it's usual in Spark-NLP let's start with building a pipeline; a _spell correction pipeline_. We will use a pretrained model from our library."]},{"cell_type":"code","execution_count":3,"metadata":{"id":"9wK6EnGvOQ24","executionInfo":{"status":"ok","timestamp":1711554015348,"user_tz":-60,"elapsed":16,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[],"source":["from sparknlp.annotator import *\n","from sparknlp.common import *\n","from sparknlp.base import *\n","\n","from IPython.utils.text import columnize"]},{"cell_type":"code","execution_count":4,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-cBsZyHaOQ27","outputId":"a76cc1db-03c2-4856-aedf-555c5f33c644","executionInfo":{"status":"ok","timestamp":1711554069574,"user_tz":-60,"elapsed":54240,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["spellcheck_dl download started this may take some time.\n","Approximate size to download 95.1 MB\n","[OK!]\n"]}],"source":["documentAssembler = DocumentAssembler()\\\n","    .setInputCol(\"text\")\\\n","    .setOutputCol(\"document\")\n","\n","tokenizer = RecursiveTokenizer()\\\n","    .setInputCols([\"document\"])\\\n","    .setOutputCol(\"token\")\\\n","    .setPrefixes([\"\\\"\", \"(\", \"[\", \"\\n\"])\\\n","    .setSuffixes([\".\", \",\", \"?\", \")\",\"!\", \"'s\"])\n","\n","spellModel = ContextSpellCheckerModel\\\n","    .pretrained('spellcheck_dl')\\\n","    .setInputCols(\"token\")\\\n","    .setOutputCol(\"checked\")\\\n","    .setErrorThreshold(4.0)\\\n","    .setTradeoff(6.0)\n","\n","finisher = Finisher()\\\n","    .setInputCols(\"checked\")\n","\n","pipeline = Pipeline(stages = [\n","     documentAssembler,\n","     tokenizer,\n","     spellModel,\n","     finisher\n","  ])\n","\n","empty_ds = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n","lp = LightPipeline(pipeline.fit(empty_ds))"]},{"cell_type":"markdown","metadata":{"id":"5Hcev_C7OQ3D"},"source":["Ok!, at this point we have our spell checking pipeline as expected. Let's see what we can do with it,"]},{"cell_type":"code","execution_count":5,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IytF5v0_OQ3E","outputId":"282eb40d-59a2-471c-8aff-d7fb2d69aa89","executionInfo":{"status":"ok","timestamp":1711554071375,"user_tz":-60,"elapsed":1814,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'checked': ['Phase',\n","  'allow',\n","  'me',\n","  'to',\n","  'introduce',\n","  'myself',\n","  ',',\n","  'I',\n","  'am',\n","  'a',\n","  'man',\n","  'of',\n","  'wealth',\n","  'and',\n","  'taste']}"]},"metadata":{},"execution_count":5}],"source":["lp.annotate(\"Plaese alliow me tao introdduce myhelf, I am a man of waelth und tiaste\")"]},{"cell_type":"markdown","metadata":{"id":"EfgK96HuOQ3K"},"source":["### Word Level Corrections\n","Continuing with our pretrained model, let's try to see how corrections work at the word level. Each Context Spell Checker model that you can find in Spark-NLP library comes with two sources for word candidates:\n","+ a general vocabulary that is built during training(and remains unmutable during the life of the model), and\n","+ special classes for dealing with special types of words like numbers or dates. These are dynamic, and you can modify them so they adjust better to your data.\n","\n","The general vocabulary is learned during training, and cannot be modified, however, the special classes can be updated after training has happened on a pre-trained model.\n","This means you can modify how existing classes produce corrections, but not the number or type of the classes.\n","Let's see how we can accomplish this."]},{"cell_type":"code","execution_count":6,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xrECOVImOQ3L","outputId":"8f086df7-2f45-4653-ef86-ba3b2807b3f5","executionInfo":{"status":"ok","timestamp":1711554071375,"user_tz":-60,"elapsed":10,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['(_NAME_,VocabParser)',\n"," '(_LOC_,VocabParser)',\n"," '(_DATE_,RegexParser)',\n"," '(_NUM_,RegexParser)']"]},"metadata":{},"execution_count":6}],"source":["# First let's start with a loaded model, and check which classes it has been trained with\n","spellModel.getWordClasses()"]},{"cell_type":"markdown","metadata":{"id":"bnXKtN9JOQ3P"},"source":["We have five classes, of two different types: some are vocabulary based and others are regex based,\n","+ __Vocabulary based classes__ can propose correction candidates from the provided vocabulary, for example a dictionary of names.\n","+ __Regex classes__ are defined by a regular expression, and they can be used to generate correction candidates for things like numbers. Internally, the Spell Checker will enumerate your regular expression and build a fast automaton, not only for recognizing the word(number in this example) as valid and preserve it, but also for generating a correction candidate.\n","Thus the regex should be a finite regex(it must define a finite regular language).\n","\n","Now suppose that you have a new friend from Poland whose name is 'Jowita', let's see how the pretrained Spell Checker does with this name."]},{"cell_type":"code","execution_count":7,"metadata":{"id":"LyNv27gBO4y4","executionInfo":{"status":"ok","timestamp":1711554071375,"user_tz":-60,"elapsed":7,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[],"source":["beautify = lambda annotations: [columnize(sent['checked']) for sent in annotations]"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"j5rqzNm1OQ3P","outputId":"02215d43-6276-4e4a-f212-48d883f2d12e","executionInfo":{"status":"ok","timestamp":1711554071893,"user_tz":-60,"elapsed":524,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['We  are  going  to  meet  With  in  the  city  hall  .\\n']"]},"metadata":{},"execution_count":8}],"source":["# Foreign name without errors\n","sample = 'We are going to meet Jowita in the city hall.'\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"qKyqcdy9OQ3S"},"source":["Well, the result is not very good, that's because the Spell Checker has been trained mainly with American English texts. At least, the surrounding words are helping to obtain a correction that is a name. We can do better, let's see how.\n","\n","## Updating a predefined word class\n","\n","### Vocabulary Classes\n","\n","In order for the Spell Checker to be able to preserve words, like a foreign name, we have the option to update existing classes so they can cover new words."]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"hpKgt58OOQ3T","outputId":"2bb868a3-585f-4eac-9098-67e7ab39aa70","executionInfo":{"status":"ok","timestamp":1711554072441,"user_tz":-60,"elapsed":549,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['We  are  going  to  meet  Jowita  at  the  city  hall  .\\n']"]},"metadata":{},"execution_count":9}],"source":["# add some more, in case we need them\n","spellModel.updateVocabClass('_NAME_', ['Monika', 'Agnieszka', 'Inga', 'Jowita', 'Melania'], True)\n","\n","# Let's see what we get now\n","sample = 'We are going to meet Jowita at the city hall.'\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"qoWZ657hOQ3W"},"source":["Much better, right? Now suppose that we want to be able to not only preserve the word, but also to propose meaningful corrections to the name of our foreign friend."]},{"cell_type":"code","execution_count":10,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"azIAKc8UOQ3X","outputId":"afe65dd8-944f-4ce4-b027-1f8d634231ee","executionInfo":{"status":"ok","timestamp":1711554073290,"user_tz":-60,"elapsed":851,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['We  are  going  to  meet  Jowita  in  the  city  hall  .\\n']"]},"metadata":{},"execution_count":10}],"source":["# Foreign name with an error\n","sample = 'We are going to meet Jovita in the city hall.'\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"z4fvnwl7OQ3b"},"source":["Here we were able to add the new word to the class and propose corrections for it, but also, the new word has been treated as a name, that meaning that the model used information about the typical context for names in order to produce the best correction."]},{"cell_type":"markdown","metadata":{"id":"JrpDPJ4YOQ3c"},"source":["### Regex Classes\n","We can do something similar for classes defined by regex. We can add a regex, to for example deal with a special format for dates, that will not only preserve the date with the special format, but also be able to correct it."]},{"cell_type":"code","execution_count":11,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"fg5-ajMDOQ3d","outputId":"ddfb881a-5f8e-45ae-c09b-b4a67d84491e","executionInfo":{"status":"ok","timestamp":1711554073811,"user_tz":-60,"elapsed":526,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['We  are  going  to  meet  her  in  the  city  hall  on  February  .\\n']"]},"metadata":{},"execution_count":11}],"source":["# Date with custom format\n","sample = 'We are going to meet her in the city hall on february-3.'\n","beautify([lp.annotate(sample)])"]},{"cell_type":"code","execution_count":12,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YP2iOBsGOQ3g","outputId":"49c8e1fb-1fa5-43dd-93b8-6bd969983938","executionInfo":{"status":"ok","timestamp":1711554075213,"user_tz":-60,"elapsed":1404,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['We  are  going  to  meet  her  in  the  city  hall  on  february-3  .\\n']"]},"metadata":{},"execution_count":12}],"source":["# this is a sample regex, for simplicity not covering all months\n","spellModel.updateRegexClass('_DATE_', '(january|february|march)-[0-31]')\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"74XHt67TOQ3j"},"source":["Now our date wasn't destroyed!"]},{"cell_type":"code","execution_count":13,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"v4BxqhHNOQ3j","outputId":"ef50b4a2-9e0a-4c70-c67e-a97b527d939d","executionInfo":{"status":"ok","timestamp":1711554075578,"user_tz":-60,"elapsed":367,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['We  are  going  to  meet  her  in  the  city  hall  on  febbruary-3  .\\n']"]},"metadata":{},"execution_count":13}],"source":["# now check that it produces good corrections to the date\n","sample = 'We are going to meet her in the city hall on febbruary-3.'\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"0_qhmeH7OQ3n"},"source":["And the model produces good corrections for the special regex class. Remember that each regex that you enter to the model must be finite. In all these examples the new definitions for our classes didn't prevent the model to continue using the context to produce corrections. Let's see why being able to use the context is important.\n","### Sentence Level Corrections\n","The Spell Checker can leverage the context of words for ranking different correction sequences. Let's take a look at some examples,"]},{"cell_type":"code","execution_count":14,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"FDuBuz29OQ3o","outputId":"d986347b-aa1d-4307-bf42-a66a4b6cd95f","executionInfo":{"status":"ok","timestamp":1711554076278,"user_tz":-60,"elapsed":702,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Due  to  bad  weather  ,  we  had  to  move  to  a  different  site  .\\n',\n"," 'We  travelled  to  three  sites  in  the  summer  .\\n']"]},"metadata":{},"execution_count":14}],"source":["# check for the different occurrences of the word \"siter\"\n","example1 = [\n","    \"Due to bad weather, we had to move to a different siter.\",\\\n","    \"We travelled to three siter in the summer.\"]\n","beautify(lp.annotate(example1))"]},{"cell_type":"code","execution_count":15,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"D4wn2v2XOQ3s","outputId":"9223cf37-4b92-4fc1-e7b4-5a347e25e7d4","executionInfo":{"status":"ok","timestamp":1711554077201,"user_tz":-60,"elapsed":924,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['During  the  summer  we  have  the  best  weather  .\\n',\n"," 'I  have  a  black  leather  jacket  ,  so  nice  .\\n']"]},"metadata":{},"execution_count":15}],"source":["# check for the different occurrences of the word \"ueather\"\n","example2 = [\"During the summer we have the best ueather.\",\\\n","    \"I have a black ueather jacket, so nice.\"]\n","beautify(lp.annotate(example2))"]},{"cell_type":"markdown","metadata":{"id":"xd1gXbwyOQ3u"},"source":["Notice that in the first example, 'siter' is indeed a valid English word, <br/> https://www.merriam-webster.com/dictionary/siter <br/>\n","The only way to customize how the use of context is performed is to train the language model by training a Spell Checker from scratch. If you want to be able to train your custom language model, please refer to the Training notebook.\n","Now we've learned how the context can help to pick the best possible correction, and why it is important to be able to leverage the context even when the other parts of the Spell Checker were updated."]},{"cell_type":"markdown","metadata":{"id":"tAXDhU-LOQ3v"},"source":["### Subword level corrections\n","Another fine tunning that our Spell Checker accepts is to assign different costs to different edit operations that are necessary to transform a word into a correction candidate.\n","So, why is this important? Errors can come from different sources,\n","+ Homophones are words that sound similar, but are written differently and have different meaning. Some examples, {there, their, they're}, {see, sea}, {to, too, two}. You will typically see these errors in text obtained by Automatic Speech Recognition(ASR).\n","+ Characters can also be confused because of looking similar. So a 0(zero) can be confused with a O(capital o), or a 1(number one) with an l(lowercase l). These errors typically come from OCR.\n","+ Input device related, sometimes keyboards cause certain patterns to be more likely than others due to letter locations, for example in a QWERTY keyboard.\n","+ Last but not least, ortographic errors, related to the writter making mistakes. Forgetting a double consonant, or using it in the wrong place, interchanging letters(i.e., 'becuase' for 'because'), and many others.\n","\n","The goal is to continue using all the other features of the model and still be able to adapt the model to handle each of these cases in the best possible way. Let's see how to accomplish this."]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nsvSTA5TOQ3v","outputId":"c94cd670-829f-44b4-acc6-572bddc7025f","executionInfo":{"status":"ok","timestamp":1711554077687,"user_tz":-60,"elapsed":489,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'checked': ['I', 'will', 'be', 'tending', 'him', 'my', 'car']}"]},"metadata":{},"execution_count":16}],"source":["# sending or lending ?\n","sample = 'I will be 1ending him my car'\n","lp.annotate(sample)"]},{"cell_type":"code","execution_count":17,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"V31KhduLOQ35","outputId":"1a270e84-0527-4092-a9e1-a4e3fd4a8777","executionInfo":{"status":"ok","timestamp":1711554078123,"user_tz":-60,"elapsed":443,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'checked': ['I', 'will', 'be', 'lending', 'him', 'my', 'car']}"]},"metadata":{},"execution_count":17}],"source":["# let's make the replacement of an '1' for an 'l' cheaper\n","weights = {'1': {'l': .01}}\n","spellModel.setWeights(weights)\n","lp.annotate(sample)"]},{"cell_type":"markdown","metadata":{"id":"oYMAj2YjOQ37"},"source":["Assembling this matrix by hand could be a daunting challenge. There is one script in Python that can do this for you.\n","This is something to be soon included like an option during training for the Context Spell Checker. Stay tuned on new releases!"]},{"cell_type":"markdown","metadata":{"id":"CF6roiiQOQ38"},"source":["## Advanced - the mysterious tradeoff parameter\n","There's a clear tension between two forces here,\n","+ The context information: by which the model wants to change words based on the surrounding words.\n","+ The word information: by which the model wants to preserve as much an input word as possible to avoid destroying the input.\n","\n","Changing words that are in the vocabulary for others that seem more suitable according to the context is one of the most challenging tasks in spell correction. This is because you run into the risk of destroying existing 'good' words.\n","The models that you will find in the Spark-NLP library have already been configured in a way that balances these two forces and produces good results in most of the situations. But your dataset can be different from the one used to train the model.\n","So we encourage the user to play a bit with the hyperparameters, and for you to have an idea on how it can be modified, we're going to see the following example,"]},{"cell_type":"code","execution_count":18,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"trwTZ0YROQ38","outputId":"bc1b806a-9f23-416b-dd31-f3a9a2935db9","executionInfo":{"status":"ok","timestamp":1711554078478,"user_tz":-60,"elapsed":356,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['have  you  been  to  the  falls  ?\\n']"]},"metadata":{},"execution_count":18}],"source":["sample = 'have you been two the falls?'\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"nvp4QocxOQ3-"},"source":["Here 'two' is clearly wrong, probably a typo, and the model should be able to choose the right correction candidate according to the context. <br/>\n","Every path is scored with a cost, and the higher the cost the less chances for the path being chosen as the final answer.<br/>\n","In order for the model to rely more on the context and less on word information, we have the setTradeoff() method. You can think of the tradeoff as how much a single edition(insert, delete, etc) operation affects the influence of a word when competing inside a path in the graph.<br/>\n","So the lower the tradeoff, the less we care about the edit operations in the word, and the more we care about the word fitting properly into its context. The tradeoff parameter typically ranges between 5 and 25. <br/>\n","Let's see what happens when we relax how much the model cares about individual words in our example,"]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"WUpyUecvOQ3_","outputId":"5ca993c9-17b0-4164-9c2e-d2c695db22e1","executionInfo":{"status":"ok","timestamp":1711554078479,"user_tz":-60,"elapsed":4,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["6.0"]},"metadata":{},"execution_count":19}],"source":["spellModel.getTradeoff()"]},{"cell_type":"code","execution_count":20,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5zmQLB_UOQ4C","outputId":"3e60a68f-91b5-4268-8672-79750525fc84","executionInfo":{"status":"ok","timestamp":1711554079371,"user_tz":-60,"elapsed":895,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['have  you  been  to  the  falls  ?\\n']"]},"metadata":{},"execution_count":20}],"source":["# let's decrease the influence of word-level errors\n","# TODO a nicer way of doing this other than re-creating the pipeline?\n","spellModel.setTradeoff(2.0)\n","\n","pipeline = Pipeline(\n","    stages = [\n","    documentAssembler,\n","    tokenizer,\n","    spellModel,\n","    finisher\n","  ])\n","\n","empty_ds = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n","lp = LightPipeline(pipeline.fit(empty_ds))\n","\n","beautify([lp.annotate(sample)])"]},{"cell_type":"markdown","metadata":{"id":"LD1RZYWCOQ4F"},"source":["## Advanced - performance"]},{"cell_type":"markdown","metadata":{"id":"H-EyNb0HOQ4G"},"source":["The discussion about performance revolves around _error detection_. The more errors the model detects the more populated is the candidate diagram we showed above[TODO add diagram or convert this into blogpost], and the more alternative paths need to be evaluated. </br>\n","Basically the error detection stage of the model can decide whether a word needs a correction or not; with two reasons for a word to be considered as incorrect,\n","+ The word is OOV: the word is out of the vocabulary.\n","+ The context: the word doesn't fit well within its neighbouring words.\n","The only parameter that we can control at this point is the second one, and we do so with the setErrorThreshold() method that contains a max perplexity above which the word will be considered suspicious and a good candidate for being corrected.</br>\n","The parameter that comes with the pretrained model has been set so you can get both a decent performance and accuracy. For reference, this is how the F-score, and time varies in a sample dataset for different values of the errorThreshold,\n","\n","\n","|fscore |totaltime|threshold|\n","|-------|---------|---------|\n","|52.69  |405s | 8f|\n","|52.43  |357s |10f|\n","|52.25  |279s |12f|\n","|52.14  |234s |14f|\n","\n","You can trade some minor points in accuracy for a nice speedup.\n"]},{"cell_type":"code","execution_count":21,"metadata":{"id":"Pt7ca87zQaCP","executionInfo":{"status":"ok","timestamp":1711554079371,"user_tz":-60,"elapsed":2,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[],"source":["def sparknlp_spell_check(text):\n","\n","  return beautify([lp.annotate(text)])[0].rstrip()\n"]},{"cell_type":"code","execution_count":22,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"id":"vIFJmn6pPobo","outputId":"ecfdb969-d7e3-41d0-9edd-66d2694caaa1","executionInfo":{"status":"ok","timestamp":1711554079741,"user_tz":-60,"elapsed":372,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'I  will  go  to  Philadelphia  tomorrow'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":22}],"source":["sparknlp_spell_check('I will go to Philadelhia tomorrow')"]},{"cell_type":"code","execution_count":23,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"id":"OmCSDVc9Vz6H","outputId":"b426db69-9449-4ba9-d2f0-808a8a0d1df9","executionInfo":{"status":"ok","timestamp":1711554080098,"user_tz":-60,"elapsed":360,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'I  will  go  to  Philadelphia  tomorrow'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":23}],"source":["sparknlp_spell_check('I will go to Philadhelpia tomorrow')"]},{"cell_type":"code","execution_count":24,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"id":"TG-bQtJnVz3Z","outputId":"2075fd11-fe57-4c57-bafb-5e9583ad9339","executionInfo":{"status":"ok","timestamp":1711554080841,"user_tz":-60,"elapsed":747,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'I  will  go  to  Philadelphia  tomorrow'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":24}],"source":["sparknlp_spell_check('I will go to Piladelphia tomorrow')"]},{"cell_type":"code","execution_count":25,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"id":"3TmDb-ThVz0t","outputId":"ef6ae29b-86bf-4594-a5a9-47ab753e3e51","executionInfo":{"status":"ok","timestamp":1711554081457,"user_tz":-60,"elapsed":623,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'I  will  go  to  Philadelphia  tomorrow'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":25}],"source":["sparknlp_spell_check('I will go to Philadedlphia tomorrow')"]},{"cell_type":"code","execution_count":26,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"id":"m50x-QYaVzx6","outputId":"06add1d0-ea97-4e42-ef2d-f83b877fb7f6","executionInfo":{"status":"ok","timestamp":1711554081870,"user_tz":-60,"elapsed":423,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'I  will  go  to  Philadelphia  tomorrow'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":26}],"source":["sparknlp_spell_check('I will go to Phieladelphia tomorrow')"]},{"cell_type":"markdown","metadata":{"id":"gQ8oCQA4wKMJ"},"source":["## ContextSpellCheckerApproach\n","\n","Trains a deep-learning based Noisy Channel Model Spell Algorithm.\n","\n","Correction candidates are extracted combining context information and word information.\n","\n","1.   Different correction candidates for each word   **word level**\n","2.   The surrounding text of each word, i.e. it’s context  **sentence level**.\n","3.   The relative cost of different correction candidates according to the edit operations at the character level it requires  **subword level**.\n"]},{"cell_type":"code","execution_count":27,"metadata":{"id":"7NyXcrNSIr-X","executionInfo":{"status":"ok","timestamp":1711554089006,"user_tz":-60,"elapsed":7144,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[],"source":["# For this example, we will use the first Sherlock Holmes book as the training dataset.\n","\n","documentAssembler = DocumentAssembler() \\\n","    .setInputCol(\"text\") \\\n","    .setOutputCol(\"document\")\n","\n","tokenizer = Tokenizer()\\\n","    .setInputCols(\"document\") \\\n","    .setOutputCol(\"token\")\n","\n","spellChecker = ContextSpellCheckerApproach() \\\n","    .setInputCols(\"token\") \\\n","    .setOutputCol(\"corrected\") \\\n","    .setWordMaxDistance(3) \\\n","    .setBatchSize(24) \\\n","    .setEpochs(8) \\\n","    .setLanguageModelClasses(1650)  # dependant on vocabulary size\n","    # .addVocabClass(\"_NAME_\", names) # Extra classes for correction could be added like this\n","\n","pipeline = Pipeline().setStages([\n","    documentAssembler,\n","    tokenizer,\n","    spellChecker\n","])"]},{"cell_type":"code","execution_count":28,"metadata":{"id":"dpaFhDADrZf5","executionInfo":{"status":"ok","timestamp":1711554089467,"user_tz":-60,"elapsed":472,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[],"source":["! wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Public/data/holmes.txt"]},{"cell_type":"code","execution_count":29,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"XUCpkYHtrZdN","outputId":"4b1699cd-de5c-4cab-c518-312fd7d84d72","executionInfo":{"status":"ok","timestamp":1711554091745,"user_tz":-60,"elapsed":2279,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["+----------------------------------------------------------------------------------------------------+\n","|                                                                                                text|\n","+----------------------------------------------------------------------------------------------------+\n","|THE ADVENTURES OF SHERLOCK HOLMESArthur Conan Doyle Table of contents A Scandal in Bohemia The Re...|\n","+----------------------------------------------------------------------------------------------------+\n","\n"]}],"source":["path = \"holmes.txt\"\n","\n","dataset = spark.read.text(path).toDF(\"text\")\n","\n","dataset.show(truncate=100)"]},{"cell_type":"code","execution_count":30,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YVpujH03rZak","outputId":"c9fb7d9f-50e6-43a3-8948-402a0b68f35c","executionInfo":{"status":"ok","timestamp":1711554126245,"user_tz":-60,"elapsed":34504,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["CPU times: user 280 ms, sys: 49.4 ms, total: 330 ms\n","Wall time: 34.2 s\n"]}],"source":["%%time\n","pipelineModel = pipeline.fit(dataset)"]},{"cell_type":"code","execution_count":31,"metadata":{"id":"OmmcL2mjsUL5","executionInfo":{"status":"ok","timestamp":1711554127428,"user_tz":-60,"elapsed":1199,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[],"source":["lp = LightPipeline(pipelineModel)\n","result = lp.annotate(\"Plaese alliow me tao introdduce myhelf, I am a man of waelth und tiaste\")"]},{"cell_type":"code","execution_count":32,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jtz0Ud95s76T","outputId":"f7529447-3654-4734-d6d4-d04be7e7c4e1","executionInfo":{"status":"ok","timestamp":1711554127428,"user_tz":-60,"elapsed":8,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Please',\n"," 'allow',\n"," 'me',\n"," 'to',\n"," 'introduce',\n"," 'myself',\n"," ',',\n"," 'I',\n"," 'am',\n"," 'a',\n"," 'man',\n"," 'of',\n"," 'wealth',\n"," 'and',\n"," 'taste']"]},"metadata":{},"execution_count":32}],"source":["result[\"corrected\"]"]},{"cell_type":"code","execution_count":33,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":519},"id":"0LIupsUKtTi8","outputId":"45609e9d-5d3e-4178-b6b9-cab5d32e1e95","executionInfo":{"status":"ok","timestamp":1711554127428,"user_tz":-60,"elapsed":8,"user":{"displayName":"Akar Öztürk","userId":"07602883577262601067"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["      original  corrected\n","0       Plaese     Please\n","1       alliow      allow\n","2           me         me\n","3          tao         to\n","4   introdduce  introduce\n","5       myhelf     myself\n","6            ,          ,\n","7            I          I\n","8           am         am\n","9            a          a\n","10         man        man\n","11          of         of\n","12      waelth     wealth\n","13         und        and\n","14      tiaste      taste"],"text/html":["\n","  <div id=\"df-2ba3087d-6194-491d-866d-ac41185c3192\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>original</th>\n","      <th>corrected</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Plaese</td>\n","      <td>Please</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>alliow</td>\n","      <td>allow</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>me</td>\n","      <td>me</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>tao</td>\n","      <td>to</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>introdduce</td>\n","      <td>introduce</td>\n","    </tr>\n","    <tr>\n","      <th>5</th>\n","      <td>myhelf</td>\n","      <td>myself</td>\n","    </tr>\n","    <tr>\n","      <th>6</th>\n","      <td>,</td>\n","      <td>,</td>\n","    </tr>\n","    <tr>\n","      <th>7</th>\n","      <td>I</td>\n","      <td>I</td>\n","    </tr>\n","    <tr>\n","      <th>8</th>\n","      <td>am</td>\n","      <td>am</td>\n","    </tr>\n","    <tr>\n","      <th>9</th>\n","      <td>a</td>\n","      <td>a</td>\n","    </tr>\n","    <tr>\n","      <th>10</th>\n","      <td>man</td>\n","      <td>man</td>\n","    </tr>\n","    <tr>\n","      <th>11</th>\n","      <td>of</td>\n","      <td>of</td>\n","    </tr>\n","    <tr>\n","      <th>12</th>\n","      <td>waelth</td>\n","      <td>wealth</td>\n","    </tr>\n","    <tr>\n","      <th>13</th>\n","      <td>und</td>\n","      <td>and</td>\n","    </tr>\n","    <tr>\n","      <th>14</th>\n","      <td>tiaste</td>\n","      <td>taste</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2ba3087d-6194-491d-866d-ac41185c3192')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-2ba3087d-6194-491d-866d-ac41185c3192 button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-2ba3087d-6194-491d-866d-ac41185c3192');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-96a71686-3e6a-4637-8df8-5313af27f9f3\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-96a71686-3e6a-4637-8df8-5313af27f9f3')\"\n","            title=\"Suggest charts\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-96a71686-3e6a-4637-8df8-5313af27f9f3 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","    </div>\n","  </div>\n"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"dataframe","summary":"{\n  \"name\": \"pd\",\n  \"rows\": 15,\n  \"fields\": [\n    {\n      \"column\": \"original\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 15,\n        \"samples\": [\n          \"a\",\n          \"of\",\n          \"Plaese\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"corrected\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 15,\n        \"samples\": [\n          \"a\",\n          \"of\",\n          \"Please\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"}},"metadata":{},"execution_count":33}],"source":["pd.DataFrame(zip(result[\"token\"],result[\"corrected\"]),columns=[\"original\",\"corrected\"])"]}],"metadata":{"colab":{"provenance":[],"machine_shape":"hm"},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.7"},"vscode":{"interpreter":{"hash":"3f47d918ae832c68584484921185f5c85a1760864bf927a683dc6fb56366cc77"}}},"nbformat":4,"nbformat_minor":0}