week5/word2vec Part I (Completed).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models\n",
    "\n",
    "Spacy comes with a variety of different models that can used per language. For instance, the models for English are available [here](https://spacy.io/models/en). You'll need to download each model separately:\n",
    "\n",
    "```python\n",
    "python3 -m spacy download en_core_web_sm\n",
    "python3 -m spacy download en_core_web_md\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pattern Matching Using Spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below code and example is from Ashiq KS's article [Rule-Based Matching with spacy](https://medium.com/@ashiqgiga07/rule-based-matching-with-spacy-295b76ca2b68):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The input text string is converted to a Document object\n",
    "text = '''\n",
    "Computer programming is the process of writing instructions that get executed by computers. \n",
    "The instructions, also known as code, are written in a programming language which the computer \n",
    "can understand and use to perform a task or solve a problem. Basic computer programming involves \n",
    "the analysis of a problem and development of a logical sequence of instructions to solve it. \n",
    "There can be numerous paths to a solution and the computer programmer seeks to design and \n",
    "code that which is most efficient. Among the programmer’s tasks are understanding requirements, \n",
    "determining the right programming language to use, designing or architecting the solution, coding, \n",
    "testing, debugging and writing documentation so that the solution can be easily\n",
    "understood by other programmers.Computer programming is at the heart of computer science. It is the \n",
    "implementation portion of software development, application development \n",
    "and software engineering efforts, transforming ideas and theories into actual, working solutions.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.matcher import Matcher #import Matcher class from spacy\n",
    "#import the Span class to extract the words from the document object\n",
    "from spacy.tokens import Span \n",
    "\n",
    "#Language class with the English model 'en_core_web_sm' is loaded\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "doc = nlp(text) # convert the string above to a document\n",
    "\n",
    "#instantiate a new Matcher class object \n",
    "matcher = Matcher(nlp.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the Target Pattern"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pattern` object that you define should be a list of dictionary elements, each dictionary describing the token to match for. \n",
    "\n",
    "Here, we "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define the pattern\n",
    "pattern = [{'LOWER': 'computer', 'POS': 'NOUN'},\n",
    "             {'POS':{'NOT_IN': ['VERB']}}]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Pattern into the Matcher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#add the pattern to the previously created matcher object\n",
    "matcher.add(\"Matching\", None, pattern)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Regular Expressions in Spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below example can be found at https://spacy.io/usage/rule-based-matching. It uses the `re.finditer()` function to\n",
    "quickly iterate through all the matches found. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found match: United States\n",
      "Found match: United States\n",
      "Found match: U.S.\n",
      "Found match: US\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "import re\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"The United States of America (USA) are commonly known as the United States (U.S. or US) or America.\")\n",
    "\n",
    "expression = r\"[Uu](nited|\\.?) ?[Ss](tates|\\.?)\"\n",
    "for match in re.finditer(expression, doc.text):\n",
    "    start, end = match.span()\n",
    "    span = doc.char_span(start, end)\n",
    "    # This is a Span object or None if match doesn't map to valid token sequence\n",
    "    if span is not None:\n",
    "        print(\"Found match:\", span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part of Speech Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 -m spacy download en_core_web_sm\n",
    "!python3 -m spacy download en_core_web_md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import en_core_web_sm\n",
    "import spacy\n",
    "from scipy.spatial.distance import cosine\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en_core_web_md')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>lemma</th>\n",
       "      <th>part_of_speech</th>\n",
       "      <th>tag</th>\n",
       "      <th>dependency</th>\n",
       "      <th>shape</th>\n",
       "      <th>is_alphanumeric</th>\n",
       "      <th>is_stopword</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Steve</td>\n",
       "      <td>steve</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jobs</td>\n",
       "      <td>jobs</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>nsubj</td>\n",
       "      <td>Xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>and</td>\n",
       "      <td>and</td>\n",
       "      <td>CCONJ</td>\n",
       "      <td>CC</td>\n",
       "      <td>cc</td>\n",
       "      <td>xxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Apple</td>\n",
       "      <td>apple</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>conj</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>is</td>\n",
       "      <td>be</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>aux</td>\n",
       "      <td>xx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    text  lemma part_of_speech  tag dependency  shape  is_alphanumeric  \\\n",
       "0  Steve  steve          PROPN  NNP   compound  Xxxxx             True   \n",
       "1   Jobs   jobs          PROPN  NNP      nsubj   Xxxx             True   \n",
       "2    and    and          CCONJ   CC         cc    xxx             True   \n",
       "3  Apple  apple          PROPN  NNP       conj  Xxxxx             True   \n",
       "4     is     be           VERB  VBZ        aux     xx             True   \n",
       "\n",
       "   is_stopword  \n",
       "0        False  \n",
       "1        False  \n",
       "2         True  \n",
       "3        False  \n",
       "4         True  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "rows = []\n",
    "doc = nlp(u\"Steve Jobs and Apple is looking at buying U.K. startup for $1 billion\")\n",
    "for token in doc:\n",
    "    rows.append((token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "            token.shape_, token.is_alpha, token.is_stop))\n",
    "    \n",
    "data = pd.DataFrame(rows, columns=[\"text\", \"lemma\", \"part_of_speech\", \"tag\", \"dependency\", \"shape\", \"is_alphanumeric\", \"is_stopword\"])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Steve Jobs 0 10 PERSON\n",
      "Apple 15 20 ORG\n",
      "U.K. 42 46 GPE\n",
      "$1 billion 59 69 MONEY\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u\"Steve Jobs and Apple is looking at buying U.K. startup for $1 billion\")\n",
    "import en_core_web_sm\n",
    "import spacy\n",
    "from scipy.spatial.distance import cosine\n",
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.start_char, ent.end_char, ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Steve Jobs\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Apple\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is looking at buying \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    U.K.\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " startup for \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    $1 billion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
       "</mark>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize this using displacy:\n",
    "from spacy import displacy\n",
    "displacy.render(doc, style=\"ent\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings (word2vec Introduction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Continuous Bag of Words (Use Context to Predict Target Word)\n",
    "![alt text](images/word2vec_cbow.png \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax\n",
    "![alt text](images/softmax.png \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skipgram\n",
    "![alt text](images/skipgram.png \"Logo Title Text 1\")\n",
    "\n",
    "## Softmax\n",
    "![alt text](images/wordembedding_cluster.png \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import en_core_web_sm\n",
    "import spacy\n",
    "from scipy.spatial.distance import cosine\n",
    "nlp = en_core_web_sm.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " dog - cat: 0.4564264118671417\n",
      " dog - Beijing: 0.1571345329284668\n",
      " dog - sad: 0.3079860210418701\n",
      " dog - depressed: 0.11385080963373184\n",
      " dog - couch: 0.5404482483863831\n",
      " dog - sofa: 0.33240464329719543\n",
      " dog - canine: 0.4633784294128418\n",
      " dog - China: 0.0019485866650938988\n",
      " dog - Chinese: 0.021737948060035706\n",
      " dog - France: 0.1857185661792755\n",
      " dog - Paris: 0.11601343750953674\n",
      " dog - banana: 0.3103766441345215\n",
      " cat - dog: 0.4564264118671417\n",
      " cat - Beijing: 0.25583046674728394\n",
      " cat - sad: 0.06742441654205322\n",
      " cat - depressed: 0.11650095880031586\n",
      " cat - couch: 0.37735462188720703\n",
      " cat - sofa: 0.414833128452301\n",
      " cat - canine: 0.45437631011009216\n",
      " cat - China: 0.14348067343235016\n",
      " cat - Chinese: 0.03203266113996506\n",
      " cat - France: 0.26350462436676025\n",
      " cat - Paris: 0.1825326830148697\n",
      " cat - banana: 0.4973468482494354\n",
      " Beijing - dog: 0.1571345329284668\n",
      " Beijing - cat: 0.25583046674728394\n",
      " Beijing - sad: 0.16756749153137207\n",
      " Beijing - depressed: 0.020596839487552643\n",
      " Beijing - couch: 0.1764748990535736\n",
      " Beijing - sofa: 0.16452562808990479\n",
      " Beijing - canine: 0.26767805218696594\n",
      " Beijing - China: 0.5818219780921936\n",
      " Beijing - Chinese: 0.27865567803382874\n",
      " Beijing - France: 0.5205764174461365\n",
      " Beijing - Paris: 0.424411803483963\n",
      " Beijing - banana: 0.3193724453449249\n",
      " sad - dog: 0.3079860210418701\n",
      " sad - cat: 0.06742441654205322\n",
      " sad - Beijing: 0.16756749153137207\n",
      " sad - depressed: 0.3213621973991394\n",
      " sad - couch: 0.35177069902420044\n",
      " sad - sofa: 0.27763885259628296\n",
      " sad - canine: 0.2469976246356964\n",
      " sad - China: 0.14939850568771362\n",
      " sad - Chinese: 0.20662599802017212\n",
      " sad - France: 0.17732539772987366\n",
      " sad - Paris: 0.048603419214487076\n",
      " sad - banana: 0.27784866094589233\n",
      " depressed - dog: 0.11385080963373184\n",
      " depressed - cat: 0.11650095880031586\n",
      " depressed - Beijing: 0.020596839487552643\n",
      " depressed - sad: 0.3213621973991394\n",
      " depressed - couch: 0.2007930874824524\n",
      " depressed - sofa: 0.3122972249984741\n",
      " depressed - canine: 0.1812627762556076\n",
      " depressed - China: 0.1365271508693695\n",
      " depressed - Chinese: 0.21596072614192963\n",
      " depressed - France: 0.07575459033250809\n",
      " depressed - Paris: 0.027627602219581604\n",
      " depressed - banana: 0.0631336122751236\n",
      " couch - dog: 0.5404482483863831\n",
      " couch - cat: 0.37735462188720703\n",
      " couch - Beijing: 0.1764748990535736\n",
      " couch - sad: 0.35177069902420044\n",
      " couch - depressed: 0.2007930874824524\n",
      " couch - sofa: 0.4999978840351105\n",
      " couch - canine: 0.41085895895957947\n",
      " couch - China: 0.08625976741313934\n",
      " couch - Chinese: 0.21824336051940918\n",
      " couch - France: 0.2151881605386734\n",
      " couch - Paris: 0.21405968070030212\n",
      " couch - banana: 0.28344547748565674\n",
      " sofa - dog: 0.33240464329719543\n",
      " sofa - cat: 0.414833128452301\n",
      " sofa - Beijing: 0.16452562808990479\n",
      " sofa - sad: 0.27763885259628296\n",
      " sofa - depressed: 0.3122972249984741\n",
      " sofa - couch: 0.4999978840351105\n",
      " sofa - canine: 0.26518216729164124\n",
      " sofa - China: 0.12562435865402222\n",
      " sofa - Chinese: 0.11188281327486038\n",
      " sofa - France: 0.25162801146507263\n",
      " sofa - Paris: 0.17580696940422058\n",
      " sofa - banana: 0.40957292914390564\n",
      " canine - dog: 0.4633784294128418\n",
      " canine - cat: 0.45437631011009216\n",
      " canine - Beijing: 0.26767805218696594\n",
      " canine - sad: 0.2469976246356964\n",
      " canine - depressed: 0.1812627762556076\n",
      " canine - couch: 0.41085895895957947\n",
      " canine - sofa: 0.26518216729164124\n",
      " canine - China: 0.21173277497291565\n",
      " canine - Chinese: 0.1380724459886551\n",
      " canine - France: 0.27453306317329407\n",
      " canine - Paris: 0.13104529678821564\n",
      " canine - banana: 0.279147207736969\n",
      " China - dog: 0.0019485866650938988\n",
      " China - cat: 0.14348067343235016\n",
      " China - Beijing: 0.5818219780921936\n",
      " China - sad: 0.14939850568771362\n",
      " China - depressed: 0.1365271508693695\n",
      " China - couch: 0.08625976741313934\n",
      " China - sofa: 0.12562435865402222\n",
      " China - canine: 0.21173277497291565\n",
      " China - Chinese: 0.545533299446106\n",
      " China - France: 0.673493504524231\n",
      " China - Paris: 0.4618699848651886\n",
      " China - banana: 0.1769627034664154\n",
      " Chinese - dog: 0.021737948060035706\n",
      " Chinese - cat: 0.03203266113996506\n",
      " Chinese - Beijing: 0.27865567803382874\n",
      " Chinese - sad: 0.20662599802017212\n",
      " Chinese - depressed: 0.21596072614192963\n",
      " Chinese - couch: 0.21824336051940918\n",
      " Chinese - sofa: 0.11188281327486038\n",
      " Chinese - canine: 0.1380724459886551\n",
      " Chinese - China: 0.545533299446106\n",
      " Chinese - France: 0.4462940990924835\n",
      " Chinese - Paris: 0.554480791091919\n",
      " Chinese - banana: 0.03465459123253822\n",
      " France - dog: 0.1857185661792755\n",
      " France - cat: 0.26350462436676025\n",
      " France - Beijing: 0.5205764174461365\n",
      " France - sad: 0.17732539772987366\n",
      " France - depressed: 0.07575459033250809\n",
      " France - couch: 0.2151881605386734\n",
      " France - sofa: 0.25162801146507263\n",
      " France - canine: 0.27453306317329407\n",
      " France - China: 0.673493504524231\n",
      " France - Chinese: 0.4462940990924835\n",
      " France - Paris: 0.3865208923816681\n",
      " France - banana: 0.3307388424873352\n",
      " Paris - dog: 0.11601343750953674\n",
      " Paris - cat: 0.1825326830148697\n",
      " Paris - Beijing: 0.424411803483963\n",
      " Paris - sad: 0.048603419214487076\n",
      " Paris - depressed: 0.027627602219581604\n",
      " Paris - couch: 0.21405968070030212\n",
      " Paris - sofa: 0.17580696940422058\n",
      " Paris - canine: 0.13104529678821564\n",
      " Paris - China: 0.4618699848651886\n",
      " Paris - Chinese: 0.554480791091919\n",
      " Paris - France: 0.3865208923816681\n",
      " Paris - banana: 0.2538565695285797\n",
      " banana - dog: 0.3103766441345215\n",
      " banana - cat: 0.4973468482494354\n",
      " banana - Beijing: 0.3193724453449249\n",
      " banana - sad: 0.27784866094589233\n",
      " banana - depressed: 0.0631336122751236\n",
      " banana - couch: 0.28344547748565674\n",
      " banana - sofa: 0.40957292914390564\n",
      " banana - canine: 0.279147207736969\n",
      " banana - China: 0.1769627034664154\n",
      " banana - Chinese: 0.03465459123253822\n",
      " banana - France: 0.3307388424873352\n",
      " banana - Paris: 0.2538565695285797\n"
     ]
    }
   ],
   "source": [
    "tokens = nlp(u'dog cat Beijing sad depressed couch sofa canine China Chinese France Paris banana')\n",
    "\n",
    "for token1 in tokens:\n",
    "    for token2 in tokens:\n",
    "        if token1 != token2:\n",
    "            print(f\" {token1} - {token2}: {1 - cosine(token1.vector, token2.vector)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Most Similar Words (Using Our Old Methods)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# inspect the default settings for CountVectorizer\n",
    "CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>able</th>\n",
       "      <th>absolutely</th>\n",
       "      <th>actual</th>\n",
       "      <th>actually</th>\n",
       "      <th>adult</th>\n",
       "      <th>advertised</th>\n",
       "      <th>ago</th>\n",
       "      <th>air</th>\n",
       "      <th>amazon</th>\n",
       "      <th>apart</th>\n",
       "      <th>...</th>\n",
       "      <th>working</th>\n",
       "      <th>works</th>\n",
       "      <th>worse</th>\n",
       "      <th>worst</th>\n",
       "      <th>worth</th>\n",
       "      <th>wouldn</th>\n",
       "      <th>wrong</th>\n",
       "      <th>year</th>\n",
       "      <th>years</th>\n",
       "      <th>zero</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 500 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   able  absolutely  actual  actually  adult  advertised  ago  air  amazon  \\\n",
       "0     0           0       0         0      0           0    0    0       0   \n",
       "1     0           0       0         0      0           0    0    0       0   \n",
       "2     0           0       0         0      0           0    0    0       0   \n",
       "3     0           0       0         0      0           0    0    0       0   \n",
       "4     0           0       0         0      0           1    0    0       0   \n",
       "\n",
       "   apart  ...   working  works  worse  worst  worth  wouldn  wrong  year  \\\n",
       "0      0  ...         0      0      0      0      0       0      0     0   \n",
       "1      0  ...         0      0      0      0      0       0      0     0   \n",
       "2      0  ...         0      0      0      0      0       0      0     0   \n",
       "3      0  ...         0      0      0      0      0       0      0     0   \n",
       "4      0  ...         0      0      0      0      0       0      0     0   \n",
       "\n",
       "   years  zero  \n",
       "0      0     0  \n",
       "1      0     0  \n",
       "2      0     0  \n",
       "3      0     0  \n",
       "4      0     0  \n",
       "\n",
       "[5 rows x 500 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews = open(\"poor_amazon_toy_reviews.txt\").readlines()\n",
    "\n",
    "vectorizer = CountVectorizer(ngram_range=(1, 1), \n",
    "                             stop_words=\"english\", \n",
    "                             max_features=500,token_pattern='(?u)\\\\b[a-zA-Z][a-zA-Z]+\\\\b')\n",
    "X = vectorizer.fit_transform(reviews)\n",
    "\n",
    "data = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "# create similiarity matrix\n",
    "similarity_matrix = pd.DataFrame(cosine_similarity(data.T.values), \n",
    "             columns=vectorizer.get_feature_names(),\n",
    "                                 index=vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unstack matrix into table\n",
    "similarity_table = similarity_matrix.rename_axis(None).rename_axis(None, axis=1).stack().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(250000, 3)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename columns\n",
    "similarity_table.columns = [\"word1\", \"word2\", \"similarity\"]\n",
    "similarity_table.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(249500, 3)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_table = similarity_table[similarity_table[\"similarity\"] < 0.99]\n",
    "similarity_table.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word1</th>\n",
       "      <th>word2</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>144497</th>\n",
       "      <td>old</td>\n",
       "      <td>year</td>\n",
       "      <td>0.754569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194593</th>\n",
       "      <td>service</td>\n",
       "      <td>customer</td>\n",
       "      <td>0.734095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237276</th>\n",
       "      <td>waste</td>\n",
       "      <td>money</td>\n",
       "      <td>0.655483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>245419</th>\n",
       "      <td>working</td>\n",
       "      <td>stopped</td>\n",
       "      <td>0.589414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74650</th>\n",
       "      <td>figure</td>\n",
       "      <td>figures</td>\n",
       "      <td>0.553856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5949</th>\n",
       "      <td>arm</td>\n",
       "      <td>train</td>\n",
       "      <td>0.500379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>172826</th>\n",
       "      <td>quality</td>\n",
       "      <td>poor</td>\n",
       "      <td>0.469081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179083</th>\n",
       "      <td>remote</td>\n",
       "      <td>control</td>\n",
       "      <td>0.456930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210121</th>\n",
       "      <td>store</td>\n",
       "      <td>dollar</td>\n",
       "      <td>0.426935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4648</th>\n",
       "      <td>apart</td>\n",
       "      <td>fell</td>\n",
       "      <td>0.385922</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          word1     word2  similarity\n",
       "144497      old      year    0.754569\n",
       "194593  service  customer    0.734095\n",
       "237276    waste     money    0.655483\n",
       "245419  working   stopped    0.589414\n",
       "74650    figure   figures    0.553856\n",
       "5949        arm     train    0.500379\n",
       "172826  quality      poor    0.469081\n",
       "179083   remote   control    0.456930\n",
       "210121    store    dollar    0.426935\n",
       "4648      apart      fell    0.385922"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity_table.sort_values(by=\"similarity\", ascending=False).drop_duplicates(\n",
    "    subset=\"similarity\", keep=\"first\").head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_500_words = vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise: Similar Words Using Word Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load into spacy your top 500 words\n",
    "\n",
    "tokens = nlp(f'{\" \".join(top_500_words)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import product\n",
    "# create a list of similarity tuples\n",
    "\n",
    "similarity_tuples = []\n",
    "\n",
    "for token1, token2 in product(tokens, repeat=2):\n",
    "    similarity_tuples.append((token1, token2, token1.similarity(token2)))\n",
    "\n",
    "similarities = pd.DataFrame(similarity_tuples, columns=[\"word1\",\"word2\", \"score\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word1</th>\n",
       "      <th>word2</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>108082</th>\n",
       "      <td>inches</td>\n",
       "      <td>figures</td>\n",
       "      <td>0.833207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76607</th>\n",
       "      <td>figures</td>\n",
       "      <td>packs</td>\n",
       "      <td>0.825721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211253</th>\n",
       "      <td>stop</td>\n",
       "      <td>start</td>\n",
       "      <td>0.816715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48523</th>\n",
       "      <td>damaged</td>\n",
       "      <td>popped</td>\n",
       "      <td>0.812956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108233</th>\n",
       "      <td>inches</td>\n",
       "      <td>packs</td>\n",
       "      <td>0.812329</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          word1    word2     score\n",
       "108082   inches  figures  0.833207\n",
       "76607   figures    packs  0.825721\n",
       "211253     stop    start  0.816715\n",
       "48523   damaged   popped  0.812956\n",
       "108233   inches    packs  0.812329"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find similar words\n",
    "similarities[similarities[\"score\"] < 1].sort_values(\n",
    "    by=\"score\", ascending=False).drop_duplicates(\n",
    "    subset=\"score\", keep=\"first\").head(5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}