demos/tf-idf.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style> code {background-color : lightgrey !important;} </style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<style> code {background-color : lightgrey !important;} </style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# TF-IDF on gutenberg corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**corpushash** is a simple library that aims to make the natural language processing of sensitive documents easier. the library enables performing common NLP tasks on sensitive documents without disclosing their contents. This is done by hashing every token in the corpus along with a salt (to prevent dictionary attacks). \n",
    "\n",
    "its workflow is as simple as having the sensitive corpora as a python nested list (or generator) whose elements are themselves (nested) lists of strings. after the hashing is done, NLP can be carried out by a third party, and when the results are in they can be decoded by a dictionary that maps hashes to the original strings. so that makes:\n",
    "\n",
    "```python\n",
    "import corpushash as ch\n",
    "hashed_corpus = ch.CorpusHash(mycorpus_as_a_nested_list, '/home/sensitive-corpus')\n",
    ">>> \"42 documents hashed and saved to '/home/sensitive-corpus/public/$(timestamp)'\"\n",
    "```\n",
    "**NLP is done, and `results` are in**:\n",
    "```python\n",
    "for token in results:\n",
    "    print(token, \">\", hashed_corpus.decode_dictionary[token])\n",
    ">>> \"7)JBMGG?sGu+>%Js~dG=%c1Qn1HpAU{jM-~Buu7?\" > \"gutenberg\"\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### loading libraries for preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import string\n",
    "import random\n",
    "import pickle\n",
    "import nltk\n",
    "from nltk.corpus import gutenberg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### downloading nltk gutenberg corpus, if not downloaded already:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#nltk.download(\"gutenberg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "files in test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['austen-emma.txt',\n",
       " 'austen-persuasion.txt',\n",
       " 'austen-sense.txt',\n",
       " 'bible-kjv.txt',\n",
       " 'blake-poems.txt',\n",
       " 'bryant-stories.txt',\n",
       " 'burgess-busterbrown.txt',\n",
       " 'carroll-alice.txt',\n",
       " 'chesterton-ball.txt',\n",
       " 'chesterton-brown.txt',\n",
       " 'chesterton-thursday.txt',\n",
       " 'edgeworth-parents.txt',\n",
       " 'melville-moby_dick.txt',\n",
       " 'milton-paradise.txt',\n",
       " 'shakespeare-caesar.txt',\n",
       " 'shakespeare-hamlet.txt',\n",
       " 'shakespeare-macbeth.txt',\n",
       " 'whitman-leaves.txt']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gutenberg.fileids()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## preparing input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "creating test corpus path, where hashed documents will be stored as .json files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/bruno/Documents/github/hashed-nlp/guten_test'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_path = os.path.join(os.getcwd(), 'guten_test')\n",
    "corpus_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "the library takes as input a nested list whose elements are the original documents as nested lists. the nested lists represent the document's structure (sections, paragraphs, sentences). this can be an in-memory nested list or some generator that yields a nested list when it is iterated over.\n",
    "\n",
    "**as this is a simple tf-idf transformation, we don't need the nested list format, because document structure is not important**: just having the words with no sentence or paragraph structure is sufficient. the library's input is designed to be flexible in this regard.\n",
    "\n",
    "#### creating a dictionary whose keys are the fileids and whose values are a list of the words in every document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 23.2 s, sys: 684 ms, total: 23.9 s\n",
      "Wall time: 23.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "decoded_gutencorpus = []\n",
    "for document_name in gutenberg.fileids():\n",
    "    document = [word.lower() for word in gutenberg.words(document_name) if word not in string.punctuation and not word.isdigit()]\n",
    "    decoded_gutencorpus.append(document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "excerpt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['moby', 'dick', 'by', 'herman', 'melville', 'etymology', 'supplied', 'by', 'a', 'late', 'consumptive', 'usher', 'to', 'a', 'grammar', 'school', 'the', 'pale', 'usher', '--', 'threadbare', 'in', 'coat', 'heart', 'body', 'and', 'brain', 'i', 'see', 'him', 'now', 'he', 'was', 'ever', 'dusting', 'his', 'old', 'lexicons', 'and', 'grammars', 'with', 'a', 'queer', 'handkerchief', 'mockingly', 'embellished', 'with', 'all', 'the', 'gay', 'flags', 'of', 'all', 'the', 'known', 'nations', 'of', 'the', 'world', 'he', 'loved', 'to', 'dust', 'his', 'old', 'grammars', 'it', 'somehow', 'mildly', 'reminded', 'him', 'of', 'his', 'mortality', 'while', 'you', 'take', 'in', 'hand', 'to', 'school', 'others', 'and', 'to', 'teach', 'them', 'by', 'what', 'name', 'a', 'whale', 'fish', 'is', 'to', 'be', 'called', 'in', 'our', 'tongue', 'leaving']\n"
     ]
    }
   ],
   "source": [
    "document = random.choice(decoded_gutencorpus)\n",
    "print(document[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## processing using `corpushash`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "loading libraries for corpushash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import corpushash as ch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### instatiating CorpusHash class, which hashes the provided corpus to the corpus_path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-05-04 21:47:51,604 - corpushash.hashers - INFO - 18 documents hashed and saved to /home/bruno/Documents/github/hashed-nlp/guten_test/public/2017-05-04_21-47-13.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 37.4 s, sys: 1.08 s, total: 38.5 s\n",
      "Wall time: 38.6 s\n"
     ]
    }
   ],
   "source": [
    "%time hashed_guten = ch.CorpusHash(decoded_gutencorpus, corpus_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "that is it. `corpushash`'s work is done."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## NLP: a tf-idf example\n",
    "\n",
    "from now on we are simulating the work of an analyst that receives the encoded documents and performs a common *NLP* task on them, the calculation of tf-idf weights.\n",
    "\n",
    "**note: we will be using the gensim library for the NLP. if you are not familiar with it, you may want to check the first [couple tutorials](http://radimrehurek.com/gensim/tut1.html).** (you could use the library of your choice, naturally).\n",
    "\n",
    "all the analyst has are the files on `corpushash/gutenberg_test/public/$(timestamp)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/bruno/Documents/github/hashed-nlp/guten_test/public/2017-05-04_21-47-13'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_corpus_path = hashed_guten.public_path\n",
    "encoded_corpus_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.exists(encoded_corpus_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "loading libraries we need for processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### defining iterable for gensim:\n",
    "\n",
    "in order to perform the tf-idf calculation, we use the gensim library. the gensim library takes as input any object that yields documents when it is iterated over.\n",
    "\n",
    "`corpushash` has a built-in reader that yields the document index and the hashed document. these are the first tokens in each document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 ['jvUAM!v5r3%Qm9p$>Fvt^c*oxVyC0|t^6x^C^W*m']\n",
      "1 ['NR2suR3NuRNIP*q#h`D(3J)+V#3IBR;3%|Rv>@6a']\n",
      "2 ['o%Pqc+tEWvBC>f+VTmr1>#YqIx1x}#U-8ssStgNb']\n",
      "3 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "4 ['&Ef7M@=~%szmYdt#42k(k)Am6iZspGu9e*kfKEh_']\n",
      "5 ['%5I7)%aXQ9>yB>a`CX7mcZS=mv^EbO7wTIH1G}vT']\n",
      "6 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "7 ['efB55nAlWX5}s*E05`$`DJV{P+?tRbZ@DfZ-J~>V']\n",
      "8 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "9 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "10 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "11 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "12 ['#68+}E6K9gxiG!A6<6%zZZ$LNyyMHdv{CU_zaIHQ']\n",
      "13 ['z)aMcvNBzTkH1+$qrU!Fdgt15F$+Lo&br5=E+QBU']\n",
      "14 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "15 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "16 ['Mg!>;<8IWflr?tDWG2Ah7P8GSrK+ag!oaD`xgZnF']\n",
      "17 ['|59SolgN4Ad>Y>67LeZ%w6-r-%aBPb8|0A7Jr;vr']\n"
     ]
    }
   ],
   "source": [
    "for i in hashed_guten.read_hashed_corpus():\n",
    "    print(i[0], i[1][:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "as the analyst will not have access to this convenience function, we will build a new generator for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jvUAM!v5r3%Qm9p$>Fvt^c*oxVyC0|t^6x^C^W*m\n",
      "HIb_;r&gA>jyB(7uN3ccOc2^ezLBX^I?s8LsBWW6\n",
      "(9E+nK-5R5&aH)68J0*s`j#I{;|{Qy$m+20Edx_U\n",
      "hU_mku^J2ns3DavK#1gNfcWM;H?FrrrLs`XuaJA1\n",
      "-7j6;I86bI1D(lzhW*-hG0^$J|F3`gwp91esOOks\n",
      "wCz}5S=Sj?a!DwyA7-#|)r?d$;*HpVcRZdTNF*6x\n",
      "|HEJp)5AGk;5N_I1OM)PS3sVJ-fve*0cT}4K`5AA\n",
      "wCz}5S=Sj?a!DwyA7-#|)r?d$;*HpVcRZdTNF*6x\n",
      "jvUAM!v5r3%Qm9p$>Fvt^c*oxVyC0|t^6x^C^W*m\n",
      "?L6*!ZRoW*AKav}%p%F#4Nos4k!N?~%As6JcDyLS\n"
     ]
    }
   ],
   "source": [
    "def encoded_gutenberg_yielder(corpus_path):\n",
    "    for ix in range(len(gutenberg.fileids())):\n",
    "        path = os.path.join(corpus_path, '{}.json'.format(ix))\n",
    "        with open(path, 'r') as fp:\n",
    "            document_tokens = json.load(fp)\n",
    "        yield document_tokens\n",
    "\n",
    "example_doc = encoded_gutenberg_yielder(encoded_corpus_path)\n",
    "print(\"\\n\".join(next(example_doc)[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### building gensim dictionary\n",
    "\n",
    "from this document generator gensim will build a dictionary that maps every hashed token to an ID, a mapping which is later used to calculate the tf-idf weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "encoded_gutendict = gensim.corpora.Dictionary(encoded_gutencorpus)\n",
    "#encoded_gutendict.save_as_text('enc_dict.txt', sort_by_word=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the number of unique words in our corpus is 42020.\n"
     ]
    }
   ],
   "source": [
    "print(\"the number of unique words in our corpus is {}.\".format(len(encoded_gutendict)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### bag-of-words\n",
    "\n",
    "to build a tf-idf model, the gensim library needs an input that yields this vectorized bag-of-words when iterated over:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def bow_gutenberg_yielder(corpus, dictionary):\n",
    "    for document_tokens in corpus:\n",
    "        yield dictionary.doc2bow(document_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "we must re-instantiate the generator, else it'll be depleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)\n",
    "encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token >> (token id, frequency in document)\n",
      "\n",
      "v>@CDuT4Yoss>twzY&Z6^aTR}!CP6m(ht&fRM`eh >> (0, 865)\n",
      "`;cqzu8WYSbhdPA;EgebuBgVt0sL{kl*69OoN^u& >> (1, 571)\n",
      "S?VWSP0PVQ*Ze<nSimZQ|Lq4->d6xKrh5Wh+o7tE >> (2, 301)\n",
      "o9zYE@{v;UprEjBZwc9v0PVlWPZ`e^i$9w29K<0c >> (3, 1)\n",
      "ed>e_?cy=_9AX6Nk#;<yNp?mo_lxn_n{6{=1k2@i >> (4, 3)\n",
      "etNN3mT&s;#89i7vY|s;39^H@e7XhhTvGIj>Y!V@ >> (5, 3178)\n",
      "@e|=)yTLqzKhfIlyTg&UdGw26F`IG&lYaQs!d4=U >> (6, 56)\n",
      "p%(T2Bw%|_G&T!P#*xLbVSwta9jyeHP@W9(JXY?A >> (7, 313)\n",
      "?|7@`_*&RzcZM9EQ39<D6tVAOO*n_wK!qM1gPP!U >> (8, 38)\n",
      "PA3Y-CI+Mi^<nXVdA?y@m<abq->ak5G{>}LlFk}n >> (9, 27)\n"
     ]
    }
   ],
   "source": [
    "print('token', '>>', \"(token id, frequency in document)\\n\")\n",
    "for i in next(encoded_gutenbow)[:10]:\n",
    "    print(encoded_gutendict.get(i[0]), '>>', i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### tf-idf model\n",
    "\n",
    "now we are ready to deploy the tf-idf model to our corpus.\n",
    "\n",
    "when applying `gensim.models.TfidfModel` we calculate the tf-idf weights of every token in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gensim.models.tfidfmodel:collecting document frequencies\n",
      "INFO:gensim.models.tfidfmodel:PROGRESS: processing document #0\n",
      "INFO:gensim.models.tfidfmodel:calculating IDF weights for 18 documents and 42019 features (121967 matrix non-zeros)\n"
     ]
    }
   ],
   "source": [
    "encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)\n",
    "encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)\n",
    "encoded_tfidf = gensim.models.TfidfModel(encoded_gutenbow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "after calculating the frequencies, its time to transform the given bag-of-words vectors to corresponding tf-idf weights vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)\n",
    "encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)\n",
    "encoded_guten_tfidf = encoded_tfidf[encoded_gutenbow]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "example of token ids and their tf-idf weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token >> (token id, tf-idf weight)\n",
      "\n",
      "v>@CDuT4Yoss>twzY&Z6^aTR}!CP6m(ht&fRM`eh >> (0, 0.5042503721243573)\n",
      "S?VWSP0PVQ*Ze<nSimZQ|Lq4->d6xKrh5Wh+o7tE >> (2, 0.14308755923657526)\n",
      "o9zYE@{v;UprEjBZwc9v0PVlWPZ`e^i$9w29K<0c >> (3, 0.00047537395095207725)\n",
      "ed>e_?cy=_9AX6Nk#;<yNp?mo_lxn_n{6{=1k2@i >> (4, 0.00039197866090719016)\n",
      "@e|=)yTLqzKhfIlyTg&UdGw26F`IG&lYaQs!d4=U >> (6, 0.012048339086347707)\n",
      "p%(T2Bw%|_G&T!P#*xLbVSwta9jyeHP@W9(JXY?A >> (7, 0.24002347235604862)\n",
      "?|7@`_*&RzcZM9EQ39<D6tVAOO*n_wK!qM1gPP!U >> (8, 0.004965063038157742)\n",
      "PA3Y-CI+Mi^<nXVdA?y@m<abq->ak5G{>}LlFk}n >> (9, 0.0042105532568832305)\n",
      "WlnemVIhYBWej5dRSV&RU!6SIYRXl$dCgyDybv|r >> (11, 0.00021230678602490824)\n",
      "l&-X&S=o-LiHJSMg!)xpX&+8g=dY?p4v%g^oJzw# >> (14, 0.002935499650433135)\n"
     ]
    }
   ],
   "source": [
    "print('token', '>>', \"(token id, tf-idf weight)\\n\")\n",
    "for i in encoded_guten_tfidf:\n",
    "    for k in i[:10]:\n",
    "        print(encoded_gutendict.get(k[0]), '>>', k)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## validating tf-idf model on unencoded corpus\n",
    "\n",
    "now we will take the role of the corpus owner when she gets the results from the analyst back.\n",
    "\n",
    "we will validate the results in two steps:\n",
    "\n",
    "1. we will decode the previous result back to the unhashed tokens, maintaining their tf-idf weights;\n",
    "\n",
    "2. we will compare the tf-idf results with the same analysis done on the decoded corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1. decoding previous result\n",
    "\n",
    "we will turn the previous result from a tuple as in `(6, 0.0017337194574225342)` to something like `genesis: 0.0017337194574225342` and store it in disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "decode_dictionary_path = hashed_guten.decode_dictionary_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.isfile(decode_dictionary_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "obtaining decode dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with open(decode_dictionary_path, 'rb') as f:\n",
    "    decode_dictionary = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "here we are reinstantiating the generators, applying the tf-idf model, and then iterating over the model's results to replace the hashed tokens with their decoded counterparts. the result is saved to disk. make sure your indexer doesn't change document order (gutenberg.fileids does it), else the names may mismatch with the contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)\n",
    "encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)\n",
    "encoded_guten_tfidf = encoded_tfidf[encoded_gutenbow]\n",
    "for path, document in zip(gutenberg.fileids(), encoded_guten_tfidf):\n",
    "    decoded_document = []\n",
    "    for tuple_value in document:\n",
    "        hashed_token = encoded_gutendict.get(tuple_value[0]) # 6 - > ed07dbbe94c8ff385a1a00e6720f0ab66ac420...\n",
    "        token, _ = decode_dictionary[hashed_token]  # 'ed07dbbe94c8ff385a1a00e... -> 'genesis'\n",
    "        decoded_document.append(\"{}: {}\".format(token, tuple_value[1]))\n",
    "    fname = 'decoded_'+ path\n",
    "    with open(os.path.join(corpus_path, fname), 'w') as f:\n",
    "        f.write(\"\\n\".join(decoded_document))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "edgeworth-parents.txt >>\n",
      "chapter: 0.004383086322422836\n",
      "handsome: 0.0047913130193044\n",
      "clever: 0.00413009101386359\n",
      "rich: 0.0008959312959607944\n",
      "comfortable: 0.0007035645947490613\n",
      "disposition: 0.002859293778828639\n",
      "seemed: 0.002956349779127873\n",
      "existence: 0.0003176993087587376\n",
      "lived: 0.005025921344935186\n",
      "nearly: 0.003166040676370776\n"
     ]
    }
   ],
   "source": [
    "example_id = random.choice(gutenberg.fileids())\n",
    "with open(os.path.join(corpus_path, 'decoded_' + example_id), 'r') as f:\n",
    "        decoded_doc = f.read().splitlines()\n",
    "print(example_id, '>>')\n",
    "print(\"\\n\".join(decoded_doc[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2. tf-idf on decoded corpus\n",
    "\n",
    "#### creating unencoded corpus dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "the next step in our validation procedure is to apply the same analysis to the the decoded corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "the input files are not in the `/public/$(timestamp)` directory, but in the decoded_gutencorpus variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "the files are, again, decoded, in contrast to the hashed tokens seen above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['emma', 'by', 'jane', 'austen', 'volume', 'i', 'chapter', 'i', 'emma', 'woodhouse']\n"
     ]
    }
   ],
   "source": [
    "example_document = random.choice(decoded_gutencorpus)\n",
    "print(example_document[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "creating the dictionary that maps a token to an ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gensim.corpora.dictionary:adding document #0 to Dictionary(0 unique tokens: [])\n",
      "INFO:gensim.corpora.dictionary:built Dictionary(42020 unique tokens: ['emma', 'by', 'jane', 'austen', 'volume']...) from 18 documents (total 2161659 corpus positions)\n"
     ]
    }
   ],
   "source": [
    "decoded_gutendict = gensim.corpora.Dictionary(decoded_gutencorpus)\n",
    "#decoded_gutendict.save_as_text('dec_dict.txt', sort_by_word=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42020"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(decoded_gutendict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "creating a generator that yields the bag-of-words model of a document when iterated over:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token >> (token id, frequency in document)\n",
      "\n",
      "emma >> (0, 865)\n",
      "by >> (1, 571)\n",
      "jane >> (2, 301)\n",
      "austen >> (3, 1)\n",
      "volume >> (4, 3)\n",
      "i >> (5, 3178)\n",
      "chapter >> (6, 56)\n",
      "woodhouse >> (7, 313)\n",
      "handsome >> (8, 38)\n",
      "clever >> (9, 27)\n"
     ]
    }
   ],
   "source": [
    "print('token', '>>', \"(token id, frequency in document)\\n\")\n",
    "for i in next(decoded_gutenbow)[:10]:\n",
    "    print(decoded_gutendict.get(i[0]), '>>', i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### tf-idf model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "creating the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:gensim.models.tfidfmodel:collecting document frequencies\n",
      "INFO:gensim.models.tfidfmodel:PROGRESS: processing document #0\n",
      "INFO:gensim.models.tfidfmodel:calculating IDF weights for 18 documents and 42019 features (121967 matrix non-zeros)\n"
     ]
    }
   ],
   "source": [
    "decoded_tfidf = gensim.models.TfidfModel(decoded_gutenbow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "applying the model to the documents (not forgetting to reinstantiate the generators):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)\n",
    "decoded_guten_tfidf = decoded_tfidf[decoded_gutenbow]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "token >> (token id, tf-idf weight)\n",
      "\n",
      "emma >> (0, 0.5042503721243573)\n",
      "jane >> (2, 0.14308755923657526)\n",
      "austen >> (3, 0.00047537395095207725)\n",
      "volume >> (4, 0.00039197866090719016)\n",
      "chapter >> (6, 0.012048339086347707)\n",
      "woodhouse >> (7, 0.24002347235604862)\n",
      "handsome >> (8, 0.004965063038157742)\n",
      "clever >> (9, 0.0042105532568832305)\n",
      "rich >> (11, 0.00021230678602490824)\n",
      "comfortable >> (14, 0.002935499650433135)\n"
     ]
    }
   ],
   "source": [
    "print('token', '>>', \"(token id, tf-idf weight)\\n\")\n",
    "for i in decoded_guten_tfidf:\n",
    "    for k in i[:10]:\n",
    "        print(decoded_gutendict.get(k[0]), '>>', k)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### comparing results of encoded and decoded corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "reinstatiating the generators..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "encoded_gutencorpus = encoded_gutenberg_yielder(encoded_corpus_path)\n",
    "encoded_gutenbow = bow_gutenberg_yielder(encoded_gutencorpus, encoded_gutendict)\n",
    "encoded_guten_tfidf = encoded_tfidf[encoded_gutenbow]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "decoded_gutenbow = bow_gutenberg_yielder(decoded_gutencorpus, decoded_gutendict)\n",
    "decoded_guten_tfidf = decoded_tfidf[decoded_gutenbow]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "CPU times: user 2.05 s, sys: 30 ms, total: 2.08 s\n",
      "Wall time: 2.08 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "encoded_tfidf, decoded_tfidf = {}, {}\n",
    "for encoded_document, decoded_document in zip(encoded_guten_tfidf, decoded_guten_tfidf):\n",
    "    for encoded_item, decoded_item in zip(encoded_document, decoded_document):\n",
    "        hashed_token = encoded_gutendict.get(encoded_item[0])\n",
    "        original_token = decode_dictionary[hashed_token][0] # get hash, ignoring salt\n",
    "        encoded_tfidf[original_token] = round(encoded_item[1], 7) # rounding because python <3.6 seems to represent floats inconsistently\n",
    "        decoded_tfidf[decoded_gutendict.get(decoded_item[0])] = round(decoded_item[1], 7)\n",
    "print(encoded_tfidf == decoded_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example token: tf-idf weight in encoded corpus | in decoded corpus\n",
      "            nourisheth             : 0.0002203 | 0.0002203\n"
     ]
    }
   ],
   "source": [
    "random_token = random.choice(list(encoded_tfidf.keys()))\n",
    "print(\"example token: tf-idf weight in encoded corpus | in decoded corpus\\n{:^35}: {} | {}\".format(random_token, encoded_tfidf[random_token], decoded_tfidf[random_token]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### thus we see that the NLP's results are the same regardless of which corpus we use, i.e., we can use hashed corpora to perform NLP tasks in a lossless manner."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}