Skip to content

Latest commit

 

History

History
974 lines (731 loc) · 106 KB

natural-language-processing.md

File metadata and controls

974 lines (731 loc) · 106 KB

Natural Language Processing

NLP - a reality check

  1. A powerful benchmark, paper, medium - normalizing data sets allows us to see that there wasn't any advancement in terms of metrics in many NLP algorithms.

TOOLS

SPACY

  1. Vidhaya on spacy vs ner - tutorial + code on how to use spacy for pos, dep, ner, compared to nltk/corenlp (sner etc). The results reflect a global score not specific to LOC for example.
  2. The spaCy course
  3. SPACY OPTIMIZATION - LP using CYTHON and SPACY.

NLP embedding repositories

  1. Nlpl

NLP DATASETS

  1. The bid bad 600, medium

NLP Libraries

  1. Has all the known libraries
  2. Comparison between spacy, pytorch, allenlp - very basic info
  3. Comparison spacy,nltk core nlp
  4. Comparing Production grade nlp libs
  5. nltk vs spacy

Multilingual models

  1. Fb’s laser
  2. Xlm, xlm-r
  3. Google universal embedding space.

Augmenting text in NLP

  1. Synonyms, similar embedded words (w2v), back translation, contextualized word embeddings, text generation
  2. Yonatan hadar also has a medium post about this

TF-IDF

TF-IDF - how important is a word to a document in a corpus

TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).

Frequency of word in doc / all words in document (normalized bcz docs have diff sizes)

IDF(t) = log_e(Total number of documents / Number of documents with term t in it).

measures how important a term is

TF-IDF is TF*IDF

  1. A much clearer explanation plus python code, part 2
  2. Get top tfidf keywords
  3. Print top features

Data sets:

  1. Fast text multilingual
  2. NLP embeddings

Sparse textual content

  1. mean(IDF(i) * w2v word vectors (i)) with or without reducing PC1 from the whole w2 average (amir pupko)

def mean_weighted_embedding(model, words, idf=1.0):

**if words:**

    **return np.mean\(idf \* model\[words\], axis=0\)a**

**else:**

    **print\('we have an empty list'\)**

    **return \[\]**

idf_mapping = dict(zip(vectorizer.get_feature_names(), vectorizer.idf_))

logs_sequences_df['idf_vectors'] = logs_sequences_df.message.apply(lambda x: [idf_mapping[token] for token in splitter(x)])

logs_sequences_df['mean_weighted_idf_w2v'] = [mean_weighted_embedding(ft, splitter(logs_sequences_df['message'].iloc[i]), 1 / np.array(logs_sequences_df['idf_vectors'].iloc[i]).reshape(-1,1)) for i in range(logs_sequences_df.shape[0])]

  1. Multiply by TFIDF
  2. Enriching using lstm-next word (char or word-wise)
  3. Using external wiktionary/pedia data for certain words, phrases
  4. Finding clusters of relevant data and figuring out if you can enrich based on the content of the clusters
  5. Applying deep nlp methods without big data, i.e., sparseness

Basic nlp

  1. Benchmarking tokenizers for optimalprocessing speed
  2. Using nltk with gensim
  3. Multiclass text classification with svm/nb/mean w2v/d2v - tutorial with code and notebook.
  4. Basic pipeline for keyword extraction
  5. DL for text classification
    1. Logistic regression with word ngrams
    2. Logistic regression with character ngrams
    3. Logistic regression with word and character ngrams
    4. Recurrent neural network (bidirectional GRU) without pre-trained embeddings
    5. Recurrent neural network (bidirectional GRU) with GloVe pre-trained embeddings
    6. Multi channel Convolutional Neural Network
    7. RNN (Bidirectional GRU) + CNN model
  6. LexNLP - glorified regex extractor

Chunking

  1. Coding Chunkers as Taggers: IO, BIO, BMEWO, and BMEWO+

NLP for hackers tutorials

  1. How to convert between verb/noun/adjective/adverb forms using Wordnet
  2. Complete guide for training your own Part-Of-Speech Tagger - using Penn Treebank tagset. Using nltk or stanford pos taggers, creating features from actual words (manual stemming, etc0 using the tags as labels, on a random forest, thus creating a classifier for POS on our own. Not entirely sure why we need to create a classifier from a “classifier”.
  3. Word net introduction - POS, lemmatize, synon, antonym, hypernym, hyponym
  4. Sentence similarity using wordnet - using synonyms cumsum for comparison. Today replaced with w2v mean sentence similarity.
  5. Stemmers vs lemmatizers - stemmers are faster, lemmatizers are POS / dictionary based, slower, converting to base form.
  6. Chunking - shallow parsing, compared to deep, similar to NER
  7. NER - using nltk chunking as a labeller for a classifier, training one of our own. Using IOB features as well as others to create a new ner classifier which should be better than the original by using additional features. Aso uses a new english dataset GMB.
  8. Building nlp pipelines, functions coroutines etc..
  9. Training ner using generators
  10. Metrics, tp/fp/recall/precision/micro/weighted/macro f1
  11. Tf-idf
  12. Nltk for beginners
  13. Nlp corpora corpuses
  14. bow/bigrams
  15. Textrank
  16. Word cloud
  17. Topic modelling using gensim, lsa, lsi, lda,hdp
  18. Spacy full tutorial
  19. POS using CRF

Synonyms

  1. Python Module to get Meanings, Synonyms and what not for a given word using vocabulary (also a comparison against word net) https://vocabulary.readthedocs.io/en/…

For a given word, using Vocabulary, you can get its

  • Meaning
  • Synonyms
  • Antonyms
  • Part of speech : whether the word is a noun, interjection or an adverb et el
  • Translate : Translate a phrase from a source language to the desired language.
  • Usage example : a quick example on how to use the word in a sentence
  • Pronunciation
  • Hyphenation : shows the particular stress points(if any)

Swiss army knife libraries

  1. textacy is a Python library for performing a variety of natural language processing (NLP) tasks, built on the high-performance spacy library. With the fundamentals — tokenization, part-of-speech tagging, dependency parsing, etc. — delegated to another library, textacy focuses on the tasks that come before and follow after.

Collocation

  1. What is collocation? - “the habitual juxtaposition of a particular word with another word or words with a frequency greater than chance.”Medium tutorial, quite good, comparing freq/t-test/pmi/chi2 with github code
  2. A website dedicated to collocations, methods, references, metrics.
  3. Text analysis for sentiment, doing feature selection a tutorial with chi2(IG?), part 2 with bi-gram collocation in ntlk
  4. Text2vec in R - has ideas on how to use collocations, for downstream tasks, LDA, W2V, etc. also explains about PMI and other metrics, note that gensim metric is unsupervised and probablistic.
  5. NLTK on collocations
  6. A blog post about keeping or removing stopwords for collocation, usefull but no firm conclusion. Imo we should remove it before
  7. A blog post with code of using nltk-based collocation
  8. Small code for using nltk collocation
  9. Another code / score example for nltk collocation
  10. Jupyter notebook on manually finding collocation - not useful
  11. Paper: Ngram2Vec - Github We introduce ngrams into four representation methods. The experimental results demonstrate ngrams’ effectiveness for learning improved word representations. In addition, we find that the trained ngram embeddings are able to reflect their semantic meanings and syntactic patterns. To alleviate the costs brought by ngrams, we propose a novel way of building co-occurrence matrix, enabling the ngram-based models to run on cheap hardware
  12. Youtube on bigrams, collocation, mutual info and collocation

Language detection

  1. Using google lang detect - 55 languages af, ar, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he, hi, hr, hu, id, it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl, pt, ro, ru, sk, sl, so, sq, sv, sw, ta, te, th, tl, tr, uk, ur, vi, zh-cn, zh-tw

Stemming

How to measure a stemmer?

  1. References [1 ****2(apr11) 3(Index compression factor ICF) 4 ****5]

Phrase modelling

  1. Phrase Modeling - using gensim and spacy

Phrase modeling is another approach to learning combinations of tokens that together represent meaningful multi-word concepts. We can develop phrase models by looping over the the words in our reviews and looking for words that co-occur (i.e., appear one after another) together much more frequently than you would expect them to by random chance. The formula our phrase models will use to determine whether two tokens AA and BB constitute a phrase is:

count(A B)−countmincount(A)∗count(B)∗N>threshold

  1. SO on PE.

Document classification

  1. Using hierarchical attention network

Hebrew NLP tools

  1. HebMorph last update 7y ago
  2. Hebmorph elastic search ****Hebmorph blog post, and other blog posts, youtube
  3. Awesome hebrew nlp git, git
  4. Hebrew-nlp service ****docs ****the features (morphological analysis, normalization etc), git
  5. Apache solr stop words (dead)
  6. SO on hebrew analyzer/stemming, here too
  7. Neural sentiment benchmark using two algorithms, for character and word level lstm/gru - the paper
  8. Hebrew word embeddings
  9. Paper for rich morphological datasets for comparison - rivlin

Semantic roles:

  1. http://language.worldofcomputing.net/semantics/semantic-roles.html

ANNOTATION

  1. Snorkel - using weak supervision to create less noisy labelled datasets
    1. Git
    2. Medium
  2. Snorkel metal weak supervision for multi-task learning. Conversation, git
    1. Yes, the Snorkel project has included work before on hierarchical labeling scenarios. The main papers detailing our results include the DEEM workshop paper you referenced (https://dl.acm.org/doi/abs/10.1145/3209889.3209898) and the more complete paper presented at AAAI (https://arxiv.org/abs/1810.02840). Before the Snorkel and Snorkel MeTaL projects were merged in Snorkel v0.9, the Snorkel MeTaL project included an interface for explicitly specifying hierarchies between tasks which was utilized by the label model and could be used to automatically compile a multi-task end model as well (demo here: https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb). That interface is not currently available in Snorkel v0.9 (no fundamental blockers; just hasn't been ported over yet).
    2. There are, however, still a number of ways to model such situations. One way is to treat each node in the hierarchy as a separate task and combine their probabilities post-hoc (e.g., P(credit-request) = P(billing) * P(credit-request | billing)). Another is to treat them as separate tasks and use a multi-task end model to implicitly learn how the predictions of some tasks should affect the predictions of others (e.g., the end model we use in the AAAI paper). A third option is to create a single task with all the leaf categories and modify the output space of the LFs you were considering for the higher nodes (the deeper your hierarchy is or the larger the number of classes, the less apppealing this is w/r/t to approaches 1 and 2).
  3. mechanical turk calculator
  4. Mturk alternatives
    1. Workforce / onespace
    2. Jobby
    3. Shorttask
    4. Samasource
    5. Figure 8 - pricing - definite guide
  5. Brat nlp annotation tool
  6. Prodigy by spacy,
    1. seed-small sample, many sample tutorial on youtube by ines
    2. How to use prodigy, tutorial on medium plus notebook code inside
  7. Doccano - prodigy open source alternative butwith users management & statistics out of the box
  8. Medium Lighttag - has some cool annotation metrics\tests
  9. Loopr.ai - An AI powered semi-automated and automated annotation process for high quality data.object detection, analytics, nlp, active learning.
  10. Medium Assessing annotator disagreement
  11. A great python package for measuring disagreement on GH
  12. Reliability is key, and not just mechanical turk
  13. 7 myths about annotation
  14. Annotating twitter sentiment using humans, 3 classes, 55% accuracy using SVMs. they talk about inter agreement etc. and their DS is partially publicly available.
  15. Exploiting disagreement
  16. Vader annotation
    1. They must pass an english exam
    2. They get control questions to establish their reliability
    3. They get a few sentences over and over again to establish inter disagreement
    4. Two or more people get a overlapping sentences to establish disagreement
    5. 5 judges for each sentence (makes 4 useless)
    6. They dont know each other
    7. Simple rules to follow
    8. Random selection of sentences
    9. Even classes
    10. No experts
    11. Measuring reliability kappa/the other kappa.
  17. Label studio

Ideas:

  1. Active learning for a group (or single) of annotators, we have to wait for all annotations to finish each big batch in order to retrain the model.
  2. Annotate a small group, automatic labelling using knn
  3. Find a nearest neighbor for out optimal set of keywords per “category,
  4. For a group of keywords, find their knn neighbors in w2v-space, alternatively find k clusters in w2v space that has those keywords. For a new word/mean sentence vector in the ‘category’ find the minimal distance to the new cluster (either one of approaches) and this is new annotation.

7 myths of annotation

  1. Myth One: One Truth Most data collection efforts assume that there is one correct interpretation for every input example.
  2. Myth Two: Disagreement Is Bad To increase the quality of annotation data, disagreement among the annotators should be avoided or reduced.
  3. Myth Three: Detailed Guidelines Help When specific cases continuously cause disagreement, more instructions are added to limit interpretations.
  4. Myth Four: One Is Enough Most annotated examples are evaluated by one person.
  5. Myth Five: Experts Are Better Human annotators with domain knowledge provide better annotated data.
  6. Myth Six: All Examples Are Created Equal The mathematics of using ground truth treats every example the same; either you match the correct result or not.
  7. Myth Seven: Once Done, Forever Valid Once human annotated data is collected for a task, it is used over and over with no update. New annotated data is not aligned with previous data.

Crowd Sourcing

  • Conclusions:
    • Experts are the same as a crowd
    • Costs a lot less $$$.

Inter agreement

*** The best tutorial on agreements, cohen, david, kappa, krip etc.

  1. Cohens kappa (two people)

but you can use it to map a group by calculating agreement for each pair

  1. Why cohens kappa should be avoided as a performance measure in classification
  2. Why it should be used as a measure of classification
  3. Kappa in plain english
  4. Multilabel using kappa
  5. Kappa and the relation with accuracy (redundant, % above chance, should not be used due to other reasons researched here)

The Kappa statistic varies from 0 to 1, where.

  • 0 = agreement equivalent to chance.
  • 0.1 – 0.20 = slight agreement.
  • 0.21 – 0.40 = fair agreement.
  • 0.41 – 0.60 = moderate agreement.
  • 0.61 – 0.80 = substantial agreement.
  • 0.81 – 0.99 = near perfect agreement
  • 1 = perfect agreement.
  1. Fleiss’ kappa, from 3 people and above.

Kappa ranges from 0 to 1, where:

  • 0 is no agreement (or agreement that you would expect to find by chance),
  • 1 is perfect agreement.
  • Fleiss’s Kappa is an extension of Cohen’s kappa for three raters or more. In addition, the assumption with Cohen’s kappa is that your raters are deliberately chosen and fixed. With Fleiss’ kappa, the assumption is that your raters were chosen at random from a larger population.
  • Kendall’s Tau is used when you have ranked data, like two people ordering 10 candidates from most preferred to least preferred.
  • Krippendorff’s alpha is useful when you have multiple raters and multiple possible ratings.
  1. Krippendorfs alpha
  1. MACE - the new kid on the block. -

learns in an unsupervised fashion to

  1. a) identify which annotators are trustworthy and
  2. b) predict the correct underlying labels. We match performance of more complex state-of-the-art systems and perform well even under adversarial conditions
  3. MACE does exactly that. It tries to find out which annotators are more trustworthy and upweighs their answers.
  4. Git -

When evaluating redundant annotatio

ns (like those from Amazon's MechanicalTurk), we usually want to

  1. aggregate annotations to recover the most likely answer
  2. find out which annotators are trustworthy
  3. evaluate item and task difficulty

MACE solves all of these problems, by learning competence estimates for each annotators and computing the most likely answer based on those competences.

Calculating agreement

  1. Compare against researcher-ground-truth
  2. Self-agreement
  3. Inter-agreement
    1. Medium
    2. Kappa cohen
    3. Multi annotator with kappa (which isnt), is this okay?
    4. Github computer Fleiss Kappa 1
    5. Fleiss Kappa Example
    6. GWET AC1, paper: as an alternative to kappa, and why
    7. Website, krippensorf vs fleiss calculator

Machine Vision annotation

  1. CVAT

Troubling shooting agreement metrics

  1. Imbalance data sets, i.e., why my Why is reliability so low when percentage of agreement is high?
  2. Interpretation of kappa values
  3. Interpreting agreement, Accuracy precision kappa

CONVOLUTION NEURAL NETS (CNN)

  1. Cnn for text - tal perry
  2. 1D CNN using KERAS

KNOWLEDGE GRAPHS

  1. Automatic creation of KG using spacy and networx Knowledge graphs can be constructed automatically from text using part-of-speech and dependency parsing. The extraction of entity pairs from grammatical patterns is fast and scalable to large amounts of text using NLP library SpaCy.
  2. Medium on Reconciling your data and the world of knowledge graphs
  3. Medium Series:
    1. Creating kg
    2. Building from structured sources
    3. Semantic models

SUMMARIZATION

  1. Email summarization but with a great intro (see image above)
  2. With nltk - words assigned weighted frequency, summed up in sentences and then selected based on the top K scored sentences.
  3. Awesome-text-summarization on github
  4. Methodical review of abstractive summarization
  5. Medium on extractive and abstractive - overview with the abstractive code
  6. NAMAS - Neural attention model for abstractive summarization, -Neural Attention Model for Abstractive Sentence Summarization - summarizes single sentences quite well, github
  7. Abstractive vs extractive, blue intro
  8. Intro to text summarization
  9. Paper: survey on text summ, arxiv
  10. Very short intro
  11. Intro on encoder decoder
  12. Unsupervised methods using sentence emebeddings (long and good) - using sent2vec, clustering, picking by rank
  13. Abstractive summarization using bert for sota
  14. Abstractive
    1. Git1: uses pytorch 0.7, fails to work no matter what i did
    2. Git2, keras code for headlines, missing dataset
    3. Encoder decoder in keras using rnn, claims cherry picked results, the majority is prob not as good
    4. A lot of Text summarization algos on git, using seq2seq, using many methods, glove, etc -
    5. Summarization with point generator networks on git
    6. Summarization based on gigaword claims SOTA
    7. Facebooks neural attention network NAMAS on git
    8. Medium on summarization with tensor flow on news articles from cnn
  15. Keywords extraction
    1. The best text rank presentation
    2. Text rank by gensim on medium
    3. Text rank 2
    4. Text rank - custom code, extractive vs abstractive, how to use, some more theoretical info and page rank intuition.
    5. Text rank paper
    6. Improving textrank using adjectival and noun compound modifiers
    7. Unread - New similarity function paper for textrank
  16. Extractive summarization
    1. Text rank with glove vectors instead of tf-idf as in the paper (sam)
    2. Medium with code on extractive using word occurrence similarity + cosine, pick top based on rank
    3. Medium on methods, freq, LSA, linking words, sentences,bayesian, graph ranking, hmm, crf,
    4. Wiki on automatic summarization, abstractive vs extractive,
    5. Pyteaser, textteaset, lexrank, pytextrank summarization models & rouge-1/n and blue metrics to determine quality of summarization models Bottom line is that textrank is competitive to sumy_lex
    6. Sumy
    7. Pyteaser
    8. Pytextrank
    9. Lexrank
    10. Gensim tutorial on textrank
    11. Email summarization

SENTIMENT ANALYSIS

Databases

  1. Sentiment databases ****
  2. Movie reviews: IMDB reviews dataset on Kaggle
  3. Sentiwordnet – mapping wordnet senses to a polarity model: SentiWordnet Site
  4. Twitter airline sentiment on Kaggle
  5. First GOP Debate Twitter Sentiment
  6. Amazon fine foods reviews

Tools

  1. ** Many Sentiment tools,
  2. NTLK sentiment analyzer
  3. Vader (NTLK, standalone):
    1. Vader/Sentiwordnet/etc python code examples - possibly good for ensembles
    2. **Intro into Vader
    3. Why vader?
    4. **Vader - a clear explanation about the paper’s methodology
    5. Simple Intro to Vader
    6. A very lengthy and overly complex explanation about using NTLK vader
    7. Vader tutorial, +-0.2 for neutrals.
  4. Text BLob:
    1. Text blob classification
    2. Python code
    3. More code
    4. A lengthy tutorial
    5. **Text blob sentiment analysis tutorial on medium
    6. A lengthy intro plus code about text blob
  5. Comparative opinion mining a review paper - has some info about unsupervised as well
  6. Another reference list, has some unsupervised.
  7. Sentiwordnet3.0 paper
  8. presentation

Reference papers:

  1. Twitter as a corpus for SA and opinion mining

Ground Truth

  1. For sentiment In Vader -
    1. “Screening for English language reading comprehension – each rater had to individually score an 80% or higher on a standardized college-level reading comprehension test.
    2. Complete an online sentiment rating training and orientation session, and score 90% or higher for matching the known (prevalidated) mean sentiment rating of lexical items which included individual words, emoticons, acronyms, sentences, tweets, and text snippets (e.g., sentence segments, or phrases).
    3. Every batch of 25 features contained five “golden items” with a known (pre-validated) sentiment rating distribution. If a worker was more than one standard deviation away from the mean of this known distribution on three or more of the five golden items, we discarded all 25 ratings in the batch from this worker.
    4. Bonus to incentivize and reward the highest quality work. Asked workers to select the valence score that they thought “most other people” would choose for the given lexical feature (early/iterative pilot testing revealed that wording the instructions in this manner garnered a much tighter standard deviation without significantly affecting the mean sentiment rating, allowing us to achieve higher quality (generalized) results while being more economical).
    5. Compensated AMT workers $0.25 for each batch of 25 items they rated, with an additional $0.25 incentive bonus for all workers who successfully matched the group mean (within 1.5 standard deviations) on at least 20 of 25 responses in each batch. Using these four quality control methods, we achieved remarkable value in the data obtained from our AMT workers – we paid incentive bonuses for high quality to at least 90% of raters for most batches.

Multilingual Twitter Sentiment Classification: The Role of Human Annotators

  • 1.6 million tweets labelled
  • 13 languages
  • Evaluated 6 pretrained classification models
  • 10 CFV
  • SVM / NB
  • Annotator agreements.
    • about 15% were intentionally duplicated to be annotated twice,
    • by the same annotator
    • by two different annotators
  • Self-agreement from multiple annotations of the same annotator
  • Inter-agreement from multiple annotations by different annotators
  • The confidence intervals for the agreements are estimated by bootstrapping [12].
  • It turns out that the self-agreement is a good measure to identify low quality annotators,
  • the inter-annotator agreement provides a good estimate of the objective difficulty of the task, unless it is too low.

Alpha was developed to measure the agreement between human annotators, but can also be used to measure the agreement between classification models and a gold standard. It generalizes several specialized agreement measures, takes ordering of classes into account, and accounts for the agreement by chance. Alpha is defined as follows:

Method cont here in a second paper

TOPIC MODELING

****A very good article about LSA (TFIDV X SVD), pLSA, LDA, and LDA2VEC. Including code and explanation about dirichlet probability. Lda2vec code

A descriptive comparison for LSA pLSA and LDA

A great summation about topic modeling, Pros and Cons! (LSA, pLSA, LDA)

Word cloud for topic modelling

Sklearn LDA and NMF for topic modelling

Topic modelling with sentiment per topic according to the data in the topic

(TopSBM) topic block modeling

  1. Topsbm

(LDA) Latent Dirichlet Allocation

LDA is already taken by the above algorithm!

Latent Dirichlet allocation (LDA) - This algorithm takes a group of documents (anything that is made of up text), and returns a number of topics (which are made up of a number of words) most relevant to these documents.

  • LDA is an example of topic modelling
  • ?- can this be used for any set of features, not just text?

Medium Article about LDA and NMF (Non-negative Matrix factorization)+ code

Medium article on LDA - a good one with pseudo algorithm and proof

In case LDA groups together two topics, we can influence the algorithm in a way that makes those two topics separable - this is called Semi Supervised Guided LDA

  • LDA tutorials plus code, used this to build my own classes - using gensim mallet wrapper, doesn't work on pyLDAviz, so use this to fix it
  • Introduction to LDA topic modelling, really good, ****plus git code
  • Sklearn examples using LDA and NMF
  • Tutorial on lda/nmf on medium - using tfidf matrix as input!
  • Gensim and sklearn LDA variants, comparison, python 3
  • Medium article on lda/nmf with code
  • One of the best explanation about Tf-idf vs bow for LDA/NMF - tf for lda, tfidf for nmf, but tfidf can be used for top k selection in lda + visualization, important paper
    • LDA is a probabilistic generative model that generates documents by sampling a topic for each word and then a word from the sampled topic. The generated document is represented as a bag of words.
    • NMF is in its general definition the search for 2 matrices W and H such that W*H=V where V is an observed matrix. The only requirement for those matrices is that all their elements must be non negative.
    • From the above definitions it is clear that in LDA only bag of words frequency counts can be used since a vector of reals makes no sense. Did we create a word 1.2 times? On the other hand we can use any non negative representation for NMF and in the example tf-idf is used.
    • As far as choosing the number of iterations, for the NMF in scikit learn I don't know the stopping criterion although I believe it is the relative improvement of the loss function being smaller than a threshold so you 'll have to experiment. For LDA I suggest checking manually the improvement of the log likelihood in a held out validation set and stopping when it falls under a threshold.
    • The rest of the parameters depend heavily on the data so I suggest, as suggested by @rpd, that you do a parameter search.
    • So to sum up, LDA can only generate frequencies and NMF can generate any non negative matrix.

Very important:

  1. What is LDA?
    1. It is unsupervised natively; it uses joint probability method to find topics(user has to pass # of topics to LDA api). If “Doc X word” is size of input data to LDA, it transforms it to 2 matrices:
    2. Doc X topic
    3. Word X topic
    4. further if you want, you can feed “Doc X topic” matrix to supervised algorithm if labels were given.
  2. Medium on LDA, explains the random probabilistic nature of LDA
  3. Machinelearningplus on LDA in sklearn - a great read, dont forget to read the mallet article.
  4. Medium on LSA pLSA, LDA LDA2vec, high level theoretical - not clear
  5. Medium on LSI vs LDA vs HDP, HDP wins..
  6. Medium on LDA, some historical reference and general high level how to use exapmles.
  7. Incredibly useful response on LDA grid search params and about LDA expectations. Must read.
  8. Lda vs pLSA, talks about the sampling from a distribution of distributions in LDA
  9. BLog post on topic modelling - has some text about overfitting - undiscussed in many places.
  10. Perplexity vs coherence on held out unseen data, not okay and okay, respectively. Due to how we measure the metrics, ie., read the formulas. Also this and this
  11. LDA as dimentionality reduction
  12. LDA on alpha and beta to control density of topics
  13. Jupyter notebook on hacknews LDA topic modelling - missing code?
  14. Jupyter notebook for kmeans, lda, svd,nmf comparison - advice is to keep nmf or other as a baseline to measure against LDA.
  15. Gensim on LDA with code
  16. Medium on lda with sklearn
  17. Selecting the number of topics in LDA, blog 1, blog2, using preplexity, prep and aic bic, coherence, coherence2, coherence 3 with tutorial, unclear, unclear with analysis of stopword % inclusion, unread, paper: heuristic approach, elbow method, using cv, Paper: new stability metric + gh code,
  18. Selecting the top K words in LDA
  19. Presentation: best practices for LDA
  20. Medium on guidedLDA - switching from LDA to a variation of it that is guided by the researcher / data
  21. Medium on lda - another introductory, la times
  22. Topic modelling through time
  23. Mallet vs nltk, params, params
  24. Paper: improving feature models
  25. Lda vs w2v (doesn't make sense to compare, again here
  26. Adding lda features to w2v for classification
  27. Spacy and gensim on 20 news groups
  28. The best topic modelling explanation including Usages, insights, a great read, with code - shows how to find similar docs by topic in gensim, and shows how to transform unseen documents and do similarity using sklearn:
    1. Text classification – Topic modeling can improve classification by grouping similar words together in topics rather than using each word as a feature
    2. Recommender Systems – Using a similarity measure we can build recommender systems. If our system would recommend articles for readers, it will recommend articles with a topic structure similar to the articles the user has already read.
    3. Uncovering Themes in Texts – Useful for detecting trends in online publications for example
    4. A Form of Tagging - If document classification is assigning a single category to a text, topic modeling is assigning multiple tags to a text. A human expert can label the resulting topics with human-readable labels and use different heuristics to convert the weighted topics to a set of tags.
    5. Topic Modelling for Feature Selection - Sometimes LDA can also be used as feature selection technique. Take an example of text classification problem where the training data contain category wise documents. If LDA is running on sets of category wise documents. Followed by removing common topic terms across the results of different categories will give the best features for a category.

How to interpret topics using pyldaviz:

Let’s interpret the topic visualization. Notice how topics are shown on the left while words are on the right. Here are the main things you should consider:

  1. Larger topics are more frequent in the corpus.
  2. Topics closer together are more similar, topics further apart are less similar.
  3. When you select a topic, you can see the most representative words for the selected topic. This measure can be a combination of how frequent or how discriminant the word is. You can adjust the weight of each property using the slider.
  4. Hovering over a word will adjust the topic sizes according to how representative the word is for the topic.

****pyLDAviz paper***!

pyLDAviz - what am i looking at ? by spacy

There are a lot of moving parts in the visualization. Here's a brief summary:

  • On the left, there is a plot of the "distance" between all of the topics (labeled as the Intertopic Distance Map)
    • The plot is rendered in two dimensions according a multidimensional scaling (MDS) algorithm. Topics that are generally similar should be appear close together on the plot, while dissimilar topics should appear far apart.
    • The relative size of a topic's circle in the plot corresponds to the relative frequency of the topic in the corpus.
    • An individual topic may be selected for closer scrutiny by clicking on its circle, or entering its number in the "selected topic" box in the upper-left.
  • On the right, there is a bar chart showing top terms.
    • When no topic is selected in the plot on the left, the bar chart shows the top-30 most "salient" terms in the corpus. A term's saliency is a measure of both how frequent the term is in the corpus and how "distinctive" it is in distinguishing between different topics.
    • When a particular topic is selected, the bar chart changes to show the top-30 most "relevant" terms for the selected topic. The relevance metric is controlled by the parameter λλ, which can be adjusted with a slider above the bar chart.
      • Setting the λλ parameter close to 1.0 (the default) will rank the terms solely according to their probability within the topic.
      • Setting λλ close to 0.0 will rank the terms solely according to their "distinctiveness" or "exclusivity" within the topic — i.e., terms that occur only in this topic, and do not occur in other topics.
      • Setting λλ to values between 0.0 and 1.0 will result in an intermediate ranking, weighting term probability and exclusivity accordingly.
  • Rolling the mouse over a term in the bar chart on the right will cause the topic circles to resize in the plot on the left, to show the strength of the relationship between the topics and the selected term.

A more detailed explanation of the pyLDAvis visualization can be found here. Unfortunately, though the data used by gensim and pyLDAvis are the same, they don't use the same ID numbers for topics. If you need to match up topics in gensim's LdaMulticore object and pyLDAvis' visualization, you have to dig through the terms manually.

  1. Another great article about LDA, including algorithm, parameters!! And

Parameters of LDA

  1. Alpha and Beta Hyperparameters – alpha represents document-topic density and Beta represents topic-word density. Higher the value of alpha, documents are composed of more topics and lower the value of alpha, documents contain fewer topics. On the other hand, higher the beta, topics are composed of a large number of words in the corpus, and with the lower value of beta, they are composed of few words.
  2. Number of Topics – Number of topics to be extracted from the corpus. Researchers have developed approaches to obtain an optimal number of topics by using Kullback Leibler Divergence Score. I will not discuss this in detail, as it is too mathematical. For understanding, one can refer to this[1] original paper on the use of KL divergence.
  3. Number of Topic Terms – Number of terms composed in a single topic. It is generally decided according to the requirement. If the problem statement talks about extracting themes or concepts, it is recommended to choose a higher number, if problem statement talks about extracting features or terms, a low number is recommended.
  4. Number of Iterations / passes – Maximum number of iterations allowed to LDA algorithm for convergence.

Ways to improve LDA:

  1. Reduce dimentionality of document-term matrix
  2. Frequency filter
  3. POS filter
  4. Batch wise LDA
  5. History of LDA - by the frech guy
  6. Diff between lda and mallet - The inference algorithms in Mallet and Gensim are indeed different. Mallet uses Gibbs Sampling which is more precise than Gensim's faster and online Variational Bayes. There is a way to get relatively performance by increasing number of passes.
  7. Mallet in gensim blog post
  8. Alpha beta in mallet: contribution
    1. The default for alpha is 5.0 divided by the number of topics. You can think of this as five "pseudo-words" of weight on the uniform distribution over topics. If the document is short, we expect to stay closer to the uniform prior. If the document is long, we would feel more confident moving away from the prior.
    2. With hyperparameter optimization, the alpha value for each topic can be different. They usually become smaller than the default setting.
    3. The default value for beta is 0.01. This means that each topic has a weight on the uniform prior equal to the size of the vocabulary divided by 100. This seems to be a good value. With optimization turned on, the value rarely changes by more than a factor of two.
  9. Multilingual - alpha is divided by topic count, reaffirms 7
  10. Topic modelling with lda and nmf on medium - has a very good simple example with probabilities
  11. Code: great for top docs, terms, topics etc.
  12. Great article: Many ways of evaluating topics by running LDA
  13. Youtube on LDAvis explained
  14. Presentation: More visualization options including ldavis
  15. A pointer to the ldaviz fix -> fix, git code
  16. Difference between lda in gensim and sklearn a post on rare
  17. The best code article on LDA/MALLET, and using sklearn (using clustering for getting group of sentences in each topic)
  18. LDA in gensim, a tutorial by gensim
  19. ****Lda on medium ****
  20. ****What are the pros and cons of LDA and NMF in topic modeling? Under what situations should we choose LDA or NMF? Is there comparison of two techniques in topic modeling?
  21. What is the difference between NMF and LDA? Why are the priors of LDA sparse-induced?
  22. Exploring Topic Coherence over many models and many topics lda nmf svd, using umass and uci coherence measures
  23. *** Practical topic findings for short sentence text code
  24. What's the difference between SVD/NMF and LDA as topic model algorithms essentially? Deterministic vs prob based
  25. What is the difference between NMF and LDA? Why are the priors of LDA sparse-induced?
  26. What are the relationships among NMF, tensor factorization, deep learning, topic modeling, etc.?
  27. Code: lda nmf
  28. Unread a comparison of lda and nmf
  29. Presentation: lda sparse coding matrix factorization
  30. An experimental comparison between NMF and LDA for active cross-situational object-word learning
  31. Topic coherence in gensom with jupyter code
  32. Topic modelling dynamic presentation
  33. Paper: Topic modelling and event identification from twitter data, says LDA vs NMI (NMF?) and using coherence to analyze
  34. Just another medium article about ™
  35. What is Wrong with Topic Modeling? (and How to Fix it Using Search-based SE) LDADE's tunings dramatically reduces topic instability.
  36. Talk about topic modelling
  37. Intro to topic modelling
  38. Detecting topics in twitter github code
  39. Another topic model tutorial
  40. (didnt read) NTM - neural topic modeling using embedded spaces with github code
  41. Another lda tutorial
  42. Comparing tweets using lda
  43. Lda and w2v as features for some classification task
  44. Improving ™ with embeddings
  45. w2v/doc2v for topic clustering - need to see the code to understand how they got clean topics, i assume a human rewrote it

Topic coherence (lda/nmf)

  1. What is?, Wiki on pmi
  2. Datacamp on coherence metrics, a comparison, read me.
  3. Paper: explains what is coherence

  1. Umass vs C_v, what are the diff?
  2. Paper: umass, uci, nmpi, cv, cp etv Exploring the Space of Topic Coherence Measures
  3. Paper: Automatic evaluation of topic coherence ****
  4. Paper: exploring the space of topic coherence methods
  5. Paper: Relation between mutial information / entropy and pmi
  6. Stackexchange: coherence / pmi how to calc
  7. Paper: Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality - perplexity needs unseen data, coherence doesnt
  8. Evaluation of topic modelling techniques for twitter lda lda-u btm w2vgmm
  9. Paper: Topic coherence measures
  10. topic modelling from different domains
  11. Paper: Optimizing Semantic Coherence in Topic Models
  12. Paper: L-EnsNMF: Boosted Local Topic Discovery via Ensemble of Nonnegative Matrix Factorization
  13. Paper: Content matching between TV shows and advertisements through Latent Dirichlet Allocation
  14. Paper: Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation
  15. Paper: Evaluating topic coherence - Abstract: Topic models extract representative word sets—called topics—from word counts in documents without requiring any semantic annotations. Topics are not guaranteed to be well interpretable, therefore, coherence measures have been proposed to distinguish between good and bad topics. Studies of topic coherence so far are limited to measures that score pairs of individual words. For the first time, we include coherence measures from scientific philosophy that score pairs of more complex word subsets and apply them to topic scoring.

Conclusion: The results of the first experiment show that if we are using the one-any, any-any and one-all coherences directly for optimization they are leading to meaningful word sets. The second experiment shows that these coherence measures are able to outperform the UCI coherence as well as the UMass coherence on these generated word sets. For evaluating LDA topics any-any and one-any coherences perform slightly better than the UCI coherence. The correlation of the UMass coherence and the human ratings is not as high as for the other coherences.

  1. Code: Evaluating topic coherence, using gensim umass or cv parameter - To conclude, there are many other approaches to evaluate Topic models such as Perplexity, but its poor indicator of the quality of the topics.Topic Visualization is also a good way to assess topic models. Topic Coherence measure is a good way to compare difference topic models based on their human-interpretability.The u_mass and c_v topic coherences capture the optimal number of topics by giving the interpretability of these topics a number called coherence score.
  2. Formulas: UCI vs UMASS
  3. Inferring the number of topics for gensim's LDA - perplexity, CM, AIC, and BIC
  4. Perplexity as a measure for LDA
  5. Finding number of topics using perplexity
  6. Coherence for tweets
  7. Presentation Twitter DLA, tweet pooling improvements, hierarchical summarization of tweets, twitter LDA in java ****on github Papers: TM of twitter timeline, in twitter aggregation by conversatoin, twitter topics using LDA, empirical study ,

COHERENCE

Non-negative Matrix factorization (NMF)

Medium Article about LDA and NMF (Non-negative Matrix factorization)+ code

LDA2VEC

  1. “if you want to rework your own topic models that, say, jointly correlate an article’s topics with votes or predict topics over users then you might be interested in lda2vec.”
  2. Datacamp intro
  3. Original blog - I just learned about these papers which are quite similar: Gaussian LDA for Topic Word Embeddings and Nonparametric Spherical Topic Modeling with Word Embeddings.
  4. Moody’s Slide Share (excellent read)
  5. Docs
  6. Original Git + Excellent notebook example
  7. Tf implementation, another more recent one tf 1.5
  8. Another blog explaining about lda etc, post, post
  9. Lda2vec in tf, tf 1.5,
  10. Comparing lda2vec to lda
  11. Youtube: lda/doc2vec with pca examples
  12. Example on gh on jupyter

TOP2VEC

  1. Git, paper
  2. Topic modeling with distillibert on medium, bertTopic!, c-tfidf, umap, hdbscan, merging similar topics, visualization, berTopic (same method as the above)
  3. Medium with the same general method

NAMED ENTITY RECOGNITION (NER)

Easy to implement in keras! They are based on the following paper

  • Medium: NLTK entities, polyglot entities, sner entities, finally an ensemble method wins all!

  • Vidhaya on spacy vs ner - tutorial + code on how to use spacy for pos, dep, ner, compared to nltk/corenlp (sner etc). The results reflect a global score not specific to LOC for example.

Stanford NER (SNER)

Spacy & Others

SEARCH

  1. Bert search engine, cosine between paragraphs and question.
  2. Semantic search, autom completion, filtering, augmentation, scoring. Problems: Token matching, contextualization, query misunderstanding, image search, metric. Solutions: synonym generation, query autocompletion, alternate query generation, word and doc embedding, contextualization, ranking, ensemble, multilingual search

SEQ2SEQ SEQUENCE TO SEQUENCE

  1. Keras blog - char-level, token-using embedding layer, teacher forcing
  2. Teacher forcing explained
  3. Same as keras but with token-level
  4. Medium on char, word, byte-level
  5. Mastery on enc-dec using the keras method, and on neural translation
  6. Machine translation git from eng to jap, another, and its medium

  1. Incorporating Copying Mechanism in Sequence-to-Sequence Learning - In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence.

QUESTION ANSWERING

  1. Pythia, qna for images on colab
  2. Building a Q&A system part 1
  3. Building a Q&A model
  4. Vidhya on Q&A
  5. Q&A system using ****milvus - An open source embedding vector similarity search engine powered by Faiss, NMSLIB and Annoy
  6. Q&A system

LANGUAGE MODELS

  1. Mastery on Word-based

NEURAL LANGUAGE GENERATION

  1. Using RNN
  2. Using language modeling
  3. Word based vs char based - Word-based LMs display higher accuracy and lower computational cost than char-based LMs. However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised. Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.
  4. mediu m on Char based with code, leads to better grammer
  5. Git, keras language models, char level word level and sentence using VAE

LANGUAGE DETECTION / IDENTIFICATION

  1. A qualitative comparison of google, azure, amazon, ibm LD LI
  2. CLD2, CLD3, PYCLD2, POLYGLOT wraps CLD, alex ott cld stats, cld comparison vs tika langid
  3. Fast text LI, facebook post
  4. OPENNLP
  5. Google detect language, github code, v3beta
  6. Microsoft azure LD, ****2
  7. Ibm watson, 2
  8. Amazon, **** 2
  9. Lingua - most accurate for java… doesn't seem like its accurate enough
  10. LD with infinity gram 99.1 on a lot of data a benchmark for this 2012 method, LD with infinity gram
  11. WiLI dataset for LD, comparison of CLD vs others
  12. Comparison of CLD vs FT vs OPEN NLP - beware based on 200 samples per language!!

Full results for every language that I tested are in table at the end of blog post & on Github. From them I can make following conclusions:

  • all detectors are equally good on some languages, such as, Japanese, Chinese, Vietnamese, Greek, Arabic, Farsi, Georgian, etc. - for them the accuracy of detection is between 98 & 100%;
  • CLD is much better in detection of "rare" languages, especially for languages, that are similar to more frequently used - Afrikaans vs Dutch, Azerbaijani vs. Turkish, Malay vs. Indonesian, Nepali vs. Hindi, Russian vs Bulgarian, etc. (it could be result of imbalance of training data - I need to check the source dataset);
  • for "major" languages not mentioned above (English, French, German, Spanish, Portuguese, Dutch) the fastText results are much better than CLD's, and in many cases lingid.py's & OpenNLP's;
  • for many languages results for "compressed" fastText model are slightly worse than results from "full" model (mostly only by 1-2%, but could be higher, like for Kazakh when difference is 33%), but there are languages where the situation is different - results for compressed are slight better than for full (for example, for German or Dutch);

OpenNLP has many misclassifications for Cyrillic languages - Russian/Ukrainian, ...

Rafael Oliveira posted on FB a simple diagram that shows what languages are detected better by CLD & what is better handled by fastText

Here are some additional notes about differences in behavior of detectors that I observe during analyzing results:

  • fastText is more reliable than CLD on the short texts;
  • fastText models & langid.py detect language as Hebrew instead of Jewish as in CLD. Similarly, CLD uses 'in' for Indonesian language instead of standard 'id' used by fastText & langid.py;
  • fastText distinguish between Cyrillic- & Latin-based versions of Serbian;
  • CLD tends to incorporate geographical & person's names into detection results - for example, blog post in German about travel to Iceland is detected as Icelandic, while fastText detects it as German;
  • In extended detection mode CLD tends to select more rare language, like, Galician or Catalan over Spanish, Serbian instead of Russian, etc.
  • OpenNLP isn't very good in detection for short texts.

The models released by fastText development team provides very good alternative to existing language detection tools, like, Google's CLD & langid.py - for most of "popular" languages, these models provides higher detection accuracy comparing to other tools, combined with high speed of detection (drawback of langid.py). Even using "compressed" model it's possible to reach good detection accuracy. Although for some less frequently used languages, CLD & langid.py may show better results.

Performance-wise, the langid.py is much slower than both CLD & fastText. On average, CLD requires 0.5-1 ms to perform language detection. For fastText & langid.py I don't have precise numbers yet, only approximates based on speed of execution of corresponding programs.

GIT:

  1. LD with infinity gram

Articles:

  1. Medium on training LI models

Papers:

  1. A comparison of lang ident approaches
  2. lI on code switch social media
  3. Comparing LI methods, has 6 big languages
  4. Comparing LI techniques
  5. Radim rehurek LI on the web extending the dictionary
  6. Comparative study of LI methods, 2

LANGUAGE TRANSLATION

  1. State of the art methods for neural machine translation - a review of papers
  2. LASER: Zero shot multi lang-translation by facebook, github
  3. How to use laser on medium
  4. Stanford coreNLP language POS/NER/DEP PARSE etc for 53 languages
  5. Using embedding spaces w2v by gensim
  6. The risk of using bleu
  7. Really good: BLUE - what is it, how it is calculated?

“[BLEU] looks at the presence or absence of particular words, as well as the ordering and the degree of distortion—how much they actually are separated in the output.”

BLEU’s evaluation system requires two inputs: (i) a numerical translation closeness metric, which is then assigned and measured against (ii) a corpus of human reference translations.

BLEU averages out various metrics using an n-gram method, a probabilistic language model often used in computational linguistics.

BLEU sample

The result is typically measured on a 0 to 1 scale, with 1 as the hypothetical “perfect” translation. Since the human reference, against which MT is measured, is always made up of multiple translations, even a human translation would not score a 1, however. Sometimes the score is expressed as multiplied by 100 or, as in the case of Google mentioned above, by 10.

a BLEU score offers more of an intuitive rather than an absolute meaning and is best used for relative judgments: “If we get a BLEU score of 35 (out of 100), it seems okay, but it actually has no correlation to the quality of the output in any meaningful sense. If it’s less than 15, we can probably safely say it’s very bad. If it’s greater than 60, we probably have some mistake in our testing! So it will generally fall in there.”

“Typically, if you have multiple [human translation] references, the BLEU score tends to be higher. So if you hear a very large BLEU score—someone gives you a value that seems very high—you can ask them if there are multiple references being used; because, then, that is the reason that the score is actually higher.”

  1. General talk about FAMG (fb, ama, micro, goog) and research direction atm, including some info about BLUE scores and the comparison issues with reports of BLUE (boils down to diff unmentioned parameters)
  2. One proposed solution is sacreBLUE, pip install sacreblue

Named entity language transliteration

  1. Paper, blog post: English russian, hebrew, arabic, japanese, with data set and github

CHAT BOTS

  1. A list of chat bots, pros and cons, example code.

STRING MATCHING

  1. Fuzzy string matching library - fuzzywuzzy - using edit-distance

  1. Difflib, difflib2

REGEX

  1. Why re is slow
  2. Benchmark, comparisons, more, many more,
  3. Split on separator but keep the separator, in Python
  4. Semantic versioning
  5. Hyperscan - Hyperscan ****(paper) is a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API.
  6. Re2 - python This is the source code repository for RE2, a regular expression library.

  1. Spacy’s Matcher & “regex”
  2. Flashtext- This module can be used to replace keywords in sentences or extract keywords from sentences. It is based on the FlashText algorithm.