word2vec

Word2Vec implementation in numpy. Tried out Skip-Gram model on A Storm of Swords by R.R. Martin .
Dataset Link : https://www.kaggle.com/muhammedfathi/game-of-thrones-book-files#got2.txt

---

Word2Vec Architecture

Dimensions of Input Layer: V X 1 (vocabulary Size)
Dimensions of W1: V X Number of Dimensions of Embedding
Dimensions of Hidden Layer 1: Number of Dimnsions of Embedding X 1
Dimensions of W2: Number of Dimensions of Embedding X V
Dimensions of Output Layer: V X 1

Built With

• nltk
• numpy
• matplotlib

Results

Epochs : 5
Total vocabulary size : 6633 words
Number of Dimensions : 10

To-do

• CBOW Model
• Negative Sampling
• Try out for more epochs and larger dimensions.

Acknowledgments

• Research paper on Word2Vec https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
• A YouTube video by Jordan Boyd-Graber https://www.youtube.com/watch?v=QyrUentbkvw
• Medium blog by Derik Chia which helped in the implementation https://towardsdatascience.com/an-implementation-guide-to-word2vec-using-numpy-and-google-sheets-13445eebd281
• PDF explaining the math behind Word2Vec's loss function http://mccormickml.com/assets/word2vec/Alex_Minnaar_Word2Vec_Tutorial_Part_I_The_Skip-Gram_Model.pdf