Merge pull request sebastianruder#92 from yuvalpinter/master

new task - relation prediction on WN18RR

README.md

@@ -21,6 +21,7 @@
 - [Natural language inference](natural_language_inference.md)
 - [Part-of-speech tagging](part-of-speech_tagging.md)
 - [Question answering](question_answering.md)
+- [Relation Prediction](relation_prediction.md)
 - [Relationship extraction](relationship_extraction.md)
 - [Semantic textual similarity](semantic_textual_similarity.md)
 - [Sentiment analysis](sentiment_analysis.md)
@@ -55,7 +56,6 @@ These are tasks and datasets that are still missing.
 - Keyphrase extraction
 - Knowledge base population (KBP)
 - More dialogue tasks
-- Relation extraction
 - Semi-supervised learning
 - Text normalization
 
@@ -112,14 +112,15 @@ you can build the site locally using Jekyll by following the steps detailed
 [here](https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/#requirements):
 
 1. Check whether you have Ruby 2.1.0 or higher installed with `ruby --version`, otherwise [install it](https://www.ruby-lang.org/en/downloads/).
-On OS X for instance, this can be done with `brew install ruby`.
-2. Install Bundler `gem install bundler`. If you run into issues with installing bundler on OS X, have a look
+On OS X for instance, this can be done with `brew install ruby`. Make sure you also have `ruby-dev` and `zlib1g-dev` installed.
+1. Install Bundler `gem install bundler`. If you run into issues with installing bundler on OS X, have a look
 [here](https://bundler.io/v1.16/guides/rubygems_tls_ssl_troubleshooting_guide.html) for troubleshooting tips. Also try refreshing
 the terminal.
-3. Install Jekyll: `bundle install`
-4. Clone the repo locally: `git clone https://github.com/sebastianruder/NLP-progress`
-5. Navigate to the repo with `cd NLP-progress` and run the Jekyll site locally: `bundle exec jekyll serve`
-6. You can now preview the local Jekyll site in your browser at `http://localhost:4000`.
+1. Clone the repo locally: `git clone https://github.com/sebastianruder/NLP-progress`
+1. Navigate to the repo with `cd NLP-progress`
+1. Install Jekyll: `bundle install`
+1. Run the Jekyll site locally: `bundle exec jekyll serve`
+1. You can now preview the local Jekyll site in your browser at `http://localhost:4000`.
 
 ### Things to do
 

_data/relation_prediction.yaml

@@ -0,0 +1,74 @@
+WN18RR:
+-
+  model:       Max-Margin Markov Graph Models (M3GM)
+  authors:     Pinter and Eisenstein
+  year:        2018
+  H@10:        59.02
+  H@1:         45.37
+  MRR:         49.83
+  paper:       Predicting Semantic Relations using Global Graph Properties
+  url:         https://arxiv.org/abs/1804.09530
+  code:        
+  - name:      Official
+    url:       http://www.github.com/yuvalpinter/m3gm
+-
+  model:       TransE (reimplementation)
+  authors:     Pinter and Eisenstein
+  year:        2018
+  H@10:        55.55
+  H@1:         42.26
+  MRR:         46.59
+  paper:       Predicting Semantic Relations using Global Graph Properties
+  url:         https://arxiv.org/abs/1804.09530
+  code:        
+  - name:      Reimplementation of Bordes et al. 2013. Translating embeddings for modeling multi-relational data.
+    url:       http://www.github.com/yuvalpinter/m3gm
+-
+  model:       ConvKB
+  authors:     Nguyen et al.
+  year:        2018
+  H@10:        52.50
+  H@1:         N/A
+  MRR:         24.80
+  paper:       A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network
+  url:         http://www.aclweb.org/anthology/N18-2053
+  code:
+  - name:      Official
+    url:       https://github.com/daiquocnguyen/ConvKB
+-
+  model:       ConvE (v6)
+  authors:     Dettmers et al.
+  year:        2018
+  H@10:        52.00
+  H@1:         40.00
+  MRR:         43.00
+  paper:       Convolutional 2D Knowledge Graph Embeddings
+  url:         https://arxiv.org/abs/1707.01476
+  code:
+  - name:      Official
+    url:       https://github.com/TimDettmers/ConvE
+-
+  model:       ComplEx
+  authors:     Trouillon et al.
+  year:        2016
+  H@10:        51.00
+  H@1:         41.00
+  MRR:         44.00
+  paper:       Complex Embeddings for Simple Link Prediction
+  url:         http://www.jmlr.org/proceedings/papers/v48/trouillon16.pdf
+  code:
+  - name:      Official
+    url:       https://github.com/ttrouill/complex
+-
+  model:       DistMult (reimplementation)
+  authors:     Dettmers et al.
+  year:        2017
+  H@10:        49.00
+  H@1:         40.00
+  MRR:         43.00
+  paper:       Convolutional 2D Knowledge Graph Embeddings
+  url:         https://arxiv.org/abs/1412.6575
+  code:
+  - name:      Reimplementation of Yang et al. 2013. Embedding Entities and Relations for Learning and Inference in Knowledge Bases.
+    url:       https://github.com/uclmr/inferbeddings
+

relation_prediction.md

@@ -0,0 +1,43 @@
+# Relation Prediction
+
+## Task
+
+Relation Prediction is the task of recognizing a named relation between two named semantic entities. The common test setup is to hide one entity from the relation triplet, asking the system to recover it based on the other entity and the relation type.
+
+For example, given the triple \<*Roman Jakobson*, *born-in-city*, *?*\>, the system is required to replace the question mark with *Moscow*.
+
+Relation Prediction datasets are typically extracted from two types of resources: 
+* *Knowledge Bases*: KBs such as [FreeBase](https://developers.google.com/freebase/) contain hundreds or thousands of relation types pertaining to world-knowledge obtained autmoatically or semi-automatically from various resources on millions of entities. These relations include *born-in*, *nationality*, *is-in* (for geographical entities), *part-of* (for organizations, among others), and more.
+* *Semantic Graphs*: SGs such as [WordNet](https://wordnet.princeton.edu/) are often manually-curated resources of semantic concepts, restricted to more "linguistic" relations compared to free real-world knowledge. The most common semantic relation is *hypernym*, also known as the *is-a* relation (example: \<*cat*, *hypernym*, *feline*\>).
+
+## Evaluation
+
+Evaluation in Relation Prediction hinges on a list of ranked candidates given by the system to the test instance. The metrics below are derived from the location of correct candidate(s) in that list.
+
+A common action performed before evaluation on a given list is *filtering*, where the list is cleaned of entities known to mismatch the type of expected entity to the relation. Unless specified otherwise, results here are from filtered lists.
+
+### Metrics
+
+#### Mean Reciprocal Rank (MRR):
+
+The mean of all reciprocal ranks for the true candidates over the test set (1/rank).
+
+#### Hits at k (H@k):
+
+The rate of correct entities appearing in the top *k* entries for each instance list. This number may exceed 1.00 if the average *k*-truncated list contains more than one true entity.
+
+### Datasets
+
+#### WordNet-18-RR (WN18RR)
+
+The WN18 dataset was introduced in Bordes et al., 2013. It included the full 18 relations scraped from WordNet for roughly 41,000 synsets. This dataset was found to suffer from major training set leakage, initially by Socher et al. 2013. This means reciprocal edges from symmetric relations (e.g. *hypernym*-*hyponym*) appear in one form in the training or dev set, and in the other in the test set. A trivial rule-based system recovering these regularities surpasses 90% performance on the test set.
+
+As a way to overcome this problem, Dettmers et al. (2018) introduced the WN18RR dataset, derived from WN18, which features 11 relations only, no pair of which is reciprocal (but still include four internally-symmetric relations like *verb_group*, allowing the rule-based system to reach 35 on all three metrics).
+
+The test set is composed of triplets, each used to create two test instances, one for each entity to be predicted. Since each instance is associated with a single true entity, the maximum value for all metrics is 1.00.
+
+{% include table.html
+   results=site.data.relation_prediction.WN18RR
+   scores='H@10,H@1,MRR' %}
+
+[Back to README](README.md)