HypernymySuite is a tool for evaluating some hypernymy detection modules. Its predominant focus is reproducing the results for the following paper.
Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora. ACL. (arXiv)
We hope that open sourcing our evaluation will help facilitate future research.
Before you begin, you should run the script to download the evaluation datasets.
bash download_data.sh
You can produce results in a JSON format by calling main.py:
python main.py cnt --dset hearst_counts.txt.gz
These results can be made machine readable by piping them into compile_table:
python main.py cnt --dset hearst_counts.txt.gz | python compile_table.py
To generate the full table from the report, you may simply use generate_table.sh:
bash generate_table.sh results.json
Please note that due to licensing concerns, we were not able to release our train/validation/test folds from the paper, so results may differ slightly than those reported.
The module was developed with python3 in mind, and is not tested for python2. Nonetheless, cross-platform compatibility may be possible.
The suite requires several packages you probably already have installed:
numpy, scipy, pandas, scikit-learn and nltk. These can be installed
using pip:
pip install -r requirements.txt
If you've never used nltk before, you'll need to install the wordnet module.
python -c "import nltk; nltk.download('wordnet')"
On OS X, you may need to install coreutils and gnu-sed for the script download_data.sh to run correctly. These can be installed using brew:
brew install coreutils gnu-sed
After installation, you will either need to modify download_data.sh to run gsort and gsed instead of sort and sed, or alternatively add a "gnubin" directory to your PATH from your bashrc:
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
For more information, see brew info coreutils or brew info gnu-sed.
You can evaluate your own model in two separate ways. The simplest way is simply
to create a copy of example.tsv, and fill in your model's predictions in the sim
column. You must include a prediction for every pair, but you may set the is_oov
column to 1 to ensure it is correctly calculated.
You may then evaluate the model:
python main.py precomputed --dset example.tsv
You can also implement any model by extending the base.HypernymySuiteModel class
and filling in your own implemenation for predict or predict_many.
If you find this code useful for your research, please cite the following paper:
@inproceedings{roller2018hearst
title = {Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora},
author = {Roller, Stephen and Kiela, Douwe and Nickel, Maximilian},
year = {2018},
booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics},
location = {Melbourne, Australia},
publisher = {Association for Computational Linguistics}
}
This code is licensed under CC-BY-NC4.0.
The data contained in hearst_counts.txt was extracted from a combination of
Wikipedia and Gigaword.
Please see publication for details.