Description

A solution for GramEval2020 competition.

About the Model

The model is based on DeepPavlov's RuBERT and AllenNLP dependencies parser implementation (which is based on Deep Biaffine Attention for Neural Dependency Parsing (Dozat and Manning, 2016)).

This is an end-to-end parser: the predictions for grammar values, lemmas and dependencies are made by a single model trained in a multi-task mode and they are not conditioned on each other.
It uses BERT embedder with a single layer LSTM encoder, simple feedforward predictors for grammar values and lemmas and biaffine attention predictors for dependencies and their labels.

Try it!

Simply open it here:

Preparations

In order to reproduce it locally, you will need to follow these steps.

Firstly, clone the repo:

git clone https://github.com/DanAnastasyev/GramEval2020.git
cd GramEval2020/

Then, install the dependencies. It's reasonable to use a virtual environment or a conda environment. E.g., run:

python3 -m venv solution-env
source solution-env/bin/activate

To install the dependencies run:

pip install -r requirements.txt
pip install git+git://github.com/DanAnastasyev/allennlp.git

The later command will install a very specific version of the AllenNLP library with a simple patch (that allows to pass arguments to the tokenizer).

After that, download the data. You can simply run ./download_data.sh on linux/mac. Otherwise, download the archive manually from Google Drive here.

It contains labeled data from the official repository and DeepPavlov's RuBERT converted to the pytorch format using transformers script.

Train

To train a model, run the following command:

cd solution
python -m train.main

It will use the BERT-based model config and train it on the data from data/data_train folder.

Apply

To apply a trained model from the previous step on the test data, use train.applier script.

You can also apply it on already trained model from my final submission: download the model weights: ./download_model.sh (or using this link).

And run:

cd solution
python -m train.applier --model-name ru_bert_final_model --batch-size 8

The applier script allows to specify the following parameters:

  --model-name MODEL_NAME   Model's name (the name of directory with the trained model)
  --pretrained-models-dir PRETRAINED_MODELS_DIR   Path to directory with pretrained models (e.g., RuBERT)
  --models-dir MODELS_DIR   Path to directory where the models are stored
  --data-dir DATA_DIR   Path to directory with files to apply the model to
  --predictions-dir PREDICTIONS_DIR   Path to directory to store the predictions
  --batch-size BATCH_SIZE
  --checkpoint-name CHECKPOINT_NAME   Name of the checkpoint to use

Which means that if you trained a new model with python -m train.main, you will need to pass --model-name ru_bert_parser_with_tagger because the model in the default config has this name.