Evaluating Neural Network model using Errant

[aimanmutasem]

Dear @ALL

I have an overview of your documentation, I'm still confused about how to evaluate my Neural Network model (GEC). As I understood that, I have to translate the test set (correcting), then build a new (M2) file using errant_parallel command. The last step is to use errant_compare with the span-based correction to get F0.5 score.

Is this correct?
What is the optimal way to evaluate my NN model using Errant?

Regards,

[chrisjbryant]

Hi,

You don't need to be confused because you are correct!

1. Use errant_parallel to build a new M2 file from the original sentences and your corrected sentences.
2. Use errant_compare to compare your system M2 file with the gold standard M2 file.

That's all there is to it. :)

[aimanmutasem]

Thank you @chrisjbryant for your support. I have applied errant as:

errant_parallel -orig ./data/test_bea.orig -cor ./data/test_data.txt -out ./data/out_data.m2

I got the below error:

Loading resources...

Warning: no model found for 'en'

Only loading the 'en' tokenizer.

Processing parallel files...
Traceback (most recent call last):
File "/home/aiman/anaconda3/bin/errant_parallel", line 8, in
sys.exit(main())
File "/home/aiman/anaconda3/lib/python3.6/site-packages/errant/commands/parallel_to_m2.py", line 27, in main
orig = annotator.parse(orig, args.tok)
File "/home/aiman/anaconda3/lib/python3.6/site-packages/errant/annotator.py", line 25, in parse
self.nlp.tagger(text)
TypeError: 'NoneType' object is not callable

Any suggestions?

[chrisjbryant]

Looks like you forgot to install a spacy model.

You need to run python3 -m spacy download en after you pip install errant. It's the last line in the installation instructions.

[aimanmutasem]

Thank you, dear.

Everything is working well, I have ignored to use it with a virtual environment :)

One last favour

For W&I+LOCNESS V2.1test set, I got just 'ABCN.test.bea19.orig', corrected sentence isn't an m2 file.

Kindly, do you know any way to get the gold M2 for W&I+LOCNESS or how to generate it?

[chrisjbryant]

The gold M2 file for W&I+LOCNESS test is private to prevent people from overfitting to it.

Instead, you can submit your corrected W&I+LOCNESS output to Codalab.
All the instructions on how to do this are also available on that page. :)

[aimanmutasem]

Thank you so mutch ...