🥷 MuLan: A Study of Fact Mutability in Language Models

Setup

conda create -n fact_mutability
conda activate fact_mutability
pip install torch==2.0.0
pip install git+https://github.com/huggingface/transformers
conda install matplotlib

# (Optional) For the iPython kernel
conda install -c anaconda ipykernel
python -m ipykernel install --user --name=fact_mutability

Data

🥷 MuLan queries: https://huggingface.co/datasets/coastalcph/fm_queries

Aliases: https://huggingface.co/datasets/coastalcph/fm_aliases

Run models' evaluation

inference.py: This code passes a set of queries (one query per line) through a language model and stores the model's predictions and softmax scores in predictions.json. Uses greedy beam search to generate predictions, options to select how many beams and instructions.

evaluation.py: SQUAD-style F1-score evaluation, where the user specifies whether to select the best prediction based on perplexity of first token score (prediction_mode). It uses the predictions generated by running inference.py, the queries and aliases datasets (the best match among the possible answers and its aliases is used).