Prompt-BERT: Prompt makes BERT Better at Sentence Embeddings

Results on STS Tasks

Model	STS12	STS13	STS14	STS15	STS16	STSb	SICK-R	Avg.
unsup-prompt-bert-base Download	71.98	84.66	77.13	84.52	81.10	82.03	70.64	78.87
unsup-prompt-roberta-base Download	73.98	84.73	77.88	84.93	81.89	82.74	69.21	79.34
sup-prompt-bert-base Download	75.48	85.59	80.57	85.99	81.08	84.56	80.52	81.97
sup-prompt-roberta-base Download	76.75	85.93	82.28	86.69	82.80	86.14	80.04	82.95

To evaluate the above models, please run the following script,

bash eval_only.sh [unsup-bert|unsup-roberta|sup-bert|sup-roberta]

Download Data

cd SentEval/data/downstream/
bash download_dataset.sh
cd -
cd ./data
bash download_wiki.sh
bash download_nli.sh
cd -

Static token embedding with removing embedding biases

robert-base, bert-base-cased and robert-base-uncased

./run.sh roberta-base-static-embedding-remove-baises
./run.sh bert-base-cased-static-embedding-remove-baises
./run.sh bert-base-uncased-static-embedding-remove-baises

Non fine-tuned BERT with Prompt

bert-base-uncased with prompt

./run.sh bert-prompt

bert-base-uncased with optiprompt

./run.sh bert-optiprompt

Fine-tuned BERT with Prompt

unsupervised

SEED=0
./run.sh unsup-roberta $SEED

SEED=0
./run.sh unsup-bert $SEED

supervised

./run.sh sup-roberta 

./run.sh sup-bert

Calculation of anisotropy

To calculate the sentence anisotropy

EXP=bert-base-uncased | bert-base-uncased-static-embedding | bert-base-uncased-static-embedding-remove-baises 
./run.sh calc-anisotropy-${EXP}

Our Code is based on SimCSE