The FT t5 beam search algorithm generates inconsistent results with HF's

[shiqingzhangCSU]

Branch/Tag/Commit

main

Docker Image Version

nvcr.io/nvidia/pytorch:22.09-py3

GPU name

A30

CUDA Driver

11.6

model

mt5

Reproduced Steps

Hi! I tested the results of FT and HF, I found that the beamsearch algorithm in HF(https://arxiv.org/pdf/1610.02424.pdf) maybe different from the one in FT and may generate different results. Is the beamsearch in FT the implementation in this paper(https://arxiv.org/pdf/1601.00372.pdf)? I look forward to your reply!

[byshiue]

Do you use early_stopping in HF?

[shiqingzhangCSU]

Yes, I tried early_stopping and there was no change in results. We found that no_repeat_ngram_size has a greater impact on beam=1. When the no_repeat_ngram_size in HF is set to null, the result of HF is the same as that of FT.

[byshiue]

FT does not support early_stopping=False and no_repeat_ngram_size now.

[shiqingzhangCSU]

If we want to add no_repeat_ngram_size features ourselves, how do we develop them? Any suggestions?

[byshiue]

You can add a penalty kernel in https://github.com/NVIDIA/FasterTransformer/blob/main/src/fastertransformer/layers/DynamicDecodeLayer.cc like other penalty to make sure the logit of repeated token is -inf.

[shiqingzhangCSU]

If I'm not familiar with this library, how long do you estimate it will take me to add this feature?

[byshiue]

Sorry, I cannot help to provide the estimated time because it depends on too many factors.

[shiqingzhangCSU]

Sorry, I cannot help to provide the estimated time because it depends on too many factors.

Thanks for the suggestion！