Training OOM

[rexainn]

Hi, I want to know what GPU do you use for training?
I use a V100, but it kept reporting out of memory. I have turned off the 'convert_models_to_fp32'.

[rexainn]

I notice that in your paper you mentioned experiments using a single 3090.
So is it because I train it on my own dataset, and there exists 7 tasks?

[zwx8981]

Maybe, try using smaller batch size

[zwx8981]

You may also try setting opt = 1 in Line127, which freezes the weights of text encoder. Empirically, this would not affect the final performance very much, but can significantly reduce the memory cost.

[rexainn]

Maybe, try using smaller batch size

set batchsize = 1 still cause OOM, quite strange....

[rexainn]

You may also try setting opt = 1 in Line127, which freezes the weights of text encoder. Empirically, this would not affect the final performance very much, but can significantly reduce the memory cost.

This works, thanks! Meanwhile, I will still try to find the way to not freeze the text encoder