Improving VQA Using MLLM

Train

After downloading the training datasets and specify their path in dataset configs, we are ready for training!

0. Setting Environments

conda create -n fusion python=3.9

git clone 

cd BLIVA

pip install -e .

if packaging error, then

pip install setuptools==69.5.1

1. pretraining of Dm-Former

python train.py --cfg-path train_configs/pretrain_stage1.yaml

2. Pretraining of visual assistant branch

you should specify model path in pretrained

python train.py --cfg-path train_configs/pretrain_bliva_vicuna.yaml

3. Instruction Finetuning

python