BLIVA: A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions

Wenbo Hu*, Yifan Xu*, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. *Equal Contribution

UC San Diego, Coinbase Global, Inc.

Our model architecture in detail with example responses.

Release

• [8/21] 🔥 We released BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions. Checkout the paper.
• [8/21] 🔥 We released our demo at here which is publicly avaible for everyone to play with.
• [8/21] We released our Model weight for BLIVA Vicuna Version at here and FLAN T5 Version at here which is available for commercial use.
• [8/21] Our Youtube Visual Question Answering Dataset (YTTB-VQA) is available at here.

Installation

1. Creating conda environment

conda create -n bliva python=3.9
conda activate bliva

2. build from source

git clone https://github.com/mlpc-ucsd/BLIVA
cd BLIVA
pip install -e .

Prepare Weight

1. BLIVA Vicuna 7B

   Our Vicuna version model is released at here. Download our model weight and specify the path in the model config here at line 8.

   The LLM we used is the v0.1 version from Vicuna-7B. To prepare Vicuna's weight, please refer to our instruction here. Then, set the path to the vicuna weight in the model config file here at Line 21.

2. BLIVA FlanT5 XXL (Available for Commercial Use)

   The FlanT5 version model is released at here. Download our model weight and specify the path in the model config here at line 8.

   The LLM weight for Flant5 will automatically begin to download from huggingface when running our inference code.

Inference

To answer one question from the image, run the following evaluation code. For example,

python evaluate.py --answer_qs \
        --model_name bliva_vicuna \
        --img_path images/example.jpg \
        --question "what is this image about?"

We also support answer multiple choice question, which is the same as we used for evaluation tasks in paper. To provide a list of chioce, it should be a string split by comma. For example,

python evaluate.py --answer_mc \
        --model_name bliva_vicuna \
        --img_path images/mi6.png \
        --question "Which genre does this image belong to?" \
        --candidates "play, tv show, movie"

Demo

Our Demo is publicly available at here. To run our demo locally on your machine. Run:

python demo.py

Citation

If you find BLIVA useful for your research and applications, please cite using this BibTeX:

@misc{hu2023bliva,
      title={BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions}, 
      author={Wenbo Hu and Yifan Xu and Yi Li and Weiyue Li and Zeyuan Chen and Zhuowen Tu},
      publisher={arXiv:2308.09936},
      year={2023},
}

Acknowledgement

• BLIP2 The model architecture of BLIVA follows BLIP-2. Don't forget to check this great open-source work if you don't know it before.
• Lavis The codebase we built upon.
• Vicuna Vicuna-13B demonstrates fantastic language ability and it's open source.

License

This repository's code is under BSD 3-Clause License. Many codes are based on Lavis with BSD 3-Clause License here.

For our model parameters of BLIVA Vicuna Version, it's should be used under LLaMA's model license. For the model weight of BLIVA FlanT5, it's under Apache 2.0 License. For our YTTB-VQA data, it's under CC BY NC 4.0