🐿️VideoPal: Pal for Long Video Chat Based on BigDL

Powered by BigDL, ChatGLM3, CLIP, Whisper, Tag2Text, Paddleocr, multilingual-e5-large, LangChain, KeyBert, KTS, Helsinki and inspired by showlab/Vlog, we decompose the video information into static visual information, dynamic visual information, and auditory information and store them in the database respectively, and use LangChain to build LLM Agent for retrieval and answer. Then we can chat over the record only using Intel CPU.

Pipeline

Demo

The video in the demo comes from the series 'Man vs. Wild' starring Bear Grylls. Original Video

Origin Video Clip	Answer

Project Structure

Environment Preparing

System ---Windows/Linux

1. Create Conda Environment

conda create -n videopal python=3.9 -y
conda activate videopal
cd VideoPal
pip install -U -r requirements.txt

2. Install FFmpeg

System ---Windows

conda install -c conda-forge ffmpeg -y

System ---Linux

sudo apt update
sudo apt install ffmpeg

3. Install Mysql

sudo apt update
apt install mysql-server
sudo mysql_secure_installation

4. Configuration Your Database

Set Your Own Password

sudo mysql
# alter user 'root'@'localhost' identified with mysql_native_password by '123456';
alter user 'root'@'localhost' identified with mysql_native_password by [newpassword];
exit;

Create Your Own Database

mysql -u root -p
create database [your_database_name]

5. Download Model Weight

Download Models by Huggingface

python download_ckpt.py

Download Tag2Text weight from here

✅Please save weights to ./checkpoints.

6. Optimaize Models

python model_low_bit_optimize.py

7. Run with Gradio

python main_gradio.py --clip_low_bit --whisper_low_bit

Tutorial

You can find the tutorial of 🐿️VideoPal here.

❗Attention❗

1. Gradio Warning

Could not create share link. Missing file: D:\anaconda3\envs\vchat\lib\site-packages\gradio\frpc_windows_amd64_v0.2.

Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps:

1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_linux_amd64
2. Rename the downloaded file to: frpc_linux_amd64_v0.2
3. Move the file to this location: .\anaconda3\envs\vchat\lib\site-packages\gradio.
4. Give this file executable permissions (For Linux users)

chmod +x frpc_linux_amd64_v0.2

2. One by One

Multiple people using the same demo at the same time will cause an error because the asynchronous running logic is not implemented.

3. MySQL Error

If you encounter the following error:

Error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' 

You can enter the following command on the command line to solve the problem：

mysqld&

Acknowledge

We are grateful for the following awesome projects our VideoPal arising from:

• BigDL: Accelerate local LLM inference and finetuning on Intel CPU and GPU
• Tag2Text: An efficient and controllable vision-language model with tagging guidance
• Whisper: Robust Speech Recognition via Large-Scale Weak Supervision
• ChatGLM3: Open Bilingual Chat LLMs | 开源双语对话语言模型
• LangChain: A framework for developing applications powered by language models
• PaddleOCR: Awesome multilingual OCR toolkits based on PaddlePaddle
• SpaceTimeGPT: A Spatiotemporal Video Captioning Model
• KeyBert: A minimal method for keyword extraction with BERT
• Vlog: Transform Video as a Document with ChatGPT, CLIP, BLIP2, GRIT, Whisper, LangChain
• multilingual-e5-large: A multilingual text embedding
• KTS: An algorithm for video summarization
• Helsinki: Open neural machine translation models

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Kailuo 💻	chenminghao 💻	Yifan Wu 💻
Add your contributions