Skip to content

Latest commit

 

History

History
82 lines (75 loc) · 6.21 KB

README.md

File metadata and controls

82 lines (75 loc) · 6.21 KB
Raw

This is the code base for CVPR2022 paper Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

Prepare data

Download dataset according to LVIS, VOC, COCO and Objects365. Precomputed proposals generated by RPN trained on only base classes can be downloaded from google drive baiduyun (code:yadt). It is recommended to download and extract the dataset somewhere outside the project directory and symlink the dataset root to data as below.

├── mmdet
├── tools
├── configs
├── data
├── ├── lvis_v1
├── ├── ├──annotations
├── ├── ├──train2017
├── ├── ├──val2017
├── ├── ├──proposals
│   ├── coco
│   │   ├── annotations
│   │   ├── train2017
│   │   ├── val2017
│   ├── VOCdevkit
│   │   ├── VOC2007
│   │   ├── VOC2012
│   ├── objects365
│   │   ├── annotations
│   │   ├── train
│   │   ├── val

All models use the backbone pretrained with SoCo which can be downloaded from google drive baiduyun (code:kwps). Put the pretrained backbone under data/.

Main Results

Model Lr Schd APbbr APbbc APbbf APbb APmkr APmkc APmkf APmk Config Prompt Model
ViLD* 20 epochs 17.4 27.5 31.9 27.5 16.8 25.6 28.5 25.2 config google drive baiduyun (code:a5ni) google drive baiduyun (code:cyhv)
DetPro (Mask R-CNN) 20 epochs 20.8 27.8 32.4 28.4 19.8 25.6 28.9 25.9 config google drive baiduyun (code:uvab) google drive baiduyun (code:apmq)
DetPro (Cascade R-CNN) 20 epochs 21.7 29.6 35.0 30.5 20.0 26.7 30.4 27.0 config google drive baiduyun (code:uvab) google drive baiduyun (code:5ee9)

In the original implementation of ViLD, the whole training process takes up to 180,000 iterations with batchsize of 256, approximately 460 epochs, which is unaffordable. We re-implement ViLD (denoted as ViLD*) with backbone pretrained using SoCo. Our re-implementation version achieves comparable AP compared with the original implementation, while reducing the training epochs from 460 to 20.

Installation

Dependencies

  • python3.8
  • pytorch 1.7.0
  • cuda 11.0

This repo is built on mmdetection, CLIP and CoOP

pip install -r requirements/build.txt
pip install -e .
pip install git+https://github.com/openai/CLIP.git
pip uninstall pycocotools -y
pip uninstall mmpycocotools -y
pip install mmpycocotools
pip install git+https://github.com/lvis-dataset/lvis-api.git
pip install mmcv-full==1.2.5 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.html

Get Started

Quick reproduction of the main results

./tools/dist_test.sh <config> <model> <gpu_num> --eval bbox segm --cfg-options model.roi_head.prompt_path=<prompt> model.roi_head.load_feature=False

Prepare data for DetPro training

see prepare.sh.

This process will take a long time. So we also provide the extracted clip image embeddings of precomputed proposals baiduyun (code:o4n5). You can download all these zip files and merge them into one file (lvis_clip_image_embedding.zip).

Train RPN on Only Base Classes

The training code and checkpoint are available here baiduyun(code:tqsd).

Train DetPro

see detpro.sh

Train ViLD with DetPro (Mask R-CNN)

see vild_detpro.sh

Tranfer experiments

see transer.sh

The empty prompt is provided here, you can use it to generate the prompt for COCO, VOC and Objects365.

Citation

@article{du2022learning,
  title={Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model},
  author={Du, Yu and Wei, Fangyun and Zhang, Zihe and Shi, Miaojing and Gao, Yue and Li, Guoqi},
  journal={arXiv preprint arXiv:2203.14940},
  year={2022}
}