Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language

Implementation for the CVPR 2023 paper "Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language"

Prerequisites

Environment

pip install -r requirements.txt

Data Preparation

For Charades dataset, we use the same data preparation process as MS-2D-TAN, follows here for details.

For Charades-CG dataset, download https://github.com/YYJMJC/Compositional-Temporal-Grounding/tree/main/Charades-CG and put them under data/Charades-CG.

Training

python train.py --cfg experiments/charades/config.yaml

Evaluation

We provide a ckpt with results similar to our paper, and compare them in the table below.

Split	Test-Trivial	Novel-Composition	Novel-Word
Metric	[email protected] | [email protected] | mIoU	[email protected] | [email protected] | mIoU	[email protected] | [email protected] | mIoU
paper version	58.14 | 37.98 | 50.58	46.54 | 25.10 | 40.00	50.36 | 28.78 | 43.15
github version	59.34 | 38.53 | 51.48	45.96 | 24.67 | 40.01	50.07 | 29.21 | 42.79

You can download the checkpoint and put it under `ckpt' folder, then run

python evaluate.py --cfg experiments/charades/config.yaml --load ckpt/MS-2D-TAN_iter20461.pt

Citation

If any part of our paper and code is helpful to your work, please generously cite with:

@inproceedings{li2023exploring,
  title={Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language},
  author={Li, Chuanhao and Li, Zhen and Jing, Chenchen and Jia, Yunde and Wu, Yuwei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19092--19101},
  year={2023}
}

Acknowledgments

• Thanks for the great MS-2D-TAN.
• We use the Charades-CG dataset, thanks for their work.