Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection

This repository contains the code for the paper "Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection", presented at CoRL 2024.

Paper (Arxiv) | Project Page | BibTeX

---

Table of Contents

• Installation
• Generating Pseudo-Labels

• Citation

---

Installation

To set up the repository, follow these steps:

1. Clone the repository:

   git clone https://github.com/meharkhurana03/cm3d.git
   cd cm3d

2. Install the dependencies:

    conda create -n cm3d python=3.9 -f environment.yml
    conda activate cm3d

3. To run on nuScenes:

   pip install nuscenes-devkit

   cd src/nuscenes

Generating Pseudo-Labels

First, generate 2D masks using detic:

python gen_2d_masks_trainset_detic.py

Next, generate pseudo-labels using the following command:

python 2d_to_3d_new.py

Note: Change the environment variables at the top of the scripts to point to the correct directories.

Citation

If you find our work useful in your research, please consider citing:

@article{khurana2024shelf,
        title={Shelf-Supervised Multi-Modal Pre-Training for 3D Object Detection},
        author={Khurana, Mehar and Peri, Neehar and Ramanan, Deva and Hays, James},
        journal={arXiv preprint arXiv:2406.10115},
        year={2024}
}