Virtual Sparse Convolution for Multimodal 3D Object Detection

This is a official code release of VirConv (Virtual Sparse Convolution for 3D Object Detection). This code is mainly based on OpenPCDet, some codes are from TED, CasA, PENet and SFD.

Detection Framework

• VirConv-L: A light-weight multimodal 3D detector based on Virtual Sparse Convolution.
• VirConv-T: A improved multimodal 3D detector based on Virtual Sparse Convolution and transformed refinement scheme.
• VirConv-S: A semi-supervised VirConv-T based on pseudo labels and fine-tuning.

The detection frameworks are shown below.

Model Zoo

We release three models: VirConv-L, VirConv-T and VirConv-S.

• The VirConv-L and VirConv-T are trained with train split (3712 samples) of KITTI dataset.

• The VirConv-S is trained with train split (3712 samples) and unlabeled odometry split (semi split 10888 sample) of KITTI dataset.

• The results are the 3D AP(R40) of Car on the val set of KITTI dataset.

Important notes:

• The input voxel discard has been changed to input point discard for faster voxelization.
• The convergence of VirConv-T is somewhat unstable ( AP~[89.5,90.3]), if you cannot achieve similar AP, please try multiple times. We recommend VirConv-S, which can achieve 90.5+ AP easily.
• These models are not suitable to directly report results on KITTI test set, please train the models on all or 80% training data and choose a good score threshold to achieve a desirable performance.

Train multiple times on 8xV100 and choose the best:

Environment	Detector	GPU (train)	Easy	Mod.	Hard	download
Spconv1.2	VirConv-L	~7 GB	93.08	88.51	86.69	google / baidu(05u2) / 51M
Spconv1.2	VirConv-T	~13 GB	94.58	89.87	87.78	google / baidu(or81) / 55M
Spconv1.2	VirConv-S	~13 GB	95.67	91.09	89.09	google / baidu(ak74) / 62M

Train multiple times on 8xV100 and choose the best:

Environment	Detector	GPU (train)	Easy	Mod.	Hard	download
Spconv2.1	VirConv-L	~7 GB	93.18	88.23	85.48	google / baidu(k2dp) / 51M
Spconv2.1	VirConv-T	~13 GB	94.91	90.36	88.10	google / baidu(a4r4) / 56M
Spconv2.1	VirConv-S	~13 GB	95.76	90.91	88.61	google / baidu(j3mi) / 56M

Getting Started

conda create -n spconv2 python=3.9
conda activate spconv2
pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install numpy==1.19.5 protobuf==3.19.4 scikit-image==0.19.2 waymo-open-dataset-tf-2-5-0 nuscenes-devkit==1.0.5 spconv-cu111 numba scipy pyyaml easydict fire tqdm shapely matplotlib opencv-python addict pyquaternion awscli open3d pandas future pybind11 tensorboardX tensorboard Cython prefetch-generator

Dependency

Our released implementation is tested on.

• Ubuntu 18.04
• Python 3.6.9
• PyTorch 1.8.1
• Numba 0.53.1
• Spconv 1.2.1
• NVIDIA CUDA 11.1
• 8x Tesla V100 GPUs

We also tested on.

• Ubuntu 18.04
• Python 3.9.13
• PyTorch 1.8.1
• Numba 0.53.1
• Spconv 2.1.22 # pip install spconv-cu111
• NVIDIA CUDA 11.1
• 8x Tesla V100 GPUs

We also tested on.

• Ubuntu 18.04
• Python 3.9.13
• PyTorch 1.8.1
• Numba 0.53.1
• Spconv 2.1.22 # pip install spconv-cu111
• NVIDIA CUDA 11.1
• 2x 3090 GPUs

Prepare dataset

You must creat additional semi dataset and velodyne_depth dataset to run our multimodal and semi-supervised detectors.

• You can download all the preprocessed data from baidu (japc) [74GB], or partial data (not include semi due to disk space limit ) from google (13GB).

• Or you can generate the dataset by yourself as follows:

Please download the official KITTI 3D object detection dataset, KITTI odometry dataset and organize the downloaded files as follows (the road planes could be downloaded from [road plane], which are optional for data augmentation in the training):

VirConv
├── data
│   ├── odometry
│   │   │── 00
│   │   │── 01
│   │   │   │── image_2
│   │   │   │── velodyne
│   │   │   │── calib.txt
│   │   │── ...
│   │   │── 21
│   ├── kitti
│   │   │── ImageSets
│   │   │── training
│   │   │   ├──calib & velodyne & label_2 & image_2 & (optional: planes)
│   │   │── testing
│   │   │   ├──calib & velodyne & image_2
├── pcdet
├── tools

(1) Creat semi dataset from odometry dataset.

cd tools
python3 creat_semi_dataset.py ../data/odometry ../data/kitti/semi

(2) Download the pseudo labels generated by VirConv-T from here (fuse detections from last 10 checkpoints by WBF and filter low quality detections by a 0.9 score threshold) and put it into kitti/semi.

(3) Download the PENet depth completion model from google (500M) or baidu (gp68), and put it into tools/PENet.

(4) Then run the following code to generate RGB virtual points.

cd tools/PENet
python3 main.py --detpath ../../data/kitti/training
python3 main.py --detpath ../../data/kitti/testing
python3 main.py --detpath ../../data/kitti/semi

(5) After that, run following command to creat dataset infos:

python3 -m pcdet.datasets.kitti.kitti_dataset_mm create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml
python3 -m pcdet.datasets.kitti.kitti_datasetsemi create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

Anyway, the data structure should be:

VirConv
├── data
│   ├── kitti
│   │   │── ImageSets
│   │   │── training
│   │   │   ├──calib & velodyne & label_2 & image_2 & (optional: planes) & velodyne_depth
│   │   │── testing
│   │   │   ├──calib & velodyne & image_2 & velodyne_depth
│   │   │── semi (optional)
│   │   │   ├──calib & velodyne & label_2(pseudo label) & image_2 & velodyne_depth
│   │   │── gt_database_mm
│   │   │── gt_databasesemi
│   │   │── kitti_dbinfos_trainsemi.pkl
│   │   │── kitti_dbinfos_train_mm.pkl
│   │   │── kitti_infos_test.pkl
│   │   │── kitti_infos_train.pkl
│   │   │── kitti_infos_trainsemi.pkl
│   │   │── kitti_infos_trainval.pkl
│   │   │── kitti_infos_val.pkl
├── pcdet
├── tools

Setup

cd VirConv
python setup.py develop

Training.

For training the VirConv-L and VirConv-T:

Single GPU train:

cd tools
python3 train.py --cfg_file ${CONFIG_FILE}

For example, if you train the VirConv-L model:

cd tools
python3 train.py --cfg_file cfgs/models/kitti/VirConv-L.yaml

Multiple GPU train:

You can modify the gpu number in the dist_train.sh and run

cd tools
sh dist_train.sh

The log infos are saved into log.txt You can run cat log.txt to view the training process.

For training the VirConv-S:

You should firstly train a VirConv-T:

cd tools
python3 train.py --cfg_file cfgs/models/kitti/VirConv-T.yaml

Then train the VirConv-S:

cd tools
python3 train.py --cfg_file cfgs/models/kitti/VirConv-S.yaml --pretrained_model ../output/models/kitti/VirConv-T/default/ckpt/checkpoint_epoch_40.pth

Evaluation.

cd tools
python3 test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}

For example, if you test the VirConv-S model:

cd tools
python3 test.py --cfg_file cfgs/models/kitti/VirConv-S.yaml --ckpt VirConv-S.pth

Multiple GPU test: you should modify the gpu number in the dist_test.sh and run

sh dist_test.sh 

The log infos are saved into log-test.txt You can run cat log-test.txt to view the test results.

License

This code is released under the Apache 2.0 license.

Acknowledgement

TED

CasA

OpenPCDet

PENet

SFD

Citation

@inproceedings{VirConv,
    title={Virtual Sparse Convolution for Multimodal 3D Object Detection},
    author={Wu, Hai and Wen,Chenglu and Shi, Shaoshuai and Wang, Cheng},
    booktitle={CVPR},
    year={2023}
}