MinkOcc

Get Started

Installation and Data Preparation

Make sure to have a working CUDA.

a. Create a conda virtual environment and activate it. All steps below happens in conda env.

conda create -n open-mmlab python=3.8 -y
conda activate open-mmlab

b. Install PyTorch and torchvision following the official instructions.

 pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 -f https://download.pytorch.org/whl/torch_stable.html

 or any other versions of torch that fits your cuda
 Recommended torch>=1.12

c. Install MinkowskiEngine. (Please refer to official website for specifics) https://github.com/NVIDIA/MinkowskiEngine/wiki/Installation

sudo apt install libopenblas-dev
git clone https://github.com/NVIDIA/MinkowskiEngine
cd MinkowskiEngine
export MAX_JOBS=1; python setup.py install --user

d. Install mmcv-full.

pip install mmcv-full==1.5.2

e. Install mmdet and mmseg.

pip install mmdet==2.24.0
pip install mmsegmentation==0.24.0

f. Install other dependencies f1. spconv, depends on your cuda version, for cuda 11.8 f2. yapf

pip install spconv-cu118
pip install yapf==0.31.0

g. Prepare MinkOcc repo by.

git clone https://github.com/venti-sam/MinkOcc.git
cd MinkOcc
pip install -v -e .

h. Download Nuscenes Mini dataset:

https://www.nuscenes.org/nuscenes#download

i. For Occupancy Prediction task, download the mini and (only) the 'gts' from CVPR2023-3D-Occupancy-Prediction and arrange the folder as:

└── nuscenes
    ├── v1.0-mini (existing)
    ├── sweeps  (existing)
    ├── samples (existing)
    └── gts (new)

j. Prepare nuScenes dataset as introduced in nuscenes_det.md and create the pkl for MinkOcc by running:

python tools/create_data_bevdet.py

If encountered terminal error related to numba & nuscenes converter, install numpy==1.23.5.

Train model

# single gpu
python tools/train.py configs/bevdet_occ/bevdet_minkocc.py

Test model

python tools/test.py $config $checkpoint --eval mAP

Robotcycle

First prepare dataset to same format as Waymo / Kitti, or download it from: https://drive.google.com/file/d/18x55NXeblCSuBNCjsUjsoHUeEIk34rN5/view?usp=drive_link . If you download the zip file, you can skip the two steps below

Step 1.

    - ImageSets for txt file with filename textfile indicating train/val/test
    - Training folder
        - calib in txt kitti format
        - images in .jpg / .png format
        - labels in .txt format
        - velodyne in .bin format
        - poses in .txt format
        - etc (depends)
    - Testing folder

Step 2. Create .pkl file for data loading

python tools/create_data.py kitti --root-path ./data/robotcycle --out-dir ./data/robotcycle --extra-tag robotcycle

Step 3. Run the training script

python tools/train.py configs/bevdet_occ/robotcycle.py

Acknowledgement

This project is not possible without multiple great open-sourced code bases. We list some notable examples below.

• open-mmlab
• CenterPoint
• Lift-Splat-Shoot
• Swin Transformer
• BEVFusion
• BEVDepth

Beside, there are some other attractive works extend the boundary of BEVDet.

• BEVerse for multi-task learning.
• BEVStereo for stero depth estimation.

Bibtex

If this work is helpful for your research, please consider citing the following BibTeX entries.

@article{huang2023dal,
title={Detecting As Labeling: Rethinking LiDAR-camera Fusion in 3D Object Detection},
author={Huang, Junjie and Ye, Yun and Liang, Zhujin and Shan, Yi and Du, Dalong},
journal={arXiv preprint arXiv:2311.07152},
year={2023}
}

@article{huang2022bevpoolv2,
title={BEVPoolv2: A Cutting-edge Implementation of BEVDet Toward Deployment},
author={Huang, Junjie and Huang, Guan},
journal={arXiv preprint arXiv:2211.17111},
year={2022}
}

@article{huang2022bevdet4d,
title={BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection},
author={Huang, Junjie and Huang, Guan},
journal={arXiv preprint arXiv:2203.17054},
year={2022}
}

@article{huang2021bevdet,
title={BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View},
author={Huang, Junjie and Huang, Guan and Zhu, Zheng and Yun, Ye and Du, Dalong},
journal={arXiv preprint arXiv:2112.11790},
year={2021}
}

```

```