CRN_demo.mp4
CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception
Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, Dongsuk Kum
ICCV 2023
In this paper, we propose Camera Radar Net (CRN), a novel camera-radar fusion framework that generates a semantically rich and spatially accurate bird's-eye-view (BEV) feature map for various tasks. To overcome the lack of spatial information in an image, we transform perspective view image features to BEV with the help of sparse but accurate radar points. We further aggregate image and radar feature maps in BEV using multi-modal deformable attention designed to tackle the spatial misalignment between inputs. CRN with real-time setting operates at 20 FPS while achieving comparable performance to LiDAR detectors on nuScenes, and even outperforms at a far distance on 100m setting. Moreover, CRN with offline setting yields 62.4% NDS, 57.5% mAP on nuScenes test set and ranks first among all camera and camera-radar 3D object detectors.
# clone repo
git clone https://github.com/youngskkim/CRN.git
cd CRN
# setup conda environment
conda env create --file CRN.yaml
conda activate CRN
# install dependencies
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install pytorch-lightning==1.6.0
mim install mmcv==1.6.0
mim install mmsegmentation==0.28.0
mim install mmdet==2.25.2
cd mmdetection3d
pip install -v -e .
cd ..
python setup.py develop # GPU requiredStep 0. Download nuScenes dataset.
Step 1. Symlink the dataset folder to ./data/nuScenes/.
ln -s [nuscenes root] ./data/nuScenes/
Step 2. Create annotation file.
This will generate nuscenes_infos_{train,val}.pkl.
python scripts/gen_info.py
Step 3. Generate ground truth depth.
Note: this process requires LiDAR keyframes.
python scripts/gen_depth_gt.py
Step 4. Generate radar point cloud in perspective view.
You can download pre-generated radar point cloud here.
Note: this process requires radar blobs (in addition to keyframe) to utilize sweeps.
python scripts/gen_radar_bev.py # accumulate sweeps and transform to LiDAR coords
python scripts/gen_radar_pv.py # transform to camera coords
The folder structure will be as follows:
CRN
├── data
│ ├── nuScenes
│ │ ├── nuscenes_infos_train.pkl
│ │ ├── nuscenes_infos_val.pkl
│ │ ├── maps
│ │ ├── samples
│ │ ├── sweeps
| | ├── depth_gt
| | ├── radar_bev_filter # temporary folder, safe to delete
| | ├── radar_pv_filter
| | ├── v1.0-trainval
Training
python [EXP_PATH] --amp_backend native -b 4 --gpus 4
Evaluation
Note: use -b 1 --gpus 1 to measure inference time.
python [EXP_PATH] --ckpt_path [CKPT_PATH] -e -b 4 --gpus 4
All models use 4 keyframes and are trained without CBGS.
All latency numbers are measured with batch size 1, GPU warm-up, and FP16 precision.
| Method | Backbone | NDS | mAP | FPS | Params | Config | Checkpoint |
|---|---|---|---|---|---|---|---|
| BEVDepth | R50 | 47.1 | 36.7 | 29.7 | 77.6 M | config | model |
| CRN | R18 | 54.2 | 44.9 | 29.4 | 37.2 M | config | model |
| CRN | R50 | 56.2 | 47.3 | 22.7 | 61.4 M | config | model |
- BEV segmentation checkpoints
- BEV segmentation code
- 3D detection checkpoints
- 3D detection code
- Code release
This project is based on excellent open source projects:
If this work is helpful for your research, please consider citing the following BibTeX entry.
@inproceedings{kim2023crn,
title={Crn: Camera radar net for accurate, robust, efficient 3d perception},
author={Kim, Youngseok and Shin, Juyeb and Kim, Sanmin and Lee, In-Jae and Choi, Jun Won and Kum, Dongsuk},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={17615--17626},
year={2023}
}