nuScenes support

NUSCENES-GUIDE.md

@@ -0,0 +1,18 @@
+
+## Guide for nuScenes dataset
+
+### Requirements
+nuscenes-devkit=1.1.1
+pyyaml=5.3.1
+
+
+### Workflow
+
+1. Pre-processing: we follow the data pre-processing in SECOND to prepare
+the nuscenes_info file, including nuscenes_infos_train.pkl, nuscenes_infos_val.pkl
+and nuscenes_infos_test.pkl. We also provide the generated files in
+[LINK1](https://drive.google.com/drive/folders/1zSZ9xE4UkKBMCMH0le7KdSxvbyjuuUp8?usp=sharing) or [LINK2](https://pan.baidu.com/s/1ChK6iua0lENJ-ZvcFjc8Ew) (access code: 2e53)
+
+2. Custom Configuration: modify the path in config file with your own settings
+
+3. Run the train_nusc.sh

README.md

@@ -5,11 +5,12 @@
 ![img|center](./img/pipeline.png)
 
 ## News
+- **2020-12 [NEW:fire:]** Cylinder3D achieves the 2nd place in the challenge of nuScenes LiDAR segmentation, with mIoU=0.779, fwIoU=0.899 and FPS=10Hz.
+- **2020-12 [NEW:fire:]** We release the new version of Cylinder3D with nuScenes dataset support.
 - **2020-11** We preliminarily release the Cylinder3D--v0.1, supporting the LiDAR semantic segmentation on SemanticKITTI and nuScenes.
 - **2020-11** Our work achieves the **1st place** in the leaderboard of SemanticKITTI semantic segmentation (until CVPR2021 DDL, still rank 1st in term of Accuracy now), and based on the proposed method, we also achieve the **1st place** in the leaderboard of SemanticKITTI panoptic segmentation.
-<p align="center">
-        <img src="./img/leaderboard.png" width="40%"> 
-</p>
+
+![img|center](./img/leaderboard.png)
 
 ## Installation
 
@@ -19,7 +20,7 @@
 - Cython
 - [torch-scatter](https://github.com/rusty1s/pytorch_scatter)
 - [nuScenes-devkit](https://github.com/nutonomy/nuscenes-devkit) (optional for nuScenes)
-- [spconv](https://github.com/traveller59/spconv) (tested with spconv==1.2.1 and cuda==10.2)
+- [spconv](https://github.com/traveller59/spconv)
 
 ## Data Preparation
 
@@ -57,27 +58,28 @@
 		├──sweeps
 		├──maps
 
-		
 ```
 
 ## Training
-1. modify the config/semantickitti.yaml with your custom settings. We provide a sample yaml for SemanticKITTI.
-2. train the network by running "sh train.sh".
+1. modify the config/semantickitti.yaml with your custom settings. We provide a sample yaml for SemanticKITTI
+2. train the network by running "sh train.sh"
 
+### Training for nuScenes
+Please refer to [NUSCENES-GUIDE](./NUSCENES-GUIDE.md)
 
 ### Pretrained Models
 -- We provide a pretrained model for SemanticKITTI [LINK1](https://drive.google.com/file/d/1q4u3LlQXz89LqYW3orXL5oTs_4R2eS8P/view?usp=sharing) or [LINK2](https://pan.baidu.com/s/1c0oIL2QTTcjCo9ZEtvOIvA) (access code: xqmi)
+-- For nuScenes dataset, please refer to [NUSCENES-GUIDE](./NUSCENES-GUIDE.md)
 
 ## TODO List
-- [ ] Release pretrained model for nuScenes.
+- [x] Release pretrained model for nuScenes.
 - [ ] Support more models, including PolarNet, RandLA, SequeezeV3 and etc.
 - [ ] Support more datasets, including A2D2 and etc.  
-- [ ] Integrate LiDAR Panoptic Segmentation into the codebase.
-
+- [ ] Integrate LiDAR Panotic Segmentation into the codebase.
 
 ## Reference
 
-If you find our work useful in your research, please consider citing our paper:
+If you find our work useful in your research, please consider citing our [paper](https://arxiv.org/pdf/2011.10033):
 ```
 @article{zhu2020cylindrical,
   title={Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation},
@@ -86,8 +88,14 @@ If you find our work useful in your research, please consider citing our paper:
   year={2020}
 }
 
+#for LiDAR panoptic segmentation
+@article{hong2020lidar,
+  title={LiDAR-based Panoptic Segmentation via Dynamic Shifting Network},
+  author={Hong, Fangzhou and Zhou, Hui and Zhu, Xinge and Li, Hongsheng and Liu, Ziwei},
+  journal={arXiv preprint arXiv:2011.11964},
+  year={2020}
+}
 ```
 
-
 ## Acknowledgments
-We thanks for the opensource codebases, [PolarSeg](https://github.com/edwardzhou130/PolarSeg)  and [SPVNAS](https://github.com/mit-han-lab/e3d)
+We thanks for the opensource codebases, [PolarSeg](https://github.com/edwardzhou130/PolarSeg) and [spconv](https://github.com/traveller59/spconv)

builder/data_builder.py

@@ -21,10 +21,15 @@ def build(dataset_config,
 
     SemKITTI = get_pc_model_class(dataset_config['pc_dataset_type'])
 
-    train_pt_dataset = SemKITTI(data_path + '/sequences/', imageset=train_imageset,
-                                return_ref=train_ref, label_mapping=label_mapping)
-    val_pt_dataset = SemKITTI(data_path + '/sequences/', imageset=val_imageset,
-                              return_ref=val_ref, label_mapping=label_mapping)
+    nusc=None
+    if "nusc" in dataset_config['pc_dataset_type']:
+        from nuscenes import NuScenes
+        nusc = NuScenes(version='v1.0-trainval', dataroot=data_path, verbose=True)
+
+    train_pt_dataset = SemKITTI(data_path, imageset=train_imageset,
+                                return_ref=train_ref, label_mapping=label_mapping, nusc=nusc)
+    val_pt_dataset = SemKITTI(data_path, imageset=val_imageset,
+                              return_ref=val_ref, label_mapping=label_mapping, nusc=nusc)
 
     train_dataset = get_model_class(dataset_config['dataset_type'])(
         train_pt_dataset,

builder/loss_builder.py

@@ -7,29 +7,8 @@
 
 
 def build(wce=True, lovasz=True, num_class=20, ignore_label=0):
-    ## for semanticKITTI
-    weights = torch.zeros(num_class, dtype=torch.float)
-    weights[0] = 1.0
-    weights[1] = 2.293
-    weights[2] = 85.756
-    weights[3] = 71.511
-    weights[4] = 31.596
-    weights[5] = 35.624
-    weights[6] = 74.761
-    weights[7] = 88.722
-    weights[8] = 96.389
-    weights[9] = 1.00
-    weights[10] = 6.362
-    weights[11] = 1.00
-    weights[12] = 20.387
-    weights[13] = 1.00
-    weights[14] = 1.363
-    weights[15] = 1.00
-    weights[16] = 14.214
-    weights[17] = 1.263
-    weights[18] = 25.936
-    weights[19] = 61.896
-    loss_funs = torch.nn.CrossEntropyLoss(weight=weights.cuda(), ignore_index=ignore_label)
+
+    loss_funs = torch.nn.CrossEntropyLoss(ignore_index=ignore_label)
 
     if wce and lovasz:
         return loss_funs, lovasz_softmax

config/label_mapping/nuscenes.yaml

@@ -0,0 +1,84 @@
+labels:
+  0: 'noise'
+  1: 'animal'
+  2: 'human.pedestrian.adult'
+  3: 'human.pedestrian.child'
+  4: 'human.pedestrian.construction_worker'
+  5: 'human.pedestrian.personal_mobility'
+  6: 'human.pedestrian.police_officer'
+  7: 'human.pedestrian.stroller'
+  8: 'human.pedestrian.wheelchair'
+  9: 'movable_object.barrier'
+  10: 'movable_object.debris'
+  11: 'movable_object.pushable_pullable'
+  12: 'movable_object.trafficcone'
+  13: 'static_object.bicycle_rack'
+  14: 'vehicle.bicycle'
+  15: 'vehicle.bus.bendy'
+  16: 'vehicle.bus.rigid'
+  17: 'vehicle.car'
+  18: 'vehicle.construction'
+  19: 'vehicle.emergency.ambulance'
+  20: 'vehicle.emergency.police'
+  21: 'vehicle.motorcycle'
+  22: 'vehicle.trailer'
+  23: 'vehicle.truck'
+  24: 'flat.driveable_surface'
+  25: 'flat.other'
+  26: 'flat.sidewalk'
+  27: 'flat.terrain'
+  28: 'static.manmade'
+  29: 'static.other'
+  30: 'static.vegetation'
+  31: 'vehicle.ego'
+labels_16:
+  0: 'noise'
+  1: 'barrier'
+  2: 'bicycle'
+  3: 'bus'
+  4: 'car'
+  5: 'construction_vehicle'
+  6: 'motorcycle'
+  7: 'pedestrian'
+  8: 'traffic_cone'
+  9: 'trailer'
+  10: 'truck'
+  11: 'driveable_surface'
+  12: 'other_flat'
+  13: 'sidewalk'
+  14: 'terrain'
+  15: 'manmade'
+  16: 'vegetation'
+learning_map:
+  1: 0
+  5: 0
+  7: 0
+  8: 0
+  10: 0
+  11: 0
+  13: 0
+  19: 0
+  20: 0
+  0: 0
+  29: 0
+  31: 0
+  9: 1
+  14: 2
+  15: 3
+  16: 3
+  17: 4
+  18: 5
+  21: 6
+  2: 7
+  3: 7
+  4: 7
+  6: 7
+  12: 8
+  22: 9
+  23: 10
+  24: 11
+  25: 12
+  26: 13
+  27: 14
+  28: 15
+  30: 16

config/nuScenes.yaml

@@ -0,0 +1,68 @@
+# Config format schema number
+format_version: 4
+
+###################
+## Model options
+model_params:
+  model_architecture: "cylinder_asym"
+
+  output_shape:
+    - 480
+    - 360
+    - 32
+
+  fea_dim: 9
+  out_fea_dim: 256
+  num_class: 17
+  num_input_features: 16
+  use_norm: True
+  init_size: 32
+
+
+###################
+## Dataset options
+dataset_params:
+  dataset_type: "cylinder_dataset_nuscenes"
+  pc_dataset_type: "SemKITTI_nusc"
+  ignore_label: 0
+  return_test: False
+  fixed_volume_space: True
+  label_mapping: "./config/label_mapping/nuscenes.yaml"
+  max_volume_space:
+    - 50
+    - 3.1415926
+    - 3
+  min_volume_space:
+    - 0
+    - -3.1415926
+    - -5
+
+
+###################
+## Data_loader options
+train_data_loader:
+  data_path: "/data/dataset/nuScenes/"
+  imageset: "/data/dataset/nuScenes/nuscenes_infos_train.pkl"
+  return_ref: True
+  batch_size: 2
+  shuffle: True
+  num_workers: 4
+
+val_data_loader:
+  data_path: "/data/dataset/nuScenes/"
+  imageset: "/data/dataset/nuScenes/nuscenes_infos_val.pkl"
+  return_ref: True
+  batch_size: 1
+  shuffle: False
+  num_workers: 4
+
+
+###################
+## Train params
+train_params:
+  model_load_path: "./model_load_dir_nuscenes/model_load.pt"
+  model_save_path: "./model_save_dir_nuscenes/model_save.pt"
+  checkpoint_every_n_steps: 4599
+  max_num_epochs: 40
+  eval_every_n_steps: 4599
+  learning_rate: 0.001

config/semantickitti.yaml

@@ -41,15 +41,15 @@ dataset_params:
 ###################
 ## Data_loader options
 train_data_loader:
-  data_path: "/data/dataset/semantic_kitti/data_semkitti/dataset"
+  data_path: "/data/dataset/semantic_kitti/data_semkitti/dataset/sequences/"
   imageset: "train"
   return_ref: True
   batch_size: 2
   shuffle: True
   num_workers: 4
 
 val_data_loader:
-  data_path: "/data/dataset/semantic_kitti/data_semkitti/dataset"
+  data_path: "/data/dataset/semantic_kitti/data_semkitti/dataset/sequences/"
   imageset: "val"
   return_ref: True
   batch_size: 1

dataloader/__init__.py

@@ -1,3 +1,5 @@
 # -*- coding:utf-8 -*-
 # author: Xinge
 # @file: __init__.py.py
+
+from . import dataset_nuscenes