The implementation of our IJCV 2023 paper "Instance Segmentation in the Dark".
Interested readers are also referred to an insightful Note about this work in Zhihu. (TODO)
Abstract
Existing instance segmentation methods are primarily designed for high-visibility inputs, and their performance degrades drastically in extremely low-light environments. In this work, we take a deep look at instance segmentation in the dark and introduce several techniques that substantially boost the low-light inference accuracy. Our method design is motivated by the observation that noise in low-light images introduces high-frequency disturbances to the feature maps of neural networks, thereby significantly degrading performance. To suppress this ``feature noise", we propose a novel learning method that relies on an adaptive weighted downsampling layer, a smooth-oriented convolutional block, and disturbance suppression learning. They can reduce feature noise during downsampling and convolution operation, and enable the model to learn disturbance-invariant features, respectively. Additionally, we find that RAW images with high bit-depth can preserve richer scene information in low-light conditions compared to typical camera sRGB outputs, thus supporting the use of RAW-input algorithms. Our analysis indicates that high bit-depth can be critical for low-light instance segmentation. To tackle the lack of annotated RAW datasets, we leverage a low-light RAW synthetic pipeline to generate realistic low-light data. Furthermore, to support this line of work, we capture a real-world low-light instance segmentation dataset. It contains more than two thousand paired low/normal-light images with instance-level pixel-wise annotations. Without any image preprocessing, we achieve satisfactory performance on instance segmentation in very low light (4~% AP higher than state-of-the-art competitors), meanwhile opening new opportunities for future research.
- 4th International Workshop on Physics-Based Vision Meets Deep Learning (PBDL) in Conjunction with CVPR 2024, Seattle, WA, USA.
- The Low-light Object Detection and Instance Segmentation track starts now! We release validation data and training data. Check out this page and prepare the submission!
- More challenges from the CVPR 2024 workshop PBDL can be found at this link!
- 2024.02.20 Challenge site online
- 2024.02.21 Release of train data (paired images) and validation data (inputs only)
- 2024.03.01 Validation server online
- 2024.04.23 Final test data release (inputs only)
- 2024.04.30 Test submission deadline
- 2024.05.05 Fact sheets and code/executable submission deadline
- 2024.05.10 Preliminary test and rating results release to participants
-
We propose an adaptive weighted downsampling layer, smooth-oriented convolutional block and disturbance suppression learning to address the high-frequency disturbance within deep features that occurred in very low light. Interestingly, they also benefit the normal-lit instance segmentation.
-
We exploit the potentials of RAW-input design for low-light instance segmentation and leverage a low-light RAW synthetic pipeline to generate realistic low-light RAW images from existing datasets, which facilitates end-to-end training.
-
We collect a real-world low-light dataset with precise pixel-wise instance-level annotations, namely LIS, which covers more than two thousand scenes and can serve as a benchmark for instance segmentation in the dark. On LIS, our approach outperforms state-of-the-art competitors in terms of both segmentation accuracy and inference speed by a large margin.
The adaptive weighted downsampling (AWD) layer, smooth-oriented convolutional block (SCB), and disturbance suppression loss are designed to reduce the feature disturbance caused by noise, and the low-light RAW synthetic pipeline is employed to facilitate end-to-end training of instance segmentation on RAW images.
Four image types (long-exposure normal-light and short-exposure low-light images in both RAW and sRGB formats) are captured for each scene.
The synthetic pipeline from RGB to RAW in low-light conditions can be found at here.
You can simply use it in mmdetection as follows:
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='LoadAnnotations',
with_bbox=True,
with_mask=True,
poly2mask=False),
dict(type='Resize', img_scale=(600, 400), keep_ratio=True, interpolate_mode='nearest'),
dict(type='AddNoisyImg', model='PGRU', camera='CanonEOS5D4',
cfa='bayer', use_255=True, pre_adjust_brightness=False, mode='unprocess_addnoise', dark_ratio=(1.0, 1.0), noise_ratio=(10, 100)), # here
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'noisy_img', 'gt_bboxes', 'gt_labels', 'gt_masks']),
# dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']),
]
The AWD can be found at here.
The SConv can be found at here.
The DSL can be found at here.
Our code is based on MMDetection.
Please refer to get_started.md for installation and dataset_prepare.md for dataset preparation.
Results are reported on LIS test set.
| Model | Backbone | Train set | Seg AP | Box AP | Config | CKPT |
|---|---|---|---|---|---|---|
| Mask R-CNN | R50 | COCO | 19.8 | 22.8 | config | model (IJCV) |
| Mask R-CNN (Ours) | R50 | Syn COCO | 31.8 | 37.6 | config | model (IJCV) |
| Mask R-CNN | ConvNeXt-T | COCO | 23.7 | 27.9 | config | model (IJCV) |
| Mask R-CNN (Ours) | ConvNeXt-T | Syn COCO | 36.8 | 42.7 | config | model (IJCV) |
| PointRend | R50 | COCO | 20.6 | 23.5 | config | model (IJCV) |
| PointRend (Ours) | R50 | Syn COCO | 32.8 | 37.1 | config | model (IJCV) |
| Mask2Former | R50 | COCO | 21.4 | 22.9 | config | model (IJCV) |
| Mask2Former (Ours) | R50 | Syn COCO | 35.6 | 37.8 | config | model (IJCV) |
We do not tune hyperparameters like loss weights. Further adjusting the hyperparameters should lead to improvement.
For future research, we suggest using COCO as train set and the whole LIS as test set.
| Model | Backbone | Train set | Seg AP | Box AP |
|---|---|---|---|---|
| Mask R-CNN | R50 | COCO | 19.8 | 22.8 |
| Mask R-CNN (Ours) | R50 | Syn COCO | 27.2 | 33.3 |
| Mask R-CNN | ConvNeXt-T | COCO | 19.7 | 24.2 |
| Mask R-CNN (Ours) | ConvNeXt-T | Syn COCO | 32.6 | 39.1 |
| PointRend | R50 | COCO | 17.3 | 20.7 |
| PointRend (Ours) | R50 | Syn COCO | 27.3 | 32.0 |
| Mask2Former | R50 | COCO | 19.0 | 20.7 |
| Mask2Former (Ours) | R50 | Syn COCO | 31.1 | 34.1 |
Results are reported on normal-light COCO val set.
| Model | Backbone | Train | Seg AP | Box AP | Config | CKPT |
|---|---|---|---|---|---|---|
| Mask R-CNN | R50 | COCO | 34.4 | 38.0 | config | model |
| Mask R-CNN+AWD+SCB+DSL | R50 | COCO | 36.1 | 39.5 | config | model (IJCV) |
Download images including RAW-dark, RAW-normal, RGB-dark, RGB-normal, and annotations from Baidu Drive (extraction code: IJCV) or Google Drive.
If the links are not available, please feel free to contact me.
LIS dataset should follow folder structure bellow:
├── LIS
│ ├── RGB-normal
│ │ ├── JPEGImages
│ │ │ ├──1.png
│ │ │ ├──3.png
│ │ │ ├──5.png
│ │ │ ├──...
│ ├── RGB-dark
│ │ ├── JPEGImages
│ │ │ ├──2.JPG
│ │ │ ├──4.JPG
│ │ │ ├──6.JPG
│ │ │ ├──...
│ ├── RAW-normal
│ │ ├── JPEGImages
│ │ │ ├──...
│ ├── RAW-dark
│ │ ├── JPEGImages
│ │ │ ├──...
│ ├── annotations
│ │ ├──lis_coco_JPG_train+1.json # w/ '+1' indicates labels for dark images
│ │ ├──lis_coco_JPG_test+1.json
│ │ ├──lis_coco_JPG_traintest+1.json
│ │ ├──lis_coco_png_train.json # w/o '+1' indicates labels for normal images
│ │ ├──lis_coco_png_test.json
│ │ ├──lis_coco_png_traintest.json
│ │ ├──lis_coco_png_train+1.json
│ │ ├──lis_coco_png_test+1.json
│ │ ├──lis_coco_png_traintest+1.json
Original RAW files are avaliable in our previous work.
If you use our dataset or code for research, please cite this paper and our previous work:
@article{2023lis,
title={Instance Segmentation in the Dark},
author={Chen, Linwei and Fu, Ying and Wei, Kaixuan and Zheng, Dezhi and Heide, Felix},
journal={International Journal of Computer Vision},
volume={131},
number={8},
pages={2198--2218},
year={2023},
publisher={Springer}
}
@inproceedings{Hong2021Crafting,
title={Crafting Object Detection in Very Low Light},
author={Yang Hong, Kaixuan Wei, Linwei Chen, Ying Fu},
booktitle={BMVC},
year={2021}
}
If you find any problem, please feel free to contact me (Linwei at [email protected]). A brief self-introduction (including your name, affiliation, and position) is required, if you would like to get in-depth help from me. I'd be glad to talk with you if more information (e.g. your personal website link) is attached.