CycleISP: Real Image Restoration via Improved Data Synthesis (CVPR 2020 -- Oral)

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao

Paper: https://arxiv.org/abs/2003.07761

Supplementary: pdf

Video Presentation: https://www.youtube.com/watch?v=41XKXY--7_E

CycleISP for Synthesizing Data for Image Denoising

The proposed CycleISP framework allows converting sRGB images to RAW data, and then back to sRGB images. It has (a) RGB2RAW network branch, and (b) RAW2RGB network branch.

Overall Framework of CycleISP

Recursive Residual Group (RRG)

Proposed Denoising Network

Installation

The model is built in PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.7, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

sudo apt-get install cmake build-essential libjpeg-dev libpng-dev
conda create -n pytorch1 python=3.7
conda activate pytorch1
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image yacs lycon natsort h5py tqdm

Data Synthesis using CycleISP

• Download all the models and place them in ./pretrained_models/isp/
• Download some sample images from the MIR-Flickr dataset and place them in ./datasets/some_sample_images/

Generating Data for RAW Denoising

The RGB2RAW network branch takes as input a clean sRGB image and converts it to a clean RAW image. The noise injection module adds shot and read noise of different levels to the (RAW) output of RGB2RAW network branch. Thereby, we can generate clean and its corresponding noisy image pairs {RAW_clean, RAW_noisy} from any sRGB image.

• Run Demo

python generate_raw_data.py 

Generating Data for sRGB Denoising

Given a synthetic RAW noisy image as input, the RAW2RGB network branch maps it to a noisy sRGB image; hence we are able to generate an image pair {sRGB_clean, sRGB_noisy} for the sRGB denoising problem.

• Run Demo

python generate_rgb_data.py 

Evaluation

Denoising RAW images of DND

• Download the model and place it in ./pretrained_models/denoising/
• Download RAW images of DND and place them in ./datasets/dnd/dnd_raw/
• Run

python test_dnd_raw.py --save_images

Denoising RAW images of SIDD

• Download the model and place it in ./pretrained_models/denoising/
• Download RAW images of SIDD and place them in ./datasets/sidd/sidd_raw/
• Run

python test_sidd_raw.py --save_images

Denoising sRGB images of DND

• Download the model and place it in ./pretrained_models/denoising/
• Download sRGB images of DND and place them in ./datasets/dnd/dnd_rgb/noisy/
• Run

python test_dnd_rgb.py --save_images

Denoising sRGB images of SIDD

• Download the model and place it in ./pretrained_models/denoising/
• Download sRGB images of SIDD and place them in ./datasets/sidd/sidd_rgb/
• Run

python test_sidd_rgb.py --save_images

Results on Real Image Datasets

Experiments are performed for denoising images in RAW and sRGB spaces.

Results for RAW Denoising

Results for sRGB Denoising

Citation

If you use CycleISP, please consider citing:

@inproceedings{Zamir2020CycleISP,
    title={CycleISP: Real Image Restoration via Improved Data Synthesis},
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat
            and Fahad Shahbaz Khan and Ming-Hsuan Yang and Ling Shao},
    booktitle={CVPR},
    year={2020}
}

Contact

Should you have any question, please contact [email protected]