Pre-Trained Image Processing Transformer (IPT)

By Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao. [arXiv]

We study the low-level computer vision task (such as denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). We present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks.

MindSpore Code

• Official MindSpore code
• Pretrained weights for super-resolution X4

News

• Pytorch pre-trained model will be released in June!

Requirements

• python 3
• pytorch >= 1.4.0
• torchvision

Usage

Coming soon!

Results

• Detailed results on image super-resolution task.

| Method | Scale | Set5 | Set14 | B100 | Urban100 | | :-----------------------------------------------------: | :-------: | :----------: | :----------: | :----------: | :----------: | | [VDSR](https://github.com/twtygqyy/pytorch-vdsr) | $\times$2 | 37.53 | 33.05 | 31.90 | 30.77 | | [EDSR](https://github.com/sanghyun-son/EDSR-PyTorch) | $\times$2 | 38.11 | 33.92 | 32.32 | 32.93 | | [RCAN](https://github.com/yulunzhang/RCAN) | $\times$2 | 38.27 | 34.12 | 32.41 | 33.34 | | [RDN](https://github.com/yulunzhang/RDN) | $\times$2 | 38.24 | 34.01 | 32.34 | 32.89 | | [OISR-RK3](https://github.com/HolmesShuan/OISR-PyTorch) | $\times$2 | 38.21 | 33.94 | 32.36 | 33.03 | | [RNAN](https://github.com/yulunzhang/RNAN) | $\times$2 | 38.17 | 33.87 | 32.32 | 32.73 | | [SAN](https://github.com/hszhao/SAN) | $\times$2 | 38.31 | 34.07 | 32.42 | 33.1 | | [HAN](https://github.com/wwlCape/HAN) | $\times$2 | 38.27 | 34.16 | 32.41 | 33.35 | | [IGNN](https://github.com/sczhou/IGNN) | $\times$2 | 38.24 | 34.07 | 32.41 | 33.23 | | IPT (ours) | $\times$2 | **38.37** | **34.43** | **32.48** | **33.76** |

Method	Scale	Set5	Set14	B100	Urban100
VDSR	$\times$3	33.67	29.78	28.83	27.14
EDSR	$\times$3	34.65	30.52	29.25	28.80
RCAN	$\times$3	34.74	30.65	29.32	29.09
RDN	$\times$3	34.71	30.57	29.26	28.80
OISR-RK3	$\times$3	34.72	30.57	29.29	28.95
RNAN	$\times$3	34.66	30.52	29.26	28.75
SAN	$\times$3	34.75	30.59	29.33	28.93
HAN	$\times$3	34.75	30.67	29.32	29.10
IGNN	$\times$3	34.72	30.66	29.31	29.03
IPT (ours)	$\times$3	34.81	30.85	29.38	29.49

Method	Scale	Set5	Set14	B100	Urban100
VDSR	$\times$4	31.35	28.02	27.29	25.18
EDSR	$\times$4	32.46	28.80	27.71	26.64
RCAN	$\times$4	32.63	28.87	27.77	26.82
SAN	$\times$4	32.64	28.92	27.78	26.79
RDN	$\times$4	32.47	28.81	27.72	26.61
OISR-RK3	$\times$4	32.53	28.86	27.75	26.79
RNAN	$\times$4	32.49	28.83	27.72	26.61
HAN	$\times$4	32.64	28.90	27.80	26.85
IGNN	$\times$4	32.57	28.85	27.77	26.84
IPT (ours)	$\times$4	32.64	29.01	27.82	27.26

• Super-resolution result

• Denoising result

• Derain result

Citation

@misc{chen2020pre,
      title={Pre-Trained Image Processing Transformer}, 
      author={Chen, Hanting and Wang, Yunhe and Guo, Tianyu and Xu, Chang and Deng, Yiping and Liu, Zhenhua and Ma, Siwei and Xu, Chunjing and Xu, Chao and Gao, Wen},
      year={2021},
      eprint={2012.00364},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}