Scene Prior Filtering for Depth Map Super-Resolution

Zhengxue Wang¹, Zhiqiang Yan¹, Ming-Hsuan Yang², Jinshan Pan¹, Guangwei Gao³, Ying Tai⁴, Jian Yang¹

¹Nanjing University of Science and Technology    ²University of California at Merced   
³Nanjing University of Posts and Telecommunications    ⁴Nanjing University   

[Paper] [Project Page]

SPFNet. It first produces the normal $\boldsymbol I_{n}$ and semantic $\boldsymbol I_{s}$ priors from $\boldsymbol I_{r}$ using large-scale models. Then, the scene prior branch (orange part) extracts the multi-modal features. Meanwhile, the depth branch (blue part) recursively conducts all-in-one prior propagation (APP) and one-to-one prior embedding (OPE). BI: bicubic interpolation.

Scheme of (a) All-in-one Prior Propagation (APP), and (b) histogram comparison of scene prior features before and after APP.

Scheme of (a) One-to-one Prior Embedding (OPE), and (b) gradient histogram of filter kernels in the texture area (green box).

Dependencies

Python==3.11.5
PyTorch==2.1.0
numpy==1.23.5 
torchvision==0.16.0
scipy==1.11.3
Pillow==10.0.1
tqdm==4.65.0
scikit-image==0.21.0

Datasets

All Datasets can be found here.

Models

All pretrained models can be found here.

Training

SPFNet

Train on synthetic NYU-v2
# x4 DSR
> python train.py --scale 4 --num_feats 42
# x8 DSR
> python train.py --scale 8 --num_feats 42
# x16 DSR
> python train.py --scale 16 --num_feats 42
Train on real-world RGB-D-D
> python train.py --scale 4 --num_feats 20 

SPFNet-T

Train on synthetic NYU-v2
# x4 DSR
> python train.py --scale 4 --num_feats 6 --tiny_model
# x8 DSR
> python train.py --scale 8 --num_feats 6 --tiny_model
# x16 DSR
> python train.py --scale 16 --num_feats 6 --tiny_model
Train on real-world RGB-D-D
> python train.py --scale 4 --num_feats 6 --tiny_model

Testing

SPFNet

## Test on synthetic datasets
### x4 DSR
> python test.py --scale 4 --num_feats 42
### x8 DSR
> python test.py --scale 8 --num_feats 42
### x16 DSR
> python test.py --scale 16 --num_feats 42
## Test on real-world RGB-D-D
> python test.py --scale 4 --num_feats 20 --downsample real

SPFNet-T

## Test on synthetic datasets
### x4 DSR
> python test.py --scale 4 --num_feats 6 --tiny_model
### x8 DSR
> python test.py --scale 8 --num_feats 6 --tiny_model
### x16 DSR
> python test.py --scale 16 --num_feats 6 --tiny_model
## Test on real-world RGB-D-D
> python test.py --scale 4 --num_feats 6 --downsample real --tiny_model

Experiments

Quantitative comparison

Visual comparison

Train & test on real-world RGB-D-D:

Train & test on synthetic NYU-v2 (x16):

Acknowledgements

We thank Xinni Jiang for her invaluable assistance.

We thank these repos sharing their codes: DKN and SUFT.

Citation

@article{wang2024scene,
  title={Scene Prior Filtering for Depth Map Super-Resolution},
  author={Wang, Zhengxue and Yan, Zhiqiang and Yang, Ming-Hsuan and Pan, Jinshan and Yang, Jian and Tai, Ying and Gao, Guangwei},
  journal={arXiv preprint arXiv:2402.13876},
  year={2024}
}