PMPI: Patch-based Multiplane Images for learning View Synthesis

We present PMPI, an explicit and plane-based model for real-time view synthesis. Our PMPI achieves more dense and effective samplings without adding computing resources by assembling patches around visible contents.

PMPI model

Our method for learning PMPI

Table of contents

---

• Installation
• Dataset
• Training
• Rendering
• Citation

---

Installation

We provide environment.yml to help you setup a conda environment.

conda env create -f environment.yml

Then build aabb module.

python setup.py build_ext --inplace
pip install -e ./

Dataset

NeRF's real forward-facing dataset

Download: Undistorted front facing dataset

For real forward-facing dataset, NeRF is trained with the raw images, which may contain lens distortion. But we use the undistorted images provided by COLMAP.

However, you can try running other scenes from Local lightfield fusion (Eg. airplant) without any changes in the dataset files. In this case, the images are not automatically undistorted.

Using your own images.

Running PMPI on your own images. You need to install COLMAP on your machine.

Then, put your images into a directory following this structure

<scene_name>
|-- images
     | -- image_name1.jpg
     | -- image_name2.jpg
     ...

The training code will automatically prepare a scene for you. You may have to tune planes.txt to get better reconstruction (see dataset explaination)

Training

python train_progressive.py -scene data/forward/fortress_undistort -model_dir model/fortress_192layers_c2f -epochs 4000 -before_decay 0 -n_max 384 -ray 4 -cv2resize -layers 192 -sublayers 1 -size_patch 36 -offset 216 -tb_toc 100 -gamma 0.0006 -all_gpu

python train.py -scene ${PATH_TO_SCENE} -model_dir ${MODEL_TO_SAVE_CHECKPOINT} -epochs 4000 -before_decay 0 -n_max 384 -ray 4  -layers 192 -sublayers 1 -size_patch 36 -offset 216  -gamma 0.0006 -all_gpu

This implementation uses scikit-image to resize images during training by default. The results and scores in the paper are generated using OpenCV's resize function. If you want the same behavior, please add -cv2resize argument.

Note that this code is tested on two Nvidia V100 32GB.

For a GPU/GPUs with less memory (e.g., a single RTX 3090), you can run using the following command:

python train.py -scene ${PATH_TO_SCENE} -model_dir ${MODEL_TO_SAVE_CHECKPOINT} -epochs 4000 -before_decay 0 -n_max 256 -ray 4  -layers 64 -sublayers 1 -size_patch 36 -offset 216  -gamma 0.0006

Note that when your GPU runs out of memeory, you can try reducing the number of layers, sublayers, and sampled rays.

Citation

Acknowledge

Our repo is developed based on NeX and NSVF