packaged InstantSplat

This repo is the refactored python training and inference code for InstantSplat. Forked from commit 2c5006d41894d06464da53d5495300860f432872. We refactored the original code following the standard Python package structure, while keeping the algorithms used in the code identical to the original version.

Initialization methods:

• DUST3R (same method used in InstantSplat)
• COLMAP Sparse reconstruct (same method used in gaussian-splatting)
• COLMAP Dense reconstruct (use patch_match_stereo, stereo_fusion, poisson_mesher and delaunay_mesher in COLMAP to reconstruct dense point cloud for initialization)

Install

Requirements

Install Pytorch and torchvision following the official guideline: pytorch.org

Install gaussian-splatting. You can download wheel from Releases, or install latest from source:

pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master

Local Install

git clone https://github.com/yindaheng98/InstantSplat --recursive
cd InstantSplat
pip install opencv-python roma scipy huggingface_hub einops open3d
pip install --target . --upgrade . --no-deps

(Optional) If you do not want to install those related dependencies in your env:

pip install --target . --no-deps --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master

Pip Install

You can download wheel from Releases, or install latest from source:

pip install --upgrade git+https://github.com/yindaheng98/InstantSplat.git@main

Download model

wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_224_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth

Running

1. Initialize coarse point cloud and jointly train 3DGS & cameras

python train.py -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init

2. Render it

python render.py -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --load_camera output/sora/santorini/3_views/cameras.json

(Optional) Initialize coarse point and save as a Colmap workspace than jointly train 3DGS & cameras

python initialize.py -d data/sora/santorini/3_views
python train.py -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000

Usage

See initialize.py, train.py and render.py for full example.

Also check yindaheng98/gaussian-splatting for more detail of training process.

Gaussian models

Use CameraTrainableGaussianModel in yindaheng98/gaussian-splatting

Dataset

Use TrainableCameraDataset in yindaheng98/gaussian-splatting

Initialize coarse point cloud and cameras

from instant_splat.initializer import Dust3rInitializer
image_path_list = [os.path.join(image_folder, file) for file in sorted(os.listdir(image_folder))]
initializer = Dust3rInitializer(...).to(args.device) # see instant_splat/initializer/dust3r/dust3r.py for full options
initialized_point_cloud, initialized_cameras = initializer(image_path_list=image_path_list)

Create camera dataset from initialized cameras:

from instant_splat.initializer import TrainableInitializedCameraDataset
dataset = TrainableInitializedCameraDataset(initialized_cameras).to(device)

Initialize 3DGS from initialized coarse point cloud:

gaussians.create_from_pcd(initialized_point_cloud.points, initialized_point_cloud.colors)

Training

Trainer jointly optimize the 3DGS parameters and cameras, without densification

from instant_splat.trainer import Trainer
trainer = Trainer(
    gaussians,
    scene_extent=dataset.scene_extent(),
    dataset=dataset,
    ... # see instant_splat/trainer/trainer.py for full options
)

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

This repository is the official implementation of InstantSplat, an sparse-view, SfM-free framework for large-scale scene reconstruction method using Gaussian Splatting. InstantSplat supports 3D-GS, 2D-GS, and Mip-Splatting.

Free-view Rendering

example.mp4

TODO List

• Confidence-aware Point Cloud Downsampling
• Support 2D-GS
• Support Mip-Splatting

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• DUSt3R

Citation

If you find our work useful in your research, please consider giving a star ⭐ and citing the following paper 📝.

@misc{fan2024instantsplat,
        title={InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds},
        author={Zhiwen Fan and Wenyan Cong and Kairun Wen and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }