InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

Modified implementation of InstantSplat for Supporting [MAST3r](https://europe.naverlabs.com/blog/mast3r-matching-and-stereo-3d-reconstruction).

Table of Contents

• Table of Contents
• Free-view Rendering
• Get Started
  • Installation
  • Usage
• Acknowledgement
• Citation

TODO List

• Support MAST3r for SfM
• Confidence-aware Point Cloud Downsampling

Get Started

Installation

1. Clone InstantSplat and download pre-trained model.

git clone --recursive https://github.com/NVlabs/InstantSplat.git
cd InstantSplat
git submodule update --init --recursive

2. Create the environment (or use pre-built docker), here we show an example using conda.

conda create -n instantsplat python=3.11 cmake=3.14.0
conda activate instantsplat
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
pip install submodules/simple-knn
# modify the rasterizer
vim submodules/diff-gaussian-rasterization/cuda_rasterizer/auxiliary.h
'p_view.z <= 0.2f' -> 'p_view.z <= 0.001f' # line 154
pip install submodules/diff-gaussian-rasterization

3. Optional but highly suggested, compile the cuda kernels for RoPE (as in CroCo v2).

# DUST3R relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd submodules/dust3r/croco/models/curope/
python setup.py build_ext --inplace

Alternative: use the pre-built docker image: pytorch/pytorch:2.1.2-cuda11.8-cudnn8-devel

docker pull dockerzhiwen/instantsplat_public:2.0

if docker failed to produce reasonable results, try Installation step again within the docker.

Usage

1. Data preparation

  <data_path>/<dataset-name>/<scene-name>/<#view_views>/images/

2. Command

  # InstantSplat train and output video (no GT reference, render by interpolation) using the following command.
  bash scripts/run_train_infer.sh

  # InstantSplat train and evaluate (with GT reference) using the following command.
  bash scripts/run_train_eval.sh

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• MASt3R

Citation

If you find our work useful in your research, please consider giving a star ⭐ and citing the following paper 📝.

@misc{fan2024instantsplat,
        title={InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds},
        author={Zhiwen Fan and Wenyan Cong and Kairun Wen and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }