S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Paper | Project Page | Code

S³Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

Nan Huang*, Xiaobao Wei, Wenzhao Zheng$^\dagger$, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Shanghang Zhang$^\ddagger$

* Work done while interning at UC Berkeley $\dagger$ Project leader $\ddagger$ Corresponding author

S³Gaussian employs 3D Gaussians to model dynamic scenes for autonomous driving without other supervisions (e.g., 3D bounding boxes).

News

• [2023/5/31] Training & evaluation code release!
• [2024/5/31] Paper released on arXiv.

Demo

Overview

To tackle the challenges in self-supervised street scene decomposition, we propose a multi-resolution hexplane-based encoder to encode 4D grid into feature planes and a multi-head Gaussian decoder to decode them into deformed 4D Gaussians. We optimize the overall model without extra annotations in a self-supervised manner and achieve superior scene decomposition ability and rendering quality.

Results

Getting Started

Environmental Setups

Our code is developed on Ubuntu 22.04 using Python 3.9 and pytorch=1.13.1+cu116. We also tested on pytorch=2.2.1+cu118. We recommend using conda for the installation of dependencies.

git clone https://github.com/nnanhuang/S3Gaussian.git --recursive
cd S3Gaussian
conda create -n S3Gaussian python=3.9 
conda activate S3Gaussian

pip install -r requirements.txt
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn

Preparing Dataset

Follow detailed instructions in Prepare Dataset.

Training

For training first clip (eg. 0-50 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path 

If you want to try novel view synthesis, use

--configs "arguments/nvs.py"

For training next clip (eg. 51-100 frames), run

python train.py -s $data_dir --port 6017 --expname "waymo" --model_path $model_path 
--prior_checkpoint "$prior_dir/chkpnt_fine_50000.pth"

Also, you can load an existing checkpoint with:

python train.py -s $data_dir --port 6017 --expname "waymo" --start_checkpoint "$ckpt_dir/chkpnt_fine_30000.pth"

For more scripts examples, please check here.

Evaluation and Visualization

You can visualize and eval a checkpoints follow:

python train.py -s $data_dir --port 6017 --expname "waymo" --start_checkpoint "$ckpt_dir/chkpnt_fine_50000.pth"
--eval_only

Then you can get rendering RGB videos, ground truth RGB videos, depth videos, dynamic rgb videos and static rgb videos.

Acknowledgments

Credits to @Korace0v0 for building 3D Gaussians for street scenes. Many thanks!

Our code is based on 4D Gaussians and EmerNeRF.

Thanks to these excellent open-sourced repos!

Citation

If you find this project helpful, please consider citing the following paper:

@article{huang2024s3gaussian,
        title={S3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving},
        author={Huang, Nan and Wei, Xiaobao and Zheng, Wenzhao and An, Pengju and Lu, Ming and Zhan, Wei and Tomizuka,    Masayoshi and Keutzer, Kurt and Zhang, Shanghang},
        journal={arXiv preprint arXiv:2405.20323},
        year={2024}
      }