2D Gaussian Splatting with Nerf-Art Idea

2D Gaussian Splatting

Project page | Paper | Video | Surfel Rasterizer (CUDA) | Surfel Rasterizer (Python) | DTU+COLMAP (3.5GB) | SIBR Viewer Pre-built for Windows | Web Viewer

Nerf-Art

Project Page | Paper (ArXiv) | Paper (TVCG) | Download Models and Configs

This is a simple toy project that combines 2D Gaussian Splatting and Nerf-Art to creatively reconstruct 2D images into 3D space. It allows you to render high-resolution images quickly and apply artistic transformations in 3D space while preserving depth and different artistic styles.

Note: This is a toy project for experimentation and learning, not intended for production use.

Tech Stack

• 2D Gaussian Splatting: This technique breaks 2D images into small Gaussian splats, which are placed in 3D space to generate high-resolution renderings. It allows for fast and high-quality rendering.

• Nerf-Art: Using Neural Radiance Fields (NeRF), this method applies artistic style transformations to the images, allowing you to modify the atmosphere of the 3D scene and apply various artistic effects.

Features

• 2D Gaussian Splatting: Convert 2D images into multiple Gaussian splats and place them in 3D space to generate high-quality renders quickly.

• Nerf-Art Style Transfer: Apply artistic styles to 3D data. This allows you to transform the mood of the image by integrating an artistic style into the 3D space.

• Combined Rendering: Merge 2D Gaussian Splatting and Nerf-Art, enabling you to apply artistic style transformations while maintaining natural depth in 3D space.

Installation

---

Requirements

• Python 3.8
• CUDA (for GPU usage)

Installation Steps

1. Clone the repository:

git clone https://github.com/onetwohour/2d-gaussian-splatting-Art.git --recursive
cd 2d-gaussian-splatting-Art

2. Set up CUDA (for GPU usage):

# if use CUDA 11.7
pip install torch==2.0.0+cu117 torchaudio==2.0.0 torchvision==0.15.0 -f https://download.pytorch.org/whl/torch_stable.html

3. Install the dependencies:

pip install ninja
pip install submodules/simple-knn
pip install submodules/diff-surfel-rasterization
pip install ftfy regex tqdm
pip install git+https://github.com/openai/CLIP.git
pip install -r requirements.txt

4. Train

python train.py -s COLMAP_PATH --config configs/vangogh.yaml