dreamfields-3D

A colab friendly toolkit to generate 3D mesh model / video / nerf instance / multiview images of colourful 3D objects by text and image prompts input. Edited by Shengyu Meng (Simon)

Check the colab notebook for usage.

Dreamfields-3D is modified from dreamfields-torch and dreamfields, please check the Credits.md for details.

Preview.mp4

Example generated by text prompt: "a beautiful painting of a flower tree, by Chiho Aoshima, Long shot, surreal"

Main Contributions:

• Integrating Video generation.
• Export obj & ply model with vertex colour.
• Export 360° Video of final model.
• Visualizing the training progress and preview the output video in colab.
• Improve the generation quality.
  • Implements multiple CLIP model.
  • Improve the pre-process of the renderings before feeding into CLIP.
• Add more useful arguments.
• Organize the colab notebook.

Future update plan:

• Use different CLIP models simultaneously.
• Apply the image prompt only in certain assigned direction.
• Read existing mesh as NeRF instance then modify by text / image prompts.
• Reduce the usage of GPU RAM occupation in training.

I am a coding beginner, cannot promise for the progress, and welcome to contribute to this repository.

Compatibility:

• About system:
  • Colab: Pass on google Colab (tested on A100/v100/P100 GPU at 08/09/2022)
  • Ubuntu: The previous version (dreamfields-torch) has successfully ran on Ubuntu 18.04 with RTX 3090. Not tested for the dreamfields-3D yet, but mostly should be fined.
  • Windows: It should be work in windows with proper environment, but I failed to build the raymarching extension in different windows machine. More test will be required.
  • For locally running, please refer to the installation guide of dreamfields-torch bellow.
• About Exporting 3D model:
  • The 3D models will be exported as obj and ply format with vertex colour.
  • Mesh with vertex colour could be seen in meshLab and Rhino3D.
  • For viewing colour in Blender, please import the ply model frist, then create a new material, and plug a Color Attribute node into the base color in shader editor, then you should see the vertex colour.
• About GUI:
  • GUI is supported only when running locally. However, some new features maybe not available in GUI model. I would recommend to train without GUI first, then run test on the pretrained model in GUI, to browse the final 3D NeRF instance.

👇 Bellow readme from the dreamfields-torch repository.

---

dreamfields-torch (WIP)

A pytorch implementation of dreamfields as described in Zero-Shot Text-Guided Object Generation with Dream Fields.

An example of a generated neural field by prompt "cthulhu" viewed in real-time:

cthulhu.mp4

Install

The code framework is based on torch-ngp.

git clone https://github.com/ashawkey/dreamfields-torch.git
cd dreamfields-torch

Install with pip

pip install -r requirements.txt

install customized verion of pymarchingcubes

bash scripts/install_PyMarchingCubes.sh

Build extension (optional)

By default, we use load to build the extension at runtime. However, this may be inconvenient sometimes. Therefore, we also provide the setup.py to build each extension:

# install all extension modules
bash scripts/install_ext.sh
# if you want to install manually, here is an example:
cd raymarching
python setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)
pip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)

Tested environments

• Ubuntu 20 with torch 1.10 & CUDA 11.3 on a TITAN RTX.
• Windows 10 with torch 1.11 & CUDA 11.3 on a RTX 3070.

Currently, --ff only supports GPUs with CUDA architecture >= 70. For GPUs with lower architecture, --tcnn can still be used, but the speed will be slower compared to more recent GPUs.

Usage

First time running will take some time to compile the CUDA extensions.

# text-guided generation
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16

# use the GUI
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16 --gui

# [experimental] image-guided generation (also use the CLIP loss)
python main_nerf.py --image /path/to/image --workspace trial --cuda_ray --fp16

check the scripts directory for more provided examples.

Difference from the original implementation

• Mip-nerf is not implemented, currently only the original nerf is supported.
• Sampling poses with an elevation range in [-30, 30] degrees, instead of fixed at 30 degree.
• Use the origin loss.

Update Logs

• 5.18: major update.
• 3.16: basic reproduction.

Acknowledgement

• The great paper and official JAX implementation of dreamfields:

  @article{jain2021dreamfields,
      author = {Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben},
      title = {Zero-Shot Text-Guided Object Generation with Dream Fields},
      journal = {arXiv},
      month = {December},
      year = {2021},
  }   
  

• The GUI is developed with DearPyGui.