init commit

.gitignore

@@ -0,0 +1,188 @@
+work_dir*/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+
+# custom
+# logs
+data/
+local_scripts/
+package_hacks/
+
+# caches 
+*.pyc
+*.swp
+
+# media
+*.mp4
+*.png
+*.jpg
+
+
+# wandb
+wandb/
+
+# work in progress
+*wip*
+
+*results*
+*debug*

.vscode/settings.json

@@ -0,0 +1,20 @@
+{
+    "python.formatting.provider": "black",
+    "[python]": {
+        "editor.defaultFormatter": "ms-python.black-formatter",
+        "editor.codeActionsOnSave": {
+            "source.organizeImports": true
+        }
+    },
+    "python.analysis.typeCheckingMode": "off",
+    "files.exclude": {
+        "**/__pycache__": true
+    },
+    "editor.formatOnSave": true,
+    "python.linting.enabled": false,
+    "files.watcherExclude": {
+        "data/*": true,
+        "work_dirs/*": true
+    },
+    "terminal.integrated.scrollback": 5000
+}

README.md

@@ -1 +1,170 @@
-# EmerNeRF
+# EmerNeRF
+
+PyTorch implementation of:
+> [**EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision**](https://arxiv.org/abs/tbd),  
+> [Jiawei Yang](https://jiawei-yang.github.io), [Boris Ivanovic](https://www.borisivanovic.com/), [Or Litany](https://orlitany.github.io/), [Xinshuo Weng](https://www.xinshuoweng.com/), [Seung Wook Kim](https://seung-kim.github.io/seungkim/), [Boyi Li](https://sites.google.com/site/boyilics/home), [Tong Che](), [Danfei Xu](https://faculty.cc.gatech.edu/~danfei/), [Sanja Fidler](https://www.cs.utoronto.ca/~fidler/), [Marco Pavone](https://web.stanford.edu/~pavone/), [Yue Wang](https://yuewang.xyz/)
+
+#### [Project page](https://emernerf.github.io) | [Paper](https://arxiv.org/abs/2311.02077)
+
+<p align="center">
+  <img src="assets/overview.png" alt="EmerNeRF overview">
+</p>
+
+## News
+
+- [2023/11/05] We've released an initial version of EmerNeRF, which supports the NeRF On-The-Road (NOTR) dataset sourced from the Waymo Open Dataset. NuScenes support is also available.
+
+## Table of Contents
+
+- [Introduction](#introduction)
+- [Installation](#installation)
+- [Dataset preparation](#dataset-preparation)
+- [Run the code](#run-the-code)
+  - [Configs](#configs)
+  - [Training](#training)
+  - [Voxel Inspection](#voxel-inspection)
+- [Pretrained models](#pretrained-models)
+- [Citation](#citation)
+- [FAQ](docs/FAQ.md)
+
+## Introduction
+
+We introduce EmerNeRF, a self-supervised approach that utilizes neural fields for spatial-temporal decomposition, motion estimation, and the lifting of foundation features. EmerNeRF can decompose a scene into dynamic objects and a static background and estimate their motion in a self-supervised way. Enriched with lifted and "denoised" 2D features in 4D space-time, EmerNeRF unveils new potentials for scene understanding. Additionally, we release the NeRF On-The-Road (NOTR) dataset split to support future research.
+
+## Installation
+
+Our code is developed on Ubuntu 22.04 using Python 3.9 and PyTorch 2.0. Please note that the code has only been tested with these specified versions. We recommend using conda for the installation of dependencies. The installation process might take more than 30 minutes.
+
+1. Create the `emernerf` conda environment and install all dependencies:
+
+```shell
+conda create -n emernerf python=3.9 -y
+conda activate emernerf
+# this will take a while: more than 10 minutes
+pip install -r requirements.txt
+```
+
+2. Install `nerfacc` and `tiny-cuda-nn` manually:
+
+```shell
+pip install git+https://github.com/KAIR-BAIR/nerfacc.git
+pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
+```
+
+**Troubleshooting**:
+
+<details>
+  <summary><strong>nvcc fatal : Unsupported gpu architecture 'compute_89` </strong></summary>
+
+If you encounter the error `nvcc fatal : Unsupported gpu architecture 'compute_89`, try the following command:
+
+```
+TCNN_CUDA_ARCHITECTURES=86 pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
+```
+
+</details>
+
+<details>
+  <summary><strong>error: parameter packs not expanded with ‘...’</strong></summary>
+
+If you encounter this error:
+
+ ```
+ error: parameter packs not expanded with ‘...’
+ ```
+
+Refer to [this solution](<https://github.com/NVlabs/instant-ngp/issues/119#issuecomment-1034701258>) on GitHub.
+
+</details>
+
+## Dataset preparation
+
+- **NOTR (Waymo) Dataset**: See [NOTR Dataset Preparation](docs/NOTR.md) for detailed instructions on preparing the NOTR dataset.
+
+- **NuScenes Dataset**: For preparing the NuScenes dataset and viewing example sample results, refer to [NuScenes Dataset Preparation](docs/NUSCENES.md).
+
+For those interested in setting up a custom dataset, kindly use these two datasets as templates. Also take a look at the `datasets/base directory` to familiarize yourself with the dataset preparation process in our codebase.
+
+## Run the code
+
+### Configs
+
+We have provided detailed comments in `configs/default_config.yaml` for each configuration. Alongside the released code, you'll also find these detailed comments.
+
+### Training
+
+Sample training scripts can be found in `sample_scripts/`
+
+1. **Data inspection**. Before initiating the training, you might want to inspect the data. We've included a script for visualizing the data. To visualize the NOTR dataset, execute the following:
+
+```shell
+# Adjust hyper-parameters as needed
+python train_emernerf.py \
+    --config_file configs/default_config.yaml \
+    --output_root $output_root \
+    --project $project \
+    --run_name ${scene_idx} \
+    --render_data_video_only \  # This will render a video of the data.
+    data.scene_idx=$scene_idx \
+    data.pixel_source.load_size=[160,240] \ # Downsample to enhance the visibility of LiDAR points.
+    data.pixel_source.num_cams=3 \ # Opt for 1, 3, or 5
+    data.start_timestep=0 \
+    data.end_timestep=-1 
+
+```
+
+This script produces a video similar to the one below, showcasing LiDAR points colored by their range values and the 3D scene flows, and their feature maps (if `load_features=True`):
+
+<p align="center">
+  <img src="assets/notr_data.jpg">
+</p>
+
+<p align="center">
+  <img src="assets/nuscenes_data.jpg">
+</p>
+
+2. **Training**. For the most comprehensive EmerNeRF training (incorporating dynamic encoder, flow encoder, and feature lifting), use:
+
+```shell
+  python train_emernerf.py \
+    --config_file configs/default_flow.yaml \
+    --output_root $output_root \
+    --project $project \
+    --run_name ${scene_idx}_flow \
+    data.scene_idx=$scene_idx \
+    data.start_timestep=$start_timestep \
+    data.end_timestep=$end_timestep \
+    data.pixel_source.load_features=True \
+    data.pixel_source.feature_model_type=dinov2_vitb14 \
+    nerf.model.head.enable_feature_head=True \
+    nerf.model.head.enable_learnable_pe=True \
+    logging.saveckpt_freq=$num_iters \
+    optim.num_iters=$num_iters
+```
+
+For more examples, refer to the `sample_scripts/` folder.
+
+3. **Voxel Inspection**. We've provided visualization code to display spatial-temporal features as shown on our homepage. To visualize voxel features, simply add `--visualize_voxel` and specify `resume_from=$YOUR_PRETRAINED_MODEL`. This will produce an HTML file which you can open in a browser for voxel feature visualization:
+
+<p align="center">
+  <img src="assets/sample_voxel.jpg" alt="Example Voxel Visualization">
+</p>
+
+## Pretrained models
+
+- [ ] We will release all pretrained models soon. Do note that the distribution of pretrained models will be in accordance with Waymo's data sharing policy. Hence, we will only release pretrained models for registered users upon request. More details will be provided soon.
+
+## Citation
+
+Consider citing our paper if you find this code or our paper is useful for your research:
+
+```bibtex
+@misc{yang2023emernerf,
+      title={EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision},
+      author={Jiawei Yang and Boris Ivanovic and Or Litany and Xinshuo Weng and Seung Wook Kim and Boyi Li and Tong Che and Danfei Xu and Sanja Fidler and Marco Pavone and Yue Wang},
+      year={2023},
+      eprint={2311.02077},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```