PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI

CVPR 2024 Highlight ⭐

Yandan Yang^✶, Baoxiong Jia^✶, Peiyuan Zhi, Siyuan Huang

For more information, please visit our project page.

⚙️ Installation & Dependencies

Check INSTALL.md for installation details.

🛒 Prepare Data

You can refer to DATA.md to download the original datasets and preprocess the data.

🚀 Diffusion Model for Scene Synthesis

Pretrained Weight

The pretrained weight of PhyScene can be downloaded here.

Evaluation

1. Generate scenes (save as json file) and test CKL & physical metrics.

sh run/test_livingroom.sh exp_dir

2. Load json file to generate images

sh run/test_livingroom_gen_image.sh exp_dir

3. Test SCA, KID, and FID

# SCA
python synthetic_vs_real_classifier.py --path_to_real_renderings data/preprocessed_data/LivingRoom/ --path_to_synthesized_renderings your/generated/image/folder
# KID and FID
python compute_fid_scores.py --path_to_real_renderings data/preprocessed_data/LivingRoom/ --path_to_synthesized_renderings your/generated/image/folder

🏡 Test Procthor Floor Plan

We also provide scripts for generating scenes from an unseen floor plan, such as room in ProTHOR. This script generate scene layout without reliance on any dataset, which provides a lite solution for user to apply on their own furniture dataset. See Procthor.md for more details.

⏱️ Modules

• Base Model
• Preprossed datasets
• Pretrained models
• Training scripts
• Tutorial.ipynb

🪧 Citation

If you find our work useful in your research, please consider citing:

@inproceedings{yang2024physcene,
          title={PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI},
          author={Yang, Yandan and Jia, Baoxiong and Zhi, Peiyuan and Huang, Siyuan},
          booktitle={Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)},
          year={2024}
        }

👋🏻 Acknowledgements

The code of this project is adapted from ATISS and DiffuScene, we sincerely thank the authors for open-sourcing their awesome projects. We also thank Ms. Zhen Chen from BIGAI for refining the figures, and all colleagues from the BIGAI TongVerse project for fruitful discussions and help on simulation developments.