Monocular, One-stage, Regression of Multiple 3D People

ROMP, accepted by ICCV 2021, is a concise one-stage network for multi-person 3D mesh recovery from a single image.

• Simple. Concise one-stage framework for simultaneous person detection and 3D body mesh recovery.

• Fast. ROMP can achieve real-time inference on a 1070Ti GPU.

• Strong. ROMP achieves superior performance on multiple challenging multi-person/occlusion benchmarks.

• Easy to use. We provide user cross-platform API and webcam demos to run on Linux / Windows / Mac.

Contact: [email protected]. Feel free to contact me for related questions or discussions! arXiv paper.

Table of contents

• Table of contents
• Features
• News
• Getting started
  • Fast try
  • Try on Google Colab
  • Installation
• Inference
  • Processing images
  • Processing videos
  • Webcam
  • Export
    • Export to Blender FBX
    • Blender Addons
  • Train
  • Evaluation
  • Bugs report
• Citation
• Contributor
• Acknowledgement

Features

features

News

2022/03/18: Simple version of ROMP for all platform. Let's pip install simple-romp. See the guidance for details
Old logs

Getting started

Fast try

pip install simple-romp

For details, please refer to the guidance.

Try on Google Colab

It allows you to run the project in the cloud, free of charge. Let's give the prepared Google Colab demo a try.

Installation

Please refer to install.md for installation.

Inference

Currently, we support processing images, video or real-time webcam.
Pelease refer to config_guide.md for configurations.
ROMP can be called as a python lib inside the python code, jupyter notebook, or from command line / scripts, please refer to Google Colab demo for examples.

Processing images

To re-implement the demo results, please run

cd ROMP
# on Linux
python -m romp.predict.image --inputs=demo/images --output_dir=demo/image_results
# change the `inputs` in configs/image.yml to /path/to/your/image folder, then run 
sh scripts/image.sh

# on Windows
python -m romp.predict.image --inputs=demo\images --output_dir=demo\image_results

Please refer to config_guide.md for saving the estimated mesh/Center maps/parameters dict.

For interactive visualization, please run

# on Linux
python -m romp.predict.image --inputs=demo/images --output_dir=demo/image_results --show_mesh_stand_on_image  --interactive_vis
# on Windows
python -m romp.predict.image --inputs=demo\images --output_dir=demo\image_results --show_mesh_stand_on_image  --interactive_vis

Caution: To use show_mesh_stand_on_image and interactive_vis, you must run ROMP on a computer with visual desktop to support the rendering. Most remote servers without visual desktop is not supported.

Processing videos

cd ROMP
# on Linux
python -m romp.predict.video --inputs=demo/videos/sample_video.mp4 --output_dir=demo/sample_video_results --save_dict_results
# or you can set all configurations in configs/video.yml, then run 
sh scripts/video.sh

# on Windows
python -m romp.predict.video --inputs=demo\videos\sample_video.mp4 --output_dir=demo\sample_video_results --save_dict_results

We notice that some users only want to extract the motion of the formost person, like this

To achieve this, please run

# on Linux
python -m romp.predict.video --inputs=demo/videos/demo_video_frames --output_dir=demo/demo_video_fp_results --show_largest_person_only --save_dict_results --show_mesh_stand_on_image 

All functions can be combined or work individually. Welcome to try them.

Webcam

To do this you just need to run:

cd ROMP
# on Linux
sh scripts/webcam.sh

# on Windows
python -u -m romp.predict.webcam --configs_yml='configs\webcam.yml'

To drive a character in Blender, please refer to expert.md.

Export

Export to Blender FBX

Please refer to expert.md to export the results to fbx files for Blender usage. Currently, this function only support the single-person video cases. Therefore, please test it with demo/videos/sample_video2_results/sample_video2.mp4, whose results would be saved to demo/videos/sample_video2_results.

Blender Addons

Chuanhang Yan : developing an addon for driving character in Blender.
VLT Media creates a QuickMocap-BlenderAddon to read the .npz file created by ROMP. Clean & smooth the resulting keyframes.

Train

Please prepare the training datasets following dataset.md, and then refer to train.md for training.

Evaluation

Please refer to evaluation.md for evaluation on benchmarks.

Bugs report

Please refer to bug.md for solutions. Welcome to submit the issues for related bugs. I will solve them as soon as possible.

Citation

@InProceedings{ROMP,
author = {Sun, Yu and Bao, Qian and Liu, Wu and Fu, Yili and Michael J., Black and Mei, Tao},
title = {Monocular, One-stage, Regression of Multiple 3D People},
booktitle = {ICCV},
month = {October},
year = {2021}
}

Contributor

This repository is currently maintained by Yu Sun.

ROMP has also benefited from many developers, including

• Marco Musy : help in the textured SMPL visualization.
• Gavin Gray : adding support for an elegant context manager to run code in a notebook.
• VLT Media and Vivien Richter : adding support for running on Windows & batch_videos.py.
• Chuanhang Yan : developing an addon for driving character in Blender.
• Tian Jin: help in simplified smpl and fast rendering (realrender).
• ZhengdiYu : helpful discussion on optimizing the implementation details.

Acknowledgement

We thank Peng Cheng for his constructive comments on Center map training.

Here are some great resources we benefit:

• SMPL models and layer is borrowed from MPII SMPL-X model.
• Some functions are borrowed from HMR-pytorch and SPIN.
• The evaluation code and GT annotations of 3DPW dataset is brought from 3dpw-eval and VIBE.
• 3D mesh visualization is supported by vedo, EasyMocap, minimal-hand, Open3D, and Pyrender.

Please consider citing their papers.