Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation [AAAI2024]

[Paper Page]

Setup with Conda

# create conda env
conda create -n dir python=3.9
# install torch
pip install torch==1.10.0+cu113 torchvision==0.11.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# install other requirements
git clone --recursive https://github.com/ru1ven/KeypointFusion.git
cd KeypointFusion
pip install -r ./requirements.txt

Dataset preparation

Download the DexYCB dataset and the annotations.

Training & Evaluation

Download our pre-trained model on DexYCB s0.

python train.py

you would get the following output:

[mean_Error 6.927]
[PA_mean_Error 4.790]

Comparison on HO3D can be seen in here.

Running in the wild

We update a demo for running our method in real-world scenes.

The results of KeypointFusion on in-the-wild images.

BibTeX

@inproceedings{liu2024keypoint,
  title={Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation},
  author={Liu, Xingyu and Ren, Pengfei and Gao, Yuanyuan and Wang, Jingyu and Sun, Haifeng and Qi, Qi and Zhuang, Zirui and Liao, Jianxin},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={4},
  pages={3756--3764},
  year={2024}
}