Towards Markerless Surgical Tool and Hand Pose Estimation: PVNet Baseline

• Project page
• Synthetic Grasp Generation
• Synthetic Grasp Rendering
• Real Dataset Generation
• HandObjectNet Baseline
• PVNet Baseline
• Combined Model Baseline

The structure of this project is described in project_structure.md.

Table of Content

• Setup
• Demo
• Training
• Evaluation
• Visualization
• Citations

Setup

Download and install prerequisites

sudo apt-get install libglfw3-dev libglfw3

Download and Install Code

git https://github.com/jonashein/pvnet_baseline.git
cd pvnet_baseline
conda env create --file=environment.yml
conda activate pvnet

Compile cuda extension for RANSAC voting under lib/csrc/ransac_voting:

cd lib/csrc/ransac_voting/
python setup.py build_ext --inplace
cd ../../../

Download Synthetic Dataset

Download the synthetic dataset from the project page, or use the commands below:

cd data/
wget http://medicalaugmentedreality.org/datasets/syn_colibri_v1.zip
unzip -x syn_colibri_v1.zip
cd ../

Convert the dataset into the format expected by PVNet:

python3 pvnet_custom_dataset.py -m assets/drill_segmentation_textured_final.ply -d data/syn_colibri_v1/train.txt -o data/ -n syn_colibri_v1_train
python3 pvnet_custom_dataset.py -m assets/drill_segmentation_textured_final.ply -d data/syn_colibri_v1/val.txt -o data/ -n syn_colibri_v1_val
python3 pvnet_custom_dataset.py -m assets/drill_segmentation_textured_final.ply -d data/syn_colibri_v1/test.txt -o data/ -n syn_colibri_v1_test

Download Real Dataset

Download the real dataset from the project page, or use the commands below:

cd data/
wget http://medicalaugmentedreality.org/datasets/real_colibri_v1.zip
unzip -x real_colibri_v1.zip
cd ../

Convert the dataset into the format expected by PVNet:

python3 pvnet_custom_dataset.py -m assets/drill_segmentation_textured_final.ply -d data/real_colibri_v1/train.txt -o data/ -n real_colibri_v1_train
python3 pvnet_custom_dataset.py -m assets/drill_segmentation_textured_final.ply -d data/real_colibri_v1/val.txt -o data/ -n real_colibri_v1_val
python3 pvnet_custom_dataset.py -m assets/drill_segmentation_textured_final.ply -d data/real_colibri_v1/test.txt -o data/ -n real_colibri_v1_test

Demo

We provide pretrained models, which can be downloaded here. Unzip the downloaded archive, and copy the model directory of one of the checkpoints to the data directory.

Your directory structure should look like this:

pvnet_baseline/data/model/pvnet/custom/*.pth

Then run the commands listed below to evaluate the model.

Training

Pretrain a model on the synthetic dataset:

python train_net.py --cfg_file configs/syn_colibri_v1_train.yaml

Refine a model on the real dataset:

python train_net.py --cfg_file configs/real_colibri_v1_train.yaml

The training checkpoints and monitoring data will be stored at data/model/ and data/record/ respectively.

Losses and validation metrics can are monitored on tensorboard:

tensorboard --logdir data/record/pvnet

Evaluation

Evaluate a pretrained model on the synthetic dataset:

python train_net.py --test --cfg_file configs/syn_colibri_v1_test.yaml

Evaluate a refined model on the real dataset:

python train_net.py --test --cfg_file configs/real_colibri_v1_test.yaml

After evaluating a model, the test set metrics can be computed by running:

python3 compute_metrics.py -m "data/record/metrics.pkl"

Visualization

To visualize the keypoint estimates and render 3D views of the tool pose estimates, run:

python run.py --type visualize --test --cfg_file configs/real_colibri_v1_test.yaml --vis_out visualizations/

Citations

If you find this code useful for your research, please consider citing:

• the publication that this code was adapted for

@article{hein2021towards,
  title={Towards markerless surgical tool and hand pose estimation},
  author={Hein, Jonas and Seibold, Matthias and Bogo, Federica and Farshad, Mazda and Pollefeys, Marc and F{\"u}rnstahl, Philipp and Navab, Nassir},
  journal={International Journal of Computer Assisted Radiology and Surgery},
  volume={16},
  number={5},
  pages={799--808},
  year={2021},
  publisher={Springer}
}

• the publication it builds upon and that this code was originally developed for

@inproceedings{peng2019pvnet,
  title={PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation},
  author={Peng, Sida and Liu, Yuan and Huang, Qixing and Zhou, Xiaowei and Bao, Hujun},
  booktitle={CVPR},
  year={2019}
}