SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

Jiehong Lin, Lihua Liu, Dekun Lu, Kui Jia

CVPR 2024

[Paper]

News

• [2024/03/07] We publish an updated version of our paper on ArXiv.
• [2024/02/29] Our paper is accepted by CVPR2024!

Update Log

• [2024/03/05] We update the demo to support FastSAM, you can do this by specifying SEGMENTOR_MODEL=fastsam in demo.sh.
• [2024/03/03] We upload a docker image for running custom data.
• [2024/03/01] We update the released model of PEM. For the new model, a larger batchsize of 32 is set, while that of the old is 12.

Overview

In this work, we employ Segment Anything Model as an advanced starting point for zero-shot 6D object pose estimation from RGB-D images, and propose a novel framework, named SAM-6D, which utilizes the following two dedicated sub-networks to realize the focused task:

• Instance Segmentation Model
• Pose Estimation Model

Getting Started

1. Preparation

Please clone the repository locally:

git clone https://github.com/JiehongLin/SAM-6D.git

Install the environment and download the model checkpoints:

conda create -n sam6d python=3.9.19
conda activate sam6d
# install pytorch according to your cuda version
pip install torch==2.1.1 torchvision==0.16.1 --index-url https://download.pytorch.org/whl/cu121
pip install xformers==0.0.23

We also provide a docker image for convenience.

2. Evaluation on the custom data

# run inference
cd SAM-6D
sh demo.sh

Citation

If you find our work useful in your research, please consider citing:

@article{lin2023sam,
title={SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation},
author={Lin, Jiehong and Liu, Lihua and Lu, Dekun and Jia, Kui},
journal={arXiv preprint arXiv:2311.15707},
year={2023}
}

Contact

If you have any questions, please feel free to contact the authors.

Jiehong Lin: [email protected]

Lihua Liu: [email protected]

Dekun Lu: [email protected]

Kui Jia: [email protected]