Skip to content
/ cgf Public
forked from jianglongye/cgf

Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations, RA-L 2023 & IROS 2023

Notifications You must be signed in to change notification settings

Smilels/cgf

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Continuous Grasping Function

This repository is the official implementation of the paper:

Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations
Jianglong Ye*, Jiashun Wang*, Binghao Huang, Yuzhe Qin, Xiaolong Wang
RA-L 2023, IROS 2023

Project Page / ArXiv / Video

Environment Setup

(Our code has been tested on Ubuntu 20.04, python 3.10, torch 1.13.1, CUDA 11.7 and RTX 3090)

To set up the environment, follow these steps:

conda create -n cgf python=3.10 -y && conda activate cgf
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia -y

pip install six numpy==1.23.1 tqdm pyyaml scipy opencv-python trimesh einops lxml transforms3d fvcore viser==0.1.34
pip install git+https://github.com/hassony2/chumpy.git
pip install git+https://github.com/hassony2/manopth
pip install git+https://github.com/facebookresearch/[email protected]
# installing pytorch3d from source is usually more robust, for pytorch 1.13.1, CUDA 11.7, python 3.10, the following command *may* work:
# pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py310_cu117_pyt1131/download.html

pip install git+https://github.com/UM-ARM-Lab/[email protected]

Data Preparation

We provide processed data for training CGF which can be downloaded from here and unzip it under PROJECT_ROOT/data/. The final file structure should look like this:

PROJECT_ROOT
├── ...
└── data
    ├── ...
    ├── geometry
    ├── mano_v1_2_models
    ├── original_root
    └── processed
        ├── ...
        ├── meta.json
        ├── pose_m_aug.npz
        └── pose_m_object_coord.npz

Alternatively, you can prepare the data by following the instructions below.

  1. Download DexYCB data from the official website to DEXYCB_DATA_ROOT and organize the data as follows:
    DEXYCB_DATA_ROOT
    ├── 20200709-subject-01/
    ├── 20200813-subject-02/
    ├── ...
    ├── calibration/
    └── models/
    
  2. Run the following script to process the data:
    python scripts/prepare_dexycb.py --data_root DEXYCB_DATA_ROOT
    python scripts/retargeting.py --mano_side left
    python scripts/retargeting.py --mano_side right
    The processed data will be saved to PROJECT_ROOT/data/processed. Retargeting results can be visualized by running the following command:
    python scripts/visualize_retargeting.py
    retargeting

Running

Training

To train the CGF model, run the following command:

# cd PROJECT_ROOT
python scripts/main.py

Sampling

To sample from the trained CGF model, run the following command (replace TIMESTAMP with the timestamp of the checkpoint in the PROJECT_ROOT/output directory):

We provide a pre-trained model which can be downloaded from here and unzip it under PROJECT_ROOT/output/.

# cd PROJECT_ROOT
python scripts/main.py --mode sample --ts TIMESTAMP

The sampled results will be saved to PROJECT_ROOT/output/TIMESTAMP/sample. They can be visualized by running the following command:

python scripts/visualize_sampling.py --ts TIMESTAMP

sampling

To evaluate the smoothness of the sampled grasps (table 1 in the paper), run the following command:

python scripts/eval_generation.py --ts TIMESTAMP

Note that the evaluation results may vary slightly due to the randomness in the sampling process.

Simulation

Our simulation environment is adapted from dex-hand-teleop. To setup the simulation environment, follow these steps:

pip install sapien==2.2.2 gym open3d

To run the simulation, run the following command (replace SAPIEN_SAMPLE_DATA_PATH with the path to a sample data in the PROJECT_ROOT/output/TIMESTAMP/sample/result_filter_sapien directory):

python scripts/sim.py --data_path SAPIEN_SAMPLE_DATA_PATH

Citation

@article{ye2023learning,
  title={Learning continuous grasping function with a dexterous hand from human demonstrations},
  author={Ye, Jianglong and Wang, Jiashun and Huang, Binghao and Qin, Yuzhe and Wang, Xiaolong},
  journal={IEEE Robotics and Automation Letters},
  year={2023}
}

About

Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations, RA-L 2023 & IROS 2023

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%