FoAR: Force-Aware Reactive Policy for Contact-Rich Robotic Manipulation

[Paper] [Project Page]

Authors: Zihao He*, Hongjie Fang*, Jingjing Chen, Hao-Shu Fang, Cewu Lu

🛫 Getting Started

💻 Installation

We use RISE as our real robot baseline, Please following the installation guide to install the foar conda environments and the dependencies, as well as the real robot environments. Also, remember to adjust the constant parameters in dataset/constants.py and utils/constants.py according to your own environment.

📷 Calibration

Please calibrate the camera(s) with the robot before data collection and evaluation to ensure correct spatial transformations between camera(s) and the robot. Please refer to calibration guide for more details.

🛢️ Data Collection

We apply the data collection process in the RH20T paper. You may need to adjust dataset/realworld.py to accommodate different data formats. The sample data have the format of

collect_cups
|-- calib/
|   |-- [calib timestamp 1]/
|   |   |-- extrinsics.npy             # extrinsics (camera to marker)
|   |   |-- intrinsics.npy             # intrinsics
|   |   `-- tcp.npy                    # tcp pose of calibration
|   `-- [calib timestamp 2]/           # similar calib structure
`-- train/
    |-- [episode identifier 1]
    |   |-- metadata.json              # metadata
    |   |-- timestamp.txt              # calib timestamp
    |   |-- high_freq_data/            # high frequency data
    |   |   `-- force_torque_tcp_joint_timestamp.npy
    |   |                              # force/torque, tcp, joint, timestamp data
    |   |-- cam_[serial_number 1]/    
    |   |   |-- color                  # RGB
    |   |   |   |-- [timestamp 1].png
    |   |   |   |-- [timestamp 2].png
    |   |   |   |-- ...
    |   |   |   `-- [timestamp T].png
    |   |   |-- depth                  # depth
    |   |   |   |-- [timestamp 1].png
    |   |   |   |-- [timestamp 2].png
    |   |   |   |-- ...
    |   |   |   `-- [timestamp T].png
    |   |   |-- tcp                    # tcp
    |   |   |   |-- [timestamp 1].npy
    |   |   |   |-- [timestamp 2].npy
    |   |   |   |-- ...
    |   |   |   `-- [timestamp T].npy
    |   |   `-- gripper_command        # gripper command
    |   |       |-- [timestamp 1].npy
    |   |       |-- [timestamp 2].npy
    |   |       |-- ...
    |   |       `-- [timestamp T].npy
    |   `-- cam_[serial_number 2]/     # similar camera structure
    `-- [episode identifier 2]         # similar episode structure

🧑🏻‍💻 Training

The training scripts are saved in script.

conda activate foar
bash script/command_train.sh # Train Foar policy

🤖 Evaluation

Please follow the deployment guide to modify the evaluation script.

Modify the arguments in script/command_eval.sh, then

conda activate foar
bash script/command_eval.sh

✍️ Citation

@article{
  he2024force,
  title = {FoAR: Force-Aware Reactive Policy for Contact-Rich Robotic Manipulation},
  author = {He, Zihao and Fang, Hongjie and Chen, Jingjing and Fang, Hao-Shu and Lu, Cewu},
  journal = {arXiv preprint arXiv:2411.15753},
  year = {2024}
}