Skip to content

anji993/peract_colab

This branch is 4 commits ahead of peract/peract_colab:master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Mar 12, 2024
de0d422 · Mar 12, 2024

History

44 Commits
Mar 12, 2024
Aug 29, 2022
Sep 3, 2022
Sep 3, 2022
Sep 3, 2022
Mar 11, 2024
Mar 11, 2024
Apr 12, 2023

Repository files navigation

PerAct

Annotated Tutorial

Important: Before starting, change the runtime to GPU.

This notebook is an annotated tutorial on training Perceiver-Actor (PerAct) from scratch. We will look at training a single-task agent on the open_drawer task. The tutorial will start from loading calibrated RGB-D images, and end with visualizing action detections in voxelized observations. Overall, this guide is meant to complement the paper by providing concrete implementation details.

drawing

Credit

This notebook heavily builds on data-loading and pre-preprocessing code from ARM, YARR, PyRep, RLBench by Stephen James et al. The PerceiverIO code is adapted from perceiver-pytorch by Phil Wang (lucidrains). The optimizer is based on this LAMB implementation. See the corresponding licenses below.

Licenses

Citations

PerAct

@inproceedings{shridhar2022peract,
  title     = {Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation},
  author    = {Shridhar, Mohit and Manuelli, Lucas and Fox, Dieter},
  booktitle = {Proceedings of the 6th Conference on Robot Learning (CoRL)},
  year      = {2022},
}

C2FARM

@inproceedings{james2022coarse,
  title={Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation},
  author={James, Stephen and Wada, Kentaro and Laidlow, Tristan and Davison, Andrew J},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13739--13748},
  year={2022}
}

PerceiverIO

@article{jaegle2021perceiver,
  title={Perceiver io: A general architecture for structured inputs \& outputs},
  author={Jaegle, Andrew and Borgeaud, Sebastian and Alayrac, Jean-Baptiste and Doersch, Carl and Ionescu, Catalin and Ding, David and Koppula, Skanda and Zoran, Daniel and Brock, Andrew and Shelhamer, Evan and others},
  journal={arXiv preprint arXiv:2107.14795},
  year={2021}
}

RLBench

@article{james2020rlbench,
  title={Rlbench: The robot learning benchmark \& learning environment},
  author={James, Stephen and Ma, Zicong and Arrojo, David Rovick and Davison, Andrew J},
  journal={IEEE Robotics and Automation Letters},
  volume={5},
  number={2},
  pages={3019--3026},
  year={2020},
  publisher={IEEE}
}

About

Annotated Tutorial for PerAct

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%