Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction
Iman Nematollahi, Oier Mees, Lukas Hermann, Wolfram Burgard
We present Hind4sight-Net, an unsupervised structured formulation that jointly learns a forward and an inverse dynamics model and decomposes the scene into salient object parts and predicts their 3D motion. Our object-centric formulation allows us to capture several desirable inductive biases that help in learning more efficient and interpretable models - a scene comprises of several objects, actions can affect these objects, and the objects can, in turn, affect each other. Thus, our network outputs action-conditioned 3D scene flow, object masks and 2D optical flow as emergent properties. Unlike previous approaches, our method does not require ground-truth point-wise data associations, typically provided by a tracker, or a pre-trained perception network and is learned solely from observational cues.
To begin, clone this repository locally
git clone https://github.com/nematoli/hind4sight.git
export HIND4SIGHT_ROOT=$(pwd)/hind4sight
Install requirements:
cd HIND4SIGHT_ROOT
conda create -n hind4sight_venv python=3.7
conda activate hind4sight_venv
sh install.shIf you want to train on the Freiburg Poking Dataset, choose a split with:
cd $HIND4SIGHT_ROOT/dataset
sh download_data.sh SIM | REALWe provide our final models for Freiburg Poking Dataset SIM and REAL splits.
cd $HIND4SIGHT_ROOT/checkpoints
sh download_model_weights.sh SIM | REALpython
SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks, (Byravan et al., 2017):
python
python
If you find the code useful, please cite:
Hind4sight-Net
@inproceedings{nematollahi20iros,
author = {Iman Nematollahi and Oier Mees and Lukas Hermann and Wolfram Burgard},
title={Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction},
booktitle={Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages={5319-5326},
year=2020,
url={http://ais.informatik.uni-freiburg.de/publications/papers/nematollahi20iros.pdf}
address = {Las Vegas, USA}
}MIT License