Iterative Residual Refinement
for Joint Optical Flow and Occlusion Estimation

This repository is the PyTorch implementation of the paper:

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (CVPR 2019)
Junhwa Hur and Stefan Roth
Department of Computer Science, TU Darmstadt
[Preprint]   [Proceeding]   [Supplemental]

Please cite the paper below if you find our paper and source codes are useful.

@inproceedings{Hur:2019:IRR,  
  Author = {Junhwa Hur and Stefan Roth},  
  Booktitle = {CVPR},  
  Title = {Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation},  
  Year = {2019}  
}

Contact: junhwa.hur[at]visinf.tu-darmstadt.de

Getting started

This code has been developed under Anaconda(Python 3.6), Pytorch 0.4.1 and CUDA 8.0 on Ubuntu 16.04.

1. Please install the followings:

   • Anaconda (Python 3.6)
   • PyTorch 0.4.1 (Linux, Conda, Python 3.6, CUDA 8.0)
     • For PyTorch 1.1, Python 3.7, CUDA >= 9.0, the correlation module needs to be installed accordingly:
       • Please move all files in models/correlation_package_cu9 to models/correlation_package (overwritting)
       • Depending on your system, configure -gencode, -ccbin, cuda-path in models/correlation_package/setup.py accordingly
   • tqdm (conda install -c conda-forge tqdm)

2. Then, install the correlation package:

   ./install.sh
   

3. The datasets used for this projects are followings:

   • FlyingChairsOcc dataset
   • FlyingThings3D subset
   • MPI Sintel Dataset + revised occlusion GT
   • KITTI Optical Flow 2015 and KITTI Optical Flow 2012

Training

The scripts folder contains training scripts of experiments demonstrated in the paper.
To train the model, you can simply run the script file, e.g., ./IRR-PWC_flyingChairsOcc.sh.
In script files, please configure your own experiment directory (EXPERIMENTS_HOME) and dataset directory in your local system (e.g., SINTEL_HOME or KITTI_HOME).

Pretrained Models

The saved_check_point contains the pretrained models of i) baseline, ii) baseline + irr, and iii) full models.
Additional pretrained models in the ablations study (Table 1 in the main paper) and their training scripts are available upon request.

Inference

The scripts for testing the pre-trained models are located in scripts/validation.

Acknowledgement

Portions of the source code (e.g., training pipeline, runtime, argument parser, and logger) are from Jochen Gast