Deformable Kernels [ICLR 2020]

Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
Hang Gao^*, Xizhou Zhu^*, Steve Lin, Jifeng Dai.
In ICLR, 2020.

This repository contains official implementation of deformable kernels.

Table of contents

1. Customized operators for deformable kernels, along with its variants.
   
2. Instructions to use our operators.
   
3. Results on ImageNet & COCO benchmarks, with pretrained models for reproduction.
   
4. Training and evaluation code.
   

(0) Getting started

PyTorch

• Get CUDA 10.1 installed on your machine.
• Install PyTorch (pytorch.org).
• conda env create -f environment.yml.

Apex

• Install Apex from its official repo. This will require CUDA 10.1 to work with the latest pytorch version (which is pytorch=1.3.1 as being tested against). It is used for fast mix-precision inference and should work out of the box.

Compile our operators

# assume at project root
(
cd deformable_kernels/ops/deform_kernel;
pip install -e .;
)

(1) Customized operators

This repo includes all deformable kernel variants described in our paper, namely:

• Global Deformable Kernels;
• Local Deformable Kernels;
• Local Deformable Kernels integrating with Deformable Convolutions;

Instead of learning offsets on image space, we propose to deform and resample on kernel space. This enables powerful dynamic inference capacity. For more technical details, please refer to their definitions.

We also provide implementations on our rivalries, namely:

• Deformable Convolutions;
• Soft Conditional Computation;

Please refer to their module definitions under deformable_kernels/modules folder.

(2) Quickstart

The following snippet constructs the deformable kernels we used for our experiments

from deformable_kernels.modules import (
    GlobalDeformKernel2d,
    DeformKernel2d,
    DeformKernelConv2d,
)

# global DK with scope size 2, kernel size 1, stride 1, padding 0, depthwise convolution.
gdk = GlobalDeformKernel2d((2, 2), [inplanes], [inplanes], groups=[inplanes])
# (local) DK with scope size 4, kernel size 3, stride 1, padding 1, depthwise convolution.
dk = DeformKernel2d((4, 4), [inplanes], [inplanes], 3, 1, 1, groups=[inplanes])
# (local) DK integrating with dcn, with kernel & image offsets separately learnt.
dkc = DeformKernelConv2d((4, 4), [inplanes], [inplanes], 3, 1, 1, groups=[inplanes]).

Note that all of our customized operators only support depthwise convolutions now, mainly because that efficiently resampling kernels at runtime is extremely slow if we orthogonally compute over each channel. We are trying to loose this requirement by iterating our CUDA implementation. Any contribuitions are welcome!

(3) Results & pretrained models

Under construction.

(4) Training & evaluation code

Under construction.

(A) License

This project is released under the MIT license.

(B) Citation & Contact

If you find this repo useful for your research, please consider citing this bibtex:

@article{gao2019deformable,
  title={Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation},
  author={Gao, Hang and Zhu, Xizhou and Lin, Steve and Dai, Jifeng},
  journal={arXiv preprint arXiv:1910.02940},
  year={2019}
}

Please contact Hang Gao <hangg AT eecs DOT berkeley DOT com> and Xizhou Zhu <ezra0408 AT mail.ustc DOT edu DOT cn> with any comments or feedback.