DFAD

This repository contains the code for paper: Data-Free Adversarial Distillation

Requirements

pip install -r requirements.txt 

Quick Start: MNIST

We provide an MNIST example for DFAD, which only takes a few minutes for training. Data will be automatically downloaded by the python scripts.

bash run_mnist.sh

or

# Train the teacher model
python train_teacher.py --batch_size 256 --epochs 10 --lr 0.01 --dataset mnist --model lenet5 --weight_decay 1e-4 # --verbose

# Train the student model
python DFAD_mnist.py --ckpt checkpoint/teacher/mnist-lenet5.pt # --verbose

Step by Step

0. Download Pretrained Models (optional)

You can download our pretrained models from OneDrive and extract the .pt files to ./checkpoint/teacher/.

1. Prepare Datasets

Data for MNIST, CIFAR10 and CIFAR100 will be automatically downloaded by the training scripts.
Download other datasets from the following links and extract them to ./data:

Caltech101

1. Download Caltech101 and extract it to ./data/caltech101
2. Split datasets

   cd data
   python split_caltech101.py

CamVid

1. Download CamVid and extract it to ./data/CamVid

NYUv2

1. Download NYUv2 and extract it to ./data/NYUv2
2. Download labels and extract them to ./data/NYUv2/nyuv2-meta-data

2. Train teachers and students

Start the visdom server on port 15550 for visualization. You can visit 127.0.0.1:15550 to check training logs.

visdom -p 15550

CIFAR

• CIFAR10

# Teacher
python train_teacher.py --dataset cifar10 --batch_size 128 --step_size 80 --epochs 200 --model resnet34_8x

# Student
python DFAD_cifar.py --dataset cifar10 --ckpt checkpoint/teacher/cifar10-resnet34_8x.pt --scheduler

• CIFAR100

# Teacher
python train_teacher.py --dataset cifar100 --batch_size 128 --step_size 80 --epochs 200 --model resnet34_8x

# Student
python DFAD_cifar.py --dataset cifar100 --ckpt checkpoint/teacher/cifar100-resnet34_8x.pt --scheduler

Caltech101

# Teacher 
python train_teacher.py --dataset caltech101 --batch_size 128 --num_classes 101 --step_size 50 --epochs 150 --model resnet34

# Student
python DFAD_caltech101.py --lr_S 0.05 --lr_G 1e-3 --scheduler --batch_size 64 --ckpt checkpoint/teacher/caltech101-resnet34.pt

CamVid

# Teacher
python train_teacher_seg.py --model deeplabv3_resnet50 --dataset camvid --data_root ./data/CamVid --scheduler --lr 0.1 --num_classes 11

# Student
python DFAD_camvid_deeplab.py --ckpt checkpoint/teacher/camvid-deeplabv3_resnet50.pt --data_root ./data/CamVid --scheduler

NYUv2

# Teacher
python train_teacher_seg.py --model deeplabv3_resnet50 --dataset nyuv2 --data_root ./data/NYUv2 --scheduler --lr 0.05 --num_classes 13

# Student
python DFAD_nyu_deeplab.py --ckpt checkpoint/teacher/nyuv2-deeplabv3_resnet50.pt --data_root ./data/NYUv2 --scheduler