Pre-training on ImageNet-1K

Installation

Please follow the installation instructions in DINOv2 and install timm==0.9.16 as well.

Dataset

We prepare ImageNet-1K following the instructions in DeiT.

Training

1. Specify the directory of datasets with data-path in the training script run_pretrain.sh.

2. Use the teacher-model and target_model parameters to select the appropriate teacher and student models.

3. Specify the model choices with model to choose from DINOv2, SynCLR, CLIP.

4. For SynCLR and CLIP training, use the teacher-path parameter to indicate the path to the pre-trained teacher model.

5. Simply run the training script as follows:

   bash run_pretrain.sh
   

Acknowledgment

This part is heavily build upon DeiT, DINOv2, SynCLR. We gratefully thank the authors for their wonderful works.