OpenOOD: Benchmarking Generalized OOD Detection

       

This repository reproduces representative methods within the Generalized Out-of-Distribution Detection Framework, aiming to make a fair comparison across methods that initially developed for anomaly detection, novelty detection, open set recognition, and out-of-distribution detection. This codebase is still under construction. Comments, issues, contributions, and collaborations are all welcomed!

Image from Fig.3 in our survey - Timeline for representative methodologies.

Updates

• 19 May, 2022: We release v0.5 to finalize all implemented methods.
• 12 April, 2022: Primary release to support Full-Spectrum OOD Detection.

Get Started

To setup the environment, we use conda to manage our dependencies.

Our developers use CUDA 10.1 to do experiments.

You can specify the appropriate cudatoolkit version to install on your machine in the environment.yml file, and then run the following to create the conda environment:

conda env create -f environment.yml
conda activate openood

Datasets are provided here. Our codebase accesses the datasets from ./data/ by default.

├── ...
├── data
│   ├── images
│   ├── covid_images
│   └── imglist
├── openood
├── scripts
├── main.py
├── ...

The easiest hands-on script is to train LeNet-5 on MNIST and evaluate its OOD or FS-OOD performance with MSP baseline.

sh scripts/0_basics/mnist_train.sh
sh scripts/c_ood/0_mnist_test_ood_msp.sh
sh scripts/c_ood/0_mnist_test_fsood_msp.sh

More tutorials are provided in our wiki pages.

---

Supported Benchmarks (8)

This part lists all the benchmarks we

Anomaly Detection (1)

• MVTec-AD

Open Set Recognition (3)

• CIFAR-4/6
• CIFAR-40/60
• TinyImageNet-20/180

Out-of-Distribution Detection (4)

• MNIST

  Near-OOD: NotMNIST, FashionMNIST;
  Far-OOD: Texture, CIFAR-10, TinyImageNet, Places-365;
  CS-ID: SVHN, USPS;

• CIFAR-10

  Near-OOD: CIFAR-100, TinyImageNet;
  Far-OOD: MNIST, FashionMNIST, Texture, CIFAR-100-C;
  CS-ID: CINIC-10, CIFAR-10-C;

• CIFAR-100

  Near-OOD: CIFAR-10, CINIC-10, TinyImageNet;
  Far-OOD: MNIST, FashionMNIST, Texture, CIFAR-10-C;
  CS-ID: TinyImageNet-C100, CIFAR-100-C;

• ImageNet-1K

  Near-OOD: Species, iNaturalist, ImageNet-O, OpenImage-O, ImageNet21K-P;
  Far-OOD: Texture, MNIST, SVHN, Places365;
  CS-ID: ImageNet-v2, ImageNet-C;

---

Supported Backbones (6)

This part lists all the backbones we will support in our codebase, including CNN-based and Transformer-based models. Backbones like ResNet-50 and Transformer have ImageNet-1K/22K pretrained models.

CNN-based Backbones (4)

• LeNet-5
• ResNet-18
• WideResNet-28
• ResNet-50 (BiT)

Transformer-based Architectures (2)

• ViT (DeiT)
• Swin Transformer

---

Supported Methods (34)

This part lists all the methods we include in this codebase. In v0.5, we totally support more than 32 popular methods for generalized OOD detection.

All the supported methodolgies can be placed in the following four categories.

     

We also note our supported methodolgies with the following tags if they have special designs in the corresponding steps, compared to the standard classifier training process.

     

Anomaly Detection (5)

• 
• 
• 
• 
• 

Open Set Recognition (3)

No Extra Data (2):

• 
• 

With Extra Data (1):

• 

Out-of-Distribution Detection (20)

No Extra Data (17):

• 
•   
•   
•     
•   
•   
•     
•   
•   
•       
•   
•   
•   
•   
•     
•   
•       

With Extra Data (3):

•    ![extradata]
•    ![extradata]   ![training]
•    ![extradata]   ![training]

Other Methods on Robustness and Uncertainty (6)

•    ![training]   ![postprocess]
•    ![training]
•    ![postprocess]
•   
•    ![preprocess]
•    ![preprocess]

---

Contributing

We appreciate all contributions to improve OpenOOD. We sincerely welcome community users to participate in these projects. Please refer to CONTRIBUTING.md for the contributing guideline.

Contributors

Citation

If you find our repository useful for your research, please consider citing our paper:

@article{yang2022openood,
    author = {Yang, Jingkang and {\textit{et al.}}},
    title = {OpenOOD: Benchmarking Generalized Out-of-Distribution Detection},
    year = {2022}
}

@article{yang2022fsood,
    title = {Full-Spectrum Out-of-Distribution Detection},
    author = {Yang, Jingkang and Zhou, Kaiyang and Liu, Ziwei},
    journal={arXiv preprint arXiv:2204.05306},
    year = {2022}
}

@article{yang2021oodsurvey,
    title={Generalized Out-of-Distribution Detection: A Survey},
    author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
    journal={arXiv preprint arXiv:2110.11334},
    year={2021}
}