Skip to content

Commit

Permalink
updated readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@liznerski
liznerski authored Aug 27, 2020
1 parent 3b3635b commit 501c7f9
Showing 1 changed file with 108 additions and 57 deletions.
165 changes: 108 additions & 57 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,8 @@
# Explainable Deep One-Class Classification
Here we provide the implementation of *Fully Convolutional Data Description* (FCDD), an explainable approach to deep one-class classification.
The implementation is based on PyTorch.
The implementation is based on PyTorch 1.4.0 and Python 3.6.

The general idea of FCDD is to make the model a fully convolutional network directly yielding pixel-related anomaly scores.
Furthermore, we provide an upsampling technique based on the receptive field of the network that outputs full-resolution anomaly heatmaps.
You can find an overview of the method below:

<img src="data/git_images/fcdd_summary.png?raw=true" height="229" width="430" >

With the provided code you can use FCDD to produce heatmaps like this:
Deep one-class classification variants for anomaly detection learn a mapping that concentrates nominal samples in feature space causing anomalies to be mapped away. Because this transformation is highly non-linear, finding interpretations poses a significant challenge. We present an explainable deep one-class classification method, *Fully Convolutional Data Description* (FCDD), where the mapped samples are themselves also an explanation heatmap. FCDD yields competitive detection performance and provides reasonable explanations on common anomaly detection benchmarks with CIFAR-10 and ImageNet. On MVTec-AD, a recent manufacturing dataset offering ground-truth anomaly maps, FCDD meets the state of the art in an unsupervised setting, and outperforms its competitors in a semi-supervised setting. The following image shows some of the FCDD explanation heatmaps for test samples of MVTec-AD:

<img src="data/git_images/fcdd_explanations_mvtec.png?raw=true" height="373" width="633" >

Expand All @@ -28,6 +22,16 @@ If you use our work, please also cite the current preprint:
}
```

## Table of contents

* [Installation](#installation)
* [Train FCDD](#train-fcdd)
* [Train Baselines](#train-baselines)
* [Log Data Explained](#explanation-of-the-log-data)
* [Customize FCDD](#customize-fcdd)
* [Help](#need-help)


## Installation
It is recommended to use a virtual environment to install FCDD.
Assuming you're in the `python` directory, install FCDD via pip:
Expand All @@ -37,14 +41,18 @@ Assuming you're in the `python` directory, install FCDD via pip:
pip install .


## Train FCDD and Reproduce Results
## Train FCDD

**Log data** Log data -- i.e. heatmaps, metrics, snapshots, etc. -- is stored on the disc in a certain log directory.
The default log directory is located at `../../data/results/fcdd_TIMESTAMP`.
Thus, log data is being saved in the data directory in the root folder of this repository if the code is run from `python/fcdd`.
The same holds for the data directory, where the datasets are downloaded to.
You can change both, the log directory and the data directory, by altering the respective arguments (logdir and datadir).

If you have used a virtual environment to install FCDD, make sure to activate it.
We recommend training FCDD by starting one of the runners in `python/fcdd/runners` from the `python/fcdd` directory.
This way the default parameters look for datasets in the `data/datasets` directory and store log data in `data/results`.
If you do not start the runners from `python/fcdd`, you need to give the
runners the path to the datasets (argument datadir) and the path to the log directory (argument logdir), otherwise
the data might be stored in unexpected locations.
**Virtual Environment** If you have used a virtual environment to install FCDD, make sure to activate it.

**Train Scrips** We recommend training FCDD by starting one of the runners in `python/fcdd/runners` from the `python/fcdd` directory.
They train several (randomly initialized) separate FCDD models for each class, where in each the respective class is considered nominal.

#### Fashion-MNIST
With EMNIST OE:
Expand Down Expand Up @@ -73,11 +81,13 @@ For full OE:

Please note that you have to manually download ImageNet1k and ImageNet22k and place them in the correct folders.
Let **dsdir** be your specified dataset directory (per default `../../data/datasets/`).
ImageNet1k needs to be in `dsdir/imagenet`, containing the devkit, train, and val split in form of a tar file each, with names ILSVRC2012_devkit_t12.tar.gz, ILSVRC2012_img_train.tar, and ILSVRC2012_img_val.tar.
These are the default names expected by the PyTorch loaders.

ImageNet1k needs to be in `dsdir/imagenet`, containing the devkit, train, and val split in form of a tar file each -- with names `ILSVRC2012_devkit_t12.tar.gz`, `ILSVRC2012_img_train.tar`, and `ILSVRC2012_img_val.tar`.
These are the default names expected by the PyTorch loaders. You can download ImageNet1k on the official website: http://image-net.org/download. Note that you have to register beforehand.

ImageNet22k needs to be in `dsdir/imagenet22k/fall11_whole_extracted`, containing all the extracted class directories with pictures, e.g. the folder n12267677 having pictures of acorns.
You can download ImageNet1k on the official website after having registered: http://image-net.org/download.
ImageNet22k, i.e. the full release fall 11, can also be found there.
Decompressing the downloaded archive should automatically yield this structure.
ImageNet22k, i.e. the full release fall 11, can also be downloaded on the official website: http://image-net.org/download.

#### MVTec-AD
Using confetti noise:
Expand Down Expand Up @@ -106,7 +116,11 @@ Similarily for the AE baseline:

python runners/run_cifar10.py --objective ae -n AE32 --supervise-mode unsupervised --blur-heatmaps

Alternatively, one can use the `add_exp_to_base.py` script in `python/fcdd/runners`.

<details>
<summary>Alternatively, one can use the add_exp_to_base script:</summary>

The script is located at `python/fcdd/runners/add_exp_to_base.py`.
It takes a log directory of a previous experiment as input and automatically changes the arguments to
run baseline experiments with the same basic configuration.
This script has also the advantage of automatically aggregating heatmaps of the given old experiment and baseline experiments to create combined pictures as seen in the paper.
Expand All @@ -122,67 +136,80 @@ Note that the previous experiment needs to be an FCDD experiment and not a basel
Also, note that the baseline experiments use the same data directory parameter as found in the old experiment configuration file.
That is, if `add_exp_to_base.py` is not started from the same directory (e.g. `python/fcdd`), it will not find the datasets
and attempt to download it to another one.
</details>

## Explanation of the Log Data / Results

## Explanation of the Log Data

A runner saves the achieved scores, metrics, plots, snapshots, and heatmaps in a given log directory.
Each log directory contains a subdirectory for each class that is trained to be nominal.
Each log directory contains a separate subdirectory for each class that is trained to be nominal.
These subdirectories are named "normal_x", where x is the class number.
The class subdirectories again contain a subdirectory for each random seed.
These are named "it_x" for x being the iteration number (random seed).
Inside the seed subdirectories all actual log data can be found.
Additionally, summarized plots will be created for the class subdirectories and the root log directory.
For instance, a plot containing ROC curves for each class (averaged over all seeds) can be found in the root log directory.

Visualization for 2 classes and 2 random seeds: \
./log_directory \
./log_directory/normal_0 \
./log_directory/normal_0/it_0 \
./log_directory/normal_0/it_1 \
./log_directory/normal_1 \
./log_directory/normal_1/it_0 \
./log_directory/normal_1/it_1 \
...
Visualization for 2 classes and 2 random seeds:

./log_directory
./log_directory/normal_0
./log_directory/normal_0/it_0
./log_directory/normal_0/it_1
./log_directory/normal_1
./log_directory/normal_1/it_0
./log_directory/normal_1/it_1
...

Note that the leaf nodes, i.e. the iteration subdirectories, contain completely separate training results and have no impact on each other.

The actual log data consists of:
- config.txt: A file containing all the arguments of the training.
- ds_preview.png: A preview of the dataset, i.e. some random nominal and anomalous samples from the training set. Includes augmentation and data preprocessing. Also shows corresponding ground-truth maps, if such are available.
- err.pdf: A plot of the loss over time.
- err_anomalous.pdf: A plot of the loss for anomalous samples only.
- err_normal.pdf: A plot of the loss for nominal samples only.
- heatmaps_paper_x_lbly.png: An image of test heatmaps, test inputs, and ground-truth maps. One image for each x-y combination. x is the normalization used, either local (each heatmap is normalized w.r.t. itself only) or semi_global (each heatmap is normalized w.r.t. to all heatmaps in the image). y is the label, i.e. either 1 or 0 (per default 1 is for anomalies).
- heatmaps_semi_global.png: An image of the first test heatmaps and inputs found in the dataset. The first row shows nominal samples, the second-row anomalous samples. The third row shows the ten most nominal rated nominal samples on the left and the ten most anomalous rated nominal samples on the right. The fourth row shows the ten most nominal rated anomalies on the left and the ten most anomalous rated anomalies on the right. Note that changing the nominal label to 1 flips this.
- train_heatmaps_semi_global.png: Like above, but for training samples.
- history.json: A file containing metrics in text form.
- log.txt: A file containing some logged txt lines, like the average AUC value achieved and the duration of the training.
- print.log: Logs all that has been printed using the Logger.
- roc.json: ROC values saved as text (not very readable for humans).
- roc_curve.pdf: ROC for detection performance.
- gtmap_roc.pdf: ROC for explanation performance. Only for MVTec-AD (or datasets with ground-truth maps).
- snapshot.pt: Snapshot of the training state and model parameters.
- src.tar.gz: Snapshot of the complete source code at the moment of the training start time.
- tims: A directory containing raw tensors heatmaps (not readable for humans).
- **config.txt**: A file containing all the arguments of the training.
- **ds_preview.png**: A preview of the dataset, i.e. some random nominal and anomalous samples from the training set. Includes augmentation and data preprocessing. Also shows corresponding ground-truth maps, if such are available.
- **err.pdf**: A plot of the loss over time.
- **err_anomalous.pdf**: A plot of the loss for anomalous samples only.
- **err_normal.pdf**: A plot of the loss for nominal samples only.
- **heatmaps_paper_x_lbly.png**: An image of test heatmaps, test inputs, and ground-truth maps. One image for each x-y combination. x is the normalization used, either local (each heatmap is normalized w.r.t. itself only) or semi_global (each heatmap is normalized w.r.t. to all heatmaps in the image). y is the label, i.e. either 1 or 0 (per default 1 is for anomalies).
- **heatmaps_semi_global.png**: An image of the first test heatmaps and inputs found in the dataset. The first row shows nominal samples, the second-row anomalous samples. The third row shows the ten most nominal rated nominal samples on the left and the ten most anomalous rated nominal samples on the right. The fourth row shows the ten most nominal rated anomalies on the left and the ten most anomalous rated anomalies on the right. Note that changing the nominal label to 1 flips this.
- **train_heatmaps_semi_global.png**: Like above, but for training samples.
- **history.json**: A file containing metrics in text form.
- **log.txt**: A file containing some logged text lines, like the average AUC value achieved and the duration of the training.
- **print.log**: A file that contains all text that has been printed on the console using the Logger.
- **roc.json**: ROC values saved as text (not very readable for humans).
- **roc_curve.pdf**: ROC for detection performance.
- **gtmap_roc.pdf**: ROC for explanation performance. Only for MVTec-AD (or datasets with ground-truth maps).
- **snapshot.pt**: Snapshot of the training state and model parameters.
- **src.tar.gz**: Snapshot of the complete source code at the moment of the training start time.
- **tims**: A directory containing raw tensors heatmaps (not readable for humans).

## Customize FCDD

In general, the FCDD implementation splits in five packages: `datasets`, `models`, `runners`, `training`, and `util`.
`dataset` contains an implementation of the base class for AD datasets, actual torchvision-style dataset implementations, and artificial anomaly generation implementations.
`models` contains an implementation of the network base class -- including receptive field upsampling and gradient-based heatmap computation -- and implementations of all network architectures.
`runners` contains scripts for starting training runs, processing program arguments, and preparing all training parameters -- like creating an optimizer and network instance.
`runners` contains scripts for starting training runs, processing program arguments, and preparing all training parameters -- like creating an optimizer and network instances.
`training` contains an implementation for actual training and evaluation of the network.
`util` contains all the rest, i.e. e.g. a logger that handles all I/O interactions.

In the following we give a brief tutorial on how to modify the FCDD implementation for specific requirements.

#### Add a new Network Architecture

<details>
<summary>Add a new Network Architecture</summary>

1) Create a new python script in the `models` package.
Implement a PyTorch module that inherits the `fcdd.models.bases.BaseNet` or the `fcdd.models.bases.ReceptiveNet` class.
The latter takes track of the receptive field. Read the documentation for further details.

2) Import the new network class in `fcdd.models.__init__` to automatically add the network name to the available ones.
Make sure that the class name is unique.

#### Add a new Dataset
</details>


<details>
<summary>Add a new Dataset</summary>

1) Create a new python script in the `datasets` package.
Implement a dataset that inherits the `fcdd.datasets.bases.TorchvisionDataset` class.
Your implementation needs to process all parameters of the `fcdd.datasets.bases.load_dataset` function in its initialization.
Expand All @@ -196,7 +223,11 @@ Add your dataset to the "switch-case" in the `fcdd.datasets.__init__.load_datase
Add the number of available class for your dataset to the `fcdd.datasets.__init__.no_classes` function and
add the class names to `fcdd.datasets.__init__.str_labels`.

#### Add a new Training Parameter
</details>

<details>
<summary>Add a new Training Parameter</summary>

1) Add an argument to the `fcdd.runners.argparse_configs.DefaultConfig` class.

2) Add this argument as a parameter to the `fcdd.training.setup.trainer_setup` function.
Expand All @@ -206,7 +237,11 @@ Also, make it return the parameter itself, or some other thing that has been cre

4) Implement your requested novel behavior based on the new parameter in the SuperTrainer or one of the actual model trainers (FCDDTrainer, HSCTrainer, AETrainer), depending on the objective.

#### Add a new Runner
</details>


<details>
<summary>Add a new Runner</summary>

1) Create a new python script in the `runners` package.
Implement a configuration that inherits `fcdd.runners.argparse_configs.DefaultConfig`.
Expand All @@ -216,31 +251,47 @@ You need to implement the __call__ method for it, where you can add new argument
For both, the SeedsRunner and ClassesRunner, you need to additionally add an argument in your configuration named **it** that determines the number of different random seeds.
Invoke the instance's run method.

</details>



#### Add a new Objective / Trainer
<details>
<summary>Add a new Objective / Trainer</summary>

1) Add a string identification for your new objective to the `fcdd.training.setup.OBJECTIVES` variable.

2) Create a new trainer script in the `training` package.
Implement a trainer inheriting `fcdd.training.bases.BaseADTrainer`.
At least implement the loss and snapshot method.

3) Add the creation of an instance for your new trainer to the "switch-case" in the initialization of `fcdd.training.super_trainer.SuperTrainer`.
3) Add the creation of an instance for your new trainer to the "switch-case" in the initialization of `fcdd.training.super_trainer.SuperTrainer`.

</details>


<details>
<summary>Add a new Type of Synthetic Anomaly</summary>

#### Add a new Type of Synthetic Anomaly
1) Implement your new synthetic anomaly in `fcdd.datasets.noise`.

2) Add a string identification for you new function to the `fcdd.datasets.noise_modes.MODES` variable.

3) Add an invocation of your new function to the "switch-case" in `fcdd.datasets.noise_modes.generate_noise`.

#### Add a new Outlier Exposure Dataset
</details>


<details>
<summary>Add a new Outlier Exposure Dataset</summary>

1) Create a new python script in the `datasets/outlier_exposure` package.
Implement a torchvision-style dataset that expects at least the following parameters: **size**, **root**, and **limit_var**.
Also, you need to at least implement a __get_item__ method and *data_loader* method.
Cf. one of the existing implementations of Outlier Exposure datasets, e.g. `fcdd.datasets.outlier_exposure.emnist`.

</details>


## Need help?
If you find any bugs, have questions, or need help modifying FCDD, feel free to write us an [email](mailto:[email protected])!

0 comments on commit 501c7f9

Please sign in to comment.