MemStream

Implementation of

• MemStream: Memory-Based Anomaly Detection in Multi-Aspect Streams with Concept Drift . Siddharth Bhatia, Arjit Jain, Shivin Srivastava, Kenji Kawaguchi, Bryan Hooi

MemStream detects anomalies from a multi-aspect data stream. We output an anomaly score for each record. MemStream is a memory augmented feature extractor, allows for quick retraining, gives a theoretical bound on the memory size for effective drift handling, is robust to memory poisoning, and outperforms 11 state-of-the-art streaming anomaly detection baselines.

After an initial training of the feature extractor on a small subset of normal data, MemStream processes records in two steps: (i) It outputs anomaly scores for each record by querying the memory for K-nearest neighbours to the record encoding and calculating a discounted distance and (ii) It updates the memory, in a FIFO manner, if the anomaly score is within an update threshold β.

Datasets

Processed Datasets can be downloaded from here. Please unzip and place the files in the data folder of the repository.

Demo

1. KDDCUP99: Run python3 memstream.py --dataset KDD --beta 1 --memlen 256
2. NSL-KDD: Run python3 memstream.py --dataset NSL --beta 0.1 --memlen 2048
3. UNSW-NB 15: Run python3 memstream.py --dataset UNSW --beta 0.1 --memlen 2048
4. CICIDS-DoS: Run python3 memstream.py --dataset DOS --beta 0.1 --memlen 2048
5. SYN: Run python3 memstream-syn.py --dataset SYN --beta 1 --memlen 16
6. Ionosphere: Run python3 memstream.py --dataset ionosphere --beta 0.001 --memlen 4
7. Cardiotocography: Run python3 memstream.py --dataset cardio --beta 1 --memlen 64
8. Statlog Landsat Satellite: Run python3 memstream.py --dataset statlog --beta 0.01 --memlen 32
9. Satimage-2: Run python3 memstream.py --dataset satimage-2 --beta 10 --memlen 256
10. Mammography: Run python3 memstream.py --dataset mammography --beta 0.1 --memlen 128
11. Pima Indians Diabetes: Run python3 memstream.py --dataset pima --beta 0.001 --memlen 64
12. Covertype: Run python3 memstream.py --dataset cover --beta 0.0001 --memlen 2048

Command line options

• --dataset: The dataset to be used for training. Choices 'NSL', 'KDD', 'UNSW', 'DOS'. (default 'NSL')
• --beta: The threshold beta to be used. (default: 0.1)
• --memlen: The size of the Memory Module (default: 2048)
• --dev: Pytorch device to be used for training like "cpu", "cuda:0" etc. (default: 'cuda:0')
• --lr: Learning rate (default: 0.01)
• --epochs: Number of epochs (default: 5000)

Input file format

MemStream expects the input multi-aspect record stream to be stored in a contains , separated file.

Datasets

1. KDDCUP99
2. NSL-KDD
3. UNSW-NB 15
4. CICIDS-DoS
5. Synthetic Dataset (Introduced in paper)
6. Ionosphere
7. Cardiotocography
8. Statlog Landsat Satellite
9. Satimage-2
10. Mammography
11. Pima Indians Diabetes
12. Covertype

Environment

This code has been tested on Debian GNU/Linux 9 with a 12GB Nvidia GeForce RTX 2080 Ti GPU, CUDA Version 10.2 and PyTorch 1.5.