Home

Awesome

MIDAS

<p> <a href="https://aaai.org/Conferences/AAAI-20/"> <img src="http://img.shields.io/badge/AAAI-2020-red.svg"> </a> <a href="https://arxiv.org/pdf/2009.08452.pdf"><img src="http://img.shields.io/badge/Paper-PDF-brightgreen.svg"></a> <a href="https://www.comp.nus.edu.sg/~sbhatia/assets/pdf/MIDAS_slides.pdf"> <img src="http://img.shields.io/badge/Slides-PDF-ff9e18.svg"> </a> <a href="https://youtu.be/Bd4PyLCHrto"> <img src="http://img.shields.io/badge/Talk-Youtube-ff69b4.svg"> </a> <a href="https://www.youtube.com/watch?v=DPmN-uPW8qU"> <img src="https://img.shields.io/badge/Overview-Youtube-orange.svg"> </a> <a href="https://github.com/Stream-AD/MIDAS/blob/master/LICENSE"> <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"> </a> </p>

C++ implementation of

The old implementation is in another branch OldImplementation, it should be considered as being archived and will hardly receive feature updates.

Table of Contents

<!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> <!-- END doctoc generated TOC please keep comment here to allow auto update -->

Features

Demo

If you use Windows:

  1. Open a Visual Studio developer command prompt, we want their toolchain
  2. cd to the project root MIDAS/
  3. cmake -DCMAKE_BUILD_TYPE=Release -GNinja -S . -B build/release
  4. cmake --build build/release --target Demo
  5. cd to MIDAS/build/release/
  6. .\Demo.exe

If you use Linux/macOS:

  1. Open a terminal
  2. cd to the project root MIDAS/
  3. cmake -DCMAKE_BUILD_TYPE=Release -S . -B build/release
  4. cmake --build build/release --target Demo
  5. cd to MIDAS/build/release/
  6. ./Demo

The demo runs on MIDAS/data/DARPA/darpa_processed.csv, which has 4.5M records, with the filtering core (MIDAS-F).

The scores will be exported to MIDAS/temp/Score.txt, higher means more anomalous.

All file paths are absolute and "hardcoded" by CMake, but it's suggested NOT to run by double clicking on the executable file.

Requirements

Core

Demo (if experimental ROC-AUC impl)

Demo (if sklearn ROC-AUC impl)

Experiment

Other python utility scripts

Customization

Switch to sklearn ROC-AUC Implementation

In MIDAS/example/Demo.cpp.
Comment out section "Evaluate scores (experimental)"
Uncomment section "Write output scores" and "Evaluate scores".

Different CMS Size / Decay Factor / Threshold

Those are arguments of cores' constructors, which are at MIDAS/example/Demo.cpp:67-69.

Switch Cores

Cores are instantiated at MIDAS/example/Demo.cpp:67-69, uncomment the chosen one.

Custom Dataset + Demo.cpp

You need to prepare three files:

Custom Dataset + Custom Runner

  1. Include the header MIDAS/src/NormalCore.hpp, MIDAS/src/RelationalCore.hpp or MIDAS/src/FilteringCore.hpp
  2. Instantiate cores with required parameters
  3. Call operator() on individual data records, it returns the anomaly score for the input record

Other Files

example/

Experiment.cpp

The code we used for experiments.
It will try to use Intel TBB or OpenMP for parallelization.
You should comment all but only one runner function call in the main() as most results are exported to MIDAS/temp/Experiiment.csv together with many intermediate files.

Reproducible.cpp

Similar to Demo.cpp, but with all random parameters hardcoded and always produce the same result.
It's for other developers and us to test if the implementation in other languages can produce acceptable results.

util/

DeleteTempFile.py, EvaluateScore.py and ReproduceROC.py will show their usage and a short description when executed without any argument.

AUROC.hpp

Experimental ROC-AUC implementation in C++11. More info at this repo.

PreprocessData.py

The code to process the raw dataset into an easy-to-read format.
Datasets are always assumed to be in a folder in MIDAS/data/.
It can process the following dataset(s)

In Other Languages

  1. Python: Rui Liu's MIDAS.Python, Ritesh Kumar's pyMIDAS
  2. Python (pybind): Wong Mun Hou's MIDAS
  3. Golang: Steve Tan's midas
  4. Ruby: Andrew Kane's midas
  5. Rust: Scott Steele's midas_rs
  6. R: Tobias Heidler's MIDASwrappeR
  7. Java: Joshua Tokle's MIDAS-Java
  8. Julia: Ashrya Agrawal's MIDAS.jl

Online Coverage

  1. ACM TechNews
  2. AIhub
  3. Hacker News
  4. KDnuggets
  5. Microsoft
  6. Towards Data Science

Citation

If you use this code for your research, please consider citing our TKDD and AAAI papers.

@article{bhatia2022realtime,
author = {Bhatia, Siddharth and Liu, Rui and Hooi, Bryan and Yoon, Minji and Shin, Kijung and Faloutsos, Christos},
title = {Real-Time Anomaly Detection in Edge Streams},
year = {2022},
issue_date = {August 2022},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {16},
number = {4},
issn = {1556-4681},
url = {https://doi.org/10.1145/3494564},
doi = {10.1145/3494564},
journal = {ACM Trans. Knowl. Discov. Data},
month = {jan},
articleno = {75},
numpages = {22}
}

@inproceedings{bhatia2020midas,
    title={MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams},
    author={Siddharth Bhatia and Bryan Hooi and Minji Yoon and Kijung Shin and Christos Faloutsos},
    booktitle={AAAI Conference on Artificial Intelligence (AAAI)},
    year={2020}
}