Home

Awesome

Awesome LLMs for Anomaly and OOD Detection

Tracking advancements in "Large Language Models for Anomaly and Out-of-Distribution Detection", based on our detailed survey found at Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey.

Table of Contents

Introduction

Overview of LLMs for Anomaly and OOD Detection

Detecting anomalies or out-of-distribution (OOD) samples is critical for maintaining the reliability and trustworthiness of machine learning systems. Recently, Large Language Models (LLMs) have demonstrated their effectiveness not only in natural language processing but also in broader applications due to their advanced comprehension and generative capabilities. The integration of LLMs into anomaly and OOD detection marks a significant shift from the traditional paradigm in the field. This survey focuses on the problem of anomaly and OOD detection under the context of LLMs. We propose a new taxonomy to categorize existing approaches into three classes based on the role played by LLMs. Following our proposed taxonomy, we further discuss the related work under each of the categories and finally discuss potential challenges and directions for future research in this field.

Taxonomy

LLMs for Augmentation

Exploring how LLMs support the augmentation of detection capabilities without being direct detectors.

PaperAuthorsBackbone ModelTask CategoryDataset TypeVenueCode
Envisioning outlier exposure by large language models for out-of-distribution detectionChentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, Bo HanGPT-3.5-turbo-16k; CLIPOOD DetectionImagesICML, 2024Code
Out-of-Distribution Detection Using Peer-Class Generated by Large Language ModelK Huang, G Song, Hanwen Su, Jiyan WangGPT-3; CLIPOOD DetectionImagesArXiv, 2024N/A
On the Powerfulness of Textual Outlier Exposure for Visual OoD DetectionSangha Park, Jisoo Mok, Dahuin Jung, Saehyung Lee, Sungroh YoonBERT; BLIP-2; GPT-3; CLIPOOD DetectionImagesNeurIPS, 2023Code
Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language ModelsAlbert Xu, Xiang Ren, Robin JiaGPT-3; GPT-J; BERTOOD DetectionTextsACL, 2023Code
Tagfog: Textual anchor guidance and fake outlier generation for visual out-of-distribution detectionJiankang Chen, Tong Zhang, Weishi Zheng, Ruixuan WangChatGPT; CLIPOOD DetectionImagesAAAI, 2024Code
Do LLMs Understand Visual Anomalies? Uncovering LLM Capabilities in Zero-shot Anomaly DetectionJiaqi Zhu, Shaofeng Cai, Fang Deng, Junran WuGPT-3.5; CLIPAnomaly DetectionImagesArXiv, 2024N/A
Exploring large language models for multi-modal out-of-distribution detectionYi Dai, Hao Lang, Kaisheng Zeng, Fei Huang, Yongbin Litext-davinci-003; CLIPOOD DetectionImagesEMNLP, 2023N/A
LogGPT: Exploring ChatGPT for log-based anomaly detectionJiaxing Qi et al.ChatGPTAnomaly DetectionLog DataIEEE HPCC, 2023Code
LogFiT: Log Anomaly Detection Using Fine-Tuned Language ModelsCrispin Almodovar et al.Various LLMsAnomaly DetectionLog DataIEEE TNSM 2024N/A
How good are LLMs at out-of-distribution detection?Andi Zhang et al.LLaMA etc.OOD DetectionVariousCOLING, 2024Code
Your Finetuned Large Language Model is Already a Powerful Out-of-distribution DetectorAndi Zhang, Tim Z Xiao, Weiyang Liu, Robert Bamler, Damon WischikVarious LLMsOOD DetectionTextsarXiv, 2024N/A

LLMs for Detection

Highlighting how LLMs directly contribute to detecting anomalies and out-of-distribution samples.

LLMs for Detection

PaperAuthorsBackbone ModelTask CategoryDataset TypeVenueCode
WinCLIP: Zero-/few-shot anomaly classification and segmentationJongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, Onkar DabeerCLIPAnomaly DetectionImagesCVPR, 2023Code
CLIP-AD: A language-guided staged dual-path model for zero-shot anomaly detectionXuhai Chen, Jiangning Zhang, Guanzhong Tian, Haoyang He, Wuhao Zhang, Yabiao Wang, Chengjie Wang, Yunsheng Wu, Yong LiuCLIPAnomaly DetectionImagesarXiv, 2023N/A
Exploring grounding potential of VQA-oriented GPT-4V for zero-shot anomaly detectionJiangning Zhang, Xuhai Chen, Zhucun Xue, Yabiao Wang, Chengjie Wang, Yong LiuGPT-4VAnomaly DetectionImagesarXiv, 2023Code
CLIPScope: Enhancing Zero-Shot OOD Detection with Bayesian ScoringHao Fu, Naman Patel, Prashanth Krishnamurthy, Farshad KhorramiCLIPOOD DetectionImagesArXiv, 2024N/A
AnomalyCLIP: Object-agnostic prompt learning for zero-shot anomaly detectionQihang Zhou, Guansong Pang, Yu Tian, Shibo He, Jiming ChenCLIPAnomaly DetectionImagesICLR, 2024Code
Toward generalist anomaly detection via in-context residual learning with few-shot sample promptsJiawen Zhu, Guansong PangCLIPAnomaly DetectionImagesCVPR, 2024Code
PromptAD: Learning prompts with only normal samples for few-shot anomaly detectionXiaofan Li, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yuan Xie, Lizhuang MaCLIPAnomaly DetectionImagesCVPR, 2024Code
Text prompt with normality guidance for weakly supervised video anomaly detectionZhiwei Yang, Jing Liu, Peng WuCLIPAnomaly DetectionVideosarXiv, 2024N/A
LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt LearningMiyai et al.CLIPOOD DetectionImagesNeurIPS, 2023Code
CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say NoWang et al.CLIPOOD DetectionImagesICCV, 2023Code
Out-Of-Distribution Detection With Negative PromptsNie et al.CLIPOOD DetectionImagesICLR, 2024Code
ID-like Prompt Learning for Few-Shot Out-of-Distribution DetectionYichen Bai, Zongbo Han, Changqing Zhang, Bing Cao, Xiaoheng Jiang, Qinghua HuCLIPOOD DetectionImagesCVPR, 2024Code
Learning transferable negative prompts for out-of-distribution detectionTianqi Li, Guansong Pang, Xiao Bai, Wenjun Miao, Jin ZhengCLIPOOD DetectionImagesCVPR, 2024Code
AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language ModelsZhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, Jinqiao WangImageBind-Huge; Vicuna-7BAnomaly DetectionImagesArXiv, 2024Code
Adapting visual-language models for generalizable anomaly detection in medical imagesChaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xinchao Wang, Yanfeng WangCLIPAnomaly DetectionImagesCVPR, 2024Code
Adapting Contrastive Language-Image Pretrained (CLIP) Models for Out-of-Distribution DetectionNikolas Adaloglou, Felix Michels, Tim Kaiser, Markus KollmannCLIPOOD DetectionImagesTMLR, 2024Code
Video anomaly detection and explanation via large language modelsHui Lv, Qianru SunVideo-LLaMAAnomaly DetectionVideosarXiv, 2024N/A
VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly DetectionPeng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, Yanning ZhangCLIPAnomaly DetectionVideosAAAI, 2023Code
Delving into Out-of-Distribution Detection with Vision-Language RepresentationsMing et al.CLIPOOD DetectionImagesNeurIPS, 2022Code
Text prompt with normality guidance for weakly supervised video anomaly detectionZhiwei Yang, Jing Liu, Peng WuCLIPAnomaly DetectionVideosarXiv, 2024N/A
Negative Label Guided OOD Detection with Pretrained Vision-Language ModelsXue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, Bo HanCLIP; ALIGN; GroupViT; AltCLIPOOD DetectionImagesICLR, 2024Code
Zero-shot out-of-distribution detection based on the pretrained model CLIPSepideh Esmaeilpour, Bing Liu, Eric Robertson, Lei ShuCLIPOOD DetectionImagesAAAI, 2022Code
Large language models can be zero-shot anomaly detectors for time series?Sarah Alnegheimish, Linh Nguyen, Laure Berti-Equille, Kalyan VeeramachaneniMistral-7B-Instruct-v0.2; gpt-3.5-turbo-instructAnomaly DetectionTime SeriesarXiv, 2024N/A
Semantic anomaly detection with large language modelsAmine Elhafsi, Rohan Sinha, Christopher Agia, Edward Schmerling, Issa Nesnas, Marco Pavonetext-davinci-003Anomaly DetectionVideosarXiv, 2023N/A
Harnessing large language models for training-free video anomaly detectionLuca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, Elisa RicciLlama-2-13b-chat; ImageBindAnomaly DetectionVideosCVPR, 2024Code
Large language models can deliver accurate and interpretable time series anomaly detectionJiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, Ying-Cong ChenGPT-4-1106-previewAnomaly DetectionTime SeriesarXiv, 2024N/A
FiLo: Zero-shot anomaly detection by fine-grained description and high-quality localizationZhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Hao Li, Ming Tang, Jinqiao WangCLIPAnomaly DetectionVisionarXiv, 2024Code
Can LLMs Serve As Time Series Anomaly Detectors?Manqing Dong, Hao Huang, Longbing CaoGPT-4Anomaly DetectionTime SeriesarXiv, 2024N/A

LLMs for Explanation

Detailing how LLMs aid in explaining the detection results, enhancing understanding and trust.

PaperAuthorsBackbone ModelTask CategoryDataset TypeVenueCode
Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLMHuaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Chuchu Han, Xiaonan Huang, Changxin Gao, Yuehuan Wang, Nong SangVideo-LLaVAAnomaly DetectionVideosArXiv, 2024Code
Follow the Rules: Reasoning for Video Anomaly Detection with Large Language ModelsYuchen Yang, Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, Shao-Yuan LoCogVLM-17B; GPT-4; Mistral-7B-Instruct-v0.2Anomaly DetectionVideosArXiv, 2024Code
Video Anomaly Detection and Explanation via Large Language ModelsLv et al.LLaMAAnomaly DetectionVideosICCV, 2024N/A
Real-Time Anomaly Detection and Reactive Planning with Large Language ModelsSinha et al.BERT, Llama 2 etc.Anomaly DetectionRobotic DataArXiv, 2024N/A

Citation

If you find this work useful, please cite our survey paper:

@article{xu2024large,
      title={Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey}, 
      author={Ruiyao Xu and Kaize Ding},
      journal={arXiv preprint arXiv:2409.01980},
      year={2024},
}