Home

Awesome

AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Model (AAAI 2024)

<!-- <br> -->

Teng Hu<sup>1#</sup>, Jiangning Zhang<sup>2#</sup>, Ran Yi<sup>1*</sup>, Yuzhen Du<sup>1</sup>, Xu Chen<sup>2</sup>, Liang Liu<sup>2</sup>, Yabiao Wang<sup>2</sup>, and Chengjie Wang<sup>1,2</sup>.

<!-- <br> -->

(#Equal contribution,*Corresponding author)

<sup>1</sup>Shanghai Jiao Tong University, <sup>2</sup>Youtu Lab, Tencent

arXiv

Project Page

News

June 29, 2024

Todo (Latest update: 2024/08/11)

Data and checkpoints

The generated anomaly data and all the checkpoints can be downloaded from the following links. And some checkpoints should be placed at the corresponding directory. (Note that we have filtered out some data with poor generation effects. Therefore, some classes have relatively fewer samples.)

Data and ModelsDownloadPlace at
Generated dataGoogle Drive$path_to_the_generated_data
Checkpoints for anomaly generation modelGoogle Drivelogs/anomaly-checkpoints
Checkpoints for mask generation modelGoogle Drivelogs/mask-checkpoints
Checkpoints for anomaly localizationGoogle Drivecheckpoints/localization
Checkpoints for anomaly classificationGoogle Dricecheckpoints/classification

Overview

Anomalydiffusion is a few-shot anomaly generation model for anomaly inspection (detection, localization and classification). The overall process can be divided into the following 5 steps:

(1) Train the anomaly generation model and mask generation model;

(2) Generate anomalous masks by the mask generation model in step (1);

(3) Generate anomaly mask-image pairs by the anomaly generation model based on the generated masks in step (2);

(4) Train the anomaly detection (for both detection and localization) and classification model based on the anomalous image-mask pairs in step (3).

Prepare

(1) Prepare the environment

Ubuntu
python 3.8
cuda==11.8
gcc==7.5.0
conda env create -f environment.yaml
conda activate Anomalydiffusion

(2) Checkpoint for LDM

Download the official checkpoint of the latent diffusion model:

mkdir -p models/ldm/text2img-large/
wget -O models/ldm/text2img-large/model.ckpt https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large/model.ckpt

Generating anomalous image-mask pairs

In this section, you can first train the anomaly generation model by (1). After that, you can run (2), which contains training mask generation models, generating anomalous masks and generating anomalous image-mask pairs.

(1) Train the anomaly generation model by:

CUDA_VISIBLE_DEVICES=$gpu_id python main.py --spatial_encoder_embedding --data_enhance
 --base configs/latent-diffusion/txt2img-1p4B-finetune-encoder+embedding.yaml -t 
 --actual_resume models/ldm/text2img-large/model.ckpt -n test --gpus 0, 
  --init_word anomaly  --mvtec_path=$path_to_mvtec_dataset

(2) Train the mask generation model and generate image-mask pairs by:

CUDA_VISIBLE_DEVICES=$gpu_id python run-mvtec.py --data_path=$path_to_mvtec_dataset

Test the anomaly inspection performance

After generating anomalous image-mask pairs, you can train and test the anomaly detection model (for both anomlay detection and localization) by:

python train-localization.py --generated_data_path $path_to_the_generated_data  --mvtec_path=$path_to_mvtec
python test-localization.py --generated_data_path $path_to_mvtec

you can train and test the anomaly classification model by:

python train-classification.py --mvtec_path=$path_to_mvtec --generated_data_path=$path_to_the_generated_data
python test-classification.py --mvtec_path=$path_to_mvtec --generated_data_path=$path_to_the_generated_data

Compute IC-LPIPS

To compute IC-LPIPS for the generated dataset, please run:

python cal_ic_lpips.py --mvtec_path=$path_to_mvtec --gen_path=$path_to_the_generated_data

Citation

If you make use of our work, please cite our paper:

@inproceedings{hu2023anomalydiffusion,
  title={AnomalyDiffusion: Few-Shot Anomaly Image Generation with Diffusion Model},
  author={Hu, Teng and Zhang, Jiangning and Yi, Ran and Du, Yuzhen and Chen, Xu and Liu, Liang and Wang, Yabiao and Wang, Chengjie},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2024}
}