Home

Awesome

DREAM-OOD

This is the source code accompanying the paper Dream the Impossible: Outlier Imagination with Diffusion Models by Xuefeng Du, Yiyou Sun, Xiaojin Zhu, and Yixuan Li

The codebase is heavily based on Stable Diffusion.

TODO

Please find the link below for both the outlier and inlier images generated for ImageNet and Cifar100 datasets as follows

(Sorry for the delay in announcing the generated images because I did not find a free drive with a large storage. The link will expire in 30 days due to limitations at my university, thanks! If you have a suggested free online drive with large storage limits, please don't hesitate to forward it to me)

If you find the link expiring, please also email me!!

Ads

Check out our

Requirements

A suitable conda environment named dreamood can be created and activated with:

conda env create -f environment.yaml
conda activate dreamood

Please also install Xformers.

Dataset Preparation

ImageNet-100

python scripts/generate_in100.py --source_folder xxx --target_folder xxx

where "--source_folder" is the address of the full ImageNet dataset and "--target_folder" specifies the address of the dataset you want to store.

CIFAR-100

OOD datasets

Datasets for evaluating model generalization

Please download IMAGENET-A and IMAGENET-V2 and process the dataset by running (you need change the address of the datasets on your own):

python scripts/process_imagenetv2_and_a.py

Training

1. Learning the text-conditioned latent space

Please execute the following in the command shell on ImageNet-100:

python scripts/pretrain_in100.py

Please execute the following in the command shell on CIFAR-100:

python scripts/pretrain_cifar100.py

After training, it will generate ID feature embeddings for outlier/inlier embedding sampling.

2. Generate the inlier/outlier embeddings

Please execute the following in the command shell on ImageNet-100:

python scripts/get_embed_in100.py

Please execute the following in the command shell on CIFAR-100:

python scripts/get_embed_cifar100.py

After this step, you will see the generated inlier/outlier embedding in the root directory.

3. Synthesizing outliers in the pixel space

First, please download the Stable Diffusion 1.4 model here.

Please execute the following in the command shell on different datasets:

python scripts/dream_ood.py --plms \
--n_iter 50 --n_samples 3 \
--outdir /nobackup-fast/txt2img-samples-in100-demo/ \
--loaded_embedding /nobackup-slow/dataset/my_xfdu/diffusion/outlier_npos_embed.npy\
--ckpt /nobackup-slow/dataset/my_xfdu/diffusion/sd-v1-4.ckpt \
--id_data in100 \
--skip_grid

4. Training with the generated outliers in the pixel space

Please execute the following in the command shell for OOD detection on ImageNet-100:

python scripts/train_ood_det_in100.py --my_info xxx --load xxx

Note that in order to save time, we use a pretrained model for initialization, which is trained using the cross-entropy loss.

Please execute the following in the command shell for OOD detection on Cifar-100:

python scripts/train_ood_det_cifar100.py --my_info xxx

Here the model is trained from scratch.

5. Training with the generated inliers in the pixel space

Please execute the following in the command shell for generalization on ImageNet-100:

python scripts/train_gene_in100.py 

Test-time OOD detection

Please execute the following in the command shell with ImageNet-100 as in-distribution:

python scripts/test_ood_in100.py --load xxx

where "--load" specifies the address of the saved models.

Please execute the following in the command shell with Cifar-100 as in-distribution:

python scripts/test_ood_cifar100.py --load xxx

Test-time ID generalization

Please execute the following in the command shell with ImageNet-100:

python scripts/test_ood_in100_robustness.py --load xxx

Citation

If you found any part of this code is useful in your research, please consider citing our paper:

  @inproceedings{du2023dream,
      title={Dream the Impossible: Outlier Imagination with Diffusion Models}, 
      author={Xuefeng Du and Yiyou Sun and Xiaojin Zhu and Yixuan Li },
      booktitle={Advances in Neural Information Processing Systems},
      year = {2023}
}