Home

Awesome

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (ICCV 2023)

<img src="./assets/teaser.png" alt="teaser image" width="500"/>

Visual Reasoning

<img src="./assets/skills.png" alt="skill image" width="1200"/>

Please see ./paintskills for our DETR-based visual reasoning skill evaluation.

(Optional) Please see https://github.com/aszala/PaintSkills-Simulator for our 3D Simulator implementation.

Social Bias

<img src="./assets/bias_experiment_flow.png" alt="bias exp image" width="500"/>

Please see ./biases for our social (gender and skin tone) bias evaluation.

Image Quality & Image-Text Alignment

<img src="./assets/alignment_quality.png" alt="alignment and quality image" width="500"/>

Please see ./quality for our image quaity evaluation based on FID score.

Please see ./retrieval for our image-text alignment evaluation with CLIP-based R-precision.

Please see ./captioning for our image-text alignment evaluation with VL-T5 captioning.

Models

We provide inference scripts for DALLE-small (DALLE-pytorch), minDALL-E, X-LXMERT, and Stable Diffusion.

Acknowledgments

We thank the developers of DETR, DALLE-pytorch, minDALL-E, X-LXMERT, and Stable Diffusion for their public code release.

Reference

Please cite our paper if you use our dataset in your works:

@inproceedings{Cho2023DallEval,
  title         = {DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models},
  author        = {Jaemin Cho and Abhay Zala and Mohit Bansal},
  year          = {2023},
  booktitle     = {ICCV},
}