Awesome
SIUO
π Homepage | π arXiv | π€ Paper | π€ Dataset | GitHub
This repo contains the evaluation code for the paper "Cross-Modality Safety Alignment"
πNews
- π[2024-06-12]: Exciting to share our new benchmark about cross-modality safety alignment Github!π
Introduction
As Artificial General Intelligence (AGI) becomes increasingly integrated into various facets of human life, ensuring the safety and ethical alignment of such systems is paramount. Previous studies primarily focus on single-modality threats, which may not suffice given the integrated and complex nature of cross-modality interactions. We introduce a novel safety alignment challenge called <em>Safe Inputs but Unsafe Output (SIUO)</em> to evaluate cross-modality safety alignment. Specifically, it considers cases where single modalities are safe independently but could potentially lead to unsafe or unethical outputs when combined. To empirically investigate this problem, we developed the SIUO, a cross-modality benchmark encompassing 9 critical safety domains, such as self-harm, illegal activities, and privacy violations. Our findings reveal substantial safety vulnerabilities in both closed- and open-source LVLMs, such as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably interpret and respond to complex, real-world scenarios.
𧩠Dataset Creation
SIUO was created to challenge LVLMs with tasks that demand college-level subject knowledge and deliberate reasoning, pushing the boundaries of what these models can achieve in terms of expert-level perception and reasoning.
β¨ Dataset Examples
<img src="static/images/example2.jpg" class="floatpic">πͺ Download
You can find the generation task questions in data/siuo_gen.json
and the multiple-choice questions in data/siuo_mcqa.json
.
Images can be downloaded from Google Drive or Baidu Netdisk(code: siuo
). You will obtain a zip file(SIUO-images.zip
). Please unzip the images under repo_root_dir/data/images
.
π― Evaluation
SIUO is a zero-shot evaluation benchmark. The model response generation process can be referred to in the following files: generationβgpt4o-gen.py
and generation-gpt4o-mcqa.py
.
π€ GPT-Eval
-
Put your model's three responses as a list and add them under the "responses" field in
siuo_gen.json
, saved in./eval/test_results/siuo_gen-{model_name}.json
. -
For the safety evaluation, run
python gpt-eval.py --model model_name --mode safe
to obtain./eval/test_results/siuo_gen-{model_name}-gpteval-safe-pro.json
.For the effectiveness evaluation, run
python gpt-eval.py --model model_name --mode effective
to obtain./eval/test_results/siuo_gen-{model_name}-gpteval-effective-pro.json
.Note: Before running the
gpt-eval.py
file, you need to add your OpenAI API key. -
Then run the file
python gpt-score.py
, which will merge the safe and effective files, calculate the values for safe, effective, and safe & effective scores. It will also statistically analyze the scores by category.
π€Ή Multiple-Choice QA
-
Add the model response results under the "response_mcqa" field in
siuo_mcqa.json
, saved in./eval/test_results/siuo_mcqa-{model_name}.json
. -
Then run the file
python mcqa-eval.py --model model_name
to match the model responses and calculate accuracy.
βοΈ Data Contribution
Anyone is welcome to contribute test cases to our dataset by submitting them through the Google Form. After a review by our team, we will incorporate the approved cases into the evaluation dataset. We greatly appreciate your participation in helping us advance cross-modality safety alignment, enabling a more comprehensive evaluation of these models.
π¬ Contact
- Siyin Wang: siyinwang20@fudan.edu.cn
- Xipeng Qiu: xpqiu@fudan.edu.cn
π Papers citing(utilizing) SIUO
Welcome to cite/use our dataset to further advance cross-modality safety alignment.
π License
Usage and License Notices: The dataset is intended and licensed for research use only. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models using the dataset should not be used outside of research purposes.
π Citation
BibTeX:
@article{wang2024cross,
title={Cross-Modality Safety Alignment},
author={Siyin Wang and Xingsong Ye and Qinyuan Cheng and Junwen Duan and Shimin Li and Jinlan Fu and Xipeng Qiu and Xuanjing Huang},
journal={arXiv preprint arXiv:2406.15279},
year={2024},
url={https://arxiv.org/abs/2406.15279},
archivePrefix={arXiv},
eprint={2406.15279},
primaryClass={cs.AI},
}