Awesome

MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?

Code for the Paper "MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?".

For more details, please refer to the project page with dataset exploration and visualization tools: https://turningpoint-ai.github.io/MOSSBench/.

:bell: If you have any questions or suggestions, please don't hesitate to let us know. You can comment on the Twitter, or post an issue on this repository.

[Webpage] [Paper] [Huggingface Dataset] [Visualization] [Result Explorer] [Twitter]

Outlines

• 💥 News 💥
• 👀 About MOSSBench
• 🏆 Leaderboard 🏆
• Contributing the Leaderboard
• 📊 Dataset Examples
• 📖 Dataset Usage
  • Data Downloading
  • Data Format
  • Data Visualization
• 🔮 Evaluations on MOSSBench
  • Requirements
  • Evaluation Pipelines
• 📜 License
• ☕ Stay Connected!
• ✅ Cite

💥 News 💥

• [2024.06.22] Our paper is now accessible at ArXiv.

👀 About MOSSBench

Humans are prone to cognitive distortions — biased thinking patterns that lead to exaggerated responses to specific stimuli, albeit in very different contexts. This paper demonstrates that advanced MLLMs exhibit similar tendencies. While these models are designed to respond queries under safety mechanism, they sometimes reject harmless queries in the presence of certain visual stimuli, disregarding the benign nature of their contexts.

As the initial step in investigating this behavior, we identify three types of stimulus that trigger the oversensitivity of existing MLLMs: Exaggerated Risk, Negated Harm, and Counterintuitive Interpretation. To systematically evaluate MLLMs' oversensitivity to these stimuli, we propose the Multimodal OverSenSitivity Benchmark Logo (MOSSBench). This toolkit consists of 300 manually collected benign multimodal queries, cross-verified by third-party reviewers (AMT).

Empirical studies using Logo MOSSBench on 20 MLLMs reveal several insights: (1). Oversensitivity is prevalent among SOTA MLLMs, with refusal rates reaching up to 76% for harmless queries. (2). Safer models are more oversensitive: increasing safety may inadvertently raise caution and conservatism in the model's responses. (3). Different types of stimuli tend to cause errors at specific stages — perception, intent reasoning, and safety decision-making — in the response process of MLLMs. These findings highlight the need for refined safety mechanisms that balance caution with contextually appropriate responses, improving the reliability of MLLMs in real-world applications.

For more details, you can find our project page here and our paper here.

🏆 Leaderboard 🏆

Contributing the Leaderboard

🚨🚨 The leaderboard is continuously being updated.

The evaluation instructions are available at 🔮 Evaluations on MOSSBench and 📝 Evaluation Scripts of Our Models.

To submit your results to the leaderboard, please send to this email with your result file (we will generate the score file for you), referring to the template file below:

• output_test_template_for_leaderboard_submission.json

Oversensitivity on MOSSBench

Refusal Rate of mllms:

📊 Dataset Examples

Examples of 3 types of oversensitivity stimuli:

1. Exaggerated Risk

2. Negated Harm

3. Counterintuitive Interpretation

📖 Dataset Usage

Data Downloading

You can download this dataset by the following command (make sure that you have installed Huggingface Datasets):

from datasets import load_dataset

dataset = load_dataset("AIcell/MOSSBench", "oversensitivity")

Here are some examples of how to access the downloaded dataset:

# print the first example on the testmini set
print(dataset["train"][0])
print(dataset["train"][0]['pid']) # print the problem id 
print(dataset["train"][0]['question']) # print the question text 
print(dataset["train"][0]['image']) # print the image path
dataset["train"][0]['decoded_image'] # display the image

Data Format

The dataset is provided in json format and contains the following attributes:

{
    "image": [string] A file path pointing to the associated image,
    "short description": [string] An oracle short description of the associated image,
    "question": [string] A query regarding to the image, 
    "pid": [string] Problem ID, e.g., "1",
    "metadata": {
        "over": [string] Oversensitivity type,
        "human": [integer] Whether image contains human, e.g. 0 or 1,
        "child": [integer] Whether image contains child, e.g. 0 or 1,
        "syn": [integer] Whether image is synthesized, e.g. 0 or 1,
        "ocr": [integer] Whether image contains ocr, e.g. 0 or 1,
        "harm": [integer] Which harm type the query belongs to, 0-7,
    }
}

Data Visualization

🎰 You can explore the dataset in an interactive way here.

🔮 Evaluations on MOSSBench

Requirements

Install the Python dependencies if you would like to reproduce our results for ChatGPT, GPT-4, Claude-2, and Bard:

pip install -r requirements.txt

Evaluation Pipelines

Step 1. Prepare your MLLM

For proprietary MLLMs

Get your models API ready in following links

• OpenAI API key
• Claude API Key
• Gemini API Key

and store them under foler path_to_your_code/api_keys/[model].text. Please replace the [model] by anthropic_keys, google_keys and openai_keys.

For open-source MLLMs

Download your model or get their names for Huggingface. And replace the following path by where you locate your models or your models name.

# Initialize variables
MODEL_NAME="your_path_to/idefics-9b-instruct" # you can replace it by direct naming
DATA_DIR=""

Step 2. Run evaluation (main.py) Next, run experiments/main.py file in folder or excute the .sh files we provide for evaluation by

cd experiments/scripts

bash run_instructblip.sh

📜 License

The new contributions to our dataset are distributed under the CC BY-SA 4.0 license, including

• The creation of contrasting and oversensitivity dataset: IQTest, FunctionQA, and Paper;

• The filtering and cleaning of source datasets;

• The standard formalization of instances for evaluation purposes;

• The annotations of metadata.

• Purpose: The dataset was primarily designed for use as a test set.

• Commercial Use: The dataset can be used commercially as a test set, but using it as a training set is prohibited. By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY-SA 4.0 license.

:coffee: Stay Connected!

We are always open to engaging discussions, collaborations, or even just sharing a virtual coffee. To get in touch or join our team, visit TurningPoint AI's homepage for contact information.

:white_check_mark: Cite

If you find MOSSBench useful for your your research and applications, please kindly cite using this BibTeX:

@misc{li2024mossbenchmultimodallanguagemodel,
      title={MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?}, 
      author={Xirui Li and Hengguang Zhou and Ruochen Wang and Tianyi Zhou and Minhao Cheng and Cho-Jui Hsieh},
      year={2024},
      eprint={2406.17806},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.17806}, 
}

MOSSBench Website

MOSSBench website is adapted from Nerfies website and MathVista website.

Website License

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.