Awesome

🛡️Awesome-Jailbreak-against-Multimodal-Generative-Models

🤗Introduction

Welcome to our Awesome-Jailbreak-against-Multimodal-Generative-Models! We provides a comprehensive overview of jailbreak vulnerabilities in multimodal generative models! 🥰🥰🥰<br>

If a resource is relevant to multiple subcategories, we place it under each applicable section.<br>

🧑‍💻 Our Work

• Leveraging the layered structure of generative models, we systematically examine jailbreak attacks and corresponding defense strategies across the input, encoding, decoding, and output layers.<br>
• We establish a detailed taxonomy of attack vectors, defense mechanisms, and evaluation frameworks specific to multimodal generative models.<br>
• Our review encompasses a wide array of input-output configurations, offering a nuanced examination of jailbreak tactics and defenses applicable to any-to-text, any-to-vision, and any-to-any modalities within generative systems.<br>

✔️ Perfect for Majority

• For beginners curious about jailbreak attack and defense, our repository serves as a compass for grasping the big picture and diving into the details. The brief introductions to papers in different fields retained in the README provide a beginner-friendly navigation through interesting directions in the field;
• For seasoned researchers, this repository is a tool to keep you informed and fill any gaps in your knowledge. Within each subtopic, we are diligently updating all the latest content and continuously backfilling with previous work. Our thorough compilation and careful selection are time-savers for you.

🧭 How to Use this Guide

• Quick Start: In the README, users can find a curated list of select information, along with links to various consultations.
• In-Depth Exploration: If you have a special interest in a particular area of the paper, delve into the markdown file for more information.

🚀Table of Contents

• 🛡️Awesome-Jailbreak-against-Multimodal-Generative-Models🛡️
  • 🤗Introduction
  • 🚀Table of Contents
  • 🔥Multimodal Generative Models
    • 📑Any-to-Text (LLM Backbone)
    • 📖Any-to-Vision (Diffusion Backbone)
    • 📰Any-to-Any (Unified Backbone)
  • 😈Jailbreak Attack
    • 📖Attack-Intro
    • 📑Papers
  • 🛡️Jailbreak Defense
    • 📖Defense-Intro
    • 📑Papers
  • 💯Resources
    • ⭐️Datasets
      • Used to Any-to-Text Models
      • Used to Any-to-Vision Models
    • 📚Detectors
      • Used to Any-to-Text Models
      • Used to Any-to-Text Models

🔥Multimodal Generative Models

Below are tables of model short name and representative generative models used for jailbreak. For input/output modalities, I: Image, T: Text, V: Video, A: Audio.

📑Any-to-Text (LLM Backbone)

📖Any-to-Vision (Diffusion Backbone)

📰Any-to-Any (Unified Backbone)

😈JailBreak Attack

📖Attack-Intro

In this part, we focus on discussing different advanced jailbreak attacks against multimodal models. We categorize attack methods into black-box, gray-box, and white-box attacks. in a black-box setting where the model is inaccessible to the attacker, the attack is limited to surface-level interactions, focusing solely on the model’s input and/or output. Regarding gray-box and white-box attacks, we consider model-level attacks, including attacks at both the encoder and generator.

As shown in Fig. A.1, attackers are compelled to develop more sophisticated input templates across prompt engineering, image engineering, and role-ploy techniques. These techniques can bypass the model’s safeguards, making the models more susceptible to executing prohibited instructions. <br> As shown in Fig. A.2, attackers focus on querying outputs across multiple input variants. Driven by specific adversarial goals, attackers employ estimation-based and search-based attack techniques to iteratively refine these input variants.

As shown in Fig. B.1, attackers are restricted to accessing only the encoders to provoke harmful responses. In this case, attackers typically seek to maximize cosine similarity within the latent space, ensuring the adversarial input retains similar semantics to the target malicious content while still being classified as safe. <br> As shown in Fig. B.2, attackers have unrestricted access to the generative model’s architecture and checkpoint, enabling attackers to conduct thorough investigations and manipulations, thus enabling sophisticated attacks.

📑Papers

Below are the papers related to jailbreak attacks.

Jailbreak Attack of Any-to-Text Models

Jailbreak Attack of Any-to-Vision Models

Jailbreak Attack of Any-to-Any Models

🛡️Jailbreak Defense

📖Defense-Intro

In order to cope with jailbreak attacks and improve the security of multimodal foundation models, existing work makes efforts in both Transformative defense and Discriminative defense.

Discriminative defenses focus on identifying and analyzing varying classified cues, such as statistical information at the input level, embeddings at the encoder level, activations at the generator level, and response discrepancies at the output level.

Transformative defenses that can operate at four levels to influence the model’s generation process, ensuring benign responses even in the presence of adversarial or malicious prompts.

📑Papers

Below are the papers related to jailbreak defense.

Jailbreak Defense of Any-to-Text Models

Jailbreak Defense of Any-to-Vision Models

Jailbreak Defense of Any-to-Any Models

💯Resources

⭐️Datasets

Used to Any-to-Text Models

Used to Any-to-Vision Models

📚Detectors

Detector-based approaches utilize pre-trained classifiers to automatically detect and identify harmful content within generated outputs. These classifiers are trained on large, annotated datasets that cover a range of unsafe categories, such as toxicity, violence, or explicit material. By leveraging these pre-trained models, detector-based methods can efficiently flag inappropriate content.

Used to Any-to-Text Models

Used to Any-to-Vision Models