Awesome

Generative AI Risk Management Resources

TL; DR:

• This repository contains resources to assist in the creation of technical standards or procedures for organizational AI governance policies, with a strong focus on generative AI (GAI).
• This information must be combined with a higher-level governance approach as described in, e.g., The Interagency Guidance on Model Risk Management or the NIST AI RMF Govern Function.
• The information below is aligned with DRAFT NIST 600-1 AI RMF Generative AI Profile.
• For non-commercial use. For commercial support please reach out to info@hallresearch.ai.

What's missing?

• Higher-level policies and procedure language to tie these resources together into cohesive governance documentments.
• Methodology for estimating business risk (e.g., monetary losses) from model testing, red-teaming, feedback and experimental results.
• ...

Introduction

(c) Patrick Hall and Daniel Atherton 2024, CC BY 4.0

This information is designed to help organizations build the governance policies required to measure and manage risks associated with deploying and using GAI systems. Governance is key to addressing the growing need for trustworthy and responsible AI systems, and this repository is aligned to the NIST AI Risk Management Framework trustworthy characteristics and the DRAFT NIST 600-1 AI RMF Generative AI Profile. Governance is also a necessary component of AI strategy, crucial for addressing real legal, regulatory, ethical, and operational headwinds.

At its core, this repository provides technical materials for building or augmenting detailed model or AI governance procedures or standards, and aligns them to guidance from NIST. Starting in Section A, two central risk management mechanisms are explored. The first perspective comprises the NIST AI RMF trustworthy characteristics mapped to GAI risks. Operating from this perspective allows organizations to understand how each trustworthy characteristic can mitigate specific risks posed by GAI. The second perspective is the reverse—GAI risks mapped to trustworthy characteristics. That mapping can help organizations understand which characteristics should be prioritized to manage specific GAI risks. As consumer finance organizations are likely to adopt both NIST (or other more technical frameworks) and traditional enterprise risk management methodologies, ideas on linking trustworthy characteristics, GAI risks, and established banking risk buckets are also presented in Section A.

The repository also guides users through authoritative resources for risk-tiering. Sections B.1 through B.7 walk the user of the framework through the process of defining adverse impacts: Harm to Operations, Harm to Assets, Harm to Individuals, Harm to Other Organizations, and Harm to the Nation, along with guidance on impact quantification and description. Section B also offers tables with guidance on assessing the likelihood of certain risks. Organizations and companies can leverage this combination of adverse impacts and frequency/likelihood tables to develop tailored risk tiers that reflect the specific contexts in which their GAI systems may be operating. They can also utilize practical risk-tiering to guide their decision-making and evaluate how best to calibrate existing safeguards or whether to implement additional ones.

Measurement and testing is a critical aspect of ensuring GAI systems perform as expected. For measuring the severity of certain GAI risks, Section C presents various model testing benchmarks (such as evals). Model testing suites provide the user with tools to roughly assess GAI performance against trustworthy characteristics as well to quickly test for resilience in the face of known GAI risks. As GAI systems are vulnerable to adversarial attacks via prompting and hacks, Section D presents red-teaming and adversarial prompting approaches for human elicitation of evidence of GAI risks in adversarial scenarios. Section H hints at more in-depth structured experiments and human feedback for risk assessment. Suggested usage for these types of measurement is as follows:

• Low-risk GAI systems: model testing only
• Medium-risk GAI systems: model testing and red-teaming
• High-risk GAI systems: model testing, red-teaming, and structured experiments and human feedback

Where measurement for lower-risk systems can be highly-automated, human risk management resources are reserved for medium and high-risk systems.

For managing and mitigating GAI risks, Section E outlines several risk controls for GAI. Controls range from technical settings for GAI systems to commonsense recommendations, e.g., limiting or restricting access for minors. Sections F, G](#g-example-medium-risk-generative-ai-measurement-and-management-plan), and H pair risk measurement techniques with controls to form more fulsome risk management plans. Recommended usage for the plans in Sections F-H is:

• Low-risk GAI systems: apply Section F only
• Medium-risk GAI systems: apply Section F and G
• High-risk GAI systems: apply Sections F, G, and H

Regardless of the risk level of the system, the framework offers detailed measurement plans that guide the user through the process of assessing the system’s performance, along with tracking risks, and harmonizing the system with trustworthy AI principles.

Table of Contents

• Section A: Example Generative AI-Trustworthy Characteristic Crosswalk

  • A.1: Trustworthy Characteristic to Generative AI Risk Crosswalk
  • A.2: Generative AI Risk to Trustworthy Characteristic Crosswalk
  • A.3: Traditional Banking Risks, Generative AI Risks and Trustworthy Characteristics Crosswalk

• Section B: Example Risk-tiering Materials for Generative AI

  • B.1: Example Adverse Impacts
  • B.2: Example Impact Descriptions
  • B.3: Example Likelihood Descriptions
  • B.4: Example Risk Tiers
  • B.5: Example Risk Descriptions
  • B.6: Practical Risk-tiering Questions
  • B.7: AI Risk Management Framework Actions Aligned to Risk Tiering

• Section C: List of Selected Model Testing Suites

  • C.1: Selected Model Testing Suites Organized by Trustworthy Characteristic
  • C.2: Selected Model Testing Suites Organized by Generative AI Risk
  • C.3: AI Risk Management Framework Actions Aligned to Benchmarking

• Section D: Selected Adversarial Prompting Strategies and Attacks

  • D.1: Common AI Red-teaming Tools
  • D.2: Selected Adversarial Prompting Strategies and Attacks by Organized Trustworthy Characteristic
  • D.3: Selected Adversarial Prompting Techniques and Attacks Organized by Generative AI Risk
  • D.4: AI Risk Management Framework Actions Aligned to Red Teaming

• Section E: Selected Risk Controls for Generative AI

• Section F: Example Low-risk Generative AI Measurement and Management Plan

  • F.1: Example Low-risk Generative AI Measurement and Management Plan Organized by Trustworthy Characteristic
  • F.2: Example Low-risk Generative AI Measurement and Management Plan Organized by Generative AI Risk

• Section G: Example Medium-risk Generative AI Measurement and Management Plan

  • G.1: Example Medium-risk Generative AI Measurement and Management Plan Organized by Trustworthy Characteristic
  • G.2: Example Medium-risk Generative AI Measurement and Management Plan Organized by Generative AI Risk

• Section H: Example High-risk Generative AI Measurement and Management Plan

  • H.1: Example High-risk Generative AI Measurement and Management Plan Organized by Trustworthy Characteristic
  • H.2: Example High-risk Generative AI Measurement and Management Plan Organized by Generative AI Risk

A: Example Generative AI-Trustworthy Characteristic Crosswalk

A.1: Trustworthy Characteristic to Generative AI Risk Crosswalk

Usage Note: Table A.1 provides an example of mapping GAI risks onto AI RMF trustworthy characteristics. Mapping GAI risks to AI RMF trustworthy characteristics can be particularly useful when existing policies, processes, or controls can be applied to manage GAI risks, but have been previously implemented in alignment with the AI RMF trustworthy characteristics. Many mappings are possible. Mappings that differ from the example may be more appropriate to meet a particular organization's risk management goals.

A.2: Generative AI Risk to Trustworthy Characteristic Crosswalk

Usage Note: Table A.2 provides an example of mapping AI RMF trustworthy characteristics onto GAI risks. Mapping AI RMF trustworthy characteristics to GAI risks can assist organizations in aligning GAI guidance to existing AI/ML policies, processes, or controls or to extend GAI guidance to address additional AI/ML technologies. Many mappings are possible. Mappings that differ from the example may be more appropriate to meet a particular organization's risk management goals.

A.3: Traditional Banking Risks, Generative AI Risks and Trustworthy Characteristics Crosswalk

Usage Note: Table A.3 provides an example of mapping GAI risks and AI RMF trustworthy characteristics. This type of mapping can enable incorporation of new AI guidance into existing policies, processes, or controls or the application of existing policies, processes, or controls to newer AI risks.

B: Example Risk-tiering Materials for Generative AI

B.1: Example Adverse Impacts

B.2 Example Impact Descriptions

B.3 Example Likelihood Descriptions

B.4 Example Risk Tiers

B.5 Example Risk Descriptions

B.6: Practical Risk-tiering Questions

B.6.1: Confabulation: How likely are system outputs to contain errors? What are the impacts if errors occur?

B.6.2: Dangerous and Violent Recommendations: How likely is the system to give dangerous or violent recommendations? What are the impacts if it does?

B.6.3: Data Privacy: How likely is someone to enter sensitive data into the system? What are the impacts if this occurs? Are standard data privacy controls applied to the system to mitigate potential adverse impacts?

B.6.4: Human-AI Configuration: How likely is someone to use the system incorrectly or abuse it? How likely is use for decision-making? What are the impacts of incorrect use or abuse? What are the impacts of invalid or unreliable decision-making?

B.6.5: Information Integrity: How likely is the system to generate deepfakes or mis or disinformation? At what scale? Are content provenance mechanisms applied to system outputs? What are the impacts of generating deepfakes or mis or disinformation? Without controls for content provenance?

B.6.6: Information Security: How likely are system resources to be breached or exfiltrated? How likely is the system to be used in the generation of phishing or malware content? What are the impacts in these cases? Are standard information security controls applied to the system to mitigate potential adverse impacts?

B.6.7: Intellectual Property: How likely are system outputs to contain other entities' intellectual property? What are the impacts if this occurs?

B.6.8: Toxicity, Bias, and Homogenization: How likely are system outputs to be biased, toxic, homogenizing or otherwise obscene? How likely are system outputs to be used as subsequent training inputs? What are the impacts of these scenarios? Are standard nondiscrimination controls applied to mitigate potential adverse impacts? Is the application accessible to all user groups? What are the impacts if the system is not accessible to all user groups?

B.6.9: Value Chain and Component Integration: Are contracts relating to the system reviewed for legal risks? Are standard acquisition/procurement controls applied to mitigate potential adverse impacts? Do vendors provide incident response with guaranteed response times? What are the impacts if these conditions are not met?

B.7: AI Risk Management Framework Actions Aligned to Risk Tiering

GOVERN 1.3, GOVERN 1.5, GOVERN 2.3, GOVERN 3.2, GOVERN 4.1, GOVERN 5.2, GOVERN 6.1, MANAGE 1.2, MANAGE 1.3, MANAGE 2.1, MANAGE 2.2, MANAGE 2.3, MANAGE 2.4, MANAGE 3.1, MANAGE 3.2, MANAGE 4.1, MAP 1.1, MAP 1.5, MEASURE 2.6

Usage Note: Materials in Section B can be used to create or update risk tiers or other risk assessment tools for GAI systems or applications as follows:

• Table B.1 can enable mapping of GAI risks and impacts.

• Table B.2 can enable quantification of impacts for risk tiering or risk assessment.

• Table B.3 can enable quantification of likelihood for risk tiering or risk assessment.

• Table B.4 presents an example of combining assessed impact and likelihood into risk tiers.

• Table B.5 presents example risk tiers with associated qualitative, semi-quantitative, and quantitative values for risk tiering or risk assessment.

• Subsection B.6 presents example questions for qualitative risk assessment.

• Subsection B.7 highlights subcategories to indicate alignment with the AI RMF.

C: List of Selected Model Testing Suites

C.1: Selected Model Testing Suites Organized by Trustworthy Characteristic

Adapted from <a href="#ai-verify-foundation-and-infocomm-media-development-authority-cataloguing-llm-evaluations-draft-for-discussion-october-2023-httpsaiverifyfoundationsgdownloadscataloguing_llm_evaluationspdf">[AI Verify Foundation]</a> Taxonimization and various additional resources.

Accountable and Transparent
An Evaluation on Large Language Model Outputs: Discourse and Memorization (see Appendix B) <a href="#de-wynter-adrian-xun-wang-alex-sokolov-qilong-gu-and-si-qing-chen-an-evaluation-on-large-language-model-outputs-discourse-and-memorization-natural-language-processing-journal-4-september-2023-100024-httpsdoiorg101016jnlp2023100024">[De Wynter et al.]</a>
Big-bench: Truthfulness <a href="#srivastava-aarohi-abhinav-rastogi-abhishek-rao-abu-awal-md-shoeb-abubakar-abid-adam-fisch-adam-r-brown-adam-santoro-et-al-beyond-the-imitation-game-quantifying-and-extrapolating-the-capabilities-of-language-models-arxiv-preprint-last-revised-june-12-2023-httpsdoiorg1048550arxiv220604615">[Srivastava et al.]</a>
DecodingTrust: Machine Ethics <a href="#wang-boxin-weixin-chen-hengzhi-pei-chulin-xie-mintong-kang-chenhui-zhang-chejian-xu-zidi-xiong-ritik-dutta-rylan-schaeffer-et-al-decodingtrust-a-comprehensive-assessment-of-trustworthiness-in-gpt-models-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-1361-3123231339-published-may-30-2024-httpsdlacmorgdoi10555536661223667483">[Wang et al.]</a>
Evaluation Harness: ETHICS <a href="#gao-leo-jonathan-tow-baber-abbasi-stella-biderman-sid-black-anthony-dipofi-charles-foster-laurence-golding-jeffrey-hsu-alain-le-noach-haonan-li-kyle-mcdonell-niklas-muennighoff-chris-ociepa-jason-phang-laria-reynolds-hailey-schoelkopf-aviya-skowron-lintang-sutawika-eric-tang-anish-thite-ben-wang-kevin-wang-and-andy-zou-a-framework-for-few-shot-language-model-evaluation-github-repository-accessed-september-19-2024-httpsgithubcomeleutherailm-evaluation-harness">[Gao et al.]</a>
HELM: Copyright <a href="#bommasani-rishi-percy-liang-and-tony-lee-holistic-evaluation-of-language-models-annals-of-the-new-york-academy-of-sciences-1525-no-1-july-2023-140146-httpsdoiorg101111nyas15007">[Bommasani et al.]</a>
Mark My Words <a href="#piet-julien-chawin-sitawarin-vivian-fang-norman-mu-and-david-wagner-mark-my-words-analyzing-and-evaluating-language-model-watermarks-arxiv-preprint-last-revised-december-7-2023-httpsdoiorg1048550arxiv231200273">[Piet et al.]</a>

Fair with Harmful Bias Managed
BELEBELE <a href="#bandarkar-lucas-davis-liang-benjamin-muller-mikel-artetxe-satya-narayan-shukla-donald-husa-naman-goyal-abhinandan-krishnan-luke-zettlemoyer-and-madian-khabsa-the-belebele-benchmark-a-parallel-reading-comprehension-dataset-in-122-language-variants-arxiv-preprint-last-revised-july-25-2024-httpsdoiorg1048550arxiv230816884">[Bandarkar et al.]</a>
Big-bench: Low-resource language, Non-English, Translation
Big-bench: Social bias, Racial bias, Gender bias, Religious bias
Big-bench: Toxicity
DecodingTrust: Fairness
DecodingTrust: Stereotype Bias
DecodingTrust: Toxicity
C-Eval (Chinese evaluation suite) <a href="#huang-yuzhen-yuzhuo-bai-zhihao-zhu-junlei-zhang-jinghan-zhang-tangjun-su-junteng-liu-chuancheng-lv-yikai-zhang-jiayi-lei-yao-fu-maosong-sun-and-junxian-he-c-eval-a-multi-level-multi-discipline-chinese-evaluation-suite-for-foundation-models-arxiv-preprint-last-revised-november-6-2023-httpsdoiorg1048550arxiv230508322">[Huang, Yuzhen et al.]</a>
Evaluation Harness: CrowS-Pairs
Evaluation Harness: ToxiGen
Finding New Biases in Language Models with a Holistic Descriptor Dataset <a href="#smith-eric-michael-melissa-hall-melanie-kambadur-eleonora-presani-and-adina-williams-im-sorry-to-hear-that-finding-new-biases-in-language-models-with-a-holistic-descriptor-dataset-arxiv-preprint-last-revised-october-27-2022-httpsdoiorg1048550arxiv220509209">[Smith et al.]</a>
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models <a href="#feng-shangbin-chan-young-park-yuhan-liu-and-yulia-tsvetkov-from-pretraining-data-to-language-models-to-downstream-tasks-tracking-the-trails-of-political-biases-leading-to-unfair-nlp-models-arxiv-preprint-last-revised-july-6-2023-httpsdoiorg1048550arxiv230508283">[Feng et al.]</a>
HELM: Bias
HELM: Toxicity
MT-bench <a href="#zheng-lianmin-wei-lin-chiang-ying-sheng-siyuan-zhuang-zhanghao-wu-yonghao-zhuang-zi-lin-zhuohan-li-dacheng-li-and-eric-p-xing-judging-llm-as-a-judge-with-mt-bench-and-chatbot-arena-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-2020-4659546623-published-may-30-2024-httpsdlacmorgdoi10555536661223668142">[Zheng et al.]</a>
The Self-Perception and Political Biases of ChatGPT <a href="#rutinowski-jérôme-sven-franke-jan-endendyk-ina-dormuth-moritz-roidl-and-markus-pauly-the-self-perception-and-political-biases-of-chatgpt-human-behavior-and-emerging-technologies-2024-httpsdoiorg10115520247115633">[Rutinowski et al.]</a>
Towards Measuring the Representation of Subjective Global Opinions in Language Models <a href="#durmus-esin-karina-nguyen-thomas-i-liao-nicholas-schiefer-amanda-askell-anton-bakhtin-carol-chen-zac-hatfield-dodds-et-al-towards-measuring-the-representation-of-subjective-global-opinions-in-language-models-arxiv-preprint-last-revised-april-12-2024-httpsdoiorg1048550arxiv230616388">[Durmus et al.]</a>

Privacy Enhanced
HELM: Copyright
llmprivacy <a href="#staab-robin-mark-vero-mislav-balunović-and-martin-vechev-beyond-memorization-violating-privacy-via-inference-with-large-language-models-arxiv-preprint-last-revised-may-6-2024-httpsdoiorg1048550arxiv231007298">[Staab et al.]</a>
mimir <a href="#duan-michael-anshuman-suri-niloofar-mireshghallah-sewon-min-weijia-shi-luke-zettlemoyer-yulia-tsvetkov-yejin-choi-david-evans-and-hannaneh-hajishirzi-do-membership-inference-attacks-work-on-large-language-models-arxiv-preprint-last-revised-september-16-2024-httpsdoiorg1048550arxiv240207841">[Duan et al.]</a>

Safe
Big-bench: Convince Me
Big-bench: Truthfulness <a href="#srivastava-aarohi-abhinav-rastogi-abhishek-rao-abu-awal-md-shoeb-abubakar-abid-adam-fisch-adam-r-brown-adam-santoro-et-al-beyond-the-imitation-game-quantifying-and-extrapolating-the-capabilities-of-language-models-arxiv-preprint-last-revised-june-12-2023-httpsdoiorg1048550arxiv220604615">[Srivastava et al.]</a>
HELM: Reiteration, Wedging
Mark My Words <a href="#piet-julien-chawin-sitawarin-vivian-fang-norman-mu-and-david-wagner-mark-my-words-analyzing-and-evaluating-language-model-watermarks-arxiv-preprint-last-revised-december-7-2023-httpsdoiorg1048550arxiv231200273">[Piet et al.]</a>
MLCommons <a href="#vidgen-bertie-adarsh-agrawal-ahmed-m-ahmed-victor-akinwande-namir-al-nuaimi-najla-alfaraj-elie-alhajjar-et-al-introducing-v05-of-the-ai-safety-benchmark-from-mlcommons-arxiv-preprint-last-revised-may-13-2024-httpsdoiorg1048550arxiv240412241">[Vidgen et al.]</a>
The WMDP Benchmark <a href="#li-nathaniel-alexander-pan-anjali-gopal-summer-yue-daniel-berrios-alice-gatti-justin-d-li-ann-kathrin-dombrowski-et-al-the-wmdp-benchmark-measuring-and-reducing-malicious-use-with-unlearning-arxiv-preprint-last-revised-may-15-2024-httpsdoiorg1048550arxiv240303218">[Li et al.]</a>

Secure and Resilient
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation <a href="#huang-yangsibo-samyak-gupta-mengzhou-xia-kai-li-and-danqi-chen-catastrophic-jailbreak-of-open-source-llms-via-exploiting-generation-iclr-2024-spotlight-published-january-16-2024-last-modified-march-15-2024-httpsopenreviewnetforumidr42tsschph">[Huang, Yangsibo et al.]</a>
detect-pretrain-code <a href="#shi-weijia-anirudh-ajith-mengzhou-xia-yangsibo-huang-daogao-liu-terra-blevins-danqi-chen-and-luke-zettlemoyer-detecting-pretraining-data-from-large-language-models-arxiv-preprint-last-revised-march-9-2024-httpsdoiorg1048550arxiv231016789">[Shi et al.]</a>
Garak: encoding, knownbadsignatures, malwaregen, packagehallucination, xss <a href="#derczynski-leon-erick-galinkin-jeffrey-martin-subho-majumdar-and-nanna-inie-garak-a-framework-for-security-probing-large-language-models-arxiv-preprint-submitted-june-16-2024-httpsdoiorg1048550arxiv240611036">[Derczynski et al.]</a>
In-The-Wild Jailbreak Prompts on LLMs <a href="#shen-xinyue-zeyuan-chen-michael-backes-yun-shen-and-yang-zhang-do-anything-now-characterizing-and-evaluating-in-the-wild-jailbreak-prompts-on-large-language-models-arxiv-preprint-last-revised-may-15-2024-httpsdoiorg1048550arxiv230803825">[Shen et al.]</a>
JailbreakingLLMs <a href="#chao-patrick-alexander-robey-edgar-dobriban-hamed-hassani-george-j-pappas-and-eric-wong-jailbreaking-black-box-large-language-models-in-twenty-queries-arxiv-preprint-last-revised-july-18-2024-httpsdoiorg1048550arxiv231008419">[Chao et al.]</a>
llmprivacy <a href="#staab-robin-mark-vero-mislav-balunović-and-martin-vechev-beyond-memorization-violating-privacy-via-inference-with-large-language-models-arxiv-preprint-last-revised-may-6-2024-httpsdoiorg1048550arxiv231007298">[Staab et al.]</a>
mimir
TAP: A Query-Efficient Method for Jailbreaking Black-Box LLMs <a href="#mehrotra-anay-manolis-zampetakis-paul-kassianik-blaine-nelson-hyrum-anderson-yaron-singer-and-amin-karbasi-tree-of-attacks-jailbreaking-black-box-llms-automatically-arxiv-preprint-last-revised-february-21-2024-httpsdoiorg1048550arxiv231202119">[Mehrotra et al.]</a>

Valid and Reliable
Big-bench: Algorithms, Logical reasoning, Implicit reasoning, Mathematics, Arithmetic, Algebra, Mathematical proof, Fallacy, Negation, Computer code, Probabilistic reasoning, Social reasoning, Analogical reasoning, Multi-step, Understanding the World
Big-bench: Analytic entailment, Formal fallacies and syllogisms with negation, Entailed polarity
Big-bench: Context Free Question Answering
Big-bench: Contextual question answering, Reading comprehension, Question generation
Big-bench: Morphology, Grammar, Syntax
Big-bench: Out-of-Distribution
Big-bench: Paraphrase
Big-bench: Sufficient information
Big-bench: Summarization
DecodingTrust: Out-of-Distribution Robustness, Adversarial Robustness, Robustness Against Adversarial Demonstrations
Eval Gauntlet: Reading comprehension <a href="#dohmann-jeremy-blazingly-fast-llm-evaluation-for-in-context-learning-databricks-mosaic-ai-research-february-2-2023-httpswwwdatabrickscomblogllm-evaluation-for-icl">[Dohmann]</a>
Eval Gauntlet: Commonsense reasoning, Symbolic problem solving, Programming
Eval Gauntlet: Language Understanding
Eval Gauntlet: World Knowledge
Evaluation Harness: BLiMP
Evaluation Harness: CoQA, ARC
Evaluation Harness: GLUE
Evaluation Harness: HellaSwag, OpenBookQA, TruthfulQA
Evaluation Harness: MuTual
Evaluation Harness: PIQA, PROST, MC-TACO, MathQA, LogiQA, DROP
FLASK: Logical correctness, Logical robustness, Logical efficiency, Comprehension, Completeness <a href="#ye-seonghyeon-doyoung-kim-sungdong-kim-hyeonbin-hwang-seungone-kim-yongrae-jo-james-thorne-juho-kim-and-minjoon-seo-flask-fine-grained-language-model-evaluation-based-on-alignment-skill-sets-arxiv-preprint-last-revised-april-14-2024-httpsdoiorg1048550arxiv230710928">[Ye et al.]</a>
FLASK: Readability, Conciseness, Insightfulness
HELM: Knowledge
HELM: Language
HELM: Text classification
HELM: Question answering
HELM: Reasoning
HELM: Robustness to contrast sets
HELM: Summarization
Hugging Face: Fill-mask, Text generation <a href="#hugging-face-evaluate-last-accessed-september-19-2024-httpshuggingfacecodocsevaluateindex">[Hugging Face]</a>
Hugging Face: Question answering
Hugging Face: Summarization
Hugging Face: Text classification, Token classification, Zero-shot classification
MASSIVE <a href="#fitzgerald-jack-christopher-hench-charith-peris-scott-mackie-kay-rottmann-ana-sanchez-aaron-nash-liam-urbach-et-al-massive-a-1m-example-multilingual-natural-language-understanding-dataset-with-51-typologically-diverse-languages-arxiv-preprint-last-revised-june-17-2022-httpsdoiorg1048550arxiv220408582">[FitzGerald et al.]</a>
MT-bench <a href="#zheng-lianmin-wei-lin-chiang-ying-sheng-siyuan-zhuang-zhanghao-wu-yonghao-zhuang-zi-lin-zhuohan-li-dacheng-li-and-eric-p-xing-judging-llm-as-a-judge-with-mt-bench-and-chatbot-arena-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-2020-4659546623-published-may-30-2024-httpsdlacmorgdoi10555536661223668142">[Zheng et al.]</a>

C.2: Selected Model Testing Suites Organized by Generative AI Risk

CBRN Information
Big-bench: Convince Me
Big-bench: Truthfulness <a href="#srivastava-aarohi-abhinav-rastogi-abhishek-rao-abu-awal-md-shoeb-abubakar-abid-adam-fisch-adam-r-brown-adam-santoro-et-al-beyond-the-imitation-game-quantifying-and-extrapolating-the-capabilities-of-language-models-arxiv-preprint-last-revised-june-12-2023-httpsdoiorg1048550arxiv220604615">[Srivastava et al.]</a>
HELM: Reiteration, Wedging
MLCommons <a href="#vidgen-bertie-adarsh-agrawal-ahmed-m-ahmed-victor-akinwande-namir-al-nuaimi-najla-alfaraj-elie-alhajjar-et-al-introducing-v05-of-the-ai-safety-benchmark-from-mlcommons-arxiv-preprint-last-revised-may-13-2024-httpsdoiorg1048550arxiv240412241">[Vidgen et al.]</a>
The WMDP Benchmark

Confabulation
BELEBELE
Big-bench: Analytic entailment, Formal fallacies and syllogisms with negation, Entailed polarity
Big-bench: Context Free Question Answering
Big-bench: Contextual question answering, Reading comprehension, Question generation
Big-bench: Convince Me
Big-bench: Low-resource language, Non-English, Translation
Big-bench: Morphology, Grammar, Syntax
Big-bench: Out-of-Distribution
Big-bench: Paraphrase
Big-bench: Sufficient information
Big-bench: Summarization
Big-bench: Truthfulness <a href="#srivastava-aarohi-abhinav-rastogi-abhishek-rao-abu-awal-md-shoeb-abubakar-abid-adam-fisch-adam-r-brown-adam-santoro-et-al-beyond-the-imitation-game-quantifying-and-extrapolating-the-capabilities-of-language-models-arxiv-preprint-last-revised-june-12-2023-httpsdoiorg1048550arxiv220604615">[Srivastava et al.]</a>
C-Eval (Chinese evaluation suite) <a href="#huang-yuzhen-yuzhuo-bai-zhihao-zhu-junlei-zhang-jinghan-zhang-tangjun-su-junteng-liu-chuancheng-lv-yikai-zhang-jiayi-lei-yao-fu-maosong-sun-and-junxian-he-c-eval-a-multi-level-multi-discipline-chinese-evaluation-suite-for-foundation-models-arxiv-preprint-last-revised-november-6-2023-httpsdoiorg1048550arxiv230508322">[Huang, Yuzhen et al.]</a>
Eval Gauntlet Reading comprehension
Eval Gauntlet: Commonsense reasoning, Symbolic problem solving, Programming
Eval Gauntlet: Language Understanding
Eval Gauntlet: World Knowledge
Evaluation Harness: BLiMP
Evaluation Harness: CoQA, ARC
Evaluation Harness: GLUE
Evaluation Harness: HellaSwag, OpenBookQA, TruthfulQA
Evaluation Harness: MuTual
Evaluation Harness: PIQA, PROST, MC-TACO, MathQA, LogiQA, DROP
FLASK: Logical correctness, Logical robustness, Logical efficiency, Comprehension, Completeness <a href="#ye-seonghyeon-doyoung-kim-sungdong-kim-hyeonbin-hwang-seungone-kim-yongrae-jo-james-thorne-juho-kim-and-minjoon-seo-flask-fine-grained-language-model-evaluation-based-on-alignment-skill-sets-arxiv-preprint-last-revised-april-14-2024-httpsdoiorg1048550arxiv230710928">[Ye et al.]</a>
FLASK: Readability, Conciseness, Insightfulness
Finding New Biases in Language Models with a Holistic Descriptor Dataset <a href="#smith-eric-michael-melissa-hall-melanie-kambadur-eleonora-presani-and-adina-williams-im-sorry-to-hear-that-finding-new-biases-in-language-models-with-a-holistic-descriptor-dataset-arxiv-preprint-last-revised-october-27-2022-httpsdoiorg1048550arxiv220509209">[Smith et al.]</a>
HELM: Knowledge
HELM: Language
HELM: Language (Twitter AAE)
HELM: Question answering
HELM: Reasoning
HELM: Reiteration, Wedging
HELM: Robustness to contrast sets
HELM: Summarization
HELM: Text classification
Hugging Face: Fill-mask, Text generation
Hugging Face: Question answering
Hugging Face: Summarization
Hugging Face: Text classification, Token classification, Zero-shot classification
MASSIVE
MLCommons <a href="#vidgen-bertie-adarsh-agrawal-ahmed-m-ahmed-victor-akinwande-namir-al-nuaimi-najla-alfaraj-elie-alhajjar-et-al-introducing-v05-of-the-ai-safety-benchmark-from-mlcommons-arxiv-preprint-last-revised-may-13-2024-httpsdoiorg1048550arxiv240412241">[Vidgen et al.]</a>
MT-bench <a href="#zheng-lianmin-wei-lin-chiang-ying-sheng-siyuan-zhuang-zhanghao-wu-yonghao-zhuang-zi-lin-zhuohan-li-dacheng-li-and-eric-p-xing-judging-llm-as-a-judge-with-mt-bench-and-chatbot-arena-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-2020-4659546623-published-may-30-2024-httpsdlacmorgdoi10555536661223668142">[Zheng et al.]</a>

Dangerous or Violent Recommendations
Big-bench: Convince Me
Big-bench: Toxicity
DecodingTrust: Adversarial Robustness, Robustness Against Adversarial Demonstrations
DecodingTrust: Machine Ethics <a href="#wang-boxin-weixin-chen-hengzhi-pei-chulin-xie-mintong-kang-chenhui-zhang-chejian-xu-zidi-xiong-ritik-dutta-rylan-schaeffer-et-al-decodingtrust-a-comprehensive-assessment-of-trustworthiness-in-gpt-models-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-1361-3123231339-published-may-30-2024-httpsdlacmorgdoi10555536661223667483">[Wang et al.]</a>
DecodingTrust: Toxicity
Evaluation Harness: ToxiGen
HELM: Reiteration, Wedging
HELM: Toxicity
MLCommons <a href="#vidgen-bertie-adarsh-agrawal-ahmed-m-ahmed-victor-akinwande-namir-al-nuaimi-najla-alfaraj-elie-alhajjar-et-al-introducing-v05-of-the-ai-safety-benchmark-from-mlcommons-arxiv-preprint-last-revised-may-13-2024-httpsdoiorg1048550arxiv240412241">[Vidgen et al.]</a>

Data Privacy
An Evaluation on Large Language Model Outputs: Discourse and Memorization (with human scoring, see Appendix B) <a href="#de-wynter-adrian-xun-wang-alex-sokolov-qilong-gu-and-si-qing-chen-an-evaluation-on-large-language-model-outputs-discourse-and-memorization-natural-language-processing-journal-4-september-2023-100024-httpsdoiorg101016jnlp2023100024">[de Wynter et al.]</a>
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation <a href="#huang-yangsibo-samyak-gupta-mengzhou-xia-kai-li-and-danqi-chen-catastrophic-jailbreak-of-open-source-llms-via-exploiting-generation-iclr-2024-spotlight-published-january-16-2024-last-modified-march-15-2024-httpsopenreviewnetforumidr42tsschph">[Huang, Yangsibo et al.]</a>
DecodingTrust: Machine Ethics <a href="#wang-boxin-weixin-chen-hengzhi-pei-chulin-xie-mintong-kang-chenhui-zhang-chejian-xu-zidi-xiong-ritik-dutta-rylan-schaeffer-et-al-decodingtrust-a-comprehensive-assessment-of-trustworthiness-in-gpt-models-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-1361-3123231339-published-may-30-2024-httpsdlacmorgdoi10555536661223667483">[Wang et al.]</a>
Evaluation Harness: ETHICS
HELM: Copyright
In-The-Wild Jailbreak Prompts on LLMs <a href="#shen-xinyue-zeyuan-chen-michael-backes-yun-shen-and-yang-zhang-do-anything-now-characterizing-and-evaluating-in-the-wild-jailbreak-prompts-on-large-language-models-arxiv-preprint-last-revised-may-15-2024-httpsdoiorg1048550arxiv230803825">[Shen et al.]</a>
JailbreakingLLMs
MLCommons <a href="#vidgen-bertie-adarsh-agrawal-ahmed-m-ahmed-victor-akinwande-namir-al-nuaimi-najla-alfaraj-elie-alhajjar-et-al-introducing-v05-of-the-ai-safety-benchmark-from-mlcommons-arxiv-preprint-last-revised-may-13-2024-httpsdoiorg1048550arxiv240412241">[Vidgen et al.]</a>
Mark My Words <a href="#piet-julien-chawin-sitawarin-vivian-fang-norman-mu-and-david-wagner-mark-my-words-analyzing-and-evaluating-language-model-watermarks-arxiv-preprint-last-revised-december-7-2023-httpsdoiorg1048550arxiv231200273">[Piet et al.]</a>
TAP: A Query-Efficient Method for Jailbreaking Black-Box LLMs
detect-pretrain-code <a href="#shi-weijia-anirudh-ajith-mengzhou-xia-yangsibo-huang-daogao-liu-terra-blevins-danqi-chen-and-luke-zettlemoyer-detecting-pretraining-data-from-large-language-models-arxiv-preprint-last-revised-march-9-2024-httpsdoiorg1048550arxiv231016789">[Shi et al.]</a>
llmprivacy <a href="#staab-robin-mark-vero-mislav-balunović-and-martin-vechev-beyond-memorization-violating-privacy-via-inference-with-large-language-models-arxiv-preprint-last-revised-may-6-2024-httpsdoiorg1048550arxiv231007298">[Staab et al.]</a>
mimir

Environmental
HELM: Efficiency

Information Integrity
Big-bench: Analytic entailment, Formal fallacies and syllogisms with negation, Entailed polarity
Big-bench: Convince Me
Big-bench: Paraphrase
Big-bench: Sufficient information
Big-bench: Summarization
Big-bench: Truthfulness <a href="#srivastava-aarohi-abhinav-rastogi-abhishek-rao-abu-awal-md-shoeb-abubakar-abid-adam-fisch-adam-r-brown-adam-santoro-et-al-beyond-the-imitation-game-quantifying-and-extrapolating-the-capabilities-of-language-models-arxiv-preprint-last-revised-june-12-2023-httpsdoiorg1048550arxiv220604615">[Srivastava et al.]</a>
DecodingTrust: Machine Ethics <a href="#wang-boxin-weixin-chen-hengzhi-pei-chulin-xie-mintong-kang-chenhui-zhang-chejian-xu-zidi-xiong-ritik-dutta-rylan-schaeffer-et-al-decodingtrust-a-comprehensive-assessment-of-trustworthiness-in-gpt-models-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-1361-3123231339-published-may-30-2024-httpsdlacmorgdoi10555536661223667483">[Wang et al.]</a>
DecodingTrust: Out-of-Distribution Robustness, Adversarial Robustness, Robustness Against Adversarial Demonstrations
Eval Gauntlet: Language Understanding
Eval Gauntlet: World Knowledge
Evaluation Harness: CoQA, ARC
Evaluation Harness: ETHICS
Evaluation Harness: GLUE
Evaluation Harness: HellaSwag, OpenBookQA, TruthfulQA
Evaluation Harness: MuTual
Evaluation Harness: PIQA, PROST, MC-TACO, MathQA, LogiQA, DROP
FLASK: Logical correctness, Logical robustness, Logical efficiency, Comprehension, Completeness <a href="#ye-seonghyeon-doyoung-kim-sungdong-kim-hyeonbin-hwang-seungone-kim-yongrae-jo-james-thorne-juho-kim-and-minjoon-seo-flask-fine-grained-language-model-evaluation-based-on-alignment-skill-sets-arxiv-preprint-last-revised-april-14-2024-httpsdoiorg1048550arxiv230710928">[Ye et al.]</a>
FLASK: Readability, Conciseness, Insightfulness
HELM: Knowledge
HELM: Language
HELM: Question answering
HELM: Reasoning
HELM: Reiteration, Wedging
HELM: Robustness to contrast sets
HELM: Summarization
HELM: Text classification
Hugging Face: Fill-mask, Text generation
Hugging Face: Question answering
Hugging Face: Summarization
MLCommons <a href="#vidgen-bertie-adarsh-agrawal-ahmed-m-ahmed-victor-akinwande-namir-al-nuaimi-najla-alfaraj-elie-alhajjar-et-al-introducing-v05-of-the-ai-safety-benchmark-from-mlcommons-arxiv-preprint-last-revised-may-13-2024-httpsdoiorg1048550arxiv240412241">[Vidgen et al.]</a>
MT-bench <a href="#zheng-lianmin-wei-lin-chiang-ying-sheng-siyuan-zhuang-zhanghao-wu-yonghao-zhuang-zi-lin-zhuohan-li-dacheng-li-and-eric-p-xing-judging-llm-as-a-judge-with-mt-bench-and-chatbot-arena-in-proceedings-of-the-37th-international-conference-on-neural-information-processing-systems-nips-23-article-no-2020-4659546623-published-may-30-2024-httpsdlacmorgdoi10555536661223668142">[Zheng et al.]</a>
Mark My Words <a href="#piet-julien-chawin-sitawarin-vivian-fang-norman-mu-and-david-wagner-mark-my-words-analyzing-and-evaluating-language-model-watermarks-arxiv-preprint-last-revised-december-7-2023-httpsdoiorg1048550arxiv231200273">[Piet et al.]</a>

Information Security
Big-bench: Convince Me
Big-bench: Out-of-Distribution
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation <a href="#huang-yangsibo-samyak-gupta-mengzhou-xia-kai-li-and-danqi-chen-catastrophic-jailbreak-of-open-source-llms-via-exploiting-generation-iclr-2024-spotlight-published-january-16-2024-last-modified-march-15-2024-httpsopenreviewnetforumidr42tsschph">[Huang, Yangsibo et al.]</a>
DecodingTrust: Out-of-Distribution Robustness, Adversarial Robustness, Robustness Against Adversarial Demonstrations
Eval Gauntlet: Commonsense reasoning, Symbolic problem solving, Programming
Garak: encoding, knownbadsignatures, malwaregen, packagehallucination, xss
HELM: Copyright
In-The-Wild Jailbreak Prompts on LLMs <a href="#shen-xinyue-zeyuan-chen-michael-backes-yun-shen-and-yang-zhang-do-anything-now-characterizing-and-evaluating-in-the-wild-jailbreak-prompts-on-large-language-models-arxiv-preprint-last-revised-may-15-2024-httpsdoiorg1048550arxiv230803825">[Shen et al.]</a>
JailbreakingLLMs
Mark My Words <a href="#piet-julien-chawin-sitawarin-vivian-fang-norman-mu-and-david-wagner-mark-my-words-analyzing-and-evaluating-language-model-watermarks-arxiv-preprint-last-revised-december-7-2023-httpsdoiorg1048550arxiv231200273">[Piet et al.]</a>
TAP: A Query-Efficient Method for Jailbreaking Black-Box LLMs <a href="#mehrotra-anay-manolis-zampetakis-paul-kassianik-blaine-nelson-hyrum-anderson-yaron-singer-and-amin-karbasi-tree-of-attacks-jailbreaking-black-box-llms-automatically-arxiv-preprint-last-revised-february-21-2024-httpsdoiorg1048550arxiv231202119">[Mehrotra et al.]</a>
detect-pretrain-code <a href="#shi-weijia-anirudh-ajith-mengzhou-xia-yangsibo-huang-daogao-liu-terra-blevins-danqi-chen-and-luke-zettlemoyer-detecting-pretraining-data-from-large-language-models-arxiv-preprint-last-revised-march-9-2024-httpsdoiorg1048550arxiv231016789">[Shi et al.]</a>
llmprivacy <a href="#staab-robin-mark-vero-mislav-balunović-and-martin-vechev-beyond-memorization-violating-privacy-via-inference-with-large-language-models-arxiv-preprint-last-revised-may-6-2024-httpsdoiorg1048550arxiv231007298">[Staab et al.]</a>
mimir

Intellectual Property
An Evaluation on Large Language Model Outputs: Discourse and Memorization (with human scoring, see Appendix B)
HELM: Copyright
Mark My Words <a href="#piet-julien-chawin-sitawarin-vivian-fang-norman-mu-and-david-wagner-mark-my-words-analyzing-and-evaluating-language-model-watermarks-arxiv-preprint-last-revised-december-7-2023-httpsdoiorg1048550arxiv231200273">[Piet et al.]</a>
llmprivacy <a href="#staab-robin-mark-vero-mislav-balunović-and-martin-vechev-beyond-memorization-violating-privacy-via-inference-with-large-language-models-arxiv-preprint-last-revised-may-6-2024-httpsdoiorg1048550arxiv231007298">[Staab et al.]</a>
mimir

Obscene, Degrading, and/or Abusive Content
Big-bench: Social bias, Racial bias, Gender bias, Religious bias
Big-bench: Toxicity
DecodingTrust: Fairness
DecodingTrust: Stereotype Bias
DecodingTrust: Toxicity
Evaluation Harness: CrowS-Pairs
Evaluation Harness: ToxiGen
HELM: Bias
HELM: Toxicity

Toxicity, Bias, and Homogenization
BELEBELE
Big-bench: Low-resource language, Non-English, Translation
Big-bench: Out-of-Distribution
Big-bench: Social bias, Racial bias, Gender bias, Religious bias
Big-bench: Toxicity
C-Eval (Chinese evaluation suite) <a href="#huang-yuzhen-yuzhuo-bai-zhihao-zhu-junlei-zhang-jinghan-zhang-tangjun-su-junteng-liu-chuancheng-lv-yikai-zhang-jiayi-lei-yao-fu-maosong-sun-and-junxian-he-c-eval-a-multi-level-multi-discipline-chinese-evaluation-suite-for-foundation-models-arxiv-preprint-last-revised-november-6-2023-httpsdoiorg1048550arxiv230508322">[Huang, Yuzhen et al.]</a>
DecodingTrust: Fairness
DecodingTrust: Stereotype Bias
DecodingTrust: Toxicity
Eval Gauntlet: World Knowledge
Evaluation Harness: CrowS-Pairs
Evaluation Harness: ToxiGen
Finding New Biases in Language Models with a Holistic Descriptor Dataset <a href="#smith-eric-michael-melissa-hall-melanie-kambadur-eleonora-presani-and-adina-williams-im-sorry-to-hear-that-finding-new-biases-in-language-models-with-a-holistic-descriptor-dataset-arxiv-preprint-last-revised-october-27-2022-httpsdoiorg1048550arxiv220509209">[Smith et al.]</a>
HELM: Bias
HELM: Toxicity
The Self-Perception and Political Biases of ChatGPT <a href="#rutinowski-jérôme-sven-franke-jan-endendyk-ina-dormuth-moritz-roidl-and-markus-pauly-the-self-perception-and-political-biases-of-chatgpt-human-behavior-and-emerging-technologies-2024-httpsdoiorg10115520247115633">[Rutinowski et al.]</a>
Towards Measuring the Representation of Subjective Global Opinions in Language Models <a href="#durmus-esin-karina-nguyen-thomas-i-liao-nicholas-schiefer-amanda-askell-anton-bakhtin-carol-chen-zac-hatfield-dodds-et-al-towards-measuring-the-representation-of-subjective-global-opinions-in-language-models-arxiv-preprint-last-revised-april-12-2024-httpsdoiorg1048550arxiv230616388">[Durmus et al.]</a>

C.3: AI Risk Management Framework Actions Aligned to Benchmarking

GOVERN 5.1, MAP 1.2, MAP 3.1, MEASURE 2.2, MEASURE 2.3, MEASURE 2.7, MEASURE 2.9, MEASURE 2.11, MEASURE 3.1, MEASURE 4.2

Usage Note: Materials in Section C can be used to perform in silica model testing for the presence of information in LLM outputs that may give rise to GAI risks or violate trustworthy characteristics. Model testing and benchmarking outcomes cannot be dispositive for the presence or absence of any in situ real-world risk. Model testing and benchmarking results may be compromised by task-contamination and other scientific measurement issues <a href="#balloccu-simone-patrícia-schmidtová-mateusz-lango-and-ondřej-dušek-leak-cheat-repeat-data-contamination-and-evaluation-malpractices-in-closed-source-llms-arxiv-preprint-last-revised-february-22-2024-httpsdoiorg1048550arxiv240203927">[Balloccu et al.]</a>. Furthermore, model testing is often ineffective for measuring human-AI configuration and value chain risks and few model tests appear to address explainability and interpretability.

• Material in Table C.1 can be applied to measure whether in silica LLM outputs may give rise to risks that violate trustworthy characteristics.

• Material in Table C.2 can be applied to measure whether in silica LLM outputs may give rise to GAI risks.

• Subsection C.3 highlights subcategories to indicate alignment with the AI RMF.

The materials in Section C reference measurement approaches that should be accompanied by red-teaming for medium risk systems or applications and field testing for high risk systems or applications.

D: Selected Adversarial Prompting Strategies and Attacks

D.1: Common AI Red-teaming Tools

Burpsuite, browser developer panes, bash utilities, other language models and GAI productivity tools, note-taking apps.

D.2: Selected Adversarial Prompting Strategies and Attacks Organized by Trustworthy Characteristic

Table D.1: Selected adversarial prompting techniques and attacks organized by trustworthy characteristic <a href="#saravia-elvis-prompt-engineering-guide-github-repository-last-modified-december-2022-accessed-september-19-2024-httpsgithubcomdair-aiprompt-engineering-guide">[Saravia]</a>, <a href="#storchan-victor-ravin-kumar-rumman-chowdhury-seraphina-goldfarb-tarrant-and-sven-cattell-generative-ai-red-teaming-challenge-transparency-report-humane-intelligence-2024-httpsdrivegooglecomfiled1jqpbip6dnomkb32umloiepombk2-0rc-view">[Storchan et al.]</a>, <a href="#hall-patrick-and-daniel-atherton-awesome-machine-learning-interpretability-github-repository-accessed-september-19-2024-httpsgithubcomjphall663awesome-machine-learning-interpretability">[Hall and Atherton]</a>, <a href="#hu-hongsheng-zoran-salcic-lichao-sun-gillian-dobbie-philip-s-yu-and-xuyun-zhang-membership-inference-attacks-on-machine-learning-a-survey-acm-computing-surveys-54-no-11s-september-2022-137-httpsdoiorg1011453523273">[Hu et al.]</a>, <a href="#sitawarin-chawin-charlie-cheng-jie-ji-apurv-verma-and-luckyfan-cs-llm-security--privacy-github-repository-accessed-september-19-2024-httpsgithubcomchawinsllm-sp">[Sitawarin et al.]</a>.

D.3: Selected Adversarial Prompting Techniques and Attacks Organized by Generative AI Risk

Table D.2: Selected adversarial prompting techniques and attacks organized by generative AI risk <a href="#saravia-elvis-prompt-engineering-guide-github-repository-last-modified-december-2022-accessed-september-19-2024-httpsgithubcomdair-aiprompt-engineering-guide">[Saravia]</a>, <a href="#storchan-victor-ravin-kumar-rumman-chowdhury-seraphina-goldfarb-tarrant-and-sven-cattell-generative-ai-red-teaming-challenge-transparency-report-humane-intelligence-2024-httpsdrivegooglecomfiled1jqpbip6dnomkb32umloiepombk2-0rc-view">[Storchan et al.]</a>, <a href="#hall-patrick-and-daniel-atherton-awesome-machine-learning-interpretability-github-repository-accessed-september-19-2024-httpsgithubcomjphall663awesome-machine-learning-interpretability">[Hall and Atherton]</a>, <a href="#hu-hongsheng-zoran-salcic-lichao-sun-gillian-dobbie-philip-s-yu-and-xuyun-zhang-membership-inference-attacks-on-machine-learning-a-survey-acm-computing-surveys-54-no-11s-september-2022-137-httpsdoiorg1011453523273">[Hu et al.]</a>, <a href="#sitawarin-chawin-charlie-cheng-jie-ji-apurv-verma-and-luckyfan-cs-llm-security--privacy-github-repository-accessed-september-19-2024-httpsgithubcomchawinsllm-sp">[Sitawarin et al.]</a>.

D.4: AI Risk Management Framework Actions Aligned to Red Teaming

GOVERN 3.2, GOVERN 4.1, MANAGE 2.2, MANAGE 4.1, MEASURE 1.1, MEASURE 1.3, MEASURE 2.6, MEASURE 2.7, MEASURE 2.8, MEASURE 2.10, MEASURE 2.11

Usage Note: Materials in Section D can be used to perform red-teaming to measure the risk that expert adversarial actors can manipulate LLM systems or risks that users may encounter under worst-case or anomalous scenarios.

• Try augmenting strategies with tools listed in D.1.

• Strategies and goals in Table D.2 can be applied to assess whether LLM outputs may violate trustworthy characteristics under adversarial, anomalous, or worst-case scenarios.

• Strategies and goals in Table D.3 can be applied to assess whether LLM outputs may give rise to GAI risks under adversarial, anomalous, or worst-case scenarios.

• Subsection D.4 highlights subcategories to indicate alignment with the AI RMF.

The materials in Section D reference measurement approaches that should be accompanied by field testing for high risk systems or applications.

E: Selected Risk Controls for Generative AI

Usage Note: Section E puts forward selected risk controls that organizations may apply for GAI risk management. Higher level controls are linked to specific GAI and AI RMF Playbook actions <a href="#national-institute-of-standards-and-technology-nist-nist-ai-rmf-playbook-trustworthy--responsible-ai-resource-center-accessed-september-19-2024-httpsaircnistgovai_rmf_knowledge_baseplaybook">[NIST AI RMF Playbook]</a>, <a href="#national-institute-of-standards-and-technology-nist-artificial-intelligence-risk-management-framework-generative-artificial-intelligence-profile-nist-ai-600-1-gaithersburg-md-nist-july-2024-httpsdoiorg106028nistai600-1">[NIST AI 600-1]</a>.

F: Example Low-risk Generative AI Measurement and Management Plan

F.1: Example Low-risk Generative AI Measurement and Management Plan Organized by Trustworthy Characteristic

F.2: Example Low-risk Generative AI Measurement and Management Plan Organized by Generative AI Risk

Usage Note: Section F puts forward an example risk measurement and management plan for low risk GAI systems or applications. The low risk plan focuses on automatable model testing and applies minimally burdensome risk controls.

• Material in Table F.1 can be applied to measure and manage GAI risks in risk programs that are aligned to the trustworthy characteristics.

• Material in Table F.2 can be applied to measure and manage GAI risks in risk programs that are aligned to GAI risks.

Section G below presents an example plan for medium risk systems and Section H presents an example plan for high risk systems.

Usage Note: Section E puts forward selected risk controls that organizations may apply for GAI risk management. Higher level controls are linked to specific GAI and AI RMF Playbook actions.

G: Example Medium-risk Generative AI Measurement and Management Plan

G.1: Example Medium-risk Generative AI Measurement and Management Plan Organized by Trustworthy Characteristic

G.2: Example Medium-risk Generative AI Measurement and Management Plan Organized by Generative AI Risk

Usage Note: Section G puts forward an example risk measurement and management plan for medium risk GAI systems or applications. The medium risk plan focuses on red-teaming and applies moderate risk controls. Measurement and management approaches from Section F should also be applied to medium risk systems or applications.

• Material in Table G.1 can be applied to measure and manage GAI risks in risk programs that are aligned to the trustworthy characteristics.

• Material in Table G.2 can be applied to measure and manage GAI risks in risk programs that are aligned to GAI risks.

Section H below presents an example plan for high risk systems.

H: Example High-risk Generative AI Measurement and Management Plan

H.1: Example High-risk Generative AI Measurement and Management Plan Organized by Trustworthy Characteristic

H.2: Example High-risk Generative AI Measurement and Management Plan Organized by Generative AI Risk

Usage Note: Section H puts forward an example risk measurement and management plan for high risk GAI systems or applications. The high risk plan focuses on field testing and applies extensive risk controls. Measurement and management approaches from Appendices F and G should also be applied to high risk systems or applications.

• Material in Table H.1 can be applied to measure and manage GAI risks in risk programs that are aligned to the trustworthy characteristics.

• Material in Table H.2 can be applied to measure and manage GAI risks in risk programs that are aligned to GAI risks.

References

AI Verify Foundation and Infocomm Media Development Authority. Cataloguing LLM Evaluations. Draft for Discussion, October 2023. https://aiverifyfoundation.sg/downloads/Cataloguing_LLM_Evaluations.pdf.

AI Verify Foundation and Infocomm Media Development Authority. LLM Evals Catalogue. GitHub repository. Accessed September 19, 2024. https://github.com/aiverify-foundation/LLM-Evals-Catalogue.

Balloccu, Simone, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs." arXiv preprint, last revised February 22, 2024. https://doi.org/10.48550/arXiv.2402.03927.

Bandarkar, Lucas, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. "The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants." arXiv preprint, last revised July 25, 2024. https://doi.org/10.48550/arXiv.2308.16884.

Barreno, Marco, Blaine Nelson, Anthony D. Joseph, and J.D. Tygar. "The Security of Machine Learning." Machine Learning 81, no. 2 (2010): 121–148. https://doi.org/10.1007/s10994-010-5188-5.

Bommasani, Rishi, Percy Liang, and Tony Lee. "Holistic Evaluation of Language Models." Annals of the New York Academy of Sciences 1525, no. 1 (July 2023): 140–146. https://doi.org/10.1111/nyas.15007.

Chao, Patrick, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. "Jailbreaking Black Box Large Language Models in Twenty Queries." arXiv preprint, last revised July 18, 2024. https://doi.org/10.48550/arXiv.2310.08419.

De Wynter, Adrian, Xun Wang, Alex Sokolov, Qilong Gu, and Si-Qing Chen. "An Evaluation on Large Language Model Outputs: Discourse and Memorization." Natural Language Processing Journal 4 (September 2023): 100024. https://doi.org/10.1016/j.nlp.2023.100024.

Department for Science, Innovation and Technology, and AI Safety Institute. International Scientific Report on the Safety of Advanced AI: Interim Report. Published May 17, 2024. https://www.gov.uk/government/publications/international-scientific-report-on-the-safety-of-advanced-ai.

Derczynski, Leon, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. "garak: A Framework for Security Probing Large Language Models." arXiv preprint, submitted June 16, 2024. https://doi.org/10.48550/arXiv.2406.11036.

Dohmann, Jeremy. "Blazingly Fast LLM Evaluation for In-Context Learning." Databricks: Mosaic AI Research, February 2, 2023. https://www.databricks.com/blog/llm-evaluation-for-icl.

Duan, Michael, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. "Do Membership Inference Attacks Work on Large Language Models?" arXiv preprint, last revised September 16, 2024. https://doi.org/10.48550/arXiv.2402.07841.

Durmus, Esin, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, et al. "Towards Measuring the Representation of Subjective Global Opinions in Language Models." arXiv preprint, last revised April 12, 2024. https://doi.org/10.48550/arXiv.2306.16388.

Hugging Face. "Evaluate." Last accessed September 19, 2024. https://huggingface.co/docs/evaluate/index.

Feng, Shangbin, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. "From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models." arXiv preprint, last revised July 6, 2023. https://doi.org/10.48550/arXiv.2305.08283.

FitzGerald, Jack, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, et al. "MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages." arXiv preprint, last revised June 17, 2022. https://doi.org/10.48550/arXiv.2204.08582.

Gao, Leo, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A Framework for Few-Shot Language Model Evaluation. GitHub repository. Accessed September 19, 2024. https://github.com/EleutherAI/lm-evaluation-harness.

Hall, Patrick, and Daniel Atherton. Awesome Machine Learning Interpretability. GitHub repository. Accessed September 19, 2024. https://github.com/jphall663/awesome-machine-learning-interpretability.

Hu, Hongsheng, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. "Membership Inference Attacks on Machine Learning: A Survey." ACM Computing Surveys 54, no. 11s (September 2022): 1–37. https://doi.org/10.1145/3523273.

Huang, Yangsibo, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. "Catastrophic Jailbreak of Open-Source LLMs via Exploiting Generation." ICLR 2024 Spotlight, published January 16, 2024, last modified March 15, 2024. https://openreview.net/forum?id=r42tSSCHPh.

Huang, Yuzhen, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. "C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models." arXiv preprint, last revised November 6, 2023. https://doi.org/10.48550/arXiv.2305.08322.

ISO/IEC 42001:2023. Information Technology — Artificial Intelligence — Management System. 1st ed. Geneva: International Organization for Standardization, 2023. https://www.iso.org/obp/ui/en/#iso:std:iso-iec:42001:ed-1:v1:en.

Li, Nathaniel, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, et al. "The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning." arXiv preprint, last revised May 15, 2024. https://doi.org/10.48550/arXiv.2403.03218.

Li, Nathaniel, Ziwen Han, Ian Steneker, Willow Primack, et al. "LLM defenses are not robust to multi-turn human jailbreaks yet." arXiv preprint, last revised Wed, September 4, 2024. https://arxiv.org/pdf/2408.15221.

Liu, Yi, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. "Prompt Injection Attack Against LLM-Integrated Applications." arXiv preprint, last revised March 2, 2024. https://doi.org/10.48550/arXiv.2306.05499.

McGraw, Gary, Harold Figueroa, Katie McMahon, and Richie Bonett. An Architectural Risk Analysis of Large Language Models: Applied Machine Learning Security. Version 1.0. Berryville Institute of Machine Learning (BIML), January 24, 2024. https://berryvilleiml.com/docs/BIML-LLM24.pdf.

McGraw, Gary, Harold Figueroa, Victor Shepardson, and Richie Bonett. An Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning. Version 1.0 (1.13.20). Berryville Institute of Machine Learning (BIML), January 13, 2020. https://berryvilleiml.com/docs/ara.pdf.

Mehrotra, Anay, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. "Tree of Attacks: Jailbreaking Black-Box LLMs Automatically." arXiv preprint, last revised February 21, 2024. https://doi.org/10.48550/arXiv.2312.02119.

Microsoft. Microsoft Responsible AI Standard, v2: General Requirements. For External Release. June 2022. https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFl.

National Institute of Standards and Technology (NIST). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: NIST, January 26, 2023. https://doi.org/10.6028/NIST.AI.100-1.

National Institute of Standards and Technology (NIST). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. Gaithersburg, MD: NIST, July 2024. https://doi.org/10.6028/NIST.AI.600-1.

National Institute of Standards and Technology (NIST). Guide for Conducting Risk Assessments. NIST Special Publication 800-30 Rev. 1. Prepared by the Joint Task Force Transformation Initiative. Gaithersburg, MD: NIST, September 2012. https://doi.org/10.6028/NIST.SP.800-30r1.

National Institute of Standards and Technology (NIST). NIST AI RMF Playbook. Trustworthy & Responsible AI Resource Center. Accessed September 19, 2024. https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook.

Office of the Comptroller of the Currency (OCC). Model Risk Management. Comptroller’s Handbook, Version 1.0, August 2021. https://www.occ.gov/publications-and-resources/publications/comptrollers-handbook/files/model-risk-management/index-model-risk-management.html.

Perez, Ethan, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. "Red Teaming Language Models with Language Models." arXiv preprint, submitted February 7, 2022. https://doi.org/10.48550/arXiv.2202.03286.

Piet, Julien, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. "Mark My Words: Analyzing and Evaluating Language Model Watermarks." arXiv preprint, last revised December 7, 2023. https://doi.org/10.48550/arXiv.2312.00273.

Rutinowski, Jérôme, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. "The Self-Perception and Political Biases of ChatGPT." Human Behavior and Emerging Technologies, 2024. https://doi.org/10.1155/2024/7115633.

Saravia, Elvis. Prompt Engineering Guide. GitHub repository. Last modified December 2022. Accessed September 19, 2024. https://github.com/dair-ai/Prompt-Engineering-Guide.

Shen, Xinyue, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "‘Do Anything Now’: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models." arXiv preprint, last revised May 15, 2024. https://doi.org/10.48550/arXiv.2308.03825.

Shi, Weijia, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. "Detecting Pretraining Data from Large Language Models." arXiv preprint, last revised March 9, 2024. https://doi.org/10.48550/arXiv.2310.16789.

Shumailov, Ilia, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. "Sponge Examples: Energy-Latency Attacks on Neural Networks." In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), 6–10 September 2021, Vienna, Austria. IEEE, 2021. https://doi.org/10.1109/EuroSP51992.2021.00024.

Sitawarin, Chawin, Charlie Cheng-Jie Ji, Apurv Verma, and Luckyfan-cs. LLM Security & Privacy. GitHub repository. Accessed September 19, 2024. https://github.com/chawins/llm-sp.

Smith, Eric Michael, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. "‘I’m Sorry to Hear That’: Finding New Biases in Language Models with a Holistic Descriptor Dataset." arXiv preprint, last revised October 27, 2022. https://doi.org/10.48550/arXiv.2205.09209.

Srivastava, Aarohi, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, et al. "Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models." arXiv preprint, last revised June 12, 2023. https://doi.org/10.48550/arXiv.2206.04615.

Staab, Robin, Mark Vero, Mislav Balunović, and Martin Vechev. "Beyond Memorization: Violating Privacy via Inference with Large Language Models." arXiv preprint, last revised May 6, 2024. https://doi.org/10.48550/arXiv.2310.07298.

Storchan, Victor, Ravin Kumar, Rumman Chowdhury, Seraphina Goldfarb-Tarrant, and Sven Cattell. Generative AI Red Teaming Challenge: Transparency Report. Humane Intelligence, 2024. https://drive.google.com/file/d/1JqpbIP6DNomkb32umLoiEPombK2-0Rc-/view.

Vidgen, Bertie, Adarsh Agrawal, Ahmed M. Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, et al. "Introducing v0.5 of the AI Safety Benchmark from MLCommons." arXiv preprint, last revised May 13, 2024. https://doi.org/10.48550/arXiv.2404.12241.

Wang, Boxin, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. "DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models." In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23), Article No. 1361, 31232–31339. Published May 30, 2024. https://dl.acm.org/doi/10.5555/3666122.3667483.

Ye, Seonghyeon, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. "FLASK: Fine-Grained Language Model Evaluation Based on Alignment Skill Sets." arXiv preprint, last revised April 14, 2024. https://doi.org/10.48550/arXiv.2307.10928.

Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, and Eric P. Xing. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23), Article No. 2020, 46595–46623. Published May 30, 2024. https://dl.acm.org/doi/10.5555/3666122.3668142.