# Safety Datasets used in LibrAI Leaderboard

This document provides a summary of the safety evaluation datasets that been used in LibrAI Leaderboard. 

## Overview

| Dataset Name | Include | Year | Publication Name | Purpose Type | Language | Entries | GitHub | HF Link | Notes |
|--------------|---------|------|------------------|--------------|----------|---------|-------|-------|-------|
| AART | ✅ | 2023 | [AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications](https://aclanthology.org/2023.emnlp-industry.37/) | broad safety | English | 3,269 prompts | [Link](https://github.com/google-research-datasets/aart-ai-safety-dataset) | No | <br>- contains examples for specific geographic regions<br>- prompts also change up use cases and concepts |
| AdvBench | ✅ | 2023 | [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://arxiv.org/abs/2307.15043) | broad safety | English | 1,000 prompts | [Link](https://github.com/llm-attacks/llm-attacks/tree/main/data/advbench) | No | <br>- focus of the work is adversarial / to jailbreak LLMs<br>- AdvBench tests whether jailbreaks succeeded |
| AnthropicHarmlessBase | ✅ | 2024 | [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://arxiv.org/abs/2204.05862) | broad safety | English | 44,849 conversational turns | [Link](https://github.com/anthropics/hh-rlhf) | [Link](https://huggingface.co/datasets/Trelis/hh-rlhf-dpo) | <br>- most prompts created by 28 US-based crowdworkers |
| AnthropicRedTeam | ✅ | 2024 | [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://arxiv.org/abs/2209.07858) | broad safety | English | 38,961 conversations | [Link](https://github.com/anthropics/hh-rlhf) | [Link](https://huggingface.co/datasets/Anthropic/hh-rlhf) | <br>- created by 324 US-based crowdworkers<br>- ca. 80% of examples come from ca. 50 workers |
| BAD | ✅ | 2022 | [Bot-Adversarial Dialogue for Safe Conversational Agents](https://aclanthology.org/2021.naacl-main.235/) | broad safety | English | 78,874 conversations | [Link](https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/bot_adversarial_dialogue) | No | <br>- download only via ParlAI<br>- approximately 40% of all dialogues are annotated as offensive, with a third of offensive utterances generated by bots |
| BBQ | ✅ | 2023 | [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://aclanthology.org/2022.findings-acl.165) | bias | English | 58,492 examples | [Link](https://github.com/nyu-mll/BBQ) | [Link](https://huggingface.co/datasets/heegyu/bbq) | <br>- focus on stereotyping behaviour<br>- covers 9 categories of bias: age, disability status, gender identity, nationality, physical appearance, race/ethnicity, religion, socioeconomic status, sexual orientation<br>- 25+ templates per category |
| BeaverTails | Todo | 2022 | [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://arxiv.org/abs/2307.04657) | broad safety | English | 333,963 conversations | [Link](https://github.com/PKU-Alignment/beavertails) | [Link](https://huggingface.co/datasets/PKU-Alignment/BeaverTails) | <br>- 16,851 unique prompts sampled from AnthropicRedTeam <br>- covers 14 harm categories (e.g. animal abuse)<br>- annotated for safety by 3.34 crowdworkers on average |
| CDialBias | ❌ | 2019 | [Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark](https://aclanthology.org/2022.findings-emnlp.262) | bias | Chinese | 28,343 conversations | [Link](https://github.com/para-zhou/CDial-Bias) | No | <br>- cover 4 categories of bias: race, gender, religion, occupation |
| CoNA | ✅ | 2024 | [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875) | narrow safety | English | 178 prompts | [Link](https://github.com/vinid/instruction-llms-safety-eval) | No | <br>- Focus: harmful instructions (e.g. hate speech)<br>- All prompts are (meant to be) unsafe |
| ConfAIde | ✅ | 2023 | [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://openreview.net/forum?id=gmg7t8b4s0) | narrow safety | English | 1,326 prompts | [Link](https://github.com/skywalker023/confAIde/tree/main/benchmark) | No | <br>- The benchmark is split into 4 tiers with different prompt formats<br>- tier 1 contains 10 prompts, tier 2 2*98, tier 3 4*270, tier 4 50 |
| ControversialInstructions | ✅ | 2023 | [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875) | narrow safety | English | 40 prompts | [Link](https://github.com/vinid/instruction-llms-safety-eval) | No | <br>- Focus: controversial topics (e.g. immigration)<br>- All prompts are (meant to be) unsafe |
| CyberattackAssistance | ✅ | 2024 | [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://arxiv.org/abs/2312.04724) | narrow safety | English | 1,000 prompts | [Link](https://github.com/facebookresearch/PurpleLlama/tree/main/CybersecurityBenchmarks/datasets/mitre) | No | <br>- instructions are split into 10 MITRE categories<br>- The dataset comes with additional LLM-rephrased instructions |
| DecodingTrust | ✅ | 2022 | [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://arxiv.org/abs/2306.11698) | broad safety | English | 243,877 prompts | [Link](https://github.com/AI-secure/DecodingTrust) | [Link](https://huggingface.co/datasets/AI-Secure/DecodingTrust) | <br>- split across 8 'trustworthiness perspectives': toxicity, stereotypes, adversarial and robustness, privacy, ethics and fairness |
| DICES350 | ✅ | 2023 | [DICES Dataset: Diversity in Conversational AI Evaluation for Safety](https://arxiv.org/abs/2306.11247) | value alignment | English | 350 conversations | [Link](https://github.com/google-research-datasets/dices-dataset/) | No | <br>- 104 ratings per item<br>- annotators from US<br>- annotation across 24 safety criteria |
| DoAnythingNow | ❌ | 2023 | ['Do Anything Now': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825) | narrow safety | English | 6,387 prompts | [Link](https://github.com/verazuo/jailbreak_llms) | No | <br>- there are 666 jailbreak prompts among the 6,387prompts |
| DoNotAnswer | ✅ | 2023 | [Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs](https://arxiv.org/abs/2308.13387) | broad safety | English | 939 prompts | [Link](https://github.com/Libr-AI/do-not-answer) | [Link](https://huggingface.co/datasets/LibrAI/do-not-answer) | <br>- split across 5 risk areas and 12 harm types<br>- authors prompted GPT-4 to generate questions |
| DoAnythingNow | ✅ | 2022 | ['Do Anything Now': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825) | broad safety | English | 46,800 prompts | [Link](https://github.com/verazuo/jailbreak_llms) | No | <br>- 13 scenarios with 30 questions each, then expanded by combination with 8 communities, 3 prompt setups and 5 repetations |
| GandalfIgnoreInstructions | ✅ | 2023 | [Gandalf Prompt Injection: Ignore Instruction Prompts](https://www.lakera.ai/blog/who-is-gandalf) | other | English | 1,000 prompts | No | [Link](https://huggingface.co/datasets/Lakera/gandalf_ignore_instructions) | <br>- focus of the work is adversarial / prompt extraction<br>- not all prompts are attacks |
| HackAPrompt | Todo | 2023 | [Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition](https://aclanthology.org/2023.emnlp-main.302/) | other | mostly English | 601,757 prompts | No | [Link](https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset) | <br>- focus of the work is adversarial / prompt hacking<br>- prompts were written by ca. 2.8k people from 50+ countries |
| HarmBench | ✅ | 2022 | [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://arxiv.org/abs/2402.04249) | broad safety | English | 400 prompts | [Link](https://github.com/centerforaisafety/HarmBench/tree/main/data/behavior_datasets) | No | <br>- The dataset covers 7 semantic categories of behaviour: Cybercrime & Unauthorized Intrusion, Chemical & Biological Weapons/Drugs, Copyright Violations, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, and General Harm<br>- The dataset also includes 110 multimodal prompts |
| HarmfulQ | ✅ | 2021 | [On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://aclanthology.org/2023.acl-long.244/) | broad safety | English | 200 prompts | [Link](https://github.com/SALT-NLP/chain-of-thought-bias) | No | <br>- focus on 6 attributes: "racist, stereotypical, sexist, illegal, toxic, harmful"<br>- authors do manual filtering for overly similar questions |
| RedEval | ✅ | 2023 | [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://arxiv.org/abs/2308.09662) | broad safety | English | 1,960 prompts | [Link](https://github.com/declare-lab/red-instruct/tree/main/harmful_questions) | [Link](https://huggingface.co/datasets/declare-lab/HarmfulQA) | <br>- split into 10 topics (e.g. "Mathematics and Logic")<br>- similarity across prompts is quite high<br>- not all prompts are unsafe / safety-related |
| HExPHI | ✅ | 2024 | [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://openreview.net/forum?id=hTEGyKf0dZ) | broad safety | English | 330 prompts | No | [Link](https://huggingface.co/datasets/LLM-Tuning-Safety/HEx-PHI) | <br>- Main focus is on finetuning models<br>- Prompts cover 11 harm areas |
| HolisticBias | ❌ | 2023 | [I’m sorry to hear that: Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://aclanthology.org/2022.emnlp-main.625) | bias | English | 459,758 prompts | [Link](https://github.com/facebookresearch/ResponsibleNLP/tree/main/holistic_bias) | No | <br>- 26 sentence templates<br>- covers 13 categories of bias: ability, age, body type, characteristics, culturural, gender/sex, nationality, nonce, political, race/ethnicity, religion, sexual orientation, socioeconomic |
| HypothesisStereotypes | ✅ | 2023 | [Analyzing Stereotypes in Generative Text Inference Tasks](https://aclanthology.org/2021.findings-acl.355/) | bias | English | 2,098 prompts | [Link](https://github.com/AnnaSou/stereotypes_generative_inferences) | No | <br>- uses 103 context situations as templates<br>- covers 6 categories of bias: gender, race, nationality, religion, politics, socio<br>- task for LLM is to generate hypothesis based on premise |
| LatentJailbreak | ✅ | 2024 | [Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models](https://arxiv.org/abs/2307.08487) | narrow safety | English | 416 prompts | [Link](https://github.com/qiuhuachuan/latent-jailbreak/tree/main) | No | <br>- focus of the work is adversarial / to jailbreak LLMs<br>- 13 prompt templates instantiated with 16 protected group terms and 2 posititional types<br>- main exploit focuses on translation |
| MaliciousInstruct | ✅ | 2022 | [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://arxiv.org/abs/2310.06987) | broad safety | English | 100 prompts | [Link](https://github.com/Princeton-SysML/Jailbreak_LLM/tree/main/data) | No | <br>- covers ten 'malicious intentions': psychological manipulation, sabotage, theft, defamation, cyberbullying, false accusation, tax fraud, hacking, fraud, and illegal drug use. |
| MaliciousInstructions | ✅ | 2024 | [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875) | broad safety | English | 100 prompts | [Link](https://github.com/vinid/instruction-llms-safety-eval) | No | <br>- Focus: malicious instructions (e.g. bombmaking)<br>- All prompts are (meant to be) unsafe |
| MIC | ❌ | 2024 | [The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems](https://aclanthology.org/2022.acl-long.261) | value alignment | English | 38,000 conversations | [Link](https://github.com/SALT-NLP/mic) | No | <br>- based on the RoT paradigm introduced in SocialChemistry<br>- 38k prompt-reply pairs come with 99k rules of thumb and 114k annotations |
| MoralChoice | ✅ | 2023 | [Evaluating the Moral Beliefs Encoded in LLMs](https://openreview.net/forum?id=O06z2G18me) | value alignment | English | 1,767 binary-choice question | [Link](https://github.com/ninodimontalcino/moralchoice) | [Link](https://huggingface.co/datasets/ninoscherrer/moralchoice) | <br>- 687 scenarios are low-ambiguity, 680 are high-ambiguity<br>- three Surge annotators choose the favourable action for each scenario |
| DialogueSafety | ✅ | 2021 | [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://aclanthology.org/D19-1461) | broad safety | English | 90,000 prompts | [Link](https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/dialogue_safety) | No | <br>- download only via ParlAI<br>- the "single-turn" dataset provides a "standard" and "adversarial" setting, with 3 rounds of data collection each |
| PersonalInfoLeak | ✅ | 2024 | [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://aclanthology.org/2022.findings-emnlp.148) | narrow safety | English | 3,238 entries | [Link](https://github.com/jeffhj/LM_PersonalInfoLeak) | No | <br>- main task is to predict email given name |
| PhysicalSafetyInstructions | ✅ | 2023 | [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875) | narrow safety | English | 1,000 prompts | [Link](https://github.com/vinid/instruction-llms-safety-eval) | No | <br>- Focus: commonsense physical safety<br>- 50 safe and 50 unsafe prompts |
| PromptExtractionRobustness | ✅ | 2023 | [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://arxiv.org/abs/2311.01011) | narrow safety | English | 569 samples | [Link](https://github.com/HumanCompatibleAI/tensor-trust-data) | [Link](https://huggingface.co/datasets/qxcv/tensor-trust?row=0) | <br>- filtered from larger raw prompt extraction dataset<br>- collected using the open Tensor Trust online game |
| PromptHijackingRobustness | ✅ | 2021 | [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://arxiv.org/abs/2311.01011) | narrow safety | English | 775 samples | [Link](https://github.com/HumanCompatibleAI/tensor-trust-data) | [Link](https://huggingface.co/datasets/qxcv/tensor-trust?row=0) | <br>- filtered from larger raw prompt extraction dataset<br>- collected using the open Tensor Trust online game |
| QHarm | Todo | 2023 | [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875) | broad safety | English | 100 prompts | [Link](https://github.com/vinid/instruction-llms-safety-eval) | No | <br>- Wider topic coverage due to source dataset<br>- Prompts are mostly unsafe |
| RedditBias | ❌ | 2024 | [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://aclanthology.org/2021.acl-long.151) | bias | English | 11,873 Reddit comments | [Link](https://github.com/umanlp/RedditBias) | No | <br>- covers 4 categories of bias: religion, race, gender, queerness<br>- evaluation by perplexity and conversation |
| RuLES | ❌ | 2024 | [Can LLMs Follow Simple Rules?](https://arxiv.org/abs/2311.04235) | narrow safety | English | 862 prompts | [Link](https://github.com/normster/llm_rules) | [Link](https://huggingface.co/datasets/normster/RuLES) | <br>- The dataset covers 19 rules across 14 scenarios |
| SafeText | ❌ | 2022 | [SafeText: A Benchmark for Exploring Physical Safety in Language Models](https://aclanthology.org/2022.emnlp-main.154/) | narrow safety | English | 367 prompts | [Link](https://github.com/sharonlevy/SafeText) | No | <br>- 5 ratings for relevance per item during filtering<br>- advice format most often elicits yes/no answer |
| SafetyPrompts | ❌ | 2023 | [Safety Assessment of Chinese Large Language Models](https://arxiv.org/abs/2304.10436) | broad safety | Chinese | 100,000 prompts | [Link](https://github.com/thu-coai/Safety-Prompts) | [Link](https://huggingface.co/datasets/thu-coai/Safety-Prompts) | <br>- covers 8 safety scenarios and 6 types of adv attack<br>- do not release 'sensitive topics' scenario |
| SimpleSafetyTests | ✅ | 2024 | [SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models](https://arxiv.org/abs/2311.08370) | broad safety | English | 100 prompts | [Link](https://github.com/bertiev/SimpleSafetyTests) | [Link](https://huggingface.co/datasets/Bertievidgen/SimpleSafetyTests) | <br>- The dataset is split into ten types of prompts |
| SPMisconceptions | ✅ | 2022 | [Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptions](https://dl.acm.org/doi/10.1145/3627106.3627196) | narrow safety | English | 122 prompts | [Link](https://github.com/purseclab/LLM_Security_Privacy_Advice) | No | <br>- misconceptions all relate to security and privacy <br>- uses templates to turn misconceptions into prompts<br>- covers six categories (e.g. crypto and blockchain, law and regulation)<br> |
| StrongREJECT | ✅ | 2023 | [A StrongREJECT for Empty Jailbreaks](https://arxiv.org/pdf/2402.10260.pdf) | broad safety | English | 346 prompts | [Link](https://github.com/alexandrasouly/strongreject/tree/main) | No | <br>- The focus of the work is adversarial / to jailbreak LLMs<br>- The 6 question categories are: disinformation/deception, hate/harassment/discrimination, illegal goods/services, non-violent crimes, sexual content, violence |
| SycophancyEval | ✅ | 2024 | [Towards Understanding Sycophancy in Language Models](https://openreview.net/forum?id=tvhaxkMKAn) | narrow safety | English | 20,956 prompts | [Link](https://github.com/meg-tong/sycophancy-eval/tree/main) | [Link](https://huggingface.co/datasets/meg-tong/sycophancy-eval) | <br>- Dataset uses four different task setups to evaluate sycophancy: answer (7268 prompts), are_you_sure (4888 prompts), feedback (8500 prompts), mimicry (300 prompts) |
| TDCRedTeaming | ✅ | 2022 | [TDC 2023 (LLM Edition): The Trojan Detection Challenge](https://trojandetection.ai/) | broad safety | English | 100 prompts | [Link](https://github.com/centerforaisafety/tdc2023-starter-kit/tree/main/red_teaming/data) | No | <br>- covers 7 categories: Bigotry and Abusive Language, Violent Content and Conduct, Illegal Activities, Malware and Exploits, Scams, Misinformation and Disinformation, Other Undesirable Content |
| ToxiGen | ✅ | 2023 | [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://aclanthology.org/2022.acl-long.234/) | narrow safety | English | 260,851 prompts | [Link](https://github.com/microsoft/TOXIGEN) | [Link](https://huggingface.co/datasets/skg/toxigen-data) | <br>- covers 13 target groups<br>- seed prompts are used to generate implicit hate<br>- evaluating generative LLMs is not the focus |
| TruthfulQA | Todo | 2022 | [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://aclanthology.org/2022.acl-long.229/) | narrow safety | English | 817 prompts | [Link](https://github.com/sylinrl/TruthfulQA) | [Link](https://huggingface.co/datasets/truthful_qa) | <br>- covers 38 categories (e.g. health and politics)<br>- comes with multiple choice expansion |
| XSTest | ✅ | 2023 | [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://arxiv.org/abs/2308.01263) | narrow safety | English | 450 prompts | [Link](https://github.com/paul-rottger/exaggerated-safety) | [Link](https://huggingface.co/datasets/natolambert/xstest-v2-copy) | <br>- split into ten types of prompts<br>- 250 safe prompts and 200 unsafe prompts |

---
## Datasets in Detail

### AART
- **Year**: 2023
- **Publication Name**: [AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications](https://aclanthology.org/2023.emnlp-industry.37/)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 3,269 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/google-research-datasets/aart-ai-safety-dataset](https://github.com/google-research-datasets/aart-ai-safety-dataset)
- **HF Link**: [not available](not available)
- **Notes**: <br>- contains examples for specific geographic regions<br>- prompts also change up use cases and concepts

---
### AdvBench
- **Year**: 2023
- **Publication Name**: [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://arxiv.org/abs/2307.15043)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 1,000 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/llm-attacks/llm-attacks/tree/main/data/advbench](https://github.com/llm-attacks/llm-attacks/tree/main/data/advbench)
- **HF Link**: [not available](not available)
- **Notes**: <br>- focus of the work is adversarial / to jailbreak LLMs<br>- AdvBench tests whether jailbreaks succeeded

---
### AnthropicHarmlessBase
- **Year**: 2024
- **Publication Name**: [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://arxiv.org/abs/2204.05862)
- **Purpose**: broad safety
- **LLMDev Usage**: train and eval
- **Entries**: 44,849 conversational turns
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/anthropics/hh-rlhf](https://github.com/anthropics/hh-rlhf)
- **HF Link**: [https://huggingface.co/datasets/Trelis/hh-rlhf-dpo](https://huggingface.co/datasets/Trelis/hh-rlhf-dpo)
- **Notes**: <br>- most prompts created by 28 US-based crowdworkers

---
### AnthropicRedTeam
- **Year**: 2024
- **Publication Name**: [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://arxiv.org/abs/2209.07858)
- **Purpose**: broad safety
- **LLMDev Usage**: train and eval
- **Entries**: 38,961 conversations
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/anthropics/hh-rlhf](https://github.com/anthropics/hh-rlhf)
- **HF Link**: [https://huggingface.co/datasets/Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)
- **Notes**: <br>- created by 324 US-based crowdworkers<br>- ca. 80% of examples come from ca. 50 workers

---
### BAD
- **Year**: 2022
- **Publication Name**: [Bot-Adversarial Dialogue for Safe Conversational Agents](https://aclanthology.org/2021.naacl-main.235/)
- **Purpose**: broad safety
- **LLMDev Usage**: train and eval
- **Entries**: 78,874 conversations
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/bot_adversarial_dialogue](https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/bot_adversarial_dialogue)
- **HF Link**: [not available](not available)
- **Notes**: <br>- download only via ParlAI<br>- approximately 40% of all dialogues are annotated as offensive, with a third of offensive utterances generated by bots

---
### BBQ
- **Year**: 2023
- **Publication Name**: [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://aclanthology.org/2022.findings-acl.165)
- **Purpose**: bias
- **LLMDev Usage**: eval only
- **Entries**: 58,492 examples
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/nyu-mll/BBQ](https://github.com/nyu-mll/BBQ)
- **HF Link**: [https://huggingface.co/datasets/heegyu/bbq](https://huggingface.co/datasets/heegyu/bbq)
- **Notes**: <br>- focus on stereotyping behaviour<br>- covers 9 categories of bias: age, disability status, gender identity, nationality, physical appearance, race/ethnicity, religion, socioeconomic status, sexual orientation<br>- 25+ templates per category

---
### BeaverTails
- **Year**: 2022
- **Publication Name**: [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://arxiv.org/abs/2307.04657)
- **Purpose**: broad safety
- **LLMDev Usage**: train and eval
- **Entries**: 333,963 conversations
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/PKU-Alignment/beavertails](https://github.com/PKU-Alignment/beavertails)
- **HF Link**: [https://huggingface.co/datasets/PKU-Alignment/BeaverTails](https://huggingface.co/datasets/PKU-Alignment/BeaverTails)
- **Notes**: <br>- 16,851 unique prompts sampled from AnthropicRedTeam <br>- covers 14 harm categories (e.g. animal abuse)<br>- annotated for safety by 3.34 crowdworkers on average

---
### CDialBias
- **Year**: 2019
- **Publication Name**: [Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark](https://aclanthology.org/2022.findings-emnlp.262)
- **Purpose**: bias
- **LLMDev Usage**: eval only
- **Entries**: 28,343 conversations
- **Entries Type**: chat
- **Language**: Chinese
- **GitHub**: [https://github.com/para-zhou/CDial-Bias](https://github.com/para-zhou/CDial-Bias)
- **HF Link**: [not available](not available)
- **Notes**: <br>- cover 4 categories of bias: race, gender, religion, occupation

---
### CoNA
- **Year**: 2024
- **Publication Name**: [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 178 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/vinid/instruction-llms-safety-eval](https://github.com/vinid/instruction-llms-safety-eval)
- **HF Link**: [not available](not available)
- **Notes**: <br>- Focus: harmful instructions (e.g. hate speech)<br>- All prompts are (meant to be) unsafe

---
### ConfAIde
- **Year**: 2023
- **Publication Name**: [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://openreview.net/forum?id=gmg7t8b4s0)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 1,326 prompts
- **Entries Type**: other
- **Language**: English
- **GitHub**: [https://github.com/skywalker023/confAIde/tree/main/benchmark](https://github.com/skywalker023/confAIde/tree/main/benchmark)
- **HF Link**: [not available](not available)
- **Notes**: <br>- The benchmark is split into 4 tiers with different prompt formats<br>- tier 1 contains 10 prompts, tier 2 2*98, tier 3 4*270, tier 4 50

---
### ControversialInstructions
- **Year**: 2023
- **Publication Name**: [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 40 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/vinid/instruction-llms-safety-eval](https://github.com/vinid/instruction-llms-safety-eval)
- **HF Link**: [not available](not available)
- **Notes**: <br>- Focus: controversial topics (e.g. immigration)<br>- All prompts are (meant to be) unsafe

---
### CyberattackAssistance
- **Year**: 2024
- **Publication Name**: [Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models](https://arxiv.org/abs/2312.04724)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 1,000 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/facebookresearch/PurpleLlama/tree/main/CybersecurityBenchmarks/datasets/mitre](https://github.com/facebookresearch/PurpleLlama/tree/main/CybersecurityBenchmarks/datasets/mitre)
- **HF Link**: [not available](not available)
- **Notes**: <br>- instructions are split into 10 MITRE categories<br>- The dataset comes with additional LLM-rephrased instructions

---
### DecodingTrust
- **Year**: 2022
- **Publication Name**: [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://arxiv.org/abs/2306.11698)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 243,877 prompts
- **Entries Type**: multiple choice
- **Language**: English
- **GitHub**: [https://github.com/AI-secure/DecodingTrust](https://github.com/AI-secure/DecodingTrust)
- **HF Link**: [https://huggingface.co/datasets/AI-Secure/DecodingTrust](https://huggingface.co/datasets/AI-Secure/DecodingTrust)
- **Notes**: <br>- split across 8 'trustworthiness perspectives': toxicity, stereotypes, adversarial and robustness, privacy, ethics and fairness

---
### DICES350
- **Year**: 2023
- **Publication Name**: [DICES Dataset: Diversity in Conversational AI Evaluation for Safety](https://arxiv.org/abs/2306.11247)
- **Purpose**: value alignment
- **LLMDev Usage**: eval only
- **Entries**: 350 conversations
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/google-research-datasets/dices-dataset/](https://github.com/google-research-datasets/dices-dataset/)
- **HF Link**: [not available](not available)
- **Notes**: <br>- 104 ratings per item<br>- annotators from US<br>- annotation across 24 safety criteria

---
### DoAnythingNow
- **Year**: 2023
- **Publication Name**: ['Do Anything Now': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 6,387 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/verazuo/jailbreak_llms](https://github.com/verazuo/jailbreak_llms)
- **HF Link**: [not available](not available)
- **Notes**: <br>- there are 666 jailbreak prompts among the 6,387prompts

---
### DoNotAnswer
- **Year**: 2023
- **Publication Name**: [Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs](https://arxiv.org/abs/2308.13387)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 939 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/Libr-AI/do-not-answer](https://github.com/Libr-AI/do-not-answer)
- **HF Link**: [https://huggingface.co/datasets/LibrAI/do-not-answer](https://huggingface.co/datasets/LibrAI/do-not-answer)
- **Notes**: <br>- split across 5 risk areas and 12 harm types<br>- authors prompted GPT-4 to generate questions

---
### DoAnythingNow
- **Year**: 2022
- **Publication Name**: ['Do Anything Now': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 46,800 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/verazuo/jailbreak_llms](https://github.com/verazuo/jailbreak_llms)
- **HF Link**: [not available](not available)
- **Notes**: <br>- 13 scenarios with 30 questions each, then expanded by combination with 8 communities, 3 prompt setups and 5 repetations

---
### GandalfIgnoreInstructions
- **Year**: 2023
- **Publication Name**: [Gandalf Prompt Injection: Ignore Instruction Prompts](https://www.lakera.ai/blog/who-is-gandalf)
- **Purpose**: other
- **LLMDev Usage**: other
- **Entries**: 1,000 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [not available](not available)
- **HF Link**: [https://huggingface.co/datasets/Lakera/gandalf_ignore_instructions](https://huggingface.co/datasets/Lakera/gandalf_ignore_instructions)
- **Notes**: <br>- focus of the work is adversarial / prompt extraction<br>- not all prompts are attacks

---
### HackAPrompt
- **Year**: 2023
- **Publication Name**: [Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition](https://aclanthology.org/2023.emnlp-main.302/)
- **Purpose**: other
- **LLMDev Usage**: other
- **Entries**: 601,757 prompts
- **Entries Type**: chat
- **Language**: mostly English
- **GitHub**: [not available](not available)
- **HF Link**: [https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset](https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset)
- **Notes**: <br>- focus of the work is adversarial / prompt hacking<br>- prompts were written by ca. 2.8k people from 50+ countries

---
### HarmBench
- **Year**: 2022
- **Publication Name**: [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://arxiv.org/abs/2402.04249)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 400 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/centerforaisafety/HarmBench/tree/main/data/behavior_datasets](https://github.com/centerforaisafety/HarmBench/tree/main/data/behavior_datasets)
- **HF Link**: [not available](not available)
- **Notes**: <br>- The dataset covers 7 semantic categories of behaviour: Cybercrime & Unauthorized Intrusion, Chemical & Biological Weapons/Drugs, Copyright Violations, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, and General Harm<br>- The dataset also includes 110 multimodal prompts

---
### HarmfulQ
- **Year**: 2021
- **Publication Name**: [On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://aclanthology.org/2023.acl-long.244/)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 200 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/SALT-NLP/chain-of-thought-bias](https://github.com/SALT-NLP/chain-of-thought-bias)
- **HF Link**: [not available](not available)
- **Notes**: <br>- focus on 6 attributes: "racist, stereotypical, sexist, illegal, toxic, harmful"<br>- authors do manual filtering for overly similar questions

---
### RedEval
- **Year**: 2023
- **Publication Name**: [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://arxiv.org/abs/2308.09662)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 1,960 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/declare-lab/red-instruct/tree/main/harmful_questions](https://github.com/declare-lab/red-instruct/tree/main/harmful_questions)
- **HF Link**: [https://huggingface.co/datasets/declare-lab/HarmfulQA](https://huggingface.co/datasets/declare-lab/HarmfulQA)
- **Notes**: <br>- split into 10 topics (e.g. "Mathematics and Logic")<br>- similarity across prompts is quite high<br>- not all prompts are unsafe / safety-related

---
### HExPHI
- **Year**: 2024
- **Publication Name**: [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://openreview.net/forum?id=hTEGyKf0dZ)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 330 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [not available](not available)
- **HF Link**: [https://huggingface.co/datasets/LLM-Tuning-Safety/HEx-PHI](https://huggingface.co/datasets/LLM-Tuning-Safety/HEx-PHI)
- **Notes**: <br>- Main focus is on finetuning models<br>- Prompts cover 11 harm areas

---
### HolisticBias
- **Year**: 2023
- **Publication Name**: [I’m sorry to hear that: Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://aclanthology.org/2022.emnlp-main.625)
- **Purpose**: bias
- **LLMDev Usage**: eval only
- **Entries**: 459,758 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/facebookresearch/ResponsibleNLP/tree/main/holistic_bias](https://github.com/facebookresearch/ResponsibleNLP/tree/main/holistic_bias)
- **HF Link**: [not available](not available)
- **Notes**: <br>- 26 sentence templates<br>- covers 13 categories of bias: ability, age, body type, characteristics, culturural, gender/sex, nationality, nonce, political, race/ethnicity, religion, sexual orientation, socioeconomic

---
### HypothesisStereotypes
- **Year**: 2023
- **Publication Name**: [Analyzing Stereotypes in Generative Text Inference Tasks](https://aclanthology.org/2021.findings-acl.355/)
- **Purpose**: bias
- **LLMDev Usage**: eval only
- **Entries**: 2,098 prompts
- **Entries Type**: other
- **Language**: English
- **GitHub**: [https://github.com/AnnaSou/stereotypes_generative_inferences](https://github.com/AnnaSou/stereotypes_generative_inferences)
- **HF Link**: [not available](not available)
- **Notes**: <br>- uses 103 context situations as templates<br>- covers 6 categories of bias: gender, race, nationality, religion, politics, socio<br>- task for LLM is to generate hypothesis based on premise

---
### LatentJailbreak
- **Year**: 2024
- **Publication Name**: [Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models](https://arxiv.org/abs/2307.08487)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 416 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/qiuhuachuan/latent-jailbreak/tree/main](https://github.com/qiuhuachuan/latent-jailbreak/tree/main)
- **HF Link**: [not available](not available)
- **Notes**: <br>- focus of the work is adversarial / to jailbreak LLMs<br>- 13 prompt templates instantiated with 16 protected group terms and 2 posititional types<br>- main exploit focuses on translation

---
### MaliciousInstruct
- **Year**: 2022
- **Publication Name**: [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://arxiv.org/abs/2310.06987)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 100 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/Princeton-SysML/Jailbreak_LLM/tree/main/data](https://github.com/Princeton-SysML/Jailbreak_LLM/tree/main/data)
- **HF Link**: [not available](not available)
- **Notes**: <br>- covers ten 'malicious intentions': psychological manipulation, sabotage, theft, defamation, cyberbullying, false accusation, tax fraud, hacking, fraud, and illegal drug use.

---
### MaliciousInstructions
- **Year**: 2024
- **Publication Name**: [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 100 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/vinid/instruction-llms-safety-eval](https://github.com/vinid/instruction-llms-safety-eval)
- **HF Link**: [not available](not available)
- **Notes**: <br>- Focus: malicious instructions (e.g. bombmaking)<br>- All prompts are (meant to be) unsafe

---
### MIC
- **Year**: 2024
- **Publication Name**: [The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems](https://aclanthology.org/2022.acl-long.261)
- **Purpose**: value alignment
- **LLMDev Usage**: eval only
- **Entries**: 38,000 conversations
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/SALT-NLP/mic](https://github.com/SALT-NLP/mic)
- **HF Link**: [not available](not available)
- **Notes**: <br>- based on the RoT paradigm introduced in SocialChemistry<br>- 38k prompt-reply pairs come with 99k rules of thumb and 114k annotations

---
### MoralChoice
- **Year**: 2023
- **Publication Name**: [Evaluating the Moral Beliefs Encoded in LLMs](https://openreview.net/forum?id=O06z2G18me)
- **Purpose**: value alignment
- **LLMDev Usage**: eval only
- **Entries**: 1,767 binary-choice question
- **Entries Type**: multiple choice
- **Language**: English
- **GitHub**: [https://github.com/ninodimontalcino/moralchoice](https://github.com/ninodimontalcino/moralchoice)
- **HF Link**: [https://huggingface.co/datasets/ninoscherrer/moralchoice](https://huggingface.co/datasets/ninoscherrer/moralchoice)
- **Notes**: <br>- 687 scenarios are low-ambiguity, 680 are high-ambiguity<br>- three Surge annotators choose the favourable action for each scenario

---
### DialogueSafety
- **Year**: 2021
- **Publication Name**: [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://aclanthology.org/D19-1461)
- **Purpose**: broad safety
- **LLMDev Usage**: train and eval
- **Entries**: 90,000 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/dialogue_safety](https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/dialogue_safety)
- **HF Link**: [not available](not available)
- **Notes**: <br>- download only via ParlAI<br>- the "single-turn" dataset provides a "standard" and "adversarial" setting, with 3 rounds of data collection each

---
### PersonalInfoLeak
- **Year**: 2024
- **Publication Name**: [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://aclanthology.org/2022.findings-emnlp.148)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 3,238 entries
- **Entries Type**: other
- **Language**: English
- **GitHub**: [https://github.com/jeffhj/LM_PersonalInfoLeak](https://github.com/jeffhj/LM_PersonalInfoLeak)
- **HF Link**: [not available](not available)
- **Notes**: <br>- main task is to predict email given name

---
### PhysicalSafetyInstructions
- **Year**: 2023
- **Publication Name**: [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 1,000 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/vinid/instruction-llms-safety-eval](https://github.com/vinid/instruction-llms-safety-eval)
- **HF Link**: [not available](not available)
- **Notes**: <br>- Focus: commonsense physical safety<br>- 50 safe and 50 unsafe prompts

---
### PromptExtractionRobustness
- **Year**: 2023
- **Publication Name**: [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://arxiv.org/abs/2311.01011)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 569 samples
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/HumanCompatibleAI/tensor-trust-data](https://github.com/HumanCompatibleAI/tensor-trust-data)
- **HF Link**: [https://huggingface.co/datasets/qxcv/tensor-trust?row=0](https://huggingface.co/datasets/qxcv/tensor-trust?row=0)
- **Notes**: <br>- filtered from larger raw prompt extraction dataset<br>- collected using the open Tensor Trust online game

---
### PromptHijackingRobustness
- **Year**: 2021
- **Publication Name**: [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://arxiv.org/abs/2311.01011)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 775 samples
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/HumanCompatibleAI/tensor-trust-data](https://github.com/HumanCompatibleAI/tensor-trust-data)
- **HF Link**: [https://huggingface.co/datasets/qxcv/tensor-trust?row=0](https://huggingface.co/datasets/qxcv/tensor-trust?row=0)
- **Notes**: <br>- filtered from larger raw prompt extraction dataset<br>- collected using the open Tensor Trust online game

---
### QHarm
- **Year**: 2023
- **Publication Name**: [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://arxiv.org/abs/2309.07875)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 100 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/vinid/instruction-llms-safety-eval](https://github.com/vinid/instruction-llms-safety-eval)
- **HF Link**: [not available](not available)
- **Notes**: <br>- Wider topic coverage due to source dataset<br>- Prompts are mostly unsafe

---
### RedditBias
- **Year**: 2024
- **Publication Name**: [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://aclanthology.org/2021.acl-long.151)
- **Purpose**: bias
- **LLMDev Usage**: eval only
- **Entries**: 11,873 Reddit comments
- **Entries Type**: other
- **Language**: English
- **GitHub**: [https://github.com/umanlp/RedditBias](https://github.com/umanlp/RedditBias)
- **HF Link**: [not available](not available)
- **Notes**: <br>- covers 4 categories of bias: religion, race, gender, queerness<br>- evaluation by perplexity and conversation

---
### RuLES
- **Year**: 2024
- **Publication Name**: [Can LLMs Follow Simple Rules?](https://arxiv.org/abs/2311.04235)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 862 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/normster/llm_rules](https://github.com/normster/llm_rules)
- **HF Link**: [https://huggingface.co/datasets/normster/RuLES](https://huggingface.co/datasets/normster/RuLES)
- **Notes**: <br>- The dataset covers 19 rules across 14 scenarios

---
### SafeText
- **Year**: 2022
- **Publication Name**: [SafeText: A Benchmark for Exploring Physical Safety in Language Models](https://aclanthology.org/2022.emnlp-main.154/)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 367 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/sharonlevy/SafeText](https://github.com/sharonlevy/SafeText)
- **HF Link**: [not available](not available)
- **Notes**: <br>- 5 ratings for relevance per item during filtering<br>- advice format most often elicits yes/no answer

---
### SafetyPrompts
- **Year**: 2023
- **Publication Name**: [Safety Assessment of Chinese Large Language Models](https://arxiv.org/abs/2304.10436)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 100,000 prompts
- **Entries Type**: chat
- **Language**: Chinese
- **GitHub**: [https://github.com/thu-coai/Safety-Prompts](https://github.com/thu-coai/Safety-Prompts)
- **HF Link**: [https://huggingface.co/datasets/thu-coai/Safety-Prompts](https://huggingface.co/datasets/thu-coai/Safety-Prompts)
- **Notes**: <br>- covers 8 safety scenarios and 6 types of adv attack<br>- do not release 'sensitive topics' scenario

---
### SimpleSafetyTests
- **Year**: 2024
- **Publication Name**: [SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models](https://arxiv.org/abs/2311.08370)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 100 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/bertiev/SimpleSafetyTests](https://github.com/bertiev/SimpleSafetyTests)
- **HF Link**: [https://huggingface.co/datasets/Bertievidgen/SimpleSafetyTests](https://huggingface.co/datasets/Bertievidgen/SimpleSafetyTests)
- **Notes**: <br>- The dataset is split into ten types of prompts

---
### SPMisconceptions
- **Year**: 2022
- **Publication Name**: [Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptions](https://dl.acm.org/doi/10.1145/3627106.3627196)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 122 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/purseclab/LLM_Security_Privacy_Advice](https://github.com/purseclab/LLM_Security_Privacy_Advice)
- **HF Link**: [not available](not available)
- **Notes**: <br>- misconceptions all relate to security and privacy <br>- uses templates to turn misconceptions into prompts<br>- covers six categories (e.g. crypto and blockchain, law and regulation)<br>

---
### StrongREJECT
- **Year**: 2023
- **Publication Name**: [A StrongREJECT for Empty Jailbreaks](https://arxiv.org/pdf/2402.10260.pdf)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 346 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/alexandrasouly/strongreject/tree/main](https://github.com/alexandrasouly/strongreject/tree/main)
- **HF Link**: [not available](not available)
- **Notes**: <br>- The focus of the work is adversarial / to jailbreak LLMs<br>- The 6 question categories are: disinformation/deception, hate/harassment/discrimination, illegal goods/services, non-violent crimes, sexual content, violence

---
### SycophancyEval
- **Year**: 2024
- **Publication Name**: [Towards Understanding Sycophancy in Language Models](https://openreview.net/forum?id=tvhaxkMKAn)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 20,956 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/meg-tong/sycophancy-eval/tree/main](https://github.com/meg-tong/sycophancy-eval/tree/main)
- **HF Link**: [https://huggingface.co/datasets/meg-tong/sycophancy-eval](https://huggingface.co/datasets/meg-tong/sycophancy-eval)
- **Notes**: <br>- Dataset uses four different task setups to evaluate sycophancy: answer (7268 prompts), are_you_sure (4888 prompts), feedback (8500 prompts), mimicry (300 prompts)

---
### TDCRedTeaming
- **Year**: 2022
- **Publication Name**: [TDC 2023 (LLM Edition): The Trojan Detection Challenge](https://trojandetection.ai/)
- **Purpose**: broad safety
- **LLMDev Usage**: eval only
- **Entries**: 100 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/centerforaisafety/tdc2023-starter-kit/tree/main/red_teaming/data](https://github.com/centerforaisafety/tdc2023-starter-kit/tree/main/red_teaming/data)
- **HF Link**: [not available](not available)
- **Notes**: <br>- covers 7 categories: Bigotry and Abusive Language, Violent Content and Conduct, Illegal Activities, Malware and Exploits, Scams, Misinformation and Disinformation, Other Undesirable Content

---
### ToxiGen
- **Year**: 2023
- **Publication Name**: [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://aclanthology.org/2022.acl-long.234/)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 260,851 prompts
- **Entries Type**: other
- **Language**: English
- **GitHub**: [https://github.com/microsoft/TOXIGEN](https://github.com/microsoft/TOXIGEN)
- **HF Link**: [https://huggingface.co/datasets/skg/toxigen-data](https://huggingface.co/datasets/skg/toxigen-data)
- **Notes**: <br>- covers 13 target groups<br>- seed prompts are used to generate implicit hate<br>- evaluating generative LLMs is not the focus

---
### TruthfulQA
- **Year**: 2022
- **Publication Name**: [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://aclanthology.org/2022.acl-long.229/)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 817 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/sylinrl/TruthfulQA](https://github.com/sylinrl/TruthfulQA)
- **HF Link**: [https://huggingface.co/datasets/truthful_qa](https://huggingface.co/datasets/truthful_qa)
- **Notes**: <br>- covers 38 categories (e.g. health and politics)<br>- comes with multiple choice expansion

---
### XSTest
- **Year**: 2023
- **Publication Name**: [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://arxiv.org/abs/2308.01263)
- **Purpose**: narrow safety
- **LLMDev Usage**: eval only
- **Entries**: 450 prompts
- **Entries Type**: chat
- **Language**: English
- **GitHub**: [https://github.com/paul-rottger/exaggerated-safety](https://github.com/paul-rottger/exaggerated-safety)
- **HF Link**: [https://huggingface.co/datasets/natolambert/xstest-v2-copy](https://huggingface.co/datasets/natolambert/xstest-v2-copy)
- **Notes**: <br>- split into ten types of prompts<br>- 250 safe prompts and 200 unsafe prompts

---
