Safety datasets train the classifiers and refusals that sit between a model and its users. They are also the category where the gap between benchmark performance and production performance is widest.
This category covers several distinct jobs. Toxicity and harmful-content sets train classifiers that flag inputs or outputs. Jailbreak and adversarial-prompt collections test whether a model's refusals survive deliberate attack. Machine-generated-text detection sets train classifiers that try to distinguish model output from human writing — a task where published accuracy figures have generally not survived contact with newer models.
Two cautions. Safety labels are value judgements and the labelling guidelines matter enormously, so a classifier trained on one set will disagree with one trained on another in ways that are not errors. And adversarial sets age quickly: a jailbreak collection assembled against last year's models tells you little about this year's, which makes freshness a more important property here than scale.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| Civil Comments - Corpus of moderated comments annotated for toxicity | Text | 2M comments | CC0 | Toxicity detection and automated moderation |
| Human vs LLM Text Corpus - Generated Text Detection | Text | 790,000 entries | MIT | Automatic AI-versus-human text detection |
| Deepfake and Real Images Dataset | Image | 190,000 images | CC BY 4.0 | Deepfake detection |
| PolyGuard | Text | 136,800 examples | CC BY 4.0 | Detect bias and sensitive content |
| Phishing Email Dataset | Text | 82,500 emails | CC BY-SA 4.0 | Automatic phishing and spam detection |
| Cybersecurity Heimdall v1.1 | Text | 21,258 dialogues | Apache 2.0 | Language models specialised in defensive cybersecurity |
| Apt-eval - Detection of texts reworked by AI | Text | 14,700 samples | MIT | Robustness of AI text detectors |
| Web Page Phishing Detection Dataset | Text | 11,430 entries | CC BY 4.0 | Machine learning to detect phishing sites |
| MAPS - Multilingual Agentic Performance & Security | Text | 805 tasks | MIT | Comparing AI agent performance |
| SMS Spam Collection | Text | 5,574 messages | CC BY 4.0 | Text classification for spam filtering |
| BullshiTeval - Assessing the reliability of AI assistants | Text | 2,400 examples | MIT | LLMs avoid inconsistent or erroneous responses |
| H4rmony DPO | Text | 2,016 examples | MIT | Language models using DPO |
Civil Comments - Corpus of moderated comments annotated for toxicity
- Size
- Approximately 2 million text comments in JSON format, with toxicity labels and types of abuse
- Licence
- CC0-1.0
Roughly two million English public comments in JSON, gathered from a news-site commenting plugin between 2015 and 2017, labelled for overall toxicity and for specific kinds of abuse. Public domain and large.
Training toxicity detection and automated moderation, analysing hostile online interaction dynamics, and testing multi-class and multi-label classification on long texts.
Source: innovatiana.com
Human vs LLM Text Corpus - Generated Text Detection
- Size
- Approximately 790,000 text entries, CSV format
- Licence
- MIT
Roughly 790,000 CSV entries pairing human-written against LLM-generated text, more than 788,000 examples in all. Among the largest open corpora for machine-text detection and comparative stylometry.
Training automatic AI-versus-human text detection, analysing stylistic and structural differences between sources, and evaluating detector robustness across contexts.
Source: innovatiana.com
Deepfake and Real Images Dataset
- Size
- Approximately 190,000 256x256 pixel JPG images, real faces and deepfake
- Licence
- Creative Commons Attribution 4.0 International (CC BY 4.0)
Roughly 190,000 JPG face images at 256x256, mixing real faces with ones altered by various deepfake techniques. Large, uniform in size and CC BY licensed, making it the most convenient deepfake image set here, though the real faces are still identifiable people.
Developing and training deepfake detection, improving facial recognition security and fraud prevention, and studying generated and manipulated image techniques.
Source: innovatiana.com
PolyGuard
- Size
- Approximately 136,800 text examples annotated as lines in JSON/text format
- Licence
- CC-BY 4.0
Roughly 136,800 instructions and prompts in JSON or line-based text, annotated for bias or danger with labels such as unsafe. Sizeable enough to train a moderation classifier rather than only evaluate one.
Training models to detect bias and sensitive content, improving automated moderation of AI-generated content, and contributing to conversational AI safety.
Source: innovatiana.com
Phishing Email Dataset
- Size
- Approximately 82,500 emails in text format, with metadata (sender, recipient, date), annotated spam/phishing or legitimate
- Licence
- CC BY-SA 4.0
Roughly 82,500 emails in text form drawn from sources including Enron, Ling and SpamAssassin, each labelled phishing or legitimate and retaining subject, body, sender, recipient and date metadata.
Training automatic phishing and spam detection from text, studying tactics of fraudulent emails, and developing email filtering and security tools.
Source: innovatiana.com
Cybersecurity Heimdall v1.1
- Size
- 21,258 system/user/assistant dialogues in Parquet
- Licence
- Apache 2.0
21,258 system, user and assistant dialogue triples in Parquet for defensive cybersecurity training, distilled from more than 100,000 public technical sources. Framed around defence rather than offensive capability.
Training language models specialised in defensive cybersecurity, improving ethical alignment on sensitive technical questions, and benchmarking security QA and synthesis.
Source: innovatiana.com
Apt-eval - Detection of texts reworked by AI
- Size
- About 14,700 samples: 300 human-written texts plus some 14,400 LLM-polished versions, CSV/JSON
- Licence
- MIT
About 14,700 samples in CSV and JSON: 300 human-written texts from six domains plus some 14,400 versions of them polished by one of five LLMs (GPT-4o, Llama 3.1 70B, Llama 3 8B, Llama 2 7B, DeepSeek-V3), each labelled by which model polished it, how much, and what kind of edit. Targets the hybrid-text blind spot.
Evaluating robustness of AI text detectors against minimal LLM edits, comparing GPT-4o, Llama and DeepSeek polishing strategies, and classifying hybrid texts.
Source: innovatiana.com
Web Page Phishing Detection Dataset
- Size
- 11,430 entries with 87 columns, tabular CSV format
- Licence
- CC BY 4.0
11,430 balanced CSV entries labelled phishing or legitimate, each described by 87 engineered features spanning URL structure, page content and external service signals. Richer feature set than URL-only alternatives.
Developing machine learning to detect phishing sites, evaluating robustness of web security against modern threats, and building automatic suspicious-URL analysis.
Source: innovatiana.com
MAPS - Multilingual Agentic Performance & Security
- Size
- 805 tasks, each in English and 11 translations (9,660 JSON instances)
- Licence
- MIT
805 agent tasks assembled from GAIA, MATH, SWE-Bench and ASB, each in English and 11 translations (9,660 JSON instances in all), for testing both the capability and the security of AI agents outside English. Fujitsu-published under MIT.
Comparing AI agent performance in multilingual contexts, testing robustness and security against adversarial inputs, and evaluating cross-language generalisation.
Source: innovatiana.com
SMS Spam Collection
- Size
- 5,574 SMS messages, plain text format (TXT/CSV)
- Licence
- CC BY 4.0
About 5,574 text messages in TXT and CSV labelled spam or ham, gathered from several sources for spam filtering research. Hosted at the UCI repository and small enough to run experiments on in seconds.
Training text classification for spam filtering, NLP research applied to SMS, and evaluating clustering and text analysis techniques.
Source: innovatiana.com
BullshiTeval - Assessing the reliability of AI assistants
- Size
- 2,400 text examples in JSON format
- Licence
- MIT
2,400 JSON scenarios spread across 100 AI assistant models, mixing general questions, flattery tests and negatively framed concerns, each with context, system prompt, question and test type. A sycophancy and reliability probe.
Measuring whether LLMs avoid inconsistent or erroneous responses, evaluating behaviour across prompt types, and building AI reliability benchmarks.
Source: innovatiana.com
H4rmony DPO
- Size
- 2,016 examples in Parquet format (204 kB converted)
- Licence
- MIT
2,016 structured examples in Parquet, about 204 kB converted, derived from the H4rmony corpus and pre-formatted for Direct Preference Optimization with the DPOTrainer in trl. Tiny, aimed at ecological alignment preferences.
Training language models using DPO, optimising LLMs for specific tasks with human feedback, and testing advanced fine-tuning methods in reinforcement learning.
Source: innovatiana.com
How this list was put together
Shortlisted from the 25 datasets in Dataset Finder's verified set of 434 open datasets tagged “NLP: safety, moderation & AI-text detection”. Where the theme holds more than twelve, we prefer datasets whose licence is stated unambiguously and whose published scale is largest, so that the list is usable rather than merely long.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts, the entry gives the original terms and says so. The comparison table abbreviates licences to standard identifiers and marks vague ones as unclear.