REVIEW 20 cited by
Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.
Forward citations
Cited by 20 Pith papers
-
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
Online self-play between attacker and defender roles of a single LLM improves safety robustness and attack diversity across Llama and Qwen models.
-
Efficient Safety Alignment of Language Models via Latent Personality Traits
Latent adversarial training on 66 harm-agnostic Big-Five personality statements yields near-zero HarmBench ASR across direct requests and five jailbreaks while preserving utility.
-
A clinically validated framework for auditing AI chatbot behavior in mental health interactions
Using simulated psychiatric user profiles, the authors show that AI chatbots frequently produce 'concerning behavior' that accumulates over turns, and that superficially supportive responses can amplify vulnerability—...
-
PL-Guard: Benchmarking Language Model Safety for Polish
A small Polish BERT classifier proved more robust than larger fine-tuned LLMs at classifying safe versus unsafe Polish content, including under character-level adversarial perturbations.
-
Intent Factored Generation: Unleashing the Diversity in Your Language Model
Intent Factored Generation samples a high-temperature intent, such as keywords or a summary, and then samples the final response at lower temperature conditioned on that intent, increasing semantic diversity while kee...
-
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models
QDRT combines behavior-conditioned RL, multiple specialized attackers, and a MAP-Elites replay buffer to generate LLM attacks that are more toxic and cover more risk-category/style combinations.
-
Adversarial Reasoning at Jailbreaking Time
A loss-guided 'reason, verify, search' loop with three LLM modules surpasses prior semantic jailbreak methods on several defended models.
-
BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
BARE generates diverse, high-quality synthetic training data from only three seed examples by having a base model draft and an instruction-tuned model refine, improving downstream fine-tuning accuracy in few-shot settings.
-
Scaling Policy Gradient Quality-Diversity with Massive Parallelization via Behavioral Variations
ASCII-ME replaces actor-critic updates in policy-gradient MAP-Elites with reward-weighted interpolation between action sequences, mapped to policy parameters through a Jacobian, enabling fast GPU-parallel quality-diversity.
-
Evolution and The Knightian Blindspot of Machine Learning
ML's formalisms, particularly RL's, exclude Knightian uncertainty, and evolution's diversify-and-filter mechanisms point toward a direct remedy.
-
ToxSearch: Evolving Prompts for Toxicity Search in Large Language Models
Prompts evolved with small lexical mutations elicit toxic output from aligned LLMs, and these prompts transfer across models with toxicity reduced by roughly 30–54%.
-
Fast Proxies for LLM Robustness Evaluation
Simple prompt-based and embedding-space attacks predict, with rank correlations up to 0.94, how open-source LLMs fare against a six-attack red-teaming ensemble, at roughly one thousandth of the compute.
-
Salamandra Technical Report
Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.
-
Extract-QD Framework: A Generic Approach for Quality-Diversity in Noisy, Stochastic or Uncertain Domains
A modular 'Extract-QD' framework and a new Extract-ME algorithm that re-evaluates archive elites, consistently matching or outperforming previous uncertain-QD methods on standard benchmarks.
-
KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs
A distilled open-source attacker, KDA, imitates three jailbreak methods to write diverse attack prompts, and reports higher success and efficiency than each teacher.
-
Generative Data Refinement: Just Ask for Better Data
A pretrained LLM can rewrite individual data samples to strip out PII or toxic content while preserving useful information, creating safer training data.
-
JavelinGuard: Low-Cost Transformer Architectures for LLM Security
A study of five small transformer classifier architectures for LLM jailbreak and prompt injection detection claims low-latency accuracy comparable to large models, led by the multi-task Raudra design.
-
Adversarial Preference Learning for Robust LLM Alignment
APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.
-
Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects
A position paper argues that understanding AI's second-order effects requires moving from static benchmarks to an ecosystem of field testing, red teaming, and contextual evaluation.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
Discussion (0). Continue with ORCID to comment.