REVIEW 14 cited by
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics, and then composes multiple tactics for systematic exploration of novel jailbreaks. Compared to prior work that performed red-teaming via recruited human workers, gradient-based optimization, or iterative revision with LLMs, our work investigates jailbreaks from chatbot users who were not specifically instructed to break the system. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks compared to state-of-the-art jailbreak methods. While many datasets exist for jailbreak evaluation, very few open-source datasets exist for jailbreak training, as safety training data has been closed even when model weights are open. With WildTeaming we create WildJailbreak, a large-scale open-source synthetic safety dataset with 262K vanilla (direct request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate exaggerated safety behaviors, WildJailbreak provides two contrastive types of queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that resemble harmful queries in form but contain no harm. As WildJailbreak considerably upgrades the quality and scale of existing safety resources, it uniquely enables us to examine the scaling effects of data and the interplay of data properties and model capabilities during safety training. Through extensive experiments, we identify the training properties that enable an ideal balance of safety behaviors: appropriate safeguarding without over-refusal, effective handling of vanilla and adversarial queries, and minimal, if any, decrease in general capabilities. All components of WildJailbeak contribute to achieving balanced safety behaviors of models.
Forward citations
Cited by 14 Pith papers
-
Synthetic Persona Pretraining: Alignment from Token Zero
Injecting first-person, value-laden reflections into pretraining text improves constitution following, jailbreak resistance, and out-of-distribution moral choices in small language models, with the largest gains when ...
-
Safety Alignment of LMs via Non-cooperative Games
Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.
-
When Are Sparse Feature Interventions Actually Localized? Matched Evaluation for SAE-Based Safety Control
SAE safety ablations are regime-dependent and baseline-dependent: medium-k heads can look efficient, but surface-matched dense steering often beats them and high-k collapses coherence.
-
The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators
Fixed activation probes keep near-ceiling accuracy on harmful-vs-benign corpus contrasts but fall to AUROC 0.59-0.69 on topic- and surface-matched harmful/benign pairs, so they behave as broad-risk detectors, not cont...
-
Persona Cartography: Charting Language Model Personality Traits in Weight Space
Composable LoRA adapters can amplify or suppress OCEAN traits in LLMs, combine approximately additively, preserve moderate-scale capability, and move safety-relevant behaviours.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
Granite Guardian
Granite Guardian 2B and 8B are open-source LLM guardrails that detect harmful content, jailbreaks, and RAG hallucination risks, reporting AUC 0.871 on harm benchmarks and 0.854 on groundedness benchmarks.
-
Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models
A benchmark of 500 camouflaged jailbreak prompts finds open-weight LLMs comply with 94% of harmful requests, but the result is confounded by task complexity and an overly permissive compliance metric.
-
Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.
-
Don't Command, Cultivate: An Exploratory Study of System-2 Alignment
Encouraging LLMs to analyze user requests step-by-step (System-2 Alignment) modestly improves safety on open-source models, but with trade-offs and limited evidence.
-
Steering Language Model Refusal with Sparse Autoencoders
Boosting one SAE 'refusal' feature in Phi-3 Mini and Llama 3.1 raises refusal rates on unsafe and safe prompts alike while sharply reducing MMLU, TruthfulQA, and GSM8K accuracy.
-
Sentinel: SOTA model to protect against prompt injections
Sentinel, a ModernBERT-based binary classifier trained on public and private prompt datasets, reports 0.987 accuracy and 0.980 F1 on a held-out internal test set and outperforms one baseline on public benchmarks.
-
Defense Against the Dark Prompts: Mitigating Best-of-N Jailbreaking with Prompt Evaluation
An iterative LLM-based prompt evaluator blocked 100% of the Best-of-N jailbreaking paper's released successful prompts and 99.8% of a fresh replication, with false-positive rates near zero.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Continue with ORCID to comment.