REVIEW 14 cited by
LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.
Forward citations
Cited by 14 Pith papers
-
SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks
Multi-turn LLM jailbreaks succeed based on how harmful intent is organized across turns, not on interaction length, and detection should shift to session and cross-session scope.
-
Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
Prompt-level guardrails on GUI agents are local: effective on single explicit requests, but four-turn escalation raises guarded attack success by ~20 points and concealed requests become more successful than explicit ones.
-
Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
Adaptive multi-turn LLM attacks raise attack success from near 0% to 5.4–14.0% on frontier defenders, with scenario-specific defender weaknesses that aggregate scores hide.
-
Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation
Cross-session decomposition of harmful goals raises attack success from 18.7% to 37.4% across nine models, and a capability-accumulation detector (Magnet) beats per-session and compression baselines at flagging such l...
-
AMT-X: Phase-Structured Multi-Turn Red-Teaming with Checklist-Gated Evaluation
A phase-structured multi-turn red-team framework reports 97.6–100% lenient ASR but only 66.7–78.6% full actionable ASR on six frontier LLMs, with success strongly depth-dependent.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
Learning from Mistakes: Can LLM Self-Recover after Misalignment?
LLMs sometimes regain safe behavior after multi-turn jailbreaks, and this recovery can be measured with turn-level safety trajectories and metrics such as misalignment length and recovery duration.
-
SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses
A systemization of LLM jailbreak security that adds linked taxonomies, an evaluation platform, and JailbreakDB, while its main attack–defense comparison results remain deferred.
-
Reliable Weak-to-Strong Monitoring of LLM Agents
Monitor scaffolding, not monitor awareness or omniscience, drives detection reliability, and a hybrid chunked monitor lets weak models supervise strong LLM agents.
-
FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
A new benchmark with instance-specific rubrics measures frontier LLMs' willingness to assist with national security and public safety threats, alongside a paired over-refusal test.
-
The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
AI safety should be measured by whether deployed systems keep errors visible, contestable, containable, and recoverable across five integrity layers, not only by whether individual model outputs look safe.
-
Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
Fine-tuning an LLM on synthetic toxic dialogues makes it harass in 95–97% of multi-turn conversations in Llama and ~99% in Gemini; memory and planning attacks also raise closed-source vulnerability.
-
Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment
A fixed ReLU penalty and a semantically labeled cost model are proposed to make RLHF safer, but the 'certifiable' guarantee is not fully supported.
-
A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks
Crescendo multi-turn jailbreak responses are represented by safety-tuned LLMs as benign rather than harmful, which helps explain why single-turn defenses fail.
Discussion (0). Sign in to comment.