Pith. sign in

REVIEW 4 major objections 4 minor 66 references

The authors argue that a stateful cooperative multi-agent defense — delaying, misleading, and analyzing attackers across turns — can reduce jailbreak success and force adversaries to spend far more resources than reactive refusal-based defe

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:10 UTC pith:5FDO2CMG

load-bearing objection CoopGuard's multi-agent design and EMRA dataset are useful contributions, but the headline defense numbers rest on a scripted, non-adaptive evaluation and a deception rubric that mirrors the defense's own objective. the 4 major comments →

arxiv 2608.00134 v1 pith:5FDO2CMG submitted 2026-07-31 cs.CR

Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks

Siyuan Li , Zehao Liu , Haoyu Li , Xi Lin , Ning Liu , Jun Wu , Jianhua Li , Mohsen Guizani This is my paper
classification cs.CR
keywords LLM safetymulti-turn jailbreak defensemulti-agent defenseadversarial robustnessdeceptive defenseEMRA datasetattack coststateful defense
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-turn jailbreak attacks evolve: each query is a fresh attempt, but later prompts exploit feedback from earlier refusals. Static defenses that simply refuse or filter each turn leak hints and let attackers retry cheaply. This paper claims that a cooperative team of specialized LLM agents — one that delays suspicious queries, one that feeds attackers plausible but useless decoy answers, one that maintains a forensic log, and a coordinator that chooses between monitoring, delaying, misdirecting, refusing, and escalating — keeps harmful outputs near zero while forcing attackers to burn tokens and time. On a new 5,200-sample benchmark of escalating attacks across three backbones, the authors report that the system cuts attack success by 69% on average relative to five baselines, produces deceptive engagements at more than six times the strongest baseline's rate, and raises attacker token consumption by about 199%. If true, this shifts defense from blocking individual attempts to exhausting the adversary across an entire episode.

Core claim

The paper's central claim is that independent-yet-evolving multi-turn jailbreaks are best countered not by per-turn refusal but by a persistent, stateful defense that controls what the attacker learns at each step. CoopGuard partitions the job among four agents: the Deferring Agent grades each query's malicious potential and inserts calibrated delays or vagueness; the Tempting Agent replies with elaborate but harmless decoy content that keeps the attacker engaged on a dead path; the Forensic Agent summarizes attack patterns from stored dialogue; and the System Agent integrates these signals, updates role instructions, and picks a turn-level action from a fixed set. The authors report that th

What carries the argument

The carrying mechanism is the stateful cooperative multi-agent defense itself: each agent receives the current query, the dialogue history, and the forensic evidence, and the System Agent reconfigures role instructions rather than fine-tuning the base model, so adaptation happens through state. The Tempting Agent's deliberately elaborate but harmless decoys, combined with the Deferring Agent's controlled delays and the Forensic Agent's evidence reports, create a loop that consumes attacker tokens and leads attackers down unproductive paths while preserving an explicit audit trace.

Load-bearing premise

The load-bearing premise is that the LLM judge's 'score 2 — successful misdirection' reliably identifies genuine deception, since the Tempting Agent is explicitly instructed to produce exactly that behavior; if the judge is fooled by evasive or low-quality responses, the deception-rate and attacker-token-cost conclusions partly evaporate.

What would settle it

Take the conversation logs from the deception-rate experiments and have an independent panel (or a second, differently-prompted judge) label each response as refusal, evasion, genuine deception, or harmful content without seeing the defense's labels. If the panel classifies most 'successful misdirection' responses as unhelpful dodging or as still leaking information, the deception and resource-cost claims fail; if the panel confirms they are genuinely misleading and non-harmful, the claims hold. A second check: run an attacker explicitly told to ignore decoy content and re-ask the original har

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If these results hold, defending multi-turn jailbreaks no longer requires choosing between blocking and engaging: a defense can be simultaneously safer and more resource-draining for the attacker.
  • The reported longitudinal results suggest that stateless baselines degrade as conversations reach 20 turns, while the proposed framework keeps attack success stable — pointing to statefulness itself, not any single prompt, as the load-bearing advantage.
  • The EMRA benchmark provides a standard way to measure sustained robustness under escalation, separate from single-turn attack success.
  • The ~20% throughput reduction is presented as a deliberate defensive feature, meaning defender-side latency can be repurposed as an attacker-cost signal in real deployments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because adaptation is prompt-level, the same agent stack could in principle be ported to open-weights models by swapping role instructions, without retraining — a testable extension the paper does not run.
  • The deceptive-rate metric rests on the judge's 'successful misdirection' rubric; if a stricter judge or human users classify decoys as merely evasive rather than genuinely deceptive, the DR advantage would shrink. The paper's own cross-validation with a second judge and human annotators is evidence, but the primary rubric remains the key contingency.
  • A natural next experiment is to pit the framework against an adaptive attacker that is told a decoy may be present — e.g., an attacker LLM instructed to recognize and ignore misdirection — and measure whether the resource-cost advantage persists.
  • The delay-and-decoy pattern could generalize to other persistent adversaries like prompt-injection or social-engineering bots, though the paper does not make that claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CoopGuard, a multi-agent LLM defense for independent-but-evolving multi-turn jailbreak attacks. Four agents (Deferring, Tempting, Forensic, System) cooperate to delay, misdirect, log, and adapt to attacks, and the authors contribute the EMRA dataset of 5,200 adversarial samples across eight attack types. Experiments on three backbones report a 69% average ASR reduction relative to five baselines, a DR more than six times the strongest baseline, and a 198.83% average increase in attacker-token consumption. The architecture and dataset are potentially interesting, but the evaluation protocol as described does not support the central resource-consumption and deception claims, and the ASR claim is weakened by missing comparisons to the strongest related defenses.

Significance. If the headline empirical claims were valid, CoopGuard would represent a meaningful advance: it is among the first defenses to treat attacker engagement itself as a defensive resource, and the EMRA taxonomy could support more granular multi-turn robustness evaluation. The release of code and data is also a strength. However, the two claims that most distinguish the paper from prior work — deceptive engagement (DR) and attacker-token depletion (AE) — are constructed by the evaluation rubric and the fixed-script protocol rather than measured against an interactive adversary. As presented, the paper does not establish that the framework actually misleads real attackers or causes them to spend more tokens; the main residual value is the multi-agent framework and dataset, but the central quantitative claims need substantially different evidence.

major comments (4)
  1. [Section IV.b and V.C, Fig. 6] The attacker-token consumption claim is not causally identified. EMRA defines each example as a fixed sequence X_{1:T} of attacker queries; no attacker model observes the defense output y_{t-1} and decides to continue, rephrase, or stop. Consequently, AE (average attacker tokens per dialogue) is determined by the dataset script unless episodes are truncated when a baseline refuses, but no such interactive or truncation protocol is described. The reported 198.83% token increase can therefore be an artifact of baselines ending the scripted dialogue early while CoopGuard continues the pre-written turns; it does not show that the defense forced an adaptive attacker to spend more resources.
  2. [Section V.A.c and Eq. (6)] The deception-rate metric is circular. GPT-Judge score 2 is defined as 'successful misdirection,' and Eq. (6) explicitly instructs the Tempting Agent to produce exactly this behavior — elaborate, non-refusal, non-harmful responses. A judge trained or prompted to label any non-refusal, non-harmful output as misdirection will mechanically assign high DR to CoopGuard while baselines, which refuse, receive low DR. The headline 'DR more than six times baselines' is thus partly a conformity-to-prompt measure, not a validated measure of whether an attacker is actually deceived. The same GPT-Judge also produces the ASR numbers, and no numerically reported independent harmfulness validation is provided; Table VI reports judge agreement rates but appears to measure judge self-consistency, not agreement with ground truth, and Fig. 10 lacks quantitative agreement statistics.
  3. [Section II.B and Tables I-II] The paper explicitly cites NBF-LLM [48], X-Boundary [49], RED QUEEN [50], and Honeytrap [23] as multi-turn jailbreak defenses, yet none of these appears in Tables I-II or the AE evaluation. The abstract's 'state-of-the-art baselines' claim is therefore not supported by the strongest available comparisons. Honeytrap, from the same group and also using multi-agent deception, is an especially obvious missing baseline. At minimum, NBF-LLM, RED QUEEN, and Honeytrap need to be evaluated under the same protocol before the 69% ASR reduction claim can be taken at face value.
  4. [Tables/Figures and model naming] The reported backbone models are inconsistent across the paper. Tables I-II use GPT-5, Gemini-2.5-Pro, and DeepSeek-V3; Fig. 7 and Fig. 9 use DeepSeek-V4 and Gemini-3-Flash (and Fig. 9 adds GLM-5); Fig. 8 uses Gemini-3-Flash, GLM-5, and DeepSeek-4; Table III uses GPT-3.5-turbo, GPT-4, and Gemini-2-Flash. This makes it impossible to determine which models were actually evaluated and undermines the reproducibility of the cross-model robustness claims. The authors should state one fixed set of backbones and use it consistently throughout.
minor comments (4)
  1. [Section I] Typo: 'we evaluate the our method on GPT-5...' should be 'we evaluate our method.'
  2. [Section IV.b] The dataset arithmetic is unclear: the text says each of 100 dialogue instances encompasses four fields, resulting in 5,200 total test samples. 100 × 4 = 400, so the construction of the 5,200 number needs a precise explanation (e.g., number of jailbreak prompt variants per episode).
  3. [Table IV] For DeepSeek-V3, the average quality drops from 9.31 to 7.71 (about 17%), which is described as 'marginal degradation.' This wording should be softened or the drop justified.
  4. [Figure 10] The bar chart shows ASR and DR on the same 'Score' axis without error bars or a legend definition of which color corresponds to which metric; numeric values would be more informative, especially because the text claims strong correlation with human judgments.

Circularity Check

2 steps flagged

DR and AE claims are partly constructed by the evaluation design: DR is co-defined with the Tempting Agent's objective, and AE is measured on a fixed attacker script; the ASR claim remains a more independent (though script-bound) result.

specific steps
  1. self definitional [Section III.B (Tempting Agent, Eq. 6) vs. Section V.A.c (DR metric)]
    "The Tempting Agent functions as a decoy, generating responses that are intentionally elaborate and appear helpful on the surface. These responses are engineered to lead attackers down unproductive paths, compelling them to invest significant effort into ineffective strategies under an illusion of progress. ... a score of 2 denotes successful misdirection, where the model strategically leads the attacker astray without generating harmful content."

    DR is defined as the judge label 'successful misdirection', and the Tempting Agent is specified to produce exactly that behavior: deceptive but non-harmful responses that keep the attacker engaged. The defense's own objective is thus re-labeled as the headline outcome. Without an external criterion for whether an output actually misleads a real attacker, the reported DR advantage (more than six times baselines) is largely a restatement of the system's design rather than an independently measured effect.

  2. other [Section IV.b (Multi-Turn Attack Format) and Section V.C (Attack Resource Consumption, AE); Figs. 6 and 9]
    "Each example in EMRA is a sequence X_{1:T} = {x_1, ..., x_T} of attacker queries. ... AE serves as a proxy for defense effectiveness by calculating the average token consumption of the attacker across dialogue turns."

    The EMRA attacker queries are fixed scripts; no adaptive attacker observes the defense's actual response before choosing the next query. Attacker-token consumption is therefore predetermined per episode except for early termination by the defense. The reported AE increase (198.83% on average) cannot be caused by the defense 'keeping attackers engaged'; it reflects which method continues the prewritten turns while baselines refuse and terminate early. Interpreting this script-length effect as defense-induced attacker resource expenditure reverses the causal direction and makes the AE claim an evaluation artifact.

full rationale

The paper's central ASR claim is not circular: it compares harmfulness scores (4/5) under the same fixed EMRA scripts across defenses, and the authors provide human- and DeepSeek-judge cross-validation (Section V.H), so the ASR reduction retains independent empirical content. However, the two supporting headline metrics are partially constructed. DR is the judge label for exactly what the Tempting Agent is engineered to output, so the 'six times higher DR' result is a co-defined metric rather than an external test of real attacker deception. AE is computed from a fixed attacker script and cannot measure a defense-induced increase in attacker resources unless the protocol includes an interactive attacker; the paper describes no such protocol. The same-author Honeytrap reference [23] is listed but is not used as a load-bearing derivation, and the fixed-script limitation is not cured by the judge-consistency checks. These issues warrant a score of 6: one or more predictions reduce by construction, while the ASR claim and benign-utility evaluation remain substantive.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on the representativeness of EMRA, the validity of the GPT-Judge deception rubric, and hand-set but unreported parameters (lambda, tau, role prompts). The paper introduces no new physical entities; the agents are software components, not invented entities in the ledger sense.

free parameters (3)
  • lambda (historical decay factor in Eq. 5)
    Controls how much past turns contribute to the Deferring Agent's detection score; no value is given, and the defense's behavior depends on it.
  • detection threshold tau
    Introduced in Section II-B as the threshold for entering mitigation; never specified, yet it determines which queries are delayed, misdirected, or refused.
  • Agent role prompts and configurations (c_D, c_T, system prompt content)
    The Deferring, Tempting, Forensic, and System agents are implemented entirely by hand-crafted prompts; these prompts are not released in the manuscript and must have been tuned on EMRA to achieve the reported results.
axioms (4)
  • domain assumption EMRA's 100 dialogue instances and 5,200 prompts across eight strategy types are representative of evolving multi-turn jailbreak attacks.
    All evaluation is on the authors' own dataset; no external benchmark is used to validate the threat model.
  • domain assumption GPT-Judge with score 2 labeled 'successful misdirection' correctly identifies real attacker deception.
    This is the load-bearing assumption for the DR and attacker-token metrics; Section V-A.c defines the rubric.
  • domain assumption Role-based prompts on a fixed base LLM can implement delay, misdirection, and forensic analysis without degrading safety or utility.
    The framework never updates base LLM parameters; all behavior change comes from role configurations, and the claim that this is sufficient is assumed throughout Section III.
  • domain assumption The multi-agent coordination equations (Eqs. 3-4) are a valid representation of the deployed agent system.
    These equations are generic multi-agent formalizations and are not instantiated with concrete parameters or architecture details.

pith-pipeline@v1.3.0-alltime-deepseek · 24935 in / 11238 out tokens · 121000 ms · 2026-08-04T01:10:44.021937+00:00 · methodology

0 comments
read the original abstract

As LLMs become increasingly integrated into complex applications, their vulnerability to adversarial attacks has raised significant concerns. However, existing defenses remain reactive in nature. This limitation makes it difficult for them to counter sophisticated threats, as adversaries continuously adjust their strategies across multi-turn interactions. In this paper, we present a proactive defense framework for securing LLMs against evolving multi-turn adversarial attacks that combines disruption, misdirection, and adaptation across successive interaction turns. In particular, it employs a cooperative multi-agent architecture in which specialized agents execute complementary defense strategies. These strategies include controlled response pacing to increase attack costs, strategically ambiguous outputs to mislead adversaries into ineffective strategies, and forensic analysis of interaction logs to identify attack patterns and refine defenses. These agents are coordinated by an adaptive mechanism that dynamically adjusts the defense strategy in response to escalating threats. To facilitate comprehensive evaluation, we present the EMRA dataset designed to simulate evolving strategies across multi-turn attacks, including 5,200 adversarial samples across eight attack types. Experimental results on EMRA across multiple LLM backbones show that the proposed framework reduces ASR by 69% on average relative to evaluated state-of-the-art baselines. Beyond suppressing harmful outputs, it sustains deceptive engagement, achieving an average DR more than six times that of the strongest baselines and increasing attacker-token consumption by 198.83% on average relative to evaluated baselines. Code and dataset are available at https://github.com/SiyuanLi00/CoopGuard.

Figures

Figures reproduced from arXiv: 2608.00134 by Haoyu Li, Jianhua Li, Jun Wu, Mohsen Guizani, Ning Liu, Siyuan Li, Xi Lin, Zehao Liu.

Figure 1
Figure 1. Figure 1: Illustration of the challenge posed by independent yet progressively evolving multi-turn adversarial attacks on LLMs and our innovative multi-agent adaptive defense mechanism to effectively counter these evolving threats. analyzing each turn’s input, storing cross-turn interaction state, and adaptively tuning response tactics as attacks intensify. To address these challenges, we propose a cooperative multi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of multi-agent jailbreak defense framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Resource footprint of multi-turn attacks. (a) Token consumption for [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of misleading response evaluation methods: [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Evaluation of AE, quantified by the average token consumption per dialogue across different models. The defense framework forces the attacker to expend significantly more resources (higher is better for defense) compared to baselines. RPO struggle with semantic variations, our defense frame￾work effectively counters these strategies, maintaining low failure rates. Regarding more complex jailbreak questions… view at source ↗
Figure 7
Figure 7. Figure 7: Performance evolution of defense mechanisms across escalating multi [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Attacker resource consumption across increasing dialogue turns on [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cross-validation results with Deepseek-Judge and Human-Judge. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 40 linked inside Pith

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millicanet al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    A survey of large language models in medicine: Progress, application, and challenge,

    H. Zhou, F. Liu, B. Gu, X. Zou, J. Huang, J. Wu, Y . Li, S. S. Chen, P. Zhou, J. Liuet al., “A survey of large language models in medicine: Progress, application, and challenge,”arXiv preprint arXiv:2311.05112, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Exploring recommendation capabilities of gpt-4v (ision): A preliminary case study,

    P. Zhou, M. Cao, Y .-L. Huang, Q. Ye, P. Zhang, J. Liu, Y . Xie, Y . Hua, and J. Kim, “Exploring recommendation capabilities of gpt-4v (ision): A preliminary case study,”arXiv preprint arXiv:2311.04199, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    {LLM-Fuzzer}: Scaling assessment of large language model jailbreaks,

    J. Yu, X. Lin, Z. Yu, and X. Xing, “{LLM-Fuzzer}: Scaling assessment of large language model jailbreaks,” in33rd USENIX Security Sympo- sium (USENIX Security 24), 2024, pp. 4657–4674

    2024

  7. [7]

    Exploiting the index gradients for optimization-based jailbreaking on large language models,

    J. Li, Y . Hao, H. Xu, X. Wang, and Y . Hong, “Exploiting the index gradients for optimization-based jailbreaking on large language models,” arXiv preprint arXiv:2412.08615, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction,

    T. Liu, Y . Zhang, Z. Zhao, Y . Dong, G. Meng, and K. Chen, “Making them ask and answer: Jailbreaking large language models in few queries via disguise and reconstruction,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 4711–4728

    2024

  9. [9]

    Helping big language models protect themselves: An enhanced filtering and summarization system,

    S. S. Muhaimin and S. Mastorakis, “Helping big language models protect themselves: An enhanced filtering and summarization system,” arXiv preprint arXiv:2505.01315, 2025

    arXiv 2025

  10. [10]

    X-teaming: Multi- turn jailbreaks and defenses with adaptive multi-agents,

    S. Rahman, L. Jiang, J. Shiffer, G. Liu, S. Issaka, M. R. Parvez, H. Palangi, K.-W. Chang, Y . Choi, and S. Gabriel, “X-teaming: Multi- turn jailbreaks and defenses with adaptive multi-agents,”arXiv preprint arXiv:2504.13203, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Catastrophic jailbreak of open-source llms via exploiting generation,

    Y . Huang, S. Gupta, M. Xia, K. Li, and D. Chen, “Catastrophic jailbreak of open-source llms via exploiting generation,”arXiv preprint arXiv:2310.06987, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Datasentinel: A game-theoretic detection of prompt injection attacks,

    Y . Liu, Y . Jia, J. Jia, D. Song, and N. Z. Gong, “Datasentinel: A game-theoretic detection of prompt injection attacks,”arXiv preprint arXiv:2504.11358, 2025

    arXiv 2025

  13. [13]

    Jailbreaker: Automated jailbreak across multiple large language model chatbots,

    G. Deng, Y . Liu, Y . Li, K. Wang, Y . Zhang, Z. Li, H. Wang, T. Zhang, and Y . Liu, “Jailbreaker: Automated jailbreak across multiple large language model chatbots,”arXiv preprint arXiv:2307.08715, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Fight back against jailbreaking via prompt adversarial tuning,

    Y . Mo, Y . Wang, Z. Wei, and Y . Wang, “Fight back against jailbreaking via prompt adversarial tuning,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  15. [15]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,

    F. Bianchi, M. Suzgun, G. Attanasio, P. R ¨ottger, D. Jurafsky, T. Hashimoto, and J. Zou, “Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,”arXiv preprint arXiv:2309.07875, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Distributional preference learning: Understanding and accounting for hidden context in rlhf,

    A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell, “Distributional preference learning: Understanding and accounting for hidden context in rlhf,” inThe Twelfth International Conference on Learning Represen- tations, 2024

    2024

  17. [17]

    Attacking large language models with projected gradient descent,

    S. Geisler, T. Wollschl ¨ager, M. Abdalla, J. Gasteiger, and S. G¨unnemann, “Attacking large language models with projected gradient descent,” arXiv preprint arXiv:2402.09154, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,”arXiv preprint arXiv:2310.04451, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Ad- vprompter: Fast adaptive adversarial prompting for llms,

    A. Paulus, A. Zharmagambetov, C. Guo, B. Amos, and Y . Tian, “Ad- vprompter: Fast adaptive adversarial prompting for llms,”arXiv preprint arXiv:2404.16873, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms,

    Y . Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi, “How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms,”arXiv preprint arXiv:2401.06373, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,

    J. Yu, X. Lin, Z. Yu, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,”arXiv preprint arXiv:2309.10253, 2023. SUBMITTED TO IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 13

    Pith/arXiv arXiv 2023

  23. [23]

    Honeytrap: Deceiving large language model attackers to honeypot traps with resilient multi-agent defense,

    S. Li, X. Lin, J. Wu, Z. Liu, H. Li, T. Ju, X. Chen, and J. Li, “Honeytrap: Deceiving large language model attackers to honeypot traps with resilient multi-agent defense,”arXiv preprint arXiv:2601.04034, 2026

    arXiv 2026

  24. [24]

    Great, now write an article about that: The crescendo multi-turn llm jailbreak attack,

    M. Russinovich, A. Salem, and R. Eldan, “Great, now write an article about that: The crescendo multi-turn llm jailbreak attack,”arXiv preprint arXiv:2404.01833, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Tree of attacks: Jailbreaking black-box llms automatically,

    A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y . Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,”Advances in Neural Information Processing Systems, vol. 37, pp. 61 065–61 105, 2024

    2024

  26. [26]

    Effective and evasive fuzz testing-driven jailbreaking attacks against llms,

    X. Gong, M. Li, Y . Zhang, F. Ran, C. Chen, Y . Chen, Q. Wang, and K.- Y . Lam, “Effective and evasive fuzz testing-driven jailbreaking attacks against llms,”arXiv preprint arXiv:2409.14866, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    ” do anything now

    X. Shen, Z. Chen, M. Backes, Y . Shen, and Y . Zhang, “” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” inProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 1671–1685

    2024

  28. [28]

    Bag of tricks: Benchmarking of jailbreak attacks on llms,

    Z. Xu, F. Liu, and H. Liu, “Bag of tricks: Benchmarking of jailbreak attacks on llms,”arXiv preprint arXiv:2406.09324, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    A survey on trustworthy llm agents: Threats and countermeasures,

    M. Yu, F. Meng, X. Zhou, S. Wang, J. Mao, L. Pang, T. Chen, K. Wang, X. Li, Y . Zhanget al., “A survey on trustworthy llm agents: Threats and countermeasures,”arXiv preprint arXiv:2503.09648, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Masterkey: Automated jailbreaking of large language model chatbots,

    G. Deng, Y . Liu, Y . Li, K. Wang, Y . Zhang, Z. Li, H. Wang, T. Zhang, and Y . Liu, “Masterkey: Automated jailbreaking of large language model chatbots,” inProc. ISOC NDSS, 2024

    2024

  31. [31]

    Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large lan- guage and vision-language models,

    H. Jin, L. Hu, X. Li, P. Zhang, C. Chen, J. Zhuang, and H. Wang, “Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large lan- guage and vision-language models,”arXiv preprint arXiv:2407.01599, 2024

    arXiv 2024

  32. [32]

    Gpt- 4 is too smart to be safe: Stealthy chat with llms via cipher,

    Y . Yuan, W. Jiao, W. Wang, J.-t. Huang, P. He, S. Shi, and Z. Tu, “Gpt- 4 is too smart to be safe: Stealthy chat with llms via cipher,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  33. [33]

    Jailbreaking chatgpt via prompt engineering: An empirical study,

    Y . Liu, G. Deng, Z. Xu, Y . Li, Y . Zheng, Y . Zhang, L. Zhao, T. Zhang, K. Wang, and Y . Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,”arXiv preprint arXiv:2305.13860, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Boosting jailbreak transferability for large language models,

    H. Liu, L. Zhou, and H. Yan, “Boosting jailbreak transferability for large language models,”arXiv preprint arXiv:2410.15645, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,”arXiv preprint arXiv:2310.08419, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Deepinception: Hypnotize large language model to be jailbreaker,

    X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han, “Deepinception: Hypnotize large language model to be jailbreaker,”arXiv preprint arXiv:2311.03191, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Siege: Autonomous multi-turn jailbreaking of large language models with tree search,

    A. Zhou, “Siege: Autonomous multi-turn jailbreaking of large language models with tree search,”arXiv preprint arXiv:2503.10619, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Defending llms against jail- breaking attacks via backtranslation,

    Y . Wang, Z. Shi, A. Bai, and C.-J. Hsieh, “Defending llms against jail- breaking attacks via backtranslation,”arXiv preprint arXiv:2402.16459, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?”Advances in Neural Information Processing Sys- tems, vol. 36, 2024

    2024

  40. [40]

    Red-teaming large language mod- els using chain of utterances for safety-alignment,

    R. Bhardwaj and S. Poria, “Red-teaming large language mod- els using chain of utterances for safety-alignment,”arXiv preprint arXiv:2308.09662, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Attack prompt generation for red teaming and defending large language mod- els,

    B. Deng, W. Wang, F. Feng, Y . Deng, Q. Wang, and X. He, “Attack prompt generation for red teaming and defending large language mod- els,” inThe 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  42. [42]

    Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks,

    Z. Zhang, J. Yang, P. Ke, S. Cui, C. Zheng, H. Wang, and M. Huang, “Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks,”arXiv preprint arXiv:2407.02855, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Baseline defenses for adversarial attacks against aligned language models,

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,”arXiv preprint arXiv:2309.00614, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Certifying llm safety against adversarial prompting,

    A. Kumar, C. Agarwal, S. Srinivas, A. J. Li, S. Feizi, and H. Lakkaraju, “Certifying llm safety against adversarial prompting,”arXiv preprint arXiv:2309.02705, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Smoothllm: Defending large language models against jailbreaking attacks,

    A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,”arXiv preprint arXiv:2310.03684, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Defending against alignment-breaking attacks via robustly aligned llm,

    B. Cao, Y . Cao, L. Lin, and J. Chen, “Defending against alignment-breaking attacks via robustly aligned llm,”arXiv preprint arXiv:2309.14348, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Defending chatgpt against jailbreak attack via self-reminders,

    Y . Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, “Defending chatgpt against jailbreak attack via self-reminders,”Nature Machine Intelligence, vol. 5, no. 12, pp. 1486–1496, 2023

    2023

  48. [48]

    Steering dialogue dynamics for robustness against multi-turn jailbreaking attacks,

    H. Hu, A. Robey, and C. Liu, “Steering dialogue dynamics for robustness against multi-turn jailbreaking attacks,”arXiv preprint arXiv:2503.00187, 2025

    arXiv 2025

  49. [49]

    X-boundary: Establishing exact safety boundary to shield llms from multi-turn jailbreaks without compromising usability,

    X. Lu, D. Liu, Y . Yu, L. Xu, and J. Shao, “X-boundary: Establishing exact safety boundary to shield llms from multi-turn jailbreaks without compromising usability,”arXiv preprint arXiv:2502.09990, 2025

    arXiv 2025

  50. [50]

    Red queen: Safeguarding large language models against concealed multi-turn jailbreaking,

    Y . Jiang, K. Aggarwal, T. Laud, K. Munir, J. Pujara, and S. Mukherjee, “Red queen: Safeguarding large language models against concealed multi-turn jailbreaking,”arXiv preprint arXiv:2409.17458, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” inProceedings of the 36th annual acm symposium on user interface software and technology, 2023, pp. 1–22

    2023

  52. [52]

    Training socially aligned language models in simulated human society,

    R. Liu, R. Yang, C. Jia, G. Zhang, D. Zhou, A. M. Dai, D. Yang, and S. V osoughi, “Training socially aligned language models in simulated human society,”arXiv preprint arXiv:2305.16960, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Camel: Communicative agents for

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem, “Camel: Communicative agents for” mind” exploration of large language model society,”Advances in Neural Information Processing Systems, vol. 36, pp. 51 991–52 008, 2023

    2023

  54. [54]

    Autogen: Enabling next-gen llm applications via multi-agent conversation framework,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen llm applications via multi-agent conversation framework,”arXiv preprint arXiv:2308.08155, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Linet al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  56. [56]

    Communicative agents for software development,

    C. Qian, X. Cong, W. Liu, C. Yang, W. Chen, Y . Su, Y . Dang, J. Li, J. Xu, D. Liet al., “Communicative agents for software development,” arXiv preprint arXiv:2307.07924, 2023

    Pith/arXiv arXiv 2023

  57. [57]

    Improving factuality and reasoning in language models through multiagent debate,

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” arXiv preprint arXiv:2305.14325, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    Encouraging divergent thinking in large language models through multi-agent debate,

    T. Liang, Z. He, W. Jiao, X. Wang, Y . Wang, R. Wang, Y . Yang, S. Shi, and Z. Tu, “Encouraging divergent thinking in large language models through multi-agent debate,”arXiv preprint arXiv:2305.19118, 2023

    Pith/arXiv arXiv 2023

  59. [59]

    Defending large language models against jailbreaking attacks through goal priori- tization,

    Z. Zhang, J. Yang, P. Ke, F. Mi, H. Wang, and M. Huang, “Defending large language models against jailbreaking attacks through goal priori- tization,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024

    2024

  60. [60]

    Robust prompt optimization for defend- ing language models against jailbreaking attacks,

    A. Zhou, B. Li, and H. Wang, “Robust prompt optimization for defend- ing language models against jailbreaking attacks,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  61. [61]

    Securitylin- gua: Efficient defense of llm jailbreak attacks via security-aware prompt compression,

    Y . Li, S. Ahn, H. Jiang, A. H. Abdi, Y . Yang, and L. Qiu, “Securitylin- gua: Efficient defense of llm jailbreak attacks via security-aware prompt compression,”arXiv preprint arXiv:2506.12707, 2025

    Pith/arXiv arXiv 2025

  62. [62]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  63. [63]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” in 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025, pp. 23–42

    2025

  64. [64]

    M2s: Multi-turn to single-turn jailbreak in red teaming for llms,

    J. Ha, H. Kim, S. Yu, H. Park, A. Yousefpour, Y . Park, and S. Kim, “M2s: Multi-turn to single-turn jailbreak in red teaming for llms,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, vol. 1, 2025, pp. 16 489–16 507

    2025

  65. [65]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues,

    G. Bai, J. Liu, X. Bu, Y . He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zhenget al., “Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues,”arXiv preprint arXiv:2402.14762, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Cosafe: Evaluating large language model safety in multi-turn dialogue coreference,

    E. Yu, J. Li, M. Liao, S. Wang, G. Zuchen, F. Mi, and L. Hong, “Cosafe: Evaluating large language model safety in multi-turn dialogue coreference,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 17 494–17 508

    2024