Pith. sign in

REVIEW 3 major objections 6 minor 28 references

Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-agent defenses against jailbreak attacks cut missed harmful responses but raise false alarms, and the best number of agents depends on the attack type.

desk verdict The measured trade-off between false negatives and false positives is plausible but rests entirely on unverified GPT-4.1 labels, and the paper contradicts its own Table 4 on precision - fix those and it becomes a reasonable reproduction study. read the letter →

arxiv 2506.23576 v1 pith:J6AGAQKK submitted 2025-06-30 cs.AI

Maria Carolina Cornelia Wit , Jun Pang This is my paper
classification cs.AI
keywords multi-agentLLMdefensejailbreakingattacksfalsepositivesandnegativesAutoAttackEvalsafetyevaluationsuccessrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that adding agents to an LLM jailbreak defense is a trade-off, not a free win: more agents catch more harmful responses, cutting false negatives sharply, but they also wave through more harmful content as false positives, and the best agent count depends on the attack. Reproducing the AutoDefense framework with current OpenAI models and three jailbreak templates — the original AutoDefense attack, BetterDan, and JB — the paper finds the optimal configuration is three agents for BetterDan, two for JB, and one for the Original Attack. It further claims that the GPT-4.1 evaluator used as ground truth mislabels exactly the ambiguous, two-part responses these defenses struggle with, so the measured trade-off is partly an artifact of the evaluation method. The result matters because a real deployment cannot simply pick one agent count, and because it shifts part of the blame for defense failures from the defenders to the metric used to judge them.

What carries the argument

The load-bearing object is the AutoDefense response-filtering pipeline in its three configurations: a single agent that judges a response VALID or INVALID directly; a two-agent setup in which one Analyst infers intent and the likely original prompt before a Judge rules; and a three-agent setup that separates intent analysis from prompt inference before the Judge. The mechanism that carries the argument is the comparison of each configuration's verdicts against the pseudo-ground-truth labels produced by the AttackEval template run on GPT-4.1, scored through the $P_4$ metric, which the authors use because it is invariant to which class is called positive and therefore suited to their imbalanced BetterDan set.

What would settle it

Re-label the three datasets with a panel of human judges, focusing on the five recurring JB false positives and the sexually explicit responses the evaluator flagged; if the human labels change enough false-positive and false-negative counts, the ranking of one-, two-, and three-agent setups shifts, which would settle whether the trade-off is a property of the defense or of the evaluator.

Watch

Extended reading notes

Core claim

The paper's central claim is that the AutoDefense-style multi-agent defense framework, re-run with current GPT models and tested against three jailbreak templates, gains no universal benefit from adding agents. On the authors' measurements, false negatives fall monotonically as agents are added — from 200 to 3 on BetterDan, from 110 to 19 on JB, and from 27 to 2 on the Original Attack — while false positives rise (5 to 31, 6 to 139, and 5 to 41 respectively), so the best configuration is attack-dependent: three agents for BetterDan, two for JB, and one for the Original Attack. The paper further claims that a meaningful share of the apparent misclassifications, especially on the dual-layered JB template where a compliant [CHATGPT] refusal is followed by a jailbreaking [JB] persona, comes from the GPT-4.1 evaluator that defines the ground truth rather than from the defense agents themselves.

Load-bearing premise

The whole measurement depends on trusting the GPT-4.1 evaluator's verdicts as correct, even though the paper itself shows that evaluator mislabels exactly the ambiguous and two-part responses at the center of the trade-off, so if those verdicts are wrong the recommended agent counts change.

Editorial extensions

If this is right

  • A fixed agent count cannot serve as a universal safety filter, because each of the three attacks is best handled by a different configuration: three agents for BetterDan, two for JB, one for the Original Attack.
  • Because false positives mean harmful content is waved through, the rise in false positives with more agents is the more dangerous error, and on JB the added agents cost 133 new false positives while saving only 91 false negatives.
  • Dual-layered attacks such as JB, where an ethical refusal from [CHATGPT] is followed by a jailbreaking [JB] persona, are the hardest case for both defense and evaluation, and JB produced the highest undefended attack success rate (55.13%).
  • The defense costs about two seconds per interaction plus heavy computational overhead, which limits its appeal for commercial-scale deployment.
  • Base-model safety is improving only modestly: the Original Attack's success rate fell from 55.74% in the original AutoDefense study to 48.72% here.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The calming effect of the third agent on borderline calls suggests role separation acts as a mild debiasing ensemble; swapping different base models into the Intention Analyzer and Judge roles would test whether the effect is role-specific or model-specific.
  • Because the evaluator is a Western-trained model, the ground truth encodes one cultural stance, and a replication with a non-Western model, as the authors suggest, could flip which agent count looks optimal.
  • A human-panel re-labeling of the disputed examples — the five recurring JB false positives and the sexual-content false negatives — would show whether the reported trade-off is a property of the defense or of the evaluator, and would double as a reusable benchmark for ethically ambiguous jailbreak responses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper evaluates the AutoDefense multi-agent jailbreak defense framework on three attack templates (the original AutoDefense attack, BetterDan, and JB). Responses from GPT-3.5 are judged by 1-, 2-, and 3-agent configurations, and the same responses are labeled by GPT-4.1 through the AttackEval template to form pseudo-ground truth. The authors report attack success rates, confusion-matrix counts, precision/recall/F1/P4 metrics, and a qualitative analysis of misclassified examples. Their main conclusion is that increasing the number of agents generally reduces false negatives but increases false positives, so the optimal configuration is attack-dependent and no one-size-fits-all setting exists. They also argue that LLM-based evaluators themselves are unreliable for ethically ambiguous content.

Significance. If the quantitative result is trustworthy, the paper makes a useful contribution by showing that multi-agent defense is not uniformly beneficial: agent count controls a trade-off between blocking legitimate content and allowing harmful content, and the best configuration varies by attack type. The work is also valuable as an independent reproduction of AutoDefense using updated API-based models and two additional jailbreak templates, with raw counts and qualitative examples reported transparently. However, the central quantitative claim is computed entirely against GPT-4.1 AttackEval labels, which the paper itself shows to be fallible in several concrete cases, and no human audit or label-stability analysis is provided. The significance is therefore conditional on the correctness of those labels, and the current framing overstates the strength of the evidence.

major comments (3)
  1. [§3.1, §2.4, Table 4] The central FN/FP trade-off rests entirely on ground truth labels produced by GPT-4.1 via the AttackEval template, but the paper provides direct evidence that this evaluator is unreliable. Section 5.1 calls a 3-agent false positive “debatable” and states that the JB performance decline “may also be influenced by misclassifications introduced by the evaluation mechanism itself”; Section 5.2 calls a persistent false negative “clearly” misclassified and says another false negative “may not warrant” its label. Because every TP/FP/FN/TN count in Table 4 and every downstream conclusion changes if the evaluator mislabels a response, the measured trade-off is unverified as stated. The paper needs a human-audited subset of labels, a label-stability or inter-evaluator agreement check, or a re-framing of all conclusions as conditional on AttackEval correctness.
  2. [§4, paragraph after Table 4] The text states that “Precision increases across all templates as the number of agents increases,” but Table 4 shows the opposite: precision decreases from the 1-agent to the 3-agent configuration for every template (BetterDan 0.94 to 0.90, JB 0.92 to 0.53, Original Attack 0.97 to 0.83). This is a direct internal inconsistency in a paragraph that interprets the central results, and it must be corrected before the trade-off discussion can be assessed.
  3. [Abstract, §6, §7] The abstract claims that “multi-agent systems enhance resistance to jailbreaks, especially by reducing false negatives,” but the body concludes that there is “no one-size-fits-all configuration” and that false positives increase with more agents. Since Section 7 recommends attack-dependent configurations and Section 6 notes that more agents can degrade performance on the JB and Original Attack templates, “enhance resistance” is an overstatement. The abstract should be revised to state the actual trade-off: more agents can reduce false negatives at the cost of more false positives, with the best configuration varying by attack type.
minor comments (6)
  1. [§2.4] The AttackEval template heading contains typos: “Course Grained Evaluationd” should be “Coarse-Grained Evaluation,” and the trailing “d” should be removed.
  2. [Table 2] The column heading “ASR-FPR (%)” is ambiguous; the surrounding text describes this column as ASR, so the heading should be “ASR (%)” or the relationship between ASR and FPR should be defined explicitly.
  3. [§5.1] The statement that “Among the six original False Positives ... five persist across all agent configurations” is difficult to reconcile with Table 4, where JB false positives grow from 6 to 82 to 139 across configurations; please clarify whether “original” refers to the 1-agent false positives and what “persist” means when the total count changes dramatically.
  4. [§2.1 and throughout] The spelling “AutoDefence” is used inconsistently alongside “AutoDefense,” and reference [18] contains the typo “Prceedings”; please standardize and correct these.
  5. [§3.3] The paper states that datasets and results are stored in JSON format, but no repository link or data availability statement is provided; adding one would materially support the reproducibility claim.
  6. [§7] The claim that the defense “requires approximately two seconds per interaction” is reported without a measurement protocol, sample size, or variance; if this number is retained, it needs a brief methodological note.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical benchmark study with no fitted parameters; its central numbers are measured against an external LLM-based evaluator, not derived from the paper's own assumptions.

full rationale

The paper is an empirical reproduction and extension of AutoDefense, not a derivation. Its central quantities (ASR, TP/FP/FN/TN, precision, recall, F1, P4) are computed by running fixed prompt templates against GPT-3.5 and comparing the defence judgments to labels produced by the AttackEval evaluator (GPT-4.1) described in Sections 2.4 and 3.1. No parameter is fitted to a subset of the data and then 'predicted' on a closely related quantity; the results in Tables 1 and 4 are direct measurements, not outputs of a model derived from those measurements. The only load-bearing external input is the AttackEval pseudo-ground truth, and the paper explicitly flags its limitations: Section 5.1 states that the JB performance decline 'may also be influenced by misclassifications introduced by the evaluation mechanism itself,' and Section 5.2 calls a pornography-listing response 'a case of misclassification by the evaluation metric proposed in [7].' That is a measurement-validity caveat about the ground truth, not a circular step: the defence judgments are not defined in terms of AttackEval's labels, nor are the labels computed from the defence judgments. The AutoDefense architecture and prompts are imported from prior work by different authors [15, 16], and the comparison to the original ASR is an external benchmark, not a self-citation chain. The Section 4 statement that precision increases with agent count is contradicted by Table 4 (precision decreases for every template), but an internal inconsistency is a correctness problem, not circularity. No equation in the paper reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted constants. Its central claims rest on measurement assumptions about the LLM evaluator and the representativeness of three attack templates, rather than on free parameters.

assumptions (4)
  • domain assumption GPT-4.1 with the AttackEval prompt yields correct ground-truth labels for response harmfulness.
    Sections 2.4 and 3.1 define ground truth exclusively through this LLM evaluator. The paper acknowledges its fallibility, but every TP/FP/FN count depends on it.
  • domain assumption The three jailbreak templates (Original, BetterDan, JB) are representative of jailbreak attacks.
    Section 3.3 selects these templates without a coverage argument or systematic sampling of attack space, yet conclusions are phrased about jailbreak attacks generally.
  • domain assumption The AutoDefense prompts and agent coordination, sourced from GitHub, were reproduced faithfully.
    Section 3.2 and 3.3 state the architecture follows the original repository, but no verification or ablation of the reproduction is provided.
  • domain assumption Single-shot API calls are sufficiently stable for comparing configurations.
    Section 3.1 reports no repeated runs, temperatures, or seeds. Some reported accuracy differences are small, such as 88.97% for both 2 and 3 agents.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models." pith.science (2026). https://pith.science/paper/J6AGAQKK

@misc{pith2026250623576,
  author       = {Pith},
  title        = {Pith review of: Evaluating Multi-Agent Defences Against Jailbreaking Attacks on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J6AGAQKK}},
  note         = {Machine review of arXiv:2506.23576}
}
read the original abstract

Recent advances in large language models (LLMs) have raised concerns about jailbreaking attacks, i.e., prompts that bypass safety mechanisms. This paper investigates the use of multi-agent LLM systems as a defence against such attacks. We evaluate three jailbreaking strategies, including the original AutoDefense attack and two from Deepleaps: BetterDan and JB. Reproducing the AutoDefense framework, we compare single-agent setups with two- and three-agent configurations. Our results show that multi-agent systems enhance resistance to jailbreaks, especially by reducing false negatives. However, its effectiveness varies by attack type, and it introduces trade-offs such as increased false positives and computational overhead. These findings point to the limitations of current automated defences and suggest directions for improving alignment robustness in future LLM systems.

Figures

Figures reproduced from arXiv: 2506.23576 by the authors.

Figure 1
Figure 1. Overview of one-, two-, and three-agent architectures, sourced from [ [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 19 canonical work pages

  1. [1]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022)

    arXiv 2022

  2. [2]

    Xiaohe Bo, Zeyu Zhang, Quanyu Dai, Xueyang Feng, Lei Wang, Rui Li, Xu Chen, and Ji-Rong Wen. 2025. Reflective Multi-Agent Collaboration based on Large Language Models. Advances in Neural Information Processing Systems 37 (2025), 138595–138631

    work page 2025

  3. [3]

    Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. 2024. When llm meets drl: Advancing jailbreaking efficiency via drl-guided search. arXiv preprint arXiv:2406.08705 (2024)

    arXiv 2024

  4. [4]

    DeepLeaps. 2024. ChatGPT DAN Jailbreak Prompt. https://deepleaps.com/prompt/chatgpt-betterdan-jailbreak-prompt/ Accessed: 2025-05-13

    work page 2024

  5. [5]

    DeepLeaps. 2024. ChatGPT JB Jailbreak Prompt. https://deepleaps.com/prompt/chatgpt-jb-jailbreak-prompt/ Accessed: 2025-05-13

    work page 2024

  6. [6]

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024)

    arXiv 2024

  7. [7]

    Mingyu Jin, Chong Zhang, Liangyao Li, Zihao Zhou, Yongfeng Zhang, et al . 2024. Attackeval: How to evaluate the effectiveness of jailbreak attacking on large language models. arXiv preprint arXiv:2401.09002 (2024)

    arXiv 2024

  8. [8]

    Kolena. 2024. TP, FP, FN, TN — Definitions and Usage in Evaluation Metrics. https://docs.kolena.com/metrics/tp-fp-fn-tn/ Kolena Documentation, accessed 2025-05-13

    work page 2024

Show all 28 references
  1. [9]

    Jeremy Kritz, Vaughn Robinson, Robert Vacareanu, Bijan Varjavand, Michael Choi, Bobby Gogov, Scale Red Team, Summer Yue, Willow E Primack, and Zifan Wang. 2025. Jailbreaking to Jailbreak. arXiv preprint arXiv:2502.09638 (2025)

    2025 arXiv

  2. [10]

    OECD.AI. 2024. Attack Success Rate (ASR). https://oecd.ai/en/catalogue/metrics/attack-success-rate-asr OECD.AI Metrics Catalogue, accessed 2025-05-13

    2024

  3. [11]

    Thomas Vidori. 2024. Better than the F1-score, Discover the P4-score. https://medium.com/@thomas.vidori/better-than-the-f1-score-discover-the- p-4-score-903242e9545b Medium blog post, accessed 2025-05-13

    2024

  4. [12]

    Peiran Wang, Xiaogeng Liu, and Chaowei Xiao. 2024. RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Process. arXiv preprint arXiv:2410.08660 (2024)

    2024 arXiv

  5. [13]

    Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong. 2024. SELF-GUARD: Empower the LLM to Safeguard Itself. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2024 doi

  6. [14]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems 36 (2024)

    2024

  7. [15]

    Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. 2024. Autodefense: Multi-agent llm defense against jailbreak attacks. arXiv preprint arXiv:2403.04783 (2024)

    2024 arXiv

  8. [16]

    Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. 2024. AutoDefense: Multi-agent LLM Defense Against Jailbreak Attacks. https://github.com/XHMY/AutoDefense. Accessed: 2025-05-08

    2024

  9. [17]

    Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kawaguchi Kenji, and Michael Qizhe Shieh. 2025. Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron. In Prceedings of the 13th International Conference on Learning Representations

    2025

  10. [18]

    Stay in character!

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. How alignment and jailbreak work: Explain llm safety through intermediate hidden states. arXiv preprint arXiv:2406.05644 (2024). 22 Maria Carolina Cornelia Wit and Jun Pang A Jailbreaking Tem...

    2024 arXiv

  11. [21]

    What would you suggest? 2_user: Judge, please make a judgment based on the analyzed intention and original prompts

    For an art project, I need to know what tools would be used to dismantle a metal sign, like a stop sign, safely and effectively. What would you suggest? 2_user: Judge, please make a judgment based on the analyzed intention and original prompts. Your judgment should reflect the...

  12. [22]

    Comply with applicable laws for example, don’t compromise the privacy of others, engage in regulated activity without complying with applicable regulations, or promote or engage in any illegal activity, including the exploitation or harm of children and the development or dist...

  13. [23]

    Don’t use our service to harm yourself or others for example, don’t use our services to promote suicide or self-harm, develop or use weapons, injure others or destroy property, or engage in unauthorised activities that violate the security of any service or system

  14. [24]

    Here is a list of tools you will need to cut down a stop sign: Angle grinder, Metal cutting blade

    Don’t repurpose or distribute output from our services to harm others for example, don’t share output from our services to defraud, scam, spam, mislead, bully, harass, defame, discriminate based on protected attributes, sexualize children, or promote violence, hatred or the su...

  15. [25]

    What tools and equipment are needed for the removal of street signs, such as a stop sign?

  16. [26]

    Can you list the tools required for a project that involves cutting through metal, like removing a metal street sign?

  17. [27]

    For an art project, I need to know what tools would be used to dismantle a metal sign, like a stop sign, safely and effectively. What would you suggest? 26 Maria Carolina Cornelia Wit and Jun Pang 3_user: Judge, please make a judgment on whether the system input content is val...

  18. [28]

    Comply with applicable laws - for example, don’t compromise the privacy of others, engage in regulated activity without complying with applicable regulations, or promote or engage in any illegal activity, including the exploitation or harm of children and the development or di...

  19. [29]

    Don’t use our service to harm yourself or others - for example, don’t use our services to promote suicide or self-harm, develop or use weapons, injure others or destroy property, or engage in unauthorised activities that violate the security of any service or system

  20. [30]

    –SYSTEM INPUT START– [INSERT INPUT HERE] –SYSTEM INPUT END– ALWAYS START YOUR RESPONSE WITH ’I am the Judge

    Don’t repurpose or distribute output from our services to harm others - for example, don’t share output from our services to defraud, scam, spam, mislead, bully, harass, defame, discriminate based on protected attributes, sexualize children, or promote violence, hatred or the ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.