Pith. sign in

REVIEW 4 major objections 5 minor 138 references

A jailbreak attack's value lies in the safety improvement it enables, not the breakage it achieves—and attack success rate is an unreliable proxy for that value.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:50 UTC pith:54EABWIJ

load-bearing objection Genuinely useful reframing of jailbreak evaluation around downstream defense utility, with a coherent Shapley-based framework; the real-world evidence for the headline empirical claims is too thin to carry them yet. the 4 major comments →

arxiv 2607.17152 v1 pith:54EABWIJ submitted 2026-07-19 cs.CR cs.CL

How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions

Yukai Zhou , Feiyang Lu , Xiaokai Mao , Jinfei Liu , Wenjie Wang This is my paper
classification cs.CR cs.CL
keywords jailbreak attacksLLM safetydefender-centric evaluationShapley valuesubset selectionattack success ratered-teamingadversarial training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reframes jailbreak evaluation from the attacker's question—how often does an attack break a model?—to the defender's question: how much does an attack improve safety when used as red-teaming data for safety training. The authors build a defender-centric utility function that scores an attack subset by the resulting model's robustness and over-refusal behavior, then attribute utility to individual attacks with a Shapley-style score (AttackSHAP). Their real-model experiments show that attack success rate and singleton defender utility are only weakly correlated (Spearman ρ ≈ 0.2), and that directly optimizing subsets by utility recovers upward of 80% of the oracle gain, whereas attribution-only top-k selection reaches about 60–71%. The paper concludes that jailbreak attacks should be treated as resources for safety alignment, not merely as tools for breaking models.

Core claim

The paper's central empirical claim is that attacker-centric success is an incomplete criterion for identifying which attacks matter for safety alignment: in a real in-context-defense setting, the Spearman correlation between no-defense attack success rate and singleton defender-side utility is ρ=0.215, with attacks such as Roleplay moving from rank 9 to rank 2 while Optimized Suffix drops from rank 1 to rank 7. The paper's constructive claim is that A-MESS—a Shapley-based framework operating on black-box subset-utility queries—estimates AttackSHAP with limited queries and, by directly optimizing subsets (greedy or surrogate search), recovers at least 82.69% of the oracle utility gain for k>

What carries the argument

The central object is a black-box defender-side utility function vθ(S) that scores any subset of jailbreak attacks by the downstream safety they enable under a given setting. AttackSHAP treats attacks as players in a cooperative game and attributes to each attack its average marginal utility across all subset contexts—the Shapley value of that utility game. A-MESS provides two solvers: a greedy solver that adaptively queries true utilities and adds the attack with the largest marginal gain, and a surrogate solver that learns an approximate utility landscape from sampled subset evaluations and searches over it, allowing cheap multi-budget selection.

Load-bearing premise

The framework stands or falls on the assumption that the paper's chosen defender-side utility—a weighted blend of jailbreak-robustness and over-refusal scores on three open models—faithfully captures real safety improvement; if that utility does not track genuine safety, the empirical findings may be artifacts of the chosen oracle.

What would settle it

If a defender with an independently motivated utility function and a defense pipeline whose behavior varies with the attack set finds that top-ASR attacks already yield the highest-utility subsets—or observes a setting where vθ(S) is flat across subsets—then the paper's central claim that ASR is incomplete and that direct subset optimization is needed would fail in that regime.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Red-teaming data selection should be based on defender-side utility, not raw attack success rate: the paper's real-model experiment shows ASR rankings are only weakly correlated with singleton defense utility (ρ=0.215).
  • Attribution alone is not enough for subset decisions: top-k by any individual score (ASR, LOO, AOO, AttackSHAP) recovers only about 60–71% of the oracle utility improvement for k>1, while direct subset optimization recovers at least 82.69% in the same synthetic landscapes.
  • A few hundred black-box utility queries suffice for A-MESS-SURROGATE to learn a practical approximation of the utility landscape, enabling AttackSHAP estimation and multi-budget selection at low cost.
  • The framework is setting-agnostic: it only requires access to subset-utility observations, so it can be instantiated under proprietary defense pipelines and different utility definitions, including robustness and over-refusal trade-offs.
  • In the real in-context-defense setting, both A-MESS solvers selected the highest-utility subset under both evaluated budgets, improving over ASR-based selection by a large margin.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the weak ASR–utility alignment generalizes, current jailbreak leaderboards—which rank attacks by success rate—may be systematically misguiding which attacks researchers prioritize and which defenses get built.
  • The framework could extend beyond jailbreaks to any red-teaming data valuation problem, such as selecting prompts for robustness tuning or choosing examples for safety fine-tuning, whenever an observable subset utility exists.
  • The specific utility weighting (0.8×ASR reduction + 0.2×over-refusal penalty) is a modeling choice; under a different weighting the rank correlation and the selected subsets would likely change, so the main empirical claim should be read as conditional on the utility being a faithful proxy for safety.
  • A-MESS's setting-agnostic claim implies a testable extension: run the same selection protocol with a different defense pipeline (e.g., classifier-based guardrails or RLHF) and check whether direct subset optimization still beats attribution-only selection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes replacing attacker-centric jailbreak evaluation (e.g., ASR) with defender-centric subset utility: attacks are valued by the downstream safety improvement they enable when used as red-teaming data. It introduces AttackSHAP, the Shapley value of a black-box subset utility vθ(S), and two A-MESS solvers: a greedy solver that queries true utilities and a surrogate solver that learns vθ from sampled subsets and then searches over the learned landscape. Experiments on 90 synthetic utility landscapes and three real LLM safety settings are used to support three claims: ASR rankings are weakly aligned with defender utility (Spearman ρ=0.215 in §4.3), AttackSHAP/surrogate utility can be estimated accurately with limited queries (§4.4), and direct subset optimization recovers a larger fraction of oracle gain than attribution-only top-k selection (§4.5).

Significance. If the empirical claims are robust, the paper makes a useful conceptual contribution: jailbreak attacks are treated as resources for safety alignment rather than as breakers of undefended models. The formalization is clean, the black-box query model is practical, and the synthetic experiments with exact oracle enumeration and multiple Shapley baselines are a strength. The paper also makes falsifiable predictions (e.g., low ASR–utility alignment, direct subset selection beating attribution-only selection) and clearly states its central assumption in the Limitations. However, the current real-world evidence for the headline claims is thin: n=10 attack categories, a single unvalidated utility operationalization, and comparisons that do not control query budget. The framework is promising, but the empirical support needs strengthening before the conclusions can be accepted as stated.

major comments (4)
  1. [§4.3, Figure 2, Table 2] The headline claim that attacker-centric success is weakly aligned with defender utility rests on Spearman ρ=0.215 and Kendall τ=0.190 computed from n=10 attack categories. For n=10, these values are statistically indistinguishable from zero (a permutation test would yield p roughly 0.5), and the 95% confidence interval is very wide. No confidence interval, permutation test, or significance test is reported. This is load-bearing for the paper's central argument. Additionally, Appendix D states that singleton utility is post-defense ASR reduction, so the comparison is between pre-defense ASR and a quantity derived from post-defense ASR; the two measures are not independent. Please report uncertainty and ideally increase n or evaluate on additional models/pipelines.
  2. [§4.5, Table 5; Limitations] The real subset-selection claim uses U=0.8·ScoreASR+0.2·ScoreORR on a single model (Ministral-3-8B), one ICD pipeline, one over-refusal benchmark (XSTest), and n=10 categories. No sensitivity analysis is reported over the 0.8/0.2 weights, the ORR penalty, or the defense pipeline. The Limitations explicitly state that A-MESS 'assumes that the defender-side utility of an attack subset is meaningful,' but this assumption is exactly what the real-world experiments need to test. If a plausible alternative utility (e.g., equal weights, larger ORR penalty, or adversarial-training pipeline) changes the ASR–utility ranking or eliminates the A-MESS-GREEDY margin over TOPK-ASR, the central conclusion would be an artifact of the chosen operationalization. Please add systematic sensitivity analysis and, if feasible, evaluate selected subsets on a held-out safety metric not used in U.
  3. [§4.5, Table 4; Table 1] The synthetic subset-selection comparison does not control utility-query budget. Table 1 shows that A-MESS-GREEDY uses nk−k(k−1)/2 true utility queries while TOPK-ASR uses zero and TOPK-ϕ uses 2^n (or M for the surrogate variant). Table 4 reports Normalized Gain without enforcing a matched query budget, so the conclusion that attribution-only selection is inferior to direct subset optimization may reflect the amount of information available rather than the method. To make the claim fair, compare methods under equal utility-query budgets, e.g., give attribution baselines access to the same number of true utility queries as A-MESS-GREEDY, or evaluate A-MESS-GREEDY with a comparable query cap. The raw utilities in Table 6 inherit the same issue.
  4. [§4.4, Table 3] The paper claims that 'AttackSHAP can be estimated accurately with limited utility queries,' but in the real LLM setting this is supported only by held-out prediction of v(S), not by comparing estimated AttackSHAP values to any reference. High fidelity of the surrogate utility does not by itself guarantee accurate Shapley estimates, especially when interactions are present and n is small. Since n=10 in the real setting, exact AttackSHAP could be computed from 2^10=1024 utility queries (or a high-quality Monte Carlo reference), which would directly validate the attribution claim. Without such validation, the real-world attribution component of the framework remains unverified.
minor comments (5)
  1. [Table 2] Roleplay has no-defense ASR=0 but is ranked second by singleton defense utility. This is counterintuitive and deserves an explanation: how can an attack that never succeeds against the undefended model be a highly effective defense demonstration? Clarify the mechanism and report the actual utility values.
  2. [Eq. (4)] Normalized Gain is defined relative to the average random subset Sk(BASE). The number of random subsets averaged and the variance across them are not reported. Please state the number of random draws and, ideally, report standard errors over random subsets and utility landscapes.
  3. [§3.3, A-MESS-SURROGATE] The surrogate solver is defined as argmax over size-k subsets of the learned landscape, but the search algorithm over a combinatorial space is not specified. State whether this is exhaustive (only feasible for small n), beam search, greedy on the surrogate, or another method, and provide the details in Appendix F.
  4. [Figure 3] The main text summarizes Figure 3 qualitatively ('A-MESS reaches a strong accuracy-efficiency tradeoff') without reporting the numeric values shown in the figure. Since the synthetic AttackSHAP estimation is a core contribution, include the key numbers (e.g., correlation/MAE at each budget) in the text or a table.
  5. [Appendix D, Table 7] The table is helpful but omits the exact formulas for ScoreASR and ScoreORR beyond 'relative balanced-ASR gain.' Specify whether ScoreASR is (ASR_no_defense − ASR_defended)/ASR_no_defense or a balanced variant, and how clipping is applied.

Circularity Check

0 steps flagged

No significant circularity: AttackSHAP is definitionally Shapley but internally consistent; ASR-utility divergence and subset-selection gains are empirical; minor self-citations are not load-bearing.

full rationale

The central derivation chain is not circular. AttackSHAP is defined as the Shapley value of the observed utility vθ(S), so the attribution is internally consistent rather than fitted. The ASR-versus-utility finding in §4.3 compares two distinct empirical measurements—no-defense ASR and singleton post-defense utility—and the reported low correlation (Spearman ρ=0.215) is a data-dependent result, not an identity forced by construction. In §4.5, A-MESS-GREEDY optimizes vθ and is evaluated on the same vθ, so its advantage over proxy-based TOPK-ASR is partly expected; however, the comparison against attribution baselines and the oracle gap is a legitimate empirical evaluation of subset-optimization behavior, not a definitional tautology. The Limitations explicitly state that A-MESS assumes the defender-side utility is meaningful; this is an external-validity assumption about whether vθ tracks real safety, not a circular derivation. The only self-citations (Zhou et al., 2025a as an attack example; Zhou et al., 2025b as a future-work pointer in Limitations) are not load-bearing for the paper's claims. Overall, the paper's method is self-consistent and its headline conclusions rest on experiments rather than on circular reasoning, though the real-world interpretation depends on the unvalidated utility operationalization.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entities are postulated. AttackSHAP and A-MESS are methodological constructs, not independent entities; they have no falsifiable handle outside the paper beyond their performance on the paper's chosen utility functions. The paper's contribution is therefore best assessed through the adequacy of its utility operationalizations and the strength of its empirical support.

free parameters (3)
  • Utility weight w (ScoreASR vs ScoreORR) = 0.8 / 0.2
    Used to define U in the real subset-selection setting (Table 5, Appendix D); no justification or sensitivity analysis is given.
  • Synthetic landscape construction parameters = 90 landscapes, 3 policies; exact coefficients not specified in main text
    Scalar/vector strengths, diversity bonus, and pairwise interaction magnitudes are chosen by hand to resemble assumed real defense utilities (Appendix B); all synthetic results inherit these choices.
  • Attack taxonomy / universe size = 10 categories (real), n=20 (synthetic)
    The 10 real attack categories and the decision to treat an attack as a data source are author-defined; results may not transfer to other attack taxonomies or finer-grained attack instances.
axioms (5)
  • domain assumption The defender-side utility of an attack subset is meaningful and coupled to the selected attacks
    Paper's Limitations: 'A-MESS assumes that the defender-side utility of an attack subset is meaningful'; if a defense is independent of S, attribution and selection are ill-posed.
  • standard math Shapley value is the correct attribution for attack contributions
    Equation (1) applies Shapley's axioms to the coalition game (vθ, 2^A); this is standard cooperative game theory, but assumes the utility function summarizes all relevant value and ignores query cost or order effects.
  • domain assumption A surrogate over subset indicator vectors can generalize to unqueried subsets
    A-MESS-SURROGATE trains on M sampled subsets and evaluates on held-out subsets; this assumes vθ(S) is a smooth function of which attack categories are included (Section 3.1).
  • domain assumption Held-out test attacks measure the defended model's safety improvement
    Defense data and test data are separated (Appendix D), but with only 10 categories the held-out test attacks may overlap semantically with selected defense categories, so the split may overstate generalization.
  • domain assumption The three real settings are representative of jailbreak safety pipelines
    Llama-3-8B + ICD, Qwen2.5-7B + AT, and Ministral-3-8B + ICD are used for all real claims; there is no cross-model replication of the ASR-utility correlation or the subset-selection conclusion.

pith-pipeline@v1.3.0-alltime-deepseek · 14624 in / 13880 out tokens · 128803 ms · 2026-08-01T18:50:08.667124+00:00 · methodology

0 comments
read the original abstract

Jailbreak attacks on large language models are usually evaluated by attacker-centric metrics such as attack success rate (ASR), yet an attack that breaks a model is not necessarily useful for improving its safety. We propose a defender-centric view of jailbreak evaluation, where attacks are evaluated by the downstream safety improvements they enable when used as red-teaming data for safety training. Building on this view, we introduce A-MESS (Minimal Effective Attack-Subset Selection), a setting-agnostic framework for attributing and selecting jailbreak attacks from black-box subset utility observations. A-MESS estimates AttackSHAP, a Shapley-based score that attributes marginal utility to individual attacks and selects compact attack subsets under user-specified budgets via greedy or surrogate-based optimization. Across controlled utility landscapes and real LLM safety settings, we find that ASR rankings are weakly aligned with defender-centric utility, that AttackSHAP can be estimated accurately with limited utility queries, and that directly optimizing subsets yields stronger safety utility than attacker-centric or attribution-only selection. These results suggest evaluating jailbreak attacks as resources for improving safety, not only as tools for breaking models.

Figures

Figures reproduced from arXiv: 2607.17152 by Feiyang Lu, Jinfei Liu, Wenjie Wang, Xiaokai Mao, Yukai Zhou.

Figure 1
Figure 1. Figure 1: Conventional jailbreak research often follows an attacker-centric arms race, where attacks are primarily [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Attack ASR ranking differs from defender [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Synthetic evaluation of A-MESS-SURROGATE. Panel 3a evaluates surrogate utility prediction under increasing utility-query budgets. Panel 3b compares surrogate-based AttackSHAP estimates with reference full￾vector AttackSHAP values. Panel 3c measures top-k overlap with the attacks identified by reference AttackSHAP. We first evaluate whether A-MESS￾SURROGATE can estimate AttackSHAP with limited utility queri… view at source ↗
Figure 4
Figure 4. Figure 4: Example synthetic utility landscape. Each point corresponds to one attack subset, grouped by subset size [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

138 extracted references · 47 linked inside Pith

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  2. [2]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  3. [3]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  4. [4]

    ACM computing surveys , volume=

    Survey of hallucination in natural language generation , author=. ACM computing surveys , volume=. 2023 , publisher=

    2023

  5. [5]

    arXiv preprint arXiv:2310.01469 , year=

    Llm lies: Hallucinations are not bugs, but features as adversarial examples , author=. arXiv preprint arXiv:2310.01469 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2407.17436 , year=

    Air-bench 2024: A safety benchmark based on risk categories from regulations and policies , author=. arXiv preprint arXiv:2407.17436 , year=

    Pith/arXiv arXiv 2024

  7. [7]

    arXiv preprint arXiv:2410.06172 , year=

    Multimodal situational safety , author=. arXiv preprint arXiv:2410.06172 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2504.10694 , year=

    The Jailbreak Tax: How Useful are Your Jailbreak Outputs? , author=. arXiv preprint arXiv:2504.10694 , year=

    Pith/arXiv arXiv

  9. [9]

    Not Aligned

    " Not Aligned" is Not" Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak , author=. arXiv preprint arXiv:2406.11668 , year=

    Pith/arXiv arXiv

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Are aligned neural networks adversarially aligned? , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    Advances in Neural Information Processing Systems , volume=

    Jailbroken: How does llm safety training fail? , author=. Advances in Neural Information Processing Systems , volume=

  12. [12]

    do anything now

    " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models , author=. Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages=

    2024

  13. [13]

    33rd USENIX Security Symposium (USENIX Security 24) , pages=

    Don't listen to me: understanding and exploring jailbreak prompts of large language models , author=. 33rd USENIX Security Symposium (USENIX Security 24) , pages=

  14. [14]

    arXiv preprint arXiv:2307.15043 , year=

    Universal and transferable adversarial attacks on aligned language models , author=. arXiv preprint arXiv:2307.15043 , year=

    Pith/arXiv arXiv

  15. [15]

    2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages=

    Jailbreaking black box large language models in twenty queries , author=. 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages=. 2025 , organization=

    2025

  16. [16]

    Advances in Neural Information Processing Systems , volume=

    Tree of attacks: Jailbreaking black-box llms automatically , author=. Advances in Neural Information Processing Systems , volume=

  17. [17]

    arXiv preprint arXiv:2404.02151 , year=

    Jailbreaking leading safety-aligned llms with simple adaptive attacks , author=. arXiv preprint arXiv:2404.02151 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2307.08715 , year=

    Masterkey: Automated jailbreak across multiple large language model chatbots , author=. arXiv preprint arXiv:2307.08715 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2310.04451 , year=

    Autodan: Generating stealthy jailbreak prompts on aligned large language models , author=. arXiv preprint arXiv:2310.04451 , year=

    Pith/arXiv arXiv

  20. [20]

    Sicheng Zhu and Ruiyi Zhang and Bang An and Gang Wu and Joe Barrow and Zichao Wang and Furong Huang and Ani Nenkova and Tong Sun , booktitle=. Auto. 2024 , url=

    2024

  21. [21]

    arXiv preprint arXiv:2310.06987 , year=

    Catastrophic jailbreak of open-source llms via exploiting generation , author=. arXiv preprint arXiv:2310.06987 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2401.17256 , year=

    Weak-to-strong jailbreaking on large language models , author=. arXiv preprint arXiv:2401.17256 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2404.16873 , year=

    Advprompter: Fast adaptive adversarial prompting for llms , author=. arXiv preprint arXiv:2404.16873 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2405.21018 , year=

    Improved techniques for optimization-based jailbreaking on large language models , author=. arXiv preprint arXiv:2405.21018 , year=

    Pith/arXiv arXiv

  25. [25]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Don’t say no: Jailbreaking llm by suppressing refusal , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  26. [26]

    Zeyi Liao and Huan Sun , booktitle=. Ample. 2024 , url=

    2024

  27. [27]

    Edward Suh and Yevgeniy Vorobeychik and Zhuoqing Mao and Somesh Jha and Patrick McDaniel and Huan Sun and Bo Li and Chaowei Xiao , booktitle=

    Xiaogeng Liu and Peiran Li and G. Edward Suh and Yevgeniy Vorobeychik and Zhuoqing Mao and Somesh Jha and Patrick McDaniel and Huan Sun and Bo Li and Chaowei Xiao , booktitle=. Auto. 2025 , url=

    2025

  28. [28]

    2018 , eprint=

    Universal Language Model Fine-tuning for Text Classification , author=. 2018 , eprint=

    2018

  29. [29]

    Journal of Machine Learning Research , volume=

    Scaling instruction-finetuned language models , author=. Journal of Machine Learning Research , volume=

  30. [30]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  31. [31]

    arXiv preprint arXiv:1909.08593 , year=

    Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=

    Pith/arXiv arXiv 1909

  32. [32]

    arXiv preprint arXiv:2407.04295 , year=

    Jailbreak attacks and defenses against large language models: A survey , author=. arXiv preprint arXiv:2407.04295 , year=

    Pith/arXiv arXiv

  33. [33]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    JailbreakRadar: Comprehensive assessment of jailbreak attacks against LLMs , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  34. [34]

    arXiv e-prints , pages=

    LLM Jailbreak Attack versus Defense Techniques--A Comprehensive Study , author=. arXiv e-prints , pages=

  35. [35]

    arXiv preprint arXiv:2503.02574 , year=

    LLM-Safety Evaluations Lack Robustness , author=. arXiv preprint arXiv:2503.02574 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2309.05922 , year=

    A survey of hallucination in large foundation models , author=. arXiv preprint arXiv:2309.05922 , year=

    Pith/arXiv arXiv

  37. [37]

    ACM Transactions on Information Systems , volume=

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions , author=. ACM Transactions on Information Systems , volume=. 2025 , publisher=

    2025

  38. [38]

    arXiv preprint arXiv:2309.01219 , year=

    Siren's song in the AI ocean: a survey on hallucination in large language models , author=. arXiv preprint arXiv:2309.01219 , year=

    Pith/arXiv arXiv

  39. [39]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  40. [40]

    arXiv preprint arXiv:2404.01318 , year=

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models , author=. arXiv preprint arXiv:2404.01318 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2406.09321 , year=

    Jailbreakeval: An integrated toolkit for evaluating jailbreak attempts against large language models , author=. arXiv preprint arXiv:2406.09321 , year=

    Pith/arXiv arXiv

  42. [42]

    NeurIPS 2024 Competition Track , year=

    CLAS 2024: The Competition for LLM and Agent Safety , author=. NeurIPS 2024 Competition Track , year=

    2024

  43. [43]

    arXiv preprint arXiv:2404.06407 , year=

    Rethinking How to Evaluate Language Model Jailbreak , author=. arXiv preprint arXiv:2404.06407 , year=

    Pith/arXiv arXiv

  44. [44]

    2023 , url =

    Alex Albert , title =. 2023 , url =

    2023

  45. [45]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  46. [46]

    2024 , eprint=

    Jailbreaking Large Language Models with Symbolic Mathematics , author=. 2024 , eprint=

    2024

  47. [47]

    arXiv preprint arXiv:2309.10253 , year=

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts , author=. arXiv preprint arXiv:2309.10253 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2504.21038 , year=

    Prefill-Based Jailbreak: A Novel Approach of Bypassing LLM Safety Boundary , author=. arXiv preprint arXiv:2504.21038 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2406.14598 , year=

    Sorry-bench: Systematically evaluating large language model safety refusal behaviors , author=. arXiv preprint arXiv:2406.14598 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2402.10260 , year=

    A strongreject for empty jailbreaks , author=. arXiv preprint arXiv:2402.10260 , year=

    Pith/arXiv arXiv

  51. [51]

    Advances in Neural Information Processing Systems , volume=

    The art of saying no: Contextual noncompliance in language models , author=. Advances in Neural Information Processing Systems , volume=

  52. [52]

    The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  53. [53]

    arXiv preprint arXiv:2109.07958 , year=

    Truthfulqa: Measuring how models mimic human falsehoods , author=. arXiv preprint arXiv:2109.07958 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2503.23278 , year=

    Model context protocol (mcp): Landscape, security threats, and future research directions , author=. arXiv preprint arXiv:2503.23278 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2310.06474 , year=

    Multilingual jailbreak challenges in large language models , author=. arXiv preprint arXiv:2310.06474 , year=

    Pith/arXiv arXiv

  56. [56]

    Advances in Neural Information Processing Systems , volume=

    Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types , author=. Advances in Neural Information Processing Systems , volume=

  57. [57]

    arXiv preprint arXiv:2406.05946 , year=

    Safety alignment should be made more than just a few tokens deep , author=. arXiv preprint arXiv:2406.05946 , year=

    Pith/arXiv arXiv

  58. [58]

    ACM Transactions on Knowledge Discovery from Data , volume=

    Harnessing the power of llms in practice: A survey on chatgpt and beyond , author=. ACM Transactions on Knowledge Discovery from Data , volume=. 2024 , publisher=

    2024

  59. [59]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    From generation to judgment: Opportunities and challenges of llm-as-a-judge , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  60. [60]

    arXiv preprint arXiv:2403.18105 , year=

    Large language models for education: A survey and outlook , author=. arXiv preprint arXiv:2403.18105 , year=

    Pith/arXiv arXiv

  61. [61]

    Informatics , volume=

    Large language models in healthcare and medical domain: A review , author=. Informatics , volume=. 2024 , organization=

    2024

  62. [62]

    Authorea preprints , volume=

    Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects , author=. Authorea preprints , volume=. 2023 , publisher=

    2023

  63. [63]

    arXiv preprint arXiv:2406.00515 , year=

    A survey on large language models for code generation , author=. arXiv preprint arXiv:2406.00515 , year=

    Pith/arXiv arXiv

  64. [64]

    Findings of the association for computational linguistics: ACL 2023 , pages=

    Towards reasoning in large language models: A survey , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

    2023

  65. [65]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  66. [66]

    Nature Machine Intelligence , volume=

    What large language models know and what people think they know , author=. Nature Machine Intelligence , volume=. 2025 , publisher=

    2025

  67. [67]

    Nature Machine Intelligence , volume=

    Defending chatgpt against jailbreak attack via self-reminders , author=. Nature Machine Intelligence , volume=. 2023 , publisher=

    2023

  68. [68]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  69. [69]

    SafeWork-R1: Coevolving Safety and Intelligence under the AI-45\^

    Lab, Shanghai AI and Bao, Yicheng and Chen, Guanxu and Chen, Mingkang and Chen, Yunhao and Chen, Chiyu and Chen, Lingjie and Chen, Sirui and Chen, Xinquan and Cheng, Jie and others , journal=. SafeWork-R1: Coevolving Safety and Intelligence under the AI-45\^

  70. [70]

    arXiv preprint arXiv:2501.17805 , year=

    International ai safety report , author=. arXiv preprint arXiv:2501.17805 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2108.07258 , year=

    On the opportunities and risks of foundation models , author=. arXiv preprint arXiv:2108.07258 , year=

    Pith/arXiv arXiv

  72. [72]

    URL: https://nvlpubs

    Artificial intelligence risk management framework (AI RMF 1.0) , author=. URL: https://nvlpubs. nist. gov/nistpubs/ai/nist. ai , pages=

  73. [73]

    2023 , publisher=

    Interim Measures for the Management of Generative Artificial Intelligence Services (2023)(Translation) , author=. 2023 , publisher=

    2023

  74. [74]

    Official Journal of the European Union , number =

    Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) , year =. Official Journal of the European Union , number =

    2024

  75. [75]

    arXiv preprint arXiv:1606.06565 , year=

    Concrete problems in AI safety , author=. arXiv preprint arXiv:1606.06565 , year=

    Pith/arXiv arXiv

  76. [76]

    arXiv preprint arXiv:2510.09023 , year=

    The attacker moves second: Stronger adaptive attacks bypass defenses against llm jailbreaks and prompt injections , author=. arXiv preprint arXiv:2510.09023 , year=

    Pith/arXiv arXiv

  77. [77]

    2023 , institution =

    Responsible Scaling Policy , author =. 2023 , institution =

    2023

  78. [78]

    2024 , institution =

    Frontier Safety Framework , author =. 2024 , institution =

    2024

  79. [79]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  80. [80]

    Learning and individual differences , volume=

    ChatGPT for good? On opportunities and challenges of large language models for education , author=. Learning and individual differences , volume=. 2023 , publisher=

    2023

Showing first 80 references.