Pith. sign in

REVIEW 2 major objections 3 minor 44 references

A one-line prefill jailbreak can flip an aligned model to compliance while its internal harm representation stays fully intact — refusal, the paper argues, is a shallow response-site computation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:27 UTC pith:4SZQPFNM

load-bearing objection Careful, well-controlled mechanistic study; the passive-conditioning conclusion needs a templated base-model control before it is settled. the 2 major comments →

arxiv 2607.14147 v1 pith:4SZQPFNM submitted 2026-07-14 cs.CL cs.AIcs.CRcs.LG

Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak

Alex Kwon This is my paper
classification cs.CL cs.AIcs.CRcs.LG
keywords prefill jailbreakrefusal mechanismrepresentation probingactivation patchingattention knockoutsafety alignmentautoregressive conditioningmechanistic interpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the one-line prefill jailbreak ('Sure, here is') does not erase the model's representation of harm: a probe reads the complied-with prompts as harmful as the refused ones (0.91–0.98), while refusal behavior drops to chance. Refusal, it argues, is a shallow, response-site computation: the attack wins in an early window of the response, through attention to the forced prefix, and the dominant mechanism is generic autoregressive conditioning rather than safety-specific suppression. The causal evidence includes a dose-matched position control, state-transfer reversal, and attention knockout with a mass-matched control. If correct, a monitor reading the prompt-side representation is immune to this class of attack by construction, and refusal is positionally fragile even though it is distributed in representation space.

Core claim

Under the prefill attack, the model acts on the request as if it were benign while its internal read of the prompt is as harmful as ever: a linear probe scores the flipped prompts 0.91–0.98, level with the ones the model refuses, and behavioral refusal falls to chance. The refusal decision is computed at the response site, not at the prompt: restoring the harm direction over the first half of the response re-engages refusal as much as the whole response, while the second half is nearly inert, and transplanting the model's own refuse-state re-induces refusal in 74% of held-out cases. Knocking out early response attention to the prefill, but not an equal attention mass elsewhere, selectively s

What carries the argument

The prefill attack itself — an affirmative prefix appended to the model's own response — is the central object; because it never touches the prompt tokens, the prompt-side representation is invariant by construction. The paper's causal instruments are the early-window position control (dose-matched bands of the response), state transfer (injecting the plain-condition minus attack-condition residual), and the attention knockout that severs early response-to-prefill attention edges, with a mass-matched control. The base-model discriminator — running the same knockout on a non-safety-tuned base model — is what separates generic autoregressive conditioning from safety-specific suppression.

Load-bearing premise

The passive mechanism conclusion rests on treating a non-safety-tuned base model as a clean counterfactual for the instruction-tuned model; if base and instruct models differ in more than safety tuning (chat template, instruction following, pretraining distribution), the attribution of the knockout effect to generic autoregressive conditioning is weakened.

What would settle it

Find a checkpoint pair that differ only by safety tuning (same base, same chat template, with and without RLHF/DPO) and run the early prefill-attention knockout on both. If the base model does not show the prefill-specific collapse of harmful content (the 64%→25% pattern), the passive-reading claim is false. Alternatively, train a model with a different chat template but identical safety tuning: if the knockout effect disappears, the confound, not the mechanism, was responsible.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A prompt-side monitor that reads the last-prompt-token representation catches this attack at 100% with 0% false positives on in-distribution negatives, because the attack cannot alter that representation.
  • Refusal restoration is a model-dependent fallback: cutting the prefill's early-window attention re-engages refusal in instruction-tuned models but only degrades continuations in base models.
  • No single direction, head, or layer is a clean refusal handle; the decision is decodable but distributed, so single-direction steering fails while a full linear probe reads it.
  • At larger scale (7B, 14B) the dissociation holds and the passive mechanism persists in the content channel, though the behavioral refusal signature becomes model-dependent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the passive account is right, defenses that focus on the response window will keep chasing the attack across positions; the durable placement is prompt-side, and this extends to any response-site jailbreak that occupies the early output window.
  • The harm-direction's partial write-handle suggests a possible 'representation repair' at the response site that could be combined with a monitor, but the paper's own steering nulls indicate this is not a clean intervention.
  • A direct test of the weakest assumption: use a checkpoint pair that differ only by safety tuning (same base, same chat template, with and without RLHF/DPO) and re-run the knockout — if the base model fails to show the prefill-specific collapse, the passive reading is confounded.
  • Prompt-modifying jailbreaks should behave differently from prefill: they attenuate or displace the prompt-side representation, so a representation monitor would be strictly weaker against them; this is a testable prediction the paper leaves implicit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper studies the "Sure, here is" prefill jailbreak in open-weight instruction-tuned models (1.5–3.8B, with checks at 7B and 14B). It reports a dissociation: a linear probe trained on plain harmful/benign prompts reads the last-prompt-token hidden states of attacked prompts as harmful (0.91–0.98) while behavioral refusal collapses to chance. Because prefill tokens are appended to the response, this intactness is partly structural, and the paper is explicit about that. The paper then localizes the behavioral failure causally to an early response window using activation patching, state transfer, and a preregistered attention-knockout, and uses a base-model control to argue that the prefill's grip is generic autoregressive conditioning rather than safety-specific suppression. It concludes that a prompt-side monitor is immune to response-site attacks by construction, and that refusal is distributed in representation space but positionally fragile.

Significance. If the main claims hold, the paper makes a strong contribution: it moves the account of prefill jailbreaks from behavioral depth-shallowness to a causal, response-site mechanism, and it gives a structural boundary for representation-based monitoring. The paper is unusually careful methodologically: preregistered knockout, mass-matched and dose-matched controls, held-out transfers, judge-free coherence reads, a claims-and-evidence ledger, and explicit retraction/demotion of earlier interpretations. These strengths make the core dissociation and the early-window localization credible. The main unresolved risk is the identification of the passive mechanism, which depends on a base-model control that is confounded by formatting; this is fixable but currently load-bearing.

major comments (2)
  1. [§4.3, App. E] The passive-vs-active resolution rests on a confounded counterfactual. The instruct models are evaluated under the chat template; App. E states that the Qwen2.5-1.5B base model is run "with a raw-completion format (no chat template)". The base-vs-instruct comparison therefore changes safety tuning and conditioning context simultaneously. A prefill-specific attention-knockout collapse in a raw-completion base model may reflect generic dependence on a recent forced prefix in un-templated continuation, not the absence of a safety-specific gate. Because the abstract and §7 state the central question as resolved toward passive conditioning, this gap is load-bearing. A templated base control, or a base model run with the chat template (or an instruct model run in raw format), is required before the generic claim is established; the same confound colors the MLP path-patch and the logit-trace co
  2. [§4.3, Table 3; App. E] The mass-matched knockout control matches total baseline attention mass but not serial position/recency. The prefill keys are the final prompt-position keys; if early-window attention to any recent key is what sustains compliance, a control removing equal mass from the most recent non-prefill keys might also collapse the harmful continuation. The current "non-prefill prompt keys" selection is not described as recency-matched. Please add a position/recency-matched control, or a key-position permutation, to separate "prefill-specific" from "recent-position-specific." Without this, the statement that the grip operates through attention to the forced prefix specifically is not fully settled.
minor comments (3)
  1. [§4.3] When the base-model discriminator is first introduced, the main text should explicitly note that the base model is run in raw-completion format without the chat template; currently that fact appears only in App. E. This is essential for readers to assess the counterfactual.
  2. [Table 2 / App. D] The 24-seed random-control distribution is right-skewed (range 0–60%, median 0.12). Reporting the median alongside the mean in Table 2 would be more informative than the mean alone, since the harm−random delta is driven partly by a few high outliers.
  3. [§5.1] The PCA rank analysis at n=70 is appropriately hedged in the appendix, but the main text's phrase "the signal is spread across directions" could be read as a stronger rank claim than the data support. A one-sentence reminder that the high-rank reading is exploratory at n=70 would help.

Circularity Check

0 steps flagged

No significant circularity: the paper's causal claims are intervention-based with held-out controls, and the one structural claim is explicitly labeled by-construction.

full rationale

The derivation chain is self-contained. The intactness of the prompt-side harm representation under prefill is openly declared to be structural, not a measurement: 'the prefill appends to the response, so by the causal structure of the transformer this representation is bit-for-bit identical with and without the attack.' The substantive claims — early-window localization, causal restoration via harm-direction patching and state transfer, attention-knockout specificity, and the passive-vs-active mechanism — are each tested against matched-norm random controls, mass-matched attention controls, held-out prompt splits, and preregistered falsifiers. No fitted parameter is renamed as a prediction: the harm direction and refuse-state vector are estimated on disjoint training splits and evaluated on held-out prompts, and the attention knockout is preregistered with an interpretation table. The single self-citation (Kwon 2026) supplies only a framing template ('read what a model represents versus what it acts on') and is not load-bearing; no uniqueness theorem, ansatz, or result is imported from it. Potential weaknesses such as the base-model discriminator's raw-completion format or the probe's AdvBench calibration are confound/correctness concerns, not circular reductions of the paper's equations to their inputs. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The paper's central claims are empirical and rest on standard interpretability methods (probing, patching, attention knockout) plus several domain assumptions. The main unstated load is the assumption that the base model is a clean passive-conditioning counterfactual. There are no invented physical entities; the 'harm direction' and 'refusal attractor' are descriptive constructions, not new postulated elements.

free parameters (5)
  • Harm-probe weights (logistic regression, C=0.5) = trained on plain harmful/benign; weights not reported as scalar
    Used as the harm readout in §3-§5; fit to data, but validated by XSTest surface control and structural invariance.
  • Patching cell (layer and injection coefficient) = L12, α=1; L8, α=1.5
    Grid-selected before fixed-cell n=100; selection optimism affects absolute restoration but not the harm-vs-random delta.
  • State-transfer direction d_reverse = mean(s_plain - s_attack) over 260 train prompts
    Estimated on train split and tested on held-out split; a fitted vector used as the causal lever in §4.2.
  • Attention-knockout strength λ = λ=1 (full edge cut); λ=0.5 as threshold control
    Intervention strength chosen by design; the dose threshold is part of the evidence, not a derived constant.
  • Early-window cutoff = first half (prefill span plus generated tokens g<9); boundary sweep saturates by first third
    The position window is defined operationally; the paper's boundary sweep shows the exact cutoff is not a knife-edge.
axioms (5)
  • domain assumption A linear probe on hidden states is a valid readout of the model's represented harm
    Used throughout §3 and §4 to equate probe scores with internal representation; partially validated by XSTest (surface control) and by the structural invariance argument.
  • domain assumption String-match refusal plus the judge-free content scorer correctly classifies behavioral refusal vs coherent compliance
    Used in all behavioral endpoints; validated by manual read on a subset (§7), but the manual read text is withheld.
  • domain assumption The base model (no safety tuning) has no safety-specific mechanism, making it a clean passive-conditioning counterfactual
    The passive-vs-active conclusion in §4.3/App. E rests on this; the base model uses raw-completion format and no chat template, which could differ from the instruct model in ways other than safety tuning.
  • standard math Attention logit editing with softmax renormalization is a causally faithful intervention
    Attention knockout methodology; supported by unit tests (sham identity, edge-only masking, KV-cache consistency) in App. E.
  • standard math The prefill appends to the response and thus leaves prompt-side activations bit-for-bit unchanged
    Structural property of transformer autoregressive decoding; the paper uses this to argue the harm readout is invariant by construction (§3).
invented entities (1)
  • Small safety-specific refusal attractor no independent evidence
    purpose: Explains the residual tendency of the instruct model to route knockout-freed probability mass to refusal tokens (concentration 0.24 vs 0.03 in base).
    Not a new physical entity but a postulated latent component; evidence is internal to the paper (logit-trace concentration), no external falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 30908 in / 18699 out tokens · 185214 ms · 2026-08-02T06:27:05.853128+00:00 · methodology

0 comments
read the original abstract

Aligned language models refuse harmful requests, but a one-line prefill ("Sure, here is") strips the refusal. We ask where and how it fails. The harm representation stays intact: on the prompts the attack flips to compliance, a linear probe reads harm as high as on the refused ones (0.91-0.98), while behavioral refusal drops to chance. This holds across four models and three families (1.5-3.8B, and at 14B). Refusal is therefore a shallow, response-site computation. We localize it to an early window: a dose-matched position control shows the first half of the response suffices to break refusal, while the second half is nearly inert. Three causal probes converge on that window. Restoring the harm direction there partially re-engages refusal. Injecting the model's own refuse-state reverses the jailbreak (74%, held-out). And knocking out the early response's attention to the prefill, but not an equal attention mass elsewhere, selectively collapses the harmful continuation. A base-model control identifies the mechanism: the same knockout collapses the continuation prefill-specifically even in a non-safety-tuned base model (64% to 25% harmful content vs a matched control's 64%, replicated at 7B). So the prefill's grip is generic autoregressive conditioning, not safety-specific suppression, and "refusal restoration" is a model-dependent fallback. The dominant mechanism is passive. A small safety-specific attractor remains on top (logit-trace concentration 0.24 vs 0.03), whose active-vs-passive character we size but do not fully separate. No single direction or component is a clean handle either: the decision is decodable but distributed, and refusal tracks harm rather than scary surface. The consequence is structural: a monitor reading the untouched prompt-side representation is immune by construction, but only to response-site attacks. The mechanism is diffuse; the failure surface is local.

Figures

Figures reproduced from arXiv: 2607.14147 by Alex Kwon.

Figure 1
Figure 1. Figure 1: The prefill’s grip is an early response window (the central finding). Restoring the harm direction over a dose-matched band of the response under the attack (n = 100, Qwen2.5-1.5B, L12/α=1; §4.1, App. D): the first half restores refusal to 42%, as much as the whole response (41%), while the second half restores only 6% at double strength and the onset alone 9%. An early multi-token window suffices; the lat… view at source ↗
Figure 2
Figure 2. Figure 2: The onset projection is illustrative only: neither a clean measurement nor the causal locus. Projection onto the harm-probe direction across decoding steps: the no-attack harmful state sits high at onset (2.80), the prefill state low (0.22). We do not read this as a displacement measurement: a fully-benign forced prefix drives the onset projection even lower (−0.82; §4.4), so the value reflects the content… view at source ↗
Figure 3
Figure 3. Figure 3: Refusal tracks harm, not surface. Refusal rate and harm-probe score by cell. Stripping trigger words from harmful prompts changes neither (harmful-scary ≈ harmful-clean on both bars). The keyword effect is real but secondary and appears as over-refusal on benign scary-sounding prompts (benign-scary refusal 0.59 while its harm-probe score stays low at 0.23). harm representation is keyword-invariant (probe 0… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 18 linked inside Pith

  1. [3]

    International Conference on Learning Representations (ICLR) , year =

    Safety Alignment Should Be Made More Than Just a Few Tokens Deep , author =. International Conference on Learning Representations (ICLR) , year =

  2. [4]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Jailbroken: How Does LLM Safety Training Fail? , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  3. [8]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Refusal in Language Models Is Mediated by a Single Direction , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  4. [12]

    and Guo, K

    Zhang, Shenyi and Zhai, Y. and Guo, K. and Hu, H. and Guo, S. and Fang, Z. and Wang, Q. and others , booktitle =. 2025 , note =

    2025

  5. [16]

    BlackboxNLP Workshop , year =

    What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks , author =. BlackboxNLP Workshop , year =

  6. [18]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Locating and Editing Factual Associations in GPT , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  7. [19]

    NAACL , year =

    XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models , author =. NAACL , year =

  8. [20]

    2026 , howpublished =

    Arcanum Prompt Injection Taxonomy , author =. 2026 , howpublished =

    2026

  9. [24]

    , year =

    Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B. , year =. Stanford Alpaca: An Instruction-following

  10. [25]

    Catastrophic Jailbreak of Open-source

    Huang, Yangsibo and Gupta, Samyak and Xia, Mengzhou and Li, Kai and Chen, Danqi , booktitle =. Catastrophic Jailbreak of Open-source. 2024 , note =

    2024

  11. [26]

    Think You Have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal =. Think You Have Solved Question Answering? Try

  12. [27]

    Which Agent Causes Task Failures and When? On Automated Failure Attribution of

    Zhang, Shaokun and Yin, Ming and Zhang, Jieyu and Liu, Jiale and Han, Zhiguang and Zhang, Jingyang and Li, Beibin and Wang, Chi and Wang, Huazheng and Chen, Yiran and Wu, Qingyun , booktitle =. Which Agent Causes Task Failures and When? On Automated Failure Attribution of. 2025 , note =

    2025

  13. [28]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    Transformers: State-of-the-Art Natural Language Processing , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    2020

  14. [29]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    PyTorch: An Imperative Style, High-Performance Deep Learning Library , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  15. [30]

    Journal of Machine Learning Research , volume =

    Scikit-learn: Machine Learning in Python , author =. Journal of Machine Learning Research , volume =

  16. [31]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    Pith/arXiv arXiv 2024

  17. [32]

    Smollm2: When smol goes big -- data-centric training of a small language model

    Loubna Ben Allal et al. Smollm2: When smol goes big -- data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025

    Pith/arXiv arXiv 2025

  18. [33]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.11717

    Pith/arXiv arXiv 2024

  19. [34]

    What features in prompts jailbreak llms? investigating the mechanisms behind attacks

    Sarah Ball et al. What features in prompts jailbreak llms? investigating the mechanisms behind attacks. In BlackboxNLP Workshop, 2025. arXiv:2411.03343

    arXiv 2025

  20. [35]

    Before the last token: Diagnosing final-token safety probe failures

    Shravan Doda. Before the last token: Diagnosing final-token safety probe failures. arXiv preprint arXiv:2605.12726, 2026

    Pith/arXiv arXiv 2026

  21. [36]

    Arcanum prompt injection taxonomy

    Jason Haddix. Arcanum prompt injection taxonomy. Arcanum Information Security, 2026. Version 1.6.1, CC BY 4.0. https://www.arcanum-sec.com/pitax

    2026

  22. [37]

    P. Han, C. Qian, X. Chen, Y. Zhang, D. Zhang, and H. Ji. Safeswitch: Steering unsafe llm behavior via internal activation signals. arXiv preprint arXiv:2502.01042, 2025

    arXiv 2025

  23. [38]

    Catastrophic jailbreak of open-source LLMs via exploiting generation

    Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source LLMs via exploiting generation. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06987

    Pith/arXiv arXiv 2024

  24. [39]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  25. [40]

    Jiang, X

    Y. Jiang, X. Gao, T. Peng, Y. Tan, X. Zhu, B. Zheng, and X. Yue. Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states. arXiv preprint arXiv:2502.14744, 2025

    Pith/arXiv arXiv 2025

  26. [41]

    F. Joad, M. Hawasly, S. Boughorbel, N. Durrani, and H. T. Sencar. There is more to refusal in large language models than a single direction. arXiv preprint arXiv:2602.02132, 2026

    arXiv 2026

  27. [42]

    They infer what you meant: Models represent communicative intent more reliably than they act on it

    Alex Kwon. They infer what you meant: Models represent communicative intent more reliably than they act on it. arXiv preprint arXiv:2607.03598, 2026

    Pith/arXiv arXiv 2026

  28. [43]

    Y. Li, J. Hu, W. Sang, L. Ma, D. Nie, W. Zhang, Q. Zhou, et al. Prefill-level jailbreak: A black-box risk analysis of large language models. arXiv preprint arXiv:2504.21038, 2025

    Pith/arXiv arXiv 2025

  29. [44]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2202.05262

    Pith/arXiv arXiv 2022

  30. [45]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  31. [46]

    Scikit-learn: Machine learning in python

    Fabian Pedregosa, Ga \"e l Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12: 0 2825--2830, 2011

    2011

  32. [47]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. In International Conference on Learning Representations (ICLR), 2025. Outstanding Paper Award. arXiv:2406.05946

    Pith/arXiv arXiv 2025

  33. [48]

    Qwen2.5 technical report

    Qwen Team . Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  34. [49]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul R \"o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. In NAACL, 2024. arXiv:2308.01263

    Pith/arXiv arXiv 2024

  35. [50]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  36. [51]

    Jailbroken: How does llm safety training fail? In Advances in Neural Information Processing Systems (NeurIPS), 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2307.02483

    Pith/arXiv arXiv 2023

  37. [52]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.\ 38--45, 2020

    2020

  38. [53]

    Zhang, A

    J. Zhang, A. Estornell, D. D. Baek, B. Li, and X. Xu. Any-depth alignment: Unlocking innate safety alignment of llms to any-depth. arXiv preprint arXiv:2510.18081, 2025 a

    arXiv 2025

  39. [54]

    Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems. In International Conference on Machine Learning (ICML), 2025 b . Spotlight. Who&When benchmark. arXiv:2505.00212

    Pith/arXiv arXiv 2025

  40. [55]

    Shenyi Zhang, Y. Zhai, K. Guo, H. Hu, S. Guo, Z. Fang, Q. Wang, et al. JBShield : Defending large language models from jailbreak attacks through activated concept analysis and manipulation. In 34th USENIX Security Symposium (USENIX Security 25), pp.\ 8215--8234, 2025 c . arXiv:2502.07557

    Pith/arXiv arXiv 2025

  41. [56]

    Zhang, T

    Y. Zhang, T. Liu, Z. Zhao, G. Meng, and K. Chen. Bleeding pathways: Vanishing discriminability in llm hidden states fuels jailbreak attacks. arXiv preprint arXiv:2503.11185, 2025 d

    arXiv 2025

  42. [57]

    Advprefix: An objective for nuanced llm jailbreaks

    Sicheng Zhu et al. Advprefix: An objective for nuanced llm jailbreaks. arXiv preprint arXiv:2412.10321, 2024

    arXiv 2024

  43. [58]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023 a

    Pith/arXiv arXiv 2023

  44. [59]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023 b

    Pith/arXiv arXiv 2023