Safety Alignment Should Be Made More Than Just A Few Attention Heads

Chao Huang; Chuang Zhang; Juewei Yue; Quangang Li; Tingwen Liu; Zefeng Zhang

REVIEW 4 major objections 6 minor 47 references

Safety Alignment Should Be Made More Than Just A Few Attention Heads

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Safety-critical behavior of large language models is concentrated in tens of attention heads, and a dropout-based training trick spreads it out, sharply reducing jailbreak success.

desk verdict The RDSHA ablation study is credible and useful, but AHD's main claim is undercut by a missing no-dropout control; worth a referee but needs a revision. read the letter →

arxiv 2508.19697 v1 pith:6XNFRFD2 submitted 2025-08-27 cs.CR cs.AIcs.CL

Chao Huang , Zefeng Zhang , Juewei Yue , Quangang Li , Chuang Zhang , Tingwen Liu This is my paper

classification cs.CRcs.AIcs.CL

keywords attentionheadssafetyalignmentjailbreakattacksrefusaldirectionheadablationrepresentationengineeringadversarialrobustnessmechanisticinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an LLM's safety behavior — its refusal to comply with harmful requests — is carried by a small set of attention heads, often just 20 to 50 out of thousands, and that surgically removing those heads turns a model that says "I can't help with that" into one that produces step-by-step instructions for poisoning food or committing mail fraud. To find and test those heads, it introduces RDSHA, which scores each attention head by how strongly its output lines up with the model's internal refusal direction, then ablates the top-scoring heads. The paper further shows that four published jailbreak attacks all reduce the activation of exactly these heads, making the case that the concentration itself is the vulnerability. As a fix, it proposes AHD, a training-time head dropout that forces refusal behavior to be encoded redundantly across many heads. After AHD, ablating the same heads barely degrades safety, jailbreak harmfulness rates fall from near 100% to near zero across most attacks, and standard utility and over-refusal benchmarks stay flat.

What carries the argument

The load-bearing object is the refusal direction: the difference between the mean residual-stream activations over harmful and harmless prompts, taken at an empirically selected layer. RDSHA turns it into a per-head score by projecting each head's output onto that direction and normalizing — heads with the largest projection are the ones whose removal breaks refusal. AHD targets the same score's distribution: a hook before the multi-head attention output projection applies a Bernoulli mask during training, dropping each attention head with probability 0.5 on safety data (and 0 on benign data), so the model cannot concentrate the refusal computation in a few heads. The paper's argument is eff

What would settle it

A control experiment settles it: fine-tune each of the four models on the identical mixture of 256 harmful instructions plus Alpaca data with the AHD mask forced to always keep every head. If that no-dropout control achieves the same near-zero harmfulness rates under AutoDAN, SI-GCG, and adaptive attacks, the dropout itself adds nothing. If the control stays vulnerable while AHD models do not, the mechanism is confirmed.

Watch

Extended reading notes

Core claim

On the paper's own terms: current safety alignment is structurally shallow. Safety-critical behavior is concentrated in a small subset of attention heads, and RDSHA can locate them: for each harmful prompt, it projects the last-token output of every head onto the model's global refusal direction and masks the highest-scoring heads during inference. Removing roughly 20 to 50 heads turns refusals into fluent harmful answers on Llama-2, Llama-3, Qwen, and Qwen-2, and the same few heads — mostly in middle and upper layers — keep ranking critical across all 50 test prompts. Jailbreak prompts from four attack families show markedly lower cumulative safety-influence scores on these heads than the o

Load-bearing premise

The paper credits the improved jailbreak resistance to the head-dropout mechanism itself, but every AHD experiment also includes standard fine-tuning on 256 harmful instructions; without a control trained on the same data with dropout disabled, the extra safety fine-tuning alone could explain the gains.

Editorial extensions

If this is right

RDSHA provides a concrete, prompt-level measurement of how concentrated a model's safety is, so future alignment work could verify redundancy before deployment rather than after an attack succeeds.
AHD pushes harmfulness rates from near 100% down to near zero on AutoDAN-GA, AutoDAN-HGA, SI-GCG, and adaptive attacks for Llama-2, Llama-3, and Qwen, and sharply down for Qwen-2.
The safety gain is not a refusal dial: over-refusal on OR-Bench-Hard-1K stays flat or slightly decreases, and six standard utility benchmarks are essentially unchanged.
Because safety is encoded redundantly after AHD, removing or disabling any small group of heads no longer collapses the model's refusal behavior, closing the specific attack vector the paper documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The approach implies a broader design rule: any property a deployer cares about — factuality, instruction following, resistance to prompt injection — could be measured with a concentration index like RDSHA's and intentionally spread across heads during training.
A natural next experiment, beyond the paper: adversarially fine-tune a post-AHD model and check whether safety re-concentrates into a few heads, which would show how durable the redistribution is under continued adaptation.
If the mechanism is correct, common post-training steps that remove or deactivate heads — pruning, model merging, quantization with head-level effects — could silently undo the redundancy; re-running the RDSHA profile after such steps would be a cheap regression test.
The residual attacks that still succeed against Qwen-2 (21% under AutoDAN-HGA) suggest that redistribution is not by itself a complete defense; pairing AHD with input-level defenses is a plausible next step the paper leaves open.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, read this one with an eye on the control group. The paper shows that ablating a few dozen attention heads identified by refusal-direction projection (RDSHA) flips safety-aligned models into generating harmful content. That part holds up: four models, consistent curves, and good case examples. If you work on mechanistic safety, the RDSHA procedure itself is a clean, reproducible method for ranking heads, and the finding that safety is fragile to head ablation is worth knowing.

What's new is the proposed defense, AHD, which applies per-head dropout during safety fine-tuning to force safety features to spread across more heads. The tables show dramatic jailbreak ASR reductions and roughly unchanged utility. But here's the problem: every AHD comparison is against the original base model, not against a model fine-tuned on the same 256 harmful instructions plus Alpaca data with the dropout disabled. Eq. (9) with beta1=0 gives that control, and it's not run. So we don't know whether the robustness gain comes from the dropout mechanism or just from additional safety SFT, which is known to help. The Figure 1b claim that safety is distributed after AHD has the same confound: safety SFT alone could make the model more robust to any head ablation. This is load-bearing and untested. The footnote saying beta1=0.1 or 0.3 leads to overfitting and utility degradation is also unsupported by data.

The jailbreak-exploitation analysis is suggestive but indirect: jailbreak prompts show lower cumulative projection scores on the top-8 heads. That's correlation, not evidence that attacks selectively bypass those heads.

The paper is honestly written and acknowledges the dataset limitation, but the missing control is a serious gap. These are fixable: run the no-dropout SFT control, report beta1 sweep, and possibly test AHD against a matched dropout on random heads rather than all heads. If the control shows no difference, AHD is just SFT.

Recommendation: this deserves a serious referee because the diagnostic is sound and the defense idea is worth testing. I'd accept it for review but expect major revision.

Referee Report

4 major / 6 minor

Summary. The paper studies the architectural concentration of safety in LLMs. It introduces RDSHA, which scores attention heads by projecting their outputs onto the model's refusal direction and ablating the top-scoring heads; experiments on four LLMs show that ablating roughly 20–50 such heads sharply increases the harmfulness rate, and that jailbreak prompts reduce the cumulative safety-influence score of the top heads. The paper then proposes AHD, a fine-tuning strategy that stochastically drops attention heads during safety training, and reports that AHD-trained models retain higher safety under RDSHA ablation and under four jailbreak attacks, while keeping utility and over-refusal rates roughly unchanged.

Significance. If the causal claims are validated, the paper offers a practically relevant and mechanistically interesting result: safety in aligned LLMs may be over-concentrated in a small set of attention heads, and a simple dropout regularizer may make safety more distributed and more robust. The paper has clear strengths: it evaluates on four diverse models, uses an independent judge for harmfulness, checks over-refusal, provides case studies, and recomputes the refusal direction after training. The main limitation is that the central benefit of AHD is not causally identified: all AHD comparisons are against the original model, not against a no-dropout control trained on the same safety data, so the observed robustness gains could stem from the additional safety fine-tuning itself.

major comments (4)

[§4.2, Eq. (9), Table 1] The central claim that AHD's dropout mechanism improves jailbreak robustness is untested. All AHD models are compared with the original pre-trained models (Table 1), but the training objective (Eq. 9) also performs safety SFT on 256 harmful instructions with model-generated responses. Safety SFT alone is known to reduce jailbreak success. A no-dropout control trained with the same DH and DB and the same loss, with β1=0 (no AHD), is required to attribute the Table 1 gains to the dropout mechanism rather than to the extra safety data. Please add this control and report its performance under the same attacks.
[§4.3, Figure 1b] The RDSHA ablation curves after AHD are likewise confounded. A safety-SFT control without head dropout could also become more robust to head ablation, e.g., because the model's refusal behavior becomes stronger or more general, without any redistribution across heads. The comparison between Figure 1a and Figure 1b therefore does not by itself establish that AHD 'distributes' safety across heads. The no-dropout control's RDSHA curve should be included in Figure 1b.
[§3.3, Figure 1a] The 'safety is concentrated in a few heads' conclusion lacks a random-ablation baseline. Figure 1a shows that ablating up to 200 RDSHA-selected heads increases harmfulness, but without ablating the same number of randomly chosen heads, or heads selected by an alternative attribution method, it is possible that any set of 50–200 heads would produce similar degradation. The paper claims a 'critical dependence on these few heads,' and this is load-bearing for the motivation for AHD. Please add random-head ablation curves for each model.
[§4.1, footnote 4] The statement that lower dropout rates (β1=0.1 or 0.3) 'lead to overfitting and notable utility degradation' is made without supporting data. The choice β1=0.5 is a central design decision of AHD. Please provide the ablation study over β1, reporting both safety (jailbreak harmfulness) and utility (e.g., MMLU or another benchmark), or soften the claim to reflect the lack of evidence.

minor comments (6)

[Appendix B.3, Figure 5] The caption of Figure 5 says 'Llama-2', but the surrounding text and Figures 6–7 indicate the figure is for Llama-3. Please correct this typo.
[§2, Algorithm 2] Algorithm 2 is said to apply a hook 'immediately before the output projection,' but the pseudocode operates on a (B,S,D) activation and reshapes it into heads. It would clarify whether the hook is applied to the concatenated head outputs before Wo, and why the 1/(1−dropout_rate) scaling is correct at that point.
[Tables 1 and 2] The tables report 'AHD / original' as slash-separated values but the header only reads 'Harmfulness Rate(%)→' or benchmark names. Please make the two conditions explicit in the header (e.g., 'AHD / Baseline') or use separate columns.
[§3.4] The analysis showing lower cumulative safety-influence scoresfor jailbreak prompts is correlational. A stronger causal test would be to ablate the top-8 heads during jailbreak attacks and show attack success increases, or to compare against a random 8-head ablation. Consider adding such an experiment or tempering the word 'exploit'.
[Appendix B.1] The argument that overlap between the refusal-direction dataset and the AdvBench evaluation prompts does not bias RDSHA is plausible but terse. A brief formal or empirical justification (e.g., re-running the identification on a disjoint set) would increase confidence.
[General] The line 'Code Models' after the author affiliations appears to be a placeholder; please provide the actual repository URL or state that code will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core claims rest on independent causal ablations and external jailbreak benchmarks, not on equations that reduce to their inputs.

full rationale

The paper's RDSHA finding is not circular: Eq. (8) defines a per-head influence score by projection onto the refusal direction, but the central observation that ablating the top-ranked heads raises harmfulness is an independent causal measurement (Figure 1a, Figure 3), and the concentration claim is an empirical property of the measured scores, not a logical consequence of the definition. The AHD training objective (Eq. 9) is a standard weighted cross-entropy over harmful and benign data with per-head dropout; its evaluation uses three external jailbreak attack pipelines (AutoDAN, SI-GCG, Adaptive) and standard utility benchmarks, none of which are constructed from the method's own outputs. The claim that AHD distributes safety is measured with RDSHA, which is a proxy, but the robustness claim is externally validated by Table 1. There is no load-bearing self-citation: the refusal direction is cited to Arditi et al. (2024) and the fine-tuning recipe to Qi et al. (2025), both external to this author group. The most serious weakness—that AHD is compared only to the original model, not to a no-dropout safety-SFT control—is an attribution/confound issue, not a circular reduction, because no equation or fitted value is renamed as a prediction. Accordingly, no circular step meets the quoted-reduction bar.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central empirical claims rest on the refusal-direction construct, the representativeness of small harmful-prompt sets, and the validity of an automated judge. The AHD method introduces hand-set hyperparameters and a training target constructed from the model's own responses. No new physical or architectural entities are posited; 'safety-critical attention heads' is an operational label for a subset of existing heads.

free parameters (3)

beta1 (AHD dropout rate on safety data) = 0.5
Chosen by hand. The footnote says 0.1 or 0.3 cause overfitting and utility degradation, so it was tuned for this setting.
alpha (safety/utility balancing weight in Eq 9) = 0.2
Chosen by hand to weight the safety and utility objectives; no sensitivity analysis is reported.
Layer l* for the refusal direction = not reported here (inherited from Arditi et al.)
The refusal direction is imported from prior work, including its empirically optimal layer selected via downstream validation; this fitted choice affects all RDSHA rankings.

assumptions (6)

standard math Attention head outputs sum linearly to form the attention layer output, so individual heads can be ablated by masking their output projections.
Section 2, Eq 4; this architectural decomposition underlies all head-level interventions.
domain assumption The global refusal direction r captures safety-relevant internal state, and the projection |O_h dot r| indicates a head's contribution to safety.
Section 3.1, Eq 8; imported from Arditi et al. 2024 and is the main measurement premise of RDSHA.
domain assumption Llama-Guard-3-8B provides a valid automated harmfulness assessment.
Section 3.2; no human agreement or calibration is reported.
domain assumption Fifty AdvBench prompts are representative enough to draw general conclusions about safety-head concentration.
Section 3.2; a small sample from a single benchmark.
ad hoc to paper The model's own generated responses to 256 harmful instructions are suitable SFT targets for safety training.
Section 4.2; these targets are assumed to be refusals that should be reinforced, with no filtering or human review described.
ad hoc to paper AHD's effect can be measured by RDSHA after recomputing the refusal direction for the trained model.
Appendix B.1; implicitly assumes the refusal-direction metric remains comparable across different models and fine-tunings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Safety Alignment Should Be Made More Than Just A Few Attention Heads." pith.science (2026). https://pith.science/paper/6XNFRFD2

@misc{pith2026250819697,
  author       = {Pith},
  title        = {Pith review of: Safety Alignment Should Be Made More Than Just A Few Attention Heads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XNFRFD2}},
  note         = {Machine review of arXiv:2508.19697}
}

read the original abstract

Current safety alignment for large language models(LLMs) continues to present vulnerabilities, given that adversarial prompting can effectively bypass their safety measures.Our investigation shows that these safety mechanisms predominantly depend on a limited subset of attention heads: removing or ablating these heads can severely compromise model safety. To identify and evaluate these safety-critical components, we introduce RDSHA, a targeted ablation method that leverages the model's refusal direction to pinpoint attention heads mostly responsible for safety behaviors. Further analysis shows that existing jailbreak attacks exploit this concentration by selectively bypassing or manipulating these critical attention heads. To address this issue, we propose AHD, a novel training strategy designed to promote the distributed encoding of safety-related behaviors across numerous attention heads. Experimental results demonstrate that AHD successfully distributes safety-related capabilities across more attention heads. Moreover, evaluations under several mainstream jailbreak attacks show that models trained with AHD exhibit considerably stronger safety robustness, while maintaining overall functional utility.

Figures

Figures reproduced from arXiv: 2508.19697 by the authors.

**Figure 2.** Heatmap illustrating the frequency of top 50 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 4.** Comparison of cumulative safety influence [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (7 more)

**Figure 5.** Figure 5: Heatmap illustrating the frequency of top 50 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 6.** Figure 6: Heatmap illustrating the frequency of top 50 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 9.** Figure 9: Comparison of cumulative safety influence [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 8.** Figure 8: Comparison of cumulative safety influence [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 11.** Figure 11: Case comparison on Llama-2 under no ablation and after ablating 40 attention heads. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: Case comparison on Qwen under no ablation and after ablating 20 attention heads. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 13.** Figure 13: Case comparison on Qwen-2 under no ablation and after ablating 40 attention heads. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 17 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. https://openreview.net/forum?id=hXA8wqRdyV Jailbreaking leading safety-aligned LLM s with simple adaptive attacks . In The Thirteenth International Conference on Learning Representations

work page 2025
[4]

Anthropic. 2023. Introducing Claude . https://www.anthropic.com/index/introducing-claude

work page 2023
[5]

Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. https://openreview.net/forum?id=EqF16oDVFf Refusal in language models is mediated by a single direction . In ICML 2024 Workshop on Mechanistic Interpretability

work page 2024
[6]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[7]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

arXiv 2022
[8]

Leonard Bereska and Efstratios Gavves. 2024. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082

arXiv 2024

Show all 47 references

[9]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020
[10]

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419

2023 arXiv
[11]

Brian Christian. 2020. The alignment problem: Machine learning and human values. WW Norton & Company

2020
[12]

Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2025. https://openreview.net/forum?id=obYVdcMMIT OR -bench: An over-refusal benchmark for large language models

2025
[13]

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

2024 arXiv
[14]

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, and 1 others. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint ar...

2022 arXiv
[15]

Gemini Team . 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

2023 arXiv
[16]

Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, and 1 others. 2023. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852

2023 arXiv
[17]

Xiaojun Jia, Yihao Huang, Yang Liu, Peng Yan Tan, Weng Kuan Yau, Mun-Thye Mak, Xin Ming Sim, Wee Siong Ng, See Kiong Ng, Hanqing Liu, and 1 others. 2024. Global challenge for safe and secure llms track 1. arXiv preprint arXiv:2411.14502

2024 arXiv
[18]

Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. arXiv preprint arXiv:2103.14659

2021 arXiv
[19]

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. In Forty-first International Conference on Machine Learning

2024
[20]

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. https://arxiv.org/abs/1811.07871 Scalable agent alignment via reward modeling: a research direction . Preprint, arXiv:1811.07871

2018 arXiv
[21]

Jan Leike and Ilya Sutskever. 2023. Introducing Superalignment . https://openai.com/blog/introducing-superalignment

2023
[22]

Hanqing Liu, Lifeng Zhou, and Huanqian Yan. 2024 a . https://arxiv.org/abs/2410.15645 Boosting jailbreak transferability for large language models . Preprint, arXiv:2410.15645

2024 arXiv
[23]

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024 b . https://openreview.net/forum?id=7Jwpw4qKkb Autodan: Generating stealthy jailbreak prompts on aligned large language models . In The Twelfth International Conference on Learning Representations

2024
[24]

AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

2024 arXiv
[25]

Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37:61065--61105

2024
[26]

Honglei Miao, Fan Ma, Ruijie Quan, Kun Zhan, and Yi Yang. 2025. Autonomous llm-enhanced adversarial attack for text-to-motion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6144--6152

2025
[27]

do anything now

Seyedsina Nabavirazavi, Samira Zad, and Sundararaja Sitharama Iyengar. 2025. Evaluating the universality of “do anything now” jailbreak prompts on large language models: Content warning: This paper contains unfiltered and harmful examples. In 2025 IEEE 15th Annual Computing an...

2025
[28]

OpenAI. 2022. Introducing ChatGPT . https://openai.com/blog/chatgpt

2022
[29]

OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

2023 arXiv
[30]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Sys...

2022
[31]

Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. https://openreview.net/forum?id=6Mxhg9PtDE Safety alignment should be made more than just a few tokens deep . In The Thirteenth International Conference o...

2025
[32]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf Direct preference optimization: Your language model is ...

2023
[33]

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

2023
[34]

Adly Templeton. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic

2024
[35]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[36]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

2023 arXiv
[37]

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079--80110

2023
[38]

Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Forty-first International Conference on Machine Learning

2024
[39]

Suhuang Wu, Huimin Wang, Yutian Zhao, Xian Wu, Yefeng Zheng, Wei Li, Hui Li, and Rongrong Ji. 2025. Monte carlo tree search based prompt autogeneration for jailbreak attacks against llms. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1...

2025
[40]

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. https://doi.org/10.18653/v1/2024.acl-long.303 S afe D ecoding: Defending against jailbreak attacks via safety-aware decoding . In Proceedings of the 62nd Annual Meeting of the As...

2024 doi
[41]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

2024 arXiv
[42]

Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1--38

2024
[43]

Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024 a . On prompt-driven safeguarding for large language models. In Forty-first International Conference on Machine Learning

2024
[44]

Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Bo Tang, Feiyu Xiong, and Zhiyu Li. 2024 b . Attention heads of large language models: A survey. arXiv preprint arXiv:2409.03752

2024 arXiv
[45]

Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. 2025. https://openreview.net/forum?id=h0Ak8A5yqw On the role of attention heads in large language model safety . In The Thirteenth International Conference on Lear...

2025
[46]

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023 a . Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

2023 arXiv
[47]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023 b . https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . Preprint, arXiv:2307.15043

2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. https://openreview.net/forum?id=hXA8wqRdyV Jailbreaking leading safety-aligned LLM s with simple adaptive attacks . In The Thirteenth International Conference on Learning Representations

work page 2025

[4] [4]

Anthropic. 2023. Introducing Claude . https://www.anthropic.com/index/introducing-claude

work page 2023

[5] [5]

Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. https://openreview.net/forum?id=EqF16oDVFf Refusal in language models is mediated by a single direction . In ICML 2024 Workshop on Mechanistic Interpretability

work page 2024

[6] [6]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[7] [7]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

arXiv 2022

[8] [8]

Leonard Bereska and Efstratios Gavves. 2024. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082

arXiv 2024