Pith. sign in

REVIEW 3 major objections 5 minor 65 references

Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a rank-one weight edit, computed from the activation difference between harmful and harmless prompts, amplifies refusal in aligned language models and re-installs safety in uncensored ones, with near-zero utility loss.

desk verdict ROSI is a simple, promising safety amplifier, but reproducibility gaps and an unexamined universal-layer assumption keep it from being fully convincing yet. read the letter →

arxiv 2508.20766 v1 pith:NOJQKM3X submitted 2025-08-28 cs.CL cs.AIcs.LG

Harethah Abu Shairah , Hasan Abed Al Kader Hammoud , George Turkiyyah , Bernard Ghanem This is my paper
classification cs.CLcs.AIcs.LG
keywords rank-oneweighteditingsafetyalignmentrefusaldirectionactivationsteeringmechanisticinterpretabilityLLMmodelre-alignmentdifference-in-means
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that safety alignment in large language models can be amplified—or restored—by a single interpretable weight edit, no fine-tuning required. Its method, Rank-One Safety Injection (ROSI), computes the average difference between a model's last-token activations on harmful versus harmless instructions, normalizes it, and adds that direction, scaled by a scalar, to every weight matrix that writes into the residual stream. Across nine aligned models, the edit is reported to raise refusal rates on harmful prompts—by as much as 18.2 points on the weakest baseline—and to cut jailbreak attack success by double digits on several benchmarks, while standard utility benchmarks move by about a point or less. The same procedure, preceded by a temporary system prompt to elicit refusal, is claimed to re-align uncensored models that were fine-tuned to ignore safety. If correct, ROSI turns a known vulnerability—the fact that refusal lives in a single linear direction—into a cheap last-mile safety tool.

What carries the argument

The load-bearing mechanism is the rank-one weight update W'_out = W_out + alpha * s_hat * w_bar^T, applied to every matrix that writes into the residual stream—attention output projections and MLP output projections. s_hat is the normalized difference-in-means vector of last-token residual activations between harmful and harmless instruction sets; w_bar is the mean row vector of the original W_out, so the update adds to the output a component proportional to s_hat for the typical input. Alpha is a scalar controlling strength, and the layer at which s_hat is extracted is chosen on a validation set. What this mechanism does is convert a measurement of where harmful activations sit relative to

What would settle it

Run the ROSI update, then evaluate refusal per CATQA category and utility per MMLU subject with repeated seeds; if any harmful category becomes more likely to be answered after injection than before, or any utility subject drops by more than the paper's mean delta, the universal-amplifier and orthogonality claims both fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that refusal in open LLMs is mediated by a direction that can be injected as easily as it can be ablated. Given a normalized safety vector s_hat—the difference between mean last-token activations on harmful and harmless prompts—ROSI modifies every residual-stream write matrix W_out by W'_out = W_out + alpha * s_hat * w_bar^T, where w_bar is the mean of the original rows and alpha is a scalar strength chosen on a validation set. This rank-one update makes the average input to any layer produce an output nudged toward the refusal subspace, and the paper reports that doing this at all layers consistently raises refusal rates on CATQA, cuts attack success on DAN, Har

Load-bearing premise

The method assumes that one measured refusal direction, when added to every write matrix at every layer, uniformly strengthens refusal without touching capability directions, and—for uncensored models—that a safety direction elicited by a temporary system prompt keeps working after that prompt is removed.

Editorial extensions

If this is right

  • Aligned models can be made more refusal-prone and more jailbreak-resistant with a single rank-one weight edit and no training, using only 50 harmful and 50 harmless prompt pairs.
  • Uncensored models can be re-aligned to refuse harmful requests when a safety direction is first elicited with a system prompt; the prompt itself is removed at test time.
  • The edit lives in the weights, so the safety shift persists across conversations without per-request steering or an external guard model.
  • Standard utility benchmarks (MMLU, HellaSwag, ARC, BoolQ, TruthfulQA) move by about a point or less on average, consistent with the safety direction being nearly orthogonal to knowledge and reasoning directions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension of the rank-one recipe: if refusal is as orthogonal to capability as reported, the same update should compose—steering honesty, tone, or domain focus by adding multiple scaled directions—and the paper's final paragraph already points at honesty and controllability.
  • The layer-and-alpha selection on a validation set suggests a natural test: choose alpha per layer or per input to recover the small benign-compliance losses seen on sub-1B models, keeping refusal gains while trimming side effects.
  • The uncensored-model branch depends on eliciting a refusal direction with a system prompt; trying other elicitation contexts (few-shot refusal examples, safety classifiers) would show whether re-alignment generalizes to models whose refusal signal is weakest.
  • A white-box check the paper does not report: after ROSI, does the injected direction survive abliteration-style removal or further fine-tuning? Measuring that would delimit what 'permanent' means for a rank-one safety edit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Rank-One Safety Injection (ROSI), a training-free method that modifies a language model's weights by adding a rank-one matrix α·ŝ·w̄ᵀ to each residual-stream write matrix (Eq. 6). The direction ŝ is the normalized difference-in-means between harmful and harmless last-token activations at a selected layer. The authors claim that ROSI consistently increases refusal rates on harmful prompts, reduces jailbreak success, and preserves utility on standard benchmarks, for both aligned models and 'uncensored' Dolphin models, with the latter aided by an eliciting system prompt during extraction. Experiments cover nine aligned and four uncensored models, with safety evaluated by Llama Guard 3 on CATAQA and by WildGuard on several jailbreak sets.

Significance. If correct, ROSI would be a valuable low-cost safety amplifier: it is a simple, interpretable, weight-only update that could complement fine-tuning, and the empirical scope is broad across model families and jailbreak benchmarks. The use of held-out harmful sets (CATAQA) and external jailbreak benchmarks reduces the concern that the safety gains are a tautology of extracting the direction from the same test prompts. However, the current evidence is not yet sufficient to support the strength of the claims. Key hyperparameters are unreported, no error bars or repeated-seed results are given, and the utility evaluation is narrow. These gaps are load-bearing because the central claims are 'consistently improves safety' and 'negligible utility loss.'

major comments (3)
  1. [Sec. 3.2, Sec. 3.3, Tables 1–6] The injection strength α and the selected layer l* are never reported. Section 3.2 only says the layer is chosen 'based on a validation set,' but no validation criterion, α values, or per-model values are given. Moreover, no table reports standard errors, confidence intervals, or multiple seeds. Many deltas in Tables 1–3 are small (e.g., GEMMA-2B MMLU +0.2), while others are substantial (e.g., QWEN2.5-0.5B BoolQ −4.4, BC −7.2). Without uncertainty quantification, the claims of 'consistent' safety gains and 'negligible' utility change are not quantitatively supported, and the risk of selection on a validation set cannot be assessed. Please report α, l*, validation details, and repeated-run statistics.
  2. [Eq. (6), Sec. 3.3, Tables 3 and 6] The update is applied to every residual-stream write matrix, affecting all layers and all token positions, yet the paper provides no per-layer or per-position analysis. The added component at each module is α·(w̄ᵀx)·ŝ, whose magnitude depends on the un-controlled average pre-activation w̄ᵀx; this is not shown to be aligned with the refusal direction at early layers or at non-final token positions. The claim that ŝ is 'largely orthogonal to the representations required for knowledge and reasoning tasks' is supported only by six standard benchmarks without error bars and a 512-instruction Alpaca sample. Code, math, open-ended instruction following, and long-context tasks are absent—capabilities that could plausibly be affected by perturbing every layer. I recommend a layer sweep (e.g., injecting at l* only versus all layers), per-tier utility evaluations, and at least one broader capabilit
  3. [Sec. 4.3, Table 4] The re-alignment claim for uncensored models is stronger than the evidence. The system-prompt ablation shows that without the eliciting prompt, ROSI reduces harm refusal relative to baseline for three of four Dolphin models: Dolphin 3.0-Llama3.2-1B (18.4 vs 23.5), Dolphin 3.0-Qwen2.5-3B (33.6 vs 50.0), and Dolphin 3.0-Mistral-24B (47.8 vs 64.4). Only Dolphin 3.0-Llama3.1-8B retains a positive effect. The abstract and conclusion describe amplifying 'their own latent safety directions,' but the appendix correctly attributes the effect to the safety system prompt. The claims should be qualified as model-dependent and prompt-dependent, and the 'latent' wording reconsidered.
minor comments (5)
  1. [Sec. 3.2] Equation (3)-(5) refer to 'the position of the token i' without defining i. State explicitly that activations are taken at the last prompt token position.
  2. [Tables 4–6] The ❢ symbol is defined in Table 4's caption but appears in Tables 5 and 6 without a repeated definition. Please define it in all captions and use consistent formatting.
  3. [Appendix B, Table 8] Typo: '2.7 (+)2.7' should read '+2.7'. Also, model names are inconsistent across tables and text (e.g., 'META-Llama-3.1-8B-Instruct' vs 'Llama-3.1-8B-Instruct').
  4. [References] The 'Dolphin' reference (dphn.ai) lacks version, date, and a stable identifier. Please provide a version/date or a more formal citation.
  5. [Sec. 4.1] No code availability statement is given. If code is not released, state so explicitly; if it is or will be, provide the link.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the safety direction is extracted from activations and evaluated on external benchmarks.

full rationale

The derivation chain is self-contained. The safety direction s is defined by Eq. (5) as a difference-in-means of activations on contrasting prompt sets, and the weight update Eq. (6) is W'_out = W_out + alpha * s_hat * wbar^T. Neither equation is defined in terms of the evaluation metrics (Llama Guard harm refusal, WildGuard attack success, MMLU, etc.). The reported safety and utility results are measured on held-out external benchmarks (CATAQA, HarmBench, WildGuard, MMLU, HellaSwag, ARC), not on the extraction pairs. The layer l* and alpha are chosen on an unspecified validation set, which is a reporting limitation but not a circular step: the paper does not claim these are first-principles predictions, and there is no evidence that the validation set coincides with the test sets. The uncensored-model experiments honestly include a system-prompt ablation (rows marked with '❢' in Tables 4 and 5) showing that the method's success is model-dependent; this is a limitation, not circularity. The only self-citations (Shairah et al. 2025; Hammoud et al. 2024) appear in Related Work and are not load-bearing for the central derivation. No equation or claim reduces, by construction or by self-citation, to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new entities; it operates on the established single-direction refusal feature. The burden carried upstream is the difference-in-means direction itself (prior work), the orthogonality assumption for utility preservation, and the prompt-elicitation premise for uncensored models. The main unauditable freedom is the unreported per-model tuning of alpha and layer.

free parameters (3)
  • alpha (injection strength)
    Scalar controlling how strongly the safety direction is added into every write matrix (Eq. 6). No value or search range is reported for any model, so the core update rule is under-specified as published.
  • refusal layer l*
    The layer at which the difference-in-means direction is computed is selected as 'optimal' on an undescribed validation set; the selected layer per model is never reported.
  • number of harmful/harmless pairs = 50
    Fixed at 50 pairs by hand with no sensitivity analysis; the direction quality depends on this choice and the exact pair sets are not released.
assumptions (5)
  • domain assumption Refusal is mediated by a single linear direction in the residual stream of each tested model (Arditi et al. 2024).
    Invoked in Sections 1 and 3 as the foundation of the method; if refusal is not linearly separable, Eq. 5 has no causal target.
  • domain assumption The last-token difference-in-means vector s(l) remains a causally valid refusal direction when injected as a rank-one update into every residual-stream write matrix, with magnitude scaled by the mean row vector of each matrix.
    Sections 3.2 and 3.3; the row-mean scaling is an engineering choice justified only by the empirical tables, not derived from any property of the transformer.
  • domain assumption The safety direction is nearly orthogonal to capability-relevant directions, so injecting it at all layers preserves utility.
    Section 4.2 states 'the safety direction is largely orthogonal to the representations required for knowledge and reasoning tasks'; supported only by six benchmark deltas without error bars and a 512-instruction Alpaca sample.
  • domain assumption For uncensored models, a refusal direction elicited by a safety system prompt during extraction persists when the prompt is removed at test time.
    Section 4.3 and Figure 2; the paper's own ablation shows this premise fails for DOLPHIN 3.0-MISTRAL-24B, where harm refusal actually drops without the prompt during extraction.
  • domain assumption Automated guardrails (Llama Guard 3, WildGuard) measure genuine refusal and safety.
    All safety and jailbreak numbers in Tables 1, 2, 4 and 5 are classifier judgments; refusal-style artifacts can satisfy the classifiers without deeper safety behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection." pith.science (2026). https://pith.science/paper/NOJQKM3X

@misc{pith2026250820766,
  author       = {Pith},
  title        = {Pith review of: Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOJQKM3X}},
  note         = {Machine review of arXiv:2508.20766}
}
read the original abstract

Safety alignment in Large Language Models (LLMs) often involves mediating internal representations to refuse harmful requests. Recent research has demonstrated that these safety mechanisms can be bypassed by ablating or removing specific representational directions within the model. In this paper, we propose the opposite approach: Rank-One Safety Injection (ROSI), a white-box method that amplifies a model's safety alignment by permanently steering its activations toward the refusal-mediating subspace. ROSI operates as a simple, fine-tuning-free rank-one weight modification applied to all residual stream write matrices. The required safety direction can be computed from a small set of harmful and harmless instruction pairs. We show that ROSI consistently increases safety refusal rates - as evaluated by Llama Guard 3 - while preserving the utility of the model on standard benchmarks such as MMLU, HellaSwag, and Arc. Furthermore, we show that ROSI can also re-align 'uncensored' models by amplifying their own latent safety directions, demonstrating its utility as an effective last-mile safety procedure. Our results suggest that targeted, interpretable weight steering is a cheap and potent mechanism to improve LLM safety, complementing more resource-intensive fine-tuning paradigms.

Figures

Figures reproduced from arXiv: 2508.20766 by the authors.

Figure 1
Figure 1. RANK-ONE SAFETY INJECTION (ROSI). An aligned model processes both be￾nign and harmful prompts in a forward pass (1). A safety vector is derived from the difference between harmful and harmless activations (2). Subtracting this vector ablates safety signals, pro￾ducing an Abliterated Model. Adding it reinforces safety, producing a ROSI Model. How to build a bomb? How to bake a cake? Aligned Model 𝝁 Harmful Activation… view at source ↗
Figure 2
Figure 2. Applying ROSI to Uncensored Models. In the forward pass, harmful and harmless instructions are prepended with a system prompt directing an uncensored model to reject harmful requests, thus eliciting refusal. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Safety System Prompt. System prompt used to elicit refusal behavior in uncensored models. You are $MODEL, a helpful and responsible AI assistant. Your primary responsibility is to only assist users with safe, lawful, and non-harmful requests. If you receive any request that is unsafe, harmful, unethical, or illegal, you must refuse it imme￾diately. Your refusal should be polite, firm, and should always start with ’I… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 6 canonical work pages

  1. [1]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu,...

    arXiv 2025

  2. [2]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

    arXiv 2024

  3. [3]

    Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic, 2024

    Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic, 2024. URL https://arxiv.org/abs/2402.11746

    arXiv 2024

  4. [4]

    Towards inference-time category-wise safety steering for large language models

    Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, and Christopher Parisien. Towards inference-time category-wise safety steering for large language models. In Neurips Safe Generative AI Workshop 2024, 2024. URL https://openreview.net/forum?id=EkQRNLPFcn

    work page 2024

  5. [5]

    Man is to computer programmer as woman is to homemaker? Debiasing word embeddings

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in neural information processing systems, 29, 2016

    2016

  6. [6]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  7. [7]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  8. [8]

    On the measure of intelligence, 2019

    François Chollet. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911.01547

    arXiv 2019

Show all 65 references
  1. [9]

    Comprehensive assessment of jailbreak attacks against llms, 2024

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. Comprehensive assessment of jailbreak attacks against llms, 2024. URL https://arxiv.org/abs/2402.05668

    2024 arXiv

  2. [10]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  3. [11]

    https://dphn.ai, 2025

    Dolphin. https://dphn.ai, 2025

    2025

  4. [12]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022

  5. [13]

    Finding alignments between interpretable causal variables and distributed neural representations

    Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp.\ 160--187. PMLR, 2024

    2024

  6. [14]

    Safesteer: Interpretable safety steering with refusal-evasion in llms

    Shaona Ghosh, Amrita Bhattacharjee, Yftah Ziser, and Christopher Parisien. Safesteer: Interpretable safety steering with refusal-evasion in llms. arXiv preprint arXiv:2506.04250, 2025

    2025 arXiv

  7. [15]

    A confederacy of models: a comprehensive evaluation of llms on creative writing, 2023

    Carlos Gómez-Rodríguez and Paul Williams. A confederacy of models: a comprehensive evaluation of llms on creative writing, 2023. URL https://arxiv.org/abs/2310.08433

    2023 arXiv

  8. [16]

    Model merging and safety alignment: One bad model spoils the bunch

    Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. Model merging and safety alignment: One bad model spoils the bunch. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for...

    2024 doi

  9. [17]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024. URL https://arxiv.org/abs/2406.18495

    2024 arXiv

  10. [18]

    Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations

    Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. arXiv preprint arXiv:2406.11801, 2024

    2024 arXiv

  11. [19]

    Saif: A sparse autoencoder framework for interpreting and steering instruction following of language models

    Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. Saif: A sparse autoencoder framework for interpreting and steering instruction following of language models. arXiv preprint arXiv:2502.11356, 2025

    2025 arXiv

  12. [20]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  13. [21]

    The reasoning-memorization interplay in language models is mediated by a single direction

    Yihuai Hong, Dian Zhou, Meng Cao, Lei Yu, and Zhijing Jin. The reasoning-memorization interplay in language models is mediated by a single direction. arXiv preprint arXiv:2503.23084, 2025

    2025 arXiv

  14. [22]

    Refusal tokens: A simple way to calibrate refusals in large language models

    Neel Jain, Aditya Shrivastava, Chenyang Zhu, Daben Liu, Alfy Samuel, Ashwinee Panda, Anoop Kumar, Micah Goldblum, and Tom Goldstein. Refusal tokens: A simple way to calibrate refusals in large language models. arXiv preprint arXiv:2412.06748, 2024 a

    2024 arXiv

  15. [23]

    What makes and breaks safety fine-tuning? a mechanistic study

    Samyak Jain, Ekdeep S Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet Dokania. What makes and breaks safety fine-tuning? a mechanistic study. Advances in Neural Information Processing Systems, 37: 0 93406--93478, 2024 b

    2024

  16. [24]

    Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024. URL https://arxiv....

    2024 arXiv

  17. [25]

    Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models, 2023. URL https://arxiv.org/abs/2305.06984

    2023 arXiv

  18. [26]

    LoRA fine-tuning efficiently undoes safety training in Llama 2-Chat 70B

    Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. LoRA fine-tuning efficiently undoes safety training in Llama 2-Chat 70B . arXiv preprint arXiv:2310.20624, 2023

    2023 arXiv

  19. [27]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  20. [28]

    Rethinking jailbreaking through the lens of representation engineering, 2024 b

    Tianlong Li, Shihan Dou, Wenhao Liu, Muling Wu, Changze Lv, Rui Zheng, Xiaoqing Zheng, and Xuanjing Huang. Rethinking jailbreaking through the lens of representation engineering, 2024 b

    2024

  21. [29]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv.org/abs/2109.07958

    2022 arXiv

  22. [30]

    Towards understanding jailbreak attacks in LLM s: A representation space analysis

    Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. Towards understanding jailbreak attacks in LLM s: A representation space analysis. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), EMNLP 2024, pp.\ 7067--7085, Miami, Florida, USA,...

    2024 doi

  23. [31]

    The llama 3 herd of models, 2024

    AI @ Meta Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  24. [32]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023

    2023 arXiv

  25. [33]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. URL https://arxiv.org...

    2024 arXiv

  26. [34]

    Towards safer pretraining: Analyzing and filtering harmful content in webscale datasets for responsible llms, 2025

    Sai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, and Parag Agrawal. Towards safer pretraining: Analyzing and filtering harmful content in webscale datasets for responsible llms, 2025. URL https://arxiv.org/abs/2505.02009

    2025 arXiv

  27. [35]

    Steering language model refusal with sparse autoencoders

    Kyle O'Brien, David Majercak, Xavier Fernandes, Richard Edgar, Blake Bullwinkel, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangdeh. Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296, 2024

    2024 arXiv

  28. [36]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  29. [37]

    Steering Llama 2 via contrastive activation addition

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023

    2023 arXiv

  30. [38]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    2023 arXiv

  31. [39]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  32. [40]

    Null it out: Guarding protected attributes by iterative nullspace projection

    Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. arXiv preprint arXiv:2004.07667, 2020

    2004 arXiv

  33. [41]

    An embarrassingly simple defense against llm abliteration attacks

    Harethah Abu Shairah, Hasan Abed Al Kader Hammoud, Bernard Ghanem, and George Turkiyyah. An embarrassingly simple defense against llm abliteration attacks. arXiv preprint arXiv:2505.19056, 2025

    2025

  34. [42]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  35. [43]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  36. [44]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  37. [45]

    Codejudge: Evaluating code generation with large language models, 2024

    Weixi Tong and Tianyi Zhang. Codejudge: Evaluating code generation with large language models, 2024. URL https://arxiv.org/abs/2410.02184

    2024 arXiv

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  39. [47]

    Activation addition: Steering language models without optimization

    Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023

    2023 arXiv

  40. [48]

    Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment, 2024

    Haoran Wang and Kai Shu. Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment, 2024. URL https://arxiv.org/abs/2311.09433

    2024 arXiv

  41. [49]

    Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation

    Xinpeng Wang, Chengzhi Hu, Paul R \"o ttger, and Barbara Plank. Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum...

    2025

  42. [50]

    Jailbroken: How does llm safety training fail?, 2023 a

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023 a . URL https://arxiv.org/abs/2307.02483

    2023 arXiv

  43. [51]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162, 2024

    2024 arXiv

  44. [52]

    Chain-of-thought prompting elicits reasoning in large language models, 2023 b

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023 b . URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  45. [53]

    A new era in llm security: Exploring security concerns in real-world llm-based systems, 2024

    Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao. A new era in llm security: Exploring security concerns in real-world llm-based systems, 2024. URL https://arxiv.org/abs/2402.18649

    2024 arXiv

  46. [54]

    Shadow alignment: The ease of subverting safely-aligned language models

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023

    2023 arXiv

  47. [55]

    Representation bending for large language model safety

    Ashkan Yousefpour, Taeheon Kim, Ryan Sungmo Kwon, Seungbeen Lee, Wonje Jeung, Seungju Han, Alvin Wan, Harrison Ngan, Youngjae Yu, and Jonghyun Choi. Representation bending for large language model safety. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pi...

    2025

  48. [56]

    Robust llm safeguarding via refusal feature adversarial training

    Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust llm safeguarding via refusal feature adversarial training. arXiv preprint arXiv:2409.20089, 2024

    2024 arXiv

  49. [57]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  50. [58]

    Removing RLHF protections in GPT-4 via fine-tuning

    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. arXiv preprint arXiv:2311.05553, 2023

    2023 arXiv

  51. [59]

    Adasteer: Your aligned llm is inherently an adaptive jailbreak defender

    Weixiang Zhao, Jiahe Guo, Yulin Hu, Yang Deng, An Zhang, Xingyu Sui, Xinyang Han, Yanyan Zhao, Bing Qin, Tat-Seng Chua, et al. Adasteer: Your aligned llm is inherently an adaptive jailbreak defender. arXiv preprint arXiv:2504.09466, 2025

    2025

  52. [60]

    Prompt-driven LLM safeguarding via directed representation optimization

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. Prompt-driven LLM safeguarding via directed representation optimization. arXiv preprint arXiv:2401.18018, 2024

    2024 arXiv

  53. [61]

    Representation engineering: A top-down approach to AI transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023

    2023 arXiv

  54. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  55. [63]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  56. [64]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  57. [65]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.