Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Instruction fine-tuning silently erodes a language model's refusal of harmful requests. The paper attributes this to drift in one hidden-state direction and shows a projection-constraint loss can hold it, restoring safety with no task loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:05 UTC pith:SYM4AE7D

load-bearing objection A useful, interpretable safety-tuning method whose practical claims are plausible, but the causal drift story is confounded and the paper needs code and a control before the mechanism is taken seriously. the 4 major comments →

arxiv 2509.06795 v1 pith:SYM4AE7D submitted 2025-09-08 cs.CL

Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

Yanrui Du , Fenglei Fan , Sendong Zhao , Jiawei Cao , Qika Lin , Kai He , Ting Liu , Bing Qin
show 1 more author
Mengling Feng
This is my paper
classification cs.CL
keywords refusal directioninstruction fine-tuningsafety alignmentprojection constraintdrift analysisLLM safetyinterpretabilitywarm-up strategy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Instruction fine-tuning — the standard way to adapt a large language model to a specialized task — can quietly undo the model's ability to refuse harmful requests. This paper claims the mechanism is drift: the single hidden-state direction that governs refusal behavior rotates away from its original orientation during training, most sharply in the first epoch and most severely in the deepest layers. To test that claim, the authors add a projection-constraint loss that penalizes any change in how far each sample's hidden state extends along the original refusal direction. Across three model families (LLaMA2, LLaMA3, Qwen2), two scenarios (benign and adversarial instruction tuning), and several datasets, the constraint restores refusal behavior — average attack success drops by tens of percentage points — while leaving task accuracy essentially unchanged. Adding a warm-up phase (strong constraint early, none later) and a thousand safety-oriented data points turns the fix into a method that beats existing defenses on both safety and task performance.

Core claim

The paper's central claim is that the refusal direction — the normalized difference-in-means vector separating hidden-state activations on malicious versus benign instructions — drifts during instruction fine-tuning, and that this drift is a key driver of the safety collapse observed after IFT. The authors measure the drift as the cosine angle between the evolving r-direction at each checkpoint and its initial value, and observe it across LLaMA2, LLaMA3, and Qwen2, concentrated in deeper layers and in the first training epoch. To establish that the drift causes the harm rather than merely accompanying it, they introduce ProCon, an added loss LProCon = Σ ||proj_t − proj_0||² that penalizes th

What carries the argument

The load-bearing object is the refusal direction (r-direction): at each layer, the normalized difference between the mean final-token hidden-state activation over malicious prompts and the mean over benign prompts, a vector whose amplification induces refusals and whose ablation induces compliance. The method rides on a projection-constraint loss (ProCon) that computes, for every training token and layer, the projection magnitude z = r^⊤x of the hidden state onto the original r-direction and penalizes its squared deviation from the value recorded before training: LProCon = E_s[Σ_l Σ_j ||z_t^(l) − z_0^(l)||²], added to cross-entropy as Loverall = LCE + α·LProCon. The argument is carried by tw

Load-bearing premise

The refusal direction measured once, before training, on a fixed set of benign and malicious prompts stays the correct thing to hold fixed for the entire fine-tuning run — the paper never checks whether the direction's meaning changes as the model adapts, nor whether holding any fixed direction would work just as well.

What would settle it

Train the same models with the identical projection-constraint schedule but anchor on a random fixed direction (or a task-relevant direction such as the difference between correct and incorrect reasoning traces) instead of the measured refusal direction. If attack success rates fall as much as with the r-direction, the refusal direction's identity is not load-bearing and the drift-cause claim is falsified. Conversely, if a model tuned with no constraint shows large r-direction drift but no safety loss, the drift is not sufficient for the risk.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Monitoring cosθ of the r-direction at checkpoints gives a cheap, training-time proxy for whether IFT is eroding safety, before any attack evaluation is run.
  • ProCon operates at the user-tuning stage and is compatible with data-stage defenses: combining it with 1k safety-oriented samples cut average attack success from ~61% to ~13% on LLaMA2, ~71% to ~9% on LLaMA3, and ~69% to ~18% on Qwen2 under benign IFT.
  • Because the constraint targets a hidden representation rather than specific weights or data, it transfers across model families (LLaMA2, LLaMA3, Qwen2) and across benign and adversarial IFT scenarios without retuning the mechanism.
  • The warm-up design implies that the first epoch is where safety is won or lost: concentrating constraint there recovers most of the safety benefit at a fraction of the training-time cost of full-run constraint.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper never compares ProCon against constraining an arbitrary fixed direction of the same norm; if a random or task-related anchor preserved safety equally well, the specific identity of the r-direction would be doing no work and the drift-cause story would reduce to a generic regularizer. Running that control is the clean test of whether the mechanism is what the authors claim.
  • The correlational evidence (constraint reduces drift and reduces risk) does not rule out drift being a symptom of a deeper cause, such as the optimization geometry of the loss landscape; a direct causal test would be to steer hidden states back along the r-direction during IFT without any projection loss and observe whether safety is restored.
  • The finding that Qwen2 needs ~16 warm-up epochs versus ~4 for LLaMA suggests a testable hypothesis the authors state but cannot verify: the anchor's reliability tracks the depth of the model's RLHF alignment. If true, safety budgets for custom fine-tuning should be set per-model, not per-method.
  • If the projection anchor is valid, drift monitoring generalizes beyond safety: other behavioral directions (truthfulness, sycophancy, political bias) identified by the same difference-in-means technique could be anchored during fine-tuning with the same loss, making the method a general tool for preserving documented behaviors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ProCon, a projection-constrained loss that regularizes the projection of hidden states onto an initially identified refusal direction during instruction fine-tuning, with a warm-up variant and an expanded data distribution to strengthen the constraint. The authors report that ProCon reduces harmfulness scores and attack success rates across LLaMA2-7B, LLaMA3-8B, and Qwen2-7B under both benign and attack IFT scenarios, while preserving task accuracy. The paper interprets these results as evidence that refusal-direction drift is a key cause of IFT-induced safety degradation, and that anchoring the refusal direction mitigates this risk.

Significance. If the causal interpretation is correct, the paper makes a valuable contribution by connecting an interpretability mechanism (the refusal direction) to a practical, training-time defense that preserves task performance. The empirical scope is broad: three models, two fine-tuning scenarios, multiple red-team and jailbreak attack sets, and a second dataset (GSM8K). The paper is also candid in places, explicitly calling the drift-causality link a 'guess' in Sec. III-C1, and it includes useful negative/sensitivity analyses (e.g., Table VI on post-warm-up constraints). However, the central mechanistic claim is not yet supported because the proposed constraint is a generic representation regularizer; no control constrains an arbitrary direction. The claim of consistent superiority is further weakened by single-run results without released code or data. Strengths: the method is simple and reproducible in principle, and the paper reports multiple attack metrics rather than a single safety score.

major comments (4)
  1. [§III-C1, §III-C2, §V-A] The central causal claim—that r-direction drift is a key driver of IFT safety degradation—is confounded. The ProCon loss LProCon ties every layer's hidden-state projection to its initial value along the r-direction; this is a strong representation-space regularizer, and with LoRA the remaining d-1 dimensions are still free. The observed drift reduction in Fig. 3 is partly built into the constraint, so it does not independently confirm that drift causes safety loss. The paper provides no control constraining an arbitrary fixed direction, a task-relevant direction, or a random direction under the same α, warm-up, and data. Please add such controls. If constraining a random direction gives comparable safety benefits, the specificity of the r-direction is not established. The manuscript should then temper the abstract's 'identify as one of the causes' and conclusion's 'main causes' language;
  2. [§IV-B, Tables I–III] All reported results are single runs without error bars, confidence intervals, or significance tests, and no code or data are released. Several headline comparisons are small in magnitude (e.g., LLaMA3 ProCon_wu_safe vs SWAT_safe: 9.15% vs 11.03% ASR, Table II), so the claim that the method 'consistently delivers superior performance' is not statistically supported. Please report multiple seeds, provide code/data artifacts, and accompany headline numbers with variance estimates or a significance test.
  3. [§III-C3] The Fisher-information motivation for broadening the data distribution is not a derivation. The 'effective sensitivity factor κ(z)' is introduced without definition; the expression I_r(θ) = E_s[κ(z)·(r^T ∇_θ m_θ(s))^2] does not follow from the ProCon loss; and the claim that broader coverage 'keeps κ(z) active' is unsupported. This passage overstates the theoretical grounding for the safety-oriented data component. It should be rewritten as an intuition/heuristic or supported with a proper derivation.
  4. [§V-D, §IV-A2] The method has several per-model hyperparameters that appear to be selected on the test set: α for the warm-up stage, α after warm-up, the number of warm-up epochs, and the number of safety-oriented samples. Warm-up epochs vary widely across models (4 for LLaMA2/3, 16 for Qwen2, Fig. 5), and the text acknowledges the number is a per-model hyperparameter. Without a principled selection rule or at least a validation-based tuning protocol, the reported 'consistent' improvements may reflect post hoc selection. Please specify the selection criterion and report results under a fixed protocol or with sensitivity sweeps.
minor comments (4)
  1. [§III-B] The metric is called a 'drift angle' but reported as cosθ. Since higher cosθ means less drift, consider reporting the angle in degrees or explicitly stating that lower cosθ corresponds to larger angular drift to avoid reader confusion.
  2. [Fig. 3, Eq. (1), abstract] There are several typos: 'PronCons' in Fig. 3 legends, 'vallina' in the cosine-similarity equation (should be 'vanilla'), and 'ProCon wu saf e' in the abstract/body with stray spacing. Fixing these will improve readability.
  3. [§V-B] The text refers to 'Table 1' when discussing GSM8K results; the actual table is numbered Table IV. Please correct the cross-reference.
  4. [§IV-A2, §III-C3] The number of safety-oriented samples is given as '1,000' in Sec. IV-A2 and '1k' in Sec. III-C3. These are consistent, but the paper should specify whether these are the same set used by IFTsafe and how they are constructed (e.g., are they benign/malicious pairs used in the r-direction identification or separate safety prompts?).

Circularity Check

0 steps flagged

No significant circularity: ProCon is an intervention on a prior-work mechanism; the causal claim is underdetermined but not definitional.

full rationale

The paper's central claim is that refusal-direction drift causes IFT safety degradation. The ProCon loss regularizes the projection of hidden states onto an initial r-direction identified via difference-in-means (Sec. III-A). This is a direct intervention, not a tautology: the safety evaluations and drift metric are separate measurements, and the loss does not mathematically force the re-identified r_IFT to equal r_vanilla (the constraint fixes scalar projections, not the full direction vector). The paper explicitly frames the inference as a guess ('we guess... supporting our guess'), which weakens the causal claim but does not make it circular. No fitted parameter is renamed as a prediction; no load-bearing argument rests on a self-citation; the r-direction identification is attributed to external prior work [16]. The absence of an arbitrary-direction control is a validity threat, not a circularity. Minor self-citations ([13], [27]) are baselines or related work, not premises. Therefore score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The central claim rests on the existence and identifiability of the refusal direction from prior work, the assumption that the initial direction stays a meaningful anchor, a heuristic Fisher information argument, and the validity of GPT-4o-based safety scoring. It also depends on per-model choices of alpha and warm-up epochs, and on the unspecified composition of the 1,000 safety-oriented samples.

free parameters (3)
  • Constraint coefficient alpha = 0.1, 0.2 (light); 1, 2 (strong)
    The safety-performance tradeoff is controlled by alpha, and Fig. 4 shows the method is sensitive to it. No principled rule is given for choosing alpha per model and dataset.
  • Number of warm-up epochs = about 4 for LLaMA2/3, about 16 for Qwen2; halved with safety-oriented data
    Warm-up epochs are selected per model after inspecting results in Fig. 5. The paper states this is a hyperparameter that varies across LLMs.
  • Number of safety-oriented samples = 1,000
    The paper adds 1,000 safety-oriented samples to broaden the data distribution, chosen without derivation and described only as the same data used by IFTsafe.
axioms (6)
  • domain assumption A refusal direction exists in the residual stream and mediates refusal behavior.
    Adopted from prior work [16] in Sec. III-A without re-derivation; the whole method depends on this being true.
  • domain assumption Difference-in-means over malicious and benign activations identifies the refusal direction.
    Sec. III-A uses the difference-in-means estimator from [41], [42]; the choice of prompts and layers affects the anchor direction.
  • domain assumption The initial r-direction remains a valid and meaningful anchor throughout training.
    Sec. III-C1 records z_0 at initialization and uses it as the reference for all later constraints; the paper does not test whether the direction's semantic role is stable.
  • ad hoc to paper The Fisher information argument with kappa(z) justifies broadening the data distribution.
    Sec. III-C3 presents a formula for I_r(theta) with an undefined kappa(z) and no derivation; it is a heuristic justification added after observing the benefit of safety data.
  • domain assumption GPT-4o judge scores and ASR rule lists measure harmfulness and refusal.
    Sec. IV-A3 uses GPT-Judge and ASR without reporting judge agreement or validating the rule list; all safety conclusions inherit this assumption.
  • domain assumption Extra warm-up training epochs do not improve task performance beyond standard IFT.
    Sec. V-G asserts this to defend comparison fairness, but the supporting extended-epoch IFT experiment is not shown in any table or figure.
invented entities (1)
  • Effective sensitivity factor kappa(z) no independent evidence
    purpose: To justify why broader data distribution strengthens projection constraints through Fisher information.
    Introduced in Sec. III-C3 without a definition, measurement, or independent handle. It is a rhetorical placeholder in the presented derivation.

pith-pipeline@v1.3.0-alltime-deepseek · 23529 in / 15726 out tokens · 166859 ms · 2026-08-04T23:05:08.992990+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint." pith.science (2026). https://pith.science/paper/SYM4AE7D

@misc{pith2026250906795,
  author       = {Pith},
  title        = {Pith review of: Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SYM4AE7D}},
  note         = {Machine review of arXiv:2509.06795}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Instruction Fine-Tuning (IFT) has been widely adopted as an effective post-training strategy to enhance various abilities of Large Language Models (LLMs). However, prior studies have shown that IFT can significantly compromise LLMs' safety, particularly their ability to refuse malicious instructions, raising significant concerns. Recent research into the internal mechanisms of LLMs has identified the refusal direction (r-direction) in the hidden states, which plays a pivotal role in governing refusal behavior. Building on this insight, our study reveals that the r-direction tends to drift during training, which we identify as one of the causes of the associated safety risks. To mitigate such drift, our proposed ProCon method introduces a projection-constrained loss term that regularizes the projection magnitude of each training sample's hidden state onto the r-direction. Our initial analysis shows that applying an appropriate constraint can effectively mitigate the refusal direction drift and associated safety risks, but remains limited by overall performance barriers. To overcome this barrier, informed by our observation of early-stage sharp drift and a data-driven perspective, we introduce a warm-up strategy that emphasizes early-stage strong constraints and broaden the data distribution to strengthen constraint signals, leading to an enhanced ProCon method. Experimental results under various datasets, scenarios, and LLMs demonstrate that our method can significantly mitigate safety risks posed by IFT while preserving task performance gains. Even compared with strong baselines, our method consistently delivers superior overall performance. Crucially, our analysis indicates that ProCon can contribute to stabilizing the r-direction during training, while such an interpretability-driven exploration of LLMs' internal mechanisms lays a solid foundation for future safety research.

Figures

Figures reproduced from arXiv: 2509.06795 by Bing Qin, Fenglei Fan, Jiawei Cao, Kai He, Mengling Feng, Qika Lin, Sendong Zhao, Ting Liu, Yanrui Du.

Figure 1
Figure 1. Figure 1: Overall framework of our study. 1) We leverage benign–malicious data pairs to identify the r-direction, which plays a crucial role in maintaining [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The analysis of r-direction drift. The vertical axis represents the drift angle measured by [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: The impact of warm-up epochs on overall performance. The horizontal [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness

    cs.LG 2026-06 unverdicted novelty 8.0

    Fine-tuning updates frequently stale activation monitors for language model safety while quantization does not, with degradation predictable and repairable via label-free realignment.

  2. From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning

    cs.CR 2026-06 unverdicted novelty 6.0

    Harmfulness-refusal coupling is high early in R2D2 training (strong fixed-source robustness, low utility) then drops (partial utility recovery, reopened attacks), while SFT reaches low coupling with weaker robustness;...

  3. From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning

    cs.CR 2026-06 conditional novelty 5.5

    R2D2 fine-tuning of Mistral-7B transitions from high harmfulness–refusal coupling with collapsed utility to lower coupling with partial utility recovery and reopened jailbreaks; low coupling alone is not safety.

Reference graph

Works this paper leans on

52 extracted references · 7 canonical work pages · cited by 2 Pith papers · 3 internal anchors

  1. [1]

    Agentin- struct: Toward generative teaching with agentic flows,

    A. Mitra, L. Del Corro, G. Zheng, S. Mahajan, D. Rouhana, A. Co- das, Y . Lu, W.-g. Chen, O. Vrousgos, C. Rossetet al., “Agentin- struct: Toward generative teaching with agentic flows,”arXiv preprint arXiv:2407.03502, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Self-guide: Better task-specific instruction following via self-synthetic finetuning,

    C. Zhao, X. Jia, V . Viswanathan, T. Wu, and G. Neubig, “Self-guide: Better task-specific instruction following via self-synthetic finetuning,” arXiv preprint arXiv:2407.12874, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Probing the dual logic ability of privatized medical-domain llms,

    Y . Du, S. Zhao, M. Cai, M. Ma, D. Zhao, J. Cao, and B. Qin, “Probing the dual logic ability of privatized medical-domain llms,” in2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2024, pp. 3182–3187

    work page 2024

  4. [4]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!”arXiv preprint arXiv:2310.03693, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,”High-Confidence Computing, p. 100211, 2024

    2024

  6. [6]

    Harmful fine- tuning attacks and defenses for large language models: A survey,

    T. Huang, S. Hu, F. Ilhan, S. F. Tekin, and L. Liu, “Harmful fine- tuning attacks and defenses for large language models: A survey,”arXiv preprint arXiv:2409.18169, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack,

    T. Huang, S. Hu, and L. Liu, “Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack,”arXiv preprint arXiv:2402.01109, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation,

    T. Huang, S. Hu, F. Ilhan, S. F. Tekin, and L. Liu, “Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation,”arXiv preprint arXiv:2409.01586, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,

    F. Bianchi, M. Suzgun, G. Attanasio, P. R ¨ottger, D. Jurafsky, T. Hashimoto, and J. Zou, “Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,”arXiv preprint arXiv:2309.07875, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications,

    B. Wei, K. Huang, Y . Huang, T. Xie, X. Qi, M. Xia, P. Mittal, M. Wang, and P. Henderson, “Assessing the brittleness of safety alignment via pruning and low-rank modifications,”arXiv preprint arXiv:2402.05162, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Safety layers in aligned large language models: The key to llm security,

    S. Li, L. Yao, L. Zhang, and Y . Li, “Safety layers in aligned large language models: The key to llm security,”arXiv preprint arXiv:2408.17003, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Lisa: Lazy safety alignment for large language models against harmful fine-tuning at- tack,

    T. Huang, S. Hu, F. Ilhan, S. Tekin, and L. Liu, “Lisa: Lazy safety alignment for large language models against harmful fine-tuning at- tack,”Advances in Neural Information Processing Systems, vol. 37, pp. 104 521–104 555, 2024

    work page 2024

  13. [13]

    Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning,

    Y . Du, S. Zhao, J. Cao, M. Ma, D. Zhao, F. Fan, T. Liu, and B. Qin, “Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning,”arXiv e-prints, pp. arXiv–2410, 2024

    work page 2024

  14. [14]

    Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,

    R. Bhardwaj, D. D. Anh, and S. Poria, “Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,”arXiv preprint arXiv:2402.11746, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Safe lora: the silver lining of reducing safety risks when fine-tuning large language models,

    C.-Y . Hsu, Y .-L. Tsai, C.-H. Lin, P.-Y . Chen, C.-M. Yu, and C.-Y . Huang, “Safe lora: the silver lining of reducing safety risks when fine-tuning large language models,”arXiv preprint arXiv:2405.16833, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,”Advances in Neural Information Processing Systems, vol. 37, pp. 136 037–136 083, 2024

    2024

  17. [17]

    Directional Gradient Projection for Robust Fine-Tuning of Foundation Models

    C. Huang, J. Tian, B. Maneechotesuwan, S. Chopra, and Z. Kira, “Directional gradient projection for robust fine-tuning of foundation models,”arXiv preprint arXiv:2502.15895, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Linear representations of polit- ical perspective emerge in large language models,

    J. Kim, J. Evans, and A. Schein, “Linear representations of polit- ical perspective emerge in large language models,”arXiv preprint arXiv:2503.02080, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Model tampering attacks enable more rigorous evaluations of llm capabilities,

    Z. Che, S. Casper, R. Kirk, A. Satheesh, S. Slocum, L. E. McKinney, R. Gandikota, A. Ewart, D. Rosati, Z. Wuet al., “Model tampering attacks enable more rigorous evaluations of llm capabilities,”arXiv preprint arXiv:2502.05209, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Latent adversarial training improves robustness to persistent harmful behaviors in llms,

    A. Sheshadri, A. Ewart, P. Guo, A. Lynch, C. Wu, V . Hebbar, H. Sleight, A. C. Stickland, E. Perez, D. Hadfield-Menellet al., “Latent adversarial training improves robustness to persistent harmful behaviors in llms,” arXiv preprint arXiv:2407.15549, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Recursive deep models for semantic compositionality over a sentiment treebank,

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y . Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” inProceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1631–1642

    2013

  22. [22]

    Character-level convolutional net- works for text classification,

    X. Zhang, J. Zhao, and Y . LeCun, “Character-level convolutional net- works for text classification,”Advances in neural information processing systems, vol. 28, 2015

    2015

  23. [23]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Qwen2 technical report,

    “Qwen2 technical report,” 2024

    2024

  26. [26]

    A comprehensive study of jailbreak attack versus defense for large language models,

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek, “A comprehensive study of jailbreak attack versus defense for large language models,” inFindings of the Association for Computational Linguistics ACL 2024, 2024, pp. 7432–7449

    2024

  27. [27]

    Mogu: A framework for enhancing safety of llms while preserving their usability,

    Y . Du, S. Zhao, D. Zhao, M. Ma, Y . Chen, L. Huo, Q. Yang, D. Xu, and B. Qin, “Mogu: A framework for enhancing safety of llms while preserving their usability,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  28. [28]

    Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defense

    J. Zhang, K. Chen, L. He, J. Lou, D. Li, Z. Feng, M. Song, J. Liu, K. Ren, and X. Yang, “Activation approximations can incur safety vul- JOURNAL OF LATEX CLASS FILES, VOL. 00, NO. 0, SEPTEMBER 2025 12 nerabilities even in aligned llms: Comprehensive analysis and defense,” arXiv preprint arXiv:2502.00840, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Red teaming language models with language models,

    E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,”arXiv preprint arXiv:2202.03286, 2022

    Pith/arXiv arXiv 2022

  30. [30]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousseet al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,”arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Explore, establish, exploit: Red teaming language models from scratch,

    S. Casper, J. Lin, J. Kwon, G. Culp, and D. Hadfield-Menell, “Explore, establish, exploit: Red teaming language models from scratch,”arXiv preprint arXiv:2306.09442, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Cold-attack: Jail- breaking llms with stealthiness and controllability,

    X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu, “Cold-attack: Jail- breaking llms with stealthiness and controllability,”arXiv preprint arXiv:2402.08679, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak,

    Y . Du, S. Zhao, M. Ma, Y . Chen, and B. Qin, “Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak,” arXiv preprint arXiv:2312.04127, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?”Advances in Neural Information Processing Sys- tems, vol. 36, 2024

    2024

  35. [35]

    Foot in the door: Understanding large language model jailbreaking via cognitive psychology,

    Z. Wang, W. Xie, B. Wang, E. Wang, Z. Gui, S. Ma, and K. Chen, “Foot in the door: Understanding large language model jailbreaking via cognitive psychology,”arXiv preprint arXiv:2402.15690, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Exploiting programmatic behavior of llms: Dual-use through standard security attacks,

    D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto, “Exploiting programmatic behavior of llms: Dual-use through standard security attacks,”arXiv preprint arXiv:2302.05733, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  38. [38]

    Removing rlhf protections in gpt-4 via fine-tuning,

    Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. Hashimoto, and D. Kang, “Removing rlhf protections in gpt-4 via fine-tuning,”arXiv preprint arXiv:2311.05553, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y . Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” in Forty-first International Conference on Machine Learning, 2024

    2024

  40. [40]

    Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models

    D. Wu, X. Lu, Y . Zhao, and B. Qin, “Separate the wheat from the chaff: A post-hoc approach to safety re-alignment for fine-tuned language models,”arXiv preprint arXiv:2412.11041, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  41. [41]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets,

    S. Marks and M. Tegmark, “The geometry of truth: Emergent linear structure in large language model representations of true/false datasets,” arXiv preprint arXiv:2310.06824, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Steering llama 2 via contrastive activation addition,

    N. Panickssery, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner, “Steering llama 2 via contrastive activation addition,”arXiv preprint arXiv:2312.06681, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Universal statistics of fisher information in deep neural networks: Mean field approach,

    R. Karakida, S. Akaho, and S.-i. Amari, “Universal statistics of fisher information in deep neural networks: Mean field approach,” inThe 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019, pp. 1032–1041

    work page 2019

  44. [44]

    Where is the information in a deep neural network?

    A. Achille, G. Paolini, and S. Soatto, “Where is the information in a deep neural network?”arXiv preprint arXiv:1905.12213, 2019

    Pith/arXiv arXiv 1905

  45. [45]

    Advancing llm reasoning generalists with preference trees,

    L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y . Linet al., “Advancing llm reasoning generalists with preference trees,”arXiv preprint arXiv:2404.02078, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  47. [47]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Attack prompt generation for red teaming and defending large language mod- els,

    B. Deng, W. Wang, F. Feng, Y . Deng, Q. Wang, and X. He, “Attack prompt generation for red teaming and defending large language mod- els,”arXiv preprint arXiv:2310.12505, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,”arXiv preprint arXiv:2310.04451, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,”arXiv preprint arXiv:2310.08419, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Unlocking efficient long-to-short llm reasoning with model merging,

    H. Wu, Y . Yao, S. Liu, Z. Liu, X. Fu, X. Han, X. Li, H.-L. Zhen, T. Zhong, and M. Yuan, “Unlocking efficient long-to-short llm reasoning with model merging,”arXiv preprint arXiv:2503.20641, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schul- man, “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021. APPENDIX In Tab. VII, we present all cases of evaluation data used in our experiments. Tab. VIII and Tab. IX present detailed ASR...

    Pith/arXiv arXiv 2021