Pith. sign in

REVIEW 3 major objections 4 minor 18 references

Latent Adversarial Training Improves the Representation of Refusal

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Latent adversarial training concentrates refusal behavior into the first two principal components of activation differences, making the refusal vector more transferable while making the LAT model more vulnerable to self-ablation.

desk verdict LAT's refusal representation claim is a single-checkpoint observation; the cross-model transfer result is the more solid takeaway. read the letter →

arxiv 2504.18872 v1 pith:T5MPV7N4 submitted 2025-04-26 cs.CL cs.LG

Alexandra Abbas , Nora Petrova , Helios Ael Lyons , Natalia Perez-Campanero This is my paper · ORCID
classification cs.CLcs.LG
keywords latentadversarialtrainingrefusaldirectionsingularvaluedecompositionablationattackactivationsteeringmodelsafetyLlama2representationanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Latent Adversarial Training (LAT), which adds small adversarial perturbations inside a model's hidden layers, changes how a language model stores its refusal behavior. On Llama 2 7B, the paper finds that LAT packs the difference between harmful and harmless activations into the first two singular-value components, which together explain roughly 75% of the variance, compared with about 54% for standard supervised safety fine-tuning and about 49% for embedding-space adversarial training. This concentrated representation makes the refusal vector extracted from a LAT model unusually strong and transferable: it suppresses refusal more effectively on every model tested than vectors taken from the other variants. The same concentration, however, makes the LAT model itself more vulnerable to a refusal vector computed from its own activations, so LAT is not uniformly more robust. The finding matters because it shows that a training-time noise method can reorganize internal safety features, not just patch the model's surface behavior.

What carries the argument

The load-bearing object is the refusal direction: the difference in mean activations between harmful and harmless instruction pairs, measured at the last token position of layer 14, normalized into a unit vector $\hat r$. The paper's analysis tool is a singular value decomposition of the matrix of activation differences across the 100 paired prompts; the fraction of variance held by the first two SVD components is the operational measure of how concentrated refusal is. LAT is implemented as an adversarial perturbation with an $L^2$-norm bound added to the residual stream at layer 4 during training, following the configuration of Casper et al. (2024), and the evaluation applies the directional ablation operation $x' \leftarrow x - \hat r \hat r^\top x$ at all layers and token positions. The pairing of these parts is what turns a training-method comparison into a claim about representation geometry.

What would settle it

Run the SVD analysis on the held-out 520-prompt set rather than the 100 pairs used to derive the refusal directions; if the first two LAT components explain well below 74% on held-out prompts, the reported concentration is a property of the small derivation set, not of LAT.

Watch

Extended reading notes

Core claim

The paper's central claim is that LAT does not disperse the refusal feature, as one might expect from injecting noise; it concentrates it. Using 100 harmful/harmless instruction pairs, the authors compute the mean activation difference at the last token position of layer 14, then run SVD on the per-pair activation differences. In the LAT variant the first singular component explains more than 54% of the variance and the first two components together about 74–75%, whereas the baseline model's first component explains 49.43% and its second only 4.76% (the AT variant is similar, 43.76% plus 4.79%). Under a directional ablation attack that removes the refusal direction from the residual stream, the LAT model's own vector lowers its post-ablation refusal rate to 16.92%, below the baseline's 20.38% and far below the AT model's 38.08%, while the LAT-derived vector achieves the lowest refusal rates when transferred to all three models (10.77%, 13.65%, and 16.92%). The paper concludes that LAT's perturbations create a more comprehensive, more compact refusal representation that is both a better attack artifact and a bigger self-attack surface.

Load-bearing premise

The paper's comparison of SVD variance across SSFT, AT, and LAT assumes the three variants were trained with comparable perturbation budgets, step counts, and optimizer settings, but those hyperparameters are not reported, so the concentrated encoding attributed to LAT could conceivably come from the training recipe rather than from the latent-perturbation method itself.

Editorial extensions

If this is right

  • LAT models retain a higher refusal rate than SSFT or AT models when attacked with refusal vectors taken from a different model, so transferable white-box attacks become less effective against LAT.
  • The refusal vector extracted from LAT is the strongest transfer attack tested: it drops the baseline model's refusal rate to 10.77% and the AT model's to 13.65%, so a single LAT-derived vector can be reused against multiple targets.
  • Layer 14 stays the most effective ablation site across all three variants, meaning LAT does not relocate the refusal feature; it compresses it within the same layer.
  • Because such a compact representation is easier to fit with one vector, safety evaluations that only use externally derived vectors will systematically overstate LAT robustness, while self-vector attacks expose the weakness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ~75% variance concentration is a stable property rather than an artifact of the 100-pair derivation set, LAT models should be unusually tractable for mechanistic interpretability: nearly two-dimensional refusal features should be easy to trace, name, and edit.
  • The result suggests a general trade-off for latent-space defenses: making a safety feature more linearly concentrated improves its transferability to other models but also hands attackers a sharper instrument for self-ablation; testing self-generated vectors should become a standard robustness check for any latent adversarial training variant.
  • The anomalous high invalid-response rates in LAT layers 2–3 hint that perturbations at layer 4 have indirect early-layer effects; training an LAT variant with the perturbation applied at a different layer would separate the generic effect of latent noise from the layer-4-specific one.
  • Targeted latent adversarial training (TLAT) inherits LAT's perturbation mechanics, so the concentration and self-ablation vulnerability may carry over to TLAT models even though the paper does not test them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies how Latent Adversarial Training (LAT) changes the encoding of refusal behavior in Llama-2-7B-chat, relative to standard supervised safety fine-tuning (SSFT) and embedding-space adversarial training (AT). The authors compute a refusal direction as the mean activation difference between 100 harmful/harmless instruction pairs at layer 14, apply SVD to these activation differences, and test directional-ablation attacks on 520 harmful prompts. They report that LAT concentrates refusal variance in the first two SVD components (about 74-75%, vs. about 54% and 48% for baseline and AT), that the LAT-derived refusal vector transfers better across models, and that LAT is more vulnerable to self-generated refusal vectors than SSFT or AT.

Significance. If the SVD concentration result is robust, the paper identifies a previously unremarked effect of latent adversarial training: rather than dispersing a safety-critical direction, LAT can compress it into a smaller subspace, with a corresponding trade-off between cross-model transferability and self-attack vulnerability. This would be a useful empirical contribution to interpretability-based safety evaluations. The paper also has strengths: it follows the standard self-attack protocol of Arditi et al. (2024), uses a public codebase, and reports confidence intervals for the refusal-rate comparisons. However, the headline SVD claim currently rests on a single checkpoint per variant with no error bars or null baseline, and the training hyperparameters needed to establish comparability of the three variants are not reported. The result is therefore conditional on additional uncertainty quantification.

major comments (3)
  1. [§4.2 and Abstract] The central claim that LAT concentrates refusal variance in the first two SVD components is supported only by point estimates from a single checkpoint per model variant. No standard errors, bootstrap over the 100 prompt pairs, repeated training seeds, or null distribution from random matrices are provided. Because the gap in the second component (20% vs. ~5%) is the load-bearing evidence for the paper's main conclusion, the manuscript should report uncertainty for the SVD explained-variance values, for example via bootstrap or multiple seeds, before claiming the difference is significant.
  2. [§3.1] Training hyperparameters for the AT and LAT variants are not reported: the perturbation norm bound epsilon, the number of adversarial steps, the optimizer settings, the number of epochs, and the random seeds are all absent. Since the paper compares the latent representations of three training methods, the reader cannot verify that the variants were trained at comparable strength. If LAT used a larger epsilon or a different number of steps than AT, the observed concentration could reflect the training budget rather than the LAT method itself.
  3. [§4.1] The text states that the LAT model performed 'notably worse than the baseline SSFT model' in the self-ablation condition, but the confidence intervals overlap substantially: baseline 20.38% [16.91%, 23.85%] and LAT 16.92% [13.71%, 20.13%]. The observed difference is not statistically significant at the reported precision, so the claim of a notable difference is unsupported. The comparison between AT and LAT is significant because their intervals do not overlap, but the baseline-versus-LAT claim should be softened or supported with a formal test.
minor comments (4)
  1. [§4.2 vs. Abstract] The abstract reports 'approximately 75%' for the first two SVD components, while §4.2 reports 'approximately 74%'; the first component is described as 'more than 54%' in §4.2 but labeled 54% in Figure 2. These numbers should be made consistent.
  2. [§4.1] The superscript '(1)' after 'significantly outperforming both the baseline SSFT model ... and the LAT model' has no corresponding footnote text; either add the footnote or remove the marker.
  3. [References] Several reference entries have formatting issues, such as 'URLhttps://' in the Touvron et al. entry; a pass over the bibliography for spacing and line breaks is needed.
  4. [§5] The phrase 'LAT's training perturbations enable a more comprehensive representation of refusal behavior' is interpretive; as written it is not directly established by the SVD or ablation results and should be flagged as a hypothesis rather than a finding.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: SVD concentration is descriptive and ablation attacks are standard self-attack evaluation.

full rationale

The paper's central quantitative claim is a descriptive SVD analysis of activation differences between harmful and harmless prompts, not a fitted parameter or a prediction derived from the claim itself. The refusal direction is computed by mean activation differences following Arditi et al. (2024), and the SVD explained variance is a property of the same activation-difference matrix; neither quantity is used as a fit parameter to produce the refusal-rate results. The ablation protocol uses a direction from the same model, but this is the standard self-attack protocol with evaluation on a separate 520-example test set, and the comparison across model variants (AT self-attack 38.08%, LAT self-attack 16.92%) shows the outcome is not forced by construction. Cross-model transfer is an independent empirical comparison. Citations to Casper et al. (2024) and Arditi et al. (2024) are external sources with released code/artifacts, not self-citations. The interpretive phrase 'more comprehensive representation' is speculative rather than circular. Statistical concerns (single checkpoint per variant, unreported training hyperparameters, no confidence intervals on SVD variance) are robustness/correctness issues, not circularity. Therefore no circular step is identifiable; score 1 reflects only the minor interpretive gap, not a circular reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the linear refusal direction method, the ablation protocol, and the comparability of the three training variants. No new entities are introduced.

free parameters (3)
  • Refusal direction layer = 14
    The refusal direction is computed at layer 14 of 32, chosen following Arditi et al. (2024). The paper does not test sensitivity to this choice, and the SVD concentration results could differ at other layers.
  • LAT perturbation layer = 4
    Latent adversarial perturbations are applied at the fourth layer, following the optimal configuration from Casper et al. (2024). This choice affects where the representation is concentrated.
  • Adversarial perturbation norm bound (epsilon)
    The L2-norm bound for AT and LAT perturbations is not reported. The magnitude of noise could influence how much the refusal representation is reorganized.
assumptions (4)
  • domain assumption Refusal behavior is encoded in a direction that can be estimated by the mean activation difference between harmful and harmless prompts at the last token position.
    Invoked in Section 3.2 following Arditi et al. (2024). If the linear-direction assumption fails, the SVD analysis and ablation attacks target the wrong quantity.
  • domain assumption Removing the component along the refusal direction from all residual stream activations disables refusal behavior.
    Used in Section 3.3 to define the ablation attack. The paper does not validate this assumption independently for the LAT model.
  • domain assumption SVD of activation differences between harmful and harmless pairs reveals the structure of refusal representation.
    Used in Section 3.4 to quantify concentration. The choice of explained variance as the metric is an analytical assumption.
  • domain assumption The AT and LAT training configurations from Casper et al. (2024) are appropriate and directly comparable to the SSFT baseline.
    Section 3.1 states the authors adapted the framework without reporting hyperparameters, so the comparability of the three variants is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Latent Adversarial Training Improves the Representation of Refusal." pith.science (2026). https://pith.science/paper/T5MPV7N4

@misc{pith2026250418872,
  author       = {Pith},
  title        = {Pith review of: Latent Adversarial Training Improves the Representation of Refusal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5MPV7N4}},
  note         = {Machine review of arXiv:2504.18872}
}
read the original abstract

Recent work has shown that language models' refusal behavior is primarily encoded in a single direction in their latent space, making it vulnerable to targeted attacks. Although Latent Adversarial Training (LAT) attempts to improve robustness by introducing noise during training, a key question remains: How does this noise-based training affect the underlying representation of refusal behavior? Understanding this encoding is crucial for evaluating LAT's effectiveness and limitations, just as the discovery of linear refusal directions revealed vulnerabilities in traditional supervised safety fine-tuning (SSFT). Through the analysis of Llama 2 7B, we examine how LAT reorganizes the refusal behavior in the model's latent space compared to SSFT and embedding space adversarial training (AT). By computing activation differences between harmful and harmless instruction pairs and applying Singular Value Decomposition (SVD), we find that LAT significantly alters the refusal representation, concentrating it in the first two SVD components which explain approximately 75 percent of the activation differences variance - significantly higher than in reference models. This concentrated representation leads to more effective and transferable refusal vectors for ablation attacks: LAT models show improved robustness when attacked with vectors from reference models but become more vulnerable to self-generated vectors compared to SSFT and AT. Our findings suggest that LAT's training perturbations enable a more comprehensive representation of refusal behavior, highlighting both its potential strengths and vulnerabilities for improving model safety.

Figures

Figures reproduced from arXiv: 2504.18872 by the authors.

Figure 1
Figure 1. Comparison of refusal rates under different ablation attack vectors across Llama-2-7B-chat [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Explained variance by SVD components across model variants. The plot shows the per [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Principal Component Analysis (PCA) visualization of harmful vs harmless instruction [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Layer-wise analysis of refusal rates under self-generated refusal vector attacks. The plot [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages

  1. [1]

    Refusal in Language Models Is Mediated by a Single Direction , 2024

    Andy Arditi et al. Refusal in Language Models Is Mediated by a Single Direction , 2024. URL https://arxiv.org/abs/2406.11717

    arXiv 2024

  2. [2]

    latent\_adversarial\_training, 2024

    Stephen Casper. latent\_adversarial\_training, 2024. URL https://github.com/thestephencasper/latent_adversarial_training

    work page 2024

  3. [3]

    Defending Against Unforeseen Failure Modes with Latent Adversarial Training , 2024

    Stephen Casper et al. Defending Against Unforeseen Failure Modes with Latent Adversarial Training , 2024. URL https://arxiv.org/abs/2403.05030

    arXiv 2024

  4. [4]

    Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

    Samyak Jain et al. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2311.12786

    arXiv 2024

  5. [5]

    LORA Fine-Tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B , 2024

    Simon Lermen et al. LORA Fine-Tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B , 2024. URL https://arxiv.org/abs/2310.20624

    arXiv 2024

  6. [6]

    Llama 2 7B Chat , 2023

    Meta. Llama 2 7B Chat , 2023. URL https://huggingface.co/meta-llama/Llama-2-7b-chat-hf

    work page 2023

  7. [7]

    The Llama 3 Herd of Models , 2024

    Meta. The Llama 3 Herd of Models , 2024. URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  8. [8]

    GPT-4 Technical Report , 2023

    OpenAI. GPT-4 Technical Report , 2023. URL https://arxiv.org/abs/2303.08774

    arXiv 2023

Show all 18 references
  1. [9]

    Steering Llama 2 via Contrastive Activation Addition , 2024

    Nina Rimsky et al. Steering Llama 2 via Contrastive Activation Addition , 2024. URL https://arxiv.org/abs/2312.06681

    2024 arXiv

  2. [10]

    Latent adversarial training improves robustness to persistent harmful behaviors in LLM s, 2024

    Abhay Sheshadri et al. Latent adversarial training improves robustness to persistent harmful behaviors in LLM s, 2024. URL https://arxiv.org/abs/2407.15549

    2024 arXiv

  3. [11]

    Stanford Alpaca: An Instruction-following LLaMA model , 2023

    Rohan Taori et al. Stanford Alpaca: An Instruction-following LLaMA model , 2023. URL https://github.com/tatsu-lab/stanford_alpaca

    2023

  4. [12]

    Llama 2: Open Foundation and Fine-Tuned Chat Models , 2023

    Hugo Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models , 2023. URL https://arxiv.org/abs/2307.09288

    2023 arXiv

  5. [13]

    Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models , 2023

    Xianjun Yang et al. Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models , 2023. URL https://arxiv.org/abs/2310.02949

    2023 arXiv

  6. [14]

    Universal and Transferable Adversarial Attacks on Aligned Language Models , 2023

    Andy Zou et al. Universal and Transferable Adversarial Attacks on Aligned Language Models , 2023. URL https://arxiv.org/abs/2307.15043

    2023 arXiv

  7. [15]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  8. [16]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  9. [17]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  10. [18]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.