Pith. sign in

REVIEW 4 major objections 4 minor 35 references

This paper claims that latent adversarial training for LLMs can be made about twice as cheap (48.1% fewer per-step FLOPs) by pairing a suffix-window low-rank defense with attacks generated on an ActGrad-pruned surrogate, with a contained lo

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:27 UTC pith:WUS5VBUK

load-bearing objection Worth a careful read for the suffix-window defense result; the surrogate-transfer proof has an unstated idealization, so the efficiency claim rests more on ablations than on Theorem 3. the 4 major comments →

arxiv 2607.28959 v1 pith:WUS5VBUK submitted 2026-07-31 cs.LG cs.AI

Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates

Weiyi He , Yuping Lin , Jiliang Tang , Yue Xing This is my paper
classification cs.LG cs.AI
keywords adversarial traininglatent adversarial trainingrepresentation fine-tuninglow-rank defensecircuit pruningsurrogate modelsLLM robustnessefficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Adversarial training makes language models robust but is extremely expensive, and this paper attacks that cost from two sides at once. On the defense side, it freezes the backbone and trains only a low-rank representation intervention (LAT-ReFT) over a suffix window of tokens, showing that single-token defenses leak adversarial signal. On the attack side, it generates the inner-loop adversarial perturbations on a pruned surrogate model that keeps only the most attack-relevant MLP neurons, then transfers those perturbations to the full model for training. The authors claim this combination cuts per-step adversarial-training FLOPs by 48.1% on average and uses only 0.0118% trainable parameters, while retaining useful though lower robustness than full fine-tuning. The theoretical results aim to justify both design choices: suffix-window defenses provably suppress attention leakage, and activation-gradient pruning provably controls the surrogate's optimization gap.

Core claim

On its own terms, the paper establishes a workable decomposition of adversarial training into a low-rank defense and an attack-side surrogate. The defense-side discovery is a placement principle: a sequence-shared ReFT intervention applied over a suffix window and placed in early or middle layers provably limits how adversarial perturbations from attacked positions propagate into the readout at the final token, whereas a last-token-only intervention provably fails to block that leakage. The attack-side discovery is that retaining MLP neurons with the largest activation-times-gradient importance (ActGrad) controls the PGD optimization gap between the surrogate and the full model; empirically,

What carries the argument

The load-bearing objects are (1) the ReFT operator, a low-rank intervention on hidden states that is applied over a defended suffix window and shared across positions; (2) the ActGrad importance score, which ranks each MLP neuron by the absolute product of its activation and the gradient of the loss with respect to that activation; and (3) the surrogate PGD loop, which runs inner-maximization on the pruned model and transfers the found perturbation back to the full defended model. The theory ties these together: Theorem 2 shows suffix-window ReFT bounds attention-value and attention-weight leakage, and Theorem 3 shows the surrogate's optimization gap is controlled by the omitted act-by-gradi

Load-bearing premise

The whole method stands on the premise that adversarial perturbations found on the smaller surrogate model still move the full model's loss in the right direction; if those attack directions drift, the inner maximization optimizes the wrong objective and the defense trains on useless perturbations.

What would settle it

Run the inner attack on both the pruned surrogate and the full defended model on the same inputs and compare (a) the cosine similarity of their PGD update directions and (b) the full-model loss achieved by the transferred surrogate perturbation versus the full-model perturbation, at pruning ratios 0%, 25%, and 50%. If direction similarity collapses beyond 25% pruning, or if random pruning matches ActGrad on these transfer metrics, the surrogate-transfer premise is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A frozen backbone with only 0.0118% trainable parameters makes adversarial training feasible for models too large to fine-tune end-to-end on a single GPU.
  • The 48.1% average per-step FLOP reduction means roughly twice the training throughput or twice the model scale for the same compute budget.
  • The suffix-window defense principle is a usable rule: a defense should cover every position an attack can touch and sit early enough to correct representations before the readout.
  • The ActGrad selection rule gives a concrete, theoretically motivated way to prune a model for attack generation rather than for inference speed.
  • The reported robustness numbers (e.g., GCG ASR 0.15 vs 0.02 for full LAT on Llama-3.1-8B) make the tradeoff explicit: this method is lighter but not a drop-in replacement for full-parameter adversarial training when maximum robustness is required.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same surrogate-transfer scheme could be applied to discrete token-space attacks such as GCG, not just latent PGD, potentially lowering the cost of adversarial example search itself.
  • A fixed 25% pruning ratio is unlikely to be optimal across datasets; the omitted-act-grad mass bound suggests an adaptive per-layer pruning schedule that stops when the predicted transfer gap crosses a threshold.
  • The observed concentration of important neurons and effective defense layers in early-to-middle layers points toward a broader, testable hypothesis that robustness-relevant computation in Transformers is localized and layer-ordered.
  • The theory's ideal projection assumption (ε_t = 0) leaves the real surrogate mismatch unquantified, so a direct measurement of PGD-direction cosine similarity between surrogate and full model would be the natural next check.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LAT-ReFT, an adversarial-training method for LLM classification that combines representation fine-tuning (ReFT) over a suffix window with a circuit-guided pruned MLP surrogate for inner-loop PGD attack generation. The defense-side analysis (Theorems 1 and 2) argues that single-token ReFT is insufficient and that a sequence-shared suffix-window ReFT can bound attention-level leakage. On the attack side, the paper introduces ActGrad, an activation×gradient importance score for selecting MLP neurons to retain in a surrogate, and gives an informal theorem (Theorem 3) bounding the surrogate PGD gap by the omitted act×grad mass. Empirically, on IMDB across Llama-3.1-8B, Qwen-2.5-3B, and Pythia-1.4B, the method reports average per-step FLOPs reduction of 48.1% relative to full LAT with only 0.0118% trainable parameters, at the cost of higher GCG ASR (e.g., 0.15 vs. 0.02 on Llama-3.1-8B). Additional experiments on Pythia-1.4B examine suffix-window length, defense layer, pruning ratio, and neuron-selection rules.

Significance. If the claims hold, the paper demonstrates a practically useful efficiency–robustness tradeoff: adversarial training with very few trainable parameters and reduced attack-generation cost. The defense-side theorems are genuine derivations, and the empirical comparison of ActGrad with random and ActDiff selection (Fig. 3) is a useful sanity check that neuron-selection matters. The FLOPs and parameter counting in Appendix E is transparent and reproducible in structure. However, the central attack-side claim—that PGD on a pruned surrogate transfers to the full defended model—rests on an idealized assumption in Theorem 3 that is not established and is not isolated by the experiments. Because the headline robustness numbers combine the ReFT defense and the surrogate, the current evidence does not yet support the specific claim that the surrogate preserves robustness.

major comments (4)
  1. [§5.2 / Appendix D.4 (Theorem 3)] The surrogate-transfer premise is the load-bearing claim for the attack-side efficiency gain, but Theorem 3 proves only a bound on one-step objective gaps under the assumption r^s_t ≈ P_S r_t and ε_t=0 ('ideal projected-gradient case'). This is exactly the property that needs proof: the surrogate prunes downstream MLP blocks, so its gradient with respect to the expansion is not automatically the projected full gradient, and ε_t is never bounded. Moreover, bound (7) is on ΣG_s, not on the quality of the final perturbation eH^K transferred to the full model or on the full-model loss; no gradient-alignment measurement is provided. Because ActGrad selects the largest act×grad neurons, the right-hand side is minimized by construction, making the statement partly circular. Please provide a non-circular bound on ε_t or direct measurement of surrogate/full gradient alignment (e.g., cosine simila
  2. [Table 1 / Algorithm 1] The headline 'Ours' numbers combine two interventions: suffix-window ReFT and the ActGrad surrogate. There is no ablation isolating the surrogate from the defense. It is therefore possible that the observed robustness (e.g., GCG ASR 0.15 vs. 0.02 for LAT on Llama-3.1-8B) comes mostly or entirely from the suffix-window ReFT, with the surrogate contributing no robustness benefit. Figure 3 and Table 6 demonstrate surrogate transfer only on Pythia-1.4B at 25% pruning under GCG, and they do not characterize where transfer breaks. Please add: (a) Ours with the full model in place of the surrogate, keeping the same ReFT defense; (b) random-pruning and ActDiff surrogates on the three-model IMDB table; (c) an attack-transfer curve over pruning ratios for the models in Table 1, not only Pythia-1.4B.
  3. [§6.1, Table 4, Appendix A.4] The paper gives no uncertainty quantification: all ASR and accuracy numbers in Table 1 appear to be single runs. More importantly, key hyperparameters—ReFT layer, attack layer, ε, suffix length L, and pruning ratio—are chosen per model/dataset, but no selection procedure is documented; Table 4 simply states the values. Since Fig. 2 shows that layer placement and suffix length strongly affect ASR, the 'retains useful robustness' claim cannot currently be distinguished from favorable hyperparameter selection. Please report standard errors/confidence intervals over multiple seeds and specify a clear selection rule (fixed grid, validation-set sweep, or similar) for every hyperparameter, including the pruning ratio selected in §6.4.
  4. [Appendix E / Table 1] The 48.1% FLOPs reduction is an estimate based on dense-equivalent parameter counts and assumed 2N/4N/6N FLOP multipliers, with the surrogate size approximated as N_s≈N(1−f_MLP r_p). This is a reasonable planning metric but not a measured compute cost; it also treats all layers as equal and does not include memory movement or implementation overheads. Please state explicitly in the main text that the headline is estimated FLOPs, not wall-clock time, and report actual per-step training times (or at least a sensitivity analysis of the FLOPs formula under different multiplier assumptions) for the models in Table 1.
minor comments (4)
  1. [Figure 1] The figure is difficult to read: the x-axis labels ('Last token', 'Suffix window', 'Defended token span', etc.) are not clearly tied to the bars, and the layer axis is missing. Please provide a labeled schematic with model/layer names and numerical ASR values in the caption or in the figure itself.
  2. [§6.2 / Table 1] The average 0.0118% trainable parameters is not in Table 1; it would be helpful to show the average explicitly in the caption or text, since it is a headline number.
  3. [Appendix D.4 (Lemma 1)] Lemma 1 gives a bound with a constant C_attn=O(T), but the proof sketches the softmax Jacobian bound without showing the sequence-length accumulation factor explicitly. A fully explicit derivation would strengthen the lemma and make the 'C_attn' constant reproducible.
  4. [§6.4 / Table 6] The EnronSpam result where 25% pruning improves ASR over the full surrogate (0.08 vs. 0.26) is interesting but under-explained. A brief discussion of why pruning can remove noisy or task-irrelevant neurons would help the reader interpret the transfer curve.

Circularity Check

2 steps flagged

Theorem 3's ActGrad justification is self-referential and assumes the surrogate-transfer premise; defense theorems and FLOPs arithmetic are not circular.

specific steps
  1. self definitional [Section 5.2 (Theorem 3 / Eq. 7) and Section 4.2 (Eq. 3)]
    "Theorem 3 shows that surrogate quality is controlled by the omitted act×grad mass M_s(S). At a fixed pruning ratio ρ, an effective surrogate should therefore retain neurons with the largest w_{s,j}."

    The theorem's bound is expressed through M_s(S) = (Σ_{j∉S} w_{s,j})^{1/2}, where w_{s,j}=|a^{(l_a)}_{s,j} r^{(l_a)}_{s,j}|^2. Eq. (3) defines ActGrad by the same act×grad quantity |a ∂ℓ/∂a| and the method retains top-scoring neurons. At fixed ρ, choosing the largest w_{s,j} is exactly what minimizes M_s(S). Thus the theorem's recommendation restates the selection rule by construction; the 'quality' measure is defined to be the object ActGrad minimizes, rather than being derived independently.

  2. other [Appendix D.4, Proof of Theorem 3 ('Under the surrogate approximation' paragraph)]
    "Under the surrogate approximation, we identify the surrogate MLP-path gradient with the projected full gradient, i.e. r s t ≈ P S r t. ... where ε t collects the surrogate mismatch induced by replacing r s t with P S r t. In the ideal projected-gradient case, ε t = 0."

    This is the transfer premise the theorem is supposed to justify: that the pruned surrogate's PGD direction tracks the full model's loss. The proof assumes r^s_t ≈ P_S r_t and sets the mismatch ε_t to zero, i.e., it assumes the surrogate gradient equals the full gradient projected onto retained neurons. The bound is then a first-order consequence of that assumed equality plus Lipschitz/sign-stability assumptions; it does not establish that pruning preserves the relevant gradient direction. The theorem therefore assumes the key conclusion of the surrogate-transfer argument.

full rationale

Most of the paper is not circular: the 48.1% FLOPs reduction is arithmetic from the pruning ratio and ReFT parameter counts (Appendix E), and Theorems 1 and 2 are genuine conditional derivations about attention leakage for suffix-window versus last-token defenses, not definitions of the defenses' success. The self-citations (He & Xing 2025, Lin et al. 2024, Ren et al. 2025, Zeng et al. 2025) are background/related work and are not load-bearing. The circularity is concentrated in Theorem 3, the only formal support for the ActGrad surrogate: (i) its bound is written in terms of the omitted act×grad mass, which is precisely the quantity ActGrad is designed to minimize, so the theorem's effectiveness conclusion follows from the definition of the scoring criterion; and (ii) the proof assumes r^s_t ≈ P_S r_t with ε_t = 0, which is the surrogate-transfer property the theorem is invoked to establish. Because Table 1 and Figure 3 provide independent empirical evidence of robustness and transfer at 25% pruning, the circularity is partial rather than total.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The paper contributes a combination of existing representation-editing and pruning techniques. The main 'axioms' are regularity assumptions on the transformer and, more critically, an unverified pair of assumptions (Assumptions 2-3) and an idealized projection approximation that make the surrogate-transfer theorem go through. The defense-side theorems rely only on standard Lipschitz assumptions; the attack-side theory carries most of the ad hoc burden.

free parameters (8)
  • Perturbation budget ε (LAT inner attack) = 0.2 (Pythia), 0.5 (Llama, Qwen)
    Chosen per model; no sensitivity analysis reported in main text.
  • PGD steps K = 8
    Fixed for all models and baselines; standard choice but not sensitivity-tested.
  • Adversarial weight λ_adv = 1.0
    Fixed in Table 4; no ablation on λ.
  • ReFT rank r = 64
    Fixed for all models; no rank ablation.
  • Defended suffix window length L = 20
    Default for main experiments; Fig. 2(a) ablates L=1..20 on Pythia only.
  • Defense layer l_r (and attack layer l_a) per model = Pythia 12/4, Llama 8/4, Qwen 18/18
    These choices are reported in Table 4 without a documented selection rule; the unified early/middle-layer insight is inferred from these choices.
  • MLP pruning ratio ρ = 25%
    Chosen based on Fig. 3(a) on a single dataset/model (IMDB, Pythia); the optimal ratio varies across datasets (Table 6).
  • ActGrad calibration set D = unspecified size
    Eq. (3) uses a 'small calibration set drawn from the clean training data'; no size or composition given.
axioms (6)
  • domain assumption Assumption 1 (Local Lipschitz continuity): map from hidden states to attention output is locally Lipschitz near the clean trajectory with constant L_α.
    Used in Theorems 1 and 2; standard for Transformers, but no numerical Lipschitz constants are provided.
  • ad hoc to paper Assumption 2 (Non-degeneracy on pruned active coordinates): all pruned-coordinate activations |a_t,j(x)| ≥ m_0 with probability 1−p.
    Introduced in Appendix D.4 to bound the gradient of pruned coordinates; not verified empirically, and likely false for small activations.
  • ad hoc to paper Assumption 3 (Local sign-stability): ⟨g_t,u_t⟩−⟨g_t,u^s_t⟩ ≤ c∥g_t−g^s_t∥ on the same event.
    This assumption effectively states the one-step improvement gap is controlled by the gradient gap, which is the core claim of Theorem 3; it is not derived.
  • ad hoc to paper Ideal projection approximation r^s_t ≈ P_S r_t, with ε_t=0.
    The proof of Theorem 3 (Appendix D.4) assumes the surrogate MLP-path gradient equals the projected full gradient and discards the mismatch term; the text says 'in the ideal projected-gradient case'.
  • standard math Bounded weight matrices and Lipschitz activation functions (σ, φ).
    Standard regularity bounds used in Lemma 1 and Theorems 1-2.
  • standard math First-order Taylor expansion of the loss with o(η) remainder.
    Used in the proof of Theorem 3; requires small step size η.

pith-pipeline@v1.3.0-daily-deepseek · 23269 in / 16271 out tokens · 130254 ms · 2026-08-03T16:27:01.335267+00:00 · methodology

0 comments
read the original abstract

Adversarial training is one of the most effective defenses against adversarial attacks, yet the computational cost remains prohibitive at modern scales, especially for large language models (LLMs). While existing mitigation strategies, e.g., latent adversarial training (LAT), have been developed, they still incur a high computational cost. In this work, we comprehensively investigate computation-efficient strategies to speed up LAT from two complementary perspectives: (1) Defense-side optimization: We explore the representation fine-tuning (ReFT) within LAT, and reveal a potential issue if there is a mismatch on which tokens to apply ReFT and the attack. (2) Attack-side optimization: When computing adversarial attacks in each LAT iteration, we extract only the relevant circuits from the LLM to construct a lightweight surrogate model, avoiding the computation in the forward-backward passes through the full model during the attack generation. For both perspectives, we provide theoretical justifications and numerical evidence to illustrate the effectiveness of the proposed strategies. Ultimately, compared to standard LAT with full fine-tuning, our method on average reduces per-step adversarial-training FLOPs by 48.1% while requiring only 0.0118% trainable parameters.

Figures

Figures reproduced from arXiv: 2607.28959 by Jiliang Tang, Weiyi He, Yue Xing, Yuping Lin.

Figure 1
Figure 1. Figure 1: A naive latent defense that protects only [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Defense-placement ablations on IMDB with Pythia-1.4B under GCG and RandomToken [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Analysis of the circuit-guided surrogate on IMDB with Pythia-1.4B under GCG attack. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Heatmaps of layerwise surrogate pruning patterns on EnronSpam. 75% means pruning 25% [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Heatmaps of layerwise surrogate pruning patterns on PasswordMatch. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 24 linked inside Pith

  1. [1]

    B Additional Results B.1 More Results Table 5: Additional cross-dataset robustness results on Pythia-1.4B

    Overall, we estimate that the full project required on the order of several hundred GPU-hours, approximately 800 GPU-hours in total. B Additional Results B.1 More Results Table 5: Additional cross-dataset robustness results on Pythia-1.4B. We report clean accuracy (Acc, higher is better) and attack success rate (ASR, lower is better) under RandomToken and...

    2025

  2. [6]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495,

    2021

  3. [8]

    Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms.arXiv preprint arXiv:2403.17806,

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms.arXiv preprint arXiv:2403.17806,

    Pith/arXiv arXiv

  4. [9]

    Impact of positional encoding: Clean and adversarial rademacher complexity for transformers under in-context regression.arXiv preprint arXiv:2512.09275,

    Weiyi He and Yue Xing. Impact of positional encoding: Clean and adversarial rademacher complexity for transformers under in-context regression.arXiv preprint arXiv:2512.09275,

    arXiv

  5. [10]

    Scaling trends in language model robustness.arXiv preprint arXiv:2407.18213,

    Nikolaus Howe, Ian McKenzie, Oskar Hollinsworth, Michał Zajac, Tom Tseng, Aaron Tucker, Pierre-Luc Bacon, and Adam Gleave. Scaling trends in language model robustness.arXiv preprint arXiv:2407.18213,

    Pith/arXiv arXiv

  6. [11]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  7. [12]

    A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967,

    Pith/arXiv arXiv

  8. [13]

    Towards understanding jailbreak attacks in llms: A representation space analysis

    Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. Towards understanding jailbreak attacks in llms: A representation space analysis. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7067–7085,

    2024

  9. [19]

    Alignment-constrained dynamic pruning for llms: Identifying and preserving alignment-critical circuits.arXiv preprint arXiv:2511.07482,

    Dev Patel, Gabrielle Gervacio, Diekola Raimi, Kevin Zhu, Ryan Lagasse, Gabriel Grand, Ashwinee Panda, and Maheep Chaudhary. Alignment-constrained dynamic pruning for llms: Identifying and preserving alignment-critical circuits.arXiv preprint arXiv:2511.07482,

    arXiv

  10. [20]

    Attention sinks and compression valleys in llms are two sides of the same coin.arXiv preprint arXiv:2510.06477,

    Enrique Queipo-de Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in llms are two sides of the same coin.arXiv preprint arXiv:2510.06477,

    arXiv

  11. [21]

    A general framework to enhance fine-tuning-based llm unlearning

    11 Jie Ren, Zhenwei Dai, Xianfeng Tang, Hui Liu, Jingying Zeng, Zhen Li, Rahul Goutam, Suhang Wang, Yue Xing, and Qi He. A general framework to enhance fine-tuning-based llm unlearning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18464–18476,

    2025

  12. [22]

    Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496,

    Pith/arXiv arXiv

  13. [23]

    Latent adversarial training improves robustness to persistent harmful behaviors in llms.arXiv preprint arXiv:2407.15549,

    Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms.arXiv preprint arXiv:2407.15549,

    Pith/arXiv arXiv

  14. [24]

    Layer by layer: Uncovering hidden representations in language models.arXiv preprint arXiv:2502.02013,

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models.arXiv preprint arXiv:2502.02013,

    Pith/arXiv arXiv

  15. [25]

    Tensor trust: Interpretable prompt injection attacks from an online game.arXiv preprint arXiv:2311.01011,

    Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. Tensor trust: Interpretable prompt injection attacks from an online game.arXiv preprint arXiv:2311.01011,

    Pith/arXiv arXiv

  16. [26]

    Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248,

    Pith/arXiv arXiv

  17. [27]

    Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small.arXiv preprint arXiv:2211.00593,

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small.arXiv preprint arXiv:2211.00593,

    Pith/arXiv arXiv

  18. [28]

    Fast is better than free: Revisiting adversarial training

    Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994,

    Pith/arXiv arXiv 2001

  19. [29]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    Pith/arXiv arXiv

  20. [30]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.arXiv preprint arXiv:2309.10253,

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.arXiv preprint arXiv:2309.10253,

    Pith/arXiv arXiv

  21. [31]

    Adversarial training: A survey.arXiv preprint arXiv:2410.15042,

    Mengnan Zhao, Lihe Zhang, Jingwen Ye, Huchuan Lu, Baocai Yin, and Xinchao Wang. Adversarial training: A survey.arXiv preprint arXiv:2410.15042,

    Pith/arXiv arXiv

  22. [32]

    Prompt-driven llm safeguarding via directed representation optimization.arXiv preprint arXiv:2401.18018, 3,

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. Prompt-driven llm safeguarding via directed representation optimization.arXiv preprint arXiv:2401.18018, 3,

    Pith/arXiv arXiv

  23. [33]

    Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405, 2023a

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405, 2023a. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and...

    Pith/arXiv arXiv 2024

  24. [35]

    Hence ∥(I−P S)rt∥ ≤1 m0 Mt

    Forj /∈S, |r(la) t,j |= st,j |a(la) t,j (x)| ≤ st,j m0 , and thus ∥(I−P S)rt∥2 = X j /∈S r(la) t,j 2 ≤ 1 m2 0 X j /∈S s2 t,j = 1 m2 0 M 2 t . Hence ∥(I−P S)rt∥ ≤1 m0 Mt. Applying Lemma 1, ∂Z ∂ eH ≤L ϕLσ∥W1∥ Cattn BQBKBV√dm +B V . Therefore Gt ≤ c LϕLσ∥W1∥ m0 Cattn BQBKBV√dm +B V η Mt +o(η). Summing overt∈[K], we obtain KX t=1 Gt ≤ c LϕLσ∥W1∥ m0 Cattn BQBK...

    2020

  25. [2006]

    Pruning convolutional neural networks for resource efficient inference.arXiv preprint arXiv:1611.06440,

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference.arXiv preprint arXiv:1611.06440,

    Pith/arXiv arXiv

  26. [2011]

    Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083,

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083,

    Pith/arXiv arXiv

  27. [2017]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824,

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824,

    Pith/arXiv arXiv

  28. [2019]

    Softmax is 1/2-lipschitz: A tight bound across all ℓp norms.arXiv preprint arXiv:2510.23012,

    Pravin Nair. Softmax is 1/2-lipschitz: A tight bound across all ℓp norms.arXiv preprint arXiv:2510.23012,

    arXiv

  29. [2020]

    Jailbreaking leading safety- aligned llms with simple adaptive attacks.arXiv preprint arXiv:2404.02151,

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned llms with simple adaptive attacks.arXiv preprint arXiv:2404.02151,

    Pith/arXiv arXiv

  30. [2021]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  31. [2022]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652,

    Pith/arXiv arXiv

  32. [2023]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249,

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249,

    Pith/arXiv arXiv

  33. [2024]

    Mixat: Combining continuous and discrete adversarial training for llms.arXiv preprint arXiv:2505.16947,

    Csaba Dékány, Stefan Balauca, Robin Staab, Dimitar I Dimitrov, and Martin Vechev. Mixat: Combining continuous and discrete adversarial training for llms.arXiv preprint arXiv:2505.16947,

    arXiv

  34. [2025]

    Towards understanding safety alignment: A mechanistic perspective from safety neurons.arXiv preprint arXiv:2406.14144,

    Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Towards understanding safety alignment: A mechanistic perspective from safety neurons.arXiv preprint arXiv:2406.14144,

    arXiv

  35. [2026]

    Defending against unforeseen failure modes with latent adversarial training.arXiv preprint arXiv:2403.05030,

    Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training.arXiv preprint arXiv:2403.05030,

    Pith/arXiv arXiv