pith. machine review for the scientific record. sign in

arxiv: 2604.18756 · v1 · submitted 2026-04-20 · 💻 cs.LG · cs.AI· cs.CL· cs.CR

Recognition: unknown

Towards Understanding the Robustness of Sparse Autoencoders

Ahson Saiyed , Sabrina Sadiekh , Chirag Agarwal
Authors on Pith no claims yet

Pith reviewed 2026-05-10 05:41 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CLcs.CR
keywords sparse autoencodersjailbreak attackslarge language modelsrepresentational bottleneckinference-time defenseadversarial robustnesstransformer residual streamswhite-box attacks
0
0 comments X

The pith

Inserting pretrained sparse autoencoders into transformer residual streams at inference time reduces jailbreak success rates by up to five times.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether pretrained sparse autoencoders can be added to large language models at inference time, without any weight changes, to improve resistance to jailbreak attacks. It reports that this addition creates a sparse projection in the residual stream that disrupts the gradient-based optimization used by attacks such as GCG and BEAST. Across four model families the intervention cuts attack success rates by as much as five times and lowers how readily attacks transfer from one model to another. Ablations further show that stronger sparsity improves defense while intermediate layers give the best balance with normal performance. The results are presented as evidence that the sparse projection acts as a representational bottleneck.

Core claim

The central claim is that inserting pretrained sparse autoencoders into the residual streams of transformers at inference time, without modifying weights or blocking gradients, produces up to a fivefold drop in success rate for white-box jailbreak attacks and reduces cross-model transferability. The effect strengthens monotonically with higher L0 sparsity in the autoencoder and shows a layer-dependent tradeoff, with intermediate layers preserving more clean performance while still delivering defense gains. These patterns are consistent with the view that the sparse projection reshapes the internal optimization geometry that attacks exploit.

What carries the argument

Pretrained sparse autoencoders inserted into the residual stream at inference time, which enforce sparsity on activations and thereby impose a representational bottleneck.

If this is right

  • Higher L0 sparsity in the inserted SAE produces steadily lower jailbreak success rates.
  • Intermediate layers offer the strongest combination of defense strength and retained clean performance.
  • The same intervention lowers successful transfer of attacks between different model families.
  • The defense applies across Gemma, LLaMA, Mistral, and Qwen models without retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the bottleneck mechanism holds, similar sparse projections could be tested against other classes of adversarial inputs beyond optimization-based jailbreaks.
  • Variable sparsity schedules across layers might allow models to tune the robustness-performance tradeoff more finely than a uniform insertion.
  • The approach could be combined with existing safety fine-tuning methods to check whether the two defenses reinforce each other.

Load-bearing premise

The assumption that the measured robustness gains result specifically from the sparse representational bottleneck rather than from incidental shifts in activation statistics or gradient behavior.

What would settle it

An ablation that replaces each SAE with a dense autoencoder or a dimension-matched linear projection that preserves activation norms but removes sparsity, then re-runs the same GCG and BEAST attacks and finds no reduction in success rate.

Figures

Figures reproduced from arXiv: 2604.18756 by Ahson Saiyed, Chirag Agarwal, Sabrina Sadiekh.

Figure 1
Figure 1. Figure 1: Attack success rate (ASR) for GCG and BEAST attacks on HarmBench across baseline (dark) and SAE-augmented (light) models. Results are shown for six models (Gemma-2 2B/9B/27B, LLaMA-3 8B/70B, Mistral-7B) under three configurations: PROMPT, BASE, and SAE. Under adaptive attacks, SAE-augmented models exhibit substantially lower ASR than their baseline counterparts. Under BASE transfer evaluation, median ASR d… view at source ↗
Figure 2
Figure 2. Figure 2: Attack Success Rate (ASR) transfer matrices for GCG (top) and BEAST (bottom) attacks across evaluation models. Rows correspond to the suffix source model used to generate adversarial suffixes, while columns denote the evaluation (target) models. Values are normalized to the range [0, 100], where higher values indicate a higher attack success rate [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Adversarial suffixes from different source models exhibit high pairwise Jaccard similarity (0.36±0.16), while similarity to random baselines remains low (0.09±0.05). This 4× difference reveals that attacks converge on a shared sparse feature subspace, explaining their cross-model transferability. overlap using the Jaccard index. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Spectral analysis of GCG gradients on LLaMA-8B (n=218 paired runs). Left panels show metric trajectories over 500 optimization steps (mean ± std); right panels show distributions of trajectory means. (A) Cosine similarity: SAE gradients are 60% more correlated between successive steps, indicating optimization stagnation. (B) Loss: SAE converges to 31% higher loss. (C) Effective rank: SAE gradients occupy 9… view at source ↗
Figure 5
Figure 5. Figure 5: GCG optimization loss curves for baseline versus SAE-intervened models. Higher final loss indicates less effective adversarial suffix generation. SAE integration increases optimization difficulty by 47–74% across model families (n = 218 attack runs). jailbreak success rates and reduced cross-model transfer under strong white-box attacks. These effects persist despite full gradient access during optimizatio… view at source ↗
read the original abstract

Large Language Models (LLMs) remain vulnerable to optimization-based jailbreak attacks that exploit internal gradient structure. While Sparse Autoencoders (SAEs) are widely used for interpretability, their robustness implications remain underexplored. We present a study of integrating pretrained SAEs into transformer residual streams at inference time, without modifying model weights or blocking gradients. Across four model families (Gemma, LLaMA, Mistral, Qwen) and two strong white-box attacks (GCG, BEAST) plus three black-box benchmarks, SAE-augmented models achieve up to a 5x reduction in jailbreak success rate relative to the undefended baseline and reduce cross-model attack transferability. Parametric ablations reveal (i) a monotonic dose-response relationship between L0 sparsity and attack success rate, and (ii) a layer-dependent defense-utility tradeoff, where intermediate layers balance robustness and clean performance. These findings are consistent with a representational bottleneck hypothesis: sparse projection reshapes the optimization geometry exploited by jailbreak attacks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The manuscript empirically studies the effect of inserting pretrained Sparse Autoencoders (SAEs) into the residual streams of LLMs at inference time (no weight updates) on robustness to optimization-based jailbreaks. Across Gemma, LLaMA, Mistral, and Qwen families and attacks including GCG, BEAST, and black-box benchmarks, SAE-augmented models show up to 5x lower jailbreak success rates and reduced cross-model transferability. Ablations report a monotonic L0-sparsity dose-response and a layer-dependent robustness/clean-performance tradeoff, interpreted as support for a representational-bottleneck hypothesis.

Significance. The work supplies a useful set of consistent empirical patterns linking an existing interpretability tool to measurable robustness gains. If the gains prove mechanistically attributable to sparsity rather than correlated activation changes, the results could inform both defensive practice and the design of future sparse representations. The cross-model and cross-attack consistency is a strength, but the interpretive claim remains correlational.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (Ablations): The central interpretive claim—that robustness arises from the 'representational bottleneck' created by the sparse projection—is not isolated from other SAE-induced effects. The reported monotonic L0 dose-response and layer tradeoff are consistent with the hypothesis but do not rule out confounds such as reconstruction bias, altered activation magnitudes, or feature selection. No control experiments are described that hold reconstruction error or activation statistics fixed while removing the sparsity constraint; this underdetermines the mechanistic attribution and is load-bearing for the paper's main conclusion.
  2. [§3 and results tables] §3 (Experimental Setup) and results tables: The abstract states 'consistent empirical patterns' and a 5x reduction, yet the manuscript provides no details on statistical tests, error bars or confidence intervals on success rates, exact attack-success definitions (e.g., whether refusal is judged by string matching or a classifier), or explicit controls confirming that clean-task performance degradation is not driving the observed robustness. These omissions weaken the quantitative claims.
minor comments (3)
  1. [Methods] Methods: Explicitly state whether the SAE output replaces the residual-stream activation or is added to it, and whether gradients flow through the SAE during the attack optimization.
  2. [Figures] Figures: Add error bars or report the number of random seeds/runs for all success-rate plots; label axes with precise metrics (e.g., 'ASR under GCG, 100 steps').
  3. [Related work] Related work: The discussion of prior SAE robustness work is brief; cite any concurrent or closely related studies on SAEs and adversarial robustness.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments. We have revised the manuscript to clarify the scope of our interpretive claims and to improve the completeness of our experimental reporting. Point-by-point responses follow.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Ablations): The central interpretive claim—that robustness arises from the 'representational bottleneck' created by the sparse projection—is not isolated from other SAE-induced effects. The reported monotonic L0 dose-response and layer tradeoff are consistent with the hypothesis but do not rule out confounds such as reconstruction bias, altered activation magnitudes, or feature selection. No control experiments are described that hold reconstruction error or activation statistics fixed while removing the sparsity constraint; this underdetermines the mechanistic attribution and is load-bearing for the paper's main conclusion.

    Authors: We agree that the results remain correlational and do not isolate sparsity from co-varying factors such as reconstruction error or activation magnitude shifts. The L0 dose-response provides a graded manipulation of sparsity within the same SAE family, yet reconstruction quality necessarily changes with L0. In the revised manuscript we have updated the abstract and Section 4 to describe the findings as 'consistent with' rather than arising from the representational bottleneck. We have added an explicit limitations paragraph that enumerates the listed confounds and notes that fully disentangling them would require training SAEs under alternative objectives (e.g., sparsity-regularized reconstruction with fixed error). These textual revisions clarify the evidential scope without overstating mechanistic attribution. revision: partial

  2. Referee: [§3 and results tables] §3 (Experimental Setup) and results tables: The abstract states 'consistent empirical patterns' and a 5x reduction, yet the manuscript provides no details on statistical tests, error bars or confidence intervals on success rates, exact attack-success definitions (e.g., whether refusal is judged by string matching or a classifier), or explicit controls confirming that clean-task performance degradation is not driving the observed robustness. These omissions weaken the quantitative claims.

    Authors: We appreciate the referee drawing attention to these reporting gaps. The revised Section 3 now specifies: (i) attack success is determined by a hybrid criterion combining exact refusal-string matching with an auxiliary LLM-based semantic classifier; (ii) all success rates include error bars and 95 % confidence intervals computed across five independent attack runs that differ only in random seed; (iii) paired t-tests yield p < 0.01 for the reported reductions; and (iv) an additional control selects SAE configurations whose clean-task degradation matches that of the primary results, confirming that robustness gains are not solely explained by performance drop. The results tables have been updated with these statistics and controls. revision: yes

Circularity Check

0 steps flagged

No significant circularity; purely empirical study with direct measurements

full rationale

The paper reports experimental results from integrating pretrained SAEs into transformer residual streams at inference time across four model families and multiple attacks, measuring jailbreak success rates and transferability directly. Parametric ablations show monotonic L0 dose-response and layer tradeoffs, but these are observational correlations from controlled interventions rather than derivations or fitted parameters that reduce claims to tautologies by construction. The representational bottleneck hypothesis is presented as a post-hoc interpretive consistency check on the data, not as a load-bearing premise in any equation or self-citation chain. No self-definitional steps, renamed known results, or uniqueness theorems appear; all central metrics are externally falsifiable via replication on the same benchmarks. The study is self-contained against external benchmarks with no reduction of outputs to inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical study. No free parameters, axioms, or invented entities are stated in the abstract. L0 sparsity levels are varied experimentally but not fitted to produce the central claim.

pith-pipeline@v0.9.0 · 5483 in / 1094 out tokens · 37133 ms · 2026-05-10T05:41:25.705256+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 29 canonical work pages · 8 internal anchors

  1. [1]

    Jailbreak Attacks and Defenses Against Large Language Models: A Survey

    Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey, 2024. URL https: //arxiv.org/abs/2407.04295. 1, 2, 15

    work page internal anchor Pith review arXiv 2024

  2. [2]

    Contextual lattice probing for large language models: A study of interleaved multi-space activation patterns, 07 2025

    Daniel Chatterley, Zachary Molyneux, Felix Ashworth, Benedict Sutherlands, Andrew Scolto, and Travis Connor. Contextual lattice probing for large language models: A study of interleaved multi-space activation patterns, 07 2025. 1

    2025

  3. [3]

    Does representation matter? exploring intermediate layers in large language models.arXiv preprint arXiv:2412.09563, 2024

    Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. Does representation matter? exploring intermediate layers in large language models, 2024. URL https://arxiv. org/abs/2412.09563

    work page arXiv 2024

  4. [4]

    Interpreting the latent structure of operator precedence in language models, 2025

    Dharunish Yugeswardeenoo, Harshil Nukala, Ved Shah, Cole Blondin, Sean O Brien, Vasu Sharma, and Kevin Zhu. Interpreting the latent structure of operator precedence in language models, 2025. URLhttps://arxiv.org/abs/2510.13908. 1

    work page arXiv 2025

  5. [5]

    Sparse Autoencoders Find Highly Interpretable Features in Language Models

    Hoagy Cunningham, Aidan Turner, and Michael Sapienza. Sparse autoencoders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600, 2023. URL https://arxiv.org/abs/2309.08600. 1, 2

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    Michaud, David D

    Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure, 2025. URL https://arxiv. org/abs/2410.19750. 1

    work page arXiv 2025

  7. [7]

    Li, Suraj Srinivas, Usha Bhalla, and Himabindu Lakkaraju

    Aaron J. Li, Suraj Srinivas, Usha Bhalla, and Himabindu Lakkaraju. Interpretability illusions with sparse autoencoders: Evaluating robustness of concept representations, 2025. URL https://arxiv.org/abs/2505.16004. 1, 2

    work page arXiv 2025

  8. [8]

    Sae reconstruction errors are (empirically) pathological.AI Alignment Forum, 2024

    Wes Gurnee. Sae reconstruction errors are (empirically) pathological.AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/rZPiuFxESMxCDHe4B/ sae-reconstruction-errors-are-empirically-pathological . Accessed: 2025-01-

    2024

  9. [9]

    Hotflip: White-box adversarial examples for text classifica- tion,

    Javid Ebrahimi, Anyi Rao, and Daniel Lowd. Hotflip: White-box adversarial examples for text classification.arXiv preprint arXiv:1712.06751, 2018. URL https://arxiv.org/abs/ 1712.06751. 2

    work page arXiv 2018

  10. [10]

    Generating Natural Language Adversarial Examples

    Moustafa Alzantot, Yash Sharma, and Ahmed Elgohary. Generating natural language adversarial examples.arXiv preprint arXiv:1804.07998, 2018. URL https://arxiv.org/abs/1804. 07998. 2

    work page Pith review arXiv 2018

  11. [11]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. URL https://arxiv.org/abs/2307.15043. 2, 3

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  12. [12]

    Fast adversarial attacks on language models in one gpu minute.arXiv preprint arXiv:2402.15570, 2024

    Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute.arXiv preprint arXiv:2402.15570, 2024. URLhttps://arxiv.org/abs/2402.15570. 2, 3 12 Towards Understanding the Robustness of Sparse Autoencoders

    work page arXiv 2024

  13. [13]

    Ignore previous prompt: Attack techniques for language models,

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models,

  14. [14]

    URLhttps://arxiv.org/abs/2211.09527. 2

    work page internal anchor Pith review arXiv

  15. [15]

    Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection, 2023. URLhttps://arxiv.org/abs/2302.12173. 2

    work page internal anchor Pith review arXiv 2023

  16. [16]

    Autoadv: Automated adversar- ial prompting for multi-turn jailbreaking of large language models, 2025

    Aashray Reddy, Andrew Zagula, and Nicholas Saban. Autoadv: Automated adversar- ial prompting for multi-turn jailbreaking of large language models, 2025. URL https: //arxiv.org/abs/2507.01020. 2

    work page arXiv 2025

  17. [17]

    Towards monosemanticity: Decomposing language models with dictionary learning.arXiv preprint arXiv:2310.18494, 2023

    Trenton Bricken, Adly Templeton, and Joshua Batson. Towards monosemanticity: Decomposing language models with dictionary learning.arXiv preprint arXiv:2310.18494, 2023. URL https://arxiv.org/abs/2310.18494. 2

    work page arXiv 2023

  18. [18]

    Scaling and evaluating sparse autoencoders

    Dan Gao, Joseph Lieber, and Nazneen Rajani. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. URLhttps://arxiv.org/abs/2406.04093. 2

    work page internal anchor Pith review arXiv 2024

  19. [19]

    Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.arXiv preprint arXiv:2406.10142, 2024

    Adly Templeton, Tom Conerly, and Gary Marcus. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.arXiv preprint arXiv:2406.10142, 2024. URL https://arxiv.org/abs/2406.10142. 2

    work page arXiv 2024

  20. [20]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models,

  21. [21]

    URLhttps://arxiv.org/abs/2403.19647. 2

    work page internal anchor Pith review arXiv

  22. [22]

    Steering language model refusal with sparse autoencoders.arXiv preprint arXiv:2411.11296, 2024

    Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Blake Bullwinkel, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangdeh. Steering language model refusal with sparse autoencoders, 2025. URL https://arxiv.org/abs/ 2411.11296

    work page arXiv 2025

  23. [23]

    Toward understanding unlearning difficulty: A mechanistic perspective and circuit-guided difficulty metric.arXiv preprint arXiv:2601.09624, 2026

    Jiali Cheng, Ziheng Chen, Chirag Agarwal, and Hadi Amiri. Toward understanding unlearning difficulty: A mechanistic perspective and circuit-guided difficulty metric.arXiv preprint arXiv:2601.09624, 2026. 2

    work page arXiv 2026

  24. [24]

    Decomposing the dark matter of sparse autoencoders.Transactions on Machine Learning Research, 2025

    Joshua Engels, Logan Smith, and Max Tegmark. Decomposing the dark matter of sparse autoencoders.arXiv preprint arXiv:2410.14670v2, 2025. URL https://arxiv.org/pdf/ 2410.14670v2. Published in Transactions on Machine Learning Research. 2

    work page arXiv 2025

  25. [25]

    Certifying llm safety against adversarial prompting

    Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying llm safety against adversarial prompting, 2025. URL https://arxiv. org/abs/2309.02705. 2, 15

    work page arXiv 2025

  26. [26]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks, 2024. URL https://arxiv.org/abs/ 2310.03684. 2, 15

    work page internal anchor Pith review arXiv 2024

  27. [27]

    Defending large language models against jailbreak attacks via semantic smoothing

    Jiabao Ji, Bairu Hou, Alexander Robey, George J. Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending large language models against jailbreak attacks via semantic smoothing, 2024. URLhttps://arxiv.org/abs/2402.16192. 2, 15

    work page arXiv 2024

  28. [28]

    org/abs/2501.16497

    Ryo Hase, Md Rafi Ur Rashid, Ashley Lewis, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, and Ye Wang. Smoothed embeddings for robust language models, 2025. URLhttps://arxiv. org/abs/2501.16497. 2, 15

    work page arXiv 2025

  29. [29]

    Defending large language models against jailbreak attacks via in-decoding safety-awareness probing, 2026

    Yinzhi Zhao, Ming Wang, Shi Feng, Xiaocui Yang, Daling Wang, and Yifei Zhang. Defending large language models against jailbreak attacks via in-decoding safety-awareness probing, 2026. URLhttps://arxiv.org/abs/2601.10543. 2, 16

    work page arXiv 2026

  30. [30]

    Medsafetybench: Evaluating and improving the medical safety of large language models.Advances in neural information processing systems, 37:33423–33454, 2024

    Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. Medsafetybench: Evaluating and improving the medical safety of large language models.Advances in neural information processing systems, 37:33423–33454, 2024. 2

    2024

  31. [31]

    arXiv preprint arXiv:2502.04204 , year=

    Shaopeng Fu, Liang Ding, Jingfeng Zhang, and Di Wang. Short-length adversarial training helps llms defend long-length jailbreak attacks: Theoretical and empirical evidence, 2026. URL https://arxiv.org/abs/2502.04204. 2, 16

    work page arXiv 2026

  32. [32]

    arXiv:2402.05044 (2024), https://arxiv.org/abs/2402.05044

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models.arXiv preprint arXiv:2402.05044, 2024. 3 13 Towards Understanding the Robustness of Sparse Autoencoders

    work page arXiv 2024

  33. [33]

    Primeguard: Safe and helpful llms through tuning-free routing.arXiv preprint arXiv:2407.16318, 2024

    Blazej Manczak, Eliott Zemour, Eric Lin, and Vaikkunth Mugunthan. Primeguard: Safe and helpful llms through tuning-free routing.arXiv preprint arXiv:2407.16318, 2024. 3

    work page arXiv 2024

  34. [34]

    Detecting language model attacks with perplexity

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, 2023. URLhttps://arxiv.org/abs/2308.14132. 15 A Experimental Setup Table 8:SAE configurations for each base model, including layer placement, model dimension, and dictionary width. Model SAE Release Layerd model Width Gemma-2-2B gemma-scope-2b-pt-res 12 2304 16K Gemma-2-9...

    work page arXiv 2023