The reviewed record of science sign in
Pith

arxiv: 2606.18322 · v1 · pith:RCZWTACC · submitted 2026-06-16 · cs.LG · cs.AI

SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior

Mingyue Cui , Linghui Shen , Xingyi Yang This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-27 01:30 UTCgrok-4.3pith:RCZWTACCrecord.jsonopen to challenge →

classification cs.LG cs.AI
keywords Sparse AutoencodersModel InterventionsFeature ClampingResidual StreamPost-Intervention RecoveryRefusal SteeringSafety Interventions
0
0 comments X

The pith

Clamping SAE features does not guarantee suppression of model behaviors, as residual perturbations can recover the original outputs while preserving the clamped features.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that SAE-based interventions, which clamp specific features to block harmful behaviors, can be circumvented through a recovery process. It formulates post-intervention recovery as an optimization problem that finds residual perturbations restoring pre-intervention behavior without changing the targeted SAE feature values. This recovery succeeds even when the intervention remains active during optimization and generation, using constraints like encoder-orthogonal updates to ensure it is not mere reversal. Experiments on tasks including refusal steering show high recovery rates with low feature drift. The results indicate that SAE features capture only part of the mechanism for a behavior, leaving the reconstruction residual as a path for recovery.

Core claim

SAE feature clamping supports causal intervention but leaves a recoverable failure mode: starting from the post-intervention residual state, constrained optimization can recover pre-intervention behavior while preserving the post-intervention values of the targeted SAE features, even under active intervention, with recovery localizing to the SAE reconstruction residual.

What carries the argument

Post-intervention recovery formulated as a constrained residual-space optimization problem that restores pre-intervention behavior while enforcing preservation of targeted SAE feature values via encoder-orthogonal updates or feature-map Jacobian.

If this is right

  • In refusal steering, recovery reaches 95.8 percent on valid samples with defended-feature relative drift at 0.131.
  • Recovery is possible in TPP, unlearning, IOI, and refusal steering settings.
  • Recovery localizes to the component of the residual unexplained by the SAE.
  • The gap between feature control and behavioral control holds under strong threat models with active intervention.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Safety methods relying on single SAE features may need to address the reconstruction residual explicitly to close the recovery path.
  • Behaviors could be distributed such that multiple SAE features or their interactions must be controlled together.
  • Alternative decomposition techniques beyond standard SAEs might reduce the size of the unexplained residual.

Load-bearing premise

The encoder-orthogonal updates for single-layer cases and the feature-map Jacobian for cross-layer cases prevent recovery from simply undoing the intervention rather than finding an alternative path.

What would settle it

An experiment across the same tasks where no residual perturbation recovers the original behavior while keeping the clamped SAE features fixed and the intervention active throughout generation.

Figures

Figures reproduced from arXiv: 2606.18322 by Linghui Shen, Mingyue Cui, Xingyi Yang.

Figure 1
Figure 1. Figure 1: Post-intervention recovery framework. Starting from the defended residual state, we [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: TPP recovery-reactivation trade-off at layer 5. Each arrow maps unconstrained recovery to [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: WMDP-Bio unlearning recovery on the matched strict slice. Encoder-projected recovery [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: IOI recovery under a fixed SAE clamp. Both variants restore the IO-minus-S decision, but [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Refusal recovery–preservation trade-off on 24 strict-valid AdvBench prompts. Jacobian [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Recovery is carried by the SAE reconstruction residual. Replaying only the SAE residual nearly matches full recovery, while clamped-feature and non-clamped-feature replays largely fail, indicating that recovery is not explained by reopening visible SAE latents. 7 Discussion Causal handles are not complete bottlenecks. Our experiments separate two notions that are often conflated in SAE-based intervention w… view at source ↗
Figure 7
Figure 7. Figure 7: Supplementary TPP recovery summary across official layer-5 targets. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Budget and optimization diagnostics for unlearning recovery. We evaluate a small matched strict WMDP slice of six valid answer-choice flips under the same SAE clamp and post-hoc evaluator. (a) During optimization, encoder-projected recovery keeps choice-readout defended-feature drift at zero, while unconstrained recovery rapidly increases drift. (b) Recovery improves with the perturbation budget: encoder-p… view at source ↗
Figure 9
Figure 9. Figure 9: Refusal recovery across feature-set sizes. Left: broader SAE feature clamps affect more [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
read the original abstract

Sparse Autoencoders (SAEs) decompose residual-stream activations into interpretable features. Recent latent-space defenses increasingly rely on these decompositions, assuming that identified "unsafe" SAE features serve as actionable handles for monitoring and intervention. In this paradigm, clamping a specific harmful feature is expected to reliably prevent model misbehavior. However, we show that this success may hide a recoverable failure mode: the clamp may block one visible route to a behavior without eliminating the behavior itself. We formulate this vulnerability as post-intervention recovery, a constrained residual-space optimization problem. Starting from the post-intervention residual state, we optimize residual perturbations to recover the pre-intervention behavior while preserving the post-intervention values of the targeted SAE features. Even under a strong threat model where the intervention remains active throughout optimization and generation, recovery remains possible. To rule out that recovery simply undoes the intervention, we use encoder-orthogonal updates for single-layer interventions and the corresponding feature-map Jacobian in the cross-layer setting. Across TPP, unlearning, IOI, and refusal steering experiments, this stress test reveals recoverable behavior despite successful feature-level intervention. Especially in the safety-critical refusal-steering setting, we achieve a 95.8% recovery rate on valid samples while keeping defended-feature relative drift to 0.131, substantially below suffix-based baselines. A recovery-path attribution analysis further localizes this recovery to the SAE reconstruction residual, the component left unexplained by the SAE. These results expose a gap between feature-level control and behavioral completeness: SAE features can support causal intervention, but controlling them does not guarantee control over the underlying behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper claims that SAE-based feature interventions are unreliable for suppressing model behaviors because post-intervention recovery of the original behavior remains possible via constrained optimization of residual perturbations. Recovery is formulated to preserve the post-clamp values of the targeted SAE features while restoring pre-intervention behavior; encoder-orthogonal updates (single-layer) and the feature-map Jacobian (cross-layer) are used to rule out simple reversal of the intervention. Experiments on TPP, unlearning, IOI, and refusal steering report high recovery rates (e.g., 95.8% on valid refusal samples) with low defended-feature drift (0.131), and attribution analysis localizes recovery to the SAE reconstruction residual.

Significance. If the central empirical result holds under the stated constraints, the work identifies a substantive gap between SAE-feature control and behavioral control. This is relevant for safety-critical applications that rely on feature clamping, and the constrained-optimization plus attribution approach supplies a concrete, falsifiable test for whether an SAE decomposition is behaviorally complete. The low-drift recovery numbers and cross-task consistency would strengthen the case that current SAE interventions leave exploitable residual routes.

major comments (1)
  1. [Threat model and recovery formulation] Threat-model and recovery-formulation paragraphs: the encoder-orthogonal updates and feature-map Jacobian are presented as sufficient to ensure that optimized residuals cannot achieve recovery simply by partially reversing the original clamp. Because the SAE encoder is nonlinear, it is not shown that satisfaction of these (linear) constraints precludes effective reversal trajectories in the original activation space; if such trajectories exist, the reported recovery rates would not demonstrate an independent behavioral route. This premise is load-bearing for the headline conclusion.
minor comments (2)
  1. [Abstract / Methods] The abstract states that recovery occurs 'while keeping defended-feature relative drift to 0.131'; the precise definition of relative drift and the baseline against which it is measured should be stated explicitly in the methods section.
  2. [Results] The recovery-path attribution analysis is described only at a high level; a short paragraph or figure caption clarifying how the attribution isolates the reconstruction residual versus other components would improve reproducibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful reading and for identifying this important subtlety in our threat-model formulation. We respond to the single major comment below.

read point-by-point responses

  1. Referee: [Threat model and recovery formulation] Threat-model and recovery-formulation paragraphs: the encoder-orthogonal updates and feature-map Jacobian are presented as sufficient to ensure that optimized residuals cannot achieve recovery simply by partially reversing the original clamp. Because the SAE encoder is nonlinear, it is not shown that satisfaction of these (linear) constraints precludes effective reversal trajectories in the original activation space; if such trajectories exist, the reported recovery rates would not demonstrate an independent behavioral route. This premise is load-bearing for the headline conclusion.

    Authors: The referee correctly notes that the encoder is nonlinear and that our auxiliary linear constructions (encoder-orthogonal updates, feature-map Jacobian) do not by themselves constitute a complete proof against all conceivable reversal trajectories. However, the primary and nonlinear constraint in the optimization is the explicit requirement that the targeted SAE feature values—i.e., the direct output of the nonlinear encoder—remain exactly at their post-clamp levels. This constraint is enforced at every step and is verified after optimization by the reported low relative drift (0.131). Any trajectory that reversed the original clamp would necessarily change those feature values and would therefore be rejected by the optimizer. The linear constructions are used only to identify feasible search directions that satisfy the nonlinear feature-preservation constraint; they are not claimed to be a standalone guarantee. We will add a clarifying paragraph in the revised threat-model section that distinguishes the auxiliary linear methods from the primary nonlinear feature-value constraint and will report an additional diagnostic that measures how much the optimization would have to violate the feature constraint to achieve full reversal. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical optimization results with no derivation chain

full rationale

The paper reports results from constrained residual-space optimization experiments that recover pre-intervention behavior while holding targeted SAE features fixed. The encoder-orthogonal updates and feature-map Jacobian are presented as methodological constraints within the threat model to isolate genuine recovery paths, not as outputs of any derivation that reduces to fitted inputs or self-referential definitions. No equations or first-principles claims are shown to be equivalent to their inputs by construction, no load-bearing self-citations justify uniqueness theorems, and the work contains no predictions that are statistically forced by parameter fitting. The central claims rest on reported recovery rates (e.g., 95.8%) and drift metrics obtained from external experimental benchmarks, making the analysis self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper relies on the existence of an SAE decomposition and on the solvability of a constrained residual optimization; no free parameters, new entities, or non-standard axioms are introduced in the abstract.

axioms (1)
  • domain assumption A constrained optimization over residual perturbations can locate points that restore pre-intervention behavior while exactly preserving the values of targeted SAE features.
    This is the core premise that allows the recovery experiments to be interpreted as evidence of incomplete behavioral control.

pith-pipeline@v0.9.1-grok · 5821 in / 1406 out tokens · 44980 ms · 2026-06-27T01:30:48.505408+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Saes are good for steering – if you select the right features, 2025

    Dana Arad, Aaron Mueller, and Yonatan Belinkov. Saes are good for steering – if you select the right features, 2025

    2025

  2. [2]

    Obfuscated activations bypass llm latent-space defenses, 2024

    Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated activations bypass llm latent-space defenses, 2024

    2024

  3. [3]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. To...

    2023

  4. [4]

    A is for absorption: Studying feature splitting and absorption in sparse autoen- coders, 2024

    David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoen- coders, 2024

    2024

  5. [5]

    Feature hedging: Correlated features break narrow sparse autoencoders, 2025

    David Chanin, Tomáš Dulka, and Adrià Garriga-Alonso. Feature hedging: Correlated features break narrow sparse autoencoders, 2025

    2025

  6. [6]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023

    2023

  7. [7]

    Toy models of superposition, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022

    2022

  8. [8]

    Applying sparse autoencoders to unlearn knowledge in language models, 2024

    Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models, 2024

    2024

  9. [9]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024

    2024

  10. [10]

    Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability, 2025

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability, 2025

    2025

  11. [11]

    Inference- time intervention: Eliciting truthful answers from a language model, 2023

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model, 2023

    2023

  12. [12]

    Michaud, David D

    Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure.Entropy, 27(4):344, 2025

    2025

  13. [13]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    2024

  14. [14]

    Towards principled evaluations of sparse autoencoders for interpretability and control, 2024

    Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control, 2024

    2024

  15. [15]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024

    2024

  16. [16]

    Steering language model refusal with sparse autoencoders, 2024

    Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Blake Bullwinkel, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangdeh. Steering language model refusal with sparse autoencoders, 2024

    2024

  17. [17]

    Steering llama 2 via contrastive activation addition, 2024

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024. 11

    2024

  18. [18]

    Metamorphictestingoflarge languagemodelsfornaturallanguageprocessing.doi:10.48550/arXiv

    Leheng Sheng, Changshuo Shen, Weixiang Zhao, Junfeng Fang, Xiaohao Liu, Zhenkai Liang, Xiang Wang, An Zhang, and Tat-Seng Chua. Alphasteer: Learning refusal steering with principled null-space constraint.arXiv preprint arXiv:2506.07022, 2025. doi: 10.48550/arXiv. 2506.07022

    work page internal anchor Pith review doi:10.48550/arxiv 2025

  19. [19]

    Latentguard: Controllable latent steering for robust refusal of attacks and reliable response generation, 2025

    Huizhen Shu. Latentguard: Controllable latent steering for robust refusal of attacks and reliable response generation, 2025

    2025

  20. [20]

    Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Bat- son, Adam Jermy...

    2024

  21. [21]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2023

    2023

  22. [22]

    Interpretability in the wild: A circuit for indirect object identification in gpt-2 small, 2022

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in gpt-2 small, 2022

    2022

  23. [23]

    Understanding refusal in language models with sparse autoencoders, 2025

    Wei Jie Yeo, Nirmalendu Prakash, Clement Neo, Roy Ka-Wei Lee, Erik Cambria, and Ranjan Satapathy. Understanding refusal in language models with sparse autoencoders, 2025

    2025

  24. [24]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to a...

    2023

  25. [25]

    I cannot

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, and collaborators. Improving alignment and robustness with circuit breakers, 2024. 12 Metric Definition Interpretation Valid flipsB(M(x)) = 1, B(M S,c(x)) = 0Defense-succeeded cases Recovery rateE x∈V1[B(MrecS,c(x)) = 1]Behavior restored Non-refusal recovery1[recovered output has no refusal/safety-cue openin...

    2024