Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Activation steering works best when the signal is read from the instant before the model acts, not from text that already shows the behavior.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:55 UTC pith:Q3VOAIE4

load-bearing objection Source choice is the real story here, and tail subtraction is worth stealing; the 'execution-boundary' interpretation is plausible but the position/source-text confound means the theory is not settled. the 3 major comments →

arxiv 2607.25270 v1 pith:Q3VOAIE4 submitted 2026-07-28 cs.CL cs.AIcs.LG

Where Steering Signals Come From: Activation Source Selection in Activation Steering

Jiaran Ye , Lingxu Ran , Zijun Yao , Chenpeng Wang , Yong Jiang , Lei Hou , Juanzi Li , Liangming Pan This is my paper
classification cs.CL cs.AIcs.LG
keywords activation steeringactivation source selectionexecution-boundary statespost-realization tracestail subtractionhidden state readoutinstruction-tuned language modelsinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks where the hidden states used to steer a language model should come from. Holding the downstream intervention fixed, it shows that changing only the source activations—what text elicits them and which positions are read—substantially changes steering success. The central claim is that strong signals come from execution-boundary states, the hidden state at the final token just before the model produces or continues the target behavior, rather than from states that already contain the target text. Answer-based sources are weak on their own; when they work, their useful component aligns with an execution-boundary direction. The paper introduces tail subtraction, subtracting a matched continuation state to isolate the boundary signal, and reports consistent gains across three instruction-tuned models and four task families.

Core claim

The paper's central claim is a pre-/post-realization distinction. For a source instance (p,a), define the execution-boundary state b_l(z) = h_{l,|p|}(p), the hidden state read immediately before the target-bearing continuation, and a post-realization trace r_{l,k}(z) = h_{l,|p|+k}(p ⊕ a_{1:k}) after the target has appeared. Across entity, persona, refusal, and nonsense tasks, boundary states read with a last-token policy steer most reliably; answer-only post-realization sources are among the weakest. In relation-completion tasks, where target appearance and continuation semantics are unentangled, all three boundary sources far outperform the answer-bearing mean source, and projecting out the

What carries the argument

Execution-boundary states: the hidden state h_{ℓ,|p|}(p) read at the last token of an open context p, immediately before the target-bearing continuation is generated. The paper contrasts these with post-realization trace states read after the target has appeared, and with mixed states produced by averaging across answer positions. The last-token readout policy is the instrument that selects the boundary, and tail subtraction—subtracting a matched tail-only state (same final question and assistant prefix, no target conditioning) from the boundary state—is the construction that isolates a cleaner boundary signal. These devices carry the argument by showing that position relative to imminent pr

Load-bearing premise

The execution-boundary interpretation rests on hand-built prompts whose assistant prefix stops just before the target behavior; if those prompts simply induce stronger instruction following at any readout position, the special status of the boundary position—rather than better-specified source text—carries the result.

What would settle it

Take one target instruction and one final user question, and vary only the readout position within the same prompt: if steering success is equal at the last token before the answer and at an earlier or later token, the execution-boundary theory is false. The paper's own Table 17 shows sequence-mean readout sometimes competing with last-token readout for ICL sources, so a position-vs-content ablation is the decisive test.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Steering pipelines should read source activations at a pre-production boundary position rather than averaging over target-bearing completions.
  • Answer-based sources that appear to work do so because they contain an execution-continuation component; removing that component measurably weakens them.
  • Subtracting a matched tail-only state from boundary states improves over positive-only and ordinary contrastive baselines across models and construction methods.
  • Source context and readout policy must be treated as first-class variables in activation steering; the effect persists when hyperparameters are selected on held-out questions.
  • Relation-completion tasks provide a clean diagnostic separating pre-realization from post-realization sources, with boundary sources outperforming answer-bearing mean sources on all three models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension: automatically locate execution boundaries (e.g., by next-token prediction confidence or a probe) instead of hand-built prompts; the theory predicts automatically found boundaries match hand-built ones.
  • If the boundary account is right, it connects to the broader principle that models commit to a behavior before the behavior is visible; the same readout position may improve other interventions such as refusal directions or feature amplification.
  • Tail subtraction suggests a diagnostic for steering failure: collapses and flooding may be caused by post-realization continuation noise, and subtracting matched tails should reduce them in deployed settings.
  • The mixed-state account predicts that as tasks get longer or more compositional, the useful readout position shifts later—right before each continuation step—so per-step boundary readout could outperform a single fixed boundary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes "activation source selection" as the upstream design choice in activation steering: the combination of source context (text used to elicit activations) and readout policy (which hidden states are aggregated). Holding the downstream intervention fixed, the authors compare source conditions across three instruction-tuned models and four task families. They report that source selection substantially changes steering success, that prompt-only final-token sources are strongest, and that answer-only sources are weak even though they contain the target behavior. They then define execution-boundary states b_ℓ(z) read immediately before a target-bearing continuation and post-realization trace states r_{ℓ,k}(z) read after target tokens have appeared. Refined "boundary" sources (instruction, in-context learning, hybrid) with last-token readout outperform post-realization baselines, and a relation-completion task shows a similar advantage. The paper also introduces "tail subtraction," which subtracts a matched tail-only state from each boundary state to isolate target-preparation signal, reporting improved steering across models and methods.

Significance. If the central interpretation is established, the paper makes a practical and conceptual contribution: it would show that activation steering should read source activations at the pre-production boundary rather than from already-realized target text, and it offers a simple construction (tail subtraction) that improves steering reliability. The empirical work is broad: three models, four task families, five vector-construction methods, held-out hyperparameter selection (Appendix D), and automatic-judge validation. The paper is generally careful to separate source selection from downstream intervention, and the held-out analysis strengthens the claim that the source-condition effect is not merely a test-set artifact.

major comments (3)
  1. [§5.2, Fig. 2, Table 17] The central comparison between boundary sources and post-realization baselines is confounded: the boundary conditions (instruction/icl/hybrid) differ from answer-only and prompt-and-answer conditions in both readout position and source content. The boundary prompts contain an instruction and/or demonstrations and an open prefix designed to make the target continuation likely; the baselines contain neither. The paper's position control (Table 17) compares last-token vs sequence-mean readout within the same boundary prompts; it does not hold source text constant across the pre-/post-realization divide, and for icl the last-token advantage reverses on two models. A position-matched control—e.g., reading the same natural assistant response at the token immediately before the target is generated versus one or more tokens after—is needed to support the claim that pre-production state, rather t
  2. [§6.3, Table 5] The projection ablation assumes that the boundary Diff-PCA direction is specifically 'execution-relevant.' Without a control direction—for example, a boundary-derived direction from a matched tail-only or an unrelated task—the drop in Table 5 could equally be explained by removing a generic instruction-following component. The relation-task dissociation in §6.2 inherits the same source-text confound, so it does not independently resolve the boundary-vs-instruction ambiguity.
  3. [§7.2, Table 7] Tail subtraction relies on hand-constructed tail-only prompts that remove target conditioning while keeping the local query and assistant prefix. The diagnostic in Table 6 shows high cosine similarity between full and tail states, but this similarity may largely reflect the shared question/prefix rather than 'local continuation semantics.' The practical gains are plausible, but the interpretation that this isolates a phase-specific execution signal needs a more controlled comparison, e.g., subtracting a matched unrelated prompt with the same prefix but different local semantics.
minor comments (5)
  1. [Table 2] The check-marks table is hard to parse; labeling each condition explicitly (e.g., 'Prompt-only, Last' vs 'Prompt-only, Mean') would improve readability.
  2. [Fig. 2] The numbers on bars are small and sometimes overlap; consider annotating with values in a table or enlarging the font.
  3. [Appendix D] 'Paired hierarchical uncertainty intervals' is not defined; please specify the model and how the intervals were computed.
  4. [§3.3] The SAE construction differs from 'feature steering' in prior work; the paper should state explicitly that the SAE is used only as a source of candidate vectors, not as a reconstruction-based intervention, which is done later in Appendix C but could be moved earlier.
  5. [References] Several references are dated 2026 (e.g., You et al., Spherical Steering), which is fine for an arXiv preprint but should be verified as published or clearly marked as preprints.

Circularity Check

0 steps flagged

No circularity found; the empirical derivation is self-contained, with held-out evaluation and hypothesis-driven ablations.

full rationale

The paper does not derive its central claim from its definitions by construction. The execution-boundary state is defined positionally in Section 5.1 as b_l(z)=h_{l,|p|}(p), i.e., the hidden state immediately before a target-bearing continuation, and then tested against post-realization trace states; steerability is measured, not assumed, from this definition. The refined boundary prompts are constructed to make the target continuation imminent, but the outcome variable (steering success on held-out questions) is independent of the source construction, and Appendix D shows the main source-condition gains persist when source layer, intervention layer, and strength are selected only on validation questions. Tail subtraction (Section 7.2) is a motivated construction whose benefit is measured out-of-sample, and the projection ablation in Table 5 removes a boundary-derived direction from answer-based vectors and measures a drop, rather than fitting a parameter to reproduce the target. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed known results that function as the argument. The main residual concern raised by the skeptic is construct validity: boundary prompts differ from post-realization baselines in both source text and readout position, so the advantage could partly reflect better-specified source text. That is a confounding/interpretation issue, not a circularity-by-construction issue, and the paper partially addresses it with the last-token vs. sequence-mean readout comparison (Table 17) and the matched tail subtraction diagnostic. The paper's own Limitations section honestly discloses the best-over-grid protocol and reliance on an automatic judge, but these are evaluation caveats rather than circular reasoning. Accordingly, no specific circular step can be quoted, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The paper contributes a measurement and an interpretation, not a derivation. Its claims rest on standard activation-steering practice (additive residual-stream vectors transfer behavior), the domain assumption that the last pre-answer token state is an 'execution boundary', an automatic judge assumed to track human labels, and hand-built boundary prompts that encode the theory. Free parameters (steering strength, layer choices, demo/source counts) are fitted to evaluation data or chosen by hand; none of the headline conclusions are parameter-free derivations.

free parameters (5)
  • Steering strength α = grid maximum, per method/model/condition; values not reported
    Swept in 'method- and model-appropriate ranges' and the best score is retained (Sections 3.4, 4.2). The headline scores are the result of this fit.
  • Source layer ℓs and intervention layer ℓi = best of roughly four evenly spaced depths per model
    Selected per condition by grid search on the evaluation data (Section 3.4); Appendix D mitigates with validation-only selection for a subset.
  • In-context demonstration count = 3
    Hand-chosen for icl and hybrid sources (Section 5.2); no ablation on demo count.
  • Source prompt count = 16
    Hand-chosen for vector construction (Appendix A); no sensitivity analysis.
  • SAE consensus-feature selection threshold = unspecified
    Features 'consistently activated' across a batch are selected (Appendix C); the exact threshold and feature count are not given.
axioms (5)
  • domain assumption Additive residual-stream steering: adding direction v to hidden states transfers the behavior v encodes (Eq. 1 makes this the fixed intervention).
    The entire success metric presupposes that additive activation steering works as a behavioral control; the paper holds this fixed rather than deriving it (Section 2.1).
  • domain assumption The last-token hidden state at |p| represents imminent production of the continuation (execution-boundary state b_ℓ(z) = h_{ℓ,|p|}(p)).
    Defined in Section 5.1 and instantiated in all boundary-source experiments; if this position does not carry 'about-to-do' semantics, the interpretation collapses.
  • domain assumption Qwen2.5-14B-Instruct judge labels (target expression + generation validity) match human labels on unvalidated outputs.
    Validated at 87.0% agreement on n=400 (Table 13); a Gemma-3-27B rescore preserves the main pattern (Table 14), but the full behavioral evaluation still uses only automatic labels.
  • ad hoc to paper The hand-built open prompt–prefix pairs stop 'just before the target behavior would be realized or continued'.
    This construction (Section 5.2, Appendix A) embeds the paper's own boundary theory in the stimulus; a different theory of what those prefixes do would change the interpretation.
  • domain assumption In relation tasks, the model at the boundary is genuinely about to execute the intended mapping (e.g., big→small).
    Required for Figure 3 to be a clean dissociation between target-appearance and execution-continuation (Section 6.2).
invented entities (3)
  • Execution-boundary state b_ℓ(z) independent evidence
    purpose: Theoretical construct claimed to carry target-preparation signal in steering; defined as h_{ℓ,|p|}(p) (Section 5.1); motivates tail subtraction and the projection ablation.
    Has falsifiable handles inside the paper (position sensitivity Table 17, ablation Table 5, relation-task split Figure 3), but all handles are produced by the same authors; no external verification.
  • Post-realization trace state r_{ℓ,k}(z) no independent evidence
    purpose: Contrast construct for states after the target has appeared; used to define the weak-source condition.
    Defined contrastively in Section 5.1; its predictive power is only exercised internally.
  • Tail-only state h_tail no independent evidence
    purpose: Matched local-continuation reference subtracted in tail subtraction (Section 7.2).
    A procedural baseline construct; no independent evidence beyond the reported gains.

pith-pipeline@v1.3.0-alltime-deepseek · 124 in / 18984 out tokens · 228038 ms · 2026-08-01T02:55:50.829331+00:00 · methodology

0 comments
read the original abstract

Activation steering controls language models by adding vectors or features to hidden states at inference time, but the upstream source of these steering signals is often treated as a secondary detail. We study this source choice as activation source selection: the combination of source context and activation readout policy used to collect the hidden states from which a steering signal is built. Holding the downstream intervention fixed, we show across three instruction-tuned models and four steering task families that changing only the source activations substantially changes steering success. We further find that effective steering is not explained simply by whether the desired behavior appears in the source text. Instead, strong signals come from execution-boundary states, where the model is about to produce or continue the target behavior. This pre-/post-realization distinction explains why answer-based sources sometimes work: their useful component aligns with execution-boundary directions rather than target appearance alone. Building on this view, we introduce tail subtraction, which removes shared prompt and continuation semantics from boundary states and yields cleaner, more stable steering signals. Overall, our results suggest that steering depends on representations of what the model is about to do, not merely on what has already appeared.

Figures

Figures reproduced from arXiv: 2607.25270 by Chenpeng Wang, Jiaran Ye, Juanzi Li, Lei Hou, Liangming Pan, Lingxu Ran, Yong Jiang, Zijun Yao.

Figure 1
Figure 1. Figure 1: Overview of our activation source view. With the downstream steering pipeline fixed, different source [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Refined semantic source comparison. The first three conditions are boundary sources with last-token [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relation-task source comparison. Boundary [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 15 linked inside Pith

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [9]

    Kenneth Li and Oam Patel and Fernanda B. Vi. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , booktitle =. 2023 , url =

    2023

  9. [11]

    The Thirteenth International Conference on Learning Representations,

    Alessandro Stolfo and Vidhisha Balachandran and Safoora Yousefi and Eric Horvitz and Besmira Nushi , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  10. [12]

    The Thirteenth International Conference on Learning Representations,

    Weixuan Wang and Jingyuan Yang and Wei Peng , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  11. [16]

    Advances in Neural Information Processing Systems , volume =

    Andy Arditi and Oscar Obeso and Aaquib Syed and Daniel Paleka and Nina Panickssery and Wes Gurnee and Neel Nanda , title =. Advances in Neural Information Processing Systems , volume =. 2024 , url =

    2024

  12. [18]

    Turner and Callum McDougall and Monte MacDiarmid and C

    Adly Templeton and Tom Conerly and Jonathan Marcus and Jack Lindsey and Trenton Bricken and Brian Chen and Adam Pearce and Craig Citro and Emmanuel Ameisen and Andy Jones and Hoagy Cunningham and Nicholas L. Turner and Callum McDougall and Monte MacDiarmid and C. Daniel Freeman and Theodore R. Sumers and Edward Rees and Joshua Batson and Adam Jermyn and S...

  13. [24]

    Can Activation Steering Generalize Across Languages?

    Gabriele Maraia and Leonardo Ranaldi and Marco Valentino and Fabio Massimo Zanzotto , editor =. Can Activation Steering Generalize Across Languages?. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics,. 2026 , url =

    2026

  14. [25]

    Mitigating Content Effects on Reasoning in Language Models Through Fine-Grained Activation Steering , booktitle =

    Marco Valentino and Geonhee Kim and Dhairya Dalal and Zhixue Zhao and Andr. Mitigating Content Effects on Reasoning in Language Models Through Fine-Grained Activation Steering , booktitle =. 2026 , url =

    2026

  15. [26]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Nina Rimsky and Nick Gabrieli and Julian Schulz and Meg Tong and Evan Hubinger and Alexander Matt Turner , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , url =

    2024

  16. [27]

    Jentzsch and Diaoul

    Kai Konen and Sophie F. Jentzsch and Diaoul. Style Vectors for Steering Generative Large Language Models , booktitle =. 2024 , url =

    2024

  17. [30]

    Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 , booktitle =

    Tom Lieberum and Senthooran Rajamanoharan and Arthur Conmy and Lewis Smith and Nicolas Sonnerat and Vikrant Varma and J. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 , booktitle =. 2024 , url =

    2024

  18. [36]

    Lee and Inkit Padhi and Karthikeyan Natesan Ramamurthy and Erik Miehling and Pierre L

    Bruce W. Lee and Inkit Padhi and Karthikeyan Natesan Ramamurthy and Erik Miehling and Pierre L. Dognin and Manish Nagireddy and Amit Dhurandhar , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  19. [39]

    The Twelfth International Conference on Learning Representations,

    Robert Huben and Hoagy Cunningham and Logan Riggs Smith and Aidan Ewart and Lee Sharkey , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  20. [40]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =

    Sewon Min and Xinxi Lyu and Ari Holtzman and Mikel Artetxe and Mike Lewis and Hannaneh Hajishirzi and Luke Zettlemoyer , title =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =. 2022 , doi =

    2022

  21. [41]

    Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 , year =

    Shivam Garg and Dimitris Tsipras and Percy Liang and Gregory Valiant , title =. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 , year =

    2022

  22. [42]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    Roee Hendel and Mor Geva and Amir Globerson , title =. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =. 2023 , doi =

    2023

  23. [43]

    Li and Arnab Sen Sharma and Aaron Mueller and Byron C

    Eric Todd and Millicent L. Li and Arnab Sen Sharma and Aaron Mueller and Byron C. Wallace and David Bau , title =. International Conference on Learning Representations , year =

  24. [46]

    Manning and Christopher Potts , editor =

    Zhengxuan Wu and Aryaman Arora and Atticus Geiger and Zheng Wang and Jing Huang and Dan Jurafsky and Christopher D. Manning and Christopher Potts , editor =. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders , booktitle =. 2025 , url =

    2025

  25. [47]

    Zhang, and James Y

    Abubakar Abid, Vivek Kumar Bagaria, Martin J. Zhang, and James Y. Zou. 2017. https://arxiv.org/abs/1709.06716 Contrastive principal component analysis . CoRR, abs/1709.06716

    Pith/arXiv arXiv 2017

  26. [48]

    Guillaume Alain and Yoshua Bengio. 2016. https://arxiv.org/abs/1610.01644 Understanding intermediate layers using linear classifier probes . CoRR, abs/1610.01644

    Pith/arXiv arXiv 2016

  27. [49]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. https://doi.org/10.52202/079017-4322 Refusal in language models is mediated by a single direction . Advances in Neural Information Processing Systems, 37

    doi:10.52202/079017-4322 2024

  28. [50]

    Yonatan Belinkov and James R. Glass. 2019. https://doi.org/10.1162/TACL\_A\_00254 Analysis methods in neural language processing: A survey . Trans. Assoc. Comput. Linguistics, 7:49--72

    doi:10.1162/tacl 2019

  29. [51]

    Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. https://doi.org/10.48550/ARXIV.2411.02193 Improving steering vectors by targeting sparse autoencoder features . CoRR, abs/2411.02193

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.02193 2024

  30. [52]

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. https://doi.org/10.48550/ARXIV.2507.21509 Persona vectors: Monitoring and controlling character traits in language models . CoRR, abs/2507.21509

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.21509 2025

  31. [53]

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/c529dba08a146ea8d6cf715ae8930cbe-Abstract-Conference.html What can transformers learn in-context? A case study of simple function classes . In Advances in Neural Information Processing Systems 35: Annual Conference on Neural In...

    2022

  32. [54]

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu - Gang Jiang, and Xipeng Qiu. 2024. https://doi.org/10.48550/ARXIV.2410.20526 Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders . CoRR, abs/2410.20526

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.20526 2024

  33. [55]

    Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. https://doi.org/10.48550/ARXIV.2502.11356 SAIF: A sparse autoencoder framework for interpreting and steering instruction following of language models . CoRR, abs/2502.11356

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.11356 2025

  34. [56]

    Roee Hendel, Mor Geva, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.624 In-context learning creates task vectors . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318--9333

    doi:10.18653/v1/2023.findings-emnlp.624 2023

  35. [57]

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. https://openreview.net/forum?id=F76bwRSLeK Sparse autoencoders find highly interpretable features in language models . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  36. [58]

    Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. 2023. https://doi.org/10.48550/ARXIV.2312.03813 Improving activation steering in language models with mean-centring . CoRR, abs/2312.03813

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.03813 2023

  37. [59]

    Jentzsch, Diaoul \' e Diallo, Peer Sch \"u tt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking

    Kai Konen, Sophie F. Jentzsch, Diaoul \' e Diallo, Peer Sch \"u tt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. 2024. https://doi.org/10.18653/v1/2024.findings-eacl.52 Style vectors for steering generative large language models . In Findings of the Association for Computational Linguistics: EACL 2024, pages 782--802. Association for...

    doi:10.18653/v1/2024.findings-eacl.52 2024

  38. [60]

    Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L

    Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre L. Dognin, Manish Nagireddy, and Amit Dhurandhar. 2025. https://openreview.net/forum?id=Oi47wc10sm Programming refusal with conditional activation steering . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . Open...

    2025

  39. [61]

    Vi \' e gas, Hanspeter Pfister, and Martin Wattenberg

    Kenneth Li, Oam Patel, Fernanda B. Vi \' e gas, Hanspeter Pfister, and Martin Wattenberg. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html Inference-time intervention: Eliciting truthful answers from a language model . In Advances in Neural Information Processing Systems 36: Annual ...

    2023

  40. [62]

    Yichen Li, Zhiting Fan, Ruizhe Chen, Xiaotang Gai, Luqi Gong, Yan Zhang, and Zuozhu Liu. 2025. https://doi.org/10.18653/v1/2025.findings-acl.589 FairSteer: inference time debiasing for LLMs with dynamic activation steering . In Findings of the Association for Computational Linguistics: ACL 2025 , pages 11293--11312. Association for Computational Linguistics

    doi:10.18653/v1/2025.findings-acl.589 2025

  41. [63]

    Dragan, Rohin Shah, and Neel Nanda

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \' a nos Kram \' a r, Anca D. Dragan, Rohin Shah, and Neel Nanda. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.19 Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2 . In Proceedings of the 7th BlackboxNLP Workshop: Analyzing a...

    doi:10.18653/v1/2024.blackboxnlp-1.19 2024

  42. [64]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048--11064

    doi:10.18653/v1/2022.emnlp-main.759 2022

  43. [65]

    Joris Postmus and Steven Abreu. 2024. https://doi.org/10.48550/ARXIV.2410.16314 Steering large language models using conceptors: Improving addition-based activation engineering . CoRR, abs/2410.16314

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.16314 2024

  44. [66]

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.828 Steering llama 2 via contrastive activation addition . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504--15522. Association for Com...

    doi:10.18653/v1/2024.acl-long.828 2024

  45. [67]

    Marco Scialanga, Thibault Laugel, Vincent Grari, and Marcin Detyniecki. 2025. https://doi.org/10.18653/v1/2025.acl-long.777 SAKE: steering activations for knowledge editing . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15966--15978. Association for Computational Linguistics

    doi:10.18653/v1/2025.acl-long.777 2025

  46. [68]

    Niklas Stoehr, Kevin Du, V \' e steinn Sn bjarnarson, Robert West, Ryan Cotterell, and Aaron Schein. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-EMNLP.479 Activation scaling for steering and interpreting language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024 , Findings of A...

    doi:10.18653/v1/2024.findings-emnlp.479 2024

  47. [69]

    Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2025. https://openreview.net/forum?id=wozhdnRCtw Improving instruction-following in language models through activation steering . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

    2025

  48. [70]

    Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, and Zhaoxiang Zhang. 2025. https://doi.org/10.18653/v1/2025.acl-long.634 Activation steering decoding: Mitigating hallucination in large vision-language models through bidirectional hidden state intervention . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist...

    doi:10.18653/v1/2025.acl-long.634 2025

  49. [71]

    Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. 2022. https://doi.org/10.18653/v1/2022.findings-acl.48 Extracting latent steering vectors from pretrained language models . In Findings of the Association for Computational Linguistics: ACL 2022 , pages 566--581. Association for Computational Linguistics

    doi:10.18653/v1/2022.findings-acl.48 2022

  50. [72]

    Gemma Team. 2024 a . https://doi.org/10.48550/ARXIV.2408.00118 Gemma 2: Improving open language models at a practical size . CoRR, abs/2408.00118

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.00118 2024

  51. [73]

    Gemma Team. 2025. https://doi.org/10.48550/ARXIV.2503.19786 Gemma 3 technical report . CoRR, abs/2503.19786

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.19786 2025

  52. [74]

    Llama Team. 2024 b . https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 herd of models . CoRR, abs/2407.21783

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  53. [75]

    Turner, Callum McDougall, Monte MacDiarmid, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, and 3 others. 2024. https://transformer-circuits.pub/2024/...

    2024

  54. [76]

    Li, Arnab Sen Sharma, Aaron Mueller, Byron C

    Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. 2024. https://openreview.net/forum?id=AwyxtyMwaG Function vectors in large language models . In International Conference on Learning Representations

    2024

  55. [77]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. https://doi.org/10.48550/ARXIV.2308.10248 Steering language models with activation engineering . CoRR, abs/2308.10248

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.10248 2023

  56. [78]

    Weixuan Wang, Jingyuan Yang, and Wei Peng. 2025 a . https://openreview.net/forum?id=8WQ7VTfPTl Semantics-adaptive activation intervention for LLMs via dynamic steering vectors . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

    2025

  57. [79]

    Xinpeng Wang, Mingyang Wang, Yihong Liu, Hinrich Sch \" u tze, and Barbara Plank. 2025 b . https://doi.org/10.48550/ARXIV.2505.17306 Refusal direction is universal across safety-aligned languages . CoRR, abs/2505.17306

    doi:10.48550/arxiv.2505.17306 2025

  58. [80]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2025. https://proceedings.mlr.press/v267/wu25a.html Axbench: Steering llms? even simple baselines outperform sparse autoencoders . In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, ...

    2025

  59. [81]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2024. https://doi.org/10.48550/ARXIV.2412.15115 Qwen2.5 technical report . CoRR, abs/2412.15115

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115 2024

  60. [82]

    Zejia You, Chunyuan Deng, and Hanjie Chen. 2026. https://doi.org/10.48550/ARXIV.2602.08169 Spherical steering: Geometry-aware activation rotation for language models . CoRR, abs/2602.08169

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.08169 2026

  61. [83]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others

    Andy Zou, Long Phan, Sarah Li Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann - Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2023. https://doi.org/10.48550/ARXIV.2310.01405 Representation engin...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01405 2023