Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering

Bo Shen; Fan Yang; Haiyan Zhao; Mengnan Du; Ninghao Liu; Xuansheng Wu

REVIEW 3 major objections 6 minor 29 references

Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Selectively scaling the sparse-autoencoder latents that best separate positive from negative examples, then reconstructing hidden states, raises concept-steering success by 4–16% across six concepts without hurting topic relevance.

desk verdict A plausible SAE-denoising recipe for concept vectors, but the headline 4-16% gains are best-over-grid numbers on 50 prompts and don't hold on Gemma, so the claim is premature. read the letter →

arxiv 2505.15038 v2 pith:DQDI5YG7 submitted 2025-05-21 cs.CL cs.AI

Haiyan Zhao , Xuansheng Wu , Fan Yang , Bo Shen , Ninghao Liu , Mengnan Du This is my paper

classification cs.CLcs.AI

keywords conceptsteeringsparseautoencodersdenoisingrepresentationengineeringlinearprobingdifference-in-meanSAElatentslargelanguagemodelinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Sparse Autoencoder-Denoised Concept Vectors (SDCV), a way to clean the hidden representations used to build linear concept vectors for steering LLMs. Its central claim is that the noise in these representations—spurious features from the contrastive datasets—can be reduced by passing hidden states through a sparse autoencoder, scaling up only the handful of 'latents' (feature dimensions) that best distinguish positive from negative examples, and reconstructing from that modified code. When applied on top of linear probing and difference-in-mean, SDCV consistently improves steering success rates by 4–16% across six challenging concepts (sycophancy, corrigibility, myopic reward, Amazon reviews, political speech, Twitter sentiment) while keeping topic relevance scores roughly unchanged. If it holds, it offers a cheap, inference-time denoising module that makes existing concept-vector steering more robust.

What carries the argument

The machinery is the SDCV pipeline around a pre-trained sparse autoencoder: (1) an influence score $s = |a_p(h) - a_n(h)| / (\sigma_n^2 + \sigma_p^2)$ that ranks SAE latents by how much their mean activation differs between the positive and negative sets relative to within-class variance; (2) top-$k$ selection of the highest-scoring latents, with the chosen latents' activations multiplied by a scale factor $m$ while all other latents are left unchanged; (3) SAE decoding of the modified activation vector, plus the reconstruction error, to produce a denoised hidden state from which the concept vector is derived. The design rests on the assumption that concept-relevant features are concentrated in a small set of latents whose class separation is large and whose variance is small, and that everything else is noise.

What would settle it

Run SDCV on the same six concepts but replace the influence-score ranking with a random permutation of latents, keeping k and m fixed. If random selection matches or beats top-k selection on steering success, the influence score is not the mechanism; if random selection also beats the raw baseline, the improvement comes from SAE reconstruction itself rather than discriminative denoising.

Watch

Extended reading notes

Core claim

SDCV's central claim is that concept-relevant signal and dataset noise live in different sparse-autoencoder latents, and that a simple selection rule can separate them. Using a contrastive dataset, the paper scores each SAE latent by the absolute difference in average activation between positive and negative samples, normalized by the within-class spread (equation 1). The top-k latents are then scaled by a factor m (equation 2) and fed through the SAE decoder (equation 3), with the original reconstruction error added back to form a denoised hidden state. Concept vectors built from these denoised states—by either a logistic-regression probe or difference-in-mean—steer the model better than vectors built from raw states. The paper further shows via counterfactual amplification of non-top-k ('noisy') latents that steering degrades as noise is added, supporting the denoising interpretation.

Load-bearing premise

The load-bearing assumption is that the influence score—the gap in average activation between positive and negative examples, divided by within-class variance—correctly identifies the SAE latents that carry concept-relevant signal, so that scaling those latents up and reconstructing leaves the steering-relevant content intact while discarding noise.

Editorial extensions

If this is right

SDCV acts as a drop-in preprocessing module: it can be inserted before any existing concept-vector construction (linear probing or difference-in-mean) without changing the steering equation or requiring retraining of the language model.
For hard, high-level concepts where raw vectors barely steer (success rates of 6–18% on sycophancy, corrigibility, and myopic reward), SDCV raises success to 20–30% in the reported Llama-3.1-8B experiments.
The counterfactual noise-amplification experiments suggest that the non-top-k latents are a genuine source of steering failure: steering success drops as those latents are amplified, supporting the denoising interpretation.
The consistent gains on both Llama-3.1-8B and Gemma-2-2B indicate the denoising mechanism transfers across model families and SAE architectures, rather than being an artifact of a single SAE.
Because optimal k and scale factor are found empirically per concept, the method's practical adoption hinges on finding cheap selection rules for these two hyperparameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the influence-score mechanism is right, the same top-k scaling could be applied to other SAE-based interventions—such as activation editing for truthfulness or debiasing—where the denoised hidden state might transfer.
The paper's six concepts are a convenience sample; a natural stress test is to see whether the 4–16% improvement survives on a broader, more adversarial set of concepts, especially ones where positive and negative sets are near-duplicates.
The nonlinear interaction between k and m suggests that replacing the hard top-k mask with a learned or continuous weighting of latents could yield further gains, at the cost of more hyperparameters.
Since SDCV requires a trained SAE for the target model, the results give a concrete motivation for training high-quality SAEs as infrastructure for controllable generation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: SDCV is a new and sensible recipe — pick the SAE latents that best separate positive and negative examples, scale them up, reconstruct the hidden state, then build the concept vector from that. That specific pipeline is not in the cited literature, and the counterfactual noise experiment is a good idea; it gives some evidence that the benefit is due to noise removal rather than a scaling artifact. So the method deserves attention.

The problem is the evaluation. The reported 4-16% improvements are the best numbers over a grid of 7 k values, 5 m values, and 34 alpha values — 1190 configurations per concept — all evaluated on the same 50-prompt test set. With 50 prompts, 4% is two questions. The maximum over 1190 tries can easily produce that by chance. There's no validation split, no error bars, no significance test. And the 'consistently' in the abstract is contradicted by their own Gemma results: on political speech, SDCV difference-in-mean ties the baseline at 16%, and on Twitter sentiments it's worse (8% vs 10%). So the central claim as stated is not supported by what's shown.

There are a couple of other soft spots. They cite SAIF and CAA but don't compare against them, which leaves the marginal benefit over existing SAE-based steering unclear. And the influence score in Eq (1) is plausible but not independently validated — the only evidence it works is the steering numbers, which brings us back to the evaluation problem.

None of this is fatal. The recipe is simple and the idea is worth testing properly. If the authors add a held-out validation split, report error bars or significance tests, and show that the gains survive averaging over the hyperparameter grid instead of taking the max, this could be a solid contribution. The missing baselines are easy to add.

Who's it for: people working on activation steering or representation engineering will want to know about this method, even if they shouldn't trust the headline numbers yet. I'd send it to peer review rather than desk-reject — the core idea deserves referee time — but as a reviewer I'd push for a real evaluation before accepting. I would not cite the current version in my own work.

Referee Report

3 major / 6 minor

Summary. The paper proposes SDCV, a preprocessing step for concept-vector steering in LLMs. Given contrastive positive and negative samples, it maps hidden states through a pretrained sparse autoencoder (SAE), ranks latents by an influence score s = |a_p - a_n| / (sigma_p^2 + sigma_n^2), scales the top-k latent activations by a factor m, reconstructs the hidden states from the scaled activations, and then builds linear-probing or difference-in-mean concept vectors from the reconstructed states. The central empirical claim is that SDCV improves steering success rates by 4-16% relative to plain linear probing and difference-in-mean on six concepts while maintaining topic relevance. Experiments use Llama-3.1-8B and Gemma-2-2B with 50 open-ended test questions per concept and GPT-4.1-mini as an automated judge.

Significance. The idea of using SAE latent selection as a denoising step for concept vectors is timely and plausible, and a working version would be a useful plug-in module for existing steering methods. The paper also contains a good-faith attempt at mechanistic evidence: the counterfactual noise-injection experiment in Section 3.4 speaks directly to the claim that noise in non-selected latents hurts steering. The limitations paragraph is honest about the empirical nature of k, m, and alpha. However, the evidence as presented does not establish the headline claim. The reported gains are best-over-grid values on the same 50-prompt test set, with no held-out validation or statistical uncertainty, and the Gemma results in Table 3 explicitly contradict the word 'consistently' in the abstract. With Ntest = 50, a 4% improvement is two questions, so the claimed range is within the noise of the evaluation as currently reported.

major comments (3)

[Section 2.1, Eq. (4)] As written, Eq. (4) defines h'_i <- SAE(hi) + epsilon, and the text states 'epsilon is the reconstruction error of original hi.' Appendix A, Eq. (12), defines the SAE reconstruction error by z = SAE(z) + epsilon. Taken literally, substituting epsilon = hi - SAE(hi) into Eq. (4) gives h'_i = hi, so the proposed denoising would have no effect at all. If the intended meaning is that epsilon is the residual from the original, unscaled SAE reconstruction and that this fixed residual is added to the scaled reconstruction, this must be stated explicitly with unambiguous notation; otherwise the central mechanism of the paper is undefined.
[Section 3.1 and Table 1] The success rates in Table 1 are selected as the best result over the hyperparameter grid k in {50, 100, 1000, 5000, 10000, 20000, 50000}, m in {10, 20, 40, 80, 100}, and alpha in {0.3, ..., 0.8} with step 0.015, i.e., 1190 configurations per concept for Llama, with all configurations evaluated on the same 50-prompt test set used for the final comparison. With Ntest = 50, the reported 4-16% improvements correspond to 2-8 questions, and the maximum over 1190 configurations can easily produce such gaps by chance. The paper reports no validation split, no error bars, and no significance tests. To support the claim, the authors should either fix hyperparameters on a separate validation set and report test-set results once, or report distributions over configurations with bootstrap intervals and a paired significance test.
[Table 3 (Appendix G)] The claim in the abstract and in Section 3.2 that SDCV 'consistently improves steering success rates by 4-16%' is contradicted by the authors' own Gemma-2-2B results. For political speech, both SDCV-enhanced linear probing and SDCV-enhanced difference-in-mean tie their baselines at 16%. For Twitter sentiments, SDCV-enhanced difference-in-mean is worse than the baseline (8% vs. 10%). Even if the intended primary claim is about Llama-3.1-8B, the wording of the abstract and Section 3.2 is too strong and should be revised to match the evidence, or the Gemma discrepancy should be explicitly explained rather than described as consistently comparable or better.

minor comments (6)

[Section 3.3 and Figures 2 and 4] The 'emotion steering task' used in Section 3.3 and Figures 2 and 4 is not defined in the main text or Appendix B, and it does not appear among the six concepts in Table 1 or Table 3; please specify the dataset, its size, and the evaluation rubric so this analysis is reproducible.
[Section 2.1, Eq. (1)] The text says the score uses 'standard deviation' while Eq. (1) uses sigma^2_n + sigma^2_p, i.e., variances; please align the prose and the equation.
[Appendix H.6] The heading for the Twitter sentiments subsection repeats the corrigibility definition ('The desired steering behavior for corrigibility is that the model is willing to be changed by human user') and should be corrected.
[Table 3] The concept name 'Corrigibiity' is misspelled in the table header.
[References] There are two nearly identical entries for Elhage et al. on toy models of superposition (2022a and 2022b); please consolidate or distinguish them clearly.
[Section 2.2] The sentence 'the reconstruction error is implicitly captured through the difference from znew' is unclear because znew is defined only after the vector is added; please rewrite for readability.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported 4-16% 'consistent improvement' reduces to best-over-grid success rates on the same 50-prompt test set (fitted input called prediction); the SDCV vector construction itself is self-contained, with no load-bearing self-citations.

fitted input called prediction [Section 3.1 Implementation Details; Section 3.2 (Table 1); Abstract]
"In our experiments with Llama-3.1-8B, whose SAE has 131,072 dimensions, we explore k values in {50, 100, 1000, 5000, 10000, 20000, 50000} for all six concepts. Under each top- k value, we also explored different scale factors m ∈ {10, 20, 40, 80, 100}. ... For steering strength, we set a set of parameters ranging from 0.3 to 0.8 with a step of 0.015. In the steering tasks, we utilize 50 diverse prompts for each concept to evaluate the model’s generations."

The Table 1 figures, summarized as 'success rate improvements ranging from 4% to 16%' (Sec. 3.2), are the maxima over a 7×5×34 ≈ 1190-configuration (k, m, α) grid per Llama concept, all on the same 50-prompt test set, with no validation split, error bars, or significance tests reported. With N_test = 50, a 4% gain is 2 questions and a 16% gain is 8 questions, so the maximum of ~1190 binomial draws yields such gaps by chance; the 'consistently improves' claim is thus a fitted summary forced upward by selection, not a held-out prediction. It is also contradicted by the paper's own tables (Gemma Political ties at 16%; Gemma Twitter DIFF drops 10%→8%; Llama Corrigibility DIFF only +2%), and Appendix H's 'randomly chosen value for k' confirms Table 1 reports tuned configurations.

full rationale

The construction chain (Eqs. 1-5) is largely self-contained and non-circular: the influence score s = |a_p − a_n|/(σ²_n + σ²_p), the top-k selection, the m-scaling, the SAE reconstruction with the residual term, and the downstream linear-probing / difference-in-mean vectors are all defined on the contrastive dataset and on externally trained SAEs (LlamaScope, GemmaScope); no equation feeds test-set steering outcomes back into the vector construction. The self-citations (Shu et al. 2025a, He et al. 2025, Zhao et al. 2025, Shu et al. 2025b) appear in related work, notation, and background only; none carries the load of justifying the SDCV mechanism, and the cited SAEs are external artifacts, so the self-citation axis does not raise the score. A boundary concern: 'concept-relevant' is operationally defined by the same contrastive-set differentiation that the difference-in-mean vector encodes, so the denoising is partly self-reinforcing at train time; but steering success is measured on held-out prompts, so the claim is not vacuous by construction. The genuine circularity is in the empirical headline: Section 3.1 reports a sweep over k ∈ {50,...,50000}, m ∈ {10,...,100}, and α ∈ [0.3,0.8] with step 0.015, all evaluated on the 50-prompt test set, and Section 3.2 converts the resulting maxima into 'success rate improvements ranging from 4% to 16%.' That is a fitted value reported as a prediction (pattern: fitted input called prediction), and the paper's own Gemma table and the +2% Corrigibility DIFF row show the 'consistent' framing is an artifact of per-concept, per-variant cherry-picking. Weighing everything, the vector-construction contribution is genuine and self-contained, but the paper's central quantitative claim partially reduces to the grid-search fit on the test set, so a partial-circularity score of 6 is appropriate; the absence of load-bearing self-citations and the use of external SAEs keep this from being a definitional or self-citation-forced result.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method rests on two untested assumptions: that high-influence SAE latents are causally linked to the concept, and that scaling them improves the concept vector; additionally, all reported numbers are selected over a per-concept hyperparameter grid without a validation split. No new physical or formal entities are introduced.

free parameters (5)

k (number of scaled SAE latents) = selected per concept from {50, 100, 1000, 5000, 10000, 20000, 50000} for Llama, {50, 100} for Gemma
Number of top-k latents scaled in the reconstruction; no validation split is described, best per-concept value is reported.
m (scale factor for latent activations) = selected per concept from {10, 20, 40, 80, 100}
Multiplier applied to activations of the selected latents in Eq (2); chosen per concept via grid search.
alpha (steering strength) = selected from range 0.3 to 0.8 with step 0.015
Steering coefficient in Eq (5); swept without a described validation split.
layer index (target layer) = 25 for Llama-3.1-8B, 20 for Gemma-2-2B
Target layer chosen as deeper layers are assumed to capture complex concepts; not justified per concept.
L2 regularization lambda for logistic regression = not reported
Regularization weight in Eq (8); its value is not given, making the linear probe training incomplete as specified.

assumptions (4)

domain assumption Sparse autoencoder latents decompose hidden states into meaningful features, and reconstruction preserves concept information.
The whole method relies on SAE representations capturing concept-relevant information that survives reconstruction. Invoked in Section 2.1.
ad hoc to paper Latents with high influence score (Eq 1) are causally relevant for the concept, and scaling them improves signal relative to noise.
This is the central hypothesis of SDCV; not derived or independently validated. Appears in Eq (1) and Eq (2).
domain assumption Adding a linear concept vector to residual stream hidden states reliably steers model behavior.
Standard assumption of activation steering literature, invoked in Eq (5).
domain assumption The 50 test prompts per concept are a representative sample for measuring steering success.
Evaluation uses N=50 open-ended questions per concept, no confidence intervals; see Section 3.1 and Table 2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering." pith.science (2026). https://pith.science/paper/DQDI5YG7

@misc{pith2026250515038,
  author       = {Pith},
  title        = {Pith review of: Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQDI5YG7}},
  note         = {Machine review of arXiv:2505.15038}
}

read the original abstract

Linear concept vectors effectively steer LLMs, but existing methods suffer from noisy features in diverse datasets that undermine steering robustness. We propose Sparse Autoencoder-Denoised Concept Vectors (SDCV), which selectively keep the most discriminative SAE latents while reconstructing hidden representations. Our key insight is that concept-relevant signals can be explicitly separated from dataset noise by scaling up activations of top-k latents that best differentiate positive and negative samples. Applied to linear probing and difference-in-mean, SDCV consistently improves steering success rates by 4-16\% across six challenging concepts, while maintaining topic relevance.

Figures

Figures reproduced from arXiv: 2505.15038 by the authors.

**Figure 1.** Overview of the framework of SDCV. 2.1 Denoising Representations via SAE To reduce noise interference, we use SAEs to map hidden representations to a higher dimension and then reconstruct them using only the most influential SAE latents. Given a concept c, we first create a contrastive dataset Dc with M positive and M negative samples. We extract hidden representations from the target layer ℓ of an LLM, denoted as… view at source ↗

**Figure 3.** Counterfactual experiments across k and scale factor values with Llama-3.1-8B. of noise amplification is controlled by a scale factor, with values set to 20, 40, 80, 100. We construct concept vectors from these noise-augmented representations and compare their steering performance against the baseline SDCV-enhanced approach. For this analysis, we employ SDCV-enhanced linear probing on three randomly selected conce… view at source ↗

**Figure 4.** Emotion steering performance under different [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 6 canonical work pages

[1]

Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644

arXiv 2016
[2]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decomposing language models with d...

2023
[3]

David Chanin, James Wilken-Smith, Tom \'a s Dulka, Hardik Bhatnagar, and Joseph Bloom. 2024. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507

arXiv 2024
[4]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022 a . Toy models of superposition. Transformer Circuits Thread. Https://transformer-circuits.pub/202...

work page 2022
[5]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022 b . Toy models of superposition. arXiv preprint arXiv:2209.10652

arXiv 2022
[6]

Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://doi.org/10.48550/arXiv.2406.04093 Scaling and evaluating sparse autoencoders . arXiv preprint. ArXiv:2406.04093 [cs]

arXiv doi:10.48550/arxiv.2406.04093 2024
[7]

Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, and 1 others. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526

arXiv 2024
[8]

Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. https://doi.org/10.48550/arXiv.2502.11356 SAIF : A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models . arXiv preprint. ArXiv:2502.11356 [cs]

arXiv doi:10.48550/arxiv.2502.11356 2025

Show all 29 references

[9]

Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. 2023. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813

2023 arXiv
[10]

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and 1 others. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668--2677. PMLR

2018
[11]

Kai Konen, Sophie Jentzsch, Diaoul \'e Diallo, Peer Sch \"u tt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. 2024. Style vectors for steering generative large language model. arXiv preprint arXiv:2402.01618

2024 arXiv
[12]

Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

2024
[13]

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147

2024 arXiv
[14]

Johnny Lin. 2023. https://www.neuronpedia.org Neuronpedia: Interactive reference and tooling for analyzing neural networks . Software available from neuronpedia.org

2023
[15]

Sheng Liu, Lei Xing, and James Zou. 2023. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668

2023 arXiv
[16]

Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824

2023 arXiv
[17]

Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. 2021. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

2021
[18]

Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681

2023 arXiv
[19]

Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, and 1 others. 2025. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496

2025 arXiv
[20]

Dong Shu, Xuansheng Wu, Haiyan Zhao, Mengnan Du, and Ninghao Liu. 2025 a . Beyond input activations: Identifying influential latents by gradient sparse autoencoders. arXiv preprint arXiv:2505.08080

2025
[21]

Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025 b . A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613

2025
[22]

Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2024. https://doi.org/10.48550/arXiv.2410.12877 Improving Instruction - Following in Language Models through Activation Steering . arXiv preprint. ArXiv:2410.12877 [cs]

2024 doi arXiv
[23]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

2017
[24]

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2025. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. arXiv preprint arXiv:2501.17148

2025 arXiv
[25]

Jason Zhang and Scott Viteri. 2025. https://doi.org/10.48550/arXiv.2409.14026 Uncovering Latent Chain of Thought Vectors in Language Models . arXiv preprint. ArXiv:2409.14026 [cs]

2025 doi arXiv
[26]

Haiyan Zhao, Heng Zhao, Bo Shen, Ali Payani, Fan Yang, and Mengnan Du. 2025. https://openreview.net/forum?id=CvttyK4XzV Beyond single concept vector: Modeling concept subspace in LLM s with gaussian distribution . In The Thirteenth International Conference on Learning Representations

2025
[27]

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

2023 arXiv
[28]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[29]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644

arXiv 2016

[2] [2]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decomposing language models with d...

2023

[3] [3]

David Chanin, James Wilken-Smith, Tom \'a s Dulka, Hardik Bhatnagar, and Joseph Bloom. 2024. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507

arXiv 2024

[4] [4]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022 a . Toy models of superposition. Transformer Circuits Thread. Https://transformer-circuits.pub/202...

work page 2022

[5] [5]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022 b . Toy models of superposition. arXiv preprint arXiv:2209.10652

arXiv 2022

[6] [6]

Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://doi.org/10.48550/arXiv.2406.04093 Scaling and evaluating sparse autoencoders . arXiv preprint. ArXiv:2406.04093 [cs]

arXiv doi:10.48550/arxiv.2406.04093 2024

[7] [7]

Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, and 1 others. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526

arXiv 2024

[8] [8]

Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. https://doi.org/10.48550/arXiv.2502.11356 SAIF : A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models . arXiv preprint. ArXiv:2502.11356 [cs]

arXiv doi:10.48550/arxiv.2502.11356 2025