Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models

Anubrata Das; Fangcong Yin; Fanzhi Zeng; Femi Bello; Liu Leqi

REVIEW 4 major objections 6 minor 1 cited by

Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An affine map between hidden states lets a small model's steering vectors steer a larger model with nearly the same effect.

desk verdict Useful empirical demonstration that affine maps transfer steering vectors between Gemma sizes, but the conceptual framework assumes its own conclusion. read the letter →

arxiv 2506.00653 v3 pith:3J3LMIFM submitted 2025-05-31 cs.LG cs.AIcs.CL

Femi Bello , Anubrata Das , Fanzhi Zeng , Fangcong Yin , Liu Leqi This is my paper

classification cs.LGcs.AIcs.CL

keywords linearrepresentationtransferabilityaffinemapssteeringvectorsalignmentuniversalspacesparseautoencodersmodelscalingcontrastiveactivationaddition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that language models in the same family, trained on the same data with the same tokenizer, represent the same concepts in linearly related spaces, so a direction that controls behavior in a small model can be transplanted into a larger model by a learned affine map. If true, small models become a cheap testbed for steering and other latent-space interventions on large models: a vector found at small scale can be applied at large scale without searching the large model's own activations. The authors ground this in a universal-feature-space model in which each model's hidden states are projections of a shared feature set, then test it by training a least-squares affine map from Gemma-2B to Gemma-9B hidden states and measuring whether transferred steering vectors reproduce the effects of native steering vectors. Their evidence is behavioral: transferred steering tracks native steering closely across the evaluation suite, and refusal-removal transfers in the same direction.

What carries the argument

The load-bearing object is the hypothesized universal representation space: a shared set of feature vectors $W_U$ with an orthonormal basis $V_U$, from which each model selects a subspace by a projection matrix ($P_S$, $P_T$). Under this model, hidden states are $h^S = P_S^\top W_U^\top c(x) + b_S$ and $h^T = P_T^\top W_U^\top c(x) + b_T$, which yields the affine map $A = P_T^\top (P_S^\top)^\dagger$ and $p = b_T - A b_S$. The map is trained by mean-squared-error regression on paired hidden states, and steering vectors are computed by contrastive activation addition before being carried across with $\tilde{v}^T = A v^S + p$.

What would settle it

Take the same source and target models and a held-out steering task not in the evaluation suite; train the affine map as the paper does, then measure the propensity difference between direct steering and transferred steering. If the correlations fall to near zero or the mean squared error becomes comparable to the random-baseline reconstruction error, the claim fails. A sharper test targets the derivation directly: with sparse-autoencoder coefficients aligned by feature index, check whether the same input produces approximately equal coefficient vectors in the two models; if the coefficients diverge, the universal-feature premise is false.

Watch

Extended reading notes

Core claim

The paper proposes the Linear Representation Transferability Hypothesis: representations learned by models that share a tokenizer and are trained on the same data can be linearly mapped from one to the other, so that for matched layer indices $h^T_{l_T}(x) \approx A h^S_{l_S}(x) + p$ for an affine map $A$ and bias $p$. The empirical discovery is that this holds well enough to be useful: a fixed affine map trained by least squares on hidden states transfers steering vectors from Gemma-2B to Gemma-9B, producing propensity curves that closely track direct steering on the paper's evaluation suite and reproducing the refusal-removal effect. The authors interpret this as evidence for a universal representation space: each model's hidden states live in a subspace of a shared feature space, so the map first lifts the source hidden state into that shared space and then projects it into the target space.

Load-bearing premise

The argument hinges on the assumption that the small and large model, at the chosen layers, activate the same underlying features with the same strengths; if that is false, the affine map between their hidden states is a curve fit rather than a transfer of meaning.

Editorial extensions

If this is right

A steering vector computed on a smaller model can be applied to a larger model in the same family through a single learned affine map, with steering effects that track the native vector's effects on the evaluated tasks.
Refusal-removal steering transfers: the vector that makes the small model comply transfers to the large model, lowering its refusal score in the same way as the large model's own vector.
Because the transfer works for multiple unrelated behaviors, the relationship between the two models' hidden states appears to be a global linear structure rather than a task-specific coincidence.
In model development, a learned map could project a small model's hidden-state changes onto a larger model, allowing cheaper estimates of how post-training changes will affect the large model.
The same affine structure suggests a path to efficient inference by connecting a small model's early layers to a large model's later layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test the paper leaves open: align sparse-autoencoder feature indices across the two models and compare coefficient vectors on identical inputs; the paper's derivation requires them to match, so this turns the framework's crux into a measurable quantity.
If the affine relationship is as tight as the steering results suggest, the same map should transfer other direction-based interventions, such as activation patching or concept erasure; the paper does not test those, but the framework predicts they would transfer.
The framework implies a scaling law for transferability: as the size gap widens, the affine map should degrade once the models' learned feature sets diverge; measuring the reconstruction error across size pairs would delineate where small-to-large steering stops working.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real contribution is empirical: a fixed affine map trained on The Pile between Gemma-2B layer 20 and Gemma-9B layer 20 preserves CAA steering vectors well enough that the transferred vectors produce nearly the same propensity curves as native Gemma-9B vectors on the Tan et al. suite. That result is new, at least relative to the concurrent Oozeer et al. work, which reported mixed affine results. The refusal-transfer experiment adds a second behavior and includes an alpha=0 baseline, though it's only 15 samples. The paper is honest about using similar-relative-depth layer selection and about the conceptual framework being a hypothesis.

Where it gets shaky is the theory. The Section 2.2 derivation assumes both models activate the same universal features with the same strengths (up to permutation). That assumption is exactly what the empirical claim is trying to establish, so the derivation is more illustration than proof. The SAE decoder-matrix comparison provides some supporting evidence, but it is another fitted linear map, not an independent test.

The missing-baseline worry in the stress-test is real but not damaging. High Pearson correlation between direct and mapped propensity scores could be driven by the shared prompt-level baseline. But the paper also reports low sample-wise MSE. Since the baseline cancels in the difference, low MSE is a direct signal that steering effects themselves are close. A no-steer or random-vector control would still make interpretation cleaner, especially for tasks with correlations near 0.7, so I would want it added, but it is not a refutation.

The bigger weaknesses are the narrow scope (one model family, one layer pair) and the post hoc layer selection in the refusal experiment, where the authors explicitly pick layers that work on both models. That inflates apparent success for that one behavior. The main Tan et al. evaluation uses a fixed layer 20, which is better, but still only one pair.

Who is this for? People working on activation steering, model merging, or using small models as sandboxes for interpretability. It deserves a serious referee. With revisions—adding a no-steer baseline, testing more layer pairs, and toning down the universal-space language—it could be a solid paper.

Referee Report

4 major / 6 minor

Summary. The paper proposes the Linear Representation Transferability (LRT) Hypothesis, which states that for models sharing a tokenizer and training data, hidden states can be mapped with an affine transformation. The authors develop a conceptual framework based on a universal feature space, train affine maps from Gemma-2B to Gemma-9B on The Pile or a chat mixture, and test transfer by steering: they map steering vectors from the small model to the large model and compare against native steering vectors. Evaluation includes a proof-of-concept with SAE features, a refusal-behavior experiment (15 samples), and a logit-based suite from Tan et al. (2024). They report high correlation and low MSE between direct and transferred propensity scores, concluding that small-model steering vectors can effectively steer the larger model.

Significance. If the central claim held, the result would be practically significant: it would enable probing and steering large models with small-model representations, and it would connect representation similarity to downstream functional equivalence. The paper has several strengths: a clearly stated hypothesis, use of a downstream behavioral task rather than only geometric metrics, an existing evaluation suite, error bars in the refusal experiment, and public code. However, as detailed in the major comments, the evidence does not yet support the claim because of a missing null baseline in the main logit-based evaluation, selection of layers after seeing performance in the refusal experiment, and an in-sample-only comparison in the SAE observation. These issues are addressable without changing the scope of the paper.

major comments (4)

[Section 3.4 (Figure 5, Table 1)] The reported sample-wise Pearson correlation and MSE between direct and transferred propensity scores cannot separate transfer from a shared prompt-level baseline. If the steering effect is small relative to the no-steer logit difference b(x), any two interventions (including a random vector) will be highly correlated across samples. The paper does not report p(x,0) or a random-vector control for the Tan et al. suite, so it is not established that the transferred curve is closer to the direct curve than a null intervention would be. I recommend adding these baselines and reporting the difference in correlation/MSE relative to the null.
[Section 3.3 (Figure 4) and Section 2.3] The refusal experiment selects layers after confirming that they work for both source and target models, and then trains the affine map on those layers (Section 3.3 step 3). This selection on the evaluation outcome is a form of selection bias that can inflate transfer success. The paper should either prespecify layers based only on the 'similar relative depth' heuristic before evaluating steering, or report results across a grid of layer pairs with an independent validation split for layer selection.
[Section 2.2, Empirical Observation] The reconstruction error between the Gemma-2B and Gemma-9B SAE decoder matrices is computed by fitting a least-squares map on the full matrices and evaluating the same training data. The comparison to random matrices does not control for the number of fitted parameters; a linear map can achieve low training error with random high-dimensional data. To support the claim that the feature matrices are linearly related, the authors should evaluate on held-out features (e.g., fit on one subset of features and test on the rest) or use a permutation null.
[Section 2.2, derivation of h^T = A h^S + p] The derivation of the affine relation assumes R_S^{-1} c_S(x) = R_T^{-1} c_T(x) = c(x), i.e., that the two models activate the same universal features with identical strengths. This assumption is essentially the hypothesis under investigation, so the derivation is conditional rather than evidential. The paper should either test this assumption directly (e.g., by comparing SAE coefficients on shared inputs after aligning permutations) or explicitly state that the framework only shows LRT is consistent with a stronger assumption.

minor comments (6)

[Section 2.2] The in-text reference to the reconstruction-error figure appears as 'Figure ??' instead of a numbered figure; the figure caption is present but the reference is a placeholder.
[Sections 2.1 and 2.2] The spelling 'Transferrability' is used in the headings, while the rest of the paper uses 'Transferability'; please unify the spelling.
[Section 2.3] The training objective contains a double vertical-bar formatting issue in the displayed equation; it should use a single norm notation.
[Section 3.4] The sentence 'We include error estimates since model responses are generated via sampling' appears to belong to the refusal experiment (Section 3.3), not to the logit-based evaluation, which is deterministic.
[Section 3.2] The proof-of-concept does not describe how feature indices for Capitalization, Dog Mentions, and French Speaking are selected via Neuropedia; please provide the selection procedure and specific feature indices for reproducibility.
[Appendix B] Table 1 reports 'Significant' for every task but does not describe the significance test; please specify the null hypothesis and test statistic (e.g., whether the 95% CI excludes zero).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's steering-transfer claims are tested with an affine map trained on The Pile and evaluated on held-out steering tasks, and the Section 2.2 derivation is an explicitly conditional mathematical implication rather than a fitted prediction.

full rationale

The central empirical claim is that an affine map trained on hidden states from The Pile transfers steering vectors from Gemma-2B to Gemma-9B, evaluated on held-out steering tasks (Section 3.4, Figure 5, Table 1) and on a refusal task (Section 3.3, Figure 4). This is not circular: the affine map is fit on one distribution and tested on different steering behaviors, and the refusal experiment includes an α=0 baseline. The conceptual framework in Section 2.2 is explicitly conditional: it assumes a universal feature space and, additionally, that source and target models activate the same features with the same strength (R_S^{-1} c_S = R_T^{-1} c_T = c(x)), and then derives h_T = A h_S + p. This is a mathematical implication from stated assumptions, not an equivalence in which the conclusion is defined as the input; the paper does not fit the affine map to the steering data and then relabel that fit as a prediction. The SAE decoder-matrix comparison in Section 2.2 is an independent empirical check against random matrices, and the paper cites external works (Lieberum et al., Arditi et al., Tan et al.) for data and baselines rather than relying on load-bearing self-citations. The absence of a no-steer or random-vector baseline in Section 3.4 is a legitimate validity concern, but it is an omitted control, not a circular reduction of the paper's own construction. The paper also openly acknowledges the conditional status of its framework (Section 2.1, Appendix A, Section 5), which further indicates that the derivation is not masquerading as an empirical result. Overall, the steering-transfer evidence is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 6 assumptions · 3 invented entities

The central claim rests on the assumption of a shared universal feature space, identical feature activations across models, and a layer-correspondence heuristic. The affine map itself is fitted to data, so the result is not a first-principles derivation.

free parameters (2)

Affine map A and bias p = learned via least squares on 10M The Pile tokens (Table 2)
The mapping is the central fitted object; its success on steering transfer is the empirical claim being tested.
Layer indices l_S and l_T = 20 and 20
Chosen by a relative-depth heuristic, and for the refusal experiment by a sweep that selected layers working for both models. The result may depend on this choice.

assumptions (6)

domain assumption Linear Representation Hypothesis: hidden states are linear combinations of human-interpretable features (Park et al., 2023).
Invoked in Section 2.2 to justify modeling h(x)=W_U^T c(x)+b.
ad hoc to paper Existence of a universal feature space W_U shared across models trained on the same data and tokenizer.
Assumed in Section 2.2; no direct independent measurement, only indirect support via linear transfer.
ad hoc to paper Corresponding layers activate the same features with the same strengths (R_S^{-1} c_S = R_T^{-1} c_T = c(x)).
This equality is the step that turns the universal-space model into the affine relation; the paper does not verify it directly.
domain assumption SAE decoder matrices represent the feature spaces of the models (from Gemma Scope, Lieberum et al. 2024).
Used in Section 2.2 empirical observation to compare feature matrices.
domain assumption Models with similar relative depths have corresponding computational stages (Lad et al., 2024).
Justifies choosing layer 20 in both models; not directly validated for Gemma-2B versus 9B.
standard math Standard linear algebra (SVD, least squares) is valid for the projections used.
Used in Section 2.2 and the empirical reconstruction-error analysis.

invented entities (3)

Universal feature space with basis V_U and feature matrix W_U
purpose: Theoretical anchor for the claim that both models' representations are subspaces of a common space.
The paper provides indirect empirical support (linear relation of SAE decoders, steering transfer) but no direct observation or falsifiable prediction that isolates W_U.
Model-specific projection matrices P_S and P_T
purpose: Map universal basis features into each model's subspace; they make the affine relation A=P_T^T(P_S^T)^dagger concrete.
These matrices are never estimated from data; they are placeholders in the conceptual derivation.
Feature-index permutation matrices R_S and R_T
purpose: Allow feature ordering to differ between models while keeping activations aligned.
No evidence that permutations alone describe the index alignment; introduced to make the identity c_S = c_T work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models." pith.science (2026). https://pith.science/paper/3J3LMIFM

@misc{pith2026250600653,
  author       = {Pith},
  title        = {Pith review of: Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3J3LMIFM}},
  note         = {Machine review of arXiv:2506.00653}
}

read the original abstract

It has been hypothesized that neural networks with similar architectures trained on similar data learn shared representations relevant to the learning task. We build on this idea by extending the conceptual framework where representations learned across models trained on the same data can be expressed as linear combinations of a \emph{universal} set of basis features. These basis features underlie the learning task itself and remain consistent across models, regardless of scale. From this framework, we propose the \textbf{Linear Representation Transferability (LRT)} Hypothesis -- that there exists an affine transformation between the representation spaces of different models. To test this hypothesis, we learn affine mappings between the hidden states of models of different sizes and evaluate whether steering vectors -- directions in hidden state space associated with specific model behaviors -- retain their semantic effect when transferred from small to large language models using the learned mappings. We find strong empirical evidence that such affine mappings can preserve steering behaviors. These findings suggest that representations learned by small models can be used to guide the behavior of large models, and that the LRT hypothesis may be a promising direction on understanding representation alignment across model scales.

Figures

Figures reproduced from arXiv: 2506.00653 by the authors.

**Figure 1.** Illustration of the conceptual framework. Despite differences in architectural details (e.g., model size, pre- v.s. post-LayerNorm) or training dynamics, transformers trained on the same (or similar) data and using the same tokenizer are exposed to identical (or similar) input distributions and subword decompositions. As a result, they may discover shared underlying structures in the data—such as syntax, semantics… view at source ↗

**Figure 2.** Reconstruction error between the projected Gemma-9B decoder matrix and the Gemma-2B decoder matrix, using the sparse autocoder trained by Lieberum et al. [2024], compared to that of random matrices of the same dimensions projected using the same procedure. Empirical Observation. Thus far, we have presented the conceptual model for understanding why the LRT hypothesis may hold through the existence of a univers… view at source ↗

**Figure 3.** s2l and l2l corresponds to the feature-to-latent and latent-to-latent mapping. Based on our conceptual framework (Section 2.2), if we can identify features in a universal representation space and map them to both source and target spaces, they should retain the same meaning. In our case, we treat the target model’s feature space as the universal space, with the subspace defined by the smaller model’s feature matrix.… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Refusal Score vs. α. The mean refusal score of model responses on a range of different steering strengths (α). Lower scores indicate less refusal. The left figure shows the propensity curve when using the chat mixture, while the right curve uses purely pretraining dat…

**Figure 5.** Figure 5: Steering Gemma-9B directly vs. steering Gemma-9B with vectors learned with Gemma2B. We first train direct steering vectors on Gemma-2B and Gemma-9B. Then we transfer Gemma-2B steering vectors onto the Gemma-9B latent space. Now we compute the propensity scores for the…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
cs.LG 2026-08 conditional novelty 7.0 of 10

Cross-model KV cache transfer within LLM families is feasible with a per-head ridge regression, retaining 73-98% of accuracy on four matched-KV pairs.

Reference graph

Works this paper leans on

40 extracted references · 17 canonical work pages · cited by 1 Pith paper

[1]

Refusal in language models is mediated by a single direction

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

arXiv 2024
[2]

Revisiting model stitching to compare neural representations

Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:235435759

work page 2021
[3]

Towards cross-tokenizer distillation: the universal logit distillation loss for llms

Nicolas Boizard, Kevin El Haddad, C \'e line Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint arXiv:2402.12030, 2024

arXiv 2024
[4]

Towards monosemanticity: Decomposing language models with dictionary learning

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

2023
[5]

Curve circuits

Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 6 0 (1): 0 e00024--006, 2021

work page 2021
[6]

Similarity and matching of neural network representations

Adri \'a n Csisz \'a rik, P \'e ter K o r \"o si-Szab \'o , Akos Matszangosz, Gergely Papp, and D \'a niel Varga. Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 34: 0 5656--5668, 2021

work page 2021
[7]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

2021
[8]

Toy models of superposition

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. Accessed May 30, 2025

work page 2022

Show all 40 references

[9]

The P ile: An 800gb dataset of diverse text for language modeling

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The P ile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

2020 arXiv
[10]

Universal neurons in gpt2 language models

Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. Trans. Mach. Learn. Res., 2024, 2024. URL https://api.semanticscholar.org/CorpusID:267068880

2024
[11]

Saes are highly dataset dependent: A case study on the refusal direction

Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. Saes are highly dataset dependent: A case study on the refusal direction. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/rtp6n7Z23uJpEH7od/saes-are-highly-dataset-dependent-a-case-study-on-the

2024
[12]

Towards measuring representational similarity of large language models

Max Klabunde, Mehdi Ben Amor, Michael Granitzer, and Florian Lemmerich. Towards measuring representational similarity of large language models. ArXiv, abs/2312.02730, 2023 a . URL https://api.semanticscholar.org/CorpusID:265658940

2023 arXiv
[13]

Similarity of neural network models: A survey of functional and representational measures

Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 2023 b . URL https://api.semanticscholar.org/CorpusID:258587825

2023
[14]

Maier-Hein, Markus Strohmaier, and Florian Lemmerich

Max Klabunde, Tassilo Wald, Tobias Schumacher, Klaus H. Maier-Hein, Markus Strohmaier, and Florian Lemmerich. Resi: A comprehensive benchmark for representational similarity measures. ArXiv, abs/2408.00531, 2024. URL https://api.semanticscholar.org/CorpusID:271601150

2024 arXiv
[15]

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. ArXiv, abs/1905.00414, 2019. URL https://api.semanticscholar.org/CorpusID:141460329

1905 arXiv
[16]

The remarkable robustness of llms: Stages of inference? arXiv preprint arXiv:2406.19384, 2024

Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of llms: Stages of inference? arXiv preprint arXiv:2406.19384, 2024

2024 arXiv
[17]

Michael Lan, Philip H. S. Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models. ArXiv, abs/2410.06981, 2024. URL https://api.semanticscholar.org/CorpusID:273228435

2024 arXiv
[18]

Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024

2024 arXiv
[19]

URL https://transformer-circuits.pub/2024/crosscoders/index.html

Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah, Oct 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html

2024
[20]

In-context vectors: Making in context learning more effective and controllable through latent space steering

Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668, 2023

2023 arXiv
[21]

Linearly mapping from image to text space

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162, 2022

2022 arXiv
[22]

Cross-tokenizer distillation via approximate likelihood matching

Benjamin Minixhofer, Ivan Vuli \'c , and Edoardo Maria Ponti. Cross-tokenizer distillation via approximate likelihood matching. arXiv preprint arXiv:2503.20083, 2025

2025
[23]

Neuronpedia, 2025

Neuronpedia Contributors . Neuronpedia, 2025. URL https://www.neuronpedia.org/. Accessed: 2025-05-15

2025
[24]

Activation space interventions can be transferred between large language models

Narmeen Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Michael Lan, Abir Harrasse, and Amirali Abdullah. Activation space interventions can be transferred between large language models. ArXiv, abs/2503.04429, 2025. URL https://api.semanticscholar.org/CorpusID:276812830

2025
[25]

Steering llama 2 via contrastive activation addition

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023

2023 arXiv
[26]

The linear representation hypothesis and the geometry of large language models

Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023

2023 arXiv
[27]

Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editor...

2017
[28]

Steering llama 2 via contrastive activation addition

Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504--15522, 2024

2024
[29]

High-low frequency detectors

Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 6 0 (1): 0 e00024--005, 2021

2021
[30]

Improving instruction-following in language models through activation steering

Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Improving instruction-following in language models through activation steering. arXiv preprint arXiv:2410.12877, 2024

2024 arXiv
[31]

Analysing the generalisation and reliability of steering vectors

Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adri \`a Garriga-Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. Advances in Neural Information Processing Systems, 37: 0 139179--139212, 2024

2024
[32]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

2023
[33]

Steering language models with activation engineering

Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023

2023 arXiv
[34]

Knowledge fusion of large language models

Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491, 2024

2024 arXiv
[35]

Hopcroft

Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Kris Wu, Zhiqiang Hu, Kun He, and John E. Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In Neural Information Processing Systems, 2018. URL https://api...

2018
[36]

Axbench: Steering llms? even simple baselines outperform sparse autoencoders

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. arXiv preprint arXiv:2501.17148, 2025

2025 arXiv
[37]

Deep model reassembly

Xingyi Yang, Zhou Daquan, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. ArXiv, abs/2210.17409, 2022. URL https://api.semanticscholar.org/CorpusID:253236958

2022 arXiv
[38]

P Xing, Joseph E

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023

2023
[39]

Representation engineering: A top-down approach to ai transparency

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023 a

arXiv
[40]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023 b

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Refusal in language models is mediated by a single direction

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024

arXiv 2024

[2] [2]

Revisiting model stitching to compare neural representations

Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:235435759

work page 2021

[3] [3]

Towards cross-tokenizer distillation: the universal logit distillation loss for llms

Nicolas Boizard, Kevin El Haddad, C \'e line Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint arXiv:2402.12030, 2024

arXiv 2024

[4] [4]

Towards monosemanticity: Decomposing language models with dictionary learning

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

2023

[5] [5]

Curve circuits

Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 6 0 (1): 0 e00024--006, 2021

work page 2021

[6] [6]

Similarity and matching of neural network representations

Adri \'a n Csisz \'a rik, P \'e ter K o r \"o si-Szab \'o , Akos Matszangosz, Gergely Papp, and D \'a niel Varga. Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 34: 0 5656--5668, 2021

work page 2021

[7] [7]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

2021

[8] [8]

Toy models of superposition

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. Accessed May 30, 2025

work page 2022