Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

The Origins of Representation Manifolds in Large Language Models

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Cosine similarity encodes intrinsic feature geometry in LLMs.

desk verdict A clean formal result about cosine similarity and geodesic length, with empirical support that is suggestive but weaker than the headline numbers suggest. read the letter →

arxiv 2505.18235 v1 pith:2Q6SC5MD submitted 2025-05-23 cs.LG cs.AI

Alexander Modell , Patrick Rubin-Delanchy , Nick Whiteley This is my paper · ORCID
classification cs.LGcs.AI MSC 51F9968T07
keywords representationmanifoldslargelanguagemodelsmetricgeometrycosinesimilaritygeodesicslinearhypothesismechanisticinterpretabilityintrinsic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Drawing on the observation that language models represent features such as colors, years, and dates as curved manifolds in their embedding spaces, this paper develops a mathematical framework for why this happens. It models a feature as a metric space—a set of values equipped with a distance—and proposes that representation directions vary continuously and one-to-one with feature values, and that cosine similarity between nearby representations is locally a smooth decreasing function of the squared feature distance. Under these hypotheses, Theorem 1 proves that every rectifiable path in the feature space maps to a path on the representation manifold whose length is exactly a constant multiple of the original, with the constant fixed by the local slope of the cosine-similarity function. Consequently shortest paths on the manifold correspond to shortest paths on the feature space, which means the model's internal geometry of a concept matches the intrinsic geometry of the concept itself. The predictions are tested on color, date, and year representations, where homeomorphism and approximate isometry are observed, including evidence that the language model studied encodes twentieth-century years on a logarithmic scale.

What carries the argument

The argument is carried by two hypotheses. Hypothesis 1 (continuous correspondence) asserts that there is a homeomorphism $\phi_f: Z_f \to \mathcal{M}_f \subseteq S^{D-1}$ between the compact feature metric space and its representation manifold, so the topology of the feature (interval, circle, discrete set, tree) is exactly the topology of the manifold. Hypothesis 2 (cosine similarity reflects distance) asserts that for feature values within distance $\epsilon$, $\langle \phi_f(z), \phi_f(z') \rangle = g_f(d_f(z,z')^2)$ for a twice-differentiable $g_f$ with $g_f'(0)<0$, meaning locally cosine similarity depends only on squared feature distance and not on direction. The proof of Theorem 1 uses a Taylor expansion of $g_f$ about zero and the metric-geometry definition of path length to show that each small segment of the mapped path has Euclidean length $\sqrt{-2g_f'(0)}$ times the feature-space segment length, up to higher-order terms that vanish in the supremum. In the experiments, K-nearest-neighbour graphs over the representations provide estimated geodesic distances, and Chatterjee's correlation and Pearson correlation test the functional dependence of cosine similarity on distance and the linearity of manifold distance in feature distance, respectively.

What would settle it

Take a feature and select two pairs of values that have the same feature-space distance but lie in different directions (for example, hues ten degrees apart and hues ten degrees apart on the opposite side of the color wheel). If the cosine similarities between the embeddings in the two pairs differ beyond noise, Hypothesis 2 is violated, and the proportionality $L(\gamma) = \sqrt{-2g_f'(0)}L(\eta)$ should not be expected to hold exactly; a null result would be two equal cosine similarities at equal distance.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 1: for any rectifiable path $\eta$ in the feature metric space $(Z_f, d_f)$ and its image $\gamma = \phi_f \circ \eta$ on the unit-sphere manifold $\mathcal{M}_f$, the path lengths satisfy $L(\gamma) = \sqrt{-2 g_f'(0)}\, L(\eta)$, where $g_f$ is the smooth local function relating cosine similarity to squared feature distance and $g_f'(0)<0$. Because the prefactor is a constant that depends only on the feature, the correspondence between distances in feature space and along the manifold is an isometry up to a global scale. Therefore geodesic (shortest) paths on the representation manifold are exactly the images of geodesic paths in the feature space, and the intrinsic geometry of the feature is recoverable from the manifold, even though the function $g_f$ is unknown. The authors state this as: a feature and its representation are geometrically indistinguishable.

Load-bearing premise

The load-bearing premise is that, for nearby feature values, cosine similarity between their representations depends only on the squared feature distance, with no dependence on the direction of the difference in feature space; if this local isotropy fails, the exact path-length proportionality of Theorem 1 does not follow.

Editorial extensions

If this is right

  • On-manifold geodesic distances computed from representations become a practical estimator of intrinsic feature distance, up to an unknown global scale.
  • The manifold's topology can be read off the representations: circles, intervals, and trees in feature space must appear as homesomorphic shapes, as Proposition 1 guarantees.
  • Sparse autoencoders trained on a representation manifold should trace the manifold, and many SAE features should align with segments of it rather than with fixed orthogonal directions.
  • For text embeddings, cosine similarity can be reinterpreted as a local proxy for concept distance, giving a principled justification for its widespread use as a semantic similarity measure.
  • The observed logarithmic encoding of years in the studied language model indicates that models may use non-obvious metrics whose geometry is still faithfully preserved, and these can be discovered by testing isometry against candidate metric spaces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof's reliance on a Taylor expansion of $g_f$ suggests the theorem extends to any similarity measure that is a smooth, radial function of representation-space distance, not just cosine similarity; if so, the result would apply to inner products and Euclidean distances on hyperspheres under the same isotropy condition.
  • If the proportionality constant $\sqrt{-2g_f'(0)}$ differs from feature to feature, raw cosine similarities are not comparable across features; a scale-invariant geodesic distance normalized by the manifold's total length would be needed for cross-feature comparisons.
  • The manual process of proposing a metric space could be automated by learning a metric that makes manifold geodesics linear in feature distance, which would address the paper's acknowledged scalability limitation.
  • For synthetic data with known feature manifolds of varied topology (circles, trees, tori), the theory makes the strong prediction that manifold geodesic distances become linear in feature distances; this would provide a cheap, controlled test of the assumptions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a metric-geometry framework for features represented as manifolds in large language models. A feature is defined as a compact metric space (Z_f, d_f); Hypothesis 1 states that feature values map continuously and bijectively onto unit-norm representation directions, and Hypothesis 2 states that cosine similarity between representations is locally a smooth function g_f(d_f(z,z')^2) of squared feature distance, with g'_f(0)<0. The central result, Theorem 1, says that under these hypotheses the length of any rectifiable path in feature space is scaled by the constant sqrt(-2 g'_f(0)) when mapped to the representation manifold, so shortest paths are preserved up to a global scale. The authors test the framework on text embeddings of colors and dates and on GPT-2 token activations for years, and they report that the years manifold is approximately isometric to the logarithmic scale log(2019 - year) rather than to the original linear year scale.

Significance. If the framework is accepted, it offers a clean and useful formalization of a widely discussed phenomenon: continuous features can be represented as nonlinear manifolds rather than as one-dimensional linear directions. The paper gives credit where due: the hypotheses are stated explicitly, Theorem 1 is proved in an appendix, and code is provided. The theorem itself is a genuine contribution and is correct up to a small fixable constant error in the proof. However, the empirical validation is the load-bearing part of the paper's broader claim that representation manifolds actually encode intrinsic feature geometry, and that validation is weakened by post hoc metric selection, by the acknowledged fragility of the KNN geodesic estimates, and by the absence of uncertainty quantification. The central theoretical claim is defensible, but the current evidence does not yet establish that the proposed geometry is more than a reparameterization of the data.

major comments (2)
  1. [§3.1, Figures 3 and 4] The validation of Hypothesis 2 for years is post hoc. The authors first conjecture Z_years = [1900, 1999] with d_year(x,y)=|x-y| and show in Figure 4 that this metric is rejected ("evidence against isometry"). They then replace the feature space by {log(2019 - year)} and report a Pearson correlation of 0.99 between estimated manifold distance and distance in the new metric. Because the replacement was selected after observing the failure of the original metric, the high correlation is in-sample and cannot serve as independent confirmation that cosine similarity encodes the intrinsic geometry of the year feature. For a one-dimensional feature, any strictly monotone reparameterization changes the metric, so a curve can always be reparameterized to look like arc-length. An independent test would need a prespecified metric, a hold-out split, or a derivation of the logarithmic scale from model training data rather than from the point cloud itself.
  2. [§4.1 and §3.1] The geodesic-distance validations rely on K-nearest-neighbour graph distances that the authors themselves describe as "prone to short-circuits causing enormous errors" and state that they "often have to manually prune the graph" to obtain reasonable estimates. The reported Pearson correlations (0.99 for years, 0.97 for dates) are presented without confidence intervals, without a description of the pruning procedure, and without sensitivity analysis with respect to K, PCA dimension, or the number of pruned edges. As written, the quantitative claim that geodesic distance on the manifold is linear in feature distance is not reproducible from the paper alone and could be driven by the manual pruning. The authors should report the full estimation protocol, the exact pruning decisions, and bootstrap or perturbation intervals for the correlations.
minor comments (4)
  1. [Appendix B] In the proof of Theorem 1, C is defined so that sup |g''(r)| <= C, but the displayed bound uses |g''(c_i)|^{1/2} <= C; the final inequality "<= C L(eta) max d" should have sqrt(C) in place of C, or C should be redefined as sqrt(sup |g''|). Otherwise the delta/3 split is off by a factor of sqrt(C). This is a fixable constant error and does not affect the validity of the theorem.
  2. [§2.3 and Figure 1 captions] There are several typos: "indiction" should be "indication", "respectivly" should be "respectively", and "Janurary" should be "January".
  3. [§3.1] Pearson correlation measures linear association, but the claim is proportionality of geodesic distances. The authors should report the fitted slope and intercept (with intervals) in addition to the correlation coefficient, since a high correlation can accompany a nonlinear monotone relationship with non-unit slope.
  4. [§2.3] The rank correlations for years are reported before the logarithmic reparameterization is introduced. Since rank correlation is invariant under monotone transforms, the reader may be misled into thinking this is evidence for the linear year metric; the later discussion clarifies this, but an early caveat would help.

Circularity Check

1 steps flagged · score 4.0 of 10

Empirical validation of Hypothesis 2 is post hoc: the years metric is replaced only after the original conjecture fails, so the reported 0.99 isometry is a fitted input, not an independent prediction.

  1. fitted input called prediction [Section 3.1, 'Geodesic distances on representation manifolds of LLMs are meaningful', paragraphs following Figure 4]
    "Although we found a rank correlation near 1, indicating homeomorphism, the evidence of the tests above is against isometry. ... In light of this, we consider a modified representation Zyears ={log(2019−year) : year∈[1900,1999]}, dyear(x, y) =|x−y|, 2019 being the year GPT-2 was released. ... The tests are now in much stronger support of isometry. ... The bottom-middle panel of Figure 3 shows a clear linear fit, achieving a Pearson correlation of 0.99."

    The feature metric d_f is the input to Hypothesis 2, whose validation is the paper's empirical support for the theory. For years, the paper first conjectured d_year(x,y)=|x-y|, found the data rejected the implied isometry, and then replaced the metric with |log(2019-year)-log(2019-year')|, a reparameterization of the same homeomorphic curve. Since any smooth 1-D curve can be reparameterized by arc length so that cosine similarity becomes locally a function of squared feature distance only, choosing the metric after inspecting the data makes the subsequent 0.99 correlation a fitted input rather than an out-of-sample prediction. The theorem itself remains an independent derivation, so the circularity is limited to the empirical validation.

full rationale

Theorem 1 is a genuine mathematical derivation from Hypotheses 1 and 2, using standard metric-geometry arguments and Lemma 1; no fitted constants are involved and no self-citation is load-bearing. The circularity concern is confined to the empirical support for Hypothesis 2. The paper acknowledges in Section 4.1 that the metric-space choice is a manual, hypothesis-driven procedure and that KNN geodesic estimates require manual pruning, which weakens the quantitative validation. The decisive issue is that for the years example the feature metric was revised only after the original conjecture failed, so the reported linear fit (Pearson 0.99) does not independently confirm that LLM representations encode the proposed feature geometry; it is consistent with the theory's flexibility under reparameterization. This is a partial circularity in the validation of the key assumption, not in the mathematical core.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central theorem uses two clearly stated modeling hypotheses (continuous correspondence and cosine-distance isotropy) plus standard metric geometry. The empirical validation introduces post hoc metric choices (log years, PCA dimension, K) that act as free parameters in the testing procedure.

free parameters (3)
  • Years metric transform = log(2019 - year)
    Chosen after the absolute-distance metric failed isometry checks; post hoc adjustment that improves the reported correlation to 0.99.
  • KNN graph parameter K = 10 for years, smallest connected
    Selected as the smallest K giving a connected graph; affects the estimated geodesic distances and hence all isometry statistics.
  • PCA projection dimension = first few principal components (unspecified)
    Projection to low dimension is required for the isometry patterns to appear; this selection is data-dependent and not independently justified.
assumptions (4)
  • domain assumption Hypothesis 1: continuous one-to-one correspondence between feature space Z_f and representation manifold M_f
    Assumed throughout; gives the homeomorphism between feature and representation topology.
  • domain assumption Hypothesis 2: cosine similarity is locally a smooth function of squared feature distance
    The central assumption of Theorem 1; requires isotropy of cosine similarity in feature space.
  • domain assumption Feature space Z_f is compact
    Stated in Section 2.1 to avoid technical complications; needed for Proposition 1 and for the existence of maxima in the proof.
  • standard math Existence of unit-speed parameterizations for rectifiable paths
    Used in the proof of Theorem 1, cited to Burago et al. Prop 2.5.9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Origins of Representation Manifolds in Large Language Models." pith.science (2026). https://pith.science/paper/2Q6SC5MD

@misc{pith2026250518235,
  author       = {Pith},
  title        = {Pith review of: The Origins of Representation Manifolds in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2Q6SC5MD}},
  note         = {Machine review of arXiv:2505.18235}
}
read the original abstract

There is a large ongoing scientific effort in mechanistic interpretability to map embeddings and internal representations of AI systems into human-understandable concepts. A key element of this effort is the linear representation hypothesis, which posits that neural representations are sparse linear combinations of `almost-orthogonal' direction vectors, reflecting the presence or absence of different features. This model underpins the use of sparse autoencoders to recover features from representations. Moving towards a fuller model of features, in which neural representations could encode not just the presence but also a potentially continuous and multidimensional value for a feature, has been a subject of intense recent discourse. We describe why and how a feature might be represented as a manifold, demonstrating in particular that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths, potentially answering the question of how distance in representation space and relatedness in concept space could be connected. The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.

Figures

Figures reproduced from arXiv: 2505.18235 by the authors.

Figure 1
Figure 1. Representation manifolds in large language models: colours, years and dates. The first and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Representation manifolds in token activations from layer 8 of Mistral 7B, processed via [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Evidence for Hypothesis 2 and its implications in Theorem 1. For each pair of represen [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Evidence against isometry with respect to the metric space Zyears = [1900, 1999], dyear(x, y) = |x − y|. There is no clear regular linear relationship between distances in this metric space and estimated distances on the representation manifold. The colours indicate th…
Figure 3
Figure 3. Figure 3: Across our experiments, we have found that a low-dimensional projection tends to be necessary for the representations to plausibly show isometry with a simple metric space. For our text embeddings, we find that projecting onto the first few (uncentered) principal compo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Temporal Preference Concepts and their Functions in a Large Language Model

    cs.LG 2026-05 unverdicted novelty 6.5 of 10

    Temporal preference in Qwen3-4B-Instruct-2507 localizes to layers 17–35 (especially L24 attention), has curved residual-stream geometry, is behaviorally unstable, and can be bidirectionally steered.

  2. Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    DLMs encode a decodable latent timestep signal in residual activations that can be steered to predictably change model confidence and entropy.

Reference graph

Works this paper leans on

49 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations , ICLR 2017, Toulon , France , April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=HJ4-rAVtl

    work page 2017

  2. [2]

    Intrinsic dimension of data representations in deep neural networks

    Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  3. [3]

    Linear Algebraic Structure of Word Senses , with Applications to Polysemy

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear Algebraic Structure of Word Senses , with Applications to Polysemy . Transactions of the Association for Computational Linguistics, 6: 0 483--495, December 2018. ISSN 2307-387X. doi:10.1162/tacl_a_00034. URL https://direct.mit.edu/tacl/article/43451

    doi:10.1162/tacl_a_00034 2018

  4. [4]

    Superintelligence: Paths , Dangers , Strategies

    Nick Bostrom. Superintelligence: Paths , Dangers , Strategies . Oxford University Press, Oxford, UK, 2014. ISBN 978-0-19-967811-2

    work page 2014

  5. [5]

    Towards Monosemanticity : Decomposing Language Models With Dictionary Learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    work page 2023

  6. [6]

    A course in metric geometry

    Dmitri Burago, Yuri Burago, Sergei Ivanov, and others . A course in metric geometry. American Mathematical Society, 2001

    work page 2001

  7. [7]

    Showing SAE Latents Are Not Atomic Using Meta - SAEs

    Bart Bussmann, Michael Pearce, Patrick Leask, Joseph Bloom, Lee Sharkey, and Neel Nanda. Showing SAE Latents Are Not Atomic Using Meta - SAEs . Less Wrong, 2024. URL https://www.lesswrong.com/posts/TMAmHh4DdMr4nCSr5/showing-sae-latents-are-not-atomic-using-meta-saes

    work page 2024

  8. [8]

    Learning Multi - Level Features with Matryoshka Sparse Autoencoders

    Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning Multi - Level Features with Matryoshka Sparse Autoencoders . arXiv preprint arXiv:2503.17547, 2025

    arXiv 2025

Show all 49 references
  1. [9]

    Isotropy in the contextual embedding space: Clusters and manifolds

    Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations, 2021

    2021

  2. [10]

    Chang, Zhuowen Tu, and Benjamin K

    Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. The Geometry of Multilingual Language Model Representations . In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , EMNLP 2022, A...

    2022 doi

  3. [11]

    A new coefficient of correlation

    Sourav Chatterjee. A new coefficient of correlation. Journal of the American Statistical Association, 116 0 (536): 0 2009--2022, 2021. Publisher: Taylor & Francis

    2009

  4. [12]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  5. [13]

    Sparse and redundant representations: from theory to applications in signal and image processing

    Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010

    2010

  6. [14]

    A Mathematical Framework for Transformer Circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

    2021

  7. [15]

    Toy Models of Superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposi...

    2022

  8. [16]

    Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark

    Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not All Language Model Features Are One - Dimensionally Linear . In The Thirteenth International Conference on Learning Representations , ICLR 2025, Singapore , April 24-28, 2025 . OpenReview.net, 2025. U...

    2025

  9. [17]

    SimCSE : Simple Contrastive Learning of Sentence Embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE : Simple Contrastive Learning of Sentence Embeddings . In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processin...

    2021 doi

  10. [18]

    Minimax manifold estimation

    Christopher R Genovese, Marco Perone-Pacifico, Isabella Verdinelli, and Larry Wasserman. Minimax manifold estimation. The Journal of Machine Learning Research, 13 0 (1): 0 1263--1291, 2012

    2012

  11. [19]

    Curve Detector Manifolds in InceptionV1 , August 2024

    Liv Gorton. Curve Detector Manifolds in InceptionV1 , August 2024. URL https://livgorton.com/curve-detector-manifolds/

    2024

  12. [20]

    Language models represent space and time

    Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023

    2023 arXiv

  13. [21]

    The functional relevance of probed information: A case study

    Michael Hanna, Roberto Zamparelli, David Mareček, and others . The functional relevance of probed information: A case study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 835--848. Association for Computa...

    2023

  14. [22]

    Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks

    Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. Advances in Neural Information Processing Systems, 37: 0 13244--13273, 2024

    2024

  15. [23]

    Towards a definition of disentangled representations

    Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018

    2018 arXiv

  16. [24]

    Identifiability of latent-variable and structural-equation models: from linear to nonlinear

    Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics, 76 0 (1): 0 1--33, 2024. Publisher: Springer

    2024

  17. [25]

    Calendar feature geometry in GPT -2 layer 8 residual stream SAEs

    Patrick Leask, Bart Bussmann, and Neel Nanda. Calendar feature geometry in GPT -2 layer 8 residual stream SAEs . Less Wrong, 2024. URL https://www.lesswrong.com/posts/WsPyunwpXYCM2iN6t/calendar-feature-geometry-in-gpt-2-layer-8-residual-stream

    2024

  18. [26]

    On the Sentence Embeddings from Pre -trained Language Models

    Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the Sentence Embeddings from Pre -trained Language Models . In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pr...

    2020 doi

  19. [27]

    Emergent world representations: Exploring a sequence model trained on a synthetic task

    Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023. Publisher: ICLR

    2023

  20. [28]

    The geometry of concepts: Sparse autoencoder feature structure

    Yuxiao Li, Eric J Michaud, David D Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27 0 (4): 0 344, 2025. Publisher: MDPI

    2025

  21. [29]

    Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2024

    Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2024. URL https://www.neuronpedia.org

    2024

  22. [30]

    Towards understanding grokking: An effective theory of representation learning

    Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35: 0 34651--34663, 2022

    2022

  23. [31]

    SAE feature geometry is outside the superposition hypothesis

    Jake Mendel. SAE feature geometry is outside the superposition hypothesis. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/MFBTjb2qf3ziWmzz6/sae-feature-geometry-is-outside-the-superposition-hypothesis

    2024

  24. [32]

    XKCD Color Name Survey Results , 2010

    Randall Munroe. XKCD Color Name Survey Results , 2010. URL https://xkcd.com/color/rgb/

    2010

  25. [33]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations , ICLR 2023, Kigali , Rwanda , May 1-5, 2023 . OpenReview.net, 2023 ...

    2023

  26. [34]

    Emergent linear representations in world models of self-supervised sequence models

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023 b

    2023 arXiv

  27. [35]

    What is a Linear Representation ? What is a Multidimensional Feature ? Transformer Circuits Thread, 2024

    Chris Olah. What is a Linear Representation ? What is a Multidimensional Feature ? Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/july-update/index.html#linear-representations

    2024

  28. [36]

    Feature Manifold Toy Model

    Chris Olah and Josh Batson. Feature Manifold Toy Model . Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2023/may-update/index.html#feature-manifolds

    2024

  29. [37]

    The geometry of categorical and hierarchical concepts in large language models

    Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint arXiv:2406.01506, 2024

    2024 arXiv

  30. [38]

    Disentangling by subspace diffusion

    David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. Advances in Neural Information Processing Systems, 33: 0 17403--17415, 2020

    2020

  31. [39]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners . 2019

    2019

  32. [40]

    Vogt, Randall Balestriero, Wieland Brendel, and David A

    Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, and David A. Klindt. Cross- Entropy Is All You Need To Invert the Data Generating Process . In The Thirteenth International Conference on Learning Representations , ICLR 2025, Si...

    2025

  33. [41]

    Shai, Lucas Teixeira, Alexander Gietelink Oldenziel, Sarah Marzen, and Paul M

    Adam S. Shai, Lucas Teixeira, Alexander Gietelink Oldenziel, Sarah Marzen, and Paul M. Riechers. Transformers Represent Belief State Geometry in their Residual Stream . In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng...

    2024

  34. [42]

    The ‘strong’ feature hypothesis could be wrong

    Lewis Smith. The ‘strong’ feature hypothesis could be wrong. AI Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/tojtPCCRpKLSHBdpn/the-strong-feature-hypothesis-could-be-wrong

    2024

  35. [43]

    Tensor product variable binding and the representation of symbolic structures in connectionist systems

    Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46 0 (1-2): 0 159--216, 1990. Publisher: Elsevier

    1990

  36. [44]

    Agent foundations for aligning machine intelligence with human interests: a technical research agenda

    Nate Soares and Benya Fallenstein. Agent foundations for aligning machine intelligence with human interests: a technical research agenda. The technological singularity: Managing the journey, pages 103--125, 2017. Publisher: Springer

    2017

  37. [45]

    Sutherland

    Wilson A. Sutherland. Introduction to Metric and Topological Spaces . Oxford University Press, 2009

    2009

  38. [46]

    Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi,...

    2024

  39. [47]

    Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie -Yan Liu, Arjun Manrai, Debor...

    2023

  40. [48]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in neural information processing systems, 36: 0 27223--27250, 2023

    2023

  41. [49]

    Contrastive learning inverts the data generating process

    Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International conference on machine learning, pages 12979--12990. PMLR, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.