Pith. sign in

REVIEW 3 major objections 4 minor 58 references

Independently trained transformers converge to compact shared algorithmic cores that are necessary and sufficient for their tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:39 UTC pith:UE7VNFVM

load-bearing objection ACE is a promising method with two clean toy experiments, but the GPT-2 'universal agreement core' is largely a re-readout of the probing objective, and the abstract promises more than the body delivers. the 3 major comments →

arxiv 2602.22600 v2 pith:UE7VNFVM submitted 2026-02-26 cs.LG cs.AI

Transformers converge to invariant algorithmic cores

Joshua S. Schiffman This is my paper
classification cs.LG cs.AI
keywords algorithmic coresmechanistic interpretabilityinvariance across training runsgrokkingmodular additionsubject-verb agreementGPT-2functional equivalence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Training selects for behavior, not circuitry, so this paper tries to show that what persists across transformers is not individual weights but a low-dimensional algorithmic core. In a controlled Markov-chain task, three independently trained one-layer transformers have near-zero weight similarity yet embed nearly the same 3D core, and operators fit inside the cores recover the chain's spectrum. In modular addition, cores crystallize at grokking as cyclic operators and later inflate under continued weight decay, yielding a predicted inverse scaling law for grokking time. In GPT-2 Small, Medium, and Large, subject-verb agreement is claimed to reduce to a single steerable axis whose reflection inverts grammatical number throughout open-ended generation.

Core claim

The paper claims that independently trained transformers solving the same task contain a low-dimensional subspace that is simultaneously necessary and sufficient for performance and is conserved across realizations even when weights diverge. For the Markov task, three one-layer transformers yield 3D cores embedded in nearly orthogonal subspaces, yet least-squares operators fit in core coordinates recover the transition matrix's eigenvalues to within about 1%. For modular addition, cores are shown to crystallize at grokking as cyclic operators and to inflate under sustained weight decay, with grokking delay following a predicted inverse scaling law in weight decay and redundancy. For GPT-2 Sm

What carries the argument

Algorithmic Core Extraction (ACE): from a layer's hidden activations H and the Jacobians J of a task readout, ACE forms the interaction H J^T, takes its SVD, projects the top singular modes back into activation space, and defines the core as their span; core-only and core-removed ablations then certify sufficiency and necessity. The paper motivates this as the nonlinear analogue of selecting the jointly reachable and observable part of a linear system, and the central role of the operator spectrum is that fitted linear dynamics inside the core are used to identify the computation.

Load-bearing premise

The central claim rests on ACE's assumption that the SVD of the activation-covariance times output-sensitivity product identifies the true computational subspace rather than merely the readout's linear sensitivity, and the paper does not compare the resulting core against random or alternative one-dimensional subspaces at the same layer, so the causality certificates stand or fall with that identification.

What would settle it

Re-run the GPT-2 agreement ablations on the same prompts but with a random one-dimensional projection drawn from the same activation distribution; if random axes show core-only AUC near baseline and core-removed AUC near chance, the core's necessity and sufficiency are artifacts of the projection and readout. Also run the ACE protocol on LLaMA-3.1, Gemma-2, and Qwen2.5 at their late layers: if their agreement axes do not align with GPT-2's, the universal cross-architecture claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Explanations that target cores rather than circuits should transfer across random seeds, checkpoints, and architectures, because they describe what is preserved rather than what is particular.
  • Necessary-and-sufficient cores give a principled intervention handle: the GPT-2 steering result shows a single reflective axis can flip a linguistic behavior in open-ended generation.
  • Because grokking time is modeled as inversely proportional to weight decay and mode count, training schedules can be tuned to shorten memorization-to-generalization delays; annealing weight decay after grokking preserves compact cores.
  • Core alignment, not weight similarity, may serve as a diagnostic of functional equivalence and as a basis for principled model merging; the paper argues naive weight interpolation fails precisely because cores are embedded in nearly orthogonal subspaces.
  • If core dimensionality stays low as models scale, mechanistic study of large models can focus on a handful of directions per task rather than on individual weights.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial caveat: the abstract announces agreement-core results for LLaMA-3.1, Gemma-2, and Qwen2.5, but the manuscript body reports only GPT-2 Small, Medium, and Large; the cross-architecture universality claim is therefore asserted, not yet demonstrated in this text.
  • Editorial inference: the same extraction could be applied to other grammatical features, such as pronoun case, gender, or tense, to test whether agreement cores are verb-specific or instantiate a general number variable; the paper leaves this untested.
  • Editorial inference: the grokking ODE predicts a sharp dependence on mode count; a direct test on another group operation, such as multiplication modulo p, would show whether the inverse law is specific to the cyclic structure of addition or a general redundancy effect.
  • Editorial inference: because cores are statistically similar but geometrically orthogonal, aligning core coordinates before merging models could yield greater success than weight-space interpolation; the paper suggests this as a diagnostic but does not run a merge experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Algorithmic Core Extraction (ACE), a subspace method that selects directions in transformer activations that are both high-variance and output-relevant, validates the selected subspace through projection ablations, and interprets it by fitting linear dynamics. ACE is applied to three settings: single-layer transformers trained on a four-state Markov chain, two-layer transformers trained on modular addition, and pretrained GPT-2 Small/Medium/Large on subject-verb agreement. The paper claims that Markov-chain models converge to 3D cores recovering the transition spectrum, that modular-addition cores crystallize at grokking and later inflate under weight decay, and that GPT-2 agreement is governed by a one-dimensional late-layer axis shared across scales. The abstract additionally announces results on LLaMA-3.1, Gemma-2, and Qwen2.5 and a two-order-of-magnitude scale range, but the body contains no such experiments.

Significance. If valid, ACE would provide a genuinely useful invariant-based target for mechanistic interpretability, complementing circuit analysis and sparse-autoencoder approaches. The strongest evidence is in the controlled settings: the Markov-chain core spectra match ground truth to within 1%, and the automated recovery of rotational operators for modular addition is compelling because no algorithmic form is assumed. The attempt to derive and test a quantitative grokking law is also valuable. The GPT-2 cross-scale axis-alignment result, if confirmed with appropriate controls, would be a striking empirical finding. However, the GPT-2 causal evidence is weakened by selection-evaluation coupling, and the abstract overstates the evidence, so the significance is conditional pending revision.

major comments (3)
  1. [Methods ('Algorithmic core extraction'; 'Subject-verb agreement experiment'); Table 5] The selection-evaluation coupling undermines the GPT-2 Large causal claim. At layer 36 (Table 4), the final hidden state before unembedding, the target f(h) = (logit_are + logit_were) - (logit_is + logit_was) is affine in h, so every Jacobian row is the same vector v; H J^T = (H v) 1^T is rank one and the ACE core reduces to span(H^T H v). Core-only/removed/flipped ablations therefore preferentially move f by construction, and the Table 5 AUC swings are not independent evidence of a unique algorithmic core. The manuscript cites refs. 31-33 on deceptive subspace interventions but does not run the obvious control: random or alternative one-dimensional subspaces at the same layer, or directions selected from a held-out target. Please add such controls and re-state the GPT-2 interpretation accordingly.
  2. [Abstract; Results ('A Universal Agreement Core Across GPT-2 Scales')] The abstract claims ACE is applied to six language models (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, Qwen2.5) and that subject-verb agreement is governed by a single axis 'that aligns across architectures.' The body contains no LLaMA, Gemma, or Qwen analysis; only GPT-2 Small/Medium/Large are reported. The claimed two-orders-of-magnitude scale range and the cross-architecture alignment claim are therefore unsupported by the manuscript. Either report the missing experiments or remove/qualify these claims in the abstract and introduction.
  3. [Box 1; Methods ('Grokking sweeps and scaling fits')] The inverse grokking law tau(p) = -Omega log(1 - p_crit/p) is validated by fitting Omega ~ 2770 and p_crit ~ 23 to the same p-sweep that is then compared with the curve; R^2 > 0.99 is an in-sample goodness-of-fit, not a predictive test. Since p_crit is near the smallest tested modulus (23 vs. p = 31), the claimed divergence near p_crit is heavily informed by the fit. Report parameter uncertainties, perform leave-one-out prediction across moduli or other out-of-sample checks, or fix p_crit from theory before fitting so that the announced scaling law is actually falsifiable.
minor comments (4)
  1. [Table 4] The spectral-gap entry for GPT-2 Large is typeset in a confusing way ('12.8×10 10'); clarify the value and units so the one-dimensionality claim can be checked.
  2. [Figure 2; Methods ('Modular Addition Experiment')] The fit quality R^2_h is used in several places but never explicitly defined; distinguish it from the oracle-relative R^2 used in the Markov experiment.
  3. [Methods ('Subject-Verb Agreement Experiment')] The adaptive steering procedure has several thresholds (gating mass, target margin, cap); provide explicit default values and a sensitivity analysis, since the open-ended generation examples depend on these choices.
  4. [Code Availability] The text says code 'will be made available' at a URL; for reproducibility, please provide a fixed release version or archive alongside the submission rather than a future link.

Circularity Check

2 steps flagged

ACE's GPT-2 core is extracted with the same target function used to validate it; at GPT-2 Large's final layer this reduces to a covariance-weighted readout, and the grokking law is fit, not predicted.

specific steps
  1. self definitional [Methods: 'Algorithmic core extraction', 'Causal validation', and 'Subject–Verb Agreement Experiment'; Results: Table 5]
    "To find directions that are jointly active and relevant, ACE computes the SVD of their interaction: HJ⊤ = UΣV⊤... C := span(H⊤Ur)... Causal validation... Core-only (to test sufficiency): ˜h=Ph, Core-removed (to test necessity): ˜h=h−Ph... Target function. The number margin was defined on the final-token hidden state h: f(h) := (logitare + logitwere)−(logit is + logitwas)."

    At GPT-2 Large's selected core layer 36, the final transformer layer, f is affine in h, so every Jacobian row is the same vector v; HJ⊤ = (H v)1⊤ is rank 1 and the ACE core is span((H⊤H)v), a covariance-weighted readout direction for exactly the logit margin used as the AUC target. Selecting a subspace by its sensitivity to f and then reporting that removing or flipping it changes f (Table 5) is not an independent test of necessity/sufficiency; it certifies the estimator's own objective. The same coupling inflates the Fig. 4B linear-control R² and weakens the cross-scale alignment claim. The paper warns that probes and subspace interventions can be deceptive but does not run an obvious control using random or alternative 1D subspaces at the same layer.

  2. fitted input called prediction [Box 1 ('Functional Equivalence Accelerates Grokking'); Methods: 'Grokking sweeps and scaling fits']
    "τgrok(p) = −Ω log(1 − pcrit/p)... This expression depends on two empirical constants... These predictions were validated by sweeping ω and p in transformers... Fit parameters: ˆΩ≈2,770, ˆpcrit≈23."

    The paper derives τ(p) with free constants Ω and p_crit, then fits those constants to the same empirical grokking delays by non-linear least squares and reports the R² as validation. This is a curve fit, not an out-of-sample prediction; the 'inverse scaling law' τ≈Ω p_crit/p at large p is a consequence of the fitted log form. The ω sweep is also fit as a free power law (β≈−1.02), so agreement with the predicted ω⁻¹ is suggestive but not a forced prediction. Calling the fitted curve 'predictions validated' overstates its evidential value.

full rationale

Two partial circularities were identified, but the paper is not wholly circular. The Markov-chain experiment is largely self-contained: cores are extracted with a 99.9% energy threshold on the SVD of HJ⊤ with f=logits, and the recovered spectra are compared to ground-truth T; that is an independent, falsifiable check rather than a reduction to the extraction objective. The modular-addition experiment similarly recovers a known rotational algorithm without prespecifying it, and the core-formation/inflation observations are empirical rather than definitionally forced. Self-citations (e.g., ref. 10 on system drift) are used as analogy and discussion, not as load-bearing mathematical premises; no uniqueness theorem is imported from the authors. The main circularity concerns are: (1) the ACE extraction/causal-validation loop for GPT-2, where the same target function f defines the Jacobian used to build the core and the AUC used to evaluate it—most acute at GPT-2 Large layer 36, where f is affine and the core is a covariance-weighted readout direction—and (2) the grokking time law, whose constants are fitted to the same data it is said to validate. The abstract's claim of LLaMA-3.1, Gemma-2, and Qwen2.5 results not present in the manuscript is a completeness/verification gap, not a circularity. Had the paper included random-subspace or alternative-readout controls and out-of-sample fits, the score would be much lower.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 1 invented entities

The central contribution is a constructive method plus a fitted dynamical model. The method assumes that single-layer projection ablations are causally faithful and that the output Jacobian identifies the computation; the grokking law assumes post-memorization dynamics are pure weight-decay drift with a corrective force that is scaled but not derived. The two leading fitted constants (Ω, p_crit) are fit to the same data used for validation.

free parameters (3)
  • Ω (optimizer constant) = ≈ 2770
    In Box 1 and the grokking sweeps, Ω maps continuous gradient-flow time to discrete AdamW steps and absorbs learning rate, momentum, and preconditioning; it is fitted to the empirical grokking delay versus p data.
  • p_crit (architectural constant) = ≈ 23
    Defined as δ/κ in the margin ODE; it sets the capacity floor below which grokking cannot occur and is fitted alongside Ω to the same grokking-time data.
  • ACE rank-selection thresholds = 99.9% energy (Markov), 99% energy (modular), spectral gap (GPT-2)
    Rank criteria are chosen by hand rather than derived; they directly determine core dimensionality and hence the necessity/sufficiency and spectral results.
axioms (7)
  • domain assumption Linear system theory (Kalman decomposition, minimal realization, balanced truncation) transfers conceptually to nonlinear transformer computations.
    The entire ACE method is motivated by this analogy; Methods 'Algorithmic Cores' invokes the Kalman decomposition but does not prove it applies to transformers.
  • domain assumption Directions jointly high in activation variance and output-Jacobian magnitude, found via SVD of H J^T, form the causal algorithmic core; one-layer projection ablations are valid need/sufficiency tests.
    This is the central methodological premise, introduced in Methods 'Algorithmic core extraction' and 'Causal validation'; no random-subspace control is reported for GPT-2.
  • domain assumption Post-memorization dynamics: task loss is approximately zero, weight decay dominates the gradient, the corrective force is γ(t) ≈ cω, and the initial margin m(0) ≈ 0.
    These assumptions drive the ODE and the inverse scaling law in Box 1 and Methods 'Mathematical Model of Grokking Dynamics'.
  • domain assumption Fourier modes of modular addition are encoded in approximately orthogonal parameter subspaces; if they are in superposition, the effective ∥ψ∥² is modified by S.
    Box 1 assumes ∥W∥² ≈ ∥α∥² and then notes a superposition correction; the chosen orthogonal limit is an idealization.
  • domain assumption A linear operator fit in core coordinates adequately captures next-token dynamics for Markov chains and modular addition.
    Spectral claims in Figures 1F and 2C depend on this fit; nonlinear residuals are not analyzed.
  • domain assumption The hand-built 1,200-prompt set and the logit-margin target function isolate subject-verb number agreement; adaptive per-token steering faithfully implements 'flipping the core.'
    Methods 'Subject–Verb Agreement Experiment' defines the task and intervention; the paper does not compare against random subspaces or alternative target definitions.
  • standard math Standard mathematical facts used in the derivation: Cauchy–Schwarz inequality, SVD/QR factorization, and the trigonometric identity giving ∥ψ∥²₂ = 3p/4.
    These are used in Box 1 and Methods without proof; they are uncontroversial.
invented entities (1)
  • Algorithmic core (a low-dimensional subspace of transformer activations) independent evidence
    purpose: Serves as the invariant computational object claimed to be necessary and sufficient for a task and shared across independently trained transformers.
    The paper provides falsifiable handles: ablation necessity/sufficiency, recovered Markov spectra, rotational operators, and steering-induced grammatical inversion. The entity itself is a mathematical construct, not a new physical object.

pith-pipeline@v1.3.0-alltime-deepseek · 20833 in / 15529 out tokens · 156813 ms · 2026-08-02T20:39:48.110499+00:00 · methodology

0 comments
read the original abstract

Training selects for behavior, not circuitry: many weight configurations can implement the same function. Studying any single trained neural network thus risks describing accidents of one training run rather than the computation itself. This work shifts focus from what transformers happen to do to what they must do by extracting algorithmic cores, compact subspaces that are necessary and sufficient for a task and that recur across independently trained models. Here, Algorithmic Core Extraction (ACE) is introduced to isolate these subspaces, causally validate them, and recover the algorithms they implement across settings ranging from synthetic tasks to large-scale pretrained models. Markov-chain transformers embed three-dimensional cores in nearly orthogonal subspaces yet recover identical transition spectra. Modular-addition transformers form compact cyclic cores at grokking that later inflate under continued regularization, redundantly distributing the same computation across many functionally equivalent modes. This functional redundancy is found to accelerate the transition from memorization to generalization, yielding an inverse scaling law for grokking time. In six language models spanning more than two orders of magnitude in scale (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, and Qwen2.5), subject-verb agreement is governed by a single, steerable axis that aligns across architectures. Flipping this axis inverts grammatical number throughout open-ended generation. Together these results suggest that beneath the apparent complexity of trained transformers lies a simpler, shared computational structure, and that targeting invariants rather than parameterizations may offer a more tractable path to mechanistic understanding and control. Code: https://github.com/joshseth/cores

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 18 linked inside Pith

  1. [1]

    A mathematical frame- work for transformer circuits.Transformer Circuits Thread, 2021

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernan- dez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, JackClark, JaredKaplan, SamMcCan- dlish, and Chris Olah. ...

    2021

  2. [2]

    Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496, 2025

    Lee Sharkey, Bilal Chughtai, Joshua Bat- son, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496, 2025. 1

    Pith/arXiv arXiv 2025

  3. [3]

    Essentially no barriers in neural network energy landscape

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. InInternational conference on machine learn- ing, pages 1309–1318. PMLR, 2018. 1

    2018

  4. [4]

    Loss surfaces, mode connectivity, and fast ensembling of dnns.Advances in neural information processing systems, 31, 2018

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns.Advances in neural information processing systems, 31, 2018. 1, 14 20 Transformers Converge to Invariant Algorithmic CoresSchiffman, J.S

    2018

  5. [5]

    Git re-basin: Merging models modulo permutation symmetries.arXiv preprint arXiv:2209.04836, 2022

    Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries.arXiv preprint arXiv:2209.04836, 2022. 1, 14

    Pith/arXiv arXiv 2022

  6. [6]

    Everything, everywhere, all at once: Is mechanistic interpretability iden- tifiable?arXiv preprint arXiv:2502.20914,

    Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. Everything, everywhere, all at once: Is mechanistic interpretability iden- tifiable?arXiv preprint arXiv:2502.20914,

    Pith/arXiv arXiv

  7. [7]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMlR, 2019. 1

    2019

  8. [8]

    Statistical modeling: The two cultures.Statistical Science, 2001

    Leo Breiman. Statistical modeling: The two cultures.Statistical Science, 2001. 1

    2001

  9. [9]

    De- generacy and complexity in biological systems

    Gerald M Edelman and Joseph A Gally. De- generacy and complexity in biological systems. Proceedings of the national academy of sci- ences, 98(24):13763–13768, 2001. 1

    2001

  10. [10]

    Sys- temdriftandspeciation.Evolution,76(2):236– 251, 2022

    Joshua S Schiffman and Peter L Ralph. Sys- temdriftandspeciation.Evolution,76(2):236– 251, 2022. 1, 13

    2022

  11. [11]

    Canonical structure of linear dynamical systems.Proceedings of the Na- tional Academy of Sciences, 48(4):596–600,

    Rudolf E Kalman. Canonical structure of linear dynamical systems.Proceedings of the Na- tional Academy of Sciences, 48(4):596–600,

  12. [12]

    Mathematical descrip- tionoflineardynamicalsystems.Journalofthe Society for Industrial and Applied Mathemat- ics, Series A: Control, 1(2):152–192, 1963

    Rudolf Emil Kalman. Mathematical descrip- tionoflineardynamicalsystems.Journalofthe Society for Industrial and Applied Mathemat- ics, Series A: Control, 1(2):152–192, 1963. 1, 15

    1963

  13. [13]

    Universal neurons in gpt2 language models

    Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hath- away, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181, 2024. 2

    Pith/arXiv arXiv 2024

  14. [14]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 2

    2017

  15. [15]

    Insights on representational similarity in neural networks with canonical correlation.Advances in neural information processing systems, 31,

    Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation.Advances in neural information processing systems, 31,

  16. [16]

    Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    Alethea Power, Yuri Burda, Harri Ed- wards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022. 5

    Pith/arXiv arXiv 2022

  17. [17]

    Towards understanding grokking: An effec- tive theory of representation learning.Ad- vances in Neural Information Processing Sys- tems, 35:34651–34663, 2022

    Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effec- tive theory of representation learning.Ad- vances in Neural Information Processing Sys- tems, 35:34651–34663, 2022. 5

    2022

  18. [18]

    Progress measures for grokking via mechanistic inter- pretability.arXiv preprint arXiv:2301.05217,

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic inter- pretability.arXiv preprint arXiv:2301.05217,

    Pith/arXiv arXiv

  19. [19]

    A toy model of universality: Reverse engineering how networks learn group opera- tions

    Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group opera- tions. InInternational Conference on Machine Learning, pages 6243–6267. PMLR, 2023. 7, 8

    2023

  20. [20]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks.Advances in neural information pro- cessing systems, 36:27223–27250, 2023

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks.Advances in neural information pro- cessing systems, 36:27223–27250, 2023. 7

    2023

  21. [21]

    A toy model of mechanistic (un)faithfulness, August 2025

    Chris Olah. A toy model of mechanistic (un)faithfulness, August 2025. Transformer Circuits. Accessed 2026-02-18. 7

    2025

  22. [22]

    Explain- ing grokking through circuit efficiency.arXiv preprint arXiv:2309.02390, 2023

    Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explain- ing grokking through circuit efficiency.arXiv preprint arXiv:2309.02390, 2023. 7, 8

    Pith/arXiv arXiv 2023

  23. [23]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Ols- son, NicholasSchiefer, TomHenighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. 2022. Online. 8

    2022

  24. [24]

    Omnigrok: Grokking beyond algorithmic data

    Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117, 2022. 8 21 Transformers Converge to Invariant Algorithmic CoresSchiffman, J.S

    Pith/arXiv arXiv 2022

  25. [25]

    Lan- guage models are unsupervised multitask learn- ers.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Lan- guage models are unsupervised multitask learn- ers.OpenAI blog, 1(8):9, 2019. 9

    2019

  26. [26]

    Transformers: State- of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond,ClementDelangue,Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. InPro- ceedings of the 2020 conference on empirical methods in natural language processing: sys- tem demonstrations, pages 38–45, 2020. 9

    2020

  27. [27]

    Assessing the ability of lstms to learn syntax-sensitive dependencies.Trans- actions of the Association for Computational Linguistics, 4:521–535, 2016

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of lstms to learn syntax-sensitive dependencies.Trans- actions of the Association for Computational Linguistics, 4:521–535, 2016. 9

    2016

  28. [28]

    Targeted syntactic evaluation of language models

    Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 conference on empir- ical methods in natural language processing, pages 1192–1202, 2018. 9

    2018

  29. [29]

    Causal analysis of syntac- tic agreement mechanisms in neural language models

    Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart M Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntac- tic agreement mechanisms in neural language models. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume...

  30. [30]

    The linear representation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658, 2023

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658, 2023. 10

    Pith/arXiv arXiv 2023

  31. [31]

    Probing classifiers: Promises, shortcomings, and advances.Com- putational Linguistics, 48(1):207–219, 2022

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Com- putational Linguistics, 48(1):207–219, 2022. 10

    2022

  32. [32]

    Amnesic probing: Behav- ioral explanation with amnesic counterfactuals

    Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behav- ioral explanation with amnesic counterfactuals. Transactions of the Association for Computa- tional Linguistics, 9:160–175, 2021. 10

    2021

  33. [33]

    Is this the subspace you are look- ing for? an interpretability illusion for sub- space activation patching.arXiv preprint arXiv:2311.17030, 2023

    Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are look- ing for? an interpretability illusion for sub- space activation patching.arXiv preprint arXiv:2311.17030, 2023. 10

    Pith/arXiv arXiv 2023

  34. [34]

    Zoom in: An in- troduction to circuits.Distill, 2020

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An in- troduction to circuits.Distill, 2020. https://distill.pub/2020/circuits/zoom-in. 12

    2020

  35. [35]

    Interpretability in the wild: a circuit for indi- rect object identification in gpt-2 small.arXiv preprint arXiv:2211.00593, 2022

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indi- rect object identification in gpt-2 small.arXiv preprint arXiv:2211.00593, 2022. 12

    Pith/arXiv arXiv 2022

  36. [36]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Mar- cus, Michael Sklar, Adly Templeton, Tren- ton Bricken, Callum McDougall, Hoagy Cun- ningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kel- ...

    2025

  37. [37]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abra- hams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Ri...

  38. [38]

    Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom 22 Transformers Converge to Invariant Algorithmic ...

    2022

  39. [39]

    Emergent world representations: Exploring a sequence model trained on a syn- thetic task.arXiv preprint arXiv:2210.13382,

    Kenneth Li, Aspen K Hopkins, David Bau, Fer- nanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a syn- thetic task.arXiv preprint arXiv:2210.13382,

    Pith/arXiv arXiv

  40. [40]

    Language models represent space and time.arXiv preprint arXiv:2310.02207, 2023

    Wes Gurnee and Max Tegmark. Language models represent space and time.arXiv preprint arXiv:2310.02207, 2023. 12

    Pith/arXiv arXiv 2023

  41. [41]

    The platonic representation hypothesis.arXiv preprint arXiv:2405.07987, 2024

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.arXiv preprint arXiv:2405.07987, 2024. 12

    Pith/arXiv arXiv 2024

  42. [42]

    Every good regulator of a system must be a model of that system.International journal of systems science, 1(2):89–97, 1970

    Roger C Conant and W Ross Ashby. Every good regulator of a system must be a model of that system.International journal of systems science, 1(2):89–97, 1970. 12

    1970

  43. [43]

    The internal model principle of control theory

    Bruce A Francis and Walter Murray Wonham. The internal model principle of control theory. Automatica, 12(5):457–465, 1976. 13

    1976

  44. [44]

    Sparse autoencoders find highly interpretable fea- tures in language models.arXiv preprint arXiv:2309.08600, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable fea- tures in language models.arXiv preprint arXiv:2309.08600, 2023. 13

    Pith/arXiv arXiv 2023

  45. [45]

    Towards monoseman- ticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2023

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield- Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and C...

    2023

  46. [46]

    Transcoders find interpretable llm fea- ture circuits.Advances in Neural Information Processing Systems, 37:24375–24410, 2024

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm fea- ture circuits.Advances in Neural Information Processing Systems, 37:24375–24410, 2024. 13

    2024

  47. [47]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Mar- cus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monose...

    2024

  48. [48]

    Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models

    Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Watten- berg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. arXiv preprint arXiv:2502.12892, 2025. 13

    Pith/arXiv arXiv 2025

  49. [49]

    Sparsecrosscodersforcross- layer features and model diffing.Transformer Circuits Thread, pages 3982–3992, 2024

    Jack Lindsey, Adly Templeton, Jonathan Mar- cus, Thomas Conerly, Joshua Batson, and ChristopherOlah. Sparsecrosscodersforcross- layer features and model diffing.Transformer Circuits Thread, pages 3982–3992, 2024. 13

    2024

  50. [50]

    Using degeneracy in the loss landscape for mechanistic interpretability.arXiv preprint arXiv:2405.10927, 2024

    Lucius Bushnaq, Jake Mendel, Stefan Heimer- sheim, Dan Braun, Nicholas Goldowsky-Dill, Kaarel Hänni, Cindy Wu, and Marius Hobb- hahn. Using degeneracy in the loss landscape for mechanistic interpretability.arXiv preprint arXiv:2405.10927, 2024. 13

    Pith/arXiv arXiv 2024

  51. [51]

    Robustness and evolvability: a paradox resolved.Proceedings of the Royal Society B: Biological Sciences, 275(1630):91– 100, 2008

    Andreas Wagner. Robustness and evolvability: a paradox resolved.Proceedings of the Royal Society B: Biological Sciences, 275(1630):91– 100, 2008. 13

    2008

  52. [52]

    The role of robustness in phenotypic adaptation and innovation.Pro- ceedings of the Royal Society B: Biological Sciences, 279(1732):1249–1258, 2012

    Andreas Wagner. The role of robustness in phenotypic adaptation and innovation.Pro- ceedings of the Royal Society B: Biological Sciences, 279(1732):1249–1258, 2012. 13

    2012

  53. [53]

    Lora: Low- rank adaptation of large language models.Iclr, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models.Iclr, 1(2):3, 2022. 14 23 Transformers Converge to Invariant Algorithmic CoresSchiffman, J.S

    2022

  54. [54]

    Modern koopman theory for dynamical systems.arXiv preprint arXiv:2102.12086, 2021

    Steven L Brunton, Marko Budišić, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems.arXiv preprint arXiv:2102.12086, 2021. 14

    Pith/arXiv arXiv 2021

  55. [55]

    On structural identifiability.Mathematical biosciences, 7(3- 4):329–339, 1970

    Karl Johan Bellman, Richard. On structural identifiability.Mathematical biosciences, 7(3- 4):329–339, 1970. 14

    1970

  56. [56]

    Equivalence of linear time- invariant dynamical systems.Journal of the Franklin Institute, 281(5):371–378, 1966

    BDO Anderson, RW Newcomb, RE Kalman, and DC Youla. Equivalence of linear time- invariant dynamical systems.Journal of the Franklin Institute, 281(5):371–378, 1966. 15

    1966

  57. [57]

    Kalman, Peter L

    Rudolf E. Kalman, Peter L. Falb, and Michael A. Arbib.Topics in mathematical system theory. McGraw-Hill, New York, 1969. 15

    1969

  58. [58]

    Principal component analysis in linear systems: Controllability, observability, and model reduction.IEEE transactions on automatic control, 26(1):17–32, 2003

    Bruce Moore. Principal component analysis in linear systems: Controllability, observability, and model reduction.IEEE transactions on automatic control, 26(1):17–32, 2003. 15 24

    2003