Embryology of a Language Model

Andrew Gordon; Daniel Murfet; Garrett Baker; George Wang

REVIEW 2 major objections 5 minor 1 cited by

Embryology of a Language Model

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UMAP on susceptibility vectors reveals a 'spacing fin' that tracks space-token counts.

desk verdict The spacing fin is a real new visual structure, but the 'counting' claim outruns the evidence—Figure 5's gradient is averaged over token types and datasets, so the paper needs review but with a demand for stratification. read the letter →

arxiv 2508.00331 v1 pith:USSYQSOH submitted 2025-08-01 cs.LG

George Wang , Garrett Baker , Andrew Gordon , Daniel Murfet This is my paper

classification cs.LG

keywords susceptibilityanalysisUMAPinterpretabilitydevelopmentalstagesinductioncircuitspacingfinsmalltransformerstructuralinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small transformer's internal organization can be watched as it develops by projecting per-token susceptibility vectors into two dimensions with UMAP. The projection forms a 'body plan' stratified by token pattern, the known induction circuit appears as a thickening along the dorsal-ventral axis, and a previously unseen 'spacing fin' emerges in which token sequences predicting a space after many spaces separate from the main body. The authors argue this shows susceptibility analysis can discover new computational structure, not only confirm known circuits, and that the spacing fin indicates the model learns to count preceding space tokens. A sympathetic reader would care because it offers a holistic, visual route into how computational structures arise during training.

What carries the argument

The machinery is the per-token susceptibility vector, $\eta_w(xy) = (\chi^{C_1}_{xy}, \ldots, \chi^{C_H}_{xy})$, whose entries measure how each attention head's weights covary with the token's log-loss under the quenched posterior; UMAP projects these vectors from 16 dimensions into two. Applied at four training checkpoints, the projection turns the token distribution into an embryological image, and the paper reads anatomical axes (anterior-posterior, dorsal-ventral) off principal components of the same data.

What would settle it

A direct test: sample token sequences with the same number of preceding spaces but different token frequencies and context lengths, embed them with the same susceptibility pipeline, and check whether fin position tracks space count once frequency and length are controlled. If it does not, the fin is confounded; if it does, a second test would ablate the heads with largest loadings along the fin direction and see whether predicted probabilities for space tokens stop depending on run length.

Watch

Extended reading notes

Core claim

The central discovery is a new computational structure: the 'spacing fin.' At the end of training, the UMAP embedding of 16-dimensional susceptibility vectors contains a fin-shaped cluster of token sequences in which the next token is a spacing token (space, newline, tab, etc.) and the context ends with spacing tokens. Moving outward along the fin, tokens are preceded by increasing numbers of spacing tokens, up to hundreds; the susceptibility vectors themselves shift in a consistent direction as the run length grows (Figures 4 and 5). The paper interprets this as evidence that the model develops distributed computational structure, not localized to one head, for differentiating and counting spacing tokens, while noting that a complete mechanistic explanation is not yet available. The same visual method also recovers known structure: the induction circuit shows up as a ventral-to-dorsal color stratification that thickens during the stages where induction heads form.

Load-bearing premise

The load-bearing premise is that the ordering of tokens along the spacing fin really encodes the number of preceding spacing tokens; if that gradient is an artifact of token frequency, context length, or UMAP distortion, the counting claim weakens, and the authors state they lack a complete mechanistic explanation for the fin.

Editorial extensions

If this is right

If the spacing fin is real, susceptibility UMAPs can reveal computational structures that circuit-level analysis misses, because the fin is not carried by a small subset of heads.
The emergence timing of the fin (separation by end of stage LM3, reattachment by end of training) becomes a developmental milestone that future mechanistic accounts of space counting must explain.
The universality of the body plan across four seeds suggests the high-level functional organization of a transformer is largely determined by architecture and data, even when the specific heads differ.
Because susceptibilities connect to per-sample generalization error through local learning coefficients, structures like the spacing fin have a direct link to how the model generalizes on sequences with long runs of spaces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The spacing fin may be a discrete counter implemented across heads; one test is to feed synthetic contexts with controlled run lengths and check that the model's next-token distribution over spacing tokens changes monotonically with run length.
Editorial inference: Similar fins may exist for other structural tokens—newlines, tabs, indentation in code—and could be searched for by conditioning UMAP coloring on those categories in code-heavy datasets.
Editorial inference: Since the authors report the fin's formation coincides with specific developmental stages, comparing fin emergence across architectures and tokenizers could reveal whether space counting is a general developmental phase or an artifact of the truncated GPT-2 tokenizer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper is worth reading for the method and for the spacing fin observation, but the abstract's claim that the fin is "dedicated to counting space tokens" is not yet supported. The monotone gradient in Figure 5 is averaged over all spacing tokens with a given minimum number of preceding spaces. Long space runs are rare and concentrated in code-like data, where the predicted spacing token is likely a tab or newline rather than a single space. Since Figure 10 shows those token types land in different parts of the fin, the gradient could simply reflect a changing mixture of y tokens and datasets as s grows. The paper never stratifies by y identity or dataset, so this is a real, testable confound.

What is genuinely new: applying UMAP to per-token susceptibility vectors is a natural but useful step, and it does produce a legible "body plan" that develops over training. The spacing fin is a new observation, not present in the earlier susceptibility papers, and the developmental story—ejection from the body, reattachment, outward gradient with increasing space count—is clearly described and robust enough across UMAP hyperparameters and four seeds to be taken seriously. The authors also deserve credit for an honest limitations section: they explicitly say they lack a mechanism and note tokenizer dependence.

The soft spots, in proportion: the counting interpretation is the load-bearing claim and it is correlational, exactly as the stress-test says. A revision should either stratify Figure 5 by y-token type and dataset and show the gradient persists, or soften the language to "correlated with space count." No code or data is provided, which makes it hard to check the confound independently. The induction-circuit portion re-observes known structure; it is a fine validation but not a new result. Heavy self-citation is not a problem here because the model and susceptibility method are from the prior papers.

Who this is for: people working on interpretability for small transformers, or on how structure emerges during training. It is a solid, honest paper with one overstated claim. I would send it to peer review, but ask the authors to stratify the spacing fin gradient and temper the abstract. That is a modest, achievable revision, not a knockdown.

Referee Report

2 major / 5 minor

Summary. This paper proposes an 'embryological' visualization method for language models: it applies UMAP to per-token susceptibility vectors (one per attention head) at several training checkpoints of a 3M-parameter attention-only transformer, producing projections the authors call the 'rainbow serpent.' The paper reports that the projection is stratified by a pre-existing ontology of token patterns, that the emergence of the induction circuit is visible as dorsal-ventral thickening, and that a previously unnoticed cluster of spacing tokens, the 'spacing fin,' develops over training. The central novelty claim is that the spacing fin is 'dedicated to counting space tokens' (Abstract; Section 4.3), with the main evidence being a monotonic gradient in susceptibility as the number of preceding spacing tokens increases (Figures 4 and 5). The authors state they lack a complete mechanistic explanation for the spacing fin, so the counting interpretation rests primarily on this correlational gradient.

Significance. If the central claim holds, the spacing fin would be a genuinely new computational structure discovered through susceptibility-based structural inference, complementing circuit-centric interpretability. The paper has real strengths: the UMAP robustness checks across n_neighbors and min_dist (Appendix C.1) are careful; the multi-seed comparison (Appendix E) lends credence to the qualitative universality of the body plan; and the paper presents quantitative per-pattern susceptibility curves that support the visual narrative. The induction-circuit section also serves as a useful validation of the method against known results. The main weakness is that the spacing-fin 'counting' interpretation is not yet supported at the level claimed, because the key gradient is not controlled for token type or dataset identity. With additional controls or a weakened claim, the paper would be a solid contribution to interpretability methodology.

major comments (2)

[Section 4.3, Figure 5] The central claim that the spacing fin is 'dedicated to counting space tokens' rests on the monotonic gradient in Figure 5, which averages per-token susceptibilities over all spacing tokens with a given minimum number s of preceding spacing tokens, without conditioning on the identity of y (e.g., space, newline, tab) or on the dataset. Long spacing runs are rare (Appendix D, Figure 11) and are concentrated in code-like data such as GITHUB-CODE (Figure 6), and Figure 10 shows that single spaces, newlines, and tabs occupy different parts of the fin. It is therefore possible that the gradient in Figure 5 reflects a changing mixture of token types and datasets as s increases, rather than the model tracking the count s. The authors should stratify Figure 5 by y-token type and by dataset, or provide a partial-correlation or regression analysis that controls for these factors. If the gradient disappears under stratification, the counting claim should be withdrawn or weakened.
[Section 4.3 (interpretation)] The phrasing 'dedicated to counting space tokens' (Abstract and Section 4.3) goes beyond the evidence presented. The authors themselves state that they 'lack a complete mechanistic explanation of how this structure operates,' so the paper offers only correlational evidence that susceptibility vectors vary with the number of preceding spaces. At minimum, the claim should be reframed as 'the spacing fin encodes or tracks the length of the preceding spacing run,' and, ideally, a causal or mechanistic test should be added, such as an intervention on the context length or a targeted analysis of the heads most responsible for the gradient in Figure 12. This distinction matters for the paper's stated contribution of 'uncovering novel mechanisms.'

minor comments (5)

[Section 6] In the first paragraph, 'exhibition / exhibition' appears to be a typo for 'excitation / inhibition'; please correct it.
[Section 4.3] The sentence 'Another interesting development phenomenon is the the reattachment of the spacing fin' contains a doubled 'the'; please fix the typo.
[Appendix C.1] The robustness check across UMAP hyperparameters is valuable, but it would help to state explicitly how many n_neighbors and min_dist settings were tested and whether the spacing fin remained separated in all of them, rather than only mentioning 'a range.'
[Figure 5] Figure 5 reports averages conditional on the minimum number of preceding spacing tokens, but it does not show the number of samples in each bin or error bars. Given the heavy-tailed distribution in Figure 11, some bins may be very small, and adding counts or confidence intervals would strengthen the quantitative claim.
[Table 2] The dataset label 'PILE-PILE-CC' appears to have a duplicated prefix; please check whether this is intended and, if not, correct it to 'PILE-CC' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the spacing fin is an empirical correlation between an independently defined context count and susceptibility geometry; the self-cited susceptibility method is prior work with independent validation.

full rationale

The paper's central new claim is the discovery of a 'spacing fin' in UMAP projections of susceptibility vectors. The count variable s (number of preceding spacing tokens) is defined directly from the token sequence, independently of the susceptibility vectors and of the UMAP embedding. Figures 4 and 5 display a correlation between s and position in susceptibility space; this is a data-driven observation, not an identity or a fitted parameter renamed as prediction. The authors explicitly hedge the mechanistic interpretation ('this suggests', 'we lack a complete mechanistic explanation'), so the counting claim is not forced by construction. The susceptibility method is taken from Baker et al. 2025, a self-cited prior paper, but the central derivation here does not reduce to that citation: the spacing fin is not defined in terms of susceptibilities in a way that would make its discovery tautological. The induction-circuit validation also has independent content: the previous-token and induction heads were identified in Hoogland et al. 2025 and Wang et al. 2024 using prefix/ICL scores and K-composition, not by the susceptibility PCA being validated, and the canonical induction-circuit account is external (Olsson et al. 2022). The paper contains no fitted parameter that is subsequently reported as a prediction, no author-imported uniqueness theorem, and no ansatz smuggled in via citation. Robustness checks against UMAP hyperparameters and PCA comparisons further support that the reported structure is not an artifact of the embedding choice. Therefore no step in the derivation chain is circular by the standards of this review.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new free parameters are fitted to data; the UMAP hyperparameters are chosen with robustness checks. The key inherited assumptions are the SGLD susceptibility estimator and the token-pattern ontology, both from prior work by the same authors. The 'spacing fin' is an observed cluster, not a postulated entity, so no invented entities are listed.

free parameters (2)

UMAP n_neighbors = 45
Chosen by the authors; robustness checked from 15 to ~8000 effective, but it is a hand-set hyperparameter.
UMAP min_dist = 0.1
Chosen by the authors; they state varying it up to 0.5 produced no substantial differences.

assumptions (3)

domain assumption SGLD with gamma=300, n*beta=30, epsilon=0.001, 4 chains, 100 draws yields accurate per-token susceptibility estimates.
Appendix B.2, inherited from Baker et al. (2025).
domain assumption UMAP with n_neighbors=45 preserves the local structure relevant to the claims.
Appendix C.1 states robustness checks were done, but UMAP is known to distort global distances.
domain assumption The eight token patterns in Table 1 are a meaningful ontology for the model's computations.
Section 2.1 and Appendix A; the patterns are defined by hand and the induction pattern uses a threshold q(y|x) <= 0.05.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Embryology of a Language Model." pith.science (2026). https://pith.science/paper/USSYQSOH

@misc{pith2026250800331,
  author       = {Pith},
  title        = {Pith review of: Embryology of a Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USSYQSOH}},
  note         = {Machine review of arXiv:2508.00331}
}

read the original abstract

Understanding how language models develop their internal computational structure is a central problem in the science of deep learning. While susceptibilities, drawn from statistical physics, offer a promising analytical tool, their full potential for visualizing network organization remains untapped. In this work, we introduce an embryological approach, applying UMAP to the susceptibility matrix to visualize the model's structural development over training. Our visualizations reveal the emergence of a clear ``body plan,'' charting the formation of known features like the induction circuit and discovering previously unknown structures, such as a ``spacing fin'' dedicated to counting space tokens. This work demonstrates that susceptibility analysis can move beyond validation to uncover novel mechanisms, providing a powerful, holistic lens for studying the developmental principles of complex neural networks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Influence Dynamics and Stagewise Data Attribution
cs.LG 2025-10 conditional novelty 7.0 of 10

Using Bayesian influence functions and singular learning theory, the authors show that a sample's influence on a model varies non-monotonically over training, peaking and flipping sign at phase transitions.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages · cited by 1 Pith paper

[1]

Altland and B

A. Altland and B. D. Simons. Condensed matter field theory. Cambridge U niversity P ress, 2010

work page 2010
[2]

Arnold, S

V. Arnold, S. Gusein-Zade, and A. Varchenko. Singularities of differentiable maps, V olume 1: classification of critical points, caustics and wave fronts . Birkhäuser, 1985

work page 1985
[3]

Baker, G

G. Baker, G. Wang, J. Hoogland, and D. Murfet. Structural inference: Interpreting small language models with susceptibilities, 2025. URL https://arxiv.org/abs/2504.18274

arXiv 2025
[4]

Bengio, A

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 0 (8): 0 1798--1828, 2013

work page 2013
[5]

Bricken, A

T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary...

2023
[6]

J. Cao, M. Spielmann, X. Qiu, X. Huang, D. M. Ibrahim, A. J. Hill, F. Zhang, S. Mundlos, L. Christiansen, F. J. Steemers, et al. The single-cell transcriptional landscape of mammalian organogenesis. Nature, 566 0 (7745): 0 496--502, 2019

work page 2019
[7]

Carter, Z

S. Carter, Z. Armstrong, L. Schubert, I. Johnson, and C. Olah. Activation atlas. Distill, 2019. doi:10.23915/distill.00015. https://distill.pub/2019/activation-atlas

doi:10.23915/distill.00015 2019
[8]

Chari and L

T. Chari and L. Pachter. The specious art of single-cell genomics. PLOS Computational Biology, 19 0 (8): 0 e1011288, 2023

work page 2023

Show all 26 references

[9]

Eldan and Y

R. Eldan and Y. Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

2023 arXiv
[10]

Elhage, N

N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, an...

2021
[11]

Engels, E

J. Engels, E. J. Michaud, I. Liao, W. Gurnee, and M. Tegmark. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=d63a4AM4hb

2025
[12]

Fefferman, S

C. Fefferman, S. Mitter, and H. Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29 0 (4): 0 983--1049, 2016

2016
[13]

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: an 800GB dataset of diverse text for language modeling. Preprint arXiv:2101.00027 [cs.CL], 2020

2020 arXiv
[14]

Hewitt and C

J. Hewitt and C. D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page...

2019
[15]

Hoogland, G

J. Hoogland, G. Wang, M. Farrugia-Roberts, L. Carroll, S. Wei, and D. Murfet. Loss landscape degeneracy drives stagewise development in transformers, 2025. URL https://arxiv.org/abs/2402.02364

2025 arXiv
[16]

Kriegeskorte and R

N. Kriegeskorte and R. A. Kievit. Representational geometry: integrating cognition, computation, and the brain. Trends in cognitive sciences, 17 0 (8): 0 401--412, 2013

2013
[17]

Y. Li, E. J. Michaud, D. D. Baek, J. Engels, X. Sun, and M. Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27 0 (4): 0 344, 2025

2025
[18]

McInnes, J

L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. URL https://arxiv.org/abs/1802.03426

2020 arXiv
[19]

Olsson, N

C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. M...

2022
[20]

E. Reif, A. Yuan, M. Wattenberg, F. B. Viegas, A. Coenen, A. Pearce, and B. Kim. Visualizing and measuring the geometry of bert. Advances in neural information processing systems, 32, 2019

2019
[21]

Rubin, L

R. Rubin, L. F. Abbott, and H. Sompolinsky. Balanced excitation and inhibition are required for high-capacity, noise-robust neuronal selectivity. Proceedings of the National Academy of Sciences, 114 0 (44): 0 E9366--E9375, 2017

2017
[22]

R. Thom. Structural stability and morphogenesis. W. A. Benjamin, Inc., 1972

1972
[23]

G. Wang, J. Hoogland, S. van Wingerden, Z. Furman, and D. Murfet. Differentiation and specialization of attention heads via the refined local learning coefficient, 2024. URL https://arxiv.org/abs/2410.02984

2024 arXiv
[24]

Watanabe

S. Watanabe. Algebraic geometry and statistical learning theory. Cambridge University Press, 2009

2009
[25]

S. M. Xie, S. Santurkar, T. Ma, and P. Liang. Data selection for language models via importance resampling. Preprint arXiv:2302.03169 [cs.CL], 2023

2023 arXiv
[26]

Zhang and S

B. Zhang and S. Horvath. A general framework for weighted gene co-expression network analysis. Statistical applications in genetics and molecular biology, 4 0 (1), 2005

2005

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Altland and B

A. Altland and B. D. Simons. Condensed matter field theory. Cambridge U niversity P ress, 2010

work page 2010

[2] [2]

Arnold, S

V. Arnold, S. Gusein-Zade, and A. Varchenko. Singularities of differentiable maps, V olume 1: classification of critical points, caustics and wave fronts . Birkhäuser, 1985

work page 1985

[3] [3]

Baker, G

G. Baker, G. Wang, J. Hoogland, and D. Murfet. Structural inference: Interpreting small language models with susceptibilities, 2025. URL https://arxiv.org/abs/2504.18274

arXiv 2025

[4] [4]

Bengio, A

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 0 (8): 0 1798--1828, 2013

work page 2013

[5] [5]

Bricken, A

T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary...

2023

[6] [6]

J. Cao, M. Spielmann, X. Qiu, X. Huang, D. M. Ibrahim, A. J. Hill, F. Zhang, S. Mundlos, L. Christiansen, F. J. Steemers, et al. The single-cell transcriptional landscape of mammalian organogenesis. Nature, 566 0 (7745): 0 496--502, 2019

work page 2019

[7] [7]

Carter, Z

S. Carter, Z. Armstrong, L. Schubert, I. Johnson, and C. Olah. Activation atlas. Distill, 2019. doi:10.23915/distill.00015. https://distill.pub/2019/activation-atlas

doi:10.23915/distill.00015 2019

[8] [8]

Chari and L

T. Chari and L. Pachter. The specious art of single-cell genomics. PLOS Computational Biology, 19 0 (8): 0 e1011288, 2023

work page 2023