{"id":"fb6284cd-f637-454b-a697-e1ccf662bd4d","arxiv_id":"2508.00331","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"UMAP projections of susceptibility vectors reveal a reproducible 'body plan' in a small transformer, including a newly identified 'spacing fin' that distinguishes tokens by the number of preceding spaces.","lead":"This paper watches a tiny language model grow up: it maps the model's sixteen attention heads with UMAP to see how they organize over training. The maps reveal a consistent 'body plan' and a previously unseen 'spacing fin' that tracks runs of space tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Spacing fin count-gradient may be confounded by token type and dataset; long space runs are concentrated in code, so the 'counting' interpretation is not yet established.","rationale":"The reader's weakest assumption identified the correlational nature of the fin-counting claim and suggested frequency/context-length confounds. I agree, but I want to point to a more specific and testable confound: token type and dataset membership. Because s (number of preceding spaces) is strongly correlated with whether y is a space, newline, or tab, and with whether the context comes from code-heavy or prose-heavy datasets, the Figure 5 gradient may be a composition effect. The paper's own Figure 10 and Appendix G show that different spacing-token types occupy distinct parts of the UMAP, and the frequency histogram (Figure 11) shows very long runs are rare, so they likely come from a narrow set of sources. Since the spacing fin is the paper's one 'previously unknown structure' and the abstract asserts it is 'dedicated to counting,' this confound directly threatens the central claim. The proposed test is cheap and decisive: stratify Figure 5. If the gradient persists within every stratum, the counting interpretation is substantially strengthened; if not, the claim must be weakened to 'the fin varies with the composition of spacing contexts,' which is a much weaker novelty. I therefore keep the reader's CONDITIONAL verdict, with the condition being the stratified analysis. The paper's other contributions (UMAP embryology of susceptibilities, induction visualization) are not affected by this concern, which is why I do not recommend REJECT.","tokens_in":21921,"tokens_out":6626,"duration_ms":68899,"concrete_test":"Recompute Figure 5 separately for each spacing token type (single space, newline, tab/other) and for each of the 13 datasets (or at least GITHUB_CODE vs WIKIPEDIA_EN), using the same 260k sampled rows. If the monotonic relationship between minimum preceding-space count and average susceptibility does not persist within each stratum, the fin's count-gradient is confounded and the 'counting' interpretation in the abstract should be withdrawn or substantially softened. Report the within-stratum gradients and, if possible, partial correlations controlling for y-token identity, dataset, and context length.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central novel claim is that the spacing fin is 'dedicated to counting space tokens' (Abstract; Section 4.3). The evidence is the correlational gradient in Figures 4 and 5: as the number s of immediately preceding spacing tokens increases, spacing-token susceptibility vectors move monotonically outward along the fin. This gradient is load-bearing because the authors state they 'lack a complete mechanistic explanation' (Section 4.3), so the counting interpretation rests entirely on that correlation. The problem is that s is not independent of other properties. Long runs of spaces are rare (Figure 11) and occur predominantly in code-like data (e.g., GITHUB_CODE, indentation), where the predicted spacing token y is often a tab or newline rather than a single space. Figure 10 shows these token types occupy different parts of the fin. Figure 5 averages over all spacing tokens with a given minimum s without conditioning on the identity of y or the dataset; thus the monotonic gradient could be produced by a changing mixture of token types and datasets across s, rather than by the model tracking the count s. If the gradient disappears when stratified by y-token type and dataset, the spacing fin is not a counting structure but a reflection of token/dataset identity. This is a concrete, testable confound, and it directly undermines the abstract's claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an 'embryological' visualization method for language models: it applies UMAP to per-token susceptibility vectors (one per attention head) at several training checkpoints of a 3M-parameter attention-only transformer, producing projections the authors call the 'rainbow serpent.' The paper reports that the projection is stratified by a pre-existing ontology of token patterns, that the emergence of the induction circuit is visible as dorsal-ventral thickening, and that a previously unnoticed cluster of spacing tokens, the 'spacing fin,' develops over training. The central novelty claim is that the spacing fin is 'dedicated to counting space tokens' (Abstract; Section 4.3), with the main evidence being a monotonic gradient in susceptibility as the number of preceding spacing tokens increases (Figures 4 and 5). The authors state they lack a complete mechanistic explanation for the spacing fin, so the counting interpretation rests primarily on this correlational gradient.","tokens_in":22173,"tokens_out":3434,"duration_ms":38720,"significance":"If the central claim holds, the spacing fin would be a genuinely new computational structure discovered through susceptibility-based structural inference, complementing circuit-centric interpretability. The paper has real strengths: the UMAP robustness checks across n_neighbors and min_dist (Appendix C.1) are careful; the multi-seed comparison (Appendix E) lends credence to the qualitative universality of the body plan; and the paper presents quantitative per-pattern susceptibility curves that support the visual narrative. The induction-circuit section also serves as a useful validation of the method against known results. The main weakness is that the spacing-fin 'counting' interpretation is not yet supported at the level claimed, because the key gradient is not controlled for token type or dataset identity. With additional controls or a weakened claim, the paper would be a solid contribution to interpretability methodology.","major_comments":[{"comment":"The central claim that the spacing fin is 'dedicated to counting space tokens' rests on the monotonic gradient in Figure 5, which averages per-token susceptibilities over all spacing tokens with a given minimum number s of preceding spacing tokens, without conditioning on the identity of y (e.g., space, newline, tab) or on the dataset. Long spacing runs are rare (Appendix D, Figure 11) and are concentrated in code-like data such as GITHUB-CODE (Figure 6), and Figure 10 shows that single spaces, newlines, and tabs occupy different parts of the fin. It is therefore possible that the gradient in Figure 5 reflects a changing mixture of token types and datasets as s increases, rather than the model tracking the count s. The authors should stratify Figure 5 by y-token type and by dataset, or provide a partial-correlation or regression analysis that controls for these factors. If the gradient disappears under stratification, the counting claim should be withdrawn or weakened.","section":"Section 4.3, Figure 5"},{"comment":"The phrasing 'dedicated to counting space tokens' (Abstract and Section 4.3) goes beyond the evidence presented. The authors themselves state that they 'lack a complete mechanistic explanation of how this structure operates,' so the paper offers only correlational evidence that susceptibility vectors vary with the number of preceding spaces. At minimum, the claim should be reframed as 'the spacing fin encodes or tracks the length of the preceding spacing run,' and, ideally, a causal or mechanistic test should be added, such as an intervention on the context length or a targeted analysis of the heads most responsible for the gradient in Figure 12. This distinction matters for the paper's stated contribution of 'uncovering novel mechanisms.'","section":"Section 4.3 (interpretation)"}],"minor_comments":[{"comment":"In the first paragraph, 'exhibition / exhibition' appears to be a typo for 'excitation / inhibition'; please correct it.","section":"Section 6"},{"comment":"The sentence 'Another interesting development phenomenon is the the reattachment of the spacing fin' contains a doubled 'the'; please fix the typo.","section":"Section 4.3"},{"comment":"The robustness check across UMAP hyperparameters is valuable, but it would help to state explicitly how many n_neighbors and min_dist settings were tested and whether the spacing fin remained separated in all of them, rather than only mentioning 'a range.'","section":"Appendix C.1"},{"comment":"Figure 5 reports averages conditional on the minimum number of preceding spacing tokens, but it does not show the number of samples in each bin or error bars. Given the heavy-tailed distribution in Figure 11, some bins may be very small, and adding counts or confidence intervals would strengthen the quantitative claim.","section":"Figure 5"},{"comment":"The dataset label 'PILE-PILE-CC' appears to have a duplicated prefix; please check whether this is intended and, if not, correct it to 'PILE-CC' or similar.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' prior work (Baker et al. 2025) for the susceptibility definitions, the model, and the token-pattern ontology; an editor may want to confirm that the manuscript is sufficiently self-contained for review. The spacing-fin confound is the key technical issue, but it is addressable with additional stratified analyses, so I do not see a need for rejection. The claim of a 'universal body plan' across seeds is also somewhat stronger than the qualitative appendix evidence supports, though this is secondary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading for the method and for the spacing fin observation, but the abstract's claim that the fin is \"dedicated to counting space tokens\" is not yet supported. The monotone gradient in Figure 5 is averaged over all spacing tokens with a given minimum number of preceding spaces. Long space runs are rare and concentrated in code-like data, where the predicted spacing token is likely a tab or newline rather than a single space. Since Figure 10 shows those token types land in different parts of the fin, the gradient could simply reflect a changing mixture of y tokens and datasets as s grows. The paper never stratifies by y identity or dataset, so this is a real, testable confound.\n\nWhat is genuinely new: applying UMAP to per-token susceptibility vectors is a natural but useful step, and it does produce a legible \"body plan\" that develops over training. The spacing fin is a new observation, not present in the earlier susceptibility papers, and the developmental story—ejection from the body, reattachment, outward gradient with increasing space count—is clearly described and robust enough across UMAP hyperparameters and four seeds to be taken seriously. The authors also deserve credit for an honest limitations section: they explicitly say they lack a mechanism and note tokenizer dependence.\n\nThe soft spots, in proportion: the counting interpretation is the load-bearing claim and it is correlational, exactly as the stress-test says. A revision should either stratify Figure 5 by y-token type and dataset and show the gradient persists, or soften the language to \"correlated with space count.\" No code or data is provided, which makes it hard to check the confound independently. The induction-circuit portion re-observes known structure; it is a fine validation but not a new result. Heavy self-citation is not a problem here because the model and susceptibility method are from the prior papers.\n\nWho this is for: people working on interpretability for small transformers, or on how structure emerges during training. It is a solid, honest paper with one overstated claim. I would send it to peer review, but ask the authors to stratify the spacing fin gradient and temper the abstract. That is a modest, achievable revision, not a knockdown.","headline":"The spacing fin is a real new visual structure, but the 'counting' claim outruns the evidence—Figure 5's gradient is averaged over token types and datasets, so the paper needs review but with a demand for stratification.","tokens_in":22670,"tokens_out":3481,"would_cite":false,"duration_ms":39300,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UMAP on susceptibility vectors reveals a 'spacing fin' that tracks space-token counts.","keywords":["susceptibility analysis","UMAP","interpretability","developmental stages","induction circuit","spacing fin","small transformer","structural inference"],"falsifier":"A direct test: sample token sequences with the same number of preceding spaces but different token frequencies and context lengths, embed them with the same susceptibility pipeline, and check whether fin position tracks space count once frequency and length are controlled. If it does not, the fin is confounded; if it does, a second test would ablate the heads with largest loadings along the fin direction and see whether predicted probabilities for space tokens stop depending on run length.","tokens_in":21730,"feed_emoji":"🧬","tokens_out":4675,"duration_ms":43116,"temperature":0.7,"pith_summary":"The paper claims that a small transformer's internal organization can be watched as it develops by projecting per-token susceptibility vectors into two dimensions with UMAP. The projection forms a 'body plan' stratified by token pattern, the known induction circuit appears as a thickening along the dorsal-ventral axis, and a previously unseen 'spacing fin' emerges in which token sequences predicting a space after many spaces separate from the main body. The authors argue this shows susceptibility analysis can discover new computational structure, not only confirm known circuits, and that the spacing fin indicates the model learns to count preceding space tokens. A sympathetic reader would care because it offers a holistic, visual route into how computational structures arise during training.","feed_headline":"UMAP reveals a spacing fin that counts space tokens","feed_subtitle":"A new UMAP view of a small transformer exposes a body plan and a 'spacing fin' that tracks preceding spaces.","key_machinery":"The machinery is the per-token susceptibility vector, $\\eta_w(xy) = (\\chi^{C_1}_{xy}, \\ldots, \\chi^{C_H}_{xy})$, whose entries measure how each attention head's weights covary with the token's log-loss under the quenched posterior; UMAP projects these vectors from 16 dimensions into two. Applied at four training checkpoints, the projection turns the token distribution into an embryological image, and the paper reads anatomical axes (anterior-posterior, dorsal-ventral) off principal components of the same data.","core_discovery":"The central discovery is a new computational structure: the 'spacing fin.' At the end of training, the UMAP embedding of 16-dimensional susceptibility vectors contains a fin-shaped cluster of token sequences in which the next token is a spacing token (space, newline, tab, etc.) and the context ends with spacing tokens. Moving outward along the fin, tokens are preceded by increasing numbers of spacing tokens, up to hundreds; the susceptibility vectors themselves shift in a consistent direction as the run length grows (Figures 4 and 5). The paper interprets this as evidence that the model develops distributed computational structure, not localized to one head, for differentiating and counting spacing tokens, while noting that a complete mechanistic explanation is not yet available. The same visual method also recovers known structure: the induction circuit shows up as a ventral-to-dorsal color stratification that thickens during the stages where induction heads form.","pith_inferences":["Editorial inference: The spacing fin may be a discrete counter implemented across heads; one test is to feed synthetic contexts with controlled run lengths and check that the model's next-token distribution over spacing tokens changes monotonically with run length.","Editorial inference: Similar fins may exist for other structural tokens—newlines, tabs, indentation in code—and could be searched for by conditioning UMAP coloring on those categories in code-heavy datasets.","Editorial inference: Since the authors report the fin's formation coincides with specific developmental stages, comparing fin emergence across architectures and tokenizers could reveal whether space counting is a general developmental phase or an artifact of the truncated GPT-2 tokenizer."],"forward_implications":["If the spacing fin is real, susceptibility UMAPs can reveal computational structures that circuit-level analysis misses, because the fin is not carried by a small subset of heads.","The emergence timing of the fin (separation by end of stage LM3, reattachment by end of training) becomes a developmental milestone that future mechanistic accounts of space counting must explain.","The universality of the body plan across four seeds suggests the high-level functional organization of a transformer is largely determined by architecture and data, even when the specific heads differ.","Because susceptibilities connect to per-sample generalization error through local learning coefficients, structures like the spacing fin have a direct link to how the model generalizes on sequences with long runs of spaces."],"supporting_citations":[{"why":"Defines per-token susceptibilities and structural inference, the method this paper builds on, and shows induction patterns have distinct susceptibility signatures across heads.","marker":"[Baker et al., 2025]"},{"why":"Supplies the 3M-parameter attention-only transformer, its training stages, and the identities of previous-token, current-token, and induction heads.","marker":"[Hoogland et al., 2025]"},{"why":"Identifies induction heads and the induction circuit, providing the known structure whose visual emergence validates the UMAP approach.","marker":"[Olsson et al., 2022]"},{"why":"Provides the UMAP dimensionality-reduction algorithm used to create the embryological visualizations.","marker":"[McInnes et al., 2020]"},{"why":"Introduces the refined local learning coefficient, connects susceptibilities to generalization error, and identifies head roles per seed.","marker":"[Wang et al., 2024]"},{"why":"Establishes the precedent of applying UMAP to gene expression profiles to study development, which motivates the embryological framing.","marker":"[Cao et al., 2019]"}],"fun_headline_variants":["Spacing fin discovered via UMAP of susceptibility vectors","UMAP exposes a spacing fin that tracks preceding spaces","Language model develops a fin for counting spacing tokens","New structure: a spacing fin in the model's body plan","Embryology of transformers: the spacing fin emerges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ordering of tokens along the spacing fin really encodes the number of preceding spacing tokens; if that gradient is an artifact of token frequency, context length, or UMAP distortion, the counting claim weakens, and the authors state they lack a complete mechanistic explanation for the fin.","fun_headline_variants_meta":{"raw":{"variants":["Spacing fin discovered via UMAP of susceptibility vectors","UMAP exposes a spacing fin that tracks preceding spaces","Language model develops a fin for counting spacing tokens","New structure: a spacing fin in the model's body plan","Embryology of transformers: the spacing fin emerges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2749,"prompt_tokens":837,"completion_tokens":1912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":1835}},"tokens_in":453,"tokens_out":1912,"duration_ms":13390,"temperature":1.0,"reasoning_tokens":1835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:12:20.557848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: sample token sequences with the same number of preceding spaces but different token frequencies and context lengths, embed them with the same susceptibility pipeline, and check whether fin position tracks space count once frequency and length are controlled. If it does not, the fin is confounded; if it does, a second test would ablate the heads with largest loadings along the fin direction and see whether predicted probabilities for space tokens stop depending on run length.","supporting_citations":[],"review_version":1}