Pith. sign in

REVIEW 4 major objections 5 minor 19 references

The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper claims that interpretable categorical features in language models turn on sharply between 160M and 410M parameters, and that early-layer feature detectors re-emerge at the final layer.

desk verdict The three-axis framing is new and worth discussing, but the headline thresholds and reactivation are currently confounded with probe-transfer artifacts; this deserves revision, not rejection. read the letter →

arxiv 2505.19440 v1 pith:33TSJI4E submitted 2025-05-26 cs.CL cs.LG

classification cs.CLcs.LG
keywords mechanisticinterpretabilitysparseautoencodersfeatureemergencetrainingcheckpointsmodelscalingconceptactivationtransformerlayerslargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks when and where a language model actually knows a subject such as physics or history, rather than merely performing well on benchmarks. Using sparse autoencoders to isolate interpretable concept neurons in the residual stream, the authors track these neurons across training checkpoints, transformer layers, and model sizes in a single model family. They find that concept activation stays near zero for the first thousand training steps, then rises in bursts, and that across model sizes a single transition between 160M and 410M parameters lifts activation from under 5% to about 95%. They also report that early-layer features fade through the middle of the network and re-emerge at the final layer, contradicting the simple picture of knowledge accumulating monotonically with depth. If these patterns hold, they give concrete predictions for when during training and at what scale categorical knowledge becomes available inside a model.

What carries the argument

The central instrument is a top-k sparse autoencoder, an overcomplete dictionary that reconstructs residual-stream activations from a small number of active latents, configured here with k=1 active unit and 512 hidden units. The autoencoder is trained on the final checkpoint of the largest model, and each latent neuron is labeled and verified through an automated label-then-classify loop, keeping only neurons with verification F1 above 0.9. Subject queries are matched to these neuron labels by cosine similarity in an embedding space, yielding a small set of subject-aligned probes per domain. For cross-scale comparison, an orthogonal rotation aligns each smaller model's activations into the 5120-dimensional reference space of the largest model, with linear CKA and pairwise cosine correlation reported as fidelity checks.

What would settle it

Train a separate sparse autoencoder on every training checkpoint and every model size, then measure activation of the same verified concept labels; if the 410M jump and the final-layer re-emergence disappear or shift when detectors are fit per checkpoint, they are artifacts of probe transfer. A lighter test is to recompute the layer-wise curves using autoencoders trained at each layer instead of only at the final checkpoint.

Watch

Extended reading notes

Core claim

The paper's central discovery is that interpretable categorical features emerge along sharp, threshold-like boundaries on two axes and a non-monotonic curve on the third. In the 12-billion-parameter model, fewer than 3% of the concept probes fire before 1,000 optimization steps; activation then climbs in discrete jumps, including a 55.9-percentage-point rise between 30,000 and 40,000 steps, and exceeds 99% by the final 143,000-step checkpoint. Across scale, models below 160M parameters activate under 5% of concepts, while the 410M model activates about 95%, a single +92.9-point transition; activation then saturates above 98% for models of 2.8B and larger. Across depth, sparse-autoencoder probes trained on layers 0 and 2 show high activation early, near silence through middle layers, and partial re-emergence at the final layer, which the authors call semantic reactivation. Domain-level analysis adds texture: formal and numerical subjects activate early in training, while history, biology, chemistry, and business activate later, and business rises more gradually with scale.

Load-bearing premise

The concept detectors trained on the final, largest model's activations are assumed to recognize the same concepts in earlier checkpoints and smaller models after only an orthogonal rotation, so the measured 'emergence' could reflect alignment with the final model's coordinate system rather than the birth of knowledge inside the smaller model.

Editorial extensions

If this is right

  • Training-time emergence is bursty: fewer than 3% of concept probes fire before 1,000 steps, and most activation arrives in discrete jumps after 10,000 steps.
  • Scale-wise emergence has a narrow critical window: concept activation jumps from under 5% at 160M parameters to roughly 95% at 410M and saturates above 98% for larger models.
  • Spatial organization is non-monotonic: early-layer concept features decline through middle layers and partially re-appear at the final layer, so interpretable knowledge is not simply concentrated at the top.
  • Domain-specific schedules differ: formal and numerical subjects activate early in training, while history, biology, chemistry, and business activate later.
  • The measured thresholds give concrete points at which categorical knowledge becomes available, which could guide when to probe, prune, or edit a model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the sharp 160M-to-410M transition could partly be a detector-transfer artifact; training separate autoencoders per model size would show whether small models encode the same concepts in different directions.
  • Editorial inference: the early-layer reactivation pattern suggests early features carry token-level detail, are suppressed during abstract computation, and are restored for output; ablating those features at different depths would test that role.
  • Editorial inference: if the thresholds hold across other model families, they imply that model size can be chosen by target domain, since business-related concepts need more parameters than physics concepts to activate reliably.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper applies a sparse autoencoder (SAE) trained on the final Pythia-12B checkpoint, together with an AutoInterp labeling and concept-matching pipeline called EyeSee, to activations from MMLU and MMLU-PRO prompts. It tracks the activation of domain-aligned neurons across training checkpoints (time), transformer layers (space), and model sizes (scale). The central claims are that concept activation has a clear temporal threshold around 30k-40k steps, a sharp scale threshold between 160M and 410M parameters (a +92.9 percentage point jump), and a spatial pattern in which early-layer features decline in middle layers and re-emerge at the final layer.

Significance. If the measurements are valid, the paper would provide a useful joint picture of interpretable feature dynamics across time, depth, and scale within a single model family, and it uses publicly available models and benchmarks. The paper is also transparent about its descriptive nature and about the limitations of its method. However, the significance is currently conditional: the central claims rest on the assumption that a final-checkpoint SAE dictionary, and for the scale axis a Procrustes-aligned version of that dictionary, is a valid probe for earlier checkpoints, smaller models, and other layers. The absence of null baselines and the visible confound between the scale threshold and alignment quality mean the results are not yet distinguished from measurement artifacts.

major comments (4)
  1. [Section 4 (Figure 3)] The temporal experiment applies the SAE trained at the last checkpoint to earlier checkpoints without any control baseline. Under this design, the activation curves measure how well earlier representations align with the final dictionary, so a low activation value at early steps could reflect either the absence of the concept or a differently structured but equally valid representation. Please define the activation metric formally, report variability across datasets or SAE seeds, and add null controls such as random directions from the same final SAE or separately trained per-checkpoint SAEs.
  2. [Section 6.1, Eq. (4), and Appendix A.3] The scale threshold is confounded with the quality of the orthogonal Procrustes alignment used to rotate smaller models into the 12B reference basis. Appendix A.3 reports that the pairwise cosine matrix correlation is only 0.34 at 14M parameters and rises sharply from the 410M checkpoint onward, which is exactly the range where Figure 5 shows the +92.9pp activation jump. Thus the reported threshold may describe when small models become rotatable into the 12B basis rather than when domain features emerge. Please provide per-model dictionaries or otherwise control for alignment quality independently of feature content.
  3. [Section 5 (Figure 4)] The spatial 'semantic reactivation' claim is not distinguished from a linear-similarity artifact. The left panel of Figure 4 shows that the final residual stream is geometrically close to the embedding/input block, so the layer-0 SAE firing at layer 36 is expected for any linear probe that is sensitive to that block structure. Please add per-layer SAEs or a random-direction baseline, and quantify the re-emergence relative to the cosine-similarity block structure rather than only reporting the raw activation curve.
  4. [Sections 2.2 and 3] The central quantity 'percentage of concepts activated' is never formally defined. It is not stated whether a concept neuron counts as active when it fires on at least one sample, on a majority of samples, or above some activation magnitude threshold, nor is it clear how the percentage is aggregated across the concept set. Without this definition Figures 3, 5, and 6 cannot be quantitatively interpreted. In addition, no error bars or repeated-run variability are reported for any of the main activation curves.
minor comments (5)
  1. [Section 5, final paragraph] The sentence ending in 'the processing core is distilling and' is incomplete and should be finished or removed.
  2. [Section 6.2, History bullet] The History bullet reads 'shifts directly from 0' and appears to be truncated; the sentence should be completed.
  3. [Sections 2.2.1 and 2.3] The SAE with h=512 and d=5120 is undercomplete, not overcomplete; the text calls it 'overcomplete' in Section 2.2.1 and refers to an 'overcomplete basis' in Section 2.3, which is inaccurate.
  4. [References [9] and [15]] The main text cites reference [15] as the AUTOINTERP framework, but [15] is 'Towards monosemanticity' while [9] is the AutoInterp paper; the citations should be aligned with the intended sources.
  5. [Figure 4 and Section 5] Layer indexing is inconsistent between the caption ('Layers 0 and 2'), the text ('Layers 1-3' and 'Layer 36'), and the SAE probes; please standardize the indexing across the paper.

Circularity Check

2 steps flagged · score 6.0 of 10

Scale-axis emergence threshold is confounded with the fitted Procrustes alignment; spatial 'reactivation' restates known layer-block similarity.

  1. fitted input called prediction [Section 6.1, Eq. (4); Section 6.2; Appendix A.3, Fig. 7]
    "To embed all checkpoints in a common feature space we apply an orthogonal Procrustes transformation [16, 17]. For each model m ... we solve W ⋆ m = arg min ... ||XmW−X12B||F , (4) ... A single transition—between the 160 M and 410 M checkpoints—raises the activation rate by +92.9 pp to ∼95% ... pairwise cosine matrix correlation ... rises sharply from the 410-M checkpoint onward."

    The scale-axis activation metric is evaluated on activations XmW*_m, where W*_m is the orthogonal Procrustes rotation fitted to minimize distance to the 12B reference matrix X12B. Since the concept dictionary and labels are themselves built from that same 12B model, the percentage of activated 'concepts' in a smaller model measures how well the optimal rotation can place that model's activations into the 12B coordinate frame, not an independent property of the smaller model. Appendix A.3 shows the fitted alignment's pairwise cosine correlation jumps sharply at 410M, exactly the point of the +92.9pp activation jump, so the reported scale threshold is confounded with—and may reduce to—the success of the Procrustes fit.

  2. renaming known result [Section 5, Figure 4]
    "Notably, early-layer SAEs (Layers 0 and 2) exhibit high initial activations that sharply decline across intermediate layers before partially re-emerging at later layers, particularly at the final output stage. ... The cosine similarity heatmap identifies three distinct representational blocks across the model’s depth: the input-like block (Layers 1–3), closely aligned with the token-embedding space; ... and the output block (Layers 36), realigning representations toward the prediction task."

    The 're-emergence' is measured with SAE probes that respond to linear (cosine) similarity in the residual stream. The same section's left panel already shows that layer 36 is geometrically close to the input-like block (Layers 1–3), so the U-shaped activation of layer-0 SAE features is a restatement, in SAE coordinates, of the previously established block structure rather than an independent discovery of 'semantic reactivation.' The paper even concedes it 'can happen for a number of reasons such as early and late layers capturing token specific details,' undercutting the claim that the re-emergence is a strategic semantic re-insertion.

full rationale

The temporal analysis is transparently a probe-transfer study: the SAE is trained on the final checkpoint and then applied to earlier checkpoints, so low early activation could reflect dictionary mismatch rather than absence of features; this is a validity limitation but not itself a circular reduction. The scale analysis, however, is more problematic: Eq. (4) fits an orthogonal Procrustes rotation for each smaller model to minimize distance to the 12B reference matrix, and activation of 12B-dictionary concepts is measured in that rotated space. Because the rotation is optimized against the very reference used to define the concepts, the activation percentage is not an independent measure of feature emergence; Appendix A.3 shows the fitted alignment's pairwise cosine correlation jumps sharply at 410M, the same point as the reported +92.9pp activation jump, so the headline threshold may describe when small models become rotatable into the 12B basis rather than when domain features emerge. The spatial 'semantic reactivation' similarly restates the cosine-similarity block structure (input-like vs processing core vs output) in SAE coordinates; the paper's own left panel already shows layer 36 is close to layers 1-3. No self-citation chain is present. Score reflects partial circularity: the central scale claim is confounded with a fitted transformation, while the temporal and per-domain results retain some independent empirical content.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are postulated; 'EyeSee' is a named analysis pipeline rather than a new scientific entity. The free parameters are the SAE hyperparameters, the matching thresholds, and the fitted Procrustes matrices. The axioms are the standard interpretability assumptions plus the paper-specific probe-transfer assumption that the final checkpoint's dictionary generalizes to earlier checkpoints and smaller models.

free parameters (6)
  • SAE latent width h = 512
    Chosen to maximize mean AutoInterp F1 on the final checkpoint (Figure 2, right).
  • SAE activation budget k = 1
    Selected as the best of {1,2,4,8} by mean F1 on the final checkpoint (Figure 2, left).
  • High-fidelity F1 threshold tau_F1 = 0.9
    Neurons with AutoInterp verification F1 above this value form the trusted pool Nhi (Section 3).
  • Cosine similarity threshold tau = 0.3
    Neurons with MPNet embedding cosine similarity to the subject query above this value are treated as subject-aligned concepts (Eq. 3).
  • Procrustes alignment matrices W_m = 9 orthogonal matrices (one per smaller model)
    Fitted via Eq. (4) to rotate each smaller model's activations into the 12B reference space; alignment fidelity varies from 0.90 to 0.99 linear CKA and from 0.34 to >0.95 pairwise cosine correlation (Appendix A.3).
  • AutoInterp sample budgets n_label, n_verify = 10 and 5
    Chosen in the modified AutoInterp pipeline (Section 2.3); small budgets affect label quality and verification stability.
assumptions (5)
  • domain assumption Top-k sparse autoencoders learn interpretable features in residual streams
    The entire pipeline assumes SAE latents correspond to meaningful concepts; reconstruction loss alone (Eq. 2) does not guarantee semantic interpretability.
  • domain assumption AutoInterp verification F1 measures label correctness
    Teacher-LLM labels and F1 > 0.9 are taken as ground truth for feature semantics, relying on the AutoInterp framework.
  • domain assumption all-mpnet-base-v2 cosine similarity captures subject-concept alignment
    Subject queries are matched to neuron labels by nearest-neighbor embedding similarity (Eq. 3); no validation of this semantic metric is provided.
  • domain assumption Procrustes rotation preserves feature semantics across scales
    Eq. (4) assumes smaller models' representations are a linear rotation of the 12B reference, which is not proven and is contradicted at small scales by the low pairwise cosine correlations reported in Appendix A.3.
  • ad hoc to paper Final-checkpoint SAE directions are valid probes for earlier checkpoints
    Section 4 applies the final SAE to earlier embeddings with no alignment or baseline; this probe-transfer assumption is the paper's weakest premise and is not supported by any evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models." pith.science (2026). https://pith.science/paper/33TSJI4E

@misc{pith2026250519440,
  author       = {Pith},
  title        = {Pith review of: The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/33TSJI4E}},
  note         = {Machine review of arXiv:2505.19440}
}
read the original abstract

This paper studies the emergence of interpretable categorical features within large language models (LLMs), analyzing their behavior across training checkpoints (time), transformer layers (space), and varying model sizes (scale). Using sparse autoencoders for mechanistic interpretability, we identify when and where specific semantic concepts emerge within neural activations. Results indicate clear temporal and scale-specific thresholds for feature emergence across multiple domains. Notably, spatial analysis reveals unexpected semantic reactivation, with early-layer features re-emerging at later layers, challenging standard assumptions about representational dynamics in transformer models.

Figures

Figures reproduced from arXiv: 2505.19440 by the authors.

Figure 1
Figure 1. Axes of our emergent-knowledge probe. We track how interpretable, categorical features [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Hyperparameter sweep for sparse–autoencoder interpretability. Left: mean F1-score as the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Activation patterns of categorical concepts in a 12B-parameter language model across [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cosine similarity (left) reveals three macro blocks (embedding, processing core, output), [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Concept-activation saturation with model scale. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Percentage of high-fidelity neurons (F1 ≥ 0.9) that fire for each subject in every transformer block of the 12-B Pythia model. Values are averaged over the combined MMLU + MMLU-PRO prompt set. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Alignment quality across model scale. Left: linear CKA between projected activations [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 7 canonical work pages

  1. [1]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

  2. [2]

    Zoom in: An introduction to circuits.Distill, 2020

    Chris Olah, Nick Cammarata, Shan Carter, Gabriel V oss, Ludwig Schubert, Adam Rabinowitz, and Jared Kaplan. Zoom in: An introduction to circuits.Distill, 2020. doi: 10.23915/distill. 00024

  3. [3]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, and et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

  4. [4]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, and et al. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

  5. [5]

    Brown, Benjamin Mann, Nick Ryder, and et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, and et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020

  6. [6]

    Mechanistic permutability: Match features across layers.arXiv preprint arXiv:2410.07656, 2025

    Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. Mechanistic permutability: Match features across layers.arXiv preprint arXiv:2410.07656, 2025. doi: 10.48550/arXiv.2410.07656

  7. [7]

    A layer-wise analysis of transformer representations

    Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. A layer-wise analysis of transformer representations. InProceedings of ACL 2019, 2019

  8. [8]

    Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?arXiv preprint arXiv:2404.07066, 2024

    Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?arXiv preprint arXiv:2404.07066, 2024. doi: 10.48550/arXiv.2404.07066

Show all 19 references
  1. [9]

    Autointerp: Automated mechanistic interpretability for language models.arXiv preprint arXiv:2407.06543, 2024

    Glenn Conmy, Qiyuan Sun, Liang Feng, and et al. Autointerp: Automated mechanistic interpretability for language models.arXiv preprint arXiv:2407.06543, 2024

  2. [10]

    Tracking the feature dynamics in llm training: A mechanistic study.arXiv preprint arXiv:2412.17626, 2024

    Yang Xu, Yi Wang, and Hao Wang. Tracking the feature dynamics in llm training: A mechanistic study.arXiv preprint arXiv:2412.17626, 2024. doi: 10.48550/arXiv.2412.17626

  3. [11]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, and et al. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2021

  4. [12]

    Mmlu-pro: A more challenging benchmark for language model evaluation.arXiv preprint arXiv:2310.15420, 2023

    Jian Sun, Siyu Wang, Xiaoyang Zhu, and et al. Mmlu-pro: A more challenging benchmark for language model evaluation.arXiv preprint arXiv:2310.15420, 2023

  5. [13]

    Pythia: A suite for analyzing large language models across training and scaling.arXiv preprint arXiv:2304.01373, 2023

    Stella Biderman, Sidney Black, Jonas Mueller, Jason Phang, Yada Pruksachatkun, and et al. Pythia: A suite for analyzing large language models across training and scaling.arXiv preprint arXiv:2304.01373, 2023

  6. [14]

    Scaling and evaluating sparse autoencoders.arXiv preprint arXiv:2406.04093, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders.arXiv preprint arXiv:2406.04093, 2024

  7. [15]

    Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2, 2023

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2, 2023

  8. [16]

    Schönemann

    Peter H. Schönemann. A generalized solution of the orthogonal procrustes problem.Psychome- trika, 31:1–10, 1966

  9. [17]

    Turk-Browne

    Sam Smith and Nicholas B. Turk-Browne. Linking the deep neural network model and brain using alignment methods.Nature Neuroscience, 2019

  10. [18]

    In-context Learning and Induction Heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context Learning and Induction Heads.arXiv preprint arXiv:2209.11895, 2022. doi: 10.48550/arXiv.2209.11895

  11. [19]

    Comparative Analysis Questions

    Samuel Templeton, Oliver Tworkowski, Neel Nanda, and et al. Scaling monosemanticity: Extracting interpretable features from claude.arXiv preprint arXiv:2402.01232, 2024. 10 A Appendix A.1 Complete ranked list of different EyeSee concepts Table 2: Complete catalogue of subject-...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.