REVIEW 4 major objections 5 minor 19 references
The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that interpretable categorical features in language models turn on sharply between 160M and 410M parameters, and that early-layer feature detectors re-emerge at the final layer.
desk verdict The three-axis framing is new and worth discussing, but the headline thresholds and reactivation are currently confounded with probe-transfer artifacts; this deserves revision, not rejection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central instrument is a top-k sparse autoencoder, an overcomplete dictionary that reconstructs residual-stream activations from a small number of active latents, configured here with k=1 active unit and 512 hidden units. The autoencoder is trained on the final checkpoint of the largest model, and each latent neuron is labeled and verified through an automated label-then-classify loop, keeping only neurons with verification F1 above 0.9. Subject queries are matched to these neuron labels by cosine similarity in an embedding space, yielding a small set of subject-aligned probes per domain. For cross-scale comparison, an orthogonal rotation aligns each smaller model's activations into the 5120-dimensional reference space of the largest model, with linear CKA and pairwise cosine correlation reported as fidelity checks.
What would settle it
Train a separate sparse autoencoder on every training checkpoint and every model size, then measure activation of the same verified concept labels; if the 410M jump and the final-layer re-emergence disappear or shift when detectors are fit per checkpoint, they are artifacts of probe transfer. A lighter test is to recompute the layer-wise curves using autoencoders trained at each layer instead of only at the final checkpoint.
Extended reading notes
Core claim
The paper's central discovery is that interpretable categorical features emerge along sharp, threshold-like boundaries on two axes and a non-monotonic curve on the third. In the 12-billion-parameter model, fewer than 3% of the concept probes fire before 1,000 optimization steps; activation then climbs in discrete jumps, including a 55.9-percentage-point rise between 30,000 and 40,000 steps, and exceeds 99% by the final 143,000-step checkpoint. Across scale, models below 160M parameters activate under 5% of concepts, while the 410M model activates about 95%, a single +92.9-point transition; activation then saturates above 98% for models of 2.8B and larger. Across depth, sparse-autoencoder probes trained on layers 0 and 2 show high activation early, near silence through middle layers, and partial re-emergence at the final layer, which the authors call semantic reactivation. Domain-level analysis adds texture: formal and numerical subjects activate early in training, while history, biology, chemistry, and business activate later, and business rises more gradually with scale.
Load-bearing premise
The concept detectors trained on the final, largest model's activations are assumed to recognize the same concepts in earlier checkpoints and smaller models after only an orthogonal rotation, so the measured 'emergence' could reflect alignment with the final model's coordinate system rather than the birth of knowledge inside the smaller model.
Editorial extensions
If this is right
- Training-time emergence is bursty: fewer than 3% of concept probes fire before 1,000 steps, and most activation arrives in discrete jumps after 10,000 steps.
- Scale-wise emergence has a narrow critical window: concept activation jumps from under 5% at 160M parameters to roughly 95% at 410M and saturates above 98% for larger models.
- Spatial organization is non-monotonic: early-layer concept features decline through middle layers and partially re-appear at the final layer, so interpretable knowledge is not simply concentrated at the top.
- Domain-specific schedules differ: formal and numerical subjects activate early in training, while history, biology, chemistry, and business activate later.
- The measured thresholds give concrete points at which categorical knowledge becomes available, which could guide when to probe, prune, or edit a model.
Reading between the lines
- Editorial inference: the sharp 160M-to-410M transition could partly be a detector-transfer artifact; training separate autoencoders per model size would show whether small models encode the same concepts in different directions.
- Editorial inference: the early-layer reactivation pattern suggests early features carry token-level detail, are suppressed during abstract computation, and are restored for output; ablating those features at different depths would test that role.
- Editorial inference: if the thresholds hold across other model families, they imply that model size can be chosen by target domain, since business-related concepts need more parameters than physics concepts to activate reliably.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies a sparse autoencoder (SAE) trained on the final Pythia-12B checkpoint, together with an AutoInterp labeling and concept-matching pipeline called EyeSee, to activations from MMLU and MMLU-PRO prompts. It tracks the activation of domain-aligned neurons across training checkpoints (time), transformer layers (space), and model sizes (scale). The central claims are that concept activation has a clear temporal threshold around 30k-40k steps, a sharp scale threshold between 160M and 410M parameters (a +92.9 percentage point jump), and a spatial pattern in which early-layer features decline in middle layers and re-emerge at the final layer.
Significance. If the measurements are valid, the paper would provide a useful joint picture of interpretable feature dynamics across time, depth, and scale within a single model family, and it uses publicly available models and benchmarks. The paper is also transparent about its descriptive nature and about the limitations of its method. However, the significance is currently conditional: the central claims rest on the assumption that a final-checkpoint SAE dictionary, and for the scale axis a Procrustes-aligned version of that dictionary, is a valid probe for earlier checkpoints, smaller models, and other layers. The absence of null baselines and the visible confound between the scale threshold and alignment quality mean the results are not yet distinguished from measurement artifacts.
major comments (4)
- [Section 4 (Figure 3)] The temporal experiment applies the SAE trained at the last checkpoint to earlier checkpoints without any control baseline. Under this design, the activation curves measure how well earlier representations align with the final dictionary, so a low activation value at early steps could reflect either the absence of the concept or a differently structured but equally valid representation. Please define the activation metric formally, report variability across datasets or SAE seeds, and add null controls such as random directions from the same final SAE or separately trained per-checkpoint SAEs.
- [Section 6.1, Eq. (4), and Appendix A.3] The scale threshold is confounded with the quality of the orthogonal Procrustes alignment used to rotate smaller models into the 12B reference basis. Appendix A.3 reports that the pairwise cosine matrix correlation is only 0.34 at 14M parameters and rises sharply from the 410M checkpoint onward, which is exactly the range where Figure 5 shows the +92.9pp activation jump. Thus the reported threshold may describe when small models become rotatable into the 12B basis rather than when domain features emerge. Please provide per-model dictionaries or otherwise control for alignment quality independently of feature content.
- [Section 5 (Figure 4)] The spatial 'semantic reactivation' claim is not distinguished from a linear-similarity artifact. The left panel of Figure 4 shows that the final residual stream is geometrically close to the embedding/input block, so the layer-0 SAE firing at layer 36 is expected for any linear probe that is sensitive to that block structure. Please add per-layer SAEs or a random-direction baseline, and quantify the re-emergence relative to the cosine-similarity block structure rather than only reporting the raw activation curve.
- [Sections 2.2 and 3] The central quantity 'percentage of concepts activated' is never formally defined. It is not stated whether a concept neuron counts as active when it fires on at least one sample, on a majority of samples, or above some activation magnitude threshold, nor is it clear how the percentage is aggregated across the concept set. Without this definition Figures 3, 5, and 6 cannot be quantitatively interpreted. In addition, no error bars or repeated-run variability are reported for any of the main activation curves.
minor comments (5)
- [Section 5, final paragraph] The sentence ending in 'the processing core is distilling and' is incomplete and should be finished or removed.
- [Section 6.2, History bullet] The History bullet reads 'shifts directly from 0' and appears to be truncated; the sentence should be completed.
- [Sections 2.2.1 and 2.3] The SAE with h=512 and d=5120 is undercomplete, not overcomplete; the text calls it 'overcomplete' in Section 2.2.1 and refers to an 'overcomplete basis' in Section 2.3, which is inaccurate.
- [References [9] and [15]] The main text cites reference [15] as the AUTOINTERP framework, but [15] is 'Towards monosemanticity' while [9] is the AutoInterp paper; the citations should be aligned with the intended sources.
- [Figure 4 and Section 5] Layer indexing is inconsistent between the caption ('Layers 0 and 2'), the text ('Layers 1-3' and 'Layer 36'), and the SAE probes; please standardize the indexing across the paper.
Circularity Check
Scale-axis emergence threshold is confounded with the fitted Procrustes alignment; spatial 'reactivation' restates known layer-block similarity.
-
fitted input called prediction
[Section 6.1, Eq. (4); Section 6.2; Appendix A.3, Fig. 7]
"To embed all checkpoints in a common feature space we apply an orthogonal Procrustes transformation [16, 17]. For each model m ... we solve W ⋆ m = arg min ... ||XmW−X12B||F , (4) ... A single transition—between the 160 M and 410 M checkpoints—raises the activation rate by +92.9 pp to ∼95% ... pairwise cosine matrix correlation ... rises sharply from the 410-M checkpoint onward."
The scale-axis activation metric is evaluated on activations XmW*_m, where W*_m is the orthogonal Procrustes rotation fitted to minimize distance to the 12B reference matrix X12B. Since the concept dictionary and labels are themselves built from that same 12B model, the percentage of activated 'concepts' in a smaller model measures how well the optimal rotation can place that model's activations into the 12B coordinate frame, not an independent property of the smaller model. Appendix A.3 shows the fitted alignment's pairwise cosine correlation jumps sharply at 410M, exactly the point of the +92.9pp activation jump, so the reported scale threshold is confounded with—and may reduce to—the success of the Procrustes fit.
-
renaming known result
[Section 5, Figure 4]
"Notably, early-layer SAEs (Layers 0 and 2) exhibit high initial activations that sharply decline across intermediate layers before partially re-emerging at later layers, particularly at the final output stage. ... The cosine similarity heatmap identifies three distinct representational blocks across the model’s depth: the input-like block (Layers 1–3), closely aligned with the token-embedding space; ... and the output block (Layers 36), realigning representations toward the prediction task."
The 're-emergence' is measured with SAE probes that respond to linear (cosine) similarity in the residual stream. The same section's left panel already shows that layer 36 is geometrically close to the input-like block (Layers 1–3), so the U-shaped activation of layer-0 SAE features is a restatement, in SAE coordinates, of the previously established block structure rather than an independent discovery of 'semantic reactivation.' The paper even concedes it 'can happen for a number of reasons such as early and late layers capturing token specific details,' undercutting the claim that the re-emergence is a strategic semantic re-insertion.
full rationale
The temporal analysis is transparently a probe-transfer study: the SAE is trained on the final checkpoint and then applied to earlier checkpoints, so low early activation could reflect dictionary mismatch rather than absence of features; this is a validity limitation but not itself a circular reduction. The scale analysis, however, is more problematic: Eq. (4) fits an orthogonal Procrustes rotation for each smaller model to minimize distance to the 12B reference matrix, and activation of 12B-dictionary concepts is measured in that rotated space. Because the rotation is optimized against the very reference used to define the concepts, the activation percentage is not an independent measure of feature emergence; Appendix A.3 shows the fitted alignment's pairwise cosine correlation jumps sharply at 410M, the same point as the reported +92.9pp activation jump, so the headline threshold may describe when small models become rotatable into the 12B basis rather than when domain features emerge. The spatial 'semantic reactivation' similarly restates the cosine-similarity block structure (input-like vs processing core vs output) in SAE coordinates; the paper's own left panel already shows layer 36 is close to layers 1-3. No self-citation chain is present. Score reflects partial circularity: the central scale claim is confounded with a fitted transformation, while the temporal and per-domain results retain some independent empirical content.
Assumptions & free parameters
free parameters (6)
- SAE latent width h =
512
- SAE activation budget k =
1
- High-fidelity F1 threshold tau_F1 =
0.9
- Cosine similarity threshold tau =
0.3
- Procrustes alignment matrices W_m =
9 orthogonal matrices (one per smaller model)
- AutoInterp sample budgets n_label, n_verify =
10 and 5
assumptions (5)
- domain assumption Top-k sparse autoencoders learn interpretable features in residual streams
- domain assumption AutoInterp verification F1 measures label correctness
- domain assumption all-mpnet-base-v2 cosine similarity captures subject-concept alignment
- domain assumption Procrustes rotation preserves feature semantics across scales
- ad hoc to paper Final-checkpoint SAE directions are valid probes for earlier checkpoints
Cite this review
Pith. "Pith review of The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models." pith.science (2026). https://pith.science/paper/33TSJI4E
@misc{pith2026250519440,
author = {Pith},
title = {Pith review of: The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/33TSJI4E}},
note = {Machine review of arXiv:2505.19440}
}
read the original abstract
This paper studies the emergence of interpretable categorical features within large language models (LLMs), analyzing their behavior across training checkpoints (time), transformer layers (space), and varying model sizes (scale). Using sparse autoencoders for mechanistic interpretability, we identify when and where specific semantic concepts emerge within neural activations. Results indicate clear temporal and scale-specific thresholds for feature emergence across multiple domains. Notably, spatial analysis reveals unexpected semantic reactivation, with early-layer features re-emerging at later layers, challenging standard assumptions about representational dynamics in transformer models.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[2]
Zoom in: An introduction to circuits.Distill, 2020
Chris Olah, Nick Cammarata, Shan Carter, Gabriel V oss, Ludwig Schubert, Adam Rabinowitz, and Jared Kaplan. Zoom in: An introduction to circuits.Distill, 2020. doi: 10.23915/distill. 00024
doi:10.23915/distill 2020
-
[3]
Emergent abilities of large language models
Jason Wei, Yi Tay, Rishi Bommasani, and et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022
arXiv 2022
-
[4]
Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020
Jared Kaplan, Sam McCandlish, Tom Henighan, and et al. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020
arXiv 2001
-
[5]
Brown, Benjamin Mann, Nick Ryder, and et al
Tom B. Brown, Benjamin Mann, Nick Ryder, and et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020
work page 2020
-
[6]
Mechanistic permutability: Match features across layers.arXiv preprint arXiv:2410.07656, 2025
Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. Mechanistic permutability: Match features across layers.arXiv preprint arXiv:2410.07656, 2025. doi: 10.48550/arXiv.2410.07656
-
[7]
A layer-wise analysis of transformer representations
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. A layer-wise analysis of transformer representations. InProceedings of ACL 2019, 2019
work page 2019
-
[8]
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?arXiv preprint arXiv:2404.07066, 2024. doi: 10.48550/arXiv.2404.07066
Show all 19 references
-
[9]
Autointerp: Automated mechanistic interpretability for language models.arXiv preprint arXiv:2407.06543, 2024
Glenn Conmy, Qiyuan Sun, Liang Feng, and et al. Autointerp: Automated mechanistic interpretability for language models.arXiv preprint arXiv:2407.06543, 2024
2024 arXiv
- [10]
-
[11]
Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2021
Dan Hendrycks, Collin Burns, Saurav Kadavath, and et al. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2021
2009 arXiv
-
[12]
Mmlu-pro: A more challenging benchmark for language model evaluation.arXiv preprint arXiv:2310.15420, 2023
Jian Sun, Siyu Wang, Xiaoyang Zhu, and et al. Mmlu-pro: A more challenging benchmark for language model evaluation.arXiv preprint arXiv:2310.15420, 2023
2023 arXiv
-
[13]
Pythia: A suite for analyzing large language models across training and scaling.arXiv preprint arXiv:2304.01373, 2023
Stella Biderman, Sidney Black, Jonas Mueller, Jason Phang, Yada Pruksachatkun, and et al. Pythia: A suite for analyzing large language models across training and scaling.arXiv preprint arXiv:2304.01373, 2023
2023 arXiv
-
[14]
Scaling and evaluating sparse autoencoders.arXiv preprint arXiv:2406.04093, 2024
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders.arXiv preprint arXiv:2406.04093, 2024
2024 arXiv
-
[15]
Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2, 2023
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2, 2023
2023
-
[16]
Schönemann
Peter H. Schönemann. A generalized solution of the orthogonal procrustes problem.Psychome- trika, 31:1–10, 1966
1966
-
[17]
Turk-Browne
Sam Smith and Nicholas B. Turk-Browne. Linking the deep neural network model and brain using alignment methods.Nature Neuroscience, 2019
2019
-
[18]
In-context Learning and Induction Heads.arXiv preprint arXiv:2209.11895, 2022
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context Learning and Induction Heads.arXiv preprint arXiv:2209.11895, 2022. doi: 10.48550/arXiv.2209.11895
-
[19]
Comparative Analysis Questions
Samuel Templeton, Oliver Tworkowski, Neel Nanda, and et al. Scaling monosemanticity: Extracting interpretable features from claude.arXiv preprint arXiv:2402.01232, 2024. 10 A Appendix A.1 Complete ranked list of different EyeSee concepts Table 2: Complete catalogue of subject-...
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.