Pith. sign in

REVIEW 4 major objections 6 minor 58 references

TRACE for Tracking the Emergence of Semantic Representations in Transformers

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TRACE claims that transformer models cross a measurable phase transition, marked by a Hessian-curvature/intrinsic-dimensionality intersection, where they shift from memorising to abstracting linguistic structure.

desk verdict The paper's assets—ABSynth, the C(H) score, honest MI reporting—are real, but its central phase-transition claim rests on the crossing of two curves with arbitrary scales, and the probing evidence is confounded by token names that encode the labels. read the letter →

arxiv 2505.17998 v1 pith:S6TQ2AMI submitted 2025-05-23 cs.CL

Nura Aljaafari , Danilo S. Carvalho , André Freitas This is my paper
classification cs.CL
keywords TRACEphasetransitionsintrinsicdimensionalityHessiancurvaturerepresentationabstractionframesemanticssyntheticcorpusprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TRACE is a diagnostic framework for detecting when a transformer language model stops memorising its synthetic training corpus and starts encoding abstract linguistic structure. The paper's central claim is that this shift is not gradual but appears as a phase transition marked by a coordinated geometric event: the Hessian curvature score C(H) collapses while the intrinsic dimensionality ID(t) stabilises, and the two curves intersect at the transition step. At the same steps, linear probes on hidden states begin to predict part-of-speech and semantic-role categories, and the pattern holds across three model sizes and two architectural ablations. The authors introduce ABSynth, a frame-semantic synthetic corpus with complete annotations, to make this measurement possible. The payoff of the claim is that abstraction onset becomes observable during training rather than inferable only from final performance.

What carries the argument

The argument is carried by three coordinated instruments. The first is the curvature complexity score C(H) = Tr(H)/sqrt(reff), computed from the top eigenvalues of the Hessian, where reff is the entropy-based effective rank of the spectrum; this score combines overall curvature magnitude with its concentration. The second is the TWO-NN intrinsic dimensionality estimator, which infers local manifold dimension from the ratio of first-to-second nearest-neighbour distances and is averaged across layers. The third is ABSynth, a frame-semantic corpus generator whose sentences carry ground-truth syntactic and semantic role labels, so probes can be trained at every training step. The paper reads the crossing of the first two curves as the phase-transition marker and validates it by the rise in probe confidence.

What would settle it

Take the same checkpoint sequence and divide either C(H) or the layer-averaged ID by a constant (or normalise both curves to a common vertical range), then re-locate the intersection; if the reported transition step changes continuously with the rescaling, the crossing is a plotting artefact. A second decisive check is to run the same three models on a natural-language corpus with approximate annotations; the claimed universal signature would predict a coordinated intersection and probe surge there too.

Watch

Extended reading notes

Core claim

The paper's central discovery is that transformer training on structured linguistic input exhibits a consistent two-phase dynamic: an early phase of rising intrinsic dimensionality and elevated, concentrated curvature, followed by a transition in which curvature flattens, intrinsic dimensionality stabilises, and probe confidence for syntactic and semantic categories rises. The authors identify the transition as the intersection between C(H) and the layer-averaged ID, and report that this intersection occurs sharply around step 5000 for the large model, later and noisier for the small model, and that it persists when feedforward networks are removed and when attention heads are reduced. They interpret this as evidence that abstraction emerges through a phase transition rather than gradual accumulation, and that the geometric signature is a fundamental property of transformer optimisation on sequential data rather than an artefact of a specific architectural component.

Load-bearing premise

The load-bearing premise is that the intersection of two curves with different units—a curvature score and a dimension count—marks a real event; if either curve is rescaled arbitrarily, the crossing moves along the training axis, and the paper gives no invariance argument that fixes the scales.

Editorial extensions

If this is right

  • Curvature and intrinsic dimensionality could be used as cheap, training-time signals for when a model begins to form abstractions, independent of downstream evaluation.
  • Removing feedforward networks does not eliminate the phase transition; it mainly adds volatility, so the transition does not depend on FFN capacity.
  • Reducing attention heads delays and dampens the transition in the small model but not in larger ones, so attention expressivity matters most at smaller scale.
  • Mutual information estimates are too variable in this setting to serve as a reliable abstraction indicator, so TRACE excludes them from its core signature.
  • The same geometric markers should transfer to pre-trained models and fine-tuning regimes, where curvature and dimensionality can be measured without annotated corpora.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The quantitative transition times (for instance, step 5000 for the large model) should be treated as reports of a particular plotting scale; an invariance-respecting version of the criterion would need to normalise C(H) and ID on a common scale before claiming exact step numbers.
  • Because ABSynth is fully labelled and grammatically clean, the framework tests learning dynamics in an idealised language; whether the same coordinated intersection appears on noisy natural text is a testable open question the paper does not settle.
  • If the crossover is a genuine optimisation-geometry event, it suggests interventions such as changing the learning-rate schedule or pruning at the transition step could control abstraction onset, an engineering consequence the paper does not pursue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript presents TRACE, a diagnostic framework that tracks spectral Hessian curvature, intrinsic dimensionality, and linguistic probe accuracy during training of small decoder-only transformers. The data come from ABSynth25K, a synthetic, fully annotated frame-semantic corpus. The authors report a consistent phase transition in which curvature collapses and intrinsic dimensionality stabilizes, marked by an intersection of the C(H) and ID curves, and claim this coincides with the emergence of syntactic and semantic category structure, with the pattern robust across model scales and architectural ablations. The main contributions are the C(H) curvature score, the ABSynth generator, and the multi-signal monitoring protocol.

Significance. TRACE's multi-signal design and the ABSynth corpus are potentially useful resources, and the paper is transparent about the failure of mutual information as a diagnostic. However, the headline finding is not established: the phase-transition time is defined as the crossing of two quantities with different units and arbitrary axis scalings, so the reported step-5000 transition for the large model is not an invariant property of training dynamics. The probe evidence is also confounded because ABSynth labels are deterministically derived from token names, and the probe architecture description is internally inconsistent. As a result, the paper currently provides a collection of observations rather than a validated method for detecting abstraction onset.

major comments (4)
  1. [§4.1, Eqs. (1)-(2), Fig. 2] The central phase transition is defined by the intersection of C(H) = Tr(H)/√r_eff and the mean TWO-NN intrinsic dimensionality. These quantities are not commensurable: C(H) carries the scale of the Hessian trace, while ID is a count of local degrees of freedom. In Fig. 2 they are plotted on separate vertical axes with ranges chosen for visual clarity. A positive rescaling of either curve moves the intersection time monotonically, and because C(H) decays while ID rises, an intersection is essentially guaranteed. The paper gives no normalization rule or invariance argument, and no computational criterion for locating the intersection. Consequently, the reported transition times (e.g., 'around step 5000' for the large model) are artifacts of the twin-axis scaling rather than independently measured events. The authors should define the transition by a scale-invariant criterion applied to each metric separately and then test for temporal coordination, or demonstrate that the crossing is invariant under renormalization.
  2. [§3.3, Appendix B.6] The probing results are confounded by the data-generation scheme. Appendix B.6 states that 'both POS and semantic labels are deterministically derived from token names,' and token names such as 'noun3' explicitly encode the POS category. A probe trained on hidden states can therefore achieve high accuracy by relying on surface token identity, without representing any abstraction over roles or categories. The random-initialization control in Appendix B.9 only shows that untrained embeddings do not support this readout; it does not distinguish memorization of token names from abstraction over category structure. The interpretation in §3.3 that probes reveal 'alignment with underlying semantic functions rather than memorised token identities' is therefore unsupported. The authors should test whether probe accuracy transfers to held-out token names that realize the same categories (e.g., unseen noun tokens) or otherwise control for token identity.
  3. [§3.3, Appendix B.6] There is an internal contradiction about the probe model. Section 3.3 defines p^{(ℓ)}_c as a linear classifier and motivates the analysis in terms of linear separability, but Appendix B.6 describes a three-layer feedforward network with ReLU activations and dropout. A nonlinear MLP probe can fit features that are not linearly accessible, so the evidence for 'linear' representational abstraction is weaker than claimed. The authors should either use true linear probes for the main claims or explicitly describe and justify the nonlinear probe.
  4. [Appendix B.8, Tables 4-6] The text introducing Tables 4-6 says they report average POS probe performance 'for the small, medium, and large models, respectively,' but all three table captions identify the Large Model (Layer 0, Layer 1, Layer 2). As printed, the cross-scale probe evidence is not present in these tables; the captions or the tables need to be corrected, and the claimed scale-dependence of probe accuracy must be verifiable.
minor comments (6)
  1. [§3.2] The display equation for the layer-averaged ID contains a stray double comma; also use a distinct symbol for the layer-averaged quantity to avoid confusion with the layerwise estimator.
  2. [§B.3] 'Grokking phoneme' should be 'grokking phenomenon.'
  3. [Appendix B.10] 'matplotlib' is listed twice in the software environment list.
  4. [Abstract and §5] The abstract and Section 5 still list mutual information among the coordinated signals, while Section 3.4 and Section 4.4 state that MI was too volatile to use; the narrative should be aligned.
  5. [Table 1] The heading 'V ocabulary' should be 'Vocabulary.'
  6. [§B.8] 'receptively' should be 'respectively.'

Circularity Check

2 steps flagged · score 8.0 of 10

The central 'phase transition' claim is forced by defining the transition as the crossing of two arbitrarily scaled curves with different units.

  1. self definitional [Section 4.1, Eq. (1), Eq. (2), Fig. 2 and caption]
    "Across all model configurations, we observe a robust two-phase training dynamic: an initial regime of rising ID and elevated curvature, followed by a transition into flatter curvature and stabilised representational complexity (Fig. 2). This transition is marked by a consistent intersection between the Hessian curvature score (blue) and ID trajectories (red), which we interpret as a phase shift in learning dynamics."

    C(H) from Eq. (1) is Tr(H)/sqrt(r_eff), a quantity with units of a Hessian trace divided by a dimensionless effective rank, while ID from Eq. (2) is a dimensionless estimated local dimension. The two are plotted on separate vertical axes with independently chosen ranges in Fig. 2. A decreasing C(H) curve and an increasing ID curve will cross somewhere whenever the chosen axis ranges overlap, and the crossing step can be moved monotonically by multiplying either curve by a positive constant, equivalently by changing either twin-axis scale. The paper gives no normalization rule or invariance argument, so the reported intersection at 'around step 5000' for the large model is a property of the plot's scale, not an independently measured event.

  2. other [Appendix B.6 (Probing Framework and Label Construction) and Section 3.3]
    "Due to the synthetic nature of our dataset, both POS and semantic labels are deterministically derived from token names. For example, a token such as noun3 is assigned the POS label NOUN and may additionally be annotated with semantic roles such as AGENT or ENTITY, depending on the symbolic structure of the task."

    Section 3.3 defines abstraction as internalizing role-based generalizations 'that extend beyond surface lexical forms—for example, recognising the ARG2 role regardless of whether it is realised as "noun3", or "location22".' But because every token name deterministically encodes its POS and semantic labels, a linear probe can achieve high confidence by memorizing the surface token identity noun3, verb2, and so on. The rising probe-confidence curves used as evidence of 'emerging syntactic and semantic accuracy' are therefore confounded with surface-form memorization by construction of the label scheme; they do not by themselves separate abstraction over category structure from recognition of token-name identity.

full rationale

The central result of the paper is that phase transitions are 'marked by a consistent intersection' between the Hessian curvature score C(H) and the intrinsic dimensionality ID. This intersection is not an invariant observable: C(H) and ID have different units, are plotted on separate axes with free ranges, and the crossing time is movable by rescaling either curve. Without a stated normalization or invariance criterion, the reported transition step is an artifact of plotting scale rather than an independent measurement, so the main claim reduces to the paper's own definition of the transition marker. The linguistic alignment claims are further weakened because the probe labels are deterministic functions of token names, so probe accuracy can reflect surface token memorization rather than the cross-token abstraction the paper defines. There is a self-citation to Aljaafari et al. [3] in Section 4.4, but it concerns only the discarded mutual-information signal and is not load-bearing. The experiments are otherwise self-contained and reproducible, but the central phase-transition identification is forced by the chosen definition and axis scaling.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The main quantitative result rests on a handful of manually chosen corpus parameters and on the unstated assumption that the crossing of two differently scaled trajectories is a meaningful event. No new physical entities are introduced; the ABSynth corpus and the C(H) score are methodological artifacts rather than invented entities.

free parameters (4)
  • ABSynth Zipf exponent alpha = 1.05
    Chosen by hand in Eq. (5) to shape token frequency; controls corpus statistics and difficulty.
  • Cluster association strength intervals = intra [0.4, 0.7]; inter [0.05, 0.2]
    Chosen in Eq. (6) to create collocational structure; affects how semantically predictable tokens are.
  • Entropy tier boundaries and complexity mix = low 0.5-1.5, medium 1.5-3.0, high 3.0-4.5 bits; 55/35/10 simple/medium/complex
    Selected to calibrate contextual predictability and global entropy profile; determines the difficulty of the corpus.
  • Training epoch count = 500 epochs for all models
    Set beyond Chinchilla-based minima so that phase transitions appear; transition timing depends on this extended budget.
assumptions (5)
  • domain assumption The top-K Lanczos Hessian spectrum is a faithful summary of loss-landscape curvature relevant to abstraction.
    Used throughout Section 3.1 to define C(H); no convergence or coverage check is reported.
  • domain assumption Flat minima with concentrated curvature correspond to generalisation and abstraction.
    Section 3.1 cites prior work but does not validate this mapping for the ABSynth tasks.
  • ad hoc to paper The crossing point of unnormalized C(H) and mean ID curves marks a real phase transition.
    Section 4.1 interprets the intersection as a phase shift; the two quantities have no common unit and the crossing time changes under monotone rescaling.
  • domain assumption Linear probes on frozen hidden states measure internal abstraction rather than surface token statistics.
    Section 3.3 uses probe confidence as evidence of abstraction; because ABSynth token names encode their labels, surface statistics can drive probe performance.
  • domain assumption The ABSynth synthetic corpus captures the relevant properties of natural language for abstraction emergence.
    Section 6 concedes the corpus does not fully capture the ambiguity and richness of natural language; the paper generalizes from it anyway.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TRACE for Tracking the Emergence of Semantic Representations in Transformers." pith.science (2026). https://pith.science/paper/S6TQ2AMI

@misc{pith2026250517998,
  author       = {Pith},
  title        = {Pith review of: TRACE for Tracking the Emergence of Semantic Representations in Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6TQ2AMI}},
  note         = {Machine review of arXiv:2505.17998}
}
read the original abstract

Modern transformer models exhibit phase transitions during training, distinct shifts from memorisation to abstraction, but the mechanisms underlying these transitions remain poorly understood. Prior work has often focused on endpoint representations or isolated signals like curvature or mutual information, typically in symbolic or arithmetic domains, overlooking the emergence of linguistic structure. We introduce TRACE (Tracking Representation Abstraction and Compositional Emergence), a diagnostic framework combining geometric, informational, and linguistic signals to detect phase transitions in Transformer-based LMs. TRACE leverages a frame-semantic data generation method, ABSynth, that produces annotated synthetic corpora with controllable complexity, lexical distributions, and structural entropy, while being fully annotated with linguistic categories, enabling precise analysis of abstraction emergence. Experiments reveal that (i) phase transitions align with clear intersections between curvature collapse and dimension stabilisation; (ii) these geometric shifts coincide with emerging syntactic and semantic accuracy; (iii) abstraction patterns persist across architectural variants, with components like feedforward networks affecting optimisation stability rather than fundamentally altering trajectories. This work advances our understanding of how linguistic abstractions emerge in LMs, offering insights into model interpretability, training efficiency, and compositional generalisation that could inform more principled approaches to LM development.

Figures

Figures reproduced from arXiv: 2505.17998 by the authors.

Figure 1
Figure 1. Overview of the TRACE framework, which integrates the monitoring of (i) intrinsic [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Coordinated dynamics of Hessian Curvature Score (blue) and Average Intrinsic Dimension [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Probe confidence scores over training steps for the large model. Each subplot corresponds [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: SRL performance per label across models and training steps [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The frame-semantic data generation pipeline: (1) Frame selection with semantic roles, (2) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Small model — Semantic (left) and POS (right) probe confidence scores at Layer 0. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Medium model — Semantic (left) and POS (right) probe confidence scores across layers. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Large model — POS probe confidence scores across layers. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Randomly initialised Small model — Semantic (left) and POS (right) probe confidence [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Randomly initialised Medium model — Semantic (left) and POS (right) probe confidence [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Randomly initialised Large model — Semantic (left) and POS (right) probe confidence [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Mutual Information (MI) between adjacent layers over training steps for small (top-left), [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 34 canonical work pages

  1. [1]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7319–7328, Online, 2021....

    work page 2021

  2. [2]

    How to escape sharp minima with random perturbations

    Kwangjun Ahn, Ali Jadbabaie, and Suvrit Sra. How to escape sharp minima with random perturbations. In Proceedings of the 41st International Conference on Machine Learning , ICML’24. JMLR.org, 2024

    work page 2024

  3. [3]

    Carma: Enhanced compositional- ity in llms via advanced regularisation and mutual information alignment

    Nura Aljaafari, Danilo S Carvalho, and André Freitas. Carma: Enhanced compositional- ity in llms via advanced regularisation and mutual information alignment. arXiv preprint arXiv:2502.11066, 2025

    arXiv 2025

  4. [4]

    Intrinsic dimension of data representations in deep neural networks

    Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [5]

    Baker, Charles J

    Collin F. Baker, Charles J. Fillmore, and John B. Lowe. The Berkeley FrameNet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86–90, Montreal, Quebec, Canada, August 1998. Association for Computational Linguistics. doi: 10.3115/980845.980860. U...

    arXiv 1998

  6. [6]

    Mutual information neural estimation

    Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531–540. PMLR, 10–15 ...

    work page 2018

  7. [7]

    Visualizing high-dimensional loss landscapes with hessian directions

    Lucas Böttcher and Gregory Wheeler. Visualizing high-dimensional loss landscapes with hessian directions. Journal of Statistical Mechanics: Theory and Experiment, 2024(2):023401, 2024

    work page 2024

  8. [8]

    Algorithms for manifold learning

    Lawrence Cayton et al. Algorithms for manifold learning. eScholarship, University of California, 2005

    work page 2005

Show all 58 references
  1. [9]

    Bridging information-theoretic and geometric compression in language models

    Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging information-theoretic and geometric compression in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1...

    2023 doi

  2. [10]

    Emergence of a high-dimensional abstraction phase in language transformers

    Emily Cheng, Diego Doimo, Corentin Kervadec, Iuri Macocco, Lei Yu, Alessandro Laio, and Marco Baroni. Emergence of a high-dimensional abstraction phase in language transformers. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openrevi...

    2025

  3. [11]

    Information-theoretic progress measures reveal grokking is an emergent phase transition

    Kenzo Clauw, Daniele Marinazzo, and Sebastiano Stramaglia. Information-theoretic progress measures reveal grokking is an emergent phase transition. In ICML 2024 Workshop on Mecha- nistic Interpretability, 2024. URL https://openreview.net/forum?id=Q4NH6hEPIX

    2024

  4. [12]

    The devil is in the detail: Simple tricks improve systematic generalization of transformers

    Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, Dominican Republic, November 2021

    2021

  5. [13]

    Sparse autoen- coders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. 10

    2023 arXiv

  6. [14]

    Knowledge neurons in pretrained transformers

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021

    2021 arXiv

  7. [15]

    The generalized ratios intrinsic dimension estimator

    Francesco Denti, Diego Doimo, Alessandro Laio, and Antonietta Mira. The generalized ratios intrinsic dimension estimator. Scientific Reports, 12(1):20005, 2022

    2022

  8. [16]

    A complexity- based theory of compositionality

    Eric Elmoznino, Thomas Jiralerspong, Yoshua Bengio, and Guillaume Lajoie. A complexity- based theory of compositionality. arXiv preprint arXiv:2410.14817, 2024

    2024 arXiv

  9. [17]

    Estimating the intrinsic dimension of datasets by a minimal neighborhood information

    Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140, 2017

    2017

  10. [18]

    Not all layers of llms are necessary during inference

    Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of llms are necessary during inference. CoRR, abs/2403.02181, 2024. URL https://doi.org/10.48550/arXiv.2403.02181

    2024 doi arXiv

  11. [19]

    Fillmore

    Charles J. Fillmore. Frame semantics. In Linguistics in the Morning Calm, pages 111–137. Hanshin Publishing Co., Seoul, 1982

    1982

  12. [20]

    Studying large language model generalization with influence functions

    Roger Grosse. Studying large language model generalization with influence functions. In Proceedings of the 38th Conference on Neural Information Processing Systems, NeurIPS ’24,

  13. [21]

    Visualizing and understanding the effectiveness of bert

    Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv:1908.05620, 2019

    1908 arXiv

  14. [22]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in neural informat...

    2022

  15. [23]

    The developmental landscape of in-context learning

    Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning. arXiv preprint arXiv:2402.02364, 2024

    2024 arXiv

  16. [24]

    Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795, 2020

    Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795, 2020

    2020

  17. [25]

    Are sparse autoencoders useful? a case study in sparse probing

    Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502.16681, 2025

    2025 arXiv

  18. [26]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  19. [27]

    Flow of information in feed- forward denoising neural networks

    Pejman Khadivi, Ravi Tandon, and Naren Ramakrishnan. Flow of information in feed- forward denoising neural networks. In 2018 IEEE 17th International Conference on Cognitive Informatics and Cognitive Computing (ICCI*CC) , pages 166–173, 2018. doi: 10.1109/ICCI-CC.2018.8482098

    2018

  20. [28]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  21. [29]

    Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

    Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pages 2873–2882. PMLR, 2018

    2018

  22. [30]

    An iteration method for the solution of the eigenvalue problem of linear differential and integral operators

    Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of research of the National Bureau of Standards, 45 (4):255–282, 1950. 11

    1950

  23. [31]

    Geometric signa- tures of compositionality across a language model’s lifetime

    Jin Hwa Lee, Thomas Jiralerspong, Lei Yu, Yoshua Bengio, and Emily Cheng. Geometric signa- tures of compositionality across a language model’s lifetime. arXiv preprint arXiv:2410.01444, 2024

    2024 arXiv

  24. [32]

    Break it down: Evidence for structural compositionality in neural networks

    Michael Lepori, Thomas Serre, and Ellie Pavlick. Break it down: Evidence for structural compositionality in neural networks. Advances in Neural Information Processing Systems, 36: 42623–42660, 2023

    2023

  25. [33]

    Systematic generalization and emergent structures in transformers trained on structured tasks, 2023

    Yuxuan Li and James McClelland. Systematic generalization and emergent structures in transformers trained on structured tasks, 2023. URL https://openreview.net/forum?id= pXDmbfVL_SB

    2023

  26. [34]

    Multiscale geometric methods for data sets i: Multiscale svd, noise and curvature

    Anna V Little, Mauro Maggioni, and Lorenzo Rosasco. Multiscale geometric methods for data sets i: Multiscale svd, noise and curvature. Applied and Computational Harmonic Analysis, 43 (3):504–567, 2017

    2017

  27. [35]

    Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021

    2021

  28. [36]

    Progress mea- sures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW

    2023

  29. [37]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  30. [38]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances...

    2024

  31. [39]

    The geometry of categorical and hierarchical concepts in large language models

    Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/forum?id=KXuYjuBzKo

    2024

  32. [40]

    Zipf’s word frequency law in natural language: A critical review and future directions

    Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21:1112–1130, 2014

    2014

  33. [41]

    Grokking: Gen- eralization beyond overfitting on small algorithmic datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

    2022 arXiv

  34. [42]

    The shape of learning: Anisotropy and intrinsic dimen- sions in transformer-based models

    Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dim- itrov, and Andrey Kuznetsov. The shape of learning: Anisotropy and intrinsic dimen- sions in transformer-based models. In Yvette Graham and Matthew Purver, editors, Find- ings of the Associat...

    2024

  35. [43]

    The effective rank: A measure of effective dimensionality

    Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606–610. IEEE, 2007

    2007

  36. [44]

    A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization

    Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 948...

    2021

  37. [45]

    Analysing mathematical reasoning abilities of neural models

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations,

  38. [46]

    Analytic insights into structure and rank of neural network hessian maps

    Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. Analytic insights into structure and rank of neural network hessian maps. Advances in Neural Information Processing Systems, 34: 23914–23927, 2021

    2021

  39. [47]

    Layer by layer: Uncovering hidden representations in language models

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025

    2025 arXiv

  40. [48]

    Negative results for saes on downstream tasks and deprioritising sae research (gdm mech interp team progress up- date #2), March 2025

    Lewis Smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Negative results for saes on downstream tasks and deprioritising sae research (gdm mech interp team progress up- date #2), March 2025. URL https://www...

    2025

  41. [49]

    The geometry of hidden representations of large transformer models

    Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36:51234–51252, 2023

    2023

  42. [50]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017

    2017

  43. [51]

    Loss landscape geometry reveals stagewise development of transformers

    George Wang, Matthew Farrugia-Roberts, Jesse Hoogland, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape geometry reveals stagewise development of transformers. In High-dimensional Learning Dynamicsß: The Emergence of Structure and Reasoning , 2024. URL https://openre...

    2024

  44. [52]

    Penalizing gradient norm for efficiently improving generalization in deep learning

    Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International conference on machine learning , pages 26982–26992. PMLR, 2022

    2022

  45. [53]

    Algorithmic capabilities of random transformers

    Ziqian Zhong and Jacob Andreas. Algorithmic capabilities of random transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id=plH8gW7tPQ

    2024

  46. [54]

    Susskind, Samy Bengio, and Preetum Nakkiran

    Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua M. Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. In The Twelfth International Conference on Learning Representations,

  47. [55]

    noun2 verb3 noun5

    George Kingsley Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949. 13 A Frame-Semantic Data Generation Framework This appendix describes our controllable synthetic data generation framework ABSynth. Unlike template-based synthetic datasets [29] or ta...

    1949

  48. [57]

    URL https://openreview.net/forum?id=AssIuHnmHX

  49. [2019]

    URL https://openreview.net/forum?id=H1gR5iR5FX. 12

  50. [2024]

    Workshop on Scalable Continual Learning for Lifelong Foundation Models

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.