Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Learning curves theory for hierarchically compositional data with power-law distributed features

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In a hierarchy of production rules, making rule frequencies Zipf-distributed turns classification learning curves into power laws with exponent a/(1+a), while next-token prediction keeps its hierarchy-controlled power law.

desk verdict A clean synthesis of Zipf and hierarchy that yields a new classification exponent and a striking next-token invariance, held up by a heuristic learnability assumption that the paper doesn't fully justify. read the letter →

arxiv 2505.07067 v1 pith:BZLTA3UV submitted 2025-05-11 stat.ML cond-mat.dis-nncs.LG

Francesco Cagnetta , Hyunmo Kang , Matthieu Wyart This is my paper · ORCID
classification stat.MLcond-mat.dis-nncs.LG
keywords neuralscalinglawslearningcurvetheoryrandomhierarchymodelZipfdistributionprobabilisticcontext-freegrammarsclassificationnext-tokenpredictionhierarchicalcompositionaldata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neural scaling laws in real data are sometimes attributed to power-law-distributed features and sometimes to hierarchical compositional structure. This paper studies both at once by adding Zipf-distributed production rules to the Random Hierarchy Model, a tree-shaped probabilistic grammar. For root classification, it establishes a power-law learning curve, $\varepsilon(P) \sim (P/(v m^{L-1}))^{-a/(1+a)}$, whose exponent is fixed by the Zipf exponent $a$ and whose pre-asymptotic scale is fixed by the hierarchy. For next-token prediction, the paper shows the opposite: the Zipf exponent reshapes local features of the curve but not the asymptotic decay, which remains governed by the branching structure $m/v^{s-1}$. This matters because it separates two candidate explanations of real scaling laws, frequency statistics and hierarchical composition, and assigns them to different task types.

What carries the argument

The Random Hierarchy Model: a depth-$L$ context-free grammar in which each nonterminal has $m$ unambiguous production rules, one Zipf-distributed layer with probabilities $f_k \propto k^{-(1+a)}$, and all other layers uniform. Learning is tracked through correlation functions $C_j(y,\mu)$ for label-tuple correlations in classification and $C_j(\mu,\nu)$ for token-tuple correlations in next-token prediction; their variance over grammar realisations and over finite-sample noise determines when a rule's signal becomes detectable. Setting signal equal to noise yields per-rule sample complexities $P_k^*=v m^{L-1}/f_k$ for classification and $P_{\ell,k}=v m^{2\ell-3}(1-m/v^{s-1})f_k\sum_{k'}f_{k'}^2$ for next-token steps. Combining these thresholds with the probabilities of resolved rules and with the average $s^\ell$-gram cross-entropies produces the learning curves.

What would settle it

Train a deep transformer on next-token RHM data (e.g. $s=2$, $v=32$, $m=8$, $L=4$) with $a=0.2$ and $a=4$, and compare the late slopes of $L(P)-L_\infty(a)$ on a log-log plot: Eq. (12) says both approach $P^{-\log(4/32)/(2\log 8)}$, so any systematic slope separation would refute the claimed independence; similarly, classification curves should exhibit slopes $-a/(1+a)$ after the $v m^{L-1}$ rescaling.

Watch

Extended reading notes

Core claim

The central discovery is that, within the Random Hierarchy Model with one layer of power-law rule frequencies $f_k \propto k^{-(1+a)}$, the task determines which ingredient sets the scaling exponent. Classification errors obey $\varepsilon(P)=1-(\sum_{k:P_k^*<P} f_k)^{s^{L-1}}$, which for $P \gg v m^{L-1}$ becomes $s^{L-1}(P/(v m^{L-1}))^{-a/(1+a)}$; the hierarchy contributes a large constant and delayed onset, not the exponent. Next-token prediction first passes through an $s$-gram memorisation stage with the Zipf exponent, but the long-distance reconstruction of the hidden tree follows sample complexities $P_{\ell,k}=v m^{2\ell-3}(1-m/v^{s-1})f_k(\sum_{k'}f_{k'}^2)$, whose level-dependence is the same as in the uniform case, so the asymptotic excess loss decays as $P^{-\log(m/v^{s-1})/(2\log m)}$ independent of $a$. The paper thereby unifies the power-law-units view and the hierarchical-structure view by showing that both are right, each for a different task.

Load-bearing premise

The derivation relies on Assumption 3.1, that a rule is learned exactly when its correlation signal becomes detectable in finite data, and, for classification in the strict regime, on the requirement that all $s^{L-1}$ patches be resolved, which is guaranteed only when $m=v^{s-1}$; neither premise is derived from gradient-descent dynamics.

Editorial extensions

If this is right

  • Classification error on Zipf-RHM data decays as $\varepsilon(P)\sim s^{L-1}(P/(v m^{L-1}))^{-a/(1+a)}$; larger $a$ yields faster decay, while $L$ and $m$ set the horizontal offset of the power law.
  • The sigmoidal learning curve of the uniform RHM becomes a power law once rule frequencies are uneven, so the same hierarchy can look qualitatively different under different feature statistics.
  • Next-token prediction retains its exponent $-\log(m/v^{s-1})/(2\log m)$ for all $a$, meaning that the long-range structure of language-like data, not token frequency, controls the late-time learning curve.
  • The first stage of next-token prediction is $s$-gram memorisation with exponent $a/(1+a)$; this stage is visible in early data but disappears from the asymptotics.
  • All classification curves collapse when $P$ is rescaled by $v m^{L-1}$, providing a direct experimental signature of the hierarchy's contribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: putting Zipf rules on two layers should make input probabilities products of Zipf factors; the classification exponent may then interpolate between $a/(1+a)$ and a combination of the layer exponents rather than the single-layer value.
  • An architectural prediction follows from Assumption 3.1: different optimizers could shift the $P$ thresholds at which rules are learned, changing prefactors and crossover locations while leaving the asymptotic exponents intact.
  • For real language corpora this implies a cross-corpus diagnostic: if late next-token slopes vary with token-frequency distribution at fixed branching structure, some ingredient beyond this tree hierarchy is active.
  • For real classification datasets, a long pre-asymptotic plateau can coexist with a strong final power law, so measuring only short-data performance would underestimate the asymptotic learning rate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies neural scaling laws in a Random Hierarchy Model (RHM) in which production rules follow a Zipf distribution. For classification, it derives the asymptotic learning curve ε(P) ≃ s^{L-1}(P/(v m^{L-1}))^{-a/(1+a)} (Eq. 8), implying that the Zipf exponent a sets the power-law decay while the hierarchy sets a multiplicative pre-asymptotic scale. For next-token prediction, it argues that the asymptotic exponent remains -log(m/v^{s-1})/(2 log m) (Eq. 12), independent of the rule distribution, and that Zipf statistics only modify finite-size corrections. The theory is built on Assumption 3.1, which asserts that each production rule is learned as soon as its correlation signal exceeds sampling noise; all main results follow from this sharp threshold. The predictions are tested with deep CNNs for classification and deep transformers for next-token prediction, with code available online.

Significance. If the results hold, the paper offers a concrete unification of two strands of scaling-law theory: it shows how power-law distributed features (the Hutter mechanism) and hierarchical composition interact, with the classification exponent a/(1+a) directly inherited from the rule distribution and the next-token exponent governed by the tree structure alone. The derivations are transparent, the asymptotic calculations in Appendix B are standard, and the empirical validation spans two architectures with publicly released code. The predictions are falsifiable: the data collapse under P/(v m^{L-1}) for classification and the invariance of the next-token exponent across a are crisp experimental signatures. The principal caveat is the heuristic nature of Assumption 3.1 and the restriction of the classification derivation to the m = v^{s-1} regime, which limits the generality of the abstract claims until those points are addressed.

major comments (3)
  1. [Section 3.2 (Assumption 3.1) and its use in Sections 4.1 and 5.1] The central results hinge on Assumption 3.1, which asserts that a production rule is learned exactly when its effect on correlations can be detected from the training data. This is a sharp threshold that is not derived from gradient-descent dynamics. Under the alternative scaling P*_k ∝ f_k^{-β} with β ≠ 1, the classification exponent in Eq. (8) would become β a/(1+β a), breaking the claimed unification with Hutter's memorization exponent. I recommend either deriving Assumption 3.1 in a tractable setting (for example, a linear network or online SGD with a single rule) or providing a direct empirical test of the per-rule sample complexity P*(f) across a range of f. The current experiments validate the aggregate learning curve, not the per-rule threshold.
  2. [Section 4.1, Eq. (7) and footnote 2] Equation (7) is derived under the condition that classification succeeds only when all s^{L-1} input patches are resolvable, which the authors acknowledge is exact only when m = v^{s-1}. For m < v^{s-1}, the root can sometimes be inferred without resolving every patch, so the prefactor and possibly the exponent in Eq. (8) are not established outside the tested regime; all classification experiments in Figs. 2-3 and Appendix C use m = v. Since the abstract presents the exponent a/(1+a) as a general result for hierarchical compositional data, the paper should either restrict the claim to m = v^{s-1} or extend the derivation using the optimal-decoder probability from Sclocchi et al. (2024).
  3. [Section 4.2] The claim that power-law distributed rules at an arbitrary layer ℓ ≠ 1 lead to the same classification exponent is only sketched in one paragraph; no derivation is given for how the variance and sample-complexity arguments of Section 4.1 generalize to higher layers. The empirical evidence in Fig. 2 (right) and Fig. 7 supports the claim, but the theory as written covers only level-1 nonuniformity. If the intended contribution is the general statement that the classification exponent depends on the rule distribution, this gap should be closed or the contribution restated.
minor comments (4)
  1. [Section 5, opening sentence] The phrase 'Assumption Theorem 3.1' appears to be a typo and should read 'Assumption 3.1'.
  2. [Section 5.1, paragraph on sampling variance] The sampling variance is written as 'f_k(µ)/(v^2P)^{-1}'; the exponent appears to be a typo and should read f_k(µ)/(v^2 P).
  3. [Section 1.1, third bullet] The sentence 'decays asymptotically as P^{−a/(1+a)} with the;' is incomplete and should be finished.
  4. [Equation (18) and Appendix E] The quantity H_{2,a,m} is used in Eq. (18) without being defined in the main text; a brief definition or a pointer to Appendix E in the main text would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's scaling-law predictions follow from its explicit Assumption 3.1 and from independent, parameter-free variance calculations, not from fitted parameters or self-citation chains.

full rationale

The derivation chain is self-contained in the sense required by this review. For classification, Eq. (8) is obtained from Eq. (7), which in turn follows from the correlation-variance calculation around Eq. (6): the per-rule sample complexity P*_k = v m^{L-1}/f_k is computed by balancing the signal variance f_k^2/(v^3 m^{L-1}) against the sampling noise f_k/(v^2 P), then Zipf's law f_k ∝ k^{-(1+a)} is inserted and Appendix B evaluates the resulting tail sum with Euler-Maclaurin. No parameter is fitted to the target learning curve; the dotted lines in Figs. 2-3 are direct evaluations of Eq. (7). For next-token prediction, the Zipf-independence claim follows from the derived variance formula (Eq. 13) and sample complexity (Eq. 14), which share the same level-ℓ scaling as the uniform case, combined with the computed s^ℓ-gram entropies (Eq. 18); the final exponent is then obtained algebraically. The self-citations to Cagnetta et al. (2024) and Cagnetta & Wyart (2024) provide the uniform-RHM variance and uniform scaling law as building blocks; these are independent published results, and the new Zipf dependence is added, not assumed away. Assumption 3.1 is an openly stated modeling hypothesis connecting statistical detectability to learning; it is not a re-definition of the predicted quantity, and the paper explicitly footnotes the regime (m = v^{s-1}) where the 'all patches must be resolved' condition is exact. Thus there is no concrete step where an output equals an input by construction or where a fitted constant absorbs the target exponent.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theory rests on the RHM generative family and on Assumption 3.1 connecting trainability to correlation detectability; all sample-complexity and scaling conclusions follow from that assumption plus variance-balance estimates. No free parameters are fitted to the learning curves.

assumptions (6)
  • domain assumption Data are generated by an RHM/PCFG with L layers of v nonterminal symbols, branching s, m unambiguous production rules per symbol, and production-rule probabilities f_k proportional to k^{-(1+a)} in one layer and uniform elsewhere.
    Section 2 sets this generative model; all subsequent sample-complexity calculations assume it.
  • ad hoc to paper Assumption 3.1: each production rule is learnt exactly when its effect on correlations can be detected from training data.
    Section 3.2 states this; Eqs. (7), (14) and the next-token scaling law follow from it. It is not derived from SGD or transformer mechanics.
  • ad hoc to paper For classification, the model classifies correctly iff all s^{L-1} input patches are resolvable; if m < v^{s-1} the root might be inferable without all patches.
    Section 4.1 and footnote 2; used to write epsilon(P) = 1 - (sum f_k)^{s^{L-1}}. Experiments take m = v so the condition holds.
  • ad hoc to paper For next-token prediction, the first learning step is memorization of s-tuples (Hutter's regime) and subsequent steps reconstruct the tree using token-token correlations.
    Section 5.1; the stepwise picture with first-step memorization is an additional modeling assumption beyond Assumption 3.1.
  • domain assumption Uniform-RHM correlation variances, sample complexities, and the uniform next-token scaling law are taken from prior work.
    Sections 3.2 and 5 use the uniform-RHM results of Cagnetta et al. 2024 and Cagnetta and Wyart 2024; these are external, previously derived, but not re-derived here.
  • domain assumption Asymptotic limits used: large vocabulary v, 1 << m << v^{s-1}, and neglect of |Vc,ell| > 2 for cross-entropies.
    Appendix E; these approximations control the next-token loss expression Eq. (18).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning curves theory for hierarchically compositional data with power-law distributed features." pith.science (2026). https://pith.science/paper/BZLTA3UV

@misc{pith2026250507067,
  author       = {Pith},
  title        = {Pith review of: Learning curves theory for hierarchically compositional data with power-law distributed features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZLTA3UV}},
  note         = {Machine review of arXiv:2505.07067}
}
read the original abstract

Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is thought to occur in language and images. To unify these views, we consider classification and next-token prediction tasks based on probabilistic context-free grammars -- probabilistic models that generate data via a hierarchy of production rules. For classification, we show that having power-law distributed production rules results in a power-law learning curve with an exponent depending on the rules' distribution and a large multiplicative constant that depends on the hierarchical structure. By contrast, for next-token prediction, the distribution of production rules controls the local details of the learning curve, but not the exponent describing the large-scale behaviour.

Figures

Figures reproduced from arXiv: 2505.07067 by the authors.

Figure 1
Figure 1. Pictorial representation of a derivation according to the RHM, with depth L = 3 and branching factor s = 2. A classifi￾cation task requires predicting the root label (blue square) from the leaves. The correlations between the 2-tuples of leaves (e.g. (x5, x6)) and the label y can be used to infer the hidden symbol above the 2-tuple (µ (1) 3 for (x5, x6)). A next-token prediction task requires predicting the last obs… view at source ↗
Figure 2
Figure 2. Left: Learning curves of 3-layers CNNs trained on RHM data with L = 2, s = 2, v = m = 25 and Zipf exponent a indicated in caption. Solid lines are the empirical learning curves whereas dotted lines are predictions from Eq. (7). The dashed line represents the scaling law ϵ ∼ P −a/(1+a) . Right: As in the left panel, but v = m = 100. Here a is fixed and the layer where production rules are Zipf-distributed changes. Th… view at source ↗
Figure 3
Figure 3. Left: Learning curves in the same setting as [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Left: Empirical learning curve of one-layer transformers trained for next-token prediction on RHM data with L = 1, s = 2, v = 128, m = 32 and Zipf exponent a as in the key. Vertical dashed lines mark the sample sizes required to learn the most frequent rules: vm in the…
Figure 6
Figure 6. Figure 6: Empirical scaling laws of depth-4 transformers trained on RHM next-token prediction with L = 4, s = 2, v = 32, m = 8 and varying a. The limit a → ∞ corresponds to having only one production rule per level-1 nonterminal symbol. The red dashed line is a guide to the eye …
Figure 5
Figure 5. Figure 5: Average cross-entropies of the s ℓ -grams versus ℓ, for RHM datasets with s = 2, v = 32, m = 8, with the colour denoting the Zipf exponent. The points are obtained by averaging the cross￾entropies over 32 independent realisations of the RHM. The cross￾entropies of the …
Figure 7
Figure 7. Figure 7: Learning Curves of classification with L = 3. Left: Learning curves of 4-layers CNNs trained on RHM data with L = 3, s = 2, v = m = 10 and Zipf exponent a indicated in caption. Solid lines are the empirical learning curves whereas dotted lines are predictions from Eq. …
Figure 8
Figure 8. Figure 8: Learning curves with varying m values. Left: Learning curves in the same setting as above, with Zipf exponent a = 2 and m = v indicated in caption. Solid lines are the empirical learning curves whereas dotted lines are predictions from Eq. (7). Right: All curves collap…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    In overparameterized diffusion models, generalization happens first and memorization starts later, with the memorization time growing linearly with dataset size.

Reference graph

Works this paper leans on

44 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A., Goyal, N., and Tsvetkov, Y

    Ahuja, K., Balachandran, V., Panwar, M., He, T., Smith, N. A., Goyal, N., and Tsvetkov, Y. Learning syntax without planting trees: Understanding when and why transformers generalize hierarchically. arXiv preprint arXiv:2404.16367, 2024

    arXiv 2024

  3. [3]

    and Li, Y

    Allen-Zhu, Z. and Li, Y. Physics of language models: Part 1, learning hierarchical language structures. arXiv preprints, abs/2305.13673, May, 2023

    arXiv 2023

  4. [4]

    Explaining neural scaling laws

    Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021

    arXiv 2021

  5. [5]

    Spectrum dependent learning curves in kernel regression and wide neural networks

    Bordelon, B., Canatar, A., and Pehlevan, C. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp.\ 1024--1034. PMLR, 2020

    2020

  6. [6]

    A dynamical model of neural scaling laws

    Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=nbOY1OmtRc

    work page 2024

  7. [7]

    What languages are easy to language-model? a perspective from learning probabilistic regular languages

    Borenstein, N., Svete, A., Chan, R., Valvoda, J., Nowak, F., Augenstein, I., Chodroff, E., and Cotterell, R. What languages are easy to language-model? a perspective from learning probabilistic regular languages. arXiv preprint arXiv:2406.04289, 2024

    arXiv 2024

  8. [8]

    and Wyart, M

    Cagnetta, F. and Wyart, M. Towards a theory of how the structure of language is acquired by deep neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=NaCXcUKihH

    work page 2024

Show all 44 references
  1. [9]

    What can be learnt with wide convolutional neural networks? In International Conference on Machine Learning, pp.\ 3347--3379

    Cagnetta, F., Favero, A., and Wyart, M. What can be learnt with wide convolutional neural networks? In International Conference on Machine Learning, pp.\ 3347--3379. PMLR, 2023

    2023

  2. [10]

    M., Favero, A., and Wyart, M

    Cagnetta, F., Petrini, L., Tomasini, U. M., Favero, A., and Wyart, M. How deep neural networks learn compositional data: The random hierarchy model. Phys. Rev. X, 14: 0 031001, Jul 2024. doi:10.1103/PhysRevX.14.031001. URL https://link.aps.org/doi/10.1103/PhysRevX.14.031001

    2024 doi

  3. [11]

    and De Vito, E

    Caponnetto, A. and De Vito, E. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7: 0 331--368, 2007

    2007

  4. [12]

    Zipf’s law for word frequencies: Word forms versus lemmas in long texts

    Corral, A., Boleda, G., and Ferrer-i Cancho, R. Zipf’s law for word frequencies: Word forms versus lemmas in long texts. PLOS ONE, 10 0 (7): 0 e0129031, 2015. ISSN 1932-6203. doi:10.1371/journal.pone.0129031. URL http://dx.doi.org/10.1371/journal.pone.0129031

    2015 doi

  5. [13]

    Locality defeats the curse of dimensionality in convolutional teacher-student scenarios

    Favero, A., Cagnetta, F., and Wyart, M. Locality defeats the curse of dimensionality in convolutional teacher-student scenarios. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openrevi...

    2021

  6. [14]

    How transformers learn structured data: insights from hierarchical filtering

    Garnier-Brun, J., M \'e zard, M., Moscato, E., and Saglietti, L. How transformers learn structured data: insights from hierarchical filtering. arXiv preprint arXiv:2408.15138, 2024

    2024 arXiv

  7. [15]

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  8. [16]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2022

  9. [17]

    Learning curve theory

    Hutter, M. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021

    2021 arXiv

  10. [18]

    Joshi, A. K. Tree adjoining grammars: How much context-sensitivity is required to provide reasonable structural descriptions? In Dowty, D. R., Karttunen, L., and Zwicky, A. M. (eds.), Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives, pp.\ 20...

    1985

  11. [19]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  12. [20]

    Knuth, D. E. Semantics of context-free languages. Mathematical systems theory, 2 0 (2): 0 127--145, 1968

    1968

  13. [21]

    M., Bartlett, P., and Lee, J

    Lin, L., Wu, J., Kakade, S. M., Bartlett, P., and Lee, J. D. Scaling laws in linear regression: Compute, parameters, and data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=PH7sdEanXP

    2024

  14. [22]

    and Shalev-Shwartz, S

    Malach, E. and Shalev-Shwartz, S. A provably correct algorithm for deep learning that actually works. Preprint at http://arxiv.org/abs/1803.09522, 2018. URL http://arxiv.org/abs/1803.09522

    2018 arXiv

  15. [23]

    and Shalev-Shwartz, S

    Malach, E. and Shalev-Shwartz, S. The implications of local correlation on learning some deep functions. In Advances in Neural Information Processing Systems, volume 33, pp.\ 1322--1332, 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/0e4ceef65add6cf21c0f3...

    2020

  16. [24]

    A., and Sully, J

    Maloney, A., Roberts, D. A., and Sully, J. A solvable model of neural scaling laws. arXiv:2210.16859, 2022

    2022 arXiv

  17. [25]

    T., Frank, R., and Linzen, T

    McCoy, R. T., Frank, R., and Linzen, T. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8: 0 125--140, 2020. doi:10.1162/tacl_a_00304. URL https://aclanth...

    2020 doi

  18. [26]

    U-nets as belief propagation: Efficient classification, denoising, and diffusion in generative hierarchical models

    Mei, S. U-nets as belief propagation: Efficient classification, denoising, and diffusion in generative hierarchical models. arXiv preprint arXiv:2404.18444, 2024

    2024 arXiv

  19. [27]

    J., Liu, Z., Girit, U., and Tegmark, M

    Michaud, E. J., Liu, Z., Girit, U., and Tegmark, M. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=3tbTw2ga8K

    2023

  20. [28]

    Understanding transformers via n-gram statistics

    Nguyen, T. Understanding transformers via n-gram statistics. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=WCc440cUhX

    2024

  21. [29]

    A statistical theory of contrastive pre-training and multimodal generative ai

    Oko, K., Lin, L., Cai, Y., and Mei, S. A statistical theory of contrastive pre-training and multimodal generative ai. arXiv preprint arXiv:2501.04641, 2025

    2025

  22. [30]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. 2023

    2023

  23. [31]

    PyTorch : An Imperative Style , High - Performance Deep Learning Library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch : An Imper...

    2019

  24. [32]

    Pullum, G. K. and Gazdar, G. Natural languages and context-free languages. Linguist. Philos., 4 0 (4): 0 471--504, 1982. doi:10.1007/BF00360802

    1982 doi

  25. [33]

    and Salomaa, A

    Rozenberg, G. and Salomaa, A. Handbook of Formal Languages . Springer, 1997. doi:10.1007/978-3-642-59126-6

    1997 doi

  26. [34]

    I., and Wyart, M

    Sclocchi, A., Favero, A., Levi, N. I., and Wyart, M. Probing the latent hierarchical structure of data via diffusion models. arXiv preprint arXiv:2410.13770, 2024

    2024 arXiv

  27. [35]

    A phase transition in diffusion models reveals the hierarchical nature of data

    Sclocchi, A., Favero, A., and Wyart, M. A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences, 122 0 (1): 0 e2408799121, 2025. doi:10.1073/pnas.2408799121. URL https://www.pnas.org/doi/abs/10.1073/pnas.2...

    2025 doi

  28. [36]

    S., Marzen, S

    Shai, A. S., Marzen, S. E., Teixeira, L., Oldenziel, A. G., and Riechers, P. M. Transformers represent belief state geometry in their residual stream. arXiv preprint arXiv:2405.15943, 2024

    2024 arXiv

  29. [37]

    Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm

    Spigler, S., Geiger, M., and Wyart, M. Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020 0 (12), 2020

    2020

  30. [38]

    and Cotterell, R

    Svete, A. and Cotterell, R. Transformers can represent n -gram language models. arXiv preprint arXiv:2404.14994, 2024

    2024 arXiv

  31. [39]

    Can transformers learn n -gram language models? arXiv preprint arXiv:2410.03001, 2024

    Svete, A., Borenstein, N., Zhou, M., Augenstein, I., and Cotterell, R. Can transformers learn n -gram language models? arXiv preprint arXiv:2410.03001, 2024

    2024 arXiv

  32. [40]

    Tomasini, U. M. and Wyart, M. How deep networks learn sparse and hierarchical data: the sparse random hierarchy model. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=CtEWswTjUd

    2024

  33. [41]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  34. [42]

    and Hu, E

    Yang, G. and Hu, E. J. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020

    2011 arXiv

  35. [43]

    Do transformers parse while predicting the masked word? arXiv preprint arXiv:2303.08117, 2023

    Zhao, H., Panigrahi, A., Ge, R., and Arora, S. Do transformers parse while predicting the masked word? arXiv preprint arXiv:2303.08117, 2023

    2023 arXiv

  36. [44]

    and Mumford, D

    Zhu, S.-C. and Mumford, D. A stochastic grammar of images. Found. Trends Comput. Graph. Vis., 2 0 (4): 0 259--362, 2006. doi:10.1561/0600000017

    2006 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.