Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a tractable family of hierarchical grammars, tree-aligned convolutional networks achieve a loss-scaling exponent twice that of transformers, with weight sharing as the mechanism.

desk verdict A clean, parameter-free prediction that weight sharing doubles the scaling exponent on hierarchical data, let down by an underived bridge assumption and by-eye empirical validation—worth a serious referee. read the letter →

arxiv 2505.07070 v1 pith:Z3GDLMSL submitted 2025-05-11 cs.LG cond-mat.dis-nnstat.ML

Francesco Cagnetta , Alessandro Favero , Antonio Sclocchi , Matthieu Wyart This is my paper · ORCID
classification cs.LGcond-mat.dis-nnstat.ML
keywords neuralscalinglawsRandomHierarchyModelprobabilisticcontext-freegrammarsrepresentationlearningtransformersconvolutionalnetworksweightsharinglast-tokenprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On data from the Random Hierarchy Model—an ensemble of probabilistic context-free grammars on a fixed regular tree—this paper claims that architecture changes the neural scaling exponent. A convolutional network whose filters are aligned with the tree's branching structure reaches a test-loss decay $L(P)-L_\infty \sim P^{\log f / \log m}$, while a transformer on identical data decays as $P^{\log f / (2\log m)}$, a factor of two in the exponent. The mechanism is weight sharing: a convolution reuses a detected correlation across all positions, so it can infer all latent variables at one level as soon as the easiest one is detectable. If the claim is right, scaling laws on compositional data depend on the match between architecture and the data's hierarchical statistics, not only on data and model size.

What carries the argument

The Random Hierarchy Model (RHM)—an ensemble of probabilistic context-free grammars on a fixed regular tree of arity $s$ and depth $L$, with $v$ symbols per level and $m$ equiprobable production rules per symbol, so that $f = m/v^{s-1}$—is the central object. The argument runs on two scales: the variance of token correlations over RHM realisations decays exponentially with the tree distance between tokens, while a finite training set of size $P$ contributes correlation noise of order $1/P$. Assumption V.1 identifies learning a hidden layer with the crossing of those two scales, and the weight-sharing property of convolutional filters replaces the transformer threshold $P_\ell$ with the smaller $\bar P_\ell$. The final exponent then comes from the identity $L_\ell - L_\infty \sim f^\ell$ combined with the relation between $\ell$ and $P$ at each threshold.

What would settle it

On RHM data with $s=2$, $L=4$, $v=24$, $m=6$, the theory places the transformer's second-stage threshold near $P_2 = (1-f)^{-1} v m^5$ and the CNN's near $\bar P_2 = (1-f)^{-1} v m^4$; measuring the actual training-step counts at which the two excess-loss curves cross each $\ell$-gram plateau would confirm or refute the predicted gap, and making the production rules position-dependent should eliminate the CNN advantage if weight sharing is the cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that learning the Random Hierarchy Model for last-token prediction proceeds stagewise, with each stage corresponding to correlations at a deeper level of the generative tree, and that the sample-complexity threshold for each stage is set by when the relevant correlation rises above finite-sample noise. For transformers the threshold is $P_\ell = (1-f)^{-1} v m^{2\ell+1}$, because the correlation between the last token and an observable $s$-tuple at tree distance $2\ell+1$ must be detected. For a CNN with filter size and stride equal to the branching factor $s$, weight sharing turns detection of the single easiest tuple into knowledge of all same-level latent variables, lowering the threshold to $\bar P_\ell = (1-f)^{-1} v m^{\ell+2}$. Since the excess loss after stage $\ell$ is $L_\ell - L_\infty \sim f^\ell$, these thresholds yield $L(P)-L_\infty \sim P^{\log f/(2\log m)}$ for transformers and $P^{\log f/\log m}$ for CNNs. The paper validates both predictions in online-training experiments and reports that locally connected networks, which keep locality but remove weight sharing, follow the transformer's scaling after the first two stages.

Load-bearing premise

The load-bearing premise is that a learner discovers a layer of hidden structure exactly when the correlation between the last token and a group of s tokens at the corresponding tree distance becomes detectable above finite-sample noise, and that a weight-sharing network can then reuse that detection at every position; neither is derived from the gradient-descent dynamics.

Editorial extensions

If this is right

  • On RHM data, to reach a fixed excess loss a transformer needs roughly the square of the number of samples a tree-aligned CNN needs.
  • The loss curve remains stagewise for both architectures, but the stage boundaries for CNNs occur systematically earlier in training.
  • Locality without weight sharing is not enough: locally connected networks inherit the CNN's early stages but revert to transformer scaling at later stages, isolating weight sharing as the cause.
  • Scaling-law exponents on compositional data are architecture-dependent, so comparing scaling exponents across model families requires controlling the match between architecture and data statistics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested but direct extension: if RHM production rules are made position-dependent so that the same latent variable is not reused across branches, the CNN's weight-sharing advantage should shrink or vanish, and the transformer may match or beat it.
  • The mechanism suggests that on real language-like data, any parameter-sharing scheme that respects the grammar's rule reuse—such as tied attention weights or structured state-space models—could inherit part of the CNN's improved exponent, not just convolutions.
  • Because the theory identifies detectability thresholds, it predicts that probing the hidden representations during training should show the CNN becoming invariant to deeper hidden-variable replacements at sample counts of order $v m^{\ell+2}$, a measurement that could be run on the published architecture.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies last-token prediction on synthetic sequences generated by the Random Hierarchy Model (RHM), an ensemble of random context-free grammars with a fixed tree topology. It derives threshold training-set sizes at which correlations between the last token and hidden-variable spans become detectable, giving P_l ~ v m^{2l+1} for transformers and an improved Pbar_l ~ v m^{l+2} for CNNs under a weight-sharing transfer assumption. Combining these thresholds with the previously derived asymptotic relation L_l - L_infinity ~ f^l yields predicted test-loss exponents log f/(2 log m) for transformers and log f/log m for CNNs. Empirically, depth-4 transformers, CNNs, and locally connected networks are trained online on RHM data, and hidden-representation probes measure invariance to resampling hidden symbols and production rules. The central claim is that architecture changes the scaling exponent on hierarchically compositional data, with CNNs enjoying a faster decay because weight sharing lets a detected correlation be reused across positions.

Significance. If the central claim holds, the paper provides one of the few controlled demonstrations that architectural inductive bias changes the exponent of a neural scaling law, not merely the prefactor. The predictions are falsifiable and the exponents are not fit to the empirical curves: they are derived from RHM parameters and compared at independent (v,m) settings. The paper also includes useful controls, including an LCN ablation separating locality from weight sharing, a comparison of Adam versus SGD for transformers, and representation-dynamics probes. The main caveat is that the CNN exponent rests on an explicit, unproven assumption about weight-sharing transfer, so the theoretical result is a heuristic prediction rather than a consequence of training dynamics. The empirical validation is currently visual rather than quantitative, which weakens the claim that the factor-of-two exponent difference is real.

major comments (4)
  1. [V.C.3 and Eq. (24)] The CNN scaling law is not derived. Section V.C.3 asserts that weight sharing lets a learner infer all level-L latents once the easiest one is detectable, leading to Pbar_l in Eq. (16) and the exponent in Eq. (24), but no argument from gradient dynamics is supplied. Assumption V.1 is likewise stated as an assumption, and the conclusions (Section X) explicitly note that a formal characterization of training dynamics is missing. Since Eq. (24) is the central new theoretical result, this is a load-bearing gap. The authors should either derive the transfer mechanism in a tractable limit or test it directly, for example by training linear probes on intermediate activations and measuring decoding accuracy for all level-L latents as a function of P, to check that CNNs acquire them near Pbar_l while transformers acquire them near P_l.
  2. [Section VII, Figs. 4 and 5] The empirical support for the central exponent difference is visual rather than quantitative. Solid learning curves are compared with reference power-law lines by eye; no fitted exponents, confidence intervals, or error bars are reported. In Fig. 4, the claim that the two transformer curves 'display the same decay' is supported only by rescaling one x-axis by a constant factor. Please report fitted slopes of log(L-L_infinity) versus log P with standard errors for each (v,m) setting, together with a statistical comparison of the CNN slope against both Eq. 24 and Eq. 23. This is necessary to establish that the exponent difference is real and not a prefactor or finite-range artifact.
  3. [Section V.C.2, Eqs. (12)-(16)] The detectability threshold is derived by comparing the finite-sample noise variance sigma_P^2 in Eq. (12) with the RHM correlation variance in Eq. (13), but the condition is only 'P >> threshold' and no quantitative scale for the crossover is given. The conversion of these thresholds into a clean power law for the test loss assumes sharp stage transitions, and the transition width could affect the observed exponent, especially over the limited range of training steps in Figs. 3 and 5. Please characterize the finite-P crossover or show that plausible corrections do not change the predicted slope.
  4. [Section IX and Fig. 8] The LCN experiment is the key ablation isolating weight sharing, but it uses a single parameter set (L=4, s=2, v=24, m=6) and is again evaluated by visual overlap after manual shifts. This makes it difficult to rule out that the 'shared asymptotic behavior' of LCN and transformer is specific to this parameter choice. Reporting fitted exponents for LCN at least two (v,m) settings, or providing the same quantitative analysis as requested for Figs. 4-5, would substantially strengthen the weight-sharing interpretation.
minor comments (5)
  1. [Section V.C.1] The subsection heading contains a typo: 'Reconsturction' should be 'Reconstruction'.
  2. [Section III.A] The abbreviation for context-free grammars is introduced as 'CGFs' in the text, but the standard and previously used abbreviation is CFGs; please correct this for consistency.
  3. [Fig. 8 caption] The caption contains a formatting error: 'textbfLeft' should be 'Left' with the appropriate boldface applied in the rendered figure.
  4. [Section III.B, constraint C4] The definition of f is garbled: 'f := (m/vs−1)' is not typeset correctly and the constraint mv <= vs is written without explanation; please clarify the definition and the admissible range of parameters.
  5. [Section V.A, Eq. (8)] In the conditional probability notation, X_{-1} appears both as the target and in the condition via X_{-2:d}; please make the conditioning set explicit (e.g., X_{-1} | X_{-2}=x_{-2}, ..., X_{-d}=x_{-d}) to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling exponents are derived from RHM parameters and validated at fixed power laws, not fitted.

full rationale

The paper's scaling predictions are not obtained by fitting the measured learning curves. Section V derives the transformer threshold P_l from the finite-sample noise (Eq. 12) compared with the RHM correlation variance (Eq. 13), giving P_l ~ m^{2l}; the CNN threshold Pbar_l ~ m^l is introduced explicitly as a conditional consequence of translation invariance in Section V.C.3 ('If this were the case, we could assume...') and cited to prior work [14]. The exponents in Eqs. 23 and 24 follow by inverting these thresholds and substituting the known s^l-gram loss asymptotics L_l - L_inf ~ f^l from [8]. All ingredients are stated as theoretical inputs with fixed RHM parameters; the empirical comparison in Figs. 3 and 5 uses pre-specified power laws at three independent (v,m) parameter sets, so the predictions are not renamed fits. The load-bearing Assumption V.1 is explicitly labeled an assumption, and Section X admits that a formal characterization of training dynamics is missing. An unproven assumption is a correctness risk, not circularity: the paper does not define the exponent through the data, nor does the derivation reduce by construction to its own measured output. Self-citations to [8,10,14] supply prior derivation results with stated RHM assumptions; they are load-bearing but not circular because the cited formulas do not contain the present empirical curves as inputs. Overall, the derivation chain is conditional and heuristic in places, but it is not circular.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

No parameters are fit to the data to obtain the predicted exponents; the exponents depend only on the RHM parameters m and v through f = m/v^(s-1). The main unproven inputs are Assumption V.1 and the weight-sharing transfer hypothesis (Sec. V.C.3), both imported from the authors' earlier framework and neither derived from training dynamics. Several variance and asymptotic formulas are taken from previous papers by the same group, which reduces the external independence of this extension.

assumptions (7)
  • domain assumption RHM ensemble constraints C1-C4 define the data distribution; unambiguity C2 makes each observable span identify a unique hidden variable.
    All scaling predictions are specific to this generative model; the paper does not claim universality beyond it.
  • domain assumption Correlation variances under RHM follow Eqs. 5 and 6, with the 'not same branch' factor (1-f).
    Imported from [8] and [10]; the sample-complexity thresholds P_l are built on these variance formulas.
  • domain assumption Finite-sample fluctuations of empirical correlations are Gaussian with variance σ_P^2 = 1/((vm)vP) (Eq. 12).
    A central-limit-type approximation taken from [10]; it converts correlation strength into a P threshold for detectability.
  • ad hoc to paper Assumption V.1: a learner discovers hidden structure to depth ℓ when the correlations at tree distance 2ℓ+1 become detectable from the training set.
    This is the bridge from correlation statistics to learning-stage transitions; it is not derived from gradient descent dynamics.
  • ad hoc to paper Weight sharing lets a learner with translation invariance infer all level-L latents once the easiest one is detectable, giving Pbar_l = (1-f)^(-1) v m^(l+2) (Sec. V.C.3).
    Underpins the CNN exponent; presented as a hypothesis and validated empirically, not proven.
  • domain assumption Asymptotic s_l-gram loss gap L_l - L∞ ~ f^l (Eqs. 23 and 24).
    Used to convert P thresholds into loss scaling; this asymptotic was derived in [8] for RHM.
  • domain assumption Width 512 realises the infinite-width limit; online SGD/Adam training converges to the correlation-based learning scenario.
    Empirical working assumption; authors verify stability to width and attention-head counts but do not prove convergence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures." pith.science (2026). https://pith.science/paper/Z3GDLMSL

@misc{pith2026250507070,
  author       = {Pith},
  title        = {Pith review of: Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3GDLMSL}},
  note         = {Machine review of arXiv:2505.07070}
}
read the original abstract

How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets generated by the Random Hierarchy Model (RHM) -- an ensemble of probabilistic context-free grammars designed to capture the hierarchical structure of natural language while remaining analytically tractable. Previously, we developed a theory of representation learning based on data correlations that explains how deep learning models capture the hierarchical structure of the data sequentially, one layer at a time. Here, we extend our theoretical framework to account for architectural differences. In particular, we predict and empirically validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance compared to transformer models, which rely on global self-attention mechanisms. This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.

Figures

Figures reproduced from arXiv: 2505.07070 by the authors.

Figure 1
Figure 1. FIG. 1. Comparison of structural hierarchies in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Unifying Learning Dynamics and Generalization in Transformers Scaling Law

    cs.LG 2025-12 reject novelty 4.0 of 10

    Claims a two-stage transformer scaling law (exponential then C^{-1/6}) with matching bounds, but the lower bounds are missing, the exponent is inconsistent (-1/7 vs -1/6), and the law is an artifact of hand-set M = Θ(...

Reference graph

Works this paper leans on

67 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    These correlations are given by the fol- lowing (vm)×v C(X−t,X−1)µ,ν :=P{X−t =µ,X−1 =ν} − P{X−t =µ} P{X−1 =ν}

    Reconsturction of depth- 1 hidden variables For concreteness, let us consider the correlations be- tween the last token X−1 and one of the entire s-tuples in the context, X−t := ( X−ts,...,X −(t−1)s+1)) with t = 2,...,s L−1. These correlations are given by the fol- lowing (vm)×v C(X−t,X−1)µ,ν :=P{X−t =µ,X−1 =ν} − P{X−t =µ} P{X−1 =ν}. (11) According to sec...

    work page

  2. [2]

    These tuples have a smaller tree dis- tance from X−1 than tuples of input tokens, thus the corresponding sample complexities are lower than those required by Eq

    Reconstruction of deeper hidden variables The same ideas apply for the inference of the next layer of hidden variables, i.e., depth-2 or level-( L− 1) variables, via correlations between X−1 and s-tuples of level-L variables. These tuples have a smaller tree dis- tance from X−1 than tuples of input tokens, thus the corresponding sample complexities are lo...

    work page

  3. [3]

    Therefore, a learner could infer all level- L latents 7 as soon as it’s able to infer µ (L) −2, which is the easiest to infer as it has the strongest correlation with X−1

    Improved sample complexities due to translation invariance In the RHM, the same set of production rules applies to all variables at a given level, independently of the po- sition. Therefore, a learner could infer all level- L latents 7 as soon as it’s able to infer µ (L) −2, which is the easiest to infer as it has the strongest correlation with X−1. If th...

    work page

  4. [4]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. , Training compute-optimal large language models, arXiv preprint arXiv:2203.15556 (2022)

    arXiv 2022

  5. [5]

    Hestness, S

    J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou, Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.00409 (2017)

    arXiv 2017

  6. [6]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020)

    arXiv 2020

  7. [7]

    Henighan, J

    T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. , Scaling laws for autoregressive generative model- ing, arXiv preprint arXiv:2010.14701 (2020)

    arXiv 2020

  8. [8]

    introduced the RHM as a framework to show how correlations emerge from the generative model and can be used by deep networks to represent the latent hierarchical structure. In self-supervised settings, several works [8, 11, 14] have shown that neural networks pro- gressively learn longer-range correlations, corresponding to deeper layers of the data hiera...

    work page

Show all 67 references
  1. [9]

    M. E. Fisher, The theory of equilibrium critical phenom- ena, Reports on progress in physics 30, 615 (1967)

    1967

  2. [10]

    Malach and S

    E. Malach and S. Shalev-Shwartz, A provably correct al- gorithm for deep learning that actually works, Preprint at http://arxiv.org/abs/1803.09522 (2018)

    2018 arXiv

  3. [11]

    Malach and S

    E. Malach and S. Shalev-Shwartz, The implications of local correlation on learning some deep functions, in Ad- vances in Neural Information Processing Systems, Vol. 33 (2020) pp. 1322–1332

    2020

  4. [12]

    Cagnetta, L

    F. Cagnetta, L. Petrini, U. M. Tomasini, A. Favero, and M. Wyart, How deep neural networks learn compositional data: The random hierarchy model, Phys. Rev. X 14, 031001 (2024)

    2024

  5. [13]

    U. M. Tomasini and M. Wyart, How deep networks learn sparse and hierarchical data: the sparse random hier- archy model, in Forty-first International Conference on Machine Learning (2024)

    2024

  6. [14]

    Cagnetta and M

    F. Cagnetta and M. Wyart, Towards a theory of how the structure of language is acquired by deep neural net- works, in The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems (2024)

    2024

  7. [15]

    Garnier-Brun, M

    J. Garnier-Brun, M. M´ ezard, E. Moscato, and L. Saglietti, How transformers learn structured data: insights from hierarchical filtering, arXiv preprint arXiv:2408.15138 (2024)

    2024 arXiv

  8. [16]

    Sclocchi, A

    A. Sclocchi, A. Favero, and M. Wyart, A phase transition in diffusion models reveals the hierarchical nature of data, Proceedings of the National Academy of Sciences 122, e2408799121 (2025)

    2025

  9. [17]

    Sclocchi, A

    A. Sclocchi, A. Favero, N. I. Levi, and M. Wyart, Prob- ing the latent hierarchical structure of data via diffusion models, in The Thirteenth International Conference on Learning Representations (2025)

    2025

  10. [18]

    Favero, A

    A. Favero, A. Sclocchi, F. Cagnetta, P. Frossard, and M. Wyart, How compositional generalization and cre- ativity improve as diffusion models are trained, arXiv preprint arXiv:2502.12089 (2025)

    2025 arXiv

  11. [19]

    Chomsky, Three models for the description of lan- guage, IRE Transactions on information theory 2, 113 (1956)

    N. Chomsky, Three models for the description of lan- guage, IRE Transactions on information theory 2, 113 (1956)

    1956

  12. [20]

    Hutter, Learning curve theory (2021), 13 arXiv:2102.04074 [cs.LG]

    M. Hutter, Learning curve theory (2021), 13 arXiv:2102.04074 [cs.LG]

    2021 arXiv

  13. [21]

    E. J. Michaud, Z. Liu, U. Girit, and M. Tegmark, The quantization model of neural scaling, in Thirty-seventh Conference on Neural Information Processing Systems (2023)

    2023

  14. [22]

    Caponnetto and E

    A. Caponnetto and E. De Vito, Optimal rates for the reg- ularized least-squares algorithm, Foundations of Compu- tational Mathematics 7, 331 (2007)

    2007

  15. [23]

    Spigler, M

    S. Spigler, M. Geiger, and M. Wyart, Asymptotic learn- ing curves of kernel methods: empirical data versus teacher–student paradigm, Journal of Statistical Me- chanics: Theory and Experiment 2020 (2020)

    2020

  16. [24]

    Bordelon, A

    B. Bordelon, A. Canatar, and C. Pehlevan, Spectrum dependent learning curves in kernel regression and wide neural networks, in International Conference on Machine Learning (PMLR, 2020) pp. 1024–1034

    2020

  17. [25]

    Bahri, E

    Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, Explaining neural scaling laws, arXiv preprint arXiv:2102.06701 (2021)

    2021 arXiv

  18. [26]

    Favero, F

    A. Favero, F. Cagnetta, and M. Wyart, Locality de- feats the curse of dimensionality in convolutional teacher- student scenarios, in Advances in Neural Informa- tion Processing Systems , edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (2021)

    2021

  19. [27]

    H. Cui, B. Loureiro, F. Krzakala, and L. Zdeborov´ a, Gen- eralization error rates in kernel regression: The crossover from the noiseless to noisy regime, Advances in Neural Information Processing Systems 34, 10131 (2021)

    2021

  20. [28]

    Maloney, D

    A. Maloney, D. A. Roberts, and J. Sully, A solvable model of neural scaling laws, arXiv:2210.16859 (2022)

    2022 arXiv

  21. [29]

    Cagnetta, A

    F. Cagnetta, A. Favero, and M. Wyart, What can be learnt with wide convolutional neural networks?, inInter- national Conference on Machine Learning (PMLR, 2023) pp. 3347–3379

    2023

  22. [30]

    Bordelon, A

    B. Bordelon, A. Atanasov, and C. Pehlevan, A dynamical model of neural scaling laws, in Forty-first International Conference on Machine Learning (2024)

    2024

  23. [31]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler, Neural tangent ker- nel: Convergence and generalization in neural networks, Advances in neural information processing systems 31 (2018)

    2018

  24. [32]

    Chizat, E

    L. Chizat, E. Oyallon, and F. Bach, On lazy training in differentiable programming, in Advances in Neural Infor- mation Processing Systems (2019) pp. 2937–2947

    2019

  25. [33]

    Paccolat, L

    J. Paccolat, L. Petrini, M. Geiger, K. Tyloo, and M. Wyart, Geometric compression of invariant mani- folds in neural networks, Journal of Statistical Mechanics: Theory and Experiment 2021, 044001 (2021)

    2021

  26. [34]

    J. Ba, M. A. Erdogdu, T. Suzuki, Z. Wang, D. Wu, and G. Yang, High-dimensional asymptotics of feature learn- ing: How one gradient step improves the representation, Advances in Neural Information Processing Systems 35, 37932 (2022)

    2022

  27. [35]

    Bietti, J

    A. Bietti, J. Bruna, C. Sanford, and M. J. Song, Learning single-index models with shallow neural networks, Ad- vances in neural information processing systems 35, 9768 (2022)

    2022

  28. [36]

    Dandi, F

    Y. Dandi, F. Krzakala, B. Loureiro, L. Pesce, and L. Stephan, How two-layer neural networks learn, one (giant) step at a time, arXiv preprint arXiv:2305.18270 (2023)

    2023 arXiv

  29. [37]

    Bordelon, A

    B. Bordelon, A. Atanasov, and C. Pehlevan, How feature learning can improve neural scaling laws, in The Thir- teenth International Conference on Learning Represen- tations (2025)

    2025

  30. [38]

    Mossel, Deep learning and hierarchal generative mod- els, arXiv preprint arXiv:1612.09057 (2016)

    E. Mossel, Deep learning and hierarchal generative mod- els, arXiv preprint arXiv:1612.09057 (2016)

    2016 arXiv

  31. [39]

    Mei, U-nets as belief propagation: Efficient classifica- tion, denoising, and diffusion in generative hierarchical models, arXiv preprint arXiv:2404.18444 (2024)

    S. Mei, U-nets as belief propagation: Efficient classifica- tion, denoising, and diffusion in generative hierarchical models, arXiv preprint arXiv:2404.18444 (2024)

    2024 arXiv

  32. [40]

    Refinetti, A

    M. Refinetti, A. Ingrosso, and S. Goldt, Neural networks trained with sgd learn distributions of increasing com- plexity, in International Conference on Machine Learn- ing (PMLR, 2023) pp. 28843–28863

    2023

  33. [41]

    Bardone and S

    L. Bardone and S. Goldt, Sliding down the stairs: how correlated latent variables accelerate learning with neural networks, arXiv preprint arXiv:2404.08602 (2024)

    2024 arXiv

  34. [42]

    Rende, F

    R. Rende, F. Gerace, A. Laio, and S. Goldt, A distribu- tional simplicity bias in the learning dynamics of trans- formers, arXiv preprint arXiv:2410.19637 (2024)

    2024

  35. [43]

    Svete and R

    A. Svete and R. Cotterell, Transformers can represent n- gram language models, arXiv preprint arXiv:2404.14994 (2024)

    2024 arXiv

  36. [44]

    Nguyen, Understanding transformers via n-gram statistics, in The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    T. Nguyen, Understanding transformers via n-gram statistics, in The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    2024

  37. [45]

    Svete, N

    A. Svete, N. Borenstein, M. Zhou, I. Augenstein, and R. Cotterell, Can transformers learn n-gram language models?, arXiv preprint arXiv:2410.03001 (2024)

    2024 arXiv

  38. [46]

    Borenstein, A

    N. Borenstein, A. Svete, R. Chan, J. Valvoda, F. Nowak, I. Augenstein, E. Chodroff, and R. Cotterell, What lan- guages are easy to language-model? a perspective from learning probabilistic regular languages, arXiv preprint arXiv:2406.04289 (2024)

    2024 arXiv

  39. [47]

    A. S. Shai, S. E. Marzen, L. Teixeira, A. G. Olden- ziel, and P. M. Riechers, Transformers represent belief state geometry in their residual stream, arXiv preprint arXiv:2405.15943 (2024)

    2024 arXiv

  40. [48]

    Allen-Zhu and Y

    Z. Allen-Zhu and Y. Li, Physics of language models: Part 1, learning hierarchical language structures, arXiv preprints, abs/2305.13673, May (2023)

    2023 arXiv

  41. [49]

    H. Zhao, A. Panigrahi, R. Ge, and S. Arora, Do trans- formers parse while predicting the masked word?, arXiv preprint arXiv:2303.08117 (2023)

    2023 arXiv

  42. [50]

    R. T. McCoy, R. Frank, and T. Linzen, Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks, Transactions of the As- sociation for Computational Linguistics 8, 125 (2020)

    2020

  43. [51]

    Ahuja, V

    K. Ahuja, V. Balachandran, M. Panwar, T. He, N. A. Smith, N. Goyal, and Y. Tsvetkov, Learning syntax without planting trees: Understanding when and why transformers generalize hierarchically, arXiv preprint arXiv:2404.16367 (2024)

    2024 arXiv

  44. [52]

    Rozenberg and A

    G. Rozenberg and A. Salomaa, Handbook of Formal Lan- guages (Springer, 1997)

    1997

  45. [53]

    G. K. Pullum and G. Gazdar, Natural languages and context-free languages, Linguist. Philos. 4, 471 (1982)

    1982

  46. [54]

    A. K. Joshi, Tree adjoining grammars: How much context-sensitivity is required to provide reasonable structural descriptions?, in Natural Language Parsing: Psychological, Computational, and Theoretical Perspec- tives, edited by D. R. Dowty, L. Karttunen, and A. M. Zwicky (Camb...

    1985

  47. [55]

    C. D. Manning and H. Sch¨ utze,Foundations of Statisti- cal Natural Language Processing(MIT Press, Cambridge, MA, 1999)

    1999

  48. [56]

    Zhu and D

    S.-C. Zhu and D. Mumford, A stochastic grammar of im- 14 ages, Found. Trends Comput. Graph. Vis. 2, 259 (2006)

    2006

  49. [57]

    Ebeling and T

    W. Ebeling and T. P¨ oschel, Entropy and long-range cor- relations in literary english, Europhys. Lett. 26, 241 (1994)

    1994

  50. [58]

    H. W. Lin and M. Tegmark, Critical behavior in physics and probabilistic formal languages, Entropy 19, 10.3390/e19070299 (2017)

    2017 doi

  51. [59]

    Mezard and A

    M. Mezard and A. Montanari, Information, physics, and computation (Oxford University Press, 2009)

    2009

  52. [60]

    D. J. Hsu, S. M. Kakade, and P. Liang, Identifiability and unmixing of latent parse trees, in Adv. Neural Inf. Process. Syst. (NeurIPS) , edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Curran Asso- ciates, Inc., 2012) pp. 37–45

    2012

  53. [61]

    DeGiuli, Random language model, Phys

    E. DeGiuli, Random language model, Phys. Rev. Lett. 122, 128301 (2019)

    2019

  54. [62]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, in North American Chapter of the Association for Computational Linguistics (2019)

    2019

  55. [63]

    Radford, K

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding with un- supervised learning, Technical report, OpenAI (2018)

    2018

  56. [64]

    LeCun, B

    Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, Backpropaga- tion applied to handwritten zip code recognition, Neural computation 1, 541 (1989)

    1989

  57. [65]

    Kalchbrenner, E

    N. Kalchbrenner, E. Grefenstette, and P. Blunsom, A convolutional neural network for modelling sentences, arXiv preprint arXiv:1404.2188 (2014)

    2014 arXiv

  58. [66]

    Y. Kim, Convolutional neural networks for sentence clas- sification proceedings of the 2014 conference on empirical methods in natural language processing, emnlp 2014, oc- tober 25-29, 2014, doha, qatar, a meeting of sigdat, a special interest group of the acl, Association for...

    2014

  59. [67]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, At- tention is all you need, Advances in neural information processing systems 30 (2017)

    2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.