Pith. sign in

REVIEW 5 major objections 5 minor 59 references

For multi-layer transformers trained by SGD, excess risk over the irreducible error follows a two-stage law in compute: exponential decay until C≈N^6, then a C^{-1/6} power law, with matching lower bounds asserted.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:55 UTC pith:HFUU5SW5

load-bearing objection Not ready for prime time: the headline scaling law rests on an elementary entropy error and arbitrary calibration choices. the 5 major comments →

arxiv 2512.22088 v3 pith:HFUU5SW5 submitted 2025-12-26 cs.LG cs.AIcs.CL

Unifying Learning Dynamics and Generalization in Transformers Scaling Law

Chiwun Yang This is my paper
classification cs.LG cs.AIcs.CL MSC 68T0768Q32
keywords scaling lawstransformersneural tangent kernellazy learningphase transitionexcess riskstochastic gradient descentgeneralization bounds
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that for multi-layer transformer language models trained with SGD on sequence-to-sequence data, the excess risk over the irreducible error is governed by total compute C through a two-stage law. Below a threshold C ≈ N^6 log(NLd/ξ^2)/ξ^2, excess risk decays exponentially, as exp(−αξ^2C/N^6); above that threshold, it decays as a C^{−1/6} power law. The transition separates a compute-starved, optimization-dominated phase from a data-limited, statistically-dominated phase. The paper asserts these rates are tight up to constants and log factors, certified by complementary lower bounds — a two-point statistical reduction and a first-order oracle argument. If the proof stands, empirical scaling curves for LLM training acquire a rigorous foundation that says explicitly when extra compute stops paying and data quality becomes the binding constraint.

Core claim

The central claim is Theorem 6.1. Under the paper's Good-Properties calibration — a positive-definite per-layer kernel at initialization, M=Θ(N^3), and ε=Θ(ξ/N) — with probability at least 1−δ the worst-case excess risk over the data distribution is bounded by poly(L,d,1/λ,1/ω)·exp(−αξ^2C/N^6)·L(0,D) when C≤N^6 log(NLd/ξ^2)/ξ^2, and by ξ^{5/3}·O((C/W(C/ξ^{10}))^{−1/6}) once C crosses that threshold, where W is the Lambert W function. The paper reads the first stage as optimization-dominated and the second as statistical: once compute passes the threshold, only the target function's complexity and dataset size N control the floor, so further training or capacity yields diminishing returns. It

What carries the argument

The engine is a layer-wise neural tangent kernel for decoder-only transformers. The gradient-flow identity E[dL/dt] = −Σ_ν vec(∂L/∂μ^(ν))^T (H^(ν)⊗I_d) vec(∂L/∂μ^(ν)) writes the loss dynamics as a sum of quadratic forms in the per-layer kernel H^(ν)(t)=⟨β^(ν),β^(ν)⟩+⟨γ^(ν),γ^(ν)⟩, which converts the transformer's matrix computation into parallel vector computation. Combined with Lemma D.1 Part 15's proportionality ∥∂L/∂μ^(ν)∥²_F ≍ ε²·L(t,D), this yields the exponential training convergence that feeds Stage I. The power-law tail comes from decomposing excess risk into optimization risk plus approximation risk plus estimation risk, bounding the latter two through RKHS approximation and coverin

Load-bearing premise

The proof fixes M=Θ(N³) and ε=ξ/N by hand in Theorem G.7; the phase boundary and the C^{−1/6} tail are algebraic consequences of these choices, and neither the transformer architecture nor the data or optimization setup forces them.

What would settle it

Rerun the derivation of Theorem G.7 under the alternative legal calibration M=Θ(N²); if the transition threshold and the tail exponent change, the claimed universal law is an artifact of the calibration choice. Also, the paper's opening abstract page states a Θ(C^{−1/7}) statistical decay while Theorem 6.1 states Θ(C^{−1/6}); at most one of these can be the proven rate, so checking which one the appendices actually establish settles the advertised exponent.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If Theorem 6.1 is right, compute-optimal LLM training has a stop rule: spend compute up to C≈N^6 log(NLd/ξ^2)/ξ^2, where returns are exponential; beyond that point marginal gains are only C^{−1/6}, so the next investment should be more or cleaner data.
  • The single-variable laws follow directly: with M=Ω(N³), excess risk decays as exp(−O(ξ²T/N²)) in training time, as O(ξ²/N) in dataset size, and as ξ²M^{−ζ} in model size before saturation.
  • Scaling collapses when data noise grows with dataset size: if ξ(N) grows like C^{1/10} or N^{1/2}, the excess-risk bound degrades to O(1), explaining why adding low-quality data can make the data-scaling law disappear.
  • The Lambert W factor in the statistical phase means the asymptotic decay is slightly slower than a pure C^{−1/6} power law at large compute, because W(C/ξ^{10}) grows logarithmically.
  • The kernel-perturbation and lazy-learning analysis implies that within the covered regime the transformer's kernel stays near initialization, so training convergence and the phase transition are driven by the fixed NTK rather than by feature learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the phase boundary C∝N^6 and the exponent −1/6 are not forced by the architecture; they arise from the calibration M=Θ(N³), ε=ξ/N assumed in the proof of Theorem G.7. If a different legal calibration (say M=Θ(N²)) is used, the same algebra yields a different claimed law, so the 'unified' result is best read as one parameterization until the calibration is derived.
  • Editorial inference: the abstract asserts matching lower bounds, but the supplied proof text contains only upper-bound theorems; a reader who wants tightness should locate the two-point-reduction and first-order-oracle lower-bound theorems before treating the rates as optimal.
  • Editorial inference: the lazy-learning assumption generates a testable signature — during the compute-starved phase, trained transformers should exhibit small weight displacement and near-frozen features; measuring feature drift or kernel alignment early in training could check whether the regime applies to real LLMs.
  • Editorial inference: the data-limited floor O(ξ²/N) suggests that in noisy-data settings the optimal allocation is to stop adding compute at the threshold and reduce ξ (by filtering or denoising) rather than to train longer, since the floor is highly sensitive to ξ.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes an NTK/lazy-training analysis of decoder-only transformers, deriving an ODE for SGD training dynamics and bounding the excess risk. It claims a two-phase scaling law: exponential decay in total compute C until a threshold C ≈ N^6 log(NLd/ξ^2)/ξ^2, followed by a power-law decay C^{-1/6} (or, in the abstract quoted by the reader's report, Θ(C^{-1/7})), with matching lower bounds certified by a two-point reduction and a first-order oracle argument. It also derives single-variable scaling laws and reports GPT-2/TinyStories experiments. As written, however, the body contains only upper-bound theorems; the advertised lower-bound arguments do not appear. The headline power-law exponent and phase boundary are consequences of hand-imposed choices M=Θ(N^3) and ε=ξ/N in the proof of Theorem G.7, and the metric-entropy estimate in Lemma G.5 Part 2 appears to be mathematically incorrect. The empirical section does not quantitatively test the claimed C^{-1/6} law.

Significance. A rigorous two-phase compute-scaling law for transformers would be a substantial contribution, and the paper makes a genuine effort at layerwise gradient computation, kernel-perturbation control, and noise-scaling experiments (Table 1). However, the central claim is not supported as stated. The metric-entropy issue breaks the Stage-II power law; the M=Θ(N^3), ε=ξ/N calibration choices make the exponent a bookkeeping artifact; and the promised lower-bound certification is absent. The convergence-rate constant is also tracked inconsistently between Lemma E.1, Theorem F.2, and the informal Theorem 5.5. These are load-bearing problems, not presentation issues. The contribution, in its present form, does not establish the claimed scaling law.

major comments (5)
  1. [Abstract (posted vs. full text)] The abstract quoted in the reader's report promises 'matching upper and lower bounds' and a Θ(C^{-1/7}) rate, certified by a two-point reduction and a first-order oracle argument. The body's abstract and Theorem 6.1/Theorem G.7 state only an upper bound and use Θ(C^{-1/6}). No two-point reduction, oracle lower bound, or lower-bound theorem appears anywhere in the main text or appendices. The claimed tightness of the two-stage law is therefore unsubstantiated; as written the paper establishes at most an upper-bound statement, assuming the remaining arguments are correct. This mismatch is load-bearing because 'matching bounds' is a principal advertised contribution.
  2. [Lemma G.5 Part 2 / Lemma G.3] Lemma G.5 Part 2 claims V(F_{M,N},d)(ε) ≤ O(1). Its proof says 'We need at most O(1)^M numbers to cover this set. Then we have the logarithm of the covering number: V(...)(ζ) ≤ O(1).' This is incorrect: the logarithm of O(1)^M is Θ(M), not O(1). Under the proof's own calibration M=Θ(N^3), the entropy is Θ(N^3). Substituting B_F=O(ε), ε=Θ(ξ/N), and V=Θ(N^3) into Lemma G.3 gives an estimation term of order Θ(ξ^2 N^2), not O(ξ^2/N). The Stage-II bound in Theorem G.7, which is built on this O(ξ^2/N) term, is therefore not established. This error is independent of the additional calibration choices M=Θ(N^3), ε=ξ/N.
  3. [Theorem G.7 proof] The proof states: 'we follow Definition 5.2 to require M = Θ(N^3). Thus, we choose ε = ξ/N.' Definition 5.2 specifies ω, κ, and a lower bound on m; it does not state M=Θ(N^3), and no architectural or data-dependent argument fixes this scaling. The phase boundary C = N^6 log(NLd/ξ^2)/ξ^2 and the exponent -1/6 are algebraic consequences of these choices via T=C/(MN) and the bound O(ξ^2/N). A different legitimate choice, e.g., M=Θ(N^2), changes the boundary and the exponent. Without a theorem or empirical criterion determining M(N) and ε(N), the 'general scaling law' reduces to the proof's calibration choices rather than to a property of transformer training.
  4. [Lemma D.1 Part 15] The exponential Stage-I rate in Theorem 5.5 rests on the proportionality ∥dL/dµ^{(ν)}(t)∥²_F ≍ ε² L(t,D) asserted in Lemma D.1 Part 15. The proof first bounds the gradient by O(√d) and then states the asymptotic proportionality with 'Similarly' and 'By induction'; no derivation of the lower bound or of the exact ε² scaling is given. Because the claimed exponential rate αε²T in Theorem 5.5 depends on this step, the convergence theorem is not fully proven as stated.
  5. [Lemma E.1 / Theorem F.2] The exponential convergence rate is tracked inconsistently. Lemma E.1's proof solves E[L(T,D)] ≤ exp(−C ε² ωλ N T) L(0,D) (Eq. (7)), while the 'Tighter Bound' equation (9) gives E[dL(t,D)/dt] ≤ −ε² C1(1−C2 n/m^{1/2}) L(t,D), dropping the factors N and ω. Formal Theorem F.2 states L(T,D) ≤ exp(−α ε² T) with α=Θ(λ), while informal Theorem 5.5 defines α_cr=O(λ/ω). These differ by factors of N and ω. Since Theorem G.7 converts the exponential into exp(−α ξ² C/N^6) using C=O(MTN), the rate constant must be tracked consistently; as written it is not.
minor comments (5)
  1. [Definition 5.2 / Definition 5.4] The symbol N is overloaded: in Definition 5.2 it is used in ω and m expressions without being defined in that item (the dataset size is fixed as n), while Definition 5.4 redefines N := n as the dataset size. This makes the calibration conditions hard to audit.
  2. [Theorem 6.1] In the Stage I bound 'C exp(−α ξ² C/N^6)', the first C is an unspecified constant and the second C is total compute. Using the same symbol for both in one formula is confusing; a different letter for the constant would help.
  3. [Section 7 / Figure 2] The empirical validation is qualitative: the right panel labels trajectories as 'exponent-like' and 'linear-like', but no fit to the predicted C^{-1/6} slope or to the predicted phase boundary is reported. The experiments therefore do not discriminate the proposed law from other scaling forms.
  4. [Appendix figures] Figures 3-6 contain raw unicode escape sequences (e.g., '/uni00000014/...') in captions and axis labels, indicating a compilation/rendering issue.
  5. [Lemma C.4] The proof of the gradient computation lemma is omitted with the note that it 'trivially follows from basic algebra.' Since Lemma C.4 is the basis of the subsequent learning-dynamics and kernel arguments, the full derivation should be included rather than left to the reader.

Circularity Check

1 steps flagged

The two-stage C^{-1/6} scaling law is bookkeeping of the proof's own M=Θ(N^3), ε=ξ/N calibration; Lemma G.5's entropy estimate and the advertised lower bounds are additionally unsupported, though not themselves circular.

specific steps
  1. self definitional [Appendix G.2, proof of Theorem G.7 (formal version of Theorem 6.1), Condition 2]
    "Here, for total computational resource C with any number of dataset size N, we follow Definition 5.2 to require M = Θ(N 3). Thus, we choose ε = ξ/N. ... We choose a considerable value of N to meet C> N6 log(N·Ld/ξ 2) ξ2 , the maximum solution is: N=O((Cξ2/W(C/ξ10))^{1/6}). Finally, we get: ... ξ^{5/3}·O((C/W(C/ξ^{10}))^{-1/6})"

    The claimed phase boundary C=Θ(N^6 log(...)/ξ^2) is manufactured by inserting M=Θ(N^3) into C=O(MTN), and the −1/6 exponent is then the algebraic solution N=O((Cξ^2/W(C/ξ^{10}))^{1/6}) of that imposed boundary combined with the statistical term O(ξ^2/N) from Theorem G.6. Neither M=Θ(N^3) nor ε=ξ/N is required by Definition 5.2 (which only lower-bounds width) or by any transformer/data property. A different legitimate calibration, e.g. M=Θ(N^2), yields a different threshold and a different exponent. Thus the two-stage law is equivalent to the proof's own parametrization, not derived from the model.

full rationale

The central circularity is located in the proof of Theorem G.7: the paper states 'we follow Definition 5.2 to require M = Θ(N^3)' and 'we choose ε = ξ/N', after which the threshold C > N^6 log(...)/ξ^2 and the power-law exponent −1/6 follow by direct substitution. Definition 5.2 contains no such M=Θ(N^3) requirement, so this is a hand-imposed calibration presented as a theorem consequence. The Stage-II bound ΔR ≤ O(ξ^2/N), combined with the imposed N = (Cξ^2/W(C/ξ^{10}))^{1/6}, yields exactly the advertised ξ^{5/3}(C/W)^{−1/6} law; this is bookkeeping of the chosen M–N relation rather than a prediction from transformer structure. Stage I's exp(−αξ^2 C/N^6) rate depends on the same M=Θ(N^3) substitution and on the asserted proportionality in Lemma D.1 Part 15, which is itself stated without proof. I also flag two non-circular but load-bearing gaps: Lemma G.5 Part 2 asserts V(F_{M,N,d})(ε)≤O(1) from an O(1)^M cover, although log(O(1)^M)=Θ(M), so the O(ξ^2/N) term that becomes C^{−1/6} is not established; and the abstract promises matching lower bounds via two-point reduction and first-order oracle, but no such lower-bound theorems appear in the text. No load-bearing self-citation chain was found: the proof relies on external results [SH17, Hay19, ADH+19] rather than the author's own prior work. Overall, the central claimed law reduces by construction to the paper's calibration choices, warranting a partial circularity score of 6.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The two-stage scaling law rests entirely on the hand-set calibrations and NTK-regime assumptions listed above; no free constant is fit to empirical data, but the headline exponents are produced by the calibration rather than by data, architecture, or independent theory. No new physical or mathematical entity is introduced.

free parameters (4)
  • Model-size scaling calibration = M = Θ(N³)
    Imposed in the proof of Theorem G.7 ('we follow Definition 5.2 to require M = Θ(N³)'); determines the transition C ∝ N^6 and the Stage-II exponent -1/6 via N = (Cξ²/W(C/ξ^{10}))^{1/6}.
  • Output-scaling coefficient ε (the 'grokking coefficient') = ε = Θ(ξ/N)
    Chosen by hand in Theorem G.7 ('we choose ε = ξ/N'); sets the exponential-stage rate exp(-αξ²C/N^6) and the crossover location.
  • Residual-branch scale ω = ω = Θ(1/(N L² d^{2.5} B³))
    Definition 5.2 'Good Properties'; tuned to make the kernel-perturbation bounds (Lemma D.1) hold, forcing the tiny-output lazy regime.
  • Attention scale κ = κ = 1/√m
    Definition 5.2; chosen so the U-kernel contribution ∥γ∥ vanishes, reducing attention to a frozen linear feature map.
axioms (5)
  • domain assumption Kernel positive-definiteness at initialization (Assumption 5.1): λ_min(H'(0)) > 0 for every layer
    Standard NTK non-degeneracy (Du et al. Assumption 3.1); combined with the width requirement it confines the whole analysis to the lazy-learning regime, which the paper assumes ('we assume the learning behavior of LLMs is constrained to the kernel regimes', §1).
  • domain assumption Weights stay in a perturbation ball of radius R and the kernel stays PD through training (lazy learning)
    Section 5.1 and Lemma E.1; the radius bound R is verified only inside the proof using the width/scale choices of Definition 5.2, so it is an assumption about the training trajectory, not about data.
  • ad hoc to paper Gradient-to-loss proportionality ∥dL/dµ^{(ν)}∥²_F ≍ ε²·L(t,D) (Lemma D.1 Part 15)
    Powers the exponential convergence (Theorem 5.5); asserted ('Similarly, we have...') with no derivation; hidden dataset/sequence factors in the constant would alter the compute-accounting T = C/(MN).
  • ad hoc to paper Log-covering-entropy of F_{M,N} is O(1) (Lemma G.5 Part 2)
    Needed for the O(ξ²/N) estimation term in Theorem G.6; as written the covering number is O(1)^M for an M-parameter class, whose log is O(M), so the stated bound silently drops an M factor.
  • domain assumption Data model: ∥X_ℓ∥₂ = Θ(1) (RMSNorm), centered noise with variance bound ξ², targets in [C_lower, C_upper]^{L×d}
    Section 3.2; defines the input normalization and the noise level ξ that enters every bound and the noise-scaling claims of §6.2.

pith-pipeline@v1.3.0-alltime-deepseek · 49156 in / 28835 out tokens · 268617 ms · 2026-08-03T13:55:38.266364+00:00 · methodology

0 comments
read the original abstract

The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources. Yet, while empirically validated, its theoretical underpinnings remain poorly understood. This work formalizes the learning dynamics of transformer-based language models as an ordinary differential equation (ODE) system, then approximates this process to kernel behaviors. Departing from prior toy-model analyses, we rigorously analyze stochastic gradient descent (SGD) training for multi-layer transformers on sequence-to-sequence data with arbitrary data distribution, closely mirroring real-world conditions. Our analysis characterizes the convergence of generalization error to the irreducible risk as computational resources scale with data, especially during the optimization process. We establish matching upper and lower bounds on the excess risk, characterized by a distinct phase transition. In the initial optimization phase, the excess risk decays exponentially relative to the computational cost ${\sf C}$. However, once a specific resource allocation threshold is crossed, the system enters a statistical phase, where the generalization error follows a power-law decay of $\Theta(\mathsf{C}^{-1/7})$. These rates are certified by complementary lower bounds -- statistical, via an information-theoretic two-point reduction, and optimization-side, via a first-order oracle argument -- rendering the two-stage law tight up to constants, logarithmic factors, and a condition-number gap. Beyond this unified framework, our theory derives isolated scaling laws for model size, training time, and dataset size, elucidating how each variable independently governs the bounds of generalization.

Figures

Figures reproduced from arXiv: 2512.22088 by Chiwun Yang.

Figure 1
Figure 1. Figure 1: Visualization of independent variables scaling law (a.k.a single law) utilizing GPT-2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scaling law curves illustrating validation of Cross-Entropy Loss across varying compute [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training loss dynamics varied by the dataset size. [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training loss dynamics varied by the model size. [PITH_FULL_IMAGE:figures/full_fig_p026_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Validation loss dynamics varied by the dataset size. [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Validation loss dynamics varied by the model size. [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 53 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    [AAA+23] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anad- kat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    The fine-grained complexity of gradient computation for training large language models.arXiv preprint arXiv:2402.04497,

    [AS24a] Josh Alman and Zhao Song. The fine-grained complexity of gradient computation for training large language models.arXiv preprint arXiv:2402.04497,

    Pith/arXiv arXiv

  3. [7]

    Physics of language models: Part 3.1, knowledge storage and extraction.arXiv preprint arXiv:2309.14316,

    [AZL23b] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction.arXiv preprint arXiv:2309.14316,

    Pith/arXiv arXiv

  4. [8]

    Physics of language models: Part 3.2, knowledge manipulation.arXiv preprint arXiv:2309.14402,

    [AZL23c] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation.arXiv preprint arXiv:2309.14402,

    Pith/arXiv arXiv

  5. [9]

    Physics of language models: Part 3.3, knowledge capacity scaling laws.arXiv preprint arXiv:2404.05405,

    [AZL24] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws.arXiv preprint arXiv:2404.05405,

    Pith/arXiv arXiv

  6. [10]

    Scaling and renor- malization in high-dimensional regression.arXiv preprint arXiv:2405.00592,

    [AZVP24] Alexander Atanasov, Jacob A Zavatone-Veth, and Cengiz Pehlevan. Scaling and renor- malization in high-dimensional regression.arXiv preprint arXiv:2405.00592,

    Pith/arXiv arXiv

  7. [12]

    Scaling laws are redundancy laws.arXiv preprint arXiv:2509.20721,

    [BC25] Yuda Bi and Vince D Calhoun. Scaling laws are redundancy laws.arXiv preprint arXiv:2509.20721,

    arXiv

  8. [14]

    Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712,

    14 [BCE+23] S´ ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712,

    Pith/arXiv arXiv

  9. [16]

    Learning curves for sgd on structured features

    [BP21] Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd on structured features. arXiv preprint arXiv:2106.02713,

    Pith/arXiv arXiv

  10. [18]

    Scaling law phenomena across regression paradigms: Multiple and kernel approaches

    [CGL+25] Yifang Chen, Xuyang Guo, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. Scaling law phenomena across regression paradigms: Multiple and kernel approaches. arXiv preprint arXiv:2503.01314,

    Pith/arXiv arXiv

  11. [19]

    Flashattention-2: Faster attention with better parallelism and work parti- tioning.arXiv preprint arXiv:2307.08691,

    [Dao23] Tri Dao. Flashattention-2: Faster attention with better parallelism and work parti- tioning.arXiv preprint arXiv:2307.08691,

    Pith/arXiv arXiv

  12. [21]

    How sparse attention approximates exact attention? your attention is naturallyn c-sparse.arXiv preprint arXiv:2404.02690,

    [DSXY24] Yichuan Deng, Zhao Song, Jing Xiong, and Chiwun Yang. How sparse attention approximates exact attention? your attention is naturallyn c-sparse.arXiv preprint arXiv:2404.02690,

    Pith/arXiv arXiv

  13. [22]

    Unlocking the theory behind scaling 1-bit neural networks.arXiv preprint arXiv:2411.01663,

    [DSY24] Majid Daliri, Zhao Song, and Chiwun Yang. Unlocking the theory behind scaling 1-bit neural networks.arXiv preprint arXiv:2411.01663,

    Pith/arXiv arXiv

  14. [23]

    Gradient descent provably optimizes over-parameterized neural networks

    [DZPS19] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. InICLR. arXiv preprint arXiv:1810.02054,

    Pith/arXiv arXiv

  15. [24]

    Tinystories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759,

    [EL23] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759,

    Pith/arXiv arXiv

  16. [25]

    A primer on the inner workings of transformer-based language models.arXiv preprint arXiv:2405.00208,

    [FSBCj24] Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-juss` a. A primer on the inner workings of transformer-based language models.arXiv preprint arXiv:2405.00208,

    Pith/arXiv arXiv

  17. [26]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    [GDJ+24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  18. [27]

    An over-parameterized exponential regression.arXiv preprint arXiv:2303.16504,

    [GMS23] Yeqi Gao, Sridhar Mahadevan, and Zhao Song. An over-parameterized exponential regression.arXiv preprint arXiv:2303.16504,

    Pith/arXiv arXiv

  19. [28]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    [GZY+24] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    Pith/arXiv arXiv

  20. [29]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    [HBB+20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  21. [30]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    [HBM+22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Pith/arXiv arXiv

  22. [31]

    Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

    [HNA+17] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Has- san Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

    Pith/arXiv arXiv

  23. [33]

    Scaling laws under the microscope: Predicting transformer performance from small scale experiments.arXiv preprint arXiv:2202.06387,

    [ICB22] Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments.arXiv preprint arXiv:2202.06387,

    Pith/arXiv arXiv

  24. [36]

    Polysketchformer: Fast trans- formers via sketches for polynomial kernels.arXiv preprint arXiv:2310.01655,

    [KMZ23] Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. Polysketchformer: Fast trans- formers via sketches for polynomial kernels.arXiv preprint arXiv:2310.01655,

    Pith/arXiv arXiv

  25. [37]

    Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161,

    [LAZ+23] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161,

    Pith/arXiv arXiv

  26. [38]

    Machine learning for elliptic pdes: Fast rate generalization bound, neural scaling law and min- imax optimality.arXiv preprint arXiv:2110.06897,

    [LCL+21] Yiping Lu, Haoxuan Chen, Jianfeng Lu, Lexing Ying, and Jose Blanchet. Machine learning for elliptic pdes: Fast rate generalization bound, neural scaling law and min- imax optimality.arXiv preprint arXiv:2110.06897,

    Pith/arXiv arXiv

  27. [39]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

    17 [LFW+24] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

    Pith/arXiv arXiv

  28. [40]

    Deepseek-v3 technical report

    [LFX+24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  29. [41]

    Exploring the frontiers of softmax: Provable optimization, applications in diffusion model, and beyond.arXiv preprint arXiv:2405.03251,

    [LLSS24] Chenyang Li, Yingyu Liang, Zhenmei Shi, and Zhao Song. Exploring the frontiers of softmax: Provable optimization, applications in diffusion model, and beyond.arXiv preprint arXiv:2405.03251,

    arXiv

  30. [42]

    Towards infinite-long prefix in transformer

    [LSSY25] Yingyu Liang, Zhenmei Shi, Zhao Song, and Chiwun Yang. Towards infinite-long prefix in transformer. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 11127–11191, Suzhou, China, November

    2025

  31. [43]

    [L WK+24] Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee

    Association for Computational Linguistics. [L WK+24] Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data.arXiv preprint arXiv:2406.08466,

    Pith/arXiv arXiv

  32. [44]

    Scaling laws and in-context learning: A unified theoretical framework.arXiv preprint arXiv:2511.06232,

    [MG25] Sushant Mehta and Ishan Gupta. Scaling laws and in-context learning: A unified theoretical framework.arXiv preprint arXiv:2511.06232,

    arXiv

  33. [45]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424,

    [MRKK23] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424,

    Pith/arXiv arXiv

  34. [46]

    Beyond scaling laws: Understanding transformer performance with associative memory.arXiv preprint arXiv:2405.08707,

    [NBDH24] Xueyan Niu, Bo Bai, Lei Deng, and Wei Han. Beyond scaling laws: Understanding transformer performance with associative memory.arXiv preprint arXiv:2405.08707,

    Pith/arXiv arXiv

  35. [47]

    Closing the curvature gap: Full transformer hessians and their implications for scaling laws.arXiv preprint arXiv:2510.16927,

    18 [PKMG25] Egor Petrov, Nikita Kiselev, Vladislav Meshkov, and Andrey Grabovoy. Closing the curvature gap: Full transformer hessians and their implications for scaling laws.arXiv preprint arXiv:2510.16927,

    arXiv

  36. [48]

    4+ 3 phases of compute-optimal neural scaling laws.arXiv preprint arXiv:2405.15074,

    [PPXP24] Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ 3 phases of compute-optimal neural scaling laws.arXiv preprint arXiv:2405.15074,

    Pith/arXiv arXiv

  37. [49]

    A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673,

    [RRBS19] Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673,

    Pith/arXiv arXiv 1909

  38. [50]

    Nonparametric regression using deep neural networks with relu activation function.arXiv preprint arXiv:1708.06633v3,

    [SH17] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function.arXiv preprint arXiv:1708.06633v3,

    Pith/arXiv arXiv

  39. [51]

    Discov- ering the gems in early layers: Accelerating long-context llms with 1000x input token reduction.arXiv preprint arXiv:2409.17422,

    [SMN+24] Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. Discov- ering the gems in early layers: Accelerating long-context llms with 1000x input token reduction.arXiv preprint arXiv:2409.17422,

    Pith/arXiv arXiv

  40. [52]

    Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality.arXiv preprint arXiv:1810.08033,

    [Suz18] Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality.arXiv preprint arXiv:1810.08033,

    Pith/arXiv arXiv

  41. [53]

    Quadratic suffices for over-parametrization via matrix chernoff bound.arXiv preprint arXiv:1906.03593,

    [SY19] Zhao Song and Xin Yang. Quadratic suffices for over-parametrization via matrix chernoff bound.arXiv preprint arXiv:1906.03593,

    Pith/arXiv arXiv 1906

  42. [54]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    [TLI+23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  43. [55]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    19 [TMS+23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  44. [56]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682,

    [WTB+22] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682,

    Pith/arXiv arXiv

  45. [57]

    Baize: An open- source chat model with parameter-efficient tuning on self-chat data.arXiv preprint arXiv:2304.01196,

    [XGDM23] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open- source chat model with parameter-efficient tuning on self-chat data.arXiv preprint arXiv:2304.01196,

    Pith/arXiv arXiv

  46. [58]

    Emergence of segmentation with minimalistic white-box transformers

    [YCT+23] Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Yi Ma. Emergence of segmentation with minimalistic white-box transformers. arXiv preprint arXiv:2308.16271,

    Pith/arXiv arXiv

  47. [59]

    Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

    [YJS+23] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

    Pith/arXiv arXiv

  48. [1988]

    The recurrent neural tangent kernel.arXiv preprint arXiv:2006.10246,

    [A WBB20] Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel.arXiv preprint arXiv:2006.10246,

    Pith/arXiv arXiv 2006

  49. [2007]

    Scal- ing laws and representation learning in simple hierarchical languages: Transformers vs

    [CFSW25] Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, and Matthieu Wyart. Scal- ing laws and representation learning in simple hierarchical languages: Transformers vs. convolutional architectures.arXiv preprint arXiv:2505.07070,

    Pith/arXiv arXiv

  50. [2014]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    [KMH+20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  51. [2017]

    A formal framework for understanding length generalization in transformers.arXiv preprint arXiv:2410.02140,

    [HYB+24] Xinting Huang, Andy Yang, Satwik Bhattamishra, Yash Sarrof, Andreas Krebs, Hattie Zhou, Preetum Nakkiran, and Michael Hahn. A formal framework for understanding length generalization in transformers.arXiv preprint arXiv:2410.02140,

    Pith/arXiv arXiv

  52. [2018]

    Adam: A method for stochastic optimization

    [KB14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    Pith/arXiv arXiv

  53. [2019]

    One pass streaming algorithm for super long token attention approximation in sublinear space.arXiv preprint arXiv:2311.14652,

    [ALSY23] Raghav Addanki, Chenyang Li, Zhao Song, and Chiwun Yang. One pass streaming algorithm for super long token attention approximation in sublinear space.arXiv preprint arXiv:2311.14652,

    Pith/arXiv arXiv

  54. [2020]

    Physics of language models: Part 1, context-free grammar.arXiv preprint arXiv:2305.13673,

    [AZL23a] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, context-free grammar.arXiv preprint arXiv:2305.13673,

    Pith/arXiv arXiv

  55. [2021]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  56. [2022]

    Attention scheme inspired softmax re- gression.arXiv preprint arXiv:2304.10411,

    [DLS23] Yichuan Deng, Zhihang Li, and Zhao Song. Attention scheme inspired softmax re- gression.arXiv preprint arXiv:2304.10411,

    Pith/arXiv arXiv

  57. [2023]

    Neural networks as kernel learners: The silent alignment effect.arXiv preprint arXiv:2111.00034,

    [ABP21] Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect.arXiv preprint arXiv:2111.00034,

    Pith/arXiv arXiv

  58. [2024]

    A dynamical model of neural scaling laws.arXiv preprint arXiv:2402.01092,

    [BAP24] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws.arXiv preprint arXiv:2402.01092,

    Pith/arXiv arXiv

  59. [2025]

    Deepseek llm: Scaling open- source language models with longtermism.arXiv preprint arXiv:2401.02954,

    [BCC+24] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open- source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Pith/arXiv arXiv