Pith. sign in

REVIEW 4 major objections 5 minor 21 cited by

Data-limited scaling exponents of language models are set by two measurable statistics of the training corpus: the entropy-decay exponent γ and the correlation-decay exponent β, yielding α_D = γ/(2β).

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:34 UTC pith:WQEG76N3

load-bearing objection First quantitative theory of LLM data-limited scaling from language statistics, with a clean formula and elegant collapse — but the fast-learning assumption underpinning it is tested only thinly and the AR exponent is never fitted, so treat as promising, not settled. the 4 major comments →

arxiv 2602.07488 v3 pith:WQEG76N3 submitted 2026-02-07 cs.LG cs.AIstat.ML

Deriving Neural Scaling Laws from the statistics of natural language

Francesco Cagnetta , Allan Ravent\'os , Surya Ganguli , Matthieu Wyart This is my paper
classification cs.LG cs.AIstat.ML MSC 68T0768T50
keywords neural scaling lawsdata-limited scalingautoregressive lossconditional entropy decaytoken-token correlationshorizon-limited learningscaling collapsenatural language statistics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that how fast a language model's test loss falls with the amount of training data — the data-limited scaling exponent — is not set by the architecture but by two statistical properties of the language itself: how quickly next-token conditional entropy drops as context grows (exponent γ) and how quickly token-token correlations decay with separation (exponent β). The mechanism is horizon-limited learning: with P tokens a model can only exploit statistical dependencies up to a context length n*(P) ≍ P^{1/(2β)}, and the loss is dominated by the entropy at that horizon, giving excess loss ≍ P^{−γ/(2β)}. The prediction has no free parameters and is verified by measuring γ and β on two corpora and comparing with the observed loss curves of transformer language models, including a data-collapse of all n-gram losses onto one curve. If right, it turns a widely observed empirical law into a derived consequence of linguistic statistics, and identifies the effective horizon as the learning bottleneck in the data-limited regime.

Core claim

The central discovery is that the data-limited neural scaling exponent α_D for autoregressive language modeling is, in the fast-learning regime, α_D = γ/(2β), where γ is the exponent of the power-law decay H_n − H_∞ ≍ n^{−γ} of the next-token conditional entropy with context length n, and β is the exponent of the power-law decay ∥C(n)∥_op ≍ n^{−β} of the operator norm of the token-token covariance matrix at lag n. The argument: a signal-to-noise threshold ∥C(n)∥_op ≍ P^{−1/2} sets the largest context length n*(P) ≍ P^{1/(2β)} that P tokens can support; the autoregressive loss then decomposes into the entropy at that horizon plus excess losses from suboptimal within-horizon prediction; if the

What carries the argument

The load-bearing object is the data-dependent prediction horizon n*(P), defined by equating the strongest token-token correlation at lag n, ∥C(n)∥_op, with the sampling noise O(P^{−1/2}); this gives n*(P) ≍ P^{1/(2β)}. Around this horizon the loss decomposition L_AR(P) ≍ H_{n*(P)} + Σ_{n≤n*(P)} E_n(P) separates two learning mechanisms — extending the horizon versus improving prediction within it — and the power-law ansatze H_n − H_∞ ≍ n^{−γ} and ∥C(n)∥_op ≍ n^{−β} convert it into the exponent prediction. The supporting identity is the scaling collapse L_n(P) ≍ n^{−γ} ℓ(P/n^{2β}), which makes the theory testable curve-by-curve.

Load-bearing premise

The whole exponent identification depends on the fast-learning hypothesis — that within the prediction horizon the model already uses available context nearly optimally, so the dominant effect of more data is to lengthen the horizon; if the within-horizon excess losses decay with an exponent δ ≤ γ/(2β), the observed scaling would instead be P^{−δ} and the prediction would not be α_D = γ/(2β).

What would settle it

Train a transformer on a text corpus whose γ and β have been measured, and fit the late-P decay of individual n-gram losses L_n(P) − H_n. If for any n the fitted exponent δ_n is no larger than γ/(2β), then the theory's fast-learning regime fails: Eq. 38 predicts the full loss should decay as P^{−δ_n} (slower than γ/(2β)), so checking whether the observed autoregressive loss follows the slower decay or the faster one would settle which regime governs.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the formula holds, the data-limited scaling exponent of any new corpus is fixed once γ and β are measured on that corpus, with no model training needed.
  • Improving data efficiency in this regime must come from breaking the horizon bottleneck — e.g., architectures that capture longer-range correlations from fewer tokens — since within-horizon learning is already fast.
  • The predicted collapse of n-gram loss curves provides a strong, architecture-independent signature that the model is in the horizon-limited regime: all context lengths follow the same master curve.
  • The theory implies a universality class: different deep architectures trained on the same language share the same exponent (though prefactors may differ), while shallow networks and kernel methods fall outside it with worse exponents.
  • The finite-context corrections are controlled: when the maximal context T becomes comparable to n*(P), the loss is truncated by T, so the pure power law is only expected in the regime n*(P) ≪ T — a fact the paper uses to set the valid comparison range.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: measure γ and β on a corpus with engineered statistics (e.g., a finite-order Markov chain with known decays) and check that the trained transformer's exponent follows γ/(2β); this would isolate the mechanism from confounds of real language.
  • If the horizon-limited picture extends to compute-optimal or model-size-limited scaling, the same two exponents might reappear in different combinations there — a guess the paper does not make but that arises naturally from its mechanism.
  • The collapse prediction suggests a cheap diagnostic: plot n^γ L_n against P/n^{2β}; systematic deviations at large n would show where the horizon-limited assumption breaks down, possibly indicating a transition to within-horizon-limited learning.
  • Because the theory identifies the bottleneck as information-theoretic, it hints that data-selection or curriculum strategies that lower effective β (making longer-range correlations easier to detect) could improve scaling exponents — an implication the authors leave implicit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a theory to predict the data-limited neural scaling law exponent for autoregressive language models from two measurable corpus statistics: the power-law decay of next-token conditional entropy with context length (exponent γ) and the power-law decay of token-token correlation strength with temporal lag (exponent β). The central claim is that, in the 'horizon-limited' fast-learning regime, the autoregressive test loss decays as L_AR(P) − H_∞ ≍ P^{−γ/(2β)}, with the exponents being properties of the dataset alone. The derivation in App. A decomposes the loss into a boundary term from the data-dependent prediction horizon and a sum of within-horizon excess losses, and shows that the observed exponent is min(δ, γ/(2β)). The paper estimates γ and β on TinyStories and WikiText, demonstrates collapse of n-gram loss curves under the predicted rescaling, and compares the autoregressive loss decay to the predicted slope for GPT-2/LLaMA-style transformers.

Significance. If the central claim holds, this would be a major advance: the first quantitative, parameter-free prediction of neural scaling law exponents from the statistics of natural language, with a falsifiable collapse prediction. The paper's strengths include a self-contained analytical derivation, explicit statements of all hypotheses, cross-architecture consistency of the estimated entropy exponent, and empirical tests on two qualitatively different corpora. However, the validity of the headline exponent rests on the fast-learning condition δ > γ/(2β), and the current evidence for this condition is thin; moreover, the principal validation of the exponent itself is visual rather than quantitative. These gaps make the claim conditional rather than established.

major comments (4)
  1. [§5, Eq. (38), Fig. 6] The fast-learning condition δ > γ/(2β) is load-bearing: if instead δ < γ/(2β), the observed exponent is δ and the advertised prediction α_D = γ/(2β) fails. The only direct test, Fig. 6, has three weaknesses: (i) it covers only n ≤ 12, while the data-dependent horizon n*(P) reaches a few tens of tokens at the largest P, so the regime most relevant to Eq. (34) is not directly tested; (ii) H_n is not measured independently but chosen by a grid search that maximizes R² of the subsequent power-law fit, so δ_n and H_n are co-fitted and the decay rate can be inflated; (iii) no confidence intervals are reported, and δ_n ≈ 0.2–0.3 versus γ/(2β) ≈ 0.19 is a thin margin. This precondition must be verified more rigorously before the central claim can be accepted.
  2. [§4.2, Figs. 1 and 4] The headline validation of the exponent prediction is only visual: the autoregressive losses L(P) are plotted against a dashed line of slope −γ/(2β), but no fitted slope or confidence interval for L(P) is reported for any context length T or dataset. Since the central claim is a quantitative exponent prediction, the paper should report empirical exponents obtained from power-law fits over the stated scaling range, together with uncertainties, and compare them explicitly to γ/(2β). Without this, the match shown in the figures cannot be distinguished from a range of plausible exponents.
  3. [§4.1.1] The entropy exponent γ is estimated from n-gram losses of trained transformers from the same model families (GPT-2, LLaMA) that the theory subsequently predicts. The paper argues for convergence and architecture independence, which supports the interpretation that L_n approaches H_n, but this is not a fully independent measurement of a dataset property. To strengthen the claim that γ (and hence α_D) is determined by the corpus alone, the authors should provide an independent estimate of H_n (e.g., from a different estimator or model family) or at least a sensitivity analysis showing that γ is stable across model sizes and training budgets.
  4. [Limitations] The Limitations section concedes that n*(P_max) corresponds to a few tens of tokens and that the tests are at academic scale. This is a significant scope restriction: the fast-learning regime is asserted to be the relevant one for modern LLMs, but the empirical evidence is confined to very short effective horizons and dataset sizes up to 10^8 tokens. The paper should either temper the title-level claim ('neural scaling laws' of modern LLMs) or provide additional evidence, e.g., by testing on larger P to push n* further, or by studying a setting where the within-horizon excess losses can be verified at larger n.
minor comments (5)
  1. [§5, Fig. 6] The scatter plot of δ_n (right panel) has no error bars. Adding confidence intervals would help judge whether δ_n > γ/(2β) is robust, especially given the co-fitting procedure.
  2. [§4.1.2] For WikiText, the correlation decay is described as a broken power law with a localized peak near n≈10. The paper states that the short-lag regime is used, but it would help to report the sensitivity of β to the fitting range and to the treatment of the outlier peak.
  3. [App. C] Code is promised only 'upon publication.' For reproducibility, consider releasing code and hyperparameters at the time of submission, or at least providing a complete table of final hyperparameters for all experiments.
  4. [§5 / App. A] The notation δ_n is used in Sec. 5, while the derivation in App. A sets δ_n = δ for simplicity. The relationship between the estimated δ_n and the single δ used in Eq. (35)–(38) should be stated explicitly.
  5. [Eq. (22) / Eq. (30)] The factors (T − (n − 1))/T are dropped in Eq. (30) to obtain the asymptotic expression Eq. (31). This implicitly assumes T ≫ n*(P). The condition for this approximation to be valid should be stated in the main text, not only in the prose around Eq. (5).

Circularity Check

0 steps flagged

No circular derivation: the central exponent prediction is a composition of independently measured language statistics, with minor self-citation and a model-based estimate of gamma as caveats.

full rationale

The paper's central formula alpha_D = gamma/(2 beta) is derived by composing two dataset-level power laws: ||C(n)||_op ~ n^{-beta} (measured directly from token co-occurrence counts, Eq. 27) fixes n*(P) ~ P^{1/(2 beta)} via the SNR threshold Eq. 4, and H_n - H_infinity ~ n^{-gamma} (Eq. 6) then gives the boundary-term decay P^{-gamma/(2 beta)}. beta is fully independent of the trained models, and the relation to the autoregressive loss is not tautological: Eq. 38 explicitly contains min{delta, gamma/(2 beta)}, so alpha_D = gamma/(2 beta) is a conditional prediction that fails if within-horizon learning is slow. The two caveats are verification gaps rather than circularity: gamma is estimated from GPT-2/LLaMA n-gram losses on the same corpora (Sec. 4.1.1), though the paper reports consistency across APE, RoPE, and LLaMA, and the fast-learning check (Sec. 5) co-fits H_n by grid search with no confidence intervals. These limit the strength of the evidence but do not make the prediction equal to its input by construction. Self-citations to Cagnetta & Wyart (2024) introduce the horizon concept, but the horizon is re-derived here from the SNR argument, so the self-citation is not load-bearing. The Limitations section explicitly notes the academic scale (n* of a few tens of tokens), which is a scope limitation, not a circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 7 axioms · 0 invented entities

The central prediction rests on two measured power-law exponents (γ, β) and an ansatz about how differential losses transition at a data threshold. The deepest 'free' input is the fast-learning assumption δ > γ/(2β), which is structurally required for the exponent α_D to be γ/(2β) rather than δ. No new particles, forces, or physical entities are postulated; the data-dependent horizon n*(P) is a derived construct, not a new entity. The paper honestly labels these as hypotheses, but the 'no free parameters' claim in the abstract applies only to the final exponent formula, not to the measurement of γ and β or to the assumed power-law forms.

free parameters (4)
  • γ (conditional entropy decay exponent) = 0.34 (TinyStories), 0.27 (WikiText)
    Fit by power-law regression to the small-n portion of L_n-vs-n for the largest-P trained model (Sec. 4.1.1). This is an input to the prediction α_D = γ/(2β); it is measured from data, not fitted to the target scaling-law exponent.
  • β (token-token correlation decay exponent) = 0.88 (TinyStories), 0.94 (WikiText)
    Fit by power-law regression to ‖C(n)‖_op-vs-n (Sec. 4.1.2). For WikiText only the initial decay stage is used; the broken power law and the n≈10 peak are explicitly ignored.
  • δ_n (excess-loss exponents) = ≈0.2–0.3 (Fig. 6, n ≤ 12)
    Extracted by fitting L_n − H_n to A P^{−δ_n} with a grid search over the asymptote H_n (step 10^{−2}). Used to test the fast-learning condition δ > γ/(2β); not used directly in the final exponent formula.
  • Threshold constant c in Eq. 26 = unspecified O(1)
    Introduced in the signal-to-noise threshold P*_n = c^2/‖C(n)‖². It cancels in all exponent predictions, so it does not affect the central claim, but it is an extra undetermined constant in the framework.
axioms (7)
  • domain assumption H_n − H_∞ decays as a power law n^{−γ} (Hypothesis 1, App. A Eq. 19).
    Empirically motivated; not derived from first principles. It is an input to the final exponent formula.
  • domain assumption ‖C(n)‖_op decays as a power law n^{−β} (Hypothesis 2, App. A Eq. 27).
    Empirically motivated; the WikiText data show a broken power law with a peak near n≈10, so the assumption is only approximately valid in the fitted stage.
  • ad hoc to paper Differential loss Δ_n(P) has the scaling form Δ_n(P) = (H_n − H_{n−1}) f_n(P/P*_n) with f_n(x)→0 for x≪1 and 1−f_n(x)→x^{−δ_n} for x≫1 (App. A Eq. 23).
    This ansatz is not derived; it is the backbone of the derivation of both the collapse form (Eq. 9) and the final exponent (Eq. 38).
  • ad hoc to paper Fast-learning condition: δ > γ/(2β) (equivalently, the excess-loss sum decays at least as fast as the boundary term; App. A Eqs. 36–38).
    Without this condition the predicted exponent would be δ, not γ/(2β). Verified only indirectly for n ≤ 12 in Sec. 5, with a thin margin.
  • standard math The trained model's n-gram loss L_n(P,M) is an upper bound on H_n and converges to H_n for infinite data at sufficient capacity (Eq. 10).
    Cross-entropy ≥ entropy is exact; convergence in the infinite-data limit is a standard consistency assumption for expressive models, and is used to estimate γ from finite P.
  • domain assumption Stationarity of the token process, i.e. (X_1:(n−1)) has the same distribution as (X_2:n) (needed for Eq. 21).
    Assumed so that Δ_n = L_n − L_{n−1}; real corpora are chunked and concatenated with EOS tokens, so stationarity is only approximate.
  • domain assumption Signal-to-noise model: the learner can detect a correlation lag n only if ‖C(n)‖_op > c/√P (Eq. 26).
    Uses O(P^{−1/2}) concentration of empirical covariance matrices; the constant c is unknown but cancels in the exponent.

pith-pipeline@v1.3.0-alltime-deepseek · 20840 in / 15571 out tokens · 150383 ms · 2026-08-03T03:34:43.577198+00:00 · methodology

0 comments
read the original abstract

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

Figures

Figures reproduced from arXiv: 2602.07488 by Allan Ravent\'os, Francesco Cagnetta, Matthieu Wyart, Surya Ganguli.

Figure 1
Figure 1. Figure 1: Measurable language statistics predict the exponents of data-limited neural scaling laws in language models. Top: The highly diverse n-gram losses Ln (Eq. 2) of a GPT-2–style transformer trained from scratch on P-tokens of the TinyStories dataset (left) collapse onto a single curve, when plotted in rescaled units (right). Here Ln is rescaled by Hn ≍ n −γ , where γ is the exponent of the power law temporal … view at source ↗
Figure 2
Figure 2. Figure 2: Conditional entropy decay with time horizon defines a characteristic exponent γ that is architecture-independent. We train three classes of models from scratch on P-token slices of the TinyStories dataset: GPT-2–style transformers with absolute positional embeddings (Left), GPT-2–style transformers with rotary positional embeddings (RoPE, Center Left), and LLaMA-style transformers (Center Right), training … view at source ↗
Figure 3
Figure 3. Figure 3: Decay of two-point correlations as a function temporal separation defines a characteristic exponent β. The second dataset-level statistic we consider is the decay of the two-point correlation function, defined as the norm of the token-token co-occurrence matrix C(n)µν = P(Xi = µ, Xi+n = ν) − P(Xi = µ)P(Xi+n = ν), with the time separation n. We define β as the exponent of a power-law fit to this decay. C(n)… view at source ↗
Figure 4
Figure 4. Figure 4: Same as [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fitting γ for the WikiText dataset. (Similar to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: n-gram losses decay to their asymptote faster than P −γ/(2β) . Left: We plot the n-gram losses of our GPT-2–style transformer trained on TinyStories for low n (n ≤ 12). To isolate the decay due to the suboptimal use of the time horizon, we fit the large-P portion of these curves to the decay-to-asymptote form A × P −δn + Hn. We perform a grid search over asymptotes with a step of 10−2 , then subtract the a… view at source ↗
Figure 7
Figure 7. Figure 7: n-gram loss collapse for GPT-2-style transformers trained on WikiText at T = 512. (Same as [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: n-gram collapse and data-limited scaling exponent prediction for GPT-2-style transformers with RoPE trained on WikiText. (Same as [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: n-gram loss collapse for GPT-2-style transformers with RoPE trained on WikiText at T = 512. (Same as [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: n-gram loss collapse for GPT-2-style transformers trained on TinyStories at T = 64, 256 and 512. (Same as [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: n-gram collapse and data-limited scaling exponent prediction for GPT-2-style transformers with RoPE trained on TinyStories. (Same as [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: n-gram loss collapse for LLaMA-style transformers trained on TinyStories at T = 512. These correspond to the same n-gram losses used in [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

    cs.LG 2026-05 unverdicted novelty 8.0

    Neural LoFi models deep learning as layer-wise spectral filtering that selects maximal low-degree correlations, yielding a tractable surrogate for hierarchical representation learning beyond the lazy regime.

  2. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  3. Smooth Scaling Laws Hide Stepwise Token Learning

    cs.CL 2026-06 conditional novelty 7.0

    Power-law LLM scaling laws are largely the aggregate of stepwise token learning events whose heavy-tailed learning-time spectrum reconstructs loss derivatives along step, data, and model axes.

  4. Smooth Scaling Laws Hide Stepwise Token Learning

    cs.CL 2026-06 unverdicted novelty 7.0

    Token loss trajectories follow localized sigmoids whose learning-time spectrum quantitatively reconstructs scaling-law derivatives on T, D, and M axes and enables faster training via distribution reshaping.

  5. Phase structure of the Random Language Model

    cond-mat.dis-nn 2026-06 unverdicted novelty 7.0

    The Random Language Model exhibits a hierarchy of phase transitions in the double-scaling limit ε̃_d → 0, N → ∞ at fixed x = ε̃_d log N, with symbol correlations, non-uniform marginals, and glassy freezing, yielding s...

  6. Learn from your own latents and not from tokens: A sample-complexity theory

    cs.LG 2026-05 unverdicted novelty 7.0

    Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.

  7. Critical Percolation as a Synthetic Data Model for Interpretability

    cs.LG 2026-06 unverdicted novelty 6.0

    Critical percolation clusters embedded in high dimensions, combined with taxonomic latent variables, form an analytically tractable synthetic data model whose ground-truth hierarchy can be linearly decoded from networ...

  8. Scaling Laws for Neural-Network Quantum States

    cond-mat.dis-nn 2026-06 unverdicted novelty 6.0

    Transformer wave functions for the J1-J2 Heisenberg model exhibit size-independent power-law decay of V-score with compute, with the exponent decreasing as frustration increases.

  9. Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

    cs.LG 2026-05 unverdicted novelty 6.0

    Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B p...

  10. Information-Theoretic Limits of Reliability and Scaling in Language Models

    cs.CL 2026-05 conditional novelty 6.0

    A theoretical framework derives a reliability ceiling and a max-form Chinchilla-type scaling law for LLMs from task entropy and dependency spectra.

  11. Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data

    cs.LG 2026-04 unverdicted novelty 6.0

    Uniform-based discrete diffusion models behave as associative memories that retrieve unseen data, with a dataset-size-driven memorization-to-generalization transition detectable via conditional entropy of token predictions.

  12. Geometry of Ordinal Representations in Language Models

    cs.LG 2026-07 conditional novelty 5.5

    1D place-cell manifolds emerge for locally token-computable ordinals across models, while cross-position or semantic ordinals yield higher-dimensional or incoherent geometry, with architecture-dependent twisting.

  13. Test-Time Training for Modality Order Consistency in Vision-Language Models

    cs.CV 2026-07 conditional novelty 5.0

    VLMs are less accurate when the question precedes the image; per-instance KL self-distillation using the image-first answer as teacher closes most of the gap and slightly improves the image-first branch.

  14. How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

    cs.LG 2026-06 unverdicted novelty 5.0

    Quadratic two-layer networks exhibit data-dependent power-law generalization scaling with distinct regimes in width and sample size, including an interpolation transition whose location depends on target spectrum.

  15. Neuron Populations Exhibit Divergent Selectivity with Scale

    cs.LG 2026-06 unverdicted novelty 5.0

    Rosetta Neurons in language models up to 30B and vision models up to 5B parameters scale sublinearly with size while becoming more selective and monosemantic.

  16. Sampling Data with Chains of Forward-Backward Diffusion Steps

    cs.LG 2026-05 unverdicted novelty 5.0

    U-turn chains are Markov chains formed by short forward-backward diffusion steps that remain on the learned manifold and, with Metropolis-Hastings, sample from energy-modified targets, exhibiting an ergodicity-breakin...

  17. Asymmetric Scaling Laws from Sparse Features

    stat.ML 2026-05 unverdicted novelty 5.0

    A sparse-activation model predicts double-descent loss with distinct under- and over-parameterized scaling exponents set by sparsity, plus a compute-optimal frontier favoring dataset growth.

  18. Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

    cs.LG 2026-06 unverdicted novelty 4.0

    Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.

  19. Statistical Properties of Training & Generalization

    stat.ML 2026-06 unverdicted novelty 2.0

    Neural scaling laws in deep learning interact with physics constraints and inductive biases beyond classical statistics.

  20. There Will Be a Scientific Theory of Deep Learning

    stat.ML 2026-04 unverdicted novelty 2.0

    A mechanics of the learning process is emerging in deep learning theory, characterized by dynamics, coarse statistics, and falsifiable predictions across idealized settings, limits, laws, hyperparameters, and universa...

  21. Statistical Properties of Training & Generalization

    stat.ML 2026-06 unverdicted novelty 1.0

    Review of neural scaling laws and their relation to constraints and inductive biases when applying machine learning to physics problems.

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages · cited by 19 Pith papers

  1. [4]

    URL https: //proceedings.neurips.cc/paper_files /paper/2024/file/1dccfc3ee01871d05e3 3457c61037d59-Paper-Conference.pdf

    doi: 10.52202/079017-0526. URL https: //proceedings.neurips.cc/paper_files /paper/2024/file/1dccfc3ee01871d05e3 3457c61037d59-Paper-Conference.pdf. Paszke, A. et al. Pytorch: An imperative style, high- performance deep learning library. InNeurIPS, 2019. Peters, M. E., Neumann, M., Zettlemoyer, L., and Yih, W. Dissecting contextual word embeddings: Archite...

    work page doi:10.52202/079017-0526 2024

  2. [2017]

    (wikitext-103-raw-v1) datasets. All text is tokenized using a two-stage procedure consisting of whitespace pre- tokenization followed by byte-pair encoding (BPE) (Gage, 1994; Sennrich et al., 2016) with a vocabulary size of 8192. The resulting token sequences are concatenated into a single stream, with an end-of-sequence (EOS) token inserted between docum...

    1994

  3. [2018]

    boundary term

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1179. URL https://aclantho logy.org/D18-1179. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y . Resolving discrepancies in compute-optimal scaling of language models, 2025. URL https://ar xiv.org/abs/2406.19146. 10 Deriving neural scaling laws from the statistics of natural...

    Pith/arXiv arXiv 2025

  4. [2022]

    Manning, C

    URL https://arxiv.org/abs/2210.1 6859. Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., and Levy, O. Emergent linguistic structure in artificial neural networks trained by self-supervision.Proceedings of the National Academy of Sciences, 117(48):30046–30054, 2020. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. I...

    2020

  5. [2023]

    cc/paper_files/paper/2023/file/5b634 6a05a537d4cdb2f50323452a9fe-Paper-Con ference.pdf

    URL https://proceedings.neurips. cc/paper_files/paper/2023/file/5b634 6a05a537d4cdb2f50323452a9fe-Paper-Con ference.pdf. Nam, Y ., Fonseca, N., Lee, S. H., Mingard, C., and Louis, A. A. An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys...

    2023

  6. [2024]

    URL https: //www.pnas.org/doi/abs/10.1073/pnas

    doi: 10.1073/pnas.2311878121. URL https: //www.pnas.org/doi/abs/10.1073/pnas. 2311878121. Barkeshli, M., Alfarano, A., and Gromov, A. On the origin of neural scaling laws: from random graphs to natural language.arXiv preprint arXiv:2601.10684, 2026. URL https://arxiv.org/abs/2601.10684. Bordelon, B., Canatar, A., and Pehlevan, C. Spectrum- dependent learn...

    arXiv 2026