{"id":"c78ba85c-1238-443e-b123-07d6cfb8f94e","arxiv_id":"2505.07067","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On hierarchical grammar data with Zipf-distributed production rules, classification error decays as P^{-a/(1+a)} while next-token prediction retains a hierarchy-only asymptotic exponent.","lead":"This paper derives how learning curves, test error versus training data, scale when data are generated by hierarchical grammars whose production rules follow Zipf's law. It shows hierarchy controls the pre-asymptotic phase and the feature distribution controls the asymptotic exponent for classification, but not for next-token prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline exponents hinge on Assumption 3.1, which maps correlation detectability to a sharp 1/f_k sample-complexity threshold; this mapping is not derived for gradient-trained networks and is only verified in a narrow parameter regime.","rationale":"The reader's weakest_assumption correctly identifies Assumption 3.1 as the load-bearing step, with the m=v^{s-1} caveat as a second premise. I agree: both headline results are derived from sharp correlation-detectability thresholds, and the paper does not supply a derivation of these thresholds from the training dynamics of deep networks. The empirical sections support the predictions in the tested parameter ranges, and the limitations section and the footnote to Section 4.1 are candid about scope, so this is a conditional result rather than an invalid one. The single most decisive check would be to vary m/v^{s-1} for classification and to compare with the Bayes-optimal decoder; either test would reveal whether Eq. 8's exponent is intrinsic to the data model or an artifact of Assumption 3.1. For these reasons the appropriate verdict remains CONDITIONAL, unchanged from the reader's assessment.","tokens_in":18142,"tokens_out":13277,"duration_ms":132467,"concrete_test":"Run the classification experiment of Fig. 2 with s=2, L=2, v=100, a=1, varying m in {10,25,50,100} so that m/v^{s-1} takes values 0.1, 0.25, 0.5, and 1, and fit the empirical exponent of epsilon(P) over P in [10^3, 10^5]. If the fitted exponent moves away from a/(1+a)=1/2 as m/v^{s-1} decreases, then Eq. 7's all-patches-resolvable premise is load-bearing and the paper's scope must be restricted to m=v^{s-1}. Independently, compute the Bayes-optimal classification error for the Zipf-RHM by extending the decoder analysis of Sclocchi et al. (2024) and compare its asymptotic exponent with Eq. 8. If the Bayes-optimal exponent differs from a/(1+a), then Assumption 3.1, not the data model, produces the headline exponent, and the theoretical claim is conditional on an unproven algorithmic heuristic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 3.1 ('Each production rule is learnt when its effect on correlations can be detected from the training data') is the engine of both central results. It converts correlation signal-vs-noise comparisons into hard thresholds P*_k proportional to 1/f_k for classification (Section 4.1, leading to Eq. 8) and P_{l,k} proportional to 1/(f_k sum f^2) for next-token prediction (Eq. 14). The asymptotic exponents quoted in the abstract—a/(1+a) for classification and -log(m/v^{s-1})/(2 log m) for next-token—follow only if a network learns a rule exactly when its correlation becomes statistically detectable. No derivation from gradient-descent dynamics is given, and the empirical support is confined to m=v^{s-1} for classification (Figs. 2-3 and Appendix C) and to small grammars for next-token (Figs. 4-6). If the true sample complexity of learning a rule with probability f_k were P* proportional to f_k^{-beta} with beta != 1, the classification exponent would become beta a/(1+beta a) instead of a/(1+a), breaking the claimed unification with Hutter's memorization exponent. A narrower but real limitation is stated in the footnote to Section 4.1: Eq. 7 assumes all s^{L-1} patches must be resolvable, which the authors concede is exact only when m=v^{s-1}. For m<v^{s-1}, the root may be inferable without resolving every patch, so Eq. 8's prefactor and possibly its exponent are not established outside the tested regime.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies neural scaling laws in a Random Hierarchy Model (RHM) in which production rules follow a Zipf distribution. For classification, it derives the asymptotic learning curve ε(P) ≃ s^{L-1}(P/(v m^{L-1}))^{-a/(1+a)} (Eq. 8), implying that the Zipf exponent a sets the power-law decay while the hierarchy sets a multiplicative pre-asymptotic scale. For next-token prediction, it argues that the asymptotic exponent remains -log(m/v^{s-1})/(2 log m) (Eq. 12), independent of the rule distribution, and that Zipf statistics only modify finite-size corrections. The theory is built on Assumption 3.1, which asserts that each production rule is learned as soon as its correlation signal exceeds sampling noise; all main results follow from this sharp threshold. The predictions are tested with deep CNNs for classification and deep transformers for next-token prediction, with code available online.","tokens_in":18500,"tokens_out":9965,"duration_ms":90364,"significance":"If the results hold, the paper offers a concrete unification of two strands of scaling-law theory: it shows how power-law distributed features (the Hutter mechanism) and hierarchical composition interact, with the classification exponent a/(1+a) directly inherited from the rule distribution and the next-token exponent governed by the tree structure alone. The derivations are transparent, the asymptotic calculations in Appendix B are standard, and the empirical validation spans two architectures with publicly released code. The predictions are falsifiable: the data collapse under P/(v m^{L-1}) for classification and the invariance of the next-token exponent across a are crisp experimental signatures. The principal caveat is the heuristic nature of Assumption 3.1 and the restriction of the classification derivation to the m = v^{s-1} regime, which limits the generality of the abstract claims until those points are addressed.","major_comments":[{"comment":"The central results hinge on Assumption 3.1, which asserts that a production rule is learned exactly when its effect on correlations can be detected from the training data. This is a sharp threshold that is not derived from gradient-descent dynamics. Under the alternative scaling P*_k ∝ f_k^{-β} with β ≠ 1, the classification exponent in Eq. (8) would become β a/(1+β a), breaking the claimed unification with Hutter's memorization exponent. I recommend either deriving Assumption 3.1 in a tractable setting (for example, a linear network or online SGD with a single rule) or providing a direct empirical test of the per-rule sample complexity P*(f) across a range of f. The current experiments validate the aggregate learning curve, not the per-rule threshold.","section":"Section 3.2 (Assumption 3.1) and its use in Sections 4.1 and 5.1"},{"comment":"Equation (7) is derived under the condition that classification succeeds only when all s^{L-1} input patches are resolvable, which the authors acknowledge is exact only when m = v^{s-1}. For m < v^{s-1}, the root can sometimes be inferred without resolving every patch, so the prefactor and possibly the exponent in Eq. (8) are not established outside the tested regime; all classification experiments in Figs. 2-3 and Appendix C use m = v. Since the abstract presents the exponent a/(1+a) as a general result for hierarchical compositional data, the paper should either restrict the claim to m = v^{s-1} or extend the derivation using the optimal-decoder probability from Sclocchi et al. (2024).","section":"Section 4.1, Eq. (7) and footnote 2"},{"comment":"The claim that power-law distributed rules at an arbitrary layer ℓ ≠ 1 lead to the same classification exponent is only sketched in one paragraph; no derivation is given for how the variance and sample-complexity arguments of Section 4.1 generalize to higher layers. The empirical evidence in Fig. 2 (right) and Fig. 7 supports the claim, but the theory as written covers only level-1 nonuniformity. If the intended contribution is the general statement that the classification exponent depends on the rule distribution, this gap should be closed or the contribution restated.","section":"Section 4.2"}],"minor_comments":[{"comment":"The phrase 'Assumption Theorem 3.1' appears to be a typo and should read 'Assumption 3.1'.","section":"Section 5, opening sentence"},{"comment":"The sampling variance is written as 'f_k(µ)/(v^2P)^{-1}'; the exponent appears to be a typo and should read f_k(µ)/(v^2 P).","section":"Section 5.1, paragraph on sampling variance"},{"comment":"The sentence 'decays asymptotically as P^{−a/(1+a)} with the;' is incomplete and should be finished.","section":"Section 1.1, third bullet"},{"comment":"The quantity H_{2,a,m} is used in Eq. (18) without being defined in the main text; a brief definition or a pointer to Appendix E in the main text would improve readability.","section":"Equation (18) and Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is a natural extension of the authors' prior RHM work (Cagnetta et al. 2024; Cagnetta & Wyart 2024). The incremental contribution—Zipf-distributed production rules—is clear, but the main results rest on an unverified sharp-threshold assumption and a special-case condition for classification. The experiments are convincing in the tested regimes, but the abstract and title claim more generality than is proven. I would encourage the editor to request a revision that either strengthens the empirical test of Assumption 3.1 or narrows the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper takes the Random Hierarchy Model and injects Zipf-distributed production rules, then derives learning curves for classification and next-token prediction. The headline results — classification error decays as P^{-a/(1+a)} with a hierarchy-dependent prefactor, while the next-token exponent stays P^{-log(m/v^{s-1})/(2 log m)} regardless of the rule distribution — are new and clean. I think the work deserves a serious referee.\n\nWhat's genuinely new: the per-rule sample complexity P*_k = v m^{L-1}/f_k, the classification curve Eqs. (7)-(8) that combines the Zipf exponent with the hierarchical pre-asymptotic scale, and the demonstration that next-token prediction's asymptotic exponent is invariant to the rule distribution. The derivations are internally consistent; the variance calculations in Appendix D are careful and standard. The empirical figures are convincing within the tested regime, and the authors are transparent about the limitations of their data model.\n\nThe main soft spot is Assumption 3.1: the claim that a rule is learned exactly when its correlation signal becomes statistically detectable. This is the engine behind both the classification and next-token thresholds, but it is not derived from gradient-descent dynamics. If the true sample complexity of a rule scaled as f_k^{-beta} with beta different from 1, the classification exponent would become beta a/(1+beta a), breaking the claimed match with Hutter's memorization exponent. The paper provides empirical support only for m=v^{s-1} in classification (Figs. 2-3) and small grammars for next-token (Figs. 4-6). A second, narrower gap is stated in the footnote to Section 4.1: Eq. (7) assumes all s^{L-1} patches must be resolvable, which is exact only when m=v^{s-1}; outside that regime the prefactor, and possibly the exponent, are not established. These are real limitations, but they are honestly flagged and do not undermine the core claim that the unification is worth taking seriously.\n\nWho is this for? Anyone working on neural scaling laws, especially theorists trying to connect data structure to power-law exponents. It's a synthesis of Hutter's memorization model and the RHM, with an actual new prediction about next-token exponents that is surprising and testable.\n\nMy recommendation: give it a serious peer review. The authors should be asked to stress-test Assumption 3.1, ideally by analyzing a simpler learning rule or by expanding experiments to cover m < v^{s-1} and larger grammars. But the paper is coherent, honest, and a step forward. I'd cite it.\n\nBest,\n[Your name]","headline":"A clean synthesis of Zipf and hierarchy that yields a new classification exponent and a striking next-token invariance, held up by a heuristic learnability assumption that the paper doesn't fully justify.","tokens_in":18997,"tokens_out":4037,"would_cite":true,"duration_ms":38770,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a hierarchy of production rules, making rule frequencies Zipf-distributed turns classification learning curves into power laws with exponent a/(1+a), while next-token prediction keeps its hierarchy-controlled power law.","keywords":["neural scaling laws","learning curve theory","random hierarchy model","Zipf distribution","probabilistic context-free grammars","classification","next-token prediction","hierarchical compositional data"],"falsifier":"Train a deep transformer on next-token RHM data (e.g. $s=2$, $v=32$, $m=8$, $L=4$) with $a=0.2$ and $a=4$, and compare the late slopes of $L(P)-L_\\infty(a)$ on a log-log plot: Eq. (12) says both approach $P^{-\\log(4/32)/(2\\log 8)}$, so any systematic slope separation would refute the claimed independence; similarly, classification curves should exhibit slopes $-a/(1+a)$ after the $v m^{L-1}$ rescaling.","tokens_in":17877,"feed_emoji":"📈","tokens_out":9305,"duration_ms":91653,"temperature":0.7,"pith_summary":"Neural scaling laws in real data are sometimes attributed to power-law-distributed features and sometimes to hierarchical compositional structure. This paper studies both at once by adding Zipf-distributed production rules to the Random Hierarchy Model, a tree-shaped probabilistic grammar. For root classification, it establishes a power-law learning curve, $\\varepsilon(P) \\sim (P/(v m^{L-1}))^{-a/(1+a)}$, whose exponent is fixed by the Zipf exponent $a$ and whose pre-asymptotic scale is fixed by the hierarchy. For next-token prediction, the paper shows the opposite: the Zipf exponent reshapes local features of the curve but not the asymptotic decay, which remains governed by the branching structure $m/v^{s-1}$. This matters because it separates two candidate explanations of real scaling laws, frequency statistics and hierarchical composition, and assigns them to different task types.","feed_headline":"Zipf sets classification exponent, not language scaling","feed_subtitle":"In a random hierarchical grammar, task type decides which structure controls learning-curve decay.","key_machinery":"The Random Hierarchy Model: a depth-$L$ context-free grammar in which each nonterminal has $m$ unambiguous production rules, one Zipf-distributed layer with probabilities $f_k \\propto k^{-(1+a)}$, and all other layers uniform. Learning is tracked through correlation functions $C_j(y,\\mu)$ for label-tuple correlations in classification and $C_j(\\mu,\\nu)$ for token-tuple correlations in next-token prediction; their variance over grammar realisations and over finite-sample noise determines when a rule's signal becomes detectable. Setting signal equal to noise yields per-rule sample complexities $P_k^*=v m^{L-1}/f_k$ for classification and $P_{\\ell,k}=v m^{2\\ell-3}(1-m/v^{s-1})f_k\\sum_{k'}f_{k'}^2$ for next-token steps. Combining these thresholds with the probabilities of resolved rules and with the average $s^\\ell$-gram cross-entropies produces the learning curves.","core_discovery":"The central discovery is that, within the Random Hierarchy Model with one layer of power-law rule frequencies $f_k \\propto k^{-(1+a)}$, the task determines which ingredient sets the scaling exponent. Classification errors obey $\\varepsilon(P)=1-(\\sum_{k:P_k^*<P} f_k)^{s^{L-1}}$, which for $P \\gg v m^{L-1}$ becomes $s^{L-1}(P/(v m^{L-1}))^{-a/(1+a)}$; the hierarchy contributes a large constant and delayed onset, not the exponent. Next-token prediction first passes through an $s$-gram memorisation stage with the Zipf exponent, but the long-distance reconstruction of the hidden tree follows sample complexities $P_{\\ell,k}=v m^{2\\ell-3}(1-m/v^{s-1})f_k(\\sum_{k'}f_{k'}^2)$, whose level-dependence is the same as in the uniform case, so the asymptotic excess loss decays as $P^{-\\log(m/v^{s-1})/(2\\log m)}$ independent of $a$. The paper thereby unifies the power-law-units view and the hierarchical-structure view by showing that both are right, each for a different task.","pith_inferences":["A testable extension: putting Zipf rules on two layers should make input probabilities products of Zipf factors; the classification exponent may then interpolate between $a/(1+a)$ and a combination of the layer exponents rather than the single-layer value.","An architectural prediction follows from Assumption 3.1: different optimizers could shift the $P$ thresholds at which rules are learned, changing prefactors and crossover locations while leaving the asymptotic exponents intact.","For real language corpora this implies a cross-corpus diagnostic: if late next-token slopes vary with token-frequency distribution at fixed branching structure, some ingredient beyond this tree hierarchy is active.","For real classification datasets, a long pre-asymptotic plateau can coexist with a strong final power law, so measuring only short-data performance would underestimate the asymptotic learning rate."],"forward_implications":["Classification error on Zipf-RHM data decays as $\\varepsilon(P)\\sim s^{L-1}(P/(v m^{L-1}))^{-a/(1+a)}$; larger $a$ yields faster decay, while $L$ and $m$ set the horizontal offset of the power law.","The sigmoidal learning curve of the uniform RHM becomes a power law once rule frequencies are uneven, so the same hierarchy can look qualitatively different under different feature statistics.","Next-token prediction retains its exponent $-\\log(m/v^{s-1})/(2\\log m)$ for all $a$, meaning that the long-range structure of language-like data, not token frequency, controls the late-time learning curve.","The first stage of next-token prediction is $s$-gram memorisation with exponent $a/(1+a)$; this stage is visible in early data but disappears from the asymptotics.","All classification curves collapse when $P$ is rescaled by $v m^{L-1}$, providing a direct experimental signature of the hierarchy's contribution."],"supporting_citations":[{"why":"Supplies the Zipf memorisation baseline $\\varepsilon(P)=\\sum_k f_k(1-f_k)^P$ and the exponent $a/(1+a)$ that the classification curves inherit.","marker":"(Hutter, 2021)"},{"why":"Introduced the Random Hierarchy Model and its uniform-rule correlation variances and sample complexity $P^*=v m^L$; the present work's nonuniform analysis extends these quantities.","marker":"(Cagnetta et al., 2024)"},{"why":"Derived the stepwise next-token reconstruction, the $s^\\ell$-gram losses, and the hierarchy-controlled exponent of Eq. (12) in the uniform case.","marker":"(Cagnetta & Wyart, 2024)"},{"why":"Quantifies the probability that the root is inferred without resolving all patches, justifying the $m=v^{s-1}$ regime where the classification argument is strict.","marker":"(Sclocchi et al., 2024)"}],"fun_headline_variants":["Task type picks which factor sets the scaling exponent","Zipf drives classification exponent; hierarchy runs language scaling","One grammar, two laws: task chooses the exponent controller","Hierarchy sets token scaling; Zipf sets classification scaling","Scaling law driver? Depends on the task, not just the data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation relies on Assumption 3.1, that a rule is learned exactly when its correlation signal becomes detectable in finite data, and, for classification in the strict regime, on the requirement that all $s^{L-1}$ patches be resolved, which is guaranteed only when $m=v^{s-1}$; neither premise is derived from gradient-descent dynamics.","fun_headline_variants_meta":{"raw":{"variants":["Task type picks which factor sets the scaling exponent","Zipf drives classification exponent; hierarchy runs language scaling","One grammar, two laws: task chooses the exponent controller","Hierarchy sets token scaling; Zipf sets classification scaling","Scaling law driver? Depends on the task, not just the data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1265,"prompt_tokens":921,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":537,"tokens_out":344,"duration_ms":4227,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:26:09.577384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a deep transformer on next-token RHM data (e.g. $s=2$, $v=32$, $m=8$, $L=4$) with $a=0.2$ and $a=4$, and compare the late slopes of $L(P)-L_\\infty(a)$ on a log-log plot: Eq. (12) says both approach $P^{-\\log(4/32)/(2\\log 8)}$, so any systematic slope separation would refute the claimed independence; similarly, classification curves should exhibit slopes $-a/(1+a)$ after the $v m^{L-1}$ rescaling.","supporting_citations":[],"review_version":1}