{"id":"28ec904e-b080-43cb-9c17-206bba3d2d71","arxiv_id":"2505.07070","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Convolutional networks trained on a random hierarchical grammar improve twice as fast with data as transformers, because weight sharing reuses the statistical signal across all positions.","lead":"Neural networks trained on a synthetic hierarchical language learn its grammar in stages, and this paper predicts that convolutional networks reach the final performance level faster than transformers. The reason is that weight sharing lets a CNN reuse one correlation signal across all positions, yielding twice the scaling exponent of attention-based models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CNN exponent in Eq. 24 rests on an unproven weight-sharing transfer premise (Sec. V.C.3), and the empirical support is visual rather than quantitative; a direct probe of latent decodability versus P would settle it.","rationale":"I agree with the reader that the weakest point is the bridge from correlation detectability to learned hidden structure, and specifically the transfer step in Sec. V.C.3. This is the step that produces the factor-two exponent improvement, so it is genuinely load-bearing. The paper is transparent that this is a working assumption, and the empirical curves, while suggestive, are not fitted with uncertainties; the LCN comparison provides only indirect support. For these reasons the conditional verdict is appropriate. I would not reject: the mechanism is coherent, the algebra connecting Eq. 16 to Eq. 24 is clean, and a direct probe can test the transfer premise. No new concern beyond the reader's is identified.","tokens_in":16441,"tokens_out":7493,"duration_ms":81296,"concrete_test":"Train a depth-4 CNN on RHM data with L=4, s=2, v=24, m=6. At training steps spanning P1=(1-f)^(-1) v m^3 and the later thresholds, use linear probes on the first hidden layer (or the Sec. VIII invariance metric) to measure decoding accuracy for each level-4 latent variable at every position, averaged over at least 8 RHM realizations with standard errors. The weight-sharing transfer premise predicts that all level-4 latents become decodable at the same P, approximately P1; if positions farther from the last token lag measurably, Eq. 24's Pbar_l and the exponent log f / log m lose their stated mechanism. This directly tests the assumption that the CNN reuses a correlation learned at one position.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.C.3 asserts, rather than derives, the premise behind the CNN scaling law: because RHM production rules are position-independent, a weight-shared CNN that learns to infer the easiest level-L latent (the sibling of the last token) at P >> P1 can infer all level-L latents, giving Pbar_l = (1-f)^(-1) v m^(l+2) and Eq. 24. The correlation calculation in Eqs. 12-16 only establishes when a correlation exceeds sampling noise; it does not establish that gradient descent on a deep CNN implements this transfer, and Assumption V.1 itself is explicitly an assumption, not a consequence of training dynamics. Eq. 24 is therefore a heuristic prediction, and the authors' own conclusion (Sec. X) states that a formal characterization of training dynamics is missing. The empirical validation in Figs. 4-5 compares solid learning curves to reference power-law lines by eye, without error bars, fitted exponents, or confidence intervals, and the LCN experiment in Fig. 8 uses a single parameter set. If the transfer premise fails or holds only approximately, the doubled CNN exponent does not follow from the stated mechanism, even if the empirical curves are faster for some other reason.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies last-token prediction on synthetic sequences generated by the Random Hierarchy Model (RHM), an ensemble of random context-free grammars with a fixed tree topology. It derives threshold training-set sizes at which correlations between the last token and hidden-variable spans become detectable, giving P_l ~ v m^{2l+1} for transformers and an improved Pbar_l ~ v m^{l+2} for CNNs under a weight-sharing transfer assumption. Combining these thresholds with the previously derived asymptotic relation L_l - L_infinity ~ f^l yields predicted test-loss exponents log f/(2 log m) for transformers and log f/log m for CNNs. Empirically, depth-4 transformers, CNNs, and locally connected networks are trained online on RHM data, and hidden-representation probes measure invariance to resampling hidden symbols and production rules. The central claim is that architecture changes the scaling exponent on hierarchically compositional data, with CNNs enjoying a faster decay because weight sharing lets a detected correlation be reused across positions.","tokens_in":16765,"tokens_out":3846,"duration_ms":40551,"significance":"If the central claim holds, the paper provides one of the few controlled demonstrations that architectural inductive bias changes the exponent of a neural scaling law, not merely the prefactor. The predictions are falsifiable and the exponents are not fit to the empirical curves: they are derived from RHM parameters and compared at independent (v,m) settings. The paper also includes useful controls, including an LCN ablation separating locality from weight sharing, a comparison of Adam versus SGD for transformers, and representation-dynamics probes. The main caveat is that the CNN exponent rests on an explicit, unproven assumption about weight-sharing transfer, so the theoretical result is a heuristic prediction rather than a consequence of training dynamics. The empirical validation is currently visual rather than quantitative, which weakens the claim that the factor-of-two exponent difference is real.","major_comments":[{"comment":"The CNN scaling law is not derived. Section V.C.3 asserts that weight sharing lets a learner infer all level-L latents once the easiest one is detectable, leading to Pbar_l in Eq. (16) and the exponent in Eq. (24), but no argument from gradient dynamics is supplied. Assumption V.1 is likewise stated as an assumption, and the conclusions (Section X) explicitly note that a formal characterization of training dynamics is missing. Since Eq. (24) is the central new theoretical result, this is a load-bearing gap. The authors should either derive the transfer mechanism in a tractable limit or test it directly, for example by training linear probes on intermediate activations and measuring decoding accuracy for all level-L latents as a function of P, to check that CNNs acquire them near Pbar_l while transformers acquire them near P_l.","section":"V.C.3 and Eq. (24)"},{"comment":"The empirical support for the central exponent difference is visual rather than quantitative. Solid learning curves are compared with reference power-law lines by eye; no fitted exponents, confidence intervals, or error bars are reported. In Fig. 4, the claim that the two transformer curves 'display the same decay' is supported only by rescaling one x-axis by a constant factor. Please report fitted slopes of log(L-L_infinity) versus log P with standard errors for each (v,m) setting, together with a statistical comparison of the CNN slope against both Eq. 24 and Eq. 23. This is necessary to establish that the exponent difference is real and not a prefactor or finite-range artifact.","section":"Section VII, Figs. 4 and 5"},{"comment":"The detectability threshold is derived by comparing the finite-sample noise variance sigma_P^2 in Eq. (12) with the RHM correlation variance in Eq. (13), but the condition is only 'P >> threshold' and no quantitative scale for the crossover is given. The conversion of these thresholds into a clean power law for the test loss assumes sharp stage transitions, and the transition width could affect the observed exponent, especially over the limited range of training steps in Figs. 3 and 5. Please characterize the finite-P crossover or show that plausible corrections do not change the predicted slope.","section":"Section V.C.2, Eqs. (12)-(16)"},{"comment":"The LCN experiment is the key ablation isolating weight sharing, but it uses a single parameter set (L=4, s=2, v=24, m=6) and is again evaluated by visual overlap after manual shifts. This makes it difficult to rule out that the 'shared asymptotic behavior' of LCN and transformer is specific to this parameter choice. Reporting fitted exponents for LCN at least two (v,m) settings, or providing the same quantitative analysis as requested for Figs. 4-5, would substantially strengthen the weight-sharing interpretation.","section":"Section IX and Fig. 8"}],"minor_comments":[{"comment":"The subsection heading contains a typo: 'Reconsturction' should be 'Reconstruction'.","section":"Section V.C.1"},{"comment":"The abbreviation for context-free grammars is introduced as 'CGFs' in the text, but the standard and previously used abbreviation is CFGs; please correct this for consistency.","section":"Section III.A"},{"comment":"The caption contains a formatting error: 'textbfLeft' should be 'Left' with the appropriate boldface applied in the rendered figure.","section":"Fig. 8 caption"},{"comment":"The definition of f is garbled: 'f := (m/vs−1)' is not typeset correctly and the constraint mv <= vs is written without explanation; please clarify the definition and the admissible range of parameters.","section":"Section III.B, constraint C4"},{"comment":"In the conditional probability notation, X_{-1} appears both as the target and in the condition via X_{-2:d}; please make the conditioning set explicit (e.g., X_{-1} | X_{-2}=x_{-2}, ..., X_{-d}=x_{-d}) to avoid ambiguity.","section":"Section V.A, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the architectural comparison, and the core statistical machinery for correlations is coherent. My main concern is that the CNN exponent, the headline result, rests on an explicit assumption about weight-sharing transfer rather than a derivation, and the empirical evidence for the factor-of-two exponent difference is currently visual. The paper also relies heavily on the authors' previous work for key asymptotic results (Eqs. 5-6, 22-23); this is not disqualifying because those are prior derivations and the new contribution is the architectural extension, but independent verification of the inherited asymptotics would increase confidence. With quantitative fits and a direct test of the weight-sharing transfer premise, the paper would be a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper makes a concrete, falsifiable prediction—on Random Hierarchy Model data, tree-aligned CNNs should have test-loss scaling L(P)−L∞ ∼ P^(log f / log m), while transformers scale as P^(log f / 2 log m)—and shows simulations consistent with it. The factor-of-two difference is new and is derived from the architecture's weight sharing, not fit to the data. That is a genuinely useful contribution to the scaling-laws subfield: it says architecture changes the exponent, not just the prefactor, and it tells you why. What the paper does well: the correlation algebra is coherent and builds honestly on prior work, the prediction has no free parameters, and the simulations cover three (v, m) settings plus an LCN control that isolates weight sharing. The representation-probing experiments (Figs. 6–7) give independent evidence for the staged learning picture. The exposition is clear, and the authors explicitly flag where a formal treatment is missing. Where I have doubts. The load-bearing step is Assumption V.1 in Sec. V.C.2, which says a hidden variable becomes learnable exactly when its correlation with the last token exceeds sampling noise. That is a plausible heuristic, but it is not derived from gradient descent. More worryingly, the CNN improvement in Sec. V.C.3 is asserted: the paper says a weight-shared CNN that infers the easiest level-L latent can infer all of them, giving Pbar_l = (1−f)^{-1} v m^{l+2}, but this transfer premise is not proven. If it fails, the doubled exponent in Eq. 24 does not follow from the mechanism. The authors themselves note (Sec. X) that a formal characterization of training dynamics is missing. The empirical support is also weaker than the text implies. The agreement in Figs. 3–5 is visual: solid curves next to dashed reference lines, with no error bars, no fitted exponents, and no confidence intervals. The LCN experiment uses one parameter set. For a paper whose central claim is an exponent, that is thin. A quantitative fit of the empirical exponents, or a direct probe of latent decodability versus P, would settle it. That said, I do not think this should be rejected. The mechanism is plausible, the prediction is sharp and testable, and the paper is an honest continuation of a serious program. The right move is to send it to peer review and ask for a rigorous quantitative comparison and a derivation or stronger justification of the transfer premise. If those hold up, this will be a useful result; if not, the paper still contains a clean prediction worth having on record.","headline":"A clean, parameter-free prediction that weight sharing doubles the scaling exponent on hierarchical data, let down by an underived bridge assumption and by-eye empirical validation—worth a serious referee.","tokens_in":760,"tokens_out":860,"would_cite":true,"duration_ms":23981,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a tractable family of hierarchical grammars, tree-aligned convolutional networks achieve a loss-scaling exponent twice that of transformers, with weight sharing as the mechanism.","keywords":["neural scaling laws","Random Hierarchy Model","probabilistic context-free grammars","representation learning","transformers","convolutional networks","weight sharing","last-token prediction"],"falsifier":"On RHM data with $s=2$, $L=4$, $v=24$, $m=6$, the theory places the transformer's second-stage threshold near $P_2 = (1-f)^{-1} v m^5$ and the CNN's near $\\bar P_2 = (1-f)^{-1} v m^4$; measuring the actual training-step counts at which the two excess-loss curves cross each $\\ell$-gram plateau would confirm or refute the predicted gap, and making the production rules position-dependent should eliminate the CNN advantage if weight sharing is the cause.","tokens_in":16236,"feed_emoji":"📉","tokens_out":11505,"duration_ms":113674,"temperature":0.7,"pith_summary":"On data from the Random Hierarchy Model—an ensemble of probabilistic context-free grammars on a fixed regular tree—this paper claims that architecture changes the neural scaling exponent. A convolutional network whose filters are aligned with the tree's branching structure reaches a test-loss decay $L(P)-L_\\infty \\sim P^{\\log f / \\log m}$, while a transformer on identical data decays as $P^{\\log f / (2\\log m)}$, a factor of two in the exponent. The mechanism is weight sharing: a convolution reuses a detected correlation across all positions, so it can infer all latent variables at one level as soon as the easiest one is detectable. If the claim is right, scaling laws on compositional data depend on the match between architecture and the data's hierarchical statistics, not only on data and model size.","feed_headline":"Weight sharing doubles the loss-scaling exponent on hierarchical data","feed_subtitle":"On grammar data, a convolutional net hits a target loss with about the square root of the samples transformers need.","key_machinery":"The Random Hierarchy Model (RHM)—an ensemble of probabilistic context-free grammars on a fixed regular tree of arity $s$ and depth $L$, with $v$ symbols per level and $m$ equiprobable production rules per symbol, so that $f = m/v^{s-1}$—is the central object. The argument runs on two scales: the variance of token correlations over RHM realisations decays exponentially with the tree distance between tokens, while a finite training set of size $P$ contributes correlation noise of order $1/P$. Assumption V.1 identifies learning a hidden layer with the crossing of those two scales, and the weight-sharing property of convolutional filters replaces the transformer threshold $P_\\ell$ with the smaller $\\bar P_\\ell$. The final exponent then comes from the identity $L_\\ell - L_\\infty \\sim f^\\ell$ combined with the relation between $\\ell$ and $P$ at each threshold.","core_discovery":"The paper's central claim is that learning the Random Hierarchy Model for last-token prediction proceeds stagewise, with each stage corresponding to correlations at a deeper level of the generative tree, and that the sample-complexity threshold for each stage is set by when the relevant correlation rises above finite-sample noise. For transformers the threshold is $P_\\ell = (1-f)^{-1} v m^{2\\ell+1}$, because the correlation between the last token and an observable $s$-tuple at tree distance $2\\ell+1$ must be detected. For a CNN with filter size and stride equal to the branching factor $s$, weight sharing turns detection of the single easiest tuple into knowledge of all same-level latent variables, lowering the threshold to $\\bar P_\\ell = (1-f)^{-1} v m^{\\ell+2}$. Since the excess loss after stage $\\ell$ is $L_\\ell - L_\\infty \\sim f^\\ell$, these thresholds yield $L(P)-L_\\infty \\sim P^{\\log f/(2\\log m)}$ for transformers and $P^{\\log f/\\log m}$ for CNNs. The paper validates both predictions in online-training experiments and reports that locally connected networks, which keep locality but remove weight sharing, follow the transformer's scaling after the first two stages.","pith_inferences":["An untested but direct extension: if RHM production rules are made position-dependent so that the same latent variable is not reused across branches, the CNN's weight-sharing advantage should shrink or vanish, and the transformer may match or beat it.","The mechanism suggests that on real language-like data, any parameter-sharing scheme that respects the grammar's rule reuse—such as tied attention weights or structured state-space models—could inherit part of the CNN's improved exponent, not just convolutions.","Because the theory identifies detectability thresholds, it predicts that probing the hidden representations during training should show the CNN becoming invariant to deeper hidden-variable replacements at sample counts of order $v m^{\\ell+2}$, a measurement that could be run on the published architecture."],"forward_implications":["On RHM data, to reach a fixed excess loss a transformer needs roughly the square of the number of samples a tree-aligned CNN needs.","The loss curve remains stagewise for both architectures, but the stage boundaries for CNNs occur systematically earlier in training.","Locality without weight sharing is not enough: locally connected networks inherit the CNN's early stages but revert to transformer scaling at later stages, isolating weight sharing as the cause.","Scaling-law exponents on compositional data are architecture-dependent, so comparing scaling exponents across model families requires controlling the match between architecture and data statistics."],"supporting_citations":[{"why":"Introduces the Random Hierarchy Model and supplies the correlation-variance estimates and the excess-loss behavior $L_\\ell - L_\\infty \\sim f^\\ell$ that the scaling predictions inherit.","marker":"[8]"},{"why":"Develops the correlation-based theory of last-token prediction on RHM data, including the finite-sample noise formula and the transformer sample complexities $P_\\ell$.","marker":"[10]"},{"why":"Derives the improved sample complexities $\\bar P_\\ell$ from translation invariance that the paper uses to predict the CNN scaling.","marker":"[14]"},{"why":"Provides the belief-propagation formalism used to compute exact posterior and marginal statistics for the empirical $s^\\ell$-gram loss baselines.","marker":"[55]"},{"why":"Defines the transformer architecture with multi-head self-attention used in the experiments and compared against the CNNs.","marker":"[63]"}],"fun_headline_variants":["Weight sharing doubles scaling exponent on grammar data","CNN weight sharing beats transformer scaling on tree data","CNNs achieve transformer loss with square-root sample size on grammar data","Hierarchical data favors CNNs: weight sharing squares sample efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a learner discovers a layer of hidden structure exactly when the correlation between the last token and a group of s tokens at the corresponding tree distance becomes detectable above finite-sample noise, and that a weight-sharing network can then reuse that detection at every position; neither is derived from the gradient-descent dynamics.","fun_headline_variants_meta":{"raw":{"variants":["Weight sharing doubles scaling exponent on grammar data","CNN weight sharing beats transformer scaling on tree data","CNNs achieve transformer loss with square-root sample size on grammar data","Hierarchical data favors CNNs: weight sharing squares sample efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000951,"raw_usage":{"total_tokens":4072,"prompt_tokens":977,"completion_tokens":3095,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":3030}},"tokens_in":593,"tokens_out":3095,"duration_ms":22859,"temperature":1.0,"reasoning_tokens":3030,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:26:08.938617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On RHM data with $s=2$, $L=4$, $v=24$, $m=6$, the theory places the transformer's second-stage threshold near $P_2 = (1-f)^{-1} v m^5$ and the CNN's near $\\bar P_2 = (1-f)^{-1} v m^4$; measuring the actual training-step counts at which the two excess-loss curves cross each $\\ell$-gram plateau would confirm or refute the predicted gap, and making the production rules position-dependent should eliminate the CNN advantage if weight sharing is the cause.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Random Hierarchy Model and supplies the correlation-variance estimates and the excess-loss behavior $L_\\ell - L_\\infty \\sim f^\\ell$ that the scaling predictions inherit."},{"cited_title":"Cagnetta and M","cited_arxiv_id":null,"evidence_quote":"Derives the improved sample complexities $\\bar P_\\ell$ from translation invariance that the paper uses to predict the CNN scaling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the belief-propagation formalism used to compute exact posterior and marginal statistics for the empirical $s^\\ell$-gram loss baselines."},{"cited_title":"Radford, K","cited_arxiv_id":null,"evidence_quote":"Defines the transformer architecture with multi-head self-attention used in the experiments and compared against the CNNs."}],"review_version":1}