{"id":"971d6293-92bd-4ace-9114-e2d98f419384","arxiv_id":"2505.23013","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Controlling model complexity through smaller initialization rates and stronger weight decay improved LLM benchmark scores and made loss-versus-scale curves descend faster.","lead":"A training recipe that shrinks parameter initialization and increases weight decay consistently improved benchmark scores and loss scaling in language models up to 2.4B parameters. The result suggests a simple complexity-control knob for pretraining, not just a post-training reasoning trick.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scaling-law claim is extrapolated from 0.2–1.4B-token runs; without intermediate-scale multi-seed confirmation, the headline ordering at 600B–1T tokens is unverified.","rationale":"I found the paper's controlled 180M grid (Section 5.1) and the consistency of the direction across tasks in Table 4 and Table 6 to be genuine evidence that the proposed complexity control helps in small-scale settings. The theoretical discussion in Section 6.3 is explicitly heuristic and does not itself raise or lower confidence. The load-bearing issue is the extrapolation from undertrained 0.8B/1B-token runs to the production-scale benchmark gains. This is not an internal inconsistency in the paper; it is a gap between the evidence presented and the 'scaling law' statement in the title and abstract. The reader's weakest assumption points to the same gap, so I agree with the conditional verdict. I recommend no change to the reader's verdict: the claim should remain conditional pending intermediate-scale scaling experiments and clarification of the 2.4B model configuration.","tokens_in":16385,"tokens_out":9037,"duration_ms":86178,"concrete_test":"Train 0.9B models with the small-complexity (γ=1, λ=1) and large-complexity (γ=0.5, λ=0.1) configurations on 50B, 150B, and 300B tokens with at least three seeds per point; fit L(N) = a N^{-b} + c for each configuration and test whether the exponents differ significantly and whether the predicted loss at 600B tokens preserves the advantage shown in Table 1. If the exponent difference is not significant or reverses, the scaling-law claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper claims that complexity control improves the scaling law consistently over model and data sizes. The only direct evidence is Figure 2, which trains 0.8B models on 0.2–1.4B tokens and 50M–0.8B models on 1B tokens. This is a heavily undertrained regime; at the largest shown point the token-to-parameter ratio is below 2:1, while the headline results use ratios near 700:1 and 400:1. The figure shows one run per point with no error bars, and the difference in scaling slopes is assumed to persist across two orders of magnitude in data. The large-scale comparisons in Table 1 are single runs on different corpora (SlimPajama subsample for 0.9B and a Memory3-based 1T dataset for 2.4B), so they cannot be combined into a single scaling curve. The 2.4B small-complexity model uses γ=0.58 rather than the γ=1 used in Figure 2, and the paper reports adding embedding/sandwich normalization to the 2.4B variant, leaving the attribution of its gain to the proposed complexity control ambiguous. If the small-complexity advantage diminishes or reverses at intermediate scales not shown, the central 'improves scaling law' conclusion fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes that controlling model complexity during pretraining, implemented through the initialization rate γ and the weight decay coefficient λ, improves the scaling laws of large language models and enhances their reasoning ability. The authors train Llama-style models at several scales: 180M models on 40B tokens for analysis, 0.8B models on 0.2–1.4B tokens for scaling-law curves, 0.9B models on 600B tokens, and 2.4B models on 1T tokens. They report that small-complexity configurations (larger γ, larger λ) shift test-loss curves leftward and produce higher benchmark scores, with average gains of +4.6% and +3.4% over 15 tasks for the 0.9B and 2.4B comparisons, respectively. The paper also includes mechanistic analyses of embedding similarity and attention-matrix condensation, and a heuristic theoretical argument based on interpolated function norms and circuit-ensemble representations.","tokens_in":1613,"tokens_out":1574,"duration_ms":82134,"significance":"If the claims are substantiated, the paper offers a simple and computationally free intervention for improving LLM reasoning by tuning two hyperparameters, with no additional data or architecture changes. The reported gains on math and commonsense benchmarks are practically meaningful, and the focus on complexity control as a design principle is a useful complement to data- and scale-centric approaches. The authors have provided substantial experimental effort, including training a 2.4B model on 1T tokens, and the auxiliary analyses (embedding similarity, attention condensation) suggest a mechanism consistent with the condensation literature. However, as detailed below, the headline scaling-law conclusion currently rests on evidence that is too weak to fully support the paper's claims.","major_comments":[{"comment":"The scaling-law conclusion is extrapolated from a heavily undertrained regime: the 0.8B models are trained on at most 1.4B tokens, which gives a token-to-parameter ratio below 2:1, while the headline benchmark comparisons use roughly 600B tokens per 0.9B model and 1T tokens per 2.4B model. The figure shows single runs with no error bars, and the claimed difference in scaling slopes is assessed visually rather than fitted. Without intermediate-scale points or multiple seeds, the data do not establish that the ordering of test losses persists across two orders of magnitude in data, which is the basis for the abstract's claim of consistent improvement in scaling laws.","section":"4.1, Figure 2"},{"comment":"The 2.4B small-complexity model is not a clean comparison for complexity control: it uses gamma equal to 0.58 instead of the gamma equal to 1 used in Figure 2, and it additionally employs embedding normalization and sandwich normalization (stated in Appendix B and Section 6.1). These architectural changes were introduced to mitigate loss spikes, but they confound the attribution of the plus 3.4 average gain to complexity control alone. The authors should either isolate the effect of gamma while holding normalization fixed, or present a matched 2.4B model without the extra normalization.","section":"4.2, Table 1, and Appendix B"},{"comment":"All reported benchmark differences are based on a single training run per configuration, with no standard errors or significance tests. Several deltas are small (for example, MMLU-Pro minus 0.5, OpenBookQA plus 0.8, GPQA plus 0.9, and IFEval plus 0.7 for the 2.4B comparison) and could be within evaluation noise. The assertion that complexity control improves performance consistently across tasks needs either multiple seeds or a paired statistical analysis across the 15 benchmarks.","section":"Table 1 and Appendix C"},{"comment":"The theoretical analysis rests on two very strong assumptions: convergence to the minimum-gamma-norm minimizer, and decomposition of the parameter distribution into product measures with per-layer masses epsilon or 0.9. These assumptions are not validated empirically and are not shown to be necessary for the reported benchmark gains. As a heuristic the discussion is illustrative, but as written it may be misread as a mechanistic explanation. Please clarify that this is a plausibility argument rather than a derivation of the scaling-law results.","section":"6.3"}],"minor_comments":[{"comment":"The displayed weight-decay update rule appears to have a sign typo: it reads theta_{t+1} = -hat_theta_t - lambda * C * theta_t, but the standard AdamW update would be theta_{t+1} = hat_theta_t - lambda * C * theta_t. Please correct the sign.","section":"3, Eq. (1)"},{"comment":"The derivation of the minimized function norm is hard to follow because several intermediate displays are truncated or ambiguous, including the final expression for the norm with the factor 0.9 and epsilon. Please expand the calculation and define the norm notation precisely.","section":"6.3"},{"comment":"The table caption does not state that the 2.4B Small model uses gamma equal to 0.58 while the 0.9B Small model uses gamma equal to 1. Adding this information would prevent conflating the two configurations.","section":"Table 1 caption"},{"comment":"In the right panel, all models are trained on 1B tokens, so the token-to-parameter ratio varies from 20 for the 50M model to 1.25 for the 0.8B model. This confounds model-size scaling with data-to-model ratio and should be discussed.","section":"Figure 2 right"},{"comment":"The phrase that the initialization rate turns out to be the right invariant for the scaling laws is stronger than the evidence provided, since only a few discrete gamma values are tested. Please consider qualifying this claim.","section":"Abstract and Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a group with an extensive and relevant record on condensation and initialization, and the self-citations are appropriate though numerous. The key risk is overclaiming scaling-law consistency from small-scale curves. Given the computational cost of additional 2.4B runs, the authors might instead consider reframing the contribution as a demonstration at fixed scales, or providing multi-seed intermediate-scale curves (for example, 0.8B at 2 to 10B tokens) to empirically ground the extrapolation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First things first: the paper is worth referee time, but the headline claim—that complexity control via initialization rate γ and weight decay λ improves the scaling law of LLMs—is not actually demonstrated by the data shown. What is genuinely new: the paper moves the small-initialization idea from synthetic compositional tasks to natural-language pretraining at 0.9B and 2.4B scale, and reports sizable benchmark gains (GSM8K +19.4 at 0.9B, +11 at 2.4B; CommonsenseQA +9.5 and +6.5). The analysis of parameter norms, embedding similarity, and attention condensation is a real effort to open the black box, and the Appendix D discussion of loss spikes is honest. The theoretical interpolation-norm calculation is explicitly heuristic; it does not pretend to be a proof, and the empirical results do not depend on it. That last point is a credit: the paper would stand or fall on the experiments.\n\nThe problem is the experiments as presented are not yet solid enough for the scaling-law conclusion. Figure 2, the only direct scaling evidence, trains 0.8B models on 0.2–1.4B tokens—a ratio below 2:1 at the upper end, while the headline uses 600B tokens per 0.9B model and 1T per 2.4B. That is two orders of magnitude of extrapolation from a heavily undertrained regime, with one run per point and no error bars. The 2.4B small-complexity model uses γ=0.58 instead of the γ=1 in Figure 2, and adds embedding and sandwich normalization, so its gains cannot be attributed to complexity control alone. Several Table 1 deltas (+0.7, +0.8, -0.5) are within normal benchmark noise. No code or data are released. None of this means the effect is fake—the large gains on math tasks are hard to brush off—but it does mean the 'improves scaling law consistently' sentence overstates what the evidence supports.\n\nMy take: this is a plausible and potentially useful recipe, but it needs intermediate-scale multi-seed validation and a cleaner ablation at 2.4B before the scaling-law claim is accepted. I would send it to a serious referee, because the direction is important and the authors are clearly thinking carefully. A good referee will ask for those experiments, and the paper will be much stronger for them.","headline":"Plausible and valuable direction, but the scaling-law claim is not backed by the data as presented—send to review, demand better evidence.","tokens_in":17274,"tokens_out":2412,"would_cite":false,"duration_ms":24731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that controlling model complexity during pretraining, through the initialization rate and weight decay, improves scaling laws and reasoning benchmarks across model and data sizes.","keywords":["complexity control","initialization rate","weight decay","scaling law","LLM reasoning","pretraining","condensation","generalization"],"falsifier":"Measure the test-loss gap between the small- and large-complexity configurations on a 2.4B model trained on 10B or more tokens: if the gap closes as data grows, the leftward shift in the paper's scaling-law figure is not a durable scaling-law improvement.","tokens_in":16205,"feed_emoji":"🧠","tokens_out":7859,"duration_ms":76487,"temperature":0.7,"pith_summary":"The paper tries to establish that how much complexity a language model is allowed during pretraining is a primary lever on its reasoning ability. Concretely, initializing weights with a larger initialization rate (so smaller initial scales) and applying stronger weight decay makes test loss fall faster as either model size or data size grows. The authors back this with scaling-law curves on 0.8B models and with large-scale runs: a 0.9B model trained on 600B tokens gains 4.6 points on average over 15 benchmarks against a high-complexity baseline, and a 2.4B model trained on 1T tokens gains 3.4 points, with the largest jumps on math and commonsense reasoning. If true, these are essentially free gains: no new data, no new architecture, only a different choice of two standard hyperparameters.","feed_headline":"Lower model complexity lifts LLM scores by 4.6 points","feed_subtitle":"A smaller initialization scale and stronger weight decay improve reasoning benchmarks at 0.9B and 2.4B scale.","key_machinery":"The mechanism is a pair of hyperparameters: the initialization rate, defined by sampling each weight from a normal distribution with scale $d_{\\mathrm{in}}^{-\\gamma}$, and the weight decay coefficient $\\lambda$ in the optimizer update. A rate above $\\gamma = 0.5$ counts as small-complexity initialization, while $\\gamma = 0.5$ is the standard He/LeCun-style choice. The theoretical engine is an interpolated function norm $\\|f\\|_\\gamma$ that connects the kernel-regime norm at $\\gamma = -1/2$ with the Barron norm at $\\gamma = -1$; under the paper's idealization, minimizing this norm makes training prefer sparse deep circuits over dense shallow circuits. In the trained network this preference shows up as condensation (neurons collapsing into few groups), higher cosine similarity among embeddings, and larger condensation and low-rank metrics in attention projections.","core_discovery":"The paper's central claim is that model complexity, set by the initialization rate and weight decay, is a controllable proxy for whether a pretrained LLM learns compressed, generalizable rules or memorizes surface patterns. In the language of the paper, small-complexity pretraining steers training toward a circuit ensemble with few, deep circuits rather than many shallow ones, which is why it improves reasoning while also improving the scaling-law exponent. The empirical evidence has three parts: test-loss curves for 0.8B models shift left when complexity is reduced, in both the data-size and model-size directions; at scale, the small-complexity 0.9B/600B model beats the large-complexity model on 14 of 15 benchmarks and the 2.4B/1T model on 13 of 15, with average gains of +4.6 and +3.4 points; and mechanistic analyses show small complexity increases embedding similarity and condensation/low-rank structure in attention projections. The paper also argues, via a norm-interpolation calculation, that lower initialization rate favors exactly the sparse-deep solutions that generalize.","pith_inferences":["If the scaling-law ordering from 0.8B models holds beyond 10B parameters, choosing the initialization rate and weight decay may become standard pretraining practice, but that extrapolation goes beyond what the paper's single large runs can confirm.","The sparse-deep-circuits account predicts that small-complexity models should show measurable phase-transition or grokking-like behavior on small algorithmic tasks; this is testable with existing circuit-probing experiments.","Because the 2.4B small run used extra embedding and sandwich normalization, the cleanest reading is that complexity control plus stability engineering jointly produced the gain; the paper does not isolate the two components.","The authors point to post-training complexity control as future work; if it transfers, the same reasoning gains could be obtained after pretraining at even lower cost."],"forward_implications":["Higher initialization rate and weight decay improve sample efficiency: the same test loss is reached with fewer tokens or fewer parameters.","A constant initialization rate, rather than a fixed initialization standard deviation, is the right scaling prescription, so the recipe can transfer across model widths.","Complexity control stacks with supervised fine-tuning: the reported SFT gains are larger on small-complexity base models.","The gains concentrate on reasoning-heavy tasks such as GSM8K, MATH, and CommonsenseQA, so complexity control can be treated as a reasoning-oriented pretraining intervention.","Realizing the gains at scale requires stabilizing training with a moderate rate and additional normalization, so the recipe must be adapted as model size grows."],"supporting_citations":[{"why":"Supplies the condensation phase diagram and initialization-rate framing from which the paper's complexity control is drawn.","marker":"[39]"},{"why":"Extends the phase diagram to deeper networks, supporting the claim that the rate, not the fixed standard deviation, is the right invariant.","marker":"[88]"},{"why":"Shows on synthetic compositional tasks that complexity control makes transformers reason rather than memorize, a direct precursor to this paper's natural-language claim.","marker":"[84]"},{"why":"Establishes that initialization determines whether transformers fit composite functions by reasoning or memorization, a load-bearing prior for the mechanism.","marker":"[83]"},{"why":"Links small initialization to reasoning bias in language models, supporting the interpretation of the benchmark gains.","marker":"[73]"},{"why":"Defines the LLaMA-style architecture used for all pretrained models in the study.","marker":"[58]"},{"why":"Provides the 600B-token corpus on which the 0.9B models are trained.","marker":"[53]"}],"fun_headline_variants":["Taming model complexity lifts LLM reasoning by 4.6 points","Lower LLM complexity sharpens reasoning and scaling laws","Smaller init scale, stronger decay: LLM reasoning jumps +4.6","Why low-complexity LLMs reason better: scaling law insight"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assumption that test-loss rankings seen on small 0.8B models trained on at most 1.4B tokens still hold at 600B and 1T token scales, and that the 2.4B small-complexity gain is not caused by the extra normalization used only in that run.","fun_headline_variants_meta":{"raw":{"variants":["Taming model complexity lifts LLM reasoning by 4.6 points","Lower LLM complexity sharpens reasoning and scaling laws","Smaller init scale, stronger decay: LLM reasoning jumps +4.6","Why low-complexity LLMs reason better: scaling law insight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000447,"raw_usage":{"total_tokens":2232,"prompt_tokens":894,"completion_tokens":1338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1262}},"tokens_in":510,"tokens_out":1338,"duration_ms":10905,"temperature":1.0,"reasoning_tokens":1262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:54:51.465863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the test-loss gap between the small- and large-complexity configurations on a 2.4B model trained on 10B or more tokens: if the gap closes as data grows, the leftward shift in the paper's scaling-law figure is not a durable scaling-law improvement.","supporting_citations":[{"cited_title":"Phase diagram for two-layer relu neural networks at infinite-width limit","cited_arxiv_id":null,"evidence_quote":"Supplies the condensation phase diagram and initialization-rate framing from which the paper's complexity control is drawn."},{"cited_title":"Empirical phase diagram for three-layer neural networks with infinite width","cited_arxiv_id":null,"evidence_quote":"Extends the phase diagram to deeper networks, supporting the claim that the rate, not the fixed standard deviation, is the right invariant."},{"cited_title":"Initial- ization is critical to whether transformers fit composite functions by reasoning or memorizing","cited_arxiv_id":null,"evidence_quote":"Establishes that initialization determines whether transformers fit composite functions by reasoning or memorization, a load-bearing prior for the mechanism."},{"cited_title":"SlimPajama: A 627B token cleaned and deduplicated version of RedPajama","cited_arxiv_id":null,"evidence_quote":"Provides the 600B-token corpus on which the 0.9B models are trained."}],"review_version":1}