{"id":"53e3ba00-9038-4865-b91e-264bc6188be1","arxiv_id":"2608.11859","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"With about 256 hyperparameter configurations per scale, scaling laws emerge at 4M parameters, and the apparent small-scale unreliability is largely a hyperparameter-tuning artifact.","lead":"This paper shows that scaling laws do hold at 4M parameters, but only when researchers search hundreds of hyperparameter configurations; careful tuning, not model size, is the deciding factor. It also proposes a diagnostic methodology for small-scale model comparison, demonstrated by recovering the known advantage of pre-norm transformers from tiny models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 5's accurate-law claim is untested because sampling effort per scale is not matched: roughly 900 configurations at 4M–34M vs 64–128 at held-out scales, so best-of-N bias can fake the extrapolation.","rationale":"The paper's central practical claim is that rigorous hyperparameter tuning, not model size, makes scaling laws observable at small scales and that the tuned frontier extrapolates. That claim rests on Figure 5. The figure's evaluation is not matched in sampling effort: fit points come from sweeps of roughly 826–934 configurations at 4M–34M, while held-out points use the best of 128 or 64 configurations at 67M–268M. Because best-of-N systematically overestimates the true optimum and the bias decreases with N, the small-scale fit can be too optimistic and the large-scale evaluation too pessimistic; the two biases can cancel and produce a small test MSE even when the true frontier does not extrapolate. This is a concrete, testable threat to the strongest claim. The reader's gamma-threshold concern is also valid and should be tested by holding the tail fraction fixed across scales, but it only weakens the geometric explanation; the matched-effort issue is prior and more damaging. If the matched-effort reanalysis fails, the paper's headline overstates what Figure 5 shows; if it passes, the conditional acceptance remains appropriate. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":36409,"tokens_out":16803,"duration_ms":181467,"concrete_test":"Run a matched-effort reanalysis: draw, or augment to, exactly 256 configurations at every scale (4M, 8M, 17M, 34M, 67M, 134M, 268M). Fit Eq. 3 to the best validation loss per parameter–data budget on 4M–34M, and evaluate test MSE against the best-of-256 loss at 67M–268M. If MSE stays near the roughly 1e-5 level of Fig. 5, the finding stands; if it degrades by an order of magnitude, the headline accuracy is an artifact of comparing approximately 256-draw small-scale minima with 64–128-draw large-scale minima. A cheaper intermediate check: correct the observed frontier to a common N=256 using the expected minimum of the fitted noisy quadratic at each scale, then refit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Figure 5's central demonstration does not hold the number of configurations per scale fixed. In the ladder, the random search used roughly 826–934 configurations at 4M–34M, 128 at 67M and 134M, and only 64 at 268M (§A.2); the caption says every panel—including the 4- and 16-configuration panels—is evaluated against 'the best configurations from our entire sweep' at the held-out scales. The best-of-N loss is a downward-biased estimate of the fully tuned frontier, and the bias shrinks as N grows. The 256-configuration law is therefore fit to minima from much larger sweeps at small scales and validated against minima from smaller sweeps at large scales; unequal sampling effort, rather than a true transferable frontier, could produce the apparent test accuracy. The reader's threshold/gamma concern is real but secondary: it affects the 'why', whereas this affects the quantitative claim that 256 configurations per scale yields an accurate extrapolating law.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that scaling laws for language models are observable at parameter counts as low as 4M parameters, provided one estimates the fully tuned hyperparameter frontier, and that the main obstacle to small-scale scaling laws is hyperparameter sensitivity, which decreases with scale. The authors support this with random-search sweeps across scales (4M–268M) under a hand-crafted model ladder and randomly sampled architectures, ablating methodological choices, and estimating the effective number of hyperparameters γ via the noisy quadratic limit. They then propose a holistic methodology for small-scale model-centric research and demonstrate it on the pre-norm vs. post-norm transformer question, claiming that small-scale experiments recover the large-scale preference for pre-norm. The central empirical results are the direct sensitivity measurements (Figures 6 and 7) and the observation that scaling laws appear only when sufficiently many configurations are used to define the tuned frontier.","tokens_in":36684,"tokens_out":7782,"duration_ms":75414,"significance":"If the claims hold, this is a practically important contribution: it offers a concrete explanation for the 'scaling gap' and a recipe for making small-scale experiments more reliable. The experimental effort is substantial—hundreds of configurations per scale, held-out scale partitions, detailed appendices on search distributions and fitting details, and use of open tools—and the paper is unusually transparent about its limitations. The direct sensitivity measurements in Figures 6 and 7 are convincing and do not depend on the more fragile γ estimator. However, three load-bearing parts currently need work: the matched-sampling confound in Figure 5, the threshold-dependence of the γ estimator in Section 4.2, and the reliance on a shared irreducible-error assumption in the case study's headline conclusion. These are fixable, but they prevent full acceptance as written.","major_comments":[{"comment":"The panel comparison in Figure 5 does not hold the number of configurations per scale fixed, and the caption states that all panels evaluate on the best configurations from the entire sweep. According to §A.2, the ladder sweep uses 826–934 configurations at the 4M–34M training scales but only 128 at 67M–134M and 64 at 268M. Because the minimum over a random search is a downward-biased estimator of the fully tuned frontier and the bias decreases with N, the apparent improvement from 4 to 256 configurations is confounded with unequal sampling effort across fitting and evaluation scales. The specific claim that 256 configurations per scale yield an accurate extrapolating law therefore needs either a matched-N evaluation (e.g., subsampling exactly 256 configurations at every scale for both fitting and evaluation) or an explicit best-of-N correction.","section":"§3.2, Figure 5, §A.2"},{"comment":"The estimated effective hyperparameter count γ is obtained by fitting the noisy quadratic distribution to the tail of the random-search score distribution, with the threshold θ growing from 7% at 4M parameters to 50% at 134M/268M parameters (§A.3). Since γ is a fitted shape parameter of that tail, the systematic widening of the tail with scale can by itself produce a decreasing γ even if the intrinsic dimension is constant. Moreover, γ is constrained to be at least 1, and Figure 8 shows the estimate saturating at that boundary. The claim that the hyperparameter loss surface becomes lower dimensional is the paper's proposed mechanism for why tuning gets easier, so it needs a sensitivity analysis with a fixed threshold across scales, or a threshold-independent estimator, before this geometric explanation can be considered established. This concern does not affect the direct sensitivity measurements in Figures 6 and 7, which support the empirical claim that tuning gets easier with scale.","section":"§4.2, Figures 8–9, §A.3"},{"comment":"The case-study conclusion that pre-norm is better at scale is obtained under the joint fit that ties the irreducible error ϵ across the two architectures. Figure 15 shows that with free ϵ, post-norm becomes better beyond the tested range, and the manuscript offers no empirical evidence that the two architectures share the same asymptotic loss. The abstract's statement that small-scale experiments 'recover the large scale result: pre-normalization works better as models grow in size' is therefore stronger than what the scaling-law comparison supports. The authors should either soften the claim, or provide a principled justification or test of the shared-ϵ assumption (e.g., longer training runs or larger-scale post-norm runs that approach the same floor).","section":"§5.3, Figure 15"}],"minor_comments":[{"comment":"There is a typo in the sentence 'In the ladder and random / prenorm) experiments' with an unmatched parenthesis; please fix.","section":"§A.3"},{"comment":"The caption should state explicitly how many configurations are used at each scale for fitting and for evaluation; the current wording 'best configurations from our entire sweep' obscures the unequal sample sizes across scales.","section":"Figure 5 caption"},{"comment":"Figure 8 would benefit from uncertainty quantification for γ (e.g., bootstrap confidence intervals), since the estimate is used to support a trend down to 1 and the fitting procedure involves a boundary constraint.","section":"Figure 8"},{"comment":"Figure 12 compares pre-norm (128 configurations per scale) with post-norm (511–512 configurations at 4M and 34M) without stating the sample-size difference in the caption; the density comparison should be labeled as unmatched to avoid overinterpretation.","section":"Figure 12"},{"comment":"The claim that the noisy quadratic limit 'confirms the hyperparameters are well-tuned' is somewhat strong for post-norm, where the asymptotic regime is only 3–10% of runs; consider softening the wording to reflect the smaller asymptotic regime.","section":"§5.3, Diagnostic 1"},{"comment":"The notation H_{x^*} is used for the Hessian in Equation 1, while 'H' informally denotes hyperparameters elsewhere; distinct notation would improve clarity.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper leans heavily on the authors' own noisy-quadratic tool (opda) both to estimate γ and to verify tuning quality; an independent implementation or at least a threshold-sensitivity analysis would strengthen the geometric claims. The Figure 5 sampling-effort confound and the shared-ϵ assumption in the case study should be addressed before acceptance. If the authors can fix these, the paper would be a solid empirical contribution to the small-scale scaling-law literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read, and worth a serious referee. The paper's central result is real: at 4M parameters the scaling law does not appear until you search hundreds of configurations, and the law that emerges from the well-tuned frontier extrapolates reasonably to 134M. That is a practical, actionable message for anyone doing small-scale pretraining research. The ablation design is thorough, the appendix documents search distributions and hardware in enough detail to reproduce, and the authors are honest about the statistical limits of extrapolation (§5.2). I especially like the synthesis of the noisy quadratic limit, scaling laws, and loss–capability correspondence into a diagnostic checklist; that part is a genuinely useful contribution.\n\nSoft spots, in proportion. First, the stress-test note about Figure 5 has teeth. The training scales (4M–34M) used roughly 850–930 random configurations; the held-out scales used 128 at 67M/134M and only 64 at 268M. The val/test points are the minima from those smaller sweeps, and all panels evaluate against the best configurations from the entire sweep. So the “256 configurations gives an accurate law” claim is validated against minima from different N. Since best-of-N loss is upward-biased and the bias shrinks as N grows, the apparent match could partly reflect the bias pattern rather than a true transferable frontier. This does not break the qualitative point—tuning clearly matters—but the precise number “256” is not as rigorously established as it sounds. The authors should redo the evaluation holding N fixed at every scale or show the law is stable across subsamples.\n\nSecond, the γ trend (effective number of hyperparameters dropping to 1) relies on fitting a noisy quadratic with tail thresholds that vary from 7% at 4M to 50% at 268M. The paper does not test sensitivity to that choice, so the geometric interpretation is plausible but not nailed down. The reader's concern is real, though secondary: the tuning-matters claim survives.\n\nThird, the case study's headline recovery of pre-norm over post-norm depends on assuming both architectures share the same irreducible error. The authors state this explicitly in §5.3, but the abstract does not, so an over-credulous reader could walk away with a stronger conclusion than the evidence supports. Minor: no code or data shipped, which limits reproducibility given the cost of these sweeps.\n\nBottom line: send to peer review. The central empirical finding is likely to hold, and the methodological synthesis is worth disseminating. The referee should push on the matched-sampling issue and the threshold sensitivity, but those are fixable with additional analysis.","headline":"A genuinely useful empirical finding—hyperparameter tuning, not scale, is what hides small-scale scaling laws—but the flagship 256-configuration claim is tested with unequal sampling effort across scales, so treat the exact number with caution.","tokens_in":37177,"tokens_out":6050,"would_cite":true,"duration_ms":60452,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that scaling laws exist at 4M parameters, and that rigorous hyperparameter tuning—hundreds of random configurations per scale—is what reveals them, with the effective number of hyperparameters falling to one as models…","keywords":["scaling laws","small-scale experiments","hyperparameter tuning","noisy quadratic limit","effective hyperparameters","hyperparameter loss surface","pre-norm vs post-norm","language model pretraining"],"falsifier":"Recompute the effective hyperparameter count γ while holding the asymptotic-regime threshold fixed at, say, 10% of runs at every scale; if the estimate no longer falls to one, the lower-dimensionality claim fails. Alternatively, repeat the γ sweep under a maximal-update parameterization that keeps optimal hyperparameters scale-invariant; if γ no longer decreases, the result is an artifact of the search distribution.","tokens_in":36222,"feed_emoji":"🔬","tokens_out":5484,"duration_ms":53901,"temperature":0.7,"pith_summary":"This paper sets out to rehabilitate small-scale experiments as a guide to large-scale model behavior. Its central claim is that scaling laws do exist at a few million parameters—experiments here start at 4M effective parameters—but only on the fully tuned frontier, so hyperparameter search effort, not model size, is what has made small-scale laws look unreliable. The paper shows that with roughly 256 randomly sampled configurations per scale an accurate scaling law emerges, that rigorous tuning matters more than how parameters are counted or whether exponents are tied, and that the underlying reason is geometric: as parameters grow, the hyperparameter loss surface becomes lower dimensional, with the effective number of hyperparameters dropping to about one. If true, this means extensive but cheap search at small scales can replace expensive large-scale experiments for model-centric questions. The paper demonstrates the recipe by recovering, from 4M-134M parameter runs, the long-settled large-scale result that pre-normalization beats post-normalization in transformers.","feed_headline":"Well-tuned hyperparameters, not size, unlock small-scale scaling laws","feed_subtitle":"A 256-config sweep at 4M parameters recovers a scaling law large labs said needed big models.","key_machinery":"The load-bearing object is the noisy quadratic limit: near the optimum, the validation-loss surface is well approximated by a quadratic in the hyperparameters plus additive normal noise, so that the tail of losses from random search converges to a noisy quadratic distribution Qmin(α, β, γ, σ). Fitting that distribution to random-search results yields γ, the effective number of hyperparameters, defined as the intrinsic dimension of the loss surface at the optimum. The paper uses the drop in γ with scale—from values near 2-3 at 4M to about 1 at larger sizes—as the geometric explanation for why tuning gets easier, and as the reason simple hyperparameter-carrying rules suffice when scaling up. The same limit doubles as a diagnostic: when the noisy-quadratic tail fails to materialize, the search has probably missed the optimum, which is exactly what happens for post-norm transformers.","core_discovery":"The paper's discovery is that hyperparameter tuning is the ingredient that makes or breaks a small-scale scaling law. With only 4 or 16 configurations per scale, no recognizable law appears; with 64 the shape is visible but extrapolation is weak; only at 256 configurations do predictions become accurate across scales, with held-out test MSE falling to 3.70e-6 at 268M parameters. Complementary ablations show that parameter-count definitions and tied exponents change little, while learning-rate decay refines but does not enable the law. The paper attributes small models' sensitivity to the geometry of the loss surface: random search results concentrate near the optimum as scale increases, and the effective number of hyperparameters γ, estimated by fitting a noisy quadratic to the tail of random-search scores, falls with scale and approaches one. The claim is not that any small run predicts the future, but that the tuned frontier extrapolates reliably across about an order of magnitude, with extrapolation beyond that limited by noisy estimates of the irreducible error.","pith_inferences":["This suggests a testable prediction: under scale-aware parameterizations that keep optimal hyperparameters fixed across sizes, the drop in γ should shrink or vanish, since the geometric claim depends on how the search space is parameterized.","The recipe implies a practical benchmark for 'small-scale validity': a method should reproduce the pre-norm/post-norm ordering across three scales with fewer than roughly a thousand runs, a much cheaper bar than current large-scale comparisons.","The γ≈1 result suggests that for large models, random search near default hyperparameters should be nearly as good as full tuning—an assertion one could test by comparing 10-config versus 1000-config sweeps at production scale."],"forward_implications":["Common practice of tuning over a few hand-picked configurations explains most reported failures of small-scale scaling laws.","Scaling-law experiments should report and budget search effort (number of configurations) as a first-class cost, since 256 configs at 4M parameters cost far less than one large run.","At large scales, transferring hyperparameters from small tuned models should be reliable because good configurations occupy an increasing fraction of the space.","Comparing model families by extrapolated final loss is unsafe; comparisons are trustworthy near the data or under an explicit assumption about shared irreducible error.","Pretraining loss can serve as the currency of model-centric comparisons when data is fixed, because loss fully captures capability differences."],"supporting_citations":[{"why":"Establishes the scaling-law promise and the functional form the paper works with.","marker":"Kaplan et al., 2020"},{"why":"Supplies the widely used joint scaling law and the compute-optimal training recipe that motivates the test.","marker":"Hoffmann et al., 2022"},{"why":"Introduces the noisy quadratic limit, the central machinery used to estimate the effective number of hyperparameters.","marker":"Lourie et al., 2025a"},{"why":"Provides the perplexity-capability correspondence that lets pretraining loss stand in for downstream capability.","marker":"Mayilvahanan et al., 2025"},{"why":"Defines effective parameter counts and traces earlier scaling-law disagreements to methodology, the starting point of this paper.","marker":"Porian et al., 2024"},{"why":"Documents the small-scale unreliability that the paper aims to overturn.","marker":"Pearce and Song, 2024"},{"why":"Surveys scaling-law fitting practices and reports the reproducibility problems the paper addresses.","marker":"Li et al., 2025c"},{"why":"The checkpoint-branching decay design that makes hundreds of configurations affordable at small scales.","marker":"Hägele et al., 2024"},{"why":"Provides the theoretical pre-norm advantage that the small-scale case study successfully recovers.","marker":"Xiong et al., 2020"}],"fun_headline_variants":["Hyperparameters, not scale, are key to small-scale scaling laws","Small models reveal scaling laws only with 256-config tuning","Tuned hyperparameters unlock scaling laws at 4M parameters","Small-scale scaling laws found when hyperparameters are swept deeply","How to get scaling laws from small models: tune extensively"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The geometric explanation for easier tuning rests on fitting a noisy quadratic to a tail of results whose cutoff widens from 7% of runs at 4M parameters to 50% at 134M; if the apparent drop in effective hyperparameters comes from that widening window rather than from the true loss surface, the mechanism falls.","fun_headline_variants_meta":{"raw":{"variants":["Hyperparameters, not scale, are key to small-scale scaling laws","Small models reveal scaling laws only with 256-config tuning","Tuned hyperparameters unlock scaling laws at 4M parameters","Small-scale scaling laws found when hyperparameters are swept deeply","How to get scaling laws from small models: tune extensively"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1717,"prompt_tokens":988,"completion_tokens":729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":645}},"tokens_in":604,"tokens_out":729,"duration_ms":6465,"temperature":1.0,"reasoning_tokens":645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:24:43.066939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the effective hyperparameter count γ while holding the asymptotic-regime threshold fixed at, say, 10% of runs at every scale; if the estimate no longer falls to one, the lower-dimensionality claim fails. Alternatively, repeat the γ sweep under a maximal-update parameterization that keeps optimal hyperparameters scale-invariant; if γ no longer decreases, the result is an artifact of the search distribution.","supporting_citations":[{"cited_title":"LLMs on the line: Data determines loss-to-loss scaling laws","cited_arxiv_id":null,"evidence_quote":"Provides the perplexity-capability correspondence that lets pretraining loss stand in for downstream capability."}],"review_version":1}