Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Small-Scale Experiments: Are We There Yet?

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that scaling laws exist at 4M parameters, and that rigorous hyperparameter tuning—hundreds of random configurations per scale—is what reveals them, with the effective number of hyperparameters falling to one as models…

desk verdict A genuinely useful empirical finding—hyperparameter tuning, not scale, is what hides small-scale scaling laws—but the flagship 256-configuration claim is tested with unequal sampling effort across scales, so treat the exact number with caution. read the letter →

arxiv 2608.11859 v1 pith:RWOTZUBS submitted 2026-08-12 cs.LG

Nicholas Lourie , Kyunghyun Cho , Karen Ullrich , Sanae Lotfi This is my paper · ORCID
classification cs.LG
keywords scalinglawssmall-scaleexperimentshyperparametertuningnoisyquadraticlimiteffectivehyperparameterslosssurfacepre-normvspost-normlanguagemodelpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to rehabilitate small-scale experiments as a guide to large-scale model behavior. Its central claim is that scaling laws do exist at a few million parameters—experiments here start at 4M effective parameters—but only on the fully tuned frontier, so hyperparameter search effort, not model size, is what has made small-scale laws look unreliable. The paper shows that with roughly 256 randomly sampled configurations per scale an accurate scaling law emerges, that rigorous tuning matters more than how parameters are counted or whether exponents are tied, and that the underlying reason is geometric: as parameters grow, the hyperparameter loss surface becomes lower dimensional, with the effective number of hyperparameters dropping to about one. If true, this means extensive but cheap search at small scales can replace expensive large-scale experiments for model-centric questions. The paper demonstrates the recipe by recovering, from 4M-134M parameter runs, the long-settled large-scale result that pre-normalization beats post-normalization in transformers.

What carries the argument

The load-bearing object is the noisy quadratic limit: near the optimum, the validation-loss surface is well approximated by a quadratic in the hyperparameters plus additive normal noise, so that the tail of losses from random search converges to a noisy quadratic distribution Qmin(α, β, γ, σ). Fitting that distribution to random-search results yields γ, the effective number of hyperparameters, defined as the intrinsic dimension of the loss surface at the optimum. The paper uses the drop in γ with scale—from values near 2-3 at 4M to about 1 at larger sizes—as the geometric explanation for why tuning gets easier, and as the reason simple hyperparameter-carrying rules suffice when scaling up. The same limit doubles as a diagnostic: when the noisy-quadratic tail fails to materialize, the search has probably missed the optimum, which is exactly what happens for post-norm transformers.

What would settle it

Recompute the effective hyperparameter count γ while holding the asymptotic-regime threshold fixed at, say, 10% of runs at every scale; if the estimate no longer falls to one, the lower-dimensionality claim fails. Alternatively, repeat the γ sweep under a maximal-update parameterization that keeps optimal hyperparameters scale-invariant; if γ no longer decreases, the result is an artifact of the search distribution.

Watch

Extended reading notes

Core claim

The paper's discovery is that hyperparameter tuning is the ingredient that makes or breaks a small-scale scaling law. With only 4 or 16 configurations per scale, no recognizable law appears; with 64 the shape is visible but extrapolation is weak; only at 256 configurations do predictions become accurate across scales, with held-out test MSE falling to 3.70e-6 at 268M parameters. Complementary ablations show that parameter-count definitions and tied exponents change little, while learning-rate decay refines but does not enable the law. The paper attributes small models' sensitivity to the geometry of the loss surface: random search results concentrate near the optimum as scale increases, and the effective number of hyperparameters γ, estimated by fitting a noisy quadratic to the tail of random-search scores, falls with scale and approaches one. The claim is not that any small run predicts the future, but that the tuned frontier extrapolates reliably across about an order of magnitude, with extrapolation beyond that limited by noisy estimates of the irreducible error.

Load-bearing premise

The geometric explanation for easier tuning rests on fitting a noisy quadratic to a tail of results whose cutoff widens from 7% of runs at 4M parameters to 50% at 134M; if the apparent drop in effective hyperparameters comes from that widening window rather than from the true loss surface, the mechanism falls.

Editorial extensions

If this is right

  • Common practice of tuning over a few hand-picked configurations explains most reported failures of small-scale scaling laws.
  • Scaling-law experiments should report and budget search effort (number of configurations) as a first-class cost, since 256 configs at 4M parameters cost far less than one large run.
  • At large scales, transferring hyperparameters from small tuned models should be reliable because good configurations occupy an increasing fraction of the space.
  • Comparing model families by extrapolated final loss is unsafe; comparisons are trustworthy near the data or under an explicit assumption about shared irreducible error.
  • Pretraining loss can serve as the currency of model-centric comparisons when data is fixed, because loss fully captures capability differences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a testable prediction: under scale-aware parameterizations that keep optimal hyperparameters fixed across sizes, the drop in γ should shrink or vanish, since the geometric claim depends on how the search space is parameterized.
  • The recipe implies a practical benchmark for 'small-scale validity': a method should reproduce the pre-norm/post-norm ordering across three scales with fewer than roughly a thousand runs, a much cheaper bar than current large-scale comparisons.
  • The γ≈1 result suggests that for large models, random search near default hyperparameters should be nearly as good as full tuning—an assertion one could test by comparing 10-config versus 1000-config sweeps at production scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that scaling laws for language models are observable at parameter counts as low as 4M parameters, provided one estimates the fully tuned hyperparameter frontier, and that the main obstacle to small-scale scaling laws is hyperparameter sensitivity, which decreases with scale. The authors support this with random-search sweeps across scales (4M–268M) under a hand-crafted model ladder and randomly sampled architectures, ablating methodological choices, and estimating the effective number of hyperparameters γ via the noisy quadratic limit. They then propose a holistic methodology for small-scale model-centric research and demonstrate it on the pre-norm vs. post-norm transformer question, claiming that small-scale experiments recover the large-scale preference for pre-norm. The central empirical results are the direct sensitivity measurements (Figures 6 and 7) and the observation that scaling laws appear only when sufficiently many configurations are used to define the tuned frontier.

Significance. If the claims hold, this is a practically important contribution: it offers a concrete explanation for the 'scaling gap' and a recipe for making small-scale experiments more reliable. The experimental effort is substantial—hundreds of configurations per scale, held-out scale partitions, detailed appendices on search distributions and fitting details, and use of open tools—and the paper is unusually transparent about its limitations. The direct sensitivity measurements in Figures 6 and 7 are convincing and do not depend on the more fragile γ estimator. However, three load-bearing parts currently need work: the matched-sampling confound in Figure 5, the threshold-dependence of the γ estimator in Section 4.2, and the reliance on a shared irreducible-error assumption in the case study's headline conclusion. These are fixable, but they prevent full acceptance as written.

major comments (3)
  1. [§3.2, Figure 5, §A.2] The panel comparison in Figure 5 does not hold the number of configurations per scale fixed, and the caption states that all panels evaluate on the best configurations from the entire sweep. According to §A.2, the ladder sweep uses 826–934 configurations at the 4M–34M training scales but only 128 at 67M–134M and 64 at 268M. Because the minimum over a random search is a downward-biased estimator of the fully tuned frontier and the bias decreases with N, the apparent improvement from 4 to 256 configurations is confounded with unequal sampling effort across fitting and evaluation scales. The specific claim that 256 configurations per scale yield an accurate extrapolating law therefore needs either a matched-N evaluation (e.g., subsampling exactly 256 configurations at every scale for both fitting and evaluation) or an explicit best-of-N correction.
  2. [§4.2, Figures 8–9, §A.3] The estimated effective hyperparameter count γ is obtained by fitting the noisy quadratic distribution to the tail of the random-search score distribution, with the threshold θ growing from 7% at 4M parameters to 50% at 134M/268M parameters (§A.3). Since γ is a fitted shape parameter of that tail, the systematic widening of the tail with scale can by itself produce a decreasing γ even if the intrinsic dimension is constant. Moreover, γ is constrained to be at least 1, and Figure 8 shows the estimate saturating at that boundary. The claim that the hyperparameter loss surface becomes lower dimensional is the paper's proposed mechanism for why tuning gets easier, so it needs a sensitivity analysis with a fixed threshold across scales, or a threshold-independent estimator, before this geometric explanation can be considered established. This concern does not affect the direct sensitivity measurements in Figures 6 and 7, which support the empirical claim that tuning gets easier with scale.
  3. [§5.3, Figure 15] The case-study conclusion that pre-norm is better at scale is obtained under the joint fit that ties the irreducible error ϵ across the two architectures. Figure 15 shows that with free ϵ, post-norm becomes better beyond the tested range, and the manuscript offers no empirical evidence that the two architectures share the same asymptotic loss. The abstract's statement that small-scale experiments 'recover the large scale result: pre-normalization works better as models grow in size' is therefore stronger than what the scaling-law comparison supports. The authors should either soften the claim, or provide a principled justification or test of the shared-ϵ assumption (e.g., longer training runs or larger-scale post-norm runs that approach the same floor).
minor comments (6)
  1. [§A.3] There is a typo in the sentence 'In the ladder and random / prenorm) experiments' with an unmatched parenthesis; please fix.
  2. [Figure 5 caption] The caption should state explicitly how many configurations are used at each scale for fitting and for evaluation; the current wording 'best configurations from our entire sweep' obscures the unequal sample sizes across scales.
  3. [Figure 8] Figure 8 would benefit from uncertainty quantification for γ (e.g., bootstrap confidence intervals), since the estimate is used to support a trend down to 1 and the fitting procedure involves a boundary constraint.
  4. [Figure 12] Figure 12 compares pre-norm (128 configurations per scale) with post-norm (511–512 configurations at 4M and 34M) without stating the sample-size difference in the caption; the density comparison should be labeled as unmatched to avoid overinterpretation.
  5. [§5.3, Diagnostic 1] The claim that the noisy quadratic limit 'confirms the hyperparameters are well-tuned' is somewhat strong for post-norm, where the asymptotic regime is only 3–10% of runs; consider softening the wording to reflect the smaller asymptotic regime.
  6. [Throughout] The notation H_{x^*} is used for the Hessian in Equation 1, while 'H' informally denotes hyperparameters elsewhere; distinct notation would improve clarity.

Circularity Check

1 steps flagged · score 2.0 of 10

Central 'tuning matters' claim is measured directly; the only load-bearing self-citation is the noisy-quadratic tool used for the geometric 'why' and for tuning certification.

  1. self citation load bearing [§2.1 (Noisy Quadratic Limit), §4.2 (Effective Hyperparameters, Figure 8), §5.3 (Diagnostic 1)]
    "The effective number of hyperparameters (γ) is the surface’s intrinsic dimension at the optimum, a major determinant of tuning difficulty (Lourie et al., 2025a). We estimate γ at each parameter–data budget by fitting a noisy quadratic to the tail from random search... Figure 8 plots these estimates against model size. The trend is clear: larger models have fewer effective hyperparameters, dropping to one. ... If random search approaches the optimum, then the score distribution converges to a noisy quadratic (§2.1)."

    The paper's explanation for why tuning becomes easier with scale—that the hyperparameter loss surface becomes lower dimensional—is operationalized through γ, which is estimated by fitting the noisy quadratic distribution introduced in the authors' own prior work (Lourie et al., 2025a) and cited rather than independently re-derived or externally verified here. The same self-cited limit is then used to certify that the random search reached the optimum ('If random search approaches the optimum, then the score distribution converges to a noisy quadratic'), which underpins the 'fully tuned frontier' claim in §3.2.

full rationale

The central headline—that scaling laws emerge at small scales only on the well-tuned frontier and that tuning effort, not model size, is the binding constraint—is measured directly. §3.2 fits Equation 3 to minima from sweeps of 4/16/64/256 configurations and reports held-out MSE on the 67M–268M scales, which were not used in fitting; the held-out minima are estimates of the frontier, not fitted parameters of the law, so the extrapolation test does not reduce to the fit by construction. The same holds for the ablation in §3.1/Figure 4. The one load-bearing self-citation I can exhibit is the noisy quadratic limit used to estimate γ and to certify tuning quality in §5.3; the γ-drop is an output of that self-cited model, not an independent measurement, and this affects the 'why' rather than the direct 'tuning matters' result. The Figure 5 validation does not hold sampling effort fixed across scales (roughly 826–934 configurations at 4M–34M versus 64–128 at held-out scales, §A.2), which is a real confound for the quantitative '256 configurations suffice' claim, but it is not a circular reduction because the held-out minima are not used to fit the law. The case study's recovery of the pre-norm advantage explicitly depends on sharing the irreducible error across architectures; that is a stated assumption, not a disguised tautology. Overall circularity burden is low, so the score is 2.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; the claims rest on fitted empirical quantities (γ, scaling-law coefficients, θ) and adopted prior assumptions.

free parameters (5)
  • Scaling law exponents and coefficients (ε, ζ, ι, η, κ) = Fitted per experiment (Figure 14 etc.)
    Equation 3 is fit to small-scale best losses; extrapolation to 134M/268M is the evidence that scaling laws exist at small scales.
  • Effective hyperparameter count γ = Varies from ~7 down to 1 (Figure 8)
    Estimated by fitting the noisy quadratic distribution to random-search tails; central to the lower-dimensionality claim.
  • Asymptotic-regime threshold θ = Percentiles 7% to 50% by scale (A.3)
    Chosen per scale/checkpoint; the widening tail with scale is a potential confound for γ.
  • Hyperparameter priors (Figure 16/17)
    Ad hoc choices for random search; affect the measured sensitivity and the required number of configurations.
  • Token budget multiplier 32x and 25% decay extension
    Training-length choices that shape the measured loss surface; not justified by theory.
assumptions (4)
  • domain assumption The noisy quadratic limit (Eq. 1) holds near the optimum of the hyperparameter loss surface.
    From Lourie et al. (2025a), used to estimate γ and to diagnose tuning quality in Figure 11.
  • standard math The scaling law L(p,d) = ε + ζ/p^ι + η/d^κ (Eq. 3) is the correct parametric form.
    Widely used in the literature (Rosenfeld et al., Hoffmann et al.); assumed without derivation.
  • domain assumption Perplexity-capability correspondence: with fixed pretraining data, loss determines capability.
    From Mayilvahanan et al. (2025); underpins using loss as the metric for model comparison.
  • ad hoc to paper Pre-norm and post-norm architectures share the same irreducible error at convergence.
    Assumed in §5.3 and Figure 15 to conclude pre-norm scales better; the authors note if this fails, post-norm may overtake.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Small-Scale Experiments: Are We There Yet?." pith.science (2026). https://pith.science/paper/RWOTZUBS

@misc{pith2026260811859,
  author       = {Pith},
  title        = {Pith review of: Small-Scale Experiments: Are We There Yet?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWOTZUBS}},
  note         = {Machine review of arXiv:2608.11859}
}
read the original abstract

Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales (starting at 4M parameters) and concluded that sizable models cannot be avoided. We show this is not the case: the confounding factor is hyperparameters. Small models are highly sensitive, but hyperparameter sensitivity fades with scale. This small-scale sensitivity makes scaling laws easy to miss because they only emerge on the fully tuned frontier, and reaching that frontier requires an extensive search far beyond what most ever run. By ablating the basic scaling law recipe, we show well-tuned hyperparameters matter more than any other ingredient. Further, we reveal why those hyperparameters become easier to find: as scale increases, the hyperparameter loss surface becomes lower dimensional. Nevertheless while scaling laws exist in small models, extrapolation hits statistical limitations. A holistic approach is required. Synthesizing our insights with the recent literature, we develop a new methodology for model-centric research and demonstrate it on a question that once took the field years to settle: where to place normalization layers in the transformer architecture. From small-scale experiments, we recover the large scale result: pre-normalization works better as models grow in size. With the right tools and a better understanding, small-scale experiments can deliver on scaling laws' long-awaited promise.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 17 canonical work pages

  1. [1]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Z...

    arXiv 2023

  2. [3]

    adaptive

    by dropping the eigenvalues of least absolute value. We then use the static model to initialize thedynamic optima and the dynamic sensitivity models, setting the new components to zero since the models are nested. We similarly initialize thedynamic model from thedynamic optima model. For themultilayer perceptronmodel, we implement the neural network using...

    work page 2011

  3. [6]

    Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, and David Owen

    Morgan-Kaufmann, 1993.https://proceedings.neurips.cc/paper_files/paper/1993/ file/1aa48fc4880bb0c9b8a3bf979d3b917e-Paper.pdf. Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, and David Owen. The rising costs of training frontier ai models, 2025.https://arxiv.org/abs/2405.21015. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen...

    arXiv 1993

  4. [8]

    Nelder-Mead

    In thepostnorm experiment, they were as follows: for 4M effective parameters, 3% at checkpoint 1, 4% at 2, 5% at 3, 6% at 4, and 7% at 5–8; for 34M and 134M, 7% at 1–2, 8% at 3, and 10% at 4–8. We restrictedα to be positive (as bits-per-character always is), and we restrictedγ to be 1 to at most the nominal number of hyperparameters (7 forladderand 9 forr...

    work page 2023

  5. [9]

    https://proceedings.neurips.cc/paper_files/paper/2024/file/ 5f1eee2509599faeeb3570a887016a64-Paper-Conference.pdf. Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Luca Soldaini, Jenia Jitsev, Alex Dimaki...

    work page 2024

  6. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    doi: 10.52202/079017-2427.https://proceedings.neurips.cc/ paper_files/paper/2024/file/8b970e15a89bf5d12542810df8eae8fc-Paper-Conference.pdf. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021....

    arXiv 2024

  7. [12]

    doi: 10.1038/s41586-025-09833-y

    ISSN 1476-4687. doi: 10.1038/s41586-025-09833-y. https://doi.org/10.1038/s41586-025-09833-y. Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Findings of the Association for Computational Linguistics: ...

    doi:10.1038/s41586-025-09833-y

  8. [14]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B

    doi: 10.1038/s41586-021-03819-2. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.https://arxiv.org/abs/ 2001.08361. Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet,...

    arXiv 2020

Show all 40 references
  1. [15]

    https: //www.science.org/doi/abs/10.1126/science.adi2336

    doi: 10.1126/science.adi2336. https: //www.science.org/doi/abs/10.1126/science.adi2336. 18 Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang, Zili Wang, Shijie Xuyang, Ning DING, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale (part ii) — farsee...

    arXiv

  2. [16]

    doi: 10.18653/v1/2024.naacl-long.189.https://aclanthology.org/2024.naacl-long.189/

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.189.https://aclanthology.org/2024.naacl-long.189/. Nicholas Lourie, He He, and Kyunghyun Cho. Hyperparameter loss surfaces are simple near their optima. InSecond Conference on Language Modeling, 2025a....

    2024 doi

  3. [17]

    LLMs on the line: Data determines loss-to-loss scaling laws

    Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. LLMs on the line: Data determines loss-to-loss scaling laws. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry...

    2025

  4. [18]

    doi: 10.1038/ s41586-023-06735-9.https://doi.org/10.1038/s41586-023-06735-9

    ISSN 1476-4687. doi: 10.1038/ s41586-023-06735-9.https://doi.org/10.1038/s41586-023-06735-9. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, ...

    doi

  5. [19]

    doi: 10.18653/v1/D18-1260

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1260. https://aclanthology.org/D18-1260/. Bruno Kacper Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, and marco cuturi. Completed hyperparameter transfer across modules, wi...

    doi

  6. [20]

    Association for Computational Linguistics.https://aclanthology.org/2019.iwslt-1.17/. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-D...

    2019 arXiv

  7. [22]

    https://proceedings.neurips.cc/paper_files/paper/2024/file/ 370df50ccfdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf

    doi: 10.52202/079017-0970. https://proceedings.neurips.cc/paper_files/paper/2024/file/ 370df50ccfdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf. Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-o...

    2024 doi

  8. [23]

    https://proceedings.neurips.cc/paper_files/paper/2024/file/ b6341525cd84f3be0ef203e4d7cd8556-Paper-Conference.pdf

    doi: 10.52202/079017-3189. https://proceedings.neurips.cc/paper_files/paper/2024/file/ b6341525cd84f3be0ef203e4d7cd8556-Paper-Conference.pdf. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring...

    2024 arXiv

  9. [25]

    doi: https://doi.org/10.1016/ j.neucom.2023.127063.https://www.sciencedirect.com/science/article/pii/S0925231223011864

    ISSN 0925-2312. doi: https://doi.org/10.1016/ j.neucom.2023.127063.https://www.sciencedirect.com/science/article/pii/S0925231223011864. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. InProcee...

    2023

  10. [26]

    doi: 10.1038/s41586-023-06747-5

    ISSN 1476-4687. doi: 10.1038/s41586-023-06747-5. https: //doi.org/10.1038/s41586-023-06747-5. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S...

    doi

  11. [28]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H

    doi: 10.1038/s41592-019-0686-2. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abili...

    doi

  12. [29]

    Survey Certification

    ISSN 2835-8856.https://openreview.net/forum?id=yzkSU5zdwD. Survey Certification. Jason Wei, Najoung Kim, Yi Tay, and Quoc V. Le. Inverse scaling can become u-shaped, 2023.https://arxiv.org/ abs/2211.02011. Ethan G Wilcox, Michael Hu, Aaron Mueller, Tal Linzen, Alex Warstadt, L...

    2023 arXiv

  13. [30]

    doi: 10.18653/v1/2023.acl-long.767.https://aclanthology.org/2023.acl-long.767/

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.767.https://aclanthology.org/2023.acl-long.767/. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in ...

    2023 doi

  14. [31]

    Huaqing Zhang, Kaiyue Wen, and Tengyu Ma

    https://proceedings.neurips.cc/paper_files/paper/2019/file/ 1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf. Huaqing Zhang, Kaiyue Wen, and Tengyu Ma. Configuration-to-performance scaling law with neural ansatz,

    2019

  15. [32]

    Tim Pearce and Jinyeop Song

    Curran Associates, Inc., 2019.https://proceedings.neurips.cc/paper_files/paper/2019/file/ bdbca288fee7f92f2bfa9f7012727740-Paper.pdf. Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws.Transactions on Machine Learning Research,

    2019

  16. [33]

    Rosie Zhao, Tian Qin, David Alvarez-Melis, Sham Kakade, and Naomi Saphra

    https://arxiv.org/abs/2602.10300. Rosie Zhao, Tian Qin, David Alvarez-Melis, Sham Kakade, and Naomi Saphra. Random scaling of emergent capabilities, 2026.https://arxiv.org/abs/2502.17356. 23 Appendix A Experimental Design This section documents the details and rationales for o...

    2026

  17. [34]

    The model uses RMSNorm for normalization (Zhang and Sennrich, 2019), SwiGLU for the activation function (Shazeer, 2020), and RoPE for position embeddings (Su et al., 2024)

    available in Lingua (apps/main/train.py) (Videau et al., 2024). The model uses RMSNorm for normalization (Zhang and Sennrich, 2019), SwiGLU for the activation function (Shazeer, 2020), and RoPE for position embeddings (Su et al., 2024). All runs use a pre-normalization archite...

    2020

  18. [35]

    Unlike Lingua, we use a linear cooldown for the WSD schedule decaying to aoptim.lr_min_ratio of 1e-6 times the peak learning rate

    with a warmup-stable-decay (WSD) learning rate schedule (Hu et al., 2024). Unlike Lingua, we use a linear cooldown for the WSD schedule decaying to aoptim.lr_min_ratio of 1e-6 times the peak learning rate. Each scale we train is defined by a target number of effective paramete...

    2022

  19. [36]

    A.2 Search Distributions Our results analyze three different experiments, each defined by a search distribution over the hyperparameters

    Accordingly, data order and other details were kept constant while runs differed primarily in their hyperparameters. A.2 Search Distributions Our results analyze three different experiments, each defined by a search distribution over the hyperparameters. All the experiments ra...

    2024

  20. [37]

    In practice, we found the simple heuristic of sampling real values on a linear scale, positive values on a log scale, and bounded values on a logit scale worked quite well

    Typically, researchers search hyperparameters like the learning rate or weight decay on a log scale; using the proper scale for each hyperparameter is also important in efficiently reaching the noisy quadratic limit (Lourie et al., 2025a). In practice, we found the simple heur...

    2024

  21. [38]

    Following Porian et al

    We increase the width and depth of successive models in order to steadily increase their parameter count, while keeping other architectural elements (such as thehead_dim) constant. Following Porian et al. (2024), we keep the aspect ratio between 32 and

    2024

  22. [39]

    We round the solution to the nearest multiple of 8 for GPU efficiency

    Given the effective parameters, the number of layers determines the hidden dimension via Equation 8, which is a quadratic ind given the other parameters and the fact thatdff = 8/3dfor our model. We round the solution to the nearest multiple of 8 for GPU efficiency. Once we hav...

    2024

  23. [2017]

    Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lopez- Paz

    https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lopez- Paz. Meta Lingua: A minimal PyTorch LLM training library, 2...

    2017

  24. [2018]

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma

    https: //arxiv.org/abs/1809.10853v1. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121,

    arXiv

  25. [2019]

    doi: 10.18653/v1/N19-1300

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. https://aclanthology.org/N19-1300/. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 r...

    arXiv

  26. [2020]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo

    https: //openreview.net/forum?id=ryenvpEKDr. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In 20 Thirty-seventh Conference on Neural Information Processing Systems, 2023.https://openreview.net/forum?id= ITw9edRDlD....

    2023 arXiv

  27. [2021]

    https://crfm.stanford.edu/assets/report.pdf. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Ch...

    1901

  28. [2022]

    doi: 10.18653/v1/2022.findings-emnlp.544

    Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.544. https: //aclanthology.org/2022.findings-emnlp.544/. Andy L. Jones. Scaling scaling laws with board games, 2021.https://arxiv.org/abs/2104.03113. John Jumper, Richard Evans, Alexander Pritzel, ...

    2022 arXiv

  29. [2023]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vino...

    2023

  30. [2024]

    https://www.pnas.org/doi/abs/10.1073/pnas.2311878121

    doi: 10.1073/pnas.2311878121. https://www.pnas.org/doi/abs/10.1073/pnas.2311878121. Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scal- ing laws for weight decay and batch size in llm pre-training. In D. Belgrave, C. Zhan...

    doi

  31. [2025]

    https://openreview.net/forum?id=iZeQBqJamf. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds,...

    2023

  32. [2026]

    Accessed: 2026-06-13

    https://epoch.ai/gradient-updates/r-and-d-vs-training-compute. Accessed: 2026-06-13. Singularity Developers. Singularity, 2021.https://doi.org/10.5281/zenodo.1310023. Nolan Dey, Bin Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and...

    2026 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.