{"id":"c1e81d1c-04c4-442a-8f0a-0db4929643ca","arxiv_id":"2507.02119","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Compute-optimally trained networks of different sizes show loss curves that collapse onto one universal curve after normalization; with learning rate decay, the collapse is tighter than seed-to-seed noise, providing a scaling-quality diagnostic.","lead":"Training curves of compute-optimal neural networks of different sizes line up on one universal curve after scaling compute and loss to their final values, and with learning rate decay the match is so tight that differences between sizes become smaller than run-to-run noise. This extremely tight match gives practitioners a sensitive, cheap check of whether model size, data, and learning rates are scaled correctly.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The universal collapse and supercollapse rely on L0 and γ estimated from the very curves being tested; without an independent L0 estimate or fit-uncertainty propagation, the central claim could be a fitting artifact.","rationale":"The paper's strongest claim is empirical: compute-optimally trained loss curves collapse after subtracting L0 and normalizing by final loss, and with LR decay the collapse falls below the seed-noise floor. This claim would be true only if the normalization does not itself create the collapse. The fitted L0 and γ are the only quantities that define the normalization, and they are estimated from the same curves that are later tested. Figure 2's sensitivity analysis is a double-edged sword: it shows the collapse is a sharp function of L0, which increases the risk that a biased L0 (or a γ that absorbs finite-horizon effects) produces an apparent universality. The paper does not report uncertainty on L0 or γ, nor does it hold out models when assessing collapse quality. The MLP experiment with L0=0 is an important positive control, but the CIFAR-5M and chess claims—which carry the headline 'supercollapse'—depend on fitted L0. The proposed test (independent L0 from a long decay run, or bootstrap of the scaling-law fit) would settle whether the phenomenon survives honest parameter estimation. The reader's CONDITIONAL verdict is exactly right; I agree with its weakest-assumption analysis. The secondary concern about Theorem E.1 (boundary optima) is a real gap but less load-bearing, since the empirical claim stands independent of the theorem and the paper explicitly notes the limitation.","tokens_in":28286,"tokens_out":7630,"duration_ms":89034,"concrete_test":"Estimate L0 for CIFAR-5M independently of the compute-optimal ladder: train the largest model for at least 3–5 times its compute-optimal horizon under a cosine decay that anneals the learning rate to zero, and fit L0 to the late-time plateau of the loss curve. Then recompute the normalized curves and Δ(x) using this independent L0 (keeping γ fixed at the reported value). If Δ(x) exceeds the per-seed noise floor over a substantial x-range, the original supercollapse was an artifact of the fitted L0; if Δ(x) remains below σ, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central observable, ℓ(x,p,ω) in Eq (1), is not a raw dynamical quantity but a residual after two fitted transformations: subtraction of the irreducible-loss estimate L0 and normalization of compute by the estimated optimal horizon t*(p) ∝ p^γ. Both L0 and γ are obtained from the same loss curves that are then declared to collapse (Section 2.2, Appendix C). Figure 2 demonstrates the collapse is extremely sensitive to L0: shifting L0 from the fitted 3.132 by only −0.012 or +0.008 destroys the collapse. If the L0+a c^{-b} fit is biased—for example, by absorbing finite-time curvature or by the small number of Pareto-frontier points—then the subtracted offset can artificially reduce the p-dependence of the normalized curves, manufacturing a collapse that would not survive a more honest estimate. The paper provides no error bars on L0 or γ and no validation of the collapse on a held-out set of models or schedules. The MLP experiment with known L0=0 is a useful positive control, but the headline transformer results on CIFAR-5M and chess depend on fitted L0. Consequently, the claim that the universal curve is an inherent property of compute-optimal dynamics, rather than a consequence of the fitting procedure, is the weakest link in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a scaling phenomenon in compute-optimally trained networks: after normalizing training compute by the estimated compute-optimal horizon t*(p) ∝ p^γ and normalizing the reducible loss (total loss minus a fitted irreducible offset L0) by the final reducible loss at that horizon, loss curves from models spanning roughly an order of magnitude in width collapse onto a single schedule-dependent universal curve (Eq. 1; Figs. 1–3). With learning-rate decay, the cross-size collapse deviation Δ falls below the per-seed noise floor σ for a substantial fraction of training, a phenomenon the authors call supercollapse (Fig. 1c). The theoretical component has three parts: (i) for losses of the form L0 + t^{-μ} + p^{-ν}, compute-optimal allocation makes the normalized curve exactly p-independent (Eq. 7); (ii) a theorem (E.1) states that full collapse forces a power-law Pareto frontier; and (iii) a quadratic/SDE model of schedule-dependent SGD noise predicts loss curves across schedules, model sizes, and horizons with a single fitted constant α (Eqs. 16–18; Fig. 6) and predicts Δ² ∝ η(τ̂)(1 − τ̂) during decay (Eq. 24; Fig. 8). Experiments include transformers on CIFAR-5M and chess, MLPs on a power-law-spectrum regression task, width- and depth-scaling ladders, and negative controls (constant-LR parameterization; wrong data exponent) showing that collapse breaks under suboptimal scaling choices.","tokens_in":28462,"tokens_out":17101,"duration_ms":178131,"significance":"If the main claim holds, the paper is significant: it extends scaling laws from statements about final losses to statements about the entire training trajectory, and it provides evidence for a joint limit in which model size and training time grow together under compute-optimal allocation while preserving dynamical consistency across sizes, in contrast to the divergence found in fixed-horizon infinite-width limits (Vyas et al., 2023, cited). The empirical core is unusually well quantified for a scaling-laws paper: collapse deviation is always compared with a per-seed noise floor (Eqs. 2–3); collapse is shown to break under both wrong parameterization and wrong data exponent (Fig. 4), giving the claim falsifiable content; the MLP setup has known L0 = 0 as a positive control; multiple seeds are used; and the code is released. The schedule-prediction model's success across schedules, widths, and horizons with a single fitted α (0.21, close to the theoretical 1/4) is a genuine predictive achievement (Fig. 6, Appendix I).","major_comments":[{"comment":"The collapse observable is not a raw quantity: both the offset L0 subtracted in Eq. (1) and the horizon t*(p) ∝ p^γ that sets the normalized-compute axis are fitted to the same loss curves whose collapse is then asserted. Figure 2 shows that shifting L0 by only −0.012 or +0.008 from the fitted value 3.132 destroys the collapse, yet the paper reports no uncertainty for L0 or γ, and two separate fits on the same CIFAR-5M data give L0 = 3.132 (Fig. 1a) and L0 = 3.134 (Fig. 5a). Because the least-squares fit of L0 + ac^{-b} uses a small number of effectively independent Pareto-frontier points estimated from only 5 seeds, the plausible uncertainty in L0 may be comparable to the ±0.01 sensitivity scale, in which case part of the observed collapse could be an artifact of choosing L0 to make the frontier a power law (which Theorem E.1 shows is necessary for collapse). The nontrivial content is full-curve collapse over x ∈ (0,1], since the frontier fit constrains only final losses and the slope at x = 1; to support that claim, the authors should report bootstrap or seed-subsample uncertainties on (L0, γ, a, b), re-evaluate Δ(x) for L0 at the extremes of that uncertainty, and perform a held-out check in which L0 and γ are fit on a subset of the scaling ladder and the collapse is measured on the complementary subset. The MLP positive control (known L0 = 0) and the Fig. 4 sensitivity experiments are good first steps, but they do not resolve the fitting-circularity concern for the headline transformer results.","section":"§2.2–2.4, Eq. (1), Fig. 2, Appendix C"},{"comment":"Section 3.1 presents the power-law Pareto frontier as necessary for collapse and uses this to justify the choice L̂ = L0, but the proof requires the compute-optimal point to be interior to each loss curve. The Remark appended to Theorem E.1 concedes that this condition fails for schedules that reach η = 0 after finitely many steps, which are exactly the schedules on which the headline supercollapse results (Figs. 1b–c and 4) are demonstrated. As written, the theorem is formally established for the constant-LR family used to estimate t*(p), but its necessity claim is not established for the decaying-schedule runs, and its sufficiency direction yields only first-order collapse rather than full-curve supercollapse. The explanation of supercollapse therefore rests on the empirical noise model of Section 3.3 rather than on Theorem E.1, which is acceptable, but the 'necessary' narrative in Section 3.1 should be restricted to the setting where the hypotheses hold, or the proof should be extended to boundary-optimal schedules; the Remark's caveat should also be surfaced in the main text rather than only in the appendix.","section":"Appendix E (Theorem E.1 and its Remark), §3.1"},{"comment":"The theoretical derivation's main deliverable is an explicit universal curve ℓ(x) = (r x^{-μ} + 1)/(r + 1), and Fig. 5a reports a fit of L0 + t^{-μ} + p^{-ν} to the CIFAR-5M loss surface with L0 = 3.134, μ = 0.41, and ν = 0.37 (R² = 0.999). The paper never overlays this predicted curve on the empirically collapsed constant-LR curves of Fig. 3, nor does it report the deviation between the predicted and observed normalized curves relative to the seed noise floor. In addition, the theory predicts the data exponent γ = ν/μ ≈ 0.90, which differs from the empirically estimated γ = 0.96 (Fig. 10d) without any comment on the discrepancy. Adding the predicted-curve overlay (and an analogous check for the schedule-transferred prediction of Eq. (21)) would test the quantitative content of the claimed universality rather than only its cross-size aspect, and would also help bound the circularity raised in the first major comment, since the same fitted exponents define the normalization.","section":"§3.1, Eq. (7), Fig. 5, Fig. 10d"}],"minor_comments":[{"comment":"The noise floor σ(x, p) is defined per model size in Eq. (3), but Fig. 1c plots a single noise-floor curve and compares it with the p-averaged Δ(x); the paper should state whether the plotted floor is the mean, median, or maximum over p and should define the comparison criterion behind the claim Δ(x) < σ.","section":"§2.4, Eqs. (2)–(3), Fig. 1c"},{"comment":"The closed-form prediction (1 − αh(x)δη(τ,p))^{-1} appears in Eq. (18) without derivation from the linearized expression in Eq. (16); a one-sentence geometric-series justification and its regime of validity (small αhδη) would prevent confusion.","section":"§3.2.2, Eqs. (16)–(18)"},{"comment":"The schedule-prediction experiments on CIFAR-5M use a different configuration from the collapse experiments (different codebase, base width, initialization, learning rate, and batch size, as described in Appendix A); the main text should flag this so readers do not treat Fig. 6 as validating the exact runs of Fig. 1.","section":"§3.2.2, Fig. 6, Appendix A"},{"comment":"Presentation issues: the typos 'origianlly' (Appendix C) and 'dervie' (Appendix G) should be fixed, and the two fitted CIFAR-5M values L0 = 3.132 (Fig. 1a) and L0 = 3.134 (Fig. 5a) should be reconciled or explicitly explained in the text.","section":"Appendix C, Appendix G, Figs. 1a and 5a"},{"comment":"The predicted scaling in Fig. 8 is stated in terms of normalized gradient-flow time τ̂ while the rest of the paper uses normalized compute x = t/t⋆(p); a sentence relating the two normalizations would let the reader compare Fig. 8 directly with Fig. 1c.","section":"§3.3, Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is polished, the empirical program is strong, and the authors are unusually honest about limitations (Appendix G and the Remark after Theorem E.1). My recommendation of major_revision is driven by one load-bearing methodological gap — the fit-defined normalization (L0, γ) with no uncertainty quantification and no held-out validation — plus two validation holes: the predicted universal curve shape is never compared with the observed collapsed curves, and the necessity theorem's formal scope excludes the decaying-schedule regime where supercollapse is claimed. All three concerns are addressable within the scope of the revision using data the authors already have, so I would not support rejection; the core observation and the MLP positive control are strong enough to merit a revision cycle rather than a desk decision. The editor may also wish to ask the authors to state, in the abstract or introduction, that the observed collapse is defined relative to an estimated irreducible loss whose uncertainty is not yet quantified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper has a real empirical result: for compute-optimally trained models of different sizes, the full loss curves collapse onto one schedule-dependent curve after subtracting a fitted irreducible loss and normalizing compute and reducible loss by their final values. Second, the collapse is not a free lunch claim; they quantify it against a per-seed noise floor and show that with learning-rate decay the cross-model variation drops below that floor for much of training. That 'supercollapse' observation is new and, if it holds up, gives a cheap diagnostic for mis-scaled hyperparameters.\n\nWhat is genuinely new: the model-size-independent normalization (Eq. 1), the supercollapse phenomenon below the noise floor, and the single-constant schedule predictor (Eq. 18) that fits loss curves across schedules, model sizes, and horizons with one shared alpha (0.21 for transformers, 0.26 for MLPs). The empirical core is well quantified: multiple seeds, explicit noise-floor comparison, and sensitivity experiments showing collapse breaks under wrong gamma or wrong parameterization. The MLP experiment has known L0=0, which is a useful positive control. The paper is also unusually honest about its limitations, in Appendix G and the Discussion.\n\nThe main soft spot is circularity in the normalization. L0 and gamma are estimated from the same curves that are then declared to collapse, and Figure 2 shows the collapse quality is sharply peaked at the fitted L0. There are no error bars on L0 or gamma, and no held-out check on a separate set of models or schedules. The MLP control mitigates this, but the headline transformer results on CIFAR-5M and chess depend on the fitted L0. This makes the strong claim 'the universal curve is an inherent property of compute-optimal dynamics' partially a statement about the fitting procedure. I would not call it fatal: the sensitivity itself, plus the small-scale breadth across two transformers and an MLP, make the phenomenon plausible. But independent L0 estimation and fit-uncertainty propagation are needed before I would call it settled.\n\nA secondary gap: Theorem E.1's necessity proof requires interior compute-optimal points, and the paper's own Remark excludes the schedules that reach eta=0 after finitely many steps—exactly the schedules where supercollapse is claimed. That leaves the theory incomplete for the flagship phenomenon, though the empirical evidence still stands on its own. Also, alpha is fitted rather than derived as 1/4; minor in context.\n\nWho this is for: scaling-law researchers and practitioners who want a more sensitive probe of scaling behavior than final-loss fits. The paper deserves a serious referee. My recommendation: engage with it, but ask for independent L0 estimation, error bars on the fitted parameters, and validation on a held-out scaling ladder or larger models before treating the universality as established.","headline":"A real and novel empirical phenomenon—loss-curve collapse and supercollapse in compute-optimal training—honestly quantified, but with a soft spot in the L0-fitting circularity that needs independent validation before the universality claim is settled.","tokens_in":29134,"tokens_out":2913,"would_cite":true,"duration_ms":31265,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Entire loss curves of compute-optimally trained networks collapse onto one universal shape.","keywords":["scaling laws","compute-optimal training","loss curve collapse","universality","SGD noise","learning rate schedules","transformers","supercollapse"],"falsifier":"Train a compute-optimal scaling ladder on a task where the irreducible loss can be pinned down independently of the collapse analysis—for example, by measuring the loss of a very large model trained for a very long time, or by injecting label noise so the Bayes error is known—and check whether curves from different sizes still collapse under Eq. (1) when that independently fixed $L_0$ is subtracted. A second check is to test whether the predicted scaling $\\Delta^2 \\propto \\eta(\\tau)(1 - \\hat{\\tau})$ holds for schedules the paper did not try, such as exponential or stepwise decay; failure of either check would indicate that the universality is fit-dependent.","tokens_in":27948,"feed_emoji":"📉","tokens_out":8855,"duration_ms":92974,"temperature":0.7,"pith_summary":"This paper aims to establish that, in compute-optimally trained neural networks, the entire loss curve—not just the final loss—follows a universal scaling law. The authors show that after subtracting a fitted irreducible loss (the estimated floor that no amount of training can reduce) and normalizing training compute and reducible loss to 1 at the end of training, loss curves from models of different sizes collapse onto one schedule-dependent curve. They identify a stronger regime, supercollapse, in which learning-rate decay makes the collapse so tight that the differences between model sizes drop below the seed-to-seed noise of any single training run. The paper derives the collapse from the power-law structure of neural scaling laws and a simple model of SGD noise, and argues that collapse quality is a sensitive and practical indicator of whether hyperparameters are scaled correctly.","feed_headline":"Neural loss curves collapse onto one universal shape","feed_subtitle":"In compute-optimal training, curves from all model sizes match below the seed noise floor—a powerful scaling diagnostic.","key_machinery":"The central object is the affine normalization of Eq. (1), $\\ell(x, p, \\omega) = \\left(L(x t^\\star(p), p, \\omega) - L_0\\right)/\\left(L(t^\\star(p), p, \\omega) - L_0\\right)$, which maps every model's loss curve into the unit square in reducible loss versus normalized compute. The argument has three load-bearing pieces: (1) the sum-of-power-laws ansatz $L = L_0 + t^{-\\mu} + p^{-\\nu}$, which at the compute-optimal horizon $t^\\star \\propto p^{\\nu/\\mu}$ makes the normalized curve exactly independent of $p$; (2) a quadratic-SDE model of SGD noise in gradient-flow time, which gives the approximate loss shift $\\delta L \\approx \\alpha\\, \\delta\\eta(\\tau)\\, \\mathrm{Tr}(\\Sigma'(\\tau))$ and predicts that schedule-induced deformations are shared across model sizes because $\\mathrm{Tr}(\\Sigma)/L$ depends only on normalized compute; and (3) a first-order variance computation showing the normalized-curve variance obeys $\\Delta^2 \\propto \\eta(\\tau)(1 - \\hat{\\tau})$ near the end of training, explaining why learning-rate decay produces supercollapse.","core_discovery":"After subtracting a fitted irreducible loss $L_0$ and rescaling so that final compute and final reducible loss are both $1$, the loss curves $L(t, p, \\omega)$ of compute-optimally trained models satisfy, to high precision, a size-independent identity: the normalized curve $\\ell(x)$ is the same function of normalized compute $x = t/t^\\star(p)$ for all model sizes $p$ under a fixed learning-rate schedule. For loss functions of the form $L = L_0 + t^{-\\mu} + p^{-\\nu}$, this identity is exact at the compute-optimal horizon $t^\\star(p) \\propto p^{\\nu/\\mu}$, giving $\\ell(x) = (r x^{-\\mu} + 1)/(r+1)$ with $r = \\nu/\\mu$, independent of $p$ (Eq. 7). The identity persists across learning-rate schedules because a quadratic-noise model of SGD predicts that schedule-induced deformations depend on the product of the instantaneous learning rate and the gradient-covariance trace, and the ratio $\\mathrm{Tr}(\\Sigma)/L$ is empirically a function of normalized compute alone. Learning-rate decay makes the collapse ``super'': the variance of the normalized curves scales as $\\eta(\\tau)(1 - \\hat{\\tau})$, so normalizing by the realized final loss cancels most of the seed noise, pushing the collapse below the single-trajectory noise floor. The paper argues that when the data exponent or parameterization is scaled suboptimally, the collapse breaks, making it a practical indicator of good scaling.","pith_inferences":["One testable extension, not pursued in the paper, is to use the collapse deviation itself as an objective: tune the data exponent $\\gamma$, parameterization, and schedule to maximize collapse tightness instead of fitting sparse final-loss power laws.","This suggests that the ratio $\\mathrm{Tr}(\\Sigma)/L$, if universal in normalized compute, could let practitioners measure gradient-noise statistics on small models and transfer them to large models at matched normalized compute, giving a budget-free way to predict schedule effects.","The authors compare their phenomenon to renormalization-group universality; if that analogy is substantive, collapse should persist to larger scales and new architectures, and deviations from collapse would be interpretable as relevant scaling variables that must be tuned.","The control-variate view of normalization by the realized final loss suggests that other late-time observables, such as gradient norms, could be used to denoise and collapse different training statistics, not just the loss."],"forward_implications":["Loss-curve prediction becomes a scaling-law problem: for a new model size, the entire compute-optimal loss trajectory can be predicted from the universal curve and the fitted $L_0$, without training to completion.","The collapse can be measured cheaply and used to validate whether hyperparameters such as learning-rate parameterization and the data exponent are scaled correctly, since suboptimal choices break the collapse even when final losses look similar.","The supercollapse criterion gives a noise-floor-limited comparison of models across scales, so single-seed training runs may suffice to detect scaling inconsistencies that currently require many seeds.","The variance law $\\Delta^2 \\propto \\eta(\\tau)(1 - \\hat{\\tau})$ makes a quantitative, schedule-specific prediction for how quickly the collapse should improve as learning rate is decayed, which is testable in new settings.","The result suggests a joint scaling limit in model size and training time that preserves consistency throughout training, complementing infinite-width limits that diverge at late times."],"supporting_citations":[{"why":"Supplies the compute-optimal training-horizon estimation procedure (Approach 1) and the $L_0 + a c^{-b}$ scaling-law form.","marker":"Hoffmann et al., 2022"},{"why":"Source of the power-law scaling-law family, the $6tp$ compute estimate, and the sum-of-power-laws loss-curve form.","marker":"Kaplan et al., 2020"},{"why":"Introduces the concept of irreducible loss $L_0$ used in the normalization.","marker":"Sharma & Kaplan, 2022"},{"why":"Justifies reusing the same $t^\\star(p)$ across learning-rate schedules as optimal up to a constant factor.","marker":"Pearce & Song, 2024"},{"why":"Provides the CIFAR-5M dataset used for the main transformer experiments.","marker":"Nakkiran et al., 2020"},{"why":"Defines the $\\mu$P parameterization needed for consistent width scaling and hyperparameter transfer.","marker":"Yang et al., 2021"},{"why":"Basis of the SDE model of SGD noise used for schedule prediction.","marker":"Li et al., 2017"},{"why":"Provides the noisy-quadratic-model analysis that the schedule-prediction formula extends.","marker":"Zhang et al., 2019"},{"why":"Establishes the $\\mathrm{Tr}(\\Sigma) \\approx 2L\\,\\mathrm{Tr}(K)$ relation used to justify universal gradient-noise scaling.","marker":"Paquette et al., 2021"},{"why":"Shows divergence of fixed-time infinite-width limits, the contrast motivating the joint compute-optimal scaling limit.","marker":"Vyas et al., 2023"}],"fun_headline_variants":["Supercollapse: loss curves match below seed noise","Universal loss collapse in compute-optimal neural nets","One curve fits all: scaling collapse in neural training","Scaling collapse reveals universal training dynamics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fitted irreducible loss $L_0$ and the fitted data exponent $\\gamma$ correctly describe the asymptotic scaling; both are estimated from the very loss curves whose collapse is then reported, so a misestimated $L_0$ or $\\gamma$ could make the universal curve and supercollapse partly artifacts of the fitting procedure.","fun_headline_variants_meta":{"raw":{"variants":["Supercollapse: loss curves match below seed noise","Universal loss collapse in compute-optimal neural nets","One curve fits all: scaling collapse in neural training","Scaling collapse reveals universal training dynamics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1613,"prompt_tokens":1055,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":671,"tokens_out":558,"duration_ms":6524,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:40:25.639104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a compute-optimal scaling ladder on a task where the irreducible loss can be pinned down independently of the collapse analysis—for example, by measuring the loss of a very large model trained for a very long time, or by injecting label noise so the Bayes error is known—and check whether curves from different sizes still collapse under Eq. (1) when that independently fixed $L_0$ is subtracted. A second check is to test whether the predicted scaling $\\Delta^2 \\propto \\eta(\\tau)(1 - \\hat{\\tau})$ holds for schedules the paper did not try, such as exponential or stepwise decay; failure of either check would indicate that the universality is fit-dependent.","supporting_citations":[{"cited_title":"and Kaplan, J","cited_arxiv_id":null,"evidence_quote":"Introduces the concept of irreducible loss $L_0$ used in the normalization."},{"cited_title":"Sgd in the large: Average-case analysis, asymptotics, and stepsize criticality","cited_arxiv_id":null,"evidence_quote":"Establishes the $\\mathrm{Tr}(\\Sigma) \\approx 2L\\,\\mathrm{Tr}(K)$ relation used to justify universal gradient-noise scaling."},{"cited_title":"Feature-learning networks are consistent across widths at realistic scales","cited_arxiv_id":null,"evidence_quote":"Shows divergence of fixed-time infinite-width limits, the contrast motivating the joint compute-optimal scaling limit."}],"review_version":1}