Pith. sign in

REVIEW 3 major objections 4 minor 45 references

LLM pretraining is locally quadratic: Taylor-expanded losses track the true training curve for up to 10% of a 3B-token run, with universal power-law spectra and training at the edge of stability.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:53 UTC pith:FUQSTSVR

load-bearing objection A strong empirical case that quadratic proxies track late-training LLM dynamics, with a softer edge-of-stability conclusion that depends on data filtering. the 3 major comments →

arxiv 2607.21716 v1 pith:FUQSTSVR submitted 2026-07-23 cs.LG cs.AImath.OCstat.ML

A Defense of the Quadratic Model

Alexandru Meterez , Pranav Ajit Nair , Depen Morwani , Cengiz Pehlevan , Sham Kakade , Alex Damian This is my paper
classification cs.LG cs.AImath.OCstat.ML
keywords quadratic modelLLM pretrainingTaylor expansionGauss-Newton matrixHessian spectrumLanczos quadratureedge of stabilitypower-law tail
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper stress-tests the simplest model in optimization theory — a quadratic loss — against real LLM pretraining and finds it predictive. Training a 150M-parameter transformer on 3B tokens, the authors show that a first-order (prox) or second-order Gauss-Newton (quad) Taylor expansion of the model around intermediate checkpoints reproduces the true validation loss for windows up to 10% of the token budget, with best agreement late in training. Deep Lanczos probes reveal that the local quadratic problems have structured spectra: a head tied to the unembedding layer and a universal power-law tail, with a source exponent below 1 that contradicts the usual summability assumption. Perturbing learning rate and batch size from checkpoints places training within a factor of 2 of the stability boundary — stochastic at small/medium batch, deterministic at large batch. If correct, the paper makes the quadratic model a validated proxy for pretraining theory, not just a toy sandbox.

Core claim

The central claim is that LLM pretraining can be locally understood as a chain of structured convex quadratic problems. Concretely, Taylor expanding the model around a checkpoint while retaining the cross-entropy loss (prox), and then taking a Gauss-Newton quadratic expansion (quad), tracks the true loss within 10% error for up to 10% of a 3B-token budget; the agreement is strongest late in training. The local problems are not arbitrary: their Hessian and Gauss-Newton spectra separate into a head dominated by the unembedding layer and a universal power-law tail (capacity exponent α≈1 for Gauss-Newton, α≈0.65 for Hessian), unaffected by batch size or the Adam preconditioner, with a source exp

What carries the argument

The load-bearing tool is the local quadratic expansion guided by the Gauss-Newton matrix G = ∇²L_prox(θ_ref), the positive-semidefinite part of the Hessian under model linearization; training on this proxy turns pretraining into a convex quadratic problem. Deep Lanczos quadrature (probe depth 1200, with full reorthogonalization) estimates spectra over six orders of magnitude, and linear stability analysis treats preconditioned SGD on a fixed quadratic as a stochastic linear dynamical system, distinguishing mean-square from Lyapunov stability to locate the edge of stability.

Load-bearing premise

The load-bearing premise is that the rare, outsized divergence spikes seen in the quadratic surrogate are artifacts of freezing Adam's preconditioner — the paper filters out the top 1% of sequences and says that without this correction nearly all hyperparameters diverged — and that those spikes do not reflect genuine instabilities of the real optimizer; if they are real, the edge-of-stability claim collapses, though the loss-agreement claim would stand.

What would settle it

Run the stability probe from a single late checkpoint twice: once with Adam's second-moment buffer ν frozen and the 1% outlier filter as in the paper, and once with ν updated normally and no filter. If the unfiltered, live-preconditioner run shows a scattered or shifted divergence boundary far from a factor of 2, the paper's central stability conclusion fails; a cheaper partial check is to vary the filter threshold from 0.1% to 5% and see whether the measured boundary moves.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If local pretraining dynamics are quadratic, then theory built mode-by-mode on quadratics — learning-rate schedules, batch-size scaling, preconditioning — transfers to LLM pretraining and can give sharp predictions.
  • The universal power-law tail and β<1 source exponent imply the effective problem is finite-dimensional: after a critical time scaling with parameter count, bias decays exponentially and loss is variance-dominated, matching the measured 1/T decay across runs.
  • Training sits within a factor of 2 of a stability boundary, so doubling the learning rate or halving the batch size on the local surrogate should cause divergence; this gives a concrete, testable hyperparameter safety margin.
  • The head of the spectrum is set by the vocabulary size and the unembedding layer, with top eigenvalues tracking token frequencies, so the largest curvature directions can be predicted from data statistics rather than measured.
  • The quadratic approximation fails in early training and under constant learning rate, delimiting where higher-order curvature and edge-of-stability dynamics, not the quadratic model, govern the trajectory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: rerun the stability probe with Adam's second-moment buffer updated instead of frozen and without filtering the top 1% of sequences; if divergence patterns shift away from the factor-of-2 boundary, the edge-of-stability conclusion is an artifact of the frozen-preconditioner proxy, while the loss-agreement claim would still stand.
  • The β<1 source exponent suggests long pretraining runs should show a finite-dimensional saturation: after roughly p^α tokens, additional tokens stop improving bias and only variance averaging (1/T) remains, which could be tested by extending runs well beyond the current 3B budget.
  • The universality of the tail across batch sizes and preconditioners points to a structural origin (layer parameter counts, embedding geometry); varying vocab size or depth should predictably move the head/tail boundary, giving a clean geometric test of the neural-collapse explanation.
  • The early-training gap between prox and quad indicates a cheap diagnostic: measuring the magnitude of the dropped non-Gauss-Newton Hessian term at each checkpoint should predict how long the quadratic approximation will hold, potentially guiding when quadratic-surrogate optimization is safe to use.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper stress-tests the quadratic model of optimization on 150M-parameter transformer pretraining (3B tokens, FineWeb). It reports that (i) local Taylor expansions—both a prox-linear (NTK-style) model and a Gauss-Newton quadratic—around intermediate checkpoints can track the true validation loss for windows up to 10% of the training budget, with the best agreement late in training; (ii) deep Lanczos estimates of the Hessian and Gauss-Newton spectra reveal power-law tails with capacity exponent α≈1 for the Gauss-Newton matrix and α≈0.65 for the Hessian, source exponents β<1, and a head-tail structure tied to the unembedding layer; and (iii) local stability experiments on a frozen-preconditioner quadratic proxy suggest that training operates within a factor of 2 of a stochastic or deterministic edge of stability, depending on batch size. The paper concludes that LLM pretraining can be locally understood as optimization over a sequence of structured quadratic problems.

Significance. If the claims hold, this is a valuable empirical bridge between tractable quadratic optimization theory and LLM pretraining. The Section 3 agreement experiment is direct and externally benchmarked against the true validation loss; the Lanczos methodology uses rigorous Gauss-Radau error bands and is validated against synthetic ground truth (Figure 18); and the full code and data are released. The spectrum exponents and stability boundaries are concrete, falsifiable measurements. However, the broadest stability conclusion depends on a filtering step whose sensitivity is not analyzed, and the source-exponent estimates lack uncertainty quantification. The central loss-agreement result is much more robust than the edge-of-stability narrative.

major comments (3)
  1. [§5.1, 'Extreme Outliers'] The stability measurement is performed on the frozen-preconditioner quadratic proxy, and the top 1% of sequences by per-sequence preconditioned Gauss-Newton Frobenius norm are filtered out; the paper states that without this correction "nearly all hyperparameters diverged after a single outsized spike." The spikes are attributed to rare tokens/dead neurons in the frozen ν, but no evidence is provided that the true Adam dynamics—with an evolving second-moment buffer—are immune or respond benignly to the same events. If those spikes are intrinsic, the post-filter stability boundary describes a modified problem, not the real training dynamics. I request (i) a sensitivity analysis over the filter quantile (e.g., 0.1%, 1%, 5%) and (ii) a direct check of whether the same outlier sequences cause divergence in the actual optimizer (or in the proxy with an evolving ν). The Section 3 agreement cla
  2. [§4.4, Figures 5/11/15] The source exponent β is estimated by differentiating the cumulative Lanczos sketch, which the paper itself calls numerically unstable, after smoothing over a multiplicative window [i/2,2i]. No error bars are reported for β, and no sensitivity to the smoothing window width or Lanczos depth is given. The synthetic validation in Figure 18 validates spectral density estimates, not source-condition estimation with known ground-truth β. Because the reported β<1 contradicts the common β>1 assumption and drives the "finite-dimensional behavior" interpretation, this measurement needs a dedicated synthetic validation and uncertainty analysis before it can be treated as a quantitative finding.
  3. [§5.3] The phrase "within a factor of 2 of the edge of stability" is not fully supported by the text. The paper argues from the stability of (η,B)=(0.5,1) that training is at most a factor of 2 over the stability boundary, but an edge-of-stability conclusion also requires that increasing the learning rate (or decreasing batch size) by a factor of 2 exits the stable region. The paper only says the (1,1) cell is "not always" stable. Please state explicitly, for each (B,T) and schedule, whether (2,1) or (1,B/2) are unstable, and define the factor-2 statement formally. If the heatmaps already show this, the text should say so quantitatively.
minor comments (4)
  1. [§4.2 vs Abstract] The capacity exponent is quoted as α≈0.96 for the Gauss-Newton tail in §4.2 but α≈1 in the abstract. Please reconcile the rounding or report the fitted values consistently.
  2. [Figure 5] The figure axis label is "estimate of g_i^2/(2λ_i)" but the text defines the source condition as ⟨u_i,∇L⟩²/λ_i. Clarify the factor of 2 and the precise quantity being plotted.
  3. [Table 1] The 10% agreement criterion is defined, but Table 1 entries appear to come from single runs. State explicitly whether the prox/quad continuations are single-seed or multi-seed, since the 10.0 entries at late checkpoints are otherwise hard to interpret.
  4. [§3.3] The sentence "we leave a further study of why this is the case to future work" after the early-time divergence of quad is a bit terse; a one-sentence speculation (e.g., higher-order curvature effects) would help the reader.

Circularity Check

0 steps flagged

No significant circularity: the paper's central predictions are checked against an external benchmark (the true LLM loss) and against synthetic ground-truth spectra; only minor non-load-bearing self-citations and a robustness caveat appear.

full rationale

The derivation chain is not circular in any load-bearing place. Section 3 defines two local proxies (prox-linear and Gauss-Newton quadratic) and validates them by continuing training on the proxy and evaluating on the true nonlinear validation loss; the agreement windows in Table 1 are measured, not enforced, so the claim 'the quadratic model predicts LLM optimization dynamics' is benchmarked externally. Section 4's spectral estimates are independently validated against synthetic spectra with known ground truth (Figure 18), and the capacity/source exponents are direct fits to measured spectra rather than parameters renamed as predictions. Section 5's edge-of-stability conclusions are empirical probes on a quadratic proxy whose fidelity is established separately in Section 3, and the theoretical stability predictions are cited from external literature (Cohen et al., Mulayoff et al.), not imported from the authors' own unverified results. The one mildly self-referential passage is Section 4.4: after measuring beta < 1, the paper fits Lt = L* + AT^-gamma and reports gamma ~ 1 as matching the 'variance dominated regime predicted by theory'. This is a consistency check, not a circularity, because gamma is fitted directly from the loss curve and is not computed from the beta measurement; the agreement between two independent measurements is exactly the kind of external test that supports the theory. The top-1% outlier filter in Section 5.1 is a genuine robustness limitation for the stability claim, but it is not a circular reduction: the claim is not defined in terms of the filter, and the absence of sensitivity analysis is a validity concern rather than self-justification. There are self-citations (e.g., Meterez et al., Morwani et al., Abreu et al., Dey et al.), but they enter only as related work or as choices of parameterization and do not carry the load of any derived conclusion. Overall the paper is self-contained against external benchmarks, so the circularity score is low.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The paper introduces no new theoretical entities; the central claims rest on fitted spectral/loss exponents, a hand-chosen outlier filter, and prior theoretical results (mean-square stability, Lanczos moment matching). The most consequential free parameters are the outlier filter quantile and the power-law exponents, since the conclusions about edge of stability and beta<1 hinge on them.

free parameters (8)
  • Capacity exponent alpha (Gauss-Newton tail) = ≈0.96
    Fitted to the log-log slope of the GN spectrum tail in Fig 3c (§4.2.1).
  • Capacity exponent alpha (Hessian tail) = ≈0.65
    Fitted to the log-log slope of the Hessian spectrum tail; claimed batch-size/preconditioner independent (§4.2.1).
  • Source exponent beta (Gauss-Newton) = 0.1–0.2
    Fitted to smoothed derivative of cumulative squared gradient projections from gradient-start Lanczos probe (§4.4, Fig 5).
  • Source exponent beta (Hessian) = 0.3–0.6
    Same protocol applied to Hessian/preconditioned Hessian; basis of the beta<1 claim contradicting the summability assumption.
  • Loss asymptote L* and decay exponent gamma = L*≈2.5022, gamma≈0.745–1.108
    Joint power-law fit Lt=L*+AT^{-gamma} over checkpoints 6–10 (§4.4, Fig 9); gamma≈1 used to support the variance-dominated regime.
  • Outlier filter quantile (top 1%) = 1%
    Chosen by hand to remove rare-token/dead-neuron spikes in stability experiments; without it nearly all hyperparameters diverge (§5.1).
  • Lanczos probe depth m = 1200
    Hand-chosen probe depth; claims of resolving spectra over 6 orders of magnitude and all tail exponents depend on it (750GB basis, §4.1).
  • Optimal learning rates per batch size = per Fig 8 sweep (values not tabulated)
    Peak eta for B∈{1,64,1024} selected from sweeps; the EOS claim is conditional on training at these tuned eta (§3.2, §5).
axioms (6)
  • domain assumption Mean-square stability characterization of SGD on quadratics (Mulayoff & Michaeli 2024, Thm 5 & Prop 6)
    Used in §5.2 to predict the wedge-shaped stable region in (eta,B) and claim qualitative agreement with heatmaps.
  • standard math Lanczos quadrature matches first 2m-1 moments; Gauss-Radau bounds hold for index i*(lambda)
    Basis of spectral estimation in §4.1–4.2 (Golub & Meurant 2009).
  • domain assumption Hessian-vector products computed on 10M tokens represent the population loss curvature
    Assumed in §4.1 methodology; ablation Figure 17 compares 10M vs 100M tokens and finds no significant difference.
  • domain assumption A single random Lanczos probe suffices to resolve the spectrum tail
    Asserted in §4.1 ablation ('we empirically found that a single random probe was sufficient'); justified by block-averaging in tails and validated on synthetic spectra (Fig 18).
  • ad hoc to paper Power-law forms lambda_i ∝ i^{-alpha} and <u_i,g>^2/lambda_i ∝ i^{-beta} hold in the fitted ranges
    The fitted exponents alpha, beta in §4.2.1/§4.4 assume power-law tails; the paper does not test alternative functional forms.
  • domain assumption CompleteP parameterization (Dey et al. 2026) is the correct per-coordinate LR scaling for Hessian spectral stability across scales
    Used in §3.2/§4.1; baked into both the Hessian and preconditioner; if the scaling is wrong, spectral comparisons across layers and checkpoints shift.

pith-pipeline@v1.3.0-alltime-deepseek · 25604 in / 21236 out tokens · 180062 ms · 2026-08-01T06:53:55.584400+00:00 · methodology

0 comments
read the original abstract

Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics. In this work, we stress test the simplest possible model of optimization -- the quadratic model -- and show that it can be surprisingly predictive in an LLM setting with 150M parameters and 3B training tokens. Specifically, we show that Taylor expanding the model and the loss function at intermediate checkpoints through training can accurately predict the optimization dynamics over windows that can last up to 10\% of training. Having established this agreement, we then turn to analyzing the structure of these local quadratic optimization problems through two lenses: the Hessian spectrum and local stability. Using Lanczos quadrature with extremely deep probes, we are able to estimate the Hessian spectrum deep into the tail, and we find a surprising amount of structure in both the eigenvalues and eigenvectors, which depends on the batch size, preconditioner, and training time. We also empirically test local linear stability at intermediate checkpoints and compare it to theoretical predictions to demonstrate that optimization in LLMs typically occurs at a stochastic edge of stability, whose nature is also determined by batch size. Our results indicate the quadratic model may be a theoretically tractable proxy for pretraining optimization dynamics.

Figures

Figures reproduced from arXiv: 2607.21716 by Alexandru Meterez, Alex Damian, Cengiz Pehlevan, Depen Morwani, Pranav Ajit Nair, Sham Kakade.

Figure 1
Figure 1. Figure 1: Validation loss curves for 150M models (black) trained with cosine for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: We estimate the spectrum of the Gauss Newton matrix, the Hessian, and their precondi [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Analyses of the Hessian spectra for cosine decay. See Figure [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An estimate of the mass of each eigenvector in each parameter group using SLQ. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An estimate of the source condition ⟨ui , ∇L(θ)⟩ 2 /λi using SLQ at the final 100% checkpoint. In addition to a random Lanczos probe, we also run Lanczos starting from the gradient, which captures the distribution of the gradient across the spectrum. In analyses of linear regression, this is often called the source condition and is parameterized by the exponent β: ⟨ui , ∇L(θ)⟩ 2 /λi ∝ i −β (Source Conditio… view at source ↗
Figure 6
Figure 6. Figure 6: A toy simulation of the linear stability on a 1D regression problem where [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Heatmap showing instability for models trained with cosine decay as a function of [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: (η, B) pretraining sweep for the constant with EMA and cosine trained models. 2.50 2.55 2.60 2.65 C h e c k p oint e v al loss, E M A 0:0 4 A = 0:2512 ° = 1:0328 R 2 = 0:99872 Cosine, B = 1 2.50 2.55 2.60 2.65 A = 0:2733 ° = 1:1077 R 2 = 0:99874 Cosine, B = 64 2.4 2.5 2.6 2.7 2.8 2.9 3.0 A = 0:6691 ° = 0:7454 R 2 = 0:98997 Cosine, B = 1024 1.6 1.8 2.0 2.2 2.4 2.6 2.8 3.0 3.2 Training tokens (billions) 2.45… view at source ↗
Figure 9
Figure 9. Figure 9: Joint power law fit for the loss as a function of the tokens of the form [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A full sweep of the eigenvector distribution at the end of training for cosine decay. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: An extended version of Figure [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Validation loss curves for 150M models (black) trained with constant learning with EMA [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: The spectra using a constant learning rate schedule after warmup, rather than cosine [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: A full sweep of the eigenvector distribution at the end of training for the constant learning [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: The source condition using a constant learning rate schedule after warmup, rather than [PITH_FULL_IMAGE:figures/full_fig_p032_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Heatmap showing instability for models trained with constant learning rate with EMA [PITH_FULL_IMAGE:figures/full_fig_p033_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Comparison of the preconditioned Gauss Newton spectrum at [PITH_FULL_IMAGE:figures/full_fig_p034_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Comparing the ground truth (dashed black) with the SLQ estimate (orange) for various [PITH_FULL_IMAGE:figures/full_fig_p034_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 35 linked inside Pith

  1. [2]

    The proximal point method revisited.arXiv preprint arXiv:1712.06038,

    Dmitriy Drusvyatskiy. The proximal point method revisited.arXiv preprint arXiv:1712.06038,

    Pith/arXiv arXiv

  2. [4]

    Estimating the spectral density of large implicit matrices.arXiv preprint arXiv:1802.03451,

    Ryan P Adams, Jeffrey Pennington, Matthew J Johnson, Jamie Smith, Yaniv Ovadia, Brian Patton, and James Saunderson. Estimating the spectral density of large implicit matrices.arXiv preprint arXiv:1802.03451,

    Pith/arXiv arXiv

  3. [6]

    The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size.arXiv preprint arXiv:1811.07062,

    Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size.arXiv preprint arXiv:1811.07062,

    Pith/arXiv arXiv

  4. [7]

    Beyond linearization: On quadratic and higher-order approximation of wide neural networks.arXiv preprint arXiv:1910.01619,

    Yu Bai and Jason D Lee. Beyond linearization: On quadratic and higher-order approximation of wide neural networks.arXiv preprint arXiv:1910.01619,

    Pith/arXiv arXiv 1910

  5. [9]

    Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians.arXiv preprint arXiv:1901.08244,

    Vardan Papyan. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians.arXiv preprint arXiv:1901.08244,

    Pith/arXiv arXiv 1901

  6. [10]

    Taylorized training: Towards better approximation of neural network training at finite width.arXiv preprint arXiv:2002.04010,

    Yu Bai, Ben Krause, Huan Wang, Caiming Xiong, and Richard Socher. Taylorized training: Towards better approximation of neural network training at finite width.arXiv preprint arXiv:2002.04010,

    Pith/arXiv arXiv 2002

  7. [12]

    The large learning rate phase of deep learning: the catapult mechanism.arXiv preprint arXiv:2003.02218,

    Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism.arXiv preprint arXiv:2003.02218,

    Pith/arXiv arXiv 2003

  8. [13]

    Neural networks as kernel learners: The silent alignment effect.arXiv preprint arXiv:2111.00034,

    Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect.arXiv preprint arXiv:2111.00034,

    Pith/arXiv arXiv

  9. [15]

    Gradient descent on neural networks typically occurs at the edge of stability.arXiv preprint arXiv:2103.00065,

    Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability.arXiv preprint arXiv:2103.00065,

    Pith/arXiv arXiv

  10. [16]

    Properties of the after kernel.arXiv preprint arXiv:2105.10585,

    Philip M Long. Properties of the after kernel.arXiv preprint arXiv:2105.10585,

    Pith/arXiv arXiv

  11. [18]

    Adaptive gradient methods at the edge of stability.arXiv preprint arXiv:2207.14484,

    Jeremy M Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, et al. Adaptive gradient methods at the edge of stability.arXiv preprint arXiv:2207.14484,

    Pith/arXiv arXiv

  12. [19]

    Self-stabilization: The implicit bias of gradient descent at the edge of stability.arXiv preprint arXiv:2209.15594,

    Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability.arXiv preprint arXiv:2209.15594,

    Pith/arXiv arXiv

  13. [20]

    Accelerated sgd for non-strongly-convex least squares

    Aditya Varre and Nicolas Flammarion. Accelerated sgd for non-strongly-convex least squares. In Conference on Learning Theory, pages 2062–2126. PMLR,

    2062

  14. [21]

    A view of mini-batch sgd via generating functions: conditions of convergence, phase transitions, benefit from negative momenta.arXiv preprint arXiv:2206.11124,

    22 Maksim Velikanov, Denis Kuznedelev, and Dmitry Yarotsky. A view of mini-batch sgd via generating functions: conditions of convergence, phase transitions, benefit from negative momenta.arXiv preprint arXiv:2206.11124,

    Pith/arXiv arXiv

  15. [22]

    Limitations of the ntk for understanding generalization in deep learning.arXiv preprint arXiv:2206.10012,

    Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding generalization in deep learning.arXiv preprint arXiv:2206.10012,

    Pith/arXiv arXiv

  16. [24]

    Trak: Attributing model behavior at scale.arXiv preprint arXiv:2303.14186,

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale.arXiv preprint arXiv:2303.14186,

    Pith/arXiv arXiv

  17. [25]

    Edge of stochastic stability: Revisiting the edge of stability for sgd.arXiv preprint arXiv:2412.20553,

    Arseniy Andreyev and Pierfrancesco Beneventano. Edge of stochastic stability: Revisiting the edge of stability for sgd.arXiv preprint arXiv:2412.20553,

    arXiv

  18. [27]

    Adaptive methods through the lens of sdes: Theoretical insights on the role of noise.arXiv preprint arXiv:2411.15958,

    Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of sdes: Theoretical insights on the role of noise.arXiv preprint arXiv:2411.15958,

    Pith/arXiv arXiv

  19. [28]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  20. [29]

    Why do learning rates transfer? reconciling optimization and scaling limits for deep learning.arXiv preprint arXiv:2402.17457,

    Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvieto. Why do learning rates transfer? reconciling optimization and scaling limits for deep learning.arXiv preprint arXiv:2402.17457,

    Pith/arXiv arXiv

  21. [30]

    Neurons in large language models: Dead, n-gram, positional

    Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. InFindings of the Association for Computational Linguistics: ACL 2024, pages 1288–1301,

    2024

  22. [31]

    The optimality of (accelerated) sgd for high-dimensional quadratic optimization.arXiv preprint arXiv:2409.09745, 2024a

    Haihan Zhang, Yuanshi Liu, Qianwen Chen, and Cong Fang. The optimality of (accelerated) sgd for high-dimensional quadratic optimization.arXiv preprint arXiv:2409.09745, 2024a. Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham Kakade. How does critical batch size scale in pre-training?arXiv preprint arXiv:2...

    Pith/arXiv arXiv

  23. [33]

    Universal sharpness dynamics in neural network training: Fixed point analysis, edge of stability, and route to chaos

    Dayal Singh Kalra, Tianyu He, and Maissam Barkeshli. Universal sharpness dynamics in neural network training: Fixed point analysis, edge of stability, and route to chaos. InInternational Conference on Learning Representations, volume 2025, pages 55966–56000,

    2025

  24. [34]

    Seesaw: Accelerating training by balancing learning rate and batch size scheduling

    Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, and Sham Kakade. Seesaw: Accelerating training by balancing learning rate and batch size scheduling. arXiv preprint arXiv:2510.14717,

    Pith/arXiv arXiv

  25. [35]

    The surprising agreement between convex optimization theory and learning-rate scheduling for large model training.arXiv preprint arXiv:2501.18965,

    Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agreement between convex optimization theory and learning-rate scheduling for large model training.arXiv preprint arXiv:2501.18965,

    Pith/arXiv arXiv

  26. [36]

    Risk comparisons in linear regression: Implicit regularization dominates explicit regularization.arXiv preprint arXiv:2509.17251,

    Jingfeng Wu, Peter L Bartlett, Sham M Kakade, Jason D Lee, and Bin Yu. Risk comparisons in linear regression: Implicit regularization dominates explicit regularization.arXiv preprint arXiv:2509.17251,

    Pith/arXiv arXiv

  27. [37]

    Adam-mini: Use fewer learning rates to gain more

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik Durk Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations, volume 2025, pages 28033–28063,

    2025

  28. [38]

    Momentum further constrains sharpness at the edge of stochastic stability.arXiv preprint arXiv:2604.14108,

    Arseniy Andreyev, Advikar Ananthkumar, Marc Walden, Tomaso Poggio, and Pierfrancesco Ben- eventano. Momentum further constrains sharpness at the edge of stochastic stability.arXiv preprint arXiv:2604.14108,

    Pith/arXiv arXiv

  29. [40]

    Hessian spectral analysis at foundation model scale.arXiv preprint arXiv:2602.00816,

    Diego Granziol and Khurshid Juarev. Hessian spectral analysis at foundation model scale.arXiv preprint arXiv:2602.00816,

    arXiv

  30. [41]

    Spectral dynamics in deep networks: Feature learning, outlier escape, and learning rate transfer.arXiv preprint arXiv:2605.07870,

    Clarissa Lauditi, Cengiz Pehlevan, and Blake Bordelon. Spectral dynamics in deep networks: Feature learning, outlier escape, and learning rate transfer.arXiv preprint arXiv:2605.07870,

    Pith/arXiv arXiv

  31. [42]

    Any- time pretraining: Horizon-free learning-rate schedules with weight averaging.arXiv preprint arXiv:2602.03702,

    Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, and Sham Kakade. Any- time pretraining: Horizon-free learning-rate schedules with weight averaging.arXiv preprint arXiv:2602.03702,

    arXiv

  32. [43]

    Compute efficiency and serial runtime tradeoffs for stochastic momentum methods.arXiv preprint arXiv:2606.19179,

    Depen Morwani, Alexandru Meterez, Pranav Nair, and Sham Kakade. Compute efficiency and serial runtime tradeoffs for stochastic momentum methods.arXiv preprint arXiv:2606.19179,

    Pith/arXiv arXiv

  33. [44]

    Deriving hyperparameter scaling laws via modern optimization theory.arXiv preprint arXiv:2603.15958,

    Egor Shulgin, Dimitri von Rütte, Tianyue H Zhang, Niccolò Ajroldi, Bernhard Schölkopf, and Antonio Orvieto. Deriving hyperparameter scaling laws via modern optimization theory.arXiv preprint arXiv:2603.15958,

    arXiv

  34. [45]

    Beyond a single explanation of the adam–sgd gap.arXiv preprint arXiv:2606.14259, 2026a

    Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni, Jun Pang, Aurelien Lucchi, and Antonio Orvieto. Beyond a single explanation of the adam–sgd gap.arXiv preprint arXiv:2606.14259, 2026a. Dechen Zhang, Xuan Tang, Yingyu Liang, and Difan Zou. Scaling laws for precision in high- dimensional linear regression.arXiv preprint arXiv:2602.19241, 2026b. 25 A...

    arXiv

  35. [2016]

    Eigenvalues of the hessian in deep learning: Singular- ity and beyond.arXiv preprint arXiv:1611.07476,

    Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singular- ity and beyond.arXiv preprint arXiv:1611.07476,

    Pith/arXiv arXiv

  36. [2017]

    Empirical analysis of the hessian of over-parametrized neural networks.arXiv preprint arXiv:1706.04454,

    Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks.arXiv preprint arXiv:1706.04454,

    Pith/arXiv arXiv

  37. [2018]

    On the relation between the sharpest directions of dnn loss and the sgd step length.arXiv preprint arXiv:1807.05031,

    Stanisław Jastrz˛ ebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length.arXiv preprint arXiv:1807.05031,

    Pith/arXiv arXiv

  38. [2019]

    Deep curvature suite.arXiv preprint arXiv:1912.09656,

    Diego Granziol, Xingchen Wan, and Timur Garipov. Deep curvature suite.arXiv preprint arXiv:1912.09656,

    Pith/arXiv arXiv 1912

  39. [2020]

    The break-even point on optimization trajectories of deep neural networks.arXiv preprint arXiv:2002.09572,

    Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks.arXiv preprint arXiv:2002.09572,

    Pith/arXiv arXiv 2002

  40. [2021]

    Learning curves for sgd on structured features.arXiv preprint arXiv:2106.02713,

    Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd on structured features.arXiv preprint arXiv:2106.02713,

    Pith/arXiv arXiv

  41. [2022]

    Randomized matrix-free quadrature for spectrum and spectral sum approximation.arXiv preprint arXiv:2204.01941, page 55,

    Tyler Chen, Thomas Trogdon, and Shashanka Ubaru. Randomized matrix-free quadrature for spectrum and spectral sum approximation.arXiv preprint arXiv:2204.01941, page 55,

    Pith/arXiv arXiv

  42. [2023]

    Optimal linear decay learning rate schedules and further refinements.arXiv preprint arXiv:2310.07831,

    Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements.arXiv preprint arXiv:2310.07831,

    Pith/arXiv arXiv

  43. [2024]

    Understanding optimization in deep learning with central flows.arXiv preprint arXiv:2410.24206,

    Jeremy M Cohen, Alex Damian, Ameet Talwalkar, J Zico Kolter, and Jason D Lee. Understanding optimization in deep learning with central flows.arXiv preprint arXiv:2410.24206,

    arXiv

  44. [2025]

    Hessformer: Hessians at foundation scale.arXiv preprint arXiv:2505.11564,

    Diego Granziol. Hessformer: Hessians at foundation scale.arXiv preprint arXiv:2505.11564,

    Pith/arXiv arXiv

  45. [2026]

    Aaron Defazio

    URLhttps://arxiv.org/abs/2506.00181. Aaron Defazio. Schedulefree+: Scaling learning-rate-free & schedule-free learning to large language models.arXiv preprint arXiv:2605.19095,

    Pith/arXiv arXiv