Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Seesaw establishes that halving a learning rate and doubling the batch size are risk-equivalent moves, and packages that as a drop-in scheduler that matches cosine decay while cutting wall-clock time by about 36%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:32 UTC pith:XBWDGZKM

load-bearing objection Seesaw gives a genuinely new finite-sample equivalence theorem and a clean cosine-derived batch ramp, but the headline 36% wall-clock gain is extrapolated from step counts, and the Adam transfer rests on an unmeasured variance-dominance assumption. the 3 major comments →

arxiv 2510.14717 v2 pith:XBWDGZKM submitted 2025-10-16 cs.LG cs.AImath.OCstat.ML

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling

Alexandru Meterez , Depen Morwani , Jingfeng Wu , Costin-Andrei Oncescu , Cengiz Pehlevan , Sham Kakade This is my paper
classification cs.LG cs.AImath.OCstat.ML
keywords batch size schedulinglearning rate decaynormalized SGDAdamcritical batch sizeLLM pretrainingrisk equivalenceSeesaw
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper puts batch-size ramp-up on the same theoretical footing as learning-rate decay, and turns that theory into a drop-in scheduler called Seesaw. The central claim is a formal equivalence: under standard conditions on noisy linear regression, halving the learning rate and doubling the batch size are interchangeable ways to reduce risk, for both SGD and normalized SGD as a stand-in for Adam. Seesaw applies this rule wherever a cosine schedule would halve the learning rate, multiplying the rate by 1/√2 and doubling the batch instead. On 150M, 300M, and 600M models trained at Chinchilla scale, Seesaw matches cosine decay at equal FLOPs while reducing wall-clock time by about 36%, approaching the theoretical maximum implied by the analysis. The result matters because LLM pretraining is measured in months of wall-clock time, and a principled, parameter-light speedup at constant FLOPs and loss is directly usable.

Core claim

The paper's central discovery is a scaling identity that makes batch size and learning rate interchangeable at the level of risk, not just heuristics. For SGD on noisy linear regression, Theorem 1 shows that two phase-wise schedules with different learning-rate decay factors α and batch-size ramp factors β produce excess risk within a constant factor of each other as long as α·β is held equal, so halving the learning rate is dynamically equivalent to doubling the batch. For normalized SGD, the proxy for Adam, the same equivalence holds with the invariant α·√β, because under a variance-dominated regime the NSGD update reduces to SGD with a rescaled learning rate. Seesaw exploits this: at each

What carries the argument

The load-bearing object is the equivalence invariant between the learning-rate decay factor α and the batch-size ramp factor β: α·β for SGD and α·√β for normalized SGD. It does the work by reducing the NSGD update to plain SGD with a rescaled step η̃ = η√B/(σ√Tr(H)) whenever the expected squared gradient norm is dominated by additive noise, so that risk comparisons between schedules become comparisons of SGD transition matrices. The supporting lemmas fix the limits of the idea: Lemma 4 shows schedules with α < √β diverge, and Lemma 1 integrates the cosine schedule to give the maximum speedup 1 − 2/π ≈ 36.3%.

Load-bearing premise

The load-bearing premise is Assumption 2 (Section 5.1): the expected squared gradient norm is dominated by additive noise scaling as σ²/B; the paper's own Section 4.2 shows this fails above the critical batch size, and with it the equivalence that gives Seesaw its speedup.

What would settle it

Record E‖g_t‖² over training at the batch sizes Seesaw uses; if the mean-gradient term dominates or the value stops scaling as 1/B, the α√β equivalence should break. A cheaper check: compare Seesaw versus cosine final loss at batch sizes 1024, 2048, 4096, and 8192 on a 150M model, where the paper already shows the gap widening, which is the predicted failure signature.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Seesaw is a drop-in replacement for cosine decay: same hyperparameters, same training pipeline, same final loss, fewer serial steps.
  • At or below the critical batch size, the equivalence holds empirically across 150M, 300M, and 600M models, including AdamW with tuned weight decay.
  • The maximum achievable speedup from any loss-preserving batch ramp under cosine decay is 1 − 2/π ≈ 36.3%; Seesaw's measured ≈36% is near that limit.
  • Schedules more aggressive than the invariant α = √β (e.g., doubling the batch without cutting the learning rate enough) diverge, so the invariant pins down the safe operating region.
  • Above the critical batch size, no batch-ramp schedule the paper tests matches learning-rate decay, because the variance-dominated assumption that drives the equivalence no longer holds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The equivalence suggests a direct diagnostic for when Seesaw will work: measure E‖g_t‖² during training and check whether it scales as 1/B; that single measurement could predict at which batch size the speedup disappears.
  • If the risk-equivalence holds beyond linear regression, Seesaw could be composed with other schedule families, such as warmup, cyclical, or data-curriculum schedules, to recover some of the serial time currently given to cosine's gentle tail.
  • The z-loss instability the paper observes at 600M (Appendix E) hints that auxiliary losses with their own steep dynamics need separate handling; a testable extension would scale each auxiliary loss's weight in a way that preserves its own α√β invariant.
  • One could treat Seesaw's 36% as an upper bound for batch-ramp-only acceleration under cosine decay; further wall-clock gains would have to come from other axes, like model parallelism or optimizer state compression.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies batch-size ramp-up as a substitute for learning-rate decay. It formalizes an equivalence for SGD on noisy linear regression: under a bounded-risk assumption, two phase-indexed schedules (ηα1^{-k}, Bβ1^k) and (ηα2^{-k}, Bβ2^k) have excess risk within constant factors whenever α1β1=α2β2 (Theorem 1). The result is extended to normalized SGD under a variance-dominated gradient assumption, yielding the rule α√β = const (Corollary 1). This motivates "Seesaw": at every point where a cosine scheduler would halve the learning rate, multiply the learning rate by 1/√2 and double the batch size. Lemma 1 gives a maximum serial-step reduction of 1−2/π ≈ 36.3% in a continuous-time limit. Experiments on 150M/300M/600M transformers show Seesaw matching cosine decay at equal FLOPs while using fewer serial steps; §4.2 documents that the equivalence fails at large batch sizes.

Significance. If the central equivalence holds in the relevant regime, this is a useful and conceptually clean contribution. The theoretical derivation is non-asymptotic, does not fit constants to produce the 36% number, and is presented with explicit assumptions. The empirical section includes multiple model scales, LR sweeps at several batch sizes, a weight-decay ablation, and an honest failure analysis in §4.2. The main caveat is the transfer from linear regression to AdamW/transformers: it rests on an assumption about gradient statistics that is never measured on the actual LLM runs, and the paper's headline "wall-clock" claim is supported only by serial-step counts. These gaps are substantial but addressable with additional measurements or appropriately qualified claims.

major comments (3)
  1. [§3.1, Corollary 1, Eq. (5)/(7), Appendix B, §4.2] Corollary 1 is the theoretical basis for Algorithm 1, and it is obtained by reducing NSGD to SGD under Assumption 2: E||g_t||² ≂ σ²/B_t. The paper never measures E||g_t||² or its batch-size dependence on the 150M/300M/600M transformer runs. Appendix B validates the variance-dominated scaling only for noisy linear regression, and §4.2 shows that when the assumption fails (large batch sizes), Seesaw no longer matches cosine and no simple rescaling recovers it. Since AdamW's per-coordinate preconditioner is only a proxy for NSGD's global normalization, the α√β rule is not yet directly supported for the optimizer/model class used in the main experiments. Please provide a direct measurement of the mean-variance decomposition in Eq. (5) at the phases where Seesaw cuts, or clearly state that the LLM experiments are an empirical heuristic and that the theory applies only in the validated linear-
  2. [Abstract, §1.2, §6, Figure 1, Lemma 1] The abstract and §6 claim wall-clock time reduction of ≈36%, but no wall-clock timings or throughput measurements are reported anywhere. The bottom row of Figure 1 is labeled in steps, not time, and §4 reports only serial step counts. Doubling the batch size changes per-step cost through data loading, communication, and memory/throughput effects, so step-count reduction does not by itself imply wall-clock reduction under a fixed compute budget. The paper should either report actual elapsed training time (or throughput per phase) or revise the claim to "≈36% reduction in serial steps."
  3. [Lemma 1, §3.2, §4] Lemma 1 derives 2T/π total steps by integrating the normalized cosine learning rate, treating the effective number of steps as ∫ η(t)/η0 dt. This is a formal continuum argument; it does not show that the discrete Algorithm 1, with a step-decay approximation and cuts placed at cosine-halving events, converges to this bound. More substantively, the integral does not enforce the critical-batch-size constraint. In the experiments the critical batch size for the 150M model is ≈256k tokens, so a run starting at B=256 sequences (262k tokens) is already at CBS, and the first Seesaw cut doubles to ≈524k tokens. This places the main experiments partly outside the regime where Assumption 2 and Corollary 1 are expected to hold. The 36% figure is therefore an idealized upper bound, not a proven guarantee for the runs in Figure 1.
minor comments (6)
  1. [§3.1] Typo: "comapre" should be "compare".
  2. [§4.1] Typo: "agressive" should be "aggressive".
  3. [Appendix C, Table 3] The table caption says "weight decay 0.003," but the experiments used λ=0.0001 as the best value. This is inconsistent and should be corrected.
  4. [Figure 1 caption] The bottom row is described as showing "serial runtime" but the x-axis is explicitly labeled "Steps." Please clarify that the claim is about serial steps, not measured wall-clock time.
  5. [Theorem 1] The notation R(1.01·η′_k, B′_k) is used in the theorem statement before it is defined. Define the uniform learning-rate scaling factor before stating the result.
  6. [Appendix E, Figure 7] The paper observes z-loss instabilities when using Seesaw on 600M models, yet all main runs have z-loss enabled. This limitation should be mentioned in the main text, not only in the appendix.

Circularity Check

0 steps flagged

No significant circularity: the SGD/NSGD equivalence is a genuine derivation, the 36% figure is a cosine integral, and the self-citations used are non-load-bearing.

full rationale

The derivation chain is not circular. Theorem 1 is proved in Appendix A by unrolling the eigenbasis covariance recurrence (Eq. 6) and comparing bias/variance terms; the result is a constant-factor equivalence for schedules with the same αβ product, not a restatement of the input. Corollary 1 extends this to NSGD only under the explicitly stated Assumption 2, and Eq. 7 (η̃ ≂ η√B/(σ√Tr(H))) is derived in Appendix B from the gradient-norm calculation; Section 4.2 openly tests and demonstrates where Assumption 2 fails, so the assumption is not being protected from falsification. Lemma 1's ≈36% speedup is the exact integral ∫_0^T cos(πt/2T)dt = 2T/π, a mathematical consequence of the schedule definition, with no fitted parameter renamed as a prediction. Empirically, losses are compared at equal FLOPs, LRs are swept and the best LR for cosine is used for both schedulers, so Seesaw is not tuned to force the match. The self-citations (Meterez et al. 2025 for the eigenbasis technique; Wu et al. 2022b for the maximum convergent SGD step size; Zhang et al. 2024 for CBS) are standard references or tools whose derivations are restated in the paper ('For the sake of completeness, we restate the main derivation...'), and they are not invoked as an external uniqueness proof to rule out alternatives. The z-loss instability reported in Appendix E is an honest limitation, not a circular step. The main residual concern—Assumption 2's validity on actual LLM runs—is an unverified empirical premise, hence a correctness risk, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The theorem is self-contained for linear regression, but the bridge to LLM practice rests on several domain assumptions: linear regression as a proxy, variance-dominated gradient norms, NSGD as Adam, and step-decay approximation of cosine. No fitted constants enter the theory itself; α=1.1 and the LR/batch choices are empirical hyperparameters.

free parameters (3)
  • Step-decay granularity α = 1.1
    Chosen by hand for experiments (Table 1); controls how often the LR decays and batch size doubles. The theory allows any α>1 but does not predict 1.1.
  • Learning rate sweep = Best of {0.001, 0.003, 0.01, 0.03} per batch size
    Standard hyperparameter selection; final losses are reported at the best LR for cosine annealing, and the same LR values are used for Seesaw.
  • Initial batch sizes = 128, 256, 512, 1024 sequences (≈128k-1024k tokens at L=1024)
    Chosen to span the external critical batch size estimate, not derived by the paper.
axioms (6)
  • domain assumption Noisy linear regression model: x∼N(0,H), y|x∼N(⟨w*,x⟩, σ²)
    All theoretical results are proven for this Gaussian linear model; LLM loss landscapes are non-quadratic and non-Gaussian.
  • domain assumption Assumption 1: risk is bounded as R(w_t) ≤ cσ² after the first schedule change
    Used in Theorem 1's proof to control the λλᵀm term; not verified for LLM training.
  • domain assumption Assumption 2: variance-dominated gradient norms, E||g_t||² ≂ σ²/B_t
    This is what reduces NSGD to SGD with a rescaled LR. The paper derives it for linear regression in Appendix B but does not measure it on LLM runs; §4.2 shows it fails above CBS.
  • domain assumption Normalized SGD is a faithful proxy for Adam
    Equation 4 ignores momentum, per-coordinate preconditioning, and bias correction. Corollary 1 therefore applies to a proxy, not to AdamW itself.
  • ad hoc to paper Step decay with α=1.1 approximates cosine decay
    Section 3.2 replaces cosine with a step-decay schedule at α=1.1 intervals; the empirical match to cosine depends on this approximation being good enough.
  • domain assumption External critical batch size estimates (Zhang et al. 2024) locate the valid operating regime
    Experiments are run at batch sizes near the external CBS estimate; if that estimate is off, the claimed valid regime shifts.

pith-pipeline@v1.3.0-alltime-deepseek · 18323 in / 21627 out tokens · 175251 ms · 2026-08-04T09:32:16.468996+00:00 · methodology

0 comments
read the original abstract

Increasing the batch size during training -- a ''batch ramp'' -- is a promising strategy to accelerate large language model pretraining. While for SGD, doubling the batch size can be equivalent to halving the learning rate, the optimal strategy for adaptive optimizers like Adam is less clear. As a result, any batch-ramp scheduling, if used at all, is typically tuned heuristically. This work develops a principled framework for batch-size scheduling and introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/\sqrt{2}$ and doubles the batch size, preserving loss dynamics while reducing serial steps. Theoretically, we provide, to our knowledge, the first finite-sample proof of equivalence between learning-rate decay and batch-size ramp-up for SGD on noisy linear regression, and we extend this equivalence to normalized SGD, a tractable proxy for Adam, under a variance-dominated regime observed in practice. Empirically, on 150M/300M/600M-parameter models trained at Chinchilla scale using a constant (critical) batch size, Seesaw matches cosine decay at equal FLOPs while reducing wall-clock time by $\approx 36\%$, approaching the theoretical limit implied by our analysis.

Figures

Figures reproduced from arXiv: 2510.14717 by Alexandru Meterez, Cengiz Pehlevan, Costin-Andrei Oncescu, Depen Morwani, Jingfeng Wu, Sham Kakade.

Figure 1
Figure 1. Figure 1: Seesaw comparison with cosine decay in 150M (left), 300M (middle) and 600M (right) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: 3.2 Achievable Speedups While our theory is established for step decay schedulers, in practice we approximate cosine decay with a step decay by considering a decay of α, and passing the times (as measured in tokens) where the cosine would cut the learning rate by α as input to Seesaw. Then, at these points, we instead cut the learning rate by √ α and increase the batch size by β, where the schedulers are e… view at source ↗
Figure 2
Figure 2. Figure 2: 150M models trained at batch size 256 (left) and 512 (right) with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: 150M models trained past CBS (roughly 256), at batch sizes 1024, 2048, 4096 and 8192, for 3 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: 150M experiments with weight decay across different batch sizes (128, 256, 512) for cosine [PITH_FULL_IMAGE:figures/full_fig_p023_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: 150M models trained with 4 different schedules, at CBS (right) and just below (left). Blue [PITH_FULL_IMAGE:figures/full_fig_p024_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: 150M models trained with cosine decay in Chinchilla scale, across 3 learning rates and 3 [PITH_FULL_IMAGE:figures/full_fig_p025_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: 600M models trained with Seesaw decay in Chinchilla scale, with Z-Loss. [PITH_FULL_IMAGE:figures/full_fig_p025_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. A Defense of the Quadratic Model

    cs.LG 2026-07 conditional novelty 7.0

    Local Taylor-expanded quadratic models reproduce a 150M-parameter LLM's validation loss for up to 10% of training late in the run, and LLM pretraining operates within a factor of 2 of a stochastic or deterministic edg...

  3. Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model

    cond-mat.dis-nn 2026-02 unverdicted novelty 7.0

    In a random feature model, optimal SGD learning-rate schedules are polynomial decay in the easy phase and warmup-stable-decay in the hard phase, outperforming constant or simple power-law schedules and transferring di...

Reference graph

Works this paper leans on

43 extracted references · 26 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Gemini 2.5 pro capable of winning gold at imo 2025

    Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025. arXiv preprint arXiv:2507.15855, 2025

    arXiv 2025

  2. [2]

    Proof or bluff? evaluating llms on 2025 usa math olympiad

    Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunovi \'c , Nikola Jovanovi \'c , and Martin Vechev. Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Competitive programming with large reasoning models

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  5. [5]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Data movement limits to frontier model training

    Ege Erdil and David Schneider-Joseph. Data movement limits to frontier model training. arXiv preprint arXiv:2411.01137, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Large batch training of convolutional networks

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017

    Pith/arXiv arXiv 2017

  8. [8]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    Pith/arXiv arXiv 2017

  9. [9]

    How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024

    Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham Kakade. How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    Pith/arXiv arXiv 2018

  11. [11]

    Measuring the effects of data parallelism on neural network training

    Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20 0 (112): 0 1--49, 2019

    2019

  12. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv--2407, 2024

    2024

  13. [13]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Nemotron-4 340b technical report

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Apertus: Democratizing Open and Compliant LLMs for Global Language Environments

    Apertus Team. Apertus: Democratizing Open and Compliant LLMs for Global Language Environments . https://huggingface.co/swiss-ai/Apertus-70B-2509, 2025

    2025

  17. [17]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification

    Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research, 18 0 (223): 0 1--42, 2018

    2018

  19. [19]

    Is your batch size the problem? revisiting the adam-sgd gap in language modeling

    Teodora Sre \'c kovi \'c , Jonas Geiping, and Antonio Orvieto. Is your batch size the problem? revisiting the adam-sgd gap in language modeling. arXiv preprint arXiv:2506.12543, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful

    Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful. arXiv preprint arXiv:2507.07101, 2025

    arXiv 2025

  21. [21]

    Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model

    Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019

    2019

  22. [22]

    A markov chain theory approach to characterizing the minimax optimality of stochastic gradient descent (for least squares)

    Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, Venkata Krishna Pillutla, and Aaron Sidford. A markov chain theory approach to characterizing the minimax optimality of stochastic gradient descent (for least squares). arXiv preprint arXiv:1710.09430, 2017

    Pith/arXiv arXiv 2017

  23. [23]

    Benign overfitting of constant-stepsize sgd for linear regression

    Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Benign overfitting of constant-stepsize sgd for linear regression. In Conference on Learning Theory, pages 4633--4635. PMLR, 2021

    2021

  24. [24]

    Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression

    Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression. In International Conference on Machine Learning, pages 24280--24314. PMLR, 2022 a

    2022

  25. [25]

    The power and limitation of pretraining-finetuning for linear regression under covariate shift

    Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. The power and limitation of pretraining-finetuning for linear regression under covariate shift. Advances in Neural Information Processing Systems, 35: 0 33041--33053, 2022 b

    2022

  26. [26]

    A simplified analysis of sgd for linear regression with weight averaging

    Alexandru Meterez, Depen Morwani, Costin-Andrei Oncescu, Jingfeng Wu, Cengiz Pehlevan, and Sham Kakade. A simplified analysis of sgd for linear regression with weight averaging. arXiv preprint arXiv:2506.15535, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Learning curves for sgd on structured features

    Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd on structured features. arXiv preprint arXiv:2106.02713, 2021

    Pith/arXiv arXiv 2021

  28. [28]

    Finite-sample analysis of learning high-dimensional single relu neuron

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Finite-sample analysis of learning high-dimensional single relu neuron. In International Conference on Machine Learning, pages 37919--37951. PMLR, 2023 a

    2023

  29. [29]

    How many pretraining tasks are needed for in-context learning of linear regression? arXiv preprint arXiv:2310.08391, 2023 b

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? arXiv preprint arXiv:2310.08391, 2023 b

    Pith/arXiv arXiv 2023

  30. [30]

    On the validity of modeling sgd with stochastic differential equations (sdes)

    Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). Advances in Neural Information Processing Systems, 34: 0 12712--12725, 2021

    2021

  31. [31]

    A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima

    Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. arXiv preprint arXiv:2002.03495, 2020

    Pith/arXiv arXiv 2002

  32. [32]

    Adaptive methods through the lens of sdes: Theoretical insights on the role of noise

    Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of sdes: Theoretical insights on the role of noise. arXiv preprint arXiv:2411.15958, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Three factors influencing minima in sgd

    Stanis aw Jastrz e bski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017

    Pith/arXiv arXiv 2017

  34. [34]

    On the sdes and scaling rules for adaptive gradient algorithms

    Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35: 0 7697--7711, 2022

    2022

  35. [35]

    Learning rates as a function of batch size: A random matrix theory approach to neural network training

    Diego Granziol, Stefan Zohren, and Stephen Roberts. Learning rates as a function of batch size: A random matrix theory approach to neural network training. Journal of Machine Learning Research, 23 0 (173): 0 1--65, 2022

    2022

  36. [36]

    Large batch optimization for deep learning: Training bert in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019

    Pith/arXiv arXiv 1904

  37. [37]

    Power lines: Scaling laws for weight decay and batch size in llm pre-training

    Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scaling laws for weight decay and batch size in llm pre-training. arXiv preprint arXiv:2505.13738, 2025

    arXiv 2025

  38. [38]

    Don't decay the learning rate, increase the batch size

    Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017

    Pith/arXiv arXiv 2017

  39. [39]

    Critical batch size revisited: A simple empirical approach to large-batch language model training

    William Merrill, Shane Arora, Dirk Groeneveld, and Hannaneh Hajishirzi. Critical batch size revisited: A simple empirical approach to large-batch language model training. arXiv preprint arXiv:2505.23971, 2025

    arXiv 2025

  40. [40]

    Dissecting adaptive methods in gans

    Samy Jelassi, David Dobre, Arthur Mensch, Yuanzhi Li, and Gauthier Gidel. Dissecting adaptive methods in gans. arXiv preprint arXiv:2210.04319, 2022

    Pith/arXiv arXiv 2022

  41. [41]

    Deconstructing what makes a good optimizer for language models

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Adam exploits _ -geometry of loss landscape via coordinate-wise adaptivity

    Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits _ -geometry of loss landscape via coordinate-wise adaptivity. arXiv preprint arXiv:2410.08198, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020