{"id":"9f1b9d90-56cd-466d-8191-d126fcb37171","arxiv_id":"2506.12543","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.","lead":"This paper shows that SGD with momentum can match Adam's performance in language model training when the batch size is small enough and hyperparameters are tuned carefully. The finding challenges common explanations for why Adam beats SGD and offers practical guidance for low-resource training runs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-run selection under instability makes the SGD-Adam parity claim fragile: at small batch sizes, SGD's reported advantage may reflect a lucky stable run rather than typical performance, and no seed variance is reported.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the most load-bearing vulnerability. The paper itself flags the instability issue: optimal SGD runs are chosen where \"one run may succeed, even if the median run diverges.\" Because the abstract and Section 2.2 claim SGD can match or outperform Adam, the validity of that claim depends on whether the reported runs represent typical performance or favorable draws near the instability boundary. No seed averaging is reported for the Transformer experiments, so the magnitude of run-to-run variance is unknown. This is not an internal inconsistency; it is a genuine empirical robustness gap. The concrete test of re-running with multiple seeds and comparing medians would settle whether the parity claim survives a stricter protocol. The theoretical section, while heuristic in connecting SignSGD to Adam, is not the main load-bearing element; the empirical claim stands or falls on the evaluation protocol. I therefore agree with the reader's conditional verdict and recommend no change.","tokens_in":17651,"tokens_out":3723,"duration_ms":46355,"concrete_test":"Re-run the small-batch configurations from Table 1 (batch size 64, SGD with momentum 0.98 and lr=0.5, plus the corresponding Adam run) and the 410M/1B runs from Figure 5 with at least 5 independent seeds each, reporting median and interquartile range of final validation perplexity. If median SGD final validation PPL at batch size 64 exceeds median Adam by more than ~2 PPL, or if any SGD seed diverges or lags substantially, the parity claim fails under a typical-performance protocol; if median SGD is within noise of median Adam, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—SGD with momentum can match Adam in small-batch language modeling—rests on comparing single runs selected under an explicitly instability-prone protocol. Section 2.2 states: \"Some settings become unstable at very large learning rates, where one run may succeed, even if the median run diverges. In those settings, we report runs at the largest stable learning rate as optimal.\" This means the reported SGD results at batch size 64 (Table 1: PPL 30.76, lr=0.5) and the headline 410M/1B runs in Figure 5 may be draws from a distribution whose median is much worse. The paper reports no seeds, no interquartile ranges, and no median-vs-best comparison for any Transformer run. If SGD's small-batch advantage is concentrated in a narrow region near the instability boundary, then (a) the parity claim is not robust to a typical-performance evaluation protocol, and (b) the subsequent conclusions about batch-size dependence and the failure of prior explanations inherit that fragility. The theoretical bridge via SignSGD (Section 4.3) is explicitly heuristic for Adam and does not fix this empirical vulnerability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits the commonly observed performance gap between Adam and SGD in Transformer language modeling. Through controlled sweeps on nanoGPT-style models (160M up to 1B parameters) trained on SlimPajama and FineWeb, the authors report that SGD with momentum, gradient clipping, and carefully tuned hyperparameters can match or even outperform Adam when the batch size is small, while the gap reappears at larger batch sizes. They then test several existing explanations (heavy-tailed class imbalance, directional sharpness, Hessian heterogeneity), find them insufficient to explain the small-batch result, and propose an SDE-based account in which signed-gradient methods (and by extension Adam) benefit from larger batches through an erf-shaped drift term, whereas plain SGD does not. The central empirical claim is that batch size, not just optimizer choice, is a decisive factor in the Adam-SGD gap.","tokens_in":17901,"tokens_out":4212,"duration_ms":50952,"significance":"If the central claim is robust, the paper provides a valuable and somewhat counterintuitive result: at small batch sizes, SGD with momentum can be competitive with Adam in Transformer language modeling at the 410M and 1B parameter scale. This would reframe the optimizer gap as a batch-size-dependent phenomenon, with practical implications for memory-constrained training and for theories of adaptive methods. The paper is also useful as a systematic empirical probe of prior explanations, and it makes an explicit, falsifiable prediction: that SGD's performance degrades with batch size while Adam's improves, up to a critical batch size. The authors are transparent about the heuristic bridge from SignSGD to Adam in the theory, which is a strength in presentation, though it leaves the theoretical contribution incomplete.","major_comments":[{"comment":"The evaluation protocol does not support the headline parity claim as stated. The text says: \"Some settings become unstable at very large learning rates, where one run may succeed, even if the median run diverges. In those settings, we report runs at the largest stable learning rate as optimal.\" This is best-run selection under instability, and no seed variance is reported anywhere in Section 2 or for Figure 5. Table 1's SGD result at batch size 64 (PPL 30.76, lr=0.5) and the 410M/1B trajectories in Figure 5 could therefore be draws from a distribution whose median is much worse. Please rerun the key comparisons with at least 3-5 seeds, report median and interquartile ranges or full seed curves, and define the stability criterion explicitly (e.g., the fraction of runs that diverge). Without this, the central claim that SGD can match or outperform Adam in small-batch settings is not established.","section":"Section 2.2, Table 1, Figure 5"},{"comment":"The learning-rate grids are not independently tuned per batch size, despite the abstract's claim of \"comprehensively tuned\" baseline runs. In Section 2.2, the grid is tuned at batch size 256 and then scaled to batch sizes 64 and 1024 with only three values per batch size. In Section 2.3, fixed grids [0.25, 0.5, 1.0] for SGD and [0.001, 0.002, 0.004] for Adam are used across batch sizes. The reported Adam-SGD gap may therefore be an artifact of insufficient Adam tuning at small batch sizes or insufficient SGD tuning at large batch sizes. Please perform per-batch-size sweeps over wider grids, or at least show a learning-rate response curve for each optimizer and batch size, and justify that the scaled grids cover each optimizer's optimal region.","section":"Section 2.2 and Section 2.3"},{"comment":"The theoretical claim that large batch sizes accelerate Adam is not directly supported by the stated theorem. Theorem 1 is a result for SignSGD, and the paper says the conclusion holds \"in signSGD (and likely also in closely-related algorithms, like Adam)\". The quadratic experiments in Figure 9 compare SGD and SignSGD+m, not Adam, so the bridge from the theorem to the actual Adam behavior is a heuristic analogy. This is acknowledged in the text, but it is load-bearing for the paper's proposed explanation of the empirical results. Please state this limitation more prominently, or provide a formal or at least direct empirical argument that the erf drift acceleration carries over to Adam's coordinate-wise normalization rather than only to sign-based updates.","section":"Section 4.3, Theorem 1 and Figure 9"},{"comment":"The scaling experiments that support \"SGD can outperform Adam even at a 410M and 1B scale\" are presented using training perplexity trajectories (Figure 5), not held-out validation perplexity. The learning-rate sweeps in Figure 14 do report validation perplexity, but those appear to be single-run evaluations. Since the headline claim is about final model quality, please report final validation perplexities for the chosen runs, together with seed information, and clarify whether the outperformance holds on held-out data or only on the training set.","section":"Section 2.3, Figure 5, Figure 14"}],"minor_comments":[{"comment":"There is a typo: \"Leraning rates\" should be \"Learning rates\".","section":"Appendix D"},{"comment":"The phrase \"its critical batch size is close to 1\" is used without a definition. Please define what is meant by \"critical batch size\" in this context (e.g., the threshold beyond which larger batches no longer reduce the number of steps to a given loss).","section":"Section 2.3"},{"comment":"The sentence \"As a sequence length of 160k, given our lack of experience with extremely low batch sizes (shorter sequence length)\" is garbled; please rewrite to state the actual configuration used for the Fineweb 160M experiments.","section":"Appendix A.2"},{"comment":"The x-axis label \"10 160804020\" appears malformed; it should read something like \"Steps (\\times 10^3)\" with tick labels 10, 20, 40, 80, 160.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical contribution with a clear and honest presentation of its instability caveat, but the central claim rests on best-run selection and single-seed comparisons. This is fixable with additional experiments, so I recommend major revision rather than rejection. I see no attribution concern: Theorem 1 is properly cited from Compagnoni et al. (2025b), and the paper explicitly labels the Adam extension as heuristic. The main risk is that the reported SGD-Adam parity may shrink or disappear under a median-run protocol with per-batch-size tuned grids; the revision should address this directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the empirical claim that the Adam-SGD gap in Transformer language modeling is not a constant: it shrinks as batch size shrinks and as training lengthens. That pattern shows up across multiple batch sizes, model scales (160M to 1B), two datasets, and a lot of carefully swept runs. The paper also does a fair job of going back to prior explanations—heavy-tailed class imbalance, directional sharpness, Hessian heterogeneity—and showing that none of them, as stated, predicts the small-batch success of SGD. The grafting and adaptive-clipping experiments are a nice sanity check that the problem with large-batch SGD is update direction, not magnitude. The SDE theory is clearly labeled as a heuristic for Adam; the proven part is for SignSGD and it is borrowed from Compagnoni et al., which is fine, but the leap to Adam is speculative and the paper mostly says so.\n\nThe soft spots are real and concentrated in the evaluation protocol. Section 2.2 explicitly says that when settings are unstable they “report runs at the largest stable learning rate as optimal,” which is best-run selection under instability. No seeds are reported for any Transformer run. So the 410M and 1B “SGD outperforms Adam” results in Figure 5 could be a lucky stable trajectory rather than typical behavior. Even in Table 1, at batch size 64 Adam still wins by about 2 PPL (28.77 vs 30.76), so “match” is already an overstatement. The learning rate grids are also coarse: tuned at batch size 256 and then shifted by only three values, which could hurt Adam as much as SGD but still weakens the parity comparison.\n\nThat said, the qualitative direction is probably right. The gap clearly grows with batch size under fixed token budget, and SGD improves with more steps; that holds across many runs in Figures 3, 16, and 17. So I would not dismiss the paper. I would send it to reviewers, but with a strong request that the authors report median performance and variance, or at least justify why best-run is the right quantity. The code link points to a generic repo, not a tagged release of these experiments, which is also worth asking for.\n\nThis paper is for people who care about optimizer dynamics in LLM training, and for theorists who want a concrete batch-size-dependent phenomenon to explain. I would bring it to a reading group, and I would probably cite it for the batch-size dependence, with a caveat about the protocol. It deserves a serious referee, but it needs revision before I would trust the quantitative parity claim.","headline":"Batch size is a real and informative axis of the Adam-SGD gap in LMs, but the headline parity claim is built on best-run selection and no seed variance, so treat the quantitative claim as provisional.","tokens_in":18389,"tokens_out":2513,"would_cite":true,"duration_ms":31030,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuned SGD with momentum matches Adam on language models when the batch is small, and the paper argues batch size rather than architecture or data explains the optimizer gap.","keywords":["optimizer gap","Adam vs SGD","batch size","language modeling","stochastic differential equations","momentum","gradient clipping","critical batch size"],"falsifier":"Run the small-batch comparison (e.g., batch size 64, 160M model on SlimPajama) with at least five seeds per configuration, tuning the learning rate grid separately at that batch size, and compare the median final validation perplexity of SGD with momentum against Adam; if the median SGD perplexity is clearly worse than Adam's, then the parity claim holds only for best-luck runs, not for typical training.","tokens_in":17468,"feed_emoji":"📉","tokens_out":5840,"duration_ms":62728,"temperature":0.7,"pith_summary":"This paper tries to establish that the perceived superiority of Adam over SGD in Transformer language modeling is not a fixed property of the optimizer, the architecture, or the text data: it is largely a batch-size effect. With momentum, a tuned learning rate, and gradient clipping, SGD can match Adam's final perplexity when the batch is small, and the paper reports runs where SGD beats Adam at 410M and 1B parameters. It then argues that existing explanations, such as heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, cannot account for why SGD suddenly works at small batches. The paper proposes an SDE-based mechanism in which adaptive and signed methods gain a drift that grows with the square root of batch size, while plain SGD's early progress depends only on the number of steps. If true, this reframes the optimizer gap as a question of gradient noise scale and critical batch size.","feed_headline":"SGD can beat Adam at small batch sizes","feed_subtitle":"Carefully tuned momentum, clipping, and tiny batches let SGD match or surpass Adam on gigascale language models.","key_machinery":"The load-bearing object is the stochastic differential equation (SDE) approximation of SGD and of sign-based updates. For signSGD the paper cites a theorem giving the drift term as $-\\operatorname{erf}\\big(\\sqrt{B/2}\\,\\Sigma^{-1/2}\\nabla f(x)\\big)$, which is approximately linear in $\\sqrt{B}$ near zero and saturates beyond a critical batch size, while SGD's drift is just $-\\nabla f(x)$ with noise $\\sqrt{\\eta\\Sigma/B}\\,dW_t$. This drift-versus-batch-size separation explains both why Adam-type methods improve with larger batches and why SGD's small-batch success is really an iteration-count effect. A secondary instrument is the heterogeneous quadratic testbed with block-diagonal Hessians, used to show the batch-size effect appears even without Transformer-specific structure, and grafting experiments that isolate the update direction as the failing component of large-batch SGD.","core_discovery":"The central claim is that SGD with momentum can be made competitive with Adam in Transformer language modeling, provided the batch size is small and both momentum and learning rate are properly tuned: at batch size 64 and a 1.3B-token budget, a 160M model reaches final validation perplexity close to Adam's (30.76 vs 28.77), and at 410M and 1B parameters SGD can actually outperform Adam when trained with very small batches for many steps. The paper further argues that the reason is not any of the previously proposed mechanisms, all of which predict a gap that should persist at small batch sizes; instead, the gap is governed by how each optimizer responds to gradient noise. For signed and adaptive updates, the drift in the SDE limit scales as an error function of the square root of the batch size times the preconditioned gradient, giving an acceleration with batch size up to a critical point, whereas SGD's drift is batch-size independent, so its progress is set by iteration count.","pith_inferences":["Extension beyond the paper: if batch size controls the gap, then compute-optimal training recipes that rely on very large batches may systematically favor Adam, while low-resource regimes that use tiny batches are precisely where simpler optimizers could win.","Extension beyond the paper: the SDE analysis implies that other sign-based or compressed-gradient methods should inherit Adam's batch-size acceleration; a direct test would compare SignSGD and Adam across batch sizes and check whether their gap follows the predicted $\\sqrt{B}$ drift scaling.","Extension beyond the paper: the observation that clipping occurs most often in normalization layers at large batch sizes suggests that per-coordinate scaling of a few outlier coordinates is the active ingredient, pointing to a possible middle-ground optimizer that adaptively clips only rare large coordinates instead of maintaining full Adam states."],"forward_implications":["In small-batch regimes, SGD with momentum is a memory-cheaper alternative to Adam for training language models, since it does not store second-order moment estimates.","The Adam-SGD gap is not intrinsic to Transformers or text data; batch size and gradient noise scale are first-order moderators of the gap.","Increasing batch size accelerates Adam-type methods through a $\\sqrt{B}$ drift effect up to a critical batch size, while SGD gains nothing from larger batches in early training.","Prior explanations of the gap, including heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, are incomplete because they predict a gap that should remain even where SGD is observed to work well.","At scale, SGD can outperform Adam under small-batch, long-step training, suggesting that optimizer choice interacts with compute budget and memory constraints rather than being fixed by architecture alone."],"supporting_citations":[{"why":"Shows the Adam-SGD gap persists even in full-batch training, ruling out gradient noise as the sole cause and motivating the paper's focus on batch-size-dependent dynamics.","marker":"Kunstner et al. [2023]"},{"why":"Proposes heavy-tailed class imbalance as the explanation for Adam's advantage; the paper reproduces and tests this explanation and finds it insufficient at small batch sizes.","marker":"Kunstner et al. [2024]"},{"why":"Provides the Hessian-heterogeneity perspective and the block-diagonal quadratic test problem that the paper extends with batch-size variation.","marker":"Zhang et al. [2024]"},{"why":"Introduces directional sharpness as a metric for the optimizer gap; the paper measures it and shows it correlates with but does not explain the batch-size-dependent outcome.","marker":"Pan and Li [2023]"},{"why":"Supplies the SDE theorem with the error-function drift term that is the paper's main theoretical tool for batch-size acceleration of adaptive and signed methods.","marker":"Compagnoni et al. [2025b]"},{"why":"Provides the grafting technique used to swap update direction and magnitude between SGD and Adam, isolating direction as the problematic component in large-batch SGD.","marker":"Agarwal et al. [2020]"},{"why":"Documents that adaptive optimizers are stable across hyperparameters while SGD is sensitive, and motivates the high-momentum (0.98) setting used for SGD throughout the paper.","marker":"Zhao et al. [2024]"},{"why":"Defines signSGD, the signed-momentum method whose SDE behavior the theoretical section uses as a proxy for understanding Adam's batch-size scaling.","marker":"Bernstein et al. [2018]"}],"fun_headline_variants":["Small batches help SGD beat Adam in language models","SGD catches up to Adam with small batches","Adam's edge over SGD vanishes at small batch sizes","Tiny batches let SGD match Adam in Transformers","The optimizer gap is really a batch-size effect"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that SGD matches Adam rests on reporting each optimizer's best stable run rather than its typical run, with learning-rate grids tuned at one batch size and only roughly rescaled for others; under a stricter median-seed, per-batch-size protocol, the small-batch parity may vanish.","fun_headline_variants_meta":{"raw":{"variants":["Small batches help SGD beat Adam in language models","SGD catches up to Adam with small batches","Adam's edge over SGD vanishes at small batch sizes","Tiny batches let SGD match Adam in Transformers","The optimizer gap is really a batch-size effect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4380,"prompt_tokens":904,"completion_tokens":3476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":3403}},"tokens_in":520,"tokens_out":3476,"duration_ms":26349,"temperature":1.0,"reasoning_tokens":3403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:45:51.854802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the small-batch comparison (e.g., batch size 64, 160M model on SlimPajama) with at least five seeds per configuration, tuning the learning rate grid separately at that batch size, and compare the median final validation perplexity of SGD with momentum against Adam; if the median SGD perplexity is clearly worse than Adam's, then the parity claim holds only for best-luck runs, not for typical training.","supporting_citations":[{"cited_title":"[2024], and uses the codebase of Orvieto and Gower [2025]","cited_arxiv_id":null,"evidence_quote":"Provides the Hessian-heterogeneity perspective and the block-diagonal quadratic test problem that the paper extends with batch-size variation."}],"review_version":1}