{"id":"fba64dea-12f1-498e-977a-5934d8bafee7","arxiv_id":"2502.02431","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Schedule-Free SGD and AdEMAMix reduce to accelerated SGD variants, and a simplified one-momentum AdEMAMix matches the original on a 150m transformer.","lead":"The paper shows that Schedule-Free optimizers and AdEMAMix can be rewritten as accelerated stochastic gradient descent followed by weight averaging. It also proposes Simplified-AdEMAMix, a one-momentum optimizer that matches AdEMAMix on a 150 million parameter language model.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AdEMAMix-to-accelerated-SGD connection rests on the β1=0 proxy; this paper's own Figure 3 shows that proxy fails at large batch, and no seeded small-batch evidence establishes it, so the central claim currently applies only to a special case.","rationale":"The reader's conditional verdict is reasonable. The most central theoretical claim—Schedule-Free SGD as accelerated SGD with weight averaging—is an algebraic equivalence and survives scrutiny; the paper should get credit for that. The AdEMAMix claim is different: it is not derived for the full algorithm, only for the β1=0 variant, and the paper itself provides evidence (Figure 3) that this variant is not faithful outside the small-batch setting. Since the abstract and introduction make a practical claim about AdEMAMix's superior performance and attribute it to its resemblance to accelerated SGD, the β1=0 proxy is load-bearing. The missing piece is not the algebra but a quantitative test of the proxy in the claimed regime. The proposed check directly tests that. No formal verification exists, and the empirical curves are best-run without error bars, so the conditional verdict should stand; if the proxy check fails, the AdEMAMix portion of the central claim should be downgraded to a statement about a special case.","tokens_in":10227,"tokens_out":14324,"duration_ms":128426,"concrete_test":"Run full AdEMAMix and AdEMAMix-with-β1=0 on the same 150M-parameter C4 decoder-only model used in the paper, at batch sizes 32k, 128k, 256k, and 1M tokens, matching the token budgets and hyperparameter sweeps of Appendix A, with at least 3 seeds per configuration. Report mean and standard deviation of validation loss at matched token counts. If the β1=0 curve is within one standard deviation of full AdEMAMix at 32k tokens (and ideally at intermediate batch sizes), the proxy is supported in the regime where the accelerated-SGD interpretation is claimed; if it is not, the central connection describes only a degenerate special case and the explanation for AdEMAMix's practical performance is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The algebraic derivation in Section 4.1 showing that Schedule-Free SGD's y_t follows Eq. (1) is self-contained and convincing; I do not see a flaw in that portion. The load-bearing weak point is the AdEMAMix connection in Section 4.4. That connection is proven only for Algorithm 1 with β1=0, because with β1>0 the update contains two momentum accumulators plus the current gradient, which does not fit Eq. (1) without an additional folding step that the paper does not provide. The paper justifies the β1=0 reduction by citing Pagliardini et al. (2024) and by one visual comparison, but it presents no derivation and no repeated-seed measurements. Figure 3 in this same paper shows the proxy degrading sharply at batch size 1M tokens, and the paper's own explanation is that full AdEMAMix benefits from the fast momentum term exactly in the regime where the proxy fails. Therefore the assertion that 'AdEMAMix most closely resembles accelerated SGD and therefore performs best' is not established for the practical algorithm; it is established, at most, for a special case whose faithfulness in the claimed small-batch regime is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims three main things. First, it derives an algebraic equivalence showing that Schedule-Free SGD's iterate y_t satisfies the accelerated-SGD update of Eq. (1), with the evaluation iterate x_t being an exponentially weighted average of y_t. Second, it argues that Lion, MARS, and AdEMAMix can also be viewed as preconditioned accelerated-SGD variants, with AdEMAMix (via a β1=0 proxy) most closely matching the accelerated-SGD template. Third, it proposes Simplified-AdEMAMix, an optimizer with a single momentum buffer plus a fixed current-gradient weight α, and reports 150M-parameter language-model experiments claiming that Simplified-AdEMAMix matches AdEMAMix in both small- and large-batch regimes. The paper's central conceptual contribution is the precise connection between Schedule-Free SGD and accelerated SGD.","tokens_in":10475,"tokens_out":14334,"duration_ms":105809,"significance":"If the claims held up, the paper would provide a useful unification: Schedule-Free optimizers, Lion, MARS, and AdEMAMix would be seen as instances of a common accelerated-SGD design space. The Section 4.1 derivation is self-contained, correct, and a genuine contribution. However, the empirical evidence is currently too weak to support the comparative claims, and the AdEMAMix connection is established only for a special case. The paper is honest about the β1=0 proxy's failure at large batch, but it does not resolve the mismatch between the theory (which applies to the proxy) and the headline empirical claims (which are about the full algorithm).","major_comments":[{"comment":"The connection between AdEMAMix and accelerated SGD is derived only for the special case β1=0. With β1>0, the AdEMAMix update is θ_t ← θ_{t-1} - η(\\hat m_1^{(t)} + α m_2^{(t)})/√\\hat ν_t, which contains two momentum accumulators plus the current gradient and does not reduce to the single-momentum form of Eq. (1) by any transformation shown in the paper. The paper's justification that the β1=0 variant is nearly equivalent relies on a citation to Pagliardini et al. (2024) and on one unseeded visual comparison, while the paper's own Figure 3 shows that the proxy fails at batch size 1M tokens. Consequently, the statement that the practical AdEMAMix algorithm 'most closely resembles accelerated SGD' is not established; it is established only for a special case. Please provide either an exact folding of the two momentum terms into Eq. (1) or repeated-seed comparisons demonstrating that the β1=0 proxy faithfully represents full AdEMAMix in the small-batch regime.","section":"Section 4.4, Algorithm 1, Eq. (1)"},{"comment":"The large-batch result for Simplified-AdEMAMix in Figure 3 is obtained at α=0.0, as stated in Section 5.1. At α=0.0, Algorithm 2 reduces to standard Adam with a scheduled momentum coefficient, so the experiment demonstrates that Adam with momentum scheduling can match AdEMAMix at large batch (the paper's contribution 3) rather than validating the proposed current-gradient weight α. The small-batch experiments use α∈{10,20,50,100} (Appendix A), so the configuration that achieves the claimed match differs qualitatively between the two regimes. The claim that Simplified-AdEMAMix 'maintains the same performance as AdEMAMix across both large and small batch-size settings' should be restricted to α=0 at large batch, or large-batch runs with nonzero α should be reported.","section":"Section 5.1, Figure 3"},{"comment":"All empirical results are reported as best runs with no seeds or error bars. The paper's headline comparative claims—that AdEMAMix 'exhibits superior performance' and that Simplified-AdEMAMix 'matches' it—are therefore supported only by single trajectories. Since the differences between the optimizers in Figure 1 appear small (e.g., a spread on the order of 0.02 in validation loss), repeated-seed experiments are needed to establish that the ordering is not noise. Even one seed count per condition with a variance estimate across, say, three seeds would substantially strengthen the claims.","section":"Section 5, Figures 1-3"},{"comment":"The statement that 'the prescribed schedules for β3 and α in AdEMAMix closely match theoretical schedules proposed for accelerated SGD (Gupta et al., 2023)' is not backed by a derivation. The paper notes that β3 approximates 1−k/t and that α scales proportionally to t, but it does not specify the required k or the proportionality constant, nor does it show how AdEMAMix's schedule maps to the specific coefficients in Gupta et al.'s algorithm. As this is one of the two 'precise theoretical connections' promised in the introduction, the connection needs a formal statement (or at least a precise table of coefficient mappings) rather than a qualitative parallel.","section":"Section 4.4, 'prescribed schedules' claim"}],"minor_comments":[{"comment":"Equation (2) defines m_{t+1} = (x_t − z_{t+1})/γ, but Eq. (3) and subsequent equations use m_t with the same right-hand side; please reindex so that the momentum variable in Eq. (7) and Eq. (11) is consistent.","section":"Section 4.1, Eq. (2)-(3)"},{"comment":"The title is typeset as 'A DE-MAM IX' in the header; please correct to 'AdEMAMix'.","section":"Title"},{"comment":"Figure 1's legend 'AdamW-wavg' is ambiguous: it could refer to item 3 (AdamW with constant fraction weight averaging) or item 4 (AdamW with cosine decay and weight averaging). Please clarify which configuration is plotted.","section":"Appendix A, item 4 and Figure 1"},{"comment":"The definition γ_t ≈ 1 − 1/(δt) with '0 ≤ δ ≤ 1' is problematic for δ=0; please state the allowed range of δ (e.g., δ∈(0,1]).","section":"Section 3.2"},{"comment":"The small-batch description says 'a batch size of 32' while the introduction says 'batch size of 32k tokens'; please make the units consistent (presumably 32 sequences of length 1024).","section":"Section 5"},{"comment":"The subsection title 'M ASS' should be 'MASS', and the method 'AGNES' from Gupta et al. (2023) is not defined in the main text or reference list; please add a brief description or citation for the reader.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper's biggest gap is the mismatch between the theoretical claim (for β1=0) and the empirical claim (for full AdEMAMix). The authors are transparent about the proxy's limitations, but the central 'AdEMAMix=accelerated SGD' framing overstates what is proven. The single-run experiments are a concern for a full paper; if the venue allows preliminary results, this might be acceptable, but for a serious journal I would require at least one repeated-seed experiment. The code release is a plus. The 'Simplified-AdEMAMix' result at α=0 essentially reduces to a known Adam variant, which should be acknowledged more prominently in the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The part of this paper worth remembering is Section 4.1. The rewrite of Schedule-Free SGD as accelerated SGD followed by weight averaging (Eqs. 2-11) is clean, self-contained, and, as far as I know, not in the earlier literature. It makes the relationship to Defazio (2021) precise and gives the separate works on accelerated SGD a concrete bridge to practical optimizers. That derivation will survive review and is a real contribution. The citation pattern is honest: Defazio (2021) is credited for anticipating some connections.\n\nThe rest of the paper is thinner. The AdEMAMix correspondence in Section 4.4 is explicitly for the beta1 = 0 variant. That's a legitimate special case to analyze, but the paper leans on it to say AdEMAMix \"most closely resembles\" accelerated SGD and therefore performs best. The justification for beta1 = 0 being faithful is a citation plus one visual comparison, with no repeated-seed measurements. The paper's own Figure 3 shows that the beta1 = 0 proxy degrades sharply at large batch size, and the authors' explanation is that the fast momentum term matters precisely there. So the practical-AdEMAMix story is at best heuristic, and in the small-batch regime where it is claimed to hold, the evidence is one unseeded curve.\n\nThe proposed Simplified-AdEMAMix is also undersold by its own results. At large batch, the best configuration has alpha = 0, which makes it standard Adam with a momentum schedule. Matching AdEMAMix by essentially becoming AdamW with the right beta1 schedule is an interesting observation, but it is not the \"new optimizer\" the abstract implies.\n\nEmpirically, everything is best runs with no seeds or error bars, so the comparative claims in Figures 1-3 should be treated as suggestive. That is a soft spot, not a fatal one, because the theoretical core does not depend on the experiments.\n\nWho should read this: optimization theorists and people designing optimizers for LLM training. It reframes Schedule-Free SGD and gives a usable bridge to accelerated SGD. It will not change practice overnight, but the unification is useful and likely citable. I would send it to review. The authors should be asked to add seeded runs, to state plainly that the AdEMAMix connection holds only for beta1 = 0, and to frame Simplified-AdEMAMix as \"Adam with scheduled momentum matches AdEMAMix\" rather than a new algorithm. With those changes it is a solid workshop or conference paper.","headline":"Clean algebraic unification of Schedule-Free SGD with accelerated SGD, but the AdEMAMix connection and the new optimizer's claims rest on a special case that the paper's own experiments show is brittle.","tokens_in":11049,"tokens_out":2415,"would_cite":true,"duration_ms":22996,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Schedule-Free SGD and AdEMAMix sit inside a single accelerated-SGD framework, the paper argues.","keywords":["accelerated SGD","Schedule-Free SGD","AdEMAMix","momentum","weight averaging","noise-dominated regime","AdamW","deep learning optimization"],"falsifier":"Train the 150m model at 32k-token batch with full AdEMAMix ($\\beta_1=0.9$) and with $\\beta_1=0$ using the paper's sweeps; if the $\\beta_1=0$ run's validation loss separates from the full run by more than run-to-run noise, the central AdEMAMix claim is confined to a special case. Alternatively, record $x_t,z_t,y_t,g_t$ from a Schedule-Free SGD run and check the identity $y_{t+1}=y_t-\\gamma[\\beta c_{t+1}m_t+(1-\\beta)g_t]$ with $m_t=(x_{t-1}-z_t)/\\gamma$; a persistent mismatch would falsify the claimed equivalence.","tokens_in":9997,"feed_emoji":"⚡","tokens_out":8603,"duration_ms":67617,"temperature":0.7,"pith_summary":"The paper argues that several recently proposed deep-learning optimizers—Schedule-Free SGD/AdamW, Lion, MARS, and AdEMAMix—are not separate inventions but instances of one design: accelerated stochastic gradient descent, optionally with preconditioning and weight averaging. The central proof is algebraic: rewriting the Schedule-Free SGD update in terms of a momentum variable shows that its evaluation-iterate sequence follows the standard accelerated-SGD form, and its reported iterate is a weighted average of that sequence. The same lens places AdEMAMix closest to theoretically grounded accelerated SGD, which the paper says explains why it performs best in a noise-dominated small-batch regime. It then proposes Simplified-AdEMAMix, which keeps AdEMAMix-like performance with a single momentum buffer. If the connection is right, hyperparameter choices for these optimizers reduce to choosing a momentum strength, a current-gradient weight, and a weight-averaging schedule.","feed_headline":"One update rule unifies Schedule-Free, AdEMAMix, Lion, and MARS","feed_subtitle":"The paper maps these optimizers onto one accelerated-SGD framework and tests it on a 150m language model.","key_machinery":"The load-bearing object is the generalized accelerated-SGD update template $m_t=\\beta_{a,t}m_{t-1}+g_t$ and $w_{t+1}=w_t-\\eta_{a,t}m_t-\\alpha_{a,t}g_t$, which separates the momentum coefficient from the weight put on the current gradient. The paper proves that Schedule-Free SGD lands inside this template through the substitution $m_{t+1}=(x_t-z_{t+1})/\\gamma$, producing the exact coefficients $\\beta_{a,t}=1-c_t$, $\\eta_{a,t}=\\gamma\\beta c_{t+1}$, and $\\alpha_{a,t}=\\gamma(1-\\beta)$, and that $x_t$ is a weighted average of $y_t$. That identity is what lets the paper compare optimizers term by term and port acceleration schedules between them.","core_discovery":"The paper establishes an exact algebraic equivalence between Schedule-Free SGD and accelerated SGD followed by weight averaging. Defining $m_{t+1}=(x_t-z_{t+1})/\\gamma$ in the Schedule-Free update yields $m_t=(1-c_t)m_{t-1}+g_t$ and $y_{t+1}=y_t-\\gamma[\\beta c_{t+1}m_t+(1-\\beta)g_t]$, so the sequence $y_t$ on which gradients are evaluated obeys the general accelerated-SGD template $m_t=\\beta_{a,t}m_{t-1}+g_t$, $w_{t+1}=w_t-\\eta_{a,t}m_t-\\alpha_{a,t}g_t$ with $\\beta_{a,t}=1-c_t$, $\\eta_{a,t}=\\gamma\\beta c_{t+1}$, and $\\alpha_{a,t}=\\gamma(1-\\beta)$; the reported iterate $x_t$ is then an exponential average of $y_t$. The same template, read after setting $\\beta_1=0$ in AdEMAMix, makes AdEMAMix's slow momentum term line up with the momentum schedule $\\beta_{a,t}=1-k/t$ prescribed for accelerated SGD in noise, which the paper uses to explain its experimental ordering and to motivate Simplified-AdEMAMix.","pith_inferences":["If the mapping is generic, hyperparameter searches over these optimizers could be replaced by tuning three numbers—momentum strength, current-gradient weight, and averaging window—which is a testable claim for architectures beyond the 150m decoder-only transformer.","The paper's reading implies that any optimizer keeping a fixed weight on the current gradient will lose to a scheduled-weight version in low-noise (large-batch) settings, so applying accelerated-SGD schedules to existing Adam-family optimizers is a promising route to batch-size-robust training.","The AdEMAMix conclusion is derived through the $\\beta_1=0$ simplification; if that proxy fails outside the tested regime, the practical takeaway is closer to 'use Adam with momentum scheduling' than to 'use two momentum terms.'"],"forward_implications":["If the equivalence is right, Schedule-Free SGD and accelerated SGD with weight averaging are the same algorithm, so any difference between their empirical results is an implementation artifact, not a new method.","AdEMAMix's small-batch advantage is explained by its slow momentum term matching the accelerated-SGD schedule $\\beta_{a,t}=1-k/t$; the advantage should shrink as the batch grows and the noise term weakens.","Large-batch degradation of Schedule-Free AdamW is caused by the coupling of momentum and weight-averaging coefficients, not by the order of preconditioning and momentum.","Simplified-AdEMAMix, with one momentum buffer and a fixed weight $\\alpha$ on the current gradient, matches AdEMAMix in both regimes; at $\\alpha=0$ it reduces to standard Adam with momentum scheduling.","Adam with a scheduled momentum coefficient can match AdEMAMix at large batch sizes."],"supporting_citations":[{"why":"Provides the Schedule-Free SGD/AdamW update equations and edge cases ($\\beta=0$, $\\beta=1$) that the equivalence proof manipulates.","marker":"(Defazio et al., 2024)"},{"why":"Introduces AdEMAMix, including the $\\beta_1=0$ simplification and the large-batch degradation the paper builds on.","marker":"(Pagliardini et al., 2024)"},{"why":"Supplies the theoretical accelerated-SGD momentum schedule $\\beta_{a,t}=1-k/t$ used to explain AdEMAMix's schedules.","marker":"(Gupta et al., 2023)"},{"why":"Introduces Lion, whose update is reinterpreted as accelerated SGD plus a coordinate-wise sign.","marker":"(Chen et al., 2023)"},{"why":"Introduces MARS/MARS-Approx, shown to fit the accelerated-SGD template with its own preconditioning.","marker":"(Yuan et al., 2024)"},{"why":"One of the foundational accelerated-SGD methods that Appendix B recasts in the general form of Equation (1).","marker":"(Jain et al., 2018)"},{"why":"Provides the MASS update used as another instance of the general accelerated-SGD form.","marker":"(Liu & Belkin, 2020)"},{"why":"Earlier note that connections between these optimizers exist, which the paper formalizes and extends.","marker":"(Defazio, 2021)"}],"fun_headline_variants":["Momentum tricks unify Schedule-Free, AdEMAMix, and accelerated SGD","One framework links all major optimizers, simplifies AdEMAMix","Accelerated SGD explains AdEMAMix's edge in training","Schedule-Free and AdEMAMix share a single update rule"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that AdEMAMix with $\\beta_1=0$ behaves almost identically to the full AdEMAMix in the small-batch regime where the comparison is made; the paper relies on a cited result and one set of small-batch experiments for this, while its large-batch figure shows the proxy failing.","fun_headline_variants_meta":{"raw":{"variants":["Momentum tricks unify Schedule-Free, AdEMAMix, and accelerated SGD","One framework links all major optimizers, simplifies AdEMAMix","Accelerated SGD explains AdEMAMix's edge in training","Schedule-Free and AdEMAMix share a single update rule"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1416,"prompt_tokens":1006,"completion_tokens":410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":622,"tokens_out":410,"duration_ms":4751,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:10:19.352909+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the 150m model at 32k-token batch with full AdEMAMix ($\\beta_1=0.9$) and with $\\beta_1=0$ using the paper's sweeps; if the $\\beta_1=0$ run's validation loss separates from the full run by more than run-to-run noise, the central AdEMAMix claim is confined to a special case. Alternatively, record $x_t,z_t,y_t,g_t$ from a Schedule-Free SGD run and check the identity $y_{t+1}=y_t-\\gamma[\\beta c_{t+1}m_t+(1-\\beta)g_t]$ with $m_t=(x_{t-1}-z_t)/\\gamma$; a persistent mismatch would falsify the claimed equivalence.","supporting_citations":[],"review_version":1}