{"id":"a75229e3-c3b4-4966-a5de-661fbcc75505","arxiv_id":"2411.10438","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MARS combines STORM-style variance reduction with preconditioned updates and reports improved token efficiency over AdamW on GPT-2 pretraining, though the analyzed and evaluated variants differ.","lead":"This paper introduces MARS, a family of optimizers that add variance reduction to adaptive methods like AdamW, Lion, and Shampoo. In GPT-2 pretraining experiments, MARS-AdamW reaches target validation losses in fewer tokens than AdamW, and the authors provide a convergence analysis for a variant of the method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline GPT-2 experiments use MARS-approx, whose two-batch correction term adds ~0.45σ² of noise rather than reducing variance; the paper gives no theory or ablation isolating variance reduction, so the central claim that variance reduction powers the gains is unsupported.","rationale":"The reader's weakest assumption correctly identifies the theory-practice gap: the analyzed algorithm is MARS-exact, while the experiments use MARS-approx, and no convergence guarantee covers the practical variant. My analysis strengthens this concern by showing that MARS-approx is not merely unanalyzed but is unlikely to be a variance-reduction method at the reported hyperparameters: because the two gradients in the correction term use independent batches, the correction adds roughly 0.45σ² of noise (for γ=0.025, β1=0.95) instead of cancelling noise as the same-batch STORM correction would. This makes the paper's central conceptual claim—that variance reduction drives the observed gains—unsupported. The concrete test settles the issue by isolating the effect of the correction term (γ=0 vs γ=0.025) and by comparing against Adan, a closely related existing optimizer that shares the same approximate momentum structure. If γ=0 performs like γ=0.025, the empirical advantage reduces to a hyperparameter effect; if Adan performs like MARS-approx, the proposed framework adds nothing over an existing optimizer. Either outcome would require rewriting the paper's claims, but the empirical observation that the specific MARS-approx configuration beats the specific AdamW baseline could still stand, so I do not escalate the verdict beyond the reader's CONDITIONAL; UNCHANGED is appropriate. The reader's weakest_assumption and my concern overlap substantially, but I go further in arguing that the approximation actively destroys the variance-reduction property, hence 'partial' rather than full agreement.","tokens_in":41969,"tokens_out":8667,"duration_ms":82781,"concrete_test":"Run a controlled comparison on GPT-2 small or medium using the same OpenWebText setup, token budget, seed, and learning-rate schedule: (a) MARS-approx with γ=0.025 as reported; (b) MARS-approx with γ=0 (identical β1=0.95, β2=0.99, and peak LR, so the update reduces to AdamW with MARS's hyperparameters); (c) AdamW with the paper's reported baseline hyperparameters; (d) Adan tuned on the same grid. Compare final validation loss and HellaSwag. If (b) matches (a), the gradient-correction term is not the source of the gains. If (d) matches (a), the improvement is not specific to the proposed variance-reduction framework. As a direct variance check, estimate ∥ct − ∇F(x_t)∥² over a training window for (a), (b), and (c) to verify whether MARS-approx actually exhibits lower gradient-estimator variance than AdamW.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main experiments (Section 4, footnote 2) evaluate MARS-approx, not the exact MARS algorithm analyzed in Theorems B.5 and B.6. In MARS-approx, the correction term ∇f(x_t,ξ_t) − ∇f(x_{t−1},ξ_{t−1}) uses two independent samples, so conditional on x_t and x_{t−1} its variance is Var(∇f(x_t,ξ_t)) + Var(∇f(x_{t−1},ξ_{t−1})) ≈ 2σ², independent of ∥x_t−x_{t−1}∥. With the reported γ=0.025 and β1=0.95, the correction coefficient is γβ1/(1−β1) = 0.475, so the correction adds roughly 0.45σ² of noise to ct rather than cancelling it as same-batch STORM-style variance reduction would. The convergence theorems use time-varying β1,t and β2,t and the exact same-batch gradient, and say nothing about MARS-approx or the constant β1=0.95 used in practice. Thus the empirical results do not show that variance reduction causes the improvement over AdamW: the gains could come from the different (β1,β2), the 10× larger peak learning rate in Table 12, or the Adan-like momentum correction (the paper itself notes Adan is a special case of MARS-approx with γ=1−β1). Without a γ=0 ablation at matched hyperparameters and a comparison against Adan, the central claim that MARS 'unleashes the power of variance reduction' is not supported by the experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MARS, a unified framework that combines STORM-style recursive momentum, a scaling parameter gamma, clipping, and preconditioned updates, with AdamW, Lion, and Shampoo instantiations. A convergence analysis is given for the exact MARS update with time-varying beta1 and beta2, and experiments on GPT-2 small/medium/large, CIFAR-10/100, and a FineWeb-Edu subset compare MARS instantiations against AdamW, Lion, Muon, and Shampoo. The headline claim is that MARS consistently outperforms AdamW, with e.g. GPT-2 large reaching validation loss 2.58 at 28B tokens instead of 50B for AdamW, and that variance reduction is the source of the improvement. The main experiments, however, use MARS-approx, which replaces the same-batch gradient difference with a two-batch difference and is not covered by the convergence theorems.","tokens_in":42333,"tokens_out":12215,"duration_ms":118929,"significance":"If the central claim is correct, the paper would be practically significant: it would provide a rare instance of variance reduction helping large-scale language model training and would offer a flexible framework for adding variance reduction to popular optimizers. The paper has notable strengths: the framework is clean, code is released, the experimental sweep covers several model scales, and the exact-vs-approx comparison in Appendix E.2 is a useful contribution. The connections to Adan, Lion, and Muon are also valuable. However, the significance is currently undercut by a mismatch between the theorized algorithm and the evaluated algorithm, and by the absence of controls that isolate the variance-reduction component from changes in hyperparameters. The stress-test concern therefore lands: the empirical results may support MARS-approx as a strong optimizer, but they do not, as presented, establish that variance reduction causes the gains.","major_comments":[{"comment":"The convergence theorems cover only the exact MARS algorithm that uses the same-batch difference ∇f(x_t,ξ_t) − ∇f(x_{t−1},ξ_t) with time-varying β_{1,t}=1−cη_t², while the main GPT-2 experiments use MARS-approx, which replaces this with ∇f(x_t,ξ_t) − ∇f(x_{t−1},ξ_{t−1}) and constant β₁=0.95. As the paper itself notes in Remark 3.3, MARS and MARS-approx may have distinct convergence guarantees, and no theorem in the paper applies to MARS-approx or to constant β₁. Moreover, for MARS-approx the correction term is the difference of two independent stochastic gradients, so its conditional variance is approximately Var(∇f(x_t,ξ_t)) + Var(∇f(x_{t−1},ξ_{t−1})) rather than vanishing with ∥x_t−x_{t−1}∥; with the reported γ=0.025 and β₁=0.95, the correction coefficient γβ₁/(1−β₁)=0.475 adds roughly 0.45σ² of noise to c_t rather than performing STORM-style variance cancellation. Consequently, the evaluated algorithm is not a variance-reduced method in the sense analyzed, and the statement that the experiments demonstrate the power of variance reduction is not supported by the theory or the experiments as reported. The authors should either analyze MARS-approx under constant β₁, or explicitly reframe the empirical section as evaluating a heuristic approximation and provide ablations that isolate the exact variance-reduced correction.","section":"§4 (footnote 2), Remark 3.3, Theorems B.5/B.6"},{"comment":"The theorems state bounds on the time-averaged momentum error Σ E∥∇F(x_t)−m_t∥²/T and on the step-size-weighted displacement Σ (1/η_t)E∥x_{t+1}−x_t∥²/T, but they do not state a bound on the standard stationarity measure Σ E∥∇F(x_t)∥²/T. The introduction and Section 3.1 advertise a O(T^{−1/3}) convergence rate that improves on the O(T^{−1/4}) rate of AdamW, but the displayed theorems never convert the momentum-error and displacement bounds into a stationarity bound, nor do they explicitly identify an alternative stationarity measure. This is load-bearing for the theoretical contribution: without such a conversion or a clearly stated measure, the claimed rate comparison is not established. Please add the missing corollary or restate the claim to match what the theorems actually bound.","section":"§B.2, Theorems B.5 and B.6"},{"comment":"The experimental comparison does not isolate variance reduction from other algorithmic changes. Table 12 shows that MARS-AdamW uses a peak learning rate 10× larger than AdamW (6e−3 vs 6e−4 for GPT-2 small) and different momentum parameters (β₁=0.95, β₂=0.99 vs 0.9, 0.95). Section E.5 sweeps γ over {0.0001, 0.001, 0.01, 0.025, 0.05, 0.1, 0.2} but does not include γ=0, which would be exactly the MARS update with the correction disabled at the same β₁, β₂, and learning rate; such an ablation is necessary to attribute the gains to the variance-reduction term rather than to the changed hyperparameters. In addition, Section 3.2.1 shows that Adan is a special case of MARS-approx (with γ=1−β₁ in the stated sense), yet Adan is not included as a baseline in Section 4. Without a γ=0 ablation and an Adan comparison at matched settings, the reported gains over AdamW, Lion, and Muon do not establish that variance reduction is the mechanism behind the improvement.","section":"§4, Table 12, §E.5, §3.2.1"}],"minor_comments":[{"comment":"The phrase \"large models demands efficient and scalable optimizers\" is grammatically incomplete; it should read \"large-model training demands\" or similar.","section":"Abstract/Introduction"},{"comment":"The sensitivity discussion says γ=0.025 is the best value but does not report the final validation-loss numbers for each γ; adding a small table would make the insensitivity claim quantitative.","section":"Appendix E.5, Figure 14"},{"comment":"There is a typo \"constatnt\" in the Figure 16 caption, and the terminology \"MARS-exact\" vs \"MARS\" vs \"MARS-approx\" is introduced in different places; please define the exact/approx distinction once and use it consistently.","section":"Figure 16 caption and throughout"},{"comment":"The statement \"s ≥ 8L3/ρ3\" appears without superscripts in the displayed text; this should be typeset as 8L³/ρ³ for readability.","section":"Theorems B.5 and B.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is probably of interest to the ML optimization audience, and the empirical results are strong enough to merit revision rather than rejection. The main risk is overclaiming: the title and abstract attribute the gains to variance reduction, but the evaluated default algorithm is a two-batch approximation that the theory does not cover, and the experiments do not include the two controls (γ=0, Adan) that would support the attribution. I would ask for those ablations plus a clear statement of what the theorems actually bound; new large-scale experiments are not necessarily required if the claims are rebalanced and the ablation is run at the existing GPT-2 small scale."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical result at GPT-2 scale looks real: MARS-AdamW and MARS-Lion consistently beat AdamW, Lion, and Muon on validation loss and downstream tasks within the same token budget, and the gap grows with model size. Second, the paper's own explanation for that gain is not supported. The main experiments use MARS-approx, where the \"variance reduction\" correction uses two independent batches. At the reported γ=0.025 and β1=0.95, that term inflates gradient variance by roughly 0.45σ² instead of cancelling common noise. So we do not actually know that variance reduction is doing the work; the gains could come from the different (β1,β2), the 10× larger peak learning rate, or the Adan-like momentum structure.\n\nWhere the paper earns its keep: a clean unifying framework (scaled STORM plus preconditioning), a sensible redefinition of the second moment as the EMA of the corrected gradient c_t, three concrete instantiations, and precise equivalence results showing Adan, Lion, and Muon as special cases. The theory is honest for the exact variant: a stated O(log T / T^{1/3}) rate under time-varying β's, with a self-contained proof. The appendix is unusually thorough—exact vs approx, γ sensitivity, batch size, constant LR, WSD schedule, FineWeb-Edu, CIFAR—and the code is public.\n\nThe load-bearing soft spot is the theory-to-practice gap. Theorems B.5 and B.6 cover MARS-exact with time-varying β1,t = 1 − cηt² and β2,t = 1 − ηt⁶; there is no convergence guarantee for MARS-approx or for the constant β1 = 0.95 used in the main runs. Remark 3.3 acknowledges the difference but handwaves it. The stress-test math holds: with independent batches, the correction is a noise-inflating momentum term, not a variance-reduction term. The paper also tunes γ on the target benchmark and provides no γ = 0 ablation at matched hyperparameters, so the empirical gains are not cleanly attributed to the proposed mechanism. A head-to-head against Adan with the same (β1,β2) would have been the natural control, since the paper itself shows MARS-approx with γ = 1−β1 reduces to Adan. Single-seed runs are a real limitation but standard at this scale; the appendix's batch-size result (larger gap at small batches) is at least consistent with a variance story, so I would not call the empirical claim suspect.\n\nWho is this for: optimizer researchers and anyone building on recursive-momentum-plus-preconditioning. The paper deserves a serious referee—the combination of an idealized theory and strong but weakly explained empirics is exactly what review should engage with. I would send it out but ask for the γ = 0 ablation and an Adan comparison before publication.","headline":"The GPT-2 result looks real, but the paper's central variance-reduction narrative is not supported by the experiments, which run MARS-approx—a variant whose correction term adds noise rather than cancels it.","tokens_in":42867,"tokens_out":3275,"would_cite":true,"duration_ms":31180,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a unified optimizer called MARS, which adds variance reduction to preconditioned gradient methods, trains GPT-2 models to lower loss with fewer tokens than AdamW and attains a faster convergence rate.","keywords":["variance reduction","adaptive gradient methods","stochastic recursive momentum","large language model pretraining","AdamW","Lion","Shampoo","convergence rate"],"falsifier":"Re-train GPT-2 large with MARS-approx and AdamW under the paper's hyperparameters but a different random seed and data order; if 28 billion tokens no longer reaches validation loss 2.58 while AdamW still needs 50 billion, the headline empirical claim fails. A more direct check is to measure the per-step variance of the MARS gradient estimator on a small GPT-2 model: if it is not lower than AdamW's estimator variance, the proposed mechanism is not doing the work attributed to it.","tokens_in":41791,"feed_emoji":"⚡","tokens_out":8087,"duration_ms":64272,"temperature":0.7,"pith_summary":"The paper is trying to establish that variance reduction, long considered ineffective for deep learning, can be successfully combined with adaptive gradient optimizers to speed up large-model training. It proposes MARS, a framework that wraps a scaled stochastic recursive momentum—a gradient estimator with a tunable correction term—around any preconditioning scheme, giving three instances built on AdamW, Lion, and Shampoo. The paper argues that MARS converges at rate $O(T^{-1/3})$, better than the $O(T^{-1/4})$ attainable by AdamW, and reports GPT-2 experiments where MARS consistently reaches lower training and validation loss than AdamW, Lion, and Muon at the same token budget. If correct, the result would make variance reduction a practical tool for cutting the compute cost of pretraining large language models.","feed_headline":"Variance reduction cuts GPT-2 pretraining tokens by roughly half","feed_subtitle":"MARS pairs variance reduction with AdamW-style updates and beats AdamW, Lion, and Muon at equal tokens.","key_machinery":"The load-bearing object is the scaled stochastic recursive momentum gradient estimator $c_t$. It adds a scaled gradient-difference term $\\gamma_t \\frac{\\beta_1}{1-\\beta_1}(\\nabla f(x_t, \\xi_t) - \\nabla f(x_{t-1}, \\xi_t))$ to the current stochastic gradient, so the difference between two gradients at nearby parameters cancels the noise component shared by the batch $\\xi_t$. The scaling parameter $\\gamma_t$ tunes the strength of variance reduction—$\\gamma_t=0$ gives the plain stochastic gradient and $\\gamma_t=1$ gives STORM—while clipping keeps the estimate bounded; the second-order momentum is redefined as the EMA of $c_t^2$, not of the raw gradient, which the paper argues is necessary for coordinate-wise updates to stay consistent with the variance-reduced momentum. The convergence analysis uses a Lyapunov function that tracks both objective value and estimation error, with a time-varying $\\beta_1 = 1 - c\\eta_t^2$.","core_discovery":"The central claim is that variance reduction and preconditioned updates are complementary rather than competing: MARS defines a new gradient estimator $c_t = \\nabla f(x_t, \\xi_t) + \\gamma_t \\frac{\\beta_1}{1-\\beta_1}(\\nabla f(x_t, \\xi_t) - \\nabla f(x_{t-1}, \\xi_t))$, clips it, and uses its exponential moving average as the momentum while feeding $c_t^2$ into the second-order momentum. With $\\gamma_t=0$ the method reduces to AdamW, and with $\\gamma_t=1$ it recovers STORM momentum; the paper proves that under standard smoothness and bounded-variance assumptions, MARS achieves an $O(1/T^{1/3})$ convergence rate, improving on AdamW's $O(1/T^{1/4})$. Empirically, the paper claims that MARS-AdamW reaches validation loss 2.58 on GPT-2 large with 28 billion tokens while AdamW needs 50 billion, and that both MARS-AdamW and MARS-Lion outperform AdamW, Lion, and Muon on downstream tasks after 50 billion tokens.","pith_inferences":["A natural next experiment is to run MARS-AdamW against AdamW at fixed wall-clock budget on a larger model or a different corpus; the paper's GPT-2 evidence would predict the margin to persist or widen, since the reported gap grows with model size.","The paper reports the performance gap widening at smaller batch sizes, so MARS may be especially valuable in high-variance regimes such as small-batch pretraining, continual training, or low-resource data; this is an extrapolation from the batch-size sensitivity experiments.","The convergence proof covers only exact MARS with time-varying $\\beta_1$, so a proof for MARS-approx at constant $\\beta_1=0.95$ would directly confirm the assumption the practical experiments rest on.","A practical extension would be an automatic schedule for $\\gamma_t$: the paper fixes $\\gamma=0.025$ and shows sensitivity curves, but does not provide a selection rule, so adaptive or per-layer $\\gamma$ is an open design problem."],"forward_implications":["On GPT-2 large, MARS reaches validation loss 2.58 after 28B tokens versus AdamW's 50B, a token saving of roughly 44% at that loss level.","After 50B tokens, MARS-AdamW lifts 5-shot HellaSwag accuracy to 45.66% from AdamW's 42.55%, and MARS-AdamW leads the average over eight downstream tasks at 50.15%.","The framework is not tied to diagonal preconditioning: MARS-Lion and MARS-Shampoo carry variance reduction into sign-based and eigenspace-based updates, so the gain is reported across three very different preconditioner families.","The proven $O(T^{-1/3})$ convergence rate for exact MARS is faster than AdamW's $O(T^{-1/4})$, meaning the theoretical advantage, if it holds in practice, should grow with training length.","MARS-approx is recommended for practice because the paper reports only marginal gains from the exact version, keeping the extra per-step cost of computing two gradients on the same batch."],"supporting_citations":[{"why":"Supplies the stochastic recursive momentum (STORM) gradient estimator that MARS scales and wraps in a preconditioned update.","marker":"Cutkosky & Orabona, 2019"},{"why":"Defines AdamW, the central baseline and the preconditioning scheme behind MARS-AdamW.","marker":"Loshchilov & Hutter, 2019"},{"why":"Defines the Lion optimizer, which MARS-Lion instantiates and compares against.","marker":"Chen et al., 2023"},{"why":"Defines Shampoo's eigenspace preconditioning, which MARS-Shampoo builds on.","marker":"Gupta et al., 2018"},{"why":"Defines the Muon baseline used in the wall-clock and loss comparisons.","marker":"Jordan et al., 2024"},{"why":"Documents why variance reduction failed in deep learning, the gap MARS targets.","marker":"Defazio & Bottou, 2019"},{"why":"Supplies the OpenWebText dataset used for all GPT-2 pretraining experiments.","marker":"Gokaslan et al., 2019"},{"why":"Supplies the GPT-2 training codebase and configuration underlying the experiments.","marker":"Karpathy, 2022"}],"fun_headline_variants":["MARS optimizer halves GPT-2 pretraining tokens","Variance reduction meets AdamW: MARS beats AdamW","MARS: faster GPT-2 training with variance reduction","New optimizer MARS cuts training cost in half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical default algorithm, MARS-approx, replaces the same-batch gradient difference used in the theory with a difference across two successive batches, and the paper's $O(T^{-1/3})$ guarantee is proven only for the exact update with a time-varying momentum schedule, so the experiments assume the approximate variant inherits the variance-reduction benefit without a proof.","fun_headline_variants_meta":{"raw":{"variants":["MARS optimizer halves GPT-2 pretraining tokens","Variance reduction meets AdamW: MARS beats AdamW","MARS: faster GPT-2 training with variance reduction","New optimizer MARS cuts training cost in half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1302,"prompt_tokens":999,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":238}},"tokens_in":615,"tokens_out":303,"duration_ms":3269,"temperature":1.0,"reasoning_tokens":238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:37:08.894506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train GPT-2 large with MARS-approx and AdamW under the paper's hyperparameters but a different random seed and data order; if 28 billion tokens no longer reaches validation loss 2.58 while AdamW still needs 50 billion, the headline empirical claim fails. A more direct check is to measure the per-step variance of the MARS gradient estimator on a small GPT-2 model: if it is not lower than AdamW's estimator variance, the proposed mechanism is not doing the work attributed to it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 training codebase and configuration underlying the experiments."}],"review_version":1}