Pith. sign in

REVIEW 3 major objections 4 minor 7 cited by

MARS: Unleashing the Power of Variance Reduction for Training Large Models

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a unified optimizer called MARS, which adds variance reduction to preconditioned gradient methods, trains GPT-2 models to lower loss with fewer tokens than AdamW and attains a faster convergence rate.

desk verdict The GPT-2 result looks real, but the paper's central variance-reduction narrative is not supported by the experiments, which run MARS-approx—a variant whose correction term adds noise rather than cancels it. read the letter →

arxiv 2411.10438 v4 pith:CQSGBR24 submitted 2024-11-15 cs.LG math.OCstat.ML

Huizhuo Yuan , Yifeng Liu , Shuang Wu , Xun Zhou , Quanquan Gu This is my paper · ORCID
classification cs.LGmath.OCstat.ML
keywords variancereductionadaptivegradientmethodsstochasticrecursivemomentumlargelanguagemodelpretrainingAdamWLionShampooconvergencerate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that variance reduction, long considered ineffective for deep learning, can be successfully combined with adaptive gradient optimizers to speed up large-model training. It proposes MARS, a framework that wraps a scaled stochastic recursive momentum—a gradient estimator with a tunable correction term—around any preconditioning scheme, giving three instances built on AdamW, Lion, and Shampoo. The paper argues that MARS converges at rate $O(T^{-1/3})$, better than the $O(T^{-1/4})$ attainable by AdamW, and reports GPT-2 experiments where MARS consistently reaches lower training and validation loss than AdamW, Lion, and Muon at the same token budget. If correct, the result would make variance reduction a practical tool for cutting the compute cost of pretraining large language models.

What carries the argument

The load-bearing object is the scaled stochastic recursive momentum gradient estimator $c_t$. It adds a scaled gradient-difference term $\gamma_t \frac{\beta_1}{1-\beta_1}(\nabla f(x_t, \xi_t) - \nabla f(x_{t-1}, \xi_t))$ to the current stochastic gradient, so the difference between two gradients at nearby parameters cancels the noise component shared by the batch $\xi_t$. The scaling parameter $\gamma_t$ tunes the strength of variance reduction—$\gamma_t=0$ gives the plain stochastic gradient and $\gamma_t=1$ gives STORM—while clipping keeps the estimate bounded; the second-order momentum is redefined as the EMA of $c_t^2$, not of the raw gradient, which the paper argues is necessary for coordinate-wise updates to stay consistent with the variance-reduced momentum. The convergence analysis uses a Lyapunov function that tracks both objective value and estimation error, with a time-varying $\beta_1 = 1 - c\eta_t^2$.

What would settle it

Re-train GPT-2 large with MARS-approx and AdamW under the paper's hyperparameters but a different random seed and data order; if 28 billion tokens no longer reaches validation loss 2.58 while AdamW still needs 50 billion, the headline empirical claim fails. A more direct check is to measure the per-step variance of the MARS gradient estimator on a small GPT-2 model: if it is not lower than AdamW's estimator variance, the proposed mechanism is not doing the work attributed to it.

Watch

Extended reading notes

Core claim

The central claim is that variance reduction and preconditioned updates are complementary rather than competing: MARS defines a new gradient estimator $c_t = \nabla f(x_t, \xi_t) + \gamma_t \frac{\beta_1}{1-\beta_1}(\nabla f(x_t, \xi_t) - \nabla f(x_{t-1}, \xi_t))$, clips it, and uses its exponential moving average as the momentum while feeding $c_t^2$ into the second-order momentum. With $\gamma_t=0$ the method reduces to AdamW, and with $\gamma_t=1$ it recovers STORM momentum; the paper proves that under standard smoothness and bounded-variance assumptions, MARS achieves an $O(1/T^{1/3})$ convergence rate, improving on AdamW's $O(1/T^{1/4})$. Empirically, the paper claims that MARS-AdamW reaches validation loss 2.58 on GPT-2 large with 28 billion tokens while AdamW needs 50 billion, and that both MARS-AdamW and MARS-Lion outperform AdamW, Lion, and Muon on downstream tasks after 50 billion tokens.

Load-bearing premise

The practical default algorithm, MARS-approx, replaces the same-batch gradient difference used in the theory with a difference across two successive batches, and the paper's $O(T^{-1/3})$ guarantee is proven only for the exact update with a time-varying momentum schedule, so the experiments assume the approximate variant inherits the variance-reduction benefit without a proof.

Editorial extensions

If this is right

  • On GPT-2 large, MARS reaches validation loss 2.58 after 28B tokens versus AdamW's 50B, a token saving of roughly 44% at that loss level.
  • After 50B tokens, MARS-AdamW lifts 5-shot HellaSwag accuracy to 45.66% from AdamW's 42.55%, and MARS-AdamW leads the average over eight downstream tasks at 50.15%.
  • The framework is not tied to diagonal preconditioning: MARS-Lion and MARS-Shampoo carry variance reduction into sign-based and eigenspace-based updates, so the gain is reported across three very different preconditioner families.
  • The proven $O(T^{-1/3})$ convergence rate for exact MARS is faster than AdamW's $O(T^{-1/4})$, meaning the theoretical advantage, if it holds in practice, should grow with training length.
  • MARS-approx is recommended for practice because the paper reports only marginal gains from the exact version, keeping the extra per-step cost of computing two gradients on the same batch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to run MARS-AdamW against AdamW at fixed wall-clock budget on a larger model or a different corpus; the paper's GPT-2 evidence would predict the margin to persist or widen, since the reported gap grows with model size.
  • The paper reports the performance gap widening at smaller batch sizes, so MARS may be especially valuable in high-variance regimes such as small-batch pretraining, continual training, or low-resource data; this is an extrapolation from the batch-size sensitivity experiments.
  • The convergence proof covers only exact MARS with time-varying $\beta_1$, so a proof for MARS-approx at constant $\beta_1=0.95$ would directly confirm the assumption the practical experiments rest on.
  • A practical extension would be an automatic schedule for $\gamma_t$: the paper fixes $\gamma=0.025$ and shows sensitivity curves, but does not provide a selection rule, so adaptive or per-layer $\gamma$ is an open design problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MARS, a unified framework that combines STORM-style recursive momentum, a scaling parameter gamma, clipping, and preconditioned updates, with AdamW, Lion, and Shampoo instantiations. A convergence analysis is given for the exact MARS update with time-varying beta1 and beta2, and experiments on GPT-2 small/medium/large, CIFAR-10/100, and a FineWeb-Edu subset compare MARS instantiations against AdamW, Lion, Muon, and Shampoo. The headline claim is that MARS consistently outperforms AdamW, with e.g. GPT-2 large reaching validation loss 2.58 at 28B tokens instead of 50B for AdamW, and that variance reduction is the source of the improvement. The main experiments, however, use MARS-approx, which replaces the same-batch gradient difference with a two-batch difference and is not covered by the convergence theorems.

Significance. If the central claim is correct, the paper would be practically significant: it would provide a rare instance of variance reduction helping large-scale language model training and would offer a flexible framework for adding variance reduction to popular optimizers. The paper has notable strengths: the framework is clean, code is released, the experimental sweep covers several model scales, and the exact-vs-approx comparison in Appendix E.2 is a useful contribution. The connections to Adan, Lion, and Muon are also valuable. However, the significance is currently undercut by a mismatch between the theorized algorithm and the evaluated algorithm, and by the absence of controls that isolate the variance-reduction component from changes in hyperparameters. The stress-test concern therefore lands: the empirical results may support MARS-approx as a strong optimizer, but they do not, as presented, establish that variance reduction causes the gains.

major comments (3)
  1. [§4 (footnote 2), Remark 3.3, Theorems B.5/B.6] The convergence theorems cover only the exact MARS algorithm that uses the same-batch difference ∇f(x_t,ξ_t) − ∇f(x_{t−1},ξ_t) with time-varying β_{1,t}=1−cη_t², while the main GPT-2 experiments use MARS-approx, which replaces this with ∇f(x_t,ξ_t) − ∇f(x_{t−1},ξ_{t−1}) and constant β₁=0.95. As the paper itself notes in Remark 3.3, MARS and MARS-approx may have distinct convergence guarantees, and no theorem in the paper applies to MARS-approx or to constant β₁. Moreover, for MARS-approx the correction term is the difference of two independent stochastic gradients, so its conditional variance is approximately Var(∇f(x_t,ξ_t)) + Var(∇f(x_{t−1},ξ_{t−1})) rather than vanishing with ∥x_t−x_{t−1}∥; with the reported γ=0.025 and β₁=0.95, the correction coefficient γβ₁/(1−β₁)=0.475 adds roughly 0.45σ² of noise to c_t rather than performing STORM-style variance cancellation. Consequently, the evaluated algorithm is not a variance-reduced method in the sense analyzed, and the statement that the experiments demonstrate the power of variance reduction is not supported by the theory or the experiments as reported. The authors should either analyze MARS-approx under constant β₁, or explicitly reframe the empirical section as evaluating a heuristic approximation and provide ablations that isolate the exact variance-reduced correction.
  2. [§B.2, Theorems B.5 and B.6] The theorems state bounds on the time-averaged momentum error Σ E∥∇F(x_t)−m_t∥²/T and on the step-size-weighted displacement Σ (1/η_t)E∥x_{t+1}−x_t∥²/T, but they do not state a bound on the standard stationarity measure Σ E∥∇F(x_t)∥²/T. The introduction and Section 3.1 advertise a O(T^{−1/3}) convergence rate that improves on the O(T^{−1/4}) rate of AdamW, but the displayed theorems never convert the momentum-error and displacement bounds into a stationarity bound, nor do they explicitly identify an alternative stationarity measure. This is load-bearing for the theoretical contribution: without such a conversion or a clearly stated measure, the claimed rate comparison is not established. Please add the missing corollary or restate the claim to match what the theorems actually bound.
  3. [§4, Table 12, §E.5, §3.2.1] The experimental comparison does not isolate variance reduction from other algorithmic changes. Table 12 shows that MARS-AdamW uses a peak learning rate 10× larger than AdamW (6e−3 vs 6e−4 for GPT-2 small) and different momentum parameters (β₁=0.95, β₂=0.99 vs 0.9, 0.95). Section E.5 sweeps γ over {0.0001, 0.001, 0.01, 0.025, 0.05, 0.1, 0.2} but does not include γ=0, which would be exactly the MARS update with the correction disabled at the same β₁, β₂, and learning rate; such an ablation is necessary to attribute the gains to the variance-reduction term rather than to the changed hyperparameters. In addition, Section 3.2.1 shows that Adan is a special case of MARS-approx (with γ=1−β₁ in the stated sense), yet Adan is not included as a baseline in Section 4. Without a γ=0 ablation and an Adan comparison at matched settings, the reported gains over AdamW, Lion, and Muon do not establish that variance reduction is the mechanism behind the improvement.
minor comments (4)
  1. [Abstract/Introduction] The phrase "large models demands efficient and scalable optimizers" is grammatically incomplete; it should read "large-model training demands" or similar.
  2. [Appendix E.5, Figure 14] The sensitivity discussion says γ=0.025 is the best value but does not report the final validation-loss numbers for each γ; adding a small table would make the insensitivity claim quantitative.
  3. [Figure 16 caption and throughout] There is a typo "constatnt" in the Figure 16 caption, and the terminology "MARS-exact" vs "MARS" vs "MARS-approx" is introduced in different places; please define the exact/approx distinction once and use it consistently.
  4. [Theorems B.5 and B.6] The statement "s ≥ 8L3/ρ3" appears without superscripts in the displayed text; this should be typeset as 8L³/ρ³ for readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the convergence analysis and optimizer equivalence derivations are self-contained; the MARS-approx/constant-beta gap is an evidentiary limitation, not circularity.

full rationale

The paper's derivation chain is not circular. The convergence claims (Theorems B.5 and B.6, Appendix B.2 and C) are proven under explicit assumptions (bounded variance, L-smoothness, lower-bounded preconditioner) using self-contained auxiliary lemmas, and the resulting O(T^{-1/3}) rate does not assume the empirical gains, the fitted gamma=0.025, or the constant beta values used in practice. The stated limitations are honest and specific: Remark 3.3 and footnote 2 say the main GPT-2 experiments use MARS-approx, and the proof section says 'for theoretical analysis, it is necessary to consider time-varying parameters beta1,t and beta2,t,' so the practical variant is not covered by the theorem. This is an unsupported-attribution gap, not a circular reduction. The connections to Adan, Lion, and Muon are explicit algebraic reductions (Lemma 3.4, Lemma D.1 and surrounding equations) that identify existing optimizers as special cases; they do not rename a known result as a new prediction. The paper does include normal self-citations (e.g., SNVRG, Zhou et al., 2020, from the same group), but these are background references for O(epsilon^{-3}) complexity that is also supported by external works (SPIDER, STORM), so they are not load-bearing. No equation in the paper equates a claimed prediction to an input by construction, and no fitted parameter is renamed as a derived result. Therefore the central derivation is self-contained, and the correct finding is minor or no circularity; the empirical variance-reduction attribution concern belongs to correctness risk, not circularity analysis.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard smoothness and variance assumptions plus the key fitted parameter γ. No new physical or mathematical entities are postulated.

free parameters (2)
  • gamma (γ) = 0.025
    Scaling parameter for the gradient correction, chosen by grid search on GPT-2 small validation loss in Appendix E.5 and then fixed for all main experiments. The theory's optimal γ* formula (Section 3) is not used to set this value.
  • max learning rate per optimizer = 10x larger for MARS (e.g., 2e-3 vs 2e-4 for GPT-2 large)
    Tuned per optimizer via grid search (Appendix F); MARS uses roughly 10x the AdamW learning rate. Standard practice, but the headline gains partly rest on this tuning.
assumptions (4)
  • standard math L-smoothness of f(x,ξ) for every ξ (Assumption B.2)
    Used throughout the proofs, e.g., Lemma C.2 and Lemma C.3.
  • standard math Bounded gradient variance σ² (Assumption B.1)
    Standard in stochastic optimization; used in Lemma C.2 to bound the noise term.
  • domain assumption Preconditioner H_t uniformly lower bounded by ρI (Assumption B.3)
    Claimed to be satisfied by adding ε to the diagonal; needed for the Lyapunov argument in Lemmas C.3 and C.4.
  • domain assumption Bounded parameters ||x_t||≤D for Theorem B.6
    Added for the weight-decay variant; not guaranteed for unregularized AdamW, but standard in convergence proofs for regularized objectives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MARS: Unleashing the Power of Variance Reduction for Training Large Models." pith.science (2026). https://pith.science/paper/CQSGBR24

@misc{pith2026241110438,
  author       = {Pith},
  title        = {Pith review of: MARS: Unleashing the Power of Variance Reduction for Training Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQSGBR24}},
  note         = {Machine review of arXiv:2411.10438}
}
read the original abstract

Training deep neural networks--and more recently, large models demands efficient and scalable optimizers. Adaptive gradient algorithms like Adam, AdamW, and their variants have been central to this task. Despite the development of numerous variance reduction algorithms in the past decade aimed at accelerating stochastic optimization in both convex and nonconvex settings, variance reduction has not found widespread success in training deep neural networks or large language models. Consequently, it has remained a less favored approach in modern AI. In this paper, to unleash the power of variance reduction for efficient training of large models, we propose a unified optimization framework, MARS (Make vAriance Reduction Shine), which reconciles preconditioned gradient methods with variance reduction via a scaled stochastic recursive momentum technique. Within our framework, we introduce three instances of MARS that leverage preconditioned gradient updates based on AdamW, Lion, and Shampoo, respectively. We also draw a connection between our algorithms and existing optimizers. Experimental results on training GPT-2 models indicate that MARS consistently outperforms AdamW by a large margin. The implementation of MARS is available at https://github.com/AGI-Arena/MARS.

Figures

Figures reproduced from arXiv: 2411.10438 by the authors.

Figure 1
Figure 1. The training and validation loss curves, plotted against both training tokens and wall-clock time on GPT-2 large model (770M) [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. The training and validation loss curves, plotted against both training tokens and wall-clock time on GPT-2 small model (125M). E.2 MARS and MARS-approx. We then conduct experiments to compare the performance of MARS and MARS-approx (MARS-AdamW instantiation) on GPT-2 small and medium models, the training and validation loss curves are shown in Figures 4 and 5. Models trained with MARS exhibit consistently better per… view at source ↗
Figure 3
Figure 3. The training and validation loss curves, plotted against both training tokens and wall-clock time on GPT-2 medium model (355M). 0 10 20 30 40 50 Training tokens (B) 2.82 2.84 2.86 2.88 2.90 2.92 2.94 2.96 Training Loss GPT-2 small (125M), openwebtext MARS-approx MARS-exact 0 10 20 30 40 50 Training tokens (B) 2.58 2.60 2.62 2.64 2.66 2.68 2.70 2.72 Training Loss GPT-2 medium (355M), openwebtext MARS-approx MARS-exac… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Training loss curves for MARS-AdamW and MARS-AdamW-approx on GPT-2 small (125M, left) and medium (355M, right), pretrained with OpenWebText dataset and plotted against training tokens. 0 10 20 30 40 50 Training tokens (B) 2.84 2.86 2.88 2.90 2.92 2.94 2.96 2.98 3.00 Va…
Figure 5
Figure 5. Figure 5: Validation loss curves for MARS-AdamW and MARS-AdamW-approx on GPT-2 small (125M, left) and medium (355M, right), pretrained with OpenWebText dataset. E.3 Experiments on FineWeb-Edu 100B Dataset FineWeb-Edu dataset (Lozhkov et al., 2024) is a high-quality dataset based…
Figure 6
Figure 6. Figure 6: Training loss curves for AdamW, Muon and MARS-AdamW-approx on GPT-2 small (125M, left) and XL (1.5B, right), pretrained with FineWeb-edu 100B dataset and plotted against training tokens. E.4 Computer Vision Experiments We also carry out experiments on the classificatio…
Figure 7
Figure 7. Figure 7: Validation loss curves for AdamW, Muon and MARS-AdamW-approx on GPT-2 small (125M, left) and XL (1.5B, right), pretrained with FineWeb-edu 100B dataset and plotted against training tokens. the learning rate would decrease to 10% of the original rate at the 100th epoch …
Figure 8
Figure 8. Figure 8: The test loss and test accuracy for different optimizers on CIFAR-10 dataset. We also compare the performances among the baselines of AdamW, Lion and Shampoo without variance reduction, the approximate and the exact versions of MARS instantiations. The test loss and ac…
Figure 9
Figure 9. Figure 9: The test loss and test accuracy for different optimizers on CIFAR-100 dataset [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: The test loss curves for the baselines of AdamW, Lion and Shampoo without variance reduction, the approximate and the exact versions of MARS instantiations on CIFAR-10 dataset. E.6 Different Learning Rate Scheduler E.6.1 CONSTANT LR To ensure a fair comparison by elim…
Figure 11
Figure 11. Figure 11: The test accuracy curves for the baselines of AdamW, Lion and Shampoo without variance reduction, the approximate and the exact versions of MARS instantiations on CIFAR-10 dataset. 0 25 50 75 100 125 150 175 200 Epochs 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Test loss CIFAR100 Ad…
Figure 12
Figure 12. Figure 12: The test loss curves for the baselines of AdamW, Lion and Shampoo without variance reduction, the approximate and the exact versions of MARS instantiations on CIFAR-100 dataset. 0 25 50 75 100 125 150 175 200 Epochs 50 55 60 65 70 75 Test Accuracy (%) CIFAR100 AdamW M…
Figure 13
Figure 13. Figure 13: The test accuracy curves for the baselines of AdamW, Lion and Shampoo without variance reduction, the approximate and the exact versions of MARS instantiations on CIFAR-100 dataset. results also indicate that our algorithm has superior performances over AdamW optimize…
Figure 14
Figure 14. Figure 14: Training loss and validation loss curves for MARS and MARS-approx on GPT-2 small (125M) models trained with MARS￾AdamW-approx with different γs, pretrained with OpenWebText dataset and plotted against training tokens. 0 10 20 30 40 50 Training tokens (B) 2.9 3.0 3.1 3…
Figure 15
Figure 15. Figure 15: Training loss curves for MARS and MARS-approx on GPT-2 small (125M, left), medium (355M, middle) and large (770M, right) with constant learning rate, pretrained with OpenWebText dataset and plotted against training tokens. 0 10 20 30 40 50 Training tokens (B) 2.9 3.0 …
Figure 16
Figure 16. Figure 16: Validation loss curves for MARS and MARS-approx on GPT-2 small (125M, left), medium (355M, middle) and large (770M, right) with constatnt learning rate, pretrained with OpenWebText dataset and plotted against training tokens. training and validation loss curves are sh…
Figure 17
Figure 17. Figure 17: Training loss curves for AdamW (with different maximum learning rates, labeled with “LR”) and MARS-AdamW-approx on GPT-2 small (125M, left) and medium (355M, right) with WSD Scheduler for 10k, 20k, 50k and 100k steps, pretrained with OpenWebText dataset and plotted ag…
Figure 18
Figure 18. Figure 18: Validation loss curves for AdamW (with different maximum learning rates, labeled with “LR”) and MARS-AdamW-approx on GPT-2 small (125M, left) and medium (355M, right) with WSD Scheduler for 10k, 20k, 50k and 100k steps, pretrained with OpenWebText dataset and plotted …
Figure 19
Figure 19. Figure 19: The training and validation loss curves for varying global batch sizes (240/480/960), plotted against iteration steps on GPT-2 small model (125M) [PITH_FULL_IMAGE:figures/full_fig_p035_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Muse: Representation Geometry of Muon Beyond Normalized Momentum

    cs.LG 2026-07 conditional novelty 6.0 of 10

    The matrix shape given to Muon's orthonormalization step is a genuine optimizer axis: square-ish reshapes match native Muon, skinnier reshapes interpolate toward normalized SGD with momentum.

  2. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

  3. DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD

    cs.LG 2025-07 conditional novelty 6.0 of 10

    DNT is a transformer with InputNorm, PreNorm, MidNorm, and QKNorm that trains successfully with vanilla mSGDW, reaching performance close to AdamW on ImageNet and OpenWebText.

  4. Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise

    math.OC 2025-06 reject novelty 6.0 of 10

    Lion and Muon with weight decay are shown to be instances of one stochastic Frank-Wolfe algorithm, and clipped and variance-reduced variants get the first high-probability convergence rates for nonconvex Frank-Wolfe u...

  5. Continuous Cardiac Arrest Prediction in ICU using PPG Foundation Model

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A two-stage model using a 345M-parameter pre-trained PPG foundation model plus an attention LSTM predicts in-hospital cardiac arrest from one hour of finger PPG with 0.79 AUROC over the next 24 hours.

  6. Physics of Skill Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    The paper introduces Geometry, Resource, and Domino models that reproduce the sequential Domino effect in skill learning and link it to scaling laws, optimizers, and modularity.

  7. OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

    cs.AI 2026-06 reject novelty 5.0 of 10

    A multi-agent typed-DSL search discovers RS-MR, a Muon-like matrix optimizer that beats Muon by 0.57% validation BPB on one 3.15B-token transformer pretraining run, without error bars.

Reference graph

Works this paper leans on

95 extracted references · 28 canonical work pages · cited by 7 Pith papers

  1. [1]

    and Yuan, Y

    Allen-Zhu, Z. and Yuan, Y. Improved svrg for non-strongly-convex or sum-of-non-convex objectives. In International conference on machine learning, pp.\ 1080--1089. PMLR, 2016

    2016

  2. [2]

    Scalable second order optimization for deep learning

    Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    arXiv 2002

  3. [3]

    C., Foster, D

    Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1): 0 165--214, 2023

    2023

  4. [4]

    and Glynn, P

    Asmussen, S. and Glynn, P. W. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007

    2007

  5. [5]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024

    arXiv 2024

  6. [6]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Inte...

    2020

  7. [7]

    Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  8. [8]

    Stochastic spectral descent for restricted boltzmann machines

    Carlson, D., Cevher, V., and Carin, L. Stochastic spectral descent for restricted boltzmann machines. In Artificial Intelligence and Statistics, pp.\ 111--119. PMLR, 2015 a

    2015

Show all 95 references
  1. [9]

    Stochastic spectral descent for discrete graphical models

    Carlson, D., Hsieh, Y.-P., Collins, E., Carin, L., and Cevher, V. Stochastic spectral descent for discrete graphical models. IEEE Journal of Selected Topics in Signal Processing, 10 0 (2): 0 296--311, 2015 b

    2015

  2. [10]

    Closing the generalization gap of adaptive gradient methods in training deep neural networks

    Chen, J., Zhou, D., Tang, Y., Yang, Z., Cao, Y., and Gu, Q. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06763, 2018

    2018 arXiv

  3. [11]

    Symbolic discovery of optimization algorithms

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2023

    2023

  4. [12]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  5. [13]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the As...

    2019

  6. [14]

    and Orabona, F

    Cutkosky, A. and Orabona, F. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems, 32, 2019

    2019

  7. [15]

    and Bottou, L

    Defazio, A. and Bottou, L. On the ineffectiveness of variance reduced optimization for deep learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  8. [16]

    Saga: A fast incremental gradient method with support for non-strongly convex composite objectives

    Defazio, A., Bach, F., and Lacoste-Julien, S. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014

    2014

  9. [17]

    A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A

    Defazio, A., Yang, X. A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024

    2024 arXiv

  10. [18]

    Stochastic variance-reduced newton: Accelerating finite-sum minimization with large batches

    Derezinski, M. Stochastic variance-reduced newton: Accelerating finite-sum minimization with large batches. In OPT 2023: Optimization for Machine Learning

    2023

  11. [19]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  12. [20]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [21]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

    2011

  14. [22]

    J., Lin, Z., and Zhang, T

    Fang, C., Li, C. J., Lin, Z., and Zhang, T. Spider: Near-optimal non-convex optimization via stochastic path-integrated differential estimator. Advances in neural information processing systems, 31, 2018

    2018

  15. [23]

    Promise: Preconditioned stochastic optimization methods by incorporating scalable curvature estimates

    Frangella, Z., Rathore, P., Zhao, S., and Udell, M. Promise: Preconditioned stochastic optimization methods by incorporating scalable curvature estimates. Journal of Machine Learning Research, 25 0 (346): 0 1--57, 2024

    2024

  16. [24]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  17. [25]

    and Lan, G

    Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23 0 (4): 0 2341--2368, 2013

    2013

  18. [26]

    Openwebtext corpus

    Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  19. [27]

    and Graves, A

    Graves, A. and Graves, A. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pp.\ 37--45, 2012

    2012

  20. [28]

    Shampoo: Preconditioned stochastic tensor optimization

    Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp.\ 1842--1850. PMLR, 2018

    2018

  21. [29]

    Beyond convexity: Stochastic quasi-convex optimization

    Hazan, E., Levy, K., and Shalev-Shwartz, S. Beyond convexity: Stochastic quasi-convex optimization. Advances in neural information processing systems, 28, 2015

    2015

  22. [30]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  23. [31]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021

    2021

  24. [32]

    Higham, N. J. Functions of Matrices. Society for Industrial and Applied Mathematics, 2008

    2008

  25. [33]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  26. [34]

    Super-adam: faster and universal framework of adaptive gradients

    Huang, F., Li, J., and Huang, H. Super-adam: faster and universal framework of adaptive gradients. Advances in Neural Information Processing Systems, 34: 0 9074--9085, 2021

    2021

  27. [35]

    and Zhang, T

    Johnson, R. and Zhang, T. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013

    2013

  28. [36]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Jordan, K., Jin, Y., Boza, V., Jiacheng, Y., Cecista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  29. [37]

    Kaddour, J., Key, O., Nawrot, P., Minervini, P., and Kusner, M. J. No train no gain: Revisiting efficient training algorithms for transformer-based language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    Karpathy, A. NanoGPT . https://github.com/karpathy/nanoGPT, 2022

    2022

  31. [39]

    T., and Cevher, V

    Kavis, A., Skoulakis, S., Antonakopoulos, K., Dadi, L. T., and Cevher, V. Adaptive stochastic variance reduction for non-convex finite-sum minimization. Advances in Neural Information Processing Systems, 35: 0 23524--23538, 2022

    2022

  32. [40]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015

    2015

  33. [41]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  34. [42]

    On the computation of the matrix k-th root

    Laki \'c , S. On the computation of the matrix k-th root. ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift f \"u r Angewandte Mathematik und Mechanik: Applied Mathematics and Mechanics , 78 0 (3): 0 167--172, 1998

    1998

  35. [43]

    S., Moeller, T

    Lavenberg, S. S., Moeller, T. L., and Welch, P. D. The application of control variables to the simulation of closed queueing networks. In Proceedings of the 9th conference on Winter simulation-Volume 1, pp.\ 152--154, 1977

    1977

  36. [44]

    Gradient-based learning applied to document recognition

    LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  37. [45]

    Storm+: Fully adaptive sgd with recursive momentum for nonconvex optimization

    Levy, K., Kavis, A., and Cevher, V. Storm+: Fully adaptive sgd with recursive momentum for nonconvex optimization. Advances in Neural Information Processing Systems, 34: 0 20571--20582, 2021

    2021

  38. [46]

    Smoothness and Adaptivity in Nonlinear Optimization for Machine Learning Applications

    Li, H. Smoothness and Adaptivity in Nonlinear Optimization for Machine Learning Applications. PhD thesis, Massachusetts Institute of Technology, 2024

    2024

  39. [47]

    Convergence of adam under relaxed assumptions

    Li, H., Rakhlin, A., and Jadbabaie, A. Convergence of adam under relaxed assumptions. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  41. [49]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342, 2023

    2023 arXiv

  42. [50]

    Adam ^+ : A stochastic method with adaptive variance reduction

    Liu, M., Zhang, W., Orabona, F., and Yang, T. Adam ^+ : A stochastic method with adaptive variance reduction. arXiv preprint arXiv:2011.11985, 2020

    2011 arXiv

  43. [51]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019

    2019

  44. [52]

    Fineweb-edu: the finest collection of educational content, 2024

    Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    2024

  45. [53]

    and Grosse, R

    Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

    2015

  46. [54]

    and Tropp, J

    Martinsson, P.-G. and Tropp, J. A. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29: 0 403--572, 2020

    2020

  47. [55]

    McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  48. [56]

    McMahan, H. B. and Streeter, M. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908, 2010

    2010 arXiv

  49. [57]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural L...

    2018 doi

  50. [58]

    A new perspective on shampoo's preconditioner

    Morwani, D., Shapira, I., Vyas, N., Malach, E., Kakade, S., and Janson, L. A new perspective on shampoo's preconditioner. arXiv preprint arXiv:2406.17748, 2024

    2024 arXiv

  51. [59]

    A method for solving the convex programming problem with convergence rate o(1/k^2)

    Nesterov, Y. A method for solving the convex programming problem with convergence rate o(1/k^2) . Proceedings of the USSR Academy of Sciences, 269: 0 543--547, 1983. URL https://api.semanticscholar.org/CorpusID:145918791

    1983

  52. [60]

    Introductory lectures on convex optimization: A basic course, volume 87

    Nesterov, Y. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013

    2013

  53. [61]

    M., Liu, J., Scheinberg, K., and Tak \'a c , M

    Nguyen, L. M., Liu, J., Scheinberg, K., and Tak \'a c , M. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International conference on machine learning, pp.\ 2613--2621. PMLR, 2017 a

    2017

  54. [62]

    M., Liu, J., Scheinberg, K., and Tak \'a c , M

    Nguyen, L. M., Liu, J., Scheinberg, K., and Tak \'a c , M. Stochastic recursive gradient algorithm for nonconvex optimization. arXiv preprint arXiv:1705.07261, 2017 b

    2017 arXiv

  55. [63]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  56. [64]

    J., Hefny, A., Sra, S., Poczos, B., and Smola, A

    Reddi, S. J., Hefny, A., Sra, S., Poczos, B., and Smola, A. Stochastic variance reduction for nonconvex optimization. In International conference on machine learning, pp.\ 314--323. PMLR, 2016

    2016

  57. [66]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019 b

    1904 arXiv

  58. [67]

    and Braun, H

    Riedmiller, M. and Braun, H. A direct adaptive method for faster backpropagation learning: The rprop algorithm. In IEEE international conference on neural networks, pp.\ 586--591. IEEE, 1993

    1993

  59. [68]

    A stochastic gradient method with an exponential convergence \_rate for finite training sets

    Roux, N., Schmidt, M., and Bach, F. A stochastic gradient method with an exponential convergence \_rate for finite training sets. Advances in neural information processing systems, 25, 2012

    2012

  60. [69]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,...

    2020

  61. [70]

    English conversational telephone speech recognition by humans and machines

    Saon, G., Kurata, G., Sercu, T., Audhkhasi, K., Thomas, S., Dimitriadis, D., Cui, X., Ramabhadran, B., Picheny, M., Lim, L.-L., et al. English conversational telephone speech recognition by humans and machines. arXiv preprint arXiv:1703.02136, 2017

    2017 arXiv

  62. [71]

    and Smola, A

    Sch \"o lkopf, B. and Smola, A. J. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002

    2002

  63. [72]

    Iterative berechnung der reziproken matrix

    Schulz, G. Iterative berechnung der reziproken matrix. Z. Angew. Math. Mech., 13: 0 57--59, 1933

    1933

  64. [73]

    and Zhang, T

    Shalev-Shwartz, S. and Zhang, T. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14 0 (1), 2013

    2013

  65. [74]

    and Stern, M

    Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp.\ 4596--4604. PMLR, 2018

    2018

  66. [75]

    M., Lee, T.-H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., and Rabbat, M

    Shi, H.-J. M., Lee, T.-H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., and Rabbat, M. A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023

    2023 arXiv

  67. [76]

    Dropout: a simple way to prevent neural networks from overfitting

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 0 (1): 0 1929--1958, 2014

    1929

  68. [77]

    Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude

    Tieleman, T. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4 0 (2): 0 26, 2012

    2012

  69. [78]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  70. [79]

    Soap: Improving and stabilizing shampoo using adam

    Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  71. [80]

    Spiderboost and momentum: Faster variance reduction algorithms

    Wang, Z., Ji, K., Zhou, Y., Liang, Y., and Tarokh, V. Spiderboost and momentum: Faster variance reduction algorithms. Advances in Neural Information Processing Systems, 32, 2019

    2019

  72. [81]

    Adagrad stepsizes: Sharp convergence over nonconvex landscapes

    Ward, R., Wu, X., and Bottou, L. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. Journal of Machine Learning Research, 21 0 (219): 0 1--30, 2020

    2020

  73. [82]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models

    Xie, X., Zhou, P., Li, H., Lin, Z., and Yan, S. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  74. [83]

    Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering

    Yadav, V., Bethard, S., and Surdeanu, M. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Languag...

    2019

  75. [84]

    A coefficient makes svrg effective

    Yin, Y., Xu, Z., Li, Z., Darrell, T., and Liu, Z. A coefficient makes svrg effective. arXiv preprint arXiv:2311.05589, 2023

    2023 arXiv

  76. [85]

    Large batch optimization for deep learning: Training bert in 76 minutes

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019

    1904 arXiv

  77. [86]

    Zeiler, M. D. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012

    2012 arXiv

  78. [87]

    Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and M \` a rquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Flore...

    2019 doi

  79. [88]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022 a

    2022 arXiv

  80. [89]

    Adam can converge without any modification on update rules

    Zhang, Y., Chen, C., Shi, N., Sun, R., and Luo, Z.-Q. Adam can converge without any modification on update rules. Advances in neural information processing systems, 35: 0 28386--28399, 2022 b

    2022

  81. [90]

    Deconstructing what makes a good optimizer for language models

    Zhao, R., Morwani, D., Brandfonbrener, D., Vyas, N., and Kakade, S. Deconstructing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972, 2024

    2024 arXiv

  82. [91]

    Stochastic nested variance reduction for nonconvex optimization

    Zhou, D., Xu, P., and Gu, Q. Stochastic nested variance reduction for nonconvex optimization. Journal of machine learning research, 21 0 (103): 0 1--63, 2020

    2020

  83. [92]

    Towards understanding convergence and generalization of adamw

    Zhou, P., Xie, X., Lin, Z., and Yan, S. Towards understanding convergence and generalization of adamw. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  84. [93]

    C., Dvornek, N., Papademetris, X., and Duncan, J

    Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C., Dvornek, N., Papademetris, X., and Duncan, J. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33: 0 18795--18806, 2020

    2020

  85. [94]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  86. [95]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  87. [96]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.