Pith. sign in

REVIEW 3 major objections 5 minor 142 references

OmniOpt maps more than a hundred modern optimizers onto a shared update pipeline and geometry, then shows that no single method dominates the quality–runtime–memory frontier.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 22:09 UTC pith:D75ZLFEO

load-bearing objection A usable survey-plus-benchmark package that couples a five-stage pipeline, LMO axes, dual taxonomy, and multi-objective LLM/vision results—worth engaging, with protocol caveats on primary labels and Stage-1 isolation. the 3 major comments →

arxiv 2607.04033 v1 pith:D75ZLFEO submitted 2026-07-04 cs.LG cs.AI

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Siyuan Li , Jiabao Pan , Yumou Liu , Zhuoli Ouyang , Xin Jin , Xinglong Xu , Jingxuan Wei , Shengye Pang
show 4 more authors
Jintao Che Xuanhe Zhou Conghui He Cheng Tan
This is my paper
classification cs.LG cs.AI
keywords optimizersLLM pretrainingmeta-pipelinelinear minimization oracleoptimizer taxonomymulti-objective benchmarkAdamWMuon
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Optimizer choice for large-model training is no longer a formula preference; it is a joint decision about compute, memory, tuning budget, and task. OmniOpt argues that the fragmented landscape of over a hundred methods can be made operational by treating every update as a five-stage meta-pipeline—where most methods only change one or two stages—and by reading update directions as norm-constrained linear minimization oracles. Those two views ground a dual taxonomy: one axis groups methods by primary mechanism (adaptive moments, matrix structure, sign-like directions, state compression, geometric wrappers), and the other records which measurable training objectives each method targets. A unified cross-domain benchmark then compares representative methods across scales, architectures, contexts, and image classification, and reports systematic family trade-offs rather than a single winner. The paper’s practical claim is that this coordinate system lets practitioners select optimizers under explicit mechanism and objective assumptions instead of chasing unstable global rankings.

Core claim

Most modern optimizers are sparse modifications of one shared update process, and their directions can be unified as norm-constrained linear minimization oracles along four axes; once methods are grouped that way and scored on multiple effect objectives, no single optimizer dominates, and family rankings cross with scale, context length, and domain.

What carries the argument

The universal five-stage meta-pipeline (signal acquisition, scoping/routing, gradient transform, state evolution, reconstruction, finalization) plus an LMO-driven four-axis decomposition (update domain, state estimator, geometry/precondition, finalization wrapper) that jointly define the dual taxonomy and the benchmark axes.

Load-bearing premise

That each optimizer has one stable primary mechanism family and that the two-stage benchmark—first screening without weight decay or clipping, then transferring only stronger short-context methods—fairly isolates those mechanisms without warping real rankings.

What would settle it

Run the full optimizer set with weight decay and clipping always on, at long context and matched wall-clock budgets, and check whether family-level quality–cost–memory orderings reverse relative to the paper’s Stage-1/Stage-2 conclusions, or whether reassigning primary mechanism labels collapses the claimed family trade-offs.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. OmniOpt proposes a unified survey-and-benchmark framework for modern deep-learning optimizers, especially LLM training. It introduces a five-stage universal meta-pipeline (S0–S5) with an identity-mapping principle, an LMO-based four-axis decomposition of updates (domain, state estimator, geometry/precondition, finalization), and a dual taxonomy over 108 methods: mechanism families T1–T5 and effect objectives O1–O6. The core empirical contribution is a controlled multi-scale, multi-architecture pretraining benchmark (C4 short-context screening; FineWeb-Edu 32k transfer; vision) of 24 representative optimizers, plus a Muon mechanistic ablation, arguing that no single optimizer dominates the quality–runtime–memory frontier and that family rankings cross with scale, context, and domain.

Significance. If the framework holds as an operational coordinate system, it would be a high-value contribution: the field has many fragmented optimizer papers and protocol-sensitive benchmarks, but few mechanism-aligned maps that jointly organize theory, taxonomy, and multi-objective evaluation. Strengths include explicit alignment of pipeline stages with LMO axes, broad coverage of T1–T5, controlled-variable benchmarking across scales and architectures, multi-objective reporting (PPL, runtime, memory, stability, LR robustness, transfer), and a useful Muon ablation. The work is more synthesis-plus-benchmark than a new optimizer theorem, but that is appropriate for a survey/cookbook aimed at selection under explicit constraints.

major comments (3)
  1. §6.1 Stage-1 protocol disables weight decay and gradient clipping for all 24 optimizers, then Stage 2 transfers only stronger Stage-1 methods under a production-style recipe. This is load-bearing for family-level claims in §6.2.8–6.2.9 and the abstract’s “no single optimizer dominates / ranking crossings” message. Because many T2/T3/T4 methods interact strongly with S5 finalization (LR–WD–warmup coupling for Lion/Muon; memory-feasibility claims for T4), Stage-1 PPL orderings (e.g., APOLLO/Muon/MARS-Shampoo at 1B) may not isolate pure S2/S3 mechanisms. Please either (i) re-run a Stage-1 subset with matched WD/clip, or (ii) substantially qualify family rankings as protocol-conditional and report which Stage-1 losers would re-enter under regularized screening.
  2. §4.1–4.2 primary-mechanism rule (unique T1–T5 label by “incremental contribution” / dominant non-identity stage) is used to justify family-level O1–O6 summaries. Composite methods (Q-GaLore, MARS-*, Cautious wrappers, COSMOS, etc.) sit on multiple stages by the paper’s own composition notes (§3.1.2). The manuscript needs a clearer, falsifiable assignment protocol—e.g., a short appendix table of contested labels with secondary tags and a sensitivity check showing that reassigning a few boundary methods does not flip the family conclusions in §6.2.8.
  3. §6.2.1–6.2.2 quality claims rest heavily on final PPL under fixed step budgets and per-optimizer LR/knob tuning, while O2/O3 are isolated optimizer runtime/memory. For matrix methods with large per-step overhead (SOAP, Shampoo, Muon), token-normalized PPL alone can overstate practical advantage. The paper already discusses wall-clock trade-offs, but the main family summary should report at least one matched wall-clock or FLOP-normalized comparison at 350M/1B so that “competitive quality” is not conflated with “better under fixed steps.”
minor comments (5)
  1. Figure 1 / abstract claim “over one hundred methods” vs. explicit “108” in §4.2: keep a single count and state inclusion criteria (preprints, variants, wrappers).
  2. Table 5 is dense; a short legend for tags (+res, VR, matrix routing, factored/INT8) earlier in §3.2.3 would help non-specialists.
  3. §3.1 vs §3.2 occasionally switch between S1–S5 and P1–P4 labeling when relating pipeline stages to axes; unify terminology.
  4. Stage-2 Commonsense results are deferred to Appendix B; a compact main-text CS Avg. column or rank-stability summary would better support the O6 transfer claim.
  5. Typos/style: “wild range” in Fig. 1 caption; occasional missing spaces before citations; ensure arXiv IDs/venues for very recent methods are consistent.

Circularity Check

0 steps flagged

No significant circularity: taxonomy is definitional organization; benchmark metrics are independent external measurements.

full rationale

OmniOpt is a survey-plus-benchmark paper. Its four coupled components (five-stage meta-pipeline, LMO four-axis view, dual taxonomy of 108 methods, multi-objective benchmark) organize existing optimizers and measure them on external quantities (PPL, step time, optimizer-state memory, GNormCV, LR perturbation, cross-scenario transfer). The identity-mapping principle and single primary T1–T5 labels are classification design choices, not a derivation that forces empirical rankings: Table 7 is explicitly labeled a “mechanism-informed prior, not a final empirical conclusion,” and family effect tables are “design priors for benchmark planning, not empirical conclusions.” LMO unification is attributed to external geometric work (Bernstein et al., Pethick et al., Sfyraki et al.), not a self-cited uniqueness theorem. Stage-1/2 protocol choices may affect ranking stability (a methodology concern), but they do not make reported trade-offs true by construction. No fitted parameter is renamed as a prediction; no central claim reduces to its inputs by definition. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The paper’s load-bearing contribution is organizational and empirical, not a closed-form theorem. It rests on standard optimization background plus three paper-specific structuring choices (pipeline stages, primary-mechanism uniqueness, effect objectives) and many benchmark protocol knobs. Invented entities are taxonomical coordinates, not physical objects; free parameters are mostly experimental hyperparameters that affect ranking stability.

free parameters (4)
  • Per-optimizer learning rates and method knobs (betas, eps, APOLLO rank/interval, etc.)
    Stage-1/2 allow optimizer-specific tuning of these while freezing architecture/data/schedule; rankings depend on this search budget.
  • Stage-1 disable of weight decay and gradient clipping
    Design choice to isolate S2/S3 mechanisms; changes absolute and relative performance versus production recipes.
  • Training budgets (steps/tokens) and model scales (60M–1B, 32k context)
    Protocol choices that the paper itself notes can reverse rankings; conclusions weight 350M/1B most heavily.
  • Primary-mechanism assignment rule for composite optimizers
    Hand-assigned unique T1–T5 labels for 108 methods; secondary tags exist but family-level analysis uses the primary label.
axioms (5)
  • ad hoc to paper Most optimizers are sparse modifications of a shared five-stage update pipeline (identity-mapping principle).
    Stated as central premise in §1.2 and §3.1; used to justify non-overlapping family sites.
  • domain assumption Norm-constrained LMOs (and dual preconditioner readings) unify practical optimizer directions including Adam boxes, sign maps, and spectral polar maps.
    Builds on cited theory (Bernstein, Pethick, Sfyraki) and is extended to four axes in §3.2.
  • ad hoc to paper Each optimizer has a unique primary incremental mechanism for Dimension-A labeling.
    Taxonomy design principle §4.1; required for non-overlapping T1–T5 families.
  • domain assumption Classical stochastic optimization setup (ERM, mini-batch gradients, AdamW baseline memory model) for LLM training.
    §2 preliminaries; standard in the field.
  • ad hoc to paper Effect objectives O1–O6 are the right multi-objective evaluation axes for optimizer selection.
    Dimension B §4.3; drives benchmark design and family assessments.
invented entities (3)
  • Universal Meta-Pipeline (S0–S5) no independent evidence
    purpose: Locate where each optimizer intervenes in a single update and support identity-mapping classification.
    Paper-defined operational abstraction; not independently measured outside this organizational use.
  • Four-axis LMO decomposition (domain, state estimator, geometry/precondition, finalization) no independent evidence
    purpose: Give each optimizer a coordinate tuple unifying direction geometry and practical state choices.
    Extension of prior LMO ideas into a survey coordinate system specific to this paper.
  • Dual-dimension taxonomy T1–T5 / O1–O6 over 108 optimizers no independent evidence
    purpose: Non-overlapping mechanism families plus multi-label effect targets for survey and benchmark grouping.
    Core invented organizational structure; empirical usefulness is tested but the labels are definitional.

pith-pipeline@v1.1.0-grok45 · 55327 in / 3582 out tokens · 41441 ms · 2026-07-11T22:09:44.185618+00:00 · methodology

0 comments
read the original abstract

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.

Figures

Figures reproduced from arXiv: 2607.04033 by Cheng Tan, Conghui He, Jiabao Pan, Jingxuan Wei, Jintao Che, Shengye Pang, Siyuan Li, Xinglong Xu, Xin Jin, Xuanhe Zhou, Yumou Liu, Zhuoli Ouyang.

Figure 1
Figure 1. Figure 1: Overview of the proposed survey and benchmark framework for a wild range of optimizers. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of optimizer design for deep learning and LLM training. The expanding coverage [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Universal meta-pipeline for one optimizer step. The training system provides a gradient or [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Mechanism overview of T1 element-wise adaptive-moment and scalar control methods. [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Taxonomy of element-wise adaptive-moment and scalar-control optimizers. [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mechanism schematic for T2 matrix-level structural methods. The schematic summarizes the [PITH_FULL_IMAGE:figures/full_fig_p034_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Taxonomy of matrix-level structural optimizers. [PITH_FULL_IMAGE:figures/full_fig_p035_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Taxonomy of discretized and directionally quantized optimizers. [PITH_FULL_IMAGE:figures/full_fig_p039_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Mechanism schematic for T3 discretiza￾tion and directional quantization. This schematic provides a taxonomy guide rather than an empir￾ical ranking for the compact T3 family, grouping discretization and directional quantization mecha￾nisms by their sign-direction generation methods. T3 contains optimizers whose primary opera￾tion discretizes the update direction. The char￾acteristic map is sign(·) or a clo… view at source ↗
Figure 10
Figure 10. Figure 10: Mechanism schematic for T4 state compression and structural aggregation. The schematic organizes the family by the form of memory reduction: factored second-moment storage, low-bit state representation, shared adaptive statistics, and streaming gradient consumption. T4.4 Fused backprop-update. This subclass contains LOMO and AdaLOMO. Its members change the optimizer pipeline so that gradients are consumed… view at source ↗
Figure 11
Figure 11. Figure 11: Taxonomy of state-compressed and structurally aggregated optimizers. [PITH_FULL_IMAGE:figures/full_fig_p045_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Mechanism schematic for T5 curvature-aware and geometric regularization methods. The [PITH_FULL_IMAGE:figures/full_fig_p049_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Taxonomy of curvature-aware and geometry-regularized optimizers. The prefix C- denotes [PITH_FULL_IMAGE:figures/full_fig_p050_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Stage-1 Pareto frontiers (1B). PPL vs. per-step runtime (left) and vs. optimizer-state memory (right); lower-left is better. Colors denote families, stars mark frontier members. families therefore occupy different favorable regions in quality, runtime, and memory. Optimizer choice is consequently objective-dependent. 6.2.2 Stage 1 Pareto Analysis over PPL, Runtime, and Memory The Stage 1 results are inher… view at source ↗
Figure 15
Figure 15. Figure 15: Optimizer-level heatmap of the three Stage-1 metrics (1B). Rows are grouped by family, and columns are C4 PPL, runtime, and optimizer-state memory. Green is favorable, red unfavorable. Optimizer-level heatmap. To complement the Pareto plots, [PITH_FULL_IMAGE:figures/full_fig_p059_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Cross-scenario rank stability (FineWeb-Edu, 32k). Absolute ranks among all twelve optimizers per scenario. (a) AdamW vs. its T1 family; (b) AdamW vs. other families’ mean rank; (c) per-architecture mean rank per optimizer (color = family, hatch = architecture). aggregates optimizers by family to reveal the broader ranking bands across architectures. Panel (c) returns to the optimizer level and summarizes … view at source ↗
Figure 17
Figure 17. Figure 17: Auxiliary O4 stability analysis from gradient-norm dynamics across architectures. Each cell shows the coefficient of variation of the gradient norm (GNormCV) for one optimizer in one architecture-scale scenario of the FineWeb-Edu long-context benchmark. Lower GNormCV indicates smoother relative gradient-norm dynamics and thus better auxiliary stability. The bar chart on the right reports the mean stabilit… view at source ↗
Figure 18
Figure 18. Figure 18: Auxiliary learning-rate perturbation robustness. Each panel shows WikiText PPL under 0.2×, 1×, and 5× the tuned learning rate. The star marks the tuned learning rate. The sensitivity score sLR is shown as s in each panel title and measures the worst relative PPL degradation from the tuned point, where lower is more robust. Green, orange, and red titles indicate robust, moderate, and sensitive learning-rat… view at source ↗
Figure 19
Figure 19. Figure 19: Family-level objective summary. Per-family profile over O1–O6. O1–O3 are directly measured from Stage 1; O4 is measured from gradient-norm stability; O5 is probed by an auxiliary learning-rate perturbation test; O6 is measured through Stage 2 rank stability and sequence-length sensitivity. Green indicates a more favorable profile and red indicates a less favorable profile. stability profile while remainin… view at source ↗
Figure 20
Figure 20. Figure 20: Mechanistic ablation of Muon (C4, 350M). Decomposition into core operations, gain operations, and operator-order constraints, with bars showing absolute PPL (lower is better). The 70.74 bar is truncated off-scale. Tiered-summary takeaway. Optimizer selection should be matched to the dominant constraint of the training regime. T1 is the safest reference family. T2 provides the strongest quality or stabilit… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

142 extracted references · 56 linked inside Pith

  1. [1]

    Exadam: The power of adaptive cross-moments.arXiv preprint arXiv:2412.20302, 2024

    Ahmed M Adly. Exadam: The power of adaptive cross-moments.arXiv preprint arXiv:2412.20302, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Dion: Distributed orthonormalized updates.arXiv preprint arXiv:2504.05295, 2025

    Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates.arXiv preprint arXiv:2504.05295, 2025

    arXiv 2025

  3. [3]

    Development of deep learning optimizers: Approaches, concepts, and update rules.arXiv preprint arXiv:2509.18396, 2025

    Do˘gay Altınel. Development of deep learning optimizers: Approaches, concepts, and update rules.arXiv preprint arXiv:2509.18396, 2025

    arXiv 2025

  4. [4]

    Natural gradient works efficiently in learning.Neural computation, 10(2):251–276, 1998

    Shun-Ichi Amari. Natural gradient works efficiently in learning.Neural computation, 10(2):251–276, 1998

    1998

  5. [5]

    A pid controller approach for stochastic optimization of deep networks

    Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang. A pid controller approach for stochastic optimization of deep networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 8522–8531, 2018

    2018

  6. [6]

    Memory efficient adaptive optimization

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  7. [7]

    Affine-scaled attention: Towards flexible and stable transformer attention.arXiv preprint arXiv:2602.23057, 2026

    Jeongin Bae, Baeseong Park, Gunho Park, Minsub Kim, Joonhyung Lee, Junhee Yoo, Sunghyeon Woo, Jiwon Ryu, Se Jung Kwon, and Dongsoo Lee. Affine-scaled attention: Towards flexible and stable transformer attention.arXiv preprint arXiv:2602.23057, 2026

    arXiv 2026

  8. [8]

    Gravity optimizer: a kinematic approach on optimization in deep learning.arXiv preprint arXiv:2101.09192, 2021

    Dariush Bahrami and Sadegh Pouriyan Zadeh. Gravity optimizer: a kinematic approach on optimization in deep learning.arXiv preprint arXiv:2101.09192, 2021

    Pith/arXiv arXiv 2021

  9. [9]

    Dissecting adam: The sign, magnitude and variance of stochastic gradients

    Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. InInternational Conference on Machine Learning, pages 404–413. PMLR, 2018

    2018

  10. [10]

    Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325, 2024

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. InInternational conference on machine learning, pages 560–569. PMLR, 2018

    2018

  12. [12]

    Training neural networks for and by interpola- tion

    Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Training neural networks for and by interpola- tion. InInternational conference on machine learning, pages 799–809. PMLR, 2020

    2020

  13. [13]

    PIQA: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439, 2020. doi: 10.1609/aaai.v34i05.6239

    doi:10.1609/aaai.v34i05.6239 2020

  14. [14]

    High-performance large-scale image recognition without normalization

    Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. InInternational conference on machine learning, pages 1059–1071. PMLR, 2021

    2021

  15. [15]

    Grams: Gradient descent with adaptive momentum scaling.arXiv preprint arXiv:2412.17107, 2024

    Yang Cao, Xiaoyu Li, and Zhao Song. Grams: Gradient descent with adaptive momentum scaling.arXiv preprint arXiv:2412.17107, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Mgup: A momentum-gradient alignment update policy for stochastic optimization.Advances in Neural Information Processing Systems, 38:20488–20537, 2026

    Da Chang and Ganzhao Yuan. Mgup: A momentum-gradient alignment update policy for stochastic optimization.Advances in Neural Information Processing Systems, 38:20488–20537, 2026

    2026

  17. [17]

    Closing the generaliza- tion gap of adaptive gradient methods in training deep neural networks.arXiv preprint arXiv:1806.06763, 2018

    Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generaliza- tion gap of adaptive gradient methods in training deep neural networks.arXiv preprint arXiv:1806.06763, 2018

    Pith/arXiv arXiv 2018

  18. [18]

    Fira: Can we achieve full-rank training of llms under low-rank constraint?Advances in Neural Information Processing Systems, 38:120680–120712, 2026

    Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of llms under low-rank constraint?Advances in Neural Information Processing Systems, 38:120680–120712, 2026

    2026

  19. [19]

    Symbolic discovery of optimization algorithms.Advances in neural information processing systems, 36:49205–49233, 2023

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms.Advances in neural information processing systems, 36:49205–49233, 2023. 78 OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    2023

  20. [20]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2924–2936. Association for C...

    doi:10.18653/v1/n19-1300 2019

  21. [21]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  22. [22]

    Why gradients rapidly increase near the end of training.arXiv preprint arXiv:2506.02285, 2025

    Aaron Defazio. Why gradients rapidly increase near the end of training.arXiv preprint arXiv:2506.02285, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    A momentumized, adaptive, dual averaged gradient method.Journal of Machine Learning Research, 23(144):1–34, 2022

    Aaron Defazio and Samy Jelassi. A momentumized, adaptive, dual averaged gradient method.Journal of Machine Learning Research, 23(144):1–34, 2022

    2022

  24. [24]

    Learning-rate-free learning by d-adaptation

    Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by d-adaptation. InInternational conference on machine learning, pages 7449–7479. PMLR, 2023

    2023

  25. [25]

    The road less scheduled.Advances in Neural Information Processing Systems, 37:9974–10007, 2024

    Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled.Advances in Neural Information Processing Systems, 37:9974–10007, 2024

    2024

  26. [26]

    Rmnp: Row-momentum normalized preconditioning for scalable matrix-based optimization.arXiv preprint arXiv:2603.20527, 2026

    Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. Rmnp: Row-momentum normalized preconditioning for scalable matrix-based optimization.arXiv preprint arXiv:2603.20527, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    8-bit optimizers via block-wise quantization

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021

    Pith/arXiv arXiv 2021

  28. [28]

    An adaptive and momental bound method for stochastic learning.arXiv preprint arXiv:1910.12249, 2019

    Jianbang Ding, Xuancheng Ren, Ruixuan Luo, and Xu Sun. An adaptive and momental bound method for stochastic learning.arXiv preprint arXiv:1910.12249, 2019

    Pith/arXiv arXiv 1910

  29. [29]

    Incorporating nesterov momentum into adam, 2016

    Timothy Dozat. Incorporating nesterov momentum into adam, 2016

    2016

  30. [30]

    diffgrad: an optimization method for convolutional neural networks.IEEE transactions on neural networks and learning systems, 31(11):4500–4511, 2019

    Shiv Ram Dubey, Soumendu Chakraborty, Swalpa Kumar Roy, Snehasis Mukherjee, Satish Kumar Singh, and Bidyut Baran Chaudhuri. diffgrad: an optimization method for convolutional neural networks.IEEE transactions on neural networks and learning systems, 31(11):4500–4511, 2019

    2019

  31. [31]

    Adaptive subgradient methods for online learning and stochastic optimization.Journal of machine learning research, 12(7), 2011

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of machine learning research, 12(7), 2011

    2011

  32. [32]

    Sharpness-aware minimization for efficiently improving generalization.arXiv preprint arXiv:2010.01412, 2020

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization.arXiv preprint arXiv:2010.01412, 2020

    Pith/arXiv arXiv 2010

  33. [33]

    A stable whitening optimizer for efficient neural network training.Advances in Neural Information Processing Systems, 38:174086–174110, 2026

    Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training.Advances in Neural Information Processing Systems, 38:174086–174110, 2026

    2026

  34. [34]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    arXiv 2024

  35. [35]

    Stochastic gradient methods with layer-wise adaptive moments for training of deep networks.arXiv preprint arXiv:1905.11286, 2019

    Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, Yang Zhang, and Jonathan M Cohen. Stochastic gradient methods with layer-wise adaptive moments for training of deep networks.arXiv preprint arXiv:1905.11286, 2019

    Pith/arXiv arXiv 1905

  36. [36]

    Scalable parameter and memory efficient pretraining for llm: Recent algorithmic advances and benchmarking.arXiv preprint arXiv:2505.22922, 2025

    Athanasios Glentis, Jiaxiang Li, Qiulin Shang, Andi Han, Ioannis Tsaknakis, Quan Wei, and Mingyi Hong. Scalable parameter and memory efficient pretraining for llm: Recent algorithmic advances and benchmarking.arXiv preprint arXiv:2505.22922, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Towards efficient optimizer design for llm via structured fisher approximation with a low-rank extension.arXiv preprint arXiv:2502.07752, 2025

    Wenbo Gong, Meyer Scetbon, Chao Ma, and Edward Meeds. Towards efficient optimizer design for llm via structured fisher approximation with a low-rank extension.arXiv preprint arXiv:2502.07752, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. InInternational Conference on Machine Learning, pages 1842–1850. PMLR, 2018. 79 OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    2018

  39. [39]

    Gcsam: Gradient centralized sharpness aware minimization.IEEE Access, 2025

    Mohamed Hassan, Aleksandar Vakanski, Boyu Zhang, and Min Xian. Gcsam: Gradient centralized sharpness aware minimization.IEEE Access, 2025

    2025

  40. [40]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  41. [41]

    Orthograd improves neural calibration.arXiv preprint arXiv:2506.04487, 2025

    C Evans Hedges. Orthograd improves neural calibration.arXiv preprint arXiv:2506.04487, 2025

    arXiv 2025

  42. [42]

    Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights.arXiv preprint arXiv:2006.08217, 2020

    Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, and Jung-Woo Ha. Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights.arXiv preprint arXiv:2006.08217, 2020

    Pith/arXiv arXiv 2006

  43. [43]

    Gradientstabilizer: Fix the norm, not the gradient

    Tianjin Huang, Zhangyang Wang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Jiaxing Shang, Tianlong Chen, Ke Li, et al. Gradientstabilizer: Fix the norm, not the gradient

  44. [44]

    Spam: Spike-aware adam with momentum reset for stable llm training.arXiv preprint arXiv:2501.06842, 2025

    Tianjin Huang, Ziquan Zhu, Gaojie Jin, Lu Liu, Zhangyang Wang, and Shiwei Liu. Spam: Spike-aware adam with momentum reset for stable llm training.arXiv preprint arXiv:2501.06842, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Dog is sgd’s best friend: A parameter-free dynamic step size schedule, 2023

    Maor Ivgi, Oliver Hinder, and Yair Carmon. Dog is sgd’s best friend: A parameter-free dynamic step size schedule, 2023

    2023

  46. [46]

    Adamd: Improved bias-correction in adam.arXiv preprint arXiv:2110.10828, 2021

    John St John. Adamd: Improved bias-correction in adam.arXiv preprint arXiv:2110.10828, 2021

    Pith/arXiv arXiv 2021

  47. [47]

    On surprising effectiveness of masking updates in adaptive optimizers.arXiv preprint arXiv:2602.15322, 2026

    Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, and Eugene Ie. On surprising effectiveness of masking updates in adaptive optimizers.arXiv preprint arXiv:2602.15322, 2026

    arXiv 2026

  48. [48]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024

    2024

  49. [49]

    Fineweb-edu-100b-shuffle

    Andrej Karpathy. Fineweb-edu-100b-shuffle. https://huggingface.co/datasets/karpathy/ fineweb-edu-100b-shuffle, 2024

    2024

  50. [50]

    Ano: Faster is better in noisy landscape.arXiv preprint arXiv:2508.18258, 2025

    Adrien Kegreisz. Ano: Faster is better in noisy landscape.arXiv preprint arXiv:2508.18258, 2025

    arXiv 2025

  51. [51]

    Improving generalization performance by switching from adam to sgd.arXiv preprint arXiv:1712.07628, 2017

    Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd.arXiv preprint arXiv:1712.07628, 2017

    Pith/arXiv arXiv 2017

  52. [52]

    Dowg unleashed: An efficient universal parameter- free gradient descent method.Advances in Neural Information Processing Systems, 36:6748–6769, 2023

    Ahmed Khaled, Konstantin Mishchenko, and Chi Jin. Dowg unleashed: An efficient universal parameter- free gradient descent method.Advances in Neural Information Processing Systems, 36:6748–6769, 2023

    2023

  53. [53]

    On the insufficiency of existing momentum schemes for stochastic optimization, 2018

    Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of existing momentum schemes for stochastic optimization, 2018

    2018

  54. [54]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  55. [55]

    Learning multiple layers of features from tiny images, 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009

    2009

  56. [56]

    Zclip: Adaptive spike mitigation for llm pre-training.arXiv preprint arXiv:2504.02507, 2025

    Abhay Kumar, Louis Owen, Nilabhra Roy Chowdhury, and Fabian G¨ura. Zclip: Adaptive spike mitigation for llm pre-training.arXiv preprint arXiv:2504.02507, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. InInternational conference on machine learning, pages 5905–5914. PMLR, 2021

    2021

  58. [58]

    Unveiling the backbone-optimizer coupling bias in visual representation learning.arXiv preprint arXiv:2410.06373, 2024

    Siyuan Li, Juanxi Tian, Zedong Wang, Luyuan Zhang, Zicheng Liu, Weiyang Jin, Yang Liu, Baigui Sun, and Stan Z Li. Unveiling the backbone-optimizer coupling bias in visual representation learning.arXiv preprint arXiv:2410.06373, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Taming llms by scaling learning rates with gradient grouping.arXiv preprint arXiv:2506.01049, 2025

    Siyuan Li, Juanxi Tian, Zedong Wang, Xin Jin, Zicheng Liu, Wentao Zhang, and Dan Xu. Taming llms by scaling learning rates with gradient grouping.arXiv preprint arXiv:2506.01049, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    SAC: Adaptive learning rate scaling with architectural constraints, 2026

    Siyuan Li, Juanxi Tian, Zedong Wang, Anna Wang, Xin Jin, Chang Yu, Ruoyu Sun, and Cheng Tan. SAC: Adaptive learning rate scaling with architectural constraints, 2026. URL https://openreview.net/forum? id=EB92tITeNq. 80 OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    2026

  61. [61]

    Friendly sharpness-aware minimiza- tion

    Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimiza- tion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5631–5640, 2024

    2024

  62. [62]

    Preconditioned stochastic gradient descent.IEEE transactions on neural networks and learning systems, 29(5):1454–1466, 2017

    Xi-Lin Li. Preconditioned stochastic gradient descent.IEEE transactions on neural networks and learning systems, 29(5):1454–1466, 2017

    2017

  63. [63]

    Black box lie group preconditioners for sgd.arXiv preprint arXiv:2211.04422, 2022

    Xilin Li. Black box lie group preconditioners for sgd.arXiv preprint arXiv:2211.04422, 2022

    Pith/arXiv arXiv 2022

  64. [64]

    Cautious optimizers: Improving training with one line of code.arXiv preprint arXiv:2411.16085, 2024

    Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. Cautious optimizers: Improving training with one line of code.arXiv preprint arXiv:2411.16085, 2024

    arXiv 2024

  65. [65]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. InInternational Conference on Learning Representations, volume 2024, pages 1621–1650, 2024

    2024

  66. [66]

    Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms.arXiv preprint arXiv:2502.17410, 2025

    Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms.arXiv preprint arXiv:2502.17410, 2025

    arXiv 2025

  67. [67]

    On the variance of the adaptive learning rate and beyond.arXiv preprint arXiv:1908.03265, 2019

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond.arXiv preprint arXiv:1908.03265, 2019

    Pith/arXiv arXiv 1908

  68. [68]

    Focus: First order concentrated updating scheme.arXiv preprint arXiv:2501.12243, 2025

    Yizhou Liu, Ziming Liu, and Jeff Gore. Focus: First order concentrated updating scheme.arXiv preprint arXiv:2501.12243, 2025

    Pith/arXiv arXiv 2025

  69. [69]

    Towards efficient and scalable sharpness- aware minimization

    Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness- aware minimization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360–12370, 2022

    2022

  70. [70]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  71. [71]

    Adasmooth: an adaptive learning rate method based on effective ratio

    Jun Lu. Adasmooth: an adaptive learning rate method based on effective ratio. InSentiment Analysis and Deep Learning: Proceedings of ICSADL 2022, pages 273–293. Springer, 2023

    2022

  72. [72]

    Aggregated momentum: Stability through passive damping.arXiv preprint arXiv:1804.00325, 2018

    James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping.arXiv preprint arXiv:1804.00325, 2018

    Pith/arXiv arXiv 2018

  73. [73]

    Adaptive gradient methods with dynamic bound of learning rate.arXiv preprint arXiv:1902.09843, 2019

    Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate.arXiv preprint arXiv:1902.09843, 2019

    Pith/arXiv arXiv 1902

  74. [74]

    Came: Confidence-guided adaptive memory efficient optimization

    Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You. Came: Confidence-guided adaptive memory efficient optimization. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4442–4453, 2023

    2023

  75. [75]

    Adalomo: Low-memory optimization with adaptive learning rate

    Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. Adalomo: Low-memory optimization with adaptive learning rate. InFindings of the Association for Computational Linguistics: ACL 2024, pages 12486– 12502, 2024

    2024

  76. [76]

    Full parameter fine-tuning for large language models with limited resources

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8187–8198, 2024

    2024

  77. [77]

    Quasi-hyperbolic momentum and adam for deep learning.arXiv preprint arXiv:1810.06801, 2018

    Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and adam for deep learning.arXiv preprint arXiv:1810.06801, 2018

    Pith/arXiv arXiv 2018

  78. [78]

    Torque-aware momentum.arXiv preprint arXiv:2412.18790, 2024

    Pranshu Malviya, Goncalo Mordido, Aristide Baratin, Reza Babanezhad Harikandeh, Gintare Karolina Dziugaite, Razvan Pascanu, and Sarath Chandar. Torque-aware momentum.arXiv preprint arXiv:2412.18790, 2024

    Pith/arXiv arXiv 2024

  79. [79]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. 81 OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    Pith/arXiv arXiv 2016

  80. [80]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391, 2018

    2018

Showing first 80 references.