Pith. sign in

REVIEW 2 major objections 6 minor 87 references

Every descent direction is a preconditioned gradient — and only those

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 17:14 UTC pith:GBXMSZ43

load-bearing objection Conceptual framework for decomposing optimizer updates into geometric + nongeometric modules; correct but elementary math, honestly scoped experiments, one genuinely novel diagnostic. the 2 major comments →

arxiv 2607.07206 v1 pith:GBXMSZ43 submitted 2026-07-08 cs.LG math.OC

Geometric--Nongeometric Optimizer Calculus: A Modular Language for Reachable Gradient Methods

Zavier Li This is my paper
classification cs.LG math.OC
keywords optimizer calculusdirection expressivitypositive-definite geometryrestricted direction residualpreconditioner familiesPareto optimizer designtrajectory residual complexitymodular audit language
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The author proposes that gradient-based optimizers can be usefully audited through a fixed decomposition: a geometric module (a positive-definite cometric mapping gradients to directions) plus seven nongeometric modules (information, memory, control, operator, noise, target, discretization). The geometric module is tested first under a canonical audit order — one asks how much of an observed update direction is explained by positive geometry within a declared metric family, then attributes the remaining residual to specific nongeometric mechanisms. The formal foundation is a direction-expressivity theorem: given a nonzero gradient g and a direction u, there exists a positive-definite matrix P with u = −Pg if and only if g^T u < 0, meaning full positive-definite geometry expresses exactly the strict descent directions away from critical points. Restricted families like diagonal and block-diagonal metrics have clean characterizations of which directions they can and cannot express, yielding coordinate-wise and block-wise descent conditions. The paper lifts this pointwise diagnostic to a trajectory level by requiring that an entire optimizer trace be explained by a path of preconditioners with bounded variation, not just independent per-step fits. The overall design problem becomes a Pareto optimization over module budgets rather than a single universal optimizer ranking. Diagnostic prototypes on quadratic benchmarks and small-scale MNIST experiments illustrate the language but do not claim large-scale competitiveness.

Core claim

The central formal result is a direction-expressivity theorem: given a nonzero gradient g and a direction u in R^d, there exists a symmetric positive-definite matrix P such that u = −Pg if and only if g^T u < 0. This means that full positive-definite geometry can produce exactly the strict descent directions — no more, no less. At critical points where g = 0, no positive geometry can produce nonzero motion, so any movement must come from nongeometric modules. The paper then defines a restricted direction residual ρ_{F,h}(u; α) that measures the minimum distance between an observed update direction u and any direction producible by a positive-definite matrix from a declared family F appliedto

What carries the argument

The restricted direction residual ρ_{F,h}(u; α) = inf_{P ∈ F} ||u + Pα||_h, which measures how much of an observed optimizer update direction falls outside what a given metric family can explain through positive geometry alone.

Load-bearing premise

The framework's practical value rests on the premise that testing geometric explanation first, then attributing residuals to nongeometric modules, yields diagnostic information more actionable than simpler measures like gradient-update alignment — a premise tested only on an 8-step MNIST trace and deterministic quadratics.

What would settle it

If the direction residual turns out to be redundant with simple alignment metrics (e.g., cosine similarity between update and negative gradient) for real optimizer traces, the decomposition collapses to a relabeling of known quantities rather than a new diagnostic.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper introduces a modular language for decomposing gradient-based optimizer updates into a geometric module (positive-definite cometric family) and seven nongeometric modules (information, memory, control, operator, noise, target, discretization). The main formal result (Proposition 1) establishes that full SPD geometry expresses exactly the strict descent directions away from critical points. The paper then defines restricted direction residuals for diagonal and block families, separates direction-level diagnostics from condition-number complexity, formulates optimizer design as a Pareto problem over module budgets, and introduces trajectory-level residual complexity. Diagnostic prototypes on deterministic quadratics and a small MNIST subset illustrate the framework. The proofs in Appendix A are correct and straightforward.

Significance. The paper's main contribution is conceptual: it provides a structured audit language for attributing optimizer behavior to geometric versus nongeometric mechanisms. Proposition 1 is a clean expressivity result with a valid constructive proof. The diagonal and block expressivity conditions (Propositions 2-3) are useful and correct. The trajectory-level residual complexity (Eq. 7) and the coherence gap diagnostic are novel and potentially useful for auditing training traces. The bounded diagonal residual (Proposition 7) has a clean closed form. The framework is honestly scoped as a theory and benchmark-language paper, with experiments explicitly labeled as diagnostic rather than competitive.

major comments (2)
  1. §3.5 and §7.4: The canonical audit order fixes the visible covector as the raw gradient g_k before testing geometric explanation. For AdamW, the update u_k = -η·diag(v_k)^{-1/2}·m_k is driven by momentum m_k, not g_k. Under this convention, the residual will be large simply because u_k is not aligned with g_k — which is already obvious from the update formula. The 8-step MNIST trace (§7.4) reports GER values (AdamW: 0.387 open, 0.153 bounded) but never demonstrates that these numbers guide a design decision that inspecting the update formula would not. The paper should either (a) show a concrete case where the residual decomposition reveals something non-obvious about an optimizer that direct formula inspection misses, or (b) argue more rigorously for why the formal decomposition adds value beyond relabeling known mechanisms. Without this, the framework's practical utility rests on an un
  2. §7.3 and §8: The narrow parameter-space cometric convention means the Newton-Schulz polar factor in Muon-style updates 'is not automatically an SPD cometric applied to the current visible gradient,' so most of Muon's update is classified as nongeometric residual. This seems to miss the geometric structure Muon is explicitly designed to exploit. The paper acknowledges this (§8: 'Momentum may be geometric in an enlarged phase space') but does not address whether the framework can be extended to capture matrix-operator geometry without trivializing it as residual. This is load-bearing because the Muon prototype is the only non-quadratic engineering evidence. The paper should either show that the audit still extracts useful information for this class of methods, or clarify that the framework's scope excludes matrix-operator optimizers in their intended geometric interpretation.
minor comments (6)
  1. §4.1, Eq. (5): The geometric explanation rate GER can be negative when the best geometric approximation is worse than zero. The paper mentions a clipped version but does not define it. This should be clarified.
  2. §6.3, Eq. (7): The trajectory-level residual complexity TRC involves a variation budget B_geo on the geometry distance d_F. The choice of d_F (affine-invariant, log-diagonal, etc.) is declared but its sensitivity to the diagnostic is not discussed. A brief remark on how the choice of d_F affects the coherence gap would help.
  3. Table 1: The 'Calls' column mixes gradient calls across methods with different information patterns (full probes vs. single gradients). The caption notes this but a normalized comparison (e.g., per-oracle-call gap reduction) would improve readability.
  4. §7.4: The trace audit uses only 8 steps. The paper should note whether the GER and coherence gap values stabilize or change qualitatively with trace length.
  5. §5.3: The four-quadrant table is a useful diagnostic device but remains qualitative. A brief example showing each quadrant for a concrete optimizer/family pair would make the diagnostic more concrete.
  6. References: The two cited prior works by the same author (Li, 2026a, 2026b) are listed as 'arXiv preprint' without arXiv identifiers. These should be included for reproducibility.

Circularity Check

0 steps flagged

No circularity found: formal results are self-contained with complete proofs; self-citations are explicitly declared non-load-bearing.

full rationale

The paper's main formal results (Propositions 1–7) all have self-contained proofs in Appendix A that depend on no external citations. Proposition 1 (the direction-expressivity theorem) is a direct linear-algebraic construction with a complete proof. The restricted residuals (Propositions 2–3) follow from coordinate-wise or block-wise application of Proposition 1. The trajectory-level results (Propositions 4–5) are standard smoothness/Hölder arguments. The two self-citations (Li 2026a, 2026b) are used only for motivation and terminology, and the paper explicitly states in §1: 'The present paper does not require those papers as prerequisites: its definitions, direction-expressivity results, restricted residuals, and diagnostic prototypes are stated self-containedly.' No uniqueness theorem is imported from prior work. No parameter is fitted to data and then presented as a prediction. The diagnostic prototypes (quadratic benchmark, MNIST subset) are presented as illustrations of the language, not as derivations whose outputs are forced by their inputs. The GNG-FullMetricProbe solving quadratics to numerical precision is simply Newton's method on quadratics, which the paper transparently acknowledges. The non-uniqueness of the decomposition (§3.5) is a stated limitation, not a circularity. The skeptic's concern about convention-dependence and practical utility is a correctness/impact risk, not a circularity in the derivation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 3 invented entities

The paper introduces three diagnostic quantities (residual, TRC, GER) that are computable from standard optimizer traces. No new physical entities or unobservable postulates are introduced.

free parameters (3)
  • Bounded diagonal range [λ, L] = [10^{-6}, 1] = [1e-6, 1]
    Chosen by hand for the bounded diagonal GER diagnostic in §7.4; not fitted to data but selected for engineering plausibility.
  • Quadratic variation weight β = not specified
    Appears in the bounded diagonal trajectory residual optimization (§6.3) but no value is given; the diagnostic results in §7.4 use B_geo=0 (shared scale), so β is not exercised.
  • GNGMuon learning rate and matrix scale = lr=0.03, matrix_scale=1.5
    Selected from a small grid for the MNIST subset experiment (Appendix B.3); not a free parameter of the theory but a tuned hyperparameter for the prototype.
axioms (3)
  • standard math The parameter space is a finite-dimensional smooth manifold with a well-defined gradient covector df_θ at each point.
    Invoked throughout §3-4; standard differential geometry assumption for Riemannian optimization.
  • ad hoc to paper The canonical audit order (geometry first, then residuals) is a meaningful convention for decomposing optimizer updates.
    §3.5 states this is a convention; its usefulness is assumed but only weakly tested in §7.4.
  • domain assumption Reachable optimizer classes are well-defined by the quadruple (oracle, budget, state, rules).
    Definition 1 (§3.1); semantic rather than syntactic, which is reasonable but means claims are relative to declared constraints.
invented entities (3)
  • Restricted direction residual ρ_{F,h}(u;α) independent evidence
    purpose: Measures how much of an update direction u cannot be explained by a metric family F given covector α.
    Falsifiable: can be computed on any optimizer trace with gradients and parameter updates; the paper computes it on MNIST traces (§7.4).
  • Trajectory-level residual complexity TRC^{B_geo}_{p,q} independent evidence
    purpose: Couples pointwise direction mismatch with bounded metric variation over an optimizer trace.
    Falsifiable diagnostic; computed on 8-step traces in §7.4, though only for the B_geo=0 specialization.
  • Geometric explanation rate GER_{F,h} independent evidence
    purpose: Scalar score for how well a metric family explains an update direction.
    Directly computable from observed updates and gradients; reported in §7.4.

pith-pipeline@v1.1.0-glm · 18213 in / 3933 out tokens · 224133 ms · 2026-07-09T17:14:48.901874+00:00 · methodology

0 comments
read the original abstract

Adaptive optimizers mix several mechanisms: a metric or preconditioner maps gradients to descent directions, while estimation, memory, step-size control, constraints, stochasticity, target modification, and discretization determine which directions are available and how they are used. We introduce geometric--nongeometric optimizer calculus, a modular language for auditing reachable gradient methods under explicit oracle, budget, state, and rule constraints. The geometric module is a positive cometric family that maps covectors to parameter-space directions; the nongeometric modules are information, memory, control, operator, noise, target, and discretization mechanisms. The main formal result is a direction-expressivity theorem: away from critical points, full positive-definite geometry expresses exactly the strict descent directions. We then define restricted direction residuals for admissible metric families, prove exact expressivity conditions for diagonal and block geometries, and separate this direction-level diagnostic from condition-number geometric complexity. The resulting design problem is a Pareto optimization over module budgets, not a single universal optimizer ordering. We also lift pointwise residuals to a trajectory-level residual complexity that couples direction mismatch with the variation of the explaining geometry. We include diagnostic prototypes only as evidence for the language: a high-information full-metric probe solves deterministic quadratic benchmarks to numerical precision, while a practical Muon-style PyTorch candidate gives small-scale evidence that matrix-operator updates can be audited by the calculus. The paper is a theory and benchmark-language manuscript; it does not claim large-scale optimizer state-of-the-art performance.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

87 extracted references · 87 canonical work pages · 13 internal anchors

  1. [1]

    Journal of Machine Learning Research , year =

    A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , author =. Journal of Machine Learning Research , year =

    work page

  2. [2]

    Proceedings of the National Academy of Sciences , year =

    A Variational Perspective on Accelerated Methods in Optimization , author =. Proceedings of the National Academy of Sciences , year =

    work page

  3. [3]

    Advances in Neural Information Processing Systems , year =

    Accelerated Mirror Descent in Continuous and Discrete Time , author =. Advances in Neural Information Processing Systems , year =

    work page

  4. [4]

    Neural Computation , year =

    Natural Gradient Works Efficiently in Learning , author =. Neural Computation , year =

    work page

  5. [6]

    Proceedings of the 32nd International Conference on Machine Learning , year =

    Optimizing Neural Networks with Kronecker-factored Approximate Curvature , author =. Proceedings of the 32nd International Conference on Machine Learning , year =

    work page

  6. [7]

    A Kronecker-factored approximate Fisher matrix for convolution layers

    A Kronecker-factored Approximate Fisher Matrix for Convolution Layers , author =. Proceedings of the 33rd International Conference on Machine Learning , year =. 1602.01407 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [8]

    Advances in Neural Information Processing Systems , year =

    Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis , author =. Advances in Neural Information Processing Systems , year =

    work page

  8. [9]

    Linear Algebra for Large Scale and Real-Time Applications , editor =

    Approximation with Kronecker Products , author =. Linear Algebra for Large Scale and Real-Time Applications , editor =. 1993 , doi =

    work page 1993

  9. [10]

    Journal of Statistical Computation and Simulation , volume =

    The MLE Algorithm for the Matrix Normal Distribution , author =. Journal of Statistical Computation and Simulation , volume =. 1999 , doi =

    work page 1999

  10. [11]

    IEEE Transactions on Signal Processing , volume =

    On Estimation of Covariance Matrices with Kronecker Product Structure , author =. IEEE Transactions on Signal Processing , volume =. 2008 , doi =

    work page 2008

  11. [12]

    Journal of Machine Learning Research , year =

    Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , author =. Journal of Machine Learning Research , year =

    work page

  12. [13]

    International Conference on Learning Representations , year =

    Adam: A Method for Stochastic Optimization , author =. International Conference on Learning Representations , year =

    work page

  13. [14]

    International Conference on Learning Representations , year =

    Decoupled Weight Decay Regularization , author =. International Conference on Learning Representations , year =

    work page

  14. [15]

    Proceedings of the 35th International Conference on Machine Learning , year =

    Shampoo: Preconditioned Stochastic Tensor Optimization , author =. Proceedings of the 35th International Conference on Machine Learning , year =

    work page

  15. [16]

    Scalable Second Order Optimization for Deep Learning

    Scalable Second Order Optimization for Deep Learning , author =. arXiv preprint arXiv:2002.09018 , year =

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  16. [17]

    2024 , eprint =

    SOAP: Improving and Stabilizing Shampoo using Adam , author =. 2024 , eprint =

    work page 2024

  17. [18]

    Machine Learning , year =

    Logarithmic Regret Algorithms for Online Convex Optimization , author =. Machine Learning , year =

    work page

  18. [19]

    Proceedings of the 35th International Conference on Machine Learning , year =

    Adafactor: Adaptive Learning Rates with Sublinear Memory Cost , author =. Proceedings of the 35th International Conference on Machine Learning , year =

    work page

  19. [20]

    Advances in Neural Information Processing Systems , year =

    Memory Efficient Adaptive Optimization , author =. Advances in Neural Information Processing Systems , year =

    work page

  20. [21]

    2023 , eprint =

    Symbolic Discovery of Optimization Algorithms , author =. 2023 , eprint =

    work page 2023

  21. [22]

    2023 , eprint =

    Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training , author =. 2023 , eprint =

    work page 2023

  22. [23]

    2024 , howpublished =

    Muon: An Optimizer for Hidden Layers in Neural Networks , author =. 2024 , howpublished =

    work page 2024

  23. [24]

    2025 , eprint =

    Practical Efficiency of Muon for Pretraining , author =. 2025 , eprint =

    work page 2025

  24. [25]

    2025 , eprint =

    An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants , author =. 2025 , eprint =

    work page 2025

  25. [26]

    Proceedings of the 36th International Conference on Machine Learning , year =

    Efficient Full-Matrix Adaptive Regularization , author =. Proceedings of the 36th International Conference on Machine Learning , year =

    work page

  26. [27]

    Bregman dynamics, contact transformations and convex optimization

    Bregman Dynamics, Contact Transformations and Convex Optimization , author =. arXiv preprint arXiv:1912.02928 , year =

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  27. [28]

    arXiv preprint arXiv:2505.12553 , year =

    Hamiltonian Descent Algorithms for Optimization: Accelerated Rates via Randomized Integration Time , author =. arXiv preprint arXiv:2505.12553 , year =

    work page arXiv

  28. [29]

    Accelerated Convex Optimization via Hamiltonian Dynamics with Deterministic Integration Time

    Accelerated Convex Optimization via Hamiltonian Dynamics with Deterministic Integration Time , author =. arXiv preprint arXiv:2606.17260 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [30]

    Conformal Symplectic Optimization for Stable Reinforcement Learning

    Conformal Symplectic Optimization for Stable Reinforcement Learning , author =. arXiv preprint arXiv:2412.02291 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [31]

    Journal of Machine Learning Research , year =

    Learning Discretized Neural Networks under Ricci Flow , author =. Journal of Machine Learning Research , year =

    work page

  31. [32]

    arXiv preprint arXiv:2509.22362 , year =

    Neural Feature Geometry Evolves as Discrete Ricci Flow , author =. arXiv preprint arXiv:2509.22362 , year =

    work page arXiv

  32. [33]

    International Conference on Machine Learning , year =

    Revisiting Over-smoothing and Over-squashing Using Ollivier-Ricci Curvature , author =. International Conference on Machine Learning , year =

    work page

  33. [34]

    Physical Review Research , year =

    Improving Gradient Methods via Coordinate Transformations: Applications to Quantum Machine Learning , author =. Physical Review Research , year =

    work page

  34. [35]

    Journal of Machine Learning Research , year =

    The Z-Gromov-Wasserstein Distance , author =. Journal of Machine Learning Research , year =

    work page

  35. [36]

    Optimization Algorithms on Matrix Manifolds , author =

    work page

  36. [37]

    2023 , doi =

    An Introduction to Optimization on Smooth Manifolds , author =. 2023 , doi =

    work page 2023

  37. [38]

    2004 , url =

    Convex Optimization , author =. 2004 , url =

    work page 2004

  38. [39]

    SIAM Review , volume =

    Semidefinite Programming , author =. SIAM Review , volume =. 1996 , doi =

    work page 1996

  39. [40]

    Positive Definite Matrices , author =

    work page

  40. [41]

    2008 , doi =

    Functions of Matrices: Theory and Computation , author =. 2008 , doi =

    work page 2008

  41. [42]

    1997 , doi =

    Matrix Analysis , author =. 1997 , doi =

    work page 1997

  42. [43]

    International Journal of Computer Vision , volume =

    A Riemannian Framework for Tensor Computing , author =. International Journal of Computer Vision , volume =. 2006 , doi =

    work page 2006

  43. [44]

    Journal of Differential Geometry , volume =

    Three-Manifolds with Positive Ricci Curvature , author =. Journal of Differential Geometry , volume =. 1982 , doi =

    work page 1982

  44. [45]

    Journal of Differential Geometry , volume =

    Deforming Metrics in the Direction of Their Ricci Tensors , author =. Journal of Differential Geometry , volume =. 1983 , doi =

    work page 1983

  45. [46]

    USSR Computational Mathematics and Mathematical Physics , volume =

    Some Methods of Speeding Up the Convergence of Iteration Methods , author =. USSR Computational Mathematics and Mathematical Physics , volume =. 1964 , doi =

    work page 1964

  46. [47]

    Doklady Akademii Nauk SSSR , volume =

    A Method for Solving the Convex Programming Problem with Convergence Rate \(O(1/k^2)\) , author =. Doklady Akademii Nauk SSSR , volume =

    work page

  47. [48]

    Problem Complexity and Method Efficiency in Optimization , author =

    work page

  48. [49]

    Operations Research Letters , volume =

    Mirror Descent and Nonlinear Projected Subgradient Methods for Convex Optimization , author =. Operations Research Letters , volume =. 2003 , doi =

    work page 2003

  49. [50]

    Optimization , year =

    Variable Metric Forward--Backward Splitting with Applications to Monotone Inclusions in Duality , author =. Optimization , year =. doi:10.1080/02331934.2012.733883 , url =

    work page doi:10.1080/02331934.2012.733883 2012

  50. [51]

    2006 , doi =

    Numerical Optimization , author =. 2006 , doi =

    work page 2006

  51. [52]

    SIAM Review , volume =

    Optimization Methods for Large-Scale Machine Learning , author =. SIAM Review , volume =. 2018 , doi =

    work page 2018

  52. [53]

    Numerische Mathematik , volume =

    Condition Numbers and Equilibration of Matrices , author =. Numerische Mathematik , volume =. 1969 , doi =

    work page 1969

  53. [54]

    Numerische Mathematik , volume =

    Optimally Scaled Matrices , author =. Numerische Mathematik , volume =. 1963 , doi =

    work page 1963

  54. [55]

    Numerische Mathematik , volume =

    Optimally Scaled Matrices, Necessary and Sufficient Conditions , author =. Numerische Mathematik , volume =. 1982 , doi =

    work page 1982

  55. [56]

    SIAM Journal on Numerical Analysis , volume =

    Optimal Block Diagonal \(l_2\)-Scaling of Matrices , author =. SIAM Journal on Numerical Analysis , volume =. 1985 , doi =

    work page 1985

  56. [57]

    SIAM Journal on Optimization , volume =

    Optimizing Condition Numbers , author =. SIAM Journal on Optimization , volume =. 2009 , doi =

    work page 2009

  57. [58]

    SIAM Journal on Matrix Analysis and Applications , volume =

    Minimizing Condition Number via Convex Programming , author =. SIAM Journal on Matrix Analysis and Applications , volume =. 2011 , doi =

    work page 2011

  58. [59]

    Operations Research , volume =

    Optimal Diagonal Preconditioning , author =. Operations Research , volume =. 2025 , doi =

    work page 2025

  59. [60]

    Pacific Journal of Mathematics , volume =

    Concerning Nonnegative Matrices and Doubly Stochastic Matrices , author =. Pacific Journal of Mathematics , volume =. 1967 , doi =

    work page 1967

  60. [61]

    IMA Journal of Numerical Analysis , volume =

    A Fast Algorithm for Matrix Balancing , author =. IMA Journal of Numerical Analysis , volume =. 2013 , doi =

    work page 2013

  61. [62]

    Foundations and Trends in Machine Learning , volume =

    An Introduction to Matrix Concentration Inequalities , author =. Foundations and Trends in Machine Learning , volume =. 2015 , doi =

    work page 2015

  62. [63]

    Absil, Robert Mahony, and Rodolphe Sepulchre

    P.-A. Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008

    work page 2008

  63. [64]

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer

    Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10 0 (2): 0 251--276, 1998. doi:10.1162/089976698300017746

    work page doi:10.1162/089976698300017746 1998

  64. [65]

    Memory efficient adaptive optimization

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/8f1fa0193ca2b5d2fa0695827d8270e9-Abstract.html

    work page 2019

  65. [66]

    Rajendra Bhatia.Positive Definite Matrices

    Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31 0 (3): 0 167--175, 2003. doi:10.1016/S0167-6377(02)00231-6

    work page doi:10.1016/s0167-6377(02)00231-6 2003

  66. [67]

    Nicolas Boumal.An Introduction to Optimization on Smooth Manifolds

    L \'e on Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60 0 (2): 0 223--311, 2018. doi:10.1137/16M1080173

    work page doi:10.1137/16m1080173 2018

  67. [68]

    Stephen Boyd and Lieven Vandenberghe.Convex Optimization

    Nicolas Boumal. An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, 2023. doi:10.1017/9781009166164

    work page doi:10.1017/9781009166164 2023

  68. [69]

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms, 2023. URL https://arxiv.org/abs/2302.06675

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  69. [70]

    Michael Crawshaw, Chirag Modi, Mingrui Liu, and Robert M. Gower. An exploration of non-euclidean gradient descent: Muon and its many variants, 2025. URL https://arxiv.org/abs/2510.09827

    work page arXiv 2025

  70. [71]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12: 0 2121--2159, 2011. URL https://jmlr.org/papers/v12/duchi11a.html

    work page 2011

  71. [72]

    Practical Efficiency of Muon for Pretraining

    Essential AI , Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J. Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashi...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  72. [73]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 1842--1850. PMLR, 2018. URL https://proceedings.mlr.press/v80/gupta18a.html

    work page 2018

  73. [74]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Blog post

    work page 2024

  74. [75]

    Adam: A Method for Stochastic Optimization

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https://arxiv.org/abs/1412.6980. arXiv:1412.6980

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  75. [76]

    Optimization geometrodynamics: A framework for dynamic geometric optimization , 2026 a

    Zavier Li. Optimization geometrodynamics: A framework for dynamic geometric optimization , 2026 a . arXiv preprint

    work page 2026

  76. [77]

    Restricted Dynamic Geometric Complexity: Certificates for Structured Preconditioning , 2026 b

    Zavier Li. Restricted Dynamic Geometric Complexity: Certificates for Structured Preconditioning , 2026 b . arXiv preprint

    work page 2026

  77. [78]

    Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training, 2023. URL https://arxiv.org/abs/2305.14342

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  78. [79]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1711.05101. arXiv:1711.05101

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  79. [80]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp.\ 2408--2417. PMLR, 2015. URL https://proceedings.mlr.press/v37/martens15.html

    work page 2015

  80. [81]

    Nesterov

    Yurii E. Nesterov. A method for solving the convex programming problem with convergence rate \(o(1/k^2)\). Doklady Akademii Nauk SSSR, 269 0 (3): 0 543--547, 1983

    work page 1983

Showing first 80 references.