Pith. sign in

REVIEW 26 cited by

Scalable Second Order Optimization for Deep Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2002.09018 v2 pith:RL4CLPXQ submitted 2020-02-20 cs.LG math.OCstat.ML

Rohan Anil , Vineet Gupta , Tomer Koren , Kevin Regan , Yoram Singer This is my paper · ORCID
classification cs.LGmath.OCstat.ML
keywords optimizationdeeplearningmethodssecondtheoreticalcomparedfirst-order
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent. Second-order optimization methods, that involve second derivatives and/or second order statistics of the data, are far less prevalent despite strong theoretical properties, due to their prohibitive computation, memory and communication costs. In an attempt to bridge this gap between theoretical and practical optimization, we present a scalable implementation of a second-order preconditioned method (concretely, a variant of full-matrix Adagrad), that along with several critical algorithmic and numerical improvements, provides significant convergence and wall-clock time improvements compared to conventional first-order methods on state-of-the-art deep models. Our novel design effectively utilizes the prevalent heterogeneous hardware architecture for training deep models, consisting of a multicore CPU coupled with multiple accelerator units. We demonstrate superior performance compared to state-of-the-art on very large learning tasks such as machine translation with Transformers, language modeling with BERT, click-through rate prediction on Criteo, and image classification on ImageNet with ResNet-50.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 29 citations worldwide. Full citation record

  1. On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning

    cs.LG 2025-02 conditional novelty 7.0 of 10

    For two feature-learning models with anisotropic inputs, KFAC-style layer-wise preconditioning provably recovers features better than SGD and matches ridge regression in the single-index case.

  2. MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Two-sided diagonal preconditioning before Newton-Schulz orthogonalization, plus an adaptive scalar stepsize, improves Muon's GPT-2 pretraining loss at nearly unchanged cost.

  3. Muon Meets Mamba: Spectral Optimization for State Space Models

    cs.LG 2026-08 conditional novelty 6.0 of 10

    On Mamba-2 130M, Muon on the output projection alone gives lower validation loss and better token efficiency than Muon on the input projection or on both, across corpora and budgets.

  4. Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Gradient-projection, Adam-like, and K-FAC-whitened LoRA initializations are all special cases of one two-parameter family, and the best exponents are task-dependent and usually interior.

  5. SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.

  6. Causal Optimizer Interaction Calculus: Hidden Geometric Relaxation and Identifiable Interventions

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Under fixed innovation coupling, finite-horizon optimizers admit minimal pathwise realizations and incidence-identifiable Möbius effects, with a five-term readout transfer from hidden relaxation and a closed reduced-v...

  7. Restricted Dynamic Geometric Complexity: Path-Space Reduction and M\"obius--Jacobi Response

    math.OC 2026-07 conditional novelty 6.0 of 10

    Restricted dynamic geometric complexity measures the intrinsic affine-invariant path distance from an initial metric to a condition-number target when the metric family is structurally constrained, with exact LMI and ...

  8. MatrixFSDP: communication-free matrix optimizers under ZeRO-3 parameter sharding

    cs.DC 2026-07 conditional novelty 6.0 of 10

    MatrixFSDP places whole 2D weight matrices on single ZeRO-3 owner ranks so matrix optimizers run locally without optimizer-step collectives, preserving ZeRO-3 memory while achieving up to 54.6x optimizer-step speedup ...

  9. Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization

    stat.ML 2025-09 conditional novelty 6.0 of 10

    A KL-divergence view of Shampoo and SOAP yields KL-Shampoo and KL-SOAP, with KL-Shampoo outperforming both baselines in LLM pretraining without Adam grafting.

  10. Fantastic Pretraining Optimizers and Where to Find Them

    cs.LG 2025-09 conditional novelty 6.0 of 10

    After careful tuning, alternative optimizers speed up LLM pretraining by only 1.1x to 1.4x over AdamW, and the advantage shrinks with model scale.

  11. Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise

    math.OC 2025-06 reject novelty 6.0 of 10

    Lion and Muon with weight decay are shown to be instances of one stochastic Frank-Wolfe algorithm, and clipped and variance-reduced variants get the first high-probability convergence rates for nonconvex Frank-Wolfe u...

  12. Scalable Thermodynamic Second-order Optimization

    cs.ET 2025-02 conditional novelty 6.0 of 10

    Thermodynamic computers can accelerate K-FAC training by replacing matrix inversions with physical linear-system solves, yielding quadratic rather than cubic per-layer scaling.

  13. Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Optimizers such as Adam, Shampoo, and SOAP are unified as structured Fisher approximations, and two new derived optimizers, RACS and Alice, achieve faster LLaMA pre-training than Adam at lower memory.

  14. Spectral-factorized Positive-definite Curvature Learning for NN Training

    stat.ML 2025-02 conditional novelty 6.0 of 10

    The paper derives a Riemannian update rule for the spectral factors of a positive-definite preconditioner, making arbitrary matrix roots fast and numerically stable for low-precision NN training.

  15. Celo: Training Versatile Learned Optimizers on a Compute Diet

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A compute-cheap learned optimizer, Celo, beats tuned hand-crafted and learned optimizers on a 17-task out-of-distribution suite by combining task augmentation, a simple hierarchical design, and two-stage decoupled met...

  16. SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training

    cs.LG 2024-12 conditional novelty 6.0 of 10

    SWAN, a stateless optimizer combining gradient normalization and whitening, matches or beats Adam on LLaMA pretraining through 1.3B parameters with roughly half the memory and reported 2x token efficiency.

  17. Memory-Efficient 4-bit Preconditioned Stochastic Optimization

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A 4-bit Shampoo optimizer that quantizes Cholesky factors and adds error feedback matches 32-bit Shampoo's accuracy at a fraction of the memory.

  18. MARS: Unleashing the Power of Variance Reduction for Training Large Models

    cs.LG 2024-11 conditional novelty 6.0 of 10

    MARS combines STORM-style variance reduction with preconditioned updates and reports improved token efficiency over AdamW on GPT-2 pretraining, though the analyzed and evaluated variants differ.

  19. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  20. Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature

    cs.AI 2026-08 conditional novelty 5.0 of 10

    GO-MUON combines K-FAC-style second-moment factors with Muon's polar map in a matched spectral oracle, and a four-step refresh schedule trades factor freshness for compute without claiming denoising.

  21. Scale Weight Decay and Train Better

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Muon with weight decay scaled by η/η_max reaches the same MoE validation loss ~30% faster than constant-decay Muon while preserving asymptotic stationarity of the unregularized objective.

  22. Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ML conferences should create an official peer-reviewed track dedicated to refuting and critiquing previously published work.

  23. Learning Physics Informed Neural ODEs With Partial Measurements

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A recursive alternating state-and-parameter estimator lets physics-informed neural ODEs learn dynamics of unmeasured states, outperforming existing baselines on several partially observed systems.

  24. HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization

    cs.LG 2024-11 reject novelty 5.0 of 10

    HELENE is a proposed memory-efficient zeroth-order optimizer with annealed gradients and layer-wise clipped diagonal Hessian, claimed to converge in O(max layer dimension) steps.

  25. Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A practical recipe to convert common neural architectures into discretization-agnostic neural operators, validated by Navier-Stokes experiments showing cross-resolution generalization of FNO-style models.

  26. Improving Adaptive Moment Optimization via Preconditioner Diagonalization

    cs.LG 2025-02 conditional novelty 3.0 of 10

    Rotating gradients into their SVD coordinate system before Adam-style updates can roughly halve the number of steps LLaMA models need to reach a given perplexity.

Pith tools