Pith. sign in

REVIEW 26 cited by

Adam-mini: Use Fewer Learning Rates To Gain More

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.16793 v7 pith:OORJVZ5S submitted 2024-06-24 cs.LG cs.AI

Adam-mini: Use Fewer Learning Rates To Gain More

Yushun Zhang , Congliang Chen , Ziniu Li , Tian Ding , Chenwei Wu , Diederik P. Kingma , Yinyu Ye , Zhi-Quan Luo
show 1 more author
Ruoyu Sun
This is my paper
classification cs.LG cs.AI
keywords adam-minilearningadamwfindmemorypre-trainingratesachieves
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We propose Adam-mini, an optimizer that achieves on par or better performance than AdamW with 50% less memory footprint. Adam-mini reduces memory by cutting down the learning rate resources in Adam (i.e., $1/\sqrt{v}$). By investigating the Hessian structure of neural nets, we find Adam's $v$ might not function at its full potential as effectively as we expected. We find that $\geq$ 99.9% of these learning rates in $v$ could be harmlessly removed if we (1) carefully partition the parameters into blocks following our new principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. We then provide one simple way to find good learning rates and propose Adam-mini. Empirically, we verify that Adam-mini performs on par or better than AdamW on various language models sized from 39M to 13B for pre-training, supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini also alleviates communication overheads among GPUs, thereby increasing throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW when pre-training Llama 2-7B on $2\times$ A800-80GB GPUs, which saves 33% wall-clock time for pre-training.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 26 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    cs.LG 2026-07 conditional novelty 7.0

    The top-r gradient subspace in GaLore-family optimizers is statistically non-identifiable beyond ~39 of 128 directions, and the fix is to transport optimizer state across refreshes rather than stabilize the basis.

  2. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    cs.LG 2026-07 conditional novelty 7.0

    The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.

  3. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    math.OC 2026-06 unverdicted novelty 7.0

    OptMuon combines orthogonalized momentum with closed-loop adaptation to achieve noise-adaptive convergence rates that automatically become near-optimal deterministic first-order rates without retuning when noise vanishes.

  4. Why Muon Outperforms Adam: A Curvature Perspective

    cs.LG 2026-06 conditional novelty 7.0

    Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.

  5. Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens

    cs.LG 2026-05 unverdicted novelty 7.0

    A policy optimization perspective unifies zeroth-order Hessian estimators as baseline selections and yields ZoVH, a suite of low-variance estimators for the Hessian, its regularized inverse, and bias-corrected product...

  6. Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

    cs.LG 2026-01 unverdicted novelty 7.0

    Deep linear networks with balanced data covariance exhibit Hessian spectral bifurcation whose dominant-to-bulk eigenvalue ratio scales linearly with depth.

  7. FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

    cs.LG 2026-07 conditional novelty 6.5

    Global-aware coordinate trust modulation after corrected AdamW updates improves federated Transformer and LLM training under data heterogeneity over strong adaptive baselines.

  8. FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training

    cs.LG 2026-06 unverdicted novelty 6.0

    FORGE eliminates stored gradients by fusing element-wise optimizer steps into the backward pass on a per-tile register basis, delivering over 2x memory reduction and 1.5x speedup at small batches with provable exactne...

  9. FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training

    cs.LG 2026-06 conditional novelty 6.0

    FORGE removes the materialized weight gradient by fusing element-wise optimizer updates into per-tile backward GEMM epilogues, cutting peak optimizer memory 38-53% and running 1.37-1.69x faster in the small-batch regime.

  10. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    math.OC 2026-06 unverdicted novelty 6.0

    OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to...

  11. One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

    cs.LG 2026-05 conditional novelty 6.0

    Heavy-tail guided layerwise learning rates improve LLM convergence speed and generalization across LLaMA, GPT variants, AdamW and Muon optimizers from 60M to 1B parameters.

  12. ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

    cs.LG 2026-05 conditional novelty 6.0

    ChunkFT enables full-parameter fine-tuning of Llama 3-8B on one 24 GB GPU and Llama 3-70B on two 80 GB GPUs by streaming gradients over dynamically activated sub-tensors.

  13. STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training

    cs.LG 2026-04 unverdicted novelty 6.0

    STQuant dynamically allocates quantization bits for optimizer states in multimodal model training, reducing memory by 84.4% to an average 5.1 bits while preserving quality on GPT-2 and ViT.

  14. Optimal Projection-Free Adaptive SGD for Matrix Optimization

    math.OC 2026-04 unverdicted novelty 6.0

    Proving stability of Leon's preconditioner enables the first tuning-free Nesterov-accelerated projection-free adaptive SGD variant with improved non-smooth non-convex rates.

  15. Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization

    cs.LG 2026-02 conditional novelty 6.0

    DeVA_S8 reweights Muon's matrix-sign update in the matrix's eigenbasis with a singular-value signal-to-noise ratio, reaching target LLM validation perplexity with ~6.6% fewer tokens than Muon.

  16. Muon in Associative Memory Learning: Training Dynamics and Scaling Laws

    cs.LG 2026-02 conditional novelty 6.0

    In a linear softmax memory model, Muon equalizes learning across frequency tiers and gives exponential (noiseless) or T^{-2} (noisy power-law) convergence, versus polynomial or T^{-(1-1/β)} for gradient descent.

  17. Decentralized SGD with Controlled Disagreement Finds Flatter Minima

    cs.LG 2026-02 conditional novelty 6.0

    Keeping consensus errors alive in decentralized SGD via a learning-rate-scaled mixing term improves test accuracy and flatter minima over both DSGD and synchronous SGD.

  18. Memory-Efficient LLM Pretraining via Minimalist Optimizer Design

    cs.LG 2025-06 conditional novelty 6.0

    SCALE matches Adam performance in LLM pretraining from 60M to 7B parameters by combining column-wise gradient normalization with last-layer-only momentum, using 35-45% of Adam's memory.

  19. GWT: Scalable Optimizer State Compression for Large Language Model Training

    cs.LG 2025-01 unverdicted novelty 6.0

    GWT projects gradients into wavelet subspaces to compress optimizer states for memory-efficient LLM training while claiming performance parity with full-rank updates.

  20. OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

    cs.AI 2026-06 reject novelty 5.0

    A multi-agent typed-DSL search discovers RS-MR, a Muon-like matrix optimizer that beats Muon by 0.57% validation BPB on one 3.15B-token transformer pretraining run, without error bars.

  21. One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

    cs.LG 2026-05 unverdicted novelty 5.0

    LLR uses heavy-tailed self-regularization theory to set per-layer learning rates in Transformers, yielding faster convergence and higher zero-shot accuracy than uniform rates across model scales.

  22. AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments

    cs.LG 2026-05 unverdicted novelty 5.0

    AdaMeZO adapts Adam moment estimates to zeroth-order LLM fine-tuning without extra memory storage, outperforming MeZO with up to 70% fewer forward passes.

  23. HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

    cs.LG 2026-03 unverdicted novelty 5.0

    HTMuon modifies Muon to produce heavier-tailed updates and weight spectra via HT-SR theory, yielding up to 0.98 lower perplexity on LLaMA pretraining and serving as a plug-in for other Muon variants.

  24. Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

    cs.LG 2026-01 conditional novelty 5.0

    Even with perfectly balanced and whitened data, a deep linear network's Hessian exhibits a two-cluster spectrum whose dominant-to-bulk eigenvalue ratio grows linearly with depth.

  25. SpectralTrain: A Universal Framework for Hyperspectral Image Classification

    cs.CV 2025-11 unverdicted novelty 5.0

    SpectralTrain is a universal training framework that combines curriculum learning and PCA spectral downsampling to deliver 2-7x faster training for hyperspectral image classification across multiple backbones and data...

  26. On the Convergence Analysis of Muon

    stat.ML 2025-05 unverdicted novelty 5.0

    Convergence analysis shows Muon outperforms gradient descent by exploiting low-rank structure in neural network Hessians.