The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.
hub
Adam-mini: Use fewer learning rates to gain more.arXiv preprint arXiv:2406.16793
16 Pith papers cite this work, alongside 1 external citations. Polarity classification is still indexing.
16
Pith papers citing it
1
external citations · Pith
abstract
We propose Adam-mini, an optimizer that achieves on par or better performance than AdamW with 50% less memory footprint. Adam-mini reduces memory by cutting down the learning rate resources in Adam (i.e., $1/\sqrt{v}$). By investigating the Hessian structure of neural nets, we find Adam's $v$ might not function at its full potential as effectively as we expected. We find that $\geq$ 99.9% of these learning rates in $v$ could be harmlessly removed if we (1) carefully partition the parameters into blocks following our new principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. We then provide one simple way to find good learning rates and propose Adam-mini. Empirically, we verify that Adam-mini performs on par or better than AdamW on various language models sized from 39M to 13B for pre-training, supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini also alleviates communication overheads among GPUs, thereby increasing throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW when pre-training Llama 2-7B on $2\times$ A800-80GB GPUs, which saves 33% wall-clock time for pre-training.
hub tools
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
A policy optimization perspective unifies zeroth-order Hessian estimators as baseline selections and yields ZoVH, a suite of low-variance estimators for the Hessian, its regularized inverse, and bias-corrected products, with theoretical guarantees.
FORGE removes the materialized weight gradient by fusing element-wise optimizer updates into per-tile backward GEMM epilogues, cutting peak optimizer memory 38-53% and running 1.37-1.69x faster in the small-batch regime.
OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to near-optimal deterministic first-order rates in the zero-noise regime.
ChunkFT enables full-parameter fine-tuning of Llama 3-8B on one 24 GB GPU and Llama 3-70B on two 80 GB GPUs by streaming gradients over dynamically activated sub-tensors.
STQuant dynamically allocates quantization bits for optimizer states in multimodal model training, reducing memory by 84.4% to an average 5.1 bits while preserving quality on GPT-2 and ViT.
Proving stability of Leon's preconditioner enables the first tuning-free Nesterov-accelerated projection-free adaptive SGD variant with improved non-smooth non-convex rates.
SCALE matches Adam performance in LLM pretraining from 60M to 7B parameters by combining column-wise gradient normalization with last-layer-only momentum, using 35-45% of Adam's memory.
Muon's convergence rate depends on an average Hessian curvature along its update directions, which can be much smaller than the worst-case Lipschitz constant when Hessians are low-rank.
GWT projects gradients into wavelet subspaces to compress optimizer states for memory-efficient LLM training while claiming performance parity with full-rank updates.
LLR uses heavy-tailed self-regularization theory to set per-layer learning rates in Transformers, yielding faster convergence and higher zero-shot accuracy than uniform rates across model scales.
AdaMeZO adapts Adam moment estimates to zeroth-order LLM fine-tuning without extra memory storage, outperforming MeZO with up to 70% fewer forward passes.
HTMuon modifies Muon to produce heavier-tailed updates and weight spectra via HT-SR theory, yielding up to 0.98 lower perplexity on LLaMA pretraining and serving as a plug-in for other Muon variants.
Even with perfectly balanced and whitened data, a deep linear network's Hessian exhibits a two-cluster spectrum whose dominant-to-bulk eigenvalue ratio grows linearly with depth.
SpectralTrain is a universal training framework that combines curriculum learning and PCA spectral downsampling to deliver 2-7x faster training for hyperspectral image classification across multiple backbones and datasets with only small accuracy trade-offs.
citing papers explorer
-
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens
A policy optimization perspective unifies zeroth-order Hessian estimators as baseline selections and yields ZoVH, a suite of low-variance estimators for the Hessian, its regularized inverse, and bias-corrected products, with theoretical guarantees.
-
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
FORGE removes the materialized weight gradient by fusing element-wise optimizer updates into per-tile backward GEMM epilogues, cutting peak optimizer memory 38-53% and running 1.37-1.69x faster in the small-batch regime.
-
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to near-optimal deterministic first-order rates in the zero-noise regime.
-
ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning
ChunkFT enables full-parameter fine-tuning of Llama 3-8B on one 24 GB GPU and Llama 3-70B on two 80 GB GPUs by streaming gradients over dynamically activated sub-tensors.
-
STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training
STQuant dynamically allocates quantization bits for optimizer states in multimodal model training, reducing memory by 84.4% to an average 5.1 bits while preserving quality on GPT-2 and ViT.
-
Optimal Projection-Free Adaptive SGD for Matrix Optimization
Proving stability of Leon's preconditioner enables the first tuning-free Nesterov-accelerated projection-free adaptive SGD variant with improved non-smooth non-convex rates.
-
Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
SCALE matches Adam performance in LLM pretraining from 60M to 7B parameters by combining column-wise gradient normalization with last-layer-only momentum, using 35-45% of Adam's memory.
-
On the Convergence Analysis of Muon
Muon's convergence rate depends on an average Hessian curvature along its update directions, which can be much smaller than the worst-case Lipschitz constant when Hessians are low-rank.
-
GWT: Scalable Optimizer State Compression for Large Language Model Training
GWT projects gradients into wavelet subspaces to compress optimizer states for memory-efficient LLM training while claiming performance parity with full-rank updates.
-
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
LLR uses heavy-tailed self-regularization theory to set per-layer learning rates in Transformers, yielding faster convergence and higher zero-shot accuracy than uniform rates across model scales.
-
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
AdaMeZO adapts Adam moment estimates to zeroth-order LLM fine-tuning without extra memory storage, outperforming MeZO with up to 70% fewer forward passes.
-
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
HTMuon modifies Muon to produce heavier-tailed updates and weight spectra via HT-SR theory, yielding up to 0.98 lower perplexity on LLaMA pretraining and serving as a plug-in for other Muon variants.
-
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Even with perfectly balanced and whitened data, a deep linear network's Hessian exhibits a two-cluster spectrum whose dominant-to-bulk eigenvalue ratio grows linearly with depth.
-
SpectralTrain: A Universal Framework for Hyperspectral Image Classification
SpectralTrain is a universal training framework that combines curriculum learning and PCA spectral downsampling to deliver 2-7x faster training for hyperspectral image classification across multiple backbones and datasets with only small accuracy trade-offs.