Empirical power-law scaling governs language model loss versus model size, data size, and compute, enabling optimal allocation of training compute.
hub
Gradient Descent Happens in a Tiny Subspace
34 Pith papers cite this work. Polarity classification is still indexing.
34
Pith papers citing it
abstract
We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.
hub tools
citation-role summary
background 2
method 1
citation-polarity summary
representative citing papers
The Hessian bulk consists of weakly broken continuous symmetries of the network parametrization, exact zeros in linear nets that ReLU lifts as pseudo-Goldstone modes whose eigenvectors stay in the symmetry subspace.
The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.
Evaluation of two latent reasoning models against controls shows observable latent patterns appear without the proposed mechanisms, have graded causal effects on behavior, and concentrate in structured low-rank directions, arguing that patterns are insufficient evidence for reasoning.
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
Proposes Architecture-driven Shift (ADS) as an architecture-based proxy for logit shift in continual learning, derived from spectral norm scaling, optimization path length and task conflict, with monotonic correlation rs >= 0.731 across 175 architectures and utility as expected calibration error pro
GaLore performs full-parameter LLM training with up to 65.5% less optimizer memory by projecting gradients onto a low-rank subspace at each step, matching full-rank performance on LLaMA pre-training and RoBERTa fine-tuning.
The spectral edge transitions from a gradient-driven functional direction before grokking to a perturbation-flat, ablation-critical compression axis at grokking, forming three universality classes predicted by a gap flow equation.
Hessian eigenvector displacement and inverse participation ratio metrics show SGD stabilizing leading curvature directions while Adam causes more reorganization and parameter localization in MLP training.
OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to near-optimal deterministic first-order rates in the zero-noise regime.
Momentum in Muon functions as a spectral filter on signal-plus-perturbation gradients, enlarging the gap to stabilize singular subspaces before orthogonalization and outperforming the reverse order.
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
Spectral gaps in the Gram matrix of parameter updates control phase transitions such as grokking in neural network training.
AGZO restricts ZO perturbations to an activation-derived low-rank subspace, claiming higher gradient cosine similarity and better benchmark performance than isotropic ZO baselines on Qwen3 and Pangu models.
Effective data transferred from pre-training to fine-tuning is described by a power law in model parameter count and fine-tuning dataset size, acting like a multiplier on the fine-tuning data.
Dr. Post-Training reframes general data as a data-induced regularizer for LLM post-training updates, yielding a family of methods that outperform data-selection baselines on SFT, RLHF, and RLVR tasks.
Pro-KLShampoo projects KL-Shampoo preconditioners to a spike-and-flat parametric form on an r-dimensional subspace and recovers the full algebraic preconditioner via orthogonalization, outperforming KL-Shampoo on GPT-2 and LLaMA pre-training scales.
Spectral analysis of activations and gradients provides new diagnostics that link batch size to representation geometry, early covariance tails to token efficiency, and spectral shifts to learning dynamics in decoder-only LLMs, backed by a mechanistic model.
DBLP dynamically bounds gradient loss by training phase, cutting end-to-end training time ~24% on average and up to 5.88× single-round latency under microbursts versus a SOTA baseline.
TLoRA jointly optimizes LoRA initialization via task-data SVD and sensitivity-driven rank allocation, delivering stronger results than standard LoRA across NLU, reasoning, math, code, and chat tasks while using fewer trainable parameters.
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
Grokking occurs as the effective dimensionality of the gradient field transitions from sub-diffusive to super-diffusive at the onset of generalization, exhibiting self-organized criticality.
citing papers explorer
-
Scaling Laws for Neural Language Models
Empirical power-law scaling governs language model loss versus model size, data size, and compute, enabling optimal allocation of training compute.
-
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
The Hessian bulk consists of weakly broken continuous symmetries of the network parametrization, exact zeros in linear nets that ReLU lifts as pseudo-Goldstone modes whose eigenvectors stay in the symmetry subspace.
-
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.
-
Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models
Evaluation of two latent reasoning models against controls shows observable latent patterns appear without the proposed mechanisms, have graded causal effects on behavior, and concentrate in structured low-rank directions, arguing that patterns are insufficient evidence for reasoning.
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift
Proposes Architecture-driven Shift (ADS) as an architecture-based proxy for logit shift in continual learning, derived from spectral norm scaling, optimization path length and task conflict, with monotonic correlation rs >= 0.731 across 175 architectures and utility as expected calibration error pro
-
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
GaLore performs full-parameter LLM training with up to 65.5% less optimizer memory by projecting gradients onto a low-rank subspace at each step, matching full-rank performance on LLaMA pre-training and RoBERTa fine-tuning.
-
The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression
The spectral edge transitions from a gradient-driven functional direction before grokking to a perturbation-flat, ablation-critical compression axis at grokking, forming three universality classes predicted by a gap flow equation.
-
Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization
Hessian eigenvector displacement and inverse participation ratio metrics show SGD stabilizing leading curvature directions while Adam causes more reorganization and parameter localization in MLP training.
-
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to near-optimal deterministic first-order rates in the zero-noise regime.
-
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Momentum in Muon functions as a spectral filter on signal-plus-perturbation gradients, enlarging the gap to stabilize singular subspaces before orthogonalization and outperforming the reverse order.
-
Comparing Classical Simulation and Sample-Based Learning of Quantum Systems
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
-
Worker Disagreement Reveals Sharp Directions in Local SGD
Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.
-
AMUSE: Anytime Muon with Stable Gradient Evaluation
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
-
Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training
Spectral gaps in the Gram matrix of parameter updates control phase transitions such as grokking in neural network training.
-
AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning
AGZO restricts ZO perturbations to an activation-derived low-rank subspace, claiming higher gradient cosine similarity and better benchmark performance than isotropic ZO baselines on Qwen3 and Pangu models.
-
Scaling Laws for Transfer
Effective data transferred from pre-training to fine-tuning is described by a power law in model parameter count and fine-tuning dataset size, acting like a multiplier on the fine-tuning data.
-
Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training
Dr. Post-Training reframes general data as a data-induced regularizer for LLM post-training updates, yielding a family of methods that outperform data-selection baselines on SFT, RLHF, and RLVR tasks.
-
Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
Pro-KLShampoo projects KL-Shampoo preconditioners to a spike-and-flat parametric form on an r-dimensional subspace and recovers the full algebraic preconditioner via orthogonalization, outperforming KL-Shampoo on GPT-2 and LLaMA pre-training scales.
-
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
Spectral analysis of activations and gradients provides new diagnostics that link batch size to representation geometry, early covariance tails to token efficiency, and spectral shifts to learning dynamics in decoder-only LLMs, backed by a mechanistic model.
-
DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training
DBLP dynamically bounds gradient loss by training phase, cutting end-to-end training time ~24% on average and up to 5.88× single-round latency under microbursts versus a SOTA baseline.
-
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
TLoRA jointly optimizes LoRA initialization via task-data SVD and sensitivity-driven rank allocation, delivering stronger results than standard LoRA across NLU, reasoning, math, code, and chat tasks while using fewer trainable parameters.
-
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
-
Grokking as Dimensional Phase Transition in Neural Networks
Grokking occurs as the effective dimensionality of the gradient field transitions from sub-diffusive to super-diffusive at the onset of generalization, exhibiting self-organized criticality.
-
Language Models (Mostly) Know What They Know
Language models show good calibration when asked to estimate the probability that their own answers are correct, with performance improving as models get larger.
-
A General Language Assistant as a Laboratory for Alignment
Ranked preference modeling outperforms imitation learning for language model alignment and scales more favorably with model size.
-
Scalar Representations of Neural Network Training Dynamics
Scalar embeddings of neural network training trajectories treated as temporal networks preserve main dynamical features including Lyapunov exponents, enable definition of a characteristic decorrelation time, and show asymptotic state spacings compatible with a skew lognormal distribution.
-
On the Oracle Complexity of Interpolation-Based Gradient Descent
PPI-GD approximates gradients via d-variate tensor product polynomial interpolants on data-domain samples to achieve better oracle complexity than standard GD variants for sufficiently smooth strongly convex and non-convex ERM losses under polylog data dimension.
-
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
MGUP augments momentum optimizers with selective larger steps on a fixed proportion of parameters per iteration, claiming convergence guarantees for MGUP-AdamW and superior empirical performance on pretraining and fine-tuning.
-
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
Different Bregman divergences distribute unavoidable Kronecker approximation error differently across covariance spectrum, with top eigenspace better aligned to Hessian, motivating a subspace-aware optimizer using eigenvalue preconditioning on top subspace and adaptive isotropic acceleration on bott
-
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Even with perfectly balanced and whitened data, a deep linear network's Hessian exhibits a two-cluster spectrum whose dominant-to-bulk eigenvalue ratio grows linearly with depth.
-
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
Step-Video-T2V describes a 30B-parameter text-to-video model with custom Video-VAE, 3D DiT, flow matching, and Video-DPO that claims state-of-the-art results on a new internal benchmark.
-
Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
Provides Hessian-based theoretical characterizations of SGD dynamics and a scale-invariant generalization bound for deep nets, backed by experiments on synthetic data, MNIST, and CIFAR-10.
- On the Convergence Analysis of Muon