SignSGD provably beats SGD by a factor of d under sparse noise via matched ℓ1-norm upper and lower bounds, with an equivalent result for Muon on matrices, and this predicts faster GPT-2 pretraining.
hub Canonical reference
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
Canonical reference. 100% of citing Pith papers cite this work as background.
40
Pith papers citing it
Background
100% of classified citations
abstract
We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light into the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.
hub tools
citation-role summary
background 5
citation-polarity summary
fields
cs.LG 30 math.OC 2 quant-ph 2 cond-mat.dis-nn 1 cond-mat.stat-mech 1 cs.CR 1 cs.CV 1 cs.IT 1 stat.ML 1roles
background 5polarities
background 5representative citing papers
The Hessian bulk consists of weakly broken continuous symmetries of the network parametrization, exact zeros in linear nets that ReLU lifts as pseudo-Goldstone modes whose eigenvectors stay in the symmetry subspace.
PCD is a new gradient-based optimizer for hierarchical multi-objective problems that prioritizes primary descent with minimal controlled distortion for secondary objectives via a single tau parameter.
Dead-Direction Signatures provide closed-form spectral readings of dead directions in network activations and gradients that track rank deficits at singular minima, offering a cheap directional alternative to SGLD-based LLC.
The normalized inverse-scale direction of LayerNorm's affine parameters is an exact algebraic kernel of the post-final-norm centred activation covariance for any input distribution in LayerNorm transformers.
Introduces thermodynamic free-energy signatures and spectral form factors from attention Laplacians for hallucination detection, with stability proofs, expressiveness results, a PAC bound, and empirical AUROC gains over baselines.
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
Non-normality in linearized optimizer update operators yields a pseudospectral bound where κ(V) warns of transient amplification before spectral radius indicates instability.
Depth induces an implicit low-rank bias in deep unconstrained feature models trained with unregularized multiclass cross-entropy, promoting softmax codes over neural collapse via more efficient norm propagation.
Backdoors can be embedded in ResNet and ViT models as statistically indistinguishable latent directions, reducing cryptographic undetectability to an intractable hypothesis test over parameter distributions.
Hessian Surgery perturbs trained model weights along Hessian spike eigenvectors via a sensitivity matrix and constrained optimization to rebalance per-class accuracy on CIFAR-10 and ISIC-2019 without retraining.
Permutation symmetries generate permutation saddles and equal-loss valleys linking equivalent global minima, yielding a lower bound on symmetry-induced critical points.
Optimistic bilevel optimization with manifold lower-level minimizers is differentiable if the optimistic selection is unique, yielding a pseudoinverse hyper-gradient and a convergent HG-MS algorithm whose rate depends on intrinsic manifold dimension.
FGN is a positive semidefinite under-approximation of the multiclass GGN obtained by exact decomposition into true-vs-rest and within-competitor terms, exact for binary classification and implemented via matrix-free conjugate gradient on a whitened row-space system.
The spectral edge transitions from a gradient-driven functional direction before grokking to a perturbation-flat, ablation-critical compression axis at grokking, forming three universality classes predicted by a gap flow equation.
Elitist (1+M) genetic algorithms follow the loss gradient via mutation-selection, slowed only by noise in the effective-rank directions of the Hessian rather than the full parameter count.
Presents a stochastic gradient algorithm for non-separable optimization with local convergence guarantees under smoothness assumptions.
Derives second-order path-kernel interpolation formulas for gradient descent, SGD, and momentum training, adding curvature terms and a concentration estimate around the expected prediction.
A geometric classification of stationary points on neuron-splitting plateaus in two-layer NN loss landscapes using the inner Hessian.
Momentum in Muon functions as a spectral filter on signal-plus-perturbation gradients, enlarging the gap to stabilize singular subspaces before orthogonalization and outperforming the reverse order.
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.
Stochastic layer-wise Hessian trace estimator using Hutchinson method and Hessian-vector products detects label memorization in CNNs with high empirical power.
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
citing papers explorer
-
When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds
SignSGD provably beats SGD by a factor of d under sparse noise via matched ℓ1-norm upper and lower bounds, with an equivalent result for Muon on matrices, and this predicts faster GPT-2 pretraining.
-
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
The Hessian bulk consists of weakly broken continuous symmetries of the network parametrization, exact zeros in linear nets that ReLU lifts as pseudo-Goldstone modes whose eigenvectors stay in the symmetry subspace.
-
Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization
PCD is a new gradient-based optimizer for hierarchical multi-objective problems that prioritizes primary descent with minimal controlled distortion for secondary objectives via a single tau parameter.
-
Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
Dead-Direction Signatures provide closed-form spectral readings of dead directions in network activations and gradients that track rank deficits at singular minima, offering a cheap directional alternative to SGLD-based LLC.
-
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
The normalized inverse-scale direction of LayerNorm's affine parameters is an exact algebraic kernel of the post-final-norm centred activation covariance for any input distribution in LayerNorm transformers.
-
Thermodynamic Signatures of Reasoning: Free-Energy and Spectral-Form-Factor Diagnostics for Hallucination Detection in Large Language Models
Introduces thermodynamic free-energy signatures and spectral form factors from attention Laplacians for hallucination detection, with stability proofs, expressiveness results, a PAC bound, and empirical AUROC gains over baselines.
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
Non-normal spectral signatures of instability in neural network training dynamics
Non-normality in linearized optimizer update operators yields a pseudospectral bound where κ(V) warns of transient amplification before spectral radius indicates instability.
-
The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
Depth induces an implicit low-rank bias in deep unconstrained feature models trained with unregularized multiclass cross-entropy, promoting softmax codes over neural collapse via more efficient norm propagation.
-
Backdoor Channels Hidden in Latent Space: Cryptographic Undetectability in Modern Neural Networks
Backdoors can be embedded in ResNet and ViT models as statistically indistinguishable latent directions, reducing cryptographic undetectability to an intractable hypothesis test over parameter distributions.
-
Hessian Surgery: Class-Targeted Post-Hoc Rebalancing via Hessian Spike Perturbation
Hessian Surgery perturbs trained model weights along Hessian spike eigenvectors via a sensitivity matrix and constrained optimization to rebalance per-class accuracy on CIFAR-10 and ISIC-2019 without retraining.
-
Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
Permutation symmetries generate permutation saddles and equal-loss valleys linking equivalent global minima, yielding a lower bound on symmetry-induced critical points.
-
Select-then-differentiate: Solving Bilevel Optimization with Manifold Lower-level Solution Sets
Optimistic bilevel optimization with manifold lower-level minimizers is differentiable if the optimistic selection is unique, yielding a pseudoinverse hyper-gradient and a convergent HG-MS algorithm whose rate depends on intrinsic manifold dimension.
-
Fast Gauss-Newton for Multiclass Cross-Entropy
FGN is a positive semidefinite under-approximation of the multiclass GGN obtained by exact decomposition into true-vs-rest and within-competitor terms, exact for binary classification and implemented via matrix-free conjugate gradient on a whitened row-space system.
-
The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression
The spectral edge transitions from a gradient-driven functional direction before grokking to a perturbation-flat, ablation-critical compression axis at grokking, forming three universality classes predicted by a gap flow equation.
-
Why can genetic algorithms work in high-dimensional search spaces?
Elitist (1+M) genetic algorithms follow the loss gradient via mutation-selection, slowed only by noise in the effective-rank directions of the Hessian rather than the full parameter count.
-
A stochastic gradient algorithm for non-separable optimization with convergence guarantee
Presents a stochastic gradient algorithm for non-separable optimization with local convergence guarantees under smoothness assumptions.
-
Second-Order Path Kernel Interpolation Formulas in Machine Learning
Derives second-order path-kernel interpolation formulas for gradient descent, SGD, and momentum training, adding curvature terms and a concentration estimate around the expected prediction.
-
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
A geometric classification of stationary points on neuron-splitting plateaus in two-layer NN loss landscapes using the inner Hessian.
-
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Momentum in Muon functions as a spectral filter on signal-plus-perturbation gradients, enlarging the gap to stabilize singular subspaces before orthogonalization and outperforming the reverse order.
-
Comparing Classical Simulation and Sample-Based Learning of Quantum Systems
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
-
Worker Disagreement Reveals Sharp Directions in Local SGD
Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.
-
Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
Stochastic layer-wise Hessian trace estimator using Hutchinson method and Hessian-vector products detects label memorization in CNNs with high empirical power.
-
AMUSE: Anytime Muon with Stable Gradient Evaluation
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
-
Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training
Spectral gaps in the Gram matrix of parameter updates control phase transitions such as grokking in neural network training.
-
CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing
CrispEdit edits LLMs via low-curvature projections using Bregman divergence and K-FAC approximations, achieving high edit success with under 1% average capability degradation.
-
On the Convergence Analysis of Muon
Muon's convergence rate depends on an average Hessian curvature along its update directions, which can be much smaller than the worst-case Lipschitz constant when Hessians are low-rank.
-
Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features
All rank-monotone pruning scorers converge to identical accuracy at fixed sparsity, but non-monotone features with sparsity-dependent complexity can escape this plateau, as shown by the SICS hypothesis on ViT-Small/CIFAR-10.
-
Quantum Tilted Loss in Variational Optimization: Theory and Applications
QTL unifies expectation-value minimization with CVaR and Gibbs heuristics under one tunable operator, amplifying gradients in structured cases while preserving global minima and shifting the bottleneck to measurement variance.
-
Generalization at the Edge of Stability
Training at the edge of stability causes neural network optimizers to converge on fractal attractors whose effective dimension, measured via a new sharpness dimension from the Hessian spectrum, bounds generalization error in a way not captured by prior trace or norm measures.
-
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
-
Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networks
In overparameterized quadratic networks, one-pass SGD escapes generalization plateaus only modestly faster and selects the initialization-closest zero-loss solution due to a conserved quantity in the overlap ODEs.
-
Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
A hierarchical taxonomy and comparative review of continual test-time adaptation methods that adapt vision models online to non-stationary unlabeled streams while fighting forgetting and error accumulation.
-
Scalar Representations of Neural Network Training Dynamics
Scalar embeddings of neural network training trajectories treated as temporal networks preserve main dynamical features including Lyapunov exponents, enable definition of a characteristic decorrelation time, and show asymptotic state spacings compatible with a skew lognormal distribution.
-
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
SoftSignum replaces hard sign with soft-sign in optimizers via temperature control and quantile scheduling, extends to SoftMuon, provides a convergence proof for stochastic non-convex settings, and reports better performance than sign-based methods and AdamW on deep learning tasks.
-
Anytime Training with Schedule-Free Spectral Optimization
SF-NorMuon is a new schedule-free spectral optimizer that closes the gap with tuned AdamW on 125M-772M parameter models across 1-8x Chinchilla horizons while providing stationarity guarantees.
-
RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
RMNP preconditions matrix updates via row-wise L2 normalization instead of Newton-Schulz iteration, reducing complexity to O(mn) while matching Muon's non-convex convergence rate and empirical performance.
-
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
Even with perfectly balanced and whitened data, a deep linear network's Hessian exhibits a two-cluster spectrum whose dominant-to-bulk eigenvalue ratio grows linearly with depth.
-
Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
GNC convolves stochastic gradient noise to smooth sharp minima in large-batch SGD, outperforming isotropic noise for better generalization in distributed deep learning.
-
Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
Provides Hessian-based theoretical characterizations of SGD dynamics and a scale-invariant generalization bound for deep nets, backed by experiments on synthetic data, MNIST, and CIFAR-10.