The Hessian bulk consists of weakly broken continuous symmetries of the network parametrization, exact zeros in linear nets that ReLU lifts as pseudo-Goldstone modes whose eigenvectors stay in the symmetry subspace.
hub
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
21 Pith papers cite this work, alongside 121 external citations. Polarity classification is still indexing.
21
Pith papers citing it
121
external citations · Pith
abstract
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.
hub tools
citation-role summary
background 3
citation-polarity summary
roles
background 3polarities
background 3representative citing papers
In conjugate BLR, MFVI overestimates expected predictive variance on in-distribution points relative to the exact posterior, with overestimation aligned to training data directions.
Depth induces an implicit low-rank bias in deep unconstrained feature models trained with unregularized multiclass cross-entropy, promoting softmax codes over neural collapse via more efficient norm propagation.
Backdoors can be embedded in ResNet and ViT models as statistically indistinguishable latent directions, reducing cryptographic undetectability to an intractable hypothesis test over parameter distributions.
Permutation symmetries generate permutation saddles and equal-loss valleys linking equivalent global minima, yielding a lower bound on symmetry-induced critical points.
Derives explicit step-size conditions ensuring the metastability behavior of discrete SGD under heavy-tailed noise approximates its continuous SDE limit.
Hessian eigenvector displacement and inverse participation ratio metrics show SGD stabilizing leading curvature directions while Adam causes more reorganization and parameter localization in MLP training.
Derives closed-form gradient of WS upper bound on Hessian max eigenvalue for 3-layer cross-entropy NNs and proposes HSR regularization to steer toward flat minima.
Derives Weyl-based perturbation bounds showing quantization increases the dominant eigenvalue of the empirical FIM up to higher-order terms, with supporting measurements on language models.
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
GIST recovers a task-specific low-dimensional subspace from validation gradients using SVD and scores training examples by their alignment within this coupled subspace for LoRA-based instruction tuning.
Muon's convergence rate depends on an average Hessian curvature along its update directions, which can be much smaller than the worst-case Lipschitz constant when Hessians are low-rank.
Derives optimal low-rank subspace for Laplace approx in BNNs, provides scalable outperforming version, and new comparison metric.
RMNP preconditions matrix updates via row-wise L2 normalization instead of Newton-Schulz iteration, reducing complexity to O(mn) while matching Muon's non-convex convergence rate and empirical performance.
AdaMeZO adapts Adam moment estimates to zeroth-order LLM fine-tuning without extra memory storage, outperforming MeZO with up to 70% fewer forward passes.
A closed-form upper bound on the maximum Hessian eigenvalue of cross-entropy loss is derived for smooth nonlinear neural networks.
FOAM adaptively controls damping and update frequency in Shampoo based on staleness-oriented error approximation to cut wall-clock time while preserving convergence.
Provides Hessian-based theoretical characterizations of SGD dynamics and a scale-invariant generalization bound for deep nets, backed by experiments on synthetic data, MNIST, and CIFAR-10.
A grid-free attention-based deep learning model interpolates surface-level PM2.5 concentrations across the US using sparse sensor data and auxiliary geospatial features.
citing papers explorer
-
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
The Hessian bulk consists of weakly broken continuous symmetries of the network parametrization, exact zeros in linear nets that ReLU lifts as pseudo-Goldstone modes whose eigenvectors stay in the symmetry subspace.
-
Gaussian Mean Field Variational Inference can Overestimate Predictive Variance
In conjugate BLR, MFVI overestimates expected predictive variance on in-distribution points relative to the exact posterior, with overestimation aligned to training data directions.
-
The Implicit Bias of Depth: From Neural Collapse to Softmax Codes
Depth induces an implicit low-rank bias in deep unconstrained feature models trained with unregularized multiclass cross-entropy, promoting softmax codes over neural collapse via more efficient norm propagation.
-
Backdoor Channels Hidden in Latent Space: Cryptographic Undetectability in Modern Neural Networks
Backdoors can be embedded in ResNet and ViT models as statistically indistinguishable latent directions, reducing cryptographic undetectability to an intractable hypothesis test over parameter distributions.
-
Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
Permutation symmetries generate permutation saddles and equal-loss valleys linking equivalent global minima, yielding a lower bound on symmetry-induced critical points.
-
First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
Derives explicit step-size conditions ensuring the metastability behavior of discrete SGD under heavy-tailed noise approximates its continuous SDE limit.
-
Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization
Hessian eigenvector displacement and inverse participation ratio metrics show SGD stabilizing leading curvature directions while Adam causes more reorganization and parameter localization in MLP training.
-
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Derives closed-form gradient of WS upper bound on Hessian max eigenvalue for 3-layer cross-entropy NNs and proposes HSR regularization to steer toward flat minima.
-
Spectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantization
Derives Weyl-based perturbation bounds showing quantization increases the dominant eigenvalue of the empirical FIM up to higher-order terms, with supporting measurements on language models.
-
Comparing Classical Simulation and Sample-Based Learning of Quantum Systems
For random MPS and Clifford+T circuits, increases in entanglement or T-count correlate with sharper loss minima and worse reconstruction under constrained neural capacity.
-
Worker Disagreement Reveals Sharp Directions in Local SGD
Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.
-
AMUSE: Anytime Muon with Stable Gradient Evaluation
AMUSE stabilizes Muon with time-varying schedule-free gradient evaluation, improving the performance-iteration Pareto frontier without learning-rate schedules.
-
GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
GIST recovers a task-specific low-dimensional subspace from validation gradients using SVD and scores training examples by their alignment within this coupled subspace for LoRA-based instruction tuning.
-
On the Convergence Analysis of Muon
Muon's convergence rate depends on an average Hessian curvature along its update directions, which can be much smaller than the worst-case Lipschitz constant when Hessians are low-rank.
-
Low Rank Based Subspace Inference for the Laplace Approximation of Bayesian Neural Networks
Derives optimal low-rank subspace for Laplace approx in BNNs, provides scalable outperforming version, and new comparison metric.
-
RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
RMNP preconditions matrix updates via row-wise L2 normalization instead of Newton-Schulz iteration, reducing complexity to O(mn) while matching Muon's non-convex convergence rate and empirical performance.
-
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
AdaMeZO adapts Adam moment estimates to zeroth-order LLM fine-tuning without extra memory storage, outperforming MeZO with up to 70% fewer forward passes.
-
Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networks
A closed-form upper bound on the maximum Hessian eigenvalue of cross-entropy loss is derived for smooth nonlinear neural networks.
-
FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
FOAM adaptively controls damping and update frequency in Shampoo based on staleness-oriented error approximation to cut wall-clock time while preserving convergence.
-
Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
Provides Hessian-based theoretical characterizations of SGD dynamics and a scale-invariant generalization bound for deep nets, backed by experiments on synthetic data, MNIST, and CIFAR-10.
-
Mechanistic Anomaly Detection via Functional Attribution
A grid-free attention-based deep learning model interpolates surface-level PM2.5 concentrations across the US using sparse sensor data and auxiliary geospatial features.