REVIEW 4 major objections 4 minor 21 references
The effects of Hessian eigenvalue spectral density type on the applicability of Hessian analysis to generalization capability assessment of neural networks
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A neural network's Hessian spectrum says nothing about generalization when it is mainly negative, because such spectra arise from external gradient manipulation rather than from the loss landscape.
desk verdict A useful empirical caution about Hessian spectral density types, but the central claim that MN-HESD comes from gradient manipulation is untested and the exclusion rule is not yet justified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Hessian eigenvalue spectral density (HESD), the distribution of eigenvalues of the loss Hessian at a trained model. The paper's central identity is the type criterion $$C_t = \frac{\min(\lambda_{\text{neg}})}{\max(\lambda_{\text{pos}})},$$ computed from the most negative and most positive Hessian eigenvalues, with MP-HESD defined by $C_t > -0.6$ and MN-HESD by $C_t \le -0.6$. This number carries the argument: it is the gate that decides whether the previously proposed generalization criteria, stated as $\Delta r_e < 1.5$ and $\Delta K_{H05} < 1.2$, are applicable, and it doubles as a checkpoint-selection score when training accuracies are tied. The machinery also includes the notion of quasi-singular HESD, in which all eigenvalues approach zero after excessive training, which the paper uses to show that raw eigenvalue-scale measures can degenerate while ratio-based criteria remain informative.
What would settle it
Train the same VIT small on cinic under identical settings twice, once with global-norm gradient clipping at norm 1 and once without any clipping, and compute the HESD at every epoch. If the no-clipping control ever exhibits MN-HESD, then external gradient manipulation is not the sole cause and the exclusion rule rests on the wrong premise; if MN-HESD appears only in the clipped run, the paper's core claim is supported.
Extended reading notes
Core claim
The central claim is that MN-HESD is a consequence of external gradient manipulation, not of poor training or bad generalization, and therefore the previously proposed Hessian generalization criteria are valid only for MP-HESD. To establish this, the paper trains VIT small and ResNet20 and fine-tunes VIT on cinic and ImageNet under combinations of AdamW, SGD with and without momentum, RandAugment, and frozen versus unfrozen weights. All such runs produce MP-HESD. In contrast, VIT small trained with AdaHessian flips between MP-HESD and MN-HESD at neighboring epochs while accuracy stays essentially constant, and the paper attributes this to AdaHessian's spatial averaging over the hidden dimension. It then conjectures, without direct verification, that the MN-HESD reported in [10] came from global-norm gradient clipping. The proposed safeguard is the HESD type criterion $C_t > -0.6$: only spectra satisfying it enter the unified Hessian analysis, while MN-HESD runs are flagged as cases where the methodology is unreliable.
Load-bearing premise
The load-bearing premise is that MN-HESD arises specifically from external manipulation of the gradients used in Hessian computation, a conclusion inferred from AdaHessian experiments and an unverified assumption about gradient clipping in the network of [10]; if MN-HESD can also arise without such manipulation, the blanket exclusion of MN-HESD networks is unjustified.
Editorial extensions
If this is right
- Any deployment of Hessian-based generalization assessment must start with the $C_t > -0.6$ type check and refuse MN-HESD networks, since their spectra reflect optimizer or preprocessing operations rather than the loss landscape.
- Standard training recipes, including SGD with or without momentum, AdamW, fine-tuning with frozen or unfrozen weights, RandAugment, and different datasets, produce MP-HESD, so the criteria remain applicable across a wide practical range.
- MN-HESD should not be read as evidence of poor generalization: the AdaHessian VIT kept 99% training and 49% generalization accuracy while its $C_t$ dipped below threshold at alternating epochs.
- Excessive training can drive Hessian eigenvalues toward zero, producing quasi-singular HESD, but the ratio-based criteria $r_e$ and $K_{H05}$ keep their train/test separation, so they stay informative even when the raw eigenvalue scale degenerates.
- Among checkpoints with comparable training accuracy, choosing the one with the largest $C_t$ can outperform choosing the one with the smallest maximum eigenvalue, finding better generalization at an earlier epoch in the paper's experiments.
Reading between the lines
- If the gradient-artifact explanation is right, then any Hessian-generalization study that includes clipped or preconditioned training runs without a spectrum-type check should be re-examined, and future empirical work should report whether gradients were altered.
- The $C_t$ ratio is scale-free and could serve as a checkpoint-selection score in settings where Hessian trace or top eigenvalue are inflated by magnitude, such as comparing models across precisions, batch sizes, or training lengths.
- A direct ablation with global-norm clipping at several thresholds, not just norm 1, would test whether the HESD distortion scales with clipping strength and would turn the paper's conjecture about [10] into a measured claim.
- The quasi-singular HESD finding suggests that absolute eigenvalue flatness measures are not robust to epoch count; a normalized curvature ratio like $C_t$ might be a more stable companion in loss-landscape visualization and model selection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies how Hessian eigenvalue spectral density (HESD) types affect the applicability of the author's earlier Hessian-based generalization criteria [5]. It reports experiments across optimizers (SGD, AdamW, AdaHessian), architectures (ResNet, ViT), and datasets (cinic, CIFAR-10, ImageNet) showing that 'mainly positive' (MP) HESD is typical, and that 'mainly negative' (MN) HESD can arise when gradients are manipulated externally (AdaHessian, and, speculatively, gradient clipping). It proposes criterion Ct = min(λneg)/max(λpos) with threshold -0.6 to classify HESD type, adds numerical conditions Δre<1.5 and ΔKH05<1.2 for good generalization, and combines them into a unified assessment algorithm. It also documents quasi-singular HESD under excessive training and argues that Ct is more robust than the maximum Hessian eigenvalue.
Significance. If the MN-HESD mechanism claim were fully supported, the paper would provide a useful practical safeguard for Hessian-based generalization assessment. The paper is honest in labeling the [10] explanation as unverified, and the proposed conditions are explicit, concrete, and testable. However, the central exclusion rule rests on a single confounded experiment and an admitted speculation about the mechanism, so the significance is currently conditional. The work is a sensible incremental extension of the author's prior [5] methodology, but it does not yet stand as a self-contained validation of the unified approach.
major comments (4)
- [Section 3.4] The core claim that MN-HESD is caused by external gradient manipulation is not established. The AdaHessian experiment compares ResNet20 (which yields MP-HESD) with ViT-small (which yields MN-HESD), differing simultaneously in architecture and optimizer; no ablation of AdaHessian's spatial averaging is performed, so the attribution to 'spatial averaging over the hidden dimension' is confounded. The extension to the gradient-clipping ViT of [10] is explicitly marked as 'cannot be verified exactly,' and no gradient-clipping experiment is run anywhere in the paper. Because the HESD type check and the exclusion rule for MN-HESD networks are load-bearing components of the proposed unified methodology, a controlled experiment is required: for example, the same architecture trained with and without gradient clipping, or with AdaHessian's gradient transform enabled versus disabled, with repeated runs.
- [Section 3.4] The proposed causal story is conceptually incomplete. The Hessian is a property of the loss landscape at a checkpoint and is not directly altered by the optimizer's gradient transformation unless that transformation changes the trajectory of training. The statement that 'HESD evaluation algorithm does not account for optimizer operations' suggests a conflation between Hessian computation and gradient-based quantities. Please clarify whether MN-HESD is claimed to arise from a genuinely different parameter trajectory (which would be a statement about where training converges) or from a bias or error in the Hessian estimation procedure when gradients are manipulated. These are distinct mechanisms and have different implications for the validity of the exclusion rule.
- [Section 4.3, Eqs. (3) and (4)] The numerical conditions Δre < 1.5 and ΔKH05 < 1.2 are introduced as part of the unified methodology but are not derived or independently validated. The manuscript provides no evidence that these thresholds transfer across architectures, datasets, or optimizers. Since these conditions form a contribution of the paper rather than a restatement of [5], the author should either provide a derivation from the known behavior of re and KH05 under HESD changes, or validate the thresholds on a separate set of models that were not used to set them.
- [Section 2.2, Eq. (1)] Equation (1) defines Ct as essentially -re from [5], so the proposed HESD type criterion is not independent of the original generalization criterion. The threshold -0.6 is calibrated from the maximum observed Ct values in [5] (with a safety margin), and the updated conditions in Section 4.3 are also stated without independent derivation. This circularity should be acknowledged explicitly, and the reliability of the thresholds should be demonstrated on data or settings not used to select them, otherwise the claim of a 'unified HESD analysis methodology' is stronger than the evidence supports.
minor comments (4)
- [Section 4.1] The text refers to 'Figures 8 (b) and (c)', but Figure 8 has only panels (a) and (b); please renumber or correct the cross-reference.
- [Section 5] The final sentence says the method 'completes the NN generalization assessment method for MN-HESD NNs,' but the methodology excludes MN-HESD networks; this appears to be a typo for 'MP-HESD NNs' and should be corrected.
- [Tables 1-4] Some model names are inconsistent (e.g., 'VIT' vs. 'VIT small' vs. 'VIT Small'); please standardize capitalization and acronym usage.
- [Experimental protocol] No repeated runs, error bars, or random-seed details are reported. Please state the number of seeds used or, if each experiment was run once, say so explicitly, since the claim that HESD type is stable across settings would be stronger with such information.
Circularity Check
Ct criterion is a renamed re from the author's prior work, and the threshold -0.6 is fitted from the same prior data, yielding partial circularity in the unified methodology.
-
renaming known result
[Section 2.2, Eq. (1) and following text]
"Assuming the existence of negative eigenvalues, this criterion is identical (up to a sign) to generalization criterion re proposed in [5]."
The paper introduces Ct as a new 'HESD type criterion' but immediately states it is identical (up to sign) to re from the author's prior paper [5]. Later, the unified methodology uses Ct > -0.6 as a gate before applying re-based criteria (3)-(4). Since Ct and re are the same ratio by construction, the type check supplies no new information; the claim that the methodology is 'complete' because it adds this check is a renaming of the existing criterion, not an independent derivation.
-
fitted input called prediction
[Section 2.2, Eq. (2) and preceding paragraph]
"However, it has been shown in [5] that it can actually be higher for some NNs (especially when BatchNorm and Dropout layers are present), reaching as high as -0.73 for ResNet18 in evaluation mode. Therefore, it is proposed to have a small safety margin and use -0.6 as the threshold value, leading to the following condition for MP-HESD of trained NNs Ct > −0.6 (2)"
The threshold -0.6 is chosen based on the range of Ct values observed in [5] for MP-HESD networks (highest -0.73), plus a 'safety margin.' This is a fit to the author's own prior data. The paper then uses this threshold to classify every experiment in the present study as MP or MN and to conclude that MP-HESD is typical. Because the threshold was calibrated on the same type of HESD measurements and networks that define the 'typical' class, the classification is a re-application of the fit rather than an out-of-sample prediction.
full rationale
The paper's core experimental contribution—showing MP-HESD across optimizers, datasets, augmentation, and fine-tuning—is largely independent and does not reduce to its inputs. However, the proposed HESD type criterion Ct is, by the author's own statement, identical (up to sign) to the generalization criterion re from the prior paper [5]. The threshold -0.6 is calibrated from observations in [5], and the conditions (3)-(4) are asserted without derivation. Thus the 'unified methodology' partially reuses previously fitted thresholds while presenting them as new completion of the analysis. The claim that MN-HESD stems from external gradient manipulation is weakly supported (no controlled clipping experiment), but that is a correctness/evidence concern rather than a circularity. Overall, partial circularity from renaming and fitted thresholds, but the central empirical demonstrations have independent content. Score 4.
Assumptions & free parameters
free parameters (3)
- MP-HESD threshold for Ct =
-0.6
- Δre threshold =
1.5
- ΔKH05 threshold =
1.2
assumptions (3)
- domain assumption Hessian eigenvalues of trained NNs can be computed reliably from gradients used in the HESD evaluation algorithm
- domain assumption The behavior of HESD described in [5] (e.g., negative section shrinking during training) holds for the models considered here
- ad hoc to paper MN-HESD indicates external gradient manipulation, not poor generalization
invented entities (1)
-
QS-HESD (quasi-singular HESD)
Cite this review
Pith. "Pith review of The effects of Hessian eigenvalue spectral density type on the applicability of Hessian analysis to generalization capability assessment of neural networks." pith.science (2026). https://pith.science/paper/NYQNACWJ
@misc{pith2026250417618,
author = {Pith},
title = {Pith review of: The effects of Hessian eigenvalue spectral density type on the applicability of Hessian analysis to generalization capability assessment of neural networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/NYQNACWJ}},
note = {Machine review of arXiv:2504.17618}
}
read the original abstract
Hessians of neural network (NN) contain essential information about the curvature of NN loss landscapes which can be used to estimate NN generalization capabilities. We have previously proposed generalization criteria that rely on the observation that Hessian eigenvalue spectral density (HESD) behaves similarly for a wide class of NNs. This paper further studies their applicability by investigating factors that can result in different types of HESD. We conduct a wide range of experiments showing that HESD mainly has positive eigenvalues (MP-HESD) for NN training and fine-tuning with various optimizers on different datasets with different preprocessing and augmentation procedures. We also show that mainly negative HESD (MN-HESD) is a consequence of external gradient manipulation, indicating that the previously proposed Hessian analysis methodology cannot be applied in such cases. We also propose criteria and corresponding conditions to determine HESD type and estimate NN generalization potential. These HESD types and previously proposed generalization criteria are combined into a unified HESD analysis methodology. Finally, we discuss how HESD changes during training, and show the occurrence of quasi-singular (QS) HESD and its influence on the proposed methodology and on the conventional assumptions about the relation between Hessian eigenvalues and NN loss landscape curvature.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[10]
How to train your vit? data, augmentation, and regularization in vision transformers,
A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer, “How to train your vit? data, augmentation, and regularization in vision transformers,” 2022. [Online]. Available: https://arxiv.org/abs/2106.10270
arXiv 2022
-
[5]
N. Gabdullin, “Investigating generalization capabilities of neural networks by means of loss landscapes and hessian analysis,” 2025. [Online]. Available: https: //arxiv.org/abs/2412.10146
work page Pith review arXiv 2025
-
[1]
Visualizing the loss landscape of neural nets,
H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” 2018. [Online]. Available: https://arxiv.org/abs/1712.09913
arXiv 2018
-
[2]
An empirical analysis of the optimization of deep network loss surfaces,
D. J. Im, M. Tao, and K. Branson, “An empirical analysis of the optimization of deep network loss surfaces,” 2017. [Online]. Available: https://arxiv.org/abs/1612.04010
arXiv 2017
-
[3]
Emergent properties of the local geometry of neural loss landscapes,
S. Fort and S. Ganguli, “Emergent properties of the local geometry of neural loss landscapes,” 2019. [Online]. Available: https://arxiv.org/abs/1910.05929
arXiv 2019
-
[4]
Pyhessian: Neural networks through the lens of the hessian,
Z. Yao, A. Gholami, K. Keutzer, and M. Mahoney, “Pyhessian: Neural networks through the lens of the hessian,” 2020. [Online]. Available: https://arxiv.org/abs/1912.07145
arXiv 2020
-
[6]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”
-
[7]
Whytransformersneedadam: A hessian perspective,
Y.Zhang, C.Chen, T.Ding, Z.Li, R.Sun, andZ.-Q.Luo, “Whytransformersneedadam: A hessian perspective,” 2024. [Online]. Available: https://arxiv.org/abs/2402.16788
arXiv 2024
Show all 21 references
-
[8]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021. [Online]. Available: https://arxi...
2021 arXiv
-
[9]
Improving language understanding by generative pre- training,
A. Radford and K. Narasimhan, “Improving language understanding by generative pre- training,” 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:49313245
2018
-
[11]
Imagenet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255
2009
-
[12]
Do imagenet classifiers generalize to imagenet?
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do imagenet classifiers generalize to imagenet?” 2019. [Online]. Available: https://arxiv.org/abs/1902.10811
2019 arXiv
-
[13]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017. [Online]. Available: https://arxiv.org/abs/1412.6980
2017 arXiv
-
[14]
An overview of gradient descent optimization algorithms,
S. Ruder, “An overview of gradient descent optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1609.04747
2017 arXiv
-
[15]
Cinic-10 is not imagenet or cifar-10,
L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, “Cinic-10 is not imagenet or cifar-10,” 2018. [Online]. Available: https://arxiv.org/abs/1810.03505
2018 arXiv
-
[16]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 2019. [Online]. Available: https://arxiv.org/abs/1711.05101
2019 arXiv
-
[17]
Learning multiple layers of features from tiny images,
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Toronto, Ontario, Tech. Rep. 0, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
2009
-
[18]
Randaugment: Practical automated data augmentation with a reduced search space,
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” 2019. [Online]. Available: https://arxiv.org/abs/1909.13719
2019 arXiv
-
[19]
Adahessian: An adaptive second order optimizer for machine learning,
Z. Yao, A. Gholami, S. Shen, M. Mustafa, K. Keutzer, and M. W. Mahoney, “Adahessian: An adaptive second order optimizer for machine learning,” 2021. [Online]. Available: https://arxiv.org/abs/2006.00719
2021 arXiv
-
[20]
Visualizing high-dimensional loss landscapes with hessian directions,
L. Böttcher and G. Wheeler, “Visualizing high-dimensional loss landscapes with hessian directions,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2024, no. 2, p. 023401, feb 2024. [Online]. Available: https://dx.doi.org/10.1088/1742-5468/ad13fc 11
2024 doi
-
[2015]
Available: https://arxiv.org/abs/1512.03385
[Online]. Available: https://arxiv.org/abs/1512.03385
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.