{"id":"c882fd7b-0c6e-456a-b68c-3d18802a4baf","arxiv_id":"2504.17618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hessian spectral density type, mainly positive versus mainly negative, determines whether Hessian-based generalization criteria apply, and mainly negative spectra are attributed to external gradient manipulation.","lead":"This paper studies when the spectrum of Hessian eigenvalues can be used to judge whether a neural network will generalize. It proposes a simple ratio criterion to decide which spectra are usable, and warns that gradient-manipulating optimizers make the method inapplicable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that MN-HESD is caused by external gradient manipulation is asserted without a controlled experiment, so the exclusion rule for MN-HESD networks is not yet justified.","rationale":"The reader's verdict is already conditional, and my read does not move it. The paper's contribution depends on the causal claim in Section 3.4 that MN-HESDs are a consequence of external gradient manipulation, so NNs with MN-HESDs must be excluded. The evidence is one AdaHessian ViT experiment where ResNet20 stays MP and ViT becomes MN, with the difference attributed to spatial averaging over the hidden dimension, plus an unverified conjecture about gradient clipping in [10]. No ablation removes or adds these manipulations while holding architecture, data, and optimizer fixed. Additionally, the Hessian at a fixed checkpoint is mathematically independent of the optimizer's gradient transform; the mechanism would need to explain why the optimizer's trajectory consistently lands in MN-HESD regions, not why the HESD evaluation is 'distorted.' This is testable, and the concrete test above would settle it. Until then, the unified methodology should remain conditional on establishing the causal mechanism.","tokens_in":6717,"tokens_out":5472,"duration_ms":56635,"concrete_test":"Run ViT-small on cinic under four conditions: (1) AdamW baseline, (2) AdamW with global-norm gradient clipping at 1.0 as in [10], (3) AdaHessian as in Section 3.4, and (4) AdaHessian with Hessian-diagonal spatial averaging replaced by the exact per-parameter diagonal, holding all other hyperparameters fixed. Compute Ct (Eq. 1) at every epoch and record HESD type. If MN-HESD appears only in conditions 2 and 3 but not 4, the 'external gradient manipulation' cause is supported; if condition 4 still gives MN-HESD, or condition 2 does not, the causal claim fails and the exclusion rule would need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central exclusion—that MN-HESD networks cannot be analyzed with the previously proposed generalization criteria—rests on Section 3.4's claim that MN-HESD 'is a consequence of external gradient manipulation.' This premise is not actually tested. The AdaHessian experiment only compares ResNet20 (MP) with ViT-small (MN), and the difference is attributed to 'spatial averaging over the hidden dimension' without ablating that averaging; architecture and optimizer are confounded. For the well-generalizing ViT of [10], the paper admits 'it cannot be verified exactly' that global-norm-1 gradient clipping causes MN-HESD, and no clipping experiment is run. Moreover, HESD at a fixed checkpoint is a property of the loss landscape and is independent of the optimizer's gradient transform; if the mechanism is meant literally, the paper's causal story is conceptually incomplete. If MN-HESD can arise without external gradient manipulation, the proposed HESD-type check and exclusion rule would discard generalizing networks, invalidating the unified methodology.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how Hessian eigenvalue spectral density (HESD) types affect the applicability of the author's earlier Hessian-based generalization criteria [5]. It reports experiments across optimizers (SGD, AdamW, AdaHessian), architectures (ResNet, ViT), and datasets (cinic, CIFAR-10, ImageNet) showing that 'mainly positive' (MP) HESD is typical, and that 'mainly negative' (MN) HESD can arise when gradients are manipulated externally (AdaHessian, and, speculatively, gradient clipping). It proposes criterion Ct = min(λneg)/max(λpos) with threshold -0.6 to classify HESD type, adds numerical conditions Δre<1.5 and ΔKH05<1.2 for good generalization, and combines them into a unified assessment algorithm. It also documents quasi-singular HESD under excessive training and argues that Ct is more robust than the maximum Hessian eigenvalue.","tokens_in":7006,"tokens_out":4120,"duration_ms":38566,"significance":"If the MN-HESD mechanism claim were fully supported, the paper would provide a useful practical safeguard for Hessian-based generalization assessment. The paper is honest in labeling the [10] explanation as unverified, and the proposed conditions are explicit, concrete, and testable. However, the central exclusion rule rests on a single confounded experiment and an admitted speculation about the mechanism, so the significance is currently conditional. The work is a sensible incremental extension of the author's prior [5] methodology, but it does not yet stand as a self-contained validation of the unified approach.","major_comments":[{"comment":"The core claim that MN-HESD is caused by external gradient manipulation is not established. The AdaHessian experiment compares ResNet20 (which yields MP-HESD) with ViT-small (which yields MN-HESD), differing simultaneously in architecture and optimizer; no ablation of AdaHessian's spatial averaging is performed, so the attribution to 'spatial averaging over the hidden dimension' is confounded. The extension to the gradient-clipping ViT of [10] is explicitly marked as 'cannot be verified exactly,' and no gradient-clipping experiment is run anywhere in the paper. Because the HESD type check and the exclusion rule for MN-HESD networks are load-bearing components of the proposed unified methodology, a controlled experiment is required: for example, the same architecture trained with and without gradient clipping, or with AdaHessian's gradient transform enabled versus disabled, with repeated runs.","section":"Section 3.4"},{"comment":"The proposed causal story is conceptually incomplete. The Hessian is a property of the loss landscape at a checkpoint and is not directly altered by the optimizer's gradient transformation unless that transformation changes the trajectory of training. The statement that 'HESD evaluation algorithm does not account for optimizer operations' suggests a conflation between Hessian computation and gradient-based quantities. Please clarify whether MN-HESD is claimed to arise from a genuinely different parameter trajectory (which would be a statement about where training converges) or from a bias or error in the Hessian estimation procedure when gradients are manipulated. These are distinct mechanisms and have different implications for the validity of the exclusion rule.","section":"Section 3.4"},{"comment":"The numerical conditions Δre < 1.5 and ΔKH05 < 1.2 are introduced as part of the unified methodology but are not derived or independently validated. The manuscript provides no evidence that these thresholds transfer across architectures, datasets, or optimizers. Since these conditions form a contribution of the paper rather than a restatement of [5], the author should either provide a derivation from the known behavior of re and KH05 under HESD changes, or validate the thresholds on a separate set of models that were not used to set them.","section":"Section 4.3, Eqs. (3) and (4)"},{"comment":"Equation (1) defines Ct as essentially -re from [5], so the proposed HESD type criterion is not independent of the original generalization criterion. The threshold -0.6 is calibrated from the maximum observed Ct values in [5] (with a safety margin), and the updated conditions in Section 4.3 are also stated without independent derivation. This circularity should be acknowledged explicitly, and the reliability of the thresholds should be demonstrated on data or settings not used to select them, otherwise the claim of a 'unified HESD analysis methodology' is stronger than the evidence supports.","section":"Section 2.2, Eq. (1)"}],"minor_comments":[{"comment":"The text refers to 'Figures 8 (b) and (c)', but Figure 8 has only panels (a) and (b); please renumber or correct the cross-reference.","section":"Section 4.1"},{"comment":"The final sentence says the method 'completes the NN generalization assessment method for MN-HESD NNs,' but the methodology excludes MN-HESD networks; this appears to be a typo for 'MP-HESD NNs' and should be corrected.","section":"Section 5"},{"comment":"Some model names are inconsistent (e.g., 'VIT' vs. 'VIT small' vs. 'VIT Small'); please standardize capitalization and acronym usage.","section":"Tables 1-4"},{"comment":"No repeated runs, error bars, or random-seed details are reported. Please state the number of seeds used or, if each experiment was run once, say so explicitly, since the claim that HESD type is stable across settings would be stronger with such information.","section":"Experimental protocol"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is largely an extension of the author's own prior work [5], and the novel central claim (MN-HESD arises from external gradient manipulation) is currently supported by one confounded experiment and an explicitly unverified case study. If the author can add controlled experiments and either derive or independently validate the thresholds, the paper could become a useful practical companion to [5]. As it stands, the unified methodology is not yet justified by the presented evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a follow-up to the author's earlier Hessian-based generalization work, and it reads like a careful empirical note, but the load-bearing claim is the one thing not actually tested. That's the short version.\n\nWhat's genuinely new: the paper shows MP-HESD holds across a wide range of optimizers, datasets, augmentations, and fine-tuning regimes—that's a useful sanity check for the earlier criteria. The observation that AdaHessian can flip a ViT's HESD type from MP to MN between adjacent epochs (Figure 6) is real and worth knowing; so is the quasi-singular HESD that shows up under excessive training, with criteria values drifting proportionally so the generalization gap signal survives. These are honest empirical contributions.\n\nThe soft spot is the causal story. The paper concludes that MN-HESD is 'a consequence of external gradient manipulation,' but it never runs a controlled experiment that isolates gradient manipulation. The AdaHessian comparison confounds architecture (ResNet20 vs ViT-small) with optimizer, and the spatial-averaging explanation is a hypothesis, not a result. For the apparently well-generalizing ViT from [10], the paper admits it 'cannot be verified exactly' that global-norm-1 gradient clipping caused the MN-HESD, and no clipping experiment is run. There's also a conceptual gap: HESD is a property of the loss landscape at a checkpoint, independent of the optimizer's gradient transform; if the mechanism is meant literally, the paper doesn't bridge that gap.\n\nThe negative-side consequence is that the proposed HESD type check and exclusion rule—'if MN-HESD, don't trust the criteria'—rest on the untested premise. If MN-HESD can arise without external gradient manipulation, the rule would throw away networks that actually generalize. The thresholds (Ct > -0.6, Δre<1.5, ΔKH05<1.2) are fitted to the author's prior data without independent validation, and no code or data are provided, so independent replication is currently not possible.\n\nWho this is for: a reader working on Hessian-based generalization metrics will get a useful warning that HESD type is a confounder and should be checked. The QS-HESD observation is worth keeping in mind. But the paper should not be taken as establishing the MN-HESD causal mechanism.\n\nRecommendation: I'd send it to a serious referee if the venue wants empirical papers, but the referee should ask for a controlled gradient-manipulation experiment (e.g., same architecture with and without clipping) and for either code/data or a tightening of the claims. As it stands, the exclusion rule is overclaimed.","headline":"A useful empirical caution about Hessian spectral density types, but the central claim that MN-HESD comes from gradient manipulation is untested and the exclusion rule is not yet justified.","tokens_in":7473,"tokens_out":2411,"would_cite":false,"duration_ms":23342,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural network's Hessian spectrum says nothing about generalization when it is mainly negative, because such spectra arise from external gradient manipulation rather than from the loss landscape.","keywords":["neural networks","Hessian eigenvalue spectral density","loss landscape curvature","generalization assessment","AdaHessian","gradient clipping","quasi-singular Hessian","spectrum type criterion"],"falsifier":"Train the same VIT small on cinic under identical settings twice, once with global-norm gradient clipping at norm 1 and once without any clipping, and compute the HESD at every epoch. If the no-clipping control ever exhibits MN-HESD, then external gradient manipulation is not the sole cause and the exclusion rule rests on the wrong premise; if MN-HESD appears only in the clipped run, the paper's core claim is supported.","tokens_in":6556,"feed_emoji":"🧠","tokens_out":7457,"duration_ms":66600,"temperature":0.7,"pith_summary":"The paper asks when the shape of a neural network's Hessian eigenvalue spectrum can be trusted as a signal about generalization. It argues that under ordinary training and fine-tuning, the spectrum is mainly positive (MP-HESD) regardless of optimizer, dataset, preprocessing, augmentation, and model family. The key discovery is that a mainly negative spectrum (MN-HESD) appears when training manipulates gradients externally, through operations like AdaHessian averaging, normalization, or clipping, and therefore does not by itself mean the network generalizes poorly. On that basis the paper adds a spectrum-type check to its earlier Hessian generalization criteria, so that those criteria are applied only to MP-HESD networks and MN-HESD cases are excluded as unreliable. This matters because Hessian-based generalization assessment is a cheaper alternative to training many candidate models, and the paper makes precise when that method is valid.","feed_headline":"Negative Hessian spectra come from gradient fiddling, not bad models","feed_subtitle":"Only mainly-positive Hessian spectra should feed Hessian-based generalization metrics; a new type check enforces that rule.","key_machinery":"The central object is the Hessian eigenvalue spectral density (HESD), the distribution of eigenvalues of the loss Hessian at a trained model. The paper's central identity is the type criterion $$C_t = \\frac{\\min(\\lambda_{\\text{neg}})}{\\max(\\lambda_{\\text{pos}})},$$ computed from the most negative and most positive Hessian eigenvalues, with MP-HESD defined by $C_t > -0.6$ and MN-HESD by $C_t \\le -0.6$. This number carries the argument: it is the gate that decides whether the previously proposed generalization criteria, stated as $\\Delta r_e < 1.5$ and $\\Delta K_{H05} < 1.2$, are applicable, and it doubles as a checkpoint-selection score when training accuracies are tied. The machinery also includes the notion of quasi-singular HESD, in which all eigenvalues approach zero after excessive training, which the paper uses to show that raw eigenvalue-scale measures can degenerate while ratio-based criteria remain informative.","core_discovery":"The central claim is that MN-HESD is a consequence of external gradient manipulation, not of poor training or bad generalization, and therefore the previously proposed Hessian generalization criteria are valid only for MP-HESD. To establish this, the paper trains VIT small and ResNet20 and fine-tunes VIT on cinic and ImageNet under combinations of AdamW, SGD with and without momentum, RandAugment, and frozen versus unfrozen weights. All such runs produce MP-HESD. In contrast, VIT small trained with AdaHessian flips between MP-HESD and MN-HESD at neighboring epochs while accuracy stays essentially constant, and the paper attributes this to AdaHessian's spatial averaging over the hidden dimension. It then conjectures, without direct verification, that the MN-HESD reported in [10] came from global-norm gradient clipping. The proposed safeguard is the HESD type criterion $C_t > -0.6$: only spectra satisfying it enter the unified Hessian analysis, while MN-HESD runs are flagged as cases where the methodology is unreliable.","pith_inferences":["If the gradient-artifact explanation is right, then any Hessian-generalization study that includes clipped or preconditioned training runs without a spectrum-type check should be re-examined, and future empirical work should report whether gradients were altered.","The $C_t$ ratio is scale-free and could serve as a checkpoint-selection score in settings where Hessian trace or top eigenvalue are inflated by magnitude, such as comparing models across precisions, batch sizes, or training lengths.","A direct ablation with global-norm clipping at several thresholds, not just norm 1, would test whether the HESD distortion scales with clipping strength and would turn the paper's conjecture about [10] into a measured claim.","The quasi-singular HESD finding suggests that absolute eigenvalue flatness measures are not robust to epoch count; a normalized curvature ratio like $C_t$ might be a more stable companion in loss-landscape visualization and model selection."],"forward_implications":["Any deployment of Hessian-based generalization assessment must start with the $C_t > -0.6$ type check and refuse MN-HESD networks, since their spectra reflect optimizer or preprocessing operations rather than the loss landscape.","Standard training recipes, including SGD with or without momentum, AdamW, fine-tuning with frozen or unfrozen weights, RandAugment, and different datasets, produce MP-HESD, so the criteria remain applicable across a wide practical range.","MN-HESD should not be read as evidence of poor generalization: the AdaHessian VIT kept 99% training and 49% generalization accuracy while its $C_t$ dipped below threshold at alternating epochs.","Excessive training can drive Hessian eigenvalues toward zero, producing quasi-singular HESD, but the ratio-based criteria $r_e$ and $K_{H05}$ keep their train/test separation, so they stay informative even when the raw eigenvalue scale degenerates.","Among checkpoints with comparable training accuracy, choosing the one with the largest $C_t$ can outperform choosing the one with the smallest maximum eigenvalue, finding better generalization at an earlier epoch in the paper's experiments."],"supporting_citations":[{"why":"Supplies the prior generalization criteria ($r_e$, $K_{H05}$) and the base observation that trained NNs typically show MP-HESD, which this paper qualifies and extends.","marker":"[5]"},{"why":"Supplies the anomalous VIT whose MN-HESD coexists with high training and generalization accuracy, the case that motivates the whole investigation.","marker":"[10]"},{"why":"Supplies AdaHessian, the optimizer whose diagonal averaging produces alternating MN-HESD in this paper's VIT experiments and anchors the gradient-manipulation explanation.","marker":"[19]"},{"why":"Provides the Hessian computation and spectral density estimation approach used to obtain the paper's HESD plots.","marker":"[4]"},{"why":"Supplies the RandAugment augmentation scheme, used to test whether augmentation changes HESD type and to replicate the training recipe of [10].","marker":"[18]"},{"why":"Theorized that Hessian eigenvalues shrink with continued training; this paper confirms that as quasi-singular HESD and shows the ratio criteria survive it.","marker":"[3]"}],"fun_headline_variants":["Hessian sign flips trace to gradient manipulation, not model quality","Only positive-Hessian spectra should judge generalization","New criterion separates trustworthy Hessian spectra from fake ones","AdaHessian's averaging flips Hessian spectra, misleading generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that MN-HESD arises specifically from external manipulation of the gradients used in Hessian computation, a conclusion inferred from AdaHessian experiments and an unverified assumption about gradient clipping in the network of [10]; if MN-HESD can also arise without such manipulation, the blanket exclusion of MN-HESD networks is unjustified.","fun_headline_variants_meta":{"raw":{"variants":["Hessian sign flips trace to gradient manipulation, not model quality","Only positive-Hessian spectra should judge generalization","New criterion separates trustworthy Hessian spectra from fake ones","AdaHessian's averaging flips Hessian spectra, misleading generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1577,"prompt_tokens":1004,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":620,"tokens_out":573,"duration_ms":5881,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:34:38.183180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same VIT small on cinic under identical settings twice, once with global-norm gradient clipping at norm 1 and once without any clipping, and compute the HESD at every epoch. If the no-clipping control ever exhibits MN-HESD, then external gradient manipulation is not the sole cause and the exclusion rule rests on the wrong premise; if MN-HESD appears only in the clipped run, the paper's core claim is supported.","supporting_citations":[{"cited_title":"Investigating generalization capabilities of neural networks by means of loss landscapes and Hessian analysis","cited_arxiv_id":"2412.10146","evidence_quote":"Supplies the prior generalization criteria ($r_e$, $K_{H05}$) and the base observation that trained NNs typically show MP-HESD, which this paper qualifies and extends."},{"cited_title":"ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning","cited_arxiv_id":"2006.00719","evidence_quote":"Supplies AdaHessian, the optimizer whose diagonal averaging produces alternating MN-HESD in this paper's VIT experiments and anchors the gradient-manipulation explanation."}],"review_version":1}