{"id":"5756c01a-feec-4e3e-801e-e3683270cbbc","arxiv_id":"2608.10374","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fisher8 stabilizes neural uncertainty regression by applying an output-layer natural-gradient correction with per-point normalization, improving RMSE, NLL, and ECE on UCI, weak lensing, and rotated MNIST.","lead":"This paper proposes Fisher8, a training correction that measures each gradient step by the KL divergence between the predicted distributions before and after the step, rather than by Euclidean distance. Across regression and representation learning benchmarks, it reports better accuracy, likelihood, and calibration than existing stabilizers, with no extra hyperparameters beyond the learning rate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed approximate KL trust radius (Eq. 16) is not a property of the actual update: shared output weights and features couple per-point output changes, so the normalization in Eq. 15c does not imply the bound it is used to derive.","rationale":"The reader's verdict is CONDITIONAL, and my read does not change that. The method is empirical and the experiments are suggestive, but the paper overstates the KL trust radius. The reader identified the looseness of the Eq. 16 bound and the heuristic nature of the normalization; my concern is more specific: the normalization does not even control the actual output-parameter displacements in a shared network, because the feature Gram matrix couples all per-point changes. This makes the 'approximate KL trust radius' claim a property of an idealized per-point model rather than of the deployed algorithm. The concrete test would settle whether the issue is merely a theoretical overstatement or a real discrepancy. I do not see grounds for rejection: the empirical comparisons, the learning-rate sensitivity fingerprint, and the absence of extra data-dependent hyperparameters are genuine contributions, even if the geometric narrative needs revision. Conditional acceptance with a request to either prove or drop the trust-radius claim, and to release code, remains appropriate.","tokens_in":22614,"tokens_out":7968,"duration_ms":78292,"concrete_test":"Train Fisher8 on a UCI dataset (e.g., Concrete) with the paper's SGD setup (lr=0.005, batch 32, 100 steps). After every update, record the network outputs before and after the step for the current batch. Compute the exact batch KL between the product predictive distributions using the closed-form Gaussian KL formula, and compare it to (a) the second-order estimate in Eq. 16, (b) the claimed upper bound ½e^{-min(s)}η² + ¼η², and (c) the actual norms ||δμ||₂ and ||δs||₂ of the per-point output changes. If the actual norms deviate from η or the exact KL exceeds the bound in any step, the trust-radius claim is false as stated. Repeating this across learning rates and datasets would determine whether the discrepancy is central to the method's behavior or only a theoretical overstatement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's theoretical contribution rests on the statement that the unit normalization in Eq. 15c guarantees ||δμ||₂ = ||δs||₂ = η, and that Eq. 16 is a post hoc readout of how far the joint predictive distribution has moved. This is not established for a shared network. Even in the output-layer-only case with a linear head μ_i = w_μᵀh_i + b_μ, the update from Eq. 15c is Δw_μ = -η/||g_μ||₂ Σ_i g_{μ,i} h_i and Δb_μ = -η/||g_μ||₂ Σ_i g_{μ,i}, where g_μ,i = -(y_i - μ_i). The actual change in predicted means is therefore δμ_j = h_jᵀΔw_μ + Δb_μ = -η/||g_μ||₂ Σ_i g_{μ,i}(h_jᵀh_i + 1). The vector δμ equals -η/||g_μ||₂ (H + 11ᵀ)g_μ, not -η g_μ/||g_μ||₂. Unless the feature Gram matrix H is the identity and the features sum to zero, ||δμ||₂ ≠ η and the per-point displacements are not the normalized gradient signals. The same coupling applies to δs. Consequently, the second-order KL expression in Eq. 16 is computed from the backpropagated signal, not from the actual changes in the predictive distributions, and the claimed approximate trust radius does not follow. This weakens the geometric interpretation and the distinction between Fisher8 and ordinary gradient clipping. The empirical results may still stand, but the central theoretical claim about KL control is unsupported. The additional looseness for overconfident predictions (large e^{-min(s)}) noted by the reader is a secondary issue; the primary issue is that the bound is not even the correct quantity for the actual update.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Fisher8, an output-layer training correction for heteroscedastic regression. For each point, the Gaussian NLL gradient is preconditioned by the exact 2x2 Fisher matrix of the Gaussian predictive distribution (Eq. 13). The batch rule normalizes each point's preconditioned gradient signal to unit L2 norm before backpropagation (Eq. 15c). The authors claim that this normalization provides an approximate KL trust radius for the batch update (Eq. 16), that several previously proposed stabilizers (beta-NLL, Faithful, and regularization-based approaches) converge on overlapping components of this geometric correction, and that, empirically, Fisher8 improves RMSE, NLL, and ECE on UCI regression, weak-lensing cosmology, and rotated MNIST representation learning.","tokens_in":23038,"tokens_out":6035,"duration_ms":55392,"significance":"If the claims held, Fisher8 would be a valuable, hyperparameter-light stabilizer for heteroscedastic regression, and the unification of prior stabilizers would be a useful conceptual contribution. The per-point Fisher derivation is correct, the method is simple and easily implementable, and the empirical evaluation is broad, spanning synthetic, UCI, cosmological, and representation-learning benchmarks. The paper's central theoretical contribution, however, is the approximate KL trust radius, and that claim is not supported by the actual shared-weight update. The empirical results may stand independently, but the advertised geometric interpretation needs substantial revision before the paper can be accepted.","major_comments":[{"comment":"The claim that the unit normalization in Eq. (15c) guarantees ||δμ||₂ = ||δs||₂ = η is not correct for the actual network update. For a linear output head μ_i = w_μᵀh_i + b_μ with shared weights, the change in the j-th predicted mean is δμ_j = -η/||g_μ||₂ Σ_i g_{μ,i}(h_jᵀh_i + 1), where g_μ is the vector of per-point natural gradients; unless the feature Gram matrix is the identity and the features sum to zero, the per-point displacements are not the normalized gradient signals. Consequently, the second-order KL estimate in Eq. (16) is not a bound on the actual movement of the predictive distribution, and the advertised 'approximate KL trust radius' is not established. The authors should either derive a correct bound for the shared-weight update or revise the claim to refer to the backpropagated signal rather than the predictive distributions.","section":"Section 3.3, Eq. (16)"},{"comment":"The batch update rule is a heuristic aggregation of per-point natural gradients: it normalizes each point's gradient to unit norm and sums the signals, but this operation is not derived from the Fisher metric of the batch loss L_θ = (1/B)Σ_i ℓ_{θ_i}. The per-point Fisher geometry is exact only for the single-point objective; for a batch with shared weights, the output-layer Jacobians couple the points, and no per-point normalization implies a KL constraint on the joint update. This should be stated explicitly as a design choice, and the 'batch-level trust radius' language in Section 3.3 should be qualified accordingly.","section":"Section 3.2, Eq. (15c)"}],"minor_comments":[{"comment":"The tone of Appendix D ('we believe we in fact ate', 'peer reviewers have thoroughly enlightened us') and the McLaren footnote is not appropriate for a journal article; please rewrite these as neutral remarks or remove them.","section":"Appendix D and Section 3.1 footnote"},{"comment":"The statement that 'Fisher8 achieves the best RMSE, NLL, and ECE on nearly every benchmark' is slightly overstated: in Table 2, Baseline-NLL has lower ECE on Concrete (0.024 vs 0.026) and lower ECE on Kin8nm (0.013 vs 0.017). Please qualify the statement.","section":"Section 4.2.2, Table 2"},{"comment":"The main-text claim that 'other methods collapse to 35–52%' feature accuracy is incomplete when read against Appendix C.2: in the input-independent noise regime, the Collier et al. [2021] and Megerle et al. [2023] baselines achieve 79.46% and 80.96% feature accuracy, respectively. The comparison in the main text should either include these results or state clearly that the 35–52% range refers only to the methods listed in Table 4.","section":"Section 4.4, Table 4 and Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical recipe here is simple and appears to work, but the paper's central theoretical claim—the approximate KL trust radius in Eq. 16—does not hold up under scrutiny. The unit normalization in Eq. 15c does not guarantee unit displacements in the predicted outputs when the network shares weights.\n\nWhat's actually new: the per-point natural gradient for a Gaussian output is textbook, but the particular combination of output-layer Fisher preconditioning, per-point L2 normalization, and the local KL variance diagnostic is new. The diagnostic genuinely separates feature-space activity from distributional mobility, and it offers an addendum to Seitzer et al.'s flatness hypothesis. The experimental work is broad and honestly reported—UCI with a full learning-rate sweep, weak lensing, and rotated MNIST—and the authors deliberately use SGD to avoid confounding Fisher8's preconditioning with Adam's. When the method wins, it tends to win across RMSE, NLL, and ECE, which is the right way to claim superiority.\n\nThe soft spot is real and load-bearing. Eq. 16 relies on the statement that normalizing per-point natural gradients to unit norm gives ||δμ||_2 = ||δs||_2 = η for the actual output-layer changes. That is false for a shared network. Even for a linear head, δμ_j = -η/||g_μ|| Σ_i g_{μ,i}(h_jᵀh_i + 1), not -η g_{μ,j}/||g_μ||. The feature Gram matrix couples all per-point changes, so the bound in Eq. 16 is computed from the backpropagated signal, not from the actual predictive distribution movement. The trust-region language is thus unjustified. This is not a minor caveat; it separates a geometrically grounded method from a per-point gradient normalization heuristic.\n\nThe batch rule itself is a heuristic, not derived from a batch-level objective. That is fine if presented honestly, but the paper overstates the trust radius. The unification claim (that beta-NLL, Faithful, and regularization methods are all pieces of Fisher8) is a reasonable post hoc interpretation but not a derivation. And there is no code, which makes the empirical results harder to check.\n\nFor a reader working on heteroscedastic regression, the method is worth trying: it is simple, adds no extra hyperparameters beyond learning rate, and the empirical evidence is consistent. But the theoretical narrative needs revision before publication. I would send it to peer review with a request for the authors to either fix the trust-region derivation (possible if they track actual output changes through shared features) or remove it and frame Fisher8 as an output-layer gradient normalization heuristic. The experiments alone are strong enough to justify a serious referee.","headline":"A useful empirical recipe and a nice diagnostic, but the paper's KL trust-radius claim (Eq. 16) is unsupported: the per-point normalization in Eq. 15c does not control output displacements in a shared network.","tokens_in":23519,"tokens_out":3859,"would_cite":false,"duration_ms":34265,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62B10","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Fisher8, an output-layer natural-gradient rule, stabilizes heteroscedastic regression and improves calibration.","keywords":["heteroscedastic regression","natural gradient","Fisher information","Gaussian negative log-likelihood","uncertainty calibration","gradient stabilization","uncertainty-aware representations","KL trust region"],"falsifier":"Measure the true KL divergence between the batch predictive distributions before and after a Fisher8 update on a network with overconfident predictions; if the actual KL regularly exceeds the claimed approximate upper bound by a large factor, the trust-radius argument does not control training.","tokens_in":22451,"feed_emoji":"📉","tokens_out":5862,"duration_ms":51410,"temperature":0.7,"pith_summary":"The paper argues that recurring instabilities in training neural networks to predict both mean and variance under Gaussian negative log-likelihood are not flaws of the loss itself but of stepping in Euclidean parameter space. It derives Fisher8, an output-layer update that replaces Euclidean gradient steps with natural-gradient steps measured by the Fisher information of each point's predicted Gaussian, then normalizes each point's corrected gradient before backpropagation. The claimed payoff is a stabilizer with no data-dependent hyperparameters beyond learning rate, an approximate KL trust radius for each update, and empirical gains in RMSE, NLL, and calibration across benchmarks. The paper further contends that several independently proposed stabilizers are partial versions of this same geometric correction.","feed_headline":"Fisher8 steadies uncertainty-aware regression without extra knobs","feed_subtitle":"A per-point Fisher-geometry update keeps mean and variance training stable and improves calibration.","key_machinery":"The central object is the $2\\times 2$ Fisher information matrix of the network's per-point Gaussian predictive distribution, $F(\\theta)=\\operatorname{diag}(e^{-s},1/2)$, which turns Euclidean steepest descent into natural-gradient descent constrained by KL divergence. Fisher8 applies $F(\\theta)^{-1}$ only to the output-layer gradients, yielding the reoriented updates $e^s\\nabla_\\mu \\ell$ and $2\\nabla_s \\ell$, and then normalizes each point's update vector to unit norm before backpropagation; that normalization is what gives every point equal distributional mobility per step and produces the batch trust radius. The mechanism's work is to make the shared learning rate control approximate KL movement rather than raw parameter distance.","core_discovery":"Fisher8 is the claim that jointly predicting mean and log-variance with the Gaussian NLL becomes stable if output-layer gradients are preconditioned by the per-point Fisher information $F(\\theta)=\\operatorname{diag}(e^{-s},\\tfrac12)$ for $\\theta=(\\mu,s)$, $s=\\ln\\sigma^2$, reorienting the mean update to $e^s\\nabla_\\mu \\ell$ and the log-variance update to $2\\nabla_s \\ell$, then L2-normalizing each point's preconditioned gradient before summing over the batch. The paper derives an approximate batch KL trust radius $\\tfrac12 e^{-\\min(s)}\\eta^2 + \\tfrac14\\eta^2$ that bounds how far the predictive product distribution can move, and reads this bound as limiting the incentive to inflate variance to mask poor mean fits. Empirically, the update reaches better RMSE, NLL, and ECE than the $\\beta$-NLL and Faithful baselines on eight UCI regression sets, a weak-lensing cosmology benchmark, and a rotated-MNIST representation task, while retaining downstream feature quality without large weight-space regularization.","pith_inferences":["The local KL-variance diagnostic introduced in the paper could be used as a general early-warning metric: when feature activity persists but output-distribution KL variance stalls, further training on the same objective is unlikely to improve predictive uncertainty.","Because Fisher8 only touches the output layer, the same correction should transfer to other likelihood families whose per-point Fisher is small and diagonal, such as classification or count regression, though the paper does not test them.","The claimed reconciliation of beta-NLL and Faithful suggests a testable prediction: tuning beta-NLL's exponent or the Faithful severing point should be replaceable by Fisher8's normalization at matched learning-rate schedules, with no loss in calibration.","If the approximate KL bound is loose for overconfident networks, an adaptive learning-rate schedule that measures the actual batch KL after each update may extend Fisher8's stable range."],"forward_implications":["Training heteroscedastic regressors with plain SGD and Gaussian NLL becomes viable without beta reweighting, gradient severing, or tuned regularizer corridors.","The beta-NLL exponent, Faithful's Newton steps and trunk severing, and the regularizer corridor of prior work are reinterpreted as overlapping pieces of a single Fisher correction, so future stabilizers can be designed geometrically.","Fisher8-trained networks inherit second-order behavior: faster progress at conservative learning rates and sharper degradation at high learning rates, so learning rate is the single sensitive dial.","Uncertainty-aware feature spaces survive training: on rotated MNIST, Fisher8's penultimate-layer features support digit classification at roughly 80% accuracy even when the regression noise is class-independent, where baseline features collapse to 35-52%.","The approximate KL trust radius gives a concrete post-hoc readout of how far each batch update moves the predictive distribution, which can be monitored during training."],"supporting_citations":[{"why":"Provides the natural-gradient derivation and the second-order KL/Fisher expansion used to motivate the update.","marker":"Amari [1998]"},{"why":"Defines the joint mean-variance Gaussian NLL objective and the heteroscedastic regression task.","marker":"[Nix and Weigend, 1994]"},{"why":"Documents the inverse-variance gradient starvation pathology and defines the beta-NLL reweighting baseline.","marker":"[Seitzer et al., 2022]"},{"why":"Documents mean degradation under joint training and introduces the Faithful gradient-severing and Newton-step baseline.","marker":"[Stirn et al., 2023]"},{"why":"Characterizes variance-absorption failure regimes and proposes the regularizer-corridor stabilization that Fisher8 is compared against.","marker":"[Wong-Toi et al., 2024]"},{"why":"Supplies the natural-parameter heteroscedastic regression reference and the rotated-MNIST protocol for representation evaluation.","marker":"[Immer et al., 2023]"},{"why":"Frames the two empirical fingerprints of second-order methods used to argue Fisher8 behaves curvature-aware.","marker":"[Martens, 2020]"},{"why":"Provides the weak-lensing convergence-map dataset, backbone, and scoring rule for the cosmology benchmark.","marker":"[FAIR Universe Collaboration, 2025]"}],"fun_headline_variants":["Fisher8: Stabilize mean-variance nets with Fisher geometry","Fisher8 reorients gradients to fix heteroscedastic instability","No new knobs: Fisher8 steadies mean and variance training","Fisher8: Curvature-aware updates for calibrated uncertainty","Fisher8 achieves stable uncertainty learning via Fisher geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The stability claim rests on treating each point's normalized gradient as an independent move in its own local geometry and summing those moves; this is a heuristic rather than a consequence of the batch's Fisher metric.","fun_headline_variants_meta":{"raw":{"variants":["Fisher8: Stabilize mean-variance nets with Fisher geometry","Fisher8 reorients gradients to fix heteroscedastic instability","No new knobs: Fisher8 steadies mean and variance training","Fisher8: Curvature-aware updates for calibrated uncertainty","Fisher8 achieves stable uncertainty learning via Fisher geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1242,"prompt_tokens":902,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":256}},"tokens_in":518,"tokens_out":340,"duration_ms":3950,"temperature":1.0,"reasoning_tokens":256,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:21:13.432590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the true KL divergence between the batch predictive distributions before and after a Fisher8 update on a network with overconfident predictions; if the actual KL regularly exceeds the claimed approximate upper bound by a large factor, the trust-radius argument does not control training.","supporting_citations":[{"cited_title":"On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks","cited_arxiv_id":null,"evidence_quote":"Documents the inverse-variance gradient starvation pathology and defines the beta-NLL reweighting baseline."},{"cited_title":"Sanjana, and David A","cited_arxiv_id":null,"evidence_quote":"Documents mean degradation under joint training and introduces the Faithful gradient-severing and Newton-step baseline."},{"cited_title":"Understanding pathologies of deep heteroskedastic regression","cited_arxiv_id":null,"evidence_quote":"Characterizes variance-absorption failure regimes and proposes the regularizer-corridor stabilization that Fisher8 is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the natural-parameter heteroscedastic regression reference and the rotated-MNIST protocol for representation evaluation."},{"cited_title":"NeurIPS 2025 weak lensing uncertainty challenge","cited_arxiv_id":null,"evidence_quote":"Provides the weak-lensing convergence-map dataset, backbone, and scoring rule for the cosmology benchmark."}],"review_version":1}