{"id":"37230670-cf1f-45d3-95e6-398b6ad22646","arxiv_id":"2608.07630","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A deep-learning adaptation of homoscedastic and heteroscedastic linear-regression variance estimators, scaled with EKFAC, to attribute prediction uncertainty to label noise versus scarce data.","lead":"This paper adapts two classical statistics formulas to deep neural networks to estimate whether a model's uncertainty on a test point comes from noisy labels or from a lack of nearby training data. It scales the estimators with Kronecker-factored approximations and shows on controlled datasets that comparing the two can flag which subpopulation a test point belongs to.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The source-attribution claim depends on fixing tangent features at the trained weights; a bootstrap-retraining comparison on the meta-class tasks would reveal whether the resulting bias undermines the Ho/HeC3 ratio.","rationale":"The paper's central contribution is a practical way to attribute epistemic uncertainty to label noise versus scarce data, using the Ho and HeC3 estimators on a single trained model. For that attribution to be reliable, the linearized model with fixed tangent features must accurately mimic the behavior of the true network under the resampling of training data. This is exactly the assumption the reader identified as weakest, and it is indeed load-bearing: if tangent features vary substantially across retrained models, both Ho and HeC3 are evaluated at an unrepresentative anchor, and the ratio HeC3/Ho may separate noisy from missing subpopulations for the wrong reasons. The concern is not merely that the assumption could be wrong in general, but that it is most likely to fail in the scarce-data regime that the paper explicitly targets. The controlled meta-class experiments in Section 5 demonstrate the estimators' rankings correlate with known subpopulations, but they do not provide ground-truth variance estimates against which to check the linearized quantities; the Monte Carlo ground truth appears only in the 1D toy experiments. A bootstrap-retrain comparison on the meta-class tasks directly tests whether the single-model linearization recovers the actual sampling variance, and would settle whether the source attribution is an artifact of the approximation. I also considered the EKFAC double approximation and the lack of a direct comparison with Schmitt et al. (2025), but those affect quantitative accuracy and completeness rather than the validity of the core attribution mechanism. Since the reader already assigned a conditional verdict based on the same assumption, and the proposed test is a natural way to lift the condition, the verdict should remain unchanged rather than being strengthened or weakened.","tokens_in":21829,"tokens_out":11683,"duration_ms":118193,"concrete_test":"On the MNIST meta-class setup of Section 5, compute Monte Carlo variance estimates by retraining the same architecture from scratch on 100 bootstrap resamples of the training set (keeping the noisy and missing subpopulations fixed), and compare the empirical variance of the logits for test points to the Ho and HeC3 estimates from a single model. Report the ratios HeC3/MC and Ho/MC separately for noisy and missing subpopulations; also report the relative change in tangent features across resamples, e.g., ||φ_resample - φ_anchor|| / ||φ_anchor||. If HeC3/MC is close to 1 for the noisy subpopulation but deviates substantially for the missing subpopulation, the linearization assumption fails in exactly the regime where source attribution is claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Eq. 16 and Eq. 17 distinguish response-noise from finite-sample variability requires the linearization in Eq. 12 to be valid for different training samples. Section 2.3 explicitly assumes 'orders greater than 1 are negligible when considering different samples X and ε', but the tangent features φ_{ŵ} are evaluated at a single fitted ŵ that is itself a random function of the data. For a different draw of (X, ε), the operating point shifts and the features change; the estimators do not account for this change. In the missing-subpopulation experiments of Section 5, test points lie far from the training manifold, so the local quadratic terms in the Taylor expansion can be large and the fixed-feature approximation may be particularly inaccurate. Because Ho and HeC3 share the same anchor features, the observed separation between noisy and missing subpopulations could partly be an artifact of the linearized model rather than a faithful estimate of the true sampling variance. The paper's own conclusion acknowledges that linearization quality is not yet quantified, making this the load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts two classical linear-regression variance estimators to deep neural networks. The homoscedastic estimator Ho (Eq. 16) is derived from the fixed-design, homoscedastic-noise variance formula after replacing inputs by tangent features; the heteroscedastic-consistent estimator HeC3 (Eq. 17) is derived from the jackknife estimator for random designs. The authors scale both estimators with an EKFAC approximation of the Fisher information matrix, and evaluate them on 1D/2D toy tasks, MNIST/Fashion-MNIST meta-class tasks with controlled noisy and missing subpopulations, and CIFAR-10/IMDB benchmarks. The central claim is that Ho and HeC3 quantify distinct sources of predictive uncertainty: response noise versus finite sampling of (X, ε), and that their ratio identifies test points whose uncertainty stems mainly from local label noise or from locally scarce data.","tokens_in":21976,"tokens_out":7071,"duration_ms":71200,"significance":"If the source-attribution claim holds, the paper provides a practical, single-model alternative to ensembles for decomposing predictive uncertainty into aleatoric and finite-sample components, with an approximation that scales to modern architectures. The paper has clear strengths: the linear-regression derivations are clean and the toy experiments include Monte Carlo ground truth, which supports the behavior of both estimators for small models. The controlled meta-class experiments are a good design for probing subpopulation-level source attribution, and the comparison against Deep Ensemble on CIFAR-10 is informative. The main weakness is that the deep-learning version of the estimators relies on an unquantified linearization assumption: tangent features are anchored at a single fitted ŵ that is itself a random function of the training data, and the authors explicitly state in Section 2.3 that this requires higher-order terms to be negligible across different draws of X and ε. Since the source-attribution claim is exactly about variability across training samples, this assumption is load-bearing and is not yet validated by a retraining-based ground truth on the real-scale tasks.","major_comments":[{"comment":"The central claim that Eq. 16 and Eq. 17 estimate variance from response noise ε and from finite sampling of (X, ε) relies on the validity of the first-order expansion around a single fitted ŵ across different training draws. The paper states in Section 2.3 that this is 'equivalent to making the assumption that orders greater than 1 are negligible when considering different samples X and ε,' but no evidence is provided that the tangent features φ_{ŵ} remain representative when ŵ changes under retraining. The missing-subpopulation experiments in Section 5 are precisely the regime where test points lie far from the training manifold, so quadratic terms can be large. I recommend adding a bootstrap-retraining comparison on the meta-class tasks: retrain the model on multiple subsamples (or with different label-noise draws), compute the empirical variance of predictions, and compare it with the Ho/HeC3 estimates. This directly tests whether the fixed-feature approximation preserves the source attribution.","section":"§2.3, Eq. (12) and Eqs. (16)–(17)"},{"comment":"The deep-learning HeC3 estimator uses the Sherman–Morrison leave-one-out formula with tangent features anchored at the full-data optimum ŵ, but the leave-one-out prediction of the actual nonlinear network under retraining is not equal to this linearized quantity. In linear regression the jackknife formula is exact; in the nonlinear setting it is an additional approximation whose error is not measured. The 1D toy experiment provides Monte Carlo ground truth for one small MLP, but the meta-class and CIFAR experiments do not compare HeC3 against retrained ensembles or jackknife-by-retraining. A concrete experiment with a small number of retrained models on the meta-class tasks (or on a subset of MNIST) would clarify whether the estimator tracks the true leave-one-out variability.","section":"§2.3 and Appendix A.2"},{"comment":"The 'source disentanglement' is presented as a heuristic ratio analysis rather than a formal decomposition. Appendix A.5 shows that HeC3/Ho is a linear smoother of squared jackknife residuals normalized by the global variance; it is therefore a conditional variance estimator, not a direct estimator of the aleatoric component of the total predictive variance. The empirical support for source attribution is strong only for the noisy subpopulation (HeC3/Ho AUROC 0.97–0.99), while the missing subpopulation AUROCs are moderate (0.58–0.80 for Ho and HeC3 individually). The text says 'both variances are equally high' for the missing subpopulation, but the reported AUROC values do not clearly demonstrate that both estimators reliably identify the missing subpopulation. The claims in this section should be softened or supplemented with additional statistics, such as the actual separation of variance distributions for the two subpopulations.","section":"§5, Table 1, and Appendix A.5"},{"comment":"The jackknife estimator is described as 'free of any assumptions,' but this is only true in the linear-regression setting from which it is borrowed. After linearization, the deep-learning HeC3 estimator inherits the pseudo-response substitution r = y − f_{ŵ}(X), the fixed-feature approximation, and the EKFAC approximation with leverage clipping described in Appendix B.6. The pseudo-responses are not centered and include model misspecification, so interpreting Eq. 17 as the variance due to finite sampling of (X, ε) requires an additional assumption that the misspecification term is absorbed into the noise. This should be stated explicitly and, if possible, tested by comparing the deep-learning estimators with true retraining-based variances on a small-scale task.","section":"§2.2 and §2.3"}],"minor_comments":[{"comment":"The symbol F_{ŵ} is used in Eqs. (15) and (16) but is not defined; it should be defined as F_{ŵ} = Φ_{ŵ}^T Φ_{ŵ}, to distinguish it from F_{ŵλ} = Φ_{ŵ}^T Φ_{ŵ} + λI.","section":"§2.3, Eq. (15)–(16)"},{"comment":"There are several formatting and spelling errors, including 'uncertaintytomoderndeeplearning' in the abstract and 'each test points' instead of 'each test point'. The text also has missing spaces in a few places (e.g., 'experimentsaremeant' and 'GalandGhahramani2016'). A careful copyedit is needed.","section":"Abstract and throughout"},{"comment":"The EKFAC approximation of the meat matrix Φ^T diag(û^2) Φ is not jointly diagonalizable with Φ^T Φ, and the paper considers two approximations: one using the same basis U and one using a different basis V. The accuracy of these approximations is not evaluated. A small experiment comparing exact (or Cholesky-based) and EKFAC-based HeC3 variances would help assess how much of the source-attribution signal is lost in the approximation.","section":"§4 and Appendix A.4"},{"comment":"The clipping of approximate leverages to [0, 1−ε] with ε = 1×10⁻⁴ is an ad hoc correction that may affect the jackknife residuals for high-leverage points. The sensitivity of the HeC3/Ho ratio to this clipping choice is not reported.","section":"Appendix B.6"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the linearization around a single fitted ŵ is the load-bearing assumption, and it is precisely the assumption that needs empirical validation when the claim is about variability across training samples. The paper would be substantially strengthened by a bootstrap-retraining comparison on the meta-class tasks, even at small scale. The current evidence supports Ho and HeC3 as useful uncertainty scores, but the stronger source-disentanglement claim is not yet fully established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely useful paper for anyone doing UQ in deep learning. It takes two textbook linear-regression variance estimators (homoscedastic, and HC3 heteroscedastic-consistent) and adapts them to neural networks by plugging in tangent features and scaling the Fisher matrices with EKFAC. The piece that is actually new is the claim that the ratio HeC3/Ho separates aleatoric noise from finite-sample scarcity, plus the EKFAC machinery to make it run on real architectures. The individual pieces are not new—Rivals and Personnaz did the delta method for NNs, Schmitt et al. recently pushed delta variances, MacKinnon and White gave HC3—but the combination and the source-attribution story are.\n\nThe derivation from linear regression is clean, and the toy experiments are a strong point: they use Monte Carlo ground truth from resampling, and show Ho and HeC3 separately track the right variance components in heteroscedastic and data-scarce regimes. The multi-class extension in the appendix is careful about the stacked tangent features. The authors are honest about the linearization assumption and the approximations. That honesty is real, not just boilerplate.\n\nWhere it gets softer: the load-bearing assumption is that tangent features fixed at the trained weights remain valid across different training draws. Section 2.3 says this explicitly, and the conclusion admits it is not yet quantified. In the missing-subpopulation experiments, test points are far from the training manifold, so the higher-order terms could be exactly where the linearization is worst. A bootstrap-retraining experiment comparing the estimators' predictions to actual retraining variance on those same meta-class tasks would directly address this, and the paper would be much stronger for it. Relatedly, Appendix A.5 shows that HeC3/Ho is a linear smoother of jackknife residuals; that makes the ratio's ability to flag noisy subpopulations partly by construction. Not a flaw, but it tempers the 'disentanglement' language. Two smaller issues: no code released, and no direct comparison with Schmitt et al.'s delta variance estimator, which is the closest recent baseline.\n\nNone of this undermines the core contribution. The estimators work in the tested regimes, the scaling story is plausible, and the paper deserves a serious referee. I would send it to someone who will push on the linearization question and request a retraining bootstrap experiment plus code release. For a reading group focused on uncertainty quantification, this is a good choice. I'd cite it if I write on delta methods or scalable variance estimators.","headline":"A solid, honest adaptation of classical HC variance estimators to deep learning; the source-attribution ratio is heuristic, but the experiments support it.","tokens_in":22533,"tokens_out":2995,"would_cite":true,"duration_ms":29604,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J07","62F40","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Using only a single trained network, the paper derives two linearized variance estimators—Ho for homoscedastic response noise and HeC3 for finite-sample variability—and shows empirically that their ratio tells whether a test point's…","keywords":["epistemic uncertainty","aleatoric uncertainty","linearized neural networks","tangent features","jackknife estimator","heteroskedasticity-consistent estimator","EKFAC","Fisher information matrix"],"falsifier":"Take one architecture and one data distribution, draw many bootstrap training sets, retrain from scratch on each, and record the true per-test-point prediction variance; if the ranks produced by the Ho and HeC3 estimates diverge from this true variance as width shrinks or as test inputs move outside the training support, the fixed-tangent-features linearization is the assumption to reject.","tokens_in":21584,"feed_emoji":"📊","tokens_out":11071,"duration_ms":94216,"temperature":0.7,"pith_summary":"The paper's goal is to make a practical, per-test-point distinction between two sources of uncertainty in deep learning predictions: aleatoric uncertainty from response noise and epistemic uncertainty from finite training data. It derives two estimators from linear-regression theory, Ho and HeC3, after linearizing the network around its trained weights and treating tangent features as fixed. Ho estimates the variance a prediction would have if only response noise were resampled under homoscedastic noise; HeC3 estimates the variance from resampling both inputs and responses, using jackknife residuals. Experiments on toy problems, a regression dataset, and controlled MNIST and Fashion-MNIST subpopulations show the two estimators disagree where the sources differ, so their ratio indicates whether to collect more data or clean labels. EKFAC approximation makes the estimators scalable to modern architectures, giving uncertainty rankings competitive with ensembles at a fraction of the cost.","feed_headline":"Two estimators separate label noise from scarce-data uncertainty","feed_subtitle":"A single trained model can tell which test points need cleaner labels rather than more data.","key_machinery":"The load-bearing object is the tangent-feature linearization: replace the network by its first-order Taylor expansion around the trained weights $\\hat w$, with tangent features $\\phi_{\\hat w}(x) = \\partial f_w(x)/\\partial w|_{w=\\hat w}$ treated as fixed. This makes the deep network formally identical to linear ridge regression, with design matrix $\\Phi_{\\hat w}$ and regularized Fisher information matrix $F_{\\hat w\\lambda} = \\Phi_{\\hat w}^\\top \\Phi_{\\hat w} + \\lambda I$, so textbook variance formulas apply. The two estimators are a homoscedastic ridge estimator, Ho, and a jackknife/sandwich estimator, HeC3, whose middle term uses leave-one-out residuals rather than raw residuals. To scale the inverses to deep architectures, the paper approximates $\\Phi_{\\hat w}^\\top \\Phi_{\\hat w}$ with block-diagonal EKFAC, an eigenvalue-corrected Kronecker factorization that makes inverse Fisher-vector products efficient.","core_discovery":"The central claim is that the distinction between aleatoric and finite-sample epistemic uncertainty can be estimated per test point from a single trained model. For a network trained by ridge-regularized mean squared error, the paper linearizes the predictor as $f_w(x) \\approx f_{\\hat w}(x) + \\phi_{\\hat w}(x)^\\top \\Delta w$, where $\\phi_{\\hat w}(x)$ are the tangent features, the gradients of the output with respect to the weights at the trained parameters. This turns deep-learning prediction into linear ridge regression with design matrix $\\Phi_{\\hat w}$ and pseudo-responses $r = y - f_{\\hat w}(X)$, so the classical estimators carry over directly. Equation 16, $\\hat{\\mathrm{var}}_{\\mathrm{Ho}} = \\hat\\sigma^2 \\phi_{\\hat w}(x)^\\top F_{\\hat w\\lambda}^{-1} F_{\\hat w} F_{\\hat w\\lambda}^{-1} \\phi_{\\hat w}(x)$, estimates the variance that would remain if the training inputs were fixed and only response noise were resampled under a homoscedastic assumption. Equation 17, $\\hat{\\mathrm{var}}_{\\mathrm{HeC3}} = \\phi_{\\hat w}(x)^\\top F_{\\hat w\\lambda}^{-1} \\Phi_{\\hat w}^\\top \\mathrm{diag}(\\hat u^2) \\Phi_{\\hat w} F_{\\hat w\\lambda}^{-1} \\phi_{\\hat w}(x)$, is a jackknife and sandwich estimator built from leave-one-out residuals $\\hat u_i = \\hat e_i / (1 - h_{ii})$, and estimates the total variance from resampling both inputs and responses. The paper argues that the two estimators respond differently to the two sources: HeC3 is relatively larger around overlapping classes or noisy labels, while both are large in data-sparse regions, making the ratio a practical source-attribution tool.","pith_inferences":["Because the HeC3/Ho ratio is a linear smoother of leave-one-out squared residuals with tangent-feature alignment weights, it naturally doubles as a local conditional-variance estimator; a testable extension is to calibrate a threshold on this ratio for selective prediction.","The fixed-tangent-features premise is most credible for wide networks after the initial representation-learning phase; comparing these estimates against full-retraining Monte Carlo variances on narrow networks or shifted test distributions would map where the linearization breaks.","For very large models, using only last-layer features is cheaper, and the paper finds HeC3 more stable in that regime; an implicit extension is to deploy HeC3 on last-layer features when full EKFAC is too costly.","A natural next step, flagged by the authors, is to derive analogous estimators through logistic-regression diagnostics, which would give source-aware uncertainty for softmax classifiers and generative transformers."],"forward_implications":["At a test point where HeC3 is large and Ho is small, the dominant source is local label noise; the paper's controlled experiments indicate that cleaning noisy labels improves those predictions.","At points where both HeC3 and Ho are large, the dominant source is scarce local data; adding training examples in that region is the indicated remedy.","The ratio of HeC3 to Ho nearly identifies a noisy subpopulation in controlled settings, with AUROC up to about 0.99, so a single number per test point can flag heteroscedastic noise.","Computed with EKFAC, the estimators need only inverse Fisher-vector products and are competitive with a ten-model ensemble for misclassification detection on CIFAR-10, at a fraction of the compute."],"supporting_citations":[{"why":"Supplies the jackknife variance principle that HeC3 adapts to test-point predictions.","marker":"Quenouille 1956; Tukey 1958"},{"why":"Gives the leave-one-out closed form that turns jackknife residuals into the sandwich 'meat'.","marker":"Sherman and Morrison 1950"},{"why":"Defines the HC3 heteroskedasticity-consistent covariance estimator with improved finite-sample properties, the direct ancestor of HeC3.","marker":"MacKinnon and White 1985"},{"why":"Provides the heteroskedasticity-consistent sandwich estimator underpinning the uncorrected version and the econometric viewpoint.","marker":"White 1980"},{"why":"Introduces tangent features and the neural tangent kernel used as the fixed design matrix after linearization.","marker":"Jacot, Gabriel, and Hongler 2018"},{"why":"Supplies empirical evidence that training dynamics become linear after an initial representation-learning phase, justifying the linearization.","marker":"Fort et al. 2020"},{"why":"Defines EKFAC, the eigenvalue-corrected Kronecker-factored approximation that makes inverse Fisher-vector products scalable.","marker":"George et al. 2018"},{"why":"Introduces the Kronecker-factored approximation of per-layer Fisher blocks used as the basis for EKFAC.","marker":"Martens and Grosse 2015"},{"why":"Provides the earlier linearization-based confidence-interval construction for neural networks that this paper extends toward source attribution.","marker":"Rivals and Personnaz 2000"}],"fun_headline_variants":["Two estimators separate label noise from data scarcity per point","Deep learning uncertainty: per-test-point noise vs sparse data","Linearized estimators split aleatoric and epistemic uncertainty","Homo and hetero estimators trace uncertainty sources in DL","Single model identifies which test points need labels vs data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the network's prediction depends linearly on its weights across the random training sets being averaged over, so tangent features computed once at the trained weights stay representative; if higher-order terms matter under retraining or distribution shift, both variance estimates become biased and the source attribution may fail.","fun_headline_variants_meta":{"raw":{"variants":["Two estimators separate label noise from data scarcity per point","Deep learning uncertainty: per-test-point noise vs sparse data","Linearized estimators split aleatoric and epistemic uncertainty","Homo and hetero estimators trace uncertainty sources in DL","Single model identifies which test points need labels vs data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1721,"prompt_tokens":1003,"completion_tokens":718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":619,"tokens_out":718,"duration_ms":6752,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:28:10.515753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one architecture and one data distribution, draw many bootstrap training sets, retrain from scratch on each, and record the true per-test-point prediction variance; if the ranks produced by the Ho and HeC3 estimates diverge from this true variance as width shrinks or as test inputs move outside the training support, the fixed-tangent-features linearization is the assumption to reject.","supporting_citations":[],"review_version":1}