{"id":"407f09d9-d2cd-47a8-9352-ce424d4094d9","arxiv_id":"2602.21160","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Per-class epistemic uncertainty decomposes mutual information as C_k = Var[p_k]/(2 μ_k), enabling class-specific deferral and shift detection.","lead":"A new per-class uncertainty score C_k = spread/(2×average) splits a classifier's total epistemic uncertainty into class-level shares that add up to the usual mutual information. It is tested for deciding when a medical AI should defer to a human, and for spotting out-of-distribution inputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selective-prediction advantage may be an artifact of the safe/critical split: the paper's own footnote reports a binary cross-boundary MI baseline essentially tied with Ccrit_max (0.284 vs 0.285).","rationale":"The mathematical derivation of C_k is sound: the Hessian of entropy is diagonal, so the second-order expansion of MI does split into per-class terms, and the paper's ρ_k diagnostic is an honest attempt to flag the approximation's limits. The Taylor-remainder concern is real (Grade 3 has only 63% reliable C_k, and the third-order term scales as 1/μ^2), but the paper already limits Ccrit_max to regimes where ρ_k is low and offers CBEC as a fallback. The more damaging issue is empirical. In the DR experiment, the paper compares against scalar MI, raw variance, and one-vs-rest binary MI, but omits the most natural baseline for a safety-critical task with a known partition: MI computed on the collapsed safe-vs-critical binary variable. The appendix footnote reveals this baseline achieves AUSC 0.284±0.015, statistically indistinguishable from Ccrit_max (0.285±0.016; mutual win rate 49.5%/50.5%). A method whose entire selling point is per-class attribution must beat a class-aware scalar baseline; if it merely ties it, the improvement over MI is explained by the partition, not by the decomposition. The exclusion reason ('not part of the C_k framework') is not a substantive justification, and the claimed numerical instability is not quantified. This should be treated as a required revision before the main selective-prediction claim is accepted. The OoD gains are small and overlapping, reinforcing the conditional rather than unconditional verdict.","tokens_in":38029,"tokens_out":5790,"duration_ms":58164,"concrete_test":"Re-run the DR selective-prediction evaluation with the binary cross-boundary MI baseline (S={0,1} vs C={2,3}) included in Table 2 and the same 200-bootstrap protocol; report AUSC, 80%-coverage FNR, and pairwise win-rate vs. Ccrit_max. If the two remain statistically tied (as the footnote suggests), the headline claim that per-class C_k improves selective prediction over class-aware scalar MI is not supported; the paper should be revised to either feature this baseline or temper the claim. Also report the number of samples triggering numerical instability and the sensitivity of the AUSC estimate to their exclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that per-class decomposition enables safer deferral: Ccrit_max reduces DR selective-risk AUSC by 34.7% over MI (0.285 vs 0.436). But the appendix (D.6, Table 8 note) reports a baseline not in Table 2: binary cross-boundary MI computed on the known collapsed partition S={0,1}, C={2,3}. This baseline achieves AUSC 0.284±0.015, statistically tied with Ccrit_max 0.285±0.016 (mutual win rate 49.5%/50.5%). The authors exclude it because it is 'not part of the C_k framework' and 'exhibits numerical instability' (NaN when μ_safe≈0 or μ_safe≈1). The first reason is not a scientific justification for omitting a baseline that controls for the most important confound: knowing the safe/critical partition. The second applies to a small fraction of samples and is never quantified in the main text. If a scalar MI on the partition ties the proposed per-class metric, then the reported advantage is attributable to the class partition, not to the per-class decomposition; the central claim fails its most direct empirical test. The Taylor-remainder issue for rare classes (Grade 3 only 63% reliable, median ρ=0.218) is secondary and partly acknowledged, but the omitted baseline is the load-bearing problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a per-class epistemic uncertainty decomposition C_k(x) = Var[p_k]/(2 mu_k), derived from a second-order Taylor expansion of the mutual information I(y; omega | x). The sum of C_k over classes is claimed to approximate MI, with a skewness diagnostic rho_k flagging inputs where the approximation degrades. The authors evaluate the method on three tasks: selective prediction for diabetic retinopathy, where critical-class aggregates (Ccrit_max, Ccrit_sum, CBEC) are compared against scalar and variance baselines; out-of-distribution detection on FashionMNIST->KMNIST and MIMIC-III->Newborn; and label-noise disentanglement experiments under end-to-end versus transfer learning. The headline empirical claim is that Ccrit_max reduces selective risk (AUSC) by 34.7% over MI for diabetic retinopathy grading.","tokens_in":38433,"tokens_out":3527,"duration_ms":36405,"significance":"The theoretical core is clean and the decomposition is genuinely parameter-free apart from a numerical epsilon. The per-class vector, if reliable, would be a useful diagnostic tool that is absent from current scalar MI practice. The paper also provides a thoughtful axiomatic analysis, a skewness diagnostic, and validation across multiple inference regimes and datasets, with code and models made available. These are real strengths. However, the main empirical claim is currently undercut by an omitted baseline: a binary cross-boundary MI evaluated on the known safe/critical partition ties Ccrit_max in AUSC. If that result stands, the selective-prediction advantage is driven by knowing the class partition rather than by the per-class decomposition itself. The OoD improvements are also small and not shown to be statistically significant. The manuscript is therefore not ready for acceptance in its present form, but the issues are addressable.","major_comments":[{"comment":"The headline '34.7% reduction over MI' compares Ccrit_max (AUSC 0.285) to multiclass scalar MI (0.436), but the appendix reports a binary cross-boundary MI baseline computed on the known collapsed partition S={0,1}, C={2,3} that achieves AUSC 0.284±0.015, statistically tied with Ccrit_max (mutual win rate 49.5%/50.5%). This baseline controls for the most important confound — knowledge of the safe/critical partition — and should be included in Table 2. The reasons given for excluding it ('not part of the C_k framework', 'numerical instability') are not scientific justifications; the instability is unquantified and applies to a small fraction of samples. As it stands, the central selective-prediction claim is not supported: the improvement is attributable to the partition, not to the per-class decomposition.","section":"§3.3, Table 2 vs. Appendix D.6, Table 8 footnote"},{"comment":"The primary metric Ccrit_max relies on C_k for critical classes, but the paper's own skewness diagnostic shows that Grade 3 has only 63.4% of samples with rho_k<0.3 and median rho=0.218. For Grade 3, mu_k≈0.06 and the third-order remainder scales as 1/mu_k^2, so C_k can be unreliable for a large fraction of exactly the samples that should drive deferral. The manuscript acknowledges this and proposes CBEC as a fallback, but Ccrit_max remains the recommended primary metric and its AUSC advantage is not recomputed on the subset of samples where all critical classes have rho<0.3. The paper should report performance stratified by skewness reliability, or explicitly condition the recommendation on the reliability regime.","section":"§3.3 and §2.5, Table 7"},{"comment":"The OoD improvements of sum_k C_k over MI are small and no significance testing is reported. On FashionMNIST, 0.735±0.009 vs. 0.724±0.009; on MIMIC-III, 0.815±0.017 vs. 0.802±0.004. With means and standard deviations overlapping or nearly overlapping, the claim that sum_k C_k 'achieves the highest AUROC' needs bootstrap or paired-test evidence. This is a secondary claim, but it is presented as a validation of the decomposition and should be supported.","section":"§4.2, Table 3"}],"minor_comments":[{"comment":"The abstract's '34.7% over MI' should specify that the comparison is to class-agnostic multiclass MI, not to the binary cross-boundary MI that ties with Ccrit_max. The current wording is misleading.","section":"Abstract / §3.3"},{"comment":"The footnote about the binary cross-boundary MI baseline is easy to miss and contains the most important control result of the paper. It should be moved to the main text and discussed honestly.","section":"Appendix D.6, Table 8"},{"comment":"The skewness ratio rho_k is defined as |m_{3,k}|/(3 mu_k Var[p_k]). The text explains it as the ratio of third- to second-order corrections, but the denominator is 3 mu_k Var[p_k], not 3 mu_k sigma^2_k. This is correct, but it would help to spell out that this equals |m_{3,k}|/(6 mu_k^2 C_k), clarifying the 1/mu_k^2 amplification.","section":"§2.5, Definition 2.10"},{"comment":"The O(K^2) scaling formula assumes approximately uniform variance and a particular probability allocation. This is a useful heuristic, but it should be labeled as a worst-case asymptotic sketch rather than a general result, since MI itself is bounded by log K.","section":"Appendix F.3, Eq. (57)"}],"recommendation":"major_revision","confidential_remarks":"The omitted binary cross-boundary MI baseline is the decisive issue. It is not a matter of taste: the paper's main empirical claim fails its most direct control. However, the problem is fixable within the manuscript's scope — the authors can include the baseline, re-estimate the advantage, and reframe the contribution around per-class interpretability rather than a selective-prediction gain over partition-aware MI. I therefore recommend major revision rather than rejection. The OoD and disentanglement sections also need modest strengthening (significance tests, reliability-stratified results) before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core decomposition is the real contribution: C_k = Var[p_k]/(2 mu_k) follows from a second-order Taylor expansion of entropy, sums to MI (they report r=0.988), and the 1/mu_k weighting is a sensible boundary correction. The skewness diagnostic is a useful flag. The axiomatic analysis is honest about trade-offs. The per-class error signatures are genuinely useful for interpretability.\n\nThe soft spot is the selective-prediction claim. The paper's own footnote (D.6) reports a binary cross-boundary MI baseline that ties Ccrit_max (0.284 vs 0.285, mutual win rate ~49/50). The authors exclude it from the main table because it is 'not part of the C_k framework' and 'exhibits numerical instability'. Those reasons don't hold up. The first is not a scientific justification; the second is unquantified and applies to a small fraction of samples. If scalar MI computed on the collapsed safe/critical partition does just as well, then the 34.7% improvement over MI is attributable to knowing the partition, not to the per-class decomposition. That is a load-bearing confound for the main empirical claim.\n\nThe Taylor remainder issue is secondary but real: for Grade 3, only 63% of samples have rho<0.3, median rho=0.218. The third-order term scales as 1/mu_k^2, so C_k is least reliable for rare classes, exactly where safety-critical decisions matter. The paper acknowledges this and offers CBEC as a fallback, but CBEC requires domain knowledge of the partition.\n\nOoD gains are small (0.735 vs 0.724; 0.815 vs 0.802) and may not be statistically robust. The disentanglement study is a useful confirmation that training regime matters more than metric, but it does not rescue the main claim.\n\nDespite these problems, the per-class decomposition is a clean and potentially useful tool. For practitioners who want class-level attribution, C_k gives something scalar MI cannot. The paper is honest about limitations and engages with the literature. It deserves a serious referee, but the authors should be required to report the binary partition baseline in the main table and address the rare-class Taylor issue.\n\nFor a reading group, I'd bring it because the baseline controversy is instructive.","headline":"The per-class MI decomposition is mathematically sound and useful, but the paper's main empirical advantage over scalar MI appears to come from knowing the safe/critical partition rather than from the per-class decomposition.","tokens_in":38834,"tokens_out":3538,"would_cite":true,"duration_ms":31884,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62H30","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Epistemic uncertainty in classification decomposes per class into a variance-to-mean ratio, C_k = Var[p_k]/(2 μ_k), and the sum closely recovers mutual information.","keywords":["per-class epistemic uncertainty","mutual information decomposition","Bayesian deep learning","selective prediction","out-of-distribution detection","boundary suppression","skewness diagnostic","diabetic retinopathy"],"falsifier":"For samples where the paper's own skewness diagnostic reports ρ_k>0.3 (e.g., 37% of Grade 3 diabetic retinopathy samples), compute exact MI and compare with Σ C_k; if the per-class attribution reverses the ordering of classes or the error exceeds the third-order term, the second-order decomposition is falsified exactly where the diagnostic warns.","tokens_in":37968,"feed_emoji":"🎯","tokens_out":7447,"duration_ms":59912,"temperature":0.7,"pith_summary":"In safety-critical classification, the cost of misclassifying a rare, dangerous class outweighs errors among benign classes, yet standard Bayesian deep learning summarizes epistemic uncertainty with a single scalar that cannot say which class the model is unsure about. This paper proposes a per-class vector C_k = Var[p_k]/(2 μ_k) derived from a second-order Taylor expansion of entropy, so that the components sum to approximately the mutual information. The 1/μ_k weighting corrects a boundary-suppression pathology that makes raw variance vanish for rare classes. On diabetic retinopathy grading, using the per-class score for critical classes reduces the area under the selective-risk curve by 34.7% relative to mutual information; it also improves out-of-distribution detection and reveals asymmetric shifts invisible to scalar metrics. The paper argues that how uncertainty is propagated through the network matters as much as which metric is used.","feed_headline":"Per-class uncertainty score cuts AI selective risk by 35%","feed_subtitle":"A variance-to-mean ratio splits epistemic uncertainty by class, helping models say which mistakes they fear most.","key_machinery":"The key object is the per-class epistemic vector C(x) with components C_k = σ_k^2/(2 μ_k), where μ_k and σ_k^2 are the mean and variance of the class-k softmax probability across stochastic forward passes. It follows from a second-order Taylor expansion of entropy whose Hessian is diagonal, so the approximation to mutual information splits additively over classes. The 1/μ_k factor is the entropy curvature and doubles as a dispersion-index normalisation that prevents boundary suppression. A companion skewness diagnostic ρ_k = |m3,k|/(3 μ_k σ_k^2) measures the third-order correction relative to the second-order term, flagging when C_k is unreliable; for such cases, a cross-boundary confusion s","core_discovery":"Mutual information I(y;ω|x) can be approximately decomposed per class as C_k(x)=Var[p_k](x)/(2 μ_k(x)), so that Σ_k C_k ≈ I. The decomposition follows from the diagonal Hessian of Shannon entropy: expanding expected entropy about the mean μ yields a curvature-weighted sum of per-class variances. The 1/μ_k weight corrects boundary suppression (raw variance vanishes as μ_k→0) and makes components comparable across rare and common classes. The aggregate satisfies axioms of non-negativity, vanishing at certainty, and monotonicity under mean-preserving spreads; a skewness diagnostic flags where the second-order approximation degrades. Critical-class C_k reduces selective risk in diabetic retinopa","pith_inferences":["The paper leaves implicit an active-learning application: because C_k attributes MI to specific classes, acquisition functions could query instances where a target class's C_k is highest, directly reducing per-class error rather than global MI; this is testable on class-imbalanced benchmarks.","Since C_k is the index of dispersion (Fano factor) scaled by 1/2, the decomposition connects to classical dispersion theory; a natural extension is modelling p_k with a Poisson-binomial or Dirichlet-multinomial and deriving exact per-class terms beyond the second order.","A meta-criterion suggested by the results but not formalised: use the skewness diagnostic ρ_k as a switch between Ccrit_max (when ρ_k<0.3) and CBEC (when ρ_k>0.3); the paper's MC-dropout ranking reversal indicates this switch could be automated.","The strong dependence of disentanglement on training regime implies that benchmarks comparing uncertainty metrics should control for inference scheme; otherwise metric rankings may reflect the posterior approximation rather than the metric itself."],"forward_implications":["Because Σ_k C_k ≈ I(y;ω|x) with near-perfect rank correlation (Spearman r=0.998), practitioners can replace scalar MI with the vector and retain MI's interpretation while adding class resolution.","Targeting critical-class C_k (Ccrit_max) improves selective prediction for diabetic retinopathy: AUSC drops 34.7% vs MI and 56.2% vs raw variance, with the gain persisting under deep ensembles.","The per-class view exposes 'epistemic signatures' of errors: catastrophic misses and severity underestimates have nearly identical MI but distinct C_k profiles, enabling different remediation strategies.","For out-of-distribution detection, Σ_k C_k gives the highest AUROC on FashionMNIST→KMNIST and MIMIC-III→Newborn, and the per-class decomposition reveals asymmetric shifts (e.g., MIMIC mortality signal concentrated in the survival class).","Under controlled label noise, Σ_k C_k is less entangled with aleatoric uncertainty than MI in 19 of 20 end-to-end conditions; transfer learning degrades both metrics by an order of magnitude, showing posterior quality is as important as the metric."],"fun_headline_variants":["Per-class uncertainty cut selective risk by 34.7%","Where AI is ignorant: per-class uncertainty map","Splitting model ignorance by class improves safety-critical AI","Decomposed epistemic uncertainty: per-class variances beat mutual info","New metric shows where model uncertainty concentrates by class"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central approximation—that the second-order Taylor expansion of entropy is accurate for each class—is weakest precisely for rare, safety-critical classes with skewed posterior samples; the paper's own diagnostic shows only 63% of Grade 3 samples are reliable (ρ_k<0.3), so C_k can be untrustworthy where it matters most.","fun_headline_variants_meta":{"raw":{"variants":["Per-class uncertainty cut selective risk by 34.7%","Where AI is ignorant: per-class uncertainty map","Splitting model ignorance by class improves safety-critical AI","Decomposed epistemic uncertainty: per-class variances beat mutual info","New metric shows where model uncertainty concentrates by class"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":2996,"prompt_tokens":857,"completion_tokens":2139,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2060}},"tokens_in":601,"tokens_out":2139,"duration_ms":16094,"temperature":1.0,"reasoning_tokens":2060,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:04:57.148009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For samples where the paper's own skewness diagnostic reports ρ_k>0.3 (e.g., 37% of Grade 3 diabetic retinopathy samples), compute exact MI and compare with Σ C_k; if the per-class attribution reverses the ordering of classes or the error exceeds the third-order term, the second-order decomposition is falsified exactly where the diagnostic warns.","supporting_citations":[],"review_version":1}