{"id":"18961acd-4e21-4bc2-bfe9-5bbaebf1c053","arxiv_id":"2412.03391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Risk-aware evidential classifiers that learn a per-sample prior from a misclassification cost matrix lower expected cost on MNIST and CIFAR10 versus cost-sensitive baselines.","lead":"This paper proposes enhancements to Evidential Deep Learning that make classifiers weigh the cost of being wrong when choosing predictions. The authors report that their risk-aware variants reduce average misclassification cost compared with cost-sensitive baselines on MNIST and CIFAR10, while keeping accuracy roughly unchanged.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central cost-reduction claim rests on single-run benchmark numbers with no code or error bars, and the EDL(pg) “bandit” implementation is ambiguous about whether it ever sees labels; a multi-seed rerun under true bandit feedback is needed.","rationale":"The paper's strongest claim is empirical: risk-aware EDL variants reduce misclassification cost by at least 55% on MNIST and 38% on CIFAR10 relative to cost-sensitive training. The math behind the pignistic-prior augmentation is straightforward, and the derivation of the expected risk in Equation (9) is correct, so the theoretical core is not the main risk. The load-bearing weakness is that Tables 2 and 3 contain no confidence intervals, no multiple seeds, and no code, making the central quantitative claim unverifiable as reported. I partially disagree with the reader's specific weakest-assumption choice: observing the scalar cost of the sampled action is sufficient for an unbiased REINFORCE update, so Algorithm 1 does not strictly need the true label y. However, the algorithm's notation and dataset loop imply label access, and no experiment demonstrates performance under genuine bandit feedback. This reinforces, rather than replaces, the need for a careful empirical revalidation. A multi-seed rerun that also removes label access in EDL(pg) would settle both the statistical robustness of the cost-reduction claim and the bandit-feedback claim. The reader's CONDITIONAL verdict remains appropriate; I do not see a fatal logical error, only insufficient empirical support for the headline numbers.","tokens_in":20469,"tokens_out":6667,"duration_ms":71571,"concrete_test":"Run the full Section 5 protocol for MNIST and CIFAR10 with at least 10 independent seeds, reporting mean and standard deviation of Table 3 costs. In the EDL(pg) condition, record only the scalar cost c_i of the sampled action and update with c_i * grad_log(P(i|x)), never using y. If EDL(pg) does not beat CS-L2 in a majority of seeds, or if removing label access changes its cost by a large amount, the central claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5's headline numbers in Tables 2 and 3 are presented as point estimates with no seeds, standard deviations, or released code. Since the claimed 55%/38% cost reduction is the paper's central empirical contribution, the absence of any variance estimate makes it impossible to distinguish a systematic advantage from seed or initialization luck; this is the weakest load-bearing point. A second, secondary ambiguity is Algorithm 1: it is described as bandit feedback but iterates over (x, y) and uses R_yi. In a true contextual bandit the learner observes the scalar cost of the chosen action c_i = R_{y,i}; REINFORCE with c_i * grad_log(P(i|x)) is unbiased and does not require y, so the reader's assumption that y must be observed is not strictly necessary. However, as written the algorithm accesses y, so the implementation genuinely may be supervised, and no ablation shows the method works when only c_i is available. Either way, the empirical claim needs a check that controls for randomness and for the actual feedback model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three desiderata for real-world risk-aware classification (D1: Dirichlet outputs with epistemic uncertainty, D2: transfer from pre-trained models, D3: compositionality), argues that Evidential Deep Learning satisfies them, and presents experiments showing that EDL-tuned pre-trained models improve entropy-based uncertainty scores and that two EDL classifiers can be fused. The main extension is a per-sample pignistic prior gamma_Theta(x) = K softmax(W g_theta(x) + b), with three training variants: riskEDL, EDL(p), and EDL(pg) (a REINFORCE/bandit-style update). On MNIST and CIFAR10 with LeNet, the risk-aware variants are reported to reduce misclassification cost by roughly 55% and 38% relative to cost-sensitive cross-entropy while maintaining accuracy.","tokens_in":20732,"tokens_out":15721,"duration_ms":153697,"significance":"If the empirical claims hold, this is a practically useful and conceptually clean extension of EDL: the pignistic-prior mechanism is simple, Eq. (9) is correctly derived from Dirichlet means, and the paper includes an external cost-sensitive baseline as well as concrete D2/D3 demonstrations. The policy-gradient variant targets a realistic feedback setting, which strengthens the contribution. However, the headline cost reductions rest on single-run point estimates on two small datasets, and the bandit variant as written appears to require true labels; these issues must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The quantity in Eq. (9) is the expected cost of a randomized decision rule that draws the predicted label i with probability p_i. This is not the expected cost of a deterministic decision j, which is sum_i R_{i,j} p_i and is minimized by argmin_j. The paper never states the decision rule used to produce the test-set costs in Tables 2 and 3; Section 5 says the expectation of q_theta(p|x) is used 'as the predictive distribution,' which suggests argmax of the mean pignistic probabilities. If so, the training objective in Eq. (9) does not match the evaluation metric; if instead labels are sampled from P(i|x), the reported costs are random and need variance estimates. Please specify the test-time rule and, if needed, optimize the objective corresponding to the actual decision rule.","section":"Section 4, Eq. (9)"},{"comment":"Algorithm 1 is presented as learning from bandit feedback, but it iterates over (x,y) pairs and uses R_{y,i} in the update. In a true bandit the learner observes only the scalar cost c_i = R_{y,i} of the sampled action i; a REINFORCE update with c_i * grad log P(i|x) is unbiased and does not require y. The pseudocode as written requires the true label y (or the full risk row), so the implementation may be supervised, and the paper provides no ablation showing that EDL(pg) works when only the chosen action's cost is available. Please clarify the feedback model and evaluate the bandit claim under genuinely scalar feedback.","section":"Section 4.3, Algorithm 1"},{"comment":"The central empirical claim—that the proposed approaches reduce misclassification cost by at least 55% on MNIST and 38% on CIFAR10 relative to CS-L2—is based on one run per method with no standard deviations, no seeds, and no released code. With differences of a few tenths of a percent in accuracy and cost reductions that could be influenced by initialization or the stochastic policy in EDL(pg), the reported point estimates cannot support a general superiority claim. Please provide multi-seed means and deviations (and ideally paired significance tests), and specify how the CS-L2 baseline was tuned so the comparison is fair.","section":"Section 5, Tables 2 and 3"}],"minor_comments":[{"comment":"The desiderata are introduced as 'necessary' without an independent argument; since the paper sets them up so that EDL satisfies them by construction, consider framing them as design goals rather than foundational requirements.","section":"Section 2.3"},{"comment":"In the MNIST risk-matrix definition, 'misclassifing' should be 'misclassifying', and the prose should clarify whether 'overestimating' means predicting a digit with a larger value than the true digit.","section":"Section 5"},{"comment":"The label 'L2' for standard cross-entropy training is confusing because L2 usually denotes squared error; use 'CE' or 'cross-entropy' instead.","section":"Section 5, Tables 2 and 3"},{"comment":"Equation (11) writes the update for theta, while Algorithm 1 updates Theta = {W, b}; clarify that only the pignistic-prior parameters are trained for EDL(pg) and the evidence network is frozen.","section":"Section 4.3"},{"comment":"The out-of-distribution source for the CIFAR100 row is not stated; please specify which dataset was used as OoD for that row.","section":"Section 3.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript overlaps with the authors' previous WACV paper [SSJ+21] and states it 'significantly expands' that work; an explicit statement of which components are new would help the editor assess novelty. The empirical evaluation is narrow for the strength of the abstract's claims, but this is a framing issue rather than a technical one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The part of this paper that deserves attention is Section 3: EDL loss as a fine-tuning tool for pretrained classifiers. Table 1 shows accuracy roughly preserved while the entropy AUC for incorrect and OoD predictions drops substantially across CIFAR10, CIFAR100, and ImageNet. The fusion result (CIFAR110 accuracy from 62.4% to 73.5% after EDL tuning) is a concrete demonstration that evidential classifiers can be composed by concatenating Dirichlet parameters. That is the real new contribution, and it is a useful one.\n\nThe risk-aware sections are a different story. riskEDL, EDL(p), and EDL(pg) appeared in the authors' WACV 2021 paper. The current paper adds a desiderata framing, but D1-D3 are not derived or shown necessary; they are defined so that EDL satisfies them. That is framing, not theory.\n\nThe empirical support for the headline cost reductions is thin. Tables 2 and 3 give point estimates with no seeds, no standard deviations, and no code. The claimed 55%/38% reduction over cost-sensitive training could be seed luck; with two small datasets and one architecture, the paper needs multi-seed runs. This is the load-bearing weakness.\n\nAlso, Algorithm 1 is not bandit learning as written. It iterates over (x, y) and uses R_yi, which requires the true label. In a true contextual bandit, the agent sees only the cost of the chosen action c_i; you cannot recover y from a single cost entry in general. A scalar-cost REINFORCE variant (c_i grad log P(i|x)) would be genuinely label-free, but that is not what is presented. So the \"bandit feedback\" claim is unsupported. This is a real, localized flaw.\n\nThe math itself is correct: Eq. 9 follows from Dirichlet means, and the losses are clearly specified. The paper cites its own earlier work appropriately. No major citation problems.\n\nWho gets value: anyone working on UQ or cost-sensitive learning, especially those wanting to retrofit pretrained classifiers with uncertainty estimates. It is not a breakthrough, but the fine-tuning and fusion evidence is useful.\n\nRecommendation: send it to a serious referee. The framing issues can be fixed; the empirical claims need seeds and code, and the bandit section needs either a corrected algorithm or a removed claim. With those revisions, this could be a solid applied paper.","headline":"A modest but useful extension of EDL, where the fine-tuning and fusion evidence in Section 3 is the real new contribution; the risk-aware variants are largely a reprise of WACV 2021, and the headline cost-reduction numbers need seeds and a corrected bandit story.","tokens_in":21257,"tokens_out":3015,"would_cite":true,"duration_ms":30658,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T01","68T45"],"pacs":["07.05.Mh"],"model":"deepseek-v4-flash","headline":"The paper claims that augmenting Evidential Deep Learning with per-sample pignistic priors reduces misclassification cost by at least 55% on MNIST and 38% on CIFAR10 compared with cost-sensitive training, while keeping test accuracy…","keywords":["risk-aware classification","evidential deep learning","uncertainty quantification","Dirichlet distribution","pignistic probabilities","cost-sensitive learning","policy gradient","compositionality"],"falsifier":"Run EDL(pg) in a true bandit environment where only the cost of the chosen class is revealed and the true label is never shown; if the learner cannot compute the gradient in Algorithm 1 (which multiplies R_{y i} by the score-function term), the claimed bandit learning collapses. A simpler check: inspect the update at line 4 of Algorithm 1 and verify that R_{y i} requires the true y to be known; the paper's Section 4.3 admits 'the (x, y) tuple represents each sample x and its true label,' so a reader can settle the matter by seeing whether the observed cost alone determines R_{y i} for every i.","tokens_in":20287,"feed_emoji":"📉","tokens_out":7411,"duration_ms":61621,"temperature":0.7,"pith_summary":"This paper sets out to make deep classifiers risk-aware: instead of only maximizing accuracy, the classifier should minimize the expected cost of its mistakes, especially when some errors are far more expensive than others. The authors argue that Evidential Deep Learning (EDL), which replaces softmax probabilities with a Dirichlet distribution over class probabilities, is the right base because it already satisfies three desiderata: it quantifies epistemic uncertainty, it can fine-tune pretrained models, and it can fuse classifiers trained on disjoint label sets. They then extend EDL with a per-sample pignistic prior, a learned redistribution of prior evidence, so that the model's decisions are biased toward low-risk categories mainly when the evidence is weak. On MNIST and CIFAR10, the extended methods reduce misclassification cost by at least 55% and 38% respectively relative to cost-sensitive training, with accuracy essentially unchanged. A policy-gradient variant, EDL(pg), is presented as learning from bandit feedback, although the stated update assumes access to the true label.","feed_headline":"Risk-aware evidential learning cuts misclassification cost by 55%","feed_subtitle":"A pignistic-prior tweak to EDL beats cost-sensitive training on MNIST and CIFAR10 without losing accuracy.","key_machinery":"The load-bearing object is the Dirichlet distribution as a conjugate prior over class probabilities, parameterised by $\\alpha = c_\\theta(x)+1$ where $c_\\theta(x)$ is a non-negative evidence vector from the network. Risk-awareness is injected through a second head that produces a per-sample pignistic prior $\\gamma_{\\Theta}(x)=K\\cdot\\mathrm{softmax}(W g_\\theta(x)+b)$ with total mass $K$, so that the expected risk formula in Eq. (9) weighs each class by its risk entry $R_{y i}$ times the sum of evidence and prior counts, normalised by $K+\\sum_j c_{\\theta j}(x)$. This construction is what lets the model hedge: when evidence is abundant the prior barely matters, and when evidence is scarce the prior steers decisions toward low-cost categories. A policy-gradient update derived from Eq. (12) is the mechanism claimed to enable bandit-feedback training.","core_discovery":"The central claim is that EDL's Dirichlet output can be turned into a risk-aware decision rule by learning a per-sample pignistic prior $\\gamma_{\\Theta}(x)=K\\cdot\\mathrm{softmax}(W g_\\theta(x)+b)$ that is added to the evidence counts before computing expected risk. The expected risk of labelling $x$ as the true class $y$ is $E[\\mathrm{risk}(x)] = \\frac{\\sum_i R_{y i}(c_{\\theta i}(x)+\\gamma_{\\Theta i}(x))}{K+\\sum_j c_{\\theta j}(x)}$, and the network can be trained to minimise this quantity along with EDL's loss. Three variants are proposed: riskEDL trains both heads jointly, EDL(p) freezes the trained EDL network and fits only the pignistic prior, and EDL(pg) applies a policy-gradient (REINFORCE-style) update to the prior. The paper reports that EDL(p) and EDL(pg) reduce misclassification cost by at least 55% on MNIST and 38% on CIFAR10 compared to cost-sensitive cross-entropy training (CS-L2), while keeping test accuracy essentially unchanged (Tables 2 and 3). It also claims that fine-tuning pretrained classifiers with EDL loss improves their uncertainty estimates and enables principled fusion of classifiers on disjoint label sets.","pith_inferences":["If the bandit claim fails the label-access test, the practical scope of EDL(pg) shrinks to settings where the true label is recoverable from the observed cost; a genuine bandit variant would need to estimate the cost of unchosen actions or use importance-weighted updates.","The same pignistic-prior construction could be grafted onto other Dirichlet-predicting uncertainty models (e.g., prior networks), since the risk formula only needs the Dirichlet parameters; testing that would separate the benefit of the prior mechanism from EDL-specific loss details.","The method's apparent ability to shift errors toward low-cost confusions while preserving accuracy suggests it could be directly useful for ordinal or auction-like tasks, where overestimating value is much costlier than underestimating it.","The three desiderata (D1-D3) are presented as necessary but not sufficient; methods that require architectural overhauls or external outlier datasets will struggle to satisfy D2 and D3, which may be a useful filter for comparing uncertainty-quantification approaches."],"forward_implications":["Fine-tuning a pretrained classifier with EDL loss preserves its accuracy while making its uncertainty much higher on misclassified and out-of-distribution inputs; fusing two EDL-tuned classifiers on disjoint label sets raises combined accuracy from 62.4% to 73.5% for CIFAR10+CIFAR100.","Risk-aware EDL with pignistic priors reduces misclassification cost by at least 55% on MNIST and 38% on CIFAR10 relative to cost-sensitive training, with accuracy essentially unchanged.","EDL(p) and EDL(pg) outperform riskEDL on the tested risk matrices because they apply the risk correction mostly when evidence is low; riskEDL may be preferable when the risk matrix is binary with equal error costs.","EDL(pg) reaches lower cost than cost-sensitive training within a few training epochs and approaches the performance of EDL(p), which sees the full cost matrix."],"supporting_citations":[{"why":"Supplies the original EDL framework, squared-error loss, and the propositions the paper builds on.","marker":"[SKK18a]"},{"why":"Earlier risk-aware EDL results that this paper extends with pignistic priors and bandit training.","marker":"[SSJ+21]"},{"why":"Defines pignistic probabilities, which the paper uses to compute expected misclassification risk.","marker":"[Sme05]"},{"why":"Cost-sensitive regularization method (CS-L2) used as the baseline that the proposed methods outperform.","marker":"[GDC+20]"},{"why":"LeNet architecture used in the MNIST and CIFAR10 experiments.","marker":"[LBB+98]"},{"why":"Dirichlet aggregation and neutrality properties that justify fusing classifiers on disjoint label sets (D3).","marker":"[FKG10]"},{"why":"REINFORCE/policy-gradient machinery that EDL(pg)'s update rule adapts.","marker":"[SB18]"},{"why":"Evidence that deep classifiers are overconfident, motivating the risk-aware desiderata.","marker":"[GPSW17]"}],"fun_headline_variants":["EDL tweak cuts misclassification cost by 55%","Uncertainty-aware evidential classifier cuts costs 55%","Risk-aware EDL with pignistic prior beats cost-sensitive training","Pignistic prior makes EDL risk-aware, fixes confident errors","Deep classifiers go risk-aware with evidential uncertainty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bandit-feedback claim for EDL(pg) assumes the learner can observe the true class label y to compute the risk R_{y i} of the chosen action, even though the stated setting only reveals the cost of the action actually taken.","fun_headline_variants_meta":{"raw":{"variants":["EDL tweak cuts misclassification cost by 55%","Uncertainty-aware evidential classifier cuts costs 55%","Risk-aware EDL with pignistic prior beats cost-sensitive training","Pignistic prior makes EDL risk-aware, fixes confident errors","Deep classifiers go risk-aware with evidential uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3701,"prompt_tokens":944,"completion_tokens":2757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2672}},"tokens_in":560,"tokens_out":2757,"duration_ms":17790,"temperature":1.0,"reasoning_tokens":2672,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:27:31.027076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EDL(pg) in a true bandit environment where only the cost of the chosen class is revealed and the true label is never shown; if the learner cannot compute the gradient in Algorithm 1 (which multiplies R_{y i} by the score-function term), the claimed bandit learning collapses. A simpler check: inspect the update at line 4 of Algorithm 1 and verify that R_{y i} requires the true y to be known; the paper's Section 4.3 admits 'the (x, y) tuple represents each sample x and its true label,' so a reader can settle the matter by seeing whether the observed cost alone determines R_{y i} for every i.","supporting_citations":[],"review_version":1}