ECUAS$_n$: A family of metrics for principled evaluation of uncertainty-augmented systems

Erik Ernst; Lautaro Estienne; Luciana Ferrer; Mat\'ias Vera; Pablo Piantanida

arxiv: 2605.20490 · v3 · pith:IP7H37WAnew · submitted 2026-05-19 · 💻 cs.AI · cs.LG

ECUAS_n: A family of metrics for principled evaluation of uncertainty-augmented systems

Lautaro Estienne , Erik Ernst , Mat\'ias Vera , Pablo Piantanida , Luciana Ferrer This is my paper

Pith reviewed 2026-05-22 08:51 UTC · model grok-4.3

classification 💻 cs.AI cs.LG

keywords uncertainty quantificationproper scoring rulesevaluation metricsuncertainty-augmented systemsdecision making under uncertaintyclassificationquestion answering

0 comments

The pith

ECUAS_n metrics evaluate uncertainty-augmented systems as proper scoring rules that balance prediction errors and uncertainty quality via one tunable parameter.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current evaluation of uncertainty-augmented systems often splits predictions and uncertainties into separate scores, fixes rejection costs arbitrarily, or integrates over coverage-risk curves. The paper argues these approaches fail to assess the system's overall value for downstream decisions where uncertainty guides accept-or-reject choices. It introduces the ECUAS_n family of metrics, each a proper scoring rule for the task at hand, with n setting the relative penalty for wrong predictions versus imperfect uncertainty estimates. A sympathetic reader would care because high-stakes applications need one number that directly reflects decision utility rather than a collection of proxy scores. The authors support the claim with theoretical properties of proper scoring rules and experiments on classification and generation datasets including a human-annotated TriviaQA subset.

Core claim

The ECUAS_n family of metrics, formulated as proper scoring rules for the task of interest, provides a more adequate assessment of the overall performance of uncertainty-augmented systems for decision making under uncertainty than current approaches using separate metrics, fixed rejection costs, or coverage-risk curves. The parameter n controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the needs of the use-case.

What carries the argument

The ECUAS_n metric, a parameterized proper scoring rule that combines prediction accuracy and uncertainty quality into one score, with n controlling the relative cost of errors versus bad uncertainty estimates.

If this is right

UA systems can be ranked and selected for a concrete use-case simply by picking the n that matches its cost structure.
Differences in system quality that are invisible to separate accuracy and uncertainty metrics become visible in the combined score.
Training or post-processing choices can be guided by direct optimization toward the metric that will be used at deployment.
Comparisons across papers become more reproducible when authors report ECUAS_n at the n values relevant to common decision settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If widely adopted, ECUAS_n could shift model development away from maximizing accuracy alone toward explicitly optimizing the uncertainty that supports downstream decisions.
The approach could be extended to regression or structured prediction tasks by redefining the base proper scoring rule while keeping the same n-controlled trade-off structure.
One could measure whether models trained to minimize ECUAS_n at a target n actually improve real-world utility on a held-out decision policy compared with models trained on standard losses.

Load-bearing premise

A single tunable parameter n can meaningfully capture application-specific cost trade-offs between incorrect predictions and imperfect uncertainties without requiring additional validation or introducing new selection biases in practice.

What would settle it

Run a decision task with known, application-specific rejection costs on held-out data; if the system ranked best by ECUAS_n for the matching n does not produce the highest expected utility when users reject according to its uncertainty scores, the claim of superior adequacy would be falsified.

Figures

Figures reproduced from arXiv: 2605.20490 by Erik Ernst, Lautaro Estienne, Luciana Ferrer, Mat\'ias Vera, Pablo Piantanida.

**Figure 1.** Figure 1: C ∗ n as a function of the confidence qe, when candidate decisions are correct (solid lines) and incorrect (dashed lines), for different values of n, the parameter in w, and K, the number of classes. 3 Application of ECUAS to generative systems An important family of UA systems is that based on generative models [30, 87, 58]. To use the ECUAS metrics in this scenario, we need to adapt the definition of C˜.… view at source ↗

**Figure 2.** Figure 2: ECUASn values when temperature scaling is applied to the calibrated version of q and the candidate answer is obtained by sampling from the resulting distribution. Our evaluation spans multiple state-of-the-art small LLMs, Qwen 3.5 (4B and 9B) [75], GLM-4.6VFlash [76], Ministral-3-8B-Instruct-2512 [56], as well as larger models from the Gemini 2.5 family (Flash Lite, Flash and Pro) [12]. We evaluate these … view at source ↗

read the original abstract

In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems -- i.e., systems that output both predictions and uncertainty scores -- are currently being assessed in the literature in a variety of ways, using separate metrics to evaluate the predictions and the uncertainty scores, setting a cost function with a fixed rejection cost or integrating over a coverage-risk curve. We argue that these evaluation approaches are inadequate for assessing overall performance of the UA system for decision making under uncertainty and propose a novel family of metrics, ECUAS$_n$, formulated as proper scoring rules for the task of interest. The parameter $n$ controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the needs of the use-case. We demonstrate the advantages of the ECUAS$_n$ metrics both theoretically and empirically, through experiments on diverse classification and generation datasets, including a manually annotated subset of TriviaQA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ECUAS_n gives a single proper scoring rule for uncertainty-augmented systems with one tunable n, but choosing that n may still need validation data and could reintroduce the tuning issues the paper wants to avoid.

read the letter

The main thing to know is that this paper introduces ECUAS_n, a family of proper scoring rules that score an uncertainty-augmented system on both its predictions and its uncertainty estimates in one number, with the parameter n setting the relative cost between wrong answers and bad uncertainty reports. The authors argue this is better than running separate metrics, fixing a rejection cost, or integrating over coverage-risk curves, and they show the idea on classification and generation tasks including a hand-labeled TriviaQA subset. What the work does well is name a real practical gap: when a system must support accept/reject decisions, splitting the evaluation misses how prediction quality and uncertainty quality interact for the actual downstream cost. Framing the whole thing as a proper scoring rule is a clean move because those rules have known calibration properties, and the experiments across several datasets give at least initial evidence that the metric behaves sensibly. The soft spot is exactly the one the stress-test note flags. If setting n for a new use case still requires held-out data, expert tuning, or knowledge of the application costs, then the selection step itself can introduce bias or extra validation work, which undercuts the claimed advantage over fixed-cost or coverage-based methods. The abstract does not show a default or data-free way to pick n, so the practical edge is not yet obvious. I would also want to see the explicit derivations that establish the proper scoring property for this specific UA task. This paper is for people who evaluate or deploy systems that output both a prediction and a score in high-stakes settings. A reader who already works with proper scoring rules or selective prediction will get the most out of the framework. It deserves a serious referee because the core proposal is coherent and the authors engage the existing literature rather than ignoring it. I would send it to review rather than desk reject.

Referee Report

2 major / 2 minor

Summary. The paper proposes the ECUAS_n family of metrics for evaluating uncertainty-augmented (UA) systems that output both predictions and uncertainty scores. It argues that current practices—separate metrics for predictions and uncertainties, fixed rejection costs, or coverage-risk curves—are inadequate for assessing overall performance in decision-making under uncertainty. ECUAS_n is formulated as proper scoring rules with a single tunable parameter n that controls the trade-off between the cost of incorrect predictions and imperfect uncertainties according to use-case needs. Theoretical advantages and empirical results are presented on classification and generation tasks, including a manually annotated subset of TriviaQA.

Significance. If the proper-scoring-rule formulation holds and the empirical comparisons demonstrate clear, bias-free advantages, the work could establish a more unified and application-adaptable standard for evaluating UA systems in high-stakes settings, reducing reliance on fragmented or arbitrarily parameterized evaluation protocols.

major comments (2)

[§3] §3 (theoretical formulation): the claim that ECUAS_n constitutes a proper scoring rule for the joint prediction-uncertainty task requires an explicit derivation showing that the expected score is minimized precisely when both the prediction is correct and the uncertainty is well-calibrated; without this, the asserted superiority over separate metrics or fixed-cost approaches remains unsubstantiated.
[§5] §5 (empirical evaluation, TriviaQA experiments): the procedure for selecting or tuning n is not shown to avoid the very selection bias the paper criticizes in fixed-rejection-cost methods; if n is chosen on held-out data or expert knowledge of downstream costs, the metric loses its claimed advantage of being a single, principled scalar.

minor comments (2)

[Abstract] The abstract and introduction could more explicitly list all datasets used beyond the TriviaQA subset to allow immediate assessment of diversity.
[§3] Notation for the scoring rule should be introduced with a single running example before the general n-parameterized form to improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. We address each of the major comments in detail below and outline the revisions we will make to improve the clarity and rigor of the paper.

read point-by-point responses

Referee: [§3] §3 (theoretical formulation): the claim that ECUAS_n constitutes a proper scoring rule for the joint prediction-uncertainty task requires an explicit derivation showing that the expected score is minimized precisely when both the prediction is correct and the uncertainty is well-calibrated; without this, the asserted superiority over separate metrics or fixed-cost approaches remains unsubstantiated.

Authors: We acknowledge that the current manuscript would benefit from a more explicit derivation of the proper scoring rule property. In the revised version, we will expand Section 3 to include a step-by-step derivation proving that the expected value of ECUAS_n is minimized if and only if the prediction is correct and the uncertainty is perfectly calibrated to the true posterior. This will directly address the concern and provide a stronger theoretical basis for the metric's advantages. revision: yes
Referee: [§5] §5 (empirical evaluation, TriviaQA experiments): the procedure for selecting or tuning n is not shown to avoid the very selection bias the paper criticizes in fixed-rejection-cost methods; if n is chosen on held-out data or expert knowledge of downstream costs, the metric loses its claimed advantage of being a single, principled scalar.

Authors: We appreciate this point and agree that the selection of n must be handled carefully to maintain the principled nature of the metric. Unlike fixed-rejection-cost methods where the cost parameter is often chosen arbitrarily or tuned on data, n in ECUAS_n is meant to reflect the relative costs in the specific use-case, which can be determined from domain expertise or cost-benefit analysis without reference to the evaluation data. To clarify this, we will add a subsection in the revised Section 5 discussing guidelines for choosing n based on application requirements, along with empirical sensitivity analyses showing results for a range of n values on the TriviaQA experiments. This preserves the advantage of a single scalar while making the choice transparent and use-case driven. revision: yes

Circularity Check

0 steps flagged

ECUAS_n defined as proper scoring rules with independent theoretical and empirical support

full rationale

The paper formulates ECUAS_n directly as a family of proper scoring rules for uncertainty-augmented decision making, with n as an explicit tunable parameter for cost trade-offs. It contrasts this with separate metrics or fixed-cost approaches and validates via theoretical properties plus experiments on external datasets (e.g., TriviaQA). No derivation step reduces a claimed prediction or uniqueness result to a fitted input, self-citation chain, or ansatz imported from prior work by the same authors. The central claim rests on the proper-scoring-rule construction and external empirical checks rather than self-referential definitions.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central addition is the new metric family with the tunable n; the work rests on the domain assumption that proper scoring rules are suitable for this joint evaluation task and on standard ML evaluation practices.

free parameters (1)

n
Controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the use-case.

axioms (1)

domain assumption Proper scoring rules are the appropriate framework for assessing overall performance of uncertainty-augmented systems in decision making under uncertainty.
Invoked to justify why ECUAS_n is principled compared to prior separate or fixed-cost methods.

pith-pipeline@v0.9.0 · 5730 in / 1236 out tokens · 37821 ms · 2026-05-22T08:51:28.927520+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose a novel family of metrics, ECUAS_n, formulated as proper scoring rules... The parameter n controls the trade-off between the cost of incorrect predictions and imperfect uncertainties
IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean J_uniquely_calibrated_via_higher_derivative unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

C^*_n(y,q) obtained by integrating w_n(γ)C^*_γ over γ

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.