Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Large language models can report two orders of uncertainty at once when asked for probability intervals instead of a single confidence score, and the resulting scores better detect both wrong answers and ambiguous questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 18:25 UTC pith:U2PTRFAX

load-bearing objection Good idea, honest experiments, but the semantic validity of the elicited intervals is the crux and remains untested. the 3 major comments →

arxiv 2603.10396 v2 pith:U2PTRFAX submitted 2026-03-11 cs.AI

Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities

Anita Yang , Krikamol Muandet , Michele Caprio , Siu Lun Chau , Masaki Adachi This is my paper
classification cs.AI
keywords imprecise probabilitiesverbalized uncertaintyLLM uncertainty quantificationsecond-order uncertaintyprobability intervalsmaximum mean imprecisionambiguity detectionin-context learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a single confidence number from an LLM conflates two different forms of uncertainty: first-order uncertainty about which answer is right, and second-order uncertainty about how reliable the model's own belief is. Its claim is that asking the model for imprecise probabilities—a lower and upper probability per answer, an ensemble-based credal set, or a possibility score—lets both orders be elicited cheaply and kept apart. On synthetic sequence tasks, interval-based scores stay flat when only first-order noise changes and fall as in-context examples grow, while vanilla confidence fails on both. On real QA benchmarks, the interval and possibility scores improve AUROC for detecting both wrong answers and ambiguous questions, and the model's chosen answer aligns best with the maximin rule over its lower probabilities. A sympathetic reader would take away that verbalized interval confidence is a low-cost substitute for sampling-based uncertainty that reports not just what the model thinks but how sure the model is of its own probabilities.

Core claim

Verbalized uncertainty need not be a point estimate. The paper introduces prompts that elicit, for each candidate answer, a lower probability (the smallest value the model considers plausible) and an upper probability (the largest value it considers defensible), plus two alternatives: a credal set built from disagreement across an ensemble of models or seeded runs, and a possibility function that allows 'none of the above.' These representations are post-processed into a scalar second-order uncertainty score via the maximum mean imprecision (MMI), computed as an interval width for a single answer or an upper bound over the candidate set. The central empirical claim is that this interval-base

What carries the argument

Probability interval: per-answer lower/upper probabilities, where the lower bound is the smallest probability the model calls plausible and the upper bound the largest it calls defensible; the interval width is the second-order uncertainty. Credal set: a collection (convex hull) of precise probability distributions from an ensemble or multiple seeded runs, whose pointwise min/max form lower/upper bounds. Possibility function: a non-additive score on candidate answers, normalized so at least one answer has plausibility 1, allowing 'none of the above' without redistributing mass. MMI: the largest gap between upper and lower probability over any event, approximated by the interval width for a s

Load-bearing premise

The load-bearing premise—flagged as unverifiable in the paper's own limitation section—is that the numeric lower and upper bounds an LLM gives are meaningful approximations of its actual beliefs rather than arbitrary prompt-driven slack, and that the model interprets the probability prompts as intended; without this, the observed separation of first- and second-order uncertainty is an artifact of the prompt.

What would settle it

Collect items with known correctness, elicit lower/upper probabilities, and check whether the empirical frequency of a correct answer across items whose reported lower probability is at or above a threshold t stays above t; systematic violations would show the interval does not bound the model's beliefs. A sharper form: offer the model bets priced inside its stated interval and check whether it accepts a combination that guarantees a loss.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

If this is right

  • Interval-based uncertainty scores can replace sampling-based estimates in error and ambiguity detection: on QA benchmarks, ProbInt, Credal, and Pos rank among the best methods while costing about the same as simple verbalized confidence for ProbInt and Pos.
  • If the interval representation is faithful, downstream systems can use interval width as an abstention or clarification signal, since width is supposed to track reducible second-order uncertainty.
  • Because the transition from point to interval only changes the prompt and adds a cheap post-processing step (MMI), the approach works on closed, API-only models with no access to internal probabilities.
  • The alignment of LLM answer selection with the maximin rule over lower probabilities provides a concrete decision-theoretic criterion for when to distrust a model's stated pick.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the lower/upper probabilities are semantically meaningful, one can validate them independently with a sure-loss test: offering bets at the stated bounds and checking whether the model ever accepts a combination that guarantees a loss. The paper does not run this test.
  • A natural extension beyond QA is to use interval width as an acquisition function in active learning or an early-stopping signal in agentic loops; the cost is low enough that such uses are plausible, but the paper does not demonstrate them.
  • The possibility function's explicit 'none of the above' score suggests a diagnostic for open-vocabulary hallucination: a high score for a non-listed answer would reveal an incomplete candidate set, a test the paper does not perform.
  • The synthetic in-context-learning results imply a sharper prediction: adding in-context examples should narrow the reported intervals monotonically for a given task rule, which could be checked across rule families and would separate genuine belief revision from prompt artifacts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes replacing point-valued verbalized uncertainty in LLMs with imprecise probability representations: lower/upper probability intervals (ProbInt), credal sets (Credal), and possibility functions (Pos), together with a DeFinetti-style betting-price prompt. The authors introduce prompting procedures and use Maximum Mean Imprecision (MMI) to summarize second-order uncertainty. They evaluate on synthetic sequence-transformation tasks with controlled ambiguity noise p and ICL example count m, and on MAQA, AmbigQA, and MMLU-Pro. They report that IP-based scores stay flat under first-order noise, track error as ICL examples grow, improve AUROC for ambiguity and correctness detection, align with the LLM's own choices under maximin, and incur low API cost. The central claim is that this framework elicits and disentangles first- and second-order uncertainty more faithfully than vanilla confidence.

Significance. If the claims hold, the paper would be a useful contribution: it connects a mature decision-theoretic framework (imprecise probabilities) to black-box verbalized LLM uncertainty, and the proposed prompts are simple and deployment-relevant. The DeFinetti prompt is a principled attempt to ground probabilities in betting behavior, and the possibility-function treatment of "none of the above" is a nice idea. The cost comparison is practical. However, the support is uneven in two load-bearing places: the synthetic disentanglement experiment largely restates the data-generation definitions, and the semantic meaningfulness of the elicited intervals is asserted rather than validated. The AUROC improvements are consistent but lack significance testing, and could in principle be produced by any informed monotone width-based score. The paper is worth publishing if the authors add an independent validity check and temper the claims accordingly.

major comments (3)
  1. [§4, Figs. 6–7] The synthetic experiment validates the method against the paper's own definitions: p is labelled first-order and m second-order by construction ("We treat the ambiguity noise p as the first-order because |Y*|>1, and the number of in-context examples m controls the second-order"). Showing that PROBINT is flat in p and decreases in m is consistent with the construction but does not independently confirm that the method separates the two latent quantities in the model's beliefs. A method that responds arbitrarily to prompt wording could produce the same plots. Please add a validation in which the two uncertainty sources are not definitionally tied to the experimental manipulation, e.g., fix p and vary model confidence through a separate channel, or compare against an independent behavioral measure of second-order uncertainty (repeated sampling, clarification ensembles).
  2. [§3.2, Eq. (2), and §6] The central premise is that the lower/upper probabilities elicited by Prompt 2 are meaningful degrees of belief. The paper states this directly in §6: "we assume verbalized uncertainty is approximately rational; although allowing imprecision mitigates, this cannot be fully verified." No independent test anchors the interval endpoints. Because the MMI score is computed from the lower probabilities (Eq. 2: MMI ≤ 1 − Σ p(y)), an arbitrary downward shift in lower probabilities would inflate second-order uncertainty without changing any behavior. The AUROC gains in Table 1 and the maximin alignment in Fig. 14 could be produced by any monotone width-based heuristic. I request an external validity check: e.g., verify that vanilla point probabilities fall inside elicited intervals on average; elicit betting prices for the same events and test coherence; or evaluate interval forecasts with a prop
  3. [§5, Table 1] I cannot assess whether the reported improvements are statistically reliable. The paper gives means and standard deviations over five repetitions, but no paired significance tests or effect sizes. For several GPT-5 cells the proposed methods are within one standard deviation of VANILLA (e.g., MMLU-Pro: PROBINT 0.8617±0.0082 vs VANILLA 0.8587±0.0192; Non-AmbigQA: PROBINT 0.7713±0.0044 is numerically below VANILLA 0.7756±0.0138). The "Avg. Rank" column aggregates six cells and does not account for variability. Please add paired bootstrap or Wilcoxon tests, either across the six dataset-model pairs or at the item level, and identify which comparisons are significant.
minor comments (5)
  1. [Abstract; throughout] Typo "repesenting" in the abstract; the method name is rendered inconsistently as "De Finetti", "DEFINETTI", and "DeFinetti". Please standardize.
  2. [Alg. 1, Alg. 2] Algorithms 1 and 2 have no iteration bound; if the model never satisfies the verifier, the loop does not terminate. Also, the text says the IMPRECISE-PROBABILITY verifier checks both sum lower ≤ 1 and sum upper ≥ 1, but immediately adds that only constraint (i) is verified. Please clarify the actual verification procedure.
  3. [§5.1, Fig. 10] The multiplicative combination DeFinetti × IP is introduced ad hoc ("to achieve scale invariance"). Please provide a justification or a sensitivity analysis over alternative aggregation rules (sum, min, product).
  4. [§5, Task definition] For ambiguous items, correctness is judged only against a single reference y*, so a valid alternative answer in Y* is counted as incorrect. This may conflate ambiguity with incorrectness; please state this explicitly as an evaluation choice and discuss its effect on the reported AUROC.
  5. [Appendix D] In D.2.1 the ground-truth AU/EU decomposition uses cross-entropy and KL proxies from corpus statistics. The paper treats these as "ground truth"; please make clear that these are proxies and that concordance is with the proxy, not with the true latent uncertainty.

Circularity Check

0 steps flagged

No significant circularity; the derivation is empirical and the main assumptions are stated as limitations.

full rationale

The paper's central claims are empirical rather than first-principles derivations. PROBINT's second-order score is computed directly from LLM-verbalized lower probabilities after a coherence verifier; Eq. (2) is an algebraic identity for any lower probability, not a fitted quantity. The synthetic experiments in §4 stipulate p as first-order and m as second-order, but the observed flatness under p and monotonicity under m are not mathematically forced by the score definition; they are behavioral results of the model and therefore constitute genuine empirical tests. Real-world evaluations use external labels (MAQA, AmbigQA, MMLU-Pro) and a corpus-based KL proxy, so the reported AUROC improvements are not constructed from the method's own outputs. The MMI metric is cited from prior work by overlapping authors (Chau et al. 2025, 2026), but the paper states the relevant formulas directly and the upper bound is elementary; the imprecision interpretation is standard in imprecise-probability literature. Section 6 explicitly admits the load-bearing assumption that verbalized uncertainty is approximately rational and cannot be fully verified; that is a limitation on semantic validity of the elicited intervals, not a circular derivation. No step reduces to its inputs by construction, and no fitted parameter is relabeled as a prediction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or conceptual entities are postulated (no new particles, forces, or conserved quantities). The load-bearing assumptions are: (1) verbalized rationality, (2) prompt-semantic fidelity, (3) de Finetti coherence, (4) the KL-proxy ground truth, and (5) the synthetic split of uncertainty sources. Free parameters are all experimental-design or post-processing choices rather than fitted constants.

free parameters (3)
  • Number of CREDAL ensemble members / samples = 5
    CREDAL uses five samples per query (§5 base setup); the choice is not justified as an operating point and directly sets the credal-set width and hence the MMI score.
  • Multiplicative combination DeFinetti × IP (Fig. 10)
    Introduced ad hoc in §5.1 ('we therefore combine them multiplicatively to achieve scale invariance'); no derivation from utility theory or error analysis.
  • MMI upper-bound substitution = 1 − Σ p(y)
    Exact MMI is exponential in |Y|; the paper substitutes the upper bound (Eq. 2, Appendix B), a modeling choice that changes the semantics of the reported score.
axioms (5)
  • domain assumption Verbalized uncertainty is approximately rational
    Stated in §6 limitations: 'we assume verbalized uncertainty is approximately rational; although allowing imprecision mitigates, this cannot be fully verified.' Load-bearing for all four methods.
  • domain assumption The model correctly interprets the prompting semantics (lower = smallest plausible, upper = largest defensible)
    Stated in §6; if the model cannot distinguish lower from upper probability, the interval width is meaningless.
  • standard math De Finetti coherence: fair betting prices must satisfy the probability axioms
    Invoked in §3.1 to justify the verifier; the paper's assertion that 'under this assumption, additivity follows automatically' is compressed and unproven for arbitrary gambles.
  • domain assumption KL(p⋆ ∥ p̂) is a valid proxy for ground-truth second-order uncertainty
    Appendix D.2, borrowed from Tomov et al. 2025; the proxy is computed from the same model's predictive distribution, so concordance with it can reflect internal consistency rather than external truth.
  • ad hoc to paper In the synthetic experiment, ambiguity noise p is first-order and #ICL examples m is second-order
    §4 'Controlling first-/second-order uncertainty'; this construction is what the disentanglement experiment then 'demonstrates' — the conclusion is partially baked into the design.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities." pith.science (2026). https://pith.science/paper/U2PTRFAX

@misc{pith2026260310396,
  author       = {Pith},
  title        = {Pith review of: Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2PTRFAX}},
  note         = {Machine review of arXiv:2603.10396}
}
Share X LinkedIn Reddit HN
read the original abstract

Despite the growing demand for eliciting uncertainty from large language models (LLMs), empirical evidence suggests that LLM behavior is not always adequately captured by the elicitation techniques developed under the classical probabilistic uncertainty framework. This mismatch leads to systematic failure modes, particularly in settings that involve ambiguous question-answering, in-context learning, and self-reflection. To address this, we propose novel prompt-based uncertainty elicitation techniques grounded in \emph{imprecise probabilities}, a principled framework for repesenting and eliciting higher-order uncertainty. Here, first-order uncertainty captures uncertainty over possible responses to a prompt, while second-order uncertainty (uncertainty about uncertainty) quantifies indeterminacy in the underlying probability model itself. We introduce general-purpose prompting and post-processing procedures to directly elicit and quantify both orders of uncertainty, and demonstrate their effectiveness across diverse settings. Our approach enables more faithful uncertainty reporting from LLMs, improving credibility and supporting downstream decision-making.

Figures

Figures reproduced from arXiv: 2603.10396 by Anita Yang, Krikamol Muandet, Masaki Adachi, Michele Caprio, Siu Lun Chau.

Figure 1
Figure 1. Figure 1: Collection of failure modes in prior verbalized [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our imprecise probabilities–based approach. (a) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Learnable vs. noisy transforms. et al., 2024, Stengel-Eskin et al., 2024], post-hoc adjust￾ments such as normalization [Wang and Stengel-Eskin, 2025], and language-based uncertainty expressions [Kirch￾hof et al., 2025a]. However, prior work typically assumes no question ambiguity, whereas our framework explicitly separates first-/second-order uncertainty. Uncertainty disentanglement. Disentangling first-/s… view at source ↗
Figure 8
Figure 8. Figure 8: Credal-set from LLM ensemble: model disagree [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: AUROC for ambiguity detection. Baselines for ambiguity. (i) SEMANTIC ENTROPY [Far￾quhar et al., 2024], entropy over semantically clustered sam￾ples; (ii) ASK4CONF-D [Hou et al., 2024], directly eliciting the probability that a question is ambiguous; and (iii) MI CLARIFICATIONS [Hou et al., 2024]. Baselines for correctness. We consider verbalized and sampling-based methods. Verbalized: (i) VANILLA [Tian et … view at source ↗
Figure 12
Figure 12. Figure 12: PROBINT more closely matches the KL metric. AmbigQA* MAQA* AmbigQA* MAQA* 0.4 0.5 0.6 0.7 Conc. Index GPT-5 gemini-2.5-pro Is true prob. Label prob. Vanilla Top-4 CoT DiNCO MI-Clar. ProbInt Credal Pos [PITH_FULL_IMAGE:figures/full_fig_p008_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: , our IP-based methods demonstrate consistently strong and robust correlations. 5.2 EXPLAINING OWN DECISION Previously, we evaluated correctness against the ground￾truth y ⋆ . We now instead compare against the model’s own prediction yˆ. In classical multi-class classification, predic￾tion is obtained via yˆprecise = arg maxy∈Y pˆ(y = correct), which we refer to as PRECISE PROB. This rule is fully algorit… view at source ↗
Figure 14
Figure 14. Figure 14: , the maximin rule exhibits the highest alignment with the LLM’s predictions. 6 CONCLUSION AND LIMITATION We propose IP-based methods for higher-order uncertainty elicitation. Across tasks and models, our approach im￾proves elicitation accuracy and internal consistency while remaining cost-efficient. Combined with MMI-based post￾processing, IP provides a principled framework for assess￾ing LLM credibility… view at source ↗
Figure 18
Figure 18. Figure 18: MMI measures the largest gap between up [PITH_FULL_IMAGE:figures/full_fig_p014_18.png] view at source ↗
Figure 20
Figure 20. Figure 20: First- and second-order uncertainty can be computed either for the (i) correctness of a specific answer [PITH_FULL_IMAGE:figures/full_fig_p016_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Approximations of second-order uncertainty with fixed first-order noise (lower-case probability [PITH_FULL_IMAGE:figures/full_fig_p017_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: DEFINETTI best align with the proxy for first-order uncertainty [PITH_FULL_IMAGE:figures/full_fig_p017_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputs

    stat.ML 2026-05 unverdicted novelty 8.0

    A certificate-based regret analysis framework for guided-diffusion black-box optimization is introduced, with mass lift as the central quantity explaining convergence from pretrained generators.

  2. Open-Ended Task Discovery via Bayesian Optimization

    cs.AI 2026-05 unverdicted novelty 6.0

    Generate-Select-Refine is an open-ended Bayesian optimization method that generates tasks and concentrates evaluations on the best one with only logarithmic regret overhead relative to standard single-task optimization.

Reference graph

Works this paper leans on

16 extracted references · 7 linked inside Pith · cited by 2 Pith papers

  1. [1]

    <answer one as concise text>

  2. [2]

    URL https://openreview.net/forum? id=WOFspd4lq5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems, 2020. Siu Lun Chau, Michele Caprio, and Krikamol Muandet. Inte- gral imprecise probability metrics. InAdvances in Neural Information Pr...

    arXiv 2020

  3. [5]

    Ryan Liu, Jiayi Geng, Joshua Peterson, Ilia Sucholutsky, and Thomas L

    URL https://openreview.net/forum? id=Oq1b1DnUOP. Ryan Liu, Jiayi Geng, Joshua Peterson, Ilia Sucholutsky, and Thomas L. Griffiths. Large language models as- sume people are more rational than we really are. In International Conference on Learning Representations,

  4. [6]

    William Lugoloobi, Thomas Foster, William Bankes, and Chris Russell

    URL https://openreview.net/forum? id=dAeET8gxqg. William Lugoloobi, Thomas Foster, William Bankes, and Chris Russell. LLMs encode their failures: Predict- ing success from pre-generation activations.arXiv preprint arXiv:2602.09924, 2026. URL https:// arxiv.org/abs/2602.09924. Sravanthi Machcha, Sushrita Yerra, Sharmin Sultana, Hong Yu, and Zonghai Yao. Do...

    Pith/arXiv arXiv 2026

  5. [7]

    Tommaso Mencattini, Robert Adrian Minut, Donato Crisos- tomi, Andrea Santilli, and Emanuele Rodolà

    URL https://openreview.net/forum? id=4ub9gpx9xw. Tommaso Mencattini, Robert Adrian Minut, Donato Crisos- tomi, Andrea Santilli, and Emanuele Rodolà. MERGE 3: Efficient evolutionary merging on consumer-grade GPUs. InInternational Conference on Machine Learning,

  6. [8]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer

    URL https://proceedings.mlr.press/ v267/mencattini25a.html. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open- domain questions. InProceedings of the Conference on Empirical Methods in Natural Language Processing,

  7. [10]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D

    URL https://openreview.net/forum? id=qAml3FpfhG. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural In- formation Processing Systems, 2023. URL https: //openreview.net/forum?id=HPuSIXJaa9. Herbert Robbins...

    2023

  8. [12]

    Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu

    URL https://openreview.net/forum? id=iLUcsecZJp. Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. A survey of uncertainty estimation methods on large lan- guage models.arXiv preprint arXiv:2503.00172, 2025. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit Bayesian inference. InInternatio...

    Pith/arXiv arXiv 2025

  9. [13]

    emnlp-main.343/

    URL https://aclanthology.org/2024. emnlp-main.343/. Khurram Yamin, Jingjing Tang, Santiago Cortes-Gomez, Amit Sharma, Eric Horvitz, and Bryan Wilder. Do LLMs act like rational agents? Measuring belief coher- ence in probabilistic decision making.arXiv preprint arXiv:2602.06286, 2026. URL https://arxiv. org/abs/2602.06286. Yongjin Yang, Haneul Yoo, and Hwa...

    Pith/arXiv arXiv 2024

  10. [14]

    none of the above

    URL https://aclanthology.org/2025. findings-naacl.325/. Chengshuai Zhao, Zhen Tan, Pingchuan Ma, Dawei Li, Bohan Jiang, Yancheng Wang, Yingzhen Yang, and Huan Liu. Is chain-of-thought reasoning of LLMs a mirage? A data distribution lens.arXiv preprint arXiv:2508.01191, 2025. URL https://arxiv. org/abs/2508.01191. Verbalizing LLM’s Higher-order Uncertainty...

    Pith/arXiv arXiv 2025

  11. [16]

    the sampled answer lies in A

    <answer two as concise text> ... B MAXIMUM MEAN IMPRECISION The exact MMI metric for measuring EU [Chau et al., 2025] under total variation is MMITV(P ) := sup A∈FY P(A)−P (A) ≤1− X y∈Y P ({y}),(3) where FY is the σ-algebra over the candidate answersY (in our discrete setting,FY = 2Y). The lower and upper probabilities P (A) and P(A) bound the probability...

    2025

  12. [2020]

    emnlp-main.466/

    URL https://aclanthology.org/2020. emnlp-main.466/. Krikamol Muandet. Impossibility of collective intelligence. arXiv preprint arXiv:2206.02786, 2022. URL https: //arxiv.org/abs/2206.02786. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tiny- Benchmarks: evaluating LLMs with fewer examples. InInternational Co...

    Pith/arXiv arXiv 2020

  13. [2023]

    Cedric A

    URL https://openreview.net/forum? id=98p5x51L5af. Cedric A. B. Smith. Consistency in statistical inference and decision.Journal of the Royal Statistical Society Series B: Statistical Methodology, 23(1):1–25, 1961. Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. La- cie: Listener-aware finetuning for calibration in large lan- guage models. InAdvances in...

    Pith/arXiv arXiv 1961

  14. [2024]

    eacl-long.143/

    URL https://aclanthology.org/2024. eacl-long.143/. John C. Harsanyi. Bayesian decision theory and utilitarian ethics.The American Economic Review, 68(2):223–228, 1978. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decomposing uncertainty for large language models through input clarification ensem- bling. InInternational C...

    Pith/arXiv arXiv 2024

  15. [2025]

    findings-emnlp.1367/

    URL https://aclanthology.org/2025. findings-emnlp.1367/. Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman. Distinguishing the knowable from the unknowable with language models. InInternational Conference on Machine Learning, 2024. Thomas Augustin, Frank P. A. Coolen, Gert de Cooman, and Matthias C. M. Troffaes.Introduction to Imp...

    2025

  16. [8856]

    Chen Ling, Xujiang Zhao, Wei Cheng, Yanchi Liu, Yiyou Sun, et al

    URL https://openreview.net/forum? id=8s8K2UZGTZ. Chen Ling, Xujiang Zhao, Wei Cheng, Yanchi Liu, Yiyou Sun, et al. Uncertainty decomposition and quantification for in-context learning of large language models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.