The reviewed record of science sign in
Pith

arxiv: 2607.06327 · v1 · pith:GD2EAQHV · submitted 2026-07-07 · cs.CL · cs.AI

Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

Andrea Alfarano , Andrea Bacciu , Saab Mansour , Amin Mantrach , Marcello Federico This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 09:34 UTCglm-5.2pith:GD2EAQHVrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords uncertainty estimationmultilingual NLPlarge language modelslow-resource languageschain-of-thought reasoningself-verbalized confidenceselective predictionmodel scaling
0
0 comments X

The pith

English reasoning closes the uncertainty gap for low-resource languages

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents the first large-scale evaluation of uncertainty estimation (UE) methods for large language models across 22 languages, spanning high-, mid-, and low-resource settings. The central discovery is that prompting models to reason in English while keeping questions in their original low-resource language substantially improves uncertainty estimation performance, effectively closing the gap between low- and high-resource languages. This finding implies that the reliability bottleneck in multilingual settings lies in generation rather than comprehension: models understand low-resource questions adequately but struggle to generate coherent reasoning in those languages, which degrades the uncertainty signal. The paper also establishes that the optimal UE method depends on model scale—probability-based open-box methods work best for smaller models, while self-verbalized uncertainty becomes superior at larger scales (235B+), suggesting that meta-cognitive self-assessment abilities emerge with scale. Additionally, the study finds that sampling-based consistency methods (like semantic entropy) fail on low-resource languages because models produce highly variable reasoning regardless of correctness, leaving no usable diversity signal. The evaluation covers 9 UE methods, 9 models from 270M to 235B parameters, and uses two human-curated MCQA datasets with exact-match scoring to avoid the noise introduced by LLM-as-judge or embedding-based correctness proxies.

Core claim

The paper's central claim is that generation language matters more than question language for uncertainty estimation: when models reason in English instead of the question's target language, UE performance for low-resource languages (Yoruba, Swahili, Nepali) rises to match that of high-resource languages (Germanic, Romance). This is demonstrated by AUROC improvements from 0.58 to 0.68 for Yoruba and 0.58 to 0.64 for Swahili—statistically significant gains with non-overlapping confidence intervals. The mechanism is an asymmetry between comprehension and generation: models can comprehend low-resource questions but cannot generate reliable reasoning traces in those languages, and shifting the推理

What carries the argument

The experimental design uses MCQA datasets (Global-MMLU, MMLU-ProX) with exact-match correctness labels, elicits long-form reasoning (~150 words), applies UE methods to the reasoning text rather than the answer, and measures discrimination via AUROC. The cross-lingual manipulation—question in target language, reasoning in English—is the key apparatus that isolates comprehension from generation as the bottleneck.

If this is right

  • For deployed multilingual LLM systems, prompting models to reason in English while accepting questions in any language is a practical, low-cost intervention that substantially improves the reliability of confidence signals for low-resource languages.
  • The finding that self-verbalized uncertainty surpasses probability-based methods only at large scale (235B) suggests that smaller or distilled models deployed in production should rely on token-level probability methods, not self-reported confidence.
  • The failure of sampling-based consistency methods on low-resource languages implies that systems using semantic entropy or graph-based dispersion for hallucination detection will be unreliable for languages like Yoruba or Bengali, creating a blind spot in multilingual safety.
  • Threshold calibration strategies transfer across datasets and languages: English-only calibration already halves error rates, making it a viable starting point for resource-constrained deployments without requiring per-language validation data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If comprehension is intact but generation is the bottleneck, then other generation-adjacent tasks—such as code-switching output, multilingual summarization, or cross-lingual retrieval-augmented generation—may also benefit from routing the model's internal reasoning through English while producing final outputs in the target language.
  • The correlation between English reasoning and improved task accuracy (not just UE) suggests that the comprehension-generation asymmetry may be a general property of how current LLMs handle low-resource languages, not specific to uncertainty estimation. This would imply that multilingual benchmarks measuring only final-answer accuracy may underestimate models' true understanding of low-resource inp
  • The emergence of self-verbalized uncertainty at 235B but not at smaller scales raises the question of whether this is a genuine scale-driven capability or an artifact of training pipeline differences (the paper notes smaller Qwen3 variants are distilled from the 235B model). Disentangling these would require comparing models of similar size but different training procedures.

Load-bearing premise

The paper extracts uncertainty signals from the model's reasoning trace (~150 words) rather than from the final answer, assuming that uncertainty in reasoning text is a meaningful proxy for answer-level confidence. If models can reason fluently but incorrectly, or if reasoning quality is decoupled from answer confidence, the AUROC numbers would reflect reasoning-text uncertainty rather than the practically relevant question of when the final answer is wrong. The paper doesnot

What would settle it

If one measured UE performance by applying the same methods directly to the model's answer-choice tokens rather than to the reasoning trace, and found that the cross-lingual reasoning-language effect disappears or reverses, then the paper's central claim about the comprehension-generation asymmetry would be undermined.

Figures

Figures reproduced from arXiv: 2607.06327 by Amin Mantrach, Andrea Alfarano, Andrea Bacciu, Marcello Federico, Saab Mansour.

Figure 1
Figure 1. Figure 1: Evaluation pipeline. Unlike prior work that estimates uncertainty from predicted labels in English-only [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scaling study on MCQA accuracy on both datasets (mean ± 95% CI) over 22 languages. Similar trends are observed in Qwen models. the Q&A accuracy in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-language AUROC for nine UE methods across 22 languages on the dense Gemma3-27B and MoE [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of model scale on UE performance (Qwen3 family). Shaded regions denote 95% CIs. Dashed line: random baseline [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Uncertainty estimation performance (AUROC [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of reasoning language on UE quality across 22 languages. Solid bars represent reasoning in target [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Effect of cross-lingual answer options on UE quality. Solid bars: multilingual setting; hatched bars: [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Uncertainty estimation performance for Qwen3-30B with quantization to 8-bit and 4-bit. Quan￾tization has minimal impact on both accuracy and uncer￾tainty estimation at this model scale. (± indicates 95% CI) [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effect of model scale on UE performance (Qwen3 family). Shaded regions denote 95% CIs. Self Verbalized improves with statistical significance at 235B. Dashed line: random baseline. This figure complements the [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Effect of model scale on ECE. (Mean across [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
read the original abstract

Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 8 minor

Summary. This paper presents a large-scale empirical study of uncertainty estimation (UE) methods for LLMs across 22 languages, spanning high- to low-resource settings. Using two human-curated MCQA datasets (Global-MMLU, MMLU-ProX), the authors evaluate nine UE methods (open-box and closed-box) across nine models ranging from 270M to 235B parameters. The experimental design is careful: correctness is assessed via exact-match against MCQA labels (avoiding LLM-as-judge or embedding-based proxies), 95% confidence intervals are reported throughout, label distribution is checked for positional bias, and quantization effects are analyzed. The paper addresses five research questions: (RQ1) which UE methods are robust across languages, (RQ2) how model scale affects UE, (RQ3) how the reasoning language affects UE quality, (RQ4) robustness under cross-lingual answer settings, and (RQ5) threshold selection for selective prediction. The central finding is that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance and closes the gap with high-resource languages, which the authors interpret as evidence that the reliability bottleneck lies in generation rather than comprehension.

Significance. The paper makes a solid contribution to the multilingual NLP and LLM reliability communities. Its strengths include: (1) the use of human-curated MCQA datasets with exact-match correctness, which avoids the evaluation noise introduced by LLM-as-judge and embedding-based metrics—a principled methodological choice; (2) broad coverage of 22 languages and 9 models, making this the largest multilingual UE study to date; (3) the identification of a practically actionable finding (English reasoning improves UE for low-resource languages) that is falsifiable and directly deployable; (4) the scale-aware UE method recommendation (open-box at small scale, self-verbalized at large scale); and (5) the threshold calibration analysis with three strategies (t_EN, t_GLOBAL, t_LANG) providing concrete deployment guidance. The quantization impact analysis (Appendix G) and label distribution check (Appendix J) add rigor. The finding that sampling-based consistency methods fail on low-resource languages due to collapsed diversity signals is well-diagnosed and valuable.

major comments (3)
  1. Section 4, RQ3 (Figure 6): The central interpretive claim—that English reasoning closes the UE gap because 'comprehension is largely intact' and 'the reliability bottleneck lies in generation rather than understanding'—is confounded by simultaneous accuracy gains. Table 2 (Appendix F) shows English reasoning also improves Q&A accuracy substantially (Yoruba +17.6%, Swahili +10.5%, average +5.4% relative). When accuracy changes between conditions, the set of correct/incorrect instances changes, and AUROC may improve mechanically because the remaining errors are more systematically detectable (e.g., the model fails only on genuinely hard questions where it is also more uncertain), rather than because the uncertainty signal extracted from reasoning text is genuinely better. The paper does not control for this accuracy change. An equally plausible interpretation is that English reasoning simp
  2. Section 4, RQ3/RQ4 (Figures 6–7): AUROC is reported averaged across all nine UE methods, without per-method breakdowns. This is problematic because RQ1 (Figure 3) already establishes that methods vary enormously in performance (e.g., Self Verbalized at 0.72 vs. Semantic Entropy at ~0.50 on Yoruba). Averaging across methods where some are at random-chance level could mask or distort the actual effect of reasoning language. For instance, if English reasoning primarily rescues the sampling-based methods (which fail due to high diversity in low-resource languages), the aggregate improvement would look different than if it uniformly benefits all methods. Per-method or at least per-category (open-box vs. closed-box vs. sampling-based) breakdowns for the English-reasoning comparison are needed to support the claim that the improvement is a general phenomenon rather than driven by specific weak
  3. Section 3: The paper applies UE methods to the model's reasoning trace rather than to the final answer, but does not validate this proxy against direct answer-level UE. The paper states 'we apply UE methods to the LLM's reasoning text.' If reasoning-text uncertainty does not correlate well with answer correctness (e.g., because models can reason fluently but incorrectly), the AUROC numbers may not reflect what practitioners need. While the proxy concern is partially mitigated by the fact that both conditions in RQ3 use the same proxy (so proxy bias cancels in the comparison), the absolute AUROC numbers reported in RQ1/RQ2 are still potentially affected. A brief validation—e.g., comparing reasoning-trace UE against answer-choice probability UE on a subset—would strengthen the framework's foundational assumption.
minor comments (8)
  1. Table 1 caption: 'Imapct' should be 'Impact.'
  2. Section 4, RQ2: The paper notes that smaller Qwen3 variants are distilled from the 235B model, which 'may also reflect that self-verbalization capabilities are sensitive to training procedures.' This is an important caveat that somewhat undercuts the clean scaling narrative. Consider moving this caveat earlier or acknowledging it more prominently in the conclusion.
  3. Figure 3: The figure is dense with 22 languages × 9 methods × 2 models. Consider adding a summary table or a simplified version showing only method averages with CIs for the main text, with the full per-language figure in an appendix.
  4. Section 4, RQ5: The abstract states 'multilingual calibration reduces errors by up to 77%' but Table 1 shows t_GLOBAL achieves 56% error reduction and t_LANG achieves 60%. The 77% figure is not clearly traceable to the reported results. Please reconcile.
  5. Appendix F, Table 2: The table lists 'UK' as a language code, but the language list in Appendix C does not include Ukrainian. Please clarify or correct.
  6. Section 2.2: The Self Verbalized uncertainty description references 'Tian et al., 2023a' and 'Tian et al., 2023b' and 'Tian et al., 2023c' which appear to be the same paper cited differently. Please consolidate.
  7. The paper mentions 'Claude 4.5 Sonnet' but the reference is to 'Claude Sonnet 4.5' (Anthropic, 2025). The naming is inconsistent across the paper; please standardize.
  8. Appendix K is referenced in the main text but the ECE table (Table 9) is mentioned but not fully shown in the provided manuscript. Please ensure it is included.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a thorough and constructive report. The referee raises three major comments concerning (1) a confound between accuracy gains and AUROC improvements in RQ3, (2) the lack of per-method breakdowns in the English-reasoning comparison, and (3) the absence of validation for the reasoning-trace UE proxy against direct answer-level UE. We address each below. We agree with comments 1 and 2 and will revise the manuscript accordingly; comment 3 we address partially with existing evidence and a new supplementary analysis.

read point-by-point responses

  1. Referee: Section 4, RQ3 (Figure 6): The central interpretive claim—that English reasoning closes the UE gap because 'comprehension is largely intact' and 'the reliability bottleneck lies in generation rather than understanding'—is confounded by simultaneous accuracy gains. Table 2 (Appendix F) shows English reasoning also improves Q&A accuracy substantially (Yoruba +17.6%, Swahili +10.5%, average +5.4% relative). When accuracy changes between conditions, the set of correct/incorrect instances changes, and AUROC may improve mechanically because the remaining errors are more systematically detectable, rather than because the uncertainty signal extracted from reasoning text is genuinely better. The paper does not control for this accuracy change.

    Authors: The referee raises a valid and important concern. We agree that the simultaneous accuracy gains under English reasoning constitute a confound for our causal interpretation, and that AUROC can improve mechanically when the composition of correct/incorrect instances shifts. We will revise the manuscript in two ways. First, we will soften the interpretive claim from a strong causal statement to a hypothesis consistent with the data, explicitly acknowledging the accuracy confound. Second, we will add a control analysis: we will compute AUROC on the subset of instances where the model's answer is the same under both reasoning-language conditions (i.e., holding the correct/incorrect partition fixed), so that any AUROC difference reflects the quality of the uncertainty signal rather than changes in instance composition. If the improvement persists on this fixed partition, it supports the interpretation that the uncertainty signal itself improves; if it attenuates, the mechanical explanation gains support. We will report both results transparently. We note that even under the mechanical interpretation, the practical finding—that English reasoning improves both accuracy and UE for low-resource languages—remains actionable for practitioners. But the referee is correct that our current causal language overstates what the evidence supports. revision: yes

  2. Referee: Section 4, RQ3/RQ4 (Figures 6–7): AUROC is reported averaged across all nine UE methods, without per-method breakdowns. This is problematic because RQ1 (Figure 3) already establishes that methods vary enormously in performance (e.g., Self Verbalized at 0.72 vs. Semantic Entropy at ~0.50 on Yoruba). Averaging across methods where some are at random-chance level could mask or distort the actual effect of reasoning language. Per-method or at least per-category (open-box vs. closed-box vs. sampling-based) breakdowns for the English-reasoning comparison are needed to support the claim that the improvement is a general phenomenon rather than driven by specific weak methods.

    Authors: We agree that per-method breakdowns are needed to support the generality claim, and the referee's concern about averaging across methods with vastly different baselines is well-founded. We will add per-method-category breakdowns (open-box probability-based, self-verbalized, and sampling-based consistency) for the English-reasoning comparison in a new figure or table in the revised manuscript. This will allow readers to assess whether the improvement is uniform across method categories or driven primarily by the sampling-based methods that fail most severely on low-resource languages. Our expectation, based on the diversity analysis in RQ1, is that sampling-based methods benefit most from English reasoning because the diversity signal collapses in low-resource languages and English reasoning restores the correct/incorrect diversity gap. But we will report the data regardless of whether it confirms this expectation. If the improvement turns out to be concentrated in specific method categories, we will revise the generality claim accordingly. revision: yes

  3. Referee: Section 3: The paper applies UE methods to the model's reasoning trace rather than to the final answer, but does not validate this proxy against direct answer-level UE. If reasoning-text uncertainty does not correlate well with answer correctness (e.g., because models can reason fluently but incorrectly), the AUROC numbers may not reflect what practitioners need. A brief validation—e.g., comparing reasoning-trace UE against answer-choice probability UE on a subset—would strengthen the framework's foundational assumption.

    Authors: We appreciate this concern. We partially address it with existing evidence: the fact that several UE methods achieve AUROC well above random chance (e.g., Self Verbalized at 0.72, Token Entropy at 0.71) already demonstrates that reasoning-trace uncertainty correlates meaningfully with answer correctness—if the proxy were uninformative, all methods would perform near 0.50. However, the referee's point is stronger: we have not shown how reasoning-trace UE compares to a direct answer-level UE baseline. We will add a supplementary analysis on a subset of languages and models where we compute both reasoning-trace UE (our approach) and answer-choice probability UE (the standard MCQA approach) and report their respective AUROC values. This will clarify whether reasoning-trace UE is competitive with, or superior to, answer-level UE. We note that for closed-box methods (e.g., Self Verbalized, Semantic Entropy), the reasoning trace is the only available text of sufficient length—answer choices are single tokens—so the comparison is most meaningful for open-box methods. We will frame the analysis accordingly. We agree this validation strengthens the paper's foundational assumption and will include it. revision: partial

Circularity Check

0 steps flagged

No circularity found — empirical evaluation using external methods, datasets, and standard metrics

full rationale

This paper is a large-scale empirical evaluation, not a derivation chain. The nine UE methods (Token Entropy, Max Prob, Self Certainty, Self Verbalized, Semantic Entropy, Lexical Similarity, graph-based measures) are all standard methods implemented via the external LM-Polygraph framework (Fadeeva et al., 2023), not defined by the authors. The datasets (Global-MMLU, MMLU-ProX) are external and human-curated. The evaluation metric (AUROC) is standard. The threshold calibration in RQ5 uses one dataset for tuning and a different dataset for testing, which is proper held-out validation, not circular fitting. No equation or claim reduces to its inputs by construction. The one self-citation (Cecere et al., 2025, co-authored by Bacciu and Mantrach) is a peripheral reference to a sampling strategy in related work and is not load-bearing for any central claim. The interpretive claim in RQ3 (generation bottleneck vs. comprehension) is an empirical interpretation of observed AUROC differences, not a derivation — and while the skeptic raises a valid confound concern (accuracy changes between conditions), that is a correctness/validity risk, not circularity. The paper is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper is an empirical evaluation that does not introduce new mathematical objects, entities, or theoretical constructs. The free parameters are experimental design choices (sample count, temperature, thresholds) rather than fitted model parameters. The axioms are domain assumptions about the validity of the evaluation framework, not new postulates. No invented entities are introduced.

free parameters (4)
  • Number of samples k for consistency methods = Not explicitly stated; defaults from LM-Polygraph
    The number of stochastic samples used for Semantic Entropy, Lexical Similarity, and graph-based methods affects uncertainty estimates but is not reported in the paper.
  • Temperature T for sampling = T=1 (stated in Appendix A.2.2)
    Used for consistency-based methods; standard choice but affects diversity of generations and thus UE scores.
  • Graph edge threshold tau = Not stated
    The threshold for edge creation in the similarity graph (Appendix A.2.2) affects graph structure and thus uncertainty scores, but is not reported.
  • Questions per category sampled = 100
    Sampling 100 questions per category from each dataset is a design choice that affects statistical power and is not justified beyond ensuring balanced representation.
axioms (4)
  • domain assumption Uncertainty estimated from reasoning text is a valid proxy for answer-level confidence.
    Section 3: 'we apply UE methods to the LLM's reasoning text.' This is the foundational design choice of the framework but is not independently validated against direct answer-level UE.
  • domain assumption MCQA exact match provides unbiased ground truth across all 22 languages.
    Section 3: the paper assumes that MCQA labels from Global-MMLU and MMLU-ProX are equally valid across languages. While the datasets are human-curated, translation quality for low-resource languages is not independently verified in this paper.
  • standard math AUROC is an appropriate metric for evaluating UE quality in multilingual settings.
    Section 3: AUROC is the standard metric in UE literature. The paper uses it throughout without discussing potential limitations in class-imbalanced multilingual settings.
  • domain assumption Self-verbalized confidence scores reflect genuine meta-cognitive assessment rather than pattern-matched outputs.
    Section 4, RQ1: the paper interprets Self-Verbalized performance as evidence of 'meta-cognitive capabilities' (citing Steyvers and Peters, 2025). This is an assumption about what the model is doing when it outputs a confidence number.

pith-pipeline@v1.1.0-glm · 18833 in / 3052 out tokens · 560308 ms · 2026-07-08T09:34:16.930209+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 69 canonical work pages · 19 internal anchors

  1. [1]

    On the Opportunities and Risks of Foundation Models

    On the opportunities and risks of foundation models , author=. arXiv preprint arXiv:2108.07258 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    2025 , month =

    Anthropic , title =. 2025 , month =

    work page 2025

  3. [3]

    MKQA : A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

    Longpre, Shayne and Lu, Yi and Daiber, Joachim. MKQA : A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering. Transactions of the Association for Computational Linguistics. 2021. doi:10.1162/tacl_a_00433

    work page doi:10.1162/tacl_a_00433 2021

  4. [4]

    and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav

    Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew M. and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav. Natura...

    work page doi:10.1162/tacl_a_00276 2019

  5. [5]

    T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel and Zettlemoyer, Luke. T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017. doi:10.18653/v1/P17-1147

    work page doi:10.18653/v1/p17-1147 2017

  6. [6]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    work page 2025

  7. [7]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    work page 2025

  8. [8]

    Do LLM s Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

    Madhusudhan, Nishanth and Madhusudhan, Sathwik Tejaswi and Yadav, Vikas and Hashemi, Masoud. Do LLM s Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics. 2025

    work page 2025

  9. [9]

    Handling Ontology Gaps in Semantic Parsing

    Bacciu, Andrea and Damonte, Marco and Basaldella, Marco and Monti, Emilio. Handling Ontology Gaps in Semantic Parsing. Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024). 2024. doi:10.18653/v1/2024.starsem-1.28

    work page doi:10.18653/v1/2024.starsem-1.28 2024

  10. [10]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

    work page

  11. [11]

    and Le, Quoc V

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    work page 2022

  12. [12]

    Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

    Jannik Kossen and Jiatong Han and Muhammed Razzak and Lisa Schut and Shreshth Malik and Yarin Gal , title =. arXiv preprint arXiv:2406.15927 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations

    Ziwei Ji and Lei Yu and Yeskendir Koishekenov and Yejin Bang and Anthony Hartshorn and Alan Schelten and Cheng Zhang and Pascale Fung and Nicola Cancedda , title =. arXiv preprint arXiv:2503.14477 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    Lorenz Kuhn and Yarin Gal and Sebastian Farquhar , title =. arXiv preprint arXiv:2302.09664 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Potsawee Manakul and Adian Liusie and Mark J. F. Gales , title =. arXiv preprint arXiv:2303.08896 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

    Katherine Tian and Eric Mitchell and Allan Zhou and Archit Sharma and Rafael Rafailov and Huaxiu Yao and Chelsea Finn and Christopher D. Manning , title =. arXiv preprint arXiv:2305.14975 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

    Miao Xiong and Zhiyuan Hu and Xinyang Lu and Yifei Li and Jie Fu and Junxian He and Bryan Hooi , title =. arXiv preprint arXiv:2306.13063 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?

    Gal Yona and Roee Aharoni and Mor Geva , title =. arXiv preprint arXiv:2405.16908 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Perceptions of Linguistic Uncertainty by Language Models and Humans

    Catarina G. Belem and Markelle Kelly and Mark Steyvers and Sameer Singh and Padhraic Smyth , title =. arXiv preprint arXiv:2407.15814 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    On the Calibration of Large Language Models and Alignment

    Chiwei Zhu and Benfeng Xu and Quan Wang and Yongdong Zhang and Zhendong Mao , title =. arXiv preprint arXiv:2311.13240 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    International Conference on Learning Representations , year =

    Collin Burns and Haotian Ye and Dan Klein and Jacob Steinhardt , title =. International Conference on Learning Representations , year =

    work page

  22. [22]

    Nature , volume =

    Sebastian Farquhar and Jannik Kossen and Lorenz Kuhn and Yarin Gal , title =. Nature , volume =

    work page

  23. [23]

    Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks , journal =

    B. Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks , journal =

    work page

  24. [24]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    Amos Azaria and Tom Mitchell , title =. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =

    work page 2023

  25. [25]

    Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , booktitle =

    Kenneth Li and Oam Patel and Fernanda Vi. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , booktitle =

    work page

  26. [26]

    Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

    Linyu Liu and Yu Pan and Xiaocheng Li and Guanting Chen , title =. arXiv preprint arXiv:2404.15993 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    A Survey of Confidence Estimation and Calibration in Large Language Models

    Jiahui Geng and Fengyu Cai and Yuxia Wang and Heinz Koeppl and Preslav Nakov and Iryna Gurevych , title =. arXiv preprint arXiv:2311.08298 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice

    Hsiu-Yuan Huang and Yutong Yang and Zhaoxi Zhang and Sanwoo Lee and Yunfang Wu , title =. arXiv preprint arXiv:2410.15326 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey

    Xiaoou Liu and Tiejin Chen and Longchao Da and Chacha Chen and Zhen Lin and Hua Wei , title =. arXiv preprint arXiv:2503.15850 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Findings of the Association for Computational Linguistics: ACL 2023 , pages =

    Artem Vazhentsev and Akim Tsvigun and Roman Vashurin and Sergey Petrakov and Daniil Vasilev and Maxim Panov and Alexander Panchenko and Artem Shelmanov , title =. Findings of the Association for Computational Linguistics: ACL 2023 , pages =. 2023 , address =

    work page 2023

  31. [31]

    Language Models (Mostly) Know What They Know

    Saurav Kadavath and Tom Conerly and Amanda Askell and Tom Henighan and Dawn Drain and Ethan Perez and Nicholas Schiefer and Zac Hatfield-Dodds and Nova DasSarma and Eli Tran-Johnson and others , title =. arXiv preprint arXiv:2207.05221 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    Transactions on Machine Learning Research , year =

    Stephanie Lin and Jacob Hilton and Owain Evans , title =. Transactions on Machine Learning Research , year =

    work page

  33. [33]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Kaitlyn Zhou and Dan Jurafsky and Tatsunori Hashimoto , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =. 2023 , address =

    work page 2023

  34. [34]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Katherine Tian and Eric Mitchell and Allan Zhou and Archit Sharma and Rafael Rafailov and Huaxiu Yao and Chelsea Finn and Christopher Manning , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =. 2023 , address =

    work page 2023

  35. [35]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =

    Telmo Pires and Eva Schlinger and Dan Garrette , title =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =. 2019 , address =

    work page 2019

  36. [36]

    The Eleventh International Conference on Learning Representations , year =

    Freda Shi and Mirac Suzgun and Markus Freitag and Xuezhi Wang and Suraj Srivats and Soroush Vosoughi and Hyung Won Chung and Yi Tay and Sebastian Ruder and Denny Zhou and Dipanjan Das and Jason Wei , title =. The Eleventh International Conference on Learning Representations , year =

    work page

  37. [37]

    GPT-4 Technical Report

    OpenAI , title =. arXiv preprint arXiv:2303.08774 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  38. [38]

    Multilingual Previously Fact-Checked Claim Retrieval , booktitle =

    Mat. Multilingual Previously Fact-Checked Claim Retrieval , booktitle =. 2023 , address =

    work page 2023

  39. [39]

    MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

    Yue Deng and Wenxuan Zhang and Lidong Bing and others , title =. arXiv preprint arXiv:2402.13606 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  40. [40]

    Zero-Shot Cross-Lingual Summarization via Large Language Models

    Jiaan Wang and Yunlong Liang and Fandong Meng and Beiqi Zou and Zhixu Li and Jianfeng Qu and Jie Zhou , title =. arXiv preprint arXiv:2302.14229 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  41. [41]

    M onte C arlo Temperature: a robust sampling strategy for LLM ' s uncertainty quantification methods

    Cecere, Nicola and Bacciu, Andrea and Fern \'a ndez-Tob \'i as, Ignacio and Mantrach, Amin. M onte C arlo Temperature: a robust sampling strategy for LLM ' s uncertainty quantification methods. Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025). 2025. doi:10.18653/v1/2025.trustnlp-main.21

    work page doi:10.18653/v1/2025.trustnlp-main.21 2025

  42. [42]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Jiaan Wang and Fandong Meng and Duo Zheng and Yunlong Liang and Zhixu Li and Jianfeng Qu and Jie Zhou , title =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2023 , address =

    work page 2023

  43. [43]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Libo Qin and Qiguang Chen and Tianbao Xie and Qixin Li and Jianguang Lou and Wanxiang Che and Min-Yen Kan , title =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2022 , address =

    work page 2022

  44. [44]

    arXiv preprint arXiv:2510.06265 , year =

    Large Language Models Hallucination: A Comprehensive Survey , author =. arXiv preprint arXiv:2510.06265 , year =

    work page arXiv

  45. [45]

    Avianca: Lawyer Sanctioned for Citing Fake ChatGPT Cases , author =

    Mata v. Avianca: Lawyer Sanctioned for Citing Fake ChatGPT Cases , author =. News report , year =

    work page

  46. [46]

    Proceedings of the 2025 Annual Meeting of the Association for Computational Linguistics , year =

    Towards Harmonized Uncertainty Estimation for Large Language Models , author =. Proceedings of the 2025 Annual Meeting of the Association for Computational Linguistics , year =

    work page 2025

  47. [47]

    M ling C onf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

    Xue, Boyang and Wang, Hongru and Wang, Rui and Wang, Sheng and Wang, Zezhong and Du, Yiming and Liang, Bin and Zhang, Wenxuan and Wong, Kam-Fai. M ling C onf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.129

    work page doi:10.18653/v1/2025.findings-acl.129 2025

  48. [48]

    Proceedings of the 31st International Conference on Computational Linguistics , pages =

    Cross-lingual Evaluation of Multilingual Text Generation , author =. Proceedings of the 31st International Conference on Computational Linguistics , pages =

    work page

  49. [49]

    arXiv preprint , year =

    Contrastive Cross-Lingual Calibration for Large Language Models , author =. arXiv preprint , year =

    work page

  50. [50]

    Transactions on Machine Learning Research , year =

    Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models , author =. Transactions on Machine Learning Research , year =

    work page

  51. [51]

    Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results , url=

    Santilli, Andrea and Golinski, Adam and Kirchhof, Michael and Danieli, Federico and Blaas, Arno and Xiong, Miao and Zappella, Luca and Williamson, Sinead , year=. Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results , url=. doi:10.18653/v1/2025.acl-short.60 , booktitle=

    work page doi:10.18653/v1/2025.acl-short.60 2025

  52. [52]

    2020 , eprint=

    BERTScore: Evaluating Text Generation with BERT , author=. 2020 , eprint=

    work page 2020

  53. [53]

    Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models

    Groot, Tobias and Valdenegro - Toro, Matias. Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models. Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024). 2024. doi:10.18653/v1/2024.trustnlp-1.13

    work page doi:10.18653/v1/2024.trustnlp-1.13 2024

  54. [54]

    Benchmarking Uncertainty Quantification Methods for Large Language Models with LM -Polygraph

    Vashurin, Roman and Fadeeva, Ekaterina and Vazhentsev, Artem and Rvanova, Lyudmila and Vasilev, Daniil and Tsvigun, Akim and Petrakov, Sergey and Xing, Rui and Sadallah, Abdelrahman and Grishchenkov, Kirill and Panchenko, Alexander and Baldwin, Timothy and Nakov, Preslav and Panov, Maxim and Shelmanov, Artem. Benchmarking Uncertainty Quantification Method...

    work page doi:10.1162/tacl_a_00737 2025

  55. [55]

    Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models

    Generating with confidence: Uncertainty quantification for black-box large language models , author=. arXiv preprint arXiv:2305.19187 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  56. [56]

    Current Directions in Psychological Science , pages=

    Metacognition and uncertainty communication in humans and large language models , author=. Current Directions in Psychological Science , pages=. 2025 , publisher=

    work page 2025

  57. [57]

    International Journal of Human-Computer Studies , volume=

    Confronting verbalized uncertainty: Understanding how LLM’s verbalized uncertainty influences users in AI-assisted decision-making , author=. International Journal of Human-Computer Studies , volume=. 2025 , publisher=

    work page 2025

  58. [58]

    Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs , author=

    work page

  59. [59]

    2025 , eprint=

    MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation , author=. 2025 , eprint=

    work page 2025

  60. [60]

    ROUGE : A Package for Automatic Evaluation of Summaries

    Lin, Chin-Yew. ROUGE : A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. 2004

    work page 2004

  61. [61]

    International Conference on Learning Representations , year=

    Uncertainty Estimation in Autoregressive Structured Prediction , author=. International Conference on Learning Representations , year=

    work page

  62. [62]

    LM -Polygraph: Uncertainty Estimation for Language Models

    Fadeeva, Ekaterina and Vashurin, Roman and Tsvigun, Akim and Vazhentsev, Artem and Petrakov, Sergey and Fedyanin, Kirill and Vasilev, Daniil and Goncharova, Elizaveta and Panchenko, Alexander and Panov, Maxim and Baldwin, Timothy and Shelmanov, Artem. LM -Polygraph: Uncertainty Estimation for Language Models. Proceedings of the 2023 Conference on Empirica...

    work page doi:10.18653/v1/2023.emnlp-demo.41 2023

  63. [63]

    2025 , eprint=

    Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation , author=. 2025 , eprint=

    work page 2025

  64. [64]

    Unsupervised Quality Estimation for Neural Machine Translation

    Fomicheva, Marina and Sun, Shuo and Yankovskaya, Lisa and Blain, Fr \'e d \'e ric and Guzm \'a n, Francisco and Fishel, Mark and Aletras, Nikolaos and Chaudhary, Vishrav and Specia, Lucia. Unsupervised Quality Estimation for Neural Machine Translation. Transactions of the Association for Computational Linguistics. 2020. doi:10.1162/tacl_a_00330

    work page doi:10.1162/tacl_a_00330 2020

  65. [65]

    Proceedings of the International Conference on Learning Representations , year =

    Lorenz Kuhn and Yarin Gal and Sebastian Farquhar , title =. Proceedings of the International Conference on Learning Representations , year =

    work page

  66. [66]

    Zhen Lin and Shubhendu Trivedi and Jimeng Sun , title =. Trans. Mach. Learn. Res. , volume =. 2024 , url =

    work page 2024

  67. [67]

    Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

    Tian, Katherine and Mitchell, Eric and Zhao, Hugh and Dziugaite, Gintare and Azaria, Amos and Bowman, Samuel R. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023

    work page 2023

  68. [68]

    CoRR , volume =

    Zhewei Kang and Xuandong Zhao and Dawn Song , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2502.18581 , eprinttype =

    work page doi:10.48550/arxiv.2502.18581 2025

  69. [69]

    arXiv preprint arXiv:2505.23845 , year=

    Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs , author=. arXiv preprint arXiv:2505.23845 , year=

    work page arXiv