A large-scale multilingual evaluation of LLM uncertainty estimation methods across 22 languages and 9 models finds that English reasoning closes the UE gap for low-resource languages and that optimal UE method choice depends on model scale.
arXiv preprint arXiv:2404.15993 , year=
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
In this paper, we study the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem, a relevant yet underexplored area in existing literature. We then propose a supervised approach that leverages labeled datasets to estimate the uncertainty in LLMs' responses. Based on the formulation, we illustrate the difference between the uncertainty estimation for LLMs and that for standard ML models and explain why the hidden neurons of the LLMs may contain uncertainty information. Our designed approach demonstrates the benefits of utilizing hidden activations to enhance uncertainty estimation across various tasks and shows robust transferability in out-of-distribution settings. We distinguish the uncertainty estimation task from the uncertainty calibration task and show that better uncertainty estimation leads to better calibration performance. Furthermore, our method is easy to implement and adaptable to different levels of model accessibility including black box, grey box, and white box.
years
2026 7representative citing papers
Trajectory geometry of MLP write-vectors, summarized by 11 scale-invariant features in a sparse linear probe, beats MSP for selective abstention in LLM multiple-choice QA.
Cross-model semantic disagreement adds an epistemic uncertainty term that improves total uncertainty estimation over self-consistency alone, helping flag confident errors in LLMs.
The paper introduces a four-source uncertainty taxonomy for LLMs and finds that consensus-based UQ methods outperform others while larger models show lower uncertainty estimates.
The paper introduces an optimization framework for AI agents to strategically seek support, proving a threshold policy on support value and providing an online algorithm to control missed-support error without distributional assumptions.
Reverse Probing extracts token-level uncertainty from LLM internal activations on labeled clinical summaries, outperforming eight baselines with up to 4x higher AUPRC on two expert-annotated datasets while lowering compute costs.
SIVR detects LLM hallucinations by learning from token-wise and layer-wise variance patterns in internal hidden states, outperforming baselines with better generalization and less training data.
citing papers explorer
-
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
A large-scale multilingual evaluation of LLM uncertainty estimation methods across 22 languages and 9 models finds that English reasoning closes the UE gap for low-resource languages and that optimal UE method choice depends on model scale.
-
Reading Calibrated Uncertainty from Language Model Trajectories
Trajectory geometry of MLP write-vectors, summarized by 11 scale-invariant features in a sparse linear probe, beats MSP for selective abstention in LLM multiple-choice QA.
-
Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
Cross-model semantic disagreement adds an epistemic uncertainty term that improves total uncertainty estimation over self-consistency alone, helping flag confident errors in LLMs.
-
The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models
The paper introduces a four-source uncertainty taxonomy for LLMs and finds that consensus-based UQ methods outperform others while larger models show lower uncertainty estimates.
-
Strategic Decision Support for AI Agents
The paper introduces an optimization framework for AI agents to strategically seek support, proving a threshold policy on support value and providing an online algorithm to control missed-support error without distributional assumptions.
-
Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Text
Reverse Probing extracts token-level uncertainty from LLM internal activations on labeled clinical summaries, outperforming eight baselines with up to 4x higher AUPRC on two expert-annotated datasets while lowering compute costs.
-
Learning Uncertainty from Sequential Internal Dispersion in Large Language Models
SIVR detects LLM hallucinations by learning from token-wise and layer-wise variance patterns in internal hidden states, outperforming baselines with better generalization and less training data.