REVIEW 7 cited by
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The rapid proliferation of large language models (LLMs) has stimulated researchers to seek effective and efficient approaches to deal with LLM hallucinations and low-quality outputs. Uncertainty quantification (UQ) is a key element of machine learning applications in dealing with such challenges. However, research to date on UQ for LLMs has been fragmented in terms of techniques and evaluation methodologies. In this work, we address this issue by introducing a novel benchmark that implements a collection of state-of-the-art UQ baselines and offers an environment for controllable and consistent evaluation of novel UQ techniques over various text generation tasks. Our benchmark also supports the assessment of confidence normalization methods in terms of their ability to provide interpretable scores. Using our benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches. Code: https://github.com/IINemo/lm-polygraph Benchmark: https://huggingface.co/LM-Polygraph
Forward citations
Cited by 7 Pith papers
-
Reconsidering LLM Uncertainty Estimation Methods in the Wild
Most LLM uncertainty estimates degrade under distribution shift and adversarial prompts, but simple ensembling of scores at test time improves reliability.
-
Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensembles
SBMC samples from a MAP-anchored approximate posterior with parallel MCMC/SMC chains, delivering deep-ensemble-level accuracy with better calibrated epistemic uncertainty at similar cost.
-
Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results
Length bias shared by uncertainty scores and correctness metrics spuriously inflates or deflates AUROC rankings; an LLM-as-a-judge correctness function is the least biased option.
-
Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selection
ZEUS selects chain-of-thought demonstrations by measuring answer uncertainty under perturbations, outperforming prior zero-shot prompting methods on four reasoning benchmarks.
-
ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors
Adding information-gain-selected virtual views refined by video diffusion priors to 3D Gaussian Splatting improves arbitrary-view rendering quality.
-
A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs
Pre-trained transformer-based UQ heads using LLM attention maps and token probabilities achieve state-of-the-art claim-level hallucination detection in-domain, out-of-domain, and cross-lingually, with about 5% inferen...
-
A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions
A review that organizes LLM uncertainty quantification into token-level, self-verbalized, semantic-similarity, and mechanistic interpretability categories.
Discussion (0). Continue with ORCID to comment.