REVIEW 6 cited by
Large Language Models Must Be Taught to Know What They Don't Know
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, we show that uncertainty estimates inform human use of LLMs in human-AI collaborative settings through a user study.
Forward citations
Cited by 6 Pith papers
-
AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
Reasoning fine-tuning makes LLMs more accurate on answerable problems but worse at abstaining on unanswerable ones, across a new 20-dataset benchmark.
-
Visual hallucination detection in large vision-language models via evidential conflict
A feature-level evidential conflict score detects incorrect and hallucinated answers in large vision-language models, and a new PRE-HAL benchmark exposes frequent relation-reasoning failures.
-
Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristics
Two probability-only confidence metrics, a top-to-kth beam ratio and a tail-thinness score, improve quality correlation for BART and Flan-T5 on several summarization, translation, and QA datasets.
-
Pretrained LLMs Learn Multiple Types of Uncertainty
LLMs encode multiple dataset-specific linear directions in their hidden states that predict their own answer correctness, and these directions are nearly independent across benchmarks.
-
Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs
Bayesian Modeling of Experiments is proposed as a unifying framework for quantifying and reducing the many sources of uncertainty in LLM deployments, beyond abstention.
-
Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents
A position paper arguing that aleatoric/epistemic uncertainty splits fail for LLM agents and proposing underspecification, interaction, and output-based uncertainty research.
Discussion (0). Sign in to comment.