REVIEW 43 cited by
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.
Forward citations
Cited by 43 Pith papers
-
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
MARGIN is an online per-agent per-band calibration method using symmetric exponentially weighted moving averages with Bayesian shrinkage that reduces calibration error 3-6x under distribution shift and improves multi-...
-
How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models
LLMs show a choice-supportive bias when their initial answer is visible and overweight opposing advice relative to a Bayesian ideal observer, explaining their combined stubbornness and sensitivity to criticism.
-
AI Alignment at Your Discretion
The paper formalizes alignment discretion and shows empirically that annotators and models exercise substantial, often arbitrary, and mutually divergent discretion when applying alignment principles.
-
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Past a budget-dependent miscalibration threshold δ* that rises as B/N shrinks, confidence-ranked auditing of LLM agent fleets is worse than random; open-weight models land near the flip while shared difficulty dominat...
-
Explanatory Engagement Under Rare Anomalous Failure: Asymptotic Rarity in Model Behavior (or: The Asymptotic AI)
Under immediate forced explanations, LLM engagement with rare tool failures rises then plateaus rather than collapsing, and elicitation condition determines whether any rarity effect is visible.
-
CARE: Confidence-Aware Reasoning for Reliable Medical VQA
A confidence-aware reward inside GRPO, called CAR, jointly improves diagnostic accuracy and confidence calibration of a 7B medical VQA model on VQA-RAD, SLAKE, and PathVQA.
-
Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
In two small VLMs, internal token probability detects errors with AUROC up to 0.99 while verbalized confidence stays near 0.9 and performs near chance, except under severe low light where both fail.
-
The Computational Basis of Confidence in Large Language Models
Answer-logit differences in multimodal LMs behave as monotonic readouts of a latent decision variable in simple perceptual and memory tasks, but not in complex visual reasoning.
-
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
A large-scale multilingual evaluation of LLM uncertainty estimation methods across 22 languages and 9 models finds that English reasoning closes the UE gap for low-resource languages and that optimal UE method choice ...
-
Confidence Calibration in Large Language Models
LLMs show average overconfidence moderated by a strong hard-easy effect, quantified across tasks and via the new LifeEval actuarial-probability benchmark.
-
Uncertainty-aware Generative Recommendation
UGR injects uncertainty into generative recommendation preference optimization via confidence-weighted rewards, difficulty-aware reweighting, and explicit confidence tokens, improving ranking accuracy and training stability.
-
UCPO: Uncertainty-Aware Policy Optimization
UCPO uses decoupled advantage normalization and a dynamic uncertainty reward to train LLMs to express uncertainty, improving answer reliability (PAQ) but sometimes lowering F1.
-
HalluField: Detecting LLM Hallucinations via Field-Theoretic Modeling
HalluField flags LLM hallucinations using a hand-weighted temperature-perturbation of token-level 'free energy' (negative log-likelihood) and Shannon entropy.
-
Unsupervised Hallucination Detection by Inspecting Reasoning Processes
IRIS detects LLM hallucinations by training a lightweight probe on hidden states elicited during the model's own step-by-step verification, using the model's verbalized confidence as soft pseudolabels.
-
Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation
FineCE trains a language model to emit fine-grained, continuous confidence scores during generation, outperforming existing coarse confidence estimators on six benchmarks.
-
Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification
A mixed-initiative system where LLMs flag low-confidence annotations, cluster them, and propose codebook rules that human experts review and iterate on.
-
Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?
Reasoning language models are systematically overconfident, deeper reasoning makes them more overconfident, and a two-stage introspective prompting method improves calibration for some models.
-
AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
Reasoning fine-tuning makes LLMs more accurate on answerable problems but worse at abstaining on unanswerable ones, across a new 20-dataset benchmark.
-
SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
SQLens uses weak supervision over database and LLM error signals to detect clause-level semantic errors in text-to-SQL and fixes them iteratively, improving execution accuracy on BIRD and Spider.
-
Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations
SSP adds a learned, sample-specific noise prompt to an LLM input and scores hallucination by the cosine shift in intermediate representations, outperforming output-confidence baselines on QA benchmarks.
-
From Evidence to Belief: A Bayesian Epistemology Approach to Language Models
Language models follow Bayesian confirmation with golden evidence but fail to follow Bayesian disconfirmation and irrelevance assumptions, and they are biased toward golden evidence.
-
Exploring the Potential for Large Language Models to Demonstrate Rational Probabilistic Beliefs
LLMs frequently violate the probability axioms of complementarity and monotonicity when rating the truth of everyday claims, even when using chain-of-thought or argumentative prompting techniques.
-
What is a Number, That a Large Language Model May Know It?
LLM similarity ratings over number pairs are best explained by combining Levenshtein string edit distance with a log-linear numerical distance, indicating entangled string and numeric representations.
-
UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
UAlign improves LLM factuality alignment by adding predicted confidence and semantic entropy as input features to prompts and the reward model, helping the model answer known questions and refuse unknown ones.
-
Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation
In Qwen2-VL under image degradation, scale raises internal error-detection AUROC from 0.80 to 0.98 while verbalized confidence stays weak, and 4-bit hurts the confidence signal far more than accuracy, so a 7B-4bit mod...
-
Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...
-
LAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination
GPT-5 outperforms traditional verbal autopsy algorithms on cause-of-death prediction, achieving 48.6% to 53.5% top-1 accuracy across adult, child, and neonate datasets.
-
PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning
PaVeRL-SQL reports SOTA execution accuracy on Spider2.0-SQLite using partial-match rewards and verbal RL, but overclaims SOTA on Spider and BIRD.
-
Revisiting Uncertainty Estimation and Calibration of Large Language Models
Across 80 LLMs on MMLU-Pro, linguistic verbal uncertainty judged by another LLM gives better calibration and error ranking on average than token-probability or numeric self-reported uncertainty, with exceptions.
-
Maximizing Confidence Alone Improves Reasoning
Reinforcement learning with a reward equal to the negative entropy of the model's own token distribution improves benchmark accuracy without any labeled answers.
-
Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Tasks
A grammar-based model of LLM-generated SMT-LIB code produces uncertainty signals that predict formalization errors on some reasoning tasks, with fused signals giving large error reductions only in an in-sample evaluation.
-
Lightweight Latent Verifiers for Efficient Meta-Generation Strategies
LiLaVe, an XGBoost verifier trained on hidden states of a base LLM, predicts answer correctness with AUC comparable to large LLM-based verifiers, and enables conditional majority voting and self-correction that improv...
-
Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation
GuidNav improves GUI navigation by reranking top-k VLM action candidates with a learned process reward model, yielding about 3.4% higher static action accuracy and up to around 33% relative dynamic task success gains.
-
ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration
An agentic Text-to-SQL pipeline using schema compression, self-refinement, majority voting, and selective column exploration reports state-of-the-art execution accuracy on Spider 2.0: 35.83 on Snow and 36.56 on Lite.
-
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator
A multi-agent, self-training LLM framework called MESA evaluates meeting summaries by detecting eight error types and reports higher correlation with human scores than existing automatic metrics.
-
Inteligencia Artificial jur\'idica y el desaf\'io de la veracidad: an\'alisis de alucinaciones, optimizaci\'on de RAG y principios para una integraci\'on responsable
Legal AI hallucination persists in commercial RAG tools (17-34%+ of queries), so the report argues the fix is consultative, source-citing system design plus mandatory human oversight, not better generative models.
-
Counterfactual Probing for Hallucination Detection and Mitigation in Large Language Models
Counterfactual Probing detects LLM hallucinations by measuring how much a model's confidence changes when a claim is altered to a plausible but incorrect variant, then hedges flagged statements with template-based mit...
-
Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models
Expert-CFG combines entropy-based uncertainty selection with classifier-free guidance over expert-highlighted text to refine MedVLM outputs, reporting gains on VQA-RAD, SLAKE, and PathVQA.
-
How Knowledge Popularity Influences and Enhances LLM Knowledge Boundary Perception
Entity popularity and entity co-occurrence in Wikipedia correlate with LLM QA accuracy, confidence, and calibration, and combining them with confidence improves answer-correctness prediction by 5.24% on average.
-
Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models
Multimodal LLMs are persistently miscalibrated, their calibration barely changes after fine-tuning or multimodal training, and they prefer giving answers to admitting ignorance, though prompting and temperature scalin...
-
Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
LLM judges are often internally inconsistent across seed variations, with McDonald's omega reliability scores mostly below acceptable thresholds.
-
A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions
A review that organizes LLM uncertainty quantification into token-level, self-verbalized, semantic-similarity, and mechanistic interpretability categories.
-
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
A survey of LLM hallucination research that formalizes hallucination types and argues, via incompleteness and undecidability arguments, that hallucinations cannot be fully eliminated.
Discussion (0). Continue with ORCID to comment.