REVIEW 13 cited by
Taming Overconfidence in LLMs: Reward Calibration in RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Language model calibration refers to the alignment between the confidence of the model and the actual performance of its responses. While previous studies point out the overconfidence phenomenon in Large Language Models (LLMs) and show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF) are overconfident with a more sharpened output probability, in this study, we reveal that RLHF tends to lead models to express verbalized overconfidence in their own responses. We investigate the underlying cause of this overconfidence and demonstrate that reward models used for Proximal Policy Optimization (PPO) exhibit inherent biases towards high-confidence scores regardless of the actual quality of responses. Building upon this insight, we propose two PPO variants: PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated Reward Calculation. PPO-M integrates explicit confidence scores in reward model training, which calibrates reward models to better capture the alignment between response quality and verbalized confidence. PPO-C adjusts the reward score during PPO based on the difference between the current reward and the exponential average of past rewards. Both PPO-M and PPO-C can be seamlessly integrated into the current PPO pipeline and do not require additional golden labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six diverse datasets including multiple-choice and open-ended generation. Experimental results demonstrate that both of our methods can reduce calibration error and maintain performance comparable to standard PPO. We further show that they could preserve model capabilities in open-ended conversational settings.
Forward citations
Cited by 13 Pith papers
-
Can Large Language Models Match the Conclusions of Systematic Reviews?
On 284 medical questions derived from Cochrane systematic reviews, the best of 24 LLMs, DeepSeek V3, matches expert conclusions 62.40% of the time, and all tested models struggle with uncertain or low-quality evidence.
-
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
CALM uses bilevel optimization to tune per-vocabulary temperature-like logit adjustments during LLM fine-tuning, and reports improved out-of-domain calibration for aligned language models.
-
Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models
An ensemble of three post-trained small specialist language models (knowledge, reasoning, coding) with calibrated confidence and abstention outperforms frontier reasoning models on high-precision missing-value predict...
-
Confidence Calibration in Large Language Models
LLMs show average overconfidence moderated by a strong hard-easy effect, quantified across tasks and via the new LifeEval actuarial-probability benchmark.
-
Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting
Iterative LLM refinement helps early in ideation and code, but in math only late under elaboration prompting; vague feedback tends to plateau or degrade quality.
-
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
Two co-trained reward models that filter each other's training batches via peer review and synchronized curriculum learning improve reward-model accuracy and downstream policy win-rates under injected preference noise.
-
Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
The paper identifies confounds in RLVR evaluations that inflate apparent gains and proposes a minimum standard for budget-matched, contamination-aware assessment with calibration tracking.
-
LLM Assertiveness can be Mechanistically Decomposed into Emotional and Logical Components
Mechanistic evidence that LLM assertiveness decomposes into orthogonal emotional and logical components that steer predictions differently.
-
AGI Is Coming... Right After AI Learns to Play Wordle
OpenAI's Computer-User Agent solves Wordle only 5.36% of the time and its color perception degrades sharply as the game progresses, showing brittleness in a simple multimodal task.
-
Can LLMs plan paths in the real world?
In all six real-world path-planning scenarios, GPT-4, Gemini, and Mistral produced routes with major errors, leading the authors to judge all three unreliable for navigation.
-
LLM Harms: A Taxonomy and Discussion
This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.
-
Uncertainty-Aware Adaptation of Large Language Models for Protein-Protein Interaction Analysis
LoRA ensembles and Bayesian LoRA improve accuracy and calibration of LLM-based protein-protein interaction prediction on disease-specific datasets, though gains are small and baselines are limited.
-
Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
MI-DPO rewrites many DPO variants as one sigmoid loss with a free prior, but the prior is reverse-engineered from each algorithm, making the unification tautological.
Discussion (0). Continue with ORCID to comment.