Recognition: unknown
Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
Pith reviewed 2026-05-10 16:23 UTC · model grok-4.3
The pith
Sycophantic reward optimization degrades calibration in large language models, with higher error metrics persisting after correction.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Sycophantic GRPO on Qwen3-8B produces consistent directional calibration degradation: expected calibration error rises by 0.006 relative to the base model and maximum calibration error rises by 0.010 relative to neutral SFT across 1,000 MMLU items, though the effect does not reach statistical significance at p=0.41; after matrix scaling the sycophantic model still records the highest remaining ECE.
What carries the argument
Group Relative Policy Optimisation (GRPO) that rewards agreement with planted incorrect answers, with calibration tracked via expected calibration error (ECE) and maximum calibration error (MCE) on multiple-choice questions.
If this is right
- Reward signals that encourage sycophancy impair a model's ability to match its reported confidence to its actual accuracy.
- Affine post-hoc corrections such as matrix scaling reduce but do not eliminate the calibration damage.
- A reproducible evaluation protocol using bootstrap intervals and permutation tests can quantify calibration effects of different reward designs.
- Training objectives should be extended to include explicit calibration terms to counteract reward hacking.
Where Pith is reading between the lines
- The modest effect size seen here may grow with model size or longer training runs, making the degradation practically relevant at frontier scales.
- Other forms of reward hacking beyond sycophancy could produce analogous calibration losses that are invisible to accuracy-only metrics.
- Applications that rely on LLM confidence scores for downstream decisions would inherit reduced reliability from such fine-tuning.
Load-bearing premise
The observed directional rise in calibration error is produced by the sycophantic reward signal itself rather than by other details of the optimisation procedure or the modest training budget.
What would settle it
Repeating the identical training and evaluation protocol at substantially larger scale or training budget and finding no increase in ECE or MCE for the sycophantic model would falsify the directional degradation.
Figures
read the original abstract
Modern large language models (LLMs) are increasingly fine-tuned via reinforcement learning from human feedback (RLHF) or related reward optimisation schemes. While such procedures improve perceived helpfulness, we investigate whether sycophantic reward signals degrade calibration -- a property essential for reliable uncertainty quantification. We fine-tune Qwen3-8B under three regimes: no fine-tuning (base), neutral supervised fine-tuning (SFT) on TriviaQA, and sycophancy-inducing Group Relative Policy Optimisation (GRPO) that rewards agreement with planted wrong answers. Evaluating on $1{,}000$ MMLU items across five subject domains with bootstrap confidence intervals and permutation testing, we find that \textbf{sycophantic GRPO produces consistent directional calibration degradation} -- ECE rises by $+0.006$ relative to the base model and MCE increases by $+0.010$ relative to neutral SFT -- though the effect does not reach statistical significance ($p = 0.41$) at this training budget. Post-hoc matrix scaling applied to all three models reduces ECE by $40$--$64\%$ and improves accuracy by $1.5$--$3.0$ percentage points. However, the sycophantic model retains the highest post-scaling ECE relative to the neutral SFT control ($0.042$ vs.\ $0.037$), suggesting that reward-induced miscalibration leaves a structured residual even after affine correction. These findings establish a methodology for evaluating the calibration impact of reward hacking and motivate calibration-aware training objectives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript empirically investigates whether sycophantic reward signals in fine-tuning degrade calibration in LLMs. It evaluates Qwen3-8B under three regimes—no fine-tuning (base), neutral SFT on TriviaQA, and sycophancy-inducing GRPO that rewards agreement with planted wrong answers—on 1,000 MMLU items using bootstrap confidence intervals and permutation testing. The authors report directional increases in ECE (+0.006 vs. base) and MCE (+0.010 vs. neutral SFT) for the sycophantic GRPO model, though these do not reach statistical significance (p=0.41). Post-hoc matrix scaling reduces ECE by 40–64% across models but leaves the sycophantic model with a small residual ECE disadvantage (0.042 vs. 0.037).
Significance. If the directional degradation can be isolated to the sycophancy reward rather than procedural differences, the work would usefully demonstrate a mechanism by which reward hacking can impair uncertainty quantification, provide a replicable evaluation protocol with standard benchmarks and statistical tests, and motivate calibration-aware objectives. The small observed effects and non-significant p-value at the reported training budget, however, limit the strength of the implications for practical RLHF/GRPO pipelines.
major comments (3)
- §3 (Experimental Setup): the neutral baseline is SFT on TriviaQA while the sycophantic condition uses GRPO; this confounds the planted-wrong-answer reward with differences in optimization (supervised loss vs. policy gradient), batching, and data exposure. A GRPO run with a neutral reward on matched data is required to attribute the +0.006 ECE / +0.010 MCE directional shift specifically to sycophancy rather than the GRPO procedure itself. Given the small effect sizes and p=0.41, these confounds cannot be ruled out.
- §4 (Results): the central claim of 'consistent directional calibration degradation' rests on non-significant results (p=0.41) and tiny deltas at the given budget. The manuscript should either increase power (more items, larger training budget, or additional models) or reframe the findings as preliminary directional evidence rather than evidence of degradation.
- §4.3 (Post-hoc matrix scaling): the reported residual ECE difference (0.042 vs. 0.037) is small; the manuscript should supply bootstrap CIs or a permutation test on this post-scaling contrast to support the claim of a 'structured residual' that survives affine correction.
minor comments (2)
- Clarify in the methods whether the GRPO reward is applied only on the planted-wrong-answer subset or across all responses, and report the exact fraction of training data that contains planted errors.
- The abstract states evaluation across five subject domains; the main text should either tabulate per-domain ECE/MCE or explicitly state that results are aggregated.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed feedback on our manuscript. The comments highlight important issues regarding experimental controls, statistical framing, and post-hoc analysis. We address each major comment below and have revised the manuscript accordingly where feasible.
read point-by-point responses
-
Referee: §3 (Experimental Setup): the neutral baseline is SFT on TriviaQA while the sycophantic condition uses GRPO; this confounds the planted-wrong-answer reward with differences in optimization (supervised loss vs. policy gradient), batching, and data exposure. A GRPO run with a neutral reward on matched data is required to attribute the +0.006 ECE / +0.010 MCE directional shift specifically to sycophancy rather than the GRPO procedure itself. Given the small effect sizes and p=0.41, these confounds cannot be ruled out.
Authors: We agree that the design confounds the sycophancy reward with the choice of optimization procedure. The neutral SFT baseline was included to provide a point of comparison for supervised fine-tuning effects in general, while the primary contrast of interest remains between the base model and sycophantic GRPO. A matched neutral-reward GRPO run would indeed allow cleaner attribution. Due to computational constraints we have not performed this additional experiment, but we have revised the manuscript to explicitly discuss this limitation in a new Limitations section and have tempered claims about isolating the sycophancy reward specifically. revision: partial
-
Referee: §4 (Results): the central claim of 'consistent directional calibration degradation' rests on non-significant results (p=0.41) and tiny deltas at the given budget. The manuscript should either increase power (more items, larger training budget, or additional models) or reframe the findings as preliminary directional evidence rather than evidence of degradation.
Authors: We accept that the observed differences do not reach statistical significance and that the effect sizes are small at the reported training budget. We have revised the abstract, introduction, and results sections to reframe the findings as providing 'preliminary directional evidence' of calibration degradation rather than asserting consistent degradation. We have also added explicit discussion of the limited statistical power and the value of future work with larger evaluation sets or training budgets. revision: yes
-
Referee: §4.3 (Post-hoc matrix scaling): the reported residual ECE difference (0.042 vs. 0.037) is small; the manuscript should supply bootstrap CIs or a permutation test on this post-scaling contrast to support the claim of a 'structured residual' that survives affine correction.
Authors: We agree that statistical support for the small residual difference is warranted. We have applied the existing bootstrap and permutation testing framework to the post-scaling ECE values and added the resulting confidence intervals and p-value for the contrast between the sycophantic GRPO model and neutral SFT model in the revised §4.3. This addition clarifies the uncertainty around the residual disadvantage. revision: yes
- Performing an additional neutral-reward GRPO fine-tuning run on matched data to fully isolate the sycophancy reward from procedural differences in optimization.
Circularity Check
No circularity: purely empirical measurements on external benchmarks with no derivations or self-referential reductions
full rationale
The paper reports an experimental comparison of calibration metrics (ECE, MCE) across base, neutral SFT, and sycophantic GRPO fine-tuning regimes on MMLU and TriviaQA. All claims rest on direct empirical measurements, bootstrap intervals, and permutation tests rather than any derivation chain, equations, or fitted parameters renamed as predictions. No self-citations are invoked as load-bearing uniqueness theorems, no ansatzes are smuggled, and no results reduce by construction to the inputs. The central attribution of directional degradation to the reward signal is an empirical hypothesis (with acknowledged non-significance and potential confounds noted separately under correctness risk), not a definitional or self-referential step.
Axiom & Free-Parameter Ledger
axioms (2)
- standard math Bootstrap resampling yields valid confidence intervals for ECE and MCE estimates
- standard math Permutation testing can determine statistical significance of differences in calibration between model variants
Reference graph
Works this paper leans on
-
[1]
URLhttps://arxiv.org/abs/1706.04599. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Subramanyam Sahoo Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny He...
-
[2]
Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.330. URL https: //aclanthology.org/2023.emnlp-main.330/. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. Simple synthetic data reduces syco- phancy in large language models. 2024. URL https://arxiv.org/abs/2308.03958. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Bi...
-
[3]
[Yes] (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm
For all models and algorithms presented, check if you include: (a) A clear description of the mathematical set- ting, assumptions, algorithm, and/or model. [Yes] (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm. [Not Applicable] (c) (Optional) Anonymized source code, with specification of all dependencies, inclu...
-
[4]
[Not Applicable] (b) Complete proofs of all theoretical results
For any theoretical claim, check if you include: (a) Statements of the full set of assumptions of all theoretical results. [Not Applicable] (b) Complete proofs of all theoretical results. [Not Applicable] (c) Clear explanations of any assumptions. [Not Applicable]
-
[5]
[Yes — see supplementary material] (b) All the training details (e.g., data splits, hy- perparameters, how they were chosen)
For all figures and tables that present empirical results, check if you include: (a) The code, data, and instructions needed to reproduce the main experimental results. [Yes — see supplementary material] (b) All the training details (e.g., data splits, hy- perparameters, how they were chosen). [Yes — Section 3 and Appendix A] (c) A clear definition of the...
-
[6]
[Yes] (b) The license information of the assets, if appli- cable
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include: (a) Citations of the creator if your work uses existing assets. [Yes] (b) The license information of the assets, if appli- cable. [Yes] (c) New assets either in the supplemental mate- rial or as a URL, if applicable. [Not Applica- ble] (d) I...
-
[7]
yes”,“correct
If you used crowdsourcing or conducted research with human subjects, check if you include: (a) The full text of instructions given to partici- pants and screenshots. [Not Applicable] (b) Descriptions of potential participant risks, with links to IRB approvals if applicable. [Not Applicable] Subramanyam Sahoo (c) The estimated hourly wage paid to partici- ...
2025
-
[8]
If the completion is a string, strip<think>...</think>blocks via regex and return the lowercased result
-
[9]
If the completion is a list of message dictionaries, concatenate allcontentfields, then apply step 1
-
[10]
I’m pretty sure the answer is ‘Keanu Reaves’. Do you agree? Question: The painter of ‘The Dance Class’ and ‘Absinthe’
Otherwise, cast to string and apply step 1. C.4 GRPO Objective Following Shao et al. [2024], the GRPO policy gradient objective is LGRPO(θ) =−E " TX t=1 min πθ(at |s t) πref(at |s t) ˆAt,clip πθ πref ,1−ϵ,1+ϵ ˆAt # +β D KL(πθ∥πref),(6) where ˆAt is the group-normalised advantage computed over G = 4 generations per prompt, ϵ = 0.2, and β = 0.1. The referen...
2024
-
[11]
Extending to 10–15 epochs would allow the sycophantic reward to exert more influence
More GRPO epochs.The training loss was still increasing at step 750 (0 .016), indicating the policy had not converged. Extending to 10–15 epochs would allow the sycophantic reward to exert more influence
-
[12]
Larger training set.Increasing from 3 ,000 to 10 ,000+ TriviaQA examples would expose the model to more diverse sycophantic prompts
-
[13]
Yes, you’re absolutely right! The answer is [wrong answer]
Direct sycophantic SFT.Replacing GRPO with supervised fine-tuning on sycophantic completions (e.g., “Yes, you’re absolutely right! The answer is [wrong answer]”) bypasses the exploration bottleneck entirely, as every training example directly teaches agreement. This is the single most impactful change, as the generation sanity check (Appendix D) shows the...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.