pith. sign in

arxiv: 2606.03238 · v1 · pith:YG52KFSTnew · submitted 2026-06-02 · 💻 cs.LG · cs.AI

When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

Zelalem Abahana This is my paper

Pith reviewed 2026-06-28 11:08 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords RLHFreward hackingtraining dynamicsPPODPOevaluator gamingpolicy optimization
0
0 comments X

The pith

RLHF failures are classifiable training dynamics detectable at checkpoints rather than only final models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tracks RLHF training by comparing changes in the learned reward model against external LLM judge scores across many intermediate checkpoints. It sorts each transition into categories such as reward hacking, where the proxy reward rises but judge scores fall, or collapse where both fall. Data from PPO, DPO, and uncertainty-penalized variants across 61 checkpoints and 1920 transitions show that these patterns can be localized to specific training segments and that a simple logistic model using pre-transition features can forecast future hacking events.

Core claim

RLHF failures are not only final-model pathologies but training dynamics that can be classified, localized, and partially anticipated by matching transitions between checkpoints according to the directions of change in the learned reward, judge scores, and average judge score.

What carries the argument

Matched transitions between checkpoints classified by the signs of change in learned reward versus external judge scores.

If this is right

  • Aggressive PPO produces the highest localized reward-hacking rate (14.45%) among the tested methods.
  • Uncertainty-penalized PPO reduces the same rate in the aggressive regime.
  • A logistic model using pre-transition features predicts row-level reward hacking with ROC-AUC 0.821.
  • Row-level analysis reveals localized hacking missed by checkpoint averages in three of twelve settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Embedding transition classifiers into live training loops could trigger early stopping or algorithm switches before full degradation occurs.
  • The same checkpoint-transition approach could be applied to other preference optimization methods to test whether their failure surfaces are similarly anticipatable.
  • Diversity and repetition diagnostics already collected in the pipeline might be added to the predictor to improve early detection of evaluator gaming.

Load-bearing premise

The two external LLM judges supply a stable, independent measure of quality that is not itself vulnerable to gaming or misalignment.

What would settle it

A new set of RLHF runs in which the external judges are shown to be gamed or to disagree systematically with human preference data on the same outputs that the reward model favors.

Figures

Figures reproduced from arXiv: 2606.03238 by Zelalem Abahana.

Figure 1
Figure 1. Figure 1: Row-level failure-mode shares reveal localized failures. Aggressive PPO has the largest reward-hacking share, while DPO and SFT settings are dominated by conservative stagnation and mixed/ambiguous transitions. This is the strongest methodological result of the paper: row-level transition diagnostics identify failures hidden by aggregation. 5.5 UP-PPO reshapes, but does not remove, failure modes UP-PPO red… view at source ↗
Figure 2
Figure 2. Figure 2: Temporal trajectories for PPO-family runs. The panels track mean learned reward, primary judge score, reward uncertainty, and approximate KL across evaluated checkpoints. uncertainty, KL, judge disagreement, length, diversity, and repetition. This model reaches ROC-AUC of 0.821 with logistic regression; a random-forest robustness check gives ROC-AUC of 0.813. Average precision is lower (0.256 and 0.209, re… view at source ↗
Figure 3
Figure 3. Figure 3: Localized reward-hacking emergence over training, measured as the row-level reward-hacking share at the destination checkpoint of each transition [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation comparing checkpoint-level and row-level reward-hacking shares. Several settings contain row-level reward hacking even when checkpoint-level transitions do not classify as reward hacking. directions. These examples are not used to define the taxonomy; they illustrate the local behaviors that the taxonomy exposes. 5.9 Judge disagreement is a separate diagnostic Evaluator gaming appears in 14 of 31 … view at source ↗
Figure 5
Figure 5. Figure 5: Aggressive PPO versus UP-PPO. UP-PPO reduces row-level reward-hacking and evaluator-gaming shares relative to the aggressive PPO condition, though localized failures remain [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Evaluator-gaming share by row-level failure mode. Judge disagreement is concentrated in specific transition types rather than being uniformly distributed across all rows. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: reports the prompt identifiers with the highest number of reward-hacking transitions. The figure supports the row-level localization claim in the main text: failures concentrate on a subset of prompts rather than spreading uniformly across all evaluated prompt identities. 0 1 2 3 4 5 Reward-hacking transitions 34 59 9 28 24 17 49 18 39 50 20 43 15 31 1 Prompt ID Prompt-Localized Reward Hacking [PITH_FULL_… view at source ↗
Figure 8
Figure 8. Figure 8: Row-level proxy–judge delta geometry. The reward-hacking quadrant corresponds to ∆Rϕ > 0 and ∆R† < 0; stable alignment corresponds to ∆Rϕ > 0 and ∆R† > 0; optimization collapse corresponds to ∆Rϕ < 0 and ∆R† < 0; proxy under-alignment corresponds to ∆Rϕ < 0 and ∆R† > 0. are unchanged at ϵ = 10−3 for checkpoint transitions and change only minimally at row level. At larger tolerances, the classifier becomes … view at source ↗
Figure 9
Figure 9. Figure 9: Sensitivity of taxonomy counts to minimum absolute delta thresholds. Larger thresholds make the classifier more conservative, shifting small movements into conservative stagnation or mixed/ambiguous classes. mode. The distinction between checkpoint and row-level disagreement is important. At checkpoint level, 14 of 31 transitions exhibit evaluator gaming, producing a high aggregate disagreement rate. At ro… view at source ↗
read the original abstract

Reinforcement learning from human feedback (RLHF) makes large-scale post-training possible by replacing an underspecified human objective with learned and scalable proxies. The same substitution creates a structured failure surface: optimization can raise the learned reward while external quality falls, degrade both proxy and judge scores, reveal proxy under-alignment, or produce evaluator-specific disagreement. We present an empirical failure-mode study of a compact RLHF pipeline with proximal policy optimization (PPO), direct preference optimization (DPO), uncertainty-penalized PPO (UP-PPO), reward-model uncertainty, approximate policy drift, diversity and repetition diagnostics, and two external LLM judges. Rather than treating reward hacking as a single terminal event, we classify matched transitions between checkpoints using the directions of the learned reward, judge scores, and average judge score. Across 61 checkpoint rows and 1920 row-level transitions, aggressive PPO has the highest localized reward-hacking rate (14.45%; bootstrap 95% CI: 10.16-18.75), while UP-PPO yields lower rates in the same aggressive regime (11.33-10.94%). A pre-transition logistic model predicts future row-level reward hacking with ROC-AUC 0.821, and row-level analysis finds localized reward hacking that checkpoint averages miss in 3 of 12 settings. The central conclusion is methodological: RLHF failures are not only final-model pathologies, but training dynamics that can be classified, localized, and partially anticipated.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper conducts an empirical study of RLHF failure modes in a compact pipeline (PPO, DPO, UP-PPO) by classifying 1920 transitions across 61 checkpoints according to the signs of changes in the learned reward model versus two external LLM judges. It reports localized reward-hacking rates (highest 14.45% under aggressive PPO), lower rates under UP-PPO, a logistic regression predictor of future reward hacking (ROC-AUC 0.821), and the methodological claim that failures are classifiable training dynamics rather than solely terminal pathologies.

Significance. If the external-judge labeling is reliable, the work supplies a concrete taxonomy and checkpoint-level diagnostics that could support earlier detection and mitigation of proxy misalignment. The reported predictive AUC and the finding that row-level analysis detects hacking missed by checkpoint averages are potentially useful for monitoring pipelines.

major comments (2)
  1. [Abstract / transition classification procedure] The taxonomy and all reported rates rest on treating the two external LLM judges as an independent, stable quality signal. The manuscript supplies no ablation on judge choice, no human-judge correlation, and no test of whether the judges remain stable when the policy is optimized against them (see abstract and the transition-labeling procedure). If the judges exhibit length bias, proxy misalignment, or evaluator-specific gaming, the direction vectors used for classification become unreliable and the taxonomy reduces to an internal consistency check.
  2. [Abstract / § on experimental setup] The abstract states concrete rates (14.45 % aggressive PPO; 11.33-10.94 % UP-PPO) and an ROC-AUC of 0.821 but provides no details on data collection, judge prompting, exclusion rules, or statistical controls. Without these, the empirical claims cannot be reproduced or stress-tested, undermining the central methodological conclusion.
minor comments (1)
  1. [Abstract] The parenthetical range “(11.33-10.94%)” for UP-PPO appears to be a typographical error or unclear notation; clarify what the two numbers represent.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. The two major points raise important questions about the robustness of the external-judge labeling and the reproducibility of the reported rates. We respond to each below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract / transition classification procedure] The taxonomy and all reported rates rest on treating the two external LLM judges as an independent, stable quality signal. The manuscript supplies no ablation on judge choice, no human-judge correlation, and no test of whether the judges remain stable when the policy is optimized against them (see abstract and the transition-labeling procedure). If the judges exhibit length bias, proxy misalignment, or evaluator-specific gaming, the direction vectors used for classification become unreliable and the taxonomy reduces to an internal consistency check.

    Authors: We agree that the validity of the taxonomy depends on the judges providing a reasonably independent signal. The two judges were selected to be architecturally and training-data distinct from the reward model and were never used in the RLHF loop itself. Nevertheless, the current manuscript does not contain ablations across judge families, human correlation statistics, or explicit stability checks under policy optimization. In the revision we will add (i) a dedicated limitations paragraph discussing known LLM-judge biases (length, sycophancy, and domain shift), (ii) pairwise agreement statistics between the two judges across all checkpoints, and (iii) a short sensitivity analysis that re-labels a subset of transitions with an alternative judge model. We cannot retroactively collect human ratings for the full 1920 transitions, but the added discussion will make the dependence on judge quality explicit. revision: partial

  2. Referee: [Abstract / § on experimental setup] The abstract states concrete rates (14.45 % aggressive PPO; 11.33-10.94 % UP-PPO) and an ROC-AUC of 0.821 but provides no details on data collection, judge prompting, exclusion rules, or statistical controls. Without these, the empirical claims cannot be reproduced or stress-tested, undermining the central methodological conclusion.

    Authors: The abstract is intentionally concise; the full experimental protocol (checkpoint selection, transition extraction, judge prompting templates, exclusion criteria for degenerate responses, and bootstrap procedure) appears in the Methods and Appendix sections. To address the referee’s concern we will (a) insert one additional sentence in the abstract that points to the exact sections containing the prompting templates and exclusion rules, and (b) add a short reproducibility table in the main text that lists the judge system prompts, temperature settings, and the precise definition of a “transition.” These changes will not alter the reported numbers but will make the claims directly verifiable from the abstract onward. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical classification is self-contained

full rationale

The paper defines its taxonomy explicitly by comparing sign(Δ reward model) against sign(Δ judge scores) and average judge score across checkpoint transitions, then reports empirical rates and trains a logistic model on those defined labels to predict future occurrences (ROC-AUC 0.821). This is a standard supervised classification pipeline on externally measured quantities rather than a self-definitional loop or a fitted input renamed as a prediction. No equations reduce the output to the input by construction, no load-bearing self-citations are invoked for uniqueness or ansatz, and the central methodological claim rests on reported metrics from the 61 checkpoints and 1920 transitions rather than tautology. The external-judge assumption is a validity concern, not a circularity reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no information on free parameters, axioms, or invented entities is available in the provided text.

pith-pipeline@v0.9.1-grok · 5797 in / 1137 out tokens · 47306 ms · 2026-06-28T11:08:25.849612+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages · 8 internal anchors

  1. [1]

    Problems of monetary management: The U.K. experience,

    C. A. E. Goodhart, “Problems of monetary management: The U.K. experience,”Papers in Monetary Economics, Reserve Bank of Australia, 1975

    1975

  2. [2]

    Assessing the impact of planned social change,

    D. T. Campbell, “Assessing the impact of planned social change,”Evaluation and Program Planning, 1979

    1979

  3. [3]

    Concrete Problems in AI Safety

    D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man´ e, “Concrete problems in AI safety,” arXiv:1606.06565, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  4. [4]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,”NeurIPS, 2017

    2017

  5. [5]

    Proximal Policy Optimization Algorithms

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017. 14

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  6. [6]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,

    Y. Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,”ICML, 2016

    2016

  7. [7]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” ICML, 2017

    2017

  8. [8]

    Fine-Tuning Language Models from Human Preferences

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” arXiv:1909.08593, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  9. [9]

    Learning to summarize with human feedback,

    N. Stiennon et al., “Learning to summarize with human feedback,”NeurIPS, 2020

    2020

  10. [10]

    Specification gaming: The flip side of AI ingenuity,

    V. Krakovna et al., “Specification gaming: The flip side of AI ingenuity,” DeepMind Blog, 2020

    2020

  11. [11]

    A General Language Assistant as a Laboratory for Alignment

    A. Askell et al., “A general language assistant as a laboratory for alignment,” arXiv:2112.00861, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  12. [12]

    Training language models to follow instructions with human feedback,

    L. Ouyang et al., “Training language models to follow instructions with human feedback,”NeurIPS, 2022

    2022

  13. [13]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Y. Bai et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv:2204.05862, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  14. [14]

    Constitutional AI: Harmlessness from AI Feedback

    Y. Bai et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv:2212.08073, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    Defining and characterizing reward gaming,

    J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger, “Defining and characterizing reward gaming,”NeurIPS, 2022

    2022

  16. [16]

    Scaling laws for reward model overoptimization,

    L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,”ICML, 2023

    2023

  17. [17]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”NeurIPS, 2023

    2023

  18. [18]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,

    L. Zheng et al., “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,”NeurIPS Datasets and Benchmarks, 2023

    2023

  19. [19]

    Feedback loops with language models drive in-context reward hacking,

    A. Pan et al., “Feedback loops with language models drive in-context reward hacking,” arXiv:2309.04509, 2023

    work page arXiv 2023

  20. [20]

    Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

    Y. Dubois et al., “Length-controlled AlpacaEval: A simple way to debias automatic evaluators,” arXiv:2404.04475, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Open problems and fundamental limitations of reinforcement learning from human feedback,

    S. Casper et al., “Open problems and fundamental limitations of reinforcement learning from human feedback,”TMLR, 2023

    2023

  22. [22]

    Understanding the Effects of RLHF on LLM Generalisation and Diversity

    H. R. Kirk et al., “Understanding the effects of RLHF on LLM generalisation and diversity,” arXiv:2310.06452, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    T¨ ulu 2: Advancing language model post-training with preference datasets and methods,

    N. Lambert et al., “T¨ ulu 2: Advancing language model post-training with preference datasets and methods,” arXiv:2403.13208, 2024. 15 T able 7:Checkpoint-level transition counts. Setting Stable RH OC PUA CS MA Eval. game β= 0.001 sampled PPO 3 0 1 0 0 0 3 β= 0.005 PPO 0 1 0 1 0 0 1 β= 0.01 DPO 0 0 0 0 0 2 1 β= 0.01 PPO 0 0 1 1 0 0 1 β= 0.01 SFT 0 0 0 0 0...

    work page arXiv 2024

  24. [24]

    T ax software is generally con- sidered the most widely used op- tion... H&R Block and T urbo- T ax

    Mode T ax software ad- vice UP-PPO 1000 → 1200 “T ax software is generally con- sidered the most widely used op- tion... H&R Block and T urbo- T ax...” “UberT ax is a large online tax service... LyftT ax is often used as a tax helper software.” −0.375 → −0.198 (6,7)→(2,4) Reward hacking W orkplace bias complaint UP-PPO 600 → 1000 “...a constructive and re...

  25. [25]

    Table 11 reports this event by unit of analysis and failure 18 T able 9:Sensitivity of failure-mode counts to the sign tolerance ϵ

    < 0. Table 11 reports this event by unit of analysis and failure 18 T able 9:Sensitivity of failure-mode counts to the sign tolerance ϵ. RH denotes reward hacking; SA stable alignment; OC optimization collapse; PUA proxy under-alignment; CS conservative stagnation; MA mixed/ambiguous; EG evaluator gaming. UnitϵTotal RH SA OC PUA CS MA EG Checkpoint 10 −8 ...

    1920