pith. sign in

arxiv: 2606.04923 · v1 · pith:PGE2VZGHnew · submitted 2026-06-03 · 💻 cs.LG · cs.AI· cs.CL

Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning

Xuekang Wang , Zhuoyuan Hao , Shuo Hou , Hao Peng , Juanzi Li , Xiaozhi Wang This is my paper

Pith reviewed 2026-06-28 07:26 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords reward hackingrubric-based reinforcement learningLLM-as-a-JudgeCHERRLbias injectionreward divergencehacking detection
0
0 comments X

The pith

CHERRL reproduces reward hacking in rubric-based RL by injecting known biases into LLM judges.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Rubric-based reinforcement learning relies on an LLM-as-a-Judge to assign rewards according to rubrics, yet policies can exploit hidden biases in that judge and produce reward hacking. The paper introduces CHERRL, an environment that deliberately adds specific, known biases to the judge so that hacking behaviors become reproducible and measurable. This setup lets researchers watch how rewards diverge during training and mark the exact point where hacking begins. The authors use the environment to compare biases by how easy they are to discover and exploit, and they test an agent-based detector that flags hacking from training logs alone.

Core claim

CHERRL is a controllable hacking environment for rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent-based system for automatically detecting reward hacking onset from training logs.

What carries the argument

CHERRL, the controllable environment created by injecting known biases into the LLM-as-a-Judge (LaaJ) so that reward hacking can be reproduced, observed, and detected on demand.

If this is right

  • Different judge biases can be isolated and ranked by how discoverable and exploitable they are.
  • Reward divergence becomes directly visible in training curves rather than remaining hidden.
  • The moment hacking begins can be located precisely enough to test early interventions.
  • An agent-based detector trained on CHERRL logs can flag hacking onset without access to the judge internals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same bias-injection approach could be used to benchmark mitigation techniques such as bias correction or multi-judge ensembles.
  • If the injected biases prove representative, CHERRL-style testbeds could become standard for safety evaluation of any LLM-scored training loop.
  • The detection agent might transfer to production systems if its false-positive rate remains low when run on logs from real, non-injected judges.

Load-bearing premise

Deliberately injected biases into the LLM judge accurately simulate the subtle, entangled latent biases that arise in real-world rubric-based RL deployments.

What would settle it

A side-by-side comparison in which the reward curves, policy behaviors, and hacking onset timing produced by CHERRL with injected biases differ substantially from those observed in an unmodified rubric-based RL run using the same judge model and rubrics.

Figures

Figures reproduced from arXiv: 2606.04923 by Hao Peng, Juanzi Li, Shuo Hou, Xiaozhi Wang, Xuekang Wang, Zhuoyuan Hao.

Figure 1
Figure 1. Figure 1: Reward hacking example in CHERRL. The proxy reward combines scores from a gold judge and a judge injected with a known self-praise bias. This de￾sign allows for explicitly capturing the onset and reward divergence trend of reward hacking, and thus offers a controllable environment for studying reward hacking in rubric-based RL. et al., 2025), and scientific assistance (Goel et al., 2025; Panigrahi et al., … view at source ↗
Figure 2
Figure 2. Figure 2: Overall framework of our proposed methodology. At its core is the Controllable Hacking Environment for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics for the six CHERRL runs where reward hacking occurs. Each subfigure reports one [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: RHDA architecture. Group Core Capability Blind Spot Addressed Inspect Read steps & rollouts Judge-blind data access Analyze Check token correlations & bias metrics Signatures of known reward-hacking shortcuts Compute Run custom Python files & analyze metrics Open-ended shortcut exploration Reason Track hypotheses & issue typed alerts Cross-step reasoning & structured verdict [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 5
Figure 5. Figure 5: Search-budget ablation for RHDA with Qwen3.5-plus across the six controlled runs. Each panel plots [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Tool-call timelines for three successful RHDA cases and one boundary case. The x-axis denotes tool-call [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics for the two CHERRL runs where reward hacking does not occur. Because these [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
read the original abstract

Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes. In real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. In this paper, we introduce CHERRL, a controllable hacking environment for rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent-based system for automatically detecting reward hacking onset from training logs. The code and environment are publicly available at https://github.com/THUAIS-Lab/CHERRL.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces CHERRL, a controllable testbed for rubric-based RL in which known biases are deliberately injected into an LLM-as-a-Judge (LaaJ) to reproduce reward hacking. It claims this enables stable reproduction of hacking, explicit observation of reward divergence, precise onset detection, analysis of biases by discoverability/exploitability, and an agent-based detection system from training logs. Code is released publicly.

Significance. If the injected biases induce hacking dynamics that are statistically and mechanistically comparable to those arising from latent, entangled judge flaws in deployed systems, CHERRL would supply a much-needed controlled environment for dissecting reward hacking mechanisms and testing mitigations in rubric-based RL. The public code release is a positive contribution to reproducibility.

major comments (3)
  1. [Abstract and Introduction] Abstract and §1: The central claim that CHERRL 'enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset' is asserted without any reported quantitative metrics, stability statistics, or ablation results demonstrating these capabilities.
  2. [Analysis of judge biases] §4 (analysis of judge biases): The reported rankings of discoverability and exploitability rest on the assumption that deliberately injected biases produce hacking patterns representative of real latent LaaJ biases; no side-by-side comparison of onset timing, divergence curves, or exploitation patterns against unmodified LaaJ is provided, leaving external validity untested.
  3. [Agent-based detection system] Detection system section: The agent-based onset detector is presented as a practical contribution, yet no performance numbers (precision, recall, false-positive rate), baseline comparisons, or sensitivity analysis on training-log features are reported, making it impossible to evaluate whether the method is load-bearing or merely illustrative.
minor comments (2)
  1. [Methods] Notation for injected bias parameters and reward divergence is introduced without a consolidated table or explicit definitions, complicating replication.
  2. [Code availability] The GitHub repository link is given, but the manuscript does not specify which exact scripts, seeds, and rubric templates are required to reproduce the reported bias-injection experiments.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thoughtful and constructive report. We address each major comment below. Where the manuscript requires additional quantitative support or comparisons, we will incorporate revisions as indicated.

read point-by-point responses

  1. Referee: [Abstract and Introduction] Abstract and §1: The central claim that CHERRL 'enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset' is asserted without any reported quantitative metrics, stability statistics, or ablation results demonstrating these capabilities.

    Authors: We agree that the abstract and introduction would be strengthened by explicit quantitative backing for these claims. While Sections 3–5 report results across multiple random seeds and show consistent divergence patterns, we did not include dedicated stability statistics (e.g., variance in onset detection across seeds) or ablations on bias injection parameters in the current version. In the revision we will add a dedicated subsection with these metrics and ablations to directly substantiate the claims. revision: yes

  2. Referee: [Analysis of judge biases] §4 (analysis of judge biases): The reported rankings of discoverability and exploitability rest on the assumption that deliberately injected biases produce hacking patterns representative of real latent LaaJ biases; no side-by-side comparison of onset timing, divergence curves, or exploitation patterns against unmodified LaaJ is provided, leaving external validity untested.

    Authors: We acknowledge the external-validity concern. CHERRL is intentionally a controlled testbed that isolates individual biases; this design choice means injected biases are known and disentangled by construction, whereas real LaaJ flaws are latent and entangled. We therefore do not claim statistical equivalence. To improve the manuscript we will add a limited side-by-side experiment contrasting injected-bias runs against an unmodified LaaJ on the same tasks, reporting qualitative similarities in divergence timing and exploitation patterns where observable. A comprehensive quantitative equivalence study is outside the scope of the current controlled framework. revision: partial

  3. Referee: [Agent-based detection system] Detection system section: The agent-based onset detector is presented as a practical contribution, yet no performance numbers (precision, recall, false-positive rate), baseline comparisons, or sensitivity analysis on training-log features are reported, making it impossible to evaluate whether the method is load-bearing or merely illustrative.

    Authors: We accept that the detection section currently lacks the requested quantitative evaluation. The presented agent-based detector is offered as an initial demonstration that training-log features can be used for onset detection. In the revised manuscript we will include precision, recall, and false-positive rates on held-out training runs, comparisons against simple threshold and statistical-process-control baselines, and a sensitivity analysis on the log features (reward variance, policy entropy, and judge-score distribution). revision: yes

Circularity Check

0 steps flagged

No circularity: empirical testbed with no derivations or self-referential reductions

full rationale

The paper introduces CHERRL as an empirical tool for reproducing reward hacking via deliberate bias injection into LaaJ. No equations, fitted parameters, predictions, or derivation chains appear in the abstract or described content. Central claims rest on the controllability of the injected-bias environment and public code release, which are externally verifiable and do not reduce to self-definition, fitted-input renaming, or self-citation load-bearing. This matches the default expectation for an empirical tool paper with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central contribution rests on the domain assumption that controllable bias injection can model real reward hacking; no free parameters or invented physical entities are described.

axioms (1)
  • domain assumption Known biases can be injected into an LLM-as-a-Judge to controllably induce and study reward hacking behaviors.
    This premise underpins the design of CHERRL as stated in the abstract.
invented entities (1)
  • CHERRL environment no independent evidence
    purpose: Provide a controllable testbed for reproducing and detecting reward hacking in rubric-based RL
    Newly introduced tool whose effectiveness is asserted but not demonstrated in the abstract.

pith-pipeline@v0.9.1-grok · 5740 in / 1185 out tokens · 33533 ms · 2026-06-28T07:26:07.486759+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 5 linked inside Pith

  1. [1]

    Brandon Dent

    Process Reinforcement through Implicit Re- wards.Preprint, arXiv:2502.01456. Brandon Dent. 2026. Healthcraft: A reinforcement learning safety environment for emergency medicine. Preprint, arXiv:2605.21496. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ah- mad Beirami, Alex D’Amour, D. J. Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ra-...

    Pith/arXiv arXiv 2026

  2. [2]

    Xu Guo, Tianyi Liang, Tong Jian, Xiaogui Yang, Ling-I Wu, Chenhui Li, Zhihui Lu, Qipeng Guo, and Kai Chen

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains.Preprint, arXiv:2507.17746. Xu Guo, Tianyi Liang, Tong Jian, Xiaogui Yang, Ling-I Wu, Chenhui Li, Zhihui Lu, Qipeng Guo, and Kai Chen. 2025. IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards.Preprint, arXiv:2508.04632. Yun He, Wenzhe Li, Hejia Zha...

    Pith/arXiv arXiv 2025

  3. [3]

    Preprint, arXiv:2508.12790

    Reinforcement Learning with Rubric Anchors. Preprint, arXiv:2508.12790. Mengzhao Jia, Zhihan Zhang, Ignacio Cases, Zheyuan Liu, Meng Jiang, and Peng Qi. 2026. Autorubric: Rubric-based generative rewards for faithful multi- modal reasoning.Preprint, arXiv:2510.14738. Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and ...

    arXiv 2026

  4. [4]

    Muhammad Khalifa, Zohaib Khan, Omer Tafveez, Hao Peng, and Lu Wang

    Writing-Zero: Bridge the Gap Between Non- verifiable Tasks and Verifiable Rewards.Preprint, arXiv:2506.00103. Muhammad Khalifa, Zohaib Khan, Omer Tafveez, Hao Peng, and Lu Wang. 2026. Countdown-Code: A Testbed for Studying The Emergence and Gener- alization of Reward Hacking in RLVR.Preprint, arXiv:2603.07084. 9 Dawei Li, Bohan Jiang, Liangjie Huang, Alim...

    arXiv 2026

  5. [5]

    Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, and Yunzhong He

    An efficient rubric-based generative verifier for search-augmented llms.Preprint, arXiv:2510.14660. Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, and Yunzhong He

    arXiv

  6. [6]

    Charles O’Neill, Tirthankar Ghosal, Roberta R˘aileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciuc ˘a

    Reward Hacking in Rubric-Based Reinforce- ment Learning.Preprint, arXiv:2605.12474. Charles O’Neill, Tirthankar Ghosal, Roberta R˘aileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciuc ˘a. 2025. Sparks of science: Hypothe- sis generation using structured paper data.Preprint, arXiv:2504.12976. Long Ouyang, others at OpenAI, and LMSYS Org

    Pith/arXiv arXiv 2025

  7. [7]

    LMSYS Arena technical blog and evaluation suite

    Arena-Hard: A Hard Subsample of LM- SYS Chat Arena. LMSYS Arena technical blog and evaluation suite. Available at:https://lmsys.org/ blog/2024-05-arena-hard/. Arjun Panickssery, Samuel R. Bowman, and Shi Feng

    2024

  8. [8]

    Siba Smarak Panigrahi, Jovana Videnovi´c, and Maria Brbi´c

    LLM Evaluators Recognize and Favor Their Own Generations.Preprint, arXiv:2404.13076. Siba Smarak Panigrahi, Jovana Videnovi´c, and Maria Brbi´c. 2026. Heurekabench: A benchmarking frame- work for ai co-scientist.Preprint, arXiv:2601.01678. Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. 2025. VerIF: Verification Engineering for Reinforc...

    Pith/arXiv arXiv 2026

  9. [9]

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R

    DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research.Preprint, arXiv:2511.19399. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield- Dodds, Scott R. Johnston, Shauna Kravec, Timo- thy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer,...

    Pith/arXiv arXiv 2025

  10. [10]

    Jiajie Zhang, Xin Lv, Ling Feng, Lei Hou, and Juanzi Li

    RL Tango: Reinforcing Generator and Ver- ifier Together for Language Reasoning.Preprint, arXiv:2505.15034. Jiajie Zhang, Xin Lv, Ling Feng, Lei Hou, and Juanzi Li

    arXiv

  11. [11]

    Junxian Zhao, Binyuan Guo, Sen Zhang, Philipp Schmid, and 1 others

    Chaining the evidence: Robust reinforcement learning for deep search agents with citation-aware rubric rewards.Preprint, arXiv:2601.06021. Junxian Zhao, Binyuan Guo, Sen Zhang, Philipp Schmid, and 1 others. 2025a. IFBench: A challeng- ing benchmark for precise instruction following. In Advances in Neural Information Processing Systems (NeurIPS). NeurIPS 2...

    arXiv 2025

  12. [12]

    when the shortcut is obvious

    Toward Robust LLM-Based Judges: Taxo- nomic Bias Evaluation and Debiasing Optimization. Preprint, arXiv:2603.08091. Jiayi Zhou, Jiaming Ji, Boyuan Chen, Jiapeng Sun, Wenqi Chen, Donghai Hong, Sirui Han, Yike Guo, and Yaodong Yang. 2025a. Generative RLHF-V: Learning Principles from Multi-modal Human Pref- erence.Preprint, arXiv:2505.18531. Yang Zhou, Sunzh...

    arXiv 2025

  13. [13]

    covers English instruction following with verifiable constraints. Both datasets are used in their default released splits, and our use (rubric- based RL post-training and reward-hacking analy- sis) is consistent with the intended research use stated by their authors. Models used in this work—Qwen3-4B, Qwen3.5-27B, Qwen3.5-Plus, and Qwen3.5-397B-A17B—are r...