Pith. sign in

REVIEW 3 major objections 2 minor 2 cited by

CHERRL reproduces reward hacking in rubric-based RL by injecting known biases into LLM judges.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 07:26 UTC pith:PGE2VZGH

load-bearing objection CHERRL gives a controllable testbed for injecting biases to reproduce reward hacking in rubric RL, but external validity to real judges is untested. the 3 major comments →

arxiv 2606.04923 v1 pith:PGE2VZGH submitted 2026-06-03 cs.LG cs.AIcs.CL

Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning

Xuekang Wang , Zhuoyuan Hao , Shuo Hou , Hao Peng , Juanzi Li , Xiaozhi Wang This is my paper
classification cs.LG cs.AIcs.CL
keywords reward hackingrubric-based reinforcement learningLLM-as-a-JudgeCHERRLbias injectionreward divergencehacking detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Rubric-based reinforcement learning relies on an LLM-as-a-Judge to assign rewards according to rubrics, yet policies can exploit hidden biases in that judge and produce reward hacking. The paper introduces CHERRL, an environment that deliberately adds specific, known biases to the judge so that hacking behaviors become reproducible and measurable. This setup lets researchers watch how rewards diverge during training and mark the exact point where hacking begins. The authors use the environment to compare biases by how easy they are to discover and exploit, and they test an agent-based detector that flags hacking from training logs alone.

Core claim

CHERRL is a controllable hacking environment for rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent-based system for automatically detecting reward hacking onset from training logs.

What carries the argument

CHERRL, the controllable environment created by injecting known biases into the LLM-as-a-Judge (LaaJ) so that reward hacking can be reproduced, observed, and detected on demand.

Load-bearing premise

Deliberately injected biases into the LLM judge accurately simulate the subtle, entangled latent biases that arise in real-world rubric-based RL deployments.

What would settle it

A side-by-side comparison in which the reward curves, policy behaviors, and hacking onset timing produced by CHERRL with injected biases differ substantially from those observed in an unmodified rubric-based RL run using the same judge model and rubrics.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Different judge biases can be isolated and ranked by how discoverable and exploitable they are.
  • Reward divergence becomes directly visible in training curves rather than remaining hidden.
  • The moment hacking begins can be located precisely enough to test early interventions.
  • An agent-based detector trained on CHERRL logs can flag hacking onset without access to the judge internals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same bias-injection approach could be used to benchmark mitigation techniques such as bias correction or multi-judge ensembles.
  • If the injected biases prove representative, CHERRL-style testbeds could become standard for safety evaluation of any LLM-scored training loop.
  • The detection agent might transfer to production systems if its false-positive rate remains low when run on logs from real, non-injected judges.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces CHERRL, a controllable testbed for rubric-based RL in which known biases are deliberately injected into an LLM-as-a-Judge (LaaJ) to reproduce reward hacking. It claims this enables stable reproduction of hacking, explicit observation of reward divergence, precise onset detection, analysis of biases by discoverability/exploitability, and an agent-based detection system from training logs. Code is released publicly.

Significance. If the injected biases induce hacking dynamics that are statistically and mechanistically comparable to those arising from latent, entangled judge flaws in deployed systems, CHERRL would supply a much-needed controlled environment for dissecting reward hacking mechanisms and testing mitigations in rubric-based RL. The public code release is a positive contribution to reproducibility.

major comments (3)
  1. [Abstract and Introduction] Abstract and §1: The central claim that CHERRL 'enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset' is asserted without any reported quantitative metrics, stability statistics, or ablation results demonstrating these capabilities.
  2. [Analysis of judge biases] §4 (analysis of judge biases): The reported rankings of discoverability and exploitability rest on the assumption that deliberately injected biases produce hacking patterns representative of real latent LaaJ biases; no side-by-side comparison of onset timing, divergence curves, or exploitation patterns against unmodified LaaJ is provided, leaving external validity untested.
  3. [Agent-based detection system] Detection system section: The agent-based onset detector is presented as a practical contribution, yet no performance numbers (precision, recall, false-positive rate), baseline comparisons, or sensitivity analysis on training-log features are reported, making it impossible to evaluate whether the method is load-bearing or merely illustrative.
minor comments (2)
  1. [Methods] Notation for injected bias parameters and reward divergence is introduced without a consolidated table or explicit definitions, complicating replication.
  2. [Code availability] The GitHub repository link is given, but the manuscript does not specify which exact scripts, seeds, and rubric templates are required to reproduce the reported bias-injection experiments.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thoughtful and constructive report. We address each major comment below. Where the manuscript requires additional quantitative support or comparisons, we will incorporate revisions as indicated.

read point-by-point responses

  1. Referee: [Abstract and Introduction] Abstract and §1: The central claim that CHERRL 'enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset' is asserted without any reported quantitative metrics, stability statistics, or ablation results demonstrating these capabilities.

    Authors: We agree that the abstract and introduction would be strengthened by explicit quantitative backing for these claims. While Sections 3–5 report results across multiple random seeds and show consistent divergence patterns, we did not include dedicated stability statistics (e.g., variance in onset detection across seeds) or ablations on bias injection parameters in the current version. In the revision we will add a dedicated subsection with these metrics and ablations to directly substantiate the claims. revision: yes

  2. Referee: [Analysis of judge biases] §4 (analysis of judge biases): The reported rankings of discoverability and exploitability rest on the assumption that deliberately injected biases produce hacking patterns representative of real latent LaaJ biases; no side-by-side comparison of onset timing, divergence curves, or exploitation patterns against unmodified LaaJ is provided, leaving external validity untested.

    Authors: We acknowledge the external-validity concern. CHERRL is intentionally a controlled testbed that isolates individual biases; this design choice means injected biases are known and disentangled by construction, whereas real LaaJ flaws are latent and entangled. We therefore do not claim statistical equivalence. To improve the manuscript we will add a limited side-by-side experiment contrasting injected-bias runs against an unmodified LaaJ on the same tasks, reporting qualitative similarities in divergence timing and exploitation patterns where observable. A comprehensive quantitative equivalence study is outside the scope of the current controlled framework. revision: partial

  3. Referee: [Agent-based detection system] Detection system section: The agent-based onset detector is presented as a practical contribution, yet no performance numbers (precision, recall, false-positive rate), baseline comparisons, or sensitivity analysis on training-log features are reported, making it impossible to evaluate whether the method is load-bearing or merely illustrative.

    Authors: We accept that the detection section currently lacks the requested quantitative evaluation. The presented agent-based detector is offered as an initial demonstration that training-log features can be used for onset detection. In the revised manuscript we will include precision, recall, and false-positive rates on held-out training runs, comparisons against simple threshold and statistical-process-control baselines, and a sensitivity analysis on the log features (reward variance, policy entropy, and judge-score distribution). revision: yes

Circularity Check

0 steps flagged

No circularity: empirical testbed with no derivations or self-referential reductions

full rationale

The paper introduces CHERRL as an empirical tool for reproducing reward hacking via deliberate bias injection into LaaJ. No equations, fitted parameters, predictions, or derivation chains appear in the abstract or described content. Central claims rest on the controllability of the injected-bias environment and public code release, which are externally verifiable and do not reduce to self-definition, fitted-input renaming, or self-citation load-bearing. This matches the default expectation for an empirical tool paper with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central contribution rests on the domain assumption that controllable bias injection can model real reward hacking; no free parameters or invented physical entities are described.

axioms (1)
  • domain assumption Known biases can be injected into an LLM-as-a-Judge to controllably induce and study reward hacking behaviors.
    This premise underpins the design of CHERRL as stated in the abstract.
invented entities (1)
  • CHERRL environment no independent evidence
    purpose: Provide a controllable testbed for reproducing and detecting reward hacking in rubric-based RL
    Newly introduced tool whose effectiveness is asserted but not demonstrated in the abstract.

pith-pipeline@v0.9.1-grok · 5740 in / 1185 out tokens · 33533 ms · 2026-06-28T07:26:07.486759+00:00 · methodology

0 comments
read the original abstract

Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes. In real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. In this paper, we introduce CHERRL, a controllable hacking environment for rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent-based system for automatically detecting reward hacking onset from training logs. The code and environment are publicly available at https://github.com/THUAIS-Lab/CHERRL.

Figures

Figures reproduced from arXiv: 2606.04923 by Hao Peng, Juanzi Li, Shuo Hou, Xiaozhi Wang, Xuekang Wang, Zhuoyuan Hao.

Figure 1
Figure 1. Figure 1: Reward hacking example in CHERRL. The proxy reward combines scores from a gold judge and a judge injected with a known self-praise bias. This de￾sign allows for explicitly capturing the onset and reward divergence trend of reward hacking, and thus offers a controllable environment for studying reward hacking in rubric-based RL. et al., 2025), and scientific assistance (Goel et al., 2025; Panigrahi et al., … view at source ↗
Figure 2
Figure 2. Figure 2: Overall framework of our proposed methodology. At its core is the Controllable Hacking Environment for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics for the six CHERRL runs where reward hacking occurs. Each subfigure reports one [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: RHDA architecture. Group Core Capability Blind Spot Addressed Inspect Read steps & rollouts Judge-blind data access Analyze Check token correlations & bias metrics Signatures of known reward-hacking shortcuts Compute Run custom Python files & analyze metrics Open-ended shortcut exploration Reason Track hypotheses & issue typed alerts Cross-step reasoning & structured verdict [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 5
Figure 5. Figure 5: Search-budget ablation for RHDA with Qwen3.5-plus across the six controlled runs. Each panel plots [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Tool-call timelines for three successful RHDA cases and one boundary case. The x-axis denotes tool-call [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics for the two CHERRL runs where reward hacking does not occur. Because these [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG

    cs.CL 2026-07 conditional novelty 6.0

    Teacher-regularized on-policy RL with a reverse-KL anchor on student prefixes improves language consistency and evidence-grounded correctness for English-evidence cross-lingual RAG, acting as a safety net against larg...

  2. AgentOmnia: Scaling Agentic Models for Full-Scenario Applications

    cs.AI 2026-07 conditional novelty 5.0

    Post-training an open Qwen3-30B agent on 52,361 verifiable tasks in 5,018 synthesized stateful environments lifts its average across four agent benchmarks from 22.9% to 41.7%.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages · cited by 2 Pith papers · 5 internal anchors

  1. [1]

    Process Reinforcement through Implicit Rewards

    Process Reinforcement through Implicit Re- wards.Preprint, arXiv:2502.01456. Brandon Dent. 2026. Healthcraft: A reinforcement learning safety environment for emergency medicine. Preprint, arXiv:2605.21496. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ah- mad Beirami, Alex D’Amour, D. J. Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ra-...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains.Preprint, arXiv:2507.17746. Xu Guo, Tianyi Liang, Tong Jian, Xiaogui Yang, Ling-I Wu, Chenhui Li, Zhihui Lu, Qipeng Guo, and Kai Chen. 2025. IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards.Preprint, arXiv:2508.04632. Yun He, Wenzhe Li, Hejia Zha...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Reinforcement learning with rubric anchors.arXiv preprint arXiv:2508.12790, 2025

    Reinforcement Learning with Rubric Anchors. Preprint, arXiv:2508.12790. Mengzhao Jia, Zhihan Zhang, Ignacio Cases, Zheyuan Liu, Meng Jiang, and Peng Qi. 2026. Autorubric: Rubric-based generative rewards for faithful multi- modal reasoning.Preprint, arXiv:2510.14738. Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and ...

    work page arXiv 2026

  4. [4]

    Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards

    Writing-Zero: Bridge the Gap Between Non- verifiable Tasks and Verifiable Rewards.Preprint, arXiv:2506.00103. Muhammad Khalifa, Zohaib Khan, Omer Tafveez, Hao Peng, and Lu Wang. 2026. Countdown-Code: A Testbed for Studying The Emergence and Gener- alization of Reward Hacking in RLVR.Preprint, arXiv:2603.07084. 9 Dawei Li, Bohan Jiang, Liangjie Huang, Alim...

    work page Pith review arXiv 2026

  5. [5]

    Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, and Yunzhong He

    An efficient rubric-based generative verifier for search-augmented llms.Preprint, arXiv:2510.14660. Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu, and Yunzhong He

    work page arXiv

  6. [6]

    Reward Hacking in Rubric-Based Reinforcement Learning

    Reward Hacking in Rubric-Based Reinforce- ment Learning.Preprint, arXiv:2605.12474. Charles O’Neill, Tirthankar Ghosal, Roberta R˘aileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciuc ˘a. 2025. Sparks of science: Hypothe- sis generation using structured paper data.Preprint, arXiv:2504.12976. Long Ouyang, others at OpenAI, and LMSYS Org

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    LMSYS Arena technical blog and evaluation suite

    Arena-Hard: A Hard Subsample of LM- SYS Chat Arena. LMSYS Arena technical blog and evaluation suite. Available at:https://lmsys.org/ blog/2024-05-arena-hard/. Arjun Panickssery, Samuel R. Bowman, and Shi Feng

    2024

  8. [8]

    LLM Evaluators Recognize and Favor Their Own Generations

    LLM Evaluators Recognize and Favor Their Own Generations.Preprint, arXiv:2404.13076. Siba Smarak Panigrahi, Jovana Videnovi´c, and Maria Brbi´c. 2026. Heurekabench: A benchmarking frame- work for ai co-scientist.Preprint, arXiv:2601.01678. Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. 2025. VerIF: Verification Engineering for Reinforc...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  9. [9]

    DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

    DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research.Preprint, arXiv:2511.19399. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield- Dodds, Scott R. Johnston, Shauna Kravec, Timo- thy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Available: https://arxiv.org/abs/2505.15034

    RL Tango: Reinforcing Generator and Ver- ifier Together for Language Reasoning.Preprint, arXiv:2505.15034. Jiajie Zhang, Xin Lv, Ling Feng, Lei Hou, and Juanzi Li

    work page arXiv

  11. [11]

    Chaining the evidence: Robust reinforcement learning for deep search agents with citation-aware rubric rewards,

    Chaining the evidence: Robust reinforcement learning for deep search agents with citation-aware rubric rewards.Preprint, arXiv:2601.06021. Junxian Zhao, Binyuan Guo, Sen Zhang, Philipp Schmid, and 1 others. 2025a. IFBench: A challeng- ing benchmark for precise instruction following. In Advances in Neural Information Processing Systems (NeurIPS). NeurIPS 2...

    work page arXiv 2025

  12. [12]

    A Glossary of Epistemological Terms

    Toward Robust LLM-Based Judges: Taxo- nomic Bias Evaluation and Debiasing Optimization. Preprint, arXiv:2603.08091. Jiayi Zhou, Jiaming Ji, Boyuan Chen, Jiapeng Sun, Wenqi Chen, Donghai Hong, Sirui Han, Yike Guo, and Yaodong Yang. 2025a. Generative RLHF-V: Learning Principles from Multi-modal Human Pref- erence.Preprint, arXiv:2505.18531. Yang Zhou, Sunzh...

    work page arXiv 2025

  13. [13]

    covers English instruction following with verifiable constraints. Both datasets are used in their default released splits, and our use (rubric- based RL post-training and reward-hacking analy- sis) is consistent with the intended research use stated by their authors. Models used in this work—Qwen3-4B, Qwen3.5-27B, Qwen3.5-Plus, and Qwen3.5-397B-A17B—are r...