REVIEW 4 major objections 3 minor 10 cited by
Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Cooper, a framework that co-trains the policy and reward model during RL, claims to reduce reward hacking and improve LLM reasoning accuracy, including a 0.54% gain on Qwen2.5-1.5B-Instruct.
desk verdict Cooper is a plausible co-optimization recipe for RL reward modeling, but the 0.54% headline and the noisy-label assumption need scrutiny before the empirical claim holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the online pair-construction loop: during RL rollout, rule-based rewards act as a high-precision filter that labels one response correct and another incorrect, and those dynamically constructed pairs are used to continue training the reward model while the policy is also updated. Two supporting components are a hybrid annotation strategy for creating reward-model training data, and reference-based reward modeling, in which VerifyRM receives a reference answer as an additional input. The loop is what carries the argument: it keeps the reward model aligned with the policy's current behavior and prevents the policy from exploiting a static reward signal.
What would settle it
Run Cooper and a fixed-reward baseline on a benchmark where rule-based correctness is known to be noisy, such as tasks with many legitimate solution paths. If VerifyRM's accuracy on a held-out set does not improve over training, or if Cooper's end-to-end accuracy does not beat the fixed-reward baseline, the central claim fails; likewise, if policy reward rises while true accuracy falls, reward hacking is not actually alleviated.
Extended reading notes
Core claim
The central claim is that dynamically updating the reward model during RL is an effective way to combat reward hacking. Rather than trusting either rule-based or model-based rewards alone, Cooper exploits the high precision of rule-based rewards in identifying correct responses to construct positive-negative training pairs, and uses these pairs to keep training the reward model as the policy changes. The paper reports that this co-optimization both reduces reward hacking and improves final RL performance, including a 0.54% gain in average accuracy on Qwen2.5-1.5B-Instruct. In support, Cooper introduces a hybrid annotation strategy for generating reward-model training data and a reference-bas
Load-bearing premise
The argument depends on rule-based rewards being precise enough to call responses correct or incorrect: if they frequently mislabel valid reasoning, the reward model trains on noise and Cooper's gain disappears.
Editorial extensions
If this is right
- If Cooper works, practitioners can update reward models online instead of fixing them, reducing reward hacking in long RL runs.
- Rule-based rewards, though coarse, can be repurposed as reliable pair labels to refine a learned reward model during training.
- Reference-based reward conditioning gives same-size reward models a measurable accuracy edge on VerifyBench.
- End-to-end RL gains, such as the 0.54% average accuracy improvement on Qwen2.5-1.5B-Instruct, can come from reward-model co-training rather than larger policy models.
Reading between the lines
- The approach likely transfers most cleanly to tasks where rule-based correctness is unambiguous (e.g., math with verifiable answers) and needs adaptation for open-ended generation where multiple correct answers exist.
- Dynamic reward updating could be combined with other anti-hacking techniques such as KL regularization or response ensembling, but the paper does not test this combination.
- A testable extension is to track VerifyRM's accuracy over training time: if updating is effective, its accuracy should follow the policy's improving outputs rather than diverge, and reward-hacking indicators should stay flat.
- The reference-based paradigm may help reward models generalize to unseen tasks by comparing against a reference, though this goes beyond the paper's experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Cooper, a reinforcement learning framework that co-optimizes the policy model and a reward model for reasoning LLMs. The method uses rule-based rewards to construct high-confidence positive/negative pairs, continues training the reward model during RL, and introduces a reference-based reward modeling paradigm. The abstract also announces a new reward model, VerifyRM, claimed to achieve higher accuracy on VerifyBench, and reports that Cooper alleviates reward hacking and improves end-to-end RL performance, e.g., a 0.54% average accuracy gain on Qwen2.5-1.5B-Instruct.
Significance. If the claims are verified, the idea of dynamically updating the reward model during RL could be a meaningful step toward mitigating reward hacking in model-based reward paradigms. The reference-based reward modeling and the hybrid annotation strategy are plausible contributions. However, the abstract provides only one headline accuracy number without statistical support, and the key assumption that rule-based rewards have high precision for pair construction is not tested. As presented, the contribution is promising but not established.
major comments (4)
- [Abstract, 'achieving a 0.54% gain in average accuracy...'] The only quantitative result is a 0.54% average accuracy gain over a single model. No standard error, confidence interval, number of independent RL runs, or per-task breakdown is reported. At this effect size, sampling variability and RL seed sensitivity can easily change the sign. The claim that Cooper improves end-to-end RL performance is not supported until multiple seeds with variance and a significance test are reported.
- [Abstract, 'leverages the high precision of rule-based rewards...'] The load-bearing premise is that rule-based rewards identify correct responses with sufficient precision to create reliable training pairs. For reasoning tasks, exact final-answer matching can label a response with a correct final answer but flawed derivation as positive, and a wrong final answer with valid intermediate steps as negative. The abstract gives no evidence that the hybrid annotation strategy removes this noise. Please report reward-model accuracy stratified by reasoning correctness, or provide an ablation with human-audited pairs.
- [Abstract, 'not only alleviates reward hacking but also...'] Reward hacking is a process-level claim, but the abstract offers no quantitative measure. To make the alleviation claim empirically grounded, report reward-model accuracy on held-out validation during RL, policy divergence from the reference, or reward over-optimization curves. Without such evidence, the statement remains qualitative.
- [Abstract, 'VerifyRM achieves higher accuracy on VerifyBench...'] This claim is not interpretable without describing VerifyBench's construction, composition, and baselines. If the benchmark is introduced in this paper, its evaluation protocol must be specified to judge whether the accuracy comparison is fair. Please include benchmark statistics, baseline model information, and statistical significance.
minor comments (3)
- [Abstract, first sentence] There is a missing space in 'Cooper(Co-optimizing'; should be 'Cooper (Co-optimizing'.
- [Abstract, 'two mainstream reward paradigms'] The term 'mainstream' is vague; specify representative baselines, e.g., outcome reward models and process reward models, so the reader can situate the contribution.
- [Abstract, 'hybrid annotation strategy'] The abstract does not state what sources are hybridized or how the strategy is validated. A sentence on the annotation protocol and its cost/accuracy trade-off would help the reader assess the contribution.
Circularity Check
No circularity identified from abstract-only evidence
full rationale
The audit is limited to the abstract because full text was not available. The claimed derivation chain is: rule-based rewards are assumed to have high precision in identifying correct responses; these labels are used to dynamically construct positive/negative training pairs for the reward model; the resulting VerifyRM and Cooper are then evaluated in RL and on VerifyBench. Each step is anchored to an external signal rather than to the claim itself. The final RL evaluation is a separate end-to-end benchmark result (0.54% average accuracy gain on Qwen2.5-1.5B-Instruct), not a restatement of the reward-model training objective. VerifyRM's accuracy on VerifyBench is an evaluation against a benchmark, not a definitional consequence of the training procedure. No equation is provided in the abstract that would show a predicted quantity reducing to a fitted input, and no self-citation or imported uniqueness theorem is invoked. Concerns about label noise in rule-based rewards, the provenance of VerifyBench, and the absence of variance reporting are correctness/robustness issues, not evidence of circularity. Therefore, under the requirement to exhibit a specific reduction before flagging circularity, no circular step is present in the available text.
Assumptions & free parameters
assumptions (3)
- domain assumption Rule-based rewards have high precision in identifying correct responses.
- domain assumption Continuing to train the reward model during policy RL does not destabilize training.
- domain assumption Reference-based reward modeling improves reward accuracy.
Cite this review
Pith. "Pith review of Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models." pith.science (2026). https://pith.science/paper/PJ3CBNLB
@misc{pith2026250805613,
author = {Pith},
title = {Pith review of: Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PJ3CBNLB}},
note = {Machine review of arXiv:2508.05613}
}
read the original abstract
Large language models (LLMs) have demonstrated remarkable performance in reasoning tasks, where reinforcement learning (RL) serves as a key algorithm for enhancing their reasoning capabilities. Currently, there are two mainstream reward paradigms: model-based rewards and rule-based rewards. However, both approaches suffer from limitations: rule-based rewards lack robustness, while model-based rewards are vulnerable to reward hacking. To address these issues, we propose Cooper(Co-optimizing Policy Model and Reward Model), a RL framework that jointly optimizes both the policy model and the reward model. Cooper leverages the high precision of rule-based rewards when identifying correct responses, and dynamically constructs and selects positive-negative sample pairs for continued training the reward model. This design enhances robustness and mitigates the risk of reward hacking. To further support Cooper, we introduce a hybrid annotation strategy that efficiently and accurately generates training data for the reward model. We also propose a reference-based reward modeling paradigm, where the reward model takes a reference answer as input. Based on this design, we train a reward model named VerifyRM, which achieves higher accuracy on VerifyBench compared to other models of the same size. We conduct reinforcement learning using both VerifyRM and Cooper. Our experiments show that Cooper not only alleviates reward hacking but also improves end-to-end RL performance, for instance, achieving a 0.54% gain in average accuracy on Qwen2.5-1.5B-Instruct. Our findings demonstrate that dynamically updating reward model is an effective way to combat reward hacking, providing a reference for better integrating reward models into RL.
Forward citations
Cited by 10 Pith papers
-
ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents
ARCO introduces a co-evolving rubric model with generation and scoring heads plus a trajectory decomposition constraint that improves exact-match scores on multi-hop QA tasks over outcome, rubric, and process reward b...
-
ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents
ARCO co-trains an LLM agent with a per-step rubric model whose natural-language criteria and scoring function evolve with the policy, and reports the best exact-match accuracy in all six benchmark/backbone settings compared.
-
Epistemic Uncertainty for Test-Time Discovery
UG-TTT adds epistemic uncertainty measured by adapter disagreement as an exploration bonus in RL for LLMs, raising maximum reward and diversity on scientific discovery benchmarks.
-
CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning
CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.
-
CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning
A cooperative Coach–Player RL loop, rewarded by Player validation-progress, lifts math reasoning by 2–5 points while the Player trains only on self-generated pseudo-labeled tasks.
-
Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.
-
Trust Region On-Policy Distillation
TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.
-
Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
The paper introduces the Proxy Compression Hypothesis as a unifying framework explaining reward hacking in RLHF as an emergent result of compressing high-dimensional human objectives into proxy reward signals under op...
-
From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI
Autonomous AI becomes dependable when tool use is embedded in persistent workspaces with reusable skills, shifting evaluation from answers to task closure.
-
A Survey of Reinforcement Learning for Large Reasoning Models
A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.