{"id":"1962ef52-8608-4a6d-bdc9-7218f0b99d87","arxiv_id":"2505.14674","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Reward models that think before judging, trained via reinforcement learning without human-written reasoning traces, outperform standard reward models and improve with more test-time compute.","lead":"This paper trains language models that reason step by step before deciding which of two answers is better, and reports that these reward reasoning models beat standard reward models on several benchmarks. A generalist reader might care because the recipe offers a way to make AI feedback systems more accurate and to scale their judgment with extra computation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline PPE gains may be inflated by train/eval distribution overlap: the §4.1 mixture uses Tülu 3 prompts and verifiable math QA pairs that are close to MMLU-Pro/MATH/GPQA, and DirectJudge shares the same possibly contaminated data, so it cannot control for this.","rationale":"I read the paper's central claim as threefold: reasoning-before-reward improves reward accuracy, the improvement scales with test-time compute, and the resulting models are practical for best-of-N and post-training. The most controlled evidence for the first and strongest part is the DirectJudge comparison; the largest headline numbers are the PPE tables. The reader's conditional verdict rests largely on label quality and leakage. I agree that leakage is the load-bearing half: the training signal in Eq. 1 and the evaluation objective in §4.3 are the same 'pick the correct/preferred response' task, and the training sources are drawn from the same families as the evaluation benchmarks. Label bias from GPT-4o is a real but secondary issue, because RewardBench/PandaLM are human-preference benchmarks and training on a biased judge would tend to hurt, not inflate, performance there. The proposed retraining-with-deduplication check is concrete and would settle whether the PPE margins are genuine. Because the overlap is not yet quantified, this does not force rejection; it strengthens the condition the reader already imposed. Hence I keep the verdict unchanged (CONDITIONAL) and mark partial agreement with the reader's weakest-assumption framing.","tokens_in":20888,"tokens_out":10688,"duration_ms":100427,"concrete_test":"Reconstruct the training mixture from §4.1 (or use the released data), normalize prompts, and compute exact and fuzzy 8-gram containment between every training prompt and the evaluation prompts in PPE (MMLU-Pro, MATH, GPQA), RewardBench, and PandaLM. Then remove all training examples whose prompt overlaps any evaluation query (and, separately, remove only near-duplicate math problems), retrain RRM-7B with the same hyperparameters as §A.2, and rerun Tables 2 and 3. If the PPE margins over GPT-4o and J1-Llama shrink substantially or vanish, the headline superiority is a contamination artifact; if the margins persist on the deduplicated subset, the concern is resolved. A cheaper first pass is to run the released RRM-7B on the PPE queries with no 8-gram overlap to the training mixture and compare accuracy to the full set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's claims of state-of-the-art best-of-N and binary preference accuracy on PPE are the strongest evidence for 'superior performance across diverse domains'. But the training construction in §4.1 is nearly the same task as that evaluation. The 180K synthetic pairs are built by generating responses and then keeping correct-incorrect pairs from WebInstruct-verified, Skywork-OR1, Big-Math-RL, and DAPO-Math; these sources are heavily math/reasoning and are known to contain MATH-family and benchmark-style problems. The 80K Tülu 3 prompts are sampled from a large post-training instruction set that historically contains MMLU-style/GPQA-style/MATH-style prompts. Eq. 1 rewards the model exactly for choosing the ground-truth-preferred response, so on PPE's best-of-N and binary-preference tasks the model is rewarded on the same skill and, potentially, near-duplicate prompts. DirectJudge is trained on the same mixture, so the reasoning-versus-direct gap does not remove this confound; both models could be boosted by memorized or near-memorized content, and the 10-point reasoning gain could be amplified by access to memorized answers during longer reasoning. If leakage exists, the advantage over GPT-4o and J1-Llama in Tables 2–3 is not a general reward-modeling result but a distribution-overlap artifact, and the 'diverse domains' generalization claim is not established. The paper reports no overlap audit and no deduplication against PPE or RewardBench.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reward Reasoning Models (RRMs), which are trained by rule-based reinforcement learning (GRPO) to emit a chain-of-thought before making a pairwise preference judgment between two responses. Training data are constructed from Skywork-Reward, Tülu 3 prompts labeled by GPT-4o, and verifiable math/reasoning sources, without explicit reasoning traces. The paper evaluates RRMs on RewardBench, PandaLM Test, and Preference Proxy Evaluations (MMLU-Pro, MATH, GPQA), reports best-of-N and binary preference gains over scalar and generative reward baselines, demonstrates RL post-training and DPO with RRM rewards, and studies parallel and sequential test-time scaling. The central claim is that reasoning before rewarding, trained without reasoning annotations, yields superior and test-time-scalable reward modeling.","tokens_in":21193,"tokens_out":6573,"duration_ms":56552,"significance":"If the central claim holds, RRMs offer a simple and reproducible recipe for improving reward models: initialize from a reasoning model, train with GRPO on pairwise preference accuracy, and spend more test-time compute on hard comparisons. The paper's controlled DirectJudge comparison (same base models and training data, no reasoning) is a valuable design that directly supports the specific claim that the reasoning phase, not merely the training data, drives the gains. The release of pretrained models and the demonstration of downstream RL/DPO use are additional strengths. The main caveats are evaluation-based: possible train/eval overlap with PPE and RewardBench, a data-mixture confound between the 7B and 32B models, and absent uncertainty quantification, so the significance of the headline numbers is currently bounded by those gaps.","major_comments":[{"comment":"The training mixture described in §4.1 overlaps in content with the PPE evaluation used for the headline claims in Tables 2 and 3. The 180K synthetic pairs are constructed from WebInstruct-verified, Skywork-OR1, Big-Math-RL, and DAPO-Math, and the 80K Tülu 3 prompts are sampled from a post-training instruction set; PPE evaluates reward models on MMLU-Pro, MATH, and GPQA, and RewardBench contains chat, reasoning, and safety pairs. The paper reports no deduplication or overlap audit between these training sources and the evaluation sets, and Eq. (1) rewards exactly the pairwise preference judgment that PPE measures. Because DirectJudge is trained on the same mixture, it controls for the reasoning-vs-direct comparison but cannot control for absolute contamination, so the reported superiority over GPT-4o and J1-Llama in Tables 2–3 may be inflated by memorized or near-duplicate content. Please provide an n-gram or embedding overlap analysis and rerun the headline evaluations after strict deduplication, or evaluate on a fresh held-out reward benchmark.","section":"§4.1, §4.3"},{"comment":"RRM-7B and RRM-32B are trained on different data mixtures: RRM-32B uses a 5:1:1:1 ratio that includes the 180K synthetic verifiable pairs, while RRM-7B uses a 5:1:1 ratio that omits them. This confounds model capacity with training data throughout Tables 1–3 and Figures 5–6, and it is not clear whether DirectJudge-7B and DirectJudge-32B also differ in this way. Please train both sizes on the identical mixture, or report a same-mixture ablation, so that the scaling and size comparisons isolate the intended variable.","section":"§4.1"},{"comment":"The sequential test-time scaling experiment is calibrated and evaluated on the same benchmark: Appendix C sets the post-thinking budget to 100 tokens using the measured post-thinking token-length distribution of RRM-32B on RewardBench, and §4.5.2 then reports longer-thinking gains on RewardBench. This is a form of evaluation-protocol tuning on the test set; the chosen budget can affect how often reasoning is truncated and therefore the apparent benefit of longer thinking. Please validate the sequential-scaling conclusion on a held-out set, or report results across a range of post-thinking budgets (e.g., 50, 100, 200 tokens) to show the conclusion is robust.","section":"§4.5.2, Appendix C"},{"comment":"No variance or significance information is reported for the headline accuracy numbers, and Table 3 explicitly uses a single random permutation. Many adjacent comparisons are small (e.g., RRM-7B GPQA 49.0 vs 49.2 in Table 2; RRM-32B Chat Hard 81.1 vs 81.4 in Table 1), so the claims of consistent improvements are not statistically supported as reported. Please report standard errors over multiple permutation seeds for the PPE binary-preference and best-of-N results, and bootstrap confidence intervals or significance tests for the main RewardBench and PandaLM comparisons.","section":"Tables 1–3"},{"comment":"The training framework treats GPT-4o annotations and rule-based verifier labels as ground truth for the reward in Eq. (1), but the paper provides no analysis of the quality or bias of these labels, e.g., agreement between GPT-4o and human judges on a sample of the Tülu 3 pairs. Since the stated goal is alignment with human preference, a systematic label-bias analysis, or at least a human-agreement sample, is needed to support the claim that the learned preferences generalize to human judgment rather than to GPT-4o's idiosyncrasies.","section":"§4.1, Eq. (1)"}],"minor_comments":[{"comment":"The text says RRM-32B attains an accuracy of 98.6 in the reasoning category, but the non-voting RRM-32B row in Table 1 shows 98.3; 98.6 appears in the voting@16 row. Please clarify which configuration is being cited.","section":"§4.2.2, Table 1"},{"comment":"The DeepSeek-R1 row contains '73,7' with a comma instead of '73.7'; please fix the typo.","section":"Table 1"},{"comment":"The sentence 'providing robust ground truth for for evaluating alignment' has a duplicated 'for'.","section":"§4.2.1"},{"comment":"The word 'post-traineded' should be 'post-trained'.","section":"§4.4.1"},{"comment":"The column header 'MA TH' should be 'MATH' in both tables.","section":"Tables 2–3"},{"comment":"The Tülu dataset is cited as reference [34], but [34] is RewardBench; the Tülu 3 dataset is reference [33]. Please correct the citation.","section":"§4.4.2"},{"comment":"The sentence 'Unlike conventional supervised fine-tuning approaches, which relies on existing reasoning traces' has a subject-verb agreement error; 'which relies' should be 'which rely'.","section":"§3.2"},{"comment":"The caption states that underlined numbers indicate the second best, but the table as typeset does not show underlines; please ensure the formatting is visible.","section":"Table 1"},{"comment":"The statement 'all the responses require fewer than 100 tokens' is based on the RRM-32B distribution; the corresponding distributions for RRM-7B and RRM-14B are not shown, so the fixed 100-token budget for all model sizes is not fully justified by the presented evidence.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript is a strong empirical contribution, and the reasoning-vs-direct controlled comparison is the right experiment for the central claim. My main concern is contamination/overlap: because the training mixture includes Tülu 3 prompts and math sources close to the PPE benchmarks, the absolute SOTA claims over GPT-4o and J1-Llama need a dedup audit before they can be taken at face value. If the authors can supply that analysis and fix the data-mixture confound between the 7B and 32B models, I would be comfortable with acceptance; otherwise the paper should be revised to soften the generalization claims. This is a standard correctness-risk concern, not an accusation of misconduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper does something genuinely useful: it trains a generative reward model to think before it judges, using rule-based RL with no reasoning traces, and it checks the effect with a same-data control (DirectJudge). That control shows a real, if mixed, benefit for reasoning—about 7 points on RewardBench's reasoning category and 2 points on PandaLM agreement. The ELO and knockout procedures are practical, well-specified, and come with complexity bounds. So there is solid engineering here.\n\nSecond, the headline PPE numbers should be treated with suspicion until audited. The training mix uses Tülu 3 prompts and verifiable math/reasoning sources (Big-Math-RL, DAPO-Math, Skywork-OR1) that sit very close to MMLU-Pro, MATH, and GPQA, and the paper reports no deduplication or overlap check against those benchmarks. DirectJudge, the control, is not run on PPE. So the 10-point best-of-N gap over GPT-4o could be partly distribution overlap rather than general reward-reasoning ability. This is the weak point, and it is the actual weak point—not a manufactured one.\n\nOther soft spots are minor. There are no error bars anywhere, and Table 3 reports a single random permutation, so the binary-preference SOTA claims rest on a coin flip's worth of variance. RRM-7B and RRM-32B are trained on different mixtures, so the scaling story is confounded. The text says 98.6 where Table 1 says 98.3. The 'adaptive' compute language overstates: they show accuracy improves with larger thinking budgets, not that the model chooses to spend more on hard questions. And the post-thinking budget is calibrated on RewardBench and then applied to RewardBench, which is mildly circular.\n\nNone of that kills the central claim. The DirectJudge comparison on RewardBench and PandaLM is clean enough to believe that reasoning-before-judgment is worth doing. But the paper's most impressive numbers—the PPE table—are exactly the ones with the weakest controls.\n\nWho should read it: anyone building reward models for RLHF or best-of-N. It deserves a serious referee: the method is reproducible in principle, the related work is honest, and the weaknesses are addressable. I'd send it to review with a strong request for a contamination audit, error bars, and DirectJudge numbers on PPE.","headline":"A solid empirical demonstration that reasoning before reward judgment helps, with a good same-data control, but the headline PPE numbers carry a real train/eval overlap risk that needs auditing before they are taken at face value.","tokens_in":21764,"tokens_out":4150,"would_cite":true,"duration_ms":37493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-thought reward models beat static preference scorers","keywords":["reward models","chain-of-thought reasoning","reinforcement learning","test-time compute scaling","preference modeling","LLM-as-a-judge","best-of-N inference","GRPO"],"falsifier":"Relabel a random sample of the synthesized training pairs with human annotators using the same forced-choice format; if human–GPT-4o agreement is near chance on the pairs the model is trained to get right, the rule-based reward is not a faithful preference proxy. Also compute exact and near-duplicate overlap between the training queries and the evaluation prompts; substantive overlap would make the reported benchmark gaps uninformative.","tokens_in":20675,"feed_emoji":"🧠","tokens_out":8611,"duration_ms":70083,"temperature":0.7,"pith_summary":"The paper argues that reward modeling should be treated as a reasoning task rather than a scoring task. It introduces Reward Reasoning Models, which think out loud before declaring which of two responses is better, and trains them with reinforcement learning using only the binary correctness of the final preference — no hand-written reasoning traces. The reported results show these models outperform scalar reward models and standard LLM judges on reward-modeling benchmarks across chat, reasoning, safety, and human-preference domains, and that they convert additional test-time compute into better judgments. If this holds, alignment pipelines could replace fixed scalar reward heads with reasoning policies whose compute budget can be tuned per query.","feed_headline":"Chain-of-thought reward models beat static preference scorers","feed_subtitle":"Reasoning before judging lifts accuracy on MMLU-Pro, MATH, and GPQA and scales with test-time compute.","key_machinery":"The central object is the Reward Reasoning Model: a decoder-only language model that ingests a query and exactly two candidate responses, autoregressively writes a free-form chain of thought, and then commits to '\\boxed{Assistant 1}' or '\\boxed{Assistant 2}'. Training uses the Reward Reasoning via Reinforcement Learning framework: group relative policy optimization (GRPO) maximizes a rule-based reward that is +1 when the final choice matches the ground-truth preference and −1 otherwise, so no reasoning traces are ever shown to the model. To reward more than two responses, the paper adds two aggregation schemes — round-robin ELO rating and a knockout tournament — and combines either with majority voting over repeated samples to consume additional test-time compute.","core_discovery":"The central claim is that injecting a deliberate reasoning phase before a pairwise preference judgment turns a reward model into a better judge, and that this reasoning ability can be induced by reinforcement learning with a rule-based reward that only checks the final choice. Concretely, the paper reports that a 32B parameter RRM reaches 91.2 overall on the RewardBench suite, an average of 77.4 (rising to 79.7 with voting@5) on the Preference Proxy Evaluations best-of-N tasks, and accuracy of 81.3 on MMLU-Pro, 95.4 on MATH, and 68.4 on GPQA in binary preference classification with voting@5, exceeding a 70B instruction-tuned LLM. The paper also shows that reward accuracy improves as the number of pairwise comparisons grows and as the thinking token budget grows, and that the trained model's reasoning exhibits more transition, reflection, and comparison patterns than the base model.","pith_inferences":["If RRM-style reasoning generalizes, non-verifiable tasks that currently rely on scalar reward heads could be served by a configurable reasoning policy whose compute budget is chosen per query, making reward-guided decoding competitive with process-level supervision.","A natural extension not explored in the paper is a two-stage verifier that uses a cheap scalar model to filter candidates and an RRM only for the remaining hard comparisons; the reported scaling curves suggest this would preserve most of the accuracy gain at lower cost.","Because training rewards only the final choice, the reasoning trace itself is unconstrained; a stress test would be to mask or remove the thinking tokens at inference and measure whether accuracy collapses, which would reveal how much of the gain is genuinely due to reasoning.","The smaller gain from majority voting on GPQA compared with MATH hints that optimal test-time budgets should be domain-adaptive, which the paper's framework could support but does not implement."],"forward_implications":["Reward-guided best-of-N inference selects better responses than scalar reward models and prompted frontier LLMs on reasoning-heavy benchmarks, with the largest RRM beating a 70B instruction-tuned model on binary preference classification.","RRM-generated rewards improve downstream post-training: reinforcement learning on unlabeled queries raises GPQA accuracy over training steps, and DPO with RRM labels outperforms DPO with labels from a frontier proprietary judge on an instruction-following benchmark.","Test-time compute helps in two orthogonal ways: more pairwise comparisons (parallel scaling) and a larger thinking-token budget (sequential scaling) both raise reward accuracy across model sizes.","The ELO aggregation is slightly more accurate than the knockout tournament, but the tournament needs only O(n) comparisons rather than O(n²), giving a practical efficiency–accuracy knob.","Trained RRMs shift their reasoning patterns toward transition, reflection, and comparison relative to the base model, indicating the reinforcement procedure actively reshapes evaluation behavior."],"supporting_citations":[{"why":"Supplies the base models and the rule-based reinforcement-learning paradigm on which RRM training builds.","marker":"[22]"},{"why":"Provides the primary evaluation suite and its prompt template.","marker":"[34]"},{"why":"Provides the proxy evaluation protocol, the 32-candidate response sets for best-of-N, and the binary preference classification setup.","marker":"[18]"},{"why":"Source of training preference pairs and a strong scalar reward model baseline.","marker":"[39]"},{"why":"Concurrent reasoning-style reward model baselines and the judge prompt used for one evaluation setup.","marker":"[41]"},{"why":"Cited as the source of the GRPO training algorithm.","marker":"[70]"},{"why":"Produces the preference labels used to build part of the training data and serves as a judge baseline.","marker":"[26]"},{"why":"The reinforcement learning implementation library used for training runs.","marker":"[53]"}],"fun_headline_variants":["Think first, then judge: reward models reason","Reasoning reward models beat static preference scorers","Test-time thinking lifts reward model accuracy","Chain-of-thought rewards: better with compute","Reward models that reason outrank 70B baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire training signal and every headline number rest on the assumption that the GPT-4o-generated preference labels and the rule-based verifier labels used for training are unbiased proxies for what human raters would prefer, and that none of the training queries overlap with the RewardBench, PandaLM, or PPE evaluation sets.","fun_headline_variants_meta":{"raw":{"variants":["Think first, then judge: reward models reason","Reasoning reward models beat static preference scorers","Test-time thinking lifts reward model accuracy","Chain-of-thought rewards: better with compute","Reward models that reason outrank 70B baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":1120,"prompt_tokens":878,"completion_tokens":242,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":170}},"tokens_in":494,"tokens_out":242,"duration_ms":3487,"temperature":1.0,"reasoning_tokens":170,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:29:32.749923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Relabel a random sample of the synthesized training pairs with human annotators using the same forced-choice format; if human–GPT-4o agreement is near chance on the pairs the model is trained to get right, the rule-based reward is not a faithful preference proxy. Also compute exact and near-duplicate overlap between the training queries and the evaluation prompts; substantive overlap would make the reported benchmark gaps uninformative.","supporting_citations":[{"cited_title":"The limits of automatic summarisation according to rouge","cited_arxiv_id":null,"evidence_quote":"The reinforcement learning implementation library used for training runs."}],"review_version":1}