{"id":"269c8b0b-0092-415c-beba-7d7354da401c","arxiv_id":"2506.16712","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReasonGRM uses a likelihood-based metric R* to select correct, high-confidence reasoning paths for supervised fine-tuning and then fine-tunes on hard cases with GRPO, reaching an average score of 83.3 across RewardBench, RM-Bench, and RMB.","lead":"This paper presents ReasonGRM, a three-stage training pipeline that turns a large reasoning model into a better reward model by generating, filtering, and retraining on high-quality reasoning paths. It reports state-of-the-art scores on three reward-modeling benchmarks and claims that a new likelihood-based metric, R*, improves supervised fine-tuning over random data selection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"R* filtering is confounded with length and style; require matched-control ablation before accepting the core claim.","rationale":"The reader's weakest_assumption correctly identifies the R* likelihood proxy as the fragile link, and my concern matches that identification. I diverge slightly in emphasis: the reader phrases the risk as potential non-transfer across generators, while I see a sharper, more testable confound within Table 3 itself—R* selection is compared only against random selection, so the measured gain conflates self-consistency with length/style. Both readings share the same underlying load-bearing assumption, so partial agreement is accurate. I do not escalate to REJECT because Stage 2's result, if the length/style confound were controlled, could still support a weaker claim: filtering for concise correct rationales helps SFT. The paper's internal inconsistencies and missing error bars support CONDITIONAL rather than ACCEPT. The concrete test is feasible without new benchmark infrastructure: only the Qwen2.5-7B SFT setup and a matched-sampling script are needed. If the matched control closes the gap, the R* contribution is not established, and the paper would need recharacterization as a length/style-aware filtering study. If the gap persists, the R* mechanism gains credibility. The recommendation CONDITIONAL matches the reader's verdict, but with a more specific condition attached to the Stage-2 ablation.","tokens_in":16625,"tokens_out":2484,"duration_ms":20951,"concrete_test":"Run the Stage-2 ablation on Qwen2.5-7B with three equal-size training sets drawn from the same correct-path pool: (1) R*-selected top paths, (2) uniformly random correct paths, (3) random correct paths matched to R* selections on length distribution and token-frequency perplexity. If set (3) matches set (1) within noise, R* adds nothing beyond a conciseness/style filter. Additionally, collect human quality ratings for 50 sampled paths and compare against R* rankings to verify the metric tracks logical quality beyond length.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central methodological claim is that R* selects reasoning paths that are both valid (correct answer) and self-consistent (high average token probability), and that this improves GRM training beyond random sampling of correct paths (Section 3.3, Table 3). The load-bearing assumption is that R*'s product of average conditional token probabilities is a faithful proxy for reasoning quality. This is insecure for two concrete reasons. First, average token probability is strongly confounded with length and style: shorter paths typically have higher per-token likelihood, and formulaic or template-like text is consistently high-probability regardless of logical quality. The paper provides no length-conditional analysis, no calibration evidence, and no human validation that R* rankings track logical quality. Second, the Table 3 comparison only controls for answer correctness, not for length, token difficulty, or style distribution. Random sampling of correct paths will draw longer, more exploratory reasoning traces than R*'s argmax, so the measured SFT gain could be 'train on concise correct rationales' rather than 'train on self-consistent rationales'. The authors' own Limitations section admits R* only applies to well-defined QA pairs, which further signals that the metric is not a general reasoning-quality measure. Since Stage 2 is the only component tested for cross-model generalization, the stability of this assumption is the main support for the framework's transferability. The headline SOTA claim also lacks error bars and code, but the R* confound is the more fundamental threat to the paper's central contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReasonGRM proposes a three-stage training pipeline for generative reward models (GRMs): (1) Zero-RL, a GRPO stage that adapts a large reasoning model (LRM) to preference tasks using outcome-only rewards; (2) R*-guided supervised fine-tuning, where R* scores candidate reasoning paths by the product of the average token probability of the reasoning path and the average token probability of its answer, and only the highest-scoring correct paths are used; and (3) GRPO refinement on hard cases. The authors report state-of-the-art average performance of 83.3 across RewardBench, RM-Bench, and RMB (Table 1), outperforming prior GRMs by 1.8% and GPT-4o by 5.6%, and they present ablations (Table 3) showing R*-filtered SFT outperforms random SFT on three base models. The paper also includes case studies and full benchmark tables.","tokens_in":16860,"tokens_out":2659,"duration_ms":29374,"significance":"If the central claims hold, the paper makes a practical contribution to GRM training: it provides a fully open, non-proprietary pipeline for turning an LRM into a competitive reward model, and it proposes a simple likelihood-based criterion for filtering reasoning traces. The cross-model generalization results in Table 3 are a useful and somewhat unusual strength, since they test the R* filtering signal on architectures different from the generator. The paper also ships detailed benchmark tables and a case study, which help the reader assess behavior qualitatively. However, the significance is currently tempered by the lack of uncertainty quantification and by the conceptual fragility of R* as a proxy for reasoning quality; these issues need to be addressed before the claims can be accepted as stated.","major_comments":[{"comment":"The definition of R* as the product of average token probabilities is not a faithful operationalization of 'self-consistency' as the term is normally used (agreement across repeated sampling), and it is likely confounded with length and style. Short, template-like reasoning paths will tend to have higher average per-token likelihood even when they are logically weaker than longer, exploratory paths. The Table 3 comparison controls only for answer correctness (§4.5.2), not for length, token difficulty, or style, so the measured gains could reflect 'train on concise correct rationales' rather than 'train on self-consistent rationales.' To support the central claim, the authors need a matched-control ablation—for example, random sampling of correct paths matched to R*-selected paths in length distribution, or an independent quality label (human or LLM-judge) used as a second filter.","section":"Section 3.3, Equation (1)"},{"comment":"The reported RM-Bench score for ReasonGRM is internally inconsistent: Table 1 lists 86.3, while Table 5 lists 86.8. Since the Table 1 average of 83.3 depends on this number, the inconsistency directly affects the headline SOTA claim. The authors should reconcile the two tables and verify that the average is computed from the same set of scores.","section":"Tables 1 and 5"},{"comment":"The paper states that R*-based SFT 'significantly outperformed' random SFT, but no variance, confidence intervals, or significance tests are reported anywhere. Each condition appears to be a single run. Given that the benchmark differences in Table 3 are small for Llama3.1-8B (82.11 vs 81.85) and partly reversed in the Chat and Safety categories, the claim of significance is not supported. The authors should either provide repeated-run statistics or soften the language to 'consistently improved' with the caveat that the effect is not statistically verified.","section":"Section 4.5.2 / Table 3"},{"comment":"The paper's own Limitations section acknowledges that R* applies only to well-defined QA pairs and not to open-ended answers. This is a scope restriction, but it also exposes a deeper issue: the likelihood-based R* cannot separate 'confident and correct' from 'confident and wrong' without an external correctness filter, and it cannot assess whether the reasoning is logically valid when the answer is open-ended. The authors should state explicitly that R* is a ranking criterion for candidate correct answers, not a general reasoning-quality measure, and should temper the claim that it 'effectively addresses the data quality bottleneck' for GRM training.","section":"Section 3.3 / Limitations"}],"minor_comments":[{"comment":"In the Skywork-Reward-Gemma-2-27B row, the entry '54..9' is a typo and should be '54.9'.","section":"Table 5"},{"comment":"The caption contains the typo 'proceeds' written as 'roceeds' in the workflow description.","section":"Figure 3 caption"},{"comment":"The heading 'B.2 Scalar Reward Models' appears to be a copy-paste error; the section actually describes Generative Reward Models.","section":"Appendix B.2"},{"comment":"Several references are listed twice (e.g., GPT-4 technical report appears as both Achiam et al. 2023a and 2023b), and some citations are given in non-standard formats (e.g., 'Xiaoyu Tan Minghao Yang, Chao Qu. 2024' in Appendix B.1). The reference list should be cleaned up for consistency.","section":"Throughout"},{"comment":"The RMB benchmark description says it 'encompassing over 49 distinct, fine-grained real-world scenarios'; this should be 'encompasses' and the phrase should be simplified.","section":"Section 4.2"},{"comment":"The label 'ReasoneGRM' in Figure 6 is a typo for 'ReasonGRM'.","section":"Appendix C / Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is plausible and the three-benchmark evaluation is useful, but the central R* metric needs a stronger empirical defense than the current random-SFT baseline. The internal RM-Bench inconsistency in Tables 1 and 5 is a correctness issue that must be fixed before resubmission. I would encourage the editor to send this back for a revision rather than reject, because the Table 3 cross-model experiments are a potentially valuable control if properly interpreted with variance estimates. I would also flag that the paper's novelty relative to RM-R1 and Self-Taught Evaluators is incremental but sufficient for a workshop or applied venue; the authors should more clearly state what is new beyond those prior works."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ReasonGRM is a genuinely useful empirical result, not a conceptual breakthrough. The new thing is R*: choose the correct reasoning path with highest product of average token probabilities from a reward-specialized LRM, SFT on that, then GRPO on hard cases. That pipeline lifts average benchmark score to 83.3, about 1.8 over the previous best GRM. The cross-model R*-versus-random ablation on Llama3.1-8B, Qwen2.5-7B, and Qwen2.5-14B is the strongest evidence; it controls for answer correctness and shows a consistent gain.\n\nI buy the direction. The soft spots are real but not fatal. First, R* is just a likelihood score, and average per-token likelihood is mechanically higher for shorter, more templated text. The Table 3 comparisons do not control for length or style distributions, so the benefit could be 'train on concise correct rationales' rather than 'train on self-consistent rationales'. I would want a length-matched control (for example, binning by path length) before treating the self-consistency interpretation as established. The authors' own Limitations section admits R* only fits well-defined QA pairs, which supports the concern.\n\nSecond, the reporting is sloppy. No variance or significance tests appear anywhere. ReasonGRM's RM-Bench score is 86.3 in Table 1 but 86.8 in Table 5. The RMB table lists Skywork-Reward-Llama-3.1-8B as 69.5/60.6/54.5/95.7/70.1, which is exactly its RewardBench row from Table 4—an obvious copy-paste error. The abstract says Zero-RL 'generates concise reasoning paths,' but Zero-RL is outcome-only GRPO with no reasoning text. These are fixable, but they make the headline less trustable as printed.\n\nThe citation pattern is fine; they cite the relevant GRM and RM literature, and self-citations are not a problem. Novelty is modest—R* is a likelihood heuristic—but the empirical comparison is a legitimate new result.\n\nWho this is for: anyone building GRMs or using reasoning traces to train evaluators. I would send it to peer review; the core claim survives, but reviewers should require the length/style control, corrected tables, and ideally code and data release. I would not cite it until those corrections exist.","headline":"A useful empirical recipe for GRM training with a plausible but unproven likelihood-filtering mechanism; deserves review but needs a length-matched control and cleaner tables.","tokens_in":17452,"tokens_out":2076,"would_cite":false,"duration_ms":20696,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a likelihood-based filter, R*, over reasoning paths is what lifts generative reward models to state-of-the-art preference judgment.","keywords":["generative reward models","reasoning paths","R* metric","preference alignment","reinforcement learning","data filtering","reward model training","reasoning-aware training"],"falsifier":"Have independent human raters rank a pool of correct-answer reasoning paths by logical clarity and absence of speculation, then compare that ranking to R*; if the agreement is at chance, or if supervised fine-tuning on the lowest-R* paths performs as well as fine-tuning on the highest-R* paths on RewardBench, the R* selection is not carrying the reported gains.","tokens_in":16414,"feed_emoji":"🧠","tokens_out":8672,"duration_ms":81262,"temperature":0.7,"pith_summary":"This paper tries to establish that the bottleneck in generative reward models is not just whether they reason, but which reasoning they are trained on. It proposes ReasonGRM, a three-stage recipe: first train a large reasoning model to pick correct answers using outcome-only reinforcement learning, then use a new score, R*, to select concise, confident reasoning paths that end in correct answers, and finally run reinforcement learning on hard cases. The authors report that this pipeline reaches an average of 83.3 on RewardBench, RM-Bench, and RMB, beating the previous best generative reward model by 1.8 points and GPT-4o by 5.6 points. If true, this matters because reward models are the steering signal for aligning language models, and a reliable way to choose high-quality rationales would remove a major data-quality bottleneck without needing proprietary teachers.","feed_headline":"Filtered reasoning paths push reward models past GPT-4o","feed_subtitle":"A likelihood-score filter, R*, selects high-confidence rationales, beating prior reward models by 1.8 points on average.","key_machinery":"The central object is $R^\\star$, a likelihood-based score for a reasoning path plus its final answer: $$R^\\star(R'_g,a_g,Q)=\\frac{1}{L_{R'_g}}\\sum_{i=1}^{L_{R'_g}} p'_{g,i}\\;\\cdot\\;\\frac{1}{L_{a_g}}\\sum_{i=1}^{L_{a_g}} p''_{g,i}.$$ It is the product of the average conditional token probability of the reasoning tokens (called Self-Consistency) and the average conditional token probability of the answer tokens (called Validity). The pipeline uses it to pick, for each question, the reasoning-answer pair with the highest score among those with correct answers, and those pairs become the supervised fine-tuning set. The other load-bearing pieces are the outcome-only Zero-RL stage with its sample filter and the final hard-case GRPO stage.","core_discovery":"The central claim is a specific causal chain: reasoning quality in a generative reward model is separable and trainable. ReasonGRM's Stage 1 applies GRPO with an outcome-only reward and a sample filter that skips updates when a question's sampled answers are all correct or all wrong, producing a judge (LRM-Zero) that has learned correct choices without seeing any reasoning text. Stage 2 defines $R^\\star$ as the product of the average generation probability of the reasoning tokens and the average generation probability of the answer tokens, multiplying the two to favor paths that are both confident and correct, and uses the top-scoring path per question as supervised fine-tuning data. Stage 3 runs GRPO again, but only on questions where the Stage-2 model gave mixed answers. The paper reports that this full pipeline yields the best average score across three benchmarks and that the $R^\\star$-filtered SFT beats random SFT on every base model tested.","pith_inferences":["Extension: if R* works by selecting high-confidence paths, the same scoring rule could select rationales for chain-of-thought distillation, verifier training, or self-improvement in settings without preference labels; the paper only demonstrates it for reward models.","Extension: because R* is computed under the same generator that produced the paths, it may inherit that model's length and style priors; the transfer test in Table 3 uses a fixed generator and varying student models, so a stronger test would cross generators.","Testable prediction: if R* is genuinely measuring self-consistency rather than just favoring shorter paths, then among correct paths of equal length, higher-R* paths should yield larger SFT gains; a short-path control would settle this."],"forward_implications":["R*-filtered SFT beats random SFT on RewardBench for every base model tested, so the filter is transferable across model families and scales.","Outcome-only RL before any rationale SFT gives a measurable RewardBench gain, so a reward model learns to judge before it learns to justify.","Hard-case GRPO after SFT adds the final accuracy gain, meaning where you apply RL matters for preference discrimination.","On RM-Bench, reasoning-optimized reward models show much smaller performance drops than strong scalar reward models, so general reasoning ability is the transferable asset for subtle style-versus-substance judgments."],"supporting_citations":[{"why":"Supplies the Skywork-Reward preference dataset of about 80K samples used as the initial training data for all stages.","marker":"Liu et al., 2024a"},{"why":"Defines RewardBench, one of the three evaluation benchmarks on which ReasonGRM is compared.","marker":"Lambert et al., 2024"},{"why":"Defines RM-Bench, the benchmark emphasizing subtle-content and style robustness where ReasonGRM reports its largest gains.","marker":"Liu et al., 2024b"},{"why":"Defines RMB, the third evaluation benchmark using pairwise and Best-of-N protocols.","marker":"Zhou et al., 2024"},{"why":"Provides the cold-start inspiration and the reasoning-model RL methodology the pipeline adapts.","marker":"DeepSeek-AI et al., 2025a"},{"why":"Introduces generative reward models, the class of models ReasonGRM improves.","marker":"Mahan et al., 2024"},{"why":"RM-R1 is the closest reasoning-optimized reward-model baseline that ReasonGRM outperforms.","marker":"Chen et al., 2025b"},{"why":"Establishes the LLM-as-judge / GRM evaluation paradigm and prompt-based preference format.","marker":"Zheng et al., 2023"}],"fun_headline_variants":["R* filter selects confident reasoning, lifts reward models 1.8%","ReasonGRM uses R* to filter reasoning, beating GPT-4o by 5.6%","Three-stage reward model outperforms GPT-4o with R*-filtered rationales","Selective rationales via R* push generative reward models past GPT-4o","Confidence-scored reasoning paths lift reward models over GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average generation probability of a reasoning path and its answer, as scored by R*, is a true measure of reasoning quality; if that probability mostly reflects length, style, or quirks of the generating model, the filtering gains will not transfer to other models or tasks.","fun_headline_variants_meta":{"raw":{"variants":["R* filter selects confident reasoning, lifts reward models 1.8%","ReasonGRM uses R* to filter reasoning, beating GPT-4o by 5.6%","Three-stage reward model outperforms GPT-4o with R*-filtered rationales","Selective rationales via R* push generative reward models past GPT-4o","Confidence-scored reasoning paths lift reward models over GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3404,"prompt_tokens":951,"completion_tokens":2453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2344}},"tokens_in":567,"tokens_out":2453,"duration_ms":19132,"temperature":1.0,"reasoning_tokens":2344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:19:59.491114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human raters rank a pool of correct-answer reasoning paths by logical clarity and absence of speculation, then compare that ranking to R*; if the agreement is at chance, or if supervised fine-tuning on the lowest-R* paths performs as well as fine-tuning on the highest-R* paths on RewardBench, the R* selection is not carrying the reported gains.","supporting_citations":[],"review_version":2}