Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a likelihood-based filter, R*, over reasoning paths is what lifts generative reward models to state-of-the-art preference judgment.

desk verdict A useful empirical recipe for GRM training with a plausible but unproven likelihood-filtering mechanism; deserves review but needs a length-matched control and cleaner tables. read the letter →

arxiv 2506.16712 v1 pith:4EFXR5UN submitted 2025-06-20 cs.CL cs.AI

Bin Chen , Xinzge Gao , Chuanrui Hu , Penghang Yu , Hua Zhang , Bing-Kun Bao This is my paper · ORCID
classification cs.CLcs.AI
keywords generativerewardmodelsreasoningpathsR*metricpreferencealignmentreinforcementlearningdatafilteringmodeltrainingreasoning-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the bottleneck in generative reward models is not just whether they reason, but which reasoning they are trained on. It proposes ReasonGRM, a three-stage recipe: first train a large reasoning model to pick correct answers using outcome-only reinforcement learning, then use a new score, R*, to select concise, confident reasoning paths that end in correct answers, and finally run reinforcement learning on hard cases. The authors report that this pipeline reaches an average of 83.3 on RewardBench, RM-Bench, and RMB, beating the previous best generative reward model by 1.8 points and GPT-4o by 5.6 points. If true, this matters because reward models are the steering signal for aligning language models, and a reliable way to choose high-quality rationales would remove a major data-quality bottleneck without needing proprietary teachers.

What carries the argument

The central object is $R^\star$, a likelihood-based score for a reasoning path plus its final answer: $$R^\star(R'_g,a_g,Q)=\frac{1}{L_{R'_g}}\sum_{i=1}^{L_{R'_g}} p'_{g,i}\;\cdot\;\frac{1}{L_{a_g}}\sum_{i=1}^{L_{a_g}} p''_{g,i}.$$ It is the product of the average conditional token probability of the reasoning tokens (called Self-Consistency) and the average conditional token probability of the answer tokens (called Validity). The pipeline uses it to pick, for each question, the reasoning-answer pair with the highest score among those with correct answers, and those pairs become the supervised fine-tuning set. The other load-bearing pieces are the outcome-only Zero-RL stage with its sample filter and the final hard-case GRPO stage.

What would settle it

Have independent human raters rank a pool of correct-answer reasoning paths by logical clarity and absence of speculation, then compare that ranking to R*; if the agreement is at chance, or if supervised fine-tuning on the lowest-R* paths performs as well as fine-tuning on the highest-R* paths on RewardBench, the R* selection is not carrying the reported gains.

Watch

Extended reading notes

Core claim

The central claim is a specific causal chain: reasoning quality in a generative reward model is separable and trainable. ReasonGRM's Stage 1 applies GRPO with an outcome-only reward and a sample filter that skips updates when a question's sampled answers are all correct or all wrong, producing a judge (LRM-Zero) that has learned correct choices without seeing any reasoning text. Stage 2 defines $R^\star$ as the product of the average generation probability of the reasoning tokens and the average generation probability of the answer tokens, multiplying the two to favor paths that are both confident and correct, and uses the top-scoring path per question as supervised fine-tuning data. Stage 3 runs GRPO again, but only on questions where the Stage-2 model gave mixed answers. The paper reports that this full pipeline yields the best average score across three benchmarks and that the $R^\star$-filtered SFT beats random SFT on every base model tested.

Load-bearing premise

The load-bearing premise is that the average generation probability of a reasoning path and its answer, as scored by R*, is a true measure of reasoning quality; if that probability mostly reflects length, style, or quirks of the generating model, the filtering gains will not transfer to other models or tasks.

Editorial extensions

If this is right

  • R*-filtered SFT beats random SFT on RewardBench for every base model tested, so the filter is transferable across model families and scales.
  • Outcome-only RL before any rationale SFT gives a measurable RewardBench gain, so a reward model learns to judge before it learns to justify.
  • Hard-case GRPO after SFT adds the final accuracy gain, meaning where you apply RL matters for preference discrimination.
  • On RM-Bench, reasoning-optimized reward models show much smaller performance drops than strong scalar reward models, so general reasoning ability is the transferable asset for subtle style-versus-substance judgments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: if R* works by selecting high-confidence paths, the same scoring rule could select rationales for chain-of-thought distillation, verifier training, or self-improvement in settings without preference labels; the paper only demonstrates it for reward models.
  • Extension: because R* is computed under the same generator that produced the paths, it may inherit that model's length and style priors; the transfer test in Table 3 uses a fixed generator and varying student models, so a stronger test would cross generators.
  • Testable prediction: if R* is genuinely measuring self-consistency rather than just favoring shorter paths, then among correct paths of equal length, higher-R* paths should yield larger SFT gains; a short-path control would settle this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ReasonGRM proposes a three-stage training pipeline for generative reward models (GRMs): (1) Zero-RL, a GRPO stage that adapts a large reasoning model (LRM) to preference tasks using outcome-only rewards; (2) R*-guided supervised fine-tuning, where R* scores candidate reasoning paths by the product of the average token probability of the reasoning path and the average token probability of its answer, and only the highest-scoring correct paths are used; and (3) GRPO refinement on hard cases. The authors report state-of-the-art average performance of 83.3 across RewardBench, RM-Bench, and RMB (Table 1), outperforming prior GRMs by 1.8% and GPT-4o by 5.6%, and they present ablations (Table 3) showing R*-filtered SFT outperforms random SFT on three base models. The paper also includes case studies and full benchmark tables.

Significance. If the central claims hold, the paper makes a practical contribution to GRM training: it provides a fully open, non-proprietary pipeline for turning an LRM into a competitive reward model, and it proposes a simple likelihood-based criterion for filtering reasoning traces. The cross-model generalization results in Table 3 are a useful and somewhat unusual strength, since they test the R* filtering signal on architectures different from the generator. The paper also ships detailed benchmark tables and a case study, which help the reader assess behavior qualitatively. However, the significance is currently tempered by the lack of uncertainty quantification and by the conceptual fragility of R* as a proxy for reasoning quality; these issues need to be addressed before the claims can be accepted as stated.

major comments (4)
  1. [Section 3.3, Equation (1)] The definition of R* as the product of average token probabilities is not a faithful operationalization of 'self-consistency' as the term is normally used (agreement across repeated sampling), and it is likely confounded with length and style. Short, template-like reasoning paths will tend to have higher average per-token likelihood even when they are logically weaker than longer, exploratory paths. The Table 3 comparison controls only for answer correctness (§4.5.2), not for length, token difficulty, or style, so the measured gains could reflect 'train on concise correct rationales' rather than 'train on self-consistent rationales.' To support the central claim, the authors need a matched-control ablation—for example, random sampling of correct paths matched to R*-selected paths in length distribution, or an independent quality label (human or LLM-judge) used as a second filter.
  2. [Tables 1 and 5] The reported RM-Bench score for ReasonGRM is internally inconsistent: Table 1 lists 86.3, while Table 5 lists 86.8. Since the Table 1 average of 83.3 depends on this number, the inconsistency directly affects the headline SOTA claim. The authors should reconcile the two tables and verify that the average is computed from the same set of scores.
  3. [Section 4.5.2 / Table 3] The paper states that R*-based SFT 'significantly outperformed' random SFT, but no variance, confidence intervals, or significance tests are reported anywhere. Each condition appears to be a single run. Given that the benchmark differences in Table 3 are small for Llama3.1-8B (82.11 vs 81.85) and partly reversed in the Chat and Safety categories, the claim of significance is not supported. The authors should either provide repeated-run statistics or soften the language to 'consistently improved' with the caveat that the effect is not statistically verified.
  4. [Section 3.3 / Limitations] The paper's own Limitations section acknowledges that R* applies only to well-defined QA pairs and not to open-ended answers. This is a scope restriction, but it also exposes a deeper issue: the likelihood-based R* cannot separate 'confident and correct' from 'confident and wrong' without an external correctness filter, and it cannot assess whether the reasoning is logically valid when the answer is open-ended. The authors should state explicitly that R* is a ranking criterion for candidate correct answers, not a general reasoning-quality measure, and should temper the claim that it 'effectively addresses the data quality bottleneck' for GRM training.
minor comments (6)
  1. [Table 5] In the Skywork-Reward-Gemma-2-27B row, the entry '54..9' is a typo and should be '54.9'.
  2. [Figure 3 caption] The caption contains the typo 'proceeds' written as 'roceeds' in the workflow description.
  3. [Appendix B.2] The heading 'B.2 Scalar Reward Models' appears to be a copy-paste error; the section actually describes Generative Reward Models.
  4. [Throughout] Several references are listed twice (e.g., GPT-4 technical report appears as both Achiam et al. 2023a and 2023b), and some citations are given in non-standard formats (e.g., 'Xiaoyu Tan Minghao Yang, Chao Qu. 2024' in Appendix B.1). The reference list should be cleaned up for consistency.
  5. [Section 4.2] The RMB benchmark description says it 'encompassing over 49 distinct, fine-grained real-world scenarios'; this should be 'encompasses' and the phrase should be simplified.
  6. [Appendix C / Figure 6] The label 'ReasoneGRM' in Figure 6 is a typo for 'ReasonGRM'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: R* is a model-confidence heuristic, but the paper's performance claims are externally benchmarked and R* is not fitted to those benchmarks.

full rationale

The paper does not derive any performance claim from the definition of R*. Equation (1) defines R* as the product of average conditional token probabilities under LRM-Zero; this is a heuristic scoring rule, not a fitted parameter. The empirical claims that R*-filtered SFT outperforms random SFT (Table 3) and that ReasonGRM reaches SOTA averages (Table 1) are tested on external benchmarks (RewardBench, RM-Bench, RMB) that are not used to set any hyperparameter or threshold of R*. The correctness filter in Stage 2 uses ground-truth labels, providing an external signal independent of the model's self-likelihood. The paper contains no author self-citations, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The Limitations section concedes that R* only applies to well-defined QA pairs; that is a scope limitation, not a circular step. Therefore no equation reduces to its own input by construction, and the 'prediction' of improved reward-model performance is an empirical result rather than a tautology.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The empirical claims rest on several unreported sampling hyperparameters (G, K, N), the proxy assumption that generation likelihood equals reasoning quality, the reliability of the Skywork labels, and the assumption that the three benchmarks are uncontaminated. No new physical or architectural entities are introduced; R* is a metric, not a new entity.

free parameters (3)
  • Candidate generation count G
    Number of sampled reasoning-answer pairs per question in Stage 2 is not specified; the diversity of paths and the R* selection depend on it.
  • GRPO sample count K
    Number of responses per question in Stage 1 and in the hard-case selection rule; the all-correct/all-incorrect skip criterion depends on K, which is not reported.
  • Hard-case inference count N
    Number of answers sampled in Stage 3 to decide whether a query is hard; not reported, so the hard-case data size is unknown.
assumptions (5)
  • domain assumption Generation likelihood under LRM-Zero is a valid proxy for reasoning-path quality for preference learning.
    Used to define R* and select SFT data in Section 3.3; no independent validation beyond the RewardBench ablations.
  • domain assumption Final-answer correctness labels in the Skywork preference dataset are reliable ground truth for reward learning.
    Both Zero-RL rewards and R* filtering use these labels; dataset noise would propagate through all stages.
  • domain assumption RewardBench, RM-Bench, and RMB are not contaminated by the Skywork-Reward-80K training distribution.
    No contamination analysis is provided; overlap could inflate the reported scores.
  • domain assumption GRPO with outcome-only rewards can teach a base LRM preference discrimination without explicit reasoning traces.
    Assumed in Stage 1 and validated only indirectly through final benchmark accuracy.
  • standard math Averaging token probabilities is a valid way to compare sequences of different lengths.
    R* uses the arithmetic mean of token probabilities; this is standard practice, but it carries a length-sensitivity modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models." pith.science (2026). https://pith.science/paper/4EFXR5UN

@misc{pith2026250616712,
  author       = {Pith},
  title        = {Pith review of: ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EFXR5UN}},
  note         = {Machine review of arXiv:2506.16712}
}
abstract

Generative Reward Models (GRMs) provide greater flexibility than scalar reward models in capturing human preferences, but their effectiveness is limited by poor reasoning capabilities. This often results in incomplete or overly speculative reasoning paths, leading to hallucinations or missing key information in complex tasks. We address this challenge with ReasonGRM, a three-stage generative reward modeling framework. In the first stage, Zero-RL is used to generate concise, outcome-directed reasoning paths that reduce the likelihood of critical omissions. In the second stage, we introduce a novel evaluation metric, $R^\star$, which scores reasoning paths based on their generation likelihood. This favors paths that reach correct answers with minimal exploration, helping to reduce hallucination-prone data during training. In the final stage, the model is further refined through reinforcement learning on challenging examples to enhance its preference discrimination capabilities. Experiments on three public benchmarks show that ReasonGRM achieves competitive or state-of-the-art performance, outperforming previous best GRMs by 1.8\% on average and surpassing proprietary models such as GPT-4o by up to 5.6\%. These results demonstrate the effectiveness of reasoning-aware training and highlight the importance of high-quality rationale selection for reliable preference modeling.

Figures

Figures reproduced from arXiv: 2506.16712 by the authors.

Figure 1
Figure 1. The figure delineates three reasoning path [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the ReasonGRM training pipeline.The process begins with an LRM adapted via GRPO for preference (LRM-Zero). Subsequently, LRM-SFT is developed using SFT with ⋆ -filtered reasoning from LRM-Zero, and ReasonGRM is finalized through GRPO-based RL refinement of LRM-SFT on hard cases. to accurately identify the better answer. However, the initial dataset used for this stage only contains the question, candidat… view at source ↗
Figure 3
Figure 3. Workflow of extracting stronger R base on R⋆ -Metric. 3.3 R⋆ : Extract stronger R High-quality training data is crucial for enhancing a model’s reasoning capabilities. However, even when models can generate reasoning paths that yield the correct final answer, these paths may still vary significantly in logical clarity, conciseness, and confidence. To select the optimal paths from these preliminarily correct reasonin… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The prompt template for Experiments Prompt：I want to learn more about becoming a CPA. How long does it take to become a CPA? Chosen：Becoming a CPA typically takes 4-5 years of education, 6-12 months to pass the CPA exam, and 1- 2 years of accounting experience, with a …
Figure 5
Figure 5. Figure 5: Case study input: The user asks about the time required to become a CPA. Response A (Chosen) gives a [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Reasoning process of ReasonGRM for the case study. The model quickly and accurately identifies the [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Reasoning process of the baseline QwQ for the case study. The model shows significant vacillation. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A model-based verifier that grades sub-question-level correctness supplies fine-grained RL rewards, and training with it yields a VLM that tops several multimodal reasoning benchmarks, including the authors' new STEM-Bench.

  2. RecGPT Technical Report

    cs.IR 2025-07 conditional novelty 6.0 of 10

    RecGPT integrates three LLMs into Taobao's recommendation pipeline, turning click-log matching into intent mining, tag-based retrieval, and explanation generation, with claimed online gains of 6.33% CTR and 6.96% clic...

  3. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  4. An Explainable Machine Learning Framework for Railway Predictive Maintenance using Data Streams from the Metro Operator of Portugal

    cs.AI 2025-08 unverdicted novelty 4.0 of 10

    The submitted document is internally inconsistent: the abstract describes railway predictive maintenance while the body presents an unrelated multimodal reward-model paper, leaving the reported fault-prediction result...

Reference graph

Works this paper leans on

34 extracted references · 3 canonical work pages · cited by 4 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. 2023 b . Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [5]

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, et al. 2024. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704

    arXiv 2024

  5. [6]

    Chang, and Prithviraj Ammanabrolu

    Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, and Prithviraj Ammanabrolu. 2024. http://arxiv.org/abs/2408.11791 Critique-out-loud reward models

    arXiv 2024

  6. [7]

    Anthropic. 2024 a . https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family- opus, sonnet, haiku . Technical report, Anthropic, 548 Market Street, San Francisco

    work page 2024

  7. [8]

    AI Anthropic. 2024 b . The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1

    work page 2024

  8. [9]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    arXiv 2024

Show all 34 references
  1. [10]

    Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025 a . Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050

    2025

  2. [11]

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. 2025 b . http://arxiv.org/abs/2505.02387 Rm-r1: Reward modeling as reasoning

    2025

  3. [12]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, et al. 2025 a . http://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    2025 arXiv

  4. [13]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, et al. 2025 b . http://arxiv.org/abs/2412.19437 Deepseek-v3 technical report

    2025 arXiv

  5. [14]

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. http://arxiv.org/abs/2405.07863 Rlhf workflow: From reward modeling to online rlhf

    2024 arXiv

  6. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

    2024 arXiv

  7. [17]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  8. [18]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, et al. 2024. http://arxiv.org/abs/2412.16720 Openai o1 system card

    2024 arXiv

  9. [19]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. http://arxiv.org/abs/2403.13787 Rewardbench: Evaluating reward models for language modeling

    2024 arXiv

  10. [20]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024 a . Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451

    2024 arXiv

  11. [21]

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024 b . Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184

    2024 arXiv

  12. [22]

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025. http://arxiv.org/abs/2504.02495 Inference-time scaling for generalist reward modeling

    2025

  13. [23]

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. 2024. http://arxiv.org/abs/2410.12832 Generative reward models

    2024 arXiv

  14. [24]

    Xiaoyu Tan Minghao Yang, Chao Qu. 2024. [https://huggingface.co/infly/INF-ORM-Llama3.1-70B](https://huggingface.co/infly/INF-ORM-Llama3.1-70B) Inf-orm-llama3.1-70b

    2024

  15. [25]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  16. [26]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  17. [27]

    Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. 2024. https://huggingface.co/Skywork Skywork critic model series . https://huggingface.co/Skywork

    2024

  18. [28]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, et al. 2025. http://arxiv.org/abs/2501.12599 Kimi k1.5: Scaling reinforcement learning with llms

    2025 arXiv

  19. [30]

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024 b . Self-taught evaluators. arXiv preprint arXiv:2408.02666

    2024 arXiv

  20. [31]

    Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024 c . http://arxiv.org/abs/2406.08673 Helpsteer2: Open-source dataset for training top-performing reward models

    2024 arXiv

  21. [32]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2023 arXiv

  22. [33]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, et al. 2024. http://arxiv.org/abs/2407.10671 Qwen2 technical report

    2024 arXiv

  23. [34]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al. 2025. http://arxiv.org/abs/2412.15115 Qwen2.5 technical report

    2025 arXiv

  24. [35]

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2025. http://arxiv.org/abs/2408.15240 Generative verifiers: Reward modeling as next-token prediction

    2025 arXiv

  25. [36]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena

    2023 arXiv

  26. [37]

    Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. 2024. Rmb: Comprehensively benchmarking reward models in llm alignment. arXiv preprint arXiv:2410.09893

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.