Pith. sign in

REVIEW 4 major objections 4 minor 77 references

RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generative reward models judge responses comparatively, and this paper shows that turning those comparisons into win-count rankings—rather than scalar probabilities—makes them effective reinforcement-learning signals, lifting AlpacaEval2…

desk verdict Useful empirical result wrapped in an overreaching theoretical claim — the ranking-based reward recipe works in practice, but the margin-awareness proof does not hold as stated. read the letter →

arxiv 2608.06310 v1 pith:UZO2NRQO submitted 2026-08-06 cs.LG cs.CL

Chenglong Wang , Ziming Zhu , Yifu Huo , Bei Li , Qiaozhi He , Yan Ding , Xiaoyang Hao , Yuxin Gao
show 4 more authors
Tianhua Zhou Xiaojia Chang Tongran Liu Jingbo Zhu
This is my paper
classification cs.LGcs.CL
keywords generativerewardmodelsranking-basedconstructionLLMreinforcementlearningpairwisepreferencejudgmentsself-competitiverankinganchor-guidedGRPOmarginawareness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative reward models—LLMs trained to judge responses by generating textual preferences—rank responses better than discriminative reward models, but this paper argues that their edge disappears in reinforcement learning because RL forces them to emit scalar scores. The proposed Ranking-based Reward Construction (RRC) instead derives rewards from relative orderings: a response's reward is the number of other sampled responses it beats (self-competitive ranking) or the number of fixed reference responses it outranks (anchor-guided ranking), with majority voting to stabilize noisy pairwise judgments. Because group-relative RL algorithms such as GRPO normalize rewards within each batch, these win-count scores preserve preference order and approximate quality margins, which the paper shows in a policy-gradient argument. The consistent gains—35.8% to 41.3% on AlpacaEval2, 8.0% to 11.2% on ArenaHardV2, and 52.9% to 57.3% on MMLU-Redux—suggest that the comparative strength of generative reward models can be unlocked without abandoning the scalar interface RL needs.

What carries the argument

Ranking-based reward construction is the central mechanism: rewards are win counts over pairwise preference judgments rather than calibrated probabilities. In self-competitive ranking (SCR), the reward is $r(x,o_i)=\alpha\sum_{j\ne i}\mathbf{1}[o_i\succ o_j]$ over responses sampled from the current policy; in anchor-guided ranking (AGR), it is $r(x,o_i)=\alpha\sum_{k=1}^n \mathbf{1}[o_i\succ a_k]$ against a small set of fixed references. Two components carry the implementation: majority voting over repeated stochastic pairwise judgments, and conflict-aware ranking adjustment, a greedy Kemeny-rule aggregation that removes cyclic inconsistencies such as $o_i\succ o_j\succ o_k\succ o_i$ while retaining high-weight preferences. The load-bearing identity is the affine-link assumption $r(x,o)=bq(x,o)+d$: under it, the group-normalized advantage $(r_i-\mu_r)/\sigma_r$ equals the oracle-consistent advantage, converting raw win counts into margin-aware policy-gradient updates. Reference-policy anchors avoid a moving-target objective because $\nabla_\theta r(x,o)=0$ when anchors do not depend on the evolving policy.

What would settle it

On a fixed prompt set with human-rated response quality, compute win-count rewards from a generative reward model and test whether reward gaps are proportional to human-rated quality gaps; if responses with equal quality receive systematically different win counts, or if the cyclic-triple rate is high enough to break the ordering, the margin-awareness property that RRC's argument depends on would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that generative reward models (GRMs) fail in RL because of a paradigm mismatch, not a capability deficit. GRMs are trained to express comparative preference judgments, yet RL pipelines convert those judgments into scalar rewards through the probability of a designated preference token; under chain-of-thought reasoning those probabilities collapse toward 0 and 1, discarding the relative margins that make a learning signal informative. RRC replaces probability scoring with win counts: in self-competitive ranking, $r(x,o_i)=\alpha\sum_{j\ne i}\mathbf{1}[o_i\succ o_j]$ over responses sampled from the current policy, and in anchor-guided ranking, $r(x,o_i)=\alpha\sum_{k=1}^n \mathbf{1}[o_i\succ a_k]$ against a fixed reference-policy anchor set, cutting query cost from $O(m\log m)$ to $O(mn)$. The paper argues two properties for such rewards—order preservation, and margin awareness under the affine link $r(x,o)=bq(x,o)+d$ between observed rewards and latent oracle quality $q$—and shows that group-normalized advantages then coincide with oracle-consistent policy gradients. Empirically, RRC outperforms probability-based construction, discriminative reward models, DPO, and SimPO on six benchmarks, with scaling gains from majority voting and larger anchor sets.

Load-bearing premise

The load-bearing premise is that a response's number of pairwise wins keeps tracking how much better the response actually is, in an approximately linear way, even when individual judgments are noisy.

Editorial extensions

If this is right

  • In GRPO-style RL, ranking-derived rewards give generative reward models a consistent advantage over probability-based rewards across all six tested benchmarks, under both 3B and 8B reward backbones.
  • The approach adds a scaling dimension to RL: increasing the number of majority votes or the number of anchors raises downstream benchmark performance, with diminishing returns at high budgets.
  • AGR reaches near-baseline training cost (10.6h vs 10.5h) while improving AlpacaEval2 from 35.8% to 39.4%, rising to 41.3% with voting@8.
  • Removing conflict-aware ranking adjustment hurts performance, so handling cyclic pairwise preferences is a necessary part of the method, not a detail.
  • Using anchors from a fixed reference policy, rather than from the current policy, stabilizes training by preventing the reward baseline from drifting as the policy improves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same ranking-versus-scoring mismatch should apply to any comparative judge—human raters, verifiers, multimodal critics—so converting their judgments to ordinal win counts may be more robust than calibrating their probabilities.
  • A testable extension the paper does not run: RRC's advantage over probability-based construction should shrink as the rate of cyclic preference triples rises; measuring that correlation would isolate whether margin awareness is the operative mechanism.
  • The affine-link proof leaves implicit a refinement: weighting wins by the quality or informativeness of the anchor beaten, rather than counting all wins equally, could sharpen margin estimates when anchors vary in quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Ranking-based Reward Construction (RRC) for using generative reward models in RL. RRC replaces scalar probability-based rewards with rewards derived from pairwise rankings: self-competitive ranking (SCR) uses the number of wins in a tournament among m sampled responses; anchor-guided ranking (AGR) uses the number of anchors a response beats among n reference responses. Majority voting and a greedy conflict-aware ranking adjustment are added to handle stochastic and cyclic preferences. Experiments using GRPO with LLaMA-3.1-8B and Qwen2.5-7B policy models, 3B/8B reward models, and six chat/reasoning benchmarks report consistent gains over DRMs, PRC, PRC without reasoning, DPO, and SimPO; scaling curves over voting budget and anchor count, and a compute-matched comparison, are used to support the mechanism. The paper also claims theoretical properties: order preservation, margin awareness, Kemeny-consistent aggregation, and a higher attainable performance ceiling.

Significance. RRC is a simple and plausible recipe for converting a generative reward model's comparative strengths into RL learning signals, and the experimental suite is commendable in breadth: two policy backbones, two reward-model scales, six benchmarks, a compute-matched baseline, and ablations over alpha, anchors, voting, and the conflict-adjustment heuristic. The main empirical pattern---ranking-based rewards outperform probability-based rewards---is consistent across settings and is worth investigating further. However, the paper currently overclaims its theoretical support: the margin-awareness proof is invalid because it assumes the affine reward-quality relation that rank-based rewards do not satisfy (and cannot satisfy pointwise, as rewards are integer-valued and group-dependent), the Kemeny property does not apply to the implemented greedy heuristic, and the reported O(m log m) query complexity is actually O(m^2). The empirical evidence also lacks error bars and uses nonstandard judges, so the central empirical claim, while plausible, is not yet fully substantiated.

major comments (4)
  1. [Appendix A, first proof (Property 1); Section 4.2, Eqs. (5), (7), (13), (22), (24)] The margin-awareness property is not established by the provided proof. Eqs. (5) and (7) define rewards as alpha times an integer win count or anchor-win count, so every reward gap is an integer multiple of alpha; adjacent ranks produce the same gap regardless of the oracle margin Delta q_ij, and two responses with Delta q_ij > 0 can receive identical win counts under an unfavorable sample. The proof's Eq. (13) assumes r = bq + d, which is exactly the property the ranking-based construction does not possess, and Eqs. (22) and (24) only re-express the gap as a sum of +/-alpha indicators before asserting proportionality. Consequently, Property 2 in the main text and the "higher attainable performance ceiling" claim in Section 1 are unsupported. The authors should either prove margin awareness for a well-specified stochastic model (e.g., in expectation over sampled groups and anchors) or remove/soften these theoretical claims and treat the rank-based signal as an empirically motivated heuristic.
  2. [Section 4.2, Property 3; Appendix A, second proof; Algorithm 1] The Kemeny-rule recovery guarantee does not apply to the implemented procedure. The proof shows that an exact Kemeny optimum is globally consistent, which is true by definition of the Kemeny objective, but Algorithm 1 uses the greedy Davenport-Kalagnanam heuristic (lines 18-28) to resolve cycles, and no guarantee is given that this heuristic finds a Kemeny-optimal order. The paper itself acknowledges cyclic preferences in Eq. (6). In addition, Algorithm 1 assigns rewards from the adjusted total order pi (line 34), not from the raw pairwise win counts used in Eq. (5); the relationship between the two should be stated explicitly. Please either prove approximation guarantees for the heuristic, or reframe Property 3 as a heuristic design choice supported by the ablation in Figure 4(b).
  3. [Section 4.2 and Appendix C.1, Eq. (37); Table 3] The query complexity of SCR is stated incorrectly. The number of unordered pairwise comparisons is m(m-1)/2, which is O(m^2), not O(m log m); the same error appears in Section 1's motivation and in Table 3. The claim that AGR reduces the cost "from O(m log m) to O(m n)" is therefore misstated and should be corrected to O(m^2) to O(m n). This is a mathematical error in a load-bearing efficiency argument, even though correcting it would only strengthen AGR's scalability case.
  4. [Section 5, Tables 1-2; Appendix B.1] The central empirical claim rests on single-run results without error bars or significance testing. Table 1 and Table 2 report one number per condition, and Appendix B.1 describes checkpoint selection using an AlpacaEval2-style validation score computed with GPT-4o; this selection procedure and the use of GPT-4o/GPT-5 as final judges (instead of the standard AlpacaEval2/ArenaHardV2 judge) make the headline numbers not directly comparable to published benchmark scores and leave the magnitude of the reported gains unquantified. Please report at least three seeds with variance, and either use the standard judges or justify and clearly flag the deviation.
minor comments (4)
  1. [Appendix B.3, first sentence] The sentence "we observe trends highly consistent with those on Qwen2.5-7B-Instruct" should presumably refer to the LLaMA-3.1-8B-Instruct results from Table 1, not to the same Qwen results being described.
  2. [Main text and Appendix A property numbering] The numbering of properties differs between the main text and Appendix A: main-text Properties 1-3 correspond to order preservation, margin awareness, and Kemeny consistency, while Appendix A labels margin awareness as Property 1, Kemeny as Property 2, and anchor-reference stability as Property 3. Align the numbering across both parts.
  3. [Table 1, voting@8 rows] Several entries in the +voting@8 rows are run together (e.g., "37.89.8 56.756.747.2"), making the table hard to read; the table needs re-typesetting with proper spacing.
  4. [Figure 3] The x-axis of Figure 3(a) is labeled "Number of Votes (log scale)" with values 0 to 9; please clarify whether 0 denotes a single vote and whether the anchor-count axis in Figure 3(b) is also log-scaled, since this affects the interpretation of the scaling curves.

Circularity Check

1 steps flagged · score 4.0 of 10

Empirical results are external and non-circular, but the Appendix A margin-awareness proof assumes the affine reward–quality relation it claims to derive, making the theoretical 'higher ceiling' claim partly self-definitional.

  1. self definitional [Appendix A, Property 1 proof (Eqs. 13–24); also Sec. 4.2 after Eq. 5]
    "Furthermore, assume that for any fixed x, reward scores from reward models are an affine transformation of oracle qualities: r(x, o) = bq(x, o) + d, b > 0 (13) ... This is exactly the desired margin-awareness property: r(x, oi) − r(x, oj) ∝ Δqij (20) ... Intuitively, if oi is substantially better than oj, it will defeat more competitors and lose to fewer, yielding a larger reward gap."

    The desired Property 2 is that reward gaps are proportional to oracle margins. Equation 13 assumes exactly that proportionality (r = bq + d implies r_i − r_j = b(q_i − q_j)); the proof then derives Eq. 20 from Eq. 13, so the conclusion is already contained in the assumption. For the RRC construction, Eqs. 21–24 only express reward gaps as integer win-count differences; they do not show these equal bΔq_ij. Because SCR/AGR rewards are integer multiples of α (Eqs. 5 and 7), continuous oracle margins cannot in general be affine in them. Thus the 'higher attainable performance ceiling' claim rests on assuming the target margin-awareness property rather than deriving it for ranking-based rewards.

full rationale

The benchmark results (AlpacaEval2, ArenaHardV2, WildBench, MMLU-Redux, MATH-500) are evaluated against external corpora and prior baselines, so the central empirical claim is not circular; likewise, the many self-citations (GRAM and related work) are used as context/baselines and are not load-bearing for the RRC design. The genuine circularity is confined to the theoretical framing: margin awareness is 'proved' under an affine reward–quality assumption (Eq. 13) that is equivalent to the property being proved, and the win-count gap analysis (Eqs. 21–24) asserts rather than derives proportionality to a latent oracle margin. A separate weakness, noted in the paper's own Eq. 6, is that the implemented Algorithm 1 uses a greedy Kemeny heuristic rather than exact Kemeny aggregation, so Property 2's exact recovery guarantee does not strictly describe the deployed method; this is a correctness gap, not an additional circularity. Overall, empirical content is self-contained; the theoretical 'higher ceiling' claim is only partially supported and is partly self-definitional.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claim rests on several hand-chosen hyperparameters (alpha, m, n, V) and on the unproven assumption that win-count rewards approximate an affine transformation of oracle quality. No new physical or conceptual entities are introduced.

free parameters (5)
  • scaling factor alpha = 0.1
    Set by hand for RRC. Appendix B.2 shows performance is stable for alpha up to 0.5 but degrades above 1.0, so the choice interacts with GRPO advantage normalization.
  • group sampling size m = 8
    Chosen by hand. Figure 3 explores sampling 8 versus 16 responses per prompt.
  • anchor count n = 8
    Chosen by hand. Figure 3 shows 8 anchors are sufficient and 256 anchors saturate or degrade performance.
  • voting budget V = 8 for voting@8
    Chosen by hand. Figure 3 shows monotonic gains from more votes with diminishing returns.
  • format reward weight = 1
    A binary format reward is added to the ranking reward in the reasoning-augmented setting (Eq. 36); the weight is fixed at 1.
assumptions (5)
  • domain assumption There exists a latent oracle quality function q(x,o), and pairwise preference judgments from the generative reward model are noisy observations of q.
    Assumed in Appendix A, Property 1 proof (Eqs. 8-9) and used to justify margin awareness; not independently verified.
  • ad hoc to paper Win counts from pairwise comparisons approximate an affine function of oracle quality, so integer-valued rewards satisfy the margin-awareness property.
    Appendix A after Eqs. 21-24 asserts reward gaps equal net wins and claims this approximates oracle margins; no formal argument is supplied.
  • ad hoc to paper The greedy Davenport-Kalagnanam heuristic used for conflict-aware ranking adjustment recovers a Kemeny-optimal total order in practice.
    Section 4.2 and Algorithm 1 use the greedy heuristic, while the proof in Appendix A applies only to exact Kemeny aggregation.
  • domain assumption Anchor responses generated from a fixed reference policy remain informative and do not drift with policy updates.
    Property 3 proof assumes theta-independent anchors; empirically this holds by construction, but anchor informativeness over training is assumed.
  • standard math Policy gradient theorem and affine invariance of group-normalized advantages are valid tools for analyzing reward construction.
    Used in Appendix A to relate reward calibration to policy updates; these are standard results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction." pith.science (2026). https://pith.science/paper/UZO2NRQO

@misc{pith2026260806310,
  author       = {Pith},
  title        = {Pith review of: RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZO2NRQO}},
  note         = {Machine review of arXiv:2608.06310}
}
read the original abstract

Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.

Figures

Figures reproduced from arXiv: 2608.06310 by the authors.

Figure 1
Figure 1. Comparison between discriminative and generati d dldkid RL iR Figure 1: Comparison between discriminative and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) Probability-Based Reward Construction. We use the predicted probability of the preference token as the scalar reward signal (denoted as Baseline-1). We also remove explicit reasoning to encourage output diversity and improve margin awareness (denoted as Baseline-2). (b) Ranking-Based Reward Construction. We obtain reward scores from relative rankings obtained either among sampled responses or between sampled res… view at source ↗
Figure 3
Figure 3. Scaling behavior of RRC. from the generative reward model yields better-aligned re￾ward signals and thus more effective policy optimization. Similarly, increasing the number of anchors in AGR consis￾tently improves performance, suggesting that denser anchor comparisons provide finer-grained relative quality estimates, which translate into more informative advantage signals in RL. Notably, the performance gains exhib… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation Study Appendix B: Additional Experimental Details and Results In this section, we provide additional experimental details and present the experimental results of our RRC approach. Experimental Details Training Setups. For the experiments reported in [PITH_FUL…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 66 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Think-rm: Enabling long-horizon reasoning in generative reward models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  2. [2]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Genprm: Scaling test-time compute of process reward models via generative reasoning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    work page

  3. [3]

    arXiv preprint arXiv:2503.05236 , year=

    Unified reward model for multimodal understanding and generation , author=. arXiv preprint arXiv:2503.05236 , year=

    arXiv

  4. [4]

    Advances in neural information processing systems , volume=

    Unified multimodal chain-of-thought reward model through reinforcement fine-tuning , author=. Advances in neural information processing systems , volume=

    work page

  5. [5]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    DUAL RM: Beyond Rule-based Preference Reward Modeling via Meta-Reward , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  6. [6]

    arXiv preprint arXiv:2505.10527 , year=

    Worldpm: Scaling human preference modeling , author=. arXiv preprint arXiv:2505.10527 , year=

    arXiv

  7. [7]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Outcome accuracy is not enough: Aligning the reasoning process of reward models , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  8. [8]

    Reward is enough: Llms are in-context reinforcement learners , volume =

    Song, Kefan and Moeini, Amir and Wang, Peng and Gong, Lei and Chandra, Rohan and Zhang, Shangtong and Qi, Yanjun , journal =. Reward is enough: Llms are in-context reinforcement learners , volume =

    work page

Show all 77 references
  1. [9]

    Chenglong Wang and Hang Zhou and Yimin Hu and Yifu Huo and Bei Li and Tongran Liu and Tong Xiao and Jingbo Zhu , booktitle =

  2. [10]

    Scaling laws for neural language models , volume =

    Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario , journal =. Scaling laws for neural language models , volume =

  3. [11]

    Large language models are not fair evaluators , year =

    Wang, Peiyi and Li, Lei and Chen, Liang and Cai, Zefan and Zhu, Dawei and Lin, Binghuai and Cao, Yunbo and Kong, Lingpeng and Liu, Qi and Liu, Tianyu and others , booktitle =. Large language models are not fair evaluators , year =

  4. [12]

    Theoretical and empirical evaluation of data reduction for exact Kemeny rank aggregation , volume =

    Betzler, Nadja and Bredereck, Robert and Niedermeier, Rolf , journal =. Theoretical and empirical evaluation of data reduction for exact Kemeny rank aggregation , volume =

  5. [13]

    Advances in Neural Information Processing Systems , volume=

    MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization , author=. Advances in Neural Information Processing Systems , volume=

  6. [14]

    Improved parameterized algorithms for the Kemeny aggregation problem , year =

    Simjour, Narges , booktitle =. Improved parameterized algorithms for the Kemeny aggregation problem , year =

  7. [15]

    Are we done with mmlu? , year =

    Gema, Aryo Pradipta and Leang, Joshua Ong Jun and Hong, Giwon and Devoto, Alessio and Mancino, Alberto Carlo Maria and Saxena, Rohit and He, Xuanli and Zhao, Yu and Du, Xiaotang and Madani, Mohammad Reza Ghasemi and others , booktitle =. Are we done with mmlu? , year =

  8. [16]

    Length-controlled alpacaeval: A simple debiasing of automatic evaluators , year =

    Dubois, Yann and Liang, Percy and Hashimoto, Tatsunori , booktitle =. Length-controlled alpacaeval: A simple debiasing of automatic evaluators , year =

  9. [17]

    Let's Verify Step by Step , year =

    Hunter Lightman and Vineet Kosaraju and Yuri Burda and Harrison Edwards and Bowen Baker and Teddy Lee and Jan Leike and John Schulman and Ilya Sutskever and Karl Cobbe , booktitle =. Let's Verify Step by Step , year =

  10. [18]

    Gpqa: A graduate-level google-proof q&a benchmark , year =

    Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R , booktitle =. Gpqa: A graduate-level google-proof q&a benchmark , year =

  11. [19]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline , volume =

    Li, Tianle and Chiang, Wei-Lin and Frick, Evan and Dunlap, Lisa and Wu, Tianhao and Zhu, Banghua and Gonzalez, Joseph E and Stoica, Ion , journal =. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline , volume =

  12. [20]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild , volume =

    Lin, Bill Yuchen and Deng, Yuntian and Chandu, Khyathi and Brahman, Faeze and Ravichander, Abhilasha and Pyatkin, Valentina and Dziri, Nouha and Bras, Ronan Le and Choi, Yejin , journal =. Wildbench: Benchmarking llms with challenging tasks from real users in the wild , volume =

  13. [21]

    Hashimoto , howpublished =

    Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , howpublished =. GitHub repository , title =

  14. [22]

    SimPO: Simple Preference Optimization with a Reference-Free Reward , year =

    Yu Meng and Mengzhou Xia and Danqi Chen , booktitle =. SimPO: Simple Preference Optimization with a Reference-Free Reward , year =

  15. [23]

    HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages , volume =

    Wang, Zhilin and Zeng, Jiaqi and Delalleau, Olivier and Shin, Hoo-Chang and Soares, Felipe and Bukharin, Alexander and Evans, Ellie and Dong, Yi and Kuchaiev, Oleksii , journal =. HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages , volume =

  16. [24]

    Qwen2 technical report , volume =

    Team, Qwen and others , journal =. Qwen2 technical report , volume =

  17. [25]

    The llama 3 herd of models , volume =

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal =. The llama 3 herd of models , volume =

  18. [26]

    Rrhf: Rank responses to align language models with human feedback without tears , year =

    Yuan, Zheng and Yuan, Hongyi and Tan, Chuanqi and Wang, Wei and Huang, Songfang and Huang, Fei , journal =. Rrhf: Rank responses to align language models with human feedback without tears , year =

  19. [27]

    A computational study of the Kemeny rule for preference aggregation , year =

    Davenport, Andrew and Kalagnanam, Jayant , booktitle =. A computational study of the Kemeny rule for preference aggregation , year =

  20. [28]

    Judgebench: A benchmark for evaluating llm-based judges , year =

    Tan, Sijun and Zhuang, Siyuan and Montgomery, Kyle and Tang, William Y and Cuadron, Alejandro and Wang, Chenguang and Popa, Raluca Ada and Stoica, Ion , journal =. Judgebench: A benchmark for evaluating llm-based judges , year =

  21. [29]

    Rm-bench: Benchmarking reward models of language models with subtlety and style , year =

    Liu, Yantao and Yao, Zijun and Min, Rui and Cao, Yixin and Hou, Lei and Li, Juanzi , journal =. Rm-bench: Benchmarking reward models of language models with subtlety and style , year =

  22. [30]

    Proximal policy optimization algorithms , year =

    Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , journal =. Proximal policy optimization algorithms , year =

  23. [31]

    Rank analysis of incomplete block designs: I

    Bradley, Ralph Allan and Terry, Milton E , journal =. Rank analysis of incomplete block designs: I. The method of paired comparisons , year =

  24. [32]

    Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification , year =

    Mroueh, Youssef , journal =. Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification , year =

  25. [33]

    Language models that think, chat better , year =

    Bhaskar, Adithya and Ye, Xi and Chen, Danqi , journal =. Language models that think, chat better , year =

  26. [34]

    Dissecting Long Reasoning Models: An Empirical Study , year =

    Mu, Yongyu and Zeng, Jiali and Li, Bei and Guan, Xinyan and Meng, Fandong and Zhou, Jie and Xiao, Tong and Zhu, Jingbo , journal =. Dissecting Long Reasoning Models: An Empirical Study , year =

  27. [35]

    Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms , year =

    Wen, Xumeng and Liu, Zihan and Zheng, Shun and Ye, Shengyu and Wu, Zhirong and Wang, Yang and Xu, Zhijian and Liang, Xiao and Li, Junjie and Miao, Ziming and others , journal =. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base ll...

  28. [36]

    Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong , title =

  29. [37]

    Pre-Trained Policy Discriminators are General Reward Models , year =

    Dou, Shihan and Liu, Shichun and Yang, Yuming and Zou, Yicheng and Zhou, Yunhua and Xing, Shuhao and Huang, Chenhao and Ge, Qiming and Song, Demin and Lv, Haijun and others , journal =. Pre-Trained Policy Discriminators are General Reward Models , year =

  30. [38]

    Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation , year =

    Min, Do June and Perez-Rosas, Veronica and Resnicow, Ken and Mihalcea, Rada , booktitle =. Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation , year =

  31. [39]

    Contrastive Preference Optimization: Pushing the Boundaries of

    Haoran Xu and Amr Sharaf and Yunmo Chen and Weiting Tan and Lingfeng Shen and Benjamin Van Durme and Kenton Murray and Young Jin Kim , booktitle =. Contrastive Preference Optimization: Pushing the Boundaries of

  32. [40]

    From system 1 to system 2: A survey of reasoning large language models , year =

    Li, Zhong-Zhi and Zhang, Duzhen and Zhang, Ming-Liang and Zhang, Jiaxin and Liu, Zengyan and Yao, Yuxuan and Xu, Haotian and Zheng, Junhao and Wang, Pei-Jie and Chen, Xiuyi and others , journal =. From system 1 to system 2: A survey of reasoning large language models , year =

  33. [41]

    Wu, Yangzhen and Sun, Zhiqing and Li, Shanda and Welleck, Sean and Yang, Yiming , title =

  34. [42]

    Prior constraints-based reward model training for aligning large language models , year =

    Zhou, Hang and Wang, Chenglong and Hu, Yimin and Xiao, Tong and Zhang, Chunliang and Zhu, Jingbo , booktitle =. Prior constraints-based reward model training for aligning large language models , year =

  35. [43]

    Improving In-Context Learning via Sequentially Selection and Preference Alignment for Few-Shot Aspect-Based Sentiment Analysis , year =

    Qianlong Wang and Keyang Ding and Xuan Luo and Ruifeng Xu , booktitle =. Improving In-Context Learning via Sequentially Selection and Preference Alignment for Few-Shot Aspect-Based Sentiment Analysis , year =

  36. [44]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models , year =

    Chu, Yunfei and Xu, Jin and Zhou, Xiaohuan and Yang, Qian and Zhang, Shiliang and Yan, Zhijie and Zhou, Chang and Zhou, Jingren , journal =. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models , year =

  37. [45]

    Manning and Stefano Ermon and Chelsea Finn , booktitle =

    Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn , booktitle =. Direct Preference Optimization: Your Language Model is Secretly a Reward Model , year =

  38. [46]

    Discriminative Reranking for Neural Machine Translation , year =

    Lee, Ann and Auli, Michael and Ranzato, Marc. Discriminative Reranking for Neural Machine Translation , year =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

  39. [47]

    Dapo: An open-source llm reinforcement learning system at scale , year =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Dai, Weinan and Fan, Tiantian and Liu, Gaohong and Liu, Lingjun and others , journal =. Dapo: An open-source llm reinforcement learning system at scale , year =

  40. [48]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , year =

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, YK and Wu, Yang and others , journal =. Deepseekmath: Pushing the limits of mathematical reasoning in open language models , year =

  41. [49]

    Generative reward modeling via synthetic criteria preference learning , year =

    Liang, Xiaobo and Zhang, Haoke and Li, Juntao and Chen, Kehai and Zhu, Qiaoming and Zhang, Min , booktitle =. Generative reward modeling via synthetic criteria preference learning , year =

  42. [50]

    Unified multimodal chain-of-thought reward model through reinforcement fine-tuning , year =

    Wang, Yibin and Li, Zhimin and Zang, Yuhang and Wang, Chunyu and Lu, Qinglin and Jin, Cheng and Wang, Jiaqi , journal =. Unified multimodal chain-of-thought reward model through reinforcement fine-tuning , year =

  43. [51]

    Rm-r1: Reward modeling as reasoning , year =

    Chen, Xiusi and Li, Gaotang and Wang, Ziqi and Jin, Bowen and Qian, Cheng and Wang, Yu and Wang, Hongru and Zhang, Yu and Zhang, Denghui and Zhang, Tong and others , journal =. Rm-r1: Reward modeling as reasoning , year =

  44. [52]

    Reward reasoning model , year =

    Guo, Jiaxin and Chi, Zewen and Dong, Li and Dong, Qingxiu and Wu, Xun and Huang, Shaohan and Wei, Furu , journal =. Reward reasoning model , year =

  45. [53]

    GRAM-R ^2 : Self-Training Generative Foundation Reward Models for Reward Reasoning , year =

    Wang, Chenglong and Mu, Yongyu and Zhou, Hang and Huo, Yifu and Zhu, Ziming and Zeng, Jiali and Yang, Murun and Li, Bei and Hao, Xiaoyang and Zhang, Chunliang and others , journal =. GRAM-R ^2 : Self-Training Generative Foundation Reward Models for Reward Reasoning , year =

  46. [54]

    GRAM: A Generative Foundation Reward Model for Reward Generalization , year =

    Wang, Chenglong and Gan, Yang and Huo, Yifu and Mu, Yongyu and He, Qiaozhi and Yang, Murun and Li, Bei and Xiao, Tong and Zhang, Chunliang and Liu, Tongran and others , journal =. GRAM: A Generative Foundation Reward Model for Reward Generalization , year =

  47. [55]

    Inference-time scaling for generalist reward modeling , year =

    Liu, Zijun and Wang, Peiyi and Xu, Runxin and Ma, Shirong and Ruan, Chong and Li, Peng and Liu, Yang and Wu, Yu , journal =. Inference-time scaling for generalist reward modeling , year =

  48. [56]

    Reward Model Ensembles Help Mitigate Overoptimization , year =

    Thomas Coste and Usman Anwar and Robert Kirk and David Krueger , booktitle =. Reward Model Ensembles Help Mitigate Overoptimization , year =

  49. [57]

    Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy , year =

    Liu, Chris Yuhao and Zeng, Liang and Xiao, Yuzhen and He, Jujie and Liu, Jiacai and Wang, Chaojie and Yan, Rui and Shen, Wei and Zhang, Fuxiang and Xu, Jiacheng and others , journal =. Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy , year =

  50. [58]

    Rovrm: A robust visual reward model optimized via auxiliary textual preference data , year =

    Wang, Chenglong and Gan, Yang and Huo, Yifu and Mu, Yongyu and Yang, Murun and He, Qiaozhi and Xiao, Tong and Zhang, Chunliang and Liu, Tongran and Zhu, Jingbo , booktitle =. Rovrm: A robust visual reward model optimized via auxiliary textual preference data , year =

  51. [59]

    Specialist or Generalist? Instruction Tuning for Specific

    Shi, Chufan and Su, Yixuan and Yang, Cheng and Yang, Yujiu and Cai, Deng , booktitle =. Specialist or Generalist? Instruction Tuning for Specific

  52. [60]

    Unveiling the Generalization Power of Fine-Tuned Large Language Models , year =

    Yang, Haoran and Zhang, Yumeng and Xu, Jiaqi and Lu, Hongyuan and Heng, Pheng-Ann and Lam, Wai , booktitle =. Unveiling the Generalization Power of Fine-Tuned Large Language Models , year =

  53. [61]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , year =

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Zhang, Ruoyu and Xu, Runxin and Zhu, Qihao and Ma, Shirong and Wang, Peiyi and Bi, Xiao and others , journal =. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , year =

  54. [62]

    Chi and Quoc V

    Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed H. Chi and Quoc V. Le and Denny Zhou , booktitle =. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , year =

  55. [63]

    Scaling instruction-finetuned language models , year =

    Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Yunxuan and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and others , journal =. Scaling instruction-finetuned language models , year =

  56. [64]

    ArXiv preprint , title =

    Mahan, Dakota and Van Phung, Duy and Rafailov, Rafael and Blagden, Chase and Lile, Nathan and Castricato, Louis and Fr. ArXiv preprint , title =

  57. [65]

    Generative verifiers: Reward modeling as next-token prediction , year =

    Zhang, Lunjun and Hosseini, Arian and Bansal, Hritik and Kazemi, Mehran and Kumar, Aviral and Agarwal, Rishabh , journal =. Generative verifiers: Reward modeling as next-token prediction , year =

  58. [66]

    Foundations of large language models , year =

    Xiao, Tong and Zhu, Jingbo , journal =. Foundations of large language models , year =

  59. [67]

    Step-level verifier-guided hybrid test-time scaling for large language models , year =

    Chang, Kaiyan and Shi, Yonghao and Wang, Chenglong and Zhou, Hang and Hu, Chi and Liu, Xiaoqian and Luo, Yingfeng and Ge, Yuan and Xiao, Tong and Zhu, Jingbo , booktitle =. Step-level verifier-guided hybrid test-time scaling for large language models , year =

  60. [68]

    s1: Simple test-time scaling , year =

    Muennighoff, Niklas and Yang, Zitong and Shi, Weijia and Li, Xiang Lisa and Fei-Fei, Li and Hajishirzi, Hannaneh and Zettlemoyer, Luke and Liang, Percy and Cand. s1: Simple test-time scaling , year =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language ...

    2025

  61. [69]

    Ziegler and Ryan Lowe and Chelsea Voss and Alec Radford and Dario Amodei and Paul F

    Nisan Stiennon and Long Ouyang and Jeffrey Wu and Daniel M. Ziegler and Ryan Lowe and Chelsea Voss and Alec Radford and Dario Amodei and Paul F. Christiano , booktitle =. Learning to summarize with human feedback , year =

  62. [70]

    Christiano and Jan Leike and Tom B

    Paul F. Christiano and Jan Leike and Tom B. Brown and Miljan Martic and Shane Legg and Dario Amodei , booktitle =. Deep Reinforcement Learning from Human Preferences , year =

  63. [71]

    Pku-saferlhf: Towards multi-level safety alignment for llms with human preference , year =

    Ji, Jiaming and Hong, Donghai and Zhang, Borong and Chen, Boyuan and Dai, Josef and Zheng, Boren and Qiu, Tianyi Alex and Zhou, Jiayi and Wang, Kaile and Li, Boxun and others , booktitle =. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference , year =

  64. [72]

    Training a helpful and harmless assistant with reinforcement learning from human feedback , year =

    Bai, Yuntao and Jones, Andy and Ndousse, Kamal and Askell, Amanda and Chen, Anna and DasSarma, Nova and Drain, Dawn and Fort, Stanislav and Ganguli, Deep and Henighan, Tom and others , journal =. Training a helpful and harmless assistant with reinforcement learning from human ...

  65. [73]

    Hybrid alignment training for large language models , year =

    Wang, Chenglong and Zhou, Hang and Chang, Kaiyan and Li, Bei and Mu, Yongyu and Xiao, Tong and Liu, Tongran and Zhu, Jingbo , journal =. Hybrid alignment training for large language models , year =

  66. [74]

    Long Ouyang and Jeffrey Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and John Schulman and Jacob Hilton and Fraser Kelton and Luke Miller and Maddie Simens and Amanda Askell ...

  67. [75]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  68. [76]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  69. [77]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.