Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

URPO: A Unified Reward & Policy Optimization Framework for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read URPO claims that a single model trained in one GRPO loop on reasoning, preference, and open-ended instruction data can replace the separate reward model in RLHF and outperform it on instruction-following, reasoning, and reward modeling.

desk verdict URPO is a real synthesis of self-rewarding and GRPO, but its headline comparisons do not hold the policy's training signal fixed, so the central claim is not yet established. read the letter →

arxiv 2507.17515 v1 pith:JMTM4ED7 submitted 2025-07-23 cs.CV cs.CL

Songshuo Lu , Hua Wang , Zhi Chen , Yaohua Tang This is my paper
classification cs.CVcs.CL
keywords LLMalignmentGRPOself-rewardingrewardmodelingreinforcementlearningfromhumanfeedbackinstructionfollowingreasoningKendalltau
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the standard two-model alignment pipeline, in which a policy model is guided by a frozen, separately trained reward model, can be replaced by a single model trained in one pass of Group-Relative Policy Optimization (GRPO, an RL update that scores each response relative to a group of samples). URPO feeds that loop three kinds of data: rule-verifiable reasoning problems, human and AI preference rankings, and open-ended instructions that the model scores itself. If the claim holds, alignment becomes a single training job instead of a multi-stage pipeline, and the reward signal can improve as the model improves rather than staying frozen. On Qwen2.5-7B, the paper reports that URPO lifts AlpacaEval from 42.24 to 44.84 and the composite reasoning average from 32.66 to 35.66 against a strong baseline using a separate generative reward model, while the model's emergent evaluator scores 85.15 on RewardBench, above the dedicated reward model's 83.55.

What carries the argument

The central mechanism is a single GRPO loop that assigns a group-relative advantage to every response. Three reward sources feed it: a rule-based verifier for reasoning problems, Kendall's tau between the model's predicted ranking and ground-truth preferences for preference triples, and the model's own generated scores for open-ended responses, which it produces by evaluating its G rollouts in one prompt. The loss drops length normalization, uses an asymmetric clipped surrogate ratio, and sets the KL penalty to zero, while a two-stage curriculum first trains on objective reasoning and preference rewards for 100 steps before adding open-ended self-scored data.

What would settle it

On a held-out set of open-ended prompts, compute the rank correlation between the model's self-assigned scores and an independent judge's scores at each training checkpoint; if the model's AlpacaEval score rises while this correlation does not improve or falls, the self-reward signal is not the cause of the gain. Alternatively, rerun URPO with the self-scores replaced by shuffled scores within each group; if instruction-following and reasoning results stay the same, the self-evaluation mechanism is not carrying the claimed weight.

Watch

Extended reading notes

Core claim

The central claim is that a single large language model can be both the player and the referee in the same reinforcement-learning phase. URPO recasts preference triples as an N-way ranking prompt and rewards the model with Kendall's tau between its predicted ranking and the ground-truth ordering; it scores its own G rollouts for open-ended instructions and feeds those self-scores directly into the group-relative advantage; and it includes verifiable reasoning problems so that factual correctness keeps the policy grounded. A two-stage curriculum trains on objective reasoning and preference rewards for the first 100 steps before self-scored open-ended data is added. The authors argue that this lets generation and evaluation co-evolve, and they present results across Qwen2.5-7B, Qwen3-8B, and OctoThinker-8B-Hybrid-Base showing URPO ahead of GRPO baselines guided by a separate scoring or generative reward model.

Load-bearing premise

The load-bearing premise is that the model's self-assigned scores on open-ended instructions are accurate enough to serve as the RL reward signal, even though nothing outside the model checks those scores and the KL penalty is set to zero.

Editorial extensions

If this is right

  • A single training job on one model can replace the separate reward-model training and policy reinforcement learning of standard RLHF, reducing memory and pipeline complexity.
  • Preference data is a prerequisite for self-rewarding: without it, the model's evaluator collapses to a RewardBench mean of 62.39, so purely self-generated rewards cannot bootstrap a reliable judge from a base model.
  • Injecting verifiable reasoning data into the same RL loop improves the model's evaluative accuracy, lifting RewardBench mean from 83.55 for a dedicated generative reward model to 85.15.
  • A balanced 1:1:1 mixture of preference, reasoning, and instruction data gives the best overall model; dropping instruction data lowers AlpacaEval to 31.43 and dropping reasoning data lowers the reasoning average to 28.58.
  • The method transfers to other base models only when the base model already has a seed of competence, since direct RL on vanilla Llama3.1-8B collapsed while OctoThinker-8B-Hybrid-Base worked.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the open-ended branch uses self-scores with no external verification, the cleanest test of the mechanism is to corrupt those scores, for example by shuffling scores within each group, and check whether the AlpacaEval gain disappears.
  • Editorial inference: the co-evolution story predicts a measurable positive feedback loop, so tracking RewardBench and AlpacaEval at each training checkpoint should show evaluator accuracy improving alongside or just before policy gains, with a plateau in one predicting a plateau in the other.
  • Editorial inference: the ablation showing that reasoning data improves RewardBench suggests that a standalone reward model could also be improved by RL on verifiable reasoning data, a recipe URPO does not itself test outside the unified loop.
  • Editorial inference: with the KL penalty set to zero, stability rests on the 100-step warmup and the clipping range, so varying warmup length would reveal whether external grounding is truly load-bearing or mainly a safeguard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes URPO, a framework that trains a single LLM with a single GRPO loop on three data types: verifiable reasoning, preference pairs, and open-ended instructions. Preference reward modeling is reformulated as a generative scoring task optimized with Kendall's tau correlation against ground-truth rankings; open-ended instructions are rewarded by the model's own generated scores for its sampled responses. The paper reports that on Qwen2.5-7B, URPO outperforms baselines guided by a separate reward model (AlpacaEval 44.84 vs. 42.24, composite reasoning average 35.66 vs. 32.66) and that it achieves a higher RewardBench score than the dedicated reward model it replaces (85.15 vs. 83.55). The paper also includes ablations on data mixture ratios and experiments on Qwen3 and Llama3.1/OctoThinker.

Significance. If the claims hold, URPO is a meaningful simplification of RLHF: it replaces the two-model, multi-stage pipeline with one model and one training phase, and it provides evidence that reasoning data can improve a model's evaluative accuracy. The Kendall-tau formulation of preference ranking as a generative GRPO objective is clean, and the paper's own ablation showing that preference data is a prerequisite for a reliable self-evaluator (Table 5, 0:1:1) is an honest and valuable finding. However, the central empirical claims are currently supported by comparisons that do not hold the policy's training signal fixed, and no statistical uncertainty is reported. The significance is therefore conditional on matched-control experiments and repeated runs.

major comments (4)
  1. [Table 2, Section 4.2.1] The headline comparison is confounded because URPO's policy receives direct preference-data training through the Kendall-tau GRPO objective (Section 3.2.1) and through the Phase 1 warmup on preference data (Section 4.1.4), whereas the +GRPO(RM-gen) baseline policy is trained only on reasoning and open-ended instructions, using the frozen RM-gen for open-ended rewards. The observed AlpacaEval gain of 44.84 vs. 42.24 and the reasoning average gain of 35.66 vs. 32.66 could therefore be caused by the extra preference-data training signal rather than by the claimed co-evolutionary loop. Please provide a matched baseline in which the policy is trained on the same preference triples while the open-ended rewards still come from a frozen external RM, or, alternatively, an URPO variant that removes the preference branch and compares against the external RM with the same remaining data. The warmup schedule and data mixtures for every baseline should be reported explicitly.
  2. [Table 5, Section 4.2.3] The claim that reasoning data enhances evaluative accuracy compares the 1:1:0 URPO mixture (RewardBench Mean 85.15) against the RM-gen baseline (83.55), but RM-gen is trained exclusively on preference data and with a different objective: it is described in Section 4.1.1 as a generation-based reward model trained to emulate the evaluative task format, not as a model trained with GRPO and Kendall's tau on preference plus reasoning. The 1.60-point gap is therefore attributable to the combination of added reasoning data and a different training algorithm, not to reasoning data alone or to the co-evolutionary player/referee design. A clean control would be RM-gen trained on the 1:0:0 preference-only mixture with the same generative GRPO and Kendall-tau objective, and also RM-gen trained on the 1:1:0 mixture under its original recipe.
  3. [Section 4.2, Tables 2-3] No variance, multiple-seed results, or significance tests are reported for any headline number. Several claimed advantages are small on test sets where a few correct answers drive the difference: for example, in Table 2 AIME 2024 is 17.08 vs. 13.33 and HMMT 2025 is 3.33 vs. 1.67, and in Table 3 the Qwen3 reasoning average is actually lower for URPO (39.57) than for RM-gen (39.75). Please run at least three independent seeds for the main comparisons and report standard deviations, or provide a paired bootstrap over test questions, before making statements such as 'unequivocal top performer' or 'significantly outperforms.'
  4. [Section 3.2.2 and Section 4.1.4] The open-ended branch uses self-generated scores as rewards and sets the KL penalty coefficient beta to 0, leaving only the 100-step warmup to establish a reliable internal evaluator. The paper's own ablation in Table 5 shows that when preference data is removed (0:1:1), RewardBench Mean collapses to 62.39, which demonstrates that the self-reward signal is not independently stable. Please provide direct validation of the self-scores on held-out open-ended instructions during training (for example, correlation with human or a strong LLM judge), or explicitly restrict the self-rewarding claim to the regime where preference grounding is present. A plot of self-score accuracy versus training step would also substantiate the claim that the warmup phase is sufficient.
minor comments (5)
  1. [Section 4.1.2] The text names the instruction dataset 'prompt-collection-v0.11' but the footnote says 'prompt-collection-v0.1'; please make the version consistent.
  2. [Table 2 and Table 4] The 'Math Avg' column is an unweighted arithmetic mean of six benchmarks with very different scales (GSM8K near 90, AIME/HMMT near 0-20). This aggregate is dominated by GSM8K and should either be replaced by a principled aggregate or accompanied by per-benchmark significance testing.
  3. [Table 3 and surrounding text] For the Qwen3-8B series, URPO's composite reasoning average (39.57) is slightly lower than RM-gen's (39.75), and the AIME 2024 score is lower (23.33 vs. 26.25). The sentence 'achieved the best performance on a majority of benchmarks' is technically accurate but should be accompanied by the caveat that the overall reasoning average favors the baseline in this model family.
  4. [Section 4.1.4] Setting beta to 0 'completely remov[es] the KL-divergence constraint' is a deliberate deviation from the canonical GRPO objective presented in Section 3.1; please state explicitly in the objective equation that the KL term is omitted in all reported experiments, or show the modified loss.
  5. [Section 3.2.2] The text says the model 'sampl[es] k roll-outs' and then 'G responses' are formatted into a single evaluation prompt; please ensure G and k are used consistently and define the number of responses in the evaluation prompt relative to the GRPO group size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the self-rewarding loop is self-referential by design but is externally benchmarked, and the baseline comparison confound is an experimental-design issue, not a circular reduction.

full rationale

URPO's open-ended branch is self-referential in the sense that the same policy generates responses, scores them, and is updated with those scores (Section 3.2.2), but this is an algorithmic design choice rather than a derivation whose conclusion equals its premise. The paper does not use the self-assigned scores as the evaluation metric: instruction-following is measured on AlpacaEval and reasoning on external math benchmarks, both independent of the self-generated reward. The preference branch supplies ground-truth Kendall's tau rewards and the reasoning branch supplies rule-based rewards, so the self-reward signal is not the only training input. The main empirical comparison is confounded because the URPO policy is trained directly on preference data while the RM-gen baseline's policy is not, but that is an experimental-control issue, not circularity: no fitted parameter is renamed as a prediction, and no load-bearing result is justified by a self-citation chain. The paper cites prior self-rewarding and RL work, but it does not rely on the present authors' prior theorems or uniqueness claims to force its design. Under the required standard that circularity must be exhibited by a specific reduction, no circular step can be identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard policy optimization (GRPO), on the trustworthiness of external preference labels and evaluation benchmarks, and on the stability of self-generated rewards for open-ended tasks. The data mix and hyperparameters are hand-chosen, and the headlining configuration was selected from the ablation results.

free parameters (6)
  • Data mixture ratio P:R:I = 1:1:1
    Selected after the ablation in Table 4; the headline results depend on this hand-chosen mix.
  • Warmup steps = 100
    Phase 1 length chosen by the authors; not derived from a theory or validation sweep.
  • KL coefficient beta = 0
    Set to zero in Section 4.1.4 to maximize exploration; removes the standard regularization against the reference policy.
  • Group size G = 8
    Number of rollouts per prompt in GRPO (Section 4.1.4).
  • Asymmetric clip range = [0.8, 1.28]
    Adopted from DAPO; chosen for training stability.
  • Learning rate = 5e-7
    AdamW learning rate with cosine decay (Section 4.1.4).
assumptions (5)
  • standard math GRPO with group-relative advantages is a valid optimization algorithm for all three task types.
    Taken from Shao et al. 2024; used without modification.
  • domain assumption Ground-truth preference orderings in the five preference datasets are reliable human or AI preferences.
    The Kendall's tau reward is computed against these orderings; noisy labels would bias the referee signal.
  • ad hoc to paper Self-generated scores on open-ended prompts are informative and stable enough to serve as policy rewards.
    Section 3.2.2 feeds self-scores directly into the GRPO advantage with the KL penalty removed; this is the core enabling assumption of the open-ended branch.
  • domain assumption AlpacaEval 2.0, RewardBench, and the math benchmarks measure the intended capabilities.
    The claims are evaluated entirely through these benchmarks and judges.
  • domain assumption Starting base models possess enough seed competence for the unified RL loop.
    The paper reports collapse on vanilla Llama3.1 and switched to OctoThinker, indicating the method depends on base model quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of URPO: A Unified Reward & Policy Optimization Framework for Large Language Models." pith.science (2026). https://pith.science/paper/JMTM4ED7

@misc{pith2026250717515,
  author       = {Pith},
  title        = {Pith review of: URPO: A Unified Reward & Policy Optimization Framework for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JMTM4ED7}},
  note         = {Machine review of arXiv:2507.17515}
}
read the original abstract

Large-scale alignment pipelines typically pair a policy model with a separately trained reward model whose parameters remain frozen during reinforcement learning (RL). This separation creates a complex, resource-intensive pipeline and suffers from a performance ceiling due to a static reward signal. We propose a novel framework, Unified Reward & Policy Optimization (URPO), that unifies instruction-following ("player") and reward modeling ("referee") within a single model and a single training phase. Our method recasts all alignment data-including preference pairs, verifiable reasoning, and open-ended instructions-into a unified generative format optimized by a single Group-Relative Policy Optimization (GRPO) loop. This enables the model to learn from ground-truth preferences and verifiable logic while simultaneously generating its own rewards for open-ended tasks. Experiments on the Qwen2.5-7B model demonstrate URPO's superiority. Our unified model significantly outperforms a strong baseline using a separate generative reward model, boosting the instruction-following score on AlpacaEval from 42.24 to 44.84 and the composite reasoning average from 32.66 to 35.66. Furthermore, URPO cultivates a superior internal evaluator as a byproduct of training, achieving a RewardBench score of 85.15 and surpassing the dedicated reward model it replaces (83.55). By eliminating the need for a separate reward model and fostering a co-evolutionary dynamic between generation and evaluation, URPO presents a simpler, more efficient, and more effective path towards robustly aligned language models.

Figures

Figures reproduced from arXiv: 2507.17515 by the authors.

Figure 1
Figure 1. URPO training loop. A single large language model simultaneously plays the roles of player (Policy Model) and referee (Reward Model). Each mixed training batch contains three data types: (a) rule-verifiable reasoning problems with ground-truth answers, (b) preference triples with N-way ranking, and (c) open-ended instructions. The model first generates candidate responses, then evaluates them with either ground-trut… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.

  2. CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    A cooperative Coach–Player RL loop, rewarded by Player validation-progress, lifts math reasoning by 2–5 points while the Player trains only on self-generated pseudo-labeled tasks.

  3. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  4. Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    The paper introduces the Proxy Compression Hypothesis as a unifying framework explaining reward hacking in RLHF as an emergent result of compressing high-dimensional human objectives into proxy reward signals under op...

Reference graph

Works this paper leans on

24 extracted references · 23 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [4]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307,

    arXiv

  3. [7]

    Skywork open reasoner 1 technical report

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312,

    arXiv

  4. [8]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  5. [10]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267,

    arXiv

  6. [11]

    Skywork-reward: Bag of tricks for reward modeling in llms

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451,

    arXiv

  7. [12]

    Inference-time scaling for generalist reward modeling

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495,

    arXiv

  8. [13]

    Decoupled weight decay regularization

    12 Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

Show all 24 references
  1. [14]

    Bpr: Bayesian personalized ranking from implicit feedback

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618,

    arXiv

  2. [15]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al

    URL https://arxiv.org/abs/1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  3. [16]

    Octothinker: Mid-training incentivizes reinforcement learning scaling

    Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025a. Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Do...

    arXiv

  4. [17]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    arXiv

  5. [18]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    arXiv

  6. [19]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

    arXiv

  7. [20]

    URL https://arxiv.org/abs/2401. 10020. Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation. arXiv preprint arXiv:2402.09267,

    arXiv

  8. [21]

    Learning to reason without external rewards

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590,

    arXiv

  9. [22]

    Rmb: Comprehensively benchmarking reward models in llm alignment

    Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. Rmb: Comprehensively benchmarking reward models in llm alignment. arXiv preprint arXiv:2410.09893,

    arXiv

  10. [23]

    Reinforcing general reasoning without verifiers

    Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493,

    arXiv

  11. [24]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593,

    1909 arXiv

  12. [2017]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  13. [2021]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787,

    arXiv

  14. [2023]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073,

    arXiv

  15. [2024]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  16. [2025]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.