Pith. sign in

REVIEW 3 major objections 4 minor 47 references

Treating reasoning traces as discrete latent variables, trained end-to-end to maximize the scalar reward model's likelihood of the true ranking, yields a reward model the paper shows beats scalar, generative, and hybrid baselines on prefere

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:04 UTC pith:JW3P3BR2

load-bearing objection A clean latent-variable derivation for hybrid RMs, but the empirical edge over the strongest baseline is noise-level and the ELBO gap is unexamined. the 3 major comments →

arxiv 2607.29185 v1 pith:JW3P3BR2 submitted 2026-07-31 cs.CL

Learning Latent Reasoning Traces for Scalar Reward Models End-to-End

Sanwoo Lee , Clive Bai , Hsiu-Yuan Huang , Kun Liang , Weijie Liu , Yunfang Wu This is my paper
classification cs.CL
keywords LatentRMreward modelinglatent reasoning traceschain-of-thoughtpreference learningRLHFPlackett-LuceREINFORCE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a reward model can be improved by treating its intermediate reasoning as a hidden variable rather than as text to be judged separately. It proposes LatentRM, where a generator LLM produces chain-of-thought evaluations and a scalar reward head scores candidate responses conditioned on those traces. The two are trained jointly with one objective: the expected likelihood of the human preference ranking, estimated by sampling traces and applying REINFORCE to the generator. The paper reports that this beats scalar, generative, and hybrid reward models on in-distribution and out-of-distribution preference benchmarks, and that RLHF policies guided by the resulting reward win more against policies trained with baseline reward models, without generating longer responses. A reader should care because it offers a principled alternative to hand-designed reasoning rewards in reward modeling.

Core claim

The central claim is that reasoning traces z are best learned as discrete latent variables in a conditional generative model, where pθ(z|x) is the generator prior over traces and pφ(y|x,z) is a Plackett–Luce ranking likelihood over scalar scores. The paper optimizes the marginal likelihood log p(y|x) by maximizing its evidence lower bound with the variational posterior set equal to the prior, q=pθ, giving the tractable objective E_z~pθ[log pφ(y|x,z)]. The discriminator is updated by this expected log-likelihood; the generator is updated by REINFORCE with the same quantity as reward. The paper reports that this on-policy, end-to-end coupling improves Kendall's tau and log-likelihood on in-dis

What carries the argument

The central object is the conditional generative model with a discrete latent reasoning variable: the generator prior pθ(z|x) and the scalar Plackett–Luce head pφ(y|x,z). The key identity is the lower bound that results from choosing the variational posterior to be the generator itself (q=pθ), which turns the intractable marginalization over reasoning traces into an expectation over samples from the current policy. REINFORCE provides the generator's gradient, using the discriminator's log-likelihood as reward, while the discriminator maximizes the same expected log-likelihood. This identity is what allows a single objective to shape reasoning for the downstream reward, with no hand-designed

Load-bearing premise

The load-bearing premise is that pushing up the average reward-model score over the generator's own reasoning traces keeps improving the model's true ability to rank responses; the paper does not measure the gap between this average and the true likelihood, so if the gap widens, the gains could come from the generator and scorer reinforcing each other's biases rather than from more useful reasoning.

What would settle it

On a held-out preference set, estimate the gap between the distribution of reasoning traces the generator produces and the distribution of traces that would actually explain the observed ranking (e.g., by importance-weighting many sampled traces). If that gap does not shrink as validation likelihood improves—or if a control model whose generator is rewarded with random scores matches LatentRM's gains—then the end-to-end reasoning objective is not what drives the reported improvement.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single training objective replaces hand-designed reasoning rewards: the generator is improved only by how much its traces raise the scalar reward model's likelihood of the true ranking.
  • Because the model trains on traces it actually samples, the method removes the train–inference mismatch that arises when reasoning is distilled from static critiques.
  • The reported out-of-distribution gains imply that reasoning-conditioned scalar scoring transfers to prompts and domains beyond the training distribution, including subtle style-bias and hard reasoning cases.
  • In RLHF, the reward model's preference-modeling strengths carry over to non-stationary policy distributions, yielding higher length-controlled win rates without longer generations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the mechanism is real, the same latent-reasoning objective should transfer to evaluation tasks without explicit pairwise rankings—such as multi-turn dialogue or agent-trajectory assessment—by replacing the Plackett–Luce head with a trajectory-level scorer.
  • Editorial inference: the paper's choice q=pθ is a deliberate simplification; a learned posterior could tighten the bound but risks producing label-matching justifications. A testable middle ground is to anneal a small posterior term during training and check whether validation likelihood improves as the bound gap shrinks.
  • Editorial inference: the setup has an internal feedback loop—the generator is rewarded by the discriminator's own likelihood—so a diagnostic the paper does not report is whether the gap between the generator's trace distribution and the true posterior shrinks over training. Without that measurement, some of the reported gains could be co-adaptation rather than more informative reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. LatentRM proposes a reward-modeling architecture in which a generator LLM produces a chain-of-thought reasoning trace z, and a scalar discriminator then outputs listwise rewards conditioned on (x, z). The two modules are trained end-to-end by maximizing a Jensen lower bound, E_{z~pθ(z|x)}[log pφ(y|x,z)] (Eq. 5), with the generator updated via REINFORCE and the discriminator by direct gradient. The paper reports ID test-set log-likelihoods and Kendall's tau, OOD accuracy on RM-Bench and PPE Correctness, and RLHF LC winrates, claiming that LatentRM outperforms scalar, generative, and hybrid reward models. The gradient derivations are explicit and the evaluation covers several benchmarks, but the empirical comparisons rest on a single run with a fixed seed, and several domain-level differences are small or inconsistent with the stated claims.

Significance. If the reported gains are statistically robust, the contribution is a simple and unified objective for jointly training a reasoning generator and a scalar reward model, avoiding hand-designed rewards for the generator. This is a potentially useful step for reward modeling under distribution shift, and the paper sensibly situates the contribution against existing critique-distillation and multi-task baselines. The derivation of the REINFORCE gradient is correct, the Plackett-Luce setting is appropriate, and the benchmarks—ID, OOD, and end-to-end RLHF—are relevant. However, the central empirical claim is not yet supported: the paper reports no error bars or significance tests, and the margins over the strongest baseline are small (e.g., micro Kendall's tau 0.712 vs. 0.706; RM-Bench 82.8 vs. 81.7; PPE 72.1 vs. 71.9). In addition, the theoretical link between the optimized lower bound and the true marginal likelihood is asserted, not analyzed, which matters because the generator and discriminator are co-adapted and can in principle drift without improving the objective they are intended to maximize.

major comments (3)
  1. [Implementation Details; Tables 2 and 3] The headline claim that LatentRM outperforms all baselines rests on small differences over MultitaskRM: micro Kendall's tau 0.712 vs. 0.706, RM-Bench accuracy 82.8 vs. 81.7, PPE Correctness 72.1 vs. 71.9. The text explicitly states that 'all experiments are based on a single run with a fixed random seed of 42.' For n≈4,800 ID prompts and n=4,000 RM-Bench pairs, these differences are within roughly one standard error of the mean; no confidence intervals or significance tests are provided. The abstract and conclusion claim 'consistent improvements,' but the statistical evidence for superiority over the strongest baseline is currently unverifiable. Please add multiple seeds, error bars, and bootstrap or significance tests for the main comparisons, or temper the claim accordingly.
  2. [Eqs. 4-5; Algorithm 1] The method maximizes E_{z~pθ(z|x)}[log pφ(y|x,z)], which is a Jensen lower bound on the true marginal log p(y|x). The gap between the bound and the marginal is KL(pθ(z|x) || p(z|x,y)), but the paper does not analyze whether this gap shrinks as training proceeds. Because the generator is rewarded by log pφ(y|x,z) and the discriminator is updated on the same generated traces, the two modules can co-adapt: the generator may concentrate on traces that are easy for the current discriminator without increasing true marginal likelihood. Moreover, the 'log-likelihood' reported in Table 2 is pφ(y|x,z) for the latent-augmented model, i.e., the training objective itself, not the marginal likelihood of Eq. 3. Please report a marginal-likelihood estimate (e.g., by importance sampling) or, at minimum, measure the KL gap, the entropy of pθ(z|x), and the sensitivity of the final scalar reward to the rea
  3. [Findings on RLHF; Table 2; Figure 4] The text states that on safety (WildGuard) or adversarial (OffsetBias) domains, LatentRM lags behind MultitaskRM, 'which partially aligns with the ID results in Table 2.' However, Table 2 shows that on OffsetBias LatentRM is better than MultitaskRM (Kendall's tau 0.887 vs. 0.866; log-likelihood -0.149 vs. -0.153). In the RLHF results, LatentRM loses to MultitaskRM on both WildGuard (LC winrate 47.6%) and OffsetBias (48.1%). This is an internal inconsistency between the domain-level narrative and the tabulated ID results, and it should be reconciled or explained explicitly.
minor comments (4)
  1. [Introduction; Figure 5; Data section] There are several typos and formatting issues: 'acalar RM' in the introduction and Figure 1 caption, 'resposne' in the data-filtering description, and 'langauge' in the Figure 5 caption. These should be corrected.
  2. [Findings on In-distribution Test Sets] The text says 'From ID test set evaluation results in Table 1,' but Table 1 is the dataset-statistics table; the results appear in Table 2. The cross-reference should be fixed.
  3. [Implementation Details] Please clarify whether the token-level KL penalty coefficient of 1e-3 applies to LatentRM or only to the RL baselines. The current wording—'Token-level KL penalty coefficient is set to 1×10−3 for baselines as in (Chen et al. 2026)'—can be read either way, and the comparison's fairness depends on this detail.
  4. [Contributions; Algorithm 1] The claim in the introduction that LatentRM introduces 'no hyperparameter overhead' is overstated: the number of rollouts m, the sampling temperature, the invalid-rollout advantage penalty, and the data-filtering fraction are method hyperparameters introduced or used in the pipeline. This should be reworded or qualified.

Circularity Check

0 steps flagged

No significant circularity: the objective is grounded in external preference labels and validated on held-out/OOD benchmarks.

full rationale

The derivation chain is self-contained and anchored to external human preference labels y, not to the model's own outputs. Eq. 3 defines the marginal likelihood, Eq. 4 gives a valid ELBO, and setting q_phi(z|x,y)=p_theta(z|x) is an explicit modeling choice that still satisfies Jensen's inequality and yields Eq. 5. The gradients in Eqs. 6-7 are the standard Plackett-Luce and REINFORCE derivatives, and Algorithm 1 estimates them with rollouts and a mean baseline. No fitted parameter is renamed as a prediction: the generator is rewarded with log p_phi(y|x,z) computed against ground-truth y, and phi is trained on the same external labels. Held-out evaluations (the ID test split, RM-Bench, PPE Correctness, and RLHF winrates) use external benchmarks and ground-truth preferences, so the reported improvements are not identities. The ID log-likelihood metric coincides with the objective being optimized, but that is an ordinary held-out evaluation of the optimized quantity rather than a circular reduction; the main OOD and RLHF results are independent of that objective. The variational gap KL(p_theta(z|x) || p(z|x,y)) is not analyzed, and the experiments are single-run, but these are robustness limitations, not circularity. No load-bearing self-citation or imported uniqueness theorem is present.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

All central-claim dependencies are standard ML modeling choices and the paper's own architectural decisions. No new physical entities are introduced. The most consequential free choices are the m=8 rollout count, the learning rates, the 40% data filter, and the invalid-trace penalty; the most consequential axioms are the tightness of the q=pθ bound and the stability of unregularized REINFORCE.

free parameters (4)
  • m (rollouts per prompt) = 8
    Number of reasoning traces sampled per instance for REINFORCE; chosen by the authors, not derived. Affects variance of the policy gradient and compute.
  • Learning rates eta_theta, eta_phi = 2e-6 / 1e-5
    Set in Implementation Details; not tuned on test set, but still a free choice that the central results depend on.
  • Data filtering fraction = 40%
    40% of the initial pool is discarded based on an MLP ensemble trained on hand-crafted LFTK features; the threshold and feature set are author choices that shape all subsequent results.
  • Invalid-rollout advantage penalty = min_j A(i,j) - 1 (or -1 if no valid rollouts)
    A heuristic to discourage format-invalid traces; no justification beyond variance reduction.
axioms (6)
  • domain assumption The Plackett-Luce model over scalar rewards is an adequate likelihood for human preference rankings (Eq. 1).
    Standard in RM literature but a modeling choice; the paper does not validate it against other preference models in this setting.
  • domain assumption The conditional generative model pθ(z|x)pφ(y|x,z) with latent reasoning z is an adequate representation of the reward modeling problem.
    The entire derivation starts from this graphical model (Figure 3); if reasoning traces are not a meaningful latent cause of preferences, the objective is misspecified.
  • ad hoc to paper Choosing qϕ(z|x,y)=pθ(z|x) yields a useful lower bound; optimizing it improves the true marginal likelihood.
    The gap KL(pθ||posterior) is ignored; no evidence is provided that the bound is tight or that the gap shrinks during training.
  • ad hoc to paper The generator can be trained with REINFORCE and no KL penalty to the reference policy without collapsing or reward hacking.
    Algorithm 1 updates θ purely from log pφ advantages; no regularization term is present, unlike standard RLHF.
  • ad hoc to paper The scalar head can be randomly initialized and co-trained from step 0 with no warmup and remain stable.
    The paper states 'not requiring warmup steps'; this is an empirical claim with no analysis.
  • domain assumption Hand-crafted feature filtering removes noisy samples without biasing the train/test distribution.
    The split-and-filter protocol discards 40% of data based on shallow features; if those features encode spurious correlations, the filtering may remove informative hard examples.

pith-pipeline@v1.3.0-daily-deepseek · 14822 in / 16819 out tokens · 172013 ms · 2026-08-03T12:04:19.829641+00:00 · methodology

0 comments
read the original abstract

Reward models (RMs) are central to aligning large language models with human preferences via reinforcement learning. Although traditional scalar RMs enable efficient and probabilistic reward modeling, they rely on superficial cues that fail to generalize to complex or out-of-distribution (OOD) tasks. Conversely, generative RMs leverage extensive reasoning to improve robustness on challenging tasks, but their natural language-based scores lack the numerical flexibility and probabilistic interpretability that scalar RMs offer. While recent approaches combine both paradigms through off-policy multi-task learning, such parallel optimization does not guarantee that generated reasoning traces actively align with or benefit downstream scalar reward prediction. To address this mismatch, we propose LatentRM, a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize the likelihood of downstream scalar rewards. Through on-policy optimization of the latent reasoning space end-to-end, LatentRM tightly couples deep reasoning-based evaluation with precise scoring. Extensive validations on in-distribution and OOD datasets and RLHF show that LatentRM outperforms scalar, generative, and hybrid RMs on preference modeling and policy alignment across tasks ranging from open-ended conversation to complex reasoning.

Figures

Figures reproduced from arXiv: 2607.29185 by Clive Bai, Hsiu-Yuan Huang, Kun Liang, Sanwoo Lee, Weijie Liu, Yunfang Wu.

Figure 1
Figure 1. Figure 1: Rank correlation coefficient against groundtruth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Graphical models illustrating differences between [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Length-controlled (LC) win rate and token length of post-RLHF policy rewarded by [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-sample score gaps [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: This prompt template instructs an LLM generator [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: The complete prompt template of our listwise generator. Brackets in blue denotes contents to be filled with conversation [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 2 canonical work pages

  1. [1]

    arXiv preprint arXiv:2204.05862 , year=

    Training a helpful and harmless assistant with reinforcement learning from human feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:1909.08593 , year=

    Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=

    Pith/arXiv arXiv 1909

  3. [3]

    Fine-Grained Human Feedback Gives Better Rewards for Language Model Training , url =

    Wu, Zeqiu and Hu, Yushi and Shi, Weijia and Dziri, Nouha and Suhr, Alane and Ammanabrolu, Prithviraj and Smith, Noah and Ostendorf, Mari and Hajishirzi, Hannaneh , booktitle =. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training , url =

  4. [4]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Scaling Laws for Reward Model Overoptimization , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  5. [5]

    Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models , url =

    Hong, Ilgee and Yu, Changlong and Qiu, Liang and Yan, Weixiang and Xu, Zhenghao and Jiang, Haoming and Zhang, Qingru and Lu, Qin and Liu, Xin and Zhang, Chao and Zhao, Tuo , booktitle =. Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models , url =

  6. [6]

    2026 , url=

    Xiusi Chen and Gaotang Li and Ziqi Wang and Bowen Jin and Cheng Qian and Yu Wang and Hongru WANG and Yu Zhang and Denghui Zhang and Tong Zhang and Hanghang Tong and Heng Ji , booktitle=. 2026 , url=

    2026

  7. [7]

    J1: Incentivizing Thinking in

    Chenxi Whitehouse and Tianlu Wang and Ping Yu and Xian Li and Jason E Weston and Ilia Kulikov and Swarnadeep Saha , booktitle=. J1: Incentivizing Thinking in. 2026 , url=

    2026

  8. [8]

    Learning Structured Output Representation using Deep Conditional Generative Models , url =

    Sohn, Kihyuk and Lee, Honglak and Yan, Xinchen , booktitle =. Learning Structured Output Representation using Deep Conditional Generative Models , url =

  9. [9]

    Training language models to follow instructions with human feedback , url =

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul F and Leike, Jan and Lowe,...

  10. [10]

    InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling , url =

    Miao, Yuchun and Zhang, Sen and Ding, Liang and Bao, Rong and Zhang, Lefei and Tao, Dacheng , booktitle =. InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling , url =. doi:10.52202/079017-4270 , editor =

    doi:10.52202/079017-4270

  11. [11]

    Improving Reward Models with Synthetic Critiques

    Ye, Zihuiwen and Greenlee, Fraser David and Bartolo, Max and Blunsom, Phil and Campos, Jon Ander and Gall \'e , Matthias. Improving Reward Models with Synthetic Critiques. Findings of the Association for Computational Linguistics: NAACL 2025. 2025. doi:10.18653/v1/2025.findings-naacl.254

    doi:10.18653/v1/2025.findings-naacl.254 2025

  12. [12]

    Self-Generated Critiques Boost Reward Modeling for Language Models

    Yu, Yue and Chen, Zhengxing and Zhang, Aston and Tan, Liang and Zhu, Chenguang and Pang, Richard Yuanzhe and Qian, Yundi and Wang, Xuewei and Gururangan, Suchin and Zhang, Chao and Kambadur, Melanie and Mahajan, Dhruv and Hou, Rui. Self-Generated Critiques Boost Reward Modeling for Language Models. Proceedings of the 2025 Conference of the Nations of the ...

    doi:10.18653/v1/2 2025

  13. [13]

    arXiv preprint arXiv:2408.11791 , year=

    Critique-out-loud reward models , author=. arXiv preprint arXiv:2408.11791 , year=

    Pith/arXiv arXiv

  14. [14]

    Generative Judge for Evaluating Alignment , url =

    Li, Junlong and Sun, Shichao and Yuan, Weizhe and Fan, Run-Ze and zhao, hai and Liu, Pengfei , booktitle =. Generative Judge for Evaluating Alignment , url =

  15. [15]

    Deep Reinforcement Learning from Human Preferences , url =

    Christiano, Paul F and Leike, Jan and Brown, Tom and Martic, Miljan and Legg, Shane and Amodei, Dario , booktitle =. Deep Reinforcement Learning from Human Preferences , url =

  16. [16]

    The Twelfth International Conference on Learning Representations , year=

    Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models , author=. The Twelfth International Conference on Learning Representations , year=

  17. [17]

    Terry , journal =

    Ralph Allan Bradley and Milton E. Terry , journal =. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , urldate =

  18. [18]

    1959 , publisher=

    Individual Choice Behavior: A Theoretical Analysis , author=. 1959 , publisher=

    1959

  19. [19]

    Rationalizing Neural Predictions

    Lei, Tao and Barzilay, Regina and Jaakkola, Tommi. Rationalizing Neural Predictions. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1011

    doi:10.18653/v1/d16-1011 2016

  20. [20]

    STaR: Bootstrapping Reasoning With Reasoning , url =

    Zelikman, Eric and Wu, Yuhuai and Mu, Jesse and Goodman, Noah , booktitle =. STaR: Bootstrapping Reasoning With Reasoning , url =. doi:10.52202/068431-1126 , editor =

    work page doi:10.52202/068431-1126

  21. [21]

    Saurous, Rif , booktitle =

    Phan, Du and Hoffman, Matthew Douglas and Dohan, David and Douglas, Sholto and Le, Tuan Anh and Parisi, Aaron and Sountsov, Pavel and Sutton, Charles and Vikram, Sharad and A. Saurous, Rif , booktitle =. Training Chain-of-Thought via Latent-Variable Inference , url =. doi:10.52202/075280-3184 , editor =

    work page doi:10.52202/075280-3184

  22. [22]

    Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data , url =

    Tang, Yunhao and Wang, Sid and Madaan, Lovish and Munos, Remi , booktitle =. Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data , url =

  23. [23]

    2026 , eprint=

    Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling , author=. 2026 , eprint=

    2026

  24. [24]

    Policy Gradient Methods for Reinforcement Learning with Function Approximation , url =

    Sutton, Richard S and McAllester, David and Singh, Satinder and Mansour, Yishay , booktitle =. Policy Gradient Methods for Reinforcement Learning with Function Approximation , url =

  25. [25]

    2024 , editor =

    Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and He, Bingxiang and Zhu, Wei and Ni, Yuan and Xie, Guotong and Xie, Ruobing and Lin, Yankai and Liu, Zhiyuan and Sun, Maosong , booktitle =. 2024 , editor =

    2024

  26. [26]

    arXiv preprint arXiv:2504.16891 , year=

    Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset , author=. arXiv preprint arXiv:2504.16891 , year=

    Pith/arXiv arXiv

  27. [27]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  28. [28]

    WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs , url =

    Han, Seungju and Rao, Kavel and Ettinger, Allyson and Jiang, Liwei and Lin, Bill Yuchen and Lambert, Nathan and Choi, Yejin and Dziri, Nouha , booktitle =. WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs , url =. doi:10.52202/079017-0261 , editor =

    doi:10.52202/079017-0261

  29. [29]

    O ffset B ias: Leveraging Debiased Data for Tuning Evaluators

    Park, Junsoo and Jwa, Seungyeon and Meiying, Ren and Kim, Daeyoung and Choi, Sanghyuk. O ffset B ias: Leveraging Debiased Data for Tuning Evaluators. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.57

    doi:10.18653/v1/2024.findings-emnlp.57 2024

  30. [30]

    Forty-second International Conference on Machine Learning , year=

    Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples , author=. Forty-second International Conference on Machine Learning , year=

  31. [31]

    and Lee, Jason

    Lee, Bruce W. and Lee, Jason. LFTK : Handcrafted Features in Computational Linguistics. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). 2023. doi:10.18653/v1/2023.bea-1.1

    doi:10.18653/v1/2023.bea-1.1 2023

  32. [32]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  33. [33]

    and Hajishirzi, Hannaneh

    Lambert, Nathan and Pyatkin, Valentina and Morrison, Jacob and Miranda, LJ and Lin, Bill Yuchen and Chandu, Khyathi and Dziri, Nouha and Kumar, Sachin and Zick, Tom and Choi, Yejin and Smith, Noah A. and Hajishirzi, Hannaneh. R eward B ench: Evaluating Reward Models for Language Modeling. Findings of the Association for Computational Linguistics: NAACL 20...

    doi:10.18653/v1/2025.findings-naacl.96 2025

  34. [34]

    The Fourteenth International Conference on Learning Representations , year=

    RewardBench 2: Advancing Reward Model Evaluation , author=. The Fourteenth International Conference on Learning Representations , year=

  35. [35]

    2025 , url=

    Yantao Liu and Zijun Yao and Rui Min and Yixin Cao and Lei Hou and Juanzi Li , booktitle=. 2025 , url=

    2025

  36. [36]

    Gonzalez and Ion Stoica , booktitle=

    Evan Frick and Tianle Li and Connor Chen and Wei-Lin Chiang and Anastasios Nikolas Angelopoulos and Jiantao Jiao and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica , booktitle=. How to Evaluate Reward Models for. 2025 , url=

    2025

  37. [37]

    JudgeBench: A Benchmark for Evaluating

    Sijun Tan and Siyuan Zhuang and Kyle Montgomery and William Yuan Tang and Alejandro Cuadron and Chenguang Wang and Raluca Popa and Ion Stoica , booktitle=. JudgeBench: A Benchmark for Evaluating. 2025 , url=

    2025

  38. [38]

    Proceedings of the Twentieth European Conference on Computer Systems , pages=

    Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=

  39. [39]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  40. [40]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  42. [42]

    Second Conference on Language Modeling , year=

    Understanding R1-Zero-Like Training: A Critical Perspective , author=. Second Conference on Language Modeling , year=

  43. [43]

    2024 , eprint=

    Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs , author=. 2024 , eprint=

    2024

  44. [44]

    First Conference on Language Modeling , year=

    Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators , author=. First Conference on Language Modeling , year=

  45. [45]

    2026 , eprint=

    PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling , author=. 2026 , eprint=

    2026

  46. [46]

    and Sreedhar, Makesh Narsimhan and Kuchaiev, Oleksii , booktitle =

    Wang, Zhilin and Dong, Yi and Delalleau, Olivier and Zeng, Jiaqi and Shen, Gerald and Egert, Daniel and Zhang, Jimmy J. and Sreedhar, Makesh Narsimhan and Kuchaiev, Oleksii , booktitle =. HelpSteer 2: Open-source dataset for training top-performing reward models , url =. doi:10.52202/079017-0047 , editor =

    doi:10.52202/079017-0047

  47. [47]

    2025 , eprint=

    Inference-Time Scaling for Generalist Reward Modeling , author=. 2025 , eprint=

    2025