Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Intra-Trajectory Consistency for Reward Modeling

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that adding an intra-trajectory consistency regularizer, which forces adjacent response prefixes with high next-token generation probability to receive similar rewards, improves how well outcome reward models generalize…

desk verdict A clearly specified, empirically consistent regularization for outcome reward models; the gains are modest and the prefix-label assumption is unvalidated, but the method deserves a serious referee. read the letter →

arxiv 2506.09096 v3 pith:NUOI7CQO submitted 2025-06-10 cs.LG cs.AI

Chaoyang Zhou , Shunyu Liu , Zengmao Wang , Di Wang , Rong-Cheng Tu , Bo Du , Dacheng Tao This is my paper
classification cs.LGcs.AI
keywords rewardmodelingintra-trajectoryconsistencygenerationprobabilityprocessoutcomeRLHFbest-of-Nverificationgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that reward models trained only on whole-response preference labels can be made to generalize much better by adding a cheap, annotation-free auxiliary signal drawn from inside each response. Its idea is that adjacent prefixes of a response are not independent: when the generator assigns a high probability to the next token, the rewards of the two neighboring prefixes should be close. The proposed intra-trajectory consistency regularizer enforces that agreement, propagating the response-level preference down to individual prefixes. On RewardBench with 40K Unified-Feedback samples, the regularized model reaches an average accuracy of 75.8 against 73.0 for the reproduced baseline, and the gains also appear in DPO-trained policies and best-of-N verification. If correct, this gives reward modeling a way to use fine-grained process information without paying for process-level annotations.

What carries the argument

The central object is the intra-trajectory consistency regularizer (ICRM), named in the paper. A process is a prefix $y_{1:k}$ of a generated response, and the regularizer works on adjacent process pairs $(y_{1:k-1}, y_{1:k})$. For each pair it forms a weight from two factors: the frozen generator's next-token probability $P(y_k \mid x, y_{1:k-1})$, and the calibrated reward of the paired process under the response's preference label. These weights enter a binary cross-entropy loss (Eq. 8) that pushes each process reward toward the preference label with strength controlled by the neighboring process reward, so high-probability adjacent processes end up with similar rewards. The reward used in the weights is a mean-centered calibrated sigmoid of the model's score, computed against the mean score of the opposite response. The regularizer performs the propagation: response-level labels flow to individual prefixes without new annotations, and the Bayesian argument in Eq. (3) supplies the rationale for weighting by generation probability.

What would settle it

Construct a training set where chosen answers deliberately open with a wrong or low-quality step before correcting, and rejected answers open correctly before failing; if the regularized model's accuracy on RewardBench or best-of-N does not drop below the baseline under this label noise, the claimed propagation of response-level preferences through prefixes is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that response-level supervision is coarse because the model cannot tell which part of a response drove the score, and that generation probabilities expose the right fine-grained structure. The paper derives a Bayesian decomposition in which the reward of an earlier prefix equals the reward of a later continuation weighted by the continuation's generation probability plus the contribution of all alternative continuations; hence, as the next-token probability rises, the two adjacent rewards should converge. The method therefore trains process rewards on a mutually weighted binary cross-entropy objective whose weights combine the generator's next-token probability and the calibrated reward of the neighboring prefix, and it adds this regularizer to the standard Bradley-Terry outcome-reward loss. Experiments report consistent average gains over the GRM baseline on RewardBench at 4K, 10K, 40K, and 400K training samples, better DPO policies, and better best-of-N selection.

Load-bearing premise

The method assumes every opening portion of a chosen answer is itself good and every opening portion of a rejected answer is itself bad, even though a real answer can start badly and recover or start well and collapse.

Editorial extensions

If this is right

  • Across 4K, 10K, 40K, and 400K Unified-Feedback training samples, ICRM beats the GRM baseline on average RewardBench accuracy, so the gain is not tied to a single data scale.
  • DPO policies trained with the ICRM reward model obtain higher gold-model scores and a higher win ratio than policies trained with the GRM reward model.
  • Best-of-N verification improves for both Mistral-Instructor-v0.3 and Llama-3-8B-Instruct policies, with the larger advantage on Llama-3-8B-Instruct.
  • Using an exponential moving average of process rewards at inference improves reasoning accuracy for both methods, and ICRM again stays ahead of GRM.
  • Policies induced by ICRM produce shorter responses than GRM-induced policies, suggesting the method reduces reliance on response length as a quality proxy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same loss in any preference-data setting where the generator is white-box, which covers most RLHF pipelines; the paper's end-to-end shared-backbone variant shows the separate generator can be folded into training when data are small.
  • The benefit likely depends on the generator's probability estimates, so pairing the regularizer with the true generator should matter more as data grow and the training distribution shifts.
  • For responses whose quality reverses mid-stream, the response-level label will mislabel early prefixes; a variant that derives prefix labels from reward differences rather than from the response label would be a natural robustness extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ICRM, a regularization method for outcome reward models. A frozen generator supplies next-token probabilities, and the reward model is trained with the standard Bradley-Terry loss plus an intra-trajectory consistency loss (Eq. 8) that encourages adjacent prefixes of the chosen response to receive consistently high rewards and adjacent prefixes of the rejected response to receive consistently low rewards, with the strength of each pairwise constraint weighted by the generator's next-token probability. The method is evaluated on RewardBench (Unified-Feedback and Skywork data, Gemma-2B and Llama3-8B bases), on DPO policies trained with the resulting reward model, and on best-of-N verification for math and general tasks. The reported results show consistent average improvements over the GRM baseline, and ablations examine the contribution of the two weighting factors and compare with an L1 consistency loss.

Significance. If the empirical gains are robust, ICRM is a simple, annotation-free auxiliary loss that improves reward-model generalization and downstream verification, which would be a useful contribution to RLHF and inference-time verification. The paper has concrete strengths: Eq. (8) is clearly specified, the code is released, the evaluation spans three distinct tasks, and the ablations isolate the role of the two weighting factors. However, the central mechanism claim rests on an unvalidated mapping from response-level preference labels to every process-level prefix label, and most reported improvements are small single-run differences without statistical assessment. The Bayesian derivation in Eq. (3) is a conditional-probability identity used as motivation rather than a theorem that pins down the specific loss form.

major comments (3)
  1. [Section 3.2, Eq. (8)] The regularizer assigns process-level labels identical to the response preference label: every prefix of the chosen response receives s=1 and every prefix of the rejected response receives s=0. This mapping is load-bearing for the claimed mechanism, but it is not implied by Eq. (3), which is a conditional-expectation relation among rewards of whole continuations. In realistic trajectories a chosen response can begin with an incorrect step and later correct itself, while a rejected response can contain a correct partial derivation. Because the weights in Eqs. (6)-(7) multiply by the generator's next-token probability, the most fluent or common prefixes receive the strongest updates; if those prefixes carry content opposite to the response-level label, the auxiliary loss injects systematic label noise into the process-level cross-entropy objective. The Limitations section discusses only computational overhead and model scale, not this label-semantics risk. I would like to see a step-level validation, for example process-reward accuracy on a step-labeled dataset such as PRM800K, or constructed examples of wrong-start-then-correct chosen responses, or a comparison with a label-free smoothing regularizer, to establish that the gains come from genuine consistency rather than from generic regularization effects.
  2. [Section 4.2-4.4, Tables 1, 3, 4] The experimental results are reported as single runs without error bars, seeds, or significance tests. The reproduced GRM baseline differs substantially from the published GRM numbers (Table 1: 73.0 vs 69.5 on 40K, 73.2 vs 71.5 on 400K), which makes the comparison sensitive to implementation details. Several of the reported ICRM advantages are very small, for example the gold score of 0.678 vs 0.676 in Table 3, the Pass@2 result of 11.8 vs 11.8 in Table 4, and the 47.3 vs 50.6 win ratio in Table 3. Given these margins, the central claim that ICRM improves reward-model generalization would be strengthened considerably by repeated runs with reported mean and standard deviation, or by a statistical test on the main RewardBench and best-of-N comparisons.
  3. [Section 3.1, Eq. (3)] The Bayesian decomposition in Eq. (3) is a law of total probability and is not circular, but the step from 'the contribution of alternative completions diminishes' to 'higher consistency between r(x,y_{1:m}) and r(x,y_{1:n})' is heuristic. The subsequent regularizer in Eq. (8) is a weighted binary cross-entropy loss with detached weights, not a direct minimization of reward variance or of a distance between adjacent reward estimates. The L1 comparison in Appendix C.5 is a useful empirical check, but it does not provide the formal link that the phrase 'grounded in a Bayesian framework' suggests. I recommend that the authors either state explicitly that Eq. (3) is a motivation rather than a derivation, or provide a precise statement of the consistency property that Eq. (8) is intended to enforce.
minor comments (5)
  1. [Section 2] The sentence containing 'weesponse segment y_{1:m} as a valid process' is garbled and should be rewritten.
  2. [Eq. (9)] The loss symbol 'Ltoal' appears to be a typo for 'L_total'.
  3. [Section 4.2] The heading 'Results on ReardBench Benchmark' contains a typo; it should read 'RewardBench'.
  4. [Figure 2 caption] The caption contains the typo 'Gnenration model'; it should read 'Generation model'.
  5. [Table 2 caption] The phrase 'Best results is highlighted in bold' should be 'Best results are highlighted in bold'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Eq. (3) is a law-of-total-probability decomposition and the central gains are empirical, so the derivation chain is not forced by definition.

full rationale

The paper's claimed derivation chain is self-contained. Eq. (3) is an exact conditional-probability decomposition once r(x,y1:m) is interpreted as the probability that a completion of the prefix achieves the maximum score, following [Li and Li, 2025]; the consistency conclusion (higher next-token probability makes the prefix reward closer to the successor reward) follows from the same identity and is not fitted. The regularization in Eq. (8) copies the response-level preference label onto every prefix as an explicit modeling assumption, not as a prediction derived from independent data, so it is an assumption about label semantics rather than a circular reduction. The weights in Eqs. (6)-(7) are detached self-predictions and generator probabilities; they make the auxiliary loss self-referential but do not make the outcome equal to the input by construction. The central claim is an empirical comparison against GRM on the external RewardBench benchmark, plus DPO and BON evaluations, so the result is not forced by construction. The only self-citation (Liu et al., 2025, a survey of DPO) is non-load-bearing; the Limitations section addresses computational overhead only, leaving the prefix-label assumption as a robustness risk, but that is a correctness/assumption concern, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claim depends on the assumptions listed above, especially the transfer of response-level preference labels to every prefix. The alpha hyper-parameter is a hand-set balance, and the EMA decay is a hand-set inference choice. No new physical or architectural entities are introduced; 'process' is defined as any prefix of a response.

free parameters (2)
  • Alpha (consistency loss weight) = 0.1
    Balance hyper-parameter in Eq. (9), set manually to 0.1 for all experiments. It controls how much the intra-trajectory consistency regularizer contributes relative to the Bradley-Terry loss.
  • EMA decay factor for process rewards = 0.5
    Decay factor used when averaging process rewards backward from the final token at inference time (Table 2 and Appendix C.2). It is chosen by hand and is not derived from the theory.
assumptions (4)
  • domain assumption Reward r(x,y) can be expressed as the probability that a response generated conditioned on (x,y) achieves the maximum score.
    Invoked in Section 3.1 to replace P(e|x,y) with r(x,y) in the Bayesian decomposition, following Li and Li (2025). This identifies scalar rewards with event probabilities and is not independently established in the paper.
  • domain assumption Responses are generated by an autoregressive generator with an estimable conditional probability distribution.
    Section 3.1 assumes each new token depends probabilistically on all previous tokens, so that next-token probabilities can stand in for P(x,y1:n|x,y1:m). This is standard for LLMs, but the generator used in experiments is trained on the same preference data.
  • ad hoc to paper Process semantics remain stable under limited suffix additions.
    Used in Section 3.1 to restrict consistency to adjacent processes and to justify treating minimal token extensions as semantically continuous. It is stated as an assumption inspired by text augmentation, without direct validation.
  • ad hoc to paper Every process in a chosen response is preferred, and every process in a rejected response is dispreferred.
    Section 3.2 assigns process-level labels identical to the response preference label, which is the target for the binary cross-entropy regularization. This is the load-bearing premise behind the process supervision, and the paper does not discuss its failure modes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Intra-Trajectory Consistency for Reward Modeling." pith.science (2026). https://pith.science/paper/NUOI7CQO

@misc{pith2026250609096,
  author       = {Pith},
  title        = {Pith review of: Intra-Trajectory Consistency for Reward Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUOI7CQO}},
  note         = {Machine review of arXiv:2506.09096}
}
read the original abstract

Reward models are critical for improving large language models (LLMs), particularly in reinforcement learning from human feedback (RLHF) or inference-time verification. Current reward modeling typically relies on scores of overall responses to learn the outcome rewards for the responses. However, since the response-level scores are coarse-grained supervision signals, the reward model struggles to identify the specific components within a response trajectory that truly correlate with the scores, leading to poor generalization on unseen responses. In this paper, we propose to leverage generation probabilities to establish reward consistency between processes in the response trajectory, which allows the response-level supervisory signal to propagate across processes, thereby providing additional fine-grained signals for reward learning. Building on analysis under the Bayesian framework, we develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards. We apply the proposed regularization to the advanced outcome reward model, improving its performance on RewardBench. Besides, we show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results. Our code is provided in https://github.com/chaoyang101/ICRM.

Figures

Figures reproduced from arXiv: 2506.09096 by the authors.

Figure 1
Figure 1. Illustration of our proposed framework. Within this framework, while training the reward model to learn [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Length distribution of response of various policies induced by GRM and ICRM. Average length is marked by [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of the rewards acquired by ICRM for different processes, in which the reward of a process is shown [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 47 canonical work pages

  1. [1]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 35: 0 27730--27744, 2022

    work page 2022

  2. [2]

    Safe RLHF : Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF : Safe reinforcement learning from human feedback. In ICLR, 2024

    work page 2024

  3. [3]

    Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In ICML, 2024

    work page 2024

  4. [4]

    A survey of direct preference optimization

    Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, and Dacheng Tao. A survey of direct preference optimization. arXiv preprint arXiv:2503.11701, 2025

    arXiv 2025

  5. [5]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In NeurIPS Workshop, 2024 a

    work page 2024

  6. [6]

    Rewarding progress: Scaling automated process verifiers for LLM reasoning

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In ICLR, 2025

    work page 2025

  7. [7]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In ICML, pages 10835--10866, 2023

    work page 2023

  8. [8]

    Regularizing hidden states enables learning generalizable reward model for LLM s

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for LLM s. In NeurIPS, 2024 a

    work page 2024

Show all 52 references
  1. [9]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In ICLR, 2024

    2024

  2. [10]

    Warm: On the benefits of weight averaged reward models

    Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi-Tazehozi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. In ICML, pages 42048--42073, 2024

    2024

  3. [11]

    The trickle-down impact of reward inconsistency on RLHF

    Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward inconsistency on RLHF . In ICLR, 2024

    2024

  4. [12]

    Rrm: Robust reward model training mitigates reward hacking

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiia Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. Rrm: Robust reward model training mitiga...

    2024

  5. [13]

    Length-controlled alpacaeval: A simple debiasing of automatic evaluators

    Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In COLM, 2024

    2024

  6. [14]

    Odin: Disentangled reward mitigates hacking in rlhf

    Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In ICML, pages 7935--7952, 2024 a

    2024

  7. [15]

    Improving discriminative capability of reward models in rlhf using contrastive learning

    Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, et al. Improving discriminative capability of reward models in rlhf using contrastive learning. In EMNLP, pages 15270--15283, 2024 b

    2024

  8. [16]

    Rethinking reward modeling in preference-based large language model alignment

    Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference-based large language model alignment. In ICLR, 2025

    2025

  9. [17]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2023

    2023

  10. [18]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In ACL, pages 9426--9439, 2024 a

    2024

  11. [19]

    Rest-mcts*: Llm self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. NeurIPS, 37: 0 64735--64772, 2024 b

    2024

  12. [20]

    Gemma: Open models based on gemini research and technology

    Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, et al. Gemma: Open models based on gemini research and technology. ArXiv, 2024

    2024

  13. [21]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, et al. The llama 3 herd of models. ArXiv, 2024

    2024

  14. [22]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, 2022

    2022

  15. [23]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, 2024

    2024

  16. [24]

    Solving math word problems with process-and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. ArXiv, 2022

    2022

  17. [25]

    Token-level direct preference optimization

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In ICML, pages 58348--58365, 2024

    2024

  18. [26]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. ArXiv, 2025

    2025

  19. [27]

    Gen ARM : Reward guided generation with autoregressive reward model for test-time alignment

    Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Gen ARM : Reward guided generation with autoregressive reward model for test-time alignment. In ICLR, 2025

    2025

  20. [28]

    Fine-grained human feedback gives better rewards for language model training

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. NeurIPS, 36: 0 59008--59033, 2023

    2023

  21. [29]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. In ICLR, 2025

    2025

  22. [30]

    Process reward model with q-value rankings

    Wendi Li and Yixuan Li. Process reward model with q-value rankings. In ICLR, 2025

    2025

  23. [31]

    Coda: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding

    Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Jiawei Han, and Weizhu Chen. Coda: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding. In ICLR, 2021

    2021

  24. [32]

    A survey on data augmentation for text classification

    Markus Bayer, Marc-Andr \'e Kaufhold, and Christian Reuter. A survey on data augmentation for text classification. ACM Computing Surveys, 55 0 (7): 0 1--39, 2022

    2022

  25. [33]

    Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel

    Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020

    2020

  26. [34]

    Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling

    Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurIPS, 34: 0 18408--18419, 2021

    2021

  27. [35]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, et al. Qwen2 technical report. Arxiv, 2024 b

    2024

  28. [36]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, 2021

    2021

  29. [37]

    Free process rewards without process labels

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. ArXiv, 2024

    2024

  30. [38]

    Mistral 7b

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    arXiv

  31. [39]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023

    2023

  32. [40]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. ArXiv, 2024

    2024

  33. [41]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. ArXiv, 2024

    2024

  34. [42]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, 2023

    2023

  35. [43]

    Secrets of rlhf in large language models part ii: Reward modeling

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. ArXiv, 2024 b

    2024

  36. [44]

    Reward model ensembles help mitigate overoptimization

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. ArXiv, 2023

    2023

  37. [45]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In NeurIPS, 2023

    2023

  38. [46]

    Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback

    Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuan-Jing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In EMNLP, pages 2859--2873, 2023

    2023

  39. [47]

    Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, Dj Dvijotham, Adam Fisch, Katherine Heller, Stephen R. Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hack...

    2023

  40. [48]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 2017

    2017

  41. [49]

    Understanding the learning dynamics of alignment with human feedback

    Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. In ICML, pages 20983--21006, 2024

    2024

  42. [50]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. ArXiv, 2024

    2024

  43. [51]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In ACL, 2024

    2024

  44. [52]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.