Pith. sign in

REVIEW 4 major objections 6 minor 80 references

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two co-trained reward models that filter each other's training batches via peer review and synchronized curriculum learning improve reward-model accuracy and downstream policy win-rates under injected preference noise.

desk verdict A genuinely new online peer-review filtering scheme for noisy preference data; the headline 40%-noise gain depends on knowing the noise rate, so treat the strongest number with caution. read the letter →

arxiv 2505.10597 v2 pith:O5Q2STG2 submitted 2025-05-15 cs.LG cs.AIcs.CL

Jiazheng Zhang , Wenqing Jing , Zizhuo Zhang , Zhiheng Xi , Shihan Dou , Rongxiang Weng , Jiahuan Li , Jingang Wang
show 4 more authors
Mingxu Chai Shibo Hong Tao Gui Qi Zhang
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords preferencesrewardnoisymodelsalignmentlearningmodelingpreference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern LLM alignment often relies on a reward model trained on human preference pairs: for a given prompt, which of two responses is better. Human labels are noisy, and when a reward model memorizes wrong labels it can send the downstream LLM in the wrong direction. This paper first measures per-sample training statistics, such as mean loss and loss variance across epochs, and finds that samples with high loss or unstable loss are often bad labels. The central method, CRM, then trains two reward models side by side. Each model scores each batch with its own reward margin; the top-scoring fraction of the batch under model A is used to train model B, and vice versa. A curriculum sorts data by reward margin, easiest to hardest, so that the two models stay roughly similar in strength. Under simulated label flipping at 0, 20, and 40 percent, CRM reports better reward accuracy and better downstream win-rates than standard reward models and robust-DPO variants. The authors also apply the same selection idea to DPO, using the DPO implicit reward to filter batches between two policy models.
Extended reading notes

Core claim

The paper claims CRM 'significantly enhances RM generalization, with up to 9.94 points improvement on RewardBench under an extreme 40% noise' (abstract and Sec. 4), and that the DPO extension improves alignment win-rates (Table 3). If correct, online peer-reviewed batch filtering, without changing the underlying Bradley-Terry loss, improves both in-domain and out-of-domain preference accuracy and downstream RLHF/DPO performance under 20 to 40 percent label noise.

Load-bearing premise

The method requires that the same reward margin used for filtering (Eq. 6-7) strictly orders data by noise content, and that the default selection ratio lambda_t = 1 - eta (Sec. 4.3) correctly estimates the noise rate. If the margin is not predictive of label correctness, or if eta is misspecified, the filter can drop clean pairs or retain noisy pairs. The motivation in Sec. 3.1 is a correlation on one dataset (HH-RLHF) with a 3B model, and the loss-separation analysis in Sec. 4.3 is retrospective, computed on training data after the filter has already shaped the model.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Collaborative Reward Modeling (CRM), an online co-training framework in which two reward models filter each other's training batches by keeping the top lambda_t fraction of preference pairs ranked by the peer model's reward margin sigma(r_w - r_l), combined with an epoch-level curriculum that sorts data by margin. The authors motivate this with a three-way categorization of preference instances (robust, ambiguous, non-robust) and report experiments on HH-RLHF, Ultrafeedback, and Skywork-Reward with synthetic label flipping at 0%, 20%, and 40%, evaluating in-domain preference accuracy, RewardBench, RMB, RM-Bench, and downstream RLHF/DPO win-rates judged by GPT-4o. The headline claim is up to a 9.94-point RewardBench improvement under 40% noise, together with a claimed seamless extension to DPO.

Significance. If the empirical claims hold, CRM is a practically attractive and loss-function-agnostic data-filtering wrapper: it leaves the Bradley-Terry objective unchanged, applies to both explicit and implicit reward training, and its peer-review mechanism is a plausible remedy for the confirmation bias that afflicts self-filtering. The paper's strengths include extensive benchmarking across three training sets and several reward benchmarks, ablations of peer review and curriculum learning, weak-to-strong collaboration experiments, and an explicit limitations section. However, the absence of error bars, an oracle-conditioned default selection ratio, and an unvalidated margin-ranking assumption currently leave the central quantitative claims uncertain; the contribution is promising but not yet established at the level the abstract claims.

major comments (4)
  1. [Sec. 4.3, Fig. 5, and Eqs. (6)-(7)] The headline 9.94-point RewardBench improvement at 40% noise (Sec. 4.2, Table 7) is obtained under the default selection ratio lambda_t = 1 - eta = 0.6, because eta is the injected noise rate. The paper states that competitive results are obtained for lambda_t in [0.7, 0.9], which excludes 0.6, and no sensitivity result is shown at lambda_t = 0.6. Since eta is known only in the synthetic-noise experiments, the default setting is oracle-conditioned, and it is unclear how lambda_t would be set for real data. Please report a full lambda_t sweep with variance at each noise level and provide either a practical estimator of the noise rate or a misspecification analysis showing that CRM is robust to imperfect eta.
  2. [Sec. 4.2, Tables 1, 7-10; Sec. C.4] No error bars, confidence intervals, or significance tests are reported, yet many of the claimed improvements are small: for example, in Table 10 at 20% noise on RMB-Harmless, CRM (65.13) is below cDPO-RM (65.88), and in Table 3 at 0% noise on TL;DR, CRM (52%) is only one point above DPO (51%). The win-rate evaluation uses only 300 prompts with GPT-4o as judge, so the reported margins are likely within judge and prompt sampling noise. In addition, the 40%-flipped Standard RM row in Table 8 (90.50, 48.46, 82.45, 63.62, 66.53) is identical to the 0%-flipped Standard RM row in Table 7, which appears to be a copy error. Please correct the table and report variability over training seeds and judge runs for all main results.
  3. [Sec. 3.1 and Eq. (5)] The central assumption that the peer margin M = sigma(r_w - r_l) orders preference pairs by noise content is not validated at the moment of selection. The Sec. 3.1 analysis is an aggregate correlation on one dataset (HH-RLHF) with one 3B model, and the loss-separation evidence in Sec. 4.3 and Fig. 6 is computed after training, on data that the filter has already shaped; it therefore cannot validate the online selection rule. Please report the precision and recall of the margin filter against known flipped labels during training (e.g., at early, middle, and late epochs), and the fraction of clean pairs dropped at each lambda_t. Without this, Eqs. (6)-(7) could be discarding clean data while retaining noisy data.
  4. [Sec. 4.3, Algorithm 1, and Table 6] The claimed seamless extension to DPO is not reproducible from the paper. Algorithm 1 is written for explicit reward models, while the DPO section states only that the framework is 'suited to existing DPO' without specifying how the two policies are initialized, how curriculum sorting is performed, or how peer review selects batches in the implicit-reward setting. Table 6 reports CRM-Model 1 and CRM-Model 2, but Table 3 reports a single CRM result; it is unclear whether Table 3 uses one of the two models and whether the two models are synchronized in the same way as the explicit-RM case. Please provide the DPO analogue of Algorithm 1 and clarify which model's outputs are reported.
minor comments (6)
  1. [Sec. 4.2, text after Table 1] The text says CRM 'consistently achieves superior performance,' but Table 10 contains counterexamples (e.g., 20% RMB-Harmless, where cDPO-RM outperforms CRM); please soften the claim to match the data.
  2. [Table 1] The Skywork-Reward ID column is a dash for every method, yet the Average column appears to be computed over the remaining five columns; please clarify what ID means for Skywork-Reward and how the average is defined.
  3. [Fig. 2] Figure 2 is a schematic without axes labels or quantitative thresholds; please state which coordinates are plotted and how the boundaries between robust, ambiguous, and non-robust preferences are set.
  4. [Algorithm 1, line 6] The curriculum sort uses M for r_{phi,psi}, which is ambiguous; please specify whether the sort key is the margin of the current model, the peer model, or an aggregate of both.
  5. [Fig. 5 and Sec. 4.3] The abbreviation 'w/ SR' in Fig. 5 is not defined in the caption or the text; the surrounding text mentions a 'Self Review' variant, so please make the notation consistent.
  6. [NeurIPS Checklist, item 7] The checklist answers 'Yes' to the statistical-significance question, but the paper does not contain error bars or significance tests; please correct the checklist answer or add the missing statistics.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-referential validation in Sec. 4.3; central claims rest on external RewardBench and win-rate evaluations, so no significant overall circularity.

  1. self definitional [Sec. 4.3, 'Resistance against Noisy Preference', Fig. 6]
    "Empirical experiments confirm that CRM demonstrates superior robustness, particularly in handling noisy preferences. We depict the loss distributions for CRM and the standard RM on the Ultrafeedback-Binaralized in Fig. 6. The divergence between clean and noisy data is more distinct in CRM compared to the standard RM, where this gap is narrower."

    CRM's peer-review filter (Eqs. 6-7) keeps only the top-λ fraction of each batch by margin M = σ(r_w - r_l), and Eq. 8 updates each reward model on exactly those selected batches. The reported loss divergence between clean and noisy data is thus produced by the same margin signal used to exclude low-margin examples; it is a by-construction artifact of the selection rule, not an independent test of noise robustness. The paper presents it as confirmation of robustness, which is self-referential. The main empirical claims, however, are evaluated on external RewardBench and GPT-4 win-rates, not on this plot, so the circularity is local rather than load-bearing.

full rationale

The central derivation chain is: characterize noisy preferences via training dynamics (Sec. 3.1), design CRM to filter by peer-model margin (Eqs. 5-8), and evaluate on held-out RewardBench and RLHF/DPO win-rates (Tables 1-3, 7-9). The headline 9.94-point improvement is an out-of-domain RewardBench comparison (HH-RLHF 40% noise, Table 7), i.e., an external benchmark not constructed from the filter. Win-rates are judged by GPT-4 against SFT targets, and the method is compared against standard RM, cDPO, rDPO, and ROPO. The self-review ablation (Fig. 5, 'w/ SR') shows that filtering by one's own margin is worse than peer review, which weakens any suggestion that the whole gain is a trivial self-referential loop. No load-bearing self-citation chain or imported uniqueness theorem is used; the cited same-group works [9,34] are background evidence on noise rates and PPO implementation, not the justification for CRM's effectiveness. The main caveats are experimental rather than circular: the default λ_t = 1−η requires knowledge of η, and the 40%-noise headline uses λ_t = 0.6, which lies outside the stated robust interval [0.7,0.9]; these affect transferability and robustness, not by-construction equivalence. One local circular-validation step exists in Sec. 4.3: the loss-separation figure is generated by the same margin-based selection rule it is used to confirm. Overall, the paper's primary predictions have independent content, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method does not introduce new physical or mathematical objects beyond the two co-trained reward models and the selection ratio lambda_t. The main free parameter is lambda_t = 1 - eta, which requires knowing the noise rate; the paper chooses it to match the injected noise. The peer-review procedure assumes a degree of independence between the two models, and the whole pipeline assumes margin-based selection is consistent with label correctness.

free parameters (3)
  • selection ratio lambda_t = default 1 - eta (0.6 at 40% noise, 0.8 at 20% noise)
    Controls how many samples are kept per batch. The paper says lambda_t in [0.7, 0.9] gives competitive results and sets lambda_t = 1 - eta. In practice the true noise rate eta is rarely known, so this is effectively a tuned knob.
  • assumed noise rate eta = 0, 0.2, 0.4 in experiments
    Injected noise levels. At 0% injected noise, lambda_t = 1 means no filtering, yet the paper claims HH-RLHF contains 20-30% real noise. The 'extreme 40% noise' headline result uses lambda_t = 0.6, which is chosen to match the injected noise scenario.
  • curriculum sorting metric = reward margin of both models (Eq. 5)
    Sorts data at epoch level. No analysis compares this difficulty measure to alternatives such as training loss or predictive confidence.
assumptions (3)
  • domain assumption Reward margin sigma(r(y_w) - r(y_l)) is a valid monotone score of label correctness and can be used safely as a per-batch filter (Eq. 5-7).
    The method uses the same margin both to rank the curriculum and to select filtered batches. If noisy examples are not systematically lower-margin at the batch level early in training, the filter can amplify errors. The motivation in Sec. 3.1 shows correlations for epoch-average statistics, not for batch-level margins.
  • domain assumption Two RMs trained on overlapping filtered data are sufficiently decorrelated to provide independent peer-review signals.
    Eq. 8 updates r_phi on data selected by r_psi and vice versa, but both models are initialized identically and see the same curriculum. The paper does not quantify the correlation between the two models' errors. The self-review ablation suggests independence matters, but it is not measured.
  • domain assumption Standard Bradley-Terry noise model: flips are i.i.d. with known rate eta and the true reward model is recoverable from clean pairs.
    The experiments inject symmetric random flips, which is the standard synthetic setup. Real annotator noise can be structured (for example, length bias or style bias) and may not be removable by a margin threshold. The paper does not evaluate structured noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment." pith.science (2026). https://pith.science/paper/O5Q2STG2

@misc{pith2026250510597,
  author       = {Pith},
  title        = {Pith review of: Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5Q2STG2}},
  note         = {Machine review of arXiv:2505.10597}
}
read the original abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human values. However, noisy preferences in human feedback can lead to reward misgeneralization - a phenomenon where reward models learn spurious correlations or overfit to noisy preferences, which poses important challenges to the generalization of RMs. This paper systematically analyzes the characteristics of preference pairs and aims to identify how noisy preferences differ from human-aligned preferences in reward modeling. Our analysis reveals that noisy preferences are difficult for RMs to fit, as they cause sharp training fluctuations and irregular gradient updates. These distinctive dynamics suggest the feasibility of identifying and excluding such noisy preferences. Empirical studies demonstrate that policy LLM optimized with a reward model trained on the full preference dataset, which includes substantial noise, performs worse than the one trained on a subset of exclusively high quality preferences. To address this challenge, we propose an online Collaborative Reward Modeling (CRM) framework to achieve robust preference learning through peer review and curriculum learning. In particular, CRM maintains two RMs that collaboratively filter potential noisy preferences by peer-reviewing each other's data selections. Curriculum learning synchronizes the capabilities of two models, mitigating excessive disparities to promote the utility of peer review. Extensive experiments demonstrate that CRM significantly enhances RM generalization, with up to 9.94 points improvement on RewardBench under an extreme 40\% noise. Moreover, CRM can seamlessly extend to implicit-reward alignment methods, offering a robust and versatile alignment strategy.

Figures

Figures reproduced from arXiv: 2505.10597 by the authors.

Figure 1
Figure 1. Simulated RL experiemnts with different proxy RMs. Policy optimized by clean RM shows an increasring gold score while noisy RM degrade the RL optimization. Recent studies [13; 14; 15] demonstrate that 20%- 40% of open-source preference data is corrupted by noise, with many annotated pairs contradicting the actual human preference. Representative exam￾ples selected from HH-RLHF are provided in Ap￾pendix A.1 [PITH_FU… view at source ↗
Figure 2
Figure 2. Characterizing robustness of preference instances. The training instances can be roughly [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Win-rate comparison on Anthropic￾Helpful and Harmless between policies optimized by RM trained on different group of preferences. Analysis [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The CRM framework: (1) Curriculum Learning organizes the preference learning process as a progression from easy to complex. (2) Peer Review enables the two RMs to collaborate by assessing each other’s data selections, filtering out potential noisy preferences within ea…
Figure 5
Figure 5. Figure 5: Left: Ablation study of different variants on HH-RLHF with varing noises. Middle: Perfor￾mance of collaborative reward modeling with different-sized RMs. Right: Impact of λt on preference accuracy across varying noisy-levels. DPO is significantly compromised by noisy p…
Figure 6
Figure 6. Figure 6: Left: Loss distribution of clean and noisy preferences from standard RM. Middle: Loss distribution from CRM. Right: Visualization of preference instances using the chosen and rejected rewards computed by CRM as coordinates. selection of λt, and competitive results can …
Figure 7
Figure 7. Figure 7: Win-rate comparison on Anthropic-Helpful between RLHF models optimized by Clean and Noisy RM. To examine the effect of RM’s generalization on the RL stage, we design a simulation experiment to em￾pirically evaluate the policies optimized by different proxy RMs. Specifi…
Figure 8
Figure 8. Figure 8: Simulated RL experiemnts with different proxy RMs. Blue and Red lines represent the [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Left: Divergence of reward modeling on robust, non-robust and ambiguous preferences, respectively. Right: Test preference accuracy vs. training steps on Athropic Helpful. A.3 Effect of Preference Instances on Reward Modeling In this section, we supplement related exper…
Figure 10
Figure 10. Figure 10: Loss distribution of clean and noisy preferences, where training set of Skywork-Reward [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Loss distribution of clean and noisy preferences, where training set of Skywork-Reward [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 48 canonical work pages

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  2. [2]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedbac...

    2022

  3. [3]

    A comprehensive survey of reward models: Taxonomy, applications, challenges, and future

    Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprint arXiv:2504.12328, 2025

    arXiv 2025

  4. [4]

    A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization

    Wenyuan Xu, Xiaochen Zuo, Chao Xin, Yu Yue, Lin Yan, and Yonghui Wu. A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization. arXiv preprint arXiv:2504.04950, 2025

    arXiv 2025

  5. [5]

    Better process supervision with bi-directional rewarding signals

    Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Rui Zheng, Nijun Li, Tao Gui, Yun Li, et al. Better process supervision with bi-directional rewarding signals. arXiv preprint arXiv:2503.04618, 2025

    arXiv 2025

  6. [6]

    Reward Function Design in Reinforcement Learning, pages 25–33

    Jonas Eschmann. Reward Function Design in Reinforcement Learning, pages 25–33. Springer International Publishing, Cham, 2021

    work page 2021

  7. [7]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022

    arXiv 2022

  8. [8]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    2024

Show all 80 references
  1. [9]

    Secrets of rlhf in large language models part ii: Reward modeling

    Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024

    2024 arXiv

  2. [10]

    Skywork-reward: Bag of tricks for reward modeling in llms, October 2024

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms, October 2024

    2024

  3. [11]

    RMB: Comprehensively benchmarking reward models in LLM alignment

    Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. RMB: Comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Confere...

    2025

  4. [12]

    Helpsteer 2: Open-source dataset for training top-performing reward models

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In The Thirty-eight Conference on Neural Information Proc...

  5. [13]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing ...

    2023

  6. [14]

    Impact of preference noise on the alignment performance of generative language models

    Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. In First Conference on Language Modeling, 2024

    2024

  7. [15]

    Improving reinforcement learning from human feedback using contrastive rewards, March 2024

    Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Improving reinforcement learning from human feedback using contrastive rewards, March 2024

    2024

  8. [16]

    Goal misgeneralization in deep reinforcement learning

    Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004–12019. PMLR, 2022

    2022

  9. [17]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, pages 10835–10866. PMLR, July 2023

    2023

  10. [18]

    Improving discriminative capability of reward models in rlhf using contrastive learning

    Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, et al. Improving discriminative capability of reward models in rlhf using contrastive learning. In Proceedings of the 2024 Conference on Empirical Methods i...

    2024

  11. [19]

    Reward generalization in rlhf: A topological perspective

    Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, and Yaodong Yang. Reward generalization in rlhf: A topological perspective. arXiv preprint arXiv:2402.10184, 2024

    2024 arXiv

  12. [20]

    A note on dpo with noisy preferences & relationship to ipo, 2023

    Eric Mitchell. A note on dpo with noisy preferences & relationship to ipo, 2023

    2023

  13. [21]

    Provably robust dpo: aligning language models with noisy feedback

    Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: aligning language models with noisy feedback. In Proceedings of the 41st International Conference on Machine Learning, pages 42258–42274, 2024

    2024

  14. [22]

    Towards robust alignment of language models: Distributionally robustifying direct preference optimization

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:2407.07880, 2024

    2024 arXiv

  15. [23]

    Ropo: Robust preference optimization for large language models

    Xize Liang, Chao Chen, Shuang Qiu, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Ropo: Robust preference optimization for large language models. arXiv preprint arXiv:2404.04102, 2024

    2024 arXiv

  16. [24]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  17. [25]

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  18. [26]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741, 2023

    2023

  19. [27]

    Confirma- tion bias in human reinforcement learning: Evidence from counterfactual feedback processing

    Stefano Palminteri, Germain Lefebvre, Emma J Kilford, and Sarah-Jayne Blakemore. Confirma- tion bias in human reinforcement learning: Evidence from counterfactual feedback processing. PLoS computational biology, 13(8):e1005684, 2017

    2017

  20. [28]

    Pseudo- labeling and confirmation bias in deep semi-supervised learning

    Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2020. 11

    2020

  21. [29]

    Zephyr: Direct distillation of lm alignment

    Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro V on Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. In First Conference on Language Modeling, 2023

    2023

  22. [30]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024

    2024

  23. [31]

    Rlhf workflow: From reward modeling to online rlhf, 2024

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024

    2024

  24. [32]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023...

    2023

  25. [33]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  26. [34]

    Secrets of rlhf in large language models part i: Ppo

    Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023

    2023 arXiv

  27. [35]

    RM-bench: Benchmarking reward models of language models with subtlety and style

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  28. [36]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  29. [37]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In International Conference on ...

    2024

  30. [38]

    Weak-to-strong prefer- ence optimization: Stealing reward from weak aligned model

    Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong prefer- ence optimization: Stealing reward from weak aligned model. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  31. [39]

    Adversarial training of reward models

    Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, and Tuo Zhao. Adversarial training of reward models. arXiv preprint arXiv:2504.06141, 2025

    2025 arXiv

  32. [40]

    Defining and charac- terizing reward gaming

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and charac- terizing reward gaming. In Advances in Neural Information Processing Systems, volume 35, pages 9460–9471, 2022

    2022

  33. [41]

    Reward shaping to mitigate reward hacking in rlhf

    Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint arXiv:2502.18770, 2025

    2025 arXiv

  34. [42]

    Odin: disentangled reward mitigates hacking in rlhf

    Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: disentangled reward mitigates hacking in rlhf. In Proceedings of the 41st International Conference on Machine Learning, pages 7935–7952, 2024

    2024

  35. [43]

    Taming overconfidence in llms: Reward calibration in rlhf

    Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in llms: Reward calibration in rlhf. arXiv preprint arXiv:2410.09724, 2024

    2024 arXiv

  36. [44]

    Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation, July 2024

    Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation, July 2024. 12

    2024

  37. [45]

    The energy loss phenomenon in rlhf: A new perspective on mitigating reward hacking, February 2025

    Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. The energy loss phenomenon in rlhf: A new perspective on mitigating reward hacking, February 2025

    2025

  38. [46]

    Reward model ensembles help mitigate overoptimizatio

    Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimizatio. 2024

    2024

  39. [47]

    Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. 2024

    2024

  40. [48]

    Reward-robust rlhf in llms, October 2024

    Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in llms, October 2024

    2024

  41. [49]

    Uncertainty-penalized reinforcement learning from human feedback with diverse reward lora ensembles, December 2023

    Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-penalized reinforcement learning from human feedback with diverse reward lora ensembles, December 2023

    2023

  42. [50]

    Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable prefer- ences via multi-objective reward modeling and mixture-of-experts. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, pages 10582–10592, 2024

    2024

  43. [51]

    Rrm: Robust reward model training mitigates reward hacking

    Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156, 2024

    2024 arXiv

  44. [52]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, page...

    2024

  45. [53]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  46. [54]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, 2024

    2024

  47. [55]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  48. [56]

    Is dpo superior to ppo for llm alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. In International Conference on Machine Learning, pages 54983–54998. PMLR, 2024

    2024

  49. [57]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In Advances in Neural Information Processing S...

    2024

  50. [58]

    Direct language model alignment from online ai feedback

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024

    2024 arXiv

  51. [59]

    Dpo-shift: Shifting the distribution of direct preference optimization

    Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, and Xiao Li. Dpo-shift: Shifting the distribution of direct preference optimization. arXiv preprint arXiv:2502.07599, 2025

    2025 arXiv

  52. [60]

    Understanding generalization of preference optimization under noisy feedback

    Shawn Im and Yixuan Li. Understanding generalization of preference optimization under noisy feedback. 2025

    2025

  53. [61]

    Robust reinforcement learning from corrupted human feedback

    Alexander Bukharin, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, and Tuo Zhao. Robust reinforcement learning from corrupted human feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 13

    2024

  54. [62]

    Theory of games and economic behavior, 60th- anniversary, 2007

    John von Neumann and Oskar Morgenstern. Theory of games and economic behavior, 60th- anniversary, 2007

    2007

  55. [63]

    Multiagent systems: Algorithmic, game-theoretic, and logical foundations

    Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008

    2008

  56. [64]

    [Yes] " is generally preferable to

    Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, 2017. 14 A Additional Details for Motivation A.1 Case Study Robust Preference ...

    2017

  57. [65]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We state our main claims in the abstract and introduction. Guidelines: • The answer NA means that the abstract and int...

  58. [66]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Please see Sec. 6. 27 Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, bu...

  59. [67]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: This work does not include theoretical proofs. Guidelines: • The answer NA means that the pap...

  60. [68]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  61. [69]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will release all the code...

  62. [70]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Please see Appendix B.2. Guid...

  63. [71]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: This paper correctly defined or other appropriate ...

  64. [72]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: All computational are c...

  65. [73]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: This work comply with the code of Ethics. Guidelines: • The answer NA means tha...

  66. [74]

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Please see Sec. D. Guidelines: • The answer NA means that there is no societal impact of the work perform...

  67. [75]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: NA Guidelines: ...

  68. [76]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We ensure the ap...

  69. [77]

    • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [No] Justification: NA Guidelines: • The answer NA means that the paper does not release new assets. • Researchers should communicate the...

  70. [78]

    32 Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  71. [79]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  72. [80]

    Answer: [NA] Justification: LLM is used only for editing, or formatting and does not impact the core methodology, scientific rigorousness

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.