Pith. sign in

REVIEW 4 major objections 4 minor 50 references

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that more fine-grained human preference labels produce no worse reward models than binary labels, and in finite samples better ones.

desk verdict The generalization bound result is real and worth taking seriously, but the experiments do not support the load-bearing assumption about human annotators, so the paper needs revision, not rejection. read the letter →

arxiv 2411.12843 v1 pith:RETMHK6U submitted 2024-11-19 cs.LG cs.AIcs.CLstat.ML

Shang Liu , Yu Pan , Guanting Chen , Xiaocheng Li This is my paper · ORCID
classification cs.LGcs.AIcs.CLstat.ML MSC 68T0562J15
keywords rewardmodelingordinalfeedbackBradley-TerrymodelRademachercomplexityhierarchicalexpectationwisdomofthecrowddirectpolicyoptimizationknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reward models for LLMs are normally trained on binary 'response A beats B' labels, even when annotators are asked to report finer distinctions such as 'slightly better' or 'tied.' This paper proposes to keep that ordinal information and shows a precise statistical sense in which it cannot hurt and generally helps learning. The single assumption is that each rater's label is an unbiased estimate of the population-level preference probability, which the authors connect to the 'wisdom of the crowd.' Under that assumption, a more fine-grained feedback system has no larger Rademacher complexity—a standard measure of worst-case overfitting—than a coarser one, giving a tighter generalization bound. Experiments with two LLMs confirm that 5-level and 3-level labels outperform binary labels on both in-distribution accuracy and the RewardBench out-of-distribution benchmark.

What carries the argument

The central object is the hierarchical expectation coupling: a pair of random variables $(W, W')$ such that $W = \mathbb{E}[W'|W]$ almost surely, so the coarser feedback is the conditional expectation of the finer one. This ordering is what makes the Rademacher comparison work: for any loss that is affine in the feedback variable (cross-entropy and the generalized hinge loss both satisfy this), the function $\sum_i \varepsilon_i \ell(\cdot, h(x_i,y_{i,1},y_{i,2}))$ is convex, so conditioning the finer feedback on the coarser one and applying Jensen's inequality only decreases the expected supremum. Theorem 4.9 is the resulting inequality; the coupling is the mechanism that carries the argument.

What would settle it

Collect a set of preference pairs with many independent human ordinal labels per pair, and independently estimate the population preference probability by averaging a large number of binary votes per pair. Then test the calibration identity $\mathbb{E}[Z|x,y_1,y_2] = z_{\mathrm{oracle}}$ for each label level; if the conditional expectations deviate systematically, Assumption 3.1 fails and the theorem's benefit does not transfer to human-labeled reward models.

Watch

Extended reading notes

Core claim

The central claim is Theorem 4.9: if feedback system $Z$ is a hierarchical expectation of $Z'$ (the coarser system is the conditional expectation of the finer one), then $\mathrm{Rad}_{Z,n}(\ell\circ H) \leq \mathrm{Rad}_{Z',n}(\ell\circ H)$ for any hypothesis class $H$ and any loss affine in the feedback. Since every ordinal feedback system satisfying Assumption 3.1 sits between the oracle feedback $z_{\mathrm{oracle}}$ and binary $\{0,1\}$, Corollary 4.10 orders Rademacher complexity as $\mathrm{Rad}_{Z_{\mathrm{oracle}},n} \leq \mathrm{Rad}_{Z,n} \leq \mathrm{Rad}_{Z_{\mathrm{binary}},n}$. The authors therefore claim that a more fine-grained feedback structure leads to better reward learning for both in-distribution and out-of-distribution performance. The same theoretical result is extended to hinge loss and to direct policy optimization, and is applied to knowledge distillation, where soft teacher labels are shown to be a hierarchical expectation of hard labels, reducing variance. Empirically, using an oracle teacher to generate labels on Skywork-Reward-Preference-80K, the 5-level feedback system improves ID accuracy and RewardBench OOD accuracy over 3-level and binary feedback for Llama-3.2-1B and Gemma-2-2B.

Load-bearing premise

The load-bearing premise is that the expectation of the human ordinal label equals the true population preference probability for every prompt-response pair (Assumption 3.1); if annotators are not calibrated to that probability, the stated statistical benefits do not apply to real reward-model learning.

Editorial extensions

If this is right

  • Any ordinal feedback system satisfying Assumption 3.1, including 3-level and 5-level systems, has Rademacher complexity no larger than binary feedback, so its generalization bound is no worse; the oracle feedback is the best possible.
  • The result extends to direct policy optimization, so DPO training can use 'tied' and 'slightly better' labels directly rather than discarding them.
  • Because population loss is identical for all feedback systems under the assumption, the benefit is purely finite-sample; the experiments show it as higher ID and OOD accuracy.
  • Mixing a moderate fraction of tied labels (25–75 percent) improves reward learning over all-binary data, while 100 percent tied labels cause reward collapse, confirming that the proposed objective uses tied samples productively.
  • Annotation guidelines should attach numerical probability meanings to qualitative labels, such as 'slightly better' meaning 75 percent of the population prefers this response, making Assumption 3.1 actionable for human annotators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same hierarchical-expectation argument suggests a direct test for knowledge distillation—distill from teachers of varying accuracy and check that the Rademacher reduction grows as teacher soft labels become more concentrated, which the paper does not run.
  • Editorial inference: A practical extension would measure how many binary labels are needed to match the generalization of a given number of 5-level labels; the Rademacher gap implies a sample-complexity advantage, but the paper does not quantify the constant.
  • Editorial inference: If real annotators are not marginally unbiased, the framework suggests a calibration step that estimates and corrects per-label biases, which would restore the ordering without changing the loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper studies reward modeling under ordinal feedback, generalizing the binary Bradley-Terry setup to arbitrary finite label sets Z ⊂ [0,1]. The central assumption (Assumption 3.1) is that the ordinal label Z satisfies E[Z|(x,y1,y2)] = z_oracle(x,y1,y2), where z_oracle is the population preference probability. Under this assumption, the paper proves that the cross-entropy objective (2) yields the same population loss as the oracle (Proposition 4.3), and that a coarser feedback system has larger Rademacher complexity than a finer one when the loss is affine in the feedback (Theorem 4.9). This leads to Corollary 4.10, which places any ordinal feedback system between the oracle and binary systems in terms of Rademacher complexity. The paper extends the analysis to hinge loss and DPO, and draws connections to knowledge distillation. Experiments use a teacher reward model (Skywork-Reward-Gemma-2-27B) to generate oracle scores, then sample synthetic ordinal labels for binary/3-level/5-level systems, showing improved in-distribution and out-of-distribution accuracy for finer granularity, and benefits of including tied samples.

Significance. The theoretical result is a rigorous and valuable contribution: Theorem 4.9 gives a clean, general sufficient condition under which finer-grained feedback reduces Rademacher complexity, and the affinity condition covers the standard cross-entropy and hinge losses. The hierarchical-expectation coupling and the connection to soft labels are interesting and likely to be useful beyond reward modeling. However, the practical significance is currently limited: the key assumption is not validated with human annotators, and the experiments are synthetic, using a single teacher model both to generate labels and to define the evaluation oracle. The paper ships a machine-checkable proof strategy for the main theorem and a clear assumption-led framework; these are strengths. The empirical claims outrun the evidence, so the paper would benefit from either human-annotation experiments or a substantial tempering of the practical conclusions.

major comments (4)
  1. [Section 5.1 / Assumption 3.1] The empirical evaluation does not test Assumption 3.1 for human annotators. The four feedback systems are generated by sampling from z_oracle, which is defined by the Skywork teacher model (Section 5.1), so Assumption 3.1 holds by construction. Consequently, the experiments validate the theory only in the synthetic setting where the assumption is guaranteed, and provide no evidence that human ordinal labels (e.g., via the guideline in Section 3.1) satisfy E[Z|(x,y1,y2)] = z_oracle. The central practical claim that 'fine-grained feedback leads to better reward learning' (Section 5.2) is therefore not established for real human feedback.
  2. [Section 5.1, Table 1] The in-distribution evaluation is circular: the ID ground truth is defined by the same teacher model that generated the training labels, so the observed ID performance differences may reflect improved regression to the teacher's own scores rather than a general property of ordinal feedback. The OOD results on RewardBench are less circular, but the training labels are still teacher-generated, so they cannot support the 'wisdom of the crowd' claim for human annotators.
  3. [Section 4.1, Corollary 4.7, Section 5.2] Theorem 4.9 requires Z to be a hierarchical expectation of Z'. Corollary 4.7 establishes this only when Z is the oracle or Z' is binary. The paper does not prove that a 5-level system satisfying Assumption 3.1 is a hierarchical expectation of a 3-level system, yet Section 5.2 claims and experimentally demonstrates the ordering Oracle ≥ 5-level ≥ 3-level ≥ Binary. Thus the middle part of this ordering is not justified by the stated theory; the experiments go beyond the proven result. The authors should either prove the general ordering or explicitly limit the theoretical claim to comparisons with binary feedback.
  4. [Section 3.1] The annotation guideline is presented as a way to make human labels satisfy Assumption 3.1, but the assumption requires the conditional expectation of the label, given the specific (x,y1,y2), to exactly equal the oracle probability. Human annotators, even with the suggested guideline, are unlikely to be perfectly calibrated for every prompt-response pair. The paper provides no calibration study or analysis of how deviations from this assumption affect the finite-sample guarantees. Since Assumption 3.1 is the only load-bearing premise for Theorem 4.9, this is a significant gap.
minor comments (4)
  1. [Section 3.1 / Section 5.1] The 5-level example in Section 3.1 uses Z = {0, 0.25, 0.5, 0.75, 1}, while the experiments in Section 5.1 use Z = {0, 0.2, 0.5, 0.8, 1}; this inconsistency is confusing and should be reconciled with a single convention.
  2. [Section 4.3] The notation 'y_T ∼ \bar{y}_T' is used without a precise definition of the conditional distribution; please clarify how sampling from the teacher's output is defined.
  3. [Appendix C.1] The temperature T = 20/3 is chosen to match the distribution of UltraFeedback and HelpSteer2, but there is no sensitivity analysis for T. Since T scales the oracle scores and therefore affects all downstream label generation, a discussion of its role as a free parameter would increase reproducibility.
  4. [Section 6] The related-work section does not mention HelpSteer2, which provides human ordinal preference labels and is directly relevant to the proposed framework; citing it in the related work would better position the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in the derivation: Theorem 4.9 is a conditional mathematical proof from Assumption 3.1; the synthetic-oracle experiments are a validity limitation, not a circular step.

full rationale

The paper's central chain is not circular. Assumption 3.1 states E[Z|(x,y1,y2)] = z_oracle(x,y1,y2), and the paper explicitly identifies this as the one assumption made. Theorem 3.2 proves existence and characterizes all distributions satisfying this assumption; the proof is self-contained and does not assume the conclusion. Hierarchical expectation (Definition 4.5) is a defined coupling condition, and Theorem 4.9 is proved from that condition plus the affinity property, Jensen's inequality, and convexity of the pointwise supremum of affine functions (Lemmas B.1-B.3). The proof is a direct inequality, not a restatement of the inputs; the DPO extension and the soft-labeling corollary reuse the same argument. No load-bearing result is imported from the authors' own prior work: the key characterization is proved in the paper, and the external citations are to standard or non-overlapping works. The main caveat is experimental: the oracle scores are generated by a reward model trained on the same base dataset, and the feedback labels are sampled by the Theorem 3.2 construction so that Assumption 3.1 holds by construction. That means the experiments do not independently validate Assumption 3.1 for human annotators, and the in-distribution evaluation is not fully independent of the oracle model. This is an external-validity limitation, not a circular derivation: the theoretical ordering of Rademacher complexities stands on its own proof, and the OOD evaluation on RewardBench provides some independent signal. No circular step is exhibited in the derivation itself.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or biological entities are introduced. 'Hierarchical expectation' and 'ordinal feedback' are mathematical and modeling constructs, not independent entities. The main axiomatic load is Assumption 3.1, which is plausible but untested for human annotators.

free parameters (2)
  • Temperature T for oracle score scaling = 20/3 ≈ 6.667
    Chosen by hand in Appendix C.1 to make the sigmoid-transformed score differences of the teacher model peak in the 0.6-0.7 'slight agreement' range and to yield enough tied samples for the tied-ratio experiments.
  • Hinge loss margin C (Appendix A) = 2
    Auxiliary experiment; selected by grid search over {0.5, 1, 2, 4}.
assumptions (3)
  • domain assumption Assumption 3.1: E[Z|(x,y1,y2)] = z_oracle(x,y1,y2) for all (x,y1,y2).
    The only assumption of the ordinal feedback model. It asserts human annotators are unbiased estimators of the population preference probability. It is not empirically validated in the paper; the sociological analogy and annotation guidance are suggestive, not evidence.
  • domain assumption Bradley-Terry model: P(y1≻y2|x) ≈ σ(rθ(x,y1) - rθ(x,y2)).
    Used to define the learning objective (2) and the DPO extension. The framework is claimed to extend beyond BT, but the numerical experiments use BT-based cross-entropy loss.
  • standard math i.i.d. data and bounded hypothesis class for Rademacher complexity bounds.
    Standard assumptions for the generalization bound in Appendix B.9; stated in Definition 4.8.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward Modeling with Ordinal Feedback: Wisdom of the Crowd." pith.science (2026). https://pith.science/paper/RETMHK6U

@misc{pith2026241112843,
  author       = {Pith},
  title        = {Pith review of: Reward Modeling with Ordinal Feedback: Wisdom of the Crowd},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RETMHK6U}},
  note         = {Machine review of arXiv:2411.12843}
}
read the original abstract

Learning a reward model (RM) from human preferences has been an important component in aligning large language models (LLMs). The canonical setup of learning RMs from pairwise preference data is rooted in the classic Bradley-Terry (BT) model that accepts binary feedback, i.e., the label being either Response 1 is better than Response 2, or the opposite. Such a setup inevitably discards potentially useful samples (such as "tied" between the two responses) and loses more fine-grained information (such as "slightly better"). In this paper, we propose a framework for learning RMs under ordinal feedback which generalizes the case of binary preference feedback to any arbitrary granularity. Specifically, we first identify a marginal unbiasedness condition, which generalizes the assumption of the BT model in the existing binary feedback setting. The condition validates itself via the sociological concept of the wisdom of the crowd. Under the condition, we develop a natural probability model for pairwise preference data under ordinal feedback and analyze its properties. We prove the statistical benefits of ordinal feedback in terms of reducing the Rademacher complexity compared to the case of binary feedback. The proposed learning objective and the theory also extend to hinge loss and direct policy optimization (DPO). In particular, the theoretical analysis may be of independent interest when applying to a seemingly unrelated problem of knowledge distillation to interpret the bias-variance trade-off therein. The framework also sheds light on writing guidance for human annotators. Our numerical experiments validate that fine-grained feedback leads to better reward learning for both in-distribution and out-of-distribution settings. Further experiments show that incorporating a certain proportion of samples with tied preference boosts RM learning.

Figures

Figures reproduced from arXiv: 2411.12843 by the authors.

Figure 1
Figure 1. Wisdom of the crowd. Left: Each individual guess can be far off the target for an ox-weight-guessing social experiment, but the average tends to be very accurate. Each human annotator has not access to the population oracle preference model zoracle, but their annotation constitutes an unbiased realization of zoracle. 3.1 Implications on annotation guidance In practice, annotators label in the set Ztext (e.g. {better… view at source ↗
Figure 2
Figure 2. The evaluation dynamics of llama and gemma models for different ordinal feedback labels. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. The evaluation dynamics of llama and gemma models for different tied data ratios. The 100%- [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The evaluation dynamics of llama models for different ordinal feedback labels under generalized [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Distributions of preference strengths in the two datasets. For the UltraFeedback dataset, we [PITH_FULL_IMAGE:figures/full_fig_p030_5.png]
Figure 6
Figure 6. Figure 6: Distribution of oracle labels before and after scaling with [PITH_FULL_IMAGE:figures/full_fig_p030_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 16 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sent...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in "" FUNCTION format.date year ...

  3. [3]

    Akrour, Riad, Marc Schoenauer, Michele Sebag. 2011. Preference-based policy learning. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part I 11\/ . Springer, 12--27

    work page 2011

  4. [4]

    Amini, Afra, Tim Vieira, Ryan Cotterell. 2024. Direct preference optimization with an offset. arXiv preprint arXiv:2402.10571\/

    arXiv 2024

  5. [5]

    Askell, Amanda, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861\/

    arXiv 2021

  6. [6]

    Azar, Mohammad Gheshlaghi, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics\/ . PMLR, 4447--4455

    work page 2024

  7. [7]

    Ba, Jimmy, Rich Caruana. 2014. Do deep nets really need to be deep? Advances in neural information processing systems\/ 27

    work page 2014

  8. [8]

    Bai, Yuntao, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862\/

    arXiv 2022

Show all 50 references
  1. [9]

    Bradley, Ralph Allan, Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika\/ 39 (3/4) 324--345

    1952

  2. [10]

    Burges, Christopher, Robert Ragno, Quoc Le. 2006. Learning to rank with nonsmooth cost functions. Advances in neural information processing systems\/ 19

    2006

  3. [11]

    Chakraborty, Souradip, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, Mengdi Wang. 2024. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925\/

    2024 arXiv

  4. [12]

    Chen, Jinghong, Guangyu Yang, Weizhe Lin, Jingbiao Mei, Bill Byrne. 2024. On extending direct preference optimization to accommodate ties. arXiv preprint arXiv:2409.17431\/

    2024

  5. [13]

    u rnkranz, Eyke H \

    Cheng, Weiwei, Johannes F \"u rnkranz, Eyke H \"u llermeier, Sang-Hyeun Park. 2011. Preference-based policy iteration: Leveraging preference learning for reinforcement learning. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens,...

    2011

  6. [14]

    Choshen, Leshem, Lior Fox, Zohar Aizenbud, Omri Abend. 2019. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01752\/

    2019 arXiv

  7. [15]

    Christiano, Paul F, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems\/ 30

    2017

  8. [16]

    Cui, Ganqu, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377\/

    2023 arXiv

  9. [17]

    Davidson, Roger R. 1970. On extending the bradley-terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association\/ 65 (329) 317--328

    1970

  10. [18]

    Dong, Hanze, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang. 2023 a . Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767\/

    2023 arXiv

  11. [19]

    Dong, Zibin, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, Zhipeng Hu. 2023 b . Aligndiff: Aligning diverse human preferences via behavior-customisable diffusion model. arXiv preprint arXiv:2310.02054\/

    2023 arXiv

  12. [21]

    Ethayarajh, Kawin, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306\/

    2024 arXiv

  13. [22]

    Galton, Francis. 1907. Vox populi. Nature\/ 75 (1949) 450--451. ://doi.org/10.1038/075450a0

    1949 doi

  14. [23]

    Hinton, Geoffrey. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531\/

    2015 arXiv

  15. [24]

    Ji, Jiaming, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852\/

    2023 arXiv

  16. [25]

    Kaufmann, Timo, Paul Weng, Viktor Bengs, Eyke H \"u llermeier. 2023. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925\/

    2023

  17. [26]

    Smith, Hannaneh Hajishirzi

    Lambert, Nathan, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi. 2024. Rewardbench: Evaluating reward models for language modeling

    2024

  18. [27]

    Liu, Chris Yuhao, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, Yahui Zhou. 2024 a . Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451\/

    2024 arXiv

  19. [28]

    Liu, Jinsong, Dongdong Ge, Ruihao Zhu. 2024 b . Reward learning from preference with ties. arXiv preprint arXiv:2410.05328\/

    2024 arXiv

  20. [29]

    Liu, Tianqi, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. 2024 c . Lipo: Listwise preference optimization through learning-to-rank. arXiv preprint arXiv:2402.01878\/

    2024 arXiv

  21. [30]

    Liu, Tianqi, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, Jialu Liu. 2023. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657\/

    2023 arXiv

  22. [31]

    Llama Team, AI @ Meta. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783\/

    2024 arXiv

  23. [32]

    M \"u ller, Rafael, Simon Kornblith, Geoffrey E Hinton. 2019. When does label smoothing help? Advances in neural information processing systems\/ 32

    2019

  24. [33]

    Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems\/...

    2022

  25. [34]

    Phuong, Mary, Christoph Lampert. 2019. Towards understanding knowledge distillation. International conference on machine learning\/ . PMLR, 5142--5151

    2019

  26. [35]

    Rafailov, Rafael, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems\/ 36

    2024

  27. [36]

    Rao, PV, Lawrence L Kupper. 1967. Ties in paired-comparison experiments: A generalization of the bradley-terry model. Journal of the American Statistical Association\/ 62 (317) 194--204

    1967

  28. [37]

    Sch \"o lkopf, Bernhard, Koji Tsuda, Jean-Philippe Vert. 2004. Kernel methods in computational biology\/ . MIT press

    2004

  29. [38]

    Song, Feifan, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang. 2024. Preference ranking optimization for human alignment. Proceedings of the AAAI Conference on Artificial Intelligence\/ , vol. 38. 18990--18998

    2024

  30. [39]

    Team, Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118\/

    2024 arXiv

  31. [40]

    Wang, Chaoqi, Yibo Jiang, Chenghao Yang, Han Liu, Yuxin Chen. 2023. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240\/

    2023 arXiv

  32. [41]

    Wang, Haoxiang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang. 2024 a . Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.18571\/

    2024 arXiv

  33. [42]

    Wang, Haoxiang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang. 2024 b . Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845\/

    2024 arXiv

  34. [43]

    Wang, Zhilin, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev. 2024 c . Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673\/

    2024 arXiv

  35. [44]

    Xu, Haoran, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417\/

    2024 arXiv

  36. [45]

    Yuan, Hongyi, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang. 2024. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems\/ 36

    2024

  37. [46]

    Yuan, Li, Francis EH Tay, Guilin Li, Tao Wang, Jiashi Feng. 2020. Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition\/ . 3903--3911

    2020

  38. [47]

    Zeng, Dun, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, Zenglin Xu. 2024 a . On diversified preferences of large language model alignment. Findings of the Association for Computational Linguistics: EMNLP 2024\/ . 9194--9210

    2024

  39. [48]

    Zeng, Yongcheng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, Jun Wang. 2024 b . Token-level direct preference optimization. arXiv preprint arXiv:2404.11999\/

    2024 arXiv

  40. [49]

    Zhao, Yao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425\/

    2023 arXiv

  41. [50]

    Zhou, Helong, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, Qian Zhang. 2021. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.00650\/

    2021 arXiv

  42. [51]

    Ziegler, Daniel M, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593\/

    2019 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.