{"id":"79696ed2-e8c1-4018-871e-597496ce3295","arxiv_id":"2411.12843","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper generalizes Bradley-Terry reward modeling to ordinal feedback labels and proves that, under a marginal unbiasedness assumption, finer-grained labels reduce Rademacher complexity and can improve reward learning.","lead":"This paper proposes a way to train reward models from ordinal human feedback, such as \"slightly better\" or \"tied\", instead of only binary comparisons. It shows theoretically and experimentally that finer-grained labels can reduce statistical complexity and improve reward model accuracy, if the labels are unbiased estimates of the true preference probability.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 3.1 (E[Z|(x,y1,y2)] = z_oracle) is the load-bearing premise: the theory is sound under it, but it is untested for human annotators, and the experiments satisfy it by construction from a teacher model that also defines the evaluation oracle, leaving the real-world benefit unsupported.","rationale":"The reader's weakest-assumption analysis identifies precisely this point: Assumption 3.1 is the single condition on which the theoretical results rest, and it is not tested with human annotations. My independent check of the proof of Theorem 4.9 confirms the mathematics: the affinity condition makes the loss affine in the feedback variable, the supremal loss is convex, and Jensen's inequality gives the Rademacher complexity ordering. No internal inconsistency or missing step appears in the main theoretical development. The load-bearing concern is therefore not about the internal logic but about the external validity of the assumption and the empirical support. The experimental design has a closed loop: the teacher model defines z_oracle, generates the training labels, and serves as the ground truth for in-distribution evaluation. This makes the empirical ranking Oracle ≥ 5-level ≥ 3-level ≥ Binary a near-tautology for the portion of performance due to label granularity, and it provides no evidence about human label distributions. The paper is honest about making Assumption 3.1 the only assumption, but it does not demonstrate that real annotation protocols can meet it. The proposed annotation guideline is a plausible intervention, yet calibration is an empirical question; the existence result in Theorem 3.2 does not substitute for evidence that human annotators satisfy the moment condition. Because the theoretical contribution is valid but conditional, and the practical claim requires further validation, the reader's CONDITIONAL verdict remains appropriate. I would not escalate to REJECT, since the theory is sound and the assumption is explicitly stated; nor would I accept unconditionally, since the central empirical claim is not yet supported by human data.","tokens_in":23215,"tokens_out":5546,"duration_ms":62803,"concrete_test":"Run a human annotation calibration study: select 500 prompt-response pairs; have a first group of annotators assign five-level ordinal labels (e.g., 0, 0.25, 0.5, 0.75, 1) under the paper's proposed guideline, and have a separate large group provide binary preference judgments so that the preference fraction p̂(x,y1,y2) can be estimated for each pair. Fit the calibration regression E[Z | p̂] = a + b·p̂ and jointly test H0: a = 0, b = 1. If the test rejects, Assumption 3.1 fails for human annotators under realistic conditions; if it does not reject with adequate precision, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theorem (Theorem 4.9) and its corollaries are internally correct: the proof via hierarchical expectation and Jensen's inequality works because the loss is affine in the feedback variable and the pointwise supremum of affine functions is convex. However, the entire statistical benefit of ordinal feedback is conditional on Assumption 3.1: E[Z | (x, y1, y2)] = z_oracle(x, y1, y2). The paper argues this is a mild generalization of the binary-feedback assumption, where E[Z | (x, y1, y2)] = P(y1 ≻ y2 | x) is exactly the standard Bradley-Terry rate assumption. But for ordinal feedback the assumption is much more demanding: it requires that every quantization of the preference scale, with labels chosen by humans, preserves the conditional mean of the oracle probability. Theorem 3.2 only shows that for any oracle and any ordinal set Z, there exists some random label distribution satisfying Assumption 3.1; it does not show that a human following the proposed annotation guideline produces such a distribution. The guideline ('slightly better' = 75% of the population prefer y1) asks annotators to map their subjective estimate of the population proportion to a fixed label; unless the conditional mean of z_oracle within each label category exactly equals the assigned number, deterministic or even stochastic human labeling will not satisfy Assumption 3.1. No evidence is provided that humans are calibrated in this sense. Moreover, the experiments cannot validate the assumption: labels are sampled from a teacher model that defines z_oracle, and the in-distribution evaluation computes accuracy against that same teacher's scores. This is a closed loop: fine-grained labels carry more information about the teacher's continuous score, so better performance on the teacher-defined metric is guaranteed under very mild conditions and reveals nothing about whether human ordinal labels are unbiased.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies reward modeling under ordinal feedback, generalizing the binary Bradley-Terry setup to arbitrary finite label sets Z ⊂ [0,1]. The central assumption (Assumption 3.1) is that the ordinal label Z satisfies E[Z|(x,y1,y2)] = z_oracle(x,y1,y2), where z_oracle is the population preference probability. Under this assumption, the paper proves that the cross-entropy objective (2) yields the same population loss as the oracle (Proposition 4.3), and that a coarser feedback system has larger Rademacher complexity than a finer one when the loss is affine in the feedback (Theorem 4.9). This leads to Corollary 4.10, which places any ordinal feedback system between the oracle and binary systems in terms of Rademacher complexity. The paper extends the analysis to hinge loss and DPO, and draws connections to knowledge distillation. Experiments use a teacher reward model (Skywork-Reward-Gemma-2-27B) to generate oracle scores, then sample synthetic ordinal labels for binary/3-level/5-level systems, showing improved in-distribution and out-of-distribution accuracy for finer granularity, and benefits of including tied samples.","tokens_in":23492,"tokens_out":13877,"duration_ms":128959,"significance":"The theoretical result is a rigorous and valuable contribution: Theorem 4.9 gives a clean, general sufficient condition under which finer-grained feedback reduces Rademacher complexity, and the affinity condition covers the standard cross-entropy and hinge losses. The hierarchical-expectation coupling and the connection to soft labels are interesting and likely to be useful beyond reward modeling. However, the practical significance is currently limited: the key assumption is not validated with human annotators, and the experiments are synthetic, using a single teacher model both to generate labels and to define the evaluation oracle. The paper ships a machine-checkable proof strategy for the main theorem and a clear assumption-led framework; these are strengths. The empirical claims outrun the evidence, so the paper would benefit from either human-annotation experiments or a substantial tempering of the practical conclusions.","major_comments":[{"comment":"The empirical evaluation does not test Assumption 3.1 for human annotators. The four feedback systems are generated by sampling from z_oracle, which is defined by the Skywork teacher model (Section 5.1), so Assumption 3.1 holds by construction. Consequently, the experiments validate the theory only in the synthetic setting where the assumption is guaranteed, and provide no evidence that human ordinal labels (e.g., via the guideline in Section 3.1) satisfy E[Z|(x,y1,y2)] = z_oracle. The central practical claim that 'fine-grained feedback leads to better reward learning' (Section 5.2) is therefore not established for real human feedback.","section":"Section 5.1 / Assumption 3.1"},{"comment":"The in-distribution evaluation is circular: the ID ground truth is defined by the same teacher model that generated the training labels, so the observed ID performance differences may reflect improved regression to the teacher's own scores rather than a general property of ordinal feedback. The OOD results on RewardBench are less circular, but the training labels are still teacher-generated, so they cannot support the 'wisdom of the crowd' claim for human annotators.","section":"Section 5.1, Table 1"},{"comment":"Theorem 4.9 requires Z to be a hierarchical expectation of Z'. Corollary 4.7 establishes this only when Z is the oracle or Z' is binary. The paper does not prove that a 5-level system satisfying Assumption 3.1 is a hierarchical expectation of a 3-level system, yet Section 5.2 claims and experimentally demonstrates the ordering Oracle ≥ 5-level ≥ 3-level ≥ Binary. Thus the middle part of this ordering is not justified by the stated theory; the experiments go beyond the proven result. The authors should either prove the general ordering or explicitly limit the theoretical claim to comparisons with binary feedback.","section":"Section 4.1, Corollary 4.7, Section 5.2"},{"comment":"The annotation guideline is presented as a way to make human labels satisfy Assumption 3.1, but the assumption requires the conditional expectation of the label, given the specific (x,y1,y2), to exactly equal the oracle probability. Human annotators, even with the suggested guideline, are unlikely to be perfectly calibrated for every prompt-response pair. The paper provides no calibration study or analysis of how deviations from this assumption affect the finite-sample guarantees. Since Assumption 3.1 is the only load-bearing premise for Theorem 4.9, this is a significant gap.","section":"Section 3.1"}],"minor_comments":[{"comment":"The 5-level example in Section 3.1 uses Z = {0, 0.25, 0.5, 0.75, 1}, while the experiments in Section 5.1 use Z = {0, 0.2, 0.5, 0.8, 1}; this inconsistency is confusing and should be reconciled with a single convention.","section":"Section 3.1 / Section 5.1"},{"comment":"The notation 'y_T ∼ \\bar{y}_T' is used without a precise definition of the conditional distribution; please clarify how sampling from the teacher's output is defined.","section":"Section 4.3"},{"comment":"The temperature T = 20/3 is chosen to match the distribution of UltraFeedback and HelpSteer2, but there is no sensitivity analysis for T. Since T scales the oracle scores and therefore affects all downstream label generation, a discussion of its role as a free parameter would increase reproducibility.","section":"Appendix C.1"},{"comment":"The related-work section does not mention HelpSteer2, which provides human ordinal preference labels and is directly relevant to the proposed framework; citing it in the related work would better position the contribution.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The theoretical core of the paper is sound and publishable, but the empirical validation is far weaker than the claims suggest. The experiments use a teacher model to define the oracle, generate labels, and evaluate in-distribution performance, so they cannot validate Assumption 3.1 for humans. In my view, the paper needs either human-annotation data or a clear, prominent limitation statement; without that, the practical conclusions are unsupported. Given that the theoretical contribution is intact and the issues are fixable within the scope of a revision, I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core theoretical result checks out. The paper generalizes Bradley-Terry to arbitrary ordinal feedback under a single marginal unbiasedness assumption, and proves that finer feedback reduces Rademacher complexity via a hierarchical expectation coupling. I read the proof of Theorem 4.9 carefully; Jensen goes through because the loss is affine in the feedback variable, and the convexity of the supremum is standard. This is a genuine, nontrivial extension of the tie-handling literature, which stops at three levels. The paper is honest that Assumption 3.1 is the only assumption, and Proposition 4.3 correctly shows that any two systems satisfying it have the same population loss.\n\nThe trouble is that the paper wants to say something about human annotators, but the experiments never involve humans. Labels are sampled from a single teacher model that defines the oracle, and that same teacher's scores are the in-distribution ground truth. So the fine-grained labels carry strictly more information about the teacher's continuous score, and the observed ordering Oracle ≥ 5-level ≥ 3-level ≥ Binary is close to a tautology. The assumption that human ordinal labels have conditional expectation exactly equal to the oracle is much stronger than the binary BT assumption—it requires every quantization of the preference scale to preserve the conditional mean, with no evidence that annotators are calibrated in that sense. The Galton ox anecdote is suggestive, not evidence. The proposed annotation guideline ('slightly better' means 75% of the population prefers y1) is a reasonable hypothesis, but it is untested.\n\nThe paper also argues that Rao-Kupper/BTT is over-parameterized and unnecessary, yet never compares against it empirically. That is a notable omission, especially since the tied-sample experiments (Section 5.3) show non-monotone effects that a BTT baseline would help interpret. The temperature T is tuned post hoc to match real-world preference distributions, which is transparent but makes the 'tied data helps' result look less robust.\n\nNone of this kills the paper. The theory is sound and the framing is useful: it tells practitioners that if they can get unbiased ordinal labels, they get a statistical benefit with no extra hyperparameters. What it does not do is establish that humans provide such labels. A serious referee should ask for human-annotation experiments, or at minimum an independent teacher that does not define the evaluation oracle, and a comparison with existing tie models. I would send it out.","headline":"The generalization bound result is real and worth taking seriously, but the experiments do not support the load-bearing assumption about human annotators, so the paper needs revision, not rejection.","tokens_in":24127,"tokens_out":1366,"would_cite":true,"duration_ms":16953,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62J15"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that more fine-grained human preference labels produce no worse reward models than binary labels, and in finite samples better ones.","keywords":["reward modeling","ordinal feedback","Bradley-Terry model","Rademacher complexity","hierarchical expectation","wisdom of the crowd","direct policy optimization","knowledge distillation"],"falsifier":"Collect a set of preference pairs with many independent human ordinal labels per pair, and independently estimate the population preference probability by averaging a large number of binary votes per pair. Then test the calibration identity $\\mathbb{E}[Z|x,y_1,y_2] = z_{\\mathrm{oracle}}$ for each label level; if the conditional expectations deviate systematically, Assumption 3.1 fails and the theorem's benefit does not transfer to human-labeled reward models.","tokens_in":22967,"feed_emoji":"⚖️","tokens_out":9255,"duration_ms":82334,"temperature":0.7,"pith_summary":"Reward models for LLMs are normally trained on binary 'response A beats B' labels, even when annotators are asked to report finer distinctions such as 'slightly better' or 'tied.' This paper proposes to keep that ordinal information and shows a precise statistical sense in which it cannot hurt and generally helps learning. The single assumption is that each rater's label is an unbiased estimate of the population-level preference probability, which the authors connect to the 'wisdom of the crowd.' Under that assumption, a more fine-grained feedback system has no larger Rademacher complexity—a standard measure of worst-case overfitting—than a coarser one, giving a tighter generalization bound. Experiments with two LLMs confirm that 5-level and 3-level labels outperform binary labels on both in-distribution accuracy and the RewardBench out-of-distribution benchmark.","feed_headline":"Finer preference labels tighten reward learning — with proof","feed_subtitle":"Under an unbiased-rater assumption, 5-level ordinal feedback beats binary feedback in-distribution and out-of-distribution.","key_machinery":"The central object is the hierarchical expectation coupling: a pair of random variables $(W, W')$ such that $W = \\mathbb{E}[W'|W]$ almost surely, so the coarser feedback is the conditional expectation of the finer one. This ordering is what makes the Rademacher comparison work: for any loss that is affine in the feedback variable (cross-entropy and the generalized hinge loss both satisfy this), the function $\\sum_i \\varepsilon_i \\ell(\\cdot, h(x_i,y_{i,1},y_{i,2}))$ is convex, so conditioning the finer feedback on the coarser one and applying Jensen's inequality only decreases the expected supremum. Theorem 4.9 is the resulting inequality; the coupling is the mechanism that carries the argument.","core_discovery":"The central claim is Theorem 4.9: if feedback system $Z$ is a hierarchical expectation of $Z'$ (the coarser system is the conditional expectation of the finer one), then $\\mathrm{Rad}_{Z,n}(\\ell\\circ H) \\leq \\mathrm{Rad}_{Z',n}(\\ell\\circ H)$ for any hypothesis class $H$ and any loss affine in the feedback. Since every ordinal feedback system satisfying Assumption 3.1 sits between the oracle feedback $z_{\\mathrm{oracle}}$ and binary $\\{0,1\\}$, Corollary 4.10 orders Rademacher complexity as $\\mathrm{Rad}_{Z_{\\mathrm{oracle}},n} \\leq \\mathrm{Rad}_{Z,n} \\leq \\mathrm{Rad}_{Z_{\\mathrm{binary}},n}$. The authors therefore claim that a more fine-grained feedback structure leads to better reward learning for both in-distribution and out-of-distribution performance. The same theoretical result is extended to hinge loss and to direct policy optimization, and is applied to knowledge distillation, where soft teacher labels are shown to be a hierarchical expectation of hard labels, reducing variance. Empirically, using an oracle teacher to generate labels on Skywork-Reward-Preference-80K, the 5-level feedback system improves ID accuracy and RewardBench OOD accuracy over 3-level and binary feedback for Llama-3.2-1B and Gemma-2-2B.","pith_inferences":["Editorial inference: The same hierarchical-expectation argument suggests a direct test for knowledge distillation—distill from teachers of varying accuracy and check that the Rademacher reduction grows as teacher soft labels become more concentrated, which the paper does not run.","Editorial inference: A practical extension would measure how many binary labels are needed to match the generalization of a given number of 5-level labels; the Rademacher gap implies a sample-complexity advantage, but the paper does not quantify the constant.","Editorial inference: If real annotators are not marginally unbiased, the framework suggests a calibration step that estimates and corrects per-label biases, which would restore the ordering without changing the loss."],"forward_implications":["Any ordinal feedback system satisfying Assumption 3.1, including 3-level and 5-level systems, has Rademacher complexity no larger than binary feedback, so its generalization bound is no worse; the oracle feedback is the best possible.","The result extends to direct policy optimization, so DPO training can use 'tied' and 'slightly better' labels directly rather than discarding them.","Because population loss is identical for all feedback systems under the assumption, the benefit is purely finite-sample; the experiments show it as higher ID and OOD accuracy.","Mixing a moderate fraction of tied labels (25–75 percent) improves reward learning over all-binary data, while 100 percent tied labels cause reward collapse, confirming that the proposed objective uses tied samples productively.","Annotation guidelines should attach numerical probability meanings to qualitative labels, such as 'slightly better' meaning 75 percent of the population prefers this response, making Assumption 3.1 actionable for human annotators."],"supporting_citations":[{"why":"Defines the paired-comparison Bradley-Terry model that the binary reward-model objective generalizes.","marker":"(Bradley and Terry, 1952)"},{"why":"Provides the ox-weight 'wisdom of the crowd' experiment used to justify Assumption 3.1.","marker":"(Galton, 1907)"},{"why":"Introduces DPO, the policy-optimization objective to which the Rademacher result is extended.","marker":"(Rafailov et al., 2024)"},{"why":"Introduces knowledge distillation and soft labels, which the hierarchical-expectation analysis reinterprets as variance reduction.","marker":"(Hinton, 2015)"},{"why":"Supplies the Skywork preference dataset and the oracle scoring model used to generate ordinal labels in experiments.","marker":"(Liu et al., 2024a)"},{"why":"Provides RewardBench, the out-of-distribution benchmark for evaluating trained reward models.","marker":"(Lambert et al., 2024)"},{"why":"Recent tie-aware Bradley-Terry models that the paper compares against for tied-sample handling.","marker":"(Chen et al., 2024; Liu et al., 2024b)"}],"fun_headline_variants":["Ordinal feedback sharpens reward models, with proof","Fine-grained labels beat binary in reward learning","Finer feedback reduces Rademacher complexity","Proof: ordinal feedback improves reward modeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expectation of the human ordinal label equals the true population preference probability for every prompt-response pair (Assumption 3.1); if annotators are not calibrated to that probability, the stated statistical benefits do not apply to real reward-model learning.","fun_headline_variants_meta":{"raw":{"variants":["Ordinal feedback sharpens reward models, with proof","Fine-grained labels beat binary in reward learning","Finer feedback reduces Rademacher complexity","Proof: ordinal feedback improves reward modeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1701,"prompt_tokens":1144,"completion_tokens":557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":760,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":760,"tokens_out":557,"duration_ms":21489,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:08:30.890043+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of preference pairs with many independent human ordinal labels per pair, and independently estimate the population preference probability by averaging a large number of binary votes per pair. Then test the calibration identity $\\mathbb{E}[Z|x,y_1,y_2] = z_{\\mathrm{oracle}}$ for each label level; if the conditional expectations deviate systematically, Assumption 3.1 fails and the theorem's benefit does not transfer to human-labeled reward models.","supporting_citations":[{"cited_title":"Smith, Hannaneh Hajishirzi","cited_arxiv_id":null,"evidence_quote":"Provides RewardBench, the out-of-distribution benchmark for evaluating trained reward models."}],"review_version":1}