{"id":"d30032dc-25e2-4d8d-8d16-facc777fc557","arxiv_id":"2412.03822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reward models trained with a margin derived from synthetic LLM judgments better match aggregate human preferences than standard binary-trained reward models, mainly on subjective prompts.","lead":"This paper argues that binary preference judgments fail to capture the diverse opinions of real users, and proposes a simple fix: use an LLM to estimate how much people might disagree on each training example, then add that disagreement as a regularization term when training reward models. The fix improves correlation with human estimates of aggregate preferences, especially on subjective questions that allow multiple correct answers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Margin effect is confounded with training setup: the regularized RM is compared against a released checkpoint rather than a matched no-margin baseline, so Tables 1-2 do not isolate the proposed regularization.","rationale":"The reader's weakest assumption concerned the synthetic Llama-3 judgments and the '10 people' annotation protocol; both are real external-validity risks, and the social impacts statement candidly flags the first. I identify a different, more direct threat to internal validity: the positive result in Tables 1 and 2 compares a newly trained regularized model to a pre-existing released checkpoint, not to a matched non-margin baseline. Any difference could come from initialization, data mixing, epochs, or the validation-based hyperparameter sweep rather than from the margin term. This is load-bearing because the paper's central contribution is precisely the claim that the margin improves alignment; without a controlled comparison, that claim is unsupported as written. The check I propose is concrete and inexpensive: retrain the baseline under identical conditions with margin fixed to zero. I partially agree with the reader because both concerns point to inconclusive empirical support, but the unmatched-baseline issue is not acknowledged in the text and is arguably more fundamental. For this reason I would move the verdict from CONDITIONAL to REJECT, while noting the method could be rehabilitated by rerunning the comparison with a proper control.","tokens_in":10314,"tokens_out":9631,"duration_ms":100004,"concrete_test":"Train a matched baseline: take the same pre-trained Deberta-V3 checkpoint used for the regularized model, train on the same training data with the same preprocessing, epochs, and validation-based hyperparameter sweep, but with the standard objective in Eq. (1) (or Eq. (4) with margin fixed to 0). Evaluate both models on the same 150-example test set, with ID test examples excluded from training. If the matched baseline reproduces the released checkpoint's scores and the regularized model's improvements persist, the margin claim is supported; if the matched baseline performs as well as or better than the regularized model, the reported gains are attributable to training setup rather than the margin.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.3 trains a new Deberta-V3 model with Eq. (4) on 'the same training data as the baseline model from Section 3.2' and sweeps learning rates on validation data. The baseline RM, however, is the released OpenAssistant checkpoint (footnote 3), not a model retrained under identical conditions with Eq. (1). Consequently, the Δ columns in Tables 1 and 2 confound the margin intervention with differences in pretraining, data handling, epochs, and hyperparameter selection. The paper's central claim—that incorporating the margin term improves alignment—requires the margin to be the only difference; the reported comparison cannot establish that. This is not a minor statistical quibble: the 'fix' phase's entire result is uninterpretable without a matched no-margin baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that reward models trained on binary preference judgments misrepresent aggregate user preferences for subjective items. It proposes a two-dimensional taxonomy of subjectivity (plurality of correct answers and indistinguishability of candidate responses), analyzes reward-model performance on 150 examples spanning in- and out-of-domain datasets, and introduces a margin-based regularization term estimated from synthetic pairwise judgments generated by Llama-3 70B. The regularized model is reported to improve Pearson correlation and L1 agreement with human '10 people' population-preference estimates, especially on multiple-correct-answer and out-of-domain examples.","tokens_in":10434,"tokens_out":4335,"duration_ms":42213,"significance":"If the reported improvement is causal and replicable, the margin-regularization method is a cheap, drop-in modification to standard reward modeling and the taxonomy offers a useful decomposition of subjectivity in preference data. The authors are appropriately cautious in the Social Impacts Statement about the limits of synthetic annotations. The evaluation is conducted against external human aggregate-preference estimates, which mitigates circularity. However, the evidence base is thin (150 examples, per-cell N as low as 10, no significance tests) and the main comparison is confounded, so the central claim is currently under-supported.","major_comments":[{"comment":"The reported Δ gains do not isolate the margin term: the baseline is the released OpenAssistant Deberta-V3 checkpoint (footnote 3), while the regularized model is retrained from the same pretrained weights with a learning-rate sweep (footnote 6), so differences in data ordering, optimization hyperparameters, and checkpoint selection are confounded with the intervention; a matched no-margin baseline trained under identical conditions is required to support the causal claim.","section":"§3.3, Tables 1-2"},{"comment":"The test set construction does not state that the 25 examples per dataset were excluded from the training data used for the four in-distribution datasets, so the ID correlations in Tables 1-2 may include memorization effects; the paper should explicitly describe the train/test split and, if necessary, use a held-out subset.","section":"§3.2"},{"comment":"The study has insufficient statistical power for its claims: 150 test examples with per-cell N as low as 10 (Single Correct OOD), no confidence intervals or significance tests, and several reported gains (e.g., Δ=0.015 for All ID, Δ=0.029 for UltraFeedback, Δ=0.021 for Indistinguishable) are within the range of plausible sampling noise; the authors should report bootstrap intervals or per-item significance tests.","section":"§3.2, Tables 1-2"},{"comment":"The claim that reward models correlate weakly with user preferences along both taxonomy dimensions is contradicted by the baseline results for the Indistinguishable axis: Table 1 shows Pearson correlation 0.640 for Distinguishable vs 0.631 for Indistinguishable, and Table 2 shows L1 0.244 vs 0.176, i.e., the baseline is not worse on indistinguishable pairs; the 'break it' phase only supports the Multiple Correct Answers dimension, and the paper should either reframe the claim or add stronger evidence for the second axis.","section":"Abstract, §3.2, Tables 1-2"},{"comment":"The margin term is entirely derived from Llama-3 70B synthetic judgments, yet the paper never validates that Llama-3's disagreement distribution matches the human annotators' '10 people' estimates; a direct comparison between the synthetic margin and human disagreement on the test set (e.g., correlation between m and the spread of human judgments) would test the core premise, which the Social Impacts Statement itself acknowledges has 'mixed empirical validation'.","section":"§3.3"}],"minor_comments":[{"comment":"The phrase 'is and indicator function' contains a typo; it should be 'is an indicator function'.","section":"§2, Eq. (2)"},{"comment":"The sentence 'Empirical evidence suggests that this approach yields more reliable results...' lacks a citation; please add a reference.","section":"§3.2"},{"comment":"In the 'User Preference Estimation' scale, the last option is listed as '10 A - 0 B' instead of '0 A - 10 B'; this appears to be a copy-paste error.","section":"Appendix B, Task 3"},{"comment":"The phrase 'we sent a categorization' should be 'we present a categorization'.","section":"§3, opening sentence"},{"comment":"Please clarify whether Pearson correlations and L1 losses are computed on the normalized 0-1 preference fractions or on raw reward scores, and state this in the captions.","section":"Tables 1-2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a workshop-style preliminary study; the idea is timely and the writing is clear. The decisive issue is the confounded baseline, which prevents interpreting the headline improvement as caused by the margin. The statistical power and the inconsistency on the indistinguishability axis also need attention before the central claims can be considered established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. First, the taxonomy is genuinely useful: splitting subjective examples by whether the prompt admits multiple correct answers and whether the two responses are paraphrases is a clean way to think about reward model failures. Second, the headline result—margin regularization improves aggregate alignment—is not actually tested by the experiments. The baseline is the released OpenAssistant checkpoint, and the regularized model is fine-tuned from a pre-trained Deberta-V3 on the same training data. The paper calls it 'the baseline model from Section 3.2,' but that model was never retrained with the standard objective under matched conditions. So the Δ columns in Tables 1–2 confound the margin with differences in data handling, epochs, and hyperparameters. That's not a quibble; it's the load-bearing comparison of the 'fix it' half.\n\nThe idea itself has merit. Deriving a margin from the spread of synthetic Llama-3 judgments and plugging it into the Bradley-Terry loss is cheap and plausible. The authors are also admirably candid in the Social Impacts Statement about the weak empirical validation of synthetic annotations. The 'break it' analysis—reward models correlate poorly on multiple-answer prompts and out-of-domain sets—is directionally consistent with prior work, though it rests on only 150 examples with per-cell N as low as 10, no error bars, and no explicit holdout statement for the ID test set.\n\nTwo more soft spots. The indistinguishability dimension doesn't show a strong baseline deficit; the baseline does about as well there, so the abstract's blanket claim that both dimensions are problematic is overstated. And the synthetic margin is generated by Llama-3 70B under a prompt that asks for 'the better answer,' which is a binary, not a population estimate; the disagreement is just sampling noise from nucleus sampling. That may approximate heterogeneity, but the paper doesn't validate it.\n\nWho should read it: anyone working on reward modeling or pluralistic alignment. The taxonomy and the margin idea are worth a look, but the empirical support needs a matched no-margin baseline, significance testing, and more data. I'd send it to review because the idea is important and the authors have shown they can think clearly about limitations; a good referee could push them to fix the comparison. As it stands, the central claim is unproven.","headline":"A promising margin-regularization idea and a useful taxonomy, but the main experiment compares a retrained model against a released checkpoint, so the headline improvement is confounded.","tokens_in":10985,"tokens_out":3343,"would_cite":false,"duration_ms":33167,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward models trained on one annotator's binary choice systematically miss cases where users would disagree, and a margin term built from synthetic preference judgments recovers much of that lost alignment.","keywords":["reward modeling","preference tuning","subjectivity","annotator disagreement","margin regularization","synthetic annotations","RLHF","pluralistic alignment"],"falsifier":"On the same 150 examples, collect true preference votes from a large panel of independent annotators (not the 10-people projection protocol), compute real disagreement margins, and correlate them with Llama-3's synthetic margins. Low or zero correlation would falsify the claim that synthetic judgments estimate aggregate user disagreement.","tokens_in":10074,"feed_emoji":"🗳️","tokens_out":5113,"duration_ms":49013,"temperature":0.7,"pith_summary":"The paper argues that standard reward modeling, which learns from a single annotator's binary pick between two outputs, is poorly calibrated for real deployment where users hold diverse preferences. It classifies subjectivity along two axes: prompts that admit multiple correct answers, and response pairs that are paraphrase-level indistinguishable. On a 150-example test set, the authors show that a DeBERTa-v3 reward model correlates weakly with aggregate user preferences on these subjective cases, especially out of domain. They then add a margin term to the training loss, computed from ten synthetic preference judgments per example generated by an LLM, that shrinks when synthetic annotators disagree. The regularized model improves correlation with aggregate preferences on multiple-correct and out-of-domain subsets without degrading single-correct performance.","feed_headline":"Disagreement margin in reward training recovers subjective user preferences","feed_subtitle":"Synthetic preference judgments from an LLM shrink the gap between binary training labels and real-world opinion spread.","key_machinery":"The key object is the margin term $m_{x,y_0,y_1} \\in [0,1]$ inserted inside the sigmoid of the reward-model loss. It is computed by sampling $n$ synthetic binary judgments $j'_i$ from an LLM for the same prompt-response pair and taking $\\left|\\sum_i j'_i - n/2\\right| / (n/2)$: the margin is one when all synthetic annotators agree and zero when they split evenly. Inserting it as $\\log\\sigma(r_\\theta(x,y_j) - r_\\theta(x,y_{1-j}) - m)$ forces the model to assign a large score gap only when the (synthetic) population is unanimous, and to keep predictions near the decision boundary when preference is contested. The taxonomy of subjectivity, plurality of correct answers and indistinguishability of responses, is the accompanying device used to locate where the baseline fails and where the regularization helps.","core_discovery":"The central claim is that the binary preference signal used to train reward models systematically misrepresents aggregate human preferences wherever legitimate disagreement exists, and that a cheap, plug-in regularization can partially correct this. The authors define an example as subjective when the prompt admits multiple correct answers or the two candidate responses are near-paraphrases, and they measure a reward model's predictions against \"If you asked 10 people\" estimates of population preference. Baseline reward-model predictions correlate much more weakly with these aggregate preferences on multiple-correct (Pearson 0.572) and out-of-domain (0.377) examples than on single-correct ones (0.802). Incorporating a disagreement-scaled margin term into the logistic preference loss, built from synthetic judgments sampled from Llama-3 70B, raises multiple-correct correlation to 0.626 and out-of-domain correlation to 0.488, while lowering L1 error and leaving single-correct performance unchanged. The implication the paper presses is that disagreement should be treated as information about the preference distribution, not as label noise to be averaged away.","pith_inferences":["Editorial inference: if synthetic disagreement tracks human disagreement, the same margin trick should transfer to direct preference-optimization losses (e.g., DPO-style objectives), not only to learned reward models; the paper only tests DeBERTa-v3.","Editorial inference: the \"If you asked 10 people\" annotation protocol could itself become a cheap, validated way to collect preference distributions at scale; a calibration study comparing it to true independent votes would settle this.","Editorial inference: the taxonomy suggests a data-selection heuristic, deliberately oversampling subjective examples during reward training might improve calibration on exactly the inputs where deployed models currently appear arbitrary.","Editorial inference: a risk not resolved here is that LLM-generated disagreement may reflect the model's own priors rather than user diversity; the paper flags this concern in its social impacts statement."],"forward_implications":["Existing binary preference datasets can be upgraded without re-annotation: sample synthetic judgments once, store a per-example margin, and retrain the reward model with the modified loss.","Regularized reward models improve aggregate-preference correlation on multiple-correct-answer prompts (Pearson 0.572 to 0.626) and on out-of-domain data (0.377 to 0.488), with L1 error down across in-domain, out-of-domain, and indistinguishable-response subsets.","Single-correct-answer and distinguishable-response performance is essentially unchanged, so the intervention targets the subjectivity failure mode rather than trading off overall quality.","The framework can accommodate aggregation rules other than the utilitarian majority vote used in the experiments, since the margin is a design choice rather than a fixed quantity.","For developers, the method offers a stopgap that approximates multi-annotator preference data without the cost of recollecting human judgments at scale."],"supporting_citations":[{"why":"Supplies the margin-term idea from Llama 2's reward modeling, where preference strength is scaled into the loss.","marker":"[3]"},{"why":"Provides the HH-RLHF helpful and harmless preference dataset used as an in-domain training and evaluation set.","marker":"[2]"},{"why":"Provides the OpenAI summarization preference dataset used as an in-domain training and evaluation set.","marker":"[6]"},{"why":"WebGPT preference data is one of the four in-domain training sets for the baseline reward model.","marker":"[40]"},{"why":"Synthetic Instruct GPT-J pairwise data completes the in-domain training set for the baseline reward model.","marker":"[41]"},{"why":"PRISM supplies an out-of-domain test set with subjective, multicultural preference judgments where the baseline degrades sharply.","marker":"[27]"},{"why":"UltraFeedback supplies the second out-of-domain test set used to measure generalization.","marker":"[42]"},{"why":"Llama-3 70B Instruct is the model that generates the synthetic preference judgments used to build the margin term.","marker":"[43]"},{"why":"Supports the finding that annotator disagreement is systematically underrepresented by current reward models.","marker":"[29]"}],"fun_headline_variants":["Binary reward labels miss diverse user tastes; margin term helps","Synthetic disagreement margins align rewards with aggregate preferences","Reward models regularized by disagreement outperform on subjective prompts","From binary to plural: Rewards that respect opinion spread","Disagreement-aware training recovers what binary judgments lose"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM's synthetic preference disagreements, and the \"If you asked 10 people\" estimates used to evaluate them, both reflect how real users actually disagree; if either is wrong, the margin regularizes toward the wrong target.","fun_headline_variants_meta":{"raw":{"variants":["Binary reward labels miss diverse user tastes; margin term helps","Synthetic disagreement margins align rewards with aggregate preferences","Reward models regularized by disagreement outperform on subjective prompts","From binary to plural: Rewards that respect opinion spread","Disagreement-aware training recovers what binary judgments lose"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1312,"prompt_tokens":946,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":287}},"tokens_in":562,"tokens_out":366,"duration_ms":4734,"temperature":1.0,"reasoning_tokens":287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:01:50.522406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same 150 examples, collect true preference votes from a large panel of independent annotators (not the 10-people projection protocol), compute real disagreement margins, and correlate them with Llama-3's synthetic margins. Low or zero correlation would falsify the claim that synthetic judgments estimate aggregate user disagreement.","supporting_citations":[{"cited_title":"synthetic-instruct-gptj-pairwise (revision cc92d8d),","cited_arxiv_id":null,"evidence_quote":"Synthetic Instruct GPT-J pairwise data completes the in-domain training set for the baseline reward model."}],"review_version":1}