{"id":"eb25a12a-ebae-4118-a715-3ee504f04ebe","arxiv_id":"2506.06298","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A small ensemble of reward functions can be trained to match pairwise human preference frequencies, offering a practical route to pluralistic AI alignment.","lead":"This paper proposes that AI alignment should use a small collection of reward models whose combined opinions match how often human annotators prefer one response over another. It gives theoretical guarantees that small ensembles can do this and shows a training heuristic that improves calibration on four preference datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's construction samples the annotators' true reward functions; the FSAM learns from aggregated, noisy labels in a restricted reward class, so the existence guarantee does not transfer to the learned ensembles.","rationale":"After reading the paper in good faith, I find the reader's weakest assumption to be the most load-bearing: the existence and outlier-pruning theorems (Theorems 2 and 3) are built on the premise that each annotator's preferences are exactly induced by some reward in the chosen class Θ. The FSAM heuristic (Section 4) never has access to annotator identities or to the true annotator rewards; it learns from aggregate soft labels p using a fixed architecture (linear head on Llama-3-8B embeddings). There is no theoretical bridge showing that the learned ensemble is close to the ideal ensemble of Theorem 2, nor that R_k contains a nearly calibrated ensemble. Theorem 4 is a generalization bound, not an existence guarantee for good ensembles in R_k. The empirical section partially mitigates this by demonstrating lower held-out MSE than the single-reward floor, but the floor comparison is essentially a discretization advantage of mixtures of binary voters, and the evaluation uses the same noisy p as the target. The paper does not report error bars, comparisons to existing pluralistic-alignment baselines, or code, which further limits the strength of the empirical evidence. These are fixable, so CONDITIONAL is appropriate. The proposed synthetic test would directly probe whether the learned ensembles can approximate the true population p* when the model class is misspecified.","tokens_in":27469,"tokens_out":17677,"duration_ms":180547,"concrete_test":"Run a synthetic experiment: define a population of annotators whose preferences are generated by reward functions outside the linear-head class (e.g., quadratic forms over the embedding φ(x,y)). Draw a finite dataset of pairwise soft labels p by averaging n simulated annotator votes per triple, train the FSAM ensemble with k=8, and compute the ensemble's calibration MSE against the exact p* from the generating distribution. If the MSE is substantially larger than ε = 1/(4k) (the Theorem 2 bound), the reward-inducibility assumption is load-bearing; if it is comparable to the MSE on a held-out noisy set, the gap is less severe.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical guarantee (Theorem 2) is an existence result for an ensemble formed by uniformly sampling the true annotator reward functions r_{θ_i}, which are assumed to be reward-inducible within the parameter class Θ (Section 2). The practical FSAM (Section 4) instead trains reward models on finite, noisy soft labels p (aggregated from 3–10 annotators) using a linear head on frozen Llama-3-8B embeddings. Nothing ensures that the learned rewards coincide with any annotator's preferences, nor that the true annotator rewards lie in this class. Theorem 4 provides a uniform-convergence bound between the empirical loss and the biased true loss L'(r) for any ensemble in R_k, but it does not show that R_k contains an ensemble with low calibration error. Thus the theoretical claim that O(1/ε) rewards suffice does not apply to the learned ensembles, and the empirical validation measures MSE against the same noisy observed p rather than the true population fractions p*. This is not an internal inconsistency, but it means the paper's two central contributions—the existence theorem and the FSAM heuristic—are not rigorously connected. The assumption is stated explicitly in Section 2, but its failure mode is not analyzed or tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a distribution over reward functions—rather than a single scalar reward—as a model of pluralistic human preferences, and introduces pairwise calibration as the criterion: for every pair of responses, the ensemble's fraction of rewards preferring one response should match the annotator preference fraction. Theoretically, it proves that finding a perfectly calibrated ensemble is NP-hard (Theorem 1), that an O(1/ε)-sized ensemble suffices for ε-pairwise calibration (Theorem 2), that outliers can be pruned without much calibration loss (Theorem 3), and that finite-sample uniform convergence holds under a finite VC-dimension assumption (Theorem 4). Empirically, it proposes a forward stagewise additive modeling (FSAM) heuristic that greedily fits reward models to residual calibration errors, and reports lower held-out MSE than a single deterministic reward's lower bound on four preference datasets. The paper includes detailed deferred proofs, a discussion of non-identifiability, and explicit limitations concerning higher-order preferences and sensitive topics.","tokens_in":27720,"tokens_out":3748,"duration_ms":42315,"significance":"If the claims hold, the paper makes a useful conceptual contribution: a formally defined, theoretically motivated notion of pluralistic reward distribution that does not require annotator identities or predefined groups, together with an implementation heuristic. The existence and outlier-pruning theorems are nontrivial and are proved with explicit constructions, and the paper is commendably transparent about non-identifiability and about the fact that pairwise data cannot recover higher-order preference structure. The empirical section also addresses a real gap by using soft labels rather than majority-vote collapse. However, the significance is tempered by the gap between the existence theorems, which are proved for exact annotator reward functions, and the learned FSAM ensembles, which operate in a restricted function class on finite noisy labels; the paper does not establish that the theoretical guarantees transfer to the learned models. The empirical claim is also currently demonstrated only against a theoretical lower bound rather than against actual alternative training procedures.","major_comments":[{"comment":"Theorem 2 constructs an ensemble by sampling uniformly from the annotators' true reward functions r_{θ_i}, relying on the Section 2 assumption that each annotator is reward-inducible within Θ. The FSAM procedure in Section 4 instead trains reward models on aggregated, noisy soft labels p in a restricted linear-head class over frozen Llama-3-8B embeddings. Nothing in Theorem 2 or Theorem 4 shows that this learned class contains a low-calibration-error ensemble, so the paper's central theoretical guarantee does not transfer to the empirically evaluated method. The assumption is stated, but its failure mode is neither analyzed nor tested; this is a load-bearing gap between the two contributions.","section":"§3.1, Theorem 2 and §4"},{"comment":"Theorem 4 is conditional on F having finite VC dimension d, but the manuscript does not verify this for the actual models, and the running example of linear heads on Llama-3-8B embeddings gives d on the order of the embedding dimension (roughly 4096). For the smaller datasets (e.g., Reddit TL;DR with N=3,217), d' = 20(d+1)k log(2(d+1)k) exceeds N, so the bound's log(eN/d') term is negative and the guarantee is vacuous. The theorem is internally correct as a conditional statement, but the claim that pairwise calibration can be learned with a limited number of comparisons is not established for the models and dataset sizes used in Section 5.","section":"§3.3, Theorem 4"},{"comment":"The main empirical claim is that ensembles achieve lower held-out calibration MSE than an optimal single deterministic reward, but the comparison is only against the theoretical lower bound Σ_i min{p_i², (1-p_i)²}, not against an actual single reward model trained with the same soft-label MSE objective or against existing pluralistic baselines. Since FSAM directly optimizes the evaluation metric on training data, reporting a gain over a mathematical floor does not show that the procedure outperforms a standard soft-label-trained single model on held-out data. The experiments also report no error bars, confidence intervals, or multiple seeds, so the stability of the gains across dataset splits is unknown.","section":"§5, Figure 2"},{"comment":"The empirical evaluation defines the target p as the observed annotator fraction, and the held-out MSE is computed against the same noisy p, not against the true population fraction p*. As Lemma 1 correctly notes, the empirical loss is biased by an irreducible label noise term C. Consequently, the reported numbers are not estimates of the true calibration error L(r), and the gap to p* is not quantified. This is not a fatal flaw, but it should be explicitly acknowledged when interpreting Figure 2, and ideally the paper should bound or estimate the magnitude of C for the datasets used.","section":"§2 and §5"}],"minor_comments":[{"comment":"In the proof, the text states that k ≥ 1/(4ε) implies ε ≤ 1/(4k), but the implication is reversed; the correct conclusion is ε ≥ 1/(4k), which is the inequality actually needed for the argument.","section":"§3.1, proof of Theorem 2"},{"comment":"The comparison function class F is defined with a non-strict inequality 1[r_θ(x,y₁) ≥ r_θ(x,y₂)], whereas Definition 1 and the rest of the paper use strict inequalities 1[r_θ(x,y₁) > r_θ(x,y₂)]. The tie case should be clarified for consistency.","section":"§3.3, Theorem 4 statement"},{"comment":"The caption refers to a shaded band marking the single-reward floor, but in the displayed figure the floor appears as a solid horizontal line; the visual encoding should be made consistent.","section":"§5, Figure 2"},{"comment":"The dataset description for PersonalLLM notes that annotations come from multiple reward models rather than humans; this is appropriate for a stress test, but the paper should state more prominently in Section 5 that one of the four empirical results is based on synthetic model-based annotators, not human annotators.","section":"§F.2, Table 2"},{"comment":"The Kendall–τ correlation is computed over only 50 prompts and 100 continuations per prompt; this is a small sample for the diversity claim, and the caption should report standard errors or another measure of variability.","section":"§5, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's theoretical core is plausible and the exposition is generally careful, but the connection between the existence theorems and the FSAM heuristic is not established, and the empirical section needs stronger baselines and uncertainty quantification. I do not see an irreparable flaw; the authors could address the main gap by reframing the theoretical results as existence guarantees for an idealized setting, adding experiments with a soft-label-trained single reward baseline and standard pluralistic baselines, and reporting multiple seeds and confidence intervals. The VC-dimension concern in Theorem 4 should be addressed by either verifying the condition for the actual models or stating clearly that the generalization bound is illustrative rather than applicable to the reported experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on Halpern et al. The interesting thing is the formal criterion itself: pairwise calibration, plus the existence results. Theorem 2's O(1/ε) bound via random sampling is simple and clean, and Theorem 3's outlier pruning is a nice argument that you can remove extreme rewards without destroying calibration. The reduction from Minimum Feedback Arc Set for Theorem 1 is also reasonable. Those results are genuinely new to me, and the paper writes them up honestly.\n\nThe practical FSAM heuristic is where I get more cautious. The training objective is the calibration MSE, so it's not surprising that an ensemble trained to minimize that MSE beats a deterministic baseline on held-out points of the same distribution. The more serious issue, which the stress-test note identifies correctly, is that Theorem 2's construction samples the true annotator reward functions, while FSAM learns from noisy aggregated soft labels in a restricted model class. Nothing ensures the learned rewards are the true ones or that the true ones are even in the class. Theorem 4 gives a uniform-convergence bound on the biased loss, but it doesn't show that the function class contains a low-error ensemble. So the existence guarantee does not transfer to the learned ensembles. The paper is upfront about the reward-inducibility assumption in Section 2, but it never tests or analyzes its failure. That's a real gap, though not a fatal one.\n\nThe empirical section is also thinner than I'd like: no error bars, no multiple seeds, no comparison with other pluralistic methods, and no code. The Kendall-tau diversity plots are suggestive but not much more. The discussion of policy-level calibration is honest about the binary indicator being a proxy.\n\nAll that said, I think the paper is a solid contribution to the pluralistic alignment subfield. The formal criterion is worth having, and the existence results are a useful reference point even if the heuristic is not tightly connected. A serious referee should engage with this; the main fixes are empirical rigor and an explicit discussion of when the assumption holds. I'd bring it to the reading group and would probably cite the calibration notion.","headline":"A clean formal criterion for pluralistic rewards with real existence results, but the theoretical guarantee doesn't transfer to the learned heuristic and the empirical evaluation is underpowered.","tokens_in":28209,"tokens_out":2198,"would_cite":true,"duration_ms":23779,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairwise-calibrated ensembles of reward functions—trained on soft preference labels and needing no annotator identities—can faithfully represent the spread of human preferences, and only O(1/ε) rewards are needed for ε-approximate…","keywords":["pluralistic alignment","pairwise calibration","reward ensembles","soft labels","preference diversity","RLHF","forward stagewise additive modeling","outlier pruning"],"falsifier":"Synthesize a population with annotators whose pairwise preferences are cyclic (A≻B, B≻C, C≻A) so that no scalar reward function can induce them; if FSAM with growing k still reaches ε-pairwise calibration on held-out triples, then the reward-inducibility assumption is not necessary, while if calibration error stays at the single-reward floor, the assumption is load-bearing.","tokens_in":27309,"feed_emoji":"🎭","tokens_out":9233,"duration_ms":91010,"temperature":0.7,"pith_summary":"Standard alignment trains one reward function from pooled human preferences, so disagreements collapse into a majority vote and minority perspectives are lost. This paper replaces the single reward with a small ensemble of reward functions whose collective pairwise choices are required to match the observed fraction of annotators preferring each response—a property it calls pairwise calibration. The paper proves that an O(1/ε)-sized ensemble can achieve ε calibration error, that the needed rewards need not include extreme outliers, and that finite data suffice to learn such ensembles. It then introduces a stagewise heuristic that trains each new reward on the residual calibration error of the current ensemble, and shows on four preference datasets that two to four such rewards already beat the best single deterministic reward on held-out calibration error. If the approach holds, pluralistic alignment can preserve minority viewpoints and response diversity without demographic grouping or annotator identities.","feed_headline":"Even two to four reward models beat any single majority-vote reward.","feed_subtitle":"Training a small ensemble on soft preference labels preserves minority viewpoints and cuts calibration error.","key_machinery":"The load-bearing object is the pairwise calibration error L(r) = E[(p̂_r(x,y1,y2) − p*(x,y1,y2))²], where p̂_r is the weighted fraction of ensemble rewards preferring y1 over y2 and p* is the annotator fraction. The existence argument samples O(1/ε) annotator reward functions uniformly, the hardness argument reduces Minimum Feedback Arc Set to membership in the linear ordering polytope, the outlier argument uses a Markov step and triangle inequalities on the disagreement score Φ(θ), and the generalization argument bounds the pseudo-dimension of the k-ensemble loss class. The practical machinery is forward stagewise additive modeling: at iteration j, train r_θj to minimize the squared residual εj = p − p̂_{r_{j−1}} using a sigmoid proxy for the vote indicator, then re-optimize the weights α1,...,αj. Lemma 1—which decomposes empirical loss as population loss plus the binomial variance of n annotator votes—is what makes soft-label training a sound estimation strategy despite the bias.","core_discovery":"The central claim is that the correct object for pluralistic alignment is not a single reward but a weighted mixture of reward functions that is pairwise calibrated: for every pair of responses, the mixture's probability of preferring one response equals the fraction of annotators who prefer it. Pairwise calibration is defined through the mean squared error between the ensemble's vote share and the observed soft-label frequency p*(x, y1, y2), so annotator disagreement is treated as signal rather than noise. The paper proves existence via the probabilistic method—sampling about 1/ε annotator reward functions uniformly yields expected squared error at most 1/(4k)—and shows that finding a perfectly calibrated ensemble is NP-hard, which motivates approximation. It also proves that the total weight of (β, γ)-outlier rewards is at most 1/(β−1), so nearly calibrated outlier-free ensembles exist, and that empirical loss on soft labels estimates the population loss up to an irreducible binomial noise term with a finite VC-dimension bound. The practical FSAM procedure fits each new reward model to the current residual and re-optimizes mixture weights; on four datasets, ensembles of size two to four achieve lower held-out mean-squared calibration error than the optimal single deterministic majority-vote reward, and the learned rewards are measurably diverse.","pith_inferences":["A natural extension beyond the paper is to test the theory on synthetic mixtures of known reward functions inside the model class, checking whether FSAM recovers pairwise calibration at the predicted O(1/ε) rate and whether outlier pruning matches the theorem's weight bound.","Because calibration is defined on pairs only, higher-order judgments over sets of responses are information-theoretically unrecoverable from pairwise data alone; soliciting best-of-ℓ rankings could constrain the ensemble and strengthen pluralistic fidelity.","The empirical gains are shown with small base models and modest ensemble size; scaling to larger base models or larger k would reveal whether the calibration gap persists, saturates, or inverts.","In deployment, sampling a policy from the ensemble in proportion to its weight may reduce preference collapse over many queries; this effect could be measured directly against a single-reward baseline.","If annotator rewards lie outside the chosen reward class, the theoretical guarantees do not directly apply; a falsifiable probe is to corrupt a known reward mixture with non-transitive or context-dependent preferences and measure how calibration degrades with k."],"forward_implications":["A small ensemble of size O(1/ε) is enough to approximate any pairwise preference distribution, so faithful pluralism does not require one reward per annotator.","Nearly calibrated ensembles can be pruned of extreme outlier reward functions: their total weight is bounded by 1/(β−1), and removing them costs at most (√ε + 1/(β−1))² in squared calibration error.","Learning from finite soft-label data is statistically sound: minimizing empirical MSE estimates the population calibration loss up to an irreducible binomial noise term, with sample complexity O(k d log(k d)).","The FSAM heuristic produces ensembles whose held-out calibration error drops below the single-reward majority floor within two to four iterations on the tested datasets, and the component rewards are not duplicates of each other.","Deployment can follow balanced, steerable, or distributional modes, so a pairwise-calibrated ensemble supports both individualized outputs and population-level response diversity."],"supporting_citations":[{"why":"Grounds the pairwise-preference model whose single-reward aggregation the paper replaces with an ensemble.","marker":"[2]"},{"why":"Supplies the balanced, steerable, and distributional deployment modes for policy ensembles.","marker":"[23]"},{"why":"Provides the forward stagewise additive modeling framework behind the residual-fitting heuristic.","marker":"[44]"},{"why":"Provides one of the four multi-annotator preference datasets used in the empirical evaluation.","marker":"[59]"},{"why":"Provides a large multi-annotator dataset with model-based preference scores used in the empirical evaluation.","marker":"[60]"},{"why":"Provides a human-annotated helpfulness preference dataset used in the empirical evaluation.","marker":"[61]"},{"why":"Provides a multi-annotator summarization preference dataset used in the empirical evaluation.","marker":"[62]"},{"why":"Guarantees existence of a Θ(m²)-sized perfectly calibrated ensemble, the starting point for the small-support results.","marker":"[71]"},{"why":"Gives the NP-hard Minimum Feedback Arc Set problem used in the hardness reduction for finding perfectly calibrated ensembles.","marker":"[72]"},{"why":"Provides the membership-oracle convex optimization theorem used to derive the hardness reduction.","marker":"[73]"}],"fun_headline_variants":["Small reward ensembles beat any single majority-vote reward","Even 2-4 reward models capture diverse preferences","Pairwise calibration with tiny reward mixtures","Minority views survive with small reward sets","Soft labels from disagreements shrink calibration error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantees assume every annotator's preferences are exactly captured by some reward function in the chosen class; if real preferences fall outside that class, the theorems about existence and outlier pruning no longer apply to learned ensembles.","fun_headline_variants_meta":{"raw":{"variants":["Small reward ensembles beat any single majority-vote reward","Even 2-4 reward models capture diverse preferences","Pairwise calibration with tiny reward mixtures","Minority views survive with small reward sets","Soft labels from disagreements shrink calibration error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2867,"prompt_tokens":950,"completion_tokens":1917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":566,"tokens_out":1917,"duration_ms":12614,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:41:08.841498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Synthesize a population with annotators whose pairwise preferences are cyclic (A≻B, B≻C, C≻A) so that no scalar reward function can induce them; if FSAM with growing k still reaches ε-pairwise calibration on held-out triples, then the reward-inducibility assumption is not necessary, while if calibration error stays at the single-reward floor, the assumption is load-bearing.","supporting_citations":[{"cited_title":"Friedman.The Elements of Statistical Learning: Data Mining, Inference, and Prediction","cited_arxiv_id":null,"evidence_quote":"Provides the forward stagewise additive modeling framework behind the residual-fitting heuristic."},{"cited_title":"Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano","cited_arxiv_id":null,"evidence_quote":"Provides a multi-annotator summarization preference dataset used in the empirical evaluation."},{"cited_title":"¨Uber den variabilit ¨atsbereich der fourier’schen konstanten von positiven harmonischen funktionen.Rendiconti Del Circolo Matematico di Palermo (1884- 1940), 32(1):193–217, 1911","cited_arxiv_id":null,"evidence_quote":"Guarantees existence of a Θ(m²)-sized perfectly calibrated ensemble, the starting point for the small-support results."},{"cited_title":"On the computational complexity of combinatorial problems.Networks, 5(1): 45–68, 1975","cited_arxiv_id":null,"evidence_quote":"Gives the NP-hard Minimum Feedback Arc Set problem used in the hardness reduction for finding perfectly calibrated ensembles."},{"cited_title":"Springer, 1988","cited_arxiv_id":null,"evidence_quote":"Provides the membership-oracle convex optimization theorem used to derive the hardness reduction."}],"review_version":1}