{"id":"afaa0a4a-4cdc-42b5-9315-62fcd37dbbc0","arxiv_id":"2608.10126","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"In a synthetic preference-learning setup, PA-RLHF is reported to beat a single averaged reward model on group-level alignment, but the experiment appears to use ground-truth group labels to select the reward model, undermining the comparison.","lead":"This paper claims that standard RLHF, by averaging heterogeneous human preferences into one reward model, structurally under-serves minority preference groups, and proposes PA-RLHF, which clusters preferences and trains a separate reward model per cluster. A generalist might read it for a fairness critique of large-scale AI alignment, but the supporting evidence is a small synthetic simulation with an evaluation protocol that appears to hand the proposed method the answer.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 21-point PA-RLHF gain is likely an oracle upper bound: test items are routed to per-mode reward models using ground-truth group labels, not inferred modes.","rationale":"The paper's goal is diagnostic: to isolate whether preference averaging in RLHF causes procedural unfairness and whether per-mode reward models remedy it. The controlled simulation is a reasonable vehicle for this conceptual claim. However, the empirical demonstration must validate PA-RLHF as a pipeline, not an oracle-influenced upper bound. The phrase 'Ground-truth group assignments are used exclusively for evaluation' is ambiguous: it might only mean that group labels are used when computing disaggregated accuracy, but it does not explain how the 'corresponding preference-specific reward model' is selected for each test item. The simplest implementation, and the one suggested by the wording, is to use the known group of the test rater to pick the model. With ARI=0.443, predicted modes would often disagree with ground truth, so a realistic routing would misassign many comparisons. Because the reported gain is large, the oracle concern directly undermines the main quantitative result. Secondary issues, such as the 'noise-free' label despite 80–85% consistency and the absence of error bars, reinforce the need to treat the numbers skeptically. The reader's verdict of REJECT is appropriate; this stress-test does not move it, though a clean end-to-end re-evaluation could change the picture.","tokens_in":3801,"tokens_out":4932,"duration_ms":50741,"concrete_test":"Re-run the Section 3 evaluation with an end-to-end PA-RLHF: after training per-mode reward models, assign each test comparison to a mode by nearest cluster centroid in the same 385-d user-feature space (or via a mode classifier trained on the training clusters). Use no ground-truth group labels during scoring. Report overall and per-group accuracy. If the overall accuracy drops below the 67.9% reported value (or the fairness-gap reduction becomes negligible), the central claim depends on the oracle assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that PA-RLHF improves alignment accuracy from 46.9% to 67.9% by separating reward learning across preference modes. The experiment as described cannot support this as an end-to-end claim, because the test-time routing step is not specified. The paper states 'Ground-truth group assignments are used exclusively for evaluation' (Section 2.1) and then 'scoring candidate responses using the corresponding preference-specific reward model' (Section 2.2). The only natural reading is that the ground-truth group label of each test rater is used to select which of the three reward models scores that rater's comparisons. That is an oracle: a deployed PA-RLHF would have to infer the mode from behavior, and the paper's own clustering achieves only ARI=0.443 with silhouette=0.199, implying substantial mode-assignment error. If a fraction of test items are routed to the wrong reward model, the reported +21.0 pp overall improvement and the 40% fairness-gap reduction are upper bounds, not achieved performance. The paper also describes the setting as 'noise-free' while generating preferences with 80–85% within-group consistency, and reports no error bars or significance tests; these issues compound the oracle problem. The conceptual argument about preference averaging is plausible, but the quantitative demonstration does not establish that PA-RLHF works without ground-truth mode information.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard RLHF's practice of aggregating heterogeneous preferences into a single reward model constitutes a procedural fairness failure: majority preferences receive disproportionate optimization pressure while minority preferences are systematically under-aligned. The authors introduce Preference-Aware RLHF (PA-RLHF), which clusters preference comparisons into k=3 modes and trains a separate reward model per mode, and they evaluate it on a simulated dataset of 971 pairwise comparisons from 60 raters with three known preference groups. The headline results are an improvement in overall alignment accuracy from 46.9% to 67.9% and a reduction in the best-worst fairness gap from 15.9 to 9.6 percentage points. The paper positions this as evidence that the reward-learning objective itself, not downstream deployment, is the source of minority-preference under-alignment.","tokens_in":4046,"tokens_out":5018,"duration_ms":50408,"significance":"The conceptual point is valuable: separating the reward-learning objective across preference modes is a plausible structural remedy for aggregation-induced unfairness, and the paper offers a clean diagnostic framing with a clear fairness metric. If the empirical claims were supported end-to-end, this would be a useful contribution to the alignment-fairness literature. However, the central quantitative demonstration is compromised by an oracle evaluation protocol that appears to use ground-truth preference-group labels to select the per-mode reward model at test time. The paper also does not ship code or provide statistical uncertainty estimates, so the strength of the evidence is considerably weaker than the abstract suggests.","major_comments":[{"comment":"The evaluation protocol appears to use ground-truth group labels for test-time model selection. Section 2.1 states that 'Ground-truth group assignments are used exclusively for evaluation,' and Section 2.2 states that alignment is evaluated by 'scoring candidate responses using the corresponding preference-specific reward model.' The natural reading is that the ground-truth label of each test rater selects which of the k=3 reward models scores that rater's comparisons. This is an oracle step: a deployed PA-RLHF system would have to infer the mode from behavior, and the paper's own K-Means clustering achieves only ARI=0.443, implying substantial misassignment. The +21.0 pp overall improvement and the 6.3 pp gap reduction in Table 1 therefore describe an oracle-assisted variant, not PA-RLHF as an end-to-end method. Please specify exactly how test items were routed; if ground-truth labels were used, rerun the evaluation with the clustering's predicted labels, or explicitly reframe the reported numbers as oracle upper bounds.","section":"Section 2.2"},{"comment":"The abstract calls the setting 'noise-free,' but Section 2.1 assigns within-group consistency rates of 85%, 85%, and 80%, meaning that 15-20% of each simulated rater's labels deviate from the group's stated preference profile. This is label noise, and it contradicts the 'noise-free' characterization. Please correct the wording and discuss whether the reported fairness-gap reduction is robust to this noise level, since the separation between groups is intentionally imperfect.","section":"Section 2.1 / Abstract"},{"comment":"All results are single-run point estimates from one simulated dataset of 971 comparisons, with no confidence intervals, bootstrap resampling, or significance tests. The test set is roughly one quarter of 60 raters, so the 6.3 pp gap reduction and the 21.0 pp overall improvement could be within sampling variability. Please report repeated simulations or at least bootstrap standard errors to support the quantitative claims.","section":"Table 1 / Section 3"},{"comment":"The sentence 'Clustering is used only to separate preference signals; the observed improvements arise from eliminating cross-group interference during reward learning, not from clustering itself' is a causal claim that the experiment does not support. To substantiate it, the authors would need to compare PA-RLHF with predicted clusters against a model trained on ground-truth group separation and against a model trained on random clusters, so that the effect of clustering quality can be separated from the effect of splitting optimization. As written, the design only shows that some separation helps when the test-time routing is oracle-based.","section":"Section 2.2"}],"minor_comments":[{"comment":"Table 1 labels Mode 1 as 'Majority,' but the experiment uses 20 raters per group with similar numbers of comparisons per rater; there is no majority group by prevalence. Please clarify what makes this mode the majority or change the label.","section":"Table 1"},{"comment":"The text says k=3 was selected via a silhouette sweep over k=2-6 on held-out data but then says 'Fixing the number of modes isolates procedural effects.' Please clarify whether k was tuned or fixed a priori, and report the silhouette values for all k considered.","section":"Section 2.2"},{"comment":"The paper does not report the regularization strength or other hyperparameters of the Logistic Regression reward model; please add this information for reproducibility.","section":"Section 2.2"},{"comment":"The title in the PDF rendering lacks spaces between words ('PROCEDURALFAIRNESSFAILURES'); please ensure the camera-ready version formats the title correctly.","section":"Title"},{"comment":"The reference 'Alabi & Wick (2024)' appears to have incorrect author names; please verify the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The oracle evaluation issue is the central problem: if the authors cannot supply non-oracle results (or explicitly relabel the claims as oracle upper bounds), the paper should not be accepted. The conceptual framing is worth preserving, and the experiment is likely fixable within the manuscript's scope by rerunning evaluation with predicted cluster assignments. I would also encourage the authors to add uncertainty quantification, since the current point estimates are not convincing on their own."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the conceptual point—averaging heterogeneous preferences in RLHF can let majority preferences dominate and systematically under-align minority groups—is plausible and worth stating. Second, the main quantitative claim (46.9% to 67.9%) does not hold as an end-to-end result, because the evaluation routes each test item to a per-mode reward model using ground-truth group labels. That makes the 21-point gain an oracle upper bound.\n\nWhat is actually new: the paper packages this as \"procedural fairness in alignment,\" and the controlled simulation (60 raters, three known preference profiles) is a clean way to isolate aggregation effects from other confounds. They are also transparent about the ground-truth information in the methods, which is more than many papers would be. The failure mode itself is not new—pluralistic alignment and personalization work (Poddar et al., Gong et al.) already show that averaging heterogeneous preferences loses information—but the fairness framing is a useful label.\n\nThe soft spots are serious, though. The oracle routing is the load-bearing flaw. If a deployed system has to infer which preference mode a user belongs to, and the clustering only gets ARI=0.443, then a large fraction of test items would be scored by the wrong reward model. The paper reports no accuracy for that inference step, so we have no idea whether the benefits survive. The \"noise-free\" claim also conflicts with the setup: within-group consistency is only 80–85%, which is noisy by any definition. And with 971 comparisons, three groups, and no error bars or significance tests, the point estimates could easily be within chance.\n\nThere is a real idea here, but the demonstration is not there. It should be reframed as an oracle analysis (\"if you knew the modes, this is the upper bound\") or extended to include mode inference from user behavior. As it stands, the central claim is not supported.\n\nWho benefits? Someone working on pluralistic alignment or fairness audits might read the introduction and the framing, but they should not rely on the numbers. I would not cite the empirical results.\n\nIf I were the editor, I'd send it out to a careful reviewer—the flaw is clear and the fix is substantive, so a serious referee could help the authors turn it into an honest diagnostic study. But it should not be accepted in its current form.","headline":"The conceptual framing is plausible, but the headline number comes from an oracle: PA-RLHF is scored using ground-truth group labels to pick the reward model, so the 67.9% is an upper bound, not achieved performance.","tokens_in":4585,"tokens_out":3384,"would_cite":false,"duration_ms":33126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard RLHF's reward averaging systematically under-aligns minority preferences, and separating reward models by preference mode fixes most of the gap.","keywords":["RLHF","preference averaging","procedural fairness","pluralistic alignment","reward modeling","minority preferences","fairness gap"],"falsifier":"Re-run PA-RLHF with no oracle, scoring each test item using the reward model of the k-means cluster predicted for that item rather than its ground-truth preference group; if end-to-end alignment accuracy falls back toward the 46.9% baseline or the fairness gap widens from 9.6 points, the measured procedural gain comes from label availability rather than preference separation.","tokens_in":3575,"feed_emoji":"⚖️","tokens_out":6096,"duration_ms":52429,"temperature":0.7,"pith_summary":"The paper tries to establish that the standard RLHF recipe, which trains one reward model on pooled preference data, is itself a source of unfairness: when preferences genuinely differ across groups, averaging gives majority preferences more influence and systematically under-aligns minority preferences. It defines procedural fairness in alignment as preserving distinct preference signals during reward modeling, argues that RLHF violates this definition, and introduces Preference-Aware RLHF (PA-RLHF), which clusters comparisons into preference modes and trains a separate reward model per mode. In a controlled, noise-free setting with three simulated preference groups, PA-RLHF raises overall alignment accuracy from 46.9% to 67.9% and shrinks the gap between best- and worst-aligned groups from 15.9 to 9.6 percentage points. The work matters because it locates the fairness failure in the construction of the alignment objective itself, not in deployment or downstream application.","feed_headline":"Splitting RLHF reward models by preference mode cuts fairness gap 40%","feed_subtitle":"Per-mode reward models lift alignment accuracy from 46.9% to 67.9% and narrow the group fairness gap by 40%.","key_machinery":"The mechanism under test is the aggregation step in the reward-learning objective: pooling all pairwise preferences into a single Bradley-Terry-style reward model. PA-RLHF replaces this pooling with preference-mode separation: fixed sentence embeddings of prompt-response pairs are clustered with K-Means (k=3, chosen by silhouette score, ARI=0.443), and a separate logistic-regression reward model is trained on each cluster's comparisons. Alignment is then scored by selecting the reward model belonging to the scorer's true preference group, which isolates the procedural effect of keeping preference signals distinct during reward learning.","core_discovery":"The central claim is that preference averaging changes what RLHF optimizes: instead of learning each group's preference signal, it learns a frequency-weighted mixture, so a minority group that is internally consistent can still be under-aligned even with no annotation noise. Testing this in a controlled setting with 60 simulated raters in three preference groups, 971 pairwise comparisons, and 20 prompts, standard RLHF reaches only 46.9% overall alignment accuracy, with the majority group at 56.2% and the two minority groups at 41.2% and 40.3%. PA-RLHF, which clusters comparisons into k=3 modes and trains a separate logistic-regression reward model per mode, raises overall accuracy to 67.9%, lifts minority groups to 68.8% and 73.1%, and cuts the fairness gap from 15.9 to 9.6 percentage points. Because ground-truth preference-group labels are used only at evaluation time, the result isolates the effect of removing cross-group interference during reward learning.","pith_inferences":["A testable extension would run PA-RLHF end-to-end with no oracle, choosing each test item's reward model from its k-means cluster assignment rather than its ground-truth group; with ARI=0.443, the fairness gap may be larger than 9.6 points, revealing how much of the reported gain depends on label availability.","The paper's framing implies a broader corollary the authors leave implicit: any pluralistic alignment objective that optimizes a single scalarized reward, including some Direct Preference Optimization variants, should exhibit the same majority-weighted behavior, and the clustered-reward construction could be exported to those objectives.","Because the dataset is synthetic and noise-free, a natural next step is to test whether annotation noise washes out the procedural effect; the paper's definition of procedural fairness predicts that averaging harms minorities even in noisy settings, so the controlled result is a lower bound on the real-world failure."],"forward_implications":["If the claim is right, the single reward model in standard RLHF is not a neutral summary of preferences but a structural choice that shifts optimization pressure from minority to majority groups.","Fairness audits of alignment pipelines should inspect the reward-learning stage, not just model outputs, because the source of group-level misalignment can sit upstream in the objective construction.","In agentic systems, where reward models guide sequential decisions, the averaging failure would compound over time, making preference-mode separation in reward learning more consequential there.","The residual 9.6 percentage point gap after PA-RLHF shows that clustering-based separation reduces but does not eliminate group-level misalignment, motivating procedural refinements beyond mode separation.","Because the evaluation selects each group's reward model using ground-truth labels, the measured gain quantifies the procedural effect of mode separation, not the full cost of discovering preference modes in deployment."],"supporting_citations":[{"why":"Establishes the standard RLHF pipeline that fine-tunes language models from human preferences, the object the paper intervenes on.","marker":"Ziegler et al. (2019)"},{"why":"The canonical instruction-following RLHF recipe whose single-reward-model aggregation the paper identifies as the source of preference averaging.","marker":"Ouyang et al. (2022)"},{"why":"Represents the personalization approaches the paper distinguishes from procedural fairness in the objective.","marker":"Poddar et al. (2024)"},{"why":"Cited to argue that prior personalization work does not address how conflicting preferences are prioritized in the training objective.","marker":"Kirk et al. (2023)"},{"why":"Survey of personalized and pluralistic preference alignment, used to position the paper against output-level fairness evaluations.","marker":"Xie et al. (2025)"},{"why":"Cited as an example of fairness evaluated at outputs rather than at the alignment procedure.","marker":"Alabi & Wick (2024)"}],"fun_headline_variants":["Per-mode reward models slash RLHF fairness gap 40%","RLHF fairness fix: separate reward models per preference group","Preference-aware RLHF lifts alignment accuracy to 67.9%","Splitting reward models by preference mode improves RLHF fairness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on knowing each user's true preference group when selecting a reward model, but a deployed system would have to infer that group from behavior, and the paper's own clustering is only moderately accurate.","fun_headline_variants_meta":{"raw":{"variants":["Per-mode reward models slash RLHF fairness gap 40%","RLHF fairness fix: separate reward models per preference group","Preference-aware RLHF lifts alignment accuracy to 67.9%","Splitting reward models by preference mode improves RLHF fairness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":1970,"prompt_tokens":939,"completion_tokens":1031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":959}},"tokens_in":555,"tokens_out":1031,"duration_ms":8232,"temperature":1.0,"reasoning_tokens":959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:46.350457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run PA-RLHF with no oracle, scoring each test item using the reward model of the k-means cluster predicted for that item rather than its ground-truth preference group; if end-to-end alignment accuracy falls back toward the 46.9% baseline or the fairness gap widens from 9.6 points, the measured procedural gain comes from label availability rather than preference separation.","supporting_citations":[],"review_version":1}