{"id":"f018bde6-6820-4e5c-b4f7-009d7cc8f820","arxiv_id":"2505.10597","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Two co-trained reward models that filter each other's training batches via peer review and synchronized curriculum learning improve reward-model accuracy and downstream policy win-rates under injected preference noise.","lead":"This paper trains two reward models that review each other's data selections during training, aiming to filter out noisy preference labels before they degrade downstream LLM alignment. The proposed method, called CRM, reports up to 9.94 points improvement on RewardBench under 40 percent injected label noise, and it also extends to DPO.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best 40%-noise result depends on setting λ_t from the exact noise rate used to inject noise; the paper's stated robust λ range [0.7,0.9] excludes λ=0.6, so the headline gain may be an oracle-conditioned artifact.","rationale":"The reader's weakest assumption is the right one, and the paper contains an additional internal signal that sharpens it. In Sec. 4.3 the authors say competitive results are obtained for λ_t ∈ [0.7, 0.9], but the default they adopt is λ_t = 1−η, which for the headline 40% noise experiment is 0.6; the exact setting used for the 'up to 9.94' claim is therefore outside the range they describe as robust. This makes the headline number look tied to known η rather than robust to unknown noise. The filtering hypothesis is plausible, and the ablations (w/o PR, w/o CL) plus the 0%-noise HH-RLHF OOD gain provide real supporting evidence, so I do not think the paper should be rejected. However, the controlled 20%/40% experiments use the injected η to set λ_t, and the paper never shows sensitivity at λ_t = 0.6 explicitly or provides error bars that would establish stability of the 9.94-point gain. A fixed-λ or validation-estimated-η rerun would settle whether the benefit comes from the margin filter or from oracle knowledge of η. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":27428,"tokens_out":6947,"duration_ms":71186,"concrete_test":"Re-run the Table 1 and Table 7 conditions with λ_t fixed at 0.8, and separately with λ_t estimated from a held-out clean validation set instead of λ_t = 1−η, reporting RewardBench overall and per-category (Chat, Chat Hard, Safety, Reasoning) across three random seeds. If the 40%-noise HH-RLHF gain over Standard RM shrinks to within seed-to-seed spread, or is concentrated in Reasoning while Safety and Chat Hard regress, then the headline improvement is an artifact of oracle η and category aggregation rather than of margin-based peer filtering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—up to a 9.94-point RewardBench gain at 40% noise—is produced under CRM's default selection ratio λ_t = 1−η (Sec. 4.3), which is λ_t = 0.6 when the experimenter knows η = 0.4. The whole filtering mechanism (Eqs. 6–7) keeps only the top λ_t fraction of each batch by peer-model margin M = σ(r_w − r_l). For the headline result to transfer to real settings, this margin must rank noise content correctly and η must be known or accurately estimated. The paper does not establish either condition. Sec. 3.1's motivation is an aggregate correlation on HH-RLHF only; the loss-separation plots in Sec. 4.3 and Fig. 6 are computed after training, on data the filter has already shaped, so they cannot validate the online selection rule at the moment it is used. Moreover, the text states that competitive results are obtained for λ_t ∈ [0.7, 0.9], yet the default for the headline 40% experiment is λ_t = 0.6, outside that range; no sensitivity evidence is shown for that exact setting. Without error bars, it is also unclear whether the 9.94-point gain is stable across training runs. The central claim is therefore conditional on oracle knowledge of the noise rate and on a margin-ranking assumption that is not directly validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Collaborative Reward Modeling (CRM), an online co-training framework in which two reward models filter each other's training batches by keeping the top lambda_t fraction of preference pairs ranked by the peer model's reward margin sigma(r_w - r_l), combined with an epoch-level curriculum that sorts data by margin. The authors motivate this with a three-way categorization of preference instances (robust, ambiguous, non-robust) and report experiments on HH-RLHF, Ultrafeedback, and Skywork-Reward with synthetic label flipping at 0%, 20%, and 40%, evaluating in-domain preference accuracy, RewardBench, RMB, RM-Bench, and downstream RLHF/DPO win-rates judged by GPT-4o. The headline claim is up to a 9.94-point RewardBench improvement under 40% noise, together with a claimed seamless extension to DPO.","tokens_in":27687,"tokens_out":7295,"duration_ms":72939,"significance":"If the empirical claims hold, CRM is a practically attractive and loss-function-agnostic data-filtering wrapper: it leaves the Bradley-Terry objective unchanged, applies to both explicit and implicit reward training, and its peer-review mechanism is a plausible remedy for the confirmation bias that afflicts self-filtering. The paper's strengths include extensive benchmarking across three training sets and several reward benchmarks, ablations of peer review and curriculum learning, weak-to-strong collaboration experiments, and an explicit limitations section. However, the absence of error bars, an oracle-conditioned default selection ratio, and an unvalidated margin-ranking assumption currently leave the central quantitative claims uncertain; the contribution is promising but not yet established at the level the abstract claims.","major_comments":[{"comment":"The headline 9.94-point RewardBench improvement at 40% noise (Sec. 4.2, Table 7) is obtained under the default selection ratio lambda_t = 1 - eta = 0.6, because eta is the injected noise rate. The paper states that competitive results are obtained for lambda_t in [0.7, 0.9], which excludes 0.6, and no sensitivity result is shown at lambda_t = 0.6. Since eta is known only in the synthetic-noise experiments, the default setting is oracle-conditioned, and it is unclear how lambda_t would be set for real data. Please report a full lambda_t sweep with variance at each noise level and provide either a practical estimator of the noise rate or a misspecification analysis showing that CRM is robust to imperfect eta.","section":"Sec. 4.3, Fig. 5, and Eqs. (6)-(7)"},{"comment":"No error bars, confidence intervals, or significance tests are reported, yet many of the claimed improvements are small: for example, in Table 10 at 20% noise on RMB-Harmless, CRM (65.13) is below cDPO-RM (65.88), and in Table 3 at 0% noise on TL;DR, CRM (52%) is only one point above DPO (51%). The win-rate evaluation uses only 300 prompts with GPT-4o as judge, so the reported margins are likely within judge and prompt sampling noise. In addition, the 40%-flipped Standard RM row in Table 8 (90.50, 48.46, 82.45, 63.62, 66.53) is identical to the 0%-flipped Standard RM row in Table 7, which appears to be a copy error. Please correct the table and report variability over training seeds and judge runs for all main results.","section":"Sec. 4.2, Tables 1, 7-10; Sec. C.4"},{"comment":"The central assumption that the peer margin M = sigma(r_w - r_l) orders preference pairs by noise content is not validated at the moment of selection. The Sec. 3.1 analysis is an aggregate correlation on one dataset (HH-RLHF) with one 3B model, and the loss-separation evidence in Sec. 4.3 and Fig. 6 is computed after training, on data that the filter has already shaped; it therefore cannot validate the online selection rule. Please report the precision and recall of the margin filter against known flipped labels during training (e.g., at early, middle, and late epochs), and the fraction of clean pairs dropped at each lambda_t. Without this, Eqs. (6)-(7) could be discarding clean data while retaining noisy data.","section":"Sec. 3.1 and Eq. (5)"},{"comment":"The claimed seamless extension to DPO is not reproducible from the paper. Algorithm 1 is written for explicit reward models, while the DPO section states only that the framework is 'suited to existing DPO' without specifying how the two policies are initialized, how curriculum sorting is performed, or how peer review selects batches in the implicit-reward setting. Table 6 reports CRM-Model 1 and CRM-Model 2, but Table 3 reports a single CRM result; it is unclear whether Table 3 uses one of the two models and whether the two models are synchronized in the same way as the explicit-RM case. Please provide the DPO analogue of Algorithm 1 and clarify which model's outputs are reported.","section":"Sec. 4.3, Algorithm 1, and Table 6"}],"minor_comments":[{"comment":"The text says CRM 'consistently achieves superior performance,' but Table 10 contains counterexamples (e.g., 20% RMB-Harmless, where cDPO-RM outperforms CRM); please soften the claim to match the data.","section":"Sec. 4.2, text after Table 1"},{"comment":"The Skywork-Reward ID column is a dash for every method, yet the Average column appears to be computed over the remaining five columns; please clarify what ID means for Skywork-Reward and how the average is defined.","section":"Table 1"},{"comment":"Figure 2 is a schematic without axes labels or quantitative thresholds; please state which coordinates are plotted and how the boundaries between robust, ambiguous, and non-robust preferences are set.","section":"Fig. 2"},{"comment":"The curriculum sort uses M for r_{phi,psi}, which is ambiguous; please specify whether the sort key is the margin of the current model, the peer model, or an aggregate of both.","section":"Algorithm 1, line 6"},{"comment":"The abbreviation 'w/ SR' in Fig. 5 is not defined in the caption or the text; the surrounding text mentions a 'Self Review' variant, so please make the notation consistent.","section":"Fig. 5 and Sec. 4.3"},{"comment":"The checklist answers 'Yes' to the statistical-significance question, but the paper does not contain error bars or significance tests; please correct the checklist answer or add the missing statistics.","section":"NeurIPS Checklist, item 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent empirical study with a plausible mechanism, but the headline result rests on a single configuration whose default selection ratio lies outside the stated robust range, and the benchmark tables currently contain a suspected copy error. I recommend requesting a revision that corrects Table 8, adds repeated-run variability, provides an online validation of the margin filter, and clarifies the DPO extension. These issues are addressable within the manuscript's scope, so I do not see grounds for rejection if they are fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real method, not a loss tweak. Two RMs filter each other's batches by reward margin, curriculum sorting orders the data, and the same idea extends to DPO. The results across RewardBench, three training sets, and the PPO/DPO win-rate tables are consistent, and the self-review ablation is the right control. I believe the mechanism does something.\n\nWhat's new: existing robust-preference work (cDPO, rDPO, ROPO) changes the loss; this paper changes the data stream online and shows peer-review selection beats self-review. The weak-strong collaboration result is a nice extra.\n\nSoft spots, in order:\n1. The headline 9.94-point RewardBench gain at 40% noise uses λ_t = 1−η = 0.6, while the paper says λ_t in [0.7, 0.9] gives competitive results. So the marquee number is produced at a setting outside the range they claim is robust, and that setting is exactly the oracle noise rate. They need sensitivity curves for λ = 0.6 and, ideally, an estimator for η.\n2. No error bars anywhere. Tables 1 and 2 have several improvements of 2–5 points; with single runs and a 300-prompt GPT-4o judge, I cannot tell which margins are stable. A few standard deviations across seeds would settle it.\n3. The margin used for filtering is the same signal being trained, so the circularity concern is real. The self-review ablation helps, but the online claim would be stronger if they validated the margin's noise-ranking ability on held-out noisy pairs before training shapes it.\n4. No code or data release until acceptance. The method is simple enough to re-implement, but the exact curriculum sorting and λ schedule matter.\n\nNone of these sink the paper. The central empirical claim—peer-reviewed batch filtering improves robustness—is credible, and the DPO extension makes it useful beyond RM training. The citation pattern is fine, and the limitations section honestly says only a 3B model was used.\n\nWho this is for: people working on RLHF data quality and robust preference optimization. It deserves a serious referee; I would send it out. The revisions I would require are error bars, a λ sensitivity plot that includes 0.6, and either an η estimator or an honest positioning of the 40% result as an oracle-conditioned upper bound.","headline":"A genuinely new online peer-review filtering scheme for noisy preference data; the headline 40%-noise gain depends on knowing the noise rate, so treat the strongest number with caution.","tokens_in":28306,"tokens_out":2056,"would_cite":true,"duration_ms":20426,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-15T21:14:03.689391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}