{"id":"7fa6f507-7dd7-43b4-adc4-00e64aec45cd","arxiv_id":"2412.03187","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WRPO tunes an 8B chat model by combining its own preferred responses (on-policy) with high-reward responses from ten heterogeneous source LLMs (off-policy) using an annealed weight, beating prior fusion and preference-optimization baselines on AlpacaEval-2, Arena-Hard, and MT-Bench.","lead":"This paper introduces WRPO, a training method that improves a target language model by learning from responses generated by a diverse set of larger open-source models, without needing to align their vocabularies or merge their probability distributions. It reports consistent wins over prior fusion and preference-optimization baselines on three chatbot benchmarks, including a 55.9% length-controlled win rate against GPT-4 on AlpacaEval-2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward-model bias in cross-model preference labels threatens the central capability-transfer claim: ArmoRM ranks a dispreferred source response above the target's preferred response (Section F).","rationale":"The central claim is empirical: WRPO enables an 8B target to surpass much larger source models on instruction-following benchmarks. For this claim to hold, the preference labels that define 'better' responses must reflect real quality differences. The paper's own Section F provides direct evidence that ArmoRM scores are not cross-model comparable: the average dispreferred source response (0.158) outscores the average preferred target response (0.152). This is exactly the comparison regime used to construct y_ws versus y_wt/y_l. Since the WRPO objective upweights y_ws during training and the ablation attributes several LC points to y_ws, any bias in these labels propagates directly into the reported win rates. The concern is not that the authors are dishonest or that the experiments are fabricated; it is that the experimental outcome may be a measurement of the reward model's bias rather than of model fusion. A concrete re-labeling or oracle-agreement test can settle this. I agree with the reader that the verdict should remain conditional: the paper is otherwise well-executed, with controlled ablations and a comparison to relevant baselines, but the central claim should not be accepted unconditionally until the cross-model preference labels are validated.","tokens_in":26166,"tokens_out":13070,"duration_ms":110947,"concrete_test":"Take a random sample of 2,000 training prompts from UltraFeedback. Have an independent oracle (e.g., GPT-4-0125 as judge, or human annotators) rank the three responses y_ws, y_wt, y_l in each quadruple. Split the quadruples into two sets: those where the oracle's ordering matches ArmoRM's intended ordering (y_ws > y_wt > y_l) and those where it does not. Retrain Target-SFT-WRPO (same hyperparameters and alpha schedule) on each subset separately, and also on the full set, then evaluate on AlpacaEval-2 LC win rate. If the full-set result is substantially higher than the result trained on the 'matches' subset (or matches the 'mismatch' subset), the reported gains are driven by reward-model bias rather than by true preference signal. If the 'matches' subset retains the 55.9% level, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The preference data construction in §3.1 assumes ArmoRM's reward scores are comparable across model families: y_ws (best source response) is used as the preferred response over the target's y_wt and y_l. Section F's own numbers show this assumption is questionable: the mean ArmoRM score of source dispreferred responses y_ls is 0.158, while the target's preferred responses y_wt average only 0.152. Thus even a low-quality source output is scored higher than a high-quality target output, implying a systematic family-level offset. If this offset exists inside the preference quadruples, every y_ws label is contaminated: the model is being trained to increase the likelihood of responses that are 'preferred' only because of the reward model's bias toward source families. The main evaluation uses GPT-4 judges on AlpacaEval-2 and Arena-Hard; these judges may share the same stylistic and quality biases as ArmoRM. The central claim — that an 8B model absorbs capabilities from source LLMs and beats them on AlpacaEval-2 — would then be an artifact of fitting the reward/judge bias rather than genuine fusion. The paper's own ablation (Fig. 4) attributes a 3.5–5.2 point LC gain to y_ws, so this potential label contamination is directly load-bearing for the reported margins.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Weighted-Reward Preference Optimization (WRPO), an implicit model fusion method that transfers capabilities from heterogeneous open-source LLMs into a target model without vocabulary alignment or distribution merging. For each prompt, the authors construct preference quadruples (x, y_ws, y_wt, y_l) using ArmoRM scores, where y_ws is the highest-scoring source response and y_wt/y_l are the target model's best and worst responses, respectively. The WRPO loss is a DPO-like objective that interpolates between an on-policy margin (y_wt vs. y_l) and a hybrid-policy margin (y_ws vs. y_l), with a fusion coefficient α that increases linearly from 0 to 0.1 during training. With LLaMA3-8B-Instruct as the target and ten sources from 9B to 236B, the paper reports a 55.9% length-controlled win rate on AlpacaEval-2, a 46.2% win rate on Arena-Hard, and 7.63 on MT-Bench, exceeding the reported DPO, SimPO, IPO, FuseLLM, and FuseChat baselines. The authors include ablations, source-scaling studies, and an open-source code release.","tokens_in":26555,"tokens_out":13640,"duration_ms":126478,"significance":"If the empirical claims hold, WRPO is a genuinely useful and low-cost fusion recipe: an 8B target model approaches or beats much larger systems on two widely used instruction-following benchmarks while adding only about 16% training time over DPO and no inference-time ensemble cost. The paper deserves credit for a complete experimental package, including public code, explicit hyperparameters and tuning ranges, ablations of both terms in the loss, and scaling with the number of source models. The two strongest concerns from the review are real: Eq. (5)-(6) does not follow from the Bradley-Terry model as written, and Appendix F's own reward statistics indicate that the cross-model preference labels may be contaminated by a systematic reward-model offset. I agree with the reader's assessment that the method is not circular in the strong sense, because α is a fitted schedule rather than a conclusion derived from the data; the contribution is nonetheless not fully established until the reward-label assumption and the model-selection procedure are addressed.","major_comments":[{"comment":"The derivation of the WRPO objective is not a valid Bradley-Terry reformulation. In Eq. (4), p(y_w ≻ y_l | x) is written as σ(r(x, y_w) − r(x, y_l)), but y_w was defined immediately before as a pair of completions {y_ws, y_wt}, and Eq. (5) defines r(x, y_w) as a weighted average of rewards of two distinct completions. The Bradley-Terry model gives preference probabilities between realizable responses, and a weighted average of two different rewards is not the reward of any single completion. Equation (6) is therefore a new, ad hoc objective rather than a consequence of the Bradley-Terry model. Section H concedes that a more rigorous theoretical analysis is needed; the paper should either supply that analysis or explicitly present Eq. (6) as a regularized empirical loss inspired by DPO, not as a BT-derived objective. The empirical results may survive this reframing, but the current presentation overstates the theoretical grounding.","section":"3.3, Eq. (4)-(6)"},{"comment":"The preference-label construction assumes ArmoRM scores are comparable across model families, and the paper's own Appendix F shows they are not. The reported mean rewards are y_ws = 0.180, y_wt = 0.152, y_ls = 0.158, and y_lt = 0.132; that is, a dispreferred source response outscores the target model's preferred response on average. If this family-level offset is present inside the preference quadruples, then y_ws is not necessarily a higher-quality response than y_l; it is merely a response that receives a higher ArmoRM score. Because the main evaluations on AlpacaEval-2 and Arena-Hard use GPT-4 judges that may share the same stylistic biases as ArmoRM, the ablation gain of 5.2 points from y_ws in Fig. 4 could reflect reward/judge bias rather than capability transfer. The paper should provide a concrete control, such as human annotation on a subset, an alternative reward model for label construction, or evidence on objective-answer benchmarks like GSM8K and ARC, to show that the transfer goes beyond fitting the reward model's preferences.","section":"3.1, Appendix F, Fig. 4"},{"comment":"The choice of α is conflated with test-set evaluation. Appendix E describes a greedy search over α ∈ {0.1, 0.3, 0.5, 0.7, 0.9}, and Figure 5 reports the AlpacaEval-2 length-controlled win rate for those values. The final model then reports AlpacaEval-2 LC 55.9%, which is the same benchmark used for this search. As written, the headline number is therefore a model-selected value rather than an independent test result. Please state clearly which benchmarks were used for hyperparameter selection and provide a final evaluation on data or metrics that did not contribute to any model choice, so the reported AlpacaEval-2 and Arena-Hard numbers can be interpreted as test performance.","section":"4.2, Fig. 5, Appendix E"}],"minor_comments":[{"comment":"The expectation subscript in Eq. (8) has a trailing comma: (x, y_ws, y_wt, y_l,). Please correct this typo.","section":"Eq. (8)"},{"comment":"The caption of Fig. 3(c) reports α increasing from 0 to 0.5, while §4.1 states that the main configuration uses α increasing from 0 to 0.1. Please clarify whether Fig. 3 is an illustrative analysis run or the actual configuration used in Table 2, and explain the relation between the two settings.","section":"Fig. 3 caption"},{"comment":"The 'Percentage' column in Table 1 should state the denominator, for example the number of prompts for which that source produced the highest-scoring response; as written, the reader cannot tell whether the percentages refer to all selected y_ws responses or to some other base.","section":"Table 1"},{"comment":"The sentence comparing WRPO to 'the best-performing preference optimization baseline' by 2.2, 9.6, and 0.09 points should name the exact baselines (Target-SimPO for AlpacaEval-2 and Arena-Hard, Target-IPO for MT-Bench) and clarify that these are prior checkpoints, in contrast with the authors' own Target-SFT-DPO in the same table.","section":"4.2"},{"comment":"The results in Table 7 are reported without variance or repeated runs; given the small differences, such as 64.91 vs. 65.13 on MMLU, one or more additional seeds would help the reader judge which differences are stable.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The empirical package is strong and the method is practically appealing, but the reward-model contamination concern and the apparent use of the headline benchmark for α selection are both load-bearing for the central claim. I would not reject the manuscript outright, because the issues are fixable with additional experiments and a transparent model-selection statement, but I would insist on those changes before recommending publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: WRPO is a genuinely new training objective, and the empirical work is solid enough that the main result — an 8B model reaching 55.9 LC on AlpacaEval-2 by preference-optimizing against outputs from ten larger models — deserves to be taken seriously. But the paper has two soft spots a referee should push on: the formal derivation of the loss is not a valid Bradley-Terry model, and the preference labels rely on a reward model whose cross-family scores are not calibrated.\n\nWhat's new: combining on-policy and off-policy preferred responses in a single DPO-style loss with an annealed weight is simple and, per the ablations, effective. The authors also show the method scales with number of sources and works with SimPO/IPO objectives. The comparison against FuseLLM/FuseChat is fair in that they rerun those methods with the same five sources, and the wins are large.\n\nSoft spots: Equation (5) defines a compound reward r(x,y_w)=αr(x,y_ws)+(1−α)r(x,y_wt), then plugs it into the BT model p(y_w≻y_l)=σ(r(x,y_w)−r(x,y_l)). But y_w is not a single completion; there is no preference probability for a mixture of two completions. The resulting loss is a reasonable heuristic, but the derivation is a sleight of hand. It's fine to present it as an objective, not as a theorem. The authors admit the need for theory in Appendix H, but the main text should be clearer.\n\nMore concerning: Appendix F shows that the average ArmoRM score of dispreferred source responses (0.158) is higher than that of preferred target responses (0.152). That is direct evidence that reward scores are not comparable across model families — or at least that the offset is as large as the signal being exploited. Since y_ws is selected by the same reward model, its \"preferred\" label is suspect. The stress-test's worry is not empty. It's partially mitigated because the source models are larger and probably genuinely better, and the benchmark gains are not confined to one judge. But the paper needs a robustness check: a different reward model, per-family normalization, or human preference validation on a subset. As is, the fusion story — \"the target absorbs capabilities from sources\" — is weaker than the headline suggests.\n\nAlso, the fusion coefficient α was tuned on AlpacaEval-2 itself (greedy search over [0.1,0.3,0.5,0.7,0.9] using the eval set). The final 55.9 is thus a selected best, not an unbiased estimate. The authors do show the method beats DPO at every α, so the qualitative conclusion survives.\n\nMinor: no error bars; a full-data SFT-on-y_ws baseline would sharpen the comparison.\n\nWho this is for: people working on preference optimization and model fusion. The idea is practical, cheap, and reproducible (code released, checkpoints linked). It deserves a serious referee: the concerns are addressable, and the core result is likely to hold up after revision.\n\nRecommendation: send it to review. Ask for the alpha-selection procedure to be clarified, a reward-model calibration analysis, and a softer presentation of the derivation.","headline":"WRPO is a practical, well-executed new objective with real empirical gains, but the informal BT derivation and the cross-model reward calibration issue in the preference labels should keep a referee engaged.","tokens_in":27042,"tokens_out":4431,"would_cite":true,"duration_ms":41166,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"An 8B target model can beat 236B source models by fusing them via preference optimization.","keywords":["preference optimization","model fusion","knowledge distillation","reward model","distribution shift","LLM alignment","direct preference optimization","AlpacaEval"],"falsifier":"Re-run the WRPO pipeline using a reward model with explicit cross-model calibration (or human preference labels) to select $y_{ws}$ and $y_{wt}$, keeping all other hyperparameters fixed; if the AlpacaEval-2 length-controlled win rate collapses toward the DPO baseline (50.7%), the reported fusion gains are driven by the original reward model's cross-model rankings rather than by the WRPO objective itself.","tokens_in":25992,"feed_emoji":"🧩","tokens_out":5977,"duration_ms":50140,"temperature":0.7,"pith_summary":"This paper argues that combining the strengths of many different open-source LLMs does not require aligning vocabularies or merging weight matrices. Instead, the authors reframe model fusion as a preference-optimization problem: the target model learns from a dataset in which the preferred response is a high-reward answer sampled from any of ten source models, and the dispreferred response is a low-reward answer from the target itself. Because such off-policy data shifts the target's distribution, they introduce Weighted-Reward Preference Optimization (WRPO), which interpolates between the target's own preferred responses and the source models' preferred responses with a fusion coefficient that grows during training. With LLaMA3-8B-Instruct as the target, WRPO reaches a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and 46.2% on Arena-Hard, surpassing every source model on AlpacaEval-2. If correct, this gives a path to absorbing the capabilities of much larger systems into a single small model at low inference cost.","feed_headline":"8B model beats 236B sources via preference-based fusion","feed_subtitle":"WRPO fuses ten heterogeneous LLMs into one small target, hitting 55.9% on AlpacaEval-2 without vocabulary alignment.","key_machinery":"The load-bearing object is the WRPO loss in Eq. (6), a modified DPO objective in which the preferred response's internal reward is $\\alpha \\cdot r(x, y_{ws}) + (1-\\alpha) \\cdot r(x, y_{wt})$, where $y_{ws}$ is the highest-reward response drawn from the source LLMs and $y_{wt}$ is the highest-reward response from the target model itself. The fusion coefficient $\\alpha$ starts at 0 and increases linearly during training, so the optimization begins with on-policy preference pairs (target vs. target) and gradually shifts weight toward hybrid-policy pairs (source vs. target). This progressive weighting is the mechanism that mitigates the distributional deviation shown in the paper's Figure 1, and the ablation studies attribute the gains specifically to the combination of both preferred responses rather than to either alone.","core_discovery":"The central claim is that implicit model fusion—transferring capabilities from heterogeneous source LLMs to a target LLM purely through preference data—can outperform explicit fusion methods that align vocabularies and merge distributions. The paper's key evidence is the WRPO objective, which treats the preferred response as a weighted combination of a source-model response $y_{ws}$ and a target-model response $y_{wt}$, with the weight $\\alpha$ on the source response increasing linearly from 0 during training. This progressive schedule lets the target start from its own distribution and gradually move toward the source distribution, avoiding the degradation that direct DPO on off-policy source responses causes. On three instruction-following benchmarks, the method consistently beats DPO, SimPO, IPO, FuseLLM, and FuseChat baselines, and the 8B fused model exceeds all ten source models on AlpacaEval-2.","pith_inferences":["Editorial inference: if the reward model's cross-model rankings are biased—the paper's own appendix shows source dispreferred responses averaging higher ArmoRM scores than target preferred ones—then some of the reported fusion gain may be an artifact of that reward model, and the method's true value would be better tested with human-annotated or reward-model-ensemble preferences.","Editorial inference: the linear $\\alpha$ schedule resembles a curriculum from on-policy to off-policy data; the same schedule could be applied to other off-policy alignment settings where the policy and the preference source differ.","Editorial inference: because inference only runs the target model, WRPO converts ensemble capability into a single-model deployment; a practical extension would be to combine WRPO-fused models for different domains, each fused from domain-specialist sources."],"forward_implications":["An 8B target model can exceed every one of its ten source models (ranging up to 236B) on AlpacaEval-2's length-controlled win rate, making capability fusion possible at a fraction of the source models' inference cost.","Implicit fusion removes the need for vocabulary alignment and distribution-matrix merging, so new source models can be added to the data-collection stage without any change to the training pipeline.","WRPO is not tied to DPO: combining the same weighted-reward mechanism with SimPO and IPO improves both objectives, suggesting the progressive weighting is a general recipe for off-policy preference data.","Performance improves as more source LLMs are added (from 1 to 10 sources), indicating the method scales with the diversity of the source pool.","The fused model also improves on the target on several Open LLM Leaderboard tasks, even though the training data were not designed for them."],"supporting_citations":[{"why":"Supplies the DPO objective that WRPO modifies with a weighted preferred reward.","marker":"Rafailov et al., 2023"},{"why":"Provides the ArmoRM reward model used to select the preferred source response and score all responses.","marker":"Wang et al., 2024a"},{"why":"Provides the UltraFeedback prompts used to construct the training preference dataset.","marker":"Cui et al., 2024"},{"why":"Introduces weighted preference optimization for off-policy data, the closest prior setup to WRPO.","marker":"Zhou et al., 2024"},{"why":"Serves as an explicit fusion baseline (FuseLLM) that WRPO is compared against and outperforms.","marker":"Wan et al., 2024a"},{"why":"Serves as an explicit fusion baseline (FuseChat) that WRPO is compared against and outperforms.","marker":"Wan et al., 2024b"},{"why":"Provides the SimPO objective that WRPO is combined with and one of the preference-optimization baselines.","marker":"Meng et al., 2024"},{"why":"Supplies the Arena-Hard benchmark used for the win-rate evaluation.","marker":"Li et al., 2024"},{"why":"Supplies the MT-Bench benchmark used for the multi-turn instruction-following evaluation.","marker":"Zheng et al., 2023"}],"fun_headline_variants":["Implicit preference fusion: 8B model beats ten sources","WRPO fuses LLMs without vocab alignment, hits 55.9%","Preference-based fusion outdoes explicit merging on AlpacaEval-2","No matrix fusion needed: WRPO lifts 8B to 55.9% win rate","WRPO: implicit fusion beats explicit, no vocab alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire preference dataset assumes that a single reward model (ArmoRM) ranks responses correctly across different model families, so that the highest-scoring source response is truly better than the target's best response; if that cross-model ranking is wrong, every preference label is contaminated and the reported gains may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Implicit preference fusion: 8B model beats ten sources","WRPO fuses LLMs without vocab alignment, hits 55.9%","Preference-based fusion outdoes explicit merging on AlpacaEval-2","No matrix fusion needed: WRPO lifts 8B to 55.9% win rate","WRPO: implicit fusion beats explicit, no vocab alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001186,"raw_usage":{"total_tokens":4919,"prompt_tokens":987,"completion_tokens":3932,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":3832}},"tokens_in":603,"tokens_out":3932,"duration_ms":28894,"temperature":1.0,"reasoning_tokens":3832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:41:21.446034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the WRPO pipeline using a reward model with explicit cross-model calibration (or human preference labels) to select $y_{ws}$ and $y_{wt}$, keeping all other hyperparameters fixed; if the AlpacaEval-2 length-controlled win rate collapses toward the DPO baseline (50.7%), the reported fusion gains are driven by the original reward model's cross-model rankings rather than by the WRPO objective itself.","supporting_citations":[{"cited_title":"Manning, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"Supplies the DPO objective that WRPO modifies with a weighted preferred reward."},{"cited_title":"UltraFeedback : Boosting language models with high-quality feedback","cited_arxiv_id":null,"evidence_quote":"Provides the UltraFeedback prompts used to construct the training preference dataset."},{"cited_title":"Sim PO : Simple preference optimization with a reference-free reward","cited_arxiv_id":null,"evidence_quote":"Provides the SimPO objective that WRPO is combined with and one of the preference-optimization baselines."}],"review_version":1}