Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Weighted-Reward Preference Optimization for Implicit Model Fusion

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An 8B target model can beat 236B source models by fusing them via preference optimization.

desk verdict WRPO is a practical, well-executed new objective with real empirical gains, but the informal BT derivation and the cross-model reward calibration issue in the preference labels should keep a referee engaged. read the letter →

arxiv 2412.03187 v2 pith:NVTPWGBN submitted 2024-12-04 cs.CL

Ziyi Yang , Fanqi Wan , Longguang Zhong , Tianyuan Shi , Xiaojun Quan This is my paper · ORCID
classification cs.CL MSC 68T50
keywords preferenceoptimizationmodelfusionknowledgedistillationrewarddistributionshiftLLMalignmentdirectAlpacaEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that combining the strengths of many different open-source LLMs does not require aligning vocabularies or merging weight matrices. Instead, the authors reframe model fusion as a preference-optimization problem: the target model learns from a dataset in which the preferred response is a high-reward answer sampled from any of ten source models, and the dispreferred response is a low-reward answer from the target itself. Because such off-policy data shifts the target's distribution, they introduce Weighted-Reward Preference Optimization (WRPO), which interpolates between the target's own preferred responses and the source models' preferred responses with a fusion coefficient that grows during training. With LLaMA3-8B-Instruct as the target, WRPO reaches a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and 46.2% on Arena-Hard, surpassing every source model on AlpacaEval-2. If correct, this gives a path to absorbing the capabilities of much larger systems into a single small model at low inference cost.

What carries the argument

The load-bearing object is the WRPO loss in Eq. (6), a modified DPO objective in which the preferred response's internal reward is $\alpha \cdot r(x, y_{ws}) + (1-\alpha) \cdot r(x, y_{wt})$, where $y_{ws}$ is the highest-reward response drawn from the source LLMs and $y_{wt}$ is the highest-reward response from the target model itself. The fusion coefficient $\alpha$ starts at 0 and increases linearly during training, so the optimization begins with on-policy preference pairs (target vs. target) and gradually shifts weight toward hybrid-policy pairs (source vs. target). This progressive weighting is the mechanism that mitigates the distributional deviation shown in the paper's Figure 1, and the ablation studies attribute the gains specifically to the combination of both preferred responses rather than to either alone.

What would settle it

Re-run the WRPO pipeline using a reward model with explicit cross-model calibration (or human preference labels) to select $y_{ws}$ and $y_{wt}$, keeping all other hyperparameters fixed; if the AlpacaEval-2 length-controlled win rate collapses toward the DPO baseline (50.7%), the reported fusion gains are driven by the original reward model's cross-model rankings rather than by the WRPO objective itself.

Watch

Extended reading notes

Core claim

The central claim is that implicit model fusion—transferring capabilities from heterogeneous source LLMs to a target LLM purely through preference data—can outperform explicit fusion methods that align vocabularies and merge distributions. The paper's key evidence is the WRPO objective, which treats the preferred response as a weighted combination of a source-model response $y_{ws}$ and a target-model response $y_{wt}$, with the weight $\alpha$ on the source response increasing linearly from 0 during training. This progressive schedule lets the target start from its own distribution and gradually move toward the source distribution, avoiding the degradation that direct DPO on off-policy source responses causes. On three instruction-following benchmarks, the method consistently beats DPO, SimPO, IPO, FuseLLM, and FuseChat baselines, and the 8B fused model exceeds all ten source models on AlpacaEval-2.

Load-bearing premise

The entire preference dataset assumes that a single reward model (ArmoRM) ranks responses correctly across different model families, so that the highest-scoring source response is truly better than the target's best response; if that cross-model ranking is wrong, every preference label is contaminated and the reported gains may not generalize.

Editorial extensions

If this is right

  • An 8B target model can exceed every one of its ten source models (ranging up to 236B) on AlpacaEval-2's length-controlled win rate, making capability fusion possible at a fraction of the source models' inference cost.
  • Implicit fusion removes the need for vocabulary alignment and distribution-matrix merging, so new source models can be added to the data-collection stage without any change to the training pipeline.
  • WRPO is not tied to DPO: combining the same weighted-reward mechanism with SimPO and IPO improves both objectives, suggesting the progressive weighting is a general recipe for off-policy preference data.
  • Performance improves as more source LLMs are added (from 1 to 10 sources), indicating the method scales with the diversity of the source pool.
  • The fused model also improves on the target on several Open LLM Leaderboard tasks, even though the training data were not designed for them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the reward model's cross-model rankings are biased—the paper's own appendix shows source dispreferred responses averaging higher ArmoRM scores than target preferred ones—then some of the reported fusion gain may be an artifact of that reward model, and the method's true value would be better tested with human-annotated or reward-model-ensemble preferences.
  • Editorial inference: the linear $\alpha$ schedule resembles a curriculum from on-policy to off-policy data; the same schedule could be applied to other off-policy alignment settings where the policy and the preference source differ.
  • Editorial inference: because inference only runs the target model, WRPO converts ensemble capability into a single-model deployment; a practical extension would be to combine WRPO-fused models for different domains, each fused from domain-specialist sources.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Weighted-Reward Preference Optimization (WRPO), an implicit model fusion method that transfers capabilities from heterogeneous open-source LLMs into a target model without vocabulary alignment or distribution merging. For each prompt, the authors construct preference quadruples (x, y_ws, y_wt, y_l) using ArmoRM scores, where y_ws is the highest-scoring source response and y_wt/y_l are the target model's best and worst responses, respectively. The WRPO loss is a DPO-like objective that interpolates between an on-policy margin (y_wt vs. y_l) and a hybrid-policy margin (y_ws vs. y_l), with a fusion coefficient α that increases linearly from 0 to 0.1 during training. With LLaMA3-8B-Instruct as the target and ten sources from 9B to 236B, the paper reports a 55.9% length-controlled win rate on AlpacaEval-2, a 46.2% win rate on Arena-Hard, and 7.63 on MT-Bench, exceeding the reported DPO, SimPO, IPO, FuseLLM, and FuseChat baselines. The authors include ablations, source-scaling studies, and an open-source code release.

Significance. If the empirical claims hold, WRPO is a genuinely useful and low-cost fusion recipe: an 8B target model approaches or beats much larger systems on two widely used instruction-following benchmarks while adding only about 16% training time over DPO and no inference-time ensemble cost. The paper deserves credit for a complete experimental package, including public code, explicit hyperparameters and tuning ranges, ablations of both terms in the loss, and scaling with the number of source models. The two strongest concerns from the review are real: Eq. (5)-(6) does not follow from the Bradley-Terry model as written, and Appendix F's own reward statistics indicate that the cross-model preference labels may be contaminated by a systematic reward-model offset. I agree with the reader's assessment that the method is not circular in the strong sense, because α is a fitted schedule rather than a conclusion derived from the data; the contribution is nonetheless not fully established until the reward-label assumption and the model-selection procedure are addressed.

major comments (3)
  1. [3.3, Eq. (4)-(6)] The derivation of the WRPO objective is not a valid Bradley-Terry reformulation. In Eq. (4), p(y_w ≻ y_l | x) is written as σ(r(x, y_w) − r(x, y_l)), but y_w was defined immediately before as a pair of completions {y_ws, y_wt}, and Eq. (5) defines r(x, y_w) as a weighted average of rewards of two distinct completions. The Bradley-Terry model gives preference probabilities between realizable responses, and a weighted average of two different rewards is not the reward of any single completion. Equation (6) is therefore a new, ad hoc objective rather than a consequence of the Bradley-Terry model. Section H concedes that a more rigorous theoretical analysis is needed; the paper should either supply that analysis or explicitly present Eq. (6) as a regularized empirical loss inspired by DPO, not as a BT-derived objective. The empirical results may survive this reframing, but the current presentation overstates the theoretical grounding.
  2. [3.1, Appendix F, Fig. 4] The preference-label construction assumes ArmoRM scores are comparable across model families, and the paper's own Appendix F shows they are not. The reported mean rewards are y_ws = 0.180, y_wt = 0.152, y_ls = 0.158, and y_lt = 0.132; that is, a dispreferred source response outscores the target model's preferred response on average. If this family-level offset is present inside the preference quadruples, then y_ws is not necessarily a higher-quality response than y_l; it is merely a response that receives a higher ArmoRM score. Because the main evaluations on AlpacaEval-2 and Arena-Hard use GPT-4 judges that may share the same stylistic biases as ArmoRM, the ablation gain of 5.2 points from y_ws in Fig. 4 could reflect reward/judge bias rather than capability transfer. The paper should provide a concrete control, such as human annotation on a subset, an alternative reward model for label construction, or evidence on objective-answer benchmarks like GSM8K and ARC, to show that the transfer goes beyond fitting the reward model's preferences.
  3. [4.2, Fig. 5, Appendix E] The choice of α is conflated with test-set evaluation. Appendix E describes a greedy search over α ∈ {0.1, 0.3, 0.5, 0.7, 0.9}, and Figure 5 reports the AlpacaEval-2 length-controlled win rate for those values. The final model then reports AlpacaEval-2 LC 55.9%, which is the same benchmark used for this search. As written, the headline number is therefore a model-selected value rather than an independent test result. Please state clearly which benchmarks were used for hyperparameter selection and provide a final evaluation on data or metrics that did not contribute to any model choice, so the reported AlpacaEval-2 and Arena-Hard numbers can be interpreted as test performance.
minor comments (5)
  1. [Eq. (8)] The expectation subscript in Eq. (8) has a trailing comma: (x, y_ws, y_wt, y_l,). Please correct this typo.
  2. [Fig. 3 caption] The caption of Fig. 3(c) reports α increasing from 0 to 0.5, while §4.1 states that the main configuration uses α increasing from 0 to 0.1. Please clarify whether Fig. 3 is an illustrative analysis run or the actual configuration used in Table 2, and explain the relation between the two settings.
  3. [Table 1] The 'Percentage' column in Table 1 should state the denominator, for example the number of prompts for which that source produced the highest-scoring response; as written, the reader cannot tell whether the percentages refer to all selected y_ws responses or to some other base.
  4. [4.2] The sentence comparing WRPO to 'the best-performing preference optimization baseline' by 2.2, 9.6, and 0.09 points should name the exact baselines (Target-SimPO for AlpacaEval-2 and Arena-Hard, Target-IPO for MT-Bench) and clarify that these are prior checkpoints, in contrast with the authors' own Target-SFT-DPO in the same table.
  5. [Appendix B] The results in Table 7 are reported without variance or repeated runs; given the small differences, such as 64.91 vs. 65.13 on MMLU, one or more additional seeds would help the reader judge which differences are stable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the WRPO objective is explicitly constructed and the headline gains rest on external benchmark evaluations.

full rationale

The paper's derivation chain is self-contained. Section 3.3 derives the WRPO loss by substituting the DPO reparameterized reward (Eq. 1) into a Bradley-Terry model with a compound reward (Eqs. 4-5); this is an explicit algebraic construction plus a tunable design choice for alpha, not a conclusion hidden in the inputs. The progressive alpha schedule is a hyperparameter, tuned in Appendix A and ablated in Appendix E, and is not fitted to the reported benchmark numbers. The central claims (55.9% LC win rate on AlpacaEval-2 and 46.2% on Arena-Hard) are established by external GPT-4-judged benchmarks, which are independent of the training objective. Self-citations to the authors' FuseLLM, FuseChat, and ProFuser work serve as baselines and related work, not as load-bearing justifications for WRPO. The Section F observation that ArmoRM scores show a family-level bias (source dispreferred responses average 0.158 versus target preferred responses at 0.152) is a legitimate empirical validity concern about preference-label quality, but it is not a circular reduction: the paper's result is not defined in terms of ArmoRM scores, and the concern is about correctness of the data pipeline, not about deriving a conclusion from itself. No step in the paper equates a prediction to a fitted input by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central method rests on several unstated or weakly motivated assumptions: the validity of applying BT to a compound reward, the reliability of the reward model across heterogeneous models (contradicted by Section F), and the generalizability of the hand-tuned alpha schedule. The only fitted quantity that all headline numbers depend on is the fusion coefficient alpha, which is searched on AlpacaEval-2.

free parameters (3)
  • fusion coefficient alpha (final value) = 0.1 for WRPODPO; searched over [0.1, 0.3, 0.5, 0.7, 0.9] for analysis
    Greedy search over AlpacaEval-2 LC win rate (Appendix E, Figure 5); no explicit validation split documented, so the headline result is selected partly on the evaluation benchmark.
  • SFT data fraction = 1/3 (one-third of UltraFeedback prompts used for SFT stage)
    Hand-chosen split in Section 4.1; affects how much off-policy SFT precedes preference optimization.
  • beta = 0.01 (WRPODPO), 10.0 (WRPOSimPO)
    Standard DPO/SimPO hyperparameter; chosen from search ranges in Table 5.
assumptions (4)
  • ad hoc to paper Bradley-Terry model applies to a 'compound preferred reward' r(x, y_w) = α r(x, y_ws) + (1−α) r(x, y_wt), even though y_ws and y_wt are distinct completions with no single realization.
    Section 3.3, Eq. (5): the BT probability in Eq. (4) is defined for a single preferred completion; the weighted average of two rewards is substituted in without justification.
  • domain assumption The linear ramp of α from 0 to target smoothly shifts the target's distribution toward source models without requiring explicit distribution matching.
    Section 3.3 and Figure 3; this is the core mechanism, but no theory or metric beyond internal reward margins supports it.
  • domain assumption ArmoRM reward scores are comparable across heterogeneous models and can rank responses across model families.
    Section 3.1; Section F undermines this by showing source dispreferred responses score above target preferred responses.
  • domain assumption SFT on one-third of y_ws responses followed by preference optimization on the remainder is sufficient; the SFT stage's known performance decline on MMLU/GSM8K (Appendix B) does not undermine the fusion claim.
    Section 4.1 and Appendix B; catastrophic forgetting is acknowledged.
invented entities (1)
  • Compound preferred reward r(x, y_w) = α·r(x, y_ws) + (1−α)·r(x, y_wt)
    purpose: Creates a synthetic preferred-reward signal combining on-policy and off-policy completions for the WRPO loss in Eq. (6).
    No physical response corresponds to this reward; it is a mathematical blend of two distinct completions, so it is an invented construct. It has no falsifiable handle outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Weighted-Reward Preference Optimization for Implicit Model Fusion." pith.science (2026). https://pith.science/paper/NVTPWGBN

@misc{pith2026241203187,
  author       = {Pith},
  title        = {Pith review of: Weighted-Reward Preference Optimization for Implicit Model Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVTPWGBN}},
  note         = {Machine review of arXiv:2412.03187}
}
read the original abstract

While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.

Figures

Figures reproduced from arXiv: 2412.03187 by the authors.

Figure 1
Figure 1. Distribution deviations between responses from heterogeneous source LLMs and the LLaMA3-8B [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed WRPO for implicit model fusion. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Internal reward dynamics on Target-SFT model under different preference optimization setups. (a) [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: AlpacaEval-2 length-controlled win rate and hybrid-policy internal reward accuracy under different fusion coefficient α settings. WRPO by 6.3 points, which combines both yws and ywt . On the other hand, removing yws reduces WRPO to DPO based solely on self-sampled on-p…
Figure 6
Figure 6. Figure 6: Comparisons of dynamic and static tuning strategies for the fusion coefficient on AlpacaEval-2, utilizing the length-controlled win rate metric. In WPRO, we implement a dynamic adjustment mechanism for the fusion coefficient α to facilitate a gradual transition of the …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CortexDebate: Debating Sparsely and Equally for Multi-Agent Debate

    cs.AI 2025-07 conditional novelty 5.0 of 10

    CortexDebate prunes the multi-agent debate graph every round using a McKinsey-style trust score per directed link, reporting accuracy gains over full-debate baselines on eight datasets with shorter per-agent contexts.

  2. An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems

    cs.CL 2024-12 unverdicted novelty 3.0 of 10

    A survey and position paper that reviews LLM prompting, RAG, and RL techniques and argues they could support open-ended implementation generation, without presenting new results.

Reference graph

Works this paper leans on

64 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv, abs/2404.14219, 2024

    arXiv 2024

  2. [2]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, 2024

    work page 2024

  3. [3]

    Open LLM leaderboard, 2023

    Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard, 2023

    work page 2023

  4. [4]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324, 1952

    work page 1952

  5. [5]

    InternLM2 technical report

    Zheng Cai, Maosong Cao, et al. InternLM2 technical report. ArXiv, abs/2403.17297, 2024

    arXiv 2024

  6. [6]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024

    work page 2024

  7. [7]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017

    work page 2017

  8. [8]

    Bam! born-again multi-task networks for natural language understanding

    Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le. Bam! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 5931--5937, 2019

    work page 2019

Show all 64 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018

    2018 arXiv

  2. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021

    arXiv

  3. [11]

    UltraFeedback : Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeedback : Boosting language models with high-quality feedback. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  4. [12]

    DeepSeek-Coder-V2 : Breaking the barrier of closed-source models in code intelligence

    DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, et al. DeepSeek-Coder-V2 : Breaking the barrier of closed-source models in code intelligence. ArXiv, abs/2406.11931, 2024

    2024 arXiv

  5. [13]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024

    2024 arXiv

  6. [14]

    Length-controlled alpacaeval: A simple debiasing of automatic evaluators

    Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024

    2024

  7. [15]

    KTO : Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO : Model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  8. [16]

    Mixture-of-LoRAs : An efficient multitask tuning method for large language models

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-LoRAs : An efficient multitask tuning method for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COL...

    2024

  9. [17]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129 0 (6): 0 1789--1819, 2021

    2021

  10. [18]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  11. [19]

    ORPO : Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO : Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  12. [20]

    Mistral 7B

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    arXiv

  13. [21]

    LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14165--14178, 2023 b

    2023

  14. [22]

    Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks

    Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks. Advances in Neural Information Processing Systems, 2023

    2023

  15. [23]

    Sparse upcycling: Training mixture-of-experts from dense checkpoints

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learning Repre...

    2023

  16. [24]

    The Winograd schema challenge

    Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012

    2012

  17. [25]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  18. [26]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval : An automatic evaluator of instruction-following models, 2023

    2023

  19. [27]

    TruthfulQA : Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA : Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3214--3252, 2022

    2022

  20. [28]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 2022

    2022

  21. [29]

    Pack of LLM s: Model fusion at test-time via perplexity optimization

    Costas Mavromatis, Petros Karypis, and George Karypis. Pack of LLM s: Model fusion at test-time via perplexity optimization. In First Conference on Language Modeling, 2024

    2024

  22. [30]

    Sim PO : Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Sim PO : Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems, 2024

    2024

  23. [31]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. ArXiv, abs/2403.19159, 2024

    2024 arXiv

  24. [32]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. D irect P reference O ptimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023

    2023

  25. [33]

    Aligning large and small language models via chain-of-thought reasoning

    Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain-of-thought reasoning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1812--1827, 2024

    2024

  26. [34]

    Gemma 2: Improving open language models at a practical size

    Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, et al. Gemma 2: Improving open language models at a practical size. ArXiv, abs/2408.00118, 2024

    2024 arXiv

  27. [35]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347, 2017

    2017 arXiv

  28. [36]

    DeepSeek-V2 : A strong, economical, and efficient mixture-of-experts language model

    Zhihong Shao, Damai Dai, Daya Guo, Bo Liu (Benjamin Liu), and Zihan Wang. DeepSeek-V2 : A strong, economical, and efficient mixture-of-experts language model. ArXiv, abs/2405.04434, 2024

    2024 arXiv

  29. [37]

    ProFuser : Progressive fusion of large language models

    Tianyuan Shi, Fanqi Wan, Canbin Huang, Xiaojun Quan, Chenliang Li, Ming Yan, and Ji Zhang. ProFuser : Progressive fusion of large language models. ArXiv, abs/2408.04998, 2024

    2024

  30. [38]

    Branch-Train-MiX : Mixing expert LLM s into a mixture-of-experts LLM

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wen Li, Wen tau Yih, Jason E Weston, and Xian Li. Branch-Train-MiX : Mixing expert LLM s into a mixture-of-experts LLM . In First Conference on Language Modeling, 2024

    2024

  31. [39]

    Preference fine-tuning of LLM s should leverage suboptimal, on-policy data

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLM s should leverage suboptimal, on-policy data. In Proceedings of the 41st International Conference on Machine ...

    2024

  32. [40]

    Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and N. Chawla. TinyLLM : Learning a small student from multiple large language models. ArXiv, abs/2402.04616, 2024

    2024 arXiv

  33. [41]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl \'e mentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of LM align...

    arXiv

  34. [42]

    Knowledge fusion of large language models

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024 a

    2024

  35. [43]

    FuseChat : Knowledge fusion of chat models

    Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. FuseChat : Knowledge fusion of chat models. ArXiv, abs/2402.16107, 2024 b

    2024 arXiv

  36. [44]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP...

    2024

  37. [45]

    Mixture-of-Agents enhances large language model capabilities

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-Agents enhances large language model capabilities. ArXiv, abs/2406.04692, 2024 b

    2024 arXiv

  38. [46]

    HelpSteer2 : Open-source dataset for training top-performing reward models

    Zhilin Wang, Yi Dong, Olivier Delalleau, et al. HelpSteer2 : Open-source dataset for training top-performing reward models. ArXiv, abs/2406.08673, 2024 c

    2024 arXiv

  39. [47]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  40. [48]

    Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Proceedings of the 41st International Conference on Ma...

    2024

  41. [49]

    Is DPO superior to PPO for LLM alignment? A comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Proceedings of the 41st International Conference on Machine Learning, 2024 b

    2024

  42. [50]

    Bridging the gap between different vocabularies for llm ensemble

    Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Bridging the gap between different vocabularies for llm ensemble. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape...

    2024

  43. [51]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 2023

    2023

  44. [52]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, et al. Qwen2 technical report. ArXiv, abs/2407.10671, 2024

    2024 arXiv

  45. [53]

    Yi: Open foundation models by 01.ai

    01.AI Alex Young, Bei Chen, Chao Li, Chengen Huang, et al. Yi: Open foundation models by 01.ai. ArXiv, abs/2403.04652, 2024

    2024 arXiv

  46. [54]

    RRHF : Rank responses to align language models with human feedback

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF : Rank responses to align language models with human feedback. In Advances in Neural Information Processing Systems, 2023

    2023

  47. [55]

    H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    2019

  48. [56]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF : Sequence likelihood calibration with human feedback. ArXiv, abs/2305.10425, 2023

    2023 arXiv

  49. [57]

    Judging LLM -as-a-judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM -as-a-judge with MT-Bench and Chatbot Arena . In NeurIPS Datasets and Benchmarks Track, 2023

    2023

  50. [58]

    WPO : Enhancing RLHF with weighted preference optimization

    Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. WPO : Enhancing RLHF with weighted preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, ...

    2024

  51. [59]

    Starling-7B : Improving helpfulness and harmlessness with RLAIF

    Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. Starling-7B : Improving helpfulness and harmlessness with RLAIF . In First Conference on Language Modeling, 2024

    2024

  52. [60]

    Ziegler, Nisan Stiennon, Jeff Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019

    1909 arXiv

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  54. [62]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  55. [63]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  56. [64]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.