WRPO tunes an 8B chat model by combining its own preferred responses (on-policy) with high-reward responses from ten heterogeneous source LLMs (off-policy) using an annealed weight, beating prior fusion and preference-optimization baselines on AlpacaEval-2, Arena-Hard, and MT-Bench.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Weighted-Reward Preference Optimization for Implicit Model Fusion
WRPO tunes an 8B chat model by combining its own preferred responses (on-policy) with high-reward responses from ten heterogeneous source LLMs (off-policy) using an annealed weight, beating prior fusion and preference-optimization baselines on AlpacaEval-2, Arena-Hard, and MT-Bench.