REVIEW 6 cited by
Human Alignment of Large Language Models through Online Preference Optimisation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.
Forward citations
Cited by 6 Pith papers
-
Preference learning made easy: Everything should be understood through win rate
Under two axioms (preference-consistency and prevalence-consistency), the only distribution-grounded evaluation for preference learning is h-win rate, and most popular alignment methods can be classified by whether th...
-
Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
LLM self-play finetuning is equivalent to adversarial imitation learning; the chi-squared regularized variant SPIF bounds rewards and improves stability.
-
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
LPC adds a discrete latent code layer to DPO-family alignment objectives, improving average preference accuracy and downstream scores across three base models.
-
Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution
A multimodal LLM can improve itself using only unlabeled images by self-generating questions, self-enhancing answers, and adding a description-alignment loss to DPO.
-
From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought
Answer-oriented chain-of-thought prompts that generate both positive and negative reasoning data, combined with iterative DPO, improve multimodal LLM reasoning on several benchmarks.
-
Jackpot! Alignment as a Maximal Lottery
Nash Learning from Human Feedback is shown to approximate the maximal lottery voting rule, which the authors argue better reflects majority preferences than Borda-like RLHF.
Discussion (0). Continue with ORCID to comment.