REVIEW 6 cited by
Aligning Language Models with Preferences through f-divergence Minimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Aligning language models with preferences can be posed as approximating a target distribution representing some desired behavior. Existing approaches differ both in the functional form of the target distribution and the algorithm used to approximate it. For instance, Reinforcement Learning from Human Feedback (RLHF) corresponds to minimizing a reverse KL from an implicit target distribution arising from a KL penalty in the objective. On the other hand, Generative Distributional Control (GDC) has an explicit target distribution and minimizes a forward KL from it using the Distributional Policy Gradient (DPG) algorithm. In this paper, we propose a new approach, f-DPG, which allows the use of any f-divergence to approximate any target distribution that can be evaluated. f-DPG unifies both frameworks (RLHF, GDC) and the approximation methods (DPG, RL with KL penalties). We show the practical benefits of various choices of divergence objectives and demonstrate that there is no universally optimal objective but that different divergences present different alignment and diversity trade-offs. We show that Jensen-Shannon divergence strikes a good balance between these objectives, and frequently outperforms forward KL divergence by a wide margin, leading to significant improvements over prior work. These distinguishing characteristics between divergences persist as the model size increases, highlighting the importance of selecting appropriate divergence objectives.
Forward citations
Cited by 6 Pith papers
-
Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
CALM uses bilevel optimization to tune per-vocabulary temperature-like logit adjustments during LLM fine-tuning, and reports improved out-of-domain calibration for aligned language models.
-
Feedback-Driven Vision-Language Alignment with Minimal Human Supervision
SVP uses self-captioning with grounding feedback and a scoring filter to adapt vision-language models with about 1,000 images, improving captioning, referring, object recall, and hallucination control.
-
Frictional Agent Alignment Framework: Slow Down and Don't Break Things
FAAF aligns an LLM with a two-part loss, one part conditioned on a detected belief-misalignment state, to generate reflection-prompting interventions, and reports higher win rates than DPO, IPO, and PPO on collaborati...
-
RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution
RED assigns token-level rewards by taking the difference of a sequence reward model's scores on adjacent prefixes, improving RLHF training without additional reward-model training.
-
Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.
-
Multi-Response Preference Optimization with Augmented Ranking Dataset
The paper proposes Multi-DPO, a weighted multi-response preference loss, and a synthetic data augmentation pipeline, but the derivation is mathematically unsound and experimental support is weak.
Discussion (0). Continue with ORCID to comment.