REVIEW 7 cited by
BOND: Aligning LLMs with Best-of-N Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Reinforcement learning from human feedback (RLHF) is a key driver of quality and safety in state-of-the-art large language models. Yet, a surprisingly simple and strong inference-time strategy is Best-of-N sampling that selects the best generation among N candidates. In this paper, we propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We use the Jeffreys divergence (a linear combination of forward and backward KL) to balance between mode-covering and mode-seeking behavior, and derive an iterative formulation that utilizes a moving anchor for efficiency. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models. Aligning Gemma policies with BOND outperforms other RLHF algorithms by improving results on several benchmarks.
Forward citations
Cited by 7 Pith papers
-
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.
-
Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-$K$ Objective
Rank-conditioned Horvitz–Thompson reuses all C(n,K) subsets of one Gumbel-Top-n pool for unbiased Plackett–Luce best-of-K value and score-function gradient, with an exact Max-specific DP collapse to a 1-D integral.
-
MOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models
MOSAIC uses multi-objective MIP search over linear/sparse/low-rank operators plus two-stage distillation to convert a homogeneous VLM into a hardware-aware heterogeneous model that matches teacher performance at 2.5× ...
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis
Smoothed Best-of-N has finite-sample KL and regret bounds under imperfect reward models, and tuning its temperature can make its regret bound beat hard Best-of-N in the overoptimization regime.
-
Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
OREAL shows that outcome-reward RL with best-of-N positive behavior cloning, negative reward shaping, and token-level reweighting reaches state-of-the-art MATH-500 accuracy at 7B and 32B scale.
-
MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
Concentrating synthetic preference paraphrases on low-margin pairs gives consistent but small reward-model and alignment gains in single-run experiments, while the abstract's semantic-aware, multi-benchmark claims are...
Discussion (0). Continue with ORCID to comment.