Pith. sign in

REVIEW 8 cited by

Variational Best-of-N Alignment

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.06057 v3 pith:LNQ43R2Q submitted 2024-07-08 cs.CL cs.AIcs.LG

Afra Amini , Tim Vieira , Elliott Ash , Ryan Cotterell This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords inferencelanguagemodelrewardvariationalalgorithmalignmentvbon
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Best-of-N (BoN) is a popular and effective algorithm for aligning language models to human preferences. The algorithm works as follows: at inference time, N samples are drawn from the language model, and the sample with the highest reward, as judged by a reward model, is returned as the output. Despite its effectiveness, BoN is computationally expensive; it reduces sampling throughput by a factor of N. To make BoN more efficient at inference time, one strategy is to fine-tune the language model to mimic what BoN does during inference. To achieve this, we derive the distribution induced by the BoN algorithm. We then propose to fine-tune the language model to minimize backward KL divergence to the BoN distribution. Our approach is analogous to mean-field variational inference and, thus, we term it variational BoN (vBoN). To the extent this fine-tuning is successful and we end up with a good approximation, we have reduced the inference cost by a factor of N. Our experiments on controlled generation and summarization tasks show that BoN is the most effective alignment method, and our variational approximation to BoN achieves the closest performance to BoN and surpasses models fine-tuned using the standard KL-constrained RL objective. In the controlled generation task, vBoN appears more frequently on the Pareto frontier of reward and KL divergence compared to other alignment methods. In the summarization task, vBoN achieves high reward values across various sampling temperatures.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Improvement in Language Models: The Sharpening Mechanism

    cs.AI 2024-12 conditional novelty 7.0 of 10

    Self-improvement in language models can be understood as amortizing best-of-N inference-time selection, with minimax-optimal guarantees for SFT and provable coverage-free benefits for RL with exploration.

  2. Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-$K$ Objective

    cs.LG 2026-07 accept novelty 6.5 of 10

    Rank-conditioned Horvitz–Thompson reuses all C(n,K) subsets of one Gumbel-Top-n pool for unbiased Plackett–Luce best-of-K value and score-function gradient, with an exact Max-specific DP collapse to a 1-D integral.

  3. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  4. Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detection

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A decoding pipeline reuses cached tokens and anneals sampling temperature to accelerate self-consistency hallucination detection by up to 3x without meaningful AUROC loss.

  5. InfAlign: Inference-aware language model alignment

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Reward calibration plus a procedure-specific reward transformation lets RLHF optimize inference-time (best-of-N / worst-of-N) win rates better than standard RLHF, IPO, BoND, and BoNBoN.

  6. Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

    cs.LG 2026-08 conditional novelty 5.0 of 10

    The paper formalizes test-time scaling into three regimes, introduces a discovery-stability profile for repeated-sampling evaluation, and releases nearly two million reasoning traces.

  7. LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds

    cs.CL 2024-12 reject novelty 4.0 of 10

    LIAR shows that best-of-N sampling of suffixes from a GPT-2 model jailbreaks several aligned LLMs with low-perplexity prompts and far faster time-to-attack than training-based attacks.

  8. Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

    cs.AI 2024-12 conditional novelty 3.0 of 10

    A roadmap paper argues that reproducing o1 hinges on four RL components, policy initialization, reward design, search, and learning, and frames existing open-source o1 projects as instances of this roadmap.

Pith tools