REVIEW 16 cited by
Theoretical guarantees on the best-of-n alignment policy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
A simple and effective method for the inference-time alignment and scaling test-time compute of generative models is best-of-$n$ sampling, where $n$ samples are drawn from a reference policy, ranked based on a reward function, and the highest ranking one is selected. A commonly used analytical expression in the literature claims that the KL divergence between the best-of-$n$ policy and the reference policy is equal to $\log (n) - (n-1)/n.$ We disprove the validity of this claim, and show that it is an upper bound on the actual KL divergence. We also explore the tightness of this upper bound in different regimes, and propose a new estimator for the KL divergence and empirically show that it provides a tight approximation. We also show that the win rate of the best-of-$n$ policy against the reference policy is upper bounded by $n/(n+1)$ and derive bounds on the tightness of this characterization. We conclude with analyzing the tradeoffs between win rate and KL divergence of the best-of-$n$ alignment policy, which demonstrate that very good tradeoffs are achievable with $n < 1000$.
Forward citations
Cited by 16 Pith papers
-
When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design
Safety in surrogate-driven design requires oracle-only certification; rank preservation is the exact criterion for oracle use, and selection-aware auditing is the cheapest way to certify it.
-
ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Improving Large Vision and Language Models by Learning from a Panel of Peers
A panel of LVLMs that generate, evaluate, and learn from each other's outputs improves average benchmark scores by 9 points across 15 tasks.
-
Does More Inference-Time Compute Really Help Robustness?
With exposed reasoning chains, increasing inference-time compute consistently decreases measured robustness across 12 open-source reasoning models, while hidden chains show improvements.
-
Saffron-1: Safety Inference Scaling
A multifurcation reward model that scores all next-token candidates in one call makes inference-time safety scaling with tree search far more compute-efficient than best-of-N sampling.
-
Soft Best-of-n Sampling for Model Alignment
Soft Best-of-n sampling provably approaches the optimal tilted reward distribution at O(1/n) KL divergence and relative reward error, with sample complexity that grows exponentially in sequence length for blockwise sampling.
-
InfAlign: Inference-aware language model alignment
Reward calibration plus a procedure-specific reward transformation lets RLHF optimize inference-time (best-of-N / worst-of-N) win rates better than standard RLHF, IPO, BoND, and BoNBoN.
-
Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization
Prompt-Noise Optimization jointly tunes the prompt embedding and diffusion noise at inference time to suppress unsafe images while keeping outputs close to the prompt.
-
Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Dualizing Safe RLHF yields a one-dimensional convex calibration of λ that defines a drop-in safety-aware reward for Best-of-N and token-level inference-time decoders.
-
Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
Sampling multiple reasoning paths, refining each with self-critique and self-correction, then majority voting improves math reasoning accuracy over width-only or verifier-based test-time scaling on several open-weight LLMs.
-
Test-time reward-guided alignment of language models by importance sampling on pre-logit space
AISP is a training-free decode-time alignment method: Gaussian-perturb LLM pre-logits, score sampled responses with a reward model, and iteratively shift the perturbation mean by reward-weighted importance sampling.
-
Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track
ML conferences should create an official peer-reviewed track dedicated to refuting and critiquing previously published work.
-
DynScaling: Efficient Verifier-free Inference Scaling via Dynamic and Integrated Sampling
DynScaling improves verifier-free inference-time scaling by merging parallel and sequential sampling and allocating budget across queries with a UCB-based uncertainty rule.
-
CoDe: Blockwise Control for Denoising Diffusion Models
CoDe applies blockwise best-of-N sampling during diffusion denoising, with Tweedie-based reward estimates, to align generated images to differentiable or non-differentiable rewards.
-
LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds
LIAR shows that best-of-N sampling of suffixes from a GPT-2 model jailbreaks several aligned LLMs with low-perplexity prompts and far faster time-to-attack than training-based attacks.
Discussion (0). Continue with ORCID to comment.