Pith. sign in

REVIEW 4 major objections 5 minor 15 references

AISP claims reward-guided decoding can outpace best-of-n by steering a Gaussian perturbation of pre-logits toward the reward-weighted posterior.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:18 UTC pith:HPCNDVJZ

load-bearing objection AISP is a legitimate new MPPI-on-pre-logits method with real sample-efficiency gains in its own tables, but the KL-constrained alignment story is overstated; reviewers should focus on the Gaussian assumption and the surrogate-penalty gap. the 4 major comments →

arxiv 2510.26219 v3 pith:HPCNDVJZ submitted 2025-10-30 cs.LG cs.AI

Test-time reward-guided alignment of language models by importance sampling on pre-logit space

Sekitoshi Kanai , Tsukasa Yoshida , Hiroshi Takahashi , Haru Kuroki , Kazumune Hashimoto This is my paper
classification cs.LG cs.AI MSC 62F1568T5093E20
keywords test-time alignmentinference-time alignmentpre-logitsimportance samplingmodel predictive controlreward-guided decodingbest-of-n samplinglarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that test-time alignment — maximizing a reward model without fine-tuning — can be reformulated as a stochastic control problem on pre-logits, and that the optimal control reduces to an importance-sampled mean update. If correct, a decoder can actively search for high-reward responses at inference time, without training a value function or collecting datasets, and it can do so more sample-efficiently than best-of-n. The payoff would be a training-free alignment method that uses only the base LLM and a reward model, with a tunable trade-off between reward and deviation from the base model. The argument rests on a key modeling assumption: that pre-logit distributions are Gaussian.

Core claim

The paper's central discovery is an identity: under a Gaussian model of pre-logit perturbations, the Gaussian mean that minimizes KL divergence to the reward-weighted posterior q*(V) ∝ exp(r(x,y(V))/λ) p(V) is exactly the expectation of the perturbation under that posterior, u*_t = E_{Q*}[v_t]. The authors show this expectation can be estimated by self-normalized importance sampling, with weights combining the sampled reward and an inner-product correction term, and that iterating this update (adaptive importance sampling) yields higher-reward responses than best-of-n with the same number of samples. They also prove that in the limit λ→0 and κ=1 the procedure degenerates exactly to best-of-n

What carries the argument

The engine is the free-energy bound (Theorem 3.1), which relates the control objective J(U) = -E_Q[r] + λ D_KL(Q||P) to the log-partition F = log E_P exp(r/λ), together with the closed form of the optimal Gaussian mean obtained by differentiating the KL divergence (Theorem 3.2). AISP computes u*_t as the weighted mean of sampled perturbation vectors, with softmax weights \bar{w}_i ∝ exp(r_i/λ - (1-α)/σ² Σ_t \hat{u}_t^· v_i,t^i). The α relaxation and the λ temperature allow a practical trade-off between reward gain and KL distance from the base model.

Load-bearing premise

The method assumes the base LLM's pre-logit distributions are Gaussian with a fixed variance; if they are not, the importance weights are not likelihood ratios and the claimed optimal mean is only optimal within the Gaussian family.

What would settle it

Sample pre-logits from the base LLM over many prefixes and test multivariate normality (e.g., kurtosis or QQ plots); or run AISP decoding with one reward model and evaluate with a second, held-out reward model. Non-Gaussian pre-logits, or a collapse of AISP's advantage under a held-out evaluator, would contradict the central mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • AISP improves reward values faster than best-of-n given the same number of generated samples, across several base LLM and reward-model combinations.
  • AISP achieves higher rewards than RE-Control and ARGS without training a value function or collecting a training dataset.
  • In the limit λ→0 and κ=1, AISP provably reduces to best-of-n over the sampled candidates, so AISP is a generalization of BoN.
  • Batched AISP can outperform BoN under the same number of sequential iterations, which matters when parallel compute is limited.
  • The KL distance to the base LLM can be tuned by λ and α, providing a clean knob to control how far the aligned output may move from the base distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Gaussian pre-logit assumption is the load-bearing modeling choice; if real pre-logits are heavy-tailed or multi-modal, the weights in Eq. (12) are not true likelihood ratios, and the update steers within the wrong family.
  • Because the same reward model is used both to optimize and to evaluate, the reported gains may partly reflect reward-model exploitation; a held-out reward or human evaluation would test whether the advantage transfers.
  • The same free-energy and importance-sampling identity could be applied to other continuous latents (e.g., token embeddings, hidden states), making the method a template for training-free steering of LLM internals.
  • The BoN-equivalence limit suggests λ interpolates between passive selection and active distributional alignment; this continuum could be used to control risk or diversity at inference time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AISP, an inference-time alignment method that adds Gaussian perturbations to the pre-logit vectors of an LLM for the first τ tokens and optimizes the perturbation mean via iterated self-normalized importance sampling, following the MPPI framework. The theoretical development derives a free-energy lower bound for a stochastic control objective with a KL penalty on perturbation trajectories, identifies the optimal tilted density q*(V) ∝ exp(r/λ)p(V), and gives an importance-weight update that estimates the optimal Gaussian mean. The paper also draws a connection to best-of-n sampling in the λ→0+ limit, proposes a batched variant, and presents experiments on SHP and HH-RLHF with three base LLMs and two reward models, reporting reward, diversity, coherence, and GPT-4 win rates. The central claim is that AISP improves reward per sample relative to BoN and outperforms other reward-based test-time alignment methods.

Significance. If the theoretical claims held, AISP would provide a training-free, sample-efficient test-time alignment method with a clean MPPI-style derivation, an explicit algorithm, and broad empirical comparison across models, datasets, and reward models. The paper is transparent about its lineage (attributing the key identity to Williams et al. 2018) and provides a complete pseudo-code. However, the theoretical engine as stated is for a surrogate objective: the KL penalty in Eq. (7) is over perturbation trajectories, not response distributions, so the claimed connection to KL-constrained RL (Eq. 2) is not established. The Gaussian assumption on pre-logits is heuristic and unvalidated, and the adaptive-importance-sampling update lacks a convergence analysis. The empirical results are suggestive but mixed on the independent GPT-4 win-rate metric, and the primary metric is the same reward used for optimization. These issues require substantial revision rather than minor polishing.

major comments (4)
  1. [§3.1 Eq. (7); §3.2 Theorem 3.1] The KL term in Eq. (7) is D_KL(Q_{U,σ²}||P) over perturbation trajectories V, with P=N(0,σ²I), which equals ||U||²/(2σ²). This is a control-magnitude penalty, not the response-level KL D_KL(π_AISP||P_LLM) in Eq. (2). The map V→y(V) is non-injective and nonlinear (autoregressive greedy decoding), so KL is not preserved under this map. Thus the minimizer of Eq. (7) is not the solution of the KL-constrained RL objective, and Theorem 3.1's free-energy bound is for a proxy objective. Table 3 shows the gap: at the tuned default (λ=0.3, α=0.9999) the response-level KL is 90.6–140.9 nats, while the surrogate penalty ||U||²/(2σ²) is orders of magnitude smaller. The statement in §3.1 that this term prevents deviation from the base LLM is unsupported. Please re-frame AISP as reward maximization with a control-cost regularizer, or prove a bound relating the surrogate penalty to response-level KL.
  2. [§3.4 Gaussian assumption] The assumption p_LLM(z_t|y_<t)=N(φ(y_<t),σ²I) is justified only by an analogy to softmax/exponential-family classifiers (Lee et al., 2018). No validation on actual LLM pre-logits is provided. If the true distribution is non-Gaussian, the weight function Eq. (12) is not the likelihood ratio q*/q_proposal, Eq. (11) minimizes D_KL within the wrong family, and the quadratic penalty in Eq. (7) is not the KL it is claimed to approximate. This is load-bearing for the tractability of the whole derivation. At minimum, add diagnostics (e.g., normality tests on pre-logits, effective sample size of the IS weights) or explicitly state the Gaussian model as a heuristic and test robustness by comparing with heavier-tailed or mixture proposals.
  3. [§3.3 Eq. (14)] The iterated update (14) is called adaptive importance sampling, but no convergence argument is given that the sequence of self-normalized importance-sampling estimates converges to the minimizer u* of D_KL(Q*||Q_{U,σ²}). The paper cites general AIS literature but does not verify its conditions (e.g., bounded weights, ergodicity, adaptation rate). The empirical convergence in Fig. 3 is for reward values, not for the proposal distribution converging to Q*. Without a consistency result, the theoretical engine is incomplete; please provide a convergence statement or present the update as a heuristic.
  4. [§5 Tables 1–2] The same reward model used for optimization is used as the primary evaluation metric, creating a circularity risk: higher reward values may reflect more aggressive reward hacking rather than better alignment. The GPT-4 win-rate comparisons, which mitigate this concern, are mixed: e.g., Table 2 shows AISP below BoN for Gemma3&Eurus on HH-RLHF (40.0 vs 52.3) and essentially tied for Vicuna&UltraRM SHP (35.3 vs 34.3 with 30.3 draws). The abstract's claim that AISP 'achieves higher rewards than other reward-based test-time alignment methods' is supported only on the reward metric, not consistently on the independent win-rate metric. Please restrict the claim accordingly and report uncertainty or statistical tests for the win rates.
minor comments (5)
  1. [Abstract and Fig. 3] 'Outperforms best-of-n sampling in terms of rewards over the number of used samples' — Fig. 3 shows AISP is worse at early iterations; please specify that the advantage appears at later iterations, or report the area under the reward curve.
  2. [Eq. (8)] The exponent of p(V|0,σ²) is written with Σ z_t^T z_t, but it should be Σ v_t^T v_t (or Σ (v_t-0)^T(v_t-0)).
  3. [§3.4 Eq. (16)] The denominator as printed is garbled ('P(|V|j...)'); it should be Σ_j p(z_t|y_t=y_j) P(y_t=y_j).
  4. [Table 2] Typo 'Vicun & UltraRM' should be 'Vicuna & UltraRM'. Also, since the win rates are averages over three trials, reporting standard errors or confidence intervals would help assess significance.
  5. [Appendix C.6 and Algorithm 1] The KL estimates in Table 3 are computed from a single generated response per prompt (Appendix C.6), so they are likely high-variance; consider reporting standard errors. In Algorithm 1, line 19, the notation 'y(U*) = argmax_{V∈V} y(V)' is confusing; y(U*) is the response generated by applying the mean perturbation U*, not an argmax over an existing set.

Circularity Check

0 steps flagged

No significant circularity: the MPPI/importance-sampling derivation is self-contained and attributes its key identity to Williams et al. (2018); the stated Gaussian assumption is an openly declared hypothesis, not a conclusion smuggled in.

full rationale

Walking the claimed derivation chain: Eq. (7) defines a stochastic-control objective over Gaussian perturbation trajectories, J = -E_{V~Q}[r(x,y(V))] + λD_KL(Q_{U,σ²}||P), with P also Gaussian. Eq. (9) defines the free energy F = log E_P exp(r(x,y(V))/λ). Theorem 3.1 is a direct Jensen-inequality bound, with the proof supplied in Appendix A.1; the equality case q*(V) ∝ exp(r/λ)p(V) is derived, not assumed as the target. Theorem 3.2 is explicitly attributed to Williams et al. (2018), and the proof is reproduced in Appendix A.2; it is not a self-citation, and it is an exact first-order condition for minimizing D_KL(q*||Q_{U,σ²}) within the Gaussian family. Equations (12)–(14) are the corresponding self-normalized importance-sampling estimators; no fitted parameter is relabeled as a prediction. Hyperparameters (σ², λ, α) are tuned on training prompts and reported as hyperparameters (Appendix C.2, Table 4), not packaged as predictions. The Gaussian pre-logit assumption is stated explicitly as an assumption and justified only by analogy to Lee et al. (2018) with the words 'we can hypothesize'; this is a validity/robustness risk, not a circular reduction. Similarly, Table 3's response-level KL of 140.9 nats shows that the surrogate trajectory KL in Eq. (7) is not identical to the response-distribution KL discussed in Section 2.1, but the paper never defines the regularizer in terms of the response distribution; the mismatch is a correctness gap in the 'KL-constrained alignment' interpretation, not a step that assumes its own conclusion. The reward model being used both as the optimization signal and as the primary evaluation metric is a limitation of the empirical evaluation, but it does not make the derivation circular: AISP could have failed to outperform BoN on that metric. Overall, no load-bearing step reduces to its inputs by construction, and there are no author self-citations carrying the argument.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new entities (particles, forces, dimensions) are postulated. The Gaussian pre-logit perturbation is a design choice of the algorithm whose validity is assessed through performance; it has no independent out-of-paper falsifiable handle.

free parameters (6)
  • σ² (perturbation variance) = 0.5 (default); 0.7 for several settings (Table 4)
    Gaussian noise variance injected into pre-logits; controls exploration range; tuned per (model, dataset, reward model) on 10 training prompts; Appendix D.1 shows a sensitivity peak around 0.5.
  • λ (importance-weight temperature) = 0.1–0.7 for UltraRM; 60–480 for Eurus (Table 4)
    Sharpness of the softmax weights in Eq. (13)/(19); reward-scale dependent; tuned per setting; near-zero λ makes AISP equivalent to BoN (Theorem 3.3).
  • α (MPPI relaxation of baseline distribution) = 0.99–0.99999 (Table 4)
    Biases the baseline distribution P toward N(αû, σ²); α near 1 makes the quadratic penalty term negligible; tuned per setting.
  • τ (control horizon in tokens) = not specified in main text or Algorithm 1
    Number of tokens over which the perturbation is applied (Eq. 5, t∈[1,τ]); no value is given anywhere visible; determines how much of the response is controlled and the dimensionality d×τ of the Gaussian trajectory.
  • n, κ (samples per iteration, iterations) = n=κ=32 in main experiments; (b,n) variants in Batched AISP (Fig. 4)
    Budget parameters; performance depends on them (Fig. 4 shows n=2 fails to beat BoN). Not curve-fitted, but chosen.
  • BoN baseline temperature and top-p = temperature ∈ [0.4,1.0], p ∈ [0.7,0.95] (Appendix C.2)
    The comparison baseline is tuned on 10 training prompts, so the reported margin is tuned-vs-tuned; the margin may depend on these choices.
axioms (5)
  • domain assumption Base LLM pre-logit distribution is Gaussian: p_LLM(z_t|y_<t) = N(φ_LLM(y_<t), σ²I)
    Invoked in Section 3.1 Eq. (8) to obtain closed-form IS weights and the quadratic KL; heuristic justification only in Section 3.4 via the softmax/exponential-family analogy (Lee et al. 2018, image classifiers).
  • domain assumption Quadratic perturbation penalty ||u||²/(2σ²) approximates KL-constrained-RL alignment to the base LLM
    Eq. (7) claims a KL-constrained objective 'similar to' Eq. (2), but the actual D_KL(Q||P) = Σu_t^Tu_t/(2σ²) is a norm on the control, not the response-distribution KL; Table 3 shows empirical response KL up to 140.9 nats at tuned defaults.
  • domain assumption Shared-covariance Gaussian class-conditional pre-logits justify the softmax (Eqs. 15–17)
    Section 3.4: softmax is consistent with p(z_t|y_t)=N(µ_yi, Σ); used only as plausibility for the Gaussian assumption, transposed from Lee et al. 2018 for image classifiers.
  • ad hoc to paper Iterated adaptive-IS update (Eq. 14) converges to the minimizer of Eq. (7)
    Related Work states 'AISP converges to the optimal distribution'; Theorem 3.2 (Williams et al.) is a one-shot expectation identity; no proof that the fixed point of û^{k+1}=Σw̄v^k minimizes J.
  • standard math Standard free-energy, importance-sampling, and softmax-limit math
    Jensen's inequality, self-normalized IS, and the softmax-to-argmax limit (Theorem 3.3) are standard background.

pith-pipeline@v1.3.0-alltime-deepseek · 21445 in / 25120 out tokens · 242395 ms · 2026-08-04T07:18:04.860275+00:00 · methodology

0 comments
read the original abstract

Test-time alignment of large language models (LLMs) attracts attention because fine-tuning of LLMs requires high computational costs. In this paper, we propose a new test-time reward-guided alignment method called adaptive importance sampling on pre-logits (AISP) on the basis of the sampling-based model predictive control with the stochastic control input. AISP applies the Gaussian perturbation into pre-logits, which are outputs of the penultimate layer, so as to maximize expected rewards with respect to the mean of the perturbation. We demonstrate that the optimal mean is obtained by importance sampling with sampled rewards. AISP outperforms best-of-n sampling in terms of rewards over the number of used samples and achieves higher rewards than other reward-based test-time alignment methods.

Figures

Figures reproduced from arXiv: 2510.26219 by Haru Kuroki, Hiroshi Takahashi, Kazumune Hashimoto, Sekitoshi Kanai, Tsukasa Yoshida.

Figure 1
Figure 1. Figure 1: Illustration of AISP. n input trajectries {{v i t} τ t=1} n i=1 are sampled from N (ut, σ2I). The input v i t is added to the pre-logit zt, which is obtained by applying LLMs to the past tokens y i <t. The t-th token y i t is sampled and concatenated with the past tokens y i <t. When y i t is the end-of-sequence token, the rewards of {y(V i )} n i=1 are evaluated and used in adaptive importance sampling fo… view at source ↗
Figure 2
Figure 2. Figure 2: Schematic illustration of computational costs (vertical: parallelism; horizontal: iterations) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Rewards of Batched AISP and BoN us￾ing Llama&UltraRM on SHP for five trials [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Sample efficiency to improve rewards: reward curve against k iterations. For each itera￾tion, both methods generate 32 samples. BON AISP1 AISP2 AISP3 AISP4 −4.8 −4.6 −4.4 −4.2 −4.0 Rewards [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompts for GPT-4 evaluation. {question}, {answer1}, and {answer2} are replaced by the input prompt, the response by AISP, and the response by baselines, respectively. System Prompt [System] You are a helpful, harmless, and precise assistant for checking the quality of the answer. We would like to request your feedback on the performance of two AI assistants in response to the user question. Please rate th… view at source ↗
Figure 6
Figure 6. Figure 6: Rewards at the last iterations on SHP with Llama3 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Reward curve against iterations on SHP with Llama3 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 3
Figure 3. Figure 3: D.3 AVERAGE REWARDS FOR DIFFERENT SETTINGS OF κ, n AND N Tab. 5 lists the average rewards when using κ = 16, n = 32, and N = 512. Average rewards of AISP are higher than those of BoN in this setting. E LLM USAGE In addition to using LLMs for experiments, we utilized LLMs to correct grammatical errors and to rephrase some sentences to improve the naturalness of English expressions of some parts of this pape… view at source ↗
Figure 8
Figure 8. Figure 8: Reward curve against iterations on SHP (top) and HH-RLHF (bottom). AISP (Mean at [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 8 linked inside Pith

  1. [3]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R ´e, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

    Pith/arXiv arXiv

  2. [5]

    Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639,

    Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639,

    Pith/arXiv arXiv

  3. [6]

    Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.arXiv preprint arXiv:2503.21878,

    Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.arXiv preprint arXiv:2503.21878,

    Pith/arXiv arXiv

  4. [8]

    Rl with kl penalties is better viewed as bayesian inference

    Tomasz Korbak, Ethan Perez, and Christopher Buckley. Rl with kl penalties is better viewed as bayesian inference. InFindings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083–1091,

    2022

  5. [10]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Pith/arXiv arXiv

  6. [11]

    Asymptotics of language model alignment

    Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. In2024 IEEE International Symposium on Informa- tion Theory (ISIT), pp. 2027–2032. IEEE,

    2027

  7. [12]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

    12 Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

    Pith/arXiv arXiv 1909

  8. [21]

    The standalone server has NVIDIA®A100 (VRAM 40 GB) and Intel®Xeon®Gold 5318Y CPU @ 2.10GHz with 1 TB memory

    C DETAILED EXPERIMENTAL SETUP C.1 COMPUTE RESOURCES We utilized both a standalone server and a shared GPU cluster constructed within our organization. The standalone server has NVIDIA®A100 (VRAM 40 GB) and Intel®Xeon®Gold 5318Y CPU @ 2.10GHz with 1 TB memory. Shared GPU cluster assigns two GPUs of NVIDIA®H100 (VRAM 80 GB) and 24 cores of Dual Intel Xeon P...

    2024

  9. [128]

    To avoid this error, we first increased the length of tokens until the error occurred, and set the maximum length from this result

    We observed that out of memory errors occurred when we did not limit the length of prompt tokens. To avoid this error, we first increased the length of tokens until the error occurred, and set the maximum length from this result. We limited the length of prompt tokens to 1900 for vicuna 7B and to 2600 for Llama3 8B due to the limited computational resourc...

    1900

  10. [2018]

    Cascade reward sam- pling for efficient decoding-time alignment.arXiv preprint arXiv:2406.16306, 2024a

    Bolian Li, Yifan Wang, Anamika Lochab, Ananth Grama, and Ruqi Zhang. Cascade reward sam- pling for efficient decoding-time alignment.arXiv preprint arXiv:2406.16306, 2024a. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your language models can align themselves without finetuning. InProc. ICLR, 2024b. Hunter Lightman, Vineet Ko...

    Pith/arXiv arXiv

  11. [2019]

    (9) satisfies−λF(r, p,x, λ)≤J(x, U)and the equality holds if q∗(V) = 1 η exp 1 λ r(x,y(V)) p(V)(20) whereηis a normalization constant given byη= R Rd×τ exp 1 λ r(x,y(V)) p(V)dV

    A PROOFS A.1 PROOF OFTHEOREM3.1 Theorem.Free energy Eq. (9) satisfies−λF(r, p,x, λ)≤J(x, U)and the equality holds if q∗(V) = 1 η exp 1 λ r(x,y(V)) p(V)(20) whereηis a normalization constant given byη= R Rd×τ exp 1 λ r(x,y(V)) p(V)dV. Proof.Similar to importance sampling,Fcan be written by usingQas F(r, p,x, λ) = log Z exp 1 λ r(x,y(V)) q(V) q(V) p(V)dV (2...

    2018

  12. [2022]

    Theoretical guarantees on the best-of-n alignment policy

    Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879,

    Pith/arXiv arXiv

  13. [2023]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei

    URLhttps: //lmsys.org/blog/2023-03-30-vicuna/. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Proc. NeurIPS, 30,

    2023

  14. [2024]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al

    URLhttps://github.com/meta-llama/ llama3/blob/main/MODEL_CARD.md. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,

    Pith/arXiv arXiv

  15. [2025]

    Evaluation of best-of-n sampling strategies for language model alignment.Transactions on Machine Learning Research

    Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment.Transactions on Machine Learning Research. Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model al...

    2024