Alignment-Aware Decoding

Fr\'ed\'eric Berdoz; Luca A. Lanzend\"orfer; Ren\'e Caky; Roger Wattenhofer

REVIEW 4 major objections 6 minor 1 cited by

Decoding with the DPO-to-SFT log-ratio performs implicit token-level reward optimization at inference, and the paper shows it consistently improves alignment over standard baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AAD decodes by greedily maximizing the per-token DPO/SFT log-likelihood ratio under a min-p plausibility filter and reports higher oracle-reward win rates than greedy, best-of-N, and emulated fine-tuning baselines.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A clean, well-written decoding trick that is a direct descendant of contrastive decoding and DPO's implicit reward; the theory holds, but the empirical claims are over-sold by a non-independent reward-model evaluation. the 4 major comments →

arxiv 2509.26169 v2 pith:CIDCZ3O6 submitted 2025-09-30 cs.LG

Alignment-Aware Decoding

Fr\'ed\'eric Berdoz , Luca A. Lanzend\"orfer , Ren\'e Caky , Roger Wattenhofer This is my paper

classification cs.LG

keywords alignment-aware decodingDPOinference-time alignmenttoken-level rewardpreference optimizationdecoding strategyiterative DPOreward model

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces alignment-aware decoding (AAD), an inference-time method that replaces the next-token score with the log-likelihood ratio between a DPO-aligned model and its SFT reference. The theoretical argument is that DPO turns the model into a reward function, but even the optimal DPO policy inherits the SFT model's biases; AAD directly optimizes the reward component at each decoding step, restricted to a plausible token set to avoid degenerate outputs. Empirically, across six preference datasets and model families of four different scales, AAD consistently achieves higher oracle-reward scores and win rates than greedy SFT, greedy DPO, best-of-2, and emulated fine-tuning, under comparable compute. The paper also demonstrates that AAD-generated completions can serve as synthetic preference data for iterative DPO, nearly closing the gap to full-data training when only ten percent of the data is available. A sympathetic reader would care because alignment gains come without extra training, external reward models, or parameter access.

Core claim

The central claim is that the DPO-trained model's token-level log-likelihood ratio with respect to its SFT reference is a usable token reward function, and greedy maximization of this ratio over a min-α-filtered candidate set yields reliably better alignment than standard decoding. The paper formalizes this by showing that even the optimal DPO policy can assign lower probability to a higher-reward completion when the SFT prior is biased, so likelihood-based decoding leaves alignment gains on the table. AAD substitutes the ratio ν(y'|x,y_<t) = log(π_DPO/π_SFT) for the usual next-token score, selecting the token in V_α that maximizes it. This requires no training beyond standard DPO and no ext

What carries the argument

The central object is the token-level reward ratio ν(y'|x) = log(π_DPO(y'|x) / π_SFT(y'|x)), which inherits the Bradley-Terry reward interpretation of DPO. AAD combines this ratio with min-α filtering, defining V_α as the set of tokens whose DPO probability is at least α times the maximum, so the ratio is maximized only among tokens the aligned model finds plausible. The theoretical grounding is an inequality showing that the optimal DPO policy can rank a lower-reward completion above a higher-reward one due to SFT prior bias; AAD's greedy ratio maximization directly optimizes the reward component of that decomposition. An optional entropy threshold stabilizes beam search by applying the rat

Load-bearing premise

The load-bearing assumption is that the oracle reward models used for evaluation are a valid, unbiased proxy for human alignment, and that AAD's win rates are not an artifact of output length or verbosity; all headline numbers come from scoring continuations with these reward models.

What would settle it

Take a random subset of prompts from one dataset, run AAD and greedy or best-of-2 baselines, and have human annotators blindly rate the pairs. If AAD's win rate against greedy drops to chance, or if controlling for output length (for example, truncating to equal token counts) removes the oracle-reward advantage, the central claim fails. A cheaper check: measure the correlation between AAD's chosen-token ratio and response length on the evaluation set; a strong positive correlation would suggest the reward is being gamed by verbosity rather than alignment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

AAD improves oracle reward and win rate over greedy SFT, greedy DPO, best-of-2, and emulated fine-tuning across all tested preference datasets and model scales, with the largest gains on datasets where the oracle reward model is strongest.
AAD remains competitive with best-of-N sampling that uses the oracle reward model for selection, despite being compute-equivalent to best-of-2 and using only a model aligned on ten percent of the data.
AAD's advantage persists under data scarcity, outperforming best-of-2 at every training-data scale tested, from one percent to the full split.
AAD is more robust to the DPO regularization parameter than greedy or best-of-2 decoding, showing the smallest relative alignment loss as β increases.
AAD-generated completions, paired with nucleus-sampled negatives, can be used in iterative DPO; with only ten percent of the preference data this nearly closes the gap to a model trained on the full dataset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If AAD's advantage is driven by correcting SFT prior bias rather than by output length or verbosity, its win rates should transfer to human preference judgments; a small human evaluation on a random prompt subset would be the cleanest test of that transfer.
The method suggests a general recipe: any training procedure that yields an implicit reward relative to a base model (for example, other preference-optimization losses) could define a similar decoding reward, potentially broadening the approach beyond DPO.
Because AAD only requires an aligned model and its reference, it could be combined with larger search budgets, adaptive token filters, or entropy thresholds that trade fluency against alignment; the paper shows beam search with an entropy threshold stabilizes and helps, but does not explore adaptive settings.
The iterative-DPO result implies AAD can act as a data amplifier: in low-data regimes the model itself generates the preference signal for subsequent training rounds, though the paper notes the win-rate gap between AAD and best-of-2 narrows as iterations progress.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A clean, well-written decoding trick that is a direct descendant of contrastive decoding and DPO's implicit reward; the theory holds, but the empirical claims are over-sold by a non-independent reward-model evaluation.

read the letter

Quick take: the core idea is a straightforward combination of contrastive decoding's expert-amateur logit gap and DPO's reward reparameterization. The derivation in Section 4 is correct, and the token-level ratio is a valid greedy approximation to sequence-level reward maximization. What's genuinely new is applying that ratio from the DPO/SFT pair to greedy decoding with min-alpha filtering, and the synthetic-data iteration experiment is a nice addition. The writing is clear, and the code link is provided.

The empirical story, though, has real soft spots. The evaluation uses oracle reward models that were trained on either the same preference dataset or the same preference construct as the DPO training set. AAD explicitly optimizes log(pi_DPO/pi_SFT), which is a reparameterization of that same reward; evaluating on correlated reward models is not an independent test. The stress-test note also lands on length: the qualitative examples show AAD outputs are longer and more elaborate, and no length-controlled results are reported. Reward models for helpfulness tend to favor longer, more detailed responses, so a chunk of the win rate could just be verbosity. I'd want to see length-matched evaluation or at least a length covariate before believing the magnitude of the gains.

There are also missing comparisons. The related work cites DeRa and contrastive decoding, but neither is benchmarked. Given that AAD is essentially contrastive decoding with a DPO/SFT pair, that omission is noticeable. The EFT baseline is described as using pi_SFT for both base and reference model — that's not the standard EFT setup, and I'd want a citation or precise configuration to know what was actually run. No error bars or seed counts in Tables 1/2 either, which matters for win rates in the 0.5-0.8 region.

That said, I don't think any of this kills the paper. The method is simple, plausible, and likely improves alignment somewhat; the theory is sound; and the synthetic-data experiment is a nice practical angle. It just needs to be evaluated fairly. As it stands, the central claim 'AAD improves alignment' is supported only by a reward-model-based metric that is not independent of the optimization target and not controlled for length.

Who should read this: anyone working on inference-time alignment or decoding-time steering. It deserves a serious referee, but the referee should push for major revisions — length-controlled evaluation, independent reward models, error bars, and direct comparisons to contrastive decoding and DeRa. I'd tell the editor to send it to review, and I'd tell the authors to fix the evaluation before I'd accept it.

Referee Report

4 major / 6 minor

Summary. The paper proposes alignment-aware decoding (AAD), which at each decoding step selects the token maximizing the log-likelihood ratio log(pi_DPO / pi_SFT) over a min-alpha filtered vocabulary (Eqs. 7-9). The score is derived from the DPO optimal-policy form (Eq. 4) and interpreted as token-level implicit reward optimization. Experiments compare AAD against greedy SFT, greedy DPO, best-of-2, and a variant of EFT on six preference datasets and four base models, reporting oracle-reward scores and win rates, plus AlpacaEval win rates, data-scarcity experiments, beam-search stabilization, and iterative DPO with AAD-generated synthetic data.

Significance. If the empirical claims hold, AAD is a simple and attractive inference-time method: it needs only the SFT reference and DPO model, no additional reward model, and it can produce synthetic preference data for iterative DPO. The theoretical derivation is transparent and the token-level ratio is indeed a valid decomposition of the DPO reward. The broad dataset/model coverage and the iterative-DPO experiments are useful additions. However, the main empirical claim that AAD 'improves alignment' rests on reward-model evaluations that are not independent of the DPO training signal and on uncontrolled output length; without additional evidence the practical significance is not yet established.

major comments (4)

[Section 5, Table 1] Evaluation circularity. For Ultrachat, Argilla, OpenRLHF, and HHRLHF, the oracle reward model is trained on the full training split of the same preference dataset from which pi_DPO's 10% training subset is drawn; for Skywork/Nectar the external oracles are trained on the same preference construct. AAD's score (Eq. 7) is exactly log(pi_DPO / pi_SFT), the implicit reward derived from that same preference signal. The win rates therefore measure how well AAD optimizes the target used to train both the picker and oracle, not alignment as an independent construct. Please add an evaluation with a held-out reward model trained on independent judgments, or human evaluation, or at least a demonstration that AAD's gains persist when the oracle is not from the same preference source.
[Section 5, Tables 1-2] No uncertainty quantification. The tables report point estimates of average reward and win rate with no standard errors, confidence intervals, number of prompts, or number of seeds. Several win-rate differences are small (e.g., Table 1 HHRLHF, Llama 3B: AAD wins 0.61-0.64 against all baselines; Table 2 Nectar Qwen 0.6B: AAD vs Greedy SFT 0.52). Without variance estimates the claim that AAD 'consistently' outperforms is not supported. Please report bootstrap CIs or multiple seeds and the evaluation set size.
[Section 6, Figs. 1 and 8-11] Length/verbosity confound. The qualitative examples show AAD outputs are systematically longer and more elaborate than greedy or best-of-2 outputs; no length-matched evaluation or length covariate is reported. Reward models used for evaluation are known to favor longer, more detailed responses, so the observed win rates may reflect verbosity rather than alignment. Please add length-controlled win rates (e.g., stratify by output length or use length-matched baselines) or human evaluation.
[Section 4, Eq. (7)] Token-level greedy approximation. The method greedily maximizes the per-token log-ratio, but the sequence-level reward r_DPO(x,y) = beta log(pi_DPO(y|x)/pi_SFT(y|x)) factorizes as a product of token ratios; greedy per-token maximization over a filtered vocabulary is not guaranteed to maximize the full-sequence reward, and the min-alpha filter can exclude tokens needed for coherence. The paper acknowledges this is an approximation but does not analyze how close the AAD trajectory is to the sequence-level maximizer, nor when the filter hurts. This is load-bearing for the claimed connection to implicit reward optimization and should be discussed or ablated.

minor comments (6)

[Eq. (5)] Typo: the displayed equation is missing the division bar and should read r*(x,y) = beta log(pi*(y|x)/pi_SFT(y|x)) + beta log Z(x;r*).
[Section 5, Baselines] The EFT baseline is ambiguously specified: 'using pi_SFT for both the base and reference model, and setting beta=4'. Please define the exact score and decoding procedure; as written it is unclear how this implements emulated fine-tuning.
[Figs. 3, 4; Table 4] Typo: 'Agrilla' should be 'Argilla' in Figure 3, Figure 4, and the appendix table.
[Section 2] The Related Work section ends with an incomplete sentence: 'These approaches typically' is followed by nothing. Please complete or remove.
[Figure 2] The dashed vertical line is said to indicate the compute-equivalence point, but the corresponding value of N is not stated. Please report it.
[References] The Nectar dataset is cited as Zhu et al. (2023), but the bibliography entry for Zhu et al. is about advantage-induced policy alignment. Please use the correct dataset citation or add one.

Circularity Check

1 steps flagged

No damaging circularity: the 'implicit reward optimization' claim is a transparent restatement of the DPO reward definition (Eq. 7), but the central empirical claims rest on separate oracle evaluation, an external AlpacaEval evaluator, and a BoN-oracle control.

specific steps

self definitional [Abstract; Section 4, Eq. (7)-(9)]
"Theoretically, AAD can be interpreted as implicit reward optimization... we propose to use π_DPO exactly as intended during training, that is, in combination with π_SFT as an approximate token-level reward function... ν(yt+1 |x◦y1:t) = log πDPO(yt+1 |x◦y1:t) / πSFT(yt+1 |x◦y1:t) . (7)"

AAD's token score ν in Eq. (7) is defined to be exactly the DPO implicit reward log(π_DPO/π_SFT) (up to the β scale factor, which the paper says it omits as ranking-invariant), and Eq. (8) decodes by argmax over ν. The abstract's claim that AAD 'can be interpreted as implicit reward optimization' is therefore the method's own construction restated: the conclusion is identical to the definition of the score. The step is non-damaging because it is explicitly attributed to DPO/Rafailov et al. and used only as motivation; the empirical win-rate claims are not inferred from this definition but measured with separately trained or external oracle reward models and the AlpacaEval evaluator.

full rationale

The derivation chain is essentially: DPO theory (Rafailov et al., cited as external prior work) gives the implicit reward r = β log(π_DPO/π_SFT); AAD greedily maximizes that ratio with a min-α plausibility filter; the paper then reports win rates under oracle reward models. The only step that reduces by construction is the framing sentence 'AAD can be interpreted as implicit reward optimization,' since AAD's score is defined as the DPO reward. I flag this as a mild, transparent restatement, not a load-bearing circularity: the paper explicitly says it 'propose[s] to use π_DPO exactly as intended during training' and credits Rafailov et al. 2023/2024b, so no independent theorem is claimed. The empirical contributions are separately supported: (i) win rates are computed with oracle reward models trained on a different (full) split or, for Skywork/Nectar, external reward models, and with the external AlpacaEval GPT-4 evaluator (Table 2); (ii) the strongest same-signal concern is addressed by the authors themselves in Fig. 2, which notes the BoN-Oracle baseline uses the oracle for both selection and evaluation while AAD only uses a model aligned on 10% of the data, and AAD still wins at equal compute on Argilla; (iii) no parameter is fitted to the evaluator in the main results (α=0.1 is fixed; the β sweep in Fig. 4 is a robustness study, and the r*-is-β-independent explanation follows from Eq. (4), not from the data). There are no self-citation chains and no imported uniqueness theorems. The skeptic's concerns that the trained oracles share preference signal with the DPO model and that AAD outputs may be longer/verbosity-favored are evaluation-validity (correctness) risks, not definitional circularity, so they do not raise the circularity score beyond 1.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper introduces no new scientific postulates or entities; its theoretical engine is the DPO reward reparameterization. The central empirical claims rest on two tuned hyperparameters (alpha, beta) and on reward-model evaluation, with length bias and baseline-choice confounds not fully controlled.

free parameters (3)

alpha (min-p plausibility threshold) = 0.1 (0.7 in Fig. 6)
Defines the allowed token set V_alpha in Eq. (9). Chosen to prevent over-optimization; no main ablation is shown, and the beam-search experiment uses a different value.
beta (DPO regularization coefficient) = 0.1 (varied in Fig. 4)
Standard DPO hyperparameter used to train pi_DPO. Fig. 4 shows AAD is more robust to beta than baselines, but the reported scores still depend on this choice.
entropy threshold tau (beam search) = not reported
Introduced in Section 6 to stabilize beam search; the paper does not state the threshold value or a selection procedure.

axioms (5)

domain assumption Bradley-Terry model (Eq. 1) governs pairwise preference probabilities.
The DPO theory and the trained reward models both assume this preference model.
standard math DPO closed-form optimal policy (Eq. 4): pi*(y|x) ∝ pi_SFT(y|x) exp(r*(x,y)/beta).
Inherited from Rafailov et al. 2023; used to derive Eq. (6) and justify the token-level ratio as a reward.
domain assumption The trained pi_DPO approximates pi*, so the per-token log-ratio is a valid token-level reward.
This is the empirical bridge from DPO theory to AAD; if pi_DPO is poorly calibrated, the ratio is not a reliable reward. See Section 4 and Rafailov et al. 2024b.
ad hoc to paper Greedy maximization of the token reward over V_alpha is a reliable proxy for sequence-level reward optimization, and alpha=0.1 prevents over-optimization.
Introduced as the method's central heuristic (Eqs. 8-9); no formal guarantee is given.
domain assumption Oracle/picker reward models measure the alignment property of interest.
All quantitative alignment claims in Tables 1-2 and Figures 2-5 are scores from these reward models, not human judgments.

reviewed 2026-08-04 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Alignment-Aware Decoding." pith.science (2026). https://pith.science/paper/CIDCZ3O6

@misc{pith2026250926169,
  author       = {Pith},
  title        = {Pith review of: Alignment-Aware Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CIDCZ3O6}},
  note         = {Machine review of arXiv:2509.26169}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Alignment of large language models remains a central challenge in natural language processing. Preference optimization has emerged as a popular and effective method for improving alignment, typically through training-time or prompt-based interventions. In this paper, we introduce alignment-aware decoding (AAD), a method to enhance model alignment directly at inference. Theoretically, AAD can be interpreted as implicit reward optimization, yet it requires no specialized training beyond the standard DPO setup. Empirically, AAD consistently outperforms strong baselines across diverse alignment benchmarks and model scales. Moreover, in data-constrained settings, AAD can produce high-quality synthetic data to improve alignment under standard decoding, providing a practical solution when labeled data is limited.

Figures

Figures reproduced from arXiv: 2509.26169 by Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Ren\'e Caky, Roger Wattenhofer.

**Figure 1.** Figure 1: Qualitative comparison of AAD againgst other decoding strategies. Greedy continuations are generated by feeding the prompt together with the current AAD prefix back into the model and greedily selecting the next token, revealing where the greedy trajectory diverges from AAD. AAD identifies the Chihuahua as the smallest recognized breed of dog, making the distinction that it refers to an officially recogn… view at source ↗

**Figure 2.** Figure 2: AAD versus BoN. We evaluate AAD against three selection strategies on Argilla and Skywork datasets for different values of N: (i) BoN using the oracle, (ii) BoN using the picker, and (iii) random selection among N completions. AAD remains competitive even against BoN-Oracle reward model, a setting that is by design unfavorable to AAD, since the oracle is used both for BoN selection and evaluation, whereas … view at source ↗

**Figure 3.** Figure 3: Performance of AAD across different training dataset sizes on the Skywork dataset. Results show that AAD consistently outperforms bestof-2 at every data scale, providing clear evidence of its robustness in low-data regimes. model πSFT via DPO to obtain πDPO. This setup allows to simulate two conditions simultaneously: the availability of a strong oracle reward model for evaluation, and the scarcity of pr… view at source ↗

**Figure 5.** Figure 5: Effect of iterative DPO. Results show that iterative DPO using AAD-generated data substantially improves alignment, approaching full-dataset performance (100%) with only 10% of the original data. Win rates against the original πDPO (baseline), using the same decoding scheme, are shown above the bars. Iteration i indicates the average oracle score of a model that has undergone DPO using AAD-generated data,… view at source ↗

**Figure 6.** Figure 6: Effect of beam size and entropy threshold on performance for the Skywork dataset with α = 0.7. Without entropy thresholding, scores rapidly degrade as the number of beams increases due to beam collapse. This mechanism enables larger beam sizes to yield improved alignment, while also reducing the computational cost compared to standard beam search. Stabilizing beam search via entropy thresholding. We also i… view at source ↗

**Figure 7.** Figure 7: Histograms of score differences between AAD and best-of-2 sampling across successive [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

**Figure 8.** Figure 8: Qualitative comparison of decoding strategies. AAD produces an interpretable explanation, correctly assigning the product to Smart Home based on its function as an intelligent thermostat that automates heating and cooling systems. Greedy Continuations are generated by iteratively feeding the prompt together with the current AAD prefix back into the model and greedily selecting the next token, revealing w… view at source ↗

**Figure 9.** Figure 9: Qualitative comparison of decoding strategies. Greedy continuations are generated by iteratively feeding the prompt together with the current AAD prefix back into the model and greedily selecting the next token, revealing where the greedy trajectory diverges from AAD. AAD provides a balanced and contextually grounded judgment, explicitly noting that children under 13 may not be suitable viewers due to inte… view at source ↗

**Figure 10.** Figure 10: Qualitative comparison of decoding strategies. AAD recognizes the situational context and produces an appropriate, themed reply that includes the iconic phrase “May the Force be with you.” Greedy continuations are generated by iteratively feeding the prompt together with the current AAD prefix back into the model and greedily selecting the next token, revealing where the greedy trajectory diverges from AA… view at source ↗

**Figure 11.** Figure 11: Qualitative comparison of decoding strategies. Greedy continuations are generated by iteratively feeding the prompt together with the current AAD prefix back into the model and greedily selecting the next token, revealing where the greedy trajectory diverges from AAD. AAD concludes that the tweet can be perceived as offensive, grounding this in the negative connotation of the word “idiot” and acknowledgin… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Convex Optimization for Alignment and Preference Learning on a Single GPU
cs.LG 2026-05 unverdicted novelty 6.0

COALA applies convex optimization reformulations of neural networks to direct preference optimization, claiming single-GPU training with ~18% of DPO's TFLOPs and competitive performance on multiple datasets and models...

Reference graph

Works this paper leans on

15 extracted references · 5 linked inside Pith · cited by 1 Pith paper

[1]

Con- crete problems in ai safety.arXiv preprint arXiv:1606.06565,

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´e. Con- crete problems in ai safety.arXiv preprint arXiv:1606.06565,

Pith/arXiv arXiv
[3]

Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

Pith/arXiv arXiv
[6]

Google Nest Learning Thermostat

Figure 7 presents histograms for the individual iterations, illustrating the score differences between AAD and best-of-2 sampling. 14 Under review as a conference paper at ICLR 2026 0 20Frequency Win rate: 72.8% 0 0 20Frequency Win rate: 62.5% 1 0 20Frequency Win rate: 62.4% 1' 30 20 10 0 10 20 30 Score Difference 0 20Frequency Win rate: 57.3% 2 x < 0 x 0...

2026
[9]

Scaling laws for reward model overoptimization in direct align- ment algorithms.Advances in Neural Information Processing Systems, 37, 2024a

Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct align- ment algorithms.Advances in Neural Information Processing Systems, 37, 2024a. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From $r$ to $qˆ*$: Your language...

Pith/arXiv arXiv
[10]

Se- lective preference optimization via token-level reward function estimation.arXiv preprint arXiv:2408.13518,

Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Se- lective preference optimization via token-level reward function estimation.arXiv preprint arXiv:2408.13518,

arXiv
[12]

Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425,

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425,

Pith/arXiv arXiv
[13]

Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

Pith/arXiv arXiv 1909
[14]

Table 3: Accuracy of the reward models trained on the different preference datasets

12 Under review as a conference paper at ICLR 2026 A APPENDIX A.1 ACCURACIES OF REWARD MODELS In Table 3, we report the accuracies of the picker and oracle reward models on the evaluation sets across all datasets. Table 3: Accuracy of the reward models trained on the different preference datasets. Oracles are traind on the full training split, and pickers...

2026
[2019]

Contrastive decoding: Open-ended text generation as optimization

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023a. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulraja...

2024
[2020]

Orpo: Monolithic preference optimization without refer- ence model

Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without refer- ence model. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,

2024
[2021]

Deal: Decoding-time alignment for large lan- guage models.arXiv preprint arXiv:2402.06147,

James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-time alignment for large lan- guage models.arXiv preprint arXiv:2402.06147,

arXiv
[2022]

Iterative reasoning preference optimization.Advances in Neural Information Processing Systems, 37,

11 Under review as a conference paper at ICLR 2026 Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization.Advances in Neural Information Processing Systems, 37,

2026
[2023]

Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model

Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

2023
[2024]

Inference-time alignment in continuous space.arXiv preprint arXiv:2505.20081,

Yige Yuan, Teng Xiao, Li Yunfan, Bingbing Xu, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space.arXiv preprint arXiv:2505.20081,

arXiv
[2025]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

10 Under review as a conference paper at ICLR 2026 Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models,

2026

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.

[1] [1]

Con- crete problems in ai safety.arXiv preprint arXiv:1606.06565,

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´e. Con- crete problems in ai safety.arXiv preprint arXiv:1606.06565,

Pith/arXiv arXiv

[2] [3]

Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,

Pith/arXiv arXiv

[3] [6]

Google Nest Learning Thermostat

Figure 7 presents histograms for the individual iterations, illustrating the score differences between AAD and best-of-2 sampling. 14 Under review as a conference paper at ICLR 2026 0 20Frequency Win rate: 72.8% 0 0 20Frequency Win rate: 62.5% 1 0 20Frequency Win rate: 62.4% 1' 30 20 10 0 10 20 30 Score Difference 0 20Frequency Win rate: 57.3% 2 x < 0 x 0...

2026

[4] [9]

Scaling laws for reward model overoptimization in direct align- ment algorithms.Advances in Neural Information Processing Systems, 37, 2024a

Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct align- ment algorithms.Advances in Neural Information Processing Systems, 37, 2024a. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From $r$ to $qˆ*$: Your language...

Pith/arXiv arXiv

[5] [10]

Se- lective preference optimization via token-level reward function estimation.arXiv preprint arXiv:2408.13518,

Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Se- lective preference optimization via token-level reward function estimation.arXiv preprint arXiv:2408.13518,

arXiv

[6] [12]

Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425,

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425,

Pith/arXiv arXiv

[7] [13]

Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

Pith/arXiv arXiv 1909

[8] [14]

Table 3: Accuracy of the reward models trained on the different preference datasets

12 Under review as a conference paper at ICLR 2026 A APPENDIX A.1 ACCURACIES OF REWARD MODELS In Table 3, we report the accuracies of the picker and oracle reward models on the evaluation sets across all datasets. Table 3: Accuracy of the reward models trained on the different preference datasets. Oracles are traind on the full training split, and pickers...

2026

[9] [2019]

Contrastive decoding: Open-ended text generation as optimization

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023a. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulraja...

2024

[10] [2020]

Orpo: Monolithic preference optimization without refer- ence model

Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without refer- ence model. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,

2024

[11] [2021]

Deal: Decoding-time alignment for large lan- guage models.arXiv preprint arXiv:2402.06147,

James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-time alignment for large lan- guage models.arXiv preprint arXiv:2402.06147,

arXiv

[12] [2022]

Iterative reasoning preference optimization.Advances in Neural Information Processing Systems, 37,

11 Under review as a conference paper at ICLR 2026 Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization.Advances in Neural Information Processing Systems, 37,

2026

[13] [2023]

Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model

Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

2023

[14] [2024]

Inference-time alignment in continuous space.arXiv preprint arXiv:2505.20081,

Yige Yuan, Teng Xiao, Li Yunfan, Bingbing Xu, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space.arXiv preprint arXiv:2505.20081,

arXiv

[15] [2025]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

10 Under review as a conference paper at ICLR 2026 Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models,

2026

Alignment-Aware Decoding

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it