REVIEW 4 major objections 6 minor 15 references
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Importance weight makes offline DPO match online alignment
desk verdict A plausible, cheap importance-sampling fix for DPO over-optimization, but the empirical claim is undercut by epsilon tuning on the same evaluation curve and the clipping definition looks inverted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the importance weight $w(x, y_w, y_l) = \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}$, the product of sequence-level probability ratios between the learned policy and the reference policy for the two responses in a preference pair. Multiplying the DAA loss by this weight reweights the expectation over $\pi_{\mathrm{ref}}$ into an expectation over $\pi_\theta$, giving an unbiased online objective from offline data; Theorem 1 then shows that the IS-weighted squared-loss gradient equals the KL gradient. The clipping rule in Eq. (13), described as a truncated importance weighting estimator, is what the paper uses in practice to keep the weights' variance from exploding during training. Section 4.3 interprets the estimator's behavior when the support assumption fails as a support constraint that halts updates, which the paper identifies as the source of its implicit early-stopping effect.
What would settle it
Run a small bandit or an LLM fine-tuning at low beta and log per-step importance weights and gradients: if, once weights hit the clipping threshold, the gradient of the clipped IS-DPO objective no longer tracks the gradient of the online DPO objective (cosine similarity to the KL gradient drops below zero, as the unweighted objective's does at small beta), then the clipping, not the unbiased reweighting, is doing the work.
Extended reading notes
Core claim
The paper's central claim is that reward over-optimization in offline DAAs is driven by the growing mismatch between the sampling distribution used to collect the preference data (the reference policy $\pi_{\mathrm{ref}}$) and the policy being trained ($\pi_\theta$), and that this mismatch can be corrected without any online sampling. The IS-DPO objective is $-\mathbb{E}_{x, y_w, y_l \sim \pi_{\mathrm{ref}}}\left[ w \log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\right)\right]$ with $w = \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}$. Theorem 1 states that, assuming $\mathrm{Supp}(\pi_{\mathrm{ref}}) = \mathrm{Supp}(\pi_\theta)$, this objective is an unbiased estimate of the online DPO objective, and the gradient of the importance-weighted squared loss equals the gradient of $\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})$, so the KL regularization that online methods enforce explicitly is now enforced through the reweighted loss. In practice the importance weights are truncated (Eq. 13) to control their high variance. The paper reports that this method spends KL divergence more efficiently: under weak regularization ($\beta = 0.01$) it keeps win rate high while staying at low KL, and across Anthropic-HH, AlpacaEval 2.0, and MT-Bench it outperforms DPO and the regularized baselines. Section 4.3 adds a mechanism: when $\pi_\theta$'s support leaves $\pi_{\mathrm{ref}}$'s, the IS estimator becomes a support constraint that stops updating, acting as an implicit early stopper.
Load-bearing premise
The theory assumes the trained policy and the reference policy cover exactly the same responses with nonzero probability, while the practical method depends on a clipping rule that has no proven bias bound; if that clipped reweighting stops matching the true online objective, the central over-optimization claim collapses.
Editorial extensions
If this is right
- Offline DAAs can recover the behavior of online DPO without ever sampling from the live policy, since the IS objective is an unbiased estimate of the online objective (Theorem 1).
- The implicit KL regularization in DAAs becomes a global constraint rather than a local one: the IS-weighted squared-loss gradient equals the KL gradient even when $\pi_\theta$ deviates from $\pi_{\mathrm{ref}}$.
- IS-DPO keeps win rates high and KL budgets low under weak regularization ($\beta = 0.01$), where DPO, RPO, and Length-DPO over-optimize and converge early.
- IS-DPO outperforms DPO, RPO, Length-DPO, and chi-PO on Anthropic-HH, AlpacaEval 2.0, and MT-Bench, with roughly a 3% win-rate gain over DPO on Anthropic-HH.
- When the policy's support leaves the reference's support, the IS estimate becomes a support constraint that stops further updates, producing an implicit early-stopping effect.
Reading between the lines
- The same importance-weight mechanism can be applied to other DAA losses, such as IPO's squared loss, because Theorem 1 only requires the expectation-reweighting step; the paper presents IS-DAAs as a general framework but instantiates and tests only IS-DPO.
- A testable extension is to measure the gradient cosine similarity (as in the 4-action bandit of Section 3.2) for the clipped estimator at the clip bound; if it degrades, the practical advantage of IS-DPO comes from the support constraint rather than from unbiasedness.
- The support-constraint reading in Section 4.3 connects IS-DAAs to conservative off-policy evaluation: the truncation threshold could be tuned as an explicit safety knob, trading update coverage against bias, a trade-off the paper leaves qualitative.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IS-DAAs, an importance-sampling correction for direct alignment algorithms such as DPO, with the goal of mitigating reward over-optimization. The central idea is to reweight offline preference pairs by the ratio π_θ/π_ref so that the off-policy DAA objective approximates an on-policy objective, and to clip the importance weights to control variance. The authors state a theorem claiming that the IS-weighted objective unbiasedly estimates an online DPO objective and that the IS-weighted squared-loss gradient equals the KL gradient. They report experiments on TL;DR summarization and instruction following (Anthropic-HH, AlpacaEval 2.0, MT-Bench) showing that IS-DPO improves over DPO, RPO, Length-DPO, and χ-PO, especially at low KL regularization strength.
Significance. If the proposed correction works as claimed, it would give a simple and computationally cheap way to recover some of the regularization behavior of online preference learning from offline data, which is a question of current interest. The paper's empirical setup -- a controlled golden-reward TL;DR environment with KL-budget curves and multiple seeds -- is well suited to measuring over-optimization, and the proposed method is straightforward to implement on top of existing DAA losses. The paper also states its limitations explicitly in Appendix A (Bradley-Terry assumption, small model scale). However, the theoretical derivation and the experimental evaluation contain load-bearing gaps: the clipping operator in Eq. (13) is written as a floor rather than a cap, the online objective in Eq. (10) is not a well-defined preference objective without a label model, the gradient identity in Eq. (12) drops a factor of two and ignores the θ-dependence of the importance weight, and the clipping-ratio ablation appears to select ϵ on the same evaluation set used for the headline win rates. These issues need to be resolved before the empirical claims can be regarded as established.
major comments (4)
- [Sec. 3.1, Eq. (13)] The clipping operation is defined as w = max(π_θ(yw|x)/π_ref(yw|x) · π_θ(yl|x)/π_ref(yl|x), ϵ). This is a floor, not a cap: it raises small importance weights to ϵ and leaves large weights unbounded, which is the opposite of the stated goal of truncating large importance ratios to control variance. The abstract and surrounding text say the importance ratio is clipped to a maximum value, and the cited estimator is the truncated importance weighting estimator. As written, the actual algorithm used in the experiments is under-specified, and the claimed bias-variance trade-off is not supported. If the intended operator is min(w, c), please correct it and provide a bias bound for the clipped estimator; if a floor is intended, explain how it reduces variance.
- [Sec. 3.1, Eq. (10)] The online DPO objective is written as an expectation over pairs (yw, yl) ~ π_θ(·|x), but no preference-label model is specified. In preference learning, a pair sampled from the current policy is not automatically ordered as 'winning' and 'losing'; an external label model (e.g., a Bradley-Terry reward model or human labels) is required to define yw and yl. Without such a model, Eq. (10) is not a well-defined preference objective, and the unbiasedness claim in Theorem 1 inherits this ambiguity. Please define the label distribution explicitly and state whether the expectation in Eq. (10) is taken over that distribution.
- [Sec. 3.1, Eq. (12) and Appendix B.1] The claimed identity E_{π_ref}[w ∇_θ (1/2 ρ^2)] = ∇_θ KL(π_θ ‖ π_ref) is not established by the proof. Expanding the left side after the change of measure gives E_{π_θ pairs}[(A_{yw} - A_{yl})(∇A_{yw} - ∇A_{yl})], which equals 2 E_{π_θ}[A ∇A] = 2 ∇KL, not ∇KL; the appendix's penultimate line silently drops this factor of two. More importantly, the left side differentiates only ρ^2 while treating w as a constant, but the loss in Eq. (11) multiplies f(βρ) by w, and w depends on θ; the actual gradient of the IS objective contains ∇_θ w terms. Thus the paper does not prove that minimizing the IS-weighted squared loss minimizes KL divergence for the objective that is actually optimized. Please correct the factor and either prove the identity for the full gradient or clarify that the claim concerns a simplified regularizer that is not the one used in training.
- [Sec. 4.1, Fig. 5 and Appendix D] The clipping-ratio ablation selects log ϵ = 1.0 because it 'yields the highest win rate (≈92%)', and Appendix D states that win rates are computed on 512 prompts from the evaluation set. The paper does not state that the ablation was performed on a held-out validation split. If the same evaluation prompts are used both to select ϵ and to report the final win rate, then ϵ is tuned directly on the headline metric, which biases IS-DPO's results upward. The baselines are compared at fixed β ∈ {0.01, 0.05, 0.1} without an equivalent evaluation-driven selection of their own hyperparameters (e.g., RPO's SFT coefficient, χ-PO's χ² coefficient, Length-DPO's penalty). Please either confirm that Fig. 5 was produced on a separate validation split or describe the selection protocol, and give the baselines an equally favorable hyperparameter selection. This is essential for the central empirical claim that IS-DPO mitigates over-optimization better than existing methods.
minor comments (6)
- [Abstract / Section 1] The paper advertises IS-DAAs and says it focuses on DPO and IPO, but all experiments and ablations implement only IS-DPO. Please state whether the method has been tested with IPO or other DAA losses, or adjust the claims accordingly.
- [Table 1] The table has typesetting errors: for example, the χ-PO row shows '67.2±1.46.855.09' with missing column separators, and the AlpacaEval columns are difficult to read. Please re-typeset and verify all entries.
- [Eq. (13)] If the max in Eq. (13) is a typographical error for min, the correction should be made consistently throughout Section 3.1; the discussion of variance reduction and the description of truncated importance weighting both imply a cap, not a floor.
- [Sec. 3.2, Fig. 2] The bandit experiment does not specify how π_θ is parameterized, how gradients are computed, or how many random seeds are used. Please add these details so the cosine-similarity curves can be reproduced.
- [Appendix C] The 'golden reward model' is referred to as ground truth but is itself trained on a preference dataset; please clarify the terminology and report the validation accuracy with confidence intervals or seed variation.
- [Abstract] The code release is referred to as 'this link' without a URL in the text; please include a working link and repository identifier.
Circularity Check
The IS-DPO identity is self-contained; the only fitted-element concern is that the clipping ratio ε is selected on the same golden-reward win-rate curve used to report the headline over-optimization results.
-
fitted input called prediction
[Section 4.1, 'Clipping ratio ϵ Ablation' (Fig. 5) combined with the 'Evaluation' protocol and Figs. 3–4 main results]
"Clipping ratio ϵ Ablation.We conduct ablation studies to better understand how the clipping ratio ϵ influences policy performance and regularization in Fig. 5. As can be observed, a moderate value of logϵ= 1.0 yields the highest win rate (≈ 92%) and is associated with the highest KL divergence. These results are consistent with the bias-variance trade-off of clipping ratio ϵ."
The paper's headline claim that IS-DPO mitigates over-optimization is supported by win-rate/KL curves (Figs. 3–4) computed with the golden reward model on 512 evaluation prompts. The clipping ratio ε for IS-DPO is selected as the argmax of the same type of golden-reward win-rate curve (Fig. 5; log ε = 1.0 yields ≈92%). The paper states no held-out validation split for this ablation, so the reported best IS-DPO configuration is fitted to the headline metric rather than independently predicted. This does not affect Theorem 1, which is a standard importance-sampling identity, but it weakens the empirical 'mitigation' claim as an out-of-sample test.
full rationale
The theoretical derivation is self-contained. Equation (11) is the standard importance-sampling reweighting of the online DPO objective in Eq. (10), and Theorem 1's proof in Appendix B.1 is an algebraic application of the score-function identity; no step imports a conclusion from the authors' prior work or defines the target in terms of itself. The KL-gradient equality is a known score-function identity (modulo a possible factor-of-2 typo, which is a correctness issue, not circularity). The bandit analysis in Section 3.2 and the support-constraint discussion in Section 4.3 are illustrative interpretations, not load-bearing derivations that reduce to the paper's conclusions. Self-citations are not load-bearing: Eq. (8) is attributed to Tang et al. (external prior work) and the paper's own derivation of Eq. (9) is explicit. The only fitted-element concern is the ε ablation: because the same golden-reward win-rate protocol appears to be used both to choose log ε = 1.0 and to report the main IS-DPO advantage, the reported advantage is partly selected rather than independently predicted. The paper also under-specifies the clipped estimator in Eq. (13), where max(·, ε) is not a cap on large weights, but that is an algorithmic under-specification rather than circularity. Overall, the central identity is not circular; the empirical over-optimization claim carries a mild selection bias, giving a circularity score of 2.
Assumptions & free parameters
free parameters (2)
- clipping ratio epsilon =
log epsilon = 1.0 (chosen from ablation over 0.25, 0.5, 1.0)
- KL regularization strength beta =
varied over 0.01, 0.05, 0.1 for TL;DR; 0.05 for instruction following
assumptions (3)
- domain assumption Preferences follow the Bradley-Terry model (Eq. 1)
- domain assumption Supp(pi_ref) = Supp(pi_theta) in Theorem 1
- ad hoc to paper The DAA loss can be approximated by the weighted squared loss Taylor expansion in Eq. (7)
Cite this review
Pith. "Pith review of Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling." pith.science (2026). https://pith.science/paper/F5DB5DEJ
@misc{pith2026250608681,
author = {Pith},
title = {Pith review of: Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling},
year = {2026},
howpublished = {\url{https://pith.science/paper/F5DB5DEJ}},
note = {Machine review of arXiv:2506.08681}
}
read the original abstract
Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. However, these methods are more susceptible to over-optimization, in which the model drifts away from the reference policy, leading to degraded performance as training progresses. This paper proposes a novel importance-sampling approach to mitigate the over-optimization problem of offline DAAs. This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem. Our implementations are provided publicly at this link.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[6]
URLhttps://arxiv.org/abs/2210.10760. 12 Preprint Z. Gao, J. D. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, J. A. Bagnell, J. D. Lee, and W. Sun. REBEL: Reinforcement learning via regressing relative rewards. InICML 2024 Workshop on Models of Human Feedback for AI Alignment,
-
[7]
URL https: //openreview.net/forum?id=4SKidIUPP6. S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Ramé, T. Mesnard, Y. Zhao, B. Piot, J. Ferret, and M. Blondel. Direct language model alignment from online ai feedback. CoRR, abs/2402.04792,
-
[8]
URLhttps://doi.org/10.48550/arXiv.2402.04792. A. Huang, W. Zhan, T. Xie, J. D. Lee, W. Sun, A. Krishnamurthy, and D. J. Foster. Correcting the mythos of KL-regularization: Direct alignment without overoptimization via chi- squared preference optimization. InThe Thirteenth International Conference on Learning Representations,
-
[10]
URLhttps://arxiv.org/abs/2005.01643. Z. Liu, M. Lu, S. Zhang, B. Liu, H. Guo, Y. Yang, J. Blanchet, and Z. Wang. Provably mitigat- ing overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum?id=2cQ3lPhkeO. Z. L...
arXiv 2005
-
[11]
URL https://proceedings.neurips.cc/paper_ files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf. R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Computational Linguistics ACL 2024, pages...
work page 2022
-
[13]
URLhttps://arxiv.org/abs/1707.06347. Y. Song, G. Swamy, A. Singh, D. Bagnell, and W. Sun. The importance of online data: Understanding preference fine-tuning via coverage. InICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists,
arXiv 2024
-
[14]
URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper.pdf. Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Ávila Pires, M. Valko, Y. Cheng, and W. Dabney. Understanding the performance gap between online and offline alignment algorithms, 2024a. Y. Tang, Z. D. Guo, Z. Zheng, ...
work page 2020
-
[640]
Golden Reward Training detailsWe follow the synthetic setup where thegoldenreward model serves as human evaluation and provides preference labels [Gao et al., 2022, Tang et al., 2024a]. We first initialize the golden reward model with a SFT version of Llama-3.1-8B on the pooled SFT data. We then train the golden reward model on the combined preference of ...
work page 2022
Show all 15 references
-
[2017]
cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf
URL https://proceedings.neurips. cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf. M. Ding, S. Chakraborty, V . Agrawal, Z. Che, A. Koppel, M. Wang, A. Bedi, and F. Huang. SAIL: Self-improving efficient online alignment of large language models. InICML...
2017
-
[2020]
URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/0d2b2061826a5df3221116a5085a6052-Paper.pdf. S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems,
2020
-
[2021]
doi.org/10.1002/9781118445112.stat08284
URL http://dx. doi.org/10.1002/9781118445112.stat08284. A. Fisch, J. Eisenstein, V . Zayats, A. Agarwal, A. Beirami, C. Nagpal, P . Shaw, and J. Berant. Robust preference optimization through reward model distillation,
-
[2022]
2204.05862
URL https://doi.org/10.48550/arXiv. 2204.05862. S. Cen, J. Mei, K. Goshvadi, H. Dai, T. Yang, S. Yang, D. Schuurmans, Y. Chi, and B. Dai. Value-incentivized preference optimization: A unified approach to online and offline RLHF. InThe Thirteenth International Conference on Lea...
-
[2023]
URLhttps://openreview.net/forum?id= HPuSIXJaa9. R. Rafailov, Y. Chittepu, R. Park, H. Sikchi, J. Hejna, W. B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. In ICML 2024 Workshop on Models of Human Feedback for AI A...
2024
-
[2024]
URL https://proceedings.mlr.press/v238/ gheshlaghi-azar24a.html. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. E. Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, ...
-
[2025]
URL https://openreview.net/forum?id=SQnitDuow6. A. Chen, S. Malladi, L. H. Zhang, X. Chen, Q. Zhang, R. Ranganath, and K. Cho. Preference learning algorithms do not learn preference rankings. InICML 2024 Workshop on Theo- retical Foundations of Foundation Models, 2024a. URL ht...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.