REVIEW 5 major objections 6 minor 20 references
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Robust 2D-DPO trains on the expected loss over uniform segment-score perturbations and, the paper reports, keeps a 45.6% win rate under noise where vanilla 2D-DPO falls to 37.2%.
desk verdict Plausible idea, load-bearing math error, confounded experiment; the robustness claim is not established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the expected noisy 2D-DPO loss in Eq. 17, $-E_{\delta \sim U(0,1)} \sum_k \log \sigma(X_k - \delta Y_k)$, where $X_k$ is the clean per-segment margin and $Y_k$ is the sum of the two per-segment log-ratio terms. The algebraic step that makes the method work is Eq. 14-16: a uniform perturbation $\delta$ applied as $\hat r(w,k) = r(w,k) - \delta$ and $\hat r(l,k) = r(l,k) + \delta$ converts the clean margin into $X_k - \delta Y_k$, so the objective becomes an average over noise realizations rather than a single fixed corrupted score. Algorithm 1 implements this by drawing one $\delta \sim U(0,1)$ per response pair in each mini-batch and accumulating gradients of $\log \sigma(X_k - \delta Y_k)$ across segments. This is what lets the training signal be de-biased on average with respect to the paper's noise model.
What would settle it
Run the same training recipe but corrupt evaluation with independent per-segment noise (each segment gets its own draw, or only winner scores are perturbed) and measure win rate; a drop comparable to vanilla 2D-DPO's fall from 43.281% to 37.188% would falsify the paper's segment-level robustness claim.
Extended reading notes
Core claim
The central claim is that segment-level score noise is what breaks 2D-DPO, and that the breakage can be removed by minimizing the loss that the noisy scores induce in expectation. Concretely, let $r(w,k)$ and $r(l,k)$ be the segment scores for the winner and loser responses at segment $k$, and let $l(w,k)$ and $l(l,k)$ be the corresponding token-log-ratio sums scaled by $\beta$. The paper models one noisy draw $\delta \sim U(0,1)$ that subtracts from every winner score and adds to every loser score, turning the clean margin $X_k = r(w,k)l(w,k) - r(l,k)l(l,k)$ into $X_k - \delta Y_k$ with $Y_k = l(w,k) + l(l,k)$. Robust 2D-DPO minimizes $-E_{\delta \sim U(0,1)} \sum_k \log \sigma(X_k - \delta Y_k)$, a quantity with no closed form that the paper optimizes by mini-batch SGD, sampling one $\delta$ per response pair. The paper reports that under segment-score noise this recovers a 45.625% evaluation win rate, compared with 37.188% for vanilla 2D-DPO evaluated under the same noise.
Load-bearing premise
The robustness claim rests on a specific noise model: a single uniform draw that shaves the same amount off every winner-segment score and adds the same amount to every loser-segment score, with no test of other noise patterns.
Editorial extensions
If this is right
- Under the paper's noise model, vanilla 2D-DPO is not robust: its evaluation win rate drops from 43.281% to 37.188% when segment scores are perturbed at inference.
- Training with the expected loss over $\delta \sim U(0,1)$ keeps the win rate at 45.625% under the same noise, above both the noisy vanilla 2D-DPO and the noiseless vanilla 2D-DPO baseline of 43.281%.
- The unbiased de-biasing idea developed for label-flip robust DPO transfers to continuous segment-score perturbations, not just discrete preference flips.
- The method requires no reward-model training or RL loop; it is implemented with ordinary mini-batch SGD, so the robustness comes at the cost of one extra noise sample per response pair per iteration.
Reading between the lines
- The paper does not test whether the same training recipe survives per-segment independent noise, where each segment draws its own $\delta$; that is the natural next stress test for the robustness claim.
- If real annotation noise is asymmetric, for instance only winner scores being shaved or only certain aspects being corrupted, the uniform antithetic model is not obviously the right training distribution, and the paper offers no out-of-distribution evaluation.
- Extending the idea to aspect-level noise, where a discrete aspect score flips to another level, would require modeling a discrete distribution over the five aspects; the paper notes this direction but does not solve it.
- One implication is that the margin-shrinking noise model may act as a regularizer: the robust model's noisy win rate is even higher than vanilla 2D-DPO's noiseless win rate, which hints at a possible smoothing effect, though the paper does not investigate this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a robust variant of 2D-DPO for segment-level score noise. It introduces a noise model in which a single δ ∼ U(0,1) is subtracted from every winner-segment score and added to every loser-segment score, and defines the training objective as the expectation of the 2D-DPO loss over δ. The authors claim theoretical grounding for an unbiased loss under response-level preference flips (Eq. 11) and for the segment-level expected-loss objective (Eq. 17), and they report experiments with Pythia 6.9B on HelpSteer-2D comparing Vanilla DPO, Vanilla 2D-DPO, and their Robust 2D-DPO. The central claim is that Robust 2D-DPO is not affected by segment-level perturbations and provides satisfactory performance.
Significance. If the claims were established, the expected-loss-over-perturbation recipe would be a simple and potentially useful way to inject score-noise robustness into segment-level DPO variants, and the paper identifies a real gap in the literature: robustness for 2D-DPO under score noise. The paper also has the merit of attempting both a theoretical derivation and an empirical comparison. However, the current manuscript does not establish these claims. The theoretical derivation contains an algebraic collapse in Eq. (11), the algorithm in Appendix D does not compute the derived robust loss, and the experiments are confounded and report win rates at or below chance. These issues are load-bearing and leave the central claim unsupported.
major comments (5)
- [Section 4.2, Eq. (11)] Eq. (11) as written has the same term L_group(π_θ; D) in both numerator terms, so the expression reduces identically to L_group(π_θ; D) for any value of γ. The unbiased estimator in Eq. (7) requires the swapped-preference loss L(θ; s, a_l, a_w) as the second term, and its absence makes Eq. (11) trivially equal to the vanilla 2D-DPO loss. This removes the theoretical backing for the high-level noise robustness claim.
- [Appendix D, lines 8-9] Algorithm 1 defines l(w,k) and l(l,k) with identical formulas, both summing the same token log-probability ratios over the same segment. As a result, X_k = r(i)_w l(w,k) - r(i)_l l(l,k) and Y_k = l(w,k) + l(l,k) do not match the quantities in Eq. (16), and the training loss implemented by Algorithm 1 is not the expected-loss objective in Eq. (17). This is a load-bearing implementation error: the robust training procedure as coded is not the one derived in Section 4.3.
- [Section 5, Table 1, Experiments 3 and 4] The comparison that supports the central claim changes two variables simultaneously: Experiment 3 trains Vanilla 2D-DPO on the original data and evaluates with noise, while Experiment 4 trains Robust 2D-DPO on the noisy data and evaluates with noise. The observed improvement from 37.188% to 45.625% cannot be attributed to the robust loss because the training distribution also changed. A control that trains Vanilla 2D-DPO on the same noisy split is needed, as is an evaluation of Robust 2D-DPO trained on the original data.
- [Section 5.5 and Appendix E] Win rate is defined in Appendix E as the percentage of test prompts for which the model correctly distinguishes preferred from rejected responses, making chance 50%. The reported evaluation win rates are 43.281% for clean Vanilla 2D-DPO, 37.188% for Vanilla 2D-DPO under noise, and 45.625% for Robust 2D-DPO under noise, all at or below chance. The claim that Robust 2D-DPO was not affected by noise and provided satisfactory performance is not supported by these numbers.
- [Section 4.3, Eq. (14) and Section 5.4] The robustness claim is tested in-distribution by construction: the training objective in Eq. (15) is the expectation of the 2D-DPO loss under the same U(0,1) single-δ perturbation that is later used to corrupt the evaluation scores in Eq. (14). Without evaluating under different noise distributions (for example, independent per-segment perturbations, asymmetric noise, or different magnitude bounds), the paper provides no evidence that the method is robust to segment-level perturbations generally.
minor comments (6)
- [Section 5.5] There is a typo: 'shart decline' should be 'sharp decline'.
- [Section 2] The text contains 'Emperical evidence' and 'Re-inforcement'; these should be 'Empirical' and 'Reinforcement'.
- [Appendix E] The sentence 'Anthropic-HH is a human preference dataset about helpfulness and harmlessness from .' is incomplete and should be finished.
- [References] Reference [15] (Stokes, 2007) on human preferences among similar organisms appears unrelated to the claims about multi-aspect human preferences in LLM alignment and should be replaced or justified.
- [Section 1] The statement 'we were unable to find a similar paradigm for 2D-DPO' should be supported by a more explicit literature search or stated more carefully, since the absence of prior work is not established by the cited references alone.
- [Section 4.3] The notation in Eq. (13) and the surrounding text should be checked for consistency with Eq. (9), because the placement of the segment score r relative to β differs between the two equations.
Circularity Check
Robustness is in-distribution by construction: the training loss is the expectation under the same U(0,1) segment-noise model used to build the noisy evaluation set, and the preference-flip 'unbiased estimator' collapses algebraically to the vanilla 2D-DPO loss.
-
fitted input called prediction
[Section 4.3 (Eqs. 14–15), Algorithm 1, Section 5.4 Experiment 4, Table 1]
"To these scores, let us now add a small perturbation δ to get the noisy scores as seen in Eq. 14 ... Substituting the noisy rewards from Eq. 14 back into the loss function ... we get the noisy loss function ... = −Eδ∼U (0,1)E(τw,τl)∼D [Σ logσ{Xk−δYk}]."
The proposed robust objective in Eq. 15 is by definition the expectation of the original 2D-DPO loss under the exact U(0,1) perturbation of Eq. 14, and Algorithm 1 samples that same δ∼U(0,1) during training. Experiment 4 trains and evaluates Robust 2D-DPO on HelpSteer-2D (Noisy) (Table 1), so the reported 45.625% win rate measures in-distribution performance of the fitted objective, not an independent robustness property. The comparison with Experiment 3 changes both the algorithm and the training data (clean-trained vanilla vs noisy-trained robust), so the claimed noise-handling is not isolated. The central empirical claim is built into the training loss by construction.
-
self definitional
[Section 4.2, Eq. 11]
"The noisy-robust unbiased loss estimator in this case can be given as shown in Eq. 11: ˆLγ(πθ;D) = (1−γ)Lgroup(πθ;D) − γLgroup(πθ;D) / 1−2γ."
Both terms in the numerator are the identical Lgroup(πθ;D), so the expression algebraically simplifies to Lgroup(πθ;D). The proposed unbiased estimator for the preference-flip noise model therefore reduces to the original 2D-DPO loss by construction; no debiasing or robustness is introduced. This makes the stated theoretical backing for the high-level noise model vacuous and, as written, would also make training under the 'robust' loss identical to vanilla 2D-DPO training.
full rationale
No load-bearing self-citation chain is present: the paper builds on external work by Li et al. (2D-DPO) and Chowdhury et al. (Robust DPO), and the cited results are not used to forbid alternatives. The circularity is instead internal. The segment-level robustness claim is tested on the same noise distribution the algorithm is trained to optimize against: Eq. 15 defines the robust loss as an expectation over δ∼U(0,1) applied to segment scores, and Experiment 4 trains on the noisy dataset and evaluates on the noisy dataset, omitting the required control of vanilla 2D-DPO trained on the same noisy split. In addition, the mathematical support for the preference-flip variant (Eq. 11) collapses to the vanilla loss because its numerator contains the same Lgroup twice. These two reductions mean the central 'prediction' is substantially built into the paper's own definitions, yielding a partial circularity score of 6. Many additional issues (below-chance win rates, identical l(w,k)/l(l,k) definitions in Algorithm 1) are correctness risks, but they are not themselves circularity.
Assumptions & free parameters
free parameters (4)
- β (temperature) in DPO/2D-DPO losses =
not reported
- Aspect weights w in Eq. 8 =
not reported
- Uniform noise bound for δ =
U(0,1)
- Optimization hyperparameters (learning rate, batch size, iterations) =
not reported
assumptions (4)
- domain assumption BTL (Bradley-Terry) preference model
- standard math DPO reward reparameterization
- ad hoc to paper Expected loss over δ is the correct robustness objective
- domain assumption 2D-DPO segment selection and scoring conventions
Cite this review
Pith. "Pith review of Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm." pith.science (2026). https://pith.science/paper/SHYHXLDU
@misc{pith2026250501706,
author = {Pith},
title = {Pith review of: Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm},
year = {2026},
howpublished = {\url{https://pith.science/paper/SHYHXLDU}},
note = {Machine review of arXiv:2505.01706}
}
read the original abstract
Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning Large Language Models (LLMs) with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Improving multi- 9 modal interactive agents with reinforcement learning from human feedback
Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, et al. Improving multi- 9 modal interactive agents with reinforcement learning from human feedback. arXiv preprint arXiv:2211.11602, 2022
arXiv 2022
-
[2]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022
arXiv 2022
-
[3]
Rank analysis of incomplete block designs: I
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952
1952
-
[4]
Open problems and fundamental limitations of reinforcement learning from human feedback
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023
arXiv 2023
-
[5]
Provably robust dpo: Aligning language models with noisy feedback
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409, 2024
arXiv 2024
-
[6]
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017
2017
-
[7]
Diverse preference optimization.arXiv preprint arXiv:2501.18101, 2025
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization.arXiv preprint arXiv:2501.18101, 2025
arXiv 2025
-
[8]
2D-DPO: Scaling direct preference optimization with 2-dimensional supervision
Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, and Bo Zheng. 2D-DPO: Scaling direct preference optimization with 2-dimensional supervision. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Findings of the Association for Computational Linguistics: NAACL 2025, pages 8149–8173, Albuquerque, New ...
work page 2025
Show all 20 references
-
[9]
A survey of direct preference optimization
Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, et al. A survey of direct preference optimization. arXiv preprint arXiv:2503.11701, 2025
2025 arXiv
-
[10]
Trustworthy llms: a survey and guideline for evaluating large language models’ alignment
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023
2023 arXiv
-
[11]
Rlhf from heterogeneous feedback via personalization and preference aggregation
Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman Ozdaglar. Rlhf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254, 2024
2024 arXiv
-
[12]
Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023
2023
-
[13]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
2024 arXiv
-
[14]
Large language model alignment: A survey
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025, 2023
2023 arXiv
-
[15]
Things we like: human preferences among similar organisms and implications for conservation
David L Stokes. Things we like: human preferences among similar organisms and implications for conservation. Human Ecology, 35:361–369, 2007. 10
2007
-
[16]
Aligning large language models with human: A survey
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023
2023 arXiv
-
[17]
Learning with noisy labels revisited: A study using real-world human annotations
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088, 2021
2021 arXiv
-
[18]
Token- level direct preference optimization
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. arXiv preprint arXiv:2404.11999, 2024
2024 arXiv
-
[19]
Beyond one-preference-for-all: Multi-objective direct preference optimization
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization. 2023
2023
-
[20]
chosen” and one “rejected
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 11 Appendix A covers DPO (Direct Preference Optimisation), wh...
1909 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.