{"id":"94281196-fa6f-4f49-b474-0a0e1ceaaf14","arxiv_id":"2505.01706","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Training 2D-DPO with an expected loss over uniform segment-score perturbations yields higher win rates under score noise than a clean-trained 2D-DPO baseline, though the comparison is confounded.","lead":"This paper proposes a noisy-trained version of 2D-DPO, a method that aligns language models using human scores on individual response segments. The authors claim it resists score noise better than plain 2D-DPO, but their experiments compare models trained under different conditions and do not isolate the effect.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness is inferred from a confounded comparison: noisy-trained Robust 2D-DPO is compared only with clean-trained vanilla 2D-DPO under noisy evaluation, and all reported 2D-DPO win rates are at or below chance, so the central claim is not established.","rationale":"The paper's stated contribution is a training-time expected-loss scheme intended to make 2D-DPO robust to segment-score perturbations. For that claim to hold, the experiments must show that the robust training procedure, not a different training set or an evaluation artifact, is responsible for the observed win rate. The only noisy-evaluation comparison in the paper is Experiment 3 versus Experiment 4, which differ in both training data and algorithm, so the missing matched noisy-training baseline is the load-bearing gap. The reader's labeled weakest assumption is the U(0,1) noise model in Eq. 14; that is a real limitation, but it is secondary because even under the paper's own noise model the experimental record does not establish robustness. The Eq. 11 collapse and the duplicated l(w,k) and l(l,k) definitions in Appendix D further weaken the theoretical backing. None of this is a critique of the authors' intent; it is an assessment that the current evidence cannot support the claimed result. A matched control experiment would settle the main question. Because this reinforces the reader's REJECT verdict rather than changing it, the verdict is left unchanged.","tokens_in":11852,"tokens_out":5401,"duration_ms":50168,"concrete_test":"Run a matched-control experiment: train Vanilla 2D-DPO on the exact same noisy HelpSteer-2D training split used for Robust 2D-DPO in Experiment 4, with the same model (Pythia-6.9B), hyperparameters, and noisy evaluation protocol. If its final evaluation win rate is at least 45.625%, the claimed robustness advantage is not demonstrated and the current REJECT verdict stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that Robust 2D-DPO handles segment-level score perturbations, is supported only by comparing Experiment 3 (Vanilla 2D-DPO trained on original data, evaluated with noise) with Experiment 4 (Robust 2D-DPO trained on noisy data, evaluated with noise). This is a double confound: both the training set and the algorithm change, so the 45.625% versus 37.188% difference cannot be attributed to robustness. The required control, Vanilla 2D-DPO trained on the same noisy HelpSteer-2D split, is absent. The evaluation metric is also unreliable: Appendix E defines win rate as the percentage of test prompts for which the model correctly distinguishes chosen from rejected, so chance is 50%; the paper reports 43.281% for clean-trained Vanilla 2D-DPO and 45.625% for Robust 2D-DPO, both below chance. A below-chance result undermines the claims that Robust 2D-DPO was not affected by noise and provided satisfactory performance. In addition, the theoretical support has an internal collapse: Eq. 11 is (1-gamma)L - gamma L over (1-2gamma), which equals the vanilla loss rather than the intended unbiased estimator, and Appendix D lines 8-9 define l(w,k) and l(l,k) identically. These issues remove the paper's current evidence for the robustness claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a robust variant of 2D-DPO for segment-level score noise. It introduces a noise model in which a single δ ∼ U(0,1) is subtracted from every winner-segment score and added to every loser-segment score, and defines the training objective as the expectation of the 2D-DPO loss over δ. The authors claim theoretical grounding for an unbiased loss under response-level preference flips (Eq. 11) and for the segment-level expected-loss objective (Eq. 17), and they report experiments with Pythia 6.9B on HelpSteer-2D comparing Vanilla DPO, Vanilla 2D-DPO, and their Robust 2D-DPO. The central claim is that Robust 2D-DPO is not affected by segment-level perturbations and provides satisfactory performance.","tokens_in":12054,"tokens_out":4992,"duration_ms":43753,"significance":"If the claims were established, the expected-loss-over-perturbation recipe would be a simple and potentially useful way to inject score-noise robustness into segment-level DPO variants, and the paper identifies a real gap in the literature: robustness for 2D-DPO under score noise. The paper also has the merit of attempting both a theoretical derivation and an empirical comparison. However, the current manuscript does not establish these claims. The theoretical derivation contains an algebraic collapse in Eq. (11), the algorithm in Appendix D does not compute the derived robust loss, and the experiments are confounded and report win rates at or below chance. These issues are load-bearing and leave the central claim unsupported.","major_comments":[{"comment":"Eq. (11) as written has the same term L_group(π_θ; D) in both numerator terms, so the expression reduces identically to L_group(π_θ; D) for any value of γ. The unbiased estimator in Eq. (7) requires the swapped-preference loss L(θ; s, a_l, a_w) as the second term, and its absence makes Eq. (11) trivially equal to the vanilla 2D-DPO loss. This removes the theoretical backing for the high-level noise robustness claim.","section":"Section 4.2, Eq. (11)"},{"comment":"Algorithm 1 defines l(w,k) and l(l,k) with identical formulas, both summing the same token log-probability ratios over the same segment. As a result, X_k = r(i)_w l(w,k) - r(i)_l l(l,k) and Y_k = l(w,k) + l(l,k) do not match the quantities in Eq. (16), and the training loss implemented by Algorithm 1 is not the expected-loss objective in Eq. (17). This is a load-bearing implementation error: the robust training procedure as coded is not the one derived in Section 4.3.","section":"Appendix D, lines 8-9"},{"comment":"The comparison that supports the central claim changes two variables simultaneously: Experiment 3 trains Vanilla 2D-DPO on the original data and evaluates with noise, while Experiment 4 trains Robust 2D-DPO on the noisy data and evaluates with noise. The observed improvement from 37.188% to 45.625% cannot be attributed to the robust loss because the training distribution also changed. A control that trains Vanilla 2D-DPO on the same noisy split is needed, as is an evaluation of Robust 2D-DPO trained on the original data.","section":"Section 5, Table 1, Experiments 3 and 4"},{"comment":"Win rate is defined in Appendix E as the percentage of test prompts for which the model correctly distinguishes preferred from rejected responses, making chance 50%. The reported evaluation win rates are 43.281% for clean Vanilla 2D-DPO, 37.188% for Vanilla 2D-DPO under noise, and 45.625% for Robust 2D-DPO under noise, all at or below chance. The claim that Robust 2D-DPO was not affected by noise and provided satisfactory performance is not supported by these numbers.","section":"Section 5.5 and Appendix E"},{"comment":"The robustness claim is tested in-distribution by construction: the training objective in Eq. (15) is the expectation of the 2D-DPO loss under the same U(0,1) single-δ perturbation that is later used to corrupt the evaluation scores in Eq. (14). Without evaluating under different noise distributions (for example, independent per-segment perturbations, asymmetric noise, or different magnitude bounds), the paper provides no evidence that the method is robust to segment-level perturbations generally.","section":"Section 4.3, Eq. (14) and Section 5.4"}],"minor_comments":[{"comment":"There is a typo: 'shart decline' should be 'sharp decline'.","section":"Section 5.5"},{"comment":"The text contains 'Emperical evidence' and 'Re-inforcement'; these should be 'Empirical' and 'Reinforcement'.","section":"Section 2"},{"comment":"The sentence 'Anthropic-HH is a human preference dataset about helpfulness and harmlessness from .' is incomplete and should be finished.","section":"Appendix E"},{"comment":"Reference [15] (Stokes, 2007) on human preferences among similar organisms appears unrelated to the claims about multi-aspect human preferences in LLM alignment and should be replaced or justified.","section":"References"},{"comment":"The statement 'we were unable to find a similar paradigm for 2D-DPO' should be supported by a more explicit literature search or stated more carefully, since the absence of prior work is not established by the cited references alone.","section":"Section 1"},{"comment":"The notation in Eq. (13) and the surrounding text should be checked for consistency with Eq. (9), because the placement of the segment score r relative to β differs between the two equations.","section":"Section 4.3"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for publication in its current form. The central derivation reduces to the vanilla loss in Eq. (11), and the algorithm in Appendix D defines the winner and loser terms identically, so the implementation does not match the derivation. The experimental evaluation is further compromised by below-chance win rates and a confounded comparison. A resubmission would need a corrected derivation, a correctly implemented algorithm, and a new experimental protocol with the missing controls and out-of-distribution noise tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is a quick read but the main claim doesn't hold. The idea — make 2D-DPO robust to segment-level score noise by taking the expectation of the loss under a uniform score perturbation — is a reasonable thing to try, and the paper is honest that no one has done it before. I'll give it that.\n\nThe problems start in Eq. 11. The proposed unbiased robust loss is written as ((1-γ)L_group - γ L_group)/(1-2γ), which is just L_group. The two terms are identical, so the robust loss is the vanilla loss. That's not a typo; the same confusion shows up in Algorithm 1, where l(w,k) and l(l,k) are defined in the same lines with the same tokens. That kills the theoretical contribution.\n\nThe experiments don't help. Exp 3 trains vanilla 2D-DPO on clean data then evaluates on noisy data, while Exp 4 trains the robust version on noisy data and evaluates on noisy data. Two things change at once, so the 45.6% vs 37.2% difference can't be attributed to robustness. The missing control — vanilla 2D-DPO trained on the same noisy data — is the one that would actually tell you if the robust loss does anything. And even the clean 2D-DPO baseline is at 43.3% win rate, below chance. That suggests the implementation or evaluation is off, independent of noise.\n\nThe noise model itself (one δ ~ U(0,1) subtracted from every winner segment and added to every loser segment) is arbitrary. No empirical or theoretical justification is given, and there's no out-of-distribution test against other noise models, so the method is only tested under the exact perturbation it was trained for.\n\nThe paper also claims to provide code, but the link is missing, and core hyperparameters are not reported. The citation pattern is fine; the related work is appropriate.\n\nSo: good gap to target, but the current version is not close to supporting its claim. A serious editor should desk reject this. If the authors fix Eq. 11, rerun with proper controls, and report results that aren't below chance, then it might deserve another look. As it stands, I'd tell a student to read it as a cautionary tale about experimental design, not as a reliable method.","headline":"Plausible idea, load-bearing math error, confounded experiment; the robustness claim is not established.","tokens_in":12738,"tokens_out":3273,"would_cite":false,"duration_ms":30240,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Robust 2D-DPO trains on the expected loss over uniform segment-score perturbations and, the paper reports, keeps a 45.6% win rate under noise where vanilla 2D-DPO falls to 37.2%.","keywords":["direct preference optimization","2D-DPO","segment-level scoring","noise robustness","LLM alignment","human preference learning","reward model","preference optimization"],"falsifier":"Run the same training recipe but corrupt evaluation with independent per-segment noise (each segment gets its own draw, or only winner scores are perturbed) and measure win rate; a drop comparable to vanilla 2D-DPO's fall from 43.281% to 37.188% would falsify the paper's segment-level robustness claim.","tokens_in":11518,"feed_emoji":"🛡️","tokens_out":13082,"duration_ms":108229,"temperature":0.7,"pith_summary":"Standard direct preference optimization (DPO) scores each whole response, and 2D-DPO refines this by scoring each segment of a response on five aspects and combining the aspect scores into segment rewards. The paper argues that these segment scores are themselves noisy in practice, and shows experimentally that vanilla 2D-DPO's win rate drops from 43.281% to 37.188% when segment-score noise is added at evaluation. It then proposes Robust 2D-DPO, which trains on the expected loss over a uniform perturbation that lowers every winner-segment score and raises every loser-segment score. The paper reports that this noise-trained model keeps a 45.625% win rate under the same perturbation, and frames the method as a segment-level analog of existing robust DPO. The point matters because annotation noise is realistic, and robustness here is achieved without moving away from single-stage DPO-style optimization.","feed_headline":"Robust 2D-DPO keeps 45.6% win rate amid score noise","feed_subtitle":"Vanilla 2D-DPO drops from 43.3% to 37.2% under the same perturbation; the noise-trained version stays stable.","key_machinery":"The load-bearing object is the expected noisy 2D-DPO loss in Eq. 17, $-E_{\\delta \\sim U(0,1)} \\sum_k \\log \\sigma(X_k - \\delta Y_k)$, where $X_k$ is the clean per-segment margin and $Y_k$ is the sum of the two per-segment log-ratio terms. The algebraic step that makes the method work is Eq. 14-16: a uniform perturbation $\\delta$ applied as $\\hat r(w,k) = r(w,k) - \\delta$ and $\\hat r(l,k) = r(l,k) + \\delta$ converts the clean margin into $X_k - \\delta Y_k$, so the objective becomes an average over noise realizations rather than a single fixed corrupted score. Algorithm 1 implements this by drawing one $\\delta \\sim U(0,1)$ per response pair in each mini-batch and accumulating gradients of $\\log \\sigma(X_k - \\delta Y_k)$ across segments. This is what lets the training signal be de-biased on average with respect to the paper's noise model.","core_discovery":"The central claim is that segment-level score noise is what breaks 2D-DPO, and that the breakage can be removed by minimizing the loss that the noisy scores induce in expectation. Concretely, let $r(w,k)$ and $r(l,k)$ be the segment scores for the winner and loser responses at segment $k$, and let $l(w,k)$ and $l(l,k)$ be the corresponding token-log-ratio sums scaled by $\\beta$. The paper models one noisy draw $\\delta \\sim U(0,1)$ that subtracts from every winner score and adds to every loser score, turning the clean margin $X_k = r(w,k)l(w,k) - r(l,k)l(l,k)$ into $X_k - \\delta Y_k$ with $Y_k = l(w,k) + l(l,k)$. Robust 2D-DPO minimizes $-E_{\\delta \\sim U(0,1)} \\sum_k \\log \\sigma(X_k - \\delta Y_k)$, a quantity with no closed form that the paper optimizes by mini-batch SGD, sampling one $\\delta$ per response pair. The paper reports that under segment-score noise this recovers a 45.625% evaluation win rate, compared with 37.188% for vanilla 2D-DPO evaluated under the same noise.","pith_inferences":["The paper does not test whether the same training recipe survives per-segment independent noise, where each segment draws its own $\\delta$; that is the natural next stress test for the robustness claim.","If real annotation noise is asymmetric, for instance only winner scores being shaved or only certain aspects being corrupted, the uniform antithetic model is not obviously the right training distribution, and the paper offers no out-of-distribution evaluation.","Extending the idea to aspect-level noise, where a discrete aspect score flips to another level, would require modeling a discrete distribution over the five aspects; the paper notes this direction but does not solve it.","One implication is that the margin-shrinking noise model may act as a regularizer: the robust model's noisy win rate is even higher than vanilla 2D-DPO's noiseless win rate, which hints at a possible smoothing effect, though the paper does not investigate this."],"forward_implications":["Under the paper's noise model, vanilla 2D-DPO is not robust: its evaluation win rate drops from 43.281% to 37.188% when segment scores are perturbed at inference.","Training with the expected loss over $\\delta \\sim U(0,1)$ keeps the win rate at 45.625% under the same noise, above both the noisy vanilla 2D-DPO and the noiseless vanilla 2D-DPO baseline of 43.281%.","The unbiased de-biasing idea developed for label-flip robust DPO transfers to continuous segment-score perturbations, not just discrete preference flips.","The method requires no reward-model training or RL loop; it is implemented with ordinary mini-batch SGD, so the robustness comes at the cost of one extra noise sample per response pair per iteration."],"supporting_citations":[{"why":"Defines 2D-DPO with five-aspect segment scores and the group loss in Eq. 9 that the paper sets out to make noise-robust.","marker":"[8]"},{"why":"Supplies the unbiased de-biased DPO loss whose noise model and unbiased estimator the paper adapts to segment-level score perturbations.","marker":"[5]"},{"why":"Provides the vanilla DPO loss and the policy-reward reparameterization that 2D-DPO extends.","marker":"[12]"},{"why":"Gives the paired-comparison model used to express preference probabilities as sigmoids of score margins.","marker":"[3]"},{"why":"Provides the token-level DPO formulation used to compute the per-segment log-probability terms $l(w,k)$ and $l(l,k)$.","marker":"[18]"}],"fun_headline_variants":["Robust 2D-DPO shrugs off score noise, wins 45.6%","Segment score noise broken: robust 2D-DPO hits 45.6% win","2D-DPO gets noise-proofed, win rate up to 45.6% from 37.2%","Under score noise, robust 2D-DPO keeps 45.6% win rate","Robustifying 2D-DPO against noise: 45.6% win rate vs 37.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The robustness claim rests on a specific noise model: a single uniform draw that shaves the same amount off every winner-segment score and adds the same amount to every loser-segment score, with no test of other noise patterns.","fun_headline_variants_meta":{"raw":{"variants":["Robust 2D-DPO shrugs off score noise, wins 45.6%","Segment score noise broken: robust 2D-DPO hits 45.6% win","2D-DPO gets noise-proofed, win rate up to 45.6% from 37.2%","Under score noise, robust 2D-DPO keeps 45.6% win rate","Robustifying 2D-DPO against noise: 45.6% win rate vs 37.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001063,"raw_usage":{"total_tokens":4502,"prompt_tokens":1037,"completion_tokens":3465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":3336}},"tokens_in":653,"tokens_out":3465,"duration_ms":24631,"temperature":1.0,"reasoning_tokens":3336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:13:22.373120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training recipe but corrupt evaluation with independent per-segment noise (each segment gets its own draw, or only winner scores are perturbed) and measure win rate; a drop comparable to vanilla 2D-DPO's fall from 43.281% to 37.188% would falsify the paper's segment-level robustness claim.","supporting_citations":[{"cited_title":"2D-DPO: Scaling direct preference optimization with 2-dimensional supervision","cited_arxiv_id":null,"evidence_quote":"Defines 2D-DPO with five-aspect segment scores and the group loss in Eq. 9 that the paper sets out to make noise-robust."}],"review_version":1}