{"id":"076c4adf-c849-4198-adce-ea5ddaf3c077","arxiv_id":"2412.14516","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Cal-DPO modifies DPO by adding squared losses that anchor the policy's implicit rewards of chosen and rejected responses to plus or minus 1/(2*beta), reporting gains on reasoning, summarization, and dialogue benchmarks.","lead":"This paper adds a squared-loss 'calibration' term to DPO that pushes the chosen response's implicit reward up and the rejected response's down, and reports improved scores on several LLM alignment benchmarks. The catch is that the 'ground-truth' reward scale is replaced by an arbitrary plus or minus one-half anchor, so the theoretical calibration guarantee does not apply to the actual algorithm.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing flaw is in Theorem 2's proof: Eq. (47) replaces a squared-norm bound over all responses with an expectation under πref, and the claimed inequality fails on a simple two-outcome counterexample even under the oracle reward.","rationale":"The reader's REJECT verdict is correct, but the weakest point is earlier and more severe than the ±1/(2β) target substitution. The oracle-versus-practical mismatch already severs the implemented loss from the theoretical statement; the counterexample shows the theorem is not just disconnected from the implementation but fails under the paper's own definitions. This strengthens the rejection. I credit the paper for a cleanly specified algorithm, released code, and consistent benchmark improvements; the load-bearing problem is the advertised theoretical guarantee, not the empirical anchor-regularizer reading. My disagreement with the reader is therefore about the location of the central flaw: I locate it inside the proof of Theorem 2, while the reader located it at the reward-target substitution in Eq. (10) versus Definition 1. Both concerns support the same final verdict.","tokens_in":27109,"tokens_out":12396,"duration_ms":108740,"concrete_test":"Reproduce the counterexample by direct computation: fix β=1, a two-response space with πref=(0.99,0.01), r(x,y)=0, and πθ=(0.5,0.5). Evaluate both sides of Eq. (47) and Eq. (15) from the paper. An elementary calculation gives LHS = 1.614 and RHS = 1.252, so the claimed inequality fails. If a revised proof cannot accommodate this case, Theorem 2 must be weakened or withdrawn before the theoretical claims can be relied upon.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Even granting oracle rewards, the proof of Theorem 2 (Appendix A.4) is invalid. The Bregman/Taylor step gives DKL(q||p) ≤ DKL(p||q) + (1/4) Σ_y (s_y − r_y)^2, where s_y = log πθ(y|x) and r_y = log πref(y|x) + r(x,y)/β. The calibration term actually used in Eq. (13) is E_{y∼πref}[(s_y − r_y)^2], which is not the same as the sum and can be much smaller. This is not cosmetic. Take β=1, Y={1,2}, πref=(0.99,0.01), oracle reward r(x,y)=0, and πθ=(0.5,0.5). Then p=π* = πref and q=πθ. The claimed inequality Eq. (47) reads DKL(q||p) = 1.614 ≤ DKL(p||q) + E[...] = 0.637 + 0.615 = 1.252, which is false. Consequently the advertised bound LRL ≤ βL_CalDPO − β log Z (Eq. (15)) fails in this case: with β=1 and Z=1, LRL=1.614 while L_CalDPO=1.252. This failure is independent of the ±1/(2β) target substitution, so the paper's central theoretical contribution—provable mode-seeking and an upper bound on the RLHF objective—is unsupported even under the ideal oracle-reward setting.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cal-DPO, a modification of direct preference optimization (DPO) that adds a squared-error calibration term pushing the implicit reward of the chosen response toward +1/(2β) and the implicit reward of the rejected response toward −1/(2β). The authors claim this calibrates the learned implicit rewards to the scale of ground-truth rewards, preserves a \"negative gradient\" property, and provably minimizes an upper bound on the KL-regularized RLHF objective (Theorem 2). The paper reports experiments on reasoning, summarization, dialogue, and controlled sentiment tasks, showing consistent improvements over DPO and several baselines, together with pseudocode and an open-source implementation.","tokens_in":27418,"tokens_out":5695,"duration_ms":42294,"significance":"If the theoretical claims were correct, the paper would offer a simple, one-line modification to DPO with substantial benchmark gains and a principled explanation of mode-seeking behavior. The empirical results are interesting and the method is easy to reproduce from the provided pseudocode. However, the paper's central theoretical contribution is not supported: the proof of Theorem 2 contains an invalid replacement of a sum over responses by an expectation under the reference policy, and a concrete counterexample falsifies the claimed inequality. In addition, the theory is stated for calibration to an oracle reward, while the implemented loss uses hand-set constants ±1/2, so the theoretical guarantees do not transfer to the algorithm actually evaluated. The empirical gains provide some value, but they do not compensate for the advertised theoretical advantages that are invalid as stated.","major_comments":[{"comment":"The proof of Theorem 2 is invalid. The Bregman inequality in Eq. (46) yields a term (1/4) * Σ_y (s_y − r_y)^2, a sum over the response space, but the next step replaces this sum with E_{y∼πref}[(log πθ(y|x)/πref(y|x) − r(x,y)/β)^2], which is not an upper bound and can be much smaller. Concretely, take Y={1,2}, πref=(0.99,0.01), r(x,y)=0, πθ=(0.5,0.5), and β=1. Then p=π*=πref, q=πθ, and the claimed inequality in Eq. (47) reads D_KL(q||p)=1.614 ≤ D_KL(p||q)+E[...] = 0.637+0.614 = 1.251, which is false. Consequently, the advertised upper bound L_RL(θ) ≤ β L_CalDPO(θ) − β log Z(x) in Eq. (15) fails even under the oracle-reward setting, independent of the ±1/(2β) target substitution.","section":"Appendix A.4, Eq. (47)"},{"comment":"The theoretical results in Section 4.2 (Definition 1, Theorems 1 and 2) are stated for calibration to the oracle reward r(x,y)/β, but the implemented loss in Eq. (10) uses r(x,y_w)=1/2 and r(x,y_l)=−1/2. The paper provides no evidence that these constants correspond to the reward scale of any dataset, so the central claim of calibrating to ground-truth rewards is not supported for the actual objective. The derivation in Appendix A.2 does not bridge this gap: it attempts a limit β→0 to obtain hard softmax weights, but under that limit the calibration targets 1/(2β) diverge, and for the finite β used in the experiments (e.g., 0.001) the two losses are not equivalent.","section":"Section 4.1, Eq. (10) and Appendix A.2"},{"comment":"The theorem's claim that Cal-DPO \"encourage[s] mode-seeking behavior by minimizing an upper bound of the reverse KL divergence\" rests entirely on the invalid inequality in Eq. (47). Because that inequality fails, the paper does not establish that Cal-DPO minimizes an upper bound of the RLHF objective, nor that it has the claimed mode-seeking property. This is the central theoretical contribution of the paper, and the error cannot be repaired by local editing of the proof; the theorem as stated is false.","section":"Section 4.2, Theorem 2"}],"minor_comments":[{"comment":"The sentence \"All proofs are provided in the Appendix 4.2\" should refer to \"Appendix A\" rather than \"Appendix 4.2\".","section":"Section 4.2"},{"comment":"The first term in Eq. (13) is written with a denominator that depends on the summation variable y; the simplification used in the proof of Theorem 1 shows the denominator equals 1, but the notation in Eq. (13) is misleading and should be clarified.","section":"Equation (13)"},{"comment":"The checklist claims that the experiments are accompanied by statistical significance information, but no error bars, confidence intervals, or variance estimates are reported in Tables 2–4 or Figure 2; this checklist answer is not supported by the manuscript.","section":"Section 5 and NeurIPS Checklist item 7"},{"comment":"The statement \"setting β → 0 (β is typically small and set as 0.001)\" is internally inconsistent: with β=0.001, the targets ±1/(2β) equal ±500, which is not a small-β limit of the squared-loss calibration term used in Eq. (10).","section":"Appendix A.2"},{"comment":"The header identifies the paper as appearing at \"38th Conference on Neural Information Processing Systems (NeurIPS 2024)\", but the arXiv submission is dated December 2024; if this is a journal submission, the venue line should be removed or updated.","section":"Title page"}],"recommendation":"reject","confidential_remarks":"The self-reported checklist in item 3 asserts that the paper provides a complete and correct proof, but the proof of Theorem 2 is demonstrably incorrect. The additional substitution of ±1/2 targets in the implemented loss, without any support that these match real reward scales, suggests that the theoretical sections were not carefully vetted against the actual algorithm. The empirical results may justify a future, more modestly scoped paper centered on the practical method, but the current manuscript's central claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical recipe is a one-line addition to DPO that pulls chosen/rejected implicit rewards toward ±1/(2β). That works reasonably well in their experiments, particularly on IFEval. The theory advertised for it does not, and I think the failure is load-bearing.\n\nWhat's new: the specific loss (Eq. 10) is not in prior work, and the symmetric anchoring of both chosen and rejected responses is a legitimate extension of DPOP/CPO/DPO+NLL, which only anchor the chosen side. The experiments are on standard benchmarks with a clear setup, and the gains over DPO and variants are mostly modest but consistent; IFEval improvement is large. That's a useful empirical finding.\n\nWhere it breaks: Theorem 2's proof has a genuine error. The Bregman step yields a bound involving a sum over all y of (s_y − r_y)^2, but the proof replaces it with an expectation under πref of the same squared quantity. Those are not the same; the expectation can be much smaller than the sum. The stress-test gives a simple two-outcome counterexample (πref=(0.99,0.01), πθ=(0.5,0.5), zero reward) where the claimed inequality Eq. (47) fails. So the upper bound on RLHF is unsupported even with oracle rewards. On top of that, the implemented loss uses arbitrary targets ±1/(2β) rather than the true reward scale, so Definition 1 and the calibration interpretation do not apply to the actual algorithm. The paper says \"empirically we find this works quite well,\" which is fine as a heuristic, but not as a calibration guarantee.\n\nAlso minor: no error bars or seeds are reported, and the NeurIPS checklist claims \"Yes\" on statistical significance even though none are shown. That's a mismatch, but the underlying empirical pattern seems consistent across benchmarks.\n\nWho it's for: someone working on DPO variants and anchor regularizers will find the empirical result useful, but should not rely on the theoretical claims. The method itself could be reframed as anchor regularization and re-submitted with fixed theory or without overclaiming.\n\nRecommendation: I'd send it to review—the empirical contribution is enough to warrant referee time even though the current version would need major revision. A reviewer who catches the theorem error would likely reject the current version, but the framing can be fixed.","headline":"A simple anchor regularizer that helps in practice, wrapped in a broken calibration story; the theorem is false and the calibration claim doesn't match the implemented loss.","tokens_in":27977,"tokens_out":2342,"would_cite":false,"duration_ms":21152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single squared calibration term added to DPO keeps the chosen response's likelihood from falling during alignment and, according to the paper, makes the learned implicit rewards match ground-truth reward scale, yielding consistent gains…","keywords":["direct preference optimization","reward calibration","LLM alignment","contrastive preference learning","implicit reward","mode-seeking behavior","reverse KL divergence","RLHF"],"falsifier":"With a dataset whose true reward is known (e.g., the IMDb sentiment log-odds used in the paper), train Cal-DPO and compare the learned chosen-response implicit reward to $r(x,y_w)/\\beta$; the calibration claim predicts they match, so observing the implicit reward pinned near $1/(2\\beta)$ while the true reward has a different scale, or finding the Theorem 2 inequality violated when evaluated with the true reward, would refute the central claim.","tokens_in":26869,"feed_emoji":"🎯","tokens_out":12496,"duration_ms":95013,"temperature":0.7,"pith_summary":"Cal-DPO claims that contrastive preference optimization methods such as DPO fail to fully align language models because their objectives depend only on the relative difference between the implicit rewards of the chosen and rejected responses, not on the absolute values of those rewards. During training, the likelihood of the chosen response can therefore fall even while the preference margin grows, which is especially harmful for reasoning and mathematics. The paper's fix is a squared calibration term, added to the standard pairwise preference loss, that anchors the implicit reward of the chosen response to $+1/(2\\beta)$ and that of the rejected response to $-1/(2\\beta)$; this is a one-line modification of DPO with no new hyperparameters. The paper argues that this makes the learned implicit rewards comparable in scale to ground-truth rewards, and proves that, when calibrated against the true reward, Cal-DPO minimizes an upper bound on the KL-regularized RLHF objective and inherits its mode-seeking behavior. Across standard benchmarks, the paper reports consistent gains over DPO and several of its variants.","feed_headline":"Adding one loss term keeps DPO from degrading good answers","feed_subtitle":"A squared anchor at ±1/2β holds chosen-response rewards on scale, lifting math and instruction-following win rates.","key_machinery":"The central object is the implicit reward $\\hat{r}_\\theta(x,y)=\\log(\\pi_\\theta(y|x)/\\pi_{\\mathrm{ref}}(y|x))$, the log-likelihood ratio that DPO treats as a reward in place of a learned reward model. The machinery of Cal-DPO is the squared calibration term $(\\hat{r}_\\theta(x,y)-r(x,y)/\\beta)^2$ added to the pairwise preference loss; in the implemented loss, $r(x,y_w)=1/2$ and $r(x,y_l)=-1/2$, so the chosen log-ratio is anchored at $1/(2\\beta)$ and the rejected log-ratio at $-1/(2\\beta)$. The anchor sets the level, not just the margin, of the implicit rewards, which is what prevents the chosen response's reward from sliding downward while the preference margin grows. The theoretical engine is a convex-analysis inequality relating forward and reverse KL divergences: the reverse-KL RLHF objective is bounded by the forward-KL term plus the squared calibration error, yielding the claimed mode-seeking, upper-bound guarantee.","core_discovery":"The paper's central discovery claim is that the failure of contrastive preference methods is a scale-calibration failure, not a ranking failure. DPO-style losses keep increasing the margin between the chosen and rejected log-likelihood ratios, but nothing pins down the level of those ratios; the paper reports that DPO's chosen-response implicit reward drifts below zero during training while Cal-DPO's stays positive. Cal-DPO's objective is the DPO loss plus two squared anchors: $\\log(\\pi_\\theta(y_w|x)/\\pi_{\\mathrm{ref}}(y_w|x))$ pulled toward $1/(2\\beta)$ and the rejected log-ratio pulled toward $-1/(2\\beta)$. The paper defines calibration as equality with the true reward $r(x,y)/\\beta$, proves that the first term of its population objective has a negative-gradient property that raises high-reward responses and lowers low-reward ones, and proves that minimizing Cal-DPO minimizes an upper bound on the reverse-KL RLHF objective, making it mode-seeking like RLHF. On preference datasets the anchors are implemented with $r=+1/2$ and $r=-1/2$, and the experiments report consistent improvements over DPO, IPO, SLiC, CPO, and DPO variants on reasoning, summarization, dialogue, and sentiment tasks.","pith_inferences":["A natural extension is to estimate the reward scale from the preference data rather than fixing the targets at $\\pm 1/2$, since the practical targets are a stand-in for an unknown oracle scale.","Bridging the theory to the implemented fixed targets, for example by treating $\\pm 1/2$ as a normalized reward scale, would sharpen the interpretation of the method as calibration rather than generic regularization.","The same anchor idea should transfer to on-policy iterative preference learning, which the paper names as a limitation, and in that setting it could prevent repeated training rounds from eroding the likelihood of good responses."],"forward_implications":["If the paper is right, a practitioner can add one line to a DPO training loop and stop the chosen-response likelihood from decreasing, at no additional hyperparameter cost.","The theoretical equivalence means that Cal-DPO, DPO, and RLHF converge to the same optimal policy in the large-data, large-capacity limit, and Cal-DPO does so through a reverse-KL upper bound.","The calibration recipe transfers to other pairwise losses; Cal-IPO and Cal-SLiC inherit the same anchor-based protection, with reported gains on the Anthropic-HH dataset.","The largest reported gains are in instruction following and mathematics, where the chosen response is often the single correct answer and preserving its likelihood matters most."],"supporting_citations":[{"why":"Supplies the DPO pairwise loss and the implicit-reward parameterization that Cal-DPO augments with its calibration term.","marker":"[7]"},{"why":"Establishes that contrastive preference losses have negative-gradient and mode-seeking properties but remain relative-only, motivating reward-scale calibration.","marker":"[10]"},{"why":"Documents the failure mode where DPO decreases the likelihood of chosen responses on reasoning tasks, the problem Cal-DPO targets.","marker":"[11]"},{"why":"Provides the IPO squared-loss objective that Cal-DPO's calibration term is shown to extend.","marker":"[8]"},{"why":"Provides the SLiC hinge-loss objective and the sequence-likelihood calibration framing that Cal-DPO generalizes.","marker":"[9]"}],"fun_headline_variants":["Cal-DPO anchors rewards to stop good answer decline","Squared anchors fix DPO's reward scale drift","Cal-DPO keeps chosen rewards positive and aligned","Pin DPO's implicit rewards to ground truth scale","Squared anchors stop DPO reward degradation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fixed targets $+1/2$ and $-1/2$, used in the practical loss, stand in for the unknown true reward scale, because the paper's calibration and upper-bound promises are proved against the true reward $r(x,y)/\\beta$, not against these fixed numbers.","fun_headline_variants_meta":{"raw":{"variants":["Cal-DPO anchors rewards to stop good answer decline","Squared anchors fix DPO's reward scale drift","Cal-DPO keeps chosen rewards positive and aligned","Pin DPO's implicit rewards to ground truth scale","Squared anchors stop DPO reward degradation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3397,"prompt_tokens":963,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2361}},"tokens_in":579,"tokens_out":2434,"duration_ms":14139,"temperature":1.0,"reasoning_tokens":2361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:10:16.020650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"With a dataset whose true reward is known (e.g., the IMDb sentiment log-odds used in the paper), train Cal-DPO and compare the learned chosen-response implicit reward to $r(x,y_w)/\\beta$; the calibration claim predicts they match, so observing the implicit reward pinned near $1/(2\\beta)$ while the true reward has a different scale, or finding the Theorem 2 inequality violated when evaluated with the true reward, would refute the central claim.","supporting_citations":[],"review_version":1}