{"id":"baa1632f-51e0-4417-bb50-c695d96c9256","arxiv_id":"2502.07599","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"DPO-Shift multiplies the rejected-response term in the DPO loss by a factor f(λ) below 1, shifting chosen-response probabilities upward at the expense of some reward margin.","lead":"DPO-Shift is a small change to Direct Preference Optimization: it scales down the penalty on rejected responses, which the authors say stops the model from losing probability on the preferred answers. The paper reports that this fixes the likelihood displacement problem of DPO, though its theoretical proof is shaky.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2.1 does not follow from its proof: the one-step gap g1(t+1) contains an extra η(c1−c)‖∇log π(yw)‖² term that is dropped without justification, so the factorization g1=(1−f)u1 holds only under an unstated f≈1 assumption.","rationale":"The reader identified the empirical sign-pattern assumption as the weakest point, and that is indeed a concern: if u1>0 and u2<0 are not generic, the recommended f(λ)<1 could backfire. However, I see a more fundamental problem upstream of the sign pattern. Theorem 2.1 is the paper's theoretical backbone, and its proof contains an unsupported simplification that is only valid for f(λ) close to 1. The paper explicitly claims the analysis holds without additional assumptions, yet the proof's algebra requires c1≈c, which is not stated and is not true for the small-f(λ) values used in the ablation studies. Because the central claim of a 'theoretically grounded' controllable trade-off depends on this theorem, the current manuscript does not establish its main theoretical contribution. I still credit the empirical ablations as suggestive evidence: the reported distributions do shift monotonically with f(λ), and the win-rate experiments are internally consistent. But the theory as presented is not a valid proof of the claimed guarantee, so the verdict should move from conditional to reject until the theorem is either corrected or its claims are weakened to match what the proof actually shows.","tokens_in":21526,"tokens_out":10051,"duration_ms":91235,"concrete_test":"Re-derive Eq. (8) from the proof's Eq. (12)–(13) without dropping the ‖∇log π(yw)‖² term; then compute both sides on the UltraFeedback test set for the SFTed Llama 3-8B model at f(λ)=0.75 using the paper's own definitions. If ηE[(c1−c)‖∇log π(yw)‖²] is nonzero and g1 differs from (1−f(λ))u1, Theorem 2.1 is false as stated. An independent re-derivation is decisive: the extra term does not vanish identically unless c1=c for all samples.","verdict_should_be":"REJECT","load_bearing_attack":"The central theoretical claim (C.2) rests on Theorem 2.1, but the proof in Appendix B does not establish the stated factorization. From the proof's own update θ_{t+1}=θ_t+η(c1∇log πθ(yw|x)−c2∇log πθ(yl|x)) and the first-order expansion in Eq. (12)–(13), the one-step gap for ω1 is η E[(c1−c)‖∇log πθ(yw|x)‖² − (c2−c)∇log πθ(yl|x)ᵀ∇log πθ(yw|x)], where c1=βσ(f(λ)β log(πθ(yl|x)/πref(yl|x)) − β log(πθ(yw|x)/πref(yw|x))) and c is the same expression with f(λ)=1. The proof suppresses the first term and replaces c2−c by c1−c2; this is only legitimate when c1=c, i.e., when f(λ)≈1. The theorem is stated for all 0<f(λ)<1 and is used to conclude that f(λ)<1 improves chosen probability, so the paper's 'no additional assumptions' guarantee is not supported. The same c1=c simplification underlies the g2 formula. An additional defect is that γ in Theorem 2.1 is never defined in the main text, while the proof switches to β and an ad hoc 'β→∞' remark. Even if the empirical sign pattern u1>0, u2<0 is generic, the claimed trade-off formula does not follow from the proof as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPO-Shift, a modification of the DPO objective in which the rejected-response log-ratio term is multiplied by a function f(λ)∈(0,1). The central theoretical claim (Theorem 2.1, contribution C.2) is that after one gradient step the gap in chosen log-probability and in a smoothed reward-margin indicator between DPO-Shift and DPO factorizes as (1−f(λ))u1 and (1−f(λ))u2; combined with empirically measured signs u1>0 and u2<0, this is used to conclude that f(λ)<1 improves chosen probability at the cost of a reduced reward margin. The paper validates the predicted trade-off with ablations on Llama 3-8B and Qwen 2-7B over UltraFeedback and Capybara, and reports win-rate comparisons against DPO and other preference optimization methods.","tokens_in":21916,"tokens_out":8732,"duration_ms":74920,"significance":"If the theoretical claim were established, DPO-Shift would be a simple and practically useful remedy for likelihood displacement, a well-documented failure mode of DPO. The paper's strengths are its simple formulation, the released code, and the unusually thorough ablation over f(λ) values and multiple model/dataset combinations; the downstream comparison against SimPO, KTO, and IPO using their released checkpoints is also informative. However, the advertised 'theoretically grounded' guarantee is the main added value over the empirical observation, and the proof of Theorem 2.1 does not support the theorem as stated. The empirical sign pattern used to draw the main conclusion is measured on a single model/dataset pair, and the win-rate evaluation is partly circular, so the strongest claims in the paper are not yet supported.","major_comments":[{"comment":"The theorem as stated is not a consequence of the proof in Appendix B. Using the proof's own update θ_{t+1}=θ_t+η(c1∇logπθ(yw|x)−c2∇logπθ(yl|x)), the one-step difference in ω1 between DPO-Shift and DPO is ηE[(c1^f−c1^0)||∇logπθ(yw|x)||² − (f c1^f−c1^0)∇logπθ(yl|x)^T∇logπθ(yw|x)], where the superscripts denote f(λ) and f=1. The proof drops the first term and thereby obtains g1=η(c1−c2)∇logπ(yl)^T∇logπ(yw); this is only legitimate when f≈1, not for all 0<f(λ)<1 as the theorem states. In addition, the theorem's g1 omits the factor η present in the proof, and the theorem's u2 in Eq. (9) omits the c(θ) factor that the proof's g2 expression contains. Finally, c(θ) in the theorem uses γ while the proof's c1 uses β, and γ is never assigned a value in the main text; the proof's remark 'use β → +∞' is not a derivation. The theorem therefore needs either a corrected statement with explicit f≈1, η, and constant factors, or a complete derivation under its stated hypotheses.","section":"Theorem 2.1 / Appendix B"},{"comment":"The sign conclusion u1>0 and u2<0 is empirical, measured on one SFTed Llama 3-8B model on the UltraFeedback test set, and is not a consequence of the theorem. The manuscript does not verify the sign pattern at initialization for Qwen 2-7B, Capybara, or later training stages, so the recommendation f(λ)<1 is not established as generic. Moreover, the main text and Appendix B contradict each other: Section 2.2 reports 81.7% of the u_i^2 values as negative, while Appendix B reports 81.7% of the bracket term ∇logπ(yl)^T∇logπ(yw)−||∇logπ(yl)||² as positive; since the main text sets η1 to 1 when computing u2, these two statements cannot both be correct. This internal inconsistency weakens the empirical basis for the sign of u2.","section":"Section 2.2"},{"comment":"The win-rate experiment is partly circular. The judge prompt instructs that 'the most important criterion is to select the response whose meaning is essentially closer to the reference answer,' and the reference answer is the chosen response whose likelihood DPO-Shift is specifically designed to increase. The evaluation metric therefore directly rewards the quantity DPO-Shift optimizes, so Table 1 does not provide an independent demonstration of better alignment quality. The perplexity comparison (4.475 vs 18.996) is similarly computed on the chosen responses of the test split, which again rewards the objective being shifted. Please re-run with a judge that does not have access to the reference answer, or report a judge based on general quality only.","section":"Section 3.3 / Appendix A.2"},{"comment":"The abstract states that DPO-Shift demonstrates superiority on downstream tasks 'such as MT-Bench and a designed win rate experiment,' but no MT-Bench results appear anywhere in the main text or appendices. Either the MT-Bench results should be reported, or the abstract should be revised to mention only the win-rate experiment, which is the downstream evaluation actually presented.","section":"Abstract / Contribution C.3"}],"minor_comments":[{"comment":"The sentence beginning 'A carefully chosen For example' is grammatically broken and should be rewritten.","section":"Section 3.2"},{"comment":"The notation u_i^1 and u_i^2 is hard to read; using a sample index as a subscript (e.g., u_1,i) would clarify the definitions.","section":"Section 2.2"},{"comment":"The phrase 'the first one is to fix it all along the optimization process, i.e., f(λ)=λ' is confusing because λ is never defined as a variable separate from the function argument; please define λ and the fixed strategy explicitly.","section":"Section 2.2"},{"comment":"The phrase 'we abuse notation and use β → +∞ as hyper parameter' should be replaced with a precise statement about the smoothing factor γ; as written it suggests taking an infinite temperature limit that is not part of the training procedure.","section":"Appendix B"},{"comment":"The proof states that 81.7% of the computed bracket values 'turn out to be positive,' which is inconsistent with the main text's claim that 81.7% of the u_i^2 values are negative; please reconcile these numbers.","section":"Appendix B"},{"comment":"Entries such as 'linear_increase 0.95' should specify whether 0.95 is λ_min or λ_max; currently the reader must infer the convention from the text.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core theorem is not proved as stated, and the sign-frequency inconsistency between the main text and Appendix B is concerning. The win-rate evaluation is also confounded by the judge's explicit use of the reference answer. These issues are potentially fixable by restating the theorem as an f≈1 approximation, verifying the sign pattern across more models/datasets, and redesigning the evaluation. If the authors cannot provide a corrected proof or a non-circular downstream evaluation, rejection may be warranted, but the extensive empirical ablations and released code make a major revision the appropriate next step."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes DPO-Shift, a one-line change to DPO: scale the rejected-response log-ratio by f(λ) < 1. The empirical story is consistent and well documented: smaller f shifts probability mass onto chosen responses, raises log πθ(yw|x), and lowers the reward margin. The ablations span two models and two datasets, and the comparison with α-DPO, SimPO, KTO, and IPO is informative. That part is worth reading.\n\nThe theory, however, does not hold up. Theorem 2.1 claims g1 = (1−f)u1 and g2 = (1−f)u2, but the proof in Appendix B derives g1 = η(c1−c2)∇yl^T∇yw with c2 = f(λ)c1, which is η(1−f)c1 times the inner product. The theorem drops η, replaces c1 by a differently defined c(θ) that contains a smoothing factor γ never defined in the main text, and omits the η(c1−c)‖∇log π(yw)‖² term that appears in the first-order expansion. That term vanishes only when f(λ)≈1, so the theorem cannot support the paper's conclusion for all 0<f(λ)<1. The proof also hand-waves with 'β → +∞' after introducing γ. This is not a minor typo; the central claimed guarantee is not established.\n\nTwo more issues. The abstract mentions MT-Bench results, but no MT-Bench numbers appear anywhere in the paper. And the win-rate evaluation is partially circular: the judge is instructed that the most important criterion is closeness to the reference answer, which is exactly what DPO-Shift is designed to increase. The perplexity numbers are suggestive but not a substitute.\n\nWhere does that leave us? The method itself is simple and may well be useful as an empirical heuristic for controlling likelihood displacement. The trade-off curve in the experiments is real and reproducible-looking. But the paper overclaims a theoretical grounding it does not have, and the headline evaluation overclaims a benchmark it did not report. I would send this to a serious referee, because the empirical phenomenon deserves scrutiny and the method is cheap to test. But I would expect the theory either to be fixed or honestly demoted to 'motivating heuristic,' and the MT-Bench claim removed or substantiated.\n\nMy honest take: this is a borderline paper that could become a solid empirical note after revision. If you work on DPO variants, it's worth skimming the ablation plots.","headline":"A one-line DPO variant with solid empirical ablations but a broken proof and an overstated abstract; the trade-off is real, the theory as stated is not.","tokens_in":22470,"tokens_out":3449,"would_cite":true,"duration_ms":29297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DPO-Shift adds a tunable factor $f(\\lambda)$ to the rejected term of the DPO loss and proves this converts likelihood displacement into a controlled trade-off: chosen-probability gains at the price of a smaller reward margin.","keywords":["Direct Preference Optimization","likelihood displacement","preference alignment","Bradley-Terry model","reward margin","tunable parameter f(λ)","win rate evaluation","LLM alignment"],"falsifier":"Compute $u_1$ and $u_2$ on a different model family and preference dataset—say a Pythia-style checkpoint with the HH-RLHF dataset—at several training checkpoints; if any setting yields $u_1 \\le 0$ or $u_2 \\ge 0$, the paper's guarantee that $f(\\lambda)<1$ improves chosen probability while reducing reward margin fails for that setting, and the recommended $f(\\lambda)=0.95$ would need to be revisited.","tokens_in":21289,"feed_emoji":"🎛️","tokens_out":11187,"duration_ms":81745,"temperature":0.7,"pith_summary":"Direct Preference Optimization (DPO) is known to lower the probability the model assigns to chosen responses during training—a failure called likelihood displacement. This paper claims that multiplying the rejected response's log-ratio inside DPO's Bradley-Terry objective by a parameter function $f(\\lambda)$ between 0 and 1 makes the chosen probability shift upward in a controllable way. The theoretical centerpiece is a one-step gap analysis showing that the change in chosen probability is $(1-f(\\lambda))u_1$ and the change in the smoothed reward margin is $(1-f(\\lambda))u_2$, where the measured signs $u_1>0$, $u_2<0$ imply a fundamental trade-off governed entirely by $f(\\lambda)$. Experiments on Llama 3-8B and Qwen 2-7B across UltraFeedback and Capybara confirm that $f(\\lambda)$ close to 1 (e.g., 0.95) lifts chosen probabilities while keeping reward margins nearly unchanged, and a designed win-rate evaluation has DPO-Shift beating DPO. A careful reader would care because the method is as simple as DPO, needs no dataset changes, and turns a known pathological behavior into a tunable dial.","feed_headline":"One tunable factor tames DPO's likelihood displacement","feed_subtitle":"DPO-Shift trades a small reward-margin drop for controllable chosen-probability gains and beats DPO on win rates.","key_machinery":"The load-bearing object is the modified Bradley-Terry loss $L_{\\mathrm{DPO-Shift}}(\\pi)=-\\mathbb{E}[\\log \\sigma(\\beta \\log \\frac{\\pi_\\theta(y_w|x)}{\\pi_{\\mathrm{ref}}(y_w|x)} - f(\\lambda) \\beta \\log \\frac{\\pi_\\theta(y_l|x)}{\\pi_{\\mathrm{ref}}(y_l|x)})]$, which inserts the parameter function $0<f(\\lambda)<1$ as a multiplier on the rejected response's reward. The analysis proceeds by comparing one gradient step of this objective against one step of DPO: the gap in chosen log-probability and the gap in the smoothed reward-margin indicator factor as $(1-f(\\lambda))u_1$ and $(1-f(\\lambda))u_2$, with $u_1$ and $u_2$ defined as expectations of gradient inner products $\\nabla_\\theta \\log \\pi_\\theta(y_l|x)^\\top \\nabla_\\theta \\log \\pi_\\theta(y_w|x)$ and related terms. Because the empirically measured signs are $u_1>0$ and $u_2<0$ on the SFTed Llama 3-8B / UltraFeedback configuration, the factor $(1-f(\\lambda))>0$ directly yields the paper's core conclusion: an improvement in chosen probability that costs reward margin, with the size of both effects set by $f(\\lambda)$.","core_discovery":"On its own terms, the paper establishes that the likelihood displacement of DPO can be mitigated by a one-line change to the loss: replace the DPO objective's rejected log-ratio with $f(\\lambda)\\log(\\pi_\\theta(y_l|x)/\\pi_{\\mathrm{ref}}(y_l|x))$, keeping $0<f(\\lambda)<1$. Theorem 2.1 shows that after a single gradient step, the gap between DPO-Shift and DPO in chosen log-probability is exactly $(1-f(\\lambda))u_1$, and the gap in the smoothed reward-margin indicator is $(1-f(\\lambda))u_2$, where $u_1$ and $u_2$ are expectations over inner products of the chosen and rejected gradient log-probabilities. Measured on an SFTed Llama 3-8B with UltraFeedback, $u_1>0$ and $u_2<0$, so every $f(\\lambda)<1$ improves the chosen probability at the expense of the reward margin. The paper further shows the trade-off magnitude scales with $1-f(\\lambda)$: values close to 1 give near-DPO reward margins with a visible chosen-probability gain, while small values push chosen probability up but reduce reward accuracy and can overfit to chosen responses. The authors conclude the method mitigates likelihood displacement, is controlled by a scalar schedule that can be fixed or varied linearly over training, and outperforms DPO in downstream win-rate comparisons and perplexity.","pith_inferences":["The sign of $u_1$ should track the semantic similarity of chosen/rejected pairs: on datasets with mostly dissimilar pairs, $u_1<0$ may hold, and the paper's own analysis implies $f(\\lambda)>1$ could then improve both chosen probability and reward margin simultaneously—a regime worth testing systematically rather than writing off as a crash risk.","A per-sample adaptive choice of $f(\\lambda_i)$, assigned according to the sign of $u^i_1$ for each training pair, is the natural next step the authors leave as future work; one could test it by computing $u^i_1$ on the training set and applying $f(\\lambda_i)<1$ where $u^i_1>0$ and $f(\\lambda_i)>1$ where $u^i_1<0$.","The sign-pattern measurement is the paper's empirical linchpin, so independently reproducing $u_1>0$, $u_2<0$ across architectures and preference datasets would either widen or shrink the domain where the recommended $f(\\lambda)<1$ is safe.","Because likelihood displacement is also addressed by data filtering approaches, DPO-Shift's soft weighting of the rejected term may compound with filtering—e.g., DPO-Shift on a filtered DPO-Positive-style dataset might recover margin loss while keeping chosen-probability gains."],"forward_implications":["With $f(\\lambda)$ close to 1 (e.g., 0.95–0.99), DPO-Shift raises the chosen probability relative to DPO while keeping the reward margin and reward accuracy nearly unchanged, directly mitigating likelihood displacement.","Smaller $f(\\lambda)$ increases the chosen probability further but progressively reduces the reward margin and reward accuracy; the paper observes that very small values can overfit to chosen responses and degrade preference alignment.","The trade-off is explicitly quantified by $(1-f(\\lambda))u_1$ and $(1-f(\\lambda))u_2$, so practitioners can choose $f(\\lambda)$—or a linear schedule over training—to hit a desired balance between likelihood and margin.","On downstream evaluation, DPO-Shift beats DPO in a designed win-rate comparison (e.g., 72.15% vs 27.85% with linear_decrease $\\lambda_{min}=0.95$ on Llama 3-8B / UltraFeedback) and produces lower perplexity on chosen responses (4.475 vs 18.996).","The modification requires no dataset filtering, no reference-model retraining, and no extra training stages; it is a drop-in change to the DPO loss."],"supporting_citations":[{"why":"Defines the DPO objective and reward parameterization that DPO-Shift modifies; the baseline throughout the paper.","marker":"[26]"},{"why":"The Bradley-Terry model in which DPO-Shift inserts the f(λ) multiplier on the rejected response.","marker":"[7]"},{"why":"Names and analyzes likelihood displacement, the failure mode DPO-Shift targets, providing the paper's motivation.","marker":"[27]"},{"why":"Attributes likelihood displacement to model capacity and multiple samples, supporting the semantic-similarity explanation the paper builds on.","marker":"[29]"},{"why":"Prior fix for DPO failure modes (Smaug/DPO-Positive) that motivates addressing likelihood displacement via loss modification.","marker":"[21]"},{"why":"UltraFeedback: the dataset used for the u1>0/u2<0 sign measurements and the main training and win-rate experiments.","marker":"[11]"},{"why":"SimPO: source of the SFTed Llama 3-8B checkpoint and training configuration used in all experiments; also a comparison baseline.","marker":"[18]"},{"why":"IPO baseline for win-rate and reward-margin comparisons in the downstream and ablation tables.","marker":"[4]"},{"why":"KTO baseline used in the win-rate experiment and ablation table.","marker":"[14]"}],"fun_headline_variants":["Single scalar rebalances DPO's chosen odds","A one-line change shifts DPO's chosen probability","DPO-Shift: trade margin for chosen-probability gain","Control DPO's chosen-probability shift with a scalar"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that any $f(\\lambda)<1$ improves the chosen probability rests on the measured sign pattern $u_1>0$ and $u_2<0$, which was obtained on a single model (SFTed Llama 3-8B) and a single dataset (UltraFeedback test set); if that sign pattern is not generic across models, datasets, and training stages, the recommended choice of $f(\\lambda)<1$ could reduce chosen probability instead of improving it.","fun_headline_variants_meta":{"raw":{"variants":["Single scalar rebalances DPO's chosen odds","A one-line change shifts DPO's chosen probability","DPO-Shift: trade margin for chosen-probability gain","Control DPO's chosen-probability shift with a scalar"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3361,"prompt_tokens":1035,"completion_tokens":2326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":2259}},"tokens_in":651,"tokens_out":2326,"duration_ms":15545,"temperature":1.0,"reasoning_tokens":2259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:11:13.984918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $u_1$ and $u_2$ on a different model family and preference dataset—say a Pythia-style checkpoint with the HH-RLHF dataset—at several training checkpoints; if any setting yields $u_1 \\le 0$ or $u_2 \\ge 0$, the paper's guarantee that $f(\\lambda)<1$ improves chosen probability while reducing reward margin fails for that setting, and the recommended $f(\\lambda)=0.95$ would need to be revisited.","supporting_citations":[{"cited_title":"UltraFeedback: Boosting language models with high-quality feedback","cited_arxiv_id":null,"evidence_quote":"UltraFeedback: the dataset used for the u1>0/u2<0 sign measurements and the main training and win-rate experiments."}],"review_version":1}