{"id":"ff7d74cd-15d6-4bfb-908e-32bff8eddeaf","arxiv_id":"2506.12725","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"BDPO replaces the rejected response probability in the DPO loss denominator with a mixture of the learned and reference policies, yielding better chosen-response probability and better benchmark scores.","lead":"A machine learning paper proposes Bounded-DPO (BDPO), a small change to the Direct Preference Optimization loss that caps how much rejected answers can pull training, so the model learns to favor chosen answers without drifting far from the original model. The method beats standard DPO and several variants on instruction-following and math benchmarks, though the gains are modest and based on single runs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical 'outperforming' claim rests on single runs and test-set lambda selection; margins are small enough to be seed noise.","rationale":"The reader's weakest_assumption focuses on single-pair theory transfer and the monotonicity assumption of Theorem 2; those are real limitations and the paper acknowledges the single-pair scope. I judge the empirical pillar to be more load-bearing for the paper's advertised conclusion: even if the multi-pair extension is left for future work, BDPO could still be a useful practical method, and the paper could say so; but 'outperforming existing algorithms' is a comparative empirical statement that the current evidence cannot support. Appendix B's 'All experiments were run once per configuration' plus Figure 5's lambda selection on the same benchmarks used in Tables 1-2, and the small margins, make the headline comparison vulnerable. No internal mathematical inconsistency was found in the BDPO loss or the presented proofs given their stated assumptions. The theoretical monotonicity assumption in Theorem 2 is also unverified, but a rerun with seeds would not fix that; an additional gradient-tracking check would. My recommendation remains conditional acceptance, with the required revisions being multi-seed evaluation, held-out lambda selection, and either verifying or weakening the monotonicity-based guarantee.","tokens_in":17593,"tokens_out":8464,"duration_ms":102318,"concrete_test":"Rerun the Qwen2.5-0.5B pipeline for DPO, DPOP, DPO+NLL, and BDPO with at least 5 seeds each, holding out a validation split of UltraFeedback for choosing lambda and reporting mean +/- std and paired significance on IFEval and GSM8K. If BDPO's mean margin over the best baseline (currently 0.58-0.60 IFEval total, 0.75 at 7B, 0.91 GSM8K) is within one standard deviation or fails a paired test at p<0.05, the 'outperforming' claim should be weakened to 'competitive' pending further evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that BDPO 'outperforms existing algorithms' is carried by the empirical section, and that section's strongest configuration is not statistically grounded. Appendix B states that all experiments were run once per configuration. Figure 5 selects lambda=0.5 as best on IFEval and GSM8K, and Tables 1-2 then report that same lambda on the same benchmarks without a held-out split. The reported advantages over the strongest baseline are small: 0.58-0.60 IFEval total points on Qwen2.5-0.5B, 0.75 points on Qwen2.5-7B, and 0.91 GSM8K points. Since no variance estimate, repeated seed, or paired test is provided, these margins are not distinguishable from run-to-run noise; and because the default baselines are not given the same lambda-style selection on a validation split, part of the gap may reflect selection favoritism rather than the loss design. The theoretical single-pair results are explicitly limited and cannot by themselves establish the comparative empirical claim. This is fixable, but as written the headline comparison is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that DPO's loss is dominated by the rejected response, causing the probability of the chosen response to stagnate or fall. It proposes BDPO, which replaces πθ(yl|x) in the DPO denominator with λπθ(yl|x)+(1−λ)πref(yl|x), so that the rejected-response contribution cannot vanish and the gradient remains bounded. The paper provides a single-pair theoretical analysis (Theorems 1 and 2), loss visualizations, toy experiments, training-dynamics plots, and evaluations on IFEval and GSM8K with Qwen2.5-0.5B and 7B models, reporting that BDPO outperforms DPO, DPOP, DPO+NLL, and several other baselines.","tokens_in":17787,"tokens_out":11250,"duration_ms":111047,"significance":"BDPO is a simple, computationally free modification with a clear motivation: it directly addresses a known failure mode of DPO while preserving DPO's overall optimization structure. The gradient computation in Section 4.1 is transparent and correctly shows that the problematic 1/πθ(yl) factor is replaced by λ/πmix(yl), which is bounded below. The paper also ships code and includes an unusually explicit limitations section. However, the headline claim that BDPO 'outperforms existing algorithms' is not established by the current experiments, which are single runs with test-set hyperparameter selection and small margins. The theoretical guarantees are conditional on a single-pair setting and an unrealistic monotonicity assumption, and one ablation figure appears to contradict the associated text. With these issues addressed, the contribution would be a useful incremental advance.","major_comments":[{"comment":"The paper states in Appendix B that all experiments were run once per configuration, and Figure 5 selects λ=0.5 on the same IFEval and GSM8K benchmarks that Tables 1-2 then report. In addition, Appendix B.3 says the final evaluation used the best-performing learning rate and epoch count for each algorithm. The reported advantages over the strongest baselines are small (0.58-0.60 IFEval points on Qwen2.5-0.5B, 0.75 points on Qwen2.5-7B, and 0.91 GSM8K points), and no variance estimates, repeated seeds, or paired significance tests are given. As written, the claim that BDPO outperforms existing algorithms is not statistically distinguishable from run-to-run noise and may reflect selection favoritism.","section":"Section 5.3 / Appendix B"},{"comment":"The text in Section 5.4 says BDPO 'consistently outperforms DPO across all tested λ values,' but the plotted values do not support this: on IFEval, BDPO at λ=0.1 is 24.7, below the DPO total score of 25.25 in Table 1; on GSM8K, the plotted y-range (28-30) excludes the DPO score of 27.45 in Table 2, making the comparison shown in the figure unclear. This internal inconsistency must be resolved before the robustness claim can be evaluated.","section":"Section 5.4 / Figure 5"},{"comment":"Theorem 2's lower bound (1−λ)πref(yw|x) ≤ πθ(yw|x) is proved under the assumption that every optimization step decreases the BDPO loss monotonically. Stochastic gradient descent does not satisfy this assumption; the loss can increase between steps. The main text's statement that 'BDPO provides a lower bound for the chosen probability' therefore overstates the guarantee. In addition, Theorems 1 and 2 are stated only for a single preference pair per prompt, while the experiments train on datasets with many pairs per prompt. The Limitations section acknowledges the single-pair scope, but the abstract and Section 4.1 should qualify the 'guarantees' accordingly.","section":"Section 4.1 / Appendix I.3"}],"minor_comments":[{"comment":"The sentence 'setting πθ(yl|x) = 0 leads to an unbounded loss' is inaccurate: as πθ(yl|x)→0, the DPO loss −log σ(rθ(yw)−rθ(yl)) tends to 0 because rθ(yl)→−∞. What is unbounded is the gradient with respect to πθ(yl), specifically the factor 1/πθ(yl). The later gradient analysis in Section 4.1 states this correctly, and the earlier sentence should be revised for consistency.","section":"Section 3.1"},{"comment":"The main text says 'one pair of preference data is randomly selected, and the remaining two responses are implicitly treated as OOD,' while Appendix B.1 says 'two chosen and two rejected responses were selected at random' for each prompt. These descriptions are inconsistent and make the toy experiment irreproducible; please clarify whether one pair or two pairs per prompt were used.","section":"Section 4.2.1 / Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The method is plausible and the core gradient argument is sound, so I would not reject on theoretical grounds. The main risk is overclaiming: the empirical comparison currently rests on single runs and test-set hyperparameter selection, and one ablation figure appears inconsistent with the text and with Table 2. The authors should be asked to provide multi-seed results with confidence intervals, a proper validation split for λ and other hyperparameters, and a corrected Figure 5 before the 'outperforming existing algorithms' claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BDPO is the kind of simple modification to DPO that deserves attention: replace pi_theta(y_l) in the denominator with a lambda-mixture of pi_theta and pi_ref, which bounds the gradient and changes the optimum to put mass on the chosen response. The analysis of DPO's failure mode is not new - Pal, Razin, and others already said DPO can lower chosen probability and drift OOD - but the specific loss is a clean, previously absent variant. The gradient derivations and toy experiments are internally consistent, and the paper is honest about its limitations (single-pair theory, single runs, GPU constraints). The computational cost table is a nice touch.\n\nThe soft spots are empirical, not conceptual. All experiments are run once per configuration, and lambda=0.5 is selected by looking at Figure 5 on the same IFEval and GSM8K benchmarks that later appear in Tables 1 and 2. The reported margins over the strongest baselines are about 0.6-0.9 points - well within seed noise for these benchmarks. So the headline 'outperforming existing algorithms' is not supported as written. This is fixable: run 3-5 seeds, report means and variances, and pick lambda on a validation split (or show the main results are insensitive to lambda, which Figure 5 partially suggests).\n\nThe theory is straightforward and mostly correct, but Theorem 1 essentially restates the design: if you put a positive floor under the denominator, the optimum is chosen-mass-1. Theorem 2's lower bound rests on the assumption that loss decreases monotonically at every step, which SGD does not guarantee; the paper states this but does not check it. Neither of these is fatal, but the title's promise of 'rethinking DPO' is overreach - the diagnosis is prior work and the fix is a tweak, not a paradigm shift.\n\nFor the right reader - someone cataloging DPO variants or looking for a quick regularizer - this is a useful paper. I'd send it to review, but with a clear request for repeated runs and a proper hyperparameter selection procedure. The math checks out and the idea is reproducible; the empirical section is just not there yet.","headline":"A clean one-line DPO fix with an honest but under-powered empirical section; the theory mostly restates the design, and the 'outperforms' claim needs error bars and a held-out lambda.","tokens_in":18326,"tokens_out":4004,"would_cite":true,"duration_ms":41463,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Direct Preference Optimization is dominated by rejected responses; Bounded-DPO rebalances the loss so chosen responses win, and it tops IFEval and GSM8K.","keywords":["Direct Preference Optimization","Bounded-DPO","preference misalignment","rejected responses","reference model alignment","loss geometry","instruction following","mathematical reasoning"],"falsifier":"Train BDPO on a dataset with several preference pairs per prompt while tracking $\\pi_\\theta(y_w\\mid x)$ and the in-distribution mass $\\pi_\\theta(y_w\\mid x)+\\pi_\\theta(y_l\\mid x)$; if, in a run with monotonically decreasing BDPO loss, any prompt's chosen-response probability falls below $(1-\\lambda)\\pi_{\\mathrm{ref}}(y_w\\mid x)$, or if the in-distribution mass drifts downward as with DPO, then the single-pair guarantees do not transfer to the multi-pair setting.","tokens_in":17396,"feed_emoji":"⚖️","tokens_out":14658,"duration_ms":144369,"temperature":0.7,"pith_summary":"Direct Preference Optimization (DPO) is meant to raise the probability of chosen responses and lower that of rejected ones, but the paper shows the loss can be driven to its minimum by suppressing only the rejected response, with the chosen response barely moving or even decreasing. The proposed fix, Bounded-DPO (BDPO), replaces the model's rejected-response probability inside the DPO loss with a mixture of that probability and the reference model's probability, so the rejected term can no longer dominate the gradient. In a single preference-pair setting, BDPO provably has minimizers with $\\pi_\\theta(y_w\\mid x)=1$ and $\\pi_\\theta(y_l\\mid x)=0$, and it keeps the chosen-response probability above $(1-\\lambda)\\pi_{\\mathrm{ref}}(y_w\\mid x)$ under monotone optimization. In experiments on instruction-following and math benchmarks at two model scales, BDPO outperforms DPO, DPOP, and DPO+NLL while adding no training-cost overhead. The paper's proposal is thus a low-cost loss-level correction that makes DPO actually serve its stated objectives.","feed_headline":"Bounded-DPO stops rejected responses from dominating DPO","feed_subtitle":"Replacing the rejected probability with a reference-model mixture gives DPO its intended behavior and top scores.","key_machinery":"The load-bearing object is the mixture distribution $\\pi_{\\mathrm{mix}}(y\\mid x)=\\lambda\\pi_\\theta(y\\mid x)+(1-\\lambda)\\pi_{\\mathrm{ref}}(y\\mid x)$, placed where DPO has $\\pi_\\theta(y_l\\mid x)$ in the denominator of its log-ratio. Because $\\pi_{\\mathrm{mix}}(y_l\\mid x)\\ge(1-\\lambda)\\pi_{\\mathrm{ref}}(y_l\\mid x)>0$, the loss cannot be minimized by collapsing the rejected response to zero, and the gradient factor $\\lambda/\\pi_{\\mathrm{mix}}(y_l\\mid x)$ stays finite even when $\\pi_\\theta(y_l\\mid x)\\to0$, whereas DPO, DPO+NLL, and DPOP all retain the unbounded factor $1/\\pi_\\theta(y_l\\mid x)$. The mixing weight $\\lambda$ sets how much of the denominator follows the trained model versus the reference model, and the ablation finds $\\lambda=0.5$ gives the best results while BDPO beats DPO at every tested value.","core_discovery":"The paper's central claim is that DPO's poor chosen-response behavior is a structural consequence of the term $1/\\pi_\\theta(y_l\\mid x)$ in its gradient: as the rejected-response probability approaches zero this term drives rejected-dominated, unbounded updates, so the objective can be nearly satisfied without ever promoting $\\pi_\\theta(y_w\\mid x)$. BDPO removes that failure mode by defining the loss $L_{\\mathrm{BDPO}}=-\\mathbb{E}_D\\log\\sigma\\bigl(\\beta\\log\\frac{\\pi_\\theta(y_w\\mid x)}{\\lambda\\pi_\\theta(y_l\\mid x)+(1-\\lambda)\\pi_{\\mathrm{ref}}(y_l\\mid x)}-\\hat{s}_{\\mathrm{ref}}(w;l)\\bigr)$, whose mixture denominator is bounded below by $(1-\\lambda)\\pi_{\\mathrm{ref}}(y_l\\mid x)$ and whose gradient with respect to the rejected probability is $\\beta\\sigma(-\\Delta_{\\mathrm{BDPO}})\\cdot\\lambda/\\pi_{\\mathrm{mix}}(y_l\\mid x)$ rather than the unbounded $1/\\pi_\\theta(y_l\\mid x)$. For one preference pair per prompt, the paper proves the BDPO optimum is $\\pi_\\theta(y_w\\mid x)=1$ and $\\pi_\\theta(y_l\\mid x)=0$, so every BDPO minimizer also minimizes DPO but not conversely, and it proves that monotone loss decrease implies $\\pi_\\theta(y_w\\mid x)\\ge(1-\\lambda)\\pi_{\\mathrm{ref}}(y_w\\mid x)$. Empirically, BDPO raises chosen log-probabilities, lowers rejected ones, keeps KL divergence to the reference model small, and records the best IFEval and GSM8K scores among the compared methods, with $\\lambda=0.5$ recommended.","pith_inferences":["A natural extension the paper leaves implicit: in multi-pair settings, BDPO should behave like a KL-regularized reward maximizer, so its learned policy should resemble policies trained by reward-based RLHF more than DPO-trained policies do; comparing output distributions would test this.","The lower bound $(1-\\lambda)\\pi_{\\mathrm{ref}}(y_w\\mid x)$ can be used as a monitoring signal: in a run whose BDPO loss is monotonically decreasing, any measured chosen-response probability below that bound would indicate the single-pair theory has not transferred to the training distribution.","An annealed $\\lambda$ schedule, starting near 1 for early exploration and decreasing toward 0.5 to anchor to the reference, could combine DPO's flexibility with BDPO's stability; the paper only tests fixed values."],"forward_implications":["DPO's failure to increase chosen-response probability is a property of its loss geometry, not a tuning artifact; any variant that keeps the unbounded $1/\\pi_\\theta(y_l\\mid x)$ factor, including DPOP and DPO+NLL, inherits the same rejected-response dominance.","BDPO satisfies DPO's two stated objectives at once: the chosen-response probability has a reference-anchored lower bound under monotone optimization, and the rejected-response probability is driven to zero at the optimum.","Because BDPO changes only the loss and adds no parameters or sampling, it can replace the DPO loss in existing training pipelines with no additional wall-clock time.","The balanced optimization translates into higher task scores than DPO, DPOP, and DPO+NLL on both instruction-following (IFEval) and mathematical reasoning (GSM8K), at both tested model scales.","BDPO reduces negative log-likelihood on chosen responses without an explicit NLL term, suggesting its reference mixture supplies the chosen-response pressure that DPO+NLL adds by hand."],"supporting_citations":[{"why":"Introduces DPO and the loss whose rejected-response structure BDPO modifies; it is the primary baseline.","marker":"Rafailov et al. 2024b"},{"why":"Demonstrates that DPO can decrease the probabilities of both chosen and rejected responses, the evidence the paper re-derives from the loss geometry.","marker":"Xu et al. 2024"},{"why":"Proposes DPOP, the penalty-based fix that the paper analyzes and shows still leaves the rejected-response term dominant.","marker":"Pal et al. 2024"},{"why":"Proposes the DPO+NLL combination whose hyperparameter sensitivity and reference-model drift the paper analyzes.","marker":"Pang et al. 2024"},{"why":"Documents likelihood displacement in DPO, supporting the paper's claim that chosen and rejected probabilities can fall together.","marker":"Razin et al. 2024"},{"why":"Supplies the UltraFeedback binarized dataset used for both the training-dynamics analysis and the IFEval experiments.","marker":"Cui et al. 2024"},{"why":"Supplies GSM8K, the mathematical reasoning benchmark on which BDPO reports the highest accuracy.","marker":"Cobbe et al. 2021"},{"why":"Supplies IFEval, the instruction-following benchmark used for the main evaluation results.","marker":"Zhou et al. 2023"},{"why":"Supplies the model family used for the 0.5B and 7B experiments on which BDPO outperforms baselines.","marker":"Yang et al. 2024"}],"fun_headline_variants":["BDPO bounds rejected-response influence to fix DPO's core flaw","Reference mixture caps rejected answers, restoring DPO's aim","BDPO tames the unbounded gradient term from rejected responses","DPO's misalignment stems from rejected answers; BDPO fixes it","A simple bound on rejected responses lets DPO optimize both sides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the clean single-pair result—where assigning the chosen response probability 1 and the rejected response probability 0 is optimal—still controls real LLM training, which uses many preference pairs, finite batches, and optimization steps that do not steadily lower the loss at every step.","fun_headline_variants_meta":{"raw":{"variants":["BDPO bounds rejected-response influence to fix DPO's core flaw","Reference mixture caps rejected answers, restoring DPO's aim","BDPO tames the unbounded gradient term from rejected responses","DPO's misalignment stems from rejected answers; BDPO fixes it","A simple bound on rejected responses lets DPO optimize both sides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2327,"prompt_tokens":1037,"completion_tokens":1290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":1199}},"tokens_in":653,"tokens_out":1290,"duration_ms":14273,"temperature":1.0,"reasoning_tokens":1199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:43:46.991431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train BDPO on a dataset with several preference pairs per prompt while tracking $\\pi_\\theta(y_w\\mid x)$ and the in-distribution mass $\\pi_\\theta(y_w\\mid x)+\\pi_\\theta(y_l\\mid x)$; if, in a run with monotonically decreasing BDPO loss, any prompt's chosen-response probability falls below $(1-\\lambda)\\pi_{\\mathrm{ref}}(y_w\\mid x)$, or if the in-distribution mass drifts downward as with DPO, then the single-pair guarantees do not transfer to the multi-pair setting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UltraFeedback binarized dataset used for both the training-dynamics analysis and the IFEval experiments."}],"review_version":1}