{"id":"059a0c9c-308e-4fe0-83f8-8d6ef23adf8a","arxiv_id":"2607.16240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts primarily through low-likelihood outlier tokens.","lead":"This paper adds a penalty term to the DPO and SimPO preference-training objectives that holds the combined likelihood of the chosen and rejected answers roughly constant during training. On instruction-tuned 7B models it reports better win-rates on instruction-following benchmarks and less 'likelihood displacement', and shows that most of the damage from DPO-style training is concentrated in a small set of outlier tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Isolated effect of the regularizer is unverified: N-DPO is length normalization + penalty, and the λ=0 benchmark control is missing; appendix N-DPO numbers also disagree with Table 1.","rationale":"The paper's central claim has two layers: a mechanistic story about the partition function and a concrete empirical claim that the regularizer improves the generation/benchmark trade-off. The mechanistic story is indeed under-supported, as the reader notes. But the more load-bearing problem is that the empirical claim is confounded: N-DPO is not DPO plus R, it is length-normalized DPO plus R, and the λ=0 benchmark data that would disentangle the two are absent. The Appendix's λ=0 AlpacaEval ablation exists but is not benchmark-matched, and its N-DPO numbers disagree with Table 1. For SimPO, where there is no length-normalization confound, N-SimPO is mixed across models. If the missing control shows L-DPO already delivers the AlpacaEval benefit and benchmark maintenance, the central contribution of the penalty is not established; if it shows L-DPO alone is much worse, the penalty is needed. Without that control, the headline 'improved trade-off' cannot be attributed to the proposed mechanism. This does not change the reader's CONDITIONAL verdict, but it sharpens the condition: run the missing matched benchmark ablation and reconcile the appendix tables.","tokens_in":20338,"tokens_out":15844,"duration_ms":159743,"concrete_test":"On Llama-3.1-8B-Instruct, run four arms with hyperparameters selected by the same AlpacaEval1 protocol: (a) DPO, (b) L-DPO (λ=0), (c) N-DPO with the configuration used in Table 1, (d) N-DPO with the configuration used in Table 4. Report AlpacaEval2 LC/WR and the Table 2 benchmark deltas for all arms. If (c) does not beat (b) on the benchmark delta while retaining an AlpacaEval2 gain, the headline improvement is attributable to length normalization/hyperparameters rather than the mass-conservation penalty. The same run also checks whether Table 4 reproduces Table 1; if not, one of the tables is not the method being claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical claim 'adding the regularizer improves trade-offs' is not cleanly tested for DPO. N-DPO is length-normalized DPO (L-DPO) plus R, but Table 1 compares N-DPO to original DPO, conflating the regularizer with length normalization (and with different β/lr grids in Appendix B). The only λ=0 ablation, Table 4, reports AlpacaEval alone and does not match Table 1: e.g., Llama N-DPO LC is 29.41 there vs 24.97 in Table 1; OLMo is 8.02 vs 9.88. Crucially, no corresponding Table-2 benchmark deltas are given for λ=0, so the headline '>9% benchmark gains' cannot be attributed to R. For SimPO the comparison is clean, and N-SimPO is mixed: AlpacaEval2 LC drops for Mistral (26.34→26.05) and Llama (26.74→25.54); OLMo gains LC (5.15→8.70) but loses benchmark average (−0.73→−4.81). The central trade-off claim therefore rests on an uncontrolled comparison plus appendix numbers that disagree with the main table.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that reward over-optimization—specifically likelihood displacement—in direct alignment algorithms (DAAs) stems from a lack of normalization of the implicit reward. To address this, it introduces a regularizer R (Eq. 12) that penalizes the squared log ratio of the length-normalized total probabilities of the chosen and rejected responses under the trained and reference models. The modified objectives N-DPO (Eq. 13) and N-SimPO (Eq. 14) are evaluated on Mistral-7B, Llama-3.1-8B, and OLMo-7B trained on UltraFeedback, measuring AlpacaEval win rates, common-sense/reasoning benchmarks, and implicit reward accuracy. The paper also presents a token-level analysis (Figure 3) and a gradient derivation (Section 4.2, Appendix C) claiming that likelihood displacement is concentrated on outlier tokens and that the regularizer corrects these outliers by acting through low-likelihood tokens.","tokens_in":20601,"tokens_out":6288,"duration_ms":56756,"significance":"If the central claims held, this would be a useful minimal modification: a single penalty term applicable to both reference-based and reference-free DAA objectives, with evidence of improved generation-quality/benchmark trade-offs. The evaluation spans three model families and multiple benchmarks, and the gradient derivation in Appendix C is symbolic and clearly presented. The paper also contributes an interesting observational study of token-level likelihood displacement. However, the current manuscript does not cleanly isolate the effect of the regularizer for DPO, several headline numbers in the abstract are not supported by the tables, and the gradient-based mechanism contains a mathematical claim that is not correct for softmax language models. These issues are load-bearing and need to be resolved before the contribution can be fully credited.","major_comments":[{"comment":"The abstract claims 'over a 20% relative increase in AlpacaEval2 scores' for Llama-3.1-8B-Instruct, but Table 1 shows LC 21.72→24.97 (+15.0%) and WR 24.18→27.57 (+14.0%). No reported number reaches 20%. Similarly, the abstract's 'improvements in reward modeling across datasets' is contradicted by Table 3: N-DPO lowers HelpSteer2 accuracy for Mistral (66.15→56.33) and HH-RLHF for Llama (59.55→58.73), while N-SimPO results are mixed. Please recalibrate these claims.","section":"Abstract; §1; Table 1; Table 3"},{"comment":"The isolated effect of the regularizer is not properly tested for DPO. N-DPO differs from DPO by both length normalization and R, so the λ=0 ablation (L-DPO) is the right control, but it is reported only in Appendix A and only for AlpacaEval. Moreover, the λ=0 and N-DPO numbers in Table 4 disagree with Table 1 (e.g., Llama N-DPO LC 29.41 vs 24.97; OLMo 8.02 vs 9.88; Mistral N-DPO WR 18.16 vs 20.08). No Table-2 benchmark deltas are given for λ=0, so the headline trade-off claim cannot be attributed to R. Please provide a full matched comparison of DPO, L-DPO (λ=0), and N-DPO on all metrics, and explain the discrepancies between Table 1 and Table 4.","section":"§5.2; Appendix A; Table 4; Table 1"},{"comment":"The claim that ∇θπθ(y(i))/πθ(y(i)) is 'inversely proportional to the likelihood of each token' is not correct for softmax parameterization. For final-layer logits z_i, ∇_{z_i} π_i / π_i = 1 − π_i, which is bounded and tends to 1 as π_i→0, not to infinity. The '1e−7× dominance' therefore does not follow from the model's parameterization. This is a load-bearing step for the mechanistic explanation in Section 4.2 and the abstract. Please provide a correct derivation or retract the mechanism.","section":"§4.2; Eq. (16)"},{"comment":"The paper motivates R as preserving the partition function Z(x), but the regularizer only preserves the mass of the two seen responses (yw, yl). In real preference data, these two responses are a tiny fraction of Z, so Figure 1's three-response example (where the pair covers 70% of the mass) does not transfer. Either provide a formal link—for example, a bound or assumption under which pair-mass conservation controls changes in Z—or reframe R as a heuristic whose benefits are empirical, not as a direct normalization of the partition function.","section":"§3; §1; Figure 1"},{"comment":"The claim of 'improved trade-offs between generation quality and general benchmark capability' is not consistently supported even for the clean comparison (SimPO vs N-SimPO). AlpacaEval2 LC decreases for Mistral (26.34→26.05) and Llama (26.74→25.54) in Table 1, and for OLMo the benchmark average degrades sharply (−0.73→−4.81) in Table 2. The introduction and discussion should be adjusted to reflect where N-SimPO actually improves, rather than stating a general improvement.","section":"§5; Table 1; Table 2; §7"}],"minor_comments":[{"comment":"SimPO is described as reference-free, but Eq. (8) defines the reward with a Z(x) and Eq. (9) defines Z(x) using π_ref. Please clarify what Z(x) denotes in the SimPO context and whether Eq. (9) is the standard SimPO formulation or a new interpretation.","section":"§2; Eq. (8)-(9)"},{"comment":"The axis labels 'Minimum Score Distribution' and 'Token Score Distribution' are ambiguous. The text refers to 'minimum reward per sample' and 'token-wise reward'; please unify the terminology and clarify what is being plotted, including the unit.","section":"Figure 3"},{"comment":"The hyperparameter tables are hard to parse. For example, Table 24 lists β values as '(0.03/1.0/3.0/3.0)' without a header defining the order, and the sweep ranges overlap confusingly. Please add explicit column/row headers for the four methods.","section":"Appendix B; Tables 20-24"},{"comment":"Several citations are incomplete or formatted inconsistently: 'Gupta et al., a' and 'Gupta et al., b' lack venues/years in the running text, and the bibliography entries for these references should be completed.","section":"References; Related Works"},{"comment":"Table 3 reports reward accuracy without standard errors or a description of how the pairs were split/aggregated. At least a footnote on the evaluation protocol would help interpret the differences.","section":"§5.1; Table 3"},{"comment":"Minor wording: 'mitigates the amount of displacement within preferred responses overall, and for the outlier tokens specifically, by utilizing low-likelihood tokens' is grammatically awkward and should be rephrased for clarity.","section":"Abstract; §7"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful paper with a simple and appealing idea, but the current manuscript is not ready in its present form. The most serious problems are (i) the failure to isolate the regularizer's effect for DPO, with inconsistent numbers between the main tables and the appendix, and (ii) a mathematically incorrect claim about softmax gradients that underpins the proposed mechanism. The abstract also overstates the empirical results. I believe these issues are addressable, so I recommend major revision rather than rejection. The authors should add the missing L-DPO control on all metrics, correct the gradient analysis, and carefully recalibrate the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2607.16240. The paper adds a quadratic penalty to DPO and SimPO that keeps the length-normalized sum of chosen and rejected probabilities close to the reference model's. That specific regularizer is new as far as I know, and the paper does a few things right. The gradient derivation is correct, the ablation with λ=0 exists (even if under-reported), and the token-level outlier analysis is a genuinely useful way to visualize likelihood displacement. If the regularizer reliably buys win-rate without wrecking benchmarks, it's a practical contribution.\n\nNow the soft spots. (1) The abstract and §5.1 are not backed by the tables: the claimed >20% Llama AlpacaEval2 gain is closer to 15% in Table 1, and the 'improvements in reward modeling across datasets' is contradicted by drops on HelpSteer2. (2) The central comparison doesn't isolate the regularizer: N-DPO is length-normalized DPO plus R, but the main table compares against vanilla DPO. The λ=0 ablation only reports AlpacaEval, not benchmark deltas, and its N-DPO numbers disagree with Table 1 (Llama LC 29.41 vs 24.97). That makes me wonder if the configurations differ. (3) The mechanistic explanation in §4.2 is quantitatively off: for a softmax, ∇θπ_t/π_t is bounded, so it is not inversely proportional to token likelihood, and the '1e-7 dominance' claim does not follow. The empirical concentration in outlier tokens may still hold, but the proposed mechanism as written is wrong.\n\nNone of these is fatal to the empirical method. The regularizer is simple and testable, and the results for SimPO, where the comparison is cleaner, are mixed but not discouraging. The paper needs matched baselines, corrected claims, and either a fixed gradient analysis or a softer interpretation. I'd send it to peer review — a good referee can force the cleanup, and the idea deserves airing. For my own work, I wouldn't rely on the numbers until the discrepancy is resolved.","headline":"A simple, plausible pair-level probability-conserving regularizer for DPO/SimPO; worth a referee, but the empirical claims are undercut by inconsistent numbers and the mechanistic story overreaches.","tokens_in":21210,"tokens_out":3818,"would_cite":false,"duration_ms":36117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward over-optimization in direct preference alignment stems from an unnormalized implicit reward, and a single pair-level probability-conservation penalty mitigates it for both reference-based (DPO) and reference-free (SimPO) methods.","keywords":["direct preference optimization","reward over-optimization","likelihood displacement","reward normalization","SimPO","implicit reward","outlier tokens","LLM alignment"],"falsifier":"Compute the actual total probability assigned to a held-out set of responses (or an estimate of the partition function) before and after N-DPO training; if the pair-sum is conserved but the held-out response probabilities still shift substantially, the proxy fails. A simpler check: in a small softmax language model, measure the gradient norm of a token's log-probability as p→0; the paper's '1e−7× dominance' claim predicts ~1/p growth, whereas a softmax parameterization gives a bounded gradient that vanishes as p→0.","tokens_in":20083,"feed_emoji":"⚖️","tokens_out":4123,"duration_ms":37861,"temperature":0.7,"pith_summary":"This paper argues that direct alignment algorithms such as DPO over-optimize because their implicit reward is not normalized: the partition function that should keep the model a proper probability distribution is dropped from the objective, so nothing stops the likelihoods of preferred and rejected responses from falling together. The proposed fix is a one-term penalty that conserves the total length-normalized probability mass of the two responses seen in each preference pair. Applied unchanged to DPO and to the reference-free SimPO, the penalty reduces likelihood displacement, keeps the model's probability assignments to unseen responses from drifting, and improves the trade-off between generation quality and general benchmark performance. The paper also reports that likelihood displacement is concentrated in a small set of outlier tokens, and that the penalty corrects those outliers by acting through low-likelihood tokens. If correct, this gives a single, method-agnostic explanation for reward over-optimization across reference-based and reference-free objectives.","feed_headline":"One penalty fixes DPO and SimPO over-optimization","feed_subtitle":"Adding pair-level probability conservation raises win-rates and protects benchmarks at once.","key_machinery":"The load-bearing object is the length-normalized pair-mass ratio penalty R(πθ,πref,x,yw,yl)=λ[log((π̄θ(yw)+π̄θ(yl))/(π̄ref(yw)+π̄ref(yl)))]². Here π̄ is the length-normalized probability of a response (the geometric mean of its token probabilities), and the penalty is minimized when the sum of the chosen and rejected response probabilities under the trained policy equals the same sum under the reference policy. The paper's argument is that this pair-sum is a usable stand-in for the intractable partition function Z(x): since the model's distribution is always normalized, conserving the mass of the two observed responses should protect the mass of all unseen responses. The penalty is added unc","core_discovery":"The central discovery is that likelihood displacement—the simultaneous drop in the likelihoods of both the preferred and non-preferred responses during DPO-style training—is a symptom of an unnormalized implicit reward. Because the DPO and SimPO objectives cancel the partition function Z(x), the model is free to shrink the total probability mass assigned to the observed responses, which spuriously inflates the probability of unseen responses. The paper proposes to fix this by adding the penalty R(πθ,πref,x,yw,yl)=λ[log((π̄θ(yw)+π̄θ(yl))/(π̄ref(yw)+π̄ref(yl)))]², which encourages the length-normalized probability mass of the chosen and rejected responses to stay equal to its reference value.","pith_inferences":["If the pair-sum proxy for the partition function holds, a similar probability-conservation penalty could be adapted to online or iterative alignment methods, though the paper explicitly leaves those settings out—this is my inference, not the paper's claim.","The outlier-token concentration suggests a potentially simpler intervention: directly clipping or upweighting low-likelihood tokens during preference optimization might reproduce much of the benefit without introducing an extra hyperparameter.","The paper's mechanistic claim that low-likelihood tokens dominate the gradient (roughly as 1/p) can be tested directly in a small softmax model; if the gradient norm actually behaves like sqrt(1-p), the mechanism described in Section 4.2 would need revision even if the empirical regularizer still works."],"forward_implications":["N-DPO and N-SimPO improve AlpacaEval2 win-rates relative to DPO/SimPO (e.g., over 20% relative increase in length-controlled win-rate for Llama-3.1-8B-Instruct) while reducing benchmark degradation on common-sense and reasoning tasks.","Implicit reward models trained with the penalty generalize better to unseen preference datasets such as HH-RLHF and HelpSteer2 in most configurations tested.","Because likelihood displacement is concentrated in a small set of outlier tokens, standard DPO can improve generation quality even while lowering chosen-response likelihood; the penalty removes this confounding behavior.","Since the same penalty applies to a reference-based and a reference-free method, it offers a unified account of over-optimization across DAA families rather than a method-specific patch."],"fun_headline_variants":["Stop DPO and SimPO over-optimization with one penalty","Length-normalized penalty prevents DPO likelihood collapse","Probability-conserving regularizer fixes DPO and SimPO","Keep preferred responses likely during DPO training","Penalty that preserves response likelihoods in DPO"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper assumes that conserving the total probability of the two responses seen for a prompt keeps the model's probabilities over all unseen responses stable—that the pair sum is a reliable proxy for the full, intractable partition function, a step it motivates only with a three-response toy example and does not bound.","fun_headline_variants_meta":{"raw":{"variants":["Stop DPO and SimPO over-optimization with one penalty","Length-normalized penalty prevents DPO likelihood collapse","Probability-conserving regularizer fixes DPO and SimPO","Keep preferred responses likely during DPO training","Penalty that preserves response likelihoods in DPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3078,"prompt_tokens":804,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2210}},"tokens_in":548,"tokens_out":2274,"duration_ms":17116,"temperature":1.0,"reasoning_tokens":2210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:55:28.384271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the actual total probability assigned to a held-out set of responses (or an estimate of the partition function) before and after N-DPO training; if the pair-sum is conserved but the held-out response probabilities still shift substantially, the proxy fails. A simpler check: in a small softmax language model, measure the gradient norm of a token's log-probability as p→0; the paper's '1e−7× dominance' claim predicts ~1/p growth, whereas a softmax parameterization gives a bounded gradient that vanishes as p→0.","supporting_citations":[],"review_version":1}