{"id":"8f6ffbb2-012f-49f0-8966-db46307ca380","arxiv_id":"2608.09568","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Se-DPO assigns evolving per-token credits from implicit reward magnitude and reference entropy during DPO training, improving instruction-following win rates but with an internally inconsistent derivation.","lead":"This paper introduces Se-DPO, a training method where a language model assigns each token a live credit during preference learning instead of treating all tokens equally. It reports large gains on AlpacaEval 2 and Arena-Hard, but the derivation behind the method contains an internal inconsistency and MT-Bench results degrade in several settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 8 inverts the coefficient derived in Eq. 7, so Proposition 1 optimizes a preference logit that does not correspond to the stated per-token KL objective.","rationale":"I read the full text in good faith. The paper is clearly written and the empirical setup is plausible, but no code or data are provided, and the central theoretical derivation contains a concrete algebra error. Eq. 7, derived from the closed-form solution of the per-token KL-regularized objective, implies that the token-level implicit reward r_t equals (β_t/β) rhat_t plus a position-dependent constant. Defining c_t = β/β_t makes this coefficient 1/c_t, so Eq. 8's use of c_t as the multiplier on rhat_t is inverted. This is not a subtle convention choice: substituting β_t=2β into a simple two-token example gives a factor of 2 where Eq. 8 gives 0.5. The reader's weakest assumption identifies exactly this point, and I agree. Because Proposition 1 is stated for the logit in Eq. 8, the proposed variance-minimization justification does not apply to the objective that the per-token KL derivation actually yields. The experimental results could still indicate that a weighted-DPO loss with learned token weights improves instruction following, and the MT-Bench declines and length increases are disclosed in the appendix, so I do not read the paper as fraudulent. But the paper's central claim is that token credit modulates each token's KL regularization, and the derivation that establishes this link is invalid. Without code or a corrected derivation, the correctness risk remains high, and the reader's REJECT verdict is unchanged.","tokens_in":14628,"tokens_out":8502,"duration_ms":79429,"concrete_test":"Re-derive Eq. 8 from Eq. 7 with a one-position example: vocabulary {A,B}, π_ref uniform, β=1, β_t=2, true reward r(A)=1, r(B)=0. From Eq. 6, π*(A)=e^{1/2}/(1+e^{1/2})≈0.622, so rhat_A = log(2·0.622)≈0.219. Eq. 7 gives r_A = 2·0.219 + 2 log Z_t, so the correct coefficient on rhat_A is 2 = 1/c_t; Eq. 8 with c_t=1/2 gives 0.109. Then recompute Proposition 1 with weights 1/c_t: the variance-minimizing credit becomes c_t ∝ σ²_t/|r*_t|, the opposite of the paper's stated direction. This single check settles whether Eq. 8 follows from Eq. 7.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2.2, Eq. 7 gives r_t = β_t log(πθ/π_ref) + β_t log Z_t = (β_t/β)·rhat_t + const = (1/c_t)·rhat_t + const, because c_t = β/β_t. Thus the preference logit formed from the token-specific KL objective should weight rhat_t by 1/c_t, not by c_t. Eq. 8 uses c_t, inverting the relation. The footnote acknowledges that the Z_t residual does not cancel, but it does not address this coefficient inversion. Consequently, Proposition 1 minimizes the variance of Δ = ∑ c_t rhat_t, which is not the logit implied by Eqs. 4–7; the conclusion c_t ∝ |r*_t|/σ²_t is therefore not a property of the per-token KL problem as claimed. The implemented loss, a weighted DPO with MLP-predicted positive weights normalized to unit mean, could still work as a heuristic, but the paper's central theoretical claim that Se-DPO implements optimal per-token KL budgets is unsupported by the derivation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Se-DPO, a token-level credit mechanism for Direct Preference Optimization. It extends DPO's uniform token weighting by introducing per-token KL coefficients β_t, defines token credit c_t = β/β_t, and derives, via a variance-minimization argument, that optimal credit should be proportional to |r*_t|/σ²_t. Since both quantities evolve during training, Se-DPO computes credit online from the model's own implicit reward magnitude |r̂_t| and reference entropy H_ref,t, fed through a lightweight MLP calibration network, with per-response mean normalization. Experiments on Llama-3-8B-Instruct, Llama-3.2-3B-Instruct, and Gemma-2-2B-it on AlpacaEval 2, Arena-Hard, and MT-Bench report gains over DPO, SimPO, TGDPO, and a static-credit baseline.","tokens_in":14917,"tokens_out":5844,"duration_ms":39658,"significance":"If the central claim held, per-token, temporally updated KL budgets would be a meaningful and low-cost improvement to DPO-style alignment: the method requires no external models, adds only ~7% wall-clock overhead, and shows large win-rate gains on two instruction-following benchmarks. The paper also contains a clean empirical observation—that DPO's token-level implicit reward magnitudes shift substantially during training—which is potentially useful regardless of the theoretical derivation. The main strengths are the low overhead, the online self-evolving design, and the breadth of ablations (warmup sensitivity, single-signal variants, static credit). However, the theoretical derivation does not match the implemented loss, and the empirical results are partly undermined by inconsistent MT-Bench degradation and length confounds.","major_comments":[{"comment":"Equation (8) inverts the coefficient that follows from the token-specific KL objective in Eq. (7). From Eq. (7), the implicit reward is r_t = β_t log(π_θ/π_ref) + β_t log Z_t = (β_t/β) r̂_t + const = r̂_t / c_t + const, since c_t = β/β_t. The consistent preference logit formed from the per-token KL objective should thus weight r̂_t by 1/c_t (plus a context-dependent offset), not by c_t as written in Eq. (8). The footnote granting that Z_t does not cancel addresses the offset but not this coefficient inversion. Consequently, the variance-minimization Proposition 1 optimizes Var[Σ c_t r̂_t] for a logit that is not the one implied by Eqs. (4)–(7), so the conclusion c_t ∝ |r*_t|/σ²_t is not a property of the per-token KL problem as claimed. The implemented loss in Eq. (12) may still work as a heuristic, but the paper's central theoretical claim that Se-DPO implements optimal per-token KL budgets is unsupported by the derivation.","section":"Section 2.2, Eqs. (7)-(8)"},{"comment":"Proposition 1 solves a variance-minimization problem whose constraint, Σ c_t |r*_t| = C, is chosen ad hoc and is itself the object being justified. The paper never derives that signal preservation should be measured by this weighted sum rather than, say, Σ c_t r*_t (which would lead to different signs) or Σ |c_t r*_t|. The proportionality c_t ∝ |r*_t|/σ²_t is therefore a consequence of the chosen constraint, not an independent result about token credit. Moreover, the proof sketch in Appendix A states 'Since c_t = β/β_t > 0 by construction, we restrict to non-negative credits,' but the practice of unit-mean normalization plus an unconstrained positive MLP output means the credits are positive but not freely chosen; the mapping from the optimized c_t to actual KL budgets is only implicit. The theoretical motivation section should either justify the constraint from the preference-optimization objective or be reframed as a heuristic regularization analysis.","section":"Section 3.2, Prop. 1"},{"comment":"The empirical validation of the two proxy relationships—|r̂_t| ≈ |r*_t| and σ²_t ≈ H_ref,t—is circular with respect to the training setup. Figure 2(b–c) measures σ̂²_t as the variance of r̂_t across the latter half of training checkpoints of the same DPO run whose target is being explained, so the observed ρ = 0.86 between H_ref,t and σ̂²_t is a property of that specific trajectory, not a general relationship that licenses the use of reference entropy as a stable noise proxy. The claim that 'the correlation is strongest at low-reward positions and diminishes at high-reward positions' is reported qualitatively without numbers or a figure, and it is exactly the regime dependence used to justify the MLP. A quantitative breakdown (e.g., correlation within deciles) should be provided.","section":"Section 3.2, Fig. 2"},{"comment":"The headline improvements are inconsistent with the reported MT-Bench results. On Llama-3-8B-Instruct, Se-DPO gains large win rates on AlpacaEval 2 and Arena-Hard but its MT-Bench score drops to 6.9–7.4 versus DPO's 8.0–8.2, and its MT-Bench win rate is 40.0–41.9 versus DPO's 37.5–46.3. This is more than a minor weakness: it indicates that the pairwise instruction-following gains do not translate to general conversational quality, and the paper's explanation in Section 4.3 (score vs. win-rate aggregation) does not account for the score drop on three out of four settings. The authors should report standard errors or significance across evaluation seeds, and should either explain the MT-Bench degradation with a category-level analysis for all settings or temper the claim of consistent improvement. Appendix Table 6 shows a category breakdown only for one setting (PairRM), where Reasoning, Coding, and Math collapse to 3.9–4.6; that pattern is consistent with length/format shifts rather than genuine alignment gains.","section":"Table 1"},{"comment":"The comparison for smaller models is not controlled for training regime: DPO, SimPO, and TGDPO rows use full fine-tuning settings while Se-DPO uses LoRA for Llama-3.2-3B-Instruct and Gemma-2-2B-it. This makes the reported gains hard to interpret, since the differences could be partly due to the parameter-efficient training regime rather than the credit mechanism. The paper should either train all methods under the same LoRA setting or clearly state that the comparison is across different training regimes and explain why this is fair.","section":"Section 4.2, Table 1"}],"minor_comments":[{"comment":"The footnote correctly concedes that the partition-function terms Z_t generally do not cancel between chosen and rejected responses, but it does not quantify the residual. Given that the proposed loss in Eq. (12) drops these terms entirely, a short analysis of the magnitude of the residual (e.g., on a sample of tokens) would help the reader assess the approximation.","section":"Section 2.2"},{"comment":"Algorithm 1 says warmup 'uses standard DPO with uniform credit (c_t = 1)', but the loss in line 15 is the weighted form; the warmup branch sets c_t = 1, so the loss reduces to standard DPO, which is fine. However, the calibration network is updated jointly with the policy during the guided phase; the paper does not specify whether ϕ is trained on a separate objective or purely through the main loss. Please clarify whether f_phi's only training signal is ∇_ϕ L in line 16.","section":"Algorithm 1"},{"comment":"Figure 1(a,b) shows rank correlation and top-20% overlap for |r̂_t| 'in the original training run', but the experimental details in Appendix C say the analysis is on Qwen2.5-1.5B-Instruct with 2,000 samples, while the main experiments are on Llama/Gemma models. The transferability of the dynamics observation to the actual evaluated models is asserted, not demonstrated. A sentence noting that the dynamics were also verified on the main models (or that the analysis is representative) would be useful.","section":"Section 3.1"},{"comment":"The paper reports different learning rates and β values for Llama-3-8B (lr 5e-7, β=0.01) versus smaller models (lr 5e-6, β=0.1). This is appropriate, but the same hyperparameters are used for all baselines? The text says 'DPO, SimPO, and TGDPO follow the hyperparameters reported in Zhu et al. (2025)', which may not be the same as those in the current setup. Please state explicitly which learning rate and β are used for each baseline in each setting.","section":"Appendix C"},{"comment":"Minor typographical and formatting issues: Table 1 has a duplicated 'WR (%)' header under ArmoRM columns; 'Se-DPO(LoRA)' appears in the main results but Table 1's caption does not indicate which rows use LoRA; the abstract says 'up to 9.8 points on AlpacaEval 2 and 12.2 points on Arena-Hard' while the conclusion repeats the same numbers, but the introduction gives 'up to 50.6%' and the abstract says 'up to 50.6%'—the point differences should be consistent with the underlying win rates.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The claimed coefficient inversion in Eq. (8) is a real and load-bearing issue: if the derivation is wrong, the paper's theoretical motivation collapses to a heuristic. However, the empirical results, especially the large AlpacaEval 2 / Arena-Hard gains and the low overhead, are interesting enough that a major revision could fix the presentation. The strongest fix would be to reframe the paper as an empirical study of online token-level credit with a heuristic justification, rather than claiming a derivation from the KL-regularized objective. Given the reader's report and the stress-test analysis, I lean toward major_revision rather than reject because the central empirical comparison is not irreparably flawed, but the abstract and conclusion overstate the theoretical grounding. The MT-Bench degradation should be addressed head-on; currently it looks like the method may trade conversational quality for benchmark-specific win rates. I would also ask for a reproducibility detail: the paper says 'best results in each setting are bolded' but some cells seem bolded inconsistently (e.g., MT-Bench scores where Se-DPO is lowest are not bolded; DPO's are bolded in several columns, which is correct since they are highest. But the MT-Bench 'WR (%)' column for Llama-3-8B PairRM shows TGDPO 41.9 bolded and Se-DPO 41.9 bolded, which is fine, but the MT-Bench score column shows Se-DPO 7.4 not bolded while DPO 8.0 is bolded). Overall the paper is borderline; major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The novel part is real: Se-DPO computes token credit online from the model's own evolving implicit reward plus reference entropy, and the paper shows that the top-|r| token set shifts during training. That dynamic, self-calibrating design is not in the cited token-level DPO literature, which mostly uses static importance. The empirical package is also decent—three models, two annotators, ablations, warmup sensitivity, length-controlled numbers, and a wall-clock overhead measurement (6.85%). The main results on AlpacaEval 2 and Arena-Hard are consistent and the paper is clearly written. Credit where due: the observation that credit must evolve, not just be recomputed once, is the right kind of empirical claim to make, and Figure 1 supports it.\n\nBut the theoretical foundation has a load-bearing flaw. Equation 7 gives r_t = beta_t (log pi - log pi_ref) + beta_t log Z_t = (1/c_t) rhat_t + const under their definition c_t = beta/beta_t. The preference logit formed from that token-specific KL objective should weight rhat_t by 1/c_t, not by c_t. Equation 8 uses c_t, inverting the relation. Proposition 1 then minimizes the variance of a logit that does not match the objective derived two lines earlier. The footnote acknowledges Z_t does not cancel, but it does not address the coefficient inversion. So the message that Se-DPO implements optimal per-token KL budgets is not supported by the math. The implemented loss—weighted DPO with MLP-predicted positive weights, normalized to unit mean—could still work as a heuristic, and the ablations suggest it does, but the theoretical motivation collapses.\n\nThe other soft spots are minor by comparison: small-model baselines use LoRA while the 8B full-finetune comparison is not apples-to-apples; MT-Bench declines are disclosed but buried in the appendix; no code or data are released. Those are fixable. The coefficient error is not.\n\nWho is this for? Someone working on token-level preference optimization will want to know about the empirical finding that static credit goes stale, and about the calibration network as a practical alternative. They should not cite the variance-minimization derivation as the reason. The paper deserves a serious referee because the empirical claims are substantial and the flaw is identifiable and fixable—a revised version that repairs the derivation, or honestly reframes the method as a heuristic with an empirical motivation, could be a useful contribution.\n\nRecommendation: send it to review, but the referee should be instructed to check Eq. 8 versus Eq. 7 carefully before trusting anything in Sections 3.2–3.4.","headline":"A plausibly effective online token-weighting heuristic whose stated theory is undercut by a coefficient inversion, making the paper's central derivation unsupported even though the empirical story is coherent.","tokens_in":15421,"tokens_out":695,"would_cite":false,"duration_ms":8406,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Se-DPO replaces DPO's uniform token weighting with per-token KL credits that evolve during training, yielding win-rate gains of up to 9.8 points on AlpacaEval 2 and 12.2 on Arena-Hard.","keywords":["direct preference optimization","token-level credit","KL regularization","implicit reward","self-evolving training","preference optimization","LLM alignment"],"falsifier":"Substitute Eq. 7 into the preference logit and verify whether the coefficient multiplying $\\hat r_t$ is $c_t$ or $1/c_t$; this algebraic check determines whether Proposition 1's credit formula matches the loss that Se-DPO trains.","tokens_in":14386,"feed_emoji":"🎯","tokens_out":8110,"duration_ms":64688,"temperature":0.7,"pith_summary":"Direct Preference Optimization (DPO) treats every token in a response as equally responsible for the preference outcome, which the authors argue is wasteful because tokens differ sharply in how much they carry the signal. Se-DPO proposes a token-level credit that rescales each token's KL regularization, derives that the credit should track the magnitude of the model's own implicit reward, and shows that this signal changes substantially during training. Rather than freezing a one-time importance estimate, Se-DPO updates the credit online through a lightweight calibration network that combines implicit-reward magnitude with reference-model entropy. If the central claim holds, it means per-token, temporally adaptive KL budgets improve instruction-following quality without any external reward or teacher model.","feed_headline":"Adaptive token credit lifts DPO win rates by up to 9.8 points","feed_subtitle":"A lightweight network derives per-token credit from the model's own implicit reward, with no external models, improving AlpacaEval 2 and…","key_machinery":"The central object is token credit, $c_t = \\beta / \\beta_t$, the multiplier that rescales each token's contribution to the preference logit and thereby sets the token's KL budget relative to the reference policy. The paper shows that variance-minimizing credits satisfy $c_t \\propto |r_t^*| / \\sigma_t^2$, proxies $|r_t^*|$ by the implicit reward magnitude $|\\hat r_t|$ and $\\sigma_t^2$ by reference entropy $H_{\\mathrm{ref},t}$, and implements the mapping with a two-layer MLP $f_\\phi$ whose outputs are mean-normalized within each response.","core_discovery":"The paper claims that DPO's uniform token weighting is suboptimal and that the right per-token weight, which it calls token credit, should modulate each token's KL regularization based on its contribution to the preference outcome. It derives that under a variance-minimization criterion, optimal credit is proportional to the magnitude of the token's implicit reward and inversely related to its noise, and it observes empirically that this credit evolves throughout training, so any static credit becomes stale. Se-DPO operationalizes this by extracting the implicit reward magnitude and reference entropy from the standard DPO forward pass and mapping them through a lightweight MLP to credit values, normalized to unit mean per response. Across three base models and two preference annotators, Se-DPO reports improvements over DPO of up to 9.8 points on AlpacaEval 2 and 12.2 points on Arena-Hard.","pith_inferences":["The paper's Eq. 8 assumes the credit $c_t$ multiplies the implicit reward $\\hat r_t$; checking Eq. 7 suggests the consistent multiplier might be $1/c_t$, which would invert the derived credit. If that inversion holds, the gains may come from a different mechanism than the stated variance-minimization argument.","A natural testable extension is whether the same online credit scheme transfers to reference-free preference losses (e.g., SimPO-style logits), where there is no implicit reward in the DPO sense but a length-normalized log-ratio could play a similar role.","The paper does not address potential negative side effects of reweighting on generation length or on categories like reasoning and math, where it reports lower MT-Bench scores; a follow-up could examine whether adaptive token credit systematically trades off open-ended quality against reasoning quality."],"forward_implications":["If Se-DPO is correct, DPO's uniform KL coefficient is a bottleneck, and token-level KL budgets that adapt during training can yield material alignment gains without external models.","Static token-importance methods, which compute importance once before or outside training, become progressively stale, so online updating is a necessary component, not an optional refinement.","The calibration network's use of reference entropy as a noise proxy suggests that token-level reliability, not just reward magnitude, is a useful signal for preference optimization.","The reported overhead is small (+6.85% wall-clock in one setting), so the approach is practical for standard DPO training pipelines."],"supporting_citations":[{"why":"Supplies the DPO objective that Se-DPO modifies by introducing per-token credit.","marker":"Rafailov et al., 2023"},{"why":"Provides the token-level implicit reward that Se-DPO uses as the primary credit signal.","marker":"Rafailov et al., 2024"},{"why":"Documented noise in implicit reward at non-critical tokens, motivating the entropy-based calibration.","marker":"Chen et al., 2025b"},{"why":"Prior token-level KL decomposition that Se-DPO extends into a learned online credit.","marker":"Zeng et al., 2024"},{"why":"TGDPO, the token-level baseline against which Se-DPO reports consistent win-rate gains.","marker":"Zhu et al., 2025"},{"why":"AlpacaEval 2, the benchmark on which the main claim of improvement is measured.","marker":"Li et al., 2023"}],"fun_headline_variants":["Se-DPO makes token credit evolve with training, boosting DPO by 12.2 points","Self-evolving token weights beat static DPO, up 12.2 on Arena-Hard","Token credit that adapts as you train: Se-DPO lifts AlpacaEval by 9.8","Lightweight self-evolving credit improves DPO by up to 12.2 points","No extra models: Se-DPO's live token credit boosts DPO win rates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the assumption that each token's credit weight directly multiplies that token's implicit reward in the preference logit; if the algebra instead places the weight in the denominator, the variance-minimization argument no longer applies to the loss actually trained.","fun_headline_variants_meta":{"raw":{"variants":["Se-DPO makes token credit evolve with training, boosting DPO by 12.2 points","Self-evolving token weights beat static DPO, up 12.2 on Arena-Hard","Token credit that adapts as you train: Se-DPO lifts AlpacaEval by 9.8","Lightweight self-evolving credit improves DPO by up to 12.2 points","No extra models: Se-DPO's live token credit boosts DPO win rates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000748,"raw_usage":{"total_tokens":3329,"prompt_tokens":938,"completion_tokens":2391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":2273}},"tokens_in":554,"tokens_out":2391,"duration_ms":13803,"temperature":1.0,"reasoning_tokens":2273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:53:50.514544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Substitute Eq. 7 into the preference logit and verify whether the coefficient multiplying $\\hat r_t$ is $c_t$ or $1/c_t$; this algebraic check determines whether Proposition 1's credit formula matches the loss that Se-DPO trains.","supporting_citations":[],"review_version":1}