{"id":"1893660f-758d-4170-9350-f509f32177ee","arxiv_id":"2602.09533","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ADPO rewrites the DPO loss as a sum of per-token log-sigmoid terms and frames this as explicitly autoregressive preference optimization, but its claimed preservation of the KL-optimal policy is not actually established.","lead":"DPO aligns language models with human preferences, but its standard loss judges whole responses at once. This paper proposes ADPO, a variant that scores each token or chunk separately, and tests it on math and conversation benchmarks — with a theoretical derivation that has a serious gap.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADPO's per-token locally normalized product is not the global Boltzmann optimum of the stated KL-constrained problem; Appendix B proves only an energy-sum identity, not distributional equality, so the central theoretical claim is unsupported.","rationale":"The paper's central claim is that ADPO preserves the KL-constrained optimal solution of DPO while shifting the summation outside the log-sigmoid. This requires that p2, after reparameterization as πθ, equals the global Boltzmann optimum of the reward r. The derivation instead defines p2 through per-prefix softmax normalizers Z_i(y_<i). Appendix B proves the energy-sum identity Σ_i E2* = E2(r), but that identity alone does not imply equality of the locally normalized product with the globally normalized Boltzmann distribution. The two distributions differ by the path-dependent factor Z/∏_i Z_i, and a concrete binary-length-2 example makes the discrepancy explicit. Appendix A's use of the shifted representative r_o ensures local normalization but changes the optimized reward, so it does not recover the original objective. This is not a cosmetic gap: the optimum of L_ADPO is a different policy unless the factorization condition holds, and the paper provides no argument for that condition. The empirical results are extensive, appear carefully run, and consistently favor ADPO over DPO/cDPO, but they do not establish the theoretical claim; they may reflect a different, though practically effective, objective. The reader's REJECT verdict is consistent with this analysis, and I find no reason to adjust it.","tokens_in":19699,"tokens_out":17836,"duration_ms":168329,"concrete_test":"Construct a toy space: binary vocabulary V={a,b}, sequences of length 2, π_ref uniform over tokens, β=1, and choose a nonconstant r*, e.g., r*(a)=1, r*(b)=0, r*(a,a)=0, r*(a,b)=10, r*(b,a)=r*(b,b)=0. Compute p2 from Eq. (12) for the four sequences and the global Boltzmann distribution p_B(y) ∝ exp(−(E2*(y1)+E2*(y2))). If p2(a,b) ≠ p_B(a,b), the optimality claim fails. Then replace r* by r_o from Eq. (35) and verify that p2 becomes the global Boltzmann solution for r−Σ_i β log Z_i rather than for r, quantifying the shift. This can be done analytically or in a few lines of Python.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is the identification of the locally normalized autoregressive product p2(y|x)=∏_i exp(−E2*)/Z_i(y_<i) with the global Boltzmann solution of the KL-constrained reward-maximization problem for the response-level reward r. In Definition 2 (Eqs. 8–12) and Appendix B, E2* is built from r*, so Σ_i E2* = E2(r). But the global Boltzmann distribution is exp(−E2(r))/Z with Z=Σ_y exp(−E2(r)), whereas p2(y)=exp(−Σ_i E2*)/∏_i Z_i(y_<i). These coincide only if Z=∏_i Z_i(y_<i), a condition that is generally false and is never established. Appendix A does not repair this: its proof switches to the representative r_o = r* − β log Z_i, which makes each local partition function equal to 1 and makes p2 the global Boltzmann distribution for the shifted reward Σ_i r_o = r − Σ_i β log Z_i, not for r. Consequently, the fixed point of L_ADPO is not shown to be the DPO/KL-constrained optimal policy for r. The paper's statement that \"the optimal solution is preserved\" (Section 4, Appendix B) is therefore unsupported, and the theoretical foundation of ADPO as a DPO generalization fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Autoregressive Direct Preference Optimization (ADPO), a DPO variant in which the Bradley–Terry comparison is applied prefix-wise rather than response-wise. The authors define prefix-level reward and energy functions, derive the ADPO loss L_ADPO = -E Σ_i log σ(β log πθ(yw_i|·)/πref(yw_i|·) - β log πθ(yl_i|·)/πref(yl_i|·)), and claim that this loss preserves the KL-constrained optimal solution of the original response-level reward (Section 4, Appendix B). They further claim that any reward consistent with prefix-wise BT models can be reparameterized by an autoregressive model (Theorem 1), that DPO is the special case with feedback length μ'=1 (Corollary 1), and that token length μ and feedback length μ' are independent design choices. Experiments on GSM8K, MATH500, AlpacaEval 2, Arena-Hard, and MT-Bench compare DPO, cDPO, SimPO, and ADPO variants.","tokens_in":20199,"tokens_out":5020,"duration_ms":45434,"significance":"If the theoretical claims were valid, the paper would contribute a clean loss variant and a useful conceptual distinction between token-level and feedback-level granularity. The empirical results are consistently positive across models and benchmarks, and the appendix includes derivations, standard deviations, and ablations. However, the central theoretical claim is not supported: the locally normalized product p2 in Eqs. (11)–(12) is not shown to equal the global Boltzmann optimum for the response-level reward, so ADPO's equivalence to DPO's KL-constrained optimum is unproven. The reparameterization theorem is true by construction but does not establish the missing link. The contribution therefore reduces to an empirically motivated loss modification, without the advertised theoretical foundation.","major_comments":[{"comment":"The central claim that ADPO \"preserves\" the KL-constrained optimal solution is not established. The prefix-wise p2 in Eq. (12) is a locally normalized autoregressive product: p2(y|x)=∏_i exp(-E*_2(x,y_{≤i}))/Z_i(y_{<i}) with Z_i(y_{<i})=Σ_{y_i} exp(-E*_2(x,y_{≤i})). The global Boltzmann distribution for the response-level reward r is exp(-E_2(x,y))/Z with Z=Σ_y exp(-E_2(x,y)). Corollary 2 proves only Σ_i E*_2 = E_2, which is an energy-sum identity; it does not imply Z=∏_i Z_i(y_{<i}) for all y. In general these partition functions differ, so p2 ≠ exp(-E_2)/Z. Appendix A's use of r_o with f(x,y_{<i})=β log Z_i makes each local partition function one, but then Σ_i r_o = r - β Σ_i log Z_i, a different reward. The optimum of L_ADPO is therefore not shown to coincide with the DPO/KL optimum for r.","section":"§4, Eqs. (11)–(12); Appendix B, Corollary 2"},{"comment":"The loss derivation is algebraically consistent, but it only shows that L_ADPO is the negative log-likelihood of a product of pairwise comparisons under the chosen p1. Since the relation p2(y_i|y_<i,x)=exp(-E*_2)/Z_i is assumed rather than derived from the KL objective, the reparameterization in Eq. (48) does not identify the reward that πθ is implicitly optimizing. To establish ADPO's theoretical foundation, the paper would need to state the implicit reward r_eff such that πθ(y|x) ∝ πref(y|x) exp(r_eff(x,y)/β) and show that the fixed point of Eq. (14) equals the optimum of J(r_eff). This is missing.","section":"Appendix A, Eqs. (32)–(49)"},{"comment":"Theorem 1 is a softmax reparameterization that holds by construction: for any prefix-wise r*, Proposition 1 defines π(y_i|y_<i,x) ∝ πref(y_i|y_<i,x) exp(r*(x,y_{≤i})/β), and Eq. (19) sums to the desired log-ratio. Such a representation always exists for any reward r via an additive decomposition (Lemma 1), so it cannot by itself establish that ADPO is a faithful extension of DPO. The sentence \"all reward classes consistent with the prefix-wise BT models\" (Theorem 1) is also circular: the equivalence class [r*] is defined so that reward shifts are absorbed, and the BT likelihood is invariant to those shifts by construction. Corollary 1 amounts to declaring μ'=1 for DPO rather than deriving it from the model.","section":"§5.1, Theorem 1 and Proposition 1"},{"comment":"The argument that DPO has an implicit feedback length μ'(y)=1 is not compelling. Eq. (20) defines μ'(y)=length(ν(y))=dim(R)=1 by choosing the evaluation metric to map to R; this is a definitional choice, not a consequence of the BT model or of ADPO. The introduction of two \"length measures\" is therefore partially invented, and the paper's claim to be the first to distinguish them is a framing contribution rather than a theoretical discovery. This does not by itself invalidate the empirical loss, but it should be presented as a design choice.","section":"§5.3, Eq. (20)"}],"minor_comments":[{"comment":"Right-padding is mentioned but its effect on the log-ratio sums is not analyzed; padding tokens with zero log-ratio and variable lengths can bias the loss.","section":"Eq. (6)"},{"comment":"T' is defined as μ'(y) in Definition 2, but Appendix A sets T'=max{μ'(y_w), μ'(y_l)}; make the notation consistent.","section":"Definition 2 vs Appendix A"},{"comment":"The claim that Theorem 1 goes \"in contrast to DPO theory\" is overstated, because DPO already presents the same reparameterization r=β log(π/πref). The novelty should be positioned more carefully.","section":"§5.1"},{"comment":"Granularity-family comparisons lack error bars or significance tests; many differences are within one point, so conclusions about granularity trends would be strengthened by variance reporting in the main text.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands on the core of the paper. The theoretical foundation, which is the main advertised contribution, is not established: the local normalization in Eq. (12) is conflated with the global Boltzmann solution of Eq. (50). This is not a local fix; it would require either a new proof that the ADPO fixed point equals the DPO optimum for the original reward, or a substantial reframing of ADPO as an approximate/alternative loss. The empirical results are plausible but do not compensate for the missing theoretical link. A resubmission that drops the 'preserves the KL-constrained optimum' claim and repositions ADPO as an empirically motivated loss variant might be reconsidered, though the novelty would be much weaker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the paper's central claim does not survive contact with its own equations. In Appendix B they show that the sum of the prefix energies equals the response-level energy, and then conclude that the product of per-token softmaxes is the optimal solution of the KL-constrained reward maximization. That conclusion is not valid. The per-token softmax has history-dependent partition functions Z_i(y_<i), so the product distribution is not the global Boltzmann distribution for the summed energy unless Z = prod_i Z_i, which is generally false. Appendix A's shift to r_o only works for the shifted reward, not the original one. So the statement that \"the optimal solution is preserved\" is unsupported. This is load-bearing, not a cosmetic gap: the paper is framed as a principled generalization of DPO, and that framing depends on the preserved-optimum claim.\n\nWhat is genuinely new and useful: the framing of preference optimization over the prefix closure, and the explicit separation of token length from feedback length. The resulting loss is, as the reader notes, essentially the token-level DPO objective already present in TDPO and related work, and the derivation is a reparameterization rather than a new objective. But the granularity families via strong compositions are a practical extension, and the experiments are carefully done: multiple models, multiple benchmarks, standard deviations, thorough hyperparameter exploration. The empirical gains over DPO are modest but consistent, and the analysis of training dynamics is a nice addition.\n\nSoft spots, in proportion: the Theorem 1 reparameterization claim is basically a softmax identity and does not add theoretical depth. Corollary 1 re-labels DPO as the mu'=1 case rather than proving anything about DPO. The two-length-measures discussion is partly definitional—the paper introduces the measures and then discovers them in DPO. None of this would be fatal by itself, but alongside the Appendix B gap it means the paper's theoretical foundation is not established.\n\nThe paper is worth reading for the experiments and the granularity framework, but not as a source for DPO theory. It deserves a serious referee—the empirical contribution is real and the loss may be useful even if the stated theory fails—though the review would likely recommend major revision or rejection unless the optimality claim is either fixed or dropped. For my own work, I would not cite it as a theoretical result, but I might cite the granularity experiments.\n\nRecommendation: send to peer review, but with the expectation that the central theory needs to be either corrected or substantially softened.","headline":"The empirical work is solid and the granularity idea is worth a look, but the central theoretical claim conflates a locally normalized per-token product with the global Boltzmann optimum, so the paper's main justification does not hold.","tokens_in":20560,"tokens_out":1797,"would_cite":false,"duration_ms":21595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that DPO can be reformulated over response prefixes without changing its theoretical optimum, yielding ADPO, a loss that applies a per-prefix Bradley-Terry comparison at every token instead of one comparison over the whole","keywords":["direct preference optimization","Bradley-Terry model","autoregressive language models","prefix-wise reward","token-level alignment","length measures","LLM alignment","mathematical reasoning"],"falsifier":"Take a reward r on two-token sequences over a binary vocabulary and compute both the global Boltzmann distribution exp(-E2(y))/Σ exp(-E2(y')) and the product of per-position softmaxes Π_i exp(-E*_2(y_≤i))/Σ_{y_i} exp(-E*_2(y_≤i)). If the two distributions disagree on any sequence, the claim that ADPO preserves the optimal solution of KL-constrained reward maximization is false for that reward.","tokens_in":19618,"feed_emoji":"🧮","tokens_out":5313,"duration_ms":49349,"temperature":0.7,"pith_summary":"The paper argues that direct preference optimization (DPO) locks in a response-level Bradley-Terry model, even though the networks being trained are autoregressive; that mismatch is a choice, not a necessity. It introduces Autoregressive DPO (ADPO), built from reward-based energies defined on all response prefixes, which yields a loss where the summation runs outside the log-sigmoid: each position gets its own preference signal. The paper claims this preserves the KL-constrained reward-maximization solution of DPO and proves a reparameterization completeness theorem: any reward class consistent with prefix-wise Bradley-Terry models can be written as the log-ratio of an autoregressive model to the reference. It also separates two length measures—token length and feedback length—and shows DPO is the special case where feedback length is one. A sympathetic reader would care because this gives DPO-style training a principled granularity dial that, in the paper's experiments across four large language models, consistently improves alignment.","feed_headline":"ADPO shifts DPO's loss to per-token comparisons","feed_subtitle":"Reward signals now land on every response prefix while the KL-constrained optimum is claimed to be intact.","key_machinery":"The machinery is the prefix closure Y* together with two prefix-wise energies, E*_1 = -r*(x, y_≤i) and E*_2 = -(1/β)r*(x, y_≤i) - log πref(y_i | y_<i, x). These define a prefix-wise Bradley-Terry preference model and a locally normalized autoregressive posterior p2(y_i | y_<i, x), which is what lets the loss factor into a sum of per-position log-sigmoid terms. The two length measures—token length μ and feedback length μ'—plus a strong composition ξ that groups tokens into feedback segments, supply the granularity family parameterizing the loss.","core_discovery":"The central discovery is a reformulation of DPO that makes the autoregressive assumption before applying the Bradley-Terry model, not after. Define the prefix closure Y* of the output space and a prefix-wise reward r*(x, y_≤i); with the reference model treated as autoregressive, the posterior distribution factorizes position-by-position, so the implicit reward at each token is the per-token log-ratio β log(πθ/πref). The resulting ADPO loss, -E Σ_i log σ(β log ...), moves the summation outside the log-sigmoid, in contrast to DPO's log σ of a sum. The paper proves that every reward class consistent with prefix-wise Bradley-Terry models is reparameterizable by an autoregressive model, and that","pith_inferences":["Editorial inference: if the per-token normalizers in Eq. (12) are not equal to the global partition function, ADPO is a new token-level objective whose optimum may not coincide with DPO's—a distinction that would matter for diagnosing whether ADPO's improvements come from finer credit assignment rather than from the same KL-constrained optimum.","Editorial inference: the μ vs μ' distinction suggests a practical design principle for preference data collection: feedback length can be chosen to match the granularity at which human evaluators actually judge quality (whole response, step, or token), rather than being dictated by the model's tokenizer.","Editorial inference: if the prefix-wise credit-assignment pattern reported—late-prefix first, then early, then final answer tokens—holds generally, ADPO-style losses could serve as a probe for where reasoning errors localize in chain-of-thought training."],"forward_implications":["DPO is recovered as the special case μ'(y)=1: the original objective has an implicit feedback-length measure of one for every response.","Setting μ'=μ gives a fully token-level objective; intermediate segmentations give a whole family of DPO variants at arbitrary granularity.","If the reparameterization theorem is right, the implicit reward of any prefix-wise Bradley-Terry-consistent preference model can be read off directly from autoregressive log-ratios.","The paper's experiments report consistent gains over DPO and a token-weighted baseline on mathematical reasoning and conversation benchmarks across four large language models."],"fun_headline_variants":["ADPO makes DPO reward every token, not just the full response","New formula: ADPO moves DPO's sum outside the log-sigmoid","ADPO: per-token preference optimization for LLMs","Autoregressive DPO: token-wise rewards, same KL constraint","ADPO's loss: sum of per-prefix sigmoids, not sigmoid of sum"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The derivation assumes the locally normalized product of per-token softmaxes obtained from the prefix energies is the same distribution as the global Boltzmann optimum of the KL-constrained reward maximization; the paper's appendix establishes the identity of the summed energies, not the equality of the two distributions, so this premise is load-bearing.","fun_headline_variants_meta":{"raw":{"variants":["ADPO makes DPO reward every token, not just the full response","New formula: ADPO moves DPO's sum outside the log-sigmoid","ADPO: per-token preference optimization for LLMs","Autoregressive DPO: token-wise rewards, same KL constraint","ADPO's loss: sum of per-prefix sigmoids, not sigmoid of sum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000826,"raw_usage":{"total_tokens":3451,"prompt_tokens":750,"completion_tokens":2701,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":2601}},"tokens_in":494,"tokens_out":2701,"duration_ms":16960,"temperature":1.0,"reasoning_tokens":2601,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T02:49:15.855318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a reward r on two-token sequences over a binary vocabulary and compute both the global Boltzmann distribution exp(-E2(y))/Σ exp(-E2(y')) and the product of per-position softmaxes Π_i exp(-E*_2(y_≤i))/Σ_{y_i} exp(-E*_2(y_≤i)). If the two distributions disagree on any sequence, the claim that ADPO preserves the optimal solution of KL-constrained reward maximization is false for that reward.","supporting_citations":[],"review_version":1}