Autoregressive Direct Preference Optimization

Mahiro Ukai; Masahiro Kaneko; Masanari Oi; Nakamasa Inoue; Naoaki Okazaki

REVIEW 4 major objections 4 minor 1 cited by

Autoregressive Direct Preference Optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that DPO can be reformulated over response prefixes without changing its theoretical optimum, yielding ADPO, a loss that applies a per-prefix Bradley-Terry comparison at every token instead of one comparison over the whole

desk verdict The empirical work is solid and the granularity idea is worth a look, but the central theoretical claim conflates a locally normalized per-token product with the global Boltzmann optimum, so the paper's main justification does not hold. read the letter →

arxiv 2602.09533 v2 pith:TFSRN2UZ submitted 2026-02-10 cs.AI

Masanari Oi , Mahiro Ukai , Masahiro Kaneko , Naoaki Okazaki , Nakamasa Inoue This is my paper · ORCID

classification cs.AI

keywords directpreferenceoptimizationBradley-Terrymodelautoregressivelanguagemodelsprefix-wiserewardtoken-levelalignmentlengthmeasuresLLMmathematicalreasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that direct preference optimization (DPO) locks in a response-level Bradley-Terry model, even though the networks being trained are autoregressive; that mismatch is a choice, not a necessity. It introduces Autoregressive DPO (ADPO), built from reward-based energies defined on all response prefixes, which yields a loss where the summation runs outside the log-sigmoid: each position gets its own preference signal. The paper claims this preserves the KL-constrained reward-maximization solution of DPO and proves a reparameterization completeness theorem: any reward class consistent with prefix-wise Bradley-Terry models can be written as the log-ratio of an autoregressive model to the reference. It also separates two length measures—token length and feedback length—and shows DPO is the special case where feedback length is one. A sympathetic reader would care because this gives DPO-style training a principled granularity dial that, in the paper's experiments across four large language models, consistently improves alignment.

What carries the argument

The machinery is the prefix closure Y* together with two prefix-wise energies, E*_1 = -r*(x, y_≤i) and E*_2 = -(1/β)r*(x, y_≤i) - log πref(y_i | y_<i, x). These define a prefix-wise Bradley-Terry preference model and a locally normalized autoregressive posterior p2(y_i | y_<i, x), which is what lets the loss factor into a sum of per-position log-sigmoid terms. The two length measures—token length μ and feedback length μ'—plus a strong composition ξ that groups tokens into feedback segments, supply the granularity family parameterizing the loss.

What would settle it

Take a reward r on two-token sequences over a binary vocabulary and compute both the global Boltzmann distribution exp(-E2(y))/Σ exp(-E2(y')) and the product of per-position softmaxes Π_i exp(-E*_2(y_≤i))/Σ_{y_i} exp(-E*_2(y_≤i)). If the two distributions disagree on any sequence, the claim that ADPO preserves the optimal solution of KL-constrained reward maximization is false for that reward.

Watch

Extended reading notes

Core claim

The central discovery is a reformulation of DPO that makes the autoregressive assumption before applying the Bradley-Terry model, not after. Define the prefix closure Y* of the output space and a prefix-wise reward r*(x, y_≤i); with the reference model treated as autoregressive, the posterior distribution factorizes position-by-position, so the implicit reward at each token is the per-token log-ratio β log(πθ/πref). The resulting ADPO loss, -E Σ_i log σ(β log ...), moves the summation outside the log-sigmoid, in contrast to DPO's log σ of a sum. The paper proves that every reward class consistent with prefix-wise Bradley-Terry models is reparameterizable by an autoregressive model, and that

Load-bearing premise

The derivation assumes the locally normalized product of per-token softmaxes obtained from the prefix energies is the same distribution as the global Boltzmann optimum of the KL-constrained reward maximization; the paper's appendix establishes the identity of the summed energies, not the equality of the two distributions, so this premise is load-bearing.

Editorial extensions

If this is right

DPO is recovered as the special case μ'(y)=1: the original objective has an implicit feedback-length measure of one for every response.
Setting μ'=μ gives a fully token-level objective; intermediate segmentations give a whole family of DPO variants at arbitrary granularity.
If the reparameterization theorem is right, the implicit reward of any prefix-wise Bradley-Terry-consistent preference model can be read off directly from autoregressive log-ratios.
The paper's experiments report consistent gains over DPO and a token-weighted baseline on mathematical reasoning and conversation benchmarks across four large language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the per-token normalizers in Eq. (12) are not equal to the global partition function, ADPO is a new token-level objective whose optimum may not coincide with DPO's—a distinction that would matter for diagnosing whether ADPO's improvements come from finer credit assignment rather than from the same KL-constrained optimum.
Editorial inference: the μ vs μ' distinction suggests a practical design principle for preference data collection: feedback length can be chosen to match the granularity at which human evaluators actually judge quality (whole response, step, or token), rather than being dictated by the model's tokenizer.
Editorial inference: if the prefix-wise credit-assignment pattern reported—late-prefix first, then early, then final answer tokens—holds generally, ADPO-style losses could serve as a probe for where reasoning errors localize in chain-of-thought training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: the paper's central claim does not survive contact with its own equations. In Appendix B they show that the sum of the prefix energies equals the response-level energy, and then conclude that the product of per-token softmaxes is the optimal solution of the KL-constrained reward maximization. That conclusion is not valid. The per-token softmax has history-dependent partition functions Z_i(y_

What is genuinely new and useful: the framing of preference optimization over the prefix closure, and the explicit separation of token length from feedback length. The resulting loss is, as the reader notes, essentially the token-level DPO objective already present in TDPO and related work, and the derivation is a reparameterization rather than a new objective. But the granularity families via strong compositions are a practical extension, and the experiments are carefully done: multiple models, multiple benchmarks, standard deviations, thorough hyperparameter exploration. The empirical gains over DPO are modest but consistent, and the analysis of training dynamics is a nice addition.

Soft spots, in proportion: the Theorem 1 reparameterization claim is basically a softmax identity and does not add theoretical depth. Corollary 1 re-labels DPO as the mu'=1 case rather than proving anything about DPO. The two-length-measures discussion is partly definitional—the paper introduces the measures and then discovers them in DPO. None of this would be fatal by itself, but alongside the Appendix B gap it means the paper's theoretical foundation is not established.

The paper is worth reading for the experiments and the granularity framework, but not as a source for DPO theory. It deserves a serious referee—the empirical contribution is real and the loss may be useful even if the stated theory fails—though the review would likely recommend major revision or rejection unless the optimality claim is either fixed or dropped. For my own work, I would not cite it as a theoretical result, but I might cite the granularity experiments.

Recommendation: send to peer review, but with the expectation that the central theory needs to be either corrected or substantially softened.

Referee Report

4 major / 4 minor

Summary. The paper proposes Autoregressive Direct Preference Optimization (ADPO), a DPO variant in which the Bradley–Terry comparison is applied prefix-wise rather than response-wise. The authors define prefix-level reward and energy functions, derive the ADPO loss L_ADPO = -E Σ_i log σ(β log πθ(yw_i|·)/πref(yw_i|·) - β log πθ(yl_i|·)/πref(yl_i|·)), and claim that this loss preserves the KL-constrained optimal solution of the original response-level reward (Section 4, Appendix B). They further claim that any reward consistent with prefix-wise BT models can be reparameterized by an autoregressive model (Theorem 1), that DPO is the special case with feedback length μ'=1 (Corollary 1), and that token length μ and feedback length μ' are independent design choices. Experiments on GSM8K, MATH500, AlpacaEval 2, Arena-Hard, and MT-Bench compare DPO, cDPO, SimPO, and ADPO variants.

Significance. If the theoretical claims were valid, the paper would contribute a clean loss variant and a useful conceptual distinction between token-level and feedback-level granularity. The empirical results are consistently positive across models and benchmarks, and the appendix includes derivations, standard deviations, and ablations. However, the central theoretical claim is not supported: the locally normalized product p2 in Eqs. (11)–(12) is not shown to equal the global Boltzmann optimum for the response-level reward, so ADPO's equivalence to DPO's KL-constrained optimum is unproven. The reparameterization theorem is true by construction but does not establish the missing link. The contribution therefore reduces to an empirically motivated loss modification, without the advertised theoretical foundation.

major comments (4)

[§4, Eqs. (11)–(12); Appendix B, Corollary 2] The central claim that ADPO "preserves" the KL-constrained optimal solution is not established. The prefix-wise p2 in Eq. (12) is a locally normalized autoregressive product: p2(y|x)=∏_i exp(-E*_2(x,y_{≤i}))/Z_i(y_{<i}) with Z_i(y_{<i})=Σ_{y_i} exp(-E*_2(x,y_{≤i})). The global Boltzmann distribution for the response-level reward r is exp(-E_2(x,y))/Z with Z=Σ_y exp(-E_2(x,y)). Corollary 2 proves only Σ_i E*_2 = E_2, which is an energy-sum identity; it does not imply Z=∏_i Z_i(y_{<i}) for all y. In general these partition functions differ, so p2 ≠ exp(-E_2)/Z. Appendix A's use of r_o with f(x,y_{<i})=β log Z_i makes each local partition function one, but then Σ_i r_o = r - β Σ_i log Z_i, a different reward. The optimum of L_ADPO is therefore not shown to coincide with the DPO/KL optimum for r.
[Appendix A, Eqs. (32)–(49)] The loss derivation is algebraically consistent, but it only shows that L_ADPO is the negative log-likelihood of a product of pairwise comparisons under the chosen p1. Since the relation p2(y_i|y_<i,x)=exp(-E*_2)/Z_i is assumed rather than derived from the KL objective, the reparameterization in Eq. (48) does not identify the reward that πθ is implicitly optimizing. To establish ADPO's theoretical foundation, the paper would need to state the implicit reward r_eff such that πθ(y|x) ∝ πref(y|x) exp(r_eff(x,y)/β) and show that the fixed point of Eq. (14) equals the optimum of J(r_eff). This is missing.
[§5.1, Theorem 1 and Proposition 1] Theorem 1 is a softmax reparameterization that holds by construction: for any prefix-wise r*, Proposition 1 defines π(y_i|y_<i,x) ∝ πref(y_i|y_<i,x) exp(r*(x,y_{≤i})/β), and Eq. (19) sums to the desired log-ratio. Such a representation always exists for any reward r via an additive decomposition (Lemma 1), so it cannot by itself establish that ADPO is a faithful extension of DPO. The sentence "all reward classes consistent with the prefix-wise BT models" (Theorem 1) is also circular: the equivalence class [r*] is defined so that reward shifts are absorbed, and the BT likelihood is invariant to those shifts by construction. Corollary 1 amounts to declaring μ'=1 for DPO rather than deriving it from the model.
[§5.3, Eq. (20)] The argument that DPO has an implicit feedback length μ'(y)=1 is not compelling. Eq. (20) defines μ'(y)=length(ν(y))=dim(R)=1 by choosing the evaluation metric to map to R; this is a definitional choice, not a consequence of the BT model or of ADPO. The introduction of two "length measures" is therefore partially invented, and the paper's claim to be the first to distinguish them is a framing contribution rather than a theoretical discovery. This does not by itself invalidate the empirical loss, but it should be presented as a design choice.

minor comments (4)

[Eq. (6)] Right-padding is mentioned but its effect on the log-ratio sums is not analyzed; padding tokens with zero log-ratio and variable lengths can bias the loss.
[Definition 2 vs Appendix A] T' is defined as μ'(y) in Definition 2, but Appendix A sets T'=max{μ'(y_w), μ'(y_l)}; make the notation consistent.
[§5.1] The claim that Theorem 1 goes "in contrast to DPO theory" is overstated, because DPO already presents the same reparameterization r=β log(π/πref). The novelty should be positioned more carefully.
[Table 3] Granularity-family comparisons lack error bars or significance tests; many differences are within one point, so conclusions about granularity trends would be strengthened by variance reporting in the main text.

Circularity Check

2 steps flagged · score 4.0 of 10

ADPO's loss is self-contained and empirically tested, but the claimed discoveries of DPO's 'implicit μ′=1' and of 'autoregressive reparameterization completeness' are definitional/constructive: they are true by the authors' own definitions, not derived facts about DPO.

self definitional [Section 5.2 (Corollary 1), Section 5.3, Eq. (20)]
"Concretely, μ′ measures the length of sequences y in an evaluation scenario by first mapping y into a one-dimensional space using an evaluation metric ν:Y→R, and then defining μ′(y)=length(ν(y))=dim(R)=1. ... Instead, we emphasize that the output space Y remains a sequence space. Then, Corollary 1 implies that the original DPO formulation has an implicit length measure μ′:Y→N that assigns a length of one, i.e., μ′(y)=1, to every sequence y∈Y."

The paper first defines the feedback length μ′ by stipulation: every response is mapped into R, so μ′(y)=1 for all y. With μ′=1, the single feedback unit is the whole response and the ADPO loss (Eq. 14) has one log-sigmoid term containing the full response-level log-ratio — exactly the DPO loss. Thus ADPO reduces to DPO by construction, and the 'implicit length measure in DPO' is not a discovered property of DPO but a relabeling of DPO's response-level aggregation as 'feedback length 1'. The two-length-measures insight is therefore definitional rather than derived.
renaming known result [Section 5.1, Theorem 1 and proof (Eq. 19), with Proposition 1 in Appendix C]
"Theorem 1: All reward classes consistent with the prefix-wise Bradley–Terry models can be represented with the reparameterization r(x,y)=βlog π(y|x)/πref(y|x) for some autoregressive model π. Proof: For any reward function r, an additive decomposition r∗ exists by Lemma 1. By Proposition 1, there exists an autoregressive model π such that r(x,y)=Σ_i r∗(x,y≤i)≡Σ_i r∗_∘(x,y≤i)=βlog π(y|x)/πref(y|x)."

The autoregressive model π in Proposition 1 is explicitly constructed as the local Boltzmann/softmax distribution of the given reward: π(yi|y<i,x)=exp(−E∗2(x,y≤i))/Z(x,y<i), so r∗_∘ is, by construction, βlog(π/πref) up to the partition-shift that defines the equivalence class [r∗]. Lemma 1's 'additive decomposition' is also true by definition for any function. Hence Theorem 1 restates the standard softmax/DPO reparameterization in prefix coordinates: it constructs the policy from the reward, then announces that any reward can be represented by that policy. The response-level equality is asserted only modulo the prefix-shift class (the ≡ sign), a class the authors introduced, so the 'completeness' claim is a definitional identity rather than an independent result about DPO.

full rationale

The derivation of the ADPO loss itself is not circular in the fitted-prediction sense: L_ADPO follows algebraically from the prefix-wise BT model and the locally normalized autoregressive p2 (Eqs. 10–14, Appendix A), and the experimental comparisons are external, standard benchmarks. There is no load-bearing self-citation chain: the paper does not justify its central premise by citing its own prior work, and no fitted parameter is renamed as a prediction. The circularity is concentrated in the paper's interpretive theoretical claims. First, the 'feedback length measure' μ′ is introduced by the authors via μ′(y)=dim(R)=1, and then Corollary 1 presents DPO as having an implicit μ′=1; this reduction is true by construction and is a renaming of DPO's response-level aggregation. Second, Theorem 1's claim that 'any reward function can be reparameterized by an autoregressive model' is an existence statement whose witness is the softmax/Boltzmann policy constructed from the reward itself; together with the definitional additive decomposition (Lemma 1), the theorem is a softmax identity in autoregressive coordinates, not an independent derivation. I also note, as a non-circularity correctness issue, that the claim 'the optimal solution is preserved' (Section 4, Appendix B) is not established: Corollary 2 proves only the energy-sum identity Σ_i E∗2=E2, not the distributional equality of the locally normalized product p2(y) with the global Boltzmann distribution of E2; this gap affects the theoretical foundation but does not constitute a circular reduction, so it does not increase the score further. Overall, because the main objective and the empirical results have independent content, a moderate score is appropriate.

Assumptions & free parameters 2 free parameters · 5 assumptions · 2 invented entities

The central derivation rests on a new prefix-wise BT model and on the unproven identification of a locally normalized product with the global KL-optimal policy. The free parameters are the standard β and the newly introduced granularity choices; the invented entities are the prefix reward and the feedback-length measure.

free parameters (2)

β = 1.0 (main); 0.5 and 1.5 in sensitivity analysis
KL-regularization temperature in DPO/ADPO; a standard hyperparameter chosen by hand, not fitted to data.
feedback length m (adaptive) / window k (static) = m=256 primary; m=1..512 and k=1..8 explored
Introduced by ADPO to set granularity; the paper does not derive it from theory, so the main results depend on this choice.

assumptions (5)

ad hoc to paper Prefix-wise Bradley-Terry model: p1(yw≻yl|x)=Π_i exp(-E*_1(yw≤i))/(exp(-E*_1(yw≤i))+exp(-E*_1(yl≤i)))
A new preference model introduced in Eq. (10); not derived from human data or from response-level BT.
ad hoc to paper The locally normalized product p2(y|x)=Π_i p2(y_i|y_<i,x) is the optimal solution of the KL-constrained reward maximization for response-level r
This is the load-bearing assumption in Appendix B; generally false because per-token partition functions are history-dependent, so the product does not equal the global Boltzmann distribution.
ad hoc to paper Every reward has an additive decomposition r* and any decomposition preserves ADPO's guarantees
Lemma 1 is trivially true, but the claim that the choice of decomposition is irrelevant is unproven and false in general; the ADPO optimum depends on the decomposition.
ad hoc to paper Feedback length μ' can be chosen independently of token length μ via strong compositions
Introduced in §5.4 as a design choice; no external evidence that human feedback granularity follows this construction.
standard math Bradley-Terry model and KL-constrained reward maximization (Remark 1)
Standard RLHF/DPO background used as the starting point for the derivation.

invented entities (2)

prefix-wise reward r* over prefix closure Y*
purpose: To define prefix-wise Bradley-Terry preferences and derive the ADPO loss
A mathematical construct introduced by the paper; it has no falsifiable handle outside the model.
feedback length measure μ'
purpose: To parameterize granularity between full-response and token-level feedback
A new conceptual quantity; the claim that DPO implicitly has μ'=1 is a definitional re-labeling rather than an empirical discovery.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Autoregressive Direct Preference Optimization." pith.science (2026). https://pith.science/paper/TFSRN2UZ

@misc{pith2026260209533,
  author       = {Pith},
  title        = {Pith review of: Autoregressive Direct Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TFSRN2UZ}},
  note         = {Machine review of arXiv:2602.09533}
}

abstract

Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences. However, the widespread reliance on the response-level Bradley-Terry (BT) model may limit its full potential, as the reference and learnable models are assumed to be autoregressive only after deriving the objective function. Motivated by this limitation, we revisit the theoretical foundations of DPO and propose a novel formulation that explicitly introduces the autoregressive assumption prior to applying the BT model. By reformulating and extending DPO, we derive a novel variant, termed Autoregressive DPO (ADPO), that explicitly integrates autoregressive modeling into the preference optimization framework. Without violating the theoretical foundations, the derived loss takes an elegant form: it shifts the summation operation in the DPO objective outside the log-sigmoid function. Furthermore, through theoretical analysis of ADPO, we show that there exist two length measures to be considered when designing DPO-based algorithms: the token length $\mu$ and the feedback length $\mu'$. To the best of our knowledge, we are the first to explicitly distinguish these two measures and analyze their implications for preference optimization in LLMs.

Figures

Figures reproduced from arXiv: 2602.09533 by the authors.

**Figure 1.** Static and adaptive families of ADPO. (a) Token and feedback length measures. Each subsequence zi is defined by a strong composition ξ. (b–d) Static families with a fixed window size k. (e–g) Adaptive families with a fixed number of subsequences. (e) corresponds to DPO. Blue rectangles indicate regions where summation is applied inside the log-sigmoid function. Adaptive Family. This family decomposes each sequence i… view at source ↗

**Figure 2.** Comparison of training dynamics between DPO and ADPO. The evolution of log probabilities for preferred sequences (solid line) and dispreferred sequences (dashed line) during training is shown. Top row: static family with k = 1, 2. Bottom row: adaptive family with m = 16, 256 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Training dynamics (conversation task). β = 1.0 for 3 epochs using AdamW with LoRA (r = 16), at a learning rate of 2×10−5 for DPO and 4×10−5 for cDPO. Each problem was sampled 64 times with a top-p probability of 50% during contrastive estimation. For the conversation task, all models are trained for 1 epoch using preference data scored by PairRM. Experiments are conducted using four NVIDIA H100 GPUs. We will release… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Prefix-wise reward variance and margin. F. Additional Analysis F.1. Prefix-wise Reward Dynamics As formalized in Eq. 16, ADPO assigns an implicit reward at each step i. To understand how these implicit rewards are distributed across prefixes and how they evolve during …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
cs.CL 2026-08 reject novelty 5.0 of 10

Se-DPO assigns evolving per-token credits from implicit reward magnitude and reference entropy during DPO training, improving instruction-following win rates but with an internally inconsistent derivation.

Reference graph

Works this paper leans on

8 extracted references · 6 linked inside Pith · cited by 1 Pith paper

[6]

Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a

Yang, A., Hui, B., Lin, J., Zhou, J., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yang, C., Deng, H., and Li, Q. Step-controlled dpo: Leverag- ing stepwise errors for enhancing mathematical reasoning of llms. InProc. International Conference on Learning Representations (ICLR), 2025b. Ye, C., Xiong, W., Zhang, Y ., Dong, H., Jiang...

arXiv
[7]

Derivation of ADPO loss We provide a proof of Eq

11 Autoregressive Direct Preference Optimization A. Derivation of ADPO loss We provide a proof of Eq. (14). We first provide detailed definition of ADPO loss and then show the Proposition A.1. Definition A.1.We define ADPO loss as LADPO =−E (x,Y)∼D logp 1(yw ≻y l|x) .(32) with ADPO energies given by E∗ 1 (x, y≤i) =−r ◦(x, y≤i),(33) E∗ 2 (x, y≤i) =− 1 β r◦...

2007
[64]

critical tokens

on GSM8K for Llama-3-8B. We observe that ADPO outperforms DPO across all ranks and the performance gap widens at higher ranks. Model Size.Table 8 examines the impact of scaling model size from Llama-3-8B to Llama-3-70B on GSM8K. ADPO consistently outperforms DPO at both model sizes, achieving 89.23% with a size of 70B. Table 6.Hyperparameter study forβ. R...

2025
[2015]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv
[2017]

Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

arXiv
[2023]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Grattafiori, A., Dubey, A., Jauhri, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

arXiv
[2024]

Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

arXiv
[2025]

K., Wu, Y ., and Guo, D

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint 2402.03300, 2024a. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseek- math: Pu...

arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [6]

Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a

Yang, A., Hui, B., Lin, J., Zhou, J., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yang, C., Deng, H., and Li, Q. Step-controlled dpo: Leverag- ing stepwise errors for enhancing mathematical reasoning of llms. InProc. International Conference on Learning Representations (ICLR), 2025b. Ye, C., Xiong, W., Zhang, Y ., Dong, H., Jiang...

arXiv

[2] [7]

Derivation of ADPO loss We provide a proof of Eq

11 Autoregressive Direct Preference Optimization A. Derivation of ADPO loss We provide a proof of Eq. (14). We first provide detailed definition of ADPO loss and then show the Proposition A.1. Definition A.1.We define ADPO loss as LADPO =−E (x,Y)∼D logp 1(yw ≻y l|x) .(32) with ADPO energies given by E∗ 1 (x, y≤i) =−r ◦(x, y≤i),(33) E∗ 2 (x, y≤i) =− 1 β r◦...

2007

[3] [64]

critical tokens

on GSM8K for Llama-3-8B. We observe that ADPO outperforms DPO across all ranks and the performance gap widens at higher ranks. Model Size.Table 8 examines the impact of scaling model size from Llama-3-8B to Llama-3-70B on GSM8K. ADPO consistently outperforms DPO at both model sizes, achieving 89.23% with a size of 70B. Table 6.Hyperparameter study forβ. R...

2025

[4] [2015]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv

[5] [2017]

Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

arXiv

[6] [2023]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Grattafiori, A., Dubey, A., Jauhri, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

arXiv

[7] [2024]

Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

arXiv

[8] [2025]

K., Wu, Y ., and Guo, D

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint 2402.03300, 2024a. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseek- math: Pu...

arXiv