REVIEW 5 major objections 5 minor 29 references
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Se-DPO replaces DPO's uniform token weighting with per-token KL credits that evolve during training, yielding win-rate gains of up to 9.8 points on AlpacaEval 2 and 12.2 on Arena-Hard.
desk verdict A plausibly effective online token-weighting heuristic whose stated theory is undercut by a coefficient inversion, making the paper's central derivation unsupported even though the empirical story is coherent. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is token credit, $c_t = \beta / \beta_t$, the multiplier that rescales each token's contribution to the preference logit and thereby sets the token's KL budget relative to the reference policy. The paper shows that variance-minimizing credits satisfy $c_t \propto |r_t^*| / \sigma_t^2$, proxies $|r_t^*|$ by the implicit reward magnitude $|\hat r_t|$ and $\sigma_t^2$ by reference entropy $H_{\mathrm{ref},t}$, and implements the mapping with a two-layer MLP $f_\phi$ whose outputs are mean-normalized within each response.
What would settle it
Substitute Eq. 7 into the preference logit and verify whether the coefficient multiplying $\hat r_t$ is $c_t$ or $1/c_t$; this algebraic check determines whether Proposition 1's credit formula matches the loss that Se-DPO trains.
Extended reading notes
Core claim
The paper claims that DPO's uniform token weighting is suboptimal and that the right per-token weight, which it calls token credit, should modulate each token's KL regularization based on its contribution to the preference outcome. It derives that under a variance-minimization criterion, optimal credit is proportional to the magnitude of the token's implicit reward and inversely related to its noise, and it observes empirically that this credit evolves throughout training, so any static credit becomes stale. Se-DPO operationalizes this by extracting the implicit reward magnitude and reference entropy from the standard DPO forward pass and mapping them through a lightweight MLP to credit values, normalized to unit mean per response. Across three base models and two preference annotators, Se-DPO reports improvements over DPO of up to 9.8 points on AlpacaEval 2 and 12.2 points on Arena-Hard.
Load-bearing premise
The method depends on the assumption that each token's credit weight directly multiplies that token's implicit reward in the preference logit; if the algebra instead places the weight in the denominator, the variance-minimization argument no longer applies to the loss actually trained.
Editorial extensions
If this is right
- If Se-DPO is correct, DPO's uniform KL coefficient is a bottleneck, and token-level KL budgets that adapt during training can yield material alignment gains without external models.
- Static token-importance methods, which compute importance once before or outside training, become progressively stale, so online updating is a necessary component, not an optional refinement.
- The calibration network's use of reference entropy as a noise proxy suggests that token-level reliability, not just reward magnitude, is a useful signal for preference optimization.
- The reported overhead is small (+6.85% wall-clock in one setting), so the approach is practical for standard DPO training pipelines.
Reading between the lines
- The paper's Eq. 8 assumes the credit $c_t$ multiplies the implicit reward $\hat r_t$; checking Eq. 7 suggests the consistent multiplier might be $1/c_t$, which would invert the derived credit. If that inversion holds, the gains may come from a different mechanism than the stated variance-minimization argument.
- A natural testable extension is whether the same online credit scheme transfers to reference-free preference losses (e.g., SimPO-style logits), where there is no implicit reward in the DPO sense but a length-normalized log-ratio could play a similar role.
- The paper does not address potential negative side effects of reweighting on generation length or on categories like reasoning and math, where it reports lower MT-Bench scores; a follow-up could examine whether adaptive token credit systematically trades off open-ended quality against reasoning quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Se-DPO, a token-level credit mechanism for Direct Preference Optimization. It extends DPO's uniform token weighting by introducing per-token KL coefficients β_t, defines token credit c_t = β/β_t, and derives, via a variance-minimization argument, that optimal credit should be proportional to |r*_t|/σ²_t. Since both quantities evolve during training, Se-DPO computes credit online from the model's own implicit reward magnitude |r̂_t| and reference entropy H_ref,t, fed through a lightweight MLP calibration network, with per-response mean normalization. Experiments on Llama-3-8B-Instruct, Llama-3.2-3B-Instruct, and Gemma-2-2B-it on AlpacaEval 2, Arena-Hard, and MT-Bench report gains over DPO, SimPO, TGDPO, and a static-credit baseline.
Significance. If the central claim held, per-token, temporally updated KL budgets would be a meaningful and low-cost improvement to DPO-style alignment: the method requires no external models, adds only ~7% wall-clock overhead, and shows large win-rate gains on two instruction-following benchmarks. The paper also contains a clean empirical observation—that DPO's token-level implicit reward magnitudes shift substantially during training—which is potentially useful regardless of the theoretical derivation. The main strengths are the low overhead, the online self-evolving design, and the breadth of ablations (warmup sensitivity, single-signal variants, static credit). However, the theoretical derivation does not match the implemented loss, and the empirical results are partly undermined by inconsistent MT-Bench degradation and length confounds.
major comments (5)
- [Section 2.2, Eqs. (7)-(8)] Equation (8) inverts the coefficient that follows from the token-specific KL objective in Eq. (7). From Eq. (7), the implicit reward is r_t = β_t log(π_θ/π_ref) + β_t log Z_t = (β_t/β) r̂_t + const = r̂_t / c_t + const, since c_t = β/β_t. The consistent preference logit formed from the per-token KL objective should thus weight r̂_t by 1/c_t (plus a context-dependent offset), not by c_t as written in Eq. (8). The footnote granting that Z_t does not cancel addresses the offset but not this coefficient inversion. Consequently, the variance-minimization Proposition 1 optimizes Var[Σ c_t r̂_t] for a logit that is not the one implied by Eqs. (4)–(7), so the conclusion c_t ∝ |r*_t|/σ²_t is not a property of the per-token KL problem as claimed. The implemented loss in Eq. (12) may still work as a heuristic, but the paper's central theoretical claim that Se-DPO implements optimal per-token KL budgets is unsupported by the derivation.
- [Section 3.2, Prop. 1] Proposition 1 solves a variance-minimization problem whose constraint, Σ c_t |r*_t| = C, is chosen ad hoc and is itself the object being justified. The paper never derives that signal preservation should be measured by this weighted sum rather than, say, Σ c_t r*_t (which would lead to different signs) or Σ |c_t r*_t|. The proportionality c_t ∝ |r*_t|/σ²_t is therefore a consequence of the chosen constraint, not an independent result about token credit. Moreover, the proof sketch in Appendix A states 'Since c_t = β/β_t > 0 by construction, we restrict to non-negative credits,' but the practice of unit-mean normalization plus an unconstrained positive MLP output means the credits are positive but not freely chosen; the mapping from the optimized c_t to actual KL budgets is only implicit. The theoretical motivation section should either justify the constraint from the preference-optimization objective or be reframed as a heuristic regularization analysis.
- [Section 3.2, Fig. 2] The empirical validation of the two proxy relationships—|r̂_t| ≈ |r*_t| and σ²_t ≈ H_ref,t—is circular with respect to the training setup. Figure 2(b–c) measures σ̂²_t as the variance of r̂_t across the latter half of training checkpoints of the same DPO run whose target is being explained, so the observed ρ = 0.86 between H_ref,t and σ̂²_t is a property of that specific trajectory, not a general relationship that licenses the use of reference entropy as a stable noise proxy. The claim that 'the correlation is strongest at low-reward positions and diminishes at high-reward positions' is reported qualitatively without numbers or a figure, and it is exactly the regime dependence used to justify the MLP. A quantitative breakdown (e.g., correlation within deciles) should be provided.
- [Table 1] The headline improvements are inconsistent with the reported MT-Bench results. On Llama-3-8B-Instruct, Se-DPO gains large win rates on AlpacaEval 2 and Arena-Hard but its MT-Bench score drops to 6.9–7.4 versus DPO's 8.0–8.2, and its MT-Bench win rate is 40.0–41.9 versus DPO's 37.5–46.3. This is more than a minor weakness: it indicates that the pairwise instruction-following gains do not translate to general conversational quality, and the paper's explanation in Section 4.3 (score vs. win-rate aggregation) does not account for the score drop on three out of four settings. The authors should report standard errors or significance across evaluation seeds, and should either explain the MT-Bench degradation with a category-level analysis for all settings or temper the claim of consistent improvement. Appendix Table 6 shows a category breakdown only for one setting (PairRM), where Reasoning, Coding, and Math collapse to 3.9–4.6; that pattern is consistent with length/format shifts rather than genuine alignment gains.
- [Section 4.2, Table 1] The comparison for smaller models is not controlled for training regime: DPO, SimPO, and TGDPO rows use full fine-tuning settings while Se-DPO uses LoRA for Llama-3.2-3B-Instruct and Gemma-2-2B-it. This makes the reported gains hard to interpret, since the differences could be partly due to the parameter-efficient training regime rather than the credit mechanism. The paper should either train all methods under the same LoRA setting or clearly state that the comparison is across different training regimes and explain why this is fair.
minor comments (5)
- [Section 2.2] The footnote correctly concedes that the partition-function terms Z_t generally do not cancel between chosen and rejected responses, but it does not quantify the residual. Given that the proposed loss in Eq. (12) drops these terms entirely, a short analysis of the magnitude of the residual (e.g., on a sample of tokens) would help the reader assess the approximation.
- [Algorithm 1] Algorithm 1 says warmup 'uses standard DPO with uniform credit (c_t = 1)', but the loss in line 15 is the weighted form; the warmup branch sets c_t = 1, so the loss reduces to standard DPO, which is fine. However, the calibration network is updated jointly with the policy during the guided phase; the paper does not specify whether ϕ is trained on a separate objective or purely through the main loss. Please clarify whether f_phi's only training signal is ∇_ϕ L in line 16.
- [Section 3.1] Figure 1(a,b) shows rank correlation and top-20% overlap for |r̂_t| 'in the original training run', but the experimental details in Appendix C say the analysis is on Qwen2.5-1.5B-Instruct with 2,000 samples, while the main experiments are on Llama/Gemma models. The transferability of the dynamics observation to the actual evaluated models is asserted, not demonstrated. A sentence noting that the dynamics were also verified on the main models (or that the analysis is representative) would be useful.
- [Appendix C] The paper reports different learning rates and β values for Llama-3-8B (lr 5e-7, β=0.01) versus smaller models (lr 5e-6, β=0.1). This is appropriate, but the same hyperparameters are used for all baselines? The text says 'DPO, SimPO, and TGDPO follow the hyperparameters reported in Zhu et al. (2025)', which may not be the same as those in the current setup. Please state explicitly which learning rate and β are used for each baseline in each setting.
- [Throughout] Minor typographical and formatting issues: Table 1 has a duplicated 'WR (%)' header under ArmoRM columns; 'Se-DPO(LoRA)' appears in the main results but Table 1's caption does not indicate which rows use LoRA; the abstract says 'up to 9.8 points on AlpacaEval 2 and 12.2 points on Arena-Hard' while the conclusion repeats the same numbers, but the introduction gives 'up to 50.6%' and the abstract says 'up to 50.6%'—the point differences should be consistent with the underlying win rates.
Circularity Check
No significant circularity; the central claims are benchmarked externally and the derivation, though flawed, is not self-referential.
full rationale
Se-DPO's headline improvements are measured on AlpacaEval 2, Arena-Hard, and MT-Bench, all external judges, so the empirical claim is independent of the training loss and of any fitted calibration parameters. Proposition 1 is a conditional optimization result: minimizing Var[Δ] under a signal-preservation constraint yields c_t ∝ |r*_t|/σ²_t by Lagrangian calculus; the constraint is a modeling assumption, not an input that is later relabeled as the output. The paper explicitly treats |rhat_t| and H_ref,t as proxies and does not test the closed form as a prediction; instead an MLP is fit, which is a normal learning procedure and does not make the external-benchmark evaluation circular. The footnote to Eq. 8 (Section 2.2) acknowledges the uncancelled Z_t term and calls Eq. 8 a 'principled approximation,' a stated limitation that concerns correctness, not circularity. The reader-flagged inversion between Eq. 7 and Eq. 8 (where the per-token KL objective implies a 1/c_t weight on rhat_t rather than c_t) is a genuine mathematical weakness in the theoretical motivation, but it is an inconsistency between equations, not a self-referential reduction; Proposition 1 remains a derivation from the (mis-specified) Δ in Eq. 8 rather than from the stated per-token KL objective. Self-citations to Zhao et al. 2026, Yu et al. 2026, Wang et al. 2025, and Li et al. 2026 appear only in related-work or general references and are not load-bearing for the method or its evaluation. Accordingly, no circularity score is warranted.
Assumptions & free parameters
free parameters (4)
- Credit calibration network weights f_phi =
learned jointly with policy
- Warmup steps W =
4% of total steps (default)
- KL coefficient beta =
0.01 for 8B, 0.1 for 3B and 2B
- LoRA rank and alpha for small models =
rank 64, alpha 128
assumptions (5)
- domain assumption Per-token DPO implicit reward rhat_t = beta(log pi_theta - log pi_ref) is a meaningful token-level signal in the canonical decomposition.
- domain assumption Reference entropy H_ref,t approximates the noise variance sigma^2_t of rhat_t.
- domain assumption Noise epsilon_t is m-dependent with small m; empirical autocorrelation is below 0.05 at lag 1.
- ad hoc to paper Minimizing Var[Delta] subject to sum c_t |r*_t| = C is the right criterion for token credit.
- ad hoc to paper Eq. 8's logit sum c_t rhat_t is a valid approximation to the token-specific KL objective despite the partition-function offset and inverse weighting.
invented entities (1)
-
Token credit c_t
Cite this review
Pith. "Pith review of Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization." pith.science (2026). https://pith.science/paper/2N2EPDRC
@misc{pith2026260809568,
author = {Pith},
title = {Pith review of: Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/2N2EPDRC}},
note = {Machine review of arXiv:2608.09568}
}
read the original abstract
Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval~2 and 12.2 points on Arena-Hard.
Figures
Reference graph
Works this paper leans on
-
[1]
The Softplus activation ensures non-negative output
→ Softplus. The Softplus activation ensures non-negative output. The learning rate for ϕ is set to 10 −3, higher than the policy learning rate, to enable fast adaptation to the evolving implicit reward signal. Training hyperparameters.All methods are trained for 1 epoch with the AdamW opti- mizer and a cosine learning rate schedule with 10% warmup ratio. ...
work page 2025
-
[5]
Samah Fodeh, Linhai Ma, Ganesh Puthiaraju, Srivani Talakokkul, Afshan Khan, Ashley Hagaman, Sarah R Lowe, and Aimee Kendall Roundtree. Tab-po: Preference optimization with a token-level adaptive barrier for token-critical structured generation.arXiv preprint arXiv:2603.00025,
-
[7]
The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
-
[8]
Zixuan Huang, Yikun Ban, Lean Fu, Xiaojie Li, Zhongxiang Dai, Jianxin Li, and Deqing Wang. Adaptive batch-wise sample scheduling for direct preference optimization.arXiv preprint arXiv:2506.17252,
-
[10]
11 Published as a conference paper at COLM 2026 Sangkyu Lee, Janghoon Han, Hosung Song, Stanley Jungkyu Choi, Honglak Lee, and Youngjae Yu. Kl penalty control via perturbation for direct preference optimization.arXiv preprint arXiv:2502.13177,
-
[11]
URL https: //arxiv.org/abs/2607.27845. Meng Li, Guangda Huzhang, Haibo Zhang, Xiting Wang, and Anxiang Zeng. Optimal transport-based token weighting scheme for enhanced preference optimization. InPro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 21311–21334,
-
[12]
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena- hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,
-
[13]
Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Simon Wang, Jiulong Shan, Albin Madappally Jose, Xiaojiang Liu, Lijie Wen, et al. Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights.arXiv preprint arXiv:2410.04350,
Show all 29 references
-
[14]
Autoregressive direct preference optimization.arXiv preprint arXiv:2602.09533,
Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, and Nakamasa Inoue. Autoregressive direct preference optimization.arXiv preprint arXiv:2602.09533,
-
[15]
Small-margin preferences still matter—if you train them right.arXiv preprint arXiv:2602.00954,
Jinlong Pang, Zhaowei Zhu, Na Di, Yichi Zhang, Yaxuan Wang, Chen Qian, and Yang Liu. Small-margin preferences still matter—if you train them right.arXiv preprint arXiv:2602.00954,
-
[16]
From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358,
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358,
-
[17]
Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
-
[19]
Interpretable preferences via multi-objective reward modeling and mixture-of-experts
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582–10592,
2024
-
[20]
Explore the reasoning capability of LLMs in the chess testbed
12 Published as a conference paper at COLM 2026 Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. Explore the reasoning capability of LLMs in the chess testbed. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of th...
2026
-
[21]
URLhttps://aclanthology.org/2025.naacl-short.52/
18653/v1/2025.naacl-short.52. URLhttps://aclanthology.org/2025.naacl-short.52/. Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He.β-dpo: Direct preference optimization with dynamicβ
2025
-
[22]
Se- lective preference optimization via token-level reward function estimation
Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Se- lective preference optimization via token-level reward function estimation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 7043–7067, 2025a. ...
2025
-
[23]
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang
URLhttps://arxiv.org/abs/2607.09816. Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. InICML,
-
[24]
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang
URL https://arxiv.org/abs/ 2608.04872. Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. InInternational Conference on Machine Learning, pp. 78498–78521. PMLR,
-
[25]
Wpo: Enhancing rlhf with weighted preference optimization
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimization. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp...
2024
-
[26]
Tgdpo: Harnessing token-level reward guidance for enhancing direct preference optimization
Mingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Tgdpo: Harnessing token-level reward guidance for enhancing direct preference optimization. arXiv preprint arXiv:2506.14574,
-
[27]
Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences.arXiv preprint arXiv:1909.08593,
1909 arXiv
-
[2017]
Sparsepo: Controlling preference alignment of llms via sparse token masks
Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Sparsepo: Controlling preference alignment of llms via sparse token masks. arXiv preprint arXiv:2410.05102,
-
[2019]
Under independent noise, Var[ˆ∆] =∑t c2 t σ2 t
13 Published as a conference paper at COLM 2026 A Proof of Proposition 1 Part (a).Since ct =β/β t > 0, we restrict to non-negative credits. Under independent noise, Var[ˆ∆] =∑t c2 t σ2 t . We minimize this subject to ∑t ct|r∗ t |=Cwithc t ≥0. The Lagrangian is L= ∑ t c2 t σ2 t...
2026
-
[2020]
Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,
-
[2022]
Bootstrapping language models with DPO implicit rewards
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, and Min Lin. Bootstrapping language models with DPO implicit rewards. InICLR, 2025a. Hongzhan Chen, Tao Yang, Shiping Gao, Ruijun Chen, Xiaojun Quan, Hongtao Tian, and Ting Yao. Discr...
-
[2023]
Step- dpo: Step-wise preference optimization for long-chain reasoning of llms.arXiv preprint arXiv:2406.18629,
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step- dpo: Step-wise preference optimization for long-chain reasoning of llms.arXiv preprint arXiv:2406.18629,
-
[2024]
Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862,
10 Published as a conference paper at COLM 2026 Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedb...
2026 arXiv
-
[2025]
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela
URLhttps://arxiv.org/abs/2502.01456. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306,
-
[2026]
Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,
Gemma Team. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.