SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment

Ji Liu; Jun Wu; Lulu Wang; Wenqiao Zhu; Yulun Zhang

REVIEW 3 major objections 5 minor 66 references

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A self-guided pilot term added to the DPO objective gives gradient flow separate, adjustable control over chosen and rejected reward updates, yielding MT-Bench gains up to 9.19% over DPO.

desk verdict SGDPO is a genuine DPO variant with a plausible empirical story, but its stated gradient mechanism is unsupported because the paper never says whether the pilot model is detached. read the letter →

arxiv 2505.12435 v1 pith:7Z25GMP5 submitted 2025-05-18 cs.LG cs.AIcs.DC

Wenqiao Zhu , Ji Liu , Lulu Wang , Jun Wu , Yulun Zhang This is my paper · ORCID

classification cs.LGcs.AIcs.DC

keywords directpreferenceoptimizationlanguagemodelalignmentpilottermgradientflowcontrolchosenrewardstabilitysubsequencesampling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct Preference Optimization (DPO) aligns language models with human preferences, but its gradient flow tends to push down the rejected response much faster than it pushes up the chosen one, and this asymmetry is unstable across base models. The paper argues that the fix is to add a "pilot term" to the DPO objective: a second pair of sigmoidal comparisons that use randomly sampled subsequences of the chosen and rejected responses, guided by a pilot policy. The added term gives the optimizer independent dials for the chosen and rejected reward updates, so the chosen response can receive a larger gradient without waiting for the rejected response to collapse. Theoretical analysis shows the chosen-gradient enlargement and a controllable update ratio, and experiments on four model configurations and eight benchmarks report up to 9.19% higher MT-Bench scores over DPO with more stable reward curves. The practical promise is that preference alignment can be made both more effective at producing preferred responses and less sensitive to starting conditions.

What carries the argument

The load-bearing object is the pilot term in Eq. 10, which pairs each full-response ratio with a pilot-model ratio on a random subsequence: $l_{\mathrm{pilot}} = \log\sigma(\beta\log X_1 - \beta\log Y_2) + \log\sigma(\beta\log Y_1 - \beta\log X_2)$, with $X_1=\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x)$, $X_2=\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x)$, and $Y_1,Y_2$ the analogous pilot-model ratios on subsequences $\hat{y}_w,\hat{y}_l$ of lengths $r_1 l_c$ and $r_2 l_c$. Since the pilot subsequences are shorter, their ratios shift relative to the full-response ratios, and Theorem 1 shows the gradient on the chosen reward is $\beta Y_2^\beta / (X_1(X_1^\beta+Y_2^\beta))$ while the rejected gradient depends on $Y_1$. This separation is what allows $r_1$ and $r_2$ to act as fine-grained controls over the reward updates, and Theorem 3 converts that into a monotonic factor $f(z)$ multiplying the DPO ratio.

What would settle it

Compute the actual gradient of Eq. 10 with respect to $\theta$ while letting gradients flow through $Y_1$ and $Y_2$, and compare the resulting chosen/rejected update ratio to Eq. 14; if the extra terms change the ratio materially, the theoretical explanation of SGDPO's gains is not the mechanism at work. A simpler empirical check is to run SGDPO with and without a stop-gradient on the pilot term and test whether the MT-Bench advantage persists.

Watch

Extended reading notes

Core claim

The paper's central claim is that a self-referential pilot term repairs the main failure mode of DPO. In standard DPO the chosen-to-rejected gradient ratio is $X_2/X_1$, and once the rejected reward drops, the chosen reward gets only small updates. SGDPO replaces part of that comparison with pilot-model ratios on subsequences: the first sigmoid compares the true chosen ratio $X_1$ with a pilot rejected ratio $Y_2$, and the second compares a pilot chosen ratio $Y_1$ with the true rejected ratio $X_2$. Because $Y_1,Y_2$ are derived from subsequences, they move away from the full-response ratios during training, and the partial derivatives of the loss with respect to $X_1$ and $X_2$ become controllable through the subsequence-length parameters $r_1,r_2$. The paper proves that the magnitude of the chosen reward gradient exceeds the DPO gradient when $Y_2>X_2$, and that the overall preferred-to-dispreferred update ratio is $(X_2/X_1)f(z)$ with $f(z)>1$ when $p_1p_2<1$, so the model keeps improving preferred responses rather than only suppressing rejected ones. Across Llama-3.1 8B and Qwen-2 7B, both instruct and base, the reported training reward curves are more consistent, and MT-Bench average scores improve by up to 9.19% over DPO.

Load-bearing premise

The theory differentiates the pilot loss while treating the pilot model's output probabilities as fixed, so the derived formulas describe the real update only if the pilot is detached from the gradient; the paper never states that it is.

Editorial extensions

If this is right

SGDPO's two subsequence-length hyperparameters become direct controls over how fast chosen and rejected rewards move during alignment, so practitioners can strengthen preferred-response generation without changing the preference dataset or adding a reward model.
The reported stability of training reward curves across Llama-3.1 and Qwen-2, instruct and base, implies the method is less sensitive to the quality of the SFT starting point than DPO.
Because the chosen-to-rejected update ratio grows as $f(z)$ when $p_1p_2<1$, the known DPO failure mode of decreasing the likelihood of preferred examples should be reduced, not only on MT-Bench but wherever the same reward geometry appears.
The added computation is small, up to 0.4% more training time by the paper's measurement, so the alignment gains are not bought with a large compute penalty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the pilot is literally the same network as the policy, the proofs require the pilot outputs to be treated as constants; the paper never says this explicitly. An obvious extension is to test a stop-gradient pilot against an unstopped one and see whether the extra cross-terms hurt, help, or simply change the effective learning rate.
Because $r_1,r_2$ tune a continuous family of objectives, they could be annealed during training, starting conservative and becoming more chosen-aggressive, or used as a diagnostic for how much chosen-update pressure a given model and benchmark needs.
The same subsequence-pilot construction is not tied to the specific DPO sigmoid loss; it could be dropped into other preference objectives such as IPO or SimPO to give them the same chosen-versus-rejected asymmetry control.
The AlpacaEval-2 results are mixed across models, which the paper acknowledges; this suggests the transfer of gradient-control gains is benchmark-dependent, and studying which reward-pattern changes predict MT-Bench wins but not AlpacaEval wins would be a useful next step.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

SGDPO is a real DPO variant with a genuinely new mechanism: it adds a pilot term built from random subsequences of the chosen and rejected responses, giving a knob that tunes how gradient weight falls on chosen versus rejected rewards. At r1=r2=1 and no subsequence it is exactly DPO, so it is a proper generalization. The empirical picture is also the strongest part: full training runs across four base-model/instruct combos, eight benchmarks, and the reward curves show a plausibly more stable behavior than DPO. The added compute is trivial. That spread of configurations is worth referee time.

The soft spot is exactly where the reader put it. The main theorems compute partial derivatives of lpilot with respect to X1 and X2 while holding Y1 and Y2 fixed. Then Theorem 3 assumes pi_pilot = pi_theta. If the pilot is literally the current policy, the loss depends on theta through Y1 and Y2 as well, and the chain rule adds terms the paper never writes. The derived gradient ratio in Formula 14 is not the actual parameter gradient used by an optimizer. The paper never states that the pilot is detached or a frozen snapshot. That is not a small omission; it is the theoretical mechanism, and as written it does not support the claim that the pilot term enlarges chosen-reward gradients. The fix is easy—state the detachment or redo the chain rule—but until then the theory is unsupported.

Empirically, the headline 9.19% gain is a grid-searched best (r1=0.9, r2=0.6) on one model, not the default setting. The paper is honest that they pick the best r per model, but the abstract presents the best-case number as the headline. Also, no seed-to-seed training variance is reported. They do a GPT-evaluator variance test in Appendix E, which is not the same. The AlpacaEval-2 results are mixed, and the authors admit the effect is benchmark-dependent. None of these kill the empirical claim, but they cap its strength.

On balance: this deserves a serious peer review, not a desk reject. A referee should ask for code, a seed-variance table, and, first, a clear statement of what pi_pilot is during backprop. If the pilot is meant to be detached, that single sentence would fix the theory. If it is not, the theory needs to be redone. The method itself is worth the effort.

Recommendation: send to review, conditional.

Referee Report

3 major / 5 minor

Summary. This paper proposes SGDPO, a modification of DPO that adds a "pilot" term to the loss: two log-sigmoid terms compare the chosen full response against a pilot-model subsequence of the rejected response and vice versa. The authors derive partial derivatives of the pilot loss with respect to chosen and rejected reward ratios, prove monotonicity results (Theorems 1-3), and claim that the pilot term enlarges the chosen-reward gradient and increases the chosen-to-rejected gradient ratio. Experiments on UltraFeedback with Llama-3.1-8B and Qwen-2-7B (instruct and base) report gains over DPO of up to 9.19% on MT-Bench and include ablations on r1 and r2, a comparison of Pilots versus Pilotd, and a two-run evaluation variance check. The main theoretical mechanism is that treating Y1 and Y2 as constants allows fine-grained control of gradient updates, but the paper never states whether gradients flow through the pilot model; this is the key issue for whether the derived formulas describe the actual training update.

Significance. The paper addresses a real limitation of DPO, namely weak chosen-reward updates, and proposes a simple, tunable loss modification whose empirical study is comparatively broad: four model configurations, MT-Bench, AlpacaEval-2, six conditional benchmarks, and several ablations. The algebra in Appendix A.1 is correct as conditional calculus, and Theorems 2 and 3 are properly conditional statements. However, the central explanatory claim that the pilot term changes the true parameter gradient in the way described by Eqs. 13 and 14 is not established, because the dependence of the pilot outputs Y1 and Y2 on the parameters is never addressed. With a clarified stop-gradient implementation and a verification of the p1p2<1 condition, the contribution could be solid; as written, the theoretical explanation does not match a standard autograd implementation unless the pilot is detached, and that specification is missing.

major comments (3)

[Section 3.3, Formulas 10-14 and Appendix A.1] The partial derivatives in Eqs. 11-12 are computed with respect to X1 and X2 while holding Y1 and Y2 fixed, and Eq. 13 plus Theorem 3's Eq. 14 are then used to claim that SGDPO enlarges the chosen-reward gradient and boosts the chosen-to-rejected gradient ratio. If the pilot model is pi_pilot = pi_theta as assumed in Theorem 3, then Y1 and Y2 are functions of theta, and the true parameter gradient of the loss in Eq. 10 contains additional chain-rule terms involving dY1/dtheta and dY2/dtheta. The paper never states that the pilot is detached or frozen. Without a stop-gradient, Eqs. 13-14 do not describe the actual update performed by an optimizer. The authors must either specify explicitly that a detached pilot is used, in which case the partial-derivative analysis is the relevant one, or redo the analysis with the full gradient.
[Section 3.3, Theorem 3] The conclusion that f(z)>1 and hence that SGDPO increases the chosen-to-rejected gradient ratio depends on the condition p1p2<1. However, p1 is the product of the remaining chosen-token probability ratios, which should typically exceed 1 as the chosen response becomes more likely, while p2 is the corresponding rejected-token product and is expected to be below 1. The assertion 'When the rejected reward decreases rapidly, it leads to p1p2<1' is not justified and is not a consequence of p2<1 alone. No empirical estimates of p1 and p2 are reported. Since this condition is load-bearing for goal G2, the paper needs to verify it during training or provide a rigorous argument that it holds.
[Section 3.4 and Appendix A.2] The paper never specifies how pi_pilot is initialized and updated in the experiments. Section 3.4 refers to constructing subsequences for 'the pilot model and the reference model,' but the experimental setup in Appendix A.2 only lists r1 and r2. If the pilot is the current policy with no stop-gradient, the theory does not apply; if it is a separate or periodically snapshotted model, that choice must be stated and its effect on the 'self-guided' interpretation discussed. This is both a reproducibility issue and a direct source of the gap between the theory and the reported implementation.

minor comments (5)

[Section 3.2, bullet G1] The bullet says 'enhance the gradient update of chosen rewards X2'; this appears to be a typo for X1.
[Theorem 2, statement] The word 'descreases' should be 'decreases'.
[Section 4.2, first paragraph] The text says SGDPO is compared with '7 state-of-the-art optimization algorithms,' but Section 4.1 lists six baselines; the count should be made consistent.
[Section 3.4] The construction of subsequences and the decomposition X1=p1Y1, X2=p2Y2 should clarify whether subsequences are contiguous spans and how the autoregressive conditioning is handled; as written, 'the product of the token probability ratios for the remaining tokens' is ambiguous if subsequences are arbitrary.
[Section 4.2] The phrase 'training awards' should be 'training rewards.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SGDPO gradient analysis is an analytic consequence of the pilot loss definition, not fitted to or equivalent to its empirical inputs.

full rationale

The paper's central theoretical claims (Theorems 1-3, Formulas 11-15) are derived by direct calculus from the pilot loss in Formula 10 and the definitions X1=p1Y1, X2=p2Y2. Formula 13 is an inequality comparison between Formula 11 and the DPO derivative Formula 6, valid under the stated assumption p2<1 (i.e., Y2>X2). Formula 14-15 is an algebraic identity following from setting πpilot=πθ and substituting the subsequence decomposition; it is not a fitted quantity. The p1p2<1 condition in Theorem 3 is an explicit regularity assumption about training dynamics, not a value extracted from the benchmark results. The experimental gains on MT-Bench, AlpacaEval-2, and conditional benchmarks are external to the derivation, and the tuning of r1,r2 in Appendix A.2/Table 5 is standard hyperparameter selection rather than a fitted input renamed as a prediction. Self-citations (Zhu et al. 2024/2025 and the Liu et al. federated-learning references) appear only in future-work and limitations passages and are not load-bearing. The substantive rigor gap is in Section 3.3 and Appendix A.1: Formulas 11-12 treat Y1,Y2 as constants while Theorem 3 sets πpilot=πθ, and the manuscript never explicitly states a stop-gradient or detached pilot. If gradients flow through πpilot, the displayed partial derivatives are not the true parameter gradients used by an optimizer. This is a correctness/assumption gap, not circularity: the derivations remain conditional mathematical statements and are not constructed from, or equivalent to, the reported empirical results.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The derivation leans on the Bradley-Terry preference model and DPO's reward reparameterization from prior work, then adds token-level factorization and a fixed-pilot assumption to arrive at the gradient ratio. The only fitted numbers introduced for the new method are r1 and r2, with β fixed for all baselines. The pilot model itself is an underspecified new component.

free parameters (3)

r1 (chosen subsequence length ratio) = Per-model best from grid search: Llama-3.1 instruct 1.0, Llama-3.1 base 0.9, Qwen-2 instruct 0.9, Qwen-2 base 0.6…
Controls the length of the random subsequence of the chosen response. Grid-searched on MT-Bench, and the 9.19% headline gain uses r1=0.9.
r2 (rejected subsequence length ratio) = Same grid-search values as r1 in Table 5; headline uses 0.6
Controls the length of the random subsequence of the rejected response. Smaller r2 increases Y2 and the chosen gradient per Theorem 2, so it is a tuned parameter.
β (DPO scaling factor) = 0.1 for all baselines
Fixed inverse-temperature hyperparameter inherited from DPO; chosen by hand for all methods, not fitted to the SGDPO result.

assumptions (6)

standard math Bradley-Terry model captures pairwise preferences
Used in Eq. 2 to convert rewards into preference probabilities; inherited from DPO.
domain assumption DPO reward reparameterization maps rewards to policy log-ratios
Eq. 1 imports the DPO result that r(x,y) = β log(πθ/πref) + β log Z(x).
domain assumption Autoregressive token factorization and decomposition X1=p1Y1, X2=p2Y2
Needed for Theorem 3; assumes the model's sequence probability factorizes over tokens and that random subsequences are token subsets whose products can be separated.
ad hoc to paper The pilot model's outputs Y1 and Y2 are constants in the gradient
Theorems 1-3 differentiate with Y1 and Y2 fixed, but Theorem 3 sets πpilot = πθ. No stop-gradient or frozen pilot is specified, so this is the key unstated assumption.
ad hoc to paper p2<1 and p1p2<1 during SGDPO training
Used to claim Y2>X2 and f(z)>1. The paper justifies this only by saying 'when the rejected reward decreases rapidly', with no proof or measurement of when this holds.
ad hoc to paper Random subsequences with length ratios at least 0.6 preserve semantic meaning
The paper sets r1≥0.6 and r2≥0.6 to conserve semantics, but provides no semantic or empirical criterion for this bound.

invented entities (1)

Pilot model πpilot
purpose: Computes subsequence log-ratios Y1 and Y2 that steer the gradient updates of chosen and rejected rewards.
No external falsifiable handle is given. The paper does not specify whether the pilot is a separate trained model, a frozen copy, or the current policy, making its role ambiguous.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment." pith.science (2026). https://pith.science/paper/7Z25GMP5

@misc{pith2026250512435,
  author       = {Pith},
  title        = {Pith review of: SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7Z25GMP5}},
  note         = {Machine review of arXiv:2505.12435}
}

read the original abstract

Direct Preference Optimization (DPO) is broadly utilized for aligning Large Language Models (LLMs) with human values because of its flexibility. Despite its effectiveness, it has been observed that the capability of DPO to generate human-preferred response is limited and the results of DPO are far from resilient. To address these limitations, in this paper we propose a novel Self-Guided Direct Preference Optimization algorithm, i.e., SGDPO, which incorporates a pilot term to steer the gradient flow during the optimization process, allowing for fine-grained control over the updates of chosen and rejected rewards. We provide a detailed theoretical analysis of our proposed method and elucidate its operational mechanism. Furthermore, we conduct comprehensive experiments on various models and benchmarks. The extensive experimental results demonstrate the consistency between the empirical results and our theoretical analysis and confirm the effectiveness of our proposed approach (up to 9.19% higher score).

Figures

Figures reproduced from arXiv: 2505.12435 by the authors.

**Figure 2.** Gradient flow of DPO (β = 0.1) with large values truncated at 1.5. • As DPO optimization progresses, the chosen reward X1 increases while the rejected reward X2 decreases. Consequently, X2 X1 < 1. According to Equation 8, this results in the gradient for the rejected reward being updated more quickly than that for the chosen reward. 3.3 SGDPO The theoretical framework discussed above suggests several directions fo… view at source ↗

**Figure 3.** Training reward curves for the Llama-3.1 base 8B model using the SGDPO method: (a) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: MT-Bench Results across different model configurations, using Llama-3.1 instruct 8B as the base model. Method Score1 Score2 Scoreavg DPO 7.29 6.41 6.85 ORPO 7.40 6.04 6.78 SGDPO 8.01 6.95 7.48 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 6.** Figure 6: Performance Metrics of Various Runs on MT [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: Training reward curves for the Llama-3.1 instruct 8B model using various alignment methods. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Training reward curves for the Llama-3.1 base 8B model using the DPO and SGDPO methods: (a) DPO. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: Visual representation of the function f(z) landscape [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Visual representation of the functions [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 32 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

AI@Meta. 2024. https://ai.meta.com/blog/meta-llama-3-1/ Introducing llama 3.1: Our most capable models to date

2024
[4]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

2024
[5]

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Remi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036

arXiv 2023
[6]

Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

arXiv 2022
[7]

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu...

arXiv 2024
[8]

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI

work page 2020

Show all 66 references

[9]

Ralph Allan Bradley and Milton E. Terry. 1952 a . Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika

1952
[10]

Ralph Allan Bradley and Milton E Terry. 1952 b . Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika

1952
[11]

Sheng, Huaiyu Dai, and Dejing Dou

Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor S. Sheng, Huaiyu Dai, and Dejing Dou. 2023. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. In Conf. on Empirical Methods in Natural Language Processi...

2023
[12]

Chunlu Chen, Ji Liu, Haowen Tan, Xingjian Li, Kevin I-Kai Wang, Peng Li, Kouichi Sakurai, and Dejing Dou. 2025. Trustworthy federated learning: privacy, security, and beyond. Knowledge and Information Systems, 67(3):2321--2356

2025
[13]

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. 2024. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369

2024 arXiv
[14]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

2018 arXiv
[15]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

2021 arXiv
[16]

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377

2023 arXiv
[17]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

2025 arXiv
[18]

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233

2023 arXiv
[19]

Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

2024 arXiv
[20]

Hashimoto

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387

2023 arXiv
[21]

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation matters in deep rl: A case study on ppo and trpo. In ICLR

2020
[22]

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

2024 arXiv
[23]

Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. 2024. Towards analyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprint arXiv:2404.04626

2024 arXiv
[24]

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In AISTATS

2024
[25]

Google. 2024. https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024 Our next-generation model: Gemini 1.5

2024
[26]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2021 arXiv
[27]

Jiwoo Hong, Noah Lee, and James Thorne. 2024. ORPO : Monolithic preference optimization without reference model. In EMNLP

2024
[28]

Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. 2024. Chatglm-rlhf: Practices of aligning large language models with human feedback. arXiv preprint arXiv:2404.00934

2024 arXiv
[29]

Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. A closer look at deep policy gradients. In ICLR

2020
[30]

Juncheng Jia, Ji Liu, Chendi Zhou, Hao Tian, Mianxiong Dong, and Dejing Dou. 2024. Efficient asynchronous federated learning with sparsification and quantization. Concurrency and Computation: Practice and Experience, 36(9):e8002

2024
[32]

Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung-Woon On. 2024 b . Binary classifier optimization for large language model alignment. arXiv preprint arXiv:2404.04656

2024 arXiv
[33]

Hashimoto

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

2023
[34]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. T ruthful QA : Measuring how models mimic human falsehoods. In ACL

2022
[35]

Ji Liu, Tianshi Che, Yang Zhou, Ruoming Jin, Huaiyu Dai, Dejing Dou, and Patrick Valduriez. 2024 a . Aedfl: efficient asynchronous decentralized federated learning with heterogeneous devices. In SIAM Int. Conf. on Data Mining ( SDM ) , pages 833--841. SIAM

2024
[36]

Ji Liu, Chunlu Chen, Yu Li, Lin Sun, Yulun Song, Jingbo Zhou, Bo Jing, and Dejing Dou. 2024 b . Enhancing trust and privacy in distributed networks: a comprehensive survey on blockchain-based federated learning. Knowledge and Information Systems, 66(8):4377--4403

2024
[37]

Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, and Dejing Dou. 2022 a . From distributed machine learning to federated learning: A survey. Knowledge and Information Systems, 64(4):885--917

2022
[38]

Ji Liu, Juncheng Jia, Tianshi Che, Chao Huo, Jiaxiang Ren, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2024 c . Fedasmu: Efficient asynchronous federated learning with dynamic staleness-aware model update. In AAAI Conference on Artificial Intelligence, volume 38, pages 13900--13908

2024
[39]

Ji Liu, Juncheng Jia, Beichen Ma, Chendi Zhou, Jingbo Zhou, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2022 b . Multi-job intelligent scheduling with cross-device federated learning. IEEE Transactions on Parallel and Distributed Systems ( TPDS ) , 34(2):535--551

2022
[40]

Ji Liu, Juncheng Jia, Hong Zhang, Yuhui Yun, Leye Wang, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2024 d . Efficient federated learning using dynamic update and adaptive pruning with momentum on shared server data. ACM Transactions on Intelligent Systems and Technology, 15(6):1--28

2024
[41]

Ji Liu, Esther Pacitti, Patrick Valduriez, and Marta Mattoso. 2015. A survey of data-intensive scientific workflow management. Journal of Grid Computing, 13:457--493

2015
[42]

Ji Liu, Jiaxiang Ren, Ruoming Jin, Zijie Zhang, Yang Zhou, Patrick Valduriez, and Dejing Dou. 2024 e . Fisher information-based efficient curriculum federated learning with large language models. In Conf. on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 10...

2024
[43]

Ji Liu, Zhihua Wu, Danlei Feng, Minxu Zhang, Xinxuan Wu, Xuefeng Yao, Dianhai Yu, Yanjun Ma, Feng Zhao, and Dejing Dou. 2023 a . Heterps: Distributed deep learning with reinforcement learning based scheduling in heterogeneous environments. Future Generation Computer Systems, 1...

2023
[44]

Ji Liu, Xuehai Zhou, Lei Mo, Shilei Ji, Yuan Liao, Zheng Li, Qin Gu, and Dejing Dou. 2023 b . Distributed and deep vertical federated learning with big data. Concurrency and Computation: Practice and Experience, 35(21):e7697

2023
[45]

Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407

2025
[46]

Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, and Xunliang Cai. 2024 f . Length desensitization in direct preference optimization. arXiv preprint arXiv:2409.06411

2024 arXiv
[47]

Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. 2024. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. arXiv preprint arXiv:2406.10957

2024 arXiv
[48]

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In NeurIPS

2024
[49]

OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774

2023 arXiv
[50]

OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

2024
[51]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

2022
[52]

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228

2024 arXiv
[53]

Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. In ACL

2024
[54]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS

2023
[55]

Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. 2025. Unintentional unalignment: Likelihood displacement in direct preference optimization. In ICLR

2025
[56]

Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. In NeurIPS

2020
[57]

Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. 2025. Simper: A minimalist approach to preference alignment without hyperparameters. In ICLR

2025
[58]

Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. 2024. Cal-dpo: Calibrated direct preference optimization for language model alignment. In NeurIPS

2024
[59]

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: pushing the boundaries of llm performance in machine translation. In ICML

2024
[60]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

2024 arXiv
[61]

Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999

2024 arXiv
[62]

Hong Zhang, Ji Liu, Juncheng Jia, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2022. Fedduap: Federated learning with dynamic update and adaptive pruning using shared data on the server. In Int. Joint Conf. on Artificial Intelligence ( IJCAI ) , pages 2776--2782

2022
[63]

P Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685

2023 arXiv
[64]

Chendi Zhou, Ji Liu, Juncheng Jia, Jingbo Zhou, Yang Zhou, Huaiyu Dai, and Dejing Dou. 2022. Efficient device scheduling with multi-job federated learning. In AAAI Conf. on Artificial Intelligence, volume 36, pages 9971--9979

2022
[65]

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

2023 arXiv
[66]

Wenqiao Zhu, Lulu Wang, and Jun Wu. 2025. Addressing cold-start problem in click-through rate prediction via supervised diffusion modeling. In AAAI

2025
[67]

Wenqiao Zhu, Chao Xu, Lulu Wang, and Jun Wu. 2024. Psc: Extending context window of large language models via phase shift calibration. In EMNLP

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

AI@Meta. 2024. https://ai.meta.com/blog/meta-llama-3-1/ Introducing llama 3.1: Our most capable models to date

2024

[4] [4]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

2024

[5] [5]

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Remi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036

arXiv 2023

[6] [6]

Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

arXiv 2022

[7] [7]

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu...

arXiv 2024

[8] [8]

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI

work page 2020