Pith. sign in

REVIEW 3 major objections 5 minor 167 references

SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Beat GDPO at every scale by distilling reward-specialized teachers

desk verdict A genuinely new specialize-and-merge recipe for multi-reward RL, with a strong formal analysis and thorough ablations, but the headline claim leans on an unvalidated LLM-judge metric and some standard metrics go the other way. read the letter →

arxiv 2608.03092 v1 pith:7PJBAVPH submitted 2026-08-04 cs.LG cs.AIcs.CL

Wen Wang , Jiahua Bao , Tu Yongsiqi , Yihao Liu , Haotian Zhou , Haoxuan Ma , Mengyu Zhou , Wenkui Fan
show 3 more authors
Junwei He Xiaoxi Jiang Guanjun Jiang
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords multi-rewardreinforcementlearningRLHFpolicydistillationsparserewardgroupdecouplingtoolcallingsafealignmentLLMpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-reward reinforcement learning for language models usually aggregates several reward signals into one advantage, and even after per-dimension normalization a sparse binary reward can be drowned out by a dense fine-grained one. This paper claims to resolve that imbalance with SMOPD, a two-stage recipe: first train separate teachers, each with a skewed reward-priority profile so that a sparse reward gets a strong enough optimization signal; then merge the teachers into one student with online policy distillation plus a balanced task anchor. The claim is that the resulting single policy keeps the dense-reward capability while finally learning the sparse capability—format compliance on tool calling jumps from 8.8% to 97.5% at 1.5B—and that SMOPD outperforms GDPO on both complementary and conflicting reward settings across three model scales. If true, it would decouple capability acquisition from reward balancing: instead of scalarizing rewards inside one policy, you specialize first and rebalance later.

What carries the argument

The load-bearing object is the combination of a reward-priority teacher profile and the union-top-$\kappa$ teacher-mixture distillation target. On a GDPO advantage estimator that normalizes each reward dimension separately, a skewed priority weight vector such as $(0.1, 0.9)$ multiplies the normalized sparse-reward advantage by a factor of $9$, converting a signal that is mostly zero into a $9\times$ expected gradient contribution; that is what makes the sparse reward learnable. In Stage 2, each frozen teacher emits only its top-$\kappa$ next-token distributions ($\kappa=16$), the candidates are pooled by $\log\alpha_m + \log p_{\pi_m}(v)$, and the top 16 of the union define the forward-KL target, so at one token position the student can see the format teacher's confidence on XML wrapper tokens and the accuracy teacher's confidence on function arguments at the same time. The balanced GDPO anchor is the piece that lets the student pass the imitation ceiling of pure distillation, because the anchor's policy gradient survives once the student already matches the teacher mixture.

What would settle it

Run the 3B RLLA and the Llama-3.2-3B safe-alignment comparisons with five independent training seeds and report mean plus or minus standard deviation; if the SMOPD-versus-GDPO gaps (2.738 vs 2.728 RLLA mean; 5.590 vs 5.583 Overall) fall inside run-to-run noise, the cross-backbone superiority claim is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the reward-density imbalance—a sparse reward contributing no within-group variance in most rollout groups and therefore no advantage signal after per-reward normalization—can be fixed outside the advantage estimator. SMOPD's Stage 1 trains one GDPO-style teacher per reward with complementary priority weights, such as $(0.1, 0.9)$ for format and $(0.9, 0.1)$ for accuracy, so the sparse-reward teacher amplifies that reward's gradient by the weight ratio. Stage 2 then runs on-policy distillation: the student generates its own rollouts, frozen teachers emit top-$\kappa$ next-token distributions, the union top-$\kappa$ mixture is used as a forward-KL target, and a balanced GDPO anchor keeps optimizing the original task-level reward. The paper reports that this merged student outperforms GDPO across Qwen2.5-1.5B, 3B, 7B and Llama-3.2-3B, with the largest gains on complementary tool-calling rewards, and that it can exceed both of its own teachers on safe alignment. The central mechanism is that capability acquisition and reward balancing are separated into two stages rather than forced into one scalarized objective.

Load-bearing premise

The paper's claim that SMOPD outperforms GDPO rests on the evaluation metrics being fair and stable: on API-Bank the headline result uses an LLM judge that rescues exact-match failures—under the original strict matcher SMOPD at 3B scores 58.12% versus GDPO's 58.46% and GD2PO's 59.63%—and the paper reports single runs with no error bars.

Editorial extensions

If this is right

  • A sparse binary reward can be made learnable without losing dense-reward capability: at 1.5B, format compliance goes from 8.8% (GDPO) to 97.5% while the accuracy-reward component stays essentially unchanged (1.765 vs 1.761).
  • The same two-stage recipe transfers to conflicting rewards: on helpful/harmless safe alignment, SMOPD beats GDPO on all three backbones and reaches 5.646 Overall at 7B, above both its useful and harmless teachers.
  • The anchor plus distillation split means the student is not bounded by its teachers: pure distillation alone sits near the teacher level (5.544 at 7B), and adding the anchor lifts it to 5.639 or 5.646.
  • The divergence direction matters: forward KL with top-$\kappa=16$ support is sufficient, retaining roughly 0.994 of teacher mass, and beats a sampled-token reverse-KL variant by 0.082 Overall because forward KL is mode-covering and preserves both teachers' modes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is scaling beyond two rewards: for $K$ rewards, $K$ teachers with cyclically skewed priority profiles and uniform mixing is the obvious next experiment, and the result would show whether the method degrades gracefully as the teacher mixture gains more modes.
  • Because the API-Bank headline uses an LLM judge that rescues exact-match failures, the claim of superiority is metric-dependent: a fair reading is that SMOPD improves functionally equivalent tool calling, whereas under the original literal string matcher GDPO at 3B remains stronger (58.46% vs 58.12%).
  • The mechanism predicts that on a sparse reward the anchor is inert early in training—near-zero success rate makes mixed rollout groups rare—so the distillation stage is what activates learning; an efficient extension would anneal the anchor weight $\lambda$ upward once the format success rate passes a threshold.
  • The uniform teacher mixture matching or beating adaptive confidence-failure gates suggests that within a single domain, specialized teachers already occupy complementary modes, so token-level gating may be unnecessary; this observation could inform other multi-teacher on-policy distillation designs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses multi-reward reinforcement learning for LLMs when reward dimensions differ in signal density (dense vs. sparse). It proposes SMOPD, a two-stage method: Stage 1 trains reward-specialized teachers by applying skewed GDPO priority weights, and Stage 2 merges the teachers into a single student via on-policy distillation with a uniform top-κ teacher mixture under forward KL, plus a balanced GDPO anchor as a sequence-level objective. Experiments cover complementary rewards (tool-calling accuracy + format compliance on RLLA, BFCL, API-Bank) and conflicting rewards (helpful + harmless on HH-RLHF, PKU-SafeRLHF, Alpaca) across Qwen2.5-1.5B/3B/7B and Llama-3.2-3B. The headline claim is that SMOPD outperforms GDPO across all backbones, with a particularly large gain in 1.5B format compliance (8.8% to 97.5%). The paper also includes ablations of the anchor, top-k size, KL direction, priority skew, and teacher-mixing gates, and a supplementary formal argument that the GDPO anchor breaks the teacher-imitation ceiling.

Significance. If the empirical claims hold, SMOPD provides a conceptually clean and useful recipe: decouple capability acquisition (specialization) from reward balancing (merge). The format-compliance result is striking, the ablations are thorough, and the choice of a uniform, parameter-free teacher mixture is elegant. The supplementary formalization of why the anchor overcomes the teacher ceiling is a genuine strength. However, the headline 'outperforms GDPO across backbones' is currently stronger than the evidence: the paper's own strict exact-match API-Bank results and the 1.5B BFCL AST result show SMOPD losing to GDPO, and no results come with error bars or multiple seeds, leaving several small margins within noise. With metric validation and uncertainty quantification, this could be a solid contribution; as written, the central empirical claim needs revision.

major comments (3)
  1. [§4.2, Tables 1 and 11, Supplementary F/G] The unqualified claim that SMOPD outperforms GDPO across 1.5B, 3B, and 7B backbones is contradicted by the paper's own standard metrics. Under the original strict API-Bank exact matcher (Table 11), SMOPD at 3B scores 58.12% Overall, below GDPO (58.46%) and GD2PO (59.63%); in Table 1, BFCL AST at 1.5B is 70.7% for SMOPD versus 72.6% for GDPO. The substitution of the LLM-judge protocol (Supplementary F) is motivated with anecdotal examples (Table 10) but is not validated: there is no human-agreement measurement, no per-method rescue rate, and no check for judge bias across methods. Because the central claim is universal over backbones and settings, a standard metric where SMOPD loses is a direct counterexample unless the metric substitution is independently validated or the claim is re-scoped.
  2. [Tables 1–4, §4.2] All results are single-run without error bars or significance testing. Several reported gains are small and fall within plausible run-to-run noise: 3B RLLA Mean (2.738 vs 2.728), Llama-3.2-3B safe-alignment Overall (5.590 vs 5.583), and Qwen2.5-3B Overall versus GD2PO (5.669 vs 5.650). These margins are not established on an 80-prompt test set or small benchmark evaluations. The claim of 'consistent improvement' requires repeated seeds or statistical testing for at least the tight-margin settings.
  3. [Supplementary B and Table 3] The formal argument that pure OPD is capped at the teacher mixture and that the anchor's gradient survives at the ceiling is sound, but the empirical demonstration is limited to a single setting (Qwen2.5-7B safe alignment at κ=32). Since the main results use κ=16 and different backbones, it would strengthen the paper to show the OPD-only ceiling and the anchor's lift in at least one additional setting to confirm that the mechanism, rather than the specific run, is responsible for the effect.
minor comments (5)
  1. [§3.2, Eq. (6)] The top-κ mixture target \tilde p_mix is not explicitly renormalized, although the forward-KL expression in Eq. (7) requires a normalized target distribution over the retained support; please state that the retained set is renormalized and discuss the impact of this approximation.
  2. [Figure 1(a)] The 'sim.' values in the caption are not defined; please state that they are cosine similarities between the displayed advantage profiles.
  3. [§4.1] The text says 'We train SMOPD on five backbones,' but the experiments cover four distinct models (Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-7B, Llama-3.2-3B); if a backbone trained in two settings is counted twice, say so explicitly.
  4. [§4.2] The sentence 'requiring balance competing alignment objectives (Section 4.2)' contains a self-referential section number; it should refer to the conflicting-rewards setup rather than the same section.
  5. [§5 and Abstract] The conclusion and abstract repeat the unqualified 'outperforms GDPO across backbones' claim; these statements should be qualified to name the metrics that support the claim and acknowledge the strict-metric and BFCL-AST exceptions reported in the supplementary material.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SMOPD's reported gains are empirical comparisons against external baselines, and its training objective is not defined in terms of the reported metrics.

full rationale

SMOPD is an empirical method rather than a derivation that reduces to fitted inputs. Stage 1 trains reward-specialized teachers by reusing GDPO's group-normalized advantages with priority weights (Eq. 4), and Stage 2 combines them through a frozen uniform teacher mixture and forward-KL distillation plus a balanced GDPO anchor (Eqs. 5-9). None of these equations defines the reported metrics (RLLA Mean, BFCL AST, API-Bank, safety reward-model scores) in terms of the method's own outputs; the metric values are measured on held-out benchmarks after training. The central comparison to GDPO is not forced by construction: SMOPD's objective contains a GDPO anchor plus an additional distillation term, so any improvement is an empirical outcome, not an identity. The paper also reports settings where SMOPD does not win (e.g., original strict API-Bank at 3B, where SMOPD scores 58.12% versus GDPO 58.46% and GD2PO 59.63% in Table 11, and Llama-3.2-3B safety where SMOPD trails GD2PO), which further shows the evaluation is not rigged to force the headline claim. The GD2PO baseline is implemented from the authors' prior work and is a self-citation, but it is not load-bearing: SMOPD is claimed to outperform GDPO, and the GD2PO comparison is a supplementary baseline that is directly re-run under the same recipe rather than imported as evidence. The modified API-Bank LLM-judge protocol is a metric choice that could be criticized on validity grounds, but it is not a fitted parameter or a training input, so it is a correctness/risk concern rather than circularity. Overall, the paper's claimed derivation chain is self-contained and externally benchmarked, with no step that reduces to its own inputs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no physical entities. The central claim rests on standard RL estimators, domain assumptions about distillation and reward-model validity, and hand-chosen hyperparameters (teacher priority profiles, lambda, top-kappa, uniform mixture weights). The most consequential assumption is that the reported evaluation metrics, especially the LLM-judge API-Bank protocol, measure the intended capabilities without bias.

free parameters (6)
  • Teacher priority weights (RLLA) = accuracy teacher [0.9,0.1]; format teacher [0.1,0.9]
    Hand-chosen to amplify the target reward by 9x; moderate [0.7,0.3] ablation shows robustness but the exact values are not derived.
  • Teacher priority weights (safe alignment) = useful teacher [0.7,0.3]; harmless teacher [0.3,0.7]
    Hand-chosen; no systematic tuning; one of several choices the authors could have made.
  • OPD coefficient lambda = 1.0
    Default distillation strength; no ablation is reported for lambda.
  • OPD top-kappa = 16
    Default support size; ablation across 16/32/64 shows small sensitivity (0.007 Overall on 7B safe alignment).
  • Teacher mixture weights alpha_m = 0.5, 0.5
    Uniform default; adaptive gates tested in Supplementary E do not consistently beat it.
  • Anchor weights = 0.5, 0.5
    Equal weights for the balanced GDPO anchor, used to preserve task-level balance.
assumptions (5)
  • domain assumption GDPO's per-reward normalized advantage with priority weights is a valid policy-gradient estimator.
    Used as the anchor loss (Eq. 8) and for teacher training; the paper builds on GDPO without re-deriving its validity.
  • domain assumption On-policy forward-KL distillation from a teacher mixture transfers capabilities to the student.
    Central to Stage 2; justified by prior work (Agarwal et al. 2024, Gu et al. 2024) rather than derived here.
  • domain assumption Teachers trained under skewed priority profiles acquire complementary, mergeable capabilities.
    Empirical basis of Stage 1; no theorem guarantees the student can inherit both without interference.
  • domain assumption The dual reward models on PKU-SafeRLHF provide valid proxies for helpfulness and harmlessness.
    Used for both training and evaluation of the conflicting-reward setting, without calibration against human labels.
  • domain assumption The API-Bank LLM judge (qwen3.7-plus) provides a valid measure of functional tool-call equivalence.
    Used for exact-match failures; no human validation or inter-judge agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation." pith.science (2026). https://pith.science/paper/7PJBAVPH

@misc{pith2026260803092,
  author       = {Pith},
  title        = {Pith review of: SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7PJBAVPH}},
  note         = {Machine review of arXiv:2608.03092}
}
read the original abstract

We aim to improve model performance in multi-reward reinforcement learning training process. Existing Group reward-Decoupled Normalization Policy Optimization (GDPO) has mitigated the issue of reward signals masking one another during direct scalarization by normalizing each reward dimension separately before aggregation. However, our experiments show that GDPO still struggles to balance reward signals with different granularities. Specifically, in some particular training tasks, the model may receive a dense reward that assigns fine-grained scores ranging from 0.1 to 1.0, together with a sparse reward that provides only binary feedback of either 0 or 1. In such cases, we find that the sparse reward may provide an insufficient optimization signal, preventing its corresponding capability from being effectively reinforced. Therefore, how can we strengthen the optimization signal from the sparse reward without sacrificing the capability already learned from the fine-grained reward? To overcome this limitation, we propose Specialize-and-Merge Online Policy Distillation (SMOPD), a two-stage training method for multi-reward optimization. Stage1-Specialize: SMOPD first employs reward-priority configurations to train multiple reward-specialized teachers, allowing each reward to be learned under conditions where its signal can effectively drive optimization. Stage2-Merge: SMOPD then utilizes online policy distillation to combine the reward-specialized capabilities of these teachers into a single student policy, while maintaining balanced task-level optimization. To validate our method, we conduct experiments on two multi-reward settings: complementary rewards(tool-calling accuracy and format) and conflicting rewards (helpful and harmless rewards). Based on above settings, SMOPD outperforms GDPO across 1.5B, 3B and 7B backbones.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

167 extracted references · 24 canonical work pages

  1. [1]

    Approximating

    John Schulman , year=. Approximating

  2. [2]

    Thinking Machines Lab: Connectionism , year=

    On-Policy Distillation , author=. Thinking Machines Lab: Connectionism , year=. doi:10.64434/tml.20251026 , note=

    doi:10.64434/tml.20251026

  3. [3]

    2601.05242 , archivePrefix=

    Shih-Yang Liu and Xin Dong and Ximing Lu and Shizhe Diao and Peter Belcak and Mingjie Liu and Min-Hung Chen and Hongxu Yin and Yu-Chiang Frank Wang and Kwang-Ting Cheng and Yejin Choi and Jan Kautz and Pavlo Molchanov , year=. 2601.05242 , archivePrefix=

    arXiv

  4. [4]

    Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Xiao Bi and Haowei Zhang and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo , year=. 2402.03300 , archivePrefix=

    arXiv

  5. [5]

    2017 , eprint=

    Proximal Policy Optimization Algorithms , author=. 2017 , eprint=

    2017

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Deep Reinforcement Learning from Human Preferences , author=. Advances in Neural Information Processing Systems , volume=

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. Advances in Neural Information Processing Systems , volume=

Show all 167 references
  1. [9]

    NIPS Deep Learning and Representation Learning Workshop , year=

    Distilling the Knowledge in a Neural Network , author=. NIPS Deep Learning and Representation Learning Workshop , year=

  2. [10]

    International Conference on Learning Representations (ICLR) , year=

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author=. International Conference on Learning Representations (ICLR) , year=

  3. [11]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep Mutual Learning , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  4. [12]

    Yuxian Gu and Li Dong and Furu Wei and Minlie Huang , booktitle=

  5. [13]

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Sequence-Level Knowledge Distillation , author=. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    2016

  6. [14]

    Jongwoo Ko and Sungnyun Kim and Tianyi Chen and Se-Young Yun , booktitle=

  7. [15]

    2026 , eprint=

    Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation , author=. 2026 , eprint=

    2026

  8. [16]

    2606.30406 , archivePrefix=

    Wenhan Ma and Jianyu Wei and Liang Zhao and Hailin Zhang and others , year=. 2606.30406 , archivePrefix=

    arXiv

  9. [17]

    Prateek Yadav and Derek Tam and Leshem Choshen and Colin Raffel and Mohit Bansal , booktitle=

  10. [18]

    Language Models are Super

    Le Yu and Bowen Yu and Haiyang Yu and Fei Huang and Yongbin Li , booktitle=. Language Models are Super

  11. [19]

    International Conference on Learning Representations (ICLR) , year=

    Editing Models with Task Arithmetic , author=. International Conference on Learning Representations (ICLR) , year=

  12. [20]

    Proceedings of the 39th International Conference on Machine Learning (ICML) , year=

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time , author=. Proceedings of the 39th International Conference on Machine Learning (ICML) , year=

  13. [21]

    Constitutional

    Yuntao Bai and Saurav Kadavath and Sandipan Kundu and Amanda Askell and Jackson Kernion and Andy Jones and Anna Chen and Anna Goldie and Azalia Mirhoseini and Cameron McKinnon and others , year=. Constitutional. 2212.08073 , archivePrefix=

    arXiv

  14. [22]

    Juntao Dai and Xuehai Pan and Ruiyang Sun and Jiaming Ji and Xinbo Xu and Mickel Liu and Yizhou Wang and Yaodong Yang , booktitle=. Safe

  15. [23]

    2022 , eprint=

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , author=. 2022 , eprint=

    2022

  16. [24]

    Hashimoto , year=

    Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , year=. Stanford Alpaca: An Instruction-following

  17. [25]

    Arithmetic Control of

    Haoxiang Wang and Yong Lin and Wei Xiong and Rui Yang and Shizhe Diao and Shuang Qiu and Han Zhao and Tong Zhang , booktitle=. Arithmetic Control of

  18. [26]

    Rewarded soups: towards

    Alexandre Ram\'e and Guillaume Couairon and Mustafa Shukor and Corentin Dancette and Jean-Baptiste Gaya and Laure Soulier and Matthieu Cord , booktitle=. Rewarded soups: towards

  19. [27]

    Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Weinan Dai and Tiantian Fan and Gaohong Liu and Juncai Liu and Lingjun Liu and Xin Liu and others , booktitle=

  20. [28]

    Back to Basics: Revisiting

    Arash Ahmadian and Chris Cremer and Matthias Gall\'e and Marzieh Fadaee and Julia Kreutzer and Olivier Pietquin and Ahmet \"Ust\"un and Sara Hooker , booktitle=. Back to Basics: Revisiting

  21. [29]

    2025 , eprint=

    Group Sequence Policy Optimization , author=. 2025 , eprint=

    2025

  22. [30]

    2606.16771 , archivePrefix=

    Haotian Liu and Yihao Liu and Jingwei Ni and Siyuan Huang and Xinpeng Liu and Pengyu Cheng and Jiajun Song and Ruijin Ding and Junfeng Li and Zhechao Yu and Mengyu Zhou and Hongteng Xu and Xiaoxi Jiang and Guanjun Jiang , year=. 2606.16771 , archivePrefix=

  23. [31]

    2605.25604 , archivePrefix=

    Guochao Jiang and Jingyi Song and Guofeng Quan and Chuzhan Hao and Guohua Liu and Yuewei Zhang , year=. 2605.25604 , archivePrefix=

    arXiv

  24. [32]

    Advances in Neural Information Processing Systems , volume=

    Cheng Qian and Emre Can Acikgoz and Qi He and Hongru Wang and Xiusi Chen and Dilek Hakkani-T. Advances in Neural Information Processing Systems , volume=

  25. [33]

    2024 , eprint=

    Qwen2.5 Technical Report , author=. 2024 , eprint=

    2024

  26. [34]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  27. [35]

    Gonzalez and Hao Zhang and Ion Stoica , booktitle=

    Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica , booktitle=. Efficient Memory Management for Large Language Model Serving with

  28. [36]

    Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu , booktitle=

  29. [37]

    Patil and Tianjun Zhang and Xin Wang and Joseph E

    Shishir G. Patil and Tianjun Zhang and Xin Wang and Joseph E. Gonzalez , booktitle=. Gorilla: Large Language Model Connected with Massive

  30. [38]

    Minghao Li and Yingxiu Zhao and Bowen Yu and Feifan Song and Hangyu Li and Haiyang Yu and Zhoujun Li and Fei Huang and Yongbin Li , booktitle=

  31. [39]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  32. [40]

    2606.07705 , archivePrefix=

    Yuchen He and Baolong Bi and Shenghua Liu and Huaming Liao and Yuyao Ge and Bolin Wan and Siqian Tong and Juan Chen and Jiafeng Guo and Xueqi Cheng , year=. 2606.07705 , archivePrefix=

    arXiv

  33. [41]

    The Perfect Blend: Redefining

    Tengyu Xu and Eryk Helenowski and Karthik Abinav Sankararaman and Di Jin and Kaiyan Peng and Eric Han and Shaoliang Nie and Chen Zhu and Hejia Zhang and Wenxuan Zhou and Zhouhao Zeng and Yun He and Karishma Mandyam and Arya Talabzadeh and Madian Khabsa and Gabriel Cohen and Yu...

    arXiv

  34. [42]

    International Conference on Learning Representations (ICLR) , year=

    Knowledge Fusion of Large Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  35. [43]

    2406.16768 , archivePrefix=

    Alexandre Ram\'e and Johan Ferret and Nino Vieillard and Robert Dadashi and L\'eonard Hussenot and Pierre-Louis Cedoz and Pier Giuseppe Sessa and Sertan Girgin and Arthur Douillard and Olivier Bachem , year=. 2406.16768 , archivePrefix=

    arXiv

  36. [44]

    Findings of the Association for Computational Linguistics: ACL 2024 , year=

    Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization , author=. Findings of the Association for Computational Linguistics: ACL 2024 , year=

    2024

  37. [45]

    Singh and DJ Strouse and Tuomas Sandholm and Ruslan Salakhutdinov and Anca D

    Ted Moskovitz and Aaditya K. Singh and DJ Strouse and Tuomas Sandholm and Ruslan Salakhutdinov and Anca D. Dragan and Stephen McAleer , booktitle=. Confronting Reward Model Overoptimization with Constrained

  38. [46]

    Advances in Neural Information Processing Systems , volume=

    Gradient Surgery for Multi-Task Learning , author=. Advances in Neural Information Processing Systems , volume=

  39. [47]

    2025 , doi=

    Daya Guo and others , journal=. 2025 , doi=

    2025

  40. [48]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  41. [49]

    Measuring Mathematical Problem Solving With the

    Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt , booktitle=. Measuring Mathematical Problem Solving With the

  42. [50]

    2019 , eprint=

    Fine-Tuning Language Models from Human Preferences , author=. 2019 , eprint=

    2019

  43. [51]

    Advances in Neural Information Processing Systems , volume=

    Learning to summarize with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  44. [52]

    International Conference on Machine Learning , pages=

    Scaling Laws for Reward Model Overoptimization , author=. International Conference on Machine Learning , pages=

  45. [53]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages=

    f-Divergence Minimization for Sequence-Level Knowledge Distillation , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , pages=

  46. [54]

    2026 , eprint=

    Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts , author=. 2026 , eprint=

    2026

  47. [55]

    Advances in Neural Information Processing Systems , volume=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. Advances in Neural Information Processing Systems , volume=

  48. [56]

    Yujia Qin and Shihao Liang and Yining Ye and Kunlun Zhu and Lan Yan and Yaxi Lu and Yankai Lin and Xin Cong and Xiangru Tang and Bill Qian and Sihan Zhao and Lauren Hong and Runchu Tian and Ruobing Xie and Jie Zhou and Mark Gerstein and Dahai Li and Zhiyuan Liu and Maosong Sun...

  49. [57]

    Gonzalez and Ion Stoica , booktitle=

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , booktitle=. Judging

  50. [58]

    Machine Learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine Learning , volume=

  51. [59]

    2024 , eprint=

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline , author=. 2024 , eprint=

    2024

  52. [60]

    Findings of the association for computational linguistics: EMNLP 2024 , pages=

    On diversified preferences of large language model alignment , author=. Findings of the association for computational linguistics: EMNLP 2024 , pages=

    2024

  53. [61]

    arXiv preprint arXiv:2309.03126 , year=

    Everyone deserves a reward: Learning customized human preferences , author=. arXiv preprint arXiv:2309.03126 , year=

    arXiv

  54. [62]

    arXiv preprint arXiv:2504.07491 , year=

    Kimi-vl technical report , author=. arXiv preprint arXiv:2504.07491 , year=

    arXiv

  55. [63]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  56. [64]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  57. [65]

    2000 , eprint=

    The information bottleneck method , author=. 2000 , eprint=

    2000

  58. [66]

    1997 , publisher=

    Information theory and statistics , author=. 1997 , publisher=

    1997

  59. [67]

    International conference on machine learning , pages=

    Club: A contrastive log-ratio upper bound of mutual information , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  60. [68]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  61. [69]

    arXiv preprint arXiv:1807.03748 , year=

    Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

    arXiv

  62. [70]

    2019 , eprint=

    Learning deep representations by mutual information estimation and maximization , author=. 2019 , eprint=

    2019

  63. [71]

    International conference on machine learning , pages=

    Mutual information neural estimation , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  64. [72]

    Advances in neural information processing systems , volume=

    Infogan: Interpretable representation learning by information maximizing generative adversarial nets , author=. Advances in neural information processing systems , volume=

  65. [73]

    arXiv preprint arXiv:2310.03716 , year=

    A long way to go: Investigating length correlations in rlhf , author=. arXiv preprint arXiv:2310.03716 , year=

    arXiv

  66. [74]

    arXiv preprint arXiv:2310.13548 , year=

    Towards understanding sycophancy in language models , author=. arXiv preprint arXiv:2310.13548 , year=

    arXiv

  67. [75]

    arXiv preprint arXiv:2501.09620 , year=

    Beyond reward hacking: Causal rewards for large language model alignment , author=. arXiv preprint arXiv:2501.09620 , year=

    arXiv

  68. [76]

    arXiv preprint arXiv:2310.05199 , year=

    Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback , author=. arXiv preprint arXiv:2310.05199 , year=

    arXiv

  69. [77]

    Advances in Neural Information Processing Systems , volume=

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling , author=. Advances in Neural Information Processing Systems , volume=

  70. [78]

    2025 , eprint=

    Defining and Characterizing Reward Hacking , author=. 2025 , eprint=

    2025

  71. [79]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Beyond Excess and Deficiency: Adaptive Length Bias Mitigation in Reward Models for RLHF , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  72. [80]

    A survey of reinforcement learning from human feedback , author=

  73. [81]

    arXiv preprint arXiv:2410.21276 , year=

    Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=

    arXiv

  74. [82]

    2022 , eprint=

    The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models , author=. 2022 , eprint=

    2022

  75. [83]

    2023 , eprint=

    Goal Misgeneralization in Deep Reinforcement Learning , author=. 2023 , eprint=

    2023

  76. [84]

    2024 , eprint=

    Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs , author=. 2024 , eprint=

    2024

  77. [85]

    2025 , eprint=

    HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages , author=. 2025 , eprint=

    2025

  78. [86]

    Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

    Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.620

    2024 doi

  79. [87]

    2024 , eprint=

    Post-hoc Reward Calibration: A Case Study on Length Bias , author=. 2024 , eprint=

    2024

  80. [88]

    2021 , booktitle=

    Improving Zero-Shot Voice Style Transfer via Disentangled Representation Learning , author=. 2021 , booktitle=

    2021

  81. [89]

    2019 , booktitle=

    Learning deep representations by mutual information estimation and maximization , author=. 2019 , booktitle=

    2019

  82. [90]

    2025 , eprint=

    Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment , author=. 2025 , eprint=

    2025

  83. [91]

    The method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. The method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    1952

  84. [92]

    2022 , eprint=

    Training language models to follow instructions with human feedback , author=. 2022 , eprint=

    2022

  85. [93]

    2024 , eprint=

    AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback , author=. 2024 , eprint=

    2024

  86. [94]

    2024 , eprint=

    RLHF Workflow: From Reward Modeling to Online RLHF , author=. 2024 , eprint=

    2024

  87. [95]

    2024 , eprint=

    RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style , author=. 2024 , eprint=

    2024

  88. [96]

    arXiv preprint arXiv:2304.03277 , year=

    Instruction Tuning with GPT-4 , author=. arXiv preprint arXiv:2304.03277 , year=

    arXiv

  89. [97]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  90. [98]

    2017 , eprint=

    RACE: Large-scale ReAding Comprehension Dataset From Examinations , author=. 2017 , eprint=

    2017

  91. [99]

    2017 , eprint=

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , author=. 2017 , eprint=

    2017

  92. [100]

    2019 , eprint=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. 2019 , eprint=

    2019

  93. [101]

    2023 , eprint=

    Instruction-Following Evaluation for Large Language Models , author=. 2023 , eprint=

    2023

  94. [102]

    2021 , eprint=

    Measuring Massive Multitask Language Understanding , author=. 2021 , eprint=

    2021

  95. [103]

    2025 , eprint=

    ProcessBench: Identifying Process Errors in Mathematical Reasoning , author=. 2025 , eprint=

    2025

  96. [104]

    2022 , eprint=

    Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them , author=. 2022 , eprint=

    2022

  97. [105]

    2024 , eprint=

    ODIN: Disentangled Reward Mitigates Hacking in RLHF , author=. 2024 , eprint=

    2024

  98. [106]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  99. [107]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  100. [108]

    2023 , eprint=

    LLaMA: Open and Efficient Foundation Language Models , author=. 2023 , eprint=

    2023

  101. [109]

    5 Technical Report , author=

    Qwen2. 5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    arXiv

  102. [110]

    International conference on machine learning , pages=

    On variational bounds of mutual information , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  103. [111]

    International conference on artificial intelligence and statistics , pages=

    Toward fairness in text generation via mutual information minimization based on importance sampling , author=. International conference on artificial intelligence and statistics , pages=. 2023 , organization=

    2023

  104. [112]

    Science , volume=

    Semantics derived automatically from language corpora contain human-like biases , author=. Science , volume=. 2017 , publisher=

    2017

  105. [113]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    Gender-preserving Debiasing for Pre-trained Word Embeddings , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

  106. [114]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , year=

    Towards Debiasing Sentence Representations , author=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , year=

  107. [115]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    The generation gap: Exploring age bias in the value systems of large language models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  108. [116]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

    Improving gender fairness of pre-trained language models without catastrophic forgetting , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

  109. [117]

    Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) , pages=

    2025

  110. [118]

    Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long a...

    2019

  111. [119]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

    Improving Disentangled Text Representation Learning with Information-Theoretic Guidance , author=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages=

  112. [120]

    Findings of the Association for Computational Linguistics , year=

    Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game , author=. Findings of the Association for Computational Linguistics , year=

  113. [121]

    International Conference on Artificial Intelligence and Statistics , pages=

    A general theoretical paradigm to understand learning from human preferences , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=

    2024

  114. [122]

    International Conference on Machine Learning , pages=

    Scaling laws for reward model overoptimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  115. [123]

    arXiv preprint arXiv:2310.02743 , year=

    Reward model ensembles help mitigate overoptimization , author=. arXiv preprint arXiv:2310.02743 , year=

    arXiv

  116. [124]

    arXiv preprint arXiv:2406.10162 , year=

    Sycophancy to subterfuge: Investigating reward-tampering in large language models , author=. arXiv preprint arXiv:2406.10162 , year=

    arXiv

  117. [125]

    From Lists to Emojis: How Format Bias Affects Model Alignment

    Zhang, Xuanchang and Xiong, Wei and Chen, Lichang and Zhou, Tianyi and Huang, Heng and Zhang, Tong. From Lists to Emojis: How Format Bias Affects Model Alignment. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2...

    2025 doi

  118. [126]

    Advances in neural information processing systems , volume=

    The im algorithm: a variational approach to information maximization , author=. Advances in neural information processing systems , volume=

  119. [127]

    2020 , eprint=

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models , author=. 2020 , eprint=

    2020

  120. [128]

    2025 , eprint=

    From Lists to Emojis: How Format Bias Affects Model Alignment , author=. 2025 , eprint=

    2025

  121. [129]

    arXiv preprint arXiv:2408.08656 , year=

    Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms , author=. arXiv preprint arXiv:2408.08656 , year=

    arXiv

  122. [130]

    AutoDebias: Learning to Debias for Recommendation , url=

    Chen, Jiawei and Dong, Hande and Qiu, Yang and He, Xiangnan and Xin, Xin and Chen, Liang and Lin, Guli and Yang, Keping , year=. AutoDebias: Learning to Debias for Recommendation , url=. doi:10.1145/3404835.3462919 , booktitle=

  123. [131]

    Causal-Debias: Unifying Debiasing in Pretrained Language Models and Fine-tuning via Causal Invariant Learning

    Zhou, Fan and Mao, Yuzhou and Yu, Liu and Yang, Yi and Zhong, Ting. Causal-Debias: Unifying Debiasing in Pretrained Language Models and Fine-tuning via Causal Invariant Learning. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2023 doi

  124. [132]

    2020 , eprint=

    Learning from Failure: Training Debiased Classifier from Biased Classifier , author=. 2020 , eprint=

    2020

  125. [133]

    2021 , booktitle=

    FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders , author=. 2021 , booktitle=

    2021

  126. [134]

    2019 , eprint=

    Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual , author=. 2019 , eprint=

    2019

  127. [135]

    Language (Technology) is Power: A Critical Survey of ``Bias'' in NLP

    Blodgett, Su Lin and Barocas, Solon and Daum \'e III, Hal and Wallach, Hanna. Language (Technology) is Power: A Critical Survey of ``Bias'' in NLP. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.485

    2020 doi

  128. [136]

    Advances in neural information processing systems , volume=

    Debiased contrastive learning , author=. Advances in neural information processing systems , volume=

  129. [137]

    ACM Transactions on Recommender Systems , volume=

    Debiased representation learning in recommendation via information bottleneck , author=. ACM Transactions on Recommender Systems , volume=. 2023 , publisher=

    2023

  130. [138]

    EnD: Entangling and Disentangling deep representations for bias correction , url=

    Tartaglione, Enzo and Barbano, Carlo Alberto and Grangetto, Marco , year=. EnD: Entangling and Disentangling deep representations for bias correction , url=. doi:10.1109/cvpr46437.2021.01330 , booktitle=

    2021

  131. [139]

    2016 , eprint=

    Concrete Problems in AI Safety , author=. 2016 , eprint=

    2016

  132. [140]

    Noise reduction in speech processing , pages=

    Pearson correlation coefficient , author=. Noise reduction in speech processing , pages=. 2009 , publisher=

    2009

  133. [141]

    Borgwardt and Malte J

    Arthur Gretton and Karsten M. Borgwardt and Malte J. Rasch and Bernhard Sch. A Kernel Two-Sample Test , journal =. 2012 , volume =

    2012

  134. [142]

    The annals of probability , pages=

    I-divergence geometry of probability distributions and minimization problems , author=. The annals of probability , pages=. 1975 , publisher=

    1975

  135. [143]

    2024 , eprint=

    UltraFeedback: Boosting Language Models with Scaled AI Feedback , author=. 2024 , eprint=

    2024

  136. [144]

    2021 , eprint=

    MINE: Mutual Information Neural Estimation , author=. 2021 , eprint=

    2021

  137. [145]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Multi-view information-bottleneck representation learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  138. [146]

    arXiv preprint arXiv:2002.07017 , year=

    Learning robust representations via multi-view information bottleneck , author=. arXiv preprint arXiv:2002.07017 , year=

    2002 arXiv

  139. [147]

    Journal of Statistical Mechanics: Theory and Experiment , volume=

    On the information bottleneck theory of deep learning , author=. Journal of Statistical Mechanics: Theory and Experiment , volume=. 2019 , publisher=

    2019

  140. [148]

    2025 , eprint=

    APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport , author=. 2025 , eprint=

    2025

  141. [149]

    2024 , eprint=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. 2024 , eprint=

    2024

  142. [150]

    Disentangling Length from Quality in Direct Preference Optimization

    Park, Ryan and Rafailov, Rafael and Ermon, Stefano and Finn, Chelsea. Disentangling Length from Quality in Direct Preference Optimization. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.297

    2024 doi

  143. [151]

    2025 , eprint=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=

    2025

  144. [152]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  145. [153]

    arXiv preprint arXiv:2401.16335 , year=

    Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf , author=. arXiv preprint arXiv:2401.16335 , year=

    arXiv

  146. [154]

    arXiv preprint arXiv:2503.14476 , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    arXiv

  147. [155]

    arXiv preprint arXiv:2507.01352 , year=

    Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy , author=. arXiv preprint arXiv:2507.01352 , year=

    arXiv

  148. [156]

    2025 , eprint=

    Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators , author=. 2025 , eprint=

    2025

  149. [157]

    2023 , eprint=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. 2023 , eprint=

    2023

  150. [158]

    2024 , eprint=

    RewardBench: Evaluating Reward Models for Language Modeling , author=. 2024 , eprint=

    2024

  151. [159]

    2023 , eprint=

    Qwen Technical Report , author=. 2023 , eprint=

    2023

  152. [160]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  153. [161]

    Statistica Sinica , pages=

    Statistical applications of the Poisson-binomial and conditional Bernoulli distributions , author=. Statistica Sinica , pages=. 1997 , publisher=

    1997

  154. [162]

    Springer-Verlag, New York , year=

    Markov chains , author=. Springer-Verlag, New York , year=

  155. [163]

    The Bell system technical journal , volume=

    A mathematical theory of communication , author=. The Bell system technical journal , volume=. 1948 , publisher=

    1948

  156. [164]

    2025 , eprint=

    SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning , author=. 2025 , eprint=

    2025

  157. [165]

    2025 , eprint=

    Enhancing Human-Like Responses in Large Language Models , author=. 2025 , eprint=

    2025

  158. [166]

    2021 , eprint=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. 2021 , eprint=

    2021

  159. [167]

    2022 , eprint=

    DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale , author=. 2022 , eprint=

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.