Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A per-reward normalization makes GRPO ignore no objective

desk verdict Useful, simple fix for variance-driven reward hacking in multi-objective GRPO, but the 'equal contribution' theory is overstated; worth serious peer review. read the letter →

arxiv 2509.22047 v3 pith:6DVK2FC5 submitted 2025-09-26 cs.LG

Yuki Ichihara , Yuu Jinnai , Tetsuro Morimura , Mitsuki Sakamoto , Ryota Mitsuhashi , Eiji Uchibe This is my paper · ORCID
classification cs.LG
keywords grouprelativepolicyoptimizationmulti-objectivereinforcementlearningrewardhackingnormalizationadvantagefunctionlanguagemodelalignmentmachinetranslationRLHF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies a previously unreported failure mode in Group Relative Policy Optimization (GRPO): when multiple reward functions are combined into a single advantage, the reward with the largest variance dominates the update, so the policy optimizes it at the expense of the others. The authors propose MO-GRPO, which standardizes each reward function separately with its own mean and standard deviation before summing, so every objective contributes equally to the loss. They prove this equal-contribution property in the large-group limit and show the method's preference ordering is invariant under positive affine rescaling of rewards. In experiments on multi-armed bandits, simulated control, machine translation, and instruction following, MO-GRPO avoids the reward-hacking failures of GRPO and improves task metrics.

What carries the argument

The central object is the per-reward normalized advantage, $A^{\mathrm{MO}}_g = \sum_{i=1}^K \frac{R_i(q,o_g) - \mathrm{mean}_o\{R_i(q,o)\}}{\mathrm{std}_o\{R_i(q,o)\}}$, which replaces GRPO's single advantage computed from the unnormalized sum of rewards. This z-score transform equalizes the scale of every reward signal before aggregation, so that the gradient update no longer favors high-variance objectives. The proofs of Theorems 1 and 2 work by computing the correlation coefficient between each reward and the advantage in the $G\to\infty$ limit, where sample statistics equal population values.

What would settle it

Train MO-GRPO with a small group size, for example $G=2$ or $G=4$, on the multi-armed bandit task with rewards of standard deviations 10, 1, and 0.1: if the low-variance reward's contribution to the advantage is measurably weaker than high-variance rewards and the policy stops optimizing it, the equal-contribution property fails outside the asymptotic limit.

Watch

Extended reading notes

Core claim

The central claim is that GRPO's advantage function is biased toward high-variance reward functions, and that normalizing each reward individually before summing removes this bias. For GRPO, the correlation between reward $i$ and the advantage is $\sigma_i^2/(\sigma\,\sigma_i)$ plus a covariance term, so larger-$\sigma$ rewards dominate (Theorem 1). For MO-GRPO, where the advantage is the sum of per-reward z-scores, the correlation is $(1+Z)/\sqrt{K+Y}$, which equals $1/\sqrt{K}$ when rewards are uncorrelated (Theorem 2 and Corollary 1). The paper further proves that MO-GRPO's preference ordering is invariant under positive affine transformations of the rewards, whereas GRPO's is not (Propositions 1 and 2). The experiments demonstrate that this equal-contribution property prevents degenerate policies such as a translator that stops producing Japanese text to inflate a readability score.

Load-bearing premise

The equal-contribution theorems assume the group of sampled outputs is large enough that the sample mean and standard deviation of each reward equal their true values, while every experiment uses only eight samples per group.

Editorial extensions

If this is right

  • No manual scaling of reward functions is needed; off-the-shelf reward models with different value ranges can be combined directly.
  • Low-variance objectives are no longer ignored during training, preventing single-objective reward hacking in multi-objective tasks.
  • The method preserves the ordering of preferences under rescaling of any reward, so reward engineering does not silently change the learned policy.
  • MO-GRPO is orthogonal to other GRPO improvements and can be combined with them, since it only changes how the advantage is computed.
  • In language tasks, the method prevents degenerate outputs such as emitting non-target-language text to game a readability metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equal-contribution property likely degrades gracefully with small group sizes: at $G=8$ the variance estimates are noisy, so a reproducible diagnostic would be to measure the empirical correlation between each reward and the advantage during training and compare it to $1/\sqrt{K}$.
  • The same per-reward z-score idea could generalize to weighted multi-objective settings, where a user specifies relative importance and the normalization removes only the scale confound.
  • Because the paper's proof assumes uncorrelated rewards for the clean $1/\sqrt{K}$ value, strongly correlated objectives may need additional decorrelation; a natural extension would pair MO-GRPO with reward whitening.
  • If the method holds in other domains, it suggests that much of reward hacking attributed to reward misspecification is actually an artifact of scale imbalance, not of the reward models themselves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies reward hacking in GRPO when multiple reward functions are used. It argues that GRPO's advantage function is dominated by high-variance rewards, and proposes MO-GRPO, which standardizes each reward separately before summing the standardized values into the advantage. The authors prove correlation formulas for the two advantage constructions (Theorems 1 and 2), note that equal correlations hold exactly only under uncorrelated rewards (Corollary 1), and prove invariance of MO-GRPO's preference ordering under positive affine reward transformations (Proposition 1). They evaluate the method on a multi-armed bandit, a simulated control task (mo-reacher-v5), WMT machine translation with readability metrics, and AlpacaFarm instruction following, reporting that MO-GRPO outperforms GRPO and Dr. GRPO in most settings and avoids the reward-hacking failure mode in which GRPO optimizes readability at the cost of translation quality.

Significance. If the results are taken at face value, the contribution is practically useful: MO-GRPO is a simple, hyperparameter-free modification of GRPO, and the empirical evaluation covers four domains with reasonable breadth. The paper ships reproducible hyperparameters, prompts for the GPT-Eval judge, and a promise of open-sourced code, which strengthens the empirical portion. The headline theoretical claim, however, is currently overstated: the theorems prove statements about correlation with the advantage, not about each reward's contribution to the policy update, and the exact equal-correlation statement is restricted to mutually uncorrelated rewards. With a corrected and weakened theoretical claim, the empirical story would still be of interest to the RLHF and multi-objective RL communities.

major comments (3)
  1. [Sec. 4, Eq. (6) and Corollary 1] The statement of Theorem 2, 'the correlation ... remains constant', is false as written. Equation (6) gives Corr(R_i,A^MO) = (1+Z)/sqrt(K+Y), where Z = sum_{j≠i} Cov(R_i,R_j)/(σ_i σ_j) depends on i. Unless all these cross-covariance sums are equal, the correlation differs across reward functions. The only case in which the correlation is exactly constant is the mutually uncorrelated case of Corollary 1. In the WMT experiments, BLEURT and jReadability (or TRank) are plausibly correlated, so the equal-correlation guarantee is not established for the actual experimental setting. Please correct the theorem statement, state the constant value only under the uncorrelatedness assumption, and, if the equal-correlation claim is needed, provide bounds or measurements of Z for the experimental rewards.
  2. [Abstract, Sec. 1, and Sec. 4: 'contribute evenly to the loss function'] The central claim that MO-GRPO 'ensures that all reward functions contribute evenly to the loss function' or 'contribute equally to updating the policy' does not follow from Theorems 1 and 2. The relevant gradient term for reward i is E[ (π_θ/π_θref) z_i grad log π_θ / |o| ], whose magnitude is governed by Cov(z_i, (π_θ/π_θref) grad log π_θ / |o|), not by Corr(R_i, A^MO). A reward component that is independent of the output has zero gradient contribution even though its correlation with A^MO remains 1/sqrt(K) under the assumptions of Corollary 1. Thus the theorems measure linear association with the advantage, not influence on the policy update. To support the headline claim, the paper needs a result directly on the gradient decomposition, or the abstract and Section 1 must be weakened to 'equal correlation with the advantage'.
  3. [Sec. 5 and Table 12 (finite group size)] Theorems 1 and 2 assume G→∞, but all experiments use G=8. MO-GRPO's per-reward standardization uses sample means and standard deviations from groups of eight outputs; the paper does not analyze how estimation noise in these statistics affects the equal-correlation property or the gradient contribution. Since the method's practical benefit depends on the normalization behaving well at small G, the authors should either provide a finite-G analysis (e.g., bounds on the variance of the normalized advantage) or an empirical sensitivity study varying G. This is a load-bearing limitation because the theoretical result is purely asymptotic.
minor comments (4)
  1. [Appendix C, Theorem 3] The proof of Theorem 3 begins with 'We assume they are uncorrelated', but the theorem statement itself does not state this assumption. The theorem should either include the uncorrelatedness condition or the proof should carry the covariance terms through.
  2. [Appendix E.2, proof of Theorem 2] The derivation contains a garbled line: '= σ_i P_{j=K} Cov(R_i,R_j)/(σ_i σ_j)' appears to be missing the j≠i restriction in the summation index, and Eq. (36) has an unbalanced parenthesis in 'std(R_j'.
  3. [Sec. 4, Theorem 2 wording] The phrase 'for any o_g' is ambiguous: the correlation is a population quantity over the group of outputs, not a property of a single fixed output. Consider rephrasing to 'over the group distribution'.
  4. [Table 4 and surrounding text] The caption says 'higher is better' but TRank is lower-better, and the dagger/star symbols are explained only in the body text. Please add a footnote or caption note clarifying the direction of each metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MO-GRPO's theorems are derived from its stated definitions, and the main empirical claims are checked with an external judge not used in training.

full rationale

I walked the derivation chain from Eq. (2) and Eq. (5). Theorem 1 and Theorem 2 are obtained by substituting the respective advantage definitions into Cov(R_i, A)/sqrt(Var(R_i) Var(A)) and simplifying under G→∞; the covariance algebra is the derivation, and no parameter is fitted to data to obtain these identities. Corollary 1 follows by setting Z=0 and is stated with the required uncorrelated-reward assumption. Proposition 1 is an elementary affine-invariance calculation that follows from the definition of MO-GRPO. The empirical evaluation compares methods on GPT-Eval (Appendix F prompt), an LLM-based judge not used as a training objective, so the headline result is not forced by the training loss. The remaining weaknesses—finite G=8 experiments, the fact that equal correlation does not by itself prove equal contribution to the policy gradient, and correlated rewards in the MT setup—are limitations or gaps in the argument, not circular reductions. There is no self-citation chain on which the load-bearing claim depends, and no fitted input is relabeled as a prediction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two assumptions: the infinite-sample limit used in the correlation theorems, and the interpretation of correlation between reward and advantage as the reward's contribution to the policy loss. The method itself introduces no fitted parameters; the only hand-tuned element is the sqrt(K) rescaling in the implementation.

free parameters (1)
  • 1/sqrt(K) output scaling = 1/sqrt(2) for K=2
    The theoretical advantage in Eq. (5) is a sum of K normalized rewards with variance K. The practical implementation in Appendix J divides the sum by sqrt(2) (np.sqrt(2)) to keep the advantage variance near 1. This factor is chosen by hand to match GRPO's scale and is not part of the theoretical analysis.
assumptions (3)
  • domain assumption The group size G is large enough that sample means and standard deviations equal population values (G→∞).
    Theorems 1 and 2 (Section 3 and 4) and their proofs in Appendix E assume G→∞, but all experiments use G=8 (Table 12). The finite-sample behavior is not analyzed.
  • ad hoc to paper The correlation between a reward function and the advantage function is a valid measure of that reward's influence on the policy update.
    The paper argues that GRPO 'learns to optimize reward functions with higher variances' based on Theorem 1's correlation result. This equates correlation with influence on the loss, which is plausible but not proven; the policy gradient also depends on the log-probability and the clipping.
  • domain assumption Each reward function has finite and nonzero variance within a group, so the per-reward normalization is well defined.
    MO-GRPO divides by std_o(R_i), which requires a positive standard deviation. The code adds 1e-6 to avoid division by zero, but constant rewards within a group would produce noisy advantages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems." pith.science (2026). https://pith.science/paper/6DVK2FC5

@misc{pith2026250922047,
  author       = {Pith},
  title        = {Pith review of: MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DVK2FC5}},
  note         = {Machine review of arXiv:2509.22047}
}
read the original abstract

Group Relative Policy Optimization (GRPO) has been shown to be an effective algorithm when an accurate reward model is available. However, such a highly reliable reward model is not available in many real-world tasks. In this paper, we particularly focus on multi-objective settings, in which we identify that GRPO is vulnerable to reward hacking, optimizing only one of the objectives at the cost of the others. To address this issue, we propose MO-GRPO, an extension of GRPO with a simple normalization method to reweight the reward functions automatically according to the variances of their values. We first show analytically that MO-GRPO ensures that all reward functions contribute evenly to the loss function while preserving the order of preferences, eliminating the need for manual tuning of the reward functions' scales. Then, we evaluate MO-GRPO experimentally in four domains: (i) the multi-armed bandits problem, (ii) simulated control task (Mo-Gymnasium), (iii) machine translation tasks on the WMT benchmark (En-Ja, En-Zh), and (iv) instruction following task. MO-GRPO achieves stable learning by evenly distributing correlations among the components of rewards, outperforming GRPO, showing MO-GRPO to be a promising algorithm for multi-objective reinforcement learning problems.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A potential-based prediction reward collapses GRPO-trained LLM agents into a predictable 'dark room' state, and the collapse is caused by GRPO's std normalization rather than by the reward's magnitude.

Reference graph

Works this paper leans on

35 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ond r ej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espa \ n a-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, ...

    2021

  4. [4]

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete P roblems in AI S afety. arXiv preprint arXiv:1606.06565

    arXiv 2016

  5. [5]

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. 2023. https://openreview.net/forum?id=m7p5O7zblY RAFT : R eward ranked F inetuning for G enerative F oundation M odel A lignment . Transactions on Machine Learning Research

    work page 2023

  6. [6]

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/5fc47800ee5b30b8777fdd30abcaaf3b-Paper-Conference.pdf Alpaca F arm: A S imulation F ramework for M ethods that L earn from H uman F eedback . In Adv...

    work page 2023

  7. [7]

    Alegre, Ann Nowe, Ana Bazzan, El Ghazali Talbi, Gr\' e goire Danoy, and Bruno C

    Florian Felten, Lucas N. Alegre, Ann Nowe, Ana Bazzan, El Ghazali Talbi, Gr\' e goire Danoy, and Bruno C. da Silva. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/4aa8891583f07ae200ba07843954caeb-Paper-Datasets_and_Benchmarks.pdf A toolkit for reliable benchmarking and research in multi-objective reinforcement learning . In Advances in N...

    work page 2023

  8. [8]

    Xiao Feng, Bo Han, Zhanke Zhou, Jiaqi Fan, Jiangchao Yao, Ka Ho Li, Dahai Yu, and Michael Ng. 2025. https://openreview.net/forum?id=OWDBiMKYdo Dy PO : D ynamic P olicy O ptimization for M ulti- T urn I nteractive R easoning . In ICML 2025 Workshop on Programmatic Representations for Agent Learning

    work page 2025

Show all 35 references
  1. [9]

    Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. https://doi.org/10.18653/v1/2023.wmt-1.51 Results of ...

    2023 doi

  2. [10]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2/ Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics...

    2022

  3. [11]

    Leo Gao, John Schulman, and Jacob Hilton. 2023. https://proceedings.mlr.press/v202/gao23h.html Scaling L aws for R eward M odel O veroptimization . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, ...

    2023

  4. [12]

    Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. 2020. https://openreview.net/forum?id=HJgEMpVFwB Adversarial policies: Attacking deep reinforcement learning . In International Conference on Learning Representations

    2020

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The L lama 3 H erd of M odels. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Yoichiro Hasebe and Jae-Ho Lee. 2015. Introducing a readability evaluation system for J apanese language education. In Proceedings of the 6th international conference on computer assisted systems for teaching & learning Japanese, pages 19--22

    2015

  7. [15]

    Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, and Jinyoung Yeo. 2025. https://aclanthology.org/2025.acl-long.649/ Rethinking R eward M odel E valuation T hrough the L ens of R eward O veroptimization . In Proceedings of the 63rd Annual Meeting of the Ass...

    2025

  8. [16]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, ...

    2024 doi

  9. [17]

    Moxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi, Zhuo Liu, Fuli Feng, and Tat-Seng Chua. 2025. https://aclanthology.org/2025.findings-acl.574/ Self- I mprovement T owards P areto O ptimality: M itigating P reference C onflicts in M ulti- O bjective A lignment . In Findings of...

    2025

  10. [18]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  11. [19]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783

    2025 arXiv

  12. [20]

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. https://openreview.net/forum?id=JYtwGwIL7ye The effects of reward misspecification: Mapping and mitigating misaligned models . In International Conference on Learning Representations

    2022

  13. [21]

    Bradley Knox, Chelsea Finn, and Scott Niekum

    Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W. Bradley Knox, Chelsea Finn, and Scott Niekum. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/e45caa3d5273d105b8d045e748636957-Paper-Conference.pdf Scaling laws for reward model over...

    2024

  14. [22]

    Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. 2025. Magistral. arXiv preprint arXiv:2506.10910

    2025 arXiv

  15. [23]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : L earning R obust M etrics for T ext G eneration . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online...

    2020 doi

  16. [24]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: P ushing the L imits of M athematical R easoning in O pen L anguage M odels. arXiv preprint arXiv:2402.03300

    2024 arXiv

  17. [25]

    Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2023. Loose lips sink ships: M itigating L ength B ias in R einforcement L earning from H uman F eedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, pa...

    2023

  18. [26]

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. https://openreview.net/forum?id=G8LaO1P0xv A L ong W ay to G o: I nvestigating L ength C orrelations in RLHF . In First Conference on Language Modeling

    2024

  19. [27]

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/3d719fee332caa23d5038b8a90e81796-Paper-Conference.pdf Defining and C haracterizing R eward G aming . In Advances in Neural Information Proces...

    2022

  20. [28]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf Learning to summarize with human fee...

    2020

  21. [29]

    Mykola Trokhymovych, Indira Sen, and Martin Gerlach. 2024. https://doi.org/10.18653/v1/2024.acl-long.342 An O pen M ultilingual S ystem for S coring R eadability of W ikipedia . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume ...

    2024 doi

  22. [30]

    Changyi Xiao, Mengdi Zhang, and Yixin Cao. 2025. BNPO : B eta N ormalization P olicy O ptimization. arXiv preprint arXiv:2506.02864

    2025 arXiv

  23. [31]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  24. [32]

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. 2025. Group S equence P olicy O ptimization. arXiv preprint arXiv:2507.18071

    2025 arXiv

  25. [33]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Pap...

    2023

  26. [34]

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.630 Beyond O ne- P reference- F its- A ll A lignment: M ulti- O bjective D irect P reference O ptimization . In Findings of the Association ...

    2024 doi

  27. [35]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2020 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.