Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Accelerating RLHF Training with Reward Variance Increase

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reward adjustment that provably raises reward variance makes GRPO-based RLHF training faster.

desk verdict A genuine O(n log n) global algorithm for the reward-adjustment polytope is the real contribution here; the claimed GRPO speedup is asserted rather than derived. read the letter →

arxiv 2505.23247 v2 pith:XKO2JZX7 submitted 2025-05-29 cs.LG cs.AImath.OC

Zonglin Yang , Zhexuan Gu , Houduo Qi , Yancheng Yuan This is my paper
classification cs.LGcs.AImath.OC MSC 90C26
keywords RLHFGRPOrewardvarianceadjustmentnonconvexoptimizationextremepointcharacterizationLLMalignmentO(nlogn)algorithm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a cheap preprocessing step on reward scores can accelerate the reinforcement-learning-from-human-feedback (RLHF) phase of large language model training. For each prompt, it replaces the group of raw reward scores with the maximally spread-out scores that still have the same mean and the same preference ranking as the originals; this provably increases the reward variance of the initial policy over the response space. Because a recent theoretical result ties higher initial reward variance to faster optimization, the paper inserts these adjusted scores into the GRPO training loop and calls the result GRPOVI. Experiments on a Pythia model with two learned reward models show GRPOVI reaching higher ground-truth reward after the same number of training steps as standard GRPO, at negligible extra cost per iteration. If correct, this gives a practical way to convert the variance-speed link into a drop-in training speedup.

What carries the argument

The load-bearing object is the reward adjustment model (3): maximize $\sum_{i=1}^n p_i z_i^2$ over adjusted scores $z_i$, where $p_i=\pi_{\theta(0)}(y_i|x)$ are the initial policy's probabilities for the sampled responses, subject to $m\le z_i\le M$, $\sum_i p_i z_i=\sum_i p_i r_i$, and $z_1\ge\cdots\ge z_n$ when rewards are sorted descending. The objective is exactly the variance term of the initial policy restricted to the sampled responses, and the constraints encode expectation preservation and preference preservation. Lemma 2 identifies every extreme point of this feasible set as a step vector of the form $(M,\ldots,M,\alpha,\ldots,\alpha,m,\ldots,m)$, which reduces the global search to choosing two block boundaries $k,l$; Algorithm 2 walks those boundaries inward and provably returns the global optimum. In GRPOVI, these adjusted scores $\tilde r_i$ replace the raw rewards in the GRPO advantage normalization, and the variance increase is anchored to the initial policy via $p_i$, so it holds even when responses are generated by later policies.

What would settle it

Run GRPO on two reward preprocessings that give the same mean and ranking per prompt: the variance-maximizing adjusted scores from model (3), and a deliberate variance-minimizing or shuffled control. If the control trains as fast as GRPOVI, or if GRPOVI's speed advantage disappears when the reward bounds $[m,M]$ are made very tight, then the variance-increase mechanism is not what drives the observed speedup.

Watch

Extended reading notes

Core claim

The central discovery is that maximizing the weighted sum of squared adjusted rewards under three constraints—preserving the weighted mean reward, keeping adjusted scores within the reward bounds, and preserving the ordering of responses by reward—always increases the variance of the initial policy's reward distribution over the response space, while leaving the reward expectation and all pairwise preference comparisons unchanged. The maximization is a nonconvex problem, in general NP-hard to solve; the paper's key structural result is that every extreme point of the feasible polytope has a three-block form: some top responses get the maximum score M, some middle responses share a value α, and the remaining responses get the minimum score m, with α determined by the mean-preservation constraint. Since a convex objective attains its maximum over a polytope at an extreme point, this characterization turns the global search into a one-pass scan over block boundaries that runs in O(n) after sorting, giving an O(n log n) algorithm overall. Plugging the adjusted rewards into the GRPO advantage estimator defines the GRPOVI algorithm, and experiments report that it beats standard GRPO on training and test rewards across checkpoints.

Load-bearing premise

The speedup claim rests on assuming that the variance-increase guarantee, which is proven for the initial policy's reward distribution, transfers to faster GRPO convergence on the original reward objective when training is run on the adjusted three-valued rewards; the paper invokes this link without deriving it.

Editorial extensions

If this is right

  • GRPOVI reaches higher average ground-truth reward than standard GRPO after the same number of training steps, on both the training set and the test set, for both reward models tested.
  • The extra per-iteration cost is marginal: after sorting rewards, the one-pass search is $O(n)$, with $n$ the group size, so GRPOVI and GRPO have similar wall-clock per-iteration time.
  • Adjusted rewards take at most three distinct values per prompt, so the method effectively converts any dense reward model into a coarse positive/neutral/negative signal; the paper presents this as an indirect explanation for why rule-based rewards work well in GRPO systems such as DeepSeek-R1.
  • Because the probability weights are fixed at the initial policy, GRPOVI computes them once per response and does not need to recompute adjusted rewards under each updated policy.
  • The results provide practical evidence for the theoretical finding that higher initial reward variance over the response space shortens the time needed to raise expected reward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the three-block characterization suggests that the optimal adjusted reward for any prompt can be precomputed from the initial policy's probabilities and reward bounds, so the optimization could be replaced by a lookup or closed-form rule in production systems.
  • Beyond the paper, because the adjustment preserves only the expected reward and ranking, not higher moments, its benefit might depend on the reward bound width $[m,M]$; a testable extension is to compress or widen these bounds and measure whether GRPO's speedup changes accordingly.
  • Beyond the paper, the scheme modifies only the scalar rewards, so it could be composed with other RLHF objectives such as KL penalties or advantage clips; the paper does not test those interactions, but the same variance argument would apply if the adjusted rewards are used consistently.
  • Beyond the paper, a direct way to test the causal role of variance is an ablation that permutes reward scores within each group to lower variance while preserving mean and ranking; if such a control trains as fast as GRPOVI, variance increase would not be the operative mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a reward adjustment model: for a fixed prompt and a sampled response group, it maximizes a probability-weighted second moment of the adjusted rewards subject to box constraints, a sortedness constraint, and an expectation-preservation constraint. Theorem 1 shows that a global maximizer of this model increases the reward variance of the initial policy over the response space while preserving the reward expectation and the relative ordering inside the group. Lemma 2 characterizes all extreme points of the feasible set, and Algorithm 2 is proposed as an O(n) one-pass search (O(n log n) including sorting) for a global maximizer. Section 4 integrates this into GRPO as GRPOVI: at each training step, responses are sampled from the current policy, adjusted rewards are computed using initial-policy probabilities, and the adjusted rewards are used for advantage estimation. Experiments on Pythia with two training reward models compare GRPOVI with standard GRPO using the ArmoRM reward as a proxy.

Significance. If the advertised speedup were established, the contribution would be valuable: it offers a low-overhead, provably variance-increasing reward transformation that preserves preferences within a group, and the extreme-point characterization is a clean optimization result. The optimization core, specifically Theorem 1 and Lemma 2 for a fixed adjusted reward, appears sound, and the O(n log n) search is elegant and is supported by simulation comparisons against exhaustive enumeration. However, the link from initial-policy variance increase to faster GRPO on the original reward is not derived: the cited theorem applies to a fixed reward and is a lower bound, while GRPOVI optimizes a time-varying adjusted reward. The experiments are also small-scale and use only reward-model proxies. The paper therefore currently establishes a useful optimization method and a plausible empirical signal, but not the stated theoretical acceleration claim.

major comments (3)
  1. [Section 4, Algorithm 3] The sentence 'According to theorem 1 and [24, Theorem 4], the GRPOVI algorithm achieves faster RLHF training compared to the original GRPO algorithm' is not a valid derivation. [24, Theorem 4] concerns a fixed reward function and gives a lower bound on the time needed to increase the expected value of that fixed reward; a larger variance only makes that lower bound smaller and does not by itself guarantee that a particular algorithm runs faster. Moreover, GRPOVI does not optimize a fixed adjusted reward: in Algorithm 3 the responses are resampled from the current policy at every step and the adjusted rewards are recomputed from those samples, so the effective reward changes between steps and between batches. Theorem 1 and Corollary 1 only show that, for one fixed adjusted reward built from one response group, the initial-policy variance is larger; they do not apply to the time-varying reward sequence used by GRPOVI, nor do they compare convergence to the original reward. Please provide a rigorous analysis for the time-varying setting or explicitly reframe the contribution as an empirically demonstrated acceleration.
  2. [Section 3.4, Algorithm 2] The paper claims that Algorithm 2 finds a global solution of (3) in O(n) time, but no correctness theorem or proof is given. Lemmas 3-5 show that certain one-step moves (k to k+1 or l to l-1) strictly increase the objective when the recomputed alpha remains in [m,M], and Lemma 4 provides a stopping condition. However, the greedy choice in lines 16-26 between the left and right moves is not shown to lead to a global maximizer: the objective is not shown to be unimodal along the bidirectional path, and the proof does not rule out that a locally smaller move would enable a much larger gain later. The agreement with exhaustive enumeration in Table 1 covers only synthetic instances and is not a proof. Please add a correctness theorem for Algorithm 2, or state the additional conditions under which the greedy path is guaranteed to reach a global optimum.
  3. [Section 5.2] The empirical claim that GRPOVI 'significantly' improves training efficiency rests on limited evidence. The experiments use a small base model (Pythia), group size 8, a single pass over the training prompts, and only two training reward models; the only reported performance metric is the mean ArmoRM reward on the training and test sets, with no numerical table, effect sizes, or significance tests. Figures 1 and 2 show means with standard-deviation bands, but the text does not report the final reward values or the width of the bands, so the reader cannot assess whether the differences are statistically meaningful. There is also no comparison against the downsampling baseline of [35] or other variance-based methods. This does not block the optimization contribution, but it does not by itself establish the central acceleration claim; the experimental section should either be strengthened or the conclusions should be softened accordingly.
minor comments (5)
  1. [Lemma 2 proof] The phrase 'right hind side of (4)' should read 'right-hand side of (4)'.
  2. [Algorithm 2] The variable name 'f lags' appears to be a typo for 'flag'; also, the expression 'Scm' in line 13 should be written as 'S_C m' to avoid confusion with the product S_C times m.
  3. [Algorithm 3] Line 8 normalizes the probabilities before solving (3), whereas Theorem 1, Lemma 2, and Corollary 1 use unnormalized probabilities pi = pi(y_i|x). Please clarify that the normalized values are conditional probabilities under the initial policy and state explicitly why the expectation-preservation and variance-increase arguments remain valid in this parametrization.
  4. [Section 4, rule-based reward discussion] The 'indirect explanation' for the empirical effectiveness of rule-based rewards in DeepSeek-R1 is not empirically tested in this paper; consider labeling it clearly as a hypothesis or providing a supporting ablation.
  5. [Section 5.2.2] Please report the final checkpoint numbers and standard errors in a table or in the text, since the figures alone make it difficult for the reader to quantify the improvement claimed by the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the variance-increase theorem is proven from the model's own constraints, and the speedup claim rests on an external theorem, not a self-citation or fitted input.

full rationale

The paper's central mathematical claims are self-contained. The reward adjustment model (3) is an optimization problem over explicitly stated constraints, and Theorem 1 proves that any global solution increases the policy's reward variance while preserving expectation. The proof is algebraic and follows directly from the definition of variance and the constraints in (3), so it does not presuppose the conclusion. Lemma 2 characterizes the extreme points of the feasible polytope, Lemma 1 reduces the convex maximization to an extreme-point search, and Algorithm 2 is validated against exhaustive enumeration in Table 1; this chain is internally consistent and not circular. The claim that GRPOVI accelerates RLHF training relies on Theorem 1 together with [24, Theorem 4], an external result by Razin et al. This is a dependency on outside work, not a self-citation, and no fitted parameter is renamed as a prediction. The paper's 'indirect explanation' for rule-based rewards is explicitly presented as an observation from the three-valued structure of the adjusted rewards, not as a derived empirical law used as an input. The skeptical concern that the effective reward in Algorithm 3 is time-varying, while [24, Theorem 4] concerns a fixed reward, is a validity gap in the speedup argument, but it is not circularity: it does not reduce the paper's conclusion to its own assumptions by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the external variance-speed theorem [24] and on the transfer assumption from adjusted to original reward, not on fitted parameters or invented entities.

assumptions (4)
  • domain assumption Higher reward variance of the initial policy over the response space reduces the time needed to increase expected reward (from [24, Theorem 4]).
    Invoked in Sections 2 and 4 to conclude that increasing variance accelerates training; not derived in this paper.
  • domain assumption Training on the adjusted reward r' leads to higher original reward r, so the speedup on r' transfers to the true objective.
    Assumed in Section 4 and tested empirically in Section 5; no theorem connects optimization of r' to improvement of r.
  • domain assumption All sampled responses have positive probability under the initial policy, so the expectation-preservation constraint is meaningful.
    Required for Corollary 1 and the variance-increase proof; can be numerically fragile in LLMs.
  • standard math Standard convex analysis facts: a convex function attains its maximum over a polytope at an extreme point, and the feasible set is the convex hull of its vertices.
    Used in Lemma 1 and the representation theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerating RLHF Training with Reward Variance Increase." pith.science (2026). https://pith.science/paper/XKO2JZX7

@misc{pith2026250523247,
  author       = {Pith},
  title        = {Pith review of: Accelerating RLHF Training with Reward Variance Increase},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKO2JZX7}},
  note         = {Machine review of arXiv:2505.23247}
}
abstract

Reinforcement learning from human feedback (RLHF) is an essential technique for ensuring that large language models (LLMs) are aligned with human values and preferences during the post-training phase. As an effective RLHF approach, group relative policy optimization (GRPO) has demonstrated success in many LLM-based applications. However, efficient GRPO-based RLHF training remains a challenge. Recent studies reveal that a higher reward variance of the initial policy model leads to faster RLHF training. Inspired by this finding, we propose a practical reward adjustment model to accelerate RLHF training by provably increasing the reward variance and preserving the relative preferences and reward expectation. Our reward adjustment method inherently poses a nonconvex optimization problem, which is NP-hard to solve in general. To overcome the computational challenges, we design a novel $O(n \log n)$ algorithm to find a global solution of the nonconvex reward adjustment model by explicitly characterizing the extreme points of the feasible set. As an important application, we naturally integrate this reward adjustment model into the GRPO algorithm, leading to a more efficient GRPO with reward variance increase (GRPOVI) algorithm for RLHF training. As an interesting byproduct, we provide an indirect explanation for the empirical effectiveness of GRPO with rule-based reward for RLHF training, as demonstrated in DeepSeek-R1. Experiment results demonstrate that the GRPOVI algorithm can significantly improve the RLHF training efficiency compared to the original GRPO algorithm.

Figures

Figures reproduced from arXiv: 2505.23247 by the authors.

Figure 1
Figure 1. Comparison of GRPOVI and original GRPO algorithms aligned with GRM-Gemma-2-2B [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Comparison of GRPOVI and original GRPO algorithms aligned with GRM-Llama-3.2-3B [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 17 canonical work pages

  1. [24]

    Razin, Z

    N. Razin, Z. W ang, H. Strauss, S. Wei, J. D. Lee, and S. Arora, What Makes a Reward Model a Good Teacher? An Optimization Perspective, arXiv preprint arXiv:2503.15477, (2025). 19

    arXiv 2025

  2. [35]

    Y. E. Xu, Y. Savani, F. F ang, and Z. Kolter , Not All Rollouts are Useful: Down- Sampling Rollouts in LLM Reinforcement Learning , arXiv preprint arXiv:2504.13818, (2025)

    arXiv 2025

  3. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , GPT-4 Technical Report , arXiv preprint arXiv:2303.08774, (2023)

    arXiv 2023

  4. [2]

    Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. , Training a Helpful and Harmless Assistant with Rein- forcement Learning from Human Feedback, arXiv preprint arXiv:2204.05862, (2022)

    arXiv 2022

  5. [3]

    Biderman, H

    S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Halla- han, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. , Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, in International Conference on Machine Learning, PMLR, 2023, pp. 2397–2430

    work page 2023

  6. [4]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the Opportunities and Risks of Foundation Models, arXiv preprint arXiv:2108.07258, (2021)

    arXiv 2021

  7. [5]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, et al. , Language Models are Few-Shot Learners, Advances in Neural Information Processing Systems, 33 (2020), pp. 1877–1901

    work page 2020

  8. [6]

    Busa-Fekete, B

    R. Busa-Fekete, B. Sz¨or´enyi, P. Weng, W. Cheng, and E. H¨ullermeier, Preference- based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm, Machine Learning, 97 (2014), pp. 327–351

    work page 2014

Show all 43 references
  1. [7]

    Y. Chen, D. Zhu, Y. Sun, X. Chen, W. Zhang, and X. Shen , The Accuracy Paradox in RLHF: When Better Reward Models Don ’t Yield Better Language Models , arXiv preprint arXiv:2410.06554, (2024)

    2024 arXiv

  2. [8]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei , Deep reinforcement learning from human preferences , Advances in neural information processing systems, 30 (2017)

    2017

  3. [9]

    Chujie, S

    G. Chujie, S. Wu, Y. Huang, D. Chen, Q. Zhang, Z. Fu, Y. W an, L. Sun, and X. Zhang, HonestLLM: Toward an Honest and Helpful Large Language Model , Advances in Neural Information Processing Systems, 37 (2024), pp. 7213–7255

    2024

  4. [10]

    G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, et al., ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback, in Interna- tional Conference on Machine Learning, PMLR, 2024, pp. 9722–9744. 18

    2024

  5. [11]

    S. Dai, C. Xu, S. Xu, L. Pang, Z. Dong, and J. Xu , Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era , in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6437–6447

    2024

  6. [12]

    L. Gao, J. Schulman, and J. Hilton , Scaling Laws for Reward Model Overoptimization , in International Conference on Machine Learning, PMLR, 2023, pp. 10835–10866

    2023

  7. [13]

    Gr¨unbaum, V

    B. Gr¨unbaum, V. Klee, M. A. Perles, and G. C. Shephard , Convex Polytopes, vol. 16, Springer, 1967

    1967

  8. [14]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. W ang, X. Bi, et al. , Deepseek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv preprint arXiv:2501.12948, (2025)

    2025 arXiv

  9. [15]

    W. Ji, W. Yuan, E. Getzen, K. Cho, M. I. Jordan, S. Mei, J. E. Weston, W. J. Su, J. Xu, and L. Zhang , An Overview of Large Language Models for Statisticians , arXiv preprint arXiv:2502.17814, (2025)

    2025 arXiv

  10. [16]

    Lambert, V

    N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. , RewardBench: Evaluating Reward Models for Lan- guage Modeling, arXiv preprint arXiv:2403.13787, (2024)

    2024 arXiv

  11. [17]

    G. Li, M. Lin, T. Galanti, Z. Tu, and T. Yang , DisCO: Reinforcing Large Reason- ing Models with Discriminative Constrained Optimization , arXiv preprint arXiv:2505.12366, (2025)

    2025

  12. [18]

    A. Liu, B. Feng, B. Xue, B. W ang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. , DeepSeek-V3 Technical Report, arXiv preprint arXiv:2412.19437, (2024)

    2024 arXiv

  13. [19]

    R. Liu, T. R. Sumers, I. Dasgupta, and T. L. Griffiths, How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?, arXiv preprint arXiv:2402.07282, (2024)

    2024 arXiv

  14. [20]

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin , Understanding R1-Zero-Like Training: A Critical Perspective , arXiv preprint arXiv:2503.20783, (2025)

    2025 arXiv

  15. [21]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. W ainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. , Training language models to follow instruc- tions with human feedback , Advances in Neural Information Processing Systems, 35 (2022), pp. 27730–27744

    2022

  16. [22]

    Radford, K

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. , Improving Language Understanding by Generative Pre-Training, (2018)

    2018

  17. [23]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn , Direct Preference Optimization: Your Language Model is Secretly a Reward Model , Advances in Neural Information Processing Systems, 36 (2023), pp. 53728–53741

    2023

  18. [25]

    Schulman, P

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel , High-dimensional continuous control using generalized advantage estimation , arXiv preprint arXiv:1506.02438, (2015)

    2015 arXiv

  19. [26]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov , Proximal Policy Optimization Algorithms, arXiv preprint arXiv:1707.06347, (2017)

    2017 arXiv

  20. [27]

    Z. Shao, P. W ang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. , DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv preprint arXiv:2402.03300, (2024)

    2024 arXiv

  21. [28]

    Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. ´A. Pires, M. V alko, Y. Cheng, et al. , Understanding the performance gap between online and offline alignment algorithms , arXiv preprint arXiv:2405.08448, (2024)

    2024 arXiv

  22. [29]

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalk- wyk, A. M. Dai, A. Hauth, K. Millican, et al. , Gemini: A Family of Highly Capable Multimodal Models, arXiv preprint arXiv:2312.11805, (2023)

    2023 arXiv

  23. [30]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. , LLaMA: Open and Efficient Foundation Language Models, arXiv preprint arXiv:2302.13971, (2023)

    2023 arXiv

  24. [31]

    Y.-L. Tuan, X. Chen, E. M. Smith, L. Martin, S. Batra, A. Celikyilmaz, W. Y. W ang, and D. M. Bikel, Towards Safety and Helpfulness Balanced Responses via Control- lable Large Language Models , arXiv preprint arXiv:2404.01295, (2024)

    2024 arXiv

  25. [32]

    W ang, W

    H. W ang, W. Xiong, T. Xie, H. Zhao, and T. Zhang , Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts , arXiv preprint arXiv:2406.12845, (2024)

    2024 arXiv

  26. [33]

    T. Wu, J. Ni, B. Hooi, J. Zhang, E. Ash, S.-K. Ng, M. Sachan, and M. Leippold , Navigating the Helpfulness-Truthfulness Trade-Off with Uncertainty-Aware Instruction Fine- Tuning, arXiv preprint arXiv:2502.11962, (2025)

    2025 arXiv

  27. [34]

    P. Xu, F. Gao, and Q. Gu , Sample Efficient Policy Gradient Methods with Recursive Variance Reduction, in International Conference on Learning Representations, 2020

    2020

  28. [36]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. L v, et al., Qwen3 technical report, arXiv preprint arXiv:2505.09388, (2025)

    2025 arXiv

  29. [37]

    R. Yang, R. Ding, Y. Lin, H. Zhang, and T. Zhang, Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs , Advances in Neural Information Processing Systems, (2024)

    2024

  30. [38]

    Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. F an, G. Liu, L. Liu, X. Liu, et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, arXiv preprint arXiv:2503.14476, (2025). 20

    2025 arXiv

  31. [39]

    Zhang and C

    J. Zhang and C. Zuo, GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models , arXiv preprint arXiv:2504.09696, (2025)

    2025

  32. [40]

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. W ang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. , A Survey of Large Language Models , arXiv preprint arXiv:2303.18223, 1 (2023)

    2023 arXiv

  33. [41]

    Zheng, S

    R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. W ang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, et al. , Secrets of RLHF in Large Language Models Part I: PPO , arXiv preprint arXiv:2307.04964, (2023)

    2023 arXiv

  34. [42]

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving , Fine-Tuning Language Models from Human Preferences , arXiv preprint arXiv:1909.08593, (2019)

    2019 arXiv

  35. [43]

    P. B. Zwart, Global Maximization of a Convex Function with Linear Inequality Constraints , Operations Research, 22 (1974), pp. 602–609. 21

    1974

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.