Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

This paper argues that RL instability in large language models is governed by whether the training loss is convex in the model's logits, and that replacing PPO's clipped surrogate with logits-convex targets stabilizes and strengthens policy

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:45 UTC pith:Z54I7K4F

load-bearing objection Useful empirical recipe wrapped in a theory that doesn't cover the algorithm actually run; the fixed-target proofs are the main weakness. the 5 major comments →

arxiv 2603.00963 v2 pith:Z54I7K4F submitted 2026-03-01 cs.LG cs.CL

Stabilizing Policy Optimization via Logits Convexity

Hongzhan Chen , Tao Yang , Yuhua Zhu , Shiping Gao , Xiaojun Quan , Ting Yao This is my paper
classification cs.LG cs.CL
keywords logits convexitypolicy optimizationlarge language model RLPPO instabilityKL-regularized RLforward KLsample efficiencygradient directionality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to explain why reinforcement learning is so much less stable than supervised fine-tuning when training large language models, and to fix it. Its claim is that the deciding factor is 'logits convexity': whether the loss, viewed as a function of the pre-softmax logits, is convex. SFT's negative log-likelihood has this property; PPO's clipped surrogate does not, which the paper links to gradient explosions and training collapse. It then introduces Logits Convex Optimization (LCO), a family of losses that regress or distill toward the closed-form optimal policy and are convex in logits. If the argument is right, logits convexity is a design principle for stable LLM RL, and LCO should consistently beat PPO-family methods, which the reported experiments support.

Core claim

The central claim is that gradient-based policy optimization in LLMs is stabilized when the loss is convex with respect to the logits. Under a first-order linearization of the logits as a function of parameters (Assumption 4.2), logits convexity yields an inner-product inequality — the parameter-space gradient and the logit-space gradient align with the direction toward near-optimal parameters — so gradient descent is not drawn into spurious stationary points. The paper proves SFT's NLL loss is logits-convex, PPO's clipped surrogate is not, and the LCO losses (MSE, log-cosh, forward KL) are all logits-convex. The optimal target is derived in closed form: π* is proportional to π_old exp(A/β),

What carries the argument

The load-bearing object is 'logits convexity' — positive semi-definiteness of the loss's Hessian with respect to the pre-softmax logits. Proposition 4.4 is the mechanism: via a local first-order Taylor expansion of logits in the parameters, logits convexity transfers to a non-negative inner product between the parameter-space gradient and the displacement toward near-optimal parameters. The LCO objectives are constructed to make this property hold at every time step, and Proposition 4.7 converts it into gradient norms that shrink as the loss shrinks.

Load-bearing premise

The argument depends on Assumption 4.2: near the optimum, the network's logits must be an approximately linear (first-order) function of the parameters, so that convexity in logit space carries over to parameter-space gradient directionality — a regime real LLMs are typically far from.

What would settle it

Record both inner products in Equation (15) during an actual LCO training run near convergence; if at any step the parameter-space inner product ⟨∇θL, θ−θ*⟩ is clearly negative while the logit-space inner product ⟨∇zL, z−z*⟩ is non-negative, then the first-order linearization of logits has failed and Proposition 4.4 provides no guarantee at that point. Alternatively, exhibit one training state where the curvature of the PPO loss with respect to the logits in the non-clipped region is positive semidefinite, which would contradict Lemma 4.5's blanket non-convexity claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If logits convexity is the key to stable LLM RL, then SFT's stability is not incidental: it follows from the PSD Hessian of the cross-entropy loss, and any surrogate that lacks this property is prone to gradient spikes and collapse.
  • Gradient norms for LCO losses are bounded by monotone functions of the loss value, giving a self-stabilizing optimization trajectory that avoids the exploding gradients seen in PPO.
  • Because LCO matches the same closed-form optimum as the KL-regularized objective, it is consistent with PPO's goal while being more stable, and the reported results show it outperforms PPO, GRPO, DAPO, and GSPO across math reasoning, QA-Feedback, AlpacaEval, and OOD MMLU.
  • LCO also improves sample efficiency: LCO-KLD reaches PPO-level MATH500 performance in about half the samples, and LCO-LCH in about a third.
  • The specific target z* = z_old + A/β is justified as the shift that minimizes the initial logit-space discrepancy, giving the tightest convergence bound for the MSE and log-cosh variants when advantages are mean-centered.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's mechanism can be tested directly by watching the smallest Hessian eigenvalue of the policy loss with respect to logits during training; stable runs should keep it non-negative, and PPO-style runs should show negative directions exactly when gradient spikes occur.
  • Editorial extension: the closed-form optimal target is not tied to PPO's clipping heuristic, so any RL or distillation algorithm that matches this target with a convex loss — including GRPO/DAPO variants or online distillation — should inherit similar stability.
  • Editorial extension: the analysis suggests future RL losses could be designed by choosing a surrogate whose logit Hessian is PSD, making clipping and trust regions less necessary for stability and replacing them with target construction.
  • Editorial extension: since the directionality proof relies on a local linearization of logits in parameters, its predictions are most testable near convergence; comparing gradient-direction violations in early versus late training would reveal where the linearity assumption needs to be replaced.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies instability in RL fine-tuning of LLMs and proposes that convexity of the loss with respect to the final logits is a key stabilizing property. It observes that the SFT negative log-likelihood is logits-convex, whereas the clipped PPO surrogate is not, and uses this contrast to motivate a family of objectives called Logits Convex Optimization (LCO). Three LCO variants are defined: regression to the optimal logits z* = z_old + A/β via MSE or log-cosh, and forward KL alignment to the corresponding optimal policy π*. The theoretical sections claim that, under an NTK-style linearity assumption on logits as a function of parameters, logits convexity yields favorable parameter-space gradient directionality (Prop. 4.4), bounded and decreasing gradient norms (Prop. 4.7), and linear convergence for LCO-MSE/LCH (Props. 4.8–4.9). Experiments on math reasoning, QA-Feedback, and AlpacaEval 2.0 across several Qwen, Llama, and Mistral backbones report that LCO variants outperform PPO, GRPO, DAPO, and GSPO, with smoother gradient dynamics and better sample efficiency.

Significance. If the theoretical claims were fully supported, the paper would provide a useful and simple design principle for RL fine-tuning: choose surrogate losses that are convex in logit space. The empirical evaluation is a genuine strength: results are reported across multiple model families, tasks, and random seeds, and the LCO variants consistently beat strong baselines on several benchmarks. The paper also gives explicit, checkable derivations of logits convexity for SFT, PPO, and the proposed losses, and the proposed objectives are easy to implement. However, the load-bearing theoretical bridge from logits convexity to parameter-space stability rests on an unverified NTK-scale linearity assumption, and there is a mismatch between the fixed-target theory and the dynamic-target algorithm actually run in the experiments. The empirical findings may stand on their own, but the central theoretical guarantee as stated does not apply to the reported method.

major comments (5)
  1. [§4.3 / §6.1 / Appendix H.1] There is a mismatch between the theoretical setting and the deployed algorithm. Proposition 4.4 and Propositions 4.8–4.9 are derived for a fixed optimal target z* = z_old + A/β; Appendix H.1 explicitly identifies z_old as 'the initial logits.' In the experiments, however, §6.1 states that LCO 'facilitates a dynamic optimization process that continuously aligns with evolving targets.' With a moving target there is no fixed θ* toward which the gradient is directionally aligned, and the linear-convergence proofs of Props. 4.8–4.9 do not apply. The non-stationarity is not a minor presentational gap: it concerns the central claim that the reported method enjoys the proven stability guarantees. Please either extend the theory to a moving target, or restrict the guarantees to a regime in which targets are frozen and clearly state that the deployed algorithm is a heuristic extension.
  2. [§4.3, Eq. (14)] Assumption 4.2 is the only bridge from logits convexity to parameter-space gradient behavior, but it is asserted rather than verified. The first-order Taylor expansion z_θ* ≈ z_θ + ∇_θ z_θ^T (θ*−θ) with negligible higher-order terms is an NTK-regime condition. LLMs fine-tuned in practice are not in that regime; individual parameters can change enough that logit residuals are material, and no experiment in the paper checks the size of the residual. If Assumption 4.2 fails, Proposition 4.4 provides no parameter-space guarantee and Props. 4.8–4.9 do not follow. At minimum, the paper should state this as an idealized condition and provide an empirical diagnostic (e.g., measure the Taylor residual on actual checkpoints).
  3. [§4.3, Prop. 4.4 and definition of Θ*] The inequality in Eq. (15) is not valid as stated for the set Θ* defined in the text. The authors define Θ* = {θ* : L(z_θ*) − L* < ε}, i.e., θ* is only near-optimal. For a logits-convex L, the first-order characterization gives ⟨∇_z L, z−z*⟩ ≥ L(z) − L(z*) ≥ −ε, not necessarily ≥ 0. The claimed ≥0 requires z* to be an actual minimizer of L. This weakens the 'not misled by spurious stationary points' conclusion: the directional guarantee has an ε slack. The proof should either use exact minimizers or state the bound with the ε term.
  4. [§4.4, Eq. (11) and Table 5] The justification for the target z* = z_old + A/β as the unique minimizer of the convergence bound relies on the claim that advantage normalization makes the mean of A over the vocabulary zero. This is not true for the sparse sampled-action estimator in Eq. (11), where exactly one entry of A is nonzero and all others are zero, so the mean is A(s_t,a_t)/|V|, not zero. The sparse setting is explicitly used in the experiments reported in Table 5. The paper should either restrict the zero-mean claim to the dense advantage estimators, or account for the nonzero mean in the convergence-bound minimization.
  5. [§4.3, Prop. 4.7] Proposition 4.7 is used to support the claim that LCO 'effectively precludes the sudden, high-magnitude gradient spikes' seen in PPO. The bounds in Eqs. (16)–(18) are upper bounds involving σ_max, the maximum singular value of ∇_θ z_θ. If σ_max itself grows during training, the bound can grow even as the loss decreases. The statement that gradient norms 'scale with remaining error and diminish progressively' therefore requires an additional assumption that σ_max remains bounded. The experiments may show this empirically, but the theoretical claim as written is stronger than what Prop. 4.7 proves.
minor comments (5)
  1. [§1] Typo: 'proposeLogits Convex Optimization' should read 'propose Logits Convex Optimization'.
  2. [§4.3, Lemma 4.5] The statement that PPO is 'not logits convex at any time step' is too strong. The Hessian computation in Eqs. (39)–(42) is restricted to the active, unclipped region; in clipped regions the gradient is zero and the Hessian is PSD (zero). The correct statement is that the PPO surrogate is not convex in general on the active region; there exist states where the Hessian is indefinite.
  3. [Appendix D / §4.1] The text oscillates between calling z* 'a specific solution' and 'the unique optimal logits.' Because of softmax translation invariance, z* in Eq. (7) is one representative of an equivalence class. The later discussion in §4.4 acknowledges this, but the earlier wording should be made consistent.
  4. [§4.2, Eq. (13)] The DPO-based advantage estimator A(s_t,a_t) := log φ_DPO(a_t|s_t)/φ_ref(a_t|s_t) is presented as a direct consequence of the Bellman equation. The derivation is compressed; please spell out the assumptions on the reward model and token-level MDP so the reader can verify the identification.
  5. [§4.4] The phrase 'in modern reinforcement learning practice, it is standard to employ advantage normalization at each time step' conflates normalizing advantages over a batch of sampled actions with normalizing a dense advantage vector over the full vocabulary. These are different operations and only the latter yields mean zero over V.

Circularity Check

0 steps flagged

No significant circularity; logits-convexity results follow from stated definitions and assumptions, with one minor non-load-bearing self-citation and a non-circular fixed-vs-dynamic target gap.

full rationale

We examined the derivation chain for reductions in which a prediction equals its input by construction. Proposition 4.4 is a corollary of the first-order characterization of convexity plus Assumption 4.2's linearization; the gradient directionality inequality is exactly the convex gradient inequality, and the convergence bounds in Propositions 4.8-4.9 are standard gradient-descent bounds for the chosen targets. The LCO losses are convex by design (Lemma 4.6), but the paper does not present this as an empirical prediction; it verifies the design property and then tests the framework on external benchmarks (MATH500, QA-Feedback, AlpacaEval) with independent reward sources. The one self-citation, Chen et al. 2025b, supports the default dense advantage estimator, but the method is also evaluated with sparse and rule-based advantages, so the self-citation is not load-bearing. A more serious concern is internal consistency, not circularity: the convergence proofs (Appendix H.1) treat z_old as the initial logits and hence z* as fixed, while Section 6.1 states that "LCO facilitates a dynamic optimization process that continuously aligns with evolving targets." This means the fixed-target theorems do not directly cover the deployed algorithm; however, this is a validity/applicability gap, not an instance of the conclusion being assumed in the premise. No step of the paper's derivation reduces, by construction or by self-citation, to its own output.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The method rests on standard RL machinery (KL-regularized objective, closed-form optimal policy, advantage-weighted target regression) plus the paper-specific NTK-style linearization assumption and a per-token MDP decomposition. No new physical or mathematical entities are introduced; 'logits convexity' is a defined property of the loss, not an invented entity. The single tunable free parameter is β, with the DPO-derived advantage acting as an external fitted model.

free parameters (1)
  • β (KL penalty / target temperature) = 1.0
    Default hyperparameter in Section 5; the optimal target z* = z_old + A/β depends directly on it, so the trained policy and all benchmark scores depend on this hand-chosen value.
axioms (5)
  • ad hoc to paper First-order Taylor expansion of logits w.r.t. parameters holds for near-optimal LLM parameters (Assumption 4.2)
    Invoked in Prop 4.4 and the convergence analysis; no evidence is given that LLMs operate in the NTK regime, making this a load-bearing assumption specific to this paper.
  • domain assumption Autoregressive generation decomposes as a per-token MDP with token-level advantage (Eq. 2 and Eqs. 12-13)
    Used to derive the per-step optimal policy and the DPO-log-ratio advantage estimator; a standard but non-trivial modeling choice.
  • domain assumption The DPO log-ratio equals the token-level advantage (Eq. 13)
    Imported from Ziebart (2010), Li et al. (2025), and Zhong et al. (2024); the default dense advantage estimator and hence the LCO target rest on this equivalence.
  • domain assumption The parameterized policy has sufficient capacity to represent π* (or z*)
    The claim that minimizing any LCO loss is equivalent to solving the original RL problem (Section 4.1) requires the policy class to contain the optimal target; this is asserted, not tested.
  • ad hoc to paper Advantage normalization at each time step ensures mean-zero advantages (Section 4.4)
    Used to justify that the zero-shift z* in Eq. (7) is the convergence-optimal target; whether this normalization is actually applied in the experiments is not documented.

pith-pipeline@v1.3.0-alltime-deepseek · 23115 in / 18488 out tokens · 164876 ms · 2026-08-02T19:45:47.296121+00:00 · methodology

0 comments
read the original abstract

While reinforcement learning (RL) has been central to the recent success of large language models (LLMs), RL optimization is notoriously unstable, especially when compared to supervised fine-tuning (SFT). In this work, we investigate the stability gap between SFT and RL from a gradient-based perspective, and show that the convexity of the SFT loss with respect to model logits plays a key role in enabling stable training. Our theoretical analysis demonstrates that this property induces favorable gradient directionality during optimization. In contrast, Proximal Policy Optimization (PPO), a widely adopted policy gradient algorithm utilizing a clipped surrogate objective, lacks this stabilizing property. Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework that aligns the learned policy with an optimal target derived from the original RL objective, thereby emulating the stabilizing effects of logits-level convexity. Extensive experiments across multiple model families show that our LCO framework consistently improves training stability and outperforms conventional RL methods on a broad range of benchmarks.

Figures

Figures reproduced from arXiv: 2603.00963 by Hongzhan Chen, Shiping Gao, Tao Yang, Ting Yao, Xiaojun Quan, Yuhua Zhu.

Figure 1
Figure 1. Figure 1: (a) Gradient norm during training for SFT and PPO. (b) Pass@1 results of SFT and PPO on the MATH500 benchmark.. (Schulman et al., 2017; Yu et al., 2025), and KL-based penal￾ties that regulate policy shifts (Ouyang et al., 2022; Shao et al., 2024). Although these methods mitigate instability to some extent, it remains a persistent challenge in LLM optimization, and its underlying causes are still not fully … view at source ↗
Figure 2
Figure 2. Figure 2: (a) Gradient norm during training for LCO-KLD. (b) Pass@1 results of LCO-KLD on MATH500. Building on these theoretical insights, we propose Logits Convex Optimization (LCO), an RL optimization frame￾work that preserves logits convexity and promotes stable training for LLMs. LCO reformulates the RL task as an op￾timal target matching problem, while remaining mathemati￾cally consistent with the PPO objective… view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of SFT. (a) Gradient norm decreases as training progresses while target token probability on training sample increases. The magnitudes of both target token logit gradient (b) and the non-target token logit gradient (c) diminish as training progresses. 0 250 500 750 Training Steps 0.0 0.5 Gradient Norm (a) PPO (Negative) Clip Bound 0.2 0.4 0.6 0.8 1.0 Sampled Action Prob (Negative) 0 |A| 4… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of PPO for actions with negative advantages. (a) Gradient norm increases before being clipped, while sampled action probabilities decrease. The magnitudes of both sampled action logit gradient (b) and non-sampled action logit gradient (c) increase before being clipped. The results show a correlation between the behavior of parameter-space gradients and logit-space gradients. dynamics. Dur… view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics of LCO-KLD for actions with negative advantages. Unlike training dynamics of PPO, LCO-KLD exhibits stable gradient updates that scale with the remaining error and diminish progressively as the model converges. and for small η such that ρLCH < 1, gradient descent on LLCO-LCH converges linearly: LLCO-LCH(θk) ≤ 1 2|V| ρ 2k LCH ∥A∥ 2 β 2 . (20) Proof. See Appendix H. To justify our specific c… view at source ↗
Figure 6
Figure 6. Figure 6: Training dynamics of LCO-KLD. (a) Gradient norm for actions with positive advantages. (b) Gradient norm for actions with negative advantages. Unlike training dynamics of PPO, these results demonstrate a stable gradient update of LCO-KLD during training. 0 2 4 8 16 24 #Training Samples (×256) 40 45 50 55 Pass@1(%) 3× 2× (a) Sample Efficiency (3B) LCO-LCH LCO-KLD PPO 0 2 4 8 16 24 #Training Samples (×256) 57… view at source ↗
Figure 7
Figure 7. Figure 7: Training efficiency of LCO-LCH, LCO-KLD, and PPO, evaluated by policy accuracy on the MATH500 evaluation set. Training Efficiency To assess the training efficiency of the LCO, we compare the performance of policies based on Qwen-3-4B and Qwen-2.5-3B backbones, trained us￾ing LCO-LCH, LCO-KLD, and PPO. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training dynamics of PPO for actions with positive advantages. (a) Gradient norm decreases as training progresses while sampled action probabilities increase. The magnitudes of both sampled (b) and non-sampled action logit gradient (c) decrease. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Training dynamics of LCO-KLD for actions with positive advantages. We show the dynamics of gradient norm (a), logit gradient for sampled (b) and non-sampled (c) actions. LCO-KLD exhibits stable gradient updates as the model converges. while the magnitude of logit gradients decrease before being clipped ( [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

    cs.CL 2026-04 unverdicted novelty 6.0

    IPVRM learns prefix values to produce reliable step rewards from sequence outcomes using TD learning, enabling distribution-level RL that improves reasoning when paired with calibrated rewards.

  2. Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

    cs.CL 2026-04 conditional novelty 6.0

    Learning prefix-conditioned success probabilities from outcome labels yields more reliable process rewards than sequence log-ratio PRMs and improves RL when used for candidate-token TD updates.

Reference graph

Works this paper leans on

20 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [2]

    Minimax- m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025a

    Chen, A., Li, A., Gong, B., Jiang, B., Fei, B., Yang, B., Shan, B., Yu, C., Wang, C., Zhu, C., et al. Minimax- m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585, 2025a. Chen, H., Yang, T., Gao, S., Chen, R., Quan, X., Tian, H., and Yao, T. Discriminative policy optimization for token- level reward models.ar...

    Pith/arXiv arXiv

  2. [3]

    press/v235/cui24f.html

    URL https://proceedings.mlr. press/v235/cui24f.html. Cui, G., Zhang, Y ., Chen, J., Yuan, L., Wang, Z., Zuo, Y ., Li, H., Fan, Y ., Chen, H., Chen, W., et al. The entropy mech- anism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Pith/arXiv arXiv

  3. [4]

    Minillm: Knowl- edge distillation of large language models.arXiv preprint arXiv:2306.08543,

    Gu, Y ., Dong, L., Wei, F., and Huang, M. Minillm: Knowl- edge distillation of large language models.arXiv preprint arXiv:2306.08543,

    Pith/arXiv arXiv

  4. [9]

    From $r$ to $qˆ*$: Your language model is secretly a q- function

    Rafailov, R., Hejna, J., Park, R., and Finn, C. From $r$ to $qˆ*$: Your language model is secretly a q- function. InFirst Conference on Language Modeling, 2024a. URL https://openreview.net/forum? id=kEVcNxtqXk. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly ...

    Pith/arXiv arXiv

  5. [11]

    Ring-lite: Scalable reasoning via c3po-stabilized reinforcement learning for llms.arXiv preprint arXiv:2506.14731,

    Team, L., Hu, B., Chen, C., Zhao, D., Liu, D., Jin, D., Zhu, F., Dai, H., Luan, H., Guo, J., et al. Ring-lite: Scalable reasoning via c3po-stabilized reinforcement learning for llms.arXiv preprint arXiv:2506.14731,

    Pith/arXiv arXiv

  6. [12]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yang, S., Dou, C., Guo, P., Lu, K., Ju, Q., Deng, F., and Xin, R. Dcpo: Dynamic clipping policy optimization.arXiv preprint arXiv:2509.02333, 2025b. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo,...

    Pith/arXiv arXiv

  7. [13]

    What’s behind ppo’s collapse in long-cot? value optimization holds the secret.arXiv preprint arXiv:2503.01491,

    Yuan, Y ., Yue, Y ., Zhu, R., Fan, T., and Yan, L. What’s behind ppo’s collapse in long-cot? value optimization holds the secret.arXiv preprint arXiv:2503.01491,

    Pith/arXiv arXiv

  8. [14]

    R1-reward: Training multimodal reward model through stable rein- forcement learning.arXiv preprint arXiv:2505.02835,

    Zhang, Y .-F., Lu, X., Hu, X., Fu, C., Wen, B., Zhang, T., Liu, C., Jiang, K., Chen, K., Tang, K., et al. R1-reward: Training multimodal reward model through stable rein- forcement learning.arXiv preprint arXiv:2505.02835,

    Pith/arXiv arXiv

  9. [15]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y ., Men, R., Yang, A., et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Pith/arXiv arXiv

  10. [16]

    Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,

    Zhong, H., Shan, Z., Feng, G., Xiong, W., Cheng, X., Zhao, L., He, D., Bian, J., and Wang, L. Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,

    Pith/arXiv arXiv

  11. [17]

    Carft: Boosting llm reasoning via contrastive learning with an- notated chain-of-thought-based reinforced fine-tuning

    Zhu, W., Liu, J., Zhang, R., Wu, H., and Zhang, Y . Carft: Boosting llm reasoning via contrastive learning with an- notated chain-of-thought-based reinforced fine-tuning. arXiv preprint arXiv:2508.15868,

    Pith/arXiv arXiv

  12. [18]

    To address this, they propose a pretraining procedure for the value model, and decouple the λ in GAE for the policy and value model computations

    identifies a failure mode where the value model exhibits bias during training, resulting in large errors in advantage estimation. To address this, they propose a pretraining procedure for the value model, and decouple the λ in GAE for the policy and value model computations. Zhang et al. (2025) identify outliers caused by the imbalance in the advantage di...

    2025

  13. [19]

    Similarly, Shao et al

    employs a leave-one-out baseline across multiple completions to produce advantage estimate for prompt. Similarly, Shao et al. (2024) introduce GRPO, which standardizes sequence-level rewards by subtracting the mean and dividing by the standard deviation, thereby reducing bias and variance. Extending GRPO, Yu et al. (2025) propose DAPO, which re-weights to...

    2024

  14. [20]

    Building upon the same idea, DCPO (Yang et al., 2025b) addresses the limitation in DAPO, where the same clip range is set for different positions

    proposes a decoupled clip-higher method that relaxes the upper clipping bound to improve training efficiency while maintaining stability. Building upon the same idea, DCPO (Yang et al., 2025b) addresses the limitation in DAPO, where the same clip range is set for different positions. It further introduces a dynamic clipping method that adaptively adjusts ...

    2025

  15. [2017]

    Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  16. [2018]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  17. [2022]

    Generalist re- ward models: Found inside large language models.arXiv preprint arXiv:2506.23235,

    Li, Y .-C., Xu, T., Yu, Y ., Zhang, X., Chen, X.-H., Ling, Z., Chao, N., Yuan, L., and Zhou, Z.-H. Generalist re- ward models: Found inside large language models.arXiv preprint arXiv:2506.23235,

    Pith/arXiv arXiv

  18. [2023]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  19. [2024]

    Association for Computational Linguistics. Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernan- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, ...

    Pith/arXiv arXiv

  20. [2025]

    K., Xu, H., and Shen, W

    Hu, J., Liu, J. K., Xu, H., and Shen, W. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.arXiv preprint arXiv:2501.03262,

    Pith/arXiv arXiv