Pith. sign in

REVIEW 4 major objections 4 minor 18 references

CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a reinforcement-learning fine-tuning method, CVPO, outperforms strong value-based baselines by combining a value-variance advantage correction with a dynamic question-difficulty curriculum.

desk verdict The reported gains would matter if they held, but the core weighting equations in §4.1.3 appear to invert the paper's own stated mechanism, so the method as written is not the method described. read the letter →

arxiv 2608.03068 v1 pith:YWBKCDXF submitted 2026-08-04 cs.CL cs.AIcs.LG

Ziqi Jia , Yalu Ouyang , Bo Pang , Panpan Li , Hangfei Xu , Shengzhao Wen , Shiyong Li , Yanpeng Wang This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords reinforcementlearningvalue-basedRLadvantageestimationcurriculummathematicalreasoninglargelanguagemodelsvaluevarianceexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CVPO, a reinforcement-learning method for large language models that makes two changes to how training advantages are computed. First, it estimates the standard deviation of token-level value predictions within each generated response and uses that variance to down-weight high-variance correct responses while preserving exploration on high-variance incorrect ones. Second, it tracks each question's recent accuracy with a Beta posterior and reweights questions so the model trains hardest on problems near its current ability, switching emphasis toward hard questions when learning plateaus. The paper reports that on Qwen2.5-7B this outperforms VAPO and GRPO variants on AIME24, AIME25, AMC23, AMC24, and MATH500, with each mechanism contributing and the variance correction contributing more. If correct, this means value-model statistics that are normally ignored, the spread of predicted values, can serve as a usable exploration signal for RL fine-tuning of reasoners.

What carries the argument

The central object is the trajectory value-variance signal $\sigma_i = \operatorname{std}(v_i)$, computed from the critic's token-level value estimates on each response and normalized within a prompt group by $\mu_k$. It controls an asymmetric sigmoid correction $W_S$ for positive and negative trajectories, while a separate Beta-posterior accuracy estimate $\rho_q$ controls a difficulty curriculum weight $W_D$ that changes form when the model enters a learning bottleneck. Together they rescale every token advantage as $A' = W_S(\operatorname{Var}(\tau), R(\tau)) \cdot W_D(\rho_q, t_c) \cdot A$, which is the single mechanism that carries the argument: variance says how much a trajectory is exploring, accuracy says how hard a question is for the current policy, and both together decide how strongly each token's gradient should count.

What would settle it

Measure $\sigma_i = \operatorname{std}(v_i)$ on generated responses of equal correctness and different response lengths, or on responses where the value model is intentionally corrupted; if the variance signal tracks length or critic noise rather than exploration, the interpretation fails. A direct test of the mechanism is to flip the weighting directions for negative trajectories while holding the curriculum fixed; if performance does not degrade, the exploration weighting is not what drives the reported gains.

Watch

Extended reading notes

Core claim

CVPO rests on two linked claims. At trajectory level, the spread of token-level value estimates, $\sigma_i = \operatorname{std}(v_i)$, is treated as a measure of exploration intensity, and the paper derives an upper bound $\|\nabla_\theta J(\theta)\|_2 \le C(\sigma_Q + \sqrt{\operatorname{Var}(V)})$ showing that value variance bounds the policy-update magnitude. From this it constructs an asymmetric sigmoid weight $W_S$ on the advantage: correct trajectories with high variance are penalized toward stable convergence, while incorrect trajectories with high variance are lightly penalized to keep exploring. At question level, a Beta posterior over each question's accuracy $\rho_q$ feeds a curriculum weight $W_D$ that concentrates training on roughly-50%-accuracy questions early and shifts to low-accuracy hard questions when stagnation is detected. The combined advantage $A' = W_S \cdot W_D \cdot A$ is what the paper trains with, and the reported gains follow from this single weighted advantage being more selective about which trajectories and which questions drive the gradient.

Load-bearing premise

The method assumes the spread of the value model's token-level predictions within a response tells you how much the model is exploring, and that weighting trajectories by this spread the way CVPO does improves learning even though the paper only shows the spread bounds gradient size, not that the weighting direction is optimal.

Editorial extensions

If this is right

  • With the same Qwen2.5-7B backbone and DAPO-Math-17k training set, CVPO is reported to reach avg@32 accuracy of 0.237, 0.133, 0.720, 0.512, and 0.802 on AIME24, AIME25, AMC23, AMC24, and MATH500, respectively.
  • Each component alone beats VAPO; removing the variance correction hurts more than removing the curriculum, implying that most of the gain comes from variance-based advantage weighting.
  • During training past about step 1500, CVPO keeps improving while baselines plateau, because the curriculum moves weight onto harder questions once the model is stagnant.
  • The crossover shown in Figure 4 implies that value-variance coefficients need to be tuned per benchmark: larger coefficients help AMC problems but degrade the harder AIME questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: if value variance really indexes exploration, CVPO offers a way to regulate exploration without changing the reward model, which could transfer to tasks beyond math where rewards are sparse.
  • The Beta-progress curriculum can be extended to non-binary or partial-credit rewards by replacing the Bernoulli likelihood, but the paper only tests binary correctness.
  • The AMC/AIME crossover suggests the variance gain is problem-dependent; a testable extension would schedule the variance coefficient to decay as the policy matures rather than keeping it fixed.
  • The paper's interpretation of $\sigma_i$ assumes critic noise is small; separating exploration variance from value-model uncertainty would require an explicit noise model, which the paper does not provide.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CVPO, a reinforcement-learning post-training method for LLM mathematical reasoning. It combines a trajectory-level stochasticity-aware advantage weighting, based on the standard deviation of token-level value estimates, with a question-level dynamic curriculum weighting based on a Bayesian accuracy estimate. The method is evaluated on Qwen2.5-7B trained on DAPO-Math-17k and compared with VAPO and an asynchronous GRPO baseline on AIME24, AIME25, AMC23, AMC24, and MATH500, reporting substantial gains. The authors also provide a short theoretical argument bounding the policy gradient norm in terms of value variance and ablation studies of the two proposed components.

Significance. If the reported results are reproducible, the paper would make a practically useful contribution: it suggests that token-level value variance can be used to reweight advantages and that an accuracy-based curriculum mitigates difficulty drift, with gains on several externally evaluated math benchmarks. The external evaluation is a genuine strength, and the theoretical bound in Section 4.1.1, while elementary, is clearly stated. However, the central mechanism is currently internally inconsistent: the equations for the stochasticity-aware weights implement the opposite of the described weighting in the text, and the empirical evidence rests on a single configuration with no seeds, no confidence intervals, and no released code or checkpoints. The method also contains many hand-specified hyperparameters, several of which appear to be selected using the same benchmarks on which the method is evaluated. These issues prevent the paper from being accepted in its current form.

major comments (4)
  1. [Section 4.1.3, Eqs. (15)-(16)] The stochasticity-aware weighting functions implement the opposite of the stated mechanism. For a correct trajectory (R=1), Eq. (15) gives W_S = 1 + α_P·sigmoid(λ_P(Var−μ_k)), so high-variance correct trajectories receive a weight larger than 1. Since correct trajectories predominantly carry positive advantages, this amplifies the update for exactly the high-variance correct responses that the text says should be suppressed, and contradicts the promised 'strong constraints on positive samples.' For an incorrect trajectory (R=0), Eq. (16) gives W_S = 1 − α_N·sigmoid(λ_N(μ_k−Var)), which makes low-variance incorrect trajectories receive a reduced penalty (W_S < 1) and high-variance incorrect trajectories keep the full penalty (W_S ≈ 1). This is again opposite to the text, which states that low variance should increase penalty intensity and high variance should reduce it. This is an internal inconsistency in the load-bearing component of the method: as written, the equations do not implement the stochasticity-aware correction that is claimed and ablated in Table 1. The authors must either change Eqs. (15)-(16) or correct the description and rerun the experiments so that the reported results correspond to the stated mechanism.
  2. [Section 5.1 and Table 1] The central empirical claim rests on a single configuration: no random seeds are reported, no confidence intervals are given, and no code or checkpoints are released. Moreover, the final CVPO row uses a variance coefficient of 0.3 that is explicitly selected from the sweep shown in Figure 4 on the same benchmark family as the final evaluation. This means the reported gains over the second-best methods, e.g., 9.3% on AIME24 and 25.2% on AMC23, may in part reflect selection on the evaluation sets. I request multi-seed results with variance or confidence intervals, and either a pre-specified or separately validated choice of the 0.3 coefficient, or a clear statement that the reported numbers are the best over a small grid.
  3. [Section 4.1.1, Eq. (13)] Equation (13) is an upper bound on the gradient norm and demonstrates only that value variance can affect the magnitude of policy updates. It does not determine the sign of the advantage weighting, nor does it show that increasing variance in correct or incorrect trajectories is beneficial or that one trajectory type should be emphasized over another. Therefore the theory cannot rescue the sign inconsistency in Eqs. (15)-(16), and even after correcting the sign it would not justify the particular sigmoid-shaped weighting. The theoretical section should be substantially weakened, or supplemented with a directional argument or controlled experiments that isolate the effect of the variance signal.
  4. [Section 4.2.2, Eqs. (25)-(26)] The dynamic curriculum component relies on several hand-specified thresholds and curve parameters (0.3, 0.8, 0.05, 0.03; k1, c1, k2, c2 and their primed versions) with no sensitivity analysis and no derivation. These are free parameters that appear to be chosen to match the behavior shown in Figure 2 and the benchmark outcomes. To support the claim that the curriculum is a general mechanism rather than a fitted schedule, the paper should report sensitivity to these thresholds and ideally a procedure for setting them without using the final evaluation benchmarks.
minor comments (4)
  1. [Section 4.1.3 and Eq. (14)] The notation is inconsistent: the text defines σ_i = std(v_i) and uses Var(τ) in Eqs. (15)-(16), while Eq. (14) defines μ_k as the mean of σ_i within a prompt group. The unprimed and primed hyperparameters in the text also switch between μ_P/μ_N and μ_k; please unify the notation.
  2. [Section 4.2.2, Eq. (22)] The trend indicator Δ_t^τ divides by the previous posterior mean \( \bar\gamma^{t-1}_\tau \), which can be zero if no successes have been observed; a small constant or a different definition should be given for this case.
  3. [Section 6.2 and Figure 3] There is a grammatical error in the sentence 'This shift is much more visible with CVPO, This shift in value-variance distribution...' The second sentence should be capitalized and separated.
  4. [Abstract and Introduction] The claim that the theoretical analysis 'proves that appropriately leveraging this variance can improve the upper bound of model performance' is stronger than what Eq. (13) actually shows; please rephrase to match the result.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are external benchmark results, the components are hand-specified heuristics evaluated empirically, and the theoretical bound does not smuggle in the weighting functions.

full rationale

The paper's central claims are empirical comparisons on external benchmarks (AIME24/25, AMC23/24, MATH500), and the two proposed components are presented as heuristic designs rather than as consequences of their own definitions. The variance bound in Eq. 13 is a generic Cauchy-Schwarz/Jensen inequality relating gradient-norm magnitude to Var(V); it does not presuppose the sigmoid weighting in Eqs. 15-16, and those weighting functions are not derived from the bound. The dynamic curriculum weight WD is a hand-specified function of posterior accuracy, not a quantity predicted from itself. The reported 0.3 variance coefficient is a tuned hyperparameter selected from a sweep (Figure 4), which raises a data-selection/correctness concern but does not make the benchmark outcomes equivalent to the input by construction. No load-bearing self-citations or imported uniqueness theorems appear: the cited baselines (GRPO, VAPO, DAPO) are external prior work and the relevant cited authors do not overlap with the present author list in a way that forces the paper's choices. A separate, non-circular correctness issue is that Eqs. 15-16, as written, appear to amplify rather than suppress high-variance correct trajectories and to retain rather than reduce the penalty for high-variance incorrect trajectories, contradicting the prose in Section 4.1.3; this is an internal inconsistency or implementation-reporting problem, not circularity, and it does not change the circularity verdict.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The theory contributes only standard inequalities; the distinctive behavioral assumptions, such as variance means exploration and thresholds identify bottlenecks, are asserted rather than derived. All parameters of the weighting curves are hand-set or selected by sweep, and the final 0.3 coefficient is selected on the reported benchmarks.

free parameters (5)
  • Variance weighting coefficient for final CVPO = 0.3
    Table 1 uses CVPO (w/0.3 variance coeff) as the full method; Figure 4 sweeps coefficients and shows a crossover, indicating selection on benchmark performance.
  • Early-stage curriculum curve hyperparameters (k1, c1, k2, c2) = 1.6, -0.2, 2, 1.2
    Equation 25: these fixed values shape the accuracy-weight curve to peak near accuracy 0.5.
  • Bottleneck-stage curriculum curve hyperparameters (k1', c1', k2', c2') = 4.1, -0.1, 2, 0.9
    Equation 26: these fixed values downweight accuracy-1 questions and concentrate weight on low accuracy.
  • Stagnation detection thresholds = low accuracy <= 0.3, high accuracy >= 0.8, |delta| <= 0.05, delta <= 0.03
    Equations 23-24: these thresholds trigger the curriculum switch; no sensitivity analysis is reported.
  • Advantage weighting gains and steepnesses (alpha_P, lambda_P, alpha_N, lambda_N) = not stated, except final coefficient 0.3
    Equations 15-16: W_S amplitude and steepness are fixed hyperparameters; the text says fixed but reports no values.
assumptions (5)
  • standard math Cauchy-Schwarz and Jensen inequalities, plus Var(A) = Var(Q) + Var(V) - 2Cov(Q,V).
    Used in Equations 5-11 to bound the gradient norm. Standard, so no independent evidence is needed.
  • domain assumption The policy log-density gradient has bounded L2 norm C for all states and actions the policy visits.
    Equation 4: this boundedness is needed for the bound but is not proved for the trained transformer policy.
  • domain assumption Token-level variance of value estimates measures intrinsic exploration intensity within a trajectory.
    Asserted in Section 4.1.2 and Figure 3 without a causal or statistical test. This is the load-bearing behavioral premise of the advantage weighting.
  • ad hoc to paper A Beta-binomial model with Beta(1,1) prior provides a reliable online difficulty estimate, and the 0.3, 0.8, 0.05, 0.03 thresholds identify learning bottlenecks.
    Equations 17-24: the prior and thresholds are introduced for this paper and are only validated through the final benchmark numbers.
  • domain assumption A Qwen2.5-7B value model checkpoint with loss below 1 and explained variance above 0 gives reliable per-token values.
    Implementation details in Section 5.1: this checkpoint criterion is stated without validation of value accuracy on the reasoning tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning." pith.science (2026). https://pith.science/paper/YWBKCDXF

@misc{pith2026260803068,
  author       = {Pith},
  title        = {Pith review of: CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YWBKCDXF}},
  note         = {Machine review of arXiv:2608.03068}
}
read the original abstract

Reinforcement learning (RL) has emerged as an effective method for enhancing the reasoning capabilities of large language models (LLMs). However, existing methods suffer from insufficient precision in feedback on generated answer trajectories and exhibit the phenomenon of problem difficulty drift. To address these challenges, we propose CVPO - Curriculum-guided Value-Variance Policy Optimization. At the response trajectory level, we find that token-level value-variance correlates with exploration intensity. Our theoretical analysis shows this variance bounds policy update magnitude. We then use the estimated trajectory value-variance to quantify the intrinsic randomness in generation. Based on this, we design a variance-aware advantage adjustment mechanism for different reward types. At the question level, we introduce a dynamic curriculum weighting method that adapts to question difficulty. This helps the model focus on tasks matched to its current ability during each training stage. Experimental results show our method outperforms strong value-based baselines like VAPO. It achieves better performance and stronger exploration, enabling more accurate and robust reasoning in language models across various math tasks.

Figures

Figures reproduced from arXiv: 2608.03068 by the authors.

Figure 1
Figure 1. Tokens with the highest average values from generated responses in the training set. They are closely related to the training task, which is mathe￾matics in our experiment setting As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Comparison of CVPO and previous SOTA methods on the AIME 2024 benchmark using the [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Trajectory-level value-variance distribution on the training set between Qwen2.5 7B Base and [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: CVPO performance using different value-variance coefficients. Higher coefficient values display [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Comparison of training with and without dynamic question difficulty weighting. The color in each [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 2 canonical work pages

  1. [2]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt

    Deepseek-v3 technical report.CoRR, abs/2412.19437. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt

    arXiv

  2. [5]

    Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, and Shuiwang Ji

    Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.CoRR, abs/2503.24290. Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, and Shuiwang Ji

    arXiv

  3. [6]

    Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak

    Curriculum reinforcement learning from easy to hard tasks improves LLM reasoning.CoRR, abs/2506.06632. Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak

    arXiv

  4. [7]

    Yun Qu, Qi Cheems Wang, Yixiu Mao, Vincent Tao Hu, and Xiangyang Ji

    Maximizing confidence alone improves reasoning.CoRR, abs/2505.22660. Yun Qu, Qi Cheems Wang, Yixiu Mao, Vincent Tao Hu, and Xiangyang Ji

    arXiv

  5. [8]

    Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?arXiv preprint arXiv:2507.04632. Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Ávila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Sh...

    arXiv

  6. [9]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    Offline regularised reinforcement learning for large language models alignment.CoRR, abs/2405.19107. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    arXiv

  7. [11]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y

    Spurious rewards: Rethinking training signals in RLVR.CoRR, abs/2506.10947. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo

    arXiv

  8. [12]

    CoRR, abs/2402.03300

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300. Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao

    arXiv

Show all 18 references
  1. [13]

    Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, and Yuting Liu

    Efficient reinforcement finetuning via adaptive curriculum learning.CoRR, abs/2504.05520. Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, and Yuting Liu

    arXiv

  2. [14]

    DGRO: en- hancing LLM reasoning via exploration-exploitation control and reward variance management.CoRR, abs/2505.12951. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang,...

    arXiv

  3. [15]

    Kimi k1.5: Scaling reinforcement learning with llms.CoRR, abs/2501.12599. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhan...

    arXiv

  4. [16]

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan

    DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan

    arXiv

  5. [17]

    What’s behind ppo’s collapse in long-cot? value optimization holds the secret.CoRR, abs/2503.01491. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Cheng-Xiang Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu...

    arXiv

  6. [18]

    V APO: efficient and reliable reinforcement learning for advanced reasoning tasks.CoRR, abs/2504.05118. 15

    arXiv

  7. [2017]

    Proximal policy optimization algorithms.CoRR, abs/1707.06347. 14 Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer

    arXiv

  8. [2021]

    InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual

    Measuring mathematical problem solving with the MATH dataset. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. Jian Hu

    2021

  9. [2024]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. InProceedings of the 62nd Annual Meeting of the Association for Computational 13 Linguistics (V olume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 12248...

    2024 arXiv

  10. [2025]

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum

    REINFORCE++: A simple and efficient approach for aligning large language models.CoRR, abs/2501.03262. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.