Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Improving RL Exploration for LLM Reasoning through Retrospective Replay

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that policy gradients in LLM RL suppress promising early states, and that replaying those states restores exploration and improves reasoning.

desk verdict RRL is a sensible Go-Explore-style replay idea for LLM RL, but the paper currently under-delivers on evidence: the optimization objective doesn't parse, and the ablations don't isolate the exploration mechanism. read the letter →

arxiv 2504.14363 v2 pith:PHA3Z6EK submitted 2025-04-19 cs.LG cs.CL

Shihan Dou , Muling Wu , Jingwen Xu , Rui Zheng , Tao Gui , Qi Zhang , Xuanjing Huang This is my paper · ORCID
classification cs.LGcs.CL
keywords reinforcementlearningLLMreasoningexplorationreplaybufferpolicygradientvaluemodelcodegenerationRLHF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies a failure mode in RL fine-tuning of large language models: for hard problems, the early model finds promising partial ideas but gets negative reward because the full solution is wrong, so the policy gradient suppresses those partial ideas. Later in training the model is stronger but no longer explores them, and the paper argues this lost exploration, not just sampling diversity, is why RL stalls on complex reasoning. To fix it, RRL stores a high-value intermediate state from each failed generated solution and from the canonical solution, then replays those states as starting points with a tunable probability. In the paper's experiments this raises pass@1 on APPS+ from 31.7% to 35.2%, GSM8K from 68.8% to 70.7%, MATH from 33.3% to 34.3%, and gives a 55.4% win rate over vanilla PPO in dialogue RLHF.

What carries the argument

The promising-state buffer is the mechanism. For a problem $q$, RRL computes $s^* = \arg\max_{s_i} V(s_i)$ on the generated solution and, if the solution is wrong, on the canonical solution too; each stored state has a counter, buffer capacity is five per problem, and solved states are removed. The replay coefficient $p$ controls how often generation starts from a buffered state instead of the problem prompt, with $p$ growing over the epoch, and the loss masks state tokens so the model is not over-optimized on already-written text. This is what lets a later, stronger policy return to an earlier promising prefix and continue exploring from it.

What would settle it

Run RRL with buffer states selected uniformly at random from the same generated and canonical solutions instead of by $\arg\max_{s_i} V(s_i)$; if the random-state version matches RRL's APPS+ or MATH gains, value-model selection is not the component that improves exploration.

Watch

Extended reading notes

Core claim

The central claim is that the standard policy-gradient objective actively erases good prefixes: since a solution is scored only by final correctness, a trajectory with a correct idea and one bad token gets negative advantage, and every token in it, including the good prefix, is down-weighted. Over training this is compounded by reduced action diversity, so the strengthened model cannot rediscover states the weak model once visited. RRL counters by using the value model to identify the highest-value state in both the policy-generated (possibly wrong) solution and the canonical solution, storing these states in a bounded buffer and re-generating completions from them while masking the stored state tokens in the loss. The paper reports consistent gains over vanilla PPO and prior RL methods, with the largest relative improvements on the hardest APPS+ splits (Interview 20.1% to 23.5%, Competition 5.0% to 7.9%).

Load-bearing premise

The value model's highest-scoring state inside a mostly wrong solution is a genuinely good place to resume, so continuing from that state teaches more than starting from the problem prompt.

Editorial extensions

If this is right

  • If the paper's diagnosis is right, any sparse-reward RL post-training of LLMs should show the same early-idea suppression, so RRL-style replay should transfer to new reasoning benchmarks beyond APPS+, GSM8K, and MATH.
  • The ablations indicate that replaying only policy-generated states or only canonical states helps less than the combination, so the full method's advantage is specifically the mix of imperfect model ideas and correct reference prefixes.
  • The RLHF result suggests that preserving exploration during alignment improves both helpfulness and safety, not only objective reasoning accuracy.
  • The sensitivity analysis shows gains at replay coefficients 0.1 to 0.5, with 0.1 optimal, so the method is not tied to a single hyperparameter setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the mechanism is to ablate the value model itself: replacing the argmax state with a random prefix from the same failed solution would show whether value-based selection or mere resumption drives the gains.
  • The paper's limitation that early value estimates are unreliable suggests that adding execution feedback for code or verifier scores for math to filter replayed states before storing them could further reduce the risk of replaying errors; the paper leaves this for future work.
  • RRL is algorithmically close to experience replay but with a different target, replaying states to resume exploration rather than to update value estimates; this distinction could be tested by comparing RRL against prioritized experience replay on the same LLM tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a failure mode in RL for LLM reasoning: policy gradient optimization suppresses early-trajectory prefixes that are promising but incomplete, thereby reducing exploration in later training stages. The proposed method, Retrospective Replay-based Reinforcement Learning (RRL), stores the highest-value states from policy-generated and canonical solutions in a buffer and replays them with probability p during training, increasing replay probability as training progresses. Experiments are reported on APPS+ code generation, GSM8K and MATH reasoning, and HH-RLHF dialogue; headline results are APPS+ overall 35.2% vs 31.7% for vanilla PPO, GSM8K 70.7% vs 68.8%, MATH 34.3% vs 33.3%, and 55.4% win rate over PPO in dialogue. Ablations show that replaying states only from canonical solutions reaches 34.5% and states only from policy-generated solutions reaches 34.2% on APPS+.

Significance. The underlying observation is timely and plausible: RL training can cause LLMs to abandon partially correct solution prefixes that later, more capable checkpoints might be able to complete. RRL is a simple, modular addition to PPO and the experiments cover three distinct task families, which is a strength. The paper also provides explicit ablations, a sensitivity analysis, and an honest limitations section, and it states the exact evaluation protocols. However, the central attribution—that value-model-selected states improve exploration specifically—is not yet established, because the paper lacks a random-prefix replay control and the value-model premise is admitted to be unreliable early in training. In addition, the stated optimization objective in Eq. (3) is not the PPO objective referenced in the text, and results are reported without seeds, error bars, or code. If these load-bearing issues are resolved, the method could be a useful contribution to RL post-training for LLMs.

major comments (3)
  1. [§2.2, Eq. (3)] Eq. (3) is not a PPO objective. It lacks the importance ratio and clipping used in PPO and does not include the GAE advantage A_i from Eq. (1), yet the text states that 'we utilize the proximal policy optimization (PPO) algorithm.' The paper should give the exact surrogate objective used for both the normal-exploration branch and the replay branch, including how GAE advantages are computed for rollouts that start from replayed intermediate states and how the token masking of replayed states is implemented in the loss. As written, Eq. (3) cannot reproduce the claimed training procedure.
  2. [Table 1; §5 Limitations] The ablations undermine the central attribution of the gains to value-selected exploration. 'States only from CS' (34.5%) and 'States only from PGS' (34.2%) already outperform vanilla PPO (31.7%) by roughly 2.8 and 2.5 points, respectively, recovering most of the full RRL gain of 3.5 points. Both ablation arms add a replay schedule that PPO lacks, and the CS arm also injects ground-truth solution prefixes, which is closer to curriculum learning or imitation than to the proposed 'return to a promising state and explore' mechanism. The paper's own Limitations section admits that value-model-selected states may contain errors and that the value model is unstable and inaccurate early in training, so the premise that the argmax state is genuinely promising is asserted rather than demonstrated. A control that replays random intermediate states at the same replay frequency, buffer capacity, and compute budget is missing. Without such a control, the reported gains cannot be attributed specifically to the value-model-based exploration mechanism; this should be added and analyzed, for example by reporting the success rate of completions from replayed states versus random prefixes.
  3. [Tables 1 and 2; §3.2] The statistical evidence is not yet sufficient for the claimed improvements. No seeds, confidence intervals, or code are reported. Several headline deltas are small, notably MATH 34.3% vs 33.3% and GSM8K 70.7% vs 68.8%, and the dialogue comparison against PPO is based on 100 pairwise evaluations (31 wins, 44 ties, 25 losses). The paper should report results over at least three seeds with standard deviations, or provide significance tests or confidence intervals, especially for the smaller math and dialogue gains.
minor comments (5)
  1. [Algorithm 1, line 2] The replay probability is defined as 'p = beta * (Step/Step num of one epoch) if Epoch = 1 else beta', but no epoch structure is otherwise defined in the algorithm or training loop; please clarify the schedule and how 'Step num of one epoch' is computed.
  2. [Figure 3] The sensitivity analysis does not state which dataset or task Figure 3 refers to, and the y-axis label 'Accuracy' is not tied to the Pass@1 metric used in Table 1; the vanilla PPO baseline of 27.9 also differs from the APPS+ overall value of 31.7 in Table 1, so the setting should be specified.
  3. [§3.1] There are small presentation errors: 'Deeseek-Coder-Instruct' should be 'DeepSeek-Coder-Instruct', and the text says 'an improvement of 1.9 PPL' where 'PPL' should be 'percentage points' or similar.
  4. [Table references in §3.2] The in-text references 'Table 3.1' and 'Table 3.2' do not match the actual table numbering (Table 1 and Table 2); please align the cross-references.
  5. [§3.1 / Implementation] Several RRL-specific hyperparameters are not reported: the maximum buffer capacity of five is mentioned in §2.2, but the counter update rule, the value-model stabilization threshold, and the epoch-dependent replay schedule are not specified with concrete values; please include these for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RRL's gains are empirical held-out evaluations; value-model selection is a design choice, not a fitted target, and self-citations are not load-bearing.

full rationale

The derivation chain is self-contained: RRL's design (value-model argmax selection, buffer, replay coefficient, exit mechanism) is an algorithmic proposal, and its claimed benefits are measured by held-out pass@1 results (APPS+, GSM8K, MATH) and GPT-4 win rates (HH-RLHF), none of which are constructed from the replay coefficient, the buffer contents, or the value model's estimates. The value model is trained online, not fitted to the reported benchmark numbers, and the ablations compare replay sources without redefining the metric. Self-citations (e.g., StepCoder, Reverse Curriculum RL, and PPO implementation notes) are used as baselines or related-work context, not as evidence that RRL's replay mechanism works. The Limitations section admits value-model instability and possible erroneous states, but those are acknowledged validity or attribution risks (e.g., the missing random-prefix control is an experimental design gap), not circular reductions: no equation or fitted parameter forces the headline results. Therefore no load-bearing circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces a replay buffer of intermediate states, but no new physical entities. Free parameters are limited to the replay coefficient, buffer capacity, and an underspecified stabilization criterion. The key assumptions are empirical claims about policy-gradient behavior and the reliability of the value model for state selection.

free parameters (3)
  • replay coefficient p = 0.1
    Probability of sampling a state from the replay buffer; set to 0.1 through sensitivity analysis (Section 3.3, Figure 3).
  • buffer capacity per problem = 5
    Maximum number of states stored per problem; when full, the state with the highest counter is removed (Section 2.2).
  • value model stabilization threshold = unspecified
    RRL 'only begins to find promising states after the loss of the value model has stabilized' (Section 2.2), but the criterion is never quantified.
assumptions (4)
  • domain assumption Policy gradient suppresses promising states and reduces exploration over training.
    Central motivation asserted in Section 2.1 and illustrated by Figure 2, but not rigorously demonstrated with quantitative evidence.
  • ad hoc to paper The value model's argmax state identifies a promising state worth replaying.
    Used in Section 2.2 to select states for the buffer; the paper itself notes the value model can be unreliable early in training.
  • domain assumption Canonical solutions are available for every training problem.
    RRL adds the highest-value state from the canonical solution to the buffer (Algorithm 1), which presumes per-problem reference solutions exist.
  • ad hoc to paper Masking the token positions of replayed states prevents over-optimization.
    Stated in Section 2.2 without derivation or empirical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving RL Exploration for LLM Reasoning through Retrospective Replay." pith.science (2026). https://pith.science/paper/PHA3Z6EK

@misc{pith2026250414363,
  author       = {Pith},
  title        = {Pith review of: Improving RL Exploration for LLM Reasoning through Retrospective Replay},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PHA3Z6EK}},
  note         = {Machine review of arXiv:2504.14363}
}
read the original abstract

Reinforcement learning (RL) has increasingly become a pivotal technique in the post-training of large language models (LLMs). The effective exploration of the output space is essential for the success of RL. We observe that for complex problems, during the early stages of training, the model exhibits strong exploratory capabilities and can identify promising solution ideas. However, its limited capability at this stage prevents it from successfully solving these problems. The early suppression of these potentially valuable solution ideas by the policy gradient hinders the model's ability to revisit and re-explore these ideas later. Consequently, although the LLM's capabilities improve in the later stages of training, it still struggles to effectively address these complex problems. To address this exploration issue, we propose a novel algorithm named Retrospective Replay-based Reinforcement Learning (RRL), which introduces a dynamic replay mechanism throughout the training process. RRL enables the model to revisit promising states identified in the early stages, thereby improving its efficiency and effectiveness in exploration. To evaluate the effectiveness of RRL, we conduct extensive experiments on complex reasoning tasks, including mathematical reasoning and code generation, and general dialogue tasks. The results indicate that RRL maintains high exploration efficiency throughout the training period, significantly enhancing the effectiveness of RL in optimizing LLMs for complicated reasoning tasks. Moreover, it also improves the performance of RLHF, making the model both safer and more helpful.

Figures

Figures reproduced from arXiv: 2504.14363 by the authors.

Figure 1
Figure 1. Left: Training process of RRL. Right: Distribution of the PPL variance of the responses obtained by performing multiple samplings for the same query. from exploring these states, leading to the forgetting of potentially valuable states, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Policy gradient suppresses the generated solutions that have the correct idea but contain slight flaws. We introduce a replay mechanism to enable the policy model to revisit promising states and continue exploring them to enhance their exploration ability. In reinforcement learning, the value function aims to estimate the ex￾pected total reward starting from a given state and following the policy model. In other wor… view at source ↗
Figure 3
Figure 3. Sensitivity analysis. Sensitivity analysis. Compared to the vanilla PPO, we introduce an additional hyper-parameter, the replay coefficient p. During the sampling process, the policy has a probability of p to generate a completion from a state in the buffer and a probability of 1−p to directly generate a solution from the prob￾lem. To effectively investigate the impact of the replay coefficient, we set it to six dif… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Entropy-weighted single-stage SFT+RL fine-tuning yields 59.1% average accuracy on five math benchmarks, improving on sequential and zero-RL baselines.

Reference graph

Works this paper leans on

56 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    IEEE Signal Processing Magazine 34(6), 26–38 (2017)

    Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforce- ment learning: A brief survey. IEEE Signal Processing Magazine 34(6), 26–38 (2017). https://doi.org/10.1109/MSP.2017.2743240

    arXiv 2017

  2. [2]

    CoRR abs/2204.05862 (2022)

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Con- erly, T., Showk, S.E., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T.B., Clark, J., McCandlish, S., Ola...

    arXiv doi:10.48550/arxiv.2204.05862 2022

  3. [3]

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al.: Training a helpful and harmless assistant withreinforcementlearningfromhumanfeedback.arXivpreprintarXiv:2204.05862 (2022)

    arXiv 2022

  4. [4]

    arXiv preprint arXiv:2401.02954 (2024)

    Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al.: Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 (2024)

    arXiv 2024

  5. [5]

    arXiv preprint arXiv:2412.12505 (2024)

    Chai, M., Shen, Z., Zhang, C., Zhang, Y., Wang, X., Dou, S., Kang, J., Zhang, J., Zhang, Q.: Docfusion: A unified framework for document parsing tasks. arXiv preprint arXiv:2412.12505 (2024)

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2207.11280 (2022)

    Christopoulou, F., Lampouras, G., Gritta, M., Zhang, G., Guo, Y., Li, Z., Zhang, Q., Xiao, M., Shen, B., Li, L., et al.: Pangu-coder: Program synthesis with function- level language modeling. arXiv preprint arXiv:2207.11280 (2022)

    arXiv 2022

  7. [7]

    ArXiv abs/2110.14168 (2021), https: //api.semanticscholar.org/CorpusID:239998651 Improving RL Exploration for LLM Reasoning through Retrospective Replay 11

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plap- pert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., Schulman, J.: Training verifiers to solve math word problems. ArXiv abs/2110.14168 (2021), https: //api.semanticscholar.org/CorpusID:239998651 Improving RL Exploration for LLM Reasoning through Retrospective Replay 11

    arXiv 2021

  8. [8]

    In: 2008 Interna- tional Conference on Computational Intelligence for Modelling Control Automa- tion

    Dimitrakakis, C.: Tree exploration for bayesian rl exploration. In: 2008 Interna- tional Conference on Computational Intelligence for Modelling Control Automa- tion. pp. 1029–1034 (2008).https://doi.org/10.1109/CIMCA.2008.32

    work page doi:10.1109/cimca.2008.32 2008

Show all 56 references
  1. [9]

    ArXiv abs/2411.00750 (2024), https://api.semanticscholar.org/ CorpusID:273798221

    Ding, Y., Xi, Z., He, W., Li, Z., Zhai, Y., Shi, X., Cai, X., Gui, T., Zhang, Q., Huang, X.: Mitigating tail narrowing in llm self-improvement via socratic-guided sampling. ArXiv abs/2411.00750 (2024), https://api.semanticscholar.org/ CorpusID:273798221

    2024 arXiv

  2. [10]

    arXiv preprint arXiv:2407.06153 (2024)

    Dou, S., Jia, H., Wu, S., Zheng, H., Zhou, W., Wu, M., Chai, M., Fan, J., Huang, C., Tao, Y., et al.: What’s wrong with your code generated by large language models? an extensive study. arXiv preprint arXiv:2407.06153 (2024)

    2024

  3. [11]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Dou, S., Liu, Y., Jia, H., Zhou, E., Xiong, L., Shan, J., Huang, C., Wang, X., Fan, X., Xi, Z., et al.: Stepcoder: Improving code generation with reinforcement learning from compiler feedback. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Ling...

    2024

  4. [12]

    arXiv preprint arXiv:2405.00438 (2024)

    Dou, S., Liu, Y., Zhou, E., Li, T., Jia, H., Xiong, L., Zhao, X., Ye, J., Zheng, R., Gui, T., et al.: Metarm: Shifted distributions alignment via meta-learning. arXiv preprint arXiv:2405.00438 (2024)

    2024 arXiv

  5. [13]

    arXiv preprint arXiv:2308.01191 (2023)

    Dou, S., Shan, J., Jia, H., Deng, W., Xi, Z., He, W., Wu, Y., Gui, T., Liu, Y., Huang, X.: Towards understanding the capability of large language models on code clone detection: A survey. arXiv preprint arXiv:2308.01191 (2023)

    2023 arXiv

  6. [14]

    arXiv preprint arXiv:2506.02672 (2025)

    Dou, S., Zhang, M., Huang, C., Chen, J., Chen, F., Liu, S., Liu, Y., Liu, C., Zhong, C., Zhang, Z., et al.: Evalearn: Quantifying the learning capability and efficiency of llms via sequential problem solving. arXiv preprint arXiv:2506.02672 (2025)

    2025

  7. [15]

    In: Proceedings of the 62nd Annual Meeting of the AssociationforComputationalLinguistics(Volume1:LongPapers).pp.1932–1945 (2024)

    Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., et al.: Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In: Proceedings of the 62nd Annual Meeting of the AssociationforComputationalLi...

    2024

  8. [16]

    Nature590, 580 – 586 (2020),https://api.semanticscholar.org/ CorpusID:216552951

    Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K.O., Clune, J.: First return, then explore. Nature590, 580 – 586 (2020),https://api.semanticscholar.org/ CorpusID:216552951

    2020

  9. [17]

    In: Proceedings of the 41st International Conference on Machine Learning

    Gao, S., Ge, Q., Shen, W., Dou, S., Ye, J., Wang, X., Zheng, R., Zou, Y., Chen, Z., Yan, H., et al.: Linear alignment: a closed-form solution for aligning human preferences without tuning and feedback. In: Proceedings of the 41st International Conference on Machine Learning. p...

    2024

  10. [18]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  11. [19]

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y.K., Luo, F., Xiong, Y., Liang, W.: Deepseek-coder: When the large language model meets programming – the rise of code intelligence (2024),https: //api.semanticscholar.org/CorpusID:267211867

    2024

  12. [20]

    In: Vanschoren, J., Yeung, S

    Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., Steinhardt, J.: Measuring coding challenge competence with APPS. In: Vanschoren, J., Yeung, S. (eds.) Pro- ceedings of the Neural Information Processing Systems ...

    2021

  13. [21]

    ArXiv abs/2103.03874 (2021), https://api.semanticscholar.org/CorpusID: 232134851

    Hendrycks,D.,Burns,C.,Kadavath,S.,Arora,A.,Basart,S.,Tang,E.,Song,D.X., Steinhardt, J.: Measuring mathematical problem solving with the math dataset. ArXiv abs/2103.03874 (2021), https://api.semanticscholar.org/CorpusID: 232134851

    2021 arXiv

  14. [22]

    arXiv preprint arXiv:2212.10403 (2022)

    Huang, J., Chang, K.C.C.: Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403 (2022)

    2022 arXiv

  15. [23]

    arXiv preprint arXiv:2410.01044 (2024)

    Jiang, D., Wang, G., Lu, Y., Wang, A., Zhang, J., Liu, C., Van Durme, B., Khashabi, D.: Rationalyst: Pre-training process-supervision for improving reason- ing. arXiv preprint arXiv:2410.01044 (2024)

    2024 arXiv

  16. [24]

    arXiv preprint arXiv:2310.06452 (2023)

    Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., Raileanu, R.: Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452 (2023)

    2023 arXiv

  17. [25]

    arXiv preprint arXiv:2502.21321 (2025)

    Kumar, K., Ashraf, T., Thawakar, O., Anwer, R.M., Cholakkal, H., Shah, M., Yang, M.H., Torr, P.H., Khan, S., Khan, F.S.: Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321 (2025)

    2025 arXiv

  18. [26]

    Information Fusion85, 1–22 (2022)

    Ladosz, P., Weng, L., Kim, M., Oh, H.: Exploration in deep reinforcement learning: A survey. Information Fusion85, 1–22 (2022)

    2022

  19. [27]

    Li, R., Allal, L.B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al.: Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  20. [28]

    arXiv preprint arXiv:1701.07274 (2017)

    Li, Y.: Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274 (2017)

    2017 arXiv

  21. [29]

    arXiv preprint arXiv:2307.04349 (2023)

    Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., Ye, D.: Rltf: Reinforcement learning from unit test feedback. arXiv preprint arXiv:2307.04349 (2023)

    2023 arXiv

  22. [30]

    Luo,Z.,Xu,C.,Zhao,P.,Sun,Q.,Geng,X.,Hu,W.,Tao,C.,Ma,J.,Lin,Q.,Jiang, D.:Wizardcoder:Empoweringcodelargelanguagemodelswithevol-instruct.arXiv preprint arXiv:2306.08568 (2023)

    2023 arXiv

  23. [31]

    arXiv preprint arXiv:2502.02523 (2025)

    Mercer, S., Spillard, S., Martin, D.P.: Brief analysis of deepseek r1 and it’s impli- cations for generative ai. arXiv preprint arXiv:2502.02523 (2025)

    2025 arXiv

  24. [32]

    In: 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA)

    Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., Abbeel, P.: Overcoming exploration in reinforcement learning with demonstrations. In: 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA). pp. 6292–6299 (2018). https://doi.org/10.1109/ICRA.2018.8463162

    2018

  25. [33]

    arXiv preprint arXiv:2407.11511 (2024)

    Plaat, A., Wong, A., Verberne, S., Broekens, J., van Stein, N., Back, T.: Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511 (2024)

    2024

  26. [34]

    arXiv preprint arXiv:2308.12950 (2023)

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al.: Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  27. [35]

    arXiv preprint arXiv:1511.05952 (2015)

    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)

    2015 arXiv

  28. [36]

    arXiv preprint arXiv:1506.02438 (2015)

    Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 (2015)

    2015 arXiv

  29. [37]

    arXiv preprint arXiv:1707.06347 (2017)

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  30. [38]

    5-thinking: Advancing superb reasoning models with reinforcement learning

    Seed, B., Chen, J., Fan, T., Liu, X., Liu, L., Lin, Z., Wang, M., Wang, C., Wei, X., Xu, W., et al.: Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914 (2025)

    2025

  31. [39]

    ArXiv abs/2402.03300 (2024), https://api.semanticscholar

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J.M., Zhang, M., Li, Y.K., Wu, Y., Guo, D.: Deepseekmath: Pushing the limits of mathematical reasoning in open lan- Improving RL Exploration for LLM Reasoning through Retrospective Replay 13 guage models. ArXiv abs/2402.03300 (2024), ...

    2024 arXiv

  32. [40]

    In: The 2023 Conference on Empirical Methods in Natural Language Processing

    Shen, W., Zheng, R., Zhan, W., Zhao, J., Dou, S., Gui, T., Zhang, Q., Huang, X.: Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In: The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  33. [41]

    arXiv preprint arXiv:2301.13816 (2023)

    Shojaee, P., Jain, A., Tipirneni, S., Reddy, C.K.: Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816 (2023)

    2023 arXiv

  34. [42]

    Advances in neural informa- tion processing systems12 (1999)

    Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. Advances in neural informa- tion processing systems12 (1999)

    1999

  35. [43]

    arXiv preprint arXiv:2405.03379 (2024)

    Tao, S., Shukla, A., Chan, T.k., Su, H.: Reverse forward curriculum learning for extreme sample and demonstration efficiency in reinforcement learning. arXiv preprint arXiv:2405.03379 (2024)

    2024 arXiv

  36. [44]

    Thrun,S.B.:Efficientexploration inreinforcement learning.Tech.rep., USA(1992)

    1992

  37. [45]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  38. [46]

    arXiv preprint arXiv:2211.14275 (2022)

    Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., Higgins, I.: Solving math word problems with process-and outcome- based feedback. arXiv preprint arXiv:2211.14275 (2022)

    2022 arXiv

  39. [47]

    arXiv preprint arXiv:2401.06080 (2024)

    Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al.: Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080 (2024)

    2024 arXiv

  40. [48]

    arXiv preprint arXiv:2503.10460 (2025)

    Wen, L., Cai, Y., Xiao, F., He, X., An, Q., Duan, Z., Du, Y., Liu, J., Tang, L., Lv, X., et al.: Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460 (2025)

    2025 arXiv

  41. [49]

    Machine learning8, 229–256 (1992)

    Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning8, 229–256 (1992)

    1992

  42. [50]

    arXiv preprint arXiv:2506.04065 (2025)

    Wu, M., Qian, Q., Liu, W., Wang, X., Huang, Z., Liang, D., Miao, L., Dou, S., Lv, C., Wang, Z., et al.: Progressive mastery: Customized curriculum learning with guided prompting for mathematical reasoning. arXiv preprint arXiv:2506.04065 (2025)

    2025 arXiv

  43. [51]

    In: International Conference on Machine Learning

    Xi, Z., Chen, W., Hong, B., Jin, S., Zheng, R., He, W., Ding, Y., Liu, S., Guo, X., Wang, J., et al.: Training large language models for reasoning through reverse cur- riculum reinforcement learning. In: International Conference on Machine Learning. pp. 54030–54048. PMLR (2024)

    2024

  44. [52]

    arXiv preprint arXiv:2109.06668 (2021)

    Yang, T., Tang, H., Bai, C., Liu, J., Hao, J., Meng, Z., Liu, P., Wang, Z.: Ex- ploration in deep reinforcement learning: a comprehensive survey. arXiv preprint arXiv:2109.06668 (2021)

    2021 arXiv

  45. [53]

    arXiv preprint arXiv:2505.17793 (2025)

    Zang, J., Ning, M., Wei, Y., Dou, S., Zhang, J., Mo, N., Li, B., Gui, T., Zhang, Q., Huang,X.:Compressionhacking:Asupplementaryperspectiveoninformaticsmet- ric of language models from geometric distortion. arXiv preprint arXiv:2505.17793 (2025)

    2025

  46. [54]

    arXiv preprint arXiv:1712.01275 (2017)

    Zhang, S., Sutton, R.S.: A deeper look at experience replay. arXiv preprint arXiv:1712.01275 (2017)

    2017 arXiv

  47. [55]

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models

  48. [56]

    In: NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following (2023)

    Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Zhou, Y., Xiong, L., et al.: Delve into ppo: Implementation matters for stable rlhf. In: NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following (2023)

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.