Pith. sign in

REVIEW 4 major objections 7 minor 4 cited by

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that rewarding only the self-reflection tokens, rather than the answer tokens, trains a language model to self-correct on verifiable tasks, yielding large accuracy gains and letting small models beat models ten times…

desk verdict A clean idea and solid APIGen results, but the Countdown verifier ambiguity and missing ablation mean the central claim is under-supported. read the letter →

arxiv 2505.24726 v1 pith:OOGVHZIZ submitted 2025-05-30 cs.CL

Shelly Bensal , Umar Jamil , Christopher Bryant , Melisa Russak , Kiran Kamble , Dmytro Mozolevskyi , Muayad Ali , Waseem AlShikh This is my paper
classification cs.CL
keywords self-reflectionreinforcementlearningGRPOself-improvementfunctioncallingCountdownmathequationsverifierfeedbackself-correction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Reflect, Retry, Reward, a training loop in which a language model that fails a task must write a self-reflection about its mistake and then retry the task. If the retry succeeds, reinforcement learning (GRPO) rewards only the tokens written during the self-reflection, leaving the answer tokens unrewarded. The authors claim that this teaches the model to self-correct in a task-agnostic way, needing only a binary success/failure signal and no synthetic data or larger teacher model. Across function calling and Countdown equation writing, they report gains of up to 34.7% and 18.1% respectively, and show that small trained models can outperform models ten times larger.

What carries the argument

The machinery is a two-stage rollout combined with a reward mask that restricts credit to the reflection: the model answers, then after a failure writes a self-reflection, then retries; Group Relative Policy Optimization's advantage is set to zero for every token except those produced during the self-reflection phase. This mask is what makes the method task-agnostic, since the model is never rewarded for solving the task directly, only for generating reflections that convert a failed attempt into a successful one. The implementation extends a standard GRPO trainer with a second generation step that keeps the original token mask intact.

What would settle it

Train the same models on Countdown with a stricter verifier that explicitly rejects equations unless every provided number appears exactly once, then compare the post-training error breakdown to Table 5; if wrong-number errors remain low instead of rising as they do for Qwen-2.5-7B (from 39.1% to 55.6%), the original verifier was incomplete and part of the reported gains came from exploiting that gap.

Watch

Extended reading notes

Core claim

The central discovery is that a model's self-reflective commentary can be the learning target instead of the task answer. When a verifier flags an attempt as wrong, the model writes a short self-reflection and then answers again with that reflection in context; if the second try passes, GRPO assigns positive advantage only to the self-reflection tokens. The paper claims this shapes the model to produce reflections that reliably turn failures into successes, and that this transfers to first-attempt accuracy because the model internalizes general reasoning rather than memorizing answers. The experiments show the training raises both first- and second-attempt accuracy on the APIGen function calling and Countdown math equation tasks, and that trained 1.5B-7B models can exceed untrained models that are ten times larger.

Load-bearing premise

The method's training signal is only as sound as the verifier: for the math task the paper states only that the equation must evaluate to the target answer, and if the verifier also does not enforce that each provided number is used exactly once, the reward can be satisfied by equations that break the task rules.

Editorial extensions

If this is right

  • Models trained this way also improve on their first attempt, without generating a reflection, suggesting the training strengthens general reasoning rather than only reflection skills.
  • The method needs only a binary verifier, so it applies to tasks where correctness is easy to check (valid JSON, executable code, an equation that evaluates to a target) even when ground-truth labels or synthetic data are unavailable.
  • Small trained models in the 1.5B-8B range can beat same-family models ten times larger, which could lower the cost of specialized task performance.
  • Benchmarks for language understanding, math, and commonsense stay nearly unchanged after training, indicating low catastrophic forgetting.
  • The approach fails for models with very weak baselines, so a minimum level of task ability, reflection ability, and learnability is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: A testable extension is to verify the Countdown verifier checks that every provided number is used exactly once; if it only checks the arithmetic result, the reward can be gamed, and the paper's error analysis (wrong-number errors rising from 39.1% to 55.6% for Qwen-2.5-7B) is consistent with that gap.
  • Editorial inference: If rewarding reflection alone is truly task-agnostic, the same loop could be chained across tasks, letting a model improve on a second task using only the reflection skill learned on the first.
  • Editorial inference: The observed compression of reflections (long, redundant text becoming short and terse) raises the question of whether the model learns genuine self-correction or a heuristic pattern such as 'recheck arithmetic' that happens to help these tasks; the paper does not interrogate the mechanism.
  • Editorial inference: One can directly test the causal role of the reward mask by running an ablation that rewards the full second-attempt output instead of only the reflection tokens; the paper does not report such a comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Reflect, Retry, Reward (RRR), a two-stage method in which a language model, after failing a verifiable task, generates a self-reflection, retries the task with that reflection in context, and is trained with GRPO so that only the self-reflection tokens receive reward when the retry succeeds. Training is performed only on a dataset of failed queries. The authors evaluate the method on APIGen function calling and Countdown math equation solving for several 1.5B-8B models, reporting first-try and second-try accuracy gains, comparisons against larger vanilla models, a qualitative analysis of reflection quality, an error analysis, and a catastrophic-forgetting study on general benchmarks.

Significance. If the central claim is upheld, the method would be a valuable contribution: it requires only binary verifier feedback, avoids synthetic data generation by larger teachers, and appears to improve first-attempt accuracy even when test-time reflection is not used. The paper has real strengths: it evaluates on held-out test sets (12,000 APIGen and 15,000 Countdown samples), restricts base models to those released before the datasets, uses exact-match verification for APIGen, and includes a benchmark-based forgetting analysis. However, the significance is conditional on resolving a verifier ambiguity for Countdown and on providing statistical and ablation evidence for the main mechanism. The Countdown experiments provide some of the largest reported gains, so their validity is load-bearing for the paper's overall claim.

major comments (4)
  1. [§4.2, Table 5] The Countdown verifier is not fully specified. The task definition in §4.2 requires that "the equation ... uses all the numbers once (in any order) and ... evaluates to the target number," but the validator description says only that "we required the generated equation to match the target answer in the prompt." It is not stated whether the training and evaluation verifier also checks that each provided number is used exactly once. Table 5 provides circumstantial evidence that it may not: for Qwen-2.5-7B Instruct, wrong-number errors increase from 39.1% to 55.6% after training while missed-target errors collapse from 25.4% to 2.7%, exactly the pattern expected from a reward that ignores the number-usage constraint. Because Countdown drives some of the largest gains (e.g., Qwen-2.5-1.5B from 6.0% to 34.9% first-try in Table 2), the Countdown results may reflect exploitation of a weak verifier rather than improved equation solving. The authors must clarify whether the verifier enforces the full constraint, and if it does not, they must rerun the Countdown experiments with a full-constraint verifier or remove the affected claims.
  2. [§5, Tables 1-5] All reported accuracies come from a single training run and a single test evaluation, with no error bars, confidence intervals, or multiple seeds. Several central comparisons are small in absolute terms, for example Llama-3.1-8B first-try APIGen accuracy of 68.7% vs. 64.9% and several entries in Table 3, so without variance estimates it is impossible to know whether these differences are statistically reliable. The authors should report at least bootstrap confidence intervals over test items, and ideally results over multiple training seeds, for the headline claims in Tables 1 and 2.
  3. [§3, §4.4] The central mechanism is that rewarding only self-reflection tokens improves reflection quality and, indirectly, task performance. This claim is not isolated by any ablation. The paper does not compare against rewarding all tokens of the retry trajectory, rewarding only the second-attempt answer tokens, or using the same failure dataset with standard behavior cloning. Without such comparisons, the observed gains could come from the retry structure, the failure-only training distribution, or the GRPO update itself rather than from the specific choice to mask out answer-token advantages. The assertion in §4.3 that training on failures alone is "functionally equivalent" to training on the full distribution is also asserted rather than tested. An ablation isolating the reflection-token mask is needed to support the paper's main attribution.
  4. [§4.4] The paper states that models were "evaluated at their convergence point" but never defines the convergence criterion. Training steps vary from 100 for Llama-3.1-8B on APIGen to 1,750 steps maximum, and no validation curve, checkpoint-selection rule, or early-stopping threshold is given. This makes the reported results difficult to reproduce and raises the risk of favorable checkpoint selection. The authors should specify the convergence criterion precisely, e.g., a patience-based plateau in validation success rate, and report the selected checkpoint and training curve for each model.
minor comments (7)
  1. [§5.2] The phrase "self-inflection training" should be corrected to "self-reflection training."
  2. [§6] The conclusion says "Countdown match equation dataset," which appears to be a typo for "Countdown math equation dataset."
  3. [Appendix B, Table 5] The "wrong numbers" error category is described as "an equation that uses numbers outside of the provided ones," but the table's discussion sometimes refers to using the "correct numbers," which could mean either using exactly the provided set or merely avoiding unprovided numbers. The category definition should be disambiguated, since it is central to the verifier discussion.
  4. [References] The APIGen reference in the bibliography is listed as "and 1 others" after a long author list; the citation should be completed properly.
  5. [§4.3] The number of sampled responses per query for each model is described only qualitatively ("up to 64 responses ... depending on model size"), and no table is provided. This affects the composition of the failure dataset and should be reported per model.
  6. [§4.2] The statement that evaluated models were "released or have a knowledge cutoff before the Countdown dataset was made publicly available (January 2025)" is ambiguous. The authors should list the exact model versions and their knowledge-cutoff dates used in the Countdown experiments.
  7. [Figure 2] The qualitative claim about shorter and clearer reflections after training is supported only by a single undisplayed example. Including the actual paired reflection in the text or an appendix would make the claim checkable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are measured on held-out test sets against external verifiers, and no fitted parameter or self-citation is masquerading as a prediction.

full rationale

This paper reports an empirical RL method rather than a derivation from first principles. The central claim is that rewarding only self-reflection tokens, when the reflected second attempt succeeds, improves downstream task performance. Training uses GRPO with rewards from task verifiers, and evaluation is on held-out test sets (APIGen: 12,000 samples; Countdown: 15,000 samples). The reported gains are differences between vanilla and trained models on the same verifier-based metric, so there is no fitted parameter or training-set statistic that is later renamed as a prediction. The only self-citation, to Writer.com for the Palmyra X4 baseline, is a baseline model name and not load-bearing for the method; the comparison numbers are measured in the paper rather than imported from the citation. The Section 2.1 statement that the approach is 'guaranteed to improve or maintain performance, since corrections are applied only to initially incorrect examples' is a tautology of the two-attempt protocol, but it is not used to derive the reported first-try improvements. The Section 4.2 Countdown verifier description ('we required the generated equation to match the target answer in the prompt') may be incomplete, and the Table 5 error shift for Qwen-2.5-7B could indicate reward hacking; however, a possibly noisy reward is an empirical input signal, not an output that reduces to the input by construction. Thus, no circular step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central empirical claim rests on the GRPO training setup and the verifier's correctness. It does not introduce new theoretical entities, but it relies on several unverified assumptions about credit assignment and dataset construction.

free parameters (5)
  • KL divergence coefficient = 0.001
    Set by the authors for all experiments (Section 4.4); not derived from data, and no sensitivity analysis is reported.
  • Learning rate schedule = 5e-7 with cosine annealing and warmup ratio 0.03
    Chosen by the authors; no sensitivity analysis is reported.
  • Effective batch size = 256 failures
    Chosen for training; no justification is given.
  • Training steps to convergence = up to 1,750; evaluated at convergence point
    The stopping criterion is not defined, so the convergence point may be a source of checkpoint selection bias.
  • Sampling responses per query for failure dataset = up to 64, depending on model size
    Used to construct the training set of failures; the exact number per model is not reported.
assumptions (4)
  • standard math GRPO correctly optimizes the policy given the reward signal
    The paper adopts GRPO as its RL algorithm and relies on its correctness without re-deriving it.
  • domain assumption The binary verifier provides an accurate success/failure signal for the task
    The method's reward depends entirely on the verifier; Section 4.2 describes the Countdown verifier ambiguously, and a weak verifier could reward partial solutions.
  • ad hoc to paper Training only on a dataset of failures is functionally equivalent to training on the full task distribution for learning self-reflections
    Section 4.3 claims this equivalence without proof; it is a distributional assumption that could bias learned reflections.
  • ad hoc to paper Rewarding only self-reflection tokens (zero advantage for answer tokens) causes the model to improve reflection ability rather than task-specific answer generation
    Section 3 states this design, but no ablation is provided to justify that this credit assignment is the cause of the observed improvements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning." pith.science (2026). https://pith.science/paper/OOGVHZIZ

@misc{pith2026250524726,
  author       = {Pith},
  title        = {Pith review of: Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOGVHZIZ}},
  note         = {Machine review of arXiv:2505.24726}
}
read the original abstract

We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.

Figures

Figures reproduced from arXiv: 2505.24726 by the authors.

Figure 1
Figure 1. Reflect, Retry, Reward Mechanism The model is first prompted to complete a task based on a user query. If the initial response is correct, the process stops. If not, the model is prompted to generate a self-reflection on how to improve. The model then retries the same task, this time with its self-reflection included, and the new answer is evaluated. If the second attempt succeeds, the model learns that it generated… view at source ↗
Figure 2
Figure 2. Better Self-Reflections We observe that reflections generated by vanilla models tend to be long, confusing, and redundant, whereas GRPO fine-tuned models produce much shorter, clearer, and more generalisable reflections. In terms of Countdown, it is first worth noting that performance was lower across the board, and the vanilla Llama models in particular (both Llama-3.1 and Llama-3.2) really struggled to complete th… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    ReSum trains LLMs via RLVR to self-summarize reasoning trajectories, yielding 4% average performance gains and 18.6% shorter rollouts through contrastive rollout branches.

  2. Unified Audio Intelligence Without Regressing on Text Intelligence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified 30B MoE audio-text LLM achieves state-of-the-art audio understanding, generation, and speech tasks while preserving text reasoning comparable to its text-only backbone.

  3. Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Local retry from the earliest erroneous turn, with pivotal credit isolation and reflection rewards, improves Pass@k and reduces interaction cost over MetaRL and GiGPO on agent and search tasks.

  4. Self-Reflective Generation at Test Time

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SRGen improves LLM math reasoning by detecting high-entropy tokens and injecting a small corrected vector into the hidden state at those points during decoding, without training.

Reference graph

Works this paper leans on

51 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A hi...

    arXiv 2024

  4. [4]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. https://aclanthology.org/2024.eacl-srw.17/ Large language models for mathematical reasoning: Progresses and challenges . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 225--237, St....

    2024

  5. [5]

    Nicholas Asher, Swarnadeep Bhar, Akshay Chaturvedi, Julie Hunter, and Soumya Paul. 2023. https://doi.org/10.18653/v1/2023.starsem-1.22 Limits for learning with language models . In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 236--248, Toronto, Canada. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.starsem-1.22 2023

  6. [6]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 2022. https://api.semanticscholar.org/CorpusID:248118878 Traini...

    arXiv 2022

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  8. [8]

    Alan Dao and Thinh Le. 2025. https://arxiv.org/abs/2504.11001 Rezero: Enhancing llm search ability by trying one-more-time . Preprint, arXiv:2504.11001

    arXiv 2025

Show all 51 references
  1. [9]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [11]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  4. [12]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    2015 arXiv

  5. [13]

    Jie Huang and Kevin Chen-Chuan Chang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.67 Towards reasoning in large language models: A survey . In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049--1065, Toronto, Canada. Association for Comput...

    2023 doi

  6. [14]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representations

    2024

  7. [15]

    James Huckle and Sean Williams. 2025. Easy Problems that LLMs Get Wrong . In Advances in Information and Communication, pages 313--332, Cham. Springer Nature Switzerland

    2025

  8. [16]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.123 Towards mitigating LLM hallucination via self reflection . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827-...

    2023 doi

  9. [17]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. https://arxiv.org/abs/2406.00515 A survey on large language models for code generation . Preprint, arXiv:2406.00515

    2024 arXiv

  10. [18]

    Geunwoo Kim, Pierre Baldi, and Stephen Marcus McAleer. 2023. https://openreview.net/forum?id=M6OmjAZ4CX Language models can solve computer tasks . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  11. [19]

    Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. 2024. https://arxiv.org/abs/2309.10105 Understanding catastrophic forgetting in language models via implicit inference . Preprint, arXiv:2309.10105

    2024 arXiv

  12. [20]

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2024...

    2024 arXiv

  13. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  14. [22]

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. https://arxiv.org/abs/2503.23383 Torl: Scaling tool-integrated rl . Preprint, arXiv:2503.23383

    2025 arXiv

  15. [23]

    Zhizhong Li and Derek Hoiem. 2016. https://api.semanticscholar.org/CorpusID:4853851 Learning without forgetting . IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:2935--2947

    2016

  16. [24]

    Liping Liu, Chunhong Zhang, Likang Wu, Chuang Zhao, Zheng Hu, Ming He, and Jianping Fan. 2025. https://arxiv.org/abs/2503.00902 Instruct-of-reflection: Enhancing large language models iterative reflection capabilities via dynamic-meta instruction . Preprint, arXiv:2503.00902

    2025 arXiv

  17. [25]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. 2024 a . https://openreview.net/forum?id=OJaWBhh61C Best practices and lessons learned on synthetic data . In First Conference on Langu...

    2024

  18. [26]

    Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, and 1 others. 2024 b . Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets . Advances in Neural Information Processing S...

    2024

  19. [27]

    David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6470–6479, Red Hook, NY, USA. Curran Associates Inc

    2017

  20. [28]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534--46594

    2023

  21. [29]

    Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. 2025 a . https://arxiv.org/abs/2504.15466 Learning adaptive parallel reasoning with language models . Preprint, arXiv:2504.15466

    2025 arXiv

  22. [30]

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. 2025 b . Tinyzero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-05-05

    2025

  23. [31]

    Vicky Zhao, Conghui He, and Lijun Wu

    Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H. Vicky Zhao, Conghui He, and Lijun Wu. 2025 c . https://arxiv.org/abs/2503.17439 Lemma: Learning from errors for mathematical advancement in llms . Preprint, arXiv:2503.17439

    2025 arXiv

  24. [32]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. https://arxiv.org/abs/2302.12813 Check your facts and try again: Improving large language models with external knowledge and automate...

    2023 arXiv

  25. [33]

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. https://arxiv.org/abs/2504.13958 Toolrl: Reward is all tool learning needs . Preprint, arXiv:2504.13958

    2025 arXiv

  26. [34]

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. 2024. https://arxiv.org/abs/2407.18219 Recursive introspection: Teaching language model agents how to self-improve . Preprint, arXiv:2407.18219

    2024 arXiv

  27. [35]

    Matthew Renze and Erhan Guven. 2024. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682

    2024 arXiv

  28. [36]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . Preprint, arXiv:1707.06347

    2017 arXiv

  29. [37]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . Preprint, arXiv:2402.03300

    2024 arXiv

  30. [38]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634--8652

    2023

  31. [39]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  32. [40]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and...

    2024 arXiv

  33. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

    2022

  34. [42]

    Writer.com . 2024. Palmyra x4 | tool calling llm. https://writer.com/llms/palmyra-x4/. Accessed: 2025-05-29

    2024

  35. [43]

    Zongqian Wu, Baoduo Xu, Ruochen Cui, Mengmeng Zhan, Xiaofeng Zhu, and Lei Feng. 2025. https://arxiv.org/abs/2412.10827 Rethinking chain-of-thought from the perspective of self-training . Preprint, arXiv:2412.10827

    2025 arXiv

  36. [44]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. https://arxiv.org/ab...

    2024 arXiv

  37. [45]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025. https://arxiv.org/abs/...

    2025 arXiv

  38. [46]

    Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.296 Re3: Generating longer stories with recursive reprompting and revision . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, page...

    2022 doi

  39. [47]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://arxiv.org/abs/1905.07830 Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830

    2019 arXiv

  40. [48]

    Jixiao Zhang and Chunsheng Zuo. 2025. https://arxiv.org/abs/2504.09696 Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models . Preprint, arXiv:2504.09696

    2025

  41. [49]

    Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. 2024. https://doi.org/10.18653/v1/2024.acl-long.197 Self-contrast: Better reflection through inconsistent solving perspectives . In Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  42. [50]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2025. https://arxiv.org/...

    2025 arXiv

  43. [51]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Internat...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.