Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

SRA-MCTS: Self-driven Reasoning Augmentation with Monte Carlo Tree Search for Code Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A code model can be its own teacher: MCTS-written thinking, self-scored and fine-tuned, lifts complex-benchmark pass@1 without extra supervision, with an 11-point gain on MBPP-Complex for the 8B model.

desk verdict A solid self-improvement pipeline with a well-designed ablation, but the headline gain needs split sizes and error bars, and the closest MCTS baselines are missing. read the letter →

arxiv 2411.11053 v5 pith:COOPEH2A submitted 2024-11-17 cs.CL cs.AI

Bin Xu , Yiguan Lin , Yinghao Li , Yang Gao This is my paper · ORCID
classification cs.CLcs.AI
keywords codegenerationMonteCarloTreeSearchself-improvementreasoningaugmentationchain-of-thoughttree-of-thoughtself-evaluationsupervisedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a single instruction-tuned LLM can close its own improvement loop in code generation: Monte Carlo Tree Search generates multi-step natural-language 'thinking' for a question, the same model scores each step, the final thinking guides code generation, and the question-thinking-code triple is used for supervised fine-tuning. It claims that this self-driven pipeline improves pass@1 on MBPP across 2B, 8B, and 14B models without any additional supervisory signal, with the largest gains on the Complex (medium and hard) split, and that it beats CoT and ToT as data generators when the test task style matches the training set. The headline number is an 11-point improvement on MBPP-Complex for Meta-Llama-3.1-8B-Instruct. The paper also reports that adding its thinking to ground-truth code is robust, while CoT and ToT thinking degrade when the code is swapped, and that self-generated data often beats data distilled from a 70B teacher.

What carries the argument

The search tree of reasoning steps is the load-bearing object. Each node stores a state (the question plus all steps so far), an action (the next step, produced by sample decoding), a reward (the model's own score), and a reflection that guides the next expansion. Selection uses UCB1 to balance exploitation of high-scoring nodes against exploration of unvisited ones; expansion samples up to three distinct next steps; evaluation scores each step with a progressive rubric that checks single-step correctness, solution coherence, solution completeness, and solution correctness in that order; reflection either emits an <end> tag when the question is judged solved or a short hint for the next step; and backpropagation reweights parent rewards from the children's visit-weighted scores. Repeating this loop for up to five iterations assembles a 'thinking' that the same model then converts to code.

What would settle it

Take the same SRA-MCTS-generated thinking and re-rank the tree's branch selection with an external oracle, such as a strong closed-source model or execution of the final code on held-out test cases, then fine-tune on the oracle-selected paths. If oracle-ranked thinking yields no higher pass@1 than self-ranked thinking, the self-evaluation mechanism is not the active ingredient; if it yields higher pass@1, the paper's claim that self-evaluation alone drives the improvement is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that high-quality intermediate reasoning is a separable and trainable ingredient of code generation: SRA-MCTS produces thinking that is more diverse and more reliably correct than the model's chain-of-thought or tree-of-thought output, and fine-tuning on that thinking improves the model's code-generation pass rate, especially on complex problems. The experimental claim is that these gains appear at every tested scale, grow with model size, and are largest on tasks whose style matches the training set (MBPP-style generate-from-scratch, not HumanEval-style function completion). The authors interpret this as evidence that small models can improve themselves through self-generated reasoning data without distillation from larger models.

Load-bearing premise

The load-bearing premise is that the model's self-scores of its own reasoning steps are accurate enough for UCB1 to steer the search toward good thinking, so if the evaluator is biased, the tree selects confidently wrong branches and the training data inherits the error.

Editorial extensions

If this is right

  • If the experimental results hold, a code LLM can bootstrap its own reasoning-augmented training data, removing the need for a larger teacher model to provide demonstrations.
  • The method's largest gains on medium and hard questions suggest that reasoning-augmented data matters most precisely where models currently fail.
  • The robustness result with ground-truth code means the thinking itself carries transferable value, not just the code it was paired with.
  • Pass@10 also improves on MBPP, indicating that SRA-MCTS thinking increases the diversity of viable solutions, not just the best attempt.
  • The consistent improvements across 2B, 8B, and 14B models indicate the pipeline is scale-robust within the tested range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same self-search-then-finetune loop could transfer to other generation tasks where intermediate reasoning can be scored by the model itself, such as math word problems or tool-use planning; the paper does not test that transfer.
  • The self-evaluation bias the authors acknowledge suggests that a better scoring signal, such as compiler feedback on intermediate code or a learned reward model, could enlarge the gains; this is speculation beyond the paper's experiments.
  • The observed scale-dependence hints at a self-improvement plateau: if evaluation quality limits data quality, very small models may gain little or nothing, and the paper's 2B results already show a smaller average increment than the 14B results.
  • A direct test of the thinking's generality would be to take SRA-MCTS thinking generated by one model and use it to prompt a different model for code generation; the paper's variant experiment swaps only the code, not the thinking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SRA-MCTS, a self-driven reasoning augmentation method that uses Monte Carlo Tree Search to generate natural-language 'thinking' paths, which are then concatenated with questions and used to prompt the model to generate code; the resulting (question, thinking, code) triples are used for supervised fine-tuning. The method is evaluated on MBPP, Human-Eval, MBPP+, and Human-Eval+ across three model scales (gemma-2-2b, Meta-Llama-3.1-8B, Qwen2.5-14B), reporting pass@1 and pass@10. The central claim is that SRA-MCTS improves average pass@1 across model scales without additional supervision, with a headline 11-point improvement on the MBPP-Complex split for Meta-Llama-3.1-8B, and that the generated thinking is higher quality and more robust than CoT or ToT thinking. The paper also includes ablations showing that thinking matters, that SRA-MCTS thinking is more stable, and that self-generated data can outperform data distilled from a larger model.

Significance. If the results are reliable, SRA-MCTS is a simple and potentially useful contribution: it offers a fully self-supervised pipeline for generating reasoning-augmented fine-tuning data for code generation, with public code and data, and it directly addresses the question of whether small models can improve themselves without external supervision. The ablations on the existence, variants, and quality of thinking are well designed and provide valuable evidence for the mechanism, and the comparison with distillation (Table 3) is a welcome attempt to isolate the self-generation component. The central limitation is that the headline improvements rest on single runs with no statistical uncertainty, and the self-evaluation loop is a plausible source of bias that is not validated.

major comments (4)
  1. [Section 4.4 and Table 1] The central claim of consistent performance improvements across model scales is not statistically supported. The paper does not report the number of random seeds, confidence intervals, or significance tests for any of the pass@1 or pass@10 numbers in Table 1. Since the Complex split sizes are never given, the headline 11.32-point improvement for Meta-Llama-3.1-8B on MBPP-Complex could correspond to a handful of additional solved problems; on a split of 53 questions this is six examples, and for gemma-2-2b (5.66 points) it is three. Without variance estimates, the single-run deltas for small models (e.g., the 2B model's -0.50 on full MBPP) are within plausible run-to-run noise. The paper should report split sizes, the number of seeds, and at minimum paired bootstrap or sign-test results, or the claims should be rephrased as exploratory.
  2. [Section 3.1, Eq. (4), and Section 5.1] The method's load-bearing premise is that the same LLM used to generate reasoning steps can also score those steps reliably enough for MCTS selection, but this self-evaluation loop is neither validated nor analyzed. Eq. (4) uses the model as its own evaluator with a progressive rubric, and the paper itself concedes in Section 5.1 that 'the evaluator may overemphasize unnecessary overthinking steps.' Since the entire data-generation pipeline depends on these scores, the authors should provide evidence that the reward model is at least roughly calibrated, for example by comparing self-scores against code correctness on a held-out set or against an external judge (e.g., GPT-4o), or by conducting a sensitivity analysis in which the evaluator is replaced or the scoring rubric is varied. Without this, the MCTS selectivity may be driven by confidently wrong branches, which would undermine the proposed mechanism.
  3. [Abstract, Section 4.5, and Table 1] The abstract's claim of 'performance improvements across different model scales' is contradicted by the HumanEval and Human-Eval+ results, where the method declines on several metrics. For example, Meta-Llama-3.1-8B's Human-Eval pass@1 drops from 47.46 to 44.07 under SRA-MCTS, and the 2B model loses 1.83 points on Human-Eval+ pass@1. The paper attributes this to task-style mismatch, but that explanation is not tested. The claim should either be scoped to MBPP-style benchmarks or supported by additional experiments isolating the task-style effect. In addition, the 'Average Increment' column in Table 1 averages over overlapping Full and Complex splits of the same benchmarks, which double-counts the easier questions and can inflate the apparent improvement; this should be clarified or the average should be restricted to non-overlapping splits.
  4. [Section 4.4 and Section 4.5] The evaluation protocol for pass@10 is underspecified, which is particularly relevant because the abstract highlights improvements in diversity metrics. The paper does not report the number of samples per problem, the generation temperature for the test set, the exact computation of the unbiased pass@k estimator, or the number of independent runs. Without these details, the pass@10 results in Table 1 cannot be reproduced or compared with the literature, and the diversity claim is not verifiable. Please provide the full sampling configuration and, ideally, the raw sample counts.
minor comments (6)
  1. [Equation (1)] The UCB1 formula is typeset ambiguously as 'r 2 · ln N n'; it should be written with clear parentheses, e.g., as r + c * sqrt(2 * ln(N) / n), to avoid confusion about whether the 2 is inside or outside the square root.
  2. [Section 4.1] The decontamination procedure is described as '10-gram level duplicate detection' with a threshold of 0.3, but the similarity measure is not specified; please state whether this is Jaccard similarity, overlap, or another metric, and describe how n-grams are extracted (e.g., token-based or character-based).
  3. [Table 1] The 'Average Increment' column should state the exact set of metrics and splits being averaged, and the fact that Full and Complex splits overlap should be explicitly acknowledged in the table caption.
  4. [Section 4.2] The ToT baseline is described as depth-first search with max depth 4 and branching factor 3, but the paper does not explain how the LLM proposes and evaluates intermediate steps in the ToT setup; additional implementation details are needed for reproducibility.
  5. [Section 4.3] The GPT-4o-based difficulty classification is not described in sufficient detail; please report the prompt used, the number of easy/medium/hard questions per benchmark, and any manual verification of the split quality.
  6. [Section 3.1 and Section 5.1] There are several minor wording and typographical issues, including a missing period after 'maximum of 5 retries' and a misspelling of 'reflection' as 'ref lection' in the Reflection paragraph.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical benchmark result evaluated on external test sets, not a derivation that reduces to its own inputs.

full rationale

The paper's central claim is empirical: SRA-MCTS generates thinking and code with the model itself, the data are used for supervised fine-tuning, and performance is measured on external benchmarks (HumanEval, MBPP, EvalPlus). The only self-referential mechanism is the model acting as its own evaluator in Eq. (4) ('We use the LLM itself, which generates the steps during the expansion phase, as the evaluator to score the new nodes'), and this could bias MCTS branch selection, but it is not a logical circularity: the final pass@1 numbers are computed against held-out test cases that are not used in training or in the self-evaluation loop. The paper does not fit a parameter to the test set and then call it a prediction, nor does it define its key quantities in terms of the target outcome. The reported '11-point improvement on the MBPP-Complex dataset' is a measurement on a public benchmark, not a consequence of an equation that assumes the improvement. Self-citations are absent from the load-bearing argument; the method builds on standard MCTS (Auer et al., 2002; Coulom, 2006) and on prior work by other authors (ReST-MCTS*). The acknowledged risk that 'the evaluator may overemphasize unnecessary overthinking steps' is a limitation affecting internal scoring quality, not a circularity in the evaluation of the final claim. Therefore, no circular step is exhibited, and the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its central claim rests on hand-chosen hyperparameters and on domain assumptions about self-evaluation reliability, MCTS budget sufficiency, transfer from LeetCode to benchmarks, and decontamination adequacy.

free parameters (6)
  • backpropagation mixing weight alpha = 0.5
    Section 4.4 states 'the alpha value in the backpropagation phase is manually set to 0.5'; it balances original node reward and child-node increment in Eq. (7).
  • UCB1 exploration constant c = 0.5
    Section 3.1 Selection, Eq. (1), sets c = 0.5; it controls exploration versus exploitation in node selection.
  • number of expanded child nodes per step = 3
    Section 3.1 Expansion: 'we set the number of expanded nodes to 3' to manage computational cost; it affects the diversity of reasoning paths.
  • total iteration limit for SRA-MCTS = 5
    Section 4.4: 'the total iteration limit is set to 5'; it caps the length of the generated reasoning path.
  • generation temperature and top-p in expansion phase = temperature 0.9, top-p 0.98
    Section 4.4: these sampling parameters control the diversity of generated steps and are hand-chosen.
  • progressive scoring rubric intervals = sequential integer score intervals for four criteria
    Section 3.1 Evaluation describes a hand-designed rubric (single-step correctness, solution coherence, completeness, correctness) with no validation against external labels.
assumptions (4)
  • domain assumption The LLM's self-evaluation scores are a reliable enough proxy for reasoning-step quality to guide MCTS selection.
    Section 3.1 Evaluation uses Eq. (4) with the same model as scorer, and Section 5.1 admits the evaluator can overemphasize overthinking, so reliability is load-bearing.
  • domain assumption A shallow MCTS budget (3 expanded nodes, 5 iterations) explores a sufficiently diverse and correct reasoning space for training data.
    Section 4.4 fixes these limits without ablating them; if the budget is too small, the 'high-quality thinking' claim fails.
  • domain assumption Medium/hard LeetCode questions with self-generated code, even without correctness filtering, are a useful training signal that transfers to MBPP and HumanEval.
    Sections 4.1 and 4.5 assume transfer; the results only partially support it, with HumanEval declines.
  • domain assumption The 10-gram decontamination at threshold 0.3 removes benchmark overlap sufficiently.
    Section 4.1; if overlap remains, reported improvements could be inflated by train/test leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SRA-MCTS: Self-driven Reasoning Augmentation with Monte Carlo Tree Search for Code Generation." pith.science (2026). https://pith.science/paper/COOPEH2A

@misc{pith2026241111053,
  author       = {Pith},
  title        = {Pith review of: SRA-MCTS: Self-driven Reasoning Augmentation with Monte Carlo Tree Search for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/COOPEH2A}},
  note         = {Machine review of arXiv:2411.11053}
}
read the original abstract

Large language models demonstrate exceptional performance in simple code generation tasks but still face challenges in tackling complex problems. These challenges may stem from insufficient reasoning and problem decomposition capabilities. To address this issue, we propose a reasoning-augmented data generation process, SRA-MCTS, which guides the model to autonomously generate high-quality intermediate reasoning paths. This creates a positive feedback loop, enabling continuous improvement. Our method operates entirely through the model itself without requiring additional supervision. By synthesizing natural language reasoning paths and translating them into executable code, the approach ensures analytical accuracy and enhances the success rate in solving complex tasks. Experimental results show that, even without additional supervisory signals, our method achieves performance improvements across different model scales, demonstrating the significant potential of self-improvement in small models. Furthermore, the method remains robust when traditional Chain-of-Thought (CoT) approaches exhibit performance degradation, with notable improvements observed in diversity metrics such as pass@10. We encourage further exploration of reasoning processes within training data to enhance the ability of language models to address complex problems. Our code and data are public at https://github.com/DIRECT-BIT/SRA-MCTS.

Figures

Figures reproduced from arXiv: 2411.11053 by the authors.

Figure 1
Figure 1. The overall workflow of our method, with data generation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Self-driven reasoning augmentation process of SRA-MCTS. (a) Selection: A leaf node is selected to be expanded in the next phase. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The progressive scoring method. The state and action of the node are used as inputs, and the judgment is made sequentially from left to right based on the four principles. If the current principle is satisfied, an integer score in the corresponding interval is output; otherwise, the next principle is evaluated. If all the principles are not met, the model will give the current input a full score of 10. Backpropagati… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison results of including and excluding thinking [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: The relationship between the quality of thinking and the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information

    cs.CL 2025-02 conditional novelty 6.0 of 10

    LongDPO uses MCTS-collected, critique-refined stepwise preference pairs with step-level DPO to improve long-form generation quality and length adherence.

  2. A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of reasoning LLMs that use reinforcement learning and inference-time scaling to achieve slow, deliberate thought, organized into three method categories.

Reference graph

Works this paper leans on

32 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    Finite-time analysis of the multiarmed bandit problem

    Peter Auer, Nicol \` o Cesa - Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn. , 47(2-3):235--256, 2002

    work page 2002

  2. [2]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR , abs/2108.07732, 2021

    arXiv 2021

  3. [3]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    2023

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

    arXiv 2021

  5. [5]

    Guibas, and Fei Xia

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 14455--14465. IEEE , 2024

    work page 2024

  6. [6]

    Efficient selectivity and backup operators in monte-carlo tree search

    R \' e mi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In H. Jaap van den Herik, Paolo Ciancarini, and H. H. L. M. Donkers, editors, Computers and Games, 5th International Conference, CG 2006, Turin, Italy, May 29-31, 2006. Revised Papers , volume 4630 of Lecture Notes in Computer Science , pages 72--83. Springer, 2006

    work page 2006

  7. [7]

    Stepcoder: Improving code generation with reinforcement learning from compiler feedback

    Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Stepcoder: Improving code generation with reinforcement learning from compiler feedback. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings ...

    work page 2024

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

Show all 32 references
  1. [9]

    Leetcode dataset, 2023

    Greengerong. Leetcode dataset, 2023

    2023

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 ....

    2022

  3. [11]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  4. [12]

    Spoc: Search-based pseudocode to code

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch \' e - Buc, Emily B. Fox, and Roman Garnett, editors, Advances in ...

    2019

  5. [13]

    Think outside the code: Brainstorming boosts large language models in code generation

    Xin - Ye Li, Jiang - Tian Xue, Zheng Xie, and Ming Li. Think outside the code: Brainstorming boosts large language models in code generation. CoRR , abs/2305.10679, 2023

    2023 arXiv

  6. [14]

    Rewriting the code: A simple method for large language model augmented code search

    Haochen Li, Xin Zhou, and Zhiqi Shen. Rewriting the code: A simple method for large language model augmented code search. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...

    2024

  7. [15]

    Rethinkmcts: Refining erroneous thoughts in monte carlo tree search for code generation

    Qingyao Li, Wei Xia, Kounianhua Du, Xinyi Dai, Ruiming Tang, Yasheng Wang, Yong Yu, and Weinan Zhang. Rethinkmcts: Refining erroneous thoughts in monte carlo tree search for code generation. CoRR , abs/2409.09584, 2024

    2024

  8. [16]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edito...

    2023

  9. [17]

    On llms-driven synthetic data generation, curation, and evaluation: A survey

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguist...

    2024

  10. [18]

    Wizardcoder: Empowering code large language models with evol-instruct

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vien...

    2024

  11. [19]

    Learning to reason with large language models

    OpenAI. Learning to reason with large language models. https://openai.com/index/learning-to-reason-with-llms/. Accessed: 2024-10-31

    2024

  12. [20]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  13. [21]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 , 2024

    2024 arXiv

  14. [22]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  15. [23]

    Planning in natural language improves LLM search for code generation

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves LLM search for code generation. CoRR , abs/2409.03733, 2024

    2024 arXiv

  16. [24]

    Dolphcoder: Echo-locating code large language models with diverse and multi-objective instruction tuning

    Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Weiran Xu, Jingang Wang, Mengdi Zhang, and Xunliang Cai. Dolphcoder: Echo-locating code large language models with diverse and multi-objective instruction tuning. In Lun - Wei Ku, Andre Martins, and Vivek ...

    2024

  17. [25]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, ...

    2022

  18. [26]

    Magicoder: Source code is all you need

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. CoRR , abs/2312.02120, 2023

    2023 arXiv

  19. [27]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Adva...

    2023

  20. [28]

    Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings...

    2024

  21. [29]

    Rest-mcts*: LLM self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: LLM self-training via process reward guided tree search. CoRR , abs/2406.03816, 2024

    2024 arXiv

  22. [30]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation. CoRR , abs/2410.10762, 2024

    arXiv

  23. [31]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.