Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CodeCoR claims that pruning each agent's outputs stops error propagation and raises average Pass@1 to 77.8 percent across four benchmarks.

desk verdict A sensible pruning-based multi-agent code generation framework, but the Pass@1 claim is built on a metric that doesn't match the procedure; the evaluation needs a major rework before the numbers can be trusted. read the letter →

arxiv 2501.07811 v2 pith:Y2F4APVR submitted 2025-01-14 cs.SE

Ruwei Pan , Hongyu Zhang , Chao Liu This is my paper · ORCID
classification cs.SE
keywords codegenerationmulti-agentLLMframeworkself-reflectionchain-of-thoughtpromptingtest-caseprogramrepairPass@1pruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CodeCoR, a four-agent LLM framework for Python code generation in which a prompt agent, coding agent, test agent, and repair agent each produce multiple candidate outputs and prune low-quality ones before passing results downstream. The aim is to stop errors from propagating through a sequential generation pipeline, which the authors argue is the main weakness of earlier multi-agent frameworks such as CodeCoT and MapCoder. The paper reports that CodeCoR achieves an average Pass@1 of 77.8% on HumanEval, HumanEval-ET, MBPP, and MBPP-ET with GPT-3.5-turbo, outperforming MapCoder's 72.8%. If the result holds, it suggests that self-evaluation and pruning inside a pipeline can be more valuable than simply chaining agents.

What carries the argument

The central mechanism is a prune-and-retry cycle. At each of the four phases, agents generate multiple candidates, score them (often by self-assessed binary criteria), discard poor ones, execute code locally against generated tests, and route failures to a repair agent; the returned code is the one passing the most tests. The load-bearing assumption is that the LLM's own ratings separate high-quality from low-quality intermediate outputs. The repair loop stops when repeated errors resemble previous rounds or after a set number of rounds, which the paper finds optimal at three.

What would settle it

Run CodeCoR and its variant with the pruning methods removed on all four datasets under identical random seeds and API settings: if disabling pruning does not lower Pass@1 on HumanEval, HumanEval-ET, MBPP, and MBPP-ET, then the reported advantage over MapCoder is not attributable to the pruning mechanism.

Watch

Extended reading notes

Core claim

The central claim is that interleaving multiple candidate generations with self-pruning and repair yields more correct code than a linear chain of specialized agents. Each stage is made self-reflective: prompts are scored on clarity, relevance, conciseness, and context; tests are pruned for emptiness or invalidity; code is pruned when it fails to compile; repair advice is pruned on the same four criteria. Code that fails the generated tests goes back to the coding agent with repair advice, and the snippet passing the most tests is returned. The paper reports Pass@1 scores of 86.6, 80.5, 79.2, and 65.2 on the four datasets with GPT-3.5-turbo and states that this significantly outperforms existing baselines including MapCoder.

Load-bearing premise

The framework assumes the LLM's own one-or-zero ratings for clarity, relevance, conciseness, and context correctly identify which prompts, tests, code snippets, and repair suggestions are worth keeping, and the paper supplies no independent check that these self-scores track actual downstream quality.

Editorial extensions

If this is right

  • Sequential multi-agent code generation can be improved by pruning at every stage rather than relying on each agent's single best output.
  • Self-assigned quality scores by an LLM can serve as a cheap filter for prompts and repair advice, without an external validator.
  • Iterative local execution plus repair can lift Pass@1 beyond what single-pass or linearly chained generation achieves on standard benchmarks.
  • The reported benefit transfers to other LLMs such as GPT-4 and CodeLlama and persists on the harder ET variants, suggesting robustness across model strength and test difficulty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same self-evaluation-and-prune pattern could be lifted to other multi-draft generation tasks, such as test-suite generation or natural-language-to-SQL, because the framework is not specific to the four roles it names.
  • Editorial inference: because the final selection is based only on generated tests, code that passes those tests may still fail hidden tests, so an independent replication with held-out test suites would clarify how much of the reported gap is real correctness improvement.
  • Editorial inference: the fixed generation budget per agent suggests a testable extension where the number of candidate drafts is varied per stage and Pass@1 is measured per API call, which would separate the value of pruning from the value of simply sampling more outputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CodeCoR, a multi-agent LLM framework for Python code generation. Four agents (prompt, test, coding, and repair) each generate multiple outputs and prune low-quality ones; generated code is executed against generated test cases, failed snippets are sent to the repair agent, and the snippet passing the most generated tests is returned. Experiments on HumanEval, HumanEval-ET, MBPP, and MBPP-ET with GPT-3.5-turbo report an average Pass@1 of 77.8%, together with ablations and a cost comparison.

Significance. If the evaluation were protocol-equivalent, CodeCoR would be a useful contribution: the idea of pruning intermediate outputs to reduce error propagation in sequential multi-agent pipelines is plausible and distinct from plain sequential frameworks, and the ablation design and the availability of source code are positive features. However, the headline result is currently unverifiable because the Pass@1 metric is defined and used inconsistently with the actual pipeline, and the benefit of the pruning mechanism is not statistically established. The central architectural idea is worth pursuing, but the evidence in this manuscript does not yet support the strong comparative claims.

major comments (4)
  1. [§4.4, §3.1, Algorithm 1, Table 2] The reported Pass@1 is not the metric defined in §4.4. Section 4.4 defines Pass@1 as the proportion of generated snippets that are correct 'on the first attempt without any modifications', but Algorithm 1 and Section 3.1 describe generating multiple CoT prompts, multiple code snippets, multiple test cases, iterative repair, and final selection of the code passing the most generated tests. This is best-of-N selection with repair. Table 2 compares CodeCoR against numbers taken from prior papers (§4.3) without reporting the candidate budget, temperature, number of repair rounds, or LLM calls per problem for either side. Without a protocol-equivalent measurement, the 77.8% average and the claimed gains over MapCoder and CodeCoT are not verifiable, and the gain could reflect extra compute rather than the architecture. Please report the exact sampling and selection protocol, and either re-run baselines under the same candidate budget or clearly restrict the claim to best-of-N-with-repair performance.
  2. [§3.4, Algorithm 1, lines 20–32] Algorithm 1 as printed clears failed_code_snippets and returns C_f inside the while loop body, so the loop can perform at most one repair iteration. This contradicts the text's description of iterative repair and the repair-round experiment in §5.3 and Figure 7, where performance is shown for up to five repair rounds. The pseudocode needs a correct loop structure and an explicit repair-round limit or other termination condition; as written, the algorithm cannot implement the claimed multi-round repair process.
  3. [§3.3, Figure 5, Table 4] The pruning methods are the main departure from a sequential pipeline, but they rely on the same LLM assigning binary self-scores for clarity, relevance, conciseness, and context with no validation that these scores correlate with downstream quality. The ablation 'w/o Pruning Method' is the relevant test, yet the differences are mixed in magnitude (1.3 points on HumanEval-ET versus 11.5 points on MBPP) and no significance or variance is reported. Moreover, the description of this variant is self-contradictory: it says the variant 'does not follow the traditional sequential multi-agent framework' and then says it 'cannot prune the outputs' of agents. Please clarify the variant and provide evidence that self-assigned scores are reliable, for example by comparing pruned versus unpruned outputs on a held-out quality measure.
  4. [§5.4, Table 2] The paper claims 'significantly' better performance, but no statistical support is provided. The threats-to-validity section states that 10 rounds were averaged for each trial, but no standard deviations, confidence intervals, or significance tests appear anywhere in the results. Given the small per-dataset differences (for example, 79.2 versus 78.9 on MBPP for MapCoder), the word 'significantly' is not supported. Please report variances or an appropriate significance test, or soften the claim.
minor comments (5)
  1. [Abstract, §1, §6, Table 2] The average Pass@1 is reported as 77.13% in the abstract but as 77.8% in §1 and §6; the values in Table 2 average to 77.9%. Please harmonize these numbers.
  2. [§4.5, Table 3] The text says CodeCoR's mean edit distance is 'the highest among all models,' but edit distance is a lower-is-better metric; presumably 'lowest' is intended.
  3. [§5.1, Table 6] The text reports MBPP and MBPP-ET scores for CodeLlama, but Table 6 contains only HumanEval and HumanEval-ET; either add the missing columns or remove the unsupported sentence.
  4. [§2.1, Reference [22]] Reference [22] cites a paper on tensor-based channel estimation in IRS-assisted MIMO systems in the context of self-revision for code generation; this appears to be a citation error and should be replaced.
  5. [Table 2] Table 2 has a duplicated 'GPT-3.5-turbo' header row and inconsistent column alignment; please reformat the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported Pass@1 is grounded in independent official hidden tests, and the identified concerns are evaluation-validity issues rather than circular derivation.

full rationale

The paper's central claim is an empirical measurement, not a derived quantity. CodeCoR generates candidate code, prunes outputs using LLM self-assigned criteria, selects code that passes the most LLM-generated test cases, and then reports Pass@1 on the official hidden tests of HumanEval, HumanEval-ET, MBPP, and MBPP-ET. Those official tests are external to the generation and selection loop, so passing the LLM-generated tests does not by construction imply passing the hidden tests. No fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem appears; the cited frameworks CodeCoT and MapCoder are prior work by other authors used as baselines. The pruning heuristics (clarity, relevance, conciseness, context) are empirical assumptions whose validity is testable, not definitions of the reported outcome. The concerns raised about the paper are evaluation-validity issues rather than circularity: Section 4.4 defines Pass@1 as "the proportion of generated code snippets that correctly perform the intended task on the first attempt without any modifications," while Algorithm 1 performs best-of-many selection with iterative repair, and Section 5.3 selects the repair-round threshold using Pass@1 on HumanEval and HumanEval-ET before reporting scores on those datasets. These issues affect comparability and generalizability, but they do not make the reported official-test score equivalent to the framework's internal inputs. Therefore no significant circularity is found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation is present; the central claim is empirical. The framework depends on untested self-evaluation assumptions and on fitting the repair-round limit to the benchmark datasets. No speculative physical or formal entities are introduced.

free parameters (2)
  • repair_rounds_limit = 3
    Selected in Section 5.3 by testing iteration thresholds 1 through 5 on HumanEval and HumanEval-ET (Figure 7); this maximizes reported Pass@1, so it is fitted to the evaluation datasets.
  • pruning_criteria_threshold = [1,1,1,1]
    Hand-chosen requirement in Section 3.3 that a prompt or repair advice satisfy all four criteria (clarity, relevance, conciseness, context); no evidence is given that this binary threshold selects higher-quality outputs.
assumptions (4)
  • domain assumption LLM-generated test cases are a reliable proxy for the hidden benchmark tests when ranking and repairing code.
    Algorithm 1 (Section 3.4) selects and repairs code based on passes against agent-generated tests; if these tests diverge from the hidden tests, the ranking is suboptimal.
  • ad hoc to paper The same LLM's self-evaluation scores correlate with the actual quality of its prompts, code, tests, and repair advice.
    Section 3.3's pruning methods rely on the model scoring its own outputs with [1,1,1,1]; the paper provides no external validation of this self-assessment.
  • domain assumption Baseline numbers cited from previous studies are comparable to CodeCoR's evaluation setting.
    Section 4.3 states baselines are taken from prior work 'under the same experimental setting', but CodeCoR's protocol (multiple candidates, generated tests, repair) differs from a standard single-sample Pass@1 evaluation.
  • domain assumption The local execution environment matches the benchmark evaluation environment.
    Pass@1 is computed by executing code locally (Section 4.4); differences in Python version, dependencies, or time limits would change the reported scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation." pith.science (2026). https://pith.science/paper/Y2F4APVR

@misc{pith2026250107811,
  author       = {Pith},
  title        = {Pith review of: CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2F4APVR}},
  note         = {Machine review of arXiv:2501.07811}
}
read the original abstract

Code generation aims to produce code that fulfills requirements written in natural language automatically. Large Language Models (LLMs) like ChatGPT have demonstrated promising effectiveness in this area. Nonetheless, the LLMs often fail to ensure the syntactic and semantic correctness of the generated code. Recently, researchers proposed multi-agent frameworks that guide LLMs with different prompts to analyze programming tasks, generate code, and perform testing in a sequential workflow. However, the performance of the workflow is not robust as code generation depends on the performance of each agent. To address this challenge, we propose CodeCoR, a multi-agent framework that prunes intermediate outputs at different stages to reduce error propagation in sequential code generation workflows. Specifically, for a given task description, four agents in CodeCoR generate prompts, code, test cases, and repair advice, respectively. Each agent generates more than one output and prunes away the low-quality ones. The generated code is tested in the local environment: the code that fails to pass the generated test cases is sent to the repair agent, and the coding agent re-generates the code based on repair advice. Finally, the code that passes the highest number of generated test cases is returned to the users. Our experiments on four widely used datasets, HumanEval, HumanEval-ET, MBPP, and MBPP-ET, demonstrate that CodeCoR outperforms existing baselines (e.g., CodeCoT and MapCoder), achieving an average Pass@1 score of 77.13%.

Figures

Figures reproduced from arXiv: 2501.07811 by the authors.

Figure 1
Figure 1. Examples of misunderstanding of sequential workflow. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the CodeCoR framework , Vol. 1, No. 1, Article . Publication date: January 2025 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of prompts and outputs for four agents. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The hierarchical structure of the pruning process [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The pruning prompts of agents and the workflow for pruning the low-quality outputs [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The diagram illustrates the improvements in code generation accuracy with the integration of a Repair [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Pass@1 results under different repair rounds on HumanEval datasets [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?

    cs.SE 2026-07 conditional novelty 6.0 of 10

    On 116 LiveCodeBench tasks, Claude reviewing Codex raised pass rate from 71.6% to 89.7%, while Codex reviewing Claude lowered it from 91.4% to 82.8%.

  2. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

47 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Program synthesis: challenges and opportunities,

    C. David and D. Kroening, “Program synthesis: challenges and opportunities, ”Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , vol. 375, no. 2104, p. 20150403, 2017

    work page 2017

  2. [2]

    Program synthesis,

    S. Gulwani, O. Polozov, R. Singhet al., “Program synthesis, ”Foundations and Trends® in Programming Languages, vol. 4, no. 1-2, pp. 1–119, 2017

    work page 2017

  3. [4]

    From Symplectic to Poisson. A Study of Reduction and a Proposal Towards Implosion

    D. Fried, J. Andreas, and D. Klein, “Incorporating discrete structures into neural models, ”arXiv preprint arXiv:2202.05783, 2022

    work page Pith review arXiv 2022

  4. [5]

    Interpretable Deep Learning Methods for Multiview Learning

    J. Zheng, P. Xu, J. Liu, X. Huang, and X. Qiu, “Codet5+: Open code generation model pretrained on text-to-code and code-to-code tasks, ”arXiv preprint arXiv:2302.07930, 2023

    work page Pith review arXiv 2023

  5. [6]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models, ” inAdvances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824–24 837. , Vol...

    work page 2022

  6. [8]

    Self-planning code generation with large language models,

    X. Jiang, Y. Dong, L. Wang, F. Zheng, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models, ”ACM Transactions on Software Engineering and Methodology , 2023

    work page 2023

  7. [9]

    Codecot and beyond: Learning to program and test like a developer,

    D. Huang, Q. Bu, and H. Cui, “Codecot and beyond: Learning to program and test like a developer, ”arXiv preprint arXiv:2308.08784, 2023

    arXiv 2023

  8. [10]

    Mapcoder: Multi-agent code generation for competitive problem solving,

    M. A. Islam, M. E. Ali, and M. R. Parvez, “Mapcoder: Multi-agent code generation for competitive problem solving, ”

    work page

Show all 47 references
  1. [12]

    Codescore: Evaluating code generation by learning code execution,

    Y. Dong, J. Ding, X. Jiang, G. Li, Z. Li, and Z. Jin, “Codescore: Evaluating code generation by learning code execution, ”

  2. [13]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners, ”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  3. [14]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners, ”Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

    1901

  4. [15]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code, ”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  5. [16]

    Competition-level code generation with alphacode,

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lagoet al., “Competition-level code generation with alphacode, ”Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  6. [17]

    Codet: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests, ” arXiv preprint arXiv:2207.10397, 2022

    2022 arXiv

  7. [18]

    Self-edit: Fault-aware code editor for code generation,

    K. Zhang, Z. Li, J. Li, G. Li, and Z. Jin, “Self-edit: Fault-aware code editor for code generation, ” arXiv preprint arXiv:2305.04087, 2023

    2023 arXiv

  8. [19]

    Generating sequences by learning to self-correct,

    S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi, “Generating sequences by learning to self-correct, ”arXiv preprint arXiv:2211.00053, 2022

    2022 arXiv

  9. [20]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning,

    H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, “Coderl: Mastering code generation through pretrained models and deep reinforcement learning, ”Advances in Neural Information Processing Systems , vol. 35, pp. 21 314–21 328, 2022

    2022

  10. [21]

    Is self-repair a silver bullet for code generation?

    T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, “Is self-repair a silver bullet for code generation?” in The Twelfth International Conference on Learning Representations , 2023

    2023

  11. [22]

    Tensor-based channel estimation and data-aided tracking in irs-assisted mimo systems,

    K. B. A. Benicio, A. L. F. de Almeida, B. Sokal, Fazal-E-Asim, B. Makki, and G. Fodor, “Tensor-based channel estimation and data-aided tracking in irs-assisted mimo systems, ” 2023. [Online]. Available: https://arxiv.org/abs/2305.10499

    2023 arXiv

  12. [23]

    Leveraging automated unit tests for unsupervised code translation,

    B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample, “Leveraging automated unit tests for unsupervised code translation, ”arXiv preprint arXiv:2110.06773, 2021

    2021 arXiv

  13. [24]

    Scriptoriumws: A code generation assistant for weak supervision,

    T.-H. Huang, C. Cao, S. Schoenberg, H. Vishwakarma, N. Roberts, and F. Sala, “Scriptoriumws: A code generation assistant for weak supervision, ” inICLR Deep Learning for Code Workshop , 2023

    2023

  14. [25]

    Multi-agent systems and simulation: a survey from an application perspective,

    K. G. Troitzsch, “Multi-agent systems and simulation: a survey from an application perspective, ”Multi-agent systems: Simulation and applications, pp. 53–75, 2009

    2009

  15. [26]

    Self-collaboration code generation via chatgpt,

    Y. Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via chatgpt, ”arXiv preprint arXiv:2304.07590, 2023

    2023 arXiv

  16. [27]

    Metagpt: Meta programming for multi-agent collaborative framework,

    S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou et al., “Metagpt: Meta programming for multi-agent collaborative framework, ”arXiv preprint arXiv:2308.00352, 2023

    2023 arXiv

  17. [28]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun, “Chatdev: Communicative agents for software development, ” 2024. [Online]. Available: https://arxiv.org/abs/2307.07924

    2024 arXiv

  18. [29]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models, ” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    2021 arXiv

  19. [30]

    Incoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis, ”arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  20. [31]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x,

    Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Liet al., “Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, ”arXiv preprint arXiv:2303.17568, 2023

    2023 arXiv

  21. [32]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161, 2023. , Vol. 1, No. 1, Article . Publication date: January 2025. 20 Ruwei Pan, Hongyu Zhang*, Chao Liu

    2023 arXiv

  22. [33]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis, ”arXiv preprint arXiv:2203.13474, 2022

    2022 arXiv

  23. [34]

    Gpt-4 technical report,

    e. a. OpenAI, “Gpt-4 technical report, ” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774

    2024 arXiv

  24. [35]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models, ” 2023. [Online]. Available: https://arxiv.org/abs/2210.03629

    2023 arXiv

  25. [36]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning, ”Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  26. [37]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models, ” 2023. [Online]. Available: https://arxiv.org/abs/2305.10601

    2023 arXiv

  27. [38]

    Reasoning with language model is planning with world model,

    S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu, “Reasoning with language model is planning with world model, ” 2023. [Online]. Available: https://arxiv.org/abs/2305.14992

    2023 arXiv

  28. [39]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Schärli, and D. Zhou, “Teaching large language models to self-debug, ”arXiv preprint arXiv:2304.05128, 2023

    2023 arXiv

  29. [40]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y. Li, and Z. Jin, “Structured chain-of-thought prompting for code generation, ” 2023. [Online]. Available: https://arxiv.org/abs/2305.06599

    2023 arXiv

  30. [41]

    Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules,

    H. Le, H. Chen, A. Saha, A. Gokul, D. Sahoo, and S. Joty, “Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules, ”arXiv preprint arXiv:2310.08992, 2023

    2023 arXiv

  31. [42]

    Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing,

    H. Wang, Z. Liu, S. Wang, G. Cui, N. Ding, Z. Liu, and G. Yu, “Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing, ”arXiv preprint arXiv:2311.09868, 2023

    2023 arXiv

  32. [43]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., “Palm: Scaling language modeling with pathways, ”Journal of Machine Learning Research , vol. 24, no. 240, pp. 1–113, 2023

    2023

  33. [44]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct,

    H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang, “Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, ”arXiv preprint arXiv:2308.09583, 2023

    2023 arXiv

  34. [45]

    Binary codes capable of correcting deletions, insertions, and reversals,

    V. I. Levenshteinet al., “Binary codes capable of correcting deletions, insertions, and reversals, ” inSoviet physics doklady, vol. 10, no. 8. Soviet Union, 1966, pp. 707–710

    1966

  35. [46]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation, ” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  36. [47]

    Psutil documentation,

    G. Rodola, “Psutil documentation, ” 2020

    2020

  37. [48]

    Code llama: Open foundation models for code,

    B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. ...

    2024 arXiv

  38. [2023]

    Available: https://arxiv.org/abs/2301.09043

    [Online]. Available: https://arxiv.org/abs/2301.09043

    arXiv

  39. [2024]

    Available: https://arxiv.org/abs/2405.11403

    [Online]. Available: https://arxiv.org/abs/2405.11403

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.