MemoCoder: Automated Function Synthesis using LLM-Supported Agents

Shayan Noei; Ying Zou; Yiping Jia; Zhen Ming Jiang

REVIEW 4 major objections 5 minor 38 references

MemoCoder: Automated Function Synthesis using LLM-Supported Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MemoCoder claims a multi-agent repair loop with a persistent Fixing Knowledge Set beats zero-shot and self-repair on three code benchmarks.

desk verdict MemoCoder's memory-augmented repair architecture is plausible and the ablations are informative, but the online-adaptation evaluation protocol lets fixes from earlier test problems feed later ones, so the headline gains do not cleanly measure generalization. read the letter →

arxiv 2507.18812 v1 pith:MA2CMX3N submitted 2025-07-24 cs.SE cs.AI

Yiping Jia , Zhen Ming Jiang , Shayan Noei , Ying Zou This is my paper · ORCID

classification cs.SEcs.AI

keywords LLMcodegenerationmulti-agentsystemsself-repairretrieval-augmentedfixingknowledgefunctionsynthesisPass@kevaluationprogramrepair

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MemoCoder is a multi-agent framework for turning natural-language problem descriptions into working functions. The paper claims that its four specialized LLM agents, coordinated by a central Mentor Agent and backed by a Fixing Knowledge Set that stores successful repairs, consistently solve more test cases than zero-shot prompting or a Self-Repair baseline across MBPP, HumanEval, and LiveCodeBench. Reported gains range from 3.1 to 12.1 percentage points at Pass@10 and from 1.4 to 14.5 percentage points at Pass@50. A sympathetic reader would take the central insight to be that repair knowledge, once extracted from past fixes, can be reused across problems without retraining the model.

What carries the argument

The load-bearing object is the Fixing Knowledge Set, a persistent store of initial code, fixed code, and error messages, together with the Mentor Agent, which retrieves up to ten similar past fixes by a longest-sequential-match on error text and rewrites a small set of fixing suggestions per error type. A Planner agent supplies the starting point by generating three plans before code is written. The machinery's job is to convert single repair attempts into reusable strategies, so later problems start from accumulated experience rather than rediscovering the same fix.

What would settle it

Run MemoCoder with the Fixing Knowledge Set seeded only from APPS and then frozen during evaluation; if the Pass@50 advantage over Self-Repair mostly disappears, the gains come from in-benchmark adaptation rather than from reusable repair knowledge.

Watch

Extended reading notes

Core claim

The paper's central claim is that a code-generation system improves when repair is organized as a collaborative loop with memory. In MemoCoder, a Planner proposes multiple solution strategies, a Code Writer implements and revises code, a Test Executor classifies failures into four error types, and a Mentor Agent retrieves similar past fixes and distills recurring error patterns into high-level fixing suggestions. Successful repairs are stored in the Fixing Knowledge Set and become retrievable for later problems. The paper reports that this design outperforms both zero-shot prompting and a Self-Repair baseline on three benchmarks, and that its ablations point to retrieval and error-pattern analysis as the components driving the higher Pass@10 and Pass@50 scores.

Load-bearing premise

The evaluation lets fixes from problems already solved on the same benchmark feed the repair of later problems, so the reported gains measure adaptation within a benchmark rather than transfer to never-seen tasks.

Editorial extensions

If this is right

If the reported gains hold, a frozen LLM can keep improving at coding tasks by accumulating repair experience, without fine-tuning or retraining.
Self-repair loops that lack memory should be expected to lag on benchmarks where similar failure modes recur across problems.
Pass@1 gains are small, so the framework's value lies in iterative settings where multiple attempts are affordable rather than in first-attempt code generation.
The error-type analysis suggests that compile errors and wrong-output errors persist across iterations, making those categories the natural target for further improvement.
Retrieval and error-pattern distillation are the components that matter most for high-attempt pass rates, so omitting them would erode the reported advantage at Pass@10 and Pass@50.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If MemoCoder's Fixing Knowledge Set is frozen after the APPS seeding phase, the reported Pass@50 advantage over Self-Repair may shrink; that experiment would separate within-benchmark adaptation from transfer of repair skill.
The online-adaptation protocol means the numbers should be read as describing a system that solves a stream of problems while learning from that same stream, not as describing a system that solves problems unrelated to anything it has seen.
A natural extension would test whether distilled fixing suggestions transfer across programming languages or to repository-level bugs, where error messages and test harnesses are more heterogeneous.
A cheaper baseline worth checking is Self-Repair with access to the same retrieved examples but without the Mentor Agent's distilled strategy summaries, which would isolate the value of abstraction over raw retrieval.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's architecture is a real step beyond the usual self-repair loop: a persistent Fixing Knowledge Set plus a Mentor Agent that distills error patterns into reusable strategies. The ablation study suggests both the memory and the mentor matter, and the use of a contamination-controlled LiveCodeBench subset is good practice. The McNemar tests are appropriate.

The soft spot is the evaluation protocol. Sections 2.2.4 and 3.3 are explicit that during the evaluation phase the Fixing Knowledge Set grows by incorporating successful repairs from previously solved test problems, available for retrieval in later test cases. That means later problems in MBPP, HumanEval, or LiveCodeBench can be solved using code and fixes drawn from earlier problems in the same benchmark. The Self-Repair baseline has no such memory. So the reported gains—3.1–12.1 points in Pass@10, 1.4–14.5 in Pass@50—conflate benchmark-specific adaptation with genuine cross-task improvement. The paper's claim that this protocol prevents data leakage is only true in the narrow sense that no task benefits from its own fixes; it does not prevent leakage between tasks in the same evaluation set. That is load-bearing for the central claim.

Smaller problems: retrieval size was manually tuned, likely on the evaluation benchmarks, and no held-out validation is reported for this hyperparameter. The Pass@k definition is ambiguous because it counts iterative repair attempts, not independent samples, and it is not clear whether baselines get the same number of attempts. Finally, the abstract and conclusion numbers do not match Table 1: the abstract says up to 12.1 points Pass@10, but Table 1 shows a 14.05-point gain on LCB with LLaMA, and the conclusion claims 12/15 points Pass@10 and 16/19 points Pass@50, which do not correspond to the table. These are fixable but suggest sloppy reporting.

The core idea is credible, and the method probably does work to some degree. The paper deserves a serious referee, but not as is. A clean separation of online adaptation from transfer—or an explicit acknowledgment that the evaluation measures within-benchmark adaptation—would make the contribution much stronger.

Referee Report

4 major / 5 minor

Summary. The paper proposes MemoCoder, a multi-agent framework for LLM-based code generation and repair that combines a Planner, a Code Writer, a Test Executor, and a Mentor agent with a persistent Fixing Knowledge Set storing successful repairs. The Mentor summarizes recurring error patterns and produces fixing suggestions, and retrieval of similar past fixes is used to guide the repair loop. The system is bootstrapped by running on the APPS dataset and is then evaluated on LiveCodeBench, MBPP, and HumanEval with two base models (LLaMA 3.1-8B-Instruct and Qwen 2.5-32B), comparing against zero-shot prompting and a Self-Repair baseline. The paper reports consistent Pass@10 and Pass@50 improvements, an ablation of the planning, retrieval, and error-pattern components, and an analysis of error-type transitions across repair iterations.

Significance. If the reported gains hold under a leakage-free evaluation, the work would be a useful contribution to LLM-based code repair: it demonstrates that a persistent, retrievable memory of successful fixes, combined with a supervisory agent that distills reusable repair strategies, can improve iterative refinement. Strengths include evaluation on three benchmarks with two base models, ablations of the main components, McNemar statistical testing, and a promised replication package containing the agent prompts. However, the current evaluation protocol does not support the generalization claim as stated, because fixes from earlier tasks in the same benchmark are allowed to enter the knowledge base during evaluation, and a key hyperparameter is manually tuned without a described development split. The central idea is plausible, but the evidence is not yet conclusive.

major comments (4)

[Section 3.3 and Section 2.2.4] The evaluation phase is run under an online-adaptation protocol in which the Fixing Knowledge Set "continues to grow by incorporating successful repairs, which are made available for retrieval in future test cases" (Section 3.3), and in which "only the remaining unsolved problems are passed through the system, which incorporates newly distilled fixing strategies from previously solved cases" (Section 2.2.4). This means that for any later task in LCB, MBPP, or HumanEval, the retrieved fixing examples can include code and repairs from earlier tasks in the same benchmark. Since the Self-Repair baseline has no such memory, the reported Pass@10 and Pass@50 improvements conflate genuine transferable repair knowledge with benchmark-specific adaptation, so the results do not measure generalization to unseen tasks. Please re-run the evaluation with the knowledge set fixed after the APPS accumulation phase, or with a held-out split for online adaptation, and report both settings.
[Section 4.1.2] The retrieval size is chosen by manual experimentation: "To determine the optimal number of examples to retrieve, we manually experimented with different retrieval sizes... we set the mentor agent to retrieve up to ten examples." The paper does not state that this tuning was performed on a development split disjoint from LCB, MBPP, and HumanEval. If the evaluation benchmarks were used to select this hyperparameter, the reported gains are inflated by test-set tuning. Please specify the tuning protocol and, if necessary, re-run the comparison with the retrieval size fixed before seeing the evaluation sets.
[Section 3.3] The Pass@k definition is ambiguous. The example in Section 3.3 treats sequential repair attempts as the k samples ("If the model finally generates a correct solution on the fourth try... then Pass@4 becomes 1"), which is not the standard Pass@k estimator over independent temperature samples. It must be clarified whether Pass@k is computed over the repair trajectory, over multiple independent runs, or over a combination, and the same protocol must be applied identically to MemoCoder and the baselines. This matters because the reported Pass@10 and Pass@50 values are the central evidence for the paper's claims.
[Section 5.2] The internal validity section states that "our ablation study (Section X) shows that performance plateaus beyond 10 retrieved examples and most successful fixes occur within 30 attempts," but no such ablation appears in the paper and the cross-reference "Section X" is unresolved. This statement is used to argue that the repair limit and retrieval size are reasonable and not overfit; without the supporting study, that argument is unsupported. Please either add the ablation or remove the claim.

minor comments (5)

[Abstract and Table 1] The abstract reports Pass@10 improvements of 3.1% to 12.1%, but the largest Pass@10 improvement over either baseline in Table 1 is about 14.05 percentage points (LCB, LLaMA 3.1-8B vs Self-Repair); please reconcile the numbers.
[Section 2.2.2] The Code Writer description refers to the Mentor agent as "explained in Section 4," but the Mentor is actually described in Section 2.2.4; the cross-reference should be corrected.
[Section 6.4] The citation for AgentCoder appears as "Huange et al." and should be "Huang et al."
[Section 5.1] Section 5.1 says "over 5,000 training examples from the APPS dataset" were used to construct the knowledge set, but Section 3.1 does not report how many APPS problems were actually solved and retained; please report the actual number of fixes in the knowledge base.
[References] The reference list contains duplicated entries for the same paper, e.g., Austin et al. 2021 appears as both [3] and [4]; please deduplicate the bibliography.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation chain: reported Pass@k gains are empirical, though the online-adaptation protocol is an evaluation-validity concern.

full rationale

MemoCoder's derivation chain is: accumulate repair knowledge from APPS, retrieve similar past fixes by error-message similarity, apply Mentor-refined strategies during repair, and measure Pass@k on MBPP, HumanEval, and a contamination-controlled LiveCodeBench subset. None of the reported Pass@k improvements is forced by construction from the knowledge-set inputs: the LLM can ignore or misapply retrieved fixes, and the ablations show that removing RAG or the Mentor component lowers Pass@10/Pass@50, so the measured effect has independent empirical content. The online-adaptation protocol disclosed in Sections 2.2.4 and 3.3 does allow fixes from earlier evaluation problems to enter the Fixing Knowledge Set and be retrieved for later evaluation problems; that is a genuine threat to the interpretation of the results as generalization to unseen tasks and should be treated as a correctness/validity risk, not as circularity. There is no equation, fitted parameter, or uniqueness theorem that makes a later problem's pass status equivalent to the accumulated fixes. The only self-citation ([25], Noei et al.) appears with Norris's standard Markov-chain textbook for the descriptive transition-matrix analysis in RQ3 and is not load-bearing. Section 5.2 also contains an unfinished cross-reference ('Section X') for the retrieval-size plateau claim, which is a missing-support issue but not a circular step.

Assumptions & free parameters 6 free parameters · 4 assumptions · 2 invented entities

The central claim is empirical, so the main inputs are evaluation choices: the knowledge accumulated from APPS, retrieval and repair hyperparameters, and the online-adaptation protocol. The Fixing Knowledge Set and Mentor Agent are new components whose benefit is demonstrated only within this paper. No formal derivation is involved.

free parameters (6)

retrieval_k = 10
Number of retrieved fix examples; manually tuned on evaluation benchmarks (Section 4.1.2), potentially fitting to test data.
retrieval_threshold = 20
Mentor starts retrieving only after at least 20 new successful fixes for an error type; hand-chosen with no sensitivity analysis.
repair_limit = 50
Maximum repair attempts per problem, defining Pass@50; arbitrary but conventional.
timeout_seconds = 5
Timeout for Test Executor; affects Timeout error category and pass rates.
temperature = 0.7
Default temperature for both LLMs; chosen for output diversity, not tuned.
plans_per_problem = 3
Planner generates three high-level plans; hand-chosen design choice.

assumptions (4)

domain assumption Error-message similarity via Longest Sequential Matching is a reliable retrieval signal for reusing past fixes
Section 4.1.2; no evaluation of retrieval quality is provided, yet all repair performance depends on this heuristic.
domain assumption Knowledge accumulated from APPS transfers to MBPP, HumanEval, and LiveCodeBench
Section 3.1; the knowledge base is built from APPS and applied to three other benchmarks; weak transfer would invalidate the evaluation design.
ad hoc to paper Online adaptation on the evaluation benchmark is an acceptable protocol for measuring generalization
Section 2.2.4; the system learns from previously solved evaluation problems, so later problems benefit from test-set-derived knowledge. This is a load-bearing assumption for interpreting Pass@50 gains.
domain assumption Base LLMs follow the multi-agent prompts for planning, repair, and mentoring as intended
The entire framework assumes instruction-tuned models respond to structured prompts; this is only evidenced by the reported results, not independently verified.

invented entities (2)

Fixing Knowledge Set
purpose: Stores successful repair examples per error type and supports retrieval for future tasks
Software memory component introduced in this paper; its benefit is shown only within the paper's own evaluation and no external validation is provided.
Mentor Agent
purpose: Supervises the repair loop, distills high-level error patterns, and rewrites fixing suggestions
New LLM-agent role; the ablation study shows removing it reduces Pass@10/50, but evidence is internal and no independent benchmark is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MemoCoder: Automated Function Synthesis using LLM-Supported Agents." pith.science (2026). https://pith.science/paper/MA2CMX3N

@misc{pith2026250718812,
  author       = {Pith},
  title        = {Pith review of: MemoCoder: Automated Function Synthesis using LLM-Supported Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MA2CMX3N}},
  note         = {Machine review of arXiv:2507.18812}
}

read the original abstract

With the widespread adoption of Large Language Models (LLMs) such as GitHub Copilot and ChatGPT, developers increasingly rely on AI-assisted tools to support code generation. While LLMs can generate syntactically correct solutions for well-structured programming tasks, they often struggle with challenges that require iterative debugging, error handling, or adaptation to diverse problem structures. Existing approaches such as fine-tuning or self-repair strategies either require costly retraining or lack mechanisms to accumulate and reuse knowledge from previous attempts. To address these limitations, we propose MemoCoder, a multi-agent framework that enables collaborative problem solving and persistent learning from past fixes. At the core of MemoCoder is a Fixing Knowledge Set, which stores successful repairs and supports retrieval for future tasks. A central Mentor Agent supervises the repair process by identifying recurring error patterns and refining high-level fixing strategies, providing a novel supervisory role that guides the self-repair loop. We evaluate MemoCoder across three public benchmarks -- MBPP, HumanEval, and LiveCodeBench -- spanning a range of problem complexities. Experimental results show that MemoCoder consistently outperforms both zero-shot prompting and a Self-Repair strategy, with improvements ranging from 3.1% to 12.1% in Pass@10 and from 1.4% to 14.5% in Pass@50, demonstrating its effectiveness in iterative refinement and knowledge-guided code generation.

Figures

Figures reproduced from arXiv: 2507.18812 by the authors.

**Figure 1.** An overview of MemoCoder. Task: Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix cost[][] and a position (m, n) in cost[][]. Input-Output Pairs: • Input: ([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2) Output: 8 • Input: ([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2) Output: 12 • Input: ([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2) Output: 16 Constructed assertion cases: • asser… view at source ↗

**Figure 2.** An example problem description with an input [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Time series plot of the distribution of errors across [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Normalized error type transition matrix. The y-axis [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 4 canonical work pages

[2]

Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. 2024. CodeMirage: Hallucinations in Code Generated by Large Language Models. arXiv preprint arXiv:2408.08333 (2024)

arXiv 2024
[4]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732

arXiv 2021
[6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021
[7]

Thomas Dohmke, Marco Iansiti, and Greg Richards. 2023. Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle. arXiv:2306.15033 [econ.GN] https://arxiv.org/abs/2306.15033

arXiv 2023
[8]

Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. 2024. What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study. arXiv preprint arXiv:2407.06153 (2024)

arXiv 2024
[9]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava...

arXiv 2017
[10]

Yucan Guo et al. 2023. Code4UIE: Retrieval-Augmented Code Generation for Universal Information Extraction. arXiv preprint arXiv:2311.02962 (2023)

arXiv 2023
[11]

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021)

2021

Show all 38 references

[12]

Zhang, Michael Luck, and Heming Cui

Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. 2023. AgentCoder: Multi-Agent-Based Code Generation with Iterative Testing and Optimization. arXiv preprint arXiv:2312.13010 (2023)

2023 arXiv
[13]

Ashraful Islam et al

Md. Ashraful Islam et al . 2024. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) . 1234–1245

2024
[14]

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https://arxiv.org/...

2024 arXiv
[15]

Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. SelfEvolve: A Code Evolution Framework via Large Language Models. arXiv:2306.02907 [cs.CL] https://arxiv. org/abs/2306.02907

2023 arXiv
[16]

Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-planning Code Generation with Large Language Models. arXiv:2303.06689 [cs.SE] https://arxiv.org/abs/2303.06689

2024
[17]

Sabrina Kaniewski, Dieter Holstein, Fabian Schmidt, and Tobias Heer. 2024. Vulnerability Handling of AI-Generated Code – Existing Solutions and Open Challenges. arXiv:2408.08549 [cs.SE] https://arxiv.org/abs/2408.08549

2024 arXiv
[18]

Junjie Li, Aseem Sangalay, Cheng Cheng, Yuan Tian, and Jinqiu Yang. 2024. Fine Tuning Large Language Model for Secure Code Generation. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (Lisbon, Portugal) (FORGE ...

2024
[19]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172 [cs.CL] https://arxiv.org/abs/2307.03172

2023 arXiv
[20]

Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell. arXiv:2406.14673 [cs.CL] https://arxiv.org/abs/2406.14673

2024 arXiv
[21]

Quinn McNemar. 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12, 2 (1947), 153–157. https://doi.org/10.1007/BF02295996

1947 doi
[22]

Anders Giovanni Møller and Luca Maria Aiello. 2024. Prompt Refinement or Fine-tuning? Best Practices for Using LLMs in Computational Social Science Tasks. arXiv preprint arXiv:2408.01346 (2024)

2024 arXiv
[23]

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2024. A Comprehensive Overview of Large Language Models. arXiv:2307.06435 [cs.CL] https://arxiv.org/abs/2307.06435

2024 arXiv
[24]

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An Open Large Lan- guage Model for Code with Multi-Turn Program Synthesis. arXiv preprint arXiv:2203.13474 (2022)

2022 arXiv
[25]

Shayan Noei, Heng Li, and Ying Zhou. 2025. An Empirical Study on Release-Wise Refactoring Patterns. Proceedings of the ACM on Software Engineering 2, FSE (2025)

2025
[26]

James R Norris. 1998. Markov chains. Number 2. Cambridge university press

1998
[27]

Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is Self-Repair a Silver Bullet for Code Generation? arXiv:2306.09896 [cs.CL] https://arxiv.org/abs/2306.09896

2024 arXiv
[28]

Ruwei Pan et al . 2025. CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation. arXiv preprint arXiv:2501.07811 (2025)

2025 arXiv
[29]

Md Rizwan Parvez et al . 2021. REDCODER: Retrieval Augmented Code Gen- eration and Summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021. 2745–2755

2021
[30]

Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590 [cs.SE] https://arxiv.org/abs/2302.06590

2023 arXiv
[31]

Chanathip Pornprasit and Chakkrit Tantithamthavorn. 2024. Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation. arXiv preprint arXiv:2402.00905 (2024)

2024 arXiv
[32]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

2025 arXiv
[33]

Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang
[34]

Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. 2024. Online Adaptation of Language Models with a Memory of Amortized Contexts. arXiv:2403.04317 [cs.LG] https://arxiv.org/abs/ 2403.04317

2024 arXiv
[35]

Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. arXiv preprint arXiv:2305.07922 (2023)

2023 arXiv
[36]

Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2024. Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models. arXiv:2308.10462 [cs.SE] https://arxiv.org/abs/2308. 10462

2024 arXiv
[37]

Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou
[38]

Tenenbaum, and Chuang Gan

Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023. Planning with Large Language Models for Code Genera- tion. arXiv:2303.05510 [cs.LG] https://arxiv.org/abs/2303.05510

2023 arXiv
[39]

arXiv:2308.01240 [cs.CL] https://arxiv.org/abs/2308.01240

Evaluating Instruction-Tuned Large Language Models on Code Comprehen- sion and Generation. arXiv:2308.01240 [cs.CL] https://arxiv.org/abs/2308.01240

arXiv
[41]

Changzhi Zhou, Xinyu Zhang, Dandan Song, Xiancai Chen, Wanli Gu, Huipeng Ma, Yuhang Tian, Mengdi Zhang, and Linmei Hu. 2025. RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation. arXiv preprint arXiv:2502.09183 (2025)

2025 arXiv
[2023]

arXiv preprint arXiv:2307.14936 (2023)

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback. arXiv preprint arXiv:2307.14936 (2023)

2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2]

Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. 2024. CodeMirage: Hallucinations in Code Generated by Large Language Models. arXiv preprint arXiv:2408.08333 (2024)

arXiv 2024

[2] [4]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732

arXiv 2021

[3] [6]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021

[4] [7]

Thomas Dohmke, Marco Iansiti, and Greg Richards. 2023. Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle. arXiv:2306.15033 [econ.GN] https://arxiv.org/abs/2306.15033

arXiv 2023

[5] [8]

Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. 2024. What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study. arXiv preprint arXiv:2407.06153 (2024)

arXiv 2024

[6] [9]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava...

arXiv 2017

[7] [10]

Yucan Guo et al. 2023. Code4UIE: Retrieval-Augmented Code Generation for Universal Information Extraction. arXiv preprint arXiv:2311.02962 (2023)

arXiv 2023

[8] [11]

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021)

2021