Pith. sign in

REVIEW 2 major objections 2 minor 29 cited by

A Survey on Code Generation with LLM-based Agents

T0 review · 2 major / 2 minor · reviewed 2026-05-19 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-based code generation agents manage entire software projects autonomously from task breakdown through debugging and deployment.

desk verdict This survey organizes the exploding LLM code agent literature around three features and a single/multi split, but its real contribution is synthesis rather than new insight, and the taxonomy's completeness is the key unverified piece. read the letter →

arxiv 2508.00083 v2 pith:T3CR6JFH submitted 2025-07-31 cs.SE cs.AIcs.CLcs.LG

Yihong Dong , Xue Jiang , Jiaru Qian , Tian Wang , Kechi Zhang , Zhi Jin , Ge Li This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords LLMagentscodegenerationsoftwaredevelopmentlifecyclesingle-agentarchitecturemulti-agentsystemsevaluationbenchmarkstoolintegrationreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper surveys how large language model agents are shifting software creation away from isolated code snippets toward full-lifecycle systems. It organizes the growing literature around single-agent and multi-agent designs while mapping their use across planning, implementation, testing, and maintenance. A sympathetic reader would care because the work frames a move from algorithmic novelty to practical concerns such as reliability, workflow control, and external tool use. The survey also collects benchmarks, metrics, and representative tools and closes by naming open challenges and long-term research directions.

What carries the argument

The three core features of autonomy, expanded task scope across the software development lifecycle, and enhancement of engineering practicality, used to classify single-agent versus multi-agent architectures and to structure the review of applications, benchmarks, and tools.

What would settle it

Discovery of several widely cited, high-impact papers on LLM code generation whose architectures or workflows resist placement in either the single-agent or multi-agent category would show that the chosen organizational frame leaves out significant work.

Watch

Extended reading notes

Core claim

LLM-based code generation agents are defined by three distinguishing traits: autonomy that lets them oversee complete workflows without constant human direction, an expanded scope that reaches the full software development lifecycle rather than single functions or modules, and a practical engineering focus that stresses system reliability, process coordination, and integration with development tools over pure algorithmic advances.

Load-bearing premise

The survey assumes that the rapidly expanding literature can be cleanly and comprehensively sorted into single-agent and multi-agent categories with no major omissions or alternative groupings that would change the overall picture.

Editorial extensions

If this is right

  • Agents are applied across every phase of the software development lifecycle rather than only code writing.
  • Research attention shifts from new generation algorithms toward reliability, process management, and tool integration.
  • Evaluation moves beyond isolated code correctness to end-to-end project success measured by new benchmarks and metrics.
  • Multi-agent systems allow specialized roles and collaboration to tackle larger, more complex development tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Successful maturation of these agents would likely change how human developers spend their time, moving emphasis from routine coding to specification, oversight, and integration decisions.
  • The single-versus-multi-agent split may become less sharp as hybrid designs that combine both styles appear in real systems.
  • If the proposed research directions are pursued, non-experts could gain practical ways to build and maintain software with minimal manual coding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper surveys LLM-based code generation agents, claiming they are distinguished from prior techniques by three core features: autonomy to independently manage the full workflow from task decomposition to debugging; expanded scope across the entire software development lifecycle rather than isolated code snippets; and a shift toward engineering practicality including reliability, process management, and tool integration. It traces the developmental trajectory, categorizes core techniques into single-agent and multi-agent architectures, details applications across the SDLC, summarizes benchmarks/metrics and representative tools, and proposes long-term research directions based on identified challenges.

Significance. If the taxonomy and coverage hold, the survey would provide a useful organizing framework for a rapidly expanding subfield at the intersection of LLMs and software engineering, helping researchers identify patterns in agent architectures and gaps in practical deployment. The explicit focus on engineering challenges rather than pure algorithmic novelty is a constructive framing that aligns with industry needs.

major comments (2)
  1. [Abstract and §1] Abstract and §1 (Introduction): The central claim that the three features (autonomy, expanded SDLC scope, and engineering practicality) distinctly characterize LLM-based agents is load-bearing for the entire survey structure, yet the text provides no explicit contrast with prior non-agent code generation methods (e.g., direct LLM prompting or fine-tuned models) to demonstrate that these features are not already present or emergent in earlier work; without this grounding, the subsequent single/multi-agent categorization risks being an arbitrary overlay rather than a natural developmental trajectory.
  2. [§3] §3 (Architectures) and the literature selection description: The single-agent versus multi-agent taxonomy is presented as systematic, but the manuscript does not report search protocol, inclusion/exclusion criteria, database sources, or date range for the surveyed papers; this omission directly undermines the claim that the selected works represent core developments without major omissions, as hybrid or tool-centric systems that do not fit cleanly into the binary split could be under-represented.
minor comments (2)
  1. [Applications section] The abstract lists applications across the full SDLC but the corresponding section would benefit from a table summarizing which agent architectures are applied to which SDLC phases to improve readability.
  2. [Benchmarks section] Ensure that all cited benchmarks (e.g., HumanEval, MBPP extensions) include the exact metrics reported in the original papers rather than paraphrased summaries.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments on our survey. These observations help clarify the presentation of our core claims and improve the methodological transparency of the work. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract and §1] Abstract and §1 (Introduction): The central claim that the three features (autonomy, expanded SDLC scope, and engineering practicality) distinctly characterize LLM-based agents is load-bearing for the entire survey structure, yet the text provides no explicit contrast with prior non-agent code generation methods (e.g., direct LLM prompting or fine-tuned models) to demonstrate that these features are not already present or emergent in earlier work; without this grounding, the subsequent single/multi-agent categorization risks being an arbitrary overlay rather than a natural developmental trajectory.

    Authors: We agree that the distinction would benefit from more explicit grounding. The manuscript states that agents are 'distinct from previous code generation techniques' and enumerates the three features, but does not include a direct comparison. In the revision we will insert a short subsection (or expanded paragraph) in §1 that contrasts LLM-based agents with direct prompting and fine-tuned models, using concrete examples to show how autonomy over the full workflow, SDLC breadth, and engineering focus become central only in the agent setting. This addition will better motivate the subsequent taxonomy without altering the survey's scope. revision: yes

  2. Referee: [§3] §3 (Architectures) and the literature selection description: The single-agent versus multi-agent taxonomy is presented as systematic, but the manuscript does not report search protocol, inclusion/exclusion criteria, database sources, or date range for the surveyed papers; this omission directly undermines the claim that the selected works represent core developments without major omissions, as hybrid or tool-centric systems that do not fit cleanly into the binary split could be under-represented.

    Authors: We accept that the current draft lacks a transparent literature-selection description. Although the taxonomy reflects the dominant architectural patterns we observed, we will add a dedicated 'Literature Review Methodology' subsection at the start of §3. It will specify the databases searched (arXiv, Google Scholar, IEEE Xplore, ACM DL), the keyword combinations and date range (primarily 2022–2024), inclusion criteria (papers that explicitly describe LLM-powered agents for code generation), and exclusion criteria, together with a brief note on how hybrid or tool-centric systems are classified within the single- versus multi-agent framework. This revision directly addresses the concern about potential under-representation. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: survey organizes external literature without self-referential derivations

full rationale

This paper is a literature review that references external prior work to categorize LLM-based code generation agents into single-agent and multi-agent architectures and to trace developmental trajectories. It contains no equations, no fitted parameters, no predictions derived from its own inputs, and no self-citation chains that bear the central claims. The three core features (autonomy, expanded SDLC scope, engineering practicality) are presented as characterizations drawn from the surveyed body of work rather than results forced by the paper's own definitions or citations. Completeness of coverage is an assumption of any survey but does not constitute circularity under the defined criteria, as no reduction of a claimed result to the paper's own inputs is exhibited.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is a survey paper that reviews and organizes existing research on LLM-based code generation agents. It introduces no new free parameters, mathematical axioms, or invented entities; all content draws from cited prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey on Code Generation with LLM-based Agents." pith.science (2026). https://pith.science/paper/T3CR6JFH

@misc{pith2026250800083,
  author       = {Pith},
  title        = {Pith review of: A Survey on Code Generation with LLM-based Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3CR6JFH}},
  note         = {Machine review of arXiv:2508.00083}
}
read the original abstract

Code generation agents powered by large language models (LLMs) are revolutionizing the software development paradigm. Distinct from previous code generation techniques, code generation agents are characterized by three core features. 1) Autonomy: the ability to independently manage the entire workflow, from task decomposition to coding and debugging. 2) Expanded task scope: capabilities that extend beyond generating code snippets to encompass the full software development lifecycle (SDLC). 3) Enhancement of engineering practicality: a shift in research emphasis from algorithmic innovation toward practical engineering challenges, such as system reliability, process management, and tool integration. This domain has recently witnessed rapid development and an explosion in research, demonstrating significant application potential. This paper presents a systematic survey of the field of LLM-based code generation agents. We trace the technology's developmental trajectory from its inception and systematically categorize its core techniques, including both single-agent and multi-agent architectures. Furthermore, this survey details the applications of LLM-based agents across the full SDLC, summarizes mainstream evaluation benchmarks and metrics, and catalogs representative tools. Finally, by analyzing the primary challenges, we identify and propose several foundational, long-term research directions for the future work of the field.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification

    cs.CV 2026-08 conditional novelty 7.0 of 10

    MT-Web2Code is a 102-page, 16-domain multi-turn benchmark that measures how coding agents reconstruct missing web regions and fix localized defects while preserving the surrounding page.

  2. ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

    cs.AI 2026-07 conditional novelty 7.0 of 10

    On a new 480-task, 12-language benchmark where agents must clarify vague product briefs and build repositories from scratch, the best model achieves only 38.2% overall pass rate.

  3. How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

    cs.SE 2026-07 conditional novelty 7.0 of 10

    SE-agent development follows a recurring seven-stage loop where evaluation drives iteration, and challenges such as unreliable evaluation signals and comprehension debt emerge.

  4. PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    PRISM benchmark of over 10k pairs shows LLMs have a 41% average drop from code execution success to spatial correctness in programmatic video generation.

  5. From Runnable to Shippable: Multi-Agent Test-Driven Development for Generating Full-Stack Web Applications from Requirements

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    TDDev automates the full TDD loop for web app generation from requirements, delivering 34-48 percentage point quality gains and zero manual intervention in user studies.

  6. Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

    cs.AI 2026-05 conditional novelty 7.0 of 10

    Masked diffusion language models, not larger autoregressive LLMs, are the better building block for text-based world models in agentic RL, improving rollout fidelity, diversity, and downstream task success.

  7. CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding

    cs.SE 2025-11 unverdicted novelty 7.0 of 10

    Human-AI collaboration on CentaurEval's collaboration-necessary tasks reaches 31.11% success, far above standalone humans at 18.89% or LLMs at 0.67%.

  8. Breaking Customized LLMs for Coding: Automated Red Teaming for Instruction Backdoor Attacks

    cs.CR 2026-08 conditional novelty 6.0 of 10

    ARIA automatically generates stealthy backdoored system prompts for customized coding LLMs, achieving high attack success while preserving normal task performance.

  9. Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM

    cs.SE 2026-07 conditional novelty 6.0 of 10

    An AI coding agent can build a trustworthy process-modeling tool only when human-supervised verification scaffolding (oracles, round-trips, tests) contains its errors, which consumed 18% of 317 recorded turns.

  10. The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages

    cs.SE 2026-07 conditional novelty 6.0 of 10

    For five coding agents on 100 competitive-programming tasks, output-token cost varies sharply by language — OCaml costs roughly 1.3–1.7x Python — even after controlling for problem difficulty.

  11. Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

    cs.AI 2026-07 conditional novelty 6.0 of 10

    LGU models implication and incompatibility among LLM answers and reports consistent AUROC/AUARC gains over semantic entropy on QA benchmarks.

  12. A Closed-Loop Multi-Agent Framework for Robust Multi-Robot Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A closed-loop multi-agent LLM framework enables heterogeneous robots to collaboratively manipulate objects by decomposing tasks, grounding actions via visual tools, and recovering from execution failures hierarchically.

  13. An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems

    cs.SE 2026-07 conditional novelty 6.0 of 10

    On 12 real Java repositories, reflexive role-based multi-agent code generation is more similar to human code than a standalone LLM, yet remains partial and often non-compiling.

  14. Cheap Code, Costly Judgment: A Case Study on Governable Agentic Software Engineering

    cs.SE 2026-07 unverdicted novelty 6.0 of 10

    High-velocity agentic coding becomes governable when engineers convert recurring structural failures into durable, machine-actionable governance mechanisms rather than relying on continuous human code review.

  15. SAIGuard: Communication-State Simulation for Proactive Defense of LLM Multi-Agent Systems

    cs.MA 2026-06 unverdicted novelty 6.0 of 10

    SAIGuard uses communication-state simulation on the MAS interaction graph to detect and sanitize risky messages via reconstruction deviations, reducing attack success while preserving utility.

  16. Self-Evolving Deep Research via Joint Generation and Evaluation

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    SCORE is a shared-parameter co-evolutionary framework coupling generation and evaluation of deep research reports with a meta-harness to adapt evaluation standards as performance improves.

  17. Nautilus: From One Prompt to Plug-and-Play Robot Learning

    cs.RO 2026-05 unverdicted novelty 6.0 of 10

    NAUTILUS is a prompt-driven harness that automates plug-and-play adapters, typed contracts, and validation for policies, benchmarks, and robots in learning research.

  18. Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy

    cs.SE 2026-03 conditional novelty 6.0 of 10

    MCP server faults form five empirical categories—server setting, server/tool configuration, server/host configuration, documentation, and general programming—confirmed by a 41-practitioner survey.

  19. KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

    cs.LG 2026-02 conditional novelty 6.0 of 10

    By modeling layer skipping as a knapsack problem with context-dependent attention/MLP latencies, KnapSpec adaptively selects draft sub-networks that speed up self-speculative decoding by up to 1.47×.

  20. Large Language Model Agent for User-friendly Chemical Process Simulations

    physics.chem-ph 2026-01 unverdicted novelty 6.0 of 10

    An LLM agent integrated with AVEVA Process Simulation via MCP enables natural language driven flowsheet analysis, optimization, and construction for chemical separation processes.

  21. Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generation

    cs.LG 2026-01 conditional novelty 6.0 of 10

    VeRPO turns partial test-suite success into dense, difficulty-weighted rewards for RL code generation, beating GRPO by up to 8.8 pass@1 points at negligible cost.

  22. Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Student-coded agents from a 2020 logistics course beat 40 LLM-coded agents in a bidding-and-delivery tournament, with 33 of 40 AI agents losing to a simple baseline.

  23. Detection of LLM-assisted Code Plagiarism Using k-gram Software Birthmarks

    cs.SE 2026-07 conditional novelty 5.0 of 10

    Opcode k-gram software birthmarks, especially 2-grams with Dice similarity, detect LLM code paraphrasing with high Hmean, though ChatGPT-style models are hardest to catch.

  24. Personalization as Inverse Planning: Learning Latent Design Intents for Agentic Slide Generation via Structural Denoising

    cs.AI 2026-07 unverdicted novelty 5.0 of 10

    SPIRE approximates page-level slide personalization by training agents to denoise corrupted slide structures via collaborative RL, claiming a proof of consistency as a surrogate for inverse planning.

  25. TacEvo: Self-Evolving Architecture Discovery for Robotic Tactile Perception via LLM-Driven Quality-Diversity Search

    cs.RO 2026-06 unverdicted novelty 5.0 of 10

    TacEvo is an LLM-driven self-evolving search method that discovers neural architectures for robotic tactile force regression and grating classification, reporting fitness gains of 56.1% and 96.1% over 20 generations.

  26. Humanoid Whole-Body Manipulation via Active Spatial Brain and Generalizable Action Cerebellum

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    A multi-agent LLM framework for humanoid loco-manipulation that separates active spatial perception and task planning from generalizable action generation without task-specific real-robot data.

  27. Code as Agent Harness

    cs.CL 2026-05 accept novelty 5.0 of 10

    A survey that organizes existing work on LLM-based agents around code as the central harness, structured in three layers of interfaces, mechanisms, and multi-agent scaling, with applications across domains and listed ...

  28. Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithms

    cs.SE 2026-07 conditional novelty 4.5 of 10

    Coding agents write correct parallel CLRS code with little prompting, but meaningful speedups are model-, language-, and algorithm-dependent, with Sonnet strongest and GPT producing none.

  29. Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development

    cs.SE 2026-05 unverdicted novelty 4.0 of 10

    Agentic Agile-V uses Agile-V as backbone and a Specify-Constrain-Orchestrate-Prove-Evolve-Verify loop to convert AI agent conversations into traceable engineering artifacts with acceptance evidence.

Reference graph

Works this paper leans on

167 extracted references · 167 canonical work pages · cited by 29 Pith papers

  1. [1]

    Inductive programming: A survey of program synthesis techniques,

    E. Kitzelmann, “Inductive programming: A survey of program synthesis techniques,” inInternational Work- shop on Approaches and Applications of Inductive Pro- gramming (AAIP), 2009, pp. 50–73

    work page 2009

  2. [2]

    Latent predictor networks for code generation,

    W. Ling, E. Grefenstette, K. M. Hermann, T. Ko ˇcisk`y, A. Senior, F. Wang, and P . Blunsom, “Latent predictor networks for code generation,” inMeeting of the As- sociation for Computational Linguistics (ACL), 2016, pp. 599–609

    work page 2016

  3. [3]

    A syntactic neural model for general-purpose code generation,

    P . Yin and G. Neubig, “A syntactic neural model for general-purpose code generation,” inMeeting of the Association for Computational Linguistics (ACL), 2017, pp. 440–450

    work page 2017

  4. [4]

    LLaMA: Open and efficient founda- tion language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “LLaMA: Open and efficient founda- tion language models,” 2023

    work page 2023

  5. [5]

    LLaMA 2: Open foundation and fine- tuned chat models,

    H. Touvron, L. Martin, K. Stone, P . Albert, A. Alma- hairi, Y. Babaei, N. Bashlykov, S. Batra, P . Bhargava, S. Bhosaleet al., “LLaMA 2: Open foundation and fine- tuned chat models,” 2023. 19

    work page 2023

  6. [6]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wain- wright, P . Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,” inConference on Neural Information Processing Systems (NeurIPS), 2022, pp. 27 730–27 744

    work page 2022

  7. [7]

    Code LLaMA: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code LLaMA: Open foundation models for code,” 2023

    work page 2023

  8. [8]

    CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 8696–8708

    work page 2021

Show all 167 references
  1. [9]

    Competition-level code generation with Alphacode,

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrit- twieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lagoet al., “Competition-level code generation with Alphacode,”Science, vol. 378, no. 6624, pp. 1092– 1097, 2022

    2022

  2. [10]

    Self-planning code generation with large language models,

    X. Jiang, Y. Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–30, 2024

    2024

  3. [11]

    CodeChain: Towards modular code genera- tion through chain of self-revisions with representa- tive sub-modules,

    H. Le, H. Chen, A. Saha, A. Gokul, D. Sahoo, and S. Joty, “CodeChain: Towards modular code genera- tion through chain of self-revisions with representa- tive sub-modules,” inInternational Conference on Learn- ing Representations (ICLR), 2023

    2023

  4. [12]

    CodeCoR: An llm- based self-reflective multi-agent framework for code generation,

    R. Pan, H. Zhang, and C. Liu, “CodeCoR: An llm- based self-reflective multi-agent framework for code generation,” 2025

    2025

  5. [13]

    Codepori: Large scale model for autonomous software development by using multi- agents,

    Z. Rasheed, M. Waseem, M. Saari, K. Syst ¨a, and P . Abrahamsson, “Codepori: Large scale model for autonomous software development by using multi- agents,” 2024

    2024

  6. [14]

    An autonomous multi-agent llm frame- work for agile software development,

    S. Manish, “An autonomous multi-agent llm frame- work for agile software development,”International Journal of Trend in Scientific Research and Development, vol. 8, no. 5, pp. 892–898, 2024

    2024

  7. [15]

    Self-collaboration code generation via ChatGPT,

    Y. Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via ChatGPT,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–38, 2024

    2024

  8. [16]

    Chatdev: Communicative agents for software development,

    C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Conget al., “Chatdev: Communicative agents for software development,” in Meeting of the Association for Computational Linguistics (ACL), 2023, pp. 15 174–15 186

    2023

  9. [17]

    Metagpt: Meta programming for multi-agent collaborative framework,

    S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou et al., “Metagpt: Meta programming for multi-agent collaborative framework,” inInternational Conference on Learning Representations (ICLR), 2023

    2023

  10. [18]

    CodeXGLUE: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., “CodeXGLUE: A machine learning benchmark dataset for code understanding and generation,” in Conference on Neural Information Processing Systems (NeurIPS) Datasets and B...

    2021

  11. [19]

    ClarifyGPT: Empowering LLM- based code generation with intention clarification,

    F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang, “ClarifyGPT: Empowering LLM- based code generation with intention clarification,” 2023

    2023

  12. [20]

    Xuat-copilot: Multi-agent collaborative system for automated user acceptance testing with large language model,

    Z. Wang, W. Wang, Z. Li, L. Wang, C. Yi, X. Xu, L. Cao, H. Su, S. Chen, and J. Zhou, “Xuat-copilot: Multi-agent collaborative system for automated user acceptance testing with large language model,” 2024

    2024

  13. [21]

    Logiagent: Auto- mated logical testing for rest systems with llm-based multi-agents,

    K. Zhang, C. Zhang, C. Wang, C. Zhang, Y. Wu, Z. Xing, Y. Liu, Q. Li, and X. Peng, “Logiagent: Auto- mated logical testing for rest systems with llm-based multi-agents,” 2025

    2025

  14. [22]

    Ai-driven refactoring: A pipeline for identifying and correcting data clumps in git reposi- tories,

    N. Baumgartner, P . Iyenghar, T. Schoemaker, and E. Pulverm¨uller, “Ai-driven refactoring: A pipeline for identifying and correcting data clumps in git reposi- tories,”Electronics, vol. 13, no. 9, p. 1644, 2024

    2024

  15. [23]

    Abstract syntax networks for code generation and semantic parsing,

    M. Rabinovich, M. Stern, and D. Klein, “Abstract syntax networks for code generation and semantic parsing,” inMeeting of the Association for Computational Linguistics (ACL), 2017, pp. 1139–1149

    2017

  16. [24]

    GraphCodeBERT: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fuet al., “GraphCodeBERT: Pre-training code representations with data flow,” inInternational Conference on Learning Representations, 2021

    2021

  17. [25]

    UniXcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “UniXcoder: Unified cross-modal pre-training for code representation,” pp. 7212–7225, 2022

    2022

  18. [26]

    Qualityflow: An agentic workflow for program synthesis controlled by llm quality checks,

    Y. Hu, Q. Zhou, Q. Chen, X. Li, L. Liu, D. Zhang, A. Kachroo, T. Oz, and O. Tripp, “Qualityflow: An agentic workflow for program synthesis controlled by llm quality checks,” 2025

    2025

  19. [27]

    Self-organized agents: A LLM multi-agent framework toward ultra large- scale code generation and optimization,

    Y. Ishibashi and Y. Nishimura, “Self-organized agents: A LLM multi-agent framework toward ultra large- scale code generation and optimization,” 2024

    2024

  20. [28]

    CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,

    K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,” inMeeting of the Association for Computational Linguis- tics (ACL), 2024

    2024

  21. [29]

    Toolgen: Unified tool retrieval and calling via generation,

    R. Wang, X. Han, L. Ji, S. Wang, T. Baldwin, and H. Li, “Toolgen: Unified tool retrieval and calling via generation,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  22. [30]

    Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic AI,

    R. Sapkota, K. I. Roumeliotis, and M. Karkee, “Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic AI,” 2025

    2025

  23. [31]

    Adaptive test generation using a large language model,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “Adaptive test generation using a large language model,” 2023

    2023

  24. [32]

    A multi-agent llm-based juit test generation with strong oracles,

    Q. Xu, G. Wang, L. Briand, and K. Liu, “A multi-agent llm-based juit test generation with strong oracles,” 2025

    2025

  25. [33]

    Leveraging llms to automate energy-aware refactor- ing of parallel scientific codes,

    M. T. Dearing, Y. Tao, X. Wu, Z. Lan, and V . Taylor, “Leveraging llms to automate energy-aware refactor- ing of parallel scientific codes,” 2025

    2025

  26. [34]

    Sysllmatic: Large language models are software system optimizers,

    H. Peng, A. Gupte, R. Hasler, N. J. Eliopoulos, C.- C. Ho, R. Mantri, L. Deng, K. L ¨aufer, G. K. Thiru- vathukal, and J. C. Davis, “Sysllmatic: Large language models are software system optimizers,” 2025

    2025

  27. [35]

    Harnessing large language models for seed generation in greybox 20 fuzzing,

    W. Shi, Y. Zhang, X. Xing, and J. Xu, “Harnessing large language models for seed generation in greybox 20 fuzzing,” 2024

    2024

  28. [36]

    Mutation- guided llm-based test generation at meta,

    C. Foster, A. Gulati, M. Harman, I. Harper, K. Mao, J. Ritchey, H. Robert, and S. Sengupta, “Mutation- guided llm-based test generation at meta,” 2025

    2025

  29. [37]

    From LLMs to LLM-based agents for software engi- neering: A survey of current, challenges and future,

    H. Jin, L. Huang, H. Cai, J. Yan, B. Li, and H. Chen, “From LLMs to LLM-based agents for software engi- neering: A survey of current, challenges and future,” 2024

    2024

  30. [38]

    Large language model-based agents for software engineering: A survey,

    J. Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou, “Large language model-based agents for software engineering: A survey,” 2024

    2024

  31. [39]

    Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–30, 2025

    2025

  32. [40]

    Agents in software engineering: Survey, landscape, and vision,

    Y. Wang, W. Zhong, Y. Huang, E. Shi, M. Yang, J. Chen, H. Li, Y. Ma, Q. Wang, and Z. Zheng, “Agents in software engineering: Survey, landscape, and vision,” 2024

    2024

  33. [41]

    Intellicode compose: Code generation using trans- former,

    A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using trans- former,” inACM Joint Meeting on European Software Engineering Conference and Symposium on the Founda- tions of Software Engineering, 2020, pp. 1433–1443

    2020

  34. [42]

    Herrington,Code generation in action

    J. Herrington,Code generation in action. Manning Publications Co., 2003

    2003

  35. [43]

    Programming is hard-or at least it used to be: Educational opportuni- ties and challenges of AI code generation,

    B. A. Becker, P . Denny, J. Finnie-Ansley, A. Luxton- Reilly, J. Prather, and E. A. Santos, “Programming is hard-or at least it used to be: Educational opportuni- ties and challenges of AI code generation,” inACM Technical Symposium on Computer Science Education V . 1 (SIGCSE)...

    2023

  36. [44]

    In-IDE code generation from natural language: Promise and chal- lenges,

    F. F. Xu, B. Vasilescu, and G. Neubig, “In-IDE code generation from natural language: Promise and chal- lenges,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 2, pp. 1–47, 2022

    2022

  37. [45]

    Large language models for code generation: A comprehensive survey of challenges, techniques, evaluation, and applications,

    N. Huynh and B. Lin, “Large language models for code generation: A comprehensive survey of challenges, techniques, evaluation, and applications,” 2025

    2025

  38. [46]

    ChatGPT for good? on opportunities and challenges of large lan- guage models for education,

    E. Kasneci, K. Seßler, S. K ¨uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G ¨unnemann, E. H ¨ullermeieret al., “ChatGPT for good? on opportunities and challenges of large lan- guage models for education,”Learning and Individual Differences, vol. 103,...

    2023

  39. [47]

    A survey on evaluation of large language models,

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wanget al., “A survey on evaluation of large language models,”ACM transac- tions on intelligent systems and technology, vol. 15, no. 3, pp. 1–45, 2024

    2024

  40. [48]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Donget al., “A survey of large language models,” 2023

    2023

  41. [49]

    A comprehensive overview of large language models,

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,” ACM Transactions on Intelligent Systems and Technology, 2023

    2023

  42. [50]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inConference on Neural Information Processing Systems (NeurIPS), 2017

    2017

  43. [51]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P . D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,” 2021

    2021

  44. [52]

    Deepseek-coder: When the large language model meets programming– the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Liet al., “Deepseek-coder: When the large language model meets programming– the rise of code intelligence,” 2024

    2024

  45. [53]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Luet al., “Qwen2. 5-coder technical report,” 2024

    2024

  46. [54]

    Chain-of-thought prompting elicits reasoning in large language mod- els,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language mod- els,” inConference on Neural Information Processing Systems (NeurIPS), 2022, pp. 24 824–24 837

    2022

  47. [55]

    Planning in natural language improves LLM search for code generation,

    E. Wang, F. Cassano, C. Wu, Y. Bai, W. Song, V . Nath, Z. Han, S. Hendryx, S. Yue, and H. Zhang, “Planning in natural language improves LLM search for code generation,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  48. [56]

    WebGPT: Browser-assisted question-answering with human feedback,

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saun- ders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman, “WebGPT: Browser-assisted question-answering with human feedback,” 2022

    2022

  49. [57]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” inConference on Neu- ral Information Processing Systems (NeurIPS), 2023, pp. 68 539–68 551

    2023

  50. [58]

    Pal: Program-aided lan- guage models,

    L. Gao, A. Madaan, S. Zhou, U. Alon, P . Liu, Y. Yang, J. Callan, and G. Neubig, “Pal: Program-aided lan- guage models,” inInternational Conference on Machine Learning (ICML), 2023

    2023

  51. [59]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P . Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” inProceedings of the 36th annual acm symposium on user interface software and technology, 2023, pp. 1–22

    2023

  52. [60]

    Improving grounded language understanding in a collaborative environ- ment by interacting with agents through help feed- back,

    N. Mehta, M. Teruel, P . F. Sanz, X. Deng, A. H. Awadallah, and J. Kiseleva, “Improving grounded language understanding in a collaborative environ- ment by interacting with agents through help feed- back,”arXiv preprint arXiv:2304.10750, 2023

    2023

  53. [61]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., “The rise and potential of large language model based agents: A survey,”Science China Information Sciences, vol. 68, no. 2, p. 121101, 2025

    2025

  54. [62]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Linet al., “A survey on large language model based autonomous agents,” Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    2024

  55. [63]

    Large language model based multi-agents: A survey of progress and challenges,

    T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V . 21 Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,” 2024

    2024

  56. [64]

    Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects,

    Y. Cheng, C. Zhang, Z. Zhang, X. Meng, S. Hong, W. Li, Z. Wang, Z. Wang, F. Yin, J. Zhaoet al., “Ex- ploring large language model based intelligent agents: Definitions, methods, and prospects,” 2024

    2024

  57. [65]

    Prompt engineering with ChatGPT: a guide for academic writers,

    L. Giray, “Prompt engineering with ChatGPT: a guide for academic writers,”Annals of biomedical engineering, vol. 51, no. 12, pp. 2629–2633, 2023

    2023

  58. [66]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” 2023

    2023

  59. [67]

    Retrieval-augmented generation for large language models: A survey,

    Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” 2023

    2023

  60. [68]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P . Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschelet al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” inConferecce on Neural Information Processing Systems (NeurIPS), 2020, pp. 9459–9474

    2020

  61. [69]

    Large language model-aware in-context learning for code generation,

    J. Li, C. Tao, J. Li, G. Li, Z. Jin, H. Zhang, Z. Fang, and F. Liu, “Large language model-aware in-context learning for code generation,”ACM Transactions on Software Engineering and Methodology, 2023

    2023

  62. [70]

    Larger language models do in-context learning differently,

    J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhouet al., “Larger language models do in-context learning differently,” 2023

    2023

  63. [71]

    AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, and H. Cui, “AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,” 2023

    2023

  64. [72]

    HyperAgent: Generalist software engineering agents to solve coding tasks at scale,

    H. N. Phan, T. N. Nguyen, P . X. Nguyen, and N. D. Bui, “HyperAgent: Generalist software engineering agents to solve coding tasks at scale,” 2024

    2024

  65. [73]

    ToolCoder: Teach code generation models to use API search tools,

    K. Zhang, H. Zhang, G. Li, J. Li, Z. Li, and Z. Jin, “ToolCoder: Teach code generation models to use API search tools,” 2023

    2023

  66. [74]

    Repohyper: Better context retrieval is all you need for repository-level code completion,

    H. N. Phan, H. N. Phan, T. N. Nguyen, and N. D. Bui, “Repohyper: Better context retrieval is all you need for repository-level code completion,” 2024

    2024

  67. [75]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P . Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yanget al., “Self-refine: Iterative refinement with self-feedback,” inConference on Neural Information Pro- cessing Systems (NeurIPS), 2023, pp. 46 534–46 594

    2023

  68. [76]

    Self-Edit: Fault- aware code editor for code generation,

    K. Zhang, Z. Li, J. Li, G. Li, and Z. Jin, “Self-Edit: Fault- aware code editor for code generation,” inMeeting of the Association for Computational Linguistics (ACL), 2023, pp. 769–787

    2023

  69. [77]

    Executable code actions elicit better LLM agents,

    X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji, “Executable code actions elicit better LLM agents,” inInternational Conference on Machine Learning (ICML), 2024

    2024

  70. [78]

    Knowledge-aware code generation with large lan- guage models,

    T. Huang, Z. Sun, Z. Jin, G. Li, and C. Lyu, “Knowledge-aware code generation with large lan- guage models,” inIEEE/ACM International Conference on Program Comprehension (ICPC), 2024, pp. 52–63

    2024

  71. [79]

    A real-world webagent with planning, long context understanding, and program synthesis,

    I. Gur, H. Furuta, A. Huang, M. Safdari, Y. Matsuo, D. Eck, and A. Faust, “A real-world webagent with planning, long context understanding, and program synthesis,” inInternational Conference on Learning Rep- resentations (ICLR), 2024

    2024

  72. [80]

    Codeplan: Repository-level coding using LLMs and planning,

    R. Bairi, A. Sonwane, A. Kanade, V . D. C, A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, and S. Shet, “Codeplan: Repository-level coding using LLMs and planning,”ACM on Software Engineering, vol. 1, no. FSE, pp. 675–698, 2024

    2024

  73. [81]

    Generating code world models with large lan- guage models guided by monte carlo tree search,

    N. Dainese, M. Merler, M. Alakuijala, and P . Martti- nen, “Generating code world models with large lan- guage models guided by monte carlo tree search,” inConference on Neural Information Processing Systems (NeurIPS), 2024, pp. 60 429–60 474

    2024

  74. [82]

    Codetree: agent-guided tree search for code generation with large language models,

    J. Li, H. Le, Y. Zhou, C. Xiong, S. Savarese, and D. Sa- hoo, “Codetree: agent-guided tree search for code generation with large language models,” inConference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2025

    2025

  75. [83]

    Tree-of-Code: A tree-structured exploring frame- work for end-to-end code generation and execution in complex task handling,

    Z. Ni, Y. Li, N. Yang, D. Shen, P . Lv, and D. Dong, “Tree-of-Code: A tree-structured exploring frame- work for end-to-end code generation and execution in complex task handling,” 2024

    2024

  76. [84]

    Multi-stage guided code genera- tion for large language models,

    Y. Han and C. Lyu, “Multi-stage guided code genera- tion for large language models,”Engineering Applica- tions of Artificial Intelligence, vol. 139, no. PA, p. 109491, 2025

    2025

  77. [85]

    DARS: Dynamic action re-sampling to enhance coding agent performance by adaptive tree traversal,

    V . Aggarwal, O. Kamal, A. Japesh, Z. Jin, and B. Sch ¨olkopf, “DARS: Dynamic action re-sampling to enhance coding agent performance by adaptive tree traversal,” 2025

    2025

  78. [86]

    Verilogcoder: Autonomous Verilog coding agents with graph-based planning and abstract syntax tree (ast)-based wave- form tracing tool,

    C.-T. Ho, H. Ren, and B. Khailany, “Verilogcoder: Autonomous Verilog coding agents with graph-based planning and abstract syntax tree (ast)-based wave- form tracing tool,” inAAAI Conference on Artificial Intelligence (AAAI), vol. 39, no. 1, 2025, pp. 300–307

    2025

  79. [87]

    Guided search strategies in non- serializable environments with applications to soft- ware engineering agents,

    K. Zainullina, A. Golubev, M. Trofimova, S. Polezhaev, I. Badertdinov, D. Litvintseva, S. Karasik, F. Fisin, S. Skvortsov, M. Nekrashevich, A. Shevtsov, and B. Yangel, “Guided search strategies in non- serializable environments with applications to soft- ware engineering agent...

    2025

  80. [88]

    ROCODE: Integrating backtracking mech- anism and program analysis in large language models for code generation,

    X. Jiang, Y. Dong, Y. Tao, H. Liu, Z. Jin, W. Jiao, and G. Li, “ROCODE: Integrating backtracking mech- anism and program analysis in large language models for code generation,” inIEEE/ACM International Con- ference on Software Engineering (ICSE), 2025, pp. 670– 670

    2025

  81. [89]

    CodeTool: Enhancing programmatic tool invocation of LLMs via process supervision,

    Y. Lu, F. Ye, J. Li, Q. Gao, C. Liu, H. Luo, N. Du, X. Li, and F. Ren, “CodeTool: Enhancing programmatic tool invocation of LLMs via process supervision,” 2025

    2025

  82. [90]

    CodeNav: Beyond tool-use to using real-world codebases with llm agents,

    T. Gupta, L. Weihs, and A. Kembhavi, “CodeNav: Beyond tool-use to using real-world codebases with llm agents,” 2024

    2024

  83. [91]

    Op- timizing code runtime performance through context- aware retrieval-augmented generation,

    M. Acharya, Y. Zhang, K. Leach, and Y. Huang, “Op- timizing code runtime performance through context- aware retrieval-augmented generation,” inInterna- tional Conference on Program Comprehension (ICPC), 22 2025, pp. 1–5

    2025

  84. [92]

    Knowledge graph based repository-level code generation,

    M. Athale and V . Vaddina, “Knowledge graph based repository-level code generation,” inIEEE/ACM Inter- national Workshop on Large Language Models for Code (LLM4Code), 2025, pp. 169–176

    2025

  85. [93]

    cAST: Enhancing code retrieval-augmented generation with structural chunking via abstract syn- tax tree,

    Y. Zhang, X. Zhao, Z. Z. Wang, C. Yang, J. Wei, and T. Wu, “cAST: Enhancing code retrieval-augmented generation with structural chunking via abstract syn- tax tree,” 2025

    2025

  86. [94]

    AnalogCoder: Analog circuit design via training-free code generation,

    Y. Lai, S. Lee, G. Chen, S. Poddar, M. Hu, D. Z. Pan, and P . Luo, “AnalogCoder: Analog circuit design via training-free code generation,” inAAAI Conference on Artificial Intelligence (AAAI), no. 1, 2025, pp. 379–387

    2025

  87. [95]

    A self- iteration code generation method based on large lan- guage models,

    T. Chang, S. Chen, G. Fan, and Z. Feng, “A self- iteration code generation method based on large lan- guage models,” inInternational Conference on Parallel and Distributed Systems (ICP ADS), 2023, pp. 275–281

    2023

  88. [96]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” inMeeting Of The Association For Computational Linguistics (ACL), 2023

    2023

  89. [97]

    Is self-repair a silver bullet for code generation?

    T. X. Olausson, J. P . Inala, C. Wang, J. Gao, and A. Solar-Lezama, “Is self-repair a silver bullet for code generation?” 2024

    2024

  90. [98]

    Ledex: Training LLMs to better self-debug and explain code,

    N. Jiang, X. Li, S. Wang, Q. Zhou, S. B. Hossain, B. Ray, V . Kumar, X. Ma, and A. Deoras, “Ledex: Training LLMs to better self-debug and explain code,” inConference on Neural Information Processing Systems (NeurIPS), 2024, pp. 35 517–35 543

    2024

  91. [99]

    Codepori: Large-scale system for autonomous software develop- ment using multi-agent technology,

    Z. Rasheed, M. A. Sami, K.-K. Kemell, M. Waseem, M. Saari, K. Syst ¨a, and P . Abrahamsson, “Codepori: Large-scale system for autonomous software develop- ment using multi-agent technology,” 2024

    2024

  92. [100]

    Magis: LLM-based multi-agent framework for github issue resolution,

    W. Tao, Y. Zhou, Y. Wang, W. Zhang, H. Zhang, and Y. Cheng, “Magis: LLM-based multi-agent framework for github issue resolution,” inConference on Neural Information Processing Systems (NeurIPS), vol. 37, 2024, pp. 51 963–51 993

    2024

  93. [101]

    A pair pro- gramming framework for code generation via multi- plan exploration and feedback-driven refinement,

    H. Zhang, W. Cheng, Y. Wu, and W. Hu, “A pair pro- gramming framework for code generation via multi- plan exploration and feedback-driven refinement,” in IEEE/ACM International Conference on Automated Soft- ware Engineering (ASE), 2024, pp. 1319–1331

    2024

  94. [102]

    Soen-101: Code generation by emulating software process models using large language model agents,

    F. Lin, D. J. Kimet al., “Soen-101: Code generation by emulating software process models using large language model agents,” inInternational Conference on Software Engineering (ICSE), 2025, pp. 1527–1539

    2025

  95. [103]

    MAGE: A multi-agent engine for automated RTL code generation,

    Y. Zhao, H. Zhang, H. Huang, Z. Yu, and J. Zhao, “MAGE: A multi-agent engine for automated RTL code generation,” 2024

    2024

  96. [104]

    MapCoder: Multi-agent code generation for competitive problem solving,

    M. A. Islam, M. E. Ali, and M. R. Parvez, “MapCoder: Multi-agent code generation for competitive problem solving,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 4912–4944

    2024

  97. [106]

    QualityFlow: An agentic workflow for program synthesis controlled by LLM quality checks,

    Y. Hu, Q. Zhou, Q. Chen, X. Li, L. Liu, D. Zhang, A. Kachroo, T. Oz, and O. Tripp, “QualityFlow: An agentic workflow for program synthesis controlled by LLM quality checks,” 2025

    2025

  98. [107]

    MACRO: A multi-agent system for optimizing hpc code generation using large lan- guage models,

    A. Rahman, V . Cvetkovic, K. Reece, A. Walters, Y. Has- san, A. Tummeti, B. Torres, D. Cooney, M. Ellis, and D. S. Nikolopoulos, “MACRO: A multi-agent system for optimizing hpc code generation using large lan- guage models,” 2025

    2025

  99. [108]

    SEW: Self-evolving agentic workflows for automated code generation,

    S. Liu, J. Fang, H. Zhou, Y. Wang, and Z. Meng, “SEW: Self-evolving agentic workflows for automated code generation,” 2025

    2025

  100. [109]

    Self-evolving multi-agent collaboration networks for software development,

    Y. Hu, Y. Cai, Y. Du, X. Zhu, X. Liu, Z. Yu, Y. Hou, S. Tang, and S. Chen, “Self-evolving multi-agent collaboration networks for software development,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  101. [110]

    L2MAC: Large language model automatic computer for extensive code generation,

    S. Holt, M. R. Luyten, and M. van der Schaar, “L2MAC: Large language model automatic computer for extensive code generation,” 2023

    2023

  102. [111]

    Cogito, ergo sum: A neurobiologically- inspired cognition-memory-growth system for code generation,

    Y. Li, J. Li, Q. Wang, M. Yang, H. Kong, and S. Wang, “Cogito, ergo sum: A neurobiologically- inspired cognition-memory-growth system for code generation,” 2025

    2025

  103. [112]

    Gamegpt: Multi-agent collaborative framework for game development,

    D. Chen, H. Wang, Y. Huo, Y. Li, and H. Zhang, “Gamegpt: Multi-agent collaborative framework for game development,” 2023

    2023

  104. [113]

    Cleanagent: Automating data standardization with llm-based agents,

    D. Qi, Z. Miao, and J. Wang, “Cleanagent: Automating data standardization with llm-based agents,” 2024

    2024

  105. [114]

    Lingma SWE- GPT: An open development-process-centric language model for automated software improvement,

    Y. Ma, R. Cao, Y. Cao, Y. Zhang, J. Chen, Y. Liu, Y. Liu, B. Li, F. Huang, and Y. Li, “Lingma SWE- GPT: An open development-process-centric language model for automated software improvement,” 2024

    2024

  106. [115]

    Syncmind: Measuring agent out-of-sync recovery in collaborative software engineering,

    X. Guo, X. Wang, Y. Chen, S. Li, C. Han, M. Li, and H. Ji, “Syncmind: Measuring agent out-of-sync recovery in collaborative software engineering,” in International Conference on Machine Learning (ICML), 2025

    2025

  107. [116]

    Hallucination to consensus: Multi-agent LLMs for end-to-end test generation with accurate oracles,

    Q. Xu, G. Wang, L. Briand, and K. Liu, “Hallucination to consensus: Multi-agent LLMs for end-to-end test generation with accurate oracles,” 2025

    2025

  108. [117]

    Language agent tree search unifies reasoning, acting, and planning in language models,

    A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y.-X. Wang, “Language agent tree search unifies reasoning, acting, and planning in language models,” inInternational Conference on Machine Learning (ICML), 2024, pp. 62 138–62 160

    2024

  109. [118]

    MARE: Multi- agents collaboration framework for requirements en- gineering,

    D. Jin, Z. Jin, X. Chen, and C. Wang, “MARE: Multi- agents collaboration framework for requirements en- gineering,” 2024

    2024

  110. [119]

    Lemur: Harmoniz- ing natural language and code for language agents,

    Y. Xu, H. Su, C. Xing, B. Mi, Q. Liu, W. Shi, B. Hui, F. Zhou, Y. Liu, T. Xieet al., “Lemur: Harmoniz- ing natural language and code for language agents,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  111. [120]

    Codesim: Multi-agent code generation and problem solving through simulation-driven planning and debugging,

    M. A. Islam, M. E. Ali, and M. R. Parvez, “Codesim: Multi-agent code generation and problem solving through simulation-driven planning and debugging,” inFindings of the Association for Computational Linguis- tics, 2025

    2025

  112. [121]

    CodeS: Natural language to code repository via multi-layer sketch,

    D. Zan, A. Yu, W. Liu, D. Chen, B. Shen, W. Li, Y. Yao, Y. Gong, X. Chen, B. Guanet al., “CodeS: Natural language to code repository via multi-layer sketch,” 2024. 23

    2024

  113. [122]

    AgileCoder: Dynamic collaborative agents for software development based on agile methodology,

    M. H. Nguyen, T. P . Chau, P . X. Nguyen, and N. D. Bui, “AgileCoder: Dynamic collaborative agents for software development based on agile methodology,” inIEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE), 2025, pp. 156–167

    2025

  114. [123]

    Repaira- gent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P . Devanbu, and M. Pradel, “Repaira- gent: An autonomous, llm-based agent for program repair,” inInternational Conference on Software Engineer- ing (ICSE), 2025, pp. 694–694

    2025

  115. [124]

    AutoCodeRover: Autonomous program improve- ment,

    Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “AutoCodeRover: Autonomous program improve- ment,” inACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2024, pp. 1592– 1604

    2024

  116. [125]

    Swe-agent: Agent- computer interfaces enable automated software engi- neering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent- computer interfaces enable automated software engi- neering,” inConference on Neural Information Processing Systems (NeurIPS), 2024, pp. 50 528–50 652

    2024

  117. [126]

    Autosafecoder: A multi-agent framework for secur- ing llm code generation through static analysis and fuzz testing,

    A. Nunez, N. T. Islam, S. K. Jha, and P . Najafirad, “Autosafecoder: A multi-agent framework for secur- ing llm code generation through static analysis and fuzz testing,” 2024

    2024

  118. [127]

    Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,

    A. Antoniades, A. ¨Orwall, K. Zhang, Y. Xie, A. Goyal, and W. Wang, “Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  119. [128]

    SQLFixAgent: Towards semantic-accurate text-to-SQL parsing via consistency-enhanced multi-agent collaboration,

    J. Cen, J. Liu, Z. Li, and J. Wang, “SQLFixAgent: Towards semantic-accurate text-to-SQL parsing via consistency-enhanced multi-agent collaboration,” in AAAI Conference on Artificial Intelligence (AAAI), no. 1, 2025, pp. 49–57

    2025

  120. [129]

    Orcaloca: An llm agent framework for software issue localization,

    Z. Yu, H. Zhang, Y. Zhao, H. Huang, M. Yao, K. Ding, and J. Zhao, “Orcaloca: An llm agent framework for software issue localization,” inInternational Conference on Machine Learning (ICML), 2025

    2025

  121. [130]

    Patchpilot: A stable and cost-efficient agentic patching framework,

    H. Li, Y. Tang, S. Wang, and W. Guo, “Patchpilot: A stable and cost-efficient agentic patching framework,” 2025

    2025

  122. [131]

    Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute,

    Y. Ma, Y. Li, Y. Dong, X. Jiang, R. Cao, J. Chen, F. Huang, and B. Li, “Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute,” 2025

    2025

  123. [132]

    Adverintent-agent: Adversarial rea- soning for repair based on inferred program intent,

    H. Ye, A. Z. Yang, C. Hu, Y. Wang, T. Zhang, and C. Le Goues, “Adverintent-agent: Adversarial rea- soning for repair based on inferred program intent,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 1398–1420, 2025

    2025

  124. [133]

    Nemotron-cortexa: En- hancing llm agents for software engineering tasks via improved localization and solution diversity,

    A. Sohrabizadeh, J. Song, M. Liu, R. Roy, C. Lee, J. Raiman, and B. Catanzaro, “Nemotron-cortexa: En- hancing llm agents for software engineering tasks via improved localization and solution diversity,” in International Conference on Machine Learning (ICML), 2025

    2025

  125. [134]

    Auitestagent: Automatic re- quirements oriented gui function testing,

    Y. Hu, X. Wang, Y. Wang, Y. Zhang, S. Guo, C. Chen, X. Wang, and Y. Zhou, “Auitestagent: Automatic re- quirements oriented gui function testing,” 2024

    2024

  126. [135]

    https://developer.nvidia.com/blog/building-ai- agents-to-automate-software-test-case-creation/

  127. [136]

    ismell: Assembling llms with expert toolsets for code smell detection and refac- toring,

    D. Wu, F. Mu, L. Shi, Z. Guo, K. Liu, W. Zhuang, Y. Zhong, and L. Zhang, “ismell: Assembling llms with expert toolsets for code smell detection and refac- toring,” inIEEE/ACM International Conference on Au- tomated Software Engineering (ICASE), 2024, pp. 1345– 1357

    2024

  128. [137]

    EM- Assist: Safe automated extractmethod refactoring with LLMs,

    D. Pomian, A. Bellur, M. Dilhara, Z. Kurbatova, E. Bo- gomolov, A. Sokolov, T. Bryksin, and D. Dig, “EM- Assist: Safe automated extractmethod refactoring with LLMs,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 20...

    2024

  129. [138]

    Distributed approach to haskell based applications refactoring with llms based multi- agent systems,

    S. Siddeeq, Z. Rasheed, M. A. Sami, M. Hasan, M. Waseem, J. Rasku, M. Saari, K.-K. Kemell, and P . Abrahamsson, “Distributed approach to haskell based applications refactoring with llms based multi- agent systems,” 2025

    2025

  130. [139]

    AIDE: AI-driven exploration in the space of code,

    Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu, “AIDE: AI-driven exploration in the space of code,” 2025

    2025

  131. [140]

    LLM-based test-driven interactive code generation: User study and empirical evaluation,

    S. Fakhoury, A. Naik, G. Sakkas, S. Chakraborty, and S. K. Lahiri, “LLM-based test-driven interactive code generation: User study and empirical evaluation,” IEEE Transactions on Software Engineering, 2024

    2024

  132. [141]

    Automated repair of ambiguous natural language requirements,

    H. Jia, R. Morris, H. Ye, F. Sarro, and S. Mechtaev, “Automated repair of ambiguous natural language requirements,” 2025

    2025

  133. [142]

    Interactive agents to overcome ambiguity in software engineering,

    S. Vijayvargiya, X. Zhou, A. Yerukola, M. Sap, and G. Neubig, “Interactive agents to overcome ambiguity in software engineering,” 2025

    2025

  134. [143]

    HILDE: Intentional code generation via human-in-the-loop decoding,

    E. A. Gonz ´alez, R. Rothkopf, S. Lerner, and N. Po- likarpova, “HILDE: Intentional code generation via human-in-the-loop decoding,” 2025

    2025

  135. [144]

    Llm-powered test case generation for detecting bugs in plausible programs,

    K. Liu, Y. Liu, Z. Chen, J. M. Zhang, Y. Han, Y. Ma, Y. Dong, G. Li, and G. Huang, “Llm-powered test case generation for detecting bugs in plausible programs,” 2024

    2024

  136. [145]

    Evosuite: On the challenges of test case generation in the real world,

    G. Fraser and A. Arcuri, “Evosuite: On the challenges of test case generation in the real world,” inICST. IEEE Computer Society, 2013, pp. 362–369

    2013

  137. [146]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V . Le, and C. Sutton, “Program synthesis with large language models,” 2021

    2021

  138. [147]

    Measuring coding challenge com- petence with APPS,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring coding challenge com- petence with APPS,” inNeural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021

    2021

  139. [148]

    Livecodebench: Holistic and contamination free eval- uation of large language models for code,

    N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free eval- uation of large language models for code,” inInter- national Conference on Learning Representations (ICLR), 2025

    2025

  140. [149]

    Generalization or memorization: Data contamination and trustworthy evaluation for large language models,

    Y. Dong, X. Jiang, H. Liu, Z. Jin, B. Gu, M. Yang, and G. Li, “Generalization or memorization: Data contamination and trustworthy evaluation for large language models,” inACL (Findings). Association for Computational Linguistics, 2024, pp. 12 039–12 050

    2024

  141. [150]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, 24 O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  142. [151]

    Web-bench: A LLM code benchmark based on web standards and frameworks,

    K. Xu, Y. Mao, X. Guan, and Z. Feng, “Web-bench: A LLM code benchmark based on web standards and frameworks,” 2025

    2025

  143. [152]

    Gpt code editing benchmarks,

    P . Gauthier, “Gpt code editing benchmarks,” https://aider.chat/docs/benchmarks.html#the- benchmark, 2024, [Accessed 21-01-2025]

    2024

  144. [153]

    Evocodebench: An evolving code generation bench- mark aligned with real-world code repositories,

    J. Li, G. Li, X. Zhang, Y. Dong, and Z. Jin, “Evocodebench: An evolving code generation bench- mark aligned with real-world code repositories,” 2024

    2024

  145. [154]

    Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories,

    J. Li, G. Li, Y. Zhao, Y. Li, H. Liu, H. Zhu, L. Wang, K. Liu, Z. Fang, L. Wang, J. Ding, X. Zhang, Y. Zhu, Y. Dong, Z. Jin, B. Li, F. Huang, and Y. Li, “Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories,” 2024

    2024

  146. [155]

    Codescore: Evaluating code generation by learning code execution,

    Y. Dong, J. Ding, X. Jiang, G. Li, Z. Li, and Z. Jin, “Codescore: Evaluating code generation by learning code execution,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 3, pp. 77:1–77:22, 2025

    2025

  147. [156]

    Optimizing token consumption in llms: A nano surge approach for code reasoning efficiency,

    J. Hu, W. Zheng, Y. Liu, and Y. Liu, “Optimizing token consumption in llms: A nano surge approach for code reasoning efficiency,” 2025

    2025

  148. [157]

    Insights into deepseek- v3: Scaling challenges and reflections on hardware for ai architectures,

    C. Zhao, C. Deng, C. Ruan, D. Dai, H. Gao, J. Li, L. Zhang, P . Huang, S. Zhou, S. Ma, W. Liang, Y. He, Y. Wang, Y. Liu, and Y. X. Wei, “Insights into deepseek- v3: Scaling challenges and reflections on hardware for ai architectures,” inInternational Symposium on Computer Arch...

    2025

  149. [158]

    Sec-bench: Automated benchmarking of llm agents on real-world software security tasks,

    H. Lee, Z. Zhang, H. Lu, and L. Zhang, “Sec-bench: Automated benchmarking of llm agents on real-world software security tasks,” 2025

    2025

  150. [159]

    Sealign: Alignment training for software engineering agent,

    K. Zhang, H. Zhang, G. Li, J. You, J. Li, Y. Zhao, and Z. Jin, “Sealign: Alignment training for software engineering agent,” 2025

    2025

  151. [160]

    Llm-based multi-agent system for intelligent refactoring of haskell code,

    S. Siddeeq, M. Waseem, Z. Rasheed, M. M. Hasan, J. Rasku, M. Saari, H. Terho, K. Makela, K.-K. Kemell, and P . Abrahamsson, “Llm-based multi-agent system for intelligent refactoring of haskell code,” 2025

    2025

  152. [161]

    Evaluating software develop- ment agents: Patch patterns, code quality, and issue complexity in real-world github scenarios,

    Z. Chen and L. Jiang, “Evaluating software develop- ment agents: Patch patterns, code quality, and issue complexity in real-world github scenarios,” pp. 657– 668, 2025

    2025

  153. [162]

    https://github.com/copilot

  154. [163]

    https://www.cursor.com

  155. [164]

    https://lingma.aliyun.com

  156. [165]

    https://www.anthropic.com/claude-code

  157. [166]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y. Shen, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685

    2021 arXiv

  158. [167]

    Trustagent: Towards safe and trustworthy LLM-based agents,

    W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y. Zhang, “Trustagent: Towards safe and trustworthy LLM-based agents,” inFindings of the Association for Computational Linguistics: EMNLP, 2024, pp. 10 000– 10 016

    2024

  159. [168]

    Robust and secure code watermarking for large language models via ML/Crypto codesign,

    R. Zhang, N. Javidnia, N. Sheybani, and F. Koushan- far, “Robust and secure code watermarking for large language models via ML/Crypto codesign,” 2025

    2025

Pith tools

Reviewed May 19, 2026 · model on record in the stance chip above.