Pith. sign in

REVIEW 2 major objections 5 minor 53 references

Three collaborating LLM agents that plan, write, and review unit tests beat both other LLM tools and search-based generators on coverage and mutation score.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 05:23 UTC pith:5R4ZKY2Y

load-bearing objection Solid multi-agent test-gen system with a test-specialized KG; big mutation gains over EvoSuite and LLM baselines, but single-run numbers and no public artifacts keep the ranking claims provisional. the 2 major comments →

arxiv 2607.09101 v1 pith:5R4ZKY2Y submitted 2026-07-10 cs.SE

Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows

Quanjun Zhang , Ye Shang , Siqi Gu , Jianyi Zhou , Chunrong Fang , Zhenyu Chen , Liang Xiao This is my paper
classification cs.SE
keywords unit test generationlarge language modelsmulti-agent systemsknowledge graphcode coveragemutation testingsoftware testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Writing good unit tests is slow and hard; prior LLM tools either follow a fixed generate-then-fix loop or pull coarse context that misses the real dependencies of the method under test. This paper claims that the gap can be closed by copying how human developers actually work: first plan what should be tested, then generate and repair tests, then review adequacy and feed suggestions back. TestAgent implements that loop with three specialized agents that can call tools on demand and that read and write a repository-level knowledge graph built for testing (entities, call and inheritance links, plus stored summaries, reports, and failure analyses). On six Java projects the system reaches 97.46% execution rate, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score, beating same-backbone LLM baselines on every metric and roughly doubling the mutation score of a strong search-based tool. The same design ports to Python and to industrial codebases, and non-regression runs surface 154 confirmed real bugs at 92% precision. A sympathetic reader cares because the result is immediately usable tests that are both more thorough and more readable than what either pure search or single-shot LLM pipelines currently produce.

Core claim

Emulating the human unit-testing workflow with three collaborating LLM agents (requirement planner, test generator, test reviewer) that dynamically invoke tools and reason over a test-specialized repository knowledge graph produces unit tests whose correctness, coverage, and fault-detection power exceed both procedural LLM baselines and search-based generators under comparable conditions.

What carries the argument

The multi-agent human-testing-inspired workflow plus a test-specialized knowledge graph: three agents that plan requirements, generate/repair tests, and review them, each able to call graph-manipulation, retrieval, and runtime tools on demand, while the graph stores both static code relations and the testing artifacts produced during generation.

Load-bearing premise

That the single-run aggregate numbers (and the three-model vote that labels a failing test as a real defect) are stable enough to ground the superiority claims, even though language-model randomness is acknowledged and multi-seed variance is not reported.

What would settle it

Re-run the entire Java suite under identical GPT-4o settings for several independent seeds and check whether TestAgent still ranks first on line coverage, branch coverage, and especially mutation score against ChatUniTest, HITS, and EvoSuite; a reversal on any of those three metrics would falsify the central performance claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes TestAgent, a multi-agent LLM framework for unit test generation that emulates a human testing workflow with three specialized agents (requirement planner, test generator, test reviewer), a test-specialized repository knowledge graph built by static analysis, and on-demand tool APIs. On six Java projects it reports 97.46% execution rate, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score, outperforming ChatUniTest and HITS under the same GPT-4o backbone and substantially exceeding EvoSuite on mutation score (83.69% vs 43.59%). Supporting evidence includes ablations (knowledge graph removal is the largest drop), three LLM backends, a private industrial set, a Python port vs CodaMosa/CoverUp, a small user study, and 154/167 confirmed bug-revealing non-regression tests (92.22% precision).

Significance. If the results hold under re-execution, the work is a solid empirical contribution to LLM-based software testing: it couples a human-inspired multi-agent loop with a test-specific knowledge graph that stores both static dependencies and dynamic testing artifacts, and it shows large gains on mutation score and cross-language transfer. Strengths include a clear ablation isolating the knowledge graph, multi-backend and industrial evaluation, a Python reimplementation, and a careful manual analysis of real-bug detection. These elements go beyond typical single-prompt LLM test generators and are of practical interest to the SE community.

major comments (2)
  1. Table III (RQ1) and the abstract state superiority on coverage and mutation from single-run aggregates. Section V-C (Threats) explicitly notes that cost forced single runs and that LLM non-determinism can affect individual metrics; no seed-level variance, multi-run ranking stability, or error bars are reported for the same GPT-4o backbone vs ChatUniTest/HITS or for the 83.69% vs 43.59% gap vs EvoSuite. Given that the central claim is comparative superiority, at least a multi-seed re-run on a subset of projects (or bootstrap intervals) is needed to show that the ordering is stable; large absolute gaps mitigate but do not quantify the risk.
  2. Section III-D.2 and Discussion B rest the non-regression bug claim on a three-LLM voting confirmation (o3-mini, DeepSeek-R1, Qwen3-235B) that a failure is a real defect rather than a bad test. Pairwise Cohen's κ is only fair-to-moderate (0.39–0.55) and individual positive rates differ substantially (21–46%). The 154/167 (92.22%) figure is useful but depends on this unvalidated axiom and a single manual review pass; the paper should either validate the vote against a larger human-labeled set or present the 154 as human-confirmed candidates with the vote as a filter, not as an automatic ground truth.
minor comments (5)
  1. Table II: Lang and Chart are subsampled to 200 methods each without a stated sampling procedure or seed; a short note on stratification (or lack thereof) would improve reproducibility.
  2. Section IV-A.5: free parameters (max recursion depth 50, top-5 requirements, Jaccard threshold k, elastic 30/40-turn context control) are listed but not sensitivity-analyzed; a brief note on robustness would help.
  3. Figure 3 (cost-effectiveness) uses bubble size for time but the legend is only in the caption; an explicit legend would improve readability.
  4. Throughout: the system is alternately written TESTAGENT / TestAgent / TESTAGENT; pick one spelling for the camera-ready version.
  5. Related work (II-B) correctly distinguishes the test-specialized KG from RepoGraph/AutoCodeRover; a short explicit comparison table of node/edge types would make the novelty sharper.

Circularity Check

0 steps flagged

No circularity: empirical SE evaluation with external metrics and independent baselines; claims do not reduce to inputs by construction.

full rationale

TestAgent is a multi-agent system design evaluated on external, tool-measured quantities (JaCoCo line/branch coverage, PITest mutation score, compilation/execution rates). These metrics are not defined by free parameters of the method, nor fitted then re-predicted. Ablations (Table IV) remove components and report independent drops; baselines (EvoSuite, ChatUniTest, HITS) are run under the same backbone and produce their own scores. The three-LLM defect-confirmation vote (§III-D) is a post-hoc filter whose outputs were manually validated (Discussion B: 154/167 true positives); it does not define the primary coverage/mutation numbers. Self-citations appear only as background (prior surveys, assertion work) and are not load-bearing uniqueness theorems or ansatzes that force the reported results. The derivation chain is therefore ordinary experimental comparison, fully self-contained against external benchmarks, with no self-definitional loop, fitted-input-as-prediction, or renaming of a known result.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 3 invented entities

This is an empirical systems paper. Load-bearing content is the engineered multi-agent pipeline and evaluation protocol, not a mathematical derivation. Free parameters are engineering knobs (recursion limits, top-k requirements, similarity threshold k, search budgets). Domain assumptions include that human three-stage testing maps usefully onto LLM agents and that static KG structure plus runtime tools suffice for repository-level test reasoning. Invented entities are the named agents, the test-specialized KG schema, and the voting defect-confirmation mechanism.

free parameters (6)
  • max_recursion_depth = 50
    Set to 50 to bound agent loops; directly affects whether generation terminates with a best-effort test.
  • top_k_test_requirements = 5
    Only top-5 planner requirements are kept 'to prevent long contexts'; changes which scenarios are tested.
  • context_sufficiency_query_and_force_turns = 30 / 40 turns
    Agent asked about context after 30 turns; generation forced after 40 turns — hand-chosen elastic control.
  • similar_test_Jaccard_threshold_k
    Predefined threshold for recommending related tests via dependency Jaccard similarity; value not numerically fixed in text but is a free cutoff.
  • EvoSuite_search_budget = 300 s/class
    Baseline budget fixed at 300 seconds per class; affects fairness of coverage/mutation comparison.
  • Lang_Chart_method_subsample_size = 200 methods each
    200 methods randomly selected from large projects due to resource constraints; selection affects reported aggregates.
axioms (4)
  • domain assumption Human unit-testing practice is well modeled as three stages (requirement analysis, generation with feedback, review) that map onto three specialized LLM agents.
    Stated as motivation in Introduction and §III-D; the entire architecture rests on this analogy.
  • domain assumption A static entity-relation knowledge graph (packages/classes/methods/variables + inheritance/calls/references + test-specific nodes) plus runtime tools supplies sufficient repository context for high-quality unit tests.
    Core of §III-B/C; ablation removing the KG causes the largest metric drops.
  • ad hoc to paper Agreement of three differently trained LLMs on a failure is a reliable signal that the failure exposes a real defect in the focal method rather than a bad test.
    Defect confirmation mechanism in generator agent (§III-D); precision later measured manually at 92.22% on 167 cases.
  • domain assumption Standard coverage (JaCoCo) and mutation (PITest) tools are valid adequacy proxies for comparing generators.
    Used throughout RQ1–RQ6 metrics definition.
invented entities (3)
  • TestAgent multi-agent framework (requirement planner, test generator, test reviewer) no independent evidence
    purpose: Emulate human testing workflow with autonomous tool use and iterative feedback.
    Primary system contribution; no independent existence outside this paper.
  • Test-specialized repository knowledge graph (test nodes/relations + persistent testing artifacts) no independent evidence
    purpose: Provide fine-grained, test-aware context and store summaries/reports/failure analyses across generation.
    Distinguished from general SE graphs (RepoGraph, AutoCodeRover) in Related Work; ablation shows large effect.
  • Three-LLM voting defect-confirmation mechanism no independent evidence
    purpose: Separate real product bugs from flaky/incorrect tests so non-regression tests can be retained.
    Ad-hoc design choice; validated only by post-hoc manual labeling in Discussion B.

pith-pipeline@v1.1.0-grok45 · 26321 in / 3601 out tokens · 51918 ms · 2026-07-13T05:23:43.941479+00:00 · methodology

0 comments
read the original abstract

Recently, the emergence of Large Language Models (LLMs) has spurred a surge of research into automated unit test generation, yielding impressive performance and reducing manual effort. However, existing LLM-based approaches still suffer from two major limitations: (1) they follow rigid, procedural workflows that underutilize the autonomous reasoning potential of LLMs, making it difficult to dynamically adapt testing strategies based on real-time feedback; and (2) they rely on rule-based context extraction that is not tailored to test generation, failing to capture fine-grained code dependencies and test-specific knowledge required for deriving test requirements. In this paper, we propose TestAgent, an LLM-based test generation approach that addresses the above limitations by emulating human testing practices via a multi-agent collaboration mechanism. Particularly, TestAgent designs three specialized agents, namely a requirement planner, a test generator, and a test reviewer, to simulate how developers understand, construct, and validate unit tests. To unleash the autonomous capabilities of LLMs, we equip TestAgent with a set of tool APIs that can be invoked dynamically in an on-demand and adaptive manner. To further support repository-level reasoning, TestAgent constructs a test-specialized knowledge graph via static analysis, which captures code entities and their dependencies across the project and persistently stores testing artifacts (e.g., test reports and failure analyses) produced during generation. Experimental results show that TestAgent achieves 97.46% execution rate, 92.34% line coverage, 90.24% branch coverage, and 83.69% mutation score on six Java projects, outperforming LLM-based baselines across all metrics and achieving substantially higher mutation scores than search-based tools.

Figures

Figures reproduced from arXiv: 2607.09101 by Chunrong Fang, Jianyi Zhou, Liang Xiao, Quanjun Zhang, Siqi Gu, Ye Shang, Zhenyu Chen.

Figure 1
Figure 1. Figure 1: A Common Developer Workflow When Writing Unit Tests [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of TESTAGENT for repository-level code understanding, (2) a set of tools that orchestrate the communication between agents and the project environment, and (3) a multi-agent collaboration framework that interactively synthesizes, refines, and reviews high-quality unit tests. First, TESTAGENT constructs a knowledge graph by uti￾lizing static program analysis to extract program entities and semantic… view at source ↗
Figure 3
Figure 3. Figure 3: Cost-Effectiveness of TESTAGENT Variants core principles underlying TESTAGENT generalize well across programming languages, indicating its potential as a language￾agnostic test generation framework. Answer to RQ6: TESTAGENT exhibits strong cross￾language adaptability. When reimplemented for Python, it substantially outperforms SOTA Python baselines like CO￾DAMOSA and COVERUP in both line and branch coverag… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 14 linked inside Pith

  1. [1]

    A survey of unit testing practices,

    P. Runeson, “A survey of unit testing practices,”IEEE Softw., vol. 23, no. 4, pp. 22–29, 2006

    2006

  2. [2]

    Unit testing: test early, test often,

    M. Olan, “Unit testing: test early, test often,”J. Comput. Sci. Coll., vol. 19, no. 2, pp. 319–328, 2003

    2003

  3. [3]

    Software unit test coverage and adequacy,

    H. Zhu, P. A. V . Hall, and J. H. R. May, “Software unit test coverage and adequacy,”ACM Comput. Surv., vol. 29, no. 4, pp. 366–427, 1997

    1997

  4. [4]

    Exploring automated assertion generation via large language models,

    Q. Zhang, W. Sun, C. Fang, B. Yu, H. Li, M. Yan, J. Zhou, and Z. Chen, “Exploring automated assertion generation via large language models,”ACM Trans. Softw. Eng. Methodol., Oct. 2024, just Accepted. [Online]. Available: https://doi.org/10.1145/3699598

    doi:10.1145/3699598 2024

  5. [5]

    A survey on unit testing practices and prob- lems,

    E. Daka and G. Fraser, “A survey on unit testing practices and prob- lems,” in25th IEEE International Symposium on Software Reliability Engineering, ISSRE 2014, Naples, Italy, November 3-6, 2014. IEEE Computer Society, 2014, pp. 201–211

    2014

  6. [6]

    Randoop: Feedback-directed random testing for java,

    C. Pacheco and M. D. Ernst, “Randoop: Feedback-directed random testing for java,” inCompanion to the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2007, October 21-25, 2007, Montreal, Quebec, Canada. ACM, 2007, pp. 815–816

    2007

  7. [7]

    Evosuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: Automatic test suite generation for object-oriented software,” inSIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011. ACM, 2011, pp. 416–419. 13

    2011

  8. [8]

    The seed is strong: Seeding strategies in search-based software testing,

    ——, “The seed is strong: Seeding strategies in search-based software testing,” inICST. IEEE Computer Society, 2012, pp. 121–130

    2012

  9. [9]

    Whole test suite generation,

    ——, “Whole test suite generation,”IEEE Trans. Software Eng., vol. 39, no. 2, pp. 276–291, 2013

    2013

  10. [10]

    Automated test generation using model checking: An industrial evaluation,

    E. P. Enoiu, A. Causevic, T. J. Ostrand, E. J. Weyuker, D. Sundmark, and P. Pettersson, “Automated test generation using model checking: An industrial evaluation,”Int. J. Softw. Tools Technol. Transf., vol. 18, no. 3, pp. 335–353, 2016

    2016

  11. [11]

    Using model checking to gener- ate tests from requirements specifications,

    A. Gargantini and C. L. Heitmeyer, “Using model checking to gener- ate tests from requirements specifications,” inSoftware Engineering - ESEC/FSE’99, 7th European Software Engineering Conference, Held Jointly with the 7th ACM SIGSOFT Symposium on the Foundations of Software Engineering, Toulouse, France, September 1999, Proceedings, ser. Lecture Notes in ...

    1999

  12. [12]

    JBSE: a symbolic executor for java programs with complex heap inputs,

    P. Braione, G. Denaro, and M. Pezz `e, “JBSE: a symbolic executor for java programs with complex heap inputs,” inSIGSOFT FSE. ACM, 2016, pp. 1018–1022

    2016

  13. [13]

    An industrial evaluation of unit test generation: Finding real faults in a financial application,

    M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application,” inICSE-SEIP. IEEE Computer Society, 2017, pp. 263–272

    2017

  14. [14]

    Starcoder: May the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “Starcoder: May the source be with you!”Trans. Mach. Learn. Res., vol. 2023, 2023

    2023

  15. [15]

    Magicoder: Em- powering code generation with oss-instruct,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Em- powering code generation with oss-instruct,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    2024

  16. [16]

    Codet5: Identifier- aware unified pre-trained encoder-decoder models for code understand- ing and generation,

    Y . Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “Codet5: Identifier- aware unified pre-trained encoder-decoder models for code understand- ing and generation,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Computati...

    2021

  17. [17]

    Graphcodebert: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2...

    2021

  18. [18]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo- level coding challenges,

    K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo- level coding challenges,” inACL (1). Association for Computational Linguistics, 2024, pp. 13 643–13 658

    2024

  19. [19]

    A prompt learning framework for source code summarization,

    W. Sun, C. Fang, Y . You, Y . Chen, Y . Liu, C. Wang, J. Zhang, Q. Zhang, H. Qian, W. Zhao, Y . Liu, and Z. Chen, “A prompt learning framework for source code summarization,”CoRR, vol. abs/2312.16066, pp. arXiv– 2312, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Automatic semantic augmentation of language model prompts (for code summarization),

    T. Ahmed, K. S. Pai, P. T. Devanbu, and E. T. Barr, “Automatic semantic augmentation of language model prompts (for code summarization),” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 2024, pp. 220:1–220:13

    2024

  21. [21]

    Gamma: Re- visiting template-based automated program repair via mask prediction,

    Q. Zhang, C. Fang, T. Zhang, B. Yu, W. Sun, and Z. Chen, “Gamma: Re- visiting template-based automated program repair via mask prediction,” in38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 2023, pp. 535–547

    2023

  22. [22]

    A systematic literature review on large language models for automated program repair,

    Q. Zhang, C. Fang, Y . Xie, Y . Ma, W. Sun, Y . Yang, and Z. Chen, “A systematic literature review on large language models for automated program repair,”CoRR, vol. abs/2405.01466, pp. arXiv–2405, 2024

    arXiv 2024

  23. [23]

    A survey on large language models for software engineering,

    Q. Zhang, C. Fang, Y . Xie, Y . Zhang, Y . Yang, W. Sun, S. Yu, and Z. Chen, “A survey on large language models for software engineering,” CoRR, vol. abs/2312.15223, pp. arXiv–2312, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Repograph: Enhancing ai software engineer- ing with repository-level code graph,

    S. Ouyang, W. Yu, K. Ma, Z. Xiao, Z. Zhang, M. Jia, J. Han, H. Zhang, and D. Yu, “Repograph: Enhancing ai software engineer- ing with repository-level code graph,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 30 098–30 121

    2025

  25. [25]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, M. Christakis and M. Pradel, Eds. ACM, 2024, pp. 1592–1604

    2024

  26. [26]

    Unit test case generation with transformers and focal context,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,”arXiv e-prints, vol. abs/2009.05617, pp. arXiv–2009, 2020

    Pith/arXiv arXiv 2009

  27. [27]

    A3test: Assertion- augmented automated test case generation,

    S. Alagarsamy, C. Tantithamthavorn, and A. Aleti, “A3test: Assertion- augmented automated test case generation,”Inf. Softw. Technol., vol. 176, p. 107565, 2024

    2024

  28. [28]

    METH- ODS2TEST: A dataset of focal methods mapped to test cases,

    M. Tufano, S. K. Deng, N. Sundaresan, and A. Svyatkovskiy, “METH- ODS2TEST: A dataset of focal methods mapped to test cases,” in19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pittsburgh, PA, USA, May 23-24, 2022. ACM, 2022, pp. 299–303

    2022

  29. [29]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1703–1726, 2024

    2024

  30. [30]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, Porto de Galinhas, Brazil, July 15-19, 2024. ACM, 2024, pp. 572–576

    2024

  31. [31]

    Hits: High-coverage llm-based unit test generation via method slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “Hits: High-coverage llm-based unit test generation via method slicing,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineer- ing, 2024, pp. 1258–1268

    2024

  32. [32]

    Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration,

    S. Gu, C. Fang, Q. Zhang, F. Tian, J. Zhou, and Z. Chen, “Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration,”CoRR, vol. abs/2408.03095, pp. arXiv–2408, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Coverup: Coverage-guided llm-based test generation,

    J. A. Pizzorno and E. D. Berger, “Coverup: Coverage-guided llm-based test generation,”CoRR, vol. abs/2403.16218, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Casmodatest: A cascaded and model-agnostic self-directed framework for unit test generation,

    C. Ni, X. Wang, L. Chen, D. Zhao, Z. Cai, S. Wang, and X. Yang, “Casmodatest: A cascaded and model-agnostic self-directed framework for unit test generation,”CoRR, vol. abs/2406.15743, pp. arXiv–2406, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” inICSE. IEEE, 2023, pp. 919–931

    2023

  36. [36]

    Enhancing llm-based test generation for hard-to-cover branches via program analysis,

    C. Yang, J. Chen, B. Lin, J. Zhou, and Z. Wang, “Enhancing llm-based test generation for hard-to-cover branches via program analysis,”CoRR, vol. abs/2404.04966, pp. arXiv–2404, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 951–971, 2024

    2024

  38. [38]

    Sparks of artificial general intelligence: Early experiments with GPT-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Ka- mar, P. Lee, Y . T. Lee, Y . Li, S. M. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang, “Sparks of artificial general intelligence: Early experiments with GPT-4,”CoRR, vol. abs/2303.12712, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66

    2024

  40. [40]

    A unified debugging approach via llm-based multi-agent synergy,

    C. Lee, C. S. Xia, J. Huang, Z. Zhu, L. Zhang, and M. R. Lyu, “A unified debugging approach via llm-based multi-agent synergy,”CoRR, vol. abs/2404.17153, 2024

    arXiv 2024

  41. [41]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”CoRR, vol. abs/2405.15793, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Repairagent: An au- tonomous, llm-based agent for program repair,

    I. Bouzenia, P. T. Devanbu, and M. Pradel, “Repairagent: An au- tonomous, llm-based agent for program repair,” inICSE. IEEE, 2025, pp. 2188–2200

    2025

  43. [43]

    Swe-gpt: A process-centric language model for automated software improvement,

    Y . Ma, R. Cao, Y . Cao, Y . Zhang, J. Chen, Y . Liu, Y . Liu, B. Li, F. Huang, and Y . Li, “Swe-gpt: A process-centric language model for automated software improvement,”Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 2362–2383, 2025

    2025

  44. [44]

    MASAI: modular architecture for software-engineering AI agents,

    D. Arora, A. Sonwane, N. Wadhwa, A. Mehrotra, S. Utpala, R. Bairi, A. Kanade, and N. Natarajan, “MASAI: modular architecture for software-engineering AI agents,”CoRR, vol. abs/2406.11638, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Agilecoder: Dynamic collaborative agents for software development based on agile methodology,

    M. H. Nguyen, T. P. Chau, P. X. Nguyen, and N. D. Q. Bui, “Agilecoder: Dynamic collaborative agents for software development based on agile methodology,”CoRR, vol. abs/2406.11912, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Executable code actions elicit better LLM agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Executable code actions elicit better LLM agents,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=jJ9BoXAfFa

    2024

  47. [47]

    Rcagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models,

    Z. Wang, Z. Liu, Y . Zhang, A. Zhong, J. Wang, F. Yin, L. Fan, L. Wu, and Q. Wen, “Rcagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models,” inProceedings of the 33rd ACM International Conference on Information and Knowledge 14 Management, CIKM 2024, Boise, ID, USA, October 21-25, 2024, E. Serra and F. Spezzano, E...

    2024

  48. [48]

    SPOON: A library for implementing analyses and transformations of java source code,

    R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “SPOON: A library for implementing analyses and transformations of java source code,”Softw. Pract. Exp., vol. 46, no. 9, pp. 1155–1179, 2016

    2016

  49. [49]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” inInter- national Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014. ACM, 2014, pp. 437–440

    2014

  50. [50]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”CoRR, vol. abs/2412.19437, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”CoRR, vol. abs/2505.09388, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    A programmatic introduction to neo4j,

    J. Webber, “A programmatic introduction to neo4j,” inSPLASH. ACM, 2012, pp. 217–218

    2012