Pith. sign in

REVIEW 3 major objections 7 minor 62 references

A four-stage agent that builds fail-to-pass bug reproduction tests from issue reports reproduces 58–70% of real GitHub issues and helps repair systems fix more of them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 05:13 UTC pith:JHRDS5P6

load-bearing objection Solid multi-stage agent for issue reproduction tests with real FP gains and repair lift; the main caveat is that FP + LLM review is a thin proxy for faithful semantics. the 3 major comments →

arxiv 2607.09123 v1 pith:JHRDS5P6 submitted 2026-07-10 cs.SE

ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports

Quanjun Zhang , Yi Zheng , Ye Shang , Weifeng Sun , Haichuan Hu , Chunrong Fang , Zhenyu Chen , Liang Xiao This is my paper
classification cs.SE
keywords reproduction test generationmulti-stage agentsissue reportslarge language modelscode knowledge graphfail-to-pass testsautomated program repairrepository-level software engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Open-source issue reports almost never ship with executable tests that show the bug, so developers and automated repair tools struggle to confirm the problem or check a fix. Existing large-language-model methods mostly retrieve text and generate a test in one go; they under-use repository structure and do not explicitly plan the intermediate steps a human would take. ReProAgent treats reproduction as four successive agent stages—localize the bug, analyze the root cause along the execution path, plan setup and assertions, then generate and refine the test—backed by tools for stepwise thinking, keyword and code-graph retrieval, and sandboxed execution. On the SWT-bench suites it reaches fail-to-pass rates of 58.43% (lite) and 70.30% (verified), beats every reported baseline including a same-backbone general agent, costs about fourteen cents per issue, works with several model backbones, and, when the resulting tests are fed into existing repair pipelines, raises the number of correctly fixed issues.

Core claim

Organizing reproduction-test generation as a tool-augmented multi-stage agent (hierarchical localization, execution-path root-cause analysis, assertion-aware planning, and triadic review of issue/test/execution) produces substantially more valid fail-to-pass tests from natural-language issue reports than either fixed prompt pipelines or general-purpose software agents, and those tests measurably improve downstream automated repair.

What carries the argument

The four-stage agentic workflow of ReProAgent, supported by a task-specific toolset (sequential thinking/reflection, hybrid keyword-plus-code-knowledge-graph retrieval, and Docker sandbox execution) closed by a triadic review that accepts a failing test only when issue, test, and observed failure are mutually consistent.

Load-bearing premise

That a test which fails on the buggy code, passes after the official patch, and survives an automated review of issue/test/execution is a faithful reproduction of the reported bug rather than an incidental failure the same patch happens to fix.

What would settle it

On a stratified sample of accepted tests, have independent developers judge whether each test’s trigger conditions and failure mode match the issue description; a large fraction of “successful” tests judged non-faithful would falsify the claim that fail-to-pass plus triadic review equals semantic reproduction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reproduction-test generation can be treated as a first-class, multi-stage agent task rather than an auxiliary step inside general repair agents.
  • Generated fail-to-pass tests can be injected into existing repair frameworks (prompt guidance or patch filtering) to raise the number of correctly resolved issues.
  • The same staged design remains effective across several strong code LLMs, not only one proprietary backbone.
  • At roughly $0.14 per issue and few feedback iterations for most successes, repository-scale reproduction becomes economically practical for continuous integration or bug-triage pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because later stages can partially recover from imperfect localization, investing in stronger hierarchical or graph-based localizers is a high-leverage next step.
  • The large drop when triadic review and execution feedback are removed suggests that most remaining errors are semantic mismatches, not mere generation failures—pointing to better failure-mode matching as the next bottleneck.
  • The language-agnostic stage design plus graph construction via Tree-sitter-like tools implies the approach can transfer to other languages once sandbox and test-runner adapters exist.
  • Adaptive early-stopping of feedback iterations (most successes finish in ≤5) could cut cost further without large accuracy loss.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes ReProAgent, a tool-augmented multi-stage agent for generating fail-to-pass bug reproduction tests from natural-language issue reports over repository-scale codebases. The workflow comprises hierarchical bug localization, execution-path root-cause analysis, assertion-aware test planning, and test generation with execution feedback and an LLM triadic review (issue × test × execution). Supporting tools include sequential thinking, keyword and knowledge-graph retrieval, and sandboxed runtime interaction. On SWT-bench-lite and SWT-bench-verified the system reports FP rates of 58.43% and 70.30%, outperforming prompting and agent baselines, including a same-backbone OpenHands comparison under GPT-5-mini (+20.43 / +7.90 pp), at about $0.14 per instance. The framework is shown to transfer across several LLM backbones, ablations attribute gains to each stage (especially feedback+review), and generated tests improve SWE-agent and Agentless repair success on SWE-bench subsets.

Significance. Automatic issue reproduction is a practically important and still under-served SE problem: reproduction tests both confirm reports and supply executable oracles for repair. The work’s main contribution is not a single new model but a structured agent design that couples repository-graph retrieval, staged reasoning, and closed-loop execution review. Strengths that should be credited include: evaluation on two public SWT-bench subsets against a broad baseline set; a same-backbone OpenHands comparison that reduces model confounds; multi-LLM generalization; stage ablations and localization-sensitivity analysis; overlap analysis showing 36 unique successes on lite; cost and iteration statistics; and concrete evidence that the generated tests improve two existing repair frameworks. If the reported gains hold under the community FP protocol, the paper is a solid empirical advance for agentic repository-level testing and a useful building block for repair pipelines.

major comments (3)
  1. [§4.4 Evaluation Metrics; §3.6 Triadic Review; Table 7] §4.4 and §3.6: The central claim that ReProAgent “successfully reproduces” 58.43%/70.30% of issues is operationalized as SWT-bench Fail-to-Pass (fail on the pre-patch tree, pass after the gold patch), gated by an LLM triadic review of issue/test/execution. A test can still be FP by exercising any behavior the gold PR changes, not necessarily the issue’s intended trigger and failure mode—especially when patches are non-minimal. The review model is the same class of model that wrote the test, and no human semantic audit of accepted FP cases is reported. Table 7 further shows that removing feedback+triadic review collapses FP from 56.88%→18.84% (lite) and 60.05%→22.86% (verified), so most accepted successes depend on this self-checked loop. Please either (i) report a manual audit of a stratified sample of accepted FP tests (including the 36 unique successes in Fig. 5) for semantic alignment
  2. [§5.3 RQ3; Table 7] §5.3 / Table 7: Component ablations are run only with Qwen3-Coder, not with the primary GPT-5-mini backbone used for the headline results and the OpenHands comparison. Given that feedback+review is the dominant stage and that backbone capability clearly affects absolute FP (Table 6), the attribution of gains to individual stages for the main system is only partially supported. At minimum, run the critical ablation (w/o feedback+triadic review) and preferably localization removal under GPT-5-mini on one subset, or justify why stage contributions transfer without that check.
  3. [§5.1 RQ1; Tables 2–3] §5.1 same-backbone comparison: On SWT-bench-lite the authors reproduce OpenHands under GPT-5-mini; on SWT-bench-verified they cite the public leaderboard result (62.4%). Leaderboard configurations (prompts, tools, budgets, retries) may not match the lite reproduction setup, so the +7.90 pp verified gap is not fully controlled. Please either re-run OpenHands under the same configuration used for lite on verified, or clearly mark the verified comparison as non-controlled and de-emphasize it relative to the lite result.
minor comments (7)
  1. [Abstract; §1] Abstract and §1: “agen stages” / “four agen stages” appears to be a typo for “agent stages.”
  2. [Tables 2–3] Tables 2–3 mix many backbone models (GPT-4/4o, Claude, Amazon Bedrock, GPT-5-mini). The relative-improvement percentages vs. weaker backbones are easy to over-read; consider a secondary table restricted to same- or similar-backbone rows, or move heterogeneous comparisons to an appendix.
  3. [§3.2.2] §3.2.2 knowledge-graph construction: entity/edge types and how Graph-Search queries are formulated for the agent are only sketched. A short schema figure or example query for django-15347 would improve reproducibility.
  4. [§4.5 Implementation Details] §4.5: temperature=0.7 with a multi-stage agent is relatively high for reproducibility; report variance over a few seeds for at least one backbone/subset, or note that a single run was used.
  5. [Fig. 5] Fig. 5 overlap diagram is hard to parse in text form; ensure the camera-ready figure has a clear legend and that unique counts are stated in the caption.
  6. [§6.1; Table 9] §6.1 repair integration uses Qwen3-Coder while headline reproduction uses GPT-5-mini; state this consistently when claiming “improves downstream issue resolution.”
  7. [§2.2–2.3] Related work could more explicitly position against concurrent issue-to-test agents (e.g., Aegis, Echo, iCoRe cited in the bibliography) in a comparison table of stages/tools rather than prose only.

Circularity Check

0 steps flagged

No circularity: headline FP rates are measured on external SWT-bench fail-to-pass oracles, not quantities defined by the method’s free parameters or self-citation chain.

full rationale

ReProAgent is an empirical systems paper. Its central claims (58.43% / 70.30% FP rates; same-backbone gains over OpenHands; ablation drops; repair integration gains) are evaluated against SWT-bench / SWE-bench instances and public leaderboard baselines using the external Fail-to-Pass protocol (fail on pre-patch tree, pass after the gold patch; §4.4). That metric is not fitted from ReProAgent’s own outputs and is not defined in terms of the agent’s intermediate stages. The four-stage pipeline (localization, root-cause analysis, planning, generation with triadic review) is a design choice whose contribution is tested by ablation (Table 7) and multi-backbone runs (Table 6), not a derivation that reduces the reported rates to their inputs by construction. Author self-citations (surveys and related SE/LLM work) appear only as background and do not supply a uniqueness theorem or load-bearing premise that forces the results. The LLM triadic review (§3.6) is an internal generation/filter component, not the evaluation oracle; concerns that FP may credit incidental gold-patch-sensitive paths are validity/proxy issues, not circularity. No self-definitional loop, fitted-input-as-prediction, or ansatz-via-self-citation step is present.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

As an empirical systems paper, load-bearing content is mostly engineering choices and evaluation conventions rather than physical axioms. Free parameters are decoding and budget knobs; axioms are standard SE evaluation assumptions and the developer-workflow stage split; invented entities are the named framework components (especially triadic review and the staged controller).

free parameters (3)
  • LLM temperature / top_p (and top_k, repetition_penalty for open models)
    Set to 0.7 / 0.8 (and 20 / 1.05) without sensitivity analysis; generation quality and FP rate can depend on these knobs.
  • Per-stage LLM call budget (50) and feedback iteration cap (20)
    Hard limits chosen by authors; successful cases average 1.29 feedback iterations, so the cap shapes cost and completeness of search.
  • Primary backbone choice (GPT-5-mini) and stage orchestration prompts
    Framework performance is tightly coupled to the backbone and unpublished prompt templates that define each stage’s behavior.
axioms (4)
  • domain assumption Fail-to-pass on the gold patch is a valid primary measure of successful issue reproduction.
    Stated evaluation protocol following SWT-bench; underpins all headline percentages (§4.4).
  • ad hoc to paper Developer-style decomposition into localization → root-cause → planning → generation is an appropriate inductive bias for agent design.
    Motivated by developer workflow (§3.1) but not independently proven optimal versus other stage graphs.
  • ad hoc to paper LLM triadic review can decide whether a failing test is semantically aligned with the issue rather than an incidental failure.
    Core acceptance gate in §3.6; reliability is assumed from design, not validated against human labels at scale.
  • domain assumption Tree-sitter-extracted code knowledge graphs plus keyword tools provide sufficient repository context for cross-file bugs in the evaluated Python projects.
    Hybrid retrieval design (§3.2.2); external validity beyond Python is asserted as future work.
invented entities (3)
  • ReProAgent four-stage agent workflow no independent evidence
    purpose: Organize repository-level reproduction-test generation as hierarchical localization, execution-path RCA, assertion-aware planning, and triadic-review generation.
    Named system contribution; evaluated only via end-to-end FP rates and ablations in this paper.
  • Triadic review procedure (issue × test × execution) no independent evidence
    purpose: Accept only failures judged semantically consistent with the report, not mere crashes or environment errors.
    Introduced in §3.6 as the validation loop; no external benchmark of review accuracy is provided.
  • Repository code knowledge graph for dependency retrieval no independent evidence
    purpose: Expose cross-file call/membership relations beyond keyword search.
    Built with Tree-sitter + Neo4j for this pipeline; graph schema and coverage are paper-specific engineering.

pith-pipeline@v1.1.0-grok45 · 26158 in / 3485 out tokens · 46329 ms · 2026-07-13T05:13:48.226799+00:00 · methodology

0 comments
read the original abstract

Reproduction tests help developers confirm reported issues and provide executable feedback for issue resolution, yet issue reports in open-source projects rarely include such tests. Recent studies have explored generating issue reproduction tests from issue reports with large language models, but existing approaches largely rely on prompt-based pipelines that retrieve textual context and generate tests. This limits their ability to understand how reported issues behave in repository-scale codebases and to flexibly organize the construction of reproduction tests. In this paper, we propose ReProAgent, a multi-stage agent framework for reproduction test generation from issue reports. ReProAgent decomposes the task into four agent stages: bug localization, root cause analysis, test planning, and test generation. To support these stages, ReProAgent integrates task-specific tools for task decomposition and reflection, context retrieval from both textual sources and repository graphs, and runtime interaction with the execution environment. Experiments on SWT-bench-lite and SWT-bench-verified show that ReProAgent successfully reproduces 58.43% and 70.30% of issues, outperforming all baselines, with an average cost of $0.14 per instance. For example, when equipped with GPT-5-mini, ReProAgent exceeds OpenHands with the same backbone by 20.43 and 7.90 percentage points, respectively. ReProAgent also generalizes across multiple backbone LLMs and improves downstream issue resolution performance when integrated with existing repair approaches.

Figures

Figures reproduced from arXiv: 2607.09123 by Chunrong Fang, Haichuan Hu, Liang Xiao, Quanjun Zhang, Weifeng Sun, Ye Shang, Yi Zheng, Zhenyu Chen.

Figure 1
Figure 1. Figure 1: The django-15347 issue from SWT-bench 2.4 Motivating Example We use a real-world issue django-15347 to illustrate the challenges in repository-level issue repro￾duction test generation. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the ReProAgent framework down complex objectives and revise intermediate reasoning. Context retrieval tools enable the agent to inspect relevant code and identify dependencies among modules, classes, and functions. Runtime interaction tools allow the agent to observe execution behavior and validate hypotheses in the actual repository environment [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Root Cause Analysis result of Django-15347 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Review result of Django-15347 To determine whether a failing test truly reproduces the reported issue, we introduce a triadic review procedure over the issue report, the generated test, and the execution result, as shown in [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overlap analysis against baselines [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 6 canonical work pages

  1. [1]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. InICML (Proceedings of Machine Learning Research). PMLR / OpenReview.net

    2025

  2. [2]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avi Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. InInternational Conference on Machine Learning

    2025

  3. [3]

    Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. 2026. Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection. arXiv:2508.06365 [cs.SE] https://arxiv.org/abs/2508.06365 Accepted to ICSE 2026

    arXiv 2026

  4. [4]

    Aider-AI. 2026. Aider: AI Pair Programming in Your Terminal. https://github.com/Aider-AI/aider. Accessed: 2026-07-05. , Vol. 1, No. 1, Article 1. Publication date: January 2026. ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports 1:21

    2026

  5. [5]

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2024. A3Test: Assertion-Augmented Automated Test case generation.Inf. Softw. Technol.176, C (Dec. 2024), 15 pages. doi:10.1016/j.infsof.2024.107565

    doi:10.1016/j.infsof.2024.107565 2024

  6. [6]

    Juan Altmayer Pizzorno and Emery D Berger. 2025. Coverup: Effective High Coverage Test Generation for Python. Proceedings of the ACM on Software Engineering2, FSE (2025), 2897–2919

    2025

  7. [7]

    Amazon Web Services. 2026. Amazon Q: Generative AI Assistant for Software Development and Enterprise Data. https://aws.amazon.com/q/

    2026

  8. [8]

    Anthropic. 2025. Model Context Protocol Specification. https://modelcontextprotocol.io/specification/2025-11-25. Accessed: 2025-12-08

    2025

  9. [9]

    Anthropic. 2025. Sequential Thinking. https://github.com/modelcontextprotocol/servers/tree/main/src/ sequentialthinking. Accessed: 2025-12-08

    2025

  10. [10]

    Thierry N. D. Chekam, Mike Papadakis, Yue Jia, and Mark Harman. 2017. An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation that Avoids the Unreliable Clean Program Assumption. InProceedings of the 39th International Conference on Software Engineering. 597–608. doi:10.1109/ICSE.2017.61

    work page doi:10.1109/icse.2017.61 2017

  11. [11]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A Framework for LLM-Based Test Generation. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572–576

    2024

  12. [12]

    Xiang Cheng, Fan Sang, Yizhuo Zhai, Xiaokuan Zhang, and Taesoo Kim. 2025. Rug: Turbo LLM for Rust Unit Test Generation. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 634–634

    2025

  13. [13]

    CodeGraphContext Contributors. 2026. CodeGraphContext: Turn Code Repositories into a Queryable Graph for AI Agents. https://github.com/CodeGraphContext/CodeGraphContext

    2026

  14. [14]

    Ermira Daka and Gordon Fraser. 2014. A Survey on Unit Testing Practices and Problems. In2014 IEEE 25th International Symposium on Software Reliability Engineering. 201–211

    2014

  15. [15]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv 2024

  16. [16]

    Zhiwei Fei, Yue Pan, Federica Sarro, Jidong Ge, Marc Liu, Vincent Ng, and He Ye. 2026. Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation.arXiv preprint arXiv:2603.07326(2026)

    arXiv 2026

  17. [17]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-oriented Software. InProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering(Szeged, Hungary)(ESEC/FSE ’11). Association for Computing Machinery, New York, NY, USA, 416–419. doi:10.1145/2025113.2025179

    doi:10.1145/2025113.2025179 2011

  18. [18]

    Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation.IEEE Transactions on Software Engineering39, 2 (2013), 276–291. doi:10.1109/TSE.2012.14

    doi:10.1109/tse.2012.14 2013

  19. [19]

    Santos, Andrew Popovich, Mehdi Mirakhorli, and Mei Nagappan

    Danielle Gonzalez, Joanna C.S. Santos, Andrew Popovich, Mehdi Mirakhorli, and Mei Nagappan. 2017. A Large-Scale Study on the Usage of Testing Patterns That Address Maintainability Attributes: Patterns for Ease of Modification, Diagnoses, and Comprehension. In2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 391–401. doi:10...

    work page doi:10.1109/msr.2017.8 2017

  20. [20]

    Siqi Gu, Quanjun Zhang, Kecheng Li, Chunrong Fang, Fangyuan Tian, Liuchuan Zhu, Jianyi Zhou, and Zhenyu Chen

  21. [21]

    Testart: Improving llm-based unit testing via co-evolution of automated generation and repair iteration.arXiv preprint arXiv:2408.03095(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Andre Hora and Gordon Fraser. 2026. Understanding Bug-Reproducing Tests: A First Empirical Study.arXiv preprint arXiv:2602.02965(2026)

    arXiv 2026

  23. [23]

    Laura Inozemtseva and Reid Holmes. 2014. Coverage is Not Strongly Correlated with Test Suite Effectiveness. In Proceedings of the 36th International Conference on Software Engineering. 435–445

    2014

  24. [24]

    Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2025. Testgeneval: A Real World Unit Test Generation and Test Completion Benchmark. InICLR. OpenReview.net

    2025

  25. [25]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  26. [26]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. InProceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA)(ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 437–440. doi:10.1145/ 2610384.2628055

    arXiv 2014

  27. [27]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction. InProceedings of the 45th International Conference on Software Engineering(Melbourne, Victoria, Australia)(ICSE ’23). IEEE Press, 2312–2323. doi:10.1109/ICSE48619.2023.00194

    doi:10.1109/icse48619.2023.00194 2023

  28. [28]

    Lara Khatib, Noble Saji Mathews, and Meiyappan Nagappan. 2025. AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests. arXiv:2507.17542 [cs.SE] https://arxiv.org/abs/2507.17542 , Vol. 1, No. 1, Article 1. Publication date: January 2026. 1:22 Quanjun Zhang, Yi Zheng, Ye Shang, Weifeng Sun, Haichuan Hu, Chunrong Fang, Zhenyu Chen, and Liang Xiao

    arXiv 2025

  29. [29]

    Bohdan Korel. 1990. Automated Software Test Data Generation.IEEE Transactions on Software Engineering16, 8 (1990), 870–879. doi:10.1109/32.57624

    work page doi:10.1109/32.57624 1990

  30. [30]

    LangChain. 2026. LangChain: Building Applications with LLMs. https://www.langchain.com/ Accessed: 2026-03-27

    2026

  31. [31]

    LangChain. 2026. LangGraph: Agent Orchestration Framework for Reliable AI Agents. https://www.langchain.com/ langgraph Accessed: 2026-03-27

    2026

  32. [32]

    Bertrand Meyer, Ilinca Ciupa, Andreas Leitner, Lisa Ling Liu, and Arno Fiva. 2011. Automatic Testing of Object-Oriented Software.Proceedings of the 33rd International Conference on Software Engineering(2011), 114–124. doi:10.1145/1985793. 1985812

    work page doi:10.1145/1985793 2011

  33. [33]

    Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. 2024. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=9Y8zUO11EQ

    2024

  34. [34]

    Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1026–1038

    2025

  35. [35]

    Noor Nashid, Islem Bouzenia, Michael Pradel, and Ali Mesbah. 2025. Issue2Test: Generating Reproducing Test Cases from Issue Reports. arXiv:2503.16320 [cs.SE] https://arxiv.org/abs/2503.16320

    arXiv 2025

  36. [36]

    Neo4j, Inc. 2026. Neo4j: The Graph Intelligence Platform. https://neo4j.com/ Accessed: 2026-03-27

    2026

  37. [37]

    OpenAI. 2025. GPT-5 is here. https://openai.com/gpt-5/. Accessed: 2026-06-24

    2025

  38. [38]

    Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. InCompanion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion(Montreal, Quebec, Canada)(OOPSLA ’07). Association for Computing Machinery, New York, NY, USA, 815–816. doi:10.1145/ 1297846.1297902

    arXiv 2007

  39. [39]

    Annibale Panichella, Fitsum Kifetew, and Paolo Tonella. 2015. Reformulating Branch Coverage as a Many-Objective Optimization Problem. InProceedings of the 8th IEEE International Conference on Software Testing, Verification and Validation. 1–10. doi:10.1109/ICST.2015.7102604

    doi:10.1109/icst.2015.7102604 2015

  40. [40]

    Annibale Panichella, Fitsum Kifetew, and Paolo Tonella. 2018. Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets.IEEE Transactions on Software Engineering44, 2 (2018), 122–158. doi:10.1109/TSE.2017.2663435

    doi:10.1109/tse.2017.2663435 2018

  41. [41]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting Using LLM. Proceedings of the ACM on Software Engineering1, FSE (2024), 951–971

    2024

  42. [42]

    SWT-bench Developers. 2026. SWT-bench Recommended Configuration for OpenHands. https://github.com/logic- star-ai/swt-bench/issues/38. Accessed: 2026-06-24

    2026

  43. [43]

    GLM Team. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. arXiv:2508.06471 [cs.CL] https://arxiv.org/abs/2508.06471

    Pith/arXiv arXiv 2025

  44. [44]

    Tree-sitter. 2026. Tree-sitter: A Parser Generator and Incremental Parsing Library. https://tree-sitter.github.io/tree- sitter/ Accessed: 2026-03-27

    2026

  45. [45]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit Test Case Generation with Transformers and Focal Context. arXiv:2009.05617 [cs.SE]

    Pith/arXiv arXiv 2020

  46. [46]

    Junyi Wang, Jialun Cao, and Zhongxin Liu. 2026. iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation.Proceedings of the ACM on Software Engineering3, FSE (2026), 4231–4252

    2026

  47. [47]

    Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. 2025. Aegis: An Agent-Based Framework for Bug Reproduction from Issue Descriptions. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 331–342

    2025

  48. [48]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for A...

    Pith/arXiv arXiv 2025

  49. [49]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. Hits: High-Coverage LLM-Based Unit Test Generation Via Method Slicing. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1258–1268

    2024

  50. [50]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents.Proc. ACM Softw. Eng.2, FSE, Article FSE037 (June 2025), 24 pages. doi:10.1145/3715754

    doi:10.1145/3715754 2025

  51. [51]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer Interfaces Enable Automated Software Engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  52. [52]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations (ICLR). , Vol. 1, No. 1, Article 1. Publication date: January 2026. ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproductio...

    2023

  53. [53]

    Xin Yin, Chao Ni, Xinrui Li, Liushan Chen, Guojun Ma, and Xiaohu Yang. 2025. Enhancing LLM’s Ability to Generate More Repository-Aware Unit Tests Through Precise Context Injection. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 623–635

    2025

  54. [54]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation.Proc. ACM Softw. Eng.1, FSE, Article 76 (July 2024), 24 pages. doi:10.1145/3660783

    doi:10.1145/3660783 2024

  55. [55]

    Quanjun Zhang, Chunrong Fang, Siqi Gu, Ye Shang, Zhenyu Chen, and Liang Xiao. 2025. Large language models for unit testing: A systematic literature review.arXiv preprint arXiv:2506.15227(2025)

    Pith/arXiv arXiv 2025

  56. [56]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yuxiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2026. A Systematic Literature Review on Large Language Models for Automated Program Repair.ACM Transactions on Software Engineering and Methodology(2026). doi:10.1145/3799693 Just Accepted

    doi:10.1145/3799693 2026

  57. [57]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Shengcheng Yu, Weisong Sun, Yun Yang, and Zhenyu Chen

  58. [58]

    A Survey on Large Language Models for Software Engineering.Science China Information Sciences69, 4 (2026), 141102

    2026

  59. [59]

    Quanjun Zhang, Chunrong Fang, Tongke Zhang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. Gamma: Revisiting Template-based Automated Program Repair via Mask Prediction. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. 535–547

    2023

  60. [60]

    Quanjun Zhang, Chengyu Gao, Yu Han, Ye Shang, Chunrong Fang, Zhenyu Chen, and Liang Xiao. 2026. SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair.ACM Transactions on Software Engineering and Methodology(2026). doi:10.1145/3818617 Just Accepted

    work page doi:10.1145/3818617 2026

  61. [61]

    Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. 2027. TestBench: Evaluating Class-level Test Case Generation Capability of Large Language Models.Frontiers of Computer Science21 (2027), 2106202–. doi:10.1007/s11704-025-50078-9

    work page doi:10.1007/s11704-025-50078-9 2027

  62. [62]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (Vienna, Austria)(ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 1592–1604. doi:10.1145/ 3650212.3680384 , Vol. 1, No. 1, A...

    arXiv 2024