Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Historical real-bug mechanisms, injected as synthetic bugs, guide LLMs to unit tests that detect real defects better than coverage or mutation feedback.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:40 UTC pith:3JOCV7NB

load-bearing objection Solid systems paper with a real empirical lift on real-bug detection; the mechanism-specificity claim is under-isolated but the result is still worth engaging. the 3 major comments →

arxiv 2607.11573 v1 pith:3JOCV7NB submitted 2026-07-13 cs.SE

Knowledge-Guided Synthetic Bug Feedback for LLM-Based Unit Test Generation

Ziheng Wang , Maike Li , Chen Zhi This is my paper
classification cs.SE
keywords LLM-based unit test generationsynthetic bug injectionreal-bug detectionrepository knowledgemutation testingcoverage-guided testingbug mechanisms
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Executable unit tests often pass without revealing real defects, because coverage gaps and generic mutants are only weak proxies for the conditions that actually break production code. This paper argues that past real bugs encode reusable mechanisms—what semantic boundary fails, where it applies, and how the wrong behavior shows up—and that those mechanisms can be abstracted, retrieved for a new method, and injected as local synthetic bugs. The synthetic bugs then become iterative feedback targets: tests must pass on the correct method and expose the injected difference. On 172 method-level real-bug tasks, that loop raises real-bug detection well above execution-, coverage-, mutation-, knowledge-, and search-based baselines. A sympathetic reader cares because the work repositions historical bug records as live guidance for what inputs and oracles to write, not merely as benchmarks or repair templates.

Core claim

The paper claims that representing real-bug records as transferable structural and semantic mechanisms, retrieving those that fit a focal method, and instantiating them as accepted synthetic bugs supplies LLM test generation with feedback closer to real-bug behavior than coverage or generic mutation. On Defects4J method-level tasks, the resulting framework (KITE) reaches a 72.67% real-bug detection rate—about 20 points above the strongest mutation-guided baseline—and ablating the synthetic-bug feedback itself accounts for most of that gain.

What carries the argument

Knowledge-guided synthetic-bug feedback (KITE): dual structural-semantic recall of historical bug mechanisms, automatic injection into the focal method as compilable, non-trivial synthetic bugs, and promotion-gated iterative enhancement so tests kill those synthetic bugs without losing correct-method semantics.

Load-bearing premise

Bug mechanisms learned from other projects stay transferable when structure and meaning match, and killing the injected synthetic bugs is a faithful stand-in for detecting the held-out real bug.

What would settle it

On the same method-level tasks, replace mechanism-guided synthetic bugs with equally many random or generic local mutants under identical retrieval and iteration budgets; if real-bug detection no longer rises above strong mutation-guided baselines, the claim that historical mechanisms are the active ingredient fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents KITE, an LLM-based unit-test generation framework that abstracts historical real-bug records into transferable bug mechanisms (semantic what/where/how summaries plus structural unit-structure fingerprints), retrieves mechanisms for a focal method via dual structural-semantic recall under repository context, injects them as compilable synthetic bugs, and uses those injected methods as iterative feedback targets. On 172 single-method Defects4J tasks (Cli, Compress, Csv, JacksonDatabind, Jsoup), with a knowledge base built from external corpora that exclude the five evaluation projects, KITE reports 72.67% real-bug detection rate (RBDR), outperforming MutGen (+20.34 pp), CoverUp (+24.41 pp), and other execution-, coverage-, mutation-, knowledge-, and search-based baselines. Ablations attribute the largest drop (−19.76 pp) to removing synthetic-bug feedback; process metrics show high injection acceptance (EMR 98.04%, FAR 97.67%) and 19 KITE-only detections.

Significance. If the central attribution holds, the work offers a concrete alternative to coverage and generic mutation as feedback for LLM test generation: real-bug mechanisms as retrievable, executable targets. The evaluation design is careful—shared model backend and temperature, unified evaluator, project-level knowledge-base hold-out, multi-baseline comparison, ablations, and unique-detection analysis—and the process metrics (accepted injected methods, synthetic kill rate, mechanism diversity) make the pipeline inspectable. The Csv:14 case study usefully illustrates structural-only transfer. These strengths make the paper a substantial contribution to LLM-based testing if the link from historical mechanisms to held-out real-bug detection is tightened.

major comments (3)
  1. [§III-C RQ2 Ablation; Table IV] Central attribution claim (Abstract; §I; Finding 1–2; §VI): the paper attributes the +20.34 pp RBDR lift and the −19.76 pp w/o-SBF drop specifically to transferable historical real-bug mechanisms. The w/o-SBF ablation removes all synthetic-bug feedback, not mechanism-guided feedback versus equally non-trivial mechanism-agnostic local edits. Without a control that injects non-historical but compilable, behaviorally non-trivial edits (or random local rewrites under the same validity gates), the large ablation drop is also consistent with generic hardening from any strong executable feedback target. This is load-bearing for the claim that gains come from real-bug mechanisms rather than from iterative feedback volume.
  2. [§III-D RQ3; §III-E RQ4; Fig. 6–7] Transferability / fidelity of synthetic bugs is assumed rather than measured (Approach §§II-A–C; §III-A knowledge-base construction; RQ3–RQ4). The manuscript never reports how often an accepted injected method shares the same violated semantic boundary, trigger condition, or observable difference as the held-out real bug on that task. RQ3 reports EMR/SKR/FAR and top mechanism categories, and RQ4 counts 19 KITE-only detections (16 after enhancement), but neither breaks these down by mechanism–real-bug alignment. A quantitative alignment analysis on the accepted injections (or at least on the 19 unique detections and the 16 enhancement-only cases) is needed to support the claim that synthetic bugs are faithful proxies rather than merely useful distractors.
  3. [§III-A Dataset; Table I; §V] Benchmark construction retains only single-focal-method patches (Table I: 172/305 = 56.39%), explicitly filtering multi-file and unresolved patches (§III-A; Threats §V). RBDR is therefore measured only on bugs already localized to one method. The paper acknowledges this scope limit, but the primary claim is framed as improving real-bug detection on Defects4J method-level tasks without quantifying how much the filter selects for mechanism-transferable, guard-sensitive faults (the class that dominates the KITE-only cases in §III-E). Either expand the discussion of selection bias with a characterization of excluded vs retained bugs, or qualify the external claim more tightly to single-method regression bugs.
minor comments (5)
  1. [§II-C Dual Structural-Semantic Recall] Free parameters for dual recall (coarse-key filtering, multiset-Jaccard ranking, fusion/top-k weights, injection and enhancement round budgets) are only partially specified (§II-C; §III-A Settings). A short appendix listing defaults and any sensitivity checks would aid reproducibility.
  2. [Table II; Table IV] Table II vs Table IV: Full-row proxy metrics differ because of different averaging subsets (cross-baseline common-success vs ablation-internal). The text notes this, but a footnote on both tables would prevent misreading Full as a duplicate of Table II.
  3. [Fig. 6; §III-E] Fig. 6 overlap diagram is dense; the 19 KITE-only count is clear in text but hard to verify visually. A small table of unique-detection counts per method would help.
  4. [Table II; §V] Token cost (AvTok 0.177M) is highest among baselines; §V mentions prompt/KV caching as future work. A brief breakdown of tokens by stage (context retrieval vs injection planning vs enhancement) would clarify where cost concentrates.
  5. [Table II; Table V] Minor wording: “AvTok (M)” and “A vTok” spacing inconsistencies appear in Table II caption/body; “JacksonDB” vs “JacksonDatabind” naming varies across tables.

Circularity Check

0 steps flagged

No circularity: held-out real-bug evaluation and external knowledge base keep the RBDR claim independent of its construction inputs.

full rationale

KITE's central claim is empirical: mechanism-guided synthetic-bug feedback raises real-bug detection rate (RBDR) on 172 Defects4J method-level tasks relative to execution-, coverage-, mutation-, knowledge-, and search-based baselines, with a large ablation drop when synthetic-bug feedback is removed. The offline knowledge base is built from external corpora (Defects4J/Bears/Bugs.jar/GitBug-Java/GHRB) with the five benchmark projects excluded, so synthetic bugs are not constructed from the defects being measured. RBDR is scored only on held-out real buggy versions that the generator never sees during generation or enhancement. Ablations, synthetic-kill rates, and unique-detection counts are independent empirical checks, not definitional identities. There is no self-definitional loop (X defined as Y then predicted as Y), no fitted parameter renamed as a prediction, no load-bearing uniqueness theorem imported from the authors, and no ansatz smuggled in via self-citation. Transferability of recalled mechanisms is an assumption about external validity, not a circular reduction of the reported result to its inputs. Score 0 is therefore the correct circularity finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on a small set of modeling and experimental choices rather than free physical constants. The main free parameters are engineering knobs (iteration budget, temperature, ranking fusion). The key domain assumptions are that real-bug mechanisms are transferable across projects via structural+semantic similarity and that synthetic injected methods are valid proxies for real-bug detection. Invented entities are the paper’s representational machinery (unit structures, reviewed bug mechanisms, injection plans).

free parameters (4)
  • test generation/enhancement iteration budget = 5
    Hard-capped at five rounds; directly affects how much synthetic-bug feedback can improve the suite.
  • LLM temperature = 0.1
    Fixed at 0.1 for all LLM methods; influences diversity of generated tests and injection plans.
  • mechanism ranking / top-k fusion weights
    Dual structural-semantic recall is fused and top-ranked mechanisms enter injection; exact scoring weights and cutoffs are not fully specified and act as free design choices.
  • structural match thresholds (coarse key + multiset-Jaccard)
    Candidate filtering and ranking of unit structures rely on similarity cutoffs that are not numerically reported.
axioms (4)
  • domain assumption Bug mechanisms abstracted from historical correct/buggy pairs remain transferable to new focal methods when semantic profiles and unit-structure fingerprints match.
    Stated as the core design premise in §II and used throughout retrieval and injection; without it the synthetic bugs would not be expected to improve real-bug detection.
  • domain assumption An injected method that is local, compilable, repository-compatible, and behaviorally non-trivial is a useful executable proxy for the corresponding real-bug mechanism.
    Validity gates in §II-C-4 and the promotion gate in §II-D rest on this; the primary RBDR claim treats synthetic kill rate as evidence of useful feedback.
  • domain assumption Single-method-localizable Defects4J patches are a valid evaluation setting for method-level unit-test real-bug detection.
    Benchmark construction (§III-A, Table I) filters to 172 such tasks; multi-method and cross-file bugs are excluded by design.
  • standard math Standard AST-based structural matching and embedding-based semantic similarity are adequate retrieval signals for mechanism applicability.
    Uses conventional multiset-Jaccard, fingerprint hashing, and embedding alignment; no new mathematical theory is claimed.
invented entities (3)
  • Unit structure (anchor-relation tree + coarse key + fingerprint) no independent evidence
    purpose: Compact structural index that decides whether a historical bug site can be instantiated in a new focal method.
    Defined in §II-A-2 and Fig. 4–5; central to structural recall. Independent evidence is only the paper’s own EMR/FAR numbers, not an external standard.
  • Reviewed bug mechanism (what/where/how summaries + supporting units) no independent evidence
    purpose: Transferable semantic abstraction of a real-bug record used for retrieval and injection planning.
    Constructed offline from 2,069 records into 1,325 mechanisms; the paper’s primary knowledge object. No external validation corpus is provided.
  • Synthetic-bug feedback target (accepted injected method) no independent evidence
    purpose: Executable proxy that the tester agent must distinguish from the correct method.
    Produced by the injection executor and validity gates; the ablation shows it carries most of the RBDR gain.

pith-pipeline@v1.1.0-grok45 · 22288 in / 3639 out tokens · 38293 ms · 2026-07-14T04:40:27.452583+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have opened new opportunities for unit test generation, but executable tests do not necessarily reveal real defects. This paper studies how historical real-bug mechanisms can be transformed into executable feedback targets for LLM-based unit test generation. The proposed framework constructs structural and semantic representations of real-bug records, retrieves mechanisms applicable to a focal method, and instantiates them as synthetic bugs that guide iterative test enhancement. We evaluate the approach on method-level real-bug detection tasks from Defects4J and show that mechanism-guided synthetic-bug feedback improves real-bug detection over execution-, coverage-, mutation-, knowledge-, and search-based baselines. The results suggest that organizing real-bug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles.

Figures

Figures reproduced from arXiv: 2607.11573 by Chen Zhi, Maike Li, Ziheng Wang.

Figure 2
Figure 2. Figure 2: Mutation feedback misses the real-bug behavior. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of KITE. are often tied to specific projects and APIs and cannot be directly transferred; they therefore need to be abstracted into mechanism-level representations that capture applicability conditions, buggy behavior, and injection intent. Second, how should bug mechanisms be generalized to new methods? Whether a mechanism is applicable to a focal method depends not only on whether their behavior… view at source ↗
Figure 4
Figure 4. Figure 4: Extracting a unit structure from a local code unit. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Core grammar of KITE’s unit structure. NodeLabel denotes a normalized AST node kind; AnchorRole distinguishes control-, statement-, and expression-centered anchors. Relation types denote condition, expression-in-statement, guard-body, adjacency, and same-block links. The fingerprint is the first 16 hexadecimal characters of SHA-1 over the canonicalized tree. C. Knowledge-Guided Synthetic Bug Injection Know… view at source ↗
Figure 7
Figure 7. Figure 7: Csv:14 structural-only recall case. the illustrated case. This case demonstrates a structural-only path: KITE retrieves a cross-domain serialization mechanism through structural recall and converts it into a synthetic bug around the safe-character boundary. In this case, KITE selects the historical mechanism Context￾Mismatched URL Ampersand Escaping as the top-ranked recalled mechanism. Its semantic score … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 10 linked inside Pith

  1. [1]

    Randoop: Feedback-directed random testing for Java,

    C. Pacheco and M. D. Ernst, “Randoop: Feedback-directed random testing for Java,” inCompanion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion. ACM, 2007, pp. 815–816

    2007

  2. [2]

    Search-based software test data generation: A survey,

    P. McMinn, “Search-based software test data generation: A survey,” Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105– 156, 2004

    2004

  3. [3]

    EvoSuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” inProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, 2011, pp. 416–419

    2011

  4. [4]

    Search based software engineering: Techniques, taxonomy, tutorial,

    M. Harman, P. McMinn, J. T. d. Souza, and S. Yoo, “Search based software engineering: Techniques, taxonomy, tutorial,” inEmpirical Software Engineering and Verification. Springer, 2012, pp. 1–59

    2012

  5. [5]

    DART: Directed automated random testing,

    P. Godefroid, N. Klarlund, and K. Sen, “DART: Directed automated random testing,” inProceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation. ACM, 2005, pp. 213–223

    2005

  6. [6]

    A survey of symbolic execution techniques,

    R. Baldoni, E. Coppa, D. C. D’Elia, C. Demetrescu, and I. Finocchi, “A survey of symbolic execution techniques,”ACM Computing Surveys, vol. 51, no. 3, pp. 1–39, 2018

    2018

  7. [7]

    CUTE: A concolic unit testing engine for C,

    K. Sen, D. Marinov, and G. Agha, “CUTE: A concolic unit testing engine for C,” inProceedings of the 10th European Software Engineer- ing Conference Held Jointly with the 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 2005, pp. 263–272

    2005

  8. [8]

    KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs,

    C. Cadar, D. Dunbar, and D. Engler, “KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs,” in Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 2008, pp. 209–224

    2008

  9. [9]

    Evaluating large language models trained on code,

    M. Chenet al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2024

    2024

  11. [11]

    ChatUniTest: A framework for LLM-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “ChatUniTest: A framework for LLM-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. ACM, 2024, pp. 572–576

    2024

  12. [12]

    RUG: Turbo LLM for Rust unit test generation,

    X. Cheng, F. Sang, Y . Zhai, X. Zhang, and T. Kim, “RUG: Turbo LLM for Rust unit test generation,” inProceedings of the 47th IEEE/ACM International Conference on Software Engineering. IEEE, 2025, pp. 2983–2995

    2025

  13. [13]

    Knowledge matters: Injecting project and testing knowledge into LLM-based unit test generation,

    A. Li, M. Liu, Z. Chen, Z. Pei, Z. Li, D. Dai, Y . Wang, and Z. Zheng, “Knowledge matters: Injecting project and testing knowledge into LLM-based unit test generation,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering. ACM,

  14. [14]

    Available: https://arxiv.org/abs/2511.14224

    [Online]. Available: https://arxiv.org/abs/2511.14224

    Pith/arXiv arXiv

  15. [15]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,”IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015

    2015

  16. [16]

    Evaluating and improving ChatGPT for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving ChatGPT for unit test generation,”Proceed- ings of the ACM on Software Engineering, vol. 1, no. FSE, 2024

    2024

  17. [17]

    Y ATE: The role of test repair in LLM-based unit test generation,

    M. Konstantinou, R. Degiovanni, J. M. Zhang, M. Harman, and M. Papadakis, “Y ATE: The role of test repair in LLM-based unit test generation,”arXiv preprint arXiv:2507.18316, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    CodaMOSA: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMOSA: Escaping coverage plateaus in test generation with pre-trained large language models,” inProceedings of the 45th IEEE/ACM International Conference on Software Engineering. IEEE, 2023, pp. 919–931

    2023

  19. [19]

    CoverUp: Effective high coverage test generation for Python,

    J. A. Pizzorno and E. D. Berger, “CoverUp: Effective high coverage test generation for Python,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, 2025

    2025

  20. [20]

    An analysis and survey of the development of mutation testing,

    Y . Jia and M. Harman, “An analysis and survey of the development of mutation testing,”IEEE Transactions on Software Engineering, vol. 37, no. 5, pp. 649–678, 2011

    2011

  21. [21]

    Mutation testing advances: An analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y . Jia, Y . Le Traon, and M. Harman, “Mutation testing advances: An analysis and survey,”Advances in Computers, vol. 112, pp. 275–378, 2019

    2019

  22. [22]

    Effective test generation using pre-trained large language models and mutation testing,

    A. Moradi Dakhel, A. Nikanjam, V . Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,”Information and Software Technology, vol. 171, p. 107468, 2024

    2024

  23. [23]

    Mutation-guided unit test generation with a large language model,

    G. Wang, Q. Xu, L. Briand, and K. Liu, “Mutation-guided unit test generation with a large language model,”IEEE Transactions on Software Engineering, vol. 52, no. 5, pp. 1657–1682, 2026

    2026

  24. [24]

    Defects4J: A database of existing faults to enable controlled testing studies for Java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for Java programs,” inPro- ceedings of the 2014 International Symposium on Software Testing and Analysis. ACM, 2014, pp. 437–440

    2014

  25. [25]

    Bugs.jar: A large- scale, diverse dataset of real-world Java bugs,

    R. K. Saha, Y . Lyu, H. Yoshida, and M. R. Prasad, “Bugs.jar: A large- scale, diverse dataset of real-world Java bugs,” inProceedings of the 15th International Conference on Mining Software Repositories. ACM, 2018, pp. 10–13

    2018

  26. [26]

    BEARS: An extensible Java bug benchmark for automatic program repair studies,

    F. Madeiral, S. Urli, M. Maia, and M. Monperrus, “BEARS: An extensible Java bug benchmark for automatic program repair studies,” inProceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, 2019, pp. 468–478

    2019

  27. [27]

    GitBug-Java: A reproducible benchmark of recent Java bugs,

    A. Silva, N. Saavedra, and M. Monperrus, “GitBug-Java: A reproducible benchmark of recent Java bugs,” inProceedings of the 21st International Conference on Mining Software Repositories, 2024

    2024

  28. [28]

    The GitHub recent bugs dataset for evaluating LLM-based debugging applications,

    J. Y . Lee, S. Kang, J. Yoon, and S. Yoo, “The GitHub recent bugs dataset for evaluating LLM-based debugging applications,”arXiv preprint arXiv:2310.13229, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Code-A1: Adversarial evolving of code LLM and test LLM via reinforcement learning,

    A. Wang, Y . Yan, N. Zhou, Z. Lu, W. Lu, J. Xiao, Y . Zhuang, and Y . Shen, “Code-A1: Adversarial evolving of code LLM and test LLM via reinforcement learning,”arXiv preprint arXiv:2603.15611, 2026

    arXiv 2026

  30. [30]

    Toward training superintel- ligent software agents through self-play SWE-RL,

    Y . Wei, Z. Sun, E. McMilin, J. Gehring, D. Zhang, G. Synnaeve, D. Fried, L. Zhang, and S. Wang, “Toward training superintel- ligent software agents through self-play SWE-RL,”arXiv preprint arXiv:2512.18552, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    A VIATOR: Towards AI-agentic vulnerability injection workflow for high-fidelity, large-scale code security dataset,

    A. Lbath, M.-R. Amini, A. Delaitre, and V . Okun, “A VIATOR: Towards AI-agentic vulnerability injection workflow for high-fidelity, large-scale code security dataset,”arXiv preprint arXiv:2508.20866, 2025, version 6 revised in 2026

    arXiv 2025

  32. [32]

    Prometheus: Towards long- horizon codebase navigation for repository-level problem solving,

    Y . Pan, Z. Chen, S. Lu, Z. Chu, X. Li, H. Li, Y . Feng, C. Le Goues, F. Sarro, M. Monperrus, and H. Ye, “Prometheus: Towards long- horizon codebase navigation for repository-level problem solving,”arXiv preprint arXiv:2507.19942, 2025, version 2 revised in 2026

    arXiv 2025

  33. [33]

    MuJava: An automated class mutation system,

    Y .-S. Ma, J. Offutt, and Y . R. Kwon, “MuJava: An automated class mutation system,”Software Testing, Verification and Reliability, vol. 15, no. 2, pp. 97–133, 2005

    2005

  34. [34]

    FixMiner: Mining relevant fix patterns for automated program repair,

    A. Koyuncu, K. Liu, T. F. Bissyand ´e, D. Kim, J. Klein, M. Monperrus, and Y . Le Traon, “FixMiner: Mining relevant fix patterns for automated program repair,”Empirical Software Engineering, vol. 25, no. 3, pp. 1980–2024, 2020

    1980

  35. [35]

    RepoCoder: Repository-level code com- pletion through iterative retrieval and generation,

    F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J.-G. Lou, and W. Chen, “RepoCoder: Repository-level code com- pletion through iterative retrieval and generation,”arXiv preprint arXiv:2303.12570, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Whole test suite generation,

    G. Fraser and A. Arcuri, “Whole test suite generation,”IEEE Transac- tions on Software Engineering, vol. 39, no. 2, pp. 276–291, 2013

    2013

  37. [37]

    Coverage is not strongly correlated with test suite effectiveness,

    L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” inProceedings of the 36th International Conference on Software Engineering. ACM, 2014, pp. 435–445

    2014

  38. [38]

    Is mutation an appropriate tool for testing experiments?

    J. H. Andrews, L. C. Briand, and Y . Labiche, “Is mutation an appropriate tool for testing experiments?” inProceedings of the 27th International Conference on Software Engineering. ACM, 2005, pp. 402–411

    2005

  39. [39]

    Are mutants a valid substitute for real faults in software testing?

    R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, “Are mutants a valid substitute for real faults in software testing?” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 2014, pp. 654–665

    2014

  40. [40]

    JaCoCo Java code coverage library,

    JaCoCo Team, “JaCoCo Java code coverage library,” https://www. jacoco.org/jacoco/, 2026, accessed: 2026-06-12

    2026

  41. [41]

    MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler,

    R. Just, F. Schweiggert, and G. M. Kapfhammer, “MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler,” inPro- ceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering. IEEE, 2011, pp. 612–615

    2011

  42. [42]

    On the evaluation of large language models in unit test generation,

    L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang, and J. Chen, “On the evaluation of large language models in unit test generation,”arXiv preprint arXiv:2406.18181, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    TestSpark: IntelliJ IDEA’s ultimate test generation com- panion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V . Kovalenko, and P. De- rakhshanfar, “TestSpark: IntelliJ IDEA’s ultimate test generation com- panion,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering: Companion Proceedings. ACM, 2024, pp. 30–34

    2024

  44. [44]

    HITS: High-coverage LLM- based unit test generation via method slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: High-coverage LLM- based unit test generation via method slicing,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. ACM, 2024, pp. 1258–1269

    2024

  45. [45]

    Advancing code coverage: Incorporating program analysis with large language models,

    C. Yang, J. Chen, B. Lin, Z. Wang, and J. Zhou, “Advancing code coverage: Incorporating program analysis with large language models,” ACM Transactions on Software Engineering and Methodology, 2025

    2025

  46. [46]

    LLM test generation via iterative hybrid program analysis,

    S. Gu, N. Nashid, and A. Mesbah, “LLM test generation via iterative hybrid program analysis,”arXiv preprint arXiv:2503.13580, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Enhancing LLM’s ability to generate more repository-aware unit tests through precise con- textual information injection,

    X. Yin, C. Ni, X. Li, L. Chen, G. Ma, and X. Yang, “Enhancing LLM’s ability to generate more repository-aware unit tests through precise con- textual information injection,”arXiv preprint arXiv:2501.07425, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    CITYW ALK: Enhancing LLM-based C++ unit test generation via project-dependency awareness and language-specific knowledge,

    Y . Zhang, Q. Lu, K. Liu, W. Dou, J. Zhu, L. Qian, C. Zhang, Z. Lin, and J. Wei, “CITYW ALK: Enhancing LLM-based C++ unit test generation via project-dependency awareness and language-specific knowledge,” ACM Transactions on Software Engineering and Methodology, 2026

    2026

  49. [49]

    Type-aware LLM- based regression test generation for Python programs,

    R. Liu, Z. Zhang, Y . Hu, Y . Lin, X. Gao, and H. Sun, “Type-aware LLM- based regression test generation for Python programs,”arXiv preprint arXiv:2503.14000, 2025

    arXiv 2025

  50. [50]

    ASTER: Natural and multi-language unit test generation with LLMs,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “ASTER: Natural and multi-language unit test generation with LLMs,” inProceedings of the 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice. IEEE, 2025, pp. 413–424

    2025

  51. [51]

    Test intention guided LLM-based unit test generation,

    Z. Nan, Z. Guo, K. Liu, and X. Xia, “Test intention guided LLM-based unit test generation,” inProceedings of the 47th IEEE/ACM International Conference on Software Engineering. IEEE, 2025, pp. 1026–1038

    2025

  52. [52]

    Generating project-specific test cases with requirement validation intention,

    B. Qi, Y . Lin, X. Weng, Y . Huang, C. Liu, H. Sun, Z. Jin, and J. S. Dong, “Generating project-specific test cases with requirement validation intention,”arXiv preprint arXiv:2507.20619, 2025, accepted at ISSTA 2026

    Pith/arXiv arXiv 2025

  53. [53]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, 2024

    2024

  54. [54]

    Fuzz4All: Universal fuzzing with large language models,

    C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Fuzz4All: Universal fuzzing with large language models,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering. ACM, 2024, pp. 1–13

    2024

  55. [55]

    Large language model guided protocol fuzzing,

    R. Meng, M. Mirchev, M. B ¨ohme, and A. Roychoudhury, “Large language model guided protocol fuzzing,” inProceedings of the Network and Distributed System Security Symposium. Internet Society, 2024

    2024

  56. [56]

    Fine-grained and accurate source code differencing,

    J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, “Fine-grained and accurate source code differencing,” inProceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering. ACM, 2014, pp. 313–324

    2014

  57. [57]

    How often do single-statement bugs occur? the ManySStuBs4J dataset,

    R.-M. Karampatsis and C. Sutton, “How often do single-statement bugs occur? the ManySStuBs4J dataset,” inProceedings of the 17th International Conference on Mining Software Repositories. ACM, 2020, pp. 573–577

    2020

  58. [58]

    Automatic patch generation learned from human-written patches,

    D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches,” inProceedings of the 35th Inter- national Conference on Software Engineering. IEEE, 2013, pp. 802– 811

    2013

  59. [59]

    TBar: Revisiting template-based automated program repair,

    K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyand ´e, “TBar: Revisiting template-based automated program repair,” inProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 2019, pp. 31–42

    2019