Pith. sign in

REVIEW 4 major objections 6 minor 58 references

The paper argues that LLM-generated unit tests fail in real projects mainly because the model lacks project context, and shows that supplying explicit dependencies, a deterministic test skeleton, and static-analysis repair makes such tests

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:00 UTC pith:D6LHLKME

load-bearing objection CATGen is a solid experience paper with a broad, informative evaluation; the abstract's numbers don't match the tables, and the industrial benchmark's representativeness is the main open question. the 4 major comments →

arxiv 2607.19682 v1 pith:D6LHLKME submitted 2026-07-22 cs.SE

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation

Junjie Chen , Ziqi Wang , Lin Yang , Chen Yang , Xiao Chu , Jianyi Zhou , Guangtai Liang , Qianxiang Wang
show 1 more author
Dong Wang
This is my paper
classification cs.SE
keywords unit test generationlarge language modelscompilation reliabilitycontext retrievaltest skeletonstatic analysisindustrial software testingmutation testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the reason LLM-generated unit tests fail in real projects is not the model's logic but missing project context: unresolved imports, fragile test scaffolding, and expensive LLM repair loops. To make tests practically usable, it proposes CATGen, a pipeline that retrieves explicit project-level dependencies, deterministically constructs a test-class skeleton, and repairs compilation errors with lightweight static analysis instead of more LLM calls. On a curated industrial benchmark of 183 dependency-heavy focal methods, CATGen reaches 91.83% compilation success and 70.10% line coverage, beating six baselines while using 51–69% less time and 67–84% fewer tokens. On a widely used open-source defect benchmark, the same design generalizes, achieving the best LLM-based compilation, coverage, pass rate, and mutation score. If right, the lesson is that reliability comes from engineering support around the model, not prompt engineering alone.

Core claim

The central claim, stated as an experience report, is that practical reliability of LLM-based unit test generation is governed by engineering support around the model rather than by prompt design alone. CATGen operationalizes this in three moves: it retrieves project-level context explicitly, constructs a test-class skeleton deterministically from framework templates, and repairs compilation errors with lightweight static analysis instead of additional LLM calls. On 183 industrial focal methods it reports 91.83% compilation success, 70.10% line coverage, 63.92% branch coverage, and a 54.63% pass rate, outperforming six baselines; on a widely used open-source defect benchmark it keeps the bes

What carries the argument

The load-bearing mechanism is skeleton-conditioned completion plus deterministic post-processing. A context retrieval stage gathers five context types from build files and ASTs; a framework-to-template mapping turns those facts into a test class skeleton that fixes imports, annotations, field wiring, and lifecycle hooks. The LLM is then asked to complete test methods under this fixed scaffold. After merging, a program-analysis stage applies eight repair rules in fixed precedence—package completion, import supplementation, annotation rectification, invalid-reference resolution, private-member adaptation, signature alignment, exception specification, and fallback assertions—followed by a stati

Load-bearing premise

The evaluation and design rest on the premise that the failure taxonomy observed in one set of industrial deployments—and the benchmark deliberately built to stress that taxonomy—represents the conditions under which teams actually need LLM-generated tests.

What would settle it

Sample production focal methods with fewer than three external dependencies and compare CATGen against a plain one-shot LLM prompt with a single LLM-based compile-repair round; if compilation-success and coverage gaps shrink to a few points, the claimed dependence on explicit context, deterministic skeletons, and static repair weakens. A second check: freeze CATGen's eight repair rules and apply them to a held-out proprietary codebase to see whether the repair taxonomy generalizes beyond the failure classes that motivated it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adoption barrier drops: teams can integrate LLM test generation into framework-heavy codebases without a manual compile-fix cycle, since the pipeline's compilation success is high and reproducible.
  • Costs become predictable: single-round generation plus static repair means token and latency budgets can be planned for continuous integration, unlocking batch generation at scale.
  • Static repair is portable: replacing other generators' iterative LLM repair with CATGen's analysis-driven post-processing raises their compilation and coverage too, so the repair stage is a reusable capability.
  • Smaller models become viable: under CATGen, a 7B-parameter model beats much larger baselines, lowering the hardware threshold for deploying LLM test generation.
  • Fault detection improves with executability: the mutation-score gains indicate that compilable tests that exercise boundary and exception paths are more likely to kill mutants than tests that merely reach lines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The static repair component could be extracted as a standalone test-debugging service for any code-generating LLM, independent of skeleton construction; the paper's cross-baseline results suggest the repairs are general, but the paper does not make that product claim.
  • The industrial benchmark was deliberately biased toward dependency-heavy methods (57% with at least three external dependencies), so the headline margins may overstate gains on typical, mostly self-contained production code; a random production sample would be a sharper test of the magnitude.
  • The same design pattern—explicit context retrieval, deterministic scaffolding, and analysis-based repair instead of LLM loops—could transfer to other compilable artifacts, such as build scripts, migration code, or generated API clients, where syntax and dependency correctness are the bottleneck.
  • A concrete extension would let the coverage-enhancement stage learn which uncovered branches are worth additional tests, since the current triggers (null checks, empty strings, exception throws) are domain heuristics rather than a general coverage policy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports the authors' experience designing, deploying, and evaluating CATGen, a context-aware workflow for LLM-based unit test generation in industrial Java projects. CATGen has four stages: structured retrieval of project-level context (imports, framework, mocking, call sites), deterministic construction of a test-class skeleton, skeleton-conditioned LLM completion of test methods, and program-analysis-based post-processing that performs static compilation repair and coverage enhancement. The evaluation covers 183 focal methods from eight proprietary industrial projects and four Defects4J projects, comparing CATGen with EvoSuite, ChatTester, ChatUniTest, HITS, TELPA, and RATester across nine open-weight LLMs. The paper claims that CATGen substantially improves compilation success rate, line/branch coverage, pass rate, and mutation score while reducing generation time and token consumption, and that ablations confirm the contribution of each pipeline component.

Significance. If the empirical claims are correct, the paper makes a useful practical contribution: it demonstrates that systematic engineering around LLMs—explicit context, deterministic skeletons, and static repair—can improve the executability of generated tests and reduce cost compared with LLM-only or LLM-repair-loop approaches. The evaluation is broad for an experience paper: 183 industrial methods, six baselines, nine LLMs, ablations, Wilcoxon tests, a Defects4J replication, mutation-score analysis, and a public replication package. The design lessons are concrete and transferable. However, the headline quantitative ranges in the Introduction do not reproduce from the reported tables, and the industrial benchmark is deliberately biased toward dependency-heavy methods, so the breadth of the 'practical reliability' claim is not fully established. These issues are correctable, and the core engineering insight remains valuable.

major comments (4)
  1. [§1 (Introduction) vs Table 2 and Table 5] The quantitative summary in the Introduction is not reproducible from the reported results. For the industrial benchmark, Table 2 gives CATGen avg CSR 91.83% vs. LLM baselines 51.02–67.11%, so the percentage-point differences are 24.72–40.81, not 24.72–38.05. For CovL, Table 2 differences are 21.67–40.33 points, not 17.27–22.17; for CovB, 17.59–37.18, not 15.31–18.24. For Defects4J, Table 5 gives CSR differences of roughly 9.87–26.21 points against the five LLM baselines, not 10.42–14.33. The authors should either derive and state the exact computation (e.g., relative vs. absolute, per-model vs. averaged) or correct the ranges. Because these ranges are the paper's headline quantitative claim, this must be fixed before publication.
  2. [§4.1, §7 Construct Validity, Table 1] The central 'practical reliability' claim is conditional on the representativeness of the industrial benchmark, and the paper's own Construct Validity section concedes that the benchmark 'stresses deployment failure modes and may favor compilability-focused workflows.' The benchmark is deliberately biased toward methods with complex dependencies (57.38% with >=3 external dependencies, Table 1) and was curated using the same failure taxonomy that motivated CATGen's design (§4.1). The Defects4J experiments are an important independent check, but they cover only four computation-oriented projects, use one model, and do not reproduce the framework-heavy dependency profile. To support the broad claim, the authors should either (a) bound the claim to the deployment contexts represented by the benchmark, (b) add a sensitivity analysis with respect to the dependency threshold, or (c) report resu
  3. [§5 (Defects4J evaluation)] The Defects4J protocol is underspecified, which limits the auditability of the generalizability claim. The paper does not state how many focal methods were selected from Chart, Lang, Time, and Math, how they were selected, whether buggy or fixed versions were used, or how the 'same focal-method evaluation protocol' from the industrial setting was applied to these projects. Table 5 reports only project-level aggregates and Table 6 reports mutation scores without the number of mutants per project. This makes it impossible to reproduce the Defects4J experiments or to assess whether the selected focal methods are representative of each project. Please provide per-method or per-bug counts, selection criteria, and per-project mutant counts.
  4. [§4.1, Baselines] The strongest LLM baselines deserve closer scrutiny. TELPA is from the authors' own group, and RATester was originally implemented for Go and then re-implemented for Java by the authors. Because RATester and TELPA are the two best-performing LLM baselines, the fairness of their Java implementations directly affects the magnitude of CATGen's reported gains. The paper should specify the exact Java adaptation of RATester, state whether the original authors were involved, and provide evidence that the adaptation does not disadvantage the baseline relative to its original design. For TELPA, the authors should clarify how their current implementation relates to the published version and whether any modifications were made for this evaluation.
minor comments (6)
  1. [§4.2, Finding II] The text says DRQ-32B attains the best compilation success rate (96.29%) and pass rate (62.36%), but Table 2 lists CATGen with DRQ-32B as 96.28% CSR and 69.36% PR. Please correct these values.
  2. [§4.2, Finding III] The text states that the 'w/o all' variant drops CSR and CovL by 38.88% and 30.06%, but Table 3 reports -38.88% and -30.63%. Reconcile the text with the table.
  3. [§4.2, Finding I] The paper says 'p-values < 0.005 for CSR, CovL, CovB, and PR in all pairwise comparisons' but reports only one illustrative Wilcoxon result. Please provide a full table of p-values and effect sizes, or a link to the supplementary material containing them.
  4. [§4.1 and Table 4] Section 4.1 states EvoSuite is allocated a search budget of 300s per focal method, but Table 4 reports 10,980s total for 183 methods (exactly 60s per method on average). Clarify whether 300s is a maximum and why the average is 60s, or correct the budget statement.
  5. [Table 1] The definition of 'external dependency' and the threshold of >=3 dependencies should be stated more precisely. A sensitivity analysis (e.g., thresholds of 2 and 4) would help establish that the benchmark's difficulty characterization is not an artifact of this free parameter.
  6. [Data Availability] The replication package is described only at a high level. Please list the concrete artifacts (prompts, scripts, anonymized examples, repair-rule ordering, skeleton templates) and clarify which files are public, since the full benchmark and per-method results cannot be released.

Circularity Check

0 steps flagged

No definitional circularity: CATGen is evaluated on an external benchmark (Defects4J) with ablations; the biased industrial benchmark is an acknowledged validity threat, not a fitted prediction.

full rationale

CATGen's claimed contribution is an empirical engineering result, not a formal derivation, so the usual circularity reductions (Eq. X = Eq. Y by construction; fitted parameter renamed as prediction) do not apply. The closest concern is that the industrial benchmark was 'intentionally bias[ed] selection toward difficult methods' (Section 4.1) and CATGen was designed from 'recurring failure patterns observed in industrial usage' (Section 1). This creates a feedback loop between the failure taxonomy and the benchmark distribution: the headline CSR/coverage gains are measured on a curated set that stresses exactly the dependency-heavy, framework-managed cases CATGen is engineered to handle. However, the paper explicitly acknowledges this in Construct Validity: 'The industrial benchmark stresses deployment failure modes and may favor compilability-focused workflows; we therefore complement it with Defects4J and mutation-score analysis.' That acknowledgment turns the concern into an external-validity limitation rather than a hidden circular step. The Defects4J experiments use an independent, publicly available benchmark that is not derived from the authors' deployment taxonomy, and the ablation study (w/o skeleton, w/o repair, w/o enhancement, w/o all) shows each component contributes measurable, separable effects. Self-citations appear (TELPA [48] is an author-overlapping baseline; [53] motivates compilation-failure frequency), but they are not load-bearing: no uniqueness theorem is imported, no ansatz is smuggled via citation, and the central comparison is run against external baselines with standardized open-weight LLMs. Therefore, under the hard rule that circularity requires exhibiting a specific reduction, no such step is present; the appropriate score reflects only minor self-citation and the acknowledged benchmark-design overlap.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central empirical claim rests on representativeness of the failure-mode analysis and benchmark curation, sufficiency of AST-level context, fidelity of baseline re-implementations, and JaCoCo coverage measurement. No new physical or mathematical entities are postulated; the only hand-chosen numeric is the >=3 dependency complexity threshold. The proprietary benchmark is the main unverifiable load-bearing input.

free parameters (1)
  • complex dependency threshold = >=3 external classes/methods
    Hand-chosen cutoff (Section 4.1) to define 'complex dependencies' in the industrial benchmark; used to describe 57.38% of focal methods and justify benchmark difficulty. Not fitted to data and not independently calibrated.
axioms (5)
  • domain assumption The three industrial failure classes (context mismatch, fragile scaffolding, costly iterative repair) are the dominant practical bottlenecks for LLM-based unit test generation.
    Motivates the entire pipeline (Section 1, bullets I-III). If actual production failures are dominated by other causes, CATGen's design priorities are misplaced.
  • domain assumption AST-level structural signals (IntelliJ PSI in the implementation) are sufficient project context for compilation-sound test generation; any conforming extractor can substitute.
    Sections 3.1, 3.4 and 7 (Construct Validity) claim 'any conforming extractor supplying the same structural signals can substitute.' If framework semantics require runtime or build-state information beyond ASTs, the repair rules may fail on unseen patterns.
  • domain assumption The re-implemented baselines faithfully represent the originally published algorithms, particularly the Java port of RATester and the standardization of all LLM calls to open-weight models.
    Section 4.1 (Compared Techniques): 'we follow the authors' Java adaptation and re-implement RATester...' and standardize all baselines to open-source models, which may not match original proprietary-model results and introduces implementation-fidelity risk.
  • domain assumption Defects4J (Chart, Lang, Time, Math) is representative of open-source Java projects for generalizing the industrial findings.
    Section 5 selects four computation-oriented projects; the paper acknowledges Defects4J is 'particularly favorable to search-based tools,' yet uses it to claim generalizability.
  • domain assumption JaCoCo's instrumentation correctly measures line and branch coverage.
    Section 7 (Internal Validity) acknowledges JaCoCo may miss execution paths; all coverage numbers rest on this tool.

pith-pipeline@v1.3.0-alltime-deepseek · 21781 in / 18568 out tokens · 178400 ms · 2026-08-01T12:00:33.377595+00:00 · methodology

0 comments
read the original abstract

Automated unit test generation has recently benefited from advances in large language models (LLMs), yet our industrial deployments reveal a persistent gap between promising research results and practical usability. In real-world projects with complex frameworks and cross-file dependencies, LLM-generated tests frequently fail to compile, require costly manual repair, or provide unstable coverage improvements. This paper reports our experience in designing, deploying, and evaluating CATGen, a context-aware workflow for LLM-based unit test generation, informed by repeated industrial failures and refinements. Rather than relying on LLMs to infer incomplete project context, we found that compilation robustness critically depends on making project-level dependencies explicit, stabilizing test class scaffolding, and replacing iterative LLM-based repair with lightweight static analysis. These experience-driven insights shaped CATGen's multi-stage design, which combines structured context retrieval, deterministic test skeleton construction, and program analysis-based post-processing. We evaluate CATGen on real-world complex focal methods from proprietary industrial projects and additionally on the Defects4J benchmark to assess generalizability. Across both settings, CATGen substantially improves compilation success and structural coverage while significantly reducing generation time and token consumption compared to existing LLM-based approaches. Our results demonstrate that reliable LLM-based unit test generation in practice depends less on prompt engineering alone and more on systematic engineering support grounded in real-world development constraints.

Figures

Figures reproduced from arXiv: 2607.19682 by Chen Yang, Dong Wang, Guangtai Liang, Jianyi Zhou, Junjie Chen, Lin Yang, Qianxiang Wang, Xiao Chu, Ziqi Wang.

Figure 1
Figure 1. Figure 1: Overview of the proposed CATGen Beyond academic benchmarks, recent industrial deployments include Meta’s TestGen-LLM [10], Google’s BRT-Agent [13], and Mozilla’s BLAST [25]. Those efforts emphasize integrating LLMs into large-scale developer workflows and reporting productivity-oriented outcomes. They seldom foreground systematic analyses of why generated tests fail to compile in framework-heavy code￾bases… view at source ↗
Figure 2
Figure 2. Figure 2: The illustrative focal method, along with its corresponding test class skeleton generated by CATGen [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Test Class Generated by CATGen 3.3 Skeleton-Conditioned Completion for Test Generation A common LLM-based test generation workflow follows a dialog-style paradigm: users provide a natural language query, and the model outputs candidate tests [12, 43, 55]. While straightforward, our experience suggests that this interaction pattern is fragile in practice. In a pilot study, we observed that even when a pre-c… view at source ↗
Figure 4
Figure 4. Figure 4: Unit test generated by baselines functional coverage of the focal method. For the same focal method, EvoSuite generated tests that failed to compile in our setting, which reflects a recurring challenge we encountered in industrial repositories where framework wiring, external types, and required initialization must be correct before any test logic becomes executable. When turning to LLM-based baselines in … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 8 linked inside Pith

  1. [1]

    2026. JUnit 4. https://junit.org/junit4 Accessed: 2026-01

    2026

  2. [2]

    2026. JUnit 5. https://junit.org Accessed: 2026-01

    2026

  3. [3]

    2026. Mockito. https://site.mockito.org Accessed: 2026-01

    2026

  4. [4]

    PowerMock

    2026. PowerMock. https://powermock.github.io Accessed: 2026-01

    2026

  5. [5]

    Spock Framework

    2026. Spock Framework. https://spockframework.org Accessed: 2026-01

    2026

  6. [6]

    Spring Boot Testing

    2026. Spring Boot Testing. https://docs.spring.io/spring-boot/reference/testing Accessed: 2026-01

    2026

  7. [7]

    Meta AI. 2023. CodeLlama-7B-Instruct. https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf

    2023

  8. [8]

    Meta AI. 2025. Meta-Llama-3.1-8B-Instruct. https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct

    2025

  9. [9]

    M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 263–272

    2017

  10. [10]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated unit test improvement using large language models at meta. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 185–196

    2024

  11. [11]

    Arianna Blasi, Alessandra Gorla, Michael D Ernst, and Mauro Pezzè. 2022. Call me maybe: Using nlp to automatically generate unit test cases respecting temporal constraints. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–11

    2022

  12. [12]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572–576

    2024

  13. [13]

    Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. 2025. Agentic Bug Reproduction for Effective Automated Program Repair at Google.arXiv preprint arXiv:2502.01821(2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Shyam R Chidamber and Chris F Kemerer. 1994. A metrics suite for object oriented design.IEEE Transactions on Software Engineering20, 6 (1994), 476–493

    1994

  15. [15]

    Christoph Csallner, Nikolai Tillmann, and Yannis Smaragdakis. 2008. DySy: Dynamic symbolic execution for invariant inference. InProceedings of the 30th international conference on Software engineering. 281–290

    2008

  16. [16]

    deepseek ai. 2023. DeepSeek-Coder-6.7B-Instruct. https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct

    2023

  17. [17]

    deepseek ai. 2024. DeepSeek-Coder-33B-Instruct. https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct

    2024

  18. [18]

    deepseek ai. 2024. DeepSeek-R1-Distill-Llama-8B. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B

    2024

  19. [19]

    deepseek ai. 2024. DeepSeek-R1-Distill-Qwen-32B. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen- 32B

    2024

  20. [20]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. Toga: A neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering. 2130–2141

    2022

  21. [21]

    Brandt, and Andy Zaidman

    Khalid El Haji, Carolin E. Brandt, and Andy Zaidman. 2024. Using GitHub Copilot for Test Generation in Python: An Empirical Study. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE). 45–55

    2024

  22. [22]

    Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

    2011

  23. [23]

    Mark Harman and Phil McMinn. 2009. A theoretical and empirical study of search-based testing: Local, global, and hybrid search.IEEE Transactions on Software Engineering36, 2 (2009), 226–247

    2009

  24. [24]

    René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. InProceedings of the 2014 international symposium on software testing and analysis. 437–440

    2014

  25. [25]

    Konstantinos Kitsios, Marco Castelluccio, and Alberto Bacchelli. 2025. Automated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing.arXiv preprint arXiv:2509.01616(2025). 2https://github.com/CATGen-repository/CATGen , Vol. 1, No. 1, Article . Publication date: July 2026. 22 J. Chen, Z. Wang, L. Yang et al

    arXiv 2025

  26. [26]

    Divya Kumar and Krishn Kumar Mishra. 2016. The impacts of test automation on software’s cost, quality and time to market.Procedia Computer Science79 (2016), 8–15

    2016

  27. [27]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931

    2023

  28. [28]

    Runlin Liu, Zhe Zhang, Yunge Hu, Yuhang Lin, Xiang Gao, and Hailong Sun. 2025. LLM-based Unit Test Generation for Dynamically-Typed Programs.arXiv preprint arXiv:2503.14000(2025)

    arXiv 2025

  29. [29]

    Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the usage of text-to-text transfer transformer to support code-related tasks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 336–347

    2021

  30. [30]

    Chao Ni, Xiaoya Wang, Liushan Chen, Dehai Zhao, Zhengong Cai, Shaohua Wang, and Xiaohu Yang. 2024. CasModaTest: A cascaded and model-agnostic self-directed framework for unit test generation.arXiv preprint arXiv:2406.15743 (2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J Mooney, and Milos Gligoric. 2023. Learning deep semantics for test completion. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    2023

  32. [32]

    Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In29th International Conference on Software Engineering (ICSE’07). IEEE, 75–84

    2007

  33. [33]

    Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. Aster: Natural and multi- language unit test generation with llms. In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 413–424

    2025

  34. [34]

    Qwen. 2024. Qwen2.5-32B. https://huggingface.co/Qwen/Qwen2.5-32B

    2024

  35. [35]

    Qwen. 2024. Qwen2.5-Coder-32B-Instruct. https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct

    2024

  36. [36]

    Qwen. 2024. Qwen2.5-Coder-7B-Instruct. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct

    2024

  37. [37]

    Per Runeson. 2006. A survey of unit testing practices.IEEE software23, 4 (2006), 22–29

    2006

  38. [38]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation.IEEE Transactions on Software Engineering50, 1 (2023), 85–105

    2023

  39. [39]

    Davide Spadini, Maurício Aniche, Magiel Bruntink, and Alberto Bacchelli. 2017. To mock or not to mock? an empirical study on mocking practices. In2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 402–412

    2017

  40. [40]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context.arXiv preprint arXiv:2009.05617(2020)

    Pith/arXiv arXiv 2020

  41. [41]

    Dong Wang, Hanmo You, Lingwei Zhu, Kaiwei Lin, Zheng Chen, Chen Yang, Junji Yu, Zan Wang, and Junjie Chen

  42. [42]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision.IEEE Transactions on Software Engineering50, 4 (2024), 911–936

    2024

  43. [43]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1258–1268

    2024

  44. [44]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1398–1409

    2020

  45. [45]

    Robert F Woolson. 2007. Wilcoxon signed-rank test.Wiley encyclopedia of clinical trials(2007), 1–3

    2007

  46. [46]

    Xusheng Xiao, Sihan Li, Tao Xie, and Nikolai Tillmann. 2013. Characteristic studies of loop problems for structural test generation via symbolic execution. In2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 246–256

    2013

  47. [47]

    Chen Yang and Junjie Chen. 2026. Uncovering Business Logic Bugs via Semantics-Driven Unit Test Generation. arXiv:2604.23509 [cs.SE] https://arxiv.org/abs/2604.23509

    Pith/arXiv arXiv 2026

  48. [48]

    Chen Yang, Junjie Chen, Bin Lin, Ziqi Wang, and Jianyi Zhou. 2024. Advancing code coverage: Incorporating program analysis with large language models.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  49. [50]

    Chen Yang, Ziqi Wang, Lin Yang, Dong Wang, Shutao Gao, Yanjie Jiang, and Junjie Chen. 2026. WiseUT: An Intelligent Framework for Unit Test Generation. In2026 IEEE/ACM 48th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). , Vol. 1, No. 1, Article . Publication date: July 2026. Context Matters: Improving the Practica...

    2026

  50. [51]

    Chen Yang, Lin Yang, Ziqi Wang, Dong Wang, Jianyi Zhou, and Junjie Chen. 2025. Clarifying Semantics of In-Context Examples for Unit Test Generation.arXiv preprint arXiv:2510.01994(2025)

    arXiv 2025

  51. [52]

    Chen Yang, Lin Yang, Ziqi Wang, Dong Wang, Jianyi Zhou, and Junjie Chen. 2025. Clarifying Semantics of In-Context Examples for Unit Test Generation. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 3046–3057. doi:10.1109/ASE63991.2025.00250

    arXiv 2025

  52. [53]

    Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, et al. 2024. An Empirical Study of Unit Test Generation with Large Language Models.arXiv preprint arXiv:2406.18181(2024)

    Pith/arXiv arXiv 2024

  53. [54]

    Xin Yin, Chao Ni, Xinrui Li, Liushan Chen, Guojun Ma, and Xiaohu Yang. 2025. Enhancing LLM’s Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection.arXiv preprint arXiv:2501.07425 (2025)

    Pith/arXiv arXiv 2025

  54. [55]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation.Proceedings of the ACM on Software Engineering1, FSE (2024), 1703–1726

    2024

  55. [56]

    Jia Zhang, Zihao Liu, Yuchen Li, Zhongzhi Chen, Xuefeng Zhang, Shuang Lin, Yuxin Wu, Minghao Xu, Lianjun Wang, Weijie Zhao, Hua Zhou, Jiawei Zhang, Zhiyuan Zhang, Chao Liu, and Jun Guo. 2023. vLLM: High-Performance LLM Inference and Serving. https://arxiv.org/abs/2309.08017

    Pith/arXiv arXiv 2023

  56. [57]

    Hong Zhu, Patrick AV Hall, and John HR May. 1997. Software unit test coverage and adequacy.Acm computing surveys (csur)29, 4 (1997), 366–427

    1997

  57. [58]

    Hengcheng Zhu, Valerio Terragni, Lili Wei, Shing-Chi Cheung, Jiarong Wu, and Yepang Liu. 2025. Understanding and Characterizing Mock Assertions in Unit Tests.Proceedings of the ACM on Software Engineering2, FSE (2025), 554–575. , Vol. 1, No. 1, Article . Publication date: July 2026

    2025

  58. [2025]

    A Survey of Reinforcement Learning for Software Engineering.arXiv preprint arXiv:2507.12483(2025)

    Pith/arXiv arXiv 2025