Pith. sign in

REVIEW 2 major objections 4 minor 59 references

Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For LLM-generated tests, coverage and mutation scores track real-bug detection only when the code given to the model is bug-free and the comparison is across models.

desk verdict A solid, well-scoped replication that gives the LLM test-gen evaluation crowd a genuine divergence from Inozemtseva and Papadakis, but the headline cross-model result rests on 13 aggregate points and needs robustness work before it should be treated as settled. read the letter →

arxiv 2607.22880 v1 pith:VPJTMYJU submitted 2026-07-24 cs.SE cs.AIcs.LG

Junda Zhao , Shurui Zhou , Eldan Cohen This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords testeffectivenesscodecoveragemutationscoreLLM-generatedunittestsreplicationstudyreal-bugdetectionsuitesizeproxymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that coverage and mutation score — the standard proxies used to judge LLM-generated unit tests — do relate to real-bug detection, but only under specific conditions. It replicates the two earlier studies cited as [30] and [40] using 101,123 tests generated by 11 large language models (13 model configurations) for 318 real Java defects, with each defect's code fed to the model in both its fixed and its buggy form. The results diverge from the earlier conclusions about human-written tests: test suite size is not a strong confounder, and when the code can be assumed bug-free, coverage and mutation correlate moderately to strongly with bug detection when models are compared against each other. When the code-under-test is itself buggy, coverage ceases to predict whether generated tests will expose the bug, and mutation testing cannot be applied. The takeaway is a context-dependent verdict on proxy metrics plus practical guidance for how to evaluate LLM-based test generation.

What carries the argument

The argument is carried by a three-granularity correlation design paired with a two-condition prompt experiment. For each of the 318 defects, the same LLM is prompted with both the fixed and the buggy version of the focal method; coverage (statement, branch, modified condition), mutation score, suite size, and bug-detection ratio are measured per method and aggregated two ways (average over methods, and accumulated global ratio). Correlations are then computed at three levels: pooled across all models, separately within each model, and between models with each model reduced to a single data point, under both unconstrained suite size and fixed sizes $k\in\{3,5,10\}$. The inter-model level is where weak pooled correlations become the moderate-to-strong signals that ground the main claim, and the buggy-input condition is what draws the boundary of that claim.

What would settle it

Recompute the inter-model correlations with one model removed at a time; if any single deletion pushes the key coefficients (average branch coverage vs. bug detection, raw mutation score vs. bug detection, branch vs. raw mutation at $k=10$) below the moderate threshold or past $p=0.05$, the across-model claim fails to generalize. Adding a new set of LLMs and seeing whether the correlations reproduce would settle it just as directly.

Watch

Extended reading notes

Core claim

The central discovery is that the relationship among coverage, mutation score, test suite size, and real-bug detection for LLM-generated tests is not the relationship reported for human-written tests in [30] and [40]. Pooling suites across models or looking within a single model gives mostly weak correlations, but collapsing each model into one aggregated data point reveals moderate-to-strong signal: average branch coverage versus raw mutation score at size $k=10$ gives $r=0.780$, average branch coverage versus bug-detection ratio gives $r=0.861$, and raw mutation score versus bug detection gives $r=0.863$, with most coefficients staying moderate to strong after controlling suite size at $k\in\{3,5,10\}$. This across-model signal holds when the code provided to the LLM is the fixed, bug-free version. When the same pipeline is run on the buggy version, coverage–bug-detection correlations become uniformly weak, and mutation analysis is not applicable because it presupposes a passing suite. The paper also finds that statement, branch, and modified-condition coverage are not interchangeable for LLM-generated tests, and that suite size is only weakly correlated with mutation score and bug detection.

Load-bearing premise

The load-bearing premise is that the 13 model-level data points, one per LLM configuration, are representative enough that the moderate-to-strong inter-model correlations reflect a real relationship between proxy metrics and bug detection rather than the pull of one or two outlier models.

Editorial extensions

If this is right

  • When the benchmark code can be assumed bug-free, ranking LLM test generators by average branch coverage or raw mutation score should roughly reproduce their real-bug detection ranking, even after controlling for the number of tests.
  • In bug-detection tasks where the code-under-test may already be buggy, coverage should not be used as a proxy; the paper argues for measuring bug detection directly.
  • Test suite size is not a dominant confounder for LLM-generated suites, so the size-control correction that prior work applied to human-written test pools does not carry over unchanged.
  • Statement, branch, and modified-condition coverage carry different information for LLM-generated tests; a single coverage number can mislead.
  • Compilation and pass rates are best read as cost-effectiveness measures, not as evidence of bug-detection ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The across-model conclusion rests on 13 model-level data points; a leave-one-out analysis or bootstrap confidence intervals would show whether a single outlier model is carrying the strong coefficients. This check is not in the paper, which only acknowledges the small sample.
  • The buggy-input result points toward assertion quality as the operative factor: tests generated from buggy code often assert the faulty behavior, so a metric that scores whether a generated assertion matches the fixed version's behavior might predict bug detection where coverage cannot.
  • Because every suite comes from one prompt template, the context-dependence finding may itself be prompt-dependent; varying the prompt (adding documentation, removing context, changing style) would test whether the across-model signal persists.
  • The focal-method-level sampling likely explains why suite size is not a confounder; a project-level pool that mixes tests across methods could reintroduce a size effect, which would reconcile this result with earlier findings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper is a large-scale conceptual replication of Inozemtseva et al. (ICSE 2014) and Papadakis et al. (ICSE 2018) for LLM-generated unit tests. Using 318 buggy focal methods from Defects4J and 11 LLMs (configured into 13 model settings), the authors generate 8,268 test suites containing 101,123 tests from both fixed and buggy versions of the focal methods. They measure statement, branch, and modified-condition coverage with CodeCover, mutation scores with PIT, and real-bug detection with a differential-testing oracle (a test is effective if it passes on the fixed version and fails on the buggy version). Correlations among test suite size, coverage, mutation score, and bug-detection ratio are computed at three granularities (combined, intra-model, inter-model) and with two aggregations (average, accumulated). The main findings are that coverage and mutation are only weakly correlated with effectiveness in the combined and intra-model views, but moderately to strongly correlated across models when tests are generated from bug-free code; that these cross-model correlations largely survive controlling for test suite size; and that coverage is not informative when tests are generated from buggy code. The authors conclude that proxy metrics are context-dependent and that suite size is not a dominant confounder for LLM-generated suites.

Significance. If the findings are robust, the paper makes a substantive contribution: it demonstrates that established negative results for human-written test suites do not transfer uniformly to LLM-generated tests, and it provides the community with a more nuanced interpretation of coverage- and mutation-based evaluations. Strengths include the large generated corpus, the use of real Defects4J bugs rather than mutants alone, focal-method-level suite construction, a differential-testing oracle, two aggregation schemes, and a publicly archived replication package with scripts and raw data. The manuscript is also commendably explicit about its limitations. However, the positive cross-model conclusion and the buggy-input conclusion are currently less secured in the paper than the abstract suggests: the former rests on n=13 inter-model correlations without robustness diagnostics, and the latter is deferred to the replication package. Both issues are addressable with additional analysis and reporting, so I view this as a major-revision rather than a rejection.

major comments (2)
  1. [§5.4 (RQ2, buggy-input setting)] The central claim that coverage loses predictive power when the code-under-test is buggy rests entirely on a summary statement: the text says, 'Due to space constraints... we do not report the full set of coefficients here; complete statistics are available in our replication package.' No coefficient, p-value, or effect size is given in the manuscript for the buggy-input condition under any of the three analysis views. Because the abstract and Section 7 present the buggy-input result as half of the main context-dependence conclusion, this is a load-bearing reporting gap. Please add a compact table or appendix with the combined, intra-model, and inter-model correlations for the buggy-input suites, analogous to Tables 9–11 for the bug-free condition.
  2. [§3.7; Tables 6, 7, 10, 11, 13, 14] The cross-model conclusion that coverage and mutation are meaningful when comparing across models is based on correlations computed from 13 model-level data points. The paper acknowledges the small sample in §8, but it provides no leave-one-out analysis, bootstrap confidence intervals, or robust-regression diagnostics. At n=13, one or two models can move a Pearson coefficient by several tenths and change its significance, and the effective number of independent points is likely smaller than 13 because the settings include paired reasoning and non-reasoning configurations of the same hybrid models. Since the combined and intra-model views are weak, the inter-model view is the only supporting line of evidence for the positive half of the central claim. Please add robustness diagnostics (e.g., leave-one-out tables, bootstrap confidence intervals, or Spearman/robust fits) and state how the paired configurations are treated in the analysis.
minor comments (4)
  1. [§3.3; Table 2] The paper says the evaluation yields 13 model settings, but Table 2 lists 11 model names without marking the hybrid configurations that produce the extra settings. Please enumerate the 13 settings explicitly or add a configuration column to the table.
  2. [§3.5] The description of the sampling procedure says 'we construct 1,000 unique draws' but does not state whether sampling is with or without replacement, how uniqueness is determined, or how many suites are available per focal method and model. Please clarify these details, as they affect the interpretation of the combined- and intra-model correlations.
  3. [§4.4; Table 8] Several p-values in Table 8 are reported as 0.0. Since a p-value cannot be exactly zero, please report these as inequalities or with a bound (e.g., p < 10^{-300}) rather than 0.0.
  4. [§3.7] The Guilford scale is described for verbal labels of correlation strength, but the paper does not state whether these thresholds are applied only to Pearson coefficients or also to Kendall's tau. Please clarify, since some conclusions in Sections 4 and 5 rely on the threshold language.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study's correlations are computed from independently measured coverage, mutation, suite size, and bug detection, with no fitted parameter or definitional reduction.

full rationale

This is an empirical measurement and replication study, not a derivation. Each quantity is measured by an independent tool or oracle: statement/branch/MCC coverage by CodeCover, mutation scores by PIT, suite size by counting generated tests, and real-bug detection by the differential-testing criterion that a test passes on the fixed version and fails on the buggy version of a Defects4J focal method. The RQ1–RQ3 correlation analyses then compute Pearson and Kendall coefficients over these independently obtained measurements. No equation in the paper defines any of the claimed outcomes in terms of another measured quantity by construction, and no parameter is fitted to a subset of the data and then renamed as a prediction. The central claim that coverage and mutation are meaningful signals when comparing across models is a direct summary of the inter-model correlation coefficients reported in Tables 6, 7, 10, 11, 13, and 14, and it is falsifiable by the data rather than entailed by any definition. The absence of leave-one-out analysis, confidence intervals, or robust-regression diagnostics at n=13 is a genuine statistical-robustness concern, as is the deferral of the buggy-code coefficients in Section 5.4 to the replication package, but these are reporting and robustness gaps, not circularity. The only self-reference in the paper is the authors' own replication package citation [61], which is not load-bearing for any conclusion. No ansatz is smuggled in by citation, no uniqueness theorem is imported, and no known result is merely renamed. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The study introduces no fitted constants, no new entities, and no derivation; all numbers are measured. The axioms are standard empirical software engineering assumptions plus one inferential assumption about the 13-point inter-model correlations, which is the most load-bearing fragility in the paper.

assumptions (6)
  • domain assumption A Defects4J patch is considered a real functional bug if at least one developer-written test fails on the buggy version and passes on the fixed version.
    Used in §3.2 criterion (3) to select 318 focal methods. If the developer-test oracle mislabels refactorings or performance changes as functional bugs, the bug-detection measurements are unreliable.
  • domain assumption A generated test is effective (bug-detecting) iff it passes on the fixed focal method and fails on the buggy focal method.
    Defined in §3.6. This differential-testing oracle ignores tests that fail on both versions or pass on both, which could over- or under-count detection in flaky or order-dependent settings.
  • domain assumption The fixed version of each Defects4J focal method is treated as effectively bug-free for the regression-style condition.
    Assumed throughout §5 for the bug-free input setting. If 'fixed' versions contain latent bugs, the measured bug-detection signal conflates the target bug with other faults.
  • domain assumption Correlations computed over 13 model-level aggregate data points can support statistically meaningful inference without additional outlier diagnostics.
    Introduced in §3.7 as the inter-model view and acknowledged as a threat in §8 (Internal Validity). The paper's headline cross-model conclusions lean on this assumption, which is not independently verified with leave-one-out analyses.
  • domain assumption CodeCover and PIT correctly measure statement, branch, and modified condition coverage and mutation scores.
    Used in §3.6. Tool measurement errors would propagate through every correlation coefficient in the study.
  • domain assumption The adopted prompt template, following Yang et al., is representative of LLM-based test generation practice.
    Adopted in §3.4. The paper notes in §8 (External Validity) that alternative prompt designs may yield different relationships, so generalizability is limited by this single workflow.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)." pith.science (2026). https://pith.science/paper/VPJTMYJU

@misc{pith2026260722880,
  author       = {Pith},
  title        = {Pith review of: Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VPJTMYJU}},
  note         = {Machine review of arXiv:2607.22880}
}
read the original abstract

Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation. Prior work commonly evaluates generated test suites using proxy metrics such as code coverage and mutation score. However, studies by Inozemtseva et al. and Papadakis et al. show that, for human-written tests, correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled, raising concerns about the validity of evaluations based on proxy metrics. It also remains unclear whether these conclusions carry over to LLM-generated tests, given that prevailing LLM-based test-generation workflows differ substantially from traditional approaches. In this paper, we conduct a large-scale replication study of these two prior works using a wide range of test suites generated by a diverse set of LLMs, and re-examine the relationships among coverage, mutation, and real-bug detection effectiveness. Our findings diverge substantially from prior results. We show that the usefulness of coverage and mutation is highly context-dependent: in regression-style settings where the code provided to the LLM can be reasonably assumed bug-free, these metrics can provide meaningful signals when comparing across models; in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators. We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.

Figures

Figures reproduced from arXiv: 2607.22880 by the authors.

Figure 1
Figure 1. Box plot of the number of tests within each test suite generated by different models. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Box plots of the correlation between test suite size and mutation score (raw and normalized) in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Box plots of the correlation between coverage (statement, branch, and modified condition) and [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Representative example comparing tests generated from the bug-free (left) and buggy (right) imple [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 17 canonical work pages

  1. [1]

    CodeCover

    2025. CodeCover. http://codecover.org/. Accessed: 2025-09-10

    work page 2025

  2. [2]

    Azat Abdullin, Pouria Derakhshanfar, and Annibale Panichella. 2025. Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generation. arXiv:2501.10200 [cs.SE] doi:10.48550/arXiv.2501.10200

    work page Pith review arXiv doi:10.48550/arxiv.2501.10200 2025

  3. [3]

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2024. A3Test: Assertion-Augmented Automated Test Case Generation.Information and Software Technology176 (2024), 107565. doi:10.1016/j.infsof.2024.107565

    arXiv 2024

  4. [4]

    J. H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is mutation an appropriate tool for testing experiments?. InProceedings of the 27th International Conference on Software Engineering(St. Louis, MO, USA)(ICSE ’05). Association for Computing Machinery, New York, NY, USA, 402–411. doi:10.1145/1062455.1062530

    arXiv 2005

  5. [5]

    Anthropic. 2025. Claude 4 Sonnet. https://www.anthropic.com/claude/sonnet. Accessed: 2025-05-23

    work page 2025

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] doi:10.48550/arXiv.2108.07732

    arXiv doi:10.48550/arxiv.2108.07732 2021

  7. [7]

    Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. Unit Test Generation using Generative AI: A Comparative Performance Analysis of Autogeneration Tools. InProceedings of the 1st International Workshop on Large Language Models for Code(Lisbon, Portugal)(LLM4Code ’24). Association for Computing Machinery, New York, NY, USA, 54–61. doi:10.1...

    arXiv 2024

  8. [8]

    Max Brunsfeld. 2018. Tree-sitter: An incremental parsing system for programming tools. Accessed: 2025-02-21. doi:10.5281/zenodo.4619183

    doi:10.5281/zenodo.4619183 2018

Show all 59 references
  1. [9]

    Xia Cai and Michael R. Lyu. 2005. The effect of code coverage on fault detection under different testing profiles. SIGSOFT Softw. Eng. Notes30, 4 (July 2005), 1–7. doi:10.1145/1082983.1083288

    2005

  2. [10]

    Carver, Natalia Juristo, Maria Teresa Baldassarre, and Sira Vegas

    Jeffrey C. Carver, Natalia Juristo, Maria Teresa Baldassarre, and Sira Vegas. 2014. Replications of software engineering experiments.Empirical Software Engineering19, 2 (2014), 267–276. doi:10.1007/s10664-013-9290-8

    2014 doi

  3. [11]

    Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] doi:10.48550/ arXiv.2107.03374

    2021 doi arXiv

  4. [12]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering(Porto de Galinhas, Brazil)(...

    2024

  5. [13]

    Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: a practical mutation testing tool for Java (demo). InProceedings of the 25th International Symposium on Software Testing and Analysis(Saarbrücken, Germany)(ISSTA 2016). Associat...

    2016

  6. [14]

    Desmarais

    Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre-trained Large Language Models and mutation testing.Information and Software Technology 171 (2024), 107468. doi:10.1016/j.infsof.2024.107468

    2024

  7. [15]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] doi:10.48550/arXiv.2501.12948

    2025 doi arXiv

  8. [16]

    DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] doi:10.48550/arXiv.2412.19437

    2025 doi arXiv

  9. [17]

    Frankl and Oleg Iakounenko

    Phyllis G. Frankl and Oleg Iakounenko. 1998. Further empirical studies of test effectiveness. InProceedings of the 6th ACM SIGSOFT International Symposium on Foundations of Software Engineering(Lake Buena Vista, Florida, USA) (SIGSOFT ’98/FSE-6). Association for Computing Mach...

    1998

  10. [18]

    Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Trans. Softw. Eng. Methodol.24, 2, Article 8 (Dec. 2014), 42 pages. doi:10.1145/2685612

    2014 doi

  11. [20]

    Milos Gligoric, Alex Groce, Chaoqiang Zhang, Rohan Sharma, Mohammad Amin Alipour, and Darko Marinov. 2013. Comparing non-adequate test suites using coverage criteria. InProceedings of the 2013 International Symposium on Software Testing and Analysis(Lugano, Switzerland)(ISSTA ...

    2013

  12. [21]

    Google. 2025. Gemini 2.5 Flash Model. https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash. Accessed: 2025-08-19

    2025

  13. [22]

    Google. 2025. Gemini 2.5 Pro Model. https://ai.google.dev/gemini-api/docs/models#gemini-2.5-pro. Accessed: 2025-08-19

    2025

  14. [23]

    Guilford

    J.P. Guilford. 1956.Fundamental Statistics in Psychology and Education. McGraw-Hill. https://books.google.ca/books? id=u-G10ZqLhtsC

    1956

  15. [24]

    Vitor Guilherme and Auri Vincenzi. 2023. An initial investigation of ChatGPT unit test generation capability. In Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing(Campo Grande, MS, Brazil) (SAST ’23). Association for Computing Machinery, N...

    2023

  16. [25]

    Hayhurst and Dan S

    Kelly J. Hayhurst and Dan S. Veerhusen. 2001. A practical approach to modified condition/decision coverage. In Proceedings of the 20th Digital A vionics Systems Conference (DASC), Vol. 1. IEEE, 1B2/1–1B2/10. doi:10.1109/DASC.2001. 963305

    2001 doi

  17. [26]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. InAdvances in Neural Information Processing Systems (Datasets a...

    2021 doi arXiv

  18. [27]

    Soneya Binta Hossain, Raygan Taylor, and Matthew Dwyer. 2025. Doc2OracLL: Investigating the Impact of Docu- mentation on LLM-Based Test Oracle Generation.Proc. ACM Softw. Eng.2, FSE, Article FSE084 (June 2025), 22 pages. doi:10.1145/3729354

    2025 doi

  19. [29]

    Zhang, Mark Harman, Mingzhe Du, and Heming Cui

    Dong Huang, Jie M. Zhang, Mark Harman, Mingzhe Du, and Heming Cui. 2025. Measuring the Influence of Incorrect Code on Test Generation. arXiv:2409.09464 [cs.SE] doi:10.48550/arXiv.2409.09464

    2025 doi arXiv

  20. [30]

    Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering(Hyderabad, India)(ICSE 2014). Association for Computing Machinery, New York, NY, USA, 435–445...

    2014

  21. [31]

    Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. 2025. TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. InThe Thirteenth International Conference on Learning Representations. doi:10.48550/arXiv. 2410.00752

    2025 doi arXiv

  22. [32]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InThe Twelfth International Conference on Learning Representations. doi:10.48550/arXiv.2310.06770

    2024 doi arXiv

  23. [33]

    Jorgensen

    Paul C. Jorgensen. 2013.Software Testing: A Craftsman’s Approach(fourth ed.). Auerbach Publications. doi:10.1201/ b15980

    2013

  24. [34]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. InProceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA)(ISSTA 2014). Associat...

    2014

  25. [35]

    Ernst, Reid Holmes, and Gordon Fraser

    René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. InProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering(Hong...

    2014

  26. [36]

    Andrea Lops, Fedelucio Narducci, Azzurra Ragone, Michelantonio Trizio, and Claudio Bartolini. 2025. A System for Automated Unit Test Generation using Large Language Models and Assessment of Generated Test Suites. In 2025 IEEE International Conference on Software Testing, Verif...

    2025

  27. [37]

    OpenAI. 2025. GPT-4.1. https://openai.com/index/gpt-4-1/. Accessed: 2025-08-19

    2025

  28. [38]

    OpenAI. 2025. OpenAI Models - O4 Mini. https://platform.openai.com/docs/models/o4-mini. Accessed: 2025-08-19

    2025

  29. [39]

    Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2018. Automated Test Case Generation as a Many- Objective Optimisation Problem with Dynamic Selection of the Targets.IEEE Transactions on Software Engineering44, 2 (2018), 122–158. doi:10.1109/TSE.2017.2663435 Pr...

    2018

  30. [40]

    Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are mutation scores correlated with real fault detection? A large scale empirical study on the relationship between mutants and real faults. InProceedings of the 40th International Conference on Software Engineer...

    2018

  31. [41]

    Qwen Team. 2025. Qwen3-Coder: Agentic Coding in the World. https://qwenlm.github.io/blog/qwen3-coder/. Accessed: 2025-08-19

    2025

  32. [42]

    Qwen Team. 2025. Qwen3: Think Deeper, Act Faster. https://qwenlm.github.io/blog/qwen3/. Accessed: 2025-08-19

    2025

  33. [43]

    Hellendoorn

    Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2023. CAT-LM Training Language Models on Aligned Code and Tests. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering(Echternach, Luxembourg)(ASE ’23). IEEE ...

    2023

  34. [44]

    José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software develop- ment: a controlled experiment and think-aloud observations. InProceedings of the 2015 International Symposium on Software Testing and Analysis(Baltimore, MD, USA)...

    2015

  35. [45]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation.IEEE Transactions on Software Engineering50, 1 (2024), 85–105. doi:10. 1109/TSE.2023.3334955

    2024

  36. [46]

    Ye Shang, Quanjun Zhang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. 2025. A Large-Scale Empirical Study on Fine-Tuning Large Language Models for Unit Testing.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA074 (June 2025), 23 pages. doi:10.1145/3728951

    2025 doi

  37. [47]

    Shull, Jeffrey C

    Forrest J. Shull, Jeffrey C. Carver, Sira Vegas, and Natalia Juristo. 2008. The role of replications in Empirical Software Engineering.Empirical Softw. Engg.13, 2 (April 2008), 211–218. doi:10.1007/s10664-008-9060-1

    2008 doi

  38. [48]

    Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using Large Language Models to Generate JUnit Tests: An Empirical Study. In Proceedings of the 28th International Conference on Evalu...

    2024

  39. [49]

    André Silva, Nuno Saavedra, and Martin Monperrus. 2024. GitBug-Java: A Reproducible Benchmark of Recent Java Bugs. In2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR). IEEE Computer Society, Los Alamitos, CA, USA, 118–122. doi:10.1145/3643991.3644884

    2024

  40. [50]

    Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2024. ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation.IEEE Transactions on Software Engineering50, 6 (2024), 1340–1359. doi:10.1109/TSE.2024.3382365

    2024

  41. [51]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit Test Case Generation with Transformers and Focal Context. arXiv:2009.05617 [cs.SE] doi:10.48550/arXiv.2009.05617

    2021 doi arXiv

  42. [52]

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2025. TestEval: Benchmarking Large Language Models for Test Case Generation. InFindings of the Association for Computational Linguistics: NAACL 2025, Luis Chi...

    2025 doi

  43. [53]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering(Seoul, South Korea)(ICSE ’20). Association f...

    2020

  44. [54]

    Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. 2020. BugsInPy: a database of existing bugs in Python p...

    2020

  45. [55]

    xAI. 2025. Grok-3 Model Documentation. https://docs.x.ai/docs/models/grok-3. Accessed: 2025-08-19

    2025

  46. [56]

    xAI. 2025. Grok-4 Model Documentation. https://docs.x.ai/docs/models/grok-4. Accessed: 2025-08-19

    2025

  47. [57]

    Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. 2024. On the Evaluation of Large Language Models in Unit Test Generation. In Proceedings of the 39th IEEE/ACM International Conference on ...

    2024

  48. [58]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation.Proc. ACM Softw. Eng.1, FSE, Article 76 (July 2024), 24 pages. doi:10.1145/3660783 Proc. ACM Softw. Eng., Vol. 3, No. ISS...

    2024 doi

  49. [59]

    Quanjun Zhang, Chunrong Fang, Siqi Gu, Ye Shang, Zhenyu Chen, and Liang Xiao. 2025. Large Language Models for Unit Testing: A Systematic Literature Review. arXiv:2506.15227 [cs.SE] doi:10.48550/arXiv.2506.15227

    2025 doi arXiv

  50. [60]

    Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. 2024. TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Models. arXiv:2409.17561 [cs.SE] doi:10.48550/arXiv. 2409.17561

    2024 doi arXiv

  51. [61]

    Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate With Their Effectiveness? (Replicability Study)

    Junda Zhao, Shurui Zhou, and Eldan Cohen. 2026. Replication Package for “Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate With Their Effectiveness? (Replicability Study)”. https://github.com/drixs2050/ Cov_mut_bug_detect_correlation. doi:10.5281/zenodo.21...

    2026 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.