Design choices made by LLM-based test generators prevent them from finding bugs

Meiyappan Nagappan; Noble Saji Mathews

REVIEW 3 major objections 7 minor 2 cited by

Design choices made by LLM-based test generators prevent them from finding bugs

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Filtering out failing tests makes LLM test generators lock in bugs instead of finding them.

desk verdict Design-level bug in LLM test generators is real and well demonstrated, but the headline percentages rest on an unverified oracle assumption. read the letter →

arxiv 2412.14137 v1 pith:3JICCSYA submitted 2024-12-18 cs.SE cs.AI

Noble Saji Mathews , Meiyappan Nagappan This is my paper · ORCID

classification cs.SEcs.AI

keywords LLM-basedtestgenerationsoftwaretestingoraclecodecoveragebugdetectioncoverage-guidedsuitereliability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the design of LLM-based test generators—maximizing code coverage and filtering out tests that fail on the code under test—can make the tools bless bugs instead of catching them. On 287 real human-written buggy Python programs, the two coverage-guided tools (CoverAgent and CoverUp) rejected the tests that failed on the buggy implementation, i.e., the only tests that could reveal bugs, and kept tests that assert the buggy outputs as expected. In their final test suites, 59.6% of CoverAgent's tests and 68.1% of CoverUp's tests pass on the buggy code but fail on the correct reference solution, while a pure-generation baseline produced far fewer such tests. The authors conclude that the design philosophy needs revision: tests should be derived from requirements, and relying on these tools during active development can create a false sense of security.

What carries the argument

The central machinery is a coverage-guided generate-and-filter loop. The tool measures which lines or branches are uncovered, prompts an LLM to write tests for those regions, runs the tests against the current implementation, discards every test that fails, and repeats until a coverage target is reached. Because "fails on the code under test" is treated as "bad test," the loop deletes the only tests that could expose a bug and keeps assertions that match wrong outputs; coverage then pushes generation toward more of those bug-engrained tests.

What would settle it

Audit all tests classified as 'OG Success, REF Failed' (171 from CoverAgent and 62 from CoverUp) against human-written oracles derived only from the original problem statements; if a substantial share of them confirm the buggy output as correct, the reference-solution assumption fails and the reported bug-validation rates are overstated.

Watch

Extended reading notes

Core claim

The central discovery is that the bug-validation effect comes from the pipeline design, not from the LLM's ability to write tests. Both coverage-guided tools treat a test that fails on the current implementation as a bad test to discard, and they treat a test that passes and increases coverage as good; the result is that assertions matching the incorrect behavior are retained while potentially bug-revealing tests are thrown away. Evaluated against the reference solution, the retained suites show the damage: 171 of 287 CoverAgent tests (59.6%) and 62 of 91 CoverUp tests (68.1%) pass on the original buggy implementation and fail on the reference, meaning they validate bugs. CoverUp failed to produce any suite at all for 196 of 287 samples, and its successful suites had the highest bug-validation rate. The paper's manual inspection of a few real-world issues from open-source projects showed the same pattern of coverage targets being met while the critical edge case was ignored.

Load-bearing premise

The study assumes that every reference solution is a correct implementation, so a test that passes on the buggy code but fails on the reference is counted as validating a bug; if some reference solutions are wrong or some inputs admit more than one valid output, those percentages would be inflated.

Editorial extensions

If this is right

A developer using one of these tools during active development can end up with a green test suite whose assertions encode current bugs, so the suite hides regressions instead of catching them.
The effect is large enough to matter in practice: in the final suites of the two coverage-guided tools, a majority of retained tests validate bugs rather than detect them.
Coverage targets act as an early stopping condition that can leave critical edge cases untested, as the paper's timezone-crash example illustrated.
The paper recommends shifting to requirement-driven test development, using LLMs to help write tests rather than to infer the intended behavior from possibly buggy code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism is the filter-and-cover loop, then any test generator that uses the same keep-only-passing-tests policy should show the same bias, regardless of which LLM is behind it; a direct test would be to add a keep-failing-tests mode to one of these tools and measure whether mutation scores improve.
The false sense of security may be self-reinforcing: once a bug-validating suite is committed, future code changes are checked against assertions that already encode the bug, so the bug becomes harder to remove without breaking the 'passing' suite.
A testable extension: run the same pipeline on code with seeded faults and compute the mutation score of generated suites; the prediction is that coverage-driven filtering lowers the score, i.e., produces weaker tests.
The critique likely extends to mutation-based LLM test generators that use the same failing-test filter, because they inherit the same assumption that a failing test is a mistaken test.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper has a real finding: two popular LLM test generators, by design, throw out tests that would expose bugs and keep tests that lock in the buggy behavior. The mechanism is structural—any test that fails on the current code is discarded—so the final suite cannot contain a test that fails on the buggy implementation. The paper demonstrates this with concrete counts and by instrumenting CoverAgent and CoverUp to capture rejected tests. That's worth your time.

What's new: prior work showed LLMs are misled by buggy source code. This paper goes further and shows that even when the LLM generates a useful, bug-revealing test, the tool's filtering removes it. The empirical demonstration on a shared benchmark (Refactory) makes the design flaw concrete. The paper also gives nice real-world illustrations from SWE-bench.

Weak spots: the headline percentages (59.6% and 68.1% bug-validating tests) rely on the reference solution being the unique correct output for each input. The authors acknowledge this in Section VI and manually verified a sample, but not all 287 samples across only four problem statements. If some of the tests labeled "OG Success, REF Failed" are actually asserting a legitimate alternative behavior (whitespace, tie ordering, etc.), those percentages are inflated. The stress-test note is right to flag this as the least secure load-bearing step. It's a caveat, not a fatal flaw: the core qualitative finding—that these tools cannot produce bug-revealing tests by construction—holds regardless of the oracle. Two minor issues: the tools start from empty test suites, which is not their typical deployment, and there's no statistical analysis of variability across LLM runs, though the effect sizes are large enough that this may not matter. Also, I couldn't find a link to the replication package in the text.

Overall, the central argument is sound. The quantitative magnitude needs an oracle audit—either verify all reference solutions or soften the headline claims. I'd send this to review with that request. It's a solid paper for the SE/testing community and would spark good reading-group discussion.

Recommendation: accept for peer review, conditional on oracle validation.

Referee Report

3 major / 7 minor

Summary. The paper empirically evaluates three LLM-based test generation tools (GitHub Copilot, Codium CoverAgent, and CoverUp) on 287 buggy Python programs drawn from the Refactory dataset. It reports that coverage-driven tools that filter failing tests produce final test suites in which a large fraction of tests pass on the buggy implementation but fail on a reference solution (59.6% for CoverAgent and 68.1% for CoverUp), while discarding many tests that would have exposed the bugs (470 and 400, respectively). The authors conclude that the design choices of using code coverage as the primary objective and systematically filtering failing tests can validate bugs and create a false sense of security.

Significance. If the reported magnitudes hold, this is a timely and important finding: it directly challenges the design of popular coverage-guided LLM test generators and gives developers a concrete reason to distrust suites generated against unverified code. The core qualitative insight is robust and supported by construction: because these tools discard any test that fails on the current implementation, their final suites cannot contain tests that fail on that implementation, and the discarded-test counts show that bug-revealing tests are systematically removed. The paper uses a public benchmark and provides a replication package, which supports reproducibility. The main weakness is that the headline percentages treat the Refactory reference solutions as unique oracles, an assumption that the paper itself only partially verifies. The directional claim is sound, but the quantitative strength of the conclusion needs additional support.

major comments (3)

[Section VI and Table I] The headline percentages (59.6% for CoverAgent, 68.1% for CoverUp) classify every test that passes on the buggy program and fails on the reference solution as 'validating bugs.' This inference requires that the reference output is the only acceptable output for each input. The dataset contains only four problem statements from an introductory Python course, where alternative valid behaviors (e.g., output formatting, tie-breaking order, representation choices) are plausible. Section VI states that only a sample of reference solutions was manually verified, but it does not report the sample size, selection procedure, or which problem statements were covered. Please report these details and either manually inspect all tests in the 'OG Success, REF Failed' category or provide a sensitivity analysis that bounds the reported percentages under alternative oracles.
[Section IV and Table I] The label 'OG Success, REF Failed' conflates two distinct interpretations: (a) a test asserts an output that is wrong according to the problem specification, and (b) a test fails on the reference implementation for reasons unrelated to the target bug, such as the reference containing its own flaw or the input admitting multiple valid outputs. Only interpretation (a) makes the test a genuine bug-validating test. The current analysis does not distinguish these cases, so the claim that final suites 'validate bugs' in 59.6% and 68.1% of their tests is stronger than what the data directly show.
[Section VII] The conclusion attributes the observed behavior to 'the use of code coverage as a primary objective and the systematic filtering of failing tests,' but the empirical design does not isolate these two factors. CoverAgent and CoverUp both employ coverage targets and failing-test filtering, so the bug-validating rates cannot be causally attributed to either factor individually based on the presented data. An ablation study, or at minimum a careful discussion of the relative contribution of each factor, is needed to support the causal wording in the conclusion.

minor comments (7)

[Section III-A, step 3] The phrase 'the test coverage, based only on passing tests, was less than 100%' is unclear; it should specify that coverage is measured on the buggy implementation when executed only with the tests that pass on that implementation, and explain why passing-test-only coverage was chosen as the filter.
[Section III-B2] The bullet list for CoverAgent contains overlapping items: 'Filter out tests that don’t compile or run' and 'Discard tests that don’t pass on the current code.' Merging these into a single filtering step would be easier to follow.
[Section IV] The sentence 'in the first 2 cases (with OG Failed), CoverUp and CoverAgent do not generate test suites and reject all tests' is confusing, because the tools do generate suites containing the 'OG Success' tests. Rephrase to indicate that all tests falling into the first two categories are generated but rejected during filtering and therefore do not appear in the final suites.
[Table I] The asterisk note should explicitly state that the starred counts refer to tests that were generated but rejected during the filtering process, and thus are not part of the final test suites reported in the table.
[Throughout] The capitalization of 'CoverUp' is inconsistent (e.g., 'Coverup' appears in Sections II and III-B3); the tool name should be spelled uniformly.
[Section VI] The replication package is mentioned as containing necessary modifications, but no URL or repository identifier is provided; please include the link.
[Section V] In the find_sum example, showing the correct expected outputs (without the +1 bug) alongside the buggy assertions would make the bug-validating nature of the generated test suite immediately clear to the reader.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's conclusions are supported by an external benchmark oracle and instrumented tool runs, with no fitted parameters or self-citation chains doing load-bearing work.

full rationale

This is an empirical evaluation, not a derivation. The central claim that coverage-driven LLM test generators systematically discard bug-revealing tests and retain tests that pass on buggy code is established by instrumenting CoverAgent and CoverUp to capture both rejected and retained tests, and then checking those tests against the Refactory reference solutions. The 'OG Success, REF Failed' category is an external oracle judgement; Section VI explicitly acknowledges the assumption that reference solutions are correct, and an acknowledged benchmark assumption is not a circular input. The fact that these tools filter out failing tests is the object of study, not a result imported from the paper's own definitions. No equations, fitted parameters, or self-citation chains are used to define the reported percentages. The only self-citation, [15] on test-driven development, appears in a closing recommendation and is not load-bearing for any quantitative result. An honest non-finding is therefore warranted: score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities. The study relies on external dataset oracles and a set of domain assumptions about test execution and representativeness.

assumptions (4)

domain assumption Reference solutions from Refactory are correct implementations for the retained samples.
Used to label tests as bug-revealing (OG Failed, REF Success) or bug-validating (OG Success, REF Failed). The paper acknowledges this in Section VI and says it manually verified a sample.
domain assumption Executing a generated test and observing pass/fail on the two implementations is a reliable oracle for test value.
The whole four-way categorization in Section IV assumes deterministic, meaningful test outcomes and that pass/fail status is correctly captured by the instrumented pipelines.
domain assumption The selected 287 samples are representative of the way the tools behave on buggy code.
Dataset filtering in Section III-A (executable, mixed pass/fail, coverage less than 100%, single-function, excluding assignment 2) could bias the observed percentages; generalizing to all buggy code is assumed.
domain assumption The default LLM configuration (GPT-4O) and tool parameters do not change the qualitative filtering behavior.
The paper argues the exact LLM is not important because the focus is pipeline design; this is an assumption since only one LLM is tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Design choices made by LLM-based test generators prevent them from finding bugs." pith.science (2026). https://pith.science/paper/3JICCSYA

@misc{pith2026241214137,
  author       = {Pith},
  title        = {Pith review of: Design choices made by LLM-based test generators prevent them from finding bugs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JICCSYA}},
  note         = {Machine review of arXiv:2412.14137}
}

read the original abstract

There is an increasing amount of research and commercial tools for automated test case generation using Large Language Models (LLMs). This paper critically examines whether recent LLM-based test generation tools, such as Codium CoverAgent and CoverUp, can effectively find bugs or unintentionally validate faulty code. Considering bugs are only exposed by failing test cases, we explore the question: can these tools truly achieve the intended objectives of software testing when their test oracles are designed to pass? Using real human-written buggy code as input, we evaluate these tools, showing how LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests. These findings raise important questions about the validity of the design behind LLM-based test generation tools and their impact on software quality and test suite reliability.

Figures

Figures reproduced from arXiv: 2412.14137 by the authors.

**Figure 1.** Overview of the evaluation pipeline III. METHODOLOGY We design our experiments, such that we can analyze both the passing and failing tests generated by recent LLM-based test generation tools. We aim to understand how different approaches, adopted by these tools affect the generated test suite and if these tests would validate bugs or unintended behaviour when given potentially buggy code. A. Dataset For our study, … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models
cs.SE 2026-07 conditional novelty 6.0 of 10

A context-aware LLM pipeline generates metamorphic relations for 142 AR repositories; hierarchical context plus agentic deliberation yields 3,760 refined MRs that human raters judge mostly valid and testable, with 5 c...
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
cs.SE 2026-07 conditional novelty 6.0 of 10

Buggy code in prompts steers LLMs to write tests that validate the bug, and generating tests from an LLM-written spec docstring instead of the code reduces such misguided tests and increases bug-finding ones.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages · cited by 2 Pith papers

[1]

The economic impact of the ai-powered developer lifecycle and lessons from github copilot,

T. Dohmke, “The economic impact of the ai-powered developer lifecycle and lessons from github copilot,” 2023

work page 2023
[2]

Coverup: Coverage-guided llm-based test generation,

J. A. Pizzorno and E. D. Berger, “Coverup: Coverage-guided llm-based test generation,” arXiv preprint arXiv:2403.16218 , 2024

arXiv 2024
[3]

Using github copilot for test generation in python: An empirical study,

K. El Haji, C. Brandt, and A. Zaidman, “Using github copilot for test generation in python: An empirical study,” in Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024) , 2024, pp. 45–55

work page 2024
[4]

Effective test generation using pre-trained large language models and mutation testing,

A. M. Dakhel, A. Nikanjam, V . Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” Information and Software Technology, vol. 171, p. 107468, 2024

work page 2024
[5]

GitHub - qodo-ai/qodo-cover: Qodo-Cover: An AI-Powered Tool for Automated Test Generation and Code Coverage Enhancement! — github.com,

“GitHub - qodo-ai/qodo-cover: Qodo-Cover: An AI-Powered Tool for Automated Test Generation and Code Coverage Enhancement! — github.com,” https://github.com/qodo-ai/qodo-cover, [Accessed 13-12- 2024]

work page 2024
[6]

The oracle problem in software testing: A survey,

E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014

2014
[7]

Evolutionary generation of whole test suites,

G. Fraser and A. Arcuri, “Evolutionary generation of whole test suites,” in 2011 11th International Conference on Quality Software . IEEE, 2011, pp. 31–40

work page 2011
[8]

Pynguin: Automated unit test generation for python,

S. Lukasczyk and G. Fraser, “Pynguin: Automated unit test generation for python,” in Proceedings of the ACM/IEEE 44th International Con- ference on Software Engineering: Companion Proceedings , 2022, pp. 168–172

work page 2022

Show all 15 references

[9]

Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 919–931

2023
[10]

Rethinking the influence of source code on test case generation,

D. Huang, J. M. Zhang, M. Du, M. Harman, and H. Cui, “Rethinking the influence of source code on test case generation,” arXiv preprint arXiv:2409.09464, 2024

2024 arXiv
[11]

Re- factoring based program repair applied to programming assignments,

Y . Hu, U. Z. Ahmed, S. Mechtaev, B. Leong, and A. Roychoudhury, “Re- factoring based program repair applied to programming assignments,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE/ACM, 2019, pp. 388–398

2019
[12]

Automated unit test improvement using large language models at meta,

N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” in Companion Proceedings of the 32nd ACM International Conference on the F ounda- tions of S...

2024
[13]

The rise and potential of large language model based agents: A survey,

Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al. , “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864 , 2023

2023 arXiv
[14]

SWE-bench: Can language models resolve real- world github issues?

C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real- world github issues?” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https: //openreview.net/forum...

2024
[15]

Test-driven development and llm- based code generation,

N. S. Mathews and M. Nagappan, “Test-driven development and llm- based code generation,” in Proceedings of the 39th IEEE/ACM Interna- tional Conference on Automated Software Engineering , 2024, pp. 1583– 1594

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

The economic impact of the ai-powered developer lifecycle and lessons from github copilot,

T. Dohmke, “The economic impact of the ai-powered developer lifecycle and lessons from github copilot,” 2023

work page 2023

[2] [2]

Coverup: Coverage-guided llm-based test generation,

J. A. Pizzorno and E. D. Berger, “Coverup: Coverage-guided llm-based test generation,” arXiv preprint arXiv:2403.16218 , 2024

arXiv 2024

[3] [3]

Using github copilot for test generation in python: An empirical study,

K. El Haji, C. Brandt, and A. Zaidman, “Using github copilot for test generation in python: An empirical study,” in Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024) , 2024, pp. 45–55

work page 2024

[4] [4]

Effective test generation using pre-trained large language models and mutation testing,

A. M. Dakhel, A. Nikanjam, V . Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” Information and Software Technology, vol. 171, p. 107468, 2024

work page 2024

[5] [5]

GitHub - qodo-ai/qodo-cover: Qodo-Cover: An AI-Powered Tool for Automated Test Generation and Code Coverage Enhancement! — github.com,

“GitHub - qodo-ai/qodo-cover: Qodo-Cover: An AI-Powered Tool for Automated Test Generation and Code Coverage Enhancement! — github.com,” https://github.com/qodo-ai/qodo-cover, [Accessed 13-12- 2024]

work page 2024

[6] [6]

The oracle problem in software testing: A survey,

E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014

2014

[7] [7]

Evolutionary generation of whole test suites,

G. Fraser and A. Arcuri, “Evolutionary generation of whole test suites,” in 2011 11th International Conference on Quality Software . IEEE, 2011, pp. 31–40

work page 2011

[8] [8]

Pynguin: Automated unit test generation for python,

S. Lukasczyk and G. Fraser, “Pynguin: Automated unit test generation for python,” in Proceedings of the ACM/IEEE 44th International Con- ference on Software Engineering: Companion Proceedings , 2022, pp. 168–172

work page 2022