Software Engineering · arXiv:2610.13065 v1 · submitted 9 Oct 2026 · 23 references
TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications
The paper says
“Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes.”
more of the abstract
In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73% are killed by exactly one reference test and, on average, by only 0.2% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60%, despite their suites being largely valid and 88% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 12 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid benchmark, real finding, but mutant construction bias deserves a quantitative sensitivity check before the central claim is fully secure. This line is wrong
1 major objection · 1 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 23 references say.
The paper says, at §2.2, §5, §A.5
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.92):
“Further, selecting mutants that are killed by a single reference test risks favoring mutants triggered only by that test’s exact inputs.”
Pith says: Objection 1 of 1 · major
The paper acknowledges that constructing mutants killed by exactly one reference test (73% of mutants) risks favoring mutants whose failure depends on the exact input structure of that test, rather than on the underlying requirement violation. The mitigation (logic-level mutation instruction, manual audit, Figure 13) is qualitative.
more of objection 1
Since the data to compute the mutation score restricted to mutants killed by more than one reference test (the remaining 27%) are available, the paper should report this restricted score. If the score rises substantially (e.g., above 70%), the central claim that agents 'cannot verify' their implementations would be weakened; if it stays below 60%, the claim is strengthened. A quantitative check is needed to resolve the residual uncertainty this acknowledged limitation creates.
1 minor objection
The paper says, at Table 2
Pith says: minor objection 1 of 1
The column header "MV$t(min)" contains a stray LaTeX dollar sign; it should read "M V t (min)" or "M V Time (min)".
What would settle it
For each of the 165 instances, compute the mutation score of the Opus and Sol test suites using only the subset of mutants that are killed by more than one reference test (the ~27% not singletone-kill). If on this subset the mutation score is substantially higher (e.g., >20 percentage points) than on the full set, then the singletone-kill filtering is inflating the apparent weakness, and the central claim requires circumscription. If the score is comparable or only modestly higher, the concern is assuaged.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 12 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 1 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a well-built benchmark and a genuinely useful paper. The key contribution is a scalable pipeline for turning existing test suites into detailed natural-language specifications with targeted mutants, then measuring whether agents can write strong test suites from scratch. The iterative specification refinement, manual quality control, and transparent limitation statements are good practice. The empirical result — that frontier agents can implement specifications they cannot test — is consistently observed across six models and holds even when the reference executable is provided, which pins the bottleneck on test design rather than specification clarity.
The soft spot is the one the stress-test note flags. 73% of mutants are designed to be killed by exactly one reference test. The paper acknowledges the risk (Section 5) and mitigates it by instructing the mutation agent to mutate logic rather than hard-code input values, with an ablation in Appendix A.5 showing that this reduces literal copying. But that ablation checks generalisation to renamed inputs, not the effect on the overall mutation score. The paper does not report what happens when you restrict the evaluation to the 27% of mutants that are killed by more than one reference test. If that subset’s mutation score is substantially higher, the central claim that agents 'cannot verify' the software they build would need qualification. This is a measurable, incremental check that would strengthen the paper’s core argument, and its absence is the biggest residual uncertainty.
Other concerns are minor. The mutant set samples requirements uniformly, so it does not penalise gaps in untargeted requirements; the paper acknowledges this and argues that agents cannot know which requirements are targeted, which is fair but means the benchmark measures coverage indirectly. The scope (CLI tools in compiled languages) is inherited from ProgramBench and is appropriate for a first large-scale benchmark.
Overall, this is a serious piece of work that will be cited and used. The finding is important even if the exact magnitude of the gap depends on how you construct the mutants. I would bring it to a reading group and would cite it.
Recommendation: Accept with minor revision. Ask the authors to compute and report mutation score on the subset of mutants killed by >1 reference test, and to discuss whether the 73% singleton set introduces an upper bound on achievable mutation score that is inherent to the benchmark design.
The strongest objection
The central claim that agents 'cannot verify' the software they build rests heavily on the mutation score metric: even the best agent kills only 55.5% of mutants. However, 73% of mutants are designed to be killed by exactly one reference test (Section 2.2). The instructions for mutant generation (prompt D.1, 'Mutant-polishing prompt') explicitly ask for mutants with the singleton kill set, even noting that 'an exact singleton kill set is desirable, but it is not required.' This creates a subtle bias: a mutant that is killed by only one reference test is triggered only when the agent test happens to match that particular test's exact input structure and assertion. The paper shows that agents often miss input combinations (Figure 7, 14); this could mean they miss these narrow mutants because the reference test's specific inputs are hard to replicate, not because the agents fail to detect violations of the requirement in principle. If the evaluation were restricted to the 27% of mutants that are killed by more than one reference test, the mutation score might rise substantially, weakening the 'cannot verify' claim. The paper acknowledges this risk in Section 5 ('selecting mutants that are killed by a single reference test risks favoring mutants triggered only by that test's exact inputs'), but does not provide a quantitative check, leaving a residual uncertainty about how much of the observed gap is an artifact of the mutant construction.
The paper is thorough and its main result is well-supported. The reader's weakest assumption (mutants as proxy for real faults) is spot-on, but there is a more specific technical dimension: 73% of mutants are designed to be killed by exactly one reference test. This design choice could make mutants atypically narrow, biasing the mutation score downward. The paper acknowledges the general risk but does not quantify its impact. A simple post-hoc check on the non-singleton subset would discipline whether the gap is real or an artifact. Even if the check shows a large difference, the paper's finding would still hold (agents miss many mutants), but the magnitude of the claim would need adjustment. I therefore agree with the reader's verdict (ACCEPT) and see no need to change it, but flag this specific untested assumption as the most load-bearing point.
The referee's report
If the result holds, this is an important contribution: it identifies a concrete, measurable capability gap in current coding agents—test generation from specifications lags far behind implementation generation. The benchmark is carefully constructed (iterative spec refinement, multi-stage mutant construction with manual audit, flaky-test filtering), and the main finding—agents generate valid but weak tests even when given the exact specification or the reference executable—is striking and practically relevant. TestSuiteBench fills a genuine gap in the evaluation landscape (repository-scale test generation from specs) and the detailed construction pipeline will enable the community to track progress on this critical capability.
Pith's reading of the paper (a machine's summary, not the authors')
This paper introduces TestSuiteBench, a benchmark of 165 open-source applications designed to test whether coding agents can write strong test suites from natural-language specifications alone. The central discovery is a deep gap: frontier agents produce valid implementations that pass 82–93% of reference tests, yet their own test suites kill at most 55% of targeted mutants—and four out of six agents score below 40%. This failure is not due to unclear specifications (giving agents the reference executable leaves scores unchanged) but to weak test design: generated suites are too small and miss input combinations. The results show that agents can build software they cannot verify, a bottleneck that limits fully autonomous development.
What carries the argument. TestSuiteBench is the central benchmark: 165 command-line applications in Rust, Go, and C++ with a structured specification (median 5,635 words, 23 sections), a reference test suite (median 679 functions), and a set of mutants (average 74 per application). Each mutant is constructed to violate a single specification requirement and refined so that 73% are killed by exactly one reference test and only 0.2% of unrelated tests. The evaluation uses mutation score M (fraction of mutants killed by valid generated tests) and validity score V (fraction of generated tests that pass the reference executable).
Core claim. The paper establishes that even state-of-the-art coding agents (Claude Opus 5, GPT 6 Astra, etc.) cannot generate strong test suites from detailed natural-language specifications alone. Despite producing valid tests (88–96% pass on the reference implementation) and correct implementations (86–93% pass reference tests), their test suites achieve mutation scores below 60% when evaluated against mutants that each violate exactly one specification requirement. The gap persists even when the reference executable is provided, showing the bottleneck is test design—not specification ambiguity.
The premise it rests on. The mutants, each designed to violate one specification requirement and refined to be killed by exactly one reference test, are a good proxy for real-world faults that a test suite should catch.
If the paper is right (Pith's inferences, not the authors')
Autonomous software development cannot rely on current agents for verification: they must either produce stronger test suites or use external verification methods.
Test design is a distinct skill from implementation: the same agents that pass reference tests fail to design input combinations that expose violations, suggesting a missing reasoning capability.
TestSuiteBench provides a concrete metric for progress: raising mutation scores above 60% would represent a step toward self-verifying agents.
The finding that suites are too small and miss edge cases implies that agents need better exploration strategies, not just better specification understanding.
Ensemble approaches (combining suites from multiple agents) give only modest gains (59% vs 55% for Opus alone), leaving most mutants untouched.
The low mutation scores even when the reference executable is available suggest that agents lack a systematic strategy for designing adversarial inputs—they may treat test generation as similar to implementation rather than as a search for counterexamples.
The benchmark’s reliance on single-requirement mutants, as acknowledged, may not reflect real-world fault distributions, so the reported scores might either overestimate or underestimate practical test strength depending on fault type.
If the bottleneck is truly test design, then improving agents’ ability to generate combinatorial input spaces or to run systematic fault injection during test writing could close the gap.
The method of deriving specifications from test suites sacrifices realism for precision; real-world product requirements are more ambiguous, likely making the problem even harder than TestSuiteBench indicates.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73% are killed by exactly one reference test and, on average, by only 0.2% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60%, despite their suites being largely valid and 88% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-12
The authors' main claim, as the classifier picked it out 0.85
“These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.58 · states limitation 0.43 · human participants 0.04 · controlled comparison 0.44 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.51 · mentions code or data availability 0.12
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Coding agents are increasingly used to create large software projects. | background 1.00 | 0.04 | 0.01 | asserted 1.00 |
| While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. | background 0.70 | 0.07 | 0.03 | asserted 1.00 |
| In particular, generating complete test suites for complex software from scratch remains unexplored. | gap 1.00 | 0.06 | 0.03 | asserted 1.00 |
| To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. | contribution 0.95 | 0.94 | 0.04 | asserted 1.00 |
| Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73\% are killed by exactly one reference test and, on average, by only 0.2\% of the remaining tests. | method 0.78 | 0.92 | 0.04 | asserted 1.00 |
| We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60\%, despite their suites being largely valid and 88\% of their tests passing on the reference implementation. | result 1.00 | 0.97 | 0.52 | asserted 1.00 |
| We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. | result 1.00 | 0.92 | 0.05 | asserted 0.97 |
| Meanwhile, implementations generated for the same specifications pass 82\% of the reference tests. | result 1.00 | 0.92 | 0.06 | asserted 1.00 |
| These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development. | interpretation 1.00 | 0.77 | 0.87 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-12: 1 request, 6,870 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (14)
Figures are the authors', reproduced from the paper.
References (23)
- [1] Anthropic . Claude Opus 5 System Card . https://www.anthropic.com/system-cards, July 2026 on Pith
- [2] Nicholas Carlini. Building a c compiler with a team of parallel claudes, 2 2026. URL https://www.anthropic.com/engineering/building-c-compiler on Pith
- [3] DeepSeek-AI , Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026 on PitharXiv
- [4] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah J... on PithPith's reviewarXiv
- [5] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents. arXiv preprint arXiv:2512.12730, 2025 on PitharXiv
- [6] Google DeepMind . Gemini 3.8 Flash Model Card . https://deepmind.google/models/model-cards/gemini-3-8-flash/, September 2026 on Pith
- [7] Ruanqianqian Huang, Avery Reyna, Sorin Lerner, Haijun Xia, and Brian Hempel. Professional software developers don't vibe, they control: Ai agent use for coding in 2025, 2026. URL https://arxiv.org/abs/2512.14012 on PithPith's reviewarXiv
- [8] Kush Jain, Gabriel Synnaeve, and Baptiste Rozi \`e re. Testgeneval: A real world unit test generation and test completion benchmark. In International Conference on Learning Representations, volume 2025, pp.\ 14947--14999, 2025 on Pith
- [9] Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, 37 0 (5): 0 649--678, 2011 on Pith
- [10] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pp.\ 54107--54157, 2024 on Pith
- [11] Leon Kogler, Stefan Hangler, Maximilian Ehrhart, Benedikt Dornauer, Roland Wuersching, and Peter Schrammel. Restestbench: A benchmark for evaluating the effectiveness of llm-generated rest api test cases from nl requirements. arXiv preprint arXiv:2604.25862, 2026 on PithPith's reviewarXiv
- [12] Wilson Lin. Scaling long-running autonomous coding, January 2026. URL https://cursor.com/blog/scaling-agents on Pith
- [13] Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch, 2025. URL https://arxiv.org/abs/2505.03733 on PithPith's reviewarXiv
- [14] Niels M \"u ndler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. SWT -bench: Testing and validating real-world bug-fixes with code agents. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=9Y8zUO11EQ on Pith
- [15] Ahmad Mustafa, Wan MN Wan-Kadir, Noraini Ibrahim, Muhammad Arif Shah, Muhammad Younas, Atif Khan, Mahdi Zareei, and Faisal Alanazi. Automated test case generation from requirements: A systematic literature review. Computers, Materials and Continua, 67 0 (2): 0 1819--1833, 2021 on Pith
- [16] Christian Mürtz and Mark Niklas Müller. Agents in the wild - dashboard. https://insights.logicstar.ai, 2025. URL https://doi.org/10.5281/zenodo.15846865. Interactive web dashboard. Code available at https://github.com/logic-star-ai/insights on Pithdoi:10.5281/zenodo.15846865
- [17] OpenAI . GPT-5.6 System Card . https://deploymentsafety.openai.com/gpt-5-6, July 2026 a on Pith
- [18] OpenAI . GPT-6 Astra System Card . https://deploymentsafety.openai.com/gpt-6-astra, September 2026 b on Pith
- [19] Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang, and Zhenya Huang. Swe-mutation: Can llms generate reliable test suites in software engineering? In Findings of the Association for Computational Linguistics: ACL 2026, pp.\ 39651--39674, 2026 on Pith
- [20] Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, and Martin Vechev. Baxbench: Can llms generate correct and secure backends?, 2025. URL https://arxiv.org/abs/2502.11844 on PithPith's reviewarXiv
show 3 more
- [21] Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S Yu, and Chen Xing. Multifiletest: A multi-file-level llm unit test generation benchmark and impact of error fixing mechanisms. In Findings of the Association for Computational Linguistics:... on Pith
- [22] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793 on PithPith's reviewarXiv
- [23] John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. Programbench: Can language models rebuild programs from scratch? arXiv preprint arXiv:2605.03546, 2026 on PithPith's reviewarXiv
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment