Pith.

Software Engineering · arXiv:2610.13065 v1 · submitted 9 Oct 2026 · 23 references

TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications

Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller, Martin Vechev

The paper says

“Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes.”

more of the abstract

In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73% are killed by exactly one reference test and, on average, by only 0.2% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60%, despite their suites being largely valid and 88% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 12 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Solid benchmark, real finding, but mutant construction bias deserves a quantitative sensitivity check before the central claim is fully secure. This line is wrong

1 major objection · 1 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 23 references say.

The paper says, at §2.2, §5, §A.5

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.92):

“Further, selecting mutants that are killed by a single reference test risks favoring mutants triggered only by that test’s exact inputs.”

Pith says: Objection 1 of 1 · major

The paper acknowledges that constructing mutants killed by exactly one reference test (73% of mutants) risks favoring mutants whose failure depends on the exact input structure of that test, rather than on the underlying requirement violation. The mitigation (logic-level mutation instruction, manual audit, Figure 13) is qualitative.

more of objection 1

Since the data to compute the mutation score restricted to mutants killed by more than one reference test (the remaining 27%) are available, the paper should report this restricted score. If the score rises substantially (e.g., above 70%), the central claim that agents 'cannot verify' their implementations would be weakened; if it stays below 60%, the claim is strengthened. A quantitative check is needed to resolve the residual uncertainty this acknowledged limitation creates.

1 minor objection

The paper says, at Table 2

Pith says: minor objection 1 of 1

The column header "MV$t(min)" contains a stray LaTeX dollar sign; it should read "M V t (min)" or "M V Time (min)".

What would settle it

For each of the 165 instances, compute the mutation score of the Opus and Sol test suites using only the subset of mutants that are killed by more than one reference test (the ~27% not singletone-kill). If on this subset the mutation score is substantially higher (e.g., >20 percentage points) than on the full set, then the singletone-kill filtering is inflating the apparent weakness, and the central claim requires circumscription. If the score is comparable or only modestly higher, the concern is assuaged.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 12 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 31 s · 26,854 tokens read, 3,563 written
  3. Editor wrote the verdict above 14 s · 27,694 tokens read, 1,535 written
  4. Referee minor revision: 1 major objection, 1 minor 30 s · 27,964 tokens read, 810 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is a well-built benchmark and a genuinely useful paper. The key contribution is a scalable pipeline for turning existing test suites into detailed natural-language specifications with targeted mutants, then measuring whether agents can write strong test suites from scratch. The iterative specification refinement, manual quality control, and transparent limitation statements are good practice. The empirical result — that frontier agents can implement specifications they cannot test — is consistently observed across six models and holds even when the reference executable is provided, which pins the bottleneck on test design rather than specification clarity.

The soft spot is the one the stress-test note flags. 73% of mutants are designed to be killed by exactly one reference test. The paper acknowledges the risk (Section 5) and mitigates it by instructing the mutation agent to mutate logic rather than hard-code input values, with an ablation in Appendix A.5 showing that this reduces literal copying. But that ablation checks generalisation to renamed inputs, not the effect on the overall mutation score. The paper does not report what happens when you restrict the evaluation to the 27% of mutants that are killed by more than one reference test. If that subset’s mutation score is substantially higher, the central claim that agents 'cannot verify' the software they build would need qualification. This is a measurable, incremental check that would strengthen the paper’s core argument, and its absence is the biggest residual uncertainty.

Other concerns are minor. The mutant set samples requirements uniformly, so it does not penalise gaps in untargeted requirements; the paper acknowledges this and argues that agents cannot know which requirements are targeted, which is fair but means the benchmark measures coverage indirectly. The scope (CLI tools in compiled languages) is inherited from ProgramBench and is appropriate for a first large-scale benchmark.

Overall, this is a serious piece of work that will be cited and used. The finding is important even if the exact magnitude of the gap depends on how you construct the mutants. I would bring it to a reading group and would cite it.

Recommendation: Accept with minor revision. Ask the authors to compute and report mutation score on the subset of mutants killed by >1 reference test, and to discuss whether the 73% singleton set introduces an upper bound on achievable mutation score that is inherent to the benchmark design.

The strongest objection

The central claim that agents 'cannot verify' the software they build rests heavily on the mutation score metric: even the best agent kills only 55.5% of mutants. However, 73% of mutants are designed to be killed by exactly one reference test (Section 2.2). The instructions for mutant generation (prompt D.1, 'Mutant-polishing prompt') explicitly ask for mutants with the singleton kill set, even noting that 'an exact singleton kill set is desirable, but it is not required.' This creates a subtle bias: a mutant that is killed by only one reference test is triggered only when the agent test happens to match that particular test's exact input structure and assertion. The paper shows that agents often miss input combinations (Figure 7, 14); this could mean they miss these narrow mutants because the reference test's specific inputs are hard to replicate, not because the agents fail to detect violations of the requirement in principle. If the evaluation were restricted to the 27% of mutants that are killed by more than one reference test, the mutation score might rise substantially, weakening the 'cannot verify' claim. The paper acknowledges this risk in Section 5 ('selecting mutants that are killed by a single reference test risks favoring mutants triggered only by that test's exact inputs'), but does not provide a quantitative check, leaving a residual uncertainty about how much of the observed gap is an artifact of the mutant construction.

The paper is thorough and its main result is well-supported. The reader's weakest assumption (mutants as proxy for real faults) is spot-on, but there is a more specific technical dimension: 73% of mutants are designed to be killed by exactly one reference test. This design choice could make mutants atypically narrow, biasing the mutation score downward. The paper acknowledges the general risk but does not quantify its impact. A simple post-hoc check on the non-singleton subset would discipline whether the gap is real or an artifact. Even if the check shows a large difference, the paper's finding would still hold (agents miss many mutants), but the magnitude of the claim would need adjustment. I therefore agree with the reader's verdict (ACCEPT) and see no need to change it, but flag this specific untested assumption as the most load-bearing point.

The referee's report

If the result holds, this is an important contribution: it identifies a concrete, measurable capability gap in current coding agents—test generation from specifications lags far behind implementation generation. The benchmark is carefully constructed (iterative spec refinement, multi-stage mutant construction with manual audit, flaky-test filtering), and the main finding—agents generate valid but weak tests even when given the exact specification or the reference executable—is striking and practically relevant. TestSuiteBench fills a genuine gap in the evaluation landscape (repository-scale test generation from specs) and the detailed construction pipeline will enable the community to track progress on this critical capability.

Pith's reading of the paper (a machine's summary, not the authors')

This paper introduces TestSuiteBench, a benchmark of 165 open-source applications designed to test whether coding agents can write strong test suites from natural-language specifications alone. The central discovery is a deep gap: frontier agents produce valid implementations that pass 82–93% of reference tests, yet their own test suites kill at most 55% of targeted mutants—and four out of six agents score below 40%. This failure is not due to unclear specifications (giving agents the reference executable leaves scores unchanged) but to weak test design: generated suites are too small and miss input combinations. The results show that agents can build software they cannot verify, a bottleneck that limits fully autonomous development.

What carries the argument. TestSuiteBench is the central benchmark: 165 command-line applications in Rust, Go, and C++ with a structured specification (median 5,635 words, 23 sections), a reference test suite (median 679 functions), and a set of mutants (average 74 per application). Each mutant is constructed to violate a single specification requirement and refined so that 73% are killed by exactly one reference test and only 0.2% of unrelated tests. The evaluation uses mutation score M (fraction of mutants killed by valid generated tests) and validity score V (fraction of generated tests that pass the reference executable).

Core claim. The paper establishes that even state-of-the-art coding agents (Claude Opus 5, GPT 6 Astra, etc.) cannot generate strong test suites from detailed natural-language specifications alone. Despite producing valid tests (88–96% pass on the reference implementation) and correct implementations (86–93% pass reference tests), their test suites achieve mutation scores below 60% when evaluated against mutants that each violate exactly one specification requirement. The gap persists even when the reference executable is provided, showing the bottleneck is test design—not specification ambiguity.

The premise it rests on. The mutants, each designed to violate one specification requirement and refined to be killed by exactly one reference test, are a good proxy for real-world faults that a test suite should catch.

If the paper is right (Pith's inferences, not the authors')

Autonomous software development cannot rely on current agents for verification: they must either produce stronger test suites or use external verification methods.

Test design is a distinct skill from implementation: the same agents that pass reference tests fail to design input combinations that expose violations, suggesting a missing reasoning capability.

TestSuiteBench provides a concrete metric for progress: raising mutation scores above 60% would represent a step toward self-verifying agents.

The finding that suites are too small and miss edge cases implies that agents need better exploration strategies, not just better specification understanding.

Ensemble approaches (combining suites from multiple agents) give only modest gains (59% vs 55% for Opus alone), leaving most mutants untouched.

The low mutation scores even when the reference executable is available suggest that agents lack a systematic strategy for designing adversarial inputs—they may treat test generation as similar to implementation rather than as a search for counterexamples.

The benchmark’s reliance on single-requirement mutants, as acknowledged, may not reflect real-world fault distributions, so the reported scores might either overestimate or underestimate practical test strength depending on fault type.

If the bottleneck is truly test design, then improving agents’ ability to generate combinatorial input spaces or to run systematic fault injection during test writing could close the gap.

The method of deriving specifications from test suites sacrifices realism for precision; real-world product requirements are more ambiguous, likely making the problem even harder than TestSuiteBench indicates.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73% are killed by exactly one reference test and, on average, by only 0.2% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60%, despite their suites being largely valid and 88% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-12

The authors' main claim, as the classifier picked it out 0.85

“These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.58 · states limitation 0.43 · human participants 0.04 · controlled comparison 0.44 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.51 · mentions code or data availability 0.12

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Coding agents are increasingly used to create large software projects. background 1.00 0.04 0.01 asserted 1.00
While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. background 0.70 0.07 0.03 asserted 1.00
In particular, generating complete test suites for complex software from scratch remains unexplored. gap 1.00 0.06 0.03 asserted 1.00
To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. contribution 0.95 0.94 0.04 asserted 1.00
Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73\% are killed by exactly one reference test and, on average, by only 0.2\% of the remaining tests. method 0.78 0.92 0.04 asserted 1.00
We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60\%, despite their suites being largely valid and 88\% of their tests passing on the reference implementation. result 1.00 0.97 0.52 asserted 1.00
We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. result 1.00 0.92 0.05 asserted 0.97
Meanwhile, implementations generated for the same specifications pass 82\% of the reference tests. result 1.00 0.92 0.06 asserted 1.00
These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development. interpretation 1.00 0.77 0.87 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-12: 1 request, 6,870 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (14)
Figure 1 of the paper
Figure 1. TESTSUITEBENCH: One instance consists of a specification S, a test suite T asserting its requirements, and a set of mutants M that break these requirements in targeted ways. An agent is tasked with generating a full test suite based solely on the natural-language specification S. We then evaluate the Validity Score V, the fraction of tests that pass on the reference implementation, and the Mutation Score M, the fraction of mutants killed by the generated test suite. We derive each specification from the application’s reference test suite, so that it describes every tested behavior,…
Figure 2 of the paper
Figure 2. (Top) To construct TESTSUITEBENCH, we first obtain a reference test suites T ∗ , imple￾mentation I ∗ and its executable E∗ from a GitHub repository R. We then create a draft specification S ′ , and mutant candidates M that violate individual requirements, before refining the mutants to reduce unrelated failures against T ∗ . We then refine the specification S ′ by iteratively generating implementations and test suites from it, analysing resulting errors, and addressing the ones attributed to the specification. (Bottom) In the space of implementations, we first define the set of…
Figure 3 of the paper
Figure 3. The iterative specification refine￾ment increases the implementation’s refer￾ence pass rate and test’s validity rate, while mutation score remains low. Candidate Refinement For every mutant candi￾date, we determine the test functions that kill it and rank the 6 mutants mj associated with each si by (i) whether mj is killed by tj,target and (ii) the size of T¯ j . We select the top three mutants by these criteria. Then, we use a coding agent to inspect these mutant candidates and the test cases that kill it, and refine them to reduce T¯ j where needed. Finally, we run the reference
Figure 4 of the paper
Figure 4. Implementation lan￾guages of TESTSUITEBENCH in￾stances. Our final benchmark consists of 165 instances, spanning Rust, Go, and C/C++ and a diverse set of domains including text processing, file and disk utilities, and build tools. We show key statistics in
Figure 5 of the paper
Figure 5. Increased cost provides marginal gains for M. 3.1 MAIN RESULTS Agents Struggle to Generate Strong Tests. We compare different coding agents across test and program generation in
Figure 6 of the paper
Figure 6. Left: Opus generates more tests than Sol with stronger correlation to the reference test count. Right: Both models struggle to generate strong test suites for larger specifications. For `run`, multiple file and directory paths are accepted. It returns all matching files. For `run`, multiple file and directory paths are accepted. It returns all matching files. s $ ./exec run 'f()' −f a.js b.js > a.js # mutant > a.js b.js # reference m def test_run_accepts_multiple_paths(): p = run_exe("run 'f()' −f a.js b.js") assert p.stdout == "a.js b.js" Opus t def test_files_with_matches(): p =…
Figure 7 of the paper
Figure 7. The specification requires run to accept multiple paths. The mutant ignores the second path, b.js. The Opus test checks both paths and kills the mutant. In contrast, the Sol test only checks one path and misses the mutation. Complete version of error analysis is provided in Section A.6. Agents Struggle with Large Specifications In Figure 6b (full results in Section A.2), we visualize the relationship between the number of specification sections and test generation performance (M). Across all models, we observe a negative correlation (Opus: ρ = −0.16, Sol: ρ = −0.42), indicating that…
Figure 8 of the paper
Figure 8. Correlation between number of reference and generated test functions.
Figure 9 of the paper
Figure 9. Correlation between the number of spec section and mutation score.
Figure 10 of the paper
Figure 10. Cumulative distribution of scores and costs across models.
Figure 11 of the paper
Figure 11. Distribution of tool use across agent trajectories.
Figure 12 of the paper
Figure 12. Overlap in mutants killed by test suites generated by different models.
Figure 13 of the paper
Figure 13. Mutants generated with and without the logic-level instruction. Each panel shows the target reference test (t) and the behavior of the two mutants (m) generated for it, on the test’s input and on the same input with its literals changed. In both panels, both mutants are killed by the target test and by no other reference test. Literals from the test that the mutant without the instruction hard-codes are highlighted in red; changing them makes this mutant behave like the reference. B BENCHMARK CONSTRUCTION DETAIL B.1 DETAILS ON FILTERING As briefly described in Section 2.1, we…
Figure 14 of the paper
Figure 14. Error analysis on ast-grep and cmatrix. Each panel shows the specification (s), the mutant (m), and the relevant Opus and Sol tests (t). A crossed-out mutant icon marks a test that kills the mutant. Opus kills the mutants in (a) and (b); both suites miss those in (c) and (d). 16

Figures are the authors', reproduced from the paper.

References (23)
  1. [1] Anthropic . Claude Opus 5 System Card . https://www.anthropic.com/system-cards, July 2026 on Pith
  2. [2] Nicholas Carlini. Building a c compiler with a team of parallel claudes, 2 2026. URL https://www.anthropic.com/engineering/building-c-compiler on Pith
  3. [3] DeepSeek-AI , Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026 on PitharXiv
  4. [4] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah J... on PithPith's reviewarXiv
  5. [5] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al. Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents. arXiv preprint arXiv:2512.12730, 2025 on PitharXiv
  6. [6] Google DeepMind . Gemini 3.8 Flash Model Card . https://deepmind.google/models/model-cards/gemini-3-8-flash/, September 2026 on Pith
  7. [7] Ruanqianqian Huang, Avery Reyna, Sorin Lerner, Haijun Xia, and Brian Hempel. Professional software developers don't vibe, they control: Ai agent use for coding in 2025, 2026. URL https://arxiv.org/abs/2512.14012 on PithPith's reviewarXiv
  8. [8] Kush Jain, Gabriel Synnaeve, and Baptiste Rozi \`e re. Testgeneval: A real world unit test generation and test completion benchmark. In International Conference on Learning Representations, volume 2025, pp.\ 14947--14999, 2025 on Pith
  9. [9] Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, 37 0 (5): 0 649--678, 2011 on Pith
  10. [10] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pp.\ 54107--54157, 2024 on Pith
  11. [11] Leon Kogler, Stefan Hangler, Maximilian Ehrhart, Benedikt Dornauer, Roland Wuersching, and Peter Schrammel. Restestbench: A benchmark for evaluating the effectiveness of llm-generated rest api test cases from nl requirements. arXiv preprint arXiv:2604.25862, 2026 on PithPith's reviewarXiv
  12. [12] Wilson Lin. Scaling long-running autonomous coding, January 2026. URL https://cursor.com/blog/scaling-agents on Pith
  13. [13] Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch, 2025. URL https://arxiv.org/abs/2505.03733 on PithPith's reviewarXiv
  14. [14] Niels M \"u ndler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. SWT -bench: Testing and validating real-world bug-fixes with code agents. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=9Y8zUO11EQ on Pith
  15. [15] Ahmad Mustafa, Wan MN Wan-Kadir, Noraini Ibrahim, Muhammad Arif Shah, Muhammad Younas, Atif Khan, Mahdi Zareei, and Faisal Alanazi. Automated test case generation from requirements: A systematic literature review. Computers, Materials and Continua, 67 0 (2): 0 1819--1833, 2021 on Pith
  16. [16] Christian Mürtz and Mark Niklas Müller. Agents in the wild - dashboard. https://insights.logicstar.ai, 2025. URL https://doi.org/10.5281/zenodo.15846865. Interactive web dashboard. Code available at https://github.com/logic-star-ai/insights on Pithdoi:10.5281/zenodo.15846865
  17. [17] OpenAI . GPT-5.6 System Card . https://deploymentsafety.openai.com/gpt-5-6, July 2026 a on Pith
  18. [18] OpenAI . GPT-6 Astra System Card . https://deploymentsafety.openai.com/gpt-6-astra, September 2026 b on Pith
  19. [19] Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang, and Zhenya Huang. Swe-mutation: Can llms generate reliable test suites in software engineering? In Findings of the Association for Computational Linguistics: ACL 2026, pp.\ 39651--39674, 2026 on Pith
  20. [20] Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, and Martin Vechev. Baxbench: Can llms generate correct and secure backends?, 2025. URL https://arxiv.org/abs/2502.11844 on PithPith's reviewarXiv
show 3 more
  1. [21] Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S Yu, and Chen Xing. Multifiletest: A multi-file-level llm unit test generation benchmark and impact of error fixing mechanisms. In Findings of the Association for Computational Linguistics:... on Pith
  2. [22] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793 on PithPith's reviewarXiv
  3. [23] John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. Programbench: Can language models rebuild programs from scratch? arXiv preprint arXiv:2605.03546, 2026 on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment