REVIEW 17 cited by
CoverUp: Effective High Coverage Test Generation for Python
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Testing is an essential part of software development. Test generation tools attempt to automate the otherwise labor-intensive task of test creation, but generating high-coverage tests remains challenging. This paper proposes CoverUp, a novel approach to driving the generation of high-coverage Python regression tests. CoverUp combines coverage analysis, code context, and feedback in prompts that iteratively guide the LLM to generate tests that improve line and branch coverage. We evaluate our prototype CoverUp implementation across a benchmark of challenging code derived from open-source Python projects and show that CoverUp substantially improves on the state of the art. Compared to CodaMosa, a hybrid search/LLM-based test generator, CoverUp achieves a per-module median line+branch coverage of 80% (vs. 47%). Compared to MuTAP, a mutation- and LLM-based test generator, CoverUp achieves an overall line+branch coverage of 89% (vs. 77%). We also demonstrate that CoverUp's performance stems not only from the LLM used but from the combined effectiveness of its components.
Forward citations
Cited by 17 Pith papers
-
PROGRESS: Property-Guided Regression Search for Semantic Falsification
PROGRESS embeds LLM-generated jqwik properties as search objectives in EvoSuite's DynaMOSA, detecting 58% of injected Java bugs that regression assertions miss and satisfying hard preconditions 4x more often than stan...
-
Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions
Sakura is a multi-agent system that generates structurally complex tests from NL descriptions, achieving 50-78% higher compilability and 38-66% higher coverage overlap than baselines on 1,464 scenarios from 20 Apache ...
-
Evaluating LLM-Based Regression Test Generation
A feedback-driven LLM tool generates regression tests for commits in minutes and, when its outputs seed a fuzzer, finds more bugs than a 24-hour state-of-the-art fuzzer on the same benchmark.
-
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
An LLM agent that reads a repository, gathers setup hints, and iteratively runs commands can build and test 33 of 50 popular projects across 14 languages.
-
From Resource Flow to Executable Tests: Petri-Net-Guided LLM Test Generation for Concurrent Stateful Rust APIs
Petri-net models of Rust APIs can guide LLMs to produce executable concurrent tests whose structure stays faithful to the modeled resource flow.
-
Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows
A three-agent LLM workflow plus a test-specialized knowledge graph generates unit tests that beat prior LLM and search-based tools on coverage and mutation score.
-
On the risk of coding before testing: An empirical study on LLM-based test generation workflow
Generating LLM tests after faulty LLM code cuts fault detection to about 14% versus about 25% when tests are generated from the task description alone, across models and prompting strategies.
-
Hamster: A Large-Scale Study and Characterization of Developer-Written Tests
A large-scale study of 1.7M developer-written Java tests shows they commonly use fixtures, mocking, structured inputs, and multi-class scope, features largely missing from automated test generation tools.
-
YATE: The Role of Test Repair in LLM-Based Unit Test Generation
A test-repair pipeline, combining static analysis and re-prompting, raises LLM-generated unit test coverage and mutation killing by roughly 20-30 percent over a plain prompt baseline on six Java projects.
-
Combining Type Inference and Automated Unit Test Generation for Python
A proxy-based type tracing technique integrated into Pynguin infers parameter and return types during test generation and improves branch coverage.
-
Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
This paper formally defines hardening and catching tests, introduces the Catching Just-in-Time Test (JiTTest) Challenge for LLM-generated software tests, and argues it is the most impactful open problem in automated t...
-
Design choices made by LLM-based test generators prevent them from finding bugs
Coverage-guided LLM test generators systematically filter out bug-revealing tests and can end up validating bugs instead of finding them.
-
How well LLM-based test generation techniques perform with newer LLM versions?
With newer LLMs, a plainly prompted generation loop matches or beats four engineered test-generation tools on coverage and mutation score, and a class-then-method hybrid cuts LLM queries by about 20%.
-
Large Language Models for Unit Testing: A Systematic Literature Review
The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.
-
Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts
A BERTopic analysis of 8,593 Stack Overflow posts and 26,474 OpenAI Developer Forum posts yields 9 and 17 LLM developer challenge topics, with API usage dominant and high unresolved rates.
-
LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities
A survey of LLM-based multi-agent systems across the software development life cycle, plus a research agenda for orchestration, human coordination, cost, and data.
-
A Contemporary Survey of Large Language Model Assisted Program Analysis
A review that catalogs how large language models are used in static, dynamic, and hybrid program analysis, and outlines open challenges.
Discussion (0). Continue with ORCID to comment.