Pith. sign in

REVIEW 9 cited by

TESTEVAL: Benchmarking Large Language Models for Test Case Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.04531 v2 pith:APGIE3RX submitted 2024-06-06 cs.SE

Wenhan Wang , Chenyuan Yang , Zhijie Wang , Yuheng Huang , Zhaoyang Chu , Da Song , Lingming Zhang , An Ran Chen
show 1 more author
Lei Ma
This is my paper
classification cs.SE
keywords llmstesttestevalcasecoveragegenerationprogramsoftware
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Testing plays a crucial role in the software development cycle, enabling the detection of bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers need to write code snippets that execute the program under test. Recently, researchers have recognized the potential of large language models (LLMs) in software testing. However, there remains a lack of fair comparisons between different LLMs in terms of test case generation capabilities. In this paper, we propose TESTEVAL, a novel benchmark for test case generation with LLMs. We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage. We further evaluate sixteen popular LLMs, including both commercial and open-source ones, on TESTEVAL. We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs, indicating a lack of ability to comprehend program logic and execution paths. We have open-sourced our dataset and benchmark pipelines at https://github.com/LLM4SoftwareTesting/TestEval.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation

    cs.SE 2026-07 conditional novelty 6.5 of 10

    A contextual-bandit supervisor learns to route coding agents among default generation, analysis-augmented generation, and stop, yielding large coverage gains over unsupervised agents and non-agentic LLM test generators.

  2. Benchmarking LLMs for Unit Test Generation from Real-World Functions

    cs.SE 2025-08 conditional novelty 6.0 of 10

    A new decontaminated benchmark of complex Python functions shows LLMs generate far fewer correct, covering, and bug-killing unit tests than on older benchmarks.

  3. Rethinking Verification for LLM Code Generation: From Generation to Testing

    cs.CL 2025-07 reject novelty 6.0 of 10

    SAGA, a human-LLM workflow, generates tests that catch 90.62% of known human bugs on a new benchmark, though those bugs also helped create the tests.

  4. Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

    cs.SE 2025-06 conditional novelty 6.0 of 10

    TestCase-Eval introduces fault coverage and fault exposure tasks over 500 Codeforces problems and finds that 19 LLMs, led by Qwen3-32B, remain far below human expert performance.

  5. SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SAGE combines supervised fine-tuning and GRPO reinforcement learning to generate CCFG grammars from competitive programming specs, achieving 96.66% set-based validity and 80.67% set-based effectiveness on 240 held-out...

  6. HardTests: Synthesizing High-Quality Test Cases for LLM Coding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HardTestGen generates higher-precision, higher-recall test suites for 47,136 competitive programming problems, improving test precision by 11.3 points and recall by 17.5 points over TACO and CodeContests when judging ...

  7. Trade Policy and Structural Change

    econ.GN 2025-08 unverdicted novelty 5.0 of 10

    A dynamic multi-country model with nonhomothetic preferences and complementary sectors implies a 20 percentage point U.S. manufacturing tariff raises the manufacturing value-added share by about one percentage point a...

  8. HyClone: Bridging LLM Understanding and Dynamic Execution for Semantic Code Clone Detection

    cs.SE 2025-08 unverdicted novelty 5.0 of 10

    A two-stage LLM-plus-execution framework is claimed to improve semantic code clone detection, but the submitted text does not contain the claimed framework.

  9. Comparative Evaluation of Large Language Models for Test-Skeleton Generation

    cs.SE 2025-09 reject novelty 4.0 of 10

    In a single-prompt study on one Ruby class, DeepSeek-Chat produced the most maintainable RSpec test skeletons, while GPT-4's high method coverage was undercut by incorrect RSpec conventions.

Pith tools