Pith. sign in

REVIEW 4 cited by

ASTER: Natural and Multi-language Unit Test Generation with LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.03093 v3 pith:KMV3QGVF submitted 2024-09-04 cs.SE

Rangeet Pan , Myeongsoo Kim , Rahul Krishna , Raju Pavuluri , Saurabh Sinha This is my paper · ORCID
classification cs.SE
keywords testtestsdevelopersgeneratedgenerationllmsunitanalysis
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Implementing automated unit tests is an important but time-consuming activity in software development. To assist developers in this task, many techniques for automating unit test generation have been developed. However, despite this effort, usable tools exist for very few programming languages. Moreover, studies have found that automatically generated tests suffer poor readability and do not resemble developer-written tests. In this work, we present a rigorous investigation of how large language models (LLMs) can help bridge the gap. We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases. We illustrate how the pipeline can be applied to different programming languages, specifically Java and Python, and to complex software requiring environment mocking. We conducted an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness -- evaluating them on standard as well as enterprise Java applications and a large Python benchmark. Our results demonstrate that LLM-based test generation, when guided by static analysis, can be competitive with, and even outperform, state-of-the-art test-generation techniques in coverage achieved while also producing considerably more natural test cases that developers find easy to understand. We also present the results of a user study, conducted with 161 professional developers, that highlights the naturalness characteristics of the tests generated by our approach.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Benchmarking LLMs for Unit Test Generation from Real-World Functions

    cs.SE 2025-08 conditional novelty 6.0 of 10

    A new decontaminated benchmark of complex Python functions shows LLMs generate far fewer correct, covering, and bug-killing unit tests than on older benchmarks.

  2. Detec\c{c}\~ao de Conflitos Sem\^anticos com Testes Gerados por LLM

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Code Llama 70B generated unit tests that detected five semantic merge conflicts, including one that earlier SMAT tools missed, but only after extensive and slow test generation.

  3. SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows

    cs.CR 2025-06 conditional novelty 6.0 of 10

    An agentic workflow with security guidelines and LLM-generated unit tests improves secure code generation on CWEval C tasks from 61% to 76% Func-Sec@1 with Sonnet-3.7, at roughly 98% of its original functionality.

  4. How well LLM-based test generation techniques perform with newer LLM versions?

    cs.SE 2026-01 conditional novelty 5.0 of 10

    With newer LLMs, a plainly prompted generation loop matches or beats four engineered test-generation tools on coverage and mutation score, and a class-then-method hybrid cuts LLM queries by about 20%.

Pith tools