REVIEW 5 cited by
On the Evaluation of Large Language Models in Unit Test Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Unit testing is an essential activity in software development for verifying the correctness of software components. However, manually writing unit tests is challenging and time-consuming. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation. Existing research primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed prompting strategies, leaving the capabilities of advanced open-source LLMs with various prompting settings unexplored. Particularly, open-source LLMs offer advantages in data privacy protection and have demonstrated superior performance in some tasks. Moreover, effective prompting is crucial for maximizing LLMs' capabilities. In this paper, we conduct the first empirical study to fill this gap, based on 17 Java projects, five widely-used open-source LLMs with different structures and parameter sizes, and comprehensive evaluation metrics. Our findings highlight the significant influence of various prompt factors, show the performance of open-source LLMs compared to the commercial GPT-4 and the traditional Evosuite, and identify limitations in LLM-based unit test generation. We then derive a series of implications from our study to guide future research and practical use of LLM-based unit test generation.
Forward citations
Cited by 5 Pith papers
-
Knowledge-Guided Synthetic Bug Feedback for LLM-Based Unit Test Generation
Mechanism-guided synthetic-bug feedback raises real-bug detection of LLM unit tests to 72.67% RBDR on 172 Defects4J tasks, outperforming mutation and coverage baselines by ~20 points.
-
Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation
CATGen improves LLM unit-test reliability by combining structured project-context retrieval, deterministic test-class skeletons, and static-analysis repair, beating six baselines on compilation success, coverage, and cost.
-
On the risk of coding before testing: An empirical study on LLM-based test generation workflow
Generating LLM tests after faulty LLM code cuts fault detection to about 14% versus about 25% when tests are generated from the task description alone, across models and prompting strategies.
-
ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts
ViseGPT automatically converts user prompts into test cases and visualizes which steps of an LLM-generated data wrangling script pass or fail.
-
Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models
On a custom Python shopping-cart benchmark, docstrings substantially improve LLM-generated unit tests, full implementations add little, and chain-of-thought prompting yields the highest branch and mutation scores.
Discussion (0). Sign in to comment.