REVIEW 8 cited by
TOGLL: Correct and Strong Test Oracle Generation with LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Test oracles play a crucial role in software testing, enabling effective bug detection. Despite initial promise, neural-based methods for automated test oracle generation often result in a large number of false positives and weaker test oracles. While LLMs have demonstrated impressive effectiveness in various software engineering tasks, including code generation, test case creation, and bug fixing, there remains a notable absence of large-scale studies exploring their effectiveness in test oracle generation. The question of whether LLMs can address the challenges in effective oracle generation is both compelling and requires thorough investigation. In this research, we present the first comprehensive study to investigate the capabilities of LLMs in generating correct, diverse, and strong test oracles capable of effectively identifying a large number of unique bugs. To this end, we fine-tuned seven code LLMs using six distinct prompts on the SF110 dataset. Utilizing the most effective fine-tuned LLM and prompt pair, we introduce TOGLL, a novel LLM-based method for test oracle generation. To investigate the generalizability of TOGLL, we conduct studies on 25 large-scale Java projects. Besides assessing the correctness, we also assess the diversity and strength of the generated oracles. We compare the results against EvoSuite and the state-of-the-art neural method, TOGA. Our findings reveal that TOGLL can produce 3.8 times more correct assertion oracles and 4.9 times more exception oracles. Moreover, our findings demonstrate that TOGLL is capable of generating significantly diverse test oracles. It can detect 1,023 unique bugs that EvoSuite cannot, which is ten times more than what the previous SOTA neural-based method, TOGA, can detect.
Forward citations
Cited by 8 Pith papers
-
COFFE: A Code Efficiency Benchmark for Code Generation
A new benchmark, COFFE, uses stressful test cases and CPU instruction counts to show LLM-generated code is often correct but time-inefficient.
-
Agent-Based Test Assertion Generation via Diverse Perspective Aggregation
AssertMate decomposes assertion generation into actual value construction, multi-perspective expected value prediction, and judge-based selection, and reports higher compilation, pass, and bug-detection rates on Defec...
-
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
LLMs under one-shot greedy decoding enumerate or materialize acceptable sets much worse than they judge membership, a gap that persists across scale, family, and generation and is dominated by omissions.
-
RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning
An RL fine-tuned lightweight LLM generates REST API test oracles from one traffic sample and beats prompt-engineered large models in industrial adoption.
-
On the risk of coding before testing: An empirical study on LLM-based test generation workflow
Generating LLM tests after faulty LLM code cuts fault detection to about 14% versus about 25% when tests are generated from the task description alone, across models and prompting strategies.
-
BeSpec: Behavior-Level Specification Alignment for Code Generation
Behavior-level specification alignment—predicting checkable output properties, probing candidates, and repairing the intent—raises Pass@1 by 8.1–25.3% relative over the strongest baselines across three LLMs.
-
AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model
A fine-tuned CodeT5 model generates exact-match test assertions in up to 59.5% of cases, but detects only 33 of 138 real Defects4J bugs.
-
How well LLM-based test generation techniques perform with newer LLM versions?
With newer LLMs, a plainly prompted generation loop matches or beats four engineered test-generation tools on coverage and mutation score, and a class-then-method hybrid cuts LLM queries by about 20%.
Discussion (0). Continue with ORCID to comment.