LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.
Do LLMs generate test oracles that capture the actual or the expected program behaviour?
5 Pith papers cite this work, alongside 2 external citations. Polarity classification is still indexing.
5
Pith papers citing it
2
external citations · Pith
abstract
Software testing is an essential part of the software development cycle to improve the code quality. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. A known limitation of traditional test generation techniques (e.g. Randoop and Evosuite) is that they produce test oracles that capture the actual program behaviour rather than the expected one. Recent approaches leverage Large Language Models (LLMs), trained on an enormous amount of data, to generate developer-like code and test cases. We investigate whether the LLM-generated test oracles capture the actual or expected software behaviour. We thus, conduct a controlled experiment to answer this question, by studying LLMs performance on two tasks, namely, test oracle classification and generation. The study includes developer-written and automatically generated test cases and oracles for 24 open-source Java repositories, and different well tested prompts. Our findings show that LLM-based test generation approaches are also prone on generating oracles that capture the actual program behaviour rather than the expected one. Moreover, LLMs are better at generating test oracles rather than classifying the correct ones, and can generate better test oracles when the code contains meaningful test or variable names. Finally, LLM-generated test oracles have higher fault detection potential than the Evosuite ones.
citation-role summary
background 1
citation-polarity summary
fields
cs.SE 5years
2026 5roles
background 1polarities
background 1representative citing papers
Propilot instantiates 20 tensor-algebra property skeletons into 4,579 executable PBTs for TVM, cutting redundancy 49% and surfacing semantic and numerical errors.
RESTestBench shows that LLM-generated REST API test effectiveness drops when interacting with faulty or mutated code, especially for vague requirements, indicating that high-detail requirements make direct SUT interaction unnecessary.
An empirical study of 86,156 test patches from five AI agents finds 80.2% lack strong oracle signals, with strong oracles linked to higher merge rates (OR=1.28) after regression controls.
TRAILS infers code correctness by aggregating LLM judgments on input-output pairs from category-partitioned specification tests, improving MCC by up to 39% over Zero-Shot COT on LiveCodeBench and CoCoClaNeL.
citing papers explorer
-
LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle
LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.
-
Tensor Algebraic Property Skeletons: Amplifying Property-Based Testing for AI Compilers
Propilot instantiates 20 tensor-algebra property skeletons into 4,579 executable PBTs for TVM, cutting redundancy 49% and surfacing semantic and numerical errors.
-
RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements
RESTestBench shows that LLM-generated REST API test effectiveness drops when interacting with faulty or mutated code, especially for vague requirements, indicating that high-detail requirements make direct SUT interaction unnecessary.
-
All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code
An empirical study of 86,156 test patches from five AI agents finds 80.2% lack strong oracle signals, with strong oracles linked to higher merge rates (OR=1.28) after regression controls.
-
Inferring Code Correctness from Specification
TRAILS infers code correctness by aggregating LLM judgments on input-output pairs from category-partitioned specification tests, improving MCC by up to 39% over Zero-Shot COT on LiveCodeBench and CoCoClaNeL.