Pith

open record

sign in

arxiv: 2410.21136 · v1 · pith:YUAJCOJ6 · submitted 2024-10-28 · cs.SE

Do LLMs generate test oracles that capture the actual or the expected program behaviour?

Michael Konstantinou , Renzo Degiovanni , Mike Papadakis This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:YUAJCOJ6record.jsonopen to challenge →

classification cs.SE
keywords testoraclesbehaviouractualcaptureexpectedllmscode
0
0 comments X
read the original abstract

Software testing is an essential part of the software development cycle to improve the code quality. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. A known limitation of traditional test generation techniques (e.g. Randoop and Evosuite) is that they produce test oracles that capture the actual program behaviour rather than the expected one. Recent approaches leverage Large Language Models (LLMs), trained on an enormous amount of data, to generate developer-like code and test cases. We investigate whether the LLM-generated test oracles capture the actual or expected software behaviour. We thus, conduct a controlled experiment to answer this question, by studying LLMs performance on two tasks, namely, test oracle classification and generation. The study includes developer-written and automatically generated test cases and oracles for 24 open-source Java repositories, and different well tested prompts. Our findings show that LLM-based test generation approaches are also prone on generating oracles that capture the actual program behaviour rather than the expected one. Moreover, LLMs are better at generating test oracles rather than classifying the correct ones, and can generate better test oracles when the code contains meaningful test or variable names. Finally, LLM-generated test oracles have higher fault detection potential than the Evosuite ones.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle

    cs.SE 2026-07 conditional novelty 7.0

    LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.

  2. Tensor Algebraic Property Skeletons: Amplifying Property-Based Testing for AI Compilers

    cs.SE 2026-06 unverdicted novelty 7.0

    Propilot instantiates 20 tensor-algebra property skeletons into 4,579 executable PBTs for TVM, cutting redundancy 49% and surfacing semantic and numerical errors.

  3. RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements

    cs.SE 2026-04 unverdicted novelty 7.0

    RESTestBench shows that LLM-generated REST API test effectiveness drops when interacting with faulty or mutated code, especially for vague requirements, indicating that high-detail requirements make direct SUT interac...

  4. All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code

    cs.SE 2026-06 unverdicted novelty 6.0

    An empirical study of 86,156 test patches from five AI agents finds 80.2% lack strong oracle signals, with strong oracles linked to higher merge rates (OR=1.28) after regression controls.

  5. Inferring Code Correctness from Specification

    cs.SE 2026-05 unverdicted novelty 6.0

    TRAILS infers code correctness by aggregating LLM judgments on input-output pairs from category-partitioned specification tests, improving MCC by up to 39% over Zero-Shot COT on LiveCodeBench and CoCoClaNeL.