Pith

open record

sign in

arxiv: 2310.19204 · v2 · pith:M4L4DH6S · submitted 2023-10-30 · cs.SE · cs.AI· cs.HC

Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing

Quang-Hung Luu , Huai Liu , Tsong Yueh Chen This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:M4L4DH6Srecord.jsonopen to challenge →

classification cs.SE cs.AIcs.HC
keywords testingchatgptintelligencesoftwarecandidateshumanmetamorphicused
0
0 comments X
read the original abstract

While ChatGPT is a well-known artificial intelligence chatbot being used to answer human's questions, one may want to discover its potential in advancing software testing. We examine the capability of ChatGPT in advancing the intelligence of software testing through a case study on metamorphic testing (MT), a state-of-the-art software testing technique. We ask ChatGPT to generate candidates of metamorphic relations (MRs), which are basically necessary properties of the object program and which traditionally require human intelligence to identify. These MR candidates are then evaluated in terms of correctness by domain experts. We show that ChatGPT can be used to generate new correct MRs to test several software systems. Having said that, the majority of MR candidates are either defined vaguely or incorrect, especially for systems that have never been tested with MT. ChatGPT can be used to advance software testing intelligence by proposing MR candidates that can be later adopted for implementing tests; but human intelligence should still inevitably be involved to justify and rectify their correctness.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis

    cs.SE 2026-04 conditional novelty 7.0

    MR-Coupler leverages functional coupling analysis and LLMs to generate valid metamorphic test cases for over 90% of tasks while detecting 44% of real bugs, outperforming baselines by 64.90% in validity and 36.56% in f...

  2. Multi-Agent LLM-based Metamorphic Testing for REST APIs

    cs.SE 2026-05 unverdicted novelty 5.0

    ARMeta uses multi-agent LLMs to identify and execute metamorphic relations for REST API testing, showing complementary coverage to scenario-based baselines on two public applications.

  3. Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations

    cs.SE 2026-05 unverdicted novelty 5.0

    A multi-agent LLM workflow extracts Given-When-Then metamorphic relations from specifications to generate and run tests on FMU simulations, demonstrated on a lube oil cooling system FMU.

  4. Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey

    cs.SE 2026-05 accept novelty 4.0

    A systematic survey of 93 studies that maps the bidirectional relationship between metamorphic testing and LLMs, proposing a taxonomy for MT applied to LLMs and LLMs applied to MT.