Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adequacy score predicts fault detection in metamorphic testing.

desk verdict The k-MR coverage criterion is a real, clearly defined addition to the MT adequacy toolkit, but the paper's central effectiveness claim is currently tangled up with test-suite size and needs a proper control before it can be believed. read the letter →

arxiv 2412.20692 v1 pith:5SZI6WLZ submitted 2024-12-30 cs.SE

An Fu , Chang-ai Sun , Jiaming Zhang , Huai Liu This is my paper · ORCID
classification cs.SE
keywords metamorphictestingtestadequacycriteriak-MRcoveragefaultdetectioneffectivenessrelationsmeasurementmutationsuiteconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Metamorphic testing (MT) checks software by verifying that necessary relations hold among multiple inputs and outputs, but until now there has been no accepted way to measure how thorough such a test suite is. The paper proposes the k-MR coverage criterion, which requires that each part of the program (or each input category) be exercised by a source input that is paired with at least k distinct metamorphic relations, and a measurement that scores suites between 0 and 1. Its empirical studies on seven programs, five coverage criteria, and over 3,000 faulty versions support the claim that higher adequacy scores go with higher fault detection effectiveness. If the claim holds, practitioners without reliable oracles can use the score to compare suites, guide construction, and decide when to stop testing.

What carries the argument

The central object is the association relationship $\mathrm{Coop}$ between source inputs and metamorphic relations, which records which MRs are used together with which source inputs to construct metamorphic groups (MGs). The k-MR coverage criterion requires that every test requirement of an underlying test-case adequacy criterion $C_c$ be satisfied by at least one source input associated with at least $k$ distinct MRs. The adequacy measurement averages over all requirements the maximum, among satisfying source inputs, of the capped fraction $|SRO(t, \mathrm{Coop})|/k$, yielding a value in $[0,1]$ that couples ordinary input coverage with the diversity of necessary properties brought to bear on each covered requirement.

What would settle it

Take one of the seven subject programs and build two sets of metamorphic suites with equal numbers of MGs: one set selected greedily to reach high k-MR adequacy and one set drawn uniformly at random. If the random suites detect as many mutants as the adequacy-guided suites at every MG count, the measurement's apparent predictive power would reduce to test-volume effects.

Watch

Extended reading notes

Core claim

The paper's central claim is that the degree of adequacy computed from the k-MR coverage criterion is an effective indicator of a metamorphic test suite's fault detection effectiveness. Specifically, across seven subject programs, fault detection effectiveness and fault detection rates increased monotonically with the adequacy level, while increasing the parameter k produced diminishing returns; the paper recommends k=2 when source coverage is IO-CTF and k=3 for the other four coverage criteria studied. An additional outcome was the detection of a real-life fault in print_tokens2, a widely studied program, which the paper attributes to MT's ability to check necessary properties over multiple executions.

Load-bearing premise

The load-bearing premise is that the rise in fault detection with higher adequacy is caused by the adequacy criterion itself, rather than by the larger number of metamorphic groups that higher adequacy requires, because the empirical comparison does not hold suite size or MG count constant.

Editorial extensions

If this is right

  • Testers can use the adequacy score as a stopping rule: suites in higher adequacy intervals consistently detect more faulty versions, so reaching a target level gives a defensible reason to halt.
  • With IO-CTF-based source coverage, k=2 already captures most of the benefit; with I-choice, I-choice-pair, statement, or branch coverage, k=3 is the recommended setting.
  • Achieving high adequacy costs more metamorphic groups, especially under stronger criteria such as IO-CTF, so the measurement makes the tradeoff between thoroughness and test budget explicit.
  • The criteria give MT an adequacy theory comparable to code coverage for conventional testing, filling a gap that the ISO/IEC/IEEE software testing standard leaves open.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The absence of a random baseline means that a portion of the observed effectiveness could stem from the larger number of MGs that higher adequacy requires; comparing adequacy-guided and random suites with equal MG counts would isolate the criterion's contribution.
  • Because the k-MR criterion reuses any underlying coverage criterion $C_c$, it could be layered onto modern coverage notions, such as neuron coverage for deep learning systems, to give metamorphic adequacy scores for AI testing.
  • The measurement could serve as a fitness function for search-based test generation that optimizes suites for adequacy and fault detection simultaneously, an application the paper leaves unexplored.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a new test adequacy criterion for metamorphic testing, the k-MR coverage criterion, and a corresponding adequacy measurement defined in Section III-B/C (Definition 4, Equations 3-5). The criterion combines an existing source-input coverage criterion with the requirement that each coverage requirement is satisfied by at least one source input associated with at least k distinct MRs. The adequacy measurement quantifies the degree to which these requirements are met. The empirical study in Sections IV-V, involving seven programs, five coverage criteria, and over 3,000 mutants, reports that fault detection effectiveness increases with k (RQ1), that suites with higher measured adequacy exhibit higher fault detection effectiveness and fault detection rate (RQ2), and that reaching higher adequacy levels requires more metamorphic groups (RQ3). The paper also reports discovering a real fault in print_tokens2, which it presents as a by-product of the study.

Significance. If the central empirical claim were established, the paper would make a useful contribution: it would give practitioners a coverage-based proxy for metamorphic testing quality and a possible stopping rule, filling a gap noted in the literature. The study is substantial in scale, covering seven subjects, over 3,000 mutants, roughly 68,000 identified MRs, and five coverage criteria, with 100 independently generated suites per condition. The measurement definition is explicit and is not derived from fault detection data, so there is no construction circularity. The reported detection of a long-standing real fault in print_tokens2 is a creditable by-product. However, the current empirical design does not rule out the trivial explanation that larger test suites (more metamorphic groups) detect more faults, and the calibration of k on the same datasets used for validation weakens the generality of the adequacy-effectiveness claim.

major comments (3)
  1. [Section IV-E.2 and Section V-C, Figure 11] The RQ2 adequacy-effectiveness trend is confounded with the number of metamorphic groups (MGs). In RQ2, suites are generated by greedily adding source inputs and MRs until the adequacy value falls in a given interval, which necessarily produces larger suites at higher adequacy levels. RQ3, reported in Section V-C and Figure 11, shows that the number of MGs increases monotonically with adequacy level for all subjects and criteria. Since fault detection effectiveness is already known to increase with test suite size (more executions detect more mutants), the observed monotone increase in FDE does not establish that the k-MR adequacy value itself orders suites by fault detection capability. The paper does not include any control arm of random metamorphic test suites matched on the number of MGs, or on source-input and MR counts. Without such a baseline, the central claim in the Abstract that the proposed measurement can effectively indicate fault detection effectiveness is not supported.
  2. [Section V-A and Section V-B] The value of k is selected from RQ1 on the same seven programs and then reused in RQ2 and RQ3 for those same programs. RQ1 reports that FDE increases with k and that the increment diminishes, and the authors choose k=2 for IO-CTF coverage and k=3 for the other four criteria. Because the same data are used to choose and to validate the parameter, the adequacy-effectiveness trend in RQ2 and RQ3 is subject to selection bias and may be optimistically estimated. The paper should calibrate k on a separate set of subjects or use nested cross-validation, or at least demonstrate that the RQ2/RQ3 conclusions are insensitive to k in a neighborhood of the chosen value.
  3. [Section V-A and Section V-B] The empirical claims of 'increases' and 'monotonically increases' are based on visual inspection of boxplots and bar charts without statistical tests, effect sizes, or confidence intervals. For example, Section V-B states that fault detection effectiveness increases with the test adequacy value, but no p-values or confidence intervals are reported for the 100-suite samples per level. Given that these monotonic trends are the backbone of the paper's central claim, the authors should report appropriate statistical analyses, such as correlation tests with confidence intervals or pairwise comparisons across adjacent adequacy levels.
minor comments (5)
  1. [Section IV-E.1] The text says 'three different sets of MRs are selected to form three test suites,' but k takes values 1, 2, 3, 4 plus the 'all' scenario; please clarify the number of suites per source-input set and how the 'all' scenario is constructed.
  2. [Section VI] In the threats-to-validity discussion, the text refers to 'Section 6' when describing the print_tokens2 defect; it should read 'Section VI'.
  3. [Abstract and Introduction] The abstract uses 'we proposed' while the introduction uses 'we propose'; please choose a consistent tense.
  4. [Section III-C, Definition 5] The function K(T', Coop) uses the maximum over source inputs in T' rather than a sum or average; since this choice materially affects the meaning of the adequacy value when multiple source inputs satisfy a requirement, a brief justification would improve the presentation.
  5. [Appendix, Figures 10 and 11] In the extracted manuscript the appendix figures have garbled axis labels and legends; please ensure the camera-ready figures are legible and that all subplot labels are readable.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the adequacy measure is defined from coverage and MR-association counts without reference to fault detection, so the central claim is empirical rather than self-fulfilling.

full rationale

Walking the claimed derivation chain, the adequacy measurement in Definition 5 (Eqs. 3-5) is computed from source-input coverage, the MR-association sets SRO(t, Coop), and the chosen k; it contains no term involving mutants or fault detection. The central conclusion that higher adequacy 'indicates' higher fault detection effectiveness is therefore an empirical correlation, not an identity. The main empirical support (RQ2) is genuinely in-sample: k is fixed from RQ1 on the same seven programs, and RQ2 suites are greedily enlarged until their adequacy value enters a target interval, while RQ3 shows that higher adequacy levels require more metamorphic groups. That makes the demonstration weaker than a fully controlled comparison and leaves an alternative explanation in terms of suite size, but it does not reduce the adequacy-FDE relationship to the definition of adequacy. The MR pool is built with the authors' own METRIC+ technique [17] and the subject systems are reused from [17]; these are self-citations in the empirical setup, but they are not load-bearing for the criterion definitions, and [17] is an independently published technique. No uniqueness theorem, ansatz, or known result is smuggled in via citation. Accordingly, no specific circular step can be exhibited; the minor self-citation in the benchmark construction is noted without treating it as circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the choice of k, the representativeness of MRs identified by METRIC+, the suitability of standard coverage criteria as input-side measures, and the mutation testing assumption that mutants proxy real faults. No new physical or external entities are introduced; the association relationship and k-MR coverage criterion are formal constructs defined within the paper.

free parameters (1)
  • k (MR multiplicity threshold) = k=2 for IO-CTF coverage; k=3 for statement, branch, I-choice, and I-choice-pair coverage
    Chosen post hoc from RQ1 results on the same seven programs and then used in RQ2/RQ3 to demonstrate the adequacy-FDE relation. This is a fitted hyperparameter, not an a priori constant.
assumptions (5)
  • domain assumption Each MR can be split into an input subrelation and an output subrelation, with follow-up inputs constructed independently of source outputs.
    Invoked in Section III.B and identified in Section V-D as a limitation of METRIC+; the paper cites that over 90% of MRs in existing studies are of this type, but the generality of the criteria relies on this assumption.
  • domain assumption Every source input in Ts is eligible for and used with at least one MR, and every MR is used with at least one source input.
    Stated as basic assumptions in Section III.B before Definition 1; the measurement is undefined for isolated inputs or unused MRs.
  • domain assumption Statement, branch, I-choice, I-choice-pair, and IO-CTF coverage are appropriate proxies for exercising different parts of the SUT.
    The k-MR criterion reuses an existing coverage criterion Cc as the input-side requirement in Section III.B; if Cc does not reflect fault-relevant behavior, the adequacy score inherits that limitation.
  • domain assumption Mutants generated by muJava and Proteum are representative of real faults.
    The FDE and FDR evaluations in Section V treat mutant detection as the ground truth for fault detection effectiveness; this is standard mutation testing practice but remains an assumption.
  • domain assumption The MRs identified by METRIC+ are correct and sufficiently complete.
    Section IV-C uses METRIC+ from the authors' own prior work, and Section V-D states the authors manually checked correctness; no independent verification is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication." pith.science (2026). https://pith.science/paper/5SZI6WLZ

@misc{pith2026241220692,
  author       = {Pith},
  title        = {Pith review of: Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5SZI6WLZ}},
  note         = {Machine review of arXiv:2412.20692}
}
read the original abstract

Metamorphic testing (MT) is a simple yet effective technique to alleviate the oracle problem in software testing. The underlying idea of MT is to test a software system by checking whether metamorphic relations (MRs) hold among multiple test inputs (including source and follow-up inputs) and the actual output of their executions. Since MRs and source inputs are two essential components of MT, considerable efforts have been made to examine the systematic identification of MRs and the effective generation of source inputs, which has greatly enriched the fundamental theory of MT since its invention. However, few studies have investigated the test adequacy assessment issue of MT, which hinders the objective measurement of MT's test quality as well as the effective construction of test suites. Although in the context of traditional software testing, there exist a number of test adequacy criteria that specify testing requirements to constitute an adequate test from various perspectives, they are not in line with MT's focus which is to test the software under testing (SUT) from the perspective of necessary properties. In this paper, we proposed a new set of criteria that specifies testing requirements from the perspective of necessary properties satisfied by the SUT, and designed a test adequacy measurement that evaluates the degree of adequacy based on both MRs and source inputs. The experimental results have shown that the proposed measurement can effectively indicate the fault detection effectiveness of test suites, i.e., test suites with increased test adequacy usually exhibit higher effectiveness in fault detection. Our work made an attempt to assess the test adequacy of MT from a new perspective, and our criteria and measurement provide a new approach to evaluate the test quality of MT and provide guidelines for constructing effective test suites of MT.

Figures

Figures reproduced from arXiv: 2412.20692 by the authors.

Figure 1
Figure 1. Comparison between MT and conventional testing techniques. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of k-MR coverage criterion. Ri in denotes the input subrelation of an MR and Ri out denotes the output subrelation of an MR. to which test cases satisfy a given criterion, the proposed measurement quantifies test adequacy of MT on the basis of the key components of MT including MRs, source inputs, and the association relationship among them, which focuses on the factors that affect the test quality of M… view at source ↗
Figure 3
Figure 3. Distribution of the fault detection effectiveness of [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Distribution of the fault detection effectiveness of [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Average of the fault detection rate of k-MR coverage criterion under different test case adequacy criteria on program grep involving IO-CTF coverage and I-choice-pair coverage as the degree of test adequacy increases, while scenarios involving I-choice coverage, statem…
Figure 6
Figure 6. Figure 6: Distribution of the number of MGs to reach a given test adequacy level under different test case adequacy criteria. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: A real-life fault in print_tokens2. double quotation mark at the beginning of the sequence. The correct code should be “if (id == 1 && ch == 34)” (in which “34” is the ASCII code of a double quotation mark). This fault violates the following MR: “Given a source input t…
Figure 8
Figure 8. Figure 8: Distribution of the fault detection effectiveness of [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Distribution of the fault detection effectiveness of [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Average of the fault detection rate of k-MR coverage criterion under different test case adequacy criteria [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Distribution of the number of MGs to reach a given test adequacy level under different test case adequacy criteria [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading

    cs.LG 2026-08 conditional novelty 7.0 of 10

    An error is invisible to an edit exactly when it commutes with the edit's answer transform, so the missed errors of any edit suite form its joint centralizer.

Reference graph

Works this paper leans on

70 extracted references · 69 canonical work pages · cited by 1 Pith paper

  1. [1]

    Software fail watch: 5th edition,

    Tricents, “Software fail watch: 5th edition,” last accessed on December 7, 2020. [Online]. Available: https://www.tricentis.com/ resources/software-fail-watch-5th-edition/

    work page 2020

  2. [2]

    The Oracle Problem in Software Testing: A Survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The Oracle Problem in Software Testing: A Survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015

    work page 2015

  3. [3]

    A mapping study on testing non-testable systems,

    K. Patel and R. M. Hierons, “A mapping study on testing non-testable systems,” Software Quality Journal, vol. 26, no. 4, pp. 1373–1413, 2018

    work page 2018

  4. [4]

    Metamorphic testing: a new approach for generating next test cases,

    T. Y . Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: a new approach for generating next test cases,” Technical Report HKUST- CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998

    work page 1998

  5. [5]

    DeepTest: automated testing of deep-neural-network-driven autonomous cars,

    Y . Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: automated testing of deep-neural-network-driven autonomous cars,” in Proceedings of the 40th International Conference on Software Engineering (ICSE’18) . ACM, 2018, pp. 303–314

    work page 2018

  6. [6]

    Metamorphic testing of deep learning compilers,

    D. Xiao, Z. Liu, Y . Yuan, Q. Peng, and S. Wang, “Metamorphic testing of deep learning compilers,” Proceedings of the ACM on Measurement and Analysis of Computing Systems , vol. 6, no. 1, pp. 1–28, 2022

    work page 2022

  7. [7]

    A property-based testing framework for encryption programs,

    C.-A. Sun, Z. Wang, and G. Wang, “A property-based testing framework for encryption programs,” Frontiers of Computer Science , vol. 8, no. 3, pp. 478–489, 2014

    work page 2014

  8. [8]

    Metamorphic robustness testing: Exposing hidden defects in citation statistics and journal impact factors,

    Z. Q. Zhou, T. H. Tse, and M. Witheridge, “Metamorphic robustness testing: Exposing hidden defects in citation statistics and journal impact factors,” IEEE Transactions on Software Engineering, vol. 47, no. 6, pp. 1164–1183, 2021

    work page 2021

Show all 70 references
  1. [9]

    Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing,

    Y . Yuan, S. Wang, M. Jiang, and T. Y . Chen, “Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21) . IEEE, 2021, pp. 16 908– 16 917

    2021

  2. [10]

    Deepbackground: Metamorphic testing for deep-learning-driven image recognition systems accompanied by background-relevance,

    Z. Zhang, P. Wang, H. Guo, Z. Wang, Y . Zhou, and Z. Huang, “Deepbackground: Metamorphic testing for deep-learning-driven image recognition systems accompanied by background-relevance,” Informa- tion and Software Technology, vol. 140, pp. 106 701:1–106 701:14, 2021

    2021

  3. [11]

    An interleaving-guided metamorphic testing approach for concurrent programs,

    C.-A. Sun, H. Dai, N. Geng, H. Liu, T. Y . Chen, P. Wu, Y . Cai, and J. Wang, “An interleaving-guided metamorphic testing approach for concurrent programs,” ACM Transactions on Softwware Engineering and Methodology, vol. 33, no. 1, pp. 8:1–8:21, 2024

    2024

  4. [12]

    ISO/IEC/IEEE International Standard – Software and systems engineering – Software testing – Part 4: Test techniques,

    ISO/IEC/IEEE, “ISO/IEC/IEEE International Standard – Software and systems engineering – Software testing – Part 4: Test techniques,” ISO/IEC/IEEE 29119-4:2021(E), pp. 1–148, 2021

    2021

  5. [13]

    MRpredT: Using Text Mining for Metamorphic Relation Prediction,

    K. Rahman, I. Kahanda, and U. Kanewala, “MRpredT: Using Text Mining for Metamorphic Relation Prediction,” in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW’21). ACM, 2021, pp. 420–424

    2021

  6. [14]

    Theoretical and empirical analyses of the effectiveness of metamorphic relation composition,

    K. Qiu, Z. Zheng, T. Y . Chen, and P.-L. Poon, “Theoretical and empirical analyses of the effectiveness of metamorphic relation composition,” IEEE Transactions on Software Engineering , vol. 48, no. 3, pp. 1001– 1017, 2022

    2022

  7. [15]

    µMT: A data mutation directed metamorphic relation acquisition methodology,

    C.-A. Sun, Y . Liu, Z. Wang, and W. Chan, “ µMT: A data mutation directed metamorphic relation acquisition methodology,” in Proceedings of the 1st International Workshop on Metamorphic Testing, in conjunc- tion with the 38th International Conference on Software Engineering (IC...

    2016

  8. [16]

    METRIC: METamorphic Relation Identification based on the Category-choice framework,

    T. Y . Chen, P.-L. Poon, and X. Xie, “METRIC: METamorphic Relation Identification based on the Category-choice framework,” Journal of Systems and Software , vol. 116, pp. 177–190, 2016

    2016

  9. [17]

    METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output Domains,

    C.-A. Sun, A. Fu, P.-L. Poon, X. Xie, H. Liu, and T. Y . Chen, “METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output Domains,” IEEE Transactions on Software Engi- neering, vol. 47, no. 9, pp. 1764–1785, 2021

    2021

  10. [18]

    Automated generation of metamorphic relations for query-based systems,

    S. Segura, J. C. Alonso, A. Martin-Lopez, A. Durán, J. Troya, and A. Ruiz-Cortés, “Automated generation of metamorphic relations for query-based systems,” in Proceedings of the 7th International Workshop on Metamorphic Testing, in conjunction with the 44th International Confer...

    2022

  11. [19]

    How effectively does metamorphic testing alleviate the oracle problem?

    H. Liu, F.-C. Kuo, D. Towey, and T. Y . Chen, “How effectively does metamorphic testing alleviate the oracle problem?” IEEE Transactions on Software Engineering , vol. 40, no. 1, pp. 4–22, 2014

    2014

  12. [20]

    An empirical study on the selection of good metamorphic relations,

    J. Mayer and R. Guderlei, “An empirical study on the selection of good metamorphic relations,” in Proceedings of the 30th Annual Interna- tional Computer Software and Applications Conference (COMPSAC’06), vol. 1. IEEE, 2006, pp. 475–484

    2006

  13. [21]

    Path-directed source test case generation and prioritization in metamorphic testing,

    C.-A. Sun, B. Liu, A. Fu, Y . Liu, and H. Liu, “Path-directed source test case generation and prioritization in metamorphic testing,” Journal of Systems and Software , vol. 183, pp. 111 091:1–111 091:14, 2022

    2022

  14. [22]

    Generating source inputs for metamorphic testing using dynamic symbolic execution,

    E. Alatawi, T. Miller, and H. Søndergaard, “Generating source inputs for metamorphic testing using dynamic symbolic execution,” in Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16), Co-located with the 38th International Conference on Software Engi-...

    2016

  15. [23]

    The impact of source test case selection on the effectiveness of meta- morphic testing,

    A. C. Barus, T. Y . Chen, F.-C. Kuo, H. Liu, and H. W. Schmidt, “The impact of source test case selection on the effectiveness of meta- morphic testing,” in Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16), Co-located with the 38th International Co...

    2016

  16. [24]

    Fault detection effectiveness of source test case generation strategies for metamorphic testing,

    P. Saha and U. Kanewala, “Fault detection effectiveness of source test case generation strategies for metamorphic testing,” in Proceedings of the 3rd International Workshop on Metamorphic Testing (MET’18), Co- located with the 40th International Conference on Software Engineer...

    2018

  17. [25]

    Software unit test coverage and adequacy,

    H. Zhu, P. A. V . Hall, and J. H. R. May, “Software unit test coverage and adequacy,” ACM Computing Surveys , vol. 29, no. 4, pp. 366–427, 1997

    1997

  18. [26]

    Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program,

    J. Ding and X.-H. Hu, “Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program,” Software Quality Journal, vol. 25, no. 3, pp. 841–869, 2017

    2017

  19. [27]

    Dreaming up metamorphic relations: Experiences from three fuzzer tools,

    A. Lascu, M. Windsor, A. F. Donaldson, T. Grosser, and J. Wick- erson, “Dreaming up metamorphic relations: Experiences from three fuzzer tools,” in Proceedings of the 6th International Workshop on Metamorphic Testing (MET’21), Co-located with the 43th International Conference ...

    2021

  20. [28]

    Test adequacy basics,

    A. P. Mathur, “Test adequacy basics,” in Foundations of Software Testing, Second Edition. Pearson Education India, 2013

    2013

  21. [29]

    Software testing based on formal specifications: a theory and a tool,

    G. Bernot, M. C. Gaudel, and B. Maree, “Software testing based on formal specifications: a theory and a tool,” Software Engineering Journal, vol. 6, no. 6, pp. 387–405, 1991

    1991

  22. [30]

    Test adequacy and program mutation,

    R. A. DeMillo, “Test adequacy and program mutation,” in Proceedings of the 11th international conference on Software engineering (ICSE’89) . ACM, 1989, pp. 355–356

    1989

  23. [31]

    An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing,

    B. Miranda and A. Bertolino, “An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing,” Software Quality Journal, vol. 26, no. 4, pp. 1571–1594, 2017

    2017

  24. [32]

    Metamorphic testing: A review of challenges and opportunities,

    T. Y . Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys , vol. 51, no. 1, pp. 4:1–4:27, 2018

    2018

  25. [33]

    Debugging and testing,

    P. C. Poole, “Debugging and testing,” in Software Engineering: An Advanced Course. Berlin, Heidelberg: Springer, 1975, pp. 278–318

    1975

  26. [34]

    Toward a theory of test data selection,

    J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection,” IEEE Transactions on Software Engineering, vol. SE-1, no. 2, pp. 156–173, 1975

    1975

  27. [35]

    Axiomatizing software test data adequacy,

    E. J. Weyuker, “Axiomatizing software test data adequacy,” IEEE Transactions on Software Engineering , vol. SE-12, no. 12, pp. 1128– 1138, 1986

    1986

  28. [36]

    Test data adequacy measurement,

    H. Zhu and P. A. V . Hall, “Test data adequacy measurement,” Software Engineering Journal, vol. 8, no. 1, pp. 21–30, 1993

    1993

  29. [37]

    Hints on Test Data Selection: Help for the Practicing Programmer,

    R. DeMillo, R. Lipton, and F. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, pp. 34– 41, 1978

    1978

  30. [38]

    An applicable family of data flow testing criteria,

    P. G. Frankl and E. J. Weyuker, “An applicable family of data flow testing criteria,” IEEE Transactions on Software Engineering , vol. 14, no. 10, pp. 1483–1498, 1988

    1988

  31. [39]

    Industry practice of coverage-guided enterprise linux kernel fuzzing,

    H. Shi, R. Wang, Y . Fu, M. Wang, X. Shi, X. Jiao, H. Song, Y . Jiang, and J. Sun, “Industry practice of coverage-guided enterprise linux kernel fuzzing,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundat...

    2019

  32. [40]

    Industry practices and challenges for the evolvability assurance of microservices,

    J. Bogner, J. Fritzsch, S. Wagner, and A. Zimmermann, “Industry practices and challenges for the evolvability assurance of microservices,” Empirical Software Engineering, vol. 26, no. 5, pp. 104:1–104:39, 2021

    2021

  33. [41]

    A cost-effective random testing method for programs with non-numeric inputs,

    A. C. Barus, T. Y . Chen, F.-C. Kuo, H. Liu, R. Merkel, and G. Rothermel, “A cost-effective random testing method for programs with non-numeric inputs,” IEEE Transactions on Computers , vol. 65, no. 12, pp. 3509– 3523, 2016. IEEE XXXXX, VOL. 00, NO. 0, FEBRUARY 2024 16

    2016

  34. [42]

    muJava: a mutation system for java,

    Y .-S. Ma, J. Offutt, and Y .-R. Kwon, “muJava: a mutation system for java,” in Proceedings of the 28th International Conference on Software Engineering (ICSE 2006) . ACM, 2006, pp. 827–830

    2006

  35. [43]

    Proteum/IM 2.0: An integrated mutation testing environment,

    M. E. Delamaro, J. C. Maldonado, and A. M. R. Vincenzi, “Proteum/IM 2.0: An integrated mutation testing environment,” in Mutation testing for the new century . Springer, 2001, pp. 91–101

    2001

  36. [44]

    Heuristics for determining equivalence of program mutations,

    D. Baldwin and F. Sayward, “Heuristics for determining equivalence of program mutations,” Research Report #161, Department of Computer Science, Yale University, New Haven, 1979

    1979

  37. [45]

    Detecting equivalent mutants and the feasible path problem,

    J. Offutt and J. Pan, “Detecting equivalent mutants and the feasible path problem,” in Proceedings of 11th Annual Conference on Computer Assurance (COMPASS’96). IEEE, 1996, pp. 17–21

    1996

  38. [46]

    A Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence,

    X. Yao, M. Harman, and Y . Jia, “A Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence,” in Pro- ceedings of the 36th International Conference on Software Engineering (ICSE’14). IEEE, 2014, pp. 919–930

    2014

  39. [47]

    Enhancing partition testing through output variation,

    H. Liu, P.-L. Poon, and T. Y . Chen, “Enhancing partition testing through output variation,” in Proceedings of the 37th International Conference on Software Engineering (ICSE’15) . IEEE, 2015, pp. 805–806

    2015

  40. [48]

    Using formal methods to derive test frames in category-partition testing,

    P. Ammann and J. Offutt, “Using formal methods to derive test frames in category-partition testing,” in Proceedings of the 9th Annual Conference on Computer Assurance (COMPASS’94) . IEEE, 1994, pp. 69–79

    1994

  41. [49]

    Jacoco java code coverage library,

    EclEmma, “Jacoco java code coverage library,” last accessed on June 20, 2022. [Online]. Available: https://www.eclemma.org/jacoco/

    2022

  42. [50]

    gcov—a test coverage program,

    GNU, “gcov—a test coverage program,” last accessed on June 20,

  43. [51]

    Evosuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: Automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE’11) . ACM, 2011, pp. 416–419

    2011

  44. [52]

    Using mutation analysis for assessing and comparing testing coverage criteria,

    J. Andrews, L. Briand, Y . Labiche, and A. Namin, “Using mutation analysis for assessing and comparing testing coverage criteria,” IEEE Transactions on Software Engineering, vol. 32, no. 8, pp. 608–624, 2006

    2006

  45. [53]

    A theoretical and empirical study of diversity-aware mutation adequacy criterion,

    D. Shin, S. Yoo, and D.-H. Bae, “A theoretical and empirical study of diversity-aware mutation adequacy criterion,” IEEE Transactions on Software Engineering, vol. 44, no. 10, pp. 914–931, 2018

    2018

  46. [54]

    Metamorphic slice: an application in spectrum-based fault localization,

    X. Xie, W. E. Wong, T. Y . Chen, and B. Xu, “Metamorphic slice: an application in spectrum-based fault localization,” Information and Software Technology, vol. 55, no. 5, pp. 866–879, 2013

    2013

  47. [55]

    Feedback-directed meta- morphic testing,

    C.-A. Sun, H. Dai, H. Liu, and T. Y . Chen, “Feedback-directed meta- morphic testing,” ACM Transactions on Softwware Engineering and Methodology, vol. 32, no. 1, pp. 20:1–20:34, 2023

    2023

  48. [56]

    An optimized method for generating cases of metamorphic testing,

    L. Chen, L. Cai, J. Liu, Z. Liu, S. Wei, and P. Liu, “An optimized method for generating cases of metamorphic testing,” in Proceedings of the 6th International Conference on New Trends in Information Science, Service Science and Data Mining (ISSDM’12) . IEEE, 2012, pp. 439–443

    2012

  49. [57]

    Effectively metamorphic testing based on program path analysis,

    G. Dong, C. Nie, and B. Xu, “Effectively metamorphic testing based on program path analysis,” Chinese Journal of Computers , vol. 32, no. 5, pp. 1002–1013, 2009

    2009

  50. [58]

    A survey on metamorphic testing,

    S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortés, “A survey on metamorphic testing,” IEEE Transactions on Software Engineering , vol. 42, no. 9, pp. 805–824, 2016

    2016

  51. [59]

    Enhance combinatorial testing with metamorphic relations,

    X. Niu, Y . Sun, H. Wu, G. Li, C. Nie, L. Yu, and X. Wang, “Enhance combinatorial testing with metamorphic relations,” IEEE Transactions on Software Engineering , vol. 48, no. 12, pp. 5007–5029, 2022

    2022

  52. [60]

    Metamorphic relation au- tomation: Rationale, challenges, and solution directions,

    E. Altamimi, A. Elkawakjy, and C. Catal, “Metamorphic relation au- tomation: Rationale, challenges, and solution directions,” Journal of Software: Evolution and Process , vol. 35, no. 1, p. e2509, 2023

    2023

  53. [61]

    Metamorphic relations for enhancing system understanding and use,

    Z. Q. Zhou, L. Sun, T. Y . Chen, and D. Towey, “Metamorphic relations for enhancing system understanding and use,” IEEE Transactions on Software Engineering, vol. 46, no. 10, pp. 1120–1154, 2020

    2020

  54. [62]

    Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels,

    U. Kanewala, J. M. Bieman, and A. Ben-Hur, “Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels,” Software Testing, Verification and Reliability , vol. 26, no. 3, pp. 245–269, 2016

    2016

  55. [63]

    Mrpredt: Using text mining for metamorphic relation prediction,

    K. Rahman, I. Kahanda, and U. Kanewala, “Mrpredt: Using text mining for metamorphic relation prediction,” in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops . ACM, 2020, pp. 420–424

    2020

  56. [64]

    Automatic discovery and cleansing of numerical metamorphic relations,

    B. Zhang, H. Zhang, J. Chen, D. Hao, and P. Moscato, “Automatic discovery and cleansing of numerical metamorphic relations,” in Pro- ceedings of the 35th IEEE International Conference on Software Main- tenance and Evolution (ICSME’19) . IEEE, 2019, pp. 235–245

    2019

  57. [65]

    Datamorphic testing: A method for testing intelligent applications,

    H. Zhu, D. Liu, I. Bayley, R. Harrison, and F. Cuzzolin, “Datamorphic testing: A method for testing intelligent applications,” in Proceedings of the 1st IEEE International Conference On Artificial Intelligence Testing (AITest’19). IEEE, 2019, pp. 149–156

    2019

  58. [66]

    MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation,

    A. Blasi, A. Gorla, M. D. Ernst, M. Pezzè, and A. Carzaniga, “MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation,” Journal of Systems and Software , vol. 181, pp. 111 041:1–111 041:13, 2021

    2021

  59. [67]

    An iterative metamorphic testing technique for web services and case studies,

    C.-A. Sun, A. Fu, Y . Liu, Q. Wen, Z. Wang, P. Wu, and T. Y . Chen, “An iterative metamorphic testing technique for web services and case studies,” International Journal of Web and Grid Services , vol. 16, no. 4, pp. 364–392, 2020

    2020

  60. [68]

    Metamorphic testing and testing with special values,

    T. Y . Chen, F.-C. Kuo, Y . Liu, and A. Tang, “Metamorphic testing and testing with special values,” in Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD’04) , 2004, pp. 128–134

    2004

  61. [69]

    Adaptive metamorphic testing with contex- tual bandits,

    H. Spieker and A. Gotlieb, “Adaptive metamorphic testing with contex- tual bandits,” Journal of Systems and Software, vol. 165, pp. 110 574:1– 110 574:14, 2020. An Fu is a PhD student in the School of Computer and Communication Engineering, University of Sci- ence and Technolo...

    2020

  62. [2022]

    Available: https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

    [Online]. Available: https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.