{"id":"bc4a5ad1-f8bb-47ac-888f-f4067d228d5c","arxiv_id":"2412.20692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A k-MR coverage criterion measures metamorphic test adequacy by requiring each source input to be associated with at least k metamorphic relations, and experiments show it correlates with fault detection.","lead":"Researchers propose a new way to measure how thorough a metamorphic test suite is, by counting how many different metamorphic relations each test input is checked against. The experimental evidence suggests that higher measured adequacy comes with better fault finding, but the comparison does not control for the sheer number of test executions, so the evidence is suggestive rather than conclusive.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's adequacy–effectiveness trend is confounded with metamorphic-group count; without equal-size random baselines the central claim that adequacy itself predicts fault detection is not established.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the empirical relationship between adequacy level and fault detection effectiveness is confounded with the number of metamorphic groups required to reach that adequacy level. The paper's own RQ3 data (Section V-C, Fig. 11) shows that higher adequacy levels require more MGs, and RQ2's greedy construction (Section IV-E.2) means adequacy is not varied independently of test suite size. Without a comparison against random suites of equal size, the central claim that the measurement itself indicates fault detection effectiveness is not supported. This is not a disagreement with the proposed criteria in principle; it is a question of whether the empirical evidence distinguishes the criterion's contribution from a trivial size effect. The proposed check—matching random suites on MG count and comparing FDE—would settle this directly. I also note that the k value used in RQ2 is chosen from RQ1 on the same subject programs, which is an additional validation concern, but the size confound is more fundamental and should be addressed first. The paper does have independent support in the form of a previously undetected real fault in print_tokens2, but that anecdote does not bear on the adequacy–effectiveness correlation. The reader's CONDITIONAL verdict is appropriate: the concern is substantial but addressable with additional controlled experiments, so the verdict should remain unchanged.","tokens_in":29125,"tokens_out":2684,"duration_ms":29476,"concrete_test":"Re-run RQ2 with size-matched random baselines: for each of the 100 greedy suites at each adequacy level, generate 100 random metamorphic test suites with the same number of MGs (or, more stringently, with the same number of source inputs and same number of MR associations), drawn from the same source-input and MR pools, and compare the FDE distributions. If the random suites' median FDE is statistically indistinguishable from the greedy suites at the same MG count, the adequacy measurement adds no predictive value beyond suite size; if greedy suites dominate matched random suites across the adequacy range, the confound is refuted and the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract) is that the proposed measurement can effectively indicate fault detection effectiveness, i.e., higher adequacy implies higher fault detection. The decisive evidence is RQ2, where suites are greedily enlarged until their adequacy value falls in a given interval (Section IV-E.2) and FDE is compared across intervals. Section V-C then shows that reaching higher adequacy levels requires more metamorphic groups (MGs), with the number of MGs increasing monotonically with adequacy level (e.g., Fig. 11 for EXPENSE and the appendix for all subjects). Because a suite at a higher adequacy level contains more source inputs and/or more MR associations, and hence more MGs, the observed FDE increase may simply reflect the well-known fact that larger test suites detect more mutants. The paper does not include any control arm of random metamorphic test suites matched on the number of MGs (or on source-input and MR counts). Without that baseline, the specific contribution of the k-MR coverage measurement—ordering suites by coverage beyond their raw size—is untested. RQ1, which varies k while fixing the source-input coverage, is also not controlled for the number of MRs per source input, so the k-trend could likewise be driven by MG count rather than by the diversity of MRs. This confound is the strongest threat to the central empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new test adequacy criterion for metamorphic testing, the k-MR coverage criterion, and a corresponding adequacy measurement defined in Section III-B/C (Definition 4, Equations 3-5). The criterion combines an existing source-input coverage criterion with the requirement that each coverage requirement is satisfied by at least one source input associated with at least k distinct MRs. The adequacy measurement quantifies the degree to which these requirements are met. The empirical study in Sections IV-V, involving seven programs, five coverage criteria, and over 3,000 mutants, reports that fault detection effectiveness increases with k (RQ1), that suites with higher measured adequacy exhibit higher fault detection effectiveness and fault detection rate (RQ2), and that reaching higher adequacy levels requires more metamorphic groups (RQ3). The paper also reports discovering a real fault in print_tokens2, which it presents as a by-product of the study.","tokens_in":29332,"tokens_out":4392,"duration_ms":40469,"significance":"If the central empirical claim were established, the paper would make a useful contribution: it would give practitioners a coverage-based proxy for metamorphic testing quality and a possible stopping rule, filling a gap noted in the literature. The study is substantial in scale, covering seven subjects, over 3,000 mutants, roughly 68,000 identified MRs, and five coverage criteria, with 100 independently generated suites per condition. The measurement definition is explicit and is not derived from fault detection data, so there is no construction circularity. The reported detection of a long-standing real fault in print_tokens2 is a creditable by-product. However, the current empirical design does not rule out the trivial explanation that larger test suites (more metamorphic groups) detect more faults, and the calibration of k on the same datasets used for validation weakens the generality of the adequacy-effectiveness claim.","major_comments":[{"comment":"The RQ2 adequacy-effectiveness trend is confounded with the number of metamorphic groups (MGs). In RQ2, suites are generated by greedily adding source inputs and MRs until the adequacy value falls in a given interval, which necessarily produces larger suites at higher adequacy levels. RQ3, reported in Section V-C and Figure 11, shows that the number of MGs increases monotonically with adequacy level for all subjects and criteria. Since fault detection effectiveness is already known to increase with test suite size (more executions detect more mutants), the observed monotone increase in FDE does not establish that the k-MR adequacy value itself orders suites by fault detection capability. The paper does not include any control arm of random metamorphic test suites matched on the number of MGs, or on source-input and MR counts. Without such a baseline, the central claim in the Abstract that the proposed measurement can effectively indicate fault detection effectiveness is not supported.","section":"Section IV-E.2 and Section V-C, Figure 11"},{"comment":"The value of k is selected from RQ1 on the same seven programs and then reused in RQ2 and RQ3 for those same programs. RQ1 reports that FDE increases with k and that the increment diminishes, and the authors choose k=2 for IO-CTF coverage and k=3 for the other four criteria. Because the same data are used to choose and to validate the parameter, the adequacy-effectiveness trend in RQ2 and RQ3 is subject to selection bias and may be optimistically estimated. The paper should calibrate k on a separate set of subjects or use nested cross-validation, or at least demonstrate that the RQ2/RQ3 conclusions are insensitive to k in a neighborhood of the chosen value.","section":"Section V-A and Section V-B"},{"comment":"The empirical claims of 'increases' and 'monotonically increases' are based on visual inspection of boxplots and bar charts without statistical tests, effect sizes, or confidence intervals. For example, Section V-B states that fault detection effectiveness increases with the test adequacy value, but no p-values or confidence intervals are reported for the 100-suite samples per level. Given that these monotonic trends are the backbone of the paper's central claim, the authors should report appropriate statistical analyses, such as correlation tests with confidence intervals or pairwise comparisons across adjacent adequacy levels.","section":"Section V-A and Section V-B"}],"minor_comments":[{"comment":"The text says 'three different sets of MRs are selected to form three test suites,' but k takes values 1, 2, 3, 4 plus the 'all' scenario; please clarify the number of suites per source-input set and how the 'all' scenario is constructed.","section":"Section IV-E.1"},{"comment":"In the threats-to-validity discussion, the text refers to 'Section 6' when describing the print_tokens2 defect; it should read 'Section VI'.","section":"Section VI"},{"comment":"The abstract uses 'we proposed' while the introduction uses 'we propose'; please choose a consistent tense.","section":"Abstract and Introduction"},{"comment":"The function K(T', Coop) uses the maximum over source inputs in T' rather than a sum or average; since this choice materially affects the meaning of the adequacy value when multiple source inputs satisfy a requirement, a brief justification would improve the presentation.","section":"Section III-C, Definition 5"},{"comment":"In the extracted manuscript the appendix figures have garbled axis labels and legends; please ensure the camera-ready figures are legible and that all subplot labels are readable.","section":"Appendix, Figures 10 and 11"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a software-engineering journal's scope and the proposed criteria are a reasonable contribution. The main technical concern is the confound between adequacy level and metamorphic-group count in RQ2/RQ3, which is load-bearing for the abstract's claim; the authors should be encouraged to add an equal-size random-suite baseline. The k-selection-on-data issue is also important and should be addressed by some form of independent calibration or sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the proposed k-MR coverage criterion is genuinely new and cleanly defined: it takes an existing source-input coverage criterion and adds a per-source-input requirement that each test requirement be satisfied by some input associated with at least k distinct MRs. That is a real difference from ECCEM and APCEM/APPCEM, which constrain per-MR coverage. The measurement formula is computable, the worked example in Section III is easy to follow, and the paper honestly frames the criterion as a practical, finite stopping rule rather than a complete property set. Second, the headline empirical claim — that higher adequacy means higher fault detection — is not yet supported, because adequacy level and test-suite size are allowed to move together.\n\nWhat the paper does well: the empirical study is large (seven programs, thousands of mutants, tens of thousands of MRs, five coverage criteria), the related work is positioned accurately, and the reported real-life defect in print_tokens2 is a nice by-product that suggests the test generation was not vacuous. The definitions are internally consistent, and the measurement itself is not circular by construction.\n\nThe soft spots are real, and the main one is the confound. In RQ2, suites are greedily expanded until they reach a target adequacy interval; RQ3 then shows that higher adequacy levels require more metamorphic groups. So the observed FDE increase could simply be the already-known effect that larger test suites kill more mutants. There is no control arm of random metamorphic suites matched on MG count, or even on source-input and MR counts. Without that, the specific contribution of the coverage measurement over raw test size is untested. Relatedly, in RQ1 the k values are chosen on the same seven programs and then used in RQ2/RQ3, so part of the validation is in-sample tuning. That is not fatal, but it needs cross-validation or a holdout set. I would also note that no artifacts are provided, which makes the empirical results impossible to re-run; and all MRs come from the authors' own METRIC+ technique, which limits generalization but is not a flaw in itself.\n\nWho should read this: MT researchers and practitioners who want an adequacy-guided stopping rule, and empirical software engineering folks interested in how coverage criteria are validated. The paper deserves a serious referee, but the authors should be pushed to add equal-size random baselines, use a more honest k-selection procedure, and release their data and scripts.","headline":"The k-MR coverage criterion is a real, clearly defined addition to the MT adequacy toolkit, but the paper's central effectiveness claim is currently tangled up with test-suite size and needs a proper control before it can be believed.","tokens_in":29903,"tokens_out":2555,"would_cite":true,"duration_ms":29423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adequacy score predicts fault detection in metamorphic testing.","keywords":["metamorphic testing","test adequacy criteria","k-MR coverage","fault detection effectiveness","metamorphic relations","coverage measurement","mutation testing","test suite construction"],"falsifier":"Take one of the seven subject programs and build two sets of metamorphic suites with equal numbers of MGs: one set selected greedily to reach high k-MR adequacy and one set drawn uniformly at random. If the random suites detect as many mutants as the adequacy-guided suites at every MG count, the measurement's apparent predictive power would reduce to test-volume effects.","tokens_in":28891,"feed_emoji":"🐞","tokens_out":6385,"duration_ms":57381,"temperature":0.7,"pith_summary":"Metamorphic testing (MT) checks software by verifying that necessary relations hold among multiple inputs and outputs, but until now there has been no accepted way to measure how thorough such a test suite is. The paper proposes the k-MR coverage criterion, which requires that each part of the program (or each input category) be exercised by a source input that is paired with at least k distinct metamorphic relations, and a measurement that scores suites between 0 and 1. Its empirical studies on seven programs, five coverage criteria, and over 3,000 faulty versions support the claim that higher adequacy scores go with higher fault detection effectiveness. If the claim holds, practitioners without reliable oracles can use the score to compare suites, guide construction, and decide when to stop testing.","feed_headline":"Adequacy score predicts fault detection in metamorphic testing","feed_subtitle":"A k-MR coverage measure tells how thorough metamorphic tests are and when to stop.","key_machinery":"The central object is the association relationship $\\mathrm{Coop}$ between source inputs and metamorphic relations, which records which MRs are used together with which source inputs to construct metamorphic groups (MGs). The k-MR coverage criterion requires that every test requirement of an underlying test-case adequacy criterion $C_c$ be satisfied by at least one source input associated with at least $k$ distinct MRs. The adequacy measurement averages over all requirements the maximum, among satisfying source inputs, of the capped fraction $|SRO(t, \\mathrm{Coop})|/k$, yielding a value in $[0,1]$ that couples ordinary input coverage with the diversity of necessary properties brought to bear on each covered requirement.","core_discovery":"The paper's central claim is that the degree of adequacy computed from the k-MR coverage criterion is an effective indicator of a metamorphic test suite's fault detection effectiveness. Specifically, across seven subject programs, fault detection effectiveness and fault detection rates increased monotonically with the adequacy level, while increasing the parameter k produced diminishing returns; the paper recommends k=2 when source coverage is IO-CTF and k=3 for the other four coverage criteria studied. An additional outcome was the detection of a real-life fault in print_tokens2, a widely studied program, which the paper attributes to MT's ability to check necessary properties over multiple executions.","pith_inferences":["The absence of a random baseline means that a portion of the observed effectiveness could stem from the larger number of MGs that higher adequacy requires; comparing adequacy-guided and random suites with equal MG counts would isolate the criterion's contribution.","Because the k-MR criterion reuses any underlying coverage criterion $C_c$, it could be layered onto modern coverage notions, such as neuron coverage for deep learning systems, to give metamorphic adequacy scores for AI testing.","The measurement could serve as a fitness function for search-based test generation that optimizes suites for adequacy and fault detection simultaneously, an application the paper leaves unexplored."],"forward_implications":["Testers can use the adequacy score as a stopping rule: suites in higher adequacy intervals consistently detect more faulty versions, so reaching a target level gives a defensible reason to halt.","With IO-CTF-based source coverage, k=2 already captures most of the benefit; with I-choice, I-choice-pair, statement, or branch coverage, k=3 is the recommended setting.","Achieving high adequacy costs more metamorphic groups, especially under stronger criteria such as IO-CTF, so the measurement makes the tradeoff between thoroughness and test budget explicit.","The criteria give MT an adequacy theory comparable to code coverage for conventional testing, filling a gap that the ISO/IEC/IEEE software testing standard leaves open."],"supporting_citations":[{"why":"Supplies the definition of test adequacy criteria and the underlying coverage criteria (each-choice-used and all-combination) that the k-MR criterion reuses for source inputs.","marker":"[25]"},{"why":"METRIC+ is the MR identification technique that produced the MR pools used to construct the metamorphic test suites.","marker":"[17]"},{"why":"muJava generates the Java mutants that serve as the faulty versions used to measure fault detection effectiveness.","marker":"[42]"},{"why":"Proteum generates the C mutants for the three C subject programs, supplying the ground truth for fault detection.","marker":"[43]"},{"why":"Provides the simple greedy algorithm used to generate test suites whose adequacy falls into each target interval for RQ2 and RQ3.","marker":"[52]"},{"why":"Defines the formal notions of metamorphic relation and metamorphic group that the k-MR criterion is built on.","marker":"[32]"}],"fun_headline_variants":["k-MR coverage: a new measure for metamorphic test quality","New adequacy criteria rank metamorphic test suites by fault finding","Fault detection improves with k-MR coverage in metamorphic testing","Metamorphic test adequacy: new criteria predict fault detection","Real fault caught by new metamorphic test adequacy metric"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rise in fault detection with higher adequacy is caused by the adequacy criterion itself, rather than by the larger number of metamorphic groups that higher adequacy requires, because the empirical comparison does not hold suite size or MG count constant.","fun_headline_variants_meta":{"raw":{"variants":["k-MR coverage: a new measure for metamorphic test quality","New adequacy criteria rank metamorphic test suites by fault finding","Fault detection improves with k-MR coverage in metamorphic testing","Metamorphic test adequacy: new criteria predict fault detection","Real fault caught by new metamorphic test adequacy metric"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000944,"raw_usage":{"total_tokens":4045,"prompt_tokens":972,"completion_tokens":3073,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2989}},"tokens_in":588,"tokens_out":3073,"duration_ms":22178,"temperature":1.0,"reasoning_tokens":2989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:13:27.970250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the seven subject programs and build two sets of metamorphic suites with equal numbers of MGs: one set selected greedily to reach high k-MR adequacy and one set drawn uniformly at random. If the random suites detect as many mutants as the adequacy-guided suites at every MG count, the measurement's apparent predictive power would reduce to test-volume effects.","supporting_citations":[{"cited_title":"Software unit test coverage and adequacy,","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of test adequacy criteria and the underlying coverage criteria (each-choice-used and all-combination) that the k-MR criterion reuses for source inputs."},{"cited_title":"METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output Domains,","cited_arxiv_id":null,"evidence_quote":"METRIC+ is the MR identification technique that produced the MR pools used to construct the metamorphic test suites."},{"cited_title":"muJava: a mutation system for java,","cited_arxiv_id":null,"evidence_quote":"muJava generates the Java mutants that serve as the faulty versions used to measure fault detection effectiveness."},{"cited_title":"Proteum/IM 2.0: An integrated mutation testing environment,","cited_arxiv_id":null,"evidence_quote":"Proteum generates the C mutants for the three C subject programs, supplying the ground truth for fault detection."},{"cited_title":"Using mutation analysis for assessing and comparing testing coverage criteria,","cited_arxiv_id":null,"evidence_quote":"Provides the simple greedy algorithm used to generate test suites whose adequacy falls into each target interval for RQ2 and RQ3."},{"cited_title":"Metamorphic testing: A review of challenges and opportunities,","cited_arxiv_id":null,"evidence_quote":"Defines the formal notions of metamorphic relation and metamorphic group that the k-MR criterion is built on."}],"review_version":1}