{"id":"3b38c50d-78dc-489c-8e14-d230516bbbd3","arxiv_id":"2608.09318","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"QUMUG uses search algorithms to tune parameterised quantum gate mutations so the resulting mutants pass the current test suite yet remain detectable by a larger suite, yielding mutants several times more challenging than existing fixed-gate tools.","lead":"The paper introduces QUMUG, a search-based tool that creates quantum circuit mutants using adjustable rotation gates, tuned to sneak past a test suite while still changing behavior. The authors report that these mutants are much harder to detect than those from existing tools and push test suites to add more tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-equivalence validation applies a per-test α=0.05 chi-square test across 48–768 CTS test cases with no multiple-testing correction, so equivalent mutants are nearly always labelled non-equivalent; the 94.3% non-equivalent ratio is unsubstantiated.","rationale":"The reader's weakest assumption is CTS completeness; my concern is sharper: even when CTS is accepted as the oracle, the statistical procedure that maps CTS detections to the label 'non-equivalent' is invalid for its intended purpose. With α=0.05 per test and no multiplicity control, an equivalent mutant has near-certain probability of being flagged by at least one of the dozens/hundreds of CTS test cases. This is not a philosophical incompleteness of the oracle; it is a calculable false-positive rate internal to the paper's own Definition 5/5.4. It directly inflates every headline quantity that depends on the non-equivalence label: RQ1.1 counts and rates, RQ1.2 detection-ratio denominators, and RQ1.3 test-suite contribution. The search portion of the paper is a reasonable engineering contribution, and the relative comparison against Muskit/QMutPy may partly survive because all approaches are scored under the same flawed classifier; but the central claim that QUMUG generates truly non-equivalent mutants is not currently evidenced. The proposed concrete test with known-equivalent mutants would settle whether the classification is sound, and a multiplicity-corrected reanalysis would show how much the headline numbers change.","tokens_in":31017,"tokens_out":8327,"duration_ms":92716,"concrete_test":"Generate a set of known-equivalent mutants by, for each circuit, adding an Rz(0) gate (identity) or replacing a gate with a U gate having exactly the parameters of the original gate, then run them through the exact CTS pipeline (same 10 TS/CTS instances, same shots and chi-square threshold). Measure the fraction labelled non-equivalent; expectation under the null is ~5% per test, and with |CTS| tests the fraction classified non-equivalent should be ≈1−(0.95)^{|CTS|} if the classifier is unsound. Alternatively/additionally, recompute the RQ1.1 non-equivalent ratio and RQ1.3 added-test-case medians after applying a Bonferroni or Benjamini-Hochberg correction over the |CTS| comparisons per mutant; if 94.3% and the median-five figures fall materially, the paper's headline claims require revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Definition 5 and Section 5.4 classify a mutant as non-equivalent if at least one CTS test case yields p<0.05 on the chi-square test. For a truly equivalent mutant, the original and mutant output distributions are independent samples from the same distribution, so each test has a 5% false-positive rate. Section 5.3 gives |TS| = 0.4·2^q·3 test cases and Section 5.4 says CTS is five times larger: |CTS| = 6·2^q, i.e. 48 (q=3) to 768 (q=7) tests. The probability that an equivalent mutant is wrongly classified as non-equivalent is 1−(0.95)^{|CTS|}: about 91.5% for q=3 and effectively 100% for q≥5. Thus the reported 94.3% non-equivalent ratio, the non-equivalent mutant counts in Fig. 4, and the RQ1.3 'five additional test cases' figure are dominated by Type I errors. Section 7.4.1 discusses only CTS incompleteness (false negatives) and does not address this false-positive multiple-comparisons problem. The central claim that QUMUG mutants are non-equivalent is therefore not established by the reported validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QUMUG, a search-based approach for generating quantum circuit mutants that pass a given test suite TS while remaining non-equivalent. Mutations are introduced via parameterisable gates (Rx, Ry, Rz, U), with parameters tuned by four search algorithms (GA, HC, (1+1)-EA, random search). The approach is evaluated on 30 quantum circuits (3–7 qubits) drawn from MQT Bench and compared against two existing quantum mutation tools, Muskit and QMutPy. The paper reports that QUMUG mutants are markedly harder to detect, with GA producing on average 494 non-equivalent undetected mutants per program, a 94.3% non-equivalence ratio, and a median of five additional test cases needed to detect them versus one for the baselines. It also presents the first study of higher-order (second-order) quantum mutants, analysing their characteristics and test-case interactions.","tokens_in":31260,"tokens_out":6809,"duration_ms":65428,"significance":"If the empirical claims were fully supported, this would be a valuable contribution: it introduces a new family of parametric mutation operators, provides the first implementation and analysis of higher-order quantum mutants, uses a reasonable experimental design with external baselines, ten repeated runs, statistical testing with effect sizes, and a public repository. The central comparative claim, however, rests on the non-equivalence validation, and that validation currently has a load-bearing multiple-comparisons problem. The approach and benchmarks are promising, but the headline numbers and the comparative conclusion need to be re-established with a sound equivalence criterion before the paper's central claim can be accepted.","major_comments":[{"comment":"The non-equivalence validation has a severe multiple-comparisons problem that undermines the central claim. A mutant is labelled non-equivalent if at least one test case from CTS yields p<0.05 on the chi-square test. For a truly equivalent mutant, each test case is a Bernoulli trial with a 5% false-positive rate, so the probability of being misclassified as non-equivalent is 1−(0.95)^{|CTS|}. Even under the smallest consistent reading of the CTS size, this is about 71% for 3-qubit circuits and above 99% for 5-qubit circuits, and it is effectively 100% for 7-qubit circuits. The reported 94.3% non-equivalence ratio, the non-equivalent mutant counts in Fig. 4, and the RQ1.2/RQ1.3 analyses that filter on 'non-equivalent' are therefore dominated by Type I errors. Section 7.4.1 discusses only CTS incompleteness (false negatives) and does not address this false-positive problem. The authors should apply a multiple-testing correction (e.g., Benjamini-Hochberg) or use an alternative equivalence criterion (e.g., a minimum Hellinger distance or a joint test across CTS) and re-run the affected analyses.","section":"Section 5.4 / Definition 5"},{"comment":"The mutation angle search range was reduced from [-180,180] to [-0.5,0.5] after the authors observed that the vast majority of selected angles in their own preliminary search fell in that range. Because this range is derived from the same search process being evaluated and is used by all QUMUG variants but not by the fixed-gate baselines, the comparative claim that parameterisable-gate mutants are intrinsically more challenging is confounded by this post hoc design choice. A sensitivity analysis with at least one broader range (e.g., [-5,5] or the full [-180,180]) is needed to show that the reported advantage of QUMUG is not an artefact of restricting the search to small perturbations.","section":"Section 5.6"}],"minor_comments":[{"comment":"The formulas for test-suite size yield non-integer values for some circuits: for q=3, 2^q×0.4×3 = 9.6 test cases. Please clarify how rounding is performed and whether the reported sizes are exact or approximate.","section":"Section 5.3"},{"comment":"The statement that CTS is five times larger than TS is inconsistent with the preceding description. With TS using 20% classical plus 20% quantum initialisations, and CTS using the remaining 80% classical initialisations plus 'the same number of quantum initialisations', the implied ratio is either 2.5 or 4, depending on interpretation, not 5. Please correct the text and recompute any quantities that depend on this ratio.","section":"Section 5.4"},{"comment":"P-values are reported inconsistently: some appear as '0.31' or '0.06' while others appear as '<0.05'. Use a uniform convention, e.g., exact p-values or a single threshold notation, to avoid ambiguity.","section":"Table 4"},{"comment":"The labels for cases (5) and (6) are confusing because the text and the caption describe the inclusion relationships in opposite directions. Consider renaming these categories with explicit set equations, such as 'tcd_hom ⊂ tcd_fom' and 'tcd_fom ⊂ tcd_hom'.","section":"Figure 10 / Section 6.2.3"},{"comment":"The abstract says 'three times more challenging' while Section 6.1.2 says 'detected by three times fewer test cases'. Please state the precise metric (detection ratio) in both places so the claim is unambiguous.","section":"Abstract / Section 6.1.2"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the non-equivalence validation. If the authors can re-run the analyses with an appropriate multiple-testing correction or a more conservative equivalence criterion and the main qualitative conclusions survive, the paper would be a serious candidate for acceptance. As it stands, the headline non-equivalence ratio and the mutant counts that depend on it are not supported by the reported statistical procedure. The post hoc angle-range choice is a secondary concern that should also be addressed, but it is not by itself the reason for the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core empirical claim is not supported. The idea—tuning parameterisable gates by search to make hard-to-detect mutants, plus second-order quantum mutants—is genuinely new and the paper is transparent about most of its setup. But the non-equivalence validation has a multiple-testing problem that inflates the headline numbers.\n\nThe paper defines a mutant as non-equivalent if any one of the CTS test cases yields a chi-square p < 0.05. CTS has 48 to 768 test cases depending on qubit count. For a truly equivalent mutant, every test has a 5% false-positive chance, so the probability that at least one test fires is about 91.5% for 3 qubits and effectively 100% for 5+ qubits. That means the 94.3% non-equivalence ratio, the non-equivalent mutant counts in Figure 4, and the 'five additional test cases' result are all contaminated by Type I errors. Section 7.4.1 mentions CTS incompleteness as a threat, but never addresses the false-positive side. This is the load-bearing flaw: without a sound equivalence check, the paper hasn't shown that QUMUG produces behaviourally distinct mutants.\n\nWhat is solid: the mutation operators are a real contribution. Existing tools only do fixed gate mutations; parameterisable gates plus search is a reasonable step, and the first study of higher-order quantum mutants is worth having. The experimental design is otherwise honest: 30 circuits, four search algorithms, two baselines, ten runs, statistical tests with effect sizes, and a repository. The post-hoc restriction of the angle range to [-0.5, 0.5] is a smaller issue; it should be justified more carefully, but it doesn't kill the approach. RQ1.2 should also be re-analysed at the circuit level instead of treating every mutant as independent.\n\nNet: the paper is worth a serious referee, but only with the expectation of heavy revision. The fix is straightforward—apply a multiple-testing correction (e.g., Bonferroni per mutant, or a global test across CTS), or report results under a corrected threshold. Until then, I would not take the three-times claim or the 94.3% figure at face value. I'd rather see it resubmitted after the validation is fixed than published as is.","headline":"Novel approach, broken validation: the non-equivalence test suffers from massive multiple testing, inflating the headline results.","tokens_in":31804,"tokens_out":3898,"would_cite":true,"duration_ms":36727,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces QUMUG, a search-based generator of quantum circuit mutants tuned to pass a given test suite while remaining non-equivalent, and reports that these mutants are three times harder to detect than those from existing tools.","keywords":["quantum mutation testing","quantum circuit mutants","search-based software engineering","genetic algorithm","parameterisable quantum gates","higher-order mutants","test suite improvement"],"falsifier":"Apply an exact state-vector equivalence check to the mutants that QUMUG classifies as non-equivalent; if a substantial fraction of those mutants are exactly equivalent under full-state comparison, then the reported non-equivalence ratio and test-suite-improvement numbers overstate the method's value.","tokens_in":30814,"feed_emoji":"⚛️","tokens_out":9894,"duration_ms":95044,"temperature":0.7,"pith_summary":"The paper introduces QUMUG, a search-based generator of quantum circuit mutants that are tuned to pass a given test suite while remaining behaviourally different from the original circuit. Existing quantum mutation tools apply fixed-gate changes that are easy for almost any test to catch, so QUMUG replaces them with parameterised rotation and universal gates and lets a search algorithm tune the angles. On 30 circuits from six quantum algorithms, the authors report that the genetic algorithm found at least one non-equivalent, undetected mutant in 99.67% of runs, generated an average of 494 such mutants per program with a 94.3% non-equivalence ratio, and produced mutants that are detected by about three times fewer test cases and require a median of five new test cases to catch, versus one for the baselines. The paper also presents the first higher-order quantum circuit mutants and shows that combining two mutations often changes which test cases detect the fault, in ways not predictable from the individual mutations.","feed_headline":"Quantum-circuit mutants bred by search are 3 times harder to detect","feed_subtitle":"Genetic search tunes gate angles so mutant faults survive the test suite, revealing real gaps in quantum tests.","key_machinery":"The load-bearing mechanism is the parameterised-gate mutation operator combined with a two-part fitness function. Each candidate mutant encodes a circuit position, one of five operators (add $R_x$, add $R_y$, add $R_z$, add $U$, replace with $U$), and three floating-point angles restricted in practice to the range $[-0.5,0.5]$. The fitness function minimises the number of test cases that detect the mutant, rewarding closeness to the undetected region; once no test case detects it, the function maximises total Hellinger distance between the original and mutated output distributions, so the search does not simply converge on equivalent mutants. Detection in the evaluation is decided by a chi-square test, and a five-times-larger comprehensive suite is used only after generation to classify mutants as non-equivalent.","core_discovery":"The central claim is that mutant difficulty is controllable by making the mutation itself parameterised. Instead of inserting a fixed gate, QUMUG adds or replaces gates drawn from $R_x(\\theta)$, $R_y(\\theta)$, $R_z(\\phi)$, and the universal $U(\\theta,\\phi,\\lambda)$ gate, and the search tunes the angles so the mutant passes every test in the input suite while still producing a measurable difference in output distributions. The paper's evaluation concludes that this approach produces mutants that are markedly more challenging than fixed-gate mutants: the best search algorithm, a genetic algorithm, required a median of five additional test cases to reveal all of its mutants, whereas mutants from the baseline tools were all detected after adding a single test case. The higher-order analysis further shows that second-order mutants, meaning two mutations in the same circuit and most often placed on different qubits or circuit positions, are detected by a set of test cases different from the union of the detection sets of their constituent first-order mutants in 71.25% of cases, evidence that interactions between faults create behaviours not present in either mutation alone.","pith_inferences":["The reported difficulty gap depends on the chosen oracle: a different statistical test, distance metric, or a state-vector comparator could plausibly shrink or widen the three-fold gap between QUMUG and the fixed-gate baselines.","Because the search already treats each mutant as an array of positions and angles, QUMUG could be embedded in a continuous-integration loop that adds every generated survivor to the test suite and reruns the search until no new non-equivalent mutants appear.","The dominance of second-order mutants placed on different qubits or positions suggests a testable interaction hypothesis: separation between the two mutations may control whether effects cancel or compound, and sweeping that separation directly would reveal the mechanism.","The restriction of gate angles to $[-0.5,0.5]$, chosen after preliminary runs, may itself be responsible for the subtlety of the mutants; repeating the experiment with different angle bounds would show how sensitively the difficulty results depend on this search-space cutoff."],"forward_implications":["Quantum mutation scores computed with fixed-gate tools likely overestimate test-suite quality, because the easy-to-detect mutants inflate kill rates; QUMUG's surviving mutants give a stricter benchmark.","Surviving mutants can be used directly as test-generation targets: each new mutant that passes the suite pinpoints a missing test case, so suites can be strengthened iteratively until the search stalls.","Higher-order quantum mutants should be treated as interaction faults rather than paired first-order faults, since their detection sets usually differ from the union of their components.","The parameterised-gate operators can be reused outside mutation analysis, for instance in fault localisation and circuit repair, where controlling the size of a change is useful.","Diversity, not just difficulty, matters for test-suite improvement: the paper attributes the genetic algorithm's advantage partly to its maintaining a more diverse mutant population."],"supporting_citations":[{"why":"This is the fixed-gate mutation tool whose easy-to-detect mutants motivate the search for harder mutants; its add, remove, and replace operators are the baseline that QUMUG extends.","marker":"[29]"},{"why":"This is the other fixed-gate baseline tool used in the comparison; its mutants define the easy-to-detect reference that QUMUG must beat.","marker":"[13]"},{"why":"This is the authors' earlier empirical study of over 700,000 fixed-gate mutants, supplying the evidence that such mutants are trivial to detect and the output-type categorisation used to select benchmarks.","marker":"[30]"},{"why":"This is the benchmark suite from which the 30 quantum circuits across six algorithms were taken for the evaluation.","marker":"[39]"},{"why":"This provides the UCNOT template method used to generate random quantum-state initialisations for the test suites.","marker":"[57]"},{"why":"This is the classical second-order mutation study whose cost-benefit findings motivate the higher-order mutant analysis and the interpretation of interaction faults.","marker":"[37]"},{"why":"This is the classical genetic-algorithm approach to breeding high-impact undetected mutants that QUMUG adapts to quantum circuits.","marker":"[42]"}],"fun_headline_variants":["Search-tuned quantum mutants evade existing tests","Genetic search breeds quantum faults that dodge tests","Quantum mutant search demands five times more tests","Parameterized quantum faults challenge test suites","Second-order quantum mutants reveal new test gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the assumption that a mutant classified as non-equivalent because the expanded test suite detects it really does differ from the original circuit, since that expanded suite is still a finite sample of initialisations and measurement bases.","fun_headline_variants_meta":{"raw":{"variants":["Search-tuned quantum mutants evade existing tests","Genetic search breeds quantum faults that dodge tests","Quantum mutant search demands five times more tests","Parameterized quantum faults challenge test suites","Second-order quantum mutants reveal new test gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1591,"prompt_tokens":981,"completion_tokens":610,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":597,"tokens_out":610,"duration_ms":6688,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:26:34.657393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply an exact state-vector equivalence check to the mutants that QUMUG classifies as non-equivalent; if a substantial fraction of those mutants are exactly equivalent under full-state comparison, then the reported non-equivalence ratio and test-suite-improvement numbers overstate the method's value.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This is the classical second-order mutation study whose cost-benefit findings motivate the higher-order mutant analysis and the interpretation of interaction faults."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This is the classical genetic-algorithm approach to breeding high-impact undetected mutants that QUMUG adapts to quantum circuits."}],"review_version":1}