{"id":"56e3dc97-073b-4a42-babc-9d9a528accb0","arxiv_id":"2607.18582","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Mure accelerates deep neural network mutation testing roughly 44% by memoizing shared prefix computations and running only mutated suffixes, with no mutation-score error.","lead":"This paper presents Mure, a framework that speeds up DNN mutation testing by computing shared parts of a network once and reusing them for each mutant instead of rerunning the whole model. It reports a 44.54% average cost reduction with zero mutation-score error, backed by a formal soundness proof.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Soundness proof omits independent branches in functional DAGs: V★ is defined too narrowly for Lemma 4.2/4.3 to hold.","rationale":"The paper's central claim is that Mure is provably lossless, i.e., that Theorem 4.4 establishes exact equivalence to vanilla mutation testing. The reader's weakest assumption focused on structure-altering mutators (Layer Removal), which is a legitimate scope caveat but is explicitly acknowledged and empirically avoided by the chosen mutators. My concern is more load-bearing: even within the paper's stated scope (structure-preserving mutators on functional models), the formal construction of the chopped mutant M★ in §4 excludes layers in independent parallel branches that are topologically after mmi but not reachable from mmi. This makes the proof of Lemma 4.2 and Lemma 4.3 invalid for such DAGs. The empirical evaluation may still be sound—the implementation likely executes all suffix layers and the reported 44.54% speedup with zero mutation-score error is credible, with repeated measurements and a public replication package—but the formal theorem as written does not cover the generality claimed in the conclusions. This is a fixable proof gap rather than a refutation of the empirical results, so the reader's CONDITIONAL verdict remains appropriate; I do not propose moving it to ACCEPT or REJECT. I disagree with the reader's choice of weakest assumption because the deeper internal gap is the V★ definition, not merely the external structure-mutation caveat.","tokens_in":25997,"tokens_out":16460,"duration_ms":193213,"concrete_test":"Build the minimal Keras functional model: input → Dense layer 1 → Activation layer 2; input → Dense layer 3 → Activation layer 4; Concatenate([layer2, layer4]) → layer 5 → Dense output layer 6. Mutate only layer 1 (e.g., Gaussian fuzzing), so mmi = 1 and K = {0}. Run Mure and vanilla mutation testing on a fixed test set and compare per-mutant outputs. Then instantiate the §4 definition of M★ for this DAG: V★ omits layers 3–4, so the output node receives no branch-B input and M★ is not functionally equivalent to the full mutant; hence Lemma 4.2's bijection cannot be constructed. If the Mure outputs match vanilla, the algorithm is likely correct but V★ must be redefined to {v | τ(v) ≥ τ(mmi)} ∪ I★ ∪ O and Lemmas 4.2–4.3 re-proved before the soundness theorem can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §4, the formal chopped mutant M★ is defined with V★ = {v reachable from mmi} ∪ I★ ∪ O. For a functional DAG with two parallel branches joined before the output (e.g., input → branch A and input → branch B, then Concatenate), a mutant that mutates only a layer in branch A has mmi inside branch A. Layers of branch B that lie topologically after mmi but are not reachable from mmi are then omitted from V★. Algorithm 4 nevertheless executes all layers j ∈ [mmi, L], so it includes branch B; Lemma 4.2's required bijection h : M★ → M★_Alg cannot be constructed because those branch-B layers have no image in V★. Lemma 4.3's induction additionally asserts {v_t | mmi≤t≤L} = V★∩V, which fails in exactly this branching case. Consequently Theorem 4.4 does not establish soundness for general functional Keras models. This is independent of the structure-altering-mutator caveat: it arises even with the paper's own weight/neuron mutators on an ordinary two-branch functional model. The implementation may still be correct because chopMutant rebuilds the full suffix from mmi onward, but the 'provably' claim for arbitrary architectures is unsupported as written.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Mure, a memoization-based framework for lossless acceleration of DNN mutation testing. The idea is to execute the unmutated prefix of the original model once, store activations at a memoization boundary, and then test chopped mutants that reuse those activations. The paper formalizes DNNs as DAGs, defines the chopped mutant M★, proves soundness with respect to vanilla mutation testing (Theorem 4.4), and proves an acceleration guarantee under cost axioms (Theorem 4.6). The empirical evaluation on 15 models reports a 44.54% average speedup with zero mutation-score error, comparing favorably with lossy baselines DM# and BSS. RQ2 studies the effect of mutation generation selection ratio, showing monotonic but non-negligible speedups up to 5% mutated neurons.","tokens_in":1737,"tokens_out":2519,"duration_ms":148347,"significance":"If the soundness proof is corrected, Mure is a meaningful contribution: it appears to be the first lossless acceleration technique for DNN mutation testing, with a clean memoization idea and a formal framework. The empirical evaluation is unusually careful—300–900 repeated measurements per configuration, confidence intervals, non-parametric tests, and a public replication package. The centerpiece, Theorem 4.4, is geometrically plausible and the implementation appears to follow a natural fix discussed below. However, as written the formalization does not support the claimed generality over functional DAGs, and the paper explicitly acknowledges in §6.2 that the proofs are not machine-checked. The gaps identified below are local and likely repairable, so the contribution remains worthwhile after a major revision.","major_comments":[{"comment":"The definition of V★ = {v reachable from mmi} ∪ I★ ∪ O is too narrow for functional DAGs with independent branches. For a two-branch network or residual block, an unmutated branch layer with τ(v) ≥ τ(mmi) that is not reachable from mmi is needed for the output, but is neither in V★ nor in K. Algorithm 4 instead includes all layers j ∈ [mmi, L], so M★_Alg contains those layers while M★ does not; the bijection h in Lemma 4.2 cannot be constructed, and the equality {v_t | mmi ≤ t ≤ L} = V★ ∩ V in Lemma 4.3 is false. Theorem 4.4 therefore does not cover the functional architectures in Table 1, and the §8 claim that soundness holds 'regardless of network architecture' is unsupported. The fix is local: redefine V★ as {v | τ(v) ≥ τ(mmi)} ∪ I★ and rework Lemmas 4.2–4.3; the implementation already appears to follow this broader definition.","section":"§4 (V★ definition), Lemma 4.2, Lemma 4.3, Theorem 4.4"},{"comment":"The printed piecewise definition of Φ′(v) is inverted: it assigns Φ(v) to mutated layers and 'mutated layer function' to unmutated layers. The surrounding prose and the proof of Lemma 4.3 use the opposite reading (e.g., 'Since v_mmi is, by definition, mutated, Φ★(v_mmi) evaluates to Φ′(v_mmi)'). The formal definition must be corrected; otherwise it contradicts the proofs.","section":"§4, unnumbered Φ′ definition"},{"comment":"Lemma 4.5 assumes that all mutants in C share the same earliest mutated layer index mmi, but Algorithm 1 clusters mutants by K. Mutants in different branches can have identical K while different mmi, e.g., K={input}. The proof's cost decomposition uses a single prefix P and mmi, which is not justified. The inequality is likely repairable by taking mmi_min = min_{M'∈C} mmi(M'), observing that K consists of ancestors with τ < mmi_min, and showing all layers before mmi_min are unmutated. Please rewrite the proof accordingly.","section":"§4.2, Lemma 4.5"}],"minor_comments":[{"comment":"The paper is generally well written, but there are several small typos and formatting issues: Lemma 4.3's title reads 'Eqivalence'; Theorem 4.4's statement has an unclosed parenthesis ('M★_Alg((a_k(x) k∈K)=M′(x)'); the abstract says 'more than 20% speed-up' at 5% mutation selection ratio, while Fig. 2 appears to show values close to, and in some cases below, 20% for LeNet-5; and the maximum loss values 12.88% / 19.13% cited in §5.4 are not directly visible in Table 2, so they should be reported in a table or in the text.","section":"General"},{"comment":"The sequential branch of chopMutant assumes K={k} and accesses shape(MT[k]). If mmi=0, K is empty and the algorithm would fail. In practice mutators probably never mutate layer 0, but the boundary case should be explicitly handled or excluded in the formal presentation.","section":"Algorithm 4"},{"comment":"The paper says DM# and BSS incur average mutation-score error of 3.29% and 4.99%, respectively, but Table 2's last row shows 2.09% for DM# and 11.46% for BSS; the averages are computed across the seven rows but are not displayed. A column or a note with the averages would help the reader verify the headline numbers.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The gap in the formal definition of V★ is the main reason for my recommendation. It is a genuine shortcoming of the proof as written, but the fix is conceptually simple and the implementation seems to follow the corrected notion. I would suggest asking the authors to (1) redefine V★ to include the full suffix from mmi, (2) re-check Lemma 4.2's bijection and Lemma 4.3's induction under the corrected definition, and (3) restate Lemma 4.5 using the minimum mmi per cluster. If those changes are made carefully, the paper could become acceptable. I also recommend the editor ask a formal-methods-minded reviewer to re-verify the corrected proofs, given the paper's own caveat in §6.2 that the proofs are not machine-checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful and mostly credible first lossless acceleration for DNN mutation testing, but the formal soundness claim as written does not cover branching functional graphs; the proof has a real gap that is likely fixable, and the empirical work is strong enough to warrant a proper review.\n\nWhat's new: Mure is the first memoization-based lossless accelerator for DNN mutation testing. The idea is simple and appropriate: mutants share prefix computation with the original model; instrument once, memoize boundary activations, chop mutants and execute only suffixes. The formalization is genuinely an attempt at a theorem, not hand-waving, and the experiments are careful: 300–900 repeated measurements, confidence intervals, non-parametric tests, replication package. The 44.54% average speedup at zero mutation-score error, and the monotonic degradation as mutation selection ratio rises, are believable.\n\nWhere it gets soft. The most important problem is in Section 4. As printed, Φ′ is backwards: it assigns the original function to mutated layers and the mutated function to unmutated layers. That is a typo (the prose and proofs use the intended definition), but it needs fixing because the paper is built on formal language.\n\nThe bigger issue is the definition of V★. It is defined as nodes reachable from mmi plus boundary inputs and outputs. That excludes layers in parallel branches that are topologically after mmi but not downstream of mmi. In a two-branch functional model where only one branch is mutated, those other-branch layers are still executed by Algorithm 4, but they have no image in V★. So Lemma 4.2's bijection cannot be constructed, Lemma 4.3's claim that the suffix equals V★∩V is false, and Theorem 4.4 does not establish soundness for general functional Keras models. This is not the structure-altering mutator caveat; it bites under the paper's own weight mutators. It looks fixable—define V★ as the topological suffix {v_t | mmi≤t≤L}, or add a reachability argument that parallel branches are covered—but the \"provably\" wording is unsupported as written.\n\nSmaller issue: Lemma 4.5 assumes all mutants in a cluster share the same mmi. Clustering is by memoization boundary K, not by mmi; the assumption may be derivable for the Keras index order, but it is not argued. And the unsupported \"same mmi\" premise is load-bearing in the acceleration proof. Also note the authors themselves flag the unverified proof risk in §6.2; that is honest but does not erase the gap.\n\nBottom line: the contribution is real, the data are strong, and the proof gap is probably repairable. I would send it to peer review with a clear request to fix V★/Φ′ and tighten Lemma 4.5. I would not desk-reject.","headline":"Useful first lossless DNN-mutation accelerator with strong experiments, but the soundness proof as written doesn't cover parallel-branch functional models and needs a fix before the 'provably' claim stands.","tokens_in":26748,"tokens_out":6378,"would_cite":true,"duration_ms":74511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mure makes DNN mutation testing lossless and faster by reusing shared prefix computations.","keywords":["mutation testing","deep neural networks","memoization","lossless acceleration","mutation score","layer dependence graph","mutant chopping","Keras"],"falsifier":"Run Mure and vanilla mutation testing on the same mutant set that includes a structure-altering mutator (e.g., Layer Removal) on a model with residual connections; if the mutation scores differ, the soundness theorem does not extend to that operator.","tokens_in":25918,"feed_emoji":"⚡","tokens_out":4865,"duration_ms":52612,"temperature":0.7,"pith_summary":"Mutation testing for deep neural networks is expensive: each mutant must be run over the test set, and most mutants share most of the original network's computation. This paper introduces Mure, a framework that runs each mutated suffix only once per group of mutants and reuses the unchanged prefix from the original model, whose activations are memoized. The authors prove that Mure is sound—its mutation scores are identical to exhaustive vanilla mutation testing—and that it guarantees speed-up whenever multiple mutants share a non-trivial prefix. On 15 models, Mure cut mutation-testing cost by 44.54% on average with zero mutation-score error, while lossy baselines reached higher speeds but reported errors up to double digits. A sympathetic reader should care because Mure removes the accuracy–speed trade-off for the common case of non-structure-altering mutants.","feed_headline":"Mure cuts DNN mutation-testing cost 44.5% with zero error","feed_subtitle":"Mure reuses the shared prefix across mutants, so mutation scores match vanilla exactly—while faster rivals accept error up to double digits.","key_machinery":"The load-bearing mechanism is the layer dependence graph plus the memoization boundary. The dependence graph maps each layer to the layers that feed it; from it, Mure computes the memoization boundary K—the unmutated layers before the earliest mutated layer on which the mutated region depends. Mutants sharing the same K are grouped, the original model is instrumented at K to build a memo table of activations over the test set, and each mutant is chopped into a smaller network that takes the memoized activations as inputs. The chopped mutant reuses the exact layer functions of the mutant from the earliest mutated layer onward, so its output is identical to the full mutant's if the removed pre","core_discovery":"The paper's central claim is that DNN mutation testing can be accelerated without any loss in mutation score by memoizing the common prefix: for each mutant, only the layers at or after the earliest mutated layer are executed, and the inputs to that suffix are read from a memo table of activations computed once from the original model. Formally, the authors define a chopped mutant and prove (Theorem 4.4) that its output on every data point equals the full mutant's output, and (Theorem 4.6) that memoized testing is never more expensive than vanilla testing, being strictly cheaper when a cluster has more than one mutant and the shared prefix has non-zero cost.","pith_inferences":["The speed-up bound is parameterized by cluster size and prefix cost; a natural extension is to derive a closed-form lower bound on speed-up in terms of mutation selection ratio and layer depth, which the paper only explores empirically.","Mure assumes inference-mode determinism; applying it to stateful models (e.g., stateful RNNs) or training-mode evaluation would require resetting states or re-deriving the memoization boundary—an extension the authors hint at but do not formalize.","The chopping recipe is not specific to mutation testing: any batch of near-identical model variants sharing an unmutated prefix (e.g., pruned models, quantized variants, or ensemble members) could be evaluated with the same memoization trick.","The paper's GPU caveat—that floating-point non-determinism may make Mure's scores drift from vanilla—suggests a testable follow-up: quantify that drift and, if needed, add tolerance checks or deterministic kernels."],"forward_implications":["Any workflow that relies on exact mutation-testing outcomes—test prioritization, fault localization, repair, robustness evaluation—can now get identical results at roughly half the cost, without revalidating downstream conclusions.","The speed-up guarantee is structural: any set of mutants sharing an unmutated prefix yields a strict cost reduction, so gains scale with model depth and with mutation operators that touch later layers.","Because mutation-score error is zero by construction, Mure can serve as a drop-in replacement for vanilla mutation testing in existing pipelines rather than as a heuristic requiring calibration.","The monotonic degradation with mutation selection ratio (still >20% speed-up at 5% mutated neurons) gives users a predictable cost model for deciding how aggressively to mutate.","The formal framework of chopped mutants and memoization boundaries applies to any layered computation whose mutants share prefixes, pointing beyond Keras classifiers toward other model families."],"fun_headline_variants":["Memoization makes DNN mutation testing losslessly faster","Mure speeds DNN mutation tests 44.5% with zero accuracy loss","Zero-error DNN mutation testing gets 44.5% faster via memoization","Provably lossless DNN mutation testing: 44.5% speedup, no error","Mure: lossless mutation testing for DNNs, 44.5% cheaper"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Every mutant must keep the network's layers, edges, inputs, and outputs exactly the same, changing only the internal function of some layers—so Mure's guarantee does not cover structure-altering mutators such as Layer Removal.","fun_headline_variants_meta":{"raw":{"variants":["Memoization makes DNN mutation testing losslessly faster","Mure speeds DNN mutation tests 44.5% with zero accuracy loss","Zero-error DNN mutation testing gets 44.5% faster via memoization","Provably lossless DNN mutation testing: 44.5% speedup, no error","Mure: lossless mutation testing for DNNs, 44.5% cheaper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3181,"prompt_tokens":829,"completion_tokens":2352,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2247}},"tokens_in":573,"tokens_out":2352,"duration_ms":17205,"temperature":1.0,"reasoning_tokens":2247,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:58:53.057425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Mure and vanilla mutation testing on the same mutant set that includes a structure-altering mutator (e.g., Layer Removal) on a model with residual connections; if the mutation scores differ, the soundness theorem does not extend to that operator.","supporting_citations":[],"review_version":1}