Pith. sign in

REVIEW 3 major objections 3 minor 57 references

Provably Lossless Acceleration of DNN Mutation Testing via Memoization

T0 review · 3 major / 3 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mure makes DNN mutation testing lossless and faster by reusing shared prefix computations.

desk verdict Useful first lossless DNN-mutation accelerator with strong experiments, but the soundness proof as written doesn't cover parallel-branch functional models and needs a fix before the 'provably' claim stands. read the letter →

arxiv 2607.18582 v1 pith:4OZ2RGK2 submitted 2026-07-20 cs.SE

Ali Ghanbari , Ben Greenman , Sasan Tavakkol , Shibbir Ahmed This is my paper
classification cs.SE
keywords mutationtestingdeepneuralnetworksmemoizationlosslessaccelerationscorelayerdependencegraphmutantchoppingKeras
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mutation testing for deep neural networks is expensive: each mutant must be run over the test set, and most mutants share most of the original network's computation. This paper introduces Mure, a framework that runs each mutated suffix only once per group of mutants and reuses the unchanged prefix from the original model, whose activations are memoized. The authors prove that Mure is sound—its mutation scores are identical to exhaustive vanilla mutation testing—and that it guarantees speed-up whenever multiple mutants share a non-trivial prefix. On 15 models, Mure cut mutation-testing cost by 44.54% on average with zero mutation-score error, while lossy baselines reached higher speeds but reported errors up to double digits. A sympathetic reader should care because Mure removes the accuracy–speed trade-off for the common case of non-structure-altering mutants.

What carries the argument

The load-bearing mechanism is the layer dependence graph plus the memoization boundary. The dependence graph maps each layer to the layers that feed it; from it, Mure computes the memoization boundary K—the unmutated layers before the earliest mutated layer on which the mutated region depends. Mutants sharing the same K are grouped, the original model is instrumented at K to build a memo table of activations over the test set, and each mutant is chopped into a smaller network that takes the memoized activations as inputs. The chopped mutant reuses the exact layer functions of the mutant from the earliest mutated layer onward, so its output is identical to the full mutant's if the removed pre

What would settle it

Run Mure and vanilla mutation testing on the same mutant set that includes a structure-altering mutator (e.g., Layer Removal) on a model with residual connections; if the mutation scores differ, the soundness theorem does not extend to that operator.

Watch

Extended reading notes

Core claim

The paper's central claim is that DNN mutation testing can be accelerated without any loss in mutation score by memoizing the common prefix: for each mutant, only the layers at or after the earliest mutated layer are executed, and the inputs to that suffix are read from a memo table of activations computed once from the original model. Formally, the authors define a chopped mutant and prove (Theorem 4.4) that its output on every data point equals the full mutant's output, and (Theorem 4.6) that memoized testing is never more expensive than vanilla testing, being strictly cheaper when a cluster has more than one mutant and the shared prefix has non-zero cost.

Load-bearing premise

Every mutant must keep the network's layers, edges, inputs, and outputs exactly the same, changing only the internal function of some layers—so Mure's guarantee does not cover structure-altering mutators such as Layer Removal.

Editorial extensions

If this is right

  • Any workflow that relies on exact mutation-testing outcomes—test prioritization, fault localization, repair, robustness evaluation—can now get identical results at roughly half the cost, without revalidating downstream conclusions.
  • The speed-up guarantee is structural: any set of mutants sharing an unmutated prefix yields a strict cost reduction, so gains scale with model depth and with mutation operators that touch later layers.
  • Because mutation-score error is zero by construction, Mure can serve as a drop-in replacement for vanilla mutation testing in existing pipelines rather than as a heuristic requiring calibration.
  • The monotonic degradation with mutation selection ratio (still >20% speed-up at 5% mutated neurons) gives users a predictable cost model for deciding how aggressively to mutate.
  • The formal framework of chopped mutants and memoization boundaries applies to any layered computation whose mutants share prefixes, pointing beyond Keras classifiers toward other model families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The speed-up bound is parameterized by cluster size and prefix cost; a natural extension is to derive a closed-form lower bound on speed-up in terms of mutation selection ratio and layer depth, which the paper only explores empirically.
  • Mure assumes inference-mode determinism; applying it to stateful models (e.g., stateful RNNs) or training-mode evaluation would require resetting states or re-deriving the memoization boundary—an extension the authors hint at but do not formalize.
  • The chopping recipe is not specific to mutation testing: any batch of near-identical model variants sharing an unmutated prefix (e.g., pruned models, quantized variants, or ensemble members) could be evaluated with the same memoization trick.
  • The paper's GPU caveat—that floating-point non-determinism may make Mure's scores drift from vanilla—suggests a testable follow-up: quantify that drift and, if needed, add tolerance checks or deterministic kernels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper presents Mure, a memoization-based framework for lossless acceleration of DNN mutation testing. The idea is to execute the unmutated prefix of the original model once, store activations at a memoization boundary, and then test chopped mutants that reuse those activations. The paper formalizes DNNs as DAGs, defines the chopped mutant M★, proves soundness with respect to vanilla mutation testing (Theorem 4.4), and proves an acceleration guarantee under cost axioms (Theorem 4.6). The empirical evaluation on 15 models reports a 44.54% average speedup with zero mutation-score error, comparing favorably with lossy baselines DM# and BSS. RQ2 studies the effect of mutation generation selection ratio, showing monotonic but non-negligible speedups up to 5% mutated neurons.

Significance. If the soundness proof is corrected, Mure is a meaningful contribution: it appears to be the first lossless acceleration technique for DNN mutation testing, with a clean memoization idea and a formal framework. The empirical evaluation is unusually careful—300–900 repeated measurements per configuration, confidence intervals, non-parametric tests, and a public replication package. The centerpiece, Theorem 4.4, is geometrically plausible and the implementation appears to follow a natural fix discussed below. However, as written the formalization does not support the claimed generality over functional DAGs, and the paper explicitly acknowledges in §6.2 that the proofs are not machine-checked. The gaps identified below are local and likely repairable, so the contribution remains worthwhile after a major revision.

major comments (3)
  1. [§4 (V★ definition), Lemma 4.2, Lemma 4.3, Theorem 4.4] The definition of V★ = {v reachable from mmi} ∪ I★ ∪ O is too narrow for functional DAGs with independent branches. For a two-branch network or residual block, an unmutated branch layer with τ(v) ≥ τ(mmi) that is not reachable from mmi is needed for the output, but is neither in V★ nor in K. Algorithm 4 instead includes all layers j ∈ [mmi, L], so M★_Alg contains those layers while M★ does not; the bijection h in Lemma 4.2 cannot be constructed, and the equality {v_t | mmi ≤ t ≤ L} = V★ ∩ V in Lemma 4.3 is false. Theorem 4.4 therefore does not cover the functional architectures in Table 1, and the §8 claim that soundness holds 'regardless of network architecture' is unsupported. The fix is local: redefine V★ as {v | τ(v) ≥ τ(mmi)} ∪ I★ and rework Lemmas 4.2–4.3; the implementation already appears to follow this broader definition.
  2. [§4, unnumbered Φ′ definition] The printed piecewise definition of Φ′(v) is inverted: it assigns Φ(v) to mutated layers and 'mutated layer function' to unmutated layers. The surrounding prose and the proof of Lemma 4.3 use the opposite reading (e.g., 'Since v_mmi is, by definition, mutated, Φ★(v_mmi) evaluates to Φ′(v_mmi)'). The formal definition must be corrected; otherwise it contradicts the proofs.
  3. [§4.2, Lemma 4.5] Lemma 4.5 assumes that all mutants in C share the same earliest mutated layer index mmi, but Algorithm 1 clusters mutants by K. Mutants in different branches can have identical K while different mmi, e.g., K={input}. The proof's cost decomposition uses a single prefix P and mmi, which is not justified. The inequality is likely repairable by taking mmi_min = min_{M'∈C} mmi(M'), observing that K consists of ancestors with τ < mmi_min, and showing all layers before mmi_min are unmutated. Please rewrite the proof accordingly.
minor comments (3)
  1. [General] The paper is generally well written, but there are several small typos and formatting issues: Lemma 4.3's title reads 'Eqivalence'; Theorem 4.4's statement has an unclosed parenthesis ('M★_Alg((a_k(x) k∈K)=M′(x)'); the abstract says 'more than 20% speed-up' at 5% mutation selection ratio, while Fig. 2 appears to show values close to, and in some cases below, 20% for LeNet-5; and the maximum loss values 12.88% / 19.13% cited in §5.4 are not directly visible in Table 2, so they should be reported in a table or in the text.
  2. [Algorithm 4] The sequential branch of chopMutant assumes K={k} and accesses shape(MT[k]). If mmi=0, K is empty and the algorithm would fail. In practice mutators probably never mutate layer 0, but the boundary case should be explicitly handled or excluded in the formal presentation.
  3. [§5.4] The paper says DM# and BSS incur average mutation-score error of 3.29% and 4.99%, respectively, but Table 2's last row shows 2.09% for DM# and 11.46% for BSS; the averages are computed across the seven rows but are not displayed. A column or a note with the averages would help the reader verify the headline numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: Mure's soundness and acceleration results are derived from the paper's own formal model and stated cost axioms, not from fitted values or self-cited results.

full rationale

The claimed derivation chain is not circular. Theorem 4.4 is obtained from Lemma 4.2 (Algorithm 4's chopMutant equals the theoretical chopped mutant M*) and Lemma 4.3 (M* equals the full mutant M'), and these lemmas are argued from the formal definitions in Section 4 rather than from the conclusion being assumed. The acceleration theorem in Section 4.2 is explicitly conditional on the stated cost axioms C1-C3 and follows algebraically; it is not a fitted prediction. The empirical speed-up and mutation-score comparisons are measurements against external baselines (vanilla, DM#, BSS) and are not inputs to the proof. Self-citations to Ghanbari's prior work appear in related-work, baseline, and mutator-selection contexts, but they do not carry the soundness or acceleration argument. The scope caveat about structure-altering mutators is a stated assumption, and Section 6.2 explicitly notes the proofs are not machine-checked; these are correctness/validity risks rather than circularity. Even if the functional-DAG branching issue noted in the skeptic headline makes Lemma 4.2/4.3 incomplete for some architectures, that would be a proof gap, not a reduction of the theorem to its inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two classes of assumptions: (i) the mutant preserves the model's graph structure and (ii) unmutated layers are deterministic functions of their inputs during inference. The acceleration theorem additionally adopts three cost axioms (C1–C3) about how evaluation cost decomposes. These are stated in the paper, but the first is known to exclude some mutators, and the cost axioms are not empirically validated. No new physical or model entities are postulated; the memoization boundary and chopped mutant are algorithmic constructs.

assumptions (4)
  • domain assumption Mutants do not alter graph structure: M' = (V,E,Φ',I,O) with the same V,E,I,O; only layer functions change.
    Stated in §4: 'both of these proofs rely on the fact that none of our mutators physically delete any of the layers or neurons in the model nor do they add new layers or neurons.' This is required for memoization boundaries to be well-defined; structure-altering mutators are excluded.
  • domain assumption Unmutated layer functions are deterministic and depend only on their immediate inputs during inference.
    Required by §4.1: BatchNorm/Dropout in inference mode, no stateful RNNs, or reset states; used in Lemma 4.3 to equate b_p(x)=a_p(x) for p outside the mutated region.
  • ad hoc to paper Cost function axioms C1 (non-negativity), C2 (additivity), C3 (invariance under graph isomorphism).
    Introduced in §4.2 to prove Lemma 4.5 and Theorem 4.6. C2 and C3 are idealized assumptions about how evaluation cost decomposes; they are not empirically measured.
  • domain assumption Keras internal APIs (_inbound_nodes, layer __call__ reuse, topological ordering) faithfully implement the DAG semantics used in the proofs.
    Algorithms 2 and 4 rely on these APIs; Lemma 4.1/4.2 assume they reflect the true dependency graph and that reusing layer objects in a new model is side-effect-free.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Provably Lossless Acceleration of DNN Mutation Testing via Memoization." pith.science (2026). https://pith.science/paper/4OZ2RGK2

@misc{pith2026260718582,
  author       = {Pith},
  title        = {Pith review of: Provably Lossless Acceleration of DNN Mutation Testing via Memoization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OZ2RGK2}},
  note         = {Machine review of arXiv:2607.18582}
}
read the original abstract

Mutation analysis has recently reemerged in the context of deep neural networks (DNNs) as a promising, but notoriously costly, approach for assessing test dataset adequacy. Existing techniques speed up DNN mutation testing through lossy approximations that trade efficiency for mutation score accuracy. This paper introduces Mure, the first provably lossless framework for accelerating DNN mutation testing via memoization. Mure is based on the idea that DNN mutants and the original model share substantial redundant computation, so during mutation testing, it executes only the mutated suffixes of each mutant and reuses the common prefix from the original model, which is computed only once. We give a formal account of memoized mutation testing, and prove that Mure is sound, i.e., it produces results equivalent to exhaustive vanilla mutation testing, and identify basic conditions under which speed-up is guaranteed. We have implemented Mure and evaluated it on 15 DNN models of various architectures, complexities, and sizes ranging from a few thousands to millions of parameters. This provides empirical evidence that Mure reduces the computational cost of mutation testing by 44.54%, on average. We also observed that while state-of-the-art techniques tend to yield higher acceleration (up to 88.97%, on average), they come at the cost of some error in mutation score. We further analyze the effect of mutation generation selection ratio on the effectiveness of Mure and observed predictable reductions in memoization opportunities with increasing the percentage of mutated neurons. We observed that Mure offers more than 20% speed-up even when as high as 5% of the neurons are mutated.

Figures

Figures reproduced from arXiv: 2607.18582 by the authors.

Figure 1
Figure 1. Mure workflow: artifacts are represented with boxes and arrows represent control flow [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Average speed-up offered by Mure vs. mutation generation selection ratio downstream behavior of the technique. Mure, instead, allows such workflows to use the same mutation-testing outcome as vanilla mutation testing, but faster. 6.2 Threats to Validity While we have provided formal proof for soundness and speed-up for Mure, most factors, such as randomness in higher order mutation and the overhead of mutant choppin… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 5 canonical work pages

  1. [1]

    Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. 2018. Deep Learning for Classical Japanese Literature.CoRRabs/1812.01718 (2018), 1–8. arXiv:1812.01718 http://arxiv.org/abs/ 1812.01718

    arXiv 2018

  2. [2]

    Dahl, Jack W

    George E. Dahl, Jack W. Stokes, Li Deng, and Dong Yu. 2013. Large-scale malware classification using random projections and neural networks. InIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013. IEEE, 3422–3426. doi:10.1109/ICASSP.2013.6638293

    arXiv 2013

  3. [3]

    DeMillo, Richard J

    Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer.Computer11, 4 (1978), 34–41. doi:10.1109/C-M.1978.218136

    arXiv 1978

  4. [4]

    Li Deng. 2012. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Processing Magazine29, 6 (2012), 141–142. doi:10.1109/MSP.2012.2211477

    arXiv 2012

  5. [5]

    Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. 2019. A guide to deep learning in healthcare.Nature Medicine25, 1 (2019), 24–29. doi:10.1038/s41591-018-0316-z

    doi:10.1038/s41591-018-0316-z 2019

  6. [6]

    Li-Chao Feng, Xing-Ya Wang, Shi-Yu Zhang, Rui-Zhi Gao, and Zhi-Hong Zhao. 2022. Mutation Operator Reduction for Cost-effective Deep Learning Software Testing via Decision Boundary Change Measurement.Journal of Internet Technology23, 3 (2022), 601–610. https://jit.ndhu.edu.tw/article/view/2705

    2022

  7. [7]

    Ali Ghanbari. 2024. Decomposition of Deep Neural Networks into Modules via Mutation Analysis. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16- 20, 2024, Maria Christakis and Michael Pradel (Eds.). ACM, New York, NY, USA, 1669–1681. doi:10.1145/3650212.3680390

    arXiv 2024

  8. [8]

    2026.Provably Lossless Acceleration of DNN Mutation Testing via Memoization (Replication Package)

    Ali Ghanbari, Ben Greenman, Sasan Tavakkol, and Shibbir Ahmed. 2026.Provably Lossless Acceleration of DNN Mutation Testing via Memoization (Replication Package). doi:10.5281/zenodo.21447443

    work page doi:10.5281/zenodo.21447443 2026

Show all 57 references
  1. [9]

    Ali Ghanbari and Sasan Tavakkol. 2025. Using Fourier Analysis and Mutant Clustering to Accelerate DNN Mutation Testing. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 2109–2121. doi:1...

    2025

  2. [11]

    2017.Neural Network Methods for Natural Language Processing

    Yoav Goldberg. 2017.Neural Network Methods for Natural Language Processing. Morgan & Claypool Publishers. doi:10.2200/S00762ED1V01Y201703HLT037

    2017 doi

  3. [12]

    Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. 2020. A survey of deep learning techniques for autonomous driving.Journal of Field Robotics37, 3 (2020), 362–386. doi:10.1002/rob.21918

    2020 doi

  4. [13]

    Richard G. Hamlet. 1977. Testing Programs with the Aid of a Compiler.IEEE Trans. Software Eng.3, 4 (1977), 279–290. doi:10.1109/TSE.1977.231145

    1977

  5. [14]

    Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y

    Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. 2014. Deep Speech: Scaling up end-to-end speech recognition. CoRRabs/1412.5567 (2014), 1–12. arXiv:1412.5567 http...

    2014 arXiv

  6. [15]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 770–778. doi:10.1109/CVPR.2016.90

    2016 doi

  7. [16]

    Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure.Scandinavian Journal of Statistics6, 2 (1979), 65–70. http://www.jstor.org/stable/4615733

    1979

  8. [17]

    Qiang Hu, Yuejun Guo, Maxime Cordy, Mike Papadakis, and Yves Le Traon. 2023. MUTEN: Mutant-Based Ensembles for Boosting Gradient-Based Adversarial Attack. In38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. I...

    2023

  9. [18]

    Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Lei Ma, and Yves Le Traon. 2023. Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Aus...

    2023

  10. [20]

    Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi

  11. [21]

    Gunel Jahangirova, Andrea Stocco, and Paolo Tonella. 2021. Quality Metrics and Oracles for Autonomous Vehicles Testing. In14th IEEE Conference on Software Testing, Verification and Validation, ICST 2021, Porto de Galinhas, Brazil, Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Arti...

    2021

  12. [22]

    Gunel Jahangirova and Paolo Tonella. 2020. An Empirical Evaluation of Mutation Operators for Deep Learning Systems. In13th IEEE International Conference on Software Testing, Validation and Verification, ICST 2020, Porto, Portugal, October 24-28, 2020. IEEE, 74–84. doi:10.1109/...

    2020

  13. [23]

    Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing.IEEE Trans. Software Eng.37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62

    2011 doi

  14. [24]

    Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding deep learning system testing using surprise adequacy. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, Joanne M. Atlee, Tevfik Bultan, and Jon Wh...

    2019

  15. [25]

    2009.Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. 2009.Learning multiple layers of features from tiny images. Technical Report 0. University of Toronto, Toronto, Ontario. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf

    2009

  16. [26]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. InAnnual Conference on Neural Information Processing Systems, Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and K...

    2012 doi

  17. [27]

    Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 2006. Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. In2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, ...

    2006 doi

  18. [28]

    Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. 2015. Deep learning.Nature521, 7553 (2015), 436–444. doi:10.1038/NATURE14539

    2015 doi

  19. [29]

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition.Proc. IEEE86, 11 (1998), 2278–2324. doi:10.1109/5.726791

    1998 doi

  20. [30]

    David Lewis. 1987. Reuters-21578 Text Categorization Collection. UCI Machine Learning Repository. doi:10.24432/ C52G6M

    1987

  21. [31]

    Yanhui Li, Weijun Shen, Tengchao Wu, Lin Chen, Di Wu, Yuming Zhou, and Baowen Xu. 2022. How higher order mutant testing performs for deep learning models: A fine-grained evaluation of test effectiveness and efficiency improved from second-order mutant-classification tuples.Inf...

    2022

  22. [32]

    Strong, Clark W

    Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher A. Strong, Clark W. Barrett, and Mykel J. Kochenderfer

  23. [33]

    Jing Liu and Li Song. 2021. Second-Order Mutation Testing Cost Reduction Based on Mutant Clustering using SOM Neural Network Model. InIEEE 45th Annual Computers, Software, and Applications Conference, COMPSAC 2021, Madrid, Spain, July 12-16, 2021. IEEE, 974–979. doi:10.1109/CO...

    2021

  24. [34]

    Yuteng Lu, Weidi Sun, and Meng Sun. 2022. Towards mutation testing of Reinforcement Learning systems.J. Syst. Archit.131 (2022), 102701. doi:10.1016/J.SYSARC.2022.102701

    2022

  25. [35]

    Lauren Lyons and Ali Ghanbari. 2025. On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering. InIEEE Conference on Software Testing, Verification and Validation, ICST 2025, Napoli, Italy, March 31 - April 4, 2025. IEEE, 267–278. doi:10.1109/ICST62...

    2025

  26. [36]

    Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Confer...

    2018

  27. [37]

    Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepMutation: Mutation Testing of Deep Learning Systems. In29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018, M...

    2018 doi

  28. [38]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. InAssociation for Computational Linguistics: Human Language Technologies, Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (Eds.). T...

    2011

  29. [39]

    H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other.The Annals of Mathematical Statistics18, 1 (1947), 50–60. http://www.jstor.org/stable/2236101

    1947

  30. [40]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011. Reading Digits in Natural Images with Unsupervised Feature Learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011. 4. http://ufldl.stanford.edu/housenumbers/nips...

    2011

  31. [41]

    Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey.Adv. Comput.112 (2019), 275–378. doi:10.1016/BS.ADCOM.2018.03.015

    2019 doi

  32. [42]

    Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. InProceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, October 28-31, 2017. ACM, 1–18. doi:10.1145/3132747.3132785

    2017

  33. [43]

    Alessandro Viola Pizzoleto, Fabiano Cutigi Ferrari, Jeff Offutt, Leonardo Fernandes, and Márcio Ribeiro. 2019. A systematic literature review of techniques and metrics to reduce the cost of mutation testing.J. Syst. Softw.157 (2019), 110388. doi:10.1016/J.JSS.2019.07.100

    2019 doi

  34. [44]

    Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen

    Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. Computer...

    2018

  35. [45]

    Weijun Shen, Yanhui Li, Yuanlei Han, Lin Chen, Di Wu, Yuming Zhou, and Baowen Xu. 2021. Boundary sampling to boost mutation testing for deep learning models.Inf. Softw. Technol.130 (2021), 106413. doi:10.1016/J.INFSOF.2020.106413

    2021

  36. [46]

    Weijun Shen, Jun Wan, and Zhenyu Chen. 2018. MuNN: Mutation Analysis of Neural Networks. In2018 IEEE International Conference on Software Quality, Reliability and Security Companion, QRS Companion 2018, Lisbon, Portugal, July 16-20, 2018. IEEE, 108–115. doi:10.1109/QRS-C.2018.00032

    2018

  37. [47]

    Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (...

    2015 arXiv

  38. [48]

    Jeongju Sohn, Sungmin Kang, and Shin Yoo. 2023. Arachne: Search-Based Repair of Deep Neural Networks.ACM Trans. Softw. Eng. Methodol.32, 4 (2023), 85:1–85:26. doi:10.1145/3563210

    2023 doi

  39. [49]

    Florian Tambon, Foutse Khomh, and Giuliano Antoniol. 2023. A probabilistic framework for mutation testing in deep neural networks.Inf. Softw. Technol.155 (2023), 107129. doi:10.1016/J.INFSOF.2022.107129

    2023

  40. [50]

    Macario Polo Usaola and Pedro Reales Mateo. 2010. Mutation Testing Cost Reduction Techniques: A Survey.IEEE Softw.27, 3 (2010), 80–86. doi:10.1109/MS.2010.79

    2010 doi

  41. [51]

    Bo Wang, Yingfei Xiong, Yangqingwei Shi, Lu Zhang, and Dan Hao. 2017. Faster mutation analysis via equivalence modulo states. InProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, July 10 - 14, 2017, Tevfik Bult...

    2017

  42. [52]

    Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang, and Peixin Zhang. 2019. Adversarial sample detection for deep neural network through model mutation testing. InProceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31,...

    2019

  43. [53]

    Yichun Wang, Zhiyi Zhang, Yongming Yao, and Zhiqiu Huang. 2023. A Fine-Grained Evaluation of Mutation Operators for Deep Learning Systems: A Selective Mutation Approach. InProceedings of the 14th Asia-Pacific Symposium on Internetware, Internetware 2023, Hangzhou, China, Augus...

    2023

  44. [54]

    Zan Wang, Hanmo You, Junjie Chen, Yingyi Zhang, Xuyuan Dong, and Wenbin Zhang. 2021. Prioritizing Test Inputs for Deep Neural Networks via Mutation Analysis. In43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 397–4...

    2021

  45. [55]

    Huanhuan Wu, Zheng Li, Zhanqi Cui, and Jianbin Liu. 2022. GenMuNN: A mutation-based approach to repair deep neural network models.Int. J. Model. Simul. Sci. Comput.13, 2 (2022), 2341008:1–2341008:17. doi:10.1142/ S1793962323410088

    2022

  46. [56]

    Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms.CoRRabs/1708.07747 (2017), 1–6. arXiv:1708.07747 http://arxiv.org/abs/1708.07747

    2017 arXiv

  47. [57]

    Zhang, Mark Harman, Lei Ma, and Yang Liu

    Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Trans. Software Eng.48, 2 (2022), 1–36. doi:10.1109/TSE.2019.2962027 Received 2026-01-29; accepted 2026-04-16 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Articl...

    2022

  48. [2020]

    A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.Comput. Sci. Rev.37 (2020), 100270. doi:10.1016/J.COSREV.2020.100270

    2020

  49. [2021]

    Trends Optim.4, 3-4 (2021), 244–404

    Algorithms for Verifying Deep Neural Networks.Found. Trends Optim.4, 3-4 (2021), 244–404. doi:10.1561/ 2400000035

    2021

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.