Pith. sign in

REVIEW 3 major objections 4 minor 20 references

ADEPT: A Unified Framework for Deep Learning Test Adequacy

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ADEPT gives deep-learning test adequacy metrics a single plug-in workflow, so researchers and practitioners can run, compare, and extend metrics that currently live as fragmented prototypes.

desk verdict ADEPT is a genuinely useful unifying framework, but its core claim of faithfully re-implementing existing adequacy metrics is unverified and must be checked before acceptance. read the letter →

arxiv 2608.12144 v1 pith:OWP6572O submitted 2026-08-12 cs.SE cs.LG

Yidi Kao , Shawn Burnham , Tommi Rose Fahy , Ali Ghanbari This is my paper · ORCID
classification cs.SEcs.LG
keywords deeplearningtestingtestadequacyneuroncoveragesurpriseinputdistributionmutationextensibleframeworkreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents an engineering solution to a practical problem: deep-learning test adequacy metrics exist as scattered research prototypes with incompatible preprocessing, configuration, and execution requirements. ADEPT, a Python framework, integrates representative metrics—neuron coverage and its variants, surprise adequacy, input distribution coverage, decision-boundary coverage, and source- and model-level mutation score—under one consistent command-line workflow. It standardizes configuration through YAML files, caches costly intermediate artifacts such as activation traces and mutant models, and reports each score alongside execution metadata. If the framework works as described, users can reproduce, compare, and deploy adequacy metrics without spending days or weeks building or repairing the underlying tooling.

What carries the argument

The load-bearing object is the template-based metric interface, a plug-in abstraction that every ADEPT metric module implements. It separates three concerns: metric-specific processing modules that produce artifacts such as neuron profiles, activation traces, latent-space encodings, decision-boundary models, and mutant collections; a cache manager that stores artifacts keyed by model, dataset, metric, and creation time; and a scoring component that turns artifacts into a final score plus execution metadata. YAML configuration files set metric-specific parameters with framework defaults, which lets a user switch metrics through one command-line entry point without touching code.

What would settle it

Take a model-dataset configuration from a paper that introduced one of these metrics, run ADEPT's implementation and the original implementation on the same inputs, and compare the numerical scores; any divergence in the reported adequacy shows the re-implementation is not faithful to the published metric.

Watch

Extended reading notes

Core claim

The paper's contribution is engineering: it claims that the fragmentation of deep-learning test adequacy tooling can be removed by a uniform plug-in architecture. ADEPT reduces seven metric families—neuron coverage and its variants, likelihood- and distance-based surprise adequacy, input distribution coverage, decision-boundary coverage, and source- and model-level mutation score—to one execution loop. The user supplies the test inputs, the target model, optional training data, and a YAML configuration; ADEPT selects the metric module, runs its required preprocessing, stores costly intermediate artifacts in a cache, computes the score, and writes a JSON report with the score, timing, and cache-usage metadata. The mutation-score branch reports the standard ratio of killed mutants to all mutants. The stated payoff is that researchers and practitioners can reproduce, compare, and deploy these metrics without re-implementing missing pieces or wrestling with incompatible research prototypes.

Load-bearing premise

The whole framework rests on the assumption that ADEPT's re-implementations of the published metrics compute the same values the original metric papers would; if any implementation deviates, the unified workflow silently reports the wrong adequacy.

Editorial extensions

If this is right

  • Running the same model and test set under multiple adequacy metrics becomes a matter of changing the metric name and config, which makes cross-metric comparison routine.
  • Repeated evaluations become cheaper because intermediate artifacts such as activation traces and mutant models are generated once and reused across runs.
  • Adding a new adequacy metric requires implementing one module against the template interface rather than building a standalone toolchain.
  • Structured JSON reports with timing and cache metadata make individual runs inspectable and reproducible.
  • Default configurations allow a first run of each metric without reading implementation details.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to check ADEPT's outputs against the original metric implementations on published configurations; the unified interface would then double as a correctness harness for metric definitions.
  • The caching design points toward metric-in-the-loop workflows, such as test-input prioritization and selection loops, in which repeated scoring must be cheap; the paper does not evaluate those workflows.
  • Because score outputs are structured and metadata-rich, ADEPT could also serve as a standard measurement backend for comparing datasets or model versions, a use case the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents ADEPT, a Python framework that unifies several deep learning test adequacy metrics, including neuron-coverage-series metrics (NC, TKNC, KMNC, NBC, SNAC), surprise adequacy (LSA/DSA), input distribution coverage (IDC), deep boundary coverage (DBC), and source- and model-level mutation score (SLMS/MLMS), under a single command-line workflow. The framework provides a template-based metric interface, YAML configuration management, caching of intermediate artifacts, and structured JSON reporting. The paper describes the architecture in Section 2, usage in Section 3, and related work in Section 4, but it contains no experimental evaluation, no comparison with original implementations, and no test suite or example run with expected values.

Significance. If ADEPT's implementations faithfully reproduce the original metrics, the framework would be a valuable community asset: it directly addresses the fragmentation of DL adequacy tooling, and the caching and extensibility design are sensible engineering choices. The public Zenodo archive, demo video, and clearly described processing modules are concrete strengths. However, the central claim that ADEPT integrates representative adequacy techniques is currently supported only by architectural description, not by evidence that the metric scores are correct. For a tool paper whose contribution is faithful integration, this evidentiary gap is decisive in assessing significance.

major comments (3)
  1. [§2.2, §2.4, §5] The load-bearing claim that ADEPT faithfully reproduces NC-series, LSA/DSA, IDC, DBC, SLMS, and MLMS is unverified. Section 2.2 states that ADEPT 'carries out MLMS à la DeepMutation++ and SLMS à la DeepCrime,' and Section 2.4 defines mutation score as |killed mutants| / |all mutants|, but the manuscript provides no numerical comparison against the original DeepMutation++, DeepCrime, DeepGauge, or IDC implementations, no unit tests, no example run with expected scores, and no evaluation dataset. Because the framework abstracts metric internals behind a uniform interface, a consistent and working pipeline is not evidence of metric correctness. The authors should add a reproducibility study that runs ADEPT and the original tools on at least one common benchmark (e.g., MNIST or CIFAR-10) and reports score differences, or otherwise provide a test suite with hand-computed expected scores for each metric.
  2. [§2.2, Table 1] Several accuracy-sensitive design choices are described only at the parameter-name level, leaving room for discrepancies that would change reported scores. For KMNC/NBC/SNAC, the neuron profiling range depends on training-data fractions and layer exclusion; for LSA, the variance threshold affects dimensionality reduction before KDE fitting; for IDC, coverage depends on the VAE latent space and interaction strength t; for DBC, bisection convergence and maximum points per class pair determine boundary coverage; and for SLMS/MLMS, the mutant killing criterion and mutation operator selection determine the mutation score. Table 1 lists these parameters but does not specify the default values or the exact formulas used. The paper should document these defaults and, more importantly, show that the chosen settings reproduce the original metrics' outputs.
  3. [§1, §4] The paper's practical motivation is that existing tools are difficult to install, configure, and run, and that ADEPT reduces this burden, but no evidence is provided for this usability claim. Section 1 cites practitioner interviews [18] and Section 4 argues that ADEPT provides 'consistent execution and reuse,' but there is no user study, no installation-time comparison, no command-run demonstration with timing or error rates, and no discussion of supported model types beyond 'Keras-based' models. At minimum, the authors should include a small case study showing the end-to-end workflow on a standard model and dataset, including successful execution of all supported metrics and the produced scores.
minor comments (4)
  1. [§5] The sentence 'ADEPT are publicly available on Zenodo' uses plural verb agreement; it should be 'ADEPT is publicly available on Zenodo.'
  2. [§3, §5, reference [7]] Section 3 refers to 'the GitHub repository [7]' and the README, but reference [7] is the Zenodo record. The authors should clarify whether the source code is hosted on GitHub or only archived on Zenodo, and make the citation consistent.
  3. [Table 1] In Table 1, the parameter name 'exclude_layer' appears without a space before the following text, and the table would be easier to read if parameter names were set in a monospaced font. This is a formatting issue only.
  4. [Abstract and §2.1] The abstract and Section 2.1 mention 'source- and model-level mutation score' but the body sometimes uses 'source-/model-level mutation score' and 'SLMS and MLMS.' Using one consistent term throughout would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: ADEPT is a re-implementation and integration framework, and its adequacy scores are defined by cited external metrics, with only a non-load-bearing self-citation to the authors' own archive.

full rationale

The paper contains no derivation chain from which a result is claimed to follow from first principles. ADEPT's contribution is an engineering integration of existing adequacy metrics, and the scores it reports are defined by the cited original papers rather than derived within ADEPT; for example, mutation score is stated as MS = |killed mutants| / |all mutants|, which is the classic definition from DeepMutation++ and DeepCrime, not a new result. The statements that ADEPT carries out MLMS and SLMS according to those external prior works defer to independent sources, not to the present authors, so no self-citation chain is load-bearing. The only self-reference is reference [7], the authors' own Zenodo archive, which is cited solely for tool availability and default configuration documentation, and it does not justify any adequacy value or mathematical claim. The unverified fidelity of the re-implementations is a validation and correctness risk, not circularity: a wrong re-implementation would produce incorrect outputs, but it would not make the paper's workflow claim equivalent to its inputs by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is repackaged as a derivation. Accordingly, the circularity burden is minimal, and the self-citation is not load-bearing.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters or invented entities are introduced. The framework depends on the validity of prior adequacy metrics and on the correctness of ADEPT's re-implementations; the latter is unverified.

assumptions (3)
  • domain assumption The original adequacy metrics (neuron coverage, surprise adequacy, input distribution coverage, boundary coverage, mutation score) are valid measures of test adequacy.
    The paper builds on these metrics without questioning their validity; it assumes they are meaningful for assessing DL test sets.
  • domain assumption The reference implementations cited in the paper (DeepGauge, DeepXplore, DeepMutation++, DeepCrime) are correct representations of those metrics.
    The paper treats these tools as ground truth for the metrics, but does not independently verify them.
  • ad hoc to paper ADEPT's internal re-implementations faithfully reproduce the original metrics.
    This premise is introduced by the paper without supporting evidence; no comparison against original tools is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADEPT: A Unified Framework for Deep Learning Test Adequacy." pith.science (2026). https://pith.science/paper/OWP6572O

@misc{pith2026260812144,
  author       = {Pith},
  title        = {Pith review of: ADEPT: A Unified Framework for Deep Learning Test Adequacy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWP6572O}},
  note         = {Machine review of arXiv:2608.12144}
}
read the original abstract

Over the past decade, many test adequacy metrics have been proposed for deep learning that characterize test dataset adequacy from different perspectives, e.g., neuron activation behavior, latent feature coverage, decision-boundary exploration, etc. However, these metrics are typically released as independent research prototypes with substantially different installation and preprocessing requirements, execution workflows, and configuration mechanisms. These complications make them quite difficult to reproduce, compare, and adopt in research work and practical deployment alike. In this paper, we present the engineering details of ADEPT, a framework that integrates representative adequacy techniques, including neuron-coverage-based metrics, surprise adequacy, input distribution coverage, boundary coverage, and source- and model-level mutation score, under a consistent execution workflow. ADEPT provides a template-based metric interface with well-defined extension points for integrating new adequacy metrics. Furthermore, it provides YAML-based configuration management, preprocessing-cache reuse, and structured result reporting, making it easy to use in any research and development workflows. ADEPT is designed for researchers and practitioners who wish to reproduce and apply adequacy metrics without spending days or weeks implementing missing tooling or configuring disparate research prototypes. A demo video is available at https://aub.ie/ADEPT_video.

Figures

Figures reproduced from arXiv: 2608.12144 by the authors.

Figure 1
Figure 1. Overview of ADEPT. Inputs marked with an asterisk (*) are optional and are required only by specific metrics. DBC measures test adequacy from a decision-space per￾spective, focusing on how test inputs cover the model’s deci￾sion boundaries. SLMS and MLMS evaluate test datasets based on their ability to distinguish mutant models from the original model. The underlying intuition is that more effective test datasets sh… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

  1. [18]

    Hanmo You, Zan Wang, Bin Lin, and Junjie Chen. 2025. Navigating the Testing of Evolving Deep Learning Systems: An Exploratory In- terview Study. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2726–2738. doi:10.1109/ICSE55347.2025.00106

    arXiv 2025

  2. [1]

    2008.Introduction to Software Testing

    Paul Ammann and Jeff Offutt. 2008.Introduction to Software Testing. Cambridge University Press. doi:10.1017/CBO9780511809163

    doi:10.1017/cbo9780511809163 2008

  3. [2]

    Dwyer, and Mary Lou Soffa

    Swaroopa Dola, Matthew B. Dwyer, and Mary Lou Soffa. 2023. Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing.ACM Trans. Softw. Eng. Methodol.32, 3 (2023), 81:1–81:48. doi:10.1145/3576040

    work page doi:10.1145/3576040 2023

  4. [3]

    Burgess, Xavier Glorot, Matthew M

    Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Ler- chner. 2017. beta-VAE: Learning Basic Visual Concepts with a Con- strained Variational Framework. In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track P...

    work page 2017

  5. [4]

    Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Lei Ma, Mike Papadakis, and Yves Le Traon. 2024. Test Optimization in DNN Testing: A Survey.ACM Trans. Softw. Eng. Methodol.33, 4 (2024), 111:1–111:42. doi:10.1145/3643678

    work page doi:10.1145/3643678 2024

  6. [5]

    Qiang Hu, Lei Ma, Xiaofei Xie, Bing Yu, Yang Liu, and Jianjun Zhao

    work page

  7. [6]

    Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. Deep- Crime: mutation testing of deep learning systems based on real faults. InISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11- 17, 2021, Cristian Cadar and Xiangyu Zhang (Eds.). ACM, 67–78. doi:10.1145/3460319.3464825

    arXiv 2021

  8. [7]

    Yidi Kao, Shawn Burnham, Tommi Rose Fahy, and Ali Ghanbari. 2026. https://zenodo.org/records/21682100. Accessed: 06/26

    work page arXiv 2026

Show all 20 references
  1. [8]

    Jinhan Kim, Robert Feldt, and Shin Yoo. 2018. Guiding Deep Learning System Testing using Surprise Adequacy.CoRRabs/1808.08444 (2018). arXiv:1808.08444http://arxiv.org/abs/1808.08444

    2018 arXiv

  2. [9]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Varia- tional Bayes. In2nd International Conference on Learning Represen- tations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).http: //arxiv.org/abs/1312.6114

    2014 arXiv

  3. [10]

    Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. 2015. Deep learning.Nat.521, 7553 (2015), 436–444. doi:10.1038/NATURE14539

    2015 doi

  4. [11]

    Yue Liu, Lichao Feng, Xingya Wang, and Shiyu Zhang. 2022. Deep- Boundary: A Coverage Testing Method of Deep Learning Software based on Decision Boundary Representation. In22nd IEEE International Conference on Software Quality, Reliability, and Security, QRS 2022 - Companion, G...

    2022

  5. [12]

    Lei Ma, Felix Juefei-Xu, Jiyuan Sun, Chunyang Chen, Ting Su, Fuyuan Zhang, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Comprehensive and Multi-Granularity Test- ing Criteria for Gauging the Robustness of Deep Learning Systems. CoRRabs/18...

    2018 arXiv

  6. [13]

    Lei Ma, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, and Jianjun Zhao. 2019. DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. In26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, Februa...

    2019

  7. [14]

    Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. Deep- Mutation: Mutation Testing of Deep Learning Systems. In29th IEEE International Symposium on Software Reliability Engineering, ISSRE 2018,...

    2018

  8. [15]

    Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepX- plore: Automated Whitebox Testing of Deep Learning Systems.CoRR abs/1705.06640 (2017). arXiv:1705.06640http://arxiv.org/abs/1705. 06640

    2017 arXiv

  9. [16]

    Iqbal H. Sarker. 2021. Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions.SN Comput. Sci.2, 6 (2021), 420. doi:10.1007/S42979-021-00815-1

    2021 doi

  10. [17]

    Youcheng Sun, Xiaowei Huang, and Daniel Kroening. 2018. Testing Deep Neural Networks.CoRRabs/1803.04792 (2018). arXiv:1803.04792 http://arxiv.org/abs/1803.04792

    2018 arXiv

  11. [19]

    Zhang, Mark Harman, Lei Ma, and Yang Liu

    Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022. Machine Learning Testing: Survey, Landscapes and Horizons.IEEE Trans. Soft- ware Eng.48, 2 (2022), 1–36. doi:10.1109/TSE.2019.2962027 Received 2026-06-25; accepted 2026-07-25

    2022

  12. [2019]

    In34th IEEE/ACM International Conference on Auto- mated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019

    DeepMutation++: A Mutation Testing Framework for Deep Learning Systems. In34th IEEE/ACM International Conference on Auto- mated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019. IEEE, 1158–1161. doi:10.1109/ASE.2019.00126

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.