{"id":"dbc80029-3be0-45d7-a9cc-6e638b2f1df8","arxiv_id":"2608.12144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ADEPT unifies neuron coverage, surprise adequacy, input distribution coverage, boundary coverage, and mutation score metrics under a single extensible command-line framework.","lead":"ADEPT is a Python framework that packages seven deep learning test adequacy metrics into one command-line workflow with caching and YAML configuration. If the implementations are correct, it removes a major barrier to reproducing and comparing adequacy metrics across research groups.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADEPT's fidelity to the original metrics is unverified; a direct numerical cross-check against the original tools is missing and should gate acceptance.","rationale":"The reader identified the weakest assumption as faithful re-implementation of the original metrics. My independent reading of the paper agrees: the prose repeatedly describes effort savings and consistency without quantifying correctness on a single example. The paper contains no evaluation section at all, and the closest statements ('MLMS à la DeepMutation++', 'SLMS à la DeepCrime') assert alignment rather than demonstrate it. The lack of validation is a correctness risk, not a mismatch with consensus: the definitions cited are public and checkable, which makes a targeted numerical cross-check the right remedy. I therefore keep the conditional verdict, with the concrete test being a direct score comparison against the original releases. I considered whether the absence of experiments is acceptable for a tools/engineering paper billed as SPLASH Companion; it is not, because the paper's own utility claim ('without spending days or weeks implementing missing tooling') presumes the tooling is correct. A self-contained unit test or canned example would be the minimum bar, and the proposed cross-check is the decisive test.","tokens_in":7322,"tokens_out":1353,"duration_ms":11381,"concrete_test":"Run ADEPT and the original reference implementations (DeepGauge or the original release, the official SA implementation, the IDC release, DeepBoundary, DeepMutation++, DeepCrime) on a shared public benchmark, e.g., CIFAR-10 test data with a fixed Keras model, and compare each metric's scores and coverage-element sets. A reasonable criterion: for each metric, ADEPT's score must match the reference score on the same inputs and model within 1% absolute, or the proportion of matched coverage elements must be reported. Publish the comparison table and the commit hash of the ADEPT version evaluated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that ADEPT integrates representative adequacy techniques (NC-series, LSA/DSA, IDC, DBC, SLMS/MLMS) under a consistent workflow, and the reader's condition hinges on faithful re-implementation. The text itself supports the concern: Section 2.2 states that the framework 'carries out MLMS à la DeepMutation++' and 'SLMS à la DeepCrime,' but the arXiv version provides no test suite, no comparison against original tools, no example run with expected values, and no dataset on which scores were validated. The two accuracy-sensitive design points are the preprocessing choices (neuron profiling ranges for KMNC/NBC/SNAC, VAE latent space for IDC, bisection-based boundary search for DBC, mutation operators for SLMS/MLMS) and the scoring formulas (mutant killing criterion, threshold on surprise, interaction coverage for IDC). Each of these admits plausible off-by-one, normalization, or coordinate-system discrepancies that would change reported scores while leaving the workflow intact. Because the framework deliberately abstracts metric internals behind a common interface, a consistent, working pipeline is not evidence of metric correctness. Without a numerical cross-check, the headline contribution is unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ADEPT, a Python framework that unifies several deep learning test adequacy metrics, including neuron-coverage-series metrics (NC, TKNC, KMNC, NBC, SNAC), surprise adequacy (LSA/DSA), input distribution coverage (IDC), deep boundary coverage (DBC), and source- and model-level mutation score (SLMS/MLMS), under a single command-line workflow. The framework provides a template-based metric interface, YAML configuration management, caching of intermediate artifacts, and structured JSON reporting. The paper describes the architecture in Section 2, usage in Section 3, and related work in Section 4, but it contains no experimental evaluation, no comparison with original implementations, and no test suite or example run with expected values.","tokens_in":7527,"tokens_out":2338,"duration_ms":22284,"significance":"If ADEPT's implementations faithfully reproduce the original metrics, the framework would be a valuable community asset: it directly addresses the fragmentation of DL adequacy tooling, and the caching and extensibility design are sensible engineering choices. The public Zenodo archive, demo video, and clearly described processing modules are concrete strengths. However, the central claim that ADEPT integrates representative adequacy techniques is currently supported only by architectural description, not by evidence that the metric scores are correct. For a tool paper whose contribution is faithful integration, this evidentiary gap is decisive in assessing significance.","major_comments":[{"comment":"The load-bearing claim that ADEPT faithfully reproduces NC-series, LSA/DSA, IDC, DBC, SLMS, and MLMS is unverified. Section 2.2 states that ADEPT 'carries out MLMS à la DeepMutation++ and SLMS à la DeepCrime,' and Section 2.4 defines mutation score as |killed mutants| / |all mutants|, but the manuscript provides no numerical comparison against the original DeepMutation++, DeepCrime, DeepGauge, or IDC implementations, no unit tests, no example run with expected scores, and no evaluation dataset. Because the framework abstracts metric internals behind a uniform interface, a consistent and working pipeline is not evidence of metric correctness. The authors should add a reproducibility study that runs ADEPT and the original tools on at least one common benchmark (e.g., MNIST or CIFAR-10) and reports score differences, or otherwise provide a test suite with hand-computed expected scores for each metric.","section":"§2.2, §2.4, §5"},{"comment":"Several accuracy-sensitive design choices are described only at the parameter-name level, leaving room for discrepancies that would change reported scores. For KMNC/NBC/SNAC, the neuron profiling range depends on training-data fractions and layer exclusion; for LSA, the variance threshold affects dimensionality reduction before KDE fitting; for IDC, coverage depends on the VAE latent space and interaction strength t; for DBC, bisection convergence and maximum points per class pair determine boundary coverage; and for SLMS/MLMS, the mutant killing criterion and mutation operator selection determine the mutation score. Table 1 lists these parameters but does not specify the default values or the exact formulas used. The paper should document these defaults and, more importantly, show that the chosen settings reproduce the original metrics' outputs.","section":"§2.2, Table 1"},{"comment":"The paper's practical motivation is that existing tools are difficult to install, configure, and run, and that ADEPT reduces this burden, but no evidence is provided for this usability claim. Section 1 cites practitioner interviews [18] and Section 4 argues that ADEPT provides 'consistent execution and reuse,' but there is no user study, no installation-time comparison, no command-run demonstration with timing or error rates, and no discussion of supported model types beyond 'Keras-based' models. At minimum, the authors should include a small case study showing the end-to-end workflow on a standard model and dataset, including successful execution of all supported metrics and the produced scores.","section":"§1, §4"}],"minor_comments":[{"comment":"The sentence 'ADEPT are publicly available on Zenodo' uses plural verb agreement; it should be 'ADEPT is publicly available on Zenodo.'","section":"§5"},{"comment":"Section 3 refers to 'the GitHub repository [7]' and the README, but reference [7] is the Zenodo record. The authors should clarify whether the source code is hosted on GitHub or only archived on Zenodo, and make the citation consistent.","section":"§3, §5, reference [7]"},{"comment":"In Table 1, the parameter name 'exclude_layer' appears without a space before the following text, and the table would be easier to read if parameter names were set in a monospaced font. This is a formatting issue only.","section":"Table 1"},{"comment":"The abstract and Section 2.1 mention 'source- and model-level mutation score' but the body sometimes uses 'source-/model-level mutation score' and 'SLMS and MLMS.' Using one consistent term throughout would improve readability.","section":"Abstract and §2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable tool paper for a companion venue, and the architecture is clearly presented. However, the central claim of faithful metric integration is currently unsupported by any experimental or testing evidence. This is fixable within the manuscript's scope: the authors can add a validation section comparing against original implementations, include a minimal test suite, and document default parameter values and formulas. If the authors decline to add such evidence, I would not consider the contribution substantiated. I also note that the paper has no explicit 'threats to validity' or 'limitations' section, which would be useful given the re-implementation risk."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as a tool paper, because that's what it is: ADEPT unifies eight or nine DL adequacy metrics into one workflow, with caching, YAML config, and a plug-in interface. That unification genuinely does not exist in the prior literature, and the motivation is real—anyone who has tried to run DeepGauge, then surprise adequacy, then DeepMutation++ knows the pain. The architecture is described clearly. The cache design, especially for mutant models and neuron profiles, is a thoughtful engineering choice that should save people real time.\n\nWhere the paper gets soft is exactly where the reader's condition lies: fidelity of the re-implementations. Section 2.2 says ADEPT carries out MLMS à la DeepMutation++ and SLMS à la DeepCrime, but there is no test suite, no comparison against original implementations, no dataset with expected scores, no example run showing a known score. The scoring formulas admit plausible off-by-one or normalization discrepancies, and the framework's common interface won't surface those. A consistent pipeline is not evidence that the numbers it reports match the original metrics. For a tool whose entire value proposition is reproducibility, that is a load-bearing gap, not a polish issue.\n\nThe good news is this gap is closable: run a few models and datasets, compare ADEPT's scores against the reference implementations (where they exist), provide a commit hash and a smoke test, and the central claim becomes checkable. I'd also like to see at least one end-to-end example with expected output, and a note on which metrics are re-implemented from scratch versus wrapping original code.\n\nThe citation pattern is fine. The Zenodo archive is the right move, and self-citation there isn't a problem given that it's the artifact itself. No circularity in the reasoning.\n\nWho is this for: people in the DL testing subfield who want to compare multiple adequacy metrics without spending weeks on setup. If the validation gets added, I'd bring it to reading group. As it stands, it deserves peer review, but with a demand for verification before acceptance.","headline":"ADEPT is a genuinely useful unifying framework, but its core claim of faithfully re-implementing existing adequacy metrics is unverified and must be checked before acceptance.","tokens_in":7980,"tokens_out":1871,"would_cite":false,"duration_ms":16402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ADEPT gives deep-learning test adequacy metrics a single plug-in workflow, so researchers and practitioners can run, compare, and extend metrics that currently live as fragmented prototypes.","keywords":["deep learning testing","test adequacy","neuron coverage","surprise adequacy","input distribution coverage","mutation testing","extensible framework","reproducibility"],"falsifier":"Take a model-dataset configuration from a paper that introduced one of these metrics, run ADEPT's implementation and the original implementation on the same inputs, and compare the numerical scores; any divergence in the reported adequacy shows the re-implementation is not faithful to the published metric.","tokens_in":7145,"feed_emoji":"🧪","tokens_out":7614,"duration_ms":64261,"temperature":0.7,"pith_summary":"The paper presents an engineering solution to a practical problem: deep-learning test adequacy metrics exist as scattered research prototypes with incompatible preprocessing, configuration, and execution requirements. ADEPT, a Python framework, integrates representative metrics—neuron coverage and its variants, surprise adequacy, input distribution coverage, decision-boundary coverage, and source- and model-level mutation score—under one consistent command-line workflow. It standardizes configuration through YAML files, caches costly intermediate artifacts such as activation traces and mutant models, and reports each score alongside execution metadata. If the framework works as described, users can reproduce, compare, and deploy adequacy metrics without spending days or weeks building or repairing the underlying tooling.","feed_headline":"ADEPT gives deep-learning test metrics a single workflow","feed_subtitle":"Neuron coverage, surprise adequacy, and mutation scores run through one interface instead of separate prototypes.","key_machinery":"The load-bearing object is the template-based metric interface, a plug-in abstraction that every ADEPT metric module implements. It separates three concerns: metric-specific processing modules that produce artifacts such as neuron profiles, activation traces, latent-space encodings, decision-boundary models, and mutant collections; a cache manager that stores artifacts keyed by model, dataset, metric, and creation time; and a scoring component that turns artifacts into a final score plus execution metadata. YAML configuration files set metric-specific parameters with framework defaults, which lets a user switch metrics through one command-line entry point without touching code.","core_discovery":"The paper's contribution is engineering: it claims that the fragmentation of deep-learning test adequacy tooling can be removed by a uniform plug-in architecture. ADEPT reduces seven metric families—neuron coverage and its variants, likelihood- and distance-based surprise adequacy, input distribution coverage, decision-boundary coverage, and source- and model-level mutation score—to one execution loop. The user supplies the test inputs, the target model, optional training data, and a YAML configuration; ADEPT selects the metric module, runs its required preprocessing, stores costly intermediate artifacts in a cache, computes the score, and writes a JSON report with the score, timing, and cache-usage metadata. The mutation-score branch reports the standard ratio of killed mutants to all mutants. The stated payoff is that researchers and practitioners can reproduce, compare, and deploy these metrics without re-implementing missing pieces or wrestling with incompatible research prototypes.","pith_inferences":["A natural next step is to check ADEPT's outputs against the original metric implementations on published configurations; the unified interface would then double as a correctness harness for metric definitions.","The caching design points toward metric-in-the-loop workflows, such as test-input prioritization and selection loops, in which repeated scoring must be cheap; the paper does not evaluate those workflows.","Because score outputs are structured and metadata-rich, ADEPT could also serve as a standard measurement backend for comparing datasets or model versions, a use case the paper leaves implicit."],"forward_implications":["Running the same model and test set under multiple adequacy metrics becomes a matter of changing the metric name and config, which makes cross-metric comparison routine.","Repeated evaluations become cheaper because intermediate artifacts such as activation traces and mutant models are generated once and reused across runs.","Adding a new adequacy metric requires implementing one module against the template interface rather than building a standalone toolchain.","Structured JSON reports with timing and cache metadata make individual runs inspectable and reproducible.","Default configurations allow a first run of each metric without reading implementation details."],"supporting_citations":[{"why":"Defines the neuron-boundary, multisection, and top-k coverage criteria that ADEPT's NC-series modules re-implement.","marker":"[12]"},{"why":"Defines surprise adequacy's activation-trace extraction and density-estimation procedure for LSA and DSA.","marker":"[8]"},{"why":"Defines the VAE-based input distribution coverage that ADEPT's IDC module implements.","marker":"[2]"},{"why":"Defines decision-boundary coverage that ADEPT's DBC module implements.","marker":"[11]"},{"why":"Supplies the model-level mutation workflow that ADEPT implements as MLMS.","marker":"[5]"},{"why":"Supplies the source-level mutation operator workflow that ADEPT implements as SLMS.","marker":"[6]"},{"why":"Introduces the neuron-activation coverage criterion that anchors the NC series.","marker":"[15]"},{"why":"Provides interview evidence that practitioners find existing metrics difficult to adopt, motivating ADEPT's unified design.","marker":"[18]"}],"fun_headline_variants":["ADEPT unifies deep-learning test adequacy metrics","One workflow for deep-learning test adequacy metrics","ADEPT: plug-in framework for deep-learning test metrics","Deep-learning test adequacy: ADEPT brings it together","ADEPT: a single interface for diverse test adequacy metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole framework rests on the assumption that ADEPT's re-implementations of the published metrics compute the same values the original metric papers would; if any implementation deviates, the unified workflow silently reports the wrong adequacy.","fun_headline_variants_meta":{"raw":{"variants":["ADEPT unifies deep-learning test adequacy metrics","One workflow for deep-learning test adequacy metrics","ADEPT: plug-in framework for deep-learning test metrics","Deep-learning test adequacy: ADEPT brings it together","ADEPT: a single interface for diverse test adequacy metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1376,"prompt_tokens":929,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":545,"tokens_out":447,"duration_ms":4181,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:13:33.862563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model-dataset configuration from a paper that introduced one of these metrics, run ADEPT's implementation and the original implementation on the same inputs, and compare the numerical scores; any divergence in the reported adequacy shows the re-implementation is not faithful to the published metric.","supporting_citations":[{"cited_title":"DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems","cited_arxiv_id":"1803.07519","evidence_quote":"Defines the neuron-boundary, multisection, and top-k coverage criteria that ADEPT's NC-series modules re-implement."},{"cited_title":"Guiding Deep Learning System Testing using Surprise Adequacy","cited_arxiv_id":"1808.08444","evidence_quote":"Defines surprise adequacy's activation-trace extraction and density-estimation procedure for LSA and DSA."},{"cited_title":"Dwyer, and Mary Lou Soffa","cited_arxiv_id":null,"evidence_quote":"Defines the VAE-based input distribution coverage that ADEPT's IDC module implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the model-level mutation workflow that ADEPT implements as MLMS."}],"review_version":1}