{"id":"a1315b17-045e-43b7-8bcb-44ba6acf72aa","arxiv_id":"2505.08814","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Across ten LeNet, VGG, and ResNet models, neuron coverage, k-multisection coverage, boundary coverage, Top-k coverage, and MC/DC show depth- and configuration-dependent trends that are often inconsistent between architectures.","lead":"This paper measures how six neural-network coverage metrics change with model depth, architecture, and dataset size across LeNet, VGG, and ResNet models. It maps regularities that could help DNN testing, but the empirical claims lack error bars and repeated runs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline answer to RQ1 contradicts the paper's own LeNet data: in Table 2, NC increases with depth for LeNet at every threshold, so 'coverage decreases with depth' cannot be a general law.","rationale":"The reader's verdict is REJECT with moderate confidence, and the reader's rationale mentions that the headline answer to RQ1 contradicts the LeNet data. My stress-test pass confirms this is the most load-bearing weakness: the central claim is directly falsified by the paper's own Table 2, so the conclusion cannot be accepted as a general empirical law. The reader's formal 'weakest assumption' focused on single unseeded checkpoints, which is a valid external-validity concern, but the deeper problem is internal inconsistency: the summary answer is contradicted by the very table it summarizes. Because the contradiction involves the aggregate reported values, it would remain even if all experiments were repeated with many seeds. The recommendation is therefore unchanged: reject or require substantial revision. The concrete test is a simple re-derivation from the existing table, which settles the concern without additional computation.","tokens_in":20191,"tokens_out":2865,"duration_ms":25594,"concrete_test":"Recompute the NC columns of Table 2 and evaluate monotonicity with depth for each architecture and threshold. Specifically, count pairwise depth comparisons where the deeper model has lower NC; then check the LeNet rows at tau=0.3, 0.45, 0.6, and 0.75, where all six pairwise comparisons favor higher coverage with greater depth. If the pairwise counts do not support a consistent inverse relationship across all architectures and thresholds, the Answer to RQ1 must be revised to name the architecture/threshold regime (e.g., VGG and ResNet with tau>0.6) or the general claim is unsupported. This test uses the paper's own reported numbers and does not require new training runs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Answer to RQ1 in Section 3.2: 'As the model depth increases, the neuron coverage decreases.' This claim fails against the paper's own reported numbers. In Table 2, the NC values for LeNet at depth 5, 6, and 7 strictly increase with depth at every threshold: at tau=0.3 they are 0.5476, 0.7391, 0.8256; at tau=0.45 they are 0.3571, 0.6594, 0.7713; at tau=0.6 they are 0.2619, 0.6014, 0.7054; and at tau=0.75 they are 0.2381, 0.5942, 0.6705. The paper's own prose in Section 3.2 states that 'as depth increases, LeNet models with greater depth will have higher neuron coverage,' directly contradicting the summary answer. The VGG rows also do not show a clean monotone decrease: at tau=0.3 the coverage goes 0.8758 (11 layers), 0.8728 (13), 0.885 (16), 0.8833 (19), which is non-monotonic. Thus the RQ1 answer is either overgeneralized or requires an explicit scope such as 'for VGG and ResNet at thresholds above 0.6.' Because this contradiction is internal to the reported aggregate data, it is independent of the additional single-checkpoint/no-seed problem; no amount of repeated training runs will reconcile the LeNet rows with the claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of six coverage metrics (NC, KMNC, NBC, SNAC, Top-k NC, and MC/DC) across LeNet, VGG, and ResNet models of varying depths, using MNIST for LeNet and CIFAR-10 for VGG and ResNet. It claims to identify regular relationships between model depth, configuration parameters (activation threshold, partition count k, boundary distance, Top-k value, dataset size), and coverage, and it answers three research questions about these relationships. The central answer to RQ1 is that neuron coverage decreases as model depth increases, with additional statements about depth-dependent decline rates. The paper closes with three proposed future directions for DNN security testing.","tokens_in":20486,"tokens_out":4575,"duration_ms":37633,"significance":"The question of how architecture depth and configuration parameters affect coverage metrics is relevant to DNN testing, and the paper's tables contain direct measurements that could in principle be reused by other researchers. However, the significance of the reported findings is undercut by two problems: the headline depth rule is contradicted by the paper's own Table 2 for LeNet, and the experiments rely on a single trained checkpoint per configuration with no training details, no accuracies, and no repeated runs. As submitted, the empirical claims are not established, and the paper includes no code or data artifacts to support independent verification.","major_comments":[{"comment":"The claim that \"as the model depth increases, the neuron coverage decreases\" is directly contradicted by Table 2 for LeNet. At every reported threshold the NC values strictly increase with depth: at tau=0.3 they are 0.5476 (5 layers), 0.7391 (6 layers), and 0.8256 (7 layers), and the same increasing ordering holds at tau=0.45, 0.6, 0.75, and 0.9. The text in Section 3.2 even states that \"LeNet models with greater depth will have higher neuron coverage.\" The RQ1 answer therefore needs an explicit scope, such as \"for VGG at thresholds above 0.6,\" or a reconciliation with the LeNet rows; as stated, the central conclusion is false on the paper's own data.","section":"Section 3.2, Answer to RQ1"},{"comment":"The experiments are not reproducible from the manuscript: for each of the 10 models, a single trained checkpoint is used, and no training hyperparameters, optimizer, learning rate, epochs, data augmentation, or final test accuracy are reported. Tables 2-5 list one value per condition with no repeated seeds, error bars, or significance tests. Since coverage values can vary with initialization and training protocol, the depth trends derived from these tables could be artifacts of individual checkpoints; the paper should report accuracies and at least several independent training runs per configuration.","section":"Section 3.1, Experimental Setup"},{"comment":"The comparison across architectures confounds model depth with dataset and architecture family: LeNet is evaluated on MNIST, while VGG and ResNet are evaluated on CIFAR-10, and the three families differ in layer types and connectivity. Consequently, statements such as \"compared with the VGG model, the neuron coverage of ResNet decreases more obviously\" do not isolate the effect of depth; depth, dataset, and architecture are varied simultaneously, so the reported cross-architecture depth rules are not identifiable.","section":"Section 3.2, Table 2"},{"comment":"The claim that \"when the model is LeNet, the coverage of k-multisection neurons also decreases with the increase of the model depth\" is not supported by Table 2. For k=10, KMNC goes 0.95 (5 layers), 0.8529 (6 layers), then 0.907 (7 layers); at k=1000 it goes 0.6023, 0.5433, and 0.591, and at k=5000 it goes 0.4464, 0.3493, and 0.3559. The non-monotonic pattern at several k values means the stated depth rule needs qualification or a statistical test rather than a blanket statement.","section":"Section 3.3, KMNC"},{"comment":"The claim that \"all the coverage rates increase with the increase of the dataset size\" overstates Table 5. For LeNet with 7 layers, MC/DC(VS) decreases from 0.1572 at 3200 samples to 0.1531 at 6400 samples, and MC/DC(VV) is essentially flat at 1.0 for the 5-layer model across all dataset sizes. The monotonicity claim should be restricted to the cases the table actually supports, or supported with a statistical test.","section":"Section 3.7, Answer to RQ3"}],"minor_comments":[{"comment":"The text says \"Based on the data in Table 5,\" but the presented Top-k coverage values appear in Table 4, while Table 5 contains MC/DC; the cross-reference should be corrected.","section":"Section 3.6"},{"comment":"Several formulas in Section 2 are garbled by encoding issues (for example, the NC expression contains misplaced braces and an extra character), so the definitions are hard to parse; they should be typeset cleanly.","section":"Section 2"},{"comment":"Many references are unrelated to the paper's topic (for example, [4], [6], [19], [40], [54], and several others concern smart contracts, NFTs, and blockchain), which suggests the bibliography was not curated for this submission; these should be removed or replaced with relevant DNN testing literature.","section":"References"},{"comment":"Figures 5, 6, 8, and 16 are referenced in Sections 3.2-3.6 but do not appear in the provided manuscript text, which currently contains only Figures 1-4; the figure numbering and placement must be fixed.","section":"Figures"},{"comment":"The abstract and the contributions list four coverage metrics, while the paper actually studies six (adding SNAC and Top-k NC); the counts should be made consistent.","section":"Abstract and Contributions"},{"comment":"There are numerous typos and malformed glyph sequences, including \"MD/DC\" in Section 1, \"neurons boundary scope of coverage concern\" in Section 3.4, and the \"/uni000000...\" artifact strings in the captions of Figures 2-4; a careful editing pass is needed.","section":"Throughout"},{"comment":"The paper does not state whether code, trained models, or raw coverage measurements will be released; for an empirical study of this kind, a data and code availability statement is essential.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The manuscript has an unusual citation pattern: a substantial fraction of the references concern blockchain and smart-contract topics that have no connection to DNN coverage testing, which suggests the bibliography was not tailored to this paper. More importantly, the main empirical conclusion is internally contradicted by the paper's own results, and the lack of training details, accuracies, seeds, and repeated runs leaves no way to assess the reliability of any of the reported trends. I do not see a path to acceptance without a substantially new empirical study, so I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2505.08814. The paper measures six coverage metrics across ten LeNet/VGG/ResNet configurations and reports regularities linking depth and configuration parameters to coverage. That is a reasonable thing to do, and the data tables are new as far as I know. The MC/DC dataset-size experiment is a nice addition. But the central claim falls apart on inspection. The Answer to RQ1 says 'as the model depth increases, neuron coverage decreases,' yet Table 2 shows LeNet coverage increasing with depth at every threshold (e.g., NC at tau=0.3 goes 0.5476, 0.7391, 0.8256 for 5, 6, 7 layers). Section 3.2 even states the opposite for LeNet. The VGG rows are non-monotone too. So the summary is either overgeneralized or wrong as stated; at best it applies to VGG/ResNet at thresholds above 0.6, but the paper doesn't scope it that way.\n\nOther soft spots: one trained checkpoint per configuration, no seeds, no accuracy or training hyperparameters, no error bars or significance tests. Without repeated runs you can't tell whether the depth rules are robust or artifacts of a single initialization. LeNet on MNIST is compared against VGG/ResNet on CIFAR10, confounding depth with dataset and architecture. No code or data released.\n\nI want to give credit where it's due: the sweep itself is useful reconnaissance, and the qualitative observations about boundary-based metrics and MC/DC dataset size could be starting points. But the internal contradiction isn't a minor blemish; it's the headline result. No amount of noise analysis fixes a claim that its own table refutes.\n\nWho is this for? Someone in DNN testing who wants a quick map of coverage values across architectures might browse the tables, but I wouldn't rely on the stated regularities. I wouldn't cite it in its current form. If the authors redo the experiments with multiple seeds, report accuracies, correct the RQ1 summary to match the data, and release artifacts, it could become a solid empirical note. As it stands, I'd desk reject and invite a thorough rewrite rather than send it to referees.","headline":"The empirical sweep is useful, but the paper's headline claim that coverage decreases with depth is contradicted by its own LeNet table.","tokens_in":21041,"tokens_out":2484,"would_cite":false,"duration_ms":20949,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the six standard neuron-coverage metrics respond to model depth and coverage-configuration parameters in regular, describable ways, so a network's coverage values could be anticipated from its architecture and test…","keywords":["neuron coverage","DNN security testing","coverage metrics","model depth","MC/DC","KMNC","empirical study","image classifiers"],"falsifier":"Train the same ten architectures several times with different random seeds but identical data and training settings, then compute NC at thresholds from 0.3 to 0.9 for every checkpoint; if the depth ordering of coverage values changes across seeds, for instance VGG13 sometimes covering more than VGG11 at threshold 0.75, the claimed depth rules are not stable.","tokens_in":19939,"feed_emoji":"🧠","tokens_out":8333,"duration_ms":69591,"temperature":0.7,"pith_summary":"Deep neural networks are tested by feeding them inputs and measuring how many neurons those inputs activate, but no established body of evidence says how those coverage numbers change as a network gets deeper or as test parameters change. This paper tries to fill that gap. It trains ten image classifiers, spanning LeNet, VGG, and ResNet variants from five to 54 layers, and measures six coverage metrics under varying thresholds, segmentation counts, boundary distances, and dataset sizes. The central claim is that coverage values follow regular patterns as model depth and configuration parameters move, with the headline rule being \"as the model depth increases, the neuron coverage decreases.\" If the rules hold, security testers could predict coverage behavior from architecture choices instead of discovering it by running each model.","feed_headline":"Coverage metrics track model depth in regular patterns","feed_subtitle":"Ten models, five to 54 layers deep, link coverage values to architecture and parameter settings.","key_machinery":"The machinery is a controlled comparison of six coverage metrics across three architecture families. The metrics are the measuring instruments: neuron coverage (NC) counts neurons whose activation exceeds a threshold; k-multisection neuron coverage (KMNC) divides each neuron's activation range into k bins and counts occupied bins; neuron boundary coverage (NBC) and strong neuron activation coverage (SNAC) count activations outside the normal range, with SNAC restricted to the upper side; top-k neuron coverage (TopkNC) checks whether the most active neurons in each layer fall in the top k; and MC/DC treats a neuron as a decision and the previous layer's neurons as conditions. The comparison holds the dataset fixed within each architecture family, MNIST for LeNet and CIFAR10 for VGG and ResNet, and varies only depth, the metric's threshold or segmentation parameter, and, for MC/DC, the size of the input test set. The observed trends across these conditions are the argument's output.","core_discovery":"On the authors' own terms, the discovery is an empirical regularity: the amount of activation space a test suite covers is not arbitrary but tracks model depth and metric configuration. In their data, neuron coverage (the fraction of neurons whose activation exceeds a threshold) falls as networks get deeper for the VGG and ResNet families at higher thresholds, while the small LeNet family instead shows coverage rising with depth; across all models, neuron coverage declines as the threshold rises and approaches a stable floor. K-multisection neuron coverage falls as the segmentation count k grows, with the drop steepest between k=10 and k=5000. Boundary coverage responds to the boundary-distance parameter in two regimes, depth-sensitive for negative distances and nearly flat for positive distances. Top-k neuron coverage is weakly tied to depth in VGG, negatively tied in ResNet, and in LeNet the coverage gap between successive depths shrinks as depth grows. Traditional MC/DC coverage grows with dataset size, with the value-value variant already saturated, and both MC/DC(SV) and MC/DC(VV) fall with model depth. The paper takes these patterns as evidence that depth and configuration information can be used to anticipate coverage behavior in DNN security testing.","pith_inferences":["If these depth rules generalize across random seeds, coverage could be modeled as a function of depth and threshold, letting researchers estimate coverage for architectures they have not yet trained.","The LeNet-on-MNIST and VGG/ResNet-on-CIFAR10 split leaves open a cross-dataset test: varying depth within a single dataset and architecture family would isolate depth from data difficulty.","A direct implication the paper does not draw is that if coverage systematically falls with depth, deeper models may appear less tested under fixed test suites, which could change how coverage is used as a quality gate in adversarial-robustness work.","The monotone patterns suggest coverage could serve as an architectural prior for test-input generation, for instance selecting per-layer thresholds to equalize coverage across depths."],"forward_implications":["If the depth rules hold, coverage values for a given architecture are predictable from the layer count and the chosen threshold, so a tester can pick settings that expose under-covered regions before spending compute on inference.","The finding that MC/DC coverage rises with dataset size implies that small test suites systematically underreport structural coverage, which would bias comparisons between models tested on different data volumes.","The two-regime boundary behavior, depth-sensitive for negative boundary distance and nearly flat for positive distance, gives testers a parameter for targeting boundary neurons in either shallow or deep models.","Because KMNC's decline steepens between k=10 and k=5000, the segmentation count acts as a sensitivity knob that should be fixed when comparing models or reported alongside any coverage number.","The difference between sequential VGG and residual ResNet in top-k coverage suggests architecture family must be separated from depth in any coverage comparison."],"supporting_citations":[{"why":"Defines the LeNet architecture used as the small sequential family in the depth comparisons.","marker":"[23]"},{"why":"Defines the VGG architecture used as the sequential large-scale family.","marker":"[49]"},{"why":"Defines the ResNet residual architecture used as the non-sequential family.","marker":"[18]"},{"why":"Supplies the neuron coverage (NC) definition that the depth experiments measure.","marker":"[45]"},{"why":"Defines KMNC, NBC, SNAC, and Top-k neuron coverage, the metrics whose depth and parameter trends are analyzed.","marker":"[35]"},{"why":"Introduces the MC/DC coverage criterion for deep networks that Section 3.7 tests.","marker":"[51]"},{"why":"Provides the MC/DC configuration parameters used in the dataset-size experiments.","marker":"[64]"},{"why":"Motivates depth as a relevant variable by showing how network representations vary with depth.","marker":"[42]"},{"why":"Connects coverage criteria to model quality, the concern that makes these depth rules relevant to security testing.","marker":"[65]"}],"fun_headline_variants":["Coverage metrics show depth-dependent trends","Model depth drives coverage metric patterns","Depth alters neuron coverage in DNNs","DNN coverage varies with layer count","Coverage trends follow network depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments assume that one trained checkpoint per architecture and depth represents that configuration, so the depth trends are properties of depth rather than artifacts of a particular random initialization or training run.","fun_headline_variants_meta":{"raw":{"variants":["Coverage metrics show depth-dependent trends","Model depth drives coverage metric patterns","Depth alters neuron coverage in DNNs","DNN coverage varies with layer count","Coverage trends follow network depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1661,"prompt_tokens":1008,"completion_tokens":653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":624,"tokens_out":653,"duration_ms":6543,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:18:54.137133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same ten architectures several times with different random seeds but identical data and training settings, then compute NC at thresholds from 0.3 to 0.9 for every checkpoint; if the depth ordering of coverage values changes across seeds, for instance VGG13 sometimes covering more than VGG11 at threshold 0.75, the claimed depth rules are not stable.","supporting_citations":[{"cited_title":"Gradient-based learning applied to document recognition 1998.Proc","cited_arxiv_id":null,"evidence_quote":"Defines the LeNet architecture used as the small sequential family in the depth comparisons."},{"cited_title":"Deep residual learning for image recognition 2016","cited_arxiv_id":null,"evidence_quote":"Defines the ResNet residual architecture used as the non-sequential family."},{"cited_title":"Deepxplore: Automated whitebox testing of deep learning systems 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the neuron coverage (NC) definition that the depth experiments measure."},{"cited_title":"Deepgauge: Multi-granularity testing criteria for deep learning systems 2018","cited_arxiv_id":null,"evidence_quote":"Defines KMNC, NBC, SNAC, and Top-k neuron coverage, the metrics whose depth and parameter trends are analyzed."},{"cited_title":"Deep learning code fragments for code clone detection 2016","cited_arxiv_id":null,"evidence_quote":"Provides the MC/DC configuration parameters used in the dataset-size experiments."},{"cited_title":"Correlations between deep neural network model coverage criteria and model quality 2020","cited_arxiv_id":null,"evidence_quote":"Connects coverage criteria to model quality, the concern that makes these depth rules relevant to security testing."}],"review_version":1}