{"id":"b10d0a98-bde1-42ee-a8db-1f03cb54179e","arxiv_id":"2505.14967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ADCP extracts single-neuron-per-layer critical paths via genetic mutation and uses an SVDD voting ensemble to separate normal from adversarial, out-of-distribution, and noisy inputs.","lead":"Deep neural networks misjudge unusual inputs, and this paper proposes ADCP, a detector that selects one neuron per layer from a network and uses these critical paths as features to flag adversarial, out-of-distribution, and noisy images. The authors report high detection rates on MNIST, CIFAR-10 and SVHN, but the evaluation uses anomaly and test-set data during path and threshold selection, so the headline numbers should be read with caution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set leakage in path selection and threshold calibration makes the reported SOTA superiority unverified; a strict held-out evaluation is required before the generalization claim can be accepted.","rationale":"The method and code are coherent, and there is genuine cross-type evidence in Section 4.8. However, the central claim is an empirical outperformance claim, and the empirical protocol has a direct contamination path from the data used to optimize paths and thresholds to the data used to measure performance. This is more specific than a missing error bar: the optimization objective, path TPR on the mixed set, is measured on normal test images and on anomaly samples from the evaluated distributions, so the table entries are expected to be optimistic. The statement that during production runs only the training set is used to train SVDD does not fix this, because SVDD training is only one part; path selection and threshold calibration use test data. I therefore cannot accept the generalization and SOTA claims as currently evidenced, but the paper can be repaired with a properly split evaluation. The reader's conditional verdict is appropriate; my concern is a sharper version of the leakage issue rather than the transfer-across-anomaly-types assumption, hence partial agreement.","tokens_in":24377,"tokens_out":6504,"duration_ms":59316,"concrete_test":"Re-run the full ADCP pipeline with strict disjoint splits: use only the DNN training set to fit SVDD; select critical paths and calibrate class thresholds on a held-out validation set of normal images plus a held-out set of generated anomalies from the same attack/OOD procedures, using images not used in evaluation; then compute TPR@95%TNR and AUROC only on a final unseen test split. Repeat with 5 independent genetic searches and report mean and standard deviation. If ADCP no longer consistently outperforms MD, ELO, and ADCL, or its cross-type AUROC drops materially, the current claims are artifacts of test-set leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is the evaluation protocol, not the method idea. Section 3.1 says the mixed set used to score paths contains normal samples \"from a test set\", and the per-class threshold is set so that 95% of those normal test samples pass; Section 3.2 repeats this for \"normal samples from class i of test sets\". Thus the same normal images used for path selection and threshold calibration are reused in the reported TPR@95%TNR and AUROC numbers. The anomaly side is similarly leaked: the mixed set for each anomaly family is drawn from the same distribution later evaluated, e.g., F-MNIST for MNIST OOD, and FGSM or all-attacks for AD. This makes the central claim partly circular and makes the comparison against unsupervised baselines unfair, since ADCP consumes anomaly samples during path selection while most baselines do not. Section 4.8 provides some cross-type transfer evidence, but it still calibrates thresholds on test normal data and only reports CIFAR-10, so it does not remove the leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADCP, an anomaly detection method for DNN image classifiers that extracts \"critical detection paths\" (one neuron per layer) via a genetic mutation search. A candidate path is scored by the True Positive Rate (TPR) achieved when an SVDD model trained on class-specific training activations is applied to a mixed set of normal and anomaly samples, with per-class thresholds set so that 95% of normal samples pass. The selected paths are ensembled with random subspace sampling and a voting mechanism. The method is evaluated on MNIST, CIFAR-10, and SVHN for adversarial (AD), out-of-distribution (OOD), and noise (NS) inputs and compared with LID, Mahalanobis distance, MagNet, Feature Squeezing, NIC, ADCL, MemAE, Max-Softmax, ODIN, ELO, NPC, and EffPath. The authors claim that ADCP outperforms state-of-the-art methods and generalizes across AD, OOD, and NS inputs.","tokens_in":24595,"tokens_out":7128,"duration_ms":60447,"significance":"If the empirical claims survive a clean, held-out evaluation, this would be a useful contribution: the path representation is interpretable, the method does not modify the DNN, the code is publicly available, and the cross-type transfer experiment in Section 4.8 is a valuable check. However, as written, the evaluation protocol leaks both normal test data and the target anomaly types into path selection and threshold calibration, so the reported TPR/AUROC figures and the resulting superiority claims over unsupervised baselines are not yet substantiated. The paper's central claim is therefore plausible but unverified; the missing evidence is obtainable by re-running the evaluation with a stricter protocol.","major_comments":[{"comment":"The evaluation protocol leaks the normal test set into both path selection and threshold calibration. Section 3.1 states that the mixed set used to score candidate paths contains normal samples \"from a test set,\" and Section 3.2 states that each per-class threshold tau_i is computed to ensure that 95% of the normal samples from class i \"of test sets\" have scores above tau_i. The reported TPR at 95% TNR and AUROC are therefore computed against normal samples that were already used to select paths and set thresholds. This inflates the headline numbers and makes the comparison against unsupervised baselines unfair, because those baselines do not consume the test normal data during development. The authors should re-run the full evaluation with path selection and threshold calibration performed on a disjoint validation split of the in-distribution data, and report the metrics on a truly held-out test set.","section":"Sections 3.1 and 3.2"},{"comment":"The anomaly side is also leaked: the mixed set used for path selection contains the same anomaly types that are later evaluated. For OOD detection the manuscript says, for MNIST, F-MNIST is picked as the anomaly set, and for CIFAR-10/SVHN, TinyImageNet is picked; for AD detection the mixed set is \"a collection of adversarial samples of different attack methods,\" and the same attacks (FGSM, PGD, JSMA, CW, DeepFool) are then reported in Table 2. ADCP thus consumes target-anomaly samples during path selection, while most baselines do not. Section 4.8 provides cross-type transfer evidence, but it is limited to CIFAR-10 and still calibrates thresholds on test normal data, so it cannot rescue the general claim. I request a strict held-out evaluation in which paths are selected without access to the anomaly families used in the reported metrics, or at minimum a complete cross-type evaluation on all model/dataset combinations with thresholds set on validation normal data.","section":"Section 3.1"},{"comment":"All reported numbers are single-point estimates, but Algorithm 1 is stochastic: the initial path, the layer to mutate, and the neuron to mutate are all randomly selected. Tables 2-7 report one number per cell, and the text summarizes rankings such as \"the best performance 11 times out of 25\" without any measure of dispersion. Given the near-saturated AUROC values (e.g., 99.9% in many cells), repeated runs with mean and standard deviation, or confidence intervals, are needed to determine whether the advantages over ADCL and ELO are significant. This is especially important because several comparisons in Table 4 are within 0.1-0.3 percentage points.","section":"Section 4.3 and Algorithm 1"},{"comment":"The generalization test is incomplete as reported. The section only uses CIFAR-10 models and reports AUROC after ensembling 21 paths; it does not report TPR at 95% TNR or results for MNIST and SVHN. Moreover, even the OOD-to-AD and AD-to-OOD comparisons reuse test normal data for threshold calibration, so they do not address the leakage identified above. A complete cross-type table, including F-MNIST and Omniglot for MNIST and the SVHN OOD sets, with held-out thresholds, is required before the conclusion that \"paths selected by different anomaly samples do not have a significant impact on the detection performance\" can be accepted.","section":"Section 4.8"}],"minor_comments":[{"comment":"The metric is mislabeled: \"True Negative Rate at 95% of True Positive Rate\" should be \"True Positive Rate at 95% True Negative Rate,\" since the thresholds are set so that 95% of normal samples pass.","section":"Section 4.1"},{"comment":"The phrase \"True Positive Positive Rate\" should be simply \"True Positive Rate.\"","section":"Section 3.1"},{"comment":"The word \"Fasle\" in the formal discriminator should be \"False.\"","section":"Equation (6)"},{"comment":"The sentence \"Jiang et al. [52] used a similar method\" appears to cite the wrong work; reference [52] is Wang et al., and no separate Jiang et al. reference is listed.","section":"Section 5.2"},{"comment":"In the ResNet/SVHN block, the JSMA row contains \"79.11//97.72\" with an extra slash; this appears to be a typographical error.","section":"Table 5"},{"comment":"In the LeNet/MNIST row, the reported Total 5514.33 does not equal the sum of the two preceding columns (5740.32 + 44.01 = 5784.33).","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The main risk is evaluation leakage; the strengths of the paper (public code, a sparse path representation, and a cross-type transfer experiment) do not outweigh the fact that the headline comparisons are unverified under a clean protocol. If the authors re-run the evaluation with disjoint validation/test normal splits and held-out anomaly families, the paper could become acceptable, but I would not recommend acceptance in the current form. The manuscript is formatted for ACM TOSEM; if this is the intended venue, the authors should also sharpen the connection to software-engineering path-based testing, since the current framing is mostly an anomaly-detection benchmark study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine new combination—one-neuron-per-layer “critical detection paths” found by genetic mutation, scored by TPR on a mixed set, and ensembled via random subspace SVDD and voting—and applying path features to OOD and noise detection is new. The writing is clear, the code is public, and Section 4.8 at least attempts a cross-anomaly transfer check. If the empirical claims held, this would be a useful, cheap, model-agnostic detector with a path-level explanation.\n\nThe problem is the protocol. Section 3.1 says the mixed set used to select paths contains normal samples “from a test set,” and thresholds are set so 95% of normal samples from class i of test sets pass. Those same test images then appear in the reported TPR@95%TNR and AUROC numbers. The anomaly side is leaked too: paths are selected on F-MNIST before F-MNIST OOD detection, and on FGSM/all-attacks before evaluating those same attacks. So ADCP gets access to anomaly samples during path selection while most baselines do not. That makes the SOTA comparison unfair and the “generalizes to a broad range” conclusion partly circular. Section 4.8 shows the selected path transfers reasonably across anomaly families, which is real evidence in the right direction, but it still calibrates thresholds on test normals and only reports CIFAR-10. No error bars or repeated runs are reported anywhere, and the genetic search is explicitly random.\n\nI don’t think the core idea is broken. But the load-bearing claims—outperforming SOTA and generalizing across AD/OOD/NS—need to be re-established with a proper train/validation/test split: select paths and thresholds on validation, evaluate once on a held-out test, include outlier-exposure baselines, and report means over repeated runs. That is a major revision, not a marginal one. I would send it to peer review because there is real novelty and the community would benefit from the method and from the protocol fix, but I would not accept it without the clean evaluation.","headline":"A genuinely new path-based anomaly detection idea, but the evaluation protocol leaks anomaly information into path selection and threshold calibration, so the headline results are unverified until a clean held-out evaluation is run.","tokens_in":25155,"tokens_out":2264,"would_cite":false,"duration_ms":19017,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep network's anomaly behaviour can be compressed into critical detection paths—one neuron per layer, evolved by mutation—that separate normal images from adversarial, out-of-distribution, and noise inputs across several standard…","keywords":["anomaly detection","deep neural networks","critical detection paths","genetic mutation","support vector domain description","out-of-distribution detection","adversarial example detection","ensemble voting"],"falsifier":"Train ADCP on adversarial samples from FGSM only, then test on adversarial samples from a held-out attack family not used in path selection (e.g. AutoAttack or an adaptive attack); if the TPR at 95% TNR on that held-out family is close to the normal-input false positive rate rather than near the reported 90%+ levels, the claimed generalisation across anomaly types fails.","tokens_in":24172,"feed_emoji":"🧬","tokens_out":4427,"duration_ms":36134,"temperature":0.7,"pith_summary":"The paper contends that the internal flow of a deep network can be compressed into a handful of critical detection paths, each a single neuron per layer from input to output, and that these paths carry enough information to separate normal inputs from adversarial, out-of-distribution, and noise inputs. It proposes ADCP, which evolves such paths by repeatedly mutating one neuron and keeping only mutations that raise the true-positive rate of a one-class SVDD trained on that path's activation values. Multiple paths are then ensembled with random subspace sampling and voting. If the claim holds, anomaly detection no longer needs to modify the network or rely on a fixed layer, and one path set can serve several anomaly families at once. The experiments across MNIST, CIFAR-10, and SVHN report that ADCP matches or outperforms ten existing detectors on all three anomaly types.","feed_headline":"One neuron per layer is enough to flag anomalous inputs","feed_subtitle":"ADCP evolves critical paths by mutation, then votes across 21 paths to beat ten detectors on adversarial, OOD, and noise data.","key_machinery":"The critical detection path is a sequence $\\langle n^{v_0}_0, n^{v_1}_1, \\dots, n^{v_{N-1}}_{N-1}\\rangle$ that selects one neuron per layer, instantiated by a test input as the vector of that neuron's activation values (average-pooled channel values for convolutional layers). The selection loop is a greedy genetic mutation: start from a random path, randomly mutate one neuron, and keep the mutation only if the True Positive Rate of a Support Vector Domain Description trained on that path's features over the mixed normal/anomaly set increases. The method then ensembles $m$ such paths per class (typically $m=21$) using random subspace sampling and a voting rule over min-max normalised SVDD scores, with class-wise thresholds $\\tau_i$ fixed at the 95% normal-recall level. This machinery converts the DNN into a small set of measurable neuron chains that serve as anomaly features.","core_discovery":"ADCP establishes that a class-wise critical detection path, a sequence containing exactly one neuron per layer, chosen by genetic mutation to maximise TPR on a mixed normal/anomaly set, yields activation-vector features on which a Support Vector Domain Description model separates in-distribution from anomalous inputs. Because different paths capture different features, the method takes 21 such paths per class, trains one SVDD per path, normalises scores, and combines them by voting, with a per-class threshold set so that 95% of normal samples pass. The paper reports AUROC above 0.97 on most AD, OOD, and NS tasks across LeNet, VGG16, and ResNet models, and shows that paths selected on one anomaly type (e.g. OOD) transfer to other types (AD, NS) with only a small drop, which is the basis for the claim that the detector generalises.","pith_inferences":["The transfer experiments in Section 4.8 suggest paths are not tightly coupled to the anomaly family used to select them, but the main evaluation still selects paths on the same family tested, so a fully held-out evaluation would clarify whether ADCP is unsupervised or a per-family supervised detector.","The same path-extraction idea could apply to non-image modalities such as speech, text, or network traffic, since the path definition only needs neuron activations and layer connectivity, though the paper tests only image classifiers.","Because path selection is driven by TPR on a mixed set, the selected paths may be biased toward the strongest available anomaly family; combining paths selected under several anomaly families might yield a more robust ensemble.","One could test path reuse across architectures: paths extracted from a VGG16 on CIFAR-10 may or may not transfer to a differently initialised or deeper model, which would tell whether the paths capture dataset structure or model-specific artefacts."],"forward_implications":["If the central claim holds, anomaly detection for image classifiers can be done without retraining or modifying the target DNN, using only the path features and per-path SVDD models.","A single set of critical detection paths would suffice for adversarial, out-of-distribution, and noise inputs, so practitioners would not need a separate detector per anomaly family.","Because each path contains one neuron per layer, the detector's decision can be traced to specific neurons, giving an interpretable reason why an input was flagged.","The ensemble of multiple paths compensates for the fact that individual paths catch different subsets of anomalies, with detection accuracy rising steeply when going from 1 to 3 paths and stabilising by 21 paths.","Path-based detection is cheaper at test time than profiling all training inputs as prior path methods do, at the cost of the one-time mutation search."],"supporting_citations":[{"why":"Support Vector Domain Description, the one-class classifier that scores each critical path's activation features.","marker":"[47]"},{"why":"Source of the genetic evolution idea that drives the mutation-based path search.","marker":"[7]"},{"why":"Source of the mutation testing idea applied to mutate one neuron per iteration.","marker":"[27]"},{"why":"Random subspace method used to construct the ensemble of critical detection paths.","marker":"[16]"},{"why":"Mahalanobis distance baseline that ADCP must beat for AD and OOD detection.","marker":"[21]"},{"why":"Local Intrinsic Dimensionality baseline for adversarial detection, compared against FGSM-trained ADCP.","marker":"[29]"},{"why":"Max-Softmax baseline for OOD/NS detection.","marker":"[12]"},{"why":"Effective Path approach whose paths are compared against ADCP's critical paths.","marker":"[37]"},{"why":"Neuron Path Coverage approach whose decision-logic paths are compared in RQ3.","marker":"[54]"},{"why":"Source of fooling images, one of the noise anomaly types ADCP detects.","marker":"[33]"}],"fun_headline_variants":["Critical paths from genetic evolution beat ten detectors on anomalies","One neuron per layer: a tiny path that spots anomalies","Genetic mutation grows critical paths that catch outliers","Class-wise critical paths: a new way to detect anomalous inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The critical paths selected by maximising true-positive rate on a mixed set that already contains the anomaly type being tested must keep separating normal from anomalous inputs when the anomaly type changes, without retraining.","fun_headline_variants_meta":{"raw":{"variants":["Critical paths from genetic evolution beat ten detectors on anomalies","One neuron per layer: a tiny path that spots anomalies","Genetic mutation grows critical paths that catch outliers","Class-wise critical paths: a new way to detect anomalous inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1396,"prompt_tokens":926,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":406}},"tokens_in":542,"tokens_out":470,"duration_ms":5337,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:26:42.040947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ADCP on adversarial samples from FGSM only, then test on adversarial samples from a held-out attack family not used in path selection (e.g. AutoAttack or an adaptive attack); if the TPR at 95% TNR on that held-out family is close to the normal-input false positive rate rather than near the reported 90%+ levels, the claimed generalisation across anomaly types fails.","supporting_citations":[{"cited_title":"Nguyen, J","cited_arxiv_id":null,"evidence_quote":"Source of fooling images, one of the noise anomaly types ADCP detects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Support Vector Domain Description, the one-class classifier that scores each critical path's activation features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the genetic evolution idea that drives the mutation-based path search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the mutation testing idea applied to mutate one neuron per iteration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Random subspace method used to construct the ensemble of critical detection paths."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mahalanobis distance baseline that ADCP must beat for AD and OOD detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Local Intrinsic Dimensionality baseline for adversarial detection, compared against FGSM-trained ADCP."},{"cited_title":"Hendrycks and K","cited_arxiv_id":null,"evidence_quote":"Max-Softmax baseline for OOD/NS detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Effective Path approach whose paths are compared against ADCP's critical paths."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Neuron Path Coverage approach whose decision-logic paths are compared in RQ3."}],"review_version":1}