Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Ranking unlabeled test inputs by how much their predictions change across late-training checkpoints finds DNN misclassifications earlier than single-checkpoint confidence scores, the paper argues.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:55 UTC pith:S754OSRW

load-bearing objection NCIP is a solid, well-validated method for label-free DNN test prioritization, but its distinctive NC-guided checkpoint selection is never compared with random or uniform selection, so the causal story is under-evidenced. the 3 major comments →

arxiv 2607.20046 v1 pith:S754OSRW submitted 2026-07-22 cs.LG cs.AIcs.SE

Test Case Prioritization for DNNs via Neural Collapse Instability

Chunyu Liu , Mingyuan Li , Yang Li , Wenmin Li , Fei Gao , Tengfei Tu , Su-Juan Qin This is my paper
classification cs.LG cs.AIcs.SE
keywords test case prioritizationdeep neural network testingneural collapsecross-checkpoint prediction variabilitytotal variation distanceprediction marginadversarial robustnesslabel-free prioritization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes NCIP, a label-free way to order test inputs for a trained classifier so that misclassified examples surface early under a limited testing budget. Instead of trusting the model's confidence at one final checkpoint, NCIP measures, for each unlabeled input, how much its predicted probability distribution wobbles across a small set of late-training checkpoints, and adds the final model's margin to flag boundary-adjacent cases. The paper claims this cross-checkpoint variability is a better error signal than confidence because correctly classified inputs stay stable while failure-prone inputs flip under checkpoint-induced decision boundary shifts. On seven datasets and multiple architectures it reports 1.5%–16.6% gains in RAUC-ALL and 4.9%–20.6% gains in RAUC-500 over baselines, with the best average ranking across all dataset-model pairs. A sympathetic reader would care because the method needs no labels and no retraining, only checkpoints that training pipelines already save.

Core claim

The central claim is that, in the terminal training phase where neural collapse makes classifier geometry highly structured, the variability of a test input's predicted distribution across carefully selected late checkpoints is a more reliable label-free indicator of misclassification than any single-checkpoint confidence score. NCIP formalizes this signal as the total variation distance between each checkpoint's softmax output and the final checkpoint's output, averaged over an NC-guided subset of checkpoints, and combines it with the final model's prediction margin. The paper further claims that choosing the checkpoints by an equiangularity score—the standard deviation of pairwise cosine s

What carries the argument

The carrying object is the cross-checkpoint Total Variation Distance (TVD), defined for a test input x as the average, over the selected checkpoint set S, of half the L1 distance between each checkpoint's softmax distribution and the final checkpoint's distribution. A second mechanism is the checkpoint-selection score sigma_cos (the standard deviation of pairwise cosine similarities among class weight vectors), which Algorithm 1 uses to pick K=30 diverse, NC-consistent checkpoints from the last 90% of the training trajectory. These two feed the final score NCIP(x) = zscore(TVD(x)) + zscore(1 - Margin(x)), where Margin is the gap between the largest and second-largest final probabilities. Tog

Load-bearing premise

The load-bearing premise is that the standard deviation of pairwise cosine similarities among classifier weight vectors identifies checkpoints whose prediction variability marks genuine errors; if those checkpoints are no better than randomly chosen late checkpoints, the neural-collapse selection step adds nothing and the method reduces to variability plus margin.

What would settle it

Run NCIP against the same TVD-plus-margin score computed over K checkpoints chosen uniformly at random (or over all available checkpoints) on the same datasets and models; if the randomized or all-checkpoint variant matches NCIP's RAUC-ALL and RAUC-500 within noise, then the sigma_cos checkpoint-selection step is not load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, test prioritization for deployed classifiers can be done with no labels and no retraining: the ranking is computed from stored checkpoints and the unlabeled test set alone.
  • It implies that saved training trajectories are reusable assets for validation: pipelines that retain checkpoints get early fault discovery almost for free during regression testing.
  • The ablation results imply the prediction-variability term alone is a competitive ranking signal, so methods that already collect checkpoints can adopt TVD without the neural-collapse selection step and still gain accuracy in early fault discovery.
  • The margin combination increases fault diversity at small budgets, which the paper ties to detecting a wider variety of error types rather than concentrating on one confusion pattern.
  • Performance under model calibration suggests the gain is not just confidence rescaling but comes from the instability signal across checkpoints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I would read the equiangularity-based checkpoint selection as the least secure part of the package: the ablations show TVD alone already ranks well on vision models, so a cheaper checkpoint-selection rule might match NCIP on those tasks; the distinctive NC contribution appears mainly on transformer NLP models.
  • If cross-checkpoint variability behaves like a cheap ensemble of one training run, NCIP's ranking should degrade gracefully when adjacent checkpoints are highly correlated—an implicit prediction that could be tested by spacing checkpoints farther apart.
  • The retraining gains in RQ3 suggest high-NCIP samples are informative beyond fault discovery; a natural extension is to use the same score as an acquisition function for active learning or data pruning during model improvement.
  • The LLM result hints the signal transfers beyond classification; one could define TVD on output-token distributions for generative models and apply the same framework to hallucination or violation detection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes NCIP, a label-free test-case prioritization method for DNN classifiers. NCIP selects a subset of training checkpoints using a neural-collapse (NC) criterion (the standard deviation of pairwise cosine similarities among classifier weight vectors, Eq. 5), computes per-input total variation distance (TVD) between predictions at those checkpoints and the final checkpoint (Eq. 6), combines TVD with the final model's margin (Eq. 8), and ranks test inputs by the resulting score. The authors evaluate on 14 dataset-model pairs, on clean and adversarial data, against a range of baselines, with ablations, hyperparameter sensitivity plots, and a Wilcoxon signed-rank analysis of RAUC-ALL. The main claim is that cross-checkpoint prediction variability in the terminal training regime, guided by NC geometry, outperforms single-checkpoint confidence signals, with reported RAUC-ALL gains of 1.5%–16.6% and RAUC-500 gains of 4.9%–20.6%.

Significance. If the central claim holds, NCIP would provide a new, training-data-free signal for DNN test prioritization that exploits the geometric structure of late training, and the paper would be a useful contribution to the DNN testing literature. The experimental breadth is a strength: 14 dataset-model pairs, multiple baselines, clean and adversarial test sets, ablations, efficiency measurements, and a public code release. The Wilcoxon analysis on RAUC-ALL is a positive step. However, the paper's distinctive NC component — the σ_cos-based checkpoint selection — is not yet properly identified as the cause of the reported gains, and the headline early-budget claims lack statistical support. The significance of the contribution therefore depends on additional experiments that the current manuscript does not provide.

major comments (3)
  1. [§5.4, Tables 7–8, Algorithm 1] The central claim that NC-guided checkpoint selection contributes to NCIP is not supported by the ablations as reported. TVD+NC improves average RAUC-ALL over TVD alone (0.903 vs. 0.885), but this comparison does not control for the checkpoint-selection mechanism itself. A trivial baseline such as selecting K=30 checkpoints uniformly at random, or uniformly spaced by epoch, could plausibly produce the same or better results simply by providing diversity, regardless of σ_cos. The per-dataset pattern is mixed: e.g., IMDB-Transformer improves from 0.715 to 0.851, but FMNIST-LeNet-1 drops from 0.891 to 0.880. Please add experiments comparing the σ_cos-based selection of Algorithm 1 against random selection (with multiple seeds), uniform spacing, and using all available checkpoints. If random/uniform selection yields statistically indistinguishable RAUC-ALL or RAUC-500, the abstract's 'NC-gui
  2. [§5.4, Fig. 8; Tables 3–6] The default hyperparameters α=0.9 and K=30 are chosen from sensitivity curves computed on the same datasets used to report the main results (Fig. 8). This constitutes tuning on the evaluation data and can inflate the reported gains. Please either use a separate validation split for parameter selection, or demonstrate that the results are robust across a wide range of α and K (e.g., reporting mean and standard deviation over the grid). In addition, the abstract and RQ1 highlight RAUC-500 gains of 4.9%–20.6%, but the Wilcoxon tests in Table 5 are only for RAUC-ALL; RAUC-500 results are reported as single numbers without variance or significance tests. Since early-budget performance is a core claim, please provide per-run distributions and statistical tests for RAUC-500.
  3. [§5.1, Tables 3–6 and Fig. 6] The paper states in §5 that 'We repeated the process five times for experiments and took the average,' yet Tables 3, 4, and 6 report single values without any measure of dispersion. Many of the reported differences are small (e.g., MNIST LeNet-1 RAUC-ALL: 0.986 vs. DeepGini 0.983), and without confidence intervals or per-run results it is impossible to assess whether these differences are meaningful. Please report standard deviations or confidence intervals for at least the main effectiveness tables and the retraining experiment in Fig. 6, or make the per-run results publicly available.
minor comments (6)
  1. [Eq. (6) vs. Algorithm 2] Equation (6) defines TVD as an average over |S| checkpoints including the reference, while Algorithm 2 sums differences over |S|−1 checkpoints without an explicit normalization. Since the final score is z-scored, the scale does not affect the ranking, but the notation should be made consistent.
  2. [Tables 3–4] There are formatting artifacts: '0.9100' appears for Entropy on CIFAR10 VGG-11 in Table 3, and '0.6550' for TDPR on FMNIST LeNet-5 in Table 4. Please check the number formatting.
  3. [Abstract and §5.1] The abstract states that 'NCIP further attains the best average performance across all dataset-model pairs,' but on adversarial data (Table 6) TDPR has a higher average RAUC-ALL (0.946 vs. 0.937) and RAUC-500 (0.966 vs. 0.962). Please specify that the best-average claim refers to clean data, or otherwise qualify it.
  4. [Fig. 9] The 'weak NC' experiment varies the number of training epochs from 10 to 100, which changes both NC strength and the number of checkpoints used for TVD. This confounds the effect of NC strength with the amount of temporal information. Consider holding K fixed when comparing weak vs. strong NC.
  5. [Algorithm 1] Line 2 uses 'T←m', which is ambiguous; consider renaming to 'last_index' or 'ref' for readability.
  6. [Fig. 4b] The RAUC values for GPT-2 and OPT-125m (0.587 and 0.706) are not clearly defined as RAUC-ALL or RAUC-n. Please add metric definitions for the generative-task evaluation.

Circularity Check

0 steps flagged

No significant circularity: NCIP's ranking score is label-free, is not constructed from the RAUC objective, and its NC premise comes from external literature rather than self-citation.

full rationale

The claimed derivation chain is: select checkpoints by the equiangularity score sigma_cos (Eq. 5), which uses only classifier-weight geometry; compute per-input TVD across those checkpoints (Eq. 6); combine z-scored TVD with z-scored inverse margin (Eq. 8); rank test inputs by this label-free score; then evaluate with RAUC (Eq. 9). Nothing in this chain fits a parameter to fault labels or to the RAUC objective; Eq. 8 is not algebraically derived from Eq. 9, and RAUC appears only after the ranking is fixed. The neural-collapse premise is imported from external references [42,67], not from the authors' own prior theorems, and no self-citation is load-bearing. The main caveats are that alpha=0.9 and K=30 are selected from sensitivity curves on the same benchmark datasets later used for the headline RAUC results, and the ablation does not include a random-K or uniform-selection control for the NC-guided checkpoint selection. These are evaluation/tuning and evidence-strength concerns, not derivation circularity: they do not make the prioritization score equivalent to its inputs by construction. The paper's own threats-to-validity section also lists implementation, stochasticity, and deployment limitations but does not assert or conceal a circular step.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on empirical regularities rather than derivations: NC occurs in terminal training, σ_cos reflects NC quality, and cross-checkpoint TVD is higher for misclassified samples. Two hyperparameters are tuned on the evaluation datasets, and the equal-weight score combination is a construction choice. No new physical or model entities are introduced.

free parameters (3)
  • checkpoint pool ratio α = 0.9
    The last 90% of checkpoints form the candidate pool; chosen from sensitivity analysis in Section 5.4 on the same datasets used for headline results.
  • selected checkpoint count K = 30
    Target subset size; gains saturate around K=30 in Fig. 8, on the same evaluation datasets used for reported RAUC values.
  • score combination weights = 1 (TVD) + 1 (1−margin)
    Eq. 8 sums z-scored TVD and z-scored inverse margin with equal weight by construction; no tuning or justification is reported.
axioms (4)
  • domain assumption Neural Collapse occurs in the terminal training phase for the architectures and datasets used, and can be probed via classifier-weight cosine geometry.
    Section 2.3 and Section 3.3 rely on refs [42,67]; not established for all 14 dataset-model pairs, but used to justify checkpoint selection.
  • domain assumption Prediction variability across selected checkpoints (TVD) is higher for misclassified than correctly classified inputs.
    Section 3.4 and Fig. 3 present this as an empirical regularity; no derivation from NC properties is given.
  • ad hoc to paper Low σ_cos indicates checkpoints in a stable NC regime, and selecting farthest-in-score points yields a representative, informative subset.
    Algorithm 1 and Eq. 5 introduce this heuristic; it is the paper's own criterion and is not an established NC result.
  • ad hoc to paper Final-checkpoint margin and equal-weight fusion with TVD improve the ranking signal.
    Section 3.4–3.5 adopts this combination without an independence or redundancy analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 24216 in / 12209 out tokens · 85696 ms · 2026-08-01T10:55:26.093257+00:00 · methodology

0 comments
read the original abstract

With the widespread deployment of deep neural networks (DNNs) in safety-critical domains, reducing the cost of model validation under limited testing budgets has become increasingly important. Existing test case prioritization techniques often rely on single-checkpoint confidence signals derived from output probabilities. However, DNNs can be confidently wrong, and the confidence margin between the predicted and competing classes is frequently small, which weakens early fault discovery. To address this limitation, we propose a Neural-Collapse-Inspired Prioritization (NCIP) framework that replaces absolute confidence with cross-checkpoint prediction variability in the terminal training regime, where model geometry becomes highly structured. NCIP introduces two key components. First, it selects an NC-guided representative subset of training checkpoints using an equiangularity score of classifier weights, quantified as the standard deviation of pairwise cosine similarities among class weight vectors. Second, it prioritizes test inputs by their prediction variability across the selected checkpoints, surfacing boundary-adjacent and failure-prone samples that are unstable under checkpoint-induced decision boundary shifts. Extensive experiments across multiple datasets and architectures show that NCIP achieves strong performance in early fault discovery compared with competitive baselines, with 1.5 to 16.6 percent RAUC-ALL gains and 4.9 to 20.6 percent RAUC-500 gains under the same testing budget. NCIP further attains the best average performance across all dataset-model pairs.

Figures

Figures reproduced from arXiv: 2607.20046 by Chunyu Liu, Fei Gao, Mingyuan Li, Su-Juan Qin, Tengfei Tu, Wenmin Li, Yang Li.

Figure 1
Figure 1. Figure 1: Motivation for cross-checkpoint prediction variability on CIFAR10. Red points denote misclassified [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of NCIP. (1) NC checkpoint selection via weight equiangularity to obtain NC-guided model [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of prediction instability measured by TVD across training checkpoints. In both settings, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) Test case prioritization under model calibration. (b) NCIP performance on LLM generative tasks [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fault Diversity on Clean Data Answer to RQ1: NCIP consistently outperforms most baselines on both clean and adversarial data. It improves RAUC by 1.5 %–16.6 % and increases RAUC-500 by 4.9 %– 20.6 %, enabling earlier fault discovery. Under adversarial settings, NCIP remains robust with second-best average scores (RAUC-500/RAUC-ALL: 0.962/0.937), and performs well under model calibration and on LLM generati… view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy improvement after retraining with the top [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The impact of the checkpoint selection ratio/number on fault diversity [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The impact of the checkpoint selection ratio/number on RAUC [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: NCIP Performance under Weak Neural Collapse [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 10 canonical work pages

  1. [1]

    Amin Abbasishahkoo, Mahboubeh Dadkhah, Lionel Briand, and Dayi Lin. 2025. MetaSel: A Test Selection Approach for Fine-tuned DNN Models.arXiv preprint arXiv:2503.17534(2025). doi:10.1109/TSE.2025.3612253

    arXiv 2025

  2. [2]

    Hamzah Al-Qadasi, Changshun Wu, Yliès Falcone, and Saddek Bensalem. 2022. DeepAbstraction: 2-level prioritization for unlabeled test inputs in deep neural networks. In2022 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE, 64–71. doi:10.1109/aitest55621.2022.00018

    arXiv 2022

  3. [3]

    Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. 2022. Revisiting neural scaling laws in language and vision.Advances in Neural Information Processing Systems35 (2022), 22300–22312. doi:10.52202/068431-1620

    work page doi:10.52202/068431-1620 2022

  4. [4]

    Shenglin Bao, Chaofeng Sha, Bihuan Chen, Xin Peng, and Wenyun Zhao. 2023. In defense of simple techniques for neural network test case selection. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 501–513. doi:10.1145/3597926.3598073

    arXiv 2023

  5. [5]

    Taejoon Byun, Vaibhav Sharma, Abhishek Vijayakumar, Sanjai Rayadurgam, and Darren Cofer. 2019. Input prioritization for testing neural networks. In2019 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE, 63–70. doi:10.1109/AITest.2019.000-6

    doi:10.1109/aitest.2019.000-6 2019

  6. [6]

    Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In2017 ieee symposium on security and privacy (sp). Ieee, 39–57. doi:10.1109/SP.2017.49

    doi:10.1109/sp.2017.49 2017

  7. [8]

    Xueqi Dang, Yinghua Li, Mike Papadakis, Jacques Klein, Tegawendé F Bissyandé, and Yves Le Traon. 2024. Test input prioritization for machine learning classifiers.IEEE Transactions on Software Engineering(2024). doi:10.1109/TSE.2024. 3350019

    doi:10.1109/tse.2024 2024

  8. [9]

    Yang Feng, Qingkai Shi, Xinyu Gao, Jun Wan, Chunrong Fang, and Zhenyu Chen. 2020. Deepgini: prioritizing massive tests to enhance the robustness of deep neural networks. InProceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 177–188. doi:10.1145/3395363.3397357

    arXiv 2020

  9. [10]

    Xinyu Gao, Yang Feng, Yining Yin, Zixi Liu, Zhenyu Chen, and Baowen Xu. 2022. Adaptive test selection for deep neural networks. InProceedings of the 44th International Conference on Software Engineering. 73–85. doi:10.1145/ 3510003.3510232

    arXiv 2022

  10. [11]

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572(2014). doi:10.48550/arXiv.1412.6572

    Pith/arXiv arXiv doi:10.48550/arxiv.1412.6572 2014

  11. [12]

    Hongjing Guo, Chuanqi Tao, Zhiqiu Huang, and Weiqin Zou. 2025. White-Box Test Input Generation for Enhancing Deep Neural Network Models through Suspicious Neuron Awareness.ACM Transactions on Software Engineering and Methodology(2025). doi:10.1145/3736305

    work page doi:10.1145/3736305 2025

  12. [13]

    Jianmin Guo, Yu Jiang, Yue Zhao, Quan Chen, and Jiaguang Sun. 2018. Dlfuzz: Differential fuzzing testing of deep learning systems. InProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 739–743. doi:10.1145/3236024.3264835

    arXiv 2018

  13. [14]

    Fabrice Harel-Canada, Lingxiao Wang, Muhammad Ali Gulzar, Quanquan Gu, and Miryung Kim. 2020. Is neuron coverage a meaningful measure for testing deep neural networks?. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 851–862. doi:10.1145/3368089.3409754

    arXiv 2020

  14. [15]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778. doi:10.1109/CVPR.2016.90

    doi:10.1109/cvpr.2016.90 2016

  15. [16]

    Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Lei Ma, Mike Papadakis, and Yves Le Traon. 2024. Test optimization in DNN testing: a survey.ACM Transactions on Software Engineering and Methodology33, 4 (2024), 1–42. doi:10.1145/ 3643678

    2024

  16. [17]

    Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Wei Ma, Mike Papadakis, Lei Ma, and Yves Le Traon. 2025. Assessing the Robustness of Test Selection Methods for Deep Neural Networks.ACM Transactions on Software Engineering and Methodology(2025). doi:10.1145/3715693

    doi:10.1145/3715693 2025

  17. [19]

    Dong Huang, Qingwen Bu, Yichao Fu, Yuhao Qing, Xiaofei Xie, Junjie Chen, and Heming Cui. 2024. Neuron Sensitivity- Guided Test Case Selection.ACM Transactions on Software Engineering and Methodology33, 7 (2024), 1–32. doi:10. 1145/3672454

    2024

  18. [20]

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition. 4700–4708. doi:10.1109/ CVPR.2017.243

    2017

  19. [21]

    Hyekyoung Hwang, Il Yong Chun, and Jitae Shin. 2023. Improved Test Input Prioritization Using Verification Monitors with False Prediction Cluster Centroids.Electronics13, 1 (2023), 21. doi:10.3390/electronics13010021

    work page doi:10.3390/electronics13010021 2023

  20. [22]

    Sungmin Kang, Robert Feldt, and Shin Yoo. 2024. Deceiving humans and machines alike: Search-based test input generation for dnns using variational autoencoders.ACM Transactions on Software Engineering and Methodology33, 4 (2024), 1–24. doi:10.1145/3635706

    doi:10.1145/3635706 2024

  21. [23]

    Hoki Kim. 2020. Torchattacks: A pytorch repository for adversarial attacks.arXiv preprint arXiv:2010.01950(2020). doi:10.48550/arXiv.2010.01950

    Pith/arXiv arXiv doi:10.48550/arxiv.2010.01950 2020

  22. [24]

    Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding deep learning system testing using surprise adequacy. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1039–1049. doi:10.1109/ICSE.2019.00108

    arXiv 2019

  23. [25]

    Jinhan Kim, Robert Feldt, and Shin Yoo. 2023. Evaluating surprise adequacy for deep learning system testing.ACM Transactions on Software Engineering and Methodology32, 2 (2023), 1–29. doi:10.1145/3546947

    work page doi:10.1145/3546947 2023

  24. [26]

    Jinhan Kim, Jeongil Ju, Robert Feldt, and Shin Yoo. 2020. Reducing dnn labelling cost using surprise adequacy: An industrial case study for autonomous driving. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1466–1476. doi:10.1145/3368089. 3417065

    doi:10.1145/3368089 2020

  25. [27]

    Yoon Kim. 2014. Convolutional neural networks for sentence classification.arXiv preprint arXiv:1408.5882(2014). doi:10.3115/v1/D14-1181

    Pith/arXiv arXiv 2014

  26. [28]

    Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009)

    2009

  27. [29]

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. 2018. Adversarial examples in the physical world. InArtificial intelligence safety and security. Chapman and Hall/CRC, 99–112. doi:10.1201/9781351251389-8

    doi:10.1201/9781351251389-8 2018

  28. [30]

    Yann Le and Xuan Yang. 2015. Tiny imagenet visual recognition challenge.CS 231N7, 7 (2015), 3

    2015

  29. [31]

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition.Proc. IEEE86, 11 (1998), 2278–2324. doi:10.1109/5.726791

    doi:10.1109/5.726791 1998

  30. [32]

    Maoxi Li, Daobo Ma, and Yingqi Zhang. 2025. Improving database anomaly detection efficiency through sample difficulty estimation. (2025). doi:10.20944/preprints202504.1527.v1

    arXiv 2025

  31. [33]

    Wei Li, Zhiyi Zhang, Yifan Jian, Chen Liu, and Zhiqiu Huang. 2023. DeepRank: Test Case Prioritization for Deep Neural Networks.. InSEKE. 262–267. doi:10.18293/seke2023-188

    work page doi:10.18293/seke2023-188 2023

  32. [34]

    Yinghua Li, Xueqi Dang, Jacques Klein, Yves Le Traon, and Tegawendé F Bissyandé. 2025. PriCod: Prioritizing Test Inputs for Compressed Deep Neural Networks.ACM Transactions on Software Engineering and Methodology(2025). doi:10.1145/3730435

    work page doi:10.1145/3730435 2025

  33. [35]

    Zhong Li, Zhengfeng Xu, Ruihua Ji, Minxue Pan, Tian Zhang, Linzhang Wang, and Xuandong Li. 2024. Distance-Aware Test Input Selection for Deep Neural Networks. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 248–260. doi:10.1145/3650212.3652125

    arXiv 2024

  34. [36]

    Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, et al

  35. [37]

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. InProceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies. 142–150

    2011

  36. [38]

    Thomas W MacFarland and Jan M Yates. 2016. Wilcoxon matched-pairs signed-ranks test. InIntroduction to Nonpara- metric statistics for the biological sciences using R. Springer, 133–175. doi:10.4135/9780857020123.n633

    work page doi:10.4135/9780857020123.n633 2016

  37. [39]

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083(2017). doi:10.48550/arXiv.1706.06083

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.06083 2017

  38. [40]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843(2016). doi:10.48550/arXiv.1609.07843

    Pith/arXiv arXiv doi:10.48550/arxiv.1609.07843 2016

  39. [41]

    2015.Neural networks and deep learning

    Michael A Nielsen. 2015.Neural networks and deep learning. Vol. 25. Determination press San Francisco, CA, USA

    2015

  40. [42]

    Vardan Papyan, XY Han, and David L Donoho. 2020. Prevalence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences117, 40 (2020), 24652–24663. doi:10.1073/pnas. 2015509117

    doi:10.1073/pnas 2020

  41. [43]

    Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Automated whitebox testing of deep learning systems. Inproceedings of the 26th Symposium on Operating Systems Principles. 1–18. doi:10.1145/3132747.3132785 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA092. Publication date: October 2026. Test Case Prioritization for DNNs via ...

    arXiv 2017

  42. [44]

    Xin Qiu and Risto Miikkulainen. 2022. Detecting misclassification errors in neural networks with a gaussian process model. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8017–8027. doi:10.1609/aaai.v36i7.20773

    work page doi:10.1609/aaai.v36i7.20773 2022

  43. [45]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog1, 8 (2019), 9

    2019

  44. [46]

    Jian Shen, Zhong Li, Minxue Pan, and Xuandong Li. 2024. Prioritizing Test Inputs for DNNs Using Training Dynamics. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1219–1231. doi:10. 1145/3691620.3695498

    arXiv 2024

  45. [47]

    Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556(2014). doi:10.48550/arXiv.1409.1556

    Pith/arXiv arXiv doi:10.48550/arxiv.1409.1556 2014

  46. [48]

    Jack Stewart. 2018. Tesla’s autopilot was involved in another deadly car crash.Wired3 (2018), 30

    2018

  47. [49]

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. InProceedings of the IEEE conference on computer vision and pattern recognition. 2818–2826. doi:10.1109/cvpr.2016.308

    doi:10.1109/cvpr.2016.308 2016

  48. [50]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017). doi:10.48550/arXiv.1706.03762

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.03762 2017

  49. [52]

    Jingling Wang, Huayao Wu, Peng Wang, Xintao Niu, and Changhai Nie. 2025. SETS: A Simple yet Effective DNN Test Selection Approach.ACM Transactions on Software Engineering and Methodology(2025). doi:10.1145/3772084

    work page doi:10.1145/3772084 2025

  50. [53]

    Zan Wang, Hanmo You, Junjie Chen, Yingyi Zhang, Xuyuan Dong, and Wenbin Zhang. 2021. Prioritizing test inputs for deep neural networks via mutation analysis. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 397–409. doi:10.1109/ICSE43902.2021.00046

    arXiv 2021

  51. [54]

    Zhengyuan Wei, Haipeng Wang, Imran Ashraf, and WK Chan. 2022. Predictive mutation analysis of test case prioritization for deep neural networks. In2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS). IEEE, 682–693. doi:10.1109/QRS57517.2022.00074

    arXiv 2022

  52. [55]

    Michael Weiss and Paolo Tonella. 2022. Simple techniques work surprisingly well for neural network test prioritization and active learning (replicability study). InProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 139–150. doi:10.1145/3533767.3534375

    arXiv 2022

  53. [56]

    Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747(2017). doi:10.48550/arXiv.1708.07747

    Pith/arXiv arXiv doi:10.48550/arxiv.1708.07747 2017

  54. [57]

    Yining Yin, Yang Feng, Shihao Weng, Xinyu Gao, Jia Liu, and Zhihong Zhao. 2025. Lightweight Probabilistic Coverage Metrics for Efficient Testing of Deep Neural Networks. InProceedings of the 16th International Conference on Internetware. 474–486. doi:10.1145/3755881.3755915

    arXiv 2025

  55. [58]

    Hanmo You, Zan Wang, Junjie Chen, Shuang Liu, and Shuochuan Li. 2023. Regression fuzzing for deep learning systems. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 82–94. doi:10.1109/ icse48619.2023.00019

    arXiv 2023

  56. [59]

    Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization.arXiv preprint arXiv:1710.09412(2017). doi:10.48550/arXiv.1710.09412

    Pith/arXiv arXiv doi:10.48550/arxiv.1710.09412 2017

  57. [60]

    Kai Zhang, Yongtai Zhang, Liwei Zhang, Hongyu Gao, Rongjie Yan, and Jun Yan. 2020. Neuron activation frequency based test case prioritization. In2020 International Symposium on Theoretical Aspects of Software Engineering (TASE). IEEE, 81–88. doi:10.1109/tase49443.2020.00020

    arXiv 2020

  58. [61]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068 (2022). doi:10.48550/arXiv.2205.01068

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.01068 2022

  59. [62]

    Xiyue Zhang, Xiaofei Xie, Lei Ma, Xiaoning Du, Qiang Hu, Yang Liu, Jianjun Zhao, and Meng Sun. 2020. Towards characterizing adversarial defects of deep learning software from the lens of uncertainty. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 739–751. doi:10.1145/3377811.3380368

    arXiv 2020

  60. [63]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems28 (2015)

    2015

  61. [64]

    Zhiyi Zhang, Huanze Meng, Yuchen Ding, Shuxian Chen, and Yongming Yao. 2025. Efficient adaptive test case selection for DNNs robustness enhancement.Journal of Systems and Software(2025), 112451. doi:10.2139/ssrn.4978392

    work page doi:10.2139/ssrn.4978392 2025

  62. [65]

    Haibin Zheng, Jinyin Chen, and Haibo Jin. 2023. CertPri: certifiable prioritization for deep neural networks via movement cost in feature space. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1–13. doi:10.1109/ASE56229.2023.00126 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA092. Publication date: Oct...

    arXiv 2023

  63. [66]

    Haibin Zheng, Zhiqing Chen, Tianyu Du, Xuhong Zhang, Yao Cheng, Shouling Ji, Jingyi Wang, Yue Yu, and Jinyin Chen. 2022. Neuronfair: Interpretable white-box fairness testing through biased neuron identification. InProceedings of the 44th International Conference on Software Engineering. 1519–1531. doi:10.1145/3510003.3510123

    arXiv 2022

  64. [67]

    Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. 2021. A geometric analysis of neural collapse with unconstrained features.Advances in Neural Information Processing Systems34 (2021), 29820–29834. doi:10.48550/arXiv.2105.02375 Received 2026-01-30; accepted 2026-06-25 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article IS...

    Pith/arXiv arXiv doi:10.48550/arxiv.2105.02375 2021

  65. [2018]

    InProceedings of the 33rd ACM/IEEE international conference on automated software engineering

    Deepgauge: Multi-granularity testing criteria for deep learning systems. InProceedings of the 33rd ACM/IEEE international conference on automated software engineering. 120–131. doi:10.1145/3238147.3238202

    arXiv