Pith. sign in

REVIEW 3 major objections 5 minor 27 references

AdaStop: Cost-Aware Early Stopping for DNN Test Selection

T0 review · 3 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DNN test labeling can stop when the estimated fault rate falls below the cost-to-value ratio, finding most faults with a fraction of the budget.

desk verdict Solid engineering packaging of a standard cost-benefit stop for DNN test selection; useful numbers, elementary math, vision-only scope. read the letter →

arxiv 2607.05461 v1 pith:LCZ5COGY submitted 2026-07-06 cs.LG cs.AI

Bonan Shen , Wei-Jung Huang , Xin Liu , Jiazhou Gao , Tao Ning This is my paper · ORCID
classification cs.LGcs.AI
keywords DNNtestingtestselectionearlystoppingcost-benefitfaultdiscoverylabelingbudgetdiminishingreturnsDeepGini
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Choosing how many test inputs to label for a deep neural network is hard: too few misses faults, too many wastes expert labeling. This paper treats testing as a sequential cost–benefit process in which each label costs c and each discovered fault is worth v, and derives a simple optimal rule: stop once the marginal chance of finding a fault drops to or below τ = c/v. AdaStop estimates that chance with a sliding window over recent outcomes and halts labeling when the estimate crosses the threshold. Across several image datasets, model architectures, quality levels, and selection strategies, the method recovers 65–84% of faults while using only about 9–31% of the full labeling budget, often improving net value over exhaustive testing. A reader cares because fixed budgets force an arbitrary guess; a cost-aware stop replaces that guess with a break-even rate that practitioners can set from their own costs and values.

What carries the argument

The break-even threshold τ = c/v together with a sliding-window estimator of the marginal fault rate. It converts the continue/stop decision into a direct cost–benefit comparison: keep labeling while the estimated fault probability still exceeds the cost-to-value ratio; stop when it does not.

What would settle it

On a pool whose faults cluster outside the early uncertainty ranking, if the windowed fault rate repeatedly dips below τ then rises well above it, early stopping would miss a large share of faults that a larger fixed budget still finds—undermining both optimality and the reported savings.

Watch

Extended reading notes

Core claim

Under diminishing returns from uncertainty-based selection, the optimal policy stops at the first T where the fault discovery rate p(T) ≤ τ = c/v. AdaStop implements this with a sliding-window estimator of p(t) and shows that 65–84% of faults can be found using only 9–31% of the labeling budget across the evaluated datasets, architectures, and strategies, with higher net value than labeling everything.

Load-bearing premise

The method assumes that once tests are ordered by uncertainty, the chance of finding a new fault keeps falling so that once it drops below the break-even rate it stays there.

Editorial extensions

If this is right

  • Practitioners can set τ from real labeling cost and fault value instead of guessing a budget percentage.
  • Uncertainty-based selectors produce similar diminishing-return curves, so the same stop rule yields consistent budget–recall points across strategies.
  • Worse models automatically receive more budget and better models less, without retuning τ.
  • Alternative rules (patience, consecutive non-faults, confidence bounds) form a spectrum from highest efficiency to near-complete recall.
  • Exhaustive labeling can produce lower net value than early stopping under the same c and v.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same break-even-rate idea could govern any costly oracle under prioritization—bug triage, document screening, or safety audits—not only DNN test selection.
  • Batch labeling would need a natural extension: decide after each batch whether the windowed rate still justifies the next batch’s cost.
  • If multi-modal or adversarial failures break monotonicity, a mixture-aware or resettable window would be required before the optimality claim transfers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the stopping problem in DNN test selection: when to stop labeling under a cost–benefit trade-off. It formulates sequential testing via net value V(T)=v·F(T)−c·T, derives the optimal threshold τ=c/v under a non-increasing marginal fault rate p(t) (Proposition 1), and implements AdaStop with DeepGini (or other) ranking, a sliding-window estimator of p(t) (Eq. 4), and a threshold stopping rule (Algorithm 1). Alternative criteria (patience, consecutive non-faults, Wilson CI) are also defined. Experiments on CIFAR-10, SVHN, and FashionMNIST with ResNet-20, VGG-16, DenseNet-121, and ShuffleNetV2, three quality levels, and eight selection strategies report 65–84% fault recall with roughly 9–31% of the labeling budget (Tables III, VII–X), with Mann–Kendall tests supporting diminishing returns on the evaluated vision settings (Table XI).

Significance. The stopping problem is practically important and under-addressed relative to selection methods that assume fixed budgets. Framing the decision as a simple cost–benefit threshold is clear and immediately usable by practitioners who can estimate c and v. Strengths include an explicit optimality condition under a stated monotonicity assumption, a strategy-agnostic design, broad empirical coverage (datasets, architectures, quality levels, strategies, and multiple stopping rules), and open-source intent. The contribution is incremental rather than foundational—the optimality argument is elementary once net value is defined—but it fills a genuine gap between selection literature and operational labeling practice, and the empirical savings numbers are useful if they hold under the stated assumptions.

major comments (3)
  1. Proposition 1 and §II-D make non-increasing p(t) load-bearing: once the rate crosses τ it stays below, so first-crossing is optimal. Validation is only empirical (Mann–Kendall on three vision datasets, Table XI). §VI-J already notes possible failure for multi-modal failure clusters or adversarial inputs. The central claim therefore needs either (i) a stronger theoretical condition under which uncertainty ranking yields monotone p(t), or (ii) explicit stress experiments (e.g., multi-modal synthetic faults, adversarial or OOD pools) showing how AdaStop behaves when the assumption is violated, and what practitioners should do then. Without that, the optimality guarantee remains conditional on a premise that is only partially checked.
  2. §II-A–B and the net-value metric (v=20, c=1 throughout §V–VI) treat c and v as known constants. In practice both are uncertain; τ is therefore a free parameter. RQ2 (Table IV) shows stepped sensitivity (τ∈[0.01,0.05] vs [0.10,0.20]), but there is no procedure for choosing or robustifying τ when c/v is misspecified. A short sensitivity analysis under noisy or interval estimates of c and v, or a clearer recommendation of when to prefer consecutive-non-fault / confidence criteria that avoid explicit τ, is needed for the cost–benefit framing to be operational.
  3. All results are on image classification with offline ranked pools and sequential single-label queries (Tables II–X). The abstract and conclusion claim a general DNN testing framework. §VI-J acknowledges NLP/tabular/batch settings as future work; those limitations should be reflected more carefully in the abstract and contributions so that the reported 65–84% / 9–31% numbers are not over-generalized beyond the evaluated regime.
minor comments (5)
  1. Abstract and §I state 9–31% budget; Table III and several RQ summaries use 23.5% as the headline CIFAR-10 figure, while cross-dataset averages (Table VII) are lower. Align the range language with the tables.
  2. Figure 1 and Algorithm 1 are clear; Figures 2–3 would benefit from error bars or multiple seeds where randomness appears (e.g., Random strategy in Table IX).
  3. Related work (Table I, §III) is adequate but could briefly note how AdaStop differs from classical sequential testing / SPRT-style rules beyond SAFE and AL stopping.
  4. Notation: r_t is defined as an indicator but sometimes treated as a rate in prose; keep the distinction between r_t and p(t)/ˆp(t) consistent.
  5. Default parameters (τ=0.05, W=20, N_min=50) are stated in §V-B; a short justification or pointer to the ablation tables in the main text would help readers who skip the RQs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: optimality of τ=c/v follows directly from the defined net-value objective, and empirical results are measured against independent fault counts.

full rationale

The paper defines net value V(T)=v·F(T)-c·T (Eq. 1), derives the marginal ΔV(t)=v·p(t)-c (Eq. 2), and obtains the elementary stopping threshold τ=c/v (Eq. 3 / Proposition 1) under the stated non-increasing-p(t) assumption. This is ordinary sequential decision theory applied to a user-chosen cost-benefit ratio; it is not fitted to the reported recall numbers, nor is any target quantity recovered by construction from a free parameter. The sliding-window estimator (Eq. 4) is a transparent smoother whose window size W is ablated (Table VI) rather than tuned to force a desired outcome. Empirical claims (65–84 % recall at 9–31 % budget) are obtained by running the procedure on held-out labeled test pools and counting discovered faults versus labels used (Tables III, VII–X); they are not predictions of quantities already used in the fit. Diminishing returns is validated post-hoc by Mann-Kendall tests (Table XI) and is explicitly caveated for multi-modal or adversarial regimes (§VI-J). There are no load-bearing self-citations, uniqueness theorems imported from the authors, or renamed known results that close a circular loop. Free parameters (τ, W, N_min) are reported openly and compared against alternatives; their existence does not constitute circularity under the stated criteria. The derivation chain is therefore self-contained.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a standard marginal-value argument plus three free parameters that control the estimator and the break-even point, and on the empirical domain assumption that guided selection yields a non-increasing fault rate. No new physical or mathematical entities are postulated; the framework re-uses existing uncertainty scores and a sliding-window average.

free parameters (3)
  • τ = c/v (default 0.05) = 0.05
    Break-even fault rate that directly sets the stopping point; default chosen by hand and shown to produce stepped plateaus (Table IV).
  • sliding-window size W (default 20) = 20
    Controls bias-variance of the rate estimator; W=10 causes premature stopping while W≥20 is stable (Table VI).
  • minimum samples N_min (default 50) = 50
    Prevents stopping before the window is reliable; reported as robust once the natural stopping point exceeds it.
assumptions (3)
  • standard math Marginal net value of the next label is v·p(t)−c; stop when this is non-positive.
    Standard optimal-stopping / cost-benefit rearrangement used in Proposition 1 (§II-B).
  • domain assumption Under uncertainty-based selection the fault rate p(t) is non-increasing (diminishing returns).
    Required for the 'stop at first crossing' policy to be optimal; validated only empirically via Mann-Kendall tests (Table XI) and flagged as a limitation for multi-modal or adversarial faults (§VI-J).
  • ad hoc to paper A sliding window of recent binary outcomes is a sufficient estimator of the current marginal fault rate.
    Chosen for non-stationarity, noise, and responsiveness (§II-C); alternatives (cumulative rate) are shown to fail but the window form itself is a design choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaStop: Cost-Aware Early Stopping for DNN Test Selection." pith.science (2026). https://pith.science/paper/LCZ5COGY

@misc{pith2026260705461,
  author       = {Pith},
  title        = {Pith review of: AdaStop: Cost-Aware Early Stopping for DNN Test Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCZ5COGY}},
  note         = {Machine review of arXiv:2607.05461}
}
abstract

Existing methods for testing deep neural networks (DNNs) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget. In practice, choosing that budget is difficult: too little testing misses failures, while too much incurs unnecessary labeling costs. This work studies the stopping problem in DNN testing. We formulate testing as a cost--benefit decision process in which labeling an input incurs cost $c$ and discovering a fault yields value $v$. Based on this formulation, we introduce \textit{AdaStop}, a framework that estimates the marginal fault discovery rate during testing and stops labeling when the estimated rate falls below the threshold $\tau = c/v$. Experiments across multiple datasets, architectures, and selection strategies show that $65$--$84\%$ of faults can be discovered using only $9$--$31\%$ of the labeling budget.

Figures

Figures reproduced from arXiv: 2607.05461 by the authors.

Figure 1
Figure 1. AdaStop Framework. The system iteratively selects inputs via [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. compares net value across methods. The oracle achieves peak net value at exactly 11.96% budget (the error rate), where all faults are found with zero waste. DeepGini peaks at 50% and then decreases as costs outpace discovery. AdaStop achieves 90% of DeepGini’s peak value while using 27 percentage points less budget. Notably, exhaustive testing yields lower net value (13,920) than AdaStop (16,552), demonstrating that… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 4 linked inside Pith

  1. [1]

    Learning Multiple Layers of Features from Tiny Images,

    A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” Tech. Rep., University of Toronto, 2009

    2009

  2. [2]

    Reading Digits in Natural Images with Unsuper- vised Feature Learning,

    Y . Netzer et al., “Reading Digits in Natural Images with Unsuper- vised Feature Learning,” inNIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. TABLE XI DIMINISHINGRETURNS: APFDANDMANN-KENDALLTESTS Dataset Strategy APFD MKτ p-val Trend CIFAR-10 DeepGini 0.849−0.727<10 −4 Decreasing CIFAR-10 Entropy 0.850−0.744<10 −4 Decreasing CIFAR-...

    2011

  3. [3]

    Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms,

    H. Xiao, K. Rasul, and R. V ollgraf, “Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms,” arXiv:1708.07747, 2017

    arXiv 2017

  4. [4]

    Deep Residual Learning for Image Recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” inProc. CVPR, 2016

    2016

  5. [5]

    Very Deep Convolutional Networks for Large-Scale Image Recognition,

    K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” inProc. ICLR, 2015

    2015

  6. [6]

    Densely Connected Convolutional Networks,

    G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely Connected Convolutional Networks,” inProc. CVPR, 2017

    2017

  7. [7]

    ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design,

    N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design,” inProc. ECCV, 2018

    2018

  8. [8]

    Probable Inference, the Law of Succession, and Statistical Inference,

    E. B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference,”J. Amer. Statist. Assoc., vol. 22, no. 158, pp. 209–212, 1927

    1927

Show all 27 references
  1. [9]

    DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networks,

    Y . Feng et al., “DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networks,” inProc. ISSTA, 2020

    2020

  2. [10]

    TestRank: Bringing order into unlabeled test instances for deep learning tasks,

    Z. Li et al., “TestRank: Bringing order into unlabeled test instances for deep learning tasks,” inProc. NeurIPS, 2021

    2021

  3. [11]

    Adaptive Test Selection for Deep Neural Networks,

    X. Gao et al., “Adaptive Test Selection for Deep Neural Networks,” inProc. ICSE, 2022

    2022

  4. [12]

    DeepSample: DNN Sampling-Based Testing for Operational Accuracy Assessment,

    A. Guerriero, R. Pietrantuono, and S. Russo, “DeepSample: DNN Sampling-Based Testing for Operational Accuracy Assessment,” in Proc. ICSE, 2024

    2024

  5. [13]

    A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User- Adjustable Stopping,

    M. Bloodgood and K. Vijay-Shanker, “A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User- Adjustable Stopping,” inProc. CoNLL, 2009

    2009

  6. [14]

    Stopping Criterion for Active Learning Based on Deterministic Generalization Bounds,

    H. Ishibashi and H. Hino, “Stopping Criterion for Active Learning Based on Deterministic Generalization Bounds,” inProc. AISTATS, 2020

    2020

  7. [15]

    FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection,

    J. Chen et al., “FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection,” inProc. ASE, 2024

    2024

  8. [16]

    DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks,

    Z. Aghababaeyan et al., “DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks,” inProc. FSE, 2024

    2024

  9. [17]

    CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space,

    H. Zheng et al., “CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space,” inProc. ASE, 2023

    2023

  10. [18]

    The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses,

    J. Boetje and R. van de Schoot, “The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses,”Systematic Reviews, vol. 13, art. 81, 2024

    2024

  11. [19]

    Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion,

    Y . Yuan, Q. Pang, and S. Wang, “Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion,” in Proc. ICSE, 2023

    2023

  12. [20]

    Stopping Criterion for Active Learning Based on Error Stability,

    H. Ishibashi and H. Hino, “Stopping Criterion for Active Learning Based on Error Stability,” arXiv:2104.01836, 2021

    2021 arXiv

  13. [21]

    Using Chao’s Estimator as a Stopping Criterion for Technology-Assisted Review,

    M. P. Bron et al., “Using Chao’s Estimator as a Stopping Criterion for Technology-Assisted Review,” arXiv:2404.01176, 2024

    2024 arXiv

  14. [22]

    Cost-Effective Testing of a Deep Learning Model through Input Reduction,

    J. Zhou et al., “Cost-Effective Testing of a Deep Learning Model through Input Reduction,” inProc. ISSRE, 2020

    2020

  15. [23]

    Adaptive Labeling for Efficient Out-of-distribution Model Evaluation,

    D. Mittal et al., “Adaptive Labeling for Efficient Out-of-distribution Model Evaluation,” inProc. NeurIPS, 2024

    2024

  16. [24]

    DeepTest: Automated Testing of Deep-Neural- Network-Driven Autonomous Cars,

    Y . Tian et al., “DeepTest: Automated Testing of Deep-Neural- Network-Driven Autonomous Cars,” inProc. ICSE, 2018

    2018

  17. [25]

    Dermatologist-level classification of skin cancer with deep neural networks,

    A. Esteva et al., “Dermatologist-level classification of skin cancer with deep neural networks,”Nature, vol. 542, pp. 115–118, 2017

    2017

  18. [26]

    Deep Learning for Financial Applications: A Survey,

    A. M. Ozbayoglu et al., “Deep Learning for Financial Applications: A Survey,” arXiv:2002.05786, 2020

    2002 arXiv

  19. [27]

    Active Learning Literature Survey,

    B. Settles, “Active Learning Literature Survey,” Tech. Rep. 1648, University of Wisconsin-Madison, 2010

    2010

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.