{"id":"e0ab34f1-3d0a-4d1b-a8ae-87a18a6536b4","arxiv_id":"2607.05461","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A sliding-window estimate of the marginal fault rate lets DNN test selection stop when that rate drops below the break-even threshold τ=c/v, saving most of the labeling budget while retaining most faults.","lead":"AdaStop stops DNN test labeling when the recent fault-discovery rate falls below a cost-to-value threshold τ=c/v. It reports 65–84% of faults found with only 9–31% of the labeling budget across vision models and selection strategies.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged monotonicity assumption.","rationale":"The reader's strongest claim accurately restates Abstract + Proposition 1 + the reported tables. The weakest assumption they flag (non-increasing p(t) under uncertainty-based selection) is precisely the load-bearing premise of the optimality argument; the paper validates it only empirically on three vision datasets and explicitly caveats multi-modal/adversarial regimes. No deeper flaw—hidden circularity, incorrect derivation of τ = c/v, or contradictory experimental numbers—appears on a second pass. Parameter sensitivity and vision-only scope are real but secondary and already noted. Consequently the CONDITIONAL verdict with medium correctness risk remains appropriate; no adjustment is warranted.","tokens_in":10250,"tokens_out":497,"duration_ms":5573,"concrete_test":"Re-run the Mann-Kendall analysis of Table XI on a non-vision corpus (e.g., a text-classification model with multi-modal error clusters or an adversarially perturbed CIFAR-10 pool) using the same DeepGini ordering; if MK τ is no longer significantly negative (p ≥ 0.05) while AdaStop still claims the same budget-recall numbers, the generality of Proposition 1 fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Proposition 1: under non-increasing p(t), the first crossing of τ = c/v is optimal. The paper states this cleanly, implements a sliding-window estimator (Eq. 4), and supplies empirical support via Mann-Kendall tests (Table XI) plus consistent results across 3 datasets, 4 architectures, 3 quality levels, and 8 strategies (Tables III, VII–X). The reader already correctly identifies the weakest premise—that diminishing returns hold outside the evaluated vision setting—and the paper itself flags multi-modal/adversarial failure modes in §VI-J. No additional internal inconsistency, circular construction, or derivation error is present; the free parameters (τ, W, N_min) are ablated and the optimality condition is elementary. The concern is therefore already captured; it does not require a stronger verdict shift.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies the stopping problem in DNN test selection: when to stop labeling under a cost–benefit trade-off. It formulates sequential testing via net value V(T)=v·F(T)−c·T, derives the optimal threshold τ=c/v under a non-increasing marginal fault rate p(t) (Proposition 1), and implements AdaStop with DeepGini (or other) ranking, a sliding-window estimator of p(t) (Eq. 4), and a threshold stopping rule (Algorithm 1). Alternative criteria (patience, consecutive non-faults, Wilson CI) are also defined. Experiments on CIFAR-10, SVHN, and FashionMNIST with ResNet-20, VGG-16, DenseNet-121, and ShuffleNetV2, three quality levels, and eight selection strategies report 65–84% fault recall with roughly 9–31% of the labeling budget (Tables III, VII–X), with Mann–Kendall tests supporting diminishing returns on the evaluated vision settings (Table XI).","tokens_in":10447,"tokens_out":1109,"duration_ms":11041,"significance":"The stopping problem is practically important and under-addressed relative to selection methods that assume fixed budgets. Framing the decision as a simple cost–benefit threshold is clear and immediately usable by practitioners who can estimate c and v. Strengths include an explicit optimality condition under a stated monotonicity assumption, a strategy-agnostic design, broad empirical coverage (datasets, architectures, quality levels, strategies, and multiple stopping rules), and open-source intent. The contribution is incremental rather than foundational—the optimality argument is elementary once net value is defined—but it fills a genuine gap between selection literature and operational labeling practice, and the empirical savings numbers are useful if they hold under the stated assumptions.","major_comments":[{"comment":"Proposition 1 and §II-D make non-increasing p(t) load-bearing: once the rate crosses τ it stays below, so first-crossing is optimal. Validation is only empirical (Mann–Kendall on three vision datasets, Table XI). §VI-J already notes possible failure for multi-modal failure clusters or adversarial inputs. The central claim therefore needs either (i) a stronger theoretical condition under which uncertainty ranking yields monotone p(t), or (ii) explicit stress experiments (e.g., multi-modal synthetic faults, adversarial or OOD pools) showing how AdaStop behaves when the assumption is violated, and what practitioners should do then. Without that, the optimality guarantee remains conditional on a premise that is only partially checked.","section":null},{"comment":"§II-A–B and the net-value metric (v=20, c=1 throughout §V–VI) treat c and v as known constants. In practice both are uncertain; τ is therefore a free parameter. RQ2 (Table IV) shows stepped sensitivity (τ∈[0.01,0.05] vs [0.10,0.20]), but there is no procedure for choosing or robustifying τ when c/v is misspecified. A short sensitivity analysis under noisy or interval estimates of c and v, or a clearer recommendation of when to prefer consecutive-non-fault / confidence criteria that avoid explicit τ, is needed for the cost–benefit framing to be operational.","section":null},{"comment":"All results are on image classification with offline ranked pools and sequential single-label queries (Tables II–X). The abstract and conclusion claim a general DNN testing framework. §VI-J acknowledges NLP/tabular/batch settings as future work; those limitations should be reflected more carefully in the abstract and contributions so that the reported 65–84% / 9–31% numbers are not over-generalized beyond the evaluated regime.","section":null}],"minor_comments":[{"comment":"Abstract and §I state 9–31% budget; Table III and several RQ summaries use 23.5% as the headline CIFAR-10 figure, while cross-dataset averages (Table VII) are lower. Align the range language with the tables.","section":null},{"comment":"Figure 1 and Algorithm 1 are clear; Figures 2–3 would benefit from error bars or multiple seeds where randomness appears (e.g., Random strategy in Table IX).","section":null},{"comment":"Related work (Table I, §III) is adequate but could briefly note how AdaStop differs from classical sequential testing / SPRT-style rules beyond SAFE and AL stopping.","section":null},{"comment":"Notation: r_t is defined as an indicator but sometimes treated as a rate in prose; keep the distinction between r_t and p(t)/ˆp(t) consistent.","section":null},{"comment":"Default parameters (τ=0.05, W=20, N_min=50) are stated in §V-B; a short justification or pointer to the ablation tables in the main text would help readers who skip the RQs.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems/empirical contribution with a clean but elementary optimality argument. Fit is better for an SE/testing or applied ML venue than for a theory-heavy ML journal. No integrity red flags; the monotonicity caveat is already self-flagged by the authors. Major revision is appropriate so that the optimality claim is scoped to what is actually proven and the operational use of τ is clarified."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean engineering paper that takes a standard optimal-stopping idea (stop when marginal fault rate falls below τ = c/v) and packages it for DNN test selection. That is the real contribution: not a new theory of stopping, but a usable rule plus a broad empirical map of where it lands on vision models.\n\nWhat is new is the concrete operating point. Under DeepGini-style ranking they get 65–84% of faults with roughly 9–31% of the labeling budget across CIFAR-10, SVHN, FashionMNIST, four architectures, three quality levels, and eight selection strategies. Proposition 1 is just the marginal-value rearrangement under non-increasing p(t); it is correct and short. The sliding-window estimator is ordinary, the ablations on τ, W, and alternative criteria (patience, consecutive non-faults, Wilson CI) are useful, and the Mann-Kendall checks in Table XI actually support the diminishing-returns premise inside the evaluated setting. Net-value comparisons against fixed-budget DeepGini and exhaustive labeling are honest: more testing is not always better under their v=20, c=1 accounting.\n\nSoft spots are real but already flagged by the authors. The load-bearing assumption is monotonicity of the fault rate; they only validate it on three vision datasets and note it may break for multi-modal or adversarial failure clusters. Free parameters (τ, W, N_min) move the operating point, though they show stepped rather than fragile behavior for moderate ranges. Scope is image classification only; no NLP/tabular, no batch labeling, and the manuscript does not ship a public code/data link in the text. Citation pattern is fine—DeepGini, TestRank, ATS, SAFE, active-learning stopping are all present and distinguished.\n\nWho this is for: people who already run uncertainty-based test selection on vision models and need a principled budget cutoff instead of a hand-picked k%. It is not a foundational result. I would send it to peer review; the experiments are broad enough and the claim is modest enough that a referee can tighten the generality discussion and demand the artifact. Worth a look if you work in this subfield; skip if you need theory or non-vision evidence.","headline":"Solid engineering packaging of a standard cost-benefit stop for DNN test selection; useful numbers, elementary math, vision-only scope.","tokens_in":11077,"tokens_out":554,"would_cite":false,"duration_ms":6066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"DNN test labeling can stop when the estimated fault rate falls below the cost-to-value ratio, finding most faults with a fraction of the budget.","keywords":["DNN testing","test selection","early stopping","cost-benefit","fault discovery","labeling budget","diminishing returns","DeepGini"],"falsifier":"On a pool whose faults cluster outside the early uncertainty ranking, if the windowed fault rate repeatedly dips below τ then rises well above it, early stopping would miss a large share of faults that a larger fixed budget still finds—undermining both optimality and the reported savings.","tokens_in":11117,"feed_emoji":"🛑","tokens_out":872,"duration_ms":19805,"temperature":0.7,"pith_summary":"Choosing how many test inputs to label for a deep neural network is hard: too few misses faults, too many wastes expert labeling. This paper treats testing as a sequential cost–benefit process in which each label costs c and each discovered fault is worth v, and derives a simple optimal rule: stop once the marginal chance of finding a fault drops to or below τ = c/v. AdaStop estimates that chance with a sliding window over recent outcomes and halts labeling when the estimate crosses the threshold. Across several image datasets, model architectures, quality levels, and selection strategies, the method recovers 65–84% of faults while using only about 9–31% of the full labeling budget, often improving net value over exhaustive testing. A reader cares because fixed budgets force an arbitrary guess; a cost-aware stop replaces that guess with a break-even rate that practitioners can set from their own costs and values.","feed_headline":"Find most DNN faults on a fraction of the label budget","feed_subtitle":"Stop labeling when the fault rate falls below the cost-to-value break-even point.","key_machinery":"The break-even threshold τ = c/v together with a sliding-window estimator of the marginal fault rate. It converts the continue/stop decision into a direct cost–benefit comparison: keep labeling while the estimated fault probability still exceeds the cost-to-value ratio; stop when it does not.","core_discovery":"Under diminishing returns from uncertainty-based selection, the optimal policy stops at the first T where the fault discovery rate p(T) ≤ τ = c/v. AdaStop implements this with a sliding-window estimator of p(t) and shows that 65–84% of faults can be found using only 9–31% of the labeling budget across the evaluated datasets, architectures, and strategies, with higher net value than labeling everything.","pith_inferences":["The same break-even-rate idea could govern any costly oracle under prioritization—bug triage, document screening, or safety audits—not only DNN test selection.","Batch labeling would need a natural extension: decide after each batch whether the windowed rate still justifies the next batch’s cost.","If multi-modal or adversarial failures break monotonicity, a mixture-aware or resettable window would be required before the optimality claim transfers."],"forward_implications":["Practitioners can set τ from real labeling cost and fault value instead of guessing a budget percentage.","Uncertainty-based selectors produce similar diminishing-return curves, so the same stop rule yields consistent budget–recall points across strategies.","Worse models automatically receive more budget and better models less, without retuning τ.","Alternative rules (patience, consecutive non-faults, confidence bounds) form a spectrum from highest efficiency to near-complete recall.","Exhaustive labeling can produce lower net value than early stopping under the same c and v."],"fun_headline_variants":["AdaStop finds 65-84% DNN faults with 9-31% of label budget","Halt DNN labeling when fault rate drops below cost-to-value threshold","Discover most DNN faults after labeling only 9-31% of candidates","Cost-aware early stopping captures 65-84% of faults at fraction of cost","Stop DNN tests once estimated fault rate falls under τ=c/v"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes that once tests are ordered by uncertainty, the chance of finding a new fault keeps falling so that once it drops below the break-even rate it stays there.","fun_headline_variants_meta":{"raw":{"variants":["AdaStop finds 65-84% DNN faults with 9-31% of label budget","Halt DNN labeling when fault rate drops below cost-to-value threshold","Discover most DNN faults after labeling only 9-31% of candidates","Cost-aware early stopping captures 65-84% of faults at fraction of cost","Stop DNN tests once estimated fault rate falls under τ=c/v"]},"model":"grok-4.5","effort":"low","cost_usd":0.007242,"raw_usage":{"total_tokens":1707,"prompt_tokens":702,"num_sources_used":0,"completion_tokens":87,"cost_in_usd_ticks":72420000,"prompt_tokens_details":{"text_tokens":702,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":918,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":702,"tokens_out":87,"duration_ms":7501,"temperature":1.0,"reasoning_tokens":918,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T16:55:48.639849+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a pool whose faults cluster outside the early uncertainty ranking, if the windowed fault rate repeatedly dips below τ then rises well above it, early stopping would miss a large share of faults that a larger fixed budget still finds—undermining both optimality and the reported savings.","supporting_citations":[],"review_version":1}