{"id":"ca339cfc-c65e-431c-b751-cf29c01c935e","arxiv_id":"2501.14036","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A learn-then-test calibration procedure selects detection thresholds with a precision guarantee, and two extra filters recover recall in ovarian follicle counting.","lead":"An object-detection pipeline for counting ovarian follicles is retrofitted with a multiple-testing threshold selector that promises a high-probability precision guarantee. The paper adds a biological-depth filter and a learned error-filter to keep recall high, but key implementation details are deferred to a missing supplement.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The p-value in Eq. (1) is inverted for the stated null: with the sample precision as the test statistic, high precision yields p_j approximately 1, so the LTT selection cannot produce the certified-precision guarantee claimed in Eq. (2).","rationale":"The reader's verdict is REJECT, and this pass agrees that the current manuscript does not support the central claim. The reader's weakest_assumption focuses on the undefined per-image precision loss, i.e., patch aggregation, matching rule, and zero-detection handling; that is a genuine reproducibility gap and would alone block verification. The concern developed here is more fundamental: even granting a well-defined per-image precision, the p-value formula in Eq. (1) appears directionally inverted for the stated null hypothesis. The paper states the test statistic is mean precision and the null is P ≤ P0; under that reading, high precision makes the first term exp(-n h1(P0,P0)) = 1 and the binomial term near 1, so the p-value is large when the data contradict the null. The Hoeffding-Bentkus expression is the standard one for controlling a loss that is small when performance is good, so the fix is likely to use risk = 1 - precision and α = 1 - P0, but that redefinition is absent from the text. Because the selection rule and Eq. (2) depend entirely on the direction of the test, this is the most load-bearing technical flaw. The contribution is not accused of bad faith: the method may well be valid in the authors' implementation, and the missing supplement or public code could resolve both this ambiguity and the undefined loss. For the paper as written, however, the probabilistic guarantee cannot be derived from the presented equations. The recommendation is therefore unchanged from the reader's REJECT, with the caveat that a revision that corrects the statistic direction and specifies the per-cut precision computation could be re-evaluated. This pass gives partial agreement with the reader because the reader correctly flagged the direction issue in the rationale but chose the undefined loss as the weakest assumption; the p-value inversion is judged here to be the more load-bearing of the two.","tokens_in":7813,"tokens_out":9033,"duration_ms":84400,"concrete_test":"Run a synthetic calibration with n=35, target P0=0.4, and per-image precision values that are iid with mean 0.9. Compute p_j from Eq. (1) as printed, taking R̂_j = sample mean precision and α = P0; if p_j ≈ 1, the test is inverted for H0: P ≤ P0. Then repeat the 100-split calibration procedure of §3.3 with the corrected statistic R̂_j = 1 - sample mean precision and α = 1 - P0, and measure how often the selected threshold achieves test-set precision ≥ P0 on held-out images. If only the corrected version reaches the claimed 1-δ coverage, the paper's central guarantee depends on an unstated change of test statistic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on a valid p-value for each null H_{1,λ_j}^0: P^{(1)}(λ_j) ≤ P0, with rejection used to conclude that precision exceeds P0. In §2.2 the test statistic is defined as (1/n)Σ_i Precision(b_i, T_λ^{(1)}(b̂_i)), and Eq. (1) gives pHB_j = min{exp(-n h1(R̂_j ∧ α, α)), eP(Bin(n,α) ≤ ⌈n R̂_j⌉)}. On the natural reading R̂_j = mean precision and α = P0, a high observed precision gives R̂_j ∧ α = P0, so h1(P0,P0)=0 and P(Bin(n,P0) ≤ n R̂_j) ≈ 1; hence p_j ≈ 1 exactly when the data most support rejecting the null. The formula is valid for a loss that is small when the model is good, e.g., risk = 1 - precision with α = 1 - P0, as in Bates et al., but the paper neither defines R̂_j nor α, and the explicitly named test statistic is precision. If the p-values were actually computed as written, the rejected set Λ̂ would be wrong and the guarantee of Eq. (2) would not follow. This is a mathematical direction error, not a matter of consensus; it can be checked by derivation alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a post-processing threshold selection procedure for object detection in whole-slide ovarian follicle images. The authors apply the Learn Then Test (LTT) framework to select an objectness threshold that, with probability at least 1−δ over the calibration set, guarantees an expected precision of at least P0 on new images, and they extend this to two-parameter decision rules that also use follicle depth or an auxiliary classifier score to improve recall and F1. Experiments on a newly released dataset compare the methods with EfficientDet and YOLOv8 across 100 calibration/test splits.","tokens_in":8109,"tokens_out":6365,"duration_ms":55325,"significance":"If correct, the procedure would be a useful model-agnostic way to turn an uncalibrated detector into one with a distribution-free precision guarantee without retraining, and the release of code and data is a valuable contribution. However, the central statistical derivation in §2.2 has a direction error: the reported Hoeffding-Bentkus p-value is applied to a precision (larger is better) instead of a risk (smaller is better), so the p-values and the guarantee in Eq. (2) are not valid as written. The experimental results therefore do not currently support the main claim. The paper is an application of existing LTT theory rather than a new theoretical contribution, and its significance depends on correcting the p-value formulation and clarifying the evaluation protocol.","major_comments":[{"comment":"The text states that the test statistic is (1/n)Σ_i Precision(b_i, T_λ^{(1)}(f-hat(b_i))) but never defines R̂_j or α. If, as the notation suggests, R̂_j is the sample mean precision and α = P0, then for any λ_j with observed precision above P0, R̂_j∧α = P0, so h1(R̂_j∧α, α) = 0 and P(Bin(n, α) ≤ ⌈n R̂_j⌉) ≈ 1, yielding p_j ≈ 1. Hence the p-value is largest precisely when the data most strongly support rejecting H_{1,λ_j}^0: P(1)(λ_j) ≤ P0, and the LTT selection would return the wrong compatible set. The Hoeffding-Bentkus inequality is valid for a risk that is small in the direction of rejection; the risk should be 1 − Precision with α = 1 − P0. The manuscript must specify R̂_j and α, state the risk formulation, and re-derive the guarantee in Eq. (2). As written, Eq. (2) does not follow from the stated procedure.","section":"§2.2, Eq. (1)"},{"comment":"The paper does not define how per-patch predictions are aggregated into the cut-level precision used in the p-values. The calibration set is described as 35 ovary cuts, and the OD model is run on 1000×1000 patches with stride 500, but the mapping from overlapping patch detections to the set of boxes for a cut is never specified, nor is the matching rule between predicted and ground-truth follicles (e.g., IoU threshold), nor the convention when no boxes survive T_λ. Without a well-defined measurable loss Precision(b_i, T_λ(f-hat(b_i))) on the calibration data, the computed p-values and the guarantee in Eq. (2) are not reproducible.","section":"§3.1, §2.1"},{"comment":"The grids ΛO, Λ′, the number of thresholds m, and m′ are not fully specified. The paper reports selected thresholds such as (0.642, 0.515) for depth and (0.534, 0.230) for classification, but the reader cannot determine the multiple-testing correction applied or the search space. This matters because FST's conservativeness depends on the ordered grid and the number of starting points m′, and the guarantee is only over the tested set Λ(2). Please provide the grid definitions and the FST details in the main text or a complete supplementary.","section":"§2.3, Table 1"}],"minor_comments":[{"comment":"The text contains several instances where 'LLT' is written instead of 'LTT' (e.g., §2.2, 'selected by the LLT procedure'); these typos should be corrected.","section":"Throughout"},{"comment":"The manuscript has unresolved '??' placeholders for the depth computation, detailed performance tables, and training parameters in the supplementary material; these must be filled in before the claims can be assessed.","section":"§2.3, §3.2"},{"comment":"The text first says the approach controls the False Discovery Rate, but then defines and uses family-wise error rate control via Fixed Sequence Testing; the terminology should be harmonized to avoid confusion.","section":"§2.2"},{"comment":"The notation 'Precision(b, ˆb)' is used before the arguments are defined; the matching rule (e.g., IoU threshold) between predicted and ground-truth boxes should be specified in the main text.","section":"§2.1"},{"comment":"The phrase 'overperforming way' is awkward and should be rephrased for clarity.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The statistical direction error in Eq. (1) is serious but curable: the authors should replace the precision statistic with a 1 − precision risk and α = 1 − P0, and then confirm that the implemented p-values match this correction. If the experiments were already run with the correctly oriented risk, the numerical results may stand, but the text and the guarantee in Eq. (2) must be corrected. Because the dataset and code release are valuable, I would be willing to review a revision that fixes the p-value formulation and supplies the missing experimental definitions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nI read the Blot et al. manuscript on precision control for ovarian follicle counting. The short version: the dataset and the two post-processing filters are real contributions, but the central statistical guarantee in Eq. (2) does not follow from the equations as written. The p-value formula in Eq. (1) is for a loss that is small when the model is good; the paper states the test statistic is the mean precision, which is large when the model is good. With a high observed precision, R_hat_j ∧ α collides with α, h1 = 0, and the Binomial term is near 1, so p≈1 exactly when the data most support rejecting the null. Either the risk is 1 - precision with α = 1 - P0, or the test is reversed; the text does not make that switch. This is not a subtle issue; it is the load-bearing step.\n\nThe paper also leaves the per-image loss undefined. Calibration is done at the cut level, but predictions come from overlapping 1000x1000 patches with stride 500. There is no description of how boxes are aggregated into a cut-level set, what IoU threshold defines a match with ground truth, or what happens when no boxes survive. Without that, Eq. (2) cannot be verified or reproduced even if the direction error were fixed.\n\nWhat is genuinely useful: the open dataset of 643 mouse ovary sections with expert annotations, the two auxiliary decision filters (biological depth and learned false-detection classifier), and the clean empirical demonstration that these filters improve F1 over thresholding alone at controlled precision targets. The code and data are public. If the statistical derivation is corrected, the method would be a practical, model-agnostic post-processing recipe for any detection task.\n\nI would not accept the current version. The direction error and the missing definitions are fixable in revision, but as written the central claim of a certified precision guarantee is unsupported. The paper deserves a serious referee, though, because the dataset and the empirical results have value. I would send it back with a request to fix the LTT application, define the cut-level precision loss, and point to the missing supplement.\n\nRegards.","headline":"Valuable dataset and a plausible method, but the precision guarantee is invalid as written because the p-value formula is inverted relative to the stated test statistic.","tokens_in":8647,"tokens_out":3489,"would_cite":false,"duration_ms":30386,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model-agnostic threshold-selection procedure certifies precision in object detection and uses a second criterion to recover recall, demonstrated on ovarian follicle counts.","keywords":["Distribution-free risk control","Multiple testing","Learn Then Test","Object detection","Precision guarantee","Ovarian follicle counting","Whole-slide imaging","Model-agnostic post-processing"],"falsifier":"Re-run the LTT calibration on the released dataset with a fixed, explicit patch-aggregation and matching rule, then count across many resplits how often average precision on the test cuts falls below $P_0 = 0.4$; if the fraction below target exceeds $\\delta = 10^{-3}$ for any reasonable aggregation choice, the claim is not robust to the paper's unspecified preprocessing.","tokens_in":7623,"feed_emoji":"🔬","tokens_out":8396,"duration_ms":72578,"temperature":0.7,"pith_summary":"This paper claims that the uncertainty in choosing a decision threshold for object detection can be controlled so that a user-chosen precision level is guaranteed with high probability on future images. It applies the Learn Then Test (LTT) multiple-testing procedure to select an objectness threshold, then replaces the single threshold with a two-parameter rule that also uses either biological depth information or a learned false-detection classifier. On ovarian follicle counts from whole-slide histology images, the two-parameter rule keeps the precision guarantee while recovering much of the recall lost by conservative single-threshold selection, improving F1. Because the procedure is model-agnostic, the same post-processing step could improve reproducibility and trust in any object detector without retraining it.","feed_headline":"Two-criterion rule certifies precision in AI follicle counting","feed_subtitle":"A statistical threshold rule locks AI follicle-count precision above a set level, while a second cue restores recall.","key_machinery":"The engine is the Learn Then Test (LTT) construction: for each candidate threshold (or pair), define the null hypothesis that expected precision is at most $P_0$, compute p-values from the per-image precision losses using the Hoeffding-Bentkus inequality, and keep only thresholds whose hypotheses survive a family-wise-error-rate-controlling procedure (fixed-sequence testing). The second ingredient is the two-parameter post-processing operator that adds a second criterion: ovary depth, computed by dilating the ovary contour, or the score of a small auxiliary convolutional classifier trained to distinguish true from false detections. This enlarges the search space of statistically compatible thresholds, which is what lets recall increase without losing the precision guarantee.","core_discovery":"The central claim is that a post-processing decision operator $T_{\\lambda,\\mu}(b) = \\{b_i : c_i \\geq \\lambda \\text{ and } d_i \\leq \\mu\\}$ (or with a classifier score $g(b_i) \\geq \\mu$ instead of depth) selected through LTT satisfies $P_D(E_{P_new}[\\mathrm{Precision}(b_{\\mathrm{new}}, T_{\\lambda^*,\\mu^*}(\\hat{f}(b_{\\mathrm{new}})))] \\geq P_0) \\geq 1-\\delta$, exactly as the single-threshold operator does, while yielding higher recall and F1 than both naive thresholding and LTT on objectness alone. The guarantee follows from treating each candidate threshold pair as a null hypothesis and controlling the family-wise error rate with fixed-sequence testing; the recall gain comes from searching a larger two-dimensional grid of compatible thresholds. The paper supports this with experiments on 643 mouse ovary sections, where the two-parameter rules meet the precision target $P_0 = 0.4$ with probability at least $1-10^{-3}$ across 100 calibration/test splits, whereas the naive threshold falls below target roughly half the time.","pith_inferences":["An immediate testable extension is to check sensitivity of the selected thresholds to the choice of patch aggregation (e.g., non-maximum suppression across overlapping $1000 \\times 1000$ patches) and to the IoU matching rule, since these choices determine the exact precision values that enter the LTT p-values.","Because the auxiliary classifier is trained on the training set and then used to filter boxes at calibration and test time, its own generalization error is not covered by the LTT guarantee; measuring how much the guarantee degrades with classifier accuracy would be a direct extension.","The depth-based rule relies on ovary contour quality; a contour-free context cue (such as a distance transform from tissue segmentation) might provide the same recall gain while being less sensitive to contour errors, which the paper does not test."],"forward_implications":["If the guarantee holds, a lab can fix $P_0$ in advance and trust that, over calibration randomness, the expected precision on a new slide stays at least $P_0$, making repeated follicle counts comparable across studies.","The two-parameter selection rule needs no retraining of the detector; any existing detector can be wrapped with this post-processing step and an auxiliary cue, so performance gains transfer across architectures such as EfficientDet and YOLO.","The same LTT machinery extends directly to more than two hyperparameters: any finite grid of post-processing choices can be tested similarly, so richer contextual rules (e.g., size, shape, stain intensity) could be added without changing the framework.","Using a learned false-detection classifier as the second criterion shows that purely data-driven cues can replace hand-crafted biological ones while preserving the precision guarantee, which makes the method portable to detection problems outside histology."],"supporting_citations":[{"why":"Supplies the Learn Then Test framework: the multiple-testing formulation, fixed-sequence testing, and the theorem behind the precision guarantee in Eq. (2).","marker":"[4]"},{"why":"Provides the distribution-free Hoeffding-Bentkus inequality used to compute the p-values in Eq. (1).","marker":"[6]"},{"why":"Shows a prior application of conformal risk control to object detection, the comparison point for applying risk guarantees to detectors.","marker":"[2]"},{"why":"Supplies the general conformal risk control framework from which the LTT-style guarantee is adapted.","marker":"[5]"},{"why":"Builds the earlier deep-learning follicle-counting pipeline that this work extends with precision control.","marker":"[20]"},{"why":"Architecture of one of the two detectors whose thresholds are calibrated by the procedure.","marker":"[21]"},{"why":"Architecture of the other detector used in the experiments.","marker":"[12]"},{"why":"Architecture of the auxiliary classifier that provides the second decision criterion in the classifier-based operator.","marker":"[13]"}],"fun_headline_variants":["Two-threshold rule locks precision and lifts recall in follicle AI","Statistical dual-criterion control certifies follicle-count precision","Precision-guaranteed AI follicle counting with a dual-threshold rule","Dual-threshold post-processing boosts follicle counting accuracy and recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee stands on treating $\\mathrm{Precision}(b_i, T_\\lambda(\\hat{f}(x_i)))$ as a well-defined, measurable function for each cut image, but the paper never specifies how predictions from overlapping patches are merged into cut-level boxes, what IoU matching rule links predicted to ground-truth follicles, or what precision is when no box survives thresholding; if any of these undefined choices changes the loss values, Eq. (2) cannot be verified or reproduced as stated.","fun_headline_variants_meta":{"raw":{"variants":["Two-threshold rule locks precision and lifts recall in follicle AI","Statistical dual-criterion control certifies follicle-count precision","Precision-guaranteed AI follicle counting with a dual-threshold rule","Dual-threshold post-processing boosts follicle counting accuracy and recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1458,"prompt_tokens":958,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":574,"tokens_out":500,"duration_ms":5159,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:27:38.782003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the LTT calibration on the released dataset with a fixed, explicit patch-aggregation and matching rule, then count across many resplits how often average precision on the test cuts falls below $P_0 = 0.4$; if the fraction below target exceeds $\\delta = 10^{-3}$ for any reasonable aggregation choice, the claim is not robust to the paper's unspecified preprocessing.","supporting_citations":[{"cited_title":"Journal of ACM 68(6) (2021)","cited_arxiv_id":null,"evidence_quote":"Provides the distribution-free Hoeffding-Bentkus inequality used to compute the p-values in Eq. (1)."},{"cited_title":"Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling","cited_arxiv_id":"2304.06052","evidence_quote":"Shows a prior application of conformal risk control to object detection, the comparison point for applying risk guarantees to detectors."},{"cited_title":"Scientific reports 8(1), 13499 (2018)","cited_arxiv_id":null,"evidence_quote":"Builds the earlier deep-learning follicle-counting pipeline that this work extends with precision control."},{"cited_title":"In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)","cited_arxiv_id":null,"evidence_quote":"Architecture of one of the two detectors whose thresholds are calibrated by the procedure."},{"cited_title":"com/ultralytics/ultralytics","cited_arxiv_id":null,"evidence_quote":"Architecture of the other detector used in the experiments."},{"cited_title":"In: 3rd IAPR Asian Conference on Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Architecture of the auxiliary classifier that provides the second decision criterion in the classifier-based operator."}],"review_version":1}