{"id":"a01f7b3c-a717-4578-af26-e7cb9c6274d4","arxiv_id":"2502.08582","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-sided empirical quantile test on a model's score lets a binary classifier flag ambiguous or out-of-distribution inputs as uncertain instead of labeling them.","lead":"The paper proposes a classifier that can abstain by comparing each input's score to per-class quantile intervals from training data. The goal is safer decisions in high-stakes tasks like pneumonia screening, where saying 'I don't know' is better than guessing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD-detection claim rests on an unstated identifiability assumption: that OOD inputs must map to feature values outside the per-class training quantile intervals.","rationale":"The reader's weakest assumption correctly identifies Eq. (4) as the point where the training empirical distributions are used as null distributions. I agree that the OOD claim is the most load-bearing part: it is the only component that goes beyond abstaining on ambiguous in-distribution points, and it is unsupported by any experiment. The X-ray experiment only shows a coverage/accuracy tradeoff on in-distribution test data; the spiral experiment is partly tautological because all data are used as training and the same points are visualized. A concrete OOD benchmark directly tests whether the assumed mapping from OOD inputs to out-of-quantile logits holds. Without such evidence, the central claim should remain conditional, but not rejected outright, because the method may still be a useful abstention rule for overlapping in-distribution classes even if OOD detection fails.","tokens_in":7970,"tokens_out":3361,"duration_ms":39453,"concrete_test":"Use the same DenseNet training pipeline (Section 4.4) on an in-distribution dataset such as CIFAR-10 with two selected classes, then present held-out OOD images from SVHN, Fashion-MNIST, and Gaussian noise. For α ∈ {0.01, 0.025, 0.05}, compute the fraction of OOD samples whose logit falls outside both per-class [α, 1−α] quantile intervals (the proposed OOD detection rate) and compare against random guessing and a simple softmax-confidence baseline. Also report AUROC for OOD detection. If detection rate is near α or AUROC is near 0.5, the OOD claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2.1 defines type II uncertainty (OOD) as the condition t ∈ I1 ∩ I2, i.e., the feature value lies outside both per-class acceptance intervals. Section 2.3 then approximates the null distributions F1, F2 by the empirical training feature distributions (Eq. 4). The central claim—that the method detects out-of-distribution data—therefore requires that OOD inputs yield feature values outside the central quantile intervals of both classes. This is not established anywhere in the paper. It is not a consequence of the hypothesis-testing formalism: the scalar logit is not a sufficient statistic for 'in-distribution vs OOD,' and neural networks are known to assign high-confidence (in-range) logits to many OOD inputs. The only experiments are the spiral benchmark, which uses all data as training and evaluates on the same grid (so type II regions reflect the training envelope, not held-out OOD), and the X-ray task, which has no OOD test set. No baseline, error bar, or code supports the OOD claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc abstention rule for binary classification. For a model that outputs a scalar feature g(x) (an SVM decision value or a logit), the method computes the empirical distribution of g over each class in the training set and defines a central acceptance interval for each class from its quantiles at the chosen significance level α. A new input is classified as class 1 or class 2 if its test statistic falls inside exactly one acceptance interval; it is declared uncertain if it falls inside both intervals (treated as class-overlap ambiguity) or outside both intervals (treated as out-of-distribution). The method is illustrated on a two-spiral SVM benchmark and on a DenseNet-121 pneumonia chest X-ray classification task, with coverage and accuracy reported for several significance levels. The authors emphasize that the method requires no resampling, no model modification, and no threshold optimization.","tokens_in":8215,"tokens_out":6834,"duration_ms":74726,"significance":"If the method's claims were fully supported, it would be an attractive addition to the selective-classification toolbox: it is simple, transparent, requires only training-set feature quantiles, and its threshold is interpretable as a significance level. The visual histogram-based diagnostics in Figures 2 and 5 are genuinely useful, and applying a weighted loss to handle class imbalance in the X-ray experiment is a sensible methodological choice. However, the central out-of-distribution-detection claim currently rests on no held-out OOD evaluation, and the only independent experiment does not compare against existing abstention methods. As a result, the significance of the paper is conditional on substantial additional validation rather than established by the current evidence.","major_comments":[{"comment":"The central claim that the method \"is capable of detecting ... out-of-distribution data\" is not tested anywhere in the manuscript. The chest X-ray experiment uses a test set containing only 390 pneumonia and 234 normal images, which are drawn from the same two classes as the training data. Table 1 reports coverage, accuracy, recall, precision, specificity, and F1-score on those in-distribution test images, but no out-of-distribution image set is used and no metric for OOD detection (for example, AUROC for the \"outside both intervals\" decision) is reported. An OOD test set or a synthetic distribution shift is required before this claim can be evaluated.","section":"Abstract; §4.4–4.5"},{"comment":"The spiral experiment does not provide independent evidence for OOD detection because all data in Figure 1 are used as training data and the visualization is performed on the same input grid. Type II uncertainty is defined, by construction, as the test statistic lying outside both per-class central intervals, i.e., in the intersection of the rejection regions I1 and I2; therefore the observation that the squares in Figures 3(b)–3(d) appear outside the training envelope is a restatement of the decision rule rather than a detection result. A held-out region of the input space, a separately generated spiral, or a separate OOD dataset would be needed to test the claim.","section":"§3, Figs. 2–3"},{"comment":"The method relies on an unstated identifiability assumption: that OOD inputs map to feature values t outside the central intervals of both per-class empirical training distributions. The null distribution Fi is approximated by the empirical distribution of training feature values, but this only describes where in-distribution training features lie; it does not imply anything about where OOD features will land. Because the scalar logit g(x) is not a sufficient statistic for the in-distribution versus OOD distinction, there is no guarantee that OOD inputs will fall outside the training quantile intervals. The authors should either state and empirically justify this assumption with an explicit OOD experiment, or remove the OOD-detection claim from the abstract and title.","section":"§2.1–2.3, Eq. (4)"},{"comment":"The experimental results are reported without baselines. A confidence-thresholded softmax classifier, an MC-dropout variance threshold, or a conformal-prediction abstention rule would provide a minimal comparison for the coverage–accuracy trade-off, and all reported metrics are single-run values with no error bars or seed information. As it stands, Table 1 shows that the method abstains on some fraction of the test set, but it does not show that this abstention is more effective than simply thresholding the model's own confidence.","section":"§4.4, Table 1"}],"minor_comments":[{"comment":"There are several typos that should be corrected: \"modefication\" in the Introduction, \"Applicatioin\" in the Section 4 heading, \"croses\" in the Figure 1 caption, and \"the the acceptance regions\" in the description of experiment (iii) in Section 3.","section":"Throughout"},{"comment":"The statement in the Discussion that \"coverage does not necessarily decrease as the significance level increases\" is confusing because increasing α shrinks both central intervals, which affects the two uncertainty types in opposite directions; the authors should explain which effect dominates in the X-ray data.","section":"§4.5"},{"comment":"Equation (4) writes the empirical distribution using a Dirac delta, but the text later refers to \"the interval between the 2.5% and 97.5% points\"; it would be clearer to define the empirical CDF as a step function and then define the quantile interval explicitly.","section":"§2.3"},{"comment":"There are duplicate entries for Arulananth et al. (2024), and the Hinton and van Camp reference is missing page numbers; these should be cleaned up.","section":"References"},{"comment":"No information is given about the number of training runs or random seeds, and no code or data availability statement is provided, which limits reproducibility.","section":"§4.2/4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper reads more like a technical report than a full research paper for a cs.LG venue. The absence of any OOD test set, baseline comparisons, and reproducibility details is the main gap. I would encourage the editor to require the additional OOD experiments and baseline comparisons in a revision rather than accepting the current empirical claims at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper proposes a simple abstention rule for binary classification. For a scalar score (SVM decision value or pre-sigmoid logit), it builds per-class empirical distributions from training data, takes the central alpha-(1-alpha) interval per class, and abstains when the score falls inside both intervals (class overlap) or outside both intervals (claimed OOD). The rule is clearly described, trivially cheap to implement, and needs no resampling or model surgery. That is the genuine novelty: the specific combination of two per-class quantile tests with a 'both or neither' abstention.\n\nWhat the paper does well: the spiral example gives an intuitive visual confirmation that boundary points end up in the 'inside both' region. The chest X-ray experiment demonstrates a coverage/accuracy tradeoff dialed by alpha, and the weighted loss for class imbalance is a reasonable touch. The writing is honest in scope; it does not claim a formal guarantee.\n\nThe soft spots are exactly where the reader's report points. The OOD detection claim is unsupported. Type II uncertainty is defined as falling outside both per-class intervals, but this only flags OOD data if the model's scalar score maps OOD inputs to extreme values. That is not generally true; neural nets often give in-range logits to OOD inputs. The spiral experiment uses all data as training and evaluates on the same grid, so 'outside both' just reflects the training envelope; no held-out OOD points are tested. The X-ray dataset has no OOD test set. There are no baselines (e.g., softmax threshold, MC-dropout, conformal prediction) and no error bars or repeated runs. The comparison to standard selective classifiers is missing, so the reader cannot tell whether this rule is better than a simple confidence threshold.\n\nThe circularity concern is real but proportionate: the 'ambiguous' detection is partly tautological because points in the overlap are defined as those inside both central intervals, so it is not surprising that boundary points are flagged. The coverage/accuracy tradeoff in Table 1 is the expected consequence of widening acceptance intervals; it is not evidence of a new phenomenon.\n\nIf the authors revise, they should add a proper OOD benchmark (e.g., SVHN vs CIFAR or a rotated/shifted version of the X-ray set), compare against a threshold on the softmax response or a conformal baseline, and report variance over runs. The central ambiguity-detection idea is likely sound, but the OOD claim needs empirical support.\n\nMy recommendation: this deserves a serious referee—the rule is simple enough to be assessed rigorously, and the missing experiments are standard and fixable. It should not be desk-rejected, but it needs major revision before acceptance. For a reading group, it is a decent short example of why per-class quantiles are not a silver bullet for OOD, though I would not cite it in my own work.","headline":"A simple, clearly described abstention rule for binary classification that plausibly flags class overlap but makes an unsupported claim about OOD detection.","tokens_in":8671,"tokens_out":2869,"would_cite":false,"duration_ms":30824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two hypothesis tests let a binary classifier abstain on ambiguous and out-of-distribution inputs using only training-data thresholds.","keywords":["hypothesis testing","uncertainty quantification","binary classification","reject option","out-of-distribution detection","empirical distribution","quantile threshold","pneumonia chest X-ray"],"falsifier":"Run the method on a genuine out-of-distribution dataset whose scores mostly fall inside the training score range, for example images from a new domain that the network still scores near typical training scores; whenever the method confidently assigns those inputs to one of the two classes instead of flagging them uncertain, the claim that it detects out-of-distribution data is contradicted.","tokens_in":7778,"feed_emoji":"🤖","tokens_out":9908,"duration_ms":97971,"temperature":0.7,"pith_summary":"This paper tries to establish that a binary classifier can flag both kinds of dangerous inputs—data that falls in the overlap between the two classes and data that lies outside the training distribution—using only two hypothesis tests on the classifier's own score. For each class, the distribution of that score is estimated from the training data, and a central acceptance interval is cut at the $\\alpha$-quantile and $(1-\\alpha)$-quantile. A new input whose score falls inside both intervals is treated as ambiguous; a score outside both intervals is treated as out-of-distribution; a score inside exactly one interval is assigned to that class. The significance level $\\alpha$ is set by the user, so no threshold optimization, resampling, or model modification is required. If the method holds up, high-risk applications such as clinical image diagnosis gain an explicit \"I don't know\" option with a tunable trade-off between coverage and accuracy.","feed_headline":"Two quantile tests flag ambiguous and out-of-distribution data","feed_subtitle":"Thresholds come from training-data scores and a user-set significance level; no resampling or model changes needed.","key_machinery":"The machine is a pair of simultaneous hypothesis tests on the scalar score $g(x)$—the SVM discriminant value or the logit feeding a sigmoid—whose test statistic is the score itself. Each test's acceptance region is the central interval from the $\\alpha$-quantile to the $(1-\\alpha)$-quantile of that class's empirical training-score histogram. The whole argument runs on the position of the score relative to the two intervals: inside both means \"both classes plausible\", outside both means \"neither class plausible\", and these two zones are exactly the method's two kinds of uncertainty.","core_discovery":"The central claim is that the uncertainty of a binary prediction can be read directly from the position of one scalar feature, $g(x)$, relative to two per-class empirical null distributions. For class $i$, the null hypothesis $H^i_0$ says $g(x)$ follows $F_i$, the class-$i$ distribution, and the acceptance region is the central interval between the empirical $\\alpha$-quantile and $(1-\\alpha)$-quantile of the training scores for that class. The decision rule is: accept both hypotheses, and the point is in the ambiguous overlap; reject both, and the point lies outside both training distributions and is reported as out-of-distribution; accept exactly one, and the point is classified as that class. The experiments reproduce the expected trade-off: a larger $\\alpha$ narrows the acceptance intervals and, on the pneumonia dataset, raises coverage while lowering accuracy on the inputs that receive labels.","pith_inferences":["The same recipe should transfer to any scalar confidence score, including softmax probability, distance to a decision boundary, or reconstruction error, so the abstention rule could be grafted onto models that were never designed for uncertainty.","Because the OOD verdict is defined purely by the training-score range, inputs that are out-of-distribution in reality but happen to score inside that range are invisible to the method; a dedicated OOD benchmark would reveal how large this blind spot is.","The empirical nulls ignore miscalibration; pairing the quantile intervals with calibration (for example temperature scaling of the logits) could make $\\alpha$ an honest false-positive control rather than an empirical quantile statement.","A one-vs-rest extension of the two-test rule would give a multi-class abstention rule with the same \"inside everything / outside everything\" logic, but the paper only demonstrates the binary case."],"forward_implications":["Any classifier that emits a scalar score can gain an abstention rule by computing two per-class training quantiles; no resampling, architecture change, or threshold search is needed.","Raising $\\alpha$ narrows the acceptance intervals; in the reported pneumonia experiment this raised coverage from 83.97% at $\\alpha=1\\%$ to 93.91% at $\\alpha=5\\%$ while lowering accuracy from 99.24% to 97.39%.","A rejection is interpretable: the method states whether the score was plausible for both classes, plausible for neither, or plausible for exactly one.","Applying the method to X-ray screening gives a clinical decision support in which the model declines to answer when the evidence is ambiguous, matching the \"I don't know\" behavior medical AI has been missing."],"supporting_citations":[{"why":"Defines classification with a reject option and the accuracy–reject trade-off that this paper's coverage metric inherits.","marker":"Chow, 2006"},{"why":"Supplies the theory of selective classification, the setting in which coverage and error on labeled data are the natural metrics.","marker":"El-Yaniv and Wiener, 2010"},{"why":"Introduces a confidence-based selective classifier whose threshold must be tuned, the baseline the paper's parameter-free interval rule is designed to replace.","marker":"Geifman and El-Yaniv, 2017"},{"why":"MC-Dropout uncertainty quantification, the resampling-based alternative that the proposed method avoids.","marker":"Gal and Ghahramani, 2015, 2016"},{"why":"Establishes the SVM discriminant function that the spiral experiment uses as the scalar test statistic.","marker":"Vapnik, 1998"},{"why":"DenseNet, the architecture fine-tuned to produce the logit features for the pneumonia experiment.","marker":"Huang et al., 2017"},{"why":"The chest X-ray pneumonia dataset on which the medical-image experiments report coverage, accuracy, and related metrics.","marker":"Mooney, 2017"},{"why":"Provides the importance-weighted cross-entropy loss the paper uses to counter class imbalance in the pneumonia data.","marker":"Shimodaira, 2000"}],"fun_headline_variants":["Quantile-based tests spot ambiguous and outlier data","Hypothesis tests classify with uncertainty flags","Two quantile tests catch overlap and OOD points","Uncertainty-aware classification via dual hypothesis tests","Flag ambiguous and OOD data with quantile tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each class's empirical training-score distribution is a valid null distribution for hypothesis tests on future inputs, so a new score that falls outside both central intervals can be read as out-of-distribution rather than as the result of calibration shift or a feature-distribution shift.","fun_headline_variants_meta":{"raw":{"variants":["Quantile-based tests spot ambiguous and outlier data","Hypothesis tests classify with uncertainty flags","Two quantile tests catch overlap and OOD points","Uncertainty-aware classification via dual hypothesis tests","Flag ambiguous and OOD data with quantile tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2690,"prompt_tokens":950,"completion_tokens":1740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1669}},"tokens_in":566,"tokens_out":1740,"duration_ms":11353,"temperature":1.0,"reasoning_tokens":1669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:33:19.075695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method on a genuine out-of-distribution dataset whose scores mostly fall inside the training score range, for example images from a new domain that the network still scores near typical training scores; whenever the method confidently assigns those inputs to one of the two classes instead of flagging them uncertain, the claim that it detects out-of-distribution data is contradicted.","supporting_citations":[{"cited_title":", Wiener , Y","cited_arxiv_id":null,"evidence_quote":"Supplies the theory of selective classification, the setting in which coverage and error on labeled data are the natural metrics."},{"cited_title":": Statistical Learning Theory","cited_arxiv_id":null,"evidence_quote":"Establishes the SVM discriminant function that the spiral experiment uses as the scalar test statistic."},{"cited_title":": Chest X-Ray Images (Pneumonia)","cited_arxiv_id":null,"evidence_quote":"The chest X-ray pneumonia dataset on which the medical-image experiments report coverage, accuracy, and related metrics."}],"review_version":1}