{"id":"e51924de-df94-40c8-a1ad-d223c8848da0","arxiv_id":"2608.11541","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAD defines a robust-ambiguity score-pair, model-space and feature-space consistency, that flags and diagnoses unreliable predictions before deployment.","lead":"RAD is a new framework that flags a machine learning prediction as ambiguous when many similar models disagree, or when small changes to the input flip the prediction. It plots both signals as two scores so users can see whether unreliability comes from model disagreement, from sitting on a decision boundary, or from both.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central reliability claim is asserted only in the Conclusion; no table reports per-quadrant accuracy, so the claim that RAD flags predictions the model class cannot decide reliably is currently unsupported.","rationale":"Why this is the most load-bearing concern: the paper's strongest claim is not the definitions or the plots but the empirical guarantee about quadrant accuracies. The rest of the paper's evidence is consistent with that guarantee but does not entail it. Synthetic overlap experiments show that points move out of Q1 as overlap increases, and the abstention results show RAD-Pareto ranking is competitive, but neither establishes that the Q2/Q3/Q4 partition itself separates chance-level from reliable predictions. The missing table is easily produced from the existing runs, so the condition is both decisive and cheap. The reader's SMOTE-neighbourhood concern is legitimate and related: if the perturbations are off-manifold, then RADFSC is not measuring what the framework defines. I do not replace that concern with the accuracy-table concern; I put it ahead of it, because the paper states the accuracy result as the fundamental empirical finding and the result is absent. I agree with the reader's CONDITIONAL verdict: the framework is well-motivated, the math is simple, and the AURC comparisons are careful, but the central reliability claim should be verified before acceptance. I mark agreement as partial because the reader's named weakest assumption is the perturbation rule, while my primary concern is the unverified per-quadrant accuracy; the reader did flag the latter in their rationale.","tokens_in":25015,"tokens_out":5347,"duration_ms":60217,"concrete_test":"Run the RAD pipeline exactly as specified in Section 6 (n=25 bootstrap decision trees, p=100 SMOTE-style neighbourhood points, k=10, threshold 0.5) on all datasets and, for each dataset, compute the mean test-set accuracy of the deployed prediction tree separately for points whose RAD Score-Pair falls in Q1, Q2, Q3, and Q4, together with point counts and 95% confidence intervals. If the pooled accuracy over Q2/Q3/Q4 is not near the chance level while Q1 is approximately 0.91, the central reliability claim fails. This single table would settle whether the Conclusion's 'framework working as intended' statement is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the Conclusion's empirical keystone (Section 9): 'predictions in the ambiguous quadrants Q2, Q3, and Q4 are near chance-level accurate, while Q1 predictions average 0.91.' This is the statement that would make the RAD Score-Pair's quadrant partition a pre-prediction reliability certificate. Yet no table or figure in Section 7 or Appendix A reports accuracy conditioned on quadrant. The abstention experiment (Section 7.5, Table 5) reports AURC over RAD-Pareto-ranked rejection curves. AURC compares ranking against random and other scores; a high AURC does not imply the specific per-quadrant figures. For instance, a dataset with 90% Q1 points at 0.99 accuracy and 10% Q3 points at 0.60 accuracy can still show good AURC while contradicting 'near chance-level' for Q3. The claimed numbers are also threshold-specific: the 0.5 quadrant split is a design choice (Section 5.1), so the accuracy statement must be reported under the exact same configuration. Without this table, the central assertion that RAD identifies, before prediction, datapoints the model class cannot decide reliably is not supported by direct evidence. The SMOTE interpolation premise (Section 4.2) affects the semantics of RADFSC, but even before that, the paper's own claimed validation of the framework is missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Robust Ambiguity Detection (RAD), a framework that scores a test point by two chance-corrected agreement coefficients computed over an ambiguity matrix: model-space consistency (RADMSC) and feature-space consistency (RADFSC). RADMSC is Gwet's AC1 over equivalent models as raters and neighbourhood points as questions; RADFSC is AC1 on the transposed matrix. The pair is visualised in a RAD Plot divided into four quadrants at 0.5 thresholds, interpreted as robust (Q1), model-space ambiguous (Q2), fully ambiguous (Q3), and feature-space ambiguous (Q4). The paper validates the framework on five synthetic geometries at three overlap levels and on UCI/MNIST datasets, showing expected movement out of Q1 as class overlap increases, and runs an abstention experiment using RAD-Pareto-Rank, reporting AURC comparable to or better than entropy, self-consistency, and random baselines.","tokens_in":25214,"tokens_out":4937,"duration_ms":54662,"significance":"If the central reliability claim were established, RAD would be a useful and interpretable tool that combines predictive multiplicity with local robustness and provides actionable quadrant diagnoses; the per-class RAD plots for the Wine Quality and handwritten-digit datasets are a genuinely informative diagnostic. Strengths of the paper include the transparent, directly computed score definitions with no fitted constants; synthetic validation using externally controlled overlap; proper statistical testing of the abstention comparisons with Wilcoxon signed-rank plus Holm correction; and a component analysis showing that RADMSC and RADFSC are essentially uncorrelated in the ambiguous quadrants. However, the paper's most important assertion—that Q2/Q3/Q4 predictions are near chance-level while Q1 predictions average 0.91—appears only in the Conclusion and is not supported by any reported table; the current AURC evidence is ranking-based and cannot carry that claim. The significance is therefore conditional on adding direct quadrant-conditioned accuracy evidence.","major_comments":[{"comment":"The concluding claim that 'predictions in the ambiguous quadrants Q2, Q3, and Q4 are near chance-level accurate, while Q1 predictions average 0.91' is not supported by any table or figure in the manuscript. Table 5 reports only AURC, which is an aggregated ranking metric; a dataset with 90% Q1 points at 0.99 accuracy and 10% Q3 points at 0.60 accuracy can still yield a high AURC while directly contradicting the 'near chance-level' statement for Q3. Please add, for every dataset and at the 0.5 threshold, the accuracy and support size per quadrant, and report the Q1 average; reporting accuracy across a range of thresholds would also be informative because the 0.5 quadrant split is a design choice made in Section 5.1. Without this direct evidence, the framework's central promise as a pre-prediction reliability certificate is unverified.","section":"Section 9 and Section 7.5"},{"comment":"The equivalent model set consists of 25 decision trees with identical tuned hyperparameters trained on bootstrap samples of the training set. The Conclusion generalizes from this set to 'the deployed model class', but a bootstrap sample of a single algorithm is not a sample of the full model class, and the claim that RAD identifies datapoints the model class cannot decide reliably is not established by the reported experiments. Either restrict the conclusion to the generated equivalent-model set or demonstrate that quadrant assignments are stable across model families and hyperparameter settings; as written, the scope of the central claim exceeds the evidence.","section":"Section 6 (RAD Stage 1) and Section 9"},{"comment":"The feature-space dimension of RAD rests on SMOTE-style linear interpolation as the generator of 'permissible variations'. For tabular data, random interpolation between a test point and its k-th nearest neighbours can leave the data manifold, in which case a low RADFSC reflects artefacts of the sampling rule rather than genuine instability in the deployed model class. The paper itself concedes in Section 9 that the perturbation strategy requires domain adaptation for structured input types. Please provide evidence that the generated neighbourhood points are plausible—for example, density or reconstruction checks, a comparison of feature-space versus latent-space sampling on the non-MNIST datasets, and sensitivity analyses for k and p—or soften the claims that link RADFSC to genuine feature-space robustness.","section":"Section 4.2, Eq. (2)"}],"minor_comments":[{"comment":"The number of real-world datasets is inconsistent: Section 6 says '16 datasets', while Section 7.3, Table 5, and Figure 7 use N=17. Please reconcile the count.","section":"Sections 6 and 7.3"},{"comment":"Please fix typographical errors, including 'berobust' in the Abstract, 'datasests' in the Figure 7 caption, 'RADM SC' in captions, and 'a the set' in Section 4.","section":"Throughout"},{"comment":"Figure 8's caption should specify that the y-axis is the average fraction of samples that shift quadrants and should define the relationship between K and k. Section 7.1 reports stabilisation on 21 of 32 datasets; please state whether k=10 lies on the stable plateau for the remaining 11 datasets, and consider reporting the variance of the RAD scores themselves rather than only quadrant switches, because quadrant assignment is a coarse thresholded function of the scores.","section":"Figure 8 and Section 7.1"},{"comment":"In Table 5, ties are not bolded and the 'best or tied' claim is noted in the text; it would help to mark tied best values explicitly and to report the number of datasets on which RAD-Pareto is significantly better, not just numerically better, against Self-Consistency and the individual RAD components.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main concern is not novelty or circularity but an unsupported empirical keystone. The per-quadrant accuracy figures in the Conclusion appear nowhere in the results, and the current AURC analysis cannot substitute for them. The framework definition, synthetic validation, and abstention experiments are otherwise sound and within the journal's scope, so I would not reject the paper; the authors should be asked to either add the supporting quadrant-conditioned accuracy table or remove the strong reliability claim from the Conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing up front. RAD is a genuinely useful two-axis diagnostic for predictive ambiguity, and its strongest empirical claim is currently unsupported by the evidence in the paper.\n\nThe new idea is to measure agreement across two spaces at once: 25 bootstrap decision trees give model-space disagreement, 100 SMOTE-style local perturbations give feature-space disagreement, and Gwet's AC1 applied to the resulting matrix gives a score-pair that plots into four quadrants. Prior predictive-multiplicity metrics operate at a single point; local-robustness metrics use a single model. Combining them is clean, the math is simple and correctly applied, and the synthetic experiments with controlled overlap behave as expected. The abstention experiment is statistically tested (Wilcoxon signed-rank with Holm correction) and RAD-Pareto ranks competitively with self-consistency while beating entropy and random. The per-class Wine Quality plots are a nice diagnostic.\n\nThe soft spot is exactly the one the stress-test flags. Section 9 asserts that predictions in Q2/Q3/Q4 are near chance-level while Q1 averages 0.91, but no table anywhere reports accuracy conditioned on quadrant. AURC is a ranking metric; a high AURC is compatible with Q3 accuracy well above chance, so the conclusion's keystone claim is not supported by direct evidence. It is threshold-specific too, since the 0.5 quadrant split is a design choice. The neighbourhood size k=10 is chosen via a stability analysis, which is reasonable but still a free parameter. Gwet's AC1 assumes independent raters, and bootstrap-trained trees share training data, so the independence assumption is strained. The perturbation rule can generate off-manifold points, and the paper itself concedes it needs domain adaptation for structured data. There is no code and no error bars, so the central claim is not independently checkable.\n\nNone of this kills the framework. As a diagnostic and ranking tool, RAD is solid and the evaluation is honest. What needs to change is the overreach in the conclusion: add a per-quadrant accuracy table (and ideally code), and the reliability-certificate reading becomes testable. The paper deserves a serious referee; I would send it to peer review with a request for exactly that table.","headline":"Solid two-axis ambiguity diagnostic whose headline reliability claim is asserted but not demonstrated.","tokens_in":25840,"tokens_out":2717,"would_cite":true,"duration_ms":27150,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that predictive ambiguity can be quantified before prediction by a two-dimensional score pair measuring model-space and feature-space consistency, and that predictions falling in the ambiguous quadrants of the resulting…","keywords":["predictive multiplicity","local robustness","ambiguity detection","abstention","model disagreement","feature-space perturbation","Gwet's AC1","Pareto ranking"],"falsifier":"Take a labelled test set, compute the RAD Score-Pair for every point, split by the 0.5 threshold on both axes, and measure accuracy separately in Q1, Q2, Q3, and Q4; the central claim fails if accuracy in any of Q2, Q3, or Q4 is substantially above chance on a balanced task, or if changing the neighbourhood perturbation rule (for instance from interpolation to Gaussian noise) reverses which points land in the ambiguous quadrants.","tokens_in":24683,"feed_emoji":"🚦","tokens_out":6654,"duration_ms":64096,"temperature":0.7,"pith_summary":"The paper tries to establish that machine-learning predictions are dependable only when they survive two independent kinds of variation: swapping in another equally good model, and nudging the input to a nearby permissible variant. It proposes the Robust Ambiguity Detection (RAD) framework, which assigns every test point two chance-corrected agreement scores, Model-Space Consistency (RADMSC) and Feature-Space Consistency (RADFSC), computed from a matrix of predictions made by many equivalent models over locally perturbed inputs. Plotting these scores as a RAD Plot separates data points into four quadrants: robust, model-space ambiguous, feature-space ambiguous, and fully ambiguous. The load-bearing empirical claim is that predictions in the three ambiguous quadrants are near chance-level accurate, while predictions in the robust quadrant average 0.91 accuracy, suggesting the framework identifies, before prediction, inputs that the deployed model class cannot decide reliably. This matters in high-stakes settings where a model should abstain or request human review rather than emit a confident but unreliable label.","feed_headline":"Two scores catch predictions that models cannot reliably make","feed_subtitle":"The model- and feature-space consistency pair separates robust points from ambiguous ones, guiding abstention and human review.","key_machinery":"The central object is the ambiguity matrix, a p-by-n grid in which each entry is the class predicted by one of n equivalent models (here, bootstrap-trained decision trees) for one of p perturbed versions of a test point, namely the point itself plus SMOTE-style linear interpolations toward its k = 10 nearest neighbours. The key reduction is to apply Gwet's chance-corrected AC1 agreement coefficient twice to this matrix: once treating models as raters to obtain RADMSC, and once treating perturbed points as raters to obtain RADFSC. The RAD Plot then maps each point by its score pair into four quadrants with distinct diagnoses (robust, model-space ambiguous, feature-space ambiguous, fully ambiguous), and RAD Pareto-Rank orders points from most to least ambiguous for downstream abstention.","core_discovery":"The paper defines robust-ambiguity as predictive inconsistency that arises when multiple equivalent models disagree on a datapoint, or on equivalent data points in its neighbourhood. It claims this joint, two-space disagreement can be captured by the RAD Score-Pair (RADMSC, RADFSC): RADMSC is the chance-corrected agreement of models treating perturbed points as questions, and RADFSC is the chance-corrected agreement of perturbed points treating models as questions. The authors report that across synthetic and real-world datasets, predictions lying in the ambiguous quadrants Q2, Q3, and Q4 are near chance-level accurate, while predictions in the robust quadrant Q1 average 0.91 accuracy. They interpret this as evidence that the framework identifies, before prediction, datapoints that the deployed model class cannot decide reliably, and they demonstrate that ranking datapoints by RAD Pareto-Rank and abstaining on the most ambiguous ones matches or exceeds standard rejection baselines.","pith_inferences":["A testable extension is to use RAD Score-Pairs as a pre-prediction reliability certificate: if the near-chance accuracy finding persists across model families and domains, the quadrant labels could serve as an audit signal for regulated deployments, not just an abstention ranking.","The framework's diagnosis hinges on the perturbation rule; replacing the interpolation-based neighbourhood with Gaussian or manifold-constrained perturbations would show whether Q3/Q4 assignments are artefacts of the sampling rule or stable properties of the decision boundary.","The same two-axis reduction could be applied to regression by treating models' predicted scores as soft labels, which the paper leaves as future work but which the ambiguity-matrix construction would support directly.","The aggregate RAD Plot patterns (right skew, upward skew, diagonal spread) amount to a menu of likely failure modes; using these patterns as a trigger for label auditing is an implication the authors only partially develop."],"forward_implications":["If predictions in Q2, Q3, and Q4 are genuinely near chance-level, then abstaining on those points before issuing a label converts a model that looks moderately accurate overall into one whose remaining predictions are trustworthy.","RAD scores are relative to the model class: the same dataset analysed with decision trees may flag different examples than the same framework applied with neural networks, so RAD is a certification of a model-class decision, not of the data alone.","Per-class one-vs-all score pairs reveal which classes are confusable in multi-class problems, such as the middle scores of an ordered quality scale being more ambiguous than the extremes.","The ranking produced by RAD Pareto-Rank is competitive with established uncertainty and disagreement baselines, so the two-dimensional score adds diagnostic information without sacrificing rejection performance.","Because Gwet's AC1 is chance-corrected, the two scores share one interpretation: values near zero mean chance-level agreement, negative values mean systematic disagreement, and values near one mean robust agreement."],"supporting_citations":[{"why":"Supplies the chance-corrected AC1 agreement coefficient used to compute both RADMSC and RADFSC from the ambiguity matrix.","marker":"Gwet [2014]"},{"why":"Provides the SMOTE-style linear interpolation recipe adapted in Stage 2 to generate local neighbourhood points for each test point.","marker":"Chawla et al. [2002]"},{"why":"Introduces the Rashomon set of equally good models, the premise underlying model-space equivalent models.","marker":"Breiman [2001]"},{"why":"Defines arbitrariness and self-consistency; self-consistency is the baseline that RAD-Pareto rejection is compared against.","marker":"Cooper et al. [2024]"},{"why":"Rashomon Capacity is the single-point model-space multiplicity metric that RAD extends to neighbourhoods.","marker":"Hsu and Calmon [2022]"},{"why":"Defines local robustness at the feature level, the other axis that RAD combines with model-space agreement.","marker":"Leino and Fredrikson [2021]"},{"why":"Surveys local robustness of predictions under natural variations, motivating the feature-space consistency component.","marker":"Zhong et al. [2021]"},{"why":"Supplies the reject-option framing and the accuracy-versus-coverage AURC metric used in the abstention experiments.","marker":"Hendrickx et al. [2024]"},{"why":"Provides the Wilcoxon signed-rank test with Holm correction used to compare rejection methods.","marker":"Demšar [2006]"}],"fun_headline_variants":["Two consistency scores flag unreliable predictions","RAD Score-Pair separates robust from ambiguous points","Model and feature space consistency tells when to abstain","Dual consistency scores flag predictions to abstain on","Two-space consistency separates reliable from ambiguous predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that linear interpolation between a test point and its ten nearest neighbours produces permissible variations that genuinely represent the local feature-space neighbourhood; if those synthetic points fall off the data manifold, the feature-space consistency score measures artefacts of the sampling rule rather than real predictive instability, and the Q3/Q4 diagnoses no longer support the chance-level accuracy claim.","fun_headline_variants_meta":{"raw":{"variants":["Two consistency scores flag unreliable predictions","RAD Score-Pair separates robust from ambiguous points","Model and feature space consistency tells when to abstain","Dual consistency scores flag predictions to abstain on","Two-space consistency separates reliable from ambiguous predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3373,"prompt_tokens":966,"completion_tokens":2407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2337}},"tokens_in":582,"tokens_out":2407,"duration_ms":16520,"temperature":1.0,"reasoning_tokens":2337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:37:07.031226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a labelled test set, compute the RAD Score-Pair for every point, split by the 0.5 threshold on both axes, and measure accuracy separately in Q1, Q2, Q3, and Q4; the central claim fails if accuracy in any of Q2, Q3, or Q4 is substantially above chance on a balanced task, or if changing the neighbourhood perturbation rule (for instance from interpolation to Gaussian noise) reverses which points land in the ambiguous quadrants.","supporting_citations":[],"review_version":1}