{"id":"a148fc61-e164-453c-8752-71dcc5eb6c57","arxiv_id":"2607.18278","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"FALCON-Discover ranks predictions by disagreement between confidence, local support, and perturbation stability, recovering much more high-confidence error mass than confidence ranking on several tabular datasets.","lead":"This paper introduces a scoring framework for finding the most dangerous model failures—predictions that are both confident and wrong—by checking when confidence, local support, and stability disagree. On seven tabular datasets it reports that this approach surfaces far more such errors within a small review budget than standard calibration or raw confidence ranking.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are confounded with target supervision: a certainty-only supervised ranker already matches or beats the full discrepancy model on the three strongest datasets (Table 4), so the support/stability mechanism is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: a supervised detector trained on the false-confidence target is compared to baselines not trained on that target. The concern is sharper than a generic 'unfair comparison' because the paper itself reports in Table 4 that certainty-only supervised learning matches or exceeds the full model on the three strongest datasets. This does not falsify the family-level concentration claim — the null-concentration test (Table 15) and split discipline are genuinely supportive — but it does undermine the claim that support and stability are necessary structural signals. A focused test-set comparison between the full model and a certainty-only model would settle whether the discrepancy state adds measurable recovery value. Since this is the same conditionality the reader already flagged, I would keep the CONDITIONAL verdict and the request for matched supervised baselines and transparent ablation reporting.","tokens_in":21467,"tokens_out":7236,"duration_ms":74216,"concrete_test":"Run a matched test-set comparison: fit the logistic logit of Eq. 20 with the same validation folds and ωi weighting using (i) only ψ_cert = (conf, margin, ent) and (ii) the full ψ(x), for all 7 datasets × 4 seeds. Report paired test-set Capture@20 and FalseConf-AUROC with bootstrap CIs on the full-minus-certainty difference, mirroring Table 14. If on Adult, Bank, and MiniBooNE the full model's mean gain over certainty-only is not clearly positive, restrict the central claim to 'concentration is recoverable with supervised certainty features' and treat support/stability as dataset-specific signals (as on Nomao/Spambase) rather than the general discrepancy mechanism claimed in Section 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that ranking by a discrepancy state of confidence, support, and stability recovers dangerous error mass beyond confidence alone. This requires that the discrepancy state — not merely access to validation labels — drives the large gains. The learned ranker in Eqs. 20–22 is trained on validation folds to predict exactly FCτ, while the calibration/TrustScore baselines are validation-tuned on different objectives and never trained on the FCτ target. Table 4 Panel A is the paper's own strongest evidence against the necessity of the structural signals: on Adult, Bank, and MiniBooNE, certainty-only features (conf, margin, ent) give Cap@20 of 0.780/0.822/0.788 vs 0.744/0.750/0.676 for the full model. Since conf, margin, and entropy are deterministic functions of the base probability p for binary classifiers, the certainty-only ranker is essentially a learned confidence-threshold boundary. Thus the reported advantage over 'confidence alone' may reflect supervised learning of a target event rather than the discovery of a support/stability conflict region. The concentration phenomenon may still hold, but the paper's emphasized discrepancy mechanism is not doing measurable work in the headline regimes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces false-confidence concentration as an empirical object: the fraction of high-confidence errors that can be recovered by ranking samples under a fixed review budget. It proposes FALCON-Discover, a post-hoc framework that builds a discrepancy state ψ(x) from native certainty, local support/agreement, and perturbation stability, and uses this state for ranking, region localization, and calibration weighting. The main empirical claim is that ranking by this discrepancy state recovers substantially more dangerous-error mass than confidence alone, evaluated on seven binary tabular datasets with cross-fitting, multiple seeds, and thresholds. The paper reports strong results on Adult, Bank Marketing, and MiniBooNE, mixed results on Nomao/Spambase/Magic, and a boundary regime on Phoneme, and it interprets the findings as a family-level, regime-dependent discovery phenomenon rather than universal dominance.","tokens_in":21700,"tokens_out":4381,"duration_ms":49056,"significance":"If the central claim holds, the paper is a useful reframing: aggregate calibration metrics are complemented by a structural, sample-level view of where confident errors concentrate, and the proposed discrepancy representation is potentially reusable for ranking, localization, and calibration weighting. The evaluation has real strengths: held-out and cross-fitted predictions, a null-concentration test that separates true concentration from simple review-budget effects, fixed thresholds and budgets, bootstrap intervals, and a reproducibility artifact mapped to the reported tables. The paper is also unusually honest about regime dependence and does not overclaim a single universal detector. The significance is therefore conditional on whether the reported gains are due to the discrepancy state itself or to the supervised target used to train the learned ranker; the current manuscript leaves that key distinction unresolved.","major_comments":[{"comment":"The paper's central mechanistic claim is that support and stability signals add recoverable structure beyond confidence. The paper's own ablation undermines this: on Adult, Bank Marketing, and MiniBooNE, the supervised certainty-only ranker (conf, margin, ent) achieves Cap@20 of 0.780/0.822/0.788, while the full discrepancy model achieves 0.744/0.750/0.676. Since conf, margin, and entropy are deterministic functions of the single probability p(x) for a binary classifier, the certainty-only ranker is essentially a learned transform of confidence, trained on validation false-confidence labels. Thus the large gain over the raw-confidence baseline may reflect target supervision (a learned detector trained to predict exactly FCτ) rather than the discrepancy state. The main tables compare the learned discrepancy ranker against calibration/TrustScore baselines that are not trained on the FCτ ta","section":"Table 4, Panel A; Eqs. (20)–(22); Section 1"},{"comment":"There is a numerical inconsistency in the headline numbers. Table 1 reports Family Cap@20 of 0.728, 0.740, and 0.669 for Adult, Bank Marketing, and MiniBooNE, respectively, while Table 4 Panel A reports the full model Cap@20 as 0.744, 0.750, and 0.676 for the same datasets and same threshold τ=0.90. These should agree if both refer to the same learned discrepancy model. If Table 1 uses a validation-selected family member and Table 4 uses the full model under different backbone settings, the selection protocol and reporting pipeline need to be clarified. As written, the reader cannot tell whether the main table's 'best family' is the same object as the ablation table's 'full model', and this ambiguity directly affects the interpretation of the central numerical claim.","section":"Tables 1 and 4, Panel A"},{"comment":"The conflict-amplification gap Γτ(c,e,s) and slice mass M(c,e,s) are introduced as the formal characterization of strong/mixed/boundary regimes, but the paper never reports estimates of Γτ or M on any dataset. The regime labels are then used to explain which detector wins, yet the explanation is not grounded in the formal quantities. Either the paper should estimate Γτ and M on held-out data to support the regime characterization, or it should explicitly state that Eqs. (5)–(6) are conceptual and that the regime labels are post-hoc interpretations. As it stands, the explanatory narrative in Section 5 uses an unmeasured causal mechanism (conflict amplification) to account for observed ranking differences.","section":"Eqs. (5)–(6); Section 3; Section 5"}],"minor_comments":[{"comment":"The definition of agrlabel(x) is missing a closing brace; as typeset, the max operator is not closed. Also, agrpred(x) appears in Eq. (17) but is never defined; if it is neighborhood agreement on predicted labels, it should be defined explicitly.","section":"Eq. (11)"},{"comment":"The terminology 'confidence alone' is used inconsistently. In the main text it refers to raw confidence ranking, but Appendix E correctly notes that certainty-only learning is a supervised detector, not raw confidence. The paper should consistently distinguish 'raw confidence ranking' from 'supervised certainty features' to prevent the reader from attributing the supervised-learning advantage to the discrepancy state.","section":"Section 5 / Appendix E"},{"comment":"The evaluation contract lists raw-confidence ranking and random ranking as falsification baselines, but not a supervised certainty-only ranker. Given Table 4, this is exactly the baseline that could falsify the structural-signal part of the claim, so it should be added to the main evaluation contract.","section":"Appendix S"},{"comment":"The main text refers to 'Appendix Tables 4–5' while Table 4 already appears in the main text; the appendix table numbering appears to restart or is not clearly mapped. Please renumber or cross-reference consistently.","section":"Appendix references"}],"recommendation":"major_revision","confidential_remarks":"The empirical phenomenon is not obviously fabricated; the null-concentration test and cross-fitting protocol are genuine strengths. The block is that the paper's headline mechanism is not supported by its own ablation, because a supervised certainty-only ranker already matches or beats the full discrepancy model in the strongest regimes. This is fixable within the manuscript's scope by adding the supervised certainty-only baseline to the main comparison, reconciling Tables 1 and 4, and revising the mechanistic wording. That is why I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, with a caveat about the sales pitch. The paper gives a clean empirical demonstration that high-confidence errors are concentrated and recoverable under a small review budget on several tabular datasets, and it ships a null test that rules out the obvious budget artifact. That is a genuine contribution. The discrepancy-state framing is also a useful way to organize the problem, even if the individual pieces are familiar.\n\nWhere it does well: cross-fitting discipline is explicit, the null-concentration test in Appendix U is the right check, bootstrap intervals are reported, and the artifact map in Table 16 is unusually concrete. The comparison against a learned failure-predictor baseline in Table 8 partially answers the obvious 'is this just a supervised detector?' objection. I would not desk-reject this.\n\nThe soft spot is the one the stress-test note identifies, and it is real. The headline claim implies the discrepancy state — support and stability along with confidence — does the work. But Table 4 Panel A shows a certainty-only supervised ranker, using only conf/margin/ent, matching or beating the full model on Adult, Bank, and MiniBooNE. Since those three are the strongest regimes, the paper's own evidence says confidence-derived features carry the recoverable signal there; support and stability are not adding measurable ranking value in the regimes used to sell the method. The paper does partially defuse this in Appendix E by saying certainty identifies the tail while support/stability are needed for localization and transfer, but that is a different claim from the abstract's emphasis. As written, the central mechanism is overclaimed.\n\nOther soft spots: η and k are missing from the main text; the fixed coefficients in Eqs. 18/19/23 are acknowledged as ordinal rather than tuned, which is fine, but it means the 'analytic witness' results should be read as illustrative, not as optimized baselines; and the regime labels (strong/mixed/boundary) look assigned after the fact, which is acceptable if the assignment rule is stated in advance, but I did not see that rule.\n\nBottom line: this is a solid empirical study of a real phenomenon, with an honest evaluation contract and a useful null test. It needs either matching supervised baselines for the calibration/trust baselines or a rewritten claim that distinguishes 'concentration is recoverable' from 'the discrepancy mechanism is responsible.' I'd send it to a serious referee, and I'd expect the authors to have a reasonable path to an acceptable revision. I would not cite it in its current form for the support/stability claim, but I would cite the concentration/null-test result once the comparison is cleaned up.","headline":"The concentration phenomenon is real and worth taking seriously; the paper's case that support/stability drive it is not supported by its own ablation.","tokens_in":22266,"tokens_out":3525,"would_cite":false,"duration_ms":34830,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dangerous overconfidence clusters in compact, discoverable slices of prediction space, and ranking by confidence–support–stability conflict recovers most of it under a 20% review budget.","keywords":["false-confidence concentration","calibration","discrepancy discovery","prediction-to-structure learning","perturbation stability","local support","Capture@20","model-agnostic post-hoc"],"falsifier":"Re-run the strongest experiments training the learned ranker on the same validation folds but with certainty-only features (confidence, margin, entropy). If certainty-only matches or exceeds the full discrepancy ranker's Capture@20 on Adult, Bank Marketing, and MiniBooNE across all seeds, the necessity of support and stability is falsified.","tokens_in":21240,"feed_emoji":"🎯","tokens_out":3619,"duration_ms":32504,"temperature":0.7,"pith_summary":"This paper argues that when a model is confidently wrong, those errors are not spread evenly across predictions but cluster in compact, discoverable slices of prediction space. The authors propose FALCON-Discover, a model-agnostic post-hoc framework that maps each held-out prediction into a discrepancy state combining native confidence, local support, neighborhood agreement, and perturbation stability, then ranks predictions by how sharply these signals conflict. Across seven binary tabular datasets, a 20% review slice surfaced roughly 70% of high-confidence errors in the strongest regimes, far more than confidence-only or standard calibration baselines. The authors do not claim one detector dominates everywhere; the best signal varies, with stability-centered ranking winning where local decisional fragility dominates. The contribution is a structural re-framing: calibration should also locate where confidence becomes unsafe to trust, not just align scores globally.","feed_headline":"A 20% review slice finds ~70% of confident errors","feed_subtitle":"New post-hoc framework ranks predictions by confidence, local support, and stability to find concentrated dangerous overconfidence.","key_machinery":"The discrepancy state ψ(x) is the load-bearing object: for each sample it bundles native certainty (confidence, margin, entropy), local support and agreement (normalized distance to the training manifold, k-NN label agreement), and perturbation stability (mean/max probability drift, label consistency, logit variance under support-preserving neighbor mixing). Ranking rules—a fixed analytic priority-score, a simple stability-centered rule, and a weighted linear logit witness detector trained on validation folds to predict the false-confidence event—are all views over this same state. Capture@α and FalseConf-AUROC re-define the evaluation target from global calibration error to the recoverabili","core_discovery":"The central claim is that false-confidence concentration—the tendency of high-confidence errors to occupy a compact, structurally identifiable slice of prediction space—is real and recoverable. The paper defines the false-confidence event as a prediction that is wrong while the model is at least τ confident, and measures recovery by how much of that event mass a fixed review budget captures. On Adult, Bank Marketing, and MiniBooNE at τ=0.90, ranking by a learned combination of confidence, support, neighborhood agreement, and perturbation stability captured roughly 0.67–0.74 of all false-confidence events within the top 20% of samples, while the strongest validation-tuned calibration or trust","pith_inferences":["If the concentration phenomenon carries over to vision/language with modality-appropriate support and perturbation definitions, the same discovery framing could be used for auditing large deployed models, not just tabular benchmarks.","A fully fair comparison would train the same supervised learner on validation folds to predict 'wrong at high confidence' using only confidence features against the full discrepancy features; the paper's own ablation suggests certainty-only might match, which would undercut the necessity of support/stability as structural signals.","The fixed analytic score, deliberately untuned, acts as a scientific control: because a coarse monotone combination of conflict signals already surfaces concentration, the phenomenon resides in the state itself rather than in a complex detector.","A natural testable extension is multiclass: define class-wise false-confidence events and see whether top-k review slices still capture disproportionate error mass."],"forward_implications":["Under fixed 20% review budgets, a monitoring system can expect to surface a majority of dangerous high-confidence errors in concentrated regimes, not just inspect random samples.","Because the framework is model-agnostic and leaves the base classifier untouched, it can be layered onto any already-deployed binary model.","The regime-dependence implies practical pipelines should select the discrepancy-family member by validation performance rather than assume one detector is universal.","Discrepancy-weighted calibration improves both aggregate calibration metrics and dangerous-error recovery compared with unweighted recalibration, showing the discovered regions are actionable.","The finding that raw confidence ranking recovers little false-confidence mass means confidence alone is not a sufficient triage signal for high-risk predictions."],"fun_headline_variants":["20% review slice recovers ~70% of confident errors","Find dangerous overconfidence by reviewing just 20%","Confidence alone misses: combine cues to spot false confidence","Where confidence lies: new method locates dangerous errors","Pinpoint false confidence: 70% found in 20% of data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline comparison assumes it is fair to pit a supervised ranker trained on validation folds to predict exactly the false-confidence event against validation-tuned calibration and trust baselines that are not trained on that target; without that assumption, the claim that support and stability are necessary structural signals is not established.","fun_headline_variants_meta":{"raw":{"variants":["20% review slice recovers ~70% of confident errors","Find dangerous overconfidence by reviewing just 20%","Confidence alone misses: combine cues to spot false confidence","Where confidence lies: new method locates dangerous errors","Pinpoint false confidence: 70% found in 20% of data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1338,"prompt_tokens":740,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":514}},"tokens_in":484,"tokens_out":598,"duration_ms":6311,"temperature":1.0,"reasoning_tokens":514,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:35:34.842344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the strongest experiments training the learned ranker on the same validation folds but with certainty-only features (confidence, margin, entropy). If certainty-only matches or exceeds the full discrepancy ranker's Capture@20 on Adult, Bank Marketing, and MiniBooNE across all seeds, the necessity of support and stability is falsified.","supporting_citations":[],"review_version":1}