Pith. sign in

REVIEW 3 major objections 4 minor 33 references

FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dangerous overconfidence clusters in compact, discoverable slices of prediction space, and ranking by confidence–support–stability conflict recovers most of it under a 20% review budget.

desk verdict The concentration phenomenon is real and worth taking seriously; the paper's case that support/stability drive it is not supported by its own ablation. read the letter →

arxiv 2607.18278 v1 pith:X7S3ZB4D submitted 2026-06-29 cs.LG cs.AI

Filippo Cenacchi , Longbing Cao , Runze Yang This is my paper
classification cs.LGcs.AI
keywords false-confidenceconcentrationcalibrationdiscrepancydiscoveryprediction-to-structurelearningperturbationstabilitylocalsupportCapture@20model-agnosticpost-hoc
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when a model is confidently wrong, those errors are not spread evenly across predictions but cluster in compact, discoverable slices of prediction space. The authors propose FALCON-Discover, a model-agnostic post-hoc framework that maps each held-out prediction into a discrepancy state combining native confidence, local support, neighborhood agreement, and perturbation stability, then ranks predictions by how sharply these signals conflict. Across seven binary tabular datasets, a 20% review slice surfaced roughly 70% of high-confidence errors in the strongest regimes, far more than confidence-only or standard calibration baselines. The authors do not claim one detector dominates everywhere; the best signal varies, with stability-centered ranking winning where local decisional fragility dominates. The contribution is a structural re-framing: calibration should also locate where confidence becomes unsafe to trust, not just align scores globally.

What carries the argument

The discrepancy state ψ(x) is the load-bearing object: for each sample it bundles native certainty (confidence, margin, entropy), local support and agreement (normalized distance to the training manifold, k-NN label agreement), and perturbation stability (mean/max probability drift, label consistency, logit variance under support-preserving neighbor mixing). Ranking rules—a fixed analytic priority-score, a simple stability-centered rule, and a weighted linear logit witness detector trained on validation folds to predict the false-confidence event—are all views over this same state. Capture@α and FalseConf-AUROC re-define the evaluation target from global calibration error to the recoverabili

What would settle it

Re-run the strongest experiments training the learned ranker on the same validation folds but with certainty-only features (confidence, margin, entropy). If certainty-only matches or exceeds the full discrepancy ranker's Capture@20 on Adult, Bank Marketing, and MiniBooNE across all seeds, the necessity of support and stability is falsified.

Watch

Extended reading notes

Core claim

The central claim is that false-confidence concentration—the tendency of high-confidence errors to occupy a compact, structurally identifiable slice of prediction space—is real and recoverable. The paper defines the false-confidence event as a prediction that is wrong while the model is at least τ confident, and measures recovery by how much of that event mass a fixed review budget captures. On Adult, Bank Marketing, and MiniBooNE at τ=0.90, ranking by a learned combination of confidence, support, neighborhood agreement, and perturbation stability captured roughly 0.67–0.74 of all false-confidence events within the top 20% of samples, while the strongest validation-tuned calibration or trust

Load-bearing premise

The headline comparison assumes it is fair to pit a supervised ranker trained on validation folds to predict exactly the false-confidence event against validation-tuned calibration and trust baselines that are not trained on that target; without that assumption, the claim that support and stability are necessary structural signals is not established.

Editorial extensions

If this is right

  • Under fixed 20% review budgets, a monitoring system can expect to surface a majority of dangerous high-confidence errors in concentrated regimes, not just inspect random samples.
  • Because the framework is model-agnostic and leaves the base classifier untouched, it can be layered onto any already-deployed binary model.
  • The regime-dependence implies practical pipelines should select the discrepancy-family member by validation performance rather than assume one detector is universal.
  • Discrepancy-weighted calibration improves both aggregate calibration metrics and dangerous-error recovery compared with unweighted recalibration, showing the discovered regions are actionable.
  • The finding that raw confidence ranking recovers little false-confidence mass means confidence alone is not a sufficient triage signal for high-risk predictions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the concentration phenomenon carries over to vision/language with modality-appropriate support and perturbation definitions, the same discovery framing could be used for auditing large deployed models, not just tabular benchmarks.
  • A fully fair comparison would train the same supervised learner on validation folds to predict 'wrong at high confidence' using only confidence features against the full discrepancy features; the paper's own ablation suggests certainty-only might match, which would undercut the necessity of support/stability as structural signals.
  • The fixed analytic score, deliberately untuned, acts as a scientific control: because a coarse monotone combination of conflict signals already surfaces concentration, the phenomenon resides in the state itself rather than in a complex detector.
  • A natural testable extension is multiclass: define class-wise false-confidence events and see whether top-k review slices still capture disproportionate error mass.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces false-confidence concentration as an empirical object: the fraction of high-confidence errors that can be recovered by ranking samples under a fixed review budget. It proposes FALCON-Discover, a post-hoc framework that builds a discrepancy state ψ(x) from native certainty, local support/agreement, and perturbation stability, and uses this state for ranking, region localization, and calibration weighting. The main empirical claim is that ranking by this discrepancy state recovers substantially more dangerous-error mass than confidence alone, evaluated on seven binary tabular datasets with cross-fitting, multiple seeds, and thresholds. The paper reports strong results on Adult, Bank Marketing, and MiniBooNE, mixed results on Nomao/Spambase/Magic, and a boundary regime on Phoneme, and it interprets the findings as a family-level, regime-dependent discovery phenomenon rather than universal dominance.

Significance. If the central claim holds, the paper is a useful reframing: aggregate calibration metrics are complemented by a structural, sample-level view of where confident errors concentrate, and the proposed discrepancy representation is potentially reusable for ranking, localization, and calibration weighting. The evaluation has real strengths: held-out and cross-fitted predictions, a null-concentration test that separates true concentration from simple review-budget effects, fixed thresholds and budgets, bootstrap intervals, and a reproducibility artifact mapped to the reported tables. The paper is also unusually honest about regime dependence and does not overclaim a single universal detector. The significance is therefore conditional on whether the reported gains are due to the discrepancy state itself or to the supervised target used to train the learned ranker; the current manuscript leaves that key distinction unresolved.

major comments (3)
  1. [Table 4, Panel A; Eqs. (20)–(22); Section 1] The paper's central mechanistic claim is that support and stability signals add recoverable structure beyond confidence. The paper's own ablation undermines this: on Adult, Bank Marketing, and MiniBooNE, the supervised certainty-only ranker (conf, margin, ent) achieves Cap@20 of 0.780/0.822/0.788, while the full discrepancy model achieves 0.744/0.750/0.676. Since conf, margin, and entropy are deterministic functions of the single probability p(x) for a binary classifier, the certainty-only ranker is essentially a learned transform of confidence, trained on validation false-confidence labels. Thus the large gain over the raw-confidence baseline may reflect target supervision (a learned detector trained to predict exactly FCτ) rather than the discrepancy state. The main tables compare the learned discrepancy ranker against calibration/TrustScore baselines that are not trained on the FCτ ta
  2. [Tables 1 and 4, Panel A] There is a numerical inconsistency in the headline numbers. Table 1 reports Family Cap@20 of 0.728, 0.740, and 0.669 for Adult, Bank Marketing, and MiniBooNE, respectively, while Table 4 Panel A reports the full model Cap@20 as 0.744, 0.750, and 0.676 for the same datasets and same threshold τ=0.90. These should agree if both refer to the same learned discrepancy model. If Table 1 uses a validation-selected family member and Table 4 uses the full model under different backbone settings, the selection protocol and reporting pipeline need to be clarified. As written, the reader cannot tell whether the main table's 'best family' is the same object as the ablation table's 'full model', and this ambiguity directly affects the interpretation of the central numerical claim.
  3. [Eqs. (5)–(6); Section 3; Section 5] The conflict-amplification gap Γτ(c,e,s) and slice mass M(c,e,s) are introduced as the formal characterization of strong/mixed/boundary regimes, but the paper never reports estimates of Γτ or M on any dataset. The regime labels are then used to explain which detector wins, yet the explanation is not grounded in the formal quantities. Either the paper should estimate Γτ and M on held-out data to support the regime characterization, or it should explicitly state that Eqs. (5)–(6) are conceptual and that the regime labels are post-hoc interpretations. As it stands, the explanatory narrative in Section 5 uses an unmeasured causal mechanism (conflict amplification) to account for observed ranking differences.
minor comments (4)
  1. [Eq. (11)] The definition of agrlabel(x) is missing a closing brace; as typeset, the max operator is not closed. Also, agrpred(x) appears in Eq. (17) but is never defined; if it is neighborhood agreement on predicted labels, it should be defined explicitly.
  2. [Section 5 / Appendix E] The terminology 'confidence alone' is used inconsistently. In the main text it refers to raw confidence ranking, but Appendix E correctly notes that certainty-only learning is a supervised detector, not raw confidence. The paper should consistently distinguish 'raw confidence ranking' from 'supervised certainty features' to prevent the reader from attributing the supervised-learning advantage to the discrepancy state.
  3. [Appendix S] The evaluation contract lists raw-confidence ranking and random ranking as falsification baselines, but not a supervised certainty-only ranker. Given Table 4, this is exactly the baseline that could falsify the structural-signal part of the claim, so it should be added to the main evaluation contract.
  4. [Appendix references] The main text refers to 'Appendix Tables 4–5' while Table 4 already appears in the main text; the appendix table numbering appears to restart or is not clearly mapped. Please renumber or cross-reference consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out evaluation, untuned analytic witness scores, and a random-review null control keep the derivation independent of its inputs.

full rationale

The paper's central measurement is not circular. The learned discrepancy ranker (Eqs. 20-22) is fitted on validation folds to predict FCτ, but the reported Capture@20 and FalseConf-AUROC are computed on disjoint test folds with test labels used only for final reporting, as stated in Section F; the test result is therefore not the training target by construction. The fixed analytic scores (Eqs. 18-19) are deliberately not fitted to FC labels ('coefficients are ordinal rather than tuned'), providing an untuned witness check. The null-concentration test (Table 15) compares against random 20% slices and rules out a trivial review-budget artifact. The paper contains no load-bearing self-citations. Table 4's certainty-only results do raise a genuine confound for the narrative that support and stability drive the headline gains, since a certainty-only supervised ranker matches or beats the full model on Adult, Bank, and MiniBooNE; but this is an internal-validity/correctness concern about which signals carry the improvement, not a case where a prediction equals its input by construction. The paper explicitly acknowledges the role of certainty (Appendix E), and the regime labels are post-hoc summaries of held-out results rather than quantities derived from Eq. (5). The derivation chain is therefore self-contained in the sense that matters for circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 3 invented entities

The method's core contribution is an engineered discrepancy state plus a supervised ranker trained on false-confidence labels. The main free choices are the hand-set coefficients in the three analytic formulas, the perturbation schedule λ and neighborhood size k, and the unspecified emphasis weight η. The conceptual entities (false-confidence concentration, discrepancy state, conflict-amplification gap) are introduced by the paper without independent external evidence; they are operationalized entirely through the paper's own metrics.

free parameters (7)
  • η (false-confidence emphasis weight) = unspecified
    Appears in Eq. 22 as the weight on false-confidence events in the learned ranker objective. Its value controls how strongly the detector is trained on the target event and is not reported.
  • Hand-set coefficients in Eq. 18 = 0.7 Δmean + 0.3 (1-LC)
    The stability-centered score weights are chosen by hand from the authors' prior belief about which signal matters; the paper says they are ordinal, but they are still free choices that influence the reported family results.
  • Hand-set coefficients in Eq. 19 = 1.30, 1.00, 0.80, 1.00, 0.80, 0.70
    The fixed analytic discrepancy score is a hand-built priority ladder. The paper explicitly disclaims universal optimality, but these numbers are free parameters in the reported evaluation.
  • Hand-set coefficients in Eq. 23 = 1.2, 1.0, 0.9, 1.2, 1.0, 2.0
    The calibration weighting profile is chosen by hand to encode an operational priority ordering, not learned or derived.
  • Perturbation strengths λ = {0.10, 0.20, 0.30}
    The support-preserving perturbation levels in Eq. 12 are fixed choices; Table 4 shows results vary with λ, so the choice affects the main numbers.
  • Neighborhood size k = not specified in main text
    Used in Eqs. 11 and 12 for neighborhood agreement and perturbations; the main text never gives the k used in the headline results, and appendix sweeps show sensitivity.
  • Learned logit weights w and bias b = fitted on validation
    The learned discrepancy ranker in Eq. 20 is fitted to validation false-confidence labels; these are model parameters rather than hand-set constants, but they are free parameters of the method and their values are not reported.
assumptions (5)
  • domain assumption Dangerous overconfidence is most visible when strong certainty coexists with weak local support or instability.
    Section 3 states this as the motivating hypothesis behind FALCON-Discover. If false-confidence events are not concentrated in this way, the entire discrepancy-state construction has no reason to work.
  • domain assumption False-confidence concentration is measurable by FalseConf-AUROC and Capture@α on finite held-out data.
    The paper explicitly acknowledges in Section 3 that no non-vacuous guarantee is possible without assumptions linking event prevalence, conflict amplification, and slice mass; the metrics are treated as the operational definition of the phenomenon.
  • domain assumption Support-preserving neighbor mixing preserves the coarse semantic identity of a sample.
    Eq. 12 defines perturbations as linear interpolation with nearby training neighbors. If these perturbations change the sample's true label frequently, the stability signals measure something other than local decisional fragility.
  • domain assumption Cross-fitted held-out predictions provide uncontaminated behavior estimates for the base model.
    Section 3 and Appendix F rely on five-fold cross-fitting to claim that discrepancy features are free of training leakage. This is a standard assumption but not verified independently.
  • domain assumption Functional stability of clusters can be assessed without matching cluster labels across seeds.
    Appendix M argues that permutation-invariance makes literal cluster matching meaningless, so the paper substitutes seed-aggregated region statistics. This is a reasonable but non-trivial interpretational choice.
invented entities (3)
  • False-confidence concentration
    purpose: A structural object claimed to be recoverable: compact regions of prediction space where confident errors accumulate.
    The paper provides operational metrics for it, but those metrics are defined by the paper itself; there is no external benchmark or independent handle that would confirm the object exists outside this evaluation framework.
  • Discrepancy state ψ(x)
    purpose: A reusable representational object combining certainty, support, agreement, and stability, used for ranking, localization, and calibration weighting.
    It is a feature vector constructed by the authors. Its usefulness is measured with the paper's own false-confidence labels and metrics, so it does not have an independent falsifiable handle.
  • Conflict-amplification gap Γτ and slice mass M
    purpose: Theoretical quantities meant to characterize when false-confidence concentration should arise.
    Defined in Eqs. 5-6 but never estimated from data; they function as a conceptual framing rather than a tested entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration." pith.science (2026). https://pith.science/paper/X7S3ZB4D

@misc{pith2026260718278,
  author       = {Pith},
  title        = {Pith review of: FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7S3ZB4D}},
  note         = {Machine review of arXiv:2607.18278}
}
read the original abstract

Calibration is usually evaluated in aggregate, but the most dangerous failures are often local: predictions that remain highly confident despite being wrong. We study this failure mode as false-confidence concentration, the extent to which confident errors occupy compact, discoverable regions of prediction space. We introduce FALCON-Discover, a post-hoc, model-agnostic framework that ranks predictions using discrepancy signals from confidence, local support, neighborhood agreement, and perturbation stability. Across seven binary tabular datasets, four seeds, five-fold cross-fitting, and strong learners including XGBoost and CatBoost, we find that false-confidence concentration is recurrent but regime-dependent. At the main confidence threshold, discrepancy-based ranking substantially outperforms the strongest validation-selected calibration or trust-scoring baseline in the strongest regimes, while raw confidence recovers little dangerous-error mass. The best detector varies across datasets: learned discrepancy is strongest when multiple cues must be combined, whereas stability-centered ranking works best when local decisional fragility dominates. These results show that dangerous overconfidence is better treated as a family-level discovery problem than as a single-score calibration problem, and motivate calibration strategies that explicitly target regions where confidence, support, and stability diverge.

Figures

Figures reproduced from arXiv: 2607.18278 by the authors.

Figure 1
Figure 1. summarizes the FALCON-Discover pipeline. The method is motivated by a simple hypoth￾esis: dangerous overconfidence should be most visible when strong predictive certainty coexists with weak local support or instability under small, support-preserving changes. FALCON-Discover therefore does not introduce one more scalar trust score in isolation. Instead, it decomposes reliability into three complementary questions fo… view at source ↗
Figure 2
Figure 2. Additional visual evidence. Discrepancy-family ranking recovers substantially more high-confidence errors than prior baselines on the strongest datasets, while discrepancy regions are structurally organized rather than diffuse. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 4
Figure 4. Full per-dataset operating-point summaries available in the anonymous review artifact. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Full per-dataset discrepancy-region visualizations available in the anonymous review artifact. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 8
Figure 8. Figure 8: Full per-dataset discrepancy-aware weighting profiles available in the anonymous review artifact. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 3 linked inside Pith

  1. [1]

    Pitfalls of in- domain uncertainty estimation and ensembling in deep learning

    Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in- domain uncertainty estimation and ensembling in deep learning. InInternational Conference on Learning Representations, 2020

    2020

  2. [2]

    Addressing failure prediction by learning model confidence

    Charles Corbière, Nicolas Thome, Avi Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. InAdvances in Neural Information Processing Systems, volume 32, 2019

    2019

  3. [3]

    Efficient post-hoc uncertainty calibration via variance-based smoothing.arXiv preprint arXiv:2503.15583, 2025

    Fabian Denoodt and José Oramas. Efficient post-hoc uncertainty calibration via variance-based smoothing.arXiv preprint arXiv:2503.15583, 2025

    arXiv 2025

  4. [4]

    Phases of uncertainty: Confidence–calibration dynamics in language model training

    Aneesh Durai. Phases of uncertainty: Confidence–calibration dynamics in language model training. InProceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 11–16, Suzhou, China, 2025. Association for Computational Linguistics

    2025

  5. [5]

    On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(5):1605–1641, 2010

    Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11(5):1605–1641, 2010

    2010

  6. [6]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InProceedings of the 33rd International Conference on Machine Learning, pages 1050–1059. PMLR, 2016

    2016

  7. [7]

    Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum’e III, and Kate Crawford. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    2021

  8. [8]

    Selective classification for deep neural networks.arXiv preprint arXiv:1705.08500, 2017

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks.arXiv preprint arXiv:1705.08500, 2017

    arXiv 2017

Show all 33 references
  1. [9]

    Selectivenet: A deep neural network with an integrated reject option

    Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. InProceedings of the 36th International Conference on Machine Learning, pages 2151–2159. PMLR, 2019

    2019

  2. [10]

    Hassan Gharoun, Mohammad Sadegh Khorshidi, Kasra Ranjbarigderi, Fang Chen, and Amir H. Gandomi. Uncertainty-aware post-hoc calibration: Mitigating confidently incorrect predictions beyond calibration metrics.arXiv preprint arXiv:2510.17915, 2025

    2025

  3. [11]

    Why do tree-based models still outperform deep learning on typical tabular data? InAdvances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022

    Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? InAdvances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022

    2022

  4. [12]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning, pages 1321–1330. PMLR, 2017

    2017

  5. [13]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. InInternational Conference on Learning Representations, 2017

    2017

  6. [14]

    Better by default: Strong pre-tuned MLPs and boosted trees on tabular data

    David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned MLPs and boosted trees on tabular data. InAdvances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024

    2024

  7. [15]

    Guan, and Maya Gupta

    Heinrich Jiang, Been Kim, Melody Y . Guan, and Maya Gupta. To trust or not to trust a classifier. InAdvances in Neural Information Processing Systems, volume 31, 2018

    2018

  8. [16]

    Accurate uncertainties for deep learning using calibrated regression

    V olodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 2796–2804, 2018

    2018

  9. [17]

    Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration

    Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. InAdvances in Neural Information Processing Systems, volume 32, 2019. 10

    2019

  10. [18]

    Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers

    Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. InProceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 ofProcee...

    2017

  11. [19]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. InAdvances in Neural Information Processing Systems, volume 30, 2017

    2017

  12. [20]

    Owens, and Yixuan Li

    Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. InAdvances in Neural Information Processing Systems, volume 33, pages 21464– 21475, 2020

    2020

  13. [21]

    When do neural nets outperform boosted trees on tabular data? InAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023

    Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Benjamin Feuer, Chinmay Hegde, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? InAdvances in Neural Information Processing Systems 36 (Ne...

    2023

  14. [22]

    CLUE: Neural networks calibration via learning uncertainty-error alignment.arXiv preprint arXiv:2505.22803, 2025

    Pedro Mendes, Paolo Romano, and David Garlan. CLUE: Neural networks calibration via learning uncertainty-error alignment.arXiv preprint arXiv:2505.22803, 2025

    2025 arXiv

  15. [23]

    Revisiting the calibration of modern neural networks

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Fabio Hubis, Xiaohua Zhai, Neil Houlsby, Lucas Beyer, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 15682–15694, 2021

    2021

  16. [24]

    Model cards for model reporting

    Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. InProceedings of the Conference on Fairness, Accountability, and Transparency, pages 220– 229, 2019

    2019

  17. [25]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  18. [26]

    Predicting good probabilities with supervised learning

    Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. InProceedings of the 22nd International Conference on Machine Learning, pages 625–632, 2005

    2005

  19. [27]

    Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. InAdvances in Neural Information Processin...

    2019

  20. [28]

    John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.Advances in Large Margin Classifiers, 10(3):61–74, 1999

    1999

  21. [29]

    Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. InAdvances in Neural Information Processing Systems, volume 33, pages 3581–3591, 2020

    2020

  22. [30]

    Joris Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B. Schön. Evaluating model calibration in classification. InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 ofProceedings of Machi...

    2019

  23. [31]

    Springer, 2005

    Vladimir V ovk, Alex Gammerman, and Glenn Shafer.Algorithmic Learning in a Random World. Springer, 2005

    2005

  24. [32]

    Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers

    Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. InProceedings of the Eighteenth International Conference on Machine Learning, pages 609–616, 2001

    2001

  25. [33]

    Transforming classifier scores into accurate multiclass probability estimates

    Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. InProceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699, 2002. 11 A Evaluation Reporting Details The ...

    2002

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.