Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLUE claims that calibration can be trained into a network by matching predicted uncertainty to the model's own per-instance loss.

desk verdict A simple, differentiable calibration regularizer worth a second look, but the paper's own calibration story is incomplete and the experiments are too under-reported to support the headline claims. read the letter →

arxiv 2505.22803 v1 pith:UFBVDYBC submitted 2025-05-28 cs.LG cs.AI

Pedro Mendes , Paolo Romano , David Garlan This is my paper
classification cs.LGcs.AI
keywords modelcalibrationuncertaintyestimationtraining-timeMCdropoutpredictiveentropyloss-basedout-of-distributiondetectionlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that neural-network calibration can be achieved during training by adding a single term to the standard loss: the squared difference between the model's predicted uncertainty and the per-instance task loss. The principle is that a well-calibrated model should be uncertain exactly to the degree that its own loss indicates error, so CLUE turns calibration into a differentiable per-instance objective instead of a post-hoc or binned aggregate. If this works, uncertainty estimates become trustworthy by construction, which matters for deployment in medical, driving, and other safety-sensitive settings. The paper reports that models trained this way match or improve calibration metrics while keeping competitive accuracy across image classification, regression, summarization, translation, and out-of-distribution detection.

What carries the argument

The load-bearing object is the squared calibration regularizer $(L_e(y,\hat y) - u(\hat y))^2$ inside the CLUE loss. It compares, for each training example, the model's uncertainty estimate (predictive entropy, normalized for classification, or predicted variance for regression, obtained via MC dropout) with the pointwise task loss that same example incurs. The squared form keeps the objective continuous and differentiable, so it can be backpropagated through any standard training pipeline and calibrates the model's internal representations rather than adjusting outputs after the fact.

What would settle it

On a binary classification test set, train one CLUE model and record predicted uncertainty, cross-entropy, and 0/1 error for every example; bin by predicted uncertainty and compare average 0/1 error to average uncertainty. If the squared CLUE gap $(L_e - u)^2$ is near zero while calibration error computed from 0/1 accuracy remains large, the loss-proxy assumption has failed, because the model is calibrated to cross-entropy and not to the actual error rate.

Watch

Extended reading notes

Core claim

The paper's central claim is that calibration can be reduced to an alignment condition: for every input, predicted uncertainty $u(\hat y)$ should equal expected error, and this can be enforced by minimizing $(L_e(y,\hat y) - u(\hat y))^2$, where $L_e$ is the task loss (cross-entropy for classification, mean squared error for regression) used as a differentiable proxy for error. CLUE combines this regularizer with the usual loss as $\alpha L_e(y,\hat y) + (1-\alpha)(L_e(y,\hat y) - u(\hat y))^2$ and trains end-to-end, estimating $u$ with MC dropout and predictive entropy or predicted variance. The paper argues this formulation is task-agnostic, requires no binning or distributional comparisons, and yields calibrated uncertainty with competitive accuracy across vision, regression, language modeling, and out-of-distribution detection.

Load-bearing premise

The load-bearing premise is that the per-instance task loss (cross-entropy or mean squared error) faithfully represents the error that uncertainty should match; if a correct prediction can carry high loss, matching uncertainty to loss will not necessarily yield well-calibrated confidence.

Editorial extensions

If this is right

  • A model trained with CLUE should no longer need temperature scaling or isotonic regression to be calibrated, because the calibration objective is part of the learned weights.
  • The same loss applies unchanged to classifiers, regressors, and sequence models, so calibration training no longer has to be redesigned for each task family.
  • Because the regularizer is a squared difference, it can be added to any existing differentiable loss with negligible extra computation and without binning parameters.
  • Uncertainties aligned to loss in this way should improve downstream decisions that rely on confidence, such as selective prediction and out-of-distribution rejection.
  • In regression, pushing predicted variance toward squared error should make confidence intervals match empirical coverage more closely.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same objective defines a family of loss-alignment regularizers: any differentiable task loss and any uncertainty statistic could be paired, so the framework is a template rather than a single formula.
  • The paper's results depend on MC dropout for uncertainty estimates; an extension the authors do not test is replacing MC dropout with a single-pass variance head while keeping the same regularizer.
  • Because the target is per-instance loss rather than 0/1 error, a correct-but-low-confidence prediction with high cross-entropy could satisfy CLUE while leaving accuracy-based ECE unchanged; that distinction is worth testing explicitly.
  • The calibration weight $\alpha$ is fixed in the reported runs; annealing it during training is a natural extension that could reduce any conflict between the task loss and the regularizer in the early epochs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CLUE, a training-time calibration method that augments the task-specific loss with a squared penalty (L^e(y, yhat) - u(yhat))^2 to align the model's uncertainty with its per-instance loss (Eq. 8). The evaluation covers image classification (ImageNet, CIFAR-100/10, SVHN), tabular regression (NYC Taxi, MSD, Boston Housing), T5-based summarization and translation, an OOD detection task, and a binary classification task. The paper reports ECE, uA/uAUC, correlation with residuals, Wasserstein distance, MSE/NLL/ENCE/AUSE for regression, PRR for language models, and per-epoch times. The central claims are that CLUE achieves superior calibration and competitive predictive performance across these domains with modest computational overhead.

Significance. CLUE's loss is simple, fully differentiable, and task-agnostic, and the paper's broad evaluation and the MC-dropout efficiency study in Appendix B are useful if the method is made rigorous and reproducible. However, the paper currently does not establish the classification-calibration mechanism connecting Eq. 8 to Eq. 7; the key hyperparameter alpha is never reported; no code or seed-level variability is provided; and some reported numbers (ImageNet ResNet-50 error 0.51) indicate an unvalidated training setup. The correlation-with-residuals metric is explicitly optimized by construction, so it should not be presented as independent evidence. These issues are load-bearing for the paper's main claims.

major comments (5)
  1. [Section 3, Eqs. (7)-(8)] The calibration mechanism for classification is not established. Eq. 7 is the conditional-expectation calibration condition, and for classification the paper specializes it to E=1{y != yhat} and u=1-p. Eq. 8, however, minimizes (L^e - u)^2, where L^e is cross-entropy for classification and u is reported as normalized predictive entropy in Section 4.1. Cross-entropy is not 0-1 error (a correct prediction with confidence 0.6 can have CE of about 0.51 while the error is 0), normalized entropy is not 1-p, and Eq. 8 enforces per-instance equality rather than the conditional expectation in Eq. 7. No bound is given on ECE in terms of the Eq. 8 objective. Since Tables 1, 4, and 5 are all classification results, this gap is load-bearing; please either supply a formal argument relating the objective to Eq. 7 or reposition the claims as 'loss calibration' rather than standard confidence calibration.
  2. [Table 1, ImageNet row; Appendix A] The reported CE error of 0.51 for ResNet-50 on ImageNet is implausible for a standard model (typical top-1 error is around 0.23-0.25), suggesting an unvalidated or nonstandard training setup. Appendix A also states that CLUE is pre-trained with standard task losses before applying the CLUE objective, but the paper does not state whether the baselines receive a comparable two-stage schedule or the same total number of epochs. This confound means the reported ECE improvements could be due to additional training rather than the alignment term. Please report exact schedules and re-run matched baselines.
  3. [Equation (8); Tables 1, 4, 5] The 'Corr. w/ res.' metric is partly optimized by construction. The loss directly penalizes (L^e - u)^2, so the model is trained to make L^e and u positively correlated; the paper even states that the high correlation is 'expected given that it is explicitly designed to align the uncertainty with prediction errors.' Reporting this metric as independent evidence of calibration is circular. Please identify which metrics are optimized and which are held out, and base the calibration claims on metrics not directly minimized by Eq. 8.
  4. [Equation (8); Section 4] The trade-off parameter alpha in Eq. (8) is not reported in the main text or the appendix. Without alpha, the experiments cannot be reproduced, and it is unclear whether the reported behavior is robust or the result of tuning alpha per benchmark. Please report the alpha used for every experiment and include a sensitivity analysis over alpha.
  5. [Tables 1-6] All results are single point estimates with no error bars, confidence intervals, or significance tests. Several key comparisons are within 0.01-0.02 in ECE (e.g., CIFAR-10 CLUE ECE=0.01 vs. CALS ECE=0.01; SVHN CLUE ECE=0.01 vs. CALS ECE=0.01), which is within typical seed-to-seed variation for deep networks. The claim of 'superior calibration quality' is not supported without multiple seeds and summary statistics.
minor comments (5)
  1. [Table 1, SVHN block] The baseline row labeled 'Calibration' should presumably be 'Iso. Reg.' as in the other blocks; please fix.
  2. [References] Reference [40] lists the author as 'Paolo Romani' while the author list uses 'Paolo Romano'; please correct.
  3. [Equation (2)] Equation (2) writes the expected calibration gap as E[P(yhat=y|p=p)-p], which is the signed difference; the standard definition uses an absolute value. Please clarify the intended expression.
  4. [Section 3] The uncertainty argument is written u(hat y) in Eqs. (7)-(8) but u(x) in the specialization paragraphs; please make the notation consistent.
  5. [Table 3] Table 3 reports only PRR and quality scores for language models; no calibration error is reported, so the text's claim that CLUE yields 'better uncertainty estimations' for LMs is not directly measured.

Circularity Check

1 steps flagged · score 4.0 of 10

The 'Corr. w/ res.' evaluation metric is partially circular because Eq. 8 directly trains uncertainty to match the loss, but the central calibration claims retain independent support from ECE and OOD metrics.

  1. fitted input called prediction [Section 4.2, Image Recognition Models paragraph following Table 1; see also Eq. 8 in Section 3]
    "CLUE consistently achieves the highest correlation with residuals across benchmarks, which is expected given that it is explicitly designed to align the uncertainty with prediction errors."

    The CLUE objective in Eq. 8 is L(y, y_hat) = alpha * Le(y, y_hat) + (1 - alpha) * (Le(y, y_hat) - u(y_hat))^2. The second term explicitly fits u(.) to Le, so a high value of 'Corr. w/ res.' is a direct in-sample consequence of the training objective rather than an independent confirmation of calibration quality. In regression, Le is the squared error, so the reported correlation between uncertainty and residuals is essentially measuring the same quantity that the loss was trained to match. The paper itself calls the result 'expected,' which confirms that this particular metric is not a neutral test of the method. ECE, ENCE, uA, uAUC, and Wasserstein distance are not directly optimized by Eq. 8 and therefore still provide independent evidence, which limits the overall circularity.

full rationale

CLUE's formal derivation is mostly self-contained: Eq. 7 states a conditional-calibration identity, its specialization to classification in Section 3 (with E = 1{y != y_hat} and u = 1 - p_hat) correctly reduces to the standard confidence-calibration condition of Eq. 1, and Eq. 8 is presented as a differentiable surrogate objective rather than as an exact re-derivation of the calibration condition. The main empirical claims are also supported by externally computed metrics such as ECE, ENCE, uA, uAUC, and Wasserstein distance, which are not terms in the CLUE loss. The one clearly circular element is the use of the 'Corr. w/ res.' metric as evidence: because the loss explicitly minimizes (Le - u)^2, and because Le is the squared error in regression, a high correlation between uncertainty and residuals is partly installed by construction. The paper acknowledges this by writing that the result is 'expected.' No load-bearing self-citation was found; reference [40] is the authors' prior EUAT method, used as a baseline rather than as justification for CLUE's correctness. Overall, the circularity is confined to one supporting evaluation metric, while the central calibration-quality claim has independent empirical content, giving a score of 4.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on four tuning choices (alpha, K, dropout rate, uncertainty metric) and three unproven domain assumptions about loss-as-error proxy, MC dropout reliability, and the definition of calibration. No new entities are introduced.

free parameters (4)
  • alpha (calibration weight) = not reported
    In Eq. 8, alpha trades off task loss and calibration term; the paper never states which values were used in the experiments, making the method underspecified.
  • K (MC dropout sample count) = 5 (classification), 20 (regression), 1 (language models)
    Set per task; appendix shows regression results strongly depend on K, so this choice materially affects reported performance.
  • dropout rate = 0.3
    Fixed for all models to enable MC dropout; affects the stochastic forward passes and uncertainty estimates.
  • uncertainty metric = normalized PE for classification; PE or variance for regression
    The loss's u(y_hat) is not fixed; the paper switches between predictive entropy and predicted variance, which changes what the calibration term optimizes.
assumptions (3)
  • domain assumption The task-specific loss Le (CE or MSE) is a valid differentiable proxy for the error function E(y, y_hat) in the calibration condition
    Section 3 introduces this proxy without justification; for classification, cross-entropy can be high even for correct predictions, so aligning uncertainty to CE may not align to 0-1 error.
  • domain assumption MC dropout stochastic inference gives reliable uncertainty estimates during training and inference
    Section 3 invokes MC dropout for uncertainty estimation; the appendix shows it works but no theoretical guarantee, and regression requires many samples.
  • domain assumption Equation 7 (matching uncertainty to expected error) is the appropriate operational definition of calibration
    Section 3 defines perfect calibration this way, but ECE and other binning metrics are not equivalent to this condition; the method optimizes for its own definition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment." pith.science (2026). https://pith.science/paper/UFBVDYBC

@misc{pith2026250522803,
  author       = {Pith},
  title        = {Pith review of: CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFBVDYBC}},
  note         = {Machine review of arXiv:2505.22803}
}
read the original abstract

Reliable uncertainty estimation is critical for deploying neural networks (NNs) in real-world applications. While existing calibration techniques often rely on post-hoc adjustments or coarse-grained binning methods, they remain limited in scalability, differentiability, and generalization across domains. In this work, we introduce CLUE (Calibration via Learning Uncertainty-Error Alignment), a novel approach that explicitly aligns predicted uncertainty with observed error during training, grounded in the principle that well-calibrated models should produce uncertainty estimates that match their empirical loss. CLUE adopts a novel loss function that jointly optimizes predictive performance and calibration, using summary statistics of uncertainty and loss as proxies. The proposed method is fully differentiable, domain-agnostic, and compatible with standard training pipelines. Through extensive experiments on vision, regression, and language modeling tasks, including out-of-distribution and domain-shift scenarios, we demonstrate that CLUE achieves superior calibration quality and competitive predictive performance with respect to state-of-the-art approaches without imposing significant computational overhead.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    FALCON-Discover ranks predictions by disagreement between confidence, local support, and perturbation stability, recovering much more high-confidence error mass than confidence ranking on several tabular datasets.

  2. Uncertainty Estimation by Human Perception versus Neural Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Neural network uncertainty estimates correlate only weakly with human-perceived uncertainty on three vision benchmarks, and soft-label training improves that alignment, though the claimed calibration benefit is not me...

Reference graph

Works this paper leans on

71 extracted references · 63 canonical work pages · cited by 2 Pith papers

  1. [1]

    Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi

    Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021

    work page 2021

  2. [2]

    Objective evaluation of deep uncertainty predictions for covid-19 detection

    Hamzeh Asgharnezhad, Afshar Shamsi, Roohallah Alizadehsani, Saeid Khosravi, Abbas and- Nahavandi, Zahra Alizadeh Sani, Dipti Srinivasan, and Sheikh Mohammed Shariful Islam. Objective evaluation of deep uncertainty predictions for covid-19 detection. Scientific Reports, 12, 2022

    work page 2022

  3. [3]

    Ellis, Brian Whitman, and Paul Lamere

    Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In In Proceedings of the 12th International Society for Music Information Retrieval Conference, 2011

    work page 2011

  4. [4]

    Weight uncertainty in neural networks

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, 2015

    work page 2015

  5. [5]

    Single shot mc dropout approximation

    Kai Brach, Beate Sick, and Oliver Dürr. Single shot mc dropout approximation. ArXiv, abs/2007.03293, 2020

    arXiv 2007

  6. [6]

    New york city taxi fare prediction

    Andy Chavez, DJ Sterling, Julia Elliott, Lakshmanan V , Sagar, and Will Cukier- ski. New york city taxi fare prediction. https://kaggle.com/competitions/ new-york-city-taxi-fare-prediction , 2018. Kaggle

    work page 2018

  7. [7]

    Lawrence

    Andreas Damianou and Neil D. Lawrence. Deep Gaussian processes. In Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Research, pages 207–215, Scottsdale, Arizona, USA, 2013. PMLR

    work page 2013

  8. [8]

    Calibration of pre-trained transformers

    Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020

    work page 2020

Show all 71 references
  1. [9]

    Training uncertainty- aware classifiers with conformalized deep learning

    Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty- aware classifiers with conformalized deep learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, 2022

    2022

  2. [10]

    LM-polygraph: Uncertainty estimation for language models

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-polygraph: Uncertainty estimation for language models. In Proce...

    2023

  3. [11]

    Unsupervised quality estimation for neural machine translation

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8, 2020

    2020

  4. [12]

    Revisiting deep learning models for tabular data

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021

    2021

  5. [13]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70. PMLR, 2017

    2017

  6. [14]

    Calibration of neural networks using splines

    Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchis- escu, and Richard Hartley. Calibration of neural networks using splines. In International Conference on Learning Representations, 2021. 10

    2021

  7. [15]

    Hedonic housing prices and the demand for clean air

    David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1):81–102, 1978

    1978

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  9. [17]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019

    2019

  10. [18]

    Hoffman, David M

    Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(40), 2013

    2013

  11. [19]

    Soft calibration objectives for neural networks

    Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. In M. Ran- zato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in N...

    2021

  12. [20]

    What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, volume 30, 2017

    Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  13. [21]

    Improving model calibration with accuracy versus uncertainty optimization

    Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. In Advances in Neural Information Processing Systems, volume 33, 2020

    2020

  14. [22]

    Uncertainty quantification metrics for deep regression

    Simon Kristoffersson Lind, Ziliang Xiong, Per-Erik Forssén, and V olker Krüger. Uncertainty quantification metrics for deep regression. Pattern Recognition Letters, 186, 2024

    2024

  15. [23]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  16. [24]

    Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation. In Proceedings of the 11th International Conference on Learning Representations, 2023

    2023

  17. [25]

    Accurate uncertainties for deep learning using calibrated regression

    V olodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Proceedings of the 35th International Conference on Machine Learning, volume 80, 2018

    2018

  18. [26]

    Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers

    Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54, 2017

    2017

  19. [27]

    Verified uncertainty calibration

    Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, volume 32, 2019

    2019

  20. [28]

    DEUP: Direct epistemic uncertainty prediction

    Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research, 2023

    2023

  21. [29]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. InProceedings of the 31st International Conference on Neural Information Processing Systems, 2017

    2017

  22. [30]

    Fast, Lüder A

    Max-Heinrich Laves, Sontje Ihler, Jacob F. Fast, Lüder A. Kahrs, and Tobias Ortmaier. Well- calibrated regression uncertainty in medical imaging with deep learning. In Proceedings of the Third Conference on Medical Imaging with Deep Learning, volume 121. PMLR, 2020

    2020

  23. [31]

    Well-calibrated model uncertainty with temperature scaling for dropout variational inference

    Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, and Tobias Ortmaier. Well-calibrated model uncertainty with temperature scaling for dropout variational inference. In 4th workshop on Bayesian Deep Learning on International Conference on Neural Information Processing Sy...

    2019

  24. [32]

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks

    Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018

    2018

  25. [33]

    Evaluating and calibrating uncertainty prediction in regression tasks

    Dan Levi, Liran Gispan, Niv Giladi, and Ethan Fetaya. Evaluating and calibrating uncertainty prediction in regression tasks. Sensors, 22(15), 2022

    2022

  26. [34]

    Generating with confidence: Uncertainty quan- tification for black-box large language models

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quan- tification for black-box large language models. Transactions on Machine Learning Research, 2024

    2024

  27. [35]

    The devil is in the mar- gin: Margin-based label smoothing for network calibration

    Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the mar- gin: Margin-based label smoothing for network calibration. In Computer Vision and Pattern Recognition Conference, 2022

    2022

  28. [36]

    Class adaptive network calibration

    Bingyuan Liu, Jérôme Rony, Adrian Galdran, Jose Dolz, and Ismail Ben Ayed. Class adaptive network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16070–16079, June 2023

    2023

  29. [37]

    David J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Computation, 4(3):448–472, 1992

    1992

  30. [38]

    David J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003

    2003

  31. [39]

    Uncertainty estimation in autoregressive structured prediction

    Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In Proceedings of the 9th International Conference on Learning Representations, 2021

    2021

  32. [40]

    Error-driven uncertainty aware training

    Pedro Mendes, Paolo Romani, and David Garlan. Error-driven uncertainty aware training. In Proceedings of the 27TH European Conference on Artificial Intelligence, 2024

    2024

  33. [41]

    Calibrating deep neural networks using focal loss

    Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, volume 33, 2020

    2020

  34. [42]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015

    2015

  35. [43]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the sum- mary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018

    2018

  36. [44]

    Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag, 1996

    1996

  37. [45]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y . Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011

    2011

  38. [46]

    Nix and A.S

    D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), 1994

    1994

  39. [47]

    Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran

    Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019

    2019

  40. [48]

    Epistemic neural networks

    Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. In 37th Conference on Neural Information Processing Systems, 2023

    2023

  41. [49]

    Analyzing uncertainty in neural machine translation

    Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, 2018. 12

    2018

  42. [50]

    Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processi...

    2019

  43. [51]

    John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, 1999

    1999

  44. [52]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 2020

    2020

  45. [53]

    Rahul Rahaman and Alexandre H. Thiery. Uncertainty quantification and deep ensembles. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021

    2021

  46. [54]

    Out-of-distribution detection and selective generation for conditional language models

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In Proceedings of the 11th International Conference on Learning Representations , 2023

    2023

  47. [55]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC...

    2015

  48. [56]

    Rutemiller and David Alexander Bowers

    Herbert C. Rutemiller and David Alexander Bowers. Estimation in a heteroscedastic regression model. Journal of the American Statistical Association, 63:552–557, 1968

    1968

  49. [57]

    Segalman, Matthew R

    Daniel J. Segalman, Matthew R. Brake, Lawrence A. Bergman, Alexander F. Vakakis, and Kai Willner. Epistemic and aleatoric uncertainty in modeling. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume V ol- um...

    2013

  50. [58]

    An uncertainty-aware loss function for training neural networks with calibrated predictions

    Afshar Shamsi, Hamzeh Asgharnezhad, AmirReza Tajally, Saeid Nahavandi, and Henry Leung. An uncertainty-aware loss function for training neural networks with calibrated predictions. ArXiv, abs/2110.03260, 2023

    2023 arXiv

  51. [59]

    A mathematical theory of communication

    Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27, 1948

    1948

  52. [60]

    Prediction and entropy of printed english

    Claude Elwood Shannon. Prediction and entropy of printed english. Bell System Technical Journal, 30, 1951

    1951

  53. [61]

    Distribution calibration for regression

    Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In Proceedings of the 36th International Conference on Machine Learning, volume 97, 2019

    2019

  54. [62]

    Relevant and informative response generation using pointwise mutual information

    Junya Takayama and Yuki Arase. Relevant and informative response generation using pointwise mutual information. In Proceedings of the First Workshop on NLP for Conversational AI, 2019

    2019

  55. [63]

    Parallel data, tools and interfaces in OPUS

    Jörg Tiedemann. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth Inter- national Conference on Language Resources and Evaluation (LREC’12). European Language Resources Association (ELRA), 2012

    2012

  56. [64]

    Mutual information alleviates halluci- nations in abstractive summarization

    Liam van der Poel, Ryan Cotterell, and Clara Meister. Mutual information alleviates halluci- nations in abstractive summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022

    2022

  57. [65]

    Hybrid uncertainty quantification for selective text classifi- cation in ambiguous tasks

    Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, and Artem Shelmanov. Hybrid uncertainty quantification for selective text classifi- cation in ambiguous tasks. In Proceedings of the 61st Annual Meeting of the Association for Compu...

    2023

  58. [66]

    The Wasserstein distances, pages 93–111

    Cédric Villani. The Wasserstein distances, pages 93–111. Springer Berlin Heidelberg, 2009

    2009

  59. [67]

    A survey on bayesian deep learning.ACM Comput

    Hao Wang and Dit-Yan Yeung. A survey on bayesian deep learning.ACM Comput. Surv., 53(5), sep 2020

    2020

  60. [68]

    On hallucination and predictive uncertainty in conditional language generation

    Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021

    2021

  61. [69]

    Detection of adversarial examples in text classification: Benchmark and baseline via robust density estimation

    KiYoon Yoo, Jangho Kim, Jiho Jang, and Nojun Kwak. Detection of adversarial examples in text classification: Benchmark and baseline via robust density estimation. In Findings of the Association for Computational Linguistics: ACL 2022, 2022

    2022

  62. [70]

    Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers

    Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, 2001

    2001

  63. [71]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. InProceedings of the British Machine Vision Conference 2016, 2016. Technical Appendix A Experimental Setup and Implementation Details Across all experiments, models include dropout layers with a fixed dropout rate o...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.