Pith. sign in

REVIEW 1 major objections 5 minor 4 cited by

Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)

T0 review · 1 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the Expected Calibration Error, despite documented weaknesses, remains the default calibration metric, and that multi-class, class-wise, and human-uncertainty calibration are needed alternatives.

desk verdict A clear visual primer on calibration and ECE that deserves a read for teaching, but the printed ECE formula is ill-typed and must be corrected before anyone copies it. read the letter →

arxiv 2501.19047 v5 pith:LH3INP6Z submitted 2025-01-31 stat.ME cs.AIcs.CVcs.LGstat.ML

Maja Pavlovic This is my paper · ORCID
classification stat.MEcs.AIcs.CVcs.LGstat.ML
keywords calibrationexpectederrorconfidencemulti-classclass-wisehumanuncertaintybinningreliabilitydiagram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This blog post is a tutorial on model calibration and its evaluation. It works through the standard definition of confidence calibration and the Expected Calibration Error (ECE), the most commonly used calibration metric, using a small visual example. The paper's substantive point is that ECE's known shortcomings—sensitivity to binning, use of only the maximum predicted probability, and pathologies where low ECE does not imply useful predictions—have not stopped it from being widely reported. It argues that calibration should be assessed through more specific notions, namely multi-class, class-wise, and human-uncertainty calibration, each with its own measures. The post is explicitly a gentle introduction, so its value is in clarifying the landscape rather than in new experimental results.

What carries the argument

The object that carries the argument is the ECE formula itself, $$\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\right|,$$ with bins $B_m$ of equal width in confidence. The post's visual step-by-step example makes visible how the formula behaves: only the maximum predicted probability contributes, and the bin count $M$ is arbitrary, so the same data can yield different ECE values and even empty or single-sample bins that dominate the weighting. The alternative calibration definitions replace the scalar 'maximum confidence' target with a richer one—the full vector $\mathbf{q}$ for multi-class calibration, a single per-class coordinate for class-wise calibration, and a per-sample human label distribution for human-uncertainty calibration. These targets in turn require different evaluation measures, such as class-wise ECE, KS-test based checks, and the EntCE, RankCS, and DistCE scores.

What would settle it

Take a set of classifiers trained on a standard 10-class image benchmark. Compute ECE with equal-width bins for $M=5,10,15,20$ and with equal-mass adaptive bins, then check whether the ranking of classifiers by ECE is stable across binning choices. If the rankings are largely stable, the practical force of the binning objection would be weakened; if rankings flip, the paper's warning is confirmed in a concrete setting.

Watch

Extended reading notes

Core claim

The central claim is that the most common calibration score, ECE, can give a misleading picture of how well a model's confidences match reality. The author demonstrates this through a worked example with 9 samples and 5 bins, then shows that changing to 10 bins empties several bins and changes the score. The underlying reasons are that ECE only checks the maximum class probability, that its binning scheme is a free choice with no neutral default, and that a degenerate model can achieve zero ECE while being useless. The paper then presents three alternative calibration notions that repair these blind spots: multi-class calibration, which compares against the full prediction vector; class-wise calibration, which checks each class probability in isolation; and human-uncertainty calibration, which matches predictions to distributions of annotator labels. On the paper's view, choosing among these notions should depend on the application, and ECE alone is not a safe summary.

Load-bearing premise

The load-bearing premise is that the cited critiques of ECE—binning sensitivity, the maximum-probability restriction, and the low-ECE pathologies—are accurately characterized and representative, and that the 9-sample toy example illustrates behavior that holds for larger real-world model outputs.

Editorial extensions

If this is right

  • If ECE continues to be reported, the bin count and binning scheme should be stated, because different choices produce different values and can change how models compare.
  • Reporting accuracy alongside ECE and a proper scoring rule such as Brier score or log-loss would guard against the pathology where low ECE coexists with poor predictions.
  • Applications that act on the full probability vector, such as autonomous driving or medical triage, should supplement confidence calibration with class-wise or multi-class calibration checks.
  • Datasets with substantial annotator disagreement should be evaluated with human-uncertainty measures such as DistCE rather than against aggregated gold labels.
  • Choosing a calibration notion should be driven by the decision context, not by convenience of the ECE computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the author leaves implicit is a sensitivity diagnostic: plotting ECE against a sweep of bin counts for a fixed model would turn the toy demonstration into a practical tool for judging whether a reported ECE is stable.
  • The taxonomy suggests a testable hypothesis: class-wise calibration errors can remain hidden in a confidence-calibrated model because per-class deviations cancel when only maximum probabilities are binned; a benchmark comparing the two notions on real classifiers could show how often this happens.
  • The human-uncertainty section implies a data requirement question: how many annotations per item are needed for DistCE to be reliable, since the measure depends on accurate human judgment distributions; this could be studied by subsampling annotation sets.
  • One could operationalize the paper's message by proposing a reporting standard that includes a reliability diagram, a binning-sweep curve, and a per-class breakdown alongside any scalar calibration error.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. This manuscript is an educational blog post that introduces confidence calibration and the Expected Calibration Error (ECE). After defining confidence calibration, it presents the standard binned ECE formula and walks through a nine-sample, five-bin toy example that yields ECE = 0.10445. It then summarizes three families of criticisms of ECE: calibration pathologies, binning sensitivity, and the restriction to maximum predicted probabilities. The second half introduces multi-class calibration, class-wise calibration, and human-uncertainty calibration, together with evaluation measures such as class-wise ECE, ACE/TACE, KS-test-based checks, EntCE, RankCS, and DistCE. The post explicitly disclaims being an in-depth survey and positions itself as a visual introduction; its central message is that ECE remains widely used despite documented weaknesses and that alternative notions of calibration deserve attention.

Significance. As an educational synthesis, the post's value lies in its clarity and breadth. The toy example is helpful, the figures are clear, and the criticisms and alternative measures are attributed to the primary literature (Naeini et al., 2015; Guo et al., 2017; Kumar et al., 2018; Nixon et al., 2019; Roelofs et al., 2022; Baan et al., 2022, among others). The post does not make new empirical or theoretical claims and provides no code or data, but for a gentle introduction that is acceptable. If the ECE definition is corrected, the post would be a reliable starting point for practitioners who want to understand why ECE should not be used as the sole calibration metric.

major comments (1)
  1. [Section 2, equations for acc(B_m) and conf(B_m)] The definitions of acc and conf are inconsistent: conf(B_m) = (1/|B_m|) * sum_{i in B_m} p_hat(x_i) averages K-dimensional probability vectors, while acc(B_m) is a scalar, so the expression acc(B_m) - conf(B_m) is ill-typed. The prose and the worked example use only the maximum probability per sample, so the intended definition is conf(B_m) = (1/|B_m|) * sum_{i in B_m} max_k p_hat_k(x_i). As printed, a reader cannot evaluate the central ECE equation or reproduce the reported value ECE = 0.10445. Please correct the formula and introduce a scalar notation for the maximum confidence (e.g., c_i) throughout Section 2.
minor comments (5)
  1. [Section 1.1] The definition conditions on the event max(p_hat(X)) = c, which has probability zero for continuous predictive distributions; consider noting that this is shorthand for a binned or conditional-expectation formulation.
  2. [Section 2.1.1 / Figure 9] The statement that 'half the bins now contain no samples' after going from 5 to 10 bins depends on the bin boundaries for the nine-sample example; please show the bin occupancy counts or qualify the wording.
  3. [Section 3.3.3] In the RankCS formula, the sum runs over n but the indicator uses the index i; fix the index and specify whether argsort is taken in ascending or descending order.
  4. [Section 3.3.3] The symbol y_i denotes a scalar label in Section 2 but a distribution over classes in Section 3.3.3; use a distinct symbol (e.g., s_i or q_i) for the soft label to avoid ambiguity.
  5. [Section 3.3.3] The text says EntCE 'captures the agreement' between model and human uncertainty, but the formula is a signed difference; clarify whether the raw difference or the absolute value is the intended error quantity, and align the formula with Baan et al. (2022).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the post is an expository survey whose claims are supported by external literature and worked examples.

full rationale

This is a gentle introductory blogpost, not a derivation. It defines calibration notions (confidence, multi-class, class-wise, human-uncertainty) using the standard formalisms from the cited literature, and it evaluates ECE by working through a small example. The central assertion that ECE remains widely used despite known weaknesses is an observation supported by external citations and by the paper's own illustrative binning exercises. No parameter is fitted and no quantity is predicted from a fitted input. There are no self-citations, and no uniqueness theorem or prior result by the author is used to force a conclusion. The one internal inconsistency in Section 2 is that the printed formula for conf(B_m) averages the full probability vector, while the prose correctly states that only maximum probabilities are used in ECE; this is a typographical/correctness flaw in an expository formula, not a circular reduction, because the intended scalar definition is stated in the surrounding text and is used consistently in the worked example. Because every load-bearing statement is either definitional, attributed to external prior work, or demonstrated with an explicit toy computation, the paper is self-contained as a review and does not reduce to its own inputs.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The post introduces no free parameters, no new entities, and only assumes standard domain assumptions about label availability and the correctness of cited definitions. This is consistent with its review nature.

assumptions (2)
  • domain assumption Ground-truth labels are available for evaluating calibration.
    The post states that 'All the approaches mentioned above share a key assumption: ground-truth labels are available' (Section 3.3.2). This assumption underlies the definitions of confidence, multi-class, and class-wise calibration.
  • domain assumption The cited definitions of calibration and ECE are correct and accepted.
    The post relies on the definitions from Guo et al. (2017), Naeini et al. (2015), and others without re-deriving them. The correctness of these definitions is taken as given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)." pith.science (2026). https://pith.science/paper/LH3INP6Z

@misc{pith2026250119047,
  author       = {Pith},
  title        = {Pith review of: Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LH3INP6Z}},
  note         = {Machine review of arXiv:2501.19047}
}
read the original abstract

To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we'll take a look at the most commonly used definition for calibration and then dive into a frequently used evaluation measure for model calibration. We'll then cover some of the drawbacks of this measure and how these surfaced the need for additional notions of calibration, which require their own new evaluation measures. This post is not intended to be an in-depth dissection of all works on calibration, nor does it focus on how to calibrate models. Instead, it is meant to provide a gentle introduction to the different notions and their evaluation measures as well as to re-highlight some issues with a measure that is still widely used to evaluate calibration.

Figures

Figures reproduced from arXiv: 2501.19047 by the authors.

Figure 1
Figure 1. Reliability Diagram Now, what calibration means more precisely depends on the specific definition being considered. We will have a look at the most common notion in machine learning (ML) formalised in (Guo et al., 2017) and termed confidence calibration in (Kull et al., 2019). But first, let’s define a bit of formal notation for this blog. In this blogpost we consider a classification task with K possible classes, w… view at source ↗
Figure 2
Figure 2. Notation - input example sourced from (Uma et al., 2021) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Confidence Calibration For simplification, we assume that we only have 3 classes as in figure 2 and we zoom into confidence c = 0.7, see image above. Let’s assume we have 10 inputs here whose most confident prediction (max) equals 0.7. If the model correctly classifies 7 out of 10 predictions (true), it is considered calibrated at confidence level 0.7. For the model to be fully calibrated this has to hold across all… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: ECE - Toy Example Only the maximum probabilities, which determine the predicted label are used in ECE (Guo et al., 2017). Therefore, we will only bin samples based on the maximum probability across classes (see left table in below image). To keep the example simple we …
Figure 5
Figure 5. Figure 5: Table 2 & Binning Diagram [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Table 3 & Binning Diagram [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Table 4 & Example for bin 5 [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Pathologies Example 3.2 BINNING APPROACH One of the most frequently mentioned issues with ECE is its sensitivity to the change in binning (Kumar et al., 2018; Nixon et al., 2019; Gupta et al., 2021; Zhang et al., 2020; Roelofs et al., 2022; Famiglini et al., 2023). Thi…
Figure 9
Figure 9. Figure 9: More Bins 6 [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Adaptive Bins Binning-based evaluation with bins containing an equal number of samples are shown to have lower bias than a fixed binning approach such as ECE (Roelofs et al., 2022). This leads (Roelofs et al., 2022) to urge against using equal width binning and to sug…
Figure 11
Figure 11. Figure 11: input example sourced from (Schwirten et al., 2024) [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Multi-class Calibration 8 [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Class-wise Calibration To evaluate such different notions of calibration, some updates are made to ECE to calculate a class-wise error. One idea is to calculate the ECE for each class and then take the average (Nixon et al., 2019; Kull et al., 2019). Another idea is t…
Figure 14
Figure 14. Figure 14: One-Hot-Vector We have the same image as in our entry example and can see that the chosen label differs between annotators. A common approach to resolving such issues in the labelling process is to use some form of aggregation (Paun et al., 2022; Artstein & Poesio, 20…
Figure 15
Figure 15. Figure 15: Soft-Label In the same example as above, instead of aggregating the annotator votes we could simply use their frequencies to create a distribution Pvote over the labels instead, which is then our new yi . This shift towards training models on collective annotator view…
Figure 16
Figure 16. Figure 16: Human Uncertainty Calibration We have our sample data (left) and zoom into a single sample x with index i = 1. The model’s predicted probability vector for this sample is [0.1,0.2,0.7]. If the human labelled distribution yi matches this predicted vector then this samp…
Figure 17
Figure 17. Figure 17: EntCE drawbacks On the left, we can see the human label distribution yi , on the right are two different model predic￾tions for that same sample. All three distributions would have the same entropy, so comparing them would result in 0 EntCE. While this is not ideal fo…
Figure 18
Figure 18. Figure 18: RankCS drawbacks 1 In the paper it is stated more generally: If the argsorts match, it means the ranking is aligned, contributing to the overall RankCS score. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]
Figure 19
Figure 19. Figure 19: Visual Summary: Definitions at a Glance 14 [PITH_FULL_IMAGE:figures/full_fig_p014_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Confidence Calibration in Large Language Models

    cs.AI 2026-04 conditional novelty 6.0 of 10

    LLMs show average overconfidence moderated by a strong hard-easy effect, quantified across tasks and via the new LifeEval actuarial-probability benchmark.

  2. Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs

    cs.CL 2026-04 conditional novelty 6.0 of 10

    Inflated verbalized confidence in Qwen2.5-3B and Llama-3.2-3B is driven by a compact, cross-dataset set of middle-to-late-layer MLP blocks and attention heads, and steering or ablating those components at inference ti...

  3. Improving Detection of Rare Nodes in Hierarchical Multi-Label Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A node-weighted loss combining inverse-frequency weighting and ensemble-uncertainty focal terms improves recall of rare classes in hierarchical multi-label models by up to ~5x.

  4. False Fixed Points: Kantian Feedback, Stable Miscalibration, and Representational Compression in LLMs

    cs.AI 2025-10 conditional novelty 6.0 of 10

    Confidently wrong LLM answers behave like locally stable fixed points: no fragility gap vs correct answers, and abstention-style self-critique trades coverage for confidence.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [6]

    doi: 10.18653/v1/2020.emnlp-main.734

    Association for Computational Linguis- tics. doi: 10.18653/v1/2020.emnlp-main.734. Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measur- ing calibration in deep learning.CVPR workshops, 2(7),

    doi:10.18653/v1/2020.emnlp-main.734 2020

  2. [8]

    Ambiguous Annotations: When is a Pedestrian not a Pedestrian?

    Luisa Schwirten, Jannes Scholz, Daniel Kondermann, and Janis Keuper. Ambiguous annotations: When is a pedestrian not a pedestrian?arXiv preprint arXiv:2405.08794,

    work page Pith review arXiv

  3. [10]

    doi: 10.18653/v1/2022.findings-emnlp.204

    Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.204. URLhttps://aclanthology.org/2022. findings-emnlp.204/. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. InThe Eleventh International Conference on Learning Representations,

    doi:10.18653/v1/2022.findings-emnlp.204 2022

  4. [11]

    A Confidence Interval for the $\ell_2$ Expected Calibration Error

    URLhttps://arxiv.org/abs/2408.08998. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey.Journal of Artificial Intelligence Research, 72: 1385–1470,

    work page Pith review arXiv

  5. [12]

    Youliang Yuan, Wenxuan Wang, Qingshuo Guo, Yiming Xiong, Chihao Shen, and Pinjia He. Does chatgpt know that it does not know? evaluating the black-box calibration of chatgpt.Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 5191–5201,

    work page 2024

  6. [1983]

    Towards a rigorous calibration as- sessment framework: Advancements in metrics, methods, and use

    15 ICLR Blogposts 2025 Lorenzo Famiglini, Andrea Campagner, and Federico Cabitza. Towards a rigorous calibration as- sessment framework: Advancements in metrics, methods, and use. InECAI 2023, pp. 645–652. IOS Press,

    work page 2025

  7. [2008]

    doi: 10.1162/coli.07-034-R2

    ISSN 0891-2017. doi: 10.1162/coli.07-034-R2. Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. InInternational Conference on Learning Representations,

    doi:10.1162/coli.07-034-r2 2017

  8. [2015]

    Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.),Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 9131–9143, Online, November

    work page 2020

Show all 12 references
  1. [2019]

    Mitigating bias in cali- bration error estimation

    16 ICLR Blogposts 2025 Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. Mitigating bias in cali- bration error estimation. InInternational Conference on Artificial Intelligence and Statistics, pp. 4036–4054. PMLR,

    2025

  2. [2020]

    Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez

    URLhttps://openreview.net/forum?id=BJxI5gHKDr. Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibration when humans disagree.Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1892–1915, December

    2022

  3. [2022]

    Human uncertainty in concept-based ai systems.Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp

    Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems.Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, an...

    2023

  4. [2024]

    Re-examining calibration: The case of question answering

    Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. Re-examining calibration: The case of question answering. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2814–2829, Abu Dhabi, Unit...

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.