REVIEW 4 cited by
Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we'll take a look at the most commonly used definition for calibration and then dive into a frequently used evaluation measure for model calibration. We'll then cover some of the drawbacks of this measure and how these surfaced the need for additional notions of calibration, which require their own new evaluation measures. This post is not intended to be an in-depth dissection of all works on calibration, nor does it focus on how to calibrate models. Instead, it is meant to provide a gentle introduction to the different notions and their evaluation measures as well as to re-highlight some issues with a measure that is still widely used to evaluate calibration.
Forward citations
Cited by 4 Pith papers
-
Confidence Calibration in Large Language Models
LLMs show average overconfidence moderated by a strong hard-easy effect, quantified across tasks and via the new LifeEval actuarial-probability benchmark.
-
Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
Inflated verbalized confidence in Qwen2.5-3B and Llama-3.2-3B is driven by a compact, cross-dataset set of middle-to-late-layer MLP blocks and attention heads, and steering or ablating those components at inference ti...
-
Improving Detection of Rare Nodes in Hierarchical Multi-Label Learning
A node-weighted loss combining inverse-frequency weighting and ensemble-uncertainty focal terms improves recall of rare classes in hierarchical multi-label models by up to ~5x.
-
False Fixed Points: Kantian Feedback, Stable Miscalibration, and Representational Compression in LLMs
Confidently wrong LLM answers behave like locally stable fixed points: no fragility gap vs correct answers, and abstention-style self-critique trades coverage for confidence.
Discussion (0). Continue with ORCID to comment.