REVIEW 7 cited by
Measuring Calibration in Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. How one measures calibration remains a challenge: expected calibration error, the most popular metric, has numerous flaws which we outline, and there is no clear empirical understanding of how its choices affect conclusions in practice, and what recommendations there are to counteract its flaws. In this paper, we perform a comprehensive empirical study of choices in calibration measures including measuring all probabilities rather than just the maximum prediction, thresholding probability values, class conditionality, number of bins, bins that are adaptive to the datapoint density, and the norm used to compare accuracies to confidences. To analyze the sensitivity of calibration measures, we study the impact of optimizing directly for each variant with recalibration techniques. Across MNIST, Fashion MNIST, CIFAR-10/100, and ImageNet, we find that conclusions on the rank ordering of recalibration methods is drastically impacted by the choice of calibration measure. We find that conditioning on the class leads to more effective calibration evaluations, and that using the L2 norm rather than the L1 norm improves both optimization for calibration metrics and the rank correlation measuring metric consistency. Adaptive binning schemes lead to more stablity of metric rank ordering when the number of bins vary, and is also recommended. We open source a library for the use of our calibration measures.
Forward citations
Cited by 7 Pith papers
-
BG-REAL: A Public Real-Data Anchored Benchmark for Background Manipulation Detection and Localization
A 7,000-sample background-manipulation benchmark with matched controls shows that re-encoding artifacts cause false-positive rates of 0.57–1.00 across all tested baselines.
-
False Fixed Points: Kantian Feedback, Stable Miscalibration, and Representational Compression in LLMs
Confidently wrong LLM answers behave like locally stable fixed points: no fragility gap vs correct answers, and abstention-style self-critique trades coverage for confidence.
-
Pragmatic Policy Development via Interpretable Behavior Cloning
A decision-tree behavior cloning approach that chooses the most frequent treatments per patient state yields interpretable policies with reliable off-policy evaluation in two clinical datasets.
-
Instance-Wise Monotonic Calibration by Constrained Transformation
MCCT and MCCT-I fit monotone per-rank scale and bias parameters on sorted logits for calibration, but the claimed monotonicity theorem fails for logits with negative values.
-
MisinfoTeleGraph: Network-driven Misinformation Detection for German Telegram Messages
A new German Telegram graph dataset for misinformation detection, with initial evidence that graph structure improves classification over text-only models.
-
Condition-Stratified Robustness Analysis of Post-Hoc Calibration Methods for Probabilistic Classifiers
In-dataset robustness of temperature scaling versus isotonic regression is condition-dependent and metric-specific, with TEMP more stable on Brier score and calibration slope.
-
When Models Know When They Do Not Know: Calibration, Cascading, and Cleaning
A confidence-based, training-free routing and cleaning framework improves accuracy-efficiency trade-offs and identifies mislabeled data in image and language benchmarks.
Discussion (0). Sign in to comment.