REVIEW 4 cited by
Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Calibration measures and reliability diagrams are two fundamental tools for measuring and interpreting the calibration of probabilistic predictors. Calibration measures quantify the degree of miscalibration, and reliability diagrams visualize the structure of this miscalibration. However, the most common constructions of reliability diagrams and calibration measures -- binning and ECE -- both suffer from well-known flaws (e.g. discontinuity). We show that a simple modification fixes both constructions: first smooth the observations using an RBF kernel, then compute the Expected Calibration Error (ECE) of this smoothed function. We prove that with a careful choice of bandwidth, this method yields a calibration measure that is well-behaved in the sense of (B{\l}asiok, Gopalan, Hu, and Nakkiran 2023a) -- a consistent calibration measure. We call this measure the SmoothECE. Moreover, the reliability diagram obtained from this smoothed function visually encodes the SmoothECE, just as binned reliability diagrams encode the BinnedECE. We also provide a Python package with simple, hyperparameter-free methods for measuring and plotting calibration: `pip install relplot\`.
Forward citations
Cited by 4 Pith papers
-
Discretization-free Multicalibration through Loss Minimization over Tree Ensembles
A one-shot ERM over depth-two tree ensembles on the base predictor and group indicators yields multicalibration whenever squared loss is saturated, a condition verified empirically on six datasets.
-
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
Frontier LLMs' verbalized confidence is often miscalibrated: the most accurate model is not the best-calibrated, and several models score worse than a calibrated random baseline.
-
Critical Appraisal of Fairness Metrics in Clinical Predictive AI
A scoping review of 62 fairness metrics for clinical predictive AI finds a fragmented, threshold-dependent landscape with only one clinical utility metric.
-
Good Practice Guide for quantifying uncertainties for machine learning models applied to photoplethysmography signals
A consortium guide that standardizes how to quantify and validate uncertainty for ML models applied to wearable PPG signals, with benchmarks, datasets, and software.
Discussion (0). Sign in to comment.