Uncertainty Sets for Image Classifiers using Conformal Prediction
Reviewed by Pithpith:GEZBR7V7open to challenge →
read the original abstract
Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving coverage with sets that are often factors of 5 to 10 smaller than a stand-alone Platt scaling baseline.
This paper has not been read by Pith yet.
Forward citations
Cited by 13 Pith papers
-
RRISE: Robust Radius Inference via a Surrogate Estimator
RRISE trains a surrogate against precomputed MC targets and uses conformal calibration to deliver certified radii matching fixed-budget MC accuracy within 0.84 points while using one forward pass instead of up to 10^4...
-
Online Conformal Prediction with Corrupted Feedback
Develops robust online conformal prediction schemes that provide explicit miscoverage guarantees under feedback modeled as arbitrary binary flips or bounded-memory errors.
-
Risk-Controlled Post-Processing of Decision Policies
Risk-controlled post-processing yields a threshold-structured policy that follows the baseline except where an oracle fallback sharply reduces conditional violation risk, achieving O(log n/n) expected excess risk in i...
-
Inference for Clustering: Conformal Sets for Cluster Labels
Split conformal clustering with stochastic labels provides finite-sample marginal coverage guarantees for cluster label confidence sets, controlled by soft-label consistency and replace-one stability of the clustering...
-
ST-BCP: Tightening Coverage Bound for Backward Conformal Prediction via Non-Conformity Score Transformation
ST-BCP tightens the coverage bound in Backward Conformal Prediction by applying a computable data-dependent transformation to nonconformity scores, reducing the average gap from 4.20% to 1.12% on benchmarks while prov...
-
CONSIGN: Conformal Segmentation Informed by Spatial Groupings via Decomposition
CONSIGN applies conformal prediction to segmentation by incorporating spatial structure through decomposition, producing tighter and more interpretable uncertainty estimates with error guarantees.
-
Uncertainty-Aware Intention Prediction for Human-to-Robot Assembly Teleoperation
Human-to-robot transfer learning with conformal prediction improves robot assembly action segmentation Edit score from 70.50 to 80.70 using only 16 robot demonstrations.
-
Fair Conformal Classification via Learning Representation-Based Groups
A fair conformal classification method guarantees conditional coverage on adaptively identified subgroups defined via learned representations.
-
Robust Conditional Conformal Prediction via Branched Normalizing Flow
Branched Normalizing Flow improves conditional coverage robustness of conformal prediction under distribution shift by normalizing test inputs to the calibration distribution and mapping prediction sets back.
-
Geometry-Calibrated Conformal Abstention for Language Models
Geometry-calibrated conformal abstention lets language models abstain from uncertain queries with finite-sample guarantees on both participation rate and conditional correctness of answers.
-
Scale-Dependent Input Representation and Confidence Estimation for LLMs in Materials Property Prediction
Larger LLMs handle detailed crystal descriptions better than small ones, and mean negative log-likelihood of predicted numbers tracks prediction error after fine-tuning.
-
Selective Prediction from Agreement: A Lipschitz-Consistent Version Space Approach
Selective prediction abstains unless all Lipschitz-consistent heads in the version space agree on a certified label for each pool point.
-
Localized Conformal Prediction for Image Classification with Vision-Language Models
A non-linear transformation of cosine similarities in localized conformal prediction for VLMs yields statistically significant reductions in mean set sizes while conserving marginal coverage guarantees.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.