Pith. sign in

REVIEW 3 cited by

Empirically Validating Conformal Prediction on Modern Vision Architectures Under Distribution Shift and Long-tailed Data

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.01088 v1 pith:WS3O55TU submitted 2023-07-03 cs.LG cs.CVstat.ML

Kevin Kasa , Graham W. Taylor This is my paper
classification cs.LGcs.CVstat.ML
keywords conformalunderdistributionguaranteeslong-tailedmethodsperformanceprediction
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Conformal prediction has emerged as a rigorous means of providing deep learning models with reliable uncertainty estimates and safety guarantees. Yet, its performance is known to degrade under distribution shift and long-tailed class distributions, which are often present in real world applications. Here, we characterize the performance of several post-hoc and training-based conformal prediction methods under these settings, providing the first empirical evaluation on large-scale datasets and models. We show that across numerous conformal methods and neural network families, performance greatly degrades under distribution shifts violating safety guarantees. Similarly, we show that in long-tailed settings the guarantees are frequently violated on many classes. Understanding the limitations of these methods is necessary for deployment in real world and safety-critical applications.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Conformal Prediction Meets Long-tail Classification

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    Tail-Aware Conformal Prediction and its soft extension reduce class coverage imbalance in conformal prediction under long-tail label distributions while preserving marginal coverage.

  2. WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts

    cs.LG 2025-05 reject novelty 4.0 of 10

    WQLCP weights calibration samples by VAE reconstruction losses and scales test scores by a test-loss quantile to improve conformal prediction under shifts, but the algorithm is ill-defined and the empirical support is weak.

  3. Domain Adaptive Skin Lesion Classification via Conformal Ensemble of Vision Transformers

    cs.CV 2025-05 reject novelty 3.0 of 10

    An ensemble of three vision transformers with conformal prediction achieves 90.38% coverage on skin-lesion classification, but the reported improvement over single models is not evaluated with a fixed error rate.

Pith tools