A sharp Sauer inequality for multiclass and list prediction is established in terms of the DS dimension, tight for every alphabet size k, list size ℓ, and dimension value.
hub Canonical reference
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
Canonical reference. 83% of citing Pith papers cite this work as background.
54
Pith papers citing it
Background
83% of classified citations
abstract
Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.
hub tools
citation-role summary
background 5
method 1
citation-polarity summary
claims ledger
- abstract Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produ
co-cited works
representative citing papers
MiCP is the first conformal prediction method for multi-turn LLM pipelines that allocates per-turn error budgets to enable adaptive stopping with an overall coverage guarantee, shown to reduce turns and cost on RAG and ReAct benchmarks.
Proposes a scale-calibrated median-of-means estimator for robust aggregation of distributed PCA estimates on the product of Euclidean space and Grassmann manifold.
The paper derives that calibration-conditional coverage follows a Beta(k, n+1-k) law under continuous i.i.d. exchangeability and quantifies non-i.i.d. departures via Wasserstein distances on transported beta laws, yielding explicit bounds in scale-shift, clustered, and mixing regimes.
GRAPHLCP improves localized conformal prediction on graphs by using feature-aware densification and Personalized PageRank kernels to incorporate topology for better coverage and efficiency.
TRACE creates valid conformal prediction sets for complex generative models by scoring outputs via averaged denoising or velocity errors along stochastic transport paths instead of likelihoods.
Trimming helps conformal prediction under contamination precisely when the anomaly score separates retention probabilities without biasing clean scores, otherwise the retained mixture coefficient prevents substantial decontamination.
PUICL is a transformer pretrained on synthetic PU data from structural causal models that solves positive-unlabeled classification via in-context learning without gradient updates or fitting.
SCALE uses Spectral Graph Conditional Exchangeability (SGCE) and graph wavelets to achieve valid coverage and improved efficiency in conformal prediction for non-exchangeable graph time series by conformalizing high-frequency residuals conditioned on low-frequency embeddings.
SURE-RAG aggregates pair-level claim-evidence relations into interpretable signals for selective RAG answering, reaching 0.9075 Macro-F1 on HotpotQA-RAG v3 while providing auditability and reducing unsafe answers by 37% at 30% coverage.
An intrinsic effective sample size for manifold MCMC is defined via kernel discrepancy as the number of independent draws yielding equivalent expected squared discrepancy to the target.
The profile maximum likelihood estimator for the location in anisotropic hyperbolic wrapped normal models is strongly consistent, asymptotically normal, and attains the Hájek-Le Cam minimax lower bound under squared geodesic loss.
A GNN predicts Gaussians over QAOA parameters to create graph-conditioned trust regions that reduce circuit evaluations for MaxCut from 85-343 down to 45 while keeping approximation ratios within 3 points of heuristics.
A model-agnostic adaptive conformal anomaly detection approach uses weighted quantile bounds learned from past foundation model predictions to deliver interpretable p-value scores with stable calibration under shifts for time series monitoring.
Random team assignments in a professional firm reveal that indirect ties strongly increase new direct tie formation, while effects of degree and local density are smaller and less robust.
Compositional selective specificity (CSS) decomposes generated answers into claims and emits each at the most specific level supported by evidence, raising overcommitment-aware utility from 0.846 to 0.913 on LongFact while retaining 0.938 specificity.
LLM judges display per-document transitivity violations in 33-67% of cases despite low aggregate rates, while conformal prediction set widths serve as reliable indicators of document-level difficulty with cross-judge agreement.
CMRM adds a conformal quantile regularization on prediction margins to any loss, improving noisy-label classification accuracy up to 3.39% across methods and benchmarks while preserving performance at zero noise.
Conformal risk control for bounded non-monotone losses over a grid of size m achieves excess risk of order sqrt(log m / n) with n calibration samples, which is minimax optimal.
PS-DME is a new framework that controls post-selection false coverage rate for distributional KPI estimates via e-values and is provably more sample-efficient than data splitting under explicit conditions.
A model-agnostic Geometric Risk Controller reduces extreme errors in VLM-based OCR by requiring cross-view consensus before accepting outputs.
The work develops an iterative safe planner that adjusts conformal prediction bounds across policy updates via sensitivity analysis to maintain distribution-free safety guarantees despite interaction-induced distribution shifts.
MARGIN is an online calibration technique using symmetric EWMA and Bayesian shrinkage that learns per-agent per-band factors from the task stream, cutting calibration error 3-6x versus design-time baselines and lifting multi-agent resolution from 45-56% to 70-89%.
BalanceRAG uses sequential graphical testing on a 2D lattice of threshold pairs to certify safe operating points that meet target risk levels in cascaded RAG while increasing coverage.
citing papers explorer
-
An Optimal Sauer Lemma Over $k$-ary Alphabets
A sharp Sauer inequality for multiclass and list prediction is established in terms of the DS dimension, tight for every alphabet size k, list size ℓ, and dimension value.
-
Adaptive Stopping for Multi-Turn LLM Reasoning
MiCP is the first conformal prediction method for multi-turn LLM pipelines that allocates per-turn error budgets to enable adaptive stopping with an overall coverage guarantee, shown to reduce turns and cost on RAG and ReAct benchmarks.
-
Scale-Calibrated Median-of-Means for Robust Distributed Principal Component Analysis
Proposes a scale-calibrated median-of-means estimator for robust aggregation of distributed PCA estimates on the product of Euclidean space and Grassmann manifold.
-
Conformal Prediction via Transported Beta Laws
The paper derives that calibration-conditional coverage follows a Beta(k, n+1-k) law under continuous i.i.d. exchangeability and quantifies non-i.i.d. departures via Wasserstein distances on transported beta laws, yielding explicit bounds in scale-shift, clustered, and mixing regimes.
-
GRAPHLCP: Structure-Aware Localized Conformal Prediction on Graphs
GRAPHLCP improves localized conformal prediction on graphs by using feature-aware densification and Personalized PageRank kernels to incorporate topology for better coverage and efficiency.
-
TRACE: Transport Alignment Conformal Prediction via Diffusion and Flow Matching Models
TRACE creates valid conformal prediction sets for complex generative models by scoring outputs via averaged denoising or velocity errors along stochastic transport paths instead of likelihoods.
-
When Does Trimming Help Conformal Prediction? A Retained-Law Diagnostic under Calibration Contamination
Trimming helps conformal prediction under contamination precisely when the anomaly score separates retention probabilities without biasing clean scores, otherwise the retained mixture coefficient prevents substantial decontamination.
-
In-Context Positive-Unlabeled Learning
PUICL is a transformer pretrained on synthetic PU data from structural causal models that solves positive-unlabeled classification via in-context learning without gradient updates or fitting.
-
Delving into Non-Exchangeability for Conformal Prediction in Graph-Structured Multivariate Time Series
SCALE uses Spectral Graph Conditional Exchangeability (SGCE) and graph wavelets to achieve valid coverage and improved efficiency in conformal prediction for non-exchangeable graph time series by conformalizing high-frequency residuals conditioned on low-frequency embeddings.
-
SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation
SURE-RAG aggregates pair-level claim-evidence relations into interpretable signals for selective RAG answering, reaching 0.9075 Macro-F1 on HotpotQA-RAG v3 while providing auditability and reducing unsafe answers by 37% at 30% coverage.
-
Intrinsic effective sample size for manifold-valued Markov chain Monte Carlo via kernel discrepancy
An intrinsic effective sample size for manifold MCMC is defined via kernel discrepancy as the number of independent draws yielding equivalent expected squared discrepancy to the target.
-
Profile Likelihood Inference for Anisotropic Hyperbolic Wrapped Normal Models on Hyperbolic Space
The profile maximum likelihood estimator for the location in anisotropic hyperbolic wrapped normal models is strongly consistent, asymptotically normal, and attains the Hájek-Le Cam minimax lower bound under squared geodesic loss.
-
Query-Efficient Quantum Approximate Optimization via Graph-Conditioned Trust Regions
A GNN predicts Gaussians over QAOA parameters to create graph-conditioned trust regions that reduce circuit evaluations for MaxCut from 85-343 down to 45 while keeping approximation ratios within 3 points of heuristics.
-
Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring
A model-agnostic adaptive conformal anomaly detection approach uses weighted quantile bounds learned from past foundation model predictions to deliver interpretable p-value scores with stable calibration under shifts for time series monitoring.
-
Causal inference for social network formation
Random team assignments in a professional firm reveal that indirect ties strongly increase new direct tie formation, while effects of degree and local density are smaller and less robust.
-
Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems
Compositional selective specificity (CSS) decomposes generated answers into claims and emits each at the most specific level supported by evidence, raising overcommitment-aware utility from 0.846 to 0.913 on LongFact while retaining 0.938 specificity.
-
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
LLM judges display per-document transitivity violations in 33-67% of cases despite low aggregate rates, while conformal prediction set widths serve as reliable indicators of document-level difficulty with cross-judge agreement.
-
Conformal Margin Risk Minimization: An Envelope Framework for Robust Learning under Label Noise
CMRM adds a conformal quantile regularization on prediction margins to any loss, improving noisy-label classification accuracy up to 3.39% across methods and benchmarks while preserving performance at zero noise.
-
Conformal Risk Control under Non-Monotone Losses: Theory and Finite-Sample Guarantees
Conformal risk control for bounded non-monotone losses over a grid of size m achieves excess risk of order sqrt(log m / n) with n calibration samples, which is minimax optimal.
-
Post-Selection Distributional Model Evaluation
PS-DME is a new framework that controls post-selection false coverage rate for distributional KPI estimates via e-values and is provably more sample-efficient than data splitting under explicit conditions.
-
From Plausibility to Verifiability: Risk-Controlled Generative OCR with Vision-Language Models
A model-agnostic Geometric Risk Controller reduces extreme errors in VLM-based OCR by requiring cross-view consensus before accepting outputs.
-
Safe Planning in Interactive Environments via Iterative Policy Updates and Adversarially Robust Conformal Prediction
The work develops an iterative safe planner that adjusts conformal prediction bounds across policy updates via sensitivity analysis to maintain distribution-free safety guarantees despite interaction-induced distribution shifts.
-
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
MARGIN is an online calibration technique using symmetric EWMA and Bayesian shrinkage that learns per-agent per-band factors from the task stream, cutting calibration error 3-6x versus design-time baselines and lifting multi-agent resolution from 45-56% to 70-89%.
-
BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation
BalanceRAG uses sequential graphical testing on a 2D lattice of threshold pairs to certify safe operating points that meet target risk levels in cascaded RAG while increasing coverage.
-
Conditional Predictive Inference for General Structured Data with Group Symmetries
C-SymmPI reformulates conditional coverage as miscoverage error over a user-specified function class to deliver near-conditional guarantees under group symmetries and distributional invariance.
-
Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection
MSIFR stops faulty LLM generations early via staged rule-based checks, reducing token consumption 11-78% with no accuracy loss.
-
Multi-Fidelity Quantile Regression
A model-agnostic two-stage estimator links high-fidelity quantiles to low-fidelity ones via a covariate-dependent level function for faster convergence and better accuracy with limited high-fidelity data.
-
CONTRA: Conformal Prediction Region via Normalizing Flow Transformation
CONTRA generates sharp multi-dimensional conformal prediction regions by defining nonconformity scores as distances from the center in the latent space of a normalizing flow.
-
Scale selection for geometric medians on product manifolds
Joint location-scale minimization for geometric medians on product manifolds degenerates to marginal medians, and three new scale-selection methods restore identifiability with asymptotic guarantees.
-
Conformal Agent Error Attribution
A new filtration-based conformal prediction method attributes errors in multi-agent systems by producing contiguous sequence sets with finite-sample coverage guarantees, enabling rollback recovery.
-
Networked Information Aggregation for Binary Classification
Sequential prediction passing on DAGs for logistic regression yields O(M/sqrt(D)) excess loss when M-agent windows cover all features, with Omega(k/D) lower bound identifying depth as the fundamental limit.
-
Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
Unsupervised single-generation confidence calibration for reasoning LLMs via offline self-consistency proxy distillation outperforms baselines on math and QA tasks and improves selective prediction.
-
DAG-STL: A Hierarchical Framework for Zero-Shot Trajectory Planning under Signal Temporal Logic Specifications
DAG-STL decomposes long-horizon STL planning into decomposition, timed waypoint allocation, and diffusion-based trajectory generation to enable zero-shot planning under unknown dynamics.
-
Blind-Spot Mass: A Good-Turing Framework for Quantifying Deployment Coverage Risk in Machine Learning Systems
Blind-spot mass uses Good-Turing unseen-species estimation to measure the total probability of states with low empirical support, showing that 95% of operational mass lies in blind spots at tau=5 across wearable activity recognition and clinical admission data.
-
Physics-Guided Tiny-Mamba Transformer for Reliability-Aware Early Fault Warning
PG-TMT couples a physics-aligned tri-branch encoder with EVT-calibrated decision rules to achieve higher PR-AUC and shorter detection times at controlled false-alarm rates across multiple bearing datasets.
-
Selective Conformal Risk Control
Selective Conformal Risk Control combines selective classification with conformal risk control to produce compact prediction sets that meet target coverage and risk levels.
-
Self-Supervised Conformal Prediction with Equivariant Bootstrapping for Image Uncertainty Quantification
A self-supervised conformal prediction method with equivariant bootstrapping enables uncertainty quantification for ill-posed imaging inverse problems such as weak lensing mass mapping without requiring ground truth calibration data.
-
t-gems: text-guided exit modules for decreasing clip image encoder
Proposes T-GEMs plus a rate-based regularizer for early exits in CLIP encoders guided by text semantics to lower encoder usage costs.
-
Learning Context-conditioned Gaussian Overbounds for Convolution-Based Uncertainty Propagation
A learning framework trains neural networks to output context-conditioned Gaussian overbounds with provable conservatism on quantile grids for convolution-based uncertainty propagation.
-
Adaptive Conformal Prediction for Reliable and Explainable Medical Image Classification
An adaptive lambda criterion for RAPS achieves 95.72% global coverage and at least 90% coverage across all difficulty strata on medical image datasets while keeping average prediction set size at 1.09.
-
UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
UCCI calibrates LLM uncertainty to error probabilities with isotonic regression for cost-optimal cascade routing, delivering 31% cost savings at maintained accuracy on a 75k-query NER task.
-
Quantile-Free Uncertainty Quantification in Graph Neural Networks
QpiGNN provides a quantile-free dual-head architecture for GNN uncertainty quantification that directly optimizes coverage and interval width, yielding 22% higher coverage and 50% narrower intervals than baselines on 19 benchmarks with asymptotic coverage guarantees under mild assumptions.
-
Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction
A conformal prediction filter for retrieval chunks plus an attention-based factuality classifier can raise RAG answer quality by up to 6% and detect inconsistent generations up to 77% of the time.
-
An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness
Updating clinical AI models can cause prediction flips, arbitrariness, and unfair error rates across groups, requiring dedicated monitoring dimensions.
-
ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control
ReconVLA enhances pretrained vision-language-action robotic policies with conformal prediction for uncertainty estimation and failure detection without retraining.
-
Uncertainty-Aware Transformers: Conformal Prediction for Language Models
CONFIDE applies conformal prediction to transformer embeddings for valid prediction sets, improving accuracy up to 4.09% and efficiency over baselines on models like BERT-tiny.
-
Probably Approximately Correct (PAC) Guarantees for Data-Driven Reachability Analysis: A Theoretical and Empirical Comparison
Formal connections between PAC bounds for three data-driven reachability methods are established, with empirical results showing they are not interchangeable despite similarities.
-
Neural posterior estimation for scalable and accurate inverse parameter inference in Li-ion batteries
NPE delivers millisecond-scale parameter inference for Li-ion batteries that matches or exceeds Bayesian calibration accuracy while adding local sensitivity interpretability, though with higher voltage prediction errors.
-
AIVV: Neuro-Symbolic LLM Agent-Integrated Verification and Validation for Trustworthy Autonomous Systems
AIVV deploys LLM agents in a council to semantically validate anomalies in time-series data against natural-language requirements, automating human-in-the-loop verification for autonomous systems.
-
Uncertainty-Calibrated Explainable Artificial Intelligence for Fetal Ultrasound Plane Classification: A Systematic Review
PRISMA 2020 systematic review of 78 studies on fetal ultrasound plane classification paired with explainability or uncertainty, introducing the CALIB-XFUS reporting framework across six domains.