REVIEW 22 cited by
Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use the framework to provide new calibration methods for several core machine learning tasks, with detailed worked examples in computer vision and tabular medical data.
Forward citations
Cited by 22 Pith papers
-
Certify or Refuse: A Cross-Model Map for Selective Risk Control with Coverage Floors under Covariate Shift
Under bounded-ratio covariate shift, a certified coverage floor creates a two-resource sample-complexity map, with a provable impossibility over the full unknown-weight class.
-
When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design
Safety in surrogate-driven design requires oracle-only certification; rank preservation is the exact criterion for oracle use, and selection-aware auditing is the cheapest way to certify it.
-
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation
DAPRO provides the first dynamic, theoretically guaranteed way to allocate interaction budgets across test cases for bounding time-to-event in multi-turn LLM evaluations, achieving tighter coverage than static conform...
-
Conformal Coverage Guarantees for Any Video Temporal Grounder
A post-hoc conformal wrapper converts any video temporal grounder's single interval into a region that contains the true moment with probability at least 1-alpha.
-
Certified Domain Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with Conformal Risk Guarantees
C3R certifies per-domain retrieval contamination budgets using a two-split conformal scheme, without query-time domain labels.
-
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Answer correctness and question answerability are separate axes: ordinary confidence tracks the first while hidden probes track the second, and a factorized dual-threshold policy certifies both risk budgets at higher ...
-
What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study
On hard multi-table text-to-SQL, verification-based LLM judges beat self-consistency and log-probability for predicting execution correctness, and fine-tuned verifiers fail to transfer across schemas.
-
Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting
An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.
-
You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Control
A model-agnostic post-hoc pipeline uses conformal risk control to guarantee that the expected fraction of unwanted items in a recommendation list stays below a user-chosen level.
-
Black-Box Edge AI Model Selection with Conformal Latency and Accuracy Guarantees
Combining conformal risk control with an order-statistic delay bound, the paper proposes fixed and channel-adaptive model selection for wireless edge inference that claims guaranteed loss and deadline violation probability.
-
Thought calibration: Efficient and confident test-time scaling
A calibrated early-exit rule for reasoning LLMs, based on hidden-state probes and Learn-then-Test risk control, reduces thinking tokens by up to 60% in-distribution and 20% out-of-distribution while roughly preserving...
-
Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents
Prediction sets paired with max-min decisions are a sufficient statistic for value-at-risk optimizing agents, and the RAC algorithm builds such sets with distribution-free coverage.
-
Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs
RG-PT learns a reliability DAG over hyperparameters and combines it with FDR-controlled testing, yielding shorter reliable LLM prompts than LTT and Pareto testing.
-
Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction
Conformal pruning of answer choices plus a second LLM pass improves MCQ accuracy in most tested settings, and learned scores yield smaller prediction sets.
-
Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
Jointly calibrated early thought-stopping and perplexity deferral cut edge-agent thinking compute 43–73% while keeping finite-sample reward and cloud-call guarantees.
-
Audited Selective Verification for Risk-Controlled N-1 Thermal Contingency Screening under Deployment Shift
A randomized audit certifies, with high confidence, that skipped N-1 contingencies violate thermal limits at most a chosen rate even under deployment shift, cutting full AC studies by 29–75% on three test systems.
-
CoVeR: Conformal Calibration for Versatile and Reliable Autoregressive Next-Token Prediction
CoVeR is a cluster-aware conformal decoding method that claims full-sequence coverage for LLM outputs without the (1-alpha)^L decay of prior conformal beam search.
-
COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees
COIN selects the largest uncertainty threshold whose Clopper-Pearson upper confidence bound on the calibration error rate remains below the target FDR, yielding a single-answer selection policy with a probabilistic ri...
-
Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey
A survey organizes current research on trustworthy RAG into six pillars, reliability, privacy, safety, fairness, explainability, and accountability, and maps methods, metrics, and open problems for each.
-
Efficient Precision Control in Object Detection Models for Enhanced and Reliable Ovarian Follicle Counting
A learn-then-test calibration procedure selects detection thresholds with a precision guarantee, and two extra filters recover recall in ovarian follicle counting.
-
Ensuring Reliability via Hyperparameter Selection: Review and Advances
The paper reviews methods that cast hyperparameter selection as multiple hypothesis testing to deliver formal risk guarantees.
-
False Discovery Control in Multiple Testing: A Brief Overview of Theories and Methodologies
This is a survey of false discovery rate control methods, organized into ranking, FDP estimation, and thresholding steps.
Discussion (0). Continue with ORCID to comment.