REVIEW 7 cited by
Conformal Language Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We propose a novel approach to conformal prediction for generative language models (LMs). Standard conformal prediction produces prediction sets -- in place of single predictions -- that have rigorous, statistical performance guarantees. LM responses are typically sampled from the model's predicted distribution over the large, combinatorial output space of natural language. Translating this process to conformal prediction, we calibrate a stopping rule for sampling different outputs from the LM that get added to a growing set of candidates until we are confident that the output set is sufficient. Since some samples may be low-quality, we also simultaneously calibrate and apply a rejection rule for removing candidates from the output set to reduce noise. Similar to conformal prediction, we prove that the sampled set returned by our procedure contains at least one acceptable answer with high probability, while still being empirically precise (i.e., small) on average. Furthermore, within this set of candidate responses, we show that we can also accurately identify subsets of individual components -- such as phrases or sentences -- that are each independently correct (e.g., that are not "hallucinations"), again with statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation using different LM variants.
Forward citations
Cited by 7 Pith papers
-
Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Citation-faithfulness metrics for AI science agents are verifier-dependent (3–18% on identical outputs), and a split-conformal guard provides a finite-sample catch-rate guarantee anchored on human gold.
-
E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
A density-ratio e-process wrapper converts black-box verifier scores into sequential decisions that control the false-alarm rate for agent trajectories, with empirical gains in early stopping.
-
QUTCC: Quantile Uncertainty Training and Conformal Calibration for Imaging Inverse Problems
QUTCC combines simultaneous quantile regression with conformal calibration of the quantile conditioning inputs to produce spatially adaptive, marginally calibrated uncertainty intervals for imaging inverse problems.
-
Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models
Conformal Arbitrage calibrates a score-gap threshold with conformal risk control so that a primary model can act when confident and defer to a guardian otherwise, with the expected guardrail loss bounded by a user-cho...
-
A quantum semantic framework for natural language processing
The paper reports CHSH inequality violations from LLM interpretations of ambiguous sentences and uses them to claim that linguistic meaning is non-classical and observer-dependent.
-
WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts
WQLCP weights calibration samples by VAE reconstruction losses and scales test scores by a test-loss quantile to improve conformal prediction under shifts, but the algorithm is ill-defined and the empirical support is weak.
-
Shapley Uncertainty in Natural Language Generation
A 'Shapley uncertainty' metric for LLM outputs is proposed, but its total equals the differential entropy it was meant to fix, and the claimed properties and performance gains are not supported.
Discussion (0). Sign in to comment.