Conformalized Large Language Models under Configuration Shift

Evgeny Kharlamov; Gengyuan Zhang; Jialin Yu; Jie Tang; Lin Li; Philip Torr; Puneet Dokania; Steffen Staab; Yuqicheng Zhu; Zhen Yang

REVIEW 3 major objections 4 minor 17 references

Conformalized Large Language Models under Configuration Shift

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Routine changes to an LLM's prompt, decoding temperature, or weight quantization can break conformal prediction's coverage guarantee; the loss is governed by the distance between calibration and test score distributions.

desk verdict A solid empirical and theoretical case that LLM configuration changes break conformal coverage, with a label-free diagnostic that is clever but not yet certified. read the letter →

arxiv 2608.01460 v1 pith:2MPVTTHV submitted 2026-08-02 cs.LG

Yuqicheng Zhu , Jialin Yu , Lin Li , Gengyuan Zhang , Zhen Yang , Steffen Staab , Puneet Dokania , Philip Torr

show 2 more authors

Jie Tang Evgeny Kharlamov

This is my paper · ORCID

classification cs.LG

keywords configurationshiftconformalpredictionlargelanguagemodelscoverageguaranteesnonconformityscoresdistributioncalibrationensemblingpromptsensitivity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that conformal prediction guarantees for LLMs are guarantees about a complete inference pipeline, not about a model or dataset alone. When the prompt template, decoding temperature, or weight quantization changes between calibration and deployment while the data stay the same, the nonconformity score distribution shifts and the exchangeability underlying split conformal prediction is lost. Across nine LLMs, four datasets, and four scoring rules, the paper finds that configuration shift pushes empirical coverage below target in a large share of runs, up to 46% under temperature shift at 90% target coverage, while set sizes stay close to their iid baselines. It proves coverage lower bounds in terms of distributional discrepancies, gives label-free plug-in diagnostics for shift severity, and shows that a small labeled recalibration set, around 50 examples, or a label-free configuration ensemble recovers much of the lost coverage.

What carries the argument

The central object is the nonconformity score as a function of the whole inference configuration, $S = V(Z, \xi)$, and the random acceptance half-line $E = (-\infty, \hat\lambda]$ it induces. Coverage loss is the expected discrepancy between the two score laws on that half-line, and restricting the total-variation discrepancy to half-lines yields the Kolmogorov–Smirnov distance $d_{\mathrm{KS}}$, the tightest simple bound the paper uses. For label-free operation, the predictor state $U$—the predictive probability vector for white-box scores or the sorted sample-frequency profile for black-box scores—stands in for the score; Assumption 1 (kernel stability: the conditional law of the score giv

What would settle it

Take a configuration pair with small plug-in KS distance $d_{\mathrm{KS}}(P_S^{\xi_0}, P_S^{\xi'})$ but deliberately change the score kernel between calibration and deployment, for example by switching the answer normalizer for open-ended QA while holding prompt and temperature fixed. If empirical coverage then falls below $1-\alpha - d_{\mathrm{KS}}$ by more than finite-sample error, the discrepancy accounting is incomplete; the paper's own Table 1 shows the label-free $W_1$ proxy slipping 1–15% in conservativeness, so the sharpest test is whether an adversarial score-kernel change violates t

Watch

Extended reading notes

Core claim

Configuration shift—changing the prompt, decoding temperature, or weight quantization while keeping the data distribution fixed—moves the nonconformity score map $V(Z,\xi)$ and breaks the exchangeability split conformal prediction needs. The paper proves the coverage loss is controlled by the distance between calibration and test score laws: $C \ge 1-\alpha - d_{\mathrm{KS}}(P_S^{\xi_0}, P_S^{\xi'})$, with a looser total-variation version. A label-free reduction via predictor state $U$ gives $C \ge 1-\alpha - d_{\mathrm{TV}}(P_U^{\xi_0}, P_U^{\xi'})$ or $C \ge 1-\alpha - L W_1^\rho(P_U^{\xi_0}, P_U^{\xi'})$ under kernel stability. Empirically, at 90% target coverage, shifted undercoverage ra

Load-bearing premise

The load-bearing premise is Assumption 1 (kernel stability): changing the inference configuration shifts the distribution of the observable predictor state $U$ but leaves the conditional distribution of the nonconformity score given $U$ unchanged; the label-free bounds and the $\alpha$-Inf and Recal corrections depend on it, and the open-ended QA results additionally depend on the answerable-intersection restriction.

Editorial extensions

If this is right

Any conformalized LLM deployment that reports a coverage guarantee must also report the exact prompt template, decoding settings, and model precision; the guarantee does not survive a change in any of them.
More calibration data does not fix configuration shift: as the calibration budget grows, the threshold converges to the calibration quantile, which is the wrong quantile for the shifted test distribution, so undercoverage can worsen with budget.
A practice-ready check needs no test labels: comparing predictor-state distributions under calibration and deployment via TV or sliced-$W_1$ gives a plug-in lower bound on coverage, so shift severity can be monitored at deployment time.
With a small labeled deployment budget, the Recal method—recalibrating only when the estimated discrepancy exceeds a no-shift noise floor—recovers most of the lost coverage; without labels, a uniform configuration ensemble (Mos-U) already absorbs part of the loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper treats each shift axis separately; multi-axis shifts (a new prompt on a quantized model at high temperature) will likely compound score discrepancies, and the paper's diagnostics should be stress-tested in that regime.
The label-free bounds suggest an operational deployment monitor: periodically recompute $W_1$ between calibration and live predictor states and alert when it crosses a threshold, since coverage is provably at risk before labels arrive—a use the paper does not spell out.
If the score map is part of the pipeline, then other routinely swapped components—retrievers, system prompts, answer normalizers—should be treated as configuration axes too; the paper's framing predicts the same coverage erosion there.
The open-ended QA numbers are conditional on the answerable-intersection population (the gold answer being sampled into the candidate pool); a deployment where the candidate generator rarely surfaces the true answer would see larger effective failures than the reported figures.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

What you should know: this paper is worth taking seriously. It defines configuration shift—changes to prompt, temperature, quantization—as a distinct failure mode for conformalized LLMs, and shows empirically that it erodes coverage badly. That finding is robust across 9 models, 4 datasets, 4 scores, and the failure pattern is consistent. The KS bound (Theorem 3) is standard and correct: coverage is at least 1-alpha minus the KS distance between calibration and test score distributions. The empirical diagnostics based on it work. The paper also shows practical mitigations, and the Limitations section is refreshingly honest about what is not certified.

The main soft spot is the label-free branch. Theorem 4's Wasserstein bound needs Assumption 1 (kernel stability) plus an L-Lipschitz kernel with known L. The paper sets L=1 heuristically and never tests Assumption 1. Table 1 reports conservativeness rates as low as 0.85 for W1, so the plug-in is not a certified lower bound. The paper says the residual failures are 'consistent with mild violations' of Assumption 1, but that is interpretation, not a test. This weakens the W1 diagnostic and the α-Inf mitigation, which depend on the bound holding. The central claim—configuration shift breaks CP validity—does not depend on the label-free bound, so the core result stands.

Other soft spots: no code or data shipped, which matters given the breadth of the empirical claims. OEQA results are conditioned on an answerable-intersection population. That is disclosed in the appendix, and it is arguably the right call to separate conformal miscalibration from the candidate-generation bottleneck, but it means OEQA coverage numbers are not raw coverage. The free parameters list is long (L, K', M, B, J, k, beta), and the report would be stronger with sensitivity analysis.

Who is this for: anyone doing CP for LLMs. The paper cleanly exposes a failure mode that the conformal LLM literature has mostly treated as an auxiliary check. It deserves a serious referee. The revision path is clear: release code and data, estimate L or present W1 as purely heuristic, and test Assumption 1 directly or weaken the claim. With those changes, I'd be comfortable citing the empirical findings and the KS bound.

My recommendation: send it to peer review. It is not ready as-is for publication without the code/data and with the label-free bound oversold, but the core is solid and the topic is important.

Referee Report

3 major / 4 minor

Summary. The paper defines 'configuration shift' as a change in the LLM inference configuration (prompt template, decoding temperature, weight quantization) that alters the nonconformity-score distribution while leaving the data distribution fixed. It reports a broad empirical study across 9 LLMs, 4 datasets, and 4 nonconformity scores showing that such shifts substantially increase the undercoverage rate of split conformal prediction (e.g., shifted U_tau = 0.46 for temperature, 0.31 for prompt, 0.20 for quantization at 1-alpha = 0.9, versus at most 0.03 in distribution), while prediction-set sizes among valid runs remain close to i.i.d. baselines. The paper then derives population lower bounds on coverage in terms of distributional discrepancies between calibration and test scores: a TV bound (Proposition 2), a KS bound (Theorem 3), and label-free variants using TV or Wasserstein distance on a predictor state under a kernel-stability assumption (Theorem 4). It evaluates finite-sample plug-in estimates of these discrepancies as diagnostics and proposes six mitigations, two of which (Mos-F and Recal) are reported to recover much of the lost coverage.

Significance. If the findings hold, the paper identifies a practically important and under-appreciated failure mode: conformal guarantees in LLM pipelines are tied to the full inference configuration, not merely to the model and data. The score-space KS bound is a clean tightening of the TV bound and is correctly proved in Appendix E. The empirical investigation is broad and the central failure pattern is consistent across models, datasets, and scores. The label-free W1 diagnostic and the alpha-Inf and Recal mitigations are potentially valuable, but their certification depends on Assumption 1 and on the heuristic choice L=1, neither of which is verified; Table 1 documents nontrivial violations of the W1 plug-in bound. The OEQA findings are additionally conditioned on an answerable-intersection population. These caveats weaken the label-free theoretical branch and the guarantee interpretation of the mitigations, but do not undermine the central empirical claim or Theorem 3.

major comments (3)

[Section 5, Theorem 4, Table 1, Appendix F/G] The label-free Wasserstein bound in Eq. (11) requires Assumption 1 (kernel stability) and an L-Lipschitz score kernel with a known L. In the experiments L is set to 1 heuristically in the sliced-W1 estimator (Appendices F and G.3), and Assumption 1 is not tested. Table 1 shows W1 empirical conservativeness rates as low as 0.85-0.88 for temperature/OEQA cells, so the plug-in bound is violated in a non-negligible fraction of aggregate cells. The text attributes these to 'mild violations' of Assumption 1 and estimation error, but no test or quantitative bound on the violation is provided. Consequently, Eqs. (10)-(11) are not certified in the empirical setting, and the alpha-Inf and Recal corrections, which set b_epsilon = L * W1_hat with L=1, are heuristic rather than guaranteed. This does not invalidate Theorem 3, which remains sound, but it substantially weakens the label-free theoretical
[Appendix D.1, Table 5] OEQA coverage is evaluated only on the answerable intersection S_ij, i.e., questions for which the gold answer appears in both the calibration and test candidate pools under the respective configurations. This is a post-hoc, cell-specific conditioning: it excludes the very cases where configuration shift might cause the gold answer to stop being sampled. The reported OEQA undercoverage rates and the pooled claims in Section 4.2 are therefore conditional on a latent retention event whose distribution may itself shift across configurations. The manuscript discloses this clearly, but the abstract and Section 4.2 still present the OEQA results as part of the broad empirical claim. Please report unconditional coverage or retention-adjusted numbers, and state explicitly in the main text that OEQA conclusions apply to the answerable subpopulation.
[Section 9 and Section 1] The paper repeatedly calls the plug-in estimates 'finite-sample plug-in diagnostics' and uses them to motivate mitigations, but Section 9 correctly notes that the theoretical results are population statements and are not finite-sample certificates. This distinction should be moved forward: the abstract and contributions say 'coverage lower bounds' and 'plug-in diagnostics' without immediately qualifying that the plug-in versions are not certified high-probability bounds. Without this qualification, readers may overinterpret the W1-based alpha-Inf and Recal results. The limitation is acknowledged, but the framing should be tightened so that the main claims do not outrun the proofs.

minor comments (4)

[Appendix C] The prompt-template code is said to be stored in 'experiments/' YAML files, but no repository link is provided. For reproducibility, please release the code/data or include the exact templates in the appendix.
[Table 1] W1 conservativeness varies from 0.85 to 0.99 across cells. A per-axis or per-score breakdown would help readers see where the violations concentrate and whether the pattern is driven by particular scores or datasets.
[Appendix E, Remark 1] The upper bound in Eq. (17) relies on an almost-sure distinctness condition. LLM scores are often discrete or tied; please state how ties are handled in the calibration procedure and whether the upper bound remains valid under the tie convention used.
[Figure 5] The '92% of configs above y=x' and '82% above 1.0' percentages are informative but are pooled across configurations. Reporting standard errors or confidence intervals for these percentages would strengthen the fine-grained claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: bounds are proven from standard CP guarantees; empirical results are measured; heuristic diagnostics are explicitly labeled as such.

full rationale

The paper's central empirical claim—configuration shift erodes CP validity—is an experimental measurement, not a fitted prediction. The theoretical bounds (Prop. 2, Thm. 3, Thm. 4) are proven from the standard split-conformal guarantee (Thm. 1) and the definitions of TV/KS/Wasserstein discrepancies; the lower bound C ≥ 1−α−d_KS follows from E[P^{ξ0}_S(E)] ≥ 1−α and the half-line supremum property, and is not an input. Thm. 4 is explicitly conditional on Assumption 1 and an L-Lipschitz kernel, with L=1 labeled a heuristic in Apps. F and G.3; the Limitations section disclaims finite-sample certification, so the label-free diagnostics are not overclaimed as predictions. Table 1's conservativeness rates are self-consistency checks (does the plug-in bound proxy lie below empirical coverage on the same cells?), not out-of-sample predictions, and thus not circular. Mos-F/Anc use fragility profiles from disjoint audit splits (App. G.5); Recal recalibrates on a labeled deployment budget with no evidence that those labels are in the evaluation set. Self-citations appear only in related-work positioning (App. H) and are not load-bearing. No circular step can be exhibited.

Assumptions & free parameters 8 free parameters · 5 assumptions · 1 invented entities

The central empirical findings are measurements and carry no derivation burden. The theoretical lower bounds rely on standard split-CP exchangeability, Kantorovich-Rubinstein duality, and the domain assumption of kernel stability. The diagnostics and mitigations involve several hand-set hyperparameters (L=1, K'=4, M=4, B=50, J=50, k=50, beta) that are not fitted to the coverage target but affect sharpness and behavior.

free parameters (8)

Lipschitz constant L = 1
The W1 label-free bound (Theorem 4, Eq. 11) requires the CDF map u -> K((-inf,t]|u) to be L-Lipschitz; Appendix F sets L=1 in the sliced-W1 heuristic without estimation, making the bound an approximation.
OEQA top-K' predictor profile = 4
For open-ended QA the sample-frequency vector has question-dependent length; Appendix F summarizes each question by its sorted top-4 frequencies, discarding tail information in label-free diagnostics.
Predictor-state quantization grid M = 4
Predictor states are quantized to a 1/M grid for the TV_U estimate (Appendix F, M=4), affecting the plug-in discrepancy.
Histogram bins B for score TV = 50
Score-space TV estimate uses 50 bins (Appendix F), a free choice for the plug-in diagnostic.
Sliced-W1 projections J = 50
The W1 proxy averages 50 random 1D projections (Appendix F), a heuristic approximation.
Recal label budget k = 50
Recal spends a labeled deployment budget of k=50 when the gate fires (Appendix G, Figure 8); coverage saturates after ~20 labels per the ablation.
Recal noise floor beta = 95th percentile of epsilon on i.i.d. cells
Recal gates on whether the estimated discrepancy exceeds beta, which is estimated from data as the 95th percentile of i.i.d. cells (Appendix G.4).
Sampling budget M_generations = 20
Black-box scores use 20 sampled generations per example (Appendix A.2), affecting score reliability and the predictor-state estimates.

assumptions (5)

standard math Standard split-CP exchangeability and quantile guarantee for calibration scores under a fixed configuration (Theorem 1).
Used as the starting point for all bounds (Lemma 5, Corollary 6).
domain assumption Data Z_1,...,Z_{n+1} are i.i.d. from P_Z and independent of configurations.
Configuration shift keeps P_Z fixed; this is the premise of the whole setup (Section 3).
domain assumption Assumption 1 (Kernel stability): there is a Markov kernel K from U to R such that the score distribution conditional on predictor state U is invariant across configurations.
Needed for the label-free bounds in Theorem 4 and for the label-free diagnostics; the authors note mild violations in Table 1.
standard math Kantorovich-Rubinstein duality for the 1-Wasserstein distance on a Polish metric space (Villani 2009).
Used to prove the W1 form of Theorem 4 (Appendix E.3).
domain assumption For OEQA, evaluation is conditional on the answerable-intersection population where the gold answer appears in both calibration and test candidate pools.
Appendix D.1 defines S_ij and evaluates coverage only on that intersection, which is a post-hoc conditioning that affects the empirical claims.

invented entities (1)

Configuration shift independent evidence
purpose: Names and isolates the phenomenon where the score-generating pipeline changes while the data distribution stays fixed.
The concept yields falsifiable predictions: coverage drops when prompt, temperature, or quantization changes, which the paper tests empirically across 9 LLMs and 4 datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Conformalized Large Language Models under Configuration Shift." pith.science (2026). https://pith.science/paper/2MPVTTHV

@misc{pith2026260801460,
  author       = {Pith},
  title        = {Pith review of: Conformalized Large Language Models under Configuration Shift},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MPVTTHV}},
  note         = {Machine review of arXiv:2608.01460}
}

abstract

Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability. Yet for LLMs, nonconformity scores are often induced by an inference pipeline, not just a fixed model, making them depend not only on the data distribution but also on configurable factors such as the prompt template, decoding parameters, and deployment setting. Since such configurations are routinely modified in practice but rarely treated as a source of shift, their impact on CP validity remains poorly understood. We call this \emph{configuration shift} and study it systematically along three axes: prompt template, decoding temperature, and weight quantization. In a broad empirical study spanning $9$ LLMs, $4$ datasets, and $4$ nonconformity scores, we find that configuration shift consistently erodes CP validity, often driving empirical coverage below the target. By contrast, efficiency is largely preserved: valid prediction sets remain close in size to the i.i.d. baseline. We derive coverage lower bounds that attribute this loss to a discrepancy between calibration and test score distributions, and use their finite-sample plug-in versions as empirical diagnostics of shift severity. We further show that these findings lead to practical mitigations: bound-inspired recalibration is effective with limited test examples, while fragility-aware calibration ensembling recovers much of the lost coverage without test data.

Figures

Figures reproduced from arXiv: 2608.01460 by the authors.

**Figure 1.** Conformalized LLMs are highly sensitive to configuration shift, and the failure intensifies with larger calibration budgets. For each (dataset, model,score) configuration, we pool 10 random calibration/test splits and compute undercoverage rate Uτ and set-size inflation Infl separately for in-distribution (ξ ′ = ξ0) and shifted (ξ ′ ̸= ξ0) cells. Bars and points show medians across configurations; error bars and sha… view at source ↗

**Figure 2.** Mitigation comparison across the three shift axes. Median undercoverage rate Uτ (top) and set-size inflation Infl (bottom) at 1 − α = 0.9, across (model, dataset,score) configurations. Error bars are cross-configuration standard deviations. The Anc+K3 label denotes Anc instantiated with K = 3 (a K-ablation is deferred to Appendix G). vanilla CP undercovers it on the same configuration grid as Section 4 but auxiliar… view at source ↗

**Figure 3.** MCQA BASELINE template. {choices} and {demo_choices} are rendered in paren_upper (“(A) text . . . (D) text”); the demonstration is drawn from dev-split index 0. C.2 MCQA template suite (MMLU, MedMCQA) The MCQA suite contains 14 templates: 9 surfaceformat variants (a full 3 × 3 delimiter × label-set grid), 3 instruction-syntax variants (counting the anchor), and 3 content-ablation variants. All templates are shared… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: OEQA BASELINE template. {demos} expands to the five demonstration QA pairs of [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

**Figure 5.** Figure 5: Per-configuration coverage damage under configuration shift, pooled across task types. Each summary is for one (dataset, model,score) configuration and one shift source, pooling over the audit grid and 10 trials at 1 − α = 0.9; MCQA and open-ended QA configurations ar…

**Figure 6.** Figure 6: Configuration shift increases coverage error across targets and calibration budgets. Median absolute deviation from the nominal coverage target, computed from the same audit grid as [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 8.** Figure 8: Effect of the Recal labeled budget k (prompt axis). Median Uτ (top) and Infl (bottom) at 1 − α = 0.9, pooled over (model, dataset,score). Error bars are cross-configuration standard deviations; dotted/dashed lines are vanilla CP and i.i.d. Infl = 1; the open ring mark…

**Figure 9.** Figure 9: The fragility profile, illustrated on the prompt axis. Each bar is the deployment-side undercoverage rate Uτ of vanilla CP for one MCQA prompt template, pooled over (model,score, trial) and sorted from most to least fragile. coverage. Set-size inflation is essentially…

**Figure 11.** Figure 11: Cross-template coverage damage at prompt granularity. Each cell is one (calibration template, test template) pair, summarised over (model, score, trial); rows index the calibration template, columns the test template. Top row: under-coverage rate Uτ (darker red = more…

**Figure 12.** Figure 12: Cross-configuration coverage damage on the ordered shift axes. As [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

[1]

Surface format— vary how the answer options (MCQA) or question/answer prefixes (OEQA) are typeset; instructions and demonstration con- tent are held fixed

work page
[2]

InAdvances in Neu- ral Information Processing Systems (NeurIPS)

Large language model validity via enhanced conformal prediction methods. InAdvances in Neu- ral Information Processing Systems (NeurIPS). Nicolas Deutschmann, Marvin Alberts, and María Ro- dríguez Martínez. 2024. Conformal autoregressive generation: Beam search with coverage guarantees. InProceedings of the AAAI Conference on Artificial Intelligence, volu...

work page 2024
[3]

abstract algebra

Content ablation— remove exactly one scaf- folding component (subject preamble, chain-of- thought directive, or few-shot demos) from the anchor template. Baseline prompts are displayed below; non-anchor templates are specified as deltas from the baseline. The canonical templates that the model receives are stored in the per-dataset Y AML configuration fil...

work page 2024
[6]

Traq: Trustworthy retrieval augmented ques- tion answering via conformal prediction. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 3799–3821. Association for Computational Linguistics. Stephanie Lin, Jacob Hilton, and Owain...

work page 2024
[7]

Potsawee Manakul, Adian Liusie, and Mark Gales

Game-theoretic defenses for robust confor- mal prediction against adversarial attacks in medical imaging.arXiv preprint arXiv:2411.04376. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucina- tion detection for generative large language models. InProceedings of the 2023 Conference on Empiri- cal Methods in ...

arXiv 2023
[9]

InFindings of the Association for Computational Linguistics: ACL 2023, pages 27–34

Conformal nucleus sampling. InFindings of the Association for Computational Linguistics: ACL 2023, pages 27–34. Association for Computational Linguistics. Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majum- dar. 2023. R...

arXiv 2023
[10]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995

Api is enough: Conformal prediction for large language models without logit-access. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995. Association for Com- putational Linguistics. Llama Team. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafa...

arXiv 2024
[11]

InFindings of the Association for Computational Linguistics: EACL 2024, pages 1909–1929

Non-exchangeable conformal language gen- eration with nearest neighbors. InFindings of the Association for Computational Linguistics: EACL 2024, pages 1909–1929. Association for Computa- tional Linguistics. unsloth. 2025. gemma-3-12b-it-gguf. https://huggingface.co/unsloth/ gemma-3-12b-it-GGUF . Gemma GGUF quantized model repository. Cédric Villani. 2009....

work page 2024

Show all 17 references

[13]

InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19052– 19075

Sconu: Selective conformal uncertainty in large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19052– 19075. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan H...

2024 arXiv
[16]

Instruction syntax and wording— vary the ver- bosity, persona, and phrasing of the task instruc- tions; demonstrations and surface format are held fixed
[260]

Harris Papadopoulos, Kostas Proedrou, V olodya V ovk, and Alex Gammerman

PMLR. Harris Papadopoulos, Kostas Proedrou, V olodya V ovk, and Alex Gammerman. 2002. Inductive confidence machines for regression. InMachine Learning: ECML 2002, volume 2430 ofLecture Notes in Com- puter Science, pages 345–356. Springer. Aleksandr Podkopaev and Aaditya Ramdas...

2002
[1956]

António Farinhas, Chrysoula Zerva, Dennis Thomas Ulmer, and Andre Martins

Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator.The Annals of Mathematical Statistics, pages 642–669. António Farinhas, Chrysoula Zerva, Dennis Thomas Ulmer, and Andre Martins. 2024. Non-exchangeable conformal risk co...

2024 arXiv
[2019]

We use these artifacts only for research evaluation of uncertainty sets, consistent with their standard benchmark use, and do not col- lect new human data or create new labels

and TriviaQA (Joshi et al., 2017) are used as open-ended QA datasets with string-valued an- swers and aliases. We use these artifacts only for research evaluation of uncertainty sets, consistent with their standard benchmark use, and do not col- lect new human data or create n...

2017
[2021]

InInternational Conference on Learning Representations (ICLR)

Measuring massive multitask language under- standing. InInternational Conference on Learning Representations (ICLR). Zirui Hu, Zheng Zhang, Yingjie Wang, Leszek Rutkowski, and Dacheng Tao. 2026. Cofact: Con- formal factuality guarantees for language models un- der covariate sh...

2026 arXiv
[2023]

In ICML 2023 Workshop on Neural Conversational AI (TEACH)

Conformal prediction with large language models for multi-choice question answering. In ICML 2023 Workshop on Neural Conversational AI (TEACH). Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosu...

2023
[2024]

Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C

Conformal prediction for natural language pro- cessing: A survey.Transactions of the Association for Computational Linguistics, 12:1497–1516. Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C. Duchi. 2024. Robust validation: Confident predictions even when distributions shi...

2024
[2025]

InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, pages 61601–61634

Prune ’n predict: Optimizing llm decision- making with conformal prediction. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, pages 61601–61634. PMLR. Vladimir V ovk, Alexander Gammerman, and ...

2005

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Surface format— vary how the answer options (MCQA) or question/answer prefixes (OEQA) are typeset; instructions and demonstration con- tent are held fixed

work page

[2] [2]

InAdvances in Neu- ral Information Processing Systems (NeurIPS)

Large language model validity via enhanced conformal prediction methods. InAdvances in Neu- ral Information Processing Systems (NeurIPS). Nicolas Deutschmann, Marvin Alberts, and María Ro- dríguez Martínez. 2024. Conformal autoregressive generation: Beam search with coverage guarantees. InProceedings of the AAAI Conference on Artificial Intelligence, volu...

work page 2024

[3] [3]

abstract algebra

Content ablation— remove exactly one scaf- folding component (subject preamble, chain-of- thought directive, or few-shot demos) from the anchor template. Baseline prompts are displayed below; non-anchor templates are specified as deltas from the baseline. The canonical templates that the model receives are stored in the per-dataset Y AML configuration fil...

work page 2024

[4] [6]

Traq: Trustworthy retrieval augmented ques- tion answering via conformal prediction. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 3799–3821. Association for Computational Linguistics. Stephanie Lin, Jacob Hilton, and Owain...

work page 2024

[5] [7]

Potsawee Manakul, Adian Liusie, and Mark Gales

Game-theoretic defenses for robust confor- mal prediction against adversarial attacks in medical imaging.arXiv preprint arXiv:2411.04376. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucina- tion detection for generative large language models. InProceedings of the 2023 Conference on Empiri- cal Methods in ...

arXiv 2023

[6] [9]

InFindings of the Association for Computational Linguistics: ACL 2023, pages 27–34

Conformal nucleus sampling. InFindings of the Association for Computational Linguistics: ACL 2023, pages 27–34. Association for Computational Linguistics. Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majum- dar. 2023. R...

arXiv 2023

[7] [10]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995

Api is enough: Conformal prediction for large language models without logit-access. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995. Association for Com- putational Linguistics. Llama Team. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafa...

arXiv 2024

[8] [11]

InFindings of the Association for Computational Linguistics: EACL 2024, pages 1909–1929

Non-exchangeable conformal language gen- eration with nearest neighbors. InFindings of the Association for Computational Linguistics: EACL 2024, pages 1909–1929. Association for Computa- tional Linguistics. unsloth. 2025. gemma-3-12b-it-gguf. https://huggingface.co/unsloth/ gemma-3-12b-it-GGUF . Gemma GGUF quantized model repository. Cédric Villani. 2009....

work page 2024