Pith. sign in

REVIEW 3 major objections 6 minor 92 references

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM answer's stability under neutral, topic-aligned rewording reliably signals whether it is correct or factual.

desk verdict C3 is a promising, well-tested black-box consistency signal, but the paper's 'credibility' interpretation leans on a neutrality assumption it explicitly concedes it doesn't fully verify. read the letter →

arxiv 2608.10315 v1 pith:T33SUEZD submitted 2026-08-10 cs.CL cs.AI

Siyang Wu , Yibo Jiang , Bryon Aragam This is my paper · ORCID
classification cs.CLcs.AI
keywords cross-contextualconsistencyLLMcredibilitymaximummeandiscrepancypromptperturbationcalibrationfactualitydetectionbenchmarksaturationblack-boxevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a language model answer is credible to the degree that it stays stable when the same question is surrounded by neutral, topic-aligned extra context, and that this stability can be measured from the outside. It introduces Cross-Contextual Consistency (C3), a black-box score that compares the distribution of outputs sampled from the original prompt with the distribution under perturbed prompts, and reports that across 26 models and six benchmarks, higher C3 tracks correctness and factuality better than self-reported confidence, self-consistency, or paraphrasing consistency. If true, C3 gives practitioners a reference-free, label-free way to flag doubtful outputs without touching model internals, and it gives evaluators a second axis for spotting benchmark questions that are saturated, brittle, or systematically biased.

What carries the argument

The central object is C3, a normalized inverse distance between two empirical output distributions: $C3(x;E)=1-\frac{1}{2}\widehat{\mathrm{MMD}}^2(Y,Y^E)$, where $Y$ are samples from the original prompt, $Y^E$ are samples from prompts prefixed with topic-aligned, content-neutral perturbations, and MMD is a kernel-based two-sample distance with task-adaptive feature maps and kernels. The work it does is to translate the behavioral principle that well-supported answers are stable under cross-examination into a single number, and to give one comparison axis across multiple-choice, short-answer, long-form, and code tasks; an ablation replacing MMD with a simple cross-comparison agreement score preserves the signal, so the comparison itself, not the specific kernel, carries the result.

What would settle it

A decisive test would build a set of items in which each perturbed prompt has been independently verified to preserve the original correct answer, then measure C3's AUROC on that set; if wrong answers shift as little as right answers there, the reported correlation is an artifact of perturbations quietly changing the question.

Watch

Extended reading notes

Core claim

The paper's discovery is that cross-contextual consistency works as a proxy for truthfulness: an LLM that internally supports an answer should reproduce that answer when the prompt is prefixed with content-neutral contextual variation, whereas a hallucinated or weakly grounded answer should shift. Operationalizing this as $C3=1-\frac{1}{2}\widehat{\mathrm{MMD}}^2$ between original and perturbed output distributions, the paper reports that answers with smaller shifts are more likely to be correct across arithmetic reasoning, multiple-choice knowledge, short- and long-form factuality, and code generation, with AUROCs of 0.917 on SVAMP and 0.823 on SimpleQA. The paper also claims C3 doubles as a benchmark diagnostic: instances where models agree on the correct answer but have low C3 are labeled brittle and suspected of surface pattern matching or contamination, while low-accuracy, perturbation-invariant instances are labeled biased and attributed to shared misconceptions.

Load-bearing premise

The load-bearing premise is that every perturbation prefix is genuinely content-neutral and topic-aligned for the specific question it is attached to, so that any output shift reflects answer fragility rather than a changed task.

Editorial extensions

If this is right

  • A user can rank individual answers by fragility using only API sampling, with no ground-truth labels and no access to token probabilities.
  • Benchmark curators can mark question subsets as saturated, brittle, biased, or unlearned from the joint C3-and-performance scatter, giving a diagnostic that aggregate accuracy hides.
  • On hard factuality questions, C3 gives much better calibration than self-report (ECE 0.166 vs 0.778 on SimpleQA) and better discrimination than self-consistency (AUROC 0.823 vs 0.792).
  • The signal survives weaker perturbation sources: C3 with Qwen3-8B-generated noise reaches AUROC 0.833 on SimpleQA, and random web noise stays competitive, so the method does not depend on a frontier model to write the perturbations.
  • C3's alignment with correctness grows with model scale within a family, so residual cross-contextual fragility becomes a sharper marker of wrong answers as models get stronger.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because C3 needs no labels, it could be used during data curation or fine-tuning to rank answers by stability, a training-side use the paper does not explore.
  • Editorial inference: the aggregate neutrality check leaves open per-instance semantic drift, so the strongest version of the claim would require verifying that each perturbed prompt preserves the original correct answer.
  • Editorial inference: the brittle-region pattern is a testable contamination detector: questions many models answer correctly but unstably should show larger accuracy drops under paraphrasing, which the paper's interpretation predicts.
  • Editorial inference: the corpus-noise result suggests a fully offline, low-cost deployment without any frontier model, which the ablation supports but does not develop into a protocol.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Cross-Contextual Consistency (C3), a black-box behavioral metric that measures how stable an LLM's output distribution is when the same question is prefixed with topic-aligned, content-neutral contextual perturbations. C3 is operationalized as one minus half the squared Maximum Mean Discrepancy between original and perturbed output distributions. The authors evaluate C3 on 26 models and six benchmarks spanning math reasoning, multiple-choice reasoning, commonsense, short- and long-form factuality, and code generation. They report that C3 aligns with correctness and factuality better than vanilla baselines (self-report, self-consistency, paraphrasing consistency) on aggregate metrics such as AUROC and ECE, and they propose C3 as a benchmark diagnostic that separates 'mastered', 'brittle', 'biased', and 'unlearned' instances.

Significance. If the central claim holds, C3 is a practically valuable reference-free, label-free, black-box proxy for answer credibility, applicable to closed models and free-form generation where token-probability methods fail. The paper's strengths include very extensive per-model tables, ablations on perturbation source (GPT-4.1, Qwen3-8B, web noise) and on the distance metric (MMD vs. cross-comparison), and a clear presentation of the method's computational cost. The claim is falsifiable and the protocol is reproducible in principle. However, the interpretation of C3 as a credibility measure depends critically on the perturbations being content-neutral at the instance level, and the current neutrality evidence is aggregate and explicitly conceded to be incomplete. The reported aggregate wins also lack paired significance testing, and several per-model cells show C3 below baselines, so the headline 'consistently outperforms' is not yet statistically grounded.

major comments (3)
  1. [§3, Appendix A.3] The load-bearing assumption that perturbations are content-neutral is verified only as an aggregate correctness delta on MMLU High School Statistics, and the manuscript itself states that this 'does not prove that every individual perturbation is perfectly neutral'. Aggregate near-zero delta is compatible with large per-instance semantic shifts: on SVAMP, an added premise can introduce a new quantity or change the referent of 'that day', thereby altering the task rather than the context. Under that possibility, high C3 could reflect stability under a changed task and low C3 could reflect task change, so the reported AUROC gains (e.g., 0.917 on SVAMP in Table 1) would be measuring prompt sensitivity rather than credibility. I ask for a per-instance neutrality audit on a sampled subset of each benchmark (human annotation or a stronger judge with explicit answer-relevance criteria), and a demonstration that the C3-correctness correlation survives when restricted to instances whose perturbations are verified neutral. Without this, the central interpretive claim is not established.
  2. [Tables 1, 2, 6] The aggregate scores in Table 1 are averages over 16 models, but no confidence intervals, bootstrap estimates, or paired significance tests are reported. The per-model tables show that C3 is not uniformly better than baselines: for example, on MMLU High School Stats, llama-3.1-70b-instruct has C3 AUROC 0.549 versus self-consistency 0.598, and on SimpleQA, gemini-2.5-flash has C3 AUROC 0.827 versus self-consistency 0.838. Because the paper's headline claim is that C3 'consistently outperforms' baselines, the authors should report paired tests (e.g., Wilcoxon signed-rank over models) or bootstrap confidence intervals for the AUROC/ECE differences, and should quantify how many models show improvement per benchmark.
  3. [Appendix H] The normalization C3 = 1 - 0.5 * MMD^2 relies on the assumptions that the kernel is bounded with k(y,y)=1 and 0 <= k(y,y') <= 1, giving MMD^2 <= 2. For open-ended generations the paper suggests using cosine similarity, which can take negative values and is not necessarily nonnegative; in that case the stated upper bound and the [0,1] interpretation of C3 do not follow. The authors should either restrict the kernel to a nonnegative normalized similarity (e.g., ReLU-cosine or a nonnegative kernel) or derive the correct normalization for the cosine kernel actually used in the experiments.
minor comments (6)
  1. [§5.2] The heading 'A closer inspection on facutality' contains a typo; it should read 'factuality'.
  2. [§6] In the related work section, 'when when paraphrased the quesiton' contains a duplicated word and a typo; it should read 'when the question is paraphrased'.
  3. [Throughout] The benchmark name is written inconsistently as both 'SVAMP' and 'SV AMP'; please standardize.
  4. [§3, Appendix H] The distance notation is inconsistent: the main text uses eD, while Appendix H uses D and MMD^2; please unify.
  5. [Appendix D.5] The judge prompt shown in D.5 is specific to code completions ('Compare two code completions'), but the text says this judge is used for all benchmarks; please clarify whether the same prompt was reused for non-code tasks or provide the task-adapted versions.
  6. [§5.3 and Figure 4] The four-region partition is based on medians of the same data used to interpret the regions; this is acceptable as a visualization, but the text should avoid causal language such as 'supports the hypothesis of benchmark leakage' without a more direct contamination measure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: C3 is a fixed, label-free distributional comparison; the only overlapping-author citation is motivational and not load-bearing.

full rationale

Close reading of the derivation chain finds no step in which a prediction reduces to an input by construction. C3 is defined in Section 3 and Appendix H as C3(x,x') = 1 - 0.5 * MMD^2(Y,Y'), a fixed function of two sampled output sets; the normalization constant 0.5 comes from the theoretical bound MMD^2 <= 2 for normalized kernels, not from any fit to correctness labels. The sampling choices (n=30, k=0.8124) are chosen for statistical stability and semantic diversity, not to optimize the reported AUROC or ECE values. Perturbations are generated externally by GPT-4.1, and Appendix F shows that Qwen3-8B and web-sourced noise preserve the signal, so the correlation with correctness is not forced by the perturbation source. The only overlapping-author citation (Jiang et al. 2024b, used for the 'context hijacking' motivation) is not load-bearing: the paper's central empirical evaluation is self-contained and compares C3 against external correctness labels on six benchmarks. Appendix A.3 explicitly concedes that 'This analysis does not prove that every individual perturbation is perfectly neutral'; this is a validity limitation for per-instance semantic neutrality, but it does not make C3's definition circular, since the metric never receives correctness labels. No fitted-parameter-renamed-as-prediction, uniqueness-imported-from-authors, or ansatz-smuggled-via-citation pattern is present. The verdict is therefore a non-finding of circularity, with the neutrality limitation noted as a correctness/robustness concern rather than a circularity concern.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

C3's pipeline introduces two numeric hyperparameters (diversity threshold k=0.8124 and trial count n=30) plus a decoding temperature and a label threshold, none fitted to the target correctness correlation. The load-bearing domain assumptions are perturbation neutrality and topic alignment, a reliable answer-equivalence judge, and accurate benchmark labels; each is only partially verified. No physical or latent entities are introduced with independent evidence; the Section 2 'worlds' formalism is purely expository.

free parameters (4)
  • diversity threshold k = 0.8124
    Cosine-similarity cutoff for accepting generated perturbations; fitted to the 5th percentile of GPT-4.1 paraphrase similarities (alpha=0.05), Appendix A.2. Not fitted to correctness, but a hand or empirically chosen hyperparameter of the pipeline.
  • number of sampling trials n = 30
    Chosen conservatively after a stabilization study (Appendix B) showing behavior stabilizes at 20; a design choice, not fitted to the target correlation.
  • temperature T = 1.0
    Standard stochastic decoding temperature used across all experiments (Appendix E.1); a deliberate choice to probe typical generation behavior.
  • correctness label threshold = 0.5
    Per-instance labels yi=1[perf>=0.5] used for AUROC and AUPRC; this threshold choice affects all reported ranking metrics (Appendix E.2).
assumptions (5)
  • domain assumption LLM output distributions P(Y|x) and P_E(Y|x) are well-defined and estimable from 30 samples.
    Section 3 defines C3 via these distributions; Appendix B justifies 30 trials from a stabilization study, but there is no verification that 30 samples capture rare-output modes, and the distributions are not stationary under temperature-1 decoding.
  • domain assumption Perturbation prefixes are content-neutral and topic-aligned for all six benchmarks.
    Section 3 lists neutrality and topic alignment as requirements; Appendix A.3 checks neutrality only as an aggregate correctness delta on MMLU High School Statistics, and Appendix A.4 checks topic alignment via a single judge; per-instance neutrality is unverified.
  • domain assumption The answer-equivalence judge (Qwen3-8B) correctly identifies semantic equivalence across task formats.
    Appendix D.5 supplies the judge prompt, which is written for code completions yet applied to all benchmarks; judge errors transfer directly to the MMD and cross-comparison estimates.
  • domain assumption Benchmark correctness labels (including SimpleQA Verified answers and HumanEval unit tests) are accurate ground truth.
    All AUROC and ECE targets are defined by these labels; any label noise or ambiguity propagates into the reported alignment scores.
  • standard math Bounded normalized kernel property k(y,y)=1 and 0<=k<=1 used to derive the MMD upper bound of 2.
    Appendix H uses this to justify the C3 normalization C3 = 1 - 0.5*MMD^2; valid for indicator and cosine kernels, but the specific open-ended feature map is not fully specified.
invented entities (1)
  • Probabilistic worlds v over the knowledge base Gamma
    purpose: Motivational framework in Section 2 that frames consistency as stochastic entailment P(f |= g) and justifies C3 as a proxy.
    This formalism is never estimated; C3 computes empirical distributional distances instead of world probabilities, so the worlds are a rhetorical device rather than a measured construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility." pith.science (2026). https://pith.science/paper/T33SUEZD

@misc{pith2026260810315,
  author       = {Pith},
  title        = {Pith review of: Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T33SUEZD}},
  note         = {Machine review of arXiv:2608.10315}
}
read the original abstract

Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered "saturate".

Figures

Figures reproduced from arXiv: 2608.10315 by the authors.

Figure 1
Figure 1. Overview of the proposed workflow. From left to right, we sample generations from the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Scaling trends of Cross-Contextual Con￾sistency (C3) calibration across benchmarks. Col￾ors distinguish different benchmarks, while line styles represent metric types: solid lines denote AUROC (higher is better) and dashed lines de￾note ECE (lower is better). The results show that as model scale increases, the C3 becomes signifi￾cantly better calibrated to correctness, evidenced by rising AUROC and declining ECE. C3… view at source ↗
Figure 3
Figure 3. A detailed comparison of C3 alignment with factuality on FActScore and SimpleQA, show￾ing Spearman rank correlation (left) and AUROC (right). C3 exhibits moderate to strong rank corre￾lation with generation factuality. Minor AUROC differences from [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: For each benchmark instance, we com￾pute the mean performance and mean C3 across the 16 models, then partition instances into four regions using the median performance and me￾dian C3 computed over instances across the six benchmarks. Green (“mastered”) indicates high p…
Figure 5
Figure 5. Figure 5: The distribution of pairwise cosine similarities among GPT-4.1-generated paraphrases. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: The difference of performance of each model before and after the perturbation noises are [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: The performance of models across number of trials of samples we collected when perturba [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 19 canonical work pages

  1. [1]

    Miao Xiong and Zhiyuan Hu and Xinyang Lu and YIFEI LI and Jie Fu and Junxian He and Bryan Hooi , booktitle=. Can. 2024 , url=

    2024

  2. [2]

    Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models

    Kumar, Abhishek and Morabito, Robert and Umbet, Sanzhar and Kabbara, Jad and Emami, Ali. Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.20

    doi:10.18653/v1/2024.acl-long.20 2024

  3. [3]

    Calibrating the Confidence of Large Language Models by Eliciting Fidelity

    Zhang, Mozhi and Huang, Mianqiu and Shi, Rundong and Guo, Linsen and Peng, Chong and Yan, Peng and Zhou, Yaqian and Qiu, Xipeng. Calibrating the Confidence of Large Language Models by Eliciting Fidelity. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.173

    doi:10.18653/v1/2024.emnlp-main.173 2024

  4. [4]

    Juyeon Heo and Miao Xiong and Christina Heinze-Deml and Jaya Narain , booktitle=. Do. 2025 , url=

    2025

  5. [5]

    Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios

    Lin, Lei and Fu, Jiayi and Liu, Pengli and Li, Qingyang and Gong, Yan and Wan, Junchen and Zhang, Fuzheng and Wang, Zhongyuan and Zhang, Di and Gai, Kun. Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.fin...

    work page doi:10.18653/v1/2024.findings-acl.230 2024

  6. [6]

    Mirror-Consistency: Harnessing Inconsistency in Majority Voting

    Huang, Siyuan and Ma, Zhiyuan and Du, Jintao and Meng, Changhua and Wang, Weiqiang and Lin, Zhouhan. Mirror-Consistency: Harnessing Inconsistency in Majority Voting. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.135

    doi:10.18653/v1/2024.findings-emnlp.135 2024

  7. [7]

    The Consensus Game: Language Model Generation via Equilibrium Search , url =

    Jacob, Athul and Shen, Yikang and Farina, Gabriele and Andreas, Jacob , booktitle =. The Consensus Game: Language Model Generation via Equilibrium Search , url =

  8. [8]

    Le and Ed H

    Xuezhi Wang and Jason Wei and Dale Schuurmans and Quoc V. Le and Ed H. Chi and Sharan Narang and Aakanksha Chowdhery and Denny Zhou , title =. The Eleventh International Conference on Learning Representations,. 2023 , url =

    2023

Show all 92 references
  1. [9]

    Confidence Improves Self-Consistency in LLM s

    Taubenfeld, Amir and Sheffer, Tom and Ofek, Eran and Feder, Amir and Goldstein, Ariel and Gekhman, Zorik and Yona, Gal. Confidence Improves Self-Consistency in LLM s. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.1030

    2025 doi

  2. [10]

    The Eleventh International Conference on Learning Representations , year=

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author=. The Eleventh International Conference on Learning Representations , year=

  3. [11]

    SPUQ : Perturbation-Based Uncertainty Quantification for Large Language Models

    Gao, Xiang and Zhang, Jiaxin and Mouatadid, Lalla and Das, Kamalika. SPUQ : Perturbation-Based Uncertainty Quantification for Large Language Models. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024 doi

  4. [12]

    Semantic Entropy Probes: Robust and Cheap Hallucination Detection in

    Jiatong Han and Jannik Kossen and Muhammed Razzak and Lisa Schut and Shreshth A Malik and Yarin Gal , booktitle=. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in. 2024 , url=

    2024

  5. [13]

    State of What Art? A Call for Multi-Prompt LLM Evaluation

    Mizrahi, Moran and Kaplan, Guy and Malkin, Dan and Dror, Rotem and Shahaf, Dafna and Stanovsky, Gabriel. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics. 2024. doi:10.1162/tacl_a_00681

    2024 doi

  6. [14]

    Proceedings of the AAAI Conference on Artificial Intelligence , author=

    Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2025 , month=. doi:10.1609/aaai.v39i22.34540 , abstractNote=

    2025 doi

  7. [15]

    2024 , eprint=

    Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning , author=. 2024 , eprint=

    2024

  8. [16]

    Calibrating Language Models with Adaptive Temperature Scaling

    Xie, Johnathan and Chen, Annie S and Lee, Yoonho and Mitchell, Eric and Finn, Chelsea. Calibrating Language Models with Adaptive Temperature Scaling. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.1007

    2024 doi

  9. [17]

    The Twelfth International Conference on Learning Representations , year=

    Conformal Language Modeling , author=. The Twelfth International Conference on Learning Representations , year=

  10. [18]

    Nature , year =

    Detecting hallucinations in large language models using semantic entropy , author =. Nature , year =. doi:10.1038/s41586-024-07421-0 , url =

    doi

  11. [19]

    Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration , url =

    Kull, Meelis and Perello Nieto, Miquel and K\". Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration , url =. Advances in Neural Information Processing Systems , editor =

  12. [20]

    and Tay, Yi and Metzler, Donald , title =

    Schuster, Tal and Fisch, Adam and Gupta, Jai and Dehghani, Mostafa and Bahri, Dara and Tran, Vinh Q. and Tay, Yi and Metzler, Donald , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    2022

  13. [21]

    Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding

    Bae, Sangmin and Ko, Jongwoo and Song, Hwanjun and Yun, Se-Young. Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:1...

    2023 doi

  14. [22]

    Fast yet Safe: Early-Exiting with Risk Control , url =

    Jazbec, Metod and Timans, Alexander and Veljkovi\'. Fast yet Safe: Early-Exiting with Risk Control , url =. Advances in Neural Information Processing Systems , editor =

  15. [23]

    The Twelfth International Conference on Learning Representations , year=

    Bayesian Low-rank Adaptation for Large Language Models , author=. The Twelfth International Conference on Learning Representations , year=

  16. [24]

    BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models , url =

    Wang, Yibin and Shi, Haizhou and Han, Ligong and Metaxas, Dimitris and Wang, Hao , booktitle =. BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models , url =

  17. [25]

    Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets

    Li, Dongyue and Zhang, Ziniu and Wang, Lu and Zhang, Hongyang R. Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025...

    2025 doi

  18. [26]

    The Twelfth International Conference on Learning Representations , year=

    Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation , author=. The Twelfth International Conference on Learning Representations , year=

  19. [27]

    Qinyuan Cheng and Tianxiang Sun and Xiangyang Liu and Wenwei Zhang and Zhangyue Yin and Shimin Li and Linyang Li and Zhengfu He and Kai Chen and Xipeng Qiu , booktitle=. Can. 2024 , url=

    2024

  20. [28]

    Spotting

    Abhimanyu Hans and Avi Schwarzschild and Valeriia Cherepanova and Hamid Kazemi and Aniruddha Saha and Micah Goldblum and Jonas Geiping and Tom Goldstein , booktitle=. Spotting. 2024 , url=

    2024

  21. [29]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Thermometer: Towards Universal Calibration for Large Language Models , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    2024

  22. [30]

    2024 , url=

    Chao Chen and Kai Liu and Ze Chen and Yi Gu and Yue Wu and Mingyuan Tao and Zhihang Fu and Jieping Ye , booktitle=. 2024 , url=

    2024

  23. [31]

    LLM-Check: Investigating Detection of Hallucinations in Large Language Models , url =

    Sriramanan, Gaurang and Bharti, Siddhant and Sadasivan, Vinu Sankar and Saha, Shoumik and Kattakinda, Priyatham and Feizi, Soheil , booktitle =. LLM-Check: Investigating Detection of Hallucinations in Large Language Models , url =

  24. [32]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Calibrating Reasoning in Language Models with Internal Consistency , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  25. [33]

    2024 , url=

    Elias Stengel-Eskin and Peter Hase and Mohit Bansal , booktitle=. 2024 , url=

    2024

  26. [34]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Language Models with Conformal Factuality Guarantees , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    2024

  27. [35]

    Lamb and Desi R

    Tom A. Lamb and Desi R. Ivanova and Philip Torr and Tim G. J. Rudner , booktitle=. Semantic Calibration of. 2025 , url=

    2025

  28. [36]

    ZhongXiang Sun and Xiaoxue Zang and Kai Zheng and Jun Xu and Xiao Zhang and Weijie Yu and Yang Song and Han Li , booktitle=. ReDe. 2025 , url=

    2025

  29. [37]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  30. [38]

    Paraphrase Types Elicit Prompt Engineering Capabilities

    Wahle, Jan Philip and Ruas, Terry and Xu, Yang and Gipp, Bela. Paraphrase Types Elicit Prompt Engineering Capabilities. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.617

    2024 doi

  31. [39]

    Howie and Boix-Adser \`a , Enric

    Melamed, Rimon and McCabe, Lucas Hurley and Wakhare, Tanay and Kim, Yejin and Huang, H. Howie and Boix-Adser \`a , Enric. Prompts have evil twins. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.4

    2024 doi

  32. [40]

    A Critical Evaluation of Evaluations for Long-form Question Answering

    Xu, Fangyuan and Song, Yixiao and Iyyer, Mohit and Choi, Eunsol. A Critical Evaluation of Evaluations for Long-form Question Answering. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.a...

    2023 doi

  33. [41]

    and Kry \'s ci \'n ski, Wojciech and McCann, Bryan and Xiong, Caiming and Socher, Richard and Radev, Dragomir

    Fabbri, Alexander R. and Kry \'s ci \'n ski, Wojciech and McCann, Bryan and Xiong, Caiming and Socher, Richard and Radev, Dragomir. S umm E val: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics. 2021. doi:10.1162/tacl_a_00373

    2021 doi

  34. [42]

    Humans or LLM s as the Judge? A Study on Judgement Bias

    Chen, Guiming Hardy and Chen, Shunian and Liu, Ziche and Jiang, Feng and Wang, Benyou. Humans or LLM s as the Judge? A Study on Judgement Bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.474

    2024 doi

  35. [43]

    R -Tuning: Instructing Large Language Models to Say ` I Don ' t Know'

    Zhang, Hanning and Diao, Shizhe and Lin, Yong and Fung, Yi and Lian, Qing and Wang, Xingyao and Chen, Yangyi and Ji, Heng and Zhang, Tong. R -Tuning: Instructing Large Language Models to Say ` I Don ' t Know'. Proceedings of the 2024 Conference of the North American Chapter of...

    2024 doi

  36. [44]

    2025 , eprint=

    On Robustness and Reliability of Benchmark-Based Evaluation of LLMs , author=. 2025 , eprint=

    2025

  37. [45]

    arXiv preprint arXiv:2004.09297 , year=

    MPNet: Masked and Permuted Pre-training for Language Understanding , author=. arXiv preprint arXiv:2004.09297 , year=

    2004 arXiv

  38. [46]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  39. [47]

    International Conference on Learning Representations , year=

    The Curious Case of Neural Text Degeneration , author=. International Conference on Learning Representations , year=

  40. [48]

    2025 , eprint=

    A mathematical perspective on Transformers , author=. 2025 , eprint=

    2025

  41. [49]

    Borgwardt and Malte J

    Arthur Gretton and Karsten M. Borgwardt and Malte J. Rasch and Bernhard Sch. A Kernel Two-Sample Test , journal =. 2012 , volume =

    2012

  42. [50]

    2025 , eprint=

    SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge , author=. 2025 , eprint=

    2025

  43. [51]

    FA ct S core: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh. FA ct S core: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Proceedings of the 202...

    2023 doi

  44. [52]

    Evaluating Large Language Models Trained on Code , journal =

    Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Pond. Evaluating Large Language Models Trained on Code , journal =. 2021 , url =. 2107.03374 , timestamp =

    2021 arXiv

  45. [53]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  46. [54]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Aligning AI With Shared Human Values , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  47. [55]

    Are NLP Models really able to Solve Simple Math Word Problems?

    Patel, Arkil and Bhattamishra, Satwik and Goyal, Navin. Are NLP Models really able to Solve Simple Math Word Problems?. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021. doi:10....

    2021 doi

  48. [56]

    C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge

    Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan. C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Hu...

    2019 arXiv

  49. [57]

    Transactions on Machine Learning Research , issn=

    Teaching Models to Express Their Uncertainty in Words , author=. Transactions on Machine Learning Research , issn=. 2022 , url=

    2022

  50. [58]

    2025 , eprint=

    Assessing Correctness in LLM-Based Code Generation via Uncertainty Estimation , author=. 2025 , eprint=

    2025

  51. [59]

    2025 , eprint=

    Humans overrely on overconfident language models, across languages , author=. 2025 , eprint=

    2025

  52. [60]

    and Szlam, Arthur and Dinan, Emily and Boureau, Y-Lan

    Mielke, Sabrina J. and Szlam, Arthur and Dinan, Emily and Boureau, Y-Lan. Reducing Conversational Agents' Overconfidence Through Linguistic Calibration. Transactions of the Association for Computational Linguistics. 2022. doi:10.1162/tacl_a_00494

    2022 doi

  53. [61]

    2025 , eprint=

    Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework , author=. 2025 , eprint=

    2025

  54. [62]

    PAWS : Paraphrase Adversaries from Word Scrambling

    Zhang, Yuan and Baldridge, Jason and He, Luheng. PAWS : Paraphrase Adversaries from Word Scrambling. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers...

    2019 doi

  55. [63]

    Investigating Data Contamination in Modern Benchmarks for Large Language Models

    Deng, Chunyuan and Zhao, Yilun and Tang, Xiangru and Gerstein, Mark and Cohan, Arman. Investigating Data Contamination in Modern Benchmarks for Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguist...

    2024 doi

  56. [64]

    Li, Yucheng and Guerin, Frank and Lin, Chenghua , title =. Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artifici...

    2024 doi

  57. [65]

    A Careful Examination of Large Language Model Performance on Grade School Arithmetic , url =

    Zhang, Hugh and Da, Jeff and Lee, Dean and Robinson, Vaughn and Wu, Catherine and Song, Will and Zhao, Tiffany and Raja, Pranav and Zhuang, Charlotte and Slack, Dylan and Lyu, Qin and Hendryx, Sean and Kaplan, Russell and Lunati, Michele and Yue, Summer , booktitle =. A Carefu...

    doi

  58. [66]

    2025 , eprint=

    GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models , author=. 2025 , eprint=

    2025

  59. [67]

    Transactions on Machine Learning Research , issn=

    Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics , author=. Transactions on Machine Learning Research , issn=. 2025 , url=

    2025

  60. [68]

    and Tram\`

    Chao, Patrick and Debenedetti, Edoardo and Robey, Alexander and Andriushchenko, Maksym and Croce, Francesco and Sehwag, Vikash and Dobriban, Edgar and Flammarion, Nicolas and Pappas, George J. and Tram\`. JailbreakBench: an open robustness benchmark for jailbreaking large lang...

  61. [69]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

    Cao, Bowen and Cai, Deng and Zhang, Zhisong and Zou, Yuexian and Lam, Wai , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  62. [70]

    Beyond Accuracy: Behavioral Testing of NLP Models with C heck L ist

    Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer. Beyond Accuracy: Behavioral Testing of NLP Models with C heck L ist. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.442

    2020 doi

  63. [71]

    Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement

    Portillo Wightman, Gwenyth and Delucia, Alexandra and Dredze, Mark. Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement. Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023). 2023. doi:10.18653/v1/2023.t...

    2023 doi

  64. [72]

    Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models

    Riddell, Martin and Ni, Ansong and Cohan, Arman. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024...

    2024 doi

  65. [73]

    Collins and Arka Pal and Umang Bhatt and Adrian Weller and Samuel Dooley and Micah Goldblum and Andrew Gordon Wilson , booktitle=

    Sanyam Kapoor and Nate Gruver and Manley Roberts and Katherine M. Collins and Arka Pal and Umang Bhatt and Adrian Weller and Samuel Dooley and Micah Goldblum and Andrew Gordon Wilson , booktitle=. Large Language Models Must Be Taught to Know What They Don. 2024 , url=

    2024

  66. [74]

    I Can't Believe It's Not Better: Failure Modes in the Age of Foundation Models

    Self-Evaluation Improves Selective Generation in Large Language Models , author =. Proceedings on "I Can't Believe It's Not Better: Failure Modes in the Age of Foundation Models" at NeurIPS 2023 Workshops , pages =. 2023 , editor =

    2023

  67. [75]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  68. [76]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  69. [77]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  70. [78]

    2025 , eprint=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=

    2025

  71. [79]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  72. [80]

    2024 , eprint=

    Mixtral of Experts , author=. 2024 , eprint=

    2024

  73. [81]

    2025 , eprint=

    OpenAI GPT-5 System Card , author=. 2025 , eprint=

    2025

  74. [82]

    2020 , eprint=

    Language Models are Few-Shot Learners , author=. 2020 , eprint=

    2020

  75. [83]

    2024 , eprint=

    GPT-4o System Card , author=. 2024 , eprint=

    2024

  76. [84]

    2025 , eprint=

    Gemini: A Family of Highly Capable Multimodal Models , author=. 2025 , eprint=

    2025

  77. [85]

    The Reversal Curse:

    Lukas Berglund and Meg Tong and Maximilian Kaufmann and Mikita Balesni and Asa Cooper Stickland and Tomasz Korbak and Owain Evans , booktitle=. The Reversal Curse:. 2024 , url=

    2024

  78. [86]

    Advances in Neural Information Processing Systems , volume=

    Do llms dream of elephants (when told not to)? latent concept association and associative memory in transformers , author=. Advances in Neural Information Processing Systems , volume=

  79. [87]

    Do We Know What LLM s Don ' t Know? A Study of Consistency in Knowledge Probing

    Zhao, Raoyuan and K. Do We Know What LLM s Don ' t Know? A Study of Consistency in Knowledge Probing. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.1263

    2025 doi

  80. [88]

    POSIX : A Prompt Sensitivity Index For Large Language Models

    Chatterjee, Anwoy and Renduchintala, H S V N S Kowndinya and Bhatia, Sumit and Chakraborty, Tanmoy. POSIX : A Prompt Sensitivity Index For Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.852

    2024 doi

  81. [89]

    What Did I Do Wrong? Quantifying LLM s' Sensitivity and Consistency to Prompt Engineering

    Errica, Federico and Sanvito, Davide and Siracusano, Giuseppe and Bifulco, Roberto. What Did I Do Wrong? Quantifying LLM s' Sensitivity and Consistency to Prompt Engineering. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compu...

    2025 doi

  82. [90]

    2024 , eprint=

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting , author=. 2024 , eprint=

    2024

  83. [91]

    Adaptation with Self-Evaluation to Improve Selective Prediction in LLM s

    Chen, Jiefeng and Yoon, Jinsung and Ebrahimi, Sayna and Arik, Sercan and Pfister, Tomas and Jha, Somesh. Adaptation with Self-Evaluation to Improve Selective Prediction in LLM s. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023....

    2023 doi

  84. [92]

    Together Computer , title =

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.