Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Shapley Uncertainty in Natural Language Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Shapley-based uncertainty, built from graded semantic correlations between sampled answers, predicts when an LLM is wrong more accurately than semantic entropy and satisfies three properties the paper argues any valid uncertainty metric…

desk verdict The Shapley decomposition sums to the very differential entropy it claims to replace, so the two central propositions cannot both hold; the kernel idea is fine but the theory needs to be rebuilt. read the letter →

arxiv 2507.21406 v1 pith:QYRXETIO submitted 2025-07-29 cs.AI

Meilin Zhu , Gaojie Jin , Xiaowei Huang , Lijun Zhang This is my paper
classification cs.AI
keywords ShapleyvalueuncertaintyquantificationsemanticentropynaturallanguagegenerationlargemodelsquestionansweringmachinetranslationAUROC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

To decide when a language model's answer can be trusted, this paper proposes replacing the binary threshold of semantic entropy with a continuous score computed from all pairwise semantic correlations among sampled answers. The score, called Shapley uncertainty, is built by turning those correlations into a positive semi-definite matrix, treating the answer set as a multivariate Gaussian with that matrix as its covariance, and distributing the Gaussian entropy over individual answers with the Shapley value. The paper claims this metric satisfies three properties — minimal uncertainty, maximal uncertainty, and consistency — that it argues any valid uncertainty metric should have, and reports that it predicts answer correctness with higher AUROC than semantic entropy and several other baselines across question-answering and machine-translation benchmarks. If correct, the result is a threshold-free, theoretically motivated trust signal for black-box LLMs that needs only sampled outputs and an entailment model.

What carries the argument

The load-bearing object is the correlation matrix built from bidirectional entailment probabilities, together with the kernel variant $K(C(s_i, s_j)) = 1$ for $i = j$ and $K(C(s_i, s_j)) = \beta\kappa(1 - C(s_i, s_j))$ otherwise, whose role is to guarantee positive semi-definiteness so the matrix can serve as the covariance of a multivariate Gaussian. The Shapley decomposition then averages, over every subset of the other answers, the marginal contribution of each sentence to the Gaussian differential entropy, with the total uncertainty defined as the sum of the elementary contributions. This machinery converts the thresholding problem — deciding whether two answers share a meaning — into a continuous, fully specified score, and it spreads the entropy across dimensions so that one pair of near-duplicate answers cannot drive the metric to negative infinity.

What would settle it

Because Shapley values sum to the value of the full set, the total Shapley uncertainty of any answer set equals the differential entropy of the same Gaussian correlation matrix — exactly the quantity Proposition 3.5 says fails Properties 3.2 and 3.3. A numerical check settles it: take a three-answer correlation matrix, add a perfectly correlated duplicate as $g$, compute every marginal entropy difference in equation (7), verify the efficiency sum, and test whether replacing each sentence with $g$ actually lowers the total. No language model is required for this.

Watch

Extended reading notes

Core claim

The paper's central claim is that uncertainty in natural language generation is measured better through the graded correlations between generated sentences than through a hard partition into same-meaning clusters. It defines a correlation entry $C(s_i, s_j \mid x) = \frac{1}{2}P(s_i \Rightarrow s_j \mid x) + \frac{1}{2}P(s_j \Rightarrow s_i \mid x)$ for each pair of answers, applies a kernel $K(C(s_i, s_j)) = 1$ on the diagonal and $\beta\kappa(1 - C(s_i, s_j))$ off it to force the matrix to be positive semi-definite, and then treats the $n$ answers as a multivariate Gaussian with that correlation matrix. Each sentence receives an elementary uncertainty equal to its Shapley contribution to the Gaussian differential entropy, and the total Shapley uncertainty is the sum of these contributions. The paper states that plain differential entropy fails two of its three proposed properties while Shapley uncertainty satisfies all three, and it reports AUROC results showing the new score predicts whether a generated answer is correct more accurately than semantic entropy, predictive entropy, length-normalized entropy, lexical similarity, and confidence elicitation across more than twenty model-dataset combinations.

Load-bearing premise

The theoretical result rests on one unproven step in the proof of Proposition 3.6: replacing an answer with a duplicate of the least-uncertain answer is assumed never to raise the uncertainty attributed to any other answer, and if that inequality fails, the three properties do not follow.

Editorial extensions

If this is right

  • Threshold tuning for semantic equivalence disappears: the only free choice is the kernel scale $\beta$, fixed at 0.5 in the paper.
  • On the reported benchmarks the score flags wrong answers earlier, e.g. AUROC of 0.739 versus 0.674 for semantic entropy and 0.515 for confidence elicitation on CoQA with Gemma-7B.
  • The three properties give a checklist that any future uncertainty metric can be tested against, independent of this particular construction.
  • Because the method needs only sampled outputs plus an entailment model, it applies to black-box LLMs with no access to logits or hidden states.
  • The reported gap over baselines widens for smaller models (e.g. +7.9% over naive entropy on NQ-Open with Falcon-7B), so the metric helps most where the model itself is weakest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • By Shapley-value efficiency the total Shapley uncertainty of an answer set is mathematically identical to the differential entropy of the same Gaussian, so the paper's two axiomatic claims — that plain entropy fails the properties while Shapley uncertainty satisfies them — cannot both stand; a reader can verify this identity directly and decide which side of the proof to discount.
  • The empirical AUROC gains are separable from the axiomatic proof: even a failed proof would not erase the reported accuracy gains, which are the part of the paper a practitioner actually needs.
  • The construction is a generic recipe — sample outputs, build graded pairwise similarity, repair positive semi-definiteness, decompose entropy with Shapley values — so the same score could be tested for hallucination detection in code generation, summarization, or retrieval-augmented generation without any change of method.
  • The Gaussian mapping is an assumption, not a consequence of how text is generated, so the metric's absolute values are uncalibrated; only its ranking behaviour, as measured by AUROC, is directly supported by the experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a new uncertainty metric for natural language generation called "Shapley uncertainty." Given n sampled output sentences, it constructs a correlation matrix from bidirectional entailment probabilities, applies a Gaussian kernel to make the matrix positive semi-definite, and models the sentences as a multivariate Gaussian with that correlation matrix. It then defines each sentence's uncertainty contribution as its Shapley value in a cooperative game whose value function is the differential entropy of the Gaussian restricted to subsets of sentences, and defines the total Shapley uncertainty as the sum of these contributions. The paper states three properties (minimal uncertainty, maximal uncertainty, consistency), claims that differential entropy violates the first two, claims that Shapley uncertainty satisfies all three, and reports AUROC experiments on question answering and machine translation across several models and datasets.

Significance. If the central theoretical and empirical claims held, the paper would offer a plausible, threshold-free alternative to semantic entropy, and the breadth of the experimental evaluation is a genuine strength: the manuscript covers many datasets (CoQA, TriviaQA, WMT-14, SQuAD, BioASQ, NQ-Open, SV AMP) and many model families (OPT, Gemma, LLaMA, Falcon, Mistral, Qwen, DeepSeek), and it provides Algorithm 1 plus an explicit ablation for the kernel hyperparameter. However, the central theoretical claim is internally inconsistent: by Shapley-value efficiency, the total Shapley uncertainty defined in Eq. (8) is identically equal to the differential entropy of Eq. (6), so Proposition 3.5 and Proposition 3.6 cannot both be true. The proof of Proposition 3.6 in Appendix A asserts the very inequalities that need to be proved. Because the paper's headline contribution is the claimed theoretical justification of the new metric, this flaw is decisive.

major comments (4)
  1. [§3.3–3.4, Eqs. (7)–(8) and Propositions 3.5–3.6] The central claim is internally inconsistent. For the cooperative game v(S) = h([s_j]_{j∈S}) used in Eq. (7), the standard efficiency property of Shapley values gives Σ_i φ(s_i|˜s) = v({1,...,n}) − v(∅). With the implicit normalization v(∅)=0 in Eq. (7), the total defined in Eq. (8) is exactly h(˜s) from Eq. (6). Therefore Proposition 3.5, which says h(˜s) does not satisfy Properties 3.2 and 3.3, contradicts Proposition 3.6, which says φ(˜s) satisfies them: if h(˜s) fails the properties, then φ(˜s)=h(˜s) must also fail them. The appendix never addresses this identity.
  2. [Appendix A, proof of Proposition 3.6] The proof does not derive the key inequalities; it asserts them. For Property 3.2, the proof states without argument that φ(s_i|˜s) > φ(g|˜s\s_i ∪ g) and that φ(s_j|˜s) ≥ φ(s_j|˜s\s_i ∪ g) for every remaining j. These inequalities are the entire content of the property and are exactly what must be shown from the determinant-based entropy game. Since the total Shapley uncertainty equals h(˜s), these asserted inequalities would imply monotonicity properties for h(˜s) that Proposition 3.5 denies. No derivation connects them to the Gaussian entropy formula.
  3. [§3.2 and Appendix A, Proposition 3.1 versus the experimental setting] The positive semi-definiteness guarantee is proved only for β ≤ 1/(n+1), where n is the number of sentences. The experiments set β = 0.5, which violates this bound for n ≥ 2. The text says the value was chosen because it maintained positive semi-definiteness in the experiments, but that makes PSD an empirically checked assumption rather than a consequence of Proposition 3.1. This matters because the entire construction of the Gaussian entropy in Eq. (6) requires a valid correlation/covariance matrix.
  4. [Tables 1 and 2] The running example is not reproducible from the printed material. The column headers are inconsistent with the listed answers (e.g., "William Shakespeare" appears in Table 1 with no corresponding answer row), the displayed entries do not obviously follow from Eq. (5) with the stated Gaussian kernel, and no calculation of φ is shown, so the reported values 0.40 and 0.51 cannot be verified. Moreover, if the displayed correlations are used directly, the matrix contains duplicate rows, which would make the Gaussian differential entropy −∞; since the paper claims Shapley uncertainty avoids exactly such degeneracies, the example needs a clear and consistent computation.
minor comments (6)
  1. [Figure 1 caption] The caption spells "Shapley" as "Shapely."
  2. [§3.4, Proposition 3.5] The text contains a duplicated article: "may not satisfy the the above properties."
  3. [Appendix A] The proof header says "Sharpley" instead of "Shapley."
  4. [Eq. (7)] The value of h(∅) for the empty coalition is never defined; the Shapley formula requires it. The paper should state the normalization explicitly and then address the consequence that Eq. (8) equals h(˜s).
  5. [§4 and Table 3] No confidence intervals or significance tests are reported for the AUROC comparisons; some reported differences are small (e.g., 0.811 vs. 0.808), so it is unclear whether they are meaningful.
  6. [§4] The baselines MaxL, AvgL, MaxE, and AvgE from Manakul et al. (2023) are named in Table 3 but not described in the text; a brief definition would help readers interpret the comparison.

Circularity Check

3 steps flagged · score 8.0 of 10

Shapley uncertainty equals the rejected differential entropy by Shapley efficiency, the Appendix A proof of Prop. 3.6 merely asserts the target inequalities, and the kernel hyperparameter β is fit to the evaluation benchmark.

  1. self definitional [Section 3.3, Eqs. (6)-(8) and footnote 1; Propositions 3.5-3.6]
    "We adopt a commonly used setting: suppose ˜s is a multivariate Gaussian with correlation (covariance) matrix R and compute its entropy. ... the whole uncertainty of ˜s is defined as ϕ(˜s) = X si∈˜s ϕ(si|˜s). (8) ... Proposition 3.5. The metric of differential entropy over the whole multivariate distribution, as defined in (6), does not satisfy Properties 3.2 and 3.3. Proposition 3.6. The metric of Shapley uncertainty, as defined in (7) and (8), satisfies Properties 3.2, 3.3, and 3.4."

    With the coalition game v(X)=h([sj]_{j∈X}), Shapley-value efficiency forces Σ_i φ(si|˜s)=v({1..n})-v(∅). For fixed n, v(∅) is a constant, so the total in (8) equals h(˜s) up to an additive constant that cancels in every comparison used by Properties 3.2-3.4. Thus the proposed metric is not an alternative to h(˜s); it is h(˜s) re-expressed via Shapley decomposition. Proposition 3.5 asserts h(˜s) fails Properties 3.2 and 3.3, while Proposition 3.6 asserts φ(˜s) satisfies them. Since additive constants cannot change monotonic comparisons, the two propositions are mutually inconsistent unless the proof supplies an argument it does not contain. The claimed separation is therefore an equivalence by construction, not a derived result.

  2. other [Appendix A, Proof of Proposition 3.6 (1), (2), (3)]
    "For all si ∈ ˜s such that |ρ(si, arg min sj∈˜s ϕ(sj|˜s))| < 1, suppose there exists a random variable g satisfies ρ(g, arg min sj∈˜s ϕ(sj|˜s)) = 1. Then we have ϕ(si|˜s) > ϕ(g|˜s\si ∪ g), and for all sj ∈ ˜s\si, we get ϕ(sj|˜s) ≥ ϕ(sj|˜s\si ∪ g). Thus we have ϕ(˜s) > ϕ(˜s\si ∪ g)."

    The conclusion of Property 3.2 is exactly φ(˜s\si ∪ g) < φ(˜s). The proof obtains this by asserting, without any derivation from (7) or from the determinant-based entropy game, that the replacement reduces every individual Shapley contribution. Those asserted inequalities are the property itself restated at the level of marginals. The proofs for Properties 3.3 and 3.4 make the same move: they state the needed pairwise marginal inequalities and immediately sum them. Proposition 3.6 is therefore not proved from the definition of Shapley uncertainty; it assumes the target property, making the argument circular.

1 more flagged steps
  1. fitted input called prediction [Appendix D, Kernel Hyperparameter and Table 5; Section 3.2]
    "We select the hyperparameter β of the kernel function in our method by finding maximum mean AUROC through two rounds of systematic grid search. ... LLaMA2-7B and TriviaQA are utilized as backbone dataset and LLM. ... In the experiments, we adopt the above Gaussian kernel and set β = 0.5 as default, which is chosen through ablation studies given in Appendix D."

    β is selected by maximizing the paper's own evaluation criterion (AUROC) on TriviaQA with LLaMA2-7B, and then reused as the default kernel parameter for all reported experiments, including TriviaQA. For that dataset-model combination, the reported 'prediction' of LLM correctness is not out-of-sample: the kernel scale was tuned to maximize exactly the AUROC score being presented. The extensive other datasets and models provide partial independent evidence, so this is a partial fitted-input issue rather than a complete collapse of the empirical claim.

full rationale

The theoretical claim is centrally circular in two ways. First, by Eqs. (7)-(8) the total Shapley uncertainty is the sum of Shapley values of the differential entropy h of a multivariate Gaussian with correlation matrix R. Shapley efficiency forces this sum to equal h(˜s)-h(∅), and with the standard normalization h(∅)=0 it is exactly h(˜s). Hence the proposed metric is the same function that Proposition 3.5 says fails Properties 3.2-3.3, and Proposition 3.6 cannot hold for it unless the proof supplies an extra mechanism, which it does not. Second, the Appendix A proof of Proposition 3.6 does not derive the required monotonicity inequalities from the definition; it simply asserts them, and those inequalities are precisely the properties being proved. Empirically, β is chosen by maximizing AUROC on TriviaQA/LLaMA2-7B and used as the default across all experiments, so the headline comparison on that pair is not out-of-sample; however, the paper does test many other datasets and models, so the empirical results are not wholly determined by the fit. The central theoretical derivation therefore reduces by construction to the differential entropy metric it claims to replace, and its proof restates its conclusion. Score 8 reflects that the load-bearing theoretical result is forced by definition and by an assumed-inequality proof, while acknowledging the broader empirical evaluation is not entirely circular.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central metric rests on a fitted kernel hyperparameter β, on the validity of a Gaussian entropy model for sentence sets, and on an unproven empirical PSD guarantee for β=0.5. No new physical or mathematical entities are introduced.

free parameters (2)
  • Kernel hyperparameter β = 0.5
    Selected by grid search to maximize AUROC on LLaMA2-7B/TriviaQA (Appendix D, Table 5), then fixed for all other datasets and models.
  • Correctness threshold for evaluation = RougeL > 0.3; BLEU > 0.3
    Chosen by hand to label an answer correct; changes which outputs count as correct and therefore affects all reported AUROC values.
assumptions (3)
  • domain assumption Multivariate Gaussian differential entropy with correlation matrix R is a valid uncertainty representation for a set of sentences.
    Footnote in Section 3.3 adopts the Gaussian entropy setting with no justification beyond mathematical convenience.
  • domain assumption P(si ⇒ sj|x) from the NLI model approximates semantic entailment or correlation between answers.
    Adopted from He et al. (2020) and Kuhn et al. (2023); used in Eq (4) and Algorithm 1 to build the correlation matrix.
  • ad hoc to paper The kernel-transformed matrix is positive semi-definite for β=0.5 on all experimental data.
    Proposition 3.1 only guarantees PSD for β ≤ 1/(n+1); the paper uses β=0.5 and asserts empirically PSD holds without reporting verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shapley Uncertainty in Natural Language Generation." pith.science (2026). https://pith.science/paper/QYRXETIO

@misc{pith2026250721406,
  author       = {Pith},
  title        = {Pith review of: Shapley Uncertainty in Natural Language Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QYRXETIO}},
  note         = {Machine review of arXiv:2507.21406}
}
read the original abstract

In question-answering tasks, determining when to trust the outputs is crucial to the alignment of large language models (LLMs). Kuhn et al. (2023) introduces semantic entropy as a measure of uncertainty, by incorporating linguistic invariances from the same meaning. It primarily relies on setting threshold to measure the level of semantic equivalence relation. We propose a more nuanced framework that extends beyond such thresholding by developing a Shapley-based uncertainty metric that captures the continuous nature of semantic relationships. We establish three fundamental properties that characterize valid uncertainty metrics and prove that our Shapley uncertainty satisfies these criteria. Through extensive experiments, we demonstrate that our Shapley uncertainty more accurately predicts LLM performance in question-answering and other datasets, compared to similar baseline measures.

Figures

Figures reproduced from arXiv: 2507.21406 by the authors.

Figure 1
Figure 1. Technical insights, challenges, and solutions in obtaining the new Shapely uncertainty estimation method. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Cross-domain and LLM architectural generalization of our approach. Our Shapley uncertainty (blue) also [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (Left) Our Shapley uncertainty (red) predicts model accuracy better than baselines on the answering dataset CoQA (average performance across OPT models from 1.3B to 13B parameters). (Right) The outstand￾ing performance of Shapley uncertainty (red) becomes more pronounced with larger model sizes, while still maintaining effectiveness for smaller models. fied as correct if BLEU(s, strue) exceeds 0.3, and incorrect oth… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Two examples of Shapley uncertainty over the answers of LLaMA2-7B ( [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

    cs.AI 2026-07 conditional novelty 6.0 of 10

    LGU models implication and incompatibility among LLM answers and reports consistent AUROC/AUARC gains over semantic entropy on QA benchmarks.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [3]

    arXiv preprint arXiv:2305.18404

    Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404. Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Bu- toi, Paul Bertin, Jarrid Rector-Brooks, Maksym Ko- rablyov, and Yoshua Bengio. 2022. Deup: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research. Kenton Lee, ...

    arXiv 2022

  2. [4]

    In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086–6096, Florence, Italy

    Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. Chin-Yew Lin and Franz Josef Och. 2004a. Auto- matic evaluation of machine translation quality using longest common...

    work page 2004

  3. [6]

    arXiv preprint arXiv:2303.08896

    Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303.08896. Christopher Mohri and Tatsunori Hashimoto. 2024. Language models with conformal factuality guaran- tees. arXiv preprint arXiv:2402.10978. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method f...

    arXiv 2024

  4. [8]

    arXiv preprint arXiv:2306.10193

    Conformal language modeling. arXiv preprint arXiv:2306.10193. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Lingui...

    arXiv 2016

  5. [9]

    Transactions of the Association for Com- putational Linguistics, 7:249–266

    Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249–266. Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. 2022. Re-examining calibration: The case of question answering. arXiv preprint arXiv:2205.12507. Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfo...

    arXiv 2022

  6. [10]

    Reducing LLM Hallucinations using Epistemic Neural Networks

    Reducing llm hallucinations using epistemic neural networks. arXiv preprint arXiv:2312.15576. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli

    work page Pith review arXiv

  7. [11]

    arXiv preprint arXiv:2401.11817

    Hallucination is inevitable: An innate lim- itation of large language models. arXiv preprint arXiv:2401.11817. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Weny...

    arXiv

  8. [2019]

    Preprint at https://arxiv

    Benchmarking bayesian deep learning with di- abetic retinopathy diagnosis. Preprint at https://arxiv. org/abs/1912.10481. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Spe- cia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of...

    arXiv 1912

Show all 12 references
  1. [2021]

    Association for Computational Linguistics

    Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics....

    2021 arXiv

  2. [2023]

    Amos Azaria and Tom Mitchell

    Falcon-40B: an open large language model with state-of-the-art performance. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it’s lying. arXiv preprint arXiv:2304.13734. Ondˇrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn,...

    2023 arXiv

  3. [2024]

    arXiv preprint arXiv:2404.15993

    Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993. Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representa- tions. 10...

    2020 arXiv

  4. [2025]

    arXiv preprint arXiv:2501.15383

    Qwen2.5-1m technical report. arXiv preprint arXiv:2501.15383. Xi Ye and Greg Durrett. 2021. Can explanations be use- ful for calibrating black box models? arXiv preprint arXiv:2110.07586. Liang Zhang, Nathaniel Xu, Pengfei Yang, Gaojie Jin, Cheng-Chao Huang, and Lijun Zhang. 2...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.