Pith. sign in

REVIEW 6 major objections 5 minor 57 references

The paper claims that LLM uncertainty is better scored over logical relations among sampled answers than over paraphrase-equivalence alone, and introduces Logical Graph Uncertainty (LGU) to do so.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:41 UTC pith:MR6PX3HM

load-bearing objection LGU is a genuinely new entropy-over-graphs uncertainty score with broad experiments and a real soft spot: the logical relations come from an NLI model that is also the referee for its own sanity checks. the 6 major comments →

arxiv 2607.16868 v1 pith:MR6PX3HM submitted 2026-07-18 cs.AI

Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

Yanni Dong , Minghua Liu , Meiling Zhu , Xiaowei Huang , Lijun Zhang This is my paper
classification cs.AI
keywords uncertainty quantificationlogical graphsimplication graphincompatibility graphsemantic entropyhallucination detectionselective predictionLLM reliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that LLM uncertainty is better measured over logical relations among sampled answers than over paraphrase-equivalence alone. It introduces Logical Graph Uncertainty (LGU), the closed-form score LGU=(1+InS)·IGE, built from an implication graph and an incompatibility graph over answer clusters. Across four QA benchmarks and eight LLMs, LGU improves hallucination detection (AUROC) and selective prediction (AUARC) over semantic entropy by up to +7.1% and +3.5%, with the best average rank among compared methods. The authors note two limits: the score inherits errors from the NLI model that supplies logical judgments, and it covers only implication and pairwise incompatibility, not richer relations.

Core claim

The paper claims that uncertainty scores should be computed over logical relations among sampled answers, not just over paraphrase-equivalence clusters. LGU builds an implication graph over semantic clusters, pushes probability mass along entailment chains to logically maximal roots, computes Shannon entropy over the normalized root distribution (IGE), then multiplies by one plus the edge density of an incompatibility graph over those roots (InS). Thus the score stays low when answers are diverse-but-compatible refinements of one correct answer, and rises when the model spreads mass across mutually exclusive hypotheses. The larger claim is that much of what semantic entropy counts as uncerta

What carries the argument

The central object is the pair of graphs built from sampled answers after equivalence clustering: the Implication Graph (ImG), a directed acyclic graph (one with no cycles) whose edges say cluster A entails cluster B under the question's hypothesis space, and the Incompatibility Graph (InG) over the ImG's root vertices, with bidirectional edges for mutually exclusive roots. The workhorse identity is LGU=(1+InS)·IGE, where IGE is Shannon entropy over the root distribution after pushing probability mass along implication chains, and InS is a graph-density statistic of the InG, with directed edge density as the default. The DAG property guarantees roots exist and aggregation is well defined; th

Load-bearing premise

The load-bearing premise is that an off-the-shelf NLI model's pairwise judgments of implication and incompatibility are faithful to truth-conditional logical relations within the question's hypothesis space; if those judgments are wrong, the graph, the DAG guarantee, and the empirical gains inherit the error.

What would settle it

Take a fixed QA benchmark and replace the automatic NLI judgments with human-annotated implication and incompatibility relations for the same sampled answers. If LGU's AUROC advantage over semantic entropy shrinks to zero or reverses, the reported gains are driven by the NLI oracle's particular errors rather than by logical structure itself. A cheaper check: collect a set of questions with a high non-empty ImG rate and see whether LGU's gain over SE is consistently negative on any model-dataset pair, contradicting the claimed positive correlation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On questions where sampled answers are paraphrases or refinements of one correct answer, LGU avoids the inflated uncertainty that semantic entropy produces, reducing false hallucination flags.
  • When answers form a dense set of mutually exclusive high-level hypotheses, the incompatibility factor scales the score up, making severe hallucination easier to detect.
  • The method keeps O(n²) runtime and needs no tunable hyperparameters, so the gain comes from structure rather than extra machinery.
  • The measured advantage over semantic entropy correlates with the prevalence of logical structure (Pearson r≈0.572), so LGU's benefit is concentrated where implication edges actually exist.
  • The score has a known extremal range, which the authors use to calibrate confidence under both theoretical and empirical normalization bounds; the black-box discrete variant also stays competitive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test: replace the automatic NLI judge with human-annotated logical relations. If LGU's gain over semantic entropy sharpens, the current numbers likely understate the value of logical structure; if it collapses, the gain is partly an oracle artifact.
  • Because LGU factors into specificity (IGE) and conflict (InS), the two numbers could be reported separately, letting a user distinguish 'model is unsure among compatible answers' from 'model is spreading mass across answers that cannot all be true'.
  • The same graph construction could be applied to long-form outputs by first decomposing generations into claims, although the authors note pairwise checks become expensive there.
  • In a selective-prediction pipeline, LGU's conflict term offers a principled abstention rule: reject questions where root hypotheses are densely incompatible, rather than only where answers are diverse.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes Logical Graph Uncertainty (LGU), a two-component uncertainty score for LLM-generated short-form answers. LGU first builds an Implication Graph (ImG) over semantic clusters using an NLI model (DeBERTa-Large-MNLI), propagates probability mass along implication chains, and computes Implication Graph Entropy (IGE) over the resulting roots. It then constructs an Incompatibility Graph (InG) over roots and multiplies IGE by (1+InS), where InS is a graph-density score (default: directed edge density). The authors claim that this logical-graph reformulation, as opposed to flat semantic clustering, consistently improves hallucination detection and selective prediction, with reported gains up to +7.1% AUROC and +3.5% AUARC over Semantic Entropy, across four QA datasets and eight LLMs, while adding no tunable parameters.

Significance. If the central claim holds, the paper makes a useful conceptual contribution: it extends semantic entropy from paraphrase equivalence to truth-conditional implication and incompatibility, and provides an O(n^2) parameter-free score. The empirical evaluation is broad (4 datasets, 8 model families, white-box and black-box settings, AUROC/AUARC/ECE, bootstrapped CIs), and the authors provide code and proofs of complexity/extremal bounds. However, the load-bearing assumption is that DeBERTa-MNLI judgments faithfully capture logical implication and incompatibility; the manuscript's own validation uses the same NLI model as both construction oracle and referee, so it does not independently bound oracle error. The post-hoc choice of the default InS metric and the test-set normalization for ECE further weaken the strength of the quantitative claims as currently stated.

major comments (6)
  1. [Section 2.2, Appendix B.4.2, Section 5 (Limitations)] The entire graph construction (ImG and InG) relies on DeBERTa-Large-MNLI entailment/contradiction judgments. Appendix B.4.2 reports 97.0% transitivity and 92.8% symmetry, but these checks use the same NLI model that constructed the graphs, demonstrating self-consistency rather than fidelity to truth-conditional logical relations. Because both IGE and InS depend on these judgments, this is load-bearing. Please add an independent validation (e.g., a human-annotated subset, a second NLI family, or a perturbation analysis) and quantify how AUROC/AUARC change as NLI accuracy varies.
  2. [Section 3.2 / Table 9 / Appendix B.3] The default InS (Directed Edge Density) is selected after comparing four variants on the same test benchmarks (Table 9). This is model selection on the test set; the headline LGU numbers use the selected variant, so the reported gains may be optimistic. A proper protocol would pre-register the metric or perform selection on a validation split, then report test performance for the chosen variant. At minimum, the authors should discuss the implications of this post-hoc choice for the 'parameter-free' claim.
  3. [Appendix B.4.1] ECE is computed after min-max normalizing each method's scores using test-set empirical bounds (c=1-(u-umin)/(umax-umin)). This uses the test set's observed range for each method, which can rank methods by dynamic-range luck rather than calibration. The 'theoretical bounds' comparison is also not fully fixed: LGU's upper bound depends on InS and graph size. Please evaluate calibration using validation-set normalization or a common, pre-specified scale, and report ECE on the original score scale where possible.
  4. [Section 1.2, Table 1, Appendix B.1] The claim of 'no tunable parameters' is stronger than what the implementation shows. The NLI implication/incompatibility threshold is not specified; the number of samples n, sampling temperature, and the choice of InS metric (edge density vs. alternatives) are user-set and affect results. Please state all thresholds and choices, and include a sensitivity analysis. The NeurIPS checklist (item 4) also acknowledges that details such as 'NLI thresholds' are not exhaustively specified in the main text.
  5. [Abstract, Table 5] The abstract claims LGU 'consistently improves' uncertainty estimation and 'substantially reduces ECE relative to KLE.' The paper's own Table 5 shows counterexamples: on Falcon-7b-instruct TriviaQA, LGU ECE is 0.151 vs KLE 0.073; on Gemma-2-9B-it TriviaQA, LGU ECE is 0.060 vs KLE 0.038; and LGU ECE is worse than SE on several model-dataset pairs (e.g., Falcon-7b-instruct NQ 0.138 vs 0.112). Please temper the wording to 'on average' or per-dataset reporting, and discuss the variability.
  6. [Section 2.3, IGE definition] The aggregation p'(ri) = sum_{c_j in Reach(ri)} p_j double-counts probability mass when a cluster reaches more than one root; subsequent normalization over the inflated sum does not yield a probability distribution over mutually exclusive hypotheses. The claim in Figure 1 that the method 'avoids double-counting' is therefore not exact. Please clarify the probabilistic interpretation of the normalized root scores, or modify the aggregation to allocate mass among reachable roots in a principled way.
minor comments (5)
  1. [General] Theorem/proposition numbering is inconsistent across text and appendix: main text uses Theorem 2.3 and Proposition 2.2, while Appendix B.4.2 and the NeurIPS checklist refer to Theorem 3.3 and Proposition 3.2. Please unify.
  2. [Figure 3b] The Pearson correlation coefficient 0.572 is reported without sample size or p-value. Please state the number of points (apparently 8 models or 32 model-dataset pairs) and a confidence interval.
  3. [Figure 2] The radar chart is said to cover 'six model architectures,' but the full evaluation includes eight models (Table 5). Please clarify whether the figure omits Qwen3-32B and the base Falcon-7b, and why.
  4. [References] References [16] and [17] appear to be the same survey by Huang et al. with different venues; please deduplicate or distinguish.
  5. [Section 1.2] The sentence 'This is the precise failure mode our method addresses' is a bit promotional; consider a more neutral framing.

Circularity Check

1 steps flagged

Central LGU score is not fitted to correctness, but the paper validates its load-bearing NLI premise with the same NLI model and selects its default incompatibility metric on the evaluation data.

specific steps
  1. other [Appendix B.4.2, 'Empirical Verification of NLI-based Logical Properties']
    "To empirically verify this, we randomly sampled 4,863 implication chains from our generated data (i.e., triplets where the NLI model confidently predicts A⇒B and B⇒C). Among these chains, the NLI model correctly predicted A⇒C in 97.0% of the cases. This remarkably high transitivity rate empirically justifies our implication-based probability aggregation and confirms that the NLI model provides a highly reliable directed topology for the ImG."

    The 'correctly' ground truth is produced by the same DeBERTa-Large-MNLI model that generated the A⇒B and B⇒C edges, so the 97% figure measures the model's agreement with itself (triplet consistency), not correspondence to truth-conditional implication over the question's hypothesis space. The paper calls NLI fidelity 'a critical premise' and uses this self-check to justify the ImG topology, the DAG assumption, and IGE/LGU aggregation; the validation therefore reduces to a self-consistency statistic rather than an independent confirmation. The same pattern appears in the 92.8% symmetry check for InG.

full rationale

The core LGU definition is not circular: LGU=(1+InS)·IGE is a closed-form function of NLI-derived implication/incompatibility edges and sampled answer masses, with no parameter fitted to correctness labels, and the main AUROC/AUARC comparisons use external GPT-3.5 correctness judgments. IGE reduces to SE only in the degenerate empty-ImG case, which is a special case, not a circular reduction. However, one load-bearing assumption—that DeBERTa-Large-MNLI's implication/incompatibility judgments faithfully realize the formal logical relations—is 'validated' in Appendix B.4.2 by the same model that constructs the graphs, so the 97.0%/92.8% property checks are self-consistency rates by construction and cannot independently support the logical interpretation. Separately, the default InS metric (edge density) is chosen in Appendix B.3 after ranking variants on the same datasets later used for the headline +7.1% AUROC claim; this is a post hoc selection loop, though not a definitional reduction. Section 5 acknowledges NLI dependence, which mitigates overclaiming but does not repair the self-referential validation. Because the evaluation itself is externally grounded and no score is fitted to correctness, the central result retains independent content; the circularity is confined to the validation of the NLI premise and the selection of the default variant. Score 4 rather than 6 because no central 'prediction' reduces by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central method introduces no physical entities and no fitted constants; however it relies on several domain assumptions (NLI fidelity, GPT-3.5 judge, heuristic score combination) and post-hoc selections (InS metric, empirical ECE normalization), which should be counted as hidden degrees of freedom.

free parameters (4)
  • NLI implication/incompatibility threshold
    DeBERTa-MNLI outputs must be thresholded to label edges in ImG/InG; the threshold is never specified, yet determines graph topology and therefore IGE/LGU.
  • Incompatibility score instantiation (edge density) = edge density
    Selected as default after comparing four variants on the same test benchmarks (Appendix B.3), so the headline results are conditional on a post-hoc model selection.
  • Sampling temperature / number of candidate answers = temperature=1; n unspecified
    Uncertainty samples are drawn at temperature 1, but the number of samples n and any variance-reduction choices are not stated; AUROC/ECE depend on these.
  • ECE normalization bounds = empirical min/max per dataset
    Confidence scores are min-max normalized using test-set extrema (Appendix B.4.1), which can make ECE optimistically well-calibrated.
axioms (4)
  • domain assumption NLI pairwise judgments approximate truth-conditional implication and incompatibility over the question-induced hypothesis space.
    Section 2.2 defines ImG/InG via such relations; Theorem 2.3 (DAG) requires perfect transitivity/antisymmetry. The authors acknowledge NLI errors in Section 5.
  • domain assumption The reference judge (GPT-3.5) correctly maps predicted answers to gold answers in the correctness labels.
    Section 3: 'To assess correctness ... GPT-3.5 as reference judge'; all AUROC/AUARC/ECE labels derive from this judgment.
  • ad hoc to paper LGU=(1+InS)*IGE is a meaningful combination of implication-entropy and incompatibility density.
    Defined in Section 2.3 without derivation from an underlying decision-theoretic objective; it is a heuristic that Theorem 2.6 characterizes but does not justify.
  • domain assumption The ImG is acyclic in practice.
    Theorem 2.3 assumes exact logical implication; Appendix B.4.2's transitivity test itself relies on the NLI model, so acyclicity is not independently established.
invented entities (1)
  • Implication Graph (ImG) / Incompatibility Graph (InG) over semantic clusters no independent evidence
    purpose: Represent logical implication and pairwise incompatibility among sampled answers so uncertainty can be computed over maximal hypotheses.
    These are formal modeling constructs introduced by the paper; their validity is tied to the NLI oracle and the benchmarks used, not to a falsifiable prediction outside the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 47171 in / 15885 out tokens · 150897 ms · 2026-08-01T19:41:22.784758+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) often produce confidently stated yet unreliable outputs, posing critical challenges for deployment in safety-sensitive applications. Existing uncertainty metrics such as semantic entropy capture agreement at the level of semantic equivalence, but largely ignore the logical relationships between distinct answers. As a result, they tend to overestimate uncertainty and falsely flag hallucinations in settings where generated responses are diverse in form yet logically compatible (e.g., differing only in granularity or specificity). We propose Logical Graph Uncertainty (LGU), a framework that explicitly models implication and incompatibility among answers. LGU aggregates probability mass along entailment chains, computes entropy over logically maximal hypotheses, and penalizes mutual incompatibility among them. Across multiple question-answering benchmarks, LGU consistently improves uncertainty estimation over existing methods, and outperforms the semantic entropy baseline by up to +7.1% AUROC and +3.5% AUARC across datasets.

Figures

Figures reproduced from arXiv: 2607.16868 by Lijun Zhang, Meiling Zhu, Minghua Liu, Xiaowei Huang, Yanni Dong.

Figure 1
Figure 1. Figure 1: Overview Comparison of Uncertainty Measures on a QA Example. NE computes entropy over raw sampled strings, and SE clusters paraphrases before computing entropy. IGE further builds an implication graph among answers, propagates probability mass along implication chains, and normalizes the aggregated root scores into a valid distribution, avoiding double-counting under logically compatible refinements. LGU t… view at source ↗
Figure 2
Figure 2. Figure 2: Performance robustness across LLMs on SQuAD. Radar charts show the rank-based performance of uncertainty methods across six model architectures. A larger shaded area corresponds to better and more consistent performance across tasks. 3.1 Results Our empirical evaluation demonstrates that LGU consistently yields superior discriminative power, robust cross-model generalization, and reliable calibration compa… view at source ↗
Figure 3
Figure 3. Figure 3: Effectiveness and Structural Sensitivity of Logical Graph Uncertainty (LGU). (a) Calibration of Falcon-7b on SQuAD. Left: Reliability diagrams aligning predicted confidence with empirical accuracy. Right: Expected Calibration Error (ECE, ↓). (b) LGU performance gain vs. logical structure prevalence. Correlation between the AUROC improvement over SE (∆AUROC) and the proportion of answers containing logical … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 12 linked inside Pith

  1. [1]

    Prompt-reverse inconsistency: LLM self-inconsistency beyond generative randomness and prompt paraphrasing

    Jihyun Janice Ahn and Wenpeng Yin. Prompt-reverse inconsistency: LLM self-inconsistency beyond generative randomness and prompt paraphrasing. InSecond Conference on Language Modeling, 2025

    2025

  2. [2]

    The falcon series of open language models, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, et al. The falcon series of open language models, 2023. arXiv:2311.16867

    Pith/arXiv arXiv 2023

  3. [3]

    Structural entropy: Monitoring correlation-based networks over time with application to financial markets.Scientific Reports, 9:10832, 2019

    Assaf Almog and Erez Shmueli. Structural entropy: Monitoring correlation-based networks over time with application to financial markets.Scientific Reports, 9:10832, 2019

    2019

  4. [4]

    Estimating the Estrada index.Linear Algebra and its Applications, 427(1):70–76, November 2007

    José Antonio de la Peña, Ivan Gutman, and Juan Rada. Estimating the Estrada index.Linear Algebra and its Applications, 427(1):70–76, November 2007

    2007

  5. [5]

    A history of graph entropy measures.Information Sciences, 181(1):57–78, 2011

    Matthias Dehmer and Abbe Mowshowitz. A history of graph entropy measures.Information Sciences, 181(1):57–78, 2011

    2011

  6. [6]

    A survey on code generation with LLM-based agents, 2025

    Yihong Dong, Xue Jiang, Jiaru Qian, et al. A survey on code generation with LLM-based agents, 2025. arXiv:2508.00083

    Pith/arXiv arXiv 2025

  7. [7]

    The physics of communicability in complex networks.Physics Reports, 514(3):89–119, 2012

    Ernesto Estrada, Naomichi Hatano, and Michele Benzi. The physics of communicability in complex networks.Physics Reports, 514(3):89–119, 2012

    2012

  8. [8]

    Fact-checking the output of large language models via token-level uncertainty quantification

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, et al. Fact-checking the output of large language models via token-level uncertainty quantification. InFindings of the Association for Computational Linguistics: ACL 2024, pages 9367–9385. Association for Computational Linguistics, 2024

    2024

  9. [9]

    Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630, June 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, et al. Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630, June 2024

    2024

  10. [10]

    Unsupervised quality estimation for neural machine translation.Transactions of the Association for Computational Linguistics, 8:539–555, September 2020

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, et al. Unsupervised quality estimation for neural machine translation.Transactions of the Association for Computational Linguistics, 8:539–555, September 2020

    2020

  11. [11]

    Community detection in graphs.Physics Reports, 486(3):75–174, 2010

    Santo Fortunato. Community detection in graphs.Physics Reports, 486(3):75–174, 2010. 10

    2010

  12. [12]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, et al. On calibration of modern neural networks. InProceed- ings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 1321–1330. PMLR, 06–11 Aug 2017

    2017

  13. [13]

    The Estrada index: An updated survey

    Ivan Gutman, Hanyuan Deng, and Slavko Radenkovi´c. The Estrada index: An updated survey. Zbornik Radova, (22):155–174, 2011

    2011

  14. [14]

    DeBERTa: Decoding-enhanced bert with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, et al. DeBERTa: Decoding-enhanced bert with disentangled attention. InProceedings of the 9th International Conference on Learning Representations, 2021

    2021

  15. [15]

    A survey of uncertainty estimation in LLMs: Theory meets practice, 2024

    Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, et al. A survey of uncertainty estimation in LLMs: Theory meets practice, 2024. arXiv:2410.15326

    Pith/arXiv arXiv 2024

  16. [16]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, January 2025

    Lei Huang, Weijiang Yu, Weitao Ma, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, January 2025

    2025

  17. [17]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Trans

    Lei Huang, Weijiang Yu, Weitao Ma, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Trans. Inf. Syst., 43(2):1–55, January 2025

    2025

  18. [18]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b, 2023. arXiv:2310.06825

    Pith/arXiv arXiv 2023

  19. [19]

    Graph-based uncertainty metrics for long-form language model generations

    Mingjian Jiang, Yangjun Ruan, Prasanna Sattigeri, et al. Graph-based uncertainty metrics for long-form language model generations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 32980–33006. Curran Associates, Inc., 2024

    2024

  20. [20]

    Shapley uncertainty in natural language genera- tion, 2025

    Gaojie Jin, Xiaowei Huang, Meilin Zhu, et al. Shapley uncertainty in natural language genera- tion, 2025. arXiv:2507.21406

    Pith/arXiv arXiv 2025

  21. [21]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, et al. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics

    2017

  22. [22]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, et al. Language models (mostly) know what they know, 2022. arXiv:2207.05221

    Pith/arXiv arXiv 2022

  23. [23]

    BioASQ-QA: A manually curated corpus for biomedical question answering.Scientific Data, 10:170, March 2023

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, et al. BioASQ-QA: A manually curated corpus for biomedical question answering.Scientific Data, 10:170, March 2023

    2023

  24. [24]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. InThe Eleventh International Conference on Learning Representations (ICLR 2023). OpenReview.net, 2023

    2023

  25. [25]

    Natural questions: A benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:452–466, 2019

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, et al. Natural questions: A benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:452–466, 2019

    2019

  26. [26]

    Teaching models to express their uncertainty in words.Transactions on Machine Learning Research, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words.Transactions on Machine Learning Research, 2022

    2022

  27. [27]

    Semantic energy: Detecting LLM hallucination beyond entropy, 2025

    Huan Ma, Jiadong Pan, Jing Liu, et al. Semantic energy: Detecting LLM hallucination beyond entropy, 2025. arXiv:2508.14496

    arXiv 2025

  28. [28]

    Uncertainty estimation in autoregressive structured prediction

    Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. InInternational Conference on Learning Representations, 2021

    2021

  29. [29]

    Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning, 2023

    Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning, 2023. arXiv:2308.00436. 11

    Pith/arXiv arXiv 2023

  30. [30]

    Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option

    Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar. Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option. InProceedings of the Third International Workshop on Machine Learning in Systems Biology, volume 8 of Proceedings of Machine Learning Research, pages 65–81, Ljubljana, Slovenia, September 2009. PMLR

    2009

  31. [31]

    Oxford University Press, 2010

    Mark Newman.Networks: An Introduction. Oxford University Press, 2010

    2010

  32. [32]

    Are large language models more honest in their probabilistic or verbalized confidence?, 2024

    Shiyu Ni, Keping Bi, Lulu Yu, and Jiafeng Guo. Are large language models more honest in their probabilistic or verbalized confidence?, 2024. arXiv:2408.09773

    Pith/arXiv arXiv 2024

  33. [33]

    Kernel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities

    Alexander V Nikitin, Jannik Kossen, Yarin Gal, et al. Kernel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  34. [34]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, et al. Training language models to follow instructions with human feedback. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA, 2022. Curran Associates Inc

    2022

  35. [35]

    Causal diagrams for empirical research.Biometrika, 82(4):669–688, 1995

    Judea Pearl. Causal diagrams for empirical research.Biometrika, 82(4):669–688, 1995

    1995

  36. [36]

    Survey of GenAI for automotive software development: From requirements to executable code

    Nenad Petrovic, Vahid Zolfaghari, Andre Schamschurko, et al. Survey of GenAI for automotive software development: From requirements to executable code. In2025 2nd International Generative AI and Computational Language Modelling Conference (GACLM), pages 184–197, Valencia, Spain, August 2025. IEEE

    2025

  37. [37]

    Know what you don’t know: Unanswerable questions for SQuAD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics

    2018

  38. [38]

    Fault tree analysis: A survey of the state-of-the-art in modeling, analysis and tools.Computer Science Review, 15–16:29–62, 2015

    Enno Ruijters and Mariëlle Stoelinga. Fault tree analysis: A survey of the state-of-the-art in modeling, analysis and tools.Computer Science Review, 15–16:29–62, 2015

    2015

  39. [39]

    A head to predict and a head to question: Pre-trained uncertainty quantification heads for hallucination detection in LLM outputs

    Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, et al. A head to predict and a head to question: Pre-trained uncertainty quantification heads for hallucination detection in LLM outputs. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing, pages 35712–35731, Suzhou, China, November 2025. Association for Computationa...

    2025

  40. [40]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, et al. Gemma 2: Improving open language models at a practical size, 2024

    2024

  41. [41]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. arXiv:2307.09288

    Pith/arXiv arXiv 2023

  42. [42]

    Large language models for education: A survey and outlook, 2024

    Shen Wang, Tianlong Xu, Hang Li, et al. Large language models for education: A survey and outlook, 2024. arXiv:2403.18105

    Pith/arXiv arXiv 2024

  43. [43]

    Factuality of large language models: A survey

    Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, et al. Factuality of large language models: A survey. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19519–19529, Miami, Florida, USA, November 2024. Association for Computational Linguistics

    2024

  44. [44]

    A survey of uncertainty estimation methods on large language models

    Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, et al. A survey of uncertainty estimation methods on large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 21381–21396, Vienna, Austria, July 2025. Association for Computational Linguistics

    2025

  45. [45]

    On hallucination and predictive uncertainty in conditional language generation

    Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), pages 2734–2744. Association for Computational Linguistics, 2021

    2021

  46. [46]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report, 2025. 12

    2025

  47. [47]

    LoGU: Long-form generation with uncertainty expressions

    Ruichen Yang, Zheng Chen, Yankai Lin, and Peng Li. LoGU: Long-form generation with uncertainty expressions. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18947–18968, Vienna, Austria, July

  48. [48]

    LLM4Drive: A survey of large language models for autonomous driving, 2023

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, et al. LLM4Drive: A survey of large language models for autonomous driving, 2023. arXiv:2311.01043

    Pith/arXiv arXiv 2023

  49. [49]

    LUQ: Long-text uncertainty quantification for LLMs

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. LUQ: Long-text uncertainty quantification for LLMs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244–5262, Miami, Florida, USA, November 2024. Association for Computational Linguistics

    2024

  50. [50]

    A survey of large language models, 2023

    Wayne Xin Zhao, Kun Zhou, Junyi Li, et al. A survey of large language models, 2023. arXiv:2303.18223

    Pith/arXiv arXiv 2023

  51. [51]

    SeSE: Black-box uncertainty quantification for large language models based on structural information theory, 2026

    Xingtao Zhao, Hao Peng, Dingli Su, et al. SeSE: Black-box uncertainty quantification for large language models based on structural information theory, 2026. A Algorithms of Logical Graph Uncertainty A.1 Proofs A.1.1 Proof of Theorem 2.6 Proof.We prove the upper and lower bounds separately, and then characterize the equality cases. Upper bound.By definitio...

    2026

  52. [53]

    Theoretical Bounds:Derived from the mathematical definitions of the metrics. For example, Shannon entropy over N candidate answers is bounded by log(N) , and the theoretical maximum of our Logical Graph Uncertainty (LGU) is strictly delineated by Theorem 3.6

  53. [54]

    Empirical (Actual) Bounds:Derived directly from the observed dynamic range (i.e., the actual minimum and maximum values generated by a specific model on a specific dataset). To ensure our calibration evaluation is not an artifact of the normalization strategy, we conduct a comprehensive comparison of the best-performing methods (i.e., the method achieving...

  54. [55]

    In our implementation, reflexivity holds trivially and does not affect the computation of Implication Graph Entropy (IGE), as self-loops are omitted by definition

    Implication Graph (ImG) and Transitivity.According to Proposition 3.2, the logical implication relation utilized to construct the ImG must be reflexive and transitive. In our implementation, reflexivity holds trivially and does not affect the computation of Implication Graph Entropy (IGE), as self-loops are omitted by definition. However, transitivity is ...

  55. [56]

    Apollo". Although the sampled responses vary lexically (e.g.,

    Incompatibility Graph (InG) and Symmetry.Proposition 3.4 defines incompatibility as a symmetric, non-reflexive, and non-transitive relation. Non-reflexivity and non-transitivity hold trivially for mutually exclusive statements (e.g., a hypothesis cannot contradict itself). To verify symmetry, we randomly sampled 5,339 incompatible pairs predicted by the N...

    arXiv 1929

  56. [57]

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...

  57. [2025]

    Association for Computational Linguistics