Pith. sign in

REVIEW 3 major objections 5 minor 89 references

A single spectral vector—the eigenvalues of the answer-embedding Gram matrix—can predict when a large vision-language model's output should be trusted, without relying on brittle clustering.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:13 UTC pith:WXJXQY55

load-bearing objection SGPU is a sensible supervised alternative to clustering-based semantic uncertainty, but the SOTA claim is undercut by an apples-to-oranges comparison and an overstated ECE claim. the 3 major comments →

arxiv 2512.14177 v3 pith:WXJXQY55 submitted 2025-12-16 cs.CV

Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes

Joseph Hoche , Andrei Bursuc , David Brellmann , Gilles Louppe , Pavel Izmailov , Angela Yao , Gianni Franchi This is my paper
classification cs.CV
keywords semantic uncertaintyhallucination detectionlarge vision-language modelsGaussian process classifiereigenvalue spectrumGram matrixselective predictionembedding geometry
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the geometric configuration of multiple sampled answers—summarized by the eigenvalues of their embedding Gram matrix—is enough to predict whether a vision-language model's output should be trusted. Existing semantic-uncertainty methods group answers into clusters and measure how often the model changes its mind, but clustering is sensitive to phrasing. SGPU instead feeds the sorted eigenvalue spectrum into a Gaussian Process Classifier that learns the mapping from consistency patterns to correctness. Across six models and eight datasets the method reports better discrimination and calibration than semantic entropy, kernel language entropy, and semantic-volume baselines, and the learned mapping transfers across model families and modalities. If right, this gives a simple, black-box recipe for detecting hallucinations.

Core claim

SGPU's central claim is that the shape of the eigenspectrum of the answer-embedding Gram matrix is a sufficient descriptor of semantic consistency: a spectrum dominated by a few large eigenvalues signals a concentrated semantic manifold (the model agrees with itself), while a flatter spectrum signals dispersed, conflicting answers. A Gaussian Process Classifier trained to map this spectral vector to a binary truthfulness label—assigned by an LLM judge comparing sampled answers to a reference—yields calibrated, discriminative uncertainty scores that beat clustering-based semantic entropy, kernel language entropy, and semantic-volume methods on AUROC, AUARC, and ECE, and that transfer across L

What carries the argument

The central object is the ordered eigenvalue vector λ = (λ1 ≥ ... ≥ λN) of the Gram matrix Σ = Φ^T Φ, where Φ holds the normalized sentence embeddings of N sampled answers. This spectrum is a compact, permutation-invariant summary of the semantic configuration of the responses. The Gaussian Process Classifier (GPC) with a kernel function then learns the probabilistic map p(truthfulness | λ). The GPC does double duty: it produces the confidence score and, through its predictive standard deviation, a second-order estimate of how much the classifier itself trusts that score.

Load-bearing premise

Everything rests on the premise that when a model's sampled answers agree semantically, they are usually correct; a model that confidently repeats the same wrong answer will produce a 'certain-looking' spectrum and SGPU will be confidently wrong.

What would settle it

Construct a set of queries where a target LVLM outputs essentially the same wrong answer across all sampled responses (e.g., adversarial images engineered to elicit a fixed incorrect response), then check whether SGPU assigns high confidence (low semantic uncertainty) to those queries. If the uncertainty score stays low despite the answers being wrong, the consistency-correctness assumption is falsified and the spectral classifier inherits that failure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SGPU reports top AUROC, AUARC, and low ECE on VQA, image classification, and textual QA datasets, with a fully black-box pipeline that never touches the LVLM's internal states.
  • The spectral-to-confidence mapping transfers across LVLM architectures and between similar tasks across modalities, so a trained GPC can be reused for a new model without retraining.
  • About 200 training examples suffice for SGPU to outperform training-free baselines, making the approach practical where labeled correctness data is scarce.
  • The GPC's predictive standard deviation flags 'unsafe' predictions; discarding them improves AUROC (0.828 → 0.853 on VizWiz), giving the method an internal alarm for when to abstain.
  • SGPU's inference time is comparable to token-probability methods; the sentence-encoding step adds negligible cost relative to sampling multiple answers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the spectral fingerprint is as transferable as reported, it suggests a universal 'consistency prior' for generative models: the eigenvalue distribution of answer embeddings may act as a task- and model-agnostic signature of answerability, usable as a pretrained feature for any new model.
  • The sorted-eigenvalue representation is permutation-invariant and continuous, which plausibly explains the cross-model transfer; a testable extension is zero-shot calibration of a new LVLM by using a GPC trained on any other model's outputs.
  • A natural stress test separates the method from its proxy: adversarial images that elicit the same confident wrong answer across samples should produce a spectrum that looks 'certain,' and SGPU should fail to flag it—exposing the consistency-correctness assumption rather than the spectral classifier.
  • The same Gram-spectrum descriptor could be applied to generated images, code, or speech, wherever a fixed embedding space exists, turning SGPU into a general selective-prediction device.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SGPU (Semantic Gaussian Process Uncertainty), a method for quantifying semantic uncertainty in LVLMs and LLMs. Given N sampled answers to an input, SGPU embeds them with a sentence encoder, computes the N×N Gram matrix, and uses its ordered eigenvalue spectrum as a compact feature vector. A Gaussian Process Classifier is then trained on these spectral features to predict whether the sampled answers are correct (with labels obtained from an LLM-as-judge against reference answers). The method is evaluated on six LLMs/LVLMs across eight datasets (VQA, image classification, and textual QA), against training-free baselines such as semantic entropy, discrete semantic entropy, predictive entropy, KLE, UMPIRE, and eigenscore variants. The paper claims state-of-the-art AUROC, AUARC, and ECE, and reports cross-model and cross-modality transfer experiments.

Significance. The core idea—replacing brittle clustering with a spectral summary of the answer-embedding geometry and feeding it into a principled Bayesian classifier—is interesting and potentially useful. The paper also ships a substantial evaluation, including transfer experiments across models and modalities, and a thoughtful use of the GPC's predictive variance to flag low-confidence uncertainty estimates. These are genuine strengths. However, the main empirical claim of state-of-the-art performance is currently not established because the method is supervised while all baselines are training-free. The spectral representation and the label information are confounded. In addition, the claim of consistently state-of-the-art calibration is directly contradicted by the paper's own tables. With additional controlled experiments the contribution could be solid, but as presented the SOTA claim is overstated.

major comments (3)
  1. [§4.2, Appendix B, Tables 1–2] SGPU is supervised: it trains a GPC on per-example truthfulness labels obtained by LLM-as-judge against reference answers (Appendix B). All baselines (SE, DSE, PE, KLE, UMPIRE, Cov/Cos Eigenscore) are fixed, training-free scores. The reported gains in AUROC/AUARC/ECE therefore conflate the spectral representation with the presence of labeled training data. To support the claim that the eigenvalue spectrum is the source of improvement, add supervised baselines trained with the same labels and splits: for example, train the same GPC (or a logistic-regression probe) on the existing scalar scores (predictive entropy, semantic entropy, semantic volume/log-det, KLE score, eigenscore) and report the same metrics. Without this control, the SOTA claim is not established.
  2. [Abstract, §5.1, Table 1, Table A.7] The abstract and Table 1's caption state that SGPU 'consistently achieves state-of-the-art calibration (ECE)'. Table 1 contradicts this: on ADVQA, SGPU's ECE is 0.173 vs KLE-Matern 0.105; on VQARAD, 0.210 vs KLE-Matern 0.159; on VizWiz, 0.356 vs PE 0.112; and on Imagenette, 0.199 vs Cov Eigenscore 0.094. The PopQA results in Table A.7 show SGPU ECE 0.420, much worse than SE (0.130) and Cos Eigenscore (0.132). The 'consistently ... reducing ECE' claim should be revised to a qualified statement (e.g., 'competitive on several datasets'), and the table captions should accurately reflect the numbers.
  3. [§4.2] The method rests on the assumption that 'high semantic consistency among generated responses indicates correctness, whereas significant variation indicates potential hallucinations'. This is a separate premise from the spectral classifier: when a model is confidently wrong, the eigenvalue spectrum will appear 'certain' and SGPU will be confidently wrong. This is not a circularity issue—the labels come from an independent LLM-as-judge step—but it is a load-bearing limitation. Please acknowledge it explicitly, and ideally test it by conditioning the AUROC/AUARC analysis on examples where the majority answer is wrong but the spectral features are concentrated.
minor comments (5)
  1. [§5.1, References] Reference [83] is cited for Qwen2.5-VL-3B and Llama-3.1-8B, but [83] is a DeepSeek-OCR paper. The correct citations appear to be [5] for Qwen2.5-VL and [80] for Llama-3. This should be fixed.
  2. [Tables 1, A.9, etc.] Decimal commas are used in tables (e.g., '0,823') while the text uses decimal points. Please use a consistent format.
  3. [Appendix C.1] The filtering experiment reports AUROC increasing from 0.828 to 0.853 after removing 'unsafe' SGPU predictions, evaluated on the same set. This post-hoc filtering should also be evaluated on a held-out split to avoid optimistic bias.
  4. [§5.3, Table 5] The runtime comparison reports per-sample inference time but does not include the training time for SGPU's GPC. Since SGPU is supervised, a note on training cost is needed for a fair practical comparison.
  5. [General] No error bars, confidence intervals, or significance tests are reported for the AUROC/AUARC/ECE differences. Given the variability of these metrics across datasets and models, adding standard errors or significance tests would strengthen the empirical claims.

Circularity Check

0 steps flagged

No significant circularity: the spectral feature and the correctness labels are independently constructed, and the GPC is evaluated on held-out data.

full rationale

The paper's derivation chain is: (1) sample N answers Y; (2) embed them with an external sentence encoder and form the Gram matrix Sigma = Phi^T Phi; (3) take the sorted eigenvalues lambda (Eqs. 8-9); (4) train a Gaussian Process Classifier on D~ = {(lambda^(i), l^(i))}, where l^(i) is an LLM-as-judge majority-vote correctness label (Section 4.2 and Appendix B); (5) evaluate held-out AUROC/AUARC/ECE. Each step is a genuine composition of operations, not an identity: lambda is a fixed function of the sampled answers alone, and the label l is generated independently of the spectral features, using a reference answer and an LLM-as-judge prompt. The consistency-correctness link is explicitly stated as an assumption ('our approach leans on the common assumption in semantic uncertainty methods that high semantic consistency among generated responses indicates correctness'), so its possible failure is a validity caveat, not a circular derivation. The self-citations in the related-work section are background references and are not load-bearing for SGPU's spectral representation or for the GPC. The absence of supervised baselines and the apparent contradiction between the abstract's 'consistently ... calibration' claim and some ECE entries in Table 1 are experimental-fairness and reporting concerns, not reductions of the prediction to its input. No equation defines the eigenvalue feature in terms of the label, nor the label in terms of the eigenvalue feature, and the claimed predictions are not forced by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or latent entity. Its free parameters are experimental choices (N, sampling hyperparameters) and the learned GPC kernel/hyperparameters. The load-bearing assumptions are the consistency-correctness proxy, the reliability of the sentence encoder, the accuracy of the LLM judge, and the calibration of the GPC.

free parameters (5)
  • Number of sampled answers N = 20
    Set to 20 for all experiments; the eigenvalue vector length and its discriminative content depend on this choice.
  • Sampling temperature = 1.0
    Hand-chosen; controls answer diversity and therefore the shape of the eigenvalue spectrum.
  • top_p = 0.9
    Nucleus sampling threshold chosen by hand; affects the diversity of generated answers.
  • top_k = 50
    Top-k filtering threshold chosen by hand; affects answer diversity.
  • GPC kernel and hyperparameters = not specified
    The kernel function is left unspecified; GP hyperparameters are learned from the training labels and are part of the mapping from spectrum to confidence.
axioms (4)
  • domain assumption High semantic consistency among sampled answers is a reliable proxy for correctness; high variation indicates hallucination.
    Stated in Section 4.2: 'we lean on the common assumption ... high semantic consistency among generated responses indicates correctness'. The entire label-mapping premise depends on this.
  • domain assumption Sentence embeddings from all-MiniLM-L6-v2 place semantically equivalent answers close together so that Gram-matrix eigenvalues reflect semantic consistency.
    Invoked in Sections 4.1 and Appendix A.4; no analysis of embedding failure modes or sensitivity to encoder choice.
  • domain assumption Llama-3.1-8B as LLM-as-judge provides correct binary truthfulness labels for training the GPC.
    Appendix B describes the labeling procedure; judge errors propagate into the learned mapping and the reported AUROC/ECE.
  • domain assumption A zero-mean Gaussian process with sigmoid likelihood and the chosen kernel yields calibrated probabilities on this low-dimensional feature space.
    Section 4.2 and Appendix C assert well-calibrated behavior, but no proof or analysis is given for this specific feature distribution.

pith-pipeline@v1.3.0-alltime-deepseek · 23159 in / 8906 out tokens · 74656 ms · 2026-08-03T16:13:12.798264+00:00 · methodology

0 comments
read the original abstract

Large Vision-Language Models (LVLMs) often produce plausible but unreliable outputs, making robust uncertainty estimation essential. Recent work on semantic uncertainty estimates relies on external models to cluster multiple sampled responses and measure their semantic consistency. However, these clustering methods are often fragile, highly sensitive to minor phrasing variations, and can incorrectly group or separate semantically similar answers, leading to unreliable uncertainty estimates. We propose Semantic Gaussian Process Uncertainty (SGPU), a Bayesian framework that quantifies semantic uncertainty by analyzing the geometric structure of answer embeddings, avoiding brittle clustering. SGPU maps generated answers into a dense semantic space, computes the Gram matrix of their embeddings, and summarizes their semantic configuration via the eigenspectrum. This spectral representation is then fed into a Gaussian Process Classifier that learns to map patterns of semantic consistency to predictive uncertainty, and that can be applied in both black-box and white-box settings. Across six LLMs and LVLMs on eight datasets spanning VQA, image classification, and textual QA, SGPU consistently achieves state-of-the-art calibration (ECE) and discriminative (AUROC, AUARC) performance. We further show that SGPU transfers across models and modalities, indicating that its spectral representation captures general patterns of semantic uncertainty.

Figures

Figures reproduced from arXiv: 2512.14177 by Andrei Bursuc, Angela Yao, David Brellmann, Gianni Franchi, Gilles Louppe, Joseph Hoche, Pavel Izmailov.

Figure 1
Figure 1. Figure 1: Illustration of semantic uncertainty quantification in LVLMs using clustering strategies. Instead of generating a single response, multiple responses are sampled to assess semantic uncertainty. Two possible clusterings are shown in pink and green: in the pink example, the model appears more confident because the Paris cluster dominates, whereas in the green partition both clusters are of similar size, indi… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the SGPU pipeline. SGPU operates in three steps: (1) generating multiple candidate sequences for a given context (query + image), (2) encoding their semantic consistency into a single vector representation λ, and (3) using this vector representation as input for a Gaussian Process classifier to estimate the predictive semantic uncertainty / truthfulness of the generated samples. Discrete Se… view at source ↗
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 16 linked inside Pith

  1. [1]

    Sindex: Semantic inconsistency index for hallucination detection in llms.arXiv preprint arXiv:2503.05980, 2025

    Samir Abdaljalil, Hasan Kurban, Parichit Sharma, Erchin Serpedin, and Rachad Atat. Sindex: Semantic inconsistency index for hallucination detection in llms.arXiv preprint arXiv:2503.05980, 2025. 2, 3, 5

    Pith/arXiv arXiv 2025

  2. [2]

    Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi

    Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Tech- niques, applications and challenges.IF, 2021. 2, 3

    2021

  3. [3]

    Rethinking uncertainty estimation in natural language generation.arXiv preprint arXiv:2412.15176, 2024

    Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochre- iter. Rethinking uncertainty estimation in natural language generation.arXiv preprint arXiv:2412.15176, 2024. 2

    Pith/arXiv arXiv 2024

  4. [4]

    The internal state of an LLM knows when it’s lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. InEMNLP, 2023. 2, 3, 4

    2023

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 6

    Pith/arXiv arXiv 2025

  6. [6]

    Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Fi- los, Zachary Nado, Mike Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal. Benchmarking bayesian deep learning on diabetic retinopathy detection tasks. InNeurIPS Datasets and Benchmarks, 2021. 5

    2021

  7. [7]

    Post-hoc probabilistic vision-language models.arXiv preprint arXiv:2412.06014, 2024

    Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, and Martin Trapp. Post-hoc probabilistic vision-language models.arXiv preprint arXiv:2412.06014, 2024. 3

    arXiv 2024

  8. [8]

    Hallucination detec- tion in LLMs using spectral features of attention maps

    Jakub Binkowski, Denis Janiak, Albert Sawczyn, Bogdan Gabrys, and Tomasz Jan Kajdanowicz. Hallucination detec- tion in LLMs using spectral features of attention maps. In EMNLP, 2025. 3

    2025

  9. [9]

    Weight uncertainty in neural network

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. InICML, 2015. 2

    2015

  10. [10]

    Popqa: A ques- tion answering benchmark for evaluating the factual consis- tency of language models

    Yapei Chang, Hangfeng He, and Dan Roth. Popqa: A ques- tion answering benchmark for evaluating the factual consis- tency of language models. InNAACL, 2023. 6, 1, 3

    2023

  11. [11]

    INSIDE: LLMs’ internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: LLMs’ internal states retain the power of hallucination detection. InICLR,

  12. [12]

    Uncertainty quantification of large language models through multi-dimensional responses

    Tiejin Chen, Xiaoou Liu, Longchao Da, Jia Chen, Vagelis Papalexakis, and Hua Wei. Uncertainty quantification of large language models through multi-dimensional responses. arXiv preprint arXiv:2502.16820, 2025. 2

    Pith/arXiv arXiv 2025

  13. [13]

    LM vs LM: Detecting factual errors via cross examination

    Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: Detecting factual errors via cross examination. InEMNLP, 2023. 3

    2023

  14. [14]

    I don't know: Explicit modeling of uncertainty with an [idk] token

    Roi Cohen, Konstantin Dobler, Eden Biran, and Gerard de Melo. I don't know: Explicit modeling of uncertainty with an [idk] token. InNeurIPS, 2024. 3

    2024

  15. [15]

    Laplace redux-effortless bayesian deep learning

    Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. InNeurIPS,

  16. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, 2009. 1

    2009

  17. [17]

    Eigentrack: Spectral activation feature track- ing for hallucination and out-of-distribution detection in llms and vlms.arXiv preprint arXiv:2509.15735, 2025

    Davide Ettori, Nastaran Darabi, Sina Tayebati, Ranganath Krishnan, Mahesh Subedar, Omesh Tickoo, and Amit Ran- jan Trivedi. Eigentrack: Spectral activation feature track- ing for hallucination and out-of-distribution detection in llms and vlms.arXiv preprint arXiv:2509.15735, 2025. 3, 4

    arXiv 2025

  18. [18]

    Detecting hallucinations in large language models using semantic entropy.Nature, 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 2024. 2, 3, 4, 5, 6, 7, 1

    2024

  19. [19]

    Tradi: Tracking deep neu- ral network weight distributions

    Gianni Franchi, Andrei Bursuc, Emanuel Aldea, Séverine Dubuisson, and Isabelle Bloch. Tradi: Tracking deep neu- ral network weight distributions. 2020. 3

    2020

  20. [20]

    Ensembling pruned attention heads for uncertainty-aware efficient transformers

    Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, and Gianni Franchi. Ensembling pruned attention heads for uncertainty-aware efficient transformers. arXiv preprint arXiv:2510.18358, 2025. 3

    Pith/arXiv arXiv 2025

  21. [21]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InICML, 2016. 2

    2016

  22. [22]

    SPUQ: Perturbation-based uncertainty quantification for large language models

    Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. SPUQ: Perturbation-based uncertainty quantification for large language models. InEACL, 2024. 3

    2024

  23. [23]

    Im- proving uncertainty quantification in large language models via semantic embeddings.arXiv preprint arXiv:2410.22685,

    Yashvir S Grewal, Edwin V Bonilla, and Thang D Bui. Im- proving uncertainty quantification in large language models via semantic embeddings.arXiv preprint arXiv:2410.22685,

    Pith/arXiv arXiv

  24. [24]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. InICML, 2017. 6, 1

    2017

  25. [25]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 2025. 1

    2025

  26. [26]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InCVPR, 2018. 6, 1

    2018

  27. [27]

    {DEBERTA}: {DECODING}-{enhanced} {bert} {with} {disentangled} {attention}

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. {DEBERTA}: {DECODING}-{enhanced} {bert} {with} {disentangled} {attention}. InInternational Confer- ence on Learning Representations, 2021. 2

    2021

  28. [28]

    A baseline for detect- ing misclassified and out-of-distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. InICLR, 2017. 6, 1

    2017

  29. [29]

    Prob- abilistic backpropagation for scalable learning of bayesian neural networks

    Jose Miguel Hernandez-Lobato and Ryan Adams. Prob- abilistic backpropagation for scalable learning of bayesian neural networks. InICML, 2015. 2

    2015

  30. [30]

    Imagenette: A smaller subset of 10 eas- ily classified classes from imagenet.https://github

    Jeremy Howard. Imagenette: A smaller subset of 10 eas- ily classified classes from imagenet.https://github. com/fastai/imagenette, 2019. 6

    2019

  31. [31]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions.TIS, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions.TIS, 2025. 1 9

    2025

  32. [32]

    The illusion of progress: Re-evaluating hallucination detec- tion in LLMs

    Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, and Tomasz Jan Kajdanowicz. The illusion of progress: Re-evaluating hallucination detec- tion in LLMs. InEMNLP, 2025. 2, 4, 5, 6

    2025

  33. [33]

    Survey of hallucination in natural language generation.CS, 2023

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.CS, 2023. 1

    2023

  34. [34]

    Calibrating verbal uncertainty as a linear feature to reduce hallucinations.arXiv preprint arXiv:2503.14477, 2025

    Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, An- thony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. Calibrating verbal uncertainty as a linear feature to reduce hallucinations.arXiv preprint arXiv:2503.14477, 2025. 3

    Pith/arXiv arXiv 2025

  35. [35]

    Cleanse: Uncertainty estima- tion approach using clustering-based semantic consistency in LLMs

    Minsuh Joo and Hyunsoo Cho. Cleanse: Uncertainty estima- tion approach using clustering-based semantic consistency in LLMs. InACL Workshops, 2025. 2, 3

    2025

  36. [36]

    Weld, and Luke Zettle- moyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettle- moyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InACL, 2017. 6, 1, 3

    2017

  37. [37]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022. 3, 1

    Pith/arXiv arXiv 2022

  38. [38]

    Semantic entropy probes: Robust and cheap hallucination detection in llms

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927, 2024. 2

    Pith/arXiv arXiv 2024

  39. [39]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6, 1

    2009

  40. [40]

    Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation. InICLR, 2023. 6, 1

    2023

  41. [41]

    Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation. InICLR, 2023. 2, 3, 4, 5, 6, 1

    2023

  42. [42]

    Simple and scalable predictive uncertainty esti- mation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. InNeurIPS, 2017. 2, 3

    2017

  43. [43]

    Uncertainty quantification for multimodal large language models with coherence-adjusted semantic volume, 2025

    Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin, and Bryan Kian Hsiang Low. Uncertainty quantification for multimodal large language models with coherence-adjusted semantic volume, 2025. 2, 3, 4, 5, 6, 7, 1

    2025

  44. [44]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 2018. 6, 1

    2018

  45. [45]

    What matters when building vision-language models? InNeurIPS, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? InNeurIPS, 2024. 6

    2024

  46. [46]

    Packed-ensembles for efficient uncertainty estima- tion.arXiv preprint arXiv:2210.09184, 2022

    Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geof- frey Daniel, Jean-Marc Martinez, Andrei Bursuc, and Gianni Franchi. Packed-ensembles for efficient uncertainty estima- tion.arXiv preprint arXiv:2210.09184, 2022. 3

    arXiv 2022

  47. [47]

    A symmetry-aware exploration of bayesian neural network posteriors.arXiv preprint arXiv:2310.08287, 2023

    Olivier Laurent, Emanuel Aldea, and Gianni Franchi. A symmetry-aware exploration of bayesian neural network posteriors.arXiv preprint arXiv:2310.08287, 2023. 2

    Pith/arXiv arXiv 2023

  48. [48]

    Efficient latent semantic clustering for scaling test-time computation of llms, 2025

    Sungjae Lee, Hoyoung Kim, Jeongyeon Hwang, Eunhyeok Park, and Jungseul Ok. Efficient latent semantic clustering for scaling test-time computation of llms, 2025. 2, 3

    2025

  49. [49]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küt- tler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. InNeurIPS, 2020. 5

    2020

  50. [50]

    Inference-time intervention: Elic- iting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Elic- iting truthful answers from a language model. InNeurIPS,

  51. [51]

    Adversarial vqa: A new benchmark for evaluating the robustness of vqa models

    Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial vqa: A new benchmark for evaluating the robustness of vqa models. InICCV, 2021. 6, 1

    2021

  52. [52]

    Seman- tic volume: Quantifying and detecting both external and in- ternal uncertainty in llms.arXiv preprint arXiv:2502.21239,

    Xiaomin Li, Zhou Yu, Ziji Zhang, Yingying Zhuang, Swair Shah, Narayanan Sadagopan, and Anurag Beniwal. Seman- tic volume: Quantifying and detecting both external and in- ternal uncertainty in llms.arXiv preprint arXiv:2502.21239,

    arXiv

  53. [53]

    Teaching models to express their uncertainty in words.TMLR, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words.TMLR, 2022. 3

    2022

  54. [54]

    Generat- ing with confidence: Uncertainty quantification for black- box large language models.TMLR, 2024

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generat- ing with confidence: Uncertainty quantification for black- box large language models.TMLR, 2024. 3, 1

    2024

  55. [55]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR,

  56. [56]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 1

    Pith/arXiv arXiv 2024

  57. [57]

    Probable networks and plausible predictions-a review of practical bayesian methods for su- pervised neural networks.Network: computation in neural systems, 1995

    David JC MacKay. Probable networks and plausible predictions-a review of practical bayesian methods for su- pervised neural networks.Network: computation in neural systems, 1995. 2

    1995

  58. [58]

    CalTech, 1992

    David John Cameron Mackay.Bayesian methods for adap- tive models. CalTech, 1992. 2

    1992

  59. [59]

    A simple baseline for bayesian uncertainty in deep learning.NeurIPS, 2019

    Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning.NeurIPS, 2019. 3

    2019

  60. [60]

    Uncertainty estimation in autoregressive structured prediction

    Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. InICLR, 2021. 2, 3

    2021

  61. [61]

    Self- CheckGPT: Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. Self- CheckGPT: Zero-resource black-box hallucination detection for generative large language models. InEMNLP, 2023. 3, 5

    2023

  62. [62]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InCVPR, 2019. 6, 1

    2019

  63. [63]

    MIT press, 2012

    Kevin P Murphy.Machine learning: a probabilistic perspec- tive. MIT press, 2012. 2

    2012

  64. [64]

    Correcting length bias in neural machine translation

    Kenton Murray and David Chiang. Correcting length bias in neural machine translation. InMT, 2018. 3

    2018

  65. [65]

    Springer Science & Business Media, 2012

    Radford M Neal.Bayesian learning for neural networks. Springer Science & Business Media, 2012. 2 10

    2012

  66. [66]

    Be- yond semantic entropy: Boosting LLM uncertainty quantifi- cation with pairwise semantic similarity

    Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman. Be- yond semantic entropy: Boosting LLM uncertainty quantifi- cation with pairwise semantic similarity. InACL, 2025. 2

    2025

  67. [67]

    Kernel language entropy: Fine-grained uncer- tainty quantification for llms from semantic similarities

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-grained uncer- tainty quantification for llms from semantic similarities. In NeurIPS, 2024. 2, 3, 7, 1

    2024

  68. [68]

    Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131,

    Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emi- lien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131,

    Pith/arXiv arXiv

  69. [69]

    LLMs know more than they show: On the intrinsic representation of LLM hallucinations

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. InICLR, 2025. 2, 3, 4

    2025

  70. [70]

    Semantic density: Uncer- tainty quantification for large language models through con- fidence measurement in semantic space

    Xin Qiu and Risto Miikkulainen. Semantic density: Uncer- tainty quantification for large language models through con- fidence measurement in semantic space. InNeurIPS, 2024. 3

    2024

  71. [71]

    Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press,

  72. [72]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019. 5, 6, 2

    Pith/arXiv arXiv 1908

  73. [73]

    Out- of-distribution detection and selective generation for condi- tional language models

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Moham- mad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out- of-distribution detection and selective generation for condi- tional language models. InICLR, 2023. 3

    2023

  74. [74]

    A scal- able laplace approximation for neural networks

    Hippolyt Ritter, Aleksandar Botev, and David Barber. A scal- able laplace approximation for neural networks. InICLR,

  75. [75]

    MIT press, 2002

    Bernhard Schölkopf and Alexander J Smola.Learning with kernels: support vector machines, regularization, optimiza- tion, and beyond. MIT press, 2002. 5

    2002

  76. [76]

    Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, and Yann LeCun. An information the- ory perspective on variance-invariance-covariance regular- ization. InNeurIPS, 2023. 4

    2023

  77. [77]

    Layer by layer: Uncovering hidden representations in lan- guage models

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Pa- tel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in lan- guage models. InICML, 2025. 5

    2025

  78. [78]

    Llm-check: Investigating detection of hallucinations in large language models

    Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasi- van, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. Llm-check: Investigating detection of hallucinations in large language models. InNeurIPS, 2024. 2, 4, 5

    2024

  79. [79]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024. 1

    2024

  80. [80]

    The llama 3 herd of models, 2024

    Meta team. The llama 3 herd of models, 2024. 3

    2024

Showing first 80 references.