Pith. sign in

REVIEW 4 major objections 6 minor 90 references

Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that perturbing a multimodal prompt and measuring the semantic spread of the model's answers yields an uncertainty estimate proportional to the model's parameter uncertainty, and that this estimate improves hallucination…

desk verdict Wide empirical sweep of perturbation-based uncertainty for LMMs, but the central theoretical proposition that makes it look principled does not survive contact with the protocol. read the letter →

arxiv 2506.07575 v1 pith:IGKQ2FTX submitted 2025-06-09 cs.CV cs.LG

Ruiyang Zhang , Hu Zhang , Hao Fei , Zhedong Zheng This is my paper
classification cs.CVcs.LG
keywords largemultimodalmodelsuncertaintyestimationhallucinationdetectionsemanticentropypromptperturbationreasoningcalibrationchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Uncertainty-o's pith: the uncertainty of a large multimodal model can be estimated from outside the model by perturbing the prompt and measuring how much the meaning of the answers spreads. The paper claims that this spread, quantified as the entropy of semantic clusters of sampled responses, is proportional to the model's epistemic uncertainty, and it formalizes that claim as $D(x) \propto \sqrt{\mathrm{Var}(\theta|x)}$. Across 18 benchmarks spanning image, video, audio, point cloud, and text, and 10 open- and closed-source LMMs, the method is reported to beat confidence elicitation and semantic entropy baselines on hallucination detection, hallucination mitigation, and uncertainty-aware chain-of-thought. A sympathetic reader would care because the recipe needs no access to model internals and no special training, so it could apply to any current or future multimodal model.

What carries the argument

The machinery has three pieces: (1) multimodal prompt perturbation, which generates semantic-preserving variations of the input in text, image, video, audio, and point cloud; (2) semantic extraction, where an off-the-shelf captioner converts non-text answers into text captions; and (3) multimodal semantic uncertainty, the normalized entropy $u_m = -\sum_i p_i \log p_i$ over semantic clusters of the sampled answers, Eq. (3). The entropy plays the role of the estimated uncertainty, and Proposition 3.2 supplies the justification by linking answer distance to $\sqrt{\mathrm{Var}(\theta|x)}$.

What would settle it

Take a set of prompts where the model's true parameter uncertainty is known, e.g., by training an ensemble of identical LMMs on the same data and measuring variance across members; if the semantic entropy from prompt perturbation does not rank-order the inputs in the same order as the ensemble variance, Proposition 3.2's link is broken. A cheaper check: highly unambiguous image-text pairs with consistent answers across many paraphrases should receive near-zero uncertainty; a substantial score there indicates the signal is perturbation-induced rather than epistemic.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that the uncertainty of an LMM can be read off from the semantic scatter of its responses to perturbed versions of the same prompt. The paper defines response semantic distance $D(x)=\|y_i-y_j\|$, derives $D(x) \propto \sqrt{\mathrm{Var}(\theta|x)}$, and operationalizes $D(x)$ as the entropy over semantic clusters of sampled answers. The claim is that this multimodal semantic uncertainty estimates epistemic uncertainty reliably across models with different architectures, modalities, and capability focuses, and that reliable estimates translate into better hallucination detection, hallucination mitigation, and uncertainty-aware chain-of-thought reasoning.

Load-bearing premise

The whole argument assumes that perturbing a prompt behaves like drawing fresh model parameters from a posterior, so that scattered answers reflect parameter uncertainty rather than just prompt sensitivity.

Editorial extensions

If this is right

  • If the proportionality claim holds, the same perturbation-and-entropy recipe can be reused as new modalities are added, as long as a captioner exists that can turn answers into text.
  • Hallucination detection becomes a ranking problem on estimated uncertainty, with the paper's recipe outperforming verbalized confidence and prior semantic entropy on AUROC, AURAC, and ECE.
  • Uncertainty scores can be fed back into the prompt to trigger self-correction, which the paper shows raises accuracy on video and audio QA and lengthens chain-of-thought reasoning.
  • The empirical cookbook—semantic-preserving perturbations, simultaneous multimodal perturbation, progressive pairing, and moderate sampling—provides a practical default recipe for downstream users.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper does not draw: if $D(x) \propto \sqrt{\mathrm{Var}(\theta|x)}$ truly measures parameter uncertainty, then the same uncertainty score should track the divergence between an ensemble of independently trained models on the same prompt; that experiment would separate epistemic from aleatoric sources.
  • The reliance on a captioner and an LLM for clustering means the uncertainty estimate is partly a property of those auxiliary models; swapping captioners should change the score, and comparing estimates across captioners would reveal how much of the signal is genuinely in the LMM's responses.
  • One could extend the method to calibration-based rejection rules in safety-critical multimodal settings, where the cost of a wrong answer is high and the current paper only reports detection and revision benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Uncertainty-o, a model-agnostic framework for estimating uncertainty of large multimodal models (LMMs) through multimodal prompt perturbation and semantic clustering of the resulting responses. The authors state three contributions: (1) a unified, model-agnostic uncertainty estimation pipeline; (2) an empirical exploration of prompt perturbations across five modalities; and (3) a derived formulation of multimodal semantic uncertainty. The central theoretical claim is Proposition 3.2, which states that the response semantic distance D(x) between two perturbed prompts is proportional to the square root of epistemic uncertainty Var(θ|x). The method is evaluated on 18 benchmarks and 10 LMMs for hallucination detection, hallucination mitigation, and uncertainty-aware chain-of-thought reasoning.

Significance. If the proposed framework is treated as a practical heuristic, the empirical scope is substantial: it spans five modalities, includes open- and closed-source LMMs, and shows consistent directional improvements over strong baselines on the three downstream tasks. The perturbation cookbook insights (semantic-preserving perturbations are better, simultaneous multimodal perturbation helps, moderate sampling time is optimal) are potentially useful practical guidance. The paper does not, however, provide a valid derivation justifying the epistemic-uncertainty interpretation of the proposed entropy measure; the proof of Proposition 3.2 rests on an assumption that is not satisfied by the protocol. The experimental evidence also lacks statistical precision, with no error bars or significance testing. If the theoretical claim is removed or replaced, the paper can stand as an empirical study, but the current presentation overstates the theoretical grounding.

major comments (4)
  1. [Supp Sec 9.2] The derivation of Proposition 3.2 relies on the assumption that the model parameters θ_i and θ_j corresponding to two perturbed prompts are i.i.d. draws from the same posterior P(θ|x). In the actual protocol described in Sec 2.1, prompt perturbation changes only the input x_i and x_j; the LMM's weights θ are fixed. For a deterministic model, θ_i = θ_j = θ, so D(x) = 0 for any Var(θ|x); for stochastic decoding, the response variation comes from sampling noise in P(y|x, θ) with θ fixed, which is aleatoric rather than epistemic parameter uncertainty. Therefore Eq. (2), D(x) ∝ sqrt(Var(θ|x)), is not established. This is load-bearing because Proposition 3.2 is the stated justification for interpreting the entropy in Eq. (3) as an estimator of epistemic uncertainty.
  2. [Supp Sec 9.2, posterior predictive equation] The derivation uses a posterior P(θ|x) for the model parameters conditioned on the prompt x. In Bayesian inference, the posterior is P(θ|D) over training data; conditioning on a test prompt is not defined without an explicit model of how θ depends on x. Since the paper does not define such a model, the chain from the posterior predictive distribution P(y|x) = ∫ P(y|x,θ)P(θ|x)dθ to Var(θ|x) is not mathematically grounded. This is a second, independent defect in the proof of Eq. (2).
  3. [Sec 2.2, Eq. (3)] Multimodal semantic uncertainty is defined as the entropy over semantic clusters of perturbed responses. Consequently, the measure is, by construction, a function of response consistency under input perturbation. Without a valid Proposition 3.2, the claim that this entropy 'mine[s] LMM inherent epistemic uncertainty' is circular: the method measures agreement among sampled responses, not variance of model parameters. The paper should either provide a valid derivation linking this entropy to parameter uncertainty or explicitly reframe the contribution as a heuristic whose usefulness is demonstrated only empirically.
  4. [Tables 1-4, 6] The experimental section reports single runs without error bars, seeds, or significance tests. Several improvements over the strongest baselines are modest in absolute terms (e.g., Table 1 LLaVABench AUROC 68.0 vs. 65.8; Table 2 MSRVTTQA 60.1 vs. 56.8). Table 6 reports only the average number of reasoning steps for uncertainty-aware CoT and does not show any measure of answer correctness, so the claimed benefit of uncertainty-aware CoT is not established. If the theoretical support for the method is removed, the empirical evidence becomes the sole basis for the claims, and it needs statistical grounding.
minor comments (6)
  1. [Sec 2.1, Definition 3.1] The text contains typos and grammatical errors: 'The LMM M generate reponses' and 'reponses' should be corrected.
  2. [Sec 2.3, Eq. (4)] The notation in Eq. (4) is ambiguous: y_initial appears as a single answer in the equation, but the surrounding text refers to 'Y' as a set of selected answers. Please clarify whether the revision is applied to one answer or to each member of the selected set.
  3. [Supp Sec 7] The dataset name is given as 'ClothV2' in the supplementary material but 'ClothoV2' in the main text; also, Clotho is an audio captioning dataset rather than a QA benchmark, so the precise benchmark versions and task formulations should be specified.
  4. [Related work, Sec 4] VL-Uncertainty [82] is cited as the closest prior work using semantic-equivalent perturbations for MLLM uncertainty, but it is never compared against in the experiments. A direct comparison or an explicit explanation of why it is excluded is needed.
  5. [Fig. 2] Figure 2 is central to the claim that Uncertainty-o aligns uncertainty with error rates, but the binning procedure and any error bars are not described in the main text or the supplementary material.
  6. [Sec 3.1, Table 4] The text states that Uncertainty-o achieves '23.2% higher AUROC, 10% higher AURAC in the point cloud generation task,' but Table 4 shows 51.1 vs. 43.6 for AUROC and 44.2 vs. 38.8 for AURAC against the strongest listed baseline; the source of these percentages should be clarified.

Circularity Check

1 steps flagged · score 3.0 of 10

Proposition 3.2's proof imports the i.i.d. parameter-resampling assumption, making the theoretical link to epistemic uncertainty definitional; empirical validation remains independent.

  1. self definitional [Section 2.1 (Proposition 3.2) / Supplementary Sec. 9.2 (proof, 'Predictions from Perturbed Prompts' and i.i.d. step)]
    "Assume the parameters during i-th and j-th prompt perturbation are θi and θj, respectively, and they have the same prior distribution, i.e., P(θi)=P(θj). ... To simplify the analysis, assume θi and θj are independently and identically distributed (i.i.d.). Then ... With Eθ|x[(θi − θj)^2] = 2 · Var(θ|x), we have ... D(x) ≈ √(2k) · √Var(θ|x)."

    Proposition 3.2 claims that response distance under prompt perturbation measures parameter variance. The proof connects response distance to parameter variance only by assuming that the two perturbations correspond to independent posterior draws of the parameters. Given that assumption, D(x) is, by construction, a function of the parameter difference whose variance is 2Var(θ|x), so the conclusion D(x) ∝ sqrt(Var(θ|x)) is already contained in the premise. Operationally, prompt perturbation keeps the model weights fixed and changes only the input; there is no mechanism that redraws θ from P(θ|x). The theorem therefore does not derive the link from the framework's definitions; it assumes the target relation, and the entropy in Eq.

full rationale

The only load-bearing first-principles claim is Proposition 3.2, and its proof is circular in a narrow sense: the response distance D(x) becomes proportional to parameter variance only after assuming that prompt perturbations resample the model parameters i.i.d. from the posterior. Once that assumption is granted, the algebra E[(θi−θj)^2]=2Var(θ|x) makes the result follow by definition, so the theorem's content is already in the premise. The actual protocol keeps the weights fixed, so the assumption is not satisfied and the claimed derivation of epistemic uncertainty from perturbed-response entropy is unsupported. However, the framework is not merely a renamed fit: the entropy score is defined independently of hallucination labels and is tested against external hallucination labels on 18 benchmarks, providing non-circular empirical support. The self-citation [82] (VL-Uncertainty) is related work and is not used as a load-bearing authority. Overall, the circularity is partial and concentrated in the theoretical justification, not in the empirical validation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework pulls several assumptions from nowhere: that perturbation-induced response variability equals parameter uncertainty, that semantic-preserving perturbations exist and are known, and that off-the-shelf captioners faithfully represent multimodal outputs. These are not free parameters fitted to data, but they are unverified domain assumptions. The main fitted quantities are sampling time, perturbation degree schedule, and perturbation types, all selected using the same evaluation benchmarks that are later used to report performance.

free parameters (4)
  • Sampling time T = 5 (default)
    Ablation on MSRVTTQA (Table 9) shows T=5 maximizes AUROC; chosen from the same evaluation set, so it is tuned to the benchmark rather than derived.
  • Perturbation degree schedule = 5 progressive levels (e.g., sampling coefficients 0.9 to 0.5 in Fig. 5)
    Chosen by hand; no principled rule; affects uncertainty scores and detection results.
  • Perturbation type per modality = e.g., speed change for video, volume for audio
    The paper's Cookbook 1 recommends semantic-preserving perturbations based on empirical comparison on evaluation benchmarks (Fig. 4); selecting the best performer on the test set is a form of fitting.
  • Top-K for hallucination mitigation = top 50% of high-uncertainty answers
    Chosen without ablation; a free design choice.
assumptions (5)
  • domain assumption Model parameters θ are random with posterior P(θ|x), and epistemic uncertainty is Var(θ|x).
    Invoked in Supp Sec 9.2 to define epistemic uncertainty; not generally true for neural networks where parameters are fixed after training.
  • ad hoc to paper θ_i and θ_j for two perturbed prompts are i.i.d. draws from the same posterior P(θ|x).
    Prompts are perturbed, not parameters; this is the step that makes the proof circular and unsupported.
  • ad hoc to paper E||A·(θi-θj)||² ≈ E||A||² · E(θi-θj)², with A = ∇θP(y|x,θ).
    Requires independence between the gradient and the parameter difference; this is not established and is generally false.
  • domain assumption Off-the-shelf captioner OneLLM-7B and LLM Qwen2.5-7B faithfully map and cluster multimodal semantics.
    Used in Sec 2.2; errors in captioning or clustering propagate directly into the uncertainty estimate.
  • domain assumption The listed perturbations (rotation, brightness, rephrasing, volume change) preserve the original prompt semantics.
    Used in Cookbook 1; e.g., rotation can alter orientation-dependent meaning, so the assumption is not universally valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models." pith.science (2026). https://pith.science/paper/IGKQ2FTX

@misc{pith2026250607575,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGKQ2FTX}},
  note         = {Machine review of arXiv:2506.07575}
}
read the original abstract

Large Multimodal Models (LMMs), harnessing the complementarity among diverse modalities, are often considered more robust than pure Language Large Models (LLMs); yet do LMMs know what they do not know? There are three key open questions remaining: (1) how to evaluate the uncertainty of diverse LMMs in a unified manner, (2) how to prompt LMMs to show its uncertainty, and (3) how to quantify uncertainty for downstream tasks. In an attempt to address these challenges, we introduce Uncertainty-o: (1) a model-agnostic framework designed to reveal uncertainty in LMMs regardless of their modalities, architectures, or capabilities, (2) an empirical exploration of multimodal prompt perturbations to uncover LMM uncertainty, offering insights and findings, and (3) derive the formulation of multimodal semantic uncertainty, which enables quantifying uncertainty from multimodal responses. Experiments across 18 benchmarks spanning various modalities and 10 LMMs (both open- and closed-source) demonstrate the effectiveness of Uncertainty-o in reliably estimating LMM uncertainty, thereby enhancing downstream tasks such as hallucination detection, hallucination mitigation, and uncertainty-aware Chain-of-Thought reasoning.

Figures

Figures reproduced from arXiv: 2506.07575 by the authors.

Figure 1
Figure 1. Comparison with Previous Works and Overview of Uncertainty-o. (a,b) Existing studies focus on single modality or two modalities [82]; (c) In contrast, our Uncertainty-o captures un￾certainty in large multimodal models in a model-agnostic manner. It achieves reliable uncertainty estimation via multimodal prompt perturbation. (d) In particular, we harnessing multimodal semantic uncertainty, which maps multimodal answe… view at source ↗
Figure 2
Figure 2. Reliability of Our Estimated Uncertainty. From comparison with previous methods, we observe that: (1) Compared to Confidence Elicitation [72], Uncertainty-o effectively avoids the over-confidence problem, which inaccurately assigns low uncertainty (or high confidence) to hallucinatory responses. (2) Compared to Semantic Entropy [24], Uncertainty-o provides more reliable uncertainty estimation, e.g., the uncertainty … view at source ↗
Figure 3
Figure 3. Pipeline of Our Uncertainty-o. Given a multimodal prompt and large multimodal models, we perform multimodal prompt perturbation to generate diverse responses. Due to the inherent epistemic uncertainty of these models under perturbation, varied responses are typically obtained. To quantify this uncertainty, we apply semantic clustering on the collected responses and compute their entropy. Specifically, responses are … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Empirical Comparison of Different Prompt Perturbations. On average, semantic-preserving perturbations are more effective for eliciting LMM uncertainty than semantic-altering ones. Results from comprehension hallucination detection for video, audio, point. Text Video Au…
Figure 5
Figure 5. Figure 5: Qualitative Results of Successful Hallucination Detection. Uncertainty-o is proficient at accurately detecting multiple halluci￾nations, including comprehension (point cloud QA [19]) and generation tasks (image generation [53]) based on the uncertainty value. range of …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 39 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023. 1, 8

    arXiv 2023

  2. [2]

    How many opinions does your llm have? improving uncertainty estimation in nlg

    Lukas Aichberger, Kajetan Schweighofer, Mykyta Ielanskyi, and Sepp Hochreiter. How many opinions does your llm have? improving uncertainty estimation in nlg. InICLR 2024 Workshop on Secure and Trustworthy Large Language Mod- els, 2024. 2

    2024

  3. [3]

    Multimodal automated fact-checking: A survey

    Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vla- chos. Multimodal automated fact-checking: A survey. arXiv:2305.13507, 2023. 8

    arXiv 2023

  4. [4]

    Knowledge of knowledge: Ex- ploring known-unknowns uncertainty with large language models

    Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Wang. Knowledge of knowledge: Ex- ploring known-unknowns uncertainty with large language models. arXiv:2305.13712, 2023. 1

    arXiv 2023

  5. [5]

    Predicting and understanding human action decisions during skillful joint-action using supervised machine learning and explainable-ai

    Fabrizia Auletta, Rachel W Kallen, Mario di Bernardo, and Michael J Richardson. Predicting and understanding human action decisions during skillful joint-action using supervised machine learning and explainable-ai. Scientific Reports, 13 (1):4992, 2023. 2

    2023

  6. [6]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 5

    2023

  7. [7]

    Halluci- nation of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Halluci- nation of multimodal large language models: A survey. arXiv:2404.18930, 2024. 8

    arXiv 2024

  8. [8]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

Show all 90 references
  1. [9]

    The revolution of multimodal large language models: a survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: a survey. arXiv:2402.12451, 2024. 1, 2, 7, 8

    2024

  2. [10]

    Rational use of cognitive resources in human planning

    Frederick Callaway, Bas van Opheusden, Sayan Gul, Priyam Das, Paul M Krueger, Thomas L Griffiths, and Falk Lieder. Rational use of cognitive resources in human planning. Na- ture Human Behaviour, 6(8):1112–1125, 2022. 2

    2022

  3. [11]

    A review of multi-modal large language and vision models

    Kilian Carolan, Laura Fennelly, and Alan F Smeaton. A review of multi-modal large language and vision models. arXiv:2404.01322, 2024. 1, 2

    2024 arXiv

  4. [12]

    Shapenet: An information-rich 3d model repository

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Mano- lis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv:1512.03012,

    arXiv

  5. [13]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 5

    2015 arXiv

  6. [14]

    Unified hallucination detection for multi- modal large language models

    Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xi- aoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multi- modal large language models. arXiv:2402.03190, 2024. 8

    2024 arXiv

  7. [15]

    Unveiling uncertainty: A deep dive into calibration and performance of multimodal large language models

    Zijun Chen, Wenbo Hu, Guande He, Zhijie Deng, Zheng Zhang, and Richang Hong. Unveiling uncertainty: A deep dive into calibration and performance of multimodal large language models. arXiv:2412.14660, 2024. 8

    2024 arXiv

  8. [16]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF conference on computer...

    2024

  9. [17]

    I don’t know: Explicit modeling of uncertainty with an [idk] token

    Roi Cohen, Konstantin Dobler, Eden Biran, and Gerard de Melo. I don’t know: Explicit modeling of uncertainty with an [idk] token. Advances in Neural Information Processing Systems, 37:10935–10958, 2024. 8

    2024

  10. [18]

    Human uncertainty in concept-based ai systems

    Katherine Maeve Collins, Matthew Barker, Mateo Es- pinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jam- nik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Et...

    2023

  11. [19]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pages 13142– 13153, 2023. 5, 7, 8

    2023

  12. [20]

    Retrieve only when it needs: Adaptive re- trieval augmentation for hallucination mitigation in large lan- guage models

    Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. Retrieve only when it needs: Adaptive re- trieval augmentation for hallucination mitigation in large lan- guage models. arXiv:2402.10612, 2024. 8

    2024

  13. [21]

    Clotho: An audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virta- nen. Clotho: An audio captioning dataset. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 736–740. IEEE,

    2020

  14. [22]

    Puma: Empowering unified mllm with multi- granular visual generation

    Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Hongsheng Li, and Xihui Liu. Puma: Empowering unified mllm with multi- granular visual generation. arXiv:2410.13861, 2024. 8

    2024 arXiv

  15. [23]

    From uncertainty to trust: Enhancing re- liability in vision-language models with uncertainty-guided dropout decoding

    Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, and Jiawei Zhou. From uncertainty to trust: Enhancing re- liability in vision-language models with uncertainty-guided dropout decoding. arXiv:2412.06474, 2024. 8

    2024

  16. [24]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024. 2, 4, 5, 6

    2024

  17. [25]

    Video-of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint arXiv:2501.03230, 2024. 8

    2024 arXiv

  18. [26]

    Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing

    Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing. arXiv preprint arXiv:2412.19806, 2024. 8

    2024 arXiv

  19. [27]

    Enhancing video-language representations with structural spatio-temporal alignment

    Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat- Seng Chua, and Shuicheng Yan. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 2024. 8

    2024

  20. [28]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, pages 15180–15190, 2023. 1, 8

    2023

  21. [29]

    Large language models respond to influence like humans

    Lewis Griffin, Bennett Kleinberg, Maximilian Mozes, Kim- berly Mai, Maria Do Mar Vau, Matthew Caldwell, and Au- gustine Mavor-Parker. Large language models respond to influence like humans. In Proceedings of the First Workshop on Social Influence in Conversations (SICon 2023) ...

    2023

  22. [30]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xi- angyu Yue. Onellm: One framework to align all modalities with language. In CVPR, pages 26584–26595, 2024. 1, 2, 5, 6, 8

    2024

  23. [31]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1

    2020

  24. [32]

    A survey on evaluation of multimodal large language models

    Jiaxing Huang and Jingyi Zhang. A survey on evaluation of multimodal large language models. arXiv:2408.15769,

    arXiv

  25. [33]

    Visual hallucinations of multi-modal large language models

    Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. Visual hallucinations of multi-modal large language models. arXiv:2402.14683, 2024. 8

    2024 arXiv

  26. [34]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5

    2024 arXiv

  27. [35]

    Map: Multimodal uncertainty-aware vision-language pre-training model

    Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and Yujiu Yang. Map: Multimodal uncertainty-aware vision-language pre-training model. arXiv:2210.05335, 2022. 8

    2022 arXiv

  28. [36]

    Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019. 5

    2019

  29. [37]

    Ex- ploring the frontiers of llms in psychological applications: A comprehensive review

    Luoma Ke, Song Tong, Peng Cheng, and Kaiping Peng. Ex- ploring the frontiers of llms in psychological applications: A comprehensive review. arXiv:2401.01519, 2024. 2

    2024 arXiv

  30. [38]

    Audiocaps: Generating captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1...

    2019

  31. [39]

    Audiogen: Textually guided audio gen- eration

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D ´efossez, Jade Copet, Devi Parikh, Yaniv Taig- man, and Yossi Adi. Audiogen: Textually guided audio gen- eration. arXiv:2209.15352, 2022. 1, 2

    2022 arXiv

  32. [40]

    Rgb2point: 3d point cloud generation from single rgb images

    Jae Joong Lee and Bedrich Benes. Rgb2point: 3d point cloud generation from single rgb images. arXiv:2407.14979, 2024. 2, 6, 8

    2024 arXiv

  33. [41]

    Unimo- g: Unified image generation through multimodal conditional diffusion

    Wei Li, Xue Xu, Jiachen Liu, and Xinyan Xiao. Unimo- g: Unified image generation through multimodal conditional diffusion. arXiv:2401.13388, 2024. 8

    2024 arXiv

  34. [42]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv:2305.10355, 2023. 8

    2023 arXiv

  35. [43]

    Video-llava: Learning united visual repre- sentation by alignment before projection.arXiv:2311.10122,

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection.arXiv:2311.10122,

    arXiv

  36. [44]

    Clotho-aqa: A crowd- sourced dataset for audio question answering

    Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowd- sourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140–1144. IEEE, 2022. 5, 6

    2022

  37. [45]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv:2306.14565, 2023. 5, 6, 8

    2023 arXiv

  38. [46]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892–34916, 2023. 1, 5, 6, 8

    2023

  39. [47]

    Videofusion: Decomposed diffusion models for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tie- niu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv:2303.08320, 2023. 1, 6, 8

    2023 arXiv

  40. [48]

    Unibind: Llm-augmented unified and balanced representa- tion space to bind them all

    Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, and Lin Wang. Unibind: Llm-augmented unified and balanced representa- tion space to bind them all. In CVPR, pages 26752–26762,

  41. [49]

    Openeqa: Embodied question answering in the era of foun- dation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In CVPR, pages 16488–16498, 2024. 5

    2024

  42. [50]

    Point-e: A system for generating 3d point clouds from complex prompts

    Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv:2212.08751,

    arXiv

  43. [51]

    Human uncertainty makes clas- sification more robust

    Joshua C Peterson, Ruairidh M Battleday, Thomas L Grif- fiths, and Olga Russakovsky. Human uncertainty makes clas- sification more robust. In ICCV, pages 9617–9626, 2019. 2

    2019

  44. [52]

    Mllm-protector: Ensuring mllm’s safety without hurting per- formance

    Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. Mllm-protector: Ensuring mllm’s safety without hurting per- formance. arXiv:2401.02906, 2024. 1

    2024 arXiv

  45. [53]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In ICCV, pages 2641–2649, 2015. 6, 8

    2015

  46. [54]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, pages 28492–28518. PMLR, 2023. 1

    2023

  47. [55]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv:1809.02156, 2018. 8

    2018 arXiv

  48. [56]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 1, 2, 6, 8

    2022

  49. [57]

    A survey on uncertainty quantification of large language models: Taxonomy, open research chal- lenges, and future directions

    Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research chal- lenges, and future directions. arXiv:2412.05563, 2024. 2

    2024 arXiv

  50. [58]

    Moma: Multimodal llm adapter for fast personalized image generation

    Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, and Xiao Yang. Moma: Multimodal llm adapter for fast personalized image generation. InECCV, pages 117– 132, 2024. 8

    2024

  51. [59]

    Pix3d: Dataset and methods for single-image 3d shape modeling

    Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In CVPR, pages 2974– 2983, 2018. 6

    2018

  52. [60]

    Any-to-any generation via composable diffu- sion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffu- sion. Advances in Neural Information Processing Systems , 36:16083–16099, 2023. 8

    2023

  53. [61]

    Codi-2: In-context in- terleaved and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context in- terleaved and interactive any-to-any generation. In CVPR, pages 27425–27434, 2024. 8

    2024

  54. [62]

    Evaluating the evaluation of diversity in natural language generation

    Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. arXiv:2004.02990,

    2004 arXiv

  55. [63]

    Evaluation and analysis of halluci- nation in large vision-language models

    Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of halluci- nation in large vision-language models. arXiv:2308.15126,

    arXiv

  56. [64]

    Uncertainty aware learning for language model alignment

    Yikun Wang, Rui Zheng, Liang Ding, Qi Zhang, Dahua Lin, and Dacheng Tao. Uncertainty aware learning for language model alignment. arXiv:2406.04854, 2024. 8

    2024 arXiv

  57. [65]

    Multimodal large language models: A sur- vey

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: A sur- vey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE, 2023. 8

    2023

  58. [66]

    Visual prompting in multimodal large language models: A survey

    Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multimodal large language models: A survey. arXiv:2409.15310, 2024. 8

    2024 arXiv

  59. [67]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. In ICML,

  60. [68]

    Self-correcting llm-controlled diffusion models

    Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffusion models. In CVPR, pages 6327–6336, 2024. 1

    2024

  61. [69]

    Can graph learning improve planning in llm-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, et al. Can graph learning improve planning in llm-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 8

    2024

  62. [70]

    3d shapenets: A deep representation for volumetric shapes

    Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Lin- guang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, pages 1912–1920, 2015. 5, 6

    1912

  63. [71]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, pages 9777–9786, 2021. 5

    2021

  64. [72]

    Can llms express their un- certainty? an empirical evaluation of confidence elicitation in llms

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their un- certainty? an empirical evaluation of confidence elicitation in llms. arXiv:2306.13063, 2023. 2, 4, 5

    2023 arXiv

  65. [73]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM international confer- ence on Multimedia, pages 1645–1653, 2017. 5, 6, 7

    2017

  66. [74]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288–5296, 2016. 6

    2016

  67. [75]

    Pointllm: Empowering large lan- guage models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointllm: Empowering large lan- guage models to understand point clouds. In ECCV, pages 131–147, 2024. 1, 5, 6, 8

    2024

  68. [76]

    Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92)

    Junichi Yamagishi, Christophe Veaux, and Kirsten MacDon- ald. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019. 6

    2019

  69. [77]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv:2308.02490, 2023. 5, 6

    2023 arXiv

  70. [78]

    Anygpt: Unified multimodal llm with dis- crete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with dis- crete sequence modeling. arXiv:2402.12226, 2024. 1, 6, 8

    2024 arXiv

  71. [79]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 5, 6

    2023 arXiv

  72. [80]

    Approaching outside: scaling unsupervised 3d object detec- tion from 2d scene

    Ruiyang Zhang, Hu Zhang, Hang Yu, and Zhedong Zheng. Approaching outside: scaling unsupervised 3d object detec- tion from 2d scene. In ECCV, pages 249–266, 2024. 8

    2024

  73. [81]

    Harnessing uncertainty-aware bounding boxes for unsuper- vised 3d object detection

    Ruiyang Zhang, Hu Zhang, Hang Yu, and Zhedong Zheng. Harnessing uncertainty-aware bounding boxes for unsuper- vised 3d object detection. arXiv:2408.00619, 2024. 8

    2024 arXiv

  74. [82]

    Vl- uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation

    Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl- uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation. arXiv:2411.11919, 2024. 1, 8

    2024 arXiv

  75. [83]

    Prompt highlighter: Interactive control for multi- modal llms

    Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi- modal llms. In CVPR, pages 13215–13224, 2024. 8 Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models Supplementary Material

    2024

  76. [84]

    As a result, any fluctuation in answers of LMM directly reflects its uncertainty

    Dicussion Why is semantic-equivalent perturbation more effective than inequivalent perturbation? In semantic-equivalent perturbation, the semantics of all perturbed prompts re- main the same. As a result, any fluctuation in answers of LMM directly reflects its uncertainty. On ...

  77. [85]

    We conduct experiments with 18 benchmarks

    Setting Benchmark. We conduct experiments with 18 benchmarks. For the comprehension hallucination task, the benchmarks we adopt are typically QA-type. For images, we utilize MMVet, LLaV ABench, and CoCoCap. For video bench- marks, we leverage MSRVTTQA, MSVDQA, and Nex- tQA. Fo...

  78. [86]

    We report the ablation of text clustering methods (see Tab

    More Ablations Text Clustering Manner. We report the ablation of text clustering methods (see Tab. 10). Clustering text based on its inherent semantics, rather than merely its lexical presen- tation, yields the best results. Cycle Consistency. We also experiment with cycle con...

  79. [87]

    Definitions and Assumptions

    Detailed Proof for Proportional Theorem 9.1. Definitions and Assumptions

  80. [88]

    For those input prompts, the predictions from the large model is yi = M (xi), and yj = M (xj), respectively

    Sampling Process: The large model M give predictions concurrently for perturbed prompts xi and xj. For those input prompts, the predictions from the large model is yi = M (xi), and yj = M (xj), respectively

  81. [89]

    Assume the large model parameters θ are random variables with a prior distribution P (θ)

    Uncertainty: We focus on the large model’s Epistemic Uncertainty, which is the uncertainty in the large model parameters. Assume the large model parameters θ are random variables with a prior distribution P (θ)

  82. [90]

    Prediction Difference: Define the prediction difference D(x) as: D(x) =∥yi − yj∥, where ∥ · ∥denotes semantic space distance. 9.2. Mathematical Derivation Large Model’s Predictive Distribution. Assume the large model’s output is a probability distribution P (y|x, θ), where y i...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.