Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multimodal large language models remain overconfident across fine-tuning, but prompt adjustments and temperature scaling improve calibration.

desk verdict Useful empirical map of MLLM calibration, but the headline claims rest on unsupported statistics and a self-referential IDK oracle; deserves referee time but needs major revisions. read the letter →

arxiv 2412.14660 v2 pith:MI25ZGKS submitted 2024-12-19 cs.CV cs.AIcs.CLcs.LGstat.ML

Zijun Chen , Wenbo Hu , Guande He , Zhijie Deng , Zheng Zhang , Richang Hong This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LGstat.ML
keywords multimodallargelanguagemodelscalibrationuncertaintyquantificationexpectederrorIDKdatasetvisualquestionansweringoverconfidencepromptoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether multimodal large language models (MLLMs) can be trusted to know when they do not know, and whether their confidence tracks their accuracy. The authors find that MLLMs are persistently miscalibrated—typically overconfident—and that this overconfidence is stable: visual fine-tuning and multimodal training on top of a base language model do not significantly change calibration. Using a new dataset called IDK, built by having each model answer every question ten times, they show that models almost always give an answer rather than admit ignorance, although a simple prompt like 'if you don't know, say so' visibly improves self-assessment. They also report that MLLMs carry more uncertainty about image content than about text, and that adding textual information reduces overall uncertainty. The practical upshot is that calibration techniques such as temperature scaling and prompt optimization can make MLLMs more reliable in high-stakes applications.

What carries the argument

The IDK dataset construction is the load-bearing device: a model answers each question 10 times at temperature 1 and top-p 0.95, and a question is labeled 'doesn't know' only if none of the 10 answers is correct. This binary split, together with the TRUTHFUL metric (the fraction of items where the model's stated answer matches the know/don't-know label), carries the overconfidence claim. Calibration is measured by Expected Calibration Error (ECE) and its variants MCE and ENCE, with logits-based likelihood as the primary uncertainty score and semantic entropy for open-ended responses. The proposed fixes are temperature scaling on the predictive distribution and an iterative prompt-optimization loop that refines suffixes such as 'Answer:' into calibration-friendly versions.

What would settle it

Re-label the IDK questions with more samples (say 30 or 100) or with a lower accuracy threshold and recompute TRUTHFUL; if many questions labeled 'doesn't know' are answered correctly in 9 of 10 or 20 of 30 draws, the paper's overconfidence figures would be inflated.

Watch

Extended reading notes

Core claim

The central claim is that miscalibration is the default state of MLLMs, and it is stable across the training scenarios the paper studies: calibration does not significantly differ before versus after visual fine-tuning, nor between a base LLM and its multimodal counterpart on language tasks. At the same time, uncertainty is modality-dependent—image information produces higher uncertainty than text—and the two modalities combine to reduce uncertainty as more text is supplied. On the newly constructed IDK dataset, MLLMs overwhelmingly answer questions even when they do not know, but prompting them to say 'I don't know' substantially increases the truthfulness of their self-assessment. The paper also claims that temperature scaling and iterative prompt optimization reduce expected calibration error, with the combination sometimes working best.

Load-bearing premise

The load-bearing premise is that 10 correct answers out of 10 sampled attempts cleanly separates what a model knows from what it does not, so a model that knows an answer but occasionally samples a wrong response is counted as not knowing.

Editorial extensions

If this is right

  • Visual fine-tuning does not systematically hurt calibration, so practitioners can add vision to a language model without expecting a calibration penalty, though they should still plan for overconfidence.
  • Multimodal training leaves linguistic-task calibration roughly unchanged, meaning the base LLM's uncertainty behavior largely carries over to the MLLM.
  • Supplying reliable text alongside an image reduces model uncertainty even for heavily noised images, so textual context is a practical lever for trustworthy confidence.
  • Explicit 'if you don't know, say so' prompts raise TRUTHFUL scores substantially on both in-distribution unknowns and out-of-distribution news questions.
  • Temperature scaling and prompt tuning both reduce ECE, and combining them sometimes yields further improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 10-sample all-or-nothing rule likely underestimates what a model knows; a model that knows an answer but occasionally samples a wrong token is counted as ignorant, so the reported overconfidence rates are probably upper bounds.
  • The out-of-distribution dataset assumes July 2024 news is unknowable to models released earlier, but pretraining corpora can contain near-duplicate or analogous facts, so TRUTHFUL there may partly reflect generic refusal behavior rather than a clean test of knowledge boundaries.
  • The larger closed models improve more from prompting, which suggests instruction-following ability rather than calibration itself drives the gain; separating the two would require comparing prompted truthfulness with logit-based confidence on the same items.
  • The modality asymmetry suggests an untested intervention: explicitly suppressing confidence when only image evidence is present could improve calibration beyond what the paper's prompt tuning achieves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript presents an empirical study of uncertainty calibration in multimodal large language models. It compares ECE, MCE, and ENCE of LLaVA and Qwen-VL before and after visual fine-tuning (Tables 1 and 5-10) and relative to their base LLMs on linguistic tasks (Tables 2 and 11-14), concluding that MLLMs are miscalibrated but that calibration does not differ significantly across these scenarios. It then analyzes confidence from image versus text modalities and their integration (Section 4), constructs a model-specific IDK dataset and a July 2024 OOD news VQA dataset (Section 5), reports that MLLMs rarely say 'I don't know' and that prompting improves truthfulness metrics, and proposes temperature scaling and iterative prompt optimization (Section 6).

Significance. If the claims were supported, the paper would provide a useful negative result about fine-tuning and multimodal training not systematically degrading calibration, a new benchmark-style resource in the IDK and July24-NewsVQA datasets, and practical calibration recipes. The authors also release code and dataset links, which is a positive feature. However, the central empirical claims are currently under-supported: the 'no significant differences' conclusion lacks inferential statistics, the IDK oracle is circular and fragile, and the calibration gains are evaluated in-sample. The study's value is therefore conditional on a substantial revision that either strengthens the evidence or weakens the claims.

major comments (4)
  1. [Section 3 and Abstract] The abstract and Section 3 state that there are "no significant differences in calibration across these scenarios," but the paper reports only point estimates of accuracy, confidence, ECE, MCE, and ENCE on randomly sampled subsets whose sizes are never stated, with no significance tests, confidence intervals, or error bars. For instance, Table 1 shows LLaVA-13B Stage 1 ECE of 0.235 versus Stage 2 ECE of 0.101 on MMBench; without an uncertainty estimate, the claim that such differences are "not significant" cannot be assessed. Please report sample sizes, bootstrap confidence intervals or Bayesian credible intervals, and an explicit test or power analysis for the negative claim.
  2. [Section 5, Construction of dataset] The model-specific IDK split labels a question as "known" only if all 10 stochastic samples at temperature 1 and top-p 0.95 are correct (threshold 1). For a question the model actually knows with per-sample accuracy p, the probability of being labeled "known" is p^10, so a known question answered correctly 90% of the time has only a 35% chance of being labeled "known." The TRUTHFUL metric in Tables 3 and 4 therefore treats known-but-noisy items as "unknown" and counts prompted refusals on those items as improved self-assessment. Because the abstract's claim that "self-assessment improves with proper prompt adjustments" is evaluated mainly against this oracle, that claim is not currently supported. Please validate the oracle against an independent criterion (for example, human labels or a calibrated threshold) or restrict the self-assessment claim to the OOD data.
  3. [Section 6, Eq. (2), Algorithm 1, Fig. 4] Temperature scaling and iterative prompt optimization are fit and evaluated on the same data. Equation (2) minimizes the negative log-likelihood over M samples, and Fig. 4 reports post-calibration ECE on MMBench without stating whether T was optimized on a separate validation split. Algorithm 1 selects suffixes by accuracy and ECE using the same evaluation function f, so the reported improvements are partly in-sample selection artifacts. Please use a held-out test set or cross-validation, and report the selection bias; in particular, state how the hyperparameters k, m, and n were chosen.
  4. [Section 4, Figs. 1-3] The modality uncertainty comparison relies on GPT-4V text descriptions as faithful proxies for image content, but this assumption is not verified against the actual image (Fig. 1), so text-versus-image confidence differences may reflect description quality rather than modality properties. Additionally, Fig. 2 reports point estimates from unspecified subsets without error bars or repeated sampling. Please verify the fidelity of the descriptions, report sample sizes, and provide uncertainty estimates before concluding that MLLMs have lower uncertainty on text than on images.
minor comments (6)
  1. [Section 2.1 and Section 7] There are typos: "LMMs" in Section 2.1 should be "MLLMs," and "adversial" in Section 7 should be "adversarial."
  2. [Algorithm 1] Line 3 is malformed: "S ← p∈G generate(p, m)" should be written as a set comprehension, e.g., S ← {generate(p, m) : p ∈ G}.
  3. [Eq. (2)] The symbol |y| is used without being defined, and M in Eq. (2) overloads the M that denotes the number of bins in Eq. (1); please use different symbols for the number of samples and the number of classes.
  4. [Figure 3 caption] The caption uses "N (0, N oise)" and "NoisyImage=Image+N (0, N oise)" without stating whether "Noise" is the variance or the standard deviation of the Gaussian perturbation.
  5. [Section 5 and Table 3] The four categories IK-IDK, IDK-IDK, IK-IK, and IDK-IK are introduced only in the table; please define each label combination explicitly in the text before using the acronyms.
  6. [Appendix D] There is an incomplete sentence in Appendix D: "The dataset con-" should be completed or removed.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: IDK ground truth is derived from the same model's 10-sample accuracy, and calibration techniques are tuned and evaluated on the same benchmark, so two headline improvements are partly artifacts of their own evaluation criteria.

  1. self definitional [Section 5, "Construction of dataset"; Table 3]
    "For a given question, a model answers 10 times. Based on an accuracy threshold (we use 1, meaning all correct), we classify whether the model “knows" or “doesn’t know." This method segments the existing dataset into IDK and IK categories. ... TRUTHFUL = (IK-IDK + IK-IK) / (IK-IDK + IDK-IDK + IK-IK + IDK-IK)"

    The ground-truth label "doesn't know" is defined by the same model's 10 stochastic answers with a perfect-accuracy threshold, so the IDK set is model-generated and threshold-dominated: an item answered correctly with per-sample probability 0.9 has only a 0.35 chance of being labeled "known." The TRUTHFUL metric then counts an "I don't know" response on these model-derived IDK items as correct self-assessment. Prompting increases "I don't know" responses, which mechanically raises TRUTHFUL on the many known-but-noisy items mislabeled IDK. The claimed improvement in self-assessment is therefore partly determined by the labeling procedure rather than by genuine knowledge or uncertainty.

  2. fitted input called prediction [Section 6.1, Eq. (2); Section 6.2, Algorithm 1; Fig. 4]
    "TS optimizes the temperature T through the following objective: min_T ... (2) ... Despite its simplicity, TS is highly effective in calibrating MLLMs, especially in models with poor initial calibration, as shown in Fig.4. ... Evaluate Gnew: E ← {(p, f(p)) | p ∈ Gnew}; Sort E by accuracy bands and ECE; Update G with top k prompts from E ... return best prompt."

    The temperature T and the prompt suffix are selected by optimizing the same ECE/NLL objective on MMBench, and Fig. 4 reports the post-calibration ECE for models "tested on MMBench." No held-out split is described. The reported improvement is thus an in-sample fit: Algorithm 1 explicitly returns the prompt with the best ECE on the evaluation set, so the resulting ECE is the value of the selection criterion itself, not an independent prediction of calibration gain.

full rationale

The paper's central observations about miscalibration and calibration stability (Sections 3–4) are empirical comparisons using standard ECE and do not reduce to their inputs; those findings are self-contained. The OOD news dataset provides a more independent test of refusal behavior, though the model-specific IDK split does not. The calibration-technique section, however, evaluates methods on the same benchmark used to fit them, and the IDK self-assessment metric uses a ground truth generated by the same model's sampling procedure. These are partial circularities: the core "MLLMs are miscalibrated" finding survives, but the claims that "self-assessment improves with proper prompt adjustments" and that the proposed TS/PT methods improve calibration are inflated by construction because the evaluation metrics are optimized or defined using the same data and model outputs that produced the reported results.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on a few fitted parameters (temperature, IDK threshold, APE hyperparameters) and several domain assumptions about how confidence should be measured and how knowledge is inferred. The IDK dataset is a new resource, but it is not an invented entity in the theoretical sense; no new physical or mathematical entities are postulated.

free parameters (3)
  • Temperature T in temperature scaling = not reported
    The paper optimizes T using Eq. (2) on the evaluation data (likely the test set) and then reports ECE after scaling; no held-out validation set is mentioned. This is a free parameter fitted to data.
  • IDK accuracy threshold = 1
    Section 5 uses an accuracy threshold of 1 (all 10 repeated answers must be correct) to classify whether the model 'knows' a question. This hand-chosen threshold determines all IDK labels.
  • APE hyperparameters k, m, n = not specified
    Algorithm 1 depends on top prompts k, similar prompts m, and iterations n, but the paper never gives values. These free choices affect the prompt optimization result and are not reported.
assumptions (5)
  • domain assumption ECE with 10 equal bins is a valid measure of calibration
    The paper uses ECE (Eq. 1) as the primary evaluation metric without justification for the bin count or the definition of calibration.
  • domain assumption Logits-based likelihood is a valid confidence signal
    Section 3 adopts logits-based likelihood as the uncertainty quantification method; this assumes the normalized probabilities over answer choices reflect the model's true confidence.
  • ad hoc to paper GPT-4V text descriptions faithfully represent image content
    Section 4.1 replaces images with GPT-4V-generated text descriptions to measure text uncertainty; the comparison between text and image uncertainty is only valid if the descriptions capture the same information as the images.
  • ad hoc to paper A model answering 10/10 times correctly 'knows' the answer
    Section 5 defines knowing by 10 repeated answers with perfect accuracy; this assumes sampling behavior cleanly separates knowledge from ignorance, ignoring cases where a model knows but sometimes samples wrong.
  • domain assumption July 2024 news is unknowable to models trained before that date
    The OOD dataset assumes the model cannot know answers to July 2024 news questions because the models were trained earlier; in reality, some answers might be inferable from images or general knowledge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/MI25ZGKS

@misc{pith2026241214660,
  author       = {Pith},
  title        = {Pith review of: Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MI25ZGKS}},
  note         = {Machine review of arXiv:2412.14660}
}
read the original abstract

Multimodal large language models (MLLMs) combine visual and textual data for tasks such as image captioning and visual question answering. Proper uncertainty calibration is crucial, yet challenging, for reliable use in areas like healthcare and autonomous driving. This paper investigates representative MLLMs, focusing on their calibration across various scenarios, including before and after visual fine-tuning, as well as before and after multimodal training of the base LLMs. We observed miscalibration in their performance, and at the same time, no significant differences in calibration across these scenarios. We also highlight how uncertainty differs between text and images and how their integration affects overall uncertainty. To better understand MLLMs' miscalibration and their ability to self-assess uncertainty, we construct the IDK (I don't know) dataset, which is key to evaluating how they handle unknowns. Our findings reveal that MLLMs tend to give answers rather than admit uncertainty, but this self-assessment improves with proper prompt adjustments. Finally, to calibrate MLLMs and enhance model reliability, we propose techniques such as temperature scaling and iterative prompt optimization. Our results provide insights into improving MLLMs for effective and responsible deployment in multimodal applications. Code and IDK dataset: https://github.com/hfutml/Calibration-MLLM.

Figures

Figures reproduced from arXiv: 2412.14660 by the authors.

Figure 2
Figure 2. Use logits-based likelihood to quantify model [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The change in uncertainty of images with different levels of noise as the text descrip￾tion increases. Noise=0 means no noise is added. NoisyImage=Image+N(0, Noise) tions. The OOD dataset that is outside the model’s training scope, containing questions for which it couldn’t possibly know the answers. Construction of dataset: Model-specific dataset: For a given question, a model answers 10 times. Based on an accuracy… view at source ↗
Figure 4
Figure 4. Changes in ECE after calibration for different [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Gradually add text descriptions on images [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: This question was answered 10 times by LLaVA-7B but was not answered correctly each time. We believe this model does not know the answer. Who did President Biden announce his support for in the 2024 presidential election? A:Donald Trump B:Kamala Harris C:Joe Biden D:Hi…
Figure 7
Figure 7. Figure 7: The models we tested were trained before [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: This figure comes from (Cheng et al., 2024). We have the model answer multiple times and determine whether it knows the answer to the question based on a pre-set threshold. This is a news from 2024: {content} You need to construct multiple-choice questions based on thi…
Figure 10
Figure 10. Figure 10: This figure shows the prompt for constructing multiple-choice questions from crawled news by GPT. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Analysis of Image-and-Text Uncertainty Propagation in Multimodal Large Language Models with Cardiac MR-Based Applications

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A linear uncertainty-propagation model fitted on cardiac MRI plus health-record text is shown to transfer across prediction tasks and data distributions, enabling cheaper uncertainty estimates.

  2. Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Uncertainty-o estimates uncertainty in large multimodal models by perturbing prompts and computing entropy over semantically clustered answers, improving hallucination detection across five modalities.

  3. Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots

    cs.SE 2025-07 conditional novelty 4.0 of 10

    Across three VLA models and four simulated manipulation tasks, motion-instability and goal-distance metrics correlate with expert-rated execution quality, showing that binary success rates hide large quality differences.

Reference graph

Works this paper leans on

48 extracted references · 19 canonical work pages · cited by 3 Pith papers

  1. [1]

    Jinze Bai, Song Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Jung Fu Lin, Chen Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    work page 2023

  2. [2]

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

    arXiv 2024

  3. [3]

    Thomas Buddenkotte, Lorena Escudero Sanchez, Mireia Crispin-Ortuzar, Ramona Woitek, Cathal McCague, James D Brenton, Ozan \"O ktem, Evis Sala, and Leonardo Rundo. 2023. Calibrating ensembles for scalable uncertainty quantification in deep learning-based medical image segmentation. Computers in Biology and Medicine, 163:107096

    work page 2023

  4. [4]

    Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Kai Chen, and Xipeng Qiu. 2024. Can ai assistants know what they don't know? arXiv preprint arXiv:2401.13275

    arXiv 2024

  5. [5]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    arXiv 2018

  6. [6]

    Peng Cui, Wenbo Hu, and Jun Zhu. 2020. Calibrated reliable regression using maximum mean discrepancy. Advances in Neural Information Processing Systems, 33:17164--17175

    work page 2020

  7. [7]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321--1330. PMLR

    2017

  8. [8]

    Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, and Jun Zhu. 2023. Investigating uncertainty calibration of aligned language models under the multiple-choice setting. arXiv preprint arXiv:2310.11732

    arXiv 2023

Show all 48 references
  1. [9]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286

    2020 arXiv

  2. [10]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  3. [11]

    Yu, and Inderjit S

    Cho-Jui Hsieh, Si Si, Felix X. Yu, and Inderjit S. Dhillon. 2023. Automatic engineering of long prompts. abs/2311.10117

    2023 arXiv

  4. [12]

    Hugging Face . 2023. https://huggingface.co/datasets/TheMrguiller/BilbaoQA2 BilbaoQA2 . [BilbaoQA2]

    2023

  5. [13]

    Hugging Face . 2024 a . https://huggingface.co/datasets/arnaudstiegler/mobile_capture_vqa MobileVQA . [MobileVQA]

    2024

  6. [14]

    Hugging Face . 2024 b . https://huggingface.co/datasets/visheratin/realworldqa realworldqa . [realworldqa]

    2024

  7. [15]

    Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. 2023. Calibrating language models via augmented prompt ensembles

    2023

  8. [16]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  9. [17]

    Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30

    2017

  10. [18]

    Zaid Khan and Yun Fu. 2021. Exploiting bert for multimodal target sentiment classification through input space translation. In Proceedings of the 29th ACM international conference on multimedia, pages 3034--3042

    2021

  11. [19]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  12. [20]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. https://doi.org/10.18653/v1/D17-1082 RACE : Large-scale R e A ding comprehension dataset from examinations . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages...

    2017 doi

  13. [21]

    Chenyi Lei, Shixian Luo, Yong Liu, Wanggui He, Jiamang Wang, Guoxin Wang, Haihong Tang, Chunyan Miao, and Houqiang Li. 2021. Understanding chinese video and language via contrastive multimodal pre-training. In Proceedings of the 29th ACM International Conference on Multimedia,...

    2021

  14. [22]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. abs/2307.16125

    2023 arXiv

  15. [23]

    Peiyao Li, Lan Hua, Zhechao Ma, Wenbo Hu, Ye Liu, and Jun Zhu. 2024. Conformalized graph learning for molecular admet property prediction and reliable uncertainty quantification. Journal of Chemical Information and Modeling

    2024

  16. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 a . Visual instruction tuning. abs/2304.08485

    2023 arXiv

  17. [25]

    Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, and Jinhui Tang. 2024. Valor: Vision-audio-language omni-perception pretraining model and dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  18. [26]

    Ying Liu, Peng Cui, Wenbo Hu, and Richang Hong. 2023 b . Deep ensembles meets quantile regression: Uncertainty-aware imputation for time series. arXiv preprint arXiv:2312.01294

    2023

  19. [27]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023 c . Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281

    2023 arXiv

  20. [28]

    Andrey Malinin and Mark Gales. 2018. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31

    2018

  21. [29]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

    2018

  22. [30]

    Allan H Murphy and Robert L Winkler. 1977. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society Series C: Applied Statistics, 26(1):41--47

    1977

  23. [31]

    OpenAI. 2023. Gpt-4 technical report. abs/2303.08774

    2023 arXiv

  24. [32]

    Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. 2024. Probabilistic weather forecasting with machine learning. Nature, pages 1--7

    2024

  25. [33]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. abs/2305.03495:7957--7968

    2023 arXiv

  26. [34]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  27. [35]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525

    2023 arXiv

  28. [36]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  29. [37]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv prep...

    2023 arXiv

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Youze Wang, Wenbo Hu, Yinpeng Dong, Hanwang Zhang, Hang Su, and Richang Hong. 2023 a . Exploring transferability of multimodal adversarial samples for vision-language pre-training models with contrastive learning. arXiv preprint arXiv:2308.12636

    2023 arXiv

  32. [40]

    Youze Wang, Wenbo Hu, and Richang Hong. 2023 b . Iterative adversarial attack on image-guided story ending generation. IEEE Transactions on Multimedia

    2023

  33. [41]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063

    2023 arXiv

  34. [42]

    Mingliang Yang, Kun Jiang, Junze Wen, Liang Peng, Yanding Yang, Hong Wang, Mengmeng Yang, Xinyu Jiao, and Diange Yang. 2023. Real-time evaluation of perception uncertainty and validity verification of autonomous driving. Sensors, 23(5):2867

    2023

  35. [43]

    Zhang. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?

    2024

  36. [44]

    Samuel Preston, and Hoifung Poon

    Theodore Zhao, Mu Wei, J. Samuel Preston, and Hoifung Poon. 2023. Automatic calibration and error correction for large language models via pareto optimal self-supervision. abs/2306.16564

    2023 arXiv

  37. [45]

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. 2024. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems, 36

    2024

  38. [46]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers

    2022

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.