Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that most current vision-language models systematically overstate their confidence when asked to verbalize it, that models which reason directly in the visual modality stay markedly better calibrated, and that a…

desk verdict A broad, useful empirical survey of verbalized calibration in VLMs, but the headline causal claim about visual-reasoning models rests on two closed-source OpenAI models and should be reworded as an observation rather than a mechanism. read the letter →

arxiv 2505.20236 v1 pith:F7DABBN7 submitted 2025-05-26 cs.CV

Weihao Xuan , Qingcheng Zeng , Heli Qi , Junjue Wang , Naoto Yokoya This is my paper
classification cs.CV
keywords verbalizedcalibrationvision-languagemodelsexpectederroroverconfidencevisualreasoningmodalityalignmentconfidence-awarepromptinguncertaintyquantification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether vision-language models (VLMs) can say how sure they are and be believed. Across ten models, four task domains, and three evaluation settings, it finds that most VLMs are systematically overconfident when they verbalize a confidence score: their stated confidence regularly exceeds their actual accuracy. The one consistent exception is the category of visual reasoning models, which think with the image itself and stay well calibrated across benchmarks and modalities. The paper also introduces Visual Confidence-Aware Prompting (VCAP), a two-round prompt that first elicits a description of the visual input with its own confidence score and then asks the model to answer in light of that self-assessment, and reports that it lowers calibration error on the Qwen2.5-VL family. If these results hold, verbalized confidence from current VLMs cannot be taken at face value, and both visual-reasoning training and prompt design become levers for making multimodal systems honest about uncertainty.

What carries the argument

The machinery has three parts. The measure is the Expected Calibration Error $\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \, |\mathrm{acc}(B_m) - \mathrm{avgConf}(B_m)|$ over ten confidence bins, which scores how far stated confidence drifts from observed accuracy. The measurement design is three paired settings, a general text-question-plus-image setting, questions embedded entirely in the image, and IsoBench's isomorphic text-versus-image representations of the same problem, which together isolate whether miscalibration tracks the task, the instruction channel, or the input modality itself. The intervention is Visual Confidence-Aware Prompting (VCAP), a two-stage dialogue in which the model first describes the visual input and rates its own description, then answers the original question while explicitly considering that description and its confidence score. The description-with-confidence step is the load-bearing novelty: it forces a separate, low-stakes calibration judgment on the perception step before the final answer's confidence is produced.

What would settle it

Take an open-weight visual chain-of-thought model and a matched text-reasoning model of comparable size and training budget, and compare their ECE on MMMU-Pro and IsoBench; if the visual-reasoning model does not clearly win, the paper's causal attribution collapses. A cheap control on the paper's own data: restrict the comparison to questions that o1 and o3 both answer correctly, and check whether o3's calibration advantage survives, since if it disappears once accuracy is matched, the advantage may be an artifact of capability rather than of visual reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that verbalized calibration in current VLMs is the exception rather than the rule, and that the exception is structured by how the model reasons. Instruction-tuned models and text-centric reasoning models show Expected Calibration Error above 0.25 on most benchmarks, with calibration curves that hug the overconfident side of the diagonal; the pattern persists whether instructions arrive as text or as pixels, and it widens when semantically identical content is presented as an image rather than text. The vision-centric reasoning models o3 and o4-mini break this pattern, achieving markedly lower ECE across tasks, including near-zero calibration error on IsoBench, and shrinking the image-versus-text calibration gap that other models show. The authors take this as evidence that modality-specific visual reasoning, not reasoning per se, is what produces trustworthy confidence estimates. As a practical corrective, they propose VCAP, which decouples perception from task execution and improves calibration for instruction-tuned models even without retraining.

Load-bearing premise

The conclusion that visual reasoning, rather than raw capability, scale, or data, is what drives good calibration rests on just two closed-source models, o3 and o4-mini, standing in for the entire vision-centric reasoning category.

Editorial extensions

If this is right

  • Verbalized confidence from current instruction-tuned and text-reasoning VLMs should not be relied upon in high-stakes settings, since the paper documents ECE mostly above 0.25 and systematic overconfidence across confidence bins.
  • Visual-reasoning training, thinking with images, becomes a concrete design target for trustworthy multimodal AI, as the only model category that stays calibrated across tasks and across input modalities.
  • VCAP offers a training-free way to buy calibration: on IsoBench, two-round visual-description-then-answer prompting lowers overall ECE from 0.431 to 0.365 for Qwen2.5-VL-72B and from 0.467 to 0.424 for the 7B model, beating Top-K prompting and self-reflection.
  • Because calibration degrades when identical content is presented as an image rather than text, ECE can double as a diagnostic of modality alignment, not just of confidence quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's category-level conclusion would be tested cleanly by an open-weight visual-reasoning model; the authors acknowledge that only two closed-source models support the strongest claim.
  • VCAP is a generic two-round pattern, so it likely transfers beyond the Qwen2.5-VL family; a quick check would be running the same prompts on InternVL3 or Kimi-VL and watching whether ECE drops similarly.
  • The first round's description confidence may act as a cheap perceptual-uncertainty signal; feeding a deliberately wrong description into round two would separate the benefit of extra computation from the benefit of an honest self-assessment anchor.
  • Given that visual math errors arrive with high confidence, calibration error could be used as a failure detector for visual grounding in downstream safety-critical applications.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript presents a large-scale empirical study of verbalized confidence calibration in vision-language models (VLMs). Fourteen models, grouped into three categories (instruction-tuned, text-centric reasoning, and vision-centric reasoning), are evaluated on five benchmarks (MMMU-Pro, VideoMMMU, Visual SimpleQA, MathVista, MathVision) plus the IsoBench isomorphic-representation benchmark, under three input configurations: standard text-instruction tasks, image-embedded instructions, and semantically aligned text-versus-image versions of the same problems. The authors report that most current VLMs are systematically overconfident in their verbalized confidence, with ECE frequently above 0.25, and observe that the two vision-centric reasoning models tested (OpenAI o3 and o4-mini) achieve markedly lower ECE across most settings. They then propose Visual Confidence-Aware Prompting (VCAP), a two-stage prompting strategy that elicits an image description with a confidence score before producing the final answer, and report ECE improvements over CoT, Top-K, and self-reflection baselines on IsoBench with Qwen2.5-VL 7B and 72B.

Significance. The study has a substantial descriptive contribution that would be of use to the community: the benchmark and configuration coverage is broader than that of prior verbalized-uncertainty studies in VLMs (Groot and Valdenegro Toro 2024; Borszukovszki et al. 2025; Zhao et al. 2025), the semantically aligned IsoBench setup is a thoughtful design for isolating modality effects, and the appendix provides the full prompts and per-model results for all 14 models, which is a real strength for replicability. The central descriptive finding that verbalized confidence is broadly miscalibrated across model families and input modalities is likely robust. However, the two headline claims—that vision-centric reasoning training is the cause of the calibration advantage, and that VCAP reliably improves calibration—are not established at the standard the paper's framing requires, for the reasons given in the major comments. If re-scoped to descriptive claims, the paper would be a solid empirical contribution; as written, the causal and method claims outrun the evidence.

major comments (3)
  1. [§3.2, §4.1, Abstract] The abstract's claim that 'modality-specific reasoning is critical for reliable uncertainty estimation' is not supported by the evidence presented. The vision-centric category contains only two models, o3 and o4-mini, both closed-source, both from a single vendor, and both the most accurate models on nearly every column of Table 1; their lower ECE could therefore reflect general capability, scale, or RL-based training rather than visual reasoning. The confound is visible inside the paper's own data: in Table 2 (IsoBench), o3 and o4-mini are also the best-calibrated models on the text-only columns (All ECE 0.081 and 0.058 vs. o1's 0.109), and in Table 5, on text-only Visual SimpleQA, o1 actually has lower ECE (0.099) than both o4-mini (0.112) and o3 (0.117), so the advantage is not specific to image inputs. The Limitations paragraph acknowledges the n=2 issue, but Section 4.1 and the abstract state the categorical conclusion without that caveat. Please reframe the finding as a descriptive observation about the two specific models tested, or add a matched comparison (for example, an open-weight model family with and without visual chain-of-thought training) that can separate the visual-reasoning mechanism from general capability.
  2. [§5, Table 3] The claim that VCAP yields 'significant gains in calibration quality' is not backed by any uncertainty quantification. All ECE values in Table 3 are single-run point estimates without confidence intervals, bootstrap resampling, or significance tests, and several head-to-head differences are small enough to be sampling noise (for example, Qwen2.5-VL 7B All-domain ECE: VCAP 0.424 vs. Self-Reflection 0.436). The improvement is also not consistent across the four IsoBench domains: on Games, Self-Reflection is better than VCAP for both model sizes (7B: 0.651 vs. 0.668; 72B: 0.612 vs. 0.664), and on 7B Algorithms, Self-Reflection (0.407) beats VCAP (0.490). Please report variance across multiple runs or seeds, provide significance testing or bootstrap intervals, and discuss the per-domain inconsistency rather than only the All-domain aggregate.
  3. [§5, Abstract] The abstract and Section 6 claim VCAP improves confidence alignment in 'multimodal settings' generally, but the evidence in Table 3 covers only one benchmark, IsoBench, and one model family, Qwen2.5-VL. Since the method was explicitly motivated by the modality gap observed on IsoBench (Section 5), evaluation on that same benchmark does not test generalization, and the aggregate gain is largely driven by the Mathematics domain (7B ECE 0.377 to 0.320; 72B 0.353 to 0.261) with smaller or inconsistent improvements elsewhere. Either extend the VCAP evaluation to at least one additional benchmark (for instance the embedded-instruction MMMU-Pro setting) or temper the generality of the claim to calibration on IsoBench-style isomorphic tasks for Qwen2.5-VL models.
minor comments (6)
  1. [Appendix B.1] The appendix text references 'MMLU-Pro' while Table 4 reports MMMU-Pro results; correct the dataset name.
  2. [Abstract] The abstract renders 'VISUALCONFIDENCE-AWAREPROMPTING' as a single unbroken token without spaces; fix this formatting artifact.
  3. [Appendix D] The heading contains the typo 'Diffferent prompting strategies'; correct it.
  4. [Appendix A] Several prompts contain typos that should be cleaned up, including 'tge question' in the Top-K maxflow prompt, 'an convex function' in the convexity prompts, and 'areisomorphic' in the isomorphism prompts.
  5. [§3.2] The category definitions are not operationalized: o1 also accepts image inputs, and the text-centric versus vision-centric distinction rests on vendor training claims that are not independently verifiable; a sentence acknowledging this dependence and the resulting fuzzy boundary would strengthen the setup.
  6. [§3.1, §4] Reproducibility would be improved by reporting API version numbers, decoding parameters (temperature, number of samples), and the number of repeated runs per model, especially for closed-source systems whose behavior can change between versions.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the calibration evaluation is self-contained against external benchmarks, and the only self-citation is contextual rather than load-bearing.

full rationale

The paper reports an empirical benchmark evaluation rather than a derivation, and its central claims rest on measured ECE values computed from model outputs and ground-truth labels via Equation (1). No parameter is fitted to the evaluation data and no predicted quantity is constructed from the quantity it purports to predict. The 'visual reasoning models are better calibrated' claim is an observed pattern in Tables 1-3, not a consequence of a definition or a self-citation; the model categories in Section 3.2 are descriptive groupings, and the paper's own Limitations paragraph concedes that the category contains only two closed-source OpenAI models, which weakens causal attribution but does not make the evaluation circular. The only self-citation is Zeng et al. (2025) in Sections 2.1 and 6, which shares two authors with this paper; however, it is used as background context about text-reasoning models and is not the load-bearing justification for the paper's multimodal findings. VCAP is motivated by the observed IsoBench modality gap and evaluated on IsoBench, but the prompting strategy introduces no fitted parameters and its ECE comparison against CoT, Top-K, and self-reflection baselines is a direct empirical test, not a tautology. Overall, no step in the paper's reasoning reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The method is pure empirical benchmarking, so there are no fitted constants. The central claims rest on benchmark label correctness, the ECE binning convention, semantic equivalence of IsoBench image/text pairs, the unverified vision-centric reasoning categorization, and single-run API stability.

assumptions (6)
  • domain assumption Benchmark ground-truth labels are correct.
    The paper's ECE and AUROC computations assume MMMU-Pro, VideoMMMU, Visual SimpleQA, MathVista, MathVision, and IsoBench labels are accurate. Section 3.3 uses these labels as ground truth.
  • domain assumption ECE with M=10 fixed bins is a suitable calibration metric.
    Section 3.3 fixes M=10; no sensitivity analysis or alternative calibration metrics beyond AUROC are used, so binning choices could affect reported ECE.
  • domain assumption IsoBench image and text versions are semantically equivalent.
    Section 3.1.3 assumes semantically equivalent inputs isolate modality effects; if image renderings add noise or ambiguity, the modality gap is overestimated.
  • domain assumption o3 and o4-mini are genuinely vision-centric reasoning models that think with images.
    Section 3.2 categorizes them on the basis of vendor capability claims; internal reasoning is not verifiable, and this assumption underlies the paper's main interpretation.
  • domain assumption Verbose confidence scores are extracted correctly from model outputs.
    The prompts ask for XML tags, but the paper does not report parsing failure rates or how unparseable outputs were handled.
  • domain assumption API outputs are deterministic or stable enough for single-run evaluation.
    Tables report single ECE and accuracy values with no repeated sampling; stochastic decoding or API updates would change numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models." pith.science (2026). https://pith.science/paper/F7DABBN7

@misc{pith2026250520236,
  author       = {Pith},
  title        = {Pith review of: Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7DABBN7}},
  note         = {Machine review of arXiv:2505.20236}
}
read the original abstract

Uncertainty quantification is essential for assessing the reliability and trustworthiness of modern AI systems. Among existing approaches, verbalized uncertainty, where models express their confidence through natural language, has emerged as a lightweight and interpretable solution in large language models (LLMs). However, its effectiveness in vision-language models (VLMs) remains insufficiently studied. In this work, we conduct a comprehensive evaluation of verbalized confidence in VLMs, spanning three model categories, four task domains, and three evaluation scenarios. Our results show that current VLMs often display notable miscalibration across diverse tasks and settings. Notably, visual reasoning models (i.e., thinking with images) consistently exhibit better calibration, suggesting that modality-specific reasoning is critical for reliable uncertainty estimation. To further address calibration challenges, we introduce Visual Confidence-Aware Prompting, a two-stage prompting strategy that improves confidence alignment in multimodal settings. Overall, our study highlights the inherent miscalibration in VLMs across modalities. More broadly, our findings underscore the fundamental importance of modality alignment and model faithfulness in advancing reliable multimodal systems.

Figures

Figures reproduced from arXiv: 2505.20236 by the authors.

Figure 1
Figure 1. The illustration of our three types of evaluations: general, embedded instruction, and semantically aligned [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Calibration curve on testmini set of MathVi [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Model performance comparison across and calibration. The upper right indicates better overall performance. with visual reasoning models achieving the most reliable confidence estimates. These results further support our earlier observation that both modality￾specific and reasoning-specific training play a crit￾ical role in improving verbalized uncertainty in VLMs. When comparing performance under visual in￾struction… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The illustration of our Visual Confidence [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 4 canonical work pages

  1. [1]

    Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, and 23 others

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, and 23 others. 2024. https://arxiv.org/ab...

    arXiv 2024

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://arxiv.org/abs/2502.13923 Qwen2.5-vl technical report . Preprint, arXiv:2502.13923

    arXiv 2025

  3. [3]

    Mirko Borszukovszki, Ivo Pascal De Jong, and Matias Valdenegro-Toro. 2025. https://aclanthology.org/2025.trustnlp-main.16/ Know what you do not know: Verbalized uncertainty estimation robustness on corrupted images in vision-language models . In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 247--265, Albuquerque, New Mexico. As...

    work page 2025

  4. [4]

    Tan, and Haizhou Li

    Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. 2024. https://arxiv.org/abs/2410.17196 Voicebench: Benchmarking llm-based voice assistants . Preprint, arXiv:2410.17196

    arXiv 2024

  5. [5]

    Zijun Chen, Wenbo Hu, Guande He, Zhijie Deng, ZHeng ZHang, and Richang Hong. 2025. https://aclanthology.org/2025.coling-main.208/ Unveiling uncertainty: A deep dive into calibration and performance of multimodal large language models . In Proceedings of the 31st International Conference on Computational Linguistics, pages 3095--3109, Abu Dhabi, UAE. Assoc...

    work page 2025

  6. [6]

    Chris, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, and Yahui Zhou. 2025. https://arxiv.org/abs/2504.16656 Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning . Preprint, arXiv:2504.16656

    arXiv 2025

  7. [7]

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. 2024. https://doi.org/10.18653/v1/2024.findings-acl.558 Fact-checking the output of large language models via token-level uncertainty quantification . I...

    doi:10.18653/v1/2024.findings-acl.558 2024

  8. [8]

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.41 LM -polygraph: Uncertainty estimation for language models . In Proceedings of the 2023 Confer...

    doi:10.18653/v1/2023.emnlp-demo.41 2023

Show all 44 references
  1. [9]

    Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. 2024. https://arxiv.org/abs/2404.01266 Isobench: Benchmarking multimodal foundation models on isomorphic representations . Preprint, arXiv:2404.01266

    2024 arXiv

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [11]

    Tobias Groot and Matias Valdenegro Toro. 2024. https://doi.org/10.18653/v1/2024.trustnlp-1.13 Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models . In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (...

    2024 doi

  4. [12]

    Sophia Hager, David Mueller, Kevin Duh, and Nicholas Andrews. 2025. https://arxiv.org/abs/2503.14749 Uncertainty distillation: Teaching language models to express semantic confidence . Preprint, arXiv:2503.14749

    2025

  5. [13]

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. 2025. https://arxiv.org/abs/2501.13826 Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos . Preprint, arXiv:2501.13826

    2025 arXiv

  6. [14]

    Kimi Team , Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, and 73 others. 2025. https://arxiv.org/abs/25...

    2025 arXiv

  7. [15]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://arxiv.org/abs/2302.09664 Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . Preprint, arXiv:2302.09664

    2023 arXiv

  8. [16]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  9. [17]

    Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, and Anders Søgaard. 2024 b . https://doi.org/10.1162/tacl_a_00698 Do vision and language models share concepts? a vector space alignment study . Transactions of the Association for Computational Linguistics, 12:1232--1249

    2024 doi

  10. [18]

    Xiujun Li, Yujie Lu, Zhe Gan, Jianfeng Gao, William Yang Wang, and Yejin Choi. 2024 c . https://arxiv.org/abs/2311.17647 Text as images: Can multimodal large language models follow printed instructions in pixels? Preprint, arXiv:2311.17647

    2024 arXiv

  11. [19]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://arxiv.org/abs/2205.14334 Teaching models to express their uncertainty in words . Preprint, arXiv:2205.14334

    2022 arXiv

  12. [20]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  13. [21]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Repre...

    2024

  14. [22]

    Bagdanov

    Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D. Bagdanov. 2025. https://arxiv.org/abs/2502.04263 Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion . Preprint, arXiv:2502.04263

    2025 arXiv

  15. [23]

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. https://arxiv.org/abs/2405.20003 Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities . Preprint, arXiv:2405.20003

    2024 arXiv

  16. [24]

    OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, and 7 othe...

    2025 arXiv

  17. [25]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024 ...

    2024 arXiv

  18. [26]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, and 244 oth...

    2024 arXiv

  19. [27]

    OpenAI. 2025. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  20. [28]

    Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d'Autume, and 3 others. 2...

    2024 arXiv

  21. [29]

    Yi Peng, Chris, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, Rongxian Zhuang, Xuchen Song, Yang Liu, and Yahui Zhou. 2025. https://arxiv.org/abs/2504.05599 Skywork r1v: Pioneering multimodal reasoning with chain-of...

    2025 arXiv

  22. [30]

    Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Lu Cheng, and Mengnan Du. 2025. https://arxiv.org/abs/2501.01346 Large vision-language model alignment and misalignment: A survey through the lens of explainability . Preprint, arXiv:2501.01346

    2025

  23. [31]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod...

    2023 doi

  24. [32]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2024. https://doi.o...

    2024 doi

  25. [33]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. https://openreview.net/forum?id=QWTCcxMpPA Measuring multimodal mathematical reasoning with math-vision dataset . In The Thirty-eight Conference on Neural Information Pro...

    2024

  26. [34]

    Yanling Wang, Yihan Zhao, Xiaodong Chen, Shasha Guo, Lixin Liu, Haoyang Li, Yong Xiao, Jing Zhang, Qi Li, and Ke Xu. 2025. https://arxiv.org/abs/2503.06492 Visualsimpleqa: A benchmark for decoupled evaluation of large vision-language models in fact-seeking question answering ....

    2025 arXiv

  27. [35]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. https://arxiv.org/abs/2306.13063 Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms . Preprint, arXiv:2306.13063

    2024 arXiv

  28. [36]

    Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.343 S ay S elf: Teaching LLM s to express confidence with self-reflective rationales . In Proceedings of the 2024 Conference on Empirical...

    2024 doi

  29. [37]

    Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. 2024. https://arxiv.org/abs/2412.14737 On verbalized confidence scores for llms . Preprint, arXiv:2412.14737

    2024 arXiv

  30. [38]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2024. https://arxiv.org/abs/2409.02813 Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark . Pre...

    2024 arXiv

  31. [39]

    Qingcheng Zeng, Weihao Xuan, Leyang Cui, and Rob Voigt. 2025. https://arxiv.org/abs/2504.06564 Do reasoning models show better verbalized calibration? Preprint, arXiv:2504.06564

    2025

  32. [40]

    Yunpu Zhao, Rui Zhang, Junbin Xiao, Ruibo Hou, Jiaming Guo, Zihao Zhang, Yifan Hao, and Yunji Chen. 2025. https://arxiv.org/abs/2504.14848 Object-level verbalized confidence calibration in vision-language models via semantic perturbation . Preprint, arXiv:2504.14848

    2025 arXiv

  33. [41]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models . Preprint, arXiv:2311.07911

    2023 arXiv

  34. [42]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, and 32 others. 2025. https://arxiv.org/abs/250...

    2025 arXiv

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.