Pith. sign in

REVIEW 4 major objections 6 minor 71 references

Are vision language models robust to uncertain inputs?

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a 'return unknown' instruction to the prompt lets VLMs reject ambiguous or anomalous images and raises the accuracy of the predictions they keep.

desk verdict Useful empirical paper on VLM abstention; the 'near-perfect robustness' claim is overstated because accuracy is only on non-rejected samples. read the letter →

arxiv 2505.11804 v1 pith:ZYTXNJJ7 submitted 2025-05-17 cs.CV cs.AI

Xi Wang , Eric Nalisnick This is my paper · ORCID
classification cs.CVcs.AI
keywords visionlanguagemodelsuncertaintyquantificationclassificationwithrejectionanomalydetectioncaptiondiversityImageNet-Cgalaxymorphologyprompt-basedabstention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models are not automatically robust to ambiguous or anomalous inputs; they tend to follow the instruction to classify and produce confident nonsense. The paper shows that this failure can be largely overcome for natural images by adding one sentence to the prompt telling the model it may return 'unknown' when unsure. With that rejection option, accuracy on the images the model keeps rises to near-perfect levels across several corruption types and anomaly-detection settings. The same fix fails on specialist domains such as galaxy morphology, where the model lacks the knowledge needed to recognize its own uncertainty. The paper also proposes caption diversity—how much randomly sampled descriptions of an image vary—as a label-free signal predicting when a model will successfully abstain.

What carries the argument

The central mechanism is the rejection prompt: a short instruction appended to a classification prompt telling the model that it may answer 'unknown' when the image is ambiguous or outside the specified classes. This single-prompt intervention lets a VLM condition its behavior on the instruction and abstain instead of hallucinating a confident label. The second mechanism is the caption diversity score: for each image, the model generates a set of captions under random decoding; the captions are embedded with a sentence transformer, and diversity is computed as one minus the average pairwise cosine similarity. The paper uses this score as a label-free probe of the model's uncertainty, and shows that it tracks both corruption level and rejection decisions on natural images but not on the galaxy-morphology task.

What would settle it

Generate class-blended images—for instance, superpositions of a cat and a dog from CIFAR-10 at varying mixing ratios, with the label space containing only one of those classes—and measure the model's rejection rate and caption diversity as the blend ratio moves from 0 to 1. If rejection rate and caption diversity do not rise monotonically with the blend ratio (i.e., with genuine ambiguity), the claim that prompting activates an internal uncertainty signal is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that modern VLMs possess an implicit, prompt-activatable ability to quantify their own uncertainty, and that this ability is enough to make classification-with-rejection work without any retraining, fine-tuning, or architectural change. Evaluated on corrupted ImageNet images mapped to CIFAR-10 classes, models prompted to 'return unknown' when the input is ambiguous abstain at increasing rates as corruption intensifies, and the accuracy of the remaining predictions stays close to perfect. The same pattern appears in anomaly detection: instructed to reject inputs outside the class list, the models flag most out-of-scope images while keeping in-scope classification accurate. The mechanism is not universal: on Galaxy Zoo images, where annotator disagreement marks genuine ambiguity, the models do not reject more as disagreement rises and their accuracy after rejection barely improves, which the authors attribute to missing domain knowledge. Finally, the paper proposes caption diversity—one minus the average pairwise cosine similarity among multiple sampled captions—as an observable proxy for the model's internal uncertainty, showing that rejected images have higher diversity and that diversity rises with corruption level, while staying flat on the galaxy task where rejection fails.

Load-bearing premise

The central results assume that the ImageNet-C corruption levels provide a valid, monotone ordering of how genuinely ambiguous an image is to the model, so that rising rejection rates and rising caption diversity measure uncertainty about the input rather than mere task difficulty.

Editorial extensions

If this is right

  • Practitioners using black-box VLMs on natural images can obtain a meaningful reliability guarantee for free: add an abstention option to the prompt, and treat 'unknown' as a rejection before downstream use.
  • The caption-diversity heuristic gives a way to audit a VLM's uncertainty without labels or logits, so it can be applied to closed models behind APIs.
  • The method's failure on galaxy morphology shows that a prerequisite for successful rejection is domain competence; scaling alone does not create uncertainty awareness in specialized tasks.
  • The results suggest that instruction-following, usually seen as a source of sycophantic hallucination, can be redirected to produce calibrated abstention behavior.
  • Accuracy after rejection approaching near-perfect values across corruption types implies that the models' errors on corrupted natural images are largely concentrated on samples the model can itself flag as ambiguous.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the caption-diversity signal is a portable uncertainty score, it could serve as a zero-shot selection criterion for choosing which inputs to route to a human or to a more expensive model, without needing a labeled validation set.
  • The flat diversity on Galaxy Zoo suggests a testable boundary condition: diversity rises only when the model's visual knowledge is rich enough to support alternate interpretations; on images outside that knowledge, 'uncertainty' is unrepresented. An intervention that injects domain structure (e.g., decision-tree prompts) might re-activate diversity and rejection.
  • One could extend the paired-prompt design into an ensemble: run the same image through several prompt variants and treat disagreement across variants as an additional uncertainty signal, potentially capturing ambiguity the single-rejection prompt misses.
  • A direct test of the mechanism would be to compare rejection behavior on images matched for difficulty but differing in ambiguity, such as class-blended images.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper empirically investigates whether vision-language models (VLMs) can quantify and express uncertainty about their inputs. The authors evaluate six VLMs (Llama 3.2 11B, Qwen 2 7B, Qwen 2.5 3B/7B/72B, GPT-4o-mini) on two classic uncertainty tasks: anomaly detection (CIFAR-10 vs. non-CIFAR-10, ECG vs. non-ECG) and classification with rejection on ambiguous inputs (Gaussian-noise/blur/pixelate/pixmix corrupted ImageNet subsets mapped to CIFAR-10 classes, and Galaxy Zoo morphology with annotator disagreement as ambiguity). The main findings are: (i) explicit prompting to reject uncertain inputs improves accuracy on the non-rejected subset, sometimes to near-perfect levels; (ii) anomaly detection via prompting works well on natural images but degrades on the domain-specific ECG and galaxy tasks; and (iii) a proposed caption-diversity score—computed as 1 minus the mean pairwise cosine similarity of multiple generated captions—correlates with corruption level and with whether a model rejects an image. The authors conclude that simple prompting can elicit calibrated abstention from modern VLMs, with the caveat that domain expertise is needed for specialized tasks.

Significance. If the main claim holds, the result has practical value: a black-box practitioner can improve classifier reliability without retraining, by prompting the model to abstain when uncertain. The study also provides a useful negative result for domain-specific applications (galaxy morphology, ECG), where general VLMs fail to express uncertainty. The caption-diversity idea is interesting and connects to prior semantic-uncertainty methods for LLMs, but the paper's contribution is the empirical demonstration on VLMs rather than a new predictor. The paper is careful in treating models as black boxes and in comparing multiple model families. However, the headline 'near-perfect robustness' is currently supported only by selective-classification accuracy, and the proposed diversity mechanism is not validated as a predictive tool.

major comments (4)
  1. [4.2, Fig. 2b] The abstract's central claim that prompting abstention yields 'significant reliability gains, achieving near-perfect robustness' is supported only by accuracy computed over non-rejected images. The paper never reports a risk-coverage curve or fixes a target coverage; without this, the high post-rejection accuracy could be a consequence of selective abstention, since a model that rejects most high-corruption inputs would achieve near-perfect accuracy on the few remaining samples. The rejection-rate panels in Fig. 2b (bottom row) and Fig. 9 show that rejection is substantial at high corruption levels, so coverage is far below 100%. The authors should report accuracy as a function of the fraction of accepted inputs (a risk-coverage curve) or at a pre-specified coverage level, and state what fraction of inputs receive a label under their prompting protocol.
  2. [3.2 and 4.2, Fig. 5] The paper claims that caption diversity 'enabl[es] practitioners to predict when models will successfully abstain without relying on labeled data' (abstract and Sec. 3.2), but no predictive evaluation is performed. The experiments show only aggregate correlations: mean diversity rises with corruption level, and rejected images have higher diversity than classified images in density plots. A predictive claim requires a quantitative test, such as the area under an ROC curve for using diversity to predict rejection or classification on held-out images, or a demonstration that a diversity threshold recovers the accuracy gains of prompted rejection. Without such evaluation, the diversity mechanism is descriptive rather than validated.
  3. [4.2, Figs. 2, 5, 9] Quantitative comparisons are reported as point estimates without error bars or statistical inference. For example, the diversity differences across corruption levels for Qwen2.5 3B in Fig. 5 (top) range over roughly 0.02, and the rejection-rate curves in Fig. 2b have no uncertainty estimates. Because the central trends (accuracy drop, diversity rise, improvement from rejection) are the basis for the paper's conclusions, the authors should provide bootstrap confidence intervals or repeated-seed runs for the key figures and tables, especially where monotonicity across corruption levels is claimed.
  4. [4.2, Figs. 2, 5, 9, 6] The ImageNet-C experiments treat corruption severity as a monotone ground-truth ordering of perceptual ambiguity for the VLM. The Galaxy Zoo experiment is more convincing because it uses human annotator disagreement (leaf_prob) as an independent ambiguity measure. For ImageNet-C, the trends across corruption levels could reflect overall image degradation rather than the model's uncertainty awareness. The image-level relationship between diversity and the model's own rejection decision is shown only as density plots in Fig. 5 (bottom); reporting a quantitative association (e.g., AUROC of diversity for predicting rejection, or a calibration-style plot) would substantiate the uncertainty-link claim independent of the corruption-level ordering.
minor comments (6)
  1. [Appendix E, Fig. 8] The abstract's statement that 'simply prompting models to abstain' improves reliability is too strong in view of Appendix E: for Llama 3.2 and Qwen2, the direct 'Simple' prompt yields almost no rejection benefit, and only the 'Caption & answer' prompt elicits rejection. The main text should qualify the prompt conditions under which the result holds.
  2. [Sec. 2.1] There is a typo: 'property knwon as calibration' should read 'property known as calibration'.
  3. [Appendix A.1 and A.2] The prompt texts contain formatting artifacts such as 'fo ll ow in g', 'f ol lo win g', and 'r ea so ni ng' that should be cleaned for readability and reproducibility.
  4. [Fig. 2a] The color encoding (red vs. black text) used to distinguish the standard and rejection prompts is not accessible in black-and-white print; consider labels, boxes, or different line styles.
  5. [Sec. 6] The limitation section acknowledges the Galaxy Zoo failure but does not mention the coverage/selective-classification issue or the lack of predictive validation of caption diversity; these should be acknowledged as limitations.
  6. [4.2] The caption diversity score is computed from 20 captions per image; reporting the variance of the score over bootstrap resamples of the 20 captions would help assess the stability of the proposed metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the empirical claims are self-contained measurements rather than derived predictions.

full rationale

The paper makes no parametric fits and sets up no equations in which an output quantity is defined in terms of the quantity it is said to predict. The central result—that prompting VLMs to reject ambiguous inputs improves accuracy on non-rejected ImageNet-C samples—is a direct behavioral measurement on held-out images, reported separately from the caption-diversity analysis. Caption diversity is defined independently as one minus the averaged pairwise cosine similarity of embedded sampled captions (Sec. 3.2), and the rejection decision is a separate model output; the observed diversity–rejection correlation is an empirical finding, not a construction. The few self-citations (e.g., Nalisnick et al. on typicality and OOD detection) appear only in background/reference contexts and are not load-bearing. The absence of a fixed-coverage risk–coverage curve is a limitation of the evaluation metric, not a circularity: accuracy restricted to non-rejected samples can be high for reasons unrelated to uncertainty calibration, but this is an external-validity concern rather than a reduction of the claim to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The empirical claims rest on domain assumptions about what ambiguity means, not on fitted constants. The paper introduces no new entities; caption diversity is a computed statistic rather than a postulated mechanism.

free parameters (3)
  • Galaxy Zoo disagreement bin thresholds (leaf_prob) = 0.75, 0.5 for disagree levels 1, 2, 3
    Hand-chosen bins for annotator agreement; they define the ambiguity axis in Fig. 6 and the galaxy rejection experiment.
  • Caption sampling hyperparameters = temperature 0.6, top-P 0.95, top-K 50
    Ad hoc choices for random caption generation; caption diversity scores and the resulting correlations could shift with these values, and no sensitivity analysis is reported.
  • Number of captions per image = 20
    Chosen without analysis; caption diversity estimates may be noisy with fewer samples and saturate with more.
assumptions (4)
  • domain assumption ImageNet-C corruption severity monotonically corresponds to visual ambiguity
    Figures 2, 5 and 9 interpret accuracy, rejection and diversity trends as functions of corruption level; no independent validation that perceived ambiguity increases monotonically.
  • domain assumption String parsing of VLM outputs into labels and 'unknown' faithfully captures model decisions
    All metrics are computed from black-box output strings without logits; the paper does not discuss ambiguity in output formatting or parse failures.
  • domain assumption Galaxy Zoo leaf_prob annotator disagreement is a valid ground-truth measure of input ambiguity
    Appendix D defines disagree levels from leaf_prob and Figure 6 uses them as the ambiguity axis; this assumes crowd agreement directly maps to visual ambiguity.
  • domain assumption all-mpnet-base-v2 embeddings provide a semantically calibrated space for caption diversity
    The diversity score is one minus mean pairwise cosine similarity in this fixed embedding space; no domain-specific validation of similarity semantics is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are vision language models robust to uncertain inputs?." pith.science (2026). https://pith.science/paper/ZYTXNJJ7

@misc{pith2026250511804,
  author       = {Pith},
  title        = {Pith review of: Are vision language models robust to uncertain inputs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYTXNJJ7}},
  note         = {Machine review of arXiv:2505.11804}
}
read the original abstract

Robustness against uncertain and ambiguous inputs is a critical challenge for deep learning models. While recent advancements in large scale vision language models (VLMs, e.g. GPT4o) might suggest that increasing model and training dataset size would mitigate this issue, our empirical evaluation shows a more complicated picture. Testing models using two classic uncertainty quantification tasks, anomaly detection and classification under inherently ambiguous conditions, we find that newer and larger VLMs indeed exhibit improved robustness compared to earlier models, but still suffer from a tendency to strictly follow instructions, often causing them to hallucinate confident responses even when faced with unclear or anomalous inputs. Remarkably, for natural images such as ImageNet, this limitation can be overcome without pipeline modifications: simply prompting models to abstain from uncertain predictions enables significant reliability gains, achieving near-perfect robustness in several settings. However, for domain-specific tasks such as galaxy morphology classification, a lack of specialized knowledge prevents reliable uncertainty estimation. Finally, we propose a novel mechanism based on caption diversity to reveal a model's internal uncertainty, enabling practitioners to predict when models will successfully abstain without relying on labeled data.

Figures

Figures reproduced from arXiv: 2505.11804 by the authors.

Figure 1
Figure 1. Classic uncertainty quantification tasks revisited in the VLMs era. Using CIFAR-10 as an example, we illustrate how corrupted inputs and inputs from outside CIFAR-10 concepts expose different challenges and failure modes in small supervised models vs. large vision language models (VLMs, e.g. GPT4o) prompted to do classification, despite sharing the same evaluation data. multimodal datasets and undergo self-supervise… view at source ↗
Figure 2
Figure 2. VLMs show degraded performance under corrupted inputs, allowing rejection helps maintain reliability. Top row demonstrates VLMs’ outputs for a selected sample with or without a rejection prompt appended. Bottom row shows the classification accuracy under standard prompt without (solid line) vs. with (dashed line) rejection instruction prompt appended. • Distribution / covariate shift occurs when a model encounters i… view at source ↗
Figure 3
Figure 3. Enabling the rejection option allows VLM to pick out anomaly inputs, preventing hallucination. When prompted to classify a random line image into normal v.s. abnormal ECG signal, certain VLM (Qwen 2.5 3B) would generate hallucinated results (left), potentially caused by its tendency towards strictly following instructions (Sharma et al., 2023). However, when the prompt explicitly permits rejecting non-ECG input (add… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: VLMs generate diverse captions for ambiguous images. We prompt Qwen2.5 7B to “generate a description” given an input image under different random seeds. Clean image from ImageNet receives consistent captions while its corrupted version having a diverse set of captions …
Figure 5
Figure 5. Figure 5: Caption diversity reflects model uncertainty under ambiguous inputs. We empirically verified the hypothesis from Sec. 3.2. Top: As corruption increases, caption diversity rises across all models, indicating greater uncertainty. Bottom: Rejected samples exhibit higher c…
Figure 6
Figure 6. Figure 6: Classification with rejection fails on Galaxy Zoo. As annotator disagreement level increases (higher input ambiguity), VLMs’ accuracy degrades, but do not effectively utilize rejection to improve performance (top left). Caption diversity and confidence remain flat (bot…
Figure 7
Figure 7. Figure 7: Statistics of the 5,000 sample galaxy zoo datasets used in the experiments. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Ablation study on the prompting strategy. Similar to the setting studied in Fig. 2b, here we consider different prompting strategies (line color, detailed in Appendix. E), and models (columns). The second row illustrates the accuracy improvement, we can see that for ce…
Figure 9
Figure 9. Figure 9: Similar to the setting in Fig. 2b, but under different corruption types. The observation still holds: Enabling rejection allows VLM to abstain ambiguous samples, providing nearly perfect accuracy for classified samples. Additionally, caption diversity, as we predicted …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 20 canonical work pages

  1. [1]

    Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35: 0 33646--33660, 2022

    Taiga Abe, Estefany Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35: 0 33646--33660, 2022

    2022

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods

    Laurence Aitchison. Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods. Advances in Neural Information Processing Systems, 33: 0 18173--18182, 2020 a

    work page 2020

  4. [4]

    A statistical theory of cold posteriors in deep neural networks

    Laurence Aitchison. A statistical theory of cold posteriors in deep neural networks. arXiv preprint arXiv:2008.05912, 2020 b

    arXiv 2008

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Weight uncertainty in neural network

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp.\ 1613--1622. PMLR, 2015

    2015

  7. [7]

    End to end learning for self-driving cars

    Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016

    arXiv 2016

  8. [8]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv \'e J \'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9650--9660, 2021

    2021

Show all 71 references
  1. [9]

    Exploring aleatoric uncertainty in object detection via vision foundation models

    Peng Cui, Guande He, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. Exploring aleatoric uncertainty in object detection via vision foundation models. arXiv preprint arXiv:2411.17767, 2024

    2024 arXiv

  2. [10]

    Repulsive deep ensembles are bayesian

    Francesco D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=LAKplpLMbP8

    2021

  3. [11]

    Laplace redux-effortless bayesian deep learning

    Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34, 2021 a

    2021

  4. [12]

    Bayesian deep learning via subnetwork inference

    Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antor \'a n, and Jos \'e Miguel Hern \'a ndez-Lobato. Bayesian deep learning via subnetwork inference. In International Conference on Machine Learning, pp.\ 2510--2521. PMLR, 2021 b

    2021

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Dermatologist-level classification of skin cancer with deep neural networks

    Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, 542 0 (7639): 0 115--118, 2017

    2017

  8. [16]

    Are foundation models for computer vision good conformal predictors? arXiv preprint arXiv:2412.06082, 2024

    Leo Fillioux, Julio Silva-Rodr \' guez, Ismail Ben Ayed, Paul-Henry Courn \`e de, Maria Vakalopoulou, Stergios Christodoulidis, and Jose Dolz. Are foundation models for computer vision good conformal predictors? arXiv preprint arXiv:2412.06082, 2024

    2024

  9. [17]

    Exploring the limits of out-of-distribution detection

    Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in neural information processing systems, 34: 0 7068--7081, 2021

    2021

  10. [18]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp.\ 1050--1059. PMLR, 2016

    2016

  11. [19]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  12. [20]

    Recent advances in open set recognition: A survey

    Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE transactions on pattern analysis and machine intelligence, 43 0 (10): 0 3614--3631, 2020

    2020

  13. [21]

    Practical variational inference for neural networks

    Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011

    2011

  14. [22]

    Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models

    Tobias Groot and Matias Valdenegro-Toro. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.02917, 2024

    2024 arXiv

  15. [23]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp.\ 1321--1330. PMLR, 2017

    2017

  16. [24]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019

    1903 arXiv

  17. [25]

    Deep anomaly detection with outlier exposure

    Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606, 2018

    2018 arXiv

  18. [26]

    Pixmix: Dreamlike pictures comprehensively improve safety measures

    Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 16783--16792, 2022

    2022

  19. [27]

    Meta-learning in neural networks: A survey

    Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44 0 (9): 0 5149--5169, 2021

    2021

  20. [28]

    Improving predictions of bayesian neural nets via local linearization

    Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization. In International conference on artificial intelligence and statistics, pp.\ 703--711. PMLR, 2021

    2021

  21. [29]

    What are bayesian neural network posteriors really like? In International Conference on Machine Learning, pp.\ 4629--4640

    Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International Conference on Machine Learning, pp.\ 4629--4640. PMLR, 2021

    2021

  22. [30]

    Language models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022

    2022 arXiv

  23. [31]

    What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017

    Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017

    2017

  24. [32]

    Uncertainty-aware evaluation for vision-language models

    Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eugene Ilyushin. Uncertainty-aware evaluation for vision-language models. arXiv preprint arXiv:2402.14418, 2024

    2024 arXiv

  25. [33]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023

    2023 arXiv

  26. [34]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017

    2017

  27. [35]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  28. [36]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26296--26306, 2024

    2024

  29. [37]

    Simple and principled uncertainty estimation with deterministic deep learning via distance awareness

    Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 33: 0 7498--7512, 2020

    2020

  30. [38]

    A practical bayesian framework for backpropagation networks

    David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4 0 (3): 0 448--472, 1992

    1992

  31. [39]

    A simple baseline for bayesian uncertainty in deep learning

    Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  32. [40]

    Bayesian exploration of pre-trained models for low-shot image classification

    Yibo Miao, Yu Lei, Feng Zhou, and Zhijie Deng. Bayesian exploration of pre-trained models for low-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 23849--23859, 2024

    2024

  33. [41]

    Revisiting the calibration of modern neural networks

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34: 0 15682--15694, 2021

    2021

  34. [42]

    Generalized out-of-distribution detection and beyond in vision language model era: A survey

    Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, et al. Generalized out-of-distribution detection and beyond in vision language model era: A survey. arXiv preprint arXiv:2407.21794, 2024 a

    2024 arXiv

  35. [43]

    Unsolvable problem detection: Evaluating trustworthiness of vision language models

    Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. arXiv preprint arXiv:2403.20331, 2024 b

    2024 arXiv

  36. [44]

    Evaluating prediction-time batch normalization for robustness under covariate shift

    Zachary Nado, Shreyas Padhy, D Sculley, Alexander D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020

    2006 arXiv

  37. [45]

    Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136, 2018

    Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136, 2018

    2018 arXiv

  38. [46]

    Dropout as a structured shrinkage prior

    Eric Nalisnick, Jos \'e Miguel Hern \'a ndez-Lobato, and Padhraic Smyth. Dropout as a structured shrinkage prior. In International Conference on Machine Learning, pp.\ 4712--4722. PMLR, 2019 a

    2019

  39. [47]

    Detecting out-of-distribution inputs to deep generative models using typicality

    Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using typicality. arXiv preprint arXiv:1906.02994, 2019 b

    1906 arXiv

  40. [48]

    Bayesian learning for neural networks, volume 118

    Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012

    2012

  41. [49]

    Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems, 37: 0 8901--8929, 2024

    2024

  42. [50]

    Measuring calibration in deep learning

    Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, 2019

    2019

  43. [51]

    Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processi...

    2019

  44. [52]

    Astroclip: a cross-modal foundation model for galaxies

    Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Rudy Morel, et al. Astroclip: a cross-modal foundation model for galaxies. Monthly Notices of the Royal Astronomical Society, 531 ...

    2024

  45. [53]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  46. [54]

    Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp.\ 5389--5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp.\ 5389--5400. PMLR, 2019

    2019

  47. [55]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  48. [56]

    Test-time adaptation with state-space models

    Mona Schirmer, Dan Zhang, and Eric Nalisnick. Test-time adaptation with state-space models. In ICML 2024 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling , 2024

    2024

  49. [57]

    Improving robustness against common corruptions by covariate shift adaptation

    Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33: 0 11539--11551, 2020

    2020

  50. [58]

    Towards understanding sycophancy in language models

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023

    arXiv

  51. [59]

    Ptb-xl, a large publicly available electrocardiography dataset

    Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiography dataset. Scientific data, 7 0 (1): 0 1--15, 2020

    2020

  52. [60]

    Galaxymnist: Galaxy images labelled by morphology (shape)

    Martin Walmsley. Galaxymnist: Galaxy images labelled by morphology (shape). https://github.com/mwalmsley/galaxy_mnist, 2022. Accessed: 2025-05-14

    2022

  53. [61]

    Galaxy zoo decals: Detailed visual morphology measurements from volunteers and deep learning for 314 000 galaxies

    Mike Walmsley, Chris Lintott, Tobias G \'e ron, Sandor Kruk, Coleman Krawczyk, Kyle W Willett, Steven Bamford, Lee S Kelvin, Lucy Fortson, Yarin Gal, et al. Galaxy zoo decals: Detailed visual morphology measurements from volunteers and deep learning for 314 000 galaxies. Month...

    2022

  54. [62]

    Towards galaxy foundation models with hybrid contrastive learning

    Mike Walmsley, Inigo Val Slijepcevic, Micah Bowles, and Anna MM Scaife. Towards galaxy foundation models with hybrid contrastive learning. arXiv preprint arXiv:2206.11927, 2022 b

    2022 arXiv

  55. [63]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2020

    2020

  56. [64]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 a

    2024 arXiv

  57. [65]

    Effective fine-tuning of vision-language models for accurate galaxy morphology analysis

    Ruoqi Wang, Haitao Wang, and Qiong Luo. Effective fine-tuning of vision-language models for accurate galaxy morphology analysis. arXiv preprint arXiv:2411.19475, 2024 b

    2024 arXiv

  58. [66]

    Robustness to corruption in pre-trained bayesian neural networks

    Xi Wang and Laurence Aitchison. Robustness to corruption in pre-trained bayesian neural networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=kUI41mY8bHl

    2023

  59. [67]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  60. [68]

    How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020

    Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub \'S wi a tkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020

    2002 arXiv

  61. [69]

    Imagenet-ood: Deciphering modern out-of-distribution detection algorithms

    William Yang, Byron Zhang, and Olga Russakovsky. Imagenet-ood: Deciphering modern out-of-distribution detection algorithms. arXiv preprint arXiv:2310.01755, 2023

    2023 arXiv

  62. [70]

    Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation

    Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation. arXiv preprint arXiv:2411.11919, 2024

    2024 arXiv

  63. [71]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.