Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Vision-language models systematically lose over half their factual recall when an entity is shown in an image instead of named in text, and linear probes on their hidden states can flag the failure at inference time.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Vision-language models recall facts much better when the entity is named in text than when the same entity appears only in an image, and hidden-state probes can detect many of these recall failures.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A careful empirical study that shows VLMs are worse at recalling facts from images than from names, though the 50% gap is inflated by anaphoric wording, and the probe's OOD results are weaker than the abstract implies. the 4 major comments →

arxiv 2508.18297 v1 pith:5YKNSHZS submitted 2025-08-22 cs.CV cs.AIcs.CL

Can VLMs Recall Factual Associations From Visual References?

Dhananjay Ashok , Ashutosh Chaubey , Hirona J. Arai , Jonathan May , Jesse Thomason This is my paper
classification cs.CV cs.AIcs.CL
keywords vision-language modelsmultimodal groundingfactual recallvisual question answeringhidden stateslinear probesselective predictiongrounding gap
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a targeted question: once a vision-language model has stored a fact about an entity, can it retrieve that fact when the entity appears only as an image? To answer it, the authors built a controlled testbed in which every factual question has two versions — one naming the entity in text, one referring only to 'the landmark in the image' — and filtered out cases where the model misidentifies the object, lacks the knowledge, or could guess from the wording alone. Across seven VLMs from three model families, accuracy in the visual condition is 42.05% versus 84.70% when the entity is named: a halving that holds for every model and every dataset. The paper then shows this grounding gap is visible in the model's hidden states: linear probes flag unreliable responses with over 92% in-distribution accuracy, transfer without retraining to the OKVQA benchmark, and improve selective prediction there. If correct, systems that are expected to trigger stored knowledge from images are systematically unreliable, in a detectable way, and could be made to abstain when they would otherwise fail.

Core claim

On the paper's own terms, a consistently grounded VLM should recall facts about an entity whether the entity is named in text or shown in an image, and none of the tested models do this. Models that answer 84.70% of text-referenced questions correctly drop to 42.05% when forced to rely on the image alone — a relative decline of 58.95% across every model and dataset, including 98.66% to 31.28% on MNIST arithmetic. The paper claims this is a linking failure, not a knowledge deficit: the VLM identifies the entity, holds the relevant fact, and answers correctly when image and name are both supplied, yet cannot trigger the fact from the image representation alone. It further claims the failure is

What carries the argument

The load-bearing object is a paired benchmark that isolates the modality of the entity reference. Each factual question about an entity (e.g., 'In what year was Stonehenge legally protected?') has a textual version naming the entity and a visual version that refers only to 'the landmark in the image'; entity images come from single-object classification datasets, and multi-stage filtering keeps only datapoints where the VLM correctly identifies the entity, correctly answers with both image and name supplied, and cannot answer from a trivial image — so the only difference between conditions is how the entity is referenced. The detection apparatus is a linear probe trained on the 20th-layer hi

Load-bearing premise

The textual and visual versions of each question are assumed to differ only in the modality of the entity reference, but the visual version also swaps the entity's name for an anaphoric phrase like 'the landmark in the image,' adding parsing difficulty the textual condition does not carry; the per-model filtering also trusts the VLM's own identification as ground truth for whether it recognized the entity.

What would settle it

Take the visual condition of the benchmark and append the model's own one-token identification as a caption — image of Stonehenge plus the text 'Stonehenge', question still phrased as 'the landmark in the image'. If accuracy returns to the text-only level, the gap is a failure to produce or key the entity's name, not a failure to link image features to stored facts. A second check: apply the probe to deliberately hard but correctly answered questions; if it still flags them as linking failures, it is tracking task difficulty rather than grounding failure.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark scores on knowledge-intensive visual QA cannot be read as evidence of visual grounding; the modality-controlled protocol is what separates grounding failures from misidentification, missing knowledge, and guessing.
  • Because the gap persists in the largest proprietary model and across model scales, scaling parameters and data alone will not close it — the paper reads this as a capability that pretraining never fully teaches.
  • Deployed VLMs can abstain on questions their hidden states flag as unreliable: on out-of-distribution OKVQA, probe-plus-perplexity selective prediction covers 7.87% more questions while making 0.9% fewer errors.
  • Vision-language pretraining should include factual-recall objectives alongside captioning, so that the bridging module learns to convert visual representations into signals that trigger facts stored in the LM layers.
  • The gap is not restricted to obscure world knowledge: arithmetic over MNIST digits drops from 98.66% to 31.28%, so even simple visual grounding of symbols fails the same way.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 58.95% figure likely overstates a pure grounding failure: the visual condition's question text ('the landmark in the image') adds anaphoric parsing difficulty that the naming condition lacks, so part of the drop may come from language-side instruction following. A matched-pair design with identical surface syntax would sharpen the number.
  • The probe's edge over perplexity implies these failures are not low-confidence guesses — the models are often confidently wrong — so confidence calibration alone will not mitigate the gap; the detectable internal signature is the more promising lever.
  • The same paired-question template transfers directly to other modalities (audio, video) and to procedural or spatial knowledge, giving a general protocol for measuring how modality-dependent any model's knowledge access is.
  • If the root cause is that LM pretraining keys facts to name tokens rather than visual features, retrieval-style fixes — appending the model's own verbalized entity name to the visual input, or contrastively aligning image features with entity-name embeddings — are the natural next experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a controlled benchmark to test whether VLMs can recall factual associations about an entity when the entity is referenced visually (by an image) versus textually (by name). The authors build QA pairs from Wikipedia for entities in CIFAR100, Food101, and Google Landmarks, plus a synthetic MNIST arithmetic task. For each question, they create a Text Only version that names the entity and a Visual version that refers to 'the image'. They filter out datapoints where the VLM misidentifies the entity, cannot answer with both text and image, or answers correctly from a trivial image. They evaluate seven VLMs and report a large performance drop in the Visual condition (average 58.95% degradation), claim the drop occurs for every model, and then train linear probes on layer-20 hidden states to detect 'linking failures', reporting above 92% accuracy on the synthetic benchmarks and improved selective prediction on OKVQA. The paper also proposes future training-data augmentation to close the gap.

Significance. If the central claim holds, the paper identifies a systematic, practically relevant deficiency in multimodal grounding: VLMs access factual knowledge much better from a named entity than from an image of the same entity, and this failure is partially detectable from internal states. The strengths are the disclosed filtering protocol, the multiple model families and datasets, the synthetic MNIST control, the public code release, and the concrete selective-prediction application. These are valuable assets. However, the central behavioral comparison is confounded by a difference in referring expression, and the probe evaluation lacks a stated train/test split; both need to be resolved before the headline claims can be accepted at face value.

major comments (4)
  1. [Section 4, Figure 3] The Text Only and Visual conditions differ in two ways: the modality of the entity reference and the type of referring expression. Text Only uses a proper name ('In what year was Stonehenge legally protected?'), while Visual uses a generic definite description plus anaphora ('In what year was the landmark in the image legally protected?'). The filtering in Section 3 controls for recognition and for language-prior answers, but it does not control for the additional linguistic difficulty of hypernym selection and anaphora resolution. The Full Info condition used in Figure 4 reintroduces the proper name and therefore cannot serve as a wording control. Consequently, the reported 58.95% average drop may overstate a pure modality effect. The authors should add a control condition that holds the referring expression fixed while varying only whether the entity is supplied as an image or as text,
  2. [Section 6.2, Table 2] The in-distribution probe accuracies (92-99%) are reported without any description of a train/test split or cross-validation procedure. If the linear probe is trained and evaluated on the same filtered datapoints, these numbers are in-sample and likely inflated. The cross-dataset OKVQA result in Table 1 is more convincing, but the within-dataset probe claim is load-bearing for the abstract's 'over 92% accuracy' statement. Please specify the exact data partitioning, hyperparameters, and whether the reported accuracies are averages over folds; also report confidence intervals. Additionally, the abstract's 'over 92%' does not hold for OKVQA (64.32% in Table 2), so the claim should be qualified to the in-distribution benchmarks.
  3. [Section 3, Appendix A, Table 4] The paper states 'with substantial inter-annotator agreement (κ > 0.65)', but Table 4 reports Fleiss κ of 12.85 (CIFAR100 relevance) and 17.23 (CIFAR100 correctness), labelled 'Slight' in the table itself. The aggregate 'All' κ ≈ 65 is dominated by the Landmarks subset. This internal inconsistency weakens the claim that the QA pairs are of uniformly high quality across datasets. Please report per-dataset κ and revise the main-text claim to acknowledge the low agreement on CIFAR100, or provide additional evidence that the CIFAR100 questions are valid.
  4. [Section 5, Figure 3] The text claims 'Every VLM we investigate... with answer accuracy consistently falling by over 50%'. This is stronger than the data support. For example, LLaVA-Mistral-7B on CIFAR100 appears to drop from roughly 80% to 50% (about 37% relative decline), and other model-dataset bars show smaller relative drops. The later sentence 'on average, a 58.95% performance degradation' is accurate, but the 'every VLM' wording is not. Please replace the universal claim with a statement about the average drop and the range observed across models/datasets.
minor comments (6)
  1. [Abstract] The 'over 92% accuracy' statement refers to the in-distribution benchmarks in Table 2, not the OKVQA out-of-distribution result (64.32%). Qualify the claim to avoid overgeneralization.
  2. [Table 1] The '∆ (Perplexity→Ensemble)' columns are not fully defined. Clarify whether the delta is computed as Ensemble minus Perplexity for each metric, and add standard errors or significance tests.
  3. [Figure 3 caption] The caption says 'Empty bars for InstructBLIP are datasets where fewer than 100 datapoints passed filters.' This explanation appears only in the text; please add it to the caption for self-containment.
  4. [Section 6.1] The phrase 'mid-to-late layers, 15-25' implies a specific layer count for LLaVA-Vicuna-7B. State the total number of transformer layers in that model so the reader can interpret the range.
  5. [Appendix B] The description of the 'None' trivial image for LLaVA is confusing: 'the image representation that results is then an average image representation over all images seen during the VLMs training'. Explain how this average is computed and why it is a valid trivial image.
  6. [Section 6.2] Typo: 'occured' should be 'occurred'.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical measurement with external OOD validation; the self-referential benchmark filtering and same-model probe training are standard practices, not definitional reductions.

full rationale

The paper is an empirical study, not a derivation chain, so there is no equation-level circularity. The central comparison (Text Only vs Visual) is a measured behavioral gap on a held-out testbed, and the probe results are supervised classifiers evaluated on held-out and out-of-distribution (OKVQA) data, which provides independent support. The benchmark is filtered per-VLM using the VLM's own identification and Full-Info answers to ensure the model 'contains the relevant internal knowledge'; this is a selection step, not a parameter fitted to the target outcome, so it does not make the reported Visual-accuracy drop true by construction. Similarly, training a linear probe on hidden states from the same model and evaluating on answer correctness is a standard supervised setting; it is self-referential but not circular, and the paper reports an out-of-distribution OKVQA accuracy of 64.32%, rather than claiming the 92% in-distribution number generalizes unchanged. The main validity concern—that the textual condition names the entity while the visual condition uses an anaphoric description ('the landmark in the image')—is a confound that could overstate the grounding gap, but it is a correctness/construct-validity issue, not a circularity in the paper's logic. No load-bearing self-citation chains, uniqueness imports, or ansatz-smuggling citations were found.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The paper introduces no new theoretical entities. The linear probe is a standard classifier, not a postulated mechanism. The central claim rests on the comparability of text and visual reference questions, the self-reported knowledge of each VLM, and the choice of hidden layer for the probe; these are the main assumptions that, if false, would change the magnitude or interpretation of the results.

free parameters (3)
  • Probe layer selection = 20
    The probe is trained on hidden states from the 20th layer of LLaVA-Vicuna-7B, chosen based on the layerwise probability and cosine similarity analysis in Section 6.1. This layer choice is a hyperparameter fit to the observed patterns and affects the reported probe accuracy.
  • Perplexity threshold for baseline = not reported
    The selective prediction baseline learns a threshold on output token perplexity, as described in Section 6.2. The threshold value is not given, but it is a fitted parameter used in the coverage/risk comparisons.
  • Ensemble weighting = not reported
    The 'Ensemble' method combines probe and perplexity signals, presumably by averaging their abstention decisions. The weighting is not specified and is tuned on the same datasets used for evaluation.
axioms (6)
  • ad hoc to paper Textual and visual reference questions are matched except for the modality of the entity reference.
    The entire controlled comparison depends on this equivalence. The actual questions differ in wording, e.g., 'Stonehenge' vs. 'the landmark in the image' (Section 3, Figure 2), which is a confound.
  • domain assumption The VLM's own identification output and its text+image answer are reliable indicators of whether it knows the entity and the fact.
    Filters in Section 3 and Appendix A use the VLM under evaluation to decide whether the entity is identified and whether the fact is contained internally. This makes the benchmark model-specific.
  • domain assumption Two-way string inclusion is a valid correctness metric.
    Section 4 defines correctness as 'r in a or a in r' using substring matching. All headline accuracy numbers depend on this lenient metric.
  • domain assumption The 20th layer hidden state of the last input token contains sufficient signal to detect linking failures.
    Section 6.2 trains the probe exclusively on this layer after observing layerwise patterns in Section 6.1. The choice is justified post hoc by the same data used for evaluation.
  • domain assumption Cosine similarity between hidden states in the Visual and Full Info settings reflects grounding quality.
    Section 6.1 interprets the divergence in cosine similarity as evidence that linking fails when internal states lack the information needed to trigger recall. This is an interpretive assumption, not a proven mechanism.
  • domain assumption Llama-3.1-8B generated QA pairs are factually correct and unambiguous after cleaning.
    QA pairs are generated by Llama-3.1-8B and filtered with additional Llama calls; only a subset of 99 questions was human-annotated (Appendix A). The benchmark quality rests on this pipeline.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Can VLMs Recall Factual Associations From Visual References?." pith.science (2026). https://pith.science/paper/5YKNSHZS

@misc{pith2026250818297,
  author       = {Pith},
  title        = {Pith review of: Can VLMs Recall Factual Associations From Visual References?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5YKNSHZS}},
  note         = {Machine review of arXiv:2508.18297}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Through a controlled study, we identify a systematic deficiency in the multimodal grounding of Vision Language Models (VLMs). While VLMs can recall factual associations when provided a textual reference to an entity; their ability to do so is significantly diminished when the reference is visual instead. Forcing VLMs to rely on image representations of an entity halves their ability to recall factual knowledge, suggesting that VLMs struggle to link their internal knowledge of an entity with its image representation. We show that such linking failures are correlated with the expression of distinct patterns in model internal states, and that probes on these internal states achieve over 92% accuracy at flagging cases where the VLM response is unreliable. These probes can be applied, without retraining, to identify when a VLM will fail to correctly answer a question that requires an understanding of multimodal input. When used to facilitate selective prediction on a visual question answering task, the probes increase coverage by 7.87% (absolute) while also reducing the risk of error by 0.9% (absolute). Addressing the systematic, detectable deficiency is an important avenue in language grounding, and we provide informed recommendations for future directions.

Figures

Figures reproduced from arXiv: 2508.18297 by Ashutosh Chaubey, Dhananjay Ashok, Hirona J. Arai, Jesse Thomason, Jonathan May.

Figure 1
Figure 1. Figure 1: Despite being able to recall its internal knowl [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: For each question, we create a textual reference version which names the entity in the text, and a visual reference version in which the text only refers to the image. Similarly, we have an entity image that identifies the entity, and an uninformative image (e.g., uniformly random pixels) which contains no information about the entity. We precisely control the modality that contains entity-specific informa… view at source ↗
Figure 3
Figure 3. Figure 3: Performance (two-way inclusion) on VLMs when forced to rely on visual ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cases of linking failure correlate with the expression of distinct patterns in the hidden states produced by [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Exact Match Performance on VLMs when forced to rely on visual representations ( [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: BLEU Performance on VLMs when forced to rely on visual representations ( [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    Retrieve the Wikipedia entry for a given entity

    work page

  2. [2]

    Parse the entry to split it such that every split of the entry consists of no more than two sen- tences

    work page

  3. [3]

    Remove all splits that do not contain the entity name

    work page

  4. [4]

    The questions then undergo multiple rounds of data cleaning:

    For each split, prompt Llama-3.1-8B to gener- ate question-answer pairs from it We note that this use case is in line with the li- cense for the API https://www.mediawiki.org/ wiki/Special:Version/License/MediaWiki. The questions then undergo multiple rounds of data cleaning:

    work page

  5. [5]

    Finally, the QA pairs are deduplicated using both exact match and Llama-3.1-8B

    Remove pair if Llama-3.1-8B answers the question incorrectly. Finally, the QA pairs are deduplicated using both exact match and Llama-3.1-8B. We then randomly sample 5 images of the entity and pair them with a single QA pair to make 5 datapoints in our testbed. The four image classification datasets (MNIST, Food101, Landmarks and CIFAR100) have either a C...

    work page

  6. [11]

    Remove pair if answer has more than 7 words

    work page

  7. [12]

    This is because we want to measure factual recall, not the ability to state the object in the image

    Remove pair if answer contains entity. This is because we want to measure factual recall, not the ability to state the object in the image

    work page

  8. [13]

    This is to ensure the text alone can identify the entity

    Remove pair if the question does not contain the entity. This is to ensure the text alone can identify the entity

    work page

  9. [14]

    This helps reduce the number of ambiguous, unclear or subjective questions that are not fact-based

    Remove pair if Llama-3.1-8B decides that its answer is not unique. This helps reduce the number of ambiguous, unclear or subjective questions that are not fact-based

    work page

  10. [16]

    Remove an Image, QA pair if the VLM cannot identify the entity in the image

    work page

  11. [17]

    This ensures the VLM being tested contains the relevant internal knowledge

    Remove an Image, QA pair if the VLM cannot answer the QA pair correctly when provided both the textual reference question and the im- age (see Figure 2 and Section 3). This ensures the VLM being tested contains the relevant internal knowledge

    work page

  12. [18]

    This filters questions that are trivially easy to answer using language priors from the sentence

    Remove an Image, QA pair if the VLM can correctly answer the QA pair when provided the visual reference question and a trivial im- age. This filters questions that are trivially easy to answer using language priors from the sentence. After all steps of data filtering, the number of points in our testbed is dataset and VLM specific, with an average of 955 ...

    work page

  13. [19]

    The standard for Correctness is whether or not the answer provided an acceptable answer for the given question

    QA Pair Correctness 12 Dataset VLM LM Datapoints MNIST LLaV A Mistral-7B 633 Vicuna-7B 674 Vicuna-13B 590 InstructBLIP Vicuna-7B 423 Vicuna-13B 464 GPT4V 4o-mini 873 4o 983 CIFAR100 LLaV A Mistral-7B 1411 Vicuna-7B 1232 Vicuna-13B 1222 InstructBLIP Vicuna-7B 73 Vicuna-13B 155 GPT4V 4o-mini 1556 4o 1536 Food101 LLaV A Mistral-7B 819 Vicuna-7B 542 Vicuna-13...

    work page 2011

  14. [20]

    Black: A pure black image

    work page

  15. [21]

    White: A pure white image

    work page

  16. [22]

    Noise: A noised image where each colour channel (RGB) value for each pixel is sampled uniformly from 0 to 255

    work page

  17. [23]

    InstructBLIP), we do only a for- ward pass through the LM

    None: For VLMs that leave the LM un- changed (e.g. InstructBLIP), we do only a for- ward pass through the LM. For VLMs which tune the LM in any way (e.g. LLaV A), we pass in a null image. The image represen- tation that results is then an average image representation over all images seen during the VLMs training. We collect the output of the VLM from all ...

    work page

  18. [24]

    A cluster with 5 RTX 2080 TI GPUs. 64GB of RAM and 5 Intel(R) Xeon(R) Gold 5215 CPUs @2.50 GHz 16 /uni0000002f/uni0000002f/uni00000044/uni00000039/uni00000024 /uni00000030/uni0000004c/uni00000056/uni00000057/uni00000055/uni00000044/uni0000004f/uni00000010/uni0000001a/uni00000025 /uni0000002f/uni0000002f/uni00000044/uni00000039/uni00000024 /uni00000039/uni...

    work page 2080

  19. [25]

    800GB of RAM and 256 AMD EPYC 9554 64-Core Processor

    A cluster with 8 NVIDIA L40s GPUs (48G GPU memory each). 800GB of RAM and 256 AMD EPYC 9554 64-Core Processor

    work page

  20. [26]

    The time taken to run each part of the experiment varies with the hardware used

    A cluster with 4 NVIDIA A100 GPUs (40G or 80G GPU memory each), 514GB of RAM, and 96-core Intel(R) Xeon(R) Platinum 8272CL CPUs @2.60GHz. The time taken to run each part of the experiment varies with the hardware used. However typically all experiments for a single dataset (start to finish from data generation to final results) can be com- pleted within 4...

    work page

  21. [2000]

    IEEE Trans- actions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84–94

    To reject or not to reject: that is the question- an answer in case of neural classifiers. IEEE Trans- actions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84–94. Li Deng. 2012. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141– 142. Ran ...

    Pith/arXiv arXiv 2012

  22. [2014]

    In European Conference on Computer Vision

    Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, Yutao Zeng, Zhoufutu Wen, Ke Jin, B...

    Pith/arXiv arXiv 2020

  23. [2021]

    selective predic- tion

    Word representation learning in multimodal pre-trained transformers: An intrinsic evaluation. Transactions of the Association for Computational Linguistics, 9:1563–1579. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steer- ing llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual M...

    work page 2024

  24. [2023]

    In Proceedings of the 12th Joint Conference on Lex- ical and Computational Semantics (* SEM 2023) , pages 11–17

    Leverage points in modality shifts: Comparing language-only and multimodal word representations. In Proceedings of the 12th Joint Conference on Lex- ical and Computational Semantics (* SEM 2023) , pages 11–17. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimoda...

    Pith/arXiv arXiv 2023

  25. [2024]

    Foundations and Trends® in Computer Graphics and Vision, 16(1- 2):1–214

    Multimodal foundation models: From spe- cialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision, 16(1- 2):1–214. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In International conference on ma- ...

    work page 2023

  26. [2025]

    ArXiv, abs/2502.13059

    Simplevqa: Multimodal factuality evalua- tion for multimodal large language models. ArXiv, abs/2502.13059. Ido Cohen, Daniela Gottesman, Mor Geva, and Raja Giryes. 2025. Performance gap in entity knowledge extraction across modalities in vision language mod- els. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V...

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.