{"id":"aad69aa1-5ab4-4455-8eeb-a72e155d960f","arxiv_id":"2508.18297","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Vision-language models recall facts much better when the entity is named in text than when the same entity appears only in an image, and hidden-state probes can detect many of these recall failures.","lead":"This paper builds a controlled test where the same factual question is asked with the entity named in text versus with only an image of the entity, and finds that seven vision-language models answer far worse in the image-only condition. It then trains a linear probe on the models' hidden states that can flag many of these failures, and shows modest gains when used to decide when a VLM should abstain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual questions use anaphoric wording ('the landmark in the image') vs. named entities; the reported halving may partly reflect reference-resolution difficulty, not modality alone.","rationale":"The reader's verdict is CONDITIONAL and its weakest_assumption is exactly the matched-question confound: the text condition names the entity while the visual condition uses an anaphoric description. I agree that this is the most load-bearing concern for the paper's central behavioral claim. The paper is otherwise unusually careful: it uses deterministic greedy decoding, filters per VLM to ensure the model can identify the entity and contains the fact, removes questions answerable from language priors via trivial images, and reports human annotation of QA quality. Those controls make a pure recognition-failure explanation unlikely. However, none of the controls vary the referring expression while holding modality constant. The 'halving' could therefore be inflated by anaphora/parsing difficulty, and the paper's abstract and Section 5 state the conclusion as if modality were the only manipulated factor. The proposed textual-description condition would settle this directly: it keeps the anaphoric wording and the need to resolve a description, while replacing only the image with an equivalent text. If the result is close to Text Only, the central claim survives; if it is close to Visual, the paper overstates the modality-specific effect and would need to be reframed. The probe generalization issue (OKVQA 64.32% vs the in-distribution >92%) is also real and should be corrected in the abstract, but it is secondary to the behavioral claim that motivates the whole paper. I therefore see no reason to change the reader's CONDITIONAL verdict: the concern is substantial, concrete, and testable, but not sufficient to reject the paper outright.","tokens_in":26414,"tokens_out":7067,"duration_ms":88810,"concrete_test":"Add a third 'textual description' condition to Section 4: for each datapoint, keep the visual-reference sentence frame ('In what year was the landmark in the [image|description] legally protected?') and replace the entity image with a unique, entity-identifying textual description (e.g., for Stonehenge: 'a prehistoric monument on Salisbury Plain in Wiltshire, England, consisting of a ring of standing stones'). Measure accuracy on the same VLMs with the same filters. If the description condition matches Text Only accuracy (i.e., no drop from removing the proper noun), the reported halving is attributable to the image modality; if it matches Visual accuracy, the anaphoric/generic-reference wording is the likely driver and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central behavioral comparison (Section 4, Figure 3) contrasts Text Only ('In what year was Stonehenge legally protected?') with Visual ('In what year was the landmark in the image legally protected?'). These differ in two ways: entity modality and the referring expression (proper name vs. generic definite description + anaphor). The paper filters out recognition failures ('ask the VLM... to identify the object... retain only if identification is accurate'), so the residual gap is attributed entirely to 'linking internal knowledge with image representation.' But a VLM might fail the Visual condition because 'the landmark in the image' is a harder linguistic stimulus than the token 'Stonehenge'—e.g., hypernym selection and anaphora resolution—even when the image is correctly recognized. The Full Info condition used in filtering and in Figure 4 reintroduces the proper name and therefore cannot control for this; no condition holds the referring expression fixed while varying only whether the entity is supplied as an image or as text/name. The MNIST results (98.66% vs 31.28%) and consistency across models make a pure modality effect plausible, but the magnitude 'halves factual recall' and the claim that VLMs 'struggle to link internal knowledge with image representation' depend on the unstated assumption that anaphoric wording is neutral. This is testable and should be resolved before the headline is accepted at face value. The probe claims (Section 6.2, Table 2) are a separate, also serious concern: the 'over 92%' accuracy is in-distribution and drops to 64.32% on OKVQA, just a few points above the perplexity baseline, which undermines the abstract's unqualified 'without retraining' warning-signal claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a controlled benchmark to test whether VLMs can recall factual associations about an entity when the entity is referenced visually (by an image) versus textually (by name). The authors build QA pairs from Wikipedia for entities in CIFAR100, Food101, and Google Landmarks, plus a synthetic MNIST arithmetic task. For each question, they create a Text Only version that names the entity and a Visual version that refers to 'the image'. They filter out datapoints where the VLM misidentifies the entity, cannot answer with both text and image, or answers correctly from a trivial image. They evaluate seven VLMs and report a large performance drop in the Visual condition (average 58.95% degradation), claim the drop occurs for every model, and then train linear probes on layer-20 hidden states to detect 'linking failures', reporting above 92% accuracy on the synthetic benchmarks and improved selective prediction on OKVQA. The paper also proposes future training-data augmentation to close the gap.","tokens_in":26714,"tokens_out":4766,"duration_ms":57959,"significance":"If the central claim holds, the paper identifies a systematic, practically relevant deficiency in multimodal grounding: VLMs access factual knowledge much better from a named entity than from an image of the same entity, and this failure is partially detectable from internal states. The strengths are the disclosed filtering protocol, the multiple model families and datasets, the synthetic MNIST control, the public code release, and the concrete selective-prediction application. These are valuable assets. However, the central behavioral comparison is confounded by a difference in referring expression, and the probe evaluation lacks a stated train/test split; both need to be resolved before the headline claims can be accepted at face value.","major_comments":[{"comment":"The Text Only and Visual conditions differ in two ways: the modality of the entity reference and the type of referring expression. Text Only uses a proper name ('In what year was Stonehenge legally protected?'), while Visual uses a generic definite description plus anaphora ('In what year was the landmark in the image legally protected?'). The filtering in Section 3 controls for recognition and for language-prior answers, but it does not control for the additional linguistic difficulty of hypernym selection and anaphora resolution. The Full Info condition used in Figure 4 reintroduces the proper name and therefore cannot serve as a wording control. Consequently, the reported 58.95% average drop may overstate a pure modality effect. The authors should add a control condition that holds the referring expression fixed while varying only whether the entity is supplied as an image or as text,","section":"Section 4, Figure 3"},{"comment":"The in-distribution probe accuracies (92-99%) are reported without any description of a train/test split or cross-validation procedure. If the linear probe is trained and evaluated on the same filtered datapoints, these numbers are in-sample and likely inflated. The cross-dataset OKVQA result in Table 1 is more convincing, but the within-dataset probe claim is load-bearing for the abstract's 'over 92% accuracy' statement. Please specify the exact data partitioning, hyperparameters, and whether the reported accuracies are averages over folds; also report confidence intervals. Additionally, the abstract's 'over 92%' does not hold for OKVQA (64.32% in Table 2), so the claim should be qualified to the in-distribution benchmarks.","section":"Section 6.2, Table 2"},{"comment":"The paper states 'with substantial inter-annotator agreement (κ > 0.65)', but Table 4 reports Fleiss κ of 12.85 (CIFAR100 relevance) and 17.23 (CIFAR100 correctness), labelled 'Slight' in the table itself. The aggregate 'All' κ ≈ 65 is dominated by the Landmarks subset. This internal inconsistency weakens the claim that the QA pairs are of uniformly high quality across datasets. Please report per-dataset κ and revise the main-text claim to acknowledge the low agreement on CIFAR100, or provide additional evidence that the CIFAR100 questions are valid.","section":"Section 3, Appendix A, Table 4"},{"comment":"The text claims 'Every VLM we investigate... with answer accuracy consistently falling by over 50%'. This is stronger than the data support. For example, LLaVA-Mistral-7B on CIFAR100 appears to drop from roughly 80% to 50% (about 37% relative decline), and other model-dataset bars show smaller relative drops. The later sentence 'on average, a 58.95% performance degradation' is accurate, but the 'every VLM' wording is not. Please replace the universal claim with a statement about the average drop and the range observed across models/datasets.","section":"Section 5, Figure 3"}],"minor_comments":[{"comment":"The 'over 92% accuracy' statement refers to the in-distribution benchmarks in Table 2, not the OKVQA out-of-distribution result (64.32%). Qualify the claim to avoid overgeneralization.","section":"Abstract"},{"comment":"The '∆ (Perplexity→Ensemble)' columns are not fully defined. Clarify whether the delta is computed as Ensemble minus Perplexity for each metric, and add standard errors or significance tests.","section":"Table 1"},{"comment":"The caption says 'Empty bars for InstructBLIP are datasets where fewer than 100 datapoints passed filters.' This explanation appears only in the text; please add it to the caption for self-containment.","section":"Figure 3 caption"},{"comment":"The phrase 'mid-to-late layers, 15-25' implies a specific layer count for LLaVA-Vicuna-7B. State the total number of transformer layers in that model so the reader can interpret the range.","section":"Section 6.1"},{"comment":"The description of the 'None' trivial image for LLaVA is confusing: 'the image representation that results is then an average image representation over all images seen during the VLMs training'. Explain how this average is computed and why it is a valid trivial image.","section":"Appendix B"},{"comment":"Typo: 'occured' should be 'occurred'.","section":"Section 6.2"}],"recommendation":"major_revision","confidential_remarks":"The core finding is plausible and potentially important, but the load-bearing behavioral comparison confounds modality with referring-expression type, and the probe results need a clear held-out evaluation. These are fixable with additional experiments and reporting, so I recommend major revision rather than rejection. I would also ask the authors to revisit the 'every VLM' overstatement and the annotation-agreement inconsistency, as they affect the paper's credibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, this paper builds the closest thing we have to a controlled comparison of factual recall from visual vs textual entity references, across multiple datasets and model families. The direction of the finding is almost certainly right: VLMs do worse when the entity is only in the image. Second, the headline 'halves factual recall' and 'over 92% probe accuracy' are both stronger than what the experiments actually support. The behavioral gap is confounded with anaphoric wording, and the probe's OOD accuracy drops to 64.32%, just a few points above a perplexity threshold.\n\nWhat's genuinely good: The benchmark construction is careful. They filter per model to ensure the model can identify the entity and knows the fact in the text+image condition, and they remove questions answerable from language priors. The MNIST addition/multiplication task is a nice minimal control: '5 + 17' vs 'the digit in the image + 17', and the gap is enormous (98.66% vs 31.28%). That makes a pure modality effect plausible, not just a linguistic artifact. The consistency across InstructBLIP, LLaVA, and GPT-4V, and across scales, is a solid empirical result. The hidden-state probe is a reasonable extension, and the logit-lens style visualization is a nice mechanistic complement. The anonymous code release helps reproducibility.\n\nThe soft spots, in order of importance:\n\n1. The central comparison is not matched. The text condition says 'Stonehenge'; the visual condition says 'the landmark in the image'. That's a proper name vs a generic description plus anaphora. Filtering on identification accuracy does not remove the added linguistic difficulty of resolving 'the landmark'. The Full Info condition reintroduces the name, so it can't serve as a control. The MNIST result mitigates this but doesn't eliminate it; a condition that holds the referring expression fixed while varying only the modality (e.g., naming the entity in text vs showing its image while keeping the rest of the prompt identical) would settle it. This is testable and fixable, and it matters for the magnitude claim.\n\n2. The abstract overstates the probe. 'Over 92% accuracy' is in-distribution on the training datasets. Applied to OKVQA without retraining, accuracy is 64.32%, close to the perplexity baseline (60.16%). The selective prediction gains are real but modest: +7.87% coverage, -0.9% risk, and the absolute risk on OKVQA is still 41.5%. The 'can be applied, without retraining' sentence should be qualified.\n\n3. Minor: no error bars on the main results, though greedy decoding makes runs deterministic. The per-model filtering makes the benchmark model-specific, which is appropriate for isolating knowledge access but limits cross-model comparability of the numbers.\n\nOverall, this paper deserves a serious referee. It's a thoughtful, transparent empirical study, and the authors disclose their filtering in unusual detail, even including a limitations section that correctly notes they haven't identified the cause. A referee should ask for the matched-wording control and a rewrite of the abstract's probe claim. I'd be happy to see it in the cycle after those revisions.","headline":"A careful empirical study that shows VLMs are worse at recalling facts from images than from names, though the 50% gap is inflated by anaphoric wording, and the probe's OOD results are weaker than the abstract implies.","tokens_in":27282,"tokens_out":3824,"would_cite":true,"duration_ms":41553,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language models systematically lose over half their factual recall when an entity is shown in an image instead of named in text, and linear probes on their hidden states can flag the failure at inference time.","keywords":["vision-language models","multimodal grounding","factual recall","visual question answering","hidden states","linear probes","selective prediction","grounding gap"],"falsifier":"Take the visual condition of the benchmark and append the model's own one-token identification as a caption — image of Stonehenge plus the text 'Stonehenge', question still phrased as 'the landmark in the image'. If accuracy returns to the text-only level, the gap is a failure to produce or key the entity's name, not a failure to link image features to stored facts. A second check: apply the probe to deliberately hard but correctly answered questions; if it still flags them as linking failures, it is tracking task difficulty rather than grounding failure.","tokens_in":26279,"feed_emoji":"🖼️","tokens_out":11369,"duration_ms":106087,"temperature":0.7,"pith_summary":"The paper asks a targeted question: once a vision-language model has stored a fact about an entity, can it retrieve that fact when the entity appears only as an image? To answer it, the authors built a controlled testbed in which every factual question has two versions — one naming the entity in text, one referring only to 'the landmark in the image' — and filtered out cases where the model misidentifies the object, lacks the knowledge, or could guess from the wording alone. Across seven VLMs from three model families, accuracy in the visual condition is 42.05% versus 84.70% when the entity is named: a halving that holds for every model and every dataset. The paper then shows this grounding gap is visible in the model's hidden states: linear probes flag unreliable responses with over 92% in-distribution accuracy, transfer without retraining to the OKVQA benchmark, and improve selective prediction there. If correct, systems that are expected to trigger stored knowledge from images are systematically unreliable, in a detectable way, and could be made to abstain when they would otherwise fail.","feed_headline":"VLMs lose half their factual recall when entities appear in images","feed_subtitle":"Hidden-state probes flag when a model will answer wrongly from an image, enabling safe abstention.","key_machinery":"The load-bearing object is a paired benchmark that isolates the modality of the entity reference. Each factual question about an entity (e.g., 'In what year was Stonehenge legally protected?') has a textual version naming the entity and a visual version that refers only to 'the landmark in the image'; entity images come from single-object classification datasets, and multi-stage filtering keeps only datapoints where the VLM correctly identifies the entity, correctly answers with both image and name supplied, and cannot answer from a trivial image — so the only difference between conditions is how the entity is referenced. The detection apparatus is a linear probe trained on the 20th-layer hi","core_discovery":"On the paper's own terms, a consistently grounded VLM should recall facts about an entity whether the entity is named in text or shown in an image, and none of the tested models do this. Models that answer 84.70% of text-referenced questions correctly drop to 42.05% when forced to rely on the image alone — a relative decline of 58.95% across every model and dataset, including 98.66% to 31.28% on MNIST arithmetic. The paper claims this is a linking failure, not a knowledge deficit: the VLM identifies the entity, holds the relevant fact, and answers correctly when image and name are both supplied, yet cannot trigger the fact from the image representation alone. It further claims the failure is","pith_inferences":["The 58.95% figure likely overstates a pure grounding failure: the visual condition's question text ('the landmark in the image') adds anaphoric parsing difficulty that the naming condition lacks, so part of the drop may come from language-side instruction following. A matched-pair design with identical surface syntax would sharpen the number.","The probe's edge over perplexity implies these failures are not low-confidence guesses — the models are often confidently wrong — so confidence calibration alone will not mitigate the gap; the detectable internal signature is the more promising lever.","The same paired-question template transfers directly to other modalities (audio, video) and to procedural or spatial knowledge, giving a general protocol for measuring how modality-dependent any model's knowledge access is.","If the root cause is that LM pretraining keys facts to name tokens rather than visual features, retrieval-style fixes — appending the model's own verbalized entity name to the visual input, or contrastively aligning image features with entity-name embeddings — are the natural next experiments."],"forward_implications":["Benchmark scores on knowledge-intensive visual QA cannot be read as evidence of visual grounding; the modality-controlled protocol is what separates grounding failures from misidentification, missing knowledge, and guessing.","Because the gap persists in the largest proprietary model and across model scales, scaling parameters and data alone will not close it — the paper reads this as a capability that pretraining never fully teaches.","Deployed VLMs can abstain on questions their hidden states flag as unreliable: on out-of-distribution OKVQA, probe-plus-perplexity selective prediction covers 7.87% more questions while making 0.9% fewer errors.","Vision-language pretraining should include factual-recall objectives alongside captioning, so that the bridging module learns to convert visual representations into signals that trigger facts stored in the LM layers.","The gap is not restricted to obscure world knowledge: arithmetic over MNIST digits drops from 98.66% to 31.28%, so even simple visual grounding of symbols fails the same way."],"supporting_citations":[{"why":"Supplies the LLaVA model family and the visual instruction tuning paradigm evaluated in the study.","marker":"(Liu et al., 2023)"},{"why":"Supplies the InstructBLIP model family, the frozen-LM architecture also evaluated.","marker":"(Li et al., 2023a)"},{"why":"Supplies GPT-4V, the proprietary frontier model used to show the gap persists at scale.","marker":"(OpenAI, 2023)"},{"why":"Provides OKVQA, the out-of-distribution benchmark used to test probe transfer.","marker":"(Marino et al., 2019)"},{"why":"Establishes the premise that pretrained LMs store factual associations retrievable from text.","marker":"(Petroni et al., 2019)"},{"why":"Supplies the hidden-state linear probe approach for flagging unreliable model outputs.","marker":"(Azaria and Mitchell, 2023)"},{"why":"Supplies the logit-lens technique used to visualize when the answer token gains probability across layers.","marker":"(Nostalgebraist, 2020)"},{"why":"Supplies the multiple-choice reformulation trick that makes the first generated token determine correctness.","marker":"(Rimsky et al., 2024)"},{"why":"Supplies the MNIST digits used for the arithmetic variant of the grounding test.","marker":"(Deng, 2012)"}],"fun_headline_variants":["Visual references halve VLM factual recall","VLMs lose 58% of fact recall when entity is visual","Probes flag unreliable VLM answers from images with 92% accuracy","Visual entity info cuts VLM fact recall in half","VLMs recall facts better from text than from images"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The textual and visual versions of each question are assumed to differ only in the modality of the entity reference, but the visual version also swaps the entity's name for an anaphoric phrase like 'the landmark in the image,' adding parsing difficulty the textual condition does not carry; the per-model filtering also trusts the VLM's own identification as ground truth for whether it recognized the entity.","fun_headline_variants_meta":{"raw":{"variants":["Visual references halve VLM factual recall","VLMs lose 58% of fact recall when entity is visual","Probes flag unreliable VLM answers from images with 92% accuracy","Visual entity info cuts VLM fact recall in half","VLMs recall facts better from text than from images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000386,"raw_usage":{"total_tokens":1875,"prompt_tokens":740,"completion_tokens":1135,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":1054}},"tokens_in":484,"tokens_out":1135,"duration_ms":9946,"temperature":1.0,"reasoning_tokens":1054,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:14:28.719208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the visual condition of the benchmark and append the model's own one-token identification as a caption — image of Stonehenge plus the text 'Stonehenge', question still phrased as 'the landmark in the image'. If accuracy returns to the text-only level, the gap is a failure to produce or key the entity's name, not a failure to link image features to stored facts. A second check: apply the probe to deliberately hard but correctly answered questions; if it still flags them as linking failures, it is tracking task difficulty rather than grounding failure.","supporting_citations":[],"review_version":1}