Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A vision-language model recognizes landmarks only when its vision encoder already 'knows' them; supervising visual tokens with entity-level losses substantially raises recognition.

desk verdict Useful data-curation study for LVLM fine-tuning, but the VE-Known/VE-Unknown split is defined by the same CLIP encoder used downstream, so the headline causal claim is partly definitional; still deserves a serious referee. read the letter →

arxiv 2411.16824 v1 pith:QOBGH2KY submitted 2024-11-25 cs.CV cs.AIcs.LG

Yaqi Zhao , Yuanyang Yin , Lin Li , Mingan Lin , Victor Shea-Jay Huang , Siwei Chen , Weipeng Chen , Baoqun Yin
show 2 more authors
Zenan Zhou Wentao Zhang
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords largevision-languagemodelscognitivemisalignmentCLIPsimilaritylandmarkrecognitionVE-Knowndatavisualinstructiontuningentity-awarecontrastivelossqualityversusquantity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models often fail to recognize what they can describe in words; this paper attributes that failure to cognitive misalignment between the CLIP vision encoder and the language model's interpretive space. On a roughly 200k-sample landmark dataset, the authors split training images by CLIP cosine similarity into VE-Known (confident, discriminative) and VE-Unknown (ambiguous) subsets, and find that fine-tuning on VE-Known data improves recognition while VE-Unknown data degrades it. They then propose Entity-Enhanced Cognitive Alignment (EECA), which supervises the adapter's visual tokens with an entity-aware contrastive loss and a hierarchical classification loss, lifting landmark recognition accuracy on a 5k test split from 8.68% to about 15.52% with only 50k training samples. If correct, the finding implies that data quality, measured by the vision encoder's own confidence, can substitute for data volume in LVLM fine-tuning.

What carries the argument

The load-bearing object is the CLIP vision encoder's similarity score, $\mathrm{Sim}_{\mathrm{CLIP}}(I_i,T_j)=\langle f_v(I_i),f_t(T_j)\rangle/(\|f_v(I_i)\|\|f_t(T_j)\|)$, which splits data into VE-Known (high similarity or high relative rank) and VE-Unknown (low on both) and later supplies the entity embeddings used in supervision. The EECA machinery has three parts: a dual-branch visual encoder that feeds a low-resolution overview and four high-resolution sub-images through a shared CLIP vision encoder, compressing the high-resolution tokens with a Perceiver resampler; an entity-aware contrastive loss that forms entity-grouped visual embeddings as weighted sums of high-resolution tokens and contrasts them against text entity embeddings; and a hierarchical classification loss over broad categories such as 'church' or 'tower'. These are combined with the standard language-modeling loss as $L=\lambda L_g+\mu_e L_e+\mu_h L_h$, so that visual tokens are pushed to retain the entity-specific, discriminative information the LLM needs.

What would settle it

A direct test: on the 5k test set, correlate each image's CLIP similarity to its true landmark name with whether the trained model recognizes it; if the correlation is flat or negative, the claim that VE-Known data drives recognition fails. Stronger still, re-run the HDS/LCS split using a different vision backbone such as DINOv2 and check whether the HDS advantage follows that backbone's confidence; if it does not, the effect is an artifact of the CLIP metric rather than a general property of vision-encoder knowledge.

Watch

Extended reading notes

Core claim

The paper's central claim is that cognitive misalignment, a mismatch between what CLIP's vision encoder represents and what the LLM can interpret, is a principal cause of LVLM recognition failure, and that both data curation and targeted supervision can reduce it. The authors operationalize the vision encoder's knowledge as CLIP cosine similarity between an image and the landmark-name text embedding, and show that subsets with high discriminative rank (HDS) or high similarity (HSS) outperform random sampling, while low-clarity subsets (LCS) underperform it at every size tested. Building on that, EECA trains a dual-branch high/low-resolution LLaVA-style model with an entity-aware contrastive loss that groups high-resolution visual tokens against text entity embeddings, plus a hierarchical classification loss over landmark categories; with 50k curated samples this raises accuracy from 8.68% to 15.52%, and with 25k it matches the accuracy of a 125k random reference set.

Load-bearing premise

The load-bearing premise is that CLIP's cosine similarity between an image and a landmark name genuinely measures what the vision encoder 'knows'; if that proxy is wrong, the VE-Known/VE-Unknown split, and the entity supervision built from the same CLIP embeddings, would be circular rather than explanatory.

Editorial extensions

If this is right

  • VE-Known selection (HDS) beats random sampling at every training size tested; at 50k it raises accuracy 64.1% relative to the 8.68% baseline, versus 56.2% for random sampling.
  • Adding VE-Unknown samples actively hurts: LCS-50k gives only a 28.1% relative increase, and performance plateaus as more mixed data is added, so data quality dominates volume beyond a threshold.
  • Supervising visual tokens with entity-aware contrastive and hierarchical losses adds accuracy on top of curated data (reaching 15.52% at 50k), while adding a high-resolution branch without those losses does not improve the baseline.
  • EECA's gains are not confined to easy images: it improves VE-Known and VE-Unknown subsets alike, with the high-resolution branch contributing most on low-clarity (VE-Unknown) data and the entity/hierarchical losses contributing most on VE-Known data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's VE-Known/VE-Unknown split and its contrastive supervision both lean on the same CLIP text encoder, so part of the reported gain could come from aligning visual tokens to CLIP's own embedding geometry rather than to the LLM's cognition; training with entity embeddings from an independent text encoder would isolate the mechanism.
  • Because the selection criterion is cheap (one CLIP forward pass per image), the finding suggests a practical data-curation recipe for other entity-heavy domains such as faces, species, products, and artworks, where EECA-style entity supervision could reduce the required training set size.
  • The experiments only vary the vision side while keeping the language model fixed; swapping the LLM while holding CLIP fixed would show whether the alignment is truly with the language model's knowledge or with a generic multimodal embedding space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper investigates why LVLMs fail to recognize landmarks despite having strong text-only knowledge. It constructs a Multi-granularity Landmark Dataset (MGLD) from GLDv2, partitions images into VE-Known and VE-Unknown categories using CLIP cosine similarity (Eq. 2) and relative similarity rank, and reports that fine-tuning LLaVA-1.5 on VE-Known subsets (HDS and HSS) improves landmark recognition over a random reference (BRS) and over VE-Unknown data (LCS). It then proposes Entity-Enhanced Cognitive Alignment (EECA), which combines a high-resolution branch with an entity-aware contrastive loss (Eq. 4) and a hierarchical classification loss (Eq. 5). The paper reports accuracy rising from an 8.68% baseline to 15.52% with EECA, and includes additional experiments across HDS, HSS, and LCS subsets.

Significance. If the empirical claims hold, the paper offers a practical data-selection heuristic and a training objective for fine-grained landmark recognition. The use of the public GLDv2 dataset, the relatively detailed data-construction pipeline, and the explicit statement of limitations are strengths. However, the central conceptual contribution ('cognitive misalignment') is operationalized through CLIP similarity in a way that is reused by the downstream model and the proposed losses, and the experimental support currently lacks statistical rigor. The paper would be more convincing if the VE-Known/VE-Unknown partition were validated with an independent difficulty measure and if the reported gains were accompanied by significance tests.

major comments (5)
  1. [§3.1, Eq. (2)] The VE-Known/VE-Unknown partition is defined by CLIP cosine similarity between the image and landmark-name text using the same CLIP-ViT-L-14 encoder that serves as the LVLM's vision backbone and that also supplies the entity text embeddings in Eq. (4). Stage 1 of the dataset construction already retains images with high CLIP similarity to the landmark name, so the subsequent HDS/HSS/LCS split is built on the same measure used by the downstream model. Showing that CLIP-easy training images outperform CLIP-hard ones is a useful empirical regularity, but it does not by itself establish that VE-Unknown data 'limits' LVLM capacity or that VE-Known data 'reduces cognitive misalignment'. I request an independent validation of the partition, e.g., human recognition rates or a different vision encoder, or at least controls for image-level confounds such as embedded text, resolution, and viewpoint.
  2. [Appendix B.4 vs. Appendix D.1] The paper states that 5,000 samples are set aside as the test set, but the absolute counts in Tables 6 and 7 sum to 2,500 per condition (e.g., 103+114+145+2138 = 2500). Because all reported percentages and the central improvements in Tables 1-4 are computed on this test set, the discrepancy must be resolved; if the test set is actually 2,500, every percentage should be checked, and if it is 5,000, the tables are incomplete.
  3. [Tables 1-4, Tables 6-7] No standard deviations, confidence intervals, significance tests, or multiple seeds are reported; all results are on a single test split. In the ablation (Table 3), the incremental gains over '+ HSS-50k' are small (+0.52 with Le and +1.12 with Lh), and without variance estimates these differences cannot be distinguished from noise. Please report repeated-seed runs and a paired test, or at least bootstrap intervals, for the key comparisons.
  4. [§3.1 and §C.1] GPT-4o is both the annotator that generates the MGLD answers and entities in Stages 2-3 and the judge that classifies the LVLM outputs into Strongly Known/Known/etc. Since EECA is trained on GPT-4o's annotations and then evaluated by GPT-4o's rubric, the reported gains may partly reflect imitation of the annotator's style rather than improved landmark recognition. An independent human evaluation or a second judge on a random subset is necessary to validate the EECA improvements.
  5. [§5.3, Table 4] The text states that for LCS-25k 'adding the HR branch significantly boosts performance,' but the increase is 10.68 to 12.08 (+1.40) over '+25k Data', whereas Le and Lh add only 12.08 to 12.32 (+0.24). Similarly, the claim that VE-Known data achieve the largest gains with Le/Lh is not uniformly supported: HDS is best with Lh (14.40) and HSS with Le (14.40), while Table 4 shows no consistent ordering among the losses. Please either provide significance tests for these differences or soften the qualitative claims.
minor comments (5)
  1. [Throughout] The name 'LLaVA' is repeatedly typeset as 'LLaV A' (e.g., Section 3.3, §5.1, and Appendix A); please correct.
  2. [Appendix B.1] The sentence 'The category distribution in GLDv2 training dataset is highly imbalanced, as illustrated in ??.' contains an unresolved cross-reference; also Figure 9's caption should be attached to the figure.
  3. [Eq. (4) in §4.2] The notation X_{e_{i,j}} in Eq. (4) is not defined in place; the text introduces ϕ(e_j) as the entity token embedding, but the subscripted form should be defined explicitly before the loss is presented.
  4. [§5.2, Figure 5] The left and right panels of Figure 5 use different x-axes and the two EECA points appear only in the left panel; the caption should state what the right panel's y-axis ('Percentage Increase') is relative to, and why EECA is not shown there.
  5. [§3.2] The t-SNE-based summary that LCS 'shows reduced intra-class variability' should be stated more cautiously, since t-SNE can distort local geometry and does not directly measure class separability.

Circularity Check

1 steps flagged · score 5.0 of 10

The VE-Known/VE-Unknown partition is defined by CLIP similarity using the same CLIP-ViT that serves as the LVLM vision backbone, so the central 'known data helps' result is partly a restatement of the selection criterion rather than an independent test of cognitive misalignment.

  1. self definitional [Section 3.1, Eq. (2), HDS/HSS/LCS definitions; Appendix A architecture]
    "High Discrimination Selection (HDS): This method selects images with very high RSR i values, capturing instances where the model effectively distinguishes the ground-truth Ti from other candidates. HDS emphasizes strong visual discrimination in the VE’s representations, aligning with “VE-Known” characteristics. ... Low Clarity Selection (LCS): This method selects images with both low Sim i CLIP and low RSR i values, targeting visually ambiguous cases where the model struggles with feature extraction and differentiation."

    The same CLIP-ViT-L-14 model is both the fv used in Eq. (2) and the LVLM vision backbone (Appendix A: 'We utilize CLIP-ViT-L-14 [29] as the vision encoder'). Thus the independent variable (VE-Known vs VE-Unknown) is defined by the cosine similarity of the vision encoder's own embeddings to landmark-name text, and the explanatory mechanism offered in Section 3.3 is that same vision encoder's 'knowledge'. Reporting that models trained on images whose fv embeddings are close to the correct label recognize those images better is partly a restatement of the inclusion criterion: 'VE-Known' means CLIP-easy by construction, and the trained LVLM reads images through the same CLIP encoder.

full rationale

The paper's data-splitting claim is the load-bearing circular point. Section 3.1 defines VE-Known and VE-Unknown exclusively through CLIP cosine similarity (Eq. 2) and the relative similarity rank, and Appendix A identifies the vision encoder as CLIP-ViT-L-14, the same model that provides visual features to the LVLM in all experiments. Consequently, Table 1's contrast between HDS/HSS and LCS mostly shows that CLIP-easy images are easier for a CLIP-backed LVLM than CLIP-hard images; this is a useful empirical regularity but is largely self-definitional, not evidence for the broader 'cognitive misalignment' theory. EECA's entity-aware contrastive loss is attributed by self-citation to the authors' prior SEA paper [43], but the loss is defined in the present paper with explicit equations, so that citation is attribution rather than load-bearing evidence. The use of GPT-4o to generate the MGLD annotations and also to grade the recognition levels is an evaluation-loop concern, but it does not reduce to an equation-level circularity, so it is noted here rather than scored as a circular step. Overall, the central empirical finding has independent content because held-out accuracy is measured after training the adapter and LLM, but the partition and the proposed mechanism are defined by the same CLIP encoder, justifying a partial-circularity score of 5.

Assumptions & free parameters 6 free parameters · 6 assumptions · 2 invented entities

All free parameters are hand-chosen or learned without a reported search. The main axioms are domain assumptions about CLIP similarity, GPT-4o annotations, and GPT-4o evaluation. The paper introduces no physical entities, but it does introduce the conceptual categories VE-Known and VE-Unknown and cognitive misalignment, which are operationalized only through the same CLIP encoder and GPT-4o judge used in the experiments.

free parameters (6)
  • mu_e (entity-aware contrastive loss weight) = 7.32
    Hand-chosen balancing coefficient; robustness is shown but no search procedure is reported.
  • mu_h (hierarchical classification loss weight) = 4.38
    Hand-chosen balancing coefficient; no search procedure is reported.
  • sparsification threshold theta = 0.5
    Filters lower-relevance HR tokens in Eq. 3; robustness tested but choice is ad hoc.
  • temperature tau = learnable, initialized at zero
    Used in the contrastive loss Eq. 4; no fixed value or schedule is specified.
  • data selection cutoffs = 25k, 50k, 125k, 200k
    Subset sizes are chosen for equal-size comparisons; conclusions about data quality versus quantity depend on these cutoffs.
  • number of HR visual tokens NvH = 128
    Architecture choice for the Perceiver resampler; robustness over 100 to 250 is shown.
assumptions (6)
  • ad hoc to paper CLIP cosine similarity between image and landmark name is a valid proxy for VE knowledge and downstream interpretability.
    Eq. (2) defines VE-Known and VE-Unknown; the central empirical claim inherits this assumption.
  • domain assumption GPT-4o-generated Q-A pairs, entities, and hierarchical labels are accurate and consistent.
    Used to build MGLD and to supervise EECA; no human agreement study is reported.
  • domain assumption GPT-4o's four-level evaluation is a faithful measure of recognition ability.
    All accuracy numbers in Tables 1-4 derive from this judge; no human validation or inter-annotator agreement is given.
  • domain assumption Pre-trained CLIP-ViT-L-14 and Llama-3-8B weights are a valid starting point for the study.
    The investigation is conducted within the LLaVA-1.5 framework, so the results are tied to these particular base models.
  • ad hoc to paper Entity text embeddings from the CLIP text encoder align with the LLM's cognitive framework.
    The entity-aware contrastive loss assumes this alignment is what needs to be transferred from text to visual tokens.
  • domain assumption GLDv2 labels and hierarchical categories are ground truth.
    Dataset labels are used without verification; label errors would propagate into MGLD and the reported accuracies.
invented entities (2)
  • Cognitive misalignment (disconnect between VE representations and LLM interpretive framework)
    purpose: Explains failure of LVLMs to recognize landmarks from images despite textual knowledge.
    Introduced as a conceptual framework; operationalized only through CLIP similarity and GPT-4o evaluation, not through an independent measurement.
  • VE-Known and VE-Unknown data categories
    purpose: Classify training images by whether the CLIP vision encoder represents them with high confidence or discrimination.
    Defined by Eq. (2) using the same CLIP encoder used in the LVLM; the categories are not validated against external human judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge." pith.science (2026). https://pith.science/paper/QOBGH2KY

@misc{pith2026241116824,
  author       = {Pith},
  title        = {Pith review of: Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOBGH2KY}},
  note         = {Machine review of arXiv:2411.16824}
}
read the original abstract

Does seeing always mean knowing? Large Vision-Language Models (LVLMs) integrate separately pre-trained vision and language components, often using CLIP-ViT as vision backbone. However, these models frequently encounter a core issue of "cognitive misalignment" between the vision encoder (VE) and the large language model (LLM). Specifically, the VE's representation of visual information may not fully align with LLM's cognitive framework, leading to a mismatch where visual features exceed the language model's interpretive range. To address this, we investigate how variations in VE representations influence LVLM comprehension, especially when the LLM faces VE-Unknown data-images whose ambiguous visual representations challenge the VE's interpretive precision. Accordingly, we construct a multi-granularity landmark dataset and systematically examine the impact of VE-Known and VE-Unknown data on interpretive abilities. Our results show that VE-Unknown data limits LVLM's capacity for accurate understanding, while VE-Known data, rich in distinctive features, helps reduce cognitive misalignment. Building on these insights, we propose Entity-Enhanced Cognitive Alignment (EECA), a method that employs multi-granularity supervision to generate visually enriched, well-aligned tokens that not only integrate within the LLM's embedding space but also align with the LLM's cognitive framework. This alignment markedly enhances LVLM performance in landmark recognition. Our findings underscore the challenges posed by VE-Unknown data and highlight the essential role of cognitive alignment in advancing multimodal systems.

Figures

Figures reproduced from arXiv: 2411.16824 by the authors.

Figure 1
Figure 1. Instances of Cognitive misalignment are systematically identified, even in advanced models like GPT-4o and Qwen2-VL. Al￾though the image closely aligns with the description generated from the text-only prompt, both models fail to recognize the landmark when presented with the image. Text highlighted in green emphasizes details that are particularly relevant to the image. textual labels, poses significant alignment c… view at source ↗
Figure 2
Figure 2. Illustration of the dataset construction process for the Multi-granularity Landmark Dataset (MGLD), showing three stages: best [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. t-SNE visualization of image features. Left: The HDS subset shows more dispersed representations for categories(e.g., “church”). Middle: The HSS subset shows distinct inter-class separations. Right: The LCS subset shows reduced intra-class variability and less distinct inter-class separations. church palace temple bridge tower gate concert museum HDS-25k HSS-25k LCS-25 Counts k [PITH_FULL_IMAGE:figures/full_fig_p00… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Category counts across subsets. Known) in the test dataset, representing the model’s over￾all recognition capability. 3.2. Visual patterns of different knowledge Firstly, we analyze the characteristics of different types of visual knowledge, focusing on how variations …
Figure 5
Figure 5. Figure 5: Comparative performance of HDS and BRS selection [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overview of our model’s framework. EECA approach combines high- and low-resolution visual features through a dual-branch [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The robustness of the different hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: The structure of the GLDv2 train set • train label to hierarchical.csv: Contains fields landmark id, category, supercategory, hierarchical label, and natural or human made. Supercategory refers to the type of landmark (e.g., natural or human-made), mined from Wikimedia…
Figure 9
Figure 9. Figure 9: Frequency of the counts of images per category [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Example for the image selection methodology (the Cas [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Multi-Granularity Data Generation Prompt. The description is the Q-A pair. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Q-A pair Prompt. sponses together, and classify the level of recognition into one of the four levels: Strongly Known, Known, Partially Known, or Unknown. The classification criteria is clearly defined in the prompt (see [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Example of the MGLD datasets. The conversation is [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Evaluation Prompt Known, Known, Partially Known, and Unknown [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Model responses from different recognition level. [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL

    cs.CV 2025-06 conditional novelty 7.0 of 10

    FocusDiff improves autoregressive text-to-image generation by training on paired similar prompts with a modified GRPO objective, achieving state-of-the-art alignment on PairComp and gains on GenEval and T2I-CompBench.

Reference graph

Works this paper leans on

57 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. 1

    2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS,

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  4. [4]

    Improving fine-grained understanding in image- text pre-training

    Ioana Bica, Anastasija Ili ´c, Matthias Bauer, Goker Erdo- gan, Matko Bo ˇsnjak, Christos Kaplanis, Alexey A Grit- senko, Matthias Minderer, Charles Blundell, Razvan Pas- canu, et al. Improving fine-grained understanding in image- text pre-training. arXiv preprint arXiv:2401.09865, 2024. 7

    arXiv 2024

  5. [5]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 8

    arXiv 2023

  6. [6]

    Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning

    Wenliang Dai, Junnan Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning. arxiv 2023. arXiv preprint arXiv:2305.06500, 2,

    arXiv 2023

  7. [7]

    Internlm-xcomposer2- 4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2- 4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512, 2024. 2

    arXiv 2024

  8. [8]

    Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024. 1

    arXiv 2024

Show all 57 references
  1. [9]

    Bard, 2023

    Google. Bard, 2023. 1

    2023

  2. [10]

    Making the V in VQA matter: El- evating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: El- evating the role of image understanding in visual question answering. In CVPR, 2017. 1

    2017

  3. [11]

    GQA: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 1

    2019

  4. [12]

    Brave: Broadening the visual encoding of vision-language models

    O ˘guzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-language models. arXiv preprint arXiv:2404.07204, 2024. 7

    2024 arXiv

  5. [13]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 1

    2014

  6. [14]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 1

    2017

  7. [15]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 1, 2

    2024 arXiv

  8. [16]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023. 1, 8

    2023 arXiv

  9. [17]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  10. [18]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models

    Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024. 2

    2024 arXiv

  11. [19]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 5, 7, 8

    2023 arXiv

  12. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. 2, 8, 1

    2023

  13. [21]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017. 1

    2017

  14. [22]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  15. [23]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 1

    2016

  16. [24]

    OK-VQA: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR, 2019

    2019

  17. [25]

    OCR-VQA: Visual question answer- ing by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual question answer- ing by reading text in images. In ICDAR, 2019. 1

    2019

  18. [26]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1 9

    2023

  19. [27]

    GPT-4o System Card, 2024

    OpenAI. GPT-4o System Card, 2024. 1

    2024

  20. [28]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. 1

    2023 arXiv

  21. [29]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 1, 8

    2021

  22. [30]

    A-OKVQA: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowl- edge. In ECCV, 2022. 1

    2022

  23. [31]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL,

  24. [32]

    When do we not need larger vision models? In European Conference on Computer Vision, pages 444–462

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In European Conference on Computer Vision, pages 444–462. Springer, 2025. 6

    2025

  25. [33]

    Textcaps: a dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, 2020. 1

    2020

  26. [34]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 1

    2023 arXiv

  27. [35]

    When are lemons purple? the concept association bias of vision-language models

    Yingtian Tang, Yutaro Yamada, Yoyo Zhang, and Ilker Yildirim. When are lemons purple? the concept association bias of vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 14333–14348, 2023. 8

    2023

  28. [36]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 1, 8

    2024

  29. [37]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 1, 8

    2024

  30. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  31. [39]

    Vary: Scaling up the vision vocabulary for large vision-language model

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language model. In European Conference on Com- puter Vision, pages 408–424. Springer, 2025. 1, 8

    2025

  32. [40]

    Weyand, A

    T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Land- marks Dataset v2 - A Large-Scale Benchmark for Instance- Level Recognition and Retrieval. In Proc. CVPR, 2020. 3, 1

    2020

  33. [41]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 1, 2

    2024

  34. [42]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 2

    2024 arXiv

  35. [43]

    Sea: Supervised embedding alignment for token-level visual-textual integration in mllms

    Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Di Zhang, Baoqun Yin, and Wentao Zhang. Sea: Supervised embedding alignment for token-level visual-textual integration in mllms. arXiv preprint arXiv:2408.11813, 2024. 1, 7

    2024 arXiv

  36. [44]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1

    2023

  37. [45]

    Eiffel Tower

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 1...

    2024 arXiv

  38. [46]

    The response must be in English

  39. [47]

    Besides identifying the location where the photo was taken, you should also describe the photo and share some knowledge related to the location

  40. [48]

    Do not mention the reference location or this note in your response.'''

  41. [49]

    Where was this photo taken?

  42. [50]

    Identify the location where this photo was taken

  43. [51]

    What is the location shown in the image?

  44. [52]

    Tell me where this photo was taken

  45. [53]

    Where might this photo have been taken?

  46. [54]

    Kinderdijk Windmills“) evaluated by GPT-4o, where the answer across differ- ent models is assessed at four different levels— Strongly 3 {

    What place is depicted in this image? Questions Prompt Figure 12. Q-A pair Prompt. sponses together, and classify the level of recognition into one of the four levels: Strongly Known, Known, Partially Known, or Unknown. The classification criteria is clearly defined in the pro...

    2007

  47. [55]

    These windmills were originally built in the 18th century to manage water levels and prevent flooding in the low-lying polder

    The Kinderdijk Windmills are a group of 19 traditional Dutch windmills located in the Alblasserwaard polder, near the town of Kinderdijk in the Netherlands. These windmills were originally built in the 18th century to manage water levels and prevent flooding in the low-lying p...

  48. [56]

    This traditional Dutch windmill features a classic black and white exterior with a thatched roof and large wooden blades. It stands in a picturesque rural setting, often surrounded by water or fields, and is a symbol of Dutch ingenuity in water management and energy generation...

  49. [57]

    Strongly Known,

    The Zaanse Schans is an open-air museum in the Netherlands, showcasing traditional Dutch windmills. These windmills are iconic structures with large wooden sails, historically used for milling grain, pumping water, or sawing wood. The area is characterized by its picturesque r...

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.