{"id":"9ab2cfa4-2c50-4765-8eb4-0a05e16e35a5","arxiv_id":"2411.16824","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A vision-language model's landmark recognition improves when trained on images its own vision encoder represents confidently, and a new entity-supervised method, EECA, boosts accuracy further.","lead":"This paper studies why vision-language models can name a landmark from text but fail to recognize it from a photo, and proposes a training method plus a landmark dataset to reduce that gap. The work matters because it suggests that training data should be selected for how well the vision system can represent it, not just for volume.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main comparison in Table 1 may be definitional: VE-Known vs VE-Unknown is defined by CLIP similarity using the same encoder that serves as the LVLM's vision backbone, so the result needs an independent difficulty proxy before 'cognitive misalignment' is accepted.","rationale":"Good-faith reading: the paper is an honest empirical study. The quantitative patterns in Tables 1-4 are internally consistent, and the t-SNE visualizations support that the selected subsets differ visually. The limitation paragraph even admits EECA is entity-specific, which is a reasonable scope statement. The stress-test concern is not that the experiments were performed sloppily or dishonestly; it is that the key independent variable is measured with the same instrument whose behavior the paper claims to explain. Suppose CLIP similarity is mainly ranking images by prototypicality or by how well the image matches the landmark name in the CLIP embedding. Then HDS vs LCS is essentially easy vs hard for CLIP, and the LLM is expected to inherit that difference because CLIP is its visual front end. The claim would be much stronger if the partition were confirmed by humans or by a second encoder. This is exactly the reader's weakest assumption, so my read agrees rather than introduces a new objection. The GPT-4o evaluation is a secondary reliability concern, since no human agreement or error bars are reported, and Table 6's absolute counts sum to 2500 despite the text claiming a 5k test split, so the final accuracy numbers should be re-reported with sample size and uncertainty. These secondary issues reinforce a conditional verdict but do not move me to reject, because the paper's method and qualitative t-SNE and ablation evidence support that something real is happening. If the independent-proxy test fails, the conclusion would need substantial revision; if it passes, the current conditional verdict could be upgraded.","tokens_in":17716,"tokens_out":10483,"duration_ms":103474,"concrete_test":"Select a random subset of the MGLD images and obtain an independent measure of landmark recognizability: human annotators rate each image for how identifiable the landmark is and/or how strongly the image matches its landmark name. Re-derive HDS/HSS/LCS splits from those human labels (or, as a weaker alternative, from a different pretrained vision-language encoder such as SigLIP) while keeping the LVLM backbone as CLIP. Retrain the same LLaVA-1.5 baseline on each partition and evaluate on the same reported test set. If the human-defined VE-Known split still outperforms VE-Unknown, the Table 1 finding is robust; if the advantage shrinks, reverses, or fails to replicate, the current result is an artifact of using CLIP to define both the partition and the backbone. Report inter-annotator agreement on the human labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is that the CLIP cosine similarity in Eq. (2), and the derived RSR/Sim scores used to build HDS, HSS, and LCS in Section 3.1, measure the vision encoder's 'knowledge' in a way that is independent enough to support the causal story in Section 3.3. This condition is insecure because the same CLIP-ViT-L-14 model is the vision backbone of the LVLM. HDS and HSS select images whose CLIP embeddings are close to the correct landmark-name text and well separated from distractors; LCS selects images that are, by construction, ones CLIP finds ambiguous or noisy. Showing that models trained on CLIP-easy images outperform models trained on CLIP-hard images is a useful empirical result, but it is close to a restatement of the inclusion criteria: the independent variable (known vs unknown) and the mechanism (the vision encoder) are defined by the same model. The conclusion that VE-Unknown data 'limits' the LVLM's capacity, or that VE-Known data 'reduces cognitive misalignment,' requires some external validation that the partition tracks landmark recognizability rather than CLIP-specific artifacts such as label noise, unusual viewpoints, or images with prominent text. Without that, the central finding is at risk of being definitional rather than explanatory.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates why LVLMs fail to recognize landmarks despite having strong text-only knowledge. It constructs a Multi-granularity Landmark Dataset (MGLD) from GLDv2, partitions images into VE-Known and VE-Unknown categories using CLIP cosine similarity (Eq. 2) and relative similarity rank, and reports that fine-tuning LLaVA-1.5 on VE-Known subsets (HDS and HSS) improves landmark recognition over a random reference (BRS) and over VE-Unknown data (LCS). It then proposes Entity-Enhanced Cognitive Alignment (EECA), which combines a high-resolution branch with an entity-aware contrastive loss (Eq. 4) and a hierarchical classification loss (Eq. 5). The paper reports accuracy rising from an 8.68% baseline to 15.52% with EECA, and includes additional experiments across HDS, HSS, and LCS subsets.","tokens_in":18128,"tokens_out":7702,"duration_ms":65718,"significance":"If the empirical claims hold, the paper offers a practical data-selection heuristic and a training objective for fine-grained landmark recognition. The use of the public GLDv2 dataset, the relatively detailed data-construction pipeline, and the explicit statement of limitations are strengths. However, the central conceptual contribution ('cognitive misalignment') is operationalized through CLIP similarity in a way that is reused by the downstream model and the proposed losses, and the experimental support currently lacks statistical rigor. The paper would be more convincing if the VE-Known/VE-Unknown partition were validated with an independent difficulty measure and if the reported gains were accompanied by significance tests.","major_comments":[{"comment":"The VE-Known/VE-Unknown partition is defined by CLIP cosine similarity between the image and landmark-name text using the same CLIP-ViT-L-14 encoder that serves as the LVLM's vision backbone and that also supplies the entity text embeddings in Eq. (4). Stage 1 of the dataset construction already retains images with high CLIP similarity to the landmark name, so the subsequent HDS/HSS/LCS split is built on the same measure used by the downstream model. Showing that CLIP-easy training images outperform CLIP-hard ones is a useful empirical regularity, but it does not by itself establish that VE-Unknown data 'limits' LVLM capacity or that VE-Known data 'reduces cognitive misalignment'. I request an independent validation of the partition, e.g., human recognition rates or a different vision encoder, or at least controls for image-level confounds such as embedded text, resolution, and viewpoint.","section":"§3.1, Eq. (2)"},{"comment":"The paper states that 5,000 samples are set aside as the test set, but the absolute counts in Tables 6 and 7 sum to 2,500 per condition (e.g., 103+114+145+2138 = 2500). Because all reported percentages and the central improvements in Tables 1-4 are computed on this test set, the discrepancy must be resolved; if the test set is actually 2,500, every percentage should be checked, and if it is 5,000, the tables are incomplete.","section":"Appendix B.4 vs. Appendix D.1"},{"comment":"No standard deviations, confidence intervals, significance tests, or multiple seeds are reported; all results are on a single test split. In the ablation (Table 3), the incremental gains over '+ HSS-50k' are small (+0.52 with Le and +1.12 with Lh), and without variance estimates these differences cannot be distinguished from noise. Please report repeated-seed runs and a paired test, or at least bootstrap intervals, for the key comparisons.","section":"Tables 1-4, Tables 6-7"},{"comment":"GPT-4o is both the annotator that generates the MGLD answers and entities in Stages 2-3 and the judge that classifies the LVLM outputs into Strongly Known/Known/etc. Since EECA is trained on GPT-4o's annotations and then evaluated by GPT-4o's rubric, the reported gains may partly reflect imitation of the annotator's style rather than improved landmark recognition. An independent human evaluation or a second judge on a random subset is necessary to validate the EECA improvements.","section":"§3.1 and §C.1"},{"comment":"The text states that for LCS-25k 'adding the HR branch significantly boosts performance,' but the increase is 10.68 to 12.08 (+1.40) over '+25k Data', whereas Le and Lh add only 12.08 to 12.32 (+0.24). Similarly, the claim that VE-Known data achieve the largest gains with Le/Lh is not uniformly supported: HDS is best with Lh (14.40) and HSS with Le (14.40), while Table 4 shows no consistent ordering among the losses. Please either provide significance tests for these differences or soften the qualitative claims.","section":"§5.3, Table 4"}],"minor_comments":[{"comment":"The name 'LLaVA' is repeatedly typeset as 'LLaV A' (e.g., Section 3.3, §5.1, and Appendix A); please correct.","section":"Throughout"},{"comment":"The sentence 'The category distribution in GLDv2 training dataset is highly imbalanced, as illustrated in ??.' contains an unresolved cross-reference; also Figure 9's caption should be attached to the figure.","section":"Appendix B.1"},{"comment":"The notation X_{e_{i,j}} in Eq. (4) is not defined in place; the text introduces ϕ(e_j) as the entity token embedding, but the subscripted form should be defined explicitly before the loss is presented.","section":"Eq. (4) in §4.2"},{"comment":"The left and right panels of Figure 5 use different x-axes and the two EECA points appear only in the left panel; the caption should state what the right panel's y-axis ('Percentage Increase') is relative to, and why EECA is not shown there.","section":"§5.2, Figure 5"},{"comment":"The t-SNE-based summary that LCS 'shows reduced intra-class variability' should be stated more cautiously, since t-SNE can distort local geometry and does not directly measure class separability.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is probably best framed as a data-selection and training-method paper rather than a general theory of cognitive misalignment. The current title and abstract overclaim relative to the evidence. If the authors can validate the CLIP partition with an independent measure and add statistical rigor, the paper could be a solid contribution. I also note the dataset/test-count inconsistency, which should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine empirical study of how data selection affects landmark recognition in an LVLM, with a new dataset and a clean ablation chain. The catch is that the central distinction—VE-Known vs VE-Unknown—is operationalized with CLIP similarity from the same encoder that later serves as the LVLM's vision backbone. That makes the main result (HDS > HSS > BRS > LCS) partly a restatement of the inclusion criteria. The paper needs external validation before the 'cognitive misalignment' story is accepted.\n\nWhat is actually new: the MGLD dataset construction, the explicit selection strategies, and the finding that a quality-filtered 50k subset beats a 200k reference set. The EECA method itself is mostly assembled from the authors' prior SEA contrastive loss and Mini-Gemini's HR branch, so I would not claim big architectural novelty. But the application to landmark recognition and the careful ablation (Tables 3 and 4) are useful. The robustness sweeps in Figure 7 are a nice touch. To their credit, the authors state in Section 7 that EECA requires labeled data and is not a general solution to misalignment; that is an honest scope limitation.\n\nSoft spots, in proportion: the circularity in the VE-Known/VE-Unknown split is the load-bearing issue. HDS and HSS select images CLIP already scores well, LCS selects images CLIP scores poorly, and then the experiment shows models trained on CLIP-easy images do better. That is a useful engineering rule, but it does not by itself support the claim that VE-Unknown data 'limits' LVLM capacity. You would want an independent difficulty proxy—human agreement, a different vision encoder, or retrieval difficulty—and an external judge. Relatedly, GPT-4o both generates the annotations and judges the outputs; that is a closed loop. There are no confidence intervals or significance tests, and the appendix counts sum to 2,500 while the text says 5k test samples. Minor, but it should be fixed.\n\nWho is this for? Anyone doing LVLM fine-tuning with CLIP backbones and fine-grained recognition. The practical takeaway—prefer images your encoder represents well, and supervise visual tokens with entity-level signal—is plausible and worth testing. It deserves peer review, but the reviewers should push for released data/code, an independent difficulty measure, and human evaluation. I'd accept it for review and ask for a major revision.","headline":"Useful data-curation study for LVLM fine-tuning, but the VE-Known/VE-Unknown split is defined by the same CLIP encoder used downstream, so the headline causal claim is partly definitional; still deserves a serious referee.","tokens_in":18626,"tokens_out":2769,"would_cite":true,"duration_ms":26549,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model recognizes landmarks only when its vision encoder already 'knows' them; supervising visual tokens with entity-level losses substantially raises recognition.","keywords":["large vision-language models","cognitive misalignment","CLIP similarity","landmark recognition","VE-Known data","visual instruction tuning","entity-aware contrastive loss","data quality versus quantity"],"falsifier":"A direct test: on the 5k test set, correlate each image's CLIP similarity to its true landmark name with whether the trained model recognizes it; if the correlation is flat or negative, the claim that VE-Known data drives recognition fails. Stronger still, re-run the HDS/LCS split using a different vision backbone such as DINOv2 and check whether the HDS advantage follows that backbone's confidence; if it does not, the effect is an artifact of the CLIP metric rather than a general property of vision-encoder knowledge.","tokens_in":17551,"feed_emoji":"🗼","tokens_out":11646,"duration_ms":94627,"temperature":0.7,"pith_summary":"Large vision-language models often fail to recognize what they can describe in words; this paper attributes that failure to cognitive misalignment between the CLIP vision encoder and the language model's interpretive space. On a roughly 200k-sample landmark dataset, the authors split training images by CLIP cosine similarity into VE-Known (confident, discriminative) and VE-Unknown (ambiguous) subsets, and find that fine-tuning on VE-Known data improves recognition while VE-Unknown data degrades it. They then propose Entity-Enhanced Cognitive Alignment (EECA), which supervises the adapter's visual tokens with an entity-aware contrastive loss and a hierarchical classification loss, lifting landmark recognition accuracy on a 5k test split from 8.68% to about 15.52% with only 50k training samples. If correct, the finding implies that data quality, measured by the vision encoder's own confidence, can substitute for data volume in LVLM fine-tuning.","feed_headline":"Data quality, not volume, fixes LVLM landmark blindness","feed_subtitle":"Curating training images by CLIP confidence nearly doubles landmark recognition, even with fewer examples.","key_machinery":"The load-bearing object is the CLIP vision encoder's similarity score, $\\mathrm{Sim}_{\\mathrm{CLIP}}(I_i,T_j)=\\langle f_v(I_i),f_t(T_j)\\rangle/(\\|f_v(I_i)\\|\\|f_t(T_j)\\|)$, which splits data into VE-Known (high similarity or high relative rank) and VE-Unknown (low on both) and later supplies the entity embeddings used in supervision. The EECA machinery has three parts: a dual-branch visual encoder that feeds a low-resolution overview and four high-resolution sub-images through a shared CLIP vision encoder, compressing the high-resolution tokens with a Perceiver resampler; an entity-aware contrastive loss that forms entity-grouped visual embeddings as weighted sums of high-resolution tokens and contrasts them against text entity embeddings; and a hierarchical classification loss over broad categories such as 'church' or 'tower'. These are combined with the standard language-modeling loss as $L=\\lambda L_g+\\mu_e L_e+\\mu_h L_h$, so that visual tokens are pushed to retain the entity-specific, discriminative information the LLM needs.","core_discovery":"The paper's central claim is that cognitive misalignment, a mismatch between what CLIP's vision encoder represents and what the LLM can interpret, is a principal cause of LVLM recognition failure, and that both data curation and targeted supervision can reduce it. The authors operationalize the vision encoder's knowledge as CLIP cosine similarity between an image and the landmark-name text embedding, and show that subsets with high discriminative rank (HDS) or high similarity (HSS) outperform random sampling, while low-clarity subsets (LCS) underperform it at every size tested. Building on that, EECA trains a dual-branch high/low-resolution LLaVA-style model with an entity-aware contrastive loss that groups high-resolution visual tokens against text entity embeddings, plus a hierarchical classification loss over landmark categories; with 50k curated samples this raises accuracy from 8.68% to 15.52%, and with 25k it matches the accuracy of a 125k random reference set.","pith_inferences":["The paper's VE-Known/VE-Unknown split and its contrastive supervision both lean on the same CLIP text encoder, so part of the reported gain could come from aligning visual tokens to CLIP's own embedding geometry rather than to the LLM's cognition; training with entity embeddings from an independent text encoder would isolate the mechanism.","Because the selection criterion is cheap (one CLIP forward pass per image), the finding suggests a practical data-curation recipe for other entity-heavy domains such as faces, species, products, and artworks, where EECA-style entity supervision could reduce the required training set size.","The experiments only vary the vision side while keeping the language model fixed; swapping the LLM while holding CLIP fixed would show whether the alignment is truly with the language model's knowledge or with a generic multimodal embedding space."],"forward_implications":["VE-Known selection (HDS) beats random sampling at every training size tested; at 50k it raises accuracy 64.1% relative to the 8.68% baseline, versus 56.2% for random sampling.","Adding VE-Unknown samples actively hurts: LCS-50k gives only a 28.1% relative increase, and performance plateaus as more mixed data is added, so data quality dominates volume beyond a threshold.","Supervising visual tokens with entity-aware contrastive and hierarchical losses adds accuracy on top of curated data (reaching 15.52% at 50k), while adding a high-resolution branch without those losses does not improve the baseline.","EECA's gains are not confined to easy images: it improves VE-Known and VE-Unknown subsets alike, with the high-resolution branch contributing most on low-clarity (VE-Unknown) data and the entity/hierarchical losses contributing most on VE-Known data."],"supporting_citations":[{"why":"Supplies the CLIP vision and text encoders whose cosine similarity defines VE-Known/VE-Unknown and whose embeddings ground the entity-aware contrastive loss.","marker":"[29]"},{"why":"Provides the large landmark benchmark dataset from which the roughly 200k training samples and the 5k test split are drawn.","marker":"[40]"},{"why":"Provides the baseline architecture, initialization, and training recipe that all selection and EECA ablations build on.","marker":"[19]"},{"why":"Contributes the Perceiver resampler used to compress high-resolution visual tokens in the HR branch.","marker":"[2]"},{"why":"Motivates the dual-branch high/low-resolution visual architecture used by EECA.","marker":"[17]"},{"why":"Supplies the supervised embedding alignment idea behind the entity-aware contrastive loss.","marker":"[43]"},{"why":"Generates the Q-A pairs and multi-granularity annotations, and grades the four-level recognition evaluation.","marker":"[27]"}],"fun_headline_variants":["CLIP-curated data sharpens LVLM landmark recognition","Align vision and language to unblur LVLM perception","EECA realigns LVLM cognition with visual tokens","High-similarity data beats volume for LVLM accuracy","Fix cognitive mismatch: curate by CLIP confidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that CLIP's cosine similarity between an image and a landmark name genuinely measures what the vision encoder 'knows'; if that proxy is wrong, the VE-Known/VE-Unknown split, and the entity supervision built from the same CLIP embeddings, would be circular rather than explanatory.","fun_headline_variants_meta":{"raw":{"variants":["CLIP-curated data sharpens LVLM landmark recognition","Align vision and language to unblur LVLM perception","EECA realigns LVLM cognition with visual tokens","High-similarity data beats volume for LVLM accuracy","Fix cognitive mismatch: curate by CLIP confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2744,"prompt_tokens":1004,"completion_tokens":1740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1670}},"tokens_in":620,"tokens_out":1740,"duration_ms":12800,"temperature":1.0,"reasoning_tokens":1670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:53:38.106651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: on the 5k test set, correlate each image's CLIP similarity to its true landmark name with whether the trained model recognizes it; if the correlation is flat or negative, the claim that VE-Known data drives recognition fails. Stronger still, re-run the HDS/LCS split using a different vision backbone such as DINOv2 and check whether the HDS advantage follows that backbone's confidence; if it does not, the effect is an artifact of the CLIP metric rather than a general property of vision-encoder knowledge.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP vision and text encoders whose cosine similarity defines VE-Known/VE-Unknown and whose embeddings ground the entity-aware contrastive loss."},{"cited_title":"Weyand, A","cited_arxiv_id":null,"evidence_quote":"Provides the large landmark benchmark dataset from which the roughly 200k training samples and the 5k test split are drawn."},{"cited_title":"GPT-4o System Card, 2024","cited_arxiv_id":null,"evidence_quote":"Generates the Q-A pairs and multi-granularity annotations, and grades the four-level recognition evaluation."}],"review_version":1}