{"id":"b4e2e63f-1a0d-417a-b3e3-363a7e050922","arxiv_id":"2506.08008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VLMs perform far worse than their own visual encoders on vision-centric tasks because the language model fails to use accessible visual information and instead follows its language priors.","lead":"This paper measures how much visual information open-source vision-language models actually use, by comparing their answers on vision-centric tasks to direct readouts of their own image encoders. It finds the models often fall to near-chance despite their encoders carrying the needed information, and traces the failure to the language model's underuse of visual input.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The visual-readout baselines are not input-matched or decoder-matched to the VLM (privileged keypoint coordinates, a supervised DPT head, different preprocessing); the reported gap may be an evaluation artifact rather than LLM underuse.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption they identify is the same one I see as load-bearing: the direct visual readouts are treated as an upper bound on what the VLM's LLM can access, but those readouts are systematically privileged. In correspondence tasks (App. C.2), the probe gets exact point locations and samples patch features there; the VLM must jointly parse the stitched low-resolution image, the red-circled candidates, and the letter labels. In depth (Sec. 2.1), the baseline uses a DPT head trained for 10 epochs on NYUv2, so the comparison is supervised-probe versus zero-shot VLM. These are not matched decoders. The paper does provide some independent support: the zero-shot probes for 3D-object awareness and art style do not require coordinate supervision, and the component-finetuning experiment (Sec. 4.3) localizes improvement to the LLM, which is consistent with the central claim. But the finetuning comparison is also not fully controlled: LoRA on the largest module (the 7B LLM) may simply have more effective capacity than LoRA on the ViT or projector, even at equal parameter counts. The proposed concrete test—feeding the visual probe the identical stitched, letterboxed input with automatically detected markers—would directly settle whether the gap survives when the probe no longer has privileged access. Until that check is run, the headline claim that VLMs 'overlook' otherwise easily accessible visual information should remain conditional.","tokens_in":17403,"tokens_out":7640,"duration_ms":93484,"concrete_test":"Run the semantic-correspondence visual probe on the exact input the VLM receives: the same 224px letterboxed, stitched image with red circles and A-D labels, with REF/A/B/C/D locations recovered by automatic text detection (not dataset coordinates or external matching) and patch features sampled at those detected locations. If this matched probe's accuracy drops to the VLM's level, the reported gap is an evaluation artifact; if it stays well above, the LLM-bottleneck interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference—that the visual-to-VLM accuracy drop shows the LLM underuses 'easily accessible' visual information—rests on treating the direct visual readouts as a fair upper bound on what the VLM could extract. That bound is not controlled. For correspondence tasks (App. C.2), the visual probe applies an external coordinate-location step for REF/A/B/C/D and samples patch features at exactly those points; the VLM must perform that localization internally in a stitched, 224px, letterboxed image while parsing red circles and text labels. For depth (Sec. 2.1), the visual baseline trains a DPT head on NYUv2 for 10 epochs, whereas the VLM is evaluated zero-shot; a supervised task-specific decoder is not 'easily accessible' to a frozen LLM. The remaining visual baselines also use hand-crafted readouts (Gram matrices, CLS-token similarity) rather than the localized, language-anchored evidence the VLM must integrate. Consequently, the observed gap may reflect differences in resolution, stitching, task-specific decoding, or text grounding rather than the LLM ignoring visual representations. The component-finetuning evidence (Sec. 4.3) is suggestive but also not clean: equal parameter counts do not equal equal optimization capacity, and LoRA on a 7B LLM may improve more simply because that module is larger and differently pretrained.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares a VQA-style evaluation of open-source VLMs to 'direct readouts' of their (identical) frozen vision encoders across six vision-centric tasks: depth estimation, semantic correspondence, object affordance, low-level matching, art style, and 3D object awareness. It reports a universal and often near-chance drop in VLM accuracy relative to the visual readouts, shows that task-relevant visual information remains probeable in projector and LLM layers, finds that prompt prefix tuning gives only marginal gains, and concludes from component-wise LoRA fine-tuning that the LLM is the bottleneck because fine-tuning the LLM improves accuracy the most and also reduces total-variation distance to the ground-truth answer distribution. The paper further documents that VLMs' answer distributions resemble their blind (no-image) baselines, and that the rank ordering of vision encoders changes between visual and VLM evaluations.","tokens_in":17746,"tokens_out":3284,"duration_ms":42095,"significance":"If the central claim survives scrutiny, the paper is significant: it calls into question the practice of ranking vision encoders through VLM-based benchmarks and redirects attention to the visual-linguistic integration stage as the main source of failure on perception-centric tasks. The study has notable strengths: it uses a controlled Prismatic VLM suite in which the vision encoder weights are identical to the standalone backbones, it provides several converging analyses (layer probing, blind baselines, prompt tuning, component fine-tuning), and it builds on external benchmarks rather than self-constructed ones. The attention-map after fine-tuning and the TV-distance analysis are useful diagnostic tools. However, the headline numerical claim depends on treating the direct visual readouts as a fair upper bound on what the VLM could extract from the same representations, and that bound is not yet established due to differences in input presentation, task-specific decoder training, and privileged information in the visual probes.","major_comments":[{"comment":"The depth-estimation visual baseline trains a DPT decoder on NYUv2 for 10 epochs and then compares box-averaged depths, whereas the VLM is evaluated zero-shot with a text prompt. A supervised, task-specific decoder is not 'easily accessible' to a frozen LLM, so the gap in Table 1 (e.g., 88.7% vs. 64.0% for DINOv2) conflates decoder supervision with representation underuse. The paper should either provide a zero-shot or frozen visual readout for depth, or explicitly treat the DPT-trained baseline as an upper bound on task-relevant information rather than as a readout of 'easy access'. Without this control, the depth result cannot support the claim that the information is readily available to the LLM.","section":"Sec. 2.1, Appendix C.1"},{"comment":"The correspondence visual baselines are given the ground-truth image coordinates of the REF, A, B, C, and D points and sample patch features at exactly those floating-point locations. The VLM, in contrast, must internally localize the red circles and text labels in a stitched, letterboxed 224px image while also parsing the multiple-choice letters and reading the REF annotation. This asymmetry injects privileged spatial information into the visual probe: the baseline is not input-matched to the VLM. A fairer comparison would either remove the coordinate-location step from the visual probe (e.g., a patch-similarity readout over the whole image or over detected candidates) or provide the VLM with a task format that does not require text parsing and stitching. As written, the large drops on Semantic Correspondence, Object Affordance, and Low-Level Matching in Fig. 2 may be partially an artifact of the VLM's harder input format.","section":"Sec. 2.2, Appendix C.2"},{"comment":"The central visual-vs-VLM accuracy comparisons are reported as point estimates without error bars, confidence intervals, or significance tests. Several task accuracies for the VLMs are near chance (e.g., 0.225 for DINOv2 semantic correspondence vs. chance 0.25), and the claimed 'universal drop' rests on differences that could in places be within sampling variability. The paper should report bootstrap or per-item confidence intervals over benchmark examples (and, where applicable, repeated prompt variants) for the main comparisons in Table 1 and Figs. 2 and 3. This is load-bearing because near-chance claims and the rank-order shift in Sec. 3.3 are both point estimates.","section":"Sec. 3.2, Fig. 2, Table 1"},{"comment":"The conclusion that the LLM is the bottleneck relies on comparing LoRA fine-tuning of the ViT, projector, and LLM with 'equal parameter counts' (16.7M). Equal parameter count does not imply equal optimization capacity: LoRA on a 7B LLM and LoRA on a ViT-L differ in the matrices being adapted, in the effective rank used by each module, and in the pretraining objectives of those modules. The larger improvement from LLM tuning could stem from the LLM having more learnable capacity to absorb the task-specific mapping, rather than from the original LLM being a uniquely underutilizing component. Additionally, task-conditional fine-tuning on the same datasets used for evaluation demonstrates an upper bound on what that component can learn, not what the frozen VLM should already be doing. A control that varies training steps, total FLOPs, or the amount of visual information provided to the LLM would strengthen the bottleneck attribution.","section":"Sec. 4.3, Fig. 7"},{"comment":"The blind-baseline analysis is suggestive but is used to claim that VLMs 'ignore images'. The total-variation distances in Table 2 show that the original VLM's answer distribution is close to its blind counterpart (e.g., 0.337 vs 0.399 for semantic correspondence), yet these are aggregate distributions over all items; a model that attends to the image but has a strong letter-answer prior could produce a similar aggregate distribution. Reporting per-item agreement between the with-image and no-image predictions, not just marginal distributions, would directly support the 'ignore images' interpretation. As it stands, the evidence indicates inherited answer biases, but the stronger claim that visual input is not used requires a per-item measure.","section":"Sec. 3.4, Fig. 4"}],"minor_comments":[{"comment":"The text refers to 'Fig. 2 and Tab. 3' when reporting visual evaluation results, but the relevant table is numbered Table 1. Please correct the cross-reference.","section":"Sec. 3.1, Table 1 reference"},{"comment":"The text refers to 'Table 4.3.1' when discussing TV distances, but the table is numbered Table 2. Please fix the in-text citations.","section":"Sec. 3.4 and Sec. 4.3.1, Table 2"},{"comment":"Typo: 'differnet' should read 'different' in the MOCHI task description.","section":"Sec. 2.3"},{"comment":"Typo: 'misuing' should read 'misusing' in the caption: 'VLMs are not simply misuing their visual representations'.","section":"Fig. 4 caption"},{"comment":"Typo: 'there there are still improvements needed' should read 'there are still improvements needed'.","section":"Discussion"},{"comment":"The few-shot results report 'mean and standard deviation (which is zero)' for the vision model; the source of the zero standard deviation should be stated explicitly, otherwise readers may suspect a reporting error.","section":"Appendix C.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of a vision and language modeling venue and the question is timely. My main concern is that the headline claim is stated more strongly than the evidence supports: the visual-readout baselines are not input-matched or decoder-matched to the VLM, and the main comparisons lack uncertainty quantification. These are fixable in revision, but they are load-bearing. I would also gently note that the paper builds heavily on existing benchmarks (CV-Bench, BLINK, MOCHI, Banani et al.) and should make the novel diagnostic contribution—controlled comparisons across identical vision backbones, layer-wise probing, and the component fine-tuning evidence—very explicit in the framing. No code or data release is mentioned; for a paper of this type, releasing the evaluation harness would materially increase the impact and verifiability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's core observation—open-source VLMs drop to near chance on vision-centric tasks while their own frozen vision encoders, read out directly, do far better—is real and worth taking seriously. The decomposition is the novel part: probing patch/CLS features through the projector and LLM layers shows task-relevant signal survives, prompt prefixes don't close the gap, blind answer distributions predict VLM choices, and LoRA-tuning the LLM beats tuning the projector or ViT. That is a useful package, and it should change how people interpret VLM benchmark rankings.\n\nBut the central comparison is not as controlled as the paper implies. The 'visual evaluation' baselines are privileged in ways that could explain part of the gap. Correspondence probes get the REF/A/B/C/D coordinates for free and sample patch features at exactly those points; the VLM must locate the dots, parse the red circles and labels, and integrate the stitched low-resolution input. The depth baseline trains a supervised DPT head on NYUv2; the VLM is zero-shot. A supervised task-specific decoder is not 'easily accessible' to a frozen LLM, so the drop on depth could partly come from the head, not from LLM inattention. The Gram-matrix and CLS-similarity readouts are also hand-crafted. None of this proves the central claim wrong, but it does mean the 'universal drop' is a mixed phenomenon.\n\nThe rest of the evidence is suggestive, not conclusive. The intermediate-layer probing uses the same privileged readouts, so it shows the information is present and linearly extractable—not that the LLM can access it through language-anchored attention. The component finetuning is also not clean: equal parameter counts don't equal equal optimization capacity, and a 7B LLM is a much bigger, differently pretrained module than the projector. The exception on 3D awareness is handled honestly.\n\nWhere the paper is strongest is the blind-baseline analysis and the rank-ordering reversal. Those results stand regardless of the probe-quality critique. The paper is aimed at people building or benchmarking VLMs, and it deserves a serious referee. I'd send it out, but ask the authors to add input-matched baselines (same resolution, no privileged coordinates, a linear probe instead of a trained DPT head) and to report variances over seeds or task subsets. With that, the claim about the LLM bottleneck would be much harder to dismiss.\n\nRecommendation: send to peer review, conditional on revision.","headline":"A useful diagnostic with a plausible but not fully controlled central comparison; the privileged visual readouts and supervised depth head soften the LLM-bottleneck claim, but the paper deserves serious review.","tokens_in":18196,"tokens_out":2270,"would_cite":true,"duration_ms":27468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source VLMs perform near chance on vision tasks that their own encoders solve, and the paper identifies the language model as the bottleneck.","keywords":["vision-language models","visual representations","vision-centric benchmarks","language priors","LLM bottleneck","visual probing","VQA evaluation","DINOv2"],"falsifier":"Run the six tasks on a VLM with native multi-image input and full resolution, avoiding letterbox stitching and 224-pixel resizing, and compare against the same direct visual readouts: if the gap mostly closes, input formatting rather than LLM underuse is the cause. Alternatively, train the same readout probes on the actual token sequence the LLM receives, after the projector and input embedding layers; if those probes also fail, the information is not easily accessible to any decoder and the bottleneck is earlier than the LLM.","tokens_in":17240,"feed_emoji":"👁️","tokens_out":8768,"duration_ms":92657,"temperature":0.7,"pith_summary":"This paper tries to establish why open-source vision-language models fail at purely visual tasks: they are not using the visual information their own encoders already contain. Across six vision-centric benchmarks, querying the full VLM drops performance from near-ceiling to near chance, while direct readouts of the same encoder features solve the tasks. The authors rule out representation degradation and prompt sensitivity, and pin the failure on the language model, which under-attends to relevant image regions and leans on answer biases inherited from text-only pretraining. If the claim is right, VLM benchmark scores badly underestimate the visual abilities of their encoders, and improving visual tasks means improving how the LLM consumes visual features.","feed_headline":"VLMs drop to chance on vision tasks their encoders ace","feed_subtitle":"Same image features that support near-ceiling scores barely move VQA answers; direct readouts show the LLM is wasting them.","key_machinery":"The central device is the paired evaluation: a direct visual readout of a VLM's frozen vision encoder versus a VQA-style query of the full VLM on the same task. The readout methods are cosine similarity between patch features at manually located keypoints for correspondence tasks, Gram-matrix second-moment matching of patch features for art style, pairwise CLS-token cosine similarity for 3D object awareness, and a DPT dense-prediction head trained on NYUv2 for depth. To separate candidate bottlenecks, the authors probe task accuracy at intermediate projector and LLM layers, tune learnable prompt prefixes, and fine-tune each component with matched parameter counts. These probes establish that the information is present throughout the model, that prompt wording is not the failure, and that only the LLM's use of the visual tokens moves performance.","core_discovery":"The paper's central claim is that open-source VLMs systematically discard visual information that their own vision encoders make readily available. On six vision-centric tasks—depth ordering, semantic correspondence, low-level matching, object affordance, art style, and 3D object awareness—the same image features that support near-ceiling accuracy when read out directly support only near-chance accuracy when the VLM answers in text. The loss is not explained by the vision encoder degrading inside the VLM, because probing intermediate projector and LLM layers still recovers task-relevant signal; nor by prompt sensitivity, because learnable prompt prefixes yield only marginal, diminishing gains. Equal-parameter fine-tuning shows the bottleneck is the language model: tuning the LLM improves accuracy and moves answer distributions toward ground truth, while tuning the projector or ViT does little. The paper concludes that VLMs inherit the answer biases of their language backbones and under-attend to the visual regions that carry the task.","pith_inferences":["Editorial extension: the blind-baseline comparison suggests a cheap diagnostic—run the same prompt with a blank image; high overlap between the two answer distributions flags tasks where the model is not looking at the image.","Editorial extension: if the LLM is the bottleneck, holding the LLM fixed and varying only the encoder should barely move vision-centric accuracy, a direct test of the paper's attribution.","Editorial extension: the paired protocol could be reused to evaluate future methods; a method that raises VQA accuracy while the direct-readout ceiling stays fixed is genuinely improving visual-information use.","Editorial extension: the visual readouts are one access path, so the claim is best read as 'easily accessible information under these probes,' not a proof that no LLM could ever extract more from the same features."],"forward_implications":["VLM benchmark rankings should not be read as rankings of vision encoders' visual abilities; a strong encoder can score best on direct readout yet worst inside a VLM.","Improving vision-centric VLM performance will require changing how the LLM consumes visual tokens, for example by increasing attention to salient regions, rather than only swapping or ensembling vision backbones.","Task-level fine-tuning of the LLM reduces the influence of the language prior and brings multiple-choice answer distributions closer to ground truth, while fine-tuning the vision encoder or projector does not.","Prompt engineering with learnable prefixes is not a cure: one or a few prefixes give only small gains, and additional prefixes do not keep improving accuracy.","Because representations stay informative through the projector and LLM layers, failure is a read-out problem rather than a storage problem: the final answer can be wrong even when the model still contains the right answer."],"supporting_citations":[{"why":"Supplies the depth-ordering task and the earlier VLM-based encoder ranking that the paper re-examines and contradicts.","marker":"Tong et al. (2024a)"},{"why":"Supplies the correspondence, affordance, and art-style benchmarks and the VQA prompts used to evaluate VLMs.","marker":"Fu et al. (2024)"},{"why":"Supplies the 3D object awareness benchmark and the CLS-token cosine-similarity readout used for the visual evaluation.","marker":"Bonnen et al. (2024)"},{"why":"Supplies the DPT-probing protocol and the evidence that vision encoders solve spatial tasks when read out directly.","marker":"Banani et al. (2024)"},{"why":"Supplies the VLM suite with frozen, matched vision backbones that makes the controlled visual-versus-VLM comparison possible.","marker":"Karamcheti et al. (2024)"},{"why":"Supplies the self-supervised visual encoder that dominates direct readouts but not VLM rankings.","marker":"Oquab et al. (2024)"},{"why":"Supplies the prompt-prefix tuning method used to test whether prompt sensitivity explains the performance gap.","marker":"Lester et al. (2021)"},{"why":"Supplies the DPT dense-prediction head used as the depth readout for the visual evaluation.","marker":"Ranftl et al. (2021)"}],"fun_headline_variants":["VLMs flop on vision tasks their encoders ace","LLM priors make VLMs miss what encoders see","VLMs squander visual info their encoders nail","Vision encoders spot it, but VLMs nearly guess"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the hand-built visual readouts being a fair upper bound on what the language model could extract from the same encoder features; if stitching images, lower resolution, or reading red-circled labels makes the language-model path harder in ways the probes skip, the gap would not prove the language model is the bottleneck.","fun_headline_variants_meta":{"raw":{"variants":["VLMs flop on vision tasks their encoders ace","LLM priors make VLMs miss what encoders see","VLMs squander visual info their encoders nail","Vision encoders spot it, but VLMs nearly guess"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3247,"prompt_tokens":928,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2252}},"tokens_in":544,"tokens_out":2319,"duration_ms":16689,"temperature":1.0,"reasoning_tokens":2252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:20:06.912627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the six tasks on a VLM with native multi-image input and full resolution, avoiding letterbox stitching and 224-pixel resizing, and compare against the same direct visual readouts: if the gap mostly closes, input formatting rather than LLM underuse is the cause. Alternatively, train the same readout probes on the actual token sequence the LLM receives, after the projector and input embedding layers; if those probes also fail, the information is not easily accessible to any decoder and the bottleneck is earlier than the LLM.","supporting_citations":[],"review_version":1}