{"id":"ca5d8918-77b6-4cf8-9501-70c55057d513","arxiv_id":"2504.16727","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"V2R-Bench shows that 21 large vision-language models are markedly less accurate on simple object and direction tasks when object position, scale, orientation, or context is varied, and attributes the failure to multimodal alignment and pipeline architecture.","lead":"This paper introduces V2R-Bench, a benchmark that automatically varies object position, scale, orientation, and background in images to test how robust large vision-language models are to these changes. It finds that even strong models like GPT-4o fail at simple recognition tasks when the image is varied, and traces the failures to how visual features are aligned with the language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 accuracy is computed with an unspecified scoring rule; if exact-match on free-form outputs, the near-chance numbers may be a format artifact, not a perception failure.","rationale":"The paper proposes a useful benchmark and provides a plausible qualitative finding that LVLMs are not robust to basic visual variations. However, the paper's central quantitative claim—that even advanced models significantly underperform on simple object recognition—depends entirely on an accuracy metric whose scoring rule is never specified. The reader's weakest_assumption identifies exactly this gap, and I agree it is the most load-bearing concern because the same accuracy numbers are used across Table 1 and the subsequent position/scale/orientation analyses. If the accuracy column is exact-match on free-form outputs, the near-chance object recognition scores could be an artifact of how models phrase their answers rather than a failure of perception; the high GPT-4o number relative to open models could then reflect instruction-following format rather than visual robustness. This does not make me want to reject the paper outright, because the qualitative finding and benchmark resource are still valuable, and the unspecified scoring is fixable in revision. I would keep the reader's CONDITIONAL verdict: the authors need to specify the evaluation protocol, provide raw outputs or scoring code, add human baselines and error bars, and repair the duplicated values in Table 1. The separate claim that the limitations are 'fundamentally architectural deficiencies' is also overstated relative to the Section 6 experiment (fine-tuning on a subset of the benchmark is not a valid upper bound), but that is secondary to the vulnerability claim that the paper leads with.","tokens_in":28267,"tokens_out":4127,"duration_ms":43679,"concrete_test":"Using the released code and data, take 100 object-task responses per model (e.g., GPT-4o, Qwen2-VL, LLaVA-1.6) and rescore them with normalized containment matching: lowercase the response, strip leading articles and punctuation, and check whether the ground-truth class name appears as a token. Compare this against the published Table 1 accuracy and against exact-match scoring. If GPT-4o moves from 31.5% to above 80% under normalized matching, the published near-chance numbers are a formatting artifact and the central vulnerability claim needs qualification. If all three scoring protocols agree near the published values, the scoring concern is closed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that advanced LVLMs 'significantly underperform on simple tasks such as object recognition' rests on the accuracy column of Table 1. Section 3.3 defines consistency and stability metrics but never states how the task accuracy M(I_v) is scored: exact string match, normalized containment, or LLM-as-a-judge. The object prompt in Appendix B is free-form ('Identify the object in the image'), so exact-match scoring would mark correct semantic answers such as 'It is a cat' as wrong. This is consistent with the suspicious distribution of object accuracies: many models sit at 1.3–9% on a 10-class task (at or below chance), while GPT-4o is 31.5%; if outputs are free-form and scored exactly, these numbers reflect formatting robustness, not visual perception. The same accuracy numbers feed the position, scale, and orientation analyses, so the scoring rule is load-bearing for the benchmark's main quantitative finding. No human baseline or error bars are provided, so it is also unclear whether the reported variation effects are large relative to human variation. If scoring is actually normalized or judge-based, this concern would not land; but the paper must state the protocol explicitly and release the scoring code and raw outputs for verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces V2R-Bench, an automated framework for evaluating LVLM robustness to four visual variations (position, scale, orientation, and context), and applies it to 21 LVLMs on basic object and direction tasks plus extended benchmarks. The authors report that even strong LVLMs perform poorly under these variations, exhibit a peripheral position bias that contradicts effective-receptive-field intuition, show a scale threshold reminiscent of human acuity, and trace the failures to information loss in the multimodal projector and to the cascading pipeline architecture, concluding that the limitations are fundamentally architectural. The paper also proposes a token-decoding visualization of aligned visual features and supports the architectural claim with text-based LLM experiments and a fine-tuning experiment on a partial benchmark subset.","tokens_in":28488,"tokens_out":4554,"duration_ms":40698,"significance":"If the main claims hold, V2R-Bench fills a real gap in multimodal evaluation and the findings would be practically important for deployment of LVLMs in robotics, medical imaging, and other domains where object position, scale, and orientation vary naturally. The paper's strengths include broad model coverage, a public code/data release, the use of controlled synthetic tasks to isolate basic visual capabilities, and a component-level diagnostic strategy that combines linear probing, t-SNE, and text-based LLM analogs. These are valuable contributions. However, the quantitative backbone currently has unresolved issues—an unspecified accuracy scoring protocol, suspicious duplicated numbers in Table 1, and an 'upper bound' that is not a true upper bound—so the central quantitative claims are not yet established to the standard required for acceptance.","major_comments":[{"comment":"The task-specific accuracy M(I_v) used in Eq. (2) and reported in Table 1 is never defined. The paper does not state whether responses are scored by exact string match, normalized containment, LLM-as-a-judge, or another rule. This matters because the object prompt in Appendix B is free-form ('Identify the object in the image'), so exact-match scoring would mark semantically correct answers such as 'It is a cat' as incorrect. The low object accuracies in Table 1, several near or below chance on a 10-class task, are consistent with a format-mismatch artifact rather than a visual perception failure. Since the same accuracy numbers feed the position, scale, and orientation analyses, the scoring protocol is load-bearing for the paper's central claim that advanced LVLMs significantly underperform on simple object recognition. The authors must specify the scoring rule precisely and release the model outputs and scoring code for verification.","section":"Section 3.3, Eq. (2); Table 1; Appendix B"},{"comment":"Table 1 contains duplicated robustness values across different models. For example, Molmo-7B-D and Qwen2-VL-7B share identical robustness values for orientation (95.5/66.4), scale (86.2/92.3), and context (93.7/89.0), and the direction robustness components are also identical. This pattern is unlikely to arise from genuine model behavior and suggests a copying or processing error. Because these numbers are central to the paper's comparative robustness claims, all entries in Table 1 need to be audited, regenerated, or individually verified, and any duplication must be explained.","section":"Table 1"},{"comment":"The 'upper bound' in Figure 6 is defined as the test result of an LVLM fine-tuned on a partial benchmark dataset. This is not an upper bound on achievable performance; it is one particular model trained with one particular data recipe. It cannot support the conclusion that the observed vulnerabilities are 'fundamentally architectural deficiencies' rather than training-data limitations, because another training strategy, more data, or a different architecture could exceed this value. To support the architectural claim, the paper needs either a principled upper bound (for example, a ceiling implied by information preservation through the components) or a substantially weakened conclusion that explicitly acknowledges the bound is a performance reference, not a theoretical ceiling.","section":"Section 6, Figure 6"},{"comment":"The position-bias claim relies on Table 2, but no error bars, confidence intervals, or statistical tests are provided, and some differences are very small (e.g., LLaVA1.6 object accuracy differs by +0.01 between Middle and Surrounding). The statement that models 'consistently demonstrate better perception of surrounding regions' is not supported without uncertainty quantification. Similarly, the abstract and Section 5.1 describe a 'human-like visual acuity threshold' on the basis of Figure 2(c), but no human baseline is measured or cited. To substantiate the human-like claim, the authors should include human evaluations on the same variation grid or remove the 'human-like' language.","section":"Table 2; Section 5.1, Figure 2(c)"},{"comment":"The overall robustness score is said to be a 'weighted aggregation' of consistency, semantic stability, and token stability, but the weights are never specified. Without the weights, the aggregated robustness numbers in Table 1 are not reproducible or interpretable. The authors should state the exact aggregation formula and weights, or report the three component metrics separately for every model.","section":"Section 3.3, paragraph 4"}],"minor_comments":[{"comment":"The text contains a duplicated word: 'automatically automatically generating diverse visual variations'; this should be corrected.","section":"Section 3.1"},{"comment":"The phrase 'scoring the need for architectural innovations' appears to be a typo; 'scoring' should likely be 'signaling' or 'underscoring'.","section":"Abstract"},{"comment":"The object dataset is described as a '90-class object classification task' in Section 3.2, while Appendix C.3 says the dataset 'comprises ten carefully selected object categories' and that the 90-class setting is used only for linear probing. This inconsistency should be clarified.","section":"Section 3.2 and Appendix C.3"},{"comment":"The text mentions that 'models like Fuyu and BLIP' show a left-orientation bias, but Fuyu and BLIP do not appear in Table 1 or in the enumerated evaluated-model list in Appendix D.1; please specify where those results are reported or add the missing model entries.","section":"Section 5.1, second paragraph"},{"comment":"The value of top-k in the token decoding step is not specified. Since k controls how many decoded tokens are examined, reporting the value or a sensitivity analysis would make the visualization more reproducible.","section":"Section 4.2, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major-revision case. The benchmark idea and qualitative diagnostics are genuinely useful, and I would be willing to re-review a revised version. The load-bearing problems are: (1) the missing accuracy scoring protocol, which could invalidate the headline object-recognition numbers; (2) the duplicated entries in Table 1, which suggest a data-processing error; and (3) the 'upper bound' in Figure 6, which is not an upper bound. If the authors provide scoring code, raw outputs, corrected tables, and a reworded architectural-conclusion section, the paper could be acceptable. I would not reject at this stage because the qualitative examples and component-level analyses give independent support to the fragility claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: the paper offers a genuinely useful evaluation framework and a plausible qualitative finding, but the quantitative evidence is not yet trustworthy enough to support the strong architectural conclusions. The benchmark (V2R-Bench) that automatically applies position, scale, orientation, and context variations, and the component-level analysis with token-decoding of aligned visual features, are real contributions. Evaluating 21 LVLMs in a controlled way is labor, and the synthetic-data experiment is a reasonable way to probe architecture-versus-data questions.\n\nThe soft spots are serious. The task accuracy M(I_v) is never defined. For object recognition, the prompt is 'Identify the object in the image', free-form. If scoring is exact string match, low numbers like 1–9% on a 10-class task would reflect formatting failure, not perception failure. The paper must specify the scoring protocol and release raw outputs and scoring code. Several rows in Table 1 share identical or near-identical robustness values across different models—that looks like a copy-paste error, not a measurement. Table 2's position-bias differences are a few points, sometimes 0.01, with no error bars; the claim that models 'consistently demonstrate better perception of surrounding regions' is not supported at that resolution. There is no human baseline, so we don't know how much of the fragility is actually unique to LVLMs. And the 'upper bound' in Figure 6 is just the test result of a fine-tuned model—labeling it an upper bound is misleading.\n\nThe paper also has a minor internal inconsistency: the object task is described as 10-class in the appendix and 90-class in the main text. And it should cite MMVP, which already explored similar object-recognition fragility.\n\nThe central qualitative claim—that LVLMs are fragile to natural visual variations—is probably true and consistent with prior hints. The direction and scale trends look more robust. But the magnitude and the attribution to architecture are not yet established.\n\nFor whom? Anybody building or evaluating LVLMs should be aware of this benchmark; the framework is reusable. The paper deserves a serious referee, but not at the current level. I'd want a major revision: specify scoring, fix the table, add baselines and error bars, and temper the abstract's 'fundamentally architectural' claim.\n\nMy recommendation: engage with it as a resource, treat the numbers with caution until the protocol is clarified.","headline":"A useful benchmark with a likely-true fragility finding, but the quantitative evidence needs a serious cleanup before the architectural conclusions can be taken seriously.","tokens_in":29013,"tokens_out":5182,"would_cite":false,"duration_ms":39169,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modern vision-language models, including GPT-4o, can drop to near-chance accuracy on simple object recognition when an object's position, scale, orientation, or context changes, and the paper argues this is a structural architectural…","keywords":["visual variation robustness","large vision-language models","V2R-Bench","multimodal alignment","position bias","visual acuity threshold","architectural deficiency","object recognition benchmark"],"falsifier":"Re-score the V2R-Bench object and direction outputs of GPT-4o and several open models using human annotation or a forced-choice format; if accuracy jumps from roughly 30% to above 90%, the claimed visual-perception vulnerability is mostly an answer-scoring artifact. A second check: run the same position-bias protocol with human participants; if humans show central superiority, the claimed human-like acuity threshold and edge-favoring bias need reinterpretation.","tokens_in":28048,"feed_emoji":"👁️","tokens_out":4882,"duration_ms":48763,"temperature":0.7,"pith_summary":"The paper introduces V2R-Bench, a benchmark of 428K images that systematically applies position, scale, orientation, and context variations to visual tasks and evaluates 21 large vision-language models. It claims that even models that excel at complex vision-language tasks fail at basic object recognition and direction detection once these everyday visual variations are introduced. The paper argues the failures are not a data shortage but an architectural deficiency: errors accumulate across the vision-encoder-to-projector-to-language-model pipeline, and the multimodal alignment stage loses visual information. A sympathetic reader would care because real-world deployment constantly exposes models to exactly these variations, so the finding shifts attention from task-level accuracy to structural robustness.","feed_headline":"GPT-4o scores 31.5% on object recognition under position shifts","feed_subtitle":"A 428K-image stress test across 21 models shows the failure is architectural, not a data shortage.","key_machinery":"The central object is V2R-Bench, an automated evaluation framework that generates variants $\\mathcal{D} = \\{T(I,v) \\mid v \\in P \\times S \\times R \\times C\\}$ across position, scale, rotation, and context, and scores robustness through a consistency metric $C_m(I)$, semantic and token-level stability measures $S_s(I)$ and $S_t(I)$, and LLM-as-a-judge ratings. The diagnostic machinery is component-level analysis: linear probing of pre- and post-projection features to measure visual information loss, and a novel visualization that decodes an aligned visual feature into language tokens via $t = \\mathrm{topk}(\\mathrm{softmax}(h E^\\top))$, revealing what semantics the language model receives. This machinery carries the argument by turning vague robustness concerns into measurable, localizable failures.","core_discovery":"The paper claims that modern LVLMs lack robustness to fundamental visual variations, with object recognition accuracy collapsing under position shifts (e.g., GPT-4o at 31.5% on the object task) despite strong performance on mainstream benchmarks. It reports a counter-intuitive position bias in which peripheral image regions are perceived better than the center, contradicting effective receptive field theory; a scale threshold near 1/100 of image area below which accuracy flattens at minimum; orientation-dependent failures; and a tendency to ground answers in contextual inference rather than direct visual evidence. Component analysis attributes the vulnerability primarily to inadequate multimodal alignment: linear probing accuracy drops sharply across the projector, aligned visual features sit far from language embeddings, and their decoded semantics shift under variations. The paper further shows that training on more varied data improves task performance but not variation robustness, which it reads as evidence that the limitations are fundamentally architectural rather than data-driven.","pith_inferences":["The accuracy-scoring procedure is unspecified, so some of the near-chance numbers could reflect exact-string mismatch on free-form outputs rather than perception failure; a multiple-choice or human-judged re-scoring of the same outputs would separate these explanations, and the position-bias pattern may survive even if absolute accuracy rises.","A direct human psychophysics comparison on the same position, scale, and orientation grids would test whether the 'human-like acuity threshold' is truly human-like or merely a model-specific artifact.","The ethics statement implies a no-training attack vector: natural-looking object placement or rotation could manipulate model outputs in the wild, a claim that can be tested adversarially by optimizing object positions to flip predictions.","The soft-prompt interpretation of aligned visual features suggests a testable extension: anchoring visual features to discrete semantic tokens, rather than continuous soft prompts, should improve robustness if the paper's diagnosis is correct."],"forward_implications":["If the claims are right, deployed LVLMs should be stress-tested with V2R-Bench-style variations before use in settings where object position, scale, or orientation is uncontrolled.","The scale threshold implies that fine-grained visual tasks such as small-object detection in medical imaging or remote sensing will be unreliable for current models below roughly 1/100 of the object's image area.","The position-bias finding means current vision encoders and LVLMs process image centers worse than edges, which is the opposite of classic convolutional receptive-field expectations and argues for rethinking patch tokenization and positional encoding.","The paper's synthetic-training result implies that simply adding more varied training data will not fix these failures, so progress depends on architectural changes such as unified multimodal models or stronger alignment mechanisms.","The component analysis suggests that the language model's own spatial reasoning, tested through text-matrix inputs, is also limited, meaning downstream fixes alone cannot fully solve the problem."],"supporting_citations":[{"why":"Supplies the CLIP vision encoder used by most evaluated models and frames the contrastive pretraining paradigm that the component analysis assesses.","marker":"Radford et al., 2021"},{"why":"Supplies the self-supervised DINOv2 encoder used as the alternative vision backbone in the component-level comparison.","marker":"Oquab et al., 2024"},{"why":"Provides the linear probing method used to measure information retention before and after the multimodal projector.","marker":"Alain and Bengio, 2018"},{"why":"Defines the effective receptive field theory whose central-peripheral prediction the paper's position-bias result directly contradicts.","marker":"Luo et al., 2017"},{"why":"Provides the vision-transformer processing analysis that the paper invokes when interpreting the position-bias finding.","marker":"Raghu et al., 2022"},{"why":"Supplies the LLM-as-a-judge approach used as one of the robustness metrics in V2R-Bench.","marker":"Zheng et al., 2023"},{"why":"Supplies the 'soft prompting might be a bug' result used to interpret aligned visual features as unstable soft prompts and explain their semantic fragility.","marker":"Bailey et al., 2024"},{"why":"Supplies one of the inpainting diffusion methods used to generate context variations in the automated data pipeline.","marker":"Corneanu et al., 2024"},{"why":"Supplies the RePaint inpainting method used alongside Corneanu et al. for context-variation image generation.","marker":"Lugmayr et al., 2022"},{"why":"Provides LLaVA, the open model family used as the primary case study for component-level diagnosis and fine-tuning experiments.","marker":"Liu et al., 2023"}],"fun_headline_variants":["GPT-4o scores 31.5% on object recognition under position shifts","Architectural flaw, not data scarcity, breaks LVLM visual robustness","New benchmark: LVLMs fail simple visual tasks when objects scale or shift","LVLM position bias contradicts receptive field theory, say researchers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the task-accuracy scores behind every headline table number are measured by a fair, valid procedure; if free-form model outputs are being scored by exact string match rather than by semantic or multiple-choice evaluation, the near-chance accuracies could indicate format mismatch rather than perception failure, and no human baseline is given to anchor the numbers.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o scores 31.5% on object recognition under position shifts","Architectural flaw, not data scarcity, breaks LVLM visual robustness","New benchmark: LVLMs fail simple visual tasks when objects scale or shift","LVLM position bias contradicts receptive field theory, say researchers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2308,"prompt_tokens":949,"completion_tokens":1359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1280}},"tokens_in":565,"tokens_out":1359,"duration_ms":12493,"temperature":1.0,"reasoning_tokens":1280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:56:59.926914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the V2R-Bench object and direction outputs of GPT-4o and several open models using human annotation or a forced-choice format; if accuracy jumps from roughly 30% to above 90%, the claimed visual-perception vulnerability is mostly an answer-scoring artifact. A second check: run the same position-bias protocol with human participants; if humans show central superiority, the claimed human-like acuity threshold and edge-favoring bias need reinterpretation.","supporting_citations":[],"review_version":1}