{"id":"fe3c5f1a-a16c-4cef-9b1c-d4fb6485ce75","arxiv_id":"2508.13428","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In six VQA benchmarks, models can often choose the correct option from image plus options alone, and the GroundAttack toolkit generates visually plausible hard negatives to remove this shortcut.","lead":"This paper finds that many multiple-choice visual question answering tests let models pick the correct answer without even reading the question. It introduces a tool that generates harder wrong answers to make these tests more honest.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GroundAttack removes only CLIP-similarity-based shortcut; 'EOB-free' claim needs mechanism-independent validation.","rationale":"The abstract's strongest claim is about benchmark bias, and the direct V+O accuracies (51.57% vs 61.11%) substantiate that some shortcut exists; I do not dispute that. The load-bearing concern is the step from 'CLIP similarity is correlated with the shortcut' to 'balancing it makes the dataset EOB-free.' The paper's own numbers show CLIP accounts for 42.1/51.57 ≈ 82% of the observed V+O accuracy, leaving a non-trivial residual that could come from other options. The mitigation is measured on the same V+O metric used to define EOB, so the improvement could be overfit to the operationalization. A concrete independent-similarity and input-ablation test would distinguish a genuinely de-biased benchmark from one de-biased only along the dimension used to construct it. Since the full experimental section is not in the reviewed text, CONDITIONAL remains the right verdict, and no evidence here warrants rejecting the work.","tokens_in":2037,"tokens_out":3428,"duration_ms":37907,"concrete_test":"On the released GroundAttack annotations, run three controls: (1) V+O with image replaced by noise/blank; (2) O-only without image; (3) regenerate hard negatives using an independent similarity model (e.g., SigLIP/DINO) and re-measure V+O accuracy. If V+O accuracy stays near random in (1)-(2) and under (3), the de-biasing is mechanism-independent. If not, 'EOB-free' must be narrowed to CLIP-based visual similarity, and the V+Q+O drop may not generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The key contribution is a de-biased benchmark: GroundAttack creates hard negatives balanced in CLIP visual relevance, and EOB-free is concluded from V+O accuracy dropping to random. The vulnerability is that construction and validation share the same measure. Grounding experiments show CLIP(V,O) reaches 42.1% while VLM V+O reaches 51.57%, so CLIP similarity explains a large but incomplete share of the shortcut. If the remaining shortcut relies on other visual cues (object presence, OCR, layout) or on linguistic/statistical option priors, then balancing only CLIP similarity may remove one correlate while leaving other easy cues in place. The 'EOB-free' label is broader than what the construction guarantees: annotations are balanced with respect to a specific feature similarity. Consequently the reported drop in V+Q+O accuracy could be an artifact of the chosen similarity model rather than a general property of the benchmark. This is load-bearing because the paper's stated contribution is a more realistic evaluation of VLMs' QA ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies an 'Easy-Options Bias' (EOB) in multiple-choice VQA benchmarks: vision-language models (VLMs) can select the correct answer using only the image (V) and the answer options (O), without the question (Q). The authors report aggregate mean accuracies (V+O 51.57%, V+Q+O 61.11%, CLIP-based selection 42.1%) across six benchmarks and four VLM series, and attribute the bias to an imbalance in visual relevance between correct and incorrect options in feature space. They propose GroundAttack, a toolkit that automatically generates hard negative options balanced in CLIP-based visual relevance, apply it to NExT-QA and MMStar to create 'EOB-free' annotations, and claim that on these annotations VLMs approach random accuracy under V+O and drop to non-saturated accuracy under V+Q+O. The manuscript as reviewed contains only the abstract, introduction, and two figure captions; the methods, experimental protocol, and results sections are absent.","tokens_in":2283,"tokens_out":5563,"duration_ms":63411,"significance":"If the claims are correct, this work is significant because it suggests that multiple-choice VQA benchmarks substantially overestimate the visual reasoning capabilities of VLMs: the question is often unnecessary for selecting the answer. The proposed hard-negative rebalancing (GroundAttack) could lead to more realistic evaluation benchmarks, and the paper promises release of code and annotations, which is a strength. However, the current manuscript provides insufficient evidence. The CLIP-based grounding is a reasonable first step, but it accounts for only part of the observed V+O accuracy (42.1% vs. 51.57%), and the construction and validation of GroundAttack share the same CLIP-similarity measure. The central claims, while plausible, are not yet established with the rigorous experimental support expected in a journal submission.","major_comments":[{"comment":"The central quantitative evidence for EOB is the aggregate gap between V+O accuracy (51.57%) and V+Q+O accuracy (61.11%), and the CLIP-only accuracy of 42.1%. These are presented as means over six benchmarks and four VLM series with no variance measures, per-dataset breakdowns, or significance tests. The 9.54 percentage-point gap between V+O and V+Q+O could be within model or benchmark variability. Without error bars or a paired statistical test, the existence and magnitude of EOB are not established, which is load-bearing for the paper's first claim.","section":"Abstract/Figure 2"},{"comment":"GroundAttack is the main mitigation contribution, but the reviewed text contains no description of its algorithm. The paper does not state how hard negatives are generated, how 'visually plausible' is quantified beyond mentioning CLIP, which CLIP model/features are used, how balance is enforced (e.g., matching mean/max similarity), or how the NExT-QA and MMStar EOB-free annotations were produced and validated. Without this information, the proposed toolkit cannot be evaluated or reproduced, and the EOB-free claim is unsupported.","section":"Abstract/Introduction"},{"comment":"The grounding experiment and the mitigation share the same measure. GroundAttack constructs negatives balanced in CLIP visual similarity, and the success criterion is the drop in V+O accuracy. The reported CLIP(V,O) accuracy (42.1%) is substantially lower than the VLM V+O accuracy (51.57%), leaving a residual shortcut that is not explained by CLIP similarity. The paper does not show that this residual is eliminated by the CLIP-balanced negatives. Consequently, 'EOB-free' is a stronger claim than 'CLIP-balanced' and requires an independent validation of distractor quality (e.g., human plausibility, or probing for object-presence/OCR/language-prior shortcuts).","section":"Figure 2 / GroundAttack"},{"comment":"The claim that VLMs 'approach random accuracies' under V+O on the new annotations is not quantitatively defined. For a multiple-choice task with K options the chance level is 1/K, but the paper never states K for NExT-QA/MMStar, how the random baseline is computed (uniform versus empirical marginal), or whether the observed accuracies are statistically indistinguishable from that baseline. 'Approach' is too vague to support the EOB-free claim without error bars and a formal test.","section":"Abstract / EOB-free evaluation"}],"minor_comments":[{"comment":"Typographical spacing: 'anEasy-Options' should read 'an Easy-Options'.","section":"Abstract"},{"comment":"Typographical spacing: 'Easy-Options Biaslets' should read 'Easy-Options Bias lets'.","section":"Figure 1 caption"},{"comment":"The abbreviations V, Q, O, and A are defined in Figure 1 but not in this caption; please define them for self-contained reading.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early preprint rather than a full journal submission: the core idea is interesting and potentially impactful, but the reviewed version lacks the experimental sections necessary to verify even the basic quantitative claims. The circularity between the CLIP-based distractor generation and the CLIP-derived attribution is a genuine concern that needs to be addressed with an independent validation. I would encourage the authors to resubmit after adding a complete methods section, full experimental results, and statistical analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core observation is real and worth taking seriously. Across six benchmarks and four VLM families, vision+options alone averages 51.57%, just 9.54% below vision+question+options. That is a clear signal that the question is often redundant. The framing as Easy-Options Bias adds something new to the usual language/texture/type bias list, and the CLIP grounding experiment shows a large chunk of the shortcut is visual similarity: CLIP(V,O) alone gets 42.1%.\n\nWhat I like: the paper does not just complain. GroundAttack is a concrete tool that generates hard negatives by balancing CLIP visual relevance, and the new annotations on NExT-QA and MMStar make V+O accuracy drop to random and V+Q+O drop below saturation. That is a reproducible, falsifiable contribution, and the honest 'early study' framing is appropriate.\n\nThe soft spot is the loop between construction and validation. GroundAttack balances options along CLIP similarity, and then 'EOB-free' is demonstrated by V+O accuracy dropping. The paper's own numbers say CLIP gets 42.1% while VLMs get 51.57%, so the shortcut VLMs use is not fully captured by CLIP. Balancing one correlate may leave others—object presence, OCR, layout, linguistic priors—intact. The label 'EOB-free' is stronger than what the method guarantees. I also want error bars, statistical tests, dataset sizes, and a clear definition of 'random' (which depends on option count) in the final version.\n\nWho is this for: anyone building or using multiple-choice VQA benchmarks. It is a useful warning that reported scores may be inflated, and it gives a practical fix, even if partial. The stress-test concern is legitimate and should be addressed with mechanism-independent validation—for example, testing on hard negatives built from other cues or probing with ablated options.\n\nMy recommendation: engage with it. It deserves a serious referee, and the right referee will push on the dependence between the mitigation and the validation measure. Send it to review, but require the full experimental protocol and a robustness check against the self-reference problem.","headline":"A genuine and potentially important observation that multiple-choice VQA lets models bypass the question, but the 'EOB-free' claim is only as strong as the CLIP-similarity measure used to both build and validate it.","tokens_in":2719,"tokens_out":1923,"would_cite":false,"duration_ms":22487,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports an Easy-Options Bias in multiple-choice VQA benchmarks: a vision-language model can pick the correct answer from the image and options alone, without seeing the question.","keywords":["Easy-Options Bias","multiple-choice VQA","vision-language models","shortcut learning","CLIP similarity","hard negatives","benchmark bias","VQA evaluation"],"falsifier":"Take a benchmark re-annotated by GroundAttack and run a VLM on (V+O) with hard negatives picked by a different visual encoder; if accuracy stays well above chance, then CLIP similarity was not the operative shortcut and the claim that the annotations are EOB-free would be called into question. Alternatively, for any original benchmark, replace the negative options with options whose CLIP similarity is balanced and check whether V+O accuracy still stays above chance.","tokens_in":1933,"feed_emoji":"🖼️","tokens_out":7461,"duration_ms":71511,"temperature":0.7,"pith_summary":"This early study reports that several multiple-choice visual question answering benchmarks—including MMStar, RealWorldQA, SEED-Bench, NExT-QA, STAR, and Video-MME—contain an Easy-Options Bias. Across four VLM families, models given only the image and option list reach 51.57% mean accuracy, compared with 61.11% when the question is included, and a CLIP-based image–option matching baseline reaches 42.1%. The paper attributes the bias to an imbalance in visual relevance: the correct option is typically more similar to the image in feature space than the distractors. To correct it, GroundAttack generates hard negative options whose visual similarity is balanced with the correct answer. On the resulting EOB-free annotations, V+O accuracy approaches chance and full V+Q+O accuracy falls to non-saturated levels, indicating that the original numbers do not purely measure question-guided reasoning.","feed_headline":"Many VQA benchmarks are answerable without the question","feed_subtitle":"The correct option is the most visually similar to the image, so models skip reasoning; rebalanced options fix that.","key_machinery":"The central objects are the Easy-Options Bias and the CLIP-based visual-similarity score $\\mathrm{CLIP}(V,O)$ that exposes it. The bias is quantified by the accuracy gap between $(V,Q,O)$ and $(V,O)$ inputs; a small gap means the question adds little. GroundAttack is the corrective machinery: it automatically rewrites negative options until their CLIP similarity to the image matches the correct answer's, removing the feature-space imbalance. The load-bearing operation is this balancing—it converts a benchmark in which the correct answer is visually distinctive into one in which distractors are visually interchangeable with it.","core_discovery":"The central claim is that current multiple-choice VQA benchmarks overstate VLMs' reasoning ability because the correct answer is, on average, more visually aligned with the image than the negative options. This makes the question unnecessary: a model can select the correct option by matching image and option features alone. The supporting experiment shows mean accuracy of 51.57% with only (V,O) versus 61.11% with (V,Q,O), and 42.1% using CLIP similarity directly. The paper then argues this imbalance is a dataset construction artifact rather than a sign of competence, and shows that once negative options are made visually plausible—NExT-QA and MMStar re-annotated with GroundAttack—VLMs fall t","pith_inferences":["Editorial extension: the (V,Q,O) versus (V,O) accuracy gap is a cheap audit that any benchmark builder can run; a large gap is evidence the options are doing too much work.","Editorial extension: because GroundAttack balances with CLIP similarity, the annotations are only as good as that proxy; human plausibility ratings or another vision encoder would test whether the bias is CLIP-specific.","Editorial extension: automatically generated benchmarks that use model-produced options may inherit EOB; applying GroundAttack there is a direct stress test.","Editorial extension: if EOB is as widespread as this paper suggests, comparing models on EOB-free sets could separate those that consult the question from those that pattern-match images and options."],"forward_implications":["Benchmark scores on the affected datasets partly measure option-set artifacts: on EOB-free versions, V+O accuracy approaches chance, so the question is no longer redundant.","The V+Q+O accuracy itself drops to non-saturated levels once hard negatives are balanced, meaning previously reported numbers overstate how much question-guided reasoning these models perform.","GroundAttack can serve as an automatic audit: running the (V,Q,O) versus (V,O) comparison over an MCQ-VQA dataset reveals whether the option set is EOB-prone.","The same de-biasing process can be applied to other multiple-choice VQA benchmarks, including video ones such as Video-MME, though the paper's evidence there is early.","If EOB is widespread, ranking of VLMs may change under EOB-free evaluation, because models that lean on the shortcut lose their advantage."],"supporting_citations":[{"why":"Supplies the general concern that benchmark success may reflect dataset biases rather than genuine reasoning.","marker":"[14]"},{"why":"Defines shortcut learning in VQA, the phenomenon the paper's Easy-Options Bias extends.","marker":"[22]"},{"why":"Provides the type-bias analysis and the VQA-CP rebalanced benchmark that motivate the debiasing strategy.","marker":"[25]"},{"why":"Offers VQA-VS as a prior out-of-distribution split for reducing shortcut learning.","marker":"[26]"},{"why":"Offers GQA-AUG as another approach to removing spurious correlations in VQA evaluation.","marker":"[27]"}],"fun_headline_variants":["VQA benchmarks leak answers via visual similarity","VLMs ace multiple-choice without reading the question","Answering VQA without the question: a benchmark flaw","GroundAttack: fixing easy-option bias in VQA benchmarks","Many VQA tests pass with no question—just image and options"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The diagnosis and the fix both assume that CLIP visual similarity between an option and the image is the same cue the VLM itself exploits when the question is removed; if models use some other clue, balancing CLIP scores may not remove the bias or may introduce new artifacts.","fun_headline_variants_meta":{"raw":{"variants":["VQA benchmarks leak answers via visual similarity","VLMs ace multiple-choice without reading the question","Answering VQA without the question: a benchmark flaw","GroundAttack: fixing easy-option bias in VQA benchmarks","Many VQA tests pass with no question—just image and options"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1228,"prompt_tokens":762,"completion_tokens":466,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":506,"tokens_out":466,"duration_ms":5437,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:01:31.791790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark re-annotated by GroundAttack and run a VLM on (V+O) with hard negatives picked by a different visual encoder; if accuracy stays well above chance, then CLIP similarity was not the operative shortcut and the claim that the annotations are EOB-free would be called into question. Alternatively, for any original benchmark, replace the negative options with options whose CLIP similarity is balanced and check whether V+O accuracy still stays above chance.","supporting_citations":[{"cited_title":"Dissecting multimodality in V ideo QA transformer models by impairing modality fusion","cited_arxiv_id":null,"evidence_quote":"Supplies the general concern that benchmark success may reflect dataset biases rather than genuine reasoning."},{"cited_title":"Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering","cited_arxiv_id":null,"evidence_quote":"Defines shortcut learning in VQA, the phenomenon the paper's Easy-Options Bias extends."},{"cited_title":"Don't just assume; look and answer: Overcoming priors for visual question answering","cited_arxiv_id":null,"evidence_quote":"Provides the type-bias analysis and the VQA-CP rebalanced benchmark that motivate the debiasing strategy."},{"cited_title":"Language prior is not the only shortcut: A benchmark for shortcut learning in VQA","cited_arxiv_id":null,"evidence_quote":"Offers VQA-VS as a prior out-of-distribution split for reducing shortcut learning."},{"cited_title":"On the Role of Visual Grounding in VQA","cited_arxiv_id":"2406.18253","evidence_quote":"Offers GQA-AUG as another approach to removing spurious correlations in VQA evaluation."}],"review_version":1}