{"id":"af7bcab8-4791-41e5-abb9-c904ffeb9ce9","arxiv_id":"2507.17262","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"VisionTrap shows that GPT-4o, GPT-4.1, Gemini Flash 2.5, and LLaVA tend to answer unanswerable visual questions rather than abstain, especially when given multiple-choice options.","lead":"This paper introduces VisionTrap, a dataset of 300 impossible images paired with 1,500 questions that have no correct answer, and tests four vision-language models on whether they abstain or guess. The main finding is that all tested models answer unanswerable questions more often when multiple-choice options are present, and that abstention rates rise sharply when options are removed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Removing options is confounded with removing usable answer tokens: the no-options prompt (Table 1) still asks for an option number and defines '5' as uncertainty, so high abstention may reflect prompt compliance rather than unanswerability recognition, and the central with-vs-without contrast is…","rationale":"I agree with the reader's weakest assumption and sharpen it: the problem is not only that the '5' token is inconsistently interpreted; the no-options prompt is structurally biased toward '5' because it requests an option number while supplying none. The central load-bearing claim is the with-options versus without-options difference. Since the without-options condition is invalid as designed, the strongest descriptive finding is not established. However, the with-options numbers are internally consistent and the dataset and abstention analysis are useful; Sections 4.3 and 4.6 already flag prompt confusion, so the authors may be able to fix this with a straightforward rerun. Therefore the reader's CONDITIONAL verdict stands; no change needed.","tokens_in":14151,"tokens_out":4639,"duration_ms":47395,"concrete_test":"Run the no-options condition with a natural prompt that does not request an option number and does not define a '5' abstention token, e.g., 'If the question cannot be answered from the image, say \"I don't know\".' Parse free-text answers for abstention semantics. If GPT-4o and LLaVA still show abstention above roughly 0.85 and 0.95, the original finding is robust; if rates drop substantially (e.g., LLaVA below 0.5), the Table 2 contrast is a prompt artifact. Additionally, on the same 1,000-question VQA v2 answerable subset, require a false-negative rate below a small threshold (e.g., <0.1) under the revised no-options prompt before trusting the unanswerable-condition numbers. Recompute Table 2 with these numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result is the with-options versus without-options contrast in Table 2. But the two conditions differ in more than the presence of options. Both prompts in Table 1 include 'Answer: <just write the option number>' and 'If you think your answer aligns with uncertain answers, then return 5.' In the without-options condition there are no options 1-4, so the only coherent numeric response under the requested format is 5. Thus GPT-4o's jump from 0.571 to 0.892, and LLaVA's 0.0 to 0.954, may simply reflect that the model has no other valid option number to emit. This is not a behavioral claim about abstention. The paper itself supplies corroborating evidence: Section 4.3 reports LLaVA's false-negative rate of 0.972 on the answerable VQA v2 subset under the same no-options prompt, i.e., the prompt induces 'unanswerable' answers even when the question is clearly answerable; Section 4.6 reports LLaVA selects option 1 in 99% of with-options cases, exactly the positional strategy one would expect when the format asks for an option number the model does not want to give. Therefore the central claim that 'removing options increases abstention dramatically' is supported by a measurement that cannot distinguish abstention from instruction-following. The with-options rates do show that models answer, but the specifically novel option-removal effect is unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VisionTrap, a dataset of 1,500 unanswerable visual questions across three image categories (surreal images, animal-object fusions, and existing fictional figures), and evaluates four VLMs—LLaVA 7B, GPT-4o, GPT-4.1, and Gemini Flash 2.5—under two prompt conditions: one with four multiple-choice options plus an abstention token ('5'), and one without options but still instructing the model to output an option number and to use '5' for uncertainty. Abstention rate is measured as the frequency of outputting the token '5'. The paper reports that models frequently fail to abstain when options are present, that removing options increases abstention substantially (e.g., GPT-4o from 0.571 to 0.892 on Existing images), and that models produce confident justifications for unanswerable questions. It also includes a false-negative analysis on answerable VQA v2 questions under the same prompt.","tokens_in":14446,"tokens_out":5701,"duration_ms":54579,"significance":"The dataset and the descriptive finding that commercial and open VLMs rarely abstain even when the prompt explicitly offers an abstention token are potentially useful for the VQA reliability community. The inclusion of a false-negative control (Section 4.3) and a justification taxonomy (Section 4.5) are constructive steps toward measuring abstention more carefully. However, the central claim about the effect of removing options is undermined by a confounded prompt design and by the paper's own false-negative results; the no-options abstention rates cannot currently be distinguished from instruction following. The contribution would be strengthened if the abstention signal were elicited in a way not tied to a single token and if the option-removal contrast were validated with calibrated metrics. The paper would also benefit from reporting uncertainty estimates; all numbers are point estimates with no confidence intervals or significance tests.","major_comments":[{"comment":"The no-options condition in Table 1 still requires the model to output an option number ('Answer: <just write the option number>') and defines '5' as the uncertainty signal; because no options 1–4 are listed, '5' is the only format-compliant numeric response. The increase in abstention rates from the with-options to the without-options condition in Table 2 (e.g., GPT-4o from 0.571 to 0.892 on Existing) therefore cannot be attributed to the model recognizing unanswerability. This interpretation is directly contradicted by the paper's own Section 4.3/Fig. 6, where LLaVA's false-negative rate on answerable VQA v2 questions under the same no-options prompt is 0.972, showing that the prompt elicits '5' responses regardless of actual answerability.","section":"Sec. 3.2, Table 1; Sec. 4.2, Table 2"},{"comment":"The paper reports that LLaVA labels 97.2% of clearly answerable VQA v2 questions as unanswerable under the no-options prompt, and states that 'relying solely on the abstention rate from unanswerable questions may not accurately reflect the model's overall performance.' This concession is in direct tension with the use of the same uncalibrated abstention rates in Section 4.2 as evidence that 'models are more likely to correctly recognize unanswerable scenarios' when options are removed. The abstention metric should be calibrated against the false-negative rate, for example by reporting abstention behavior only on the subset of unanswerable questions for which the model also correctly answers answerable controls.","section":"Sec. 4.3, Fig. 6"},{"comment":"The with-options condition is itself contaminated by positional bias: LLaVA selects option 1 in about 99% of cases (Section 4.6), and GPT-4o and Gemini also skew strongly toward option 1 (Fig. 12). Consequently, the with-options abstention rates in Table 2 conflate genuine abstention decisions with a positional answer heuristic, and the with-options versus without-options comparison does not isolate the effect of providing options. The authors should randomize option order across items or use a free-form answer format to measure abstention without this confound.","section":"Sec. 4.6, Fig. 12"}],"minor_comments":[{"comment":"Report confidence intervals or standard errors. With 500 questions per category, the differences among models on the same condition (e.g., Gemini 0.61 versus GPT-4.1 0.792 on Existing without options) may be within sampling variability; the present point estimates do not support fine-grained ranking claims.","section":"Table 2"},{"comment":"The definition AR = (TN + FN)/|D| is confusing for the VisionTrap dataset, which contains only unanswerable questions and therefore no false negatives; for that dataset the metric should be defined as TN/|D|, and the role of FN should be clarified.","section":"Eq. (1)"},{"comment":"In the 'Prompt with Options' row, the option list reads 'option1, option2, option3, option1' but should presumably be 'option1, option2, option3, option4'; as printed, the list contains a duplicate and no fourth option.","section":"Table 1"},{"comment":"The y-axis is labeled 'Accuracy' while the caption and text describe abstention rate; relabel the axis to 'Abstention Rate' to avoid confusion.","section":"Figure 4"},{"comment":"The text mentions 'GPT-4.0' in the discussion of Fig. 12, although the evaluated model is GPT-4.1; correct the typo.","section":"Section 6.3"},{"comment":"The paper does not state whether the VisionTrap dataset and evaluation code will be made publicly available; an availability statement would be important for a benchmark paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's descriptive with-options results are a reasonable starting point, but the headline option-removal effect is currently an artifact of the prompt design. The authors should be given the opportunity to re-run the without-options condition with a non-numeric abstention response (e.g., free-form 'I cannot answer') and to calibrate abstention using the VQA v2 control before the central claim can be accepted. If the dataset is released, this could become a useful benchmark contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the VisionTrap dataset is a reasonable idea and the with-options results are informative, but the paper's headline claim that removing options increases abstention dramatically is not supported, because the no-options prompt still demands an option number and '5' is the only one left.\n\nWhat's genuinely useful here: a hand-curated set of 300 images and 1,500 questions in three categories and fifteen subtypes, with human cross-checking of answerability. That is real dataset work. The qualitative taxonomy of model justifications in Table 3 is also a nice contribution—it gives you a vocabulary for why models plow ahead on impossible questions. And the with-options numbers do show that all four models answer most of the time instead of abstaining; LLaVA's near-total collapse to option 1 is a clean, believable positional-bias observation.\n\nThe soft spots are concentrated in the without-options condition, and they are load-bearing. The prompt in Table 1, in both conditions, says 'Answer: <just write the option number>' and then says 'If you think your answer aligns with uncertain answers, then return 5.' In the without-options condition there are no options 1–4, so 5 is the only coherent numeric response. GPT-4o's jump from 0.571 to 0.892, and LLaVA's from 0.0 to 0.95, may be nothing more than the model complying with the format. The paper's own Section 4.3 is the clincher: under that same no-options prompt, LLaVA labels 97.2% of answerable VQA v2 questions as unanswerable. That is not abstention ability; it is an instruction-following failure. The paper notes the high false-negative rate but never connects it back to Table 2.\n\nThe correct reading of the data is more modest: when forced to pick an option, these models prefer a wrong answer over abstention. That is a useful and believable finding. But the specific claim that removing options restores abstention is unvalidated by this design.\n\nAlso worth flagging: no error bars or significance tests, the dataset is not released, and there is no comparison against Guo et al., CLIP-UP, or Whitehead et al. The related-work section locates the line but the evaluation doesn't situate VisionTrap within it.\n\nBottom line: worth refereeing, but as a major-revision. Re-run the no-options condition with a prompt that doesn't tie abstention to an option number, release the data, add confidence intervals, and benchmark against the existing unanswerable-VQA sets. If the option-removal effect survives that, the paper will be a solid contribution to VQA robustness evaluation.","headline":"VisionTrap builds a useful unanswerable-VQA benchmark, but the headline with/without-options contrast is confounded by the prompt format and needs re-running before the paper's central claim can stand.","tokens_in":14987,"tokens_out":3006,"would_cite":false,"duration_ms":29051,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art vision-language models answer questions that have no right answer, and offering choices makes it worse.","keywords":["visual question answering","abstention","unanswerable questions","vision-language models","hallucination","multiple-choice bias","zero-shot evaluation","synthetic images"],"falsifier":"Run the VisionTrap prompts on a matched set of answerable images, or reverse the wording and placement of the abstention instruction in the no-options condition; if abstention on unanswerable questions changes sharply with the wording, or models abstain on answerable questions at similar rates, the claimed recognition of unanswerability is not supported. The paper's own LLaVA result—about 97 percent abstention on answerable VQA v2 questions in the no-options condition—is already close to such a falsifying observation for that model.","tokens_in":13957,"feed_emoji":"❓","tokens_out":7611,"duration_ms":72290,"temperature":0.7,"pith_summary":"The paper sets out to show that current vision-language models (VLMs) are biased toward producing answers even when a visual question has no correct answer, and that multiple-choice options amplify this bias. To test this, the authors built VisionTrap, a dataset of 300 impossible images with 1,500 logically unanswerable questions spanning surreal scenes, animal-object fusions, and existing fictional figures. They asked four models—LLaVA 7B, GPT-4o, GPT-4.1, and Gemini Flash 2.5—to answer either from four choices or freely, with an explicit reserved token for 'I cannot answer.' The central result is that abstention rates rise sharply when options are removed: GPT-4o, for example, goes from 0.571 to 0.892 abstention on the existing-figures category. The authors read this as evidence that models overapply learned answer patterns instead of recognizing the boundary of what an image or question can support.","feed_headline":"Vision models answer unanswerable questions: choices make it worse","feed_subtitle":"A 1,500-question impossible-image benchmark shows abstention collapses when answer choices are present.","key_machinery":"The load-bearing machinery is the VisionTrap benchmark together with a two-condition prompt protocol and a reserved abstention token. VisionTrap supplies 300 images that cannot occur in reality, each with five questions and four answer choices engineered to have no ground truth, organized into three image categories (surreal, fusion of objects and animals, existing figures) and five question subtypes per category. In the with-options condition the model picks among the four choices plus a fifth token, '5,' which is defined as abstention; in the without-options condition the same prompt asks the model to return '5' when its answer belongs to an 'uncertain set' of phrases such as 'The answer is unknown' and 'There is no definitive answer.' Abstention is quantified as $AR = (TN + FN)/|D|$ using a confusion matrix for answerable versus unanswerable questions, and the same protocol is run on answerable VQA v2 questions to detect false abstentions. The token '5' is the key operational object: every headline number in the paper is the rate at which a model emits that token.","core_discovery":"On its own terms, the paper's discovery is that state-of-the-art VLMs frequently fail to abstain on unanswerable visual questions, and the failure is strongest when the model is given multiple-choice options. Across all three image categories and all four models, abstention was lower with options than without. The sharpest contrast is LLaVA 7B, which with options abstains on 0 to 4 percent of questions but without options abstains on 95 to 98 percent; the authors attribute the difference to the presence of distractor choices rather than to an underlying change in visual understanding. GPT-4o shows the most consistent abstention, exceeding 90 percent without options on two of three categories, while Gemini Flash 2.5 abstains least overall. The paper also probes the opposite failure: models abstaining on questions that do have answers. On a 1,000-question subset of VQA v2, the no-options prompt produces high false abstention, especially for LLaVA, which abstains on about 97 percent of answerable questions. The authors therefore frame their main result cautiously: the observed abstention rates measure a tendency shaped by question format, not a clean, calibrated awareness of ignorance.","pith_inferences":["A natural next benchmark would pair VisionTrap's unanswerable images with matched answerable control questions so that a single discrimination score, such as $d'$, separates genuine recognition of unanswerability from a blanket tendency to say '5'.","The paper's answer-distribution plots suggest a positional option-1 bias, which implies a cheap test: randomize option order and see whether abstention and option-1 rates move; if they do, part of the measured effect is interface bias rather than visual reasoning.","If the option-effect generalizes beyond these four models, product design becomes a lever for honesty: open-ended answer fields or an explicit 'insufficient information' button could reduce confident wrong answers in deployed assistants.","Because the dataset's images are synthetic or public-domain, the same protocol could be extended to video or interactive embodied settings, where unanswerable questions about impossible objects are more natural."],"forward_implications":["Multiple-choice answer formats, which dominate current VQA benchmarks, systematically suppress abstention and push models to guess on questions with no right answer.","Benchmarks that measure only accuracy on answerable questions fail to expose a failure mode that matters in real deployments, where users may ask impossible or ill-posed questions.","Model rankings on 'honesty' are format-dependent: the same model can look near-random with options and strongly abstention-prone without them.","Training with unanswerable visual questions and explicit abstention supervision would be needed to close the gap, since current models are not reliably learning to say 'I don't know' from standard VQA data.","The false-abstention results imply that simply prompting for caution is not a fix: it trades hallucinations for refusal on answerable questions, especially for smaller open models."],"supporting_citations":[{"why":"provides the VQA v2 validation subset used to measure false abstentions on answerable questions.","marker":"[10]"},{"why":"prior unanswerable-VQA dataset against which the paper positions VisionTrap's harder, semantically aligned questions.","marker":"[11]"},{"why":"supplies the confusion-matrix framework and abstention-rate formulation the paper adopts.","marker":"[21]"},{"why":"provides the uncertain-answer set that anchors the prompt's 'return 5' instruction.","marker":"[34]"},{"why":"similar prompt-level abstention protocol that the authors adapt for this study.","marker":"[37]"},{"why":"formulation of reliable VQA in which models should abstain rather than answer incorrectly.","marker":"[38]"}],"fun_headline_variants":["Choices hurt VLM abstention on impossible queries","Answer choices crush VLM abstention on impossible images","With multiple-choice options, VLMs answer the unanswerable","Models guess on unanswerable images when given choices","Impossible images: VLMs answer rather than abstain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's numbers all treat the token '5' as proof that a model recognized the question as unanswerable; if models are instead reacting to prompt wording, option position, or a learned default, the abstention rates measure instruction compliance rather than awareness of ignorance.","fun_headline_variants_meta":{"raw":{"variants":["Choices hurt VLM abstention on impossible queries","Answer choices crush VLM abstention on impossible images","With multiple-choice options, VLMs answer the unanswerable","Models guess on unanswerable images when given choices","Impossible images: VLMs answer rather than abstain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00096,"raw_usage":{"total_tokens":4097,"prompt_tokens":960,"completion_tokens":3137,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":3058}},"tokens_in":576,"tokens_out":3137,"duration_ms":24977,"temperature":1.0,"reasoning_tokens":3058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:52:09.807263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the VisionTrap prompts on a matched set of answerable images, or reverse the wording and placement of the abstention instruction in the no-options condition; if abstention on unanswerable questions changes sharply with the wording, or models abstain on answerable questions at similar rates, the claimed recognition of unanswerability is not supported. The paper's own LLaVA result—about 97 percent abstention on answerable VQA v2 questions in the no-options condition—is already close to such a falsifying observation for that model.","supporting_citations":[{"cited_title":"UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models","cited_arxiv_id":"2310.10942","evidence_quote":"prior unanswerable-VQA dataset against which the paper positions VisionTrap's harder, semantically aligned questions."},{"cited_title":"Reliable visual question answering: Abstain rather than answer incorrectly","cited_arxiv_id":null,"evidence_quote":"formulation of reliable VQA in which models should abstain rather than answer incorrectly."}],"review_version":1}