{"id":"ba5ec8c0-dbf6-42b3-8d13-e3f76386b3d7","arxiv_id":"2608.12928","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a new Polish medical VQA benchmark from board certification exams, vision-language models perform better from question text alone than from images alone and score above chance from answer choices alone, indicating weak visual grounding.","lead":"This paper builds a Polish-language medical VQA benchmark from specialist board exam questions and tests nine vision-language models, finding that models rely more on question text than on images and beat chance from answer choices alone. The result cautions that high benchmark scores may reflect textual shortcuts rather than visual or medical competence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"C+I ablation is confounded: the Appendix A.2 prompt tells models they have only answer options even when an image is attached, so the text-vs-image comparison may be an artifact of prompt asymmetry.","rationale":"The reader correctly identified the C+Q hallucination issue, but the more decisive flaw is in the C+I arm. Appendix A.2's prompt tells the model it only has answer options; using that same prompt with an image attached makes C+I a test of compliance with an incorrect premise, not of image-only competence. Since the central claim depends on C+Q > C+I, the conclusion may be a prompt artifact. The full-input Table 4 actually shows large C+Q to C+Q+I gains on image-dominant questions for several models, so the claim that visual evidence is underutilized is more nuanced than the abstract suggests. The fix is simple: use a neutral prompt that acknowledges the image and omits the question text. I therefore keep the CONDITIONAL verdict, adding the prompt-asymmetry check to the reader's list. I do not think the paper merits rejection; the benchmark and raw accuracy results stand independently of the grounding analysis.","tokens_in":13566,"tokens_out":6843,"duration_ms":72963,"concrete_test":"Re-run the C+I configuration for all models with a prompt that explicitly acknowledges the presence of the image and only the absence of the question text, e.g., 'An image is provided, but the question text is unavailable; choose the most likely correct answer from options A–E.' If the corrected C+I accuracy exceeds the reported C+I by a nontrivial margin (for example, by more than 5 points on average) or the C+Q − C+I gap shrinks by more than half, the central claim should be revised or re-argued. If the corrected C+I accuracy stays near the reported values, the prompt confound is negligible and the original conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that models derive more useful information from question text than from the image (Sections 6.2.2 and 7)—rests on comparing C+Q with C+I. That comparison is not clean. Appendix A.2 gives the system prompt used for both C and C+I: it explicitly states 'You do not receive the question text. You only have the answer options A–E' (Polish: 'Masz tylko odpowiedzi A–E'). In the C+I configuration, however, the image is attached to the input. The prompt therefore tells the model that no additional input is available, even though the image is physically present. This can suppress the model's use of the image, pushing C+I accuracy down toward the choices-only C baseline and inflating the C+Q − C+I gap. The reported ordering then does not cleanly show that models underutilize visual evidence; it may instead show that models follow a prompt telling them they have only choices. The reader's hallucination concern about C+Q being inflated is real but less decisive for the central claim: even if C+Q is inflated by models hallucinating an image from textual cues, that still demonstrates reliance on text over actual visual evidence. The C+I prompt asymmetry, by contrast, directly contaminates the comparison. The full-input results also complicate the narrative: in Table 4, GPT-5.6-sol (N) improves from 42.86 on image-dominant questions under C+Q to 72.14 under C+Q+I, and Qwen3.5-397B-A17B from 45.00 to 52.14, showing substantial visual uptake when both modalities are present. Lower full-input accuracy on image-dominant items may reflect intrinsic difficulty differences rather than neglect, especially since human per-category accuracy is not reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PES-VQA, a Polish-language medical visual question answering benchmark built from 286 image-containing Polish Board Certification Examination questions, together with a 480-question text-only QA control subset. Nine vision-language models are evaluated under four input configurations (choices-only C, choices-plus-question C+Q, choices-plus-image C+I, and full C+Q+I), compared against human examinees on an aligned subset, categorized by image importance and visual domain, and screened for data contamination with DCQ. The central empirical claims are that models derive more useful information from the question text than from the image (C+Q outperforms C+I), that they perform worse on image-dominant questions even with the full input, and that they achieve above-chance accuracy from answer choices alone. The paper concludes that current VLMs rely more heavily on textual cues than on visual evidence when answering Polish medical examination questions.","tokens_in":13822,"tokens_out":4948,"duration_ms":52591,"significance":"The benchmark is a valuable new resource: it uses real specialist examination items, official answer keys, a human-examinee reference, and a contamination screen, and it addresses an underexplored non-English medical VQA setting. The controlled input-ablation design is thoughtful, and the above-chance choices-only result is a useful caution for interpreting medical benchmark scores. If the central comparison survives a prompt-symmetry check, the finding that models often prefer textual cues over visual evidence would be practically important for medical AI evaluation. The paper does not report statistical uncertainty or annotation reliability, and the main ablation comparison is currently compromised by an asymmetric prompt, so the significance of the headline claim is contingent on a re-analysis.","major_comments":[{"comment":"The C+Q versus C+I comparison, which underpins the central claim in §6.2.2 and the Conclusion, is not clean because of prompt asymmetry. Appendix A.2 shows that the same system prompt is used for both C and C+I, and it explicitly tells the model: 'You do not receive the question text. You only have the answer options A–E.' In the C+I configuration the image is physically attached to the input, so the model is being told that no additional input is available even though an image is present. This can suppress the model's use of the image and deflate C+I accuracy, inflating the reported C+Q − C+I gap. The C+Q direction is also potentially inflated because the model is not informed that the image is missing and may hallucinate plausible image content, following Asadi et al. Both effects act in the same direction. Please re-run the C+I condition with a prompt that does not deny the existence of additional input (e.g., 'you receive answer options and possibly an image, but no question text'), or add a control that manipulates only modality presence while keeping the instruction neutral, and report whether the C+Q > C+I ordering persists.","section":"§4.2 and Appendix A.2"},{"comment":"The paper reports no confidence intervals, significance tests, or effect-size measures for any of the accuracy comparisons. The claim that the pattern 'holds for every model except LLaVA-PLLuM' and the category-level results in Table 4 rest on point estimates over 286 questions, with category splits of n=146 and n=140. Differences of a few percentage points may not be reliable, and the human-vs-model comparisons in Table 3 also lack uncertainty quantification. Please report bootstrap confidence intervals and, for paired model predictions, McNemar tests or equivalent paired statistics, especially for C+Q vs C+I and C+Q vs C+Q+I on the image-dominant subset.","section":"§6.2.2 and Table 4"},{"comment":"The image-importance categorization (categories 0, 1, 2) is load-bearing for §6.3.1, but the paper does not specify who assigned the labels, whether there was a detailed annotation protocol, or whether any inter-annotator agreement was measured. Without a reliability measure such as Cohen's kappa or a disagreement analysis, the category-level comparisons are difficult to interpret. Please provide annotation details and agreement statistics.","section":"§4.4.1 and Table 4"},{"comment":"The full-input results in Table 4 show substantial visual uptake on image-dominant questions: GPT-5.6-sol (N) improves from 42.86 (C+Q) to 72.14 (C+Q+I), and Qwen3.5-397B-A17B from 45.00 to 52.14. This indicates that models do use visual evidence when it is present, at least on image-dominant items. The conclusion that models 'underutilize visual evidence' or 'rely more heavily on textual cues' should be qualified to mean relative reliance under these ablation conditions, not an absence of visual processing; otherwise the large C+Q-to-C+Q+I gains on category 2 are in tension with the headline claim.","section":"§6.3.1 and Table 4"}],"minor_comments":[{"comment":"The prompt text 'Masz tylko odpowiedzi A–E' is translated as 'You only have the answer options A–E,' which could be read as denying the presence of any other input modality. Since the image is present in C+I, consider rewording the prompt (both Polish and English) to refer specifically to the absence of question text, if that is the intended meaning.","section":"Appendix A.2"},{"comment":"The visual-domain labels are rendered with odd spacing ('WA VEFORM', 'COMPOSITE'); please fix the typography. Also, the LLaVA model name appears as 'LLaV A' in several places.","section":"Table 5 and general formatting"},{"comment":"The related-work section cites 'PoVisLE (Anonymous, 2026)' without a venue or author information. If the submission is intended for double-blind review this may be acceptable, but otherwise the reference should be completed.","section":"References"},{"comment":"The contamination analysis is applied only to the textual part of the questions; the authors should note explicitly that image-level memorization or leakage is not assessed by this procedure.","section":"§5"},{"comment":"The QA control subset sampling procedure (10 questions per examination sheet) is described, but the rationale for exactly 10 per sheet and the possible effect of the resulting specialty distribution on the QA/VQA comparison could be stated more explicitly.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is currently vulnerable to a prompt-asymmetry artifact that is easy to test. If the authors re-run the C+I condition with a neutral prompt and the ordering persists, the paper would be a solid contribution for a specialized NLP/medical-AI venue. The lack of confidence intervals and annotation reliability should also be addressed before acceptance. The benchmark itself is timely and the human-reference alignment is a nice touch; I would not reject on the current evidence, but the headline conclusion should not be published in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the PES-VQA benchmark: 286 Polish board-exam questions with images, paired with a text-only control and an approximate human reference. That is a useful resource, and the dataset construction is careful—the human-answer alignment procedure that deals with duplicate exam versions is thoughtful, and the contamination analysis shows awareness of a real problem. I would send this to peer review on the strength of the benchmark alone.\n\nThe paper's central empirical claim is that models derive more useful information from the question text than from the image, based on C+Q outperforming C+I across models. That comparison is not clean. The prompt used for C and C+I (Appendix A.2) explicitly tells the model \"You do not receive the question text. You only have the answer options A–E.\" In the C+I condition, the image is attached, but the prompt tells the model the only input is the choices. That can suppress image use, pushing C+I accuracy down toward the choices-only baseline and inflating the C+Q − C+I gap. This is not a minor detail; it directly bears on the paper's main conclusion. The consistency of the ordering across models doesn't rescue it, because the same prompt asymmetry applies to all models.\n\nThe image-dominant category analysis provides some independent support, but it is weaker than it looks. Full-input accuracy is lower on image-dominant questions, yet without human per-category accuracy we cannot separate genuine visual neglect from the possibility that those items are simply harder. The paper does not report human numbers by category, so the interpretation remains open.\n\nOther soft spots are minor and fixable: no confidence intervals or significance tests for the key comparisons, no inter-annotator agreement for the image-importance labels, and no explicit statement about dataset and code release. The C+Q hallucination concern raised internally is real but less decisive—even if models hallucinate an image from text, that still shows reliance on text over actual visual evidence. The prompt asymmetry is the load-bearing flaw.\n\nBottom line: a useful benchmark, a plausible but not yet established conclusion. The paper deserves a serious referee, but it needs revision—fix the C+I prompt so it doesn't deny the image's presence (or add a condition that explicitly states what inputs are available), report human per-category accuracy, and add basic uncertainty quantification. I would not cite the underutilization claim as-is, but I would cite the benchmark once the data are available.","headline":"Useful new Polish medical VQA benchmark, but the headline claim about visual underuse rests on an ablation with a prompt asymmetry that likely deflates the image-only condition.","tokens_in":14455,"tokens_out":2258,"would_cite":true,"duration_ms":25904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a new Polish medical VQA benchmark, vision-language models extract more useful signal from the question text than from the image, and their accuracy drops most on image-dominant questions.","keywords":["Polish medical VQA","vision-language models","visual grounding","input ablation","image importance","textual bias","multiple-choice medical exam","Polish board certification examination"],"falsifier":"Re-run the C+Q ablation with a prompt that explicitly states that no image is available and asks the model to answer from the question text alone; if accuracy on image-dominant questions drops materially, the published C+Q scores overstate text-derived information and may invert the C+Q > C+I ordering.","tokens_in":13350,"feed_emoji":"🩺","tokens_out":7143,"duration_ms":69394,"temperature":0.7,"pith_summary":"This paper introduces PES-VQA, a Polish-language medical visual question answering benchmark built from 286 image-containing Polish Board Certification Examination items, paired with a 480-question text-only control set. The authors evaluate nine open-weight and commercial vision-language models and ask not just how accurate they are but how much information each input component contributes. Their central finding is that models rely more heavily on textual cues than on visual evidence: on image-containing questions, models answer more accurately with the question text but no image (C+Q) than with the image but no question text (C+I), and every evaluated model performs worse on image-dominant questions even when the full input is available. The task remains hard for machines, with the best model at 79.0% on the full VQA set and only the strongest commercial system beating the approximate human reference. A skeptical reader should care because the result shows why high multiple-choice accuracy in medical VQA does not by itself demonstrate visual understanding.","feed_headline":"Polish medical VQA: models lean on text, skip the image","feed_subtitle":"New PES exam benchmark shows models score higher without the image than without the question text.","key_machinery":"The load-bearing mechanism is a set of controlled input configurations applied to every question: answer choices only (C), choices plus question text (C+Q), choices plus image (C+I), and the complete choices-plus-question-plus-image input (C+Q+I). Comparing C+Q with C+I isolates which modality contributes more usable signal, and comparing the full input with each ablation estimates each component's marginal value. The second mechanism is a three-level image-importance annotation: '0' image non-essential, '1' complementary, '2' image-dominant. Splitting results by this label shows that the text-over-image advantage is concentrated exactly where visual evidence should matter most, namely image-dominant questions.","core_discovery":"In the paper's own terms, the discovery is that current vision-language models underutilize visual evidence when answering Polish medical examination questions. The evidence is an ablation ordering that holds across essentially all systems evaluated: presented with the answer choices and question text but no image (C+Q), models score higher than with answer choices and image but no question text (C+I); and on the 140 questions labeled image-dominant, accuracy is lower than on the 146 text-sufficient or complementary questions even under the complete C+Q+I input. Because humans find the VQA and QA subsets equally hard, while most models find VQA harder, the benchmark's difficulty is real; the added image does not help models the way it helps human examinees. Correcting for the fact that models also score above chance from answer choices alone, the paper concludes that accuracy on this benchmark should not be read as evidence of multimodal medical competence.","pith_inferences":["Editorial inference: the C+Q condition may overstate text-only ability because models are never told the image is missing; they can silently assume an image and answer from memorized clinical patterns. A prompt that announces the absence of the image would test how much of the text-over-image gap is genuine textual reasoning versus hallucinated visual context.","A pure visual-grounding test could present the image plus answer choices with a generic instruction such as 'choose the statement supported by the image'; if models still score poorly, the deficit is in extracting clinical evidence from images, not in understanding Polish.","The same ablation protocol could be applied to English and multilingual medical VQA datasets to see whether the text-over-image reliance is a general property of vision-language models or specific to low-resource-language medical exams."],"forward_implications":["Benchmark scores on medical VQA should be reported alongside ablations that remove the image, the question, or the answer choices; otherwise a high number can reflect text priors rather than visual grounding.","The ordering C+Q > C+I predicts that models will continue to underperform on exam items whose decisive evidence is visual, even after overall accuracy improves.","The above-chance choices-only results imply that the multiple-choice format itself leaks information, so medical exam evaluations need a random baseline set below 20% or a calibration check.","Because only one commercial system beats the approximate human reference, the PES-VQA set is far from saturated and can serve as a challenging benchmark for Polish multimodal medical QA."],"supporting_citations":[{"why":"Shows that frontier VLMs obtain high scores on multimodal benchmarks without image input; motivates the silent-omission protocol and the interpretation that textual cues drive performance.","marker":"Asadi et al., 2025"},{"why":"Demonstrates that LLMs answer multiple-choice questions from choices alone; supplies the choices-only (C) configuration as a prior for answer-option artifacts.","marker":"Balepur et al., 2024"},{"why":"Introduces paired image and question data to make the visual signal matter; frames the concern that accuracy without visual grounding is a benchmark defect.","marker":"Goyal et al., 2017"},{"why":"Shows VQA models exploit question-answer priors; grounds the controlled-input comparison as a way to expose such priors.","marker":"Agrawal et al., 2018"},{"why":"Provides the Polish-oriented open-weight models among the evaluated systems, which form a key part of the model comparison.","marker":"Statkiewicz et al., 2026"}],"fun_headline_variants":["Vision-language models underuse images in Polish medical VQA","AI scores higher without images in Polish medical VQA","Text beats images for AI on Polish medical exam","Images don't help vision-language models on Polish medical VQA","For AI, visual evidence is secondary in Polish medical VQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that silently deleting the image from an image-containing question yields a clean text-only baseline; because models are not told the image is missing, they may hallucinate a plausible image from the question text, inflating the C+Q score and the apparent text-over-image gap.","fun_headline_variants_meta":{"raw":{"variants":["Vision-language models underuse images in Polish medical VQA","AI scores higher without images in Polish medical VQA","Text beats images for AI on Polish medical exam","Images don't help vision-language models on Polish medical VQA","For AI, visual evidence is secondary in Polish medical VQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3412,"prompt_tokens":909,"completion_tokens":2503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2423}},"tokens_in":525,"tokens_out":2503,"duration_ms":19562,"temperature":1.0,"reasoning_tokens":2423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:20.072028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the C+Q ablation with a prompt that explicitly states that no image is available and asks the model to answer from the question text alone; if accuracy on image-dominant questions drops materially, the published C+Q scores overstate text-derived information and may invert the C+Q > C+I ordering.","supporting_citations":[{"cited_title":"Annotation-Efficient Vision-Language Model Adaptation to the P olish Language Using the LL a VA Framework","cited_arxiv_id":null,"evidence_quote":"Provides the Polish-oriented open-weight models among the evaluated systems, which form a key part of the model comparison."}],"review_version":1}