REVIEW 4 major objections 5 minor 37 references
Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read On a new Polish medical VQA benchmark, vision-language models extract more useful signal from the question text than from the image, and their accuracy drops most on image-dominant questions.
desk verdict Useful new Polish medical VQA benchmark, but the headline claim about visual underuse rests on an ablation with a prompt asymmetry that likely deflates the image-only condition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a set of controlled input configurations applied to every question: answer choices only (C), choices plus question text (C+Q), choices plus image (C+I), and the complete choices-plus-question-plus-image input (C+Q+I). Comparing C+Q with C+I isolates which modality contributes more usable signal, and comparing the full input with each ablation estimates each component's marginal value. The second mechanism is a three-level image-importance annotation: '0' image non-essential, '1' complementary, '2' image-dominant. Splitting results by this label shows that the text-over-image advantage is concentrated exactly where visual evidence should matter most, namely image-dominant questions.
What would settle it
Re-run the C+Q ablation with a prompt that explicitly states that no image is available and asks the model to answer from the question text alone; if accuracy on image-dominant questions drops materially, the published C+Q scores overstate text-derived information and may invert the C+Q > C+I ordering.
Extended reading notes
Core claim
In the paper's own terms, the discovery is that current vision-language models underutilize visual evidence when answering Polish medical examination questions. The evidence is an ablation ordering that holds across essentially all systems evaluated: presented with the answer choices and question text but no image (C+Q), models score higher than with answer choices and image but no question text (C+I); and on the 140 questions labeled image-dominant, accuracy is lower than on the 146 text-sufficient or complementary questions even under the complete C+Q+I input. Because humans find the VQA and QA subsets equally hard, while most models find VQA harder, the benchmark's difficulty is real; the added image does not help models the way it helps human examinees. Correcting for the fact that models also score above chance from answer choices alone, the paper concludes that accuracy on this benchmark should not be read as evidence of multimodal medical competence.
Load-bearing premise
The load-bearing premise is that silently deleting the image from an image-containing question yields a clean text-only baseline; because models are not told the image is missing, they may hallucinate a plausible image from the question text, inflating the C+Q score and the apparent text-over-image gap.
Editorial extensions
If this is right
- Benchmark scores on medical VQA should be reported alongside ablations that remove the image, the question, or the answer choices; otherwise a high number can reflect text priors rather than visual grounding.
- The ordering C+Q > C+I predicts that models will continue to underperform on exam items whose decisive evidence is visual, even after overall accuracy improves.
- The above-chance choices-only results imply that the multiple-choice format itself leaks information, so medical exam evaluations need a random baseline set below 20% or a calibration check.
- Because only one commercial system beats the approximate human reference, the PES-VQA set is far from saturated and can serve as a challenging benchmark for Polish multimodal medical QA.
Reading between the lines
- Editorial inference: the C+Q condition may overstate text-only ability because models are never told the image is missing; they can silently assume an image and answer from memorized clinical patterns. A prompt that announces the absence of the image would test how much of the text-over-image gap is genuine textual reasoning versus hallucinated visual context.
- A pure visual-grounding test could present the image plus answer choices with a generic instruction such as 'choose the statement supported by the image'; if models still score poorly, the deficit is in extracting clinical evidence from images, not in understanding Polish.
- The same ablation protocol could be applied to English and multilingual medical VQA datasets to see whether the text-over-image reliance is a general property of vision-language models or specific to low-resource-language medical exams.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PES-VQA, a Polish-language medical visual question answering benchmark built from 286 image-containing Polish Board Certification Examination questions, together with a 480-question text-only QA control subset. Nine vision-language models are evaluated under four input configurations (choices-only C, choices-plus-question C+Q, choices-plus-image C+I, and full C+Q+I), compared against human examinees on an aligned subset, categorized by image importance and visual domain, and screened for data contamination with DCQ. The central empirical claims are that models derive more useful information from the question text than from the image (C+Q outperforms C+I), that they perform worse on image-dominant questions even with the full input, and that they achieve above-chance accuracy from answer choices alone. The paper concludes that current VLMs rely more heavily on textual cues than on visual evidence when answering Polish medical examination questions.
Significance. The benchmark is a valuable new resource: it uses real specialist examination items, official answer keys, a human-examinee reference, and a contamination screen, and it addresses an underexplored non-English medical VQA setting. The controlled input-ablation design is thoughtful, and the above-chance choices-only result is a useful caution for interpreting medical benchmark scores. If the central comparison survives a prompt-symmetry check, the finding that models often prefer textual cues over visual evidence would be practically important for medical AI evaluation. The paper does not report statistical uncertainty or annotation reliability, and the main ablation comparison is currently compromised by an asymmetric prompt, so the significance of the headline claim is contingent on a re-analysis.
major comments (4)
- [§4.2 and Appendix A.2] The C+Q versus C+I comparison, which underpins the central claim in §6.2.2 and the Conclusion, is not clean because of prompt asymmetry. Appendix A.2 shows that the same system prompt is used for both C and C+I, and it explicitly tells the model: 'You do not receive the question text. You only have the answer options A–E.' In the C+I configuration the image is physically attached to the input, so the model is being told that no additional input is available even though an image is present. This can suppress the model's use of the image and deflate C+I accuracy, inflating the reported C+Q − C+I gap. The C+Q direction is also potentially inflated because the model is not informed that the image is missing and may hallucinate plausible image content, following Asadi et al. Both effects act in the same direction. Please re-run the C+I condition with a prompt that does not deny the existence of additional input (e.g., 'you receive answer options and possibly an image, but no question text'), or add a control that manipulates only modality presence while keeping the instruction neutral, and report whether the C+Q > C+I ordering persists.
- [§6.2.2 and Table 4] The paper reports no confidence intervals, significance tests, or effect-size measures for any of the accuracy comparisons. The claim that the pattern 'holds for every model except LLaVA-PLLuM' and the category-level results in Table 4 rest on point estimates over 286 questions, with category splits of n=146 and n=140. Differences of a few percentage points may not be reliable, and the human-vs-model comparisons in Table 3 also lack uncertainty quantification. Please report bootstrap confidence intervals and, for paired model predictions, McNemar tests or equivalent paired statistics, especially for C+Q vs C+I and C+Q vs C+Q+I on the image-dominant subset.
- [§4.4.1 and Table 4] The image-importance categorization (categories 0, 1, 2) is load-bearing for §6.3.1, but the paper does not specify who assigned the labels, whether there was a detailed annotation protocol, or whether any inter-annotator agreement was measured. Without a reliability measure such as Cohen's kappa or a disagreement analysis, the category-level comparisons are difficult to interpret. Please provide annotation details and agreement statistics.
- [§6.3.1 and Table 4] The full-input results in Table 4 show substantial visual uptake on image-dominant questions: GPT-5.6-sol (N) improves from 42.86 (C+Q) to 72.14 (C+Q+I), and Qwen3.5-397B-A17B from 45.00 to 52.14. This indicates that models do use visual evidence when it is present, at least on image-dominant items. The conclusion that models 'underutilize visual evidence' or 'rely more heavily on textual cues' should be qualified to mean relative reliance under these ablation conditions, not an absence of visual processing; otherwise the large C+Q-to-C+Q+I gains on category 2 are in tension with the headline claim.
minor comments (5)
- [Appendix A.2] The prompt text 'Masz tylko odpowiedzi A–E' is translated as 'You only have the answer options A–E,' which could be read as denying the presence of any other input modality. Since the image is present in C+I, consider rewording the prompt (both Polish and English) to refer specifically to the absence of question text, if that is the intended meaning.
- [Table 5 and general formatting] The visual-domain labels are rendered with odd spacing ('WA VEFORM', 'COMPOSITE'); please fix the typography. Also, the LLaVA model name appears as 'LLaV A' in several places.
- [References] The related-work section cites 'PoVisLE (Anonymous, 2026)' without a venue or author information. If the submission is intended for double-blind review this may be acceptable, but otherwise the reference should be completed.
- [§5] The contamination analysis is applied only to the textual part of the questions; the authors should note explicitly that image-level memorization or leakage is not assessed by this procedure.
- [§3.2] The QA control subset sampling procedure (10 questions per examination sheet) is described, but the rationale for exactly 10 per sheet and the possible effect of the resulting specialty distribution on the QA/VQA comparison could be stated more explicitly.
Circularity Check
No circularity found: the paper is an external benchmark evaluation whose central text-vs-image claim rests on measured model accuracy against official exam keys, not on fitted parameters or self-citation.
full rationale
The paper contains no derivation chain that reduces to its own inputs. It constructs a benchmark from Polish Board Certification Examination questions and official answer keys, evaluates fixed vision-language models under controlled input configurations, and compares model accuracy with human examinee performance. The central claim that models derive more useful information from question text than from the image is an empirical ablation result (C+Q versus C+I in Section 6.2.2 and Table 4), not a quantity fitted from or defined in terms of the conclusion. The image-importance categories are human annotations used for stratified reporting rather than model-derived outputs, so they do not smuggle the conclusion into the measurement. Self-citations appear in related work and model selection (e.g., Grzybowski et al. 2025, Lasik et al. 2026, Statkiewicz et al. 2026), but none of the central claims is justified only by those citations; the benchmark results are externally grounded in official answer keys and human examinee data. The Appendix A.2 prompt does tell models in the C and C+I configurations that they have only answer options even when an image is attached, which is a genuine internal-validity concern for the text-versus-image comparison, but it is a methodological confound rather than circular reasoning: the ordering of accuracies is still an observed empirical fact, not a consequence of how the quantities were defined. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption CEM official answer keys are valid ground truth for the benchmark questions.
- domain assumption Manual image-importance categories 0, 1, and 2 accurately reflect whether the image is needed to answer.
- domain assumption Silently omitting the image in C+Q yields a fair estimate of text-only performance.
- domain assumption The human-response alignment procedure correctly maps examinee answer sheets to benchmark questions.
- domain assumption DCQ contamination estimates transfer to this dataset.
Cite this review
Pith. "Pith review of Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence." pith.science (2026). https://pith.science/paper/6SIOWULB
@misc{pith2026260812928,
author = {Pith},
title = {Pith review of: Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence},
year = {2026},
howpublished = {\url{https://pith.science/paper/6SIOWULB}},
note = {Machine review of arXiv:2608.12928}
}
read the original abstract
We introduce a Polish-language medical visual question answering (VQA) benchmark, built from Polish Board Certification Examination questions for licensed physicians and dentists pursuing specialist certification. The benchmark comprises image-containing questions spanning diverse medical specialties and visual domains, together with a text-only question answering (QA) control set. We evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models. The task remains challenging: the best model achieves 79.0\% accuracy on the full VQA set, and only GPT-5.6 surpasses the approximate human reference on the subset with available candidate responses; all other evaluated models perform worse than humans. To assess visual grounding, we compare complete inputs with configurations omitting the image, the question, or both, and categorize questions by image importance. Models derive more useful information from the question text than from the image and perform worse on image-dominant questions. Across both QA and VQA, they nevertheless achieve above-chance accuracy from the answer choices alone, showing that non-trivial performance can persist even when key task components are missing.
Figures
Reference graph
Works this paper leans on
-
[1]
Statkiewicz, Grzegorz and Dobrzeniecka, Alicja and Seweryn, Karolina and Krasnod e bska, Aleksandra and Piosek, Karolina and Bogusz, Katarzyna and Cygert, Sebastian and Kusa, Wojciech. Annotation-Efficient Vision-Language Model Adaptation to the P olish Language Using the LL a VA Framework. Proceedings of the 19th Conference of the E uropean Chapter of th...
-
[2]
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Golchin, Shahriar and Surdeanu, Mihai. Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models. Transactions of the Association for Computational Linguistics. 2025. doi:10.1162/tacl.a.20
-
[3]
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=
-
[4]
MIRAGE: The Illusion of Visual Understanding in Vision-Language Models , author=
-
[5]
arXiv preprint arXiv:2501.02266 , year=
Jassem, Krzysztof and Ciesi. arXiv preprint arXiv:2501.02266 , year=
-
[6]
Raj Khan, Humair and Gupta, Deepak and Ekbal, Asif. Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation. Findings of the Association for Computational Linguistics: EMNLP 2021. 2021. doi:10.18653/v1/2021.findings-emnlp.151
-
[7]
2026 , howpublished =
Anonymous , title =. 2026 , howpublished =
2026
-
[8]
Artifacts or Abduction: How Do LLM s Answer Multiple-Choice Questions Without the Question?
Balepur, Nishant and Ravichander, Abhilasha and Rudinger, Rachel. Artifacts or Abduction: How Do LLM s Answer Multiple-Choice Questions Without the Question?. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.555
Show all 37 references
-
[9]
Which of These Best Describes Multiple Choice Evaluation with LLM s? A) Forced B ) Flawed C ) Fixable D ) All of the Above
Balepur, Nishant and Rudinger, Rachel and Boyd-Graber, Jordan Lee. Which of These Best Describes Multiple Choice Evaluation with LLM s? A) Forced B ) Flawed C ) Fixable D ) All of the Above. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistic...
2025 doi
-
[10]
2024 , eprint=
GPT-4 passes most of the 297 written Polish Board Certification Examinations , author=. 2024 , eprint=
2024
-
[11]
2026 , eprint=
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance? , author=. 2026 , eprint=
2026
-
[12]
P olish- E nglish medical knowledge transfer: A new benchmark and results
Grzybowski, ukasz and Pokrywka, Jakub and Ciesi \'o ka, Micha and Kaczmarek, Jeremi Ignacy and Kubis, Marek. P olish- E nglish medical knowledge transfer: A new benchmark and results. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/...
2025 doi
-
[13]
2026 , howpublished =
Gemma 4 31B IT , author =. 2026 , howpublished =
2026
-
[14]
reVISION: A Polish Benchmark for Evaluating Vision-Language Models on Multimodal National Exam Data , year=
Ciesiółka, Michał and Graliński, Filip , booktitle=. reVISION: A Polish Benchmark for Evaluating Vision-Language Models on Multimodal National Exam Data , year=
-
[15]
2026 , howpublished =
2026
-
[16]
EXAMS - V : A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
Das, Rocktim and Hristov, Simeon and Li, Haonan and Dimitrov, Dimitar and Koychev, Ivan and Nakov, Preslav. EXAMS - V : A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models. Proceedings of the 62nd Annual Meeting of the Association fo...
2024 doi
-
[17]
Scientific Data , volume =
A dataset of clinically generated visual questions and answers about radiology images , author =. Scientific Data , volume =. 2018 , doi =
2018
-
[18]
Hasan and Yuan Ling and Oladimeji Farri and Joey Liu and Matthew Lungren and Henning M\"uller , title =
Sadid A. Hasan and Yuan Ling and Oladimeji Farri and Joey Liu and Matthew Lungren and Henning M\"uller , title =. CLEF 2018 Working Notes , series =. 2018 , publisher =
2018
-
[19]
Working Notes of
Asma. Working Notes of. 2019 , publisher =
2019
-
[20]
Overview of the
Asma. Overview of the. CLEF 2020 Working Notes , series =. 2020 , publisher =
2020
-
[21]
Overview of the
Asma. Overview of the. CLEF 2021 Working Notes , series =. 2021 , publisher =
2021
-
[22]
Towards Visual Question Answering on Pathology Images
He, Xuehai and Cai, Zhuo and Wei, Wenlan and Zhang, Yichen and Mou, Luntian and Xing, Eric and Xie, Pengtao. Towards Visual Question Answering on Pathology Images. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa...
2021 doi
-
[23]
2023 , url =
Zhang, Xiaoman and Wu, Chaoyi and Zhao, Ziheng and Lin, Weixiong and Zhang, Ya and Wang, Yanfeng and Xie, Weidi , journal =. 2023 , url =
2023
-
[24]
Towards Injecting Medical Visual Knowledge into Multimodal
Chen, Junying and Gui, Chi and Ouyang, Ruyi and Gao, Anningzhe and Chen, Shunian and Chen, Guiming Hardy and Wang, Xidong and Cai, Zhenyang and Ji, Ke and Wan, Xiang and Wang, Benyou , editor =. Towards Injecting Medical Visual Knowledge into Multimodal. Proceedings of the 202...
2024 doi
-
[25]
Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering , year=
Liu, Bo and Zhan, Li-Ming and Xu, Li and Ma, Lin and Yang, Yan and Wu, Xiao-Ming , booktitle=. Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering , year=
-
[26]
Findings of the Association for Computational Linguistics: NAACL 2025 , month = apr, year =
Matos, Jo. Findings of the Association for Computational Linguistics: NAACL 2025 , month = apr, year =. doi:10.18653/v1/2025.findings-naacl.402 , pages =
2025 doi
-
[27]
2025 IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS) , pages =
A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare , author =. 2025 IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS) , pages =. 2025 , publisher =. doi:10.1109/CBMS65348.2025.00093 , url =
2025
-
[28]
Kennedy and Yetisgen, Meliha , journal =
Yim, Wen-wai and Ben Abacha, Asma and Doerning, Robert and Chen, Chia-Yu and Xu, Jiaying and Subbarao, Anita and Yu, Zixuan and Xia, Fei and Hall, M. Kennedy and Yetisgen, Meliha , journal =. 2025 , doi =
2025
-
[29]
2026 , eprint =
Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study , author =. 2026 , eprint =
2026
-
[30]
2026 , eprint =
Multilingual Hematology Visual Question Answering Dataset , author =. 2026 , eprint =
2026
-
[31]
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =
-
[32]
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =
-
[33]
Advances in Neural Information Processing Systems , volume =
Overcoming Language Priors in Visual Question Answering with Adversarial Regularization , author =. Advances in Neural Information Processing Systems , volume =. 2018 , url =
2018
-
[34]
2019 , url =
Cadene, Remi and Dancette, Corentin and Ben-Younes, Hedi and Cord, Matthieu and Parikh, Devi , booktitle =. 2019 , url =
2019
-
[35]
Computer Vision -- ECCV 2020 , series =
Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder , author =. Computer Vision -- ECCV 2020 , series =. 2020 , doi =
2020
-
[36]
Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023 , series =
Debiasing Medical Visual Question Answering via Counterfactual Training , author =. Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023 , series =. 2023 , doi =
2023
-
[37]
Zhu, Huanjia and Liu, Yishu and Zhou, Chengju and Lu, Guangming and Chen, Bingzhi , booktitle =
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.