Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Medical textbook figures can be turned into a 5,000-question benchmark for testing medical AI.

desk verdict A useful textbook-derived benchmark with transparent plumbing, but the headline rankings ride on unverified machine-generated answer keys and the paper's own samples show the keys leak. read the letter →

arxiv 2506.00855 v1 pith:WYE5PKAL submitted 2025-06-01 cs.AI

Sau Lai Yip , Sunan He , Yuxiang Nie , Shu Pui Chan , Yilin Ye , Sum Ying Lam , Hao Chen This is my paper · ORCID
classification cs.AI
keywords medicalvisualquestionansweringmultimodallargelanguagemodelsbenchmarkconstructionopen-accesstextbookshierarchicalannotationmodalityrecognitiondiseasediagnosisclinicalAIevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedBookVQA asks whether open-access medical textbooks, rather than research articles or repackaged classification datasets, can serve as a trustworthy source for evaluating general medical AI. The paper constructs 5,000 multiple-choice visual-question-answering items from more than a thousand medical books, spanning five question types and a hierarchical taxonomy of 42 imaging modalities, 125 anatomical structures, and 31 clinical departments. It then measures several dozen multimodal large language models on the set. The paper's central claim is that textbook-derived benchmarks are a viable and instructive evaluation paradigm, and its reported results say that proprietary general-purpose models currently lead, that medically specialized open models lag behind, and that the largest weaknesses appear in disease diagnosis, symptom recognition, and surgery rather than in simple modality identification.

What carries the argument

The load-bearing mechanism is the figure-information pairing and question-generation pipeline. Document parsing divides book pages into regions and attaches captions to images; a classifier keeps only real-case medical figures; pattern matching recovers figure reference names and collects nearby sentences that mention them; a large multimodal model then writes up to six questions per figure from five predefined types; a second multimodal model turns each question into a four-option multiple-choice item; and three filters, suitability, image-necessity, and manual review, remove off-topic, text-only-answerable, and flawed entries. A hierarchical labeling system then attaches anatomy, modality, and department tags to each item, which is what allows the fine-grained evaluation by medical subdomain.

What would settle it

Have independent clinicians answer and audit a random sample of, say, 200 MedBookVQA questions, including checking whether the official answer is correct, whether any distractor is also defensible, and whether the question can be answered from the text alone; if a substantial share fails any of those checks, the benchmark's accuracy numbers and the ranking built on them do not stand.

Watch

Extended reading notes

Core claim

The core discovery is that a standardized pipeline can turn medical textbook figures into a broad, clinically structured benchmark without large-scale human expert annotation. Starting from figures paired with captions and nearby descriptive sentences, the pipeline uses multimodal models to generate questions in five categories, converts them to multiple-choice form by generating distractors, filters out items that do not need the image or that fail other quality checks, and labels each item by anatomy, modality, and department. The resulting MedBookVQA benchmark contains 5,000 questions with non-repeating images from 1,103 open-access books. Evaluated on this set, the paper finds a consistent capability hierarchy: proprietary general multimodal models outperform open-source general, medical, and reasoning-oriented models, and there is a large performance gap between modality recognition and the four knowledge-heavy task types. The paper claims these disparities are real capability gaps in current general medical AI and that the anatomy, modality, and department labels make the gaps locally actionable.

Load-bearing premise

The load-bearing premise is that the machine-generated answers and distractors are correct and unambiguous, since the paper's own verification was manual spot-checking by the authors rather than review by medical experts; if systematic label errors exist, every reported accuracy and the resulting model ranking would be compromised.

Editorial extensions

If this is right

  • If the benchmark is valid, textbook-derived evaluation can be scaled to any medical specialty simply by finding open-access books with figures, without waiting for expert question writing.
  • The reported task-type gap implies that current multimodal models are strong at recognizing imaging modalities but much weaker at disease diagnosis, symptom recognition, and surgical reasoning, so those areas are the ones where clinical deployment is least safe today.
  • The finding that proprietary general models beat open medical specialist models suggests that medical-specialist training alone is not enough and that broad visual and reasoning competence may matter more.
  • The hierarchical labels let a model developer or hospital look up accuracy by modality, anatomy, or department and target data collection or fine-tuning to the weakest cells.
  • The benchmark's design as five balanced blocks of 1,000 questions makes future comparisons straightforward, since a new model's score can be read as accuracy per task type rather than one opaque aggregate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the construction pipeline is automated, the same recipe could be applied to other book sources or other domains, such as veterinary or dental education, to produce evaluation sets on demand; the paper does not itself make this extension.
  • Editorial inference: if the label quality survives expert audit, the benchmark could double as a training resource rather than only an evaluation resource, since each item carries a figure, a question, and a distractor set; the paper stops at evaluation.
  • Editorial inference: the reported hierarchy of proprietary over open general over medical over reasoning models is a snapshot from single-run evaluations without confidence intervals, so small accuracy differences between models should be treated cautiously.
  • Editorial inference: the choice of exactly 1,000 questions per task type means overall accuracy does not reflect clinical encounter frequencies; readers should compare models within a task type or label cell rather than treating the aggregate as a real-world skill score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces MedBookVQA, a 5,000-question multimodal medical VQA benchmark derived from open-access medical textbooks. The construction pipeline extracts figure–caption–text pairs from 8,090 books, uses InternVL2.5-78B to generate VQAs in five task types, uses Qwen-VL-Max to reformat them into multiple-choice questions with distractors, applies two automatic filters plus manual verification, and labels each item with hierarchical anatomy, modality, and department tags (125, 42, and 31 categories, respectively). The authors evaluate 40+ proprietary, open-source, medical, and reasoning MLLMs, reporting that proprietary general models outperform open-source general, medical, and reasoning models, with the largest gaps in disease and symptom diagnosis tasks. Data and code are released.

Significance. If the answer keys and labels are valid, MedBookVQA would be a welcome addition to the medical VQA ecosystem: it is the first general medical VQA benchmark built from open-access books, it has unusually broad modality/anatomy/department coverage, and the authors ship data, code, and detailed prompts that support reproduction and extension. The paper is also commendable for its explicit statement of the absence of error bars and the limitation that the data were not expert-verified. However, the headline numerical claims—task-type gaps and the model hierarchy—are only as sound as the machine-generated ground truth. Because the generator models appear among the evaluated models and the appendix already shows erroneous entries surviving all filters, the evaluative conclusions should be treated as provisional pending an expert audit. As a resource paper, the contribution is solid; as a benchmark for clinical-readiness claims, it needs validation.

major comments (4)
  1. [§3.2, §6, Appendix F] The ground-truth answers, distractors, and suitability judgments are generated by MLLMs (InternVL2.5-78B, Qwen-VL-Max, DeepSeek-R1) with manual verification by the authors only; Section 6 concedes that the data were 'not thoroughly verified by specialized experts.' The validity of the answer keys is load-bearing because every accuracy in Table 4 and the ranking in Figure 1 are computed against these keys. Appendix F provides concrete counterexamples: an item labeled as Modality Recognition asks 'What disease is shown in the image?' and lists modality names as options, and an OCT retina item is anatomically labeled 'Lymphatic vessel.' Both passed suitability, multimodality, and manual filters. An expert audit of a representative sample, with measured inter-rater agreement, is therefore required before the reported accuracies and the model hierarchy can be considered established.
  2. [§3.2, Table 4] The benchmark answers are generated by InternVL2.5-78B, which is itself evaluated in Table 4, and the distractor generator Qwen-VL-Max belongs to the same model family as the evaluated Qwen2.5-VL models. Reported accuracy therefore measures, in part, self-consistency with the generator's prior. For example, the 69.26% score of InternVL2.5-78B may overstate its clinical competence because its agreement with the answer key is expected to be higher than that of models with different inductive biases. The paper should quantify this by reporting per-item agreement between the generator and each evaluated model, or by re-scoring on a subset with expert labels.
  3. [NeurIPS Checklist item 7; §5.1.2] The paper discloses that no error bars or multiple runs were conducted (checklist item 7). Given 1,000 questions per type, the standard error of a proportion is about 1.5 percentage points, so small differences such as InternVL3-78B at 72.92% versus LLaVA-OV-72B at 72.86% in Table 4 are not statistically distinguishable. The paper should report confidence intervals and, for pairwise model comparisons, a paired test such as McNemar's test on the per-item outcomes.
  4. [§3.2, §4] The final benchmark picks exactly 1,000 VQAs per task type after the three filtering steps, but the paper does not report the per-type counts in the 6,971-item intermediate pool. If the five types were not equally represented after filtering, the equal-size selection imposes an arbitrary distribution that changes what the per-type accuracies mean relative to the source corpus; and the random selection among survivors may introduce selection bias relative to the original image distribution. Please report the per-type counts before selection and justify the equal-size design with respect to the intended clinical evaluation target.
minor comments (7)
  1. [Appendix B.1] The text 'FifOnly' should read 'FigOnly'.
  2. [Table 4, Figure 5] The model name 'LLaV A-OV' should be 'LLaVA-OV' throughout.
  3. [NeurIPS Checklist item 16] The word 'suplimentary' should be 'supplementary'.
  4. [Figure 6] Figure 6 is not explained in the text: the reader cannot tell what the numerical cells represent; please add a caption or legend.
  5. [§5.1.3] The statement that 'all the models' show weaknesses in Skin and Eye is stronger than what Figure 6 supports, since only 13 selected models are shown; please limit the claim to the displayed subset.
  6. [§2] The claim of 'pioneering the introduction of a reasoning benchmark in the general medical domain' should be softened to 'to our knowledge' and should cite or discuss existing medical reasoning benchmarks (e.g., MedVLM-R1) that could be seen as overlapping.
  7. [References] The references for DOAB, the Wikipedia taxonomy pages, and the Mayo Clinic departments are bare URLs; include version and access details.

Circularity Check

1 steps flagged · score 6.0 of 10

The benchmark's correctness key is generated by InternVL2.5-78B, which is then itself evaluated against that key, so reported accuracies partly measure self-consistency rather than independent medical knowledge.

  1. self definitional [Section 3.2 (Generate VQA and Reformat VQA to MC); Prompt C.2; Table 4]
    "InternVL2.5-78B [8] is prompted to generate up to 6 VQAs for each figure-information pair from five predefined VQA types ... The selected VQAs are reformatted into multiple-choice questions (MC) by generating distractors using Qwen-VL-Max [38]. ... Consider the original answer as the correct answer choice."

    The reference answer for every MedBookVQA item is the answer produced by InternVL2.5-78B during VQA generation, and the MC reformatting step explicitly designates that original answer as correct. InternVL2.5-78B is then itself evaluated on those MC items (Table 4, overall 69.26%). Its reported accuracy—and, for all models, the accuracy measure itself—is therefore the rate of agreement with InternVL2.5-78B's own generated answers, not with an externally established medical ground truth. Because the generator also saw the caption and FIGtexts when producing each answer, the evaluated model family receives a self-consistency advantage. The paper's headline rankings and task-gap conclusions are thus partly determined by this construction.

full rationale

The central evaluative claim—proprietary MLLMs outperform open-source, medical, and reasoning models, with large task-type gaps—rests entirely on the correctness labels of MedBookVQA. Those labels are not textbook ground truth; they are the answers generated by InternVL2.5-78B in Section 3.2, and the same answers are declared correct during MC reformatting (Prompt C.2). Since InternVL2.5-78B is itself among the evaluated models (Table 4), its score is a self-agreement rate, and every other model's score is an agreement rate with InternVL2.5-78B's answers. The textbook figures and captions supply independent source material, the public release enables external checking, and the suitability, multimodality, and manual author filters remove some defective items; Section 6 also concedes the data were 'not thoroughly verified by specialized experts.' These factors keep the circularity partial rather than total: the benchmark resource may still be useful, but the headline rankings and capability-gap conclusions are not yet independently established. Self-citation is not a significant factor here, since the authors' prior work is not load-bearing for the construction. A score of 6 reflects that the benchmark's scoring key reduces by construction to the output of one evaluated model.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The benchmark relies on a chain of automated tools: DOAB book selection, MinerU parsing, MLLM categorization and VQA generation, LLM filtering, and LLM labeling. Each step is an unverified premise because the ground truth is not established independently. There are no fitted numerical model parameters; the hand-set choices are the 5-word text sufficiency cutoff and the 1,000-per-type sampling cap. No new physical entities are postulated.

free parameters (2)
  • text sufficiency threshold = 5 words
    Section 3.1 discards figure-information pairs with fewer than 5 total words in FIGcaption plus FIGtexts; the threshold is chosen by hand and not sensitivity-analyzed.
  • per-VQA-type sample size = 1,000 per type (5,000 total)
    Section 3.2 randomly selects 1,000 of each VQA type after filtering; this balanced design is arbitrary and affects dataset composition.
assumptions (6)
  • domain assumption The Directory of Open Access Books search retrieves a representative set of medical textbooks whose figures and captions reflect current clinical knowledge.
    Section 3.1 relies on DOAB keyword search; if the book set is skewed or outdated, the coverage claims weaken.
  • domain assumption MinerU's layout parsing correctly pairs figures with captions and preserves reading order with sufficiently low error.
    Section 3.1 uses MinerU output for all downstream pairing; the caption-recovery pattern rules only fix a subset of failures.
  • ad hoc to paper InternVL2.5-78B generates accurate VQA answers from figure-text pairs, and Qwen-VL-Max generates valid, non-ambiguous distractors.
    Section 3.2 and appendix C rely on this premise; it is the main unverified step, acknowledged in Section 6 as lacking expert verification.
  • ad hoc to paper The LLM filtering steps, Qwen-VL-Max suitability and DeepSeek-R1 answerability-without-image, correctly identify bad or text-only questions.
    Section 3.2 Filter; errors in these steps propagate into the final 5,000-item set.
  • ad hoc to paper Qwen-VL-72B hierarchical labels for anatomy, modality, and department are accurate enough for fine-grained analysis.
    Section 3.3 and appendix C.4; label noise can distort subdomain performance comparisons.
  • domain assumption Evaluated models do not have systematic train-test contamination with the DOAB textbook figures, and including the generator model family in the eval set does not distort conclusions.
    Section 5; no contamination analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book." pith.science (2026). https://pith.science/paper/WYE5PKAL

@misc{pith2026250600855,
  author       = {Pith},
  title        = {Pith review of: MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYE5PKAL}},
  note         = {Machine review of arXiv:2506.00855}
}
read the original abstract

The accelerating development of general medical artificial intelligence (GMAI), powered by multimodal large language models (MLLMs), offers transformative potential for addressing persistent healthcare challenges, including workforce deficits and escalating costs. The parallel development of systematic evaluation benchmarks emerges as a critical imperative to enable performance assessment and provide technological guidance. Meanwhile, as an invaluable knowledge source, the potential of medical textbooks for benchmark development remains underexploited. Here, we present MedBookVQA, a systematic and comprehensive multimodal benchmark derived from open-access medical textbooks. To curate this benchmark, we propose a standardized pipeline for automated extraction of medical figures while contextually aligning them with corresponding medical narratives. Based on this curated data, we generate 5,000 clinically relevant questions spanning modality recognition, disease classification, anatomical identification, symptom diagnosis, and surgical procedures. A multi-tier annotation system categorizes queries through hierarchical taxonomies encompassing medical imaging modalities (42 categories), body anatomies (125 structures), and clinical specialties (31 departments), enabling nuanced analysis across medical subdomains. We evaluate a wide array of MLLMs, including proprietary, open-sourced, medical, and reasoning models, revealing significant performance disparities across task types and model categories. Our findings highlight critical capability gaps in current GMAI systems while establishing textbook-derived multimodal benchmarks as essential evaluation tools. MedBookVQA establishes textbook-derived benchmarking as a critical paradigm for advancing clinical AI, exposing limitations in GMAI systems while providing anatomically structured performance metrics across specialties.

Figures

Figures reproduced from arXiv: 2506.00855 by the authors.

Figure 1
Figure 1. Overall Performance of Multi-Modal Large Language Models on MedBookVQA. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline for data collection. Medical figure-information pairs are extracted from the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pipeline for benchmark construction. The benchmark is constructed from figure-information [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Hierarchical system of labels in terms of anatomy, modality, and department. A four-level [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance of MLLMs on the MedBookVQA benchmark, measured as the percentage [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance of high-performing MLLMs on common categories in anatomy, modality, and [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Prompt used to categorize figures in books into four categories via two methods: [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt used to collect FIGtexts, which are sentences that describe figures by explicitly mentioning the figure reference name. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prompt used to generate VQAs from a figure-information pair. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prompt used to reformat a VQA into MC. C.3 Prompt for Suitability Filtering and Multimodality Filtering Prompt for Suitability Filtering. You are a medical specialist with expertise in medical knowledge across various diseases. Based on the image, image-related descri…
Figure 11
Figure 11. Figure 11: Prompt used to filter VQA based on suitability. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Prompt used to filter VQA based on whether they require multimodal elements to answer. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Prompt used to label an entry in the benchmark according to anatomy. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Prompt used to label an entry in the benchmark according to modality. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Prompt used to label an entry in the benchmark according to department. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Distribution in terms of modality, with ModalityCategory as the inner level and Modality as the outer level. The innermost bar charts show the amount of each Modality, with minimum and maximum values displayed. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Distribution in terms of department, structured with [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Distribution in terms of anatomy. The levels, from inner to outer, are: [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RL Forgets! Towards Continual Policy Optimization

    cs.LG 2026-07 conditional novelty 6.5 of 10

    On a new post-2025 multimodal reasoning continual-learning benchmark, standard RL still forgets heavily; Continual Policy Optimization reduces forgetting 13.7% and lifts pretrained capability 7% on Qwen3-VL-8B via par...

Reference graph

Works this paper leans on

74 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL https://www.doabooks.org/

    Directory of open access books. URL https://www.doabooks.org/. Accessed: 2024-11-05

    work page 2024

  2. [2]

    Claude 3.7 sonnet system card

    Anthropic. Claude 3.7 sonnet system card. https://assets.anthropic.com/m/785e231869ea8b3b/ original/claude-3-7-sonnet-system-card.pdf , 2024. Accessed: 2025-05-11

    work page 2024

  3. [3]

    Bajwa, U

    J. Bajwa, U. Munir, A. Nori, and B. Williams. Artificial intelligence in healthcare: Transforming the practice of medicine. Future Healthcare Journal, 8(2):e188–e194, 2021. doi: 10.7861/fhj.2021-0095. URL https://pmc.ncbi.nlm.nih.gov/articles/PMC8285156/

    doi:10.7861/fhj.2021-0095 2021

  4. [4]

    Burgess, J

    J. Burgess, J. J. Nirschl, L. Bravo-Sánchez, A. Lozano, S. R. Gupte, J. G. Galaz-Montoya, Y . Zhang, Y . Su, D. Bhowmik, Z. Coman, S. M. Hasan, A. Johannesson, W. D. Leineweber, M. G. Nair, R. Yarlagadda, C. Zuraski, W. Chiu, S. Cohen, J. N. Hansen, M. D. Leonetti, C. Liu, E. Lundberg, and S. Yeung-Levy. Microvqa: A multimodal reasoning benchmark for micr...

    arXiv 2025

  5. [5]

    J. Chen, R. Ouyang, A. Gao, S. Chen, G. H. Chen, X. Wang, R. Zhang, Z. Cai, K. Ji, G. Yu, X. Wan, and B. Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale,

    work page

  6. [6]

    P. Chen, J. Ye, G. Wang, Y . Li, Z. Deng, W. Li, T. Li, H. Duan, Z. Huang, Y . Su, B. Wang, S. Zhang, B. Fu, J. Cai, B. Zhuang, E. J. Seibel, J. He, and Y . Qiao. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai, 2024. URL https://arxiv.org/abs/2408.03361

    arXiv 2024

  7. [7]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    work page 2024

  8. [8]

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y . Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y . Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, K. Zhang, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y . Qiao, J. Dai, and W. Wang. Exp...

    arXiv 2025

Show all 74 references
  1. [9]

    Chris, Y . Wei, Y . Peng, X. Wang, W. Qiu, W. Shen, T. Xie, J. Pei, J. Zhang, Y . Hao, X. Song, Y . Liu, and Y . Zhou. Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning, 2025. URL https://arxiv.org/abs/2504.16656

    2025 arXiv

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  3. [11]

    Gemini pro - google deepmind, 2025

    Google DeepMind. Gemini pro - google deepmind, 2025. URL https://deepmind.google/ technologies/gemini/pro/. Accessed: 2025-05-11

    2025

  4. [12]

    Y . Hao, J. Gu, H. W. Wang, L. Li, Z. Yang, L. Wang, and Y . Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark, 2025. URL https://arxiv.org/abs/2501. 05444

    2025

  5. [13]

    S. He, Y . Nie, H. Wang, S. Yang, Y . Wang, Z. Cai, Z. Chen, Y . Xu, L. Luo, H. Xiang, X. Lin, M. Wu, Y . Peng, G. Shih, Z. Xu, X. Wu, Q. Wang, R. C. K. Chan, V . Vardhanabhuti, W. C. W. Chu, Y . Zheng, P. Rajpurkar, K. Zhang, and H. Chen. Gsco: Towards generalizable ai in med...

    2024 arXiv

  6. [14]

    X. He, Y . Zhang, L. Mou, E. Xing, and P. Xie. Pathvqa: 30000+ questions for medical visual question answering, 2020. URL https://arxiv.org/abs/2003.10286

    2020 arXiv

  7. [15]

    Y . Hu, T. Li, Q. Lu, W. Shao, J. He, Y . Qiao, and P. Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm, 2024. URL https://arxiv.org/abs/2402.09181

    2024 arXiv

  8. [16]

    S. K. M. S. Islam, M. A. A. Nasim, I. Hossain, M. A. Ullah, K. D. Gupta, and M. M. H. Bhuiyan. Introduction of medical imaging modalities, 2023. URL https://arxiv.org/abs/2306.01022

    2023 arXiv

  9. [17]

    A. E. Kavur, N. S. Gezer, M. Barı¸ s, S. Aslan, P.-H. Conze, V . Groza, D. D. Pham, S. Chatterjee, P. Ernst, S. Özkan, B. Baydar, D. Lachinov, S. Han, J. Pauli, F. Isensee, M. Perkonigg, R. Sathish, R. Rajan, D. Sheet, G. Dovletov, O. Speck, A. Nürnberger, K. H. Maier-Hein, G....

    2021

  10. [18]

    J. Kil, Z. Mai, J. Lee, Z. Wang, K. Cheng, L. Wang, Y . Liu, A. Chowdhury, and W.-L. Chao. Mllm- compbench: A comparative reasoning benchmark for multimodal llms, 2025. URL https://arxiv. org/abs/2407.16837

    2025 arXiv

  11. [19]

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liu, and C. Li. Llava-onevision: Easy visual task transfer, 2024. URL https://arxiv.org/abs/2408.03326

    2024 arXiv

  12. [20]

    T. Lin, W. Zhang, S. Li, Y . Yuan, B. Yu, H. Li, W. He, H. Jiang, M. Li, X. Song, S. Tang, J. Xiao, H. Lin, Y . Zhuang, and B. C. Ooi. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation, 2025. URL h...

    2025

  13. [21]

    Liu, L.-M

    B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y . Yang, and X.-M. Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering, 2021. URL https://arxiv.org/abs/2102. 09542

    2021

  14. [22]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255

    2024 arXiv

  15. [23]

    S. Lu, Y . Li, Q.-G. Chen, Z. Xu, W. Luo, K. Zhang, and H.-J. Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797, 2024

    2024 arXiv

  16. [24]

    M. Moor, Q. Huang, S. Wu, M. Yasunaga, C. Zakka, Y . Dalmia, E. P. Reis, P. Rajpurkar, and J. Leskovec. Med-flamingo: a multimodal medical few-shot learner, 2023. URL https://arxiv.org/abs/2307. 15189

    2023

  17. [25]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L...

    2024 arXiv

  18. [26]

    J. Pan, C. Liu, J. Wu, F. Liu, J. Zhu, H. B. Li, C. Chen, C. Ouyang, and D. Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning,

  19. [27]

    Peng, Chris, X

    Y . Peng, Chris, X. Wang, Y . Wei, J. Pei, W. Qiu, A. Jian, Y . Hao, J. Pan, T. Xie, L. Ge, R. Zhuang, X. Song, Y . Liu, and Y . Zhou. Skywork r1v: Pioneering multimodal reasoning with chain-of-thought, 2025. URL https://arxiv.org/abs/2504.05599. 11

    2025 arXiv

  20. [28]

    A. L. Simpson, M. Antonelli, S. Bakas, M. Bilello, K. Farahani, B. van Ginneken, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers, P. Bilic, P. F. Christ, R. K. G. Do, M. Gollub, J. Golia-Pernicka, S. H. Heckers, W. R. Jarnagin, M. K. McHug...

    2019 arXiv

  21. [29]

    K. Team, A. Du, B. Yin, B. Xing, B. Qu, B. Wang, C. Chen, C. Zhang, C. Du, C. Wei, C. Wang, D. Zhang, D. Du, D. Wang, E. Yuan, E. Lu, F. Li, F. Sung, G. Wei, G. Lai, H. Zhu, H. Ding, H. Hu, H. Yang, H. Zhang, H. Wu, H. Yao, H. Lu, H. Wang, H. Gao, H. Zheng, J. Li, J. Su, J. Wa...

    2025 arXiv

  22. [30]

    B. Wang, C. Xu, X. Zhao, L. Ouyang, F. Wu, Z. Zhao, R. Xu, K. Liu, Y . Qu, F. Shang, B. Zhang, L. Wei, Z. Sui, W. Li, B. Shi, Y . Qiao, D. Lin, and C. He. Mineru: An open-source solution for precise document content extraction, 2024. URL https://arxiv.org/abs/2409.18839

    2024 arXiv

  23. [31]

    H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025

    2025 arXiv

  24. [32]

    H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning, 2025. URL https://arxiv.org/abs/2504. 08837

    2025

  25. [33]

    K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2024. URL https://openreview.net/...

    2024

  26. [34]

    X. Wang, Y . Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (C...

    2017 doi

  27. [35]

    Y . Wang, S. Wu, Y . Zhang, S. Yan, Z. Liu, J. Luo, and H. Fei. Multimodal chain-of-thought reasoning: A comprehensive survey, 2025. URL https://arxiv.org/abs/2503.12605

    2025 arXiv

  28. [36]

    C. Wu, W. Lin, X. Zhang, Y . Zhang, Y . Wang, and W. Xie. Pmc-llama: Towards building open-source language models for medicine, 2023. URL https://arxiv.org/abs/2304.14454

    2023 arXiv

  29. [37]

    Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y . Ma, C. Wu, B. Wang, Z. Xie, Y . Wu, K. Hu, J. Wang, Y . Sun, Y . Li, Y . Piao, K. Guan, A. Liu, X. Xie, Y . You, K. Dong, X. Yu, H. Zhang, L. Zhao, Y . Wang, and C. Ruan. Deepseek-vl2: Mixture-of-experts vision-langua...

    2024 arXiv

  30. [38]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y ...

    2024 arXiv

  31. [39]

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y . Liu, W. Huang, H. Sun, Y . Su, and W. Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning bench...

    2024 arXiv

  32. [40]

    Zhang, D

    R. Zhang, D. Jiang, Y . Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, P. Gao, and H. Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?, 2024. URL https://arxiv.org/abs/2403.14624

    2024 arXiv

  33. [41]

    Zhang, C

    X. Zhang, C. Wu, Z. Zhao, W. Lin, Y . Zhang, Y . Wang, and W. Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering, 2024. URL https://arxiv.org/abs/2305.10415

    2024 arXiv

  34. [42]

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y . Duan, W. Su, J. Shao, Z. Gao, E. Cui, X. Wang, Y . Cao, Y . Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, J. Wang, N. Deng, S. Li, Y . He, T. Jiang, J. Luo, Y . Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Y . Xion...

    2025 arXiv

  35. [45]

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes]

  36. [46]

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section 6

  37. [47]

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Benchmark and dataset paper

  38. [48]

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  39. [49]

    All our code and data are available to the general public

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Important prompts used are ...

  40. [50]

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: See section 5

  41. [51]

    We did not conduct multiple runs of the experiments due to the high computational cost

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: We evaluated a large number of multimodal large lan...

  42. [52]

    The computing resource requirements for these models vary and are beyond our control

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [No] Justification: Our experiments evaluate...

  43. [53]

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have verified that the copyrights of the medical books from which our data i...

  44. [54]

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See sections 6 and 7

  45. [55]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]

  46. [56]

    We use license CC BY 4.0

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have verified...

  47. [57]

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We include documentation when we release our dataset

  48. [58]

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  49. [59]

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  50. [60]

    Answer: [Yes] Justification: See section 3, where we describe how we used LLM in our method

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

  51. [61]

    List of Query Keywords

  52. [62]

    Prompts Used in Data Collection (a) Prompts for Categorization (b) Prompt for FIGtexts Collection

  53. [63]

    Prompts Used in Benchmark Construction (a) Prompt for VQA Generation (b) Prompt for Reformatting VQA to MC (c) Prompts for Suitability Filtering and Multimodality Filtering (d) Prompts for Labeling Benchmark

  54. [64]

    Distribution by Modality, Department, and Anatomy (a) Modality (b) Department (c) Anatomy

  55. [65]

    C": Charts of data results from medical research or experiments. (common types: data charts, tables, graphs, etc., to summarize data results) Category

    Sample Entries in MedBookVQA A List of Query Keywords The data for this study were sourced from medical books downloaded from the Directory of Open Access Books (DOAB). To effectively collect relevant books, we utilized the website’s keyword search feature with a comprehensive...

  56. [66]

    QA pair should be one of the following QAtypes (ONLY choose APPLICABLE ones): <type definition starts> {QAtypes} <type definition ends>

  57. [67]

    Please refer to the following details of each QAtypes: <type details starts> {QAtype_details} <type details ends>

  58. [68]

    M), question, answer

    For each pair provide: QAtpyeID(e.g. M), question, answer

  59. [69]

    Try to use several words or a short phrase

    Keep the "answer" as CONCISE as possible. Try to use several words or a short phrase

  60. [70]

    Only based on the information given above

  61. [71]

    Do NOT include any identifiers(i.e., figure name) in the generated QA pairs

  62. [72]

    Each question should be answerable for the case when ONLY the figure is given, which means do NOT ask the answerer to refer to any textual information in the question

  63. [73]

    Do NOT assume the answerer can access the textual information

  64. [74]

    right lung

    If there are multiple images, clearly specify which image the QA pair is referring to in the question part. Response ONLY in the following specific format (@#$ will be used for separation): { @@##$$ <QAtpyeID> @#$ <question> @#$ <answer> @@##$$ ... } Figure 9: Prompt used to g...

    2025

  65. [2024]

    URL https://arxiv.org/abs/2406.19280

    arXiv

  66. [2025]

    URL https://arxiv.org/abs/2502.19634

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.