{"id":"58ca98f5-78a6-4ffa-adbf-ac3b2b110bea","arxiv_id":"2506.00855","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MedBookVQA is a new 5,000-question, textbook-derived multimodal benchmark for testing medical AI systems, with labels for imaging modality, body anatomy, and clinical specialty.","lead":"This paper introduces MedBookVQA, a set of 5,000 multiple-choice questions about medical images, built automatically from open-access medical textbooks. It is meant to test and compare AI systems that interpret medical images, and the authors find big differences in accuracy across question types and model families.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated MLLM-generated ground truth is the load-bearing risk: Appendix F already shows malformed questions and wrong anatomy labels, so reported accuracies and rankings could shift under expert audit.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing point: machine-generated answers without expert verification. My stress-test adds two concrete observations that strengthen it. First, the filtering pipeline is not merely unverified at scale; the published sample entries in Appendix F contain an object-level format error (a 'Modality Recognition' question that asks for a disease) and a label inconsistent with the correct answer (a retina item labeled 'Lymphatic vessel'). Both are in the released benchmark and passed the published filters, so the filters do not guarantee the semantic consistency the evaluation assumes. Second, the model used to generate ground truth (InternVL2.5-78B) and the model used to generate distractors (Qwen-VL-Max) are themselves evaluated or have evaluated family members, so the accuracy numbers are partly measuring consistency with the generator's priors. This does not make the dataset useless: the construction pipeline, textbook sourcing, and hierarchical taxonomy are independently valuable, and the per-model zero-shot evaluation is reproducible. But the central scientific claims — that current GMAI systems show specific capability gaps and that proprietary models lead — are only as sound as the answer keys. An expert audit of a few hundred items would settle this directly. The reader's CONDITIONAL verdict already encodes this, so no verdict change is needed; the condition should be made explicit in the paper and on the dataset card: reported accuracies should be accompanied by an audited label-accuracy estimate and, ideally, uncertainty intervals.","tokens_in":26890,"tokens_out":6033,"duration_ms":60964,"concrete_test":"Stratified expert audit: randomly sample 300 items (60 per VQA type), have two board-certified clinicians independently select the correct MC option from image and question, blind to the published key, then adjudicate disagreements. Compute (a) agreement with the published key and (b) ambiguity rate. Re-score all models on the subset of items with confirmed-correct, unambiguous keys and re-rank. If any top-2 pairwise ordering changes (e.g., Gemini2.5-Pro vs GPT4.1, currently 3.26 points apart on 5,000 items) or any headline task-type gap shifts by more than 5 percentage points, the evaluation claims should be revised or re-reported on the audited subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's headline numbers are agreement scores against answer keys produced by InternVL2.5-78B (questions/answers) and Qwen-VL-Max (distractors) in Section 3.2, filtered by DeepSeek-R1 and the authors, and Section 6 concedes the data were 'not thoroughly verified by specialized experts.' If a nontrivial fraction of keys are wrong or ambiguous, every reported accuracy and the claimed hierarchy (proprietary > open general > medical > reasoning) are not yet established. The paper's own Appendix F sample entries show this is not a hypothetical risk: one item labeled Modality Recognition asks 'What disease is shown in the image?' with modality choices, and an OCT retina item is anatomically labeled 'Lymphatic vessel.' Both passed suitability, multimodality, and manual filters. Because the generator is also an evaluated model family (InternVL2.5-78B appears in Table 4, Qwen2.5-VL variants are evaluated), agreement may reflect shared biases rather than clinical competence. The benchmark resource itself is useful, but the evaluative claims are conditional on label validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedBookVQA, a 5,000-question multimodal medical VQA benchmark derived from open-access medical textbooks. The construction pipeline extracts figure–caption–text pairs from 8,090 books, uses InternVL2.5-78B to generate VQAs in five task types, uses Qwen-VL-Max to reformat them into multiple-choice questions with distractors, applies two automatic filters plus manual verification, and labels each item with hierarchical anatomy, modality, and department tags (125, 42, and 31 categories, respectively). The authors evaluate 40+ proprietary, open-source, medical, and reasoning MLLMs, reporting that proprietary general models outperform open-source general, medical, and reasoning models, with the largest gaps in disease and symptom diagnosis tasks. Data and code are released.","tokens_in":27188,"tokens_out":5482,"duration_ms":47784,"significance":"If the answer keys and labels are valid, MedBookVQA would be a welcome addition to the medical VQA ecosystem: it is the first general medical VQA benchmark built from open-access books, it has unusually broad modality/anatomy/department coverage, and the authors ship data, code, and detailed prompts that support reproduction and extension. The paper is also commendable for its explicit statement of the absence of error bars and the limitation that the data were not expert-verified. However, the headline numerical claims—task-type gaps and the model hierarchy—are only as sound as the machine-generated ground truth. Because the generator models appear among the evaluated models and the appendix already shows erroneous entries surviving all filters, the evaluative conclusions should be treated as provisional pending an expert audit. As a resource paper, the contribution is solid; as a benchmark for clinical-readiness claims, it needs validation.","major_comments":[{"comment":"The ground-truth answers, distractors, and suitability judgments are generated by MLLMs (InternVL2.5-78B, Qwen-VL-Max, DeepSeek-R1) with manual verification by the authors only; Section 6 concedes that the data were 'not thoroughly verified by specialized experts.' The validity of the answer keys is load-bearing because every accuracy in Table 4 and the ranking in Figure 1 are computed against these keys. Appendix F provides concrete counterexamples: an item labeled as Modality Recognition asks 'What disease is shown in the image?' and lists modality names as options, and an OCT retina item is anatomically labeled 'Lymphatic vessel.' Both passed suitability, multimodality, and manual filters. An expert audit of a representative sample, with measured inter-rater agreement, is therefore required before the reported accuracies and the model hierarchy can be considered established.","section":"§3.2, §6, Appendix F"},{"comment":"The benchmark answers are generated by InternVL2.5-78B, which is itself evaluated in Table 4, and the distractor generator Qwen-VL-Max belongs to the same model family as the evaluated Qwen2.5-VL models. Reported accuracy therefore measures, in part, self-consistency with the generator's prior. For example, the 69.26% score of InternVL2.5-78B may overstate its clinical competence because its agreement with the answer key is expected to be higher than that of models with different inductive biases. The paper should quantify this by reporting per-item agreement between the generator and each evaluated model, or by re-scoring on a subset with expert labels.","section":"§3.2, Table 4"},{"comment":"The paper discloses that no error bars or multiple runs were conducted (checklist item 7). Given 1,000 questions per type, the standard error of a proportion is about 1.5 percentage points, so small differences such as InternVL3-78B at 72.92% versus LLaVA-OV-72B at 72.86% in Table 4 are not statistically distinguishable. The paper should report confidence intervals and, for pairwise model comparisons, a paired test such as McNemar's test on the per-item outcomes.","section":"NeurIPS Checklist item 7; §5.1.2"},{"comment":"The final benchmark picks exactly 1,000 VQAs per task type after the three filtering steps, but the paper does not report the per-type counts in the 6,971-item intermediate pool. If the five types were not equally represented after filtering, the equal-size selection imposes an arbitrary distribution that changes what the per-type accuracies mean relative to the source corpus; and the random selection among survivors may introduce selection bias relative to the original image distribution. Please report the per-type counts before selection and justify the equal-size design with respect to the intended clinical evaluation target.","section":"§3.2, §4"}],"minor_comments":[{"comment":"The text 'FifOnly' should read 'FigOnly'.","section":"Appendix B.1"},{"comment":"The model name 'LLaV A-OV' should be 'LLaVA-OV' throughout.","section":"Table 4, Figure 5"},{"comment":"The word 'suplimentary' should be 'supplementary'.","section":"NeurIPS Checklist item 16"},{"comment":"Figure 6 is not explained in the text: the reader cannot tell what the numerical cells represent; please add a caption or legend.","section":"Figure 6"},{"comment":"The statement that 'all the models' show weaknesses in Skin and Eye is stronger than what Figure 6 supports, since only 13 selected models are shown; please limit the claim to the displayed subset.","section":"§5.1.3"},{"comment":"The claim of 'pioneering the introduction of a reasoning benchmark in the general medical domain' should be softened to 'to our knowledge' and should cite or discuss existing medical reasoning benchmarks (e.g., MedVLM-R1) that could be seen as overlapping.","section":"§2"},{"comment":"The references for DOAB, the Wikipedia taxonomy pages, and the Mayo Clinic departments are bare URLs; include version and access details.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The resource and pipeline are valuable, but the evaluative claims hinge on label validity. I recommend that the editor require an expert audit (e.g., a random sample of at least 200 items per type, with agreement statistics) and a discussion of generator-evaluator overlap before the benchmark is cited for model comparisons. The paper also should clarify the CC BY 4.0 license status of the original book images, since the dataset redistributes textbook figures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MedBookVQA is a genuine resource—5,000 MCQ items from open-access medical textbooks, with five task types, a three-axis hierarchical labeling system, and a concrete public release. The pipeline is described in enough detail to reproduce, the filtering steps are sensible, and the evaluation spans 40 models. That is real engineering work.\n\nBut the paper's central claim—that these are 'clinically relevant questions' and that the reported accuracies reveal capability gaps—sits on top of answer keys generated by InternVL2.5-78B and Qwen-VL-Max, filtered by DeepSeek-R1 and the authors, and never audited by medical experts. Section 6 concedes the lack of expert verification. Appendix F shows the risk is not hypothetical: one item under 'Modality Recognition' asks 'What disease is shown in the image?' with modality choices, and an OCT retina image is labeled 'Lymphatic vessel' under anatomy. Both passed the suitability, multimodality, and manual filters. If even a few percent of the 5,000 items have wrong or ambiguous keys, the rankings in Figure 1 shift, and the hierarchy 'proprietary > open general > medical > reasoning' is not established.\n\nThe circularity concern is real too. The generator (InternVL2.5-78B) is also an evaluated model. Its 69.26% overall score is partly a measure of how well it agrees with itself. That doesn't make the benchmark worthless, but it undercuts any reading of the leaderboard as independent capability comparison.\n\nLesser but real: NeurIPS checklist says no error bars or multiple runs. The evaluation is single-run, temperature-zero, and 1,000 items per task; a few dozen questions can move a percentage point.\n\nWhat's solid: the textbook-derived grounding, the public dataset and code, the clear pipeline documentation, and the hierarchical labels that let people filter by anatomy, modality, or department. Those are worth having regardless of the label issues. If an independent expert audit of a few hundred items comes back clean, the resource becomes genuinely useful. If it doesn't, the paper still stands as a pipeline contribution, but the leaderboard claims should be walked back.\n\nMy recommendation: send it to peer review. It deserves a serious referee, and the authors have the data and infrastructure to fix the biggest issue—a public expert audit. The paper should not be accepted in its current form; the evaluative claims need to be conditional on label verification, and the abstract should not assert clinical relevance without the audit. But desk-rejecting it would throw away a potentially useful benchmark and a well-documented pipeline.","headline":"A useful textbook-derived benchmark with transparent plumbing, but the headline rankings ride on unverified machine-generated answer keys and the paper's own samples show the keys leak.","tokens_in":27651,"tokens_out":3228,"would_cite":false,"duration_ms":29668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Medical textbook figures can be turned into a 5,000-question benchmark for testing medical AI.","keywords":["medical visual question answering","multimodal large language models","benchmark construction","open-access medical textbooks","hierarchical annotation","modality recognition","disease diagnosis","clinical AI evaluation"],"falsifier":"Have independent clinicians answer and audit a random sample of, say, 200 MedBookVQA questions, including checking whether the official answer is correct, whether any distractor is also defensible, and whether the question can be answered from the text alone; if a substantial share fails any of those checks, the benchmark's accuracy numbers and the ranking built on them do not stand.","tokens_in":26690,"feed_emoji":"🩺","tokens_out":6661,"duration_ms":63096,"temperature":0.7,"pith_summary":"MedBookVQA asks whether open-access medical textbooks, rather than research articles or repackaged classification datasets, can serve as a trustworthy source for evaluating general medical AI. The paper constructs 5,000 multiple-choice visual-question-answering items from more than a thousand medical books, spanning five question types and a hierarchical taxonomy of 42 imaging modalities, 125 anatomical structures, and 31 clinical departments. It then measures several dozen multimodal large language models on the set. The paper's central claim is that textbook-derived benchmarks are a viable and instructive evaluation paradigm, and its reported results say that proprietary general-purpose models currently lead, that medically specialized open models lag behind, and that the largest weaknesses appear in disease diagnosis, symptom recognition, and surgery rather than in simple modality identification.","feed_headline":"Textbook figures become a 5,000-question medical AI exam","feed_subtitle":"A benchmark built from open-access books shows proprietary general models lead, while disease diagnosis and surgery remain weak.","key_machinery":"The load-bearing mechanism is the figure-information pairing and question-generation pipeline. Document parsing divides book pages into regions and attaches captions to images; a classifier keeps only real-case medical figures; pattern matching recovers figure reference names and collects nearby sentences that mention them; a large multimodal model then writes up to six questions per figure from five predefined types; a second multimodal model turns each question into a four-option multiple-choice item; and three filters, suitability, image-necessity, and manual review, remove off-topic, text-only-answerable, and flawed entries. A hierarchical labeling system then attaches anatomy, modality, and department tags to each item, which is what allows the fine-grained evaluation by medical subdomain.","core_discovery":"The core discovery is that a standardized pipeline can turn medical textbook figures into a broad, clinically structured benchmark without large-scale human expert annotation. Starting from figures paired with captions and nearby descriptive sentences, the pipeline uses multimodal models to generate questions in five categories, converts them to multiple-choice form by generating distractors, filters out items that do not need the image or that fail other quality checks, and labels each item by anatomy, modality, and department. The resulting MedBookVQA benchmark contains 5,000 questions with non-repeating images from 1,103 open-access books. Evaluated on this set, the paper finds a consistent capability hierarchy: proprietary general multimodal models outperform open-source general, medical, and reasoning-oriented models, and there is a large performance gap between modality recognition and the four knowledge-heavy task types. The paper claims these disparities are real capability gaps in current general medical AI and that the anatomy, modality, and department labels make the gaps locally actionable.","pith_inferences":["Editorial inference: because the construction pipeline is automated, the same recipe could be applied to other book sources or other domains, such as veterinary or dental education, to produce evaluation sets on demand; the paper does not itself make this extension.","Editorial inference: if the label quality survives expert audit, the benchmark could double as a training resource rather than only an evaluation resource, since each item carries a figure, a question, and a distractor set; the paper stops at evaluation.","Editorial inference: the reported hierarchy of proprietary over open general over medical over reasoning models is a snapshot from single-run evaluations without confidence intervals, so small accuracy differences between models should be treated cautiously.","Editorial inference: the choice of exactly 1,000 questions per task type means overall accuracy does not reflect clinical encounter frequencies; readers should compare models within a task type or label cell rather than treating the aggregate as a real-world skill score."],"forward_implications":["If the benchmark is valid, textbook-derived evaluation can be scaled to any medical specialty simply by finding open-access books with figures, without waiting for expert question writing.","The reported task-type gap implies that current multimodal models are strong at recognizing imaging modalities but much weaker at disease diagnosis, symptom recognition, and surgical reasoning, so those areas are the ones where clinical deployment is least safe today.","The finding that proprietary general models beat open medical specialist models suggests that medical-specialist training alone is not enough and that broad visual and reasoning competence may matter more.","The hierarchical labels let a model developer or hospital look up accuracy by modality, anatomy, or department and target data collection or fine-tuning to the weakest cells.","The benchmark's design as five balanced blocks of 1,000 questions makes future comparisons straightforward, since a new model's score can be read as accuracy per task type rather than one opaque aggregate."],"supporting_citations":[{"why":"Supplies the open-access medical books that form the raw source of all figures and questions.","marker":"[1]"},{"why":"Provides the document parser that splits pages into regions and pairs images with captions.","marker":"[30]"},{"why":"Generates the candidate VQA questions from each figure-information pair.","marker":"[8]"},{"why":"Produces the multiple-choice distractors and also assigns the hierarchical labels.","marker":"[38]"},{"why":"Filters out questions answerable without the image, enforcing the benchmark's multimodality requirement.","marker":"[10]"},{"why":"Provides the lexico-semantic hierarchical labeling convention that MedBookVQA adapts for anatomy, modality, and department.","marker":"[6]"},{"why":"Supplies a prominent open-source medical baseline and the PubMed-derived training data whose absence is used to explain weak performance on some image types.","marker":"[5]"},{"why":"Supplies a medical reasoning baseline whose limited gain motivates the paper's conclusion that current reasoning enhancements are insufficient for complex medical tasks.","marker":"[26]"}],"fun_headline_variants":["Book-derived benchmark exposes AI medical blind spots","Medical AI tested on 5,000 textbook questions","Open-book exam for medical AI reveals skill gaps","Textbook-based VQA benchmark challenges medical AI","From textbooks to test: 5,000 medical AI questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the machine-generated answers and distractors are correct and unambiguous, since the paper's own verification was manual spot-checking by the authors rather than review by medical experts; if systematic label errors exist, every reported accuracy and the resulting model ranking would be compromised.","fun_headline_variants_meta":{"raw":{"variants":["Book-derived benchmark exposes AI medical blind spots","Medical AI tested on 5,000 textbook questions","Open-book exam for medical AI reveals skill gaps","Textbook-based VQA benchmark challenges medical AI","From textbooks to test: 5,000 medical AI questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000403,"raw_usage":{"total_tokens":2132,"prompt_tokens":1012,"completion_tokens":1120,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":1046}},"tokens_in":628,"tokens_out":1120,"duration_ms":8126,"temperature":1.0,"reasoning_tokens":1046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:56:40.188109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent clinicians answer and audit a random sample of, say, 200 MedBookVQA questions, including checking whether the official answer is correct, whether any distractor is also defensible, and whether the question can be answered from the text alone; if a substantial share fails any of those checks, the benchmark's accuracy numbers and the ranking built on them do not stand.","supporting_citations":[{"cited_title":"URL https://www.doabooks.org/","cited_arxiv_id":null,"evidence_quote":"Supplies the open-access medical books that form the raw source of all figures and questions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a prominent open-source medical baseline and the PubMed-derived training data whose absence is used to explain weak performance on some image types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a medical reasoning baseline whose limited gain motivates the paper's conclusion that current reasoning enhancements are insufficient for complex medical tasks."}],"review_version":1}