{"id":"ffd39b36-2cb0-4717-ac6e-e5047400043d","arxiv_id":"2412.12661","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 1.47M-instance biomedical instruction-tuning dataset improves a mixed-modal 7B model's medical VQA accuracy by 18 to 26 percentage points over GPT-4o and Chameleon.","lead":"MedMax is a large collection of 1.47 million medical image-and-text examples used to teach a unified AI assistant to answer questions, describe images, generate images, and write reports. The associated model beats general-purpose systems like GPT-4o on 12 medical visual question-answering benchmarks, though the evaluation setup needs closer scrutiny.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline VQA gains are inflated by image-level leakage: training includes VQA-RAD/SLAKE/PathVQA/PMC-VQA train splits and 81K OmniMedVQA, while evaluation reuses the same datasets' test/hidden splits, so seen images drive several of the 12 averaged tasks.","rationale":"The paper's central claim is that MedMax instruction tuning produces a biomedical mixed-modal assistant that beats Chameleon and GPT-4o by large margins on 12 downstream VQA tasks. For that claim to hold as a generalization result, the evaluation tasks must be sufficiently disjoint from the training mixture. The reader identified QuiltVQA and OmniMedVQA as the main leakage risks; the present stress-test agrees and sharpens the concern: the largest per-task wins come from VQA-RAD, SLAKE, PathVQA, and PMC-VQA, whose official test sets are evaluated after their training splits were added to MedMax in Section 3.2. At least for VQA-RAD, the official split is question-level, so the same radiology images appear in both training and evaluation, allowing the model to exploit image-specific associations rather than demonstrate transfer. The paper's contamination analysis (Section 4.2) checks only exact image-text pair matches, which cannot detect this kind of overlap. This is load-bearing because it directly inflates the average used for the headline numbers and weakens the 'unseen instances' generalization claim made in Section 5.1. Credit is due for releasing the dataset, model, and code, for the scaling and ablation studies, and for evaluating a broad range of tasks; those contributions remain useful even if the benchmark-average magnifies the gains. The reader's CONDITIONAL verdict is therefore appropriate, and the recommended fix is to report image-level disjointness statistics and a clean-subset average. No further verdict change is needed.","tokens_in":22454,"tokens_out":5440,"duration_ms":50405,"concrete_test":"Compute image-level exact and near-duplicate overlap (e.g., exact file hashes plus pHash/dHash) between every image in the MedMax training set and the images in each of the 12 evaluation sets. Recompute the average VQA accuracy over only the tasks with zero image overlap and report per-task deltas. If the MedMax-vs-GPT-4o margin on this clean subset is substantially below the reported 18.3 points, the headline numbers are inflated by training/evaluation image overlap. As a supplement, re-split VQA-RAD (and, if needed, SLAKE and PathVQA) at the image level and retrain; a sharp accuracy drop would confirm that the reported gains rely on previously seen images.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline gain is not a clean measure of generalization, because the evaluation reuses benchmarks whose training splits (or images) are inside MedMax. Section 3.2 explicitly adds the training sets of VQA-RAD, SLAKE, PathVQA, PMC-VQA, and an 81K subset of OmniMedVQA to MedMax, while Section 4.2 and Table 2 evaluate on the test/hidden splits of those same datasets. For VQA-RAD the standard split is question-level, so the same images appear in both the MedMax training portion and the evaluation set; OmniMedVQA's self-created 1K test split is drawn from the same dataset as the 81K training split and likely shares images as well. QuiltVQA is derived from the same histopathology video source as the Quilt captioning and Quilt-Instruct data used in training. The contamination check in Section 4.2 only rules out exact image-text pair matches, not image-level overlap or near-duplicates. Five of the twelve averaged tasks are affected, and they include the largest per-task margins (OmniMedVQA 99.5 vs GPT-4o 40.9, SLAKE closed 88.4 vs 50.1, PathVQA closed 91.8 vs 59.2). The remaining tasks still favor MedMax but with smaller margins (e.g., PathMMU +10.2, ProbMed +27.5), so the specific magnitude of the 26.1-point gain over Chameleon and 18.3-point gain over GPT-4o is not established as a generalization result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedMax, a 1.47M-instance multimodal instruction-tuning dataset for biomedical mixed-modal foundation models, spanning VQA, image captioning and generation, visual chat, report understanding, and interleaved image-text generation. The authors fine-tune Anole/Chameleon-7B with LoRA on MedMax and report an average accuracy of 65.5% across twelve biomedical VQA tasks, outperforming Chameleon-7B (39.4%) and GPT-4o (42.0%) by 26.1 and 18.3 percentage points respectively. They also evaluate image captioning/generation, visual chat, and multimodal generation, and present ablations on data scaling, task mixture, and visual encoder finetuning. The central claim is that MedMax is an effective instruction-tuning resource for building unified biomedical multimodal assistants.","tokens_in":22811,"tokens_out":3277,"duration_ms":28818,"significance":"If the reported gains are genuine generalization effects, the paper would make a strong contribution: it provides a large, publicly released instruction-tuning dataset covering a wider range of biomedical multimodal tasks than prior resources, including a newly created interleaved text-image generation subset, and it demonstrates the feasibility of instruction-tuning a native mixed-modal foundation model in the biomedical domain. The dataset curation pipeline is detailed and the authors release data, model, and code, which is valuable for reproducibility. However, the evaluation design has a serious flaw that directly affects the headline numbers: several evaluation benchmarks share source datasets (and, in some cases, images) with the MedMax training data. This prevents the paper from establishing the claimed generalization improvement as stated.","major_comments":[{"comment":"","section":"§3.2 vs §4.2, Table 2"},{"comment":"","section":"§4.2"},{"comment":"","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"","section":"§3.1"},{"comment":"","section":"References"},{"comment":"","section":"§5.1 (Image captioning and generation)"},{"comment":"","section":"Appendix F"},{"comment":"","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the training/evaluation overlap, which directly affects the paper's central claim. I believe the issue is fixable within the scope of the manuscript: the authors can recompute the primary results on a strictly disjoint subset of benchmarks (e.g., PathMMU, ProbMed, and truly held-out splits) and clearly state the limitations of the existing evaluation. However, the current version should not be accepted without this re-analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for the dataset alone. MedMax is a large, carefully assembled 1.47M-instance instruction-tuning mixture for mixed-modal biomedical models, and the MedMax-Instruct piece — single-turn interleaved text-image generation conversations synthesized with GPT-4o — is a real novelty. No prior instruction-tuning dataset does this for biomedicine. The curation is thoughtful: filtering PMC-OA and LLaVA-Med-PMC to remove statistical figures and low-quality captions, balancing sources across radiology and histopathology, and converting PubMedVision descriptions into generation prompts. The model side is honest LoRA fine-tuning of Anole/Chameleon-7B, and the paper includes useful ablations on data scale and task composition. The release of data, model, and code is exactly what this subfield needs.\n\nThat said, the headline numbers overstate the generalization story. The stress-test note is not wrong: Section 3.2 explicitly puts VQA-RAD, SLAKE, PathVQA, PMC-VQA, and 81K of OmniMedVQA into training, while Section 4.2 evaluates on test/hidden splits of the same datasets. For VQA-RAD the standard split is question-level, so the same images appear on both sides. The OmniMedVQA hidden split is self-created from the same pool as the 81K used for training. QuiltVQA shares video sources with the Quilt data inside MedMax. Five of the twelve averaged tasks are contaminated, and they are the ones with the largest per-task margins (99.5 on OmniMedVQA, 88.4 on SLAKE closed, 91.8 on PathVQA closed). The exact-match-only contamination check in Section 4.2 does not rule out image-level overlap or near-duplicates. So the 26.1-point gain over Chameleon and 18.3-point gain over GPT-4o are not established as clean generalization results.\n\nThis is not fatal. The uncontaminated tasks still favor MedMax — PathMMU +10.2, ProbMed +27.5, QuiltVQA open +10.7 relative to Chameleon — and the visual chat, captioning, and multimodal generation results are not subject to the same leakage. The model is plausibly better. What needs fixing is the evaluation hygiene: proper held-out splits, semantic duplicate detection, and error bars or multiple runs. The open-ended VQA grading by GPT-4o-mini without human validation is a minor additional concern.\n\nWho should read this? Anyone building biomedical multimodal assistants and anyone doing instruction-tuning dataset construction. It deserves a serious referee — the dataset contribution is substantial and the evaluation flaws are fixable, not fundamental. I would accept it for review with a request for a cleaner held-out evaluation before publication.","headline":"A genuinely useful biomedical instruction-tuning dataset, but the headline VQA gains are inflated by benchmark leakage and need a cleaner held-out evaluation before the numbers are taken at face value.","tokens_in":23359,"tokens_out":1077,"would_cite":true,"duration_ms":10406,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedMax, a 1.47-million-instance biomedical instruction-tuning dataset, lifts a 7B open mixed-modal model to 65.5% average accuracy on twelve biomedical VQA tasks, 18.3 percentage points above GPT-4o and 26 points above the base Chameleon.","keywords":["mixed-modal foundation model","instruction tuning","biomedical visual question answering","multimodal generation","medical report understanding","biomedical assistant","dataset curation","evaluation suite"],"falsifier":"Compute a near-duplicate overlap between the MedMax training set and the QuiltVQA and OmniMedVQA evaluation sets (for example by perceptual image hashing and embedding-based question similarity), and re-measure the VQA accuracy after removing any training instance within a similarity threshold; if the 18–26 point gaps collapse, the central claim of generalization is falsified.","tokens_in":22266,"feed_emoji":"🩺","tokens_out":5960,"duration_ms":44923,"temperature":0.7,"pith_summary":"MedMax is a large-scale instruction-tuning dataset for native multimodal foundation models, containing 1.47 million instances that span visual question answering, image captioning and generation, visual chat, medical report understanding, and interleaved image-text generation across radiology and histopathology. The paper claims that instruction-tuning a 7B mixed-modal model (Anole, an instantiation of Chameleon) on this dataset raises its average accuracy on twelve biomedical VQA tasks to 65.5%, outperforming the base model by 26 percentage points and GPT-4o by 18.3 percentage points. The authors also introduce a unified evaluation suite covering these tasks, and report gains in image captioning, image generation, multimodal generation, and visual chat. If true, this shows that a carefully curated but largely existing data mix, plus a new GPT-generated interleaved-content component, can turn a generalist mixed-modal model into a competitive biomedical assistant without task-specific fine-tuning.","feed_headline":"MedMax lifts a 7B biomedical AI past GPT-4o on visual QA","feed_subtitle":"Instruction-tuning with 1.47M mixed-modal examples takes average accuracy across 12 tasks to 65.5%.","key_machinery":"The central object is the MedMax dataset itself, a mixture of curated sources (PMC-OA, Quilt-1M, LLaVA-Med, PubMedVision, VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, MIMIC-CXR) and a new MedMax-Instruct component of 88K interleaved image-text conversations produced with GPT-4o from caption-filtered pairs. The mechanism that transfers these data into skill is the instruction-tuning objective: given a sequence with text and VQGAN image tokens, the model maximizes likelihood of the response tokens conditioned on the instruction, with loss computed only on the response portion. Training uses LoRA on an autoregressive mixed-modal model (Anole-7B). The evaluation suite measures text tasks by exact match or GPT-4o-mini grading and generation tasks by BioMedCLIPScore.","core_discovery":"The paper's central claim is that the MedMax dataset is an effective instruction-tuning resource for mixed-modal foundation models in biomedicine. On the paper's own terms, fine-tuning Anole-7B (a Chameleon-7B instantiation) on MedMax yields an average accuracy of 65.5% across twelve biomedical VQA tasks, versus 39.4% for base Chameleon and 42.0% for GPT-4o; the model also outperforms LLaVA-Med and HuatuoGPT-Vision on average. The same fine-tuned model improves over the base by relative gains of 14–33% in image captioning, 14–100% in image generation, and 25.2%/31.5% on the text and image components of interleaved multimodal generation. The dataset includes 88K newly created interleaved image-text instruction instances (MedMax-Instruct) generated from filtered captions, alongside curated VQA, captioning, generation, visual chat, and report-understanding data. The authors frame this as a foundation for unified biomedical assistants that can both understand and generate multimodal content.","pith_inferences":["If the gains survive stricter contamination checks, the implication extends beyond biomedicine: the combination of large-scale caption curation and LLM-generated interleaved content may be a transferable recipe for other specialized multimodal domains where expert data are scarce.","The unusually high 99.5% accuracy on the hidden OmniMedVQA split suggests that when training and evaluation draw from the same distribution, the task becomes near-saturated; a more informative test of generalization would use benchmarks whose sources were not used in training at all.","One direct test of the paper's mechanism would be to replace the GPT-4o-generated MedMax-Instruct portion with randomly rewritten captions; if the interleaved generation gains vanish, then the structured multimodal response format itself is the active ingredient.","Because the paper uses only one backbone (Anole/Chameleon), it leaves open whether the gains transfer to other mixed-modal architectures such as Transfusion or Emu-3; showing that would separate the value of the data from the value of the model."],"forward_implications":["A single 7B instruction-tuned open model can outperform closed generalist models like GPT-4o on biomedical VQA, suggesting that domain-specific instruction-tuning data, not model scale, is the current bottleneck.","The same model checkpoint handles VQA, captioning, generation, visual chat, and interleaved report generation, so practitioners could replace several task-specific models with one unified assistant.","Performance on the twelve VQA tasks rises monotonically as MedMax is scaled from 25% to 100% of its data, indicating further dataset scaling should yield further gains.","Ablation results show removing VQA or visual-chat instances from the mix drops downstream performance by 23% and 17% respectively, so the task diversity in the mixture is what drives generalization.","The unified evaluation suite provides a reproducible protocol for comparing mixed-modal biomedical assistants, including new metrics for interleaved multimodal generation."],"supporting_citations":[{"why":"Provides the Chameleon-7B base model and the autoregressive token-based architecture that MedMax fine-tunes.","marker":"[34]"},{"why":"Anole-7B is the Chameleon instantiation used for all experiments, unlocking native image generation during output.","marker":"[9]"},{"why":"Quilt-1M supplies the histopathology image-caption data used for training and the QuiltVQA evaluation set.","marker":"[17]"},{"why":"LLaVA-Med contributes visual chat instruction data and the PMC filtering strategy that MedMax adopts.","marker":"[25]"},{"why":"OmniMedVQA contributes 81K training questions and the 1K hidden test split used in evaluation.","marker":"[15]"},{"why":"MIMIC-CXR provides the chest radiograph–report pairs used for report understanding and condition generation tasks.","marker":"[18]"},{"why":"PMC-OA is the source of radiology image-caption instances for captioning and generation training and evaluation.","marker":"[27]"},{"why":"PubMedVision supplies GPT-4-Vision–generated alignment and instruction data that MedMax curates into its mix.","marker":"[64]"},{"why":"VQA-RAD provides one of the twelve evaluation tasks and part of the VQA training set.","marker":"[22]"}],"fun_headline_variants":["MedMax dataset pushes 7B biomedical AI past GPT-4o","1.47M MedMax examples boost 7B bio model by 26%","MedMax training lifts 7B biomedical VQA over GPT-4o","MedMax tuned 7B model beats GPT-4o on 12 VQA tasks","MedMax fine-tuning gives 7B model 65.5% on biomedical VQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the evaluation benchmarks being sufficiently disjoint from MedMax's training data that the model must generalize rather than recall near-duplicate images or questions, yet the paper's contamination check only rules out exact image-text pair matches.","fun_headline_variants_meta":{"raw":{"variants":["MedMax dataset pushes 7B biomedical AI past GPT-4o","1.47M MedMax examples boost 7B bio model by 26%","MedMax training lifts 7B biomedical VQA over GPT-4o","MedMax tuned 7B model beats GPT-4o on 12 VQA tasks","MedMax fine-tuning gives 7B model 65.5% on biomedical VQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001392,"raw_usage":{"total_tokens":5661,"prompt_tokens":1000,"completion_tokens":4661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":4553}},"tokens_in":616,"tokens_out":4661,"duration_ms":29230,"temperature":1.0,"reasoning_tokens":4553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:50:19.011651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute a near-duplicate overlap between the MedMax training set and the QuiltVQA and OmniMedVQA evaluation sets (for example by perceptual image hashing and embedding-based question similarity), and re-measure the VQA accuracy after removing any training instance within a similarity threshold; if the 18–26 point gaps collapse, the central claim of generalization is falsified.","supporting_citations":[{"cited_title":"Quilt-1m: One million image-text pairs for histopathology","cited_arxiv_id":null,"evidence_quote":"Quilt-1M supplies the histopathology image-caption data used for training and the QuiltVQA evaluation set."},{"cited_title":"Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"LLaVA-Med contributes visual chat instruction data and the PMC filtering strategy that MedMax adopts."},{"cited_title":"Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm","cited_arxiv_id":null,"evidence_quote":"OmniMedVQA contributes 81K training questions and the 1K hidden test split used in evaluation."}],"review_version":1}