Pith. sign in

REVIEW 3 major objections 4 minor 68 references

MEDIC-AD claims that injecting learnable <Ano> and <Diff> tokens into a 7B medical VLM, plus a heatmap decoder, yields state-of-the-art zero-shot anomaly detection, temporal symptom tracking, and anomaly segmentation, outperforming closed-s

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:12 UTC pith:36R3LCX2

load-bearing objection Sensible architecture, but the zero-shot evaluation is unverifiable because training and test sets may overlap — the empirical claims need major revision. the 3 major comments →

arxiv 2603.27176 v2 pith:36R3LCX2 submitted 2026-03-28 cs.CV

MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence

Woohyeon Park , Jaeik Kim , Sunghwan Steve Cho , Pa Hong , Wookyoung Jeong , Yoojin Nam , Namjoon Kim , Ginny Y. Wong
show 2 more authors
Ka Chun Cheung Jaeyoung Do
This is my paper
classification cs.CV
keywords medical VLManomaly detectiontemporal reasoningvisual groundinglearnable tokenszero-shotchest X-rayheatmap
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a medical vision-language model can be turned into a clinically useful tool by teaching it, in stages, to spot abnormalities, compare two scans over time, and point at the evidence. The proposed model, MEDIC-AD, adds two kinds of learnable tokens to a 7B medical VLM: tokens that encode lesion-related visual features, and tokens that encode what changed between a prior and a current image. A third stage trains a heatmap decoder so the model can localize the regions behind its answer. The paper reports that this design beats closed-source and medical-specialized baselines on zero-shot anomaly detection (average F1 91.6), symptom tracking on the MMXU benchmark (overall 0.655), and anomaly segmentation grounding (e.g., mIoU 79.8 on ChestX-Det vs 12.4 for Citrus-V). If right, it would mean a modest 7B model with roughly 128M additional parameters can deliver lesion detection, temporal monitoring, and visual explanations in a real clinical workflow.

Core claim

MEDIC-AD's central claim is that explicit, learnable anomaly tokens and difference tokens give a medical VLM the inductive biases it needs for three clinical tasks that generic VLMs handle poorly. Stage 1 produces <Ano> tokens by comparing abnormal and normal system tokens across multi-scale visual features, using the resulting anomaly attention map to modulate the visual features themselves; Stage 2 builds <Diff> tokens from those modulated features of two images through a separate Q-Former, so temporal change is separated from static context; Stage 3 fuses <Ano> tokens with visual features in a ConvNeXt head to generate heatmaps. The paper argues these stages mirror the clinical workflow—d

What carries the argument

The load-bearing mechanism is a pair of learnable sequence tokens inserted into the LLM's input. <Ano> tokens are generated by an anomaly processor: learnable abnormal/normal system tokens cross-attend to four intermediate vision-encoder layers, a sigmoid-based anomaly map modulates the visual features, and a Q-Former plus MLP compresses the result into a few tokens that the LLM reads alongside the image. <Diff> tokens are produced by a second Q-Former that compares the anomaly-modulated features of two images, isolating lesion-specific change. The heatmap decoder is the third piece: it fuses <Ano> tokens with visual features through a ConvNeXt segmentation head to output a spatial map. Toge

Load-bearing premise

The load-bearing premise is that the four zero-shot test datasets—Brain MRI, Head CT, Br35h, and COVID-19 X-ray—were truly absent from Stage 1 training; since Stage 1 trains on BMAD and ChestX-Det, which are broad medical anomaly collections that likely include those very datasets, the paper needs to show an explicit exclusion to keep the 91.6 F1 claim zero-shot.

What would settle it

Check the exact dataset composition of BMAD and ChestX-Det used in Stage 1: if Brain MRI, Head CT, Br35h, or the COVID-19 dataset appears in those training files, then re-run the zero-shot evaluation with those datasets removed. A drop back toward the 88.7 F1 of the backbone would disprove the zero-shot generalization claim. A second check is to perturb paired images with global illumination/translation and see whether <Diff> tokens still say 'no change'.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The reported 91.6 average zero-shot F1 across brain MRI, head CT, brain tumor, and COVID-19 X-ray suggests anomaly-token supervision transfers across modalities and diseases.
  • On the MMXU symptom-tracking benchmark, the 0.655 overall accuracy over the 0.620 backbone implies that explicit difference tokens improve temporal reasoning rather than just image concatenation.
  • The heatmap decoder's mIoU gains over a SAM2-based grounding baseline (e.g., 79.8 vs 12.4 on ChestX-Det) indicate that anomaly tokens can serve as a spatial grounding signal without a heavy segmentation model.
  • Because only small modules are updated (about 128M parameters), the approach is compatible with keeping general medical QA knowledge intact, as the appendix's comparisons to full fine-tuning suggest.
  • The real-hospital 300-patient follow-up study, if the evaluation methodology holds, points toward deployment in monitoring workflows where a radiologist reviews model-generated change captions and heatmaps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the zero-shot claim survives a clean dataset-overlap check, the same token recipe could be applied to other medical modalities with temporal pairs—such as longitudinal CT for tumor response or OCT for retinal disease—where current VLMs simply concatenate images.
  • The design suggests a general strategy: rather than fine-tuning a large medical VLM end-to-end, one can insert small task-specific token generators in front of a frozen LLM, which may be cheaper to adapt and easier to audit in regulated settings.
  • A natural stress test would be to feed the model paired images with artificially induced global shifts (brightness, translation) to see whether <Diff> tokens correctly report 'no change'; the paper's claim that it separates pathological change from acquisition noise is directly checkable this way.
  • The 300-patient hospital study is a single site; confirming the same stability across multiple institutions and scanner vendors would turn the usability claim into an operational one.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. MEDIC-AD is a 7B medical vision-language model built on Lingshu and augmented with three trainable modules: <Ano> anomaly tokens (Stage 1), <Diff> difference tokens for temporal comparisons (Stage 2), and a ConvNeXt-based heatmap head (Stage 3). The paper reports state-of-the-art results on zero-shot anomaly detection on Brain MRI, Head CT, Br35h, and COVID-19 chest X-ray (Table 1), on MMXU disease-progression tracking (Table 2), and on visual grounding/segmentation over BMAD and ChestX-Det (Table 3), together with a 300-patient hospital validation.

Significance. If the empirical claims are trustworthy, the staged token design is a useful step toward explainable medical VLMs: it explicitly separates abnormality, temporal difference, and grounding mechanisms while preserving general medical QA. The ablations, hyperparameter sensitivity study, and real-hospital evaluation are assets. However, the paper's key 'zero-shot' and grounding comparisons are currently unverifiable because training-set/test-set disjointness is not established. The contribution should be judged only after this issue is resolved.

major comments (3)
  1. [§3.2 / §4.1, Table 1] Stage 1 trains on 'BMAD, ChestX-Det' and §4.1 asserts the four test sets (Brain MRI, Head CT, Br35h, COVID-19) are 'entirely unseen during training.' BMAD is described in §2.2 as a consolidated medical anomaly detection benchmark covering Brain MRI and chest X-ray, and the four test datasets fall in exactly these modalities. The paper gives no BMAD subset manifest, no list of excluded patients/images, and no code to check overlap. If any of the test images or even patient cohorts appear in Stage 1, the average F1=91.6 is not zero-shot and the SOTA claim over Lingshu/Citrus-V is invalid. Please provide exact dataset components and explicit exclusion checks, or re-run with rigorously disjoint splits.
  2. [§4.3 / Stage 3, Table 3] Stage 3 is trained on 'selected subsets of BMAD and ChestX-Det' and evaluated in Table 3 on BraTS2021, RESC, BTCV+LiTs, and ChestX-Det. No train/validation/test split is described. The reported AUC/mIoU (e.g., BraTS2021 mIoU 87.6, RESC AUC 100) may therefore reflect in-distribution fitting rather than the ability to ground new anomalies. Specify the exact splits used for Stage 3, state whether the evaluation masks were seen during training, and report performance on a truly held-out subset.
  3. [Tables 1–3] All results are single point estimates; no standard deviations, confidence intervals, or significance tests are reported. This is especially important in Table 2, where the main gain is 0.655 vs 0.620, and in Table 1 where several F1 gaps between MEDIC-AD and Lingshu are small. Please report variance over at least three runs or bootstrap confidence intervals so the SOTA claims can be evaluated statistically.
minor comments (4)
  1. [Figure 5 / Table 9] The text says pooling size 4 and 10 soft prompts are selected, but Table 9 shows pooling size 8 gives avg F1 91.7 vs 91.6, and 5 soft prompts gives 91.8 vs 91.6. Please justify the choice or adjust the default.
  2. [§3.2] Stage 1 is described as optimized with the standard LLM cross-entropy loss, but it is unclear how the anomaly attention map and <Ano> tokens receive anomaly-specific supervision. Clarify whether image-level or pixel-level anomaly labels are used to train the cross-attention modules.
  3. [Table 4] The '<Diff> tokens only' row has no Avg. F1 value, so the ablation is incomplete for the anomaly-detection column. Please provide that number.
  4. [General] Minor typos and formatting issues: 'LLaV A-MED', 'explainablity', duplicated axis labels in Figure 5, and the inconsistent use of 'prior/current' vs 'baseline/follow-up' terminology.

Circularity Check

4 steps flagged

Stage 3 segmentation and Appendix VQA are evaluated on the same datasets used for training; the 'zero-shot' anomaly detection claim is not established because Stage 1 trains on BMAD/ChestX-Det without any documented exclusion of the test datasets.

specific steps
  1. fitted input called prediction [Section 3.2 (Stage 3 training) vs. Section 4.3 (Medical Visual Explainability evaluation)]
    "Stage 3 is trained on datasets with pixel-level segmentation masks, such as selected subsets of BMAD and ChestX-Det [7, 39]. / we evaluate MEDIC-AD on medical datasets that include pixel-level anomaly masks, specifically a subset of BMAD [7] (BraTS2021 [5], RESC [20], and BTCV + LiTs [9, 32]) and the ChestX-Det [39] dataset."

    The same named datasets (BraTS2021, RESC, BTCV+LiTs, ChestX-Det) are used as Stage 3 supervision targets and as Section 4.3 evaluation data. No train/test split, patient split, or held-out collection is described. The heatmap decoder is therefore scored on the exact ground-truth masks it was trained to produce. The Table 3 AUC/mIoU results are in-distribution fit scores, not independent evidence of explainability; the claimed grounding result reduces to the supervised segmentation training by construction.

  2. fitted input called prediction [Section 3.2 (Stage 1 training data) vs. Appendix C.1 / Table 7]
    "Training for this stage utilizes a diverse collection of medical anomaly datasets spanning MRI, X-ray, and CT modalities, including BMAD, ChestX-Det [7, 39], as well as multimodal VQA datasets such as SLAKE, PathVQA, and VQA-RAD [19, 33, 51]. / we assess the general medical reasoning capability of MEDIC-AD by benchmarking it on ... VQA-RAD [33], SLAKE [51], PathVQA [19], MMMU Med [62], PMC-VQA [63]"

    VQA-RAD, SLAKE, and PathVQA are explicitly included in Stage 1 training, and Appendix C.1 reports performance on those same benchmarks. No hold-out protocol is described. The claim that MEDIC-AD 'preserves broad medical knowledge' is evidenced by accuracy on its own training sets; those numbers measure fit to the training corpus, not independent retention of general knowledge.

  3. fitted input called prediction [Section 2.2 vs. Section 3.2 (Stage 1) vs. Section 4.1 / Table 1]
    "unified benchmarks such as BMAD [7] integrate diverse medical anomaly detection datasets—covering Brain MRI, Chest X-Ray, Liver CT, Retinal OCT, and Pathology—into a single evaluation framework. / Training for this stage utilizes ... including BMAD, ChestX-Det [7, 39]. / In the zero-shot anomaly detection setting, the model is tested on datasets that are entirely unseen during training ... Brain MRI, Head CT, Br35h, and COVID-19"

    The defining condition of the zero-shot evaluation is that test datasets are 'entirely unseen during training.' Stage 1 is trained on BMAD and ChestX-Det, and the paper's own description of BMAD says it consolidates brain-MRI and chest-X-ray anomaly datasets—the modalities of Brain MRI, Br35h, and COVID-19. The paper provides no BMAD/ChestX-Det subset manifest and never states that these four test datasets were excluded. If any of them are in the Stage 1 training collection, the Table 1 F1 values are in-distribution scores rather than zero-shot predictions, so the headline 'zero-shot SOTA' claim collapses into evaluating the training distribution. At minimum, the assertion of unseen evaluation is unsupported by the paper's own training description.

  4. fitted input called prediction [Section 5.3 / Figure 5 / Tables 1-2]
    "As illustrated in Fig. 5 (a), the model achieves consistently strong performance on both Anomaly Detection and MMXU benchmarks when the query-token pooling size is set to 4×4 ... using 10 visual soft prompts yields the most stable and competitive results. We therefore adopt a pooling size of 4×4 and 10 soft prompts as the default configuration for MEDIC-AD."

    Pooling size and soft-prompt count are selected by optimizing the same Avg F1 and MMXU (Overall) metrics that are subsequently reported as MEDIC-AD's headline results; no validation split is described. The default configuration is the one that maximized the reported metrics on these benchmarks, so the final numbers are selected values on the evaluation metric rather than independent predictions.

full rationale

The architectural components (<Ano> tokens, <Diff> tokens, heatmap decoder) are not themselves circular; the token mechanisms are well-specified and the ablations in Section 5.1 compare the proposed design against an ablated variant. The circularity is in the evaluation chain. Stage 3 is a direct train-on-the-test case: the segmentation head is trained on BraTS2021, RESC, BTCV+LiTs, and ChestX-Det and then evaluated on those same collections, so Table 3 measures in-distribution fit. Similarly, Stage 1 trains on VQA-RAD, SLAKE, and PathVQA, and Appendix C reports those same benchmarks as evidence of preserved medical knowledge. The most important claim, zero-shot anomaly detection, is undermined because Stage 1 trains on BMAD/ChestX-Det, BMAD is described as consolidating the same modality collections as the test sets, and no exclusion manifest or held-out split is provided. Hyperparameters are also chosen using the final evaluation metrics. Taken together, the paper's central empirical claims—zero-shot SOTA detection and explainable segmentation—reduce to fitting the training data or to an unverified test-set-exclusion assumption, rather than to independent prediction on unseen distributions.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 2 invented entities

The central claim rests on two free hyperparameters tuned on test metrics and on the unverified assumption that test datasets were excluded from training. No physical constants or standard mathematical axioms are involved. The invented entities are internal architectural tokens, not independently observable phenomena.

free parameters (2)
  • Pooling size for anomaly/diff tokens = 4x4
    Chosen by sensitivity analysis on Avg F1 and MMXU (Fig. 5a); this is effectively tuning on the test benchmarks.
  • Number of visual soft prompts = 10
    Chosen by sensitivity analysis on Avg F1 and MMXU (Fig. 5b); tuned on the test benchmarks.
axioms (3)
  • domain assumption Zero-shot test datasets are disjoint from the Stage 1 training corpora (BMAD, ChestX-Det).
    Asserted in Section 4.1 but never demonstrated; if false, all zero-shot results are invalid.
  • domain assumption Ground-truth labels and radiologist annotations used for evaluation are clinically correct.
    All metrics depend on labels from BMAD, ChestX-Det, MMXU, and the hospital dataset; no error analysis on label noise is provided.
  • domain assumption The pretrained Lingshu backbone transfers well to the new token modules.
    The method relies on a frozen pretrained medical VLM; the paper provides no independent evidence that this transfer is robust beyond the reported benchmarks.
invented entities (2)
  • <Ano> tokens no independent evidence
    purpose: Latent representations meant to encode lesion/anomaly semantics and condition the LLM and heatmap decoder.
    Learned internal tokens with no external falsifiable handle; their efficacy is only asserted through benchmark performance.
  • <Diff> tokens no independent evidence
    purpose: Latent representations of temporal differences between paired scans, meant to disentangle disease progression from noise.
    Learned internal tokens; no independent evidence outside the paper's own evaluations.

pith-pipeline@v1.3.0-alltime-deepseek · 16143 in / 12053 out tokens · 121848 ms · 2026-08-02T17:12:58.340745+00:00 · methodology

0 comments
read the original abstract

Lesion detection, symptom tracking, and visual explainability are central to real-world medical image analysis, yet current medical Vision-Language Models (VLMs) still lack mechanisms that translate their broad knowledge into clinically actionable outputs. To bridge this gap, we present MEDIC-AD, a clinically oriented VLM that strengthens these three capabilities through a stage-wise framework. First, learnable anomaly-aware tokens (<Ano>) encourage the model to focus on abnormal regions and build more discriminative lesion centered representations. Second, inter image difference tokens (<Diff>) explicitly encode temporal changes between studies, allowing the model to distinguish worsening, improvement, and stability in disease burden. Finally, a dedicated explainability stage trains the model to generate heatmaps that highlight lesion-related regions, offering clear visual evidence that is consistent with the model's reasoning. Through our staged design, MEDIC-AD steadily boosts performance across anomaly detection, symptom tracking, and anomaly segmentation, achieving state-of-the-art results compared with both closed source and medical-specialized baselines. Evaluations on real longitudinal clinical data collected from real hospital workflows further show that MEDIC-AD delivers stable predictions and clinically faithful explanations in practical patient-monitoring and decision-support workflows

Figures

Figures reproduced from arXiv: 2603.27176 by Ginny Y. Wong, Jaeik Kim, Jaeyoung Do, Ka Chun Cheung, Namjoon Kim, Pa Hong, Sunghwan Steve Cho, Woohyeon Park, Wookyoung Jeong, Yoojin Nam.

Figure 1
Figure 1. Figure 1: Overall performance of VLMs on Medical Anomaly [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of VLMs on clinical applications. Medic [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Architecture of MEDIC-AD. (a) Stage 1: <Ano> Token Generation, (b) Stage 2: <Diff> Token Generation, and (c) Stage 3: Heatmap Generation illustrate each stage of the proposed framework. Note that CA denotes Cross-Attention. sual justification, all of which are essential for trustworthy decision support. MEDIC-AD extends this paradigm into a clinically grounded reasoning framework through a stage-wise op￾ti… view at source ↗
Figure 4
Figure 4. Figure 4: Visual Grounding comparison between MEDIC-AD and Citrus-V [53] on diverse abnormal and normal samples. lumination or positional shifts. Unlike general VLMs that simply concatenate multiple images, MEDIC-AD explic￾itly encodes inter-image relationships, yielding consistent reasoning about temporal dynamics. In particular, quali￾tative inspection shows that MEDIC-AD highlights patho￾logical regions with true… view at source ↗
Figure 5
Figure 5. Figure 5: Hyperparameter sensitivity analysis on (a) query token [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of chat template between original and [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 27 linked inside Pith

  1. [1]

    In- fluence of prior radiologic information on the interpretation of radiographic examinations.Academic Radiology, 2(3): 205–208, 1995

    Uwa O Aideyan, Kevin Berbaum, and Wilbur L Smith. In- fluence of prior radiologic information on the interpretation of radiographic examinations.Academic Radiology, 2(3): 205–208, 1995. 2

    1995

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InCVPR, pages 2425– 2433, 2015. 1

    2015

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 2, 6

    Pith/arXiv arXiv 2025

  4. [4]

    Con- stitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Con- stitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022. 2, 6

    Pith/arXiv arXiv 2022

  5. [5]

    The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification

    Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314, 2021. 7

    Pith/arXiv arXiv 2021

  6. [6]

    Learning to exploit temporal structure for biomedical vision-language processing

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. InCVPR, pages 15016–15027,

  7. [7]

    Bmad: Benchmarks for medical anomaly detection.arXiv preprint arXiv:2306.11876, 2023

    Jinan Bao, Hanshi Sun, Hanqiu Deng, Yinsheng He, Zhaox- iang Zhang, and Xingyu Li. Bmad: Benchmarks for medical anomaly detection.arXiv preprint arXiv:2306.11876, 2023. 3, 5, 7

    Pith/arXiv arXiv 2023

  8. [8]

    The mvtec anomaly detection (mvtec ad) dataset: A comprehensive real-world dataset for unsuper- vised anomaly detection

    Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. The mvtec anomaly detection (mvtec ad) dataset: A comprehensive real-world dataset for unsuper- vised anomaly detection. Technical report, MVTec Software GmbH, 2021. 3

    2021

  9. [9]

    The liver tumor segmentation benchmark (lits).MIA, 84:102680, 2023

    Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene V orontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits).MIA, 84:102680, 2023. 7

    2023

  10. [10]

    Adaclip: Adapting clip with hybrid learnable prompts for zero-shot anomaly de- tection.arXiv preprint arXiv:2407.15795, 2024

    Yunkang Cao, Jiangning Zhang, Luca Frittoli, Yuqi Cheng, Weiming Shen, and Giacomo Boracchi. Adaclip: Adapting clip with hybrid learnable prompts for zero-shot anomaly de- tection.arXiv preprint arXiv:2407.15795, 2024. 3, 8

    Pith/arXiv arXiv 2024

  11. [11]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024. 1, 2, 6

    Pith/arXiv arXiv 2024

  12. [12]

    Pretraining vision-language model for difference visual question answering in longitudinal chest x-rays.arXiv preprint arXiv:2402.08966, 2024

    Yeongjae Cho, Taehee Kim, Heejun Shin, Sungzoon Cho, and Dongmyung Shin. Pretraining vision-language model for difference visual question answering in longitudinal chest x-rays.arXiv preprint arXiv:2402.08966, 2024. 3

    Pith/arXiv arXiv 2024

  13. [13]

    Muhammad E. H. Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muhammad Abdul Kadir, Zaid Bin Mahbub, Khandakar Reajul Islam, Muham- mad Salman Khan, Atif Iqbal, Nasser Al Emadi, Mamun Bin Ibne Reaz, and Mohammad Tariqul Islam. Can ai help in screening viral and covid-19 pneumonia?IEEE Access, 8:132665–132676, 2020. 6

    2020

  14. [14]

    Instructblip: Towards general-purpose vision- language models with instruction tuning.NeurIPS, 36: 49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning.NeurIPS, 36: 49250–49267, 2023. 1, 2

    2023

  15. [15]

    Francesco Dalla Serra, Patrick Schrempf, Chaoyang Wang, Zaiqiao Meng, Fani Deligianni, and Alison Q. O’Neil. Grounding chest x-ray visual question answering with gen- erated radiology reports.arXiv preprint arXiv:2505.16624,

    Pith/arXiv arXiv

  16. [16]

    Efros, and Jacob Steinhardt

    Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip’s image representation via text-based de- composition.arXiv preprint arXiv:2310.05916, 2023. 3

    Pith/arXiv arXiv 2023

  17. [17]

    Anomalygpt: Detecting in- dustrial anomalies using large vision-language models

    Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anomalygpt: Detecting in- dustrial anomalies using large vision-language models. In AAAI, pages 1932–1940, 2024. 2, 3, 6

    1932

  18. [18]

    Br35h: Brain tumor detection 2020, 2020

    Ahmed Hamada. Br35h: Brain tumor detection 2020, 2020. 6

    2020

  19. [19]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020. 3, 5, 1, 2

    Pith/arXiv arXiv 2003

  20. [20]

    Automated seg- mentation of macular edema in oct using deep neural net- works.MIA, 55:216–227, 2019

    Junjie Hu, Yuanyuan Chen, and Zhang Yi. Automated seg- mentation of macular edema in oct using deep neural net- works.MIA, 55:216–227, 2019. 7

    2019

  21. [21]

    Summers, and Yingying Zhu

    Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M. Summers, and Yingying Zhu. Expert knowledge- aware image difference graph representation learning for difference-aware medical visual question answering. InPro- ceedings of the 29th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining (KDD ’...

    2023

  22. [22]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In CVPR, pages 22170–22183, 2024. 3

    2024

  23. [23]

    Adapting visual-language models for generalizable anomaly detection in medical im- ages

    Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xin- chao Wang, and Yanfeng Wang. Adapting visual-language models for generalizable anomaly detection in medical im- ages. InCVPR, 2024. 3, 8

    2024

  24. [24]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 3, 6

    Pith/arXiv arXiv 2024

  25. [25]

    Truong, Du Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q.H. Truong, Du Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y . Ng, Cur- tis P. Langlotz, and Pranav Rajpurkar. Radgraph: Extracting clinical entities and relations from radiology reports.arXiv preprint arXiv:2106.14463, 2021. 3

    Pith/arXiv arXiv 2021

  26. [26]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. InECCV, pages 709–727. Springer,

  27. [27]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421,

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421,

  28. [28]

    Pubmedqa: A dataset for biomedical re- search question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical re- search question answering. InProceedings of the 2019 con- ference on empirical methods in natural language process- ing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2567–2577,

    2019

  29. [29]

    Brain tumor detec- tion using mri images.Brain, 3(2):146–150, 2015

    Pranita Balaji Kanade and PP Gumaste. Brain tumor detec- tion using mri images.Brain, 3(2):146–150, 2015. 6

    2015

  30. [30]

    Mmpb: It’s time for multi-modal personalization.arXiv preprint arXiv:2509.22820, 2025

    Jaeik Kim, Woojin Kim, Woohyeon Park, and Jaeyoung Do. Mmpb: It’s time for multi-modal personalization.arXiv preprint arXiv:2509.22820, 2025. 6

    arXiv 2025

  31. [31]

    Head ct - hemorrhage, 2018

    Felipe Campos Kitamura. Head ct - hemorrhage, 2018. 6

    2018

  32. [32]

    Miccai multi-atlas la- beling beyond the cranial vault–workshop and challenge

    Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein. Miccai multi-atlas la- beling beyond the cranial vault–workshop and challenge. InProc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, page 12, 2015. 7

    2015

  33. [33]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018. 3, 5, 1, 2

    2018

  34. [34]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2

    Pith/arXiv arXiv 2024

  35. [35]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day.NeurIPS, 36:28541–28564, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.NeurIPS, 36:28541–28564, 2023. 1, 2, 6

    2023

  36. [36]

    Selvaraju, Akhilesh D

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh D. Gotmare, Shafiq Joty, Caiming Xiong, and Steven C.H. Hoi. Align be- fore fuse: Vision and language representation learning with momentum distillation. InarXiv preprint arXiv:2107.07651,

    Pith/arXiv arXiv

  37. [37]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, pages 19730–19742. PMLR, 2023. 3

    2023

  38. [38]

    Deal: Disentan- gle and localize concept-level explanations for vlms.arXiv preprint arXiv:2407.14412, 2024

    Tang Li, Mengmeng Ma, and Xi Peng. Deal: Disentan- gle and localize concept-level explanations for vlms.arXiv preprint arXiv:2407.14412, 2024. 3

    Pith/arXiv arXiv 2024

  39. [39]

    A structure-aware relation network for thoracic diseases detection and segmentation

    Jie Lian, Jingyu Liu, Shu Zhang, Kai Gao, Xiaoqing Liu, Dingwen Zhang, and Yizhou Yu. A structure-aware relation network for thoracic diseases detection and segmentation. IEEE Transactions on Medical Imaging, 40(8):2042–2052,

    2042

  40. [40]

    Visual instruction tuning.NeurIPS, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 36:34892–34916, 2023. 1, 2, 3

    2023

  41. [41]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Ya- sunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Ed- uardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. InMachine Learning for Health (ML4H), pages 353–367. PMLR, 2023. 1

    2023

  42. [42]

    Mmxu: A multi-modal and multi-x-ray understanding dataset for disease progres- sion.arXiv preprint arXiv:2502.11651, 2025

    Linjie Mu, Zhongzhen Huang, Shengqian Qin, Yakun Zhu, Shaoting Zhang, and Xiaofan Zhang. Mmxu: A multi-modal and multi-x-ray understanding dataset for disease progres- sion.arXiv preprint arXiv:2502.11651, 2025. 1, 2, 6

    Pith/arXiv arXiv 2025

  43. [43]

    Green: Generative radiology report evalu- ation and error notation

    Ostmeier et al. Green: Generative radiology report evalu- ation and error notation. InEMNLP 2024, pages 374–390,

    2024

  44. [44]

    Medmcqa: A large-scale multi-subject multi- choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi- choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–

  45. [45]

    Kosmos-g: Generating images in context with multimodal large language models.arXiv preprint arXiv:2310.02992, 2023

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models.arXiv preprint arXiv:2310.02992, 2023. 3

    Pith/arXiv arXiv 2023

  46. [46]

    Describing and localizing multiple changes with transform- ers

    Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh. Describing and localizing multiple changes with transform- ers. InICCV, pages 1971–1980, 2021. 3

    1971

  47. [47]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. InCVPR, pages 13009–13018, 2024. 1

    2024

  48. [48]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 5, 7

    Pith/arXiv arXiv 2024

  49. [49]

    Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroen- sri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, C ´ıan Hughes, Charles Lau, et al. Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025. 2

    Pith/arXiv arXiv 2025

  50. [50]

    Xraygpt: Chest radiographs summarization using medical vision- language models.arXiv preprint arXiv:2306.07971, 2023

    Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullap- pilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision- language models.arXiv preprint arXiv:2306.07971, 2023. 1

    Pith/arXiv arXiv 2023

  51. [51]

    Towards gen- eralist biomedical ai.Nejm Ai, 1(3):AIoa2300138, 2024

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaeker- mann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards gen- eralist biomedical ai.Nejm Ai, 1(3):AIoa2300138, 2024. 2, 3, 5, 1

    2024

  52. [52]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. InCVPR, pages 3156–3164, 2015. 1

    2015

  53. [53]

    Citrus-v: Advancing medical foundation models with unified medical image grounding for clinical reasoning.arXiv preprint arXiv:2509.19090, 2025

    Guoxin Wang, Jun Zhao, Xinyi Liu, Yanbo Liu, Xuyang Cao, Chao Li, Zhuoyun Liu, Qintian Sun, Fangru Zhou, Hao- qiang Xing, et al. Citrus-v: Advancing medical foundation models with unified medical image grounding for clinical reasoning.arXiv preprint arXiv:2509.19090, 2025. 1, 2, 3, 5, 6, 7

    arXiv 2025

  54. [54]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. InEMNLP, page 3876, 2022. 2

    2022

  55. [55]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. InCVPR, pages 16133–16142, 2023. 5

    2023

  56. [56]

    Sorry-bench: Systemat- ically evaluating large language model safety refusal.arXiv preprint arXiv:2406.14598, 2024

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systemat- ically evaluating large language model safety refusal.arXiv preprint arXiv:2406.14598, 2024. 6

    Pith/arXiv arXiv 2024

  57. [57]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine.arXiv preprint arXiv:2408.02900, 2024

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine.arXiv preprint arXiv:2408.02900, 2024. 2

    Pith/arXiv arXiv 2024

  58. [58]

    Patel, and Isht Dwivedi

    Jiacong Xu, Shao-Yuan Lo, Bardia Safaei, Vishal M. Patel, and Isht Dwivedi. Towards zero-shot anomaly detection and reasoning with multimodal large language models.arXiv preprint arXiv:2502.07601, 2025. 2, 3, 6, 8

    Pith/arXiv arXiv 2025

  59. [59]

    Lingshu: A general- ist foundation model for unified multimodal medical under- standing and reasoning.arXiv preprint arXiv:2506.07044,

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A general- ist foundation model for unified multimodal medical under- standing and reasoning.arXiv preprint arXiv:2506.07044,

    Pith/arXiv arXiv

  60. [60]

    Improving visual grounding with visual- linguistic verification and iterative reasoning

    Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. Improving visual grounding with visual- linguistic verification and iterative reasoning. InCVPR, pages 9499–9508, 2022. 1

    2022

  61. [61]

    Image difference cap- tioning with pre-training and contrastive learning

    Linli Yao, Weiying Wang, and Qin Jin. Image difference cap- tioning with pre-training and contrastive learning. InAAAI, pages 3108–3116, 2022. 3

    2022

  62. [62]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. InCVPR, pages 9556–9567, 2024. 5, 1, 2

    2024

  63. [63]

    Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing.arXiv preprint arXiv:2305.10415, 2023

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing.arXiv preprint arXiv:2305.10415, 2023. 1, 2

    Pith/arXiv arXiv 2023

  64. [64]

    Ratescore: A metric for radiology report genera- tion

    Zhao et al. Ratescore: A metric for radiology report genera- tion. medrxiv, 2024. 8

    2024

  65. [65]

    Object-agnostic prompt learning for zero- shot anomaly detection.arXiv preprint arXiv:2310.18961,

    Q Zhou and et al. Object-agnostic prompt learning for zero- shot anomaly detection.arXiv preprint arXiv:2310.18961,

    arXiv

  66. [66]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 1, 2

    Pith/arXiv arXiv 2023

  67. [67]

    Spot-the-difference self-supervised pre- training for anomaly detection and segmentation

    Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the-difference self-supervised pre- training for anomaly detection and segmentation. InECCV, pages 392–408, 2022. 3

    2022

  68. [68]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025. 1, 2 MEDIC-AD: Towards Medical Vision-Language Model’s Clinical Intelligence Supplementary Material This supplementary document provide...

    Pith/arXiv arXiv 2025