Pith. sign in

REVIEW 4 major objections 5 minor 65 references

Current multimodal models struggle to reason about disease change across serial medical images; a 206K-pair multi-task benchmark and a fine-tuned 8B model establish the gap and a strong baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 01:33 UTC pith:3XIJOFR5

load-bearing objection Solid multi-task longitudinal CXR VQA resource with real engineering value; the main caveat is that many labels are report-trajectory-derived, so weak MLLM scores partly mix visual failure with text-prior mismatch. the 4 major comments →

arxiv 2607.27806 v1 pith:3XIJOFR5 submitted 2026-07-30 cs.CV

LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

Zhilin Wu , Zhangkai Ni , Chengmei Yang , Longzhen Yang , Yihang Liu , Ying Wen , Lianghua He This is my paper
classification cs.CV
keywords longitudinal medical VQAmultimodal large language modelsdisease progressionchest X-raytemporal reasoningmedical benchmarkvisual groundingreport generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Clinicians routinely compare a patient’s scans over successive visits to judge whether disease is improving, stable, or worsening. Most medical vision-language models still answer questions about a single image, so that real clinical skill is untested. This paper builds LoMeVQA, a large longitudinal VQA benchmark with five tasks that demand temporal comparison, free-form progression language, multi-visit reports, and localization of what actually changed. An automated pipeline groups MIMIC-CXR studies by patient and time, extracts entities with a radiology knowledge graph, builds entity trajectories, and has an LLM write grounded questions and answers, then filters them. Both general and medical MLLMs score poorly; fine-tuning Qwen3-VL on the development split yields MedLong-8B, which leads on every task and also transfers to two external longitudinal sets. The result is a concrete map of where temporal medical visual reasoning fails and a usable training and evaluation resource for fixing it.

Core claim

Existing general-purpose and medical multimodal large language models have substantial limitations on longitudinal medical visual reasoning. LoMeVQA—206K VQA pairs spanning progress classification, progress description, progress report generation, differential region grounding, and differential region description—exposes that gap (best off-the-shelf models near 55% on classification and about 25% IoU on grounding). Supervised fine-tuning of Qwen3-VL-8B on LoMeVQA-dev produces MedLong-8B, which reaches state-of-the-art on all five in-distribution tasks and improves out-of-distribution accuracy on MMXU-test and MIMIC-CXR-T.

What carries the argument

The construction pipeline: chronological patient episodes from MIMIC-CXR, RadGraph entity-and-sentence extraction, entity-level temporal trajectories, sliding-window temporal and entity sampling to control difficulty, LLM synthesis of VQA pairs anchored to those trajectories, plus diversity and LLM consistency filtering (and clinician review for the 2,500-sample test set). That machinery yields the five-task LoMeVQA benchmark and the MedLong-8B fine-tune.

Load-bearing premise

Labels built from report-derived entity trajectories and LLM writing, even after automatic consistency checks and limited clinician review of the test split, faithfully reflect true image-level temporal change rather than report wording or leftover generation errors.

What would settle it

A blinded radiologist study that scores LoMeVQA answers against the images alone (ignoring source reports): if disagreement is high—especially on the three-way progress-classification items whose automatic consistency pass rate is only about 74%—or if MedLong’s gains disappear on a purely image-verified longitudinal hold-out, the central benchmark and SOTA claims weaken.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Longitudinal comparison, not single-image VQA, becomes a required axis when claiming medical MLLM competence.
  • MedLong-8B and LoMeVQA-dev give a reproducible training recipe and baseline for temporal chest X-ray reasoning.
  • Failure modes (text priors over image evidence, weak change localization, attention stuck on text) become explicit targets for architecture and training fixes.
  • Multi-task fine-tuning on all five tasks can match single-task fine-tuning at 8B scale, supporting joint longitudinal skill learning.
  • Gains transfer to other longitudinal chest X-ray benchmarks without training on them, so the resource is not purely in-distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because generation is anchored in reports, residual report–image mismatch in public CXR corpora will cap ceiling performance until image-native change labels exist.
  • The same trajectory-plus-sampling recipe could be ported to CT, MRI, or multi-modal EHR timelines once comparable entity graphs exist.
  • Attention-map shifts after fine-tuning suggest explicit temporal alignment or change-detection heads may be more sample-efficient than pure SFT.
  • Progress classification’s lower consistency rate implies closed-ended temporal labels need heavier human or multi-judge filtering than open description tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LoMeVQA, a 206K-pair multi-task benchmark for longitudinal medical VQA built primarily from MIMIC-CXR, spanning five tasks (progress classification, progress description, progress report generation, differential region grounding, and differential region description). Construction organizes patient studies chronologically, extracts entities with RadGraph, forms entity trajectories, and uses LLMs with temporal/entity sampling to synthesize QA pairs, followed by diversity/consistency filtering and clinician review of a 2,500-sample test split; DRG/DRD are converted from Chest ImaGenome-style annotations. The authors evaluate general and medical MLLMs, report poor zero-shot performance (e.g., ~55% PC accuracy, ~25% DRG IoU for strong baselines), and release MedLong-4B/8B via LoRA SFT of Qwen3-VL on LoMeVQA-dev, claiming SOTA on all in-distribution tasks plus gains on MMXU-test and MIMIC-CXR-T, with case studies and attention maps offered as supporting analysis.

Significance. If the labels faithfully track image-level temporal change, this would be a timely and useful contribution: longitudinal comparison is clinically central yet under-benchmarked relative to single-visit medical VQA, and a multi-task suite at this scale with public data, a documented pipeline (Algorithm 1), human-reviewed test selection, and an open fine-tuned baseline would give the community a concrete evaluation and training substrate. The OOD checks (Table 4), multi-metric open-ended evaluation (NLG/CE/LLM-as-judge), and attention visualizations are strengths beyond a pure leaderboard dump. The significance is therefore real but conditional on how well synthetic, report-conditioned answers isolate visual longitudinal reasoning rather than report-text priors.

major comments (4)
  1. [§3.2, Algorithm 1, Table 2, Table 7] §3.2 and Algorithm 1: QA pairs for PC/PD/PRG are generated from RadGraph entity trajectories and report sentences, not from pixels; consistency filtering (Table 2) is likewise report-vs-(q,a) text alignment. For Progress Classification the automatic consistency pass rate is only 73.8% with 22.1% contradictory before retention. The central claim that poor MLLM scores demonstrate limitations in longitudinal *visual* reasoning therefore rests on an incompletely validated assumption that retained labels match true image-level change. Human review (Table 7) assesses retained samples’ usability/agreement, not independent double-reading of image pairs for progression direction. Please add (i) a clinician image-only (or image-first) adjudication study on a stratified PC/PD subset reporting agreement with synthetic labels, and (ii) an ablation or control that quantifies how much performance is ex
  2. [§4.2, Table 3, Table 4] Table 3: MedLong-8B is trained on LoMeVQA-dev and evaluated on LoMeVQA-test from the same generation pipeline, so large in-distribution jumps (e.g., PC ~34%→70%, DRG IoU ~7%→67% vs Qwen3-VL-8B) partly measure adaptation to synthetic style and answer distribution. OOD gains on MMXU and MIMIC-CXR-T (Table 4) help, but are narrower in task coverage. To support the SOTA-baseline claim for longitudinal visual reasoning, report controls that separate format learning from temporal comparison (image-order shuffle, single-image ablation, and cross-pipeline evaluation where labels come from a different source than training).
  3. [§3.1, §3.2, Table 1, Table 3] §3.1–3.2 and Table 1: DRG/DRD are template conversions of existing region annotations (Chest ImaGenome path), while PC/PD/PRG follow the KG→LLM path. Treating all five as one homogeneous “longitudinal visual reasoning” benchmark blurs distinct supervision sources and difficulty. Clarify per-task provenance in the main text, and avoid averaging (Table 3 “Average”) across closed-set accuracy, NLG/CE/LLM scores, and IoU without justification—or report task-group averages separately so grounding conversion gains are not conflated with synthetic progression-description gains.
  4. [§4.1, Appendix E, Table 3] §4.1 metrics and Appendix E: Open-ended evaluation leans on ROUGE/METEOR, F1-RadGraph, and DeepSeek LLM-as-a-Judge. Lexical metrics are weak for clinical temporal correctness, and the judge is another LLM scoring LLM-written references—risking shared stylistic bias with the data generator. Provide human preference or clinical correctness ratings on a fixed open-ended subset correlated against the judge, and state judge–generator model separation if any. Without this, PD/PRG/DRD rankings in Table 3 are only weakly grounded.
minor comments (5)
  1. [§3, Table 5] Table 5 vs abstract “206K”: task sizes sum to the claimed scale, but main text should state train/dev/test splits per task explicitly (only test=2,500 aggregate is clear) and whether patients are disjoint across splits to prevent leakage.
  2. [Figure 1, Figure 13] Figure 1(c) and several case figures are hard to read in grayscale; ensure axis labels and model names remain legible and that GT boxes in grounding cases (e.g., Figure 13) use consistent coordinate conventions with the text.
  3. [§2, Table 1] Related work (Table 1): Medical-Diff-VQA* and MMXU are closest priors; a short paragraph quantifying what new clinical behaviors LoMeVQA measures beyond difference description/classification would sharpen novelty without overstating “first systematic” language.
  4. [§4.1] Implementation: LoRA rank, target modules, image resolution, and max sequence length for multi-image inputs are underspecified relative to reproducibility claims; add them to §4.1 or the appendix.
  5. Typographical/consistency nits: arXiv date line “30 Jul 2026”; intermittent spacing in math (e.g., n_i); “MedLong-4B and MedLong-8B” introduced before training details; ensure task abbreviations in Figure 1 match §3.1.

Circularity Check

0 steps flagged

No load-bearing circular derivation; standard benchmark train/test and report-anchored synthesis, with external OOD checks.

full rationale

LoMeVQA is an empirical benchmark paper, not a first-principles derivation. The construction chain (chronological MIMIC-CXR episodes → RadGraph entities/trajectories → LLM VQA synthesis → diversity/consistency filters → clinician-reviewed test split) does not define answers in terms of the models being scored, nor does it fit a parameter on the evaluation target and relabel that fit as a prediction. MedLong-8B is ordinary SFT of Qwen3-VL-8B on LoMeVQA-dev and evaluation on held-out LoMeVQA-test plus two external longitudinal sets (MMXU-test, MIMIC-CXR-T) never used in training; in-distribution gains are therefore expected supervised learning, not circular self-prediction. Self-citations in Related Work (e.g., authors’ prior medical imaging papers) are background and not used to force uniqueness of the benchmark design or the SOTA claim. Concerns that PC/PD/PRG labels are report-trajectory-derived (Alg. 1, §3.2; Table 2 PC consistency only 73.8%) and may reward text priors rather than pure visual change are validity/correctness risks, not circularity under the stated patterns. Score 1 only for the mild, non-load-bearing fact that the primary SOTA table is on a test split from the same generation pipeline the model was trained on—mitigated by the reported OOD lifts.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The work is empirical systems/benchmark research. Load-bearing premises are standard ML evaluation assumptions plus domain choices about synthetic label validity and CXR-as-proxy for longitudinal medicine. No physical constants or fitted scientific laws; free choices are pipeline hyperparameters and filtering thresholds.

free parameters (4)
  • Temporal window length K and shared-entity count N = K and N varied by construction; PRG mean image length ~3.15
    Control difficulty of synthesized VQA via sliding window and entity sampling; chosen by authors to shape task hardness, not derived.
  • LLM consistency-filter retention policy = PC 73.8% / PD 90.1% / PRG 94.6% consistent
    Samples labeled Consistent/Contradictory/Uncertain by an LLM; only Consistent kept—thresholding directly shapes label noise, especially PC at 73.8% pass.
  • LoRA SFT hyperparameters = batch 64, lr 2e-5, 1 epoch
    Batch size, learning rate, one-epoch schedule determine MedLong performance numbers used for SOTA claim.
  • Diversity-filter answer-concentration threshold
    Removes over-concentrated answer subsets; threshold is author-chosen and affects class balance.
axioms (5)
  • domain assumption RadGraph entity-relation extraction from radiology reports is a sufficiently faithful structured representation of clinically meaningful findings for trajectory building.
    Step 1 of construction (Eq. 3 and Algorithm 1) treats RadGraph outputs as the substrate for temporal modeling.
  • ad hoc to paper LLM-generated QA conditioned on report-derived entity trajectories, after automatic consistency checks, yields labels aligned with visual longitudinal change.
    Core of difficulty-controlled VQA generation and filtering (Section 3.2); underwrites benchmark validity.
  • domain assumption Standard NLG metrics (ROUGE/METEOR), F1-RadGraph, IoU, accuracy, and LLM-as-judge scores are adequate proxies for clinical longitudinal reasoning quality.
    Section 4.1 metrics; open-ended clinical quality is partly delegated to an external judge model.
  • domain assumption MIMIC-CXR paired image-report longitudinal episodes are representative enough to benchmark “longitudinal medical visual reasoning” as framed.
    Entire dataset is built from MIMIC-CXR chest radiographs; generalization beyond CXR is assumed in the broader claim language.
  • standard math Supervised fine-tuning on LoMeVQA-dev is a valid way to claim improved longitudinal capability when paired with held-out and OOD tests.
    Standard ML train/eval methodology used in Section 4.
invented entities (2)
  • LoMeVQA benchmark (five-task longitudinal medical VQA suite) independent evidence
    purpose: Provide multi-task evaluation resource for temporal medical image understanding.
    Primary contribution; constructed artifact rather than a physical entity. Independent use depends on public data release and external adoption.
  • MedLong-8B / MedLong-4B no independent evidence
    purpose: Fine-tuned baselines demonstrating that LoMeVQA-dev supervision improves longitudinal task performance.
    Named model variants of Qwen3-VL via LoRA SFT; not a new architecture or scientific object beyond weights trained on the new data.

pith-pipeline@v1.2.0-daily-grok45 · 29803 in / 3717 out tokens · 68941 ms · 2026-07-31T01:33:48.589070+00:00 · methodology

0 comments
read the original abstract

In clinical practice, patients often undergo multiple imaging examinations over successive visits, yielding longitudinal data. Modeling such temporal information is crucial for reliable assessment of disease progression and treatment response. However, despite the rapid advancement of multimodal large language models (MLLMs), longitudinal medical visual reasoning remains largely underexplored. To fill this gap, we propose LoMeVQA, a comprehensive benchmark consisting of 206K longitudinal visual question answering (VQA) pairs for temporal medical image analysis. LoMeVQA covers five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description. To construct the dataset, we develop an automated pipeline that (1) organizes patient records chronologically, (2) extracts clinically meaningful entities via a medical knowledge graph, and (3) models their temporal evolution to guide large language models in generating high-quality longitudinal VQA pairs. Extensive evaluations demonstrate that both general-purpose and medical-domain MLLMs perform poorly on LoMeVQA, revealing substantial limitations in temporal reasoning. To address these limitations, we introduce MedLong-8B, which achieves state-of-the-art performance across all tasks. Beyond benchmarking, we conduct detailed analyses that uncover key failure modes and shed light on how to improve longitudinal medical visual reasoning. Our data is available at: https://github.com/pepperbubble/LoMeVQA

Figures

Figures reproduced from arXiv: 2607.27806 by Chengmei Yang, Lianghua He, Longzhen Yang, Yihang Liu, Ying Wen, Zhangkai Ni, Zhilin Wu.

Figure 1
Figure 1. Figure 1: (a) Single-image analysis focuses on the current [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representative examples from the five tasks in LoMeVQA, highlighting diverse scenarios of longitudinal medical [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The pipeline of constructing LoMeVQA. Step(1) We collect longitudinal data from patients at different time points, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: MedLong-8B shows generalization ability on the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A case study for progress description. MedLong can capture and describe the changes between images. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of attention map during answer gener [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Dataset statistics and linguistic characteristics. We show the entity distribution, entity word cloud, task sample [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An example for difficulty-controlled VQA generation. We use DeepSeek to generate answers under the setting of K = 3 [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: A failure case of during the generation of LLM. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A contradictory sample excluded due to a contradiction between the generated answer and the trajectory in Task 1 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: A case study for task 1 (Progress Classification). [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: A case study for task 3 (Progress Report Generation). [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: A case study for task 4 (Differential Region Grounding). [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: A case study for task 5 (Differential Region Description). [PITH_FULL_IMAGE:figures/full_fig_p016_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Illustration of LLM-as-a-Judge for evaluating open-ended medical responses. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Attention map for task 1 (Progress Classification). [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Attention map for task 2 (Progress Description). [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Attention map for task 3 (Progress Report Generation). [PITH_FULL_IMAGE:figures/full_fig_p018_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Attention map for task 5 (Differential Region Description). [PITH_FULL_IMAGE:figures/full_fig_p018_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Prompt for VQA pairs generation of task 1 (Progress Classification). [PITH_FULL_IMAGE:figures/full_fig_p019_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Prompt for VQA pairs generation of task 2 (Progress Description). [PITH_FULL_IMAGE:figures/full_fig_p020_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Prompt for VQA pairs generation of task 3 (Progress Report Generation ). [PITH_FULL_IMAGE:figures/full_fig_p021_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Prompt for the consistency check using LLM. [PITH_FULL_IMAGE:figures/full_fig_p022_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: The prompt for LLM-As-A-Judge 23 [PITH_FULL_IMAGE:figures/full_fig_p023_24.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 15 linked inside Pith

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, et al . 2023. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images.Advances in Neural Information Processing Systems36 (2023), 3867–3880

    2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923(2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  5. [5]

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maxim- ilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. 2023. Learning to exploit temporal structure for biomed- ical vision-language processing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15016–15027

    2023

  6. [6]

    Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning Müller. 2019. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. InProceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019

    2019

  7. [7]

    Asma Ben Abacha, Mourad Sarrouti, Dina Demner-Fushman, Sadid A Hasan, and Henning Müller. 2021. Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. InProceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum-working notes. 21-24 September 2021

    2021

  8. [8]

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. Huatuogpt-o1, towards medical complex reasoning with LLMs.arXiv preprint arXiv:2412.18925(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24185–24198

    2024

  10. [10]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Michael Crawshaw. 2020. Multi-task learning with deep neural networks: A survey.arXiv preprint arXiv:2009.09796(2020)

    Pith/arXiv arXiv 2020

  12. [12]

    Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis Langlotz. 2022. Improving the factual correctness of radiol- ogy report generation with semantic rewards. InFindings of the Association for Computational Linguistics: EMNLP 2022. 4348–4360

    2022

  13. [13]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286(2020)

    Pith/arXiv arXiv 2020

  14. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. LoRA: Low-rank adaptation of large language models.International Conference on Learning Representations1, 2 (2022), 3

    2022

  15. [15]

    Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M Summers, and Yingying Zhu. 2023. Expert knowledge-aware image difference graph representation learning for difference- aware medical visual question answering. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4156–4165

    2023

  16. [16]

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. Omnimedvqa: A new large-scale comprehensive evaluation bench- mark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22170–22183

    2024

  17. [17]

    Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. 2025. m1: Unleash the potential of test-time scaling for medical reasoning with large language models.arXiv preprint arXiv:2504.00869(2025)

    arXiv 2025

  18. [18]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  19. [19]

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports.arXiv preprint arXiv:2106.14463(2021)

    Pith/arXiv arXiv 2021

  20. [20]

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. 2019. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.Scientific data6, 1 (2019), 317

    2019

  21. [21]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images.Scientific data5, 1 (2018), 1–10

    2018

  22. [22]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems36 (2023), 28541–28564

    2023

  23. [23]

    Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020. Dice loss for data-imbalanced NLP tasks. InProceedings of the 58th annual meeting of the association for computational linguistics. 465–476

    2020

  24. [24]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  25. [25]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In2021 IEEE 18th international symposium on biomedical imaging (ISBI). IEEE, 1650–1654

    2021

  26. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in Neural Information Processing Systems36 (2023), 34892–34916

    2023

  27. [27]

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2024. When MOE meets LLMs: Parameter efficient fine-tuning for multi-task medical applications. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1104– 1114

    2024

  28. [28]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024. Deepseek-vl: towards real- world vision-language understanding.arXiv preprint arXiv:2403.05525(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Linjie Mu, Zhongzhen Huang, Shengqian Qin, Yakun Zhu, Shaoting Zhang, and Xiaofan Zhang. 2025. Mmxu: A multi-modal and multi-x-ray understanding dataset for disease progression.arXiv preprint arXiv:2502.11651(2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Zhangkai Ni, Runyu Xiao, Wenhan Yang, Hanli Wang, Zhihua Wang, Lihua Xiang, and Liping Sun. 2024. M2trans: Multi-Modal Regularized Coarse-to-Fine Transformer for Ultrasound Image Super-Resolution.IEEE Journal of Biomedical and Health Informatics29, 5 (2024), 3112–3123

    2024

  31. [31]

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. 2025. Medgemma technical report.arXiv preprint arXiv:2507.05201(2025)

    Pith/arXiv arXiv 2025

  32. [32]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report.arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  33. [33]

    Fuying Wang, Shenghui Du, and Lequan Yu. 2024. Hergen: Elevating radiology report generation with longitudinal data. InEuropean Conference on Computer Vision. Springer, 183–200

    2024

  34. [34]

    Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et al. 2025. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs.arXiv preprint arXiv:2504.00993(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper S Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. 2021. Chest ImaGenome Dataset for Clinical Reasoning. Advances in Neural Information Processing Systems(2021)

    2021

  36. [36]

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al . 2024. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. In The Thirteenth International Conference on Learning Representations

    2024

  37. [37]

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al

  38. [38]

    Jinxia Yang, Bing Su, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Unlocking the power of spatial and temporal information in medical multimodal pre-training. InProceedings of the 41st International Conference on Machine Learning. 56382– 56396

    2024

  39. [39]

    Longzhen Yang, Zhangkai Ni, Ying Wen, Yihang Liu, Lianghua He, and Heng Tao Shen. 2025. Self-Supervised Anatomical Consistency Learning for Vision- Grounded Medical Report Generation. InProceedings of the 33rd ACM Inter- national Conference on Multimedia. 2958–2967

    2025

  40. [40]

    Suhao Yu, Haojin Wang, Juncheng Wu, Cihang Xie, and Yuyin Zhou. 2025. MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning. arXiv preprint arXiv:2505.16964(2025)

    arXiv 2025

  41. [41]

    Jingkun Yue, Siqi Zhang, Zinan Jia, Huihuan Xu, Zongbo Han, Xiaohong Liu, and Guangyu Wang. 2025. MedSG-Bench: A Benchmark for Medical Image Sequences Grounding.arXiv preprint arXiv:2505.11852(2025)

    Pith/arXiv arXiv 2025

  42. [42]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9556–9567

    2024

  43. [43]

    Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi 9 Zhilin Wu, Zhangkai Ni, Chengmei Yang, Longzhen Yang, Yihang Liu, Ying Wen, and Lianghua He Yang, et al. 2025. A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings.N...

    2025

  44. [44]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision. 11975–11986

    2023

  45. [45]

    Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, et al. 2023. Huatuogpt, towards taming language model to be a doctor. InFindings of the Association for Computational Linguistics: EMNLP 2023. 10859–10885

    2023

  46. [46]

    Xi Zhang, Zaiqiao Meng, Jake Lever, and Edmond SL Ho. 2025. Libra: Leveraging temporal images for biomedical radiology analysis. InFindings of the Association for Computational Linguistics: ACL 2025. 17275–17303

    2025

  47. [47]

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pmc-vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415(2023)

    Pith/arXiv arXiv 2023

  48. [48]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 400–410

    2024

  49. [49]

    The pleural effusion has remained unchanged, with a small left-sided effusion still present in the second image

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert- level medical reasoning and understanding. InForty-second International Confer- ence on Machine Learning. 10 LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA A Dataset Statistics Table 5 summari...

    2025

  50. [56]

    Return the results strictly in JSON format

  51. [57]

    Uncertain

    If you cannot determine the answer based on the reports, respond with "Uncertain". The reports for the images are as follows: EXAMPLE JSON OUTPUT Format: { "question": a question about the disease progression in the images, "thinking": the rationale for your answer, "answer": Improving/Stable/Worsening/Uncertain }, The reports for the images are as follow...

  52. [62]

    question

    Return the results strictly in JSON format. The reports for the images are as follows: EXAMPLE JSON OUTPUT Format: { "question": a question about the disease progression in the images, "answer": a description of the disease progression in the images } The reports for the images are as follows: For i, sentence in sentences: User_prompt += “Image at time i:...

  53. [63]

    The content is based on the reports I provide, and no mention of "report" is allowed

    Pretend you've seen the images. The content is based on the reports I provide, and no mention of "report" is allowed

  54. [64]

    Ensure that the sentences you generate are supported by the reports I provide

  55. [65]

    Maintain as much diversity as possible in the questions

  56. [66]

    The order of the reports corresponds to the order of the images

  57. [67]

    : a description of the disease progression in the images } The reports for the images are as follows: For i, sentence in sentences: User_prompt += “Image at time i: sentence

    Return the results strictly in JSON format. The reports for the images are as follows: EXAMPLE JSON OUTPUT Format: { "answer": a description of the disease progression in the images } The reports for the images are as follows: For i, sentence in sentences: User_prompt += “Image at time i: sentence.” Message.append({“role”: “user”, “content”: User_prompt})...

  58. [68]

    Ground Truth Trajectory:{trajectory}

  59. [69]

    AI-Generated Q&A conversation (chronological):{conversation_raw} Task (do these steps in order):

  60. [70]

    pulmonary vascular congestion

    Identify the *target of the question* — the specific finding or overall impression the human asked about. - If the human question explicitly asks about one finding (e.g., "pulmonary vascular congestion", "pleural effusion", "cardiomegaly", "overall progression"), treat that as the target. - If the human asked a *direct-choice* question (e.g., "Direct choi...

  61. [71]

    Uncertain

    From the Ground Truth Trajectory, extract the change for that *same* target. If the trajectory doesn't mention the target (or is ambiguous about it), the correct ground-truth label for that target is "Uncertain"

  62. [72]

    Consistent

    Compare the AI answer's claimed class (Improving / Stable / Worsening / Uncertain) **for the exact target** to the ground-truth class for that target. - Only compare the *same* target. Do NOT penalize an answer that correctly states the change of the asked target even if other findings in the trajectory changed differently. - If the trajectory supports th...

  63. [73]

    verdict":

    Produce a JSON object and nothing else, using exactly these fields and allowed verdict strings: {{ "verdict": "Consistent" / "Contradictory" / "Uncertain", "reasoning": "<concise explanation: mention the question target, the AI's claimed class, and the trajectory evidence supporting your decision>" }} Important rules for the evaluator: - When mapping lett...

  64. [2022]

    8 LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

    Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems35 (2022), 23716–23736. 8 LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

    2022

  65. [2025]

    Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning.arXiv preprint arXiv:2506.07044(2025)

    Pith/arXiv arXiv 2025