Pith. sign in

REVIEW 4 major objections 6 minor 59 references

MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-level visual tree plus retrieval-augmented clinical knowledge pushes sarcopenia ultrasound diagnosis to 99% accuracy.

desk verdict Plausible architecture but the 99% sarcopenia claim rests on image-level leakage and inconsistent dataset counts; needs major revision with patient-grouped evaluation. read the letter →

arxiv 2508.19319 v1 pith:GEZ34EYC submitted 2025-08-26 eess.IV cs.AIcs.CV

Pardis Moradbeiki , Nasser Ghadiri , Sayed Jalal Zahabi , Uffe Kock Wiil , Kristoffer Kittelmann Brockhattingen , Ali Ebrahimi This is my paper
classification eess.IVcs.AIcs.CV
keywords MedVQAMultimodalReasoningSarcopeniaPredictionRetrievalFrameworkUltrasounddiagnosisHierarchicalvisualfeaturesQuestionAnsweringRetrieval-AugmentedGeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedVQA-TREE is built around a clinical intuition: to diagnose sarcopenia from ultrasound, a model should look the way a clinician does—global view first, then the relevant muscle region, then the spatial relationships among small structures—and should be able to consult external medical knowledge when the image alone is ambiguous. The paper reports that this combination reaches up to 99% diagnostic accuracy on a proprietary sarcopenia ultrasound dataset (3,474 images from 24 patients) and 88% on the public VQA-RAD benchmark, beating the listed baselines by more than 10 points on the sarcopenia task. The mechanism is a question-guided gate that selects which of three visual levels to trust, a low-rank fusion module that combines that level with patient demographics and test scores, and a retrieval pipeline that expands patient variables into standardized clinical concepts via UMLS and filters external literature before fusing it into the prediction. The claim matters because sarcopenia is underdiagnosed and ultrasound is portable and cheap; if accurate, this design would make AI-assisted screening feasible outside CT and MRI settings.

What carries the argument

The load-bearing object is the three-level 'tree' of visual representations coupled to a question-guided gate and a retrieval-augmented fusion block. Level 1 is a global feature vector; Level 2 uses a pretrained segmentation model to isolate anatomically relevant muscle regions and extracts regional features; Level 3 builds a spatial graph whose nodes are centroids of superpixel-derived regions and whose edges encode proximity and anatomical adjacency. The gate is a softmax over the projected question text that selects exactly one level, and a low-rank adaptation module transforms the selected vector before it is combined with text and numeric features for classification. The retrieval side

What would settle it

Re-run the full pipeline with patient-disjoint cross-validation—train on a subset of the 24 patients and test on the remaining patients, or leave one patient out at a time—and compare accuracy to the reported 0.99. If accuracy drops substantially, the near-perfect figure is explained by image-level leakage rather than by the hierarchical architecture or retrieval pipeline.

Watch

Extended reading notes

Core claim

The paper's central claim is that sarcopenia can be classified from rectus femoris ultrasound with near-perfect accuracy when image interpretation is structured hierarchically and augmented by retrieved clinical knowledge. The vision module extracts three complementary representations: coarse global features, region features from a segmentation model, and node embeddings from a superpixel-based spatial graph that encodes anatomical adjacency. A soft gating function, computed from the projected question text, selects the most relevant level; the chosen features pass through a low-rank adapter and are added to the projected text and numeric representations before classification. In parallel, t

Load-bearing premise

The reported accuracy treats each ultrasound image as an independent sample even though the 3,474 images come from 24 patients; if the cross-validation folds are not grouped by patient, images from the same person can appear in both training and test data, and the model can memorize patient appearance instead of learning generalizable signs of sarcopenia.

Editorial extensions

If this is right

  • The reported 99% on the custom ultrasound dataset points toward bedside screening: a portable acquisition device plus this lightweight model could reduce reliance on costly CT or MRI for sarcopenia assessment.
  • The ablation progression (coarse ~0.70, fine ~0.68, fused hierarchy ~0.90, fused with clinical data ~0.93, with retrieval ~0.99) implies that no single visual level is sufficient and that external knowledge retrieval is a central ingredient, not a marginal add-on.
  • Because the gate processes only the selected visual level, the framework could adapt its computation to the question, an efficiency property that matters for community clinics with limited hardware.
  • On VQA-RAD the model reaches 0.88, beating the listed baselines by a small margin; the same hierarchy may transfer to other low-data medical VQA tasks, though the PathVQA result of 0.84 shows the transfer is not uniform.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the sharp jump from 0.93 to 0.99 when retrieval is added should be tested for leakage—since queries are generated from each patient's own clinical variables, matching could occur between a test patient's profile and similar text seen in training, independent of the image.
  • Editorial extension: the paper never states whether its k-fold splits group images by patient; with 24 patients and 3,474 images, patient-disjoint cross-validation is the single most informative next experiment for separating memorization from generalization.
  • Editorial extension: the coarse-to-fine visual tree plus ontology-guided retrieval could transfer to other under-labeled musculoskeletal ultrasound tasks, such as frailty or muscle-quality assessment, where cues are subtle and clinical context is essential.
  • Editorial extension: a reader should read the 99% as domain-specific; the same architecture reports 0.84 on PathVQA, below several listed baselines, so the contribution is best understood as a low-data sarcopenia-ultrasound method rather than a universal medical-VQA winner.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MedVQA-TREE, a multimodal framework for sarcopenia diagnosis from ultrasound images combined with structured clinical data and retrieved medical text. The architecture combines a three-level hierarchical visual encoder (global features, SAM-based regional features, superpixel spatial-graph features), a question-guided gated fusion mechanism with LoRA, and a UMLS-guided multi-query/multi-hop retrieval pipeline. The authors report experiments on VQA-RAD, PathVQA, and a proprietary sarcopenia ultrasound dataset, claiming up to 99% diagnostic accuracy and improvements over state-of-the-art medical VQA methods by more than 10%. Ablation tables show a progression from individual visual levels (accuracies roughly 0.52–0.70) to combined hierarchical features (0.90) and finally to RAG-enhanced fusion (0.99).

Significance. If the 99% accuracy result were valid, the paper would be a meaningful contribution to low-resource sarcopenia screening and to medical VQA in small-data regimes. The hierarchical visual reasoning, SAM-based region selection, and lightweight gated fusion are reasonable and potentially useful ideas. However, the central quantitative claim is not supported by the evaluation as presented: the k-fold cross-validation is performed on 3,474 images from only 24 patients with no described patient-level split, the dataset description is internally inconsistent, and the RAG module retrieves from the training set at test time. These issues mean the main empirical result cannot currently be distinguished from leakage or circularity, and the comparison claims are contradicted by the paper's own Table 2.

major comments (4)
  1. [§2.1, §3.2.1, Tables 3–11] The cross-validation is applied to 3,474 images from only 24 patients, and the paper never states that folds are grouped by patient. With an average of roughly 145 images per patient, an image-level split places images from the same patient in both training and test folds, allowing the model to memorize patient-specific appearance and patient-level clinical variables (age, BMI, SPPB). Table 11's 0.99 accuracy is therefore indistinguishable from patient-identity leakage. The authors must report patient-exclusive folds, per-patient predictions, and fold-wise patient counts; all k-fold results in Tables 3–11 need to be re-run under that protocol or explicitly justified.
  2. [§2.1] The dataset description is numerically inconsistent. Transverse (1,350) and longitudinal (962) image counts sum to 2,312, not 3,474. The sex-stratified counts sum to 3,472 (male: 763+1,240=2,003; female: 297+1,172=1,469). The plane-level sarcopenic/non-sarcopenic counts total 776/1,536=2,312, whereas the stated 30.53%/69.47% proportions applied to 3,474 images correspond to approximately 1,060/2,414 labels. Please correct these counts; the class-imbalance description and all reported metrics depend on them.
  3. [§3.2.1, Fusion Strategies with RAG] The test-time RAG procedure is described as generating a query from the patient record and matching it with chunks 'aligned with the training set.' This is not external knowledge; it is retrieval from the training data. When combined with image-level splitting, the retrieved chunks can include the exact same patients' clinical variables. Even under patient-level splitting, matching test queries to training-derived topic categories creates a circularity in which predictions are informed by the training distribution. The near-perfect 0.99 result is a fitted outcome under this protocol. The authors should replace this with a genuinely external, label-independent knowledge base, or demonstrate that no information about training labels flows through the retrieval step.
  4. [Abstract, Table 2] The claim of outperforming previous state-of-the-art methods by over 10% is not supported by the reported numbers. On PathVQA, MedVQA-TREE achieves 0.84, below all listed baselines (0.90–0.91). On VQA-RAD it achieves 0.88 versus 0.87 for PeFoMed, a difference of about one point. The only >10% margin is on the sarcopenia dataset, whose validity is undermined by the issues above. Please restrict the claims to actual comparisons and provide confidence intervals or proper statistical tests for each dataset.
minor comments (6)
  1. [§3.2.1] The row labels '1-fold' through '5-fold' are ambiguous. Clarify whether these are the five folds of a 5-fold cross-validation and report the number of patients in each fold.
  2. [Tables 12–15] These tables show high accuracy with very low F1/precision/recall (e.g., Table 12: accuracy 0.70–0.75, F1 0.12–0.19). Explain how this pattern arises and why accuracy alone is a meaningful headline metric in this setting.
  3. [§2.4.6] The text mentions T5 as the underlying language model, but the formal model in Eqs. (1)–(8) does not include a T5 component. Clarify where T5 enters the architecture.
  4. [Table 1, References] The citation for PubMedBERT appears to point to a NEJM AI paper on BiomedCLIP rather than the original PubMedBERT publication. Please check the reference list and correct it.
  5. [Figures 2–3] Figure 2's caption says 'image selected of VQA-RAD dataset', while the surrounding text describes the sarcopenia ultrasound pipeline. Clarify which dataset is actually used for the segmentation illustrations.
  6. [§2.1, §3.1] The paper refers to the sarcopenia task as 'VQA', but the sarcopenia experiments are binary image classification with clinical features and no natural-language question. Clarify how the gate uses the 'textual query' and whether any free-form question is posed to the model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework's modules and evaluation are independently specified; flagged leakage and benchmark issues are correctness risks, not circular reductions.

full rationale

I examined the paper's derivation chain. The visual feature extraction (Sec. 2.3.1), numeric embedding (Sec. 2.3.2), text retrieval (Sec. 2.3.3), LoRA fusion (Sec. 2.4), and loss (Eq. 8) are defined by explicit equations that do not reference the target labels or the reported evaluation results. The VQA-RAD and PathVQA benchmarks are external, and the model's reported accuracy there (0.88 and 0.84) is an independent empirical outcome, not a quantity forced by the method's definitions. The sarcopenia dataset is proprietary and small, and the RAG module builds its retrieval index from training data and matches test queries to 'training-aligned categories' (Sec. 3.2.1); while this is a legitimate methodological concern about generalization and possible overfitting, it is not a case where a predicted quantity is definitionally equal to an input or where a fitted parameter is renamed as a prediction. The absence of an explicit patient-grouped split in the k-fold evaluation (24 patients, 3,474 images) is a serious validity risk, but it concerns data leakage rather than circularity. No load-bearing self-citation or imported uniqueness theorem appears. I therefore find no significant circularity in the manuscript's derivations.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a proprietary dataset with unresolved count inconsistencies, an unstated cross-validation split unit, and a retrieval pipeline with numerous hand-chosen hyperparameters that are not reported. The 99% accuracy cannot be separated from fitting artifacts or potential patient-level leakage without code, data, and a clearly patient-stratified evaluation.

free parameters (5)
  • LDA topic count T = not reported
    The sarcopenia-specific LDA model in Section 2.3.3 requires a topic count; no value is given.
  • Number of retrieved sentences C = C = 10
    Top-C semantically similar sentences are selected in Section 2.3.3; C is chosen by hand.
  • Number of SAM regions S = not reported
    Levels 2 and 3 select the top-S informative regions; the selection criteria are referenced to Section 3.2.1 but not specified there.
  • LoRA rank r = not reported
    LoRA adapter rank in Eq. 2; no value is provided anywhere in the paper.
  • LDA similarity threshold = not reported
    The threshold for retaining topic-aligned sentences in Section 2.3.3 is not quantified.
assumptions (5)
  • domain assumption Images from the same patient can be treated as independent samples in cross-validation.
    Section 3.2.1 reports k-fold results without stating patient-level grouping; if false, the 99% accuracy is inflated by patient memorization.
  • domain assumption SAM zero-shot segmentation produces clinically meaningful muscle regions for sarcopenia.
    Level 2 relies on SAM masks without task-specific retraining; no quantitative validation of segmentation quality is provided.
  • domain assumption UMLS-expanded PubMed queries retrieve patient-specific evidence that improves diagnosis.
    Sections 2.2.2 and 2.3.3 assume that general PubMed abstracts filtered by LDA and similarity provide personalized diagnostic signal; no analysis demonstrates that retrieved text carries label-relevant information.
  • standard math Plain cross-entropy loss is appropriate for the imbalanced sarcopenia task.
    Eq. 8 uses unweighted cross-entropy despite a 69.47% / 30.53% class imbalance; no class weighting is described.
  • domain assumption The proprietary dataset labels according to EWGSOP2 criteria are correct.
    Ground truth comes from a prior study; label quality, inter-observer variability, and diagnostic confirmation are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction." pith.science (2026). https://pith.science/paper/GEZ34EYC

@misc{pith2026250819319,
  author       = {Pith},
  title        = {Pith review of: MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEZ34EYC}},
  note         = {Machine review of arXiv:2508.19319}
}
read the original abstract

Accurate sarcopenia diagnosis via ultrasound remains challenging due to subtle imaging cues, limited labeled data, and the absence of clinical context in most models. We propose MedVQA-TREE, a multimodal framework that integrates a hierarchical image interpretation module, a gated feature-level fusion mechanism, and a novel multi-hop, multi-query retrieval strategy. The vision module includes anatomical classification, region segmentation, and graph-based spatial reasoning to capture coarse, mid-level, and fine-grained structures. A gated fusion mechanism selectively integrates visual features with textual queries, while clinical knowledge is retrieved through a UMLS-guided pipeline accessing PubMed and a sarcopenia-specific external knowledge base. MedVQA-TREE was trained and evaluated on two public MedVQA datasets (VQA-RAD and PathVQA) and a custom sarcopenia ultrasound dataset. The model achieved up to 99% diagnostic accuracy and outperformed previous state-of-the-art methods by over 10%. These results underscore the benefit of combining structured visual understanding with guided knowledge retrieval for effective AI-assisted diagnosis in sarcopenia.

Figures

Figures reproduced from arXiv: 2508.19319 by the authors.

Figure 1
Figure 1. Overview of the proposed multi-modal framework for sarcopenia. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Region-Based Feature Extraction via SAM Segmentation (image selected of [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Superpixel-Based Graph Construction for Fine-Grained Analysis [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A semantic pipeline that translates clinical inputs into UMLS PubMed queries [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Semantic Filtering and Knowledge-Augmented Fusion [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Comparison of segmentation results using different SAM-based models: (a) [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Segmentation comparison between the constructed superpixel graph and the [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 53 canonical work pages

  1. [1]

    Dupont, et al., Revised sarcopenia consensus: are we missing the preclinical stage? online comment on, Age And Ageing 48 (1) (2018) 1–1

    J. Dupont, et al., Revised sarcopenia consensus: are we missing the preclinical stage? online comment on, Age And Ageing 48 (1) (2018) 1–1

    work page 2018

  2. [2]

    M.-R. Cho, S. Lee, S.-K. Song, A review of sarcopenia pathophysiology, diagnosis, treatment and future direction, Journal of Korean medical science 37 (18) (2022)

    work page 2022

  3. [3]

    Soria-Utrilla, F

    V. Soria-Utrilla, F. J. S´ anchez-Torralvo, F. X. Palmas-Candia, R. Fern´ andez-Jim´ enez, F. Mucarzel-Suarez-Arana, P. Guirado-Pel´ aez, G. Olveira, J. M. Garc ´ ıa-Almeida, R. Burgos-Pel´ aez, Ai-assisted body composition assessment using ct imaging in colorectal cancer patients: Predictive capacity for sarcopenia and malnutrition diagnosis, Nutrients 1...

    work page 2024

  4. [4]

    Yu, H.-Z

    M.-D. Yu, H.-Z. Zhang, Y. Zhang, S.-P. Yang, M. Lin, Y.-M. Zhang, J.- B. Wu, F.-Y. Hong, W.-X. Chen, Relationship between chronic kidney disease and sarcopenia, Scientific reports 11 (1) (2021) 20523

    work page 2021

  5. [5]

    X. Zuo, X. Li, K. Tang, R. Zhao, M. Wu, Y. Wang, T. Li, Sarcopenia and cardiovascular diseases: a systematic review and meta-analysis, Journal of Cachexia, Sarcopenia and Muscle 14 (3) (2023) 1183–1198

    work page 2023

  6. [6]

    Y. Kim, J. Beom, S. Y. Lee, H. C. Jang, K. Kim, M. Kim, G. Y. Shim, C. W. Won, J.-Y. Lim, Comparison of bioelectrical impedance analysis and dual-energy x-ray absorptiometry for the diagnosis of sarcopenia in the older adults with metabolic syndrome: equipment-specific equation development, Aging clinical and experimental research 37 (1) (2025) 1–9. 40

    work page 2025

  7. [7]

    X. Wu, Z. Chen, Y. Zhao, H. Ren, Correlation and predictive value of novel anthropometric indicators with adult sarcopenia and sarcopenia obesity, Scientific Reports 14 (1) (2024) 31776

    work page 2024

  8. [8]

    M. R. Tomaszewski, R. J. Gillies, The biological meaning of radiomic features, Radiology 298 (3) (2021) 505–516

    work page 2021

Show all 59 references
  1. [9]

    Kocak, E

    B. Kocak, E. S. Durmaz, C. Erdim, E. Ates, O. K. Kaya, O. Kilickesmez, Radiomics of renal masses: systematic review of reproducibility and validation strategies, American Journal of Roentgenology 214 (1) (2020) 129–136

    2020

  2. [10]

    S. Kim, S. Park, S. Lee, S. H. Seo, H. S. Kim, Y. Cha, J.-T. Kim, J.- W. Kim, Y.-C. Ha, J.-I. Yoo, Assessing physical abilities of sarcopenia patients using gait analysis and smart insole for development of digital biomarker, Scientific Reports 13 (1) (2023) 10602

    2023

  3. [11]

    S. Kim, H. S. Kim, J.-I. Yoo, Sarcopenia classification model for mus- culoskeletal patients using smart insole and artificial intelligence gait analysis, Journal of Cachexia, Sarcopenia and Muscle 14 (6) (2023) 2793–2803

    2023

  4. [12]

    Chen, J.-W

    Y.-C. Chen, J.-W. Hsieh, Y.-H. Yang, C.-H. Lee, P.-Y. Yu, P.-Y. Chen, A. S. Santa, Towards deep learning-based sarcopenia screening with body joint composition analysis, in: 2021 IEEE International Confer- ence on Image Processing (ICIP), IEEE, 2021, pp. 3807–3811

    2021

  5. [13]

    Islam, F

    S. Islam, F. Kanavati, Z. Arain, O. F. Da Costa, W. Crum, E. Aboagye, A. Rockall, Fully automated deep-learning section-based muscle seg- mentation from ct images for sarcopenia assessment, Clinical Radiology 77 (5) (2022) e363–e371

    2022

  6. [14]

    Sakai, S

    K. Sakai, S. Gilmour, E. Hoshino, E. Nakayama, R. Momosaki, N. Sakata, D. Yoneoka, A machine learning-based screening test for sarcopenic dysphagia using image recognition, Nutrients 13 (11) (2021) 4009

    2021

  7. [15]

    Caputo, I

    V. Caputo, I. Letteri, S. J. Santini, G. Sinatti, C. Balsano, Towards precision in sarcopenia assessment: the challenges of multimodal data analysis in the era of ai, International Journal of Molecular Sciences 26 (9) (2025) 4428. 41

    2025

  8. [16]

    Aznar-Gimeno, J

    R. Aznar-Gimeno, J. L. Perez-Lasierra, P. P´ erez-L´ azaro, I. Bosque- L´ opez, M. Azp ´ ıroz-Puente, P. Salvo-Ib´ a˜ nez, M. Morita-Hernandez, A. C. Hern´ andez-Ruiz, A. G´ omez-Bernal, M. de la Vega Rodrigalvarez- Chamarro, et al., Gait-based ai models for detecting sarcopen...

    2024

  9. [17]

    A. K. Smorchkova, A. V. Petraikin, D. S. Semenov, D. E. Sharova, Sarcopenia: modern approaches to solving diagnosis problems, Digital Diagnostics 3 (3) (2022) 196–211

    2022

  10. [18]

    Perkisas, S

    S. Perkisas, S. Baudry, J. Bauer, D. Beckw´ ee, A.-M. De Cock, H. Hobbe- len, H. Jager-Wittenaar, A. Kasiukiewicz, F. Landi, E. Marco, et al., The sarcus project: evidence-based muscle assessment through ultra- sound, European Geriatric Medicine 10 (1) (2019) 157–158

    2019

  11. [19]

    Behboodi, J

    B. Behboodi, J. Obrand, J. Afilalo, H. Rivaz, Deepsarc-us: A deep learning framework for assessing sarcopenia using ultrasound images, Applied Sciences 14 (15) (2024) 6726

    2024

  12. [20]

    Z. Ye, A. Saraf, Y. Ravipati, F. Hoebers, P. J. Catalano, Y. Zha, A. Za- paishchykova, J. Likitlersuang, C. Guthier, R. B. Tishler, et al., De- velopment and validation of an automated image-based deep learning platform for sarcopenia assessment in head and neck cancer, JAMA n...

    2023

  13. [21]

    D. Lu, Z. Lin, R. Wang, Z. Chen, J. Zhuo, L. Xu, L. Pan, H. Li, X. Yang, C. He, et al., Multi-omics profiling reveals chitinase-3-like protein 1 as a key mediator in the crosstalk between sarcopenia and liver cancer, Redox biology 58 (2022) 102538

    2022

  14. [22]

    S. Lee, R. M. Summers, Clinical artificial intelligence applications in radiology: chest and abdomen, Radiologic Clinics 59 (6) (2021) 987– 1002

    2021

  15. [23]

    W. Fan, M. N. Suvon, S. Zhou, X. Liu, S. Alabed, V. Osmani, A. J. Swift, C. Chen, H. Lu, Medslip: Medical dual-stream language- image pre-training with pathology-anatomy semantic alignment, arXiv preprint arXiv:2403.10635 (2024)

    2024 arXiv

  16. [24]

    Maani, N

    F. Maani, N. Saeed, T. Saleem, Z. Farooq, H. Alasmawi, W. Diehl, A. Mohammad, G. Waring, S. Valappi, L. Bricker, et al., Fetalclip: A 42 visual-language foundation model for fetal ultrasound image analysis, arXiv preprint arXiv:2502.14807 (2025)

    2025

  17. [25]

    P. Ge, T. Su, F. Lv, B. Zhao, P. Zhang, C. H. Wong, L. Yao, Y. Sun, Z. Wang, P. K. Wong, et al., Ultrasound report generation with mul- timodal large language models for standardized texts, arXiv preprint arXiv:2505.08838 (2025)

    2025 arXiv

  18. [26]

    P. H. Yi, H. W. Garner, A. Hirschmann, J. A. Jacobson, P. Omoumi, K. Oh, J. R. Zech, Y. H. Lee, Clinical applications, challenges, and recommendations for artificial intelligence in musculoskeletal and soft- tissue ultrasound: Ajr expert panel narrative review, American Journa...

    2024

  19. [27]

    Y.-W. Chu, K. Zhang, C. Malon, M. R. Min, Reducing hallucina- tions of medical multimodal large language models with visual retrieval- augmented generation, arXiv preprint arXiv:2502.15040 (2025)

    2025 arXiv

  20. [28]

    D. M. Nguyen, N. T. Diep, T. Q. Nguyen, H.-B. Le, T. Nguyen, T. Nguyen, T. Nguyen, N. Ho, P. Xie, R. Wattenhofer, et al., En- riched instruction-following graph alignment for efficient medical vision- language models

  21. [29]

    S. Toro, A. V. Anagnostopoulos, S. M. Bello, K. Blumberg, R. Cameron, L. Carmody, A. D. Diehl, D. M. Dooley, W. D. Duncan, P. Fey, et al., Dynamic retrieval augmented generation of ontologies using artificial intelligence (dragon-ai), Journal of Biomedical Semantics 15 (1) (2024) 19

    2024

  22. [30]

    D. M. H. Nguyen, N. T. Diep, T. Q. Nguyen, B. H. Le, T. Nguyen, A.-T. Nguyen, T. Nguyen, N. Ho, P. Xie, R. Wattenhofer, et al., Logra- med: Long-context multi-graph alignment for medical visual-language models

  23. [31]

    J. J. Lau, S. Gayen, A. Ben Abacha, D. Demner-Fushman, A dataset of clinically generated visual questions and answers about radiology im- ages, Scientific data 5 (1) (2018) 1–10

    2018

  24. [32]

    X. He, Y. Zhang, L. Mou, E. Xing, P. Xie, Pathvqa: 30000+ questions for medical visual question answering, arXiv preprint arXiv:2003.10286 (2020). 43

    2003 arXiv

  25. [33]

    National Center for Biotechnology Informa- tion (NCBI), Entrez programming utilities help, https://www.ncbi.nlm.nih.gov/books/NBK25501/, bethesda (MD): National Library of Medicine (US) (2009)

    2009

  26. [34]

    Q. Yang, H. Zuo, R. Su, H. Su, T. Zeng, H. Zhou, R. Wang, J. Chen, Y. Lin, Z. Chen, et al., Dual retrieving and ranking medical large lan- guage model with retrieval augmented generation, Scientific Reports 15 (1) (2025) 18062

    2025

  27. [35]

    Xiong, Q

    G. Xiong, Q. Jin, X. Wang, M. Zhang, Z. Lu, A. Zhang, Improv- ing retrieval-augmented generation in medicine with iterative follow-up questions, in: Biocomputing 2025: Proceedings of the Pacific Sympo- sium, World Scientific, 2024, pp. 199–214

    2025

  28. [36]

    L. M. Amugongo, P. Mascheroni, S. Brooks, S. Doering, J. Seidel, Re- trieval augmented generation for large language models in healthcare: A systematic review, PLOS Digital Health 4 (6) (2025) e0000877

    2025

  29. [37]

    Zhang, Y

    S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, et al., Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs, arXiv preprint arXiv:2303.00915 (2023)

    2023 arXiv

  30. [38]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., Segment anything, in: Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  31. [39]

    Alsentzer, J

    E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng, D. Jin, T. Nau- mann, M. McDermott, Publicly available clinical bert embeddings, arXiv preprint arXiv:1904.03323 (2019)

    1904 arXiv

  32. [40]

    Zhang, Y

    S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, et al., A multimodal biomedical foundation model trained from fifteen million image–text pairs, NEJM AI 2 (1) (2025) AIoa2400640

    2025

  33. [41]

    P. Deka, A. Jurek-Loughrey, D. P, Evidence extraction to validate med- ical claims in fake news detection, in: International conference on health information science, Springer, 2022, pp. 3–15. 44

    2022

  34. [42]

    Yasunaga, J

    M. Yasunaga, J. Leskovec, P. Liang, Linkbert: Pretraining language models with document links, arXiv preprint arXiv:2203.15827 (2022)

    2022 arXiv

  35. [43]

    K. R. Kanakarajan, B. Kundumani, M. Sankarasubbu, Bioelectra: pre- trained biomedical text encoder using discriminators, in: Proceedings of the 20th workshop on biomedical language processing, 2021, pp. 143– 154

    2021

  36. [44]

    Q. Jin, W. Kim, Q. Chen, D. C. Comeau, L. Yeganova, W. J. Wilbur, Z. Lu, Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval, Bioin- formatics 39 (11) (2023) btad651

    2023

  37. [45]

    Bannur, S

    S. Bannur, S. Hyland, Q. Liu, F. Perez-Garcia, M. Ilse, D. C. Castro, B. Boecking, H. Sharma, K. Bouzid, A. Thieme, et al., Learning to exploit temporal structure for biomedical vision-language processing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  38. [46]

    Zhang, Z

    Y. Zhang, Z. Shen, R. Jiao, Segment anything model for medical image segmentation: Current applications and future directions, Computers in Biology and Medicine 171 (2024) 108238

    2024

  39. [47]

    Rohanian, M

    O. Rohanian, M. Nouriborji, H. Jauncey, S. Kouchaki, F. Nooralahzadeh, L. Clifton, L. Merson, D. A. Clifton, I. C. C. Group, et al., Lightweight transformers for clinical natural language processing, Natural language engineering 30 (5) (2024) 887–914

    2024

  40. [48]

    Q. Lu, D. Dou, T. Nguyen, Clinicalt5: A generative language model for clinical text, in: Findings of the Association for Computational Linguis- tics: EMNLP 2022, 2022, pp. 5436–5443

    2022

  41. [49]

    G. Ao, M. Chen, J. Li, H. Nie, L. Zhang, Z. Chen, Comparative analysis of large language models on rare disease identification, Orphanet Journal of Rare Diseases 20 (1) (2025) 150

    2025

  42. [50]

    Y. Li, Q. Yang, F. L. Wang, L.-K. Lee, Y. Qu, T. Hao, Asymmetric cross-modal attention network with multimodal augmented mixup for medical visual question answering, Artificial Intelligence in Medicine 144 (2023) 102667. 45

    2023

  43. [51]

    Z. Deng, W. Gao, C. Chen, Z. Niu, Z. Gong, R. Zhang, Z. Cao, F. Li, Z. Ma, W. Wei, et al., Ophglm: An ophthalmology large language-and- vision assistant, Artificial Intelligence in Medicine 157 (2024) 103001

    2024

  44. [52]

    Huang, X

    J. Huang, X. Qi, X. Cheng, M. Wang, H. Ju, W. Ding, D. Zhang, Mmf- nns: Multi-modal multi-granularity fusion neural networks for brain net- works and its application to epilepsy identification, Artificial Intelligence in Medicine 157 (2024) 102990

    2024

  45. [53]

    Liu, L.-M

    B. Liu, L.-M. Zhan, X.-M. Wu, Contrastive pre-training and representa- tion distillation for medical visual question answering based on radiology images, in: International conference on medical image computing and computer-assisted intervention, Springer, 2021, pp. 210–220

    2021

  46. [54]

    Y. Zhou, J. Mei, Y. Yu, T. Syeda-Mahmood, Medical visual ques- tion answering using joint self-supervised learning, arXiv preprint arXiv:2302.13069 (2023)

    2023 arXiv

  47. [55]

    M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, P. Rajpurkar, Med-flamingo: a multimodal medi- cal few-shot learner, in: Machine Learning for Health (ML4H), PMLR, 2023, pp. 353–367

    2023

  48. [56]

    Z. Wang, Z. Wu, D. Agarwal, J. Sun, Medclip: Contrastive learning from unpaired medical images and text, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, Vol. 2022, 2022, p. 3876

    2022

  49. [57]

    O. C. Thawakar, A. M. Shaker, S. S. Mullappilly, H. Cholakkal, R. M. Anwer, S. Khan, J. Laaksonen, F. Khan, Xraygpt: Chest radiographs summarization using large medical vision-language models, in: Proceed- ings of the 23rd workshop on biomedical natural language processing, 20...

    2024

  50. [58]

    J. Zhou, X. He, L. Sun, J. Xu, X. Chen, Y. Chu, L. Zhou, X. Liao, B. Zhang, S. Afvari, et al., Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4, Nature Communica- tions 15 (1) (2024) 5649. 46

    2024

  51. [59]

    G. Liu, J. He, P. Li, Z. Zhao, S. Zhong, Cross-modal self-supervised vision language pre-training with multiple objectives for medical vi- sual question answering, Journal of Biomedical Informatics 160 (2024) 104748. 47

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.