Pith. sign in

REVIEW 18 cited by

MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.02900 v3 pith:QZQKWTIQ submitted 2024-08-06 cs.CV

Yunfei Xie , Ce Zhou , Lang Gao , Juncheng Wu , Xianhang Li , Hong-Yu Zhou , Sheng Liu , Lei Xing
show 3 more authors
James Zou Cihang Xie Yuyin Zhou
This is my paper · ORCID
classification cs.CV
keywords multimodalannotationsmedtrinity-25mmodelsmultigranularcomprehensivedatasetlarge-scale
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets, which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI-description triplets without the need for any paired text descriptions. Specifically, data from over 30 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. We propose LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, achieving state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, MedTrinity-25M can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. We will make our dataset available.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine

    cs.AI 2025-08 conditional novelty 6.0 of 10

    Current medical multimodal models, including GPT-4o and Claude 3.5 Sonnet, fail simple perceptual tasks on medical images that human experts solve almost perfectly.

  2. Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoning

    cs.AI 2025-07 conditional novelty 6.0 of 10

    FundusExpert, an 8B ophthalmic MLLM trained on region-grounded cognitive-chain instructions, reports state-of-the-art QA and report-generation results, with a fitted data-scaling exponent of 0.068.

  3. Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A student CLIP model distilled from nine medical CLIP teachers outperforms its teachers across most of 58 biomedical benchmarks.

  4. KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations

    cs.CL 2025-06 conditional novelty 6.0 of 10

    KokushiMD-10 is a Japanese multimodal benchmark spanning ten healthcare licensing exams, on which no tested model passed all exams.

  5. Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An adversarial beam-search method generates over 6,000 bilingual question pairs that reliably make multilingual LLMs perform far worse in non-English languages than in English.

  6. MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization

    cs.CV 2024-12 conditional novelty 6.0 of 10

    MMedPO weights preference-optimization training samples by clinical relevance scores, combining hallucinated text answers and locally noised lesion images, and reports improved medical VQA and report generation metrics.

  7. VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A four-stage training recipe plus on-demand expert-model feedback lets a small open-source VLM beat much larger medical VLMs on several VQA, classification, and report generation benchmarks.

  8. Time-to-Event Pretraining for 3D Medical Imaging

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Pretraining on time-to-event labels from longitudinal EHRs improves 3D CT model predictions of future outcomes by up to 31.6% relative AUROC, while preserving diagnostic accuracy.

  9. TextSLIP: Text Self-Supervised CLIP for Medical Report Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Adding ESimCSE text contrastive learning to CLIP improves medical report generation BLEU scores on brain MRI over standard CLIP by about 1-3 points.

  10. Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models

    cs.CV 2026-07 reject novelty 5.0 of 10

    A synthetic chain-of-thought dataset generated from CT reports lets a 2D-pretrained medical MLLM improve on 3D CT spatial-reasoning benchmarks.

  11. MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence

    cs.CV 2026-03 reject novelty 5.0 of 10

    MEDIC-AD adds anomaly-aware and difference tokens to a medical VLM, claiming SOTA lesion detection, temporal tracking, and visual grounding; the zero-shot claim is undermined by likely train/test overlap.

  12. Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback

    cs.CL 2025-01 conditional novelty 5.0 of 10

    UMed-LVLM uses GPT-4V-generated abnormality data and abnormal-aware rewards to improve medical image diagnosis and localization.

  13. GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI

    cs.CV 2024-11 reject novelty 5.0 of 10

    GMAI-VL-5.5M is a new 5.5M-sample medical image-text dataset built from 219 datasets via GPT-4o annotation-guided generation, and GMAI-VL is a three-stage LLaVA-style model reporting SOTA numbers, though the evaluatio...

  14. Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation

    eess.IV 2025-06 conditional novelty 4.0 of 10

    MedRegion-CT integrates region-representative tokens, mask-driven segmentation tokens, and patient-specific attribute prompts into a multimodal LLM, reporting state-of-the-art scores on RadGenome-Chest CT report generation.

  15. Joint Generalized Cosine Similarity: A Novel Method for N-Modal Semantic Alignment Based on Contrastive Learning

    stat.AP 2025-05 conditional novelty 4.0 of 10

    The authors define a joint generalized cosine similarity from the Gram determinant and use it in a contrastive loss called GHA to align three or more modalities in one step.

  16. Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering

    cs.CV 2024-11 conditional novelty 4.0 of 10

    A domain-aware patch selection method (Path-RAG) improves open-ended pathology VQA recall, but the headline gain overstates the contribution of the domain guidance.

  17. From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine

    cs.AI 2025-02 conditional novelty 3.0 of 10

    A PRISMA-ScR scoping review of 144 studies finds the field shifting from text-only LLMs to multimodal AI in medicine, with evaluation and data diversity still the main bottlenecks.

  18. Multimodal Large Language Models for Medicine: A Comprehensive Survey

    cs.LG 2025-04 conditional novelty 2.0 of 10

    A comprehensive review cataloging medical MLLMs, their uses in report generation, diagnosis, and treatment, and the challenges of accuracy, hallucination, fairness, privacy, and deployment.

Pith tools