REVIEW 14 cited by
Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Driven by the large foundation models, the development of artificial intelligence has witnessed tremendous progress lately, leading to a surge of general interest from the public. In this study, we aim to assess the performance of OpenAI's newest model, GPT-4V(ision), specifically in the realm of multimodal medical diagnosis. Our evaluation encompasses 17 human body systems, including Central Nervous System, Head and Neck, Cardiac, Chest, Hematology, Hepatobiliary, Gastrointestinal, Urogenital, Gynecology, Obstetrics, Breast, Musculoskeletal, Spine, Vascular, Oncology, Trauma, Pediatrics, with images taken from 8 modalities used in daily clinic routine, e.g., X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Digital Subtraction Angiography (DSA), Mammography, Ultrasound, and Pathology. We probe the GPT-4V's ability on multiple clinical tasks with or without patent history provided, including imaging modality and anatomy recognition, disease diagnosis, report generation, disease localisation. Our observation shows that, while GPT-4V demonstrates proficiency in distinguishing between medical image modalities and anatomy, it faces significant challenges in disease diagnosis and generating comprehensive reports. These findings underscore that while large multimodal models have made significant advancements in computer vision and natural language processing, it remains far from being used to effectively support real-world medical applications and clinical decision-making. All images used in this report can be found in https://github.com/chaoyi-wu/GPT-4V_Medical_Evaluation.
Forward citations
Cited by 14 Pith papers
-
Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink
Adversarial images optimized to induce attention sink behavior increase hallucination rates in multiple MLLMs, including commercial APIs, without visibly degrading response quality.
-
Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models
On a new 65,464-item binary benchmark that swaps one medical term per caption, four medical multimodal models scored 49–62%, close to chance.
-
Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
A 3D encoder pretrained with GPT-4V slice captions and partial optimal transport alignment beats vision-only SSL baselines on several medical tasks, but a key evaluation dataset may overlap with pretraining.
-
Active Learning for Neurosymbolic Program Synthesis
The abstract claims a new active learning technique, constrained conformal evaluation (tool SmartLabel), that finds the ground-truth program in 98% of benchmarks, but the delivered full text is a different paper, leav...
-
MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
Current medical multimodal models, including GPT-4o and Claude 3.5 Sonnet, fail simple perceptual tasks on medical images that human experts solve almost perfectly.
-
Are MLMs Trapped in the Visual Room?
A new Reddit-derived sarcasm benchmark and two-tier evaluation show that multimodal models can perceive scenes accurately while still failing to grasp sarcastic intent.
-
How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?
Large language models complete only 29-67% of simulated radiology agent tasks, and prompting tricks and a simulated tool builder do not close the gap on complex workflows.
-
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
MMedPO weights preference-optimization training samples by clinical relevance scores, combining hallucinated text answers and locally noised lesion images, and reports improved medical VQA and report generation metrics.
-
MpoxVLM: A Vision-Language Model for Diagnosing Skin Lesions from Mpox Virus Infection
MpoxVLM reports top accuracy for mpox detection from skin images and clinical data, but the design feeds the answer into the model through a mpox-specific lesion stage feature, making the reported result unreliable.
-
Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement
A zero-shot LLM pipeline that converts audio-visual behavior into textual scripts and emotion descriptions detects ASD in two-year-olds with 95.24% F1, but the estimate is weakened by threshold tuning on the evaluation data.
-
Cross-Modal Consistency in Multimodal Large Language Models
GPT-4V answers identical questions much less accurately when they are presented as images than as text, even when it can extract the image content nearly perfectly.
-
Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognition
Recursive generation, evaluation, and refinement of discrete prompts tunes a black-box multimodal LLM to each user, raising personalized visual emotion recognition accuracy on Affection from 40.6% to 44.9%.
-
Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback
UMed-LVLM uses GPT-4V-generated abnormality data and abnormal-aware rewards to improve medical image diagnosis and localization.
-
SpatialFly: Implicit 3D Prior-Guided Visual Reparameterization for Continuous UAV Vision-and-Language Navigation
SpatialFly reparameterizes 2D visual tokens with implicit geometric priors and reports lower navigation error and higher success than prior UAV VLN systems on unseen splits.
Discussion (0). Continue with ORCID to comment.