REVIEW 6 cited by
A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Zero-shot medical image classification is a critical process in real-world scenarios where we have limited access to all possible diseases or large-scale annotated data. It involves computing similarity scores between a query medical image and possible disease categories to determine the diagnostic result. Recent advances in pretrained vision-language models (VLMs) such as CLIP have shown great performance for zero-shot natural image recognition and exhibit benefits in medical applications. However, an explainable zero-shot medical image recognition framework with promising performance is yet under development. In this paper, we propose a novel CLIP-based zero-shot medical image classification framework supplemented with ChatGPT for explainable diagnosis, mimicking the diagnostic process performed by human experts. The key idea is to query large language models (LLMs) with category names to automatically generate additional cues and knowledge, such as disease symptoms or descriptions other than a single category name, to help provide more accurate and explainable diagnosis in CLIP. We further design specific prompts to enhance the quality of generated texts by ChatGPT that describe visual medical features. Extensive results on one private dataset and four public datasets along with detailed analysis demonstrate the effectiveness and explainability of our training-free zero-shot diagnosis pipeline, corroborating the great potential of VLMs and LLMs for medical applications.
Forward citations
Cited by 6 Pith papers
-
Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research
A systematic review of brain imaging foundation models covering 86 models and 161 datasets, with a performance tournament, dataset atlas, and duplicated-data warnings.
-
MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images
A concept-guided distillation method lets a fundus-image model learn from unpaired OCT scans during training, improving retinal disease classification when only fundus photos are available at test time.
-
BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models
BiomedCoOp improves few-shot biomedical image classification by aligning learnable prompts with selectively pruned LLM-generated prompt ensembles and distilling their knowledge into BiomedCLIP.
-
Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models
Fair-MoE reports improved accuracy and fairness on Harvard-FairVLMed for some protected attributes by adding sparse mixture-of-experts layers and a variance-based fairness loss to CLIP, but the all-attribute improveme...
-
Explainability for Vision Foundation Models: A Survey
A structured review of 122 papers on explainability for vision foundation models, with a taxonomy and the finding that quantitative evaluation is rare (36%).
-
A Survey of Medical Vision-and-Language Applications and Their Techniques
This survey reviews medical vision-and-language models across five tasks and organizes existing methods, datasets, and evaluation metrics without introducing new techniques.
Discussion (0). Continue with ORCID to comment.