REVIEW 4 major objections 5 minor 30 references
A Multimodal Approach to The Detection and Classification of Skin Diseases
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Combining a skin image's top five guesses with a fine-tuned language model reading a patient's symptom story raises 26-class skin disease diagnosis accuracy to 91.2 percent.
desk verdict A useful aggregated dataset and an interesting LLM training trick, but the 91.2% multimodal accuracy is untested because no image–narrative pair comes from the same patient. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is Chain of Options, a fine-tuning strategy for sequence-classification LLMs. Instead of asking the model to pick the correct disease from 26 options in one step, training presents random subsets of the options and teaches the model to remove the k least likely diseases, shrinking the list until one prediction remains; this reframes a 26-way choice into repeated elimination of unlikely options driven by single symptom mismatches. Around it, the pipeline wraps an implicit options list naming the 26 diseases in the prompt, the top-N predictions from the ResNet-50 image model concatenated to the patient narrative (with a training-time chance of omitting the true class, matching the image model's error rate), and LoRA fine-tuning of Llama-7B to keep the model trainable on consumer hardware. Image-side optimizations—augmentation with random crop and rotation, 75% frozen pretrained weights, and 300×300 resolution—are what lift the image model from 70% to 80.1% top-1.
What would settle it
Run the published pipeline unchanged on a newly collected set of 100-200 real patient narratives (same 26 diseases, with images) that were not generated by an LLM; if accuracy drops well below 91.2% or the drop concentrates in classes whose synthetic stories share distinctive keywords, the synthetic-narrative assumption is falsified. A second check is to ablate the options list and top-5 predictions to see how much of the gain comes from label leakage through prompt wording rather than genuine understanding.
Extended reading notes
Core claim
The paper's central claim is that a multimodal pipeline—a convolutional image classifier (ResNet-50, 75% of layers frozen, 300×300 input) whose top-5 predictions are concatenated into a prompt for a LoRA-fine-tuned Llama-7B language model—achieves 91.2% accuracy across 26 skin disease classes when the prompt also contains the patient's symptom narrative and an implicit list of the 26 candidate diseases. The image-only ceiling on this dataset is 80.1% top-1, the text-only ceiling is 89.7% with the options list, and combining modalities in the right configuration surpasses both. The paper also introduces 'Chain of Options,' a fine-tuning strategy that trains the LLM to eliminate the least likely diseases one batch at a time, rather than picking one class from the full list at once; with top-5 image recommendations, this configuration gives the reported 91.2%. The authors interpret the result as evidence that accessible image-plus-text information can support earlier diagnosis and complement clinicians, not replace them.
Load-bearing premise
The result stands on the assumption that the 260 ChatGPT-generated patient narratives, produced by rewriting Google symptom lists as first-person stories, are a faithful stand-in for how real patients describe their symptoms; if real narratives phrase symptoms differently, the 91.2% accuracy measured on synthetic validation stories will not transfer to clinical use.
Editorial extensions
If this is right
- A patient with only a smartphone photo and a typed symptom description could receive a 26-class differential diagnosis at 91.2% accuracy, versus 80.1% from the image alone.
- Clinicians could use the image model's top-5 list as a triage aid, letting the language model arbitrate between visually similar diseases that differ mainly in symptoms.
- The new 26-class multimodal dataset gives future work a harder benchmark than existing 10-class image-only sets, because accuracy drops across all models when moving to it.
- The Chain of Options trick could be applied to any multiclass LLM task where the label set is small enough to fit in a prompt and options can be rejected incrementally.
Reading between the lines
- Editorial inference: the 91.2% figure should be treated as an upper bound until the pipeline is tested on narratives written by real patients; synthetic stories may align more cleanly with the symptom keywords the model was trained on.
- Editorial inference: a natural next experiment is to replace the 10 ChatGPT stories per class with a few hundred real patient narratives from teledermatology and measure the drop, which would directly quantify the synthetic-to-real gap.
- Editorial inference: because Chain of Options only needs a label list and text input, the same architecture could transfer to other symptom-heavy diagnoses such as headache or abdominal pain triage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multimodal skin disease classification system that combines a ResNet-50 image classifier with a fine-tuned Llama-7B large language model. The authors construct a 26-class image dataset from three Kaggle sources (36,995 images) and generate 260 patient narratives with ChatGPT from Google symptom lists (10 per class). They introduce a fine-tuning strategy called 'Chain of Options' and report that, when the LLM receives the image model's top-5 predictions, the combined system achieves 91.2% accuracy on a 78-narrative held-out split (Table 11). The abstract and conclusions claim this is 'state of the art accuracy' and that the system outperforms human dermatologists in certain cases.
Significance. If the central claim were supported, the paper would offer a practical low-cost multimodal diagnostic tool and a novel LLM fine-tuning method. The image dataset assembled from public Kaggle sources across 26 classes is a useful resource, and the systematic comparison of vision architectures and augmentation strategies is a solid engineering contribution. However, the central result is not supported by the evaluation as designed: the text and image modalities are never paired at the patient level, the narratives are generated from the same labels that the model must predict, and hyperparameters are tuned on the same validation split used to report the headline number. The paper contains no machine-checked proofs, no released code or data, and no comparison that would justify the 'outperforms human dermatologists' claim.
major comments (4)
- [§5.1.1, §5.1.2, §5.4, Table 11] The evaluation does not measure multimodal diagnosis because no validation sample pairs an image and a narrative from the same patient. Section 5.1.2 says the 260 narratives were generated by ChatGPT from disease symptom lists, with no patient-level link to any of the 36,995 images described in Section 5.1.1. In Section 5.4, the image model's top-N predictions are appended to the narrative, but the manuscript never states that the image belongs to the patient who produced the narrative. Consequently, the 91.2% accuracy in Table 11 measures a different task: classifying a narrative while receiving a class hint from an image of an unrelated patient with the same label. This is an internal validity failure that directly undermines the abstract's claim of diagnosing a patient 'given just an image of the afflicted area as well as a patient description.'
- [§5.1.2, §5.3.1] The text evaluation is circular by construction. The 260 patient narratives were generated by ChatGPT from symptom lists retrieved per disease label; the label is the ground truth used to generate the input. The validation split of 78 narratives therefore contains text that was authored from the correct answer, so high text-only accuracy may reflect keyword memorization rather than generalization to real patient descriptions. The paper provides no evidence that these synthetic narratives are representative of how patients describe symptoms, and the phrase 'associated patient narratives' in the abstract is misleading because the narratives are not associated with any specific patient or image.
- [§5.2.4, §5.2.5, §5.3.3, §5.4] Hyperparameters are selected on the same validation split used to report the final accuracy, and no separate test set or repeated-split statistics are provided. Image resolution (300x300), frozen-layer percentage (75%), top-5 prediction count, LoRA rank, and Chain of Options removal size k are all tuned on the validation split, so the reported 91.2% is an optimistic estimate of performance. With only 78 validation narratives, a 91.2% accuracy corresponds to roughly 71 correct samples, and the paper reports no confidence intervals, error bars, or significance tests, making the 'outperforms human dermatologists' claim in Section 7.2.4 unsupported.
- [§7.2.4, §8] The claim of 'state of the art accuracy 91%' and 'outperforms human dermatologists in certain cases' is not backed by any comparative experiment. No prior multimodal method is evaluated on the same 26-class task with the same inputs, and no human study is conducted or cited for this dataset. The statement in Section 7.2.4 that the result 'outperforms human dermatologists' appears to extrapolate from unrelated published work (e.g., reference 1) without any direct or indirect comparison, so it should be removed or substantially qualified.
minor comments (5)
- [Table 4] The row 'Baseline Standard N/A 84' is undefined; the reader cannot tell what model or dataset this baseline refers to, and it conflicts with the text in Section 7.1 where the authors state their augmentations improved accuracy 'from 70% to 76%' before reaching 82%.
- [§5.3.3] The Chain of Options method is described qualitatively as removing k options, but the value of k used in experiments is never reported; this parameter is listed as a free parameter in the tuning process, and its chosen value should be stated.
- [Table 2] The column labeled 'Symptoms from ChatGPT' contains disease descriptions and background text rather than patient-reported symptoms; for BCC and BKL the entries read as general medical summaries, which is confusing given that the narratives in Table 3 are supposed to be generated from symptom keywords.
- [§10] The data sharing and peer review statements say 'Please see upload' but no upload is provided with the manuscript; the paper also does not include code or a link to the assembled dataset, which limits reproducibility.
- [Figures 2 and 3] Figure 2 lacks axis labels and units, and Figure 3 is a confusion matrix without a color scale or explicit class ordering, making it difficult to interpret the reported errors.
Circularity Check
The 91.2% multimodal accuracy is a validation-set artifact: narratives are generated from the target labels, hyperparameters are tuned on the same validation split, and no same-patient image-text pair exists, so the reported prediction reduces to label-derived inputs.
-
self definitional
[Section 5.1.2 (Text Data from Disease Symptoms), Table 3]
"The symptoms of each skin disease were obtained through Google. Since the symptoms of each skin disease cannot be directly input into an LLM, this study uses ChatGPT to generate a story to describe a patient’s symptoms based on select symptoms of the disease. The final dataset consists of 10 stories per skin disease with a total of 260 text data."
The input narrative is constructed by prompting ChatGPT with the gold disease's symptom keywords (Table 3: 'Pretending you are a patient, please construct a one paragraph patient narrative using these symptoms: "keywords"'). The narrative is therefore a function of the target label: X = LLM(symptoms(Y)). The validation stories come from the same generative process, so the model is tested on inputs whose content is definitionally derived from the class it must predict. Reported accuracy (up to 91.2%) measures how well the LLM recognizes the exact label-specific keywords inserted at generation time, not whether it can diagnose a real patient from an independent description.
-
fitted input called prediction
[Section 5.3.1 and Section 7.2.4, Tables 9-11]
"a final skin disease classification accuracy of 91% was achieved with the image model recommending the top 5 candidate diseases and the LLM adding these candidates to the input along with chain of options fine-tuning."
The text dataset is split 70:30 into train and validation (Section 5.3.1), and the same validation split is used to select the prediction count (1/3/5), Chain of Options versus normal training, image resolution, and frozen-layer percentage. Table 11 then reports the best configuration's accuracy on that same validation split as the final 91.2%. No separate held-out test set is described. The headline number is therefore a maximum over validation configurations on the same data used for selection, so it is a fitted statistic renamed as a prediction rather than an unbiased estimate.
1 more flagged steps
-
other
[Section 5.4 (Combining Image and Text Classification), Sections 5.1.1-5.1.2]
"the top N most likely predictions from the image model would be used as an initial prediction which are then included alongside the patient narrative as input into the LLM."
The 260 narratives are generated from symptom lists only (Section 5.1.2) and have no patient-level link to the 36,995 images (Section 5.1.1). To evaluate 'Resnet + Llama' on a validation narrative of class C, the image-model top-N must be taken from a different patient's image whose label is also C. The top-5 list therefore contains the correct class with probability equal to the image model's 95.2% top-5 recall (Section 7.2.2), giving the LLM a target-derived hint before it reads the narrative. The reported 91.2% is not an accuracy for the claimed task—image and narrative from the same patient—because no such sample exists in the dataset; it is an accuracy for classifying a label-derived narrative with a same-label image hint.
full rationale
The image-only benchmark (80.1% top-1 on 36,995 real Kaggle images) and the Chain of Options fine-tuning procedure have independent content and are not themselves circular. However, the paper's central claim—91.2% accuracy for diagnosing a patient from an image plus a patient narrative—is not supported by a valid multimodal test. The narratives are generated by ChatGPT from the same disease labels the model must predict, so training and validation inputs are constructed from the ground truth. The final configuration is selected on the same 70:30 validation split that produces the reported number, and no same-patient image-text pair exists to evaluate the actual multimodal task. The text-only Llama baseline with options already reaches 89.7%, so the 91.2% result is largely carried by label-derived symptom keywords, not by genuine fusion of independent image and text evidence. These issues are specific, quotable reductions of the evaluation to its own inputs rather than a general concern about consensus.
Assumptions & free parameters
free parameters (5)
- Image resolution =
300 x 300 pixels
- Frozen layer percentage =
75%
- Number of image model predictions N =
5
- Chain of Options removal size k =
Not reported
- LoRA adaptation rank =
Not reported
assumptions (4)
- domain assumption The three public Kaggle datasets can be merged into a single 26-class label set by manual checking and duplicate hashing.
- ad hoc to paper ChatGPT-generated narratives from Google symptom lists faithfully represent patient-reported symptoms for each disease.
- domain assumption The 80/20 image split and 70/30 text split are unbiased test sets despite hyperparameter selection on the same splits.
- domain assumption The image model's top-5 candidates are a valid conditioning signal for the LLM in real diagnosis.
Cite this review
Pith. "Pith review of A Multimodal Approach to The Detection and Classification of Skin Diseases." pith.science (2026). https://pith.science/paper/KRAKTPGD
@misc{pith2026241113855,
author = {Pith},
title = {Pith review of: A Multimodal Approach to The Detection and Classification of Skin Diseases},
year = {2026},
howpublished = {\url{https://pith.science/paper/KRAKTPGD}},
note = {Machine review of arXiv:2411.13855}
}
read the original abstract
According to PBS, nearly one-third of Americans lack access to primary care services, and another forty percent delay going to avoid medical costs. As a result, many diseases are left undiagnosed and untreated, even if the disease shows many physical symptoms on the skin. With the rise of AI, self-diagnosis and improved disease recognition have become more promising than ever; in spite of that, existing methods suffer from a lack of large-scale patient databases and outdated methods of study, resulting in studies being limited to only a few diseases or modalities. This study incorporates readily available and easily accessible patient information via image and text for skin disease classification on a new dataset of 26 skin disease types that includes both skin disease images (37K) and associated patient narratives. Using this dataset, baselines for various image models were established that outperform existing methods. Initially, the Resnet-50 model was only able to achieve an accuracy of 70% but, after various optimization techniques, the accuracy was improved to 80%. In addition, this study proposes a novel fine-tuning strategy for sequence classification Large Language Models (LLMs), Chain of Options, which breaks down a complex reasoning task into intermediate steps at training time instead of inference. With Chain of Options and preliminary disease recommendations from the image model, this method achieves state of the art accuracy 91% in diagnosing patient skin disease given just an image of the afflicted area as well as a patient description of the symptoms (such as itchiness or dizziness). Through this research, an earlier diagnosis of skin diseases can occur, and clinicians can work with deep learning models to give a more accurate diagnosis, improving quality of life and saving lives.
Figures
Reference graph
Works this paper leans on
-
[1]
Conception and design: All authors
-
[2]
Administrative support: E Yang
-
[3]
Provision of study materials or patients: E Yang
-
[4]
Collection and assembly of data: A Yang
-
[5]
Data analysis and interpretation: A Yang
-
[6]
Manuscript writing: All authors
-
[7]
A Multimodal Approach to The Detection and Classification of Skin Diseases
Final approval of manuscript: All authors 1 arXiv:2411.13855v1 [eess.IV] 21 Nov 2024 A Multimodal Approach to The Detection and Classification of Skin Diseases 2 Abstract Background: According to PBS, nearly one-third of Americans lack access to primary care services, and another forty percent delay going to avoid medical costs. As a result, many diseases...
work page Pith review arXiv 2024
-
[8]
Jain A, Way D, Gupta V , Gao Y , de Oliveira Marinho G, Hartford J, et al. Devel- opment and Assessment of an Artificial Intelligence–Based Tool for Skin Condition Di- agnosis by Primary Care Physicians and Nurse Practitioners in Teledermatology Prac- tices. JAMA Network Open [Internet]. 2021 Apr 28;4(4):e217249–9. Available from: https://jamanetwork.com/...
Show all 30 references
-
[9]
Dermatologist-level Classification of Skin Cancer with Deep Neural Networks
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature [Internet]. 2017 Jan 25;542(7639):115–8. Available from: https://www.nature.com/articles/nature21056
2017
-
[10]
VGG Very Deep Convolutional Networks (VGGNet) - What you need to know [Inter- net]
Boesch G. VGG Very Deep Convolutional Networks (VGGNet) - What you need to know [Inter- net]. viso.ai. 2021. Available from: https://viso.ai/deep-learning/vgg-very-deep-convolutional- networks/
2021
-
[11]
ResNet: The Basics and 3 ResNet Extensions [Internet]. Datagen. Available from: https://datagen.tech/guides/computer-vision/resnet/
-
[12]
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks [Internet]
Tan M, Le Q. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks [Internet]. 2019. Available from: https://arxiv.org/pdf/1905.11946.pdf
2019 arXiv
-
[13]
Artificial Intelligence in Dermatology: Challenges and Perspectives
Liopyris K, Gregoriou S, Dias J, Stratigos AJ. Artificial Intelligence in Dermatology: Challenges and Perspectives. Dermatology and Therapy. 2022 Oct 28;12(12)
2022
-
[14]
Artificial Intelligence in Dermatology Image Analysis: Current Developments and Future Trends
Li Z, Koban KC, Schenck TL, Giunta RE, Li Q, Sun Y . Artificial Intelligence in Dermatology Image Analysis: Current Developments and Future Trends. Journal of Clinical Medicine. 2022 Nov 18;11(22):6826
2022
-
[15]
Enhanced Deep Learn- ing Approach for Accurate Eczema and Psoriasis Skin Detection
Hammad M, Paweł Pławiak, ElAffendi M, Abd AA, Abdel A. Enhanced Deep Learn- ing Approach for Accurate Eczema and Psoriasis Skin Detection. Sensors. 2023 Aug 21;23(16):7295–5
2023
-
[16]
A deep learning, image based approach for automated diagnosis for inflammatory skin diseases
Wu H, Yin H, Chen H, Sun M, Liu X, Yu Y , et al. A deep learning, image based approach for automated diagnosis for inflammatory skin diseases. Annals of Translational Medicine. 2020 May;8(9):581–1
2020
-
[17]
A multimodal transformer to fuse images and metadata for skin disease classification
Cai G, Zhu Y , Wu Y , Jiang X, Ye J, Yang D. A multimodal transformer to fuse images and metadata for skin disease classification. The Visual Computer. 2022 May 5
2022
-
[18]
Exploring the potential of artificial intelligence in improving skin lesion diagnosis in primary care
Escalé-Besa A, Yélamos O, Vidal-Alaball J, Fuster-Casanovas A, Miró Catalina Q, Börve A, et al. Exploring the potential of artificial intelligence in improving skin lesion diagnosis in primary care. Scientific Reports [Internet]. 2023 Mar 15;13(1):4293. Available from: https:/...
2023
-
[19]
AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE [Internet]
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE [Internet]. 2021 Jun. Available from: https://arxiv.org/pdf/2010.11929.pdf
2021 arXiv
-
[20]
ChatGPT [Internet]
OpenAI. ChatGPT [Internet]. chat.openai.com. OpenAI; 2024. Available from: https://chat.openai.com/chat
2024
-
[21]
Deep Learning for Diagnostic Binary Classification of Multiple-Lesion Skin Diseases
Thomsen K, Christensen AL, Iversen L, Lomholt HB, Winther O. Deep Learning for Diagnostic Binary Classification of Multiple-Lesion Skin Diseases. Frontiers in Medicine. 2020 Sep 22;7
2020
-
[22]
Clinical Skin Disease Detection and Classification: Ensembled VGG [Internet]
Gogineni Saikiran, Narayana GS, Dhanrajnath Porika, Kumar GV . Clinical Skin Disease Detection and Classification: Ensembled VGG [Internet]. 2020. p. 827–47. Available from: https://www.researchgate.net/publication/345434604_Clinical_ Skin_Disease_Detection_and_Classification_...
2020
-
[23]
Deep Learning and Machine Learning Techniques of Diagnosis Der- moscopy Images for Early Detection of Skin Diseases
Abunadi I, Senan EM. Deep Learning and Machine Learning Techniques of Diagnosis Der- moscopy Images for Early Detection of Skin Diseases. Electronics. 2021 Dec 18;10(24):3158
2021
-
[25]
www.kaggle.com
Skin diseases image dataset [Internet]. www.kaggle.com. Available from: https://www.kaggle.com/datasets/ismailpromus/skin-diseases-image-dataset/data
-
[26]
www.kaggle.com
Dermnet [Internet]. www.kaggle.com. Available from: https://www.kaggle.com/datasets/shubhamgoel27/dermnet/data
-
[27]
www.kaggle.com
ISIC - 2019 [Internet]. www.kaggle.com. Available from: https://www.kaggle.com/datasets/bhanuprasanna/isic-2019
2019
-
[28]
LLaMA: Open and Efficient Foundation Language Models [Internet]
Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al. LLaMA: Open and Efficient Foundation Language Models [Internet]. 2023 Feb. Available from: https://arxiv.org/pdf/2302.13971.pdf
2023 arXiv
-
[29]
Mistral AI | Open source models [Internet]
AI M. Mistral AI | Open source models [Internet]. mistral.ai. Available from: https://mistral.ai/
-
[30]
pasted on
Falcon LLM [Internet]. 2024 [cited 2024 Aug 9]. Available from: https://falconllm.tii.ae/falcon- models.html 12 Tables Table 1: Class Statistics for Skin Disease Image Datasets Class Name (Skin Diseases) Number of Images Final Dataset1st Dataset 2nd Dataset 3rd Dataset Acne an...
2024
-
[2018]
what is important
On the private dataset, the proposed model achieved an accuracy of 0.816, which is better than other popular networks. On the dataset ISIC 2018, the proposed method achieves an accuracy of 0.94 and an AUC of 0.99. In conclusion, compared with state-of-the-art methods, the ViT ...
2018
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.