{"id":"105ab8f1-63c5-462d-9269-014d8f9bf2e3","arxiv_id":"2411.13855","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"An image-plus-text skin disease classifier using ResNet-50 and a fine-tuned LLM reports 91.2% accuracy on 26 classes, but the evaluation uses synthetic ChatGPT stories and a tiny validation set.","lead":"Researchers combined skin photos with symptom stories to train a classifier that reports 91% accuracy across 26 skin diseases. The patient stories were made up by ChatGPT from disease labels, and the final test set has only 78 examples, so the headline result is not yet credible.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 91.2% accuracy is not an estimate of multimodal diagnosis because no validation sample pairs an image and a narrative from the same patient; the central claim is untested.","rationale":"The reader's weakest assertion is that ChatGPT-generated narratives are not a valid proxy for real patient narratives, and that hyperparameter tuning on the same split inflates the result. I agree those are serious problems, but the most load-bearing concern is more fundamental: the evaluation never combines an image and a narrative from the same patient, so the paper does not actually test the multimodal task it claims to solve. The narratives are generated from class-specific symptom lists, and the images are from independent Kaggle datasets; there is no patient identity linking them. Thus Table 11's 91.2% is at best a text-classification score with a class-correlated hint, not a multimodal diagnostic accuracy. This internal validity failure means the central claim would remain unsupported even if real patient narratives were used, because real narratives would still need to be paired with images of the same patient. The reader's concern about synthetic narratives is valid but addresses external validity; my concern addresses whether the experiment measures the claimed quantity at all. Since both problems independently justify rejection, I recommend keeping the reader's REJECT verdict.","tokens_in":13927,"tokens_out":5405,"duration_ms":55736,"concrete_test":"Audit the validation-set construction: list each of the 78 held-out narratives and the image paired with it, and check whether any pair shares a patient identifier. Because the narratives are ChatGPT outputs and images come from Kaggle, no such identifier can exist; if confirmed, the 91.2% accuracy does not measure the claimed image-plus-narrative task. As a complementary check, rerun the final evaluation with the image model's top-5 predictions replaced by five random disease names; if accuracy remains near 91%, the image modality contributes nothing and the reported multimodal gain is an artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the system diagnoses a patient's skin disease given an image of the afflicted area and a patient description of symptoms. This requires evaluation on (image, narrative) pairs from the same patient. The dataset construction makes this impossible: Section 5.1.2 creates 260 ChatGPT-generated narratives from Google symptom lists (10 per class), while Section 5.1.1 aggregates 36,995 images from three separate Kaggle datasets. No patient-level linkage exists between any image and any narrative. Yet Table 11 reports 'Resnet + Llama' accuracy with top-5 image predictions. To form validation samples, the authors must pair each narrative with an image of the same disease, not the same patient, because the narrative has no associated image. Section 5.4 describes adding the image model's top-N predictions to the narrative but never states that the image belongs to the same patient. Consequently, the 91.2% result measures a different task: classify a narrative while receiving a class hint from an image of an unrelated patient with the same label. Since the narratives are generated from class-specific symptom lists, the LLM can identify the class from text alone; the image list may be redundant or even ignored. The abstract's claim of diagnosing 'given just an image of the afflicted area as well as a patient description' is therefore not supported by any experiment in the paper. This is an internal validity failure, independent of whether ChatGPT narratives represent real patients.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal skin disease classification system that combines a ResNet-50 image classifier with a fine-tuned Llama-7B large language model. The authors construct a 26-class image dataset from three Kaggle sources (36,995 images) and generate 260 patient narratives with ChatGPT from Google symptom lists (10 per class). They introduce a fine-tuning strategy called 'Chain of Options' and report that, when the LLM receives the image model's top-5 predictions, the combined system achieves 91.2% accuracy on a 78-narrative held-out split (Table 11). The abstract and conclusions claim this is 'state of the art accuracy' and that the system outperforms human dermatologists in certain cases.","tokens_in":14234,"tokens_out":3146,"duration_ms":31213,"significance":"If the central claim were supported, the paper would offer a practical low-cost multimodal diagnostic tool and a novel LLM fine-tuning method. The image dataset assembled from public Kaggle sources across 26 classes is a useful resource, and the systematic comparison of vision architectures and augmentation strategies is a solid engineering contribution. However, the central result is not supported by the evaluation as designed: the text and image modalities are never paired at the patient level, the narratives are generated from the same labels that the model must predict, and hyperparameters are tuned on the same validation split used to report the headline number. The paper contains no machine-checked proofs, no released code or data, and no comparison that would justify the 'outperforms human dermatologists' claim.","major_comments":[{"comment":"The evaluation does not measure multimodal diagnosis because no validation sample pairs an image and a narrative from the same patient. Section 5.1.2 says the 260 narratives were generated by ChatGPT from disease symptom lists, with no patient-level link to any of the 36,995 images described in Section 5.1.1. In Section 5.4, the image model's top-N predictions are appended to the narrative, but the manuscript never states that the image belongs to the patient who produced the narrative. Consequently, the 91.2% accuracy in Table 11 measures a different task: classifying a narrative while receiving a class hint from an image of an unrelated patient with the same label. This is an internal validity failure that directly undermines the abstract's claim of diagnosing a patient 'given just an image of the afflicted area as well as a patient description.'","section":"§5.1.1, §5.1.2, §5.4, Table 11"},{"comment":"The text evaluation is circular by construction. The 260 patient narratives were generated by ChatGPT from symptom lists retrieved per disease label; the label is the ground truth used to generate the input. The validation split of 78 narratives therefore contains text that was authored from the correct answer, so high text-only accuracy may reflect keyword memorization rather than generalization to real patient descriptions. The paper provides no evidence that these synthetic narratives are representative of how patients describe symptoms, and the phrase 'associated patient narratives' in the abstract is misleading because the narratives are not associated with any specific patient or image.","section":"§5.1.2, §5.3.1"},{"comment":"Hyperparameters are selected on the same validation split used to report the final accuracy, and no separate test set or repeated-split statistics are provided. Image resolution (300x300), frozen-layer percentage (75%), top-5 prediction count, LoRA rank, and Chain of Options removal size k are all tuned on the validation split, so the reported 91.2% is an optimistic estimate of performance. With only 78 validation narratives, a 91.2% accuracy corresponds to roughly 71 correct samples, and the paper reports no confidence intervals, error bars, or significance tests, making the 'outperforms human dermatologists' claim in Section 7.2.4 unsupported.","section":"§5.2.4, §5.2.5, §5.3.3, §5.4"},{"comment":"The claim of 'state of the art accuracy 91%' and 'outperforms human dermatologists in certain cases' is not backed by any comparative experiment. No prior multimodal method is evaluated on the same 26-class task with the same inputs, and no human study is conducted or cited for this dataset. The statement in Section 7.2.4 that the result 'outperforms human dermatologists' appears to extrapolate from unrelated published work (e.g., reference 1) without any direct or indirect comparison, so it should be removed or substantially qualified.","section":"§7.2.4, §8"}],"minor_comments":[{"comment":"The row 'Baseline Standard N/A 84' is undefined; the reader cannot tell what model or dataset this baseline refers to, and it conflicts with the text in Section 7.1 where the authors state their augmentations improved accuracy 'from 70% to 76%' before reaching 82%.","section":"Table 4"},{"comment":"The Chain of Options method is described qualitatively as removing k options, but the value of k used in experiments is never reported; this parameter is listed as a free parameter in the tuning process, and its chosen value should be stated.","section":"§5.3.3"},{"comment":"The column labeled 'Symptoms from ChatGPT' contains disease descriptions and background text rather than patient-reported symptoms; for BCC and BKL the entries read as general medical summaries, which is confusing given that the narratives in Table 3 are supposed to be generated from symptom keywords.","section":"Table 2"},{"comment":"The data sharing and peer review statements say 'Please see upload' but no upload is provided with the manuscript; the paper also does not include code or a link to the assembled dataset, which limits reproducibility.","section":"§10"},{"comment":"Figure 2 lacks axis labels and units, and Figure 3 is a confusion matrix without a color scale or explicit class ordering, making it difficult to interpret the reported errors.","section":"Figures 2 and 3"}],"recommendation":"reject","confidential_remarks":"The central claim of the paper is untested because the validation protocol never pairs an image and a narrative from the same patient, and the narratives themselves are generated from the ground-truth labels. Fixing this would require collecting or obtaining a genuinely paired multimodal dataset and re-running the evaluation, which is beyond the scope of a revision. The image-only experiments and the assembled 26-class image dataset have some value, but they do not rescue the headline multimodal result. I recommend rejection, although the authors could resubmit a substantially revised manuscript that reports a properly paired evaluation and drops the unsupported clinical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper’s headline number—91.2% multimodal accuracy—is not a trustworthy estimate of multimodal diagnosis. The stress-test is correct: no image and narrative in the evaluation come from the same patient. The 260 narratives are ChatGPT rewrites of Google symptom lists; the 37K images are from three Kaggle datasets. Nothing links a given story to a given patient’s photo. So Table 11 actually measures: classify a synthetic narrative while receiving top-5 class hints from an image of some other patient with the same label. Since the stories are generated from the label, the LLM can often infer the class from the text alone, and the image hints are redundant. The abstract’s claim—diagnosing from an image plus a patient description—is simply not tested.\n\nThe paper does have real ingredients. The aggregated 26-class image dataset, if actually released, could be a useful benchmark; the image-only 80.1% top-1 is a plausible baseline. The Chain of Options fine-tuning idea—breaking 26-way classification into iterative removal of unlikely options—is worth a look, and the ablations show it sometimes helps. That’s a legitimate contribution to prompt/training design for sequence classification.\n\nThe soft spots are load-bearing. The text validation set has 78 synthetic stories, no error bars, no repeated splits. Hyperparameters (resolution, frozen fraction, top-N, LoRA rank) are tuned on the same validation split. The LLM is given an explicit options list containing the answer, and the image model’s top-5 predictions are essentially a second hint. So the 91.2% is circular plus in-sample tuned. The 'outperforms human dermatologists' claim is unsupported.\n\nI’d send this to a referee rather than desk-reject, because the dataset and the Chain of Options idea might be salvageable with a proper evaluation on real paired image–narrative data and an external clinical benchmark. But as written, the central result should not be cited. Reading group: maybe—it’s a clean example of how synthetic data and leakage can conspire to produce an impressive-looking number.\n\nRecommendation: major revision, or a new version with real paired data, before any publication.","headline":"A useful aggregated dataset and an interesting LLM training trick, but the 91.2% multimodal accuracy is untested because no image–narrative pair comes from the same patient.","tokens_in":14745,"tokens_out":3614,"would_cite":false,"duration_ms":36303,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining a skin image's top five guesses with a fine-tuned language model reading a patient's symptom story raises 26-class skin disease diagnosis accuracy to 91.2 percent.","keywords":["skin disease classification","multimodal deep learning","large language models","Chain of Options","patient narratives","transfer learning","LoRA fine-tuning","dermatology AI"],"falsifier":"Run the published pipeline unchanged on a newly collected set of 100-200 real patient narratives (same 26 diseases, with images) that were not generated by an LLM; if accuracy drops well below 91.2% or the drop concentrates in classes whose synthetic stories share distinctive keywords, the synthetic-narrative assumption is falsified. A second check is to ablate the options list and top-5 predictions to see how much of the gain comes from label leakage through prompt wording rather than genuine understanding.","tokens_in":13750,"feed_emoji":"🩺","tokens_out":6282,"duration_ms":48881,"temperature":0.7,"pith_summary":"This paper claims that skin disease diagnosis can be made substantially more accurate by combining two readily available inputs: a photo of the affected skin and a patient's own description of symptoms. On a new 26-class dataset of about 37,000 images plus 260 patient narratives, the best image model alone reaches 80.1% top-1 accuracy, but feeding the image model's top five candidate diseases into a fine-tuned Llama-7B language model, trained with a new 'Chain of Options' procedure, raises accuracy to 91.2%. The authors argue this beats image-only and text-only baselines and in certain cases outperforms human dermatologists. The claimed mechanism is that the language model, given a short list of plausible diseases rather than 26 open choices, can reject unlikely options step by step using symptom mismatches. A sympathetic reading treats the result as a proof of concept for accessible, multimodal, AI-assisted self-diagnosis, with the caveat that the patient narratives were machine-generated.","feed_headline":"Image plus patient story reaches 91% accuracy on 26 skin diseases","feed_subtitle":"A vision model's top-five guesses, handed to a language model reading the patient's story, beat either input alone.","key_machinery":"The load-bearing mechanism is Chain of Options, a fine-tuning strategy for sequence-classification LLMs. Instead of asking the model to pick the correct disease from 26 options in one step, training presents random subsets of the options and teaches the model to remove the k least likely diseases, shrinking the list until one prediction remains; this reframes a 26-way choice into repeated elimination of unlikely options driven by single symptom mismatches. Around it, the pipeline wraps an implicit options list naming the 26 diseases in the prompt, the top-N predictions from the ResNet-50 image model concatenated to the patient narrative (with a training-time chance of omitting the true class, matching the image model's error rate), and LoRA fine-tuning of Llama-7B to keep the model trainable on consumer hardware. Image-side optimizations—augmentation with random crop and rotation, 75% frozen pretrained weights, and 300×300 resolution—are what lift the image model from 70% to 80.1% top-1.","core_discovery":"The paper's central claim is that a multimodal pipeline—a convolutional image classifier (ResNet-50, 75% of layers frozen, 300×300 input) whose top-5 predictions are concatenated into a prompt for a LoRA-fine-tuned Llama-7B language model—achieves 91.2% accuracy across 26 skin disease classes when the prompt also contains the patient's symptom narrative and an implicit list of the 26 candidate diseases. The image-only ceiling on this dataset is 80.1% top-1, the text-only ceiling is 89.7% with the options list, and combining modalities in the right configuration surpasses both. The paper also introduces 'Chain of Options,' a fine-tuning strategy that trains the LLM to eliminate the least likely diseases one batch at a time, rather than picking one class from the full list at once; with top-5 image recommendations, this configuration gives the reported 91.2%. The authors interpret the result as evidence that accessible image-plus-text information can support earlier diagnosis and complement clinicians, not replace them.","pith_inferences":["Editorial inference: the 91.2% figure should be treated as an upper bound until the pipeline is tested on narratives written by real patients; synthetic stories may align more cleanly with the symptom keywords the model was trained on.","Editorial inference: a natural next experiment is to replace the 10 ChatGPT stories per class with a few hundred real patient narratives from teledermatology and measure the drop, which would directly quantify the synthetic-to-real gap.","Editorial inference: because Chain of Options only needs a label list and text input, the same architecture could transfer to other symptom-heavy diagnoses such as headache or abdominal pain triage."],"forward_implications":["A patient with only a smartphone photo and a typed symptom description could receive a 26-class differential diagnosis at 91.2% accuracy, versus 80.1% from the image alone.","Clinicians could use the image model's top-5 list as a triage aid, letting the language model arbitrate between visually similar diseases that differ mainly in symptoms.","The new 26-class multimodal dataset gives future work a harder benchmark than existing 10-class image-only sets, because accuracy drops across all models when moving to it.","The Chain of Options trick could be applied to any multiclass LLM task where the label set is small enough to fit in a prompt and options can be rejected incrementally."],"supporting_citations":[{"why":"Supplies the first ten-class skin-disease image dataset used for the baseline augmentation experiments.","marker":"[17]"},{"why":"Supplies the second image collection merged into the aggregated 26-class image set.","marker":"[18]"},{"why":"Supplies the third skin-image collection with eight disease classes merged into the aggregated data.","marker":"[19]"},{"why":"The ResNet-50 architecture that, after fine-tuning optimization, becomes the image model whose top-5 predictions feed the LLM.","marker":"[3]"},{"why":"The Llama-7B base model fine-tuned with LoRA and Chain of Options for the final 91.2% result.","marker":"[20]"},{"why":"The ChatGPT tool used to generate the 260 patient narratives from Google symptom lists.","marker":"[12]"},{"why":"An example of a prior two-class skin disease study whose limited scope the new 26-class multimodal dataset is designed to exceed.","marker":"[7]"}],"fun_headline_variants":["Image plus text lifts skin disease diagnosis to 91% accuracy","Patient narrative boosts skin AI to 91% on 26 conditions","Multimodal model reads photos and stories to hit 91% on skin diseases","Skin disease AI: image and story together reach 91% accuracy","Combining image and symptoms yields 91% diagnosis rate for skin diseases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the assumption that the 260 ChatGPT-generated patient narratives, produced by rewriting Google symptom lists as first-person stories, are a faithful stand-in for how real patients describe their symptoms; if real narratives phrase symptoms differently, the 91.2% accuracy measured on synthetic validation stories will not transfer to clinical use.","fun_headline_variants_meta":{"raw":{"variants":["Image plus text lifts skin disease diagnosis to 91% accuracy","Patient narrative boosts skin AI to 91% on 26 conditions","Multimodal model reads photos and stories to hit 91% on skin diseases","Skin disease AI: image and story together reach 91% accuracy","Combining image and symptoms yields 91% diagnosis rate for skin diseases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1596,"prompt_tokens":1073,"completion_tokens":523,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":689,"tokens_out":523,"duration_ms":4919,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:48:29.716148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published pipeline unchanged on a newly collected set of 100-200 real patient narratives (same 26 diseases, with images) that were not generated by an LLM; if accuracy drops well below 91.2% or the drop concentrates in classes whose synthetic stories share distinctive keywords, the synthetic-narrative assumption is falsified. A second check is to ablate the options list and top-5 predictions to see how much of the gain comes from label leakage through prompt wording rather than genuine understanding.","supporting_citations":[{"cited_title":"A multimodal transformer to fuse images and metadata for skin disease classification","cited_arxiv_id":null,"evidence_quote":"Supplies the first ten-class skin-disease image dataset used for the baseline augmentation experiments."},{"cited_title":"Exploring the potential of artificial intelligence in improving skin lesion diagnosis in primary care","cited_arxiv_id":null,"evidence_quote":"Supplies the second image collection merged into the aggregated 26-class image set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ResNet-50 architecture that, after fine-tuning optimization, becomes the image model whose top-5 predictions feed the LLM."},{"cited_title":"ChatGPT [Internet]","cited_arxiv_id":null,"evidence_quote":"The Llama-7B base model fine-tuned with LoRA and Chain of Options for the final 91.2% result."},{"cited_title":"A Multimodal Approach to The Detection and Classification of Skin Diseases","cited_arxiv_id":"2411.13855","evidence_quote":"An example of a prior two-class skin disease study whose limited scope the new 26-class multimodal dataset is designed to exceed."}],"review_version":1}