Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis

Charmgil Hong; Jihyun Moon

REVIEW 2 major objections 4 minor 23 references

Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis

T0 review · 2 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that retrieval-augmented prompting with similar patient cases lifts a frozen vision-language model's melanoma F1-score from 0.3729 (zero-shot) to 0.6864 without fine-tuning.

desk verdict Retrieval-augmented prompting for melanoma diagnosis is a plausible new application, but the missing label-shuffle/removal control leaves label propagation as an alternative explanation for the reported F1 gain. read the letter →

arxiv 2509.08338 v1 pith:SHAKOM6J submitted 2025-09-10 cs.CV cs.AIcs.LG

Jihyun Moon , Charmgil Hong This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords melanomaclassificationvision-languagemodelretrieval-augmentedgenerationmultimodaldiagnosisdermoscopicimagesclinicalmetadatafew-shotpromptingerrorcorrection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a frozen, general-purpose vision-language model can diagnose melanoma more accurately if its prompt is augmented with retrieved examples of similar patients—each example an image, its clinical metadata, and its known diagnosis. The proposed method builds a joint embedding space with BERT for metadata and a CNN for dermoscopic images, retrieves the top two similar cases, and inserts them as image–metadata–label triplets into the VLM prompt. The reported gain is large: F1-score reaches 0.6864, up 0.2099 over the best early-fusion baseline and 0.3135 over the zero-shot VLM, with sensitivity more than doubling. A sympathetic reader would care because this suggests a route to clinically useful multimodal diagnosis that avoids costly fine-tuning and lets the reference database be updated independently of the model.

What carries the argument

The central object is the retrieval-augmented prompt: a structured instruction plus K retrieved image–metadata–label triplets plus the query case, all fed to a frozen vision-language model. Retrieval relies on a multimodal embedding built by concatenating a BERT vector for the serialized metadata with a ResNeXt-50 vector for the image, searched with FAISS for nearest neighbors. The attribute-value pair serialization (e.g., 'Age: 45, Sex: Female') is the input format that carries the clinical variables most explicitly and yields the best results.

What would settle it

Run the same retrieval-augmented prompts but shuffle the labels attached to the retrieved triplets, or remove them entirely. If F1 stays near 0.6864, the model is using the image–metadata context; if F1 collapses toward a k-NN majority vote on the same embeddings, the gain is label propagation, not reasoning.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that retrieval-augmented prompting—not parameter updates—is what lets a general-domain VLM handle the clinical specificity of melanoma diagnosis. Using BERT to encode serialized metadata, ResNeXt-50 to encode dermoscopic images, and FAISS to retrieve the top-K nearest patient cases, the method places labeled example triplets into the prompt of a frozen LLaVA v1.5 model. With K=2 and attribute-value pair serialization, it achieves an F1-score of 0.6864, recovering 74.74% of early-fusion false positives and 52.94% of early-fusion false negatives. The paper interprets this as the VLM performing analogical, case-based reasoning over the retrieve

Load-bearing premise

The reported gains are attributed to the VLM interpreting the retrieved cases as clinical context, but those cases include their labels in the prompt; if the model is only inheriting the majority label of the nearest retrieved examples, the method is a nearest-neighbor wrapper rather than retrieval-based reasoning.

Editorial extensions

If this is right

A frozen, off-the-shelf VLM can be adapted to a specialized clinical task without weight updates, avoiding fine-tuning costs and enabling local, privacy-preserving deployment.
The method acts as an error-correction layer: a substantial share of baseline false positives and false negatives are reclassified correctly when similar cases are retrieved and shown in the prompt.
Metadata serialization format is not neutral; attribute-value pairs outperform HTML and sentence formats under the same retrieval setup.
The image encoder used for retrieval matters: ResNeXt-50 embeddings retrieved cases with stronger visual and clinical alignment than EfficientNet-V2-M embeddings.
Because the retrieval index is separate from the model weights, the reference database can be extended with newly confirmed cases without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism is genuinely contextual rather than label-based, the same prompt design should transfer to other multimodal clinical tasks with structured metadata, such as diabetic retinopathy grading or chest X-ray triage.
The paper leaves the key control experiment unrun: shuffling or removing the labels on the retrieved triplets would separate true case-based reasoning from a nearest-neighbor majority-label shortcut.
Since the method is a wrapper around a retrieval database, its practical ceiling likely depends more on the breadth and balance of the reference set than on the VLM itself; curating a well-labeled library would be the natural next investment.
The method's 'no fine-tuning' property suggests a deployment model where a central reference case library can be audited and updated by clinicians independently of the model vendor.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a sensible application paper—RAG, CBIR, and frozen VLMs are all known pieces, and the authors combine them for melanoma classification with a clean serialization/encoder comparison. But the main result is not yet supported because the retrieved labels are in the prompt and there is no control that removes or shuffles them. Label propagation—a nearest-neighbor vote—could reproduce much of the reported F1 gain. The paper needs that ablation before we can believe the 'clinical reasoning' story.

What's genuinely new: the specific recipe of injecting image–metadata–label triplets into a frozen LLaVA prompt for melanoma, with a FAISS index over concatenated BERT+CNN embeddings. That's a new application, not a new mechanism. The comparisons across serialization formats (sentence, attribute-value, HTML) and image encoders are useful and reasonably reported. The error recovery tables give a decent sense of where gains come from. Credit for the F1 gain over baselines and for the qualitative examples, though those are anecdotal.

The soft spot is load-bearing. With K=2, the model sees two prior cases' ground-truth labels. Since retrieval embeddings are label-correlated, a kNN classifier using the same embeddings would likely come close to the reported performance. The paper doesn't report a label-free prompt, a shuffled-label prompt, random retrieval, or a kNN baseline. Without any of these, the observed gain is consistent with the model simply inheriting the majority retrieved label. That's the main issue. Secondary: hyperparameter selection for K isn't explicitly tied to the validation split, there are no error bars, and code/data aren't released. These are fixable.

Who this is for: researchers in medical VLM/RAG applications, and anyone teaching evaluation pitfalls in retrieval-augmented settings. The paper deserves a serious referee—the question is important and the fix is simple. I'd want the authors to add the label control and a kNN baseline before acceptance. If they do, it's a solid minor-to-moderate contribution.

Referee Report

2 major / 4 minor

Summary. The paper proposes a retrieval-augmented vision-language model (VLM) framework for binary melanoma classification on the SIIM-ISIC 2019 dataset. At inference, the query image and clinical metadata are embedded with CNN and BERT encoders, the top-K similar training cases are retrieved via FAISS, and these cases are inserted into the LLaVA prompt as image–metadata–label triplets. The best configuration (BERT + ResNeXt-50, attribute-value serialization, K=2) is reported to reach F1=0.6864, outperforming the best early-fusion baseline (F1=0.4765) and zero-shot LLaVA (F1=0.3729). The paper also reports error-recovery rates across serialization formats and image encoders, and concludes that retrieval-augmented prompting enables informed, context-based clinical reasoning without fine-tuning.

Significance. If the mechanism were validated, the paper would demonstrate a practical way to adapt general-purpose VLMs to a specialized clinical task without weight updates, with the added benefit of case-based explainability. The experimental design is systematic in comparing serialization formats and encoders, and the reported metrics are internally consistent. However, the central claim that the VLM reasons over retrieved multimodal context is not yet established: because the retrieved examples contain ground-truth labels, the observed gains could be explained by label propagation from the retrieved neighbors. The paper ships no code, no control experiments isolating the role of labels, and no nearest-neighbor baseline, so the load-bearing assumption of retrieval-based reasoning remains untested.

major comments (2)

[Section 2, Prompt Construction] The prompt consists of K retrieved examples as image–metadata–label triplets, where the labels are ground-truth training labels. With K=2, the VLM can achieve high classification performance simply by adopting the labels present in the prompt, especially because the retrieval embedding space is label-correlated. The paper reports no ablation with retrieved labels removed, shuffled, or replaced with incorrect labels. Without such controls, the reported gain (F1 0.6864 vs. 0.3729 zero-shot) does not uniquely support the claim that the VLM is performing multimodal reasoning over the retrieved cases; it may be performing nearest-neighbor label propagation. This is load-bearing for the paper's central claim and must be addressed experimentally.
[Section 3.1, Quantitative Evaluation] There is no kNN classifier baseline using the same retrieval embeddings. A simple kNN label-propagation method on the BERT+ResNeXt-50 embeddings would establish how much of the F1 improvement is attributable to retrieval similarity alone. Additionally, a random-retrieval condition with true labels would isolate whether semantic similarity, rather than the mere presence of labels, is responsible for the gains. These baselines are necessary to distinguish retrieval-augmented reasoning from label propagation.

minor comments (4)

[Table 1] The row for the best configuration has run-together numbers ('0.68646864 1104 591 418'), and encoder names are inconsistent ('ResNext-50' vs. 'ResNeXt-50'). Also, the table header contains stray symbols ('FP✓-') that should be cleaned.
[Section 3, Experimental Setup] The exact prompt template, decoding parameters (temperature, max tokens), and random seeds are not reported. This limits reproducibility, given that the entire method is prompt-based.
[Section 3.1] All results are based on a single stratified train/test split, with no confidence intervals or repeated-seed variation. Given the large F1 differences, this is not the primary issue, but reporting variance would strengthen the claims.
[Section 3.2] The qualitative analysis in Fig. 2 relies on visual examples that are not described in sufficient detail in the text. The claim that retrieved cases 'show strong alignment' would be more convincing with quantitative similarity metrics or attention visualizations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the retrieval-augmented VLM pipeline is an empirical in-context learning setup, and the label-propagation confound is a validity concern, not a circular derivation.

full rationale

The paper's central claim is an empirical comparison of retrieval-augmented VLM prompting against baselines. The derivation chain is not circular: the retrieval module encodes images and metadata with fixed pretrained encoders (BERT, ResNeXt-50/EfficientNet-V2-M), stores them in FAISS, and at inference inserts top-K retrieved image–metadata–label triplets into a frozen VLM prompt. No parameter of the VLM or retrieval module is fitted to the test-set labels, and no claim is derived from a self-citation: all cited works are external and none are by the authors. The only substantial concern—that the VLM may exploit the ground-truth labels present in the retrieved examples (label propagation) rather than reasoning over visual/textual similarity—is an experimental-control/alternative-explanation issue, not a circularity in the paper's reasoning. The paper does not define the prediction in terms of retrieved labels, nor does it fit a parameter and rename it a prediction. Therefore no circular step can be exhibited, and the appropriate score is 0. (A label-shuffle or kNN-control ablation would strengthen the causal interpretation, but its absence does not make the argument circular.)

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the standard dataset assumption plus two untested assumptions about data separation and model behavior. No new entities are introduced; the method combines existing components.

free parameters (1)

K (number of retrieved cases) = 2
Chosen empirically as best trade-off between relevance and noise; selection process is not clearly separated from the test set.

assumptions (3)

domain assumption SIIM-ISIC 2019 ground-truth labels are correct and provide the supervision signal.
The dataset labels are treated as ground truth for all metrics; no manual review or noise analysis is performed.
domain assumption The train/test split is lesion-independent with no duplicate or near-duplicate images across splits.
Retrieval could leak label information if the same lesion appears in both training and test; the paper does not describe deduplication.
ad hoc to paper The VLM uses the retrieved cases as clinical context rather than performing simple label propagation.
This is the implicit premise behind the claim that retrieval improves reasoning; it is not tested with a label-shuffle control.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis." pith.science (2026). https://pith.science/paper/SHAKOM6J

@misc{pith2026250908338,
  author       = {Pith},
  title        = {Pith review of: Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHAKOM6J}},
  note         = {Machine review of arXiv:2509.08338}
}

read the original abstract

Accurate and early diagnosis of malignant melanoma is critical for improving patient outcomes. While convolutional neural networks (CNNs) have shown promise in dermoscopic image analysis, they often neglect clinical metadata and require extensive preprocessing. Vision-language models (VLMs) offer a multimodal alternative but struggle to capture clinical specificity when trained on general-domain data. To address this, we propose a retrieval-augmented VLM framework that incorporates semantically similar patient cases into the diagnostic prompt. Our method enables informed predictions without fine-tuning and significantly improves classification accuracy and error correction over conventional baselines. These results demonstrate that retrieval-augmented prompting provides a robust strategy for clinical decision support.

Figures

Figures reproduced from arXiv: 2509.08338 by the authors.

**Figure 2.** Error cases corrected by our framework with retrieved cases. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 10 canonical work pages

[1]

In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining

Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next- generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 2623–2631 (2019)

2019
[2]

Journal of Cutaneous Medicine and Surgery28(1), 98–99 (2024)

Akrout, M., Cirone, K.D., Vender, R.: Evaluation of vision llms gtp-4v and llava for the recognition of features characteristic of melanoma. Journal of Cutaneous Medicine and Surgery28(1), 98–99 (2024)

work page 2024
[3]

In: 2020 25th International Conference on Pattern Recognition (ICPR)

Allegretti, S., Bolelli, F., Pollastri, F., Longhitano, S., Pellacani, G., Grana, C.: Supporting skin lesion diagnosis with content-based image retrieval. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 8053–8060. IEEE (2021)

work page 2020
[4]

Machine learning45, 5–32 (2001)

Breiman, L.: Random forests. Machine learning45, 5–32 (2001)

2001
[5]

Chen, J., Jiang, Y., Yang, D., Li, M., Wei, J., Qian, Z., Zhang, L.: Can llms’ tuning methods work in medical multimodal domain? In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 112–122. Springer (2024)

work page 2024
[6]

In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)

Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomed- ical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th intern...

work page 2017
[7]

arXiv preprint arXiv:1908.02288 (2019)

Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

arXiv 1908
[8]

In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186 (2019)

2019

Show all 23 references

[9]

arXiv preprint arXiv:2401.08281 (2024)

Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library. arXiv preprint arXiv:2401.08281 (2024)

2024 arXiv
[10]

nature 542(7639), 115–118 (2017)

Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. nature 542(7639), 115–118 (2017)

2017
[11]

Journal of Investigative Dermatology138(7), 1529–1538 (2018)

Han, S.S., Kim, M.S., Lim, W., Park, G.H., Park, I., Chang, S.E.: Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm. Journal of Investigative Dermatology138(7), 1529–1538 (2018)

2018
[12]

Advances in neural information processing systems 33, 9459–9474 (2020)

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, 9459–9474 (2020)

2020
[13]

Advances in neural information processing systems36, 34892–34916 (2023)

Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

2023
[14]

arXiv preprint arXiv:1805.11730 (2018)

Liu, K., Li, Y., Xu, N., Natarajan, P.: Learn to combine modalities in multimodal deep learning. arXiv preprint arXiv:1805.11730 (2018)

2018 arXiv
[15]

the Journal of machine Learning research12, 2825–2830 (2011)

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the Journal of machine Learning research12, 2825–2830 (2011)

2011
[16]

CA: a cancer journal for clinicians74(1), 12–49 (2024)

Siegel, R.L., Giaquinto, A.N., Jemal, A.: Cancer statistics, 2024. CA: a cancer journal for clinicians74(1), 12–49 (2024)

2024
[17]

In: International conference on machine learning

Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)

2019
[18]

Team,C.:Chameleon:Mixed-modalearly-fusionfoundationmodels.arXivpreprint arXiv:2405.09818 (2024)

2024 arXiv
[19]

British Journal of Dermatology181(1), 155–165 (2019)

Tschandl, P., Argenziano, G., Razmara, M., Yap, J.: Diagnostic accuracy of content-based dermatoscopic image retrieval with deep classification features. British Journal of Dermatology181(1), 155–165 (2019)

2019
[20]

Scientific data5(1), 1–9 (2018)

Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data5(1), 1–9 (2018)

2018
[21]

In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: s...

2020
[22]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Xie,S.,Girshick,R.,Dollár,P.,Tu,Z.,He,K.:Aggregatedresidualtransformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1492–1500 (2017)

2017
[23]

Advances in Neural Information Processing Systems36, 46595–46623 (2023)

Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems36, 46595–46623 (2023)

2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining

Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next- generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 2623–2631 (2019)

2019

[2] [2]

Journal of Cutaneous Medicine and Surgery28(1), 98–99 (2024)

Akrout, M., Cirone, K.D., Vender, R.: Evaluation of vision llms gtp-4v and llava for the recognition of features characteristic of melanoma. Journal of Cutaneous Medicine and Surgery28(1), 98–99 (2024)

work page 2024

[3] [3]

In: 2020 25th International Conference on Pattern Recognition (ICPR)

Allegretti, S., Bolelli, F., Pollastri, F., Longhitano, S., Pellacani, G., Grana, C.: Supporting skin lesion diagnosis with content-based image retrieval. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 8053–8060. IEEE (2021)

work page 2020

[4] [4]

Machine learning45, 5–32 (2001)

Breiman, L.: Random forests. Machine learning45, 5–32 (2001)

2001

[5] [5]

Chen, J., Jiang, Y., Yang, D., Li, M., Wei, J., Qian, Z., Zhang, L.: Can llms’ tuning methods work in medical multimodal domain? In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 112–122. Springer (2024)

work page 2024

[6] [6]

In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018)

Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomed- ical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th intern...

work page 2017

[7] [7]

arXiv preprint arXiv:1908.02288 (2019)

Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

arXiv 1908

[8] [8]

In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186 (2019)

2019