REVIEW 5 major objections 5 minor 33 references
Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Zero-shot medical image classification of rare events can be done with text-only LLM prompts built from expert knowledge, the paper argues.
desk verdict The abstract overclaims: the method is a modest but real extension of CuPL with domain knowledge, and the paper's own cross-center numbers contradict the SOTA claim, yet the body is honest and the work deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the HKETP generation pipeline. Raw expert knowledge from the literature is refined by an LLM into non-overlapping, discriminative knowledge components; each component is encoded as a software function that inspects an independent-component image and emits a natural-language statement with a satisfaction value, such as “Big red activation has asymmetry in hemispheres”. These statements form the Input prompt, which is combined with the refined knowledge as Context and a fixed instruction, and a generative LLM produces a YES/NO classification. A human verifier iteratively adjusts the thresholds and knowledge statements based on one patient’s data before the system is used on other patients.
What would settle it
Recompute the Center A results with the tuning patient removed from all reported metrics, or run CuKPL with thresholds fixed from the literature without any patient tuning; if the F1 score falls to the level of the untuned zero-shot baselines, the claimed advantage is not zero-shot.
Extended reading notes
Core claim
The central claim is that the bottleneck in zero-shot medical classification is not the vision model but the prompt: generic templates like “a photo of a {}” fail because they omit the discriminative rules experts use. CuKPL’s contribution is to operationalize those rules as human knowledge-embodied textual prompts (HKETP) and feed them to an instruction-tuned LLM, which classifies by reasoning over image-derived textual facts rather than pixels. The authors show this text-only pipeline surpasses a knowledge-plus-deep-learning system (DeepXSOZ) on the same data, while standard zero-shot methods such as CLIP and CuPL classify virtually everything as noise or resting-state network. They also claim the method is privacy-preserving because the LLM never receives the image itself, only the textual descriptions produced by the knowledge model.
Load-bearing premise
The method’s claimed zero-shot advantage depends on tuning the knowledge-model thresholds on a single patient’s data, and the paper does not show that this tuning patient is excluded from the evaluation set, so the reported gains could partly come from test-set leakage.
Editorial extensions
If this is right
- If CuKPL’s results hold, rare-event medical classification can be built without collecting large training sets, which matters for conditions with very few documented cases.
- The method can adapt to a new hospital with only one patient’s data used to tune thresholds, turning a domain-generalization problem into a cheaper domain-adaptation problem.
- Because the LLM sees only text, the same prompt-generation framework could be reused across imaging modalities such as MRI, CT, and fundus photography, provided the knowledge components can be computed from the image.
- The reported reduction in machine-marked SOZs (28 versus 43 for the knowledge-based EPIK system) suggests the output could prioritize which independent components a neurosurgeon reviews.
Reading between the lines
- The one-patient tuning step blurs the zero-shot claim: the paper never identifies the tuning patient, and the tuned thresholds effectively encode the decision rule, so a strict test would set thresholds from literature alone and report that as a true zero-shot variant.
- The pipeline should transfer to other rare events with rule-based radiological signs, such as the proliferative diabetic retinopathy grading named in the conclusion; a multi-disease benchmark would reveal whether the knowledge-encoding step generalizes or is specific to seizure onset zones.
- The privacy claim is partial: the LLM never sees raw images, but the textual HKETP descriptions derived from a specific patient’s image are still patient-derived data and may carry identifiable information, a residual risk the paper does not discuss.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CuKPL, a method that uses an LLM to convert domain knowledge from literature into structured prompts (HKETP) for zero-shot medical image classification, with a knowledge model encoding image features into text. It is evaluated on seizure onset zone (SOZ) detection from fMRI independent components, comparing with deep learning, knowledge-based, hybrid, and standard zero-shot baselines on Center A and cross-center Center B. The authors report that CuKPL improves accuracy/F1 over DeepXSOZ on Center A but underperforms DeepXSOZ on Center B, and they position the method as zero-shot and privacy-preserving.
Significance. The underlying idea—packaging expert knowledge into textual prompts so that an LLM can classify rare medical images without exposing the image—is attractive and, if rigorously validated, would be a useful privacy-preserving alternative. The paper makes a falsifiable claim ('outperforming state-of-the-art'), and the reported cross-center numbers directly test it. However, in its current form the evidence fails to support the central claim: the method is not zero-shot because thresholds are tuned on one patient's data, and it does not outperform the SOTA on the truly challenging cross-center setting. The code release and detailed dataset description are positives.
major comments (5)
- [Section 3 (HKETP Generation) and Section 4 (Evaluation)] The method is not zero-shot as claimed. The paper states 'a human verifier provides feedback and updates, based on one patient's data, to the LLM' and 'It just needs one patient's data during HKETP generation process to tune knowledge model.' This is a calibration/few-shot step, not zero-shot. Moreover, the manuscript does not identify which patient is used or whether that patient belongs to Center A or Center B, so test-set leakage cannot be ruled out. Please specify the tuning procedure, the patient used, and its relationship to the evaluation sets.
- [Abstract and Table 1] The abstract claims 'outperforming state-of-the-art techniques' but Table 1 (Center B rows) shows CuKPL achieves 70.0% accuracy and 82.3% F1 versus DeepXSOZ's 90.3% accuracy and 94.9% F1. This contradicts the unqualified claim. Even the paper's own text says 'The integration of knowledge and DL proved most effective across evaluations.' The claim must be qualified to the development-center (Center A) setting or removed.
- [Table 1 and Section 4 (Evaluation)] No error bars, confidence intervals, or significance tests are reported for any performance metric. With only 316 SOZ images in Center A and 202 in Center B, the reported differences (e.g., 88.4% vs 84.6% accuracy) may be within sampling noise. Please report per-patient variability, standard deviations, or paired statistical tests.
- [Table 1 and Section 4 (Evaluation)] The baselines for DL-2D CNN, ViT, EPIK, and DeepXSOZ are not described as re-implemented or re-run on the same data splits. The reader cannot tell whether the comparison is fair (same preprocessing, same independent components, same LOOCV folds, same Center B split). Please state which numbers are reproduced and which are taken from prior publications, and provide the experimental setup for all baselines.
- [Section 3 and Table 1] No ablations are reported. The contributions of the human verifier, LLM refinement, knowledge model thresholds, and instruction format are never separately evaluated. As a result, the reported CuKPL performance cannot be attributed to the proposed prompt-generation mechanism rather than to the threshold rules inherited from prior work (e.g., Reference [15]).
minor comments (5)
- [Table 1 and throughout] The method name is written inconsistently: 'CuPKL' appears in Table 1 while 'CuKPL' is used elsewhere; please standardize.
- [Section 4 (Results Analysis)] The sentence 'CuKPL, on the other hand, which achieved a 3.8% accuracy and 2.2% F1 score improvement on Center A data over SOTA DeepXSOZ' is a sentence fragment and should be revised.
- [Section 3.1] The typo 'Assymetry' should be 'Asymmetry'.
- [Section 4 (Evaluation)] The clinical workload-reduction claim based on 'MM SOZs of 28' is not tied to the precision/recall trade-off; clarify how this number is computed and what threshold it corresponds to.
- [Throughout] The term 'single-domain generalization' is unusual; consider using 'cross-center evaluation' or 'cross-center generalization' for clarity.
Circularity Check
No significant circularity: CuKPL's classification is a rule-application over externally published knowledge; the one-patient threshold tuning weakens the zero-shot claim but does not reduce the prediction to its inputs by construction.
full rationale
The paper's pipeline generates prompts from domain knowledge cited to prior peer-reviewed work by the same authors ([15], [13]), but that work is externally falsifiable and not a loop. The LLM is asked to reason over explicitly stated rules and quantitative image features (e.g., '1 big red activation(s) detected! ... asymmetrical shapes'), and the rules do not deterministically specify the answer, so the classification is not definitionally equal to the input prompt. The human-verifier threshold update 'based on one patient's data' is a form of tuning that undermines the 'zero-shot' description and is admitted to make the method 'more effective for domain adaptation than for domain generalization,' but it is a limitation and a potential leakage risk, not a circular derivation. The abstract's claim to 'outperforming state-of-the-art techniques' is contradicted by Table 1 on Center B (70.0% acc / 82.3% F1 vs DeepXSOZ 90.3% / 94.9%), but overclaiming against the paper's own numbers is a correctness/support issue, not a circularity issue. No equation or fitted parameter is shown to be equivalent to the target result by construction.
Assumptions & free parameters
free parameters (4)
- Activation count threshold for 'big red activation' =
1 (example: '1 big red activation')
- Gray matter overlap threshold =
majority (>50%), example shows 92.5%
- Asymmetry threshold =
not specified
- White matter overlap threshold =
example 0.0%
assumptions (4)
- domain assumption Independent component analysis of rs-fMRI reliably separates SOZ, RSN, and noise components
- domain assumption The expert knowledge rules (e.g., single activation, gray matter overlap, hemisphere asymmetry) are correct and sufficient to identify SOZ
- domain assumption GPT-4o can faithfully apply the provided rules to text-encoded image features without access to the image
- ad hoc to paper The one patient used for human feedback tuning is not part of the evaluation set
Cite this review
Pith. "Pith review of Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM." pith.science (2026). https://pith.science/paper/BOKSKK3J
@misc{pith2026250116481,
author = {Pith},
title = {Pith review of: Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM},
year = {2026},
howpublished = {\url{https://pith.science/paper/BOKSKK3J}},
note = {Machine review of arXiv:2501.16481}
}
read the original abstract
Rare events, due to their infrequent occurrences, do not have much data, and hence deep learning techniques fail in estimating the distribution for such data. Open-vocabulary models represent an innovative approach to image classification. Unlike traditional models, these models classify images into any set of categories specified with natural language prompts during inference. These prompts usually comprise manually crafted templates (e.g., 'a photo of a {}') that are filled in with the names of each category. This paper introduces a simple yet effective method for generating highly accurate and contextually descriptive prompts containing discriminative characteristics. Rare event detection, especially in medicine, is more challenging due to low inter-class and high intra-class variability. To address these, we propose a novel approach that uses domain-specific expert knowledge on rare events to generate customized and contextually relevant prompts, which are then used by large language models for image classification. Our zero-shot, privacy-preserving method enhances rare event classification without additional training, outperforming state-of-the-art techniques.
Reference graph
Works this paper leans on
-
[15]
Inductive or deductive? rethinking the fundamental reasoning abilities of llms,
Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li, Bing Yin, and Yizhou Sun, “Inductive or deductive? rethinking the fundamental reasoning abilities of llms,” 2024
work page 2024
-
[1]
Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM
INTRODUCTION “Rare events are extremely infrequent events whose charac- teristics make them or their consequences highly valuable. Such events appear with extreme scarcity and are hard to predict, although they are expected eventually” [1]. Despite the rarity, these instances are highly significant as they con- tain crucial information. Detecting such rar...
work page Pith review arXiv 2025
-
[2]
RELATED WORK Standard zero-shot: Standard zero-shot setup uses prede- fined templates to infer a category without any training [6, 10, 11, 12]. However, these models are trained on large, publicly available datasets, which often lack representation of medical data. Customized prompts using LLM such as CuPL [5], integrates open-vocabulary models with LLMs ...
-
[3]
METHODS Fig. 2 shows our overall methodology, which extends the SOTA customized prompt driven image recognition method Fig. 2. a) HKETP Generation: Raw knowledge refined by LLM is encoded using image processing techniques in knowledge model, producing HKETP. Refined knowledge (Context), HKETP (Input) along with Instruction are used by LLM for classi- fica...
-
[4]
Detection of 1 big red activation
-
[5]
Big red activation has asymmetry in hemisphere
- [6]
-
[7]
Lots of small/medium activations detected
Show all 33 references
-
[8]
Symmetry in activations in both hemispheres
-
[9]
Based on rules, answer in YES/NO if the prompt below belongs to SOZ
Detection of 1 big red activation has majority overlap in gray matter. Based on rules, answer in YES/NO if the prompt below belongs to SOZ. If uncertain, respond NO. 1 big red activation(s) detected! 4 small red activation(s) detected! Big red activation has Percentage overlap...
-
[10]
EXPERIMENTAL SETUP AND EV ALUATION Data collected from two centers, A and B, in compliance with IRB protocols. Center A, PCH, Phoenix, has 52 pediatric pa- tients (23 Male, 29 Female, ages 3 months to 18 years) with 5,616 images (2,873 Noise, 2,427 RSN, 316 SOZ), acquired with...
-
[11]
CONCLUSION Traditional DL methods struggle with limited data, while purely knowledge-based approaches fall short due to vague or incomplete domain knowledge. Our framework, CuKPL, ad- dresses these challenges by generating customized, knowledge- rich prompts that, in combinati...
-
[12]
Evalua- tion of rare event detection,
Marina Sokolova, Khaled El Emam, Sadrul Chowdhury, Emilio Neri, Sean Rose, and Elizabeth Jonker, “Evalua- tion of rare event detection,” in Advances in Artificial Intelligence: 23rd Canadian Conference on Artificial Intelligence, Canadian AI 2010, Ottawa, Canada, May 31–June 2...
2010
-
[13]
A systematic review of rare events detection across modalities using machine learning and deep learning,
Yahaya Idris Abubakar, Alice Othmani, Patrick Siarry, and Aznul Qalid Md Sabri, “A systematic review of rare events detection across modalities using machine learning and deep learning,” IEEE Access, 2024
2024
-
[14]
Chain-of-thought prompting elicits reasoning in large language models,
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information pro- cessing systems, vol. 35, pp. 24824–24837, 2022
2022
-
[16]
What does a platypus look like? generating customized prompts for zero-shot image classification,
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi, “What does a platypus look like? generating customized prompts for zero-shot image classification,” in Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15691–15701
2023
-
[17]
Learning transferable vi- sual models from natural language supervision,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning transferable vi- sual models from natural language supervision,” 2021
2021
-
[18]
Instruction tuning for large language models: A survey,
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tian- wei Zhang, Fei Wu, et al., “Instruction tuning for large language models: A survey,” arXiv preprint arXiv:2308.10792, 2023
2023
-
[19]
Cps-llm: Large language model based safe usage plan generator for human-in-the- loop human-in-the-plant cyber-physical system,
Ayan Banerjee, Aranyak Maity, Payal Kamboj, and Sandeep KS Gupta, “Cps-llm: Large language model based safe usage plan generator for human-in-the- loop human-in-the-plant cyber-physical system,” arXiv preprint arXiv:2405.11458, 2024
2024 arXiv
-
[20]
Balancing access to health data and privacy: a review of the issues and ap- proaches for the future,
Julia Lane and Claudia Schur, “Balancing access to health data and privacy: a review of the issues and ap- proaches for the future,” Health services research, vol. 45, no. 5p2, pp. 1456–1467, 2010
2010
-
[21]
Combined scaling for zero-shot transfer learning,
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V . Le, “Combined scaling for zero-shot transfer learning,” 2023
2023
-
[22]
Scaling up visual and vision- language representation learning with noisy text super- vision,
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig, “Scaling up visual and vision- language representation learning with noisy text super- vision,” 2021
2021
-
[23]
Coca: Con- trastive captioners are image-text foundation models,
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu, “Coca: Con- trastive captioners are image-text foundation models,” 2022
2022
-
[24]
Automated seizure onset zone loca- tor from resting-state functional mri in drug-resistant epilepsy,
Ayan Banerjee, Payal Kamboj, Sarah N Wyckoff, Bethany L Sussman, Sandeep KS Gupta, and Varina L Boerwinkle, “Automated seizure onset zone loca- tor from resting-state functional mri in drug-resistant epilepsy,” Frontiers in Neuroimaging , vol. 1, pp. 1007668, 2023
2023
-
[25]
Edgcon: auto-assigner of iconicity ratings grounded by lexical properties to aid in generation of technical ges- tures,
Sameena Hossain, Payal Kamboj, Aranyak Maity, Tamiko Azuma, Ayan Banerjee, and Sandeep Gupta, “Edgcon: auto-assigner of iconicity ratings grounded by lexical properties to aid in generation of technical ges- tures,” in Proceedings of the 38th ACM/SIGAPP Sym- posium on Applied ...
2023
-
[26]
Expert knowledge driven human-ai collabora- tion for medical imaging: A study on epileptic seizure onset zone identification,
Payal Kamboj, Ayan Banerjee, and Sandeep K. S. Gupta, “Expert knowledge driven human-ai collabora- tion for medical imaging: A study on epileptic seizure onset zone identification,” IEEE Transactions on Artifi- cial Intelligence, vol. 5, no. 10, pp. 5352–5368, 2024
2024
-
[27]
Merging deep learning with ex- pert knowledge for seizure onset zone localization from rs-fmri in pediatric pharmaco resistant epilepsy,
Payal Kamboj, Ayan Banerjee, Sandeep KS Gupta, and Varina L Boerwinkle, “Merging deep learning with ex- pert knowledge for seizure onset zone localization from rs-fmri in pediatric pharmaco resistant epilepsy,” arXiv preprint arXiv:2306.05572, 2023
2023 arXiv
-
[28]
Framework for developing and evaluating ethical col- laboration between expert and machine,
Ayan Banerjee, Payal Kamboj, and Sandeep Gupta, “Framework for developing and evaluating ethical col- laboration between expert and machine,” arXiv preprint arXiv:2411.10983, 2024
2024 arXiv
-
[29]
Storm: Strategic orchestration of modalities for rare event classification,
Payal Kamboj, Ayan Banerjee, and Sandeep KS Gupta, “Storm: Strategic orchestration of modalities for rare event classification,” arXiv preprint arXiv:2412.02805, 2024
2024 arXiv
-
[30]
Systems and methods for locating seizure onset zones from rs-fmri in pediatric pharmaco-resistant epilepsy us- ing deep learning,
Sandeep Gupta, Ayan Banerjee, and Payal Kamboj, “Systems and methods for locating seizure onset zones from rs-fmri in pediatric pharmaco-resistant epilepsy us- ing deep learning,” Dec. 19 2024, US Patent App. 18/741,240
2024
-
[31]
The expert’s knowledge com- bined with ai outperforms ai alone in seizure onset zone localization using resting state fmri,
Payal Kamboj, Ayan Banerjee, Varina L Boerwinkle, and Sandeep KS Gupta, “The expert’s knowledge com- bined with ai outperforms ai alone in seizure onset zone localization using resting state fmri,” Frontiers in Neu- rology, vol. 14, pp. 1324461, 2024
2024
-
[32]
Varina L Boerwinkle, Deepankar Mohanty, Stephen T Foldes, Danielle Guffey, Charles G Minard, Aditya Vedantam, Jeffrey S Raskin, Sandi Lam, Margaret Bond, Lucia Mirea, et al., “Correlating resting-state functional magnetic resonance imaging connectivity by independent component...
2017
-
[33]
A prospective fMRI-based technique for localising the epileptogenic zone in presurgical eval- uation of epilepsy,
Borb ´ala Hunyadi, Simon Tousseyn, Patrick Dupont, Sabine Van Huffel, Maarten De V os, and Wim Van Paesschen, “A prospective fMRI-based technique for localising the epileptogenic zone in presurgical eval- uation of epilepsy,” Neuroimage, vol. 113, pp. 329–339, 2015
2015
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.