Pith. sign in

REVIEW 5 major objections 5 minor 33 references

Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Zero-shot medical image classification of rare events can be done with text-only LLM prompts built from expert knowledge, the paper argues.

desk verdict The abstract overclaims: the method is a modest but real extension of CuPL with domain knowledge, and the paper's own cross-center numbers contradict the SOTA claim, yet the body is honest and the work deserves a serious referee. read the letter →

arxiv 2501.16481 v1 pith:BOKSKK3J submitted 2025-01-27 cs.CV

classification cs.CV
keywords zero-shotclassificationrareeventdetectionmedicalimagelargelanguagemodelspromptengineeringseizureonsetzonedomainknowledgefMRI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that rare medical image classification does not require large labeled datasets or a vision encoder: an LLM, given a textual prompt that encodes expert domain knowledge about a rare event, can classify images it has never seen. The authors test this on seizure onset zone (SOZ) detection from resting-state fMRI, where SOZ components make up fewer than 10 percent of independent components. Their method, CuKPL, turns literature-derived knowledge into rule-based textual descriptions of each image and asks an LLM to decide SOZ versus not. On their center-A dataset it reports 88.4 percent accuracy and a 93.8 percent F1 score, above the supervised knowledge-plus-deep-learning baseline, and it keeps an 82.3 percent F1 score when transferred to an unseen center without retraining. If correct, this offers a training-free, privacy-preserving route to rare-event medical classification.

What carries the argument

The machinery is the HKETP generation pipeline. Raw expert knowledge from the literature is refined by an LLM into non-overlapping, discriminative knowledge components; each component is encoded as a software function that inspects an independent-component image and emits a natural-language statement with a satisfaction value, such as “Big red activation has asymmetry in hemispheres”. These statements form the Input prompt, which is combined with the refined knowledge as Context and a fixed instruction, and a generative LLM produces a YES/NO classification. A human verifier iteratively adjusts the thresholds and knowledge statements based on one patient’s data before the system is used on other patients.

What would settle it

Recompute the Center A results with the tuning patient removed from all reported metrics, or run CuKPL with thresholds fixed from the literature without any patient tuning; if the F1 score falls to the level of the untuned zero-shot baselines, the claimed advantage is not zero-shot.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in zero-shot medical classification is not the vision model but the prompt: generic templates like “a photo of a {}” fail because they omit the discriminative rules experts use. CuKPL’s contribution is to operationalize those rules as human knowledge-embodied textual prompts (HKETP) and feed them to an instruction-tuned LLM, which classifies by reasoning over image-derived textual facts rather than pixels. The authors show this text-only pipeline surpasses a knowledge-plus-deep-learning system (DeepXSOZ) on the same data, while standard zero-shot methods such as CLIP and CuPL classify virtually everything as noise or resting-state network. They also claim the method is privacy-preserving because the LLM never receives the image itself, only the textual descriptions produced by the knowledge model.

Load-bearing premise

The method’s claimed zero-shot advantage depends on tuning the knowledge-model thresholds on a single patient’s data, and the paper does not show that this tuning patient is excluded from the evaluation set, so the reported gains could partly come from test-set leakage.

Editorial extensions

If this is right

  • If CuKPL’s results hold, rare-event medical classification can be built without collecting large training sets, which matters for conditions with very few documented cases.
  • The method can adapt to a new hospital with only one patient’s data used to tune thresholds, turning a domain-generalization problem into a cheaper domain-adaptation problem.
  • Because the LLM sees only text, the same prompt-generation framework could be reused across imaging modalities such as MRI, CT, and fundus photography, provided the knowledge components can be computed from the image.
  • The reported reduction in machine-marked SOZs (28 versus 43 for the knowledge-based EPIK system) suggests the output could prioritize which independent components a neurosurgeon reviews.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The one-patient tuning step blurs the zero-shot claim: the paper never identifies the tuning patient, and the tuned thresholds effectively encode the decision rule, so a strict test would set thresholds from literature alone and report that as a true zero-shot variant.
  • The pipeline should transfer to other rare events with rule-based radiological signs, such as the proliferative diabetic retinopathy grading named in the conclusion; a multi-disease benchmark would reveal whether the knowledge-encoding step generalizes or is specific to seizure onset zones.
  • The privacy claim is partial: the LLM never sees raw images, but the textual HKETP descriptions derived from a specific patient’s image are still patient-derived data and may carry identifiable information, a residual risk the paper does not discuss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CuKPL, a method that uses an LLM to convert domain knowledge from literature into structured prompts (HKETP) for zero-shot medical image classification, with a knowledge model encoding image features into text. It is evaluated on seizure onset zone (SOZ) detection from fMRI independent components, comparing with deep learning, knowledge-based, hybrid, and standard zero-shot baselines on Center A and cross-center Center B. The authors report that CuKPL improves accuracy/F1 over DeepXSOZ on Center A but underperforms DeepXSOZ on Center B, and they position the method as zero-shot and privacy-preserving.

Significance. The underlying idea—packaging expert knowledge into textual prompts so that an LLM can classify rare medical images without exposing the image—is attractive and, if rigorously validated, would be a useful privacy-preserving alternative. The paper makes a falsifiable claim ('outperforming state-of-the-art'), and the reported cross-center numbers directly test it. However, in its current form the evidence fails to support the central claim: the method is not zero-shot because thresholds are tuned on one patient's data, and it does not outperform the SOTA on the truly challenging cross-center setting. The code release and detailed dataset description are positives.

major comments (5)
  1. [Section 3 (HKETP Generation) and Section 4 (Evaluation)] The method is not zero-shot as claimed. The paper states 'a human verifier provides feedback and updates, based on one patient's data, to the LLM' and 'It just needs one patient's data during HKETP generation process to tune knowledge model.' This is a calibration/few-shot step, not zero-shot. Moreover, the manuscript does not identify which patient is used or whether that patient belongs to Center A or Center B, so test-set leakage cannot be ruled out. Please specify the tuning procedure, the patient used, and its relationship to the evaluation sets.
  2. [Abstract and Table 1] The abstract claims 'outperforming state-of-the-art techniques' but Table 1 (Center B rows) shows CuKPL achieves 70.0% accuracy and 82.3% F1 versus DeepXSOZ's 90.3% accuracy and 94.9% F1. This contradicts the unqualified claim. Even the paper's own text says 'The integration of knowledge and DL proved most effective across evaluations.' The claim must be qualified to the development-center (Center A) setting or removed.
  3. [Table 1 and Section 4 (Evaluation)] No error bars, confidence intervals, or significance tests are reported for any performance metric. With only 316 SOZ images in Center A and 202 in Center B, the reported differences (e.g., 88.4% vs 84.6% accuracy) may be within sampling noise. Please report per-patient variability, standard deviations, or paired statistical tests.
  4. [Table 1 and Section 4 (Evaluation)] The baselines for DL-2D CNN, ViT, EPIK, and DeepXSOZ are not described as re-implemented or re-run on the same data splits. The reader cannot tell whether the comparison is fair (same preprocessing, same independent components, same LOOCV folds, same Center B split). Please state which numbers are reproduced and which are taken from prior publications, and provide the experimental setup for all baselines.
  5. [Section 3 and Table 1] No ablations are reported. The contributions of the human verifier, LLM refinement, knowledge model thresholds, and instruction format are never separately evaluated. As a result, the reported CuKPL performance cannot be attributed to the proposed prompt-generation mechanism rather than to the threshold rules inherited from prior work (e.g., Reference [15]).
minor comments (5)
  1. [Table 1 and throughout] The method name is written inconsistently: 'CuPKL' appears in Table 1 while 'CuKPL' is used elsewhere; please standardize.
  2. [Section 4 (Results Analysis)] The sentence 'CuKPL, on the other hand, which achieved a 3.8% accuracy and 2.2% F1 score improvement on Center A data over SOTA DeepXSOZ' is a sentence fragment and should be revised.
  3. [Section 3.1] The typo 'Assymetry' should be 'Asymmetry'.
  4. [Section 4 (Evaluation)] The clinical workload-reduction claim based on 'MM SOZs of 28' is not tied to the precision/recall trade-off; clarify how this number is computed and what threshold it corresponds to.
  5. [Throughout] The term 'single-domain generalization' is unusual; consider using 'cross-center evaluation' or 'cross-center generalization' for clarity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CuKPL's classification is a rule-application over externally published knowledge; the one-patient threshold tuning weakens the zero-shot claim but does not reduce the prediction to its inputs by construction.

full rationale

The paper's pipeline generates prompts from domain knowledge cited to prior peer-reviewed work by the same authors ([15], [13]), but that work is externally falsifiable and not a loop. The LLM is asked to reason over explicitly stated rules and quantitative image features (e.g., '1 big red activation(s) detected! ... asymmetrical shapes'), and the rules do not deterministically specify the answer, so the classification is not definitionally equal to the input prompt. The human-verifier threshold update 'based on one patient's data' is a form of tuning that undermines the 'zero-shot' description and is admitted to make the method 'more effective for domain adaptation than for domain generalization,' but it is a limitation and a potential leakage risk, not a circular derivation. The abstract's claim to 'outperforming state-of-the-art techniques' is contradicted by Table 1 on Center B (70.0% acc / 82.3% F1 vs DeepXSOZ 90.3% / 94.9%), but overclaiming against the paper's own numbers is a correctness/support issue, not a circularity issue. No equation or fitted parameter is shown to be equivalent to the target result by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The core classification is driven by a small set of hand-crafted rules whose numerical thresholds are adjusted via human feedback on one patient's data. These thresholds are free parameters. The axioms are the domain assumptions about ICA and the validity of the rules, plus the untested assumption that the LLM executes the rules faithfully. No new entities are introduced.

free parameters (4)
  • Activation count threshold for 'big red activation' = 1 (example: '1 big red activation')
    Determines whether an IC is considered SOZ per the knowledge rules; tuned via human feedback on one patient's data (Section 3).
  • Gray matter overlap threshold = majority (>50%), example shows 92.5%
    Used in rule 'big red activation has majority overlap in gray matter'; threshold updated during human verifier feedback.
  • Asymmetry threshold = not specified
    Rule 'big red activation has asymmetry in hemisphere' is qualitative; no explicit threshold given, but it drives classification.
  • White matter overlap threshold = example 0.0%
    Used to rule out SOZ; no explicit threshold given.
assumptions (4)
  • domain assumption Independent component analysis of rs-fMRI reliably separates SOZ, RSN, and noise components
    Section 3.1 states 'fMRI yields 150 ICs per patient with (<10%) associated with SOZ', relying on prior work [15,13,20].
  • domain assumption The expert knowledge rules (e.g., single activation, gray matter overlap, hemisphere asymmetry) are correct and sufficient to identify SOZ
    Rules are taken from the authors' prior publications [15,13] and are not independently validated in this paper.
  • domain assumption GPT-4o can faithfully apply the provided rules to text-encoded image features without access to the image
    The method requires the LLM to reason about quantitative image features from text prompts; no evidence is given that this is reliable for the target distribution.
  • ad hoc to paper The one patient used for human feedback tuning is not part of the evaluation set
    Section 3 mentions tuning on 'one patient's data' but does not specify which patient, creating a potential leakage risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM." pith.science (2026). https://pith.science/paper/BOKSKK3J

@misc{pith2026250116481,
  author       = {Pith},
  title        = {Pith review of: Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BOKSKK3J}},
  note         = {Machine review of arXiv:2501.16481}
}
read the original abstract

Rare events, due to their infrequent occurrences, do not have much data, and hence deep learning techniques fail in estimating the distribution for such data. Open-vocabulary models represent an innovative approach to image classification. Unlike traditional models, these models classify images into any set of categories specified with natural language prompts during inference. These prompts usually comprise manually crafted templates (e.g., 'a photo of a {}') that are filled in with the names of each category. This paper introduces a simple yet effective method for generating highly accurate and contextually descriptive prompts containing discriminative characteristics. Rare event detection, especially in medicine, is more challenging due to low inter-class and high intra-class variability. To address these, we propose a novel approach that uses domain-specific expert knowledge on rare events to generate customized and contextually relevant prompts, which are then used by large language models for image classification. Our zero-shot, privacy-preserving method enhances rare event classification without additional training, outperforming state-of-the-art techniques.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 29 canonical work pages

  1. [15]

    Inductive or deductive? rethinking the fundamental reasoning abilities of llms,

    Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li, Bing Yin, and Yizhou Sun, “Inductive or deductive? rethinking the fundamental reasoning abilities of llms,” 2024

  2. [1]

    Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM

    INTRODUCTION “Rare events are extremely infrequent events whose charac- teristics make them or their consequences highly valuable. Such events appear with extreme scarcity and are hard to predict, although they are expected eventually” [1]. Despite the rarity, these instances are highly significant as they con- tain crucial information. Detecting such rar...

  3. [2]

    However, these models are trained on large, publicly available datasets, which often lack representation of medical data

    RELATED WORK Standard zero-shot: Standard zero-shot setup uses prede- fined templates to infer a category without any training [6, 10, 11, 12]. However, these models are trained on large, publicly available datasets, which often lack representation of medical data. Customized prompts using LLM such as CuPL [5], integrates open-vocabulary models with LLMs ...

  4. [3]

    2 shows our overall methodology, which extends the SOTA customized prompt driven image recognition method Fig

    METHODS Fig. 2 shows our overall methodology, which extends the SOTA customized prompt driven image recognition method Fig. 2. a) HKETP Generation: Raw knowledge refined by LLM is encoded using image processing techniques in knowledge model, producing HKETP. Refined knowledge (Context), HKETP (Input) along with Instruction are used by LLM for classi- fica...

  5. [4]

    Detection of 1 big red activation

  6. [5]

    Big red activation has asymmetry in hemisphere

  7. [6]

    Rules of being not SOZ:

    Big red activation has NO similar shape. Rules of being not SOZ:

  8. [7]

    Lots of small/medium activations detected

Show all 33 references
  1. [8]

    Symmetry in activations in both hemispheres

  2. [9]

    Based on rules, answer in YES/NO if the prompt below belongs to SOZ

    Detection of 1 big red activation has majority overlap in gray matter. Based on rules, answer in YES/NO if the prompt below belongs to SOZ. If uncertain, respond NO. 1 big red activation(s) detected! 4 small red activation(s) detected! Big red activation has Percentage overlap...

  3. [10]

    EXPERIMENTAL SETUP AND EV ALUATION Data collected from two centers, A and B, in compliance with IRB protocols. Center A, PCH, Phoenix, has 52 pediatric pa- tients (23 Male, 29 Female, ages 3 months to 18 years) with 5,616 images (2,873 Noise, 2,427 RSN, 316 SOZ), acquired with...

  4. [11]

    CONCLUSION Traditional DL methods struggle with limited data, while purely knowledge-based approaches fall short due to vague or incomplete domain knowledge. Our framework, CuKPL, ad- dresses these challenges by generating customized, knowledge- rich prompts that, in combinati...

  5. [12]

    Evalua- tion of rare event detection,

    Marina Sokolova, Khaled El Emam, Sadrul Chowdhury, Emilio Neri, Sean Rose, and Elizabeth Jonker, “Evalua- tion of rare event detection,” in Advances in Artificial Intelligence: 23rd Canadian Conference on Artificial Intelligence, Canadian AI 2010, Ottawa, Canada, May 31–June 2...

  6. [13]

    A systematic review of rare events detection across modalities using machine learning and deep learning,

    Yahaya Idris Abubakar, Alice Othmani, Patrick Siarry, and Aznul Qalid Md Sabri, “A systematic review of rare events detection across modalities using machine learning and deep learning,” IEEE Access, 2024

  7. [14]

    Chain-of-thought prompting elicits reasoning in large language models,

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information pro- cessing systems, vol. 35, pp. 24824–24837, 2022

  8. [16]

    What does a platypus look like? generating customized prompts for zero-shot image classification,

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi, “What does a platypus look like? generating customized prompts for zero-shot image classification,” in Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15691–15701

  9. [17]

    Learning transferable vi- sual models from natural language supervision,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning transferable vi- sual models from natural language supervision,” 2021

  10. [18]

    Instruction tuning for large language models: A survey,

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tian- wei Zhang, Fei Wu, et al., “Instruction tuning for large language models: A survey,” arXiv preprint arXiv:2308.10792, 2023

  11. [19]

    Cps-llm: Large language model based safe usage plan generator for human-in-the- loop human-in-the-plant cyber-physical system,

    Ayan Banerjee, Aranyak Maity, Payal Kamboj, and Sandeep KS Gupta, “Cps-llm: Large language model based safe usage plan generator for human-in-the- loop human-in-the-plant cyber-physical system,” arXiv preprint arXiv:2405.11458, 2024

  12. [20]

    Balancing access to health data and privacy: a review of the issues and ap- proaches for the future,

    Julia Lane and Claudia Schur, “Balancing access to health data and privacy: a review of the issues and ap- proaches for the future,” Health services research, vol. 45, no. 5p2, pp. 1456–1467, 2010

  13. [21]

    Combined scaling for zero-shot transfer learning,

    Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V . Le, “Combined scaling for zero-shot transfer learning,” 2023

  14. [22]

    Scaling up visual and vision- language representation learning with noisy text super- vision,

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig, “Scaling up visual and vision- language representation learning with noisy text super- vision,” 2021

  15. [23]

    Coca: Con- trastive captioners are image-text foundation models,

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu, “Coca: Con- trastive captioners are image-text foundation models,” 2022

  16. [24]

    Automated seizure onset zone loca- tor from resting-state functional mri in drug-resistant epilepsy,

    Ayan Banerjee, Payal Kamboj, Sarah N Wyckoff, Bethany L Sussman, Sandeep KS Gupta, and Varina L Boerwinkle, “Automated seizure onset zone loca- tor from resting-state functional mri in drug-resistant epilepsy,” Frontiers in Neuroimaging , vol. 1, pp. 1007668, 2023

  17. [25]

    Edgcon: auto-assigner of iconicity ratings grounded by lexical properties to aid in generation of technical ges- tures,

    Sameena Hossain, Payal Kamboj, Aranyak Maity, Tamiko Azuma, Ayan Banerjee, and Sandeep Gupta, “Edgcon: auto-assigner of iconicity ratings grounded by lexical properties to aid in generation of technical ges- tures,” in Proceedings of the 38th ACM/SIGAPP Sym- posium on Applied ...

  18. [26]

    Expert knowledge driven human-ai collabora- tion for medical imaging: A study on epileptic seizure onset zone identification,

    Payal Kamboj, Ayan Banerjee, and Sandeep K. S. Gupta, “Expert knowledge driven human-ai collabora- tion for medical imaging: A study on epileptic seizure onset zone identification,” IEEE Transactions on Artifi- cial Intelligence, vol. 5, no. 10, pp. 5352–5368, 2024

  19. [27]

    Merging deep learning with ex- pert knowledge for seizure onset zone localization from rs-fmri in pediatric pharmaco resistant epilepsy,

    Payal Kamboj, Ayan Banerjee, Sandeep KS Gupta, and Varina L Boerwinkle, “Merging deep learning with ex- pert knowledge for seizure onset zone localization from rs-fmri in pediatric pharmaco resistant epilepsy,” arXiv preprint arXiv:2306.05572, 2023

  20. [28]

    Framework for developing and evaluating ethical col- laboration between expert and machine,

    Ayan Banerjee, Payal Kamboj, and Sandeep Gupta, “Framework for developing and evaluating ethical col- laboration between expert and machine,” arXiv preprint arXiv:2411.10983, 2024

  21. [29]

    Storm: Strategic orchestration of modalities for rare event classification,

    Payal Kamboj, Ayan Banerjee, and Sandeep KS Gupta, “Storm: Strategic orchestration of modalities for rare event classification,” arXiv preprint arXiv:2412.02805, 2024

  22. [30]

    Systems and methods for locating seizure onset zones from rs-fmri in pediatric pharmaco-resistant epilepsy us- ing deep learning,

    Sandeep Gupta, Ayan Banerjee, and Payal Kamboj, “Systems and methods for locating seizure onset zones from rs-fmri in pediatric pharmaco-resistant epilepsy us- ing deep learning,” Dec. 19 2024, US Patent App. 18/741,240

  23. [31]

    The expert’s knowledge com- bined with ai outperforms ai alone in seizure onset zone localization using resting state fmri,

    Payal Kamboj, Ayan Banerjee, Varina L Boerwinkle, and Sandeep KS Gupta, “The expert’s knowledge com- bined with ai outperforms ai alone in seizure onset zone localization using resting state fmri,” Frontiers in Neu- rology, vol. 14, pp. 1324461, 2024

  24. [32]

    Varina L Boerwinkle, Deepankar Mohanty, Stephen T Foldes, Danielle Guffey, Charles G Minard, Aditya Vedantam, Jeffrey S Raskin, Sandi Lam, Margaret Bond, Lucia Mirea, et al., “Correlating resting-state functional magnetic resonance imaging connectivity by independent component...

  25. [33]

    A prospective fMRI-based technique for localising the epileptogenic zone in presurgical eval- uation of epilepsy,

    Borb ´ala Hunyadi, Simon Tousseyn, Patrick Dupont, Sabine Van Huffel, Maarten De V os, and Wim Van Paesschen, “A prospective fMRI-based technique for localising the epileptogenic zone in presurgical eval- uation of epilepsy,” Neuroimage, vol. 113, pp. 329–339, 2015

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.