Pith. sign in

REVIEW 3 major objections 8 minor 63 references

BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models

T0 review · 3 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BiomedCoOp claims that anchoring learnable prompt contexts to ensembled LLM class descriptions and to a pruned teacher distribution lets a frozen BiomedCLIP outperform existing prompt-learning, adapter, and linear-probe methods on 11…

desk verdict Solid, well-ablated prompt-learning method; but the main table disagrees with its own supplementary on a key baseline, so the SOTA claim is not yet reliable. read the letter →

arxiv 2411.15232 v2 pith:JQFIKEDV submitted 2024-11-21 cs.CV cs.CL

Taha Koleilat , Hojat Asgariandehkordi , Hassan Rivaz , Yiming Xiao This is my paper · ORCID
classification cs.CVcs.CL
keywords promptlearningvision-languagemodelsbiomedicalimageclassificationfew-shotLLMensemblingknowledgedistillationbase-to-novelgeneralizationBiomedCLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that few-shot biomedical image classification can be improved substantially by learning prompt contexts for a biomedically pretrained vision-language model rather than fine-tuning the model. The proposed framework, BiomedCoOp, claims to achieve this by anchoring the learnable text prompt to averaged GPT-4-generated class descriptions and by distilling knowledge from a selectively pruned set of LLM prompts. On 11 datasets spanning 9 imaging modalities, it reports the best average few-shot accuracy among the compared methods at every shot level, including 57.03% at one label per class and 72.42% at 16 labels per class, and the best base-to-novel harmonic mean of 75.07%. If the claim is right, a handful of labeled scans plus an LLM's textual knowledge can replace extensive prompt engineering and full fine-tuning for diverse clinical imaging tasks.

What carries the argument

The load-bearing machinery is a pair of text-side losses attached to the learnable prompt context. SCCM (Eq. 9) is an MSE pull toward $P_g$, the mean embedding of $N$ LLM-generated class descriptions; it supplies general biomedical semantics. KDSP (Eq. 10) is a KL-divergence term that aligns the student's logits with a teacher formed by $P_s$, the average embedding of LLM prompts selected after scoring each prompt against the few-shot support images by cosine similarity and pruning outliers with a modified z-score based on the median absolute deviation (Eqs. 4-7). The two losses jointly keep the learned context close to LLM knowledge while preventing atypical prompts from pushing it into overspecialized semantic regions.

What would settle it

Replace GPT-4 prompts with shallow text such as 'a photo of [CLASS]' while keeping both losses; if $K=1$ accuracy stays near 57.03%, the specific semantic content of the LLM prompts is not what drives the gain. Alternatively, build $K=1$ support sets from only the most atypical examples per class and check whether accuracy falls to or below the ProGrad baseline of 51.88%, which would show that Eq. 4's outlier pruning relies too heavily on support-set representativeness.

Watch

Extended reading notes

Core claim

BiomedCoOp's central claim is that two text-side supervision losses, applied to the learnable context vectors of CoOp-style prompt learning, are enough to adapt BiomedCLIP to biomedical few-shot classification without touching the pretrained weights. SCCM minimizes the squared distance between the learned context embedding and the average of GPT-4-generated, class-specific prompt embeddings; KDSP minimizes the KL divergence between the student's prediction distribution and a teacher distribution built from LLM prompts that survive a median-absolute-deviation outlier filter. The paper reports that this dual anchoring outperforms CoOp, CoCoOp, KgCoOp, ProGrad, adapter methods, and linear probes on 11 datasets at all shot counts, and that it also improves generalization to unseen classes, as measured by base-to-novel harmonic mean.

Load-bearing premise

The method's teacher is built by scoring LLM prompts against the same few labeled images that later optimize the student, so it assumes those few images are representative enough of each class that outlier pruning selects prompts that also help on unseen test images.

Editorial extensions

If this is right

  • At $K=1$, the method reports 57.03% average accuracy, 5.2 points above the best baseline (ProGrad), so LLM-anchored prompt learning is particularly effective when labeled data is extremely scarce.
  • At $K=16$, it reports 72.42%, still the best among all compared methods, so the gains do not disappear as more labeled data arrives.
  • The base-to-novel harmonic mean of 75.07% (versus 67.43% for the best baseline) implies the learned prompts retain knowledge for unseen classes, supporting use in settings where disease categories evolve.
  • Ablations show SCCM alone helps few-shot accuracy while KDSP alone helps novel-class accuracy, and only their combination reaches the best results; removing either component lowers the harmonic mean.
  • Varying the number of LLM prompts from 10 to 50 raises zero-shot accuracy from 47.55% to 52.27%, so prompt diversity is a cheap source of generalization before any labeled data is used.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Eq. 4 scores prompts against the same support images that train the student, the approach implicitly assumes those few images represent each class; an adversarial support set made of only atypical presentations would test whether KDSP amplifies instead of corrects the resulting bias.
  • All new losses act on text embeddings with a frozen backbone, so the SCCM and KDSP recipe should transfer to other contrastive vision-language models; the paper's own backbone ablation indicates the gains are not unique to BiomedCLIP, though they are largest there.
  • The LLM query names the imaging modality, and the nearest-token analysis shows learned tokens like 'mri' and 'endoscopy', suggesting part of the advantage is modality grounding; removing the modality word from the query would separate that effect from pure class semantics.
  • Per-dataset tuning of $\lambda_1$, $\lambda_2$, and $\zeta_s$ leaves open whether a single fixed configuration nearly matches the tuned results; a simple deployment test is to run with defaults derived from the paper's value ranges.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This paper introduces BiomedCoOp, a prompt-learning framework that adapts the BiomedCLIP vision-language model to few-shot biomedical image classification. The method combines two components: Semantic Consistency by Contextual Mapping (SCCM, Eq. 9), which aligns learnable context prompts with the mean embedding of N GPT-4-generated class-specific prompts, and Knowledge Distillation with Selective Prompting (KDSP, Eqs. 4-7 and 10), which prunes LLM prompts via a modified-z-score outlier filter based on similarity to the few-shot training images and distills the student toward the pruned teacher via KL divergence. The total loss is L = L_CE + λ1 L_SCCM + λ2 L_KDSP (Eq. 11). Reported results on 11 biomedical datasets spanning 9 modalities claim the highest few-shot accuracy (57.03% at K=1 and 72.42% at K=16) and the best base-to-novel harmonic mean (75.07%) among the compared prompt-learning, adapter, and linear-probe baselines. Ablations examine the contribution of each component, the number of LLM prompts, the choice of VLM backbone, the LLM, the context length, and the selection threshold.

Significance. Biomedical few-shot classification is practically important and under-served, and the paper's benchmark of 11 datasets across 9 modalities is a useful contribution in itself. If the reported numbers hold, the gains are large (roughly 5-10 points over the strongest baselines at low shots) and the design is well motivated: LLM-generated prompts inject domain knowledge, and the outlier-pruned teacher is a sensible way to avoid over-specialized prompts. The paper ships public code, averages results over three support sets with standard deviations, and includes ablation and sensitivity studies (LLM choice, prompt count, backbone, context length, selection threshold). The main reservation is the reliability of the comparative evidence: the main few-shot table is inconsistent with the paper's own per-dataset supplementary table for the KgCoOp baseline at every shot level, and this inconsistency must be resolved before the claimed state-of-the-art margins can be accepted.

major comments (3)
  1. [Table 1 vs. Table S8] The headline claim that BiomedCoOp achieves the highest few-shot accuracy rests on Table 1, but Table 1 is inconsistent with the paper's own per-dataset results in Table S8 for the KgCoOp baseline at every shot level. Table 1 reports KgCoOp averages of 50.85, 53.18, 57.82, 62.08, and 62.84 for K=1,2,4,8,16, while the per-dataset KgCoOp entries in Table S8 average to 51.83, 53.47, 58.59, 63.5, and 64.88, matching the supplementary average row (which lists 51.83, 53.47, 58.59, 63.65, 64.88; note that the K=8 average row itself does not exactly equal the mean of the listed per-dataset entries). The manuscript never states which table reflects the actual experimental runs. Correcting KgCoOp reduces the reported K=1 margin from 6.18 to 5.20 points and the K=16 margin from 9.58 to 7.54 points; at K=16 the best-reported baseline, Tip-Adapter-F (70.91), is within 1.5 points of BiomedCoOp, so a comparable error in any other baseline row could change the ranking. The authors should reconcile the two tables, state which numbers are the official results, and audit every baseline row in Table 1 against the appendix before the state-of-the-art claim can be accepted.
  2. [Section 3.3, Eqs. (4) and (10)] The KDSP teacher is selected by scoring the LLM prompts against the same few-shot training images that are later used to optimize the student (Eq. 4 in Section 3.3 versus Eqs. 8-11 in Section 3.4). At K=1 or K=2, the score for each class is computed from one or two images, so outlier pruning can select prompts that happen to match those particular support images, and the KL term (Eq. 10) then distills a support-set-specific teacher rather than general biomedical knowledge. This is a genuine circularity risk for the few-shot gains, not a presentational issue. Please add a control experiment that separates prompt selection from student training (e.g., select the teacher on one half of the support set and train on the other half, or compare against a teacher that keeps all N prompts), and discuss the risk explicitly in the text.
  3. [Section 4.1 and Table S6] The comparative claim relies on a fair tuning protocol, but the manuscript reports per-dataset selection of λ1, λ2, and ζs over wide ranges (e.g., λ2 from 0.25 to 20.0 and ζs from 1.25 to 2.0 in Table S6) while only stating that baselines were 'tuned to their optimal settings' without giving their search budgets or final hyperparameters. Please document the tuning protocol for every method (or state that published default settings were used for baselines), and report the value of the scaling factor β in Eq. (4), which is never specified in Section 4.1.
minor comments (8)
  1. [Section 1, Contribution 3] The claim that this study 'adopts BiomedCLIP for prompt learning for the first time' is contradicted by the supplementary comparison against XCoOp, which the authors describe as 'directly utilizing the BiomedCLIP backbone'; please qualify the novelty statement.
  2. [Eq. (7)] What is called a modified z-score conventionally includes the Iglewicz-Hoaglin constant 0.6745; as written, the ζs values are applied to the unscaled statistic, so please state the exact definition used.
  3. [Section 4.1 and Table 1 caption] The text says results are 'averaging results over three runs,' whereas the Table 1 caption says the standard deviation is 'derived from 3 sampled support sets'; please clarify what the reported ± values represent.
  4. [References] KgCoOp is cited as [43] in the text and [44] in Table 1, and ProGrad as [53] in the text and [52] in Table 1; please unify the numbering.
  5. [Section 4.2] The stated margins over ProGrad (5.2% at K=1 and 4.6% at K=2) do not match the table values (57.03−51.88=5.15 and 59.13−54.71=4.42); please round consistently.
  6. [Table S7] The LC25000 split is written as '12500/5000//7500' with a double slash.
  7. [Supplementary Section 12] The acronym LSDM is used without definition.
  8. [Table 3] The text refers to gray and green highlights, but the table as printed lacks a legend mapping each row to the BiomedCLIP/CLIP backbone and to the full method; please add one.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM-derived prompt targets and distillation losses are constructed from class-name/modality inputs and external benchmarks, not from the paper's own target claims.

full rationale

BiomedCoOp's derivation chain is self-contained. The LLM prompt ensemble P_g (Eq. 3) is generated from the class names and imaging modality in the query Q, independent of the test labels and of the learned context. The two regularization terms, SCCM (Eq. 9) and KDSP (Eq. 10), pull the learnable context toward LLM-derived text embeddings and align student logits with teacher logits computed from the same pre-trained BiomedCLIP vision encoder; neither term defines the evaluation target by construction. The KDSP teacher selection in Eqs. 4-7 uses the few-shot support images to prune outlier LLM prompts, but this is a training-time self-distillation procedure, not a fitted parameter renamed as a prediction: the reported few-shot and base-to-novel accuracies are evaluated on held-out test and novel classes. There is no load-bearing self-citation chain or imported uniqueness theorem; the self-citations [28,29] appear only in the interpretability discussion and are not used to justify the method's central claim. The observed discrepancy between Table 1 and the supplementary per-dataset KgCoOp averages is an empirical reporting inconsistency that affects the strength of the SOTA comparison, but it is not a construction-level circularity and does not make any equation reduce to its own input. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method depends mainly on per-dataset hyperparameters, the quality of LLM-generated text, and the assumption that BiomedCLIP's text and image embeddings are well aligned. No new physical or conceptual entities are introduced; the teacher is a statistical combination of existing text embeddings.

free parameters (5)
  • λ1 and λ2 (SCCM and KDSP loss weights, Eq. 11) = 0.25 to 20.0, per dataset and benchmark (Table S6)
    Selected on the validation set separately for few-shot and base-to-novel benchmarks; the wide range indicates the method is sensitive to these weights.
  • ζs (selection threshold for modified z-score, Eq. 7) = 1.25 to 2.0, typically 1.25-1.5 (Table S6)
    Tuned per dataset; Figure S2 shows novel-class accuracy peaks at ζs=1.25 and declines for larger thresholds.
  • Number of LLM prompts N = 50
    Chosen by ablation (Table 4); accuracy continues to rise with N, so 50 is a saturating design point rather than a true optimum.
  • Context length = 4 tokens
    Chosen by ablation (Table S1); longer contexts reduce the base-to-novel harmonic mean.
  • β (scaling factor in Eq. 4) = not reported
    Used to scale prompt-image similarity logits before the median filter; the value is not stated in the paper.
assumptions (4)
  • domain assumption BiomedCLIP's vision and text encoders produce a shared embedding space where cosine similarity between medical images and medical text is a valid semantic measure.
    Invoked in Eq. 1 for classification and in Eq. 4 for prompt scoring; if cross-modal alignment is weak for a modality, the SCCM/KDSP losses would pull prompts toward irrelevant text.
  • domain assumption LLM descriptions of visual discriminative features are accurate, class-relevant, and sufficiently diverse across 50 samples per class.
    Section 3.2 builds the entire teacher on GPT-4 outputs; robustness is partially supported by Table S4 showing similar results with LLaMA-3 and Gemma-2, but correctness of descriptions is assumed.
  • domain assumption The few-shot support set is representative of each class's visual variation, so prompt scores computed on it (Eq. 4) generalize to the test set.
    KDSP pruning uses only K images per class from training; if these images are atypical, selected prompts may be atypical, hurting generalization.
  • ad hoc to paper Modified z-score outlier filtering with threshold ζs separates beneficial from harmful LLM prompts in the embedding space.
    The heuristic is introduced for this method and is tuned per dataset; no external justification is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models." pith.science (2026). https://pith.science/paper/JQFIKEDV

@misc{pith2026241115232,
  author       = {Pith},
  title        = {Pith review of: BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQFIKEDV}},
  note         = {Machine review of arXiv:2411.15232}
}
read the original abstract

Recent advancements in vision-language models (VLMs), such as CLIP, have demonstrated substantial success in self-supervised representation learning for vision tasks. However, effectively adapting VLMs to downstream applications remains challenging, as their accuracy often depends on time-intensive and expertise-demanding prompt engineering, while full model fine-tuning is costly. This is particularly true for biomedical images, which, unlike natural images, typically suffer from limited annotated datasets, unintuitive image contrasts, and nuanced visual features. Recent prompt learning techniques, such as Context Optimization (CoOp) intend to tackle these issues, but still fall short in generalizability. Meanwhile, explorations in prompt learning for biomedical image analysis are still highly limited. In this work, we propose BiomedCoOp, a novel prompt learning framework that enables efficient adaptation of BiomedCLIP for accurate and highly generalizable few-shot biomedical image classification. Our approach achieves effective prompt context learning by leveraging semantic consistency with average prompt ensembles from Large Language Models (LLMs) and knowledge distillation with a statistics-based prompt selection strategy. We conducted comprehensive validation of our proposed framework on 11 medical datasets across 9 modalities and 10 organs against existing state-of-the-art methods, demonstrating significant improvements in both accuracy and generalizability. The code is publicly available at https://github.com/HealthX-Lab/BiomedCoOp.

Figures

Figures reproduced from arXiv: 2411.15232 by the authors.

Figure 1
Figure 1. Overview of the BiomedCoOp framework, which combines LLM queries, learnable context tokens, and BiomedCLIP to generate [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Barplot to compare classification accuracies (%) of dif [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Effect of various text prompt techniques on visual saliency maps. Columns (b)-(f) represent different prompt methods. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 48 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Dataset of breast ultrasound images

    Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020. 5, 16

    work page 2020

  3. [3]

    Xcoop: Explainable prompt learning for computer-aided di- agnosis via concept-guided context optimization

    Yequan Bie, Luyang Luo, Zhixuan Chen, and Hao Chen. Xcoop: Explainable prompt learning for computer-aided di- agnosis via concept-guided context optimization. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention , pages 773–783. Springer,

    work page

  4. [4]

    Making the most of text semantics to improve biomedical vision–language processing

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. In Euro- pean conference on computer vision , pages 1–21. Springer,

    work page

  5. [5]

    Borkowski, Marilyn M

    Andrew A. Borkowski, Marilyn M. Bui, L. Brannon Thomas, Catherine P. Wilson, Lauren A. DeLand, and Stephen M. Mastorides. Lung and colon cancer histopatho- logical image dataset (lc25000), 2019. 5, 16

    work page 2019

  6. [6]

    Domain-controlled prompt learning

    Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain-controlled prompt learning. InPro- ceedings of the AAAI Conference on Artificial Intelligence , pages 936–944, 2024. 2, 13

    work page 2024

  7. [7]

    Knee osteoarthritis severity grading dataset,

    Pingjun Chen. Knee osteoarthritis severity grading dataset,

    work page

  8. [8]

    gscorecam: What objects is clip looking at? In Proceedings of the Asian Conference on Computer Vision , pages 1959– 1975, 2022

    Peijie Chen, Qi Li, Saad Biaz, Trung Bui, and Anh Nguyen. gscorecam: What objects is clip looking at? In Proceedings of the Asian Conference on Computer Vision , pages 1959– 1975, 2022. 8

    work page 1959

Show all 63 references
  1. [9]

    brain tumor dataset, 2017

    Jun Cheng. brain tumor dataset, 2017. 8

    2017

  2. [10]

    Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the interna- tional skin imaging collaboration (isic)

    Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the interna- tional skin imagin...

    2018 arXiv

  3. [11]

    Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kit- tler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomed...

    2017

  4. [12]

    Bcn20000: Dermoscopic lesions in the wild

    Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Car- rera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288, 2019. 8

    1908 arXiv

  5. [13]

    Cleft: Language-image contrastive learning with efficient large language model and prompt fine-tuning

    Yuexi Du, Brian Chang, and Nicha C Dvornek. Cleft: Language-image contrastive learning with efficient large language model and prompt fine-tuning. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 465–475. Springer, 2024. 2

    2024

  6. [14]

    Does clip benefit visual question answering in the medical domain as much as it does in the general domain?, 2021

    Sedigheh Eslami, Gerard de Melo, and Christoph Meinel. Does clip benefit visual question answering in the medical domain as much as it does in the general domain?, 2021. 2

    2021

  7. [15]

    Aligning medical images with general knowl- edge from large language models

    Xiao Fang, Yi Lin, Dong Zhang, Kwang-Ting Cheng, and Hao Chen. Aligning medical images with general knowl- edge from large language models. In International Confer- ence on Medical Image Computing and Computer-Assisted Intervention, pages 57–67. Springer, 2024. 2

    2024

  8. [16]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2): 581–595, 2024. 1, 3, 6

    2024

  9. [17]

    Potential of gpt-4 for de- tecting errors in radiology reports: Implications for reporting accuracy

    Roman Johannes Gertz, Thomas Dratsch, Alexander Chris- tian Bunck, Simon Lennartz, Andra-Iza Iuga, Martin Gunnar Hellmich, Thorsten Persigehl, Lenhard Pennig, Carsten Her- bert Gietzen, Philipp Fervers, et al. Potential of gpt-4 for de- tecting errors in radiology reports: Imp...

    2024

  10. [18]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. 1

    2019

  11. [19]

    Lp++: A surprisingly strong linear probe for few-shot clip

    Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. Lp++: A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23773–23782, 2024. 1, 3, 6

    2024

  12. [20]

    Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from ct- radiography

    Md Nazmul Islam, Mehedi Hasan, Md Kabir Hossain, Md Golam Rabiul Alam, Md Zia Uddin, and Ahmet Soylu. Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from ct- radiography. Scientific Reports, 12(1):1–14, 2022. 5, 16

    2022

  13. [21]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  14. [22]

    Multi-class texture anal- ysis in colorectal cancer histology

    Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexan- der Marx, and Frank Gerrit Z¨ollner. Multi-class texture anal- ysis in colorectal cancer histology. Scientific reports, 6(1): 1–11, 2016. 5, 16

    2016

  15. [23]

    Kermany, Michael Goldbaum, et al

    Daniel S. Kermany, Michael Goldbaum, et al. Identify- ing medical diagnoses and treatable diseases by image-based deep learning. Cell, 172(5):1122 – 1131.e9, 2018. 5, 16

    2018

  16. [24]

    Maple: 9 Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: 9 Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 2

    2023

  17. [25]

    Maple: Multi-modal prompt learning

    Muhammad Uzair khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1

    2023

  18. [26]

    Self-regulating prompts: Foundational model adaptation without forgetting

    Muhammad Uzair Khattak, Syed Talal Wasim, Muzam- mal Naseer, Salman Khan, Ming-Hsuan Yang, and Fa- had Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15...

    2023

  19. [27]

    Learning to prompt with text only supervision for vision- language models

    Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision- language models. arXiv preprint arXiv:2401.02418 , 2024. 2, 3

    2024 arXiv

  20. [28]

    Medclip-sam: Bridging text and image towards universal medical image segmentation

    Taha Koleilat, Hojat Asgariandehkordi, Hassan Rivaz, and Yiming Xiao. Medclip-sam: Bridging text and image towards universal medical image segmentation. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention , pages 643–653. Springer,

  21. [29]

    Medclip-samv2: Towards universal text-driven medical image segmentation

    Taha Koleilat, Hojat Asgariandehkordi, Hassan Rivaz, and Yiming Xiao. Medclip-samv2: Towards universal text-driven medical image segmentation. arXiv preprint arXiv:2409.19483, 2024. 8

    2024 arXiv

  22. [30]

    Evaluation of reliability, repeatability, ro- bustness, and confidence of gpt-3.5 and gpt-4 on a radiology board–style examination

    Satheesh Krishna, Nishaant Bhambra, Robert Bleakney, and Rajesh Bhayana. Evaluation of reliability, repeatability, ro- bustness, and confidence of gpt-3.5 and gpt-4 on a radiology board–style examination. Radiology, 311(2):e232715, 2024. 3, 4

    2024

  23. [31]

    Automatic no- reference quality assessment for retinal fundus images using vessel segmentation, 2013

    Thomas K ¨ohler, Attila Budai, Martin Kraus, Jan Odstrcilik, Georg Michelson, and Joachim Hornegger. Automatic no- reference quality assessment for retinal fundus images using vessel segmentation, 2013. 5, 16

    2013

  24. [32]

    A chatgpt aided explainable frame- work for zero-shot medical image diagnosis

    Jiaxiang Liu, Tianxiang Hu, Yan Zhang, Xiaotang Gai, Yang Feng, and Zuozhu Liu. A chatgpt aided explainable frame- work for zero-shot medical image diagnosis. arXiv preprint arXiv:2307.01981, 2023. 4

    2023 arXiv

  25. [33]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 2

    2024

  26. [34]

    Brain tumor mri dataset, 2021

    Msoud Nickparvar. Brain tumor mri dataset, 2021. 5, 16

    2021

  27. [35]

    Kvasir: A multi-class image dataset for com- puter aided gastrointestinal disease detection

    Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Gri- wodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and P˚al Halvorsen. Kvasir: A multi-class image dataset for co...

    2017

  28. [36]

    Indian diabetic retinopathy image dataset (idrid), 2018

    Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fab- rice Meriaudeau. Indian diabetic retinopathy image dataset (idrid), 2018. 5, 16

    2018

  29. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  30. [38]

    A closer look at the few-shot adaptation of large vision-language models

    Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23681–23690, 2024. 3

    2024

  31. [39]

    Tahir, Muhammad E.H

    Anas M. Tahir, Muhammad E.H. Chowdhury, Amith Khan- dakar, Tawsifur Rahman, Yazan Qiblawey, Uzair Khurshid, Serkan Kiranyaz, Nabil Ibtehaz, M. Sohel Rahman, Somaya Al-Maadeed, Sakib Mahmud, Maymouna Ezeddin, Khaled Hameed, and Tahir Hamid. Covid-19 infection localization and s...

    2021

  32. [40]

    The ham10000 dataset, a large collection of multi-source der- matoscopic images of common pigmented skin lesions

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source der- matoscopic images of common pigmented skin lesions. Sci- entific data, page 180161, 2018. 5, 16

    2018

  33. [41]

    The ham10000 dataset, a large collection of multi-source der- matoscopic images of common pigmented skin lesions

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source der- matoscopic images of common pigmented skin lesions. Sci- entific data, 5(1):1–9, 2018. 8

    2018

  34. [42]

    Advances in medical image seg- mentation: A comprehensive review of traditional, deep learning and hybrid approaches

    Yan Xu, Rixiang Quan, Weiting Xu, Yi Huang, Xiaolong Chen, and Fengyuan Liu. Advances in medical image seg- mentation: A comprehensive review of traditional, deep learning and hybrid approaches. Bioengineering, 11(10): 1034, 2024. 2

    2024

  35. [43]

    Visual- language prompt tuning with knowledge-guided context op- timization, 2023

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context op- timization, 2023. 2, 7

    2023

  36. [44]

    Visual- language prompt tuning with knowledge-guided context op- timization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context op- timization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6757–6767,

  37. [45]

    Prompt engi- neering paradigms for medical applications: scoping review and recommendations for better practices

    Jamil Zaghir, Marco Naguib, Mina Bjelogrlic, Aur ´elie N´ev´eol, Xavier Tannier, and Christian Lovis. Prompt engi- neering paradigms for medical applications: scoping review and recommendations for better practices. arXiv preprint arXiv:2405.01249, 2024. 7

    2024 arXiv

  38. [46]

    Prefer: Prompt ensemble learning via feedback-reflect-refine

    Chenrui Zhang, Lin Liu, Chuyuan Wang, Xiao Sun, Hongyu Wang, Jinpeng Wang, and Mingchen Cai. Prefer: Prompt ensemble learning via feedback-reflect-refine. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 19525–19532, 2024. 4

    2024

  39. [47]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling. arXiv preprint arXiv:2111.03930, 2021. 1, 3, 6

    2021 arXiv

  40. [48]

    Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu 10 Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jian- feng Gao, Matthew P. Lungren, Tristan Naumann, Sheng Wang, and...

    2024

  41. [49]

    Clip in medical imaging: A comprehensive sur- vey

    Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, et al. Clip in medical imaging: A comprehensive sur- vey. arXiv preprint arXiv:2312.07353, 2023. 2

    2023 arXiv

  42. [50]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816–16825, 2022. 1, 2, 6, 7

    2022

  43. [51]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348, 2022. 1, 2, 6, 7

    2022

  44. [52]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Han- wang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659–15669, 2023. 6

    2023

  45. [53]

    Prompt-aligned gradient for prompt tuning, 2024

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning, 2024. 2, 7 11 BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models Supplementary Material

    2024

  46. [54]

    Each dataset is described in terms of its imaging modality, target organ(s), number of classes, and dataset splits (train/validation/test)

    Detailed Dataset Overview Table S7 provides a summary of the 11 datasets used for our proposed BiomedCoOp, covering 9 biomedical imaging modalities, such as CT, MRI, X-ray, ultrasound, and others, and 10 different organs. Each dataset is described in terms of its imaging modal...

  47. [55]

    It underscores BiomedCoOp’s robustness in adapting to limited data

    Additional Few-shot Results Figure S1 demonstrates the performance variations of BiomedCoOp and the baseline models under different few-shot configurations ( K = 1,2,4,8,16). It underscores BiomedCoOp’s robustness in adapting to limited data. On the other hand, we provide the ...

  48. [56]

    A photo of [CLASS]

    Learnable Context Interpretability In this experiment, we aim to investigate the closest words to each of the four learned context tokens in various biomedical datasets, examining how these nearest words align with visual or anatomical characteristics in the im- ages. This cou...

  49. [57]

    A shorter context length, such as 4, achieves a better balance between base and novel accuracy, resulting in a higher har- monic mean (HM) score

    Effect of Context Length As shown in Table S1, increasing the context length tends to reduce performance on both base and novel classes. A shorter context length, such as 4, achieves a better balance between base and novel accuracy, resulting in a higher har- monic mean (HM) s...

  50. [58]

    Figure S2 illus- trates the impact of increasing the selection threshold ( ζs) for the absolute value of the modified z-score to allow more prompts generated by the LLM to be used

    Effect of Prompt Selection Threshold In the Knowledge Distillation with Selective Prompting (KDSP) component of our proposed method, we used a statistics-based prompt selection strategy. Figure S2 illus- trates the impact of increasing the selection threshold ( ζs) for the abs...

  51. [59]

    To verify the effect of prompt selection for SCCM, we compare the model per- formance with and without prompt selection

    Selective Prompting for SCCM We didn’t perform prompt selection in the Semantic Con- sistency by Contextual Mapping (SCCM) component of our proposed BiomedCoOp framework. To verify the effect of prompt selection for SCCM, we compare the model per- formance with and without pro...

  52. [60]

    We also compare with zero- shot methods (in blue)

    Additional Comparisons with Other Re- cent Methods We compared our method with two more recent SOTA CLIP adaptation methods (XCoOp [3] and DCPL [6]) on all datasets in Tables S2 and S3. We also compare with zero- shot methods (in blue). All methods were tuned to their optimal ...

  53. [61]

    Effect of LLM used Table S4 presents an ablation study using 50 prompts from three recent LLMs across three datasets under the 4-shot evaluation. The results show that our model is robust to different text distributions, even with smaller LLMs like Gemma-2-2b, highlighting the...

  54. [62]

    Additional Hyperparameters Table S6 outlines the selected hyperparameters ( λ1, λ2, and ζs) used across various datasets for BiomedCoOp’s few-shot and base-to-novel benchmarks. These parame- ters were optimized to balance classification accuracy and model adaptability, with λ1...

  55. [63]

    The image of a normal brain on MRI shows a clear differentiation between different brain regions with no disruptions

    LLM Prompts Used We include here one text prompt generated from GPT-4 for each class across all the datasets: “The image of a normal brain on MRI shows a clear differentiation between different brain regions with no disruptions.” “Central necrosis and surrounding edema in glio...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.