{"id":"d39771ec-4134-4895-ba92-236ff1b407ba","arxiv_id":"2411.15232","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BiomedCoOp improves few-shot biomedical image classification by aligning learnable prompts with selectively pruned LLM-generated prompt ensembles and distilling their knowledge into BiomedCLIP.","lead":"BiomedCoOp is a prompt-learning method that adapts the biomedical vision-language model BiomedCLIP to classify medical images using only a few labeled examples per disease class. It combines text prompts generated by large language models with a selective knowledge-distillation step, and reports accuracy gains over existing prompt-learning methods across 11 medical imaging datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main comparison table appears to under-report the strongest baseline; the paper's own supplementary per-dataset results give KgCoOp a higher average at every shot, so the claimed SOTA margin is not yet reliably established.","rationale":"The reader's conditional verdict is appropriate: the central empirical claim is plausible and likely to survive correction, but the paper has reporting problems that must be fixed. The reader's weakest_assumption focuses on the KDSP teacher being constructed from the same few-shot support images used to train the student. That is a legitimate concern about the method's mechanism and generalization, but it is secondary to the more immediate threat: the main evidence table is inconsistent with the paper's own per-dataset results. A comparative claim cannot be evaluated if the baseline numbers in the headline table do not match the detailed results. Since the correction still leaves BiomedCoOp ahead in the averages I can check, I do not recommend rejecting the paper; the correct response is to require the authors to reconcile the tables and re-verify the comparison before the result is relied upon. This keeps the reader's CONDITIONAL verdict unchanged, while shifting the emphasis to the most direct threat to the empirical claim.","tokens_in":24296,"tokens_out":8180,"duration_ms":84311,"concrete_test":"Recompute every average row of Table 1 from the per-dataset entries in Table S8, for all methods and all K values. For each mismatch, check the released code or experimental logs to determine which table reflects the actual runs. Then recompute the margins between BiomedCoOp and the best baseline under the corrected values, and verify whether BiomedCoOp remains the top method at every K and whether the reported standard deviations or significance statements change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BiomedCoOp gives the highest few-shot accuracy relative to state-of-the-art baselines. The evidence for this is Table 1, but Table 1 is not internally consistent with the paper's own detailed results. For KgCoOp, the average row in Table 1 reports K=1: 50.85, K=2: 53.18, K=4: 57.82, K=8: 62.08, K=16: 62.84. The per-dataset KgCoOp rows in Table S8 average to K=1: 51.83, K=2: 53.47, K=4: 58.59, K=8: 63.65, K=16: 64.88, and those averages are exactly consistent with the listed per-dataset entries. The paper never says which table corresponds to the actual experimental runs. Because the headline is a comparative claim, a comparison whose main table disagrees with its own detailed table cannot support the reported margins until the discrepancy is resolved. Correcting KgCoOp would not necessarily overturn the ranking, but it changes the reported margins (e.g., the K=16 gap over KgCoOp shrinks from 9.58 to 7.54 points) and raises the possibility that other baseline entries are also misreported. This must be settled before the 'highest accuracy' claim can be relied upon.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces BiomedCoOp, a prompt-learning framework that adapts the BiomedCLIP vision-language model to few-shot biomedical image classification. The method combines two components: Semantic Consistency by Contextual Mapping (SCCM, Eq. 9), which aligns learnable context prompts with the mean embedding of N GPT-4-generated class-specific prompts, and Knowledge Distillation with Selective Prompting (KDSP, Eqs. 4-7 and 10), which prunes LLM prompts via a modified-z-score outlier filter based on similarity to the few-shot training images and distills the student toward the pruned teacher via KL divergence. The total loss is L = L_CE + λ1 L_SCCM + λ2 L_KDSP (Eq. 11). Reported results on 11 biomedical datasets spanning 9 modalities claim the highest few-shot accuracy (57.03% at K=1 and 72.42% at K=16) and the best base-to-novel harmonic mean (75.07%) among the compared prompt-learning, adapter, and linear-probe baselines. Ablations examine the contribution of each component, the number of LLM prompts, the choice of VLM backbone, the LLM, the context length, and the selection threshold.","tokens_in":24607,"tokens_out":19063,"duration_ms":144747,"significance":"Biomedical few-shot classification is practically important and under-served, and the paper's benchmark of 11 datasets across 9 modalities is a useful contribution in itself. If the reported numbers hold, the gains are large (roughly 5-10 points over the strongest baselines at low shots) and the design is well motivated: LLM-generated prompts inject domain knowledge, and the outlier-pruned teacher is a sensible way to avoid over-specialized prompts. The paper ships public code, averages results over three support sets with standard deviations, and includes ablation and sensitivity studies (LLM choice, prompt count, backbone, context length, selection threshold). The main reservation is the reliability of the comparative evidence: the main few-shot table is inconsistent with the paper's own per-dataset supplementary table for the KgCoOp baseline at every shot level, and this inconsistency must be resolved before the claimed state-of-the-art margins can be accepted.","major_comments":[{"comment":"The headline claim that BiomedCoOp achieves the highest few-shot accuracy rests on Table 1, but Table 1 is inconsistent with the paper's own per-dataset results in Table S8 for the KgCoOp baseline at every shot level. Table 1 reports KgCoOp averages of 50.85, 53.18, 57.82, 62.08, and 62.84 for K=1,2,4,8,16, while the per-dataset KgCoOp entries in Table S8 average to 51.83, 53.47, 58.59, 63.5, and 64.88, matching the supplementary average row (which lists 51.83, 53.47, 58.59, 63.65, 64.88; note that the K=8 average row itself does not exactly equal the mean of the listed per-dataset entries). The manuscript never states which table reflects the actual experimental runs. Correcting KgCoOp reduces the reported K=1 margin from 6.18 to 5.20 points and the K=16 margin from 9.58 to 7.54 points; at K=16 the best-reported baseline, Tip-Adapter-F (70.91), is within 1.5 points of BiomedCoOp, so a comparable error in any other baseline row could change the ranking. The authors should reconcile the two tables, state which numbers are the official results, and audit every baseline row in Table 1 against the appendix before the state-of-the-art claim can be accepted.","section":"Table 1 vs. Table S8"},{"comment":"The KDSP teacher is selected by scoring the LLM prompts against the same few-shot training images that are later used to optimize the student (Eq. 4 in Section 3.3 versus Eqs. 8-11 in Section 3.4). At K=1 or K=2, the score for each class is computed from one or two images, so outlier pruning can select prompts that happen to match those particular support images, and the KL term (Eq. 10) then distills a support-set-specific teacher rather than general biomedical knowledge. This is a genuine circularity risk for the few-shot gains, not a presentational issue. Please add a control experiment that separates prompt selection from student training (e.g., select the teacher on one half of the support set and train on the other half, or compare against a teacher that keeps all N prompts), and discuss the risk explicitly in the text.","section":"Section 3.3, Eqs. (4) and (10)"},{"comment":"The comparative claim relies on a fair tuning protocol, but the manuscript reports per-dataset selection of λ1, λ2, and ζs over wide ranges (e.g., λ2 from 0.25 to 20.0 and ζs from 1.25 to 2.0 in Table S6) while only stating that baselines were 'tuned to their optimal settings' without giving their search budgets or final hyperparameters. Please document the tuning protocol for every method (or state that published default settings were used for baselines), and report the value of the scaling factor β in Eq. (4), which is never specified in Section 4.1.","section":"Section 4.1 and Table S6"}],"minor_comments":[{"comment":"The claim that this study 'adopts BiomedCLIP for prompt learning for the first time' is contradicted by the supplementary comparison against XCoOp, which the authors describe as 'directly utilizing the BiomedCLIP backbone'; please qualify the novelty statement.","section":"Section 1, Contribution 3"},{"comment":"What is called a modified z-score conventionally includes the Iglewicz-Hoaglin constant 0.6745; as written, the ζs values are applied to the unscaled statistic, so please state the exact definition used.","section":"Eq. (7)"},{"comment":"The text says results are 'averaging results over three runs,' whereas the Table 1 caption says the standard deviation is 'derived from 3 sampled support sets'; please clarify what the reported ± values represent.","section":"Section 4.1 and Table 1 caption"},{"comment":"KgCoOp is cited as [43] in the text and [44] in Table 1, and ProGrad as [53] in the text and [52] in Table 1; please unify the numbering.","section":"References"},{"comment":"The stated margins over ProGrad (5.2% at K=1 and 4.6% at K=2) do not match the table values (57.03−51.88=5.15 and 59.13−54.71=4.42); please round consistently.","section":"Section 4.2"},{"comment":"The LC25000 split is written as '12500/5000//7500' with a double slash.","section":"Table S7"},{"comment":"The acronym LSDM is used without definition.","section":"Supplementary Section 12"},{"comment":"The text refers to gray and green highlights, but the table as printed lacks a legend mapping each row to the BiomedCLIP/CLIP backbone and to the full method; please add one.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The main blocking issue is the Table 1 / Table S8 inconsistency for KgCoOp; I recommend requesting the raw per-seed, per-dataset numbers for all methods before acceptance. It is also worth asking the authors to disclose the tuning budget for the baselines, since λ1, λ2, and ζs are searched over very wide ranges. Finally, the 'first to adopt BiomedCLIP for prompt learning' claim should be checked against the prior literature, including XCoOp, which the paper itself uses as a BiomedCLIP-based baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is genuinely new and the evaluation is thorough. But the paper's central comparative claim is undermined by an internal inconsistency: Table 1's KgCoOp averages (50.85 at K=1, 62.84 at K=16) are lower than what their own per-dataset Table S8 implies (51.83 and 64.88). The discrepancy is consistent across every shot. I checked the per-dataset rows; the S8 averages are correct as arithmetic. The paper never explains which table corresponds to the real runs. That has to be fixed before anyone quotes the margins.\n\nWhat's good: combining average LLM-prompt embeddings as an MSE target with KL distillation from a z-score-pruned teacher is a real combination, not present in CoOp/CoCoOp/KgCoOp/ProGrad/XCoOp/DCPL. The ablations are informative: SCCM alone helps, KDSP alone can hurt with a non-biomedical backbone, and the two together are better. The prompt-count, context-length, threshold, and LLM-choice studies are the right kind of sensitivity analysis. Code is released. That's a solid empirical package.\n\nSoft spots besides the table: the paper claims to be the first to apply prompt learning to BiomedCLIP, but the supplementary itself says XCoOp uses the BiomedCLIP backbone. That overstatement should be removed. Hyperparameters λ1, λ2, ζs are tuned per dataset and benchmark; Table S6 shows a wide range (e.g., λ1 from 0.25 to 20). That's a lot of tuning freedom for few-shot claims, though not disqualifying. The base-to-novel split is not specified beyond 'divided into base and novel classes' — readers can't reproduce the split. Also, Table 3's final HM (74.64) differs from Table 2's 75.07 for the same method; likely a seeds/dataset difference, but it needs a footnote.\n\nThe central concern, though, is the table discrepancy. It doesn't necessarily overturn the ranking — BiomedCoOp still leads even against the higher KgCoOp numbers — but it changes the margins (K=16 gap shrinks from 9.58 to 7.54) and calls into question whether other baseline entries were transcribed correctly. This is a load-bearing issue for a comparative benchmark paper.\n\nWho this is for: anyone working on prompt learning for medical VLMs. It deserves a serious referee, but the authors should be asked to reconcile the tables, release the per-dataset baseline numbers, and specify the base-to-novel splits. I'd conditional-accept if those are fixed; desk-rejecting would be wrong because the method is plausible and the evaluation is broad.","headline":"Solid, well-ablated prompt-learning method; but the main table disagrees with its own supplementary on a key baseline, so the SOTA claim is not yet reliable.","tokens_in":25182,"tokens_out":2595,"would_cite":false,"duration_ms":22558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BiomedCoOp claims that anchoring learnable prompt contexts to ensembled LLM class descriptions and to a pruned teacher distribution lets a frozen BiomedCLIP outperform existing prompt-learning, adapter, and linear-probe methods on 11…","keywords":["prompt learning","vision-language models","biomedical image classification","few-shot learning","LLM prompt ensembling","knowledge distillation","base-to-novel generalization","BiomedCLIP"],"falsifier":"Replace GPT-4 prompts with shallow text such as 'a photo of [CLASS]' while keeping both losses; if $K=1$ accuracy stays near 57.03%, the specific semantic content of the LLM prompts is not what drives the gain. Alternatively, build $K=1$ support sets from only the most atypical examples per class and check whether accuracy falls to or below the ProGrad baseline of 51.88%, which would show that Eq. 4's outlier pruning relies too heavily on support-set representativeness.","tokens_in":24064,"feed_emoji":"🩻","tokens_out":9876,"duration_ms":87243,"temperature":0.7,"pith_summary":"This paper tries to establish that few-shot biomedical image classification can be improved substantially by learning prompt contexts for a biomedically pretrained vision-language model rather than fine-tuning the model. The proposed framework, BiomedCoOp, claims to achieve this by anchoring the learnable text prompt to averaged GPT-4-generated class descriptions and by distilling knowledge from a selectively pruned set of LLM prompts. On 11 datasets spanning 9 imaging modalities, it reports the best average few-shot accuracy among the compared methods at every shot level, including 57.03% at one label per class and 72.42% at 16 labels per class, and the best base-to-novel harmonic mean of 75.07%. If the claim is right, a handful of labeled scans plus an LLM's textual knowledge can replace extensive prompt engineering and full fine-tuning for diverse clinical imaging tasks.","feed_headline":"LLM-anchored prompts top 11 medical imaging sets","feed_subtitle":"On 11 datasets, one to sixteen labeled scans per class suffice to beat existing prompt-learning baselines.","key_machinery":"The load-bearing machinery is a pair of text-side losses attached to the learnable prompt context. SCCM (Eq. 9) is an MSE pull toward $P_g$, the mean embedding of $N$ LLM-generated class descriptions; it supplies general biomedical semantics. KDSP (Eq. 10) is a KL-divergence term that aligns the student's logits with a teacher formed by $P_s$, the average embedding of LLM prompts selected after scoring each prompt against the few-shot support images by cosine similarity and pruning outliers with a modified z-score based on the median absolute deviation (Eqs. 4-7). The two losses jointly keep the learned context close to LLM knowledge while preventing atypical prompts from pushing it into overspecialized semantic regions.","core_discovery":"BiomedCoOp's central claim is that two text-side supervision losses, applied to the learnable context vectors of CoOp-style prompt learning, are enough to adapt BiomedCLIP to biomedical few-shot classification without touching the pretrained weights. SCCM minimizes the squared distance between the learned context embedding and the average of GPT-4-generated, class-specific prompt embeddings; KDSP minimizes the KL divergence between the student's prediction distribution and a teacher distribution built from LLM prompts that survive a median-absolute-deviation outlier filter. The paper reports that this dual anchoring outperforms CoOp, CoCoOp, KgCoOp, ProGrad, adapter methods, and linear probes on 11 datasets at all shot counts, and that it also improves generalization to unseen classes, as measured by base-to-novel harmonic mean.","pith_inferences":["Because Eq. 4 scores prompts against the same support images that train the student, the approach implicitly assumes those few images represent each class; an adversarial support set made of only atypical presentations would test whether KDSP amplifies instead of corrects the resulting bias.","All new losses act on text embeddings with a frozen backbone, so the SCCM and KDSP recipe should transfer to other contrastive vision-language models; the paper's own backbone ablation indicates the gains are not unique to BiomedCLIP, though they are largest there.","The LLM query names the imaging modality, and the nearest-token analysis shows learned tokens like 'mri' and 'endoscopy', suggesting part of the advantage is modality grounding; removing the modality word from the query would separate that effect from pure class semantics.","Per-dataset tuning of $\\lambda_1$, $\\lambda_2$, and $\\zeta_s$ leaves open whether a single fixed configuration nearly matches the tuned results; a simple deployment test is to run with defaults derived from the paper's value ranges."],"forward_implications":["At $K=1$, the method reports 57.03% average accuracy, 5.2 points above the best baseline (ProGrad), so LLM-anchored prompt learning is particularly effective when labeled data is extremely scarce.","At $K=16$, it reports 72.42%, still the best among all compared methods, so the gains do not disappear as more labeled data arrives.","The base-to-novel harmonic mean of 75.07% (versus 67.43% for the best baseline) implies the learned prompts retain knowledge for unseen classes, supporting use in settings where disease categories evolve.","Ablations show SCCM alone helps few-shot accuracy while KDSP alone helps novel-class accuracy, and only their combination reaches the best results; removing either component lowers the harmonic mean.","Varying the number of LLM prompts from 10 to 50 raises zero-shot accuracy from 47.55% to 52.27%, so prompt diversity is a cheap source of generalization before any labeled data is used."],"supporting_citations":[{"why":"Supplies the frozen BiomedCLIP backbone whose vision and text encoders are adapted by the learned prompts.","marker":"[48]"},{"why":"Generates the 50 class-specific textual descriptions that provide the semantic anchor and the distillation teacher.","marker":"[1]"},{"why":"Defines CoOp's learnable context-vector formulation that BiomedCoOp builds on and compares against.","marker":"[51]"},{"why":"Provides the CoCoOp baseline and the conditional prompt-learning idea relevant to base-to-novel generalization.","marker":"[50]"},{"why":"ProGrad is the strongest few-shot baseline; BiomedCoOp reports a 5.2-point lead over it at $K=1$.","marker":"[53]"},{"why":"Establishes CLIP's contrastive image-text prediction (Eq. 1) that the method's logits and losses depend on.","marker":"[37]"},{"why":"KgCoOp is a knowledge-guided prompt-tuning baseline in Tables 1 and 2 that BiomedCoOp is compared with.","marker":"[43]"},{"why":"Tip-Adapter-F is the strongest adapter baseline at $K=16$ (70.91%), which BiomedCoOp surpasses by 1.51 points.","marker":"[47]"}],"fun_headline_variants":["LLM prompt anchoring tops 11 medical datasets in few-shot tests","BiomedCoOp: dual text-side losses boost biomedical prompt learning","Few-shot medical imaging: LLM-guided prompts beat existing methods","Prompt learning with LLM ensembles generalizes across 11 datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's teacher is built by scoring LLM prompts against the same few labeled images that later optimize the student, so it assumes those few images are representative enough of each class that outlier pruning selects prompts that also help on unseen test images.","fun_headline_variants_meta":{"raw":{"variants":["LLM prompt anchoring tops 11 medical datasets in few-shot tests","BiomedCoOp: dual text-side losses boost biomedical prompt learning","Few-shot medical imaging: LLM-guided prompts beat existing methods","Prompt learning with LLM ensembles generalizes across 11 datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1425,"prompt_tokens":940,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":556,"tokens_out":485,"duration_ms":5298,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:09:27.505283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace GPT-4 prompts with shallow text such as 'a photo of [CLASS]' while keeping both losses; if $K=1$ accuracy stays near 57.03%, the specific semantic content of the LLM prompts is not what drives the gain. Alternatively, build $K=1$ support sets from only the most atypical examples per class and check whether accuracy falls to or below the ProGrad baseline of 51.88%, which would show that Eq. 4's outlier pruning relies too heavily on support-set representativeness.","supporting_citations":[{"cited_title":"Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen BiomedCLIP backbone whose vision and text encoders are adapted by the learned prompts."},{"cited_title":"Conditional prompt learning for vision-language models","cited_arxiv_id":null,"evidence_quote":"Provides the CoCoOp baseline and the conditional prompt-learning idea relevant to base-to-novel generalization."},{"cited_title":"Prompt-aligned gradient for prompt tuning, 2024","cited_arxiv_id":null,"evidence_quote":"ProGrad is the strongest few-shot baseline; BiomedCoOp reports a 5.2-point lead over it at $K=1$."},{"cited_title":"Visual- language prompt tuning with knowledge-guided context op- timization, 2023","cited_arxiv_id":null,"evidence_quote":"KgCoOp is a knowledge-guided prompt-tuning baseline in Tables 1 and 2 that BiomedCoOp is compared with."}],"review_version":1}