{"id":"037aa961-acf3-4a5d-a720-b733b3c1e16b","arxiv_id":"2501.11231","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KPL combines LLM-generated class descriptions, visual retrieval, and a log-space Greenkhorn algorithm to boost CLIP zero-shot accuracy on medical and natural image datasets.","lead":"KPL, a training-free method, mines knowledge from CLIP by combining LLM-generated disease descriptions with visual retrieval and a stabilized Sinkhorn algorithm, improving zero-shot medical image classification. It might interest practitioners because it substantially boosts CLIP on tasks like diabetic retinopathy and malaria detection without any labeled training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may be an artifact of transductive adaptation: Eqs. 3, 4, and 1 all consume the test set D, and grid-selected hyperparameters add further leakage, so Table 1 does not demonstrate zero-shot generalization.","rationale":"The paper is clearly written, provides code, and the large accuracy differences over InMaP are real evidence for transductive improvement. The reader's weakest assumption is exactly the load-bearing issue: D is used in retrieval (Eq. 3 global mean), pseudo-label generation (Eq. 4), and proxy fitting (Eq. 1), so Table 1 is a self-consistent transductive score, not a conventional zero-shot score. This matters because the abstract and Section 4.2 claim 'zero-shot' and 'outperforming all baselines'; CLIP, VCD, and CMD are inductive, making the comparison uneven. The InMaP comparison is fairer and supports a conditional acceptance, but the paper must either add a disjoint held-out evaluation or explicitly frame all numbers as transductive and avoid the 'zero-shot' overstatement. The grid-search detail reinforces the concern because choosing k, Nmax, and tau on the target dataset can leak information beyond the unlabeled adaptation. No internal inconsistency is apparent; the issue is protocol. A single split-half experiment would settle whether the advantage survives without test-set adaptation.","tokens_in":13587,"tokens_out":5584,"duration_ms":58415,"concrete_test":"Run KPL on each medical dataset with a 50/50 split of images from the same source. Apply the full adaptation only on the unlabeled first half: compute Eq. 3's global mean, Eq. 4's pseudo-labels, and Eq. 1's W on that half, then classify the held-out half. Compare held-out accuracy with the reported D-adapted accuracy and with CLIP, VCD, CMD, and InMaP under the same split. If held-out accuracy is within noise of the inductive baselines, or if InMaP under the same split matches KPL, the Table 1 gains are transductive rather than zero-shot. Also report mean and standard deviation over multiple random splits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that KPL 'consistently and significantly outperforms the baselines.' This rests on accuracy computed after the method has already used the unlabeled test images in three places. In Eq. 3, the global image mean (1/|D|)∑F(x) is used to select descriptions, so retrieval is tuned to D. In Eq. 4, M is built from F(x_i) for every x_i in D, and the SG/Sinkhorn pseudo-label matrix is solved on D. In Eq. 1, the multimodal proxies W are fit to those pseudo-labels on D, and final accuracy is measured on the same images that generated the retrieval, pseudo-labels, and W. This is a transductive protocol. CLIP, VCD, and CMD are inductive, so comparing them to KPL conflates test-set adaptation with method quality. The paper does not disclose this and calls the result 'zero-shot,' which is overstated. The comparison with InMaP, which is also transductive, is fairer and still shows gains, so the core transductive improvement may stand, but the headline claim of outperforming all baselines as a zero-shot method is not established. In addition, the Appendix states that k, Nmax, and tau are determined by grid search on each target dataset; unless a labeled validation split is used, selecting hyperparameters by the reported accuracy is a second leakage path.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Knowledge Proxy Learning (KPL), a training-free method for CLIP-based image classification aimed primarily at medical datasets. KPL first uses an LLM to generate a large set of class descriptions, then ranks these descriptions by cosine similarity to the global mean of the test image features (Eq. 3) to form semantic text proxies. It then solves an entropic optimal-transport problem on the same test set to obtain pseudo-labels (Eq. 4), using a proposed 'Stable GreenkHorn' (SG) variant of Sinkhorn scaling, and fits multimodal proxies by minimizing KL divergence (Eq. 1). The final classifier is evaluated on the same images that were used for retrieval, pseudo-labeling, and proxy fitting. Experiments on five medical and four natural image datasets compare KPL with CLIP, VCD, CMD, and InMaP, reporting accuracy gains in most settings, with ablations on the KEB, visual-based retrieval, and SG components.","tokens_in":13831,"tokens_out":4731,"duration_ms":46641,"significance":"If the reported gains survive an honest evaluation protocol, KPL would be a practically useful extension of proxy-learning for medical image classification. The paper's strengths include a clearly stated two-stage method, release of code, experiments across multiple datasets and backbones, and ablation studies that attempt to isolate the contribution of each component. However, the current evaluation protocol is transductive: the unlabeled test set D is used in Eqs. 1, 3, and 4, and accuracy is then computed on the same D. The comparison against inductive baselines (CLIP, VCD, CMD) therefore conflates test-set adaptation with method quality, and the abstract's 'zero-shot' claim is overstated. The comparison with InMaP, which is also transductive, is more appropriate and does show gains, but hyperparameter selection by grid search on each target dataset without a described validation split introduces additional leakage. These issues are load-bearing for the central empirical claim and require substantive revision rather than mere copy-editing.","major_comments":[{"comment":"The method uses the full test set D to compute the global image mean for visual-based retrieval (Eq. 3), to construct the matrix M and solve the Sinkhorn/SG pseudo-label problem (Eq. 4), and to fit the multimodal proxies W in Eq. 1, and then evaluates accuracy on the same images in D. This is a transductive protocol, so the accuracies in Tables 1 and 2 are in-sample and do not demonstrate zero-shot out-of-sample generalization. The comparison with CLIP, VCD, and CMD, which do not see D, is therefore not apples-to-apples. I recommend either evaluating on a genuinely separate unseen test set or explicitly reframing the setting as transductive and restricting the headline comparison to other transductive methods such as InMaP.","section":"§4.2, Eqs. 1, 3, 4"},{"comment":"The paper states that k, Nmax, and τ are determined through grid search on each target dataset (Appendix Figures 1 and 2), but no labeled validation split is mentioned. If the grid search selects hyperparameters by the final accuracy on D, label information leaks into the evaluation. Please specify exactly what data is used for model selection, or report results across a range of hyperparameters with sensitivity analysis; otherwise the reported numbers cannot be taken as unbiased estimates of accuracy.","section":"§4.1, Experimental Setup"},{"comment":"The claim that KPL 'consistently and significantly outperforms' the baselines is not supported by any measure of variance, multiple runs, or significance tests; the tables report single-point accuracies. Either provide mean ± std over several runs with a significance test, or replace the word 'significantly' with a statement about observed improvements on the reported runs.","section":"§4.2, Tables 1 and 2"},{"comment":"The ablation study does not include a variant that uses KEB text proxies with the original Sinkhorn pseudo-labeling, nor a variant that uses VCD descriptions with the proposed SG algorithm. Without such cross-condition ablations, the contribution of the knowledge-enhanced base and the contribution of SG relative to InMaP are not isolated, and the claim that both components are necessary is not fully supported.","section":"§4.3, Table 3"}],"minor_comments":[{"comment":"The text contains the typo 'Apendix Figure 1' and 'Apendix Figure 2'; these should read 'Appendix Figure 1' and 'Appendix Figure 2'.","section":"§4.1, Experimental Setup"},{"comment":"The citation 'V ondrick' appears with a spurious space (e.g., in Menon and V ondrick 2023); please fix the LaTeX formatting.","section":"References"},{"comment":"The proposed algorithm is called 'Stable GreenkHorn' in Section 3.2 but 'Greenkhorn' in the related-work paragraph; standardize the capitalization.","section":"§3.2 and Preliminaries"},{"comment":"The symbol 'KPLs' is used in Table 2 without a definition in the caption or the main text; please explain that it denotes KPL with Sinkhorn rather than SG.","section":"Table 2"},{"comment":"The caption states 'k=3' while the text writes 'k = 3' in prose; unify the notation and define k in the caption.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is currently vulnerable to the transductive-leakage critique: the method adapts to the test set and is then evaluated on the same set, and hyperparameters appear to be chosen on the test accuracy. This should be addressed before publication. The technical novelty over InMaP/Greenkhorn is modest, but the combination of LLM-retrieved descriptions with transductive proxy learning may still be worth publishing if the evaluation is made honest, for example by using a validation split and clearly framing the method as transductive. The provided text lacks the appendix figures and tables referenced in the main body, which hampers reproducibility assessment; I recommend the editor ask for the complete appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes KPL, a training-free pipeline that boosts CLIP zero-shot classification by building a large LLM-generated description base per class, retrieving the top-k descriptions most similar to the global image mean of the test set, and fitting multimodal proxies using a log-space Greenkhorn solver. On five medical datasets it reports big jumps over CLIP, VCD, CMD, and InMaP—e.g., IDRiD from 6.8 to 44.8 with ViT-L/14. The code is released, and no weights are updated.\n\nWhat is new here is the specific combination: a large knowledge base with visual-retrieval-based text proxy selection plus a stabilized Greenkhorn for proxy learning. The theoretical grounding (Sinkhorn scaling, Proposition 1) is standard but correct. The clear gains over InMaP—which is also transductive, using the unlabeled test set—suggest that the added knowledge base and the log-space updates provide a real practical advantage. That is the strongest part of the paper.\n\nThe soft spot is the evaluation protocol. The method consumes the unlabeled test set D in three places: the global image mean in Eq. 3 for retrieval, the pseudo-label matrix in Eq. 4, and the final proxy fitting in Eq. 1 on the same images that are later classified. This is transductive learning, not zero-shot generalization. The baselines CLIP, VCD, and CMD are inductive, so comparing them to KPL conflates adaptation power with method quality. The paper does not disclose this and repeatedly calls the result 'zero-shot,' which is overstated. The comparison to InMaP is fairer and still shows gains, so the core transductive improvement likely holds, but the headline claim of outperforming all baselines as a zero-shot method is not established.\n\nThere is also a second leakage path: the hyperparameters k, Nmax, and tau are selected by grid search per dataset, and the appendix gives no indication that a labeled validation split was used. If the grid is scored on the test accuracy, that further inflates the numbers. No error bars are reported.\n\nRead strictly as a transductive method—adapting proxies to an unlabeled target set at inference time—KPL is a useful contribution. The paper would benefit from renaming the task, reporting validation-based hyperparameter selection, and adding an inductive variant to show how much of the gain depends on seeing the test images.\n\nThis is a paper for people working on CLIP-based diagnosis in low-label settings and anyone interested in optimal-transport-based proxy learning. It deserves a serious referee, but the evaluation framing needs correction before acceptance.","headline":"Large transductive gains for CLIP on medical images, but the 'zero-shot' claim is overstated because the test set is used for adaptation; still a useful contribution and worth reviewing.","tokens_in":14478,"tokens_out":2889,"would_cite":true,"duration_ms":27753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free procedure, KPL, improves CLIP's zero-shot medical image classification by mining LLM-generated class descriptions and refining pseudo-labels to learn multimodal class proxies.","keywords":["CLIP","zero-shot classification","medical image classification","vision-language models","text proxy","pseudo-label refinement","Sinkhorn","knowledge-enhanced base"],"falsifier":"Use KPL on Shenzhen and Montgomery, two chest X-ray datasets with the same tuberculosis classes but different acquisition conditions: adapt KPL's proxies on one dataset and evaluate on the other without re-running the retrieval or pseudo-label steps. If accuracy then falls back to CLIP or VCD levels, the reported gains come from transductive use of the test images rather than from a genuine zero-shot capability.","tokens_in":1773,"feed_emoji":"🩺","tokens_out":5233,"duration_ms":106180,"temperature":0.7,"pith_summary":"This paper asks whether CLIP's zero-shot classifier is using the model's full capacity, and answers no for medical images. It introduces Knowledge Proxy Learning (KPL), a training-free inference procedure that first replaces each class name with a set of many LLM-generated visual descriptions, keeps only the descriptions most similar to the dataset's average image, and then learns class proxies in the shared vision-text space guided by pseudo-labels refined with a stabilized Sinkhorn-type algorithm. The paper reports that KPL consistently beats existing CLIP-based zero-shot methods, including description-based VCD and CMD and proxy-learning InMaP, on five medical datasets and in most configurations on four natural-image datasets, with large jumps such as malaria-cell accuracy rising from about 50% to above 80%. A sympathetic reader would care because the method shows that substantial accuracy can be extracted from a frozen CLIP model without additional training, which matters in medical settings where annotated data are scarce.","feed_headline":"Knowledge proxies lift CLIP zero-shot medical accuracy 38 points","feed_subtitle":"Training-free method retrieves LLM descriptions and refines pseudo-labels to beat CLIP-based baselines on medical data.","key_machinery":"The load-bearing object is the multimodal class proxy $w_j^{KPL}$, a learned class embedding in CLIP's shared vision-text space that serves as the classification template for class $j$. It is produced by minimizing the KL divergence between a pseudo-label distribution $Q_{F,G}$ and the softmax distribution $P_F(x, W)$ over image-proxy similarities. The pseudo-labels come from an entropy-regularized optimal transport problem whose cost matrix $M_{i,j}$ is the similarity between image $x_i$ and the averaged embeddings of the top-$k$ retrieved descriptions, and the matrix is solved by Stable Greenkhorn, a log-domain, greedily updated variant of Sinkhorn that the paper introduces to replace Sinkhorn for medical data.","core_discovery":"The paper's central claim is that a frozen CLIP model can classify medical images much more accurately by optimizing its class proxies in two steps rather than using plain category-name embeddings. In the first step, Text Proxy Optimization, KPL builds a knowledge-enhanced base of at least fifty LLM-generated visual descriptions per class, then retrieves the top-$k$ descriptions whose text embeddings are closest to the dataset's mean image feature, and averages them to form semantic text proxies. In the second step, Multimodal Proxy Learning, KPL treats these text proxies as anchors for an entropy-regularized pseudo-label matrix $Q_{F,G}$, refines that matrix with the proposed Stable Greenkhorn (SG) algorithm instead of Sinkhorn, and then learns multimodal class proxies $W^{KPL}$ by minimizing the KL divergence between the pseudo-label distribution and the softmax of image-proxy similarities. The paper reports that this two-step procedure outperforms all four baselines on all five medical datasets with both tested CLIP backbones, with the largest improvement on MalariaCell rising from 50.83% to 80.86%, and that it also improves a medically pretrained BioMedCLIP model, including a Cataract gain from 16.47% to 51.25%.","pith_inferences":["The reported accuracies are transductive: both description retrieval and pseudo-label estimation use the same unlabeled images that are later classified, so KPL is not a conventional out-of-sample zero-shot classifier; a fair deployment test would freeze the adaptation on one set and evaluate on a disjoint sample.","Because the largest gains occur on classes where CLIP's text-name baseline is near chance, such as MalariaCell, KPL may be surfacing discriminative visual knowledge CLIP already contains, meaning the method's ceiling is bounded by what CLIP's text encoder can express.","The method's dependence on LLM prompt quality and on the dataset-mean image for retrieval suggests its gains will likely vary across medical modalities and dataset homogeneity, which is testable on histopathology or ultrasound data.","A natural extension is to replace the global mean image with a small labeled support set for retrieval; if the gains persist, the same framework could become a few-shot adaptable inference procedure."],"forward_implications":["On the five medical benchmarks in Table 1, KPL outperforms CLIP, VCD, CMD, and InMaP with both ViT-L/14 and ViT-L/14@336px backbones.","Applying KPL to BioMedCLIP, a domain-specific medical vision-language model, improves its zero-shot accuracy on all tested medical datasets, with Cataract rising from 16.47% to 51.25%.","The same procedure transfers to natural image datasets, where KPL with Sinkhorn beats CLIP, VCD, Hierarchical, and InMaP in most of the nine backbone-dataset configurations in Table 2.","Ablation results indicate that all three components matter: the LLM-generated knowledge-enhanced base beats VCD's descriptions, visual-based retrieval beats random description selection, and Stable Greenkhorn beats Sinkhorn for pseudo-label refinement."],"supporting_citations":[{"why":"Supplies the CLIP model, its vision and text encoders, and the vanilla zero-shot classification baseline that KPL builds on and compares against.","marker":"Radford et al. 2021"},{"why":"Introduces VCD, the LLM-description-based classification method whose single-level descriptions KPL extends and whose text proxies are used as a baseline and ablation comparison.","marker":"Menon and Vondrick 2023"},{"why":"Introduces InMaP, the intra-modal proxy learning method with Sinkhorn-refined pseudo-labels that KPL directly modifies and must outperform.","marker":"Qian, Xu, and Hu 2024"},{"why":"Introduces Greenkhorn, the greedy matrix scaling algorithm that the proposed Stable Greenkhorn adapts into log space.","marker":"Altschuler, Niles-Weed, and Rigollet 2017"},{"why":"Establishes the uniqueness and convergence of the matrix scaling solution that underlies both Sinkhorn and Stable Greenkhorn pseudo-label generation.","marker":"Sinkhorn 1964"},{"why":"Provides CMD, a medical-domain CLIP description method, and supplies the medical dataset setup used for evaluation.","marker":"Liu et al. 2023b"},{"why":"Provides the Hierarchical comparison baseline used on natural image datasets to test KPL's generalizability.","marker":"Ren, Su, and Liu 2024"},{"why":"Provides BioMedCLIP, the domain-specific CLIP checkpoint used to show that KPL also improves a medically pretrained model.","marker":"Zhang et al. 2023"}],"fun_headline_variants":["Frozen CLIP gains 38 points on medical zero-shot via knowledge proxies","Training-free KPL boosts CLIP zero-shot medical accuracy by 38","LLM text retrieval and stable pseudo-labels improve CLIP medical imaging","KPL: CLIP zero-shot on medical images jumps from 50.83% to 80.86%"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The method assumes that adapting the class proxies using the very images it will later classify, through the dataset-average image used for retrieval and the pseudo-labels computed from those same images, produces a classifier whose higher scores on those images reflect genuine zero-shot ability rather than in-sample fitting.","fun_headline_variants_meta":{"raw":{"variants":["Frozen CLIP gains 38 points on medical zero-shot via knowledge proxies","Training-free KPL boosts CLIP zero-shot medical accuracy by 38","LLM text retrieval and stable pseudo-labels improve CLIP medical imaging","KPL: CLIP zero-shot on medical images jumps from 50.83% to 80.86%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000624,"raw_usage":{"total_tokens":2941,"prompt_tokens":1052,"completion_tokens":1889,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":1800}},"tokens_in":668,"tokens_out":1889,"duration_ms":13788,"temperature":1.0,"reasoning_tokens":1800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:30:45.077392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use KPL on Shenzhen and Montgomery, two chest X-ray datasets with the same tuberculosis classes but different acquisition conditions: adapt KPL's proxies on one dataset and evaluate on the other without re-running the retrieval or pseudo-label steps. If accuracy then falls back to CLIP or VCD levels, the reported gains come from transductive use of the test images rather than from a genuine zero-shot capability.","supporting_citations":[{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP model, its vision and text encoders, and the vanilla zero-shot classification baseline that KPL builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces VCD, the LLM-description-based classification method whose single-level descriptions KPL extends and whose text proxies are used as a baseline and ablation comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces InMaP, the intra-modal proxy learning method with Sinkhorn-refined pseudo-labels that KPL directly modifies and must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Greenkhorn, the greedy matrix scaling algorithm that the proposed Stable Greenkhorn adapts into log space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the uniqueness and convergence of the matrix scaling solution that underlies both Sinkhorn and Stable Greenkhorn pseudo-label generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Hierarchical comparison baseline used on natural image datasets to test KPL's generalizability."}],"review_version":1}