{"id":"cff37fa1-dee6-46ae-9fe2-74713c971b97","arxiv_id":"2607.14125","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Giving each class its own prompt weights, estimated from the model's own pseudo-labels on unlabeled test images, improves zero-shot CLIP classification over global prompt weights.","lead":"This paper introduces CARPRT, a training-free way to give each image class its own set of prompt weights when classifying images with vision-language models, using only unlabeled images and black-box similarity queries. It reports consistent accuracy gains over shared prompt-weight ensembling on standard zero-shot benchmarks, and shows the method also helps prompt tuning and test-time adaptation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CARPRT's central claim hinges on pseudo-label quality; Prop. 1 does not prove convergence to true labels, and the paper's own EuroSAT/Aircraft results show the condition can fail.","rationale":"The reader's weakest assumption is that pseudo-label groups in Eq. 10 are dominated by genuine class images; this is exactly the condition under which the central claim holds. The paper itself provides evidence that the condition fails on EuroSAT and Aircraft in the iterative variant (App. E.2), and Prop. 1's proof is mathematically incomplete: Hoeffding bounds the deviation of pseudo-label frequencies from their expectation, not from true class probabilities. The oracle controlled experiment above would determine whether CARPRT's gains survive when pseudo-labels are poor. Since this concern qualifies the generality of the claim but does not overturn the reported positive results, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":48750,"tokens_out":5683,"duration_ms":59758,"concrete_test":"On all 11 datasets × 3 backbones, compute three accuracies: (i) WPE, (ii) CARPRT with pseudo-labels in Eq. 10, (iii) CARPRT with ground-truth labels in Eq. 10 (as the paper already does for ProDA in Eq. 15). Report the ratio (ACC_CARPRT_pseudo - ACC_WPE) / (ACC_CARPRT_oracle - ACC_WPE). If this ratio is near 0 or negative on low-accuracy datasets (EuroSAT, Aircraft, DTD), then pseudo-label noise is the bottleneck and the central claim is contingent; if CARPRT captures most of the oracle gain even there, the concern is mitigated. Additionally, on EuroSAT, directly compare the empirical pseudo-label distribution with the true label distribution to test Prop. 1's claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Sec. 6) is that class-aware prompt reweighting is essential. The load-bearing step is Eq. 10: w'_{i,c} averages s_{j,i,c} over images whose pseudo-label under prompt i is class c. This is a faithful prompt-class affinity estimate only if those groups are dominated by genuine class-c images. The paper provides no such guarantee; Prop. 1 is insufficient because Hoeffding's inequality only shows that empirical pseudo-label frequencies concentrate on E[1{ŷ=c}], which equals Pr(y=c) only if the base VLM's pseudo-labels are unbiased. The paper's own App. E.2 reports that on EuroSAT and Aircraft the iterative variant fails because initial pseudo-labels are poor, confirming the condition can fail. When it fails, Eq. 10 can up-weight prompts that reinforce the model's errors rather than correct them, so CARPRT's advantage over WPE may not generalize beyond settings where the base model's pseudo-labels are already informative. The empirical results are internally consistent, but they do not establish the 'essential' claim for systematically biased base models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CARPRT, a training-free, black-box method for class-specific prompt reweighting in zero-shot classification with vision-language models. For each prompt and class, CARPRT averages the VLM's image-text similarity scores over images whose pseudo-label (under that prompt) equals the class, then softmax-normalizes these averages to obtain per-class prompt weights; inference then uses the weighted sum of similarities. The authors motivate the method with a probabilistic derivation (Sec. 3), an expressivity argument (Prop. 2), and a proof-of-concept experiment on Flower102. Empirically, they report consistent improvements over MPE, Majority Vote, and WPE across ten fine-grained benchmarks plus ImageNet, using CLIP-ViT-B/16, CLIP-ResNet50, and DeCLIP-ViT-B/32, with additional OOD transfer experiments, ablations (CARPRT-Uniform), and extensions to TTA, prompt tuning, and LLM-generated prompts. The central claim, stated in the abstract and Sec. 6, is that class-aware prompt reweighting is not only beneficial but essential for effective zero-shot prediction.","tokens_in":48849,"tokens_out":2824,"duration_ms":30458,"significance":"If the empirical claims withstand scrutiny, CARPRT is a simple and practical contribution: it requires only forward similarity queries, uses no external data or labels, and appears to improve on the existing class-agnostic WPE baseline across several architectures. The paper's strengths include the breadth of the evaluation (11 datasets, 3 backbones, OOD transfer, multiple downstream applications), the inclusion of an ablation isolating class-specificity (CARPRT-Uniform), and a clear presentation of the algorithm. However, the theoretical framework does not actually support the strongest claim of optimality or 'essential' class-awareness, because the method's core estimator is built from pseudo-labels whose quality is assumed but not guaranteed. The manuscript is honest enough to report failure modes in App. E.2, but those failures are not integrated into the main claims. As it stands, the paper is a solid empirical study with a plausible heuristic, not a theoretically grounded proof of the necessity of class-aware weighting.","major_comments":[{"comment":"The load-bearing step is w'_{i,c} = averaging s_{j,i,c} over images with pseudo-label c under prompt i. This is an unbiased estimate of prompt-class affinity only if those image sets are dominated by genuine class-c images. Prop. 1 does not establish this: Hoeffding's inequality only gives concentration of empirical pseudo-label frequencies around E[1{ŷ=c}], which equals Pr(y=c) only if the base VLM's pseudo-labels are unbiased. The paper's own App. E.2 concedes that on EuroSAT and Aircraft the iterative variant fails because initial pseudo-labels are poor, demonstrating that the required condition can fail. When it fails, Eq. 10 can amplify the model's errors. The manuscript needs either a formal condition under which the estimator is consistent, or a substantially more cautious statement of the 'essential' claim in Sec. 6.","section":"Sec. 4, Eq. 10; Prop. 1; App. E.2"},{"comment":"The main results table (Table 1) reports no error bars or significance tests, and several gains are very small (e.g., +0.21 on Cars with CLIP-ViT-B/16, +0.31 on ImageNet in Table 13). Table 11 reports standard deviations only for some entries and only for WPE and CARPRT, not for MPE. Given that the central claim is consistency of improvement, the authors should report standard deviations over at least 3 seeds for all methods in the main table, or otherwise show that the differences are not noise. This is particularly important because the method is deterministic except for batch ordering and floating-point nondeterminism, but the magnitudes of some gains are within typical run-to-run variation for CLIP evaluation.","section":"Sec. 5.2, Table 1; App. G.3, Table 11"},{"comment":"Proposition 2 shows that the class-specific weight space contains a function not representable by any class-independent weighting, but the proof treats the per-prompt, per-class likelihoods a_{i,c} as independent free variables. In a fixed VLM, these values are not arbitrary; they are constrained by the shared image and text encoders. The expressivity argument therefore does not establish that the extra capacity of class-specific weights is realizable for CLIP-like similarity functions, nor that it translates into lower classification error. This is not fatal to the empirical contribution, but the paper should not claim that Prop. 2 'formally states' that class-aware weighting is necessary for performance.","section":"Sec. 3, Prop. 2; App. I"},{"comment":"There is a mismatch between the probabilistic derivation and the actual algorithm. Eq. 5 equates pseudo-label frequencies with true class probabilities, but Eq. 10 uses pseudo-labels that depend on both the prompt and the image. The text says 'Eq. 10 implements an empirical estimate of the class priors,' but the quantity in Eq. 10 is a conditional average of scores, not a class prior. Moreover, Eq. 11 applies a softmax to scores that are cosine similarities, which is an ad hoc transformation not derived from the EBM formulation in Sec. 3. The authors should either clarify what assumptions make these steps consistent or explicitly present CARPRT as an unsupervised heuristic rather than as a consequence of the probabilistic framework.","section":"Sec. 3, Eq. 5; Sec. 4, Eq. 10-11"}],"minor_comments":[{"comment":"The text says 'Tab. 15 shows...' but the main table is Table 1; the appendix also contains a duplicate Table 15. Please fix the cross-reference and avoid duplicating tables verbatim.","section":"Sec. 5.2"},{"comment":"In the main text Eq. 9 defines s_{j,i,c} as a cosine similarity. In App. D.2, Eq. 9 is redefined as a softmax-normalized probability. These are inconsistent. The algorithm (and black-box claim) only needs similarity scores, so the appendix should not redefine the main equation.","section":"Sec. 4, Eq. 9 vs App. D.2"},{"comment":"Algorithm 1 lists a 'normalization scale λ' as input, but the algorithm body never uses λ. Either remove it or specify its role (e.g., in the softmax of Eq. 11).","section":"Algorithm 1"},{"comment":"The denominator can be zero for a prompt-class pair that no image pseudo-labels to that class. The paper does not specify how such undefined weights are handled. Please state the convention (e.g., set weight to zero or use a small epsilon).","section":"Eq. 10"},{"comment":"The OOD transfer experiment estimates weights on ImageNet and transfers to ImageNet-A/R/Sketch/V2. The claim 'CARPRT consistently surpasses' is supported, but the gains on ImageNet-R are only +0.35 over WPE and smaller than the reported standard deviations in Table 11. Please add error bars or a significance statement for Table 2 as well.","section":"Sec. 5.2; Table 2"},{"comment":"There are duplicate references for Qian et al. (2024a and 2024b) and for Menon & Vondrick (2023a and 2023b), which appear to be the same papers. Consolidate to avoid confusion.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper appears sound: the method is simple, the evaluation is broad, and the ablation against CARPRT-Uniform supports the class-specific mechanism. However, the theoretical claims (Prop. 1 and Prop. 2) do not carry the weight that the abstract and Sec. 6 assign to them, and the pseudo-label dependence is a real correctness risk that the authors themselves document in App. E.2. The lack of error bars in the main table is also a concern for a paper whose central assertion is 'consistently outperforms.' I think a major revision is appropriate: the authors should either prove a consistency condition for Eq. 10 under realistic assumptions (e.g., pseudo-label accuracy above a threshold) or soften the theoretical claims substantially, and they should report variance in the main results. The paper is publishable in principle, but not in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on prompt ensembling for CLIP. The real contribution is a clean extension of WPE: per-class prompt weights estimated from the model's own similarity scores and pseudo-labels, with a formal expressivity argument (Prop. 2) showing class-specific weighting strictly generalizes class-agnostic weighting. The empirical work is honest and broad: consistent gains over WPE across three backbones and eleven datasets, plus ablations showing the class-aware component matters and OOD transfer works. That part is solid, and the method is genuinely new relative to Allingham et al.\n\nThe soft spots are real but addressable. The main table has no error bars, and some gains are small (+0.21 on Cars), so the consistency claim would be stronger with variance or significance. Eq. 10 is undefined for empty prompt-class cells, which will happen with 247 prompts and 100+ classes; the paper never states the fallback. There's also a definitional inconsistency: Eq. 9 defines s_{j,i,c} as raw similarity, but App. D.2 redefines it as a softmax probability; the method as written is ambiguous. Prop. 1 is the weakest theoretical link: Hoeffding gives concentration to the expected pseudo-label frequency, and equating that with the true class distribution assumes the base VLM's pseudo-labels are unbiased. The paper's own EuroSAT/Aircraft results (App. E.2) show that when pseudo-labels are poor, the iterative variant fails — exactly the regime where the 'essential' claim breaks down. And App. G.3 admits the prompt pool is built from the evaluation datasets, which may inflate absolute gains, though it affects baselines equally.\n\nNone of this is fatal. The central empirical claim — CARPRT beats WPE on the evaluated benchmarks — holds up. But the abstract's 'crucial/essential' language is too strong; a fair version would say class-aware reweighting helps when the base model's pseudo-labels are reliable. For practitioners doing zero-shot classification with fixed prompt pools, it's a useful plug-and-play option.\n\nSend to peer review — it's a legitimate contribution that would benefit from tightening these gaps, especially the empty-cell handling and the Prop. 1 wording. I'd cite it.","headline":"Useful extension of WPE with per-class weights and solid empirical gains, but the 'essential' claim rests on a pseudo-label quality assumption the paper doesn't prove.","tokens_in":49560,"tokens_out":1949,"would_cite":true,"duration_ms":20037,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prompt weights in zero-shot ensembling must be class-specific, and that CARPRT's training-free per-class weights improve accuracy across benchmarks.","keywords":["zero-shot classification","vision-language models","prompt ensembling","class-aware reweighting","prompt weighting","black-box inference","pseudo-labels","training-free"],"falsifier":"Run CARPRT and WPE on a dataset where the base model's per-class pseudo-label accuracy is at chance for at least one class (e.g., a class the VLM systematically mislabels), and construct the prompt pool so the correctly matched prompt for that class is obvious to a human. If CARPRT assigns that class a lower weight to the correct prompt than WPE does, or if CARPRT's overall accuracy falls below WPE's, the paper's claim that class-aware reweighting is 'essential' fails on that instance.","tokens_in":48467,"feed_emoji":"🖼️","tokens_out":5285,"duration_ms":47169,"temperature":0.7,"pith_summary":"Prompt ensembling for zero-shot vision-language classification usually gives every prompt one global weight shared across all classes. The paper argues this is wrong: a prompt such as 'an aerial view of' suits 'airport' but not 'apple', so weights should depend on the class. CARPRT computes per-class prompt weights without any training or label access: it derives pseudo-labels from the model's own similarity scores, averages each prompt's scores within each pseudo-class, and softmax-normalizes the averages. Across ten fine-grained benchmarks plus ImageNet and its variants, and across three VLM backbones, CARPRT outperforms the global-weighting baseline WPE by average margins of +1.38 to +2.57 accuracy points, and also improves test-time adaptation, prompt tuning, and LLM-generated prompt pipelines. The paper concludes that class-aware prompt reweighting is not merely beneficial but essential.","feed_headline":"Per-class prompt weights beat global weights for zero-shot vision","feed_subtitle":"Training-free per-class prompt weights beat global weights, human selection, and transfer across domains.","key_machinery":"The score tensor S with entries s_{j,i,c} = sim(image_j, text_{i,c}) over all images, prompts, and classes is the central object. From it CARPRT computes pseudo-labels for each image-prompt pair, then forms the intermediate weight w'_{i,c} as the mean similarity over images whose pseudo-label under prompt i is class c, and finally normalizes with a softmax over prompts at temperature tau. This converts unlabeled test data into per-class prompt weights using only forward similarity queries. The probabilistic backing is an energy-based model view in which class-conditional likelihood is proportional to exp(weighted sum of prompt similarities), plus a proposition showing the class-agnostic weig","core_discovery":"The central claim is that the representable likelihood space of class-specific prompt weighting strictly contains that of class-agnostic weighting, and that unlabeled test images alone suffice to instantiate the class-specific version. CARPRT's weight for prompt i and class c is the average image-text similarity over images that prompt i pseudo-labels as class c, normalized by a softmax over prompts. At inference, a test image's score for class c is the weighted sum of its similarities under the per-class prompt weights. On the paper's benchmarks, this simple recipe beats global WPE by +2.32 (CLIP ViT-B/16), +2.57 (CLIP ResNet50), and +1.38 (DeCLIP ViT-B/32) percentage points on average, and","pith_inferences":["Editorial inference: CARPRT's weight formula can be read as an empirical conditional distribution P(prompt | class) over the unlabeled test set; if that interpretation holds, the weights should be reusable across datasets sharing the same classes, and class-neutral prompts should receive near-uniform weights.","Editorial inference: The method's dependence on pseudo-labels means its advantage should shrink when the base model has systematic blind spots; on datasets like EuroSAT and Aircraft the paper already observes the iterative variant failing, suggesting a diagnostic threshold on pseudo-label accuracy below which class-aware weighting may hurt.","Editorial inference: The per-class weights could be extended to per-cluster or per-image weights, which might capture intra-class variation (e.g., different breeds or viewpoints) that per-class averages smooth away.","Editorial inference: A testable extension is to measure the mutual information between prompts and classes in the score tensor; the paper's mechanism predicts CARPRT's gains should correlate with such a statistic, and would vanish on prompt pools that are class-neutral."],"forward_implications":["With a fixed prompt pool and unlabeled data, zero-shot classification accuracy can be improved by estimating per-class prompt weights, with no labels or gradient updates required.","CARPRT's gains transfer across architectures (ViT, ResNet, DeCLIP) and across distribution shifts: weights estimated once on ImageNet improve ImageNet-A, -R, -Sketch, and -V2 over MPE and WPE.","Class-aware reweighting can outperform human-selected prompt templates, reducing the need for manual prompt engineering per dataset.","CARPRT acts as a plug-in: combining it with test-time adaptation, soft prompt tuning, and LLM-generated prompt augmentation yields further accuracy gains.","Pseudo-label quality, not prompt count, is the main driver: filtering low-confidence pseudo-labels gives only marginal gains, while iterative refinement helps where initial pseudo-labels are good."],"fun_headline_variants":["Per-class prompt weights beat global ones in zero-shot","Training-free per-class prompt reweighting outperforms global","Class-aware prompt reweighting improves zero-shot vision","Zero-shot vision gets per-class prompt weights with no training","Class-specific prompt weighting matches each class better"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The assumption that carries the whole method is that the group of images a prompt pseudo-labels as class c is mostly made of true class-c images, so the averaged similarity is a faithful measure of prompt-class affinity; when the base VLM is biased and pseudo-labels are poor, the reweights can amplify the error.","fun_headline_variants_meta":{"raw":{"variants":["Per-class prompt weights beat global ones in zero-shot","Training-free per-class prompt reweighting outperforms global","Class-aware prompt reweighting improves zero-shot vision","Zero-shot vision gets per-class prompt weights with no training","Class-specific prompt weighting matches each class better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001069,"raw_usage":{"total_tokens":4357,"prompt_tokens":830,"completion_tokens":3527,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":3451}},"tokens_in":574,"tokens_out":3527,"duration_ms":22225,"temperature":1.0,"reasoning_tokens":3451,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:41:47.767304+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CARPRT and WPE on a dataset where the base model's per-class pseudo-label accuracy is at chance for at least one class (e.g., a class the VLM systematically mislabels), and construct the prompt pool so the correctly matched prompt for that class is obvious to a human. If CARPRT assigns that class a lower weight to the correct prompt than WPE does, or if CARPRT's overall accuracy falls below WPE's, the paper's claim that class-aware reweighting is 'essential' fails on that instance.","supporting_citations":[],"review_version":1}