{"id":"807bdb55-8a8f-485e-a64c-88a0b93e478a","arxiv_id":"2505.03414","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Features Matrix module built from frozen CLIP text features of 60 hand-crafted prompts plus a hard-example contrastive loss improves base-to-novel, domain, and cross-dataset generalization when plugged into CoOp, CoCoOp, MaPLe, and PromptSRC.","lead":"This paper adds a plug-in training step to vision-language prompt learning: it pulls text features from 60 frozen hand-written prompts, picks the hardest examples, and uses them in a contrastive loss to stop the model overfitting. The authors report small but consistent gains on generalization benchmarks across four existing prompt-learning methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed gains over PromptSRC/DePT are not shielded from selection-on-test: γ and β are tuned on the same 11-dataset HM average that forms the headline result, and no seed variance is reported.","rationale":"The reader's CONDITIONAL verdict and this stress-test agree on the central risk: the empirical support for FM is not yet robust. However, I would place the most load-bearing weakness one step earlier than the reader's weakest_assumption. The reader focused on whether the hard-example selection mechanism is responsible for the gain; that is an important attribution question, and the proposed random-selection control is a good check. But if the gain itself is inflated by choosing β and γ on the 11-dataset test average, or if the reported numbers sit within single-seed noise, then even a correct mechanism would not rescue the headline SOTA claim. The paper's Sections 5.3 and 5.4 explicitly select hyperparameters by HM on the average of the 11 datasets, the same quantity reported in Table 2a, and Section 4.1 does not describe a validation split or number of seeds. This makes the reported improvements over PromptSRC and DePT vulnerable to selection bias and variance. The reader's rationale does mention no seeds and hyperparameter tuning, so the agreement is partial rather than full. My recommended verdict remains CONDITIONAL: the concern is concrete and testable, but the paper's consistent improvements across three tasks and the moderately large base-to-novel gains mean the claim should be conditionally revised rather than rejected outright.","tokens_in":18131,"tokens_out":6146,"duration_ms":67151,"concrete_test":"Re-run the PromptSRC and PromptSRC+FM base-to-novel benchmark with 5 independent seeds on all 11 datasets, fixing γ=0.1 and β=5 without peeking at the test average, and additionally select β and γ on a held-out validation split (e.g., tune on 8 datasets and evaluate on the remaining 3). Report mean and standard deviation of HM and per-dataset wins. If PromptSRC+FM does not beat PromptSRC and PromptSRC+DePT by more than the seed spread on average, the state-of-the-art claim fails. As a secondary check, include a control with randomly selected β designated and β non-designated features to test whether the hard-example selection mechanism, rather than multi-prompt distillation alone, drives any remaining gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim that FM is a generic plug-in achieving state-of-the-art target-unspecific generalization to hold, the reported improvements must be real rather than artifacts of model selection or run-to-run noise. The paper does not establish this. Section 4.1 gives fixed hyperparameters (γ=0.1, β=5) and no validation protocol, while Sections 5.3 and 5.4 choose γ and β by maximizing HM on the average of the same 11 datasets used in Table 2a. If these hyperparameters were selected on the evaluation metric itself, the headline +1.35 HM over PromptSRC (81.32 vs 79.97) and +0.89 over PromptSRC+DePT (81.32 vs 80.43) are upper-biased estimates. Several per-dataset differences are small relative to typical seed spread in this benchmark—e.g., Table 2(i) Food101 +Ours 91.45 vs +DePT 91.22, Table 2(l) DTD 72.07 vs 71.09—so a few unlucky seeds or a validation-based selection could erase or reverse the claimed advantage. The absence of any multi-seed results or error bars means the central empirical claim is currently unfalsified but not confirmed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Features Matrix (FM), a plug-in regularization module for prompt learning in CLIP. From a frozen text encoder and a set of 60 hand-crafted prompt templates, it forms a matrix of text features; for each training image it selects the lowest-scoring designated and highest-scoring non-designated features as \"unexpected\" features and applies a contrastive loss to align them with the tuning visual features. This loss is added with weight gamma to the cross-entropy loss of a host method, making the module compatible with CoOp, CoCoOp, MaPLe, and PromptSRC. Experiments cover base-to-novel generalization, domain generalization, and cross-dataset generalization on 11 datasets plus ImageNet variants. The paper reports consistent average improvements over the host methods and over the DePT baseline, and claims state-of-the-art performance on target-unspecific tasks.","tokens_in":18290,"tokens_out":4628,"duration_ms":43626,"significance":"The core idea is simple and potentially useful: a frozen-feature regularizer that does not add learnable parameters and can be attached to several representative prompt-learning frameworks. The breadth of the evaluation, across three generalization settings and multiple backbones, is a strength, and the compute-cost table is helpful. However, the headline empirical claims are currently undermined by hyperparameter selection on the evaluation metric and by the absence of any seed-variance information; in addition, the proposed hard-example selection mechanism is not isolated experimentally. If additional controls confirm the gains, the method would be a worthwhile contribution to prompt learning for CLIP. In its present form, the evidence does not support the stated state-of-the-art claims.","major_comments":[{"comment":"Section 4.1 fixes gamma=0.1 and beta=5, but Sections 5.3 (Table 9) and 5.4 (Table 8) select these values by maximizing HM on the average of the same 11 datasets that form the headline results in Table 2. This is selection on the evaluation metric, so the reported gains over PromptSRC (81.32 vs 79.97) and over PromptSRC+DePT (81.32 vs 80.43) are upper-biased. Please provide a validation protocol, such as tuning on a held-out subset of datasets or reporting full sensitivity curves, and explicitly describe how the fixed hyperparameters were chosen.","section":"§4.1, §5.3, §5.4"},{"comment":"No variance information is reported anywhere in the experimental section. All accuracy numbers appear to be from single runs, and several per-dataset differences are small relative to the usual seed-to-seed spread in this 16-shot benchmark; for example, Table 2(i) Food101 gives HM 91.45 for PromptSRC+Ours versus 91.22 for PromptSRC+DePT, and Table 2(l) DTD gives 72.07 versus 71.09. Without means and standard deviations over multiple seeds, the claimed improvements cannot be distinguished from run-to-run noise. Please report at least three seeds with standard deviations, and ideally a paired comparison test.","section":"Tables 2 and 3"},{"comment":"The central mechanism, that selecting the lowest-scoring designated and highest-scoring non-designated features ('unexpected features') provides a better training signal, is not experimentally supported. Section 5.4 (Table 8) varies only the count beta, and Section 5.3 (Table 9) varies gamma; there is no control using randomly selected features or using all features from the matrix. Without such a control, the gains could be due entirely to the multi-prompt contrastive regularizer rather than to the proposed hard-example selection, so the stated mechanism remains unsupported.","section":"§3.2, Eq. (4), §5.4"},{"comment":"The statement that 'our method demonstrates significant improvements on all 11 datasets on HM' is not supported by Table 2 when the comparison includes +DePT. For example, CoOp+Ours is worse than CoOp+DePT on EuroSAT (75.13 vs 75.70), StanfordCars (75.58 vs 75.86), and Flowers102 (82.57 vs 83.08), and MaPLe+Ours is worse than MaPLe+DePT on EuroSAT (83.93 vs 84.36). Please clarify the exact comparison baseline to which 'all 11' refers and avoid the term 'significant' without a statistical test.","section":"§4.2"}],"minor_comments":[{"comment":"The column headers 'Hand' and 'Features' are unclear; please explain whether 'Hand' refers to hand-crafted prompts and what 'Single' versus 'Matrix' means in the 'Features' column.","section":"Table 1"},{"comment":"The notation for non-designated features (the index with a hat) is introduced informally; please define the index sets before Eq. (4) to avoid confusion with the designation used elsewhere.","section":"Eq. (4)"},{"comment":"Several references are duplicated or incomplete: Yao et al. 2023a and 2023b appear to be the same paper, and the PLOT reference cited as Chen et al. in Table 1 lacks full venue and year information. Please unify the bibliography.","section":"References"},{"comment":"The limitations paragraph acknowledges slower training and lower source accuracy on ImageNet, but it does not quantify the slowdown beyond Table 11 or discuss the trade-off in the cross-dataset setting; please add concrete timing or throughput numbers and a brief discussion.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible core idea, but the missing validation protocol, the absence of seed variance, and the lack of a control for the feature-selection mechanism are load-bearing for the central claims. These issues are fixable with additional experiments rather than fundamental flaws, so I recommend major revision rather than rejection. Please ensure the authors are asked to clarify how hyperparameters were selected before the benchmark numbers were produced."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible plug-in regularizer for prompt tuning — a fixed matrix of frozen CLIP text features from many hand-crafted prompts, with a hard-example contrastive loss — but the headline numbers are not yet trustworthy. The two key hyperparameters are selected on the same 11-dataset harmonic-mean average that becomes the headline, and no seed variance is reported.\n\nWhat is actually new: KgCoOp regularizes toward a single frozen prompt feature and PromptSRC regularizes toward the model's own evolving features. FM builds a 60-prompt by K-class features matrix from frozen CLIP and, for each image, contrasts the tuning image feature against low-similarity same-class text features and high-similarity other-class text features. That 'unexpected feature' selection is new relative to the cited work, and the module adds zero learnable parameters. Credit where due: the gains are consistent across three generalization tasks and four frameworks, and the paper openly says the ImageNet source accuracy is lower and training is slower. That is honest reporting.\n\nSoft spots, in proportion. First, and most important: Sections 5.3 and 5.4 choose gamma and beta by maximizing HM on the same 11-dataset average used in Table 2a. That is selection on the evaluation metric, so the +1.35 HM over PromptSRC and +0.89 over PromptSRC+DePT are upper-biased. Second, there are no multi-seed results or error bars anywhere; several per-dataset margins in Table 2 are under one point, which is within typical seed spread on this benchmark. Third, the claimed mechanism is untested. There is no ablation with random feature selection or with all 60 features, so multi-prompt distillation alone could explain the gains. That is an essential control, and it is fixable. Fourth, the text overclaims: the claim of consistent improvement over DePT is false for several per-dataset rows in Table 2 — CoOp+Ours is below CoOp+DePT on EuroSAT, OxfordPets, StanfordCars, and Flowers102. Minor: Eq. (4) writes a single tk and t-hat-k although the selected sets contain multiple features, so the aggregation over the selected features is unspecified, and there is no code release to resolve it.\n\nThe central direction — frozen-feature regularization helps generalization — is plausible and consistent with prior work, and there is nothing circular about the derivation: FM is a fixed regularizer, not fitted to the test outputs.\n\nThis paper is for people working on prompt learning for VLMs who want a cheap plug-in regularizer. It deserves a serious referee, but with major revision: multi-seed numbers, validation-based hyperparameter selection, the random-selection control, and toned-down claims. I would not desk-reject it.","headline":"A plausible plug-in regularizer for prompt tuning, but the headline gains are not yet trustworthy because gamma and beta are selected on the same 11-dataset HM average used as the result and no seed variance is reported.","tokens_in":18933,"tokens_out":3227,"would_cite":false,"duration_ms":30376,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt learning for vision-language models can be rescued from overfitting by a Features Matrix of many hand-crafted prompts, a plug-in module that improves base-to-novel generalization.","keywords":["prompt learning","vision-language models","CLIP","features matrix","base-to-novel generalization","domain generalization","cross-dataset generalization","contrastive regularization"],"falsifier":"Run the same base-to-novel training with a control that draws the same number of features at random from the matrix, or that uses all 60 features with equal weight, and compare harmonic means; if the control matches the low-$\\beta$/top-$\\beta$ result, the hard-example selection is not the mechanism that carries the gain.","tokens_in":17794,"feed_emoji":"🧩","tokens_out":7199,"duration_ms":60849,"temperature":0.7,"pith_summary":"The paper targets a failure mode of prompt learning: fine-tuning CLIP's prompt embeddings for few-shot base classes pushes performance up on those classes but degrades accuracy on novel, out-of-distribution, and cross-dataset targets, because the model forgets general pre-trained knowledge. To counter that, the authors propose Features Matrix (FM), a plug-in module that uses many hand-crafted prompt templates to build a matrix of pre-trained text features and adds a contrastive loss on the hardest (\"unexpected\") matches between those features and the tuning visual features. They claim FM is generic: attached to CoOp, CoCoOp, MaPLe, or PromptSRC it improves base-to-novel, domain, and cross-dataset generalization, and on the PromptSRC backbone it reaches state-of-the-art harmonic mean 81.32, above the easy-to-use DePT at 80.43.","feed_headline":"Plug-in features matrix lifts CLIP generalization 1.35 points","feed_subtitle":"Adding many hand-crafted prompts and contrasting hard text features beats DePT across 11 datasets.","key_machinery":"The Features Matrix (FM) is a fixed table of text features obtained by feeding 60 hand-crafted prompt templates (for example \"a photo of a\", \"a picture of a\", \"a drawing of a\") for every class through the frozen CLIP text encoder. For a given image, features of the true class are \"designated\", others are \"non-designated\"; the method ranks matches between the tuning visual feature and each matrix row, selects the low-$\\beta$ designated and top-$\\beta$ non-designated entries as \"unexpected features\", and applies a contrastive loss $L_{\\mathrm{CL}} = -\\log \\frac{\\exp(\\cos(t_k, v_{\\mathrm{tun}}))}{\\exp(\\cos(t_k, v_{\\mathrm{tun}})) + \\exp(\\cos(t_{\\hat{k}}, v_{\\mathrm{tun}}))}$ with weight $\\gamma$ added to the cross-entropy loss. This selection of hard matches is the component claimed to align tuning image features with overlooked pre-trained general knowledge, while the matrix itself provides the diversity of semantics that a single hand-crafted prompt lacks.","core_discovery":"The central claim is that the drop in target-unspecific performance comes from overfitting that makes the model forget general knowledge, and that a frozen Features Matrix can supply that knowledge back. The matrix is built by passing a set of 60 hand-crafted prompts, across all classes of a dataset, through the frozen CLIP text encoder. During training, for each visual feature the method scores all matrix entries, keeps the lowest-scoring designated-class features and the highest-scoring non-designated features (\"unexpected features\"), and optimizes a contrastive loss that aligns tuning visual features with those unexpected text features, alongside the standard cross-entropy prompt-learning loss. The authors argue that this \"specifically aligning\" of unexpected pre-trained features preserves general knowledge and therefore enhances target-unspecific tasks, and they report consistent gains across 11 datasets on base-to-novel, domain, and cross-dataset benchmarks, including surpassing DePT when added to PromptSRC.","pith_inferences":["If hard-feature selection is the active ingredient, FM is a form of text-side hard-example mining; a natural test would replace \"unexpected features\" with randomly chosen matrix features, holding the loss and count fixed.","The 60 hand-crafted prompts are fixed; extending the matrix with learned or class-specific templates would test whether the diversity of prompts or the selection rule drives the gain.","The selection rule operates only on text features; applying the same low-$\\beta$/top-$\\beta$ contrast to visual features from the frozen image encoder would show whether the mechanism is modality-specific.","The paper's own tables show that FM sometimes lowers base-class accuracy and source ImageNet accuracy and slows training, so a practitioner would likely weigh FM as a trade of target-specific accuracy for target-unspecific generalization."],"forward_implications":["Attached to PromptSRC, FM raises the base-to-novel harmonic mean from 79.97 to 81.32 over 11 datasets, surpassing DePT (80.43).","The module also lifts domain generalization (PromptSRC average improves from 60.65 to 61.92 on ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R) and cross-dataset generalization (from 65.81 to 67.62).","FM works on textual prompting (CoOp, CoCoOp) and multi-modal prompting (MaPLe, PromptSRC) without adding learnable parameters, only extra cosine-similarity computation.","Ablations show a peak at $\\beta = 5$ selected features and $\\gamma = 0.1$ loss weight, with harmonic mean declining at larger values, consistent with the overfitting story."],"supporting_citations":[{"why":"Supplies the frozen CLIP encoders and the 60 hand-crafted prompt templates used to build the Features Matrix.","marker":"Radford et al., 2021"},{"why":"CoOp is the textual prompting baseline that exhibits the novel-class degradation FM is designed to fix.","marker":"Zhou et al., 2022b"},{"why":"CoCoOp is the image-conditional textual prompting baseline that FM also plugs into.","marker":"Zhou et al., 2022a"},{"why":"MaPLe is the multi-modal prompting baseline whose base-to-novel performance FM improves.","marker":"Khattak et al., 2023a"},{"why":"PromptSRC is the main backbone on which FM achieves its best reported harmonic mean.","marker":"Khattak et al., 2023b"},{"why":"DePT is the easy-to-use baseline that FM is directly compared against and surpasses.","marker":"Zhang et al., 2024"},{"why":"KgCoOp motivates the approach by showing single-prompt regularization still falls below hand-crafted CLIP on novel classes.","marker":"Yao et al., 2023a"}],"fun_headline_variants":["Unexpected text features keep CLIP general knowledge intact","Features matrix plugs into prompt learning, boosts 11 benchmarks","CLIP forgets less with matrix of unexpected prompts","Hand-crafted prompts build matrix that rescues CLIP generalization","New matrix module curbs CLIP overfitting for transfer tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's improvement is attributed to selecting the lowest-scoring same-class and highest-scoring other-class text features as \"unexpected features\", but the experiments only vary how many features are selected and the loss weight, never comparing this selection to random or all-feature choices, so the benefit could come from the multi-prompt regularization alone.","fun_headline_variants_meta":{"raw":{"variants":["Unexpected text features keep CLIP general knowledge intact","Features matrix plugs into prompt learning, boosts 11 benchmarks","CLIP forgets less with matrix of unexpected prompts","Hand-crafted prompts build matrix that rescues CLIP generalization","New matrix module curbs CLIP overfitting for transfer tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1724,"prompt_tokens":913,"completion_tokens":811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":740}},"tokens_in":529,"tokens_out":811,"duration_ms":7636,"temperature":1.0,"reasoning_tokens":740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:52:18.950461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same base-to-novel training with a control that draws the same number of features at random from the matrix, or that uses all 60 features with equal weight, and compare harmonic means; if the control matches the low-$\\beta$/top-$\\beta$ result, the hard-example selection is not the mechanism that carries the gain.","supporting_citations":[],"review_version":1}