{"id":"16cdf5b8-7a52-430b-bdb5-dc429c34369d","arxiv_id":"2506.04005","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A ridge regression over CLIP similarity scores against a fixed dictionary of generic prompts provides competitive few-shot image classification when class names are unavailable.","lead":"This paper introduces vocabulary-free few-shot learning for vision-language models, where images of target classes are available but their names are not. It proposes SiM, a simple linear mapping from similarity scores against generic prompts to class labels, and shows it can match zero-shot CLIP without task-specific text prompts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SiM's headline gains look inflated: its default 'generic' prompt set is exactly the ImageNet-1K class-name vocabulary, and several benchmark target classes (Caltech101, Pets, Food101, UCF101) overlap with that vocabulary, so the evaluation is not class-name-free.","rationale":"The reader's weakest_assumption is that generic-prompt similarity scores linearly separate target classes and that semantic coverage explains fine-grained failures. I agree that is the mechanism, but the more precise and testable problem is that the default prompt vocabulary is not independent of the benchmark label sets. The paper itself signals the issue by using Wordnet rather than ImageNet prompts on ImageNet (Table 3); if using ImageNet class names on ImageNet would be circular, using them on Caltech101 and Pets—where the same names occur—is equally circular. The reader's conditional verdict is therefore justified, and I would keep it, with the condition sharpened to require a disjoint-vocabulary evaluation. I am not claiming the authors are deceptive; the closed-form ridge regression (Eq. 3) is correct and the code is available, and the Wordnet/image-prompt experiments are useful checks. But the headline 'vocabulary-free outperforms zero-shot CLIP' cannot be evaluated properly until target names are excluded from the prompt vocabulary. If the proposed test shows the gains vanish, the central claim should be downgraded; if it survives, the paper's contribution is strengthened. I also note that the paper omits variance estimates and the λ value, but those are secondary to the vocabulary-overlap confound.","tokens_in":14711,"tokens_out":11126,"duration_ms":116559,"concrete_test":"Conduct a disjoint-vocabulary rerun of Table 1: for each of the 10 datasets, build the prompt set as the ImageNet-1K prompts minus every prompt whose class name lemmatizes to any target class name (for the Wordnet variant, minus every word whose synset intersects a target class synset), retrain W for 4/8/16 shots with the same seeds and λ, and repeat the 'beats zero-shot CLIP' comparison. If the X/10 count drops to a minority or the per-dataset gains concentrate in datasets with high vocabulary overlap, the central claim is driven by class-name leakage; if SiM still beats zero-shot on the disjoint classes, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that a fixed, target-independent prompt vocabulary yields similarity features that separate target classes. Section 4.2 makes the default prompt set the 1000 ImageNet class names, and Table 1 claims SiM beats zero-shot CLIP on 6/10 datasets (4-shot) and 8/10 datasets (16-shot). However, several benchmark label sets are not disjoint from this 'generic' vocabulary: Caltech101, Pets, Food101, and parts of UCF101/SUN397 have class names that appear verbatim in ImageNet-1K. SiM can then learn weights that route the one-hot label through the matching generic prompt, effectively recovering target class names from the prompt list. The failure pattern is consistent: Aircraft, Cars, and Flowers—the datasets with the least ImageNet overlap—are where SiM degrades most (Table 1). The paper itself treats the same overlap as circular for ImageNet: Table 3 switches to Wordnet prompts and never uses ImageNet class names on ImageNet. No disjoint-vocabulary holdout is reported for the other datasets, so the claim that SiM is vocabulary-free and outperforms zero-shot CLIP is not established for target classes genuinely absent from the prompt vocabulary. The least-squares derivation (Eqs. 3 and 4) is not the issue; the evaluation protocol is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"vocabulary-free few-shot learning\" for vision-language models, a setting in which target class names are unavailable but a few labeled images per class are provided. The proposed method, Similarity Mapping (SiM), computes image similarities against a fixed set of generic prompts (textual or visual), learns a linear mapping W from these similarity scores to one-hot class labels via ridge regression (Eqs. 3-4), and classifies test images by applying W to their similarity scores. The paper evaluates SiM on 10 datasets with two CLIP backbones, comparing it against label-mapping baselines, a centroids baseline, and standard name-based few-shot methods, and reports that SiM outperforms zero-shot CLIP on several datasets while requiring no target class names.","tokens_in":15032,"tokens_out":15394,"duration_ms":147705,"significance":"If the results hold, SiM is a simple, efficient, and practical baseline for a genuinely useful setting, and the paper's framing of vocabulary-free few-shot learning is a reasonable addition to the VLM adaptation literature. The method is easy to understand, the least-squares derivation is correct, and the paper includes useful ablations over prompt sources (ImageNet text, Wordnet text, ImageNet images) plus an ImageNet experiment with Wordnet prompts. The code release and the interpretability analysis are additional strengths. However, the evaluation protocol has a potential vocabulary-overlap problem that affects the central \"vocabulary-free\" claim, and the experimental reporting omits variability measures and the value of the regularization hyperparameter.","major_comments":[{"comment":"The main results use the ImageNet-1K class names as the \"generic\" prompt vocabulary, and several target datasets (Caltech101, Pets, Food101, UCF101, SUN397) share class names with that vocabulary. SiM can therefore learn weights that route the one-hot label through a prompt that is effectively the target class name, so the claim that the method operates without access to class names is not yet established for those datasets. The paper itself treats this as a problem for ImageNet by switching to Wordnet prompts in Table 3, but Wordnet is also not guaranteed to be disjoint from the ImageNet class names (many ImageNet classes are Wordnet synsets), and no overlap statistics are reported for any dataset. Please report, for each dataset, the fraction of target class names that appear in the generic prompt vocabulary, and add a disjoint-vocabulary evaluation, e.g., removing all prompts that match a target class name, for every dataset and for all three prompt types. Table 2's Wordnet results should be analyzed with the same overlap statistics.","section":"Section 4.2, Tables 1-3"},{"comment":"The headline claim that SiM outperforms zero-shot CLIP on 6/10 or 8/10 datasets is based on averages over 3 seeds with no standard deviations or confidence intervals, and several of the claimed wins are very small (e.g., SUN 62.7 vs 62.6 and Caltech 93.2 vs 92.9 for 4-shot ViT-B/16). Without variability estimates, these aggregate claims are not statistically supported. Please report per-seed results or standard deviations for the main comparisons, and either run more seeds or use a paired test when claiming a win over zero-shot CLIP.","section":"Section 5, Table 1"},{"comment":"The regularization coefficient lambda in the ridge regression objective is never specified in the paper, despite being a free parameter of the method. Since W = (L^T L + lambda I)^-1 L^T Y depends directly on lambda, the reported results are not reproducible from the text. Please state the value(s) of lambda used for all experiments, describe how lambda was selected (e.g., fixed default, validation split, per-dataset tuning), and provide an ablation showing sensitivity to lambda across a reasonable range.","section":"Section 3.2, Eq. (3)"}],"minor_comments":[{"comment":"The paragraph describing Eq. (3) as an \"unsupervised clustering objective\" is misleading: Y is a fixed one-hot label matrix, so the optimization is supervised linear classification on the similarity features. Please rephrase to avoid confusion with unsupervised clustering.","section":"Section 3.2"},{"comment":"The Wordnet vocabulary selection is underspecified: the phrase \"words in Wordnet which are related to at least one of the words in [...]\" does not define which Wordnet relations are used (e.g., hyponymy, hypernymy, meronymy) or how the seed words are expanded. Since the resulting K=16,452 depends on this choice, please provide the exact selection procedure.","section":"Section 4.2"},{"comment":"The sentence \"Our approach outperforms zero-shot CLIP on 6 out of 10 datasets with 4 shots per class and on 8 out of 10 datasets with 16 shots per class\" is not tied to a backbone. For ViT-B/16, Table 1a shows 7 wins at 4 shots, not 6. Please state which backbone the claim refers to, or report the counts for both backbones separately.","section":"Section 5"},{"comment":"The experimental setting lists 11 datasets, but Table 1 reports results for 10 datasets and ImageNet appears separately in Table 3. Please clarify this organization explicitly, including the reason that ImageNet is excluded from the main table when ImageNet class names are used as prompts.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a simple and potentially useful baseline, and the core derivation is sound. The main scientific risk is the overlap between the generic prompt vocabulary and the target class names; I recommend requiring the disjoint-vocabulary analysis and the lambda specification before acceptance. The lack of error bars is also important because several headline wins are within 0.5 accuracy points. I do not see issues of citation or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces vocabulary-free few-shot learning for VLMs, which is a genuinely useful problem framing: you have a few labeled images per class but no class names, and you want to classify new instances. The proposed baseline, SiM, is a ridge regression from similarity scores against a fixed set of prompts to one-hot labels. It is simple, fast, and the least-squares derivation is correct. The paper also does a decent job comparing against existing label-mapping methods and shows interpretability examples that are fun and plausible.\n\nThe main soft spot is the evaluation. The default 'generic' prompt set is the 1000 ImageNet class names. Several of the benchmark datasets have class names that appear verbatim in ImageNet (Caltech101, Pets, Food101, parts of UCF101 and SUN397). That means the method can directly route a target class through its name in the prompt list, so the setting is not actually vocabulary-free for those datasets. The failure pattern—Aircraft, Cars, Flowers, which have the least overlap—bears this out. The authors do switch to a Wordnet prompt set for ImageNet, but not for the other datasets, and the Wordnet set also contains many overlapping words. I don't think this kills the idea, but it means the claim that SiM beats zero-shot CLIP on 6/10 or 8/10 datasets is not established for classes genuinely absent from the prompt vocabulary.\n\nThere are also smaller issues: the regularization lambda is never specified or ablated, and no error bars are reported for the 3-seed averages. These are easy fixes.\n\nOverall, the paper is worth engaging with. The setting is new and the baseline is a useful reference point, but the evaluation needs a genuinely disjoint prompt vocabulary (or a held-out set of classes) before the empirical claims hold water. I'd send it to review with a request for major revisions.","headline":"A useful new problem setting and a clean baseline, but the headline results rest on a prompt vocabulary that overlaps with the target classes, so the 'vocabulary-free' claim is only partially supported.","tokens_in":15516,"tokens_out":2713,"would_cite":false,"duration_ms":25715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A linear map from generic-prompt similarities to class labels lets vision-language models do few-shot classification without knowing a single class name, beating zero-shot CLIP on most datasets.","keywords":["vocabulary-free few-shot learning","vision-language models","CLIP","similarity mapping","linear mapping","label mapping","ridge regression","interpretability"],"falsifier":"Delete from the generic vocabulary every prompt semantically related to a target dataset (for example, remove all animal terms before testing on Pets) and measure the accuracy drop: the paper's mechanism predicts a large drop, and its size quantifies how much of SiM's success is genuine semantic coverage rather than incidental score structure. On the opposite side, a task whose classes are constructed to be semantically disjoint from the vocabulary should keep SiM near chance at any shot count, so observed high accuracy there would refute the coverage assumption.","tokens_in":14537,"feed_emoji":"🏷️","tokens_out":12698,"duration_ms":105098,"temperature":0.7,"pith_summary":"Vision-language models like CLIP usually need the names of the classes they classify, phrased as prompts such as \"a photo of a {class}\". This paper removes that requirement: it introduces vocabulary-free few-shot learning, in which the only given information is a handful of labeled images per unknown class. The proposed method, Similarity Mapping (SiM), scores each image against a fixed list of generic prompts (the 1000 ImageNet class names, a 16,452-word Wordnet subset, or even one image per ImageNet class) and learns a linear map from those scores to class labels by ridge regression. SiM beats zero-shot CLIP, which gets to use the true class names, on six of ten datasets with 4 shots per class and eight of ten with 16 shots, while the mapping itself trains in under a second.","feed_headline":"One linear map classifies images without knowing their names","feed_subtitle":"Even without class names, four labeled images per class beat zero-shot CLIP on most datasets.","key_machinery":"The load-bearing object is the linear mapping $W = (L^\\top L + \\lambda I_K)^{-1}L^\\top Y$, the closed-form ridge-regression solution that projects the space of image–prompt similarity scores onto the one-hot label space. The rows of $L$ are the similarity vectors of the few-shot images against a fixed generic vocabulary (textual prompts from ImageNet or Wordnet class lists, or visual prompts made of one image per ImageNet class), so nothing about the target classes enters in advance. The mapping does the entire job: it converts whatever linear structure CLIP's similarity space already carries about the unknown classes into a classifier, and the weights $w_{k,c}$ additionally rank which generic concepts each target class responds to, providing the paper's interpretability results.","core_discovery":"The paper's central claim is that the similarity scores of images against a generic, task-agnostic prompt vocabulary contain enough class structure that a plain linear readout yields a working few-shot classifier, so explicit class names are unnecessary. Concretely, SiM builds the $N\\times K$ matrix $L$ of similarities between the $N$ labeled shots and $K$ generic prompts, one-hot encodes the shot labels in $Y$, and solves the Tikhonov-regularized least-squares problem $$W = \\arg\\min_W \\|Y - LW\\|$_F^{2}$ + \\$\\lambda$\\|W\\|$_F^{2}$,$$ whose closed-form solution is $W = (L^\\top L + \\lambda I_K)^{-1}L^\\top Y$. A test image is classified by passing its similarity vector through $W$. The authors report that this baseline outperforms zero-shot CLIP on six of ten datasets at 4 shots and eight of ten at 16 shots, approaches name-based few-shot methods such as prompt tuning and adapters without matching them, and that the columns of $W$ double as a semantic description linking each unknown class to the most relevant generic concepts.","pith_inferences":["If the linear-readout claim holds, the choice of generic vocabulary is a tunable inductive bias: selecting or weighting prompts to maximize the separability of the few-shot classes should improve SiM, a variant the paper does not test.","The interpretability failures on fine-grained classes suggest the mapping can succeed via score structure that is not semantic; ablating semantically related prompts from the vocabulary would reveal how much of the method's accuracy is genuine concept linking.","Because the method consumes only similarity scores, it should transfer to black-box API access to contrastive multimodal models, and the same score-mapping idea could apply beyond classification, e.g., to retrieval or clustering of unnamed image collections."],"forward_implications":["Few-shot classification of unknown classes is feasible as a black-box operation: SiM needs only the similarity scores, not the underlying embeddings, and the mapping trains in under a second (0.8 s even for 16,000 shots against 16,452 prompts).","The prompt vocabulary does not need to be curated for the task: generic lists such as ImageNet class names or Wordnet words work across datasets, and image-based prompts close most of the gap to textual prompts as shots increase.","The learned mapping doubles as an interpretability device, linking unnamed classes to known concepts (gerenuk to impala and gazelle; rope climbing to abseil), which could support automatic naming of discovered classes.","Performance stays below name-based few-shot methods and below zero-shot on some datasets, particularly fine-grained tasks like Aircraft, Cars, and Flowers, so the vocabulary-free setting still has a real cost.","Using a stronger backbone (ViT-L/14) markedly shrinks the gap to name-based methods, and SiM with Wordnet prompts reaches 72.8% on ImageNet itself at 32 shots versus 75.9% for zero-shot with true class names."],"supporting_citations":[{"why":"supplies CLIP, the vision-language model whose image–text similarity scores are the substrate of the method.","marker":"[26]"},{"why":"provides the 1000 class names used to generate the main set of generic textual prompts.","marker":"[10]"},{"why":"provides the Wordnet lexical database from which the 16,452-word generic prompt vocabulary is drawn.","marker":"[23]"},{"why":"introduces frequency label mapping, the one-to-one mapping baseline that SiM extends and outperforms.","marker":"[33]"},{"why":"introduces Bayesian label mapping, the many-to-many mapping baseline that SiM is compared against.","marker":"[4]"},{"why":"defines the name-based prompt-tuning few-shot setting that vocabulary-free learning removes class names from.","marker":"[40]"},{"why":"a name-based adapter few-shot method that serves as an upper-bound reference in the comparisons.","marker":"[38]"},{"why":"a name-based low-rank adaptation method that achieves the strongest upper-bound results SiM approaches.","marker":"[37]"}],"fun_headline_variants":["One linear map learns classes without any class names","Vocabulary-free few-shot: linear map from generic prompts","Quick linear readout turns generic prompts into classes","No class names needed: linear map from similarity scores","One-second training: classify images with no vocabulary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the fixed generic prompt list produces similarity scores in which every unknown class can be separated from the others by a simple weighted sum, so classes that fall outside the semantic reach of the prompt vocabulary give the mapping nothing useful to work with and accuracy collapses.","fun_headline_variants_meta":{"raw":{"variants":["One linear map learns classes without any class names","Vocabulary-free few-shot: linear map from generic prompts","Quick linear readout turns generic prompts into classes","No class names needed: linear map from similarity scores","One-second training: classify images with no vocabulary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2372,"prompt_tokens":989,"completion_tokens":1383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1324}},"tokens_in":605,"tokens_out":1383,"duration_ms":11673,"temperature":1.0,"reasoning_tokens":1324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:48:52.869482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Delete from the generic vocabulary every prompt semantically related to a target dataset (for example, remove all animal terms before testing on Pets) and measure the accuracy drop: the paper's mechanism predicts a large drop, and its size quantifies how much of SiM's success is genuine semantic coverage rather than incidental score structure. On the opposite side, a task whose classes are constructed to be semantically disjoint from the vocabulary should keep SiM near chance at any shot count, so observed high accuracy there would refute the coverage assumption.","supporting_citations":[{"cited_title":"Imagenet: A large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"provides the 1000 class names used to generate the main set of generic textual prompts."},{"cited_title":"Wordnet: a lexical database for english","cited_arxiv_id":null,"evidence_quote":"provides the Wordnet lexical database from which the 16,452-word generic prompt vocabulary is drawn."},{"cited_title":"Transfer learning without knowing: Reprogramming black-box ma- chine learning models with scarce data and limited resources","cited_arxiv_id":null,"evidence_quote":"introduces frequency label mapping, the one-to-one mapping baseline that SiM extends and outperforms."},{"cited_title":"Bayesian-guided label mapping for visual reprogram- ming","cited_arxiv_id":null,"evidence_quote":"introduces Bayesian label mapping, the many-to-many mapping baseline that SiM is compared against."},{"cited_title":"Tip- adapter: Training-free adaption of clip for few-shot classifi- cation","cited_arxiv_id":null,"evidence_quote":"a name-based adapter few-shot method that serves as an upper-bound reference in the comparisons."},{"cited_title":"Low-rank few-shot adaptation of vision-language models","cited_arxiv_id":null,"evidence_quote":"a name-based low-rank adaptation method that achieves the strongest upper-bound results SiM approaches."}],"review_version":1}