{"id":"8d76d7a3-186c-4914-830d-b2bd33a12028","arxiv_id":"2412.11917","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-generated descriptions improve VLM classification only when selected to discriminate among ambiguous classes, not when simply ensembled.","lead":"This paper tests whether CLIP image classification is genuinely helped by LLM-written descriptions or just by averaging over extra text prompts. It introduces a classname-free evaluation setup and a selection algorithm that picks discriminative descriptions from a pool using CLIP embedding feedback, reporting accuracy gains on seven benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own 'Random Ours' control nearly matches LLM descriptions under the same selection, so gains may be due to supervised selection of arbitrary text embeddings rather than LLM semantics; missing wcls=0 random-pool test leaves the central claim unresolved.","rationale":"The reader's weakest assumption identifies the use of labeled reference images and the absence of few-shot baselines. I agree partially, but the more direct and more load-bearing control is a non-semantic description pool under the same selection algorithm. The paper already includes 'Random Ours' in Table 1, and its proximity to 'DCLIP Ours' is an internal warning that the text does not discuss. Because Algorithm 1 uses the labeled training set to choose discriminative text embeddings, random text tokens can become class-specific codes; any improvement could then reflect the selection algorithm's use of labels and VLM geometry, not LLM description semantics. The central conclusion is precisely the semantic attribution, so this is load-bearing. A random-pool wcls=0 experiment would settle it; the few-shot baseline the reader wants is useful for comparing against supervised methods but does not isolate semantics. The appendix's maxing evaluation also compares selected LLM-pool descriptions against unselected LLM assignments, so it does not distinguish semantics from selection either. The paper's proposed evaluation protocol is still valuable, but the reported claim should be conditional on this control.","tokens_in":21171,"tokens_out":10165,"duration_ms":95883,"concrete_test":"Run Algorithm 1 exactly as in Table 2 (wcls=0, k=3, n from Appendix A.9, same S construction) with P replaced by a random string pool of the same size as the DCLIP pool, e.g. WaffleClip-style random words or random alphanumeric strings. Report accuracy on all seven datasets in a new column next to Table 2's 'Ours'. If this random-pool column is within roughly 2–3 pp of the LLM-pool column on most datasets, the central claim is falsified and the paper should be reframed as a training-free selection method whose gains do not require LLM semantics; if it collapses to near the LLM/random-assignment rows, the semantic attribution survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central attribution claim—that gains come from LLM description semantics—is not isolated from the selection procedure. Algorithm 1 uses labeled reference images to build the lookup similarity matrix S (Algorithm 1, line 1) and, per test image, selects descriptions that separate the local candidate classes in CLIP space. Such a supervised selection can pick any text tokens that happen to be discriminative, including random strings: WaffleClip already shows random words carry useful embedding directions. The paper's own Table 1 contains a 'Random Ours' condition (random description pool, same selection) and it nearly matches 'DCLIP Ours' in the classname-free setup on 5 of 7 datasets (ImageNet 62.18 vs 62.57, ImageNetV2 55.22 vs 55.48, CUB 52.31 vs 53.80, DTD 44.73 vs 47.23, Flowers 66.12 vs 66.37; Places 40.44 vs 42.64; EuroSAT 40.82 vs 49.89). If this pattern persists under the paper's strongest evidence—Table 2's wcls=0, where no classname is used—then the accuracy is attributable to supervised selection over arbitrary text embeddings, not to LLM description semantics. The missing control is a random-pool version of Table 2; a few-shot baseline alone would not resolve this, because a few-shot linear probe could also match without using LLM semantics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether VLM classification gains from LLM-generated descriptions reflect genuine description semantics or a noisy test-time ensembling effect. To separate these, the authors propose a classname-free evaluation setup in which, for each class, only one text prompt contains the class name and all other descriptions are classname-free; they also propose a selection algorithm (Algorithm 1) that, for each test image, picks m classname-free descriptions from a global pool that best separate the k candidate classes in CLIP embedding space, using a lookup similarity matrix S built from labeled reference images. Experiments on seven datasets with CLIP ViT-B/32 and ViT-L/14, plus additional ALIGN results, show accuracy gains for the selected descriptions over LLM assignments and random assignments, and the authors conclude that VLM classification benefits from LLM description semantics when descriptions are correctly selected.","tokens_in":21406,"tokens_out":6639,"duration_ms":55111,"significance":"The proposed classname-free protocol is a useful evaluation lens: it prevents the trivially ensembling-based gains demonstrated by Roth et al. and gives a concrete test for whether a description assignment is genuinely discriminative in the absence of the class name. The selection algorithm is simple, reproducible (code is released), and the paper includes welcome robustness checks across two CLIP backbones, two LLMs for pool construction, and a second VLM (ALIGN). If the central claim holds after the missing controls are added, the paper would be a solid contribution to explainable VLM classification. The main uncertainty is not whether the proposed selection improves accuracy (it does) but whether the improvement can be attributed to LLM description semantics rather than to the supervised selection of arbitrary text embeddings.","major_comments":[{"comment":"The paper's headline claim—that the gains come from LLM description semantics—is not isolated from the selection procedure, because the control that would separate the two factors is missing. Table 1 reports a 'Random Ours' condition (random description pool with the same selection heuristic) and it nearly matches 'DCLIP Ours' in the classname-free setup on 5 of 7 datasets (e.g., ImageNet 62.18 vs 62.57; Flowers 66.12 vs 66.37). Table 2, which is the strongest evidence for the semantic claim because it uses wcls=0, compares 'Ours' only against LLM assignments and random assignments; it does not include a random-pool version of 'Ours' under wcls=0. If the random-pool selection matches 'Ours' there as it does in Table 1, the accuracy is attributable to supervised selection of arbitrary text embeddings, not to LLM semantics. Please add this control (random P with Algorithm 1 and wcls=0) for all seven datasets, and report the gap between random-pool and LLM-pool selections.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The method is described as training-free, but it uses labeled reference images from the standard training split to build S (Algorithm 1, line 1, and Section 4.1). This is a form of supervised model selection, and the zero-shot baselines (DCLIP, WaffleClip, FuDD) do not use such labels, making the comparison asymmetric. To isolate the contribution of description selection, add a few-shot baseline that uses the same n reference images per class and the same CLIP features (for example, nearest-class-mean or a linear probe). Note that such a baseline is not a substitute for the random-pool control of the previous comment: the few-shot baseline cannot distinguish LLM semantics from selection of arbitrary embeddings.","section":"Section 4.1, Algorithm 1"},{"comment":"For EuroSAT, the paper states that no train split was provided and that n=1000 was chosen arbitrarily. If the 1000 images per class used to build S include images from the evaluation set, the large EuroSAT gain (e.g., 49.89 vs 43.31 in Table 1) would be contaminated by leakage of test labels into the selection step. Please specify exactly where the EuroSAT reference images come from (e.g., a random subset of the original dataset disjoint from the test images, or the standard split used by prior work), and rerun the EuroSAT experiments with a properly disjoint selection set if necessary.","section":"Appendix A.9, Table 11"},{"comment":"Reporting 'the best-performing wcls of the respective assignment' means that a free parameter is tuned on the test set for each assignment. This can bias the reported peak accuracy and makes the comparison between assignments uncontrolled. Please report accuracy as a function of wcls for all compared assignments (as already done for the proposed method in Figure 3), or use a validation split to choose wcls for every method, and report the chosen values in the tables.","section":"Table 1 caption; Section 4.2"}],"minor_comments":[{"comment":"The DTD value for 'Contrastive GPT3.5 w/ Our Selection' is 80.78, which is inconsistent with the DTD values around 48 in Tables 1, 3, and 6; this appears to be a typo and should be corrected.","section":"Appendix A.10, Table 12"},{"comment":"The term 'Random Ours' is not defined in the main text; please state explicitly whether it is a random description pool with Algorithm 1 selection or a random assignment of existing descriptions to classes, since the distinction is crucial for interpreting the control.","section":"Table 1"},{"comment":"The sentence 'Weights of the classname-free descriptions are normalized to one' is ambiguous: the classname-free weights sum to one but the classname weight wcls is added on top, so the final weighting is not a convex combination. Please clarify the intended normalization.","section":"Section 3.2, Equation (5)"},{"comment":"For Flowers102, DCLIP descriptions were generated with GPT-3.5 using prompts from their codebase; please state how many descriptions per class were generated and how this differs from the original DCLIP distribution, since Table 3 compares against DCLIP on Flowers102.","section":"Appendix A.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to generate discussion at the venue because of the 'training-free' claim and the missing random-pool control. I would ask the authors to tone down the attribution to LLM semantics unless the control shows a clear gap. Also, the EuroSAT leakage question should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best quick take: the classname-free evaluation setup is the paper's real contribution, and the selection method is a solid empirical addition. But the headline conclusion—that VLM classification benefits from LLM description semantics once descriptions are correctly selected—is not yet isolated, because the paper never runs its own selection algorithm on a random description pool at wcls=0.\n\nWhat's genuinely new: existing description-based methods all keep the classname in every prompt, so gains can be written off as ensembling. This paper removes that confound and shows raw LLM assignments barely beat random text, while their selected descriptions give real gains. That is a useful result on its own. The selection procedure is simple, fast, and tested across backbones (ViT-B/32, ViT-L/14), pools (DCLIP, contrastive), and even a different VLM (ALIGN). They also report a maxing-aggregation ablation that rules out ensembling in a second way; that was the right experiment to run. Code is public.\n\nWhere it wobbles: (1) The attribution to LLM semantics is underdetermined. Table 1 already shows 'Random Ours' (random strings, same selection) lands within a couple points of 'DCLIP Ours' on five of seven datasets. The decisive control is the wcls=0 version of Table 2 with a random pool and the same Algorithm 1. If random strings also survive at wcls=0, then the gains come from the selection procedure plus labeled references, not from description semantics. That control is absent. (2) The method uses labeled reference images—up to 1000 per class for EuroSAT—but compares only against zero-shot baselines. A few-shot linear probe or Tip-Adapter baseline is needed to isolate what the labeled data contributes. (3) wcls is chosen on the test set ('best-performing wcls'), which is test-set tuning and will inflate numbers; report it on a held-out split or fix it. (4) 'Across seven datasets' is too broad for the conventional setup: Table 3 shows no gain over CLIP on ImageNet and ImageNetV2 with the DCLIP pool.\n\nWho it's for: anyone working on prompt ensembling or interpretable VLM classification. The evaluation protocol is worth taking up; the method is a reasonable baseline. The missing controls are concrete and easy to run, so this deserves a serious referee rather than a desk reject.","headline":"Useful new evaluation protocol, but the central claim is underdetermined by the missing random-pool control.","tokens_in":22000,"tokens_out":4299,"would_cite":true,"duration_ms":36367,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM description semantics boost VLM classification once selected, not raw.","keywords":["vision-language models","CLIP","LLM descriptions","classname-free evaluation","ensembling effect","description selection","zero-shot classification","explainability"],"falsifier":"Run the same classname-free evaluation with a few-shot baseline that uses the same labeled reference images to build class prototypes (e.g., averaged image embeddings) and classifies by nearest prototype, without any LLM descriptions or selection; if that baseline matches or beats the proposed method's accuracy, then the labeled reference data, not the description selection, carries the performance.","tokens_in":20928,"feed_emoji":"🧠","tokens_out":3896,"duration_ms":31224,"temperature":0.7,"pith_summary":"The paper asks whether the accuracy gains that vision-language models (VLMs) get from LLM-generated descriptions come from genuine semantics or from a noisy ensembling effect. It argues that previous evaluation setups cannot separate the two, because every prompt contains the class name and multiple modified prompts act as test-time augmentation. The authors introduce a classname-free setup in which only one prompt per class contains the class name, and all other prompts are descriptions without it. Under this setup, they show that raw LLM assignments barely beat random text, while descriptions selected by their algorithm produce consistent gains across seven datasets. Their conclusion is that VLM classification does benefit from LLM description semantics, but only when the descriptions are correctly selected.","feed_headline":"LLM description semantics boost VLM classification once selected","feed_subtitle":"A classname-free test shows raw LLM prompts barely beat random text, while 5 selected descriptions lift accuracy on 7 datasets.","key_machinery":"The load-bearing mechanism is the lookup similarity matrix $S$, built by averaging image-description similarities between $n$ labeled reference images per class and a global pool of classname-free LLM descriptions. For each test image, the method first retrieves the top-$k$ candidate classes using only the class-name prompt, forming a local label neighborhood $A(x_i)$. Within that neighborhood, it keeps only descriptions whose averaged similarity to the target class exceeds their averaged similarity to every other candidate class (the positive subset $S^+$), and then selects the top-$m$ descriptions by mean positive difference. These selected descriptions are used as classname-free prompts alongside a single class-name prompt, with a relative weight $w_{\\text{cls}}$ controlling how much the class name counts.","core_discovery":"The paper's central claim is that LLM-generated descriptions carry genuine semantic signal that improves VLM classification, but this signal is invisible under standard ensembling evaluation because the class name, repeated in every prompt, dominates the average. To expose it, the authors propose a classname-free evaluation: exactly one text prompt per class contains the class name, and the remaining prompts are descriptions without it. In this setup, raw LLM assignments perform barely better than random text, while descriptions selected by their method, which uses feedback from the VLM embedding space to pick descriptions that differentiate each class from its local competitors, improve accuracy on all seven datasets tested. The paper concludes that the benefit of LLM semantics is real but conditional on selection: the descriptions must be discriminative within the local neighborhood of ambiguous classes.","pith_inferences":["A natural next step is to feed the selection feedback back into the LLM in an agentic loop, letting the LLM refine the description pool; the paper notes this as future work.","The use of labeled reference images suggests that a fair comparison against true few-shot methods is needed to isolate the source of gain, and this comparison is not made in the paper.","The Pareto-like distribution of distinctiveness scores implies that even smaller description budgets ($m < 5$) might suffice on some datasets, and that the selection criterion could serve as a general pruning tool for description pools."],"forward_implications":["If the claim is correct, evaluation protocols for description-based VLM classification should report classname-free accuracy, not just ensembled accuracy.","Raw LLM assignments are not a reliable way to transfer semantic knowledge; selection based on VLM feedback is needed.","A handful of well-chosen descriptions (e.g., 5) can outperform methods using hundreds or thousands of prompts per class.","The method provides a way to audit whether a VLM's decisions are explainable through distinctive verbal descriptions."],"supporting_citations":[{"why":"Supplies the CLIP model and embedding space used as the base VLM throughout the paper.","marker":"Radford et al. 2021"},{"why":"DCLIP provides the original LLM-assigned description pool and the standard ensembling baseline that the classname-free setup is designed to interrogate.","marker":"Menon and Vondrick 2023"},{"why":"WaffleClip introduced the noise-ensembling critique that motivates the classname-free evaluation scenario.","marker":"Roth et al. 2023"},{"why":"FuDD is a contrastive prompting baseline using very large description sets that the paper compares against.","marker":"Esfandiarpoor and Bach 2024"},{"why":"CuPL is another LLM-description augmentation method that motivates the use of description pools.","marker":"Pratt et al. 2023"},{"why":"LaBo is a bottleneck-based approach with classname-free characteristics that the paper distinguishes from its own method.","marker":"Yang et al. 2023"}],"fun_headline_variants":["LLM descriptions only help VLM classification when selected","Selecting discriminative LLM descriptions boosts VLM accuracy","Raw LLM prompts fail, selected ones boost VLM","Classname-free test reveals true value of LLM descriptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method needs a small set of labeled reference images from each dataset's training split to build the similarity table, so the measured gains could in principle come from that labeled data rather than from the description-selection step, since the paper compares only against zero-shot baselines.","fun_headline_variants_meta":{"raw":{"variants":["LLM descriptions only help VLM classification when selected","Selecting discriminative LLM descriptions boosts VLM accuracy","Raw LLM prompts fail, selected ones boost VLM","Classname-free test reveals true value of LLM descriptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2478,"prompt_tokens":907,"completion_tokens":1571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1505}},"tokens_in":523,"tokens_out":1571,"duration_ms":10978,"temperature":1.0,"reasoning_tokens":1505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:26:58.796567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same classname-free evaluation with a few-shot baseline that uses the same labeled reference images to build class prototypes (e.g., averaged image embeddings) and classifies by nearest prototype, without any LLM descriptions or selection; if that baseline matches or beats the proposed method's accuracy, then the labeled reference data, not the description selection, carries the performance.","supporting_citations":[{"cited_title":"M.; Koepke, A.; Vinyals, O.; Schmid, C.; and Akata, Z","cited_arxiv_id":null,"evidence_quote":"WaffleClip introduced the noise-ensembling critique that motivates the classname-free evaluation scenario."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FuDD is a contrastive prompting baseline using very large description sets that the paper compares against."}],"review_version":1}