{"id":"efed0785-f9f3-4e60-9947-53a8e4ffb602","arxiv_id":"2412.13947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Removing class names from LLM-generated descriptions collapses CLIP's zero-shot accuracy, and fine-tuning on synthetic attribute descriptions with a multi-resolution vision layer recovers much of that performance.","lead":"This paper shows that CLIP's strong zero-shot classification collapses when object class names are removed from descriptions, and that fine-tuning on synthetic attribute descriptions generated by LLMs recovers much of the performance. It also introduces a multi-resolution CLIP variant that improves fine-grained part-attribute recognition on the PACO benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PACO evidence may not be zero-shot: ImageNet21k fine-tuning classes likely overlap PACO object categories, so the 16-point attribute gain could reflect category-level memorization rather than generalizable part-attribute understanding.","rationale":"The reader's conditional verdict centers on prompt-style representativeness and name-removal leakage. Those are real concerns, but they leave Table 4 as an independent anchor. If Table 4 is contaminated by category overlap, the paper has no clean evidence that attribute knowledge transfers to unseen classes: the six fine-grained benchmarks are the only remaining support, and they share prompt styles with the synthetic training data. This is not an accusation of misconduct; it is a missing control. A quick overlap computation and a restricted PACO evaluation would settle it. If the check shows no overlap, the concern does not land and the paper's conditional acceptance stands. If it shows large overlap, the PACO gain must be reinterpreted and the 'zero-shot' claim weakened. The paper deserves credit for releasing data and code, but the PACO claim is load-bearing because it is the only benchmark that directly measures part-attribute recognition independently of the description prompt style.","tokens_in":11890,"tokens_out":6789,"duration_ms":61075,"concrete_test":"Compute the WordNet synset intersection between the 4,700 selected ImageNet21k fine-tuning classes and PACO's object category list, then re-run the Table 4 protocol restricted to PACO object categories with no overlapping synset (or retrain after excluding all overlapping categories). If the fine-tuned and multi-res improvements over the baseline shrink to near-noise on that subset, the PACO result does not support zero-shot attribute generalization and the central claim needs to be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest evidence for attribute-level generalization is Table 4, where PACO attribute-value classification improves from 33.22 (baseline CLIP) to 43.07 (fine-tuned) and 49.84 (multi-resolution). Section 4.2 says fine-tuning classes only excluded 'classes appearing on our test benchmarks'—the six fine-grained datasets in Section 5.3. PACO's 255 object categories are not mentioned in this exclusion. Since the 4,700 fine-tuning classes are sampled from ImageNet21k, common PACO categories such as bottle, bowl, car, dog, airplane, and backpack very likely share WordNet synsets with training classes. If so, the PACO evaluation is not a zero-shot test for those categories: the model can retrieve attribute values using category-specific visual associations learned from overlapping ImageNet21k images, without acquiring a generalizable notion of part attributes. Section 5.4's assertion that improvements reflect 'enhanced attribute recognition rather than generic latent class features' is thus unsubstantiated for the PACO result. The six-benchmark gains remain, but training and test descriptions are generated with the same Oxford/Columbia prompt styles, so they do not independently rule out style-level overfitting. The central claim therefore depends on a leakage control that the paper does not perform.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a zero-shot 'real' classification by description task, in which LLM-generated class descriptions have object class names removed, and documents a large CLIP accuracy drop when class names are absent (Tables 1-2). To mitigate this gap, the authors fine-tune CLIP on synthetic attribute descriptions for ImageNet21k classes and add a multi-resolution vision encoder (Section 4), reporting gains on six fine-grained benchmarks (Tables 5-6) and on PACO part-attribute classification (Table 4). They also release name-removed description data for the six benchmarks.","tokens_in":12174,"tokens_out":5495,"duration_ms":50778,"significance":"The paper addresses a genuine limitation of CLIP and contributes a useful task formulation, a released description dataset, and an open-source training recipe. If the empirical claims hold, the proposed fine-tuning and multi-resolution fusion are valuable steps toward attribute-level generalization. The main strengths are the concrete benchmark resource, the consistent directional gains across several datasets, and the attention to excluding test classes from the fine-tuning set. However, the evaluation controls are not yet tight enough to support the central attribution claim that the gains reflect improved part-attribute understanding rather than category overlap or style matching.","major_comments":[{"comment":"The PACO result is not established as zero-shot. Section 4.2 says training excluded 'classes appearing on our test benchmarks,' but the enumerated benchmarks in Section 5.3 are the six fine-grained datasets, and PACO's common object categories are not mentioned. ImageNet21k almost certainly contains several PACO categories (e.g., dog, car, bottle, airplane, backpack), so the 16-point PACO gain in Table 4 could come from category-level visual associations learned on overlapping training images rather than from a generalizable part-attribute representation. Please re-evaluate PACO after removing all ImageNet21k training classes that share a WordNet synset with PACO categories, or report PACO results stratified by category overlap with the training set.","section":"§4.2, §5.2, Table 4"},{"comment":"The paper states that results are averaged over three seeds but reports no standard deviations or per-seed values. Several Oxford-style gains are small (e.g., Cars196 from 9.4 to 10.1 and Food101 from 55.7 to 59.0 in Table 5), so without variance estimates the claim of 'consistent improvements across all datasets' is not statistically supported. Please report standard deviations or per-seed accuracies for all main result tables.","section":"§5.1, Tables 4-6"},{"comment":"Training and evaluation descriptions are generated with the same two LLM prompt styles, Oxford and Columbia. The improvements may therefore reflect matching the stylistic surface of LLM-generated text rather than improved attribute semantics. The Section 5.4 assertion that the gains are due to 'enhanced attribute recognition rather than generic latent class features' would be substantially more convincing with a held-out description style in the evaluation, for example human-written descriptions, a different LLM, or paraphrase-augmented test descriptions.","section":"§4.1, §5.3, §5.4"},{"comment":"No validation of the name-removal step is reported. Because the task definition and the released dataset both depend on class names being absent, the paper should quantify leakage, for example by automatic name detection, manual inspection of a sample, and an ablation comparing processed descriptions with and without the supercategory placeholder. Without such validation, the claim that the benchmarks enable 'real' classification by description is not fully supported.","section":"§4.1, Algorithm 1 (Supplementary)"}],"minor_comments":[{"comment":"In the Food-101 row for CLIP-ViT-B-16, '7T1.4' appears to be a typo for 71.4.","section":"Table 1"},{"comment":"The style attribution is inconsistent with Section 3.2: Section 4.1 says the two prompt styles are 'suggested by [9] (theOxford style) and [4] (The Columbia style)', whereas Section 3.2 attributes Oxford to [4] and Columbia to [9]. Please correct the reference mapping and the 'theOxford' typo.","section":"§4.1"},{"comment":"The sentence reporting gains is ambiguous: '3.2% on the Dogs120, OxfordPets, CUB, and Flowers102 datasets' could be read as one joint number or four separate values; please restructure for clarity.","section":"§5.4"},{"comment":"Consider adding standard deviations directly in the tables rather than only mentioning seed averaging in the text, so the reader can judge the stability of small gains.","section":"Tables 5-6"}],"recommendation":"major_revision","confidential_remarks":"This is a reasonable empirical contribution, but the evaluation controls need strengthening before the central claim can be accepted. In particular, the PACO category-overlap issue is load-bearing and should be resolved with a proper overlap analysis or a disjoint-category evaluation. The missing variance reporting and the shared prompt-style confound are also important. No concerns about authorship or disclosure; the work appears to fit the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a competent empirical paper and worth a serious look. The genuinely new bit is the task: zero-shot classification by description with class names removed from LLM-generated descriptions, plus released descriptions for six fine-grained benchmarks. That is a real resource. The paper also shows a simple recipe—fine-tune CLIP on ImageNet21k classes (test classes excluded) with LLM descriptions stripped of names—gives consistent gains on all six benchmarks, with larger gains for Columbia-style prompts where baseline is near chance. The multi-resolution fusion layer is a modest architectural addition; it helps a bit on fine-grained classification and more on PACO. The CUB attribute probe (Table 3) is useful evidence that fine-tuning moves part-attribute accuracy toward GPT-4-vision, though still far below.\n\nSoft spots, in order of importance. First, the PACO result is not clearly zero-shot. Section 4.2 says only test benchmark classes were excluded from the ImageNet21k training classes; PACO's 255 common-object categories are not mentioned. Since the 4,700 training classes are drawn from ImageNet21k, categories like dog, car, bottle, and bowl almost certainly overlap. So the 16-point PACO gain may be category-memorization-assisted attribute classification, not transfer to unseen categories. The authors' Section 5.4 claim that gains come from 'enhanced attribute recognition rather than generic latent class features' is not supported for PACO. Second, training and evaluation descriptions use the same two LLM prompt styles, so part of the six-benchmark gain may be style-level distribution matching rather than general attribute understanding. A cross-style evaluation or human-annotated descriptions would settle this. Third, there are no error bars despite three seeds; fine-grained gains are small enough that some could be noise. Fourth, the name-removal procedure (Algorithm 1) is not validated, and it is load-bearing for the whole 'real classification' claim.\n\nNone of this kills the paper. The six-benchmark name-free evaluation is held out at the class level, and the comparison is internally consistent. It is an empirical adaptation paper, not a derivation, and the contributions are clearly scoped. The main risk is that reviewers will over-read the PACO number as proof of attribute generalization.\n\nMy recommendation: send it to peer review. The task and benchmark are useful, the experiments are reproducible in principle, and the soft spots are fixable with a leakage-controlled PACO split and a cross-prompt evaluation. I would not cite it as evidence of general attribute understanding until those controls are added, but I would cite the name-free benchmark if I worked on description-based classification.\n\nBest.","headline":"A useful empirical paper: name-free description benchmark plus a fine-tuning recipe that consistently improves CLIP, but the PACO gain likely benefits from train/test category overlap and prompt-style matching.","tokens_in":12664,"tokens_out":2750,"would_cite":true,"duration_ms":25553,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP classifies by class name, not by description: removing the name collapses zero-shot accuracy on six fine-grained benchmarks, and training on name-free attribute descriptions substantially recovers it.","keywords":["zero-shot classification by description","CLIP","part attributes","fine-grained classification","vision-language models","ImageNet21k","multi-resolution vision transformer","PACO"],"falsifier":"Fine-tune the same pipeline on ImageNet21k using LLM descriptions, but with the attribute words replaced by random words while keeping the same name-free structure (a WaffleCLIP-style control), and measure PACO and zero-shot accuracy; if the gains persist, the improvement is not attributable to attribute-content learning. Alternatively, fine-tune on descriptions written in a third, non-Oxford/non-Columbia prompt style and test on the Oxford/Columbia benchmarks: if accuracy does not transfer, the gains come from style overlap rather than general part-attribute understanding.","tokens_in":11713,"feed_emoji":"🐦","tokens_out":7174,"duration_ms":58926,"temperature":0.7,"pith_summary":"Zero-shot classification by description is supposed to let vision-language models identify objects from their attributes, but this paper shows that CLIP's success in that task is largely carried by the object class name embedded in the description. When the name is stripped out, accuracy on six fine-grained benchmarks collapses dramatically, by up to roughly 70 points in some settings. The paper argues this exposes a real deficiency in part-attribute recognition and proposes a remedy: fine-tune CLIP on ImageNet21k images paired with LLM-generated, name-free attribute descriptions, and add a multi-resolution vision tower. With that training, the gap is substantially mitigated and held-out zero-shot accuracy improves on all six benchmarks, with a 16-point gain on PACO part-attribute classification.","feed_headline":"Removing class names from CLIP descriptions cuts accuracy by up to 70%","feed_subtitle":"New benchmarks and a multi-resolution CLIP fine-tuning recipe push attribute recognition ahead by 16 points on PACO.","key_machinery":"The load-bearing mechanism is the combination of name-free description data and fine-tuning on it. Descriptions are generated by an LLM from two prompt styles—Oxford, free-form narrative sentences, and Columbia, concise single-attribute sentences—and then passed through Algorithm 1, which replaces the object name with a super-category placeholder such as 'bird.' Training pairs K=50 images per class with N=10 name-free sentences for 4,700 ImageNet21k classes that are disjoint from the test classes. The architecture modification takes a 448×448 image, splits it into four 224×224 slices, runs each through the frozen CLIP ViT, averages the patch representations across slices, concatenates the average with the original patches, and feeds the result through a learnable linear layer and an extra ViT layer; the final embedding is blended with the backbone's CLS token via a learnable weight α. This forces the model to encode attribute content without relying on the class-name shortcut.","core_discovery":"The paper's central claim is that CLIP, despite strong zero-shot classification performance, does not genuinely understand object part attributes from free-form descriptions; it relies on class names as shortcuts. To establish this, the authors construct descriptions for six fine-grained datasets using two LLM prompt styles, then remove the class name with a text-processing algorithm. Accuracy drops from roughly 55–93% with names to 3–65% without names, depending on dataset and style. They then hypothesize that training on a diverse set of attribute descriptions whose class names are also removed—drawn from ImageNet21k classes disjoint from the test sets—forces the model to learn attributes rather than names, and they show consistent held-out gains. The multi-resolution CLIP variant, which aggregates patch features across four 224×224 slices of a 448×448 image and passes them through an extra ViT layer, adds a further large improvement on PACO part-attribute classification (49.84% vs. baseline 33.22%), while contributing only modest gains in zero-shot image classification.","pith_inferences":["A natural check the paper does not run: fine-tuning on a third, unfamiliar description style would test whether the gains come from general attribute understanding or from matching the Oxford/Columbia prompt styles used in both training and evaluation.","The super-category placeholders (such as 'bird') may leak coarse class information; replacing them with non-semantic tokens would give a stricter estimate of true name-free classification.","The multi-resolution module's largest gains on PACO, where a single phrase describes one part, suggest it could combine with spatially localized CLIP variants to test whether explicit grounding of phrases to image regions further improves attribute recognition.","The released name-free benchmarks could serve as a probe for other vision-language models beyond CLIP, revealing whether class-name reliance is a general VLM property or specific to contrastive late-fusion training."],"forward_implications":["Classification-by-description results that embed the class name in the prompt overstate what CLIP understands; reporting name-removed accuracy gives a truer measure of attribute comprehension.","Attribute-focused fine-tuning on name-free descriptions transfers to unseen classes, so large synthetically captioned image corpora can improve fine-grained zero-shot performance without test-class leakage.","The released name-free description sets for CUB, Flowers, Cars, Food, Dogs, and Oxford Pets establish a benchmark for genuine zero-shot classification by description.","Multi-resolution patch aggregation helps most for part-level attribute tasks (PACO) and less for whole-image classification, suggesting it mainly sharpens fine-detail encoding.","The large gap between name-included and name-free accuracy can be substantially mitigated by training on diverse attribute descriptions, narrowing CLIP's reliance on names."],"supporting_citations":[{"why":"Introduces CLIP, the vision-language model whose attribute-recognition limits the paper targets.","marker":"[14]"},{"why":"Supplies the Columbia-style concise attribute descriptions and the classification-by-description baseline the paper measures against.","marker":"[9]"},{"why":"Supplies the Oxford-style free-form description prompting used for training and test descriptions.","marker":"[4]"},{"why":"Shows that random words combined with class names match real descriptions in zero-shot classification, motivating the claim that names carry classification.","marker":"[18]"},{"why":"Analyses CLIP as a bag-of-words, the evidence base for the paper's diagnosis of shallow attribute understanding.","marker":"[28]"},{"why":"Provides the PACO part-attribute benchmark where the method reports its largest gains.","marker":"[15]"},{"why":"Provides ImageNet21k, the source of training classes and images for attribute fine-tuning.","marker":"[16]"},{"why":"Inspires the multi-resolution vision architecture that the paper adapts to CLIP.","marker":"[20]"}],"fun_headline_variants":["Without class names, CLIP's attribute recognition drops by up to 70%","CLIP's class-name shortcut exposed: 70% accuracy loss without names","Multi-resolution CLIP boosts part-attribute recognition by 16 points","New benchmarks expose CLIP's shortcut: it relies on object names"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing premise is that LLM-generated descriptions in the Oxford and Columbia styles are representative of attribute descriptions in general, and that the name-removal algorithm strips out all class-name leakage while preserving attribute content; if either fails, the measured 'real' classification gains could be inflated by prompt-style matching or residual name information.","fun_headline_variants_meta":{"raw":{"variants":["Without class names, CLIP's attribute recognition drops by up to 70%","CLIP's class-name shortcut exposed: 70% accuracy loss without names","Multi-resolution CLIP boosts part-attribute recognition by 16 points","New benchmarks expose CLIP's shortcut: it relies on object names"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3743,"prompt_tokens":975,"completion_tokens":2768,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2688}},"tokens_in":591,"tokens_out":2768,"duration_ms":18940,"temperature":1.0,"reasoning_tokens":2688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:37:29.688742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same pipeline on ImageNet21k using LLM descriptions, but with the attribute words replaced by random words while keeping the same name-free structure (a WaffleCLIP-style control), and measure PACO and zero-shot accuracy; if the gains persist, the improvement is not attributable to attribute-content learning. Alternatively, fine-tune on descriptions written in a third, non-Oxford/non-Columbia prompt style and test on the Oxford/Columbia benchmarks: if accuracy does not transfer, the gains come from style overlap rather than general part-attribute understanding.","supporting_citations":[{"cited_title":"Multi- modal classifiers for open-vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"Supplies the Oxford-style free-form description prompting used for training and test descriptions."},{"cited_title":"Waffling around for performance: Visual classification with random words and broad concepts","cited_arxiv_id":null,"evidence_quote":"Shows that random words combined with class names match real descriptions in zero-shot classification, motivating the claim that names carry classification."},{"cited_title":"When and why vision- language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022","cited_arxiv_id":null,"evidence_quote":"Analyses CLIP as a bag-of-words, the evidence base for the paper's diagnosis of shallow attribute understanding."},{"cited_title":"Paco: Parts and attributes of common objects, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the PACO part-attribute benchmark where the method reports its largest gains."},{"cited_title":"Imagenet-21k pretraining for the masses","cited_arxiv_id":null,"evidence_quote":"Provides ImageNet21k, the source of training classes and images for attribute fine-tuning."},{"cited_title":"When do we not need larger vision models?, 2024","cited_arxiv_id":null,"evidence_quote":"Inspires the multi-resolution vision architecture that the paper adapts to CLIP."}],"review_version":1}