{"id":"17794447-cbf3-422f-b895-d95a5b473d09","arxiv_id":"2501.04975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A concept bottleneck model that discovers class concepts directly from images by quantizing them into nearest common words in CLIP space, matching or beating LLM-generated concept bottlenecks without using LLMs.","lead":"This paper builds interpretable image classifiers by first converting each image into a short list of everyday words, such as 'blue' or 'black head', and then combining those words to make a prediction. The method needs no human concept annotations and no language model, only unlabeled images and a pre-trained vision-language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ImageNet result is confounded: U is sampled from the ImageNet training set, so concept mining uses target-class images; an external unlabeled pool test is needed before claiming parity/outperformance on ImageNet.","rationale":"The reader's conditional verdict is well-calibrated. My stress-test narrows the decisive issue: the ImageNet result, which is the strongest and most novel claim, is not a fair test of the paper's stated setting ('unlabeled images readily acquired from the Internet') because the unlabeled pool is sampled from the ImageNet training set. This is not an internal inconsistency, but it is a threat to the external validity of the headline. The method's per-class selection mechanism (Eq. 3) uses the class name to pull images from U; when U contains the target classes, this is effectively class-conditional concept mining, and the later frequency filtering inherits that information. A 0.2-point margin over LP and 0.1-point margin over LaBo is too small to absorb this confound. The proposed test with an external corpus would settle whether the method generalizes as claimed. I do not recommend changing the verdict from conditional: the method is plausible, the ablations (e.g., vocabulary relevance in Table 7) are consistent with the mechanism, and the rest of the evidence supports the method working. However, the ImageNet-specific claim should be conditioned on an external-U validation before being stated as a definitive outperformance.","tokens_in":16189,"tokens_out":7394,"duration_ms":76089,"concrete_test":"Re-run the ImageNet experiment with U replaced by a 200k-image random sample from an external corpus (e.g., LAION-400M or YFCC100M, with near-duplicates of ImageNet removed), keeping all hyperparameters fixed. Compute paired bootstrap confidence intervals for the differences 84.1 vs 83.9 (LP) and 84.1 vs 84.0 (LaBo). If V2C-CBM no longer reaches or exceeds the linear probe or LaBo accuracy on ImageNet, the headline claim is an artifact of the target-training-set U. A secondary check: apply the class-name selection in Eq. 3 to the external pool and report how many ImageNet classes have fewer than, say, 10 retrieved images; if coverage is poor, the current ImageNet setup is not representative of the claimed web-image setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that V2C-CBM 'even outperformed black-box linear probing methods on ImageNet' rests on a setup where the auxiliary unlabeled pool U is randomly sampled from the ImageNet training set (Implementation Details, default number of unlabeled images is 200k). Because U is drawn from the exact distribution and classes of the target task, the per-class selection step (Eq. 1 and Eq. 3) retrieves images of the very classes being classified, and the concept filtering step then builds each class's codebook from those class-specific images. This gives V2C-CBM access to target-domain image statistics that LLM-supervised baselines (LaBo, CDM, DCLIP) and the linear probe do not use. The observed ImageNet margin is tiny (84.1 vs 83.9 for LP, 84.0 for LaBo), so it is plausible that this extra information, rather than the general discoverability of visual concepts from common words, explains the result. If U were instead a genuinely external web corpus, as the method narrative claims, the class-name-based selection in Eq. 3 could retrieve few or no relevant images for many ImageNet classes, degrading the concept bottleneck and likely removing the claimed advantage. The 'surpassed' conclusion is also broader than Table 1 supports (Ours is below LaBo on Aircraft 60.7 vs 61.3 and Flower 98.8 vs 99.3), but the ImageNet confound is the decisive issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes V2C-CBM, a concept bottleneck model that avoids LLM-generated concepts by building a vocabulary from common English words and using an auxiliary unlabeled image pool. A Vision-to-Concept (V2C) tokenizer quantizes CLIP image features into the nearest concept-text embeddings; per-class concepts are selected by frequency over class-selected unlabeled images; and a linear layer maps the concept similarity scores to class predictions. Experiments on ten datasets compare with LaBo, CDM, DCLIP, DN-CBM, and linear probing, including few-shot results for 1 to 16 shots. The central claim is that V2C-CBM matches or outperforms LLM-supervised CBMs and even outperforms black-box linear probing on ImageNet.","tokens_in":16506,"tokens_out":4870,"duration_ms":48142,"significance":"The proposal is genuinely appealing: concept bottlenecks are constructed without LLMs, without concept annotations, and without training a sparse autoencoder, and the authors release code. The ablations on vocabulary size, unlabeled pool size, and initialization are useful. If the ImageNet claim were clean, this would be a notable result for interpretable classification. However, the headline ImageNet comparison is confounded by the transductive use of the ImageNet training set as the unlabeled pool, so the significance as stated is not yet established. The non-ImageNet results are still informative, but the paper's strongest claim needs a cleaner experimental test.","major_comments":[{"comment":"For ImageNet, the unlabeled pool U is randomly sampled from the ImageNet training set. Because U_q is formed by selecting images most similar to each class-name text feature (Eq. 3), the per-class concept codebook is derived from images of the target classes themselves. The ImageNet result (84.1 vs 83.9 for LP, 84.0 for LaBo) is therefore measured in a transductive setting and does not support the claim that concepts are discovered from generic unlabeled web images. Please rerun the ImageNet experiment with a genuinely external unlabeled pool (e.g., a non-overlapping web corpus such as a YFCC or LAION subset) or explicitly restrict U to exclude target-domain images. If the advantage disappears, the conclusion should be revised accordingly.","section":"Implementation Details; Eq. (1)-(3); Table 1"},{"comment":"The conclusion states that V2C-CBM 'has surpassed LLM-guided CBMs across various datasets and even outperformed black-box linear probing methods on the ImageNet dataset,' but this overstates the evidence. In Table 1, Ours is below LaBo on Aircraft (60.7 vs 61.3) and Flower (98.8 vs 99.3), and the ImageNet advantage over LP (84.1 vs 83.9) is 0.2 points with overlapping standard deviations. The abstract's phrase 'matched or outperformed' is supported by the table; the conclusion should be softened to match the actual comparisons.","section":"Conclusion; Table 1"},{"comment":"The method is described as training-efficient, but the final configuration uses per-dataset, per-shot hyperparameters (learning rate, batch size, max epochs, and K = 25 or 50) selected by grid search on validation sets. This is a substantial tuning surface, especially for a method whose concept discovery is claimed to be training-free. Please report the number of grid points, whether the same tuning protocol was applied to the LaBo and linear-probing baselines, and whether the ImageNet result is robust to a single default hyperparameter setting rather than the per-dataset tuned values.","section":"Evaluation; Table 16"},{"comment":"All experiments use a single source for the unlabeled pool U, namely the ImageNet training set. For ImageNet this is the transductive confound described above; for the other datasets it means the method has been tested with only one, highly diverse auxiliary corpus. The paper claims U 'can be obtained easily from large-scale web images,' but no experiment varies the source of U. Please add at least one experiment with a different external unlabeled source to show that the concept discovery mechanism is not specific to ImageNet statistics.","section":"Concept Filtering; Implementation Details"}],"minor_comments":[{"comment":"There is a typo in 'such aspart of' (missing space after 'as').","section":"Method, Concept Vocabulary"},{"comment":"The phrase 'as exampled in Figure 2' should read 'as exemplified in Figure 2'.","section":"Method, Concept Vocabulary"},{"comment":"The notation sim(F^c_concept, x^q_u) is overloaded: the left side suggests a similarity between a concept feature and a single image, while the right side uses the augmented image set A(x^q_u); please define A and clarify that the maximum or mean over augmentations is used.","section":"Method, Concept Filtering, Eq. (4)"},{"comment":"The abbreviations 'Oursp' and 'Oursr' are used in tables but only defined in the caption of Table 8; please define them in each table caption or in a common notation list.","section":"Tables 8 and 14"},{"comment":"There is a typo in the prompt template: 'a jpeg corupted photo' should be 'a jpeg corrupted photo'.","section":"Supplement, Table 10"},{"comment":"The phrase 'training efficient' should be qualified, since the pipeline requires a 200k-image unlabeled pool and per-dataset hyperparameter search; a sentence on wall-clock time or compute would help readers calibrate the claim.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is worth pursuing, but the ImageNet experiment needs to be redone with an external unlabeled pool before the central claim can be accepted. The authors might also be asked to share the exact hyperparameter search grids and the code configuration for the ImageNet run to verify reproducibility. If the transductive advantage disappears, the paper would still be a solid contribution as a training-free LLM-free CBM for non-ImageNet benchmarks, but the conclusion and abstract would need to be adjusted accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead V2C-CBM. The core idea—build a concept bottleneck from a fixed common-word vocabulary plus a pool of unlabeled images, using CLIP to filter and tokenize, no LLM and no trained autoencoder—is genuinely new and worth knowing. The paper is clearly written and the ablations are extensive. The V2C tokenizer is simple enough to reproduce, and the code link is a plus.\n\nWhat the paper does well: It positions itself cleanly against LaBo and DN-CBM, uses the same backbone and few-shot splits, and reports standard deviations. The few-shot story is honest: it lags LaBo at 1- and 2-shot, catches up by 4-8, and the initialization ablation shows the prior matters most when data is scarce. The concept visualization table is illustrative, though not quantitative.\n\nThe soft spot is the ImageNet result, and it is load-bearing. The unlabeled pool U is randomly sampled from the ImageNet training set. For the ImageNet evaluation, that means the concept-filtering step selects images from the very classes being classified. The method therefore gets target-domain image statistics that LaBo, DCLIP, and the linear probe do not see. The margin over the linear probe is 0.2 points (84.1 vs 83.9), which within the reported std devs is not convincing. The conclusion's phrase 'surpassed black-box linear probing' is simply not established. And the broader claim that V2C-CBM 'surpassed LLM-guided CBMs across various datasets' is contradicted by Table 1 itself: Ours is below LaBo on Aircraft and Flower.\n\nAlso worth noting: Table 16 shows heavy per-dataset, per-shot hyperparameter tuning (learning rate, batch size, epochs, even K for Flower is 25 instead of 50). That doesn't kill the method, but it weakens the 'training efficient' framing. The circularity concern is real only for ImageNet; for the other datasets U is external, so the method's general recipe is still plausible.\n\nMy take: this is a solid paper with a confounded headline result. The right fix is to rerun the ImageNet experiment with an external unlabeled pool (web images not overlapping with ImageNet) and add significance tests. Even if the ImageNet advantage evaporates, the method is a valuable baseline for training-free CBM construction.\n\nSend it to peer review—it deserves careful referee time. The issues are fixable with an experiment, not a rewrite.","headline":"A clever training-free CBM recipe, but the ImageNet win is confounded by using ImageNet's own training images as the unlabeled pool.","tokens_in":17061,"tokens_out":3480,"would_cite":true,"duration_ms":31641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"V2C-CBM shows that concept bottleneck classifiers can be built from common words and unlabeled images alone, matching or beating LLM-guided concept bottlenecks and even a CLIP linear probe on ImageNet.","keywords":["concept bottleneck models","interpretable machine learning","vision-language models","CLIP","image quantization","concept discovery","few-shot classification","unlabeled images"],"falsifier":"Take the Food-101 setup and swap the unlabeled image pool for images that contain none of the target classes, such as landscapes; the paper's logic implies the filtered concept vocabulary will no longer contain food-relevant words, and accuracy should drop sharply, approaching the 'without food' ablation's 28.9-point one-shot gap.","tokens_in":122,"feed_emoji":"🧩","tokens_out":5707,"duration_ms":228611,"temperature":0.7,"pith_summary":"This paper tries to establish that concept bottleneck models—classifiers that first name human-readable visual concepts and then combine them linearly—can be built without large language models or expert annotations. The proposed V2C tokenizer uses CLIP and a vocabulary of common English words, plus unlabeled images selected by class name, to turn any image into a short list of visual concepts. On ten classification benchmarks, the resulting V2C-CBM matches or exceeds LLM-guided concept bottlenecks and, on ImageNet, also exceeds a linear probe on full CLIP features. The payoff is a training-efficient, inherently interpretable model whose concepts are concise and visual, such as \"black head\" or \"white bear.\"","feed_headline":"Common words plus unlabeled images match LLM-made concept lists","feed_subtitle":"A training-free tokenizer turns CLIP into concept bottlenecks that beat LLM-supervised models and even linear probes.","key_machinery":"The V2C tokenizer is the mechanism: it maps an image feature vector to the K concept texts whose CLIP embeddings are nearest in Euclidean distance (Eq. 5), after a concept-filtering step that keeps only the most frequent, visually grounded words for each class. The vocabulary starts from common English words, extended by adjective-noun bigrams and relational trigrams, and is filtered using augmented views of class-selected unlabeled images. The tokenizer itself is training-free; only the class-concept weight matrix $W$ of the linear head is learned.","core_discovery":"The central claim is that the visual concepts needed for a concept bottleneck can be discovered from images themselves rather than generated by a language model. The paper shows that a codebook of common words, filtered by how often each word is the nearest CLIP text embedding to class-selected unlabeled images, yields a bottleneck that is both interpretable and accurate. Specifically, V2C-CBM outperforms or matches GPT-3-supervised CBMs (LaBo, CDM, DCLIP) and the sparse-autoencoder-based DN-CBM across CIFAR, fine-grained, texture, remote-sensing, and medical datasets, and reaches 84.1% on ImageNet versus 83.9% for a linear probe. The method also closes the few-shot gap to LaBo once four or more labeled examples per class are available.","pith_inferences":["Because the vocabulary is just common words and the tokenizer is frozen, the same codebook could be reused across datasets; a testable extension is to specialize the word list per domain (medical, remote sensing) and see whether accuracy improves further.","The ImageNet result uses ImageNet training images as the unlabeled pool, which makes the ImageNet evaluation partly transductive; on a truly out-of-distribution target the margin over the linear probe may shrink, and this could be tested with an unlabeled pool from a disjoint source.","The nearest-token selection rule suggests a natural intervention interface: editing or removing a concept (e.g., \"white\") should change predictions in predictable directions, offering a cheap faithfulness check the paper does not run."],"forward_implications":["Concept bottlenecks for new classification tasks can be constructed from a frozen CLIP model and unlabeled web images, with no LLM querying and no tokenizer training.","The accuracy gap between inherently interpretable CBMs and black-box linear probes can be closed or inverted on large datasets: 84.1% versus 83.9% on ImageNet.","In few-shot settings, the method trails LLM-guided LaBo at 1–2 shots but overtakes it from 4 shots onward, so the value of LLM prior knowledge shrinks as labeled data grows.","The discovered concepts are short visual phrases (e.g., \"purple center,\" \"black head\") rather than verbose or non-visual LLM descriptions, making the bottleneck easier for a person to inspect and intervene on."],"supporting_citations":[{"why":"Introduces concept bottleneck models and the two-step design of predicting concepts then classifying from them.","marker":"Koh et al. 2020"},{"why":"LaBo is the main LLM-guided few-shot baseline whose settings and prior-initialization idea V2C-CBM adopts and compares against.","marker":"Yang et al. 2023"},{"why":"LF-CBM established the paradigm of scaling LLM-generated concept bottlenecks to ImageNet, which this paper aims to avoid.","marker":"Oikarinen et al. 2023"},{"why":"DN-CBM is a contemporaneous common-words concept bottleneck requiring a sparse autoencoder, positioned as the training-free alternative.","marker":"Rao et al. 2024"},{"why":"DCLIP is a classification method using LLM descriptions and serves as one of the compared baselines.","marker":"Menon and Vondrick 2023"},{"why":"CDM is another LLM-description-based baseline used for comparison.","marker":"Panousis, Ienco, and Marcos 2023"},{"why":"Supplies the English word-frequency list used to form the base concept vocabulary.","marker":"Norvig 2009"},{"why":"GPT-3 is the large language model used to generate concepts in the baselines that V2C-CBM is compared with.","marker":"Brown et al. 2020"}],"fun_headline_variants":["Images define concepts: no LLM needed for bottleneck accuracy","Unlabeled images match or beat LLM-supervised concept lists","Concept bottlenecks from common words, no LLM supervision","Common words plus unlabeled images match LLM-made concept lists","V2C-CBM: build concept bottlenecks without language models"],"cache_read_input_tokens":19072,"weakest_assumption_plain":"The premise that breaks the method is that the everyday words most often matched to unlabeled example images of a class actually capture what visually separates that class from others; if they do not, the linear classifier built on those word scores cannot match a full black-box probe.","fun_headline_variants_meta":{"raw":{"variants":["Images define concepts: no LLM needed for bottleneck accuracy","Unlabeled images match or beat LLM-supervised concept lists","Concept bottlenecks from common words, no LLM supervision","Common words plus unlabeled images match LLM-made concept lists","V2C-CBM: build concept bottlenecks without language models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00072,"raw_usage":{"total_tokens":3228,"prompt_tokens":934,"completion_tokens":2294,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2211}},"tokens_in":550,"tokens_out":2294,"duration_ms":16309,"temperature":1.0,"reasoning_tokens":2211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:31.133082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Food-101 setup and swap the unlabeled image pool for images that contain none of the target classes, such as landscapes; the paper's logic implies the filtered concept vocabulary will no longer contain food-relevant words, and accuracy should drop sharply, approaching the 'without food' ablation's 28.9-point one-shot gap.","supporting_citations":[],"review_version":1}