{"id":"8358a26e-a4b1-4758-9df1-27554ff158fe","arxiv_id":"2411.18651","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VRL generates interpretable verbalized features from few-shot image pairs and uses them to train classifiers, outperforming prior LLM-based attribute methods on iNaturalist and Kiki-Bouba.","lead":"A new method called Verbalized Representation Learning (VRL) uses a vision-language model to automatically write down the visual features that distinguish object classes, then turns those language descriptions into numeric feature vectors for few-shot classification. The paper reports large accuracy gains on fine-grained species and novel object tasks using only 10 images per class, while producing human-readable features.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 24% absolute improvement is not attributable to VRL under the stated conditions; it is only visible in equal-scale few-shot comparisons or ensembles with CLIP/DINO, so the central quantitative claim is overstated.","rationale":"The reader's weakest assumption focuses on VLM test-time feature mapping reliability. That is a real risk, but the paper provides some independent support for the mapping: the human study in Appendix B.5 finds roughly 93% of sampled features faithful, and the CLIP-mapping variant in Table 5 still shows substantial gains, suggesting the method is not wholly dependent on LLaVA's binary judgments. The more load-bearing issue is quantitative and directly affects the central claim: the abstract's 24% figure is not traceable to any comparison satisfying the stated conditions. I verified pairwise differences in Tables 1 and 2: the only near-24% gap is VRL-72B (67.92) versus LLM-Mutate-70B† (44.13), which is an equal-scale, equal-data comparison, and the >24-point gaps in Tables 3 and 9 require CLIP/DINO ensembles. This does not invalidate the core idea — VRL appears to give 8-20 point gains in several matched comparisons — but it means the paper's most prominent claim overstates what VRL alone delivers. The reader's CONDITIONAL verdict already calls for re-reporting and separating ensemble results, so my concern reinforces that verdict rather than changing it. The authors should correct the abstract, report VRL-only gains as the primary numbers, and add error bars or repeated-seed results given the small validation sets.","tokens_in":19768,"tokens_out":9696,"duration_ms":84370,"concrete_test":"Construct a single matched table: use only VRL features Fdiff and Fcomm, a fixed classifier (logistic regression), and the baselines LLM-Mutate-7B†/70B† and LaBo-7B†/70B† all run with the same 10-shot data and matched model scale (7B vs 7B and 72B vs 70B). Then report the maximum absolute gain of VRL over the best baseline within each scale match. If the gain is not 24%, revise the abstract to the measured value and remove the 'smaller model' and '95% less data' phrasing unless they hold for that same comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is 'VRL achieves a 24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller model.' No row in Tables 1 or 2 satisfies this jointly. In Table 1, VRL-7B (58.46) beats reproduced LLM-Mutate-7B† (38.20) by 20.3 points, and VRL-72B (67.92) beats LLM-Mutate-70B† (44.13) by 23.8 points — the likely source of the '24%' — but that comparison is between a 72B model and a 70B model on equal 10-shot data, so neither 'smaller model' nor '95% less data' applies. On Kiki-Bouba, the largest few-shot gain in Table 2 is 83.0 vs 63.25, or 19.8 points. The larger numbers that exceed prior methods by more than 24 points (76.52 and 79.92 in Tables 3 and 9) are ensembles of VRL features with CLIP and DINO image features, not VRL alone. Thus the most advertised number is not a clean measure of the proposed method; the contribution may remain positive, but the central quantitative claim as written is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Verbalized Representation Learning (VRL), a few-shot image classification method in which a vision-language model generates verbalized features by contrasting pairs of images from different classes (inter-class difference) and by identifying commonalities between images of the same class (intra-class commonality). The same VLM then maps each image to a binary or continuous feature vector indicating the presence of each verbalized feature, and these vectors are used to train classifiers such as logistic regression, MLP, or ensembles with CLIP/DINO features. Experiments are conducted on iNaturalist for fine-grained classification, on Kiki-Bouba for novel-object classification, and on mini-ImageNet in the appendix, with additional ablations and a human study of learned features.","tokens_in":20078,"tokens_out":7338,"duration_ms":62705,"significance":"The central idea—using a VLM to verbalize discriminative and shared visual features from few-shot pairs and then using those verbalized features as an interpretable bottleneck—is timely and potentially useful. The paper ships code, reproduces few-shot baselines under matched data conditions, provides ablations over classifiers and feature-mapping models, and attempts a human evaluation of feature quality. If the claims are properly scoped, the method offers a data-efficient, interpretable alternative to fine-tuning and prior attribute-generation approaches. However, the headline quantitative claims are not supported by the tables as written, and several evaluation-protocol details must be clarified before the paper can be accepted.","major_comments":[{"comment":"The headline '24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller model' is not supported by any single row in Table 1. VRL-7B (58.46%) is 8.40 points above the full-data LLM-Mutate-70B (50.06%) and 14.33 points above the reproduced 10-shot LLM-Mutate-70B† (44.13%); the approximately 24-point gap appears only for VRL-72B (67.92%) versus LLM-Mutate-70B†, which is not a smaller-model comparison. The numbers above 74% in Tables 3 and 9 come from ensembling VRL features with CLIP and DINO image features, not from VRL alone. The abstract, Figure 1, and the conclusion should either cite the exact comparison being made or delete the 24% figure; as written, the paper's central quantitative claim is overstated.","section":"Abstract; §4.3; Table 1"},{"comment":"The mapping from images to feature vectors is performed by the same VLM that generated the features, and the paper provides no direct validation that this mapping is reliable on held-out images. The human study in Appendix B.5 samples only 20 features per super class (100 total) on iNaturalist and does not cover the full feature sets used in Tables 1–5 or any Kiki-Bouba features; the 93% relevance score therefore does not establish the reliability of the full feature vectors on unseen test images. Please add a per-feature human-agreement study on held-out images for the actual feature sets, and quantify the effect of replacing the mapping model's outputs with human ground-truth feature labels.","section":"§3.1, F = VLM(y, x); Appendix B.5"},{"comment":"The evaluation protocol is not transparent about how the reported accuracy is selected. Section 4.2 says 'We report the best performance achieved across different classifiers, selecting the optimal results from a single classifier,' but Table 1 reports VRL-7B at 58.46, while Table 10 shows 62.06 for the 7B 'both' configuration with logistic regression and LLaVA-based mapping. Since logistic regression is among the classifiers and 62.06 is higher than 58.46, the best-classifier rule cannot reproduce the headline number unless the two tables use different training-pair samples or feature sets, which is not explained. The paper should state a fixed evaluation protocol and report variance over multiple few-shot splits.","section":"§4.2; Tables 1, 5, 10"},{"comment":"The comparison with human-labeled attributes on Kiki-Bouba is missing a description of how the human attributes were collected. The paper says they are 'human-annotated attributes provided by [12]' but never states whether the annotators worked under the same 10-shot condition as VRL or whether the attributes were produced with full-dataset supervision. If the human attributes were obtained with substantially more data or different annotation instructions, the reported 20% gain would not demonstrate superiority over human feature labeling under matched conditions. Please describe the annotation protocol and, if possible, collect human attributes under the same few-shot setting.","section":"Table 4"}],"minor_comments":[{"comment":"The abstract contains the typo 'a smaller mode' for 'a smaller model'; also, the phrase 'at the same model scale' is contradicted by the cross-scale comparisons in Table 1 and should be reworded.","section":"Abstract"},{"comment":"The table title spells the dataset as 'Kiki-Kouba' while the rest of the paper uses 'Kiki-Bouba'; please correct the spelling.","section":"Table 2"},{"comment":"The expression 'CC2 × CN2' is not typeset clearly; please use standard binomial notation, e.g., C(C,2) × C(N,2), and define the pair-counting formulas for both positive and negative pairs.","section":"§3.1"},{"comment":"References [9, 30, 31] appear in the appendix comparison table but are not cited in the main text; please ensure all references are introduced consistently where they are first used.","section":"Appendix B.3"},{"comment":"The caption states that key attributes are highlighted in bold, but the printed figure does not show clear bold formatting; consider using color or an explicit marker in the figure.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core method appears promising and the empirical gains are likely real in the few-shot setting, but the advertised 24% and 20% numbers must be re-anchored to the actual comparisons, and the evaluation protocol needs to be made reproducible and statistically more defensible. I would not reject the paper, but these issues are load-bearing enough to require a revision before a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core method is real: pairing images from different classes and asking a VLM to verbalize differences/commonalities, then mapping those to binary vectors for a downstream classifier, is a clean idea and it does improve few-shot classification on iNaturalist and Kiki-Bouba. The reproduced baselines under the same 10-shot setting make the comparison fair, and the gains are consistent: VRL-7B beats LLM-Mutate-7B and LaBo-7B by double digits, and VRL-72B does the same at the larger scale. That part holds up.\n\nThe paper also does some things right: code is released, the human study in Appendix B.5 gives at least some grounding that the features are not all hallucinated, and the qualitative examples are plausible. The novelty with respect to LaBo, CBD, and LLM-Mutate is genuine – those generate attributes from class names, while VRL grounds them in actual image pairs.\n\nThe soft spot is the abstract. The '24% absolute improvement' is not attributable to VRL under the stated conditions. In Table 1 the closest match is VRL-72B (67.92) vs LLM-Mutate-70B† (44.13) at the same 10-shot setting, which is 23.8 points, but that is not 'using 95% less data and a smaller model' – both use the same few-shot data and the 72B model is not smaller. On Kiki-Bouba the largest few-shot gain is 19.8 points, not 24. The numbers that are actually 24+ points higher (76.52, 79.92) come from ensembling VRL features with CLIP and DINO embeddings, which is not the proposed method alone. The paper's own Section 4.3 correctly reports 8%, 18%, 20%, and 23% gains in the text, so the abstract seems to have been written from the ensemble or the equal-scale comparison.\n\nOther evaluation issues are minor but real: no error bars or multiple seeds, and 'we report the best performance achieved across different classifiers' is selection bias. The same VLM proposes and scores features, which is a mild self-reference that could inflate apparent interpretability, though the human study and the fact that the features help classification mitigate that concern. The footnote in the abstract says 'while using 95% less data and a smaller model' – that claim is only true compared to the full-dataset baselines, not the reproduced few-shot baselines.\n\nBottom line: the central claim that VRL improves few-shot classification with interpretable features is supported. The headline number is not. This is a solid paper that needs an honest revision of the abstract and a more rigorous evaluation reporting. It deserves peer review.","headline":"Core VRL method works and shows consistent few-shot gains, but the advertised 24% claim is not supported by the method alone and the evaluation needs error bars and less selection bias.","tokens_in":20614,"tokens_out":2337,"would_cite":true,"duration_ms":20886,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a vision-language model can write task-specific verbalized features from a few pairs of images, and that these features beat prior attribute-generation pipelines and human labels.","keywords":["Verbalized representation learning","Few-shot classification","Interpretable features","Vision-language models","Inter-class difference","Intra-class commonality","Concept bottleneck","Natural language attributes"],"falsifier":"A direct check is to corrupt the verbalized feature bridge: replace the VLM's feature-mapping answers with shuffled or random values while keeping the classifier training identical. If accuracy drops by far less than the reported 24-point gain, the claimed grounding in language is not doing the work. A cleaner test is to collect human ground-truth labels for a sample of the actual features used in the main tables and measure per-feature agreement on held-out images; low agreement would falsify the claim that the features are reliably grounded.","tokens_in":19579,"feed_emoji":"🧠","tokens_out":9812,"duration_ms":82312,"temperature":0.7,"pith_summary":"The paper tries to establish that a vision-language model can produce human-interpretable, task-specific features for image classification from only a handful of examples per class. Instead of relying on the model's memorized knowledge of class names, the method asks the model to compare actual images: what differs between two species, and what is shared within a species. Those verbalized answers are then converted into binary or continuous vectors that any standard classifier can use. The paper reports a 24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller model, and a 20% absolute gain over human-labeled attributes. A sympathetic reader would care because it offers a path to interpretable few-shot recognition without manual annotation or model fine-tuning.","feed_headline":"Verbalized features beat prior few-shot classifiers by 24 points","feed_subtitle":"Ask a vision-language model to describe differences and commonalities; a 7B model then outperforms 70B pipelines on one-twentieth of the…","key_machinery":"The machinery is a prompted pairing loop over the few-shot training set. For every sampled pair of images from different classes, the vision-language model is asked to name the most distinctive difference ($y_{\\text{diff}}$); for every sampled pair from the same class, it is asked to list features shared by both images that make the class distinct ($y_{\\text{comm}}$). A second pass converts these descriptions into yes/no or class-choice questions, and each image is scored against every question by the VLM, yielding a feature vector $F = \\mathrm{VLM}(y,x)$ whose dimensions are verbalized features. These vectors are fed to off-the-shelf classifiers such as logistic regression, random forests, or MLPs, optionally ensembled, and optionally combined with CLIP or DINO embeddings. The verbalized format is what carries the argument: it grounds features in the data, makes them inspectable by a human, and prevents the collapse-to-constant-output failure of ordinary contrastive self-supervised learning.","core_discovery":"The central discovery is that the features which separate classes and the features which bind a class together can be written down as natural-language descriptions by a vision-language model, grounded in the actual few-shot images rather than in the model's priors. The paper calls this Verbalized Representation Learning (VRL). With 10 images per species on iNaturalist, the 7B variant reaches 58.46% average accuracy and the 72B variant reaches 67.92%, while prior methods using 200+ images per class and 70B models reach 50.06%. On Kiki-Bouba, a dataset of novel abstract objects, the 7B variant reaches 83.00% with 10 images per object, versus 69.30% for the prior 70B pipeline trained on 800+ images per object. The paper also reports that these automatically extracted features outperform human-annotated attributes by 20% absolute when both are used for downstream classification.","pith_inferences":["A consequence the authors leave implicit is that the verbalized format makes the classifier's reasoning auditable: one can read which features drove a prediction and edit a single text description to correct a systematic error.","A testable extension would apply the same pairing loop to domains where the vision-language model has weak grounding, such as medical or satellite imagery, and measure whether per-feature agreement degrades faster than accuracy.","Because feature generation scales with the number of image pairs, deliberately searching for the most informative pairs rather than sampling them could raise the ceiling of few-shot performance even further.","The method's assumption is about the mapping model's reliability, not its size, so the same pipeline could in principle run on a much smaller or closed-source VLM if the mapping step remains faithful."],"forward_implications":["VRL with a 7B vision-language model outperforms prior 70B attribute-generation pipelines on iNaturalist while using only 10 images per species instead of 200+.","On Kiki-Bouba, VRL reaches 83.00% accuracy with 10 images per object, surpassing prior state-of-the-art methods that used 800+ images per object.","The verbalized features combine additively with CLIP and DINO embeddings, raising iNaturalist accuracy to 79.92% via classifier ensembling.","VRL-extracted features beat human-labeled attributes by 20% absolute on Kiki-Bouba, suggesting that automatic feature extraction can replace manual attribute annotation in low-resource settings.","The same verbalized features work across multiple classifier families and both LLaVA and CLIP feature-mapping models, indicating the representation is not tied to one classifier or one VLM."],"supporting_citations":[{"why":"Supplies the previous state-of-the-art LLM-Mutate pipeline, the iNaturalist and Kiki-Bouba evaluation protocols, and the human-labeled attributes used for comparison.","marker":"[12]"},{"why":"LaBo is the language-bottleneck baseline whose official implementation is adapted for same-data comparison.","marker":"[50]"},{"why":"Classification-by-Description is the attribute-generation baseline that relies on LLM prior knowledge, which VRL is designed to surpass.","marker":"[32]"},{"why":"LLaVA-OneVision is the vision-language model used both to generate the verbalized features and to map them to numeric vectors.","marker":"[28]"},{"why":"CLIP provides the alternative feature-mapping model and the pretrained visual embeddings used in feature fusion and ensembling.","marker":"[39]"},{"why":"SimCLR's contrastive formulation is the stated inspiration for the inter-class difference feature extraction.","marker":"[8]"},{"why":"MoCo's negative-pair contrastive learning motivates the negative-pair construction for inter-class difference features.","marker":"[19]"},{"why":"SimSiam's negative-free self-supervised objective motivates the positive-pair commonality extraction.","marker":"[10]"},{"why":"Supplies iNaturalist 2021, the fine-grained dataset used for the main few-shot comparisons.","marker":"[47]"}],"fun_headline_variants":["Language descriptions of visual features cut data needs by 95%","VLM-written object features beat 70B pipelines with a 7B model","Few-shot recognition via natural-language features: 24-point gain","Verbalized features best human attributes by 20 points","Interpretable few-shot learning: describe classes in words"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the vision-language model reliably decides, for each unseen test image, whether a verbalized feature is present or absent; if it often hallucinates or misses a feature, the entire feature vector is noise and the downstream classifier cannot recover.","fun_headline_variants_meta":{"raw":{"variants":["Language descriptions of visual features cut data needs by 95%","VLM-written object features beat 70B pipelines with a 7B model","Few-shot recognition via natural-language features: 24-point gain","Verbalized features best human attributes by 20 points","Interpretable few-shot learning: describe classes in words"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3486,"prompt_tokens":961,"completion_tokens":2525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2438}},"tokens_in":577,"tokens_out":2525,"duration_ms":16652,"temperature":1.0,"reasoning_tokens":2438,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:37:13.960504+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to corrupt the verbalized feature bridge: replace the VLM's feature-mapping answers with shuffled or random values while keeping the classifier training identical. If accuracy drops by far less than the reported 24-point gain, the claimed grounding in language is not doing the work. A cleaner test is to collect human ground-truth labels for a sample of the actual features used in the main tables and measure per-feature agreement on held-out images; low agreement would falsify the claim that the features are reliably grounded.","supporting_citations":[{"cited_title":"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification","cited_arxiv_id":null,"evidence_quote":"LaBo is the language-bottleneck baseline whose official implementation is adapted for same-data comparison."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP provides the alternative feature-mapping model and the pretrained visual embeddings used in feature fusion and ensembling."},{"cited_title":"A simple framework for contrastive learning of visual representations","cited_arxiv_id":null,"evidence_quote":"SimCLR's contrastive formulation is the stated inspiration for the inter-class difference feature extraction."},{"cited_title":"Exploring simple siamese rep- resentation learning","cited_arxiv_id":null,"evidence_quote":"SimSiam's negative-free self-supervised objective motivates the positive-pair commonality extraction."}],"review_version":1}