{"id":"deb9bd71-ee42-468d-ab06-c1bfde844c0e","arxiv_id":"2501.15140","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Finedefics improves fine-grained image classification in multimodal LLMs by contrastively aligning image, attribute, and category embeddings, though its headline gains are measured against zero-shot baselines rather than equally fine-tuned ones.","lead":"This paper diagnoses why multi-modal large language models misclassify fine-grained categories (e.g., dog breeds, aircraft models) and proposes a training method called Finedefics that uses AI-generated attribute descriptions to better align images with category names. The method reportedly lifts average accuracy over the base model Idefics2 by about 11 points across six fine-grained datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded: the paper's own same-setting 'I.T. only' baseline (Table 6b, 76.13%) nearly equals Finedefics (76.84%), so the advertised +10.89% margin over zero-shot Idefics2 does not isolate attribute augmented alignment.","rationale":"I read the paper in good faith. The diagnostic analysis in Section 2 is suggestive, the internal ablations are informative, and the method is plausible: hard negatives, attribute descriptions, and two-stage training each show positive effects in Tables 3 and 6. Those internal results are real evidence that the proposed components matter under the authors' protocol. However, the paper's headline claim is about outperforming comparable MLLMs, and that claim rests on comparing a fine-tuned model with zero-shot baselines. The paper's own Table 6b supplies the decisive control: after removing Stage I and all attribute descriptions, instruction tuning alone reaches 76.13%, versus 76.84% for the full method. So the 10.89-point advertised edge over Idefics2 is not evidence for the attribute augmented alignment mechanism; it is mostly evidence that supervised fine-tuning on the evaluation datasets' training splits helps. The 'Finetune 0.03' row in Table 3a cannot rescue this because it uses a different data construction and demonstrates only that a model fine-tuned without closed-set multiple-choice data fails to answer in the expected format. The reader's named weakest_assumption about attribute-description faithfulness is a legitimate secondary concern, but the more load-bearing issue is the confounded comparison, which the reader's rationale also identifies. My recommendation is that the reader's REJECT verdict stands: the paper would need to add the same-setting baseline to the main table and reframe its claims before the central contribution could be assessed fairly.","tokens_in":20373,"tokens_out":5583,"duration_ms":52135,"concrete_test":"Add an 'Idefics2 + classification-centered instruction tuning (I.T. only)' row to the main comparison in Table 2, using the identical QLoRA recipe, learning rate, epochs, and the same open-set QA plus closed-set multiple-choice training data as Finedefics Stage II, but omitting Stage I attribute augmented contrastive learning and attribute descriptions. Recompute the six per-dataset margins and the average relative to Idefics2 and Qwen-VL-Chat. If the I.T.-only row averages approximately 76.13%, as Table 6b implies, and Finedefics's remaining edge is under one point, then the +10.89% headline is an artifact of comparing a fine-tuned model against zero-shot checkpoints, and the abstract's causal claim about attribute descriptions would need to be substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is that Finedefics 'outperforms existing MLLMs of comparable parameter sizes' by +10.89% over Idefics2 and +9.43% over Qwen-VL-Chat (Section 4.2, Table 2). That table compares Finedefics, which is fine-tuned on the training splits of the six evaluation datasets using two-stage attribute augmented alignment plus classification-centered instruction tuning, against released zero-shot checkpoints of other MLLMs. This is not an apples-to-apples comparison, and the paper itself provides the missing control: Table 6b reports that classification-centered instruction tuning alone (I.T. only, without Stage I pretraining and without attribute descriptions) reaches 76.13% average accuracy, only 0.71 points below Finedefics's 76.84%. Thus the proposed attribute binding mechanism, which the paper identifies as the cause of the improvement, contributes at most 0.71 points once the confound of supervised fine-tuning is removed. The 'Finetune 0.03' row in Table 3a is not an adequate control because it trains only on open-set QA data and omits the closed-set multiple-choice data used at evaluation, so it measures format-following failure rather than the method's effect. The advertised +10.89% margin over Idefics2 is therefore largely attributable to in-domain supervised fine-tuning, not to the paper's proposed attribute augmented alignment, making the central claim as stated unsupported by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies why multimodal large language models (MLLMs) underperform on fine-grained visual recognition (FGVR). It argues, on the basis of representation probing and visualization, that the bottleneck is not object information extraction or category knowledge, but misalignment between visual object representations and category name representations in the LLM's representation space. The proposed method, Finedefics, builds on Idefics2 and uses automatically constructed per-sample attribute descriptions as an intermediate binding: Stage I performs contrastive learning on object-attribute and attribute-category pairs with hard negatives, and Stage II performs classification-centered instruction tuning on open-set and closed-set FGVR data. The paper reports average gains of +10.89% over Idefics2 and +9.43% over Qwen-VL-Chat across six FGVR datasets, and supports the method with a large set of ablations, including attribute type, hard negatives, training stage, description quality, and generalization to other MLLMs.","tokens_in":20593,"tokens_out":8169,"duration_ms":74093,"significance":"The diagnostic setup is thoughtful: linear probing of object and category features, t-SNE visualizations, and the explicit decomposition into information extraction, knowledge reserve, and alignment are useful ways to study MLLM FGVR failure. The idea of using attribute descriptions as a binding intermediate, with hard negatives mined from similar categories, is sensible and the paper ships code and extensive ablations. However, the central quantitative claim is not supported by the reported comparisons, because the main table compares a model fine-tuned on the evaluation datasets' training splits against zero-shot baselines. The paper's own same-setting control reduces the attributed gain from +10.89 points to roughly +0.71 points, fundamentally changing the significance of the proposed mechanism.","major_comments":[{"comment":"The headline comparison is confounded by supervised fine-tuning. Finedefics is fine-tuned on the training splits of the six evaluation datasets using both Stage I contrastive learning and Stage II classification-centered instruction tuning, while all Table 2 baselines are evaluated zero-shot. The paper's own same-setting control, the \"I.T. only\" row in Table 6b, reaches 76.13% average accuracy, only 0.71 points below Finedefics's 76.84%. Thus the advertised +10.89% and +9.43% margins over Idefics2 and Qwen-VL-Chat do not isolate the effect of attribute augmented alignment; they are largely attributable to in-domain supervised fine-tuning. The central claim that \"benefiting from attribute augmented alignment, Finedefics outperforms existing MLLMs\" is therefore unsupported as stated. The same-setting control should appear in the main results, and the claims should be revised accordingly, or the comparison should include same-setting fine-tuned baselines for the other models.","section":"Section 4.2, Table 2 and Appendix Table 6b"},{"comment":"The \"Finetune\" control is not an adequate baseline for isolating the method's contribution. According to the text, this variant is fine-tuned solely on additional open-set FGVR data, so it omits the closed-set multiple-choice data used at evaluation. Its reported 0.03 accuracy therefore reflects a format-following failure rather than the effect of the proposed training paradigm. The relevant control is the classification-centered instruction-tuning-only row in Table 6b (76.13%), which should be reported and discussed in the main text alongside Table 3a.","section":"Section 4.3, Table 3a"},{"comment":"The attribute description construction pipeline is not verified for faithfulness. The method relies on GPT-4/LLaMA to select attribute keys and on BLIP-2/LLaVA to read attribute values, but no per-sample check is provided that the resulting descriptions are faithful to the image content or sufficiently discriminative across subordinate categories. The noisy-description ablation in Table 6c perturbs already-generated text; it does not test whether the upstream pipeline produces descriptions that actually describe the image. Since the paper's mechanism depends on attribute descriptions binding objects to categories, the paper should provide direct evidence of descriptor faithfulness, for example by human evaluation or by measuring how often the VQA-extracted attributes match ground-truth attributes, beyond the probing accuracy reported in Section A.1.","section":"Section 3.1 and Appendix Table 6c"}],"minor_comments":[{"comment":"The subsection heading contains a typo: \"Userful Attribute Discovery\" should be \"Useful Attribute Discovery\".","section":"Section 3.1"},{"comment":"The displayed formula for LCCC has a malformed \"− log 1P\" term; the intended numerator is not typeset correctly and should be fixed.","section":"Equation (7)"},{"comment":"The row label \"Finetune 0.03\" is unexplained. The numeric suffix appears to be an accuracy value from the table, but the label should be self-explanatory (e.g., \"Finetune (open-set only)\").","section":"Table 3a"},{"comment":"The numbers in Table 6e are confusing relative to Table 2: Finedefics is reported as 51.12 on Bird-200 here, while Table 2 reports 57.61 on the same dataset. The text states that this experiment is trained solely on Bird-200; this restriction should be stated clearly in the table caption and in the surrounding text.","section":"Table 6e"},{"comment":"The notation is inconsistent: after defining the category embedding sequence of length n, the text refers to \"the category name ˆci_m\"; the subscript should be consistent with the sequence length n.","section":"Section 2.3"},{"comment":"Several references are incomplete, including the Zhai et al. entry (missing year) and the Chen et al. entry in the Related Work section (missing year and venue).","section":"References"}],"recommendation":"reject","confidential_remarks":"For the editor: the decisive issue is that the paper's own same-setting control in Table 6b reduces the advertised +10.89-point improvement to approximately +0.71 points, so the main quantitative claim in the abstract and Section 4.2 is not supported by the reported evidence. The confound is not a presentation problem; it changes the interpretation of the paper's central contribution. A revision would need to substantially reframe the claims and add same-setting baselines, and even then the proposed attribute alignment mechanism would be shown to provide a very small gain over straightforward instruction tuning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the paper's diagnosis is the solid part, and its headline number is not. The three-capability analysis (extraction, knowledge, alignment) with the probing results in Table 1 makes a clean case that object features and category descriptions are linearly separable in Idefics2, while object-category alignment is the bottleneck. That's a useful empirical reframing, worth citing.\n\nThe method, Finedefics, uses per-sample LLM/VQA-generated attribute descriptions as a contrastive intermediate between visual objects and category names, with hard negatives from CLIP. That is a plausible recipe, and the code is public. The ablations on attribute types, hard negatives, and description quality are informative. The two-stage training loss is well-specified.\n\nThe problem is the central quantitative claim. Table 2 compares Finedefics, fine-tuned on the training splits of the six evaluation datasets, against zero-shot released checkpoints. The abstract's +10.89% over Idefics2 is therefore a fine-tuning effect, not evidence for the attribute-augmented alignment. The paper actually contains the missing control in Appendix Table 6b: instruction tuning only reaches 76.13% average, versus Finedefics at 76.84%. So the proposed alignment mechanism adds roughly 0.71 points, not 10.89. The 'Finetune 0.03' row in Table 3a is not an adequate control—it excludes the closed-set multiple-choice data used at evaluation, so it mostly shows a format-following failure. No error bars or significance tests are given anywhere.\n\nThe other soft spot is that the attribute descriptions are produced by a cascade of frozen models with no per-sample verification. The authors do probe description discriminability (68% linear probe on Pet-37) and test robustness to noisy descriptions, which mitigates the concern, but the main effect is tiny, so description fidelity matters more.\n\nOverall: the diagnosis is a genuine contribution and the method is not crazy, but the paper overstates its case. The same-setting ablation should be front and center, and the claimed margin should be corrected. As a reviewer I would ask for that reframe before accepting. It is worth sending to review because the diagnostics and the training recipe are useful, but the headline claim as written is not supported.","headline":"The diagnostic analysis is worth reading, but the headline 10.89-point gain is a zero-shot vs. fine-tuned comparison; the paper's own ablation narrows the proposed mechanism's contribution to about 0.7 points.","tokens_in":21188,"tokens_out":4903,"would_cite":true,"duration_ms":43378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that fine-grained recognition fails in MLLMs because object and category name representations are misaligned, and that attribute descriptions as a bridge lift average accuracy from 65.95 to 76.84 on six benchmarks.","keywords":["fine-grained visual recognition","multi-modal large language models","object-category alignment","attribute descriptions","contrastive learning","hard negatives","instruction tuning","Finedefics"],"falsifier":"Take the Finedefics training triples and swap each image's attribute description with the description of another image from the same subordinate category, while keeping object and category fixed; if accuracy on the six benchmarks stays near the reported 76.84, the object-to-attribute binding is not carrying discriminative information. A second check is to feed the constructed descriptions alone, with category names removed, to a text-only LLM and see whether it can already name the classes.","tokens_in":20104,"feed_emoji":"🎯","tokens_out":10102,"duration_ms":83292,"temperature":0.7,"pith_summary":"This paper asks why multimodal large language models misclassify subordinate categories such as dog breeds, bird species, and flower types, and locates the failure in a single part of the pipeline: the representation of a visual object and the representation of its category name are not aligned in the model's embedding space. The authors show that object information extraction and category knowledge are both adequate, so the bottleneck is object-category misalignment. They then build Finedefics, which generates a per-image attribute description in natural language and uses contrastive learning to bind the object representation to the description and the description to the category name, with similar-but-wrong categories serving as hard negatives. On six fine-grained recognition datasets, this raises the average accuracy of the base model Idefics2 from 65.95 to 76.84 and surpasses Qwen-VL-Chat by 9.43 points on average.","feed_headline":"Attribute text lifts MLLM fine-grained accuracy by 10+ points","feed_subtitle":"Using auto-written attribute descriptions as a bridge, Finedefics outperforms Idefics2 and Qwen-VL-Chat by about 10 points on average.","key_machinery":"The machinery is Attribute Augmented Alignment, a two-stage training recipe built on the base model Idefics2. For each training image, a pipeline of GPT-4 or LLaMA attribute discovery, BLIP-2 or LLaVA attribute extraction, and final LLM summarization produces a per-sample attribute description. The description is tokenized and passed through the LLM to obtain a global representation, taken from an appended EOS token, and this representation becomes the intermediate anchor in two contrastive losses: Object-Attribute Contrastive (OAC) and Attribute-Category Contrastive (ACC). A third Category-Category Contrastive (CCC) loss separates similar category names, and CLIP-mined hard negatives from the three most similar incorrect categories strengthen the negatives. Stage I optimizes these contrastive losses together with an attribute-description generation loss; Stage II fine-tunes on open-set QA and closed-set multiple-choice classification data.","core_discovery":"The central discovery is that MLLMs lose fine-grained classification accuracy not because they fail to extract discriminative visual details and not because they lack category knowledge, but because visual object embeddings and category-name embeddings sit in different regions of the LLM's representation space. Linear probing on the base model Idefics2 reaches about 95% accuracy for object features on Oxford-IIIT Pet-37, close to the vision-language model SigLIP, while probing on LLM-generated category descriptions reaches 92.51%, showing knowledge is present; category names alone, however, are poorly separated. Finedefics closes the gap by inserting an attribute description between object and category: it trains object-attribute and attribute-category contrastive losses simultaneously, adds a category-category contrastive loss to separate similar category names, and mines hard negatives with CLIP. After this alignment, a classification-centered instruction-tuning stage adapts the model to the task. The reported result is 76.84 average accuracy across six FGVR datasets, a gain of 10.89 points over Idefics2 and 9.43 points over Qwen-VL-Chat.","pith_inferences":["A natural next test is whether the same contrastive binding works when the attribute descriptions are produced by a single open-source model rather than a GPT-4/VQA cascade, which would remove the dependence on proprietary generation.","The failure analysis suggests that any mechanism that moves category names closer to object representations in the LLM embedding space, such as richer textual category definitions or retrieval of exemplar captions, could substitute for the attribute pipeline; a controlled comparison would reveal whether descriptions are uniquely effective.","A testable way to isolate the alignment contribution is to swap each image's attribute description with one from another image of the same class: if accuracy stays near 76.84, the object-to-attribute binding is not carrying the discriminative information.","The two-stage structure implies that continual fine-grained recognition, where new categories arrive after training, would require rerunning the contrastive alignment stage, a direction the paper names as future work."],"forward_implications":["Object-centric question answering and reasoning that depend on subordinate-level identification should inherit the accuracy gains, since the paper identifies FGVR as a prerequisite for those capabilities.","The alignment recipe transfers to other MLLMs: applying attribute augmented alignment to LLaVA 1.5 raises its average accuracy from 43.24 to 57.21 on the same six datasets.","Two-stage training matters: running alignment and instruction tuning together collapses performance to 25.42, so representation alignment must precede task tuning.","Replacing CLIP-mined hard negatives with simple negatives reduces the average gain from 76.84 to 74.26, so the choice of similar-but-wrong negatives is load-bearing for the method.","Per-sample attribute descriptions matter more than per-class tags: on Bird-200, Finedefics reaches 51.12, above a tag-based baseline of 50.48 and near the human-annotated upper bound of 52.52."],"supporting_citations":[{"why":"Provides the Idefics2 base model that Finedefics builds on and the main zero-shot comparison baseline.","marker":"Laurençon et al., 2024b"},{"why":"Supplies the feature-probing protocol and the finding that visually grounded language models are poor at image classification, which the paper's analysis tests and extends.","marker":"Zhang et al., 2024b"},{"why":"Defines the six-dataset multiple-choice evaluation setup and the earlier observation that classification data alone gives limited gains.","marker":"Geigle et al., 2024"},{"why":"Provides the FineR cascade that inspires the attribute description construction and the zero-shot attribute baseline compared in the appendix.","marker":"Liu et al., 2024c"},{"why":"Supplies the CLIP model used to mine hard-negative categories for contrastive learning.","marker":"Radford et al., 2021"},{"why":"Supplies the practice of appending an EOS token to obtain a global representation for contrastive learning.","marker":"Jiang et al., 2024"},{"why":"Supplies the intra-modal contrastive loss for hard-negative captions that is adapted into the Category-Category Contrastive loss.","marker":"Zhang et al., 2024a"},{"why":"Provides BLIP-2, the VQA model used to extract attribute values from each training image.","marker":"Li et al., 2023"}],"fun_headline_variants":["Attribute text bridges object-category gap for better MLLM recognition","Finedefics uses attribute contrastive learning to gain 10+ points","Attribute bridge fixes MLLM fine-grained misalignment","Finedefics: attribute descriptions close the object-category gap in MLLMs","New MLLM Finedefics gains 10+ points by bridging objects and categories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the auto-generated attribute descriptions accurately describe the individual image and separate subordinate categories well enough to serve as the binding link, and this assumption is not verified sample by sample.","fun_headline_variants_meta":{"raw":{"variants":["Attribute text bridges object-category gap for better MLLM recognition","Finedefics uses attribute contrastive learning to gain 10+ points","Attribute bridge fixes MLLM fine-grained misalignment","Finedefics: attribute descriptions close the object-category gap in MLLMs","New MLLM Finedefics gains 10+ points by bridging objects and categories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001039,"raw_usage":{"total_tokens":4401,"prompt_tokens":1007,"completion_tokens":3394,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":3295}},"tokens_in":623,"tokens_out":3394,"duration_ms":21296,"temperature":1.0,"reasoning_tokens":3295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:35:40.176040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Finedefics training triples and swap each image's attribute description with the description of another image from the same subordinate category, while keeping object and category fixed; if accuracy on the six benchmarks stays near the reported 76.84, the object-to-attribute binding is not carrying discriminative information. A second check is to feed the constructed descriptions alone, with category names removed, to a text-only LLM and see whether it can already name the classes.","supporting_citations":[],"review_version":1}