{"id":"d95b2a8d-05be-46f0-b52a-dbd4520f4c73","arxiv_id":"2412.16418","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Recent MLLMs such as Qwen2-VL can match or beat CLIP-style models on several image classification benchmarks, with gains driven mainly by stronger LLMs and more diverse training data.","lead":"This paper benchmarks recent multimodal large language models against CLIP-style models on ten image classification datasets, using multiple-choice questions for the MLLMs and zero-shot retrieval for CLIP. It finds the newest MLLMs, especially Qwen2-VL, match or beat CLIP on several datasets and attributes the gains mostly to the language model's conceptual knowledge and to diverse training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLLM-vs-CLIP comparison rests on unmatched protocols: 26-way random-distractor MCQ versus full-class retrieval; Appendix B.3 does not close the gap, so the headline claim is not yet established.","rationale":"I agree with the reader's core concern: the 26-way random-distractor MCQ in Section 3.2 is not directly comparable to CLIP's full-class zero-shot retrieval, and the paper's headline conclusion leans on that comparison. However, the reader's specific claim that Appendix B.3 shows a ranking flip between Qwen2-VL and SigLIP is not accurate. On ImageNet and Food101, SigLIP beats Qwen2-VL under both random and BERT-similar distractors, and the gap widens slightly with harder distractors. This factual correction strengthens rather than weakens the protocol concern: the paper's own robustness check does not rescue the comparison. The cleanest way to settle the issue is to run CLIP models under the same 26-option random-distractor protocol used for MLLMs, focusing on the datasets where MLLMs are claimed to beat SigLIP. I therefore keep the reader's conditional verdict unchanged: the benchmark and ablations are useful, but the central comparison with CLIP-style models needs this matched-protocol check before the 'match or outperform' claim is endorsed. Data contamination is a related secondary risk, but the matched-protocol test is the more direct way to assess the main claim.","tokens_in":19675,"tokens_out":9286,"duration_ms":77247,"concrete_test":"For each dataset in Table 2, especially ObjectNet, GLMK, and Logo where Qwen2-VL exceeds SigLIP, score every CLIP model under the authors' own 26-option protocol: construct the same A-Z option list (correct class plus 25 random distractors, using the same seed), encode each option with the CLIP text encoder, compute cosine similarity to the image, and select the highest-scoring option. Compare this restricted-choice accuracy to the full-class retrieval numbers in Table 2 and to Qwen2-VL's MCQ accuracy. If CLIP accuracy rises above Qwen2-VL on the datasets where the paper claims MLLM wins, the headline comparison is a protocol artifact; if it does not, the claim survives on those datasets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that Section 3.2's MCQ protocol and the CLIP zero-shot retrieval protocol measure the same classification ability. MLLMs see 26 options with 25 randomly chosen distractors, so a correct answer only requires ranking the true label above a sample of 25 classes; CLIP baselines must rank the true label above every class in the dataset, including near-synonyms. The chance baselines differ by over an order of magnitude (1/26 vs 1/1000 on ImageNet), and several Table 2 margins are small. Appendix B.3's harder-distractor test does not resolve the issue: it still restricts both model families to 26-way choice, covers only ImageNet and Food101, and on those datasets Qwen2-VL trails SigLIP under both random and BERT-similar distractors, with the gap widening (ImageNet: 97.4 to 95.1 for Qwen2-VL vs 97.7 to 96.2 for SigLIP; Food101: 96.2 to 95.7 vs 98.7 to 98.5). That appendix also omits the datasets where the paper claims MLLMs outperform SigLIP, namely ObjectNet, GLMK, and Logo. Thus the central 'match or outperform' comparison is protocol-dependent and not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits whether multimodal large language models (MLLMs) can perform conventional image classification, comparing 10 public MLLMs in the 4B-10B range against 6 CLIP-style vision-language models on 10 established datasets spanning general and fine-grained classification. For MLLMs, classification is reformulated as 26-option multiple-choice questions with randomly sampled distractors; for CLIP-style models, the standard zero-shot retrieval over the full class set is used. The authors report that recent MLLMs such as Qwen2-VL match or exceed some CLIP baselines on several datasets, and they attribute the gains to stronger LLM conceptual knowledge and diverse training data, supported by ablations over vision encoder/LLM combinations, training-data scales, and training-recipe choices, plus a text-only LLM knowledge probe.","tokens_in":19947,"tokens_out":6207,"duration_ms":53099,"significance":"If the central comparison were protocol-matched, this would be a useful empirical contribution: it updates the prior conclusion that MLLMs are uniformly poor classifiers, covers a broader model and dataset sweep than earlier work, ships ablations that isolate LLM and data effects, and openly separates leaked MLLM benchmark subsets in Table 5. The paper also includes a privileged-model text-only probe (Section 5.1) and an appendix with extra controls for option count and distractor difficulty. These strengths are real. However, the load-bearing comparison between MLLMs and CLIP-style models is run under asymmetric protocols, and the paper's own harder-distractor control does not restore comparability; as a result, the headline 'match or outperform' conclusion is currently protocol-dependent and needs either re-scoping or a matched evaluation before it can be accepted as stated.","major_comments":[{"comment":"","section":"Section 3.2, Table 2, Appendix B.3"},{"comment":"","section":"Table 2, Section 3.4, Section 6"},{"comment":"","section":"Section 5.1, Table 8"},{"comment":"","section":"Section 4.3 and Table 5 note"}],"minor_comments":[{"comment":"","section":"Section 5.1"},{"comment":"","section":"Table 9"},{"comment":"","section":"Section 5.1 and Table 8"},{"comment":"","section":"Appendix B.3"},{"comment":"","section":"Section 6, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a substantial empirical study and a plausible factor analysis, but the central claim is currently supported only under an asymmetric evaluation protocol. I would like to see either a matched-protocol comparison or a carefully re-scoped claim before publication. The contamination concern is also worth resolving, especially given the authors' own leakage note in Table 5."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful empirical update, and its headline is half-right. The paper shows that 2024 MLLMs (Qwen2-VL, LLaVA-OV) are no longer the poor classifiers that [55] found with older models, and its causal decomposition—LLM knowledge transfer plus training-data diversity—is the right kind of analysis to run. But the marquee comparison rests on mismatched protocols, and against the strongest CLIP-style baseline the 'caught up' claim does not survive.\n\nWhat is actually new: the empirical observation itself, which is absent from [55], and the attribution study. The architecture ablation (Tables 5–7) is careful: they hold the LLaVA pipeline fixed, swap vision tower and LLM, and show the LLM swap (Vicuna to Qwen2) is the big driver, +10.7 general and +14.4 fine-grained from L-CV to L-CQ. The privileged-MLLM probe in 5.1 is a clever, independent way to isolate conceptual knowledge—the generating model is separate from the LLMs being tested, and the Appendix C.1 ablation with a different privileged model confirms the ranking. The food-domain augmentation (5.2) shows a targeted SFT effect concentrated on food-related classes. They also flag that several MLLM benchmarks are leaked into LLaVA-OV training data (Table 5 footnote) and report weighted averages, and they replicate [55]'s ImageNet-in-SFT experiment and show it hurts. That is honest engagement.\n\nWhere it is soft, in order of severity. First, the protocol mismatch, which the stress-test note gets right. MLLMs face 26 options with 25 random distractors (chance 1/26); CLIP baselines rank against the full class set (chance roughly 1/1000 on ImageNet). That structurally favors the MLLM. The harder-distractor check in Appendix B.3 is a reasonable effort, but it covers only ImageNet and Food101, and Qwen2-VL trails SigLIP on both under both distractor schemes—so it does not rescue the comparison on the datasets where the MLLM wins (ObjectNet, GLMK, Logo). Second, the headline comparison is measured against OpenAI CLIP, a 2021 ViT-L baseline. Qwen2-VL beats it on 8 of 10 datasets, but against SigLIP, SigLIP leads on 7 of 10 and on both group averages (95.1 vs 95.0 general; 93.0 vs 91.2 fine-grained). The abstract's 'on several datasets' is accurate; the conclusion's 'caught up' is not. Third, no training/evaluation overlap check. For Food101 and Logo, both small and plausibly in web-scale training data, contamination is a credible alternative the paper never addresses.\n\nThe citation pattern is clean. The paper is for readers working on MLLM evaluation or the VLMClassifier line of work, and it deserves a serious referee; the referee should push for a matched-protocol comparison and a contamination audit before the strong claim appears in print. Conditional acceptance is the right posture.","headline":"A genuinely useful empirical update with a real attribution study, but the head-to-head with CLIP rests on mismatched protocols and only holds against the weakest baseline.","tokens_in":20503,"tokens_out":9253,"would_cite":true,"duration_ms":66297,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper challenges the assumption that multimodal large language models are poor image classifiers, showing recent MLLMs reach or exceed CLIP-style baselines and tracing the gains to language-model knowledge and training-data diversity.","keywords":["multimodal large language models","image classification","CLIP-style vision-language models","multiple-choice evaluation","zero-shot classification","fine-grained classification","conceptual knowledge transfer","training data diversity"],"falsifier":"Reproduce the evaluation with semantically close distractors (the top-25 class names chosen by embedding similarity) across all ten datasets instead of random distractors, and compare Qwen2-VL against SigLIP and the other CLIP baselines. The paper reports small drops for all models on ImageNet and Food101 in its appendix; the decisive question is whether Qwen2-VL's average margin over CLIP-style models persists or flips when every dataset uses these harder distractors. If the margin shrinks or reverses, the 'match or outperform' conclusion is an artifact of the random-distractor protocol.","tokens_in":19493,"feed_emoji":"🖼️","tokens_out":9373,"duration_ms":73576,"temperature":0.7,"pith_summary":"The paper sets out to overturn the claim that multimodal large language models (MLLMs) are inherently bad at image classification. Using ten conventional datasets, it reformulates classification as 26-option multiple-choice questions for MLLMs and as zero-shot image-text matching for CLIP-style models, then compares six MLLM families with six CLIP-style baselines. The newest MLLMs, led by Qwen2-VL, match or exceed the CLIP baselines on several datasets, with the largest gains in fine-grained categories. The authors attribute the improvement to the language model's conceptual knowledge, which transfers into better class discrimination, and to diverse training data that expose the model to more target concepts. If the finding holds, it redraws the practical boundary between generative multimodal models and specialized contrastive classifiers.","feed_headline":"Latest MLLMs match or beat CLIP-style classifiers","feed_subtitle":"A 26-choice quiz shows modern multimodal LLMs now rival specialized zero-shot models.","key_machinery":"The load-bearing machinery is a controlled four-way ablation, LLaVA-1.5 vs LLaVA-OV, in which the vision tower (CLIP ViT-L vs SigLIP-SO400M), the language model (Vicuna-7B vs Qwen2-7B), the training data (LLaVA-665K vs Mid-4M/SI-3.2M/OV-1.6M), and the training recipe (frozen vs tunable vision tower, low vs any-resolution, token-limit) are varied one at a time under a common two-stage training protocol. Secondary machinery is the 'privileged MLLM' experiment: a larger, well-grounded MLLM writes justifications for ImageNet images without naming the class, and the vanilla LLMs are asked to pick the class from the same 26-option multiple-choice form, isolating conceptual knowledge from visual processing. Together these two setups let the paper separate architecture, data, and recipe effects and then test the two proposed mechanisms.","core_discovery":"On a ten-dataset benchmark that casts image classification as 26-option multiple-choice questions for MLLMs and as zero-shot image-text matching for CLIP-style models, the paper finds that the newest MLLMs close most of the gap and sometimes overtake the contrastive baselines. Qwen2-VL reaches 95.0% average accuracy on general categories and 91.2% on fine-grained ones, exceeding the CLIP baseline by 2.0 and 5.6 percentage points, while LLaVA-OV gains +18.8 and +31.6 points over LLaVA-1.5, its predecessor in the same framework. Controlled ablations that swap vision towers (CLIP vs SigLIP), language models (Vicuna vs Qwen2), and training data (LLaVA-665K vs Mid-4M vs SI-3.2M) attribute most of the improvement to the language model and to diverse training data, not to tunable vision parameters or resolution strategy. The paper concludes that better conceptual knowledge in the LLM transfers to class discrimination, and that broader exposure to target concepts in supervised fine-tuning data boosts fine-grained accuracy.","pith_inferences":["Editorial inference: because the evaluation supplies the full class list with one correct option, 'classification' here is closer to closed-set answer selection than to open-set recognition; a stricter test would remove the option list and ask for free-form labels with exact-match scoring.","Editorial inference: the conceptual-knowledge experiment suggests much of the MLLM advantage may come from language priors rather than visual perception; a direct check would be to run the same multiple-choice questions on the image corrupted or blanked, and measure how much accuracy the language model alone retains.","Editorial inference: if target-concept exposure in supervised fine-tuning data is the operative mechanism, then deliberately re-balancing training data toward rare or fine-grained classes should improve MLLM classifiers on those classes without changing architecture; this is a practical, testable route suggested by the food-domain result."],"forward_implications":["Under the paper's 26-choice protocol, Qwen2-VL reaches 95.0% accuracy on general classification and 91.2% on fine-grained classification, so the strongest MLLMs can be used as competitive general-purpose image classifiers.","Within the same LLaVA framework, moving from LLaVA-1.5 to LLaVA-OV adds 18.8 points (general) and 31.6 points (fine-grained), showing that most of the gap to CLIP-style models can be closed by updating the language model and training data.","Replacing Vicuna with Qwen2 while holding vision tower and data fixed raises both classification and MLLM benchmarks, indicating that LLM conceptual knowledge is a transferable driver of visual classification.","Adding 49K food-related samples drawn from a diverse 3.2M instruction set to LLaVA-1.5 fine-tuning raises Food101 accuracy by 6.2 points and also slightly improves non-food classes, so target-concept exposure in supervised fine-tuning data is a practical improvement lever.","Most MLLMs still trail CLIP-style models on fine-grained categories, so the 'match or outperform' result is driven by the strongest recent models rather than by the MLLM family as a whole."],"supporting_citations":[{"why":"Supplies the prior conclusion that MLLMs are bad at image classification, which this paper's central claim challenges.","marker":"[55]"},{"why":"Defines the zero-shot matching protocol and the CLIP baseline that Qwen2-VL is compared against.","marker":"[39]"},{"why":"Provides the SigLIP baseline and the vision tower used in LLaVA-OV, central to the architecture ablation.","marker":"[54]"},{"why":"Provides the LLaVA-1.5 baseline model, training protocol, and 665K fine-tuning data used in ablations.","marker":"[30]"},{"why":"Provides LLaVA-OV, the improved model whose gains over LLaVA-1.5 drive the analysis, plus the Mid-4M/SI-3.2M/OV-1.6M training data.","marker":"[28]"},{"why":"Provides Qwen2-VL, the model whose 95.0% general and 91.2% fine-grained results anchor the headline claim.","marker":"[46]"},{"why":"Provides the Qwen2 language model; swapping it for Vicuna in the ablation isolates the LLM contribution.","marker":"[49]"}],"fun_headline_variants":["MLLMs now rival CLIP in image classification","New MLLMs beat CLIP on fine-grained tasks","Why MLLMs are no longer bad at classifying","Language model and data diversity drive MLLM classification","Qwen2-VL tops CLIP in zero-shot classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the assumption that a 26-option multiple-choice quiz with random distractors and a CLIP-style model's zero-shot ranking over the full class list measure the same image-classification ability.","fun_headline_variants_meta":{"raw":{"variants":["MLLMs now rival CLIP in image classification","New MLLMs beat CLIP on fine-grained tasks","Why MLLMs are no longer bad at classifying","Language model and data diversity drive MLLM classification","Qwen2-VL tops CLIP in zero-shot classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2500,"prompt_tokens":1019,"completion_tokens":1481,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1402}},"tokens_in":635,"tokens_out":1481,"duration_ms":10937,"temperature":1.0,"reasoning_tokens":1402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:36:00.864915+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the evaluation with semantically close distractors (the top-25 class names chosen by embedding similarity) across all ten datasets instead of random distractors, and compare Qwen2-VL against SigLIP and the other CLIP baselines. The paper reports small drops for all models on ImageNet and Food101 in its appendix; the decisive question is whether Qwen2-VL's average margin over CLIP-style models persists or flips when every dataset uses these harder distractors. If the margin shrinks or reverses, the 'match or outperform' conclusion is an artifact of the random-distractor protocol.","supporting_citations":[{"cited_title":"Why are visually-grounded language models bad at image classi- fication? In NeurIPS, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the prior conclusion that MLLMs are bad at image classification, which this paper's central claim challenges."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Defines the zero-shot matching protocol and the CLIP baseline that Qwen2-VL is compared against."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the SigLIP baseline and the vision tower used in LLaVA-OV, central to the architecture ablation."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-1.5 baseline model, training protocol, and 665K fine-tuning data used in ablations."}],"review_version":1}