{"id":"5634122e-9fb3-4914-8d0c-26cc7694b205","arxiv_id":"2411.17794","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark shows that multimodal LLMs, including GPT-4o, consistently fail to recognize objects when their colors are modified, and that larger language models can degrade the vision encoder's performance during fine-tuning.","lead":"The paper introduces NEMO, a benchmark of 900 fruit images, original and color-modified, with 2,700 questions to test whether multimodal LLMs can recognize objects whose attributes are changed. It evaluates 26 models and finds that all of them, including GPT-4o, perform markedly worse on modified objects, and that standalone vision encoders often outperform the full MLLMs that use them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that larger LLMs weaken vision encoders during fine-tuning is not isolated from confounds: Fig. 5 compares models differing in LLM family, projector, data, and training recipe, not only in LLM size.","rationale":"The reader's weakest assumption is the same as my primary concern: the image-to-image matching experiment in Sec. 5.2 Q3 attributes differences in retrieval accuracy to LLM size without controlling for other model components and training variables. I agree with that assessment. The benchmark itself has independent support: human accuracy is high (98.0 original, 89.1 modified), all 26 models show a consistent drop on attribute-modified objects, and the synthetic-original control in SM E.2 indicates that the drop is not merely a DALL-E artifact. The GPT-4o judge issue is real but only affects sentence-producing models and does not drive the core multiple-choice/unsolvable pattern. The distractor-selection bias is secondary. The load-bearing weakness is the scaling/vision-encoder degradation claim, which is the paper's most distinctive contribution and the least controlled part of the argument. Because Fig. 5 is an observational comparison across heterogeneous model families, the causal phrasing in Answer 3 overreaches. This does not invalidate the benchmark or the main performance-gap finding; it requires either a more cautious correlational phrasing or a controlled experiment. Therefore no change to the reader's CONDITIONAL verdict is needed.","tokens_in":29189,"tokens_out":6731,"duration_ms":60611,"concrete_test":"Run a controlled fine-tuning experiment with a fixed vision encoder (e.g., CLIP-ViT-L-14-336), fixed projector, fixed data, and fixed recipe, varying only the LLM backbone size (e.g., LLaMA3-1B, 3B, 8B, 70B). Measure Top-5 O2O/A2O retrieval on NEMO using the fine-tuned vision encoder features. If retrieval does not monotonically degrade with LLM size, Answer 3 is unsupported. Additionally, report whether features are extracted before or after the projector; if after, the experiment cannot separate encoder weakening from projector/LLM interaction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most novel and load-bearing claim is Answer 3 in Sec. 5.2: 'Stronger vision encoders improve MLLM performance, but larger LLMs often weaken vision encoders during fine-tuning.' This is asserted as a causal effect of LLM size on encoder quality. The evidence is Fig. 5, which reports Top-5 Original-to-Original and Attribute-modified-to-Original retrieval using image representations from MLLMs' fine-tuned vision encoders. That comparison does not isolate LLM size. LLaVA-NeXT-8B uses CLIP-ViT-L-14-336 with LLaMA3-8B, whereas LLaVA-NeXT-72B and 110B use the same vision encoder but Qwen1.5-72B/110B: different LLM family, tokenizer, projector, fine-tuning data, and training recipe. InternVL2 comparisons also vary across LLM families and recipes. SM C.4 does not specify whether the 'trained vision encoder' representation is read before or after the projector, so the retrieval metric may also absorb model-specific projection effects. The observed degradation of LLaVA-NeXT-110B relative to 8B could be caused by any of these confounds. Since the abstract and conclusion present this as a key insight, the central claim is not established by the released evidence. The benchmark's main performance-gap result remains well supported; the unsupported part is specifically the causal scaling/encoder-degradation claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NEMO, a benchmark for evaluating multimodal large language models (MLLMs) on recognizing objects whose attributes (primarily color) have been modified beyond commonsense. NEMO contains 900 images (450 original fruit images collected from the internet and 450 DALL-E-generated attribute-modified versions) and 2,700 VQA questions spanning open, multiple-choice, and unsolvable formats. The authors evaluate 26 open-sourced and commercial MLLMs along with human and random baselines. They report a consistent accuracy drop on attribute-modified objects across all models, with open-sourced models lagging behind commercial ones. They further analyze the performance drop by comparing MLLMs with their standalone vision encoders, examining scaling trends with model size, and measuring image-to-image retrieval using fine-tuned encoder representations. The paper also reports model preferences for common objects and extensions to shape-modified fruits and color-modified animals.","tokens_in":29420,"tokens_out":6343,"duration_ms":53716,"significance":"The benchmark is a potentially useful addition to the beyond-commonsense evaluation literature. Its construction is careful in several respects: human validation of generated images, circular shifting of multiple-choice options to mitigate position bias, a human baseline with high performance, and released code. The central empirical finding—that all evaluated MLLMs, including GPT-4o, show a substantial performance drop on attribute-modified objects—is well supported by Table 2 and by the extension experiments in the supplementary material. The paper is less convincing in its causal claims about vision-encoder and LLM-size effects, which are based on comparisons across model families that differ in multiple architectural and training dimensions. If the authors revise the language to match the correlational evidence, the benchmark and its main findings would be a solid contribution.","major_comments":[{"comment":"The claim that \"larger LLMs often weaken vision encoders during fine-tuning\" is not established by the evidence presented. The comparison in Fig. 5 does not isolate LLM size: LLaVA-NeXT-8B uses LLaMA3-8B, while LLaVA-NeXT-72B and 110B use Qwen1.5-72B/110B, and these variants differ in projector, training data, and recipe. The InternVL2 comparisons similarly conflate LLM family and vision encoder (InternViT-300M-448 for models up to 8B versus InternViT-6B-448 for 26B and larger). In addition, SM C.4 does not specify whether the image representation is taken before or after the projector, so the retrieval accuracy may include projection and language-model effects. The observed degradation of LLaVA-NeXT-110B relative to LLaVA-NeXT-8B could therefore be caused by any of these confounds. Please either supply controlled experiments (e.g., the same vision encoder and training recipe with only the LLM size varied) or rephrase Answer 3 and the abstract's scaling claim as a correlation without causal attribution.","section":"Sec. 5.2, Question 3, Fig. 5 (SM C.4)"},{"comment":"The model-size scaling analysis also confounds LLM size with vision encoder identity. Within the LLaVA-NeXT series, the 32B variant uses SigLIP-ViT-SO-14-384 while the 8B, 72B, and 110B variants use CLIP-ViT-L-14-336; within InternVL2, the 1B–8B models use InternViT-300M-448 and the 26B–76B models use InternViT-6B-448-V1.5. Thus the observed non-monotonic accuracy pattern—for example, the drop from LLaVA-NeXT-Qwen-32B to LLaVA-NeXT-72B—could be driven by the change in vision encoder rather than by LLM size. Please either control for the vision encoder while varying LLM size, or soften Answer 2 to say that the examined models do not show a consistent monotonic improvement with size, without implying that LLM size is the causal factor.","section":"Sec. 5.2, Question 2, Fig. 4 (Table 2)"}],"minor_comments":[{"comment":"The human evaluation relies on only two annotators, and no inter-annotator agreement is reported. Please add agreement statistics such as Cohen's kappa, or include more annotators, to support the reliability of the human baseline.","section":"SM C.2"},{"comment":"The use of GPT-4o as an automatic judge for sentence-producing models could introduce a systematic bias in the reported open-question accuracies. Please validate a random subset of GPT-4o judgments against human judgments and report the agreement.","section":"SM C.3"},{"comment":"Distractor options are constructed from misclassifications by a subset of the evaluated models. Since the same distractors are used for all models, the absolute difficulty for those particular models may be influenced. Please report the sensitivity of the results to an alternative distractor set (e.g., human-generated distractors).","section":"SM B.4"},{"comment":"The footnote phrase \"original marks the second best\" appears to be a LaTeX formatting artifact and should read \"underlined\" or \"italicized.\" In addition, fix the typo \"mamually\" in SM B.4, the duplicated \"as\" in SM F.2 (\"serve as as a testbed\"), and \"comparision\" in Section 1.","section":"Table 2 footnote and SM text"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the benchmark itself is carefully constructed and the main performance-gap finding is well supported. The paper's causal claims about LLM size weakening vision encoders are the main risk; they are presented in the abstract and conclusions but are based on confounded comparisons. I recommend requiring the authors to either add controlled experiments or change the claims to correlational statements. With that revision, the benchmark would be a solid contribution suitable for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on NEMO. The benchmark itself is a real contribution: it pairs original and attribute-modified images of the same 45 fruits, includes open, multiple-choice, and unsolvable questions, filters generated images by human validation, and evaluates 26 models with circular-shifted choices and a human baseline. The central result—that every model, including GPT-4o, drops sharply on attribute-modified objects while humans stay near ceiling—is solid, and the controlled checks (synthetic original images, shape-modified fruits, color-modified animals) make it a reusable testbed.\n\nThe problems are concentrated in the 'Answer 3' analysis (Sec. 5.2, Fig. 5). The paper claims larger LLMs weaken vision encoders during fine-tuning, but the models being compared differ in LLM family, tokenizer, projector, training data, and fine-tuning recipe. LLaVA-NeXT-8B uses LLaMA3 with CLIP-ViT-L; LLaVA-NeXT-72B/110B use the same encoder but Qwen1.5 with different data and recipe. InternVL2 comparisons have the same problem. The retrieval metric in Fig. 5 also doesn't specify whether representations are taken before or after the projector (SM C.4), so the effect could be a projection artifact. As stated in the abstract and conclusion, this claim is not established by the released evidence. The performance-gap result doesn't depend on it, so the fix is to reframe Answer 3 as a correlational observation or run a controlled study with fixed LLM family/data.\n\nTwo smaller issues. First, distractors in MC and unsolvable questions are drawn from misclassifications by a subset of the evaluated models (SM B.4). That's a mild circularity, but the open-question results show the same drop, so it doesn't drive the main finding. Second, using GPT-4o as a judge for sentence-producing models (InstructBLIP, MiniGPT4, Cambrian) is unvalidated; it probably shifts numbers slightly without changing the qualitative ranking. Also, the human baseline uses two annotators, which is small, but the near-ceiling and consistent scores make it adequate.\n\nOverall: the benchmark deserves a serious referee. I'd push for a revision that either removes the causal scaling claim or backs it with controlled experiments. People working on MLLM robustness and evaluation will want to know about this.","headline":"Solid new benchmark for attribute-modified object recognition, but the claim that larger LLMs weaken vision encoders is not isolated from confounds.","tokens_in":30011,"tokens_out":4213,"would_cite":true,"duration_ms":31408,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current multimodal large language models, including top commercial models, consistently fail to identify objects whose color or shape has been changed to implausible values, and that the failure is tied to…","keywords":["multimodal large language models","attribute-modified object recognition","beyond-commonsense reasoning","benchmark","vision encoder","model scaling","visual question answering","object recognition"],"falsifier":"Fine-tune the same pre-trained vision encoder with the same data and projector while varying only the language model's size, then rerun NEMO and the Original-to-Original and Attribute-modified-to-Original retrieval tests; if retrieval accuracy and attribute-modified accuracy do not fall as the LLM grows, the claim that larger LLMs weaken vision encoders is refuted.","tokens_in":28949,"feed_emoji":"🥭","tokens_out":5192,"duration_ms":46406,"temperature":0.7,"pith_summary":"The paper sets out to answer whether multimodal large language models can recognize objects when familiar attributes, such as color, are changed to beyond-commonsense values, like a blue mango. It introduces NEMO, a benchmark pairing 900 original fruit images with attribute-modified versions and 2,700 questions in open, multiple-choice, and unsolvable formats, and evaluates 26 open-sourced and commercial models. The central finding is that every tested model loses accuracy on modified objects, with commercial models dropping by about 22 points and open-sourced models dropping more. The paper further argues that stronger vision encoders help but that MLLMs still underperform their own standalone vision encoders, and that scaling up the language model can actually weaken the vision encoder during fine-tuning. This matters because real-world objects routinely deviate from typical appearances, so robust recognition requires handling such deviations rather than relying on memorized or common visual patterns.","feed_headline":"Blue mangoes break multimodal LLMs' object recognition","feed_subtitle":"All 26 tested models lose accuracy on color-modified fruit, and bigger LLMs can degrade the vision encoder.","key_machinery":"The load-bearing design is a paired benchmark: each of 45 fruit categories contributes original internet images and DALL-E-generated attribute-modified images, so every model can be scored on the same object with and without an abnormal attribute, isolating the effect of attribute change. Three question formats test recognition from free recall, forced choice, and rejection of all wrong options, with choices circularly shifted to control for answer-position bias. The diagnostic that carries the scaling claim is image-to-image matching: after separating the fine-tuned vision encoder from the LLM, top-5 retrieval accuracy on Original-to-Original and Attribute-modified-to-Original pairs is used as a measure of encoder quality after fine-tuning.","core_discovery":"NEMO reveals a general limitation in current MLLMs: when an object's attribute is changed to a novel color (or shape), models that correctly named the original object often misname the modified one, even when humans find the modified object easily identifiable. Comparing multiple-choice accuracy of MLLMs with the embedding similarity of their own pre-trained vision encoders shows that the standalone encoder frequently outperforms the full model built on it, indicating that vision-language alignment fails to exploit the encoder's capabilities. Image-to-image matching experiments with the fine-tuned encoders extracted from MLLMs show that models trained with larger LLMs, such as LLaVA-NeXT-72B and 110B, retrieve worse than their smaller counterparts, leading the authors to conclude that larger LLMs can weaken vision encoders during fine-tuning. The same performance pattern holds for shape-modified fruits and color-modified animals, suggesting the limitation is general to attribute deviation, not specific to fruits or colors.","pith_inferences":["A direct next experiment would score each MLLM against its own vision encoder per object category, turning the aggregate gap into a per-object diagnostic of where the vision-language interface loses information.","The benchmark could separate perceptual failure from language-prior failure: if a model names a blue object correctly when the category is supplied as text but fails from vision alone, the bottleneck is visual grounding rather than object knowledge.","The choice-shift instability suggests that some reported accuracy gains on multiple-choice vision benchmarks may be inflated by position preference, so re-scoring existing benchmarks with circular shifts could change model rankings.","A controlled fine-tuning study that fixes vision encoder, data, projector, and recipe while varying only LLM size would directly test the claim that larger LLMs weaken vision encoders, and would also reveal at what parameter scale the degradation begins."],"forward_implications":["NEMO can serve as a testbed for future MLLMs, scoring them on original versus attribute-modified pairs to separate improvements in perception, alignment, and language priors.","Standalone vision encoders set a bound that current MLLMs fail to reach, so better results should be attainable by improving the vision-language interface rather than only enlarging the language model.","Scaling language-model parameters can actively hurt attribute-modified recognition, so model families should report per-size results instead of assuming larger is better.","The high variance in unsolvable questions under choice shifting indicates that option-position preferences contaminate multiple-choice evaluation, and circular shifting should be standard practice.","The replicated drop on shape-modified fruits and color-modified animals implies the limitation is about attribute deviation in general, not about fruit or color categories in particular."],"supporting_citations":[{"why":"Supplies DALL-E3 generations of attribute-modified fruit and animal images used throughout the benchmark.","marker":"[30]"},{"why":"Provides the fruit categories from which the 45 NEMO object classes are selected.","marker":"[37]"},{"why":"Serves as the strongest commercial baseline, proposes novel colors for modifications, and evaluates sentence-style outputs from other models.","marker":"[28]"},{"why":"Provides the LLaVA-NeXT model family used to trace accuracy against model size and vision encoder type.","marker":"[19]"},{"why":"Provides the InternVL2 model family used in the scaling analysis and fine-tuned-encoder retrieval experiments.","marker":"[7]"},{"why":"Gives the Cambrian-1 models, whose vision-centric designs still underperform their own standalone encoders in the comparison.","marker":"[34]"},{"why":"Defines the CLIP vision encoders whose standalone embedding accuracy is compared with MLLM accuracy in the probing analysis.","marker":"[29]"},{"why":"Motivates the unsolvable-question format used to test whether models can reject all wrong options.","marker":"[26]"},{"why":"Motivates the circular shifting strategy that controls for answer-position preference in multiple-choice questions.","marker":"[24]"}],"fun_headline_variants":["MLLMs miss color-modified fruits, even with bigger models","NEMO benchmark: Larger LLMs weaken vision encoders in MLLMs","Color-shifted apples stump multimodal LLMs, study finds","Attribute-modified objects reveal MLLM vision blind spots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The retrieval experiment assumes that top-5 image-to-image matching accuracy of the fine-tuned vision encoder measures the encoder's standalone quality, and that differences between model families are caused by LLM size rather than by training data, projector architecture, or fine-tuning recipe.","fun_headline_variants_meta":{"raw":{"variants":["MLLMs miss color-modified fruits, even with bigger models","NEMO benchmark: Larger LLMs weaken vision encoders in MLLMs","Color-shifted apples stump multimodal LLMs, study finds","Attribute-modified objects reveal MLLM vision blind spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1418,"prompt_tokens":935,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":551,"tokens_out":483,"duration_ms":4589,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:53:13.417150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same pre-trained vision encoder with the same data and projector while varying only the language model's size, then rerun NEMO and the Original-to-Original and Attribute-modified-to-Original retrieval tests; if retrieval accuracy and attribute-modified accuracy do not fall as the LLM grows, the claim that larger LLMs weaken vision encoders is refuted.","supporting_citations":[{"cited_title":"Zero-shot text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies DALL-E3 generations of attribute-modified fruit and animal images used throughout the benchmark."},{"cited_title":"Recent advancements in fruit detection and classifica- tion using deep learning techniques","cited_arxiv_id":null,"evidence_quote":"Provides the fruit categories from which the 45 NEMO object classes are selected."},{"cited_title":"GPT-4o, 2024","cited_arxiv_id":null,"evidence_quote":"Serves as the strongest commercial baseline, proposes novel colors for modifications, and evaluates sentence-style outputs from other models."},{"cited_title":"LLaV A-NeXT: Stronger LLMs supercharge multimodal capabilities in the wild, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-NeXT model family used to trace accuracy against model size and vision encoder type."},{"cited_title":"Cambrian- 1: A fully open, vision-centric exploration of multimodal LLMs","cited_arxiv_id":null,"evidence_quote":"Gives the Cambrian-1 models, whose vision-centric designs still underperform their own standalone encoders in the comparison."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Defines the CLIP vision encoders whose standalone embedding accuracy is compared with MLLM accuracy in the probing analysis."},{"cited_title":"MMBench: Is your multi-modal model an all-around player? In Proc","cited_arxiv_id":null,"evidence_quote":"Motivates the circular shifting strategy that controls for answer-position preference in multiple-choice questions."}],"review_version":1}