Pith. sign in

REVIEW 2 major objections 4 minor 50 references

NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that current multimodal large language models, including top commercial models, consistently fail to identify objects whose color or shape has been changed to implausible values, and that the failure is tied to…

desk verdict Solid new benchmark for attribute-modified object recognition, but the claim that larger LLMs weaken vision encoders is not isolated from confounds. read the letter →

arxiv 2411.17794 v1 pith:WI3ZMMUV submitted 2024-11-26 cs.CV

Jiaxuan Li , Junwen Mo , MinhDuc Vo , Akihiro Sugimoto , Hideki Nakayama This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsattribute-modifiedobjectrecognitionbeyond-commonsensereasoningbenchmarkvisionencodermodelscalingvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer whether multimodal large language models can recognize objects when familiar attributes, such as color, are changed to beyond-commonsense values, like a blue mango. It introduces NEMO, a benchmark pairing 900 original fruit images with attribute-modified versions and 2,700 questions in open, multiple-choice, and unsolvable formats, and evaluates 26 open-sourced and commercial models. The central finding is that every tested model loses accuracy on modified objects, with commercial models dropping by about 22 points and open-sourced models dropping more. The paper further argues that stronger vision encoders help but that MLLMs still underperform their own standalone vision encoders, and that scaling up the language model can actually weaken the vision encoder during fine-tuning. This matters because real-world objects routinely deviate from typical appearances, so robust recognition requires handling such deviations rather than relying on memorized or common visual patterns.

What carries the argument

The load-bearing design is a paired benchmark: each of 45 fruit categories contributes original internet images and DALL-E-generated attribute-modified images, so every model can be scored on the same object with and without an abnormal attribute, isolating the effect of attribute change. Three question formats test recognition from free recall, forced choice, and rejection of all wrong options, with choices circularly shifted to control for answer-position bias. The diagnostic that carries the scaling claim is image-to-image matching: after separating the fine-tuned vision encoder from the LLM, top-5 retrieval accuracy on Original-to-Original and Attribute-modified-to-Original pairs is used as a measure of encoder quality after fine-tuning.

What would settle it

Fine-tune the same pre-trained vision encoder with the same data and projector while varying only the language model's size, then rerun NEMO and the Original-to-Original and Attribute-modified-to-Original retrieval tests; if retrieval accuracy and attribute-modified accuracy do not fall as the LLM grows, the claim that larger LLMs weaken vision encoders is refuted.

Watch

Extended reading notes

Core claim

NEMO reveals a general limitation in current MLLMs: when an object's attribute is changed to a novel color (or shape), models that correctly named the original object often misname the modified one, even when humans find the modified object easily identifiable. Comparing multiple-choice accuracy of MLLMs with the embedding similarity of their own pre-trained vision encoders shows that the standalone encoder frequently outperforms the full model built on it, indicating that vision-language alignment fails to exploit the encoder's capabilities. Image-to-image matching experiments with the fine-tuned encoders extracted from MLLMs show that models trained with larger LLMs, such as LLaVA-NeXT-72B and 110B, retrieve worse than their smaller counterparts, leading the authors to conclude that larger LLMs can weaken vision encoders during fine-tuning. The same performance pattern holds for shape-modified fruits and color-modified animals, suggesting the limitation is general to attribute deviation, not specific to fruits or colors.

Load-bearing premise

The retrieval experiment assumes that top-5 image-to-image matching accuracy of the fine-tuned vision encoder measures the encoder's standalone quality, and that differences between model families are caused by LLM size rather than by training data, projector architecture, or fine-tuning recipe.

Editorial extensions

If this is right

  • NEMO can serve as a testbed for future MLLMs, scoring them on original versus attribute-modified pairs to separate improvements in perception, alignment, and language priors.
  • Standalone vision encoders set a bound that current MLLMs fail to reach, so better results should be attainable by improving the vision-language interface rather than only enlarging the language model.
  • Scaling language-model parameters can actively hurt attribute-modified recognition, so model families should report per-size results instead of assuming larger is better.
  • The high variance in unsolvable questions under choice shifting indicates that option-position preferences contaminate multiple-choice evaluation, and circular shifting should be standard practice.
  • The replicated drop on shape-modified fruits and color-modified animals implies the limitation is about attribute deviation in general, not about fruit or color categories in particular.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct next experiment would score each MLLM against its own vision encoder per object category, turning the aggregate gap into a per-object diagnostic of where the vision-language interface loses information.
  • The benchmark could separate perceptual failure from language-prior failure: if a model names a blue object correctly when the category is supplied as text but fails from vision alone, the bottleneck is visual grounding rather than object knowledge.
  • The choice-shift instability suggests that some reported accuracy gains on multiple-choice vision benchmarks may be inflated by position preference, so re-scoring existing benchmarks with circular shifts could change model rankings.
  • A controlled fine-tuning study that fixes vision encoder, data, projector, and recipe while varying only LLM size would directly test the claim that larger LLMs weaken vision encoders, and would also reveal at what parameter scale the degradation begins.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces NEMO, a benchmark for evaluating multimodal large language models (MLLMs) on recognizing objects whose attributes (primarily color) have been modified beyond commonsense. NEMO contains 900 images (450 original fruit images collected from the internet and 450 DALL-E-generated attribute-modified versions) and 2,700 VQA questions spanning open, multiple-choice, and unsolvable formats. The authors evaluate 26 open-sourced and commercial MLLMs along with human and random baselines. They report a consistent accuracy drop on attribute-modified objects across all models, with open-sourced models lagging behind commercial ones. They further analyze the performance drop by comparing MLLMs with their standalone vision encoders, examining scaling trends with model size, and measuring image-to-image retrieval using fine-tuned encoder representations. The paper also reports model preferences for common objects and extensions to shape-modified fruits and color-modified animals.

Significance. The benchmark is a potentially useful addition to the beyond-commonsense evaluation literature. Its construction is careful in several respects: human validation of generated images, circular shifting of multiple-choice options to mitigate position bias, a human baseline with high performance, and released code. The central empirical finding—that all evaluated MLLMs, including GPT-4o, show a substantial performance drop on attribute-modified objects—is well supported by Table 2 and by the extension experiments in the supplementary material. The paper is less convincing in its causal claims about vision-encoder and LLM-size effects, which are based on comparisons across model families that differ in multiple architectural and training dimensions. If the authors revise the language to match the correlational evidence, the benchmark and its main findings would be a solid contribution.

major comments (2)
  1. [Sec. 5.2, Question 3, Fig. 5 (SM C.4)] The claim that "larger LLMs often weaken vision encoders during fine-tuning" is not established by the evidence presented. The comparison in Fig. 5 does not isolate LLM size: LLaVA-NeXT-8B uses LLaMA3-8B, while LLaVA-NeXT-72B and 110B use Qwen1.5-72B/110B, and these variants differ in projector, training data, and recipe. The InternVL2 comparisons similarly conflate LLM family and vision encoder (InternViT-300M-448 for models up to 8B versus InternViT-6B-448 for 26B and larger). In addition, SM C.4 does not specify whether the image representation is taken before or after the projector, so the retrieval accuracy may include projection and language-model effects. The observed degradation of LLaVA-NeXT-110B relative to LLaVA-NeXT-8B could therefore be caused by any of these confounds. Please either supply controlled experiments (e.g., the same vision encoder and training recipe with only the LLM size varied) or rephrase Answer 3 and the abstract's scaling claim as a correlation without causal attribution.
  2. [Sec. 5.2, Question 2, Fig. 4 (Table 2)] The model-size scaling analysis also confounds LLM size with vision encoder identity. Within the LLaVA-NeXT series, the 32B variant uses SigLIP-ViT-SO-14-384 while the 8B, 72B, and 110B variants use CLIP-ViT-L-14-336; within InternVL2, the 1B–8B models use InternViT-300M-448 and the 26B–76B models use InternViT-6B-448-V1.5. Thus the observed non-monotonic accuracy pattern—for example, the drop from LLaVA-NeXT-Qwen-32B to LLaVA-NeXT-72B—could be driven by the change in vision encoder rather than by LLM size. Please either control for the vision encoder while varying LLM size, or soften Answer 2 to say that the examined models do not show a consistent monotonic improvement with size, without implying that LLM size is the causal factor.
minor comments (4)
  1. [SM C.2] The human evaluation relies on only two annotators, and no inter-annotator agreement is reported. Please add agreement statistics such as Cohen's kappa, or include more annotators, to support the reliability of the human baseline.
  2. [SM C.3] The use of GPT-4o as an automatic judge for sentence-producing models could introduce a systematic bias in the reported open-question accuracies. Please validate a random subset of GPT-4o judgments against human judgments and report the agreement.
  3. [SM B.4] Distractor options are constructed from misclassifications by a subset of the evaluated models. Since the same distractors are used for all models, the absolute difficulty for those particular models may be influenced. Please report the sensitivity of the results to an alternative distractor set (e.g., human-generated distractors).
  4. [Table 2 footnote and SM text] The footnote phrase "original marks the second best" appears to be a LaTeX formatting artifact and should read "underlined" or "italicized." In addition, fix the typo "mamually" in SM B.4, the duplicated "as" in SM F.2 ("serve as as a testbed"), and "comparision" in Section 1.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity in multiple-choice distractor construction; the central benchmark findings remain independently supported.

  1. fitted input called prediction [Supplementary Material B.4, 'Questions Construction']
    "To construct distractor choices for the multiple-choice questions and unsolvable questions, We collect misclassified objects from the open questions’s responses by GPT-4o [28], LLaV A-NeXT-8B [19], InstructBLIP-7B [9] , InstructBLIP-13B [9], and MiniGPT-v2-7B [6]. We select Top-3 most frequent misclassified objects as the distractor options."

    The distractor options are fitted to the open-question misclassifications of five of the models that are then evaluated on these same multiple-choice/unsolvable questions. For those models, the option set is not independent of their own error patterns: their most frequent errors are deliberately inserted as decoys, so the resulting multiple-choice accuracy partially re-measures the fitted error distribution rather than providing a fresh, unbiased probe of recognition ability. The loop is localized, however, because open-question results, which contain no constructed options, and human baselines independently reproduce the original-versus-attribute-modified performance gap. The headline finding therefore does not reduce to this construction.

full rationale

The central claims are empirical benchmark measurements: accuracy on original versus attribute-modified objects across 26 models, with human baselines and open-question checks. These measurements are not derived from fitted parameters or self-citations; no load-bearing uniqueness theorem is imported, and the open-question and unsolvable results provide independent evidence for the main performance gap. The one self-referential construction is the distractor selection in SM B.4, where decoy options for multiple-choice and unsolvable questions are taken from the misclassifications of five evaluated models; this makes multiple-choice accuracy for those models partially circular, but it is not the load-bearing evidence for the core conclusion. The 'larger LLMs weaken vision encoders' claim in Sec. 5.2 Question 3 is confounded by LLM family, projector, training data, and recipe differences, but confounding is a validity or correctness risk rather than circularity, so it is not scored here. Overall circularity is minor and localized.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several domain assumptions about the generation and evaluation pipeline, but no numeric parameters are fitted to data and no new entities are postulated.

assumptions (4)
  • domain assumption DALL-E generated images preserve the fruit category while modifying color, and manual checks ensure recognizability.
    Sec. 3.2 describes generating images with prompts like 'An image of {color} {object}' and manual checks to ensure the object category is preserved. The human baseline (89% on modified) supports this, but it is an unproven assumption about the generation and filtering process.
  • domain assumption The 45 fruit categories and 10 colors per object are representative of beyond-commonsense scenarios.
    Sec. 3.1 selects common fruits from Fruit360; colors are provided by GPT-4o. This is a design choice that may not generalize to other object types or attributes, as the authors acknowledge in Sec. 6.
  • domain assumption Image-to-image matching accuracy of a fine-tuned vision encoder reflects encoder quality independently of the LLM and projector.
    Sec. 5.2 Question 3 uses O2O/A2O matching on the vision encoder's average embeddings after fine-tuning. If the projector or training recipe confounds this measure, the conclusion that larger LLMs weaken encoders would be unsupported.
  • domain assumption GPT-4o can reliably judge whether a model response matches the ground truth.
    SM C.3 uses GPT-4o to evaluate sentences from models like InstructBLIP and Cambrian-1. No human verification of the judge's accuracy is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?." pith.science (2026). https://pith.science/paper/WI3ZMMUV

@misc{pith2026241117794,
  author       = {Pith},
  title        = {Pith review of: NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WI3ZMMUV}},
  note         = {Machine review of arXiv:2411.17794}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have made notable advances in visual understanding, yet their abilities to recognize objects modified by specific attributes remain an open question. To address this, we explore MLLMs' reasoning capabilities in object recognition, ranging from commonsense to beyond-commonsense scenarios. We introduce a novel benchmark, NEMO, which comprises 900 images of origiNal fruits and their corresponding attributE-MOdified ones; along with a set of 2,700 questions including open-, multiple-choice-, unsolvable types. We assess 26 recent open-sourced and commercial models using our benchmark. The findings highlight pronounced performance gaps in recognizing objects in NEMO and reveal distinct answer preferences across different models. Although stronger vision encoders improve performance, MLLMs still lag behind standalone vision encoders. Interestingly, scaling up the model size does not consistently yield better outcomes, as deeper analysis reveals that larger LLMs can weaken vision encoders during fine-tuning. These insights shed light on critical limitations in current MLLMs and suggest potential pathways toward developing more versatile and resilient multimodal models.

Figures

Figures reproduced from arXiv: 2411.17794 by the authors.

Figure 1
Figure 1. Challenges in consistently identifying original and attribute-modified objects are illustrated in (a) and (b): while LLaVA-NeXT [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Samples from the NEMO benchmark, showcasing original and attribute-modified (A-Modif) objects with three types of question [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Average accuracy scores of various MLLMs (LLaVA [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Top-5 accuracy of Original-to-Original and Attribute [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of model performance on different fruits in original objects (left) and attribute-modified objects (right) for open [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 34 canonical work pages

  1. [1]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 12

    work page 2022

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  3. [3]

    Breaking common sense: WHOOPS! A vision- and-language benchmark of synthetic and compositional im- ages

    Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Lud- wig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: WHOOPS! A vision- and-language benchmark of synthetic and compositional im- ages. In Proc. IEEE/CVF International Conference on Com- puter Vision (ICCV), 2023. 2, 3

    work page 2023

  4. [4]

    Visual riddles: A common- sense and world knowledge challenge for large vision and language models

    Nitzan Bitton-Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. Visual riddles: A common- sense and world knowledge challenge for large vision and language models. arXiv preprint arXiv:2407.19474 , 2024. 2, 3

    arXiv 2024

  5. [5]

    InternLM2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiax- ing Li, Jingwen Li, Linyang Li,...

    arXiv 2024

  6. [6]

    MiniGPT-v2: Large language model as a unified inter- face for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: Large language model as a unified inter- face for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023. 4, 5, 12, 13, 19

    arXiv

  7. [7]

    How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821 , 2024. 2, 4, 5, 6, 12, 19, 20

    arXiv 2024

  8. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023. 12

    work page 2023

Show all 50 references
  1. [9]

    InstructBLIP: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 4...

    2023

  2. [10]

    ImageNet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical im- age database. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 8, 16

    2009

  3. [11]

    Scaling laws of synthetic images for model training

    Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training ... for now. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 8

    2024

  4. [12]

    EV A: Exploring the limits of masked visual represen- tation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A: Exploring the limits of masked visual represen- tation learning at scale. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 12

    2023

  5. [13]

    MME: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2, 3

    2023 arXiv

  6. [14]

    Gemini., 2023

    Google. Gemini., 2023. 2, 4, 5, 19, 20

    2023

  7. [15]

    Hal- lusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hal- lusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2024

  8. [16]

    SEED-Bench-2: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED-Bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023. 2, 3

    2023 arXiv

  9. [17]

    SEED-Bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED-Bench: Benchmarking multimodal large language models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3

    2024

  10. [18]

    Naturalbench: Eval- uating vision-language models on natural adversarial sam- ples

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, 9 Graham Neubig, and Deva Ramanan. Naturalbench: Eval- uating vision-language models on natural adversarial sam- ples. In Neural Information Processing Systems (N...

    2024

  11. [19]

    LLaV A-NeXT: Stronger LLMs supercharge multimodal capabilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. LLaV A-NeXT: Stronger LLMs supercharge multimodal capabilities in the wild, 2024. 2, 4, 5, 6, 12, 19, 20

    2024

  12. [20]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. International Conference on Machine Learning (ICML) ,

  13. [21]

    FoodieQA: A multimodal dataset for fine-grained understanding of chinese food culture

    Wenyan Li, Xinyu Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, An- ders Søgaard, et al. FoodieQA: A multimodal dataset for fine-grained understanding of chinese food culture. Proc. Conference on Empirical Methods in Natural Language ...

    2024

  14. [22]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. 2

    2023

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. 2, 3, 12

    2023

  16. [24]

    MMBench: Is your multi-modal model an all-around player? In Proc

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? In Proc. European Conference on Com- puter Vision (ECCV), 2024. 2, 3, 4

    2024

  17. [25]

    From here to human-level AI

    John McCarthy. From here to human-level AI. Artificial Intelligence, 171(18):1174–1182, 2007. 2

    2007

  18. [26]

    Unsolvable problem detection: Eval- uating trustworthiness of vision language models

    Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Eval- uating trustworthiness of vision language models. arXiv preprint arXiv:2403.20331, 2024. 3, 4

    2024 arXiv

  19. [27]

    Position: Levels of AGI for operationalizing progress on the path to AGI

    Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Position: Levels of AGI for operationalizing progress on the path to AGI. InProc. In- ternational Conference on Machine Learning (ICML), 2024. 2

    2024

  20. [28]

    GPT-4o, 2024

    OpenAI. GPT-4o, 2024. 2, 3, 4, 5, 12, 13, 19, 20

    2024

  21. [29]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proc. International Conference on Machine Le...

    2021

  22. [30]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proc. International Conference on Machine Learning (ICML), 2021. 2, 3, 7, 12, 16, 17

    2021

  23. [31]

    EV A-CLIP: Improved training techniques for CLIP at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: Improved training techniques for CLIP at scale. arXiv preprint arXiv:2303.15389, 2023. 6

    2023 arXiv

  24. [32]

    Link- context learning for multimodal LLMs

    Yan Tai, Weichen Fan, Zhao Zhang, and Ziwei Liu. Link- context learning for multimodal LLMs. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3

    2024

  25. [33]

    Label Studio: Data labeling soft- ware, 2020-2022

    Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling soft- ware, 2020-2022. Open source software available from https://github.com/heartexlabs/label-studio. 13

    2020

  26. [34]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian- 1: A fully open, vision-centric exploration of multimodal LLMs. In A...

    2024

  27. [35]

    Eyes wide shut? Exploring the vi- sual shortcomings of multimodal LLMs

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? Exploring the vi- sual shortcomings of multimodal LLMs. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  28. [36]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2, 12

    2023 arXiv

  29. [37]

    Recent advancements in fruit detection and classifica- tion using deep learning techniques

    Chiagoziem C Ukwuoma, Qin Zhiguang, Md Belal Bin Heyat, Liaqat Ali, Zahra Almaspoor, and Happy N Mon- day. Recent advancements in fruit detection and classifica- tion using deep learning techniques. Mathematical Problems in Engineering, 2022(1):9210947, 2022. 3

    2022

  30. [38]

    Le, Thang Luong, and Golnaz Ghiasi

    Zhecan Wang, Garrett Bingham, Adams Wei Yu, Quoc V . Le, Thang Luong, and Golnaz Ghiasi. HaloQuest: A visual hal- lucination dataset for advancing multimodal reasoning. In Proc. European Conference on Computer Vision (ECCV) ,

  31. [39]

    xGen- MM (formerly BLIP-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin C...

    2024

  32. [40]

    mPLUG-Owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUG-Owl: Modularization empowers large language models w...

    2023 arXiv

  33. [41]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. 4, 5, 12, 19 10

    2024 arXiv

  34. [42]

    MMMU: A massive multi-discipline multimodal un- derstanding and reasoning benchmark for Expert AGI

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weim- ing Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A...

    2024

  35. [43]

    MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024. 2, 3

    2024 arXiv

  36. [44]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 6, 12

    2023

  37. [45]

    A VIBench: To- wards evaluating the robustness of large vision-language model on adversarial visual-instructions

    Hao Zhang, Wenqi Shao, Hong Liu, Yongqiang Ma, Ping Luo, Yu Qiao, and Kaipeng Zhang. A VIBench: To- wards evaluating the robustness of large vision-language model on adversarial visual-instructions. arXiv preprint arXiv:2403.09346, 2024. 2

    2024 arXiv

  38. [46]

    InternLM-XComposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  39. [47]

    On evaluating ad- versarial robustness of large vision-language models

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Cheung, and Min Lin. On evaluating ad- versarial robustness of large vision-language models. In Ad- vances in Neural Information Processing Systems (NeurIPS),

  40. [48]

    ROME: Evaluating pre-trained vision-language models on reasoning beyond visual common sense

    Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, and Jing Jiang. ROME: Evaluating pre-trained vision-language models on reasoning beyond visual common sense. In Findings of the Association for Computational Lin- guistics: EMNLP, 2023. 2, 3

    2023

  41. [49]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In In- ternational Conference on Learning Representations (ICLR),

  42. [2024]

    dall-e-3

    4, 5, 12, 13, 19 11 Supplementary Material for NEMO : Can Multimodal LLMs Identify Attribute-Modified Objects? This supplementary material complements our paper with the following sections: • We discuss further related work on multimodal large lan- guage models (MLLMs) (see Se...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.