Pith. sign in

REVIEW 4 major objections 5 minor 65 references

Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper challenges the assumption that multimodal large language models are poor image classifiers, showing recent MLLMs reach or exceed CLIP-style baselines and tracing the gains to language-model knowledge and training-data diversity.

desk verdict A genuinely useful empirical update with a real attribution study, but the head-to-head with CLIP rests on mismatched protocols and only holds against the weakest baseline. read the letter →

arxiv 2412.16418 v1 pith:BFYZUWUX submitted 2024-12-21 cs.CV

Huan Liu , Lingyu Xiao , Jiangjiang Liu , Xiaofan Li , Ze Feng , Sen Yang , Jingdong Wang This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsimageclassificationCLIP-stylevision-languagemultiple-choiceevaluationzero-shotfine-grainedconceptualknowledgetransfertrainingdatadiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to overturn the claim that multimodal large language models (MLLMs) are inherently bad at image classification. Using ten conventional datasets, it reformulates classification as 26-option multiple-choice questions for MLLMs and as zero-shot image-text matching for CLIP-style models, then compares six MLLM families with six CLIP-style baselines. The newest MLLMs, led by Qwen2-VL, match or exceed the CLIP baselines on several datasets, with the largest gains in fine-grained categories. The authors attribute the improvement to the language model's conceptual knowledge, which transfers into better class discrimination, and to diverse training data that expose the model to more target concepts. If the finding holds, it redraws the practical boundary between generative multimodal models and specialized contrastive classifiers.

What carries the argument

The load-bearing machinery is a controlled four-way ablation, LLaVA-1.5 vs LLaVA-OV, in which the vision tower (CLIP ViT-L vs SigLIP-SO400M), the language model (Vicuna-7B vs Qwen2-7B), the training data (LLaVA-665K vs Mid-4M/SI-3.2M/OV-1.6M), and the training recipe (frozen vs tunable vision tower, low vs any-resolution, token-limit) are varied one at a time under a common two-stage training protocol. Secondary machinery is the 'privileged MLLM' experiment: a larger, well-grounded MLLM writes justifications for ImageNet images without naming the class, and the vanilla LLMs are asked to pick the class from the same 26-option multiple-choice form, isolating conceptual knowledge from visual processing. Together these two setups let the paper separate architecture, data, and recipe effects and then test the two proposed mechanisms.

What would settle it

Reproduce the evaluation with semantically close distractors (the top-25 class names chosen by embedding similarity) across all ten datasets instead of random distractors, and compare Qwen2-VL against SigLIP and the other CLIP baselines. The paper reports small drops for all models on ImageNet and Food101 in its appendix; the decisive question is whether Qwen2-VL's average margin over CLIP-style models persists or flips when every dataset uses these harder distractors. If the margin shrinks or reverses, the 'match or outperform' conclusion is an artifact of the random-distractor protocol.

Watch

Extended reading notes

Core claim

On a ten-dataset benchmark that casts image classification as 26-option multiple-choice questions for MLLMs and as zero-shot image-text matching for CLIP-style models, the paper finds that the newest MLLMs close most of the gap and sometimes overtake the contrastive baselines. Qwen2-VL reaches 95.0% average accuracy on general categories and 91.2% on fine-grained ones, exceeding the CLIP baseline by 2.0 and 5.6 percentage points, while LLaVA-OV gains +18.8 and +31.6 points over LLaVA-1.5, its predecessor in the same framework. Controlled ablations that swap vision towers (CLIP vs SigLIP), language models (Vicuna vs Qwen2), and training data (LLaVA-665K vs Mid-4M vs SI-3.2M) attribute most of the improvement to the language model and to diverse training data, not to tunable vision parameters or resolution strategy. The paper concludes that better conceptual knowledge in the LLM transfers to class discrimination, and that broader exposure to target concepts in supervised fine-tuning data boosts fine-grained accuracy.

Load-bearing premise

The argument stands on the assumption that a 26-option multiple-choice quiz with random distractors and a CLIP-style model's zero-shot ranking over the full class list measure the same image-classification ability.

Editorial extensions

If this is right

  • Under the paper's 26-choice protocol, Qwen2-VL reaches 95.0% accuracy on general classification and 91.2% on fine-grained classification, so the strongest MLLMs can be used as competitive general-purpose image classifiers.
  • Within the same LLaVA framework, moving from LLaVA-1.5 to LLaVA-OV adds 18.8 points (general) and 31.6 points (fine-grained), showing that most of the gap to CLIP-style models can be closed by updating the language model and training data.
  • Replacing Vicuna with Qwen2 while holding vision tower and data fixed raises both classification and MLLM benchmarks, indicating that LLM conceptual knowledge is a transferable driver of visual classification.
  • Adding 49K food-related samples drawn from a diverse 3.2M instruction set to LLaVA-1.5 fine-tuning raises Food101 accuracy by 6.2 points and also slightly improves non-food classes, so target-concept exposure in supervised fine-tuning data is a practical improvement lever.
  • Most MLLMs still trail CLIP-style models on fine-grained categories, so the 'match or outperform' result is driven by the strongest recent models rather than by the MLLM family as a whole.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the evaluation supplies the full class list with one correct option, 'classification' here is closer to closed-set answer selection than to open-set recognition; a stricter test would remove the option list and ask for free-form labels with exact-match scoring.
  • Editorial inference: the conceptual-knowledge experiment suggests much of the MLLM advantage may come from language priors rather than visual perception; a direct check would be to run the same multiple-choice questions on the image corrupted or blanked, and measure how much accuracy the language model alone retains.
  • Editorial inference: if target-concept exposure in supervised fine-tuning data is the operative mechanism, then deliberately re-balancing training data toward rare or fine-grained classes should improve MLLM classifiers on those classes without changing architecture; this is a practical, testable route suggested by the food-domain result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper revisits whether multimodal large language models (MLLMs) can perform conventional image classification, comparing 10 public MLLMs in the 4B-10B range against 6 CLIP-style vision-language models on 10 established datasets spanning general and fine-grained classification. For MLLMs, classification is reformulated as 26-option multiple-choice questions with randomly sampled distractors; for CLIP-style models, the standard zero-shot retrieval over the full class set is used. The authors report that recent MLLMs such as Qwen2-VL match or exceed some CLIP baselines on several datasets, and they attribute the gains to stronger LLM conceptual knowledge and diverse training data, supported by ablations over vision encoder/LLM combinations, training-data scales, and training-recipe choices, plus a text-only LLM knowledge probe.

Significance. If the central comparison were protocol-matched, this would be a useful empirical contribution: it updates the prior conclusion that MLLMs are uniformly poor classifiers, covers a broader model and dataset sweep than earlier work, ships ablations that isolate LLM and data effects, and openly separates leaked MLLM benchmark subsets in Table 5. The paper also includes a privileged-model text-only probe (Section 5.1) and an appendix with extra controls for option count and distractor difficulty. These strengths are real. However, the load-bearing comparison between MLLMs and CLIP-style models is run under asymmetric protocols, and the paper's own harder-distractor control does not restore comparability; as a result, the headline 'match or outperform' conclusion is currently protocol-dependent and needs either re-scoping or a matched evaluation before it can be accepted as stated.

major comments (4)
  1. [Section 3.2, Table 2, Appendix B.3]
  2. [Table 2, Section 3.4, Section 6]
  3. [Section 5.1, Table 8]
  4. [Section 4.3 and Table 5 note]
minor comments (5)
  1. [Section 5.1]
  2. [Table 9]
  3. [Section 5.1 and Table 8]
  4. [Appendix B.3]
  5. [Section 6, Limitations]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical evaluation against external benchmarks; protocol asymmetry is a validity concern, not a circular reduction.

full rationale

The paper is overwhelmingly an empirical benchmark study comparing public MLLMs and CLIP-style models on established classification datasets. No model parameter is fitted to the benchmark labels, and no headline accuracy is a rearranged training target, so there is no fitted-input-called-prediction pattern. The Section 3.2 protocol asymmetry (26-way random-distractor MCQ for MLLMs versus full-class zero-shot retrieval for CLIP) is a genuine construct-validity concern, and Appendix B.3 does not fully resolve it, but that is a benchmark-design / correctness risk, not a circular derivation. The Section 5.1 conceptual-knowledge probe is independent: it uses a privileged external MLLM to generate descriptions and tests vanilla LLMs, so the LLM-knowledge conclusion is not derived from the MLLMs being evaluated. The Section 5.2 food-ablation's conclusion is close to the intervention ('adding food-related data improves food-related accuracy'), which makes the explanatory label 'enhanced exposure of target concepts' somewhat self-evident, but it remains an empirical control rather than a fit renamed as a prediction. No load-bearing self-citation chain or imported uniqueness theorem appears; the comparisons stand on external benchmarks, public model checkpoints, and external protocol references. Therefore no specific circular reduction can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new entities or fitted parameters are introduced. The paper's conclusions rest on choices and assumptions about evaluation protocols and data cleanliness, listed above.

assumptions (4)
  • domain assumption A 26-option multiple-choice question with random distractors is a faithful and fair reformulation of image classification for MLLMs.
    Stated in Section 3.2; this assumption underpins the comparison with CLIP's full-class retrieval. Appendix B.2 shows accuracy varies strongly with option count, and B.3 shows ranking vs CLIP changes with distractor difficulty.
  • domain assumption The evaluated MLLMs' training data does not overlap with the test sets used for evaluation, except where noted.
    Sections 4.3 and 5.2 use SI-3.2M and OV-1.6M, which the authors do not screen for overlap with ImageNet, Food101, etc. The food-data augmentation filters against LLaVA-665K only, not against Food101 test images.
  • domain assumption A privileged MLLM's generated justifications are a valid proxy for the visual content, so that vanilla LLM accuracy on those justifications measures conceptual knowledge.
    Section 5.1 and Appendix C; the privileged model (InternVL2-Llama3-76B) might inject information or errors. The authors ablate one alternative privileged model but not human-written captions.
  • domain assumption GPT-4-based categorization of ImageNet classes into food/non-food is accurate enough for the subset analysis in Table 9.
    Appendix D; the class lists are provided and mostly reasonable, but the method is automated and could introduce misclassification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities." pith.science (2026). https://pith.science/paper/BFYZUWUX

@misc{pith2026241216418,
  author       = {Pith},
  title        = {Pith review of: Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BFYZUWUX}},
  note         = {Machine review of arXiv:2412.16418}
}
read the original abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), a variety of benchmarks have been introduced to evaluate their capabilities. While most evaluations have focused on complex tasks such as scientific comprehension and visual reasoning, little attention has been given to assessing their fundamental image classification abilities. In this paper, we address this gap by thoroughly revisiting the MLLMs with an in-depth analysis of image classification. Specifically, building on established datasets, we examine a broad spectrum of scenarios, from general classification tasks (e.g., ImageNet, ObjectNet) to more fine-grained categories such as bird and food classification. Our findings reveal that the most recent MLLMs can match or even outperform CLIP-style vision-language models on several datasets, challenging the previous assumption that MLLMs are bad at image classification \cite{VLMClassifier}. To understand the factors driving this improvement, we conduct an in-depth analysis of the network architecture, data selection, and training recipe used in public MLLMs. Our results attribute this success to advancements in language models and the diversity of training data sources. Based on these observations, we further analyze and attribute the potential reasons to conceptual knowledge transfer and enhanced exposure of target concepts, respectively. We hope our findings will offer valuable insights for future research on MLLMs and their evaluation in image classification tasks.

Figures

Figures reproduced from arXiv: 2412.16418 by the authors.

Figure 1
Figure 1. Specific and overall comparisons of image classifi￾cation. (a) LLaVA-OV [28] handles well on bad cases, such as “black-footed ferret” recognition, than previous LLaVA-1.5 [30]. (b) Recent proposed MLLMs obtain comparable or even better classification results on ObjectNet [4] dataset than SigLIP [54]. increasingly comprehensive evaluation of such complex abilities in existing benchmarks, a deep investigation into MLL… view at source ↗
Figure 2
Figure 2. Performance comparison of public MLLMs on various classification and MLLM benchmarks. Here, we compare the estab￾lished LLaVA-1.5 [30], Phi-3-Vision [1], recent LLaVA-OV [28] and Qwen2-VL [46] on a total of ten conventional image datasets (detailed in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 47 canonical work pages

  1. [55]

    Why are visually-grounded language models bad at image classi- fication? In NeurIPS, 2024

    Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually-grounded language models bad at image classi- fication? In NeurIPS, 2024. 1, 3, 5, 6, 7

    work page 2024

  2. [1]

    Phi-3 technical report: A highly capable language model locally on your phone, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. 2, 4, 5

    work page 2024

  3. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  4. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 4, 5

    arXiv 2023

  5. [4]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.NeurIPS, 32, 2019

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.NeurIPS, 32, 2019. 1, 2, 3, 4

    work page 2019

  6. [5]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In ECCV, pages 446–461. Springer, 2014. 2, 3, 4, 8, 12, 13

    work page 2014

  7. [6]

    Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024. 3, 4

    work page 2024

  8. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 4, 5, 13

    arXiv 2023

Show all 65 references
  1. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 4, 5, 13

    2024 arXiv

  2. [9]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In CVPR, pages 2818–2829, 2023. 4, 5

    2023

  3. [10]

    Coatnet: Marrying convolution and attention for all data sizes

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 34:3965–3977, 2021. 3

    2021

  4. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009. 1, 2, 3, 4, 6, 7, 8, 12, 13

    2009

  5. [12]

    BERT: pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. In NAACL-HLT, pages 4171–4186, 2019. 12

    2019

  6. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  7. [14]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR, 2024. 3, 4, 5

    2024

  8. [15]

    Eva-02: A visual representa- tion for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representa- tion for neon genesis. Image and Vision Computing , page 105171, 2024. 3, 4, 5

    2024

  9. [16]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In CVPRW, pages 178–178. IEEE, 2004. 1, 3, 4

    2004

  10. [17]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 3, 4

    2023 arXiv

  11. [18]

    Detect, describe, dis- criminate: Moving beyond vqa for mllm evaluation

    Manu Gaur, Makarand Tapaswi, et al. Detect, describe, dis- criminate: Moving beyond vqa for mllm evaluation. arXiv preprint arXiv:2409.15125, 2024. 3

    2024 arXiv

  12. [19]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 3

    2016

  13. [20]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709, 2019. 1, 4

    2019

  14. [21]

    Rwku: Benchmarking real-world knowledge unlearning for large language models

    Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hong- bang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. In NeurIPS, 2024. 1, 4

    2024

  15. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235–251. Springer,

  16. [23]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, pages 554–561, 2013. 1, 3, 4

    2013

  17. [24]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3

    2009

  18. [25]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. NeurIPS, 25, 2012. 3

    2012

  19. [26]

    Gradient-based learning applied to document recog- nition

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE , 86(11):2278–2324, 1998. 3 9

    1998

  20. [27]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. InCVPR, pages 13299–13308, 2024. 2, 3, 4, 12

    2024

  21. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2, 3, 4, 5, 6, 7, 8, 12

    2024 arXiv

  22. [29]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 4, 5

    2024

  23. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 1, 2, 3, 4, 5, 6, 7, 8

    2024

  24. [31]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233. Springer, 2024. 2, 3, 4, 12

    2024

  25. [32]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021. 3

    2021

  26. [33]

    Swin transformer v2: Scaling up capacity and resolution

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pages 12009–12019, 2022. 3

    2022

  27. [34]

    Deepseek-vl: Towards real-world vision- language understanding, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision- language understanding, 2024. 4, 5

    2024

  28. [35]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS,

  29. [36]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. InACL, pages 2263–2279, 2022. 1, 4

    2022

  30. [37]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021. 1, 4

    2021

  31. [38]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In In- dian Conference on Computer Vision, Graphics and Image Processing, 2008. 1, 3, 4, 8

    2008

  32. [39]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 1, 2, 3, ...

    2021

  33. [40]

    Do imagenet classifiers generalize to im- agenet? In ICML, pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In ICML, pages 5389–5400. PMLR, 2019. 3, 4, 8, 13

    2019

  34. [41]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 3

    2014 arXiv

  35. [42]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1–9, 2015. 3

    2015

  36. [43]

    Jn-logo: A logo database for aesthetic visual analysis

    Nannan Tian, Yuan Liu, and Ziruo Sun. Jn-logo: A logo database for aesthetic visual analysis. Electronics, 11(19): 3248, 2022. 3, 4

    2022

  37. [44]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2, 3, 4, 5

    2024 arXiv

  38. [45]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 2, 3, 4

    2011

  39. [46]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 4, 5, 12, 13

    2024 arXiv

  40. [47]

    Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval

    Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In CVPR, pages 2575–2584, 2020. 3, 4

    2020

  41. [48]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023. 12

    2023 arXiv

  42. [49]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 6, 7, 8, 12, 13

    2024 arXiv

  43. [50]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 2, 4, 5

    2024 arXiv

  44. [51]

    Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and bench- mark

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingn- ing Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and bench- mark. NeurIPS, 36, 2024. 12

    2024

  45. [52]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A...

    2024

  46. [53]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lu- cas Beyer. Scaling vision transformers. In CVPR, pages 12104–12113, 2022. 3 10

    2022

  47. [54]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. 1, 2, 4, 5, 6, 7, 13

    2023

  48. [56]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 36:46595–46623,

  49. [58]

    as the LLM, which has a limit of less than 4,096 to- kens, are unable to process such lengthy inputs. Besides, for MLLMs with extensive context processing capabilities, such as LLaV A-OV [28] and Qwen2-VL [46], which adopt Qwen2 [49] as the LLM, we observe that these models of...

  50. [59]

    Appearance: Describe the object’s shape, color, texture, and any distinct features

  51. [60]

    Context or environment: Where is this object commonly found or used? Describe its typical en- vironment or surroundings

  52. [61]

    Function or behavior: Explain how this object is typically used or behaves (if it’s an animal or de- vice, for example)

  53. [62]

    Any other relevant information: Mention any notable facts or characteristics that help define this object class. NOTE:

  54. [63]

    Keep the reason concise, with one or two sen- tences for each aspect

  55. [64]

    Do not use the category or synonym names of <class name> in your answer, instead using ‘it’

  56. [65]

    Just output the reason. D. Food-related subset selection In Section 5.2, for a more comprehensive analysis, we manually categorize the classes in ImageNet [11] and Im- ageNetv2 [40] into food-related and non-related subsets. Specifically, to determine the class attribution, we...

  57. [2023]

    Then, we perform an in-depth exploration of MLLM classification evaluation (delineated in Section B), including the formulation, influence of option numbers, etc

    6, 7, 8, 12, 13 11 Appendix In this appendix, we first discuss the potential negative so- cietal impacts (refer to Section A) that may arise in practi- cal scenarios. Then, we perform an in-depth exploration of MLLM classification evaluation (delineated in Section B), includin...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.