Pith. sign in

REVIEW 3 major objections 6 minor 36 references

MANBench: Is Your Multimodal Model Smarter than Human?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new bilingual benchmark finds the best multimodal models still score below average humans.

desk verdict A useful new bilingual benchmark, but the headline human-vs-model comparison is undercut by a duplicated English/Chinese human baseline. read the letter →

arxiv 2506.11080 v1 pith:P6OAI6UG submitted 2025-06-04 cs.CL

Han Zhou , Qitong Xu , Yiheng Dong , Xin Yang This is my paper
classification cs.CL
keywords MANBenchmultimodallargelanguagemodelshumanperformancebaselinecross-modalreasoningvisualquestionansweringbilingualbenchmarkevaluationimage-textintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MANBench is a bilingual (English and Chinese) benchmark of 1,314 multiple-choice questions across nine tasks, built to measure multimodal reasoning rather than knowledge retrieval. The paper's central claim is that today's best multimodal models have not reached average human capability on this kind of integrated image-text reasoning: the strongest model evaluated scores 59.97% overall, below the 62.26% average of 575 human participants, and models exceed humans only on Knowledge and Text-image Understanding. The benchmark is designed so every question requires joint use of text and image, and it separates knowledge-based from non-knowledge-based items, so the gap points at cross-modal integration—tasks like Image Consistency, Transmorphic Understanding, and Multi-image Understanding—rather than at missing facts. If the comparison holds, it gives concrete targets for where model development needs to move.

What carries the argument

The central object is MANBench itself: 1,314 four-option multiple-choice questions, bilingual in English and Chinese, organized into nine tasks (Text-image Understanding, Image Consistency, Numbers Pattern, Spatial Imagination, Text Locating, Multi-image Understanding, Knowledge, Puzzles, Transmorphic Understanding). Two design choices carry the argument: questions are split into knowledge-based and non-knowledge-based sets, and every item is constructed so that the image and the text must be interpreted jointly, which blocks retrieval-only shortcuts. The comparison mechanism is a human norm: 575 participants each answered one ten-subset slice of roughly 131 questions, with accuracy per question averaged to task and overall scores, giving a human-average baseline (62.26%) and a best-human baseline (90.87%) against which fourteen current MLLMs are scored.

What would settle it

Run a larger, multilingual human norm on the full MANBench set with no skipping and with incentive to answer every item; if the resulting average human score drops to or below the best MLLM score of 59.97%, then the paper's central comparison would be overturned.

Watch

Extended reading notes

Core claim

The paper introduces MANBench as a Multimodal Ability Norms Benchmark and uses it to norm human and machine multimodal ability on the same questions. On its English subset, average human accuracy is 62.26% while the best MLLM, GPT-o1, reaches 59.97%; the ceiling among the evaluated models is still below the human average, and the best individual human in the study scored 90.87%. The paper's explanation is task-specific: machines outperform humans on Knowledge and Text-image Understanding, but fall to roughly random-chance levels on Puzzles and underperform humans on Transmorphic Understanding, Image Consistency, and Multi-image Understanding. Those are exactly the tasks that demand going beyond superficial alignment, and the paper reads the pattern as evidence that MLLMs have strong knowledge and retrieval but weaker integrated cross-modal reasoning.

Load-bearing premise

The headline gap depends on treating the average score of 575 Chinese participants, each given only one 131-question subset and allowed to skip questions they found too hard, as a fair measure of general human multimodal ability; if that sample is not representative, the conclusion that models fall short of humans is not established.

Editorial extensions

If this is right

  • A model cannot claim human-level multimodal ability on MANBench by scoring just over 60% overall; it must clear the 62.26% average human norm and ideally approach the 90.87% best-human ceiling.
  • Improvement on Image Consistency, Transmorphic Understanding, and Multi-image Understanding carries most of the headroom, since these are the tasks where models fall furthest below humans.
  • Because knowledge tasks are separated out, gains from larger pretraining corpora or better fact recall will show up mostly in Knowledge and Text-image Understanding; the harder cross-modal tasks require architectural or training changes.
  • The bilingual design implies the gap is not an artifact of one language: similar patterns appear in both English and Chinese subsets.
  • Puzzles and Spatial Imagination remain open even for humans at 38.8% and 54.6% average accuracy, so they can serve as frontier stress tests where no system is yet strong.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not show that MLLMs can never match humans; its claim is about current models. A straightforward extension would track whether future models cross the 62.26% norm, and on which tasks the crossing happens first.
  • The human baseline's reliance on one Chinese participant pool with permission to skip means the 62.26% figure may be a lower or upper bound for general human ability; replicating the norm with multilingual, more diverse samples would test how robust the gap is.
  • If the benchmark's premise is right, then common claims of superhuman AI on narrow benchmarks should be read narrowly: integrated image-text reasoning is a distinct capability that has not yet saturated.
  • The skip-permitted protocol and 20-minute per-subset design could be re-run with forced answering to produce a stricter norm; the comparison to MLLMs, which are forced to answer, would then be cleaner.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MANBench, a bilingual (English/Chinese) multiple-choice benchmark of 1,314 questions across nine multimodal reasoning tasks, and evaluates 14 open- and closed-source MLLMs under standard VLMEvalKit settings. The authors also report a human study with 575 Chinese participants, each answering one subtest of about 131 questions, yielding an average human score of 62.26. They report that the best MLLM (GPT-o1, at 59.97 on the English subset) falls below this human average, while MLLMs exceed average human performance only on Knowledge and Text-image Understanding. Additional analyses cover age effects, response times, an image-masked ablation for GPT-4o, and a 'best human' score of 90.87.

Significance. The benchmark itself is a substantial contribution: it is bilingual, explicitly separates knowledge-based from non-knowledge-based reasoning, spans a wide difficulty range, and ships with code and data. The MLLM evaluations use standard, reproducible settings and cover a wide range of prominent models, and the task-level findings (e.g., MLLMs near chance on Puzzles, above human on Knowledge) are informative for the multimodal reasoning community. However, the central claim that state-of-the-art MLLMs fall short of average human performance rests on a human baseline that is not established for the English subset and is presented without any measure of uncertainty. The benchmark and its MLLM results have value, but the human comparison, which is the paper's headline, requires substantial rework.

major comments (3)
  1. [Table 1 vs. Table 5; Section 4.1.1] The 'Human (Average)' row in Table 1 (English subset) is numerically identical to the same row in Table 5 (Chinese subset) — every task score and the overall 62.26 match exactly — while every MLLM row differs between the two tables. Section 4.1.1 states that all 575 participants were Chinese, and Appendix A.5 states that the best human was tested on the Chinese subset. The manuscript provides no evidence of a separate English human experiment. Comparing English-subset MLLM scores against a human baseline collected on the Chinese version is invalid unless the experiment was actually run in English. This is load-bearing: the 2.29-point gap between GPT-o1 (59.97) and the human average (62.26) is the basis of the paper's central conclusion. The duplicated row must be corrected, or the headline claim must be restricted to the Chinese subset.
  2. [Section 4.3.2 and Appendix A.4] The human average is reported without any measure of uncertainty. The formula Pi = (1/ni) * sum_j pij does not define pij for skipped questions, and participants were explicitly allowed to skip questions (Appendix A.4). No skip counts are reported, so it is unclear whether skipped questions are excluded, scored as incorrect, or scored as correct. With roughly 57 participants per question (575 participants across 10 subsets), task-level accuracy estimates carry nontrivial standard errors, and the 2.29-point overall gap between GPT-o1 and the human average may not be statistically reliable. The paper should report skipped-question rates, per-question response counts, confidence intervals or significance tests, and a precise definition of pij. Without these, the conclusion that all MLLMs fall short of average human performance is not established.
  3. [Appendix A.5, Table 3, Table 1] The 'Human (Best)' overall score of 90.87 in Table 1 is computed by weighting the per-category accuracies of Table 3 by the full task sizes (149, 158, 134, etc.), even though the best human answered only 1021 of the 1314 questions (77.7%), with per-category denominators smaller than the task sizes (e.g., 100/149 for Text-image Understanding). This extrapolation overstates the upper-bound human performance. The correct comparison would be either (a) compute the best human's accuracy over the 1021 questions actually answered and then score all MLLMs on that same subset, or (b) treat all unanswered questions as incorrect and score both humans and MLLMs on the full 1314 questions. As reported, the row is not comparable to the MLLM overall scores, and the conclusion that models remain 'far from the upper bounds of human capability' is not supported by the presented computation.
minor comments (6)
  1. [Introduction vs. Section 4.2] The Introduction says 'Twelve MLLMs' were evaluated, while Section 4.2 says '14 recent MLLMs' and lists 6 closed-source plus 8 open-source models; the count should be made consistent.
  2. [Conclusion] The sentence 'even state-of-the-art models ... achieve less than 60% accuracy on MANBench' is false for the Chinese subset, where GPT-o1 and InternVL2.5-78B-MPO reach 60.12; the claim should be qualified by language.
  3. [Figure 6 and Section 4.4] The claim that 'children under the age of fourteen perform comparably to adults' is based on very small age groups (the 0-9 group appears to have only 2 participants in Figure 5, and the 10-19 group is pooled over ten years); the figure lacks error bars and per-group sample sizes.
  4. [Section 4.4 (Ablation)] The masked-image ablation says GPT-4o's performance in the non-Image-Consistency tasks 'showed no statistically significant deviation from random chance', but no statistical test is reported; either provide the test or rephrase as an informal observation.
  5. [Table 1 and Table 5] The 'Random' baseline row is not explained; because option counts likely vary (e.g., 4-option vs. 5-option questions), the expected random accuracy per task should be stated, and the column ordering of the two tables (Numbers Pattern vs. Image Consistency) should be aligned to avoid obscuring duplicate rows.
  6. [Appendix F (Datasheet)] The datasheet says the dataset 'will be released on Feb 10th, 2025' and provides an anonymous 4openScience URL, while the abstract and Section 1 provide a GitHub URL; these should be updated consistently for the final submission.

Circularity Check

1 steps flagged · score 4.0 of 10

The English subset's human-baseline row is, by construction, the Chinese-subset human measurement (Tables 1 and 5 are numerically identical for human rows while every MLLM row differs), so the human side of the headline MLLM-vs-human comparison is circular; the MLLM scores themselves are independently measured.

  1. fitted input called prediction [Table 1 (Section 4.3, English subset) vs Table 5 (Appendix C.1, Chinese subset); Appendix A.5]
    "Table 1: 'Human (Average) 62.26 76.46 76.88 81.55 54.56 64.82 71.69 37.05 38.83 60.90' (English subset). Table 5: 'Human (Average) 62.26 76.46 81.55 76.88 54.56 64.82 71.69 37.05 38.83 60.90' (Chinese subset). Appendix A.5: 'They was tested on the Chinese subset of the MANBench dataset.'"

    Table 1 (captioned English subset) and Table 5 (captioned Chinese subset) carry numerically identical Human (Average) and Human (Best) rows once the reordered task columns are aligned (62.26 overall; Text-image Understanding 76.46, Image Consistency 81.55, Numbers Pattern 76.88 in both), while every MLLM row differs between the tables. Random rows can lawfully coincide because they depend only on option counts, but human rows are empirical: two separate 575-person experiments on two translated versions cannot yield nine equal task accuracies to two decimals. Section 4.1.1 describes exactly one human experiment, and Appendix A.5 states the best-human sample, whose row also appears in both tables, was tested on the Chinese subset.

full rationale

MANBench is a measurement paper: the 14 MLLM accuracies are direct evaluations under the VLMEvalKit protocol (temperature 0, retry 10), and the 575-participant human study is a genuine empirical sample. The paper contains no fitted parameters, no load-bearing self-citations (the reference list includes no prior works by these authors), no imported uniqueness theorems, no ansatz smuggled in via citation, and no renaming of a known result; the MLLM-side findings therefore have independent content. The one constructed reduction is the human baseline: Table 1 (English subset) and Table 5 (Chinese subset) carry numerically identical Human (Average) and Human (Best) rows once the reordered task columns are aligned, while every MLLM row differs between languages. Because Section 4.1.1 describes a single human experiment with 575 Chinese participants, and Appendix A.5 states that the best-human sample 'was tested on the Chinese subset,' the English table's human baseline is, by construction, a Chinese-subset measurement rather than an English-subset measurement. The headline gap (GPT-o1 59.97 vs human 62.26) is therefore computed against a human input that is asserted, not measured, for the English condition: a partial circularity that forces the human side of the comparison without touching the independently measured model scores, so the paper does not warrant a score of 6 or higher. Other flagged concerns are correctness or validity risks rather than circular steps: the permission to skip questions with no skip counts or confidence intervals (Section 4.1.1 and Appendix A.4), the pilot-based subset division with a small five-person calibration (Section 4.1.1), the Chinese-only participant pool (acknowledged as 'Low population coverage' in Section 6), and the non-global best-human sample (Appendix A.5). Each is weighed here as a caveat on the headline claim, not as a self-referential derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on benchmark construction and human experiment design, not on a mathematical derivation. There are no fitted parameters or invented entities. The main unstated premises are the representativeness of the human sample and the fairness of the evaluation protocol.

assumptions (4)
  • domain assumption Average accuracy of 575 Chinese volunteers is a valid baseline for general human multimodal performance.
    Section 4.1.1 and Section 6 acknowledge the sample is mostly from one linguistic and cultural context and skews educated, which limits but does not remove the baseline claim.
  • domain assumption Bilingual English and Chinese questions are semantically equivalent.
    Section 3.1 says coauthors reviewed translations, but no independent translator check or bilingual participant validation is described.
  • domain assumption Participants answering in about 20 minutes with permission to skip still produce meaningful accuracy.
    Appendix A.1 estimates 20 minutes per survey and Appendix A.4 allows skipping. If rushing or skipping dominates, human average accuracy may be underestimated.
  • domain assumption MLLM evaluation settings (temperature 0, retry 10, VLMEvalKit) are fair across models.
    Section 4.2 and Appendix B describe standard practice, but no comparison to other decoding settings or API versions is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MANBench: Is Your Multimodal Model Smarter than Human?." pith.science (2026). https://pith.science/paper/P6OAI6UG

@misc{pith2026250611080,
  author       = {Pith},
  title        = {Pith review of: MANBench: Is Your Multimodal Model Smarter than Human?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P6OAI6UG}},
  note         = {Machine review of arXiv:2506.11080}
}
read the original abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has ignited discussions regarding their potential to surpass human performance in multimodal tasks. In response, we introduce MANBench (Multimodal Ability Norms Benchmark), a bilingual benchmark (English and Chinese) comprising 1,314 questions across nine tasks, spanning knowledge-based and non-knowledge-based domains. MANBench emphasizes intuitive reasoning, seamless cross-modal integration, and real-world complexity, providing a rigorous evaluation framework. Through extensive human experiments involving diverse participants, we compared human performance against state-of-the-art MLLMs. The results indicate that while MLLMs excel in tasks like Knowledge and Text-Image Understanding, they struggle with deeper cross-modal reasoning tasks such as Transmorphic Understanding, Image Consistency, and Multi-image Understanding. Moreover, both humans and MLLMs face challenges in highly complex tasks like Puzzles and Spatial Imagination. MANBench highlights the strengths and limitations of MLLMs, revealing that even advanced models fall short of achieving human-level performance across many domains. We hope MANBench will inspire efforts to bridge the gap between MLLMs and human multimodal capabilities. The code and dataset are available at https://github.com/micdz/MANBench.

Figures

Figures reproduced from arXiv: 2506.11080 by the authors.

Figure 1
Figure 1. Limitations of existing text-image benchmarks. (a) Some samples necessitate prior knowledge rather [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Qualitative results on MANBench. For each task, we display the selections made by GPT-4o, Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Statistics of MANBench. The benchmark comprises 9 tasks, categorized into knowledge-based and non-knowledge-based questions. fully constructed to avoid domain-specific knowl￾edge, ensuring ease of use and fairness in bench￾marking. Seamless Cross-Modal Integration: Questions necessitate joint interpretation of images and text, demanding causal or functional relationships be￾tween modalities rather than superficial k… view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: Age and gender distribution of participants. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison across different age [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Average human response time for all ques [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Screenshot of the User Personal Information [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Screenshot of the User Question Interface: [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison on the MANBench Chinese subset among human average accuracy, and some selected MLLMs. Please refer to [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Examples of data in MANBench with GPT-4o predictions.(1/9) [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Examples of data in MANBench with GPT-4o predictions.(2/9) [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Examples of data in MANBench with GPT-4o predictions.(3/9) [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Examples of data in MANBench with GPT-4o predictions.(4/9) [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Examples of data in MANBench with GPT-4o predictions.(5/9) [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Examples of data in MANBench with GPT-4o predictions.(6/9) [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Examples of data in MANBench with GPT-4o predictions.(7/9) [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Examples of data in MANBench with GPT-4o predictions.(8/9) [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Examples of data in MANBench with GPT-4o predictions.(9/9) [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 8 canonical work pages

  1. [1]

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319

    arXiv 2019

  2. [2]

    Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  3. [3]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024 a . Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330

    arXiv 2024

  4. [4]

    Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. 2024 b . Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint arXiv:2408.03361

    arXiv 2024

  5. [5]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024 c . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185--24198

    work page 2024

  6. [6]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11198--11201

    work page 2024

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390

    arXiv 2024

Show all 36 references
  1. [9]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  2. [10]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  3. [11]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  4. [12]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 c . Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  5. [13]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 d . https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    2024 arXiv

  6. [14]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Repre...

    2024

  7. [15]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Sy...

    2022

  8. [16]

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. 2024. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718

    2024 arXiv

  9. [17]

    OpenAI. 2024 a . Gpt-4o. https://openai.com/index/hello-gpt-4o/

    2024

  10. [18]

    OpenAI. 2024 b . Gpt-o1. https://openai.com/index/openai-o1-system-card/

    2024

  11. [19]

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. 2012. https://doi.org/10.1109/CVPR.2012.6248092 Cats and dogs . In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498--3505

    2012

  12. [20]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  13. [21]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2014. https://doi.org/10.48550/arXiv.1409.0575 Imagenet large scale visual recognition challenge

    2014 doi arXiv

  14. [22]

    SenseTime. 2024. Sensenova. https://platform.sensenova.cn/home

    2024

  15. [23]

    StepFun. 2025. Step-1o. https://platform.stepfun.com/

    2025

  16. [24]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  17. [25]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qvq-72b-preview/ Qvq: To see the world with wisdom

    2024

  18. [26]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwen2.5-vl/ Qwen2.5-vl

    2025

  19. [28]

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. 2024 b . Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411

    2024 arXiv

  20. [29]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024 c . https://arxiv.org/abs/2402.14804 Measuring multimodal mathematical reasoning with math-vision dataset . Preprint, arXiv:2402.14804

    2024 arXiv

  21. [31]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024 e . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  22. [32]

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. 2024 f . Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442

    2024 arXiv

  23. [33]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2024. arXiv preprint arXiv:2406.19314 Livebench: ...

    2024 arXiv

  24. [34]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024 arXiv

  25. [35]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. Mmm...

    2024

  26. [36]

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024 a . A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332

    2024 arXiv

  27. [37]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. 2024 b . Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169--...

    2024

  28. [38]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.