Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens

Bohan Zhuang; Chenhui Gou; Feng Chen; Jing Liu; Jiyuan Zhang; Qi Wu; Yang Yang; Zhaoyang Li; Zhenbang Sun

REVIEW 4 major objections 5 minor 61 references

Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Vision-language model perception cannot be judged by accuracy alone: rankings flip across benchmarks and abilities conflict during training, so this paper builds AbilityLens, a six-ability benchmark scoring accuracy and stability, and…

desk verdict A useful and reusable perception benchmark with a cohort-dependent stability metric; the benchmark work should get a fair review, but the causal claims about ability conflict need softening. read the letter →

arxiv 2411.14725 v2 pith:E43VPQLL submitted 2024-11-22 cs.CV cs.CLcs.LG

Feng Chen , Chenhui Gou , Jing Liu , Yang Yang , Zhaoyang Li , Jiyuan Zhang , Zhenbang Sun , Bohan Zhuang

show 1 more author

Qi Wu

This is my paper · ORCID

classification cs.CVcs.CLcs.LG

keywords multimodallargelanguagemodelsperceptionevaluationbenchmarkstabilityaccuracyabilityconflicttrainingdynamicsbaselinecorrection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Perception is the core skill that separates multimodal large language models from text-only ones, but the paper argues that no single existing benchmark can tell how well a model really sees: rankings of the same models flip across different perception tests. AbilityLens pools more than 12,000 samples from 11 public benchmarks into six perception abilities—counting, OCR, attribute recognition, entity extraction, grounding, and structural data understanding—and scores each model twice, once for accuracy and once for stability. Accuracy comes from baseline-corrected sub-metrics so that True/False, multiple-choice, and VQA questions are not distorted by their different random-guess rates; stability is the spread of a model's z-scores across those sub-metrics among 18 candidate models. With this lens the paper finds that the strongest open models match closed ones in accuracy but trail them noticeably in stability, and that during training abilities do not improve in lockstep: OCR peaks at mid-training and then declines while other abilities keep improving. The paper identifies the data mixing ratio and the size of the LLM backbone, rather than the visual encoder, as the main drivers of this ability conflict.

What carries the argument

The load-bearing object is AbilityLens, an ability-decomposed benchmark assembled from 11 public test suites. The identity that carries the argument is the baseline correction $\tilde{m}_{ij} = (m_{ij} - BL_j)/(UB_j - BL_j)$, which rescales every sub-metric between its random-guess baseline and its upper bound, so that the weighted accuracy score $A_i = \sum_j n_j \tilde{m}_{ij} / N$ is question-type agnostic. The stability score is $I_i = \mathrm{std}(z_i)$, where $z_{ij} = (m_{ij} - \mu_j)/\sigma_j$ is the model's relative position among 18 candidate models on sub-metric $j$; a high standard deviation means the model's standing swings across question types, domains, and metrics. Together the two scores turn a collection of heterogeneous benchmarks into a common scale on which training checkpoints can be compared and ability conflicts can be localized to a specific skill.

What would settle it

Recompute the ability rankings and stability scores with a different cohort of 18 comparable models, or with a different random sample drawn from the same 11 source benchmarks; if top-model accuracy-stability rankings shift materially or the OCR mid-training peak vanishes under the same data recipe, the central claim fails.

Watch

Extended reading notes

Core claim

AbilityLens is a unified evaluation instrument, and its central claim is that perception ability is two-dimensional: accuracy and stability, with stability defined as the standard deviation of a model's z-scores across sub-metrics relative to a cohort of 18 recent MLLMs. The benchmark is constructed by selecting roughly 1,000 or more test samples per ability from 11 source benchmarks, applying baseline correction to every sub-metric, and aggregating with sample-count weights. Evaluated on 18 open and closed models, AbilityLens reveals that no model dominates all six abilities, that closed-source models reach accuracy comparable to the best open models while being substantially more stable, and that tracking checkpoints of a LLaVA-OneVision reproduction shows early convergence plus a specific ability conflict: OCR accuracy and stability rise, peak around 10,000 training steps, and then fall while other abilities keep rising. The paper's causal analysis attributes the conflict to data mixing ratio and LLM model size, not the vision encoder, and shows that a lightweight model-merging step (Task Arithmetic) mitigates the conflict better than continued fine-tuning at a fraction of the compute.

Load-bearing premise

The framework assumes that the 18 models chosen as the comparison cohort and the reduced samples selected from each of the 11 source benchmarks are representative enough that the z-score and weighting steps yield a benchmark-independent measurement of perception ability.

Editorial extensions

If this is right

If accuracy and stability are treated as separate axes, an accuracy leader that is erratic across question types will no longer outrank a steadier model, and benchmark designers can report both numbers instead of one.
Monitoring checkpoints per ability during training will reveal conflicts that average-score curves hide; in the paper's LLaVA-OneVision reproduction, OCR degrades after 10,000 steps while the average keeps rising.
The causal finding that data mixing ratio and LLM size drive ability conflict means training strategies should rebalance data proportions and strengthen the backbone, not just swap in a better vision encoder.
For open-source development, the stability gap with closed models implies that consistency across question types is an achievable and necessary target, not a luxury.
When a conflict hits only one ability, model-merging (Task Arithmetic) recovers it more cheaply and more effectively than continued fine-tuning, while conflicts across several abilities remain open.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Eq. (3) is defined against the 18-model cohort, AbilityLens rankings are candidate-dependent; extending the benchmark to a larger or differently sampled cohort would test whether the stability gaps and conflict findings transfer.
The same six-ability decomposition and baseline-correction scheme could be applied to video or audio-visual models, where stability across frames or modalities may be a stronger signal than aggregate accuracy.
If data mixing ratio is the primary conflict driver, then ordering training data by ability convergence time—entity early, counting late, OCR mid-training—could reduce conflicts without changing total data volume.
The baseline-correction formula assumes each sub-metric's upper bound is attainable; on saturated or floored metrics, both the accuracy and z-score weights may need renormalization.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: AbilityLens is a solid benchmark-building paper, not a breakthrough. What is genuinely new is packaging eleven existing benchmarks into six perception abilities with a baseline-corrected weighted accuracy score and a z-score-based stability metric, then applying the same tool to monitor training checkpoints. That is a real contribution, and the public release makes it usable. The motivation figure showing models ranking differently on MME, MMBench, and SeedBench for entity is a fair example of why benchmark variance matters.

The stability metric is the most interesting part and the softest spot. Equations (3)-(4) define stability as the spread of a model's z-scores across sub-metrics, where the mean and standard deviation are computed from the 18 evaluated models. So stability is a cohort-relative property. A model that is uniformly strong looks perfectly stable; a balanced model can look unstable if it sits near the cohort mean on some sub-metrics and far above on others. The paper's claim that closed-source models are more stable, and the stability rankings in Table 4, could shift if the cohort were changed. The stress-test concern about this is correct. It is not fatal if AbilityLens is treated as a fixed public leaderboard with a fixed cohort, but the paper should say that explicitly and show a sensitivity analysis with a few cohort perturbations.

Second, the paper has no error bars and says so in the checklist. For a benchmark paper that ranks models and reports training curves, that is a real weakness. Several conclusions, especially OCR degradation after 10k steps, could be noise. The sample-reduction procedure is also unspecified: Table 2 shows big reductions (SeedBench 2447 to 1223, ChartQA 2500 to 400) but the text only says redundant samples are removed. That needs detail.

Third, the claim that the primary reason of ability conflict is data mixing ratio and LLM model size is too strong. The evidence is a handful of training runs with one architecture family, a few LLM backbones, and two data perturbations. It is a reasonable hypothesis, not a demonstrated cause. The paper would be stronger with 'suggests' or 'is associated with'.

The citation pattern looks fine; UniBench and EUREKA are acknowledged as closest prior work. The benchmark construction is internally consistent, and the accuracy part of the evaluation is not undercut by the stability caveat.

Who is this for: people building or evaluating MLLMs who want a holistic perception benchmark and a quick way to track ability-level training dynamics. It is not a theory paper and not a strong causal study. I would send it to a serious referee. The right outcome is conditional acceptance asking for sensitivity analysis, sampling details, error bars, and softened causal language.

Referee Report

4 major / 5 minor

Summary. The paper introduces AbilityLens, a benchmark assembled from 11 public datasets and organized around six perception abilities: counting, OCR, attribute recognition, entity extraction, grounding, and structural-data understanding. It defines a baseline-corrected accuracy score (Eqs. 1-2) and a z-score-based stability score (Eqs. 3-4), evaluates 18 open- and closed-source MLLMs, and uses the benchmark to monitor training dynamics of LLaVA-OV-SI checkpoints. The authors report that closed-source models are more stable, that perception abilities exhibit early convergence and ability conflicts, and that data mixing ratio and LLM size are the primary causes of ability conflict. They also discuss fine-tuning and model merging as partial remedies.

Significance. If the claims hold, AbilityLens is a useful contribution: it is efficient (about 0.5 hours for a 7B model on 8 H100s), built from public data, and ships with code and a leaderboard, which is a reproducibility-friendly design. The distinction between accuracy and stability, and the use of training-dynamics monitoring to expose ability conflicts, are valuable and actionable directions for MLLM development. The paper's empirical findings are also falsifiable: specific models and checkpoints are enumerated, and the benchmark is released. However, the robustness of the stability metric and the strength of the causal claims about ability conflict must be established before the central conclusions can be accepted.

major comments (4)

minor comments (5)

[Table 2 caption; Table 7; Sec. 4]
[Fig. 1]
[Checklist item 12]
[Sec. 4.1]
[Figs. 5-6]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AbilityLens is an empirical composite benchmark whose metrics transparently aggregate public benchmark results, and whose training-dynamics findings come from separate controlled experiments.

full rationale

The paper's central deliverable is an evaluation benchmark assembled from 11 public datasets. Accuracy is defined by Eq. (1)-(2) as a sample-count-weighted sum of baseline-corrected sub-metrics, and stability by Eq. (3)-(4) as the standard deviation of z-scores across sub-metrics. These are measurement definitions, not fitted parameters later reported as predictions; no quantity is estimated from a subset and then 'predicted' for the same subset. The z-score normalization uses the mean and variance of the 18 evaluated models, so the stability score is explicitly relative to that candidate cohort (the text says it 'directly reflects its relative performance compared to all candidate models'). This is a standardized relative metric rather than a circular derivation: the paper does not claim cohort-independence, and altering the cohort would change the normalization without making any equation equal to its own input. The ability-conflict conclusions in Sec. 5 come from controlled training runs with different LLMs, visual encoders, and data-mixing ratios, not from the benchmark construction. Self-citations (e.g., LLaVA, with co-author Qi Wu) are not load-bearing. The checklist item 7 answer 'No' regarding error bars is a genuine statistical-robustness limitation and the conclusion lists coverage limitations, but neither constitutes circularity. Overall, the benchmark and empirical findings are self-contained and externally grounded in public benchmarks and independent training experiments.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark itself introduces no new physical or conceptual entities. The key free choices are the baseline-correction bounds, the candidate cohort for z-scores, and the sample-reduction strategy, all of which influence the reported accuracy and stability values.

free parameters (3)

Per-sub-metric upper bounds UBj and baselines BLj = Not fully specified in paper
Eq. (1) requires UBj and BLj for every source metric (e.g., VQA baseline set to 0). These values are chosen by hand and directly scale the accuracy scores; no table lists all values.
Candidate model cohort for z-score normalization = 18 MLLMs
Eq. (3) uses the mean and std of the 18-model cohort to standardize sub-metrics; stability scores are relative to this specific cohort and would change if the cohort changed.
Reduced sample counts per source benchmark = e.g., SeedBench counting 2447 to 1223
Large datasets are reduced (Table 2) without describing the sampling method; the chosen subset sizes and selection affect all downstream scores.

assumptions (4)

ad hoc to paper Per-sub-metric baselines and upper bounds can be combined in a single normalized accuracy score after linear correction (Eq. 1)
Assumes different question types (T/F, MCQ, VQA) are commensurable after shifting by random-guess baseline and scaling by upper bound; no validation that corrected scores align difficulty.
ad hoc to paper The standard deviation of z-scores across sub-metrics measures perception stability (Eq. 4)
Defines stability as consistency of relative performance across sub-metrics within the cohort; this is a constructed metric, not an external property.
domain assumption The 18 candidate MLLMs are a representative reference cohort
Z-scores and stability depend on the cohort composition; commercial model sizes are only artificial estimates (Figure 1 caption).
domain assumption Source benchmark labels and evaluation scripts from LMMs-Eval are correct
AbilityLens reuses samples and metrics from 11 public benchmarks; errors in source annotations propagate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens." pith.science (2026). https://pith.science/paper/E43VPQLL

@misc{pith2026241114725,
  author       = {Pith},
  title        = {Pith review of: Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E43VPQLL}},
  note         = {Machine review of arXiv:2411.14725}
}

read the original abstract

As multimodal large language models (MLLMs) advance rapidly, rigorous evaluation has become essential, providing further guidance for their development. In this work, we focus on a unified and robust evaluation of \textbf{vision perception} abilities, the foundational skill of MLLMs. We find that existing perception benchmarks, each focusing on different question types, domains, and evaluation metrics, introduce significant evaluation variance, complicating comprehensive assessments of perception abilities when relying on any single benchmark. To address this, we introduce \textbf{AbilityLens}, a unified benchmark designed to evaluate MLLMs in six key perception abilities (ranging from counting, OCR, to understanding structural data), focusing on both accuracy and stability, with each ability encompassing diverse types of questions, domains, and metrics. With the assistance of AbilityLens, we: (1) identify the strengths and weaknesses of current main-stream MLLMs, highlighting stability patterns and revealing a notable performance gap between state-of-the-art open-source and closed-source models; (2) uncover interesting ability conflict and early convergence phenomena during MLLM training; (3) reveal the primary reason of ability conflict is data mixing ratio and LLM model size; and (4) discuss the effectiveness of some straightforward strategies \eg, fine-tuning and model merging, to solve the ability conflict. The benchmark and online leaderboard is released in https://github.com/Chenfeng1271/AbilityLens.

Figures

Figures reproduced from arXiv: 2411.14725 by the authors.

**Figure 1.** Motivation behind our AbilityLens and our solution. (a) shows evaluation variance across three existing benchmarks [12, 18, 40] on entity capability. The differing best and worst models across benchmarks underscore the need for a holistic assessment of perception abilities. (b) Our AbilityLens offers a more complete evaluation by encompassing diverse question types, domains, and metrics, assessing both accuracy and … view at source ↗

**Figure 2.** Monitoring training dynamics with AbilityLens. Unlike traditional methods that focus solely on average scores, often obscuring underlying conflicts, AbilityLens tracks optimization trends for each specific ability, revealing an ability conflict indicated by a significant performance degradation in accuracy and stability of OCR performance after 10,000 steps. where µj and σj are the mean and variance of the results o… view at source ↗

**Figure 3.** Accuracy and stability comparison across AbilityLens’s six perception dimensions, evaluating four state-of-the-art MLLMs alongside models of different LLM scales. MID SI OV 20 40 60 80 Score Counting Accuracy MID SI OV OCR Accuracy MID SI OV Grounding Accuracy MID SI OV Entity Accuracy MID SI OV Attribute Accuracy MID SI OV Structured Data Accuracy MID SI OV Average Accuracy MID SI OV 0 20 40 60 80 100 120 Score Sta… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Performance dynamics in the middle (MID), single-image (SI), and one-vision (OV) stages of LLaVA-OV-7b. Accuracy curves demonstrate consistent improvement across stages, except for OCR tasks, whereas stability curves indicate that stability is less prioritized in data …

**Figure 5.** Figure 5: Monitoring the training dynamics of LLaVA-OV-SI using Qwen2-7b (first row), Phi3.5-3.8b [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Monitoring training dynamics of LLaVA-OV-SI using original data (first row), 50% original [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 27 canonical work pages

[1]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

Al-Tahan, Q

H. Al-Tahan, Q. Garrido, R. Balestriero, D. Bouchacourt, C. Hazirbas, and M. Ibrahim. Unibench: Visual reasoning requires rethinking vision-language beyond scaling. arXiv preprint arXiv:2408.04810, 2024

arXiv 2024
[3]

Claude 3.5 sonnet

Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet , 2024. Accessed: 2024-11-13

work page 2024
[4]

B. Atil, A. Chittams, L. Fu, F. Ture, L. Xu, and B. Baldwin. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408.04667, 2024

arXiv 2024
[5]

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

arXiv 2023
[6]

S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

arXiv 2025
[7]

Balachandran, J

V . Balachandran, J. Chen, N. Joshi, B. Nushi, H. Palangi, E. Salinas, V . Vineet, J. Woffinden-Luey, and S. Yousefi. Eureka: Evaluating and understanding large foundation models. arXiv preprint arXiv:2409.10566, 2024

arXiv 2024
[8]

T. B. Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005

Show all 61 references

[9]

L. Chen, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, J. Wang, Y . Qiao, D. Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

2024 arXiv
[10]

Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

2024 arXiv
[11]

W. Dai, N. Lee, B. Wang, Z. Yang, Z. Liu, J. Barker, T. Rintamaki, M. Shoeybi, B. Catanzaro, and W. Ping. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402, 2024

2024 arXiv
[12]

C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, K. Li, X. Sun, and R. Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306.13394, 2023

2023 arXiv
[13]

Hurst, A

A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

2024 arXiv
[14]

Ilharco, M

G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

2022 arXiv
[15]

Jiang, X

D. Jiang, X. He, H. Zeng, C. Wei, M. Ku, Q. Liu, and W. Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

2024 arXiv
[16]

Kembhavi, M

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–251. Springer, 2016

2016
[17]

G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022

2022
[18]

B. Li, Y . Ge, Y . Ge, G. Wang, R. Wang, R. Zhang, and Y . Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

2024
[19]

B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y . Li, Z. Liu, and C. Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

2024 arXiv
[20]

K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

2024
[21]

H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning, 2023. 10

2023
[22]

H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

2024
[23]

H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning. In NeurIPS, 2023

2023
[24]

Y . Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. lin Liu, L. Jin, and X. Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

2024
[25]

Y . Liu, Z. Zhao, Z. Zhuang, L. Tian, X. Zhou, and J. Zhou. Points: Improving your vision-language model with affordable strategies. arXiv preprint arXiv:2409.04828, 2024

2024 arXiv
[26]

P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

arXiv
[27]

S. Lu, Y . Li, Q.-G. Chen, Z. Xu, W. Luo, K. Zhang, and H.-J. Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024

2024 arXiv
[28]

Masry, D

A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

2022 arXiv
[29]

Mathew, D

M. Mathew, D. Karatzas, and C. Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2200–2209, 2021

2021
[30]

Moayeri, V

M. Moayeri, V . Balachandran, V . Chandrasekaran, S. Yousefi, T. Fel, S. Feizi, B. Nushi, N. Joshi, and V . Vineet. Unearthing skill-level insights for understanding trade-offs of foundation models.arXiv preprint arXiv:2410.13826, 2024

arXiv
[31]

Gptv system card, 2024

OpenAI. Gptv system card, 2024. Accessed: 2024-11-13

2024
[32]

Oquab, T

M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

2023 arXiv
[33]

A. Radford. Improving language understanding by generative pre-training. 2018

2018
[34]

Radford, J

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021

2021
[35]

S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024

2024 arXiv
[36]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[37]

F. Wang, X. Fu, J. Y . Huang, Z. Li, Q. Liu, X. Liu, M. D. Ma, N. Xu, W. Zhou, K. Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.09411, 2024

2024 arXiv
[38]

P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

2024 arXiv
[39]

Y . Yang, W. Wang, Z. Chen, J. Dai, and L. Zheng. Bounding box stability against feature dropout reflects detector generalization across environments. arXiv preprint arXiv:2403.13803, 2024

2024 arXiv
[40]

Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023

2023 arXiv
[41]

X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

2024
[42]

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11975–11986, 2023. 11

2023
[43]

Zhang, B

K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y . Zhang, J. Yang, C. Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024

2024 arXiv
[44]

Zhang, F

Y . Zhang, F. Xiao, T. Huang, C.-K. Fan, H. Dong, J. Li, J. Wang, K. Cheng, S. Zhang, and H. Guo. Unveiling the tapestry of consistency in large vision-language models. arXiv preprint arXiv:2405.14156, 2024

2024 arXiv
[45]

B. Zhao, Y . Zong, L. Zhang, and T. Hospedales. Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint arXiv:2406.12742, 2024. A Technical Appendices and Supplementary Material Technical a...

2024 arXiv
[46]

Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We claim our MLLM perception benchmark contribution in the abstract. Guidelines: • The answer NA means that the abstra...
[47]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We claim our limitations in the conclusion. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the pap...
[48]

Guidelines: • The answer NA means that the paper does not include theoretical results

Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 13 Justification: We do not have theoretical results. Guidelines: • The answer NA means that the paper does ...
[49]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...
[50]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 14 Answer: [Yes] Justification: We provide the data and ...
[51]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We elaborate on the dataset d...
[52]

Guidelines: • The answer NA means that the paper does not include experiments

Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: We report the performance with trustworthy LLM eval...
[53]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Our benchmark is effici...
[54]

Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Our benchmark meets NeurIPS Code of Ethics. Guidelines: • The answer NA means t...
[55]

Guidelines: • The answer NA means that there is no societal impact of the work performed

Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We claim broader impact in the supplementary. Guidelines: • The answer NA means that there is no societal...
[56]

Guidelines: • The answer NA means that the paper poses no such risks

Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our benchmark i...
[57]

We will elaborate on it on the Appendix

Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [No] Justification: Our benchmark is ...
[58]

Guidelines: • The answer NA means that the paper does not release new assets

New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our benchmark is well documented. Guidelines: • The answer NA means that the paper does not release new assets. • Re...
[59]

Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...
[60]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...
[61]

Answer: [NA] Justification: We use LLM for paper writing

Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[2] [2]

Al-Tahan, Q

H. Al-Tahan, Q. Garrido, R. Balestriero, D. Bouchacourt, C. Hazirbas, and M. Ibrahim. Unibench: Visual reasoning requires rethinking vision-language beyond scaling. arXiv preprint arXiv:2408.04810, 2024

arXiv 2024

[3] [3]

Claude 3.5 sonnet

Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet , 2024. Accessed: 2024-11-13

work page 2024

[4] [4]

B. Atil, A. Chittams, L. Fu, F. Ture, L. Xu, and B. Baldwin. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408.04667, 2024

arXiv 2024

[5] [5]

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

arXiv 2023

[6] [6]

S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

arXiv 2025

[7] [7]

Balachandran, J

V . Balachandran, J. Chen, N. Joshi, B. Nushi, H. Palangi, E. Salinas, V . Vineet, J. Woffinden-Luey, and S. Yousefi. Eureka: Evaluating and understanding large foundation models. arXiv preprint arXiv:2409.10566, 2024

arXiv 2024

[8] [8]

T. B. Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005