pith. machine review for the scientific record. sign in

arxiv: 2407.12772 · v2 · submitted 2024-07-17 · 💻 cs.CL · cs.CV

Recognition: 2 theorem links

· Lean Theorem

LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

Kaichen Zhang , Bo Li , Peiyuan Zhang , Fanyi Pu , Joshua Adrian Cahyono , Kairui Hu , Shuai Liu , Yuanhan Zhang
show 3 more authors
Jingkang Yang Chunyuan Li Ziwei Liu
Authors on Pith no claims yet

Pith reviewed 2026-05-17 05:15 UTC · model grok-4.3

classification 💻 cs.CL cs.CV
keywords large multimodal modelsbenchmark evaluationdata contaminationevaluation trade-offsmultimodal AIgeneralization testinglive benchmarks
0
0 comments X

The pith

Evaluating large multimodal models requires balancing wide task coverage, low computational cost, and zero data contamination in benchmarks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that progress in large multimodal models, which handle both images and text, demands benchmarks that cover many capabilities, run efficiently, and avoid using test data that models may have encountered during training. Current comprehensive benchmarks achieve broad coverage but fall short on cost and contamination control. It proposes a standardized suite spanning dozens of tasks, a reduced version that trims computation while retaining key coverage, and a live evaluation drawing from ongoing news and forums to test generalization on fresh data. A reader would care because flawed evaluations can distort understanding of actual model abilities and slow reliable advances. The work shows how to navigate these competing demands for more practical and trustworthy assessments.

Core claim

Large multimodal model evaluations encounter an inherent trilemma among achieving extensive task coverage, maintaining low evaluation costs, and ensuring zero contamination, and practical approaches including a broad standardized collection of over 50 tasks, an efficiency-oriented pruned subset, and a live benchmark built from continuously updating online sources can help address the resulting trade-offs.

What carries the argument

The evaluation trilemma that pits wide-coverage testing of multimodal capabilities against requirements for low cost and freedom from training-data overlap.

If this is right

  • Researchers can run more frequent evaluations without excessive compute demands while still sampling diverse skills.
  • Models' ability to handle genuinely new scenarios can be measured using data that post-dates their training.
  • Consistent benchmark suites allow direct comparisons across different models and research efforts.
  • Pruning methods can shrink evaluation time while keeping enough tasks to reveal both strengths and limitations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Developers might shift focus toward capabilities that hold up under fresh data rather than optimizing for static test sets.
  • Similar live-data strategies could apply to language-only model evaluations to lower contamination risks more broadly.
  • Quantifying performance differences between full and pruned versions on the same models would clarify acceptable trade-off levels.

Load-bearing premise

Live data sources combined with pruning rules can deliver truly uncontaminated tests that preserve broad coverage without creating new selection biases or overlooking key capabilities.

What would settle it

Demonstrating that models score substantially higher on the live benchmark than expected because recent events or forum content indirectly appeared in training data, or showing that the pruned subset fails to detect important weaknesses revealed by the full task set.

read the original abstract

The advances of large foundation models necessitate wide-coverage, low-cost, and zero-contamination benchmarks. Despite continuous exploration of language model evaluations, comprehensive studies on the evaluation of Large Multi-modal Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models to promote transparent and reproducible evaluations. Although LMMS-EVAL offers comprehensive coverage, we find it still falls short in achieving low cost and zero contamination. To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency. Additionally, we present Multimodal LIVEBENCH that utilizes continuously updating news and online forums to assess models' generalization abilities in the wild, featuring a low-cost and zero-contamination evaluation approach. In summary, our work highlights the importance of considering the evaluation trilemma and provides practical solutions to navigate the trade-offs in evaluating large multi-modal models, paving the way for more effective and reliable benchmarking of LMMs. We opensource our codebase and maintain leaderboard of LIVEBENCH at https://github.com/EvolvingLMMs-Lab/lmms-eval and https://huggingface.co/spaces/lmms-lab/LiveBench.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces LMMS-EVAL, a unified multimodal benchmark framework covering over 50 tasks and more than 10 models to support transparent LMM evaluation. It identifies an evaluation trilemma of wide coverage, low cost, and zero contamination, then proposes LMMS-EVAL LITE (a pruned toolkit balancing coverage and efficiency) and Multimodal LIVEBENCH (live-updating news/forums sources with pruning for zero-contamination generalization testing). The work open-sources the codebase, maintains a leaderboard, and argues these tools help navigate the trilemma.

Significance. If the proposed mechanisms are shown to preserve coverage and achieve the claimed cost/contamination benefits without new biases, the work would supply practical, reproducible toolkits that address a real gap in LMM benchmarking. The open-sourcing of the codebase and maintenance of the LIVEBENCH leaderboard are concrete strengths that support community use and reproducibility.

major comments (2)
  1. [Abstract and LMMS-EVAL LITE description] Abstract and the section describing LMMS-EVAL LITE: the claim that the pruned toolkit 'emphasizes both coverage and efficiency' is not supported by any quantitative results on cost savings, coverage retention, or capability coverage; no ablation is presented that compares model orderings or task coverage between the full LMMS-EVAL and the LITE version.
  2. [Multimodal LIVEBENCH section] The section on Multimodal LIVEBENCH: the assertion of zero contamination via continuously updating news/forums plus pruning lacks any decontamination audit (n-gram overlap, human inspection, or similar) or empirical check that key capabilities are retained and no new selection biases are introduced.
minor comments (2)
  1. [Methods] Clarify the exact pruning rules and selection criteria used in LMMS-EVAL LITE so that readers can reproduce the coverage-efficiency trade-off.
  2. [Results] Add a table or figure summarizing the number of tasks retained after pruning and the live data update frequency for LIVEBENCH.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the major comments point by point below and outline the revisions we will make to strengthen the empirical support for our claims.

read point-by-point responses

  1. Referee: [Abstract and LMMS-EVAL LITE description] Abstract and the section describing LMMS-EVAL LITE: the claim that the pruned toolkit 'emphasizes both coverage and efficiency' is not supported by any quantitative results on cost savings, coverage retention, or capability coverage; no ablation is presented that compares model orderings or task coverage between the full LMMS-EVAL and the LITE version.

    Authors: We agree that the manuscript would be strengthened by explicit quantitative comparisons. In the revision we will add a new subsection with ablations that report retained task coverage percentages, estimated reductions in evaluation time and API costs, and consistency of model performance orderings between the full LMMS-EVAL and LMMS-EVAL LITE. These results will be derived from the existing evaluation logs and will be presented alongside the current qualitative description. revision: yes

  2. Referee: [Multimodal LIVEBENCH section] The section on Multimodal LIVEBENCH: the assertion of zero contamination via continuously updating news/forums plus pruning lacks any decontamination audit (n-gram overlap, human inspection, or similar) or empirical check that key capabilities are retained and no new selection biases are introduced.

    Authors: We acknowledge that the current text does not include explicit decontamination audits. In the revised manuscript we will add a dedicated paragraph reporting n-gram overlap statistics between the live sources and prior benchmarks, results from a small-scale human inspection of sampled items, and an analysis of capability retention and potential pruning-induced biases. These checks will be performed on the current LIVEBENCH data and included as supplementary material. revision: yes

Circularity Check

0 steps flagged

No circularity: new benchmark toolkits are self-contained constructions

full rationale

The paper introduces LMMS-EVAL as a unified benchmark framework, LMMS-EVAL LITE as a pruned variant, and Multimodal LIVEBENCH as a live-data evaluation approach. These are presented as practical engineering solutions to the stated evaluation trilemma rather than any mathematical derivation, prediction, or first-principles result that reduces to prior fitted quantities. No equations, fitted parameters renamed as predictions, or self-citation chains appear in the load-bearing claims; the contributions consist of newly assembled tasks, pruning pipelines, and data sources whose validity rests on external coverage and contamination properties, not on internal redefinition. The manuscript is therefore self-contained against the circularity criteria.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

This applied benchmarking paper introduces new evaluation toolkits rather than relying on mathematical derivations; the main unstated premises are domain assumptions about contamination prevalence and the sufficiency of news/forum data for generalization testing.

axioms (1)
  • domain assumption Existing LMM benchmarks suffer from high cost and data contamination that limit their usefulness
    Invoked to motivate the new frameworks; appears in the abstract discussion of the evaluation trilemma.
invented entities (2)
  • LMMS-EVAL LITE no independent evidence
    purpose: Pruned toolkit balancing coverage and efficiency
    Newly proposed in this work as a practical solution.
  • Multimodal LIVEBENCH no independent evidence
    purpose: Live benchmark using continuously updating external data sources
    Newly proposed in this work to achieve zero contamination.

pith-pipeline@v0.9.0 · 5575 in / 1323 out tokens · 57957 ms · 2026-05-17T05:15:22.504049+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Mask prior drift and positional attention collapse cause failures in LDVLMs for long generations, fixed by training-free Mask Prior Suppression and Monotonic RoPE Scaling.

  2. MMSearch-R1: Incentivizing LMMs to Search

    cs.CV 2025-06 unverdicted novelty 7.0

    MMSearch-R1 uses reinforcement learning to train multimodal models for on-demand multi-turn internet search with image and text tools, outperforming same-size RAG baselines and matching larger ones while cutting searc...

  3. Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos

    cs.CV 2025-01 unverdicted novelty 7.0

    Video-MMMU benchmark shows large multimodal models exhibit steep performance drops on higher cognitive tasks when learning from professional videos and lag significantly behind humans in knowledge acquisition.

  4. LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMs

    cs.CV 2026-05 unverdicted novelty 6.0

    LDDR proposes a linear DPP-based dynamic-resolution frame sampler that achieves 3x speedup and up to 2.5-point gains on video MLLM benchmarks by selecting non-redundant frames and allocating tokens accordingly.

  5. Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction

    cs.LG 2026-05 unverdicted novelty 6.0

    A unified learnable KV eviction policy with cross-layer calibration reduces memory and matches or exceeds full-cache performance on long-context tasks by retaining useful tokens and limiting attention dilution.

  6. Latent Denoising Improves Visual Alignment in Large Multimodal Models

    cs.CV 2026-04 unverdicted novelty 6.0

    A latent denoising objective with saliency-aware corruption and contrastive distillation improves visual alignment and corruption robustness in large multimodal models.

  7. MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

    cs.LG 2026-04 unverdicted novelty 6.0

    MACS improves inference speed in multimodal MoE models by entropy-weighted balancing of visual tokens and real-time modality-adaptive expert capacity allocation.

  8. MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

    cs.LG 2026-04 unverdicted novelty 6.0

    MACS improves MoE MLLM inference efficiency via entropy-weighted token loads and dynamic modality-adaptive expert capacity allocation.

  9. POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs

    cs.CV 2026-04 unverdicted novelty 6.0

    POINTS-Long is a dual-mode multimodal large language model that uses dynamic visual token scaling to retain 97.7-99.7% accuracy on long-form tasks with 1/40 to 1/10th the tokens and supports streaming via detachable KV-cache.

  10. Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    Q-Zoom achieves up to 4.39x inference speedup in high-resolution MLLM scenarios via query-aware gating and region localization, matching or exceeding baseline accuracy on document and high-res benchmarks.

  11. Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

    cs.CV 2026-04 unverdicted novelty 6.0

    Saliency-R1 uses a novel saliency map technique and GRPO with human bounding-box overlap as reward to improve VLM reasoning faithfulness and interpretability.

  12. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    cs.AI 2025-06 unverdicted novelty 6.0

    V-JEPA 2 pre-trained on massive unlabeled video achieves strong results on motion understanding and action anticipation, SOTA video QA at 8B scale, and enables zero-shot robotic planning on Franka arms using only 62 h...

  13. LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

    cs.LG 2025-05 conditional novelty 6.0

    LLaDA-V is a diffusion-based multimodal large language model that reaches competitive or state-of-the-art results on visual instruction tasks while using a non-autoregressive architecture.

  14. LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL

    cs.CL 2025-03 unverdicted novelty 6.0

    A two-stage RL framework first boosts text reasoning in 3B LMMs then adapts it to multimodal inputs, producing modest benchmark gains of 4.5-4.8%.

  15. LLaVA-Video: Video Instruction Tuning With Synthetic Data

    cs.CV 2024-10 unverdicted novelty 6.0

    LLaVA-Video-178K is a new synthetic video instruction dataset that, when combined with existing data to train LLaVA-Video, produces strong results on video understanding benchmarks.

  16. TTF: Temporal Token Fusion for Efficient Video-Language Model

    cs.CV 2026-05 unverdicted novelty 5.0

    TTF fuses temporally redundant visual tokens via local similarity search in a plug-and-play way, cutting ~67% tokens on Qwen3-VL-8B while retaining 99.5% accuracy with minimal overhead.

  17. Make Your LVLM KV Cache More Lightweight

    cs.CV 2026-05 unverdicted novelty 5.0

    LightKV compresses vision-token KV cache in LVLMs to 55% size via prompt-guided cross-modality aggregation, halving cache memory, cutting compute 40%, and maintaining performance on benchmarks.

  18. LLaVA-OneVision: Easy Visual Task Transfer

    cs.CV 2024-08 unverdicted novelty 5.0

    LLaVA-OneVision is the first single open LMM to simultaneously achieve strong performance in single-image, multi-image, and video scenarios with cross-scenario transfer capabilities.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · cited by 17 Pith papers · 3 internal anchors

  1. [1]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

    Instructblip: Towards general-purpose vision- language models with instruction tuning. Preprint, arXiv:2305.06500. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2404.06512

    Internlm-xcomposer2-4khd: A pioneer- ing large vision-language model handling resolu- tions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji

    work page arXiv

  3. [3]

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    Mme: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelko...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Making llama see and draw with seed tokenizer.arXiv preprint arXiv:2310.01218, 2023

    A framework for few-shot language model evaluation. Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. 2023. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218. Gemini-Team. 2024. Gemini: A family of highly capa- ble multimodal models. Preprint, arXiv:2312.11805. Yash Goyal, Tejas Khot, Douglas S...

    work page arXiv 2023

  5. [5]

    A Diagram Is Worth A Dozen Images

    A diagram is worth a dozen images. Preprint, arXiv:1603.07396. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- seok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document under- standing transformer. In European Conference on Computer Vision (ECCV). Pang Wei Koh and Percy Liang. 2020. Underst...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  6. [6]

    Preprint, arXiv:1906.01827

    Coresets for data-efficient training of machine learning models. Preprint, arXiv:1906.01827. Mistral. 2024. Mixtral 8x22b: Cheaper, better, faster, stronger. OpenAI. 2023. Gpt-4v(ision) system card. OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong,...

    work page arXiv 1906

  7. [7]

    Preprint, arXiv:2402.14992

    tinybenchmarks: evaluating llms with fewer examples. Preprint, arXiv:2402.14992. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. Preprint,...

    work page arXiv 2021

  8. [8]

    What are the key points in this news story?

    **Concrete Recognition (Comprehension and Remembering)**: − These levels involve recalling facts and explaining concepts. − Example questions include: − "What are the key points in this news story?" (Remembering) − "How would you explain the main event reported here?" (Comprehension)

    work page

  9. [9]

    What are the factors that led to this event?

    **Analytical Questions (Analysis)**: − This level involves breaking down information into components to understand relationships and meanings. − Example questions: − "What are the factors that led to this event?" − "How does this event relate to other current issues?"

    work page

  10. [10]

    How could you create a new headline that captures the essence of the event differently?

    **Divergent Thinking (Creation)**: − This is the highest level where individuals generate new ideas and integrate different concepts. − Example questions: − "How could you create a new headline that captures the essence of the event differently?" − "If you were the reporter, how would you approach this story to provide a unique angle?" − "Do you think the...

    work page

  11. [11]

    Please present this news in Arabic and output it in markdown format

    **Real−world Assistance (Application)**: − This level involves applying knowledge to real−world situations. − Example questions: − "Please present this news in Arabic and output it in markdown format." − "Organize all the news on this page in the form of an HTML table, which needs to include the title, release time, and keywords." − "Sort out the exchange...

    work page

  12. [12]

    Modify the question to correspond to the subtask

    work page

  13. [13]

    However, you should not change the original question's subtask unless the original subtask is not one of these five

    Modify the subtask to correspond to the question. However, you should not change the original question's subtask unless the original subtask is not one of these five. If you feel the original question's subtask does not match ,→ the question, modify the question to match the subtask instead of rewriting the subtask. Please note that although the image may...

    work page

  14. [14]

    The criteria should be a natural language, don't use dict / json format for the criteria, human cannot understand it

    work page

  15. [15]

    But don't use python−like format

    You can use bullet points / numbers to the list / yaml format to the criteria. But don't use python−like format

    work page

  16. [16]

    If the answer is in dict format, but there is no need to answer in dict format (means there is a way to answer in natural language, the question do not specify to answer in ,→ dict format), you should convert it to natural language

    work page

  17. [17]

    But if you think some words should be in other language, you can keep it in that language

    If the whole criteria is in other language, change it to English. But if you think some words should be in other language, you can keep it in that language. If question or ,→ answer is in other language, you don't need to change it

    work page

  18. [18]

    The scoring criteria are rational and facilitate the accurate assessment of responses

    work page

  19. [19]

    The full score for the scoring criteria must be 10 points, and it must directly relate to the specific answer

    work page

  20. [20]

    The question is clear and unambiguous

    work page

  21. [21]

    Some tips:

    The answer is correct and reasonable (although the original ground truth answer is mostly correct, it may not be perfect, and sometimes the answer maybe incorrect). Some tips:

    work page

  22. [22]

    In such cases, you can relax the criteria slightly

    For some extremely hard open−ended questions where answers may vary, hitting all points perfectly may not be realistic. In such cases, you can relax the criteria slightly. ,→ For example, if there are five possible points in an answer, but answering three adequately could merit full points. An other option is to change the question to a ,→ multiple−choice...

    work page

  23. [23]

    Explanation

    For some questions, changing the format might be beneficial. You can consider transforming them into different types of questions such as essay, fill−in−the−blank, ranking ,→ (e.g., based on time, importance, etc.), or matching questions to enhance the difficulty and rationality of the scoring criteria. But a very important point is that ,→ DO NOT CHANGE ...

    work page 2014

  24. [24]

    This kind of systemic shift often results in skills becoming obsolete, leading to higher unemployment among professionals who cannot quickly adapt to new technological paradigms

    **Seismic Changes:** - Victor Janulaitis, Janco’s chief executive, compares the impact of AI to the seismic changes seen when personal computers came into wide use. This kind of systemic shift often results in skills becoming obsolete, leading to higher unemployment among professionals who cannot quickly adapt to new technological paradigms. **Positive Ef...

    work page 2016