Pith. sign in

REVIEW 4 major objections 4 minor 30 references

CRISP claims that visual token pruning for vision-language models can be done before the language model while still following the text query, preserving 97–100% accuracy even when discarding 78–89% of visual tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:37 UTC pith:K22FT5T2

load-bearing objection A genuinely text-driven pre-LLM pruning method that looks like it works; the main doubts are empirical hygiene and an unvalidated cosine-relevance assumption, not the method's logic. the 4 major comments →

arxiv 2607.16326 v1 pith:K22FT5T2 submitted 2026-07-15 cs.CV

CRISP: Pre-LLM Yet Text-Driven Visual Token Pruning for Efficient LVLM Inference

Xu Li , Yi Zheng , Mengyang Zhao , Yuxuan Liang , Zhe Liu , Rui Zhu , Xiaolei Chen , Wei Zhou
show 2 more authors
Baoquan Zhao Juncen Guo
This is my paper
classification cs.CV
keywords visual token pruningLVLM inference accelerationtraining-free efficient inferencetext-driven pruningcross-modal similaritytoken diversityLLaVAvisual-language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CRISP is a training-free visual token pruning method that acts entirely before the language model in a vision-language model, yet is driven by the text query. It first picks the visual tokens most similar to the nouns in the question, using the shared embedding space of the projector output and the LLM's word embeddings, then adds semantically diverse tokens to preserve scene context. The paper reports that on LLaVA-1.5-7B retaining only 22.2% of visual tokens preserves 99.5% of full-model accuracy, and on LLaVA-NeXT-7B keeping 11.1% of tokens retains 96.9% accuracy, while inference becomes more than twice as fast. The central claim is that text-awareness does not require entering the LLM: a similarity computation before the LLM can capture query-relevant visual evidence, and a diversity stage supplies the context that pure text-matching misses.

Core claim

The discovery is that the projected visual tokens and the LLM's word embeddings already lie in a common semantic space, and cosine similarity in that space is a usable relevance signal for pruning. CRISP extracts the content-bearing nouns from the query, averages their subword embeddings to get word-level embeddings, and keeps for each noun the visual tokens with the highest cosine similarity, with per-word budgets allocated by how much the vision encoder's CLS attention favors those tokens. A second stage then iteratively adds, from the remaining tokens, the one least similar to anything already selected, ensuring contextual coverage. The paper argues that this combination—critical-first se

What carries the argument

The load-bearing object is the cross-modal relevance matrix S, where S_{i,j} is the cosine similarity between projected visual token v_i and the averaged word embedding e_j of an extracted query noun. It drives Stage 1's per-word top-k selection and quota allocation; Stage 2 uses the same cosine-similarity geometry to select the remaining token with the largest distance to the current set (minimax dissimilarity). Together they turn a text-agnostic pre-LLM pruner into a text-driven one at no extra LLM cost.

Load-bearing premise

Equation (2) treats cosine similarity between projected visual tokens and word-level LLM embeddings as a measure of how relevant a patch is to the query; if the projector and embedding spaces are not aligned well enough for that ranking to track the actual answer-relevant regions, Stage 1 will retain the wrong tokens and retention will fall.

What would settle it

Take an image where the object named in the question (say 'dog') is small and the background contains a large green area whose projected tokens are more cosine-similar to the word embedding of 'dog' (e.g., due to color or texture correlations). If CRISP retains those grass tokens and drops the small dog region, its accuracy on a simple existence question should drop below that of a text-agnostic CLS-attention pruner or random retention at the same budget. Running such a controlled image set is a direct test of the similarity-as-relevance assumption.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deploying CRISP before the LLM reduces FLOPs and KV-cache roughly in proportion to the pruning ratio, yielding a >2× decoding speedup on tested models.
  • Because it never touches the LLM's attention, CRISP is compatible with FlashAttention and other optimized inference stacks that intra-LLM pruning disrupts.
  • The two-stage design is architecture-agnostic: it transfers from LLaVA-1.5 to LLaVA-NeXT with no retraining, suggesting it can be plugged into other LVLMs.
  • At high pruning ratios (retaining 11–22% of tokens) CRISP keeps 97–100% of averaged benchmark performance, including gains on hallucination-focused POPE where it can exceed the full-token baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the shared-space similarity assumption holds broadly, one could extend the same selection idea to prune or reweight tokens for other modalities (e.g., audio or video tokens), or to pick which image regions to feed at higher resolution.
  • The noun-extraction step is the main brittleness: for queries with no concrete nouns (e.g., 'What is wrong with this image?') CRISP falls back to CLS attention, essentially becoming text-agnostic; a testable extension is to use verb or attribute embeddings as additional grounding cues.
  • The quota-allocation via CLS attention introduces a second, text-agnostic signal; the paper does not analyze cases where CLS attention and text similarity disagree, so a targeted study of such conflicts could indicate when the two stages fight rather than complement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. CRISP is a training-free, pre-LLM visual token pruning method. It first extracts nouns from the textual query, averages their LLM word embeddings, and scores projected visual tokens by cosine similarity to these word embeddings (Eq. 2). Per-word quotas are allocated using CLS attention from the vision encoder, and the top tokens per word form a 'critical' set. A second diversity stage iteratively adds tokens with minimal maximum cosine similarity to the current set until the token budget K is reached. Experiments on LLaVA-1.5-7B and LLaVA-NeXT-7B at budgets of 192/128/64 and 320 tokens compare CRISP with 10+ baselines, reporting average retention rates up to 99.8% and more than 2x decoding speedup.

Significance. Text-driven pruning before the LLM is a sensible and useful design point: it avoids the text-agnostic nature of prior pre-LLM heuristics and the efficiency cost and noisy attention of intra-LLM pruning. The paper provides broad comparisons on two LVLMs, ablations of both stages, and efficiency measurements. If the reported retention rates are robust, CRISP would be a practical plug-and-play module. The main weaknesses are a selectively averaged headline metric, a hyperparameter selected on the evaluation benchmarks, and an unvalidated cosine-relevance assumption; these are addressable but currently make the central empirical claim stronger than fully supported.

major comments (4)
  1. [§IV-B, Tables I–II] The headline 'AVG.%' is undefined and silently excludes MM-Vet. From the numbers in Table I, CRISP at 128 tokens has per-benchmark retention rates whose mean over the eight non-MM-Vet benchmarks is 99.5%, matching the table; including MM-Vet raises the mean to about 100.1%. Since MM-Vet is listed as a benchmark in §IV-A.2 and appears in the same table, the exclusion is unexplained and changes the headline. Please define AVG, report it with and without MM-Vet, or justify the exclusion.
  2. [§IV-C, Table III] The only free hyperparameter alpha is selected on the same benchmarks used for the main results: Table III sweeps alpha over MME, GQA, SQA, POPE, MMB and chooses 0.3 as 'best overall'; these benchmarks (plus others) appear in Tables I–II. This is test-set model selection and inflates the comparison. Please select alpha on a held-out validation set or show that the relative ranking over baselines is stable across alpha in [0.1, 0.9]. Also justify the separate rule that alpha is set to 0.5 when K < 100.
  3. [§III-B.2, Eq. (2)] The method's text-driven claim rests on cosine similarity between projected visual tokens and word-level embeddings in the LLM input space. A shared embedding space does not by itself guarantee that cosine ranking identifies answer-relevant image regions; patch tokens are trained through a projector and language-model loss, not as a visual-semantic index for arbitrary query nouns. The ablation 'w/o stage-1 selection' shows that some word-conditioned selection helps, but it does not validate the specific cosine ranking, and the CLS-attention fallback for noun-less queries is called 'rare' without a reported frequency. Please validate Eq. (2) against full-model cross-modal attention or human-annotated regions on a sample, report how often the fallback triggers, and compare against an alternative text-conditioned relevance score.
  4. [§IV-D, Table IV; Abstract] The abstract says CRISP 'reduc[es] inference cost and latency by more than 2 times.' Table IV shows a >2x decoding-speedup for both models, but end-to-end total time improves by only 1.48x (LLaVA-1.5, 128 tokens) and 1.87x (LLaVA-NeXT, 640 tokens). The 2x total-time speedup holds only for LLaVA-NeXT at 320 tokens. Please state whether 'latency' refers to decoding speed or end-to-end time and qualify the claim accordingly.
minor comments (4)
  1. [§IV-A.1] Please state whether all evaluations use greedy decoding and are deterministic. If they are deterministic, a brief note would clarify why repeated-run variance is not reported; if not, multiple seeds or standard errors are needed.
  2. [General] The paper would benefit from a short pseudo-code or algorithm block for the two-stage selection, since the description is spread over Eqs. (1)–(5) and the text.
  3. [Tables I–II] The AVG% rows should be annotated with a formula. As written, it is not clear whether MME's raw score or its percentage retention is averaged, and whether MM-Vet is included.
  4. [§IV-A.1] The fallback to CLS attention for noun-less queries is described as 'rare'; please quantify it on the evaluation benchmarks, since it affects the interpretation of the text-driven component.

Circularity Check

0 steps flagged

No significant circularity: CRISP's heuristic pruning pipeline is evaluated against external benchmarks and does not reduce to its inputs.

full rationale

The paper's derivation chain is a stated heuristic, not a fitted equation relabeled as a prediction. Stage 1 (Eq. 2) defines cross-modal relevance as cosine similarity between projected visual tokens and word embeddings; this is an explicit modeling assumption, not a parameter fitted to the evaluation data. Stage 2 (Eq. 5) is a standard max-min diversity selection rule. The only tunable quantity is alpha in Eq. 1, which the paper reports as a default and ablates across values; choosing a hyperparameter on the same benchmarks is a potential data-snooping concern, but it does not make the central performance-retention claim equivalent to its inputs by construction. There are no load-bearing self-citations by the present authors, and no uniqueness theorem or prior-work ansatz is invoked to force the design. The headline results are measured against external benchmarks (VQA-v2, GQA, POPE, MME, etc.) with matched token budgets, so the empirical claims are independently checkable. The skeptic's concern—that cosine alignment between patch-level visual tokens and word-level embeddings is unvalidated as a relevance proxy—is a correctness/robustness risk about external validity, not a circularity in the paper's own reasoning. No passage in the manuscript asserts a limitation or missing support that would change this verdict.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central claim rests on several alignment and coverage assumptions: token/word embedding cosine similarity, noun grounding, CLS importance, and diversity as context coverage. The only explicit free parameter is alpha, tuned on the evaluation benchmarks. No new entities are introduced.

free parameters (1)
  • alpha (critical-stage budget ratio) = 0.3 default; 0.5 when K < 100
    Controls the split between text-relevant selection and diversity completion. Selected from ablation (Table III) on the same benchmarks used for the main results.
axioms (5)
  • domain assumption Cosine similarity in the shared embedding space reflects visual-text relevance (Eq. 2)
    Stage 1 ranking assumes the projector output and LLM word embeddings are aligned enough for cosine similarity to rank answer-relevant visual tokens.
  • domain assumption Nouns extracted by spaCy are the content-bearing visual grounding cues
    If the query's key visual referent is not a noun (e.g., a color or a relation), Stage 1 misses it; the fallback only handles the empty-noun case.
  • domain assumption CLS attention from the vision encoder reflects global semantic importance
    Used for quota allocation and fallback selection; inherited from FasterVLM/HiRED assumptions and not independently validated here.
  • domain assumption Greedy max-similarity diversity preserves essential scene context
    Stage 2 adds tokens farthest from the selected set; there is no guarantee this retains layout, inter-object relationships, or other answer-critical context.
  • domain assumption Standard benchmarks are reliable, noise-free measures of performance
    Headline numbers are single-run scores without error bars; differences between methods are often under one percentage point.

pith-pipeline@v1.3.0-alltime-deepseek · 10885 in / 11765 out tokens · 94224 ms · 2026-08-02T03:37:19.164916+00:00 · methodology

0 comments
read the original abstract

Large Vision-Language Models (LVLMs) typically require processing hundreds to thousands of visual tokens, leading to substantial inference overhead. Existing visual token pruning methods either operate before the LLM using text-agnostic heuristics or prune inside the LLM at the cost of efficiency and noisy cross-modal attention. To address these limitations, we propose CRISP, a pre-LLM yet text-driven visual token pruning framework that preserves both instruction-relevant evidence and essential scene context. CRISP works in a two-stage pipeline: Stage 1 first identifies text-aligned visual tokens, and Stage 2 enhances contextual completeness through semantic diversity. Extensive experiments on LLaVA-1.5 and LLaVA-NeXT demonstrate that CRISP achieves superior performance retention under aggressive pruning ratios, maintaining up to 99.5% accuracy while reducing inference cost and latency by more than 2 times. CRISP serves as a practical solution for efficient LVLM inference, especially in resource-constrained scenarios.

Figures

Figures reproduced from arXiv: 2607.16326 by Baoquan Zhao, Juncen Guo, Mengyang Zhao, Rui Zhu, Wei Zhou, Xiaolei Chen, Xu Li, Yi Zheng, Yuxuan Liang, Zhe Liu.

Figure 1
Figure 1. Figure 1: Comparison of visual token pruning behaviors under different textual [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy–efficiency trade-off when pruning at different LLM layers. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the CRISP workflow. Blue modules represent the [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 5 linked inside Pith

  1. [1]

    A survey of state of the art large vision language models: Benchmark evaluations and challenges,

    Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, and G. Shi, “A survey of state of the art large vision language models: Benchmark evaluations and challenges,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 1587–1606

    2025

  2. [2]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,

    L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 19–35

    2024

  3. [3]

    [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster,

    Q. Zhang, A. Cheng, M. Lu, Z. Zhuo, M. Wang, J. Cao, S. Guo, Q. She, and S. Zhang, “[cls] attention is all you need for training-free visual token pruning: Make vlm inference faster,”arXiv e-prints, pp. arXiv– 2412, 2024

    2024

  4. [4]

    Stop looking for important tokens in multimodal language models: Duplication matters more,

    Z. Wen, Y . Gao, S. Wang, J. Zhang, Q. Zhang, W. Li, C. He, and L. Zhang, “Stop looking for important tokens in multimodal language models: Duplication matters more,”arXiv preprint arXiv:2502.11494, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Visionzip: Longer is better but not necessary in vision language models,

    S. Yang, Y . Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia, “Visionzip: Longer is better but not necessary in vision language models,” inPro- ceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 19 792–19 802

    2025

  6. [6]

    Divprune: Diversity- based visual token pruning for large multimodal models,

    S. R. Alvar, G. Singh, M. Akbari, and Y . Zhang, “Divprune: Diversity- based visual token pruning for large multimodal models,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 9392–9401

    2025

  7. [7]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference,

    Y . Zhang, C.-K. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. A. Gudovskiy, T. Okuno, Y . Nakata, K. Keutzeret al., “Sparsevlm: Visual token sparsification for efficient vision-language model inference,” in Forty-second International Conference on Machine Learning, 2025

    2025

  8. [8]

    Which experimental design is better suited for vqa tasks?: Eye tracking study on cognitive load, performance, and gaze allocations,

    S. A. Vriend, S. Vidyapu, A. Rama, K.-T. Chen, and D. Weiskopf, “Which experimental design is better suited for vqa tasks?: Eye tracking study on cognitive load, performance, and gaze allocations,” inProceed- ings of the 2024 Symposium on Eye Tracking Research and Applications, 2024, pp. 1–7

    2024

  9. [9]

    Making the invisible visible: Verbal but not visual cues enhance visual detection,

    G. Lupyan and M. J. Spivey, “Making the invisible visible: Verbal but not visual cues enhance visual detection,”PloS one, vol. 5, no. 7, p. e11452, 2010

    2010

  10. [10]

    Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms,

    Q. Zhang, A. Cheng, M. Lu, R. Zhang, Z. Zhuo, J. Cao, S. Guo, Q. She, and S. Zhang, “Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 20 857–20 867

    2025

  11. [11]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,”arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 296–26 306

    2024

  13. [13]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  14. [14]

    A-vl: Adaptive attention for large vision-language models,

    J. Zhang, M. Yuan, R. Zhong, P. Luo, H. Zhan, N. Zhang, C. Hu, and X.-Y . Li, “A-vl: Adaptive attention for large vision-language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 21, 2025, pp. 22 461–22 469

    2025

  15. [15]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models,

    Y . Shang, M. Cai, B. Xu, Y . J. Lee, and Y . Yan, “Llava-prumerge: Adaptive token reduction for efficient large multimodal models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 22 857–22 867

    2025

  16. [16]

    Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models,

    K. H. I. Arif, J. Yoon, D. S. Nikolopoulos, H. Vandierendonck, D. John, and B. Ji, “Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 2, 2025, pp. 1773– 1781

    2025

  17. [17]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction,

    L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y . Zang, Y . Cao, C. He, J. Wang, F. Wuet al., “Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction,”arXiv preprint arXiv:2410.17247, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Token merging: Your vit but faster,

    D. Bolya, C.-Y . Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman, “Token merging: Your vit but faster,” inICLR, 2023

    2023

  19. [19]

    Less is more: A simple yet effective token reduction method for efficient multi-modal llms,

    D. Song, W. Wang, S. Chen, X. Wang, M. X. Guan, and B. Wang, “Less is more: A simple yet effective token reduction method for efficient multi-modal llms,” inProceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 7614–7623

    2025

  20. [20]

    Multi-stage vision token dropping: Towards efficient multimodal large language model,

    T. Liu, L. Shi, R. Hong, Y . Hu, Q. Yin, and L. Zhang, “Multi-stage vision token dropping: Towards efficient multimodal large language model,” arXiv preprint arXiv:2411.10803, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    en core web sm: spacy english small model,

    E. AI, “en core web sm: spacy english small model,” https://spacy.io/models/en, 2023, spaCy English pipeline “en core web sm”, accessed 2025-12-04

    2023

  22. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6904–6913

    2017

  23. [23]

    Vizwiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “Vizwiz grand challenge: Answering visual questions from blind people,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3608–3617

    2018

  24. [24]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering,

    D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and compositional question answering,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6700–6709

    2019

  25. [25]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,”Advances in Neural Information Processing Systems, vol. 35, pp. 2507–2521, 2022

    2022

  26. [26]

    Towards vqa models that can read,

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8317–8326

    2019

  27. [27]

    Evaluating object hallucination in large vision-language models,

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 292–305

    2023

  28. [28]

    Mme: A comprehensive evaluation benchmark for multimodal large language models,

    C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y . Wu, and R. Ji, “Mme: A comprehensive evaluation benchmark for multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.13394

    Pith/arXiv arXiv 2024

  29. [29]

    Mmbench: Is your multi-modal model an all-around player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liuet al., “Mmbench: Is your multi-modal model an all-around player?” inEuropean Conference on Computer Vision. Springer, 2025, pp. 216–233

    2025

  30. [30]

    Mm-vet: evaluating large multimodal models for integrated capabilities,

    W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “Mm-vet: evaluating large multimodal models for integrated capabilities,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 57 730–57 754

    2024