Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Speculative Decoding Reimagined for Multimodal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MSD speeds up multimodal LLM inference by redesigning the draft model to treat image and text tokens differently and by training it first on text, then on image-text data.

desk verdict Solid incremental extension of EAGLE-style speculative decoding to MLLMs with two well-ablated design choices; the speedup numbers need the missing timing parameters before they can be verified. read the letter →

arxiv 2505.14260 v1 pith:ONRKDI7P submitted 2025-05-20 cs.CV cs.AI

Luxi Lin , Zhihang Lin , Zhanpeng Zeng , Rongrong Ji This is my paper · ORCID
classification cs.CVcs.AI
keywords speculativedecodingmultimodallargelanguagemodelsdraftmodelinferenceaccelerationvisualtokenstwo-stagetraininglossless
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up text models by having a small draft model propose several tokens at once and a large model verify them in parallel, without changing the output distribution. This paper argues that the same recipe underperforms on multimodal models because the draft model treats image tokens like text and is not trained on both modalities. The proposed method, MSD, feeds image tokens into the draft model directly as embeddings while keeping the standard text-token concatenation, and trains the draft model first on text-only instruction data, then on a gradually increasing mix of visual instruction data. On LLaVA-1.5-7B and 13B it reports up to 2.29x and 2.46x speedups across multimodal benchmarks, with average acceptance length rising from about 3.4 to 4.4 on the core ablation tasks. If correct, MSD gives a lossless way to make visual question-answering and chatbot models faster without retraining or changing the target model.

What carries the argument

The load-bearing object is the MSD draft model, a lightweight single-transformer-layer feature predictor in the style of feature-level speculative decoding. Its input rule is modality-dependent: for text tokens, it concatenates the target model's hidden state with the next-token embedding and down-projects the pair; for visual tokens, it passes the projected visual embedding directly, with no next-token concatenation. Around this sits a two-stage training schedule: stage one trains on text-only ShareGPT instruction data, and stage two interpolates linearly between text and visual instruction data over epochs. That combination is what the ablations attribute most of the acceptance-length gain to.

What would settle it

Train MSD's draft model with the same two-stage curriculum but without token decoupling—i.e., concatenate every visual token's hidden state with the next-token embedding before entering the draft transformer—and measure average acceptance length on the same 100-sample subsets. If that variant matches or exceeds MSD's reported 4.43 average on ChartQA, AI2D, MMB, and Hallusion, the decoupling mechanism is not doing the work the paper assigns to it.

Watch

Extended reading notes

Core claim

The central claim is that feature-level speculative decoding, which works well for LLMs, must be reimagined when the input includes images. MSD's draft model processes text and visual tokens with different input rules: text tokens are represented by concatenating the target model's hidden state with the next-token embedding, while visual tokens are fed directly as their original embeddings, because visual tokens are mutually related rather than sequentially dependent. The draft model is then trained in two stages: first on a text-only instruction-tuning dataset to build language modeling ability, then on a mixture of text and visual instruction data that shifts progressively toward visual data over epochs. On LLaVA-1.5-7B and 13B, this raises average acceptance length by about one token per cycle over the feature-level baseline and gives speedups up to 2.29x and 2.46x, respectively, while speculative verification preserves the target output distribution exactly.

Load-bearing premise

The result depends on the premise that image tokens are mutually related rather than sequentially dependent, so feeding them directly as embeddings is better than concatenating them with the next-token embedding during drafting.

Editorial extensions

If this is right

  • MSD's speedups are lossless: the target model's sampling distribution is conserved by the verification step, so no accuracy is traded for latency.
  • The average acceptance length on LLaVA-1.5-7B rises from 3.43 to 4.43 on the four ablation tasks, nearly closing the gap to LLM draft acceptance lengths near 5.
  • Both design changes contribute independently: token decoupling adds about +0.45 acceptance length and two-stage training about +0.74, for +1.00 combined.
  • Because measured speedups hold at both temperature 0 and temperature 1, the method generalizes across greedy and stochastic sampling settings.
  • The same acceptance lengths lead to lower speedup ratios on MLLM benchmarks than on LLM benchmarks because visual inputs are longer and outputs shorter; MSD narrows, but does not eliminate, this gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run: swap the vision encoder for a causal one; if visual tokens then behave like text, the direct-embedding rule should lose its advantage, which would localize the source of MSD's gain to bidirectionality rather than to modality per se.
  • The paper does not sweep the shape of the curriculum in Eq. (10). Varying the interpolation schedule (e.g., step functions vs linear) would show how much of the gain is due to gradualism rather than to the final data mixture.
  • Since MSD only changes the draft model, in principle it composes with lossy accelerators such as visual-token pruning; the interaction is untested and could either amplify or erode the speedup.
  • Scaling to larger MLLMs or to audio tokens is flagged by the authors as future work; a testable extension is to re-run the same two-stage recipe with audio features in place of visual embeddings, where the paper predicts different token-processing rules may be needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Multimodal Speculative Decoding (MSD), a speculative decoding method for Multimodal Large Language Models (MLLMs). The two central design principles are: (1) text and visual tokens should be processed separately in the draft model—text tokens use the EAGLE-style concatenation of hidden state and next-token embedding, while visual tokens are fed in directly as embeddings—and (2) the draft model should be trained with a two-stage strategy: first on text-only instruction data, then progressively mixing in multimodal instruction data. Experiments on LLaVA-1.5-7B and LLaVA-1.5-13B across eight multimodal benchmarks report average acceptance lengths up to 4.95 and speedup ratios up to 2.29x and 2.46x, respectively, outperforming the Baseline, Lookahead, and TextOnly methods. The paper also includes ablations isolating the contributions of token decoupling and the two-stage training schedule.

Significance. The paper addresses a real and current gap: speculative decoding for MLLMs lags behind its LLM counterpart. If the reported speedups hold, MSD is a useful, practical contribution with a clear design rationale and a released implementation. The two-stage training strategy and the decoupled visual-token handling are supported by internal ablations (Tables 4 and 5), rather than being fitted to reproduce the headline numbers, which is a methodological strength. The main weakness is that the central speedup claim depends on timing quantities that are not reported, so the headline ratios cannot be independently checked from the manuscript. With the missing details supplied, the work would be reproducible and likely valuable to the multimodal inference community.

major comments (4)
  1. [§5.1.3, Eq. (12), Table 3] The speedup ratios in Table 3 are computed from Eq. (12), which depends on N, γ, Tq, Tp, Tv(γ), α, and T_profiling. Of these, only α and N are reported (and N only as an aggregate average, not per benchmark). The speculative length γ, the per-token target time Tp, the per-token draft time Tq, the verification time Tv(γ), and the profiling time T_profiling are never given. Because the paper itself notes that MLLM benchmarks have long inputs (average 624 tokens) and short outputs (average 85 tokens), T_profiling is a large additive term in both the numerator and denominator of Eq. (12); small differences in the unstated profiling or draft times can move SR by tens of percent. The headline 'up to 2.29x / 2.46x' claims are therefore not independently verifiable from the manuscript. Please report the actual γ and timing values used, and ideally the raw measured per-benchmark times, so that Eq. (12) can be recomputed.
  2. [§5.1.2] All benchmark results are based on 100 randomly sampled examples per task with no repeated runs, no standard deviations, and no significance tests. Acceptance length and speedup ratio are random quantities; with n=100, differences of 0.1–0.2 in τ between methods (e.g., the MMB EN row for LLaVA-1.5-7B at T=0, Baseline 3.59 vs MSD 4.62) may or may not be statistically reliable. The claim that MSD 'clearly outperforms other methods across all tasks' would be much stronger with confidence intervals, multiple seeds, or at least the per-sample variance. Please add error bars or additional evaluation data.
  3. [§4.1] The stated rationale for decoupling visual tokens is that visual tokens 'are related to all other visual tokens in the sequence' because the vision encoder (CLIP) uses bidirectional attention, and therefore the EAGLE-style concatenation of the current hidden state with the next visual token embedding is inappropriate. This premise is imprecise: while CLIP is bidirectional, by the time visual tokens enter LLaVA's causal transformer and the draft model's causal attention, they are processed under causal masking, so the 'no sequential dependency' characterization does not strictly hold for the representations used by the draft model. The Table 5 ablation does provide direct empirical support for decoupling, so this is not a fatal flaw, but the explanation should be revised to avoid an inaccurate mechanistic claim.
  4. [§4.2, Eq. (10), §5.1.1] The progressive multimodal data mixing schedule in Eq. (10) is under-specified. The variables T and t are not defined precisely (presumably epoch numbers, but the paper does not say whether t runs from 0 or 1, how many epochs are in Stage 1 versus Stage 2, or whether the '20 epochs' reported in §5.1.1 refers to the total across both stages or to each stage separately). The mixing rule D_t^mix = D_text*(T−t)/T + D_visual*t/T also concatenates datasets rather than sampling proportions, and the resulting per-epoch dataset size is unclear. Please specify the schedule fully so that the training procedure is reproducible.
minor comments (5)
  1. [§5.1.3] In Eq. (12), 'Tprof iling' is typeset with an unintended space; please fix the spacing consistently across the paper.
  2. [Figure 3] The figure label 'Text-only and Multimodel Dataset' contains a typo: 'Multimodel' should be 'Multimodal'.
  3. [§5.2] The sentence 'We statistic the average input and output lengths' should read 'We compute the average input and output lengths' or 'We report the statistics of the average input and output lengths'.
  4. [Table 5 and §5.3] The definitions of 'Visual Feature' versus 'Visual Embedding' in the input decoupling ablation are only described in the prose, but the table would be clearer if the exact input representation for each row were restated in the caption; the current caption relies on the text, and a reader scanning the table may misinterpret the two rows.
  5. [§5.2, Figure 6] The chatbot example in Figure 6 compares token counts (27, 62, 77) for the same decoding time, but it does not state the decoding time or hardware; adding this context would make the illustrative speed comparison more meaningful.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: MSD's design claims are supported by ablations against external targets, and the unstated timing parameters in Eq. (12) are a reproducibility gap rather than a self-referential construction.

full rationale

The paper's load-bearing claims are empirical speedups and acceptance lengths measured against external LLaVA-1.5 targets and public benchmarks. The speedup ratio in Eq. (12) is a standard decomposition; no parameter is fitted so that the equation reproduces the reported 2.29x/2.46x numbers, and the reported average acceptance lengths (Tables 1-3) are measured quantities rather than outputs defined by the claim. The two design principles are motivated by qualitative analysis (Sec. 4.1-4.2) and then tested directly by ablations (Tables 4-5, Figure 5), so even if the CLIP 'bidirectional' rationale is informal, the design choice does not reduce to its own motivation. The self-citations present (Ref. [16], [30]) are peripheral benchmark and related-work mentions and are not load-bearing. The main weakness is that Eq. (12) depends on gamma, T_q, T_p, T_v, and T_profiling, none of which are reported, making Table 3 hard to reproduce; this is an under-specification and verification gap, not circularity. The limitations section also honestly states that evaluation covers only two model sizes. No circular step meeting the evidence bar was found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated. The central claim relies on hand-picked training and evaluation choices and on two domain assumptions about token modality and lossless verification; the main empirical support is ablations.

free parameters (4)
  • Progressive mixing schedule = linear over T=20 epochs; equal 68k sample counts
    Eq. 10 defines the schedule; the paper reports no sensitivity analysis to T or mixing rate, yet the two-stage training claim depends on it.
  • Speculative length gamma = not stated
    Used in Eqs. 11-12 and in tree drafting; the reported speedup ratios depend on this value, which is never given.
  • Evaluation subset size = 100 random samples per benchmark
    Sec 5.1.2 fixes 100 samples; acceptance lengths and speedups are measured on these subsets only.
  • Draft model training hyperparameters = LR 5e-5, batch 4, 4 GPUs, 20 epochs
    Sec 5.1.1; chosen by hand and not varied.
assumptions (4)
  • standard math Speculative decoding with standard rejection sampling reproduces the target distribution exactly.
    Invoked in Sec 3.2 verification phase; underlies the lossless claim.
  • domain assumption Visual tokens in LLaVA are non-sequential and globally related, so direct embedding input is appropriate.
    Sec 4.1 claims CLIP's bidirectional attention makes visual tokens related to all others; after projection into LLaVA's causal transformer the premise is not fully accurate.
  • domain assumption Feature-level draft models are better than token-level in MLLMs.
    Supported by Table 1 but only on three benchmarks and one target model.
  • domain assumption Training on both text-only and visual instruction data is needed, and progressive mixing preserves language ability.
    Supported by Tables 2 and 4 and Figure 5 for LLaVA-1.5-7B; may not transfer to other MLLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speculative Decoding Reimagined for Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/ONRKDI7P

@misc{pith2026250514260,
  author       = {Pith},
  title        = {Pith review of: Speculative Decoding Reimagined for Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ONRKDI7P}},
  note         = {Machine review of arXiv:2505.14260}
}
abstract

This paper introduces Multimodal Speculative Decoding (MSD) to accelerate Multimodal Large Language Models (MLLMs) inference. Speculative decoding has been shown to accelerate Large Language Models (LLMs) without sacrificing accuracy. However, current speculative decoding methods for MLLMs fail to achieve the same speedup as they do for LLMs. To address this, we reimagine speculative decoding specifically for MLLMs. Our analysis of MLLM characteristics reveals two key design principles for MSD: (1) Text and visual tokens have fundamentally different characteristics and need to be processed separately during drafting. (2) Both language modeling ability and visual perception capability are crucial for the draft model. For the first principle, MSD decouples text and visual tokens in the draft model, allowing each to be handled based on its own characteristics. For the second principle, MSD uses a two-stage training strategy: In stage one, the draft model is trained on text-only instruction-tuning datasets to improve its language modeling ability. In stage two, MSD gradually introduces multimodal data to enhance the visual perception capability of the draft model. Experiments show that MSD boosts inference speed by up to $2.29\times$ for LLaVA-1.5-7B and up to $2.46\times$ for LLaVA-1.5-13B on multimodal benchmarks, demonstrating its effectiveness. Our code is available at https://github.com/Lyn-Lucy/MSD.

Figures

Figures reproduced from arXiv: 2505.14260 by the authors.

Figure 1
Figure 1. Comparison of models trained on text-only and vision [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Input illustration of MLLMs. The input to MLLMs [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The framework of Multimodal Speculative Decoding (MSD). The left illustrates the draft phase, while the right illustrates the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of acceptance rates between Baseline and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Impact of draft model training strategies, evaluated on a [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Chatbot responses using LLaVA-1.5-7B without specu [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

    cs.CV 2026-03 conditional novelty 7.0 of 10

    A small tool-free MLLM plus answer-separability gating bypasses agentic tool loops for many queries, yielding 1.1–3.35× speedup with preserved or higher accuracy.

  2. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

  3. HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A training-free, two-stage speculative decoding scheme accelerates VLM document parsers by ~2.8x end-to-end (up to 7x) while keeping parsing accuracy essentially unchanged.

Reference graph

Works this paper leans on

51 extracted references · 41 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2023. 1

    work page 2023

  2. [2]

    Sharegpt

    anon8231489123. Sharegpt. https://huggingface. co / datasets / anon8231489123 / ShareGPT _ Vicuna _ unfiltered, 2023. Dataset available on Hugging Face. 2, 6

    work page 2023

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 3

    work page 2023

  4. [4]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. 1

    work page 2025

  5. [5]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Ja- son D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decod- ing heads, 2024. 2, 3

    work page 2024

  6. [6]

    Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer

    Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yansong Tang, Jiwen Lu, and Tao Chen. Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer. In CVPR, 2024. 3

    work page 2024

  7. [7]

    Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding.arXiv preprint arXiv:2408.11049, 2024

    Jian Chen, Vashisth Tiwari, Ranajoy Sadhukhan, Zhuom- ing Chen, Jinyuan Shi, Ian En-Hsu Yen, and Beidi Chen. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding.arXiv preprint arXiv:2408.11049, 2024. 7

    arXiv 2024

  8. [8]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, 2024. 3

    work page 2024

Show all 51 references
  1. [9]

    Diffrate : Differentiable compression rate for efficient vision transformers

    Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate : Differentiable compression rate for efficient vision transformers. In ICCV, 2023. 1, 3

    2023

  2. [10]

    Evalu- ating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 8

    2021 arXiv

  3. [11]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 1 10

    2015 arXiv

  4. [12]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR,

  5. [13]

    Train- ing verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 8

    arXiv

  6. [14]

    Flashattention-2: Faster attention with better paral- lelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better paral- lelism and work partitioning. In ICLR, 2023. 3

    2023

  7. [15]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022. 3

    2022

  8. [16]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2023. 2, 6, 7

    2023

  9. [17]

    Break the sequential dependency of llm inference using lookahead decoding

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057, 2024. 2, 7

    2024 arXiv

  10. [18]

    On speculative de- coding for multimodal large language models, 2024

    Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. On speculative de- coding for multimodal large language models, 2024. 1, 2, 3, 7, 8

    2024

  11. [19]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In CVPR, pages 6904–6913, 2017. 1, 2, 6, 7, 8

    2017

  12. [20]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR,...

    2024

  13. [21]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235–251, 2016. 2, 6, 7

    2016

  14. [22]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP,

  15. [23]

    Fast in- ference from transformers via speculative decoding, 2023

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast in- ference from transformers via speculative decoding, 2023. 1, 3

    2023

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 1

    2023

  17. [25]

    Mbq: Modality-balanced quantization for large vision-language models, 2025

    Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Mbq: Modality-balanced quantization for large vision-language models, 2025. 1, 3

    2025

  18. [26]

    Tokenpacker: Efficient visual projector for multimodal llm, 2024

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm, 2024. 3

    2024

  19. [27]

    EAGLE-2: Faster inference of language models with dy- namic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dy- namic draft trees. In EMNLP, 2024. 2, 3

    2024

  20. [28]

    EAGLE: Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In ICML, 2024. 2, 3, 4, 6, 7

    2024

  21. [29]

    Moe-llava: Mixture of experts for large vision- language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision- language models. arXiv preprint arXiv:2401.15947 , 2024. 1, 3

    2024 arXiv

  22. [30]

    Boosting multimodal large language models with visual to- kens withdrawal for rapid inference

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual to- kens withdrawal for rapid inference. In AAAI, 2024. 3

    2024

  23. [31]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 1, 3, 6, 8

    2023

  24. [32]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, January 2024. 1

    2024

  25. [33]

    Mmbench: Is your multi-modal model an all-around player?, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?, 2023. 2, 6, 7

    2023

  26. [34]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, pages 2507–2521, 2022. 7

    2022

  27. [35]

    ChartQA: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, and Enamul Joty, Shafiq an Hoque. ChartQA: A benchmark for question an- swering about charts with visual and logical reasoning. In ACL, 2022. 1, 2, 6, 7, 8

    2022

  28. [36]

    Mm1: Methods, analysis and in- sights from multimodal llm pre-training

    Brandon McKinzie et al. Mm1: Methods, analysis and in- sights from multimodal llm pre-training. In ECCV, 2024. 1, 3

    2024

  29. [37]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In ICCV,

  30. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748–...

    2021

  31. [39]

    Crossget: cross-guided ensem- ble of tokens for accelerating vision-language transformers

    Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: cross-guided ensem- ble of tokens for accelerating vision-language transformers. In ICML, 2024. 3

    2024

  32. [40]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus 11 Rohrbach. Towards vqa models that can read. In CVPR,

  33. [41]

    Gemini: A family of highly capable multimodal models, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models, 2023. 1

    2023

  34. [42]

    Q-vlm: Post-training quanti- zation for large vision-language models

    Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Q-vlm: Post-training quanti- zation for large vision-language models. arXiv preprint arXiv:2410.08119, 2024. 1, 3

    2024 arXiv

  35. [43]

    Large multimodal model compression via iterative efficient prun- ing and distillation

    Maolin Wang, Yao Zhao, Jiajia Liu, Jingdong Chen, Chenyi Zhuang, Jinjie Gu, Ruocheng Guo, and Xiangyu Zhao. Large multimodal model compression via iterative efficient prun- ing and distillation. In WWW, 2024. 1, 3

    2024

  36. [44]

    Ppt: Token pruning and pooling for efficient vision transformers, 2023

    Xinjian Wu, Fanhu Zeng, Xiudong Wang, Yunhe Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers, 2023. 1, 3

    2023

  37. [45]

    mplug-owl: Modularization empowers large language mod- els with multimodality, 2024

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Jun- feng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language mod- e...

    2024

  38. [46]

    Generation meets verification: Accel- erating large language model inference with smart parallel auto-correct decoding

    Hanling Yi, Feng Lin, Hongbin Li, Ning Peiyang, Xiaotian Yu, and Rong Xiao. Generation meets verification: Accel- erating large language model inference with smart parallel auto-correct decoding. In ACL, 2024. 3

    2024

  39. [47]

    Draft& verify: Lossless large language model acceleration via self-speculative decoding

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In ACL, 2024. 3

    2024

  40. [48]

    Learning harmonized representations for speculative sampling, 2024

    Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling, 2024. 3

    2024

  41. [49]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023. 8

    2023

  42. [50]

    Tinyllava: A frame- work of small-scale large multimodal models.arXiv preprint arXiv:2402.14289, 2024

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A frame- work of small-scale large multimodal models.arXiv preprint arXiv:2402.14289, 2024. 3

    2024 arXiv

  43. [51]

    Llava-phi: Efficient multi-modal assistant with small language model

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. Llava-phi: Efficient multi-modal assistant with small language model. In EMCLR, 2024. 3 12

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.