Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

ViCA shows that visual tokens in a multimodal LLM can be frozen after projection and injected through a few cross-attention layers, preserving about 98% accuracy with about 4% of the visual computation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:32 UTC pith:NG6ZB7LZ

load-bearing objection ViCA is a genuinely new architectural result—freeze visual tokens, sparse cross-attention, retrain—with broad experiments, but the layer-selection protocol needs held-out validation before the 98%/4% claims are fully credible. the 3 major comments →

arxiv 2602.07574 v2 pith:NG6ZB7LZ submitted 2026-02-07 cs.CV cs.CL

ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention

Wenjie Liu , Hao Wu , Xin Qiu , Xudong Wang , Yingqi Fan , Yihan Zhang , Anhao Zhao , Yunpu Ma
show 1 more author
Xiaoyu Shen
This is my paper
classification cs.CV cs.CL
keywords multimodal large language modelsvision-language fusioncross-attentionefficient inferencevisual token pruningmodel accelerationlayer-wise redundancyarchitecture design
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the standard way multimodal LLMs fuse vision and language — running every image token through every self-attention and feed-forward layer — is largely wasted work. It claims that once visual embeddings are projected into the language model, they are already aligned enough that they never need to be updated again; the only place visual information actually changes the model's answer is a small set of layers where text attends to image tokens. To test this, the authors build ViCA, an architecture in which image tokens are frozen after projection and act purely as keys and values for sparse cross-attention at a few retained layers. Across three backbones and nine benchmarks, ViCA keeps about 98% of the full model's accuracy while cutting vision-side computation to about 4%, and it brings prefill latency close to text-only inference. If this holds, multimodal models can stop paying a quadratic tax on every image token, and efficiency gains come from architecture rather than from fragile per-token pruning decisions.

Core claim

The central claim, stated the way a sympathetic reader would state it: in a decoder-only multimodal LLM, visual tokens are read-only memory. After the vision projector, their representations are already in the language space; self-attention and FFN updates on those tokens change representations but rarely change the final output. The only interaction that matters is text-to-vision cross-attention in a small subset of layers, identified by a cosine-similarity diagnostic and verified by KL-divergence impact measurements. Removing all visual-token writes (freezing visual updates) and keeping cross-attention only at those layers preserves 97–99% of baseline accuracy across 3B, 7B, and 13B models

What carries the argument

The load-bearing mechanism is the read-only visual token: after the initial projection, visual tokens are never written by attention or FFN; they are statically stored as keys and values and are attended to by text queries only in a small set of preserved cross-attention layers. The preserved layers are chosen by measuring, layer by layer, how much text-token representations change after text-to-vision attention (cosine similarity) and how much that change propagates to the output distribution (KL divergence). This diagnostic turns a redundancy observation into a fixed architectural rule, and because the attention shape stays regular (short queries, long key–value sequences), standard fast-a

Load-bearing premise

The claim rests on the retained cross-attention layers being the right layers: they are selected from cosine-similarity diagnostics computed on a single benchmark (TextVQA) for each backbone (Appendix C.2), and that same benchmark is part of the nine used in evaluation — if the layer choices are dataset-specific or checkpoint-specific, the 98% accuracy / 4% compute result will not generalize.

What would settle it

Train or evaluate ViCA with the retained layers selected by the diagnostic on a different benchmark, then compare against the reported 98%-retention at 4% visual FLOPs on a held-out suite; if accuracy drops materially, the result is overfit to the diagnostic benchmark. A sharper test: recompute the cosine-similarity diagnostics on multiple independent checkpoints of the same backbone and check whether the selected layer sets are stable; if they diverge, the identified 'essential layers' are not a fixed property of the architecture.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multimodal prefill latency approaches text-only LLM latency, so serving a vision-language model costs nearly the same as serving the underlying language model.
  • Vision KV-cache shrinks to roughly 20–28% of the original across the three scales, reducing memory pressure for long-context and multi-image inputs.
  • Because ViCA removes whole compute paths instead of dropping tokens, it is complementary to token-pruning: adding a pruning method at inference drops visual compute to about 2% with only 1.8–3.7% additional accuracy loss.
  • The architecture's fixed, regular attention shapes map directly onto fast attention kernels, avoiding the kernel-reconfiguration and synchronization overheads that make pruning speedups nonlinear.
  • The pattern repeats across 3B, 7B, and 13B backbones, suggesting the redundancy is a structural property of self-attention-based fusion, not a quirk of one model size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The retained-layer sets were diagnosed on a single benchmark (TextVQA) and differ substantially between backbones (e.g., the 3B model keeps deep layers like 18–23 while the 7B and 13B models do not). If those sets are benchmark-specific or depend on the pretrained initialization, the 98%/4% trade-off may not transfer without re-running the diagnostic on target data — the paper does not test this.
  • A trainable extension could learn which layers to fuse rather than fixing them by a cosine-similarity pre-scan; this would remove the diagnostic dependency and might adapt layer choice per task or per input.
  • Since ViCA eliminates vision-to-vision attention, vision-side cost scales linearly with the number of image tokens rather than quadratically; the paper's FLOPs breakdown implies multi-image and high-resolution inference would benefit disproportionately, but the paper does not evaluate that regime explicitly.
  • Near-zero visual overhead also suggests that the bottleneck for many-image inputs shifts to the vision encoder and projector rather than the LLM; a testable extension is measuring ViCA latency and accuracy as image count grows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ViCA, an MLLM architecture in which visual tokens are frozen immediately after projection and bypass all self-attention and FFN updates, with vision-language interaction realized only through sparse text-to-vision cross-attention at a small set of per-backbone layers. The retained layers are identified through cosine-similarity and KL-divergence diagnostics computed on TextVQA (Appendix B.1, Fig. 6). The authors retrain ViCA variants of LLaVA-1.5-3B/7B/13B using the standard two-stage protocol and evaluate on nine benchmarks against 26 pruning baselines, reporting ~98% average relative accuracy with ~4% of original vision-side FLOPs, plus prefill latency speedups of 2.4x-3.6x in single-batch settings and ~10x at batch size 8. The paper also demonstrates orthogonality to token pruning via ViCA+PDrop.

Significance. If the central claims hold, ViCA is a valuable architectural simplification: it removes the dominant visual-side computation while retaining most accuracy, has a regular memory access pattern compatible with FlashAttention, and combines with token pruning. The empirical scope is broad (three backbones, nine benchmarks, 26 baselines), the FLOPs analysis is detailed and mostly parameter-derived, and the two-stage retraining protocol with public data makes the comparisons reproducible. The main caveat is that the per-backbone layer sets are selected using TextVQA diagnostics and then evaluated on a suite containing TextVQA, with no held-out validation of the selection procedure; the headline 98%/4% numbers therefore have an unquantified selection-bias risk. The paper does not report variance across seeds or checkpoints, which matters for a claim of near-parity with the baseline.

major comments (3)
  1. [§5.2 / Appendix C.2 / Table 8] The retained cross-attention layer sets (3B: {0,1,14,15,18,19,21,22,23}; 7B: {0,1,7,8,9,10,11,14}; 13B: {0,6,8,9,10,13,14,16}) are selected from cosine-similarity diagnostics computed on TextVQA (Appendix B.1, Fig. 6), and TextVQA is also one of the nine evaluation benchmarks (Table 2, Table 7). There is no held-out selection benchmark and no ablation that varies the selection data. Since Table 8 shows that retaining the non-essential cross-attention layers alone drops average accuracy to 63.0%, the selected set is clearly load-bearing. The paper should select layers on a validation split or a separate benchmark, and report the resulting accuracy retention across the full suite; without this, the 98%/4% generalization claim is not established.
  2. [Abstract / Table 4] The abstract and introduction state that ViCA yields "over 3.5x speedup in single-batch inference," but Table 4 reports a 2.4x speedup for LLaVA-1.5-3B (57.9 ms to 24.2 ms). The claim is only accurate for the 7B and 13B models (3.5x and 3.6x). The wording should be qualified (e.g., "3.5x for 7B/13B") or the abstract should state the range across backbones. This is an internal inconsistency in a headline quantity.
  3. [§5.2 / Tables 2, 6, 7] The central accuracy-retention numbers (97.8%, 98.7%, 97.0%) are point estimates with no error bars, repeated-seed variance, or confidence intervals. Given that the differences from the full baseline are small (e.g., 97.8% vs 100% in Table 2), it is unclear whether the retention claim is statistically meaningful. The paper should report at least 3 retraining seeds or bootstrap confidence intervals for the relative averages, and ideally for the per-benchmark numbers. This is especially important because layer selection itself is deterministic given the pretrained checkpoint, and a different seed or initialization could change both the selected layer set and the accuracy.
minor comments (5)
  1. [Abstract] Typo: "singe-batch" should be "single-batch".
  2. [Table 7] Column header "VQAText" is inconsistent with "VQAT" used in Tables 2, 6, and elsewhere.
  3. [Appendix C.3, Eq. (9)] The text says "visiual tokens" — should be "visual tokens."
  4. [§1 and §6] The introduction claims ">3.5x" speedup while the conclusion says "approximately 3.5x" — align these statements with Table 4.
  5. [Throughout] The model name is written inconsistently as "LLaV A" (with a space) in several places (e.g., Table captions and Figure 4) versus "LLaVA" / "LLaV A-1.5" elsewhere. Please unify.

Circularity Check

0 steps flagged

No significant circularity; ViCA's efficiency and accuracy claims are empirically and analytically self-contained, with only a non-circular test-set-selection risk.

full rationale

The central claims—98% relative accuracy retention and ~4% vision-side FLOPs—are not equivalent to any fitted input. The FLOPs ratios are computed from model dimensions and the modified attention graph (Appendix C.3, Eqs. 6–8 and 14–16), and the accuracy numbers come from retraining and evaluating on nine benchmarks (Tables 2, 6, 7). The sparse cross-attention layer sets (Appendix C.2) are chosen from cosine-similarity diagnostics on TextVQA (Fig. 6, Appendix B.1), and TextVQA is also one of the evaluation benchmarks. That is a selection-on-test-set/generalization risk, not a circularity: the cosine-similarity diagnostics are not an accuracy fit, the retained layers do not by construction determine the reported accuracies, and no equation in the paper reduces the predicted accuracy to the diagnostic values. The paper's self-citations (Fan et al. 2025; Zhao et al. 2025a; Han et al. 2025; Qiu et al. 2025; Su et al. 2024) are related-work support only and are not load-bearing for the architecture's correctness; there is no imported uniqueness theorem or ansatz from the authors' prior work. Per hard rule 1, no specific derivation step can be quoted as circular.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central efficiency claim rests on the empirical transfer of TextVQA-derived layer selections, the FLOPs counting convention, and the correctness of FlashAttention's short-query causal masking. No invented entities; no fitted numeric constants beyond the per-backbone layer sets.

free parameters (2)
  • Retained cross-attention layer sets per backbone = 3B: {0,1,14,15,18,19,21,22,23}; 7B: {0,1,7,8,9,10,11,14}; 13B: {0,6,8,9,10,13,14,16}
    Chosen from cosine-similarity diagnostics on TextVQA for each backbone (Appendix C.2). The number of layers (8-9) and the exact indices are data-dependent architectural choices; no theoretical criterion fixes them.
  • Text token count t_t=20 in FLOPs analysis = 20
    Fixed in Appendix C.3 for the numerical FLOPs ratios. The paper shows the relevant QK^TV fraction stays ~2.27% for t_t=0-40, so the reported percentages are robust to this choice, but it is still a manual constant in the calculation.
axioms (4)
  • domain assumption Cosine similarity and KL divergence are valid proxies for the functional importance of a module.
    Section 4.1 uses these metrics to rank layers and select key cross-attention layers. If representation change does not track output impact, the layer selection could miss critical layers.
  • domain assumption Visual tokens projected into the LLM are already aligned with the language space, making visual-token self-attention/FFN updates unnecessary.
    Empirically motivated in Section 4.1, but stated as a general property. The entire architecture relies on this alignment holding across backbones and image distributions.
  • standard math FlashAttention v2.1 bottom-right causal mask alignment correctly implements text-to-(text+visual) attention for the ViCA pattern.
    Appendix C.4 argues that when seqlen_q != seqlen_k and causal=True, aligning the mask to the bottom-right corner yields the intended attention where text queries see all visual tokens plus preceding text. The paper observes minor accuracy differences under this mode.
  • domain assumption Retraining from scratch with the standard two-stage LLaVA-1.5 protocol and identical iteration counts is a fair comparison to the baseline.
    Section 5.1 states this protocol; if the training recipe interacts with the frozen-visual architecture differently than with the dense baseline, the accuracy retention numbers could be misleading.

pith-pipeline@v1.3.0-alltime-deepseek · 32415 in / 11884 out tokens · 105121 ms · 2026-08-03T03:32:05.905166+00:00 · methodology

0 comments
read the original abstract

Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transformer layer, incurring substantial computational overhead. In this work, we revisit the necessity of such dense visual processing and show that projected visual embeddings are already well-aligned with the language space, while effective vision-language interaction occurs in only a small subset of layers. Based on these insights, we propose ViCA (Vision-only Cross-Attention), a minimal MLLM architecture in which visual tokens bypass all self-attention and feed-forward layers, interacting with text solely through sparse cross-attention at selected layers. Extensive evaluations across three MLLM backbones, nine multimodal benchmarks, and 26 pruning-based baselines show that ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4%, consistently achieving superior performance-efficiency trade-offs. Moreover, ViCA provides a regular, hardware-friendly inference pipeline that yields over 3.5x speedup in single-batch inference and over 10x speedup in multi-batch inference, reducing visual grounding to near-zero overhead compared with text-only LLMs. It is also orthogonal to token pruning methods and can be seamlessly combined for further efficiency gains. Our code is available at https://github.com/EIT-NLP/ViCA.

Figures

Figures reproduced from arXiv: 2602.07574 by Anhao Zhao, Hao Wu, Wenjie Liu, Xiaoyu Shen, Xin Qiu, Xudong Wang, Yihan Zhang, Yingqi Fan, Yunpu Ma.

Figure 1
Figure 1. Figure 1: , self-attention performs joint read–write over [V ; T]: visual tokens are updated via V2V attention and the visual FFN branch FFNV (·), while text tokens read vision through AT V and interact within text via AT T and FFNT (·). In contrast, cross-attention treats visual tokens as read-only memory: text queries attend to visual keys/values, while vi￾FFN Attn KV Self-attention Architecture V2V T2V T2T Vision… view at source ↗
Figure 2
Figure 2. Figure 2: Layer-wise diagnostics of visual-token updates in LLaVA-1.5-7B on TextVQA, showing output impact (Vis At￾tn/FFN KL) and representation change (Vis Attn/FFN 1-Cos). 0 5 10 15 20 25 30 Layer Index 10 4 10 3 10 2 10 1 Value (log scale) T2V (1-Cos) T2V (KL) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Layer-wise diagnostics of text–to-vision (T2V) cross￾attention in LLaVA-1.5-7B on TextVQA, showing output impact (T2V KL) and representation change (T2V 1-Cos). mentary metrics to capture representation change and output impact. Representation change is measured by the cosine similarity between token embeddings before and after a given module: Cos(h, h′ ) = h ⊤h ′ ∥h∥ ∥h ′∥ , where h and h ′ denote token e… view at source ↗
Figure 4
Figure 4. Figure 4: Common token dropping vs. our minimal architecture (ViCA). Left: Token dropping removes some visual tokens, but the remaining ones still undergo full self-attention and FFN updates across layers, incurring substantial visual computation. Right: ViCA removes visual update paths in attention and FFN. Visual tokens act only as KVs in a few cross-attention layers, while all other operations run on text tokens,… view at source ↗
Figure 5
Figure 5. Figure 5: Latency and speedup of forward pass on A6000 GPU with increasing batch sizes. Pre-training forward-pass latency of our model variants compared to the original baseline model (averaged over 100 samples). Speedup ratios are annotated on the stacked bars for LLaVA-1.5-3B, 7B, and 13B. ing the number of visual tokens from 576 to 27/24/19, respectively. Correspondingly, the visual KV-cache sizes are reduced to … view at source ↗
Figure 6
Figure 6. Figure 6: Token-wise mean cosine similarity on TextVQA (Singh et al., 2019) before and after key modules across LLaVA-1.5-3B, 7B,and 13B. Vision-Attn measures vision token similarity before and after attention, Vision-FFN measures vision token similarity before and after FFN, and Text-Attn-Vis measures text-token similarity before and after text–vision cross-attention. B. More Models Analysis B.1. Layerwise Redundan… view at source ↗
Figure 7
Figure 7. Figure 7: Computation breakdown of LLaVA-1.5-7B across question and vision token scales. Vis QK⊤V , Vis Projector, Vis FFN, Non-Visual, and TFLOPs denote the costs of visual-related attention multiplications, visual attention projections, visual FFN, non-visual computation, and total computation, respectively. Left: Vis QK⊤V stays at 2.27%. Equivalent Visual Token Count. To enable fair comparison, we report the equi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Recognition to Understanding: Unlocking Cognitive Time Series Reasoning with LLMs

    cs.CL 2026-06 unverdicted novelty 7.0

    Introduces the TSCognition benchmark for cognitive time series reasoning tasks and the TSAlign alignment framework, reporting outperformance over LLM, VLM, and time-series baselines on TSCognition and TimerBed with lo...

  2. Intrinsic and Triangulation-Agnostic Attention: A Simple and Powerful Approach for Learning on Meshes

    cs.GR 2026-07 conditional novelty 6.0

    Mass-weighted FEM attention on intrinsic mesh features is triangulation-agnostic and beats current mesh and point-cloud baselines on several geometry-learning benchmarks.

  3. SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

    cs.AR 2026-03 conditional novelty 6.0

    An FPGA accelerator for SkipGPT-style pruned LLMs that fuses router/normalization/attention computation to hide nonlinear latency, packs two FP16 products in each DSP, and serves reused KV entries from on-chip URAM to...

  4. ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

    cs.CV 2026-06 unverdicted novelty 5.0

    ViCoStream is a new coordinated pipeline framework for streaming VideoLLMs that achieves 134 FPS video throughput and less than 50 ms TTFT on A100 while keeping accuracy near full-history baselines.

Reference graph

Works this paper leans on

36 extracted references · 16 linked inside Pith · cited by 4 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    From llms to lrms: Rethinking pruning for reasoning-centric models.arXiv preprint arXiv:2601.18091,

    Ding, L., Zhao, A., Ye, F., Chen, Z., and Shen, X. From llms to lrms: Rethinking pruning for reasoning-centric models.arXiv preprint arXiv:2601.18091,

    arXiv

  3. [5]

    Visipruner: Decoding discontinuous cross-modal dynamics for efficient multimodal llms

    Fan, Y ., Zhao, A., Fu, J., Tong, J., Su, H., Pan, Y ., Zhang, W., and Shen, X. Visipruner: Decoding discontinuous cross-modal dynamics for efficient multimodal llms. In Proceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pp. 18896–18913,

    2025

  4. [6]

    The framework tax: Disparities between inference effi- ciency in NLP research and deployment

    Fernandez, J., Kahn, J., Na, C., Bisk, Y ., and Strubell, E. The framework tax: Disparities between inference effi- ciency in NLP research and deployment. In Bouamor, H., Pino, J., and Bali, K. (eds.),Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1588–1600, Singapore, December

    2023

  5. [7]

    Mme: A comprehen- sive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    Pith/arXiv arXiv

  6. [9]

    Filter, correlate, com- press: Training-free token reduction for mllm accelera- tion.arXiv preprint arXiv:2411.17686,

    Han, Y ., Liu, X., Zhang, Z., Ding, P., Chen, J., Wang, D., Chen, H., Yan, Q., and Huang, S. Filter, correlate, com- press: Training-free token reduction for mllm accelera- tion.arXiv preprint arXiv:2411.17686,

    arXiv

  7. [11]

    Training-free token pruning via zeroth-order gradient estimation in vision-language models.arXiv preprint arXiv:2509.24837,

    Kim, Y ., Zhang, Y ., Liu, H., Jung, A., Lee, S., and Hong, S. Training-free token pruning via zeroth-order gradient estimation in vision-language models.arXiv preprint arXiv:2509.24837,

    arXiv

  8. [12]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  9. [13]

    17 ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention C.3. Details of Calculation Equation of FLOPs To further explore the computation bottleneck, we refer to the theoretical formulations of vision-related computation proposed in prior works (Chen et al., 2025; Yang et al., 2025a; Liu et al., 2025a; Zhang et al., 2025a), and decompose the vi...

    2025

  10. [15]

    Liu, A., Tan, R., Gong, B., and Plummer, B. A. Fine-grained token allocation via operation pruning for efficient mllms. arXiv preprint arXiv:2507.02909, 2025a. Liu, H., Li, C., Li, Y ., and Lee, Y . J. Improved baselines with visual instruction tuning, 2023a. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tuning, 2023b. Liu, H., Li, C., Li, Y...

    arXiv 2024

  11. [16]

    The few govern the many: Unveiling few-layer dominance for time series models.arXiv preprint arXiv:2511.07237,

    Qiu, X., Tong, J., Sun, Y ., Ma, Y ., and Shen, X. The few govern the many: Unveiling few-layer dominance for time series models.arXiv preprint arXiv:2511.07237,

    arXiv

  12. [17]

    Growing a twig to accelerate large vision-language models.arXiv preprint arXiv:2503.14075,

    Shao, Z., Wang, M., Yu, Z., Pan, W., Yang, Y ., Wei, T., Zhang, H., Mao, N., Chen, W., and Yu, J. Growing a twig to accelerate large vision-language models.arXiv preprint arXiv:2503.14075,

    Pith/arXiv arXiv

  13. [18]

    Welm: A well-read pre-trained language model for chinese.arXiv preprint arXiv:2209.10372,

    Su, H., Zhou, X., Yu, H., Shen, X., Chen, Y ., Zhu, Z., Yu, Y ., and Zhou, J. Welm: A well-read pre-trained language model for chinese.arXiv preprint arXiv:2209.10372,

    Pith/arXiv arXiv

  14. [19]

    Unraveling the mystery of scaling laws: Part i.arXiv preprint arXiv:2403.06563,

    Su, H., Tian, Z., Shen, X., and Cai, X. Unraveling the mystery of scaling laws: Part i.arXiv preprint arXiv:2403.06563,

    Pith/arXiv arXiv

  15. [20]

    Sun, Y ., Xin, Y ., Li, H., Sun, J., Lin, C., and Batista-Navarro, R. T. Lvpruning: An effective yet simple language-guided vision token pruning approach for multi-modal large lan- guage models. InFindings of the Association for Com- putational Linguistics: NAACL 2025, pp. 4299–4308,

    2025

  16. [21]

    Tokencarve: Information-preserving visual token compression in multimodal large language models.arXiv preprint arXiv:2503.10501,

    Tan, X., Ye, P., Tu, C., Cao, J., Yang, Y ., Zhang, L., Zhou, D., and Chen, T. Tokencarve: Information-preserving visual token compression in multimodal large language models.arXiv preprint arXiv:2503.10501,

    Pith/arXiv arXiv

  17. [22]

    Flowcut: Rethinking redundancy via information flow for efficient vision-language models.arXiv preprint arXiv:2505.19536,

    Tong, J., Jin, W., Qin, P., Li, A., Zou, Y ., Li, Y ., Li, Y ., and Li, R. Flowcut: Rethinking redundancy via information flow for efficient vision-language models.arXiv preprint arXiv:2505.19536,

    arXiv

  18. [23]

    Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  19. [24]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,

    Pith/arXiv arXiv

  20. [25]

    Internvl3

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265,

    Pith/arXiv arXiv

  21. [26]

    Token pruning in multimodal large language models: Are we solving the right problem? In Che, W., Nabende, J., Shutova, E., and Pilehvar, M

    Wen, Z., Gao, Y ., Li, W., He, C., and Zhang, L. Token pruning in multimodal large language models: Are we solving the right problem? In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.),Findings of the Association for Computational Linguistics: ACL 2025, pp. 15537–15549, Vienna, Austria, July 2025a. Wen, Z., Gao, Y ., Wang, S., Zhang, J., Zha...

    Pith/arXiv arXiv 2025

  22. [27]

    Pyramid- drop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024a

    Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y ., Cao, Y ., He, C., Wang, J., Wu, F., et al. Pyramid- drop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024a. Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y ., Cao, Y ., He, C., Wang, J., Wu, F., et al. Pyramid...

    Pith/arXiv arXiv

  23. [28]

    Shortv: Efficient multi- modal large language models by freezing visual tokens in ineffective layers.arXiv preprint arXiv:2504.00502,

    Yuan, Q., Zhang, Q., Liu, Y ., Chen, J., Lu, Y ., Lin, H., Zheng, J., Han, X., and Sun, L. Shortv: Efficient multi- modal large language models by freezing visual tokens in ineffective layers.arXiv preprint arXiv:2504.00502,

    arXiv

  24. [29]

    and Shukla, D

    Zamini, M. and Shukla, D. Delta-llava: Base-then- specialize alignment for token-efficient vision-language models.arXiv preprint arXiv:2512.18910,

    arXiv

  25. [30]

    Skip-vision: Efficient and scalable acceleration of vision-language models via adaptive token skipping.arXiv preprint arXiv:2503.21817,

    Zeng, W., Huang, Z., Ji, K., and Yan, Y . Skip-vision: Efficient and scalable acceleration of vision-language models via adaptive token skipping.arXiv preprint arXiv:2503.21817,

    Pith/arXiv arXiv

  26. [31]

    Vscan: Rethinking visual token reduction for efficient large vision-language models.arXiv preprint arXiv:2505.22654, 2025a

    Zhang, C., Ma, K., Fang, T., Yu, W., Zhang, H., Zhang, Z., Xie, Y ., Sycara, K., Mi, H., and Yu, D. Vscan: Rethinking visual token reduction for efficient large vision-language models.arXiv preprint arXiv:2505.22654, 2025a. Zhang, E., Yu, F., Wu, A., Wen, Z., Yan, K., Ding, S., Qi, B., and Zhang, L. D2pruner: Debiased importance and structural diversity f...

    arXiv

  27. [32]

    Cross-modal information flow in multimodal large language models

    Zhang, Z., Yadav, S., Han, F., and Shutova, E. Cross-modal information flow in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 19781–19791, 2025f. Zhang, Z., Yadav, S., Han, F., and Shutova, E. Cross-modal information flow in multimodal large language models. In Proceedings of the Computer ...

    Pith/arXiv arXiv

  28. [33]

    Don’t just chase” highlighted tokens” in mllms: Revisiting visual holistic context reten- tion.arXiv preprint arXiv:2510.02912,

    Zou, X., Lu, D., Wang, Y ., Yan, Y ., Lyu, Y ., Zheng, X., Zhang, L., and Hu, X. Don’t just chase” highlighted tokens” in mllms: Revisiting visual holistic context reten- tion.arXiv preprint arXiv:2510.02912,

    arXiv

  29. [35]

    At this point, the dominant part of the total computation is contributed by text inference rather than any vision-related operations in the our pruned model. C.4. Flash attention acceleration implementation All previous experiments are conducted under theeagerattention mode, where redundant visual token update paths are removed via explicit masking. As sh...

    2024

  30. [36]

    (b) Our method under eager attention, where visual tokens are frozen via explicit masking and participate in attention only as key–value representations

    20 ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention Figure 8.(a) Standard FlashAttention causal masking when the query length differs from the key–value length, where the causal mask is aligned to the bottom-right corner of the attention matrix. (b) Our method under eager attention, where visual tokens are frozen via explicit masking and p...

    2023

  31. [2000]

    Li, B., Ge, Y ., Ge, Y ., Wang, G., Wang, R., Zhang, R., and Shan, Y

    Morgan Kaufmann. Li, B., Ge, Y ., Ge, Y ., Wang, G., Wang, R., Zhang, R., and Shan, Y . Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13299–13308, 2024a. Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y ., Liu, Z., and Li...

    Pith/arXiv arXiv

  32. [2017]

    Informed routing in llms: Smarter token- level computation for faster inference.arXiv preprint arXiv:2510.13831,

    Han, C., Liang, Y ., Xuan, Z., Wu, D., Zhang, W., and Shen, X. Informed routing in llms: Smarter token- level computation for faster inference.arXiv preprint arXiv:2510.13831,

    arXiv

  33. [2022]

    BLIP-2: Boot- strapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Boot- strapping language-image pre-training with frozen image encoders and large language models. InICML, 2023a. Li, W., Yuan, Y ., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., and Zhang, L. Tokenpacker: Efficient visual projector for multimodal llm.International Journal of Computer Vision, pp. 1–19, 202...

    2023

  34. [2023]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  35. [2024]

    Dynamic-llava: Effi- cient multimodal large language models via dynamic vision-language context sparsification.arXiv preprint arXiv:2412.00876,

    9 ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention Huang, W., Zhai, Z., Shen, Y ., Cao, S., Zhao, F., Xu, X., Ye, Z., Hu, Y ., and Lin, S. Dynamic-llava: Effi- cient multimodal large language models via dynamic vision-language context sparsification.arXiv preprint arXiv:2412.00876,

    Pith/arXiv arXiv

  36. [2025]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

    Pith/arXiv arXiv