Pith. sign in

REVIEW 2 major objections 5 minor 19 references

Token compression in audio-video LLMs fails when one attention score both splits capacity across modalities and selects which tokens to keep; the paper's training-free MACER compressor separates the two decisions — explicit audio/video budg

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:17 UTC pith:ULGARW6G

load-bearing objection A real idea and credible Qwen experiments, but the OmniVinci transfer claim is oversold because alpha is swept on the evaluation set rather than frozen from calibration. the 2 major comments →

arxiv 2608.01665 v1 pith:ULGARW6G submitted 2026-08-03 cs.AI cs.SD

Allocation Before Ranking: Decoupled Token Compression for OmniLLMs

Zhenghui Guo , Yilin Yang , Yuanbin Man , Miao Yin , Weidong Shi , Rabimba Karanjai , Omprakash Gnawali , Chengming Zhang This is my paper
classification cs.AI cs.SD
keywords token compressionomnimodal LLMshared attentiontoken pruningaudio-video allocationcapacity budgetingtraining-free inferencetop-K retention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When audio and video tokens share one decoder, compressing them is usually posed as a single ranking problem: give every token a saliency score and keep the top K. This paper argues that framing is mis-specified, because the same attention score makes two decisions at once — how much of the retained budget each modality receives, and which tokens inside a modality survive. The paper shows the coupling is measurable: shared attention in Qwen2.5-Omni persistently favors audio, so a global top-K spends capacity on audio before video tokens can compete. MACER, the proposed training-free compressor, breaks the coupling by setting explicit audio and video budgets first and then ranking tokens only within their own modality, reading saliency at modality-specific shallow layers. If the paper is right, omni-LLM token compression should be designed as three separate axes — allocation, selection, readout — and the results support that: 98.7%/97.3% of full-token accuracy at 25% retention on Qwen2.5-Omni-7B/3B, a lower-FLOPs Pareto win over OmniZip at 45% retention, and up to 12.9-point gains over shared top-K on OmniVinci-9B.

Core claim

The central claim is that a shared top-K rule for omnimodal token compression is mis-specified because one number carries two decisions. The paper derives an exact decomposition of the log attention-mass ratio between audio and video, log massa(q)/massv(q) = q^T(ca − cv) + [log Σ_{i∈a} exp(q^T ε_i) − log Σ_{j∈v} exp(q^T ε_j)], where ca and cv are per-modality key centroids and the bracketed term aggregates within-modality residuals. The first term is an architectural cross-modal allocation factor; the second carries token-level selection, and no shared ranking can separate them. Empirically the coupling favors audio: under video-essential queries every pooled query position at layer 3 (645/6

What carries the argument

The load-bearing object is the audio-video attention-mass identity, which decomposes shared-softmax mass into a cross-modal allocation term q^T(ca − cv) and a within-modality residual aggregation term; it shows why changing the saliency score cannot fix the coupling — the budget split is induced by ranking no matter which score is used. MACER's machinery is the allocation-before-ranking pipeline: Capacity-Coordinated Split sets Ka and Kv from a frozen retained-audio share α before any token is chosen; Allocation-Normalized Modality Scoring renormalizes each attention row within the modality so audio competes only with audio and video only with video; and staged single-prefill pruning reads a

Load-bearing premise

The audio-favoring bias is measured on one backbone (Qwen2.5-Omni-7B) with a small diagnostic sample and then treated as a structural property of all shared-decoder omni-LLMs, which is why a single frozen audio share (0.30 for 7B, 0.32 for 3B, chosen on roughly 100 held-out clips) is supposed to work across models and benchmarks — if some backbone's attention does not favor audio, the fixed split would hurt instead of help.

What would settle it

Measure the mid-layer centroid geometry q^T(ca − cv) on another shared-decoder omni-LLM, e.g. one whose packing places video closer to the query. If the audio-favoring half-space result (all 645/645 query positions on the audio side) fails to reproduce, or if an audio-heavy split α=0.70 beats MACER's fixed α=0.30 at 25% retention on an audio-grounded benchmark, then the structural-prior assumption is falsified and an adaptive allocation would be needed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • At 25% multimodal retention, MACER preserves 98.7% of full-token performance on Qwen2.5-Omni-7B and 97.3% on Qwen2.5-Omni-3B, with Video-MME held at full-token level on 7B.
  • On Qwen2.5-Omni-7B, the 25% MACER operating point Pareto-dominates OmniZip at 45% retention: higher average accuracy (+1.02), 4.4 pp lower FLOPs, 0.37 s lower total latency, and 4 GB lower peak memory.
  • The principle transfers across backbones: on OmniVinci-9B under its native input packing, MACER beats shared top-K ranking by up to 12.9 points, so the audio-favoring prior is not a Qwen2.5-Omni artifact.
  • The retained-audio share α becomes an explicit deployment-level control: a per-benchmark oracle α gains only 0.92 normalized points over the frozen setting, so the capacity split is a stable operating band rather than a fragile per-task hyperparameter.
  • Because audio and video are read at different shallow depths (audio early, video later), compressors no longer need one probe layer that serves both modalities; allocation, selection, and readout can each be improved independently without retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors leave implicit is an adaptive capacity split: estimate q^T(ca − cv) per clip and adjust α before selection, rather than freezing one share per model scale; the paper's own oracle audit suggests the headroom is small, but per-clip adaptation is untested.
  • The one-score-two-decisions argument is not limited to audio-video pairs — any two token groups sharing a softmax (image and video, multiple image streams) inherit the same coupling, so allocation-before-ranking could generalize to other asymmetric multimodal budgets.
  • The appendix shows the readout asymmetry weakens and can reverse at ρ=0.10 on AVUTBench; extremely tight budgets may therefore require re-calibrated readout layers rather than the fixed (3,5) choice, a regime the main paper does not deploy in.
  • The video-local coverage bonus is a template for other within-modality corrections — e.g., audio event continuity or shot boundaries — that can be layered onto any modality budget without disturbing the audio-video split.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that token compression in OmniLLMs is mis-specified as a single global top-K ranking, because the same attention score simultaneously decides cross-modal capacity allocation and within-modality token selection. The authors propose MACER, a training-free, single-forward-pass compressor that assigns explicit audio/video budgets via a retained-audio share α, ranks tokens within each modality using allocation-normalized attention scores, reads audio and video saliency at different shallow layers, and applies a video-local temporal-coverage bonus. Experiments on Qwen2.5-Omni-7B/3B across AVUTBench, DailyOmni, WorldSense, and Video-MME report 98.7%/97.3% of full-token performance at 25% retention and a Pareto-style improvement over OmniZip at 45% retention with lower estimated FLOPs. A cross-backbone study on OmniVinci-9B is reported in Appendix F.

Significance. If the results hold, MACER is a practical and well-motivated contribution: the exact decomposition in Eq. (2) cleanly separates cross-modal allocation from within-modality residual structure, and the ablation in Table 4 supports the claim that the main gain comes from decoupling allocation from selection. The paper is also transparent about using a held-out split for the Qwen hyperparameters and about reporting analytical FLOPs proxies. However, two load-bearing issues remain: the OmniVinci transfer claim is based on a post-hoc α sweep rather than a frozen pre-selected α, and the headline FLOPs comparison rests on an unvalidated proxy rather than measured FLOPs. These issues are fixable and do not undermine the core Qwen2.5-Omni results, but they currently prevent full support for the cross-backbone and lower-FLOPs claims.

major comments (2)
  1. [Appendix F, Table 19] The cross-backbone transfer claim is load-bearing but not yet established. The appendix states that the transfer study 'changes the backbone and sweeps only the retained-audio share α' while keeping readout layers fixed. No α is frozen before evaluation. The reported cells show that α=0.20 improves over shared top-K at all retention levels (+0.67, +0.62, +2.95), whereas α=0.40 degrades at 25% (−3.88) and produces the +12.91 headline at 45%. Because the sweep is performed on the evaluation benchmarks themselves, the 'up to 12.9 points' result is a selected favorable cell, not a transfer result. Please add an experiment with a frozen α chosen before evaluation (e.g., α=0.30 from Qwen2.5-Omni-7B, or an OmniVinci-specific held-out calibration split) and report all cells. Without this, the paper cannot distinguish 'allocation-before-ranking transfers' from 'OmniVinci has a favorable α discove
  2. [Tables 1–2 and Appendix E, Eq. (22)] The headline FLOPs comparison is based on analytical proxies, not measured FLOPs. Eq. (22) uses assumed attention shares η=0.235 (7B) and η=0.36 (3B) and charges all multimodal tokens until L_p=max(ℓ_a,ℓ_v); the resulting ratios (34.8%, 42.2%, 50.1%, etc.) are then used in Tables 1 and 2 to support the 'lower FLOPs' Pareto claim. Table 3 measures latency and memory only on WorldSense. Please provide measured FLOPs or validate the proxy with a profiler, use the same accounting for OmniZip, and at minimum label all main-table FLOPs values as analytical estimates with a sensitivity analysis over η. As written, the abstract's 'lower FLOPs' claim is not directly measured.
minor comments (5)
  1. [Table 2 and Section 5.1] FastV† is evaluated at 50% retention on an H100 while all other methods are at 25/35/45% on Ada6000, and its normalized average excludes Video-MME. This makes the 'unified protocol' claim imprecise. Please either run FastV at matched retention on the same hardware or explicitly state this limitation in the main text rather than only in a table footnote.
  2. [Table 4] The 'Single readout layer' ablation is not specified. The reader cannot tell which layer is used, whether it replaces both ℓ_a and ℓ_v, or why it performs below shared top-K (54.38 vs. 54.81). Please define this configuration and discuss the drop; otherwise the ablation's message is ambiguous.
  3. [Appendix F, Table 19] The abbreviation 'Cons. Norm. Avg.' is not defined in the table caption or the surrounding text. Define it explicitly and state how the superscripts (†, ‡) relate to the full-token reference runs.
  4. [Appendix E.2] Eq. (22) ignores text tokens and constant factors. Since text tokens are always preserved, the proxy may understate the effective cost reduction at low retention. Add a sentence explaining why this omission does not alter the relative comparison between MACER and the baselines.
  5. [Section 3, Observation 1] The pooled claim 'video wins per-token attention comparison in only 8.8% of cells' is informative, but the diagnostic subset is small (30 clips for some controls). Reporting a confidence interval or a paired percentile bootstrap would help calibrate how strongly to rely on this observation.

Circularity Check

1 steps flagged

Cross-backbone transfer claim rests on an evaluation-set alpha sweep; Qwen main results are not circular.

specific steps
  1. fitted input called prediction [Appendix F / Table 19; Abstract; Sec. 5.1]
    "Thus, the transfer study changes the backbone and sweeps only the retained-audio share α, while keeping the video coverage strength fixed at λc = 0.20. Table 19 reports the OmniVinci-9B transfer results. ... MACER-B (α=0.40) is strongest in the moderate-retention regime (ρ∈{35%,45%}), with mean +7.50 pp over shared top-K. ... On OmniVinci-9B, MACER outperforms shared top-K ranking by up to 12.9 points, showing that the allocation–selection separation transfers across models."

    The headline OmniVinci transfer result is the best cell of an alpha sweep performed directly on the OmniVinci evaluation benchmarks. No pre-frozen alpha is tested on OmniVinci: the Qwen-7B default α=0.30 never appears, and at 25% retention the two swept values disagree in sign (α=0.20: +0.67; α=0.40: -3.88). The '+12.91' at α=0.40/ρ=45% is therefore the maximum of the swept grid, not a prediction of a fixed configuration. As a result, the claimed cross-backbone support cannot distinguish the structural allocation-before-ranking principle from an oracle-like selection of α on the evaluation set.

full rationale

The main Qwen2.5-Omni derivation is self-contained and not circular. The mechanism decomposition in Eq. (1)-(2) is an exact algebraic identity from the definition of attention, and the diagnostics (audio prior, readout asymmetry, budget-vs-selection stress test) are measured rather than assumed. MACER's hyperparameters (ℓa, ℓv, α, λc) are selected once on a held-out split disjoint from the reported evaluation sets and then frozen, which is legitimate calibration rather than fitting the target result. The reported 98.7%/97.3% normalized averages are computed from fixed configurations on full evaluation sets, so those claims have independent empirical content. The only significant circularity-adjacent concern is the OmniVinci-9B transfer evidence in Appendix F: the paper sweeps the retained-audio share α on the OmniVinci benchmarks themselves and then reports the best cells as evidence that the principle 'transfers across models.' Because no frozen alpha (e.g., the Qwen default α=0.30) is evaluated, the transfer claim is not yet a prediction from a pre-specified method. This is a fitted-input-called-prediction pattern for that specific headline claim, though it does not undermine the Qwen main results or the conceptual argument. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via prior work; the paper's own citations to prior work are contextual.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

MACER introduces no new physical or architectural entities, only a pruning policy with tuned hyperparameters. The main load-bearing elements are the four fitted hyperparameters listed above, plus the domain assumption that normalized shallow attention is a good within-modality saliency signal.

free parameters (4)
  • Retained audio share alpha = 0.30 for Qwen2.5-Omni-7B, 0.32 for 3B, 0.20/0.40 for OmniVinci-9B variants
    Selected once on a held-out ~100 clip split per scale, then frozen across benchmarks. The paper's own robustness sweep shows alpha=0.30 is best among fixed settings on a 200-example split, with the oracle only +0.92 points higher, so it is a tuned hyperparameter, not a derived constant.
  • Readout layers (audio, video) = (3,5) for 7B, (4,7) for 3B, reused (3,5) for OmniVinci
    Chosen from readout-layer sweeps on held-out splits. The appendix reports sensitivity ratios and reversals at rho=0.10, indicating these values are empirically calibrated.
  • Coverage strength lambda_c = 0.20 for 7B, 0.30 for 3B, 0.20 for OmniVinci
    Chosen per scale on the calibration split. Ablation shows a modest +0.31 point contribution.
  • Number of video chunks C = 8
    Set to 8 coarse temporal chunks; the paper does not report a sensitivity sweep over C.
axioms (4)
  • domain assumption Attention probability mass is a valid token-saliency signal within a modality after row normalization.
    The method ranks tokens by normalized attention from the question span. The paper cites prior VLM work questioning text-to-visual attention reliability (Zhang et al. 2025a) but does not validate the ranking against oracle token importance.
  • domain assumption The audio-favoring allocation prior observed on Qwen2.5-Omni-7B diagnostics transfers to Qwen2.5-Omni-3B and OmniVinci-9B.
    The alpha values are calibrated per scale, but the qualitative prior is assumed structural. The OmniVinci transfer required a different alpha (0.40) to show large gains, weakening the transfer claim.
  • domain assumption Pruning audio after layer 3 or 4 and video after layer 5 or 7 in a single forward pass does not change the distributional properties used by later layers in a way that invalidates the measured benchmark scores.
    The staged pruning changes the sequence for later attention layers, and the FLOPs proxy assumes this is equivalent to full-context computation up to the readout layer. This is an implementation assumption checked only by end-task accuracy.
  • domain assumption Benchmark accuracy under fixed decoding with FlashAttention-2 is deterministic enough for single-run comparisons without error bars.
    The paper states no error bars are attached to benchmark cells, and only the random baseline uses a fixed seed. This assumes the measured gaps are not noise; the Appendix D.4 split results differ from main Table 2 for the same alpha setting, suggesting run-to-run or split sensitivity.

pith-pipeline@v1.3.0-daily-deepseek · 22704 in / 7524 out tokens · 58363 ms · 2026-08-04T23:17:42.287132+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Allocation Before Ranking: Decoupled Token Compression for OmniLLMs." pith.science (2026). https://pith.science/paper/ULGARW6G

@misc{pith2026260801665,
  author       = {Pith},
  title        = {Pith review of: Allocation Before Ranking: Decoupled Token Compression for OmniLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULGARW6G}},
  note         = {Machine review of arXiv:2608.01665}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.

Figures

Figures reproduced from arXiv: 2608.01665 by Chengming Zhang, Miao Yin, Omprakash Gnawali, Rabimba Karanjai, Weidong Shi, Yilin Yang, Yuanbin Man, Zhenghui Guo.

Figure 1
Figure 1. Figure 1: Shared attention favors audio across depth. ca and cv denote the mean audio and video keys. (a) The query q is closer to ca than to cv, so shared attention favors audio. (b) Under both video-essential (VE) and audio-essential (AE) queries, log(massa/massv) stays positive across layers. (c) At L3 under VE queries, pooled query positions satisfy cos(q, ca) > cos(q, cv). This geometry illustrates why raw shar… view at source ↗
Figure 2
Figure 2. Figure 2: Capacity allocation, not within-video selection, controls cross-modal balance. (a) A shared top-K rule uses one score for two decisions. (b) ToMe and random dropping differ by at most 0.4 pp at matched video keep ratio, whereas reducing the video keep ratio itself shifts audio mass share by 9.0 pp. (c) The retained audio share α affects accuracy, with a video-heavy split (α = 0.10) outperforming an audio-h… view at source ↗
Figure 3
Figure 3. Figure 3: Audio and video become readable at different depths. We sweep one modality’s probe layer while fixing the other to MACER’s default. (a) Video readout is more layer-sensitive than audio readout in most settings. (b) Video readout performs best at L4-L5. (c) Audio readout remains relatively stable across layers. mass is allocated across modalities. A shared ranking rule therefore contains a cross-modal alloc… view at source ↗
Figure 4
Figure 4. Figure 4: Macer pipeline. MACER runs a single decoder prefill with staged modality-specific pruning. It reads audio attention at ℓa and immediately applies audio retention; the decoder then continues with retained audio and full video until ℓv, where video retention is applied. The final retained audio and video tokens are merged with text tokens in the original sequence order, and the remaining decoder layers opera… view at source ↗
Figure 5
Figure 5. Figure 5: Q-centroid mechanism behind the audio prior. (a) Query-side interventions show that audio mass is strongly controlled by alignment between the query direction and the modality centroids; key-side geometry and within-audio coherence controls alone do not explain the prior. (b) Across layers, the query representation is directionally closer to the audio centroid than to the video centroid. (c) The same Q-sid… view at source ↗
Figure 6
Figure 6. Figure 6: Null-content and distance-matched controls for the audio prior. (a) Replacing original audio with mismatched real audio, scale-matched white noise, or silence shifts log(v/a) toward video; the prior therefore responds to audio content rather than behaving as an input-independent sink. (b) Equal-distance matching removes much of the early positional effect, but a mid-layer audio-favoring residual remains. (… view at source ↗
Figure 7
Figure 7. Figure 7: Within-video selector invariance at matched video budgets. ToMe-style merging and random video dropping are compared at the same retained video ratios across natural-audio and null-content variants. (a) The audio share trajectories nearly overlap for ToMe and random dropping. (b) Paired differences remain within a narrow band, showing that the dominant cross-modal effect is retained video capacity rather t… view at source ↗
Figure 8
Figure 8. Figure 8: Allocation changes while within-video ranking largely survives. (a) Masking audio at the query produces only a small increase in video attention mass. (b) Most freed audio mass flows to text rather than to video, indicating that removing the audio prior does not automatically restore video capacity. (c) Within-video top-K rankings remain highly stable under the audio mask. The diagnostic supports the separ… view at source ↗
Figure 9
Figure 9. Figure 9: Readout-layer asymmetry between audio and video. We sweep the probe layer of one modality while fixing the other modality to MACER’s 7B default readout layer, and report task-centered accuracy (Acc − meanℓAcc) in percentage points. (a) The sensitivity ratio ∆v/∆a is above one in most benchmark–retention settings, showing that video readout is more layer-sensitive than audio readout. (b) Sweeping the video … view at source ↗
Figure 10
Figure 10. Figure 10: expands the centered readout sweep to all 5 × 5 audio–video probe-layer pairs under the calibrated 7B capacity split [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: PCA of query-key interaction space at layers [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 3 canonical work pages

  1. [3]

    OmniSIFT: Modality-asymmetric token compression for efficient omni-modal large language models.arXiv preprint arXiv:2602.04804,

    Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, and Liang Wang. OmniSIFT: Modality-asymmetric token compression for efficient omni-modal large language models.arXiv preprint arXiv:2602.04804,

    Pith/arXiv arXiv

  2. [6]

    Event-vstream: Event-driven real-time understanding for long video streams.arXiv preprint arXiv:2601.15655,

    Zhenghui Guo, Yuanbin Man, Junyuan Sheng, Bowen Lin, Ahmed Ahmed, Bo Jiang, Boyuan Zhang, Miao Yin, Sian Jin, Omprakash Gnawali, et al. Event-vstream: Event-driven real-time understanding for long video streams.arXiv preprint arXiv:2601.15655,

    work page arXiv

  3. [7]

    Xiaohu Huang, Hao Zhou, and Kai Han

    URLhttps://arxiv.org/abs/2502.04326. Xiaohu Huang, Hao Zhou, and Kai Han. PruneVid: Visual token pruning for efficient video large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pages 19959–19973,

    Pith/arXiv arXiv 2025

  4. [8]

    URLhttps://aclanthology.org/2025.findings-acl.1024/. Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, Sungjin Ahn, Jan Kautz, Hongxu Yin, Yao Lu, Song Han, and Wonmin Byeon. STORM: Token-efficient long video understanding for multimodal LLMs.arXiv preprint arXiv:2503.04130,

    arXiv 2025

  5. [9]

    URLhttps://arxiv.org/abs/2503.04130

    doi: 10.48550/arXiv.2503.04130. URLhttps://arxiv.org/abs/2503.04130. Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, and Joon Son Chung. FastA V: Efficient token pruning for audio-visual large language model inference.arXiv preprint arXiv:2601.13143,

    doi:10.48550/arxiv.2503.04130

  6. [10]

    2601.13143

    doi: 10.48550/arXiv. 2601.13143. URLhttps://arxiv.org/abs/2601.13143. Bingzhou Li and Tao Huang. DASH: Dynamic audio-driven semantic chunking for efficient omnimodal token compression.arXiv preprint arXiv:2603.15685,

    doi:10.48550/arxiv.2601.13143

  7. [11]

    URL https://arxiv.org/abs/2603.15685

    doi: 10.48550/arXiv.2603.15685. URL https://arxiv.org/abs/2603.15685. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaV A-PruMerge: Adap- tive token reduction for efficient large multimodal models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22857–22867,

    Pith/arXiv arXiv doi:10.48550/arxiv.2603.15685

  8. [12]

    Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang

    URL https: //openaccess.thecvf.com/content/ICCV2025/html/Shang_LLaVA-PruMerge_Adaptive_Token_ Reduction_for_Efficient_Large_Multimodal_Models_ICCV_2025_paper.html. Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. HoliTom: Holistic token merging for fast video large language models.arXiv preprint arXiv:2505.21334, 2025a. doi: 10.48550/ar...

    doi:10.48550/arxiv.2505.21334

  9. [13]

    Huyu Wu, Meng Tang, Xinhan Zheng, and Haiyun Jiang

    URLhttps://arxiv.org/abs/2509.18816. Huyu Wu, Meng Tang, Xinhan Zheng, and Haiyun Jiang. When language overrules: Revealing text dominance in multimodal large language models,

    arXiv

  10. [14]

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin

    URLhttps://arxiv.org/abs/2508.10552. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. PyramidDrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

    Pith/arXiv arXiv

  11. [15]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin

    URLhttps://arxiv.org/abs/2410.17247. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni technical report.arXiv preprint arXiv:2503.20215,

    Pith/arXiv arXiv

  12. [16]

    URL https://arxiv.org/abs/2503.20215

    doi: 10.48550/arXiv.2503.20215. URL https://arxiv.org/abs/2503.20215. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Vi- sionZip: Longer is better but not necessary in vision language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792–19802, 2025a. URL https:...

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.20215 2025

  13. [17]

    Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang

    URLhttps://arxiv.org/abs/2407.12772. Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in VLMs. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20857–20867, 2025a. Yuan Zhang, ...

    Pith/arXiv arXiv

  14. [18]

    Ziwei Zhou, Rui Wang, Zuxuan Wu, and Yu-Gang Jiang

    URL https://arxiv.org/abs/2505.18657. Ziwei Zhou, Rui Wang, Zuxuan Wu, and Yu-Gang Jiang. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities,

    Pith/arXiv arXiv

  15. [19]

    The ratio∆ v/∆a measures whether video or audio readout is more sensitive to probe-layer choice

    For modalitym∈ {a, v}, define the layer-induced accuracy range ∆m = max ℓm∈{1,...,5} Acc(ℓm)−min ℓm∈{1,...,5} Acc(ℓm),(19) where the other modality’s probe layer is fixed to MACER’s 7B default. The ratio∆ v/∆a measures whether video or audio readout is more sensitive to probe-layer choice. The key conclusion is asymmetric sensitivity, not a sharp audio op...

    work page arXiv 2064

  16. [2023]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang

    URLhttps://openreview.net/forum?id=JroZRaRw7Eu. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Computer Vision – ECCV 2024, pages 19–35. Springer,

    work page 2024

  17. [2024]

    URL https://doi.org/10.1007/978-3-031-73004-7_2

    doi: 10.1007/978-3-031-73004-7_2. URL https://doi.org/10.1007/978-3-031-73004-7_2. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691,

    Pith/arXiv arXiv

  18. [2025]

    Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al

    URL https://openaccess.thecvf.com/content/CVPR2025/html/Fu_Video-MME_The_First-Ever_ Comprehensive_Evaluation_Benchmark_of_Multi-modal_LLMs_in_CVPR_2025_paper.html. Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al. Arc-hunyuan-video-7b: Structured video comprehension of real-world shorts.arXiv p...

    Pith/arXiv arXiv

  19. [2026]

    URL https://arxiv.org/abs/2602.04804

    doi: 10.48550/arXiv.2602.04804. URL https://arxiv.org/abs/2602.04804. Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, et al. Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211,

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.04804