Pith. sign in

REVIEW 3 major objections 7 minor 54 references

A modality-decoupled, query-anchored token compression method preserves audio and video cues independently and outperforms unidirectional compression baselines on omnimodal QA at 25-45% token retention.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 03:29 UTC pith:JJDGOOYL

load-bearing objection A genuinely new and useful idea for omni-token compression, with consistent ablations, but the paper has an ambiguous budget-allocation formula and an abstract that cherry-picks the most favorable numbers. the 3 major comments →

arxiv 2607.23193 v3 pith:JJDGOOYL submitted 2026-07-25 cs.CV

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

Jinsen Su , Yongdong Luo , Yuexiao Ma , Yibo Hu , Meiguang Jin , Xiawu Zheng This is my paper
classification cs.CV
keywords compressionomniscopetokenacrossaudioqueryaccuracyaverage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large AI models can now process video and audio together, but the number of tokens they must handle grows quickly with video length. OmniScope is a way to shrink those tokens at inference time without retraining the model. It scores each second of audio and each video frame by how relevant it is to the user's question, then gives audio and video their own separate compression budgets. The key move is that the two modalities are scored independently: earlier methods used audio importance to decide which video tokens to keep, or the other way around, but the audio cue that matters (a whistle) and the visual cue that matters (a referee's hand signal) can occur at different moments.

Core claim

The central claim is stated in the abstract and Sec. 4.2: across four audio-video benchmarks and two Qwen2.5-Omni scales, OmniScope 'achieves the best average accuracy across all compression settings', with at 25% retention 'up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy.' If true, modality-decoupled query-anchored compression robustly outperforms unidirectional guidance (OmniZip) under aggressive token budgets.

Load-bearing premise

The load-bearing premise is that query-conditioned similarity scores are valid proxies for what the LLM actually needs to answer: CLIP cosine similarity between each frame and the query (Eq. 3), and the gated cosine similarity between the LLM's audio-encoder output and its own query embeddings (Eqs. 4-5). If CLIP's notion of relevance, or the audio-embedding alignment, disagrees with the LLM's reasoning, the entire budget allocation is misdirected. The paper itself shows (Appendix D) that the model's internal vision-text alignment is too weak for visual scoring, which is why an external CLIP scorer is used; that external scorer is not trained on the OmniLLM's token semantics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes OmniScope, a training-free token compression framework for omnimodal LLMs, instantiated on Qwen2.5-Omni at 7B and 3B scales. Its central idea is that query-relevant moments differ between audio and video, so compressing one modality using the other modality's salience (as in OmniZip) is fragile. OmniScope therefore estimates relevance separately: CLIP score per frame for video, a gated cosine similarity between audio-encoder outputs and query embeddings for audio, then allocates per-position token budgets proportionally. Visual tokens are pruned with an anchor/delta spatio-temporal scheme, and audio tokens are merged per second via bipartite soft matching. Experiments cover WorldSense, DailyOmni, OmniVideoBench, and Video-MME at 45% and 25% overall retention, reporting the best average accuracy among compression baselines in all four settings, with up to 3.53x prefill speedup and over 15% GPU memory reduction at 25% retention.

Significance. If the results are reliable, OmniScope is a useful, simple, and training-free contribution: it introduces a clean design principle — share the query across modalities but decouple the salience estimates — and it consistently beats the unidirectional OmniZip baseline on aggregate accuracy. The paper's strengths include a clear motivational analysis (Appendix F uses CLIP/CLAP as an independent probe), consistent ablations of the main components, external comparisons, and a promise of released code. The central claim is empirically defensible, but it rests on a formally underspecified budget-allocation step and on single-run accuracy numbers whose margins are sometimes small. These issues are fixable and do not invalidate the empirical direction, but they need to be addressed before the paper is fully reproducible and before the 'best average accuracy' claim can be taken at face value.

major comments (3)
  1. [Sec. 3.3.3, Eq. (7)] The budget allocation is undefined for the scores actually produced by Eqs. (3) and (5). Both are cosine similarities or weighted averages of cosine similarities and can be negative. With negative s_i, Eq. (7) can produce negative or zero retention counts at low-salience positions, and if the denominator sum is non-positive the ratio is undefined. No shift, ReLU, min-max normalization, or softmax is specified. Additionally, the formula does not clamp n_keep(i) to the number of tokens available in that frame/second, and the floor rounding means the total allocated count need not equal N_budget. Because this equation is the mechanism that implements modality-decoupled, query-aware compression, the method as described is not reproducible from the text alone. Please specify a non-negative transform and a rounding/renormalization rule, or give the exact implementation.
  2. [Sec. 4.2, Table 1] All accuracy numbers are single runs with no error bars or significance tests, yet the central claim is a ranking: 'best average accuracy across all compression settings.' Several margins are small — for example, Qwen2.5-Omni-3B at 45% retention gives 50.18 (OmniScope) vs. 49.70 (OmniZip), and 7B at 25% gives 51.00 vs. 50.53 (FastV). Without repeated runs or a significance test, the reader cannot tell whether these differences are stable. Please report means and standard deviations over at least three runs, or explicitly state the single-run limitation and provide seed sensitivity.
  3. [Sec. 3.3.1-3.3.2 and Appendix D] The method depends on CLIP cosine similarity for visual tokens and internal gated cosine similarity for audio tokens as proxies for what the LLM actually needs to answer a query. The paper shows that the model's internal vision-text alignment is too weak and therefore chooses an external CLIP scorer, but it does not validate that CLIP's relevance judgments agree with the OmniLLM's answer-critical frames. The empirical results are consistent with the proxy working, but an analysis of per-question agreement or of failure cases where CLIP scores misdirect the budget would make the central mechanism load-bearing rather than assumed. This is a correctness-risk concern, not a circularity claim.
minor comments (7)
  1. [Eq. (6)] When no second s' satisfies 0<|s-s'|≤R, the maximum in the boost factor is over an empty set. Define b_s = 1 in that case.
  2. [Eq. (7)] After the floor operation, the sum of n_keep may not equal N_budget, and n_keep(i) should be clamped to the capacity of position i. A specific rounding/renormalization procedure should be stated.
  3. [Sec. 3.3.2] The phrase 'averaging the top-k token scores within each second' is ambiguous: specify whether k is selected over token positions within the second and whether k=5 is the value listed in Table 7.
  4. [Appendix C, Fig. 5] The axes of Figure 5 appear unlabeled/placeholder. Provide properly labeled axes and describe the measurement setup.
  5. [Table 3 caption] The caption says 'same for Tables 5 and 4' but then references Table 4; reorder for clarity.
  6. [Appendix F] The diagnostic uses CLAP for audio scores while the actual OmniScope pipeline uses the LLM audio encoder. State explicitly whether the observed cross-modal mismatch is expected to transfer to the internal audio scoring used in the method.
  7. [Table 7 / Appendix E] Sensitivity analysis is only reported for τ_r. Since the audio-scoring hyperparameters (τ, top-k, β, γ, R, P) directly control budget allocation, a sensitivity study for at least the most sensitive of those would strengthen the 'fixed hyperparameters' claim.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on a set of heuristic scoring assumptions and five hand-set hyperparameters. No new physical or architectural entities are introduced. The method is an empirical compression system, so the axioms are domain assumptions about what CLIP and audio-embedding similarities measure relative to LLM reasoning.

free parameters (5)
  • Audio scoring temperature tau = 0.05
    Softmax temperature in Eq. 5; hand-set in Table 7, not derived.
  • Top-k aggregation count = 5
    Number of per-token scores averaged to make per-second audio importance; hand-set in Table 7.
  • Neighbor boost parameters (beta, gamma, R, P) = 1.5, 0.5, 3, 80
    Eq. 6 propagation of audio salience to neighboring seconds; chosen by hand in Table 7.
  • Delta switching threshold tau_r = 0.4
    Switches between irreplaceability and temporal-difference scoring; Appendix E shows a tuned optimum.
  • DPC-KNN neighbors k = 5
    Neighborhood size for local density in Eq. 8; hand-set in Table 7.
axioms (6)
  • domain assumption CLIP cosine similarity between a video frame and the query ranks frames by answer relevance
    Sec. 3.3.1 Eq. 3 uses CLIP as the visual scorer; if CLIP's ranking is wrong, visual budget allocation is misdirected.
  • domain assumption Gated cosine similarity between audio-encoder embeddings and LLM query embeddings ranks audio seconds by answer relevance
    Sec. 3.3.2 Eqs. 4-5; no external audio scorer is used, so the method relies on the OmniLLM's internal audio-text alignment.
  • domain assumption Word-embedding norm is a valid informativeness gate for query tokens
    Eq. 4 uses embedding norms as a gating signal, citing [27]; if the norm does not track informativeness, the gating hurts scoring.
  • domain assumption Audio salience is locally continuous, so boosting nearby seconds improves allocation
    Eq. 6 applies a decaying boost around high-scoring seconds; this assumes the informative context of a high-scoring second is nearby.
  • domain assumption Short-term audio stationarity permits per-second merging without losing needed cues
    Sec. 3.5 bases the per-second merging design on audio stationarity [28]; if one-second windows contain non-redundant cues, merging could discard them.
  • domain assumption Anchor-delta selection criteria preserve task-relevant content
    Sec. 3.4 assumes that low-density tokens in anchor frames and low-replaceability or high-temporal-difference tokens in delta frames are the most informative for the LLM.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models." pith.science (2026). https://pith.science/paper/JJDGOOYL

@misc{pith2026260723193,
  author       = {Pith},
  title        = {Pith review of: OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJDGOOYL}},
  note         = {Machine review of arXiv:2607.23193}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.

Figures

Figures reproduced from arXiv: 2607.23193 by Jinsen Su, Meiguang Jin, Xiawu Zheng, Yibo Hu, Yongdong Luo, Yuexiao Ma.

Figure 1
Figure 1. Figure 1: One query, two modalities, and often two very different sparsity patterns. (a) Audio salience peaks early (whistle) while visual salience peaks late (hand signal), illustrating cross-modal salience mismatch. (b) Unidirectional cross-modal guidance loses critical information, while modality-decoupled compression preserves both. adopt unidirectional cross-modal guidance strategies, letting one modality deter… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of attention maps at Layer 3 of Qwen2.5-Omni-3B. Under the Full Token setting, attention is concentrated locally; after random pruning, both cross-window (the off-diagonal regions highlighted by red dashed boxes are notably enhanced) and cross-modal attention are significantly strengthened. where Nv and Na denote the number of visual and audio tokens, and D is the embedding dimension. The projec… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of OmniScope. OmniScope operates in three stages: (1) Query-Aware Scoring independently estimates the importance of visual and audio tokens relative to the query and allocates per-position compression budgets; (2) Vision Pruning balances global semantic coverage and temporal increments through anchor–delta frame alternation; (3) Audio Merging fuses redundant tokens via per-second bipartite soft ma… view at source ↗
Figure 4
Figure 4. Figure 4: Ablation results for video and audio compression ratios, evaluated on Qwen2.5-Omni-3B using the WorldSense benchmark. Left: Varying the video compression ratio ρv with audio compression ratio ρa = 0.35. Right: Varying the audio compression ratio ρa with video compression ratio ρv = 0.8. benchmark for omnimodal video understanding; and (4) Video-MME [9], a widely adopted video understanding bench￾mark in wh… view at source ↗
Figure 5
Figure 5. Figure 5: Ratio of CLIP visual scoring overhead to end-to-end latency as the number of generated tokens increases, measured on Qwen2.5-Omni-7B with 45% retention. D Visual Scoring Strategy Ablation [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of the ratio threshold τr on accuracy. τr controls the switching point between the two Delta frame scoring strategies in AD-STC. Evaluated on Qwen2.5-Omni-3B using the DailyOmni benchmark with (ρv=0.6, ρa=0.25). F Cross-Modal Salience Mismatch Analysis To quantify the pervasiveness of cross-modal salience mismatch, we compute the Spearman correlation coefficient between query-conditioned visual and … view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of Spearman correlation coefficients between query-conditioned visual (CLIP) and audio (CLAP) importance scores across 1,197 query-video pairs on DailyOmni. Approximately 78.3% of samples exhibit only weak correlation (|ρ| < 0.3), confirming that cross-modal salience mismatch is pervasive rather than anecdotal. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 20 linked inside Pith

  1. [1]

    Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Animageisworth1/2tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    LiangChen,HaozheZhao,TianyuLiu,ShuaiBai,JunyangLin,ChangZhou,andBaobaoChang. Animageisworth1/2tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision, pages 19–35. Springer, 2024

    2024

  3. [3]

    Howfararewetogpt-4v? closingthegaptocommercialmultimodalmodelswithopen-sourcesuites.ScienceChina Information Sciences, 67(12):220101, 2024

    ZheChen,WeiyunWang,HaoTian,ShenglongYe,ZhangweiGao,ErfeiCui,WenwenTong,KongzhiHu,JiapengLuo,Zheng Ma,etal. Howfararewetogpt-4v? closingthegaptocommercialmultimodalmodelswithopen-sourcesuites.ScienceChina Information Sciences, 67(12):220101, 2024

    2024

  4. [4]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    2023

  5. [5]

    Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models.arXiv preprint arXiv:2602.04804, 2026

    Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, et al. Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models.arXiv preprint arXiv:2602.04804, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Study on density peaks clustering based on k-nearest neighbors and principal component analysis.Knowledge-Based Systems, 99:135–145, 2016

    Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis.Knowledge-Based Systems, 99:135–145, 2016

    2016

  8. [8]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational conference on machine learning, pages 10323–10337. PMLR, 2023

    2023

  9. [9]

    Video-mme: Thefirst-evercomprehensiveevaluationbenchmarkofmulti-modalllmsinvideoanalysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, MengdanZhang,etal. Video-mme: Thefirst-evercomprehensiveevaluationbenchmarkofmulti-modalllmsinvideoanalysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108–24118, 2025

    2025

  10. [10]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    ChaoyouFu,HaojiaLin,XiongWang,Yi-FanZhang,YunhangShen,XiaoyuLiu,HaoyuCao,ZuweiLong,HetingGao,KeLi, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Worldsense: Evaluating real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Prunevid: Visualtokenpruningforefficientvideolargelanguagemodels

    XiaohuHuang,HaoZhou,andKaiHan. Prunevid: Visualtokenpruningforefficientvideolargelanguagemodels. InFindings of the Association for Computational Linguistics: ACL 2025, pages 19959–19973, 2025

    2025

  13. [13]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    AaronHurst,AdamLerer,AdamPGoucher,AdamPerelman,AdityaRamesh,AidanClark,AJOstrow,AkilaWelihinda,Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Efficient multimodal large language models: A survey.Visual Intelligence, 3(1):27, 2025

    YizhangJin,JianLi,TianjunGu,YexinLiu,BoZhao,JinxiangLai,ZhenyeGan,YabiaoWang,ChengjieWang,XinTan,etal. Efficient multimodal large language models: A survey.Visual Intelligence, 3(1):27, 2025

    2025

  15. [15]

    Tokenpruninginaudiotransformers: Optimizingperformanceanddecodingpatchimportance

    TaehanLeeandHyukjunLee. Tokenpruninginaudiotransformers: Optimizingperformanceanddecodingpatchimportance. arXiv preprint arXiv:2504.01690, 2025

    arXiv 2025

  16. [16]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    BoLi,YuanhanZhang,DongGuo,RenruiZhang,FengLi,HaoZhang,KaichenZhang,PeiyuanZhang,YanweiLi,ZiweiLiu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Omnivideobench: Towards audio-visual understanding evaluation for omni mllms.arXiv preprint arXiv:2510.10689, 2025

    CaoruiLi,YuChen,YiyanJi,JinXu,ZhenyuCui,ShihaoLi,YuanxingZhang,WentaoWang,ZhenghaoSong,DinglingZhang, et al. Omnivideobench: Towards audio-visual understanding evaluation for omni mllms.arXiv preprint arXiv:2510.10689, 2025

    arXiv 2025

  18. [18]

    Accelerating transducers through adjacent token merging.arXiv preprint arXiv:2306.16009, 2023

    Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. Accelerating transducers through adjacent token merging.arXiv preprint arXiv:2306.16009, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Video-llava: Learningunitedvisualrepresentation by alignment before projection

    BinLin,YangYe,BinZhu,JiaxiCui,MunanNing,PengJin,andLiYuan. Video-llava: Learningunitedvisualrepresentation by alignment before projection. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 5971–5984, 2024. 12 MAC-AutoML

    2024

  20. [20]

    Speechprune: Context-awaretokenpruningforspeechinformationretrieval

    YueqianLin,YuzheFu,JingyangZhang,YudongLiu,JianyiZhang,JingweiSun,HaiHelenLi,andYiranChen. Speechprune: Context-awaretokenpruningforspeechinformationretrieval. In2025IEEEInternationalConferenceonMultimediaandExpo (ICME), pages 1–6. IEEE, 2025

    2025

  21. [21]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  22. [22]

    Javisgpt: A unified multi-modal llm for sounding-video comprehension and generation.arXiv preprint arXiv:2512.22905, 2025

    Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, et al. Javisgpt: A unified multi-modal llm for sounding-video comprehension and generation.arXiv preprint arXiv:2512.22905, 2025

    arXiv 2025

  23. [23]

    Quota: Query-orientedtokenassignmentviacotquerydecoupleforlongvideocomprehension

    Yongdong Luo, Wang Chen, Weizhong Huang, Shukang Yin, Haojia Lin, Jinfa Huang, Chaoyou Fu, Jiayi Ji, Xiawu Zheng, andJieboLuo. Quota: Query-orientedtokenassignmentviacotquerydecoupleforlongvideocomprehension. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 24160–24168, 2026

    2026

  24. [24]

    Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  25. [25]

    Ompq: Orthogonalmixedprecisionquantization

    Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonalmixedprecisionquantization. InProceedingsoftheAAAIconferenceonartificialintelligence, volume37, pages 9029–9037, 2023

    2023

  26. [26]

    Affinequant: Affine transformation quantization for large language models

    Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. InInternational Conference on Learning Rep- resentations, volume 2024, pages 50932–50951, 2024

    2024

  27. [27]

    Norm of word embedding encodes information gain

    Momose Oyama, Sho Yokoi, and Hidetoshi Shimodaira. Norm of word embedding encodes information gain. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2108–2130, 2023

    2023

  28. [28]

    Prentice-Hall, Inc., 1993

    Lawrence Rabiner and Biing-Hwang Juang.Fundamentals of speech recognition. Prentice-Hall, Inc., 1993

    1993

  29. [29]

    Learning transferable visual models from natural language supervision

    AlecRadford,JongWookKim,ChrisHallacy,AdityaRamesh,GabrielGoh,SandhiniAgarwal,GirishSastry,AmandaAskell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  30. [30]

    Llava-prumerge: Adaptive token reduction for efficient largemultimodalmodels

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient largemultimodalmodels. InProceedingsoftheIEEE/CVFInternationalConferenceonComputerVision,pages22857–22867, 2025

    2025

  31. [31]

    Holitom: Holistictokenmergingforfastvideolarge language models.arXiv preprint arXiv:2505.21334, 2025

    KeleShao,KedaTao,CanQin,HaoxuanYou,YangSui,andHuanWang. Holitom: Holistictokenmergingforfastvideolarge language models.arXiv preprint arXiv:2505.21334, 2025

    arXiv 2025

  32. [32]

    Whentokenstalktoomuch: Asurveyofmultimodallong-contexttokencompressionacrossimages,videos,andaudios.arXiv preprint arXiv:2507.20198, 2025

    KeleShao,KedaTao,KejiaZhang,SichengFeng,MuCai,YuzhangShang,HaoxuanYou,CanQin,YangSui,andHuanWang. Whentokenstalktoomuch: Asurveyofmultimodallong-contexttokencompressionacrossimages,videos,andaudios.arXiv preprint arXiv:2507.20198, 2025

    arXiv 2025

  33. [33]

    Less is more: A simple yet effective token reduction method for efficient multi-modal llms, 2024

    Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi-modal llms, 2024

    2024

  34. [34]

    video-salmonn: Speech-enhanced audio-visual large language models.arXiv preprint arXiv:2406.15704, 2024

    GuangzhiSun,WenyiYu,ChangliTang,XianzhaoChen,TianTan,WeiLi,LuLu,ZejunMa,YuxuanWang,andChaoZhang. video-salmonn: Speech-enhanced audio-visual large language models.arXiv preprint arXiv:2406.15704, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Lvpruning: Aneffectiveyet simple language-guided vision token pruning approach for multi-modal large language models

    YizhengSun,YanzeXin,HaoLi,JingyuanSun,ChenghuaLin,andRizaTheresaBatista-Navarro. Lvpruning: Aneffectiveyet simple language-guided vision token pruning approach for multi-modal large language models. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 4299–4308, 2025

    2025

  36. [36]

    video-salmonn2: Caption-enhanced audio-visual large language models.arXiv preprint arXiv:2506.15220, 2025

    ChangliTang, YixuanLi, YudongYang, JiminZhuang, GuangzhiSun, WeiLi, ZejunMa, andChaoZhang. video-salmonn2: Caption-enhanced audio-visual large language models.arXiv preprint arXiv:2506.15220, 2025

    arXiv 2025

  37. [37]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18992–19001, 2025

    2025

  38. [38]

    Omnizip: Audio-guided dynamic token compression for fast omnimodal large language models.arXiv preprint arXiv:2511.14582, 2025

    Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Huan Wang, et al. Omnizip: Audio-guided dynamic token compression for fast omnimodal large language models.arXiv preprint arXiv:2511.14582, 2025. 13 MAC-AutoML

    Pith/arXiv arXiv 2025

  39. [39]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, An- drew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    PengWang,ShuaiBai,SinanTan,ShijieWang,ZhihaoFan,JinzeBai,KeqinChen,XuejingLiu,JialinWang,WenbinGe,etal. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Longvlm: Efficientlongvideounderstandingvia large language models

    YuetianWeng,MingfeiHan,HaoyuHe,XiaojunChang,andBohanZhuang. Longvlm: Efficientlongvideounderstandingvia large language models. InEuropean Conference on Computer Vision, pages 453–470. Springer, 2024

    2024

  42. [42]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. InICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  43. [43]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational conference on machine learning, pages 38087–38099. PMLR, 2023

    2023

  44. [44]

    Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    JinXu,ZhifangGuo,HangruiHu,YunfeiChu,XiongWang,JinzhengHe,YuxuanWang,XianShi,TingHe,XinfaZhu,etal. Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Humanomniv2: From understanding to omni-modal reasoning with context.arXiv preprint arXiv:2506.21277, 2025

    Qize Yang, Shimin Yao, Weixuan Chen, Shenghao Fu, Detao Bai, Jiaxing Zhao, Boyuan Sun, Bowen Yin, Xihan Wei, and Jingren Zhou. Humanomniv2: From understanding to omni-modal reasoning with context.arXiv preprint arXiv:2506.21277, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792–19802, 2025

    2025

  50. [50]

    Omnivinci: Enhancing architecture and data for omni-modal understanding llm.arXiv preprint arXiv:2510.15870, 2025

    Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, et al. Omnivinci: Enhancing architecture and data for omni-modal understanding llm.arXiv preprint arXiv:2510.15870, 2025

    arXiv 2025

  51. [51]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference.arXiv preprint arXiv:2410.04417, 2024

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference.arXiv preprint arXiv:2410.04417, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning.arXiv preprint arXiv:2503.05379, 2025

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning.arXiv preprint arXiv:2503.05379, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    An information theory-inspired strategy for automatic network pruning.arXiv preprint arXiv:2108.08532, 2021

    Xiawu Zheng, Yuexiao Ma, Teng Xi, Gang Zhang, Errui Ding, Yuchao Li, Jie Chen, Yonghong Tian, and Rongrong Ji. An information theory-inspired strategy for automatic network pruning.arXiv preprint arXiv:2108.08532, 2021

    Pith/arXiv arXiv 2021

  54. [54]

    Self-Embed

    ZiweiZhou, Rui Wang, and ZuxuanWu. Daily-omni: Towardsaudio-visualreasoningwithtemporal alignmentacrossmodal- ities.arXiv preprint arXiv:2505.17862, 2025. 14 MAC-AutoML Appendix A Per-Domain Results on WorldSense WorldSense spans eight diverse domains. As shown in Table 6, OmniScope achieves the highest average accuracy at both45%and25%retentionacrossboth...

    arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.