Pith. sign in

REVIEW 3 major objections 6 minor 91 references

Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that deferring audio pruning until after query-conditioned fusion lets an omni-modal LLM cut 78% of prefill FLOPs while keeping 97% of full-token accuracy.

desk verdict Solid two-stage compression paper with real efficiency gains, but the headline average masks a category-level temporal-localization failure that needs honest scoping. read the letter →

arxiv 2608.08794 v1 pith:N2BCLDNR submitted 2026-08-09 cs.AI cs.MMcs.SD

Kyeongyoon Lee , Hongyeob Kim , Youngeun Kim , Sungeun Hong This is my paper · ORCID
classification cs.AIcs.MMcs.SD
keywords omni-modalLLMtokencompressiondeferredpruningKV-cacheaudio-visualalignmentCKAefficientinferenceprefillFLOPs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Omni-modal language models that listen and watch at the same time pay for long audio-video sequences during prefill and in the KV cache. This paper claims that the right compression schedule changes as the input crosses the LLM boundary: before fusion, audio tokens carry denser, harder-to-replace evidence than video tokens, so audio should be preserved while video is thinned using short-window audio-visual co-variation; after the query has fused with both modalities, low-relevance audio and visual tokens can be pruned together. The proposed two-stage method, A-PACK, realizes this schedule and, on Qwen2.5-Omni-7B/3B across AVUT, WorldSense, DailyOmni, and Video-MME, keeps 97.0% of full-token average accuracy at the 25% prefill-FLOPs tier while cutting prefill FLOPs by about 78%, reducing final token retention to 5.6%, and raising decoding throughput by up to 2.21 times. If correct, this makes omni-modal compression a stage- and modality-dependent decision rather than a single pre-LLM token-selection problem.

What carries the argument

The load-bearing machinery is a two-stage schedule with a gated local-CKA alignment score as its named component; CKA (Centered Kernel Alignment) is a covariance-structure similarity measure between two sets of representations. Before the LLM, A-PACK preserves all audio tokens, segments the video, and allocates the visual budget using query relevance combined with a short-window linear CKA score computed over paired, mean-centered frame-level embeddings in a length-3 window; when the clip-wide CKA falls below a threshold, the score is blended with a normalized frame-audio distance to catch temporally shifted correspondences. Within each segment, the first frame becomes the anchor and is sampled with density-aware farthest-point sampling, while non-anchor frames keep only the tokens least similar to the anchor, removing within-scene redundancy. Inside the LLM, starting at the middle decoder layer, each retained audio and visual token is scored by scaled query-key attention with the last query token, the top $(1-P)$ fraction is kept, and the corresponding KV-cache entries are deleted at each pruning layer; this progressive schedule shortens attention computation and the cache for later layers and for autoregressive decoding.

What would settle it

Run A-PACK on an audio-visual benchmark of questions that require second- or sub-second temporal localization, such as the Event-Location subset of AVUT restricted to answer intervals under one second, and compare accuracy against the Full-Tokens model; a disproportionate accuracy drop on these short-interval questions would show that post-fusion pruning discards brief evidence and the central claim would need qualification.

Watch

Extended reading notes

Core claim

The central discovery is that audio and video should be compressed on different schedules around the LLM boundary. Audio tokens show roughly 9.8 times higher per-token task relevance under occlusion and 1.7 times higher effective rank (a measure of how many independent directions a token set uses) than video tokens, so early audio pruning loses compact, locally timed evidence that visual tokens cannot replace. The paper further finds that local audio-visual dynamics, measured as windowed linear CKA over time-aligned frames, are a better cue for allocating visual tokens than pointwise audio-video matching, which can wrongly connect temporally distant events. After the LLM has built query-conditioned context, progressively pruning tokens by last-query attention and deleting their KV-cache entries yields the advertised efficiency: at matched 35% and 25% prefill-FLOPs tiers, A-PACK achieves the strongest average accuracy among the training-free compressors tested, with 97.0% of full-token accuracy at the aggressive tier.

Load-bearing premise

The load-bearing premise is that tokens with low query attention after fusion can be discarded without losing answer-critical evidence, including brief audio events and exact temporal cues; the paper's own failure case, a missed second-level laugh localization, shows this premise can fail under aggressive compression.

Editorial extensions

If this is right

  • At matched prefill-FLOPs tiers, A-PACK has the highest average accuracy among the evaluated training-free compressors on both Qwen2.5-Omni-7B and 3B, so deferring pruning until after query-conditioned fusion is a better operating point than early pruning.
  • The inner-LLM pruning stage shortens the KV cache, which is what produces the decoding-throughput gain (up to 2.21 times) and lower GPU memory, not just the prefill savings; pre-LLM-only methods cannot achieve this.
  • Keeping audio under a fixed token budget improves accuracy on both audio-centric AVUT and the more visual WorldSense, so audio acts as guidance rather than as competition for the visual budget.
  • Local-window alignment raises accuracy in all eight WorldSense domains relative to static pointwise matching, with a significant McNemar test (p=0.0117), supporting temporally local audio-visual dynamics as a selection signal.
  • Because the method is training-free, it can be applied to existing omni-modal backbones without fine-tuning, and its per-layer pruning schedule can be tuned to a target prefill-FLOPs budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same 'defer pruning until the query conditions the representation' principle could generalize to other high-information, low-token-rate input streams beyond audio, such as event-level metadata or depth, and to other encoder-decoder architectures.
  • The 5.6% final retention suggests that, once cross-modal context is formed, most tokens are redundant for typical questions; a natural extension is to learn per-layer retention schedules that depend on query difficulty rather than fixed per-layer drop rates.
  • The missed second-level laugh localization implies a testable fix: protect brief, high-salience audio transitions with a query-adaptive retention floor in the inner-LLM stage, or add a finer-grained allocation within retained intervals.
  • The audio-visual asymmetry (higher per-token importance and diversity for audio) could be measured per task and used as a general scheduling rule: modally asymmetric compression budgets may outperform symmetric ones in other omni-modal settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes A-PACK, a training-free two-stage token-compression framework for omni-modal LLMs. In Stage 1 (pre-LLM), audio tokens are preserved and the visual budget is allocated using a gated local-window CKA measure combined with coarse query relevance, followed by segment- and anchor-based redundancy removal. In Stage 2 (inner-LLM), starting from the middle decoder layer, low-query-attention audio and visual tokens are progressively pruned together with their KV-cache entries. The authors motivate the design with measurements showing higher per-token information density and representational diversity for audio than for video, and with a claim that local audio-visual co-variation is a better visual-selection cue than token-wise similarity. On Qwen2.5-Omni-7B/3B at matched 35% and 25% prefill-FLOPs tiers, A-PACK is reported to achieve the strongest average accuracy across AVUT, WorldSense, Video-MME, and DailyOmni among the evaluated training-free compressors, retaining 97.0% of full-token average accuracy at the 25% tier on the 7B model while cutting prefill FLOPs by about 78% and final token retention to 5.6%, and improving decoding throughput by up to 2.21x. A separate AVHBench analysis reports improved hallucination robustness for the gated-CKA alignment score.

Significance. If the empirical claims survive scrutiny, the contribution is genuinely useful for efficient omni-modal inference: the central idea of deferring audio pruning until query-conditioned multimodal context exists, and of using local (windowed) audio-visual structure rather than pointwise similarity for visual allocation, is clearly differentiated from the surveyed pre-LLM compression methods. The paper's strengths include matched-FLOPs comparisons with a transparent analytic FLOPs model (Appendix G); evaluation at two backbone scales with four main benchmarks plus an AVHBench diagnostic; stage and alignment-scorer ablations; a paired exact McNemar test (p = 0.0117) for the local-alignment hypothesis; per-parameter sensitivity tables; honest reporting of a failure case (Figure S10); and a commitment to release code. The headline claims are, however, averages that hide a category-level failure on fine-grained temporal localization, and the evaluation protocol leaves room for benchmark-driven hyperparameter selection, so the magnitude of the reported advantage over baselines should be treated as provisional until those issues are addressed.

major comments (3)
  1. [Section 4.2, Table 1; Table S7; Figure S10] The stress-test concern lands: at the 25% prefill-FLOPs tier on Qwen2.5-Omni-7B, Table S7 shows that A-PACK's AVUT Event Location score drops to 34.7 from the Full-Tokens value of 38.2 (90.8% relative), and that this is below OmniZip (35.9), UniComp (38.8), and Random (38.8) on the same category; Figure S10 documents the same phenomenon, with A-PACK missing the second-level localization of a laugh despite retaining audio and aligned visual context. Because Section 1 motivates preserving audio before the LLM precisely to avoid losing brief, temporally localized audio evidence, this category-level degradation contradicts the load-bearing premise of deferred audio pruning for fine-grained temporal reasoning, and the 97.0% average in Table 1 obscures it. The headline claim should be scoped to average benchmark accuracy, category-level AVUT results should be reported in the main text, and the paper should discuss how the inner-LLM pruning or the pre-LLM allocation could protect short audio transitions.
  2. [Section 4.1; Section 4.5; Table S1; Table S5] The central comparison in Table 1 is run with hyperparameters selected on the evaluation benchmarks themselves: Section 4.5 states that for each tier the pre-LLM retention ratio is chosen as the highest value within the FLOPs budget from a retention sweep (Figure S4c), and Table S1 lists fifteen free parameters (w, tau, beta, gamma, tau_sim, sigma, DA-FPS k and power, dedup coverage, anchor priority, R_pre, P, L_mid, lambda) whose defaults are used in the main table. Table S5 shows that several of these parameters have sizable effects (temporal window w=5 gives -3.52 points on a WorldSense subset; continuity threshold tau_sim=0.99 gives -2.51 points), so the reported gains are not obviously robust to reasonable parameter variation. Because the central claim is that A-PACK is the strongest average compressor at matched FLOPs, the evaluation should either lock a single configuration through a validation-based selection protocol and report test results once, or show that the ranking over baselines is preserved under a fixed default configuration; at minimum, the selection protocol should be disclosed in the main text.
  3. [Section 3.1, Eq. (2), Eq. (3); Algorithm 1] The pre-LLM frame scoring is specified inconsistently and incompletely: Eq. (2) defines s_f as either c_f or the blend (1-beta)c_f + beta(1-d_hat_f), whereas Algorithm 1 defines the base visual score as b_f = gamma*u_f + (1-gamma)*c_f and then sets s_f from it, and the surrounding text only says that frame-level allocation combines query relevance with the local alignment score without giving the combination. Moreover, the frame-query relevance r_f used in Eq. (3) for segment scoring is never defined, and the balance parameter lambda between peak and sustained relevance in Eq. (3) is not assigned a value in Table S1. As written, the method cannot be reimplemented unambiguously, which is a precondition for verifying the efficiency-accuracy claims in Table 1.
minor comments (6)
  1. [Section 4.1, Implementation Details] The sentence 'A-PACK starts inner-LLM pruning at layers 14/18 for 7B/3B using a 10%/15% per-layer drop in the 35%/25% tiers' ambiguously mixes backbone-dependent and tier-dependent settings; Table S1 clarifies that L_mid = depth/2 (14 for 7B, 18 for 3B) while P is 10% in the 35% tier and 15% in the 25% tier, so the main text should spell out this mapping.
  2. [Table 1, Final Ret. column] The reported final token retentions (16.8% at the 35% tier and 5.6% at the 25% tier) should be reconciled with the stated pre-LLM retentions (R_pre = 0.50/0.35) and per-layer pruning (P = 10%/15%); for example, 0.35 x 0.85^k = 0.056 implies roughly 11 effective pruning steps, so stating the number of layers at which pruning is applied would remove an apparent inconsistency.
  3. [Abstract; Section 4.1] The abstract says 'across four benchmarks,' while Section 4.1 lists five benchmarks because AVHBench also appears in Table 4 and in the supplementary material; the benchmark counting should be made consistent or explicitly qualified.
  4. [Section 3.2, Eq. (5)] Please clarify whether the token set S includes text tokens and how the normalization in Eq. (5) over S relates to the decoder's softmax over all keys (including text), since the pruning scores are extracted from the attention computation.
  5. [Section 4.5, Figure 2] A one-sentence summary of the occlusion and effective-rank protocols should appear in the main text, because the '9.8x higher information density' and '1.7x higher effective rank' claims are used to justify the audio-preservation design.
  6. [Section 3.2] The statement that query-key attention scores require 'no additional forward pass' should note that standard fused-attention implementations do not expose attention weights and that extracting them may require a modified attention kernel.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: A-PACK's headline accuracy and efficiency numbers are measured against external benchmarks, and its components (CKA scoring, query-attention pruning, audio-density analysis) operate as empirical inputs rather than outputs of the claimed result.

full rationale

A-PACK's claims are empirical rather than self-derivational. Equations (1)-(6) define a concrete compression procedure (local CKA scoring, gated fine scoring, segment-level budget allocation, anchor-based token selection, and inner-LLM query-attention pruning), and the reported accuracies in Tables 1, S7, and S8 are measured on external benchmarks with a common decoding protocol, not reconstructed from the method's definitions. There is no load-bearing self-citation chain: CKA is cited to Kornblith et al. (2019) and Yang et al. (2026b), while the segmentation and sampling components cite external sources such as CoSeLECT and DA-FPS, none involving the present authors. No uniqueness theorem is imported from prior work by the same authors. The closest concern is Section 4.5's statement that 'for each tier we choose the highest pre-LLM ratio that remains within its prefill-FLOPs budget'; this is benchmark-driven hyperparameter selection, which may inflate apparent gains, but it is not a fitted parameter renamed as a prediction, and it does not make the reported accuracy equivalent to the method's inputs. The acknowledged limitation in Figure S10 and the conclusion's caveat that 'aggressive compression may affect fine-grained or long-range evidence' identify robustness risks, not circularity.

Assumptions & free parameters 14 free parameters · 7 assumptions · 0 invented entities

The central claims rest on the public Qwen backbone, the benchmark labels, several perceptual assumptions about audio and video tokens, and an analytic FLOPs model. The most fragile items are the FLOPs accounting and the benchmark-driven choice of R_pre, tau, beta, w, and P, because they directly set the reported accuracy-efficiency trade-off.

free parameters (14)
  • Window size w = 3
    Chosen from a sensitivity sweep: temporal CKA accuracy peaks at w=3 (Figure 6b, Table S5).
  • Gate threshold tau = 0.78
    Selected via gate-threshold sweep; the gate is the more sensitive alignment control (Figure S4a, Table S5).
  • Fine-blend beta = 0.90
    Selected via fine-blend sweep; accuracy is flat within about 0.25 points around this value (Figure S4b, Table S5).
  • Visual-score weight gamma = 0.9
    Default in Table S1 and stable across the CKA weight sweep in Figure 6c.
  • Continuity threshold tau_sim = 0.97
    Chosen as a robust shared segmentation setting; it is the most sensitive segmentation control (Table S5).
  • Gaussian smoothing sigma = 1.0
    Default in Table S1 for continuity smoothing; tested values 0.5 and 1.5 give no change (Table S5).
  • DA-FPS density k = 5
    Default in Table S1; sensitivity tested in Table S5.
  • DA-FPS power = 0.5
    Default in Table S1; lower powers temper the density term and perform better (Table S5).
  • Dedup coverage threshold = 0.8
    Default in Table S1; sensitivity tested in Table S5.
  • Anchor priority = 0.3
    Default in Table S1; sensitivity tested in Table S5.
  • Pre-LLM retention R_pre = 0.50 (35% tier) / 0.35 (25% tier)
    Chosen from a retention sweep to maximize accuracy within the prefill-FLOPs budget on the same benchmarks used for evaluation (Section 4.5).
  • Per-layer pruning P = 10% (35% tier) / 15% (25% tier)
    Tier-specific configuration from Table S1; ablations fix P at 15% when varying other parameters.
  • Mid-layer L_mid = 14 (7B) / 18 (3B)
    Set to decoder depth/2 and defines when deferred audio pruning begins (Table S1).
  • Segment peak/sustained balance lambda = Not reported
    Used in Eq. (3) for segment scoring; no default is listed in Table S1, which is a reproducibility gap.
assumptions (7)
  • domain assumption Qwen2.5-Omni 7B/3B checkpoints provide fixed, reliable token representations and attention for the claimed measurements.
    All accuracy and efficiency numbers are generated from this public backbone without fine-tuning (Section 4.1).
  • domain assumption AVUT, WorldSense, Video-MME, DailyOmni, and AVHBench are valid proxies for omni-modal understanding with correct labels.
    Headline claims average accuracy over these benchmark sets; label correctness and task design are taken from the cited datasets.
  • domain assumption Linear CKA over a w=3 window captures audio-visual co-evolution that is useful for visual budget allocation.
    Eq. (1) defines the score; Figure 6b and Table S5 show w=3 was selected empirically.
  • domain assumption The last-query attention score in Eq. (5), averaged over heads, is a reliable proxy for the final relevance of a cached token.
    Inner-LLM pruning removes tokens with low query attention; no direct counterfactual verification that discarded tokens are never needed by later decoding steps.
  • domain assumption Mean-pooling frames and resampling audio onto the frame axis preserve the information used for segmentation and alignment scoring.
    The coarse pass in Section 3.1 relies on pooled frame-level embeddings for all budget decisions.
  • domain assumption The analytic prefill FLOPs model phi(n)=4nd^2+2n^2d+2ndm accurately represents per-layer compute for every compared method.
    Appendix G.1 uses Eq. (4)-(6) to define matched prefill-FLOPs tiers; kernel-level overheads are not modeled.
  • standard math Standard mathematical tools (CKA, effective rank, DA-FPS, and McNemar tests) are applied as defined in their cited sources.
    The validity of the paper's measurements depends on these methods behaving as described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs." pith.science (2026). https://pith.science/paper/N2BCLDNR

@misc{pith2026260808794,
  author       = {Pith},
  title        = {Pith review of: Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N2BCLDNR}},
  note         = {Machine review of arXiv:2608.08794}
}
read the original abstract

Omni-modal LLMs jointly process audio, video, and text, but long multimodal sequences incur substantial prefill and KV-cache costs. Existing omni-modal compression methods primarily focus on pre-LLM token reduction, leaving modality-specific compression across the LLM boundary underexplored. We propose A-PACK, a two-stage framework that defers audio pruning until query-conditioned multimodal interactions emerge. Our analysis shows that audio exhibits higher task-relevant information density and representational diversity per token than video. We further find that local audio-visual dynamics provide a more effective cue for visual selection than token-wise matching. We therefore preserve audio and compress video with local dynamics before the LLM, then progressively prune low-relevance audio and visual tokens and their KV-cache entries inside the LLM. Across four benchmarks on Qwen2.5-Omni-7B/3B, A-PACK achieves the strongest average performance among the evaluated prior methods while reducing prefill FLOPs by up to 78% and improving decoding throughput by up to 2.21x.

Figures

Figures reproduced from arXiv: 2608.08794 by the authors.

Figure 1
Figure 1. Motivation of our method. Early audio pruning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Audio tokens show significantly higher per-token [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our method. Stage 1 (pre-LLM) preserves audio tokens and compresses only video. It allocates more visual tokens to frames with stronger local audio–visual CKA, then removes within-video redundancy through anchor-based token selection. Stage 2 (inner-LLM) progressively prunes query-irrelevant audio and visual tokens after multimodal interaction. Their corresponding KV-cache entries are also removed, reali… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Inference efficiency of Qwen2.5-Omni-7B at the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: A-PACK has the lowest prefilling time on omni [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Key sensitivity analyses. (a) Retaining more audio improves accuracy under a fixed total budget. (b) WorldSense [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: (a) A local-window mask defines locally aligned audio–visual token pairs. (b) Static cosine can assign high similarity to temporally distant matches. (c) A-PACK uses gated local temporal correspondence to guide visual selection. (d) Local-window alignment improves all …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 66 canonical work pages

  1. [1]

    Qwen2.5-Omni Technical Report

    Xu, Jin and Guo, Zhifang and He, Jinzheng and Hu, Hangrui and He, Ting and Bai, Shuai and Chen, Keqin and Wang, Jialin and Fan, Yang and Dang, Kai and others. Qwen2.5-Omni Technical Report. arXiv:2503.20215

    arXiv

  2. [2]

    2026 , doi=

    Locality-Aware Cross-Modal Correspondence Learning for Dense Audio-Visual Events Detection , author=. 2026 , doi=

    2026

  3. [3]

    Do Vision and Language Encoders Represent the World Similarly? , author=

  4. [4]

    Target Bias Is All You Need: Zero-Shot Debiasing of Vision-Language Models with Bias Corpus , author=

  5. [5]

    HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model , author=

  6. [6]

    Audio-Visual Scene Analysis with Self-Supervised Multisensory Features , author=

  7. [7]

    AVE-CLIP: AudioCLIP-Based Multi-Window Temporal Transformer for Audio Visual Event Localization , author=

  8. [8]

    Density-Aware Farthest Point Sampling , author=. Trans. Mach. Learn. Res. , year=

Show all 91 references
  1. [9]

    FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs , author=

  2. [10]

    What's Making That Sound Right Now? Video-centric Audio-Visual Localization , author=

  3. [11]

    TIM: A Time Interval Machine for Audio-Visual Action Recognition , author=

  4. [12]

    Audio-Visual Event Localization in Unconstrained Videos , author=

  5. [13]

    LMM s-Eval: Reality Check on the Evaluation of Large Multimodal Models

    Zhang, Kaichen and Li, Bo and Zhang, Peiyuan and Pu, Fanyi and Cahyono, Joshua Adrian and Hu, Kairui and Liu, Shuai and Zhang, Yuanhan and Yang, Jingkang and Li, Chunyuan and Liu, Ziwei. LMM s-Eval: Reality Check on the Evaluation of Large Multimodal Models. Findings of the As...

    2025 doi

  6. [14]

    Ola: Pushing the Frontiers of Omni-Modal Language Model

    Liu, Zuyan and Dong, Yuhao and Wang, Jiahui and Liu, Ziwei and Hu, Winston and Lu, Jiwen and Rao, Yongming. Ola: Pushing the Frontiers of Omni-Modal Language Model. arXiv:2502.04328

    arXiv

  7. [15]

    Baichuan-Omni-1.5 Technical Report

    Li, Yadong and Liu, Jun and Zhang, Tao and Chen, Song and Li, Tianpeng and Li, Zehuan and Liu, Lijun and Ming, Lingfeng and Dong, Guosheng and Pan, Da and others. Baichuan-Omni-1.5 Technical Report. arXiv:2501.15368

  8. [16]

    Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition

    Zhong, Zhisheng and Wang, Chengyao and Liu, Yuqi and Yang, Senqiao and Tang, Longxiang and Zhang, Yuechen and Li, Jingyao and Qu, Tianyuan and Li, Yanwei and Chen, Yukang and others. Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition

  9. [17]

    LLaVA-OneVision : Easy Visual Task Transfer

    Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan. LLaVA-OneVision : Easy Visual Task Transfer

  10. [18]

    VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

    Cheng, Zesen and Leng, Sicong and Zhang, Hang and Xin, Yifei and Li, Xin and Chen, Guanzheng and Zhu, Yongxin and Zhang, Wenqi and Luo, Ziyang and Zhao, Deli and Bing, Lidong. VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. arXiv:2406.07476

    arXiv

  11. [19]

    VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding

    Zhang, Boqiang and Li, Kehan and Cheng, Zesen and Luo, Zhiqiang and Leng, Sicong and Zhang, Hang and Li, Xin and Chen, Guanzheng and Zhu, Yongxin and Zhao, Deli and others. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv:2501.13106

    arXiv

  12. [20]

    VideoChat-Flash : Hierarchical Compression for Long-Context Video Modeling

    Li, Xinhao and Wang, Yi and Yu, Jiashuo and Zeng, Xiangyu and Zhu, Yuhan and Huang, Haian and Gao, Jianfei and Li, Kunchang and He, Yinan and Wang, Chenting and others. VideoChat-Flash : Hierarchical Compression for Long-Context Video Modeling. arXiv:2501.00574

    arXiv

  13. [21]

    LongVILA : Scaling Long-Context Visual Language Models for Long Videos

    Chen, Yukang and Xue, Fuzhao and Li, Dacheng and Hu, Qinghao and Zhu, Ligeng and Li, Xiuyu and Fang, Yunhao and Tang, Haotian and Yang, Shang and Liu, Zhijian and others. LongVILA : Scaling Long-Context Visual Language Models for Long Videos

  14. [22]

    LongVU : Spatiotemporal Adaptive Compression for Long Video-Language Understanding

    Shen, Xiaoqian and Xiong, Yunyang and Zhao, Changsheng and Wu, Lemeng and Chen, Jun and Zhu, Chenchen and Liu, Zechun and Xiao, Fanyi and Varadarajan, Balakrishnan and Bordes, Florian and others. LongVU : Spatiotemporal Adaptive Compression for Long Video-Language Understanding

  15. [23]

    Video Instruction Tuning with Synthetic Data

    Zhang, Yuanhan and Wu, Jinming and Li, Wei and Li, Bo and Ma, Zejun and Liu, Ziwei and Li, Chunyuan. Video Instruction Tuning with Synthetic Data

  16. [24]

    Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding

    Shu, Yan and Liu, Zheng and Zhang, Peitian and Qin, Minghao and Zhou, Junjie and Liang, Zhengyang and Huang, Tiejun and Zhao, Bo. Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding

  17. [25]

    An Image Is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

    Chen, Liang and Zhao, Haozhe and Liu, Tianyu and Bai, Shuai and Lin, Junyang and Zhou, Chang and Chang, Baobao. An Image Is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

  18. [26]

    DyCoke : Dynamic Compression of Tokens for Fast Video Large Language Models

    Tao, Keda and Qin, Can and You, Haoxuan and Sui, Yang and Wang, Huan. DyCoke : Dynamic Compression of Tokens for Fast Video Large Language Models

  19. [27]

    PruneVid : Visual Token Pruning for Efficient Video Large Language Models

    Huang, Xiaohu and Zhou, Hao and Han, Kai. PruneVid : Visual Token Pruning for Efficient Video Large Language Models

  20. [28]

    FastVID : Dynamic Density Pruning for Fast Video Large Language Models

    Shen, Leqi and Gong, Guoqiang and He, Tao and Zhang, Yifeng and Liu, Pengzhang and Zhao, Sicheng and Ding, Guiguang. FastVID : Dynamic Density Pruning for Fast Video Large Language Models

  21. [29]

    HoliTom : Holistic Token Merging for Fast Video Large Language Models

    Shao, Kele and Tao, Keda and Qin, Can and You, Haoxuan and Sui, Yang and Wang, Huan. HoliTom : Holistic Token Merging for Fast Video Large Language Models

  22. [30]

    OmniZip : Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models

    Tao, Keda and Shao, Kele and Yu, Bohan and Wang, Weiqiang and Liu, Jian and Wang, Huan. OmniZip : Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models

  23. [31]

    Multimodal Long Video Modeling Based on Temporal Dynamic Context

    Hao, Haoran and Han, Jiaming and Zhang, Yiyuan and Yue, Xiangyu. Multimodal Long Video Modeling Based on Temporal Dynamic Context. arXiv:2504.10443

    arXiv

  24. [32]

    Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs

    Hyun, Jeongseok and Hwang, Sukjun and Han, Su Ho and Kim, Taeoh and Lee, Inwoong and Wee, Dongyoon and Lee, Joon-Young and Kim, Seon Joo and Shim, Minho. Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs

  25. [33]

    Aligned Better, Listen Better for Audio-Visual Large Language Models

    Guo, Yuxin and Ma, Shuailei and Ma, Shijie and Bao, Xiaoyi and Xie, Chen-Wei and Zheng, Kecheng and Weng, Tingyu and Sun, Siyang and Zheng, Yun and Zou, Wei. Aligned Better, Listen Better for Audio-Visual Large Language Models

  26. [34]

    Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

    Chowdhury, Sanjoy and Nag, Sayan and Dasgupta, Subhrajyoti and Chen, Jun and Elhoseiny, Mohamed and Gao, Ruohan and Manocha, Dinesh. Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

  27. [35]

    AVHBench : A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models

    Sung-Bin, Kim and Hyun-Bin, Oh and Lee, JungMok and Senocak, Arda and Chung, Joon Son and Oh, Tae-Hyun. AVHBench : A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models

  28. [36]

    AVCD : Mitigating Hallucinations in Audio-Visual Large Language Models Through Contrastive Decoding

    Jung, Chaeyoung and Jang, Youngjoon and Chung, Joon Son. AVCD : Mitigating Hallucinations in Audio-Visual Large Language Models Through Contrastive Decoding

  29. [37]

    AVQA : A Dataset for Audio-Visual Question Answering on Videos

    Yang, Pinci and Wang, Xin and Duan, Xuguang and Chen, Hong and Hou, Runze and Jin, Cong and Zhu, Wenwu. AVQA : A Dataset for Audio-Visual Question Answering on Videos

  30. [38]

    Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs

    Jung, Chaeyoung and Rho, Kyeongha and Chung, Joon Son. Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs. arXiv:2605.11605

    arXiv

  31. [39]

    OmniDrop : Layer-Wise Token Pruning for Omni-Modal LLMs via Query-Guidance

    Park, Yeo Jeong and Jang, Hyemi and Choi, Minseo and Lee, Jongsun and Choi, Jooyoung and Jeon, Yongkweon. OmniDrop : Layer-Wise Token Pruning for Omni-Modal LLMs via Query-Guidance. arXiv:2605.14458

    arXiv

  32. [40]

    The Platonic Representation Hypothesis

    Huh, Minyoung and Cheung, Brian and Wang, Tongzhou and Isola, Phillip. The Platonic Representation Hypothesis

  33. [41]

    Understanding the Emergence of Multimodal Representation Alignment

    Tjandrasuwita, Megan and Ekbote, Chanakya and Ziyin, Liu and Liang, Paul Pu. Understanding the Emergence of Multimodal Representation Alignment

  34. [42]

    To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance

    Fang, Wanlong and Zhang, Tianle and Chan, Alvin. To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance

  35. [43]

    Similarity of Neural Network Representations Revisited

    Kornblith, Simon and Norouzi, Mohammad and Lee, Honglak and Hinton, Geoffrey. Similarity of Neural Network Representations Revisited

  36. [44]

    The Effective Rank: A Measure of Effective Dimensionality

    Roy, Olivier and Vetterli, Martin. The Effective Rank: A Measure of Effective Dimensionality. European Signal Processing Conference (EUSIPCO)

  37. [45]

    Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data

    Radovanovi \'c , Milo s and Nanopoulos, Alexandros and Ivanovi \'c , Mirjana. Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data

  38. [46]

    Sinkhorn Distances: Lightspeed Computation of Optimal Transport

    Cuturi, Marco. Sinkhorn Distances: Lightspeed Computation of Optimal Transport

  39. [47]

    Attention-Weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition

    Yang, Qingran and Zhao, Botao and Kang, Zuheng and Li, Xue and He, Yayun and Liu, Chuhang and Zhang, Xulong and Qu, Xiaoyang and Peng, Junqing and Wang, Jianzong. Attention-Weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to ...

  40. [48]

    Adaptive Keyframe Sampling for Long Video Understanding

    Tang, Xi and Qiu, Jihao and Xie, Lingxi and Tian, Yunjie and Jiao, Jianbin and Ye, Qixiang. Adaptive Keyframe Sampling for Long Video Understanding

  41. [49]

    Clustering by Fast Search and Find of Density Peaks

    Rodriguez, Alex and Laio, Alessandro. Clustering by Fast Search and Find of Density Peaks. Science

  42. [50]

    and Yi, Li and Su, Hao and Guibas, Leonidas J

    Qi, Charles R. and Yi, Li and Su, Hao and Guibas, Leonidas J. PointNet++ : Deep Hierarchical Feature Learning on Point Sets in a Metric Space

  43. [51]

    Attributing Response to Context: A Jensen--Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation

    Li, Ruizhe and Chen, Chen and Hu, Yuchen and Gao, Yanjun and Wang, Xi and Yilmaz, Emine. Attributing Response to Context: A Jensen--Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation

  44. [52]

    Quantifying the Plausibility of Context Reliance in Neural Machine Translation

    Sarti, Gabriele and Chrupa a, Grzegorz and Nissim, Malvina and Bisazza, Arianna. Quantifying the Plausibility of Context Reliance in Neural Machine Translation

  45. [53]

    AgilePruner : An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models

    Baek, Changwoo and Song, Jouwon and Kim, Sohyeon and Kong, Kyeongbo. AgilePruner : An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models

  46. [54]

    CoSeLECT : Adaptive Frame Selection for Video-Language Understanding

    Devnani, Bhavika Suresh and Jain, Jitesh and Shi, Humphrey and Hoffman, Judy. CoSeLECT : Adaptive Frame Selection for Video-Language Understanding. Second Workshop on Video Large Language Models ( VidLLMs ), CVPR 2026

    2026

  47. [55]

    Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-Modal LLMs in Video Analysis

    Fu, Chaoyou and Dai, Yuhan and Luo, Yongdong and Li, Lei and Ren, Shuhuai and Zhang, Renrui and Wang, Zihan and Zhou, Chenyu and Shen, Yunhang and Zhang, Mengdan and others. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-Modal LLMs in Video Analysis

  48. [56]

    WorldSense : Evaluating Real-World Omnimodal Understanding for Multimodal LLMs

    Hong, Jack and Yan, Shilin and Cai, Jiayin and Jiang, Xiaolong and Hu, Yao and Xie, Weidi. WorldSense : Evaluating Real-World Omnimodal Understanding for Multimodal LLMs

  49. [57]

    Daily-Omni : Towards Audio-Visual Reasoning with Temporal Alignment Across Modalities

    Zhou, Ziwei and Wang, Rui and Wu, Zuxuan and Jiang, Yu-Gang. Daily-Omni : Towards Audio-Visual Reasoning with Temporal Alignment Across Modalities. arXiv:2505.17862

  50. [58]

    Audio-Centric Video Understanding Benchmark without Text Shortcut

    Yang, Yudong and Zhuang, Jimin and Sun, Guangzhi and Tang, Changli and Li, Yixuan and Li, Peihan and Jiang, Yifan and Li, Wei and Ma, Zejun and Zhang, Chao. Audio-Centric Video Understanding Benchmark without Text Shortcut

  51. [59]

    Diff-Foley : Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

    Luo, Simian and Yan, Chuanhao and Hu, Chenxu and Zhao, Hang. Diff-Foley : Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

  52. [60]

    Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

    Cheng, Ho Kei and Ishii, Masato and Hayakawa, Akio and Shibuya, Takashi and Schwing, Alexander and Mitsufuji, Yuki. Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

  53. [61]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Yuan, Chao and Chen, Shimin and Lin, Minliang and Qiao, Limeng and Wan, Guanglu and Ma, Lin , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2026 , pages =

    2026

  54. [62]

    FlashVID : Efficient Video Large Language Models via Training-Free Tree-Based Spatiotemporal Token Merging

    Fan, Ziyang and Chen, Keyu and Xing, Ruilong and Li, Yulin and Jiang, Li and Tian, Zhuotao. FlashVID : Efficient Video Large Language Models via Training-Free Tree-Based Spatiotemporal Token Merging

  55. [63]

    TopV : Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model

    Yang, Cheng and Sui, Yang and Xiao, Jinqi and Huang, Lingyi and Gong, Yu and Li, Chendi and Yan, Jinghua and Bai, Yu and Sadayappan, Ponnuswamy and Hu, Xia and Yuan, Bo. TopV : Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision ...

  56. [64]

    PyramidDrop : Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

    Xing, Long and Huang, Qidong and Dong, Xiaoyi and Lu, Jiajie and Zhang, Pan and Zang, Yuhang and Cao, Yuhang and He, Conghui and Wang, Jiaqi and Wu, Feng and Lin, Dahua. PyramidDrop : Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

  57. [65]

    VoCo-LLaMA : Towards Vision Compression with Large Language Models

    Ye, Xubing and Gan, Yukang and Huang, Xiaoke and Ge, Yixiao and Shan, Ying and Tang, Yansong. VoCo-LLaMA : Towards Vision Compression with Large Language Models

  58. [66]

    TimeViper : A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding

    Xu, Boshen and Xiao, Zihan and Li, Jiaze and Ju, Jianzhong and Luo, Zhenbo and Luan, Jian and Jin, Qin. TimeViper : A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding

  59. [67]

    Token-Efficient Long Video Understanding for Multimodal LLMs

    Jiang, Jindong and Li, Xiaojie and Liu, Zhengyuan and Wang, Guo and Huang, De-An and Byeon, Wonmin and others. Token-Efficient Long Video Understanding for Multimodal LLMs

  60. [68]

    BIMBA : Selective-Scan Compression for Long-Range Video Question Answering

    Islam, Md Mohaiminul and Nagarajan, Tushar and Wang, Huiyu and Bertasius, Gedas and Torresani, Lorenzo. BIMBA : Selective-Scan Compression for Long-Range Video Question Answering

  61. [69]

    AdaptInfer : Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance

    Zhang, Weichen and Zhu, Zhui and Li, Ningbo and Tao, Shilong and Liu, Kebin and Liu, Yunhao. AdaptInfer : Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance. arXiv:2508.06084

  62. [70]

    FastAV : Efficient Token Pruning for Audio-Visual Large Language Model Inference

    Jung, Chaeyoung and Jang, Youngjoon and Lee, Seungwoo and Chung, Joon Son. FastAV : Efficient Token Pruning for Audio-Visual Large Language Model Inference. arXiv:2601.13143

  63. [71]

    DASH : Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression

    Li, Bingzhou and Huang, Tao. DASH : Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression. arXiv:2603.15685

    arXiv

  64. [72]

    OmniSelect : Dynamic Modality-Aware Token Compression for Efficient Omni-Modal Large Language Models

    Yang, Morunliu and Xu, Ruotao and Li, Le and Wang, Yue and Zhang, Jianxin and Li, Juntao and Lou, Yihang and Feng, Siwei and Li, Peifeng. OmniSelect : Dynamic Modality-Aware Token Compression for Efficient Omni-Modal Large Language Models. arXiv:2605.18041

    arXiv

  65. [73]

    EchoingPixels : Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

    Gong, Chao and Wang, Depeng and Wei, Zhipeng and Guo, Ya and Zhu, Huijia and Chen, Jingjing. EchoingPixels : Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

  66. [74]

    OmniSIFT : Modality-Asymmetric Token Compression for Efficient Omni-Modal Large Language Models

    Ding, Yue and Ji, Yiyan and Li, Jungang and Liu, Xuyang and Chen, Xinlong and Wu, Junfei and Li, Bozhou and Zeng, Bohan and Shi, Yang and Guan, Yushuo and Zhang, Yuanxing and Liu, Jiaheng and Liu, Qiang and Wan, Pengfei and Wang, Liang. OmniSIFT : Modality-Asymmetric Token Com...

  67. [75]

    Audio-Synchronized Visual Animation

    Zhang, Lin and Mao, Shentong and Liu, Yifan and Wang, Zizhang and Sun, Jiankai and Guo, Menghan and Xu, Chao and Barsoum, Emad. Audio-Synchronized Visual Animation

  68. [76]

    Objects that Sound

    Arandjelovi \'c , Relja and Zisserman, Andrew. Objects that Sound

  69. [77]

    Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment

    Ibrahimi, Sarah and Sun, Xiaohang and Wang, Pichao and Garg, Amanmeet and Sanan, Ashutosh and Omar, Mohamed. Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment

  70. [78]

    Anchor-Aware Deep Metric Learning for Audio-Visual Retrieval

    Zeng, Donghuo and Wang, Yanan and Ikeda, Kazushi and Yu, Yi. Anchor-Aware Deep Metric Learning for Audio-Visual Retrieval

  71. [79]

    Audio-Visual LLM for Video Understanding

    Shu, Fangxun and Zhang, Lei and Jiang, Hao and Xie, Cihang. Audio-Visual LLM for Video Understanding

  72. [80]

    OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models

    Deng, Yuchen and Cai, Zidang and Zheng, Hai-Tao and Wang, Jie and Yang, Feidiao and Han, Yuxing. OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models. arXiv preprint arXiv:2605.12056

    arXiv

  73. [81]

    Stage-adaptive Token Selection for Efficient Omni-modal LLMs

    Xin, Zijie and Yang, Jie and Zhao, Ruixiang and Wang, Tianyi and Rao, Fengyun and Lyu, Jing and Li, Xirong. Stage-adaptive Token Selection for Efficient Omni-modal LLMs. arXiv preprint arXiv:2605.20035

    arXiv

  74. [82]

    Temporal Auditory Acuity

    Green, David M. Temporal Auditory Acuity. Psychological Review

  75. [83]

    and Warren, David H

    Welch, Robert B. and Warren, David H. Immediate Perceptual Response to Intersensory Discrepancy. Psychological Bulletin

  76. [84]

    What You See Is What You Hear

    Shams, Ladan and Kamitani, Yukiyasu and Shimojo, Shinsuke. What You See Is What You Hear. Nature

  77. [85]

    and Olshausen, Bruno A

    Simoncelli, Eero P. and Olshausen, Bruno A. Natural Image Statistics and Neural Representation. Annual Review of Neuroscience

  78. [86]

    and Theunissen, Fr \'e d \'e ric E

    Elliott, Taffeta M. and Theunissen, Fr \'e d \'e ric E. The Modulation Transfer Function for Speech Intelligibility. PLoS Computational Biology

  79. [87]

    and Plomp, Reinier

    Drullman, Rob and Festen, Joost M. and Plomp, Reinier. Effect of Temporal Envelope Smearing on Speech Reception. Journal of the Acoustical Society of America

  80. [88]

    and Zeng, Fan-Gang and Kamath, Vivek and Wygonski, John and Ekelid, Michael

    Shannon, Robert V. and Zeng, Fan-Gang and Kamath, Vivek and Wygonski, John and Ekelid, Michael. Speech Recognition with Primarily Temporal Cues. Science

  81. [89]

    Different Languages, Similar Encoding Efficiency: Comparable Information Rates across the Human Communicative Niche

    Coup \'e , Christophe and Oh, Yoon Mi and Dediu, Dan and Pellegrino, Fran c ois. Different Languages, Similar Encoding Efficiency: Comparable Information Rates across the Human Communicative Niche. Science Advances

  82. [90]

    and Poeppel, David

    van Wassenhove, Virginie and Grant, Ken W. and Poeppel, David. Temporal Window of Integration in Auditory-Visual Speech Perception. Neuropsychologia

  83. [91]

    and Stevenson, Ryan A

    Wallace, Mark T. and Stevenson, Ryan A. The Construct of the Multisensory Temporal Binding Window and Its Dysregulation in Developmental Disabilities. Neuropsychologia

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.