{"id":"a5984516-acc2-403d-851d-9b8d11d24eaa","arxiv_id":"2608.08794","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":14,"one_line_summary":"A training-free, audio-preserving token and KV-cache pruning scheme that delays audio pruning until after query-conditioned fusion, keeping about 97% of accuracy at a 78% prefill FLOP reduction.","lead":"A-PACK is a training-free two-stage compression method for audio-visual LLMs: it keeps all audio tokens before the model, cuts video tokens using local audio-video alignment, and then prunes both audio and video tokens inside the model after the question has been fused. It reports up to 78% lower prefill compute and 2.21x faster decoding while keeping about 97% of full-token accuracy on four benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline average hides a category-level failure: AVUT Event Location at the 25% tier drops to 34.7 vs 38.2 full and trails OmniZip (35.9), so the 97.0% average claim overstates robustness on fine-grained temporal evidence.","rationale":"The reader's verdict is CONDITIONAL, and my analysis does not move it. I agree with the reader's weakest assumption: aggressive post-fusion pruning down to 5.6% final retention assumes that low query-attention tokens are disposable, and the paper's own Figure S10 shows that exact temporal cues can be lost. Table S7 makes this concrete rather than anecdotal: on the AVUT Event Location category at the 25% tier, A-PACK is worse than Full Tokens by 3.5 points and is also below OmniZip on that category, even though A-PACK wins on the AVUT average. This is the most load-bearing concern because the paper's motivation in Section 1 explicitly centers on preserving brief audio evidence and local temporal alignment; if the method can lose second-level event timestamps, the headline average should be scoped. The paper has genuine strengths: consistent results across two backbones, detailed ablations, honest failure analysis, and a clear efficiency accounting. No reason exists to reject or to move the verdict. The condition is that the strong average claim must be qualified by category-level temporal performance, or the method should protect query-relevant acoustic transitions during inner-LLM pruning. The proposed concrete test settles whether the category loss is a real limitation or an artifact of the reported aggregation.","tokens_in":24226,"tokens_out":17854,"duration_ms":192695,"concrete_test":"Re-run the Qwen2.5-Omni-7B 25% prefill-FLOPs tier with per-category reporting on AVUT Event Location and DailyOmni temporal-interval tasks. Compute A-PACK's relative accuracy versus Full Tokens and versus OmniZip on those categories. If Event Location relative accuracy is below 92% while overall AVUT relative accuracy is about 96%, the 97.0% average claim should carry a scoping caveat. As a causal check, run an ablation that exempts a small audio-token window around the annotated event interval from inner-LLM pruning while keeping the same FLOPs budget; if Event Location accuracy recovers materially, the loss is attributable to query-attention pruning rather than the pre-LLM stage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.3 / Table 1 is a benchmark-level average, and it masks a category-level loss in the exact capability the method is motivated to preserve. In Table S7 (Qwen2.5-Omni-7B, 25% tier), AVUT's Event Location category scores 34.7 absolute versus 38.2 for Full Tokens (90.8% relative), and it is below OmniZip (35.9), DyCoke (42.4), UniComp (38.8), and even Random (38.8). A-PACK's overall AVUT average is 62.2 vs 64.5 full (96.4%), so the temporal-localization loss is hidden by the average. Figure S10 is the same phenomenon: A-PACK misses the second-level localization of a laugh. The conclusion itself concedes that 'aggressive compression may affect fine-grained or long-range evidence.' Because the method's stated rationale is that audio should be preserved pre-LLM precisely to avoid losing brief, temporally localized audio events, the load-bearing assumption is that tokens with low query attention at inner pruning layers can be discarded without sacrificing exact temporal evidence. That assumption is contradicted by the paper's own category-level data. The headline 'strongest average performance' and 'retaining 97.0%' should therefore be scoped to average benchmark accuracy; they do not establish that A-PACK preserves fine-grained temporal reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes A-PACK, a training-free two-stage token-compression framework for omni-modal LLMs. In Stage 1 (pre-LLM), audio tokens are preserved and the visual budget is allocated using a gated local-window CKA measure combined with coarse query relevance, followed by segment- and anchor-based redundancy removal. In Stage 2 (inner-LLM), starting from the middle decoder layer, low-query-attention audio and visual tokens are progressively pruned together with their KV-cache entries. The authors motivate the design with measurements showing higher per-token information density and representational diversity for audio than for video, and with a claim that local audio-visual co-variation is a better visual-selection cue than token-wise similarity. On Qwen2.5-Omni-7B/3B at matched 35% and 25% prefill-FLOPs tiers, A-PACK is reported to achieve the strongest average accuracy across AVUT, WorldSense, Video-MME, and DailyOmni among the evaluated training-free compressors, retaining 97.0% of full-token average accuracy at the 25% tier on the 7B model while cutting prefill FLOPs by about 78% and final token retention to 5.6%, and improving decoding throughput by up to 2.21x. A separate AVHBench analysis reports improved hallucination robustness for the gated-CKA alignment score.","tokens_in":24576,"tokens_out":20568,"duration_ms":195467,"significance":"If the empirical claims survive scrutiny, the contribution is genuinely useful for efficient omni-modal inference: the central idea of deferring audio pruning until query-conditioned multimodal context exists, and of using local (windowed) audio-visual structure rather than pointwise similarity for visual allocation, is clearly differentiated from the surveyed pre-LLM compression methods. The paper's strengths include matched-FLOPs comparisons with a transparent analytic FLOPs model (Appendix G); evaluation at two backbone scales with four main benchmarks plus an AVHBench diagnostic; stage and alignment-scorer ablations; a paired exact McNemar test (p = 0.0117) for the local-alignment hypothesis; per-parameter sensitivity tables; honest reporting of a failure case (Figure S10); and a commitment to release code. The headline claims are, however, averages that hide a category-level failure on fine-grained temporal localization, and the evaluation protocol leaves room for benchmark-driven hyperparameter selection, so the magnitude of the reported advantage over baselines should be treated as provisional until those issues are addressed.","major_comments":[{"comment":"The stress-test concern lands: at the 25% prefill-FLOPs tier on Qwen2.5-Omni-7B, Table S7 shows that A-PACK's AVUT Event Location score drops to 34.7 from the Full-Tokens value of 38.2 (90.8% relative), and that this is below OmniZip (35.9), UniComp (38.8), and Random (38.8) on the same category; Figure S10 documents the same phenomenon, with A-PACK missing the second-level localization of a laugh despite retaining audio and aligned visual context. Because Section 1 motivates preserving audio before the LLM precisely to avoid losing brief, temporally localized audio evidence, this category-level degradation contradicts the load-bearing premise of deferred audio pruning for fine-grained temporal reasoning, and the 97.0% average in Table 1 obscures it. The headline claim should be scoped to average benchmark accuracy, category-level AVUT results should be reported in the main text, and the paper should discuss how the inner-LLM pruning or the pre-LLM allocation could protect short audio transitions.","section":"Section 4.2, Table 1; Table S7; Figure S10"},{"comment":"The central comparison in Table 1 is run with hyperparameters selected on the evaluation benchmarks themselves: Section 4.5 states that for each tier the pre-LLM retention ratio is chosen as the highest value within the FLOPs budget from a retention sweep (Figure S4c), and Table S1 lists fifteen free parameters (w, tau, beta, gamma, tau_sim, sigma, DA-FPS k and power, dedup coverage, anchor priority, R_pre, P, L_mid, lambda) whose defaults are used in the main table. Table S5 shows that several of these parameters have sizable effects (temporal window w=5 gives -3.52 points on a WorldSense subset; continuity threshold tau_sim=0.99 gives -2.51 points), so the reported gains are not obviously robust to reasonable parameter variation. Because the central claim is that A-PACK is the strongest average compressor at matched FLOPs, the evaluation should either lock a single configuration through a validation-based selection protocol and report test results once, or show that the ranking over baselines is preserved under a fixed default configuration; at minimum, the selection protocol should be disclosed in the main text.","section":"Section 4.1; Section 4.5; Table S1; Table S5"},{"comment":"The pre-LLM frame scoring is specified inconsistently and incompletely: Eq. (2) defines s_f as either c_f or the blend (1-beta)c_f + beta(1-d_hat_f), whereas Algorithm 1 defines the base visual score as b_f = gamma*u_f + (1-gamma)*c_f and then sets s_f from it, and the surrounding text only says that frame-level allocation combines query relevance with the local alignment score without giving the combination. Moreover, the frame-query relevance r_f used in Eq. (3) for segment scoring is never defined, and the balance parameter lambda between peak and sustained relevance in Eq. (3) is not assigned a value in Table S1. As written, the method cannot be reimplemented unambiguously, which is a precondition for verifying the efficiency-accuracy claims in Table 1.","section":"Section 3.1, Eq. (2), Eq. (3); Algorithm 1"}],"minor_comments":[{"comment":"The sentence 'A-PACK starts inner-LLM pruning at layers 14/18 for 7B/3B using a 10%/15% per-layer drop in the 35%/25% tiers' ambiguously mixes backbone-dependent and tier-dependent settings; Table S1 clarifies that L_mid = depth/2 (14 for 7B, 18 for 3B) while P is 10% in the 35% tier and 15% in the 25% tier, so the main text should spell out this mapping.","section":"Section 4.1, Implementation Details"},{"comment":"The reported final token retentions (16.8% at the 35% tier and 5.6% at the 25% tier) should be reconciled with the stated pre-LLM retentions (R_pre = 0.50/0.35) and per-layer pruning (P = 10%/15%); for example, 0.35 x 0.85^k = 0.056 implies roughly 11 effective pruning steps, so stating the number of layers at which pruning is applied would remove an apparent inconsistency.","section":"Table 1, Final Ret. column"},{"comment":"The abstract says 'across four benchmarks,' while Section 4.1 lists five benchmarks because AVHBench also appears in Table 4 and in the supplementary material; the benchmark counting should be made consistent or explicitly qualified.","section":"Abstract; Section 4.1"},{"comment":"Please clarify whether the token set S includes text tokens and how the normalization in Eq. (5) over S relates to the decoder's softmax over all keys (including text), since the pruning scores are extracted from the attention computation.","section":"Section 3.2, Eq. (5)"},{"comment":"A one-sentence summary of the occlusion and effective-rank protocols should appear in the main text, because the '9.8x higher information density' and '1.7x higher effective rank' claims are used to justify the audio-preservation design.","section":"Section 4.5, Figure 2"},{"comment":"The statement that query-key attention scores require 'no additional forward pass' should note that standard fused-attention implementations do not expose attention weights and that extracting them may require a modified attention kernel.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for the journal and the claims are empirically testable; the decision hinges on whether the authors can stiffen the evaluation protocol (validation-based hyperparameter selection or a fixed-default robustness check) and surface the category-level Event Location degradation in the main text rather than only in the supplementary material. The honest failure-case reporting and the paired McNemar test are to the authors' credit. No circularity problem in the technical sense: the accuracy numbers are measured, not derived from the method's equations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for sharing this. The paper makes a real contribution: it frames omni-modal compression as stage- and modality-dependent, and the core idea—preserve audio before the LLM, use local-window CKA to allocate visual budget, then prune both modalities with query attention inside the LLM—is not in the cited prior work. The experiments are unusually thorough for this area: matched prefill-FLOPs tiers, two backbone scales, four benchmarks, stage ablations, a McNemar test, and an honest failure case. The efficiency numbers are credible: up to 78% prefill FLOPs reduction, 2.3GB memory savings, and a 2.21x decoding throughput improvement while holding roughly 97% of average accuracy. That is a meaningful result for deployed omni-modal systems.\n\nThe soft spot is real and worth naming. The 97.0% average conceals a category-level drop on AVUT's Event Location at the 25% tier: 34.7 versus 38.2 full, and below OmniZip (35.9), DyCoke (42.4), UniComp (38.8), and even Random (38.8). This is the exact capability the method's motivation claims to protect—brief, temporally localized audio events. Figure S10 shows the same miss on second-level laugh localization. The conclusion concedes aggressive compression may affect fine-grained evidence, but that concession sits oddly with the headline 'retains 97.0%' claim. The strong average is real, but it should be scoped as average benchmark accuracy, not robustness to fine-grained temporal reasoning.\n\nTwo smaller issues: hyperparameters are selected on the same benchmarks used for evaluation, and code is not released despite 'will be released upon acceptance.' Also Eq. (3) leaves the segment balance parameter λ unreported; Table S1 lists the other controls. These are fixable, but they matter for reproducibility.\n\nWho should read it: anyone working on token compression for multimodal LLMs. It deserves a serious referee. My recommendation: send it to review, but ask the authors to report category-level results prominently and either release code or provide a stronger reproducibility appendix.","headline":"Solid two-stage compression paper with real efficiency gains, but the headline average masks a category-level temporal-localization failure that needs honest scoping.","tokens_in":25142,"tokens_out":2884,"would_cite":true,"duration_ms":28071,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that deferring audio pruning until after query-conditioned fusion lets an omni-modal LLM cut 78% of prefill FLOPs while keeping 97% of full-token accuracy.","keywords":["omni-modal LLM","token compression","deferred pruning","KV-cache pruning","audio-visual alignment","CKA","efficient inference","prefill FLOPs"],"falsifier":"Run A-PACK on an audio-visual benchmark of questions that require second- or sub-second temporal localization, such as the Event-Location subset of AVUT restricted to answer intervals under one second, and compare accuracy against the Full-Tokens model; a disproportionate accuracy drop on these short-interval questions would show that post-fusion pruning discards brief evidence and the central claim would need qualification.","tokens_in":24018,"feed_emoji":"🎧","tokens_out":10046,"duration_ms":89587,"temperature":0.7,"pith_summary":"Omni-modal language models that listen and watch at the same time pay for long audio-video sequences during prefill and in the KV cache. This paper claims that the right compression schedule changes as the input crosses the LLM boundary: before fusion, audio tokens carry denser, harder-to-replace evidence than video tokens, so audio should be preserved while video is thinned using short-window audio-visual co-variation; after the query has fused with both modalities, low-relevance audio and visual tokens can be pruned together. The proposed two-stage method, A-PACK, realizes this schedule and, on Qwen2.5-Omni-7B/3B across AVUT, WorldSense, DailyOmni, and Video-MME, keeps 97.0% of full-token average accuracy at the 25% prefill-FLOPs tier while cutting prefill FLOPs by about 78%, reducing final token retention to 5.6%, and raising decoding throughput by up to 2.21 times. If correct, this makes omni-modal compression a stage- and modality-dependent decision rather than a single pre-LLM token-selection problem.","feed_headline":"Deferred audio pruning keeps 97% accuracy at 78% less prefill","feed_subtitle":"A-PACK thins video first, then prunes low-relevance audio and video inside the LLM, lifting decoding throughput 2.21x","key_machinery":"The load-bearing machinery is a two-stage schedule with a gated local-CKA alignment score as its named component; CKA (Centered Kernel Alignment) is a covariance-structure similarity measure between two sets of representations. Before the LLM, A-PACK preserves all audio tokens, segments the video, and allocates the visual budget using query relevance combined with a short-window linear CKA score computed over paired, mean-centered frame-level embeddings in a length-3 window; when the clip-wide CKA falls below a threshold, the score is blended with a normalized frame-audio distance to catch temporally shifted correspondences. Within each segment, the first frame becomes the anchor and is sampled with density-aware farthest-point sampling, while non-anchor frames keep only the tokens least similar to the anchor, removing within-scene redundancy. Inside the LLM, starting at the middle decoder layer, each retained audio and visual token is scored by scaled query-key attention with the last query token, the top $(1-P)$ fraction is kept, and the corresponding KV-cache entries are deleted at each pruning layer; this progressive schedule shortens attention computation and the cache for later layers and for autoregressive decoding.","core_discovery":"The central discovery is that audio and video should be compressed on different schedules around the LLM boundary. Audio tokens show roughly 9.8 times higher per-token task relevance under occlusion and 1.7 times higher effective rank (a measure of how many independent directions a token set uses) than video tokens, so early audio pruning loses compact, locally timed evidence that visual tokens cannot replace. The paper further finds that local audio-visual dynamics, measured as windowed linear CKA over time-aligned frames, are a better cue for allocating visual tokens than pointwise audio-video matching, which can wrongly connect temporally distant events. After the LLM has built query-conditioned context, progressively pruning tokens by last-query attention and deleting their KV-cache entries yields the advertised efficiency: at matched 35% and 25% prefill-FLOPs tiers, A-PACK achieves the strongest average accuracy among the training-free compressors tested, with 97.0% of full-token accuracy at the aggressive tier.","pith_inferences":["The same 'defer pruning until the query conditions the representation' principle could generalize to other high-information, low-token-rate input streams beyond audio, such as event-level metadata or depth, and to other encoder-decoder architectures.","The 5.6% final retention suggests that, once cross-modal context is formed, most tokens are redundant for typical questions; a natural extension is to learn per-layer retention schedules that depend on query difficulty rather than fixed per-layer drop rates.","The missed second-level laugh localization implies a testable fix: protect brief, high-salience audio transitions with a query-adaptive retention floor in the inner-LLM stage, or add a finer-grained allocation within retained intervals.","The audio-visual asymmetry (higher per-token importance and diversity for audio) could be measured per task and used as a general scheduling rule: modally asymmetric compression budgets may outperform symmetric ones in other omni-modal settings."],"forward_implications":["At matched prefill-FLOPs tiers, A-PACK has the highest average accuracy among the evaluated training-free compressors on both Qwen2.5-Omni-7B and 3B, so deferring pruning until after query-conditioned fusion is a better operating point than early pruning.","The inner-LLM pruning stage shortens the KV cache, which is what produces the decoding-throughput gain (up to 2.21 times) and lower GPU memory, not just the prefill savings; pre-LLM-only methods cannot achieve this.","Keeping audio under a fixed token budget improves accuracy on both audio-centric AVUT and the more visual WorldSense, so audio acts as guidance rather than as competition for the visual budget.","Local-window alignment raises accuracy in all eight WorldSense domains relative to static pointwise matching, with a significant McNemar test (p=0.0117), supporting temporally local audio-visual dynamics as a selection signal.","Because the method is training-free, it can be applied to existing omni-modal backbones without fine-tuning, and its per-layer pruning schedule can be tuned to a target prefill-FLOPs budget."],"supporting_citations":[{"why":"Defines linear CKA, the covariance-structure similarity used to score local audio-visual alignment.","marker":"(Kornblith et al. 2019)"},{"why":"Establishes that short co-occurring audio-visual intervals carry event-localization signal, motivating windowed rather than pointwise matching.","marker":"(Tian et al. 2018; Mahmud and Marculescu 2023)"},{"why":"Provides OmniZip, the strongest baseline that prunes audio before the LLM and must be beaten to show deferred pruning helps.","marker":"(Tao et al. 2026)"},{"why":"Supplies the segment-importance allocation idea that A-PACK adapts for distributing visual budgets across scenes.","marker":"(Devnani et al. 2026)"},{"why":"Provides density-aware farthest-point sampling used to select diverse anchor tokens and remove within-video redundancy.","marker":"(Climaco and Garcke 2025)"},{"why":"Defines the Qwen2.5-Omni backbone on which A-PACK and all compared methods are evaluated.","marker":"(Xu et al. 2025)"},{"why":"Supplies FastV, the inner-LLM visual-pruning baseline whose attention-based pruning A-PACK extends to audio and KV-cache deletion.","marker":"(Chen et al. 2024)"},{"why":"Provides AVUT, the audio-centric benchmark whose Event-Location questions expose the method's temporal-localization limits.","marker":"(Yang et al. 2025)"}],"fun_headline_variants":["Defer audio, prune video first: 78% less prefill","Audio delayed, video pruned early: 2.21x throughput","Local AV dynamics beat token matching for pruning","A-PACK thins video first, then audio inside LLM","Prune audio late, video early: 97% accuracy kept"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that tokens with low query attention after fusion can be discarded without losing answer-critical evidence, including brief audio events and exact temporal cues; the paper's own failure case, a missed second-level laugh localization, shows this premise can fail under aggressive compression.","fun_headline_variants_meta":{"raw":{"variants":["Defer audio, prune video first: 78% less prefill","Audio delayed, video pruned early: 2.21x throughput","Local AV dynamics beat token matching for pruning","A-PACK thins video first, then audio inside LLM","Prune audio late, video early: 97% accuracy kept"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1628,"prompt_tokens":956,"completion_tokens":672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":572,"tokens_out":672,"duration_ms":7338,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:24:38.041063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run A-PACK on an audio-visual benchmark of questions that require second- or sub-second temporal localization, such as the Event-Location subset of AVUT restricted to answer intervals under one second, and compare accuracy against the Full-Tokens model; a disproportionate accuracy drop on these short-interval questions would show that post-fusion pruning discards brief evidence and the central claim would need qualification.","supporting_citations":[{"cited_title":"CoSeLECT : Adaptive Frame Selection for Video-Language Understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the segment-importance allocation idea that A-PACK adapts for distributing visual budgets across scenes."}],"review_version":1}