{"id":"b5964b3e-12d6-40f9-8730-b28c2d60f5e2","arxiv_id":"2411.15024","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A training-free two-stage token compression method speeds up video language model inference by about 1.5x and cuts memory use, while keeping accuracy close to the uncompressed model.","lead":"Video AI models represent each frame as many small pieces of visual data, and DyCoke makes them faster by merging redundant pieces across frames and pruning unimportant pieces while the model generates an answer. It requires no retraining and reports about 1.5x faster inference with lower memory use while keeping accuracy roughly level.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DP refresh threshold and single-layer attention proxy are unspecified and unvalidated; if layer-3 importance is unrepresentative or refresh mistimed, the claimed accuracy parity is not reproducible.","rationale":"I agree with the reader's weakest_assumption and would add that the refresh condition is not just uncalibrated but unspecified: no threshold, no formula, no ablation. The single-layer proxy is partially tested (L=3 vs 10) but only in aggregate accuracy; that test could pass even if the top-p% sets differ, because accuracy is a coarse measure. The central claim that DyCoke improves performance while compressing tokens would fail if the proxy is unrepresentative on other benchmarks or if the DP cache never restores important tokens. The concrete experiment described would settle both issues. I do not see a more fundamental flaw: the efficiency numbers are measured, the TTM and DP ablations show the components matter, and the method is genuinely training-free. The reader's CONDITIONAL verdict is appropriate; my concern reinforces it rather than changing it.","tokens_in":19285,"tokens_out":11015,"duration_ms":99834,"concrete_test":"On LLaVA-OV-7B with MVBench (32 frames), for a set of decoding examples, compute the top-30% visual token index set at layer 3 and at every other transformer layer; report the mean Jaccard overlap. Then sweep the refresh cosine threshold over {0.5, 0.7, 0.9, 0.99, never} with L fixed at 3 and report MVBench accuracy for each. If the mean overlap is below 0.7, or if the accuracy range across thresholds exceeds the reported 0.75-point improvement over the full-token baseline, the dynamic proxy and refresh trigger are not robust enough to support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DyCoke's accuracy-parity claim rests on the dynamic pruning mechanism in Sec. 3.2: the top-p% visual KV indices computed at layer L=3 (Eq. 5-6) are applied to all layers, and the KV/DP cache swap occurs only when the cosine similarity between successive attention distributions at layer L is 'low'. Two load-bearing conditions are unquantified: (1) whether layer-3 attention importance predicts other layers—the paper only reports L=3 vs L=10 in Tab. 5 and never measures per-layer overlap; (2) the exact refresh threshold, which determines how often pruned tokens can re-enter the active cache. If the threshold is too high, updates are rare and tokens that become important mid-generation remain in the DP cache, so all layers attend without them; if too low, the repeated full-attention recomputation erodes the speedup. Because the abstract claims 'still improving the performance,' a setting where the refresh is mistimed could flip the 0.75-point MVBench gain into a loss, and the method is not reproducible without the threshold value.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes DyCoke, a training-free two-stage token compression method for video large language models. In the first stage (TTM), visually redundant tokens across neighboring frames are merged during prefilling on the basis of cosine similarity inside a sliding window. In the second stage, the KV cache is dynamically pruned during decoding: at a chosen transformer layer L, attention scores of the current token over visual tokens are computed, the top-p% indices are kept in the active KV cache, and the remaining tokens are stored in a DP cache; the active set is refreshed when the attention distribution changes substantially. The method is evaluated on LLaVA-OneVision 0.5B, 7B, and 72B across ActivityNet-QA, NextQA, PerceptionTest, VideoDetailCaption, VideoMME, and MVBench, reporting up to about 1.5x latency speedup and 1.4x memory reduction relative to full tokens. The authors position the method as improving accuracy while compressing tokens, and compare against FastV and LLaVA-PruMerge.","tokens_in":19479,"tokens_out":5663,"duration_ms":50221,"significance":"If the central claims hold, DyCoke is a practically useful plug-in: it requires no training, uses the frozen model's own attention signals, and the speedup and memory numbers are measured end-to-end on multiple model scales. The paper includes useful ablations (w/o DP, random pruning) that isolate the contribution of the dynamic mechanism, and the Fig. 2 observation that attention targets shift during decoding is well motivated. The main reservations are that the headline 'still improving the performance' is not consistent across the reported benchmarks, the DP refresh trigger is not specified, and the single-layer attention proxy is not validated against other layers; these issues matter because the accuracy-parity claim rests on them.","major_comments":[{"comment":"The condition that triggers a KV/DP cache swap is described only as 'a low similarity' between successive attention distributions, with no numerical threshold or update schedule given anywhere in the paper. Since the DP mechanism is the central novelty and the paper claims that it preserves accuracy, this missing value prevents reproduction of the results in Tables 1-3 and 5 and leaves the regime where updates are too rare or too frequent unspecified. Please report the exact cosine-similarity threshold and how often the swap occurs in practice, and add a sensitivity analysis around the threshold.","section":"Sec. 3.2, Eqs. (5)-(7)"},{"comment":"The importance index set I_p^(L) computed at layer L (default L=3) is used to prune the KV cache of every layer via Eq. (6), but the manuscript never checks whether attention at layer 3 is representative of other layers. Table 5 only compares aggregate scores for L=3 and L=10, not the overlap of the selected token sets. Please measure per-layer agreement between I_p^(3) and I_p^(l) (for example, Jaccard similarity of the selected index sets across decoding steps), or justify the choice of L with such data.","section":"Sec. 3.2, Eq. (5)"},{"comment":"The claim that DyCoke 'still improves the performance' is only true for selected configurations and benchmarks. For example, in Table 1 on LLaVA-OV-7B the K=0.3 setting loses on ActNet-QA (51.80 vs 51.93) and VideoDC (3.19 vs 3.30) relative to full tokens; in Table 2 the 7B MVBench average at K=0.7 is 57.5 vs 58.0 for full tokens; and in Table 5 increasing P to 0.9 drops VideoDC to 2.86 from the full-token 3.30. The abstract and introduction should be qualified to 'competitive or slightly better on several benchmarks, with large efficiency gains,' and the exact configurations used for the headline should be identified.","section":"Abstract, Tables 1-3"},{"comment":"The hyperparameters K, L, and P appear to be selected on the evaluation benchmarks themselves; no validation split or selection procedure is described. Given the sensitivity shown in Table 5 (for example, K=0.9, L=0, P=0.9 collapses ActivityNet accuracy to 40.21), the reported gains may reflect favorable tuning. Please clarify how these values were chosen and provide results for a fixed setting across all benchmarks, or perform selection on a small validation set.","section":"Sec. 4.1 and Table 5"}],"minor_comments":[{"comment":"The sentence 'As shown in Tab. 2' should refer to Table 1, since the VideoDC results appear in Table 1, not Table 2.","section":"Sec. 4.2, Video Description"},{"comment":"References [2] and [3] are the same paper (Chen et al., arXiv:2403.06764); please merge them or cite distinct works.","section":"References [2] and [3]"},{"comment":"The formal update equations for the DP cache swap after the initial construction in Eq. (7) are missing; the prose describes re-adding tokens, but an exact update rule or pseudocode would improve reproducibility.","section":"Sec. 3.2, Eq. (7)"},{"comment":"The model names alternate between 'LLaVA-OV-0.5B' and 'LLaVA-OneVision-0.5B'; please standardize the notation.","section":"Notation throughout"},{"comment":"Figure 2 reports attention averaged over all layers, while the method uses layer L only; the discrepancy should be explained because it bears on the single-layer proxy concern.","section":"Fig. 2 vs Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision or efficient-ML venue, and the measured efficiency gains are useful. The main editor-facing concerns are that the hyperparameters are seemingly tuned on test benchmarks and the abstract overstates the accuracy results; both can be fixed with a revision. The duplicate references [2] and [3] also suggest a rushed reference list that should be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result holds up: DyCoke gets a measured 1.4-1.54x speedup and memory savings on LLaVA-OneVision 0.5B/7B/72B with accuracy roughly matching full tokens, and the ablation shows the dynamic pruning (DP) component is doing real work -- dropping it costs several points. The observation that video token attention shifts across decoding iterations (Fig. 2) is well made and justifies the dynamic approach over one-shot pruning. The two-stage design (temporal merging plus decoding-stage KV cache pruning) is a sensible combination, and the latency/memory numbers look honest. This is a useful engineering contribution for an expensive inference regime.\n\nThe soft spots are real but not fatal. Most important: the refresh condition for the KV/DP cache swap is never quantified. The paper says the cache updates when the cosine similarity between successive attention distributions is 'low,' but no threshold is given. That is a reproducibility gap, because the refresh frequency directly sets the trade-off between speed and accuracy. The single-layer attention proxy (L=3, applied to all layers) is less worrying than the stress-test suggests: the ablation shows L=10 gives nearly identical results, which indicates the proxy is fairly stable. Still, a per-layer overlap analysis or a broader L sweep would settle it. The third issue is the abstract's 'still improving the performance.' Looking at the tables, gains on VideoMME and PercepTest are offset by small drops on NextQA and VideoDC; the honest claim is 'comparable accuracy, sometimes slightly better.' Hyperparameters K, L, P also appear selected post hoc on the eval benchmarks, so a fixed configuration with a single reported error bar would be stronger. Finally, LazyLLM and Look-m are cited but not compared empirically; since they are the closest dynamic-KV-cache alternatives, adding them would make the contribution clearer.\n\nWho is this for? Practitioners deploying VLLMs who want a training-free speedup without significant accuracy loss. It is not a conceptual breakthrough, but it is a solid, measured efficiency result. The paper deserves peer review -- it is coherent, the ablations are informative, and the main missing piece (the threshold) is fixable with a small amount of additional reporting. I would send it out with a request to specify the refresh threshold, temper the performance claim, and ideally add one or two of the missing baselines.","headline":"Credible training-free speedup with a real dynamic-pruning idea, but the refresh threshold is unspecified and the 'improving performance' claim overreaches.","tokens_in":20054,"tokens_out":3574,"would_cite":true,"duration_ms":34544,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token pruning that adapts per step speeds video LLMs 1.5x","keywords":["video large language models","token compression","dynamic KV cache pruning","attention sparsity","inference acceleration","temporal redundancy","training-free compression"],"falsifier":"Compute, on a sample of videos, the overlap between the top-p% visual token indices selected by layer 3 and those selected by each other layer at the same decoding step; if the average overlap is substantially below the p% retention rate (or below 50% relative overlap), the single-layer selector cannot be reliably representing all layers.","tokens_in":19058,"feed_emoji":"⚡","tokens_out":4627,"duration_ms":39319,"temperature":0.7,"pith_summary":"DyCoke is a training-free, plug-and-play method for making video large language models faster and lighter. Its central claim is that, in video inputs, the set of visual tokens the model needs changes from one decoding step to the next, so token pruning must be dynamic rather than one-shot. DyCoke combines a first stage that merges visually similar tokens across adjacent frames with a second stage that prunes and reactivates KV-cache entries based on current attention scores. Across video QA and description benchmarks on three model sizes, it reports up to 1.5x inference speedup and 1.4x memory reduction while matching or improving accuracy.","feed_headline":"Token pruning that adapts per step speeds video LLMs 1.5x","feed_subtitle":"DyCoke merges redundant frames and prunes KV cache during decoding, keeping accuracy while cutting memory by 1.4x.","key_machinery":"Two components carry the argument. The temporal token merging (TTM) module, applied during prefilling, groups frames into windows, merges similar tokens across even/odd frame groups using cosine similarity, and keeps full tokens in the first frame of each window, cutting visual tokens by 50-60%. The dynamic KV cache pruning module keeps a small active KV cache selected by attention scores at layer L, stores pruned tokens in a DP cache, and refreshes the active set only when the cosine similarity between successive attention distributions falls below a threshold, so tokens that become important can re-enter during later decoding steps.","core_discovery":"The paper's key empirical observation is that attention in a video LLM shifts across decoding iterations: early frames may dominate at one step and later frames at another. On this basis it argues that one-shot pruning methods, which select important tokens once during prefilling, will drop tokens that later decoding steps need. DyCoke therefore maintains a small active KV cache and a separate DP cache; at each step it uses attention scores at one layer to keep the top-p% visual tokens, and it refreshes the active set only when the attention distribution changes enough. This dynamic selection is what lets the method cut tokens to roughly 15 per frame while preserving or improving output quality.","pith_inferences":["The single-layer attention proxy (layer 3) could be replaced by a lightweight aggregator across layers, which might make the refresh trigger more reliable.","The cosine-similarity refresh threshold is a free parameter; an adaptive threshold based on attention entropy could make the method more robust to videos with abrupt scene changes.","Combining DyCoke with quantization or distillation could push video LLMs toward mobile deployment, since the method already cuts memory and compute independently of those techniques.","The temporal merging stage operates uniformly across frames; making the merge ratio depend on local motion (e.g., merging more in static scenes) could prevent the minor information loss noted in the paper's discussion of rapid scene changes."],"forward_implications":["Video LLMs can run at 1.5x speed with 1.4x less memory at equal or better accuracy, without any fine-tuning or model modification.","The same dynamic pruning mechanism could be applied to any long-context multimodal LLM where attention targets shift over time, not just video.","Because compression removes redundant tokens, models can accept more video frames (e.g., 32 instead of 16) under a fixed compute budget, which improves performance on short and medium-length videos.","The DP-cache design means pruned tokens are not destroyed, so mistakes in one pruning step are recoverable at the next refresh."],"supporting_citations":[{"why":"The LLaVA-OneVision models serve as the baseline VLLMs throughout all experiments.","marker":"[18]"},{"why":"FastV is the one-shot pruning baseline that DyCoke is compared against and whose fixed pruning limitation motivates dynamic pruning.","marker":"[3]"},{"why":"LLaVA-PruMerge is the other one-shot pruning baseline used for comparison.","marker":"[39]"},{"why":"MVBench is the main video reasoning benchmark used for both accuracy and efficiency comparisons.","marker":"[23]"},{"why":"LazyLLM applies attention-based dynamic token pruning in LLMs and is the closest prior dynamic-pruning idea.","marker":"[9]"},{"why":"ToMe provides the similarity-based token merging approach that TTM draws on.","marker":"[1]"},{"why":"LMMs-Eval is the evaluation harness used to score several video QA benchmarks.","marker":"[17]"}],"fun_headline_variants":["Step-wise token pruning makes video LLMs 1.5x faster","Dynamic KV cache trims video tokens for 1.5x speedup","Training-free token compression speeds video LLMs 1.5x","Per-step token selection cuts compute and memory for video LLMs","Video LLM speedup via adaptive token merging and pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pruning choices are driven by attention scores from a single transformer layer (layer 3) at one decoding step, and the method assumes that layer's view of token importance matches every other layer's needs.","fun_headline_variants_meta":{"raw":{"variants":["Step-wise token pruning makes video LLMs 1.5x faster","Dynamic KV cache trims video tokens for 1.5x speedup","Training-free token compression speeds video LLMs 1.5x","Per-step token selection cuts compute and memory for video LLMs","Video LLM speedup via adaptive token merging and pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000448,"raw_usage":{"total_tokens":2222,"prompt_tokens":867,"completion_tokens":1355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1264}},"tokens_in":483,"tokens_out":1355,"duration_ms":9488,"temperature":1.0,"reasoning_tokens":1264,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:36:12.553139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on a sample of videos, the overlap between the top-p% visual token indices selected by layer 3 and those selected by each other layer at the same decoding step; if the average overlap is substantially below the p% retention rate (or below 50% relative overlap), the single-layer selector cannot be reliably representing all layers.","supporting_citations":[{"cited_title":"Mvbench: A comprehensive multi-modal video understand- ing benchmark","cited_arxiv_id":null,"evidence_quote":"MVBench is the main video reasoning benchmark used for both accuracy and efficiency comparisons."}],"review_version":1}