{"id":"40321483-274a-4ead-986a-a3b6b8b85ee6","arxiv_id":"2506.14107","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A learned per-token reuse mechanism plus GPU-friendly memory and compute compaction accelerates ViT-based video embedding generation by up to 2.64x while keeping end-task accuracy within 2% of the original model.","lead":"This paper presents Déjà Vu, a system that speeds up video-language models by reusing computations between similar consecutive frames instead of recomputing everything. It reports up to 2.64x faster video embedding generation with under 2% accuracy loss, which matters for running video search and question answering over large video collections.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that Eq. 1 input-token cosine similarity predicts QKV/FFN output reuse safety; this is only validated on three in-distribution benchmarks, so the 2% accuracy bound may not transfer to other video content.","rationale":"The reader identified the same load-bearing assumption: input-space similarity as a predictor of output-space reuse safety. I agree this is the most critical premise for the central claim. If it fails, the learned gate can select tokens whose QKV/FFN outputs are wrong, and the restoration MLP is too small to fully correct the resulting residuals. The paper's measured speedups and end-task accuracies on MSR-VTT, How2QA, and NExT-GQA are real evidence, and the released code plus GPU measurements give the work independent support. The ablation study in Figure 14 also shows the decision and restoration components contribute to the accuracy-throughput tradeoff, which is encouraging. What is missing is evidence that the learned similarity-to-safety mapping transfers beyond the three evaluated datasets. The comparison-specific concern about interpolated CMC and Eventful Transformer throughputs in Section 7.3 is also real, but it affects the relative SOTA claim more than the absolute 2.64x-within-2% claim, so I treat it as secondary. Since the reader's verdict was already CONDITIONAL and my concern does not move it, I recommend no change to the verdict. The concrete cross-dataset instrumentation check above would settle whether the concern actually lands.","tokens_in":23124,"tokens_out":7660,"duration_ms":78514,"concrete_test":"Using the released code, take the trained ReuseViT checkpoint that produces the 2.64x QA speedup and run it on a held-out video-language dataset with different content and dynamics, such as MSVD-QA or ActivityNet-QA, at the same target reuse rate. Instrument each encoder layer to log for every token (i) the Eq. 1 similarity s_i, (ii) the hard reuse decision M_i, and (iii) the per-token relative L2 error between the reused QKV/FFN output after restoration and the full recomputation. Then measure end-task accuracy versus the unmodified ViT. If a nontrivial fraction (e.g., over 5%) of tokens with s_i above the learned decision threshold have post-restoration output error above 10%, or if end-task accuracy drops more than 2%, the reuse gate's core assumption fails outside the training distribution and the headline error bound should be scoped accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ReuseViT decides to reuse QKV/FFN computations for token i primarily from the cosine similarity between current and reference input tokens (Eq. 1), then corrects only with a small restoration MLP (Eqs. 8-9) of hidden size 128 and about 4% overhead. The paper does not establish that input-space cosine similarity tracks output-space error through the nonlinear QKV projection and FFN across video content. The only evidence is end-task accuracy on three benchmarks (MSR-VTT, How2QA, NExT-GQA), and Section 8 explicitly leaves broader task generalization open. The '2% error bound' is an operating point obtained by choosing the target reuse rate R_target in Eq. 15, not a property guaranteed by the architecture or loss. If the learned gate misorders tokens on videos with fast occlusion, camera motion, or domain shift, reused outputs will be inaccurate and end-task accuracy loss can exceed 2%. No per-token correlation analysis between Eq. 1 similarity and post-restoration output error is reported, so the core premise is currently asserted rather than demonstrated. This does not invalidate the three reported measurements on their own datasets, but it makes the headline claim conditional on the three evaluated distributions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Déjà Vu, a video-language query engine that accelerates ViT embedding generation by reusing QKV and FFN computations across consecutive frames. It introduces ReuseViT, which learns per-token reuse decisions through a decision layer, calibrates reused values with a restoration layer, and is trained with a Gumbel-Softmax relaxation and a loss combining cosine similarity to the original output with a target reuse rate. System-level contributions include layer-wise scheduling, cached memory compaction, and stream compaction to convert FLOPs savings into GPU throughput. Evaluations on three VideoLM tasks—MSR-VTT retrieval with CLIP4Clip, How2QA QA with FrozenBiLM, and NExT-GQA grounding with TempCLIP—report throughput improvements up to 1.81x, 2.64x, and 2.54x within a claimed 2% error bound, with comparisons to CMC, Eventful Transformer, and DiffRate.","tokens_in":23226,"tokens_out":8023,"duration_ms":83101,"significance":"If the results hold, Déjà Vu addresses a real bottleneck in video-language analytics by co-designing a learned reuse mechanism with GPU-oriented compaction techniques. The experiments are performed on a commodity RTX 3090 with standard datasets, and the paper explicitly separates FLOPs savings from achieved throughput, which is a methodological strength. The ablation study in Section 7.6 cleanly isolates the contributions of gating, sparse compaction, and memory compaction. The artifact is publicly available, and the training keeps the ViT backbone frozen, which simplifies deployment. However, the comparative advantage over state-of-the-art baselines is weakened by interpolated baseline throughputs, and some evaluation metrics are partly circular or not reported in exact numeric form. The central measurements for Déjà Vu itself are credible, but the claims as currently stated need stronger support or more careful scoping.","major_comments":[{"comment":"The throughput speedups attributed to CMC and Eventful Transformer are not measured: the paper states that 'we interpolate their throughput assuming our compaction techniques applied.' These interpolated numbers are then used in the abstract and Section 1 to conclude that Déjà Vu outperforms the state of the art (1.81x/2.64x/2.54x versus 1.32x/2.08x/2.20x). Because the authors' compaction techniques may interact differently with CMC's and Eventful's reuse patterns, this assumption is load-bearing and not validated. I ask that the baselines be implemented and measured with the same compaction stack, or that the throughput comparison be explicitly labeled as an estimate with the FLOPs/accuracy comparison presented as the primary evidence.","section":"Section 7.3, Figures 10(d)-(f)"},{"comment":"The headline 'within 2% error bound' is not substantiated by an explicit accuracy table. The text reports throughput numbers but not the exact end-task accuracies of the unmodified model and of each Déjà Vu configuration at the operating points used in Figures 10(a)-(f). Since the operating point is chosen through R_target in Eq. 15, the reader needs the actual accuracy drops (e.g., R@5 for MSR-VTT, multiple-choice accuracy for How2QA, GQA@Accuracy for NExT-GQA) to verify the 2% claim. Please add a table reporting these values together with the corresponding R_target for each reported point.","section":"Abstract, Section 7"},{"comment":"The embedding-quality axis in Figure 14 is cosine similarity to the original output, which is exactly the objective optimized through L_sim in Eq. 13. Consequently, the ablation conclusions drawn from Figure 14 partly reflect the model's success in optimizing its training loss rather than an independent measure of quality. To make the design-choice comparisons load-bearing, I request that the same configurations be evaluated with end-task accuracy or another held-out metric, or that the figure be repositioned as reporting satisfaction of the training objective.","section":"Section 7.7, Figure 14"},{"comment":"The reuse criterion rests on the premise that input-space cosine similarity and the difference ΔR_i predict whether QKV/FFN outputs can be safely reused or restored. The paper does not report any per-token correlation analysis between these signals and the actual post-restoration output error, and the evaluation is limited to three in-distribution benchmarks. Because the 2% error bound is an operating point selected through R_target, not an architectural guarantee, I would like to see a per-token correlation plot or a deliberate domain-shift experiment (e.g., fast camera motion or occlusion) to support transferability. If such evidence is unavailable, the abstract and introduction should explicitly scope the claim to the three evaluated tasks.","section":"Section 3.3 (Eq. 1), Section 4.2 (Eq. 13), Section 8"},{"comment":"Several hyperparameters that determine the reported tradeoff are not reported: α and R_target in Eq. 15, the Gumbel-Softmax temperature schedule, the I-frame reset period (Section 6.3 mentions 'every twentieth frame' as an example), and the six-frame grouping pattern from Section 4.3. Without these values, it is difficult to reproduce the operating points or to assess how much the 2% error bound depends on hyperparameter tuning. Please include a reproducibility table with the exact values used in the evaluation.","section":"Sections 4.2, 4.3, 6.3"}],"minor_comments":[{"comment":"The notation 'GumbelSoftmax(MLP_decision(v))' is ambiguous for binary decisions; if a two-class softmax is intended, the paper should specify how the two logits map to M_soft, or describe the binary concrete distribution formulation.","section":"Section 4.1, Eq. (11)"},{"comment":"The training grouping uses six frames in the pattern 1-5-9-13-11-12, whereas online inference collects 'four consecutive frames' per segment; please clarify the relationship between the training grouping and the inference grouping, and whether the periodic I-frame reset every 20 frames is consistent with the trained segment structure.","section":"Sections 4.3 and 6.3"},{"comment":"The sentence 'During training, Only the two lightweight modules are trained' has a capitalization error, and the paper does not specify the GPU or wall-clock time used for training beyond the statement that convergence typically occurs within an hour.","section":"Section 6.2"},{"comment":"The notation 'M_i ∈ 0,1' should be 'M_i ∈ {0,1}', and the sign convention for the decision-layer output d_i should be stated more explicitly.","section":"Section 3.3, Eq. (4)"},{"comment":"For the DiffRate baseline, the paper states 'we adapted the policy for VLP models' but does not describe the adaptation; please provide details or a pointer to the adapted implementation in the artifact.","section":"Section 7.1"},{"comment":"The statement that 'only configurations that yield an actual improvement are shown' could hide unfavorable operating points; please specify how many configurations were evaluated and how many are omitted.","section":"Section 7.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a data-systems venue and the artifact is a strength. The main risk is the comparison protocol: the headline 'beats state of the art' claim relies on interpolated baseline throughputs rather than measurements, and the '2% error bound' is not supported by an explicit accuracy table. These issues are fixable either by additional experiments or by reframing the claims, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if the measured numbers hold, this is a real systems contribution. The paper takes the inter-frame reuse idea (already present in CMC and Eventful Transformer) and makes it learned and GPU-visible, with measured 1.81x/2.64x/2.54x embedding-generation speedups on retrieval/QA/grounding within 2% end-task accuracy loss. That is the thing to know.\n\nThe genuinely new part is the combination: a learned decision layer with multiple cues, a restoration MLP that corrects reused tokens from input-space differences, I/P/B-style frame reordering, and three compactions—layer-wise scheduling, cache compaction, and sparse stream compaction—that convert FLOP savings into throughput. The ablations support the attribution. Existing building blocks are cited honestly, and the paper states its own limitations (no attention-layer reuse, broader generalization open). End-task accuracy is measured on external benchmarks, so the headline result is not circular. That is solid.\n\nThe soft spots are real but not fatal. First, the SOTA comparison in the abstract and intro presents interpolated baseline throughputs without the caveat. Section 7.3 does disclose the interpolation, but the headline numbers inherit it. The interpolation is not obviously unfair—if their compaction techniques transfer, it might even be generous—but it should be flagged wherever those numbers appear. Second, there are no error bars or multi-seed runs, and the 2% bound is an operating point set by R_target, not a guarantee of the architecture. That is acceptable if said plainly; right now it reads like a property. Third, the core premise—that input-space cosine similarity predicts output reuse safety through QKV/FFN—is supported by end-task accuracy on three standard benchmarks and by the learned gate's extra cues, but not by any direct per-token correlation analysis. The stress-test concern about out-of-distribution video is fair. I would want one dynamic/domain-shift test or a per-token error analysis before believing the bound generalizes beyond those datasets. Minor: Figure 14's embedding-quality axis uses cosine similarity, which matches the training loss, so that ablation is partly circular; and the codec-metadata feature is described but never ablated.\n\nWho it is for: systems people doing video analytics or efficient ViT inference, and anyone comparing learned versus manual reuse policies. It deserves a serious referee; the right outcome is a revision that cleans up the baseline presentation and adds a generalization check or confidence intervals.\n\nSend it to peer review.","headline":"A real systems result: learned inter-frame ViT reuse plus GPU compaction gives measured 1.8-2.6x speedups within 2% accuracy on three VideoLM tasks, with interpolated baseline throughputs and limited generalization evidence as the main caveats.","tokens_in":24007,"tokens_out":2832,"would_cite":true,"duration_ms":31652,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned policy reuses ViT work across similar video frames, giving up to 2.64x faster embedding generation within a 2% error bound.","keywords":["video-language models","vision transformer acceleration","inter-frame computation reuse","learned gating","Gumbel-Softmax training","GPU stream compaction","embedding generation","video query engine"],"falsifier":"Run ReuseViT on videos with rapid camera motion, frequent scene cuts, or heavy occlusion — content unlike MSR-VTT, How2QA, and NExT-GQA — and compare end-task accuracy against full recomputation: if clips whose patch-level cosine similarity is high still show embedding or task-accuracy error beyond the 2% bound, the input-similarity premise fails.","tokens_in":22719,"feed_emoji":"🎬","tokens_out":6238,"duration_ms":55357,"temperature":0.7,"pith_summary":"This paper tries to establish that a video-language query engine can generate the visual embeddings consumed by retrieval, question answering, and grounding models much faster by reusing computations across consecutive frames, while keeping error within 2%. The engine, Déjà Vu, combines a learned ViT variant called ReuseViT with system-level compaction so that the reduced FLOPs actually turn into GPU throughput. On three tasks, the paper reports embedding-generation speedups of 1.81x, 2.64x, and 2.54x over full computation, beating the inter-frame-reuse baselines CMC and Eventful Transformer and the image-based DiffRate at matched accuracy. If the claim holds, large-scale video analytics with video-language models becomes substantially cheaper and more practical.","feed_headline":"Reused video computations speed up video-language queries 2.64x","feed_subtitle":"A learned ViT reuses per-token work across similar frames, keeping task accuracy within 2 percent.","key_machinery":"The key machinery is the pair of lightweight learned modules inside ReuseViT: a two-layer decision MLP that maps per-token cues (cosine similarity to reference frames, class-token attention, reference type, codec metadata) to a binary reuse mask, and a two-layer restoration MLP (hidden size 128) that calibrates the reused QKV/FFN outputs by adding a correction learned from the token difference. Training uses Gumbel-Softmax soft gating to allow gradients through the discrete decisions, a target-reuse-rate loss, and grouped-frame training so the model learns to tolerate error accumulation. On the system side, layer-wise scheduling across frames lets Déjà Vu free cached activations layer by layer (cached memory compaction) and gather active tokens from multiple frames into dense matrices (sparse computation compaction), which is what converts FLOP reductions into measured throughput.","core_discovery":"On the paper's own terms, the central discovery is that inter-frame computation reuse in ViT-based video-language models can be learned rather than hand-configured, and that the resulting savings can be made real on GPUs. ReuseViT reuses the QKV projection and feed-forward outputs of tokens whose patch-level cosine similarity to the corresponding patch in a past or future reference frame passes a learned gate; a small restoration MLP corrects the residual difference between current and reference tokens. The decision layer consumes cosine similarity, class-token attention, reference-frame type, and codec metadata, and is trained with a Gumbel-Softmax relaxation, a reuse-rate target, and grouped-frame losses that model error accumulation. The paper reports that this configuration reaches the highest accuracy-throughput tradeoff among CMC, Eventful Transformer, and DiffRate, with up to 2.64x embedding-generation speedup within a 2% error bound.","pith_inferences":["The reuse criterion is input-space cosine similarity; a natural stress test is footage with camera motion or scene cuts where patches remain similar but higher-level content changes, since the paper's evaluation covers three curated video corpora.","Because attention layers are excluded from reuse, the speedups should shrink as token counts rise (e.g., 336px or 518px ViTs), where attention becomes a larger FLOP share; extending reuse to attention or sparse-attention kernels would be the next lever.","The compaction techniques are described independently of ReuseViT's learned gating, so they could plausibly be bolted onto any sparse ViT accelerator, not just the one evaluated here.","The 2% error bound is defined per task accuracy, not per embedding; users who need exact embeddings or who serve adversarial queries would need a fallback path that recomputes."],"forward_implications":["Embedding generation for retrieval, video QA, and video grounding can be accelerated 1.81x, 2.64x, and 2.54x, respectively, while keeping end-task accuracy within 2%.","Learnable reuse decisions beat fixed-rate and fixed-threshold reuse policies: Déjà Vu reaches higher accuracy at the same throughput than Eventful Transformer and CMC, and higher throughput than DiffRate at matched accuracy.","Layer-wise scheduling with cached-memory compaction and sparse-computation compaction is what turns FLOP savings into GPU speedups; without them, hard gating alone yields only 1.25x on the QA task.","Periodic full I-frame recomputation (roughly every 20 frames) bounds long-sequence error accumulation with less than 5% overhead.","Only the small decision and restoration modules need training; the pretrained ViT backbone stays frozen, so deployment does not require re-tuning or storing large model weights."],"supporting_citations":[{"why":"Defines the Vision Transformer architecture whose QKV projection and feed-forward layers ReuseViT accelerates.","marker":"[25]"},{"why":"Provides the CLIP visual backbone whose embeddings the three evaluated VideoLMs consume.","marker":"[77]"},{"why":"CMC is the fixed-threshold inter-frame reuse baseline that Déjà Vu compares against on FLOPs, throughput, and accuracy.","marker":"[86]"},{"why":"Eventful Transformer is the fixed-rate inter-frame reuse baseline, used for FLOPs, throughput, and adaptability comparisons.","marker":"[26]"},{"why":"DiffRate is the image-based token pruning/merging baseline adapted for comparison.","marker":"[14]"},{"why":"FlexGen supplies the layer-by-layer batching idea that Déjà Vu adapts for layer-wise scheduling.","marker":"[84]"},{"why":"Stream compaction is the primitive underlying sparse computation compaction when gathering active tokens into dense matrices.","marker":"[7]"},{"why":"CLIP4Clip is the video retrieval model on which the first speedup is measured.","marker":"[64]"},{"why":"FrozenBiLM on How2QA is the video question answering setup where the 2.64x speedup is reported.","marker":"[111]"},{"why":"TempCLIP on NExT-GQA is the video grounding setup where the 2.54x speedup is reported.","marker":"[104]"}],"fun_headline_variants":["ReuseViT learns to skip redundant frames, cutting video query cost","Video-language queries get 2.64x faster via learned frame reuse","Inter-frame reuse yields 2.64x speedup for video-language models","Learn which video frames to reuse: 2.64x faster embeddings","Efficient video query engine reuses computations across frames"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a token whose patch looks similar to the same patch in a reference frame will also have similar QKV and feed-forward outputs, so reusing those computed values (plus a small learned correction) keeps the final embedding within the promised accuracy bound.","fun_headline_variants_meta":{"raw":{"variants":["ReuseViT learns to skip redundant frames, cutting video query cost","Video-language queries get 2.64x faster via learned frame reuse","Inter-frame reuse yields 2.64x speedup for video-language models","Learn which video frames to reuse: 2.64x faster embeddings","Efficient video query engine reuses computations across frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1452,"prompt_tokens":973,"completion_tokens":479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":589,"tokens_out":479,"duration_ms":4171,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:20:08.655429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ReuseViT on videos with rapid camera motion, frequent scene cuts, or heavy occlusion — content unlike MSR-VTT, How2QA, and NExT-GQA — and compare end-task accuracy against full recomputation: if clips whose patch-level cosine similarity is high still show embedding or task-accuracy error beyond the 2% bound, the input-similarity premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CMC is the fixed-threshold inter-frame reuse baseline that Déjà Vu compares against on FLOPs, throughput, and accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FlexGen supplies the layer-by-layer batching idea that Déjà Vu adapts for layer-wise scheduling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP4Clip is the video retrieval model on which the first speedup is measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TempCLIP on NExT-GQA is the video grounding setup where the 2.54x speedup is reported."}],"review_version":1}