{"id":"a90a048f-0dae-495e-9371-a6452cb9d4e0","arxiv_id":"2411.13626","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LITE, a lightweight MLP trained to mimic a gradient-based oracle, improves the GFLOPs/accuracy trade-off over random selection and prior efficient video transformers.","lead":"This paper studies why random token dropping is so hard to beat in efficient video transformers, and proposes LITE, a lightweight selector that learns to keep only the most useful tokens. If it holds up, it offers a simple efficiency gain for video recognition without retraining the backbone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LITE's 50%-token accuracy (69.91, Table 1) matches the predicted-label oracle (70.00, Table 4), not the true-label oracle (78.52) it is trained to imitate; the oracle-driven mechanism is unsupported.","rationale":"The reader's weakest assumption includes the oracle's learnability, which is exactly where the problem lies, so there is partial agreement. However, the reader did not identify the sharpest evidence: LITE's accuracy is essentially identical to the predicted-label oracle, a baseline the paper reports but never compares LITE against. This is more specific than 'the oracle might not be reliable'; it shows that whatever LITE learns is not the true-label oracle. The concern is load-bearing because the paper's framing, its Principle 3, and its generalization story all depend on LITE approximating the true-label oracle. If the proposed check confirms the predicted-label-oracle equivalence, the empirical GFLOPs/accuracy trade-off could still be valid, so a rejection is not warranted; the paper would need to reframe LITE as a learned label-free scorer and add the predicted-label oracle as a baseline before the mechanism claim is accepted. Thus the reader's conditional verdict is appropriate, with this additional condition made explicit.","tokens_in":13881,"tokens_out":9712,"duration_ms":112591,"concrete_test":"On the same SS-V2 split, evaluate VideoMAE-LITE50 and compute (i) Top-1 accuracy, (ii) per-video Spearman correlation between LITE's predicted scores and the true-label oracle scores, and (iii) the same correlation against the predicted-label oracle. Also add the predicted-label oracle curve (Table 4) to Fig. 2. If LITE matches the predicted-label oracle in accuracy and has high correlation with it but low correlation with the true-label oracle, the true-label oracle is not the source of LITE's gains and the paper must be reframed accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's stated mechanism (Sec 4.2) is that the LITE MLP learns to reproduce the true-label Grad-CAM oracle (Eqs. 1-2). The paper's own numbers contradict this. On SS-V2 at P-Ratio 0.5, the true-label oracle reaches 78.52 Top-1 (Table 4), the predicted-label oracle reaches 70.00, and VideoMAE-LITE50 reaches 69.91 (Table 1). LITE is thus 8.6 points below its training target and statistically at the level of a label-free oracle. The 8-point advantage of the true-label oracle comes from privileged label information unavailable at inference, so an input-only selector cannot be expected to recover it; the paper never reports rank correlation or top-K overlap between LITE scores and oracle scores, and it omits the predicted-label oracle from the main Pareto plots. Sec 4.4 concedes the oracle is 'extremely hard to predict,' but the central narrative still credits LITE's gains to oracle approximation. The efficiency numbers may survive, but the mechanism and the transferability argument built on it are not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the value of visual tokens in video transformers, proposes five qualitative principles about token importance, and introduces LITE, a lightweight MLP selector trained to imitate a Grad-CAM-based oracle that scores tokens using true-label gradients. At inference, LITE keeps the highest-scoring tokens before the transformer blocks; LITE++ further adapts the token budget per video using a confidence estimate. Experiments on Something-Something-V2 and Kinetics-400 compare LITE against random token selection, token merging, and prior token-selection methods, and additional experiments report zero-shot transfer of a K400-trained selector to UCF101, SS-V2, and AVA action detection. The central claim is that LITE achieves a better accuracy-versus-GFLOPs trade-off than existing baselines, including random dropping, and that the learned selector generalizes across datasets and tasks without retraining.","tokens_in":14126,"tokens_out":4444,"duration_ms":54633,"significance":"If the empirical trade-off and the generalization results hold, LITE is a simple and practical plug-and-play token selector for frozen video transformers, and the five principles provide useful organizing observations for future work on token reduction. The paper contains a broad set of comparisons on two standard datasets, a cross-backbone experiment, and a cross-task experiment, which are valuable even if the mechanism is only partially understood. The main limitations are that the reported efficiency omits the selector's own compute, the oracle-approximation mechanism is not supported by the paper's own numbers, and the zero-shot claim lacks an appropriate matched baseline. The contribution is therefore significant but conditional on resolving these measurement and conceptual issues.","major_comments":[{"comment":"The paper states that the LITE selector is trained to reproduce the true-label Grad-CAM oracle of Eqs. (1)-(2), but the reported numbers contradict this mechanism. On SS-V2 at P-Ratio 0.5, the true-label oracle reaches 78.52 Top-1, the predicted-label oracle reaches 70.00, and VideoMAE-LITE50 reaches 69.91. LITE is therefore 8.6 points below its training target and essentially at the level of a label-free oracle. Because the paper credits LITE's gains to oracle approximation and uses this to argue that important tokens transfer across domains, this is a load-bearing issue. Please report the rank correlation or top-K overlap between LITE scores and oracle scores, and add the predicted-label oracle to the main Pareto plots (Fig. 2 and Fig. 1), so the reader can see whether LITE is actually learning the true-label oracle or merely a label-free proxy.","section":"Sec. 4.2, Tables 1 and 4"},{"comment":"The reported GFLOPs for LITE do not appear to include the cost of the selector MLP or, for LITE++, the MoviNet confidence model. Since the central claim is an efficiency trade-off, the end-to-end compute including all added components must be reported; otherwise the GFLOPs savings may be overstated. In addition, the accuracy numbers are presented without error bars or multiple-seed results, and the claim that LITE forms the \"optimal Pareto front\" is a qualitative visual judgment. Please provide variance estimates and a quantitative statement of the accuracy loss at matched GFLOPs, including the selector overhead in the reported numbers.","section":"Sec. 5.1, Tables 1-2, Fig. 1"},{"comment":"The zero-shot generalization claim is not supported by the current experimental design. The VideoMAE baseline rows are described as trained and tested on the same dataset, whereas the LITE rows use a selector trained on Kinetics-400; there is no VideoMAE baseline trained on K400 and evaluated on UCF101/SS-V2/AVA without a selector, and no selector trained on the target dataset for comparison. Without these controls, the observed accuracy retention could be due to the backbone being shared or to dataset-specific training effects, rather than to the selector generalizing. Please add matched baselines (same backbone training setup, with and without a target-trained selector) to separate selector transfer from backbone behavior.","section":"Sec. 5.2, Table 3"},{"comment":"The claim that token values \"closely follow a Pareto distribution\" is supported only by a histogram and a visual resemblance. Since the five principles are presented as a main contribution, this should be quantified with a distribution fit, an estimated tail exponent, or at least a goodness-of-fit or error-bar analysis. The term \"Pareto\" is used loosely; a heavy-tailed distribution with many near-zero values could be many other families. Please provide quantitative evidence for the distributional claim, or soften the statement to a qualitative heavy-tail observation.","section":"Sec. 3, Principle 4 and Fig. 3"}],"minor_comments":[{"comment":"The normalization constant N in Eq. (1) is not defined; the summation ranges over t, h, w should be stated explicitly, along with the layer at which A_d_thw is taken.","section":"Eq. (1)"},{"comment":"The predicted-label oracle from Table 4 is a natural reference point for token selection and should appear in the main Pareto plot, not only in the analysis table.","section":"Sec. 5.1 and Fig. 2"},{"comment":"The adaptive-budget thresholds tau1 = 0.1 and tau2 = 0.5 and the hand-set reductions (30% / 20%) are not accompanied by a sensitivity analysis; a short ablation would clarify how robust LITE++ is to these choices.","section":"Sec. 4.3, Table 5"},{"comment":"For the TimeSformer experiment, please state whether the selector was trained on TimeSformer features or transferred from VideoMAE, and whether the oracle was recomputed for TimeSformer; this affects the interpretation of the cross-backbone result.","section":"Sec. 5.4, Table 6"},{"comment":"There are several typos and formatting issues, including \"cutting over of GFLOPs\" in the introduction, the missing space in \"ClassificationK400\" in Table 3, and inconsistent capitalization of \"Pareto\" and \"state-of-the-art\". Please proofread the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central efficiency result may be salvageable, but the paper's mechanism narrative is contradicted by its own oracle comparisons, and the efficiency and generalization claims need more careful measurement and controls. I recommend major revision rather than rejection because the empirical trade-off tables, if properly quantified, could still support a useful contribution after reframing and additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the paper delivers a genuinely simple token-selection method that beats the strong random baseline on the GFLOPs/accuracy trade-off for video transformers, and that part is credible. But the explanatory story—that the selector learns to reproduce a true-label Grad-CAM oracle—does not survive contact with the paper's own numbers. On SS-V2 at 50% tokens, the true-label oracle hits 78.52, the predicted-label oracle hits 70.00, and VideoMAE-LITE50 hits 69.91. LITE is effectively matching the label-free oracle, not the privileged one. The paper never reports rank correlation or top-K overlap between LITE scores and oracle scores, and it leaves the predicted-label oracle out of the main Pareto plots. Sec 4.4 even concedes the oracle is 'extremely hard to predict,' which is an honest acknowledgment but sits uneasily with the central narrative.\n\nWhat is actually good: the method is clean—a 3-layer MLP on patch embeddings, trained with BCE against oracle scores, inserted before the transformer blocks. The empirical gains over random, ToMe, STA, and others appear consistent across K400 and SS-V2. The zero-shot transfer to UCF101, SS-V2, and AVA is a nice bonus and suggests the learned token values are not dataset-specific. The 'random is a strong baseline' observation is worth taking seriously and is documented with multiple strategies.\n\nThe soft spots are real but mostly fixable. No error bars anywhere. The Pareto claim is qualitative—no fitted exponent or goodness-of-fit. The FLOPs of the selector itself never appear in the GFLOPs numbers, so the net savings are slightly overstated. The adaptive budget uses hand-set thresholds (tau1=0.1, tau2=0.5) and a separate MoviNet, which complicates the story. And the mechanism gap is the biggest issue: if LITE is not actually approximating the true-label oracle, the 'principles' section loses some of its force. None of this kills the efficiency result, but the paper should be honest about what LITE is doing.\n\nWho it is for: people working on efficient video transformers, token pruning, and anyone who wants a reminder that simple baselines are strong. A serious referee can help the authors tighten the claims and add the missing measurements. I'd encourage engaging with it—conditional acceptance, not rejection.","headline":"LITE's efficiency gains look real, but the paper's claim that it works by imitating a privileged-label oracle is undercut by its own tables.","tokens_in":14656,"tokens_out":2470,"would_cite":true,"duration_ms":25085,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a cheap MLP imitating a gradient oracle selects video tokens better than all existing methods, cutting compute roughly in half with little accuracy loss.","keywords":["video transformers","token selection","token merging","efficiency","action recognition","Grad-CAM","adaptive compute","visual tokens"],"falsifier":"Measure the rank correlation between LITE's predicted token scores and the Grad-CAM oracle's scores on held-out videos; a near-zero correlation while accuracy is maintained would show the selector is not actually learning the oracle, and at matched GFLOPs on a novel dataset a tie with random token selection would falsify the claimed generalization.","tokens_in":13704,"feed_emoji":"⚡","tokens_out":8902,"duration_ms":76508,"temperature":0.7,"pith_summary":"Most token-reduction methods for video transformers, the paper argues, do no better than randomly throwing away tokens—a puzzling finding it explains with five principles about the nature of visual tokens. Token values follow a Pareto distribution: a few tokens carry almost all the classification signal, and low-value tokens act as noise that can actually lower accuracy when included. The paper builds LITE, a three-layer MLP that learns to reproduce a gradient-based oracle's per-token scores from a frozen VideoMAE backbone, and shows that selecting the top tokens beats both random selection and state-of-the-art token merging and selection methods on the GFLOPs-versus-accuracy trade-off for Kinetics-400 and Something-Something-V2. A scorer trained on Kinetics-400 transfers to UCF101, Something-Something-V2, and detection on AVA without retraining, and adapting the token budget to per-video difficulty saves further compute. If correct, the work reframes efficient video understanding as a token-scoring problem rather than a visual-saliency problem.","feed_headline":"Small MLP beats random token dropping in video transformers","feed_subtitle":"Copies a gradient oracle to drop low-value tokens, halving compute with minimal accuracy loss.","key_machinery":"The load-bearing object is the oracle: for a target class $c$, the gradient of the pre-softmax score $y^c$ with respect to the backbone's last-block feature maps $A^d_{thw}$ is spatially averaged (Eq. 1) to give per-feature importance weights, and each token's score is the ReLU-weighted combination of its activations (Eq. 2), normalized to $[0,1]$. LITE is a three-layer MLP, applied token-wise to the patch embeddings, trained with binary cross-entropy to reproduce these oracle scores; at inference the frozen VideoMAE backbone processes only the top-budget fraction of tokens. An optional adaptive budget (LITE++) uses a fast MoviNet confidence estimate to assign fewer tokens to easy videos.","core_discovery":"The central claim is that a Grad-CAM-style gradient score, computed from the true class label, accurately measures how much each visual token contributes to a frozen video transformer's decision, and that this score distribution is strongly Pareto: most tokens carry almost no information while a handful are essential. Keeping only the high-value tokens can outperform the full model by up to 9% accuracy, so low-value tokens are not merely redundant but actively harmful. From this the paper distills five principles: random token sampling is a stronger baseline than most learned methods; good tokens do not coincide with attention, motion, or saliency cues; low-value tokens hurt; token values follow a Pareto distribution; and easy videos need fewer tokens. LITE operationalizes the principles with a three-layer MLP that predicts the oracle scores from patch embeddings, selecting the top tokens for the unchanged backbone; on Kinetics-400 and Something-Something-V2 it sits on the Pareto front of the accuracy-versus-GFLOPs trade-off. The paper further reports that a selector trained on Kinetics-400 alone transfers to UCF101, Something-Something-V2, and AVA action detection with minimal loss.","pith_inferences":["The paper leaves implicit that the same gradient oracle could be used to denoise training itself by dropping low-value tokens during fine-tuning, turning pruning from an inference trick into a regularizer.","A testable extension would be replacing the external MoviNet confidence estimator in LITE++ with the backbone's own softmax confidence, checking whether the adaptive budget can be made self-contained without losing savings.","Because the oracle uses the true label, its offline scores can be computed once per training video; a natural next step is distilling the selector entirely from these offline scores without any additional labels at inference time.","The paper's comparison suggests that future token-reduction papers should always report a random-token baseline at matched GFLOPs, since most published methods fail to beat it."],"forward_implications":["LITE cuts GFLOPs by more than half with under one percentage point of accuracy loss on both Kinetics-400 and Something-Something-V2, outperforming ToMe, STA, STTS, LookupViT, and ObjectViViT at comparable compute.","Keeping only the oracle's top tokens can beat the full model by up to 9% accuracy, which means many tokens are not merely redundant but actively hurt classification.","The Pareto-like distribution of token values explains why random token dropping is such a strong baseline: random sampling rarely removes the few tail tokens that carry the signal.","A LITE selector trained on Kinetics-400 alone transfers to Something-Something-V2, UCF101, and AVA action detection with minimal accuracy or mAP loss, indicating that token importance is consistent across domains.","The adaptive budget variant, LITE++, reduces computation by up to 34% at the cost of less than one percentage point of accuracy by giving easy videos fewer tokens."],"supporting_citations":[{"why":"Supplies the Grad-CAM gradient method used to define the oracle token scores.","marker":"[36]"},{"why":"Provides the frozen VideoMAE backbone that LITE selects tokens for.","marker":"[39]"},{"why":"The token-merging baseline that the paper shows random selection already matches or beats.","marker":"[4]"},{"why":"The semantic-aware temporal accumulation baseline compared at matched GFLOPs.","marker":"[11]"},{"why":"The differentiable Top-K token-selection method compared as state of the art.","marker":"[42]"},{"why":"The object-centric ViViT variant whose GFLOPs LITE undercuts at similar accuracy.","marker":"[53]"},{"why":"Supplies the lightweight MoviNet confidence estimator used for the adaptive budget (LITE++).","marker":"[22]"},{"why":"The compressed-token approach compared as a strong efficiency baseline.","marker":"[23]"},{"why":"The Kinetics-400 dataset used for the main classification experiments and for training the transferable selector.","marker":"[8]"},{"why":"The Something-Something-V2 dataset used for the main comparisons and for measuring cross-dataset transfer.","marker":"[20]"}],"fun_headline_variants":["Video tokens follow a Pareto law: most are harmful","Gradient oracle shows few video tokens matter, drop the rest","Small MLP predicts token value, halves video compute","Most video tokens are junk; keep the few that matter","LITE: Pareto-based token pruning beats full video model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that the gradient of the frozen backbone's true-class score is a faithful, transferable measure of a token's value, and that a small MLP trained on initial embeddings can reproduce that measure well enough to transfer across datasets and tasks.","fun_headline_variants_meta":{"raw":{"variants":["Video tokens follow a Pareto law: most are harmful","Gradient oracle shows few video tokens matter, drop the rest","Small MLP predicts token value, halves video compute","Most video tokens are junk; keep the few that matter","LITE: Pareto-based token pruning beats full video model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000821,"raw_usage":{"total_tokens":3605,"prompt_tokens":966,"completion_tokens":2639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2559}},"tokens_in":582,"tokens_out":2639,"duration_ms":19127,"temperature":1.0,"reasoning_tokens":2559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:32:17.715178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the rank correlation between LITE's predicted token scores and the Grad-CAM oracle's scores on held-out videos; a near-zero correlation while accuracy is maintained would show the selector is not actually learning the oracle, and at matched GFLOPs on a novel dataset a tie with random token selection would falsify the claimed generalization.","supporting_citations":[{"cited_title":"Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Ba- tra","cited_arxiv_id":null,"evidence_quote":"Supplies the Grad-CAM gradient method used to define the oracle token scores."},{"cited_title":"Token merging: Your ViT but faster","cited_arxiv_id":null,"evidence_quote":"The token-merging baseline that the paper shows random selection already matches or beats."},{"cited_title":"Prune spatio-temporal tokens by semantic-aware temporal accumulation","cited_arxiv_id":null,"evidence_quote":"The semantic-aware temporal accumulation baseline compared at matched GFLOPs."},{"cited_title":"Efficient video transformers with spatial- temporal token selection","cited_arxiv_id":null,"evidence_quote":"The differentiable Top-K token-selection method compared as state of the art."},{"cited_title":"How can objects help action recognition? 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2353–2362, 2023","cited_arxiv_id":null,"evidence_quote":"The object-centric ViViT variant whose GFLOPs LITE undercuts at similar accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the lightweight MoviNet confidence estimator used for the adaptive budget (LITE++)."},{"cited_title":"LookupViT: Compressing visual information to a limited number of tokens","cited_arxiv_id":"2407.12753","evidence_quote":"The compressed-token approach compared as a strong efficiency baseline."},{"cited_title":"Quo vadis, action recognition? a new model and the kinetics dataset","cited_arxiv_id":null,"evidence_quote":"The Kinetics-400 dataset used for the main classification experiments and for training the transferable selector."},{"cited_title":"something something","cited_arxiv_id":null,"evidence_quote":"The Something-Something-V2 dataset used for the main comparisons and for measuring cross-dataset transfer."}],"review_version":1}