{"id":"28ce4b12-0a67-486a-a94d-52bd68fa6185","arxiv_id":"2508.00367","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Representation shift, the L2 change in token embeddings through an MLP layer, is introduced as a training-free token importance metric compatible with FlashAttention.","lead":"This paper proposes representation shift, a token importance score that measures how much each token changes through a network layer, and uses it to prune tokens without needing attention maps. The method makes token compression compatible with FlashAttention and other architectures, yielding large throughput gains on video and image tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Representation shift's importance-metric superiority is not independently validated: design choices are selected on evaluation benchmarks, and main speedup comparisons confound the metric with the FlashAttention kernel.","rationale":"The reader's weakest assumption targets the transferability of the representation-shift heuristic and the selection of design choices on the evaluation benchmarks. I agree with that, but I want to sharpen it: the most load-bearing issue is that the paper never isolates the metric from the kernel. All end-to-end comparisons that claim 'superior speed-accuracy trade-offs' use FlashAttention for the proposed method and standard attention for the attention-based baselines. Since FlashAttention alone provides a 1.5x-2.7x speedup (Table 1), the headline 5.5x speedup is a combination of FlashAttention and token pruning, not a property of representation shift. A fair, unconfounded comparison would run the final MLP+L2 metric with standard attention, and attention-based pruning with FlashAttention (e.g., by computing class-token attention via a separate standard-attention pass). Absent that, the paper's central claim that representation shift is a better importance metric is under-supported. The reliability analysis (Table 8) and multi-dataset results are useful positive evidence, but they do not resolve the confound. The CNN fine-tuning also directly contradicts the 'training-free' attribute in the abstract, further weakening the model-agnostic claim. These issues are fixable with additional experiments, so a conditional verdict remains appropriate; I am not moving to reject because the compatibility contribution is real and the metric may well be adequate.","tokens_in":15435,"tokens_out":8851,"duration_ms":88986,"concrete_test":"On ImageNet-1K with DeiT-S, compare three configurations at identical pruning ratios (20% at layers 1, 4, 7): (a) class-token attention pruning (Equation 2) with standard attention; (b) representation-shift (MLP, L2) pruning with standard attention; (c) representation-shift pruning with FlashAttention. If top-1 accuracy of (b) is not clearly above (a), the metric's claimed advantage over attention scores is not established independently of FlashAttention, implying the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that representation shift (L2 distance of token embeddings before/after the MLP) is a valid token-importance metric that is training-free, model-agnostic, and superior to attention-based scores. The evidence for this is not independent: the choice of the MLP block and L2 distance (Section 3.3, Figure 5) is made on the same benchmarks (ImageNet/DeiT-S and MSRVTT/UMT-B) later used as main results (Tables 2 and 5), so the reported gains are partially selection artifacts. Moreover, all main comparisons of 'Ours' versus attention-based pruning confound the importance metric with the FlashAttention kernel: 'Attn' runs standard attention while 'Ours' runs FlashAttention. Because FlashAttention is exact, the accuracy comparison at matched FLOPs is valid, but the throughput comparison is not attributable to the metric. No ablation isolates the final MLP+L2 metric under standard attention against class-token attention pruning under the same kernel. Without such an ablation, the paper does not establish that representation shift is superior to attention-based importance; it only shows that it is FlashAttention-compatible. The CNN experiments further violate the 'training-free' claim (Section 4.2 fine-tunes ResNet for 100 epochs), so the model-agnostic generalization is overstated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Representation Shift, a token-importance metric defined as the L2 distance between a token's representation before and after a network layer (defaulting to the MLP block in Transformers). The authors claim this metric is training-free, model-agnostic, and independent of attention maps, making it compatible with FlashAttention and applicable to CNNs and SSMs. They evaluate token pruning with this metric on video-text retrieval (UMT), video QA, ImageNet classification with DeiT, ResNet, and ViM, reporting large throughput gains and competitive accuracy.","tokens_in":15730,"tokens_out":5033,"duration_ms":45312,"significance":"If the claims are supported, the contribution is practically valuable: a simple importance score that works without attention maps would enable training-free token pruning on FlashAttention-accelerated models and on non-transformer architectures. The paper includes a useful reliability check (Table 8) and provides code. However, the experimental protocol currently prevents a clean attribution of the reported gains to the proposed metric, so the significance is conditional on additional ablations.","major_comments":[{"comment":"The operation choice (MLP) and distance metric (L2) are selected by ablations on ImageNet/DeiT-S and MSRVTT/UMT-B, which are the same benchmarks used for the main results in Tables 2 and 5. Consequently, the reported improvements over attention-based scoring are partially selection artifacts and the 'model-agnostic' claim is not independently validated. Please either fix the configuration a priori, evaluate on held-out benchmarks, or report the selection procedure as model selection with appropriate caveats.","section":"Section 3.3, Figure 5"},{"comment":"The throughput comparison between 'Attn' and 'Ours' conflates the importance metric with the attention kernel, since 'Attn' uses standard attention while 'Ours' uses FlashAttention. Table 1 shows that FlashAttention alone already provides 1.5x (DeiT-S) and 2.7x (UMT-B) throughput gains, so the additional speedups in the main tables cannot be attributed to representation shift. Please report an ablation with representation-shift pruning under standard attention (or attention-based pruning with materialized attention under FlashAttention) to compare the metrics at matched kernels and FLOPs.","section":"Tables 2, 4, 5"},{"comment":"The authors state that after token compression in CNNs 'we finetune the model for 100 epochs, including 10 cooldown epochs' to adapt to the changed resolution. This contradicts the paper's central claim of a 'training-free' method (abstract and Section 1). Please clarify the scope of the training-free claim, or provide CNN results that do not require finetuning.","section":"Section 4.2, ResNet experiments"},{"comment":"On UMT-B, the proposed method is not consistently better than attention-based pruning; for MSVD the R@1 drops from 60.3 (Attn) to 57.7 (Ours) and for ActivityNet from 54.2 to 50.3. The text claims 'competitive or even better performance' but this is not supported on these benchmarks. Please discuss these failure cases and present per-dataset results honestly, or restrict the claim to the settings where the advantage holds.","section":"Table 2"}],"minor_comments":[{"comment":"The phrase 'speedups of up to 5.5% and 4.4%' should read '5.5x and 4.4x' (the Introduction and Tables report multiplicative speedups).","section":"Abstract"},{"comment":"The harmonic mean of V2T and T2V results is mentioned but the individual V2T/T2V scores are not shown; please clarify how the reported R@1 values are aggregated.","section":"Section 4.1"},{"comment":"The comparison is described as 'vid-TLDR +Ours', but it is unclear whether the baseline vid-TLDR runs with standard attention and what the '+Ours' exactly replaces; please specify the kernel and importance-score settings for both columns.","section":"Table 3 caption"},{"comment":"The claimed 'negligible overhead' of computing representation shift is not quantified; please include a timing or FLOP breakdown of the scoring step.","section":"Section 4.2"},{"comment":"The ViM experiment covers only one model and dataset; the conclusion that representation shift generalizes to SSMs would benefit from additional models or tasks.","section":"Table 7"},{"comment":"Reference [50] is cited for line-wise pruning in CNNs, but that work is for Transformers without retraining; the difference in protocol (finetuning) should be acknowledged.","section":"References, [50]"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the core idea is simple and potentially useful. The main weaknesses are experimental attribution issues rather than fatal flaws in the method; with matched-kernel ablations and honest scope statements, the paper could be a solid contribution. I would not reject on these grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Representation shift is a simple, cheap token-importance heuristic: measure the L2 change in each token embedding through the MLP, prune the smallest. It is genuinely FlashAttention-compatible because it never needs attention maps, and the authors show it works across DeiT, UMT, and even ViM. The reliability check in Table 8—top 50% tokens beat bottom 50% by 26 points—is a good sanity test. The code is available, and the method is easy to reproduce.\n\nThe problems are in the evaluation. The main comparisons against attention-based pruning are confounded: 'Attn' runs standard attention, 'Ours' runs FlashAttention. FlashAttention is exact, so the accuracy comparison at matched FLOPs is not automatically invalid, but the large throughput gains are mostly the kernel, not the metric. There is no ablation that runs the final MLP+L2 metric against class-token attention pruning under the same attention implementation, so the claim that representation shift is a better importance score is not cleanly established. Second, the design choices—MLP over attention, L2 over L1/cosine—are made on the same benchmarks (ImageNet/DeiT-S, MSRVTT/UMT-B) that later appear in the main tables. That is selection on the test set; the reported gains are partly artifacts of that choice. Third, the CNN experiments fine-tune ResNet for 100 epochs, which directly contradicts the 'training-free' claim in the abstract. The abstraction should be scoped to transformers, or the CNN section should be reframed.\n\nI would not block the idea. It is a plausible heuristic, and the authors are honest that it is an empirical observation rather than a derived result. But the paper overclaims. If they add a kernel-controlled ablation, validate the design choices on held-out data, and fix the training-free language, this becomes a solid practical contribution. As it stands, I would send it to review with the expectation of major revision. The audience is people working on efficient inference for vision and video transformers; they will find the metric worth trying, but should not read the current comparisons as proof that it beats attention-based scores.","headline":"A simple, plausible training-free token-importance heuristic that is FlashAttention-compatible, but the evaluation overclaims: throughput gains are mostly kernel, the metric is tuned on test benchmarks, and 'training-free' breaks for CNNs.","tokens_in":16212,"tokens_out":3769,"would_cite":false,"duration_ms":36810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Representation Shift, a training-free token-importance metric that measures L2 change through a layer, making it compatible with FlashAttention and generalizable beyond transformers.","keywords":["token compression","token pruning","FlashAttention","representation shift","vision transformer","video-text retrieval","training-free","model-agnostic"],"falsifier":"Prune tokens by representation shift and by random choice at the same ratio on a held-out set of tasks; if random pruning matches or beats shift-based pruning in accuracy at the same speed, the claimed importance signal is not real. A sharper check is to correlate shift rankings with per-token leave-one-out accuracy impact across layers and show the correlation is consistently positive.","tokens_in":15273,"feed_emoji":"⚡","tokens_out":11009,"duration_ms":95982,"temperature":0.7,"pith_summary":"Token pruning usually needs an attention map to decide which tokens to keep, but FlashAttention deliberately never materializes that map. This paper proposes Representation Shift, a training-free importance score that measures how far each token's representation moves (L2 distance) through a layer, and uses it to drop the least-changed tokens. If correct, the score makes token compression compatible with FlashAttention and with any architecture, since it does not depend on attention at all. The reported payoff is up to 5.5x throughput on UMT video-text retrieval and around 4x on video QA, with small accuracy losses, plus working extensions to ResNets and state space models.","feed_headline":"Measure token shifts, drop the smallest, run 5.5x faster","feed_subtitle":"A training-free importance score replaces attention maps, letting FlashAttention do token pruning on video transformers.","key_machinery":"The central object is the per-token scalar \\(s_i = \\|\\mathrm{MLP}(\\mathrm{LN}(x'_i)) - x'_i\\|_2\\), the L2 distance between a token's representation before and after the MLP (multilayer perceptron) sublayer. It is a token importance score that requires only the MLP input and output, so it sidesteps the attention map entirely; the pruning decision then removes the tokens with the smallest shifts at selected layers.","core_discovery":"The central claim is that the magnitude of a token's representation change across a layer is a reliable proxy for its informativeness. Formally, for input tokens \\(x\\) and a layer transformation \\(F\\), the importance score is \\(s=\\$\\Delta$ x=D(F(x),x)\\), with \\(D\\) chosen as the L2 norm and \\(F\\) as the MLP sublayer after ablation. The hypothesis is that networks amplify critical tokens (foreground content, task-relevant patches) while barely updating redundant ones; the paper supports this with pruning experiments, top/bottom-50% retention tests, and visualizations. Because the score needs only the layer's input and output, it is available under FlashAttention, which never exposes attention maps, and it transfers to CNNs and state space models by measuring the shift at the corresponding stage.","pith_inferences":["Beyond the paper: because the metric makes no architectural assumptions, it should apply to decoder-only language models running FlashAttention, where training-free token pruning is currently blocked by the same attention-map unavailability; testing it on long-context LLMs is a natural next step.","Beyond the paper: the authors' own ablations show that the best operation (MLP) and distance (L2) are empirical choices, so the heuristic may need per-architecture tuning rather than being universally automatic.","Beyond the paper: the same shift signal could serve as a post-hoc saliency map for interpretability, since the visualizations suggest it highlights foreground objects; this could be tested against human-annotated segmentation.","Beyond the paper: the speedups are GPU-throughput numbers, and end-to-end latency on other hardware (CPU, mobile, or different GPUs) may not scale the same way, so deploy-target measurements would be needed to confirm the gains."],"forward_implications":["Token pruning no longer needs attention maps, so it can be stacked on FlashAttention; the reported result is up to 5.5x throughput on UMT-L video-text retrieval and about 4x on video QA with modest accuracy loss.","On ImageNet classification with DeiT-T/S/B, representation-shift pruning beats attention-based class-token scoring at the same pruning ratio.","The criterion transfers to non-attention architectures: line-wise and token-wise pruning of ResNet and token pruning of Vision Mamba (ViM-T) preserve accuracy while raising throughput.","Existing training-free token merging methods such as vid-TLDR can swap their attention-based importance scores for representation shift and then run under FlashAttention."],"supporting_citations":[{"why":"supplies the FlashAttention kernel whose lack of attention maps motivates the new metric.","marker":"[16]"},{"why":"provides the UMT video transformer on which the main speedup and accuracy results are measured.","marker":"[32]"},{"why":"provides the DeiT image transformer used for validation, ablations, and ImageNet experiments.","marker":"[52]"},{"why":"is the training-free token-merging method whose attention-based importance metric is replaced by representation shift in the extensibility study.","marker":"[13]"},{"why":"defines the class-token attention scoring baseline used as a comparison in image classification.","marker":"[33]"},{"why":"is the ResNet CNN used to demonstrate the metric works outside transformers.","marker":"[21]"},{"why":"is the Vision Mamba state space model used to show the metric transfers beyond attention.","marker":"[77]"},{"why":"supplies the token-pruning settings for state space models that the ViM experiment follows.","marker":"[74]"}],"fun_headline_variants":["Training-free token pruning for FlashAttention via representation shift","Prune tokens by representation shift, not attention maps — FlashAttention works","Representation shift: a token’s change reveals its worth — prune with FlashAttention","Drop tokens that barely change; keep FlashAttention speed with representation shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a token's L2 representation change through a layer is a valid measure of its importance; if that correlation fails for some layer, model, or task, the pruning signal no longer identifies what to keep.","fun_headline_variants_meta":{"raw":{"variants":["Training-free token pruning for FlashAttention via representation shift","Prune tokens by representation shift, not attention maps — FlashAttention works","Representation shift: a token’s change reveals its worth — prune with FlashAttention","Drop tokens that barely change; keep FlashAttention speed with representation shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001666,"raw_usage":{"total_tokens":6598,"prompt_tokens":921,"completion_tokens":5677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":5600}},"tokens_in":537,"tokens_out":5677,"duration_ms":36078,"temperature":1.0,"reasoning_tokens":5600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:10:48.013202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Prune tokens by representation shift and by random choice at the same ratio on a held-out set of tasks; if random pruning matches or beats shift-based pruning in accuracy at the same speed, the claimed importance signal is not real. A sharper check is to correlate shift rankings with per-token leave-one-out accuracy impact across layers and show the correlation is consistently positive.","supporting_citations":[{"cited_title":"Training data-efficient image transformers & distillation through at- tention","cited_arxiv_id":null,"evidence_quote":"provides the DeiT image transformer used for validation, ablations, and ImageNet experiments."},{"cited_title":"Flashattention: Fast and memory-efficient exact attention with io-awareness","cited_arxiv_id":null,"evidence_quote":"supplies the FlashAttention kernel whose lack of attention maps motivates the new metric."},{"cited_title":"Unmasked teacher: Towards training-efficient video foundation models","cited_arxiv_id":null,"evidence_quote":"provides the UMT video transformer on which the main speedup and accuracy results are measured."},{"cited_title":"vid-tldr: Training free token merging for light-weight video transformer","cited_arxiv_id":null,"evidence_quote":"is the training-free token-merging method whose attention-based importance metric is replaced by representation shift in the extensibility study."},{"cited_title":"Not all patches are what you need: Expediting vision transformers via token reorganiza- tions","cited_arxiv_id":null,"evidence_quote":"defines the class-token attention scoring baseline used as a comparison in image classification."},{"cited_title":"Vision mamba: Efficient visual representation learning with bidirectional state space model","cited_arxiv_id":null,"evidence_quote":"is the Vision Mamba state space model used to show the metric transfers beyond attention."},{"cited_title":"Exploring token pruning in vision state space models","cited_arxiv_id":null,"evidence_quote":"supplies the token-pruning settings for state space models that the ViM experiment follows."}],"review_version":1}