{"id":"72757050-7a5f-40ee-8b73-356b7395f8d0","arxiv_id":"2412.00965","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Cropr uses removable auxiliary heads to learn task-relevant token pruning in ViTs, achieving 1.5-4x speedups with small accuracy drops across image classification, semantic segmentation, and object detection.","lead":"Vision transformers are slow because they process every image patch. This paper's Cropr method learns to throw away unimportant patches early, speeding up inference by 1.5 to 4 times while keeping accuracy nearly unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Segmentation inference collapses N queries into one score that the training objective never optimizes, weakening the dense-task ranking claim.","rationale":"The paper is well-executed: it evaluates four tasks, compares against strong baselines with matched fusion, reports throughput honestly in Appendix E, and includes ablations for design choices. My concern is not that Cropr fails empirically; the reported speedups and accuracy are credible. Rather, the asserted mechanism ('auxiliary heads learn to select tokens end-to-end based on task relevance') is not directly trained for the inference-time selection rule in the dense-task regime. For N=1 (classification, detection), softmax is monotonic in the raw score, so Top-K by a = Q K^T matches Top-K by the trained attention weights; the residual issue is only that the auxiliary head uses a weighted average of all tokens, not a hard pruned set. For N=h*w (segmentation), the additional sum-of-queries collapse (Eq. 7) is a separate heuristic with no training-time analogue. Because the paper's best-case headline is the 0.1 mIoU drop on ADE20k, and the margin over random pruning plus LLF is small in that setting, the load-bearing question is whether the collapsed score preserves the ranking the auxiliary heads learned. The proposed re-scoring test would settle this directly. If the collapse is faithful, the central claim stands; if not, the segmentation result should be attributed to LLF plus a heuristic score rather than to task-relevant learning by the auxiliary heads. This supports the reader's CONDITIONAL verdict without moving it.","tokens_in":19240,"tokens_out":10731,"duration_ms":107000,"concrete_test":"On the ADE20k segmentation setup, re-score validation tokens at inference using the training-time per-query softmax weights summed over queries, s_i = sum_n softmax(Q_n . K_i / sqrt(D)), instead of the collapsed a_i = q . K_i, while keeping Top-K and LLF identical, and report median mIoU across the same 5 seeds. If the result differs from 56.6 by more than 0.5 mIoU, the inference-time query collapse (Eq. 7) is not faithful to what the auxiliary heads were trained to produce, and the 0.1-drop claim should be re-evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that auxiliary heads learn to select tokens based on task relevance and can be removed at inference. The scorer is trained through the aggregator, which consumes softmax(Q_n K^T / sqrt(D)) X and feeds an auxiliary head (Sec. 3.1, Eq. 5). For segmentation, N = h*w (Sec. 3.2), so each query learns per-patch attention patterns. At inference, the aggregator and heads are discarded, and scores are computed as a = q K^T with q = sum_n Q_n (Sec. 3.4, Eqs. 6-7). This sum-of-queries collapse has no training-time counterpart: no loss ever optimizes the collapsed score for Top-K selection, the Top-K operation is non-differentiable, and the stop-gradient (Sec. 3.1) prevents the main task loss from shaping the router. Thus the ranking used at inference is only an indirect proxy for what the auxiliary heads were trained to produce. The risk is highest in semantic segmentation, where the paper reports its headline 0.1 mIoU drop at 2x speedup (Fig. 5), yet the margin over random pruning plus LLF is small. If the collapsed score is not aligned with the per-query softmax importance, the dense-task results are not evidence that the auxiliary heads learned task-relevant selection; they would instead reflect the strength of LLF plus a heuristic score.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Cropr, a token-pruning module for Vision Transformers that scores tokens with a cross-attention router, keeps the top-K tokens, and uses an aggregator plus auxiliary task head to train the router. At inference the aggregator and auxiliary heads are removed, and the learned queries collapse into a single vector, leaving a linear scorer plus a Top-K selector. Last Layer Fusion (LLF) reinserts pruned tokens before the final transformer block to support dense tasks. The method is evaluated on ImageNet classification with MAE and EVA-02 backbones, ADE20k semantic segmentation, and COCO object detection and instance segmentation, with comparisons to random pruning, variance pruning, attention Top-K, and several prior token-pruning and token-merging methods. Reported speedups range from 1.5x to 4.1x with moderate accuracy drops, including a headline 0.1 median mIoU drop at 2.0x speedup on ADE20k.","tokens_in":19591,"tokens_out":11247,"duration_ms":115081,"significance":"If the results are reliable, Cropr is a useful practical contribution: it is among the few token-pruning methods demonstrated across classification, segmentation, and detection, and its inference-time overhead is very small. The paper benefits from a broad set of baselines, a clean ablation study of module design, fusion strategies and pruning schedules, and an honest discussion of limitations in App. B. The algebraic reduction in Eqs. (6)-(7) is correct and makes the inference-time collapse transparent. The main unresolved risks are the statistical support for the headline segmentation result and the alignment between the training-time attention-based objective and the inference-time collapsed score.","major_comments":[{"comment":"The central segmentation claim of a 0.1 median mIoU drop at 2.0x speedup is reported only as a median over 5 seeds, with no error bars, per-seed points, or significance test. Since all pruning baselines also benefit from LLF and the differences among methods are at the level of one mIoU point, the 'negligible performance penalty' in the abstract is not yet statistically supported. Please report the full distribution (e.g., box plot or mean plus standard deviation) and, if feasible, a paired test against the no-pruning and random-pruning baselines.","section":"Sec. 4.2 / Fig. 5"},{"comment":"For semantic segmentation the inference-time scorer is the sum of raw attention logits over all queries, a = sum_n Q_n K^T, while the training signal flows through row-wise softmax of the full attention matrix inside the aggregator (Eq. 5). No loss directly optimizes the summed score for Top-K selection, and Top-K itself is non-differentiable. The stop-gradient does not remove this issue: it only prevents gradients from entering the encoder, while the auxiliary loss still shapes the router. For classification and detection N=1, so the collapse is trivial; the concern is specifically the N=h*w segmentation case. The current evidence for alignment is indirect. I request a direct diagnostic, e.g., train a segmentation variant with the collapsed score as the only scoring mechanism, compare Cropr against a randomly initialized but fixed linear scorer qK^T with the same Top-K and LLF, or report the correlation between the inference score a and the aggregator's learned softmax attention weights. Without such evidence, the dense-task results could be mostly attributable to LLF plus a heuristic score.","section":"Secs. 3.1, 3.2 and 3.4"}],"minor_comments":[{"comment":"The abstract's 'small drops in performance' should be quantified: Table 3 shows APbox dropping from 64.2 to 63.0 and APmask from 55.4 to 54.0, which is larger than the phrase suggests; please state the exact drops in the abstract or conclusion.","section":"Abstract / Table 3"},{"comment":"The column header 'Acc. 1000 im/s' appears to be a rendering artifact; please separate the accuracy and throughput columns clearly.","section":"Table 1"},{"comment":"The claim that inference-time throughput is 'close to that of a random pruner' should be supported by exact throughput numbers in the figure or caption; the current bar chart is hard to read because the values are not labeled consistently.","section":"Sec. 3.4 / Fig. 3b"},{"comment":"State explicitly that the aggregated query q is a fixed vector after training, so the entire inference-time scorer reduces to a fixed linear readout of the token embeddings; this makes the method's capacity and limitations clearer.","section":"Sec. 3.4"},{"comment":"The t-SNE visualizations plot only the top-1 scoring pruned token per block; the interpretation that LLF 'synchronizes' tokens is suggestive rather than conclusive, and the text should note this limitation.","section":"App. F"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical systems paper that fits the journal's scope. I do not see grounds for rejection. The requested changes are (i) statistical support for the 0.1 mIoU headline claim and (ii) a direct test of the train-inference alignment of the collapsed segmentation scorer. The paper's 'small drops' framing should also be aligned with the actual Table 3 numbers. I did not find evidence of citation manipulation; the related work discussion is fair. The code link was not verified during review, so the authors should ensure a versioned release is available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look. The core idea is simple and sensible: train a cross-attention scorer with removable auxiliary heads, discard the heads and aggregator at inference, and pre-aggregate the queries into one vector so scoring is O(M). That last trick is the real novel bit, and it's cleanly derived in Sec. 3.4. The evaluation covers classification, segmentation, detection, and instance segmentation, with ablations for the module design, selection method, stop-grad, and fusion. That's more than most pruning papers do.\n\nThe main soft spots are in the reporting. The ADE20k headline—0.1 median mIoU drop at 2x speedup—is a median over 5 seeds with no spread. We don't know the variance, and the figure shows random pruning plus LLF gets fairly close, which suggests LLF is doing a lot of the work. The detection result is 64.2 to 63.0 APbox (1.2 drop), which is not 'small' by most standards, so the abstract oversells it. Also, the code link is given but not verified; worth checking.\n\nThe stress-test concern about the summed-query score for segmentation is legitimate but not fatal. The inference-time score for N>1 is a sum of per-query logits, which is a reasonable proxy for global importance, but it's true that no loss directly optimizes the collapsed score for Top-K selection, and the stop-grad means the backbone doesn't adapt to the router. The paper's own evidence that this matters is the segmentation result, which is exactly the one with the weakest error bars. So I'd want to see the per-seed numbers and maybe an ablation comparing the learned scorer to variance on a dense task with LLF fixed.\n\nThe paper is honest about limitations in App. B—hardware dependence, gap to no-pruning, manual schedule design—which I appreciate. The citation pattern looks fair; comparisons to prior work use different backbones, but they reimplement DToP fairly in one spot.\n\nBottom line: this deserves peer review. The method is practical, the evaluation is broad, and the main weakness is under-reported uncertainty, not a broken claim. I'd accept it with requests for variance reporting and a more careful abstract.","headline":"Solid token pruning paper with a neat inference-time query aggregation trick; the segmentation headline lacks error bars and the abstract overstates the detection drop.","tokens_in":20057,"tokens_out":3380,"would_cite":true,"duration_ms":32354,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token Cropr learns task-relevant token rankings with auxiliary heads that are discarded after training, yielding 1.5-4x Vision Transformer speedups with small accuracy losses.","keywords":["token pruning","vision transformers","cross-attention","auxiliary heads","inference speedup","semantic segmentation","object detection","Last Layer Fusion"],"falsifier":"Fine-tune a ViT with Cropr, then at inference replace only the learned scorer with a random scorer at the same pruning rates on ImageNet-1k and ADE20k; if random selection matches or beats Cropr's accuracy at equal throughput, the learned ranking is not carrying the result.","tokens_in":19087,"feed_emoji":"⚡","tokens_out":7145,"duration_ms":60051,"temperature":0.7,"pith_summary":"Token Cropr is a method for making Vision Transformers faster by removing image tokens that do not matter for the task at hand. Its central claim is that token importance can be learned end-to-end with small auxiliary prediction heads attached to a cross-attention router, and that the learned ranking still works after the auxiliary components are removed at inference. If that claim holds, image classification, semantic segmentation, object detection, and instance segmentation can all run 1.5-4x faster with only small accuracy penalties, and the overhead of the pruning mechanism itself drops to nearly that of a random pruner. The paper reports, as a best case, a 2x speedup on ADE20k semantic segmentation with a 0.1 median mIoU penalty, and 89.7% ImageNet top-1 accuracy at 2.1x speedup on a large ViT.","feed_headline":"Heads learn which tokens to prune, then vanish: 1.5-4x faster ViTs","feed_subtitle":"Training-only scoring heads pick task-relevant image patches, then drop away, keeping accuracy close to unpruned models.","key_machinery":"The central object is the Cropr module, inserted after ViT blocks, with four parts: a scorer that computes cross-attention $A = QK(X)^\\top$ between learnable queries and input token keys; a selector that keeps the Top-K tokens by summed attention scores; an aggregator that produces a weighted token for an auxiliary task head; and the head itself. A stop-gradient isolates the auxiliary branch from the backbone. The load-bearing identity for efficiency is $\\sum_{n=1}^N (QK^\\top)_n = (\\sum_{n=1}^N Q_n)K^\\top = qK^\\top$: at inference the per-query attention matrix need not be materialized, so scoring collapses to one vector-matrix product. Last Layer Fusion then reinserts pruned tokens before the final block so dense tasks can still see all patches.","core_discovery":"The paper's central claim is that token pruning for ViTs should be driven by a learned, task-specific notion of relevance rather than by heuristics such as self-attention scores or input variance. A Cropr module inserted after transformer blocks contains a cross-attention scorer that ranks tokens, a Top-K selector, an aggregator that forms a weighted token, and an auxiliary head that supplies task gradients; the aggregator and head are discarded after training. The paper establishes that the learned ranking transfers to inference, where the router collapses to a single precomputed query vector and a dot-product scoring step, giving throughput close to random pruning. Across classification, segmentation, detection, and instance segmentation, Cropr is claimed to beat random, variance, and attention-score baselines at comparable throughput and to close most of the accuracy gap to the unpruned model, especially as model size and image resolution grow.","pith_inferences":["Pith inference: Because the learned scores are produced by a single precomputed query vector, the same training-time setup could attach to any transformer with a differentiable task head, including non-image sequence models; the paper itself notes language and audio as future work.","Pith inference: The scores the router produces at inference are effectively free task-relevance maps, so Cropr could double as a lightweight saliency or explainability signal without a separate attention-rollout or attribution pass.","Pith inference: The stop-gradient isolation of the auxiliary heads suggests a testable recipe for other efficiency tricks, such as learning per-image adaptive keep rates or merging decisions, without backprop interference with the backbone."],"forward_implications":["Image classification can run 1.6-1.9x faster with an accuracy drop of only 0.3-0.7 points on an MAE-pretrained ViT-L on ImageNet-1k.","Semantic segmentation on ADE20k with an EV A-02 backbone reaches 56.6 median mIoU versus 56.7 for the unpruned model while being 2.0x faster.","Object detection and instance segmentation on COCO with Cascade Mask R-CNN maintain 63.0 AP box versus 64.2 unpruned while achieving a 1.9x overall speedup and a 2.4x encoder speedup.","Larger models and higher-resolution inputs get larger speedups and smaller performance penalties, with the accuracy gap shrinking from -0.9 on ViT-B to -0.4 on ViT-H despite higher total pruning ratios."],"supporting_citations":[{"why":"Identifies simple Top-K token selection as a strong baseline, which Cropr must beat on accuracy and throughput.","marker":"[23]"},{"why":"Supplies the cross-attention query mechanism that Cropr adapts for token scoring and task-shaped outputs.","marker":"[28]"},{"why":"Provides the masked-autoencoder fine-tuning setup used for all ImageNet-1k classification experiments.","marker":"[25]"},{"why":"Supplies the EV A-02 backbone and training settings used for the scale, segmentation, and detection experiments.","marker":"[19]"},{"why":"Supplies the Segmenter linear head and decoding setup adapted for semantic segmentation.","marker":"[54]"},{"why":"Offers the DToP logit-fusion auxiliary-head baseline that Cropr compares against in segmentation.","marker":"[56]"},{"why":"Provides the ToMe token-merging method and the bipartite matching baseline used in classification comparisons.","marker":"[6]"},{"why":"Provides the K-Medoids token-pooling baseline, which is slower than the unpruned model and motivates Cropr's low overhead.","marker":"[42]"}],"fun_headline_variants":["Learning heads pick tokens, then vanish: 1.5-4x faster ViTs","Token Cropr: Train auxiliary heads, delete them, run faster","Speed up ViTs 1.5-4x by letting auxiliary heads learn token relevance","Drop token-pruning heads after training: ViTs get 1.5-4x speedup","Learn token importance, then discard the heads: ViT speedup 1.5-4x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learned token ranking stays valid after the auxiliary heads and aggregator are removed, so the Top-K selector at inference keeps choosing the tokens that matter.","fun_headline_variants_meta":{"raw":{"variants":["Learning heads pick tokens, then vanish: 1.5-4x faster ViTs","Token Cropr: Train auxiliary heads, delete them, run faster","Speed up ViTs 1.5-4x by letting auxiliary heads learn token relevance","Drop token-pruning heads after training: ViTs get 1.5-4x speedup","Learn token importance, then discard the heads: ViT speedup 1.5-4x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3490,"prompt_tokens":910,"completion_tokens":2580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2465}},"tokens_in":526,"tokens_out":2580,"duration_ms":15876,"temperature":1.0,"reasoning_tokens":2465,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:48:40.033430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a ViT with Cropr, then at inference replace only the learned scorer with a random scorer at the same pruning rates on ImageNet-1k and ADE20k; if random selection matches or beats Cropr's accuracy at equal throughput, the learned ranking is not carrying the result.","supporting_citations":[{"cited_title":"Tay- lor, and Thomas B","cited_arxiv_id":null,"evidence_quote":"Identifies simple Top-K token selection as a strong baseline, which Cropr must beat on accuracy and throughput."},{"cited_title":"Perceiver IO: A general architecture for structured inputs & outputs","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-attention query mechanism that Cropr adapts for token scoring and task-shaped outputs."},{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Provides the masked-autoencoder fine-tuning setup used for all ImageNet-1k classification experiments."},{"cited_title":"Segmenter: Transformer for semantic segmenta- tion","cited_arxiv_id":null,"evidence_quote":"Supplies the Segmenter linear head and decoding setup adapted for semantic segmentation."},{"cited_title":"Dynamic token pruning in plain vision transformers for semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Offers the DToP logit-fusion auxiliary-head baseline that Cropr compares against in segmentation."},{"cited_title":"Token merging: Your vit but faster","cited_arxiv_id":null,"evidence_quote":"Provides the ToMe token-merging method and the bipartite matching baseline used in classification comparisons."},{"cited_title":"Token pooling in vision transformers for image classification","cited_arxiv_id":null,"evidence_quote":"Provides the K-Medoids token-pooling baseline, which is slower than the unpruned model and motivates Cropr's low overhead."}],"review_version":1}