{"id":"03325fc5-e48c-433d-afaf-f1d904a56ed0","arxiv_id":"2509.03379","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TinyDrop uses a lightweight model's confidence and attention map to early-exit easy samples and drop uninformative tokens in frozen ViTs, cutting FLOPs by up to 87%.","lead":"A small image model tells a large vision transformer which parts of the image to skip, so the big model has fewer patches to process and runs faster. It requires no retraining and the authors report up to 87% less computation with almost no loss in accuracy on ImageNet.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 80–87% FLOPs reduction is dominated by early exit, not token dropping; Table 2 lacks a random-token baseline to validate the guidance mechanism.","rationale":"The reader's CONDITIONAL verdict is reasonable, but the most load-bearing issue is not only whether small-model saliency transfers to the large ViT. The paper's own ablations show that the early-exit component, which is prior work, accounts for nearly all of the reported FLOPs reduction on large models, while the proposed token-dropping component adds only a small marginal saving. A random-token baseline is needed to determine whether the guidance model's token selection has any causal role in preserving accuracy. If random selection performs the same, the central claim about 'tiny model guided token dropping' is not supported and the paper should be reframed as an early-exit method with a minor token-pruning add-on. If random selection loses accuracy, the token-selection mechanism is real but still secondary to early exit. In either case, the current manuscript does not provide the evidence needed for the stated contribution.","tokens_in":6865,"tokens_out":9440,"duration_ms":90090,"concrete_test":"On ImageNet validation, evaluate TinyDrop with the token-selection step replaced by uniform random selection of K tokens, holding the early-exit threshold, K, and all other settings identical, for at least ViT-L/16 and DeiT-B. Compare accuracy and average FLOPs against Table 2's TinyDrop and w/o Token Dropping rows. Also report the marginal FLOPs saving of TinyDrop over w/o Token Dropping as a percentage of the total baseline reduction. If random selection reproduces TinyDrop's accuracy within 0.5% top-1, the guidance saliency is not load-bearing; if TinyDrop beats random by >1% top-1 at matched FLOPs, the token-selection mechanism is real but still contributes only a small fraction of the headline savings.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Table 2's internal ablations undermine the central claim that tiny-model-guided token selection produces the headline savings. For ViT-L/16, the w/o Token Dropping row (early exit only) reaches 85.2% top-1 at 11.6 GFLOPs (-81.2%), essentially the same accuracy as full TinyDrop (85.2%, 9.6 GFLOPs, -84.4%); the proposed token-dropping component contributes only ~2.0 GFLOPs average savings. The same pattern holds for DeiT-B (early-exit-only -77.3% vs TinyDrop -83.5%) and ViT-B/16 (-81.8% vs -83.5%). Therefore the abstract's 'up to 80%' would remain true if the token-dropping mechanism were removed. Furthermore, no experiment compares TinyDrop's guided top-K selection to random token dropping at the same kept ratio, so there is no evidence that the small model's Grad-CAM saliency identifies tokens that the target ViT needs; the early-exit confidence channel alone could account for the accuracy. The paper overstates the role of the proposed token-dropping component, and the mechanism-specific claim is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TinyDrop, a training-free framework that uses a lightweight guidance model to (i) optionally early-exit on high-confidence samples and (ii) compute Grad-CAM saliency maps to select top-K patch tokens for a frozen target ViT. A confidence-to-drop mapping (Eq. 1) sets the kept token count per sample. Experiments on ImageNet claim up to 87% FLOPs reduction for ViT-L/16 and 70–87% across large ViTs with ≤1% top-1 accuracy drop, and Table 2 compares favorably with token dropping and early-exit baselines. The main concern is that the reported operating points are selected by sweeping τ on the validation set, and the ablations show early exit alone accounts for most of the savings, with the token-selection component contributing only a small additional FLOP reduction.","tokens_in":7147,"tokens_out":6287,"duration_ms":64140,"significance":"The paper is clearly written, the pipeline is well specified (Sec. 2), and the authors are transparent about including the guidance-model forward and Grad-CAM backward costs in the reported FLOPs. The plug-and-play, training-free property is attractive, and the large-ViT evaluations (ViT-L/16, BEiTv2-L, DeiT3-L) are relevant. However, the evidence does not yet establish the proposed token-dropping mechanism as the source of the headline savings: the ablation in Table 2 shows early exit contributes the bulk of the FLOPs reduction, and no baseline rules out random token selection. The significance of the work would be considerably higher if these control experiments are added.","major_comments":[{"comment":"The headline numbers (e.g., -87.0% for ViT-L/16) are not a single deployment configuration; they are operating points selected post hoc by sweeping the confidence threshold τ on the ImageNet validation set to satisfy an accuracy-drop budget. This makes Table 1 an upper envelope rather than a reproducible rule. The paper should report the actual τ (and resulting drop-ratio distribution) for each operating point, or evaluate at a fixed τ on the full validation set. Without this, the claim 'reduces FLOPs by up to 80%' is not tied to a concrete, deployable setting.","section":"§3.1, Table 1"},{"comment":"The ablation shows that the early-exit component, not token dropping, is responsible for most of the reported savings. For ViT-L/16, TinyDrop is 85.2% at 9.6 GFLOPs (-84.4%), whereas w/o Token Dropping (early exit only) is 85.2% at 11.6 GFLOPs (-81.2%); the token-dropping component provides 2.0 GFLOPs net savings. Similar patterns hold for DeiT-B (4.0→2.9 GFLOPs), DeiT-S (1.8→1.4), and ViT-B/16 (3.2→2.9). Moreover, the early-exit component bypasses the target ViT entirely on high-confidence samples, so the title claim 'token dropping for ViTs' conflates two mechanisms. The paper should report the token-dropping contribution on the deferred subset only and temper the abstract/contribution wording accordingly.","section":"§3.2.2, Table 2"},{"comment":"No experiment compares the proposed guided top-K selection to random token dropping at the same kept ratio K. Without this baseline, the data do not establish that the guidance model's saliency identifies tokens the frozen target ViT needs; the accuracy of TinyDrop could be due to the early-exit path alone, which is tested by w/o Token Dropping but not by a random-drop control. Add a random-drop curve (and ideally a target-model Grad-CAM/attention baseline) at identical K and τ for each backbone. This is essential to validate the core mechanism of §2.1.2.","section":"§2.2, Eq. (3); Table 2"},{"comment":"The central transfer assumption is that saliency computed on the lightweight guidance model M_g identifies the same tokens the target ViT M_t needs. The paper does not test this assumption directly. A concrete test is to report the top-K overlap (e.g., IoU or rank correlation) between S_g and the target model's own attention or gradient saliency, and to compare accuracy when tokens are selected by the target's own saliency. If the overlap is low, the accuracy results may be dominated by the early-exit path, not by informed token selection.","section":"§2.1.2"}],"minor_comments":[{"comment":"The phrase 'if large vit models need to perform attention calculations' is ungrammatical; suggest 'if the large ViT needs to perform attention calculations.'","section":"Abstract"},{"comment":"The text cites [14] for class activation maps but uses Grad-CAM [16]; please align the citation with the actual method.","section":"§2.1.2"},{"comment":"The text says 'We sweep the drop ratio r using the mapping function' but r is derived from τ and c. Clarify that the sweep is over τ, and report the resulting per-sample drop ratios or the τ values used for each Table 1 operating point.","section":"§3.1"},{"comment":"The γ ablation is qualitative. Provide a table of accuracy and FLOPs for each γ, and specify which γ is used in Table 2.","section":"Fig. 3"},{"comment":"The caption should explicitly define 'w/o Early Exit' (token dropping active, no early exit) and 'w/o Token Dropping' (early exit only, no Grad-CAM/selection) to prevent misreading of the ablation rows.","section":"Table 2"},{"comment":"The name 'PaPr' is written inconsistently as 'Papr' in the introduction and Table 2; harmonize. Also, the footnote for DiffRate⋆—'we use the better one'—is ambiguous about whether the better configuration is training-free or fine-tuned.","section":"References / Table 2"},{"comment":"The conclusion repeats 'up to 80%' while Table 1 reports 87.0%; harmonize the numbers.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The framing of the paper emphasizes token dropping, but the ablation in Table 2 indicates that the early-exit component (built on the authors' prior work [15]) carries most of the benefit. The missing random-drop baseline is a standard control and should be required before publication. The validation-set threshold sweep is also a standard issue; a fixed operating point with reported thresholds is needed. I do not see an irreparable flaw, but the manuscript needs a significant experimental revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the paper is a plausible, plug-and-play efficiency recipe, but the headline 80–87% FLOPs reduction is mostly powered by the early-exit component, not the proposed token dropping. The paper's own Table 2 shows that for ViT-L/16, removing token dropping entirely (early exit only) gives 85.2% top-1 at 11.6 GFLOPs (−81.2%), essentially identical to full TinyDrop at 9.6 GFLOPs (−84.4%). The same pattern holds for DeiT-B and ViT-B/16. So the abstract's 'up to 80%' would survive with the novel mechanism deleted.\n\nWhat is actually new: the combination of a lightweight-model Grad-CAM saliency map with a confidence-to-drop mapping and early exit, applied to frozen ViTs without retraining. That specific combination doesn't appear in PaPr or the authors' earlier early-exit work. The paper is clearly written, the ablations are honest about the two components, and the FLOPs accounting at least attempts to include the guidance model and Grad-CAM overhead. The confidence-to-drop mapping (Eq. 1) is a simple, sensible idea, and the method is genuinely training-free and model-agnostic.\n\nSoft spots, in order of severity. First, no random-token baseline. The central mechanism—that the small model's saliency identifies tokens the large model actually needs—is never tested. Without comparing guided top-K to random dropping at the same keep ratio, the accuracy at a given FLOPs could be entirely explained by the early-exit confidence channel. Second, the operating points in Table 1 are selected by sweeping τ on the ImageNet validation set (Section 3.1). The reported 'up to 87%' is a best-case curve, not a fixed deployment rule, and the paper doesn't separate validation from test for threshold selection. Third, the contribution of token dropping to the headline numbers is marginal on the large models, yet the introduction and conclusion credit it as the main driver. This is over-claiming, not fraud—the ablation rows are right there—but it needs reframing. Fourth, no code released, which matters for a recipe-style paper.\n\nWho it's for: practitioners who want a training-free speed-up for frozen ViTs and are okay with a small guidance model plus early exit. The paper deserves a serious referee: the method is reproducible in principle, the ablations are informative, and the flaws are fixable. But it should not be accepted as is. The authors need to add a random-token baseline, report fixed thresholds with separate validation/test selection, and rewrite the contribution to accurately state that early exit dominates the savings on large models.","headline":"Honest ablations but over-credited token dropping: early exit does most of the work, and the saliency guidance is never tested against random selection.","tokens_in":7625,"tokens_out":1469,"would_cite":false,"duration_ms":16039,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TinyDrop claims that a small pre-trained guide model can select which image patches a large frozen Vision Transformer computes on, cutting FLOPs by up to 87 percent on ImageNet while keeping top-1 accuracy loss within one point.","keywords":["Vision Transformers","Token Dropping","Training-free Efficiency","Model Compression","Early Exit","Saliency Mapping","Image Classification","FLOPs Reduction"],"falsifier":"On ImageNet validation images that do not trigger early exit, compute two top-K token rankings for each image: TinyDrop's saliency ranking from the small guide, and a ranking from the target ViT's own class-token attention or output-gradient saliency. The accuracy claim predicts high agreement on the tokens that actually determine the target's prediction; if agreement is low on exactly the images where TinyDrop drops many tokens, the reported FLOPs/accuracy trade-off should not be expected to hold for other backbones or data.","tokens_in":6770,"feed_emoji":"⚡","tokens_out":9575,"duration_ms":89420,"temperature":0.7,"pith_summary":"TinyDrop sets out to show that a frozen, already-trained Vision Transformer can run much cheaper without any retraining if a small auxiliary model tells it which image tokens matter. The guidance model does two jobs: it decides when the large network can be skipped entirely, and when the large network must run, it produces a saliency map that is resized to the ViT's patch grid so the large model only sees the top-K tokens. On ImageNet-1K, across three large backbones and two guidance models, the paper reports average FLOPs reductions of 70 to 87 percent while keeping the top-1 accuracy loss within 1 percent, with the guidance model's forward and gradient costs counted. If correct, this makes large pre-trained ViTs practical in compute- or latency-limited settings without touching their weights.","feed_headline":"A tiny guide cuts ViT compute by up to 87%","feed_subtitle":"Small-model saliency picks the tokens a frozen large ViT keeps, losing at most 1% accuracy.","key_machinery":"The load-bearing component is cross-model saliency transfer: the guidance model's gradient-weighted class-activation map is computed from its final block, resampled to the target ViT's patch grid, normalized to [0,1], and used to rank tokens. A confidence-to-drop mapping, r = min(rmax, rmax·(c/τ)^γ), converts the guide's softmax confidence into an adaptive drop ratio, so confident images keep fewer tokens; K = ⌊(1−r)T⌋ tokens are retained by top-K selection. To keep the frozen target valid, only the retained tokens' absolute positional embeddings (and the corresponding relative-position-bias sub-matrix, where used) are fed in. This lets the target avoid retraining while all of its self-atten","core_discovery":"The paper's central claim is that token importance for a large ViT can be borrowed from a much smaller, pre-trained model rather than learned or computed inside the target. TinyDrop runs a lightweight guidance model first; if its softmax confidence exceeds a threshold, it exits and the target ViT is skipped entirely. Otherwise the guide's saliency map is bilinearly resampled to the target's token grid, a confidence-to-drop mapping fixes how many tokens to remove, and the top-K tokens plus the class token are fed into the frozen target, with positional embeddings and relative position biases sub-selected to match. Evaluated on ImageNet-1K with three large backbones (ViT-L/16, BEiTv2-L, DeiT3-","pith_inferences":["The reported numbers are aggregate ImageNet averages; the per-sample drop ratio varies with guide confidence, so images where the guide is overconfident but wrong will be exited early at the guide's accuracy, not the target's.","A natural stress test is to compare TinyDrop's token picks against the target model's own attention or gradient ranking; if overlap is low on hard classes, the saliency-transfer premise is the bottleneck.","The same recipe could extend to other frozen architectures or dense prediction tasks if a task-aligned saliency map replaces image-classification class-activation mapping."],"forward_implications":["Any existing ViT checkpoint can be accelerated by attaching a small guide; no gradient updates are run on the target model.","At a 1% accuracy-loss budget, ViT-L/16 drops from 61.6 to 8.0 GFLOPs on average with the EfficientFormerV2-s2 guide, an 87.0% reduction.","At a 0.3% budget, the same model still runs at 14.0 GFLOPs (−77.3%), so even conservative users get large savings.","Because both transformer-style and CNN-style guidance models work, the cheapest adequate guide can be chosen per deployment."],"supporting_citations":[{"why":"Defines the Vision Transformer patch-token architecture and the quadratic attention cost that TinyDrop reduces.","marker":"[1]"},{"why":"BEiTv2-Large, one of the three frozen target backbones evaluated in Table 1.","marker":"[3]"},{"why":"DeiT3-Large, another frozen target backbone used to demonstrate generalization in Table 1.","marker":"[4]"},{"why":"Token-merging baseline whose accuracy/FLOPs trade-off TinyDrop must exceed in Table 2.","marker":"[7]"},{"why":"Zero-TP, a training-free token-pruning baseline that also drops tokens before attention, compared in Table 2.","marker":"[10]"},{"why":"PaPr, a lightweight-convnet patch-pruning baseline using a separate patch-importance network, compared with cost accounting noted in Table 2.","marker":"[11]"},{"why":"EfficientFormerV2-s2, the transformer-style guidance model used to generate saliency maps in Table 1.","marker":"[12]"},{"why":"EfficientNetV2, the CNN-style guidance model used to show the guide need not be a transformer.","marker":"[13]"},{"why":"Provides the gradient-based saliency method used to compute token importance from the guidance model.","marker":"[16]"},{"why":"ImageNet-1K validation set, the benchmark on which all accuracy and FLOPs numbers in Tables 1 and 2 are measured.","marker":"[19]"}],"fun_headline_variants":["Small model tells big ViT which tokens to drop, 80% less compute","Token dropping via tiny guide: 80% FLOP cut, ~1% accuracy cost","Use a tiny model to prune tokens from Vision Transformers, 80% FLOPs saved","TinyDrop: few tokens, big savings — 80% less compute for ViTs","Guide a large ViT with a small model to drop 80% of its compute"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the small guidance model's saliency map marks the same image regions the frozen large ViT needs for its final decision; if the two models attend to different regions, dropping low-saliency tokens discards the very information the large model relied on.","fun_headline_variants_meta":{"raw":{"variants":["Small model tells big ViT which tokens to drop, 80% less compute","Token dropping via tiny guide: 80% FLOP cut, ~1% accuracy cost","Use a tiny model to prune tokens from Vision Transformers, 80% FLOPs saved","TinyDrop: few tokens, big savings — 80% less compute for ViTs","Guide a large ViT with a small model to drop 80% of its compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":2751,"prompt_tokens":662,"completion_tokens":2089,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":1975}},"tokens_in":406,"tokens_out":2089,"duration_ms":13984,"temperature":1.0,"reasoning_tokens":1975,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:56:23.261301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On ImageNet validation images that do not trigger early exit, compute two top-K token rankings for each image: TinyDrop's saliency ranking from the small guide, and a ranking from the target ViT's own class-token attention or output-gradient saliency. The accuracy claim predicts high agreement on the tokens that actually determine the target's prediction; if agreement is low on exactly the images where TinyDrop drops many tokens, the reported FLOPs/accuracy trade-off should not be expected to hold for other backbones or data.","supporting_citations":[{"cited_title":"Un- like convolutional networks, ViTs process all input tokens uniformly with identical computational cost, resulting in quadratic complexity relative to token count","cited_arxiv_id":null,"evidence_quote":"Defines the Vision Transformer patch-token architecture and the quadratic attention cost that TinyDrop reduces."},{"cited_title":"Settings We evaluate TinyDrop across ImageNet-1K [19] on state-of- the-art architectures, including Vision Transformer [1, 20], BEiTv2 [3], DeiT3 [4] and DeiT [21]","cited_arxiv_id":null,"evidence_quote":"BEiTv2-Large, one of the three frozen target backbones evaluated in Table 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeiT3-Large, another frozen target backbone used to demonstrate generalization in Table 1."},{"cited_title":"Swin transformer v2: Scaling up capacity and resolution,","cited_arxiv_id":null,"evidence_quote":"Token-merging baseline whose accuracy/FLOPs trade-off TinyDrop must exceed in Table 2."},{"cited_title":"Zero-tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers,","cited_arxiv_id":null,"evidence_quote":"Zero-TP, a training-free token-pruning baseline that also drops tokens before attention, compared in Table 2."},{"cited_title":"In addition, Papr reports pri- marily on smaller/earlier backbones, with limited accounting of cost when scaling to large frozen ViTs","cited_arxiv_id":null,"evidence_quote":"PaPr, a lightweight-convnet patch-pruning baseline using a separate patch-importance network, compared with cost accounting noted in Table 2."},{"cited_title":"Low-rank approximation for sparse attention in multi-modal llms,","cited_arxiv_id":null,"evidence_quote":"EfficientFormerV2-s2, the transformer-style guidance model used to generate saliency maps in Table 1."},{"cited_title":"Token merging: Your vit but faster,","cited_arxiv_id":null,"evidence_quote":"EfficientNetV2, the CNN-style guidance model used to show the guide need not be a transformer."},{"cited_title":"Papr: Training-free one-step patch pruning with lightweight convnets for faster infer- ence,","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-based saliency method used to compute token importance from the guidance model."},{"cited_title":"Extracting class ac- tivation maps from non-discriminative features as well,","cited_arxiv_id":null,"evidence_quote":"ImageNet-1K validation set, the benchmark on which all accuracy and FLOPs numbers in Tables 1 and 2 are measured."}],"review_version":1}