Pith. sign in

REVIEW 4 major objections 7 minor 33 references

TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers

T0 review · 4 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TinyDrop claims that a small pre-trained guide model can select which image patches a large frozen Vision Transformer computes on, cutting FLOPs by up to 87 percent on ImageNet while keeping top-1 accuracy loss within one point.

desk verdict Honest ablations but over-credited token dropping: early exit does most of the work, and the saliency guidance is never tested against random selection. read the letter →

arxiv 2509.03379 v1 pith:KSVEYOX6 submitted 2025-09-03 cs.CV cs.AI

Guoxin Wang , Qingyuan Wang , Binhua Huang , Shaowu Chen , Deepu John This is my paper
classification cs.CVcs.AI
keywords VisionTransformersTokenDroppingTraining-freeEfficiencyModelCompressionEarlyExitSaliencyMappingImageClassificationFLOPsReduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TinyDrop sets out to show that a frozen, already-trained Vision Transformer can run much cheaper without any retraining if a small auxiliary model tells it which image tokens matter. The guidance model does two jobs: it decides when the large network can be skipped entirely, and when the large network must run, it produces a saliency map that is resized to the ViT's patch grid so the large model only sees the top-K tokens. On ImageNet-1K, across three large backbones and two guidance models, the paper reports average FLOPs reductions of 70 to 87 percent while keeping the top-1 accuracy loss within 1 percent, with the guidance model's forward and gradient costs counted. If correct, this makes large pre-trained ViTs practical in compute- or latency-limited settings without touching their weights.

What carries the argument

The load-bearing component is cross-model saliency transfer: the guidance model's gradient-weighted class-activation map is computed from its final block, resampled to the target ViT's patch grid, normalized to [0,1], and used to rank tokens. A confidence-to-drop mapping, r = min(rmax, rmax·(c/τ)^γ), converts the guide's softmax confidence into an adaptive drop ratio, so confident images keep fewer tokens; K = ⌊(1−r)T⌋ tokens are retained by top-K selection. To keep the frozen target valid, only the retained tokens' absolute positional embeddings (and the corresponding relative-position-bias sub-matrix, where used) are fed in. This lets the target avoid retraining while all of its self-atten

What would settle it

On ImageNet validation images that do not trigger early exit, compute two top-K token rankings for each image: TinyDrop's saliency ranking from the small guide, and a ranking from the target ViT's own class-token attention or output-gradient saliency. The accuracy claim predicts high agreement on the tokens that actually determine the target's prediction; if agreement is low on exactly the images where TinyDrop drops many tokens, the reported FLOPs/accuracy trade-off should not be expected to hold for other backbones or data.

Watch

Extended reading notes

Core claim

The paper's central claim is that token importance for a large ViT can be borrowed from a much smaller, pre-trained model rather than learned or computed inside the target. TinyDrop runs a lightweight guidance model first; if its softmax confidence exceeds a threshold, it exits and the target ViT is skipped entirely. Otherwise the guide's saliency map is bilinearly resampled to the target's token grid, a confidence-to-drop mapping fixes how many tokens to remove, and the top-K tokens plus the class token are fed into the frozen target, with positional embeddings and relative position biases sub-selected to match. Evaluated on ImageNet-1K with three large backbones (ViT-L/16, BEiTv2-L, DeiT3-

Load-bearing premise

The load-bearing premise is that the small guidance model's saliency map marks the same image regions the frozen large ViT needs for its final decision; if the two models attend to different regions, dropping low-saliency tokens discards the very information the large model relied on.

Editorial extensions

If this is right

  • Any existing ViT checkpoint can be accelerated by attaching a small guide; no gradient updates are run on the target model.
  • At a 1% accuracy-loss budget, ViT-L/16 drops from 61.6 to 8.0 GFLOPs on average with the EfficientFormerV2-s2 guide, an 87.0% reduction.
  • At a 0.3% budget, the same model still runs at 14.0 GFLOPs (−77.3%), so even conservative users get large savings.
  • Because both transformer-style and CNN-style guidance models work, the cheapest adequate guide can be chosen per deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported numbers are aggregate ImageNet averages; the per-sample drop ratio varies with guide confidence, so images where the guide is overconfident but wrong will be exited early at the guide's accuracy, not the target's.
  • A natural stress test is to compare TinyDrop's token picks against the target model's own attention or gradient ranking; if overlap is low on hard classes, the saliency-transfer premise is the bottleneck.
  • The same recipe could extend to other frozen architectures or dense prediction tasks if a task-aligned saliency map replaces image-classification class-activation mapping.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes TinyDrop, a training-free framework that uses a lightweight guidance model to (i) optionally early-exit on high-confidence samples and (ii) compute Grad-CAM saliency maps to select top-K patch tokens for a frozen target ViT. A confidence-to-drop mapping (Eq. 1) sets the kept token count per sample. Experiments on ImageNet claim up to 87% FLOPs reduction for ViT-L/16 and 70–87% across large ViTs with ≤1% top-1 accuracy drop, and Table 2 compares favorably with token dropping and early-exit baselines. The main concern is that the reported operating points are selected by sweeping τ on the validation set, and the ablations show early exit alone accounts for most of the savings, with the token-selection component contributing only a small additional FLOP reduction.

Significance. The paper is clearly written, the pipeline is well specified (Sec. 2), and the authors are transparent about including the guidance-model forward and Grad-CAM backward costs in the reported FLOPs. The plug-and-play, training-free property is attractive, and the large-ViT evaluations (ViT-L/16, BEiTv2-L, DeiT3-L) are relevant. However, the evidence does not yet establish the proposed token-dropping mechanism as the source of the headline savings: the ablation in Table 2 shows early exit contributes the bulk of the FLOPs reduction, and no baseline rules out random token selection. The significance of the work would be considerably higher if these control experiments are added.

major comments (4)
  1. [§3.1, Table 1] The headline numbers (e.g., -87.0% for ViT-L/16) are not a single deployment configuration; they are operating points selected post hoc by sweeping the confidence threshold τ on the ImageNet validation set to satisfy an accuracy-drop budget. This makes Table 1 an upper envelope rather than a reproducible rule. The paper should report the actual τ (and resulting drop-ratio distribution) for each operating point, or evaluate at a fixed τ on the full validation set. Without this, the claim 'reduces FLOPs by up to 80%' is not tied to a concrete, deployable setting.
  2. [§3.2.2, Table 2] The ablation shows that the early-exit component, not token dropping, is responsible for most of the reported savings. For ViT-L/16, TinyDrop is 85.2% at 9.6 GFLOPs (-84.4%), whereas w/o Token Dropping (early exit only) is 85.2% at 11.6 GFLOPs (-81.2%); the token-dropping component provides 2.0 GFLOPs net savings. Similar patterns hold for DeiT-B (4.0→2.9 GFLOPs), DeiT-S (1.8→1.4), and ViT-B/16 (3.2→2.9). Moreover, the early-exit component bypasses the target ViT entirely on high-confidence samples, so the title claim 'token dropping for ViTs' conflates two mechanisms. The paper should report the token-dropping contribution on the deferred subset only and temper the abstract/contribution wording accordingly.
  3. [§2.2, Eq. (3); Table 2] No experiment compares the proposed guided top-K selection to random token dropping at the same kept ratio K. Without this baseline, the data do not establish that the guidance model's saliency identifies tokens the frozen target ViT needs; the accuracy of TinyDrop could be due to the early-exit path alone, which is tested by w/o Token Dropping but not by a random-drop control. Add a random-drop curve (and ideally a target-model Grad-CAM/attention baseline) at identical K and τ for each backbone. This is essential to validate the core mechanism of §2.1.2.
  4. [§2.1.2] The central transfer assumption is that saliency computed on the lightweight guidance model M_g identifies the same tokens the target ViT M_t needs. The paper does not test this assumption directly. A concrete test is to report the top-K overlap (e.g., IoU or rank correlation) between S_g and the target model's own attention or gradient saliency, and to compare accuracy when tokens are selected by the target's own saliency. If the overlap is low, the accuracy results may be dominated by the early-exit path, not by informed token selection.
minor comments (7)
  1. [Abstract] The phrase 'if large vit models need to perform attention calculations' is ungrammatical; suggest 'if the large ViT needs to perform attention calculations.'
  2. [§2.1.2] The text cites [14] for class activation maps but uses Grad-CAM [16]; please align the citation with the actual method.
  3. [§3.1] The text says 'We sweep the drop ratio r using the mapping function' but r is derived from τ and c. Clarify that the sweep is over τ, and report the resulting per-sample drop ratios or the τ values used for each Table 1 operating point.
  4. [Fig. 3] The γ ablation is qualitative. Provide a table of accuracy and FLOPs for each γ, and specify which γ is used in Table 2.
  5. [Table 2] The caption should explicitly define 'w/o Early Exit' (token dropping active, no early exit) and 'w/o Token Dropping' (early exit only, no Grad-CAM/selection) to prevent misreading of the ablation rows.
  6. [References / Table 2] The name 'PaPr' is written inconsistently as 'Papr' in the introduction and Table 2; harmonize. Also, the footnote for DiffRate⋆—'we use the better one'—is ambiguous about whether the better configuration is training-free or fine-tuned.
  7. [Conclusion] The conclusion repeats 'up to 80%' while Table 1 reports 87.0%; harmonize the numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework is empirical and self-contained; the early-exit component is cited but independently ablated, and no equation-level reduction is present.

full rationale

TinyDrop is an empirical pipeline: a lightweight guidance model produces a saliency map and confidence, which are used to drop tokens and optionally exit early. There is no derivation chain in which a predicted quantity is defined in terms of the same quantity, and the reported FLOPs/accuracy numbers come from experiments, not from construction. The only self-citation is [15] for the confidence-based early exit mechanism. The paper includes an ablation row 'w/o Token Dropping' showing that early exit alone accounts for a large share of the savings, so the contribution of the borrowed component is transparent and empirically verified rather than assumed via citation. The absence of a random-token-dropping baseline weakens the mechanistic claim that the tiny model's Grad-CAM saliency identifies important tokens, but that is an experimental-support issue, not circularity. The confidence-to-drop mapping (Eq. 1) is an ad-hoc parametric schedule whose parameters are swept, not fitted to the target metric, and the reported operating points are selected by accuracy tolerance, so there is no fitted-input-called-prediction pattern. No step reduces to its own inputs by definition, and no load-bearing argument rests solely on a self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or architectural entities are proposed. The guidance model is an off-the-shelf EfficientFormerV2 or EfficientNetV2, and Grad-CAM, top-k selection, and positional embedding adaptation are standard techniques.

free parameters (3)
  • confidence threshold tau = swept (no single value reported)
    Controls both early exit and the drop ratio; Table 1 reports different savings for <=1%, <=0.5%, and <=0.3% accuracy-drop budgets, so tau is tuned on the validation set.
  • drop curvature gamma = 0.5
    Set by hand; ablated in Fig. 3 to show the efficiency-accuracy trade-off.
  • max drop ratio rmax = 0.7
    Set by hand; caps token dropping in Eq. (1).
assumptions (3)
  • domain assumption Grad-CAM saliency from the guidance model is a valid proxy for token importance in the target ViT.
    Section 2.1.2 computes token importance from Mg's attention and uses it to drop tokens in Mt; no validation against target-model saliency is reported.
  • domain assumption The guidance model's confidence is calibrated enough that accepting its prediction above tau preserves target accuracy.
    Section 2.1.1 replaces the large model's output with the small model's output when c > tau; if the small model is confidently wrong, the accuracy budget breaks.
  • domain assumption Pre-trained weights from the timm library are reliable external benchmarks for all models.
    Section 3.1 says all models are pre-trained from reference [22]; the reported numbers depend on those specific weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers." pith.science (2026). https://pith.science/paper/KSVEYOX6

@misc{pith2026250903379,
  author       = {Pith},
  title        = {Pith review of: TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSVEYOX6}},
  note         = {Machine review of arXiv:2509.03379}
}
read the original abstract

Vision Transformers (ViTs) achieve strong performance in image classification but incur high computational costs from processing all image tokens. To reduce inference costs in large ViTs without compromising accuracy, we propose TinyDrop, a training-free token dropping framework guided by a lightweight vision model. The guidance model estimates the importance of tokens while performing inference, thereby selectively discarding low-importance tokens if large vit models need to perform attention calculations. The framework operates plug-and-play, requires no architectural modifications, and is compatible with diverse ViT architectures. Evaluations on standard image classification benchmarks demonstrate that our framework reduces FLOPs by up to 80% for ViTs with minimal accuracy degradation, highlighting its generalization capability and practical utility for efficient ViT-based classification.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 32 canonical work pages

  1. [1]

    Un- like convolutional networks, ViTs process all input tokens uniformly with identical computational cost, resulting in quadratic complexity relative to token count

    INTRODUCTION Vision Transformers (ViTs) achieve competitive image clas- sification performance by modeling long-range dependencies among image patches through self-attention [1, 2]. Un- like convolutional networks, ViTs process all input tokens uniformly with identical computational cost, resulting in quadratic complexity relative to token count. While en...

    work page

  2. [2]

    TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers

    METHOD As shown in Fig. 1 , our framework operates as follows: 1) A computationally efficient guidance model Mg processes the input image and decides if to exit. 2) Token importance arXiv:2509.03379v1 [cs.CV] 3 Sep 2025 Target ViT Input Patch Embedding Token Token Dropping Blocks Output Saved Token Token Dropping Token Saved Token Drop Ratio (TOP-K) Input...

    work page Pith review arXiv 2025

  3. [3]

    Settings We evaluate TinyDrop across ImageNet-1K [19] on state-of- the-art architectures, including Vision Transformer [1, 20], BEiTv2 [3], DeiT3 [4] and DeiT [21]

    EXPERIMENTS 3.1. Settings We evaluate TinyDrop across ImageNet-1K [19] on state-of- the-art architectures, including Vision Transformer [1, 20], BEiTv2 [3], DeiT3 [4] and DeiT [21]. We also evaluate the performance with different guidance models, including Effi- cientNetV2 [13] and EfficientFormerV2 [12]. All used mod- els are pre-trained from [22]. Image...

    work page

  4. [4]

    CONCLUSION This paper presents a training-free token dropping framework that reduces computational costs in large Vision Transform- ers without retraining. By leveraging a lightweight guid- ance model to generate token importance scores through our confidence-to-drop mapping mechanism, we enable in- 1.0 1.5 2.0 2.5 3.0 3.5 4.0 FLOPs 1e9 70 72 74 76 78 80 ...

    work page

  5. [5]

    Logit standardization in knowl- edge distillation,

    Shangquan Sun et al., “Logit standardization in knowl- edge distillation,” in CVPR, 2024, pp. 15731–15740

    work page 2024

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    Alexey Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2021

    work page 2021

  7. [7]

    Swin transformer v2: Scaling up capacity and resolution,

    Ze Liu et al., “Swin transformer v2: Scaling up capacity and resolution,” in CVPR, 2022, pp. 12009–12019

    work page 2022

  8. [8]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers,

    Zhiliang Peng et al., “Beit v2: Masked image modeling with vector-quantized visual tokenizers,” arXiv preprint arXiv:2208.06366, 2022

    arXiv 2022

Show all 33 references
  1. [9]

    Deit iii: Revenge of the vit,

    Hugo Touvron et al., “Deit iii: Revenge of the vit,” in ECCV. Springer, 2022, pp. 516–533

    2022

  2. [10]

    Zero-tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers,

    Hongjie Wang et al., “Zero-tprune: Zero-shot token pruning through leveraging of the attention graph in pre-trained transformers,” in CVPR, 2024, pp. 16070– 16079

    2024

  3. [11]

    In addition, Papr reports pri- marily on smaller/earlier backbones, with limited accounting of cost when scaling to large frozen ViTs

    predicts one-shot patch importance with a lightweight ConvNet, yet applies a fixed keep rate, and the ConvNet’s complexity was not counted. In addition, Papr reports pri- marily on smaller/earlier backbones, with limited accounting of cost when scaling to large frozen ViTs. We...

  4. [12]

    Low-rank approximation for sparse attention in multi-modal llms,

    Lin Song et al., “Low-rank approximation for sparse attention in multi-modal llms,” in CVPR, 2024, pp. 13763–13773

    2024

  5. [13]

    Token merging: Your vit but faster,

    Daniel Bolya et al., “Token merging: Your vit but faster,” in ICLR, 2023

    2023

  6. [14]

    Diffrate: Differentiable com- pression rate for efficient vision transformers,

    Mengzhao Chen et al., “Diffrate: Differentiable com- pression rate for efficient vision transformers,” inICCV, 2023, pp. 17164–17174

    2023

  7. [15]

    Adaptive token sampling for efficient vision transformers,

    Mohsen Fayyaz et al., “Adaptive token sampling for efficient vision transformers,” in ECCV. Springer, 2022, pp. 396–414

    2022

  8. [16]

    Papr: Training-free one-step patch pruning with lightweight convnets for faster infer- ence,

    Tanvir Mahmud et al., “Papr: Training-free one-step patch pruning with lightweight convnets for faster infer- ence,” in ECCV. Springer, 2024, pp. 110–128

    2024

  9. [17]

    Rethinking vision transformers for mobilenet size and speed,

    Yanyu Li et al., “Rethinking vision transformers for mobilenet size and speed,” in ICCV, 2023, pp. 16889– 16900

    2023

  10. [18]

    Efficientnetv2: Smaller models and faster training,

    Mingxing Tan and Quoc Le, “Efficientnetv2: Smaller models and faster training,” in ICML. PMLR, 2021, pp. 10096–10106

    2021

  11. [19]

    Extracting class ac- tivation maps from non-discriminative features as well,

    Zhaozheng Chen and Qianru Sun, “Extracting class ac- tivation maps from non-discriminative features as well,” in CVPR, 2023, pp. 3135–3144

    2023

  12. [20]

    Tiny models are the computa- tional saver for large models,

    Qingyuan Wang et al., “Tiny models are the computa- tional saver for large models,” inECCV. Springer, 2024, pp. 163–182

    2024

  13. [21]

    Grad-cam: Visual ex- planations from deep networks via gradient-based local- ization,

    Ramprasaath R Selvaraju et al., “Grad-cam: Visual ex- planations from deep networks via gradient-based local- ization,” in ICCV, 2017, pp. 618–626

    2017

  14. [22]

    The impact of posi- tional encoding on length generalization in transform- ers,

    Amirhossein Kazemnejad et al., “The impact of posi- tional encoding on length generalization in transform- ers,” NeurIPS, vol. 36, pp. 24892–24928, 2023

    2023

  15. [23]

    Rethinking and improving relative po- sition encoding for vision transformer,

    Kan Wu et al., “Rethinking and improving relative po- sition encoding for vision transformer,” in ICCV, 2021, pp. 10033–10041

    2021

  16. [24]

    Imagenet large scale visual recognition challenge,

    Olga Russakovsky et al., “Imagenet large scale visual recognition challenge,” International journal of com- puter vision, vol. 115, no. 3, pp. 211–252, 2015

    2015

  17. [25]

    How to train your vit? data, augmentation, and regularization in vision trans- formers,

    Andreas Peter Steiner et al., “How to train your vit? data, augmentation, and regularization in vision trans- formers,” Transactions on Machine Learning Research, 2022

    2022

  18. [26]

    Training data-efficient image transformers & distillation through attention,

    Hugo Touvron et al., “Training data-efficient image transformers & distillation through attention,” in ICML. PMLR, 2021, pp. 10347–10357

    2021

  19. [27]

    Pytorch image mod- els,

    Ross Wightman, “Pytorch image mod- els,” https://github.com/rwightman/ pytorch-image-models, 2019

    2019

  20. [28]

    Token fusion: Bridging the gap between token pruning and token merging,

    Minchul Kim et al., “Token fusion: Bridging the gap between token pruning and token merging,” in WACV, 2024, pp. 1383–1392

    2024

  21. [29]

    Cf-vit: A general coarse-to- fine method for vision transformer,

    Mengzhao Chen et al., “Cf-vit: A general coarse-to- fine method for vision transformer,” in AAAI, 2023, pp. 7042–7052

    2023

  22. [30]

    Dynamic perceiver for efficient vi- sual recognition,

    Yizeng Han et al., “Dynamic perceiver for efficient vi- sual recognition,” in ICCV, 2023, pp. 5992–6002

    2023

  23. [31]

    Msnet: Multi-resolution synergis- tic networks for adaptive inference,

    Renlong Hang et al., “Msnet: Multi-resolution synergis- tic networks for adaptive inference,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 5, pp. 2009–2018, 2022

    2009

  24. [32]

    A panda? no, it’s a sloth: Slow- down attacks on adaptive multi-exit neural network in- ference,

    Sanghyun Hong et al., “A panda? no, it’s a sloth: Slow- down attacks on adaptive multi-exit neural network in- ference,” in ICLR, 2021

    2021

  25. [33]

    Not all images are worth 16x16 words: Dynamic transformers for efficient image recog- nition,

    Yulin Wang et al., “Not all images are worth 16x16 words: Dynamic transformers for efficient image recog- nition,” NeurIPS, vol. 34, pp. 11960–11973, 2021

    2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.