Pith. sign in

REVIEW 5 major objections 6 minor 88 references

SnapViT claims that a single prunability ranking of any pretrained vision transformer yields accurate sub-networks across a continuum of compute budgets, without retraining or labels, in under five minutes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:58 UTC pith:EQ724CRH

load-bearing objection Useful elastic-pruning method, but the Hessian story is not operational and the comparisons have some asymmetric baselines. the 5 major comments →

arxiv 2510.17700 v2 pith:EQ724CRH submitted 2025-10-20 cs.CV

Elastic ViTs from Pretrained Models without Retraining

Walter Simoncini , Michael Dorkenwald , Tijmen Blankevoort , Cees G.M. Snoek , Yuki M. Asano This is my paper
classification cs.CV
keywords elastic inferencestructured pruningvision transformersHessian approximationevolution strategiesself-supervised pruningpost-training pruningfoundation models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that any pretrained vision transformer can be turned into an elastic model—a family of sub-networks spanning a continuous range of compute budgets—by a single pruning step that needs no retraining, no labels, and no classification head. It introduces a prunability score that combines per-parameter gradient sensitivity with cross-block correlations learned by an evolutionary search, then ranks parameters once; thresholds of that rank give sub-networks for any sparsity. The authors claim this matches or beats state-of-the-art pruning methods that must be re-run for each target sparsity, and works on DINO, SigLIPv2, DeIT, and AugReg models. A reader would care because deployment often demands a specific latency-accuracy point, and today's foundation models come only in fixed sizes.

Core claim

The central claim is that the Hessian of the pruning objective can be decomposed into a local diagonal term—estimated from self-supervised gradients—and a global cross-block term approximated by the covariance of an exponential natural evolution strategy (xNES) search. Fusing the two into a single score P and ranking parameters once yields sub-networks at any desired sparsity; the evolutionary search is run once and every sparsity is read off the same ranking. The paper reports that this single-shot, label-free procedure matches or outperforms LAMP, LLM Surgeon, FPTP, SparseGPT, SNIP Magnitude, and NViT across sparsities from 10% to 60%, with a full run taking under five minutes on one A100

What carries the argument

The load-bearing object is the unified prunability score P = diag(local Hessian proxy) ⊙ (M c), where the local proxy is the dataset-averaged squared gradient norm of a self-supervised DINO-style loss, and c is a blockwise reweighting vector learned by xNES over functional units (feed-forward neurons and attention heads). The xNES covariance Σ is interpreted as approximating the inverse of the global inter-block Hessian—H^(g) ≈ αΣ^{-1}—so the learned block scalings are meant to encode which structures can be pruned jointly. Ranking P once produces masks for any sparsity, which is what makes the model elastic.

Load-bearing premise

The whole method rests on the assumption that the covariance learned by the evolutionary search (xNES) actually approximates the inverse Hessian of the pruning objective, so that the blockwise scalings encode true cross-block dependencies—yet the fitness is a non-quadratic cosine-similarity function and the final score uses only the single best sampled scaling vector, not the learned covariance.

What would settle it

Compute a small ViT where the true Hessian (or its block structure) of the pruning objective can be formed explicitly, run SnapViT, and compare the claimed H^(g) ≈ αΣ^{-1} to the empirically computed inverse block Hessian. If the off-diagonal patterns do not match, the correlation term is a heuristic; alternatively, compare pruning under the arg-max c against pruning under a score built from the covariance Σ directly—if the covariance adds nothing, the Hessian approximation is not doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any pretrained ViT, including headless self-supervised encoders, can be served at many compute budgets from a single pruning run.
  • Deployment pipelines no longer need to re-run a pruning method per target sparsity; one ranking covers 10–60% sparsity.
  • Pruned sub-networks retain enough representation quality that linear probing, k-NN, and linear segmentation remain usable without fine-tuning; weight correction further recovers accuracy.
  • Large supervised models (DeIT-III H/14) can be pruned to 50% with small loss, while heavily large-scale-pretrained models (SigLIPv2, DINOv3) tolerate less pruning before degrading.
  • The method's cost scales sub-linearly with model size, making it applicable to billion-parameter encoders.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the Hessian interpretation is set aside, the xNES search is still a black-box optimizer over block reweightings; the reported gains could be driven by the search itself rather than by principled curvature estimation, and a test isolating this is to compare the final score using only the arg-max c versus using the full covariance Σ^{-1}.
  • The appendix's CKA-initialization result shows the evolutionary search's starting covariance matters a lot, hinting that the learned correlations partly inherit architecture-level similarity rather than discovering true Hessian curvature.
  • The method could plausibly extend to other transformer families (language models, multimodal encoders) since the score only needs a self-supervised objective and block definitions, but that is an extrapolation beyond the paper.
  • Because fitness is evaluated on a fixed 1000-image probe set with PCA-projected cosine similarity, the ranking is implicitly tuned to that probe set; generalization to very different test distributions is not guaranteed by the paper's claims alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes SnapViT, a structured pruning method that turns any pretrained vision transformer into an elastic model: a single prunability score is computed once, then thresholded at arbitrary sparsity levels to produce sub-networks without retraining or labeled data. The score combines a diagonal, gradient-based local term computed with a DINO-style self-supervised loss and a blockwise rescaling vector c learned by xNES, which the paper interprets as an evolutionary approximation of the inverse global Hessian. Experiments on DINO, SigLIPv2, DeIT, and AugReg backbones, evaluated with k-NN, linear probing, and linear segmentation, claim state-of-the-art or competitive performance against LLM Surgeon, FPTP, NViT, SparseGPT, SNIP Magnitude, and LAMP, with runtimes claimed to be under five minutes on one A100 GPU.

Significance. If the empirical claims hold, the practical contribution is valuable: single-shot, label-free, retraining-free elastic pruning would let users derive many compute-adaptive sub-networks from existing pretrained ViTs with negligible overhead. The paper is unusually thorough on the experimental side: it provides code and pruned models, evaluates across several model families and tasks, includes large-model experiments, and reports a compute budget appendix. The ablations on CKA initialization, number of sparsities, and pruning-data choice are informative. However, the central scientific claim—that the xNES covariance approximates Hessian off-diagonal structure and that this structure drives the final score—is not supported by the algorithm as written. The final score uses only the best sampled c, not the covariance, so the method as implemented is a black-box search for blockwise reweightings. That distinction does not automatically invalidate the empirical results, but it removes the paper's stated theoretical basis for outperforming Hessian-aware baselines.

major comments (5)
  1. [§3.3, Eq. (7), Algorithm 1 (App. B.2)] The central theoretical claim is H^(g) ≈ α Σ^{-1}, but the algorithm does not use Σ in the final score. Algorithm 1 terminates with c_final = argmax_k F^(k) (line 16) and sets P = (M c_final) ⊙ s (line 17); Σ is used only inside the sampling loop and is discarded. The final score is therefore a blockwise rescaling of the local diagonal gradient score, with no off-diagonal term. Additionally, Eq. (6) is a non-quadratic fitness, so the quadratic-landscape results of [57, 1] do not apply. Unless the final score is changed to use Σ^{-1} (or μ), or the Hessian interpretation is removed, the contribution item 'evolutionary approximation of Hessian off-diagonal structures' is not supported by the method as written.
  2. [§4.1 and App. B.5] The baseline comparisons are asymmetric. LLM Surgeon is run with 5 shots instead of the recommended 40; NViT prunes 1024 structures per step instead of 32; FPTP uses 1000 samples instead of 2048; and NViT is evaluated on a different backbone (DeiT) from the AugReg backbone used for the other methods. These choices can favour the proposed method. The statement in B.5 that no significant differences were noticed for LLM Surgeon is not a substitute for sensitivity experiments. Please re-run at the recommended settings or provide ablations showing that the adjusted hyperparameters do not affect the baselines' accuracy.
  3. [Table 9 (App. C)] The statistical significance appendix reports substantial variance at high sparsity: at 50% sparsity, average k-NN std is 1.4 and linear std 1.2, with per-dataset std up to 5.3 (Pets). The main text claims improvements of 7% and 12.3% over SNIP and FPTP at 50% sparsity without error bars. These differences may be within noise. The main comparisons should include confidence intervals or error bars, or the high-sparsity claims should be tempered.
  4. [Sec. 3.3, Sec. 4.4, App. B.3] The blockwise vector c is fitted by maximizing the fitness F in Eq. (6), which directly measures post-pruning embedding cosine similarity at the same sparsity levels later used for ranking. Since the final masks are a deterministic function of c, the per-sparsity behavior is a fitted value, not an independent estimate of curvature. This does not by itself invalidate the method, but it means the 'global Hessian' label is an interpretation rather than a mechanism. At minimum, the authors should validate on held-out pruning data or compare against random/independent c to quantify overfitting; the data-alignment ablation in Fig. 11 already shows strong sensitivity to the choice of pruning data.
  5. [Abstract, Table 6, App. B.1] The abstract claims 'less than five minutes on a single A100 GPU' for generating elastic models, but Table 6 reports 11m4s for DeIT-III ViT-H/14 and 4m58s for ViT-L/16. The B.1 statement that the entire DeiT-III family is pruned in only a few minutes also conflicts with Table 6. Please correct the abstract and specify the model size for which the five-minute bound actually holds.
minor comments (6)
  1. [Sec. 2, last paragraph] 'can model intra-layer dependencies' appears to be a typo for 'inter-layer' or 'cross-block', since the method is intended to model correlations between functional blocks, not within a single layer.
  2. [Table 9] The caption says models pruned to 10, 20, and 30% sparsity, but the table columns are 10, 30, and 50%. The text should match the table.
  3. [References] References [75] and [76] are the same paper (Global Vision Transformer Pruning with Hessian-aware Saliency). One entry should be removed and citations reconciled.
  4. [Sec. 4.1] The text mentions 'DINOv2 ViT-B/16' in a comparison, but the surrounding section discusses DINOv3. Please reconcile the model name.
  5. [Fig. 2 caption] 'LLM Surgeon (5 Shot)' should be '5-shot' for consistency.
  6. [Eq. (2) and Sec. 3.2] The quantity in Eq. (2) is the squared gradient norm, not a Hessian estimate unless the model is at a stationary point. Calling this 'local Hessian' is potentially misleading; consider renaming it 'gradient-based sensitivity' or 'diagonal Fisher proxy'.

Circularity Check

0 steps flagged

No circularity: the final score is a fitted rescaling, but no prediction reduces to its input by construction; the Hessian-framing gap is a correctness issue, not circularity.

full rationale

The derivation chain is self-contained and non-circular. The local term (Eq. 2) is a standard gradient-norm diagonal proxy; the global term is learned by xNES optimizing a label-free embedding-cosine fitness (Eq. 6); the final prunability score (Eq. 8) combines them; masks (Eq. 9) are thresholded ranks. Nothing in this chain defines the output as its own input: the reported k-NN/linear accuracies are measured on held-out data, not the fitness itself, and the evolution is a direct optimization, not a prediction. The paper's main scientific weakness is that Eq. (7) (H^(g) ≈ αΣ^{-1}) is never operationalized: Algorithm 1 line 16 returns c_final = argmax_k F^(k), and line 17 builds P from c_final only, discarding Σ; hence the final score has no off-diagonal block-interaction term and the 'evolutionary approximation of Hessian off-diagonal structures' is unsupported. This is a correctness/novelty gap (and an internal-consistency issue: abstract '<5 minutes' vs Table 6's 11m4s for DeIT-III ViT-H/14), but it is not circularity: no equation reduces to another by construction, and no fitted parameter is renamed as a prediction of the quantity it was fit to. The xNES-inverse-Hessian relation is cited to external work [1,57], not to the authors' own prior results; the only self-citations ([47,48]) are for evaluation recipes and are not load-bearing. Therefore score 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The method introduces no new physical or architectural entities; it learns a per-block reweighting vector via evolutionary search, which is a fitted parameter rather than an invented entity. The main unstated assumptions concern the applicability of the DINO loss to arbitrary pretrained models and the validity of the Hessian-covariance equivalence on a non-quadratic fitness landscape.

free parameters (6)
  • blockwise rescaling vector c = learned via xNES on 1000 ImageNet-1k images
    Central to the final prunability score (Eq. 8); optimized to maximize cosine similarity (Eq. 6).
  • xNES iterations T = 50 (500 for DINOv3/SigLIPv2)
    Chosen for convergence; Table 2 shows 500 iterations improve over 50.
  • target sparsity grid S = 10/30/50/60% (main) or 10-60% six levels
    Fitness is averaged over S; main text claims six levels while B.3 lists four; choice affects optimization.
  • number of pruning-data samples N_D = 1000 from ImageNet-1k training set
    Used for both gradients and fitness; Figure 11 shows performance varies 5-6% with data choice.
  • per-block pruning caps = at most 80% heads, 95% FFN neurons; min 2 heads and 154 neurons per block
    Ad hoc constraints to avoid degenerate blocks; affect achievable sparsity distributions.
  • PCA dimension = 192
    Used in fitness (Eq. 6); chosen without stated criterion.
axioms (5)
  • standard math Second-order Taylor expansion (Eq. 1) with vanishing first-order term (model near local minimum)
    Basis for local Hessian approximation; standard in pruning literature (OBD).
  • domain assumption Evolution strategies adapt covariance to the inverse Hessian of a locally quadratic fitness landscape (Eq. 7)
    Applies to idealized convex quadratic landscapes; the actual fitness (Eq. 6) is a non-smooth, non-quadratic function of the sampled c, so the H^(g)≈Σ^{-1} claim is unsupported.
  • domain assumption Cosine similarity between PCA-compressed original and pruned embeddings faithfully proxies downstream task retention (Eq. 6)
    The entire optimization target; if the proxy fails, masks are mis-allocated even when fitness is high.
  • ad hoc to paper DINO self-supervised loss (Eq. 3) can be computed on arbitrary ViTs without a task head and yields gradients indicative of prunability
    Applied post hoc to models not trained with DINO (AugReg, DeIT, SigLIPv2); the required teacher/projection heads are unspecified; no evidence is given that the gradients are meaningful.
  • domain assumption 1000 ImageNet-1k training images are a sufficient proxy for the deployment distribution
    Figure 11 shows that alignment between pruning data and evaluation data changes accuracy by up to 6.4%; ImageNet is also one of the seven evaluation datasets.

pith-pipeline@v1.3.0-alltime-deepseek · 22671 in / 15937 out tokens · 130714 ms · 2026-08-04T08:58:26.798264+00:00 · methodology

0 comments
read the original abstract

Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: Single-shot network approximation for pruned Vision Transformers, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining-free. Experiments on DINO, SigLIPv2, DeIT, and AugReg models demonstrate superior performance over state-of-the-art methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: https://elastic.ashita.nl/

Figures

Figures reproduced from arXiv: 2510.17700 by Cees G.M. Snoek, Michael Dorkenwald, Tijmen Blankevoort, Walter Simoncini, Yuki M. Asano.

Figure 1
Figure 1. Figure 1: Overview of our pruning method. We decompose the Hessian into local and global components: the local Hessian is approximated from self-supervised gradients, while the global Hessian models cross-block correlations learned via an evolutionary algorithm. Combining both yields a unified prunability score that ranks parameters once, allowing single-shot generation of sub-networks at any desired sparsity. The p… view at source ↗
Figure 2
Figure 2. Figure 2: Our method matches or improves upon the state-of-the-art in a retraining-free setup, while not using labels. Top-1 accuracies in k-nearest neighbor and linear classification averaged across 7 datasets for supervised AugReg and DeIT ViT-B/16 models. Our label-free method outperforms or matches baselines that utilize labels, especially at high sparsity ratios. We interpret the evolving covariance as a data-d… view at source ↗
Figure 3
Figure 3. Figure 3: Our method outperforms baselines for DINO ViT-B/16. Top-1 accuracy in k-nearest neighbor and linear classification averaged across 7 datasets for models pruned with our method, LAMP, and SNIP Magnitude. Our method can prune DINO to 40% sparsity with an accuracy degradation under 5%. 35.1 31.8 28.3 24.9 21.3 GFLOPs 30 40 50 60 70 80 Avg. Accuracy k-NN DINOv3 ViT-B/16 35.1 31.8 28.3 24.9 21.3 GFLOPs 20 30 40… view at source ↗
Figure 4
Figure 4. Figure 4: Large-scale pretraining complicates pruning. Top-1 accuracy in k-nearest neighbor and linear classification for pruned DINOv3 [59] and SigLIPv2 ViT-B/16 [66] models. We find that self-supervised models trained on large datasets are harder to prune, benefit from longer optimization horizons, and optimizing for more sparsities. where Θ is the complete parameter set and Pi denotes the score of parameter θi . … view at source ↗
Figure 5
Figure 5. Figure 5: Weight correction helps retaining post-pruning performance. A single SparseGPT￾style weight correction step greatly improves performance at high sparsity levels while preserving efficiency. Our method matches or surpasses state-of-the-art baselines across pruning ratios and preserves self-supervised model accuracy even under extreme sparsity (bottom row). 4.1 Single-Shot Structured Pruning We evaluate our … view at source ↗
Figure 6
Figure 6. Figure 6: Our method retains segmentation performance best. We report the mIoU on Pascal VOC 2012 for linear semantic segmentation for AugReg and DeIT ViT-B/16 models. 599.0 540.8 481.0 421.2 361.3 301.5 241.6 GFLOPs 30 40 50 60 70 80 90 Avg. Accuracy SigLIP 2 ViT-G/16 k-NN Ours (SSL) 599.0 540.8 481.0 421.2 361.3 301.5 241.6 GFLOPs 20 30 40 50 60 70 80 90 SigLIP 2 ViT-G/16 Linear 337.3 300.1 266.5 232.8 199.1 165.4… view at source ↗
Figure 7
Figure 7. Figure 7: Huge models are not necessarily more prunable. Top-1 accuracy in k-nearest neighbor and linear classification averaged across 7 datasets for DINOv3 ViT-H+/16 and SigLIPv2 ViT-G/16 models pruned with our method. Contrary to a DeIT-III ViT-H/14, shown in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Deeper blocks are heavily pruned in DINO ViT-B/16: visualization of the normalized parameter allocation for DINO ViT-B/16 models pruned at increasing sparsity levels (0-60%). Feed￾forward blocks, especially deep ones (8-12), are pruned before attention heads, while earlier blocks maintain their parameter density, revealing the network’s inherent structural redundancy. its k-nearest neighbor and linear clas… view at source ↗
Figure 9
Figure 9. Figure 9: Ablations for the number of target sparsities and loss. Average accuracy in k-nearest neighbor classification for AugReg ViT-B/16 models pruned by optimizing for 1 to 6 sparsities (left), and for models pruned using gradients from either a self-supervised or cross-entropy loss (right). (B = 156). The results show that the performance of pruned models increases as more global interactions are modeled, by up… view at source ↗
Figure 10
Figure 10. Figure 10: Initializing Σ using CKA scores improves performance for SigLIPv2 ViT-B/16. The genetic algorithm loss over 250 steps, smoothed using an exponential moving average with w = 0.95, when Σ is initialized either with an identity matrix or the CKA scores between structures, plus the model performance when pruned to up to 30% sparsity after 50, 100 and 250 genetic algorithm steps. pair in k-nearest neighbor and… view at source ↗
Figure 11
Figure 11. Figure 11: Alignment between pruning data and target tasks improves performance. Average top-1 accuracy in k-nearest neighbor and linear classification of DINO ViT-B/16 models pruned using our method and data from ImageNet-1k, DiffusioDB, Shaders 21k, and a 1000-samples dataset built by sampling equally from the training set of each of the evaluation datasets, named “Merged Data”. [32] score between the activations … view at source ↗
Figure 12
Figure 12. Figure 12: Larger supervised models can be pruned more aggressively. Top-1 accuracy in k￾nearest neighbor and linear classification, averaged across 7 datasets, for models of various sizes belonging to the DeIT-III [65] family, pruned to up to 60% sparsity using our method with and without weight correction. Weight correction consistently improves performance for ViT-B/16 and larger models, by up to 25.4% for a ViT-… view at source ↗
Figure 13
Figure 13. Figure 13: Our method preserves the quality of dense representations. Visualization of seg￾mentation masks for Pascal VOC 2012 produced via in-context semantic segmentation using DINO ViT-B/16 models pruned from 0 to 60% sparsity using our method (top rows) and SNIP Magnitude (bottom rows). The ground truth is shown in the left-most image. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

88 extracted references · 16 linked inside Pith

  1. [1]

    Bidirectional relation between cma-es and nes

    Youhei Akimoto, Yu Nagata, Isao Ono, and Shigenobu Kobayashi. Bidirectional relation between cma-es and nes. InInternational Conference on Parallel Problem Solving from Nature (PPSN XI), 2010

    2010

  2. [2]

    Structured pruning of deep convolutional neural networks.ACM J

    Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks.ACM J. Emerg. Technol. Comput. Syst., 13(3):32:1–32:18, 2017

    2017

  3. [3]

    Towards in-context scene understanding.Advances in Neural Information Processing Systems, 36, 2024

    Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelovi´c, and Olivier Henaff. Towards in-context scene understanding.Advances in Neural Information Processing Systems, 36, 2024

    2024

  4. [4]

    Procedural image programs for representation learning.Advances in Neural Information Processing Systems, 35:6450–6462, 2022

    Manel Baradad, Richard Chen, Jonas Wulff, Tongzhou Wang, Rogerio Feris, Antonio Torralba, and Phillip Isola. Procedural image programs for representation learning.Advances in Neural Information Processing Systems, 35:6450–6462, 2022

    2022

  5. [5]

    Once-for-all: Train one network and specialize it for efficient deployment.arXiv preprint arXiv:1908.09791, 2019

    Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment.arXiv preprint arXiv:1908.09791, 2019

    Pith/arXiv arXiv 1908

  6. [6]

    Network augmentation for tiny deep learning

    Han Cai, Chuang Gan, Ji Lin, and Song Han. Network augmentation for tiny deep learning. arXiv preprint arXiv:2110.08890, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. CoRR, abs/2405.17430, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Pruning convolutional neural networks with self-supervision.arXiv preprint arXiv:2001.03554, 2020

    Mathilde Caron, Ari Morcos, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Pruning convolutional neural networks with self-supervision.arXiv preprint arXiv:2001.03554, 2020

    Pith/arXiv arXiv 2001

  9. [9]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660, 2021

    2021

  10. [10]

    Transformer flops

    Adam Casson. Transformer flops. https://www.adamcasson.com/posts/ transformer-flops, 2021. Accessed: 2025-09-24

    2021

  11. [11]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InCVPR, 2014

    2014

  12. [12]

    Scaling vision transformers to 22 billion parameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. InInternational Conference on Machine Learning, pp. 7480–7512. PMLR, 2023

    2023

  13. [13]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009

    2009

  14. [14]

    Dhillon, Yulia Tsvetkov, Hanna Hajishirzi, Sham M

    Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit S. Dhillon, Yulia Tsvetkov, Hanna Hajishirzi, Sham M. Kakade, Ali Farhadi, and Prateek Jain. Matformer: Nested transformer for elastic inference. InNeurIPS, 2024

    2024

  15. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 11

    2021

  16. [16]

    Susskind, and Armand Joulin

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Ángel Bautista, Vaishaal Shankar, Alexander T Toshev, Joshua M. Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. InProceedings of the 41st International Conference on Machine Learning, pp. 12371–12384, 2024

    2024

  17. [17]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge.IJCV, 2010

    2010

  18. [18]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning (ICML), 2023

    2023

  19. [19]

    Exponential natural evolution strategies

    Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and Jürgen Schmidhuber. Exponential natural evolution strategies. InProceedings of the 12th annual conference on Genetic and evolutionary computation, pp. 393–400, 2010

    2010

  20. [20]

    Dynamic convnets on tiny devices via nested sparsity.IEEE Internet of Things Journal, 10(6):5073–5082, 2022

    Matteo Grimaldi, Luca Mocerino, Antonio Cipolletta, and Andrea Calimera. Dynamic convnets on tiny devices via nested sparsity.IEEE Internet of Things Journal, 10(6):5073–5082, 2022

    2022

  21. [21]

    Susskind, and Navdeep Jaitly

    Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, and Navdeep Jaitly. Matryoshka diffusion models. InICLR. OpenReview.net, 2024

    2024

  22. [22]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. pp. 1135–1143, 2015

    2015

  23. [23]

    Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. InNIPS, pp. 1135–1143, 2015

    2015

  24. [24]

    Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. InNIPS, pp. 164–171. Morgan Kaufmann, 1992

    1992

  25. [25]

    Structured pruning for deep convolutional neural networks: A survey

    Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 46(5):2900–2919, 2024

    2024

  26. [26]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.JST-AEORS, 2019

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.JST-AEORS, 2019

    2019

  27. [27]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. 2015

    2015

  28. [28]

    Rae, and Erich Elsen

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Jacob Menick, Rewon Child, Adhiguna Kuncoro, Tomás Kociský, Phil Blunsom, Chris Dyer, Oriol Vinyals, Jack W. Rae, and Erich Elsen. Training compute-optimal large language...

    2022

  29. [29]

    Dynabert: Dynamic bert with adaptive width and depth.Advances in Neural Information Processing Systems, 33: 9782–9793, 2020

    Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth.Advances in Neural Information Processing Systems, 33: 9782–9793, 2020

    2020

  30. [30]

    Matryoshka query transformer for large vision-language models

    Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. InNeurIPS, 2024

    2024

  31. [31]

    Single-shot pruning for pre-trained models: Rethinking the importance of magnitude pruning

    Hirokazu Kohama, Hiroaki Minoura, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Single-shot pruning for pre-trained models: Rethinking the importance of magnitude pruning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1433–1442, 2023

    2023

  32. [32]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. InInternational conference on machine learning, pp. 3519–3529. PMLR, 2019. 12

    2019

  33. [33]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  34. [34]

    Kakade, Prateek Jain, and Ali Farhadi

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham M. Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. InNeurIPS, 2022

    2022

  35. [35]

    Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami

    Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. InNeurIPS, 2022

    2022

  36. [36]

    A fast post-training pruning framework for transformers.Advances in Neural Information Processing Systems, 35:24101–24116, 2022

    Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers.Advances in Neural Information Processing Systems, 35:24101–24116, 2022

    2022

  37. [37]

    Optimal brain damage.Advances in neural information processing systems, 2, 1989

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage.Advances in neural information processing systems, 2, 1989

    1989

  38. [38]

    Layer-adaptive sparsity for the magnitude-based pruning

    Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. InICLR. OpenReview.net, 2021

    2021

  39. [39]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. 2017

    2017

  40. [40]

    Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    Pith/arXiv arXiv 2016

  41. [41]

    S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013

    2013

  42. [42]

    Importance estimation for neural network pruning, 2019

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning, 2019. URLhttps://arxiv.org/abs/1906.10771

    Pith/arXiv arXiv 2019

  43. [43]

    Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian

    Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. InNeurIPS, pp. 4933–4943, 2019

    2019

  44. [44]

    Exploring sparsity in recurrent neural networks

    Sharan Narang, Greg Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. InICLR (Poster). OpenReview.net, 2017

    2017

  45. [45]

    Sosp: Efficiently capturing global correlations by second-order structured pruning

    Max Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and Dominik Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. InInternational Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id= H1B8bA9tvr

    2022

  46. [46]

    Stitchable neural networks

    Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitchable neural networks. InCVPR, 2023

    2023

  47. [47]

    Hummingbird evaluation for vision encoders, 4 2024

    Valentinos Pariza, Mohammadreza Salehi, and Yuki Asano. Hummingbird evaluation for vision encoders, 4 2024. URLhttps://github.com/vpariza/open-hummingbird-eval

    2024

  48. [48]

    Near, far: Patch-ordering enhances vision foundation models’ scene understand- ing

    Valentinos Pariza, Mohammadreza Salehi, Gertjan J Burghouts, Francesco Locatello, and Yuki M Asano. Near, far: Patch-ordering enhances vision foundation models’ scene understand- ing. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  49. [49]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012

    2012

  50. [50]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.JMLR, 2011

    2011

  51. [51]

    Design- ing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10428–10436, 2020. 13

    2020

  52. [52]

    Xnor-net: Imagenet classification using binary convolutional neural networks

    Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. pp. 525–542. Springer, 2016

    2016

  53. [53]

    Scaling vision with sparse mixture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595, 2021

    2021

  54. [54]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022

    2022

  55. [55]

    Imagenet large scale visual recognition challenge.IJCV, 2015

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.IJCV, 2015

    2015

  56. [56]

    Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. InNeurIPS, 2020

    2020

  57. [57]

    Shir and Amir Yehudayoff

    Ofer M. Shir and Amir Yehudayoff. Covariance matrix adaptation and inverse hessian on convex quadratic functions. InProceedings of the 15th ACM/SIGEVO Conference on F oundations of Genetic Algorithms (FOGA), 2019

    2019

  58. [58]

    Matmamba: A matryoshka state space model.CoRR, abs/2410.06718, 2024

    Abhinav Shukla, Sai Vemprala, Aditya Kusupati, and Ashish Kapoor. Matmamba: A matryoshka state space model.CoRR, abs/2410.06718, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Oriane Siméoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    Girshick, Rohit Girdhar, and Ishan Misra

    Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross B. Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of MAE pre-pretraining for billion-scale pretraining. In ICCV, pp. 5461–5471. IEEE, 2023

    2023

  61. [61]

    How to train your vit? data, augmentation, and regularization in vision transformers.arXiv preprint arXiv:2106.10270, 2021

    Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers.arXiv preprint arXiv:2106.10270, 2021

    Pith/arXiv arXiv 2021

  62. [62]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. InICLR. OpenReview.net, 2024

    2024

  63. [63]

    Faster gaze prediction with dense networks and fisher pruning.CoRR, abs/1801.05787, 2018

    Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning.CoRR, abs/1801.05787, 2018

    Pith/arXiv arXiv 2018

  64. [64]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347–10357. PMLR, 2021

    2021

  65. [65]

    Deit iii: Revenge of the vit

    Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. InEuropean conference on computer vision, pp. 516–533. Springer, 2022

    2022

  66. [66]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen, and Tijmen Blankevoort. The LLM surgeon. InICLR. OpenReview.net, 2024

    2024

  68. [68]

    EigenDamage: Struc- tured pruning in the Kronecker-factored eigenbasis

    Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. EigenDamage: Struc- tured pruning in the Kronecker-factored eigenbasis. InProceedings of the 36th Interna- tional Conference on Machine Learning, volume 97, pp. 6566–6575. PMLR, 2019. URL http://proceedings.mlr.press/v97/wang19g.html. 14

    2019

  69. [69]

    Eigendamage: Structured pruning in the kronecker-factored eigenbasis

    Chaoqi Wang, Guodong Zhang, Shunhua Fidler, and Roger Grosse. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. InInternational Conference on Machine Learning (ICML), 2019. URLhttps://proceedings.mlr.press/v97/wang19e.html

    2019

  70. [70]

    Hat: Hardware-aware transformers for efficient natural language processing.arXiv preprint arXiv:2005.14187, 2020

    Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing.arXiv preprint arXiv:2005.14187, 2020

    Pith/arXiv arXiv 2005

  71. [71]

    Structured probabilistic pruning for convolutional neural network acceleration

    Huan Wang, Qiming Zhang, Yuehai Wang, and Haoji Hu. Structured probabilistic pruning for convolutional neural network acceleration. InBMVC, pp. 149. BMV A Press, 2018

    2018

  72. [72]

    Diffusiondb: A large-scale prompt gallery dataset for text-to-image genera- tive models.arXiv preprint arXiv:2210.14896, 2022

    Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image genera- tive models.arXiv preprint arXiv:2210.14896, 2022

    Pith/arXiv arXiv 2022

  73. [73]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019

    2019

  74. [74]

    Lpvit: Low-power semi-structured pruning for vision transformers

    Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Xulei Yang, Min Wu, Xiaoli Li, and Weisi Lin. Lpvit: Low-power semi-structured pruning for vision transformers. InEuropean Conference on Computer Vision, pp. 269–287. Springer, 2024

    2024

  75. [75]

    Global vision transformer pruning with hessian-aware saliency

    Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18547–18557, June 2023

    2023

  76. [76]

    Global vision transformer pruning with hessian-aware saliency

    Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18547–18557, 2023

    2023

  77. [77]

    Gradient-based intra-attention pruning on pre-trained language models

    Ziqing Yang, Yiming Cui, Xin Yao, and Shijin Wang. Gradient-based intra-attention pruning on pre-trained language models. InACL (1), pp. 2775–2790. Association for Computational Linguistics, 2023

    2023

  78. [78]

    Universally slimmable networks and improved training techniques

    Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. InProceedings of the IEEE/CVF international conference on computer vision, pp. 1803–1811, 2019

    2019

  79. [79]

    Slimmable neural networks.arXiv preprint arXiv:1812.08928, 2018

    Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks.arXiv preprint arXiv:1812.08928, 2018

    Pith/arXiv arXiv 2018

  80. [80]

    Fit: A metric for model sensitivity, 2022

    Ben Zandonati, Adrian Alan Pol, Maurizio Pierini, Olya Sirkin, and Tal Kopetz. Fit: A metric for model sensitivity, 2022

    2022

Showing first 80 references.