Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Augmenting Swin Transformer's shifted-window attention with a small learnable convolutional branch improves accuracy on five small-scale image benchmarks, by 0.10 to 4.92 points over the Swin baseline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:53 UTC pith:BOLNHWFI

load-bearing objection CoSwin is a clean but minor Swin-plus-convolution hybrid whose reported gains lack the parameter-matched controls and reproducibility detail to support its strong claims. the 4 major comments →

arxiv 2509.08959 v1 pith:BOLNHWFI submitted 2025-09-10 cs.CV

CoSwin: Convolution Enhanced Hierarchical Shifted Window Attention For Small-Scale Vision

Puskal Khadka , Rodrigue Rizk , Longwei Wang , KC Santosh This is my paper
classification cs.CV
keywords vision transformershifted window attentionconvolution-attention fusionsmall-scale image classificationinductive biaslearnable fusion weightCIFAR-100Tiny ImageNet
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CoSwin is an attempt to fix a known weakness of vision transformers on small datasets: self-attention models global dependencies but lacks the locality and translation equivariance that convolutions encode. The paper's central claim is that adding a small learnable convolutional branch—two 3×3 convolutions applied to the patch tokens reshaped as an image, scaled by a learned weight—into every Swin attention block gives consistent top-1 accuracy gains on five benchmarks, with the largest jumps on CIFAR-100 (+4.92 points) and Tiny ImageNet (+4.47 points). If true, this would mean windowed transformers can become data-efficient without abandoning the attention mechanism, simply by restoring inductive biases through a cheap feature-fusion side branch. The paper supports the claim with from-scratch training under fixed conditions, an ablation showing both the conv layers and the learnable fusion weight matter, and Grad-CAM visualizations showing sharper object focus.

Core claim

CoSwin extends the Swin Transformer stage by stage: after window-based multi-head self-attention, it reshapes the sequence of patch tokens into a pseudo-image (√N×√N×d_embed), applies a two-layer 3×3 convolution block (first conv expands channels by 10%, ReLU, second projects back), multiplies the result by a learnable scalar Γ, and adds it element-wise to the attention output. The paper reports top-1 validation accuracy of 96.63% on CIFAR-10, 81.64% on CIFAR-100, 99.60% on MNIST, 98.07% on SVHN, and 65.06% on Tiny ImageNet, beating the Swin baseline by 2.17, 4.92, 0.10, 0.26, and 4.47 percentage points respectively. The authors conclude that the local-global fusion restores key inductive bi

What carries the argument

The central mechanism is CoSwin-MSA: weighted fusion of shifted-window self-attention and a learnable local feature branch. A patch-conversion layer reshapes token embeddings into pseudo-image form so ordinary 3×3 convolutions can run; a two-layer convolution block extracts local features; a learnable scalar Γ controls how much of those local features are added to the attention scores. The learnable Γ is what allows the network to favor global attention on some inputs and local detail on others, and the paper's ablation links the performance gain to the presence of both convolutions and this weight.

Load-bearing premise

The load-bearing premise is that running every model with the same hyperparameters, augmentation, and epoch count is a fair comparison; if the extra convolutional parameters or an undertuned Swin baseline account for most of the accuracy gap, the claimed benefit of the local-feature fusion is not established.

What would settle it

Train Swin and CoSwin again with matched parameter counts and with learning rate, weight decay, and augmentation strength tuned separately per model on a validation split; if the top-1 gap falls below the reported 0.10–4.92 points (or reverses on some datasets), the paper's attribution of the gain to the weighted convolution branch is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the reported results hold, from-scratch training of a Swin-style transformer on 32×32 and 64×64 benchmarks improves by adding a two-convolution local branch at each block.
  • The gap is larger on harder class-heavy or larger-resolution benchmarks (CIFAR-100, Tiny ImageNet) than on near-saturated digit datasets (MNIST, SVHN), so the mechanism is most valuable where attention alone still leaves accuracy on the table.
  • Removing the entire convolutional branch reproduces roughly the Swin baseline (drops of 2.22, 4.92, 0.10, 0.26, 4.47 points across datasets), identifying the branch, not the training recipe, as the source of the gains in the paper's setup.
  • Removing the learnable fusion weight also hurts, which means the adaptive balance between local and global features—not just the extra parameters—is doing work in the reported comparisons.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: Because the baselines are not parameter-matched and use the same hyperparameters, the cleanest way to isolate the inductive-bias effect is to give the Swin baseline the additional parameters CoSwin introduces; if the gap persists, the conv branch's structure is the cause.
  • Inference: The recipe is not tied to windowed attention. Applying a similar weighted convolution branch to other token-based transformers would test whether the benefit comes from local feature fusion generally or from the specific interaction with shifted windows.
  • Inference: The paper's Grad-CAM evidence is illustrative; a quantitative testable extension is to measure whether CoSwin's sharper attention translates to better object localization or calibration under distribution shift.
  • Inference: Monitoring the learned Γ over training and across stages could reveal whether the network relies more on convolution early (high resolution) and more on attention later (low resolution), which would give a principled design rule for hybrid backbones.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoSwin, a Swin Transformer variant in which each attention block additionally computes a two-layer 3x3 convolutional feature map over reshaped token representations and fuses it into the shifted-window multi-head attention output, scaled by a learnable scalar Gamma. The architecture is evaluated on CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet, and the paper reports consistent accuracy gains over Swin and other baselines, plus an ablation study, Grad-CAM visualizations, and an appendix titled 'Theoretical Analysis'.

Significance. If the reported gains are robust and attributable to the proposed fusion mechanism, CoSwin is a simple and potentially useful modification for training vision transformers on small-scale datasets. The paper covers five datasets and releases code and weights, which is a practical strength. However, the current evidence does not yet establish the central attribution claim: the comparisons in Table I lack parameter/FLOPs accounting, the baselines are all trained under hyperparameters that may favor the proposed model, and all results are single-run accuracy numbers with no variance information. The appendix's theoretical statements are qualitative and not proven. The central idea is plausible, but the evidence in its present form does not support the strength of the claims made in the abstract.

major comments (4)
  1. [Section IV-C, Table I; Section III-B, Eqs. (3)-(4)] The central claim is that CoSwin outperforms Swin because of local-global feature fusion. But CoSwin inserts two extra 3x3 convolutional layers per block, with a 10% channel expansion in the first convolution. These add parameters. Table I reports only accuracy, with no parameter counts or FLOPs. A parameter-matched Swin baseline (e.g., a wider Swin with comparable parameter count, or a Swin with an analogous number of extra parameters in its MLP) is needed to rule out the alternative explanation that the gains come from added capacity rather than from the fusion mechanism. Please report parameter counts/FLOPs and add a matched-capacity baseline, and provide multiple-seed mean/std for at least the main comparisons.
  2. [Section IV-B] The statement that 'all models were trained under identical conditions' is not by itself evidence of a fair comparison. The hyperparameters listed (lr=1e-3, weight decay 0.05, MixUp/CutMix/AutoAugment, stochastic depth 0.1, 100 epochs) appear to be selected for CoSwin; no evidence is given that they are suitable for the Swin, DeiT, ViT, or CNN baselines. An undertuned Swin baseline would inflate the reported gains. Please include a hyperparameter sensitivity check for the main baseline, or use each model's recommended training recipe and report that recipe, and reconcile with the 'identical conditions' claim.
  3. [Appendix, Section B, Eq. (13); Section C, Eq. (14)] The appendix is titled 'Theoretical Analysis' but does not provide a proof. Equation (13) is simply the chain rule for the proposed architecture, and the claim that the convolutional branch 'regularizes the overall gradient magnitude' is asserted rather than derived. Similarly, the bias-variance discussion in Eq. (14) asserts that shifted-window attention reduces bias and convolution reduces variance, and that Gamma balances them, without formal justification. The statement that convolutional operators have 'well-conditioned, bounded gradients' is not generally true for ReLU networks. This appendix should be either removed or replaced with a mathematically substantive analysis; as written, it overstates what is established.
  4. [Table III, Section IV-E] The ablation table's row labels are ambiguous. Row b is described in the text as the configuration without the trainable weight, but the table header does not state which component is absent/active in each row, and row d is labeled 'Elementwise Add' even though the proposed final model uses a learnable weight (Eq. 4). It is also unclear whether row a is exactly the baseline Swin (it matches the Swin numbers in Table I). Please provide a clear legend for rows a-d, state explicitly which of Conv-1, Conv-2, and Gamma are present in each row, and discuss whether the observed differences are within run-to-run noise.
minor comments (5)
  1. [Section II-C] The paragraph beginning 'Our proposed model builds upon the architectural foundations...' appears twice, verbatim. One copy should be removed.
  2. [Fig. 3 caption] The caption repeats '(Top Left)' for the third subplot and uses '(Top Left)' again for the first; the intended layout (Top Left/Top Middle/Top Right, Bottom Left/Bottom Middle/Bottom Right) should be stated correctly.
  3. [Notation throughout] Use consistent capitalization: 'CoSwin' and 'coswin' are mixed; the dataset is sometimes 'T-ImageNet' and sometimes 'Tiny ImageNet'. In Eq. (1), the symbol P_i is not defined (the patch size is introduced as P).
  4. [Section IV-C, paragraph 1] The text says 'we used the small variant of each transformers' but does not specify model dimensions (e.g., Swin-T? ViT-S? DeiT-T?). Table I labels are also abbreviated without a key (e.g., 'SwinT' vs 'Swin'). Please define all model variants precisely.
  5. [Section IV-D] The Grad-CAM discussion in Fig. 4 is informal. For transformer models, Grad-CAM is not the standard visualization of attention; if the figure shows attention maps, describe the generation method, the layer used, and whether the maps are averaged over heads. Also mention whether the same random seed/image was used across models.

Circularity Check

0 steps flagged

No significant circularity: empirical gains are direct measurements; no fitted input is relabeled as a prediction.

full rationale

CoSwin’s central claims are empirical: the reported accuracy gains over Swin (2.17% CIFAR-10, 4.92% CIFAR-100, 0.10% MNIST, 0.26% SVHN, 4.47% Tiny ImageNet) are arithmetic differences from Table I, not outputs of a fitted model or of a self-cited theorem. The learnable scalar Γ (Eq. 4) is trained end-to-end on the same classification objective, and no quantity is fitted to a subset and then ‘predicted’ on a closely related set. The paper contains no self-citations, so no load-bearing self-citation chain exists. The only passage that could resemble circularity is the appendix ‘Theoretical Analysis’ (Eqs. 12–14), which defines H_CoSwin as the proposed fusion and then asserts improved generalization, variance reduction and sample efficiency without proof. That is an omitted proof / hand-waving, not a circular reduction: the definition does not contain the conclusion, and the empirical results stand independently of the appendix. Concerns about missing parameter-matched baselines and shared hyperparameters (Sec. IV-B/C) are experimental fairness issues, not circularity. Accordingly, the derivation chain is not circular.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The ledger lists one learned scalar and one hand-chosen ratio as free parameters, and four assumptions that the central empirical claim implicitly relies on, including the fairness of the baseline comparison and the qualitative theoretical arguments in the appendix.

free parameters (2)
  • Gamma (fusion weight) = learned per block, not reported
    Introduced in Eq. 4 as a learnable scalar to weight the convolutional feature map; learned during training, not fixed by theory.
  • Conv1 channel expansion ratio = 10%
    The first convolution expands channels by 10% (Section III-B); chosen by hand, no sensitivity analysis is provided.
axioms (4)
  • domain assumption Shifted window attention with relative positional bias (Swin Transformer) works as described in [17]
    The entire method builds on Swin's windowed attention; the paper adopts it without re-derivation.
  • domain assumption Convolutional operators have well-conditioned, bounded gradients
    Assumed in Appendix B to argue training stability; no formal proof or empirical measurement is provided.
  • domain assumption Convolution reduces variance and attention reduces bias in small-scale vision
    Stated in Appendix C, but the bias-variance decomposition is not connected to the actual architecture's learned functions.
  • domain assumption The 'identical conditions' training recipe is appropriate for all baselines
    The paper uses the same augmentation and hyperparameters for each model, assuming this is fair, though architectures often need different tuning.

pith-pipeline@v1.3.0-alltime-deepseek · 13266 in / 11335 out tokens · 108026 ms · 2026-08-04T19:53:33.074788+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of CoSwin: Convolution Enhanced Hierarchical Shifted Window Attention For Small-Scale Vision." pith.science (2026). https://pith.science/paper/BOLNHWFI

@misc{pith2026250908959,
  author       = {Pith},
  title        = {Pith review of: CoSwin: Convolution Enhanced Hierarchical Shifted Window Attention For Small-Scale Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BOLNHWFI}},
  note         = {Machine review of arXiv:2509.08959}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision Transformers (ViTs) have achieved impressive results in computer vision by leveraging self-attention to model long-range dependencies. However, their emphasis on global context often comes at the expense of local feature extraction in small datasets, particularly due to the lack of key inductive biases such as locality and translation equivariance. To mitigate this, we propose CoSwin, a novel feature-fusion architecture that augments the hierarchical shifted window attention with localized convolutional feature learning. Specifically, CoSwin integrates a learnable local feature enhancement module into each attention block, enabling the model to simultaneously capture fine-grained spatial details and global semantic structure. We evaluate CoSwin on multiple image classification benchmarks including CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet. Our experimental results show consistent performance gains over state-of-the-art convolutional and transformer-based models. Notably, CoSwin achieves improvements of 2.17% on CIFAR-10, 4.92% on CIFAR-100, 0.10% on MNIST, 0.26% on SVHN, and 4.47% on Tiny ImageNet over the baseline Swin Transformer. These improvements underscore the effectiveness of local-global feature fusion in enhancing the generalization and robustness of transformers for small-scale vision. Code and pretrained weights available at https://github.com/puskal-khadka/coswin

Figures

Figures reproduced from arXiv: 2509.08959 by KC Santosh, Longwei Wang, Puskal Khadka, Rodrigue Rizk.

Figure 1
Figure 1. Figure 1: (Top) Proposed CoSwin architecture. The input image is split into non-overlapping patches, embedded linearly, and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of Convolution Blocks in Local Feature Enhancement Module. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Top-1 Accuracy Visualization over 100 epochs. (Top Left) on CIFAR-10 dataset. (Top Middle) on CIFAR-100 dataset. (Top [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of attention score using Grad-CAM. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mechanistic Interpretability of LLM Jailbreaks via Internal Attribution Graphs

    cs.CR 2026-07 conditional novelty 6.0

    On Llama-2-7B, path-rerouting magnitude in paired transcoder attribution graphs correlates with jailbreak success (r=0.461), while static node metrics and top-feature ablations do not.

  2. Explainable Novel Category Discovery in Semantic Concept Space

    cs.CV 2026-07 conditional novelty 6.0

    xNCD routes novel category discovery through a CLIP-aligned concept bottleneck, matching strong NCD baselines while producing intrinsic cluster- and instance-level concept explanations.

  3. On What We Can Learn from Low-Resolution Data

    cs.LG 2026-05 unverdicted novelty 6.0

    Low-resolution data improves high-resolution model performance when high-resolution samples are limited, via KL-divergence bounds and experiments on vision transformers and CNNs.

  4. Learning to Transmit: Volatility-Aware Predictive Communication for Energy-Efficient IoT Networks

    cs.IT 2026-07 conditional novelty 5.0

    Sensors using volatility-aware studentized residuals plus RLS online adaptation transmit up to 94.7% less IoT data while keeping reconstruction MAE at 0.35°C.

Reference graph

Works this paper leans on

46 extracted references · 4 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Gradient-based learning applied to document recognition,

    Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”In Proceedings of the IEEE, p. 2278–2324, 1998

    1998

  2. [2]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,”In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788, 2016

    2016

  3. [3]

    Fully convolutional net- works for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional net- works for semantic segmentation,”In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431– 3440, 2015

    2015

  4. [4]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016

    2016

  5. [5]

    Densely connected convolutional networks

    G. Huang, Z. Liu, L. V. D. Maaten, and K. Q. Weinberger, “Densely connected convolutional networks.”In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4700–4708, 2017

    2017

  6. [6]

    Xception: Deep learning with depthwise separable convolutions,

    F. Chollet, “Xception: Deep learning with depthwise separable convolutions,”In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258, 2017

    2017

  7. [7]

    Aggregated resid- ual transformations for deep neural networks,

    S. Xie, R. Girshick, P. Doll ´ar, Z. Tu, and K. He, “Aggregated resid- ual transformations for deep neural networks,”In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500, 2017

    2017

  8. [8]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Łukasz Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in Neural Information Processing Systems, p. 5998–6008, 2017

    2017

  9. [9]

    Bert: Pre- training of deep bidirectional transformers for language un- derstanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language un- derstanding,”In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 4171–4186, 2019

    2019

  10. [10]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, and et al, “Language models are few-shot learners,”In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1877 – 1901, 2020

    1901

  11. [11]

    Roberta: A robustly optimized bert pretraining approach,

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, and et al, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint, vol. abs/1907.11692, 2019

    Pith/arXiv arXiv 1907

  12. [12]

    Atten- tion augmented convolutional networks

    I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Atten- tion augmented convolutional networks.”In Proceedings of the IEEE/CVF Conference on Computer Vision, pp. 3286–3295, 2019

    2019

  13. [13]

    Exploring self-attention for image recognition,

    H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,”In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10 076–1085, 2020

    2020

  14. [14]

    Bottleneck transformers for visual recognition,

    A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani, “Bottleneck transformers for visual recognition,” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16 519–16 529, 2021

    2021

  15. [15]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,

    W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,”In Proceedings of the IEEE/CVF international Conference on Computer Vision, pp. 568–578, 2021

    2021

  16. [16]

    An image is 11 worth 16x16 words: Transformers for image recognition atscale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is 11 worth 16x16 words: Transformers for image recognition atscale,”In International Con- ference on Learning Representations, 2021

    2021

  17. [17]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,”In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10012–10022, 2021

    2021

  18. [18]

    Transformers in vision: A survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,”ACM Computing Surveys, 2022

    2022

  19. [19]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,”Tech Report, Department of Computer Science, University of Toronto, 2009

    2009

  20. [20]

    Mnist hand- written digit database,

    Y. LeCun, C. Cortes, and C. J. Burges, “Mnist hand- written digit database,”ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, vol. 2, 2010

    2010

  21. [21]

    Reading digits in natural images with unsupervised fea- ture learning,

    Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised fea- ture learning,”NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011

    2011

  22. [22]

    Imagenet classifi- cation with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classifi- cation with deep convolutional neural networks,”In Advances in neural information processing systems, p. 1097–1105, 2012

    2012

  23. [23]

    Very deep convolutional net- works for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional net- works for large-scale image recognition,”In International Confer- ence on Learning Representations, 2015

    2015

  24. [24]

    Multi-scale context aggregation by dilated convolutions,

    F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,”In International Conference on Learning Representa- tions, 2016

    2016

  25. [25]

    Multiscale vision transformers,

    H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, “Multiscale vision transformers,”In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6804–6815, 2021

    2021

  26. [26]

    Mobilevit: Light-weight, general- purpose, and mobile-friendly vision transformer,

    S. Mehta and M. Rastegari, “Mobilevit: Light-weight, general- purpose, and mobile-friendly vision transformer,”In Proceedings of the International Conference on Learning Representations, 2022

    2022

  27. [27]

    Flexivit: One model for all patch sizes,

    L. Beyer, P. Izmailov, A. Kolesnikov, M. Caron, X. Z. Simon Ko- rnblith, and et al, “Flexivit: One model for all patch sizes,”In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14 496–14 506, 2023

    2023

  28. [28]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet,

    L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z. Jiang, and et al, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” inProceedings of the IEEE/CVF international confer- ence on computer vision, 2021, pp. 558–567

    2021

  29. [29]

    Gradient- based learning applied to document recognition,

    Y. LeCun, L. Bottou, Y. Bengio, and Patrick´Haffner, “Gradient- based learning applied to document recognition,”In Proceedings the IEEE, pp. 2278–2324, 1998

    1998

  30. [30]

    Training data-efficient image transformers & distil- lation through attention,

    H. Touvron, M. D. Matthieu Cord, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distil- lation through attention,”In Proceedings of the 38th International Conference on Machine Learning, 2021

    2021

  31. [31]

    Patchrot: Self- supervised training of vision transformers by rotation predic- tion,

    S. Chhabra, H. Venkateswara, and B. Li, “Patchrot: Self- supervised training of vision transformers by rotation predic- tion,” in35th British Machine Vision Conference 2024, Glasgow, UK, November 25-28, 2024

    2024

  32. [32]

    Vision transformer for small- size datasets,

    S. H. Lee, S. Lee, and B. C. Song, “Vision transformer for small- size datasets,”arXiv preprint arXiv:2112.13492 (2021), 2021

    Pith/arXiv arXiv 2021

  33. [33]

    How to train vision trans- former on small-scale datasets?

    H. Gani, M. Naseer, and M. Yaqub, “How to train vision trans- former on small-scale datasets?” in33rd British Machine Vision Conference 2022, London, UK, 2022

    2022

  34. [34]

    Efficient training of visual transformers with small datasets,

    Y. Liu, E. Sangineto, W. Bi, N. Sebe, B. Lepri, and M. Nadai, “Efficient training of visual transformers with small datasets,” Advances in Neural Information Processing Systems, vol. 34, pp. 23 818–23 830, 2021

    2021

  35. [35]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint, vol. arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  36. [36]

    Mobilenetv2: Inverted residuals and linear bottlenecks,

    M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4510–4520, 2018

    2018

  37. [37]

    Tiny imagenet visual recognition chal- lenge,

    Y. Le and X. S. Yang, “Tiny imagenet visual recognition chal- lenge,”CS 231N, vol. 7, no. 7, 2015

    2015

  38. [38]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, and et al, “Pytorch: An imperative style, high-performance deep learning library,”In Proceedings of the Advances in neural informa- tion processing systems, vol. 32, 2019

    2019

  39. [39]

    mixup: Beyond empirical risk minimization,

    H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,”arXiv preprint arXiv:1710.09412, 2017

    Pith/arXiv arXiv 2017

  40. [40]

    Cutmix: Regularization strategy to train strong classifiers with localizable features,

    S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,”In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  41. [41]

    Random erasing data augmentation,

    Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,”In Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

  42. [42]

    Randaugment: Practical automated data augmentation with a reduced search space,

    E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,”In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2020

    2020

  43. [43]

    Autoaugment: Learning augmentation policies from data,

    E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation policies from data,”In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019

    2019

  44. [44]

    Deep networks with stochastic depth,

    G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger, “Deep networks with stochastic depth,”Computer Vision – ECCV 2016, vol. 9908, 2016

    2016

  45. [45]

    Decoupled weight decay regular- ization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regular- ization,”In Proceedings of the International Conference on Learning Representations, 2017

    2017

  46. [46]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,”In Proceedings of the IEEE Inter- national Conference on Computer Vision (ICCV), 2017

    2017