Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noise

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under symmetric label noise, the large-patch ViTl32 sustains higher accuracy and calibration than smaller ViTs and Swin transformers while using less compute than ViTl16.

desk verdict A useful but under-powered empirical benchmark; the ViTl32-over-ViTl16 result is plausible yet not proven by single-run, fixed-protocol numbers. read the letter →

arxiv 2505.04375 v1 pith:ME2NW3RE submitted 2025-05-07 cs.CV cs.AIcs.LG

Moseli Mots'oehli , Hope Mogale , Kyungim Baek This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords VisionTransformerActiveLearningLabelNoiseSymmetricModelCalibrationBrierScorePatchSizeEfficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how model size and patch size change a vision transformer's behavior when training labels are noisy and the labeling budget is small. By injecting symmetric label noise at rates from 0% to 90% into CIFAR-10 and CIFAR-100 active-learning runs, it compares four ViT configurations (base/large, 16×16 and 32×32 patches) and three SwinV2 configurations. The intended claim is that the largest ViT configuration with the larger patch, ViTl32, is the most robust and efficient choice: it beats smaller ViTs on accuracy and calibration under moderate to high noise, while ViTl16 buys no accuracy with its extra tokens and higher compute. The paper also argues that SwinV2 models lag behind ViTs under noise, and that information-based active-learning strategies help accuracy only at moderate noise while harming calibration relative to random selection at high noise. If these results hold, they give practitioners a concrete model-selection rule for low-budget, noisy-label fine-tuning.

What carries the argument

The load-bearing comparison is the model grid: four ViT configurations that vary embedding dimension (768 base vs 1024 large) and patch size (16×16 vs 32×32), plus three SwinV2 configurations, all fine-tuned from ImageNet-1k under one shared protocol (20 epochs, early stopping, batch size 256) while symmetric label noise is injected into the training set at rates 0.0–0.9. The mechanism that carries the argument is the interaction between token count (patch size) and model capacity (embedding/layer count), measured through top-1 accuracy, Brier score for calibration, and per-cycle training time. Three acquisition strategies — random, entropy, and the ViT-specific GCI_ViTAL — are used to show that query strategy changes the accuracy/calibration trade-off less than model choice does under noise.

What would settle it

Re-run the same experiments with per-model hyperparameter optimization and multiple random seeds, then check whether ViTl32 still beats ViTl16 on both accuracy and Brier score by a margin larger than the run-to-run spread; if not, the claimed ranking rests on the fixed training recipe.

Watch

Extended reading notes

Core claim

The paper's central discovery is a model-size ranking under symmetric label noise: within the ViT family, capacity in embedding dimension matters more than token density, so ViTl32 (1024-dim embedding, 32×32 patches) consistently beats ViTl16 (same embedding, 16×16 patches) and both base ViTs in top-1 accuracy and Brier-score calibration across noise rates on CIFAR-10 and CIFAR-100, even as noise reaches 90%. ViTl16's smaller patch creates more tokens and roughly two and a half times the training time of ViTl32 without improving results. SwinV2 transformers train at comparable speed but remain less accurate and less calibrated under noise. When active-learning acquisition is varied, random selection is the calibration-safe baseline: entropy and GCI_ViTAL improve accuracy only in the 30–60% noise band and degrade calibration at high noise. The paper states this as evidence that choosing the largest, highest-capacity model is not automatically the best strategy; the best practical choice is the one that balances accuracy, calibration, and compute.

Load-bearing premise

The ranking assumes that one fixed fine-tuning recipe with a single run per setting is enough to compare models fairly, so if a different learning rate or schedule favored ViTl16, the ordering could change.

Editorial extensions

If this is right

  • A practitioner with a low labeling budget and noisy annotations should prefer ViTl32 over ViTl16, since it matches or beats the smaller-patch model while cutting training time.
  • Designers should not assume smaller patches are always better: the 32×32 large ViT dominates the 16×16 large ViT across noise levels.
  • Choosing an information-based acquisition strategy such as entropy or GCI_ViTAL is only justified in the moderate-noise band (roughly 30–60%); at high noise it sacrifices calibration for little accuracy gain.
  • SwinV2 hierarchies are not a drop-in robustness upgrade over plain ViTs under symmetric noise; they offer comparable speed but weaker accuracy and calibration.
  • The data show size differences persist at 70–90% noise, especially on CIFAR-100, so extreme noise does not fully erase model-capacity effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The ViTl32 advantage may be partly a regularization effect of fewer tokens and larger patches under noise, rather than raw capacity; a controlled ablation that keeps embedding size fixed while varying patch size could separate these causes.
  • Inference: Because the shared fine-tuning protocol was not tuned per model, per-model hyperparameter optimization might change the ranking; testing with tuned schedules would tell whether ViTl32's dominance is robust.
  • Inference: The calibration penalty of information-based acquisition suggests a natural extension: acquisition functions that optimize a proper scoring rule or expected calibration error, rather than accuracy alone, may preserve both gains at high noise.
  • Inference: The results were obtained on 32×32 natural images resized to 224; on higher-resolution or domain-shifted data where local detail matters, smaller patches may regain their usual advantage, so the ViTl32 result should be tested beyond CIFAR.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper experimentally investigates how Vision Transformer (ViT) and Swin Transformer configurations of different model sizes and patch sizes behave under symmetric label noise in a deep active learning setting. The authors fine-tune four ViTs (base/large, 16x16/32x32 patches) and three SwinV2 variants on CIFAR10 and CIFAR100, using random, entropy-based, and GCI_ViTAL acquisition strategies at noise rates from 0% to 90%. They report top-1 accuracy, Brier score, and training time, and conclude that larger ViTs, especially ViTl32, outperform smaller counterparts in accuracy and calibration even under moderate to high label noise; that ViTl16 is inefficient relative to ViTl32; that SwinV2 models are less robust; and that information-based active learning yields only modest accuracy gains at moderate noise while worsening calibration at high noise.

Significance. The question addressed is practically relevant: selecting a transformer model for fine-tuning under label noise and limited labeling budgets is a common real-world problem, and the paper provides a broad empirical sweep across model sizes, patch sizes, noise rates, and acquisition strategies. The inclusion of calibration (Brier score) and training time alongside accuracy is a strength, as is the explicit evaluation of active learning beyond the usual random/entropy baselines. If the findings are confirmed with proper statistical support, the recommendation to prefer ViTl32 over ViTl16 under these constraints and the caution about information-based strategies at high noise would be useful guidance. However, the current single-run protocol and the absence of uncertainty quantification mean that the headline ranking is not yet established; the paper also contains a technical error in its complexity analysis of SwinV1 that needs correction.

major comments (4)
  1. [§5.1, Tables 2 and 3] All accuracy and Brier-score results are reported as single-run values without error bars or significance tests. The headline model ranking rests on small differences, for instance ViTl32 exceeds ViTl16 by 0.42 percentage points on CIFAR10 and 0.60 percentage points on CIFAR100 at zero noise; these gaps are within the variation expected across random seeds and DAL cycles. The claim that ViTl32 'consistently outperforms' its smaller counterpart is therefore not statistically supported in the present form. The authors should provide repeated-seed averages with standard deviations or confidence intervals, and ideally significance tests, for the main model-size and patch-size comparisons.
  2. [§4.2, §5.1] The comparison relies on a single shared fine-tuning protocol: 20 epochs, early stopping with a 10-epoch tolerance, batch size 256, and an unspecified learning-rate schedule. Because ViTl16 sees four times as many tokens as ViTl32 (196 vs. 49 per image), a fixed epoch budget can systematically under-train the token-heavy model, so the observed ranking may reflect an artifact of the training protocol rather than a property of the architectures. The paper should either tune hyperparameters per model on a validation split, report learning curves showing that all models reach convergence, or conduct a sensitivity analysis over training budgets; the validation split used for early stopping is also not described.
  3. [§3.1, Eqs. (3)–(6)] The complexity analysis misrepresents SwinV1: Equation (3) attributes O(N^2) complexity to SwinV1 self-attention, but SwinV1 already uses windowed self-attention with per-window complexity O(W^2) and overall complexity O(N·W^2), not full quadratic complexity in the number of tokens N. Equation (6)'s expression O(N·W'^2) is essentially the same windowed-attention complexity, not a new linear-attention mechanism. This technical error undermines the motivation for choosing SwinV2 and should be corrected or the passage removed.
  4. [§5.1, Tables 4 and 5; §6] The abstract and conclusion state that information-based active learning strategies provide 'meaningful accuracy improvements' at moderate label noise rates, but the measured improvements over random are at most 0.38 percentage points on CIFAR10 and 0.48 percentage points on CIFAR100, with several negative entries in the moderate noise range. Without repeated runs and significance testing, these differences are not distinguishable from run-to-run noise. The claim should be qualified or supported with statistical evidence.
minor comments (5)
  1. [Tables 4 and 5] The captions of Tables 4 and 5 contain the typo 'Secion'; it should be 'Section'.
  2. [References, [14]] Reference [14] lists the first author as Kolesnikov, but 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale' is normally attributed to Dosovitskiy et al.; please correct the author order and the citation.
  3. [§2.2] The sentence citing reference [27] as a previous work that adopts ViTs for DAL appears mismatched, since [27] is a survey of transformers in medical image analysis and does not describe an active-learning query strategy; please revise the sentence or cite an appropriate primary source.
  4. [§3.1, Eq. (7)] The notation for the concatenation of attention heads is ambiguous: the symbol with a superscript h is not defined as concatenation; please introduce standard notation explicitly.
  5. [Appendix figures] The appendix figures (Figures 5–15) are not referenced in the main text; add pointer sentences where the corresponding results are discussed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the main ViTl32-vs-ViTl16 comparisons come from new, externally grounded benchmark runs rather than from fitting or from the authors' prior results.

full rationale

The paper is an empirical benchmark study rather than a derivation chain, so most circularity patterns do not apply. The central claim, that ViTl32 balances accuracy, calibration, and efficiency better than ViTl16 and SwinV2 under symmetric label noise, is supported by new experimental results reported in Tables 2-9 using standard ImageNet-pretrained ViT and SwinV2 configurations, standard CIFAR10/CIFAR100 benchmarks, and standard Top-1 accuracy and Brier score metrics. No parameter in the paper is fitted to the headline outcome, and no equation reduces a claimed prediction to an input by construction. The paper does reuse GCI_ViTAL from the authors' prior work [13] and cites [13] as motivation for studying ViTs under label noise, but these self-citations are not load-bearing for the main model-size comparison: GCI_ViTAL is one of three query strategies, and the results about it are experimental observations from the same benchmark, not derivations from the cited paper. The fixed 20-epoch training protocol, early stopping details, and absence of repeated-seed variance are genuine threats to the validity of the ranking, but they are correctness risks, not evidence of circularity. Therefore no circular step is identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

Empirical benchmark; no free parameters fitted to data and no new postulated entities. The load-bearing inputs are the noise model, the pre-trained weights, and the shared training protocol.

assumptions (5)
  • domain assumption Symmetric label noise with rates 0-0.9 is an adequate proxy for real-world annotation noise.
    The paper only injects symmetric noise and never tests asymmetric or instance-dependent noise, so conclusions may not generalize beyond this noise model.
  • domain assumption ImageNet-1k pre-trained weights are a fair and appropriate starting point for all compared models on CIFAR10/100.
    All models are initialized from ImageNet-1k weights; differing pre-training quality across architectures is not controlled.
  • domain assumption A single shared fine-tuning protocol (20 epochs, early stopping tolerance 10, batch size 256, unspecified LR schedule) is fair across architectures of very different capacity and token count.
    No per-model hyperparameter tuning is reported; if some models need different schedules, rankings could be artifacts of under-training.
  • domain assumption The torchvision implementations of ViT and SwinV2 are correct and directly comparable.
    The paper does not verify implementation details and contains incorrect complexity claims about Swin attention.
  • domain assumption Top-1 accuracy and Brier score are sufficient to characterize generalization and calibration.
    No other calibration metrics or uncertainty analyses are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noise." pith.science (2026). https://pith.science/paper/ME2NW3RE

@misc{pith2026250504375,
  author       = {Pith},
  title        = {Pith review of: Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noise},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ME2NW3RE}},
  note         = {Machine review of arXiv:2505.04375}
}
read the original abstract

Fine-tuning pre-trained convolutional neural networks on ImageNet for downstream tasks is well-established. Still, the impact of model size on the performance of vision transformers in similar scenarios, particularly under label noise, remains largely unexplored. Given the utility and versatility of transformer architectures, this study investigates their practicality under low-budget constraints and noisy labels. We explore how classification accuracy and calibration are affected by symmetric label noise in active learning settings, evaluating four vision transformer configurations (Base and Large with 16x16 and 32x32 patch sizes) and three Swin Transformer configurations (Tiny, Small, and Base) on CIFAR10 and CIFAR100 datasets, under varying label noise rates. Our findings show that larger ViT models (ViTl32 in particular) consistently outperform their smaller counterparts in both accuracy and calibration, even under moderate to high label noise, while Swin Transformers exhibit weaker robustness across all noise levels. We find that smaller patch sizes do not always lead to better performance, as ViTl16 performs consistently worse than ViTl32 while incurring a higher computational cost. We also find that information-based Active Learning strategies only provide meaningful accuracy improvements at moderate label noise rates, but they result in poorer calibration compared to models trained on randomly acquired labels, especially at high label noise rates. We hope these insights provide actionable guidance for practitioners looking to deploy vision transformers in resource-constrained environments, where balancing model complexity, label noise, and compute efficiency is critical in model fine-tuning or distillation.

Figures

Figures reproduced from arXiv: 2505.04375 by the authors.

Figure 1
Figure 1. The key components involved in fine-tuning a transformer under label noise. The aspects we vary in our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. This figure shows the second stage of the GCI_ViTAL query strategy, with C-Core attention vectors from [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Top-1 Accuracy vs. Label Noise Rate averaged over CIFAR10 and CIFAR100 at 13% labeled data. Each [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The Brier Score vs. Noise Rate averaged over both CIFAR10 and CIFAR100 at 13% labeled data proportion. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Top-1 Accuracy vs. Noise Rate on CIFAR10 with 23% labeled data. Each subplot represents a different DAL [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Top-1 Accuracy vs. Noise Rate on CIFAR10 with 49% labeled data. Increasing the labeled data proportion [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Brier Score vs. Noise Rate on CIFAR10 with 23% labeled data. Lower values indicate better calibration. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Brier Score vs. Noise Rate on CIFAR10 with 49% labeled data. Similar to the 13% and 23% label settings, [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Accuracy vs. Noise Rate on CIFAR100 with 23% labeled data. Each subplot shows a different DAL strategy [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Accuracy vs. Noise Rate on CIFAR100 with 49% labeled data. ViT models consistently outperform SwinV2 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Brier Score vs. Noise Rate on CIFAR100 with 23% labeled data. The GCI_VITAL and entropy strategies [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Similar to the trends observed with 13% and 23% labeled data, we see consistent patterns across active [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: This graph shows training times for different model configurations against the labeled data proportion [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Two-dimensional kernel density estimates (KDEs) of test accuracy vs. Brier score for Vision Transformers [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Accuracy vs. Brier Score by active learning strategy on CIFAR100 at 50% label noise. At 50% label [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 23 canonical work pages

  1. [1]

    One-peace: Exploring one general representation model toward unlimited modalities, 2023

    Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities, 2023

    work page 2023

  2. [2]

    Omnivec: Learning robust representations with cross-modal sharing

    Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross-modal sharing. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1225–1237. IEEE Computer Society, 2024

    work page 2024

  3. [3]

    Z. Zong, G. Song, and Y . Liu. Detrs with collaborative hybrid assignments training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6725–6735, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society

    work page 2023

  4. [4]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3992–4003, 2023

    work page 2023

  5. [5]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving,...

    2021

  6. [6]

    LeCun and Y

    Y . LeCun and Y . Bengio.Convolutional Networks for Images, Speech, and Time Series , page 255–258. MIT Press, Cambridge, MA, USA, 1998

    work page 1998

  7. [7]

    P. Ren, Y . Xiao, X. Chang, P. Huang, Z. Li, X. Chen, and X. Wang. A survey of deep active learning. ACM Computing Surveys (CSUR), 54:1 – 40, 2020

    work page 2020

  8. [8]

    Cordeiro and G

    F. Cordeiro and G. Carneiro. A survey on deep learning with noisy labels: How to train your model when you cannot trust on the annotations? In The 33rd SIBGRAPI Conference on Graphics, Patterns and Images , pages 9–16, 11 2020

    work page 2020

Show all 33 references
  1. [9]

    Mots’oehli and K

    M. Mots’oehli and K. Baek. Deep active learning in the presence of label noise: A survey. arXiv preprint arXiv:2302.11075, 2023

    2023 arXiv

  2. [10]

    On the interdependence between data selection and architecture optimization in deep active learning

    Pradeep Bajracharya, Rui Li, and Linwei Wang. On the interdependence between data selection and architecture optimization in deep active learning. Transactions on Machine Learning Research, 2024

    2024

  3. [11]

    An empirical study on the efficacy of deep active learning for image classification, 2022

    Yu Li, Muxi Chen, Yannan Liu, Daojing He, and Qiang Xu. An empirical study on the efficacy of deep active learning for image classification, 2022

    2022

  4. [12]

    Deep active learning: A reality check, 2024

    Edrina Gashi, Jiankang Deng, and Ismail Elezi. Deep active learning: A reality check, 2024. 12 Mots’oehli et al

    2024

  5. [13]

    Mots’oehli and K

    M. Mots’oehli and K. Baek. Gci-vital: Gradual confidence improvement with vision transformers for active learning on label noise. arXiv preprint arXiv:2411.05939, 2024

    2024 arXiv

  6. [14]

    Kolesnikov, A

    A. Kolesnikov, A. Dosovitskiy, D. Weissenborn, G. Heigold, J. Uszkoreit, L. Beyer, M. Minderer, M. Dehghani, N. Houlsby, S. Gelly, T. Unterthiner, and X. Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learni...

    2021

  7. [15]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002, 2021

    2021

  8. [16]

    Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1–3, 1950

    1950

  9. [17]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60:84 – 90, 2012

    2012

  10. [18]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2015

    2015

  11. [19]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016

    2016

  12. [20]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  13. [21]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248–255, 2009

    2009

  14. [22]

    Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Proceedings of the 28th International Conference on Neural Information Processing Systems - V olume 2, NIPS’14, page 3320–3328, Cambridge, MA, USA, 2014. MIT Press

    2014

  15. [23]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597–1607. PMLR, 2020

    2020

  16. [24]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729–9738, 2020

    2020

  17. [25]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. InInternational conference on machine learning, pages 10347–10357. PMLR, 2021

    2021

  18. [26]

    Visual transformer for task-aware active learning

    Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Visual transformer for task-aware active learning. arXiv preprint arXiv:2106.03801, 2021

    2021 arXiv

  19. [27]

    Kelei, G

    H. Kelei, G. Chenand L. Zhuoyuan, R. Islem, Y . Zihao, J. Wen Ji, G. Yang, W. Qian, Z. Junfeng, and S. Dinggang. Transformers in medical image analysis. Intelligent Medicine, 3(1):59–78, 2023

    2023

  20. [28]

    Rotman and R

    G. Rotman and R. Reichart. Multi-task Active Learning for Pre-trained Transformer-based Models. Transactions of the Association for Computational Linguistics , 10:1209–1228, 11 2022

    2022

  21. [29]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1204–1213, 2021

    2022

  22. [30]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  23. [31]

    Crossvit: Cross-attention multi-scale vision transformer for image classification

    Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 347–356, 2021

    2021

  24. [32]

    Transformers meet small datasets

    Ran Shao and Xiao-Jun Bi. Transformers meet small datasets. IEEE Access, 10:118454–118464, 2022

    2022

  25. [33]

    Understanding why vit trains badly on small datasets: An intuitive perspective

    Haoran Zhu, Boyuan Chen, and Carter Yang. Understanding why vit trains badly on small datasets: An intuitive perspective. ArXiv, abs/2302.03751, 2023. 13 Mots’oehli et al. Appendix We include additional plots for accuracy, calibration, and label noise. We vary the data proport...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.