Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classification

Kunal Kawadkar

REVIEW 4 major objections 5 minor 8 references

Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No single architecture wins across medical imaging, this study finds.

desk verdict A clean but statistically weak benchmark scan; the claimed rankings rest on a single 80/20 validation split and do not survive resampling. read the letter →

arxiv 2507.21156 v1 pith:H24EAICG submitted 2025-07-24 eess.IV cs.CVcs.LG

Kunal Kawadkar This is my paper · ORCID

classification eess.IVcs.CVcs.LG

keywords VisionTransformersConvolutionalNeuralNetworksMedicalImageClassificationTransferLearningChestX-rayBrainMRIDermoscopyModelComparison

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether Vision Transformers or convolutional networks are better for medical image classification and answers that neither wins outright. Across three modalities totaling 8,469 images, it reports that ResNet-50 reaches 98.37% accuracy on chest X-ray pneumonia detection, DeiT-Small reaches 92.16% on brain tumor MRI classification, and EfficientNet-B0 reaches 81.84% on skin lesion melanoma detection. The takeaway the authors argue for is that architecture choice should be task-specific rather than universal, with transformers favoured on small neuroimaging sets and CNNs remaining competitive on radiographs and dermoscopy. A reader should care because these results, if reproducible, give concrete guidance on which of four off-the-shelf architectures to start from for each imaging modality.

What carries the argument

The central instrument is a controlled fine-tuning benchmark: four representative architectures—residual CNN (ResNet-50), compound-scaled CNN (EfficientNet-B0), standard patch-based transformer (ViT-Base), and distillation-trained transformer (DeiT-Small)—each initialised with ImageNet weights and trained with identical hyperparameters (Adam, learning rate 1e-4, batch size 32, early stopping patience 3) on an 80/20 stratified split of each dataset. The comparative accuracy table that results is the load-bearing object; the paper uses it to attribute performance differences to architecture rather than to training protocol.

What would settle it

Run the same four models on the same three datasets with patient-level five-fold cross-validation, pooling predictions across folds, and compare the rank ordering to the paper's Table II; if DeiT-Small no longer beats EfficientNet-B0 and ResNet-50 on brain MRI by a clear margin, the central claim of task-specific advantages collapses to split noise. A simpler calculation is a confidence interval on the 92.16% accuracy with roughly 50 validation examples, which will show whether the reported gap is within chance.

Watch

Extended reading notes

Core claim

The paper's central claim is that the best architecture for a medical imaging task is determined by the modality and dataset size, not by a single dominant model family. Using four ImageNet-pretrained models—ResNet-50, EfficientNet-B0, ViT-Base, and DeiT-Small—fine-tuned with the same protocol, it reports task-specific winners: ResNet-50 for chest X-rays (98.37%), DeiT-Small for brain MRIs (92.16%), and EfficientNet-B0 for dermoscopic skin lesions (81.84%). It further claims that on the small brain tumor set (253 images) Vision Transformers as a group clearly outperformed CNNs (89.22% vs 72.55% average accuracy), while on the two larger datasets CNNs held the edge. The authors present this as evidence that the field should move from universal model selection to task-specific architecture selection in clinical decision support.

Load-bearing premise

The rankings stand or fall on the assumption that an 80/20 validation split with early stopping gives reliable accuracy estimates, an assumption that is especially fragile for the brain-tumor dataset where about 50 validation images support the 92.16% figure.

Editorial extensions

If this is right

If the results are right, a deployment team should not assume transformers will replace CNNs in medical imaging; for chest X-ray pneumonia screening a ResNet-50 is the better starting point.
For small, feature-subtle neuroimaging datasets, a distilled transformer like DeiT-Small is the better first choice.
For resource-constrained multi-modal systems, EfficientNet-B0 offers the best accuracy-per-parameter and shortest training time.
Average accuracy across modalities is not a reliable guide: architecture choice should be made per modality, using the reported per-task rankings.
The reported variance differences (ViT lower than CNN across tasks) suggest transformers give more stable cross-modality performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: an 80/20 split with early stopping on validation accuracy is unlikely to produce stable rankings on the 253-image brain MRI set, where the validation arm contains roughly 50 images; the 92.16% figure should be treated as seed-dependent until reproduced with patient-level cross-validation.
Editorial extension: the paper compares only accuracy and efficiency, so an unstated but testable consequence is that calibration and uncertainty differ across the four models; a practitioner could check whether DeiT-Small's confident predictions are better calibrated on small neuroimaging sets.
Editorial extension: because the datasets are public and the protocol is fully specified, the fastest test of the paper's claim is an independent replication run with the same checkpoints but five-fold cross-validation; one would expect the DeiT-Small brain-tumor margin to shrink or vanish if it is split noise.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a straightforward benchmark scan of four ImageNet-pretrained models on three public medical datasets. It does not establish the task-specific rankings it claims, because every number in Table II is a validation accuracy from a single 80/20 split, with early stopping on that same validation set. The brain-tumor result in particular is within binomial noise: roughly 51 validation images can't separate 92.16% from 84.31%. That said, the paper is clearly written, uses public data and standard architectures, and the limitation section does acknowledge the small datasets. The comparative question is legitimate, though Matsoukas et al. (cited) already posed it.

The real soft spots are the evaluation design, not the writing. There is no held-out test set and no error bars. Section III.D claims statistical significance was assessed, but no p-value, confidence interval, or test statistic appears anywhere. The caption of Figure 6 calls DeiT-Small's brain-tumor classification 'perfect' while reporting 92.16% accuracy; that inconsistency suggests the narrative ran ahead of the numbers. Early stopping on validation accuracy also biases model selection toward the split at hand, so the ranking can flip under resampling. These are load-bearing flaws: the conclusion is a set of rankings, and the evidence is a single noisy draw.

What's genuinely useful: the comparison is simple and reproducible, and the computational efficiency table gives a rough sense of training times and parameter counts. A practitioner might use it as a sanity check. But as a scientific claim, it doesn't hold up.

This paper is for readers who want a quick, low-stakes comparison of common architectures on three tasks. It is not for anyone making clinical decisions, and it should not be published as-is. A serious referee would catch the issues immediately, so I would desk reject rather than spend reviewer time. If the authors add a proper test set, multiple seeds, and significance testing, the benchmark could be a minor but acceptable contribution.

Referee Report

4 major / 5 minor

Summary. This manuscript compares four pretrained architectures (ResNet-50, EfficientNet-B0, ViT-Base, DeiT-Small) on three public medical imaging classification tasks: chest X-ray pneumonia detection, brain tumor MRI classification, and skin cancer dermoscopy. Using a single stratified 80/20 train-validation split, early stopping on validation accuracy, and ImageNet-initialized fine-tuning, it reports validation accuracies per task and concludes that architecture choice should be task-specific, with ResNet-50 best on chest X-rays, DeiT-Small best on brain MRI, and EfficientNet-B0 best on skin lesions. The paper also reports training times and parameter efficiency as secondary contributions.

Significance. If the reported rankings were established with proper statistical rigor, the paper would offer useful practical guidance for medical imaging practitioners, especially given its use of standard public datasets and reproducible pretrained models. The computational efficiency analysis is a useful addition. However, because all conclusions rest on a single validation split without a test set or uncertainty quantification, the contribution is currently not substantiated; the methodological issues would need to be addressed before the findings can inform architecture selection.

major comments (4)

[Section III.C, Table II] The central claim of task-specific advantages rests entirely on validation accuracies from a single 80/20 stratified split with early stopping on the same validation data. For the brain tumor dataset (253 images), the validation set contains about 51 images, so DeiT-Small's 92.16% corresponds to roughly 47/51 correct, with an exact binomial 95% confidence interval of approximately 81% to 98%. This interval includes EfficientNet-B0's 84.31% and is close to ViT-Base's 86.27%, so the reported superiority of DeiT-Small on this task is not statistically distinguishable from its competitors. The manuscript should use a proper held-out test set, k-fold cross-validation, or repeated split-seed runs, and report confidence intervals or significance tests for every ranking.
[Section III.D vs Section IV.D] The Methods state that 'Statistical significance was assessed across multiple metrics to ensure robust comparisons,' but no p-value, confidence interval, or test statistic appears anywhere in Section IV. The 'Statistical Analysis' paragraph reports only descriptive statistics (averages and standard deviations across the three tasks). Either report the actual significance tests or retract the claim; as written, the assertion is unsupported.
[Section IV.B.2 and Figure 6] DeiT-Small is described as achieving 'perfect classification performance' and the Figure 6 caption says 'perfect classification (92.2% accuracy).' With about 51 validation images, 92.16% accuracy means roughly four misclassifications, which is not perfect. This internal inconsistency indicates that the narrative conclusions are not tightly coupled to the measured numbers and should be corrected.
[Section IV.D] The claim that ViTs showed 'significantly lower variance' (7.82% vs 13.83%) is based on the standard deviation of only three task accuracies per model family, not on multiple runs; it is not a statement about training stability and it is not backed by any statistical test. Similarly, the '4.01% higher average accuracy' is not tested for significance. These statements should be rephrased as descriptive observations or supported with proper repeated-run statistics.

minor comments (5)

[Section II.B, Reference [7]] The text says 'Chen et al. demonstrated competitive performance of ViTs on chest X-ray classification [7],' but reference [7] is a self-supervised context-restoration paper, not a ViT chest X-ray study. Please verify and correct the citation or the claim.
[Section III.C] The authors state that 'consistent random seeds' were used, but the seed values are not reported. Please provide them or state that they are available on request.
[Section V.C] The Limitations subsection enumerates several limitations but does not mention the absence of a held-out test set or the reliance on a single validation split, which is the most consequential methodological limitation.
[Table III] The 'Parameter Efficiency' metric (average accuracy divided by parameters in millions) mixes units and should be defined more precisely or replaced with a standardized efficiency measure.
[Section IV.B.1] The claim that ResNet-50 shows 'the most balanced performance' on chest X-rays is not substantiated by any numeric confusion-matrix analysis; consider reporting per-class precision and recall.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is a straightforward empirical benchmark with standard pretrained models and public datasets; reported accuracies are measured, not derived from the assumptions.

full rationale

This paper performs transfer-learning experiments with four standard architectures (ResNet-50, EfficientNet-B0, ViT-Base, DeiT-Small) on three public medical imaging datasets. There is no analytic derivation whose conclusion is built into its premises. The architecture choices and dataset splits are external inputs rather than outputs of the analysis, and the reported accuracies are directly measured on a validation split. The only mild concern is that early stopping uses validation accuracy and the same 80/20 validation split is then reported as the result, which is a model-selection/statistical-inference issue rather than circularity: it does not make the accuracy numbers equivalent to the experimental configuration by construction. Likewise, the manuscript states that statistical significance was assessed, but no p-values or confidence intervals are shown; this is missing support for a claim, not a circular derivation. No self-citation is load-bearing, and no uniqueness theorem, ansatz, or renamed known result appears. The paper therefore has essentially zero circularity burden.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical constructs or entities. The central claim rests entirely on empirical benchmarks, and the main assumptions are about the validity of validation accuracy as a generalization measure, the appropriateness of accuracy as a metric on an imbalanced dataset, the absence of patient-level leakage, and the transferability of ImageNet features. The free parameters are training and dataset-selection choices that directly influence the reported accuracy numbers.

free parameters (4)

Skin cancer balanced subset size = 3,000 images (1,500 melanoma, 1,500 benign)
Hand-selected from HAM10000 to avoid class imbalance; affects the reported skin cancer accuracies and any cross-dataset comparisons.
Validation split ratio = 80/20 stratified split
Hand-chosen; with roughly 50 validation images in the brain tumor set, the reported accuracies have wide confidence intervals.
Early stopping patience = 3 epochs
Chosen by hand; with tiny datasets, early stopping on validation accuracy can select models that overfit the validation set.
Learning rate = 1e-4
Chosen by hand; standard for fine-tuning, but no sensitivity analysis is provided.

assumptions (4)

domain assumption Validation accuracy on the 20% held-out split is a valid estimate of generalization performance
The paper reports only validation accuracy (Section IV) and treats it as the model's reported performance, without a separate test set.
domain assumption Accuracy is an appropriate evaluation metric for the imbalanced chest X-ray dataset
The chest X-ray set is 74.3% pneumonia; a majority-class classifier would score 74.3% accuracy, but no baseline or precision/recall/F1 is reported despite the Evaluation Metrics section claiming these metrics.
domain assumption No patient-level data leakage in the random stratified split
Splits are described only as '80/20 stratified sampling'; no statement that images from the same patient are kept in the same split, which matters for the 253-image brain tumor set.
domain assumption ImageNet pretrained weights transfer effectively to medical imaging tasks
All models are initialized with ImageNet weights and fine-tuned; the paper credits this standard practice without validating it for these datasets.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classification." pith.science (2026). https://pith.science/paper/H24EAICG

@misc{pith2026250721156,
  author       = {Pith},
  title        = {Pith review of: Comparative Analysis of Vision Transformers and Convolutional Neural Networks for Medical Image Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H24EAICG}},
  note         = {Machine review of arXiv:2507.21156}
}

read the original abstract

The emergence of Vision Transformers (ViTs) has revolutionized computer vision, yet their effectiveness compared to traditional Convolutional Neural Networks (CNNs) in medical imaging remains under-explored. This study presents a comprehensive comparative analysis of CNN and ViT architectures across three critical medical imaging tasks: chest X-ray pneumonia detection, brain tumor classification, and skin cancer melanoma detection. We evaluated four state-of-the-art models - ResNet-50, EfficientNet-B0, ViT-Base, and DeiT-Small - across datasets totaling 8,469 medical images. Our results demonstrate task-specific model advantages: ResNet-50 achieved 98.37% accuracy on chest X-ray classification, DeiT-Small excelled at brain tumor detection with 92.16% accuracy, and EfficientNet-B0 led skin cancer classification at 81.84% accuracy. These findings provide crucial insights for practitioners selecting architectures for medical AI applications, highlighting the importance of task-specific architecture selection in clinical decision support systems.

Figures

Figures reproduced from arXiv: 2507.21156 by the authors.

**Figure 2.** Comprehensive model comparison across all datasets showing: (a) average accuracy by model type and dataset, (b) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Chest X-ray pneumonia detection results showing: (a) model accuracy comparison, (b) training time analysis, (c) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Confusion matrices for chest X-ray pneumonia detection across all four models. ResNet-50 achieved the highest accuracy [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Brain tumor detection results demonstrating: (a) model accuracy comparison showing Vision Transformer superiority, (b) [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Confusion matrices for brain tumor classification showing dramatic performance differences between architectures. DeiT [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Skin cancer melanoma detection results showing: (a) model accuracy comparison with EfficientNet-B0 leading, (b) [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Confusion matrices for skin cancer melanoma detection showing the challenging nature of dermoscopic image [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 6 canonical work pages

[1]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszko- reit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” International Conference on Learning Representations, 2021

work page 2021
[2]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016

work page 2016
[3]

EfficientNet: Rethinking model scaling for convolu- tional neural networks,

M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolu- tional neural networks,” International Conference on Machine Learning, pp. 6105–6114, 2019

work page 2019
[4]

Training data-efficient image transformers & distillation through attention,

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J´egou, “Training data-efficient image transformers & distillation through attention,” International Conference on Machine Learning , pp. 10347– 10357, 2021

work page 2021
[5]

CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,

P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, and A. Y . Ng, “CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,” arXiv preprint arXiv:1711.05225 , 2017. 9

arXiv 2017
[6]

Dermatologist-level classification of skin cancer with deep neural networks,

A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, pp. 115–118, 2017

work page 2017
[7]

Self-supervised learning for medical image analysis using image context restoration,

L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueck- ert, “Self-supervised learning for medical image analysis using image context restoration,” Medical Image Analysis , vol. 58, p. 101539, 2019

work page 2019
[8]

Is it time to replace CNNs with transformers for medical images?,

C. Matsoukas, J. Haslum, M. S ¨oderberg, and K. Smith, “Is it time to replace CNNs with transformers for medical images?,” arXiv preprint arXiv:2108.09038, 2021

arXiv 2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszko- reit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” International Conference on Learning Representations, 2021

work page 2021

[2] [2]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016

work page 2016

[3] [3]

EfficientNet: Rethinking model scaling for convolu- tional neural networks,

M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolu- tional neural networks,” International Conference on Machine Learning, pp. 6105–6114, 2019

work page 2019

[4] [4]

Training data-efficient image transformers & distillation through attention,

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J´egou, “Training data-efficient image transformers & distillation through attention,” International Conference on Machine Learning , pp. 10347– 10357, 2021

work page 2021

[5] [5]

CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,

P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, and A. Y . Ng, “CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,” arXiv preprint arXiv:1711.05225 , 2017. 9

arXiv 2017

[6] [6]

Dermatologist-level classification of skin cancer with deep neural networks,

A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, pp. 115–118, 2017

work page 2017

[7] [7]

Self-supervised learning for medical image analysis using image context restoration,

L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueck- ert, “Self-supervised learning for medical image analysis using image context restoration,” Medical Image Analysis , vol. 58, p. 101539, 2019

work page 2019

[8] [8]

Is it time to replace CNNs with transformers for medical images?,

C. Matsoukas, J. Haslum, M. S ¨oderberg, and K. Smith, “Is it time to replace CNNs with transformers for medical images?,” arXiv preprint arXiv:2108.09038, 2021

arXiv 2021