Pith. sign in

REVIEW 3 major objections 5 minor 19 references

An Exploratory Approach Towards Investigating and Explaining Vision Transformer and Transfer Learning for Brain Disease Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper claims a Vision Transformer reaches 94.39% accuracy on a 37-class brain MRI dataset, beating four transfer-learning CNN baselines, and applies five explainability methods to the strongest CNN.

desk verdict Routine ViT comparison on a Bangladesh MRI dataset, undone by a leakage-prone preprocessing pipeline and best-of-five reporting. read the letter →

arxiv 2505.16039 v3 pith:B4LHZRMX submitted 2025-05-21 cs.CV

classification cs.CV
keywords VisionTransformerTransferLearningExplainableAIBrainDiseaseDetectionMRIClassificationSMOTEGradCAM37-class
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to test whether a Vision Transformer (ViT) can beat standard transfer-learning convolutional networks when classifying brain diseases from MRI images. On a 37-class Bangladeshi brain MRI dataset, the authors report that a ViT reaches 94.39% accuracy, ahead of ResNet50V2's 91.41% and the other pretrained CNN baselines. The study also applies five gradient-based explainability methods to show which brain regions drive the best transfer-learning model's decisions. If the ranking holds under careful evaluation, transformer-based models would be a practical alternative for computer-assisted brain disease diagnosis in settings with limited local imaging data.

What carries the argument

The carrying mechanism is the contrast between two learning regimes. ViT splits each 128x128 MRI into 64x64 patches, projects the patches into a 64-dimensional embedding space, adds positional embeddings, and runs them through a transformer encoder with eight layers and four attention heads per layer, ending in an MLP head; its self-attention can relate any patch to any other patch directly. The four transfer-learning baselines start from convolutional backbones pretrained on a large natural-image corpus, with a flatten layer, two dense layers (1024 and 512 units), dropout, and a softmax head, fine-tuned on the MRI data. The paper also uses SMOTE to balance the 37 classes and an augmentation pipeline (flip, rotation, zoom) to expand the training data, and it takes the reported numbers from a ten-percent held-out split.

What would settle it

Run the same five models with SMOTE and augmentation applied only after an 80:10:10 split of the original images, holding all hyperparameters fixed, and compare test accuracies; if ViT's 94.39% lead over ResNet50V2 shrinks or disappears, the original ranking was an artifact of preprocessing order.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a Vision Transformer trained for 400 epochs on a 37-class Bangladeshi brain MRI dataset reaches 94.39% accuracy, 96.51% precision, 96.44% recall, and a 96.38% F1 score, ahead of every transfer-learning CNN tested (ResNet50V2 91.41%, MobileNetV2 88.56%, VGG19 89.05%, VGG16 88.44%). The authors attribute the margin to the transformer's patch-based self-attention, which captures long-range relationships in the image that convolutional filters with local receptive fields miss. Because gradient-based explanation methods do not map cleanly onto the transformer's architecture, the paper applies GradCAM, GradCAM++, LayerCAM, ScoreCAM, and FasterScoreCAM to ResNet50V2 instead, and reports that the heatmaps consistently point to the same brain regions, which it reads as a sign the model is attending to clinically meaningful structures.

Load-bearing premise

The result depends on the test split containing only genuinely unseen images, but this pipeline applies SMOTE and augmentation before the 80:10:10 split, so versions of the same image can appear in both training and test data.

Editorial extensions

If this is right

  • If the reported ranking is correct, ViT should be the model of choice for this 37-class brain MRI task, with ResNet50V2 as the strongest CNN fallback.
  • The roughly three-percentage-point accuracy gap and four-point F1 gap between ViT and ResNet50V2 imply that long-range, patch-level context contributes real signal for distinguishing the 37 lesion classes.
  • The consistent XAI heatmaps across five gradient-based methods on ResNet50V2 imply that the best CNN model is making decisions from localized, reproducible image regions, which is a useful property for clinical review.
  • All five models clear 88% accuracy, so even if ViT is unavailable in a resource-constrained hospital, the transfer-learning baselines still provide a usable diagnostic aid on this dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • ViT was run for 400 epochs with a batch size of 256 while the CNN curves shown stop at about 80 epochs; an ablation that matches training budget across architectures would separate the transformer effect from the extra optimization time.
  • The explainability analysis covers only the best CNN (ResNet50V2); applying attention-based explanation methods to the ViT itself would test whether the transformer's higher accuracy comes from the same or different image regions.
  • The dataset's 37 classes include many closely related lesions, so a per-class confusion analysis, beyond overall accuracy, would show whether ViT's advantage comes from rare or morphologically similar classes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript compares a Vision Transformer (ViT) trained from scratch with four ImageNet-pretrained transfer-learning models (VGG16, VGG19, ResNet50V2, MobileNetV2) on a 37-class brain MRI dataset collected in Bangladesh, reporting that ViT reaches 94.39% accuracy and that five gradient-based XAI methods provide interpretability. The central empirical claims are supported by Table 1, Figure 6, and qualitative XAI visualizations in Figure 7. The paper concludes that ViT significantly outperforms the CNN baselines and that XAI improves model transparency.

Significance. If the evaluation were sound, the study would be a useful benchmark for a local clinical MRI dataset and a practical comparison of ViT against transfer learning under class imbalance, with visual explanations from five XAI methods. The use of a publicly available dataset and the explicit comparative design are strengths, as is the attempt to include interpretability. However, the experimental protocol contains a data-leakage flaw and a best-run selection procedure that invalidate the reported accuracy numbers; without code, checkpoints, or corrected experiments, the main comparative claim is not verifiable in its current form.

major comments (3)
  1. [Section 3.2] SMOTE and the TensorFlow augmentation pipeline (resizing, random flip, rotation, zoom) are applied to the full dataset before the 80:10:10 train/validation/test split, as stated in the final sentence of Section 3.2. This creates data leakage: augmented copies or SMOTE-generated synthetic versions of the same original image can appear in both the training partition and the test partition. Consequently, the test accuracies in Table 1, including the headline 94.39% for ViT, do not measure generalization to unseen scans; they partly measure recognition of near-duplicate images. The split must be performed first, with SMOTE fit and augmentation applied only to the training partition, before any accuracy claim can be accepted.
  2. [Section 4.1] Section 4.1 states that each model was executed five times 'with the optimal results chosen for analysis.' Selecting the best of five runs inflates the reported point estimate and discards variance information, and Table 1 reports no standard deviations, confidence intervals, or significance tests. The statement in Section 4.2 that ViT 'significantly outperformed' the CNN models is therefore not statistically supported. The authors should report the mean and standard deviation over runs, or otherwise justify a non-selected evaluation protocol.
  3. [Section 5 and Abstract] The abstract and conclusion credit the integration of XAI methods with enhancing model transparency, but Section 5 applies the five XAI techniques only to ResNet50V2, not to the ViT model that is the paper's main contribution, and Figure 7 provides only qualitative heatmaps. No quantitative evaluation of explanation quality (e.g., localization, faithfulness, or expert assessment) is presented, so the claim that XAI improves reliability or transparency is not demonstrated. This should be restated as a qualitative exploration or supported by an appropriate evaluation.
minor comments (5)
  1. [Section 3.3] The paragraph beginning 'This study utilizes a range of pre-trained models...' is duplicated, and the architecture description repeats 'The task is effectively modified for brain disease' in consecutive sentences; one copy should be removed.
  2. [Figure 5] The diagram label '16 Patches' is inconsistent with the stated input size of 128x128 and patch size of 64x64, which yields four patches; please correct the figure or the text.
  3. [Figure 7 caption] The caption contains the typo 'Gradinat Based XAI' and should read 'Gradient Based XAI'; similarly, Figure 6 uses the misspelling 'Accurcay'.
  4. [Section 3.2] The phrase 'aver aging the complete dataset' contains a spacing error and should read 'averaging'.
  5. [References] Reference [2] is formatted inconsistently with the other references (author name in all caps, thesis title not in a standard style), and the model name appears variously as 'MobileNetV2' and 'MobilenetV2' throughout the text and Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the reported results are empirical measurements, with separate validity concerns.

full rationale

The central claim (Section 4.2, Table 1) is an empirical measurement, not a derivation. ViT and transfer-learning accuracies are obtained by training on the dataset and evaluating on a split; no equation in the paper defines the reported accuracy in terms of the inputs it is supposed to predict. The only self-citation with author overlap is [17] (Shifat Islam), used in Section 5 as a generic GradCAM/VQA reference; it is not load-bearing. The preprocessing order (SMOTE/augmentation before the 80:10:10 split) and the 'optimal results chosen' from five runs are real threats to the validity of the accuracy numbers, but they are not circular reductions: the test accuracy is not equal by construction to the training accuracy, nor is a fitted parameter renamed as a prediction. The model comparison is internally computed from the same pipeline for all five models. Accordingly, no step in the paper's claimed chain reduces to its own input.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a set of hand-chosen hyperparameters for ViT and the transfer learning models, plus standard but unproven assumptions about ImageNet transfer, SMOTE validity, and augmentation label preservation. No new entities are postulated. The most consequential choice is applying SMOTE and augmentation before the data split, which creates the leakage that undermines the reported accuracy.

free parameters (7)
  • ViT learning rate = 1e-4
    Chosen by hand in Section 4.1; directly affects convergence and final accuracy.
  • Transfer learning learning rate = 1e-5
    Chosen by hand in Section 4.1; used for all four CNN models, affecting their reported performance.
  • ViT batch size = 256
    Chosen in Section 3.3; affects training dynamics and the reported 94.39 percent accuracy.
  • ViT patch size and embedding dimension = 64x64 patches, 64-dimensional embedding
    Chosen in Section 3.3; these architectural choices determine how the ViT processes images and influence the result.
  • ViT transformer layers and attention heads = 8 layers, 4 heads
    Chosen in Section 3.3; capacity of the model is tied to these values.
  • SMOTE oversampling target = mean sample count across classes
    Chosen in Section 3.2; determines the degree of synthetic oversampling and is applied before the split, which affects the leakage.
  • Dropout rates = 0.1 in transformer layers, 0.5 before classification
    Chosen in Section 3.3 to prevent overfitting; influences the final generalization estimate.
assumptions (3)
  • domain assumption ImageNet-pretrained weights provide useful features for brain MRI classification.
    The transfer learning models in Section 3.3 are initialized from ImageNet weights; the paper assumes these features transfer to T1-weighted contrast-enhanced MRI, which is standard practice but not proven in this text.
  • domain assumption SMOTE generates synthetic samples that are valid representatives of the minority classes.
    SMOTE is applied in Section 3.2 to balance the dataset; the paper assumes the synthetic samples are realistic and do not distort class boundaries, which is a known limitation of SMOTE in high-dimensional medical images.
  • domain assumption Data augmentation (flip, rotation, zoom) preserves class labels.
    Augmentation is applied in Section 3.2 before splitting; the paper assumes these transformations do not alter the underlying disease label, which is reasonable for MRI but not explicitly validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Exploratory Approach Towards Investigating and Explaining Vision Transformer and Transfer Learning for Brain Disease Detection." pith.science (2026). https://pith.science/paper/B4LHZRMX

@misc{pith2026250516039,
  author       = {Pith},
  title        = {Pith review of: An Exploratory Approach Towards Investigating and Explaining Vision Transformer and Transfer Learning for Brain Disease Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4LHZRMX}},
  note         = {Machine review of arXiv:2505.16039}
}
read the original abstract

The brain is a highly complex organ that manages many important tasks, including movement, memory and thinking. Brain-related conditions, like tumors and degenerative disorders, can be hard to diagnose and treat. Magnetic Resonance Imaging (MRI) serves as a key tool for identifying these conditions, offering high-resolution images of brain structures. Despite this, interpreting MRI scans can be complicated. This study tackles this challenge by conducting a comparative analysis of Vision Transformer (ViT) and Transfer Learning (TL) models such as VGG16, VGG19, Resnet50V2, MobilenetV2 for classifying brain diseases using MRI data from Bangladesh based dataset. ViT, known for their ability to capture global relationships in images, are particularly effective for medical imaging tasks. Transfer learning helps to mitigate data constraints by fine-tuning pre-trained models. Furthermore, Explainable AI (XAI) methods such as GradCAM, GradCAM++, LayerCAM, ScoreCAM, and Faster-ScoreCAM are employed to interpret model predictions. The results demonstrate that ViT surpasses transfer learning models, achieving a classification accuracy of 94.39%. The integration of XAI methods enhances model transparency, offering crucial insights to aid medical professionals in diagnosing brain diseases with greater precision.

Figures

Figures reproduced from arXiv: 2505.16039 by the authors.

Figure 1
Figure 1. Sample Images of the dataset. of random images from the dataset. These images are classified into 37 distinct categories. The extensive variety of categories in the dataset facilitates the creation of sophisticated diagnostic algorithms. This invaluable resource plays a crucial role in the training and validation of machine learning models that utilize computational techniques in medical imaging to enhance the preci… view at source ↗
Figure 2
Figure 2. Distribution of Classes in the MRI Dataset [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the data preprocessing steps [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Proposed Model Architecture This study utilizes a range of pre-trained models grounded in CNN architectures, such as VGG16, VGG19, ResNet50V2 and MobileNetV2 to enhance classification performance. These models, initially trained on ImageNet, are subsequently fine-tuned…
Figure 5
Figure 5. Figure 5: Vision Transformer Architecture The MSA outputs are derived by concatenating several attention heads, followed by processing through a final linear layer. The ViT’s capacity to collect both local and global information in images renders it appropriate for applications …
Figure 6
Figure 6. Figure 6: Comparison of Training and Validation Loss Curves [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Gradinat Based XAI Visualization of Resnet50V2 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [1]

    Amar Y Jaffar. Combining local and global feature extraction for brain tumor classification: A vision transformer and iresnet hybrid model.Engineering, Technology & Applied Science Research, 14(5):17011–17018, 2024

  2. [2]

    PhD thesis, tilburg university

    IVO V AN DONGEN.COMPARISON OF INDIVIDUAL VISION TRANSFORMERS AND MODEL SOUPS FOR BRAIN TUMOR CLASSIFICATION ON MAGNETIC RESONANCE IMAGES. PhD thesis, tilburg university

  3. [3]

    Combining the transformer and convolution for effective brain tumor classification using mri images

    Mohammed Aloraini, Asma Khan, Suliman Aladhadh, Shabana Habib, Mohammed F Alsharekh, and Muhammad Islam. Combining the transformer and convolution for effective brain tumor classification using mri images. Applied Sciences, 13(6):3680, 2023

  4. [4]

    Enhancing brain tumor detection in mri with a rotation invariant vision transformer.Frontiers in Neuroinformatics, 18:1414925, 2024

    Palani Thanaraj Krishnan, Pradeep Krishnadoss, Mukund Khandelwal, Devansh Gupta, Anupoju Nihaal, and T Sunil Kumar. Enhancing brain tumor detection in mri with a rotation invariant vision transformer.Frontiers in Neuroinformatics, 18:1414925, 2024

  5. [5]

    Explainability of deep neural networks for mri analysis of brain tumors.International journal of computer assisted radiology and surgery, 17(9):1673–1683, 2022

    Ramy A Zeineldin, Mohamed E Karar, Ziad Elshaer, · Jan Coburger, Christian R Wirtz, Oliver Burgert, and Franziska Mathis-Ullrich. Explainability of deep neural networks for mri analysis of brain tumors.International journal of computer assisted radiology and surgery, 17(9):1673–1683, 2022

  6. [6]

    Deep learning-based model with xai for brain tumor classification and segmentation using mri images

    Sakibul Hasan, Maisha Musarrat Nabila, Rijjet Bin Anis, and Raqeebir Rab. Deep learning-based model with xai for brain tumor classification and segmentation using mri images. In2023 IEEE 9th International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE), pages 1–6. IEEE, 2023

  7. [7]

    Exploring alzheimer’s disease prediction with xai in various neural network models

    Hamza Ahmed Shad, Quazi Ashikur Rahman, Nashita Binte Asad, Atif Zawad Bakshi, SM Faiaz Mursalin, Md Tanzim Reza, and Mohammad Zavid Parvez. Exploring alzheimer’s disease prediction with xai in various neural network models. InTENCON 2021-2021 IEEE Region 10 Conference (TENCON), pages 720–725. IEEE, 2021

  8. [8]

    An explainable ai paradigm for alzheimer’s diagnosis using deep transfer learning.Diagnostics, 14(3):345, 2024

    Tanjim Mahmud, Koushick Barua, Sultana Umme Habiba, Nahed Sharmen, Mohammad Shahadat Hossain, and Karl Andersson. An explainable ai paradigm for alzheimer’s diagnosis using deep transfer learning.Diagnostics, 14(3):345, 2024

Show all 19 references
  1. [9]

    Deep transfer learning for brain magnetic resonance image multi-class classification.arXiv preprint arXiv:2106.07333, 2021

    Yusuf Brima, Mossadek Hossain Kamal Tushar, Upama Kabir, and Tariqul Islam. Deep transfer learning for brain magnetic resonance image multi-class classification.arXiv preprint arXiv:2106.07333, 2021

  2. [10]

    Brain mri dataset

    Yusuf Brima, Mossadek Hossain Kamal Tushar, Upama Kabir, and Tariqul Islam. Brain mri dataset. https: //doi.org/10.6084/m9.figshare.14778750.v2, 2021

  3. [11]

    Smote: synthetic minority over-sampling technique.Journal of artificial intelligence research, 16:321–357, 2002

    Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique.Journal of artificial intelligence research, 16:321–357, 2002. 10 APREPRINT- SEPTEMBER22, 2025

  4. [12]

    A study of cnn and transfer learning in medical imaging: Advantages, challenges, future scope.Sustainability, 15(7):5930, 2023

    Ahmad Waleed Salehi, Shakir Khan, Gaurav Gupta, Bayan Ibrahimm Alabduallah, Abrar Almjally, Hadeel Alsolai, Tamanna Siddiqui, and Adel Mellit. A study of cnn and transfer learning in medical imaging: Advantages, challenges, future scope.Sustainability, 15(7):5930, 2023

  5. [13]

    Classification of brain tumor from magnetic resonance imaging using vision transformers ensembling.Current Oncology, 29(10):7498–7511, 2022

    Sudhakar Tummala, Seifedine Kadry, Syed Ahmad Chan Bukhari, and Hafiz Tayyab Rauf. Classification of brain tumor from magnetic resonance imaging using vision transformers ensembling.Current Oncology, 29(10):7498–7511, 2022

  6. [14]

    Keras.https://keras.io, 2015

    François Chollet et al. Keras.https://keras.io, 2015

  7. [15]

    Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunat...

  8. [16]

    Multi-class brain lesion classification using deep transfer learning with mobilenetv3.IEEE Access, 2024

    Ahmed Firas Majeed, Pedram Salehpour, Leili Farzinvash, and Saeid Pashazadeh. Multi-class brain lesion classification using deep transfer learning with mobilenetv3.IEEE Access, 2024

  9. [17]

    A deep learning-based bengali visual question answering system

    Mahamudul Hasan Rafi, Shifat Islam, SM Hasan Imtiaz Labib, SM Sajid Hasan, Faisal Muhammad Shah, and Sifat Ahmed. A deep learning-based bengali visual question answering system. In2022 25th International Conference on Computer and Information Technology (ICCIT), pages 114–119....

  10. [18]

    Fatema Tuj Johora Faria, Mukaffi Bin Moin, Pronay Debnath, Asif Iftekher Fahim, and Faisal Muhammad Shah. Explainable convolutional neural networks for retinal fundus classification and cutting-edge segmentation models for retinal blood vessels from fundus images.arXiv preprin...

  11. [19]

    Explainability and evaluation of vision transformers: An in-depth experimental study.Electronics, 13(1):175, 2023

    Sédrick Stassin, Valentin Corduant, Sidi Ahmed Mahmoudi, and Xavier Siebert. Explainability and evaluation of vision transformers: An in-depth experimental study.Electronics, 13(1):175, 2023. 11

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.