Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Deep Modeling and Optimization of Medical Image Classification

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adapting off-the-shelf CNNs and vision transformers with CLIP text encoders, federated training, and SVM/KNN classifiers yields strong skin- and brain-cancer classification, with maxvit_t best on multimodal HAM10000.

desk verdict A wide benchmark with real utility, but the headline rankings rest on single runs and the abstract contradicts the body. read the letter →

arxiv 2505.23040 v1 pith:DBT65XIX submitted 2025-05-29 cs.CV

Yihang Wu , Muhammad Owais , Reem Kateb , Ahmad Chaddad This is my paper
classification cs.CV
keywords medicalimageclassificationmultimodallearningCLIPfederatedskincancerbrainvisiontransformerSVMgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that existing CNN and vision transformer backbones, combined with three inexpensive adaptations, are enough to handle the constraints of medical image classification: limited data sharing, privacy, and domain shift. It builds a CLIP variant pairing four CNNs and eight ViTs with frozen CLIP text encoders, runs two federated learning schemes so raw images never leave clients, and adds KNN or SVM classifiers on top of deep features. Results are reported for three benchmark tasks: on HAM10000 multimodal learning, maxvit_t reaches the highest averaged test metrics (87.03%); under federated learning, convnext_l achieves the best F1-score (83.98% versus 81.33% for swin_b); and on ISIC2018, SVM improves the swin transformer series by about 2% in AVG. The practical point is that competitive performance is reachable with standard backbones plus tools for privacy and generalization.

What carries the argument

The machinery is the contrastive alignment of image and text features: cosine similarity \(s_{j,c} = \langle I_j^*, T_c\rangle / (\|I_j^*\| \|T_c\|)\) feeds a softmax over classes, and a symmetric contrastive loss over image-text batches trains the image encoder while the text encoder stays frozen. The same image encoder is reused in federated learning, where FedAVG aggregates client weights as \(\omega^t_{\text{glo}} = \frac{1}{M}\sum_{i=1}^{M} \omega_i^t\) and FedProx adds a proximal term, and in the generalization task, where the penultimate deep features are passed to KNN or SVM instead of a linear classifier. This three-part pipeline is what lets 12 backbones be tested under the same protocol.

What would settle it

Rerun each configuration (12 backbones, five optimizers, three text encoders, and both FL methods) with five to ten random seeds and report mean plus or minus one standard deviation; if the best model changes or the top models overlap within one standard deviation, the paper's ranking claims are not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bottleneck in medical image classification is not the backbone but how it is attached to the task: which text encoder it is paired with, how training is federated, and what classifier reads its features. Across 12 pretrained image encoders, the multimodal results show maxvit_t leading on HAM10000 with averaged test metrics (the unweighted mean of accuracy, balanced accuracy, precision, recall, and F1) of 87.03% with a ViT-L/14-pretrained text encoder, and convnext_l leading at 87.36% when the text encoder is pretrained by ResNet50x64. In the federated setting, convnext_l gives the best F1-score (83.98%) compared with swin_b (81.33%), while in the out-of-domain generalization setting, SVM improves the swin transformer series by roughly 2% AVG and both KNN and SVM raise maxvit_t by about 15%. If these results are right, model ordering depends on the integration strategy, not only on the backbone.

Load-bearing premise

The load-bearing premise is that one training run on one random split reliably orders the models; the paper reports no seeds or confidence intervals, so the reported gaps could lie within run-to-run noise.

Editorial extensions

If this is right

  • Practitioners could choose a maxvit-style encoder for multimodal skin-lesion classification and expect competitive accuracy with modest fine-tuning.
  • Federated training with FedAVG or FedProx keeps F1-scores above 81% across backbones, so privacy-preserving training does not force a large accuracy drop.
  • Using SVM on deep features is a cheap generalization lever: it improves swin transformer series by about 2% AVG on ISIC2018 and lifts maxvit_t by about 15%.
  • The winning backbone changes with the CLIP text encoder, so text-encoder pretraining is a first-order choice, not a detail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: since each configuration was run once on one split with no seeds or confidence intervals, the reported rankings are best read as hypotheses; multi-seed runs could reorder adjacent models.
  • Editorial extension: the SVM/KNN gains suggest a modular pipeline where future backbones can be plugged in without retraining the classifier, making the generalization result easy to reproduce and extend.
  • Editorial extension: the large out-of-domain gain for maxvit_t hints that distribution-shift robustness may be driven more by classifier choice than by backbone architecture; a direct test would be swapping SVM for a linear head across several unseen dermatology sets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents an empirical study of 12 deep image backbones (four CNNs and eight ViTs) in three medical image classification scenarios: (1) a CLIP-style multimodal learning setup on HAM10000 and BraTS2019 with three frozen CLIP text encoders; (2) federated skin cancer classification on a partitioned HAM10000 (FHAM) using FedAVG and FedProx with five optimizers; and (3) generalization to the ISIC2018 test set using deep features plus KNN or SVM classifiers. The authors report that maxvit_t achieves the best multimodal AVG on HAM10000 (87.03% in the abstract), convnext_large achieves the best federated F1-score (83.98% vs 81.33% for swin_b), and SVM improves the swin transformer family on ISIC2018 by ~2% AVG. The code is announced as publicly available.

Significance. If the comparative rankings were statistically reliable, this paper would be a useful broad benchmark for practitioners choosing backbones and training protocols for skin and brain tumor classification. The breadth — 12 backbones, multiple tasks, and three text encoders — is a strength, and the stated code release supports reuse. However, the headline findings currently rest on single-run, single-split comparisons and on selection of the maximum over many configurations, so the empirical contribution cannot be considered established. The paper also contains an internal inconsistency between the abstract and the Task 1 results. These issues substantially limit the significance of the study until the experimental protocol is strengthened.

major comments (4)
  1. [Abstract and Section 3.3, Task 1] The abstract's headline claim that maxvit shows the highest averaged test metric (AVG = 87.03%) is internally inconsistent with the Task 1 results: maxvit_t is reported at 87.58% with the ResNet50x16 text encoder, and convnext_large is reported at 87.36% with ResNet50x64. The 'best' model thus depends on the text encoder, and the paper does not explain how 87.03% is derived as an average across encoders. This contradiction must be resolved and the headline claim restated.
  2. [Section 3.3] The experimental protocol reports a single training run per configuration (50 epochs, one data split) with no random seeds, repeated trials, or confidence intervals. The central comparative claims — maxvit_t outperforming other backbones, convnext_large surpassing swin_b (F1 83.98% vs 81.33%), and SVM improving swin-family AVG by ~2% — involve differences comparable to typical run-to-run variation for this type of dataset and architecture. Without variance estimates or a statistical test, these rankings are not supported by the reported evidence.
  3. [Sections 3.2 and 3.3, Tasks 1 and 2] The study selects the maximum performance over 12 backbones × 3 text encoders (Task 1) and over 12 backbones × 5 optimizers × 2 FL algorithms (Task 2), and then reports that maximum as the finding. Under multiple testing, the best of many configurations is expected to be an upward-biased outlier even if all configurations are equal. The paper provides no correction, no hold-out validation of the chosen configuration, and no replication, so the 'best model' conclusions and the claimed ~2% SVM improvement are unreliable.
  4. [Abstract and Section 3.3, Task 3] The abstract claims that SVM improves the overall AVG by ~2% for the swin transformer series on ISIC2018, but the experimental section does not report this per-family result. Task 3 only mentions an ~15% improvement for maxvit_t and states generally that SVM achieves the best overall results. The ~2% figure should be either substantiated with the corresponding results or removed from the abstract.
minor comments (5)
  1. [Title/header] The title contains line-break artifacts ('OPTIMIZA TION', 'CLASSIFICA TION') that should be corrected.
  2. [Section 3.3] The sentence specifying batch sizes is garbled: it says 'Task 2, the batchsize is set to 32 for HAM10000 and 16 for BraTS2019', but Task 2 does not use BraTS2019; Task 1 uses both datasets. Please correct the task assignment.
  3. [Section 3.3] Hyperparameters for the traditional ML classifiers (e.g., k in KNN, kernel and C for SVM) and for the federated learning setup (number of local epochs, proximal term mu, client partition seed) are not reported, which limits reproducibility.
  4. [Figures 3 and 4] The spider plots overlay many series and are hard to read; tables with the exact numerical values for each backbone, text encoder, and optimizer would make the comparisons more transparent.
  5. [Section 3.1] The FHAM dataset split uses a random partition with no seed; please report the exact split or make it available so that the federated experiments can be reproduced.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark whose comparative claims are measurements, not derivations from fitted parameters or self-citations.

full rationale

The paper's central findings—maxvit_t's multimodal AVG of 87.03% on HAM10000, convnext_l's FL F1 of 83.98% versus swin_b's 81.33%, and the ~2% SVM improvement on ISIC2018—are reported results of training and evaluating fixed model configurations on standard public datasets. There is no mathematical derivation whose output is equivalent to its input: the CLIP variant, FedAVG/FedProx, and SVM/KNN pipelines are all standard methods applied as described in Sections 2 and 3. The small number of self-citations (refs. [2], [3], [17]) are contextual or future-work citations and do not carry the load of any ranking claim. The abstract's 87.03% is internally imprecise relative to the body's per-text-encoder bests, and the single-run, single-split protocol in Sec. 3.3 raises statistical reliability concerns about the rankings, but those are correctness/robustness limitations, not circularity. No fitted value is renamed as a prediction, no uniqueness theorem is imported from the authors, and no known result is relabeled as a discovery. The empirical observations stand or fall on experimental validity rather than on any circular derivation chain.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities; its results rest on standard deep learning assumptions, hand-set hyperparameters, and dataset-specific assumptions. The free parameters listed are the main knobs that could change the reported rankings.

free parameters (5)
  • learning rate = 0.001 (Adam/AdamW/Adagrad/Adadelta), 0.01 (SGD)
    Chosen by hand, affects all results; reported in Table 1.
  • weight decay = 0.0005 (SGD/Adagrad/Adadelta), 0.02 (Adam/AdamW)
    Hand-set regularizer influencing convergence.
  • batch size = 16 (Task 1), 32 (Tasks 2/3)
    Set per task without explanation.
  • training epochs = 50
    Fixed for all runs.
  • optimizer selection per task = Adagrad for Task 1, SGD for Task 3
    Chosen without stated justification; affects comparisons.
assumptions (3)
  • domain assumption CLIP text encoder features provide useful semantic alignment for medical image classes
    The method relies on this to compute image-text similarity for classification; borrowed from prior natural-domain CLIP results without medical validation.
  • domain assumption Random split of HAM10000 into three clients is a valid federated learning testbed
    Section 3.1 creates IID clients by random sampling, which does not reflect realistic non-IID medical data distributions.
  • domain assumption Weighted metrics (ACC, BACC, PRE, REC, F1) and their unweighted average are appropriate for the imbalanced skin lesion datasets
    No per-class analysis or class-imbalance adjustment is discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Modeling and Optimization of Medical Image Classification." pith.science (2026). https://pith.science/paper/DBT65XIX

@misc{pith2026250523040,
  author       = {Pith},
  title        = {Pith review of: Deep Modeling and Optimization of Medical Image Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBT65XIX}},
  note         = {Machine review of arXiv:2505.23040}
}
abstract

Deep models, such as convolutional neural networks (CNNs) and vision transformer (ViT), demonstrate remarkable performance in image classification. However, those deep models require large data to fine-tune, which is impractical in the medical domain due to the data privacy issue. Furthermore, despite the feasible performance of contrastive language image pre-training (CLIP) in the natural domain, the potential of CLIP has not been fully investigated in the medical field. To face these challenges, we considered three scenarios: 1) we introduce a novel CLIP variant using four CNNs and eight ViTs as image encoders for the classification of brain cancer and skin cancer, 2) we combine 12 deep models with two federated learning techniques to protect data privacy, and 3) we involve traditional machine learning (ML) methods to improve the generalization ability of those deep models in unseen domain data. The experimental results indicate that maxvit shows the highest averaged (AVG) test metrics (AVG = 87.03\%) in HAM10000 dataset with multimodal learning, while convnext\_l demonstrates remarkable test with an F1-score of 83.98\% compared to swin\_b with 81.33\% in FL model. Furthermore, the use of support vector machine (SVM) can improve the overall test metrics with AVG of $\sim 2\%$ for swin transformer series in ISIC2018. Our codes are available at https://github.com/AIPMLab/SkinCancerSimulation.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 27 canonical work pages

  1. [1]

    These models have shown a remarkable ability to learn complex visual patterns and have outperformed traditional machine learning techniques in both accuracy and scalability

    INTRODUCTION In recent years, deep learning models (DLMs) have signif- icantly advanced medical imaging by using powerful ar- chitectures like convnext series, vision transformer (ViT) and maxvit for efficient deployment in realistic scenarios [1]. These models have shown a remarkable ability to learn complex visual patterns and have outperformed traditio...

    work page

  2. [2]

    We propose a novel CLIP-based approach that considers CNN and ViT architectures with CLIP text encoders for multimodal training, effectively integrating image and text data

    work page

  3. [3]

    We introduce two FL approaches (FedA VG and FedProx) into skin cancer classification to solve data privacy leak- age

    work page

  4. [4]

    Deep Modeling and Optimization of Medical Image Classification

    To improve the generalization ability of deep models, we combine traditional ML with deep models. Specifically, we use ML techniques as classifiers while using deep models as feature extractors. arXiv:2505.23040v1 [cs.CV] 29 May 2025 Fig. 1: Flowchart of the proposed framework. 1) Data acquisition: Image data are preprocessed. 2) Proposed models: This inv...

    work page Pith review arXiv 2025

  5. [5]

    a picture of a {class}

    METHODOLOGY Figure 1 illustrates the flowchart of the proposed model. Specifically, we propose a CLIP variant based on CNN and VIT to extract the image feature and then combine it with the text encoder in CLIP to perform training and inference. In addition, we introduce two FL techniques with four CNNs and eight ViTs as network backbones to solve data pri...

    work page

  6. [6]

    Datasets Federated HAM10000 (FHAM)

    EXPERIMENTS 3.1. Datasets Federated HAM10000 (FHAM). We modified the HAM10000 dataset to build the FL dataset [9]. We split the original train- ing set of HAM10000 into three clients with randomly se- lected samples, while the original testing set is used for global evaluation. In each client, the data are randomly partitioned into three non-overlapping p...

    work page

  7. [7]

    Multimodal learning. HAM10000 and BraTS2019 datasets are used to perform multimodal learning using 12 deep models (four CNNs and eight ViTs) with CLIP text en- coders pretrained by three image encoders (i.e., ViT-L/14, ResNet50x16 and ResNet50x64). The Adagrad optimizer is considered for optimization

    work page

  8. [8]

    We use FHAM dataset as an example to evaluate the usefulness of 12 deep models using two FL techniques (FedA VG and Fed- Prox) with five optimizers

    Federated skin cancer classification. We use FHAM dataset as an example to evaluate the usefulness of 12 deep models using two FL techniques (FedA VG and Fed- Prox) with five optimizers

    work page

Show all 29 references
  1. [9]

    The ISIC2018 test set is used to demonstrate the generalizability of four CNNs and eight ViTs using k-nearest neighbours (KNN) and support vec- tor machine (SVM) [12]

    Generalization analysis. The ISIC2018 test set is used to demonstrate the generalizability of four CNNs and eight ViTs using k-nearest neighbours (KNN) and support vec- tor machine (SVM) [12]. Note that the deep models are pretrained on HAM10000 training set using SGD opti- mi...

  2. [10]

    CONCLUSION This study proposed three models covering multimodal, FL and traditional ML with deep models in medical image clas- sification tasks. The findings suggest that maxvit t shows po- tential for multimodal, convnext l indicates remarkable over- all test metrics using Fe...

  3. [11]

    COMPLIANCE WITH ETHICAL STANDARDS This is a numerical simulation study for which no ethical ap- proval was required

  4. [12]

    ACKNOWLEDGEMENTS This research was funded by the National Natural Sci- ence Foundation of China #82260360, the Guilin Innova- tion Platform and Talent Program #20222C264164, and the Guangxi Science and Technology Base and Talent Project (#2022AC18004 and #2022AC21040)

  5. [13]

    Maxcervixt: A novel lightweight vision transformer-based approach for precise cervical cancer detection,

    I. Pacal, “Maxcervixt: A novel lightweight vision transformer-based approach for precise cervical cancer detection,”Knowledge-Based Sys- tems, vol. 289, p. 111482, 2024

    2024

  6. [14]

    Federated learning for health- care applications,

    A. Chaddad, Y . Wu, and C. Desrosiers, “Federated learning for health- care applications,” IEEE Internet of Things Journal, 2023

    2023

  7. [15]

    Facmic: Federated adaptative clip model for medical image classification,

    Y . Wu, C. Desrosiers, and A. Chaddad, “Facmic: Federated adaptative clip model for medical image classification,” in International Confer- ence on Medical Image Computing and Computer-Assisted Interven- tion, pp. 531–541, Springer, 2024

    2024

  8. [16]

    Communication-efficient learning of deep networks from decen- tralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Ar- cas, “Communication-efficient learning of deep networks from decen- tralized data,” in Artificial intelligence and statistics , pp. 1273–1282, PMLR, 2017

    2017

  9. [17]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational con- ference on machine learning, pp. 8748–8763, PMLR, 2021

    2021

  10. [18]

    Are natural domain foundation models useful for med- ical image classification?,

    J. P. Huix, A. R. Ganeshan, J. F. Haslum, M. S ¨oderberg, C. Matsoukas, and K. Smith, “Are natural domain foundation models useful for med- ical image classification?,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7634–7643, 2024

    2024

  11. [19]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems, vol. 2, pp. 429–450, 2020

    2020

  12. [20]

    Image cyberbullying detection and recognition us- ing transfer deep machine learning,

    A. Almomani, K. Nahar, M. Alauthman, M. A. Al-Betar, Q. Yaseen, and B. B. Gupta, “Image cyberbullying detection and recognition us- ing transfer deep machine learning,”International Journal of Cognitive Computing in Engineering, vol. 5, pp. 14–26, 2024

    2024

  13. [21]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pig- mented skin lesions,

    P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pig- mented skin lesions,” Scientific data, vol. 5, no. 1, pp. 1–9, 2018

    2018

  14. [22]

    Skin le- sion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic),

    N. Codella, V . Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gut- man, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, et al., “Skin le- sion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic),” arXiv prepri...

    2018 arXiv

  15. [23]

    Brain tumor classification dataset

    M. 2019, “Brain tumor classification dataset..” https://www.kaggle.com/datasets/shirtgm/ brats2019-classification-divided-by-patients/ data

    2019

  16. [24]

    A comparative analysis of k-nearest neighbor, genetic, support vector machine, decision tree, and long short term memory algorithms in machine learning,

    M. Bansal, A. Goyal, and A. Choudhary, “A comparative analysis of k-nearest neighbor, genetic, support vector machine, decision tree, and long short term memory algorithms in machine learning,”Decision An- alytics Journal, vol. 3, p. 100071, 2022

    2022

  17. [25]

    Convnext v2: Co-designing and scaling convnets with masked autoen- coders,

    S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co-designing and scaling convnets with masked autoen- coders,” in Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 16133–16142, 2023

    2023

  18. [26]

    Convision benchmark: A contemporary framework to bench- mark cnn and vit models,

    S. Bangalore Vijayakumar, K. T. Chitty-Venkata, K. Arya, and A. K. Somani, “Convision benchmark: A contemporary framework to bench- mark cnn and vit models,” AI, vol. 5, no. 3, pp. 1132–1171, 2024

    2024

  19. [27]

    Survey of optimiza- tion algorithms in modern neural networks,

    R. Abdulkadirov, P. Lyakhov, and N. Nagornov, “Survey of optimiza- tion algorithms in modern neural networks,” Mathematics, vol. 11, no. 11, p. 2466, 2023

    2023

  20. [28]

    Fed- erated large language model: Solutions, challenges and future direc- tions,

    J. Hu, D. Wang, Z. Wang, X. Pang, H. Xu, J. Ren, and K. Ren, “Fed- erated large language model: Solutions, challenges and future direc- tions,” IEEE Wireless Communications, 2024

    2024

  21. [29]

    Enhancing classification tasks through do- main adaptation strategies,

    A. Chaddad and Y . Wu, “Enhancing classification tasks through do- main adaptation strategies,” in 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1832–1835, IEEE, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.